for both $e_{*}$ and $\rho_{*}$. If the predictions are well-calibrated, the
statistics of predictive $z$-scores should resemble a standard normal
distribution. Concretely, the order statistics of a collection of $N_{test}$
$z$-scores should closely match the theoretical quantiles of a standard normal
distribution — a relationship that can be graphically inspected with a QQ-
plot. To compare different models, we visualize QQ-plots for the different
covariance functions in Figure 14. As an additional summary of calibration, we
compare the fraction of predicted examples covered by $1/2,1,2$, and 3
posterior standard deviations, summarized in Table 2. We see that the squared
exponential kernel $z$-scores and empirical coverage are not far from what is
expected by a theoretical standard normal distribution, indicating that the
estimates are well-calibrated.

We also depict model fit statistics as a function of distance from the origin
in Figure 15. In Figures 15(a) and 15(b) show that pointwise predictions are
again well-calibrated as a function of distance, though we observe a small
amount of concentration about $0$ for the extinction predictions. This
indicates that the predictive model was too conservative, or not confident
enough in the posterior expectation at nearby values. This is corroborated by
Figure 15(c), which shows the posterior variance for extinctions is high at
nearby (and distant) values, however MSE are high mostly for distant values.
Good calibration, even at far distances, is encouraging, as the goal is to
form good estimates of this density far away from the observer location.

(a) Test extinctions $e$

(b) Test $\rho(x)$ values

Figure 14: Posterior for extinctions $e$ and dust density $\rho$ are well
calibrated. QQ-plot for predicted distributions on 2,000 held out
integrated-$\rho$ test points $e_{*}$. Theoretical normal quantiles are on the
horizontal axis, and predictive $z$-scored quantiles are on the vertical axis.

| Gneit-$\alpha=1$ | Mat-$\nu=1/2$ | Mat-$\nu=3/2$ | Mat-$\nu=5/2$ | SqExp | $\mathcal{N}(0,1)$
---|---|---|---|---|---|---
$\sigma$ |  |  |  |  |  |
0.5 | 0.536 | 0.560 | 0.441 | 0.407 | 0.384 | 0.383
1.0 | 0.880 | 0.917 | 0.785 | 0.730 | 0.679 | 0.683
2.0 | 0.997 | 0.999 | 0.999 | 0.988 | 0.971 | 0.955
3.0 | 1.000 | 1.000 | 1.000 | 1.000 | 1.000 | 0.997

Table 2: Coverage fractions at various levels of z-score standard deviation
$\sigma$ for test extinctions $e_{*}$. Relative to theoretical coverage
fractions, the inferences concentrate too many predictions near zero — a sign
that we are not confident enough.

(a) Test $e$ z scores by distance

(b) Test $\rho(x)$ z scores by distance

(c) Test $e$ posterior standard deviation by distance

(d) Test $\rho(x)$ posterior standard deviation by distance

(e) Test $e$ values by distance

(f) Test $\rho(x)$ values by distance

Figure 15: Posterior summaries as a function of distance to the origin
(observation point).

##  Appendix G Additional Synthetic Experiments

In this supplemental section, we examine the effect of hyper parameter choices
on algorithm performance. Specifically, we study two choices: (i) the whitened
and standard parameterizations and (ii) the effect of the Monte Carlo semi-
integrated covariance estimator on stochastic variational inference.

###  G.1 Algorithm experiments

The inference algorithm developed in Section 4 involves two key choices,
optimizing in the whitened space and the number of Monte Carlo samples used to
approximate the semi-integrated kernel. To validate our choices, we
investigate the impact of these choices on the inference algorithm and
predictions. For simplicity, we assume the number and position of inducing
points is fixed.

Whitened parameterization comparison We compare the optimization performance
of the two parameterizations detailed above: (i) the _standard_
parameterization of variational parameters
$\boldsymbol{\mathbf{m}},\boldsymbol{\mathbf{S}}$ and covariance function
parameters ${\boldsymbol{\mathbf{\theta}}}$; and (ii) the _whitened_
parameterization of variational parameters
$\tilde{\boldsymbol{\mathbf{m}}},\tilde{\boldsymbol{\mathbf{S}}}$ and
covariance function parameters ${\boldsymbol{\mathbf{\theta}}}$.

We fit the above synthetic dataset with $N=5{,}000$ observations using
stochastic gradient optimization with batches of size $1{,}000$, for 10
epochs. We fix the learning rate for the variational parameters, and we vary
the learning rate for the covariance function parameters to be $\Delta\in$
[1e-4, 1e-5, 1e-6]. We also vary the initial value of $\sigma^{2}$ to be in
$\sigma^{2}_{0}\in[1,10,100,1{,}000]$. We fit 24 models, 12 standard and 12
whitened.

We compare the mean squared errors for the test $e$ and $\rho(x)$ values in
Figure 16, showing that the models fit in the whitened space have lower mean
squared errors. Empirically, we also find that optimizing in the whitened
space leads to better predictions in fewer epochs.

(a) Test $e$ MSEs

(b) Test $\rho(x)$ MSEs

Figure 16: Optimizing with the whitened parameterization leads to better
predictions. We ran stochastic optimization initialized at many different
initial settings (the top five shown above) for ten epochs. The resulting
predictive mean squared error for integrated values $e$ (left) and pointwise
values $\rho(x)$ (right) are plotted above. Optimizing in the whitened space
dramatically improves predictive accuracy.

Semi-Integrated Estimators Additionally, we examine the affect of the number
of Monte Carlo samples used to estimate the semi-integrated covariance
function, $L$. We leverage the fact that the squared exponential kernel admits
a closed form semi-integrated covariance function so we can directly compare
Monte Carlo estimators with different sample sizes $L$ to the analytic version
Further, we use the analytic form of the semi-integrated covariance function
to compute the true variational objective for all approaches; this true
objective value is unavailable when the semi-integrated covariance function is
unavailable. Figure 9 compares optimization traces of the variational
objective using different values of $L$, ranging from 5 to 50. We see that the
performance of the Monte Carlo gradients with $L=20$ closely mimics the model
fit with the exact gradient. We use $L=30$ when optimizing with the Gneiting
and Matérn covariance functions to be conservative.


H})$ $\triangleright$ Computation of reweighted MMCD

20: $\tilde{\alpha}=\nicefrac{{\left|H\right|}}{{n}}$

21:
$(\hat{\bm{\operatorname{M}}}_{\ast},\hat{\bm{\Sigma}}^{\operatorname{row}}_{\ast},\hat{\bm{\Sigma}}^{\operatorname{col}}_{\ast})=(\hat{\bm{\operatorname{M}}},c(\tilde{\alpha})\hat{\bm{\Sigma}}^{\operatorname{row}},\hat{\bm{\Sigma}}^{\operatorname{col}})$
$\triangleright$ Consistency scaling for reweighted MMCD

22: return
$\hat{\bm{\operatorname{M}}}_{\ast},\hat{\bm{\Sigma}}^{\operatorname{row}}_{\ast},\hat{\bm{\Sigma}}^{\operatorname{col}}_{\ast}$

23:end procedure

###  C.1 Elemental subsets

For large $n$, the probability of obtaining at least one clean subset with
$d+2$ observations among $m$ random subsets tends to

$1-(1-(1-\varepsilon)^{d+2})^{m},$

with $\varepsilon$ denoting the percentage of outliers, see also Rousseeuw and
Driessen, (1999). Hence, the number of subsets we must investigate to obtain
at least one clean subset with a probability of $\beta$ is

$\lceil\log(1-\beta)/log(1-(1-\varepsilon)^{d+2})\rceil.$ (50)

In Figure 10, we plot the number of necessary subsets according to Equation
(50) for $\beta=0.99$ for $d$ between $1$ and $50$ and $\varepsilon$ between
$0$ and $0.5$. The different green-shaded areas starting from the bottom right
indicate settings where up to $m=500$ initial subsets of size $d+2$ are
sufficient to obtain at least one clean subset with a probability of
$\beta=0.99$ and the various shades of orange indicate settings where we need
more elemental subsets.

Figure 10: Number of subsets of size $d+2$ we have to investigate for various
levels of contamination, to obtain at least one clean subset with a
probability of 99%

We assess the influence of using only 2 C-step and MLE iterations on the MMCD
estimators’ objective, the determinant of
$\hat{\bm{\Sigma}}^{\operatorname{col}}\otimes\hat{\bm{\Sigma}}^{\operatorname{row}}$.
We consider a setting with $n=200$ observations with $p=2$ rows and $q=8$
columns. The clean observations are generated by a centered matrix normal
distribution with
$\bm{\Sigma}^{\operatorname{row}}=\bm{\Sigma}^{\operatorname{fix}}(0.7)$ and
$\bm{\Sigma}^{\operatorname{col}}=\bm{\Sigma}^{\operatorname{mix}}(0.7)$, with
diagonal entries
$\sigma_{jj}^{\operatorname{fix}}=\sigma_{jj}^{\operatorname{mix}}=1$ and off-
diagonal entries $\sigma_{jk}^{\operatorname{fix}}(0.7)=0.7$ and
$\sigma_{jk}^{\operatorname{mix}}(0.7)=0.7^{\left|j-k\right|}$, respectively.
The outliers have a mean of $5$ and the same covariance as the regular
observations. We use 100 random subsets and plot
$\det(\hat{\bm{\Sigma}}^{\operatorname{col}}\otimes\hat{\bm{\Sigma}}^{\operatorname{row}})$
for subsequent C-step iterations with 40% of contamination. We compare the
setting when we limit the number of MLE iterations to 2 or iterate until
convergence and/or use elemental subsets with $d+2=6$ instead of $h$-subsets
of size $\nicefrac{{n}}{{2}}=100$.

Figure 11: Logarithm of determinant for successive C-step iterations to
analyze the effects of initial subset size and the number of ML iterations.

Comparing the top and bottom row of Figure 11, we see that there is virtually
no difference in the objective function while limiting the ML iterations
increases the computation speed. For the subset size, we see that several of
the elemental subsets yield robust solutions with a lower covariance
determinant than the larger $h$-subsets and that most of them are identified
after 1 or 2 iterations. While 40% contamination is not often encountered in
practice, it shows that the algorithm can deal with settings with such a high
level of contamination. We also analyzed settings with lower contamination,
and using elemental subsets and fewer ML iterations had no negative effects in
those settings, however, the larger $h$-subsets also led to robust solutions
more frequently.

######  Remark C.1.1.

Instead of using the consistency factor $c(\alpha)$ given in Equation (18), we
could also scale the estimators to align the MMDs with a quantile of the chi-
square distribution as in Rousseeuw and Driessen, (1999). Across the
simulations and the examples considered in this paper, we have only seen very
slight changes in the resulting estimators for both the raw and reweighted
MMCD.

##  Appendix D Shapley proofs

######  Proof of Theorem 5.2.1.

To show that the computation of the rowwise Shapley value can be simplified,
we start by rewriting the rowwise marginal contributions to the matrix
Mahalanobis distance.

$\displaystyle\Delta_{a}\operatorname{MMD}(\hat{\bm{X}}^{S}):=\operatorname{MMD}(\hat{\bm{X}}^{S\cup\\{a\\}})-\operatorname{MMD}(\hat{\bm{X}}^{S})$
$\displaystyle=$
$\displaystyle\sum_{i=1}^{p}\sum_{j=1}^{p}\sum_{k=1}^{q}\sum_{l=1}^{q}(\hat{x}^{S\cup\\{a\\}}_{ik}-m_{ik})(\hat{x}^{S\cup\\{a\\}}_{jl}-m_{jl})\omega_{lk}^{\operatorname{col}}\omega_{ij}^{\operatorname{row}}-\sum_{i=1}^{p}\sum_{j=1}^{p}\sum_{k=1}^{q}\sum_{l=1}^{q}(\hat{x}^{S}_{ik}-m_{ik})(\hat{x}^{S}_{jl}-m_{jl})\omega_{lk}^{\operatorname{col}}\omega_{ij}^{\operatorname{row}}$
$\displaystyle=$ $\displaystyle\sum_{i\in S\cup\\{a\\}}\sum_{j\in
S\cup\\{a\\}}\sum_{k=1}^{q}\sum_{l=1}^{q}(x_{ik}-m_{ik})(x_{jl}-m_{jl})\omega_{lk}^{\operatorname{col}}\omega_{ij}^{\operatorname{row}}-\sum_{i\in
S}\sum_{j\in
S}\sum_{k=1}^{q}\sum_{l=1}^{q}(x_{ik}-m_{ik})(x_{jl}-m_{jl})\omega_{lk}^{\operatorname{col}}\omega_{ij}^{\operatorname{row}}$
$\displaystyle=$ $\displaystyle\sum_{i\in S\cup\\{a\\}}\sum_{j\in
S}\sum_{k=1}^{q}\sum_{l=1}^{q}(x_{ik}-m_{ik})(x_{jl}-m_{jl})\omega_{lk}^{\operatorname{col}}\omega_{ij}^{\operatorname{row}}+\sum_{i\in
S\cup\\{a\\}}\sum_{k=1}^{q}\sum_{l=1}^{q}(x_{ik}-m_{ik})(x_{al}-m_{al})\omega_{lk}^{\operatorname{col}}\omega_{ia}^{\operatorname{row}}$
$\displaystyle-\sum_{i\in S}\sum_{j\in
S}\sum_{k=1}^{q}\sum_{l=1}^{q}(x_{ik}-m_{ik})(x_{jl}-m_{jl})\omega_{lk}^{\operatorname{col}}\omega_{ij}^{\operatorname{row}}$
$\displaystyle=$ $\displaystyle\sum_{i\in S}\sum_{j\in
S}\sum_{k=1}^{q}\sum_{l=1}^{q}(x_{ik}-m_{ik})(x_{jl}-m_{jl})\omega_{lk}^{\operatorname{col}}\omega_{ij}^{\operatorname{row}}-\sum_{i\in
S}\sum_{j\in
S}\sum_{k=1}^{q}\sum_{l=1}^{q}(x_{ik}-m_{ik})(x_{jl}-m_{jl})\omega_{lk}^{\operatorname{col}}\omega_{ij}^{\operatorname{row}}$
$\displaystyle+\sum_{j\in
S}\sum_{k=1}^{q}\sum_{l=1}^{q}(x_{ak}-m_{ak})(x_{jl}-m_{jl})\omega_{lk}^{\operatorname{col}}\omega_{aj}^{\operatorname{row}}+\sum_{i\in
S}\sum_{k=1}^{q}\sum_{l=1}^{q}(x_{ik}-m_{ik})(x_{al}-m_{al})\omega_{lk}^{\operatorname{col}}\omega_{ia}^{\operatorname{row}}$
$\displaystyle+\sum_{k=1}^{q}\sum_{l=1}^{q}(x_{ak}-m_{ak})(x_{al}-m_{al})\omega_{lk}^{\operatorname{col}}\omega_{aa}^{\operatorname{row}}$
$\displaystyle=$ $\displaystyle 2\sum_{i\in
S}\sum_{k=1}^{q}\sum_{l=1}^{q}(x_{al}-m_{al})(x_{ik}-m_{ik})\omega_{lk}^{\operatorname{col}}\omega_{ia}^{\operatorname{row}}+\sum_{k=1}^{q}\sum_{l=1}^{q}(x_{ak}-m_{ak})(x_{al}-m_{al})\omega_{lk}^{\operatorname{col}}\omega_{aa}^{\operatorname{row}}.$

Now the coordinates $\phi_{a}(\bm{X})$ of the Shapley value
$\bm{\phi}(\bm{X})$ are given by
$(w(\left|S\right|)=\frac{\left|S\right|!(p-\left|S\right|-1)!}{p!})$

$\displaystyle\phi_{a}(\bm{X})=$ $\displaystyle\sum_{S\subseteq
P\setminus\\{a\\}}w(\left|S\right|)\Delta_{a}\operatorname{MMD}(\hat{\bm{X}}^{S})$
$\displaystyle=$ $\displaystyle 2\sum_{S\subseteq
P\setminus\\{a\\}}w(\left|S\right|)\sum_{i\in
S}\sum_{k=1}^{q}\sum_{l=1}^{q}(x_{al}-m_{al})(x_{ik}-m_{ik})\omega_{lk}^{\operatorname{col}}\omega_{ia}^{\operatorname{row}}$
$\displaystyle+\sum_{S\subseteq
P\setminus\\{a\\}}w(\left|S\right|)\sum_{k=1}^{q}\sum_{l=1}^{q}(x_{ak}-m_{ak})(x_{al}-m_{al})\omega_{lk}^{\operatorname{col}}\omega_{aa}^{\operatorname{row}}$

and we can simplify the first term of the sum as

$\displaystyle 2\sum_{S\subseteq P\setminus\\{a\\}}w(\left|S\right|)\sum_{i\in
S}\sum_{k=1}^{q}\sum_{l=1}^{q}(x_{al}-m_{al})(x_{ik}-m_{ik})\omega_{lk}^{\operatorname{col}}\omega_{ia}^{\operatorname{row}}$
$\displaystyle=2\sum_{s=1}^{p-1}w(\left|S\right|)\sum_{S\subseteq
P\setminus\\{a\\},\left|S\right|=s}\sum_{i\in
S}\sum_{k=1}^{q}\sum_{l=1}^{q}(x_{al}-m_{al})(x_{ik}-m_{ik})\omega_{lk}^{\operatorname{col}}\omega_{ia}^{\operatorname{row}}$
$\displaystyle=2\sum_{s=1}^{p-1}\sum_{k=1}^{q}\sum_{l=1}^{t}w(\left|S\right|)\sum_{S\subseteq
P\setminus\\{a\\},\left|S\right|=s}\sum_{i\in
S}(x_{al}-m_{al})(x_{ik}-m_{ik})\omega_{lk}^{\operatorname{col}}\omega_{ia}^{\operatorname{row}}$
$\displaystyle=2\sum_{s=1}^{p-1}\sum_{k=1}^{q}\sum_{l=1}^{q}\frac{\left|S\right|!(p-\left|S\right|-1)!}{p!}\binom{p-2}{s-1}\sum_{i\in
P\setminus\\{a\\}}(x_{al}-m_{al})(x_{ik}-m_{ik})\omega_{lk}^{\operatorname{col}}\omega_{ia}^{\operatorname{row}}$
$\displaystyle=2\frac{1}{p(p-1)}\sum_{s=1}^{p-1}s\sum_{k=1}^{q}\sum_{l=1}^{q}\sum_{i\in
P\setminus\\{a\\}}(x_{al}-m_{al})(x_{ik}-m_{ik})\omega_{lk}^{\operatorname{col}}\omega_{ia}^{\operatorname{row}}$
$\displaystyle=2\frac{1}{p(p-1)}\frac{p(p-1)}{2}\sum_{k=1}^{q}\sum_{l=1}^{q}\sum_{i\in
P\setminus\\{a\\}}(x_{al}-m_{al})(x_{ik}-m_{ik})\omega_{lk}^{\operatorname{col}}\omega_{ia}^{\operatorname{row}}$
$\displaystyle=\sum_{k=1}^{q}\sum_{l=1}^{q}\sum_{i\in
P\setminus\\{a\\}}(x_{al}-m_{al})(x_{ik}-m_{ik})\omega_{lk}^{\operatorname{col}}\omega_{ia}^{\operatorname{row}}.$

Since the second term is independent of the subset $S$ and $\sum_{S\subseteq
P\setminus\\{a\\}}w(\left|S\right|)=1$, we obtain

$\displaystyle\phi_{a}(\bm{X})=$
$\displaystyle\sum_{k=1}^{q}\sum_{l=1}^{q}\sum_{i\in
P\setminus\\{a\\}}(x_{al}-m_{al})(x_{ik}-m_{ik})\omega_{lk}^{\operatorname{col}}\omega_{ia}^{\operatorname{row}}$
$\displaystyle+\sum_{k=1}^{q}\sum_{l=1}^{q}(x_{ak}-m_{ak})(x_{al}-m_{al})\omega_{lk}^{\operatorname{col}}\omega_{aa}^{\operatorname{row}}$
$\displaystyle=$
$\displaystyle\sum_{i=1}^{p}\sum_{k=1}^{q}\sum_{l=1}^{q}(x_{al}-m_{al})(x_{ik}-m_{ik})\omega_{lk}^{\operatorname{col}}\omega_{ia}^{\operatorname{row}},$

which completes the proof. ∎

##  Appendix E Further simulation results

Simulations that analyze the behavior of affine equivariant multivariate
estimators often use data generated from a standard multivariate normal
distribution, i.e. mean zero and covariance identity, because the data could
be transformed to this setting anyway. Lemma 3.0.1 shows that the MMCD
estimators are matrix affine equivariant but we do not consider data generated
from a standard matrix normal distribution. The reason for this is twofold. On
the one hand, consider the fact that the ML estimators are estimated
iteratively using the flip-flop algorithm (Dutilleul,, 1999), which is
commonly initialized by setting either the rowwise or columnwise covariance
matrix equal to the identity matrix. Hence, we would initialize one covariance
matrix with the theoretically correct one, which is very unlikely in practice.
On the other hand, when we know that we are dealing with data that have
independent rows and/or columns, the columns and/or rows could be analyzed
separately using standard multivariate techniques.

This section delves deeper into the simulations outlined in Section 6. The
simulations involve generating regular and outlying samples from a matrix
normal distribution. A fraction, $\varepsilon$, of the clean data is replaced
by outliers. The clean observations are drawn from a centered distribution,
while the mean of the outliers shifts based on the parameter $\gamma$—the mean
matrix of the outliers comprises entries set to $\gamma$. Three types of
covariance matrices are considered: The covariance matrix
$\bm{\Sigma}^{\operatorname{rnd}}$, as proposed by Agostinelli et al., (2015),
is randomly generated with low correlations. The covariance matrix
$\bm{\Sigma}^{\operatorname{fix}}(0.7)$ induces a relatively collinear
setting, with entries defined as:

$\displaystyle\sigma_{jk}^{\operatorname{fix}}(0.7)=\begin{cases}1&\text{ if
}j=k\\\ 0.7&\text{ if }j\neq k\end{cases}.$

The covariance matrix $\bm{\Sigma}^{\operatorname{mix}}(0.7)$ exhibits both
large and small correlations, featuring entries as follows:

$\displaystyle\sigma_{jk}^{\operatorname{mix}}(0.7)=\begin{cases}1&\text{ if
}j=k\\\ 0.7^{\left|j-k\right|}&\text{ if }j\neq k\end{cases}.$

While maintaining the same covariance structure for both outliers and clean
samples, we explore the impact of increasing the outlier covariance by scaling
the covariance of clean observations by the parameter $s$. Each simulation
setting is replicated 100 times. Unless specified otherwise, we set
$\bm{\Sigma}^{\operatorname{row}}=\bm{\Sigma}^{\operatorname{rnd}}$,
$\bm{\Sigma}^{\operatorname{col}}=\bm{\Sigma}^{\operatorname{mix}}(0.7)$, and
$s=1$ as detailed in Section 6. An overview of all parameters for the
simulations is provided in Table 1. For $(p,q)=(5,20)$, all listed parameter
combinations are considered, while for $(p,q)\in{(50,20),(100,50)}$, we only
consider $s=1$. Additionally, for $(p,q)=(5,20)$, we compute the Minimum
Covariance Determinant (MCD) estimator on the vectorized observations for
comparative analysis.

Parameter | Parameter values
---|---
Sample size $n$ | $20,50,100,200,300,400,500,750,1000$
Contamination $\varepsilon$ | $0.1,0.2,0.3,0.4$
Rowwise covariance $\bm{\Sigma}^{\operatorname{row}}$ | $\bm{\Sigma}^{\operatorname{fix}}(0.7),\bm{\Sigma}^{\operatorname{rnd}}$
Columnwise covariance $\bm{\Sigma}^{\operatorname{col}}$ | $\bm{\Sigma}^{\operatorname{mix}}(0.7),\bm{\Sigma}^{\operatorname{rnd}}$
Mean shift $\gamma$ | $1,2,3,4,5$
Covariance multiplier $s$ | $1,2,3,4$
Table 1: Parameters considered for the simulations with $p,q=(5,20)$.

To assess the quality of covariance estimation, we consider two additional
measures to the KL divergence: the relative Frobenius error given as

$\frac{\left\lVert\hat{\bm{\Sigma}}^{\operatorname{col}}\otimes\hat{\bm{\Sigma}}^{\operatorname{row}}-\bm{\Sigma}^{\operatorname{col}}\otimes\bm{\Sigma}^{\operatorname{row}}\right\rVert_{F}}{\left\lVert\bm{\Sigma}^{\operatorname{col}}\otimes\bm{\Sigma}^{\operatorname{row}}\right\rVert_{F}},$

and angle error between eigenvalues given as

$1-\frac{\hat{\bm{a}}^{\top}\bm{a}}{\sqrt{\hat{\bm{a}}^{\top}\hat{\bm{a}}}\sqrt{\bm{a}^{\top}\bm{a}}},$

where $\hat{\bm{a}}$ and ${\bm{a}}$ are the vectors of sorted eigenvalues of
$\hat{\bm{\Sigma}}^{\operatorname{col}}\otimes\hat{\bm{\Sigma}}^{\operatorname{row}}$
and $\bm{\Sigma}^{\operatorname{col}}\otimes\bm{\Sigma}^{\operatorname{row}}$,
respectively. Large values of the KL divergence and the relative Frobenius
error indicate difficulties in the estimation of the covariances. The angle
error between the eigenvalues is in the interval $[0,1]$, and a large value
means that the shape of the covariance matrix is not appropriately estimated.
To assess the efficacy of outlier detection, we extend our evaluation beyond
the F-score, considering precision and recall as additional metrics. Precision
denotes the proportion of correctly identified outliers among all detected
samples, while recall represents the proportion of correctly identified
outliers among all contaminated samples.

We begin by examining the previously discussed settings in Section 6,
expanding our analysis to include varying sample sizes
$n\in\\{20,50,100,200,300,400,500,750,1000\\}$. Figure 12 presents the scores
depicting covariance estimation. For the MMCD estimators the covariance
estimation performance is improving with the sample size across all settings.
On the other hand, the sample size has a negligible effect on the quality of
the MLE estimators in the presence of outliers and a larger mean shift
decreases performance. For small sample sizes, MLE and MMCD estimators are
close in terms of KL divergence, but the angle error and Frobenius error
indicate worse performance of the MLE estimators also for small sample sizes.
The relative Frobenius error of MMCD estimators is smaller than one and thus
only plotted on $[0,1]$. For the MLE estimators, it is often above one and
those settings are not visible in plots.

Analyzing Figure 13, we see that the recall of the MMCD estimators is close to
one across all settings, except for $(p,q)=(5,20)$ and a small mean shift. The
MLE estimators only detect the most severe outliers due to the masking effect,
leading to a median recall below $0.25$ across all settings. With an
increasing sample size, the precision of the MMCD is improving and has very
low variability. On the other hand, the MLE shows very unstable results.

Figure 14 shows the difference between a contamination of $\varepsilon=0.1$
and $\varepsilon=0.4$ with mean shift $\gamma=3$. The KL divergence reveals
that the MMCD estimator yields more accurate results across all settings.
However, for $\varepsilon=0.1$, the F-scores of the MLE are increasing with
the dimensionality and perform better than the MMCD for small sample sizes.
For $\varepsilon=0.4$, only the MMCD yields reliable results.

For the setting with $(p,q)=(5,20)$, $\varepsilon=0.2$ and $n>100$, we also
computed the MCD on the vectorized samples in addition to the matrix MLE and
MMCD and considered the true mean and covariance used to generate the data as
a benchmark. Figures 15 and 16 summarize the results and reveal that the MCD
on the vectorized observations does not lead to robust estimators. This issue
arises because the robustness of the MCD and MMCD depends on the
dimensionality of the data. For the MCD it depends on $p\cdot q$ and for the
MMCD it depends on $\nicefrac{{p}}{{q}}+\nicefrac{{q}}{{p}}$. To achieve a
$99\%$ probability of obtaining at least one clean initial subset with
$(p,q)=(5,20)$ and a contamination $\varepsilon=0.2$, MCD requires
approximately $2.8\cdot 10^{10}$ initial subsets, while MMCD only needs 16.
For the setting with the smallest mean shift, the comparison between MMCD and
the actual parameters in Figure 16 highlights the difficulty of this setting,
since even using the actual parameters the recall shows a lot of variability.

In addition to shifting the mean of the outliers by
$\gamma\in\\{1,\dots,5\\}$, we now consider the effect of scaling the
covariance by $s\in\\{1,\dots,4\\}$. The difference between $s\in\\{2,3,4\\}$
was negligible and Figures 17 and 18 summarize the results for $s=2$. While
the MLE performs quite well for outlier detection, especially compared to the
setting with $s=1$ (see Figure 14), the estimated covariance matrices are not
accurate. The overall performance of the MCD computed on the vectorized
samples improves with increasing sample size $n$ but even more samples would
be necessary to obtain similar results to the MMCD.

Finally, we compare the 4 different combinations of row- and columnwise
covariance matrices with $\varepsilon=0.2$. In Figure 19 we use $\gamma=1$ and
in Figure 20 we increase the mean shift to $\gamma=5$. The F-score based on
the true parameters is included as a reference. When
$\bm{\Sigma}^{\operatorname{row}}=\bm{\Sigma}^{\operatorname{fix}}(0.7)$,
$\bm{\Sigma}^{\operatorname{col}}=\bm{\Sigma}^{\operatorname{mix}}(0.7)$, and
$\gamma=1$, the mean shift is too small and the outliers cannot be separated
from the regular observations. Increasing the mean shift to $\gamma=5$, the
separation becomes clearer and the MMCD yields robust results. If $\gamma=1$,
we still see a lot of variability in the F-score if only the rowwise or
columnwise covariance matrix is generated randomly. However, if both are
generated randomly the distinction between outliers and regular observations
is easier.

Overall the MMCD performs very well in all settings where the outliers can be
clearly distinguished from the regular observations. It could benefit from
small sample corrections as in the multivariate case. Those factors could be
derived in a large-scale simulation study as in Pison et al., (2002).

Figure 12: Overview of simulation results with a fraction $\varepsilon=0.2$ of
contaminated samples. The quality of covariance estimation is evaluated based
on the logarithm of KL divergence, angle error between eigenvalues, and
relative Frobenius error. Figure 13: Overview of simulation results with a
fraction $\varepsilon=0.2$ of contaminated samples. The outlier detection
capabilities are measured by F-score, precision, and recall. Figure 14:
F-score and logarithm of KL divergence for simulations with mean shift
$\gamma=3$. Figure 15: Quality of covariance estimation for simulations with
$\varepsilon=0.2$ and $(p,q)=(5,20)$. Figure 16: Outlier detection
capabilities for simulations with $\varepsilon=0.2$ and $(p,q)=(5,20)$. Figure
17: Quality of covariance estimation for simulations where the covariance of
the outliers is scaled by $s=2$. Figure 18: Outlier detection capabilities for
simulations where the covariance of the outliers is scaled by $s=2$. Figure
19: F-score and logarithm of KL divergence comparing 4 different combinations
of row- and columnwise covariance matrices, $\gamma=1$, and $\varepsilon=0.2$.
Figure 20: F-score and logarithm of KL divergence comparing 4 different
combinations of row- and columnwise covariance matrices, $\gamma=5$, and
$\varepsilon=0.2$.


# Rendering Humans from Object-Occluded Monocular Videos

Tiange Xiang, Adam Sun, Jiajun Wu, Ehsan Adeli, Li Fei-Fei
Stanford University
Correspondence to<EMAIL_ADDRESS>
###### Abstract

3D understanding and rendering of moving humans from monocular videos is a
challenging task. Despite recent progress, the task remains difficult in real-
world scenarios, where obstacles may block the camera view and cause partial
occlusions in the captured videos. Existing methods cannot handle such defects
due to two reasons. First, the standard rendering strategy relies on point-
point mapping, which could lead to dramatic disparities between the visible
and occluded areas of the body. Second, the naive direct regression approach
does not consider any feasibility criteria (_i.e_., prior information) for
rendering under occlusions. To tackle the above drawbacks, we present OccNeRF,
a neural rendering method that achieves better rendering of humans in severely
occluded scenes. As direct solutions to the two drawbacks, we propose surface-
based rendering by integrating geometry and visibility priors. We validate our
method on both simulated and real-world occlusions and demonstrate our
method’s superiority. Project page:
https://cs.stanford.edu/~xtiange/projects/occnerf/

##  1 Introduction

Rendering 3D human bodies from a sequence of observations is of great interest
in various communities, including robotics [yang2021s3], motion analysis
[endo2022gaitforemer], and healthcare [gerats20223d]. This task is
challenging, since one must recover the complete human body with complex
textures and poses from sparse partial observations. It is usually cumbersome
to acquire images of the same human object from multiple camera angles
simultaneously; hence, capturing a monocular video from a single camera is
more common and feasible.

The task of rendering humans from a monocular video is not new. Progress so
far mainly focuses on rendering quality [neuralbody, weng2022humannerf] and
rendering efficiency [peng2022selfnerf, jiang2022instantavatar]. However, most
existing neural rendering methods assume that the human object is placed in a
scene with a clear view of the entire body without any external interference.
In contrast, real-world environments often contain undesired obstacles that
contaminate training data and impact the rendering quality (See Figure 1).
These real-world occlusions pose significant challenges for training when
using only monocular videos, where no other camera angles can be used to
provide complementary information. As a result, a direct application of
previous neural rendering methods on object-occluded videos leads to subpar
performance. Optimizing a neural radiance field is difficult under occlusions.
There is often no ground truth associated with the occluded area.
Additionally, radiance fields are typically optimized in a scene-specific
manner; that is, no external information can and should be used to fill in the
occluded areas.

Figure 1: Object obstacles in the scene may cause severe occlusions in the
rendered/captured videos, imposing additional challenges into the rendering
process. Top row: Ideal scene with no defects and clear view of the body;
Bottom row: Real-world scene with undesired obstacles and occluded body parts.

Two major drawbacks of previous methods impair their capabilities to train on
object-occluded videos. First, the prior work does not account for local
geometry cues in their rendering process. Following the point-based rendering
paradigm as in NeRF [nerf], most previous methods render color and density
values of a ray sample by only looking at a single 3D coordinate. However, we
explain in section 3.2 that this basic strategy may lead to dramatically
different rendering results even in very close positions. Second, methods
suffer from not properly incorporating priors. In the monocular video setting,
geometry (_e.g_., SMPL [SMPL]) and visibility priors can describe a complete
human geometry and indicate which body parts are visible to the camera.

In this work, we propose novel methods for dealing with the above drawbacks,
allowing us to accurately render occluded humans from monocular video. We
first present a surface-based rendering strategy that determines the radiance
of each 3D ray sample by conditioning it on a wide region of the human body’s
surface. A geometry prior is used to discretely parameterize the surface
segments. We then collect visibility frequencies on the human body through
training frames and formulate them as attention maps for better aggregation of
the surface regions. Finally, we design a loss function to encourage the
network to output high-density values for positions within the human body.

In summary, our contributions are three-fold: _(i)_ We are the first to study
dynamic human rendering under real-world settings with severe occlusions.
_(ii)_ We propose novel methods that include surface-based rendering, a
reformulation of body part visibility frequency as attention, and a
completeness loss to enable human rendering from object-occluded monocular
videos. _(iii)_ We empirically demonstrate that our methods achieve
significant quantitative and qualitative improvements compared to the previous
state-of-the-art, yielding the first baseline in this topic.

##  2 Related Work

3D Human Modeling. Reconstructing the appearance and geometry of humans has
always been challenging. From [matusik2000image, collet2015high,
dou2016fusion4d], techniques have been consistently designed for high-quality
and efficient human modeling. Traditional methods mainly relied on SMPLify
[Bogo:ECCV:2016] or Video-avatars [alldieck2018video] to regress SMPL [SMPL]
to parameterize a structured human body. More complex networks were
subsequently designed that can model 3D humans based on temporal priors
[humanMotionKZFM19, kocabas2019vibe], based on depth [jiang2022selfrecon,
saito2019pifu, hong2021stereopifu], or multiple human instances simultaneously
[jiang2020coherent, sun2021monocular, BEV, zhang2021body]. Although this line
of methods can generate a reasonable human body mesh fast, using parametric
SMPL models limit their ability to achieve photo-realistic view synthesis.

Neural Radiance Field for Human Rendering. Since the emergence of Neural
Radiance Fields (NeRF) [nerf], different extensions have been recently
developed to enable high-quality rendering of static scenes [hedman2021baking,
srinivasan2021nerv, barron2021mip, barron2022mip, verbin2022ref,
tancik2020fourier, sun2021direct, muller2022instant], moving objects
[gao2021dynamic, li2021neural, park2021nerfies, park2021hypernerf,
pumarola2021d, ost2021neural], and dynamic humans [neuralbody,
bergman2022generative, chen2021animatable, chibane2020implicit, chen2022gdna,
corona2021smplicit, deng2020nasa, feng2022capturing, he2020geo, he2021arch++,
huang2020arch, jiang2022selfrecon, li2022tava, liu2021neural,
noguchi2021neural, peng2021animatable, tiwari2021neural, wang2022arah,
xu2021h, peng2022selfnerf, jiang2022instantavatar, jiang2022neuman]. NeRF
predicts the color and density of each ray sample point in a 3D space and
aggregates them together through volume rendering (more details are in section
3.1). This approach enables the capture of intricate lighting effects and
textural details that are typically difficult to model in traditional methods.

Our work is built upon HumanNeRF [weng2022humannerf] due to its state-of-the-
art rendering quality for monocular videos. HumanNeRF maintains a static
T-pose human body as the canonical space and learns a motion field
[weng2020vid2actor] that maps the canonical representation to every frame of
the video in the observation space (more details are in section 3.1). We note
that a concurrent work, SelfNeRF [peng2022selfnerf], shares a similar
regression schema as ours. However, their method is designed particularly for
fast rendering and compromises rendering quality. Moreover, all of the above
approaches were developed on clean training data only, where body parts are
assumed to be clearly demonstrated in the monocular video without any
occlusions. On the other hand, our work aims to render humans under
occlusions.

Occluded Human Modelling. Rendering objects under all kinds of real-world
defects, especially partial occlusions, is a long-standing research problem.
Early works sought to estimate human poses from occluded images and videos
[sarandi2018robust, zhang2020object, kocabas2021pare, biggs20203d], while more
recent works [rockwell2020full, sun2021monocular, yang2022lasor,
kocabas2021pare] learn both SMPL shape and pose priors directly from occluded
images and videos. The generated SMPL parameters from these robust methods can
be used as good geometric prior for a subsequent rendering process.

However, optimizing a NeRF from occluded images is still an unsolved problem.
There are very few works that were specifically designed for rendering under
occlusions. NeuRay [liu2022neural] was proposed to regress not only the
radiance but also a feature vector of every ray sample to indicate visibility.
This enables the optimization of the radiance field to focus on visible
features and reduce interference from occlusions. Ha-NeRF
[chen2022hallucinated] presents an appearance hallucination module to handle
time-varying appearances and an anti-occlusion module to decompose the static
subjects for visibility accurately. Unfortunately, these existing methods are
not capable of handling dynamic objects, and the multi-view inputs used in
past work actually make it easier to learn under occlusions. In this work, we
consider visibility as an additional prior to assist in rendering under
occlusions. Our work is the very first in this field to handle occlusions for
rendering dynamic objects from only a monocular video.

##  3 Methods

In this section, we first review preliminaries and background in NeRF [nerf]
and HumanNeRF [weng2022humannerf] (section 3.1). We then present our OccNeRF
by introducing a new rendering strategy (section 3.2), a formulation of
visibility into attention (section 3.3), and a novel loss function (section
3.4) to ensure high rendering quality as well as geometry completeness under
occlusions. An overview of our OccNeRF is shown in Figure 2.

Figure 2: OccNeRF functions on video frames and optimizes a neural radiance
field for synthesizing novel views of an object-occluded human. With a pre-
computed body pose, we first adopt the motion field to map observable ray
samples $\mathbf{x}$ into coordinates $\mathbf{\hat{x}}$ in a canonical space.
Nearest parameterization vertices $\\{\mathbf{v}_{i}\\}$ of every
$\mathbf{\hat{x}}$ are searched and conditioned by our surface-based rendering
method. During training, we iteratively update the attention scores
$\\{\mathbf{a}\\}$ for all $\\{\mathbf{v}\\}$ as indications of their
visibility. This ensures more attention on frequently visible vertices to
improve rendering quality. The blended vertex $\mathbf{\hat{v}}$ along with
its signed distance to $\mathbf{\hat{x}}$ are jointly encoded via a 4D hash
grid before being fed into the regression MLP along with the encoded vertices.
Photometric and perceptual constraints are enforced against visible pixels,
while an additional loss function is designed to encourage geometry
completeness in occluded areas.

###  3.1 Preliminaries and Background

Neural Radiance Field [nerf]. Consider a (bounded) 3D scene. NeRF learns a
regression function $\mathcal{F}$ (usually an MLP) that takes the encoded
coordinates of a 3D point $\mathbf{x}\in\mathbb{R}^{3}$ in the scene as input,
and outputs the corresponding color $\mathbf{c}$ and density $\mathbf{\sigma}$
at that position:

$\mathbf{c},\mathbf{\sigma}=\mathcal{F}(\gamma(\mathbf{x})),$ (1)

where $\gamma(\cdot)$ is an encoding function. We refer to the above point-to-
point mapping as point-based rendering. Instead of sampling points
$\mathbf{x}$ randomly in the scene, NeRF casts rays $\mathbf{r}$ towards the
directions $\pi$ from the camera origin $\mathbf{o}$ to every pixel, and
sample $\mathbf{x}$ on the rays uniformly. Then, NeRF renders the pixel by
aggregating the regressed color and density at each $\mathbf{x}$ via volume
rendering [lombardi2019neural]:

$\sum_{i}\alpha(\mathbf{x}_{i})\prod_{j<i}(1-\alpha(\mathbf{x}_{j}))\mathbf{c},$
(2)

where $\alpha(\mathbf{x}_{i})=1-\exp(-\mathbf{\sigma}_{i}\delta_{i})$,
$\mathbf{x}_{i}=\mathbf{o}+z_{i}\pi$, $z_{i}$ is the z-axis position of ray
samples, and $\delta_{i}=z_{i+1}-z_{i}$ is the distance between two samples
along the ray.

HumanNeRF [weng2022humannerf]. HumanNeRF is a method based on NeRF that can
render humans from monocular videos by representing them as neural fields. The
method first defines a moving human in a static canonical space with 3D
coordinates $\hat{\mathbf{x}}$, and warps the human in different dynamic poses
by warping the canonical body pose $\mathbf{p}$ to the observation space. This
warping process also defines the transformation of 3D coordinates in the two
spaces:

$\hat{\mathbf{x}}=\mathcal{T}(\mathbf{p},\mathbf{x}),$ (3)

where $\mathcal{T}$ is a network that maps $\mathbf{x}$ in the observation
space to corresponding coordinates $\hat{\mathbf{x}}$ at the canonical space,
denoted as the motion field. The motion field achieves the mapping by
performing a weighted sum of a set of $K$ motion bases defined by rotations
$R_{i}$ and translation $t_{i}$ of the $i_{th}$ bone of the human body:

$\hat{\mathbf{x}}=\sum^{K}_{i}w_{i}(\mathbf{x})(R_{i}\mathbf{x}+t_{i}),$ (4)

where $R_{i}$ and $t_{i}$ can be directly computed from $\mathbf{p}$. $w_{i}$
serves as the weights in the observation space, which can be approximated
using the weights defined in the canonical space. Similar to
[weng2023personnerf], we removed both the non-rigid motion and the pose
correction part of the motion field.

###  3.2 Surface-based Rendering

Motivation. Although HumanNeRF and its variants can already achieve decent
rendering quality in an occlusion-free scene, they fail to excel when
obstacles block the view of the camera that causes severe occlusions. We
attribute this failure to the point-based rendering strategy (reviewed in
section 3.1). Given the ray samples at discrete 3D coordinates $\mathbf{x}$ in
a continuous 3D space, even a mild variation between two coordinates
$\mathbf{x}_{a}$ and $\mathbf{x}_{b}$ can lead to dramatic disparities on the
outputs. Let there be no overlaps between the input distributions
$\\{\mathbf{x}_{a}\\}$ and $\\{\mathbf{x}_{b}\\}$:

$\\{\mathbf{x}_{a}\\}\cap\\{\mathbf{x}_{b}\\}=\emptyset\ \mathbin{|}\
\mathbf{x}_{a}\not\equiv\mathbf{x}_{b}.$ (5)

Then, in an occluded scene, when only $\mathbf{x}_{a}$ is visible to the
camera, non-overlapping inputs may yield huge output differences, even at very
close locations. This is because $\mathbf{x}_{a}$ has visible supervisions
while $\mathbf{x}_{b}$ does not, which leads to unexpected artifacts and
unstable rendering quality at occluded regions.

This motivates us to enlarge the range of the inputs to cover a wider range of
3D space rather than a single 3D coordinate. We expect that a new rendering
strategy with range-to-point mapping will be able to reduce the output
difference at adjacent locations:

$\int_{\mathbb{R}^{3}}\min[\mathcal{N}(\mathbf{x}_{a}),\mathcal{N}(\mathbf{x}_{b})]d\mathbf{x}\gg
0,$ (6)

where $\mathcal{N}(\mathbf{x}_{a})$ and $\mathcal{N}(\mathbf{x}_{b})$ are 3D
sub-regions corresponding to the target coordinates $\mathbf{x}_{a}$ and
$\mathbf{x}_{b}$. With a focus on human rendering, we define the sub-regions
as continuous segments on the body surface. We name this rendering strategy
surface-based rendering. A high-level comparison to the standard point-based
rendering is outlined in Figure 3.

Parameterization. It is difficult to process continuous variables, especially
ones with irregular distributions, which is the case with human surfaces. We
approach this challenge by using a discretized parameterization of the
continuous 3D sub-regions. Specifically, we use the pre-computed SMPL [SMPL]
mesh as a geometry prior to roughly outline the surface of the human body. The
surface segments are then parameterized by the $k$ nearest mesh vertices
$\\{\mathbf{v}_{1},\cdots,\mathbf{v}_{k}\\}$ when using the target coordinates
$\mathbf{x}$ as queries. We denote these discrete neighboring coordinates as
parameterization vertices. With our surface-based rendering approach, we now
reformulate Equation 1 as a hybrid combination of both the target coordinate
and the parameterized surface:

$\mathbf{c},\mathbf{\sigma}=\mathcal{F}(\underbrace{\gamma(\mathbf{\hat{x}})}_{\text{point
term}}\mathbin{\|}\ \
\underbrace{\phi(\\{\gamma(\mathbf{v}_{1}),\cdots,\gamma(\mathbf{v}_{k})\\})}_{\text{surface
term}}\ ),$ (7)

where $\phi$ is a function that aggregates all $\\{\mathbf{v}_{i}\\}$ of a
query $\mathbf{x}$ and $\mathbin{\|}$ denotes concatenation. The above
formulation requires all parameterization points $\mathbf{v}$ to be as
accurately laid on the human body as possible. However, this is difficult for
the coarsely structured SMPL mesh with potential approximation errors.
Therefore, we rely on the inaccurate SMPL mesh only as an initialization and
enable the positions of $\mathbf{v}$ to be optimized jointly with the network.
This formulation is analogous to area sampling [Loubet2019Reparameterizing]
for ray tracing, which not only integrates samples along the ray but in
vicinity area.

Figure 3: Left: Point-based rendering takes as input a single 3D point
$\mathbf{x}$ that has no overlap with nearby (but not identical) points, and
is poorly conditioned at occluded areas; Right: Our surface-based rendering
approach takes as input a 3D sub-regions $\mathcal{N}(\mathbf{x})$ at location
$\mathbf{x}$ that yields a large overlap at adjacent locations for better
conditioning at occluded areas.

Multi-Scale Representations. Choosing the area of surface segments and the
number of parameterization vertices $k$ per query is another issue. A small
area leads to less overlap and more unstable results, while a large area leads
to more overlap of $\\{\mathbf{v}_{i}\\}$ at two query locations but a more
inefficient search of nearest neighbors. Taking inspiration from the multi-
scale mechanism used in I-NGP [muller2022instant], we construct the set of
parameterization vertices by finding the nearest neighbors on the SMPL mesh at
multiple scales. We define the default SMPL mesh at the finest scale and
iteratively down-sample the mesh with sparse vertices through furthest point
sampling [qi2017pointnet] with a ratio of 0.25 for 3 iterations. In practice,
we set a small $k=5$ at all 4 scales, which enables an efficient span over a
large surface area.

Surface-Aware Regression. The additional operations introduced above
inevitably slow down network training. Similar to [peng2022selfnerf,
jiang2022instantavatar], we adopt a hash grid [muller2022instant] in the
canonical space as our encoding function $\gamma(\cdot)$ instead of the
standard frequency-based positional encoding [nerf]. Furthermore, for better
awareness of the human body surface, we represent a query point in the
canonical space $\mathbf{\hat{x}}$ by the combination of its closest
parameterization vertex $\mathbf{\hat{v}}$ and their signed distance
$\mathbf{d}$. For simplicity, we reuse the previously searched $k$ nearest
vertices and blend them through normal similarities to form the closest vertex
$\mathbf{\hat{v}}$:

$\mathbf{\hat{v}}=\frac{\sum_{i}^{k}|\texttt{cos}(\mathbf{\hat{x}},\mathbf{v}_{i})|\mathbf{v}_{i}}{\sum_{i}^{k}|\texttt{cos}(\mathbf{\hat{x}},\mathbf{v}_{i})|},$
(8)

where $\texttt{cos}(\mathbf{\hat{x}},\mathbf{v}_{i})$ denotes the cosine
similarity between the vector $\mathbf{\hat{x}}-\mathbf{v}_{i}$ and the normal
vector at $\mathbf{v}_{i}$. After obtaining $\mathbf{\hat{v}}$, we can easily
determine $\mathbf{d}$ between $\mathbf{\hat{v}}$ and $\mathbf{\hat{x}}$ via a
multiplication with the normal vectors at $\mathbf{v}_{i}$. Inspired by
[peng2022selfnerf], we then rely on a 4D hash grid to encode the combination
$[\mathbf{\hat{v}},\mathbf{d}]$. Note that our formulation differs from
[peng2022selfnerf], which encodes a 4D feature vector for every nearest
neighbor first and then blends the feature vectors afterward. Our
implementation encodes every $\mathbf{\hat{x}}$ only once.

With the above formulation, we can rewrite the point term
$\gamma(\mathbf{\hat{x}})$ in Equation 7 into
$\gamma([\mathbf{\hat{v}},\mathbf{d}])$. The surface term is formulated with
visibility priors, as discussed below.

###  3.3 Visibility Attention

In occluded videos, some parts of the human body may be more frequently
visible by the camera than others. As a result, more supervision is provided
for these highly visible parts which makes $\mathcal{F}$ fit on these _visible
areas_ much better. When conditioning on a wide range of surfaces, we hope to
pay more attention to the highly visible vertices than the hardly visible
ones. We achieve this through an attentive aggregation of the neighbor
vertices $\\{\mathbf{v}_{i}\\}$ via the function $\phi$ (Equation 7) based on
their visibility frequency.

Specifically, for each of the vertices $\mathbf{v}_{i}$, we maintain a
separate attention score $\mathbf{a}_{i}$ to be updated on-the-fly as the
training proceeds. Instead of recording the visibility frequency of all sample
points in the camera rays, only the termination point $\mathbf{t}$ per ray
should be considered. However, it is computationally expensive to find the
exact intersection point between the camera rays and the human body. We
approximate $\mathbf{t}$ as the sample point with the highest $\alpha$ along
each of the rays, such that
$\mathbf{t}=\mathbf{\hat{x}}_{\operatorname*{arg\,max}\\{\alpha\\}}$. For each
$\mathbf{t}$, we again rely on the $k$ nearest vertices $\\{\mathbf{v}_{i}\\}$
found earlier to determine the visible area on the body. At each training
step, for all neighbors $\\{\mathbf{v}_{i}\\}$ of every $\mathbf{t}$, we
increment their corresponding attention scores $\\{\mathbf{a}_{i}\\}$ by 1.
Taking visibility into account, Equation 7 can be reformulated as:

$\mathbf{c},\mathbf{\sigma}=\mathcal{F}(\
\underbrace{\gamma([\mathbf{\hat{v}},\mathbf{d}])}_{\text{point term}}\
\mathbin{\|}\
\underbrace{\frac{\sum^{k}_{i}\mathbf{a}_{i}\gamma([\mathbf{v}_{i},\mathbf{d}^{(\mathbf{v}_{i})}])}{\sum^{k}_{i}\mathbf{a}_{i}}}_{\text{surface
term}}\ ),$ (9)

where the hash grid encoding $\gamma(\cdot)$ is shared for both point and
surface. Recall that all vertices $\mathbf{v}$ have learnable coordinates, and
we compute their signed distance $\mathbf{d}^{(\mathbf{v}_{i})}$ w.r.t the
vertices on the initial SMPL mesh. The updating process of our visibility
attention is demonstrated in Figure 4.

Figure 4: Formulating visibility as attention. Highly visible body parts along
with associated parameterization vertices are expected to correspond to more
attention.

###  3.4 Loss Functions

Following HumanNeRF, we mainly supervise the training of OccNeRF through
pixel-wise photometric loss $\mathcal{L}_{\text{MSE}}$ and LPIPS
[zhang2018unreasonable] loss $\mathcal{L}_{\text{LPIPS}}$ to encourage high-
quality rendering at the visible parts. Unfortunately, these constraints do
not apply to the occluded parts, where supervisions are hardly available.
Hence we design another constraint to explicitly penalize renderings with
incomplete geometry and encourage high-density values within the human body.
The previously computed signed distances $\mathbf{d}$ are good approximations
of the position of ray samples w.r.t the SMPL mesh. Instead of only enforcing
the samples near the body surface, we apply the constraint to all samples with
negative $\mathbf{d}$. Our completeness loss $\mathcal{L}_{\text{comp}}$ is
therefore defined as:

$\mathcal{L}_{\text{comp}}=m\cdot\exp(\texttt{ReLU}(-\texttt{ReLU}(\sigma)+\beta)-\beta),$
(10)

where $m=1$ if $\mathbf{d}<0$ and $0$ otherwise, and $\beta=10$ is a hyper-
parameter. Intuitively, it is designed to penalize incompleteness inside the
human body. We use ReLU to clip negative $\sigma$ in the range of $[-\beta,0]$
and use exponential trick to decrease penalty for high densities. OccNeRF is
supervised by a weighted combination of the three losses:

$\lambda_{1}\mathcal{L}_{\text{MSE}}+\lambda_{2}\mathcal{L}_{\text{LPIPS}}+\lambda_{3}\mathcal{L}_{\text{comp}}.$
(11)

##  4 Experiments

###  4.1 Datasets

ZJU-MoCap [neuralbody]. This dataset contains humans performing a wide variety
of activities. Following HumanNeRF [weng2022humannerf], we mainly evaluate our
methods on the 6 subjects (377, 386, 387, 392, 393, 394) for direct
comparisons. Videos captured by _camera 1_ are used as training data, and the
other 22 cameras are used for evaluation. Since the MoCap data was captured in
a lab environment without the interference of any obstacles, we simulate
occlusions to be applied to the training videos. Without losing generality, we
simulate the presence of a box-like obstacle right in front of the camera that
causes a rectangular area at the center of the frame to be occluded. To do so,
we first determine a center point of the valid pixels from video frames, and
then mask out 50% of these pixels (demonstrated in Figure 1). Our simulated
obstacle and the occluded area are not intended to be moving along with the
subject. Since there is no obvious horizontal movement of subjects, we further
expect that they can move out of the occluded area for a short time and
therefore only apply the mask to 80% of the frames.

OcMotion [huang2022occluded]. This dataset contains humans interacting with
various objects, subject to real-world occlusions. There are a total of 48
videos, and each video was captured at 6 different camera poses. We evaluated
on 2 videos with different extents of occlusions. Specifically, we selected
540 frames from _video 14, camera 4_ and 500 frames from _video 11, camera 2_
as benchmarks for mild and severe real-world occlusions respectively. For both
benchmarks, we use the camera matrices, human body poses, and SMPL parameters
provided by the dataset, which were computed by [huang2021dynamic] directly on
the occluded videos. We provide more results in supplementary materials. We
also show the robustness of our method to inaccurately estimated priors.

Figure 5: Qualitative results on simulated occlusions in the ZJU-MoCap dataset
[neuralbody]. Figure 6: Qualitative results on real-world occlusions in
OcMotion dataset [huang2022occluded].

###  4.2 Comparison and Metrics

We mainly compare our method with HumanNeRF [weng2022humannerf], the state-of-
the-art human rendering method. We also compare against a baseline method
Neural Body [neuralbody] in supplementary materials. Note that all methods use
identical prior information, including pre-computed binary human mask and
SMPL/camera parameters. The extra visibility prior used in OccNeRF can be
calculated from the videos.

Methods are compared qualitatively and quantitatively. For qualitative
evaluations, we directly visualize novel views to assess the quality of the
renderings. For quantitative evaluations, we rely on the commonly used peak
signal-to-noise ratio (PSNR) and structural similarity (SSIM) metrics
[neuralbody, weng2022humannerf, peng2022selfnerf]. Previous methods computed
these metrics on full-scale images, which contain a majority of transparent
backgrounds. These regions are identical between predictions and references,
which inflate the overall metrics. To focus on the quality of rendered humans,
we compute metrics on the pixels with non-zero accumulated $\alpha$. For
OcMotion, since there is no ground truth for real-world occlusions, we compute
the metrics on the visible area only. We refer to the standard metrics as
PSNR${}_{\text{full}}$/SSIM${}_{\text{full}}$ and modified metrics as
PSNR${}_{\text{vis}}$/SSIM${}_{\text{vis}}$.

###  4.3 Implementation Details

Using the loss formulated in Equation 11, we optimize OccNeRF with the Adam
optimizer [kingma2014adam]. We set the learning rate to $5\times 10^{-4}$ for
the regression MLP $\mathcal{F}$, $1\times 10^{-4}$ for the parameterization
vertices $\mathbf{v}$, and $5\times 10^{-5}$ for the rest. $\lambda_{1}$,
$\lambda_{2}$, and $\lambda_{3}$ were set to 0.2, 1.0, and 10.0 respectively.
We adopted patch-wise sampling of rays, each with 128 sample points. Due to
the usage of the hash grid, OccNeRF converges faster than HumanNeRF. As a
result, we trained our models for only 10K iterations while HumanNeRF models
for 40K iterations.

###  4.4 Results on Simulated Occlusions

Qualitative comparison on ZJU-MoCap videos with simulated occlusions between
HumanNeRF and OccNeRF is shown in Figure 5. OccNeRF is capable of rendering a
mostly completed body geometry with sensible details filled in at occluded
areas. On the contrary, HumanNeRF fails to recover occluded body parts and
produces significant artifacts in the occluded areas. Additionally, the
quantitative results in Table 1 show that OccNeRF surpasses HumanNeRF for all
subjects and under both metrics by a great margin. Note that this
straightforward simulation of occlusions is in fact uncommon in real-world
settings, where obstacles should have various shapes and humans are able to
move across the entire scene with interactions with obstacles. _More
comparisons against Neural Body[neuralbody] can be found in supplementary
materials._

###  4.5 Results on Real-world Occlusions

For better validating on real-world scenes, we present the rendering results
on OcMotion videos in Figure 6. For the video with mild occlusions (top row),
OccNeRF outperforms HumanNeRF with a higher fidelity rendering of texture
details and much fewer artifacts at non-human regions. For the video with
severe occlusions (bottom row), OccNeRF is still able to generate novel views
with high-level rendering quality. However, HumanNeRF fails completely in such
challenging cases when most body parts are occluded. This validates the
superiority of OccNeRF in real-world scenes. OccNeRF also exceeds HumanNeRF on
quantitative benchmarks as indicated in Table 1. Note that the metrics were
computed on visible pixels in training images only, which ignored most of the
artifacts HumanNeRF generated. _More comparisons on real-world scenes can be
found in supplementary materials._

ZJU-MoCap | Subject 377 | Subject 386
---|---|---
PSNR${}_{\text{vis}}$ | SSIM${}_{\text{vis}}$ | PSNR${}_{\text{full}}$ | SSIM${}_{\text{full}}$ | PSNR${}_{\text{vis}}$ | SSIM${}_{\text{vis}}$ | PSNR${}_{\text{full}}$ | SSIM${}_{\text{full}}$
HumanNeRF [weng2022humannerf] | 11.29 | 0.5649 | 22.15 | 0.9612 | 9.491 | 0.4877 | 19.89 | 0.9531
OccNeRF |  best_color13.23 |  best_color0.6097 |  best_color23.43 |  best_color0.9642 |  best_color13.44 |  best_color0.5974 |  best_color23.66 |  best_color0.9639
ZJU-MoCap | Subject 387 | Subject 392
PSNR${}_{\text{vis}}$ | SSIM${}_{\text{vis}}$ | PSNR${}_{\text{full}}$ | SSIM${}_{\text{full}}$ | PSNR${}_{\text{vis}}$ | SSIM${}_{\text{vis}}$ | PSNR${}_{\text{full}}$ | SSIM${}_{\text{full}}$
HumanNeRF [weng2022humannerf] | 9.551 | 0.4140 | 19.47 | 0.9408 | 11.04 | 0.5290 | 21.01 | 0.9543
OccNeRF |  best_color13.27 |  best_color0.5243 |  best_color22.26 |  best_color0.9513 |  best_color13.00 |  best_color0.5692 |  best_color22.13 |  best_color0.9575
ZJU-MoCap | Subject 393 | Subject 394
PSNR${}_{\text{vis}}$ | SSIM${}_{\text{vis}}$ | PSNR${}_{\text{full}}$ | SSIM${}_{\text{full}}$ | PSNR${}_{\text{vis}}$ | SSIM${}_{\text{vis}}$ | PSNR${}_{\text{full}}$ | SSIM${}_{\text{full}}$
HumanNeRF [weng2022humannerf] | 10.86 | 0.4483 | 20.92 | 0.9476 | 10.55 | 0.4764 | 20.56 | 0.9489
OccNeRF |  best_color12.00 |  best_color0.4655 |  best_color21.58 |  best_color0.9489 |  best_color13.12 |  best_color0.5317 |  best_color22.06 |  best_color0.9532
OcMotion | Video Mild | Video Severe
PSNR${}_{\text{vis}}$ | SSIM${}_{\text{vis}}$ | PSNR${}_{\text{full}}$ | SSIM${}_{\text{full}}$ | PSNR${}_{\text{vis}}$ | SSIM${}_{\text{vis}}$ | PSNR${}_{\text{full}}$ | SSIM${}_{\text{full}}$
HumanNeRF [weng2022humannerf] | 13.38 | 0.6544 | 21.18 | 0.9680 | 11.40 | 0.4545 | 17.96 | 0.9470
OccNeRF |  best_color14.56 |  best_color0.6814 |  best_color21.50 |  best_color0.9695 |  best_color14.95 |  best_color0.5998 |  best_color21.16 |  best_color0.9692
Table 1: Quantitative comparison on the ZJU-MoCap and OcMotion datasets. We
color cells that have the best metric values.

###  4.6 Ablation Studies

In this section, we conduct additional experiments by simply removing each of
the proposed components from the OccNeRF framework to prove their
effectiveness. Quantitative metrics are also presented in the figures. _More
ablation studies can be found in supplementary materials._

Figure 7: Our visibility attention improves rendering quality with more
confident predictions at occluded areas with fewer blurs.

Impact of Visibility Attention. Our ablation studies start by proving the
benefits of reformulating visibility priors as attention maps to be applied
during surface-based rendering. Figure 7 shows that when disabling the
attentive aggregation from Equation 9, the model becomes less confident in
occluded areas, resulting in more blurs.

Figure 8: Our $\mathcal{L}_{\text{comp}}$ improves geometry completeness a
step further when combined with the proposed rendering strategy.

Impact of $\mathcal{L}_{\text{comp}}$. The proposed completeness loss
$\mathcal{L}_{\text{comp}}$ is designed to encourage high-density values at
locations inside the SMPL mesh. When removing this loss, Figure 8 shows that
our method cannot render a complete geometry anymore. However, with our
surface-based rendering, we still yield better results than HumanNeRF.

Impact of Surface-based Rendering. As discussed in section 3.2, we claimed
that our proposed rendering strategy enables $\mathcal{F}(\cdot)$ to condition
on inputs better with more overlaps. Here we validate the necessity of such a
design by removing it from the framework. We, however, still keep the hash
grid encoding to see its impact. According to Figure 9, the hash grid encoding
alone is not able to achieve comparable performance to our full OccNeRF. It
has to be equipped together with the proposed rendering strategy. This
validates that major performance improvements do come from surface-based
rendering.

##  5 Discussions and Conclusion

Discussions. It is difficult to optimize scene-specific neural radiance fields
under occlusions. There is neither a ground truth for the occluded parts nor
external information from different scenes to inpaint the missing area.
OccNeRF achieves rendering of the occluded regions by referring to nearby
visible correspondences and enforcing complete geometry. However, OccNeRF can
yield subtle artifacts. This is because we have more parameters to optimize
and fewer training data due to occlusions. Since no external information is
accessible, OccNeRF is not capable of inpainting an area that has never been
seen in the video. The above limitations can be overcome with a better
geometry prior [jiang2022selfrecon] and a cross-scene training strategy
[chen2022gpnerf]. Although the hash grid encoding accelerates the convergence
at training, OccNeRF runs relatively slower than HumanNeRF at inference.

Figure 9: Our surface-based rendering method fills in the occluded parts with
both accurate geometry and appropriate appearance.

Conclusion. We proposed OccNeRF for rendering humans from object-occluded
monocular videos. Most existing methods assume clear views of the entire human
body without any interference, which is not feasible in real-world scenes. We
designed a surface-based rendering strategy that incorporates geometry and
visibility priors to assist rendering under occlusions. Moreover, our novel
loss function is also able to help maintain geometry completeness. In our
experiments, we compared OccNeRF against the state-of-the-art method under
both simulated and real-world video occlusions. Our state-of-the-art results
set up a new benchmark in this field of research.

#### Acknowledgments.

This work was partially funded by the Gordon and Betty Moore Foundation,
Panasonic Holdings Corporation, NSF RI #2211258, and Stanford HAI.

## References

  * [1] Thiemo Alldieck, Marcus Magnor, Weipeng Xu, Christian Theobalt, and Gerard Pons-Moll.  Video based reconstruction of 3d people models.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8387–8397, 2018.
  * [2] Jonathan T Barron, Ben Mildenhall, Matthew Tancik, Peter Hedman, Ricardo Martin-Brualla, and Pratul P Srinivasan.  Mip-nerf: A multiscale representation for anti-aliasing neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5855–5864, 2021.
  * [3] Jonathan T Barron, Ben Mildenhall, Dor Verbin, Pratul P Srinivasan, and Peter Hedman.  Mip-nerf 360: Unbounded anti-aliased neural radiance fields.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5470–5479, 2022.
  * [4] Alexander W Bergman, Petr Kellnhofer, Yifan Wang, Eric R Chan, David B Lindell, and Gordon Wetzstein.  Generative neural articulated radiance fields.  arXiv preprint arXiv:2206.14314, 2022.
  * [5] Benjamin Biggs, David Novotny, Sebastien Ehrhardt, Hanbyul Joo, Ben Graham, and Andrea Vedaldi.  3d multi-bodies: Fitting sets of plausible 3d human models to ambiguous image data.  Advances in Neural Information Processing Systems, 33:20496–20507, 2020.
  * [6] Federica Bogo, Angjoo Kanazawa, Christoph Lassner, Peter Gehler, Javier Romero, and Michael J. Black.  Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image.  In Computer Vision – ECCV 2016, Lecture Notes in Computer Science. Springer International Publishing, Oct. 2016.
  * [7] Jianchuan Chen, Ying Zhang, Di Kang, Xuefei Zhe, Linchao Bao, Xu Jia, and Huchuan Lu.  Animatable neural radiance fields from monocular rgb videos.  arXiv preprint arXiv:2106.13629, 2021.
  * [8] Mingfei Chen, Jianfeng Zhang, Xiangyu Xu, Lijuan Liu, Yujun Cai, Jiashi Feng, and Shuicheng Yan.  Geometry-guided progressive nerf for generalizable and efficient neural human rendering.  In ECCV, 2022.
  * [9] Xu Chen, Tianjian Jiang, Jie Song, Jinlong Yang, Michael J Black, Andreas Geiger, and Otmar Hilliges.  gdna: Towards generative detailed neural avatars.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 20427–20437, 2022.
  * [10] Xingyu Chen, Qi Zhang, Xiaoyu Li, Yue Chen, Ying Feng, Xuan Wang, and Jue Wang.  Hallucinated neural radiance fields in the wild.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12943–12952, 2022.
  * [11] Julian Chibane, Thiemo Alldieck, and Gerard Pons-Moll.  Implicit functions in feature space for 3d shape reconstruction and completion.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6970–6981, 2020.
  * [12] Alvaro Collet, Ming Chuang, Pat Sweeney, Don Gillett, Dennis Evseev, David Calabrese, Hugues Hoppe, Adam Kirk, and Steve Sullivan.  High-quality streamable free-viewpoint video.  ACM Transactions on Graphics (ToG), 34(4):1–13, 2015.
  * [13] Enric Corona, Albert Pumarola, Guillem Alenya, Gerard Pons-Moll, and Francesc Moreno-Noguer.  Smplicit: Topology-aware generative model for clothed people.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11875–11885, 2021.
  * [14] Boyang Deng, John P Lewis, Timothy Jeruzalski, Gerard Pons-Moll, Geoffrey Hinton, Mohammad Norouzi, and Andrea Tagliasacchi.  Nasa neural articulated shape approximation.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part VII 16, pages 612–628. Springer, 2020.
  * [15] Mingsong Dou, Sameh Khamis, Yury Degtyarev, Philip Davidson, Sean Ryan Fanello, Adarsh Kowdle, Sergio Orts Escolano, Christoph Rhemann, David Kim, Jonathan Taylor, et al.  Fusion4d: Real-time performance capture of challenging scenes.  ACM Transactions on Graphics (ToG), 35(4):1–13, 2016.
  * [16] Mark Endo, Kathleen L Poston, Edith V Sullivan, Li Fei-Fei, Kilian M Pohl, and Ehsan Adeli.  Gaitforemer: Self-supervised pre-training of transformers via human motion forecasting for few-shot gait impairment severity estimation.  In Medical Image Computing and Computer Assisted Intervention–MICCAI 2022: 25th International Conference, Singapore, September 18–22, 2022, Proceedings, Part VIII, pages 130–139. Springer, 2022\.
  * [17] Yao Feng, Jinlong Yang, Marc Pollefeys, Michael J Black, and Timo Bolkart.  Capturing and animation of body and clothing from monocular video.  arXiv preprint arXiv:2210.01868, 2022.
  * [18] Chen Gao, Ayush Saraf, Johannes Kopf, and Jia-Bin Huang.  Dynamic view synthesis from dynamic monocular video.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5712–5721, 2021.
  * [19] Beerend GA Gerats, Jelmer M Wolterink, and Ivo AMJ Broeders.  3d human pose estimation in multi-view operating room videos using differentiable camera projections.  Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization, pages 1–9, 2022.
  * [20] Tong He, John Collomosse, Hailin Jin, and Stefano Soatto.  Geo-pifu: Geometry and pixel aligned implicit functions for single-view human reconstruction.  Advances in Neural Information Processing Systems, 33:9276–9287, 2020.
  * [21] Tong He, Yuanlu Xu, Shunsuke Saito, Stefano Soatto, and Tony Tung.  Arch++: Animation-ready clothed human reconstruction revisited.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 11046–11056, 2021.
  * [22] Peter Hedman, Pratul P Srinivasan, Ben Mildenhall, Jonathan T Barron, and Paul Debevec.  Baking neural radiance fields for real-time view synthesis.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5875–5884, 2021.
  * [23] Yang Hong, Juyong Zhang, Boyi Jiang, Yudong Guo, Ligang Liu, and Hujun Bao.  Stereopifu: Depth aware clothed human digitization via stereo vision.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 535–545, 2021.
  * [24] Buzhen Huang, Yuan Shu, Jingyi Ju, and Yangang Wang.  Occluded human body capture with self-supervised spatial-temporal motion prior.  arXiv preprint arXiv:2207.05375, 2022.
  * [25] Buzhen Huang, Yuan Shu, Tianshu Zhang, and Yangang Wang.  Dynamic multi-person mesh recovery from uncalibrated multi-view cameras.  In 2021 International Conference on 3D Vision (3DV), pages 710–720. IEEE, 2021.
  * [26] Zeng Huang, Yuanlu Xu, Christoph Lassner, Hao Li, and Tony Tung.  Arch: Animatable reconstruction of clothed humans.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3093–3102, 2020.
  * [27] Boyi Jiang, Yang Hong, Hujun Bao, and Juyong Zhang.  Selfrecon: Self reconstruction your digital avatar from monocular video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5605–5615, 2022.
  * [28] Tianjian Jiang, Xu Chen, Jie Song, and Otmar Hilliges.  Instantavatar: Learning avatars from monocular video in 60 seconds.  arXiv preprint arXiv:2212.10550, 2022.
  * [29] Wen Jiang, Nikos Kolotouros, Georgios Pavlakos, Xiaowei Zhou, and Kostas Daniilidis.  Coherent reconstruction of multiple humans from a single image.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5579–5588, 2020.
  * [30] Wei Jiang, Kwang Moo Yi, Golnoosh Samei, Oncel Tuzel, and Anurag Ranjan.  Neuman: Neural human radiance field from a single video.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII, pages 402–418. Springer, 2022.
  * [31] Angjoo Kanazawa, Jason Y. Zhang, Panna Felsen, and Jitendra Malik.  Learning 3d human dynamics from video.  In Computer Vision and Pattern Recognition (CVPR), 2019.
  * [32] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [33] Muhammed Kocabas, Nikos Athanasiou, and Michael J. Black.  Vibe: Video inference for human body pose and shape estimation.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.
  * [34] Muhammed Kocabas, Chun-Hao P Huang, Otmar Hilliges, and Michael J Black.  Pare: Part attention regressor for 3d human body estimation.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11127–11137, 2021.
  * [35] Ruilong Li, Julian Tanke, Minh Vo, Michael Zollhöfer, Jürgen Gall, Angjoo Kanazawa, and Christoph Lassner.  Tava: Template-free animatable volumetric actors.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII, pages 419–436. Springer, 2022.
  * [36] Zhengqi Li, Simon Niklaus, Noah Snavely, and Oliver Wang.  Neural scene flow fields for space-time view synthesis of dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6498–6508, 2021.
  * [37] Lingjie Liu, Marc Habermann, Viktor Rudnev, Kripasindhu Sarkar, Jiatao Gu, and Christian Theobalt.  Neural actor: Neural free-view synthesis of human actors with pose control.  ACM Transactions on Graphics (TOG), 40(6):1–16, 2021.
  * [38] Yuan Liu, Sida Peng, Lingjie Liu, Qianqian Wang, Peng Wang, Christian Theobalt, Xiaowei Zhou, and Wenping Wang.  Neural rays for occlusion-aware image-based rendering.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7824–7833, 2022.
  * [39] Stephen Lombardi, Tomas Simon, Jason Saragih, Gabriel Schwartz, Andreas Lehrmann, and Yaser Sheikh.  Neural volumes: Learning dynamic renderable volumes from images.  arXiv preprint arXiv:1906.07751, 2019.
  * [40] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  SMPL: A skinned multi-person linear model.  ACM Trans. Graphics (Proc. SIGGRAPH Asia), 34(6):248:1–248:16, Oct. 2015.
  * [41] Guillaume Loubet, Nicolas Holzschuch, and Wenzel Jakob.  Reparameterizing discontinuous integrands for differentiable rendering.  Transactions on Graphics (Proceedings of SIGGRAPH Asia), 38(6), Dec. 2019.
  * [42] Wojciech Matusik, Chris Buehler, Ramesh Raskar, Steven J Gortler, and Leonard McMillan.  Image-based visual hulls.  In Proceedings of the 27th annual conference on Computer graphics and interactive techniques, pages 369–374, 2000.
  * [43] Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng.  Nerf: Representing scenes as neural radiance fields for view synthesis.  Communications of the ACM, 65(1):99–106, 2021.
  * [44] Thomas Müller, Alex Evans, Christoph Schied, and Alexander Keller.  Instant neural graphics primitives with a multiresolution hash encoding.  arXiv preprint arXiv:2201.05989, 2022.
  * [45] Atsuhiro Noguchi, Xiao Sun, Stephen Lin, and Tatsuya Harada.  Neural articulated radiance field.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5762–5772, 2021.
  * [46] Julian Ost, Fahim Mannan, Nils Thuerey, Julian Knodt, and Felix Heide.  Neural scene graphs for dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2856–2865, 2021.
  * [47] Keunhong Park, Utkarsh Sinha, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Steven M Seitz, and Ricardo Martin-Brualla.  Nerfies: Deformable neural radiance fields.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 5865–5874, 2021.
  * [48] Keunhong Park, Utkarsh Sinha, Peter Hedman, Jonathan T Barron, Sofien Bouaziz, Dan B Goldman, Ricardo Martin-Brualla, and Steven M Seitz.  Hypernerf: A higher-dimensional representation for topologically varying neural radiance fields.  arXiv preprint arXiv:2106.13228, 2021.
  * [49] Bo Peng, Jun Hu, Jingtao Zhou, and Juyong Zhang.  Selfnerf: Fast training nerf for human from monocular self-rotating video.  arXiv preprint arXiv:2210.01651, 2022.
  * [50] Sida Peng, Junting Dong, Qianqian Wang, Shangzhan Zhang, Qing Shuai, Xiaowei Zhou, and Hujun Bao.  Animatable neural radiance fields for modeling dynamic human bodies.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14314–14323, 2021.
  * [51] Sida Peng, Yuanqing Zhang, Yinghao Xu, Qianqian Wang, Qing Shuai, Hujun Bao, and Xiaowei Zhou.  Neural body: Implicit neural representations with structured latent codes for novel view synthesis of dynamic humans.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9054–9063, 2021.
  * [52] Albert Pumarola, Enric Corona, Gerard Pons-Moll, and Francesc Moreno-Noguer.  D-nerf: Neural radiance fields for dynamic scenes.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10318–10327, 2021.
  * [53] Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas.  Pointnet: Deep learning on point sets for 3d classification and segmentation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652–660, 2017.
  * [54] Chris Rockwell and David F Fouhey.  Full-body awareness from partial observations.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XVII 16, pages 522–539. Springer, 2020.
  * [55] Shunsuke Saito, Zeng Huang, Ryota Natsume, Shigeo Morishima, Angjoo Kanazawa, and Hao Li.  Pifu: Pixel-aligned implicit function for high-resolution clothed human digitization.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 2304–2314, 2019.
  * [56] István Sárándi, Timm Linder, Kai O Arras, and Bastian Leibe.  How robust is 3d human pose estimation to occlusion?  arXiv preprint arXiv:1808.09316, 2018.
  * [57] Pratul P Srinivasan, Boyang Deng, Xiuming Zhang, Matthew Tancik, Ben Mildenhall, and Jonathan T Barron.  Nerv: Neural reflectance and visibility fields for relighting and view synthesis.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7495–7504, 2021.
  * [58] Cheng Sun, Min Sun, and Hwann-Tzong Chen.  Direct voxel grid optimization: Super-fast convergence for radiance fields reconstruction.  CVPR, 2022.
  * [59] Yu Sun, Qian Bao, Wu Liu, Yili Fu, Michael J Black, and Tao Mei.  Monocular, one-stage, regression of multiple 3d people.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 11179–11188, 2021.
  * [60] Yu Sun, Wu Liu, Qian Bao, Yili Fu, Tao Mei, and Michael J Black.  Putting people in their place: Monocular regression of 3d people in depth.  In CVPR, 2022.
  * [61] Matthew Tancik, Pratul Srinivasan, Ben Mildenhall, Sara Fridovich-Keil, Nithin Raghavan, Utkarsh Singhal, Ravi Ramamoorthi, Jonathan Barron, and Ren Ng.  Fourier features let networks learn high frequency functions in low dimensional domains.  Advances in Neural Information Processing Systems, 33:7537–7547, 2020.
  * [62] Garvita Tiwari, Nikolaos Sarafianos, Tony Tung, and Gerard Pons-Moll.  Neural-gif: Neural generalized implicit functions for animating people in clothing.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 11708–11718, 2021.
  * [63] Dor Verbin, Peter Hedman, Ben Mildenhall, Todd Zickler, Jonathan T Barron, and Pratul P Srinivasan.  Ref-nerf: Structured view-dependent appearance for neural radiance fields.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5481–5490. IEEE, 2022.
  * [64] Shaofei Wang, Katja Schwarz, Andreas Geiger, and Siyu Tang.  Arah: Animatable volume rendering of articulated human sdfs.  In Computer Vision–ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23–27, 2022, Proceedings, Part XXXII, pages 1–19. Springer, 2022.
  * [65] Chung-Yi Weng, Brian Curless, and Ira Kemelmacher-Shlizerman.  Vid2actor: Free-viewpoint animatable person synthesis from video in the wild.  arXiv preprint arXiv:2012.12884, 2020.
  * [66] Chung-Yi Weng, Brian Curless, Pratul P Srinivasan, Jonathan T Barron, and Ira Kemelmacher-Shlizerman.  Humannerf: Free-viewpoint rendering of moving people from monocular video.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16210–16220, 2022.
  * [67] Chung-Yi Weng, Pratul P Srinivasan, Brian Curless, and Ira Kemelmacher-Shlizerman.  Personnerf: Personalized reconstruction from photo collections.  arXiv preprint arXiv:2302.08504, 2023.
  * [68] Hongyi Xu, Thiemo Alldieck, and Cristian Sminchisescu.  H-nerf: Neural radiance fields for rendering and temporal reconstruction of humans in motion.  Advances in Neural Information Processing Systems, 34:14955–14966, 2021.
  * [69] Kaibing Yang, Renshu Gu, Maoyu Wang, Masahiro Toyoura, and Gang Xu.  Lasor: Learning accurate 3d human pose and shape via synthetic occlusion-aware data and neural mesh rendering.  IEEE Transactions on Image Processing, 31:1938–1948, 2022.
  * [70] Ze Yang, Shenlong Wang, Sivabalan Manivasagam, Zeng Huang, Wei-Chiu Ma, Xinchen Yan, Ersin Yumer, and Raquel Urtasun.  S3: Neural shape, skeleton, and skinning fields for 3d human modeling.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 13284–13293, 2021.
  * [71] Jianfeng Zhang, Dongdong Yu, Jun Hao Liew, Xuecheng Nie, and Jiashi Feng.  Body meshes as points.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 546–556, 2021.
  * [72] Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 586–595, 2018.
  * [73] Tianshu Zhang, Buzhen Huang, and Yangang Wang.  Object-occluded human shape and pose estimation from a single color image.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 7376–7385, 2020.