# MetaPose: Fast 3D Pose from Multiple Views without 3D Supervision

Ben Usman1,2
<EMAIL_ADDRESS>Andrea Tagliasacchi2,3
<EMAIL_ADDRESS>Kate Saenko1,4
<EMAIL_ADDRESS>Avneesh Sud2
<EMAIL_ADDRESS>1Boston University, 2Google Research, 3University of Toronto,
4MIT-IBM Watson AI Lab

###### Abstract

In the era of deep learning, human pose estimation from multiple cameras with
unknown calibration has received little attention to date. We show how to
train a neural model to perform this task with high precision and minimal
latency overhead. The proposed model takes into account joint location
uncertainty due to occlusion from multiple views, and requires only 2D
keypoint data for training. Our method outperforms both classical bundle
adjustment and weakly-supervised monocular 3D baselines on the well-
established Human3.6M dataset, as well as the more challenging in-the-wild
Ski-Pose PTZ dataset.

##  1 Introduction

We tackle the problem of estimating 3D coordinates of human joints from RGB
images captured using synchronized (potentially moving) cameras with unknown
positions, orientations, and intrinsic parameters. We additionally assume
having access to a training set with only 2D positions of joints labeled on
captured images.

Historically, real-time capture of the human 3D pose has been undertaken only
by large enterprises that could afford expensive specialized motion capture
equipment [12]. In principle, if camera calibrations are available [1], human
body joints can be triangulated directly from camera-space observations [19,
25]. One scenario in which camera calibration cannot easily be estimated is
sports capture, in which close-ups of players are captured in front of low-
texture backgrounds, with wide-baseline, moving cameras. Plain backgrounds
preclude calibration, as not sufficiently many feature correspondences can be
detected across views; see Figure 1.

Figure 1:  We show how to train a neural network that can aggregate outputs of
multiple single-view methods, takes prediction uncertainty into consideration,
has minimal latency overhead, and requires only 2D supervision for training.
Our method mimics the structure of bundle-adjustment solvers, but using the
joints of the human body to drive camera calibration, and by implementing a
bundle-like solver with a simple feed-forward neural network.

In this work, we propose a neural network to simultaneously predict 3D human
and relative camera poses from multiple views; see Figure 1. Our approach uses
human body joints as a source of information for camera calibration. As joints
often become occluded, uncertainty must be carefully accounted for, to avoid
bad calibration and consequent erroneous 3D pose predictions. As we assume a
synchronized multi-camera setup at test-time, our algorithm should also be
able to effectively aggregate information from different viewpoints. Finally,
our approach supervised by 2D annotations alone, as ground-truth annotation of
3D data is unwieldy. As summarized in Figure 2, and detailed in what follows,
none of the existing approaches fully satisfies these fundamental
requirements.

Fully-supervised 3D pose estimation approaches yield the lowest estimation
error, but make use of known 3D camera specification during either training
[45] or both training and inference [19]. However, the prohibitively high cost
of 3D joint annotation and full camera calibration in-the-wild makes it
difficult to acquire large enough labeled datasets representative of specific
environments [37, 22], therefore rendering supervised methods not applicable
in this setup.

Monocular 3D methods, [18, 27, 44] as well as 2D-to-3D lifting networks [6,
43], relax data constraints to enable 3D pose inference using just multi-view
2D data without calibration at train time. Unfortunately, at inference time,
these methods can only be applied to a single view at a time, therefore unable
to leverage cross-view information and uncertainty.

Classical SfM (structure from motion) approaches to 3D pose estimation [25]
iteratively refine both the camera and the 3D pose from noisy 2D observations.
However, these methods are often much slower than their neural counterparts,
since they have to perform several optimization steps during inference.
Further, most of them do not consider uncertainty estimates, resulting in sub-
par performance.

Figure 2:  Prior work — Existing solutions either require 3D annotations [19],
perform inference on a single view at a time [44], or ignore uncertainty in
joint coordinates due to occlusions [25].

To overcome these limitation we propose MetaPose; see Figure 1. Our method for
3D pose estimation aggregates pose predictions and uncertainty estimates
across multiple views, requires no 3D joint annotations or camera parameters
at both train and inference time, and adds very little latency to the
resulting pipeline.

Overall, we propose the feed-forward neural architecture that can accurately
estimate the 3D human pose and the relative cameras configuration from
multiple views, taking into account joint occlusions and prediction
uncertainties, and uses only 2D joint annotations for training. We employ an
off-the-shelf weakly-supervised 3D network to form an initial guess about the
pose and the camera setup, and a neural meta-optimizer that iteratively
refines this guess using 2D joint location probability heatmaps generated by
an off-the-shelf 2D pose estimation network. This modular approach not only
yields low estimation error, leading to state-of-the-art results on Human3.6M
[17] and Ski-Pose PTZ [37], but also has low latency, as inference within our
framework executes as a feed-forward neural network.

##  2 Related Work

In this section, we review multi-view 3D human pose estimation, while we refer
to Joo et al. [22] for a survey of 3D human pose estimation in the wild.

##### Full supervision

Supervised methods [19, 42, 7] yield the lowest 3D pose estimation errors on
multi-view single person [17] and multi-person [21, 2, 7] datasets, but
require precise camera calibration during both training and inference. Other
approaches [45] use datasets with full 3D annotations and a large number of
annotated cameras to train methods that can adapt to novel camera setups in
visually similar environments, therefore somewhat relaxing camera calibration
requirements. Martinez et al. [32] use pre-trained 2D pose networks [34] to
take advantage of existing datasets with 2D pose annotations. Epipolar
transformers [16] use only 2D keypoint supervision, but require camera
calibration to incorporate 3D information in the 2D feature extractors.

Figure 3:  Method – We illustrate our method with a simple 2D example of
regressing the 3D vertices of an equilateral triangle given multi-view
observations. (left) AniPose [25] performs classical bundle adjustment to
identify camera positions and 3D vertices that minimize reprojection error to
2D landmarks on the input images. Conversely, our technique emulates classical
bundle adjustment in a “neural” fashion by a meta-optimizer: first (middle),
the EpipolarPose [27] neural network obtains a per-frame 3D estimate of the
joints, which we co-align via procrustes to estimate an initialization of both
3D cameras and 3D joints; then (right), a neural network meta-optimizer
performs a bundle adjustment and refines both joints and cameras, using per-
view keypoint localization heatmaps as input. Additional prior information,
such as the fact that the triangle is equilateral, can be elegantly integrated
in the meta-optimizer training.

##### Weak and self-supervision

Several approaches exist that do not use paired 3D ground truth data. Many
augment limited 3D annotations with 2D labels [49, 24, 33, 46]. Fitting-based
methods [24, 28, 29, 46] jointly fit a statistical 3D human body model and 3D
human pose to monocular images or videos. Analysis-by-synthesis methods [36,
30, 20] learn to predict 3D human pose by estimating appearance in a novel
view. Most related to our work are approaches that exploit the structure of
multi-view image capture. EpipolarPose [27] uses epipolar geometry to obtain
3D pose estimates from multi-view 2D predictions, and subsequently uses them
to directly supervise 3D pose regression. Iqbal et al. [18] proposes a weakly-
supervised baseline to predict pixel coordinates of joints and their depth in
each view and penalized the discrepancy between rigidly aligned predictions
for different views during training. The self-supervised CanonPose [44]
further advances state-of-the-art by decoupling 3D pose estimation in
“canonical” frame. Drover et al. [9] learn a “dictionary” mapping 2D pose
projections into corresponding realistic 3D poses, using a large collection of
simulated 3D-to-2D projections. RepNet [43] and Chen et al. [6] train similar
“2D-to-3D lifting networks” with more realistic data constraints. While all
the aforementioned methods use multi-view consistency for training, they do
not allow pose inference from multiple images.

##### Iterative refinement

Estimating camera and pose simultaneously is a long-standing problem in vision
[38]. One of the more recent successful attempts is the work of Bridgeman et
al. [4] that proposed an end-to-end network that refines the initial
calibration guess using center points of multiple players in the field. In the
absence of such external calibration signals, Takahashi et al. [41] performs
bundle adjustment with bone length constraints, but do not report results on a
public benchmark. AniPose [25] performs joint 3D pose and camera refinement
using a modified version of the robust 3D registration algorithm of Zhou et
al. [48]. Such methods ignore predicted uncertainty for faster inference, but
robustly iteratively estimate outlier 2D observations and ignores them during
refinement. In Section 5, we show that these classical approaches struggle in
ill-defined settings, such as when we have a small number of cameras. More
recently, SPIN [29], HUND [47] and Holopose [14] incorporate iterative pose
refinement for monocular inputs, however, the refinement is tightly integrated
into the pose estimation network. MetaPose effectively regularizes the multi-
view pose estimation problem with a finite-capacity neural network resulting
in both faster inference and higher precision than the classical refinement.

##  3 Method

As illustrated in Figure 3, given a collection of $\\{\mathcal{I}_{c}\\}$
images, we seek to optimize, up to a global rotation, scale, and shift:

  * •

$\mathbf{J}=\\{\mathbf{j}_{j}\in{}^{3}\\}_{j=1}^{J}$: the 3D coordinates of 3D
body joints,

  * •

$\mathbf{C}=\\{\mathbf{c}_{c}\in{}^{P}\\}_{c=1}^{C}$: the cameras parameters.

Having also observed:

  * •

$\mathbf{H}=\\{\mathbf{h}_{c}\in{}^{J\times H\times W}\\}_{c=1}^{C}$: a set of
2D heatmaps of locations on images $\\{\mathcal{I}_{c}\\}$ captured using
these cameras,

And assuming that, at training time, we are provided with:

  * •

$\mathbf{K}=\\{\mathbf{k}_{j,c}\\}$: the ground truth 2D locations of the
projection of joint $\mathbf{j}_{j}$ in camera $\mathbf{c}_{c}$.

##### Bayesian model

Formally, assuming that heatmaps depend on camera parameters and joint
positions $\mathbf{J}$ only through 2D keypoint locations (i.e.
$p(\mathbf{H}|\mathbf{K},\mathbf{J},\mathbf{C})=p(\mathbf{H}|\mathbf{K})$),
the joint distribution can be factorized as:

$p(\mathbf{J},\mathbf{C},\mathbf{K},\mathbf{H})=p(\mathbf{H}|\mathbf{K})\,p(\mathbf{K}|\mathbf{J},\mathbf{C})\,p(\mathbf{J})\,p(\mathbf{C})$
(1)

Joints and keypoints are assumed to be related by:

$p(\mathbf{K}|\mathbf{J},\mathbf{C})=\prod_{j,c}\delta(\mathbf{k}_{j,c}-\pi(\mathbf{j}_{j},\mathbf{c}_{c}))$
(2)

where $\delta$ is the Dirac distribution, and $\pi(\mathbf{j},\mathbf{c})$
projects a joint $\mathbf{j}$ to the 2D coordinates in camera $\mathbf{c}$. We
use a weak-projection camera model , hence, each camera is defined by a tuple
of rotation matrix $\mathbf{R}$, pixel shift vector $\mathbf{t}$, and single
scale parameter $s$, i.e. $\mathbf{c}=[\mathbf{R},\mathbf{t},s]$, and the
projection operator is defined as
$\pi(\mathbf{j},(\mathbf{R},\mathbf{t},s))=s\cdot\mathbf{I}_{[0{:}1]}\cdot
R\cdot\mathbf{j}+t$ where $\mathbf{I}_{[0{:}1]}$ is a truncated identity
matrix that discards the third dimension of the multiplied vector. This choice
of the camera model simplifies initialization of camera parameters from
single-view estimates (Section 3.2) and eliminates the ambiguity in focal
length-vs-distance to the camera. In Section 5 we show experimentally what
fraction of the final error comes from this camera model.

##### Inference task

Our inference task is then to estimate the $\mathbf{J}$ and $\mathbf{C}$ from
observed heatmaps $\mathbf{H}$. We first introduce a probabilistic bundle
adjustment formulation to handle joint position uncertainty, then propose a
regression model that models complex interactions between joint positions and
observed heatmaps. The overall inference task can be framed as finding the
maximum of the posterior probability of the pose and camera parameters given
observed heatmaps, marginalized over possible keypoint locations:

$\max_{\mathbf{J},\mathbf{C}}\>p(\mathbf{J},\mathbf{C}|\mathbf{H})\\!=\\!\\!\\!\int\\!\\!\frac{p(\mathbf{k}|\mathbf{H})\,p(\mathbf{k}|\mathbf{J},\mathbf{C})\,p(\mathbf{J})\,p(\mathbf{C})}{p(\mathbf{k})}\mathop{}\\!\mathrm{d}\mathbf{k}$
(3)

where, assuming that no prior information over camera parameters, keypoint
locations, and poses is given (i.e. constant $p(\mathbf{C})$, $p(\mathbf{K})$
and $p(\mathbf{J})$) and using (2) we get:

$p(\mathbf{J},\mathbf{C}|\mathbf{H})\propto\prod_{c,j}p(\mathbf{k}_{j,c}=\pi(\mathbf{j}_{j},\mathbf{c}_{c})|\mathbf{H})$
(4)

Further, assuming that each keypoint $\mathbf{k}_{c,j}$ is affected only by a
corresponding heatmap $\mathbf{h}_{c,j}$, and more specifically that the
conditional probability density is proportional to the corresponding value of
the heatmap:

$p(\mathbf{k}_{j,c}|\mathbf{H})=p(\mathbf{k}_{j,c}|\mathbf{h}_{j,c})\propto\mathbf{h}_{j,c}[\mathbf{k}_{j,c}]$
(5)

we get a probabilistic bundle adjustment problem:

$\max_{\mathbf{J},\mathbf{C}}\>\prod_{c,j}\mathbf{h}_{j,c}[\pi(\mathbf{j}_{j},\mathbf{c}_{c})]$
(6)

As we will show in Section 5, better estimation accuracy with faster inference
time can be archived if assume that each keypoint can be affected by any
heatmap via the following functional relation up to a normally distributed
residual:

$p(\mathbf{K}|\mathbf{H},\theta)=\mathcal{N}(\mathbf{K}\>|\>\pi(\mathbf{J}_{\theta}(\mathbf{H}),\mathbf{C}_{\theta}(\mathbf{H})),\mathbf{I})$
(7)

where $\mathbf{J}_{\theta},\mathbf{C}_{\theta}$ are joint and camera
regression models (e.g. neural networks) parameterized by an unknown parameter
$\theta$, and $\mathcal{N}$ is a multivariate normal density. Parameters of
this model can be found via maximum likelihood estimation using observations
from $p(\mathbf{K},\mathbf{H})$ available during training

$\displaystyle\theta_{\text{MLE}}$
$\displaystyle=\arg\max_{\theta}\>\>p(\mathbf{H},\mathbf{K}|\theta)=\arg\max_{\theta}p(\mathbf{K}|\mathbf{H},\theta)$
(8)
$\displaystyle=\arg\min_{\theta}\>\>\mathbb{E}_{\mathbf{K},\mathbf{H}}\|\mathbf{K}-\pi(\mathbf{J}_{\theta}(\mathbf{H}),\mathbf{C}_{\theta}(\mathbf{H}))\|_{2}^{2}$
(9)

Then the test-time inference reduces to evaluation of the regression model at
given heatmaps:

$\arg\max_{\mathbf{J},\mathbf{C}}\>p(\mathbf{J},\mathbf{C}|\mathbf{H},\theta)=\mathbf{J}_{\theta}(\mathbf{H}),\mathbf{C}_{\theta}(\mathbf{H})$
(10)

Intuitively, the parametric objective enables complex interactions between all
observed heatmaps and all predicted joint locations. The resulting model
outperforms the probabilistic bundle adjustment both in terms of speed and
accuracy, as we show in Section 5.

##### Solver

To solve the highly non-convex problem in (9), and to do so efficiently, we
employ a modular two stages approach; see Figure 3:

Stage 1 (S1): Initialization – Section 3.2:

We first acquire an initial guess
$(\mathbf{J}_{\text{init}},\mathbf{C}_{\text{init}})$ using single-view 3D
pose estimates for the camera configuration and the 3D pose by applying rigid
alignment to per-view 3D pose estimates obtained using a pre-trained weakly-
supervised single-view 3D network, e.g. [27, 44]

Stage 2 (S2): Refinement – Section 3.3:

We then train a neural network $f_{\theta}$ to predict a series of refinement
steps for camera and pose, staring from the initial guess so to optimize (9).

##### Advantages

This approach has several key advantages:

  1. 1)

it primes the refinement stage with a “good enough” guess to start from the
correct basin of the highly non-convex pose likelihood objective given multi-
view heatmaps;

  2. 2)

it provides us with a modular framework, letting us swap pre-trained modules
for single-view 2D and 3D without re-training the entire pipeline whenever a
better approach becomes available;

  3. 3)

the neural optimizer provides orders of magnitude faster inference than
classical iterative refinement, and allows the entire framework to be written
within the same coherent computation framework (i.e. neural networks vs.
neural networks plus classical optimization).

###  3.1 Pre-processing

We assume that we have access to a 2D pose estimation model (e.g. PoseNet
[35]) that produces 2D localization heatmaps $\mathbf{h}_{j,c}$ for each joint
$j$ from RGB image $\mathcal{I}_{c}$. We approximate each heatmap
$\mathbf{h}_{j,c}$ with an $M$-component mixture of spherical Gaussians
$\mathbf{g}_{j,c}$. This compressed format reduces the dimensionality of the
input to the neural optimizer (Section 3.3). To fit parameters
$\mathbf{g}_{j,c}$ of a mixture of spherical Gaussians to a localization 2D
histogram $\mathbf{h}_{j,c}$, we treat the heatmap as a regular grid of
samples weighted by corresponding probabilities, and apply weighted EM
algorithm [11] directly to weighted samples, as described in the supplementary
Section 8.2.

##### Single-view pose estimation

To initialize camera parameters via rigid alignment (Section 3.2), we need a
single-image 3D pose estimation model trained without 3D supervision (e.g.
EpipolarPose [27]) that produces per-camera rough 3D pose estimates
$\mathbf{Q}=\\{\mathbf{q}_{c,j}\\}$ given an image $\mathcal{I}_{c}$ from that
camera. These single-image estimates $\mathbf{q}_{c,j}$ are assumed to be in
the camera frame, meaning that first two spatial coordinates of
$\mathbf{q}_{c,j}$ correspond to pixel coordinates of joint $j$ on image
$\mathcal{I}_{c}$, and the third coordinate corresponds to its single-image
relative zero-mean depth estimate.

Figure 4:  Initialization – We form an initial guess for the 3D pose and the
cameras by taking the mean of rigid aligned 3D poses estimated from each RGB
image using an external single-view weakly-supervised 3D pose estimation
network [27, 44].

Figure 5:  Refinement – We train a neural optimizer $f_{\theta}$ to predict
iterative refinement that minimizes the reprojection error with the ground
truth re-projection, using the current guess and joint heatmaps as an input.
During inference, we do not need ground truth 2D projections.

###  3.2 Initialization – Figure 4

The goal of this stage is to acquire an initial guess for the 3D pose and
cameras $(\mathbf{J}_{\text{init}},\mathbf{C}_{\text{init}})$ using single-
view rough camera-frame 3D pose estimates $\mathbf{Q}$ made by a model trained
without 3D supervision [27, 44]. We assume fixed initial parameters of the
first camera

$\displaystyle\mathbf{c}_{0}^{\text{init}}=(\mathbf{R}^{\text{init}}_{0},\mathbf{t}^{\text{init}}_{0},s^{\text{init}}_{0})=(\mathbf{I},\bar{0},1)$
(11)

and define initial estimates of rotations, scales and translations of
remaining cameras as solutions the following orthogonal rigid alignment
problem:

$\displaystyle\operatorname*{arg\,min}_{\mathbf{R}_{c},\mathbf{t}_{c},s_{c}}\sum_{j}\|\mathbf{q}_{c,j}-(s_{c}\cdot\mathbf{R}_{c}\cdot\mathbf{q}_{0,j}+\mathbf{I}_{[0:1]}^{T}\cdot\mathbf{t}_{c})\|^{2}$
(12)

that can be solved using SVD of the outer product of mean-centered 3D poses
[40]. The initial guess for the 3D pose $\mathbf{J}_{\text{init}}$ then is the
average of single-view 3D pose predictions $\mathbf{Q}$ rigidly aligned back
into the first camera frame by corresponding estimated optimal rotations,
scales and shifts:

$\displaystyle\mathbf{J}^{\text{init}}=\tfrac{1}{C}\sum_{c}\mathbf{R}_{c}^{T}\cdot(\mathbf{q}_{c}-\mathbf{I}_{[0:1]}^{T}\cdot\mathbf{t}_{c}))/s_{c}$
(13)

###  3.3 Refinement – Figure 5

We train a neural network $f_{\theta}$ to predict a series of updates to 3D
pose and camera estimates that leads to a refined estimate starting from the
initialization from Section 3.2:

$\displaystyle\mathbf{J}^{(i+1)}$
$\displaystyle=\mathbf{J}^{(i)}+\mathop{}\\!\mathrm{d}\mathbf{J}^{(i)},$
$\displaystyle\mathbf{J}^{(0)}=\mathbf{J}_{\text{init}}$ (14)
$\displaystyle\mathbf{C}^{(i+1)}$
$\displaystyle=\mathbf{C}^{(i)}+\mathop{}\\!\mathrm{d}\mathbf{C}^{(i)},$
$\displaystyle\mathbf{C}^{(0)}=\mathbf{C}_{\text{init}}.$ (15)

To ensure that inferred camera parameters $\mathbf{C}$ stay valid under any
update $\mathop{}\\!\mathrm{d}\mathbf{C}$ predicted by a network, camera scale
(always positive) is represented in log-scale, and camera rotation uses a
continuous 6D representation [50].

At each refinement step
$\mathop{}\\!\mathrm{d}\mathbf{J}^{(i)},\mathop{}\\!\mathrm{d}\mathbf{C}^{(i)}=\mathcal{F}_{\theta}^{(i)}(\dots)$
the sub-network $\mathcal{F}_{\theta}^{(i)}$ of the overall network
$f_{\theta}$ is provided with as much information as possible to perform a
meaningful update towards the optimal solution:

  * •

$(\mathbf{J}^{(i)},\mathbf{C}^{(i)})$ – the current estimate to be refined;

  * •

$\mathbf{G}{=}\\{\mathbf{g}_{j,c}\\}$ – a collection of Gaussian mixtures
compactly representing the heatmaps density distributions;

  * •

$\mathbf{K}^{(i)}{=}\\{\mathbf{k}_{j,c}^{(i)}=\pi(\mathbf{j}_{j}^{(i)},\mathbf{c}_{c}^{(i)})\\}$
– the set of projections of each joint $\mathbf{j}{{}^{(i)}}$ into each camera
frame $\mathbf{c}^{(i)}$;

  * •

$\mathcal{L}(\mathbf{J}^{(i)},\mathbf{C}^{(i)}|\mathbf{G})$ – the likelihood
of the current estimate of joints given the heatmap mixture parameters.

These learnt updates seek to minimize the L2 distance between predicted and
ground truth 2D coordinates of keypoints in each frame, mirroring the maximum
likelihood objective (9) we defined earlier:

$\arg\min_{\theta}\>\mathcal{L}_{\mathbf{k}}(\theta)=\sum_{(i)}\sum_{j,c}\|\mathbf{k}_{j,c}^{(i+1)}-\mathbf{k}_{j,c}^{\text{gt}}\|_{2}^{2}$
(16)

where, in practice, we train refinement steps
$\mathcal{\mathcal{F}_{\theta}}^{(i)}$ progressively, one after the other.

Figure 6:  Architecture – In order for predicted updates to respect symmetries
of the problem at hand, we copy and concatenate view-invariant inputs (current
pose estimate, average heatmap likelihood - dashed line) to each row of view-
specific inputs (current cameras and joint projections, heatmaps), pass them
though a Permutation-Equivariant MLP Block shown above. To get permutation-
invariant final pose update we additionally apply MLP to averaged output pose
embeddings.

##### Architecture design

The architecture of $\mathcal{F}_{\theta}$ needs to be very carefully designed
to respect the symmetries of the problem at hand. The inferred updates to
$\mathbf{J}^{(i+1)}$ ought to be invariant to the order of cameras, while
updates to $\mathbf{C}^{(i+1)}$ ought to be permutation-equivariant w.r.t. the
current estimates of $\mathbf{C}^{(i)}$, rows of $\mathbf{K}^{(i)}$, and
Gaussian mixtures $\mathbf{G}$. Formally, for any inputs and permutation of
cameras $\sigma$:

$\displaystyle\mathop{}\\!\mathrm{d}\mathbf{J},\mathop{}\\!\mathrm{d}\mathbf{C}=\mathcal{F}_{\theta}(\mathbf{J}^{(i)},\mathbf{C}^{(i)},\mathbf{G},\mathbf{K}^{(i)},\mathcal{L})$
(17)
$\displaystyle\mathop{}\\!\mathrm{d}\mathbf{J}^{\prime},\mathop{}\\!\mathrm{d}\mathbf{C}^{\prime}=\mathcal{F}_{\theta}(\mathbf{J}^{(i)},\mathbf{C}^{(i)}_{\sigma},\mathbf{G}_{\sigma},\mathbf{K}^{(i)}_{\sigma},\mathcal{L})$
(18)

we need to guarantee that
$\mathop{}\\!\mathrm{d}\mathbf{J}=\mathop{}\\!\mathrm{d}\mathbf{J}^{\prime}$
and
$\mathop{}\\!\mathrm{d}\mathbf{C}=\mathop{}\\!\mathrm{d}\mathbf{C}^{\prime}_{\sigma}$.
To archive this, we concatenate view-invariant inputs $\mathbf{J}^{(i)}$ and
$\mathcal{L}$ to each row of view-dependant inputs
$\mathbf{C}^{(i)},\mathbf{G},\mathbf{K}^{(i)}$, pass them though a
permutation-equivariant MLP [8, 23] with aggregation layers concatenating
first and second moments of feature vectors back to these feature vectors, and
apply mean aggregation and a non-permutation-equivariant MLP to get the final
pose update, as illustrated in Figure 6.

##### Limitations

We assume a weak camera model, making our method less accurate on captures
shot using wide-angle (short-focus) lenses. To achieve best performance, our
method requires accurate 2D keypoint ground truth for training, but we also
report performance without using GT keypoints during training (Table 2). We
implicitly assume that the subject is of comparable size (in pixels) across
all views, and expect re-weighting of different components of the reprojection
loss (16) might otherwise be necessary. While tracking and pose estimation can
be used for malicious purposes, such as mass surveillance, we believe that
societal benefits brought by such technological advances [39, 13, 10] outweigh
possible abuse.

####  3.3.1 Pose prior (i.e. “bone-length” experiment)

We illustrate the modularity of our solution by effortlessly injecting a
subject-specific bone-legth prior into our meta-optimizer. Given two joints
$\mathbf{j}_{n}$ and $\mathbf{j}_{m}$ connected in the human skeleton
$\mathcal{E}$ by an edge $e=(n,m)$, we define the bone length
$b_{e}(\mathbf{J})=\|\mathbf{j}_{n}-\mathbf{j}_{m}\|_{2}$. However, as our
bundle adjustment is performed up to scale we ought to define scale-invariant
bone lengths ${b}^{N}(\mathbf{J})=b(\mathbf{J})/\hat{\mu}(b(\mathbf{J}))$ by
expressing length of each bone relative to the average length of other bones
$\hat{\mu}(b)=(\sum_{e}b_{e})/|\mathcal{E}|$. If we assume that during
training and inference we observe noisy normalized bone-lenghs vectors
$\mathbf{B}=b^{N}(\mathbf{J})+\varepsilon$, where
$\varepsilon\sim\mathcal{N}(0,\sigma_{b}^{2}\mathbf{I})$. Then, the joint
probability (1) becomes:

$p(\mathbf{J},\mathbf{C},\mathbf{K},\mathbf{H},\mathbf{B})=p(\mathbf{B}|\mathbf{J})\,p(\mathbf{H}|\mathbf{K})\,p(\mathbf{K}|\mathbf{J},\mathbf{C})\,p(\mathbf{J})\,p(\mathbf{C})$

and our parametric likelihood (7) becomes:

$p(\mathbf{K}|\mathbf{H},\mathbf{B},\theta)\propto
p(\mathbf{K}|\mathbf{H},\theta)\,\cdot\,\mathcal{N}(b^{N}(\mathbf{J}_{\theta}(\mathbf{H},\mathbf{B}))|\mathbf{B},\sigma_{b}^{2}\mathbf{I})$

and its parameters $\theta$ can be estimated equivalently to (9) via maximum
over $p(\mathbf{K},\mathbf{H},\mathbf{B}|\,\theta)$ using observations from
$p(\mathbf{K},\mathbf{H},\mathbf{B})$ available during training, effectively
resulting in an additional loss term penalizing derivations of bone lengths of
predicted poses from provided bone lengths:

$\mathcal{L}_{b}(\theta)=\sum_{(i)}\left\|b^{N}(\mathbf{J}^{(i+1)})-\mathbf{B}\right\|^{2}_{2}.$
(19)

Figure 7:  Qualitative Results – The top row shows input frames we used for
pose estimation, overlayed with the GT pose (black). Two bottom rows show
predictions made by evaluated methods on H36M with four cameras (left) and
SkiPose with two cameras (right). We include predictions for Initialization
(Stage 1), MetaPose (Stage 1+2), MetaPose with an Iterative Refinement
(S1+IR), and AniPose initialized with GT. We also provide errors in the
format: PMPJPE/NMPJPE. A video demonstration of qualitative results across
both datasets can be found in the supplementary material or at this link:
https://bit.ly/cvpr22_6639.

##  4 Experiments

In this section, we specify datasets and metrics we used to validate the
performance of the proposed method and a set of baselines and ablation
experiments we conducted to evaluate the improvement in error provided by each
stage and each supervision signal.

##### Data

We evaluated our method on Human3.6M [17] dataset with four fixed cameras and
a more challenging SkiPose-PTZ [37] dataset with six moving pan-tilt-zoom
cameras. We used standard train-test evaluated protocol for H36M [19, 27] with
subjects 1, 5, 6, 7, and 8 used for training, and 9 and 11 used for testing.
We additionally pruned the H36M dataset by taking each 16-th frame from it,
resulting in 24443 train and 8516 test examples, each example containing
information from four cameras. We evaluated our method on the subset (1035
train / 230 test) of SkiPose [37] that was used in CanonPose [44] that
excludes 280 examples with visibility obstructed by snow. In each dataset, we
used the first 64 examples from the train split as a validation set.

##### Metrics

We report Procrustes aligned Mean Per Joint Position Error (PMPJPE) and
Normalized Mean Per Joint Position Error (NMPJPE) that measure the L2-error of
3D joint estimates after applying the optimal rigid alignment (including
scale) to the predicted 3D pose and the ground truth 3D pose (for NMPJPE), or
only optimal shift and scale (for PMPJPE). We also report the total amount of
time ($\Delta t$) it takes to perform 3D pose inference from multi-view RGB.

##### Baselines

On H36M we lower-bound the error with the state-of-the-art fully-supervised
baseline of Iskakov et al. [19] that uses ground truth camera parameters to
aggregate multi-view predictions during inference. We also compare the
performance of our method to methods that use multi-view 2D supervision during
training but only perform inference on a single view at a time: self-
supervised EpipolarPose (EP) [27] and CanonPose (CP) [44], as well as the
weakly supervised baselines of Iqbal et al. [18] and Rhodin et al. [37]. On
SkiPose we compared our model with the only two baselines available in the
literature: CanonPose [44] and Rhodin et al. [37]. We did not evaluate
EpipolarPose on SkiPose because it requires fixed cameras to perform the
initial self-supervised pseudo-labeling. We did not evaluate Iqbal et al. [18]
on SkiPose because no code has been released to date and authors did not
respond to a request to share code.

We also compared our method against the “classical” bundle adjustment
initialized with ground truth extrinsic camera parameters of all cameras, and
set fixed GT intrinsics, therefore putting it into unrealistically favorable
conditions. We used the well-tested implementation of bundle adjustment in
AniPose [25] that uses an adapted version of the 3D registration algorithm of
Zhou et al. [48]. This approach takes point estimates of keypoint locations as
an input (i.e. no uncertainty) and iteratively detects outliers and refines
camera parameters and joint 3D positions using the second-order Trust Region
Reflective algorithm [5, 3].

##### Architecture

For monocular 2D pose estimation, we used the stacked hourglass network [34]
pre-trained on COCO pose dataset [15]. For monocular 3D estimation in Stage 1,
we applied EpipolarPose [27] on Human3.6M and CanonPose [44] on SkiPosePTZ. We
note that differences in the joint labeling schemes used by these monocular 3D
methods and our evaluation set do not affect the quality of camera
initialization we acquire via rigid alignment, as long as monocular 3D
estimates for all views follow a consistent labeling scheme. Each neural
optimizer step is trained separately, and stop gradient is applied to all
inputs. We refer our readers to Section 8.3 in supplementary for a more
detailed description of all components we used to train our neural optimizer
and their reference performance.

Method | PMPJPE$\downarrow$ | NMPJPE$\downarrow$ | $\Delta t$[s]
---|---|---|---
4 | 2 | 4 | 2
Isakov et al. [19] | 20 | - | - | - | -
AniPose [25] w/ GT | 75 | 167 | 103 | 230 | 7.0
Rhodin et al. [37] | 65 | - | 80 | - | -
CanonPose [44] | 53 | - | 82 | - | -
EpipolarPose (EP) [27] | 71 | - | 78 | - | -
Iqbal et al. [18] | 55 | - | 66 | - | -
MetaPose (S1) | 74 | 87 | 83 | 95 | 0.2
MetaPose (S1+S2) | 32 | 44 | 49 | 55 | 0.3

Method | PMPJPE$\downarrow$ | NMPJPE$\downarrow$ | $\Delta t$[s]
---|---|---|---
6 | 2 | 6 | 2
AniPose [25] w/ GT | 50 | 62 | 221 | 273 | 7.0
Rhodin et al. [37] | - | - | 85 | - | -
CanonPose (CP) [44] | 90 | - | 128 | - | -
MetaPose (S1) | 81 | 86 | 140 | 144 | 0.3
MetaPose (S1+S2) | 42 | 50 | 53 | 59 | 0.4

Table 1:  Quantitative comparison to prior work – Performance of different
methods with four and two cameras on Human3.6M (top) and six and two cameras
SkiPose-PTZ (bottom), Procrustes and Normalized MPJPE in millimeters,
inference time in seconds. See Tables 4 and 6 in the supplementary material
for an extended comparison and breakdown of runtime performance.

##  5 Results – Table 1

Method | PMPJPE$\downarrow$ | NMPJPE$\downarrow$ | $\Delta t$[s]
---|---|---|---
4 | 2 | 4 | 2
MetaPose (S1+S2) | 32 | 44 | 49 | 55 | 0.3
MetaPose (S1+IR) | 43 | 53 | 66 | 75 | 2.0
MetaPose (S1+S2/SS) | 39 | 50 | 56 | 63 | 0.3
MetaPose (S1+S2) | 32 | 44 | 49 | 55 | 0.3
MetaPose (RND+S2) | 36 | 51 | 52 | 64 | 0.3
MetaPose (S1+IR) | 43 | 53 | 66 | 75 | 2.0
MetaPose (RND+IR) | 200 | 385 | 265 | 444 | 2.0
MetaPose (GT+IR) | 40 | 48 | 63 | 68 | 2.0
MetaPose (S1+S2) | 32 | 44 | 49 | 55 | 0.3
MetaPose (S1+S2/MLP) | 30 | 44 | 47 | 58 | 0.3
MetaPose (S1+S2) | 32 | 44 | 49 | 55 | 0.3
MetaPose (S1+S2/BL) | 30 | 37 | 50 | 54 | 0.3
Table 2: Ablations on H36M. Notation consistent with Table 1.

The proposed method (MetaPose S1+S2) outperforms the classical bundle-
adjustment baseline initialized with ground truth cameras (AniPose [25] w/ GT)
by +40mm on H36M with four cameras, and +8mm on SkiPose with six cameras. With
fewer cameras the performance gap increases further. MetaPose also outperforms
semi-, weakly-, and self-supervised baselines reported in prior work [18, 37,
44, 27] by more then 10mm. We would like to re-iterate core advantages of the
proposed method beyond its high performance, namely:
\scriptsize\raisebox{-0.7pt}{1}⃝ that Stage 1 primes the neural optimizer with
a good enough initialization that leads it to a good solution;
\scriptsize\raisebox{-0.7pt}{2}⃝ that our solution is modular enabling
swapping existing priming and pose estimation networks, as well as additional
losses, and re-training only the neural optimizer;
\scriptsize\raisebox{-0.7pt}{3}⃝ that our method achieves lower latency then
both classical and (GPU-accelerated) probabilistic bundle adjustment. We
expand upon these and other related finding in the next subsection.

###  5.1 Ablations

##### Iterative refiner

We measured the speed gain we get from using the neural optimizer $f_{\theta}$
by replacing Stage 2 with a test-time GPU-accelerated gradient descent (Adam
[26]) over the probabilistic bundle adjustment objective (6) with GMM-
parameterized heatmaps. Section 1 in Table 2 shows that the proposed method
(S1+S2) is up to seven times faster than the iterative refinement (S1+IR), and
is at least 10mm more accurate. We also measured the contribution of keypoint
supervision towards prediction accuracy of S2 compared to iterative
refinement. To do that, we trained Stage 2 to minimize the same GMM-
parameterized probabilistic bundle adjustment objective (6) instead of the re-
projection loss (16). The resulting self-supervised model (S1+S2/SS)
outperforms the iterative refinement, suggesting that the proposed
architecture regularizes the pose estimation problem. Note that our self-
supervised results also outperform prior work that uses weak- and self-
supervision [18, 27, 44].

##### Random initialization

We measured the effect of replacing single-view pose estimates
$\mathbf{q}_{c,j}$ used to initialize the pose and cameras in Stage 1 with
random Gaussian noise. Section 2 in Table 2 shows that while the neural
optimizer (RND+S2) is more resilient to poor initialization then the classical
one (RND+IR), a good initialization is necessary to achieve the state-of-art
performance (S1+S2). Moreover, marginally better results with GT
initialization (GT+IR) show that the proposed initialization already brings
the optimizer in the neighbourhood of the correct solution, and that further
improvement in the quality of the initial guess will not provide significant
gains in accuracy.

##### Non-equivariant network

We measured the effect of letting the model “memorize” the camera order by
replacing equivariant blocks with MLPs that receive multi-view information as
a single concatenated vector. The resulting model (S2/MLP) achieved marginally
better performance on H36M and marginally worse performance on SkiPose (Table
5), likely due to fixed cameras positions in H36M and moving cameras in
SkiPose.

##### Bone lengths

We trained a model with an additional bone length prior (Sec. 3.3.1) that
improved PMPJPE with two cameras by 7mm. The two camera setup is ill-
conditioned, hence can better exploit priors like bone lengths.

##### Inputs of neural optimizer

Unsurprisingly, among all inputs passed to the neural optimizer, heatmap
$\mathbf{H}$ contributed most to the final performance, but all components
were necessary to achieve the state-of-art performance; see Table 3 in
supplementary.

##### Further ablations (supplementary)

The teacher-student loss proposed by Ma et al. [31] to draw predicted
solutions into the basin of the right solution hurts the performance in all
experiments (Table 8), suggesting that Stage 1 already provides good-enough
initialization to start in the correct basin of the objective. We also ran the
iterative refiner from ground truth initialization with re-projection losses
with different camera models: results suggests that the weak camera model
contributed to 10-15mm of error on H36M and no error on SkiPose; see Table 10.
The performance of MetaPose on H36M starts to severely deteriorate at around
5% of the training data; see Table 11. Replacing GMM with a single Gaussian
decreased the performance only in two-camera H36M setup by 4mm, and did not
significantly influence the performance in other cases; see Table 13.

##  6 Conclusions

In this paper, we propose a new modular approach to 3D pose estimation that
requires only 2D supervision for training and significantly improves upon the
state-of-the-art by fusing per-view outputs of singe-view modules with a
simple view-equivariant neural network. Our modular approach not only enables
practitioners to analyze and improve the performance of each component in
isolation, and channel future improvements in respective sub-tasks into
improved 3D pose estimation “for free”, but also provides a common “bridge”
that enables easy inter-operation of different schools of thought in 3D pose
estimation – enriching both the “end-to-end neural world” with better model-
based priors and improved interpretability, and the “iterative refinement
world” with better-conditioned optimization problems, transfer-learning, and
faster inference times. We provide a detailed ablation study dissecting
different sources of the remaining error, suggesting that future progress in
this task might come from the adoption of a full camera model, further
improvements in 2D pose localization, better pose priors and incorporating
temporal signals from video data.

##  7 Acknowledgement

We would like to thank Bastian Wandt, Nori Kanazawa and Diego Ruspini for help
with CanonPose [44], stacked hourglass pose estimator, and interfacing with
AniPose, respectively.

## References

  * Agarwal et al. [2011] Sameer Agarwal, Yasutaka Furukawa, Noah Snavely, Ian Simon, Brian Curless, Steven M. Seitz, and Richard Szeliski.  Building rome in a day.  _Commun. ACM_ , 54(10):105–112, 2011.
  * Belagiannis et al. [2014] Vasileios Belagiannis, Sikandar Amin, Mykhaylo Andriluka, Bernt Schiele, Nassir Navab, and Slobodan Ilic.  3d pictorial structures for multiple human pose estimation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1669–1676, 2014.
  * Branch et al. [1999] Mary Branch, Thomas Coleman, and Yuying li.  A subspace, interior, and conjugate gradient method for large-scale bound-constrained minimization problems.  _SIAM Journal on Scientific Computing_ , 21, 12 1999.  doi: 10.1137/S1064827595289108.
  * Bridgeman et al. [2019] Lewis Bridgeman, Marco Volino, Jean-Yves Guillemaut, and Adrian Hilton.  Multi-person 3d pose estimation and tracking in sports.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops_ , pages 0–0, 2019.
  * Byrd et al. [1988] Richard H Byrd, Robert B Schnabel, and Gerald A Shultz.  Approximate solution of the trust region problem by minimization over two-dimensional subspaces.  _Mathematical programming_ , 40(1):247–263, 1988\.
  * Chen et al. [2019] Ching-Hang Chen, Ambrish Tyagi, Amit Agrawal, Dylan Drover, Stefan Stojanov, and James M Rehg.  Unsupervised 3d pose estimation with geometric self-supervision.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5714–5724, 2019.
  * Chen et al. [2020] Long Chen, Haizhou Ai, Rui Chen, Zijie Zhuang, and Shuang Liu.  Cross-view tracking for multi-human 3d pose estimation at over 100 fps.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * Deng et al. [2021] Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas Guibas.  Vector neurons: A general framework for so (3)-equivariant networks.  _arXiv preprint arXiv:2104.12229_ , 2021.
  * Drover et al. [2018] Dylan Drover, Ching-Hang Chen, Amit Agrawal, Ambrish Tyagi, and Cong Phuoc Huynh.  Can 3d pose be learned from 2d projections alone?  In _Proceedings of the European Conference on Computer Vision (ECCV) Workshops_ , pages 0–0, 2018.
  * Feng et al. [2021] Wenxin Feng, Jiangnan Zou, Andrew Kurauchi, Carlos H Morimoto, and Margrit Betke.  Hgaze typing: Head-gesture assisted gaze typing.  In _ACM Symposium on Eye Tracking Research and Applications_ , pages 1–11, 2021.
  * Frisch and Hanebeck [2021] Daniel Frisch and Uwe D. Hanebeck.  Gaussian mixture estimation from weighted samples, 2021.
  * Gleicher [1999] Michael Gleicher.  Animation from observation: Motion capture and motion editing.  _SIGGRAPH_ , 33(4):51–54, November 1999.
  * Gu et al. [2019] Yiwen Gu, Shreya Pandit, Elham Saraee, Timothy Nordahl, Terry Ellis, and Margrit Betke.  Home-based physical therapy with an interactive computer vision system.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_ , pages 0–0, 2019.
  * Guler and Kokkinos [2019] Riza Alp Guler and Iasonas Kokkinos.  Holopose: Holistic 3d human reconstruction in-the-wild.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * Güler et al. [2018] Rıza Alp Güler, Natalia Neverova, and Iasonas Kokkinos.  Densepose: Dense human pose estimation in the wild.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7297–7306, 2018.
  * He et al. [2020] Yihui He, Rui Yan, Katerina Fragkiadaki, and Shoou-I Yu.  Epipolar transformers.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7779–7788, 2020.
  * Ionescu et al. [2013] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.  Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments.  _IEEE transactions on pattern analysis and machine intelligence_ , 36(7):1325–1339, 2013.
  * Iqbal et al. [2020] Umar Iqbal, Pavlo Molchanov, and Jan Kautz.  Weakly-supervised 3d human pose learning via multi-view images in the wild.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5243–5252, 2020.
  * Iskakov et al. [2019] Karim Iskakov, Egor Burkov, Victor Lempitsky, and Yury Malkov.  Learnable triangulation of human pose.  In _International Conference on Computer Vision (ICCV)_ , 2019.
  * Jakab et al. [2020] Tomas Jakab, Ankush Gupta, Hakan Bilen, and Andrea Vedaldi.  Self-supervised learning of interpretable keypoints from unlabelled videos.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * Joo et al. [2019] Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timothy Godisart, Bart Nabbe, Iain Matthews, and et al.  Panoptic studio: A massively multiview system for social interaction capture.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 41(1):190–204, Jan 2019.
  * Joo et al. [2020] Hanbyul Joo, Natalia Neverova, and Andrea Vedaldi.  Exemplar fine-tuning for 3d human model fitting towards in-the-wild 3d human pose estimation, 2020.
  * Joseph-Rivlin et al. [2019] Mor Joseph-Rivlin, Alon Zvirin, and Ron Kimmel.  Momen (e) t: Flavor the moments in learning to classify shapes.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_ , pages 0–0, 2019.
  * Kanazawa et al. [2018] Angjoo Kanazawa, Michael J. Black, David W. Jacobs, and Jitendra Malik.  End-to-end recovery of human shape and pose.  In _Computer Vision and Pattern Regognition (CVPR)_ , 2018.
  * Karashchuk et al. [2020] Pierre Karashchuk, Katie L Rupp, Evyn S Dickinson, Elischa Sanders, Eiman Azim, Bingni W Brunton, and John C Tuthill.  Anipose: a toolkit for robust markerless 3d pose estimation.  _BioRxiv_ , 2020.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kocabas et al. [2019] Muhammed Kocabas, Salih Karagoz, and Emre Akbas.  Self-supervised learning of 3d human pose using multi-view geometry.  In _The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * Kocabas et al. [2020] Muhammed Kocabas, Nikos Athanasiou, and Michael J Black.  Vibe: Video inference for human body pose and shape estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5253–5263, 2020.
  * Kolotouros et al. [2019] Nikos Kolotouros, Georgios Pavlakos, Michael J Black, and Kostas Daniilidis.  Learning to reconstruct 3d human pose and shape via model-fitting in the loop.  In _ICCV_ , 2019.
  * Kundu et al. [2020] Jogendra Nath Kundu, Siddharth Seth, Varun Jampani, Mugalodi Rakesh, R. Venkatesh Babu, and Anirban Chakraborty.  Self-supervised 3d human pose estimation via part guided novel image synthesis.  _2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , Jun 2020.
  * Ma et al. [2020] Wei-Chiu Ma, Shenlong Wang, Jiayuan Gu, Sivabalan Manivasagam, Antonio Torralba, and Raquel Urtasun.  Deep feedback inverse problem solver.  In _ECCV_ , 2020.
  * Martinez et al. [2017] Julieta Martinez, Rayat Hossain, Javier Romero, and James J Little.  A simple yet effective baseline for 3d human pose estimation.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 2640–2649, 2017.
  * Mitra et al. [2020] Rahul Mitra, Nitesh B. Gundavarapu, Abhishek Sharma, and Arjun Jain.  Multiview-consistent semi-supervised learning for 3d human pose estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * Newell et al. [2016] Alejandro Newell, Kaiyu Yang, and Jia Deng.  Stacked hourglass networks for human pose estimation.  In _European conference on computer vision_ , pages 483–499. Springer, 2016.
  * Papandreou et al. [2018] George Papandreou, Tyler Zhu, Liang-Chieh Chen, Spyros Gidaris, Jonathan Tompson, and Kevin Murphy.  Personlab: Person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , pages 269–286, 2018.
  * Rhodin et al. [2018a] Helge Rhodin, Mathieu Salzmann, and Pascal Fua.  Unsupervised geometry-aware representation for 3d human pose estimation.  In _Proceedings of the European Conference on Computer Vision (ECCV)_ , September 2018a.
  * Rhodin et al. [2018b] Helge Rhodin, Jorg Sporri, Isinsu Katircioglu, Victor Constantin, Frederic Meyer, Erich Mueller, Mathieu Salzmann, and Pascal Fua.  Learning monocular 3d human pose estimation from multi-view images.  _Conference On Computer Vision And Pattern Recognition (CVPR)_ , 2018b.
  * Rosales et al. [2001] Romer Rosales, Matheen Siddiqui, Jonathan Alon, and Stan Sclaroff.  Estimating 3d body pose using uncalibrated cameras.  In _Proceedings of the 2001 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. CVPR 2001_ , volume 1, pages I–I. IEEE, 2001.
  * Saraee et al. [2019] Elham Saraee, Yiwen Gu, Shreya Pandit, San Tran, Eugenia Shandelman, Saurabh Singh, Timothy J Nordahl, Terry Ellis, and Margrit Betke.  Exercisecheck: Data analytics for a remote monitoring and evaluation platform for home-based physical therapy.  In _Proceedings of the 12th ACM International Conference on PErvasive Technologies Related to Assistive Environments_ , pages 110–118, 2019\.
  * Schönemann [1966] Peter H Schönemann.  A generalized solution of the orthogonal procrustes problem.  _Psychometrika_ , 31(1):1–10, 1966.
  * Takahashi et al. [2018] Kosuke Takahashi, Dan Mikami, Mariko Isogawa, and Hideaki Kimata.  Human pose as calibration pattern; 3d human pose estimation with multiple unsynchronized and uncalibrated cameras.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops_ , pages 1775–1782, 2018.
  * Tu et al. [2020] Hanyue Tu, Chunyu Wang, and Wenjun Zeng.  Voxelpose: Towards multi-camera 3d human pose estimation in wild environment, 2020.
  * Wandt and Rosenhahn [2019] Bastian Wandt and Bodo Rosenhahn.  Repnet: Weakly supervised training of an adversarial reprojection network for 3d human pose estimation.  In _Computer Vision and Pattern Recognition (CVPR)_ , June 2019.
  * Wandt et al. [2021] Bastian Wandt, Marco Rudolph, Petrissa Zell, Helge Rhodin, and Bodo Rosenhahn.  CanonPose: Self-supervised monocular 3D human pose estimation in the wild.  In _Computer Vision and Pattern Recognition (CVPR)_ , June 2021.
  * Xie et al. [2020] Rongchang Xie, Chunyu Wang, and Yizhou Wang.  Metafuse: A pre-trained fusion model for human pose estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , June 2020.
  * Zanfir et al. [2020] Andrei Zanfir, Eduard Gabriel Bazavan, Hongyi Xu, William T. Freeman, Rahul Sukthankar, and Cristian Sminchisescu.  Weakly supervised 3d human pose and shape reconstruction with normalizing flows.  In _Computer Vision – ECCV 2020_ , pages 465–481, 2020.
  * Zanfir et al. [2021] Andrei Zanfir, Eduard Gabriel Bazavan, Mihai Zanfir, William T Freeman, Rahul Sukthankar, and Cristian Sminchisescu.  Neural descent for visual 3d human pose and shape.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14484–14493, 2021.
  * Zhou et al. [2016] Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun.  Fast global registration.  In _European Conference on Computer Vision_ , pages 766–782. Springer, 2016.
  * Zhou et al. [2017] Xingyi Zhou, Qixing Huang, Xiao Sun, Xiangyang Xue, and Yichen Wei.  Towards 3d human pose estimation in the wild: A weakly-supervised approach.  _2017 IEEE International Conference on Computer Vision (ICCV)_ , Oct 2017.
  * Zhou et al. [2019] Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li.  On the continuity of rotation representations in neural networks.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5745–5753, 2019.

$\mathbf{J},\mathbf{C}$ | ✓ | ✓ | ✓ | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✗
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
$\mathbf{H}$ | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✓ | ✓ | ✗ | ✗ | ✓ | ✗ | ✗ | ✓ | ✗
$\mathbf{K}$ | ✓ | ✓ | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗
$\mathcal{L}$ | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ | ✗ | ✗ | ✗ | ✗ | ✗ | ✓
4 | 32 | 33 | 32 | 45 | 32 | 48 | 33 | 32 | 45 | 45 | 32 | 46 | 45 | 34 | nan
3 | 36 | 35 | 36 | 49 | 37 | 50 | 38 | 39 | 49 | 50 | 37 | 50 | 49 | 44 | nan
2 | 44 | 44 | 46 | 53 | 43 | 59 | 40 | 56 | 53 | 58 | 45 | 55 | 51 | 45 | nan
Table 3:  Ablation – PMPJPE$\downarrow$ of our method on Human3.6M with different number of cameras with different inputs passed to the neural optimizer. Heatmap $\mathbf{H}$ contributes most to the final performance, but all inputs are necessary to achieve the state-of-art performance.  | PoseNet | GMM | S1 | Solver | Total | Error [mm]
---|---|---|---|---|---|---
AniPose | 0.03 $\cdot$ 4 | - | - | 7 | 7.1 | 75
MetaPose (S1) | 0.03 $\cdot$ 4 | - | 0.01 $\cdot$ 4 | - | 0.15 | 74
MetaPose (S1+S2) | 0.03 $\cdot$ 4 | 0.01 $\cdot$ 4 | 0.01 $\cdot$ 4 | 0.006 | 0.2 | 40
MetaPose (S1+IR) | 0.03 $\cdot$ 4 | 0.01 $\cdot$ 4 | 0.01 $\cdot$ 4 | 1.5 | 1.7 | 43
AniPose | 0.5 $\cdot$ 4 | - | - | 10 | 12 | 75
MetaPose (S1) | 0.5 $\cdot$ 4 | - | 0.20 $\cdot$ 4 | - | 2.8 | 74
MetaPose (S1+S2) | 0.5 $\cdot$ 4 | 0.25 $\cdot$ 4 | 0.20 $\cdot$ 4 | 0.01 | 4 | 40
MetaPose (S1+IR) | 0.5 $\cdot$ 4 | 0.25 $\cdot$ 4 | 0.20 $\cdot$ 4 | 3.5 | 7.5 | 43
Table 4: Latency breakdown in seconds for estimating the full 3D pose on H36M with four cameras on a GPU (V100, top) and a CPU (bottom) across four components: per-view 2D heatmap estimation (PoseNet), heatmap GMM fitting, per-view monocular 3D and initialization, multi-view bundle adjustment (neural network forward pass in case of MetaPose S1+S2, Adam [26] in case of S1+IR, and a 2nd-order CPU-only TRR [5, 3] solver in case of AniPose). MetaPose (S1+S2) achieves lowest error with an at least six times (on GPU; two times on CPU) faster inference as the iterative refiner.  (a) H36M |
---|---
Method | 4 | 3 | 2
Metapose (S1+S2) | 32 | 36 | 44
Metapose (S1+S2/MLP) | 30 | 35 | 41
(b) SkiPose |
Method | 6 | 4 | 2
Metapose (S1+S2) | 42 | 45 | 50
Metapose (S1+S2/MLP) | 46 | 44 | 54
Table 5: Equivariant (S1+S2) and non-equivariant (S1+S2/MLP) performance
networks have comparable performance across different numbers of cameras on
H36M (top) and SkiPose (bottom).

(a) H36M
---
Method and supervision type | PMPJPE$\downarrow$ | NMPJPE$\downarrow$ | $\Delta t$[s]
4 | 2 | 4 | 2
Isakov et al. [19] | 3D | 20 | - | - | - | -
AniPose [25] w/ GT | S | 75 | 167 | 103 | 230 | 7.1
Rhodin et al. [37] | 2/3D | 65 | - | 80 | - | -
CanonPose [44] | S | 53 | - | 82 | - | -
EpipolarPose (EP) [27] | S | 71 | - | 78 | - | -
Iqbal et al. [18] | 2D | 55 | - | 66 | - | -
MetaPose (S1) | S | 74 | 87 | 83 | 95 | 0.2
MetaPose (S1+S2) | 2D | 32 | 44 | 49 | 55 | 0.2
MetaPose (S1+IR) | S | 43 | 66 | 53 | 75 | 1.7
MetaPose (S1+S2/SS) | S | 39 | 50 | 56 | 63 | 0.2

(b) SkiPose
---
Method and supervision type | PMPJPE$\downarrow$ | NMPJPE$\downarrow$ | $\Delta t$[s]
6 | 2 | 6 | 2
AniPose [25] w/ GT | S | 50 | 62 | 221 | 273 | 7.1
Rhodin et al. [37] | 2/3D | - | - | 85 | - | -
CanonPose (CP) [44] | S | 90 | - | 128 | - | -
MetaPose (S1) | S | 81 | 86 | 140 | 144 | 0.3
MetaPose (S1+S2) | 2D | 42 | 50 | 53 | 59 | 0.4
MetaPose (S1+IR) | S | 30 | 77 | 54 | 94 | 2.5
MetaPose (S1+S2/SS) | S | 42 | 95 | 59 | 102 | 0.4

Table 6:  MetaPose outperforms SotA on H36M (top) and SkiPose (bottom) – Same
notation as in Table 1. Also includes the self-supervised (S2/SS) and
iterative solver (SS/IR) flavours of MetaPose. Supervision signal used during
training: 2D - ground truth 2D keypoints, 3D - ground truth 3D poses, S -
self-supervision (i.e. using a pose estimation network pre-trained on a
different dataset), 2/3D - 2D keypoint data with 3D poses on few subjects.
(a) H36M
---
Method | 4 | 3 | 2
Metapose S1+S2 | 32 | 36 | 44
Metapose S1+S2 \+ bone | 31 | 34 | 37
Metapose S1+IR | 43 | 52 | 53
Metapose S1+IR \+ bone | 38 | 44 | 47
Metapose S1+S2/SS | 39 | 47 | 50
Metapose S1+S2/SS \+ bone | 38 | 45 | 50
(b) SkiPose
Method | 6 | 4 | 2
Metapose S1+S2 | 41 | 43 | 47
Metapose S1+S2 \+ bone | 45 | 46 | 49
Metapose S1+IR | 30 | 32 | 77
Metapose S1+IR \+ bone | 26 | 28 | 46
Metapose S1+S2/SS | 41 | 46 | 95
Metapose S1+S2/SS \+ bone | 44 | 45 | 53

Table 7: Personalized bone lengths prior helps in all cases for H36M (top),
especially in the few-camera setup; and in the majority of cases on SkiPose
(bottom).  (a) H36M
---
Method | 4 | 3 | 2
MetaPose S1+S2 | 32 | 36 | 44
MetaPose S1+S2/TS | 38 | 45 | 45
(b) SkiPose
Method | 6 | 4 | 2
MetaPose S1+S2 | 42 | 45 | 50
MetaPose S1+S2/TS | 42 | 43 | 72

Table 8: Teacher-student loss analogous to the one proposed by Ma et al. [31]
to bring the neural optimizer into the basin of the correct solution either
hurts or does significantly affect the performance in all cases.

Metric | Train | Validation | Test
---|---|---|---
GT log-prob. | -5.17 | -5.58 | -5.06
Stage: | S1 | S1+IR | S1+S2 | S1 | S1+IR | S1+S2 | S1 | S1+IR | S1+S2
Pred log-prob. | -4.22 | -5.86 | -5.00 | -4.6 | -6.07 | -5.41 | -3.92 | -5.77 | -4.95
PMPJPE [mm] | 69 | 38 | 15 | 65 | 34 | 17 | 74 | 43 | 32
NMPJPE [mm] | 78 | 58 | 36 | 69 | 56 | 46 | 88 | 66 | 49
MSE 2D ($10^{-4}$) | 15 | 5 | 0.6 | 6 | 2 | 0.6 | 20 | 7 | 5

(a) H36M

Metric | Train | Validation | Test
---|---|---|---
GT log-prob. | -5.49 | -5.49 | -4.81
Stage: | S1 | S1+IR | S1+S2 | S1 | S1+IR | S1+S2 | S1 | S1+IR | S1+S2
Pred log-prob. | -2.83 | -5.79 | -5.49 | -2.90 | -5.75 | -4.51 | -3.04 | -5.59 | -5.33
PMPJPE [mm] | 71 | 17 | 1 | 72 | 17 | 10 | 80 | 30 | 42
NMPJPE [mm] | 139 | 35 | 1 | 143 | 38 | 15 | 140 | 54 | 53
MSE 2D ($10^{-4}$) | 34 | 6 | 0.01 | 37 | 6 | 1 | 30 | 7 | 7

(b) SkiPose

Table 9: Details about predictions across different stages: initialization
using monocular 3d (S1), iterative refinement (S1+IR), and neural refinement
(S1+S2) on H36M with four cams (top) and SkiPose with six cams (bottom). 2D
error is scaled so that the entire pose lies in $[0,1]^{2}$. The GT log
probability is the log probability of ground truth points given predicted
heatmaps and measures how well heatmaps generated by our 2D prediction network
match the ground truth. Significantly larger discrepancy between GT log
probabilities on train and test on SkiPose shows that 2D pose network overfits
much more on SkiPose than on H36M due to its limited size. (a) H36M
---
Method | 2D | S1 | 4 | 3 | 2
S1+IR | HT | EP | 43 | 52 | 53
S1+IR | HT | GT | 40 | 49 | 48
S1+IR | full-GT | EP | 17 | 20 | 24
S1+IR | full-GT | GT | 14 | 16 | 20
S1+IR | weak-GT | EP | 4 | 6 | 18
S1+IR | weak-GT | GT | 1.4 | 1.7 | 2
(b) SkiPose
Method | 2D | S1 | 6 | 4 | 2
S1+IR | HT | CP | 30 | 33 | 77
S1+IR | HT | GT | 28 | 30 | 41
S1+IR | full-GT | CP | 8 | 8 | 29
S1+IR | weak-GT | CP | 8 | 7 | 28

Table 10: MetaPose S1+IR trained with either ground truth pseudo-heatmaps centered around full and weak-projected 3D joints and with different S1 initialization (either predicted via EpipolarPose or “perfect”). This experiment shows that imperfect heatmaps contribute to at least 20mm of error in both cases, weak camera model contribute to 10mm of error on H36M and no error on SkiPose, and imperfect initialization contributes to at most 3mm of error.  # cam % | 100 | 89 | 84 | 79 | 73 | 68 | 63 | 58 | 52 | 47 | 42 | 37 | 31 | 29 | 26 | 24 | 21 | 18 | 16 | 13 | 10 | 8 | 5 | 3
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
4 | 32 | 32 | 32 | 32 | 33 | 36 | 33 | 35 | 33 | 35 | 34 | 42 | 37 | 39 | 36 | 38 | 36 | 41 | 48 | 41 | 41 | 44 | 48 | 70
3 | 36 | 35 | 36 | 37 | 37 | 36 | 37 | 39 | 37 | 37 | 38 | 39 | 40 | 39 | 41 | 45 | 42 | 43 | 44 | 45 | 46 | 51 | 53 | 70
2 | 44 | 48 | 48 | 47 | 46 | 48 | 48 | 40 | 54 | 60 | 43 | 43 | 51 | 57 | 53 | 58 | 54 | 50 | 52 | 48 | 48 | 51 | 68 | 87

Table 11: Test PMPJE of MetaPose on H36M as a function of the fraction of
training examples with 2D ground truth used (i.e. first X%). Reminder: we
never use any ground truth 3D annotations for either cameras or poses, these
are percentages of 2D labels used for training. We can see that MetaPose
produces high-accuracy predictions (within 10mm of the original performance)
with up to 1/5-th ($\approx$18%) of the H36M training 2D pose annotations
($\approx$5k training examples each containing multiple cameras). The few-
camera setup exhibits more variations in test error due to random network
initialization. (a) H36M
---
Method | GMM | 4 | 3 | 2
MetaPose S1+IR | 4 | 43 | 52 | 53
MetaPose S1+IR | 3 | 42 | 51 | 52
MetaPose S1+IR | 2 | 42 | 51 | 52
MetaPose S1+IR | 1 | 42 | 52 | 53
MetaPose S1+S2 | 4 | 32 | 39 | 44
MetaPose S1+S2 | 3 | 31 | 36 | 47
MetaPose S1+S2 | 2 | 32 | 36 | 50
MetaPose S1+S2 | 1 | 32 | 36 | 48
(b) SkiPose
Method | GMM | 6 | 4 | 2
MetaPose S1+IR | 4 | 30 | 33 | 77
MetaPose S1+IR | 3 | 30 | 32 | 77
MetaPose S1+IR | 2 | 31 | 34 | 75
MetaPose S1+IR | 1 | 43 | 43 | 58
MetaPose S1+S2 | 4 | 42 | 45 | 50
MetaPose S1+S2 | 3 | 44 | 41 | 50
MetaPose S1+S2 | 2 | 42 | 49 | 51
MetaPose S1+S2 | 1 | 41 | 43 | 47

Table 13: The number of Gaussian Mixture components does not significantly
affect the performance of the network in all cases on both SkiPose (top) and
H36M (bottom), except for MetaPose S1+IR on SkiPose with a single Gaussian.

##  8 Supplementary

###  8.1 Ablation Tables

  * •

Table 3 shows the performance of the neural optimizer trained with different
subsets of inputs;

  * •

Table 4 shows the latency breakdown across model components and models;

  * •

Table 5 shows that the performance of equivariant (S1+S2) and non-equivariant
(S1+S2/MLP) models differs by at most 4mm on both datasets;

  * •

Table 6 shows that MetaPose outperforms prior work with corresponding
supervision signals;

  * •

Table 7 shows that the personalized bone length prior improves the performance
of both the iterative and neural refiners in the majority of cases;

  * •

Table 8 shows that the student-teacher loss inspired by Ma et al. [31] to draw
the predicted solution into the correct basin of the loss hurts the
performance in all cases;

  * •

Table 9 summarizes reference performance of monocular pose estimation
components across different splits of data (train, val, test) for
reproducibility, and shows strong overfitting on SkiPose;

  * •

Table 10 shows that at least 20mm of error is due to imperfect heatmaps, up to
10mm is due to the weak camera model, and only up to 3mm is due to imperfect
init;

  * •

Table 11 shows that on H36M with just 1/5th of the entire training dataset
(i.e. 5k labeled training samples, each sample containing several cameras) we
can get a model that achieves PMPJPE within 5-10mm of the performance we
achieve on full data.

  * •

Table 13 shows the effect of varying the number of Gaussian mixture components
on the performance of different methods.

###  8.2 Weighted EM-algorithm for spherical GMM

We used grid points $x_{i}$ weighted by corresponding probabilities $p_{i}$ to
fit a GMM to a 2D probability heatmap. Following Frisch and Hanebeck [11] on
each step $t=0\dots T$ of the EM algorithm we performed usual (non-weighted)
E-step to compute the new assignment matrix $\eta_{i,m}^{(t+1)}$ between
points $x_{i}$ and spherical clusters $m=0\dots M$ with means $\mu_{m}^{(t)}$,
and standard variations $\sigma_{m}^{(t)}$, and weights $w_{m}^{(t)}$,
followed by a weighted M-step:

$\displaystyle
w^{(t+1)}_{m}=\frac{\sum_{i}\eta^{(t+1)}_{i,m}p_{i}}{\sum_{m^{\prime}}\sum_{i}\eta^{(t+1)}_{i,{m^{\prime}}}p_{i}}$
$\displaystyle\mu^{(t+1)}_{m}=\frac{\sum_{i}\eta^{(t+1)}_{i,m}p_{i}x_{i}}{\sum_{i}\eta^{(t+1)}_{i,m}p_{i}}$
$\displaystyle\sigma^{(t+1)}_{m}=\sqrt{\frac{\sum_{i}\eta^{(t+1)}_{i,m}p_{i}||x_{i}-\mu^{(t+1)}_{m}||_{2}^{2}}{\sum_{i}\eta^{(t+1)}_{i,m}p_{i}}}$

###  8.3 Implementation Details

##### Architecture

For monocular 2D pose estimation we used the stacked hourglass network [34]
pre-trained on COCO pose dataset [15]. We additionally trained a linear
regression adapter to convert between COCO and H36M label formats (see
supplementary Figure 8 for labeling format comparison). The resulting
procedure yields good generalization on H36M, as shown in supplementary Table
9). The COCO-pretrained network generalized very poorly to SkiPosePTZ dataset
because of the visual domain shift, so we fine-tuned the stacked hourglass
network using ground truth 2D labels. For monocular 3D estimates used in Stage
1, we applied EpipolarPose [27] on Human3.6M and CanonPose [44] on SkiPosePTZ.
We would like to note that, despite the significant shift in the labeling
format between predictions of these monocular 3D methods and the format used
in datasets we used for evaluation, this does not affect the quality of camera
initialization we acquired via rigid alignment. Similar to prior work [31],
each “neural optimizer step” is trained separately, and the fresh new neural
net is used at each stage, and stop gradient is applied to all inputs. For MLP
architecture, we used $L$ fully-connected 512-dimensional layers followed by a
fully-connected 128-dimensional, all with selu with L=4 for H36M and L=2 for
SkiPose. For equivalent network, the optimal network for H36M had following
layers: [512, 512, CC, 512, 512, CC, 512] and for SkiPose had following
layers: [512, 512, CC, 512, 512, CC, 512, 512, CC, 512, 512, CC, 512] - where
CC corresponds to concatenation of first two moments and numbers correspond to
dense layers with selu. We re-trained each stage multiple times until the
validation PMPJPE improved or the total number of “stage training attempts”
exceeded 100.

##### Hyperparameters

We used Adam [26] optimizer with learning rate 1e-2 for 100 steps for exact
refinement, and 1e-4 for the neural optimizer.

##### Reference 2D performance

Tables 9 shows performance of 2D pose prediction networks and the resulting
MetaPose network on different splits of different datasets. It shows that both
the 2D network and MetaPose to certain degree overfit to SkiPose because of
its smaller size.

Figure 8: Both H36M ground truth poses, COCO dataset (used to train the
hourglass network), and EpipolarPose predictions (used to generate the 3D
initialization) have different label formats from H36M. We trained a small
“adapter” to convert COCO-to-H36M, and used EpipolarPose predictions as-is.

Videos with test predictions can be found in the attached video file and
following this link: https://bit.ly/cvpr22_6639.

###  8.4 Closed Form Expressions for Stage 1

Below we describe the solution to the rigid alignment problem (12) for
monocular 3D pose estimates $\mathbf{q}_{c}$ and inferred weak camera
parameters from them. Assume that we have monocular 3D predictions
$\mathbf{q}_{c}$ in frame of the camera $c$. The parameters of the first
camera are assumed to be known and fixed

$R^{(0)}_{\text{init}}{=}I,\bm{t}^{(0)}{=}\bar{0},s^{(0)}{=}1$

whereas the rotation of other cameras are inferred using optimal rigid
alingment $R^{(c)}_{\text{init}}=(U^{(c)})^{T}V^{(c)}$ where

$U^{(c)},\Lambda,V^{(c)}=\operatorname{SVD}(\operatorname{centered}(\mathbf{q}_{c})\cdot\operatorname{centered}(\mathbf{q}_{0}))^{T})$

The scale $s$ and shift $\bm{t}$ can be acquired by comparing the original
monocular $\mathbf{q}_{c,[:,0:2]}$ in pixels to
$[R^{(c)}_{\text{init}}\operatorname{centered}(\mathbf{q}_{0})]_{[:,0:2]}$
rotated back into each camera frame, for example:

$\displaystyle
s^{(c)}_{\text{init}}=\frac{||[(R^{(c)}_{\text{init}})^{T}\operatorname{centered}(\mathbf{q}_{0})]_{[:,0:2]}||}{||\operatorname{centered}(\mathbf{q}_{c})_{[:,0:2]}||}$
(20) $\displaystyle\bm{t}^{(c)}_{\text{init}}=$
$\displaystyle\hat{\mu}([(R^{(c)}_{\text{init}})^{T}\operatorname{centered}(\mathbf{q}_{0})]_{[:,0:2]})-\hat{\mu}([\mathbf{q}_{c}]_{[:,0:2]}))$
(21)

where $\hat{\mu}(\bm{a})=(\sum_{k}^{K}\bm{a}_{k})/K$ is the center of the 3D
pose and $\operatorname{centered}(\bm{a})_{k}=(\bm{a}_{k}-\hat{\mu}(\bm{a}))$
and the initial pose estimate is the average of aligned, rotated and
predictions from other cameras. The initial guess for the pose is the average
of all monocular poses rotated into the first camera frame:

$\bm{J}_{\text{init}}=\frac{1}{C}\sum_{c=0}^{C}(s^{(c)}_{\text{init}}\cdot
R^{(c)}_{\text{init}}\operatorname{centered}(\mathbf{q}_{c}))+\hat{\mu}(\mathbf{q}_{0})$
(22)

###  8.5 6D rotation re-parameterization

We used for following parameterization:
$R(x,y)=\operatorname{stack}[n(x),n(x\times y),n(x\times(x\times y))]$ where
$n(x)$ is a normalization operation, and $a\times b$ is a vector product. This
is essentially Gram-Schmidt orthogonalization. Rows of the resulting matrix is
guaranteed to form an orthonormal basis. This rotation representation was
shown to be better suited for optimization [50].

###  8.6 Stable Gaussian Mixture Likelihood

We used the following numerically stable spherical GMM log-likelihood to
compute (6):

$\displaystyle\log\left[\sum_{r}w_{r}\cdot\frac{\exp(-\frac{1}{2}(\bm{x}-\bm{\mu}_{r})^{T}\cdot(\sigma_{r}^{2}\cdot
I)^{-1}(\bm{x}-\bm{\mu}_{r})}{\sqrt{(2\pi)^{2}\sigma_{r}^{4}}}\right]$
$\displaystyle=\log\sum_{r}\exp\left[\log\left(\frac{w_{r}}{2\pi\sigma_{r}^{2}}\right)-\frac{||\bm{x}-\bm{\mu}_{r}||^{2}}{2\sigma_{r}^{2}}\right]$
$\displaystyle=\operatorname{LSE}_{r}\left[\log\left(\frac{w_{r}}{2\pi\sigma_{r}^{2}}+\varepsilon\right)-\frac{||\bm{x}-\bm{\mu}_{r}||^{2}}{2\sigma_{r}^{2}}\right]$

where $(\mu,\sigma^{2},w)$ are mean, variance and weight of the corresponding
mixture component, $\operatorname{LSE}(l_{0},\dots,l_{r})$ is a numerically
stable “log-sum-exp” often implemented as
$\operatorname{LSE}(l_{0},\dots,l_{r})=l^{*}+\log(\sum_{k}\exp(l_{k}-l^{*}))$,
where $l^{*}=\max(l_{0},\dots,l_{r})$, and $\varepsilon$ is a small number.

###  8.7 Teacher loss

In addition to the reprojcetion loss, the student-teacher ablation (S2/TS)
used the following additional loss inspired by Ma et al. [31] to draw the
predicted solution $\bm{J}_{\text{neur}}$ into the basin of the correct
solution by penalizing its deviation from the solution $\bm{J}_{\text{ref}}$
produced by the iterative refiner (IR).

$\displaystyle\mathcal{L}_{y}\big{(}\bm{J}_{\text{neur}},$ $\displaystyle
R^{(c)}_{\text{neur}},\bm{t}^{(c)}_{\text{neur}},s^{(c)}_{\text{neur}};\bm{J}_{\text{ref}},R^{(c)}_{\text{ref}},\bm{t}^{(c)}_{\text{ref}},s^{(c)}_{\text{ref}}\big{)}$
$\displaystyle=$
$\displaystyle\lambda_{p}\cdot||\bm{J}_{\text{neur}}-\bm{J}_{\text{ref}}||_{2}^{2}+\lambda_{t}\cdot\sum||\bm{t}_{\text{neur}}^{(c)}-\bm{t}_{\text{ref}}^{(c)}||_{2}^{2}$
$\displaystyle+$
$\displaystyle\lambda_{R}\cdot\sum||(R_{\text{neur}}^{(c)})^{T}R_{\text{ref}}^{(c)}-I||_{2}^{2}$
$\displaystyle+$
$\displaystyle\lambda_{s}\cdot\sum||\log(s_{\text{neur}}^{(c)})-\log(s_{\text{ref}}^{(c)})||_{2}^{2}.$
(23)

Table 8 shows that it hurts the performance of the model.

###  8.8 Qualitative Results

We provide qualitative examples (failure cases, success cases) on the test set
of H36M and SkiPose in Figures 9-19. Videos with more test prediction
visualizations are available at: https://bit.ly/cvpr22_6639. Circles around
joints on 2D views represent the absolute reprojection error for that joint
for that view. Our qualitative findings:

  1. 1.

MetaPose considerably improves over the initial guess when a lot of self-
occlusion is present

  2. 2.

MetaPose fails on extreme poses for which monocular estimation fails (e.g.
somersaults)

  3. 3.

In two-camera SkiPose setup, AniPose often yields smaller reprojection error
while producing very bad 3D pose results

Figure 9: Full MetaPose (S1+S2) outperforms initialization (S1), Iterative
Solver (S1+IR), and AniPose w/ GT camera init. Figure 10: MetaPose improves
over the initial guess under high self-occlusion. Figure 11: MetaPose improves
over the initial guess under high self-occlusion. Figure 12: AniPose w/ GT
camera initialization can yields low re-projection error but high 3D
estimation error. Figure 13: AniPose with GT init fails due to poor choice of
2D predictions to ignore during refinement. Figure 14: MetaPose fails on few
extreme poses that have much poorer than average initialization quality.
Figure 15: MetaPose improves over the initial guess under high self-occlusion.
Figure 16: MetaPose fails on poses that have much poorer (than average)
initialization quality. Figure 17: AniPose with GT init fails due to poor
choice of 2D predictions to ignore during refinement.

Figure 18: With two cameras AniPose with GT camera init often yields low
reprojection error but bad 3D estimation error

Figure 19: With two cameras AniPose with GT camera init often yields low
reprojection error but bad 3D estimation error