[table]capposition=top

11institutetext: Computer Vision Lab, EPFL, Switzerland
11email<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
# On Triangulation as a Form of Self-Supervision for 3D Human Pose Estimation

Soumava Kumar Roy equal contribution Leonardo Citraro* Sina Honari  Pascal Fua

###### Abstract

Supervised approaches to 3D pose estimation from single images are remarkably
effective when labeled data is abundant. Therefore, much of the recent
attention has shifted towards semi and (or) weakly supervised learning.
Generating an effective form of supervision with little annotations still
poses major challenges in crowded scenes. However, since it is easy to observe
a scene from multiple cameras, we propose to impose multi-view geometrical
constraints by means of a differentiable triangulation and to use it as form
of self-supervision during training when no labels are available. We therefore
train a 2D pose estimator in such a way that its predictions correspond to the
re-projection of the triangulated 3D one and train an auxiliary network on
them to produce the final 3D poses. We complement the triangulation with a
weighting mechanism that nullify the impact of noisy predictions caused by
self-occlusion or occlusion from other subjects. Our experimental results on
Human3.6M and MPI-INF-3DHP substantiate the significance of our weighting
strategy where we obtain state-of-the-art results in the semi and weakly
supervised learning setup. We also contribute a new multi-player sports
dataset that features occlusion, and show the effectiveness of our algorithm
over baseline triangulation methods.

##  1 Introduction

Supervised approaches in capturing human 3D pose are now remarkably effective,
provided that enough annotated training data is available [23, 24, 34, 37, 44,
45, 52, 53] . However, there are many scenarios that involve unusual
activities for which not enough annotated data can be obtained. Semi-
supervised or unsupervised methods are then required [25, 36, 5, 47, 16, 4]. A
promising subset of those rely on constraints imposed by multi-view geometry:
If a scene is observed by several cameras, there is a single a priori unknown
3D pose whose projection is correct in all views [40, 18, 30]. This is used to
provide a supervisory signal with limited need for manual annotations. There
are many venues, such as a sports arena, that are outfitted with multiple
cameras, which can easily be used for this purpose.

These semi-supervised approaches, however, are sensitive to occlusions or
noisy predictions, which are prevalent in crowded scenes. In this paper, we
propose an approach that can handle them effectively. To this end, we start
from the observation that there are now many off-the-shelf pre-trained 2D pose
estimation models that do not necessarily perform well in new environments but
can be fine-tuned to do so. Our approach therefore starts with one of these
that we run on all images taken at the same time and whose results can be
triangulated. The model is then trained so that the prediction in all views
correspond to the re-projection of the pseudo 3D pose target. In particular,
we give the 2D prediction of each view a weight, that is derived from the
level of agreement of that view with respect to all other views. In doing so,
we apply a principled triangulation weighting mechanism to obtain pseudo 3D
targets that can discard erroneous views, hence helping the model bootstrap in
semi and weakly supervised setups. Our approach can be used in any multi-view
context, without restriction on camera placement. At inference time, our
retrained network can be used on single-view images and have their output
lifted to 3D by an auxiliary network.

We demonstrate the effectiveness of our approach on the traditional Human3.6M
[12] and MPI-INF-3DHP [28] datasets. To highlight its ability to handle more
crowded scenes, we acquired a multi-view dataset featuring an amateur
basketball game and use to test the robustness of our approach under more
challenging conditions. We will make this dataset publicly available along
with our code.

##  2 Related Work

With the advent of deep learning, direct regression of 2D and 3D poses from
images has become the dominant approach for human pose estimation [23, 31, 29,
41, 42]. Recent variations on this theme use volumetric representations [15],
lift 2D poses to 3D [27, 14], use graph convolutional network (GCN) [6, 2, 50,
26, 54] or generate and refine pseudo labels from video sequences [25, 36].
Given enough annotated training data, these methods are now robust and
accurate. Thus, in recent years, the attention has shifted from learning 3D
human poses in fully supervised setups to semi- or weakly-supervised ones [35,
16, 49, 32, 48, 3], with a view to drastically reduce the required amount of
annotated images.

In many venues such as sport arenas, there are multiple cameras designed to
provide different viewpoints, often for broadcasting purposes. Hence, a valid
approach is to exploit the constraints that multi-view geometry provides for
training purposes, while at inference time, the trained network operates on
single views. For example, in [18], a single-view 3D pose estimator is trained
using pseudo labels generated by triangulating the predictions of a pre-
trained 2D pose estimator in multiple images. However, as the 2D pose
estimator is fixed, pseudo label errors caused by noisy 2D detections are
never corrected.

In [40, 14], a regressor is trained to produce consistent 3D poses from
single-views, while guaranteeing consistency of the predictions across
multiple views. A problem with these approaches is that predictions can be
consistent but wrong, which makes it necessary to use some annotated 3D or 2D
data. The method of [39] relies on a similar setup but reduces the required
amount of annotated data by using multi-view unsupervised pre-training to
learn a low-dimensional latent representation from which a simple regressor
could infer 3D pose. However, due to the low-dimensionality of the learned
latent space, the structural information from the image is lost, which makes
it less accurate in practice. In [20], an encoder-decoder based self-
supervised training algorithm is used to disentangle the appearance and the
pose features of the same subject extracted from two different images. It also
uses a part-based 2D puppet model as an additional form of prior knowledge
regarding the 2D human skeleton structure and appearance to guide the learning
process. This results in a very complex pipe-line that requires a great-deal
of additional data for training purposes without delivering superior
performance as we demonstrate in the result section.

Triangulation is at the heart of our approach and is also used extensively in
the fully-supervised method of [15], which uses 3D labels to refine the
triangulation process, and of [38] where 2D poses are fused in a latent space.
While [38] uses standard triangulation that does not weigh different views,
[15] learns a weight for each view, which requires a much stronger supervision
than ours to learn an effective weighting scheme. In contrast, ours is derived
from the agreement of 2D views and can better handle semi- and weakly-
supervised setups because the weights are computed from multi-view consistency
rather than required to be learned. In the result section, we show that the
triangulation of [15] applied to our semi-supervised framework results in less
accurate poses when the amount of labeled data is limited.

Figure 1: Network Architecture. It comprises a detection network that outputs
2D joint locations in individual views and a lifting network that predicts
root relative distances to the camera which are then turned to 3D poses using
the 2D ones and the intrinsic camera parameters. They are trained jointly
using a small number of images with corresponding 3D annotations and a much
larger set of images acquired using multiple cameras. During training, the
prediction networks feed their 2D detections into a differentiable
triangulation layer that estimates 3D joint locations. These are re-projected
into the images to minimize the discrepancy between the projections and the
original detections, along with a supervised loss that operates on the frames
for which annotations are available. The estimated 3D joint locations are also
used to train the lifting network.

##  3 Method

Our goal is to train a deep neural network to estimate 3D human poses from
single images using as little labeled data as possible. To this end we acquire
multiple views of our subjects and train a 2D pose estimator to detect joint
locations consistently across views. This detection network is complemented by
a lifting network that infers 3D poses from 2D ones. During training, we check
for consistency across views of unlabeled images by triangulating the 2D
detections and re-projecting the results into the images and use the weighted
triangulated 3D points to train the lifting network.

Both networks are trained in a semi-supervised way on our target dataset, that
is, we assume that the ground-truth 2D and 3D poses are known only for a small
subset of the entire dataset. To train the detection network, in addition to a
small set of 2D labels, we use a robust differentiable triangulation layer
that takes as input the predicted 2D poses for unlabeled images in individual
views and produces the 3D estimates. They are used as pseudo labels to check
for consistency of the 2D pose estimates and to train both the detection and
the lifting networks. The complete pipeline is depicted by Fig. 1 and is end-
to-end trainable. Making the triangulation process robust to mis-detections in
some of the views is a key component of our approach because some joints are
not visible in some views, especially in crowded scenes.

###  3.1 Notations

Let
$\mathcal{L}=\\{[\boldsymbol{I}^{l,c}]_{c=1}^{N_{C}},\boldsymbol{p}^{l}_{3D}\\}_{l=1}^{N_{L}}$
be a sequence of labeled RGB images
${\boldsymbol{I}}^{l,c}\in\mathbb{R}^{h\times w\times 3}$ taken using $N_{C}$
different cameras, and $\boldsymbol{p}^{l}_{3D}$ be the ground-truth 3D pose,
where $c$ denotes the number of cameras and $l$ is the index of the labeled
images. Similarly, let
$\mathcal{U}=\\{[\boldsymbol{I}^{u,c}]_{v=1}^{N_{C}}\\}_{u=1}^{N_{U}}$, where
$u$ denotes the unlabeled, be a larger sequence of multi-view images without
associated 3D pose labels. We take $\boldsymbol{p}^{l}_{3D}$ to be the entire
set of body joints $\\{\boldsymbol{X}^{l}_{j}\\}_{j=1}^{N_{J}}$, where
$\boldsymbol{X}^{l}_{j}\in\mathbb{R}^{3}$ denotes the 3D coordinates of joint
$j$ for the $l$ sample. Similarly, let $\boldsymbol{p}^{l}_{2D}$ be
$\\{\boldsymbol{x}^{l,c}_{j}\\}_{j=1}^{N_{J}}$, where
$\boldsymbol{x}^{l,c}_{j}\in\mathbb{R}^{2}$ is the projection of
$\boldsymbol{X}^{l}_{j}$ in view $c$. In the remainder of this section, we
drop the $l$, $u$, and $c$ notations when there is no ambiguity.

Let $f_{{\boldsymbol{\theta}}}$ and $g_{{\boldsymbol{\phi}}}$ denote the
detection and lifting network with weights ${\boldsymbol{\theta}}$ and
${\boldsymbol{\phi}}$ respectively. The detection network
$f_{{\boldsymbol{\theta}}}$ takes $\boldsymbol{I}^{c}$ as input and returns a
2D pose estimate in that view
${\hat{\boldsymbol{p}}_{2D}^{c}}=\\{\hat{\boldsymbol{x}}^{c}_{j}\\}_{j=1}^{N_{J}}$.
The lifting network $g_{{\boldsymbol{\phi}}}$ takes as input such a 2D pose
predictions and returns root relative distances to the camera which are then
turned to a 3D pose estimate ${\hat{\boldsymbol{p}}}_{3D}$ using the known
intrinsic camera parameters K.

Finally, let $\Pi$ be a function that takes as input the known camera
parameters and a set of image points, one per view, and triangulates to a
location in the world
${\hat{\boldsymbol{X}}}_{j}=\Pi(\\{\hat{\boldsymbol{x}}^{c}_{j}\\}_{c=1}^{N_{C}})$.

###  3.2 Training the Detection and Lifting Networks

We train the detection network $f_{{\boldsymbol{\theta}}}$ and the lifting
network $g_{{\boldsymbol{\phi}}}$ by minimizing two loss functions that we
define below, which we describe in more details in the following subsections.

####  3.2.1 Detection Loss $L^{f}(\theta)$.

We take it to be the sum of a supervised term computed on $\mathcal{L}$ and an
unsupervised one computed on $\mathcal{U}$. We write it as

$\displaystyle L^{f}(\theta)$
$\displaystyle=L_{sup}(\theta;\mathcal{L})+L_{tri}(\theta;\mathcal{U})\;,$
$\displaystyle L_{sup}(\theta;\mathcal{L})$
$\displaystyle=\sum_{l=1}^{N_{L}}\sum_{j=1}^{N_{J}}\sum_{c=1}^{N_{C}}\left\|{\boldsymbol{\hat{x}}}^{l,c}_{j}-\boldsymbol{x}^{l,c}_{j}\right\|^{2}_{2}\;,$
(1) $\displaystyle L_{tri}(\theta,\mathcal{U})$
$\displaystyle=\sum_{u=1}^{N_{U}}\sum_{c=1}^{N_{C}}\sum_{j=1}^{N_{J}}\left\|\hat{\boldsymbol{x}}^{u,c}_{j}-{{\bar{\boldsymbol{x}}}^{u,c}_{j}}\right\|^{2}_{2}\;,$

where ${\bar{\boldsymbol{x}}}^{u,c}_{j}$ denotes the projection of the
$\hat{\boldsymbol{X}}^{u}_{j}$ obtained by triangulating the
${\hat{\boldsymbol{x}}}^{u,c}_{j}$ predicted by the network. $L_{sup}$ is the
supervised MSE loss between the predicted 2D poses and the ground-truth ones.
$L_{tri}$ is the unsupervised loss that is at its minimum when the detected 2D
poses are consistent with the projections of the triangulated poses. $L_{tri}$
pools information from the multiple views to provide a supervisory signal to
our model without using any labels. All the image location in the losses are
normalized in the range $[-1,1]$ with respect to their subject’s bounding box.

As shown in Fig. 1, $L_{tri}$ provides two sets of gradients during back-
propagation. One from the re-projected triangulated pose directly to the
detection model and one that flows into the triangulation. Blocking either one
of these results in a different behavior that makes this self-supervised loss
less stable in practice. We provide more details and experiments Section § 4.4
to highlight this.

####  3.2.2 Lifting Loss $L^{g}(\phi;f_{\theta})$.

Once $f_{{\boldsymbol{\theta}}}$ is trained, we use its predictions on
$\mathcal{U}$ and the triangulation operation $\Pi$ to create a new set with
pseudo 3D poses
$\mathcal{\hat{U}}=\\{[{\boldsymbol{I}}^{u,c}]_{c=1}^{N_{C}},{\boldsymbol{\hat{p}}}^{u}_{3D}\\}_{u=1}^{N_{U}}$.
We consider the lifting loss to be

$L^{g}(\phi;f_{\theta})=L_{semi-
sup}^{3D}(\phi;f_{\theta},\mathcal{L}^{\frown}\mathcal{\hat{U}})$ (2)

with respect to $\phi$. $L_{semi-sup}^{3D}$ is the MSE loss between the
predicted 3D poses and ground-truth or triangulated ones. The parameters of
the 2D pose estimator $f_{\theta}$ are fixed and provide the 2D poses as input
to $g_{\phi}$. ⌢ denotes the concatenation operation on $\mathcal{L}$ and
$\mathcal{\hat{U}}$.

###  3.3 Self-Supervision by Triangulation

Computing $L_{tri}(\theta,\mathcal{U})$ of Eq. 1 requires triangulating each
joint $j$ in the set of 2D locations
$\\{\hat{\boldsymbol{x}}^{c}_{j}\\}_{c=1}^{N_{C}}$. In theory, the most
reliable way to triangulate is to perform non-linear minimization [10].
However, because this computation is part of our deep learning, it must be
differentiable, which is not easy to achieve for this kind of minimization. We
therefore rely on the Direct Linear Transform (DLT) [1], which involves
transforming the projection equations to a system of linear ones whose
solution can be found using Singular Value Decomposition [9]. This is less
accurate but easily differentiable.

In practice, occlusions and mis-detections pose a much more significant
challenge than any loss of accuracy due to the use of DLT. It is inevitable
that some joints are hidden in some views and their estimated 2D locations
$\hat{\boldsymbol{x}}^{u,c}_{j}$ are erroneous, which results in inaccurate
triangulations. To prevent this, we introduce a weighing strategy that reduces
the influence of mis-detections in such problematic views. We apply these
weights in DLT and re-write the unsupervised loss of Eq. 1 as

$\displaystyle L_{tri}(\theta,\mathcal{U})$
$\displaystyle=\sum_{u=1}^{N_{U}}\sum_{c=1}^{N_{C}}\sum_{j=1}^{N_{J}}w_{j}^{u,c}||\hat{\boldsymbol{x}}^{u,c}_{j}-{{\bar{\boldsymbol{x}}}^{u,c}_{j}}||^{2}_{2}\;,$
(3)

where $w_{j}^{u,c}$ denotes the reliability of the estimate of the location of
joint $j$ in view $c$. To estimate these weights, we use the purely
geometrical approach based on robust statistics described below.

####  3.3.1 Robust Localization.

To robustly estimate the 3D location of a joint $j$, we first compute
candidate 3D locations using all pairs of 2D detections
$(\hat{\boldsymbol{x}}^{c}_{j},\hat{\boldsymbol{x}}^{c^{\prime}}_{j})$ in
pairs of views. Let

$\displaystyle T_{j}$
$\displaystyle=\\{\hat{\boldsymbol{Y}}_{j}^{c,c^{\prime}}=\Pi(\boldsymbol{x}^{c}_{j},\boldsymbol{x}^{c^{\prime}}_{j})|(c,c^{\prime})\in\binom{N_{C}}{2}\\}\;,$
(4) $\displaystyle\tilde{\boldsymbol{X}}_{j}$
$\displaystyle=\text{GeoMed}(T_{j})$

where $T_{j}$ is a detection cluster and $\tilde{\boldsymbol{X}}_{j}$ is the
center of the cluster, as shown in Fig. 2. $\Pi$ denotes the triangulation,
$\binom{N_{C}}{2}$ denotes the set of all pairs of views, and GeoMed refers to
calculating the geometric median of $T_{j}$. When most of the predicted image
locations are correct, $\tilde{\boldsymbol{X}}_{j}$ closely represent the true
location of the jth joint. Instead, when more than a half of the image
locations are erroneous, the geometrical median cannot recover the joint
location reliably. We explain how to overcome this difficulty using a measure
of homogeneity defined within the cluster $T_{j}$ in the subsequent
paragraphs.

Given $T_{j}$ and $\tilde{\boldsymbol{X}}_{j}$, we take the weights to be

$\displaystyle w^{c}_{j}$ $\displaystyle=\textrm{Median}(W^{c}_{j})$ (5)
$\displaystyle W^{c}_{j}$
$\displaystyle=\\{w_{j}^{c,c^{\prime}}=\exp(-\frac{\|\tilde{\boldsymbol{X}}_{j}-\hat{\boldsymbol{Y}}^{c,c^{\prime}}_{j}\|^{2}_{2}}{\sigma^{2}})|c^{\prime}\in
C_{j}\backslash\\{c\\}\\}$

where $W^{c}_{j}$ is a set of weights for joint jth from view $c$ to all other
views. These encode the distance of each candidate location in $T_{j}$ to its
center and are bounded in the range $[0,1]$ using a Gaussian function.
$\sigma$ is a parameter of the model that is set according to the amount of
noise present in the candidate joints. Intuitively, when a joint estimate in
$T_{j}$ is an outlier, its weights should be closer to zero, otherwise it
should be close to one. In all our experiments we set it to $10$ millimeters.

Figure 2: Visual representation of cluster $T_{j}$.

####  3.3.2 Joint Selection.

The above formulation relies on $\tilde{\boldsymbol{X}}_{j}$ obtained in Eq. 4
being reliable, which is usually true when more than $50\%$ of the 2D
detections are correct. However, this is not always the case. To handle this,
at each training iteration, we assign a score to each $T_{j}$ and momentarily
discards the joints whose score is below a certain threshold. To compute this
score, we use the normalized Within-Cluster-Sum of Squared Errors (WSS) as
follows:

$WSS_{j}=\frac{1}{|T_{j}|}\sum_{\hat{\boldsymbol{Y}}_{j}^{c,c^{\prime}}\in
T_{j}}\|\hat{\boldsymbol{Y}}_{j}^{c,c^{\prime}}-\tilde{\boldsymbol{X}}_{j}\|^{2}_{2}\;.$
(6)

WSS is a measure of homogeneity defined within an agglomeration often used in
clustering methods. Intuitively, when the image points are erroneous, the
joint estimates composing the cluster $T_{j}$ are distant from one another
resulting in a high $WSS_{j}$. Therefore, if $WSS_{j}$ is greater than a
manually defined value during training, we momentarily discard the joint. As
the training progresses, the model improves iteratively, allowing for the
possibility of previously discarded joints to contribute towards the training
of the model. In all our experiments we set this threshold to $20$ millimeters
that we found empirically.

In summary, the weights computed using our purely geometric approach of Eq. 5
allow the triangulation to output a more robust estimate of the true location
of the joint. That is the center of the cluster $\tilde{\boldsymbol{X}}_{j}$.
Instead, Eq. 6 allows to discard the noisy 2D detections, thus reducing their
negative impact on the model.

Figure 4, of the experiment section, shows a comparison between ours (with
weights) and the standard triangulation method (without weights). We show that
our method produces more accurate results than the others when less than
$50\%$ of the views are affected by the noise, while obtaining similar
performance otherwise.

###  3.4 Implementation Details

#### Lifting 2D Poses to 3D.

The second stage of our pipeline “lifts” the predictions of our finetuned 2D
pose estimator to 3D poses in the camera coordinate system. We adopt a
commonly used pose representation [44, 33, 13, 14] namely 2.5D
${\boldsymbol{p}}^{2.5D}=\\{(u_{j},v_{j},d^{root}+d^{rel}_{j})\\}_{j=1}^{N_{J}}$
where $u_{j}$ and $v_{j}$ are the 2D location of joint jth in the undistorted
image space, $d^{root}$ is a scalar representing the depth of the root joint
with respect to the camera and $d^{rel}_{j}$ is the relative depth of each
joint to the root. The advantage of using this representation lies in the fact
that the 2D pose $\\{(u_{j},v_{j})\\}_{j=1}^{N_{J}}$ is spatially coincident
with the image, which allows to fully exploit the characteristics of
convolutional neural networks. In addition, the 2D pose estimator can be
further improved using additional in-the-wild 2D pose datasets.

To obtain 3D poses we first train a multi-layer neural network as in [27] and
output root relative depths
$d^{rel}_{j}=g_{\phi}(f_{\theta}({\boldsymbol{I}}))$. Then, we recover the 3D
pose in the camera or in the world coordinate system using the inverse of the
projection equations:

$(d^{root}+d^{rel}_{j})\begin{pmatrix}u_{j}\\\ v_{j}\\\
1\end{pmatrix}={\boldsymbol{K}}\begin{pmatrix}X^{C}_{j}\\\ Y^{C}_{j}\\\
(d^{root}+d^{rel}_{j})\end{pmatrix}={\boldsymbol{K}}\left[{\boldsymbol{R}}\begin{pmatrix}X_{j}\\\
Y_{j}\\\ Z_{j}\end{pmatrix}+{\boldsymbol{t}}\right]$ (7)

where $X^{C}$ and $Y^{C}$ are the first two components of a joint in camera
space, $(X_{j},Y_{j},Z_{j})$ is the joint in the world coordinate and
${\boldsymbol{K}}\in\mathbb{R}^{3\times 3}$,
${\boldsymbol{R}}\in\mathbb{R}^{3\times 3}$, and
${\boldsymbol{t}}\in\mathbb{R}^{3\times 1}$ are the intrinsic matrix, rotation
matrix, and translation vector of the camera respectively. The above
formulation assumes that the depth of the root joint $d^{root}$ is known. In
previous works [40, 27, 44], the position of the root is taken to be the
ground-truth one or not used at all simply because the dataset itself is
provided in camera coordinate. We follow the same protocol and use the ground-
truth depth of the root, however, an approximation can be obtain analytically
as shown in [13].

A simple multi-layer neural network can effectively learn the distribution of
relative depths from the joint coordinate of 2D poses, however, the image can
still provide useful information to reduce ambiguities. For this reason, we
use an additional convolutional neural network $h_{\gamma}$, parameterized by
$\gamma$, that takes an image $\boldsymbol{I}$ as input and output a small
feature vector. We then concatenate this feature vector with the joint
coordinate of a 2D pose and feed it into the multi-layer neural network to
output the relative depth
$d^{rel}_{j}=g_{\phi}(f_{\theta}({\boldsymbol{I}})^{\frown}h_{\gamma}({\boldsymbol{I}}))$.
This additional network contribute to distinguish classes of poses such as
when subjects are standing, sitting and lying that can be more easily captured
from the images themselves. In our experiments we consider $h_{\gamma}$ to be
a ResNet50 [11] pretrained on ImageNet [43] and replace its last linear layer
with an MLP that shrinks the size of the output feature from $2048$ to $16$.

#### Detection and Lifting models.

We use Alphapose [8] trained on Crowdpose dataset [22] as our 2D pose
estimator $f_{{\boldsymbol{\theta}}}$ where we replace the original non-
differentiable Non-Maximum Suppression (NMS) by a two-dimensional Soft-argmax
as in [44]. For the lifting network $g_{{\boldsymbol{\phi}}}$ we used a multi-
layer perceptron network with 2 hidden layers with hidden size of $2048$. The
first two layers are composed of a linear layer followed by ReLU activation
and $10\%$ dropout while the last layer is a linear layer. The lifting network
takes as input the feature vector obtained by concatenating the joint
coordinates of a 2D pose and the $16$ dimensional feature obtained from
$h_{\gamma}$. $h_{\gamma}$ takes the cropped image resized to $256\times 256$
as the input. The 2D pose is defined in the undistorted image space and is
normalized in the range $[-1,1]$ with respect to the bounding box.

#### Training Procedure.

We resize the input crops to $320\times 256$ pixels by keeping the original
aspect ratio and apply random color jittering for regularization. In all our
experiments we pretrained the 2D detection network first on the small set of
labeled set and then we finetuned it on both labeled and unlabeled sets with
our losses. During training, we used mini-batches of size $24$ samples where
$8$ samples come from the labeled set and the other from the unlabeled one. We
use Adam optimizer [17] with constant learning rate of $1e-4$ for the pre-
training and $1e-5$ for the fine-tuning. We do not use weight decay nor
learning rate scheduling. All the 3D losses were computed on poses expressed
in meters and the image coordinates of the 2D losses are all normalized in the
range $[-1,1]$ with respect to the bounding boxes.

##  4 Experiments

We primarily report our results in the semi-supervised learning setup, where
we have 3D annotations only for a subset of the training images. We use the
projection of these 3D annotations to train $f_{{\boldsymbol{\theta}}}$ and
they are directly used to supervise the training of $g_{{\boldsymbol{\phi}}}$.
For the unannotated samples, we use the 3D poses triangulated from 2D
estimations and their projections to train both $g_{{\boldsymbol{\phi}}}$ and
$f_{{\boldsymbol{\theta}}}$.

###  4.1 Datasets and Metrics

We validate our proposed framework on two large 3D pose estimation datasets
and a new multi-view dataset featuring an amateur basketball match to study
the influence of occlusions in a crowded scene. We will release it publicly.

#### Human3.6M [12].

It is the most widely used indoor dataset for single and multi-view 3D pose
estimation. It consists of $3.6$ million images captured form $4$ calibrated
cameras. As in most published papers, we use subjects S$1$, S$5$, S$6$, S$7$,
S$8$ for training and S$9$, S$11$ for testing. In the semi-supervised setup we
follow two separate protocols:

  1. 1.

Subject S$1$ is the only source of supervision, while other training subjects
are treated as unlabeled, as done in [40, 32, 21].

  2. 2.

Only $10\%$ of uniformly sampled training samples are considered annotated and
the rest as unlabeled [20].

The temporal sampling rate of the unlabeled videos is always set to $5$.

#### MPI-INF-3DHP [28].

This dataset contains both constrained indoor and some complex outdoor images
for single person 3D pose estimation. It features $8$ subjects performing $8$
different actions, which are recorded using $14$ different cameras, thereby
covering a wide range of diverse 3D poses. We follow the standard protocol and
use the the 5 chest-height cameras only. In the semi-supervised setup, as for
the Human3.6M dataset, we exploit annotations for subject S$1$ of the training
set.

#### SportCenter.

We filmed an amateur basketball match using $8$ fixed and calibrated cameras.
$2$ of the $8$ cameras are mounted on the roof of the court, which makes them
useful for location tracking but less useful for pose estimation. The images
feature a variable number of subjects ranging from $10$ to $13$. They are
either running, walking, or standing still. The players are often occluded
either by others or by various objects, such as the metal frames of the nets.
There are also substantial light variations that makes it even more
challenging. We computed the players’ trajectories for the whole sequence and
manually annotated a subset with 2D poses. Thereafter, we obtained the 3D
poses by triangulating the manually annotated 2D detections. The dataset
comprises $315,000$ images out of which $560$ are provided with $3,740$ 2D
poses and $700$ 3D poses. We use two subjects for testing and the remaining
subjects are used for training. In total, we have $140$ annotated 3D poses for
the test phase, while the remaining $560$ annotated 3D poses are used for
training in the semi-supervised setup.

#### Metrics.

We report the Mean Per-Joint Position Error (MPJPE), the normalized NMPJPE,
and the procrustes aligned PMPJPE in millimeters. On the MPI-INF-3DHP dataset
we also report the Percentage of Correct Keypoints in 3D (3DPCK) with a
threshold of 150mm. The best score is always shown in bold.

(a)

(b)

Figure 3: Comparison with Pavllo  _et al_. [36], Kundu  _et al_. [20] and our
semi-supervised variant with the triangulation of Iskakov [15] on (left)
NMPJPE and (right) PMPJPE metrics on Human3.6m with different amount of
supervision.

### Quantitative Evaluation

We report our results for the semi-supervised learning setup in Tables 4, 4
and 3 for Human3.6M [12], MPI-INF-3DHP [28] and SportCenter datasets
respectively.

Considering only $10\%$ of data as labeled for Human3.6M dataset in Tables 4,
we note that our approach is able to outperform Kundu  _et al_. [20] ($46.1$mm
vs. $50.8$ mm) in terms of PMPJPE by a significant margin. Similarly, we
perform better then them when considering all samples of S$1$ as labeled (Only
S1) ($48.3$mm vs. $51$ mm PMPJPE). Note that Kundu  _et al_. [20] make use of
additional datasets such as in-the-wild YouTube videos and the MADS [51]
dataset, in addition to a part-based puppet to instill prior human skeletal
knowledge in their overall learning framework. However, our experimental
results demonstrate that we circumvent the need of such supplementary
components by designing an effective geometrical reasoning based on a
weighting strategy that nullifies the impact of noisy detections as the
training progresses. Furthermore, we also obtained better results than Iqbal
_et al_. [14] ($60.8$mm vs $62.8$mm in terms of MPJPE) in semi-supervised by
at least $2$ mm in both MPJPE and PMPJPE.

Table 4 shows the comparison on the semi-supervised setups on MPI-INF-3DHP
dataset. Here the approach of Iqbal [14] was able to produce slightly better
results then us with $113.8$ millimeters compared to $119.1$ of ours. However,
we note that [14] uses ground-truth 2D poses for the unlabeled set as source
of supervision.

Table 1: Quantitative results Human3.6M (Semi-supervised)

Singleview - semi-supervised
---
10% of All Data
Method |  MPJPE $\downarrow$ |  NMPJPE $\downarrow$ |  PMPJPE $\downarrow$
Kundu et.al. [20] | - | - | 50.8
Ours | 56.9 | 56.6 | 45.4
Only S1
Rhodin _et al_. [39] | 131.7 | 122.6 | 98.2
Pavlako _et al_. [32] | 110.7 | 97.6 | 74.5
Li _et al_. [25] | 88.8 | 80.1 | 66.5
Rhodin _et al_. [40] | - | 80.1 | 65.1
Kocabas _et al_. [18] | - | 67.0 | 60.2
Pavllo _et al_. [36] | 64.7 | 61.8 | -
Iqbal _et al_. [14] | 62.8 | 59.6 | 51.4
Kundu _et al_. [20] | - | - | 52
Ours | 60.8 | 60.4 | 48.4

Table 2: Quantitative results MPI-INF-3DHP (Semi-supervised)

Method |  MPJPE $\downarrow$ |  NMPJPE $\downarrow$ |  PMPJPE $\downarrow$ |  3DPCK $\uparrow$
---|---|---|---|---
Only S1
Rhodin _et al_. [40] | - | 121.8 | - | -
Kocabas _et al_. [18] | - | 119.9 | - | -
Iqbal _et al_. [14] | 113.8 | 102.2 | - | 79.1
Ours | 119.1 | 108.5 | 97.4 | 74.3

A comparative study between ours, Kundu [20], Pavllo [36] and Iskakov  _et
al_. [15] is also shown in Fig. 3 as we progressively increase the amount of
labeled 3D poses in the semi-supervised learning setup. It is observed that we
outperform both Kundu [20] and Pavllo [36] by a significant margin when
working with limited amount of labeled images (_e.g._ 5/10 $\%$S$1$). This
clearly shows that our model is able to take advantage of unlabeled multi-view
images to provide supervisiory signal to our model.

We evaluate the effectiveness of our triangulation method on the SportsCenter
dataset in the semi-supervised setup. We compare our proposed triangulation
approach against i) Standard DLT, which is the standard triangulation, and ii)
Ours+Iskakov using the learned triangulation approach of [15], added to our
backbone. We use the same architecture for all triangulation approaches to
better observe the impact of triangulation. We follow the setup of [15] for
learning the weights using a neural network. Our approach instead is purely
geometrical, which is an advantage when limited amount of labeled data is
available. We report the multi-view (MV) and the single-view (SV) results in
Table 3. We can note that our triangulation method is superior than Standard
DLT because of its ability to produce reliable 3D poses from multiple 2D views
even in the presence of occlusion, which is frequent in the SportCenter
dataset. This is observed in the multiview case and as a consequence also in
the single view results. Moreover, our approach also obtains better results
than Ours+Iskakov on this task. We believe Ours+Iskakov suffers from the lack
of adequate amounts of labeled data, which is needed to train precise weights
in this approach, and also from the noise in the 2D detections caused by the
occlusion.

Table 3: Quantitative results on SportCenter (semi-supervised) using different
triangulation approaches.

| Multiview | Single View
---|---|---
Method | MPJPE$\downarrow$ | NMPJPE$\downarrow$ | PMPJPE$\downarrow$ | MPJPE$\downarrow$ | NMPJPE$\downarrow$ | PMPJPE$\downarrow$
Standard DLT | 80.5 | 78.4 | 66.7 | 118.5 | 116.5 | 95.4
Ours+Iskakov [15] | 88.3 | 83.6 | 70.9 | 121.1 | 119.5 | 99
Ours | 72.0 | 71.0 | 60.7 | 112.1 | 110.1 | 88.8

Table 4: Quantitative results on the Second Spectrum dataset (semi-supervised) using different triangulation approaches. | Multiview | Single View
---|---|---
Methods | MPJPE$\downarrow$ | NMPJPE$\downarrow$ | PMPJPE$\downarrow$ | MPJPE$\downarrow$ | NMPJPE$\downarrow$ | PMPJPE$\downarrow$
Standard DLT | 61.5 | 60.3 | 52.3 | 95.7 | 93.9 | 75.8
Ours | 54.3 | 52.9 | 46.2 | 94.7 | 92.8 | 74.5

###  4.2 Second Spectrum Dataset

Here we evaluate and compare our proposed method against the _Standard DLT_ on
a basketball dataset provided by Second Spectrum. The dataset consists of $15$
players and $3$ referees playing a game of basketball at the National
Basketball Association (NBA). The videos are captured using $8$ calibrated
cameras, with $4$ placed in each half of the court. We have considered $2$
(out of $15$) players and $1$ referee (out of $3$) to be the test set, while
the remaining $13$ players and $2$ referees are considered as the train set.
Each subject has its annotated 2D and 3D ground truth poses. In the semi-
supervised learning setup, we have considered $2,160$ annotated images as the
labeled set $\mathcal{L}$ and the unlabeled set $\mathcal{U}$ consists of
$19,865$ images. The results for multiview and single view experiments are
shown in Table 4. As seen, we consistently outperform the _Standard DLT_ [1]
across all the three evaluation metrics for both the multiview and the single
view experiments; thereby clearly demonstrating the importance and
effectiveness of our proposed weighting strategy in handling noisy 2D
detections and producing reliable triangulated 3D poses.

###  4.3 Mobile Camera

In this section, we evaluate and compare our proposed methods against (a)
_Standard DLT_ [1] and (b) Ours+Iskakov [15] on a video sequence of
SportCenter dataset captured on a single mobile camera. The images are
captured using a calibrated Iphone6. Like before, we evaluate the learnt
models of Table 3 on two test subjects, thereby resulting in a test set of
$38$ images. The results using the predictions of the learnt 2D pose estimator
model $f_{\theta}$ (_i.e._ ${\boldsymbol{\hat{x}}}$) and the 2D ground truth
annotations (_i.e._ $\boldsymbol{x}$) are shown in Table 5. As expected, there
exists a significant performance gap($\sim$ 60 mm) between using
${\boldsymbol{\hat{x}}}$ and $\boldsymbol{x}$ as the input to the lifting
network $g_{{\boldsymbol{\phi}}}$. We believe this gap in performance can be
bridged provided sufficient amount of data obtained using a mobile camera is
available to improve the performance of $f_{\theta}$. Having said that, our
proposed method still remains the best performing method against the other two
baseline methods which clearly shows the effectiveness of our weighting
strategy to generate suitable triangulated 3D poses.

Table 5: Quantitative reuslts on SportCenter dataset captured on a mobile camera (Single View). | Using ${\boldsymbol{\hat{x}}}$ | Using $\boldsymbol{x}$
---|---|---
Method | MPJPE$\downarrow$ | NMPJPE$\downarrow$ | PMPJPE$\downarrow$ | MPJPE$\downarrow$ | NMPJPE$\downarrow$ | PMPJPE$\downarrow$
Standard DLT | 199 | 197.3 | 140 | 138.3 | 138.4 | 88.1
Ours + Iskakov [15] | 197.7 | 193.7 | 130.1 | 131.8 | 132.5 | 82.8
Ours | 188.9 | 186.2 | 130.5 | 129.1 | 126.8 | 85.6

###  4.4 Ablation Study

Table 6: Comparative study on the impact of unlabeled data for different levels of supervision $\mathcal{L}$. We report the MPJPE results in millimeters. $\mathcal{L}$ | 1% S1 | 5% S1 | 10% S1 | 50% S1 | S1 | S1, S5 | S1, S5, S6
---|---|---|---|---|---|---|---
Baseline | 135.3 | 110.5 | 94.0 | 83.4 | 83.9 | 69.2 | 65.1
Ours | 105.7 | 79.1 | 67.9 | 62.3 | 60.8 | 56.9 | 55.6

### Learning with Unlabeled Data

Here we aim to study the impact of the unlabeled set of images in our proposed
weighted triangulation method for semi-supervised learning. Table 6 compares
the results of “Ours” against a standard “Baseline” as we progressively
increase the amount of available 3D supervision on the Human3.6M [12] dataset
for single view learning setup. The 2D pose estimator
$f_{{\boldsymbol{\theta}}}$ is trained using Eq. 1 for all setups and is
fixed. In “Baseline”, we train the lifting network $g_{{\boldsymbol{\phi}}}$
directly on the labeled set of images $\mathcal{L}$ for each setup using the
following loss function:

$L^{g}(\phi;f_{\theta})=L^{3D}(\phi;f_{\theta},\mathcal{L})~{}~{}~{},$ (8)

where $L^{3D}$ is the MSE loss between the predicted and the ground truth 3D
poses on $\mathcal{L}$. In Ours, in addition to the labelled set we also train
the lifting network using the estimated triangulated 3D poses on the unlabeled
data. As expected, the performance of both networks improve as the amount of
labeled data in increased gradually, however, our algorithm still manages to
outperform the fully supervised baseline network. Most importantly, in cases
with low amount of available 3D supervision (such as 1% S1, 5% S1) there
exists a significant gap in performance($\sim 30$mm) between ours and the
baseline network. This clearly demonstrates that our proposed method makes the
most of the unlabeled set of images $\mathcal{U}$ in learning the important
features to generate the appropriate supervision via weighted triangulation.

### Triangulation Robustness Analysis

In this experiment we analyze how the estimated object locations, that are the
results of applying triangulation, are affected by localization error for our
triangulation approach and the standard one. To do so, we create a simulation
using six views with real camera parameters. We first generate $N$ three-
dimensional object locations and then obtain their corresponding ground-truth
projections in each one of the views using the camera parameters. We then run
experiments in which we randomly select a subset of cameras and gradually
increase the amount of 2D noise applied to the projected points in these
views. The noise that we apply to the 2D points is uniformly sampled on a
circle of radius $r$ around the ground-truth 2D locations, where $r$ is the
level of noise. For our triangulation approach with $WSS$ of Eq. 6, when $WSS$
is bigger then a threshold $\lambda=20$ millimeters we set all the weights to
a small but equal weight. This is equivalent to using standard triangulation.

Figure 4: Triangulation Error (lower is better). Triangulation error with
respect to an increasing level of noise expressed in pixels and applied to a
varying number of views. Our triangulation approach (w/ weights w/ WSS)
performs better than standard triangulation (wo/ weights) when less then
$50\%$ of the cameras are affected by the noise and on par otherwise. Without
WSS (w/ weights wo/ WSS) instead, the triangulation produces less accurate
locations when most of the image points are erroneous.

In Figure 4 we report the MPJPE between the triangulated results and the
ground-truth object locations, where noise is applied to different number of
cameras, for standard triangulation (wo/ weights), ours with weights (w/
weights wo/ WSS) and ours with weights and $WSS$ (w/ weights w/ WSS).

It can be noted that our robust triangulation produces more accurate results
than standard triangulation when less than $50\%$ of the views are affected by
the noise and on par otherwise. When $WSS$ is disabled instead, our
triangulation produces worse results when more than $50\%$ of the views are
noisy.

Figure 5: Balancing $L_{tri}$. $L_{tri}$ provides two gradients flows that we
control with parameter $\alpha$. We show the effect on the loss and MPJPE for
different values of $\alpha$ in a controlled test setup. When $\alpha=0$ the
model can potentially reach the best MPJPE but then deviates quickly as the
training continues. On the other hand, when $\alpha=0.1$ the loss is much more
stable.

### Stability of $L_{tri}$

As discussed in Section § 3.2, the self-supervised loss term $L_{tri}$
minimizes the difference between the predicted image points
$\hat{\boldsymbol{x}}$ and the projection of the triangulated one
$\bar{\boldsymbol{x}}$ using the Mean Squared Error (MSE). Since both are
function of the network $f_{\theta}$, there are two gradients that affect the
network weights $\theta$. To better understand the contribution of each
gradient we decompose $L_{tri}$ in two terms as shown in Eq. 9. In the first
term, we consider $\bar{\boldsymbol{x}}$ to be ground-truth data, that means
that there is no gradient flowing in it. In the second term instead we
consider $\hat{\boldsymbol{x}}$ as to be the ground-truth. Note that,
$\hat{\boldsymbol{x}}$ and $\bar{\boldsymbol{x}}$ are dependent on one another
and can change from one iteration to another.

$L_{tri}(\hat{\boldsymbol{x}},\bar{\boldsymbol{x}})=\alpha
L(\hat{\boldsymbol{x}},\bar{\boldsymbol{x}}_{GT})+(1-\alpha)L(\hat{\boldsymbol{x}}_{GT},\bar{\boldsymbol{x}})$
(9)

When $\alpha=0$, the gradient flows only through the differentiable
triangulation via $\bar{\boldsymbol{x}}$. When $\alpha=1$ it flows only
through the predictions ${\boldsymbol{\hat{x}}}$, with the triangulated points
considered as the ground-truth. Since we want to exploit triangulation,
setting $\alpha=0$ seems a reasonable choice, however, we demonstrate with a
simulation (Fig. 5) that it can be counterproductive.

To do so, we run an experiment in a controlled environment by isolating the
self-supervised loss $L_{tri}$ from the rest of the pipeline; neural networks
are not used here either. We first generate $N$ three-dimensional object
locations and project them in each one of the views ($3$ in our case) to
obtain ground-truth image locations $\boldsymbol{x}$ using camera parameters.
Then, to simulate the error of the 2D pose estimator, we inject a moderate
amount (1 pixel) of Gaussian noise to all the image points and a higher amount
(10 pixels) to one view only. Finally, we optimize the position of the noisy
image locations $\hat{\boldsymbol{x}}$ using $L_{tri}$ where at each iteration
we compute $\bar{\boldsymbol{x}}$ using the standard differentiable
triangulation. The problem can be expressed as
$\hat{\boldsymbol{x}}^{*}=\operatorname*{arg\,min}_{\hat{\boldsymbol{x}}}L_{tri}(\hat{\boldsymbol{x}},\bar{\boldsymbol{x}})$.

Figure 5 depicts the loss values and the MPJPE between the triangulated points
and the ground-truth ones for different values of $\alpha$. It can be noted
that when $\alpha=0$ the MPJPE can reach its lowest value but, as the training
continues, it deviates quickly. This is due to having the model aiming to put
all the joint locations at the center of the image , which is a target that
can minimize the triangulation loss without minimizing the MPJPE error. This
means that minimizing the self-supervised loss does not correspond to
minimizing the MPJPE. On the other hand, when $\alpha=1$ the error on the
noisy image key-points is hardly reduced but is rather propagated to the other
views leading to the degradation of the learnt model. When both terms in Eqn.
9 are active, the image points that are precise deviate less and the MPJPE
curve is more stable. Note that with additional supervision from labeled data
all the MPJPE curves in Fig. 5 would improve over time. In our experiment we
found that using $\alpha=0.5$ is a good compromise.

Table 7: Quantitative results Human3.6M (Weakly-supervised)

Singleview - weakly-supervised
---
Method |  MPJPE $\downarrow$ |  NMPJPE $\downarrow$ |  PMPJPE $\downarrow$
Pavlakos  _et al_. [35] | 118.4 | - | -
Kanzawa  _et al_. [16] | 106.8 | - | 67.5
Wandt  _et al_. [48] | 89.9 | - | -
Tome  _et al_. [46] | 88.4 | - | -
Kocabas  _et al_. [18] |  | 77.75 | 70.67
Chen  _et al_. [3] | - | - | 68.0
Drover  _et al_. [7] | - | - | 64.6
Kolotouros  _et al_. [19] | - | - | 62.0
Wang  _et al_. [49] | 83.0 | - | 57.5
Iqbal  _et al_. [14] | 67.4 | 64.5 | 54.5
Kundu  _et al_. [20] | - | - | 62.4
Ours | 64.7 | 64.2 | 52.1

Table 8: Quantitative results MPI-INF-3dHP (Weakly-supervised)

Method |  MPJPE $\downarrow$ |  NMPJPE $\downarrow$ |  PMPJPE $\downarrow$ |  3DPCK $\uparrow$
---|---|---|---|---
Kanzawa  _et al_. [16] | 169.5 | - | - | 59.6
Kocabas  _et al_. [18] | 126.8 | 125.7 | - | 64.7
Kolotouros  _et al_. [19] | 124.8 | - | - | 66.8
Kundu  _et al_. [20] | - | - | 93.9 | 84.6
Iqbal  _et al_. [14] | 109.3 | 107.2 | - | 79.5
Ours | 118.4 | 109.3 | 99.3 | 73.4

### Impact of Weighted Triangulation

Here we study the impact of our proposed weighting strategy to generate
suitable triangulated 3D poses for training the lifting network
$g_{{\boldsymbol{\phi}}}$. Therefore, we report our results in the weakly-
supervised setup in Tables 4 and 4 for Human3.6M [12], MPI-INF-3DHP [28]
datasets respectively, both measured in single-view. In this learning setup,
we use 2D manual annotations for all the training images to train
$f_{{\boldsymbol{\theta}}}$ and the triangulated 3D poses obtained from 2D
predictions of the training images are the sole source of supervision to train
$g_{{\boldsymbol{\phi}}}$. Note that since the lifting network is trained on
model predictions, rather than ground truth 3D targets, this setup leads to
lower accuracy compared to the semi-supervised setup. Table 4 provides the
results for H3.6M dataset. We outperform existing solutions in all the three
metrics. In Table 4 we show results on MPI dataset, where we obtain comparable
results to other baselines. It is worth noting that while we rely only on the
triangulated 3D poses from 2D model predictions to train
$g_{{\boldsymbol{\phi}}}$, while [14, 20] use additional sources of
supervisions.

|  |  |  |
---|---|---|---|---
|  |  |  |
Figure 6: Additional qualitative results for the test set of H36M. The pose in the top image correspond to the projection of the 3D prediction in this view from our single view model trained on 10% of labeled data. This projected pose is equivalent to the prediction of the 2D pose estimator. The bottom image depicts the same 3D pose but from another viewpoint. The bottom image allow us to visualise the depth information, which would be invisible otherwise. |  |  |  |  |
---|---|---|---|---|---
Figure 7: Qualitative results for the test set of SportCenter. The pose in the top image correspond to the projection of the 3D prediction in this view from our single view model trained in semi-supervised setup. This projected pose is equivalent to the prediction of the 2D pose estimator. The bottom image depicts the same 3D pose but from another viewpoint. The bottom image allow us to visualise the depth information, which would be invisible otherwise. |  |  |
---|---|---|---
|  |  |
|  |  |
Figure 8: Qualitative results for the test set of MPI-INF-3DHP. We show the
single view 3D predictions of our model train under semi-supervised learning.

##  5 Qualitative Results

Fig. 6, 7 and 8 show single view qualitative results for Human3.6M [12],
SportCenter and MPI-INF-3DHP [28] datasets, respectively.

##  6 Conclusion

In this work, we have presented an effective way of using triangulation as a
form of self-supervision for single view 3D pose estimation. Our approach
imposes geometrical constraints on multi-view images to train models with
little annotations in semi- and weakly-supervised learning setups. This was
made possible by making the pipeline end-to-end differentiable. We have
demonstrated how the proposed robust triangulation can reliably generate
pseudo labels even in crowded scenes and how to use them to supervise a single
view 3D pose estimator. The experimental results, especially in the
SportsCenter dataset, clearly shows its advantages in learning a precise
triangulated 3D pose over other triangulation methods. In future, we plan to
further investigate the impact of integrating temporal or spatial constraints
into our learning framework.

####  6.0.1 Acknowledgment.

This work was supported in part by Innosuisse, the Swiss Innovation Agency and
Second Spectrum. We would also like to thank Victor Constantin for his
valuable insights and discussions.

## References

  * [1] Abdel-Aziz, Y., Karara, H.: Direct Linear Transformation from Comparator Coordinates into Object Space Coordinates in Close-Range Photogrammetry. In: ASP/UI Symposium on Close-Range Photogrammetry. pp. 1–18 (1971)
  * [2] Cai, Y., Ge, L., Liu, J., Cai, J., Cham, T., Yuan, J., Thalmann, N.M.: Exploiting spatial-temporal relationships for 3d pose estimation via graph convolutional networks. In: ICCV (2019)
  * [3] Chen, C., Tyagi, A., Agrawal, A., Drover, D., MV, R., Stojanov, S., Rehg, J.M.: Unsupervised 3d pose estimation with geometric selfsupervision. In: Conference on Computer Vision and Pattern Recognition (2019)
  * [4] Chen, C.H., Tyagi, A., Agrawal, A., Drover, D., Stojanov, S., Rehg, J.M.: Unsupervised 3D Pose Estimation with Geometric Self-Supervision. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 5714–5724 (2019)
  * [5] Chen, Y., Shen, C., Wei, X.S., Liu, L., Yang, J.: Adversarial posenet: A structure-aware convolutional network for human pose estimation. In: Proceedings of the IEEE International Conference on Computer Vision. pp. 1212–1221 (2017)
  * [6] Ci, H., Wang, C., Ma, X., Wang, Y.: Optimizing network structure for 3d human pose estimation. In: International Conference on Computer Vision (2019)
  * [7] Drover, D., MV, R., Chen, C., Agrawal, A., Tyagi, A., Huynh, C.P.: Can 3d pose be learned from 2d projections alone? In: European Conference on Computer Vision Workshops (2018)
  * [8] Fang, H., Xie, S., Tai, Y., Lu, C.: RMPE: Regional multi-person pose estimation. In: ICCV (2017)
  * [9] Golub, G., Loan, C.V.: Matrix Computations (Johns Hopkins Studies in Mathematical Sciences). The Johns Hopkins University Press (1996)
  * [10] Hartley, R., Schaffalitzky, F.: Reconstruction from Projections Using Grassmann Tensors. In: European Conference on Computer Vision. pp. 363–375 (2004)
  * [11] He, K., Zhang, X., Ren, S., Sun, J.: Deep Residual Learning for Image Recognition. In: Conference on Computer Vision and Pattern Recognition. pp. 770–778 (2016)
  * [12] Ionescu, C., Papava, I., Olaru, V., Sminchisescu, C.: Human3.6M: Large Scale Datasets and Predictive Methods for 3D Human Sensing in Natural Environments. IEEE Transactions on Pattern Analysis and Machine Intelligence (2014)
  * [13] Iqbal, U., Molchanov, P., Gall, T.B.J., Kautz, J.: Hand Pose Estimation via Latent 2.5 D Heatmap Regression. In: European Conference on Computer Vision. pp. 118–134 (2018)
  * [14] Iqbal, U., Molchanov, P., Kautz, J.: Weakly-Supervised 3D Human Pose Learning via Multi-view Images in the Wild (2020)
  * [15] Iskakov, K., Burkov, E., Lempitsky, V., Malkov, Y.: Learnable Triangulation of Human Pose. In: International Conference on Computer Vision. pp. 7718–7727 (2019)
  * [16] Kanazawa, A., Black, M.J., Jacobs, D.W., Malik, J.: End-To-End Recovery of Human Shape and Pose. In: Conference on Computer Vision and Pattern Recognition (2018)
  * [17] Kingma, D.P., Ba, J.: Adam: A Method for Stochastic Optimisation. In: International Conference on Learning Representations (2015)
  * [18] Kocabas, M., Karagoz, S., Akbas, E.: Self-supervised learning of 3d human pose using multi-view geometry (2019)
  * [19] Kolotouros, N., Pavlakos, G., Black, M.J., Daniilidis, K.: Learning to Reconstruct 3D Human Pose and Shape via Model-Fitting in the Loop. In: Conference on Computer Vision and Pattern Recognition (2019)
  * [20] Kundu, J.N., Seth, S., Jampani, V., Rakesh, M., Babu, R.V., Chakraborty, A.: Self-Supervised 3D Human Pose Estimation via Part Guided Novel Image Synthesis. In: CVPR (2020)
  * [21] Li, B., Yan, J., Wu, W., Zhu, Z., Hu, X.: High Performance Visual Tracking with Siamese Region Proposal Network. In: Conference on Computer Vision and Pattern Recognition (June 2018)
  * [22] Li, H., Xu, Z., Taylor, G., Studer, C., Goldstein, T.: Visualizing the Loss Landscape of Neural Nets. In: Advances in Neural Information Processing Systems. pp. 6389–6399 (2018)
  * [23] Li, S., Chan, A.B.: 3D Human Pose Estimation from Monocular Images with Deep Convolutional Neural Network. In: Asian Conference on Computer Vision (2014)
  * [24] Li, S., Zhang, W., Chan, A.B.: Maximum-Margin Structured Learning with Deep Networks for 3D Human Pose Estimation. In: International Conference on Computer Vision (2015)
  * [25] Li, Z., Wang, X., Wang, F., Jiang, P.: On boosting single-frame 3d human pose estimation via monocular videos. In: ICCV (2019)
  * [26] Liu, K., Ding, R., Zou, Z., Wang, L., Tang, W.: A comprehensive study of weight sharing in graph networks for 3d human pose estimation. In: European Conference on Computer Vision (2020)
  * [27] Martinez, J., Hossain, R., Romero, J., Little, J.: A Simple Yet Effective Baseline for 3D Human Pose Estimation (2017)
  * [28] Mehta, D., Rhodin, H., Casas, D., Fua, P., Sotnychenko, O., Xu, W., Theobalt, C.: Monocular 3D Human Pose Estimation in the Wild Using Improved CNN Supervision. In: International Conference on 3D Vision (2017)
  * [29] Mehta, D., S., S., Sotnychenko, O., Rhodin, H., Shafiei, M., Seidel, H., Xu, W., Casas, D., Theobalt, C.: Vnect: Real-Time 3D Human Pose Estimation with a Single RGB Camera. In: ACM SIGGRAPH (2017)
  * [30] Mitra, R., Gundavarapu, N.B., Sharma, A., Jain, A.: Multiview-Consistent Semi-Supervised Learning for 3D Human Pose Estimation. In: Conference on Computer Vision and Pattern Recognition. pp. 6907–6916 (2020)
  * [31] Newell, A., Yang, K., Deng, J.: Stacked Hourglass Networks for Human Pose Estimation. In: European Conference on Computer Vision (2016)
  * [32] Pavlakos, G., N. Kolotouros, Daniilidis, K.: Texturepose: Supervising human mesh estimation with texture consistency. In: International Conference on Computer Vision (2019)
  * [33] Pavlakos, G., Zhou, X., Daniilidis, K.: Ordinal Depth Supervision for 3D Human Pose Estimation. In: Conference on Computer Vision and Pattern Recognition (2018)
  * [34] Pavlakos, G., Zhou, X., Derpanis, K., Konstantinos, G., Daniilidis, K.: Coarse-To-Fine Volumetric Prediction for Single-Image 3D Human Pose. In: Conference on Computer Vision and Pattern Recognition (2017)
  * [35] Pavlakos, G., Zhou, X., Konstantinos, K.D.G., Kostas, D.: Harvesting Multiple Views for Marker-Less 3D Human Pose Annotations. In: Conference on Computer Vision and Pattern Recognition (2017)
  * [36] Pavllo, D., Feichtenhofer, C., Grangier, D., Auli, M.: 3D Human Pose Estimation in Video with Temporal Convolutions and Semi-Supervised Training. In: Conference on Computer Vision and Pattern Recognition (2019)
  * [37] Popa, A.I., Zanfir, M., Sminchisescu, C.: Deep Multitask Architecture for Integrated 2D and 3D Human Sensing. In: Conference on Computer Vision and Pattern Recognition (2017)
  * [38] Remelli, E., Han, S., Honari, S., Fua, P., Wang, R.: Lightweight Multi-View 3D Pose Estimation through Camera-Disentangled Representation. In: Conference on Computer Vision and Pattern Recognition (2020)
  * [39] Rhodin, H., Salzmann, M., Fua, P.: Unsupervised Geometry-Aware Representation for 3D Human Pose Estimation. In: European Conference on Computer Vision (2018)
  * [40] Rhodin, H., Spoerri, J., Katircioglu, I., Constantin, V., Meyer, F., Moeller, E., Salzmann, M., Fua, P.: Learning Monocular 3D Human Pose Estimation from Multi-View Images. In: Conference on Computer Vision and Pattern Recognition (2018)
  * [41] Rogez, G., Weinzaepfel, P., Schmid, C.: Lcr-Net: Localization-Classification-Regression for Human Pose. In: Conference on Computer Vision and Pattern Recognition (2017)
  * [42] Rogez, G., Weinzaepfel, P., Schmid, C.: Lcr-Net++: Multi-Person 2D and 3D Pose Detection in Natural Images. In: arXiv Preprint (2018)
  * [43] Russakovsky, O., Deng, J., Su, H., Krause, J., S.Satheesh, Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A., Fei-Fei, L.: Imagenet Large Scale Visual Recognition Challenge. International Journal of Computer Vision 115(3), 211–252 (2015)
  * [44] Sun, X., B. Xiao, F.W., Liang, S., Wei, Y.: Integral Human Pose Regression. In: European Conference on Computer Vision (2018)
  * [45] Tekin, B., Katircioglu, I., Salzmann, M., Lepetit, V., Fua, P.: Structured Prediction of 3D Human Pose with Deep Neural Networks. In: British Machine Vision Conference (2016)
  * [46] Tome, D., Russell, C., Agapito, L.: Lifting from the Deep: Convolutional 3D Pose Estimation from a Single Image. In: arXiv Preprint (2017)
  * [47] Tung, H.Y., Harley, A., Seto, W., Fragkiadaki, K.: Adversarial Inverse Graphics Networks: Learning 2D-To-3D Lifting and Image-To-Image Translation from Unpaired Supervision. In: International Conference on Computer Vision (2017)
  * [48] Wandt, B., Rosenhahn, B.: Repnet: Weakly supervised training of an adversarial reprojection network for 3d human pose estimation. In: Conference on Computer Vision and Pattern Recognition (2019)
  * [49] Wang, C., Kong, C., Lucey, S.: Distill knowledge from nrsfm for weakly supervised 3d pose learning. In: ICCV (2019)
  * [50] Zeng, A., Sun, X., Huang, F., Liu, M., Xu, Q., Lin, S.: SRNet: Improving Generalization in 3D Human Pose Estimation with a Split-and-Recombine Approach. In: European Conference on Computer Vision (2020)
  * [51] Zhang, Y., Yang, Q.: A Survey on Multi-Task Learning. In: arXiv Preprint (2017)
  * [52] Zhou, X., Huang, Q., Sun, X., Xue, X., Wei, Y.: Towards 3d human pose estimation in the wild: a weakly-supervised approach. In: International Conference on Computer Vision (2017)
  * [53] Zhou, X., Sun, X., Zhang, W., Liang, S., Wei, Y.: Deep Kinematic Pose Regression. In: European Conference on Computer Vision (2016)
  * [54] Zou, Z., Tang, W.: Modulated Graph Convolutional Network for 3D Human Pose Estimation. In: International Conference on Computer Vision (2021)