(eccv) Package eccv Warning: Package ‘hyperref’ is loaded with option
‘pagebackref’, which is *not* recommended for camera-ready version

11institutetext: ShanghaiTech University 22institutetext: United Imaging
Intelligence

# Self-learning Canonical Space for Multi-view 3D Human Pose Estimation

Xiaoben Li 1122 Mancheng Meng 22 Ziyan Wu 22 Terrence Chen 22 Fan Yang
Corresponding author.22 Dinggang Shen 1122

###### Abstract

Multi-view 3D human pose estimation is naturally superior to single view one,
benefiting from more comprehensive information provided by images of multiple
views. The information includes camera poses, 2D/3D human poses, and 3D
geometry. However, the accurate annotation of these information is hard to
obtain, making it challenging to predict accurate 3D human pose from multi-
view images. To deal with this issue, we propose a fully self-supervised
framework, named cascaded multi-view aggregating network (CMANet), to
construct a canonical parameter space to holistically integrate and exploit
multi-view information. In our framework, the multi-view information is
grouped into two categories: 1) intra-view information (_i.e_., camera pose,
projected 2D human pose, view-dependent 3D human pose), 2) inter-view
information (_i.e_., cross-view complement and 3D geometry constraint).
Accordingly, CMANet consists of two components: intra-view module (IRV) and
inter-view module (IEV). IRV is used for extracting initial camera pose and 3D
human pose of each view; IEV is to fuse complementary pose information and
cross-view 3D geometry for a final 3D human pose. To facilitate the
aggregation of the intra- and inter-view, we define a canonical parameter
space, depicted by per-view camera pose and human pose and shape parameters
($\theta$ and $\beta$) of SMPL model, and propose a two-stage learning
procedure. At first stage, IRV learns to estimate camera pose and view-
dependent 3D human pose supervised by confident output of an off-the-shelf 2D
keypoint detector. At second stage, IRV is frozen and IEV further refines the
camera pose and optimizes the 3D human pose by implicitly encoding the cross-
view complement and 3D geometry constraint, achieved by jointly fitting
predicted multi-view 2D keypoints. The proposed framework, modules, and
learning strategy are demonstrated to be effective by comprehensive
experiments and CMANet is superior to state-of-the-art methods in extensive
quantitative and qualitative analysis.

###### Keywords:

Human Pose Estimation Multi-view Self-learning

##  1 Introduction

3D human pose estimation (HPE) as a fundamental task in computer vision
involves detecting the 3D locations of body keypoints and building a geometry
representation of the human body from visual data [41, 29, 40]. It has a wide
range of applications, including human-computer interaction, augmented
reality, and virtual reality. Due to the critical importance and wide
applications of 3D HPE, numerous works have been conducted in this field.
Considerable amount of efforts are spent on recovering 3D human model
(pose/shape) from a single image [3, 21, 31, 40]. However, it is an ill-posed
task because of depth ambiguity and self-occlusion. In contrast, multi-view
images inherently contain more comprehensive information, so that it is more
feasible to perform 3D HPE [2, 24, 33].

Although multi-view images contain crucial information for 3D HPE, such as
camera pose, human pose and shape from each view, and cross-view 3D geometry,
appropriately learning the information is problematic. The challenges can be
concluded into two aspects: 1) the corresponding annotations of the
aforementioned information are scarce and labor intensive; 2) the information
is heterogeneous and hard to model in an unified space. Consequently, we
propose a cascade self-supervised framework and a canonical parameter space to
handle the two challenges, respectively.

Multi-view information is composed by multiple spatially correlated single-
view images, thus in our framework it is naturally separated into intra-view
(_i.e_., camera pose and view-dependent 3D human pose) and inter-view
information (_i.e_., cross-view complement and 3D geometry constraint) (Figure
1). The framework, named cascaded multi-view aggregating network (CMANet),
encompasses two major components: intra-view module (IRV) and inter-view
module (IEV) to process the two types of information, respectively. Within the
framework, a canonical parameter space is proposed and parameterized by camera
pose and human pose and shape parameters ($\theta$ and $\beta$) of SMPL model
[26] so that the heterogeneous information can be integrated into a
homogeneous parameter space for communication (Figure 1).

Figure 1: The illustration of the two categories of multi-view information
(intra-view and inter-view) and the brief idea of proposed cascaded multi-view
aggregation pipeline and canonical parameter space. For demonstration
simplicity, two-view images are adopted here, in practice multi-view case is
compatible.

For intra-view information, we resort to 2D keypoints and SMPL model [26] to
recover the camera pose and view-dependent 3D human pose in IRV. Since the
state-of-the-art 2D keypoint detectors achieve human-competitive performance,
an off-the-shelf detector [39] is adopted to yield confident 2D keypoints,
screening by confidence score. Due to the differentiable parametric property
of SMPL model and inspiration from HMR [14] and SPIN [17], the camera pose and
view-dependent 3D human pose can be estimated by minimizing reprojection error
with the 2D keypoints and represented in a canonical parameter space. This
procedure is employed on each view of image to obtain parameterized
representation of intra-view information.

The resulting 3D human poses are inaccurate owing to the single-view
limitations: depth ambiguity and self-occlusion problems. Given camera pose,
multi-view of image can construct the geometry of 3D space. The different
views of image can complement each other to capture more complete human pose,
especially when some keypoints are exclusively visible in certain views.
Therefore, the cross-view complementary and 3D geometrical characteristics of
inter-view information can serve to mitigate the limitations. To leverage the
inter-view information, we aggregate the estimated per-view camera pose and 3D
human pose into IEV as the initial value of the canonical parameter space for
a shallow network to refine camera poses and optimize a 3D human pose by
simultaneously minimizing the reprojection error with multi-view 2D keypoints.
In this way, the inter-view information is implicitly exploited in a canonical
space for 3D human pose.

The contributions of the work are summarized into three-fold: I. A fully
unsupervised cascaded framework together with a canonical parameter space are
proposed for 3D human pose estimation; II. According to the characteristics of
multi-view information, a two-stage learning procedure is proposed to
facilitate modeling; III. Extensive experiments on large datasets validate the
efficacy and superiority of the proposed components and method.

##  2 Related Work

Single-view 3D Human Pose Estimation. With the rapid development of deep
neural networks, there are many works estimating 3D pose of human using
single-view image. These works can be further divided into: direct estimation
methods and 2D to 3D lifting methods. Direct estimation methods infer the 3D
human pose from 2D image without intermediately estimating 2D pose
representation. The capacity of deep neural networks for direct 3D human pose
estimation is fully explored by CNN-based methods [21, 31, 40] and
transformer-based methods [47, 46]. Recently, diffusion-based method is also
adopted for direct 3D human pose estimation [7] and the field is further
advancing. Although promising results have been achieved, these methods
typically require 3D supervision which is not satisfied in real-world
scenario. 2D to 3D lifting methods infer 3D human pose from the intermediately
estimated 2D human pose [3, 29]. This type of methods are demonstrated to be
effective [29] but the effect is dependent on the input 2D poses and 3D
supervision is also required. Besides, 3D human mesh recovery methods [14, 17]
predict human mesh that contains rich shape and pose information directly from
a image. In these methods the mesh is represented as parameters of a skinned
vertex-based model, SMPL [26] .

Figure 2: The architecture of the proposed cascaded multi-view aggregating
network (CMANet) consisting of two components: intra-view module (IRV) and
inter-view module (IEV). IRV estimates intra-view information, _i.e_., camera
pose and 3D human pose and shape, of each view. After screening, IEV leverages
the camera poses, global orientations, the optimal human body pose and shape
to initialize proposed canonical parameter space, then aggregates the features
from all views to refine the camera pose and human model to output final human
3D keypoints. Furthermore, a two-stage learning procedure is adopted. At the
first stage IRV learns to extract intra-view information of each view,
supervised by reprojection loss provided by 2D keypoints detector and SMPL
loss provided by SMPLify. At the second stage IRV is frozen and IEV learns to
fuse multi-view information supervised by reprojection loss and SMPL loss from
all views.

Multi-view 3D Human Pose Estimation. Although great progress have been made in
single-view 3D pose estimation, these methods still suffer from occlusion and
depth ambiguity. To tackle the problems, multi-view 3D pose estimation exploit
geometric information from multiple views to infer 3D pose. A group of methods
[2, 32] utilize Pictorial Structure Model (PSM) to represent the human body
and infer the 3D pose from 2D poses by optimizing model parameters to match
the model projection with the 2D pose. However, multi-view information is not
fully and effectively fused in these PSM-based methods. Therefore, many
efforts have been dedicated to fuse information from multiple observations of
human body [13, 33]. Volumetric representation [13] and epipolar geometry [33,
27, 28] are explored to fuse features from different views with camera
parameters and infer 3D pose through a neural network There are also works
focusing on multi-view methods without need of camera calibrations [9, 38],
which typically explore features of human body that invariant to the camera
position.

Unsupervised 3D Human Pose Estimation. Although deep neural networks have
shown powerful capabilities in the field of 3D human pose estimation, their
main drawback is the requirement of large amount of labeled data. There are
large benchmarks [12, 30] collected in constrained laboratory scenarios but
few datasets are collected in unconstrained wild environment, leading to large
gap between research and practice. Therefore, many efforts have attempted to
explore unsupervised methods for 3D human pose estimation [4, 6, 11, 16, 22,
35, 36]. Multi-view consistency constraint is utilized as supervision to train
a neural network [36], but the method still need a small amount of 3D ground-
truth data to ensure accurate prediction. Adversarial learning strategy is
also adopted to provide supervision [6]. Multi-view information is also
harvested to generate 3D annotations from multi-view input [32] and [16]. In a
recent work [11], solely multiple-view geometry is provided to estimate 3D
keypoints from multi-view images. It leverages reprojection and estimation of
human mask to ensure that the estimated 3D keypoints are meaningful. Creating
realistic synthetic data [24, 8] is another way to avoid the dependence of
ground-truth annotations while extra efforts are required.

Difference from Previous Works. 1) Our method does not need any forms of
supervision from ground-truth annotations: camera pose, 2D/3D human pose. 2)
We construct a canonical parameter space to holistically integrate and exploit
heterogeneous multi-view information. 3) Our method contains a two-stage
learning strategy to exploit intra-view and inter-view information.

##  3 Method

###  3.1 SMPL-based Human Body Representation

Skinned Multi-Person Linear Model (SMPL) [26] is a skinned vertex-based model
and can accurately represent a wide variety of body shapes in natural human
poses. It provides a function
$M(\beta,\theta;\Phi):\mathbb{R}^{|\theta|\times|\beta|}\mapsto\mathbb{R}^{3N}$,
mapping shape and pose parameters to $N=6890$ vertices to form a mesh
representation of human body. In the function $\beta\in\mathbb{R}^{10}$,
$\theta\in\mathbb{R}^{24\times 3}$, and $\Phi$ are shape, pose and learning
model parameters, respectively. The shape parameter $\beta$ is coefficient of
a low-dimensional shape space, learned from a training set of thousands of
registered scans. The pose parameter $\theta$ includes global rotation
$\theta_{g}\in\mathbb{R}^{3}$ and body pose parameter
$\theta_{b}\in\mathbb{R}^{23\times 3}$, consequently
$\theta=\\{\theta_{g},\theta_{b}\\}$.

Typically, a pre-trained linear regressor $W$ is used to obtain $k$ keypoints
of interest of the whole body, _i.e_., $J_{3D}\in\mathbb{R}^{k\times
3}=WM(\beta,\theta;\Phi)$ [17]. As for 2D keypoints, a perspective camera
model with fixed focal length and intrinsics $K$ is used to project 3D joint
positions into the 2D image plane. Each camera pose $\pi=\\{R,t\\}$ consists
of global orientation $R\in\mathbb{R}^{3}$ and translation
$t\in\mathbb{R}^{3}$. Given above parameters, 3D keypoints can be projected to
the 2D image as $J_{2D}=\Pi(K(RJ_{3D}+t))$, where $\Pi$ is a perspective
projection with camera intrinsics $K$. Since the pose parameter $\theta$
already includes a global orientation $\theta_{g}$, in practice we assume $R$
of the camera model as identity matrix and only predict camera translation
$t$.

###  3.2 Cascaded Multi-view Aggregating Network

The architecture of the proposed cascaded multi-view aggregating network
(CMANet) is depicted in Figure 2, consisting of two major components: intra-
view module (IRV) and inter-view module (IEV). Formally, given $N$-view images
$\\{I^{i}\\}_{i=1}^{N}$ of a person, IRV firstly extracts camera pose, 3D
human model and image features of each view
$\\{\theta^{i},\beta^{i},\pi^{i},F^{i}\\}_{i=1}^{N}$ with $N$ shared encoders
and $N$ regressors. IEV leverages the camera poses $\\{\pi^{i}\\}_{i=1}^{N}$,
global orientations $\\{\theta_{g}^{i}\\}_{i=1}^{N}$, the optimal human body
pose and shape $\tilde{\theta}_{b}$ and $\tilde{\beta}$ screened from all
views as initial parameter of the canonical space, and aggregates the features
of each view into a light-weight sub-network to further optimize human model
and refine camera poses. In learning phase, the weights of IRV is first
optimized independently, then it is frozen and IEV learns to estimate final
human pose. In addition, all $\beta,\theta,\pi$ are supervised by deteteced 2D
keypoints.

###  3.3 Intra-view Module

This module aims to extract intra-view information, _i.e_., camera pose
($\pi=\\{R,t\\}$) and 3D human pose and shape ($\\{\theta,\beta\\}$) of each
view. Since intra-view information is contained in each single view of image
and independent from cross-view geometry, each view of image is processed
independently. More specifically, given $N$ views of images
$\\{I^{i}\\}_{i=1}^{N}$ of a human body, the images are first forwarded into
$N$ weights-shared encoders for feature extraction. Subsequently, the
extracted feature map $F^{i}$ of $i$-th image is reshaped into a fixed-length
feature vector and fed into a corresponding $i$-th regressor to yield $i$-th
camera pose $\pi^{i}$, 3D human pose $\theta^{i}$ and shape $\beta^{i}$ of
$i$-th view.

Due to the powerful feature representation capacity of Swin-Transformer [25],
we adopt it as architecture of the encoder. Considering the success of
previous work [14], instead of directly regressing parameters in one go, we
regress camera pose and human model parameters in an iterative manner, where
mean value of camera pose and human pose and shape parameters are set as the
initial value of parameters. We further utilize the representation proposed by
[48] for the 3D rotations as in [17] for faster convergence during learning.
The $N$ regressors are constructed by $N$ weights-unshared two-layer fully
connected network.

###  3.4 Inter-view Module

The module is to extract inter-view information, _i.e_., cross-view complement
and 3D geometry constrain, and fuse complementary intra-view human and camera
pose information and cross-view 3D geometry. To facilitate the aggregation of
the intra- and inter-view information, we define a holistic canonical
parameter space for multi-view pose representation, which contains per-view
camera pose $\\{\pi^{i}\\}_{i=1}^{N}=\\{R^{i},t^{i}\\}_{i=1}^{N}$, global
orientations of each view $\\{\theta_{g}^{i}\\}_{i=1}^{N}$, and view-
independent shape and body pose parameters $\\{\theta_{b},\beta\\}$. Note that
in multi-view scenarios, different views correspond to the same human body
thus have the same shape and body pose parameters $\\{\theta_{b},\beta\\}$,
which are view-independent. On the contrary, the camera poses of different
views differ from each other, so we have $N$ global orientations
$\\{\theta_{g}^{i}\\}_{i=1}^{N}$ ($R_{i}$ is assumed to be identity matrix).

Table 1: Ablation study of each component in our CMANet on MPI-INF-3DHP dataset. Method | Metrics
---|---
IEV | IRV |  Two-stage learning | Absolute | Rigid Alignment
PCK $\uparrow$ | AUC $\uparrow$ | MPJPE $\downarrow$ | PCK $\uparrow$ | AUC $\uparrow$ | MPJPE $\downarrow$
✓ | ✗ | ✗ | 82.0 | 30.7 | 117.8 | 87.1 | 51.6 | 75.8
✓ | ✓ | ✗ | 79.1 | 29.3 | 135 | 87.0 | 51.2 | 79.8
✓ | ✓ | ✓ | 89.1 | 46.1 | 93.7 | 93.9 | 59.2 | 68.2
Table 2: Ablation study of visible keypoints for learning on MPI-INF-3DHP dataset. ‘Min_Max’ indicates that we use the view that have most visible keypoints and the view that have fewest visible keypoints, and ‘Min_2’ indicates that we use the 2 views that have fewest visible keypoints. Num of views | Metrics
---|---
Absolute | Rigid Alignment
PCK $\uparrow$ | AUC $\uparrow$ | MPJPE $\downarrow$ | PCK $\uparrow$ | AUC $\uparrow$ | MPJPE $\downarrow$
Min_Max | 70.1 | 27.8 | 154.4 | 85.0 | 42.3 | 100.6
Min_2 | 65.4 | 27.2 | 168.5 | 79.4 | 38.0 | 107.3
4 | 89.1 | 46.1 | 93.7 | 93.9 | 59.2 | 68.2
Table 3: Comparisons on MPJPE and PA-MPJPE (both in mm) on the Human3.6M test
set with other methods. ‘Multi-view’ indicates methods which take multi-view
images or videos as input. ‘Cameras’ indicates methods which need camera
extrinsic parameters. ‘Temporal’ indicates methods which use temporal
information, _i.e_., videos for human pose/shape estimation.

Method | Supervision | Multi-view | Cameras | Temporal | Metrics
---|---|---|---|---|---
MPJPE $\downarrow$ | PA-MPJPE $\downarrow$
Rhodin et al. [35] | J3D | ✓ | ✓ | ✗ | 131.70 | 98.20
HMR [14] | J2D/Adv. | ✗ | ✗ | ✗ | 106.84 | 67.45
SPIN [17] | J2D | ✗ | ✗ | ✗ | - | 62.00
Liang et al. [24] | J2D/J3D/Mesh | ✓ | ✗ | ✗ | 79.85 | 45.13
Qiu et al. [33] | J2D | ✓ | ✓ | ✗ | 26.20 | -
VIBE [15] | J2D/J3D/Mesh/Adv. | ✗ | ✗ | ✓ | 65.60 | 41.40
AdaFuse [45] | J2D | ✓ | ✓ | ✗ | 19.50 | -
Shin et al. [37] | J2D/J3D/Mesh | ✓ | ✓ | ✗ | 46.90 | 32.50
Li et al. [23] | J2D/J3D | ✓ | ✗ | ✗ | 64.80 | 43.80
ProHMR [18] | J2D/J3D/Mesh | ✗ | ✗ | ✗ | 62.20 | 34.50
TesseTrack [34] | Mesh | ✓ | ✓ | ✓ | 18.70 | -
Shuai et al.[38] | J3D | ✓ | ✗ | ✓ | 26.20 | -
Chun et al.[5] | Mesh | ✓ | ✓ | ✗ | 17.59 | -
Kocabas et al. [16] | Self | ✓ | ✗ | ✗ | 76.60 | 67.45
Kundu et al. [19] | Self | ✗ | ✗ | ✗ | 99.20 | -
Kundu et al. [20] | Self | ✗ | ✗ | ✗ | - | 89.40
Honari et al. [11] | Self | ✓ | ✓ | ✗ | 73.80 | 63.00
Honari et al. [10] | Self | ✗ | ✗ | ✓ | 100.30 | 74.90
Ours | Self | ✓ | ✗ | ✗ | 64.48 | 51.50

Table 4: Comparison results on TotalCapture test set in terms of 3D pose
errors MPJPE (mm). Note that IMU-PVH [42] uses information from the IMU
sensors as supervision.

Method | Supervision | Subjects 1,2,3 | Subjects 4,5 | Mean
---|---|---|---|---
Walking 2 | Acting 3 | Freestyle 3 | Walking 2 | Acting 3 | Freestyle 3
Tri-CPM [44] | 3D | 79.0 | 106.5 | 112.1 | 79.0 | 73.7 | 149.3 | 99.8
PVH [42] | 3D | 48.3 | 94.3 | 122.3 | 84.3 | 154.5 | 168.5 | 107.3
IMU-PVH [42] | 3D | 30.0 | 49.0 | 90.6 | 36.0 | 109.2 | 112.1 | 70.9
Trumble et al. [43] | 3D | 42.0 | 59.8 | 120.5 | 58.4 | 103.4 | 162.1 | 85.4
ProHMR [18] | Mesh | 125.7 | 118.9 | 134.3 | 131.9 | 125.2 | 135.8 | 127.8
Ours | Self | 76.6 | 74.2 | 74.3 | 80.8 | 77.2 | 86.7 | 77.9

Table 5: Comparison results on MPI-INF-3DHP with and without rigid alignment. The top half is methods using different forms of supervision while the bottom half is self-supervised methods. Method | Metrics
---|---
Absolute | Rigid Alignment
PCK $\uparrow$ | AUC$\uparrow$ | MPJPE$\downarrow$ | PCK $\uparrow$ | AUC $\uparrow$ | MPJPE$\downarrow$
HMR [14] | 59.6 | 27.9 | 169.5 | 77.1 | 40.7 | 113.2
SPIN [17] | 66.8 | 30.2 | 124.8 | 87.0 | 48.5 | 80.4
Liang et al. [24] | - | - | - | 95 | 63 | 62
VIBE [15] | - | - | 96.6 | 89.3 | - | 64.6
Shin et al. [37] | - | - | - | 97.4 | 65.5 | 50.2
Li et al. [23] | 79.2 | 39.3 | 98.7 | 92.9 | 56.1 | 65.6
ProHMR [18] | - | - | - | - | - | 65.0
Chun et al. [5] | 94.1 | 59.3 | 68.0 | - | - | -
Kocabas et al. [16] | 64.7 | - | 126.79 | - | - | -
Kundu et al. [19] | - | - | - | 83.2 | 58.7 | 97.6
Kundu et al. [20] | - | - | - | 79.2 | 43.4 | 99.2
Honari et al. [11] | - | - | 145.7 | - | - | 115.2
Honari et al. [10] | - | - | 209.5 | - | - | 140.4
Ours | 89.1 | 46.1 | 93.7 | 93.9 | 59.2 | 68.2

As shown in Figure 2, after IRV extracts camera pose and human pose of each
view, we forwarded the optimal human pose, shape, all camera poses and global
orientations into aforementioned canonical parameter space. The screened
parameters together with features from all views are fused by a single Swin
Transformer layer (STL), following by a regressor to output the final human
pose, shape and camera poses. Specifically, the IRV estimated camera pose
parameters $\\{R^{i},t^{i}\\}_{i=1}^{N}$ and global orientations
$\\{\theta_{g}^{i}\\}_{i=1}^{N}$ are used as the initial for regression in
IEV. From $\\{\theta^{i}_{b},\beta^{i}\\}_{i=1}^{N}$, we select the one with
least 2D reprojection error as the initial view-independent shape and body
pose parameters $\\{\tilde{\theta_{b}},\tilde{\beta}\\}$. The features
$\\{F^{i}\\}_{i=1}^{N}$ from all views are fused by a Swin Transformer layer
and forwarded into a regressor to refine human model and camera pose, so as to
output final human pose.

###  3.5 Self-supervised Two-stage Learning

In our CMANet, intra-view module (IRV) and intra-view module (IEV) exploit
intra-view and inter-view information, respectively. As the two modules are
designed cascadedly and decoupledly, it is natural to optimize the two module
in a staged manner. Specifically, we propose a two-stage learning procedure.
At the first stage, IRV learns to estimate camera pose and view-dependent 3D
human pose supervised by confident output of an off-the-shelf 2D keypoint
detector. At the second stage, IRV is frozen and IEV further learns to refine
the camera pose and optimize the 3D human pose by implicitly encoding the
cross-view complement and 3D geometry constraint, achieved by jointly fitting
the estimated multi-view 2D keypoints.

In learning phase, no ground truth 2D or 3D annotations are involved, instead,
we adopt a strong 2D keypoint detector [39] to yield body keypoints
$\hat{\bm{J}}_{\rm 2D}=\\{\hat{J}_{\rm 2D}^{i}\\}_{i=1}^{N}$ as well as
corresponding confidence scores $\bm{C}=\\{C^{i}\\}_{i=1}^{N}$. Then, we adopt
following strategies to process the keypoint information. First, we filter out
the keypoints whose confidence scores are less than a certain threshold
$\lambda$ and obtain a new set of confidence scores $\bm{C}_{\rm
Filt}=\\{C_{\rm Filt}^{i}\\}_{i=1}^{N}$. Second, when computing 2D
reprojection loss, we use the confidence scores to reweight keypoint-wise
loss, _i.e_., the keypoints with higher confidence score have more
contribution to total loss. At the first stage, IRV is optimized using 2D
reprojection loss function. Given predicted camera pose $\\{R^{i},t^{i}\\}$
and SMPL parameters $\beta^{i},\theta^{i}$ for $i$-th view, 2D keypoints of
the views $J_{\rm 2D}^{i}$ are calculated by regression and projection, the 2D
reprojection loss function is defined as

$L_{\rm 2D}^{i}=C_{\rm Filt}^{i}\|\hat{J}_{\rm 2D}^{i}-J_{\rm
2D}^{i}\|^{2}_{2}$ (1)

However, as argued by [17], this 2D reprojection supervisory signal is not
strong enough for the network to learn. Thus we introduce a SMPLify [1]
optimization routine in the learning loop. The loss function for SMPL
parameter is defined as

$L_{\rm SMPL}^{i}=\|\\{\beta_{\rm opt}^{i},\theta_{\rm
opt}^{i}\\}-\\{\beta^{i},\theta^{i}\\}\|^{2}_{2}$ (2)

The total loss function for $i$-th view at the first stage is

$\displaystyle L_{1}^{i}=\alpha L_{\rm 2D}^{i}+\gamma L_{\rm SMPL}^{i}$ (3)

where $\alpha$ and $\gamma$ are hyper-parameters to balance the 2D
reprojection loss and SMPL parameter loss. Note that at the first stage, we
have $N$ loss functions for $N$ views, each of them is used to supervise the
corresponding regressor and shared encoder.

At the second learning stage, the weights of IRV is frozen, and we optimize
IEV using 2D reprojection loss from $N$ views. The difference between the 2D
reprojection loss used here and that at the first stage is that instead of
backpropagating loss for each view separately, we update weights of IEV using
summed 2D reprojection loss from all views. In this way, the Swin Transformer
layer and regressor can implicitly model the complementary human pose and the
3D geometry constraint. Therefore, the 2D reprojection loss function for the
second learning stage is defined as:

$L_{\rm 2D}=\sum_{i}^{N}C_{\rm Filt}^{i}\|\hat{J}_{\rm 2D}^{i}-J_{\rm
2D}^{i}\|^{2}_{2}$ (4)

To utilize additional prior information as supervision, we also introduce a
optimization routine at the second stage, which is a multi-view adaptation of
SMPLify [1], similar to [23]. The SMPL parameter loss function for the second
stage is defined as

$\displaystyle L_{\rm SMPL}=$ $\displaystyle\|\\{\beta_{\rm
opt},(\theta_{b})_{\rm opt}\\}-\\{\beta,\theta_{b}\\}\|^{2}_{2}+$ (5)
$\displaystyle\sum_{i}^{N}\|(\theta^{i}_{g})_{\rm
opt}-\theta^{i}_{g}\|^{2}_{2}$

The total loss function for the second stage is:

$\displaystyle L_{2}=\alpha L_{\rm 2D}+\gamma L_{\rm SMPL}$ (6)

where $\alpha$ and $\gamma$ are identical to the ones in the loss function for
the first stage.

##  4 Experiments

###  4.1 Experimental Setup

Datasets. We evaluate our method on Human3.6M [12], MPI-INF-3DHP [30] and
TotalCapture [42] datasets. Human3.6M is a large 3D human pose benchmark which
contains 11 different subjects and each subject performs 15 different daily
actions indoors. The subjects are captured by 4 synchronized 50 Hz digital
cameras. The dataset provides 2D/3D keypoint annotations obtained by marked-
based motion capture system. We use subjects 9 and 11 in protocol 1 for model
learning and validation. MPI-INF-3DHP contains 8 actors (4 males and 4
females) performing 8 human activities. 2D and 3D keypoint annotations are
also provided by marker-less motion capture system. The dataset contains
multi-view images of 14 cameras with regular lenses, and we choose views 0, 2,
7, 8 from them. In addition, the standard test dataset of MPI-INF-3DHP only
have single-view images, so we use 8 for learning and validation, similar to
[37, 5]. TotalCapture dataset consists of 1.9 million frames captured from 8
calibrated full HD video cameras recording at 60Hz. The images of
“Freestyle3”, “Walking2” and “Acting3” on subjects 1,2,3,4 and 5 are used for
our experiments. In addition, these images from four cameras (1,3,5,7) in
experiments and are without the IMU sensors.

Evaluation Metrics. For Human3.6M dataset, we use mean per joint position
error (MPJPE) and reconstruction error as metrics. Reconstruction error is
MPJPE after rigid alignment of the prediction with ground truth via
Procrustes. The reconstrution error is also known as PA (procrustes
aligned)-MPJPE. For MPI-INF-3DHP, this dataset is collected indoors and
outdoors with a multi-camera marker-less MoCap system. Thus, the ground truth
3D annotations have some noise. In addition to MPJPE, the Percentage of
Correct Keypoints (PCK) thresholded at 150mm and the Area Under the Curve
(AUC) over a range of PCK thresholds before and after rigid alignment are both
reported. For TotalCapture dataset, we report mean per joint position error
(MPJPE) of different subjects and different action sequences as previous work
[42].

Implementation Details. In our implementation, input size of all images is
256$\times$256\. IRV module is optimized for 10 epochs at the first stage and
IEV is optimized for 20 epochs at the second stage with a fixed learning rate
$1e-5$. All the experiments are performed on 4 NVIDIA Quadro RTX 6000 GPUs
using PyTorch.

###  4.2 Ablation Study

The Effect of Each Component. We perform ablation study on MPI-INF-3DHP
dataset. First, in Table 1 we evaluate each component of CMANet, including an
intra-view module (IRV), an inter-view module (IEV) and two-stage learning
procedure. Observing the 1st row of 1, IEV alone is not enough to produce
accurate 3D pose. However, when adding IRV without using two-stage learning,
the performance decrease notably, we attribute the reason to follows. Although
IRV can aggregate information of multi-view, but it is harder to optimize IEV
and IRV simultaneously as cross-view complement and 3D geometry constraint is
not easy to directly explored by neural network. When introducing the proposed
two-stage learning strategy, we observe apparent improvement in performance,
which demonstrates the correctness of aforementioned reasoning.

The Effect of Visible Keypoints. We explore how the number of visible views of
keypoint affect the accuracy of 3D pose in Table 2. We call one keypoint
‘visible’ in a certain view if its confidence value is higher than a
threshold, that means if a keypoint is visible in a certain view, it can
provide 2D reprojection supervision. We design two different settings:
‘Min_Max’ and ‘Min_2’. ‘Min_Max’ indicates that we use the view that have most
visible keypoints and the view that have fewest visible keypoints, and ‘Min_2’
indicates that we use the 2 views that have fewest visible keypoints. The
results illustrate that more visible keypoints used for learning is essential
to obtain promising results. In addition, even in fewest visible scenarios,
our method can still generate reasonable results, which can be attributed to
the effective fusion of complementary cross-view information in the proposed
canonical space.

###  4.3 Comparison with State-of-the-art Methods

Human3.6M. In Table 3, we compare our method with state-of-the-art methods on
test set of Human3.6M Protocol 1. Because Human3.6M is such a popular public
benchmark, many works report results on it. So we distinguish these methods in
different ways, _e.g_., whether using multi-view information, the type of
supervision. Compared to other self-supervision methods, our method not only
is superior, but also recovers human mesh beyond human pose while not
requiring any additional information. In addition to this, our method is
comparable to some supervised human mesh recovery methods like [23, 18].

MPI-INF-3DHP. Comparison with state-of-the-art methods on MPI-INF-3DHP dataset
is listed in Table 5. Similar to the result on Human3.6M, our method
outperforms other self-supervised methods and is comparable to some of
supervised methods (including mesh recovery methods) on all metrics (PCK, AUC
and MPJPE).

Figure 3: Visualization of reconstructed human mesh and 3D pose of samples
from Human3.6M (row 1), MPI-INF-3DHP (row 2) and TotalCapture (row 3).

TotalCapture. The comparison results with state-of-the-art methods on
TotalCapture dataset are listed in Table 4. All methods excepts ours use
various forms of supervision and IMU-PVH [42] uses information from the IMU
sensors. Our method outperforms other supervised methods that do not use IMU
sensors and is comparable to IMU-PVH [42].

###  4.4 Qualitative Results

Figure 3 presents qualitative results of examples from the three datasets used
in experiments. We show the reconstructed human mesh and the final 3D pose.
Notice that the results are promising both for mesh and 3D pose even in the
presence of self-occlusion, demonstrating the effectiveness of our method to a
certain extent.

##  5 Conclusion

This paper presents a novel fully self-supervised framework, cascaded multi-
view aggregating network (CMANet), for multi-view human pose estimation, where
a canonical parameter space is constructed to learn multi-view information
holistically. CMANet includes an intra-view module to extract view-dependent
information from each view and an inter-view module to fuse multiple intra-
view information, which are complementary to each other. In addition, a two-
stage learning strategy is designed to facilitate CMANet to model human pose
more effectively. In future work, more prior information and better fusion
strategy could be explored to further improve the performance of pose
estimation.

## References

  * [1] Bogo, F., Kanazawa, A., Lassner, C., Gehler, P., Romero, J., Black, M.J.: Keep it SMPL: Automatic estimation of 3D human pose and shape from a single image. In: ECCV (2016)
  * [2] Burenius, M., Sullivan, J., Carlsson, S.: 3d pictorial structures for multiple view articulated pose estimation. In: CVPR (2013)
  * [3] Chen, C.H., Ramanan, D.: 3d human pose estimation = 2d pose estimation + matching. CVPR (2017)
  * [4] Chen, C.H., Tyagi, A., Agrawal, A., Drover, D., MV, R., Stojanov, S., Rehg, J.M.: Unsupervised 3d pose estimation with geometric self-supervision. In: CVPR (2019)
  * [5] Chun, S., Park, S., Chang, J.Y.: Learnable human mesh triangulation for 3d human pose and shape estimation. In: WACV (2023)
  * [6] Drover, D., Chen, C.H., Agrawal, A., Tyagi, A., Phuoc Huynh, C.: Can 3d pose be learned from 2d projections alone? In: ECCV (2018)
  * [7] Gong, J., Foo, L.G., Fan, Z., Ke, Q., Rahmani, H., Liu, J.: Diffpose: Toward more reliable 3d pose estimation. In: CVPR (2023)
  * [8] Gong, X., Song, L., Zheng, M., Planche, B., Chen, T., Yuan, J., Doermann, D., Wu, Z.: Progressive multi-view human mesh recovery with self-supervision. In: AAAI (2023)
  * [9] Gordon, B., Raab, S., Azov, G., Giryes, R., Cohen-Or, D.: Flex: Extrinsic parameters-free multi-view 3d human motion reconstruction. In: ECCV (2022)
  * [10] Honari, S., Constantin, V., Rhodin, H., Salzmann, M., Fua, P.: Temporal representation learning on monocular videos for 3d human pose estimation. IEEE TPAMI (2023)
  * [11] Honari, S., Fua, P.: Unsupervised 3d keypoint estimation with multi-view geometry. arXiv preprint (2022)
  * [12] Ionescu, C., Papava, D., Olaru, V., Sminchisescu, C.: Human3.6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE TPAMI (2014)
  * [13] Iskakov, K., Burkov, E., Lempitsky, V., Malkov, Y.: Learnable triangulation of human pose. In: ICCV (2019)
  * [14] Kanazawa, A., Black, M.J., Jacobs, D.W., Malik, J.: End-to-end recovery of human shape and pose. In: CVPR (2018)
  * [15] Kocabas, M., Athanasiou, N., Black, M.J.: Vibe: Video inference for human body pose and shape estimation. In: CVPR (2020)
  * [16] Kocabas, M., Karagoz, S., Akbas, E.: Self-supervised learning of 3d human pose using multi-view geometry. In: CVPR (2019)
  * [17] Kolotouros, N., Pavlakos, G., Black, M., Daniilidis, K.: Learning to reconstruct 3d human pose and shape via model-fitting in the loop. In: ICCV (2019)
  * [18] Kolotouros, N., Pavlakos, G., Jayaraman, D., Daniilidis, K.: Probabilistic modeling for human mesh recovery. In: ICCV (2021)
  * [19] Kundu, J.N., Seth, S., Jampani, V., Rakesh, M., Babu, R.V., Chakraborty, A.: Self-supervised 3d human pose estimation via part guided novel image synthesis. In: CVPR (2020)
  * [20] Kundu, J.N., Seth, S., Rahul, M., Rakesh, M., Radhakrishnan, V.B., Chakraborty, A.: Kinematic-structure-preserved representation for unsupervised 3d human pose estimation. In: AAAI (2020)
  * [21] Li, S., Chan, A.B.: 3d human pose estimation from monocular images with deep convolutional neural network. In: ACCV (2014)
  * [22] Li, Y., Li, K., Jiang, S., Zhang, Z., Huang, C., Xu, R.Y.D.: Geometry-driven self-supervised method for 3d human pose estimation. In: AAAI (2020)
  * [23] Li, Z., Oskarsson, M., Heyden, A.: 3d human pose and shape estimation through collaborative learning and multi-view model-fitting. In: WACV (2021)
  * [24] Liang, J., Lin, M.C.: Shape-aware human pose and shape reconstruction using multi-view images. In: ICCV (2019)
  * [25] Liu, Z., Lin, Y., Cao, Y., Hu, H., Wei, Y., Zhang, Z., Lin, S., Guo, B.: Swin transformer: Hierarchical vision transformer using shifted windows. In: ICCV (2021)
  * [26] Loper, M., Mahmood, N., Romero, J., Pons-Moll, G., Black, M.J.: SMPL: A skinned multi-person linear model. ACM TOG (2015)
  * [27] Ma, H., Chen, L., Kong, D., Wang, Z., Liu, X., Tang, H., Yan, X., Xie, Y., Lin, S.Y., Xie, X.: Transfusion: Cross-view fusion with transformer for 3d human pose estimation. In: BMVC (2021)
  * [28] Ma, H., Wang, Z., Chen, Y., Kong, D., Chen, L., Liu, X., Yan, X., Tang, H., Xie, X.: Ppt: token-pruned pose transformer for monocular and multi-view human pose estimation. In: ECCV (2022)
  * [29] Martinez, J., Hossain, R., Romero, J., Little, J.J.: A simple yet effective baseline for 3d human pose estimation. In: ICCV (2017)
  * [30] Mehta, D., Rhodin, H., Casas, D., Fua, P., Sotnychenko, O., Xu, W., Theobalt, C.: Monocular 3d human pose estimation in the wild using improved cnn supervision. In: 3DV (2017)
  * [31] Pavlakos, G., Zhou, X., Derpanis, K.G., Daniilidis, K.: Coarse-to-fine volumetric prediction for single-image 3D human pose. In: CVPR (2017)
  * [32] Pavlakos, G., Zhou, X., Derpanis, K.G., Daniilidis, K.: Harvesting multiple views for marker-less 3d human pose annotations. In: CVPR (2017)
  * [33] Qiu, H., Wang, C., Wang, J., Wang, N., Zeng, W.: Cross view fusion for 3d human pose estimation. In: ICCV (2019)
  * [34] Reddy, N.D., Guigues, L., Pishchulin, L., Eledath, J., Narasimhan, S.G.: Tessetrack: End-to-end learnable multi-person articulated 3d pose tracking. In: CVPR. pp. 15190–15200 (2021)
  * [35] Rhodin, H., Salzmann, M., Fua, P.: Unsupervised geometry-aware representation for 3d human pose estimation. In: ECCV (2018)
  * [36] Rhodin, H., Spörri, J., Katircioglu, I., Constantin, V., Meyer, F., Müller, E., Salzmann, M., Fua, P.: Learning monocular 3d human pose estimation from multi-view images. In: CVPR (2018)
  * [37] Shin, S., Halilaj, E.: Multi-view human pose and shape estimation using learnable volumetric aggregation. arXiv preprint (2020)
  * [38] Shuai, H., Wu, L., Liu, Q.: Adaptive multi-view and temporal fusing transformer for 3d human pose estimation. IEEE TPAMI (2022)
  * [39] Sun, K., Xiao, B., Liu, D., Wang, J.: Deep high-resolution representation learning for human pose estimation. In: CVPR (2019)
  * [40] Sun, X., Shang, J., Liang, S., Wei, Y.: Compositional human pose regression. In: ICCV (2017)
  * [41] Tekin, B., Katircioglu, I., Salzmann, M., Lepetit, V., Fua, P.: Structured prediction of 3d human pose with deep neural networks. In: BMVC (2016)
  * [42] Trumble, M., Gilbert, A., Malleson, C., Hilton, A., Collomosse, J.: Total capture: 3d human pose estimation fusing video and inertial sensors. In: BMVC (2017)
  * [43] Trumble, M., Gilbert, A., Hilton, A., Collomosse, J.: Deep autoencoder for combined human pose estimation and body model upscaling. In: ECCV (2018)
  * [44] Wei, S.E., Ramakrishna, V., Kanade, T., Sheikh, Y.: Convolutional pose machines. In: CVPR (2016)
  * [45] Zhang, Z., Wang, C., Qiu, W., Qin, W., Zeng, W.: Adafuse: Adaptive multiview fusion for accurate human pose estimation in the wild. IJCV (2020)
  * [46] Zhao, Q., Zheng, C., Liu, M., Wang, P., Chen, C.: Poseformerv2: Exploring frequency domain for efficient and robust 3d human pose estimation. In: CVPR (2023)
  * [47] Zheng, C., Zhu, S., Mendieta, M., Yang, T., Chen, C., Ding, Z.: 3d human pose estimation with spatial and temporal transformers. In: ICCV (2021)
  * [48] Zhou, Y., Barnes, C., Lu, J., Yang, J., Li, H.: On the continuity of rotation representations in neural networks. In: CVPR (2019)