# PointVoxel: A Simple and Effective Pipeline for Multi-View Multi-Modal 3D
Human Pose Estimation

Zhiyu Pan  Zhicheng Zhong  Wenxuan Guo  Yifan Chen  Jianjiang Feng
Corresponding author Jie Zhou  Department of Automation, BNRist, Tsinghua
University, China
{pzy20, zhongzc18, gwx22<EMAIL_ADDRESS>
{jfeng<EMAIL_ADDRESS>
###### Abstract

Recently, several methods have been proposed to estimate 3D human pose from
multi-view images and achieved impressive performance on public datasets
collected in relatively easy scenarios. However, there are limited approaches
for extracting 3D human skeletons from multimodal inputs (e.g., RGB and
pointcloud) that can enhance the accuracy of predicting 3D poses in
challenging situations. We fill this gap by introducing a pipeline called
PointVoxel that fuses multi-view RGB and pointcloud inputs to obtain 3D human
poses. We demonstrate that volumetric representation is an effective
architecture for integrating these different modalities. Moreover, in order to
overcome the challenges of annotating 3D human pose labels in difficult
scenarios, we develop a synthetic dataset generator for pretraining and design
an unsupervised domain adaptation strategy so that we can obtain a well-
trained 3D human pose estimator without using any manual annotations. We
evaluate our approach on four datasets (two public datasets, one synthetic
dataset, and one challenging dataset named BasketBall collected by ourselves),
showing promising results. The code and dataset will be released soon.

##  1 Introduction

Human pose estimation is a fundamental task in computer vision and has been
widely applied in various fields, such as human-computer interaction, human-
robot interaction, human activity recognition, etc. Specifically, multi-view
image datasets [22, 25, 4, 50] allow more precise 3D human pose estimation
compared to single-view ones [1, 55, 24, 34], due to the ability of multiple
views to capture 3D information from epipolar geometry. As technology
advances, researchers [23, 48, 33, 54, 43] have achieved promising results on
these multi-view images datasets. However, practical scenarios are more
challenging than existing public datasets, with diverse human motions, severe
occlusions, and large scene.

Figure 1: The overview of training a 3D human pose estimation network without
using manual annotations. First, PointVoxel is pretrained on synthetic data
generated by SyncHuman, and then unsupervisedly trained to adapt to the target
dataset. The top row shows estimated human poses on a new dataset, BasketBall,
which is collected by four camera-LiDAR sensors (only one RGB view is shown).
The bottom row shows our synthetic dataset.

Combining different sensors such as LiDAR and RGB camera has the potential of
boosting the accuracy of 3D human pose estimation performance in multiple
situations. LiDAR sensor can obtain precise but quite sparse 3D measurements
over long range, while camera can capture images of high resolution but lacks
depth perception. Therefore, LiDAR-Camera capturing system can provide texture
and depth information simultaneously in a large covering range. It makes it
possible to capture complex human activities within a large scene such as
basketball games with a small number of sensors, beneficial for system setup
and reducing cost. Therefore, it is essential to consider integrating the
information from sensors of various types and views, and fusing them into a
cohesive representation which can significantly facilitate the accuracy of 3D
human pose estimation.

In this paper, we propose PointVoxel, a 3D human pose estimation pipeline
taking multi-view multi-modal inputs. Volumetric representation is an
effective and reasonable architecture to fuse RGB and pointcloud. Researchers
[23, 48, 43, 53] have shown the effectiveness of voxel-based method. Voxel-
based architecture can naturally model a space’s geometric characteristics.
Besides, it is straightforward to either map the pointcloud information or
back-project the 2D information into the 3D volumetric representation.
Nevertheless, voxel-based method is not satisfactory for detecting pedestrians
especially in large scene due to the large computational cost while defining
volumetric space on it. Some pointcloud-based methods [42, 30, 35] illustrates
that they can detect objects accurately with the assist of pointcloud. As to
multi-modal features fusion [19, 15], we refer to the work BEVFusion [35] and
unify different modalities into a volumetric space that reserve each
modality’s geometric characteristics. Hence, PointVoxel combines human
detection and voxel-based 3D human pose estimation in a top-down manner.

Manually annotating or capturing 3D poses for multiple individuals in large
scenes is challenging, expensive, and hard to ensure generalizability to new
scenarios. To achieve better results on challenging scenarios without pose
labels, we adopt the approach of pretraining on a synthetic dataset first and
then unsupervised domain adaption training on the target dataset. We develop a
synthetic dataset generator named SyncHuman for generating pretrained data. It
can generate a large amount of synthetic data with multi-modal information,
and we can change LiDARs’ and cameras’ properties in the scene to meet
practical needs. Furthermore, the actions of avatars are diverse, as they can
be obtained from existing action files or pose annotation files sourced from
public motion capture datasets like AMASS [38]. Accordingly, the synthetic
data has accurate 3D pose groundtruth for a group of people with complex
actions.

To overcome the disparity between synthetic data and real-world scenarios and
achieve unsupervised domain adaption training, we propose an efficient
strategy including entropy-selected pseudo 3D pose supervision, pseudo 2D pose
supervision, and human pose prior constraints. Benefiting from volumetric
representation, we can get the interpretable outputs’ form: 3D human joints’
heatmap. Heatmap represents a probability distribution, and we can calculate
the information entropy of each heatmap channel to get the confidence of
corresponding joint. It is an essential indicator for identifying the quality
of predicted poses. Experiments show that entropy value is inverse
proportional to the rationality of predicted poses. Hence, we can use the
entropy to filter out unreasonable poses and get the pseudo 3D pose labels
while unsupervised domain adaption training. Off-the-shelf 2D pose estimation
methods [52, 6, 47, 12] are so robust that we can use them to generate the
pseudo 2D pose labels. Moreover, we design an intuitive human prior loss for
guaranteeing the rationality of adapted predicted 3D poses. According to the
aforementioned schemes, we are able to construct a 3D human pose estimation
algorithm based on multi-modal and multi-view data without the need for
annotations on target dataset (Figure 2).

We summarize the contributions of this paper as follows:

  * •

We proposed a simple and effective pipeline named PointVoxel to fuse multi-
view multi-modal inputs and estimate multiple person 3D pose, and it achieves
promising results on different datasets with different settings.

  * •

We proposed a training strategy without using manual annotations that involves
pretraining on synthetic data generated by SyncHuman and unsupervised domain
adaption training on a target dataset.

##  2 Related Works

In this section, we conduct the literature review according to the two
contribution points we proposed.

###  2.1 3D Human Pose Estimation

Image-based. Basic 3D pose estimation method is in two stage that estimates
the 2D pose first and then lifts it into 3D space [9, 23, 48, 33, 51, 43]. As
to multi-person setting, some methods [9, 33] match pedestrians from different
views and then locate them through 2D pose similarity. But they are not robust
to inaccurate 2D pose results. Zhang et al. [54] directly utilizes the 2D
images as inputs and regresses the 3D pose by transformer architecture.
However, their training process is time-consuming. Some voxel-based methods
[48, 43] locate each person in a 3D volumetric space and estimate 3D poses.
Such voxel-based methods greatly improve the precision of 3D human pose
estimation. However, they are not suitable for the large scene because of
large computational cost when detecting people.

Figure 2: The detailed structure of PointVoxel in 3D human pose estimation and
its unsupervised training strategy. Dashed arrows indicate calculating
corresponding loss.  represents conditional flow which means the pose will be
the pseudo 3D label for the next epoch if required condition is met.

Pointcloud-based. In the beginning, there are some 3D pose estimation methods
based on single-view depth map [13, 20, 18, 39]. They treat the depth map as
2D information with depth value. On the other hand, a few of methods back-
project depth map into 3D space as dense 3D pointcloud and use Pointnet
network to deal with [57, 3]. Moon et al. [40] propose a single-person pose
estimation approach which treats depth map as pointcloud and fill it in
volumetric space. Bekhtaoui et al. [3] use Pointnet related approach to detect
and estimate 3D human pose. Recently, Li et al. [31] use sparse pointcloud
scanned from LiDAR to estimate single-person 3D human pose. However, sparse
pointcloud cannot provide enough information for accurate 3D pose estimation.
Multi-modal fusion is helpful for concise 3D human poses’ perceiving.

Multi-modal based. Many works have been introduced for RGBD human pose
estimation [2, 58, 19, 15]. Multi-modal information can not only help detect
the person but also guarantee the accuracy of 3D human pose. Lately, some
researchers [7, 31] introduce camera-LiDAR setting datasets for outdoors and
large scene. As to the multi-modal fusion methods, some approaches [58, 3, 19]
fuse modalities in point-level by attaching extracted 2D features to every 3D
point. And a few methods [41, 32] fuse modalities in feature-level strategy.
They do not unify the 2D and 3D information in a unified interpretable space,
and fuse them in an uninterpretable feature space instead.

Our approach utilizes multi-modal information for human detection and 3D human
pose estimation. Besides, we use volumetric representation representing real-
word space to unify the 2D and 3D information.

Table 1: Comparison of the synthetic datasets about human beings.

Synthetic Dataset | Scene setup | Labels
---|---|---
| Multi-
---
View
| Multi-
---
Person
RGB Image |  Depth Image |  | LiDAR
---
Pointcloud
| 2D
---
Pose
| 3D
---
Pose
CAPE [37] |  2717 |  2717 |  2717 |  2717 |  2717 |  2717 |  2713
SURREAL [49] |  2717 |  2717 |  2713 |  2713 |  2717 |  2713 |  2713
PSP [11] |  2717 |  2717 |  2713 |  2717 |  2717 |  2713 |  2717
CALAR [10] |  2713 |  2713 |  2713 |  2713 |  2713 |  2717 |  2717
Ours |  2713 |  2713 |  2713 |  2713 |  2713 |  2713 |  2713

###  2.2 Synthetic Dataset Generation

Manually annotating 3D human pose is very tough. Therefore, synthetic dataset
is beneficial and helpful for pretraining model. There are many synthetic
datasets for 3D human pose estimation [49, 37, 11]. However, these datasets
are just single RGB view with random background. Besides, they do not consider
the rational interaction between avatars and the background. Dosovitskiy et
al. [10] provides a large-scale synthetic dataset for autonomous driving.
However, it dose not provide the 3D human pose labels, and the actions of
characters are monotonous.

We propose our synthetic dataset generator SyncHuman which can produce data
with more richness and diversity (Table 1). Moreover, we will publish the
generating tool so that researchers can change settings according to their
needs via our provided APIs.

###  2.3 Unsupervised Domain Adaption Training

There are many works in unsupervised or domain adaption manner for single view
3D human pose estimation. Kocabas et al. [27] use multi-view geometry to
supervise single view’s predicting results. Several methods [46, 28] use
teacher-student framework to conduct domain adaption. And a portion of
approaches [8, 56] utilize optical-flow or depth as their inputs which are
least affected by domain shift compared with RGB. Bigalke et al. [5] add human
prior loss according to human anatomy. Kundu et al. [29] define the
uncertainty of predictions and control the uncertainty value while training.
For multi-view 3D human pose generation method, there are some works use non-
deeplearning methods [4, 44]. 3DPS [4] and basic triangulation get 3D poses
within large computing complexity or inaccuracy. Remelli et al. [44] proposed
an efficient DLT (direct linear transformation) which can get relatively
accurate 3D results quickly.

Our work uses different views 2D pose heatmaps and depths as inputs that are
little influenced when changing domain. Besides, we choose information entropy
as the network’s uncertainty index without any extra loss while training.
Entropy can be used to select reliable results as pseudo 3D labels. Also, we
add the human prior loss to ensure the reasonability of 3D poses.

##  3 Methodology

The proposed method is composed of two parts: 3D human pose estimation and
unsupervised domain adaption training (Figure 2). Section 3.1 introduces the
approach of the top-down manner of 3D human pose estimation. We introduce the
details about voxel-based multi-modal fusion. Section 3.2 illustrates how we
generate the synthetic dataset to support the training of multi-modal pose
estimation model, and how to combine entropy with the human prior loss to
realize domain shift.

###  3.1 PointVoxel

Similar to [48, 9, 43], we adopt the top-down manner to estimate the 3D pose.
In our case, we adopt PointPillars [30] to detect the 3D bounding box of each
person. After obtaining the bounding box of each person, we can get the 2D
bounding box in each view for 2D pose estimation, and extract their
pointcloud. At first, we can define a volumetric space centered at the
poincloud’s centroid whose size is consistent with detected bounding boxes’,
and discretize it into an $X\times Y\times Z$ resolution. So that we can fill
each voxel according to each pointcloud’s coordinate. In our case, we set
value $1$ to the voxel containing poincloud. Then we can get the pointcloud-
related feature ($F_{1}\times X\times Y\times Z$) via 3D convolution backbone
V2V-Net [40].

Thanks to the development of 2D pose estimation methods [12, 52], relatively
accurate corresponding 2D pose can be predicted without extra training. Hence,
we can get each view’s 2D pose heatmap via Gaussian blur. We apply the same
projecting way as [48] to calculate each voxel’s information, and apply
V2V-Net to extract the rgb-related features ($F_{2}\times X\times Y\times Z$)
in 3D space. Then, we concatenate the two modal features in the same 3D space
for modality fusion and get the final 3D human pose heatmap via a 3D
convolution head Fusion-Net. In order to mitigate quantization error, we adopt
Soft-argmax to calculate each joints’ 3D coordinate $J^{k}$ through 3D
heatmaps and minimize the $L_{1}$ loss with the ground truth $J_{*}^{k}$:

$\displaystyle\mathcal{L}_{\text{pose}}=\sum_{k=1}^{K}\left\|J^{k}-J_{*}^{k}\right\|_{1}$
(1)

where $K$ is the number of joints.

###  3.2 Unsupervised Domain Adaptation

Due to the lack of annotated multi-view multi-modal 3D human pose dataset, we
choose to generate a synthetic dataset to assist training. SyncHuman we
designed can create 3D dynamic scenes of multiple persons with accurate
labels. Besides, we develop an efficient unsupervised domain adaption means by
designing a loss function to transfer the pretrained model from synthetic
dataset to real-world dataset.

####  3.2.1 SyncHuman Dataset

We develop the synthetic system based on Unity Engine. Due to its flexibility
and productivity, we can formulate a scene with a specific size and background
as per our specifications. Concerning sensors’ aspect, we implement Unity’s
built-in camera to obtain RGB images, and extract the depth information from
the GPU’s depth buffer with custom shaders used in the rendering pipeline.
Furthermore, we can attain the colored pointcloud by sampling the depth and
RGB images. For the sake of imitating the scanning process of LiDAR, we need
to sample points along to a scanning function of time. Currently, we consider
Livox Mid-40 LiDAR only, since its price is sufficiently low for sports or
surveillance applications. We use the following function to simulate its
scanning process:

$\displaystyle r=\alpha\times\mathbb{\cos}(3.825\times(\theta_{0}+0.0017\times
n))$ (2)

where $n\in[0,t\times 1e5]$ and $t$ is in second. $\alpha$ is the scanning
radius in pixel, and $\theta_{0}$ is a random initial angle. This equation is
defined in polar coordinate. Final sampling points can be acquired by
transforming it from polar coordinates to Cartesian coordinates and
translating it to the center of the image space.

As for the avatars, we download various human 3D models from Adobe
Mixamo111https://mixamo.com. With a view to guarantee the diversity of the
generated actions, we can drive these avatars by either ready-made action
files or other public datasets’ keypoint annotations. Currently, we have
developed the driving APIs for COCO17, COCO19 [34] and SMPL [36] standard
keypoint annotated inputs. Regarding the groundtruth matter, we can obtain the
3D human pose from the avatars’ humanoid skeleton; 2D pose label can be
acquired via projecting 3D pose into 2D view. Additionally, we can fetch the
mesh vertex of each avatar and then compute the semantic segmentation label
for each pointcloud by considering the pose label meanwhile.

####  3.2.2 Unsupervised Domain Adaptation

Similar to the 2D projection supervision utilized by other unsupervised or
weakly supervised methods [27, 16], we directly employ off-the-shelf 2D human
pose estimation model to get the pseudo 2D pose label
$\widetilde{J}_{\text{2D}}$. $\mathcal{L}_{\text{2D}}$ is obtained by
calculating the $L_{2}$ norm between 3D pose projection results and pseudo 2D
label from each view.

$\displaystyle\mathcal{L}_{\text{2D}}=\sum_{v=1}^{V}\sum_{k=1}^{K}\left\|\mathcal{P}_{v}(J^{v,k})-\widetilde{J}_{\text{2D}}^{v,k}\right\|_{2}$
(3)

where $\mathcal{P}$ is the projection function. $V$ is the number of views.
$\mathcal{L}_{\text{2D}}$ is the fundamental loss for unsupervised training.
In order to attain pseudo 3D label, we choose to use information entropy as
one uncertainty index. The entropy of one keypoint prediction heatmap (a
spatial probability distribution) $h^{k}$ is defined as:

$\displaystyle\mathcal{H}(h^{k})=-\sum_{i}h_{i}^{k}\times\log h_{i}^{k}$ (4)

where $i$ represents the voxel index. The higher the entropy is, the more
uncertain the keypoint location is. Furthermore, it relates to the rationality
of the keypoint location as observed in our experiments (Section 4.5). To
measure a person’s uncertainty, we take the maximum entropy value of all
keypoints. Different from Kundu et al. [29] who train the uncertainty value,
we consider it as an inartificial index. Our experiments demonstrate that the
magnitude of entropy values can serve as an indicator of pose estimation
quality for a specific network. Therefore, an entropy threshold $\lambda$ can
be set to sort out the reasonable predicted 3D human poses. In other words, we
select the predicted pose $\widetilde{p}$ whose entropy value is less than
$\lambda$ as the pseudo 3D pose label for the next epoch’s training, and it
can be updated after each training epoch. Therefore, the 3D pseudo pose loss
can be obtained by:

$\displaystyle\mathcal{L}_{\text{3D}}=\sum_{k=1}^{K}\left\|J^{k}-\widetilde{J}^{k}\right\|_{1}$
(5)

where $\widetilde{J}^{k}$ is the 3D joint of the selected pseudo 3D pose label
$\widetilde{p}$.

To further ensure the anatomical plausibility of the pose, we introduce a
human prior loss $\mathcal{L}_{\text{prior}}$222Please refer to Suppl. for
details. adapted from Bigalke et al. [5]. Specifically, we formulate three
losses to penalize asymmetric limb lengths $\mathcal{L}_{\text{symm}}$,
implausible joint angles $\mathcal{L}_{\text{angle}}$, and implausible bone
lengths $\mathcal{L}_{\text{length}}$. To sum up, the final loss function is
defined as:

$\displaystyle\mathcal{L}_{\text{unsup}}=\omega_{\text{1}}\mathcal{L}_{\text{2D}}+\omega_{\text{2}}\mathbbm{1}(\widetilde{p}<\lambda)\mathcal{L}_{\text{3D}}+\omega_{\text{3}}\mathcal{L}_{\text{prior}}$
(6)

where $\omega_{\text{1}}$, $\omega_{\text{2}}$ and $\omega_{\text{3}}$ are the
weights of each loss.

Figure 3: The results are from different inputs’ modalities training on
BasketBall. The first row is the 2D pose and pointcloud for model inputs, the
second row’s results are from the model trained by 2D pose heatmap (h.m.)
inputs, and the third row’s results are from the model trained by 2D pose
heatmap and pointcloud (h.m.+p.c.) inputs. The pose results shown on image are
from projected 3D ones except for the first row. Both models utilize the
complete unsupervised training strategy. Figure 4: The results show the
ablation study about unsupervised training losses on BasketBall. Baseline is
only pseudo 2D pose supervision. “Entropy” means adding entropy-selected
pseudo 3D pose supervision. “Prior” means adding human prior loss.

##  4 Experiments

In this section, we introduce all datasets and evaluation metrics used in our
experiments. We analyze the performance of our PointVoxel pipeline in
supervised and unsupervised manner on different datasets. Besides, we also
conduct ablation studies to verify the effectiveness of each strategy or loss
in unsupervised domain adaption.

###  4.1 Implementation details.

We use V2V-Net [40] as the pointcloud and heatmap voxel’s feature extractor.
The detailed V2Vnet design is the same with PRN [48]. The resolution of the
voxel grid is $64\times 64\times 64$ defined in a $2\text{m}\times
2\text{m}\times 2\text{m}$ space. We set $\omega_{\text{1}}=0.02$,
$\omega_{\text{2}}=1$, $\omega_{\text{3}}=10$, and $\lambda=6$. We train the
voxel-based pose estimation network on an NVIDIA GeForce RTX-3090 (24G) with a
batch size of eight. The learning rate is set to 0.001 and the optimizer is
Adam [26].

###  4.2 Datasets and Metrics.

We use the following datasets in our experiments:
I. CMU Panoptic Studio [25]. It is indoors with a valid scene range around
$5\text{m}\times 5\text{m}$. In order to realize the multi-modal inputs, we
select several subsets333“160422_ultimatum1”, “160224_haggling1”,
“160226_haggling1”, “161202_haggling1”, “160906_ian1”, “160906_ian2”,
“160906_ian3”, and “160906_band1” for training; “160906_pizza1”,
“160422_haggling1”, “160906_ian5”, “160906_band2” for testing. that were
evaluated in [33] and have depth information. We unify the depth information
from Kinect 1 to 5 and process these depth maps by equation 2 to get the
sparce pointcloud which is similar to Mid-40 Livox LiDAR’s scanning. About the
2D pose estimation results, we use the results provided by [33] which are
claimed to be predicted from HRNet [47].
II. MVOR [45]. It is indoors set in operating room. In this dataset, we adopt
three sampling functions (equation 2) centered at three trisection points of
image’s width to imitate a Mid-100 Livox LiDAR’s scanning. And we use the 2D
pose estimation results provided by [45] which are predicted from AlphaPose
[12]. There is a semantic keypoints’ definition gap between MVOR (a ten
keypoints standard) and ours (COCO17 standard). Therefore, we only conduct the
qualitative analysis on MVOR. We use the data with annotation from “day1,
day2, day3” for training and “day4” for testing.
III. BasketBall. It is outdoors with a valid scene range around
$35\text{m}\times 17\text{m}$ recording a basketball match. We collected this
real-world dataset with four-view multi-modal information recording a
basketball match. The pointcloud is collected by four Mid-100 Livox LiDARs.
The 2D pose estimation results are predicted by VitPose [52]. The dataset
contains two segments, each containing of two thousand frames. It has manually
marked the detection and tracking groundtruth for each frame, while lacks of
the 3D keypoints groundtruth. Therefore, it is mainly used for qualitative
analysis.
IV. BasketBallSync. It is a synthetic dataset generated by SyncHuman with the
same sensors’ configuration and setting as BasketBall. The pointcloud is
obtained by simulating the scan pattern of Mid-100 Livox LiDARs. It contains
ten avatars acting randomly with the first eight for training and the rest two
for testing among all frames (3336 frames in 10Hz). The predicted 2D human
poses are extracted by VitPose [52].
Evaluation metrics. For evaluating the 3D human pose, we calculate the
standard mean per-joint position error before and after Procrustes Alignment
[17] as MPJPE and PA-MPJPE in millimeter respectively. And we compare to the
methods with single-frame data input like ours for fairness.

Table 2: Comparison of different 3D pose estimation methods on Panoptic and
BasketBallSync. As to unsupervised manner, we pretrained PointVoxel on
BasketBallSync in Panoptic testing and on Panoptic in BasketballSync testing.

Methods | Supervision | Panoptic | BasketBallSync
---|---|---|---
MPJPE | PA-MPJPE | MPJPE | PA-MPJPE
MvP [54] | sup. | 25.78 | 25.02 | 291.11 | 240.32
PlanePose [33] | sup. | 18.54 | 11.92 | 119.55 | 65.80
VoxelPose(PRN) [48] | sup. | 14.91 | 11.88 | 40.80 | 34.34
Ours | sup. | 14.44 | 11.61 | 31.84 | 27.36
DLT [44] | no sup. | 38.26 | 44.77 | 101.49 | 62.77
Ours | no sup. | 22.00 | 15.96 | 72.92 | 62.72

Table 3: Comparison of different input modalities for PointVoxel on Panoptc
and BasketBallSync. Model tested on BasketBallSync is pretrained on Panoptic,
and is pretrained on BasketBallSync for Panoptic testing.

Input modality | supervision | Panoptic | BasketBallSync
---|---|---|---
MPJPE | PA-MPJPE | MPJPE | PA-MPJPE
p.c. | sup. | 164.30 | 148.78 | 136.01 | 123.22
h.m. | sup. | 14.91 | 11.88 | 40.80 | 34.34
h.m.+p.c. | sup. | 14.44 | 11.61 | 31.84 | 27.36
h.m. | no sup. | 25.72 | 17.24 | 263.05 | 183.85
h.m.+p.c. | no sup. | 22.00 | 15.96 | 72.92 | 62.72

Figure 5: Qualitative results of 3D human pose estimation on MVOR from three samples. Blue lines represent the predictions and green lines represent the groundtruth. The first to third columns are the projected 2D results on three views and the fourth one is the 3D pose result. Table 4: Comparison of different scene setups. We simulated four different scene setups by SyncHuman generator. This table shows the performance of PointVoxel by pretraining on different scenes and directly applying to Panoptic. “others.” means corresponding attribute is a setting that is different from Panoptic or BasketBall. “Pan.” means corresponding attribute is the same with Panoptic. “Bas.” means corresponding attribute is same with BasketBall. Scene’s Range | Sensors’ Setting | Metric
---|---|---
Number | Position | Orientation | MPJPE | PA-MPJPE
Bas. | 4 | Bas. | Bas. | 82.61 | 93.41
Bas. | 5 | others | Pan. | 77.90 | 60.42
Pan. | 5 | others | Pan. | 26.45 | 22.56
Pan. | 5 | Pan. | Pan. | 23.15 | 17.63
Table 5: Comparison of different unsupervised training strategies. The model
is pretrained on BasketBallSync for Panoptic testing, and on Panoptic for
BasketBallSync testing. “Entropy” means adding entropy selected pseudo 3D pose
supervision, and “Prior” means adding human prior loss.

Unsup. loss | Panoptic | BasketBallSync
---|---|---
Entropy | Prior | MPJPE | PA-MPJPE | MPJPE | PA-MPJPE
2717 |  2717 | 29.14 | 16.48 | 84.49 | 75.55
2713 |  2717 | 28.26 | 25.85 | 77.38 | 68.84
2717 |  2713 | 22.88 | 24.69 | 76.45 | 65.54
2713 |  2713 | 22.00 | 15.96 | 72.92 | 62.72

###  4.3 3D Pose Estimation Analysis

In this section, we evaluate the 3D human pose estimation results of different
approaches. For PlaneSweepPose [33], VoxelPose [48] and PointVoxel in
supervised manner, we utilize the groundtruth location of each person into the
process of training and testing. As to MvP [54], it is difficult to add the
groundtruth detection information into the network due to its architecture
design, which also means that it cannot utilize the depth or pointcloud
informaion directly. For fairness, we just evaluate the MPJPE@500 metrics
(only considering per joint error less than 500 mm) for the matched person as
to MvP.

We evaluate these approaches on Panoptic Studio and BasketballSync as small
scene and large scene respectively. Table 2 shows that our approach
outperforms them in terms of MPJPE and PA-MPJPE especially in large scene’s
setting. It is worth noting that our approach with unsupervised learning
manner can even outperform supervised RGB inputs multi-view approaches.

We assess our pipeline on MVOR, and our method in unsupervised manner achieved
great performance which even outperforms groundtruth in some frames (Figure
5).

Additionally, we analyze the performance with different modals inputs
(pointcloud, RGB, and both of them) in volumetric architecture. Table 3
illustrates that the Livox pointcloud information alone is unable to
accurately extract 3D human skeleton due to its sparsity, and cannot be
adopted in unsupervised manner without 2D pseudo labels from heatmaps.
However, when combined with RGB information, the performance is greatly
enhanced. Figure 3 demonstrates that multi-modal inputs increase the model’s
tolerance to 2D pose estimation error, including severe situations like miss-
prediction or predicting on a wrong person, during unsupervised domain
adaption training.

###  4.4 Unsupervised Domain Adaption

In this section, we conduct an in-depth analysis of the settings of synthetic
datasets and ablation studies of unsupervised training strategy.

Figure 6: The entropy distributions of reasonable and unreasonable predicted
3D poses.

####  4.4.1 Synthetic Setting

Tu et al. [48] have conducted an experiment by training VoxelPose with target
dataset’s camera configuration and others’ human pose, then directly applying
the model to target dataset for evaluation. In this section, we further
analyze the impact of scene settings in volumetric architecture. We simulated
four different scene setups by SyncHuman generator which changes the sensors’
position and orientation. Table 4 demonstrates that the more similar scene
setups are, the better the model performs. It also suggests that in practical
applications, setting the synthetic data to have the same sensor arrangement
and scene range as the target dataset can improve the performance of
unsupervised domain adaption or direct inferring on the new scenario.

####  4.4.2 Ablation Study on Unsupervised Training Losses

For unsupervised domain adaption, pseudo 2D pose supervision is necessary, and
it is the baseline. Besides, we adopt an interpretable human prior loss and a
pseudo 3D pose (selected by entropy value) loss to assist the learning.
Therefore, we conduct an ablation study to analyze the impact of these two
losses. From Table 5, entropy selected pseudo 3D pose loss improves the
performance, but it does not limit the rationality which means it may generate
low-entropy invalid 3D poses. Prior loss controls the results in reasonable
action range. Therefore, we choose both losses as our efficient training
strategy. Besides, these losses can enhance the robustness to 2D pose
estimation errors (Figure 4).

###  4.5 Entropy Analysis

In order to verify the relationship between entropy value and pose
rationality, we labeled some 3D poses predicted by PointVoxel as reasonable or
unreasonable. Figure 6 shows the distribution of entropy value for reasonable
and unreasonable poses. Poses can be divided naturally via an entropy
threshold. Therefore, we can judge one model’s performance on one unlabeled
dataset by its average entropy value. It is noteworthy that it cannot reduce
the entropy value for a model without proper pretraining after unsupervised
learning. Hence, entropy-based judgement is more applicable before
unsupervised adaption.

##  5 Limitation

The PointVoxel pipeline performs well in both supervised or unsupervised
manner. However, it still has some problems to be addressed. 1) The method
does not consider temporal information. 2) The unsupervised adaption
performance on single view dataset is not good. 3) It suffers from the view-
variant effect especially after unsupervised training. 4) The synthetic
dataset currently does not set complex background, and the real basketball
dataset is relatively limited in size.

##  6 Conclusion

In this paper, we propose a pipeline called PointVoxel for 3D human pose
estimation using multi-view multi-modal data. To train the model in
unsupervised manner, we develop a synthetic generator SyncHuman and design an
efficient unsupervised loss for domain adaption learning. We conduct extensive
experiments on both synthetic and real-world datasets. We analyze the powerful
learning ability of this pipeline for 3D human pose estimation. Besides, we
study the impact of scene settings and unsupervised training losses.
Furthermore, we investigate the inherent relationship between entropy value
and pose plausibility. We believe that our approach can be a good baseline for
large scene’s multi-view multi-modal 3D human pose estimation.

## References

  * Andriluka et al. [2014] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele.  2D human pose estimation: New benchmark and state of the art analysis.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3686–3693, 2014.
  * Bashirov et al. [2021] Renat Bashirov, Anastasia Ianina, Karim Iskakov, Yevgeniy Kononenko, Valeriya Strizhkova, Victor Lempitsky, and Alexander Vakhitov.  Real-time rgbd-based extended body pose estimation.  In _Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision_ , pages 2807–2816, 2021.
  * Bekhtaoui et al. [2020] Walid Bekhtaoui, Ruhan Sa, Brian Teixeira, Vivek Singh, Klaus Kirchberg, Yao-jen Chang, and Ankur Kapoor.  View invariant human body detection and pose estimation from multiple depth sensors.  _arXiv preprint arXiv:2005.04258_ , 2020.
  * Belagiannis et al. [2014] Vasileios Belagiannis, Sikandar Amin, Mykhaylo Andriluka, Bernt Schiele, Nassir Navab, and Slobodan Ilic.  3D pictorial structures for multiple human pose estimation.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 1669–1676, 2014.
  * Bigalke et al. [2022] Alexander Bigalke, Lasse Hansen, Jasper Diesel, and Mattias P Heinrich.  Domain adaptation through anatomical constraints for 3D human pose estimation under the cover.  In _International Conference on Medical Imaging with Deep Learning_ , pages 173–187, 2022.
  * Cao et al. [2017] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime multi-person 2D pose estimation using part affinity fields.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 7291–7299, 2017.
  * Cong et al. [2022] Peishan Cong, Yiteng Xu, Yiming Ren, Juze Zhang, Lan Xu, Jingya Wang, Jingyi Yu, and Yuexin Ma.  Weakly supervised 3D multi-person pose estimation for large-scale scenes based on monocular camera and single LiDAR.  _arXiv preprint arXiv:2211.16951_ , 2022.
  * Doersch and Zisserman [2019] Carl Doersch and Andrew Zisserman.  Sim2real transfer learning for 3D human pose estimation: motion to the rescue.  _Advances in Neural Information Processing Systems_ , 32, 2019.
  * Dong et al. [2019] Junting Dong, Wen Jiang, Qixing Huang, Hujun Bao, and Xiaowei Zhou.  Fast and robust multi-person 3D pose estimation from multiple views.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 7792–7801, 2019.
  * Dosovitskiy et al. [2017] Alexey Dosovitskiy, German Ros, Felipe Codevilla, Antonio Lopez, and Vladlen Koltun.  CARLA: An open urban driving simulator.  In _Conference on Robot Learning_ , pages 1–16, 2017.
  * Ebadi et al. [2021] Salehe Erfanian Ebadi, You-Cyuan Jhang, Alex Zook, Saurav Dhakad, Adam Crespi, Pete Parisi, Steven Borkman, Jonathan Hogins, and Sujoy Ganguly.  PeopleSansPeople: a synthetic data generator for human-centric computer vision.  _arXiv preprint arXiv:2112.09290_ , 2021.
  * Fang et al. [2022] Hao-Shu Fang, Jiefeng Li, Hongyang Tang, Chao Xu, Haoyi Zhu, Yuliang Xiu, Yong-Lu Li, and Cewu Lu.  Alphapose: Whole-body regional multi-person pose estimation and tracking in real-time.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 2022.
  * Garau et al. [2021] Nicola Garau, Niccolo Bisagno, Piotr Bródka, and Nicola Conci.  DECA: Deep viewpoint-equivariant human pose estimation using capsule autoencoders.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 11677–11686, 2021.
  * Geiger et al. [2012] Andreas Geiger, Philip Lenz, and Raquel Urtasun.  Are we ready for autonomous driving? the KITTI vision benchmark suite.  In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , pages 3354–3361, 2012.
  * Gerats et al. [2022] Beerend GA Gerats, Jelmer M Wolterink, and Ivo AMJ Broeders.  3D human pose estimation in multi-view operating room videos using differentiable camera projections.  _Computer Methods in Biomechanics and Biomedical Engineering: Imaging & Visualization_, pages 1–9, 2022.
  * Gholami et al. [2022] Mohsen Gholami, Ahmad Rezaei, Helge Rhodin, Rabab Ward, and Z Jane Wang.  Self-supervised 3D human pose estimation from video.  _Neurocomputing_ , 488:97–106, 2022.
  * Gower [1975] John C Gower.  Generalized procrustes analysis.  _Psychometrika_ , 40:33–51, 1975.
  * Guo et al. [2017] Hengkai Guo, Guijin Wang, Xinghao Chen, and Cairong Zhang.  Towards good practices for deep 3D hand pose estimation.  _arXiv preprint arXiv:1707.07248_ , 2017.
  * Hansen et al. [2019] Lasse Hansen, Marlin Siebert, Jasper Diesel, and Mattias P Heinrich.  Fusing information from multiple 2D depth cameras for 3D human pose estimation in the operating room.  _International Journal of Computer Assisted Radiology and Surgery_ , 14:1871–1879, 2019.
  * Haque et al. [2016] Albert Haque, Boya Peng, Zelun Luo, Alexandre Alahi, Serena Yeung, and Li Fei-Fei.  Towards viewpoint invariant 3D human pose estimation.  In _Proceedings of the European Conference on Computer Vision_ , pages 160–177, 2016.
  * Hou et al. [2020] Yunzhong Hou, Liang Zheng, and Stephen Gould.  Multiview detection with feature perspective transformation.  In _Proceedings of the European Conference on Computer Vision_ , pages 1–18, 2020.
  * Ionescu et al. [2013] Catalin Ionescu, Dragos Papava, Vlad Olaru, and Cristian Sminchisescu.  Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 36(7):1325–1339, 2013.
  * Iskakov et al. [2019] Karim Iskakov, Egor Burkov, Victor Lempitsky, and Yury Malkov.  Learnable triangulation of human pose.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 7718–7727, 2019.
  * Jin et al. [2020] Sheng Jin, Lumin Xu, Jin Xu, Can Wang, Wentao Liu, Chen Qian, Wanli Ouyang, and Ping Luo.  Whole-body human pose estimation in the wild.  In _Proceedings of the European Conference on Computer Vision_ , pages 196–214, 2020.
  * Joo et al. [2015] Hanbyul Joo, Hao Liu, Lei Tan, Lin Gui, Bart Nabbe, Iain Matthews, Takeo Kanade, Shohei Nobuhara, and Yaser Sheikh.  Panoptic studio: A massively multiview system for social motion capture.  In _Proceedings of the IEEE International Conference on Computer Vision_ , pages 3334–3342, 2015.
  * Kingma and Ba [2014] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _arXiv preprint arXiv:1412.6980_ , 2014.
  * Kocabas et al. [2019] Muhammed Kocabas, Salih Karagoz, and Emre Akbas.  Self-supervised learning of 3D human pose using multi-view geometry.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1077–1086, 2019.
  * Kundu et al. [2020] Jogendra Nath Kundu, Ambareesh Revanur, Govind Vitthal Waghmare, Rahul Mysore Venkatesh, and R Venkatesh Babu.  Unsupervised cross-modal alignment for multi-person 3D pose estimation.  In _Proceedings of the European Conference on Computer Vision_ , pages 35–52, 2020.
  * Kundu et al. [2022] Jogendra Nath Kundu, Siddharth Seth, Pradyumna YM, Varun Jampani, Anirban Chakraborty, and R Venkatesh Babu.  Uncertainty-aware adaptation for self-supervised 3D human pose estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 20448–20459, 2022.
  * Lang et al. [2019] Alex H Lang, Sourabh Vora, Holger Caesar, Lubing Zhou, Jiong Yang, and Oscar Beijbom.  PointPillars: Fast encoders for object detection from point clouds.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12697–12705, 2019.
  * Li et al. [2022] Jialian Li, Jingyi Zhang, Zhiyong Wang, Siqi Shen, Chenglu Wen, Yuexin Ma, Lan Xu, Jingyi Yu, and Cheng Wang.  Lidarcap: Long-range marker-less 3d human motion capture with lidar point clouds.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 20502–20512, 2022.
  * Liang et al. [2018] Ming Liang, Bin Yang, Shenlong Wang, and Raquel Urtasun.  Deep continuous fusion for multi-sensor 3D object detection.  In _Proceedings of the European Conference on Computer Vision_ , pages 641–656, 2018.
  * Lin and Lee [2021] Jiahao Lin and Gim Hee Lee.  Multi-view multi-person 3D pose estimation with Plane Sweep Stereo.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11886–11895, 2021.
  * Lin et al. [2014] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft COCO: Common objects in context.  In _Proceedings of the European Conference on Computer Vision_ , pages 740–755, 2014.
  * Liu et al. [2022] Zhijian Liu, Haotian Tang, Alexander Amini, Xinyu Yang, Huizi Mao, Daniela Rus, and Song Han.  BEVFusion: Multi-task multi-sensor fusion with unified bird’s-eye view representation.  _arXiv preprint arXiv:2205.13542_ , 2022.
  * Loper et al. [2015] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J Black.  SMPL: A skinned multi-person linear model.  _ACM Transactions on Graphics_ , 34(6):1–16, 2015\.
  * Ma et al. [2020] Qianli Ma, Jinlong Yang, Anurag Ranjan, Sergi Pujades, Gerard Pons-Moll, Siyu Tang, and Michael J Black.  Learning to dress 3D people in generative clothing.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6469–6478, 2020.
  * Mahmood et al. [2019] Naureen Mahmood, Nima Ghorbani, Nikolaus F Troje, Gerard Pons-Moll, and Michael J Black.  AMASS: Archive of motion capture as surface shapes.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 5442–5451, 2019.
  * Martínez-González et al. [2020] Angel Martínez-González, Michael Villamizar, Olivier Canévet, and Jean-Marc Odobez.  Residual pose: A decoupled approach for depth-based 3D human pose estimation.  In _IEEE/RSJ International Conference on Intelligent Robots and Systems_ , pages 10313–10318, 2020.
  * Moon et al. [2018] Gyeongsik Moon, Ju Yong Chang, and Kyoung Mu Lee.  V2V-Posenet: Voxel-to-voxel prediction network for accurate 3D hand and human pose estimation from a single depth map.  In _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , pages 5079–5088, 2018.
  * Piergiovanni et al. [2021] AJ Piergiovanni, Vincent Casser, Michael S Ryoo, and Anelia Angelova.  4D-net for learned multi-modal alignment.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 15435–15445, 2021.
  * Qi et al. [2019] Charles R Qi, Or Litany, Kaiming He, and Leonidas J Guibas.  Deep hough voting for 3D object detection in point clouds.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 9277–9286, 2019.
  * Reddy et al. [2021] N Dinesh Reddy, Laurent Guigues, Leonid Pishchulin, Jayan Eledath, and Srinivasa G Narasimhan.  Tessetrack: End-to-end learnable multi-person articulated 3D pose tracking.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 15190–15200, 2021.
  * Remelli et al. [2020] Edoardo Remelli, Shangchen Han, Sina Honari, Pascal Fua, and Robert Wang.  Lightweight multi-view 3D pose estimation through camera-disentangled representation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 6040–6049, 2020.
  * Srivastav et al. [2018] Vinkle Srivastav, Thibaut Issenhuth, Abdolrahim Kadkhodamohammadi, Michel de Mathelin, Afshin Gangi, and Nicolas Padoy.  Mvor: A multi-view rgb-d operating room dataset for 2D and 3D human pose estimation.  _arXiv preprint arXiv:1808.08180_ , 2018.
  * Srivastav et al. [2020] Vinkle Srivastav, Afshin Gangi, and Nicolas Padoy.  Self-supervision on unlabelled or data for multi-person 2D/3D human pose estimation.  In _Medical Image Computing and Computer Assisted Intervention_ , pages 761–771, 2020.
  * Sun et al. [2019] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang.  Deep high-resolution representation learning for human pose estimation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 5693–5703, 2019.
  * Tu et al. [2020] Hanyue Tu, Chunyu Wang, and Wenjun Zeng.  VoxelPose: Towards multi-camera 3D human pose estimation in wild environment.  In _Proceedings of the European Conference on Computer Vision_ , pages 197–212, 2020.
  * Varol et al. [2017] Gul Varol, Javier Romero, Xavier Martin, Naureen Mahmood, Michael J Black, Ivan Laptev, and Cordelia Schmid.  Learning from synthetic humans.  In _Proceedings of the IEEE Conference on computer vision and pattern recognition_ , pages 109–117, 2017.
  * Von Marcard et al. [2018] Timo Von Marcard, Roberto Henschel, Michael J Black, Bodo Rosenhahn, and Gerard Pons-Moll.  Recovering accurate 3D human pose in the wild using imus and a moving camera.  In _Proceedings of the European Conference on Computer Vision_ , pages 601–617, 2018.
  * Wu et al. [2021] Size Wu, Sheng Jin, Wentao Liu, Lei Bai, Chen Qian, Dong Liu, and Wanli Ouyang.  Graph-based 3D multi-person pose estimation using multi-view images.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , pages 11148–11157, 2021.
  * Xu et al. [2022] Yufei Xu, Jing Zhang, Qiming Zhang, and Dacheng Tao.  Vitpose: Simple vision transformer baselines for human pose estimation.  _arXiv preprint arXiv:2204.12484_ , 2022.
  * Ye et al. [2022] Hang Ye, Wentao Zhu, Chunyu Wang, Rujie Wu, and Yizhou Wang.  Faster VoxelPose: Real-time 3d human pose estimation by orthographic projection.  In _Proceedings of the European Conference on Computer Vision_ , pages 142–159, 2022.
  * Zhang et al. [2021a] Jianfeng Zhang, Yujun Cai, Shuicheng Yan, Jiashi Feng, et al.  Direct multi-view multi-person 3D pose estimation.  _Advances in Neural Information Processing Systems_ , 34:13153–13164, 2021a.
  * Zhang et al. [2019a] Song-Hai Zhang, Ruilong Li, Xin Dong, Paul Rosin, Zixi Cai, Xi Han, Dingcheng Yang, Haozhi Huang, and Shi-Min Hu.  Pose2seg: Detection free human instance segmentation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 889–898, 2019a.
  * Zhang et al. [2019b] Xiheng Zhang, Yongkang Wong, Mohan S Kankanhalli, and Weidong Geng.  Unsupervised domain adaptation for 3D human pose estimation.  In _Proceedings of the 27th ACM International Conference on Multimedia_ , pages 926–934, 2019b.
  * Zhang et al. [2021b] Zihao Zhang, Lei Hu, Xiaoming Deng, and Shihong Xia.  Sequential 3D human pose estimation using adaptive point cloud sampling strategy.  In _International Joint Conferences on Artificial Intelligence Organization_ , pages 1330–1337, 2021b.
  * Zheng et al. [2022] Jingxiao Zheng, Xinwei Shi, Alexander Gorban, Junhua Mao, Yang Song, Charles R Qi, Ting Liu, Visesh Chari, Andre Cornman, Yin Zhou, et al.  Multi-modal 3D human pose estimation with 2D weak supervision in autonomous driving.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 4478–4487, 2022.

Supplementary Material

##  7 Different Scanning Patterns of Pointcloud

There are different ways to get or scan the pointcloud: 1) randomly sampling
the depth map; 2) sampling the depth map with multiple equidistant horizontal
lines to imitate Velodyne LiDARs; 3) sampling the depth map with the “Rose
curve” sampling equation mentioned in our paper to imitate Livox LiDARs.
Figure 7 shows that the “Rose curve” sampling equation provides minimal
information due to its localized concentrated scan. However, Livox LiDARs are
more affordable than Velodyne LiDARs, and they have been used in many
applications including surveillance. Besides, our BaseketBall data is captured
by Livox LiDARs. Therefore, we use the Livox scanning pattern to simulate
scanning pointcloud in our experiments.

##  8 Datasets: BaseketBall and SyncHuman

###  8.1 BaseketBall

BasketBall is an outdoor dataset capturing a Basketball match with four sensor
nodes (each node includes one Livox LiDAR and one RGB camera) in a convergent
acquisition. The dataset is challenging due to large covering, occlusions, and
players’ dynamic motions (Figure 8). Currently, we have developed an
annotation tool to annotate the players’ 3D bounding boxes and IDs. In the
future, we will integrate the 3D human keypoints annotation into the tool with
the assist of PointVoxel.

###  8.2 SyncHuman

We can use our synthetic data generator SyncHuman to simulate any arrangement
of sensors to observe a scene. As illustrated in Experiments of our paper,
using the same scene setting for both training and test yields better
transferring performance. Figure 8 shows the datasets we generated,
PanopticSync and BaseketBallSync, compared with the original datasets.

Figure 7: Different scanning patterns of pointcloud. All the samples shown in
this figure are from the same scene at the same time, and contain the same
number of points. Figure 8: Real datasets and the synthetic datasets generated
by SyncHuman.

##  9 Human Prior Loss

We use the human prior loss similar to [5] to encourage the network to
generate human-like 3D keypoints. The human prior loss is defined as three
parts: 1) the predicting bone length should be in a reasonable range; 2) the
predicting length of symmetric bones should be similar; 3) the predicting bone
angles should be reasonable according to the kinematic of human.

Different from Bigalke et al. [5] who distribute each joint with a specific
length range, we set one range for all bones. In our case, we set
$l_{\text{min}}=0.05\text{m}$ and $l_{\text{max}}=0.7\text{m}$. So the
$\mathcal{L}_{\text{length}}$ can be defined as:

$\displaystyle\mathcal{L}_{\text{length}}=\sum_{b=1}^{N}\mathcal{C}(B_{i}-l_{\text{max}},0)+\mathcal{C}(l_{\text{min}}-B_{i},0)$
(7)

where $\mathcal{C}()$ is the clipping function that clip the value greater
than $0$, $N$ is the number of bones. As to the symmetric bones, we set the
symmetric bones as a pair, and set $L_{\text{2}}$ loss among them. So the
$\mathcal{L}_{\text{symm}}$ can be defined as:

$\displaystyle\mathcal{L}_{\text{symm}}=\sum_{b=1}^{N}\|B_{i}-B_{\text{symm}(i)}\|_{2}$
(8)

Figure 9: Definition of $\mathcal{L}_{\text{angle}}$.

where $B_{\text{symm}}$ is the symmetric bone of $B_{i}$. As to angle loss, we
limit the nose-neck-midhip angle and hip-knee-ankle angle specifically to let
nose be in front of the body and legs be bent forward. Figure 9 shows the
definition of each joint and vectors. Specially, we do not directly calculate
the angle of the bones, but calculate the dot product of corresponding
vectors. First, we calculate the forward direction vector
$\vec{d}_{\text{forward}}$ of the body, which is the cross product of the unit
vector from neck to midhip $\overrightarrow{J_{\text{0}}J_{\text{2}}}$ and
unit vector from neck to left shoulder
$\overrightarrow{J_{\text{0}}J_{\text{3}}}$:

$\centering\vec{d}_{\text{forward}}=\overrightarrow{J_{\text{0}}J_{\text{2}}}\times\overrightarrow{J_{\text{0}}J_{\text{3}}}\@add@centering$
(9)

Then, as to the nose-neck-midhip angle, we calculate the unit vector from neck
to nose $\overrightarrow{J_{\text{1}}J_{\text{0}}}$ denoted by
$\vec{d}_{\text{nose}}$, and we calculate the dot product of
$\vec{d}_{\text{nose}}$ and $\vec{d}_{\text{forward}}$ and get the head angle
loss:

$\displaystyle\mathcal{L}_{\text{head\\_ang}}=\mathcal{C}(\vec{d}_{\text{forward}}\cdot\vec{d}_{\text{nose}},0,1)$
(10)

where $\mathcal{C}()$ is the clipping function that clip the value into $0$ to
$1$. As to the hip-knee-ankle angle, we need to get the midpoint of the hip
and ankle denoted by $c_{\text{l}}$ and $c_{\text{r}}$ for left leg and right
leg respectively. Then, we calculate the unit vectors from knee point to the
leg’s midpoint as $\vec{d}_{\text{l\\_leg}}$ and $\vec{d}_{\text{r\\_leg}}$.
Therefore, we get the leg angle loss:

$\displaystyle\mathcal{L}_{\text{leg\\_ang}}=\mathcal{C}(\vec{d}_{\text{forward}}\cdot\vec{d}_{\text{l\\_leg}},0,1)+\mathcal{C}(\vec{d}_{\text{forward}}\cdot\vec{d}_{\text{r\\_leg}},0,1)$
(11)

where $\mathcal{C}()$ is the clipping function that clip the value into $0$ to
$1$. Therefore, we can calculate the angle loss:

$\displaystyle\mathcal{L}_{\text{angle}}=\mathcal{L}_{\text{head\\_ang}}+\mathcal{L}_{\text{leg\\_ang}}$
(12)

Finally, we combine the three losses together as the human prior loss:

$\displaystyle\mathcal{L}_{\text{prior}}=\gamma_{\text{1}}\mathcal{L}_{\text{length}}+\gamma_{\text{2}}\mathcal{L}_{\text{symm}}+\gamma_{\text{3}}\mathcal{L}_{\text{angle}}$
(13)

where $\gamma_{\text{1}}$, $\gamma_{\text{2}}$ and $\gamma_{\text{3}}$ are the
weights of each loss. In our case, we set all the weights as $1$.

##  10 Extended Experiments

In this section, we conduct the experiments to verify the advantages of
pointcloud input in pedestrian detection. Furthermore, we show more examples
to explain the relationship between entropy value and pose rationality.

###  10.1 Human Detection

As for evaluating human detection, we assess the performance using the
established average precision (AP) metric as described in KITTI [14]. And we
regard detections as true positives to overlap by more than 70%
($\text{AP}_{\text{70}}$) or 50% ($\text{AP}_{\text{50}}$).

In the application of our current experiment, we adopt PointPillars [30] to
detect the human bounding box. And we utilize the VoxelPose’s CPN [48] and
MVDet [21] which is more suitable in large scene’s application as for
comparison in terms of multi-view RGB-based methods. In CMU panoptic studio
setup, VoxelPose [48] can get relatively accurate center location. However, it
set the size of bounding box to be a constant value (We set 0.8m $\times$ 0.8m
$\times$ 1.9m in our case for tighting results instead of 2m $\times$ 2m
$\times$ 2m in [48]) which affects the performance of detection. In
BasketBall, we adopt MVDet to detect human beings. Table 6 shows that
pointcloud-based method outperforms the multi-view RGB-based method in terms
of $\text{AP}_{\text{50}}$ and $\text{AP}_{\text{70}}$ benefiting from
original poincloud’s 3D information. Besides, we verify the generalization
ability of the pointcloud-based method by pretraining it on our synthetic
dataset, and it still has an acceptable result.

Figure 10: The entropy value and the specific poses.  represents an increasing entropy value from left to right among row’s samples. The size of joints’ ball represents the magnitude of the joint’s entropy value. Table 6: Human detection results on different datasets. “$\\#$” means using synthetic datasets (BaseketBallSync and PanopticSync) to pretrain and directly evaluate on corresponding real-world datasets. Datasets | Methods | Metric
---|---|---
$\text{AP}_{\text{50}}$ | $\text{AP}_{\text{70}}$
BasketBall | MVDet | 69.41 | 37.66
PointPillars$\\#$ | 88.17 | 44.26
PointPillars | 89.77 | 69.96
Panoptic | VoxelPose | 21.17 | 0.19
PointPillars$\\#$ | 40.17 | 6.25
PointPillars | 73.83 | 13.97

###  10.2 Entropy Analysis

Figure 10 shows the entropy value and the specific poses, and we can find that
the 3D poses become more and more irrational while the entropy goes up.