# Rethinking the Heatmap Regression for Bottom-up Human Pose Estimation

Zhengxiong Luo  Megvii Inc 2University of Chinese Academy of Sciences (UCAS)
Center for Research on Intelligent Perception and Computing (CRIPAC)
National Laboratory of Pattern Recognition (NLPR)  Institute of Automation,
Chinese Academy of Sciences (CASIA)
<EMAIL_ADDRESS>{wangzhicheng<EMAIL_ADDRESS>{yhuang,
wangliang<EMAIL_ADDRESS>Zhicheng Wang  Megvii Inc 2University of
Chinese Academy of Sciences (UCAS)  Yan Huang  Center for Research on
Intelligent Perception and Computing (CRIPAC)
National Laboratory of Pattern Recognition (NLPR)  Institute of Automation,
Chinese Academy of Sciences (CASIA)
<EMAIL_ADDRESS>{wangzhicheng<EMAIL_ADDRESS>{yhuang,
wangliang<EMAIL_ADDRESS>Liang Wang  Center for Research on Intelligent
Perception and Computing (CRIPAC)
National Laboratory of Pattern Recognition (NLPR)  Institute of Automation,
Chinese Academy of Sciences (CASIA)
<EMAIL_ADDRESS>{wangzhicheng<EMAIL_ADDRESS>{yhuang,
wangliang<EMAIL_ADDRESS>Tieniu Tan  Center for Research on Intelligent
Perception and Computing (CRIPAC)
National Laboratory of Pattern Recognition (NLPR)  Erjin Zhou  Megvii Inc
2University of Chinese Academy of Sciences (UCAS)

###### Abstract

Heatmap regression has become the most prevalent choice for nowadays human
pose estimation methods. The ground-truth heatmaps are usually constructed via
covering all skeletal keypoints by 2D gaussian kernels. The standard
deviations of these kernels are fixed. However, for bottom-up methods, which
need to handle a large variance of human scales and labeling ambiguities, the
current practice seems unreasonable. To better cope with these problems, we
propose the scale-adaptive heatmap regression (SAHR) method, which can
adaptively adjust the standard deviation for each keypoint. In this way, SAHR
is more tolerant of various human scales and labeling ambiguities. However,
SAHR may aggravate the imbalance between fore-background samples, which
potentially hurts the improvement of SAHR. Thus, we further introduce the
weight-adaptive heatmap regression (WAHR) to help balance the fore-background
samples. Extensive experiments show that SAHR together with WAHR largely
improves the accuracy of bottom-up human pose estimation. As a result, we
finally outperform the state-of-the-art model by $+1.5AP$ and achieve $72.0AP$
on COCO test-dev2017, which is comparable with the performances of most top-
down methods.

##  1 Introduction

Multi-person human pose estimation (HPE) aims to locate skeletal keypoints of
all persons in a given RGB image. It has been widely applied in human activity
recognition, human computer interaction, animation  _etc_. Current human pose
estimation methods fall into two categories: top-down and bottom-up. In top-
down methods, all persons are firstly cropped out by a human detector and then
resized to the same size before they are input to the keypoints detector.
Oppositely, bottom-up methods directly detect keypoints of all persons
simultaneously. It is more light-weight fast but suffers from various human
scales.

Figure 1: Top row: the noses of different persons are covered by gaussian
kernels with the same standard deviation. Bottom row: the standard deviations
for keypoints of different persons are adaptively adjusted in SAHR.

Heatmap regression is widely used in both top-down and bottom-up HPE methods.
The ground-truth heatmaps are constructed by putting 2D Gaussian kernels on
all keypoints. They are used to supervise the predicted heatmaps via L2 loss.
This method is easy to be implemented and has much higher accuracy than
traditional coordinate regression [35, 34, 30]. However, in current practice,
different keypoints are covered by gaussian kernels with the same standard
deviation [36, 6, 25], which means that different keypoints are supervised by
the same constructed heatmaps.

We argue that this is unreasonable in two aspects. Firstly, keypoints of
different scales are semantically discriminative in regions of different
spatial sizes. It may cause confusion to put the same gaussian kernel on all
keypoints. As shown in the top row of Figure 1, the noses of different persons
are covered by gaussian kernels with the same deviation ($\sigma=2$). In (a),
the covered region is restricted on the top of the nose. But in (b), the
Gaussian kernel could cover the face, and in (c), the whole head is even
covered. The various covered regions for the same keypoint may cause semantic
confusion. Secondly, even humans could not label the keypoints with pixel-wise
accuracy, and the ground-truth coordinates may have inherent ambiguities [13,
8]. Thus the keypoints could be treated as distributions (instead of discrete
points) centered around the labeled coordinates. Their standard deviations
represent their uncertainties and should be proportion to the labeling
ambiguities. However, current practice keeps the standard deviations fixed. It
seems to have ignored the influence of various labeling ambiguities of
different keypoints.

From the above discussion, the standard deviation for each keypoint should be
related to its scale and uncertainty. A straightforward way to solve these
issues is manually labeling different keypoints with different standard
deviations. However, this work is extremely labor-intensive and time-
consuming. Besides, it is difficult to define customized standard deviations
for different keypoints. Towards this problem, we propose the scale-adaptive
heatmap regression (SAHR), which can adaptively adjust the standard deviation
for each keypoint by itself.

Specifically, we firstly cover all keypoints by Gaussian kernels of the same
base standard deviation $\sigma_{0}$. We add a new branch to predict scale
maps $\mathbf{s}$, which are of the same shape as ground-truth heatmaps. Then
we modify the original standard deviation for each keypoint to
$\sigma_{0}\cdot\mathbf{s}$ by a point-wise operation. Thus to some extent,
$\mathbf{s}$ represents the scales and uncertainties of corresponding
keypoints. In this way, the suitable standard deviations for different
keypoints could be adaptively learned, and thus SAHR may be more tolerant of
various human scales and labeling ambiguities. However, as shown in the bottom
row of Figure 1, SAHR may aggravate the imbalance between fore-background
samples, which potentially restricts the improvements of SAHR [21, 19].
Motivated by focal loss for classification [21], we further introduce the
weight-adaptive heatmap regression (WAHR), which can automatically down-weight
the loss of relatively easier samples, and focus more on relatively harder
samples. Experiments show that the improvements brought by SAHR can be further
advanced by WAHR.

Our contributions can be summarized as four points:

  * 1.

To the best of our knowledge, this is the first paper that focuses on the
problems in heatmap regression when tackling large variance of human scales
and labeling ambiguities. We attempt to alleviate these problems by scale and
uncertainty prediction.

  * 2.

We propose a scale-adaptive heatmap regression (SAHR), which can adaptively
adjust the standard deviation of the Gaussian kernel for each keypoint,
enabling the model to be more tolerant of various human scales and labeling
ambiguities.

  * 3.

We propose a weight-adaptive heatmap regression (WAHR) to alleviate the severe
imbalance between foreground and background samples. It could automatically
focus more on relatively harder examples and fully exploit the superiority of
SAHR.

  * 4.

Our model outperforms the state-of-the-art model by $1.5AP$ and achieves
$72.0AP$ on COCO test-dev2017, which is comparable with the performances of
most top-down methods.

##  2 Related Works

###  2.1 Bottom-up Human Pose Estimation

Bottom-up HPE methods firstly detect all identity-free keypoints and then
group them into individual persons. Compared with recent top-down HPE methods
[6, 37, 33, 3], bottom-up methods are usually inferior on accuracy. However,
since they do not rely on human detectors and could decouple the runtime with
the number of persons, bottom-up methods may have more potential superiority
on speed [4]. But on the other hand, bottom-up methods have to tackle the
grouping problem and large variance of human scales.

Recent works about bottom-up HPE mostly focus on developing better grouping
methods [24, 27, 19, 17, 38]. In [17], a Part Intensity Field (PIF) and a Part
Association Field (PAF) are used to localize and associate body parts. In
[19], the body parts are learned in the same way as keypoints by heatmaps. And
in [38], keypoints are grouped according to their offsets from corresponding
center points. In this paper, we use associative embedding proposed in [24],
which simple yet proved to be effective for points grouping [1, 10, 18].
Although the grouping method has been advanced a lot, few works are done about
the various human scales. In this paper, we mainly focus on the problems in
bottom-up HPE when tackling large variance of human scales.

###  2.2 Heatmap Regression

Heatmap regression is widely used for semantic landmarks localization, such as
keypoints of human faces [2], hands [32], bodies [36, 25] and household
objects [29]. The ground-truth heatmaps are constructed by putting 2D Gaussian
kernels on the labeled points. The pixel values on the heatmaps are usually
treated as the probabilities of corresponding pixels being the keypoints. This
method is easy to be implemented and could potentially attain pixel-wise
accuracy. Thus heatmap regression has become the dominant method for HPE.
However, current methods typically cover all keypoints by Gaussian kernels
with the same standard deviations. It may work well for top-down methods, in
which all persons are resized to the same size. But in bottom-up methods, in
which persons are of various scales, it seems to be more desirable to adjust
the standard deviation for each keypoint according to the scale of the
corresponding person.

Figure 2: During training, the ground-truth heatmaps are firstly scaled
according to predicted scale maps and then are used to supervise the whole
model via weight-adaptive loss. During testing, the predicted heatmaps and
associative embeddings are used for grouping of individual persons.

###  2.3 Uncertainty Prediction

Recently, uncertainty prediction has become an important method for many tasks
[5, 15, 11, 31]. As there are usually inevitable labeling ambiguities in the
training datasets [31], it is better to explicitly model the uncertainty for
predictions. In [13], He _et al_. treat the positions of ground-truth bounding
boxes as Gaussian distributions around the labeled coordinates, and use KL
loss [23] to supervise the model. In [8], a similar idea is adopted to predict
the coordinates and objecting scores of bounding boxes. For HPE, inherent
ambiguities may also exist in ground-truth keypoint, such as inaccurate
labeling, occlusion, or ambiguous cases. Original heatmap regression covers
keypoints by Gaussian kernels while keeping standard deviations fixed. In that
case, the ambiguities of different keypoints are assumed to be the same. This
implicit assumption may be too strong and potentially hurt the performance. In
this paper, the scale-adaptive heatmap regression alleviates this problem by
introducing scale maps to adaptively modify the standard deviation for each
keypoint.

##  3 Proposed Method

###  3.1 Formulation

Suppose $C^{p}_{k}=\\{x^{p}_{k},y^{p}_{k}\\}$ denotes the coordinate of the
$k^{th}$ keypoint of the $p^{th}$ person, and $\mathbf{h}^{p}$ denotes its
corresponding ground-truth heatmap, then the covered region for $C^{p}_{k}$ is
written as

$\displaystyle\mathbf{h}^{p}_{k,i,j}=e^{{-((i-x^{p}_{k})^{2}+(j-y^{p}_{k})^{2})}/{2\sigma^{2}}}$
(1) $\displaystyle s.t.\quad\|i-x^{p}_{k}\|_{1}\leq
3\sigma\quad\|j-y^{p}_{k}\|_{1}\leq 3\sigma,$

where $\sigma$ denotes the standard deviation, and $\\{k,i,j\\}$ indicates the
position of pixel on $\mathbf{h}^{p}$. For $\|i-x^{p}_{k}\|\textgreater
3\sigma$ or $\|j-y^{p}_{k}\|\textgreater 3\sigma$, we have
$\mathbf{h}^{p}_{k,i,j}=0$. If the number of persons is $N$, then the overall
ground-truth heatmaps are

$\mathbf{H^{\sigma}}=\max\\{\mathbf{h}^{1},\mathbf{h}^{2},\dots,\mathbf{h}^{N}\\},$
(2)

where $\max$ is pixel-wisely operated.

Suppose the predicted heatmaps are $\mathbf{P}$, then the regression loss is

$\mathcal{L}_{regressoin}=\|\mathbf{P}-\mathbf{H}^{\sigma}\|^{2}_{2}.$ (3)

###  3.2 Scale-Adaptive Heatmap Regression

In previous methods, the standard deviation $\sigma$ is fixed as $\sigma_{0}$
for all keypoints, in which case the ground-truth heatmaps are denoted as
$\mathbf{H^{\sigma_{0}}}$. However, keypoints of different scales have
semantically discriminative regions, thus they are expected to be covered by
Gaussian kernels with different deviations. Since it is hard to manually label
each keypoint, we hope that the model could learn to adjust $\sigma$ by
itself.

We add a new branch to predict the scale maps $\mathbf{s}$, which are of the
shape with ground-truth heatmaps. For keypoint
$C^{p}_{k}=\\{x^{p}_{k},y^{p}_{k}\\}$, we modify the standard deviation to
$\sigma_{0}\cdot\mathbf{s}_{k,x^{p}_{k},y^{p}_{k}}$. then the covered region
for $C^{p}_{k}$ becomes

$\displaystyle\mathbf{h}^{p}_{k,i,j}=e^{{-((i-x_{k})^{2}+(j-y_{k})^{2})}/{2(\sigma_{0}\cdot\mathbf{s}_{k,x^{p}_{k},y^{p}_{k}})^{2}}}$
(4) $\displaystyle s.t.\quad\|i-x^{p}_{k}\|_{1}\leq
3\sigma\quad\|j-y^{p}_{k}\|_{1}\leq 3\sigma.$

Since the covered region is relatively small, we may have
$\mathbf{s}_{k,x^{p}_{k},y^{p}_{k}}\approx\mathbf{s}_{k,i,j}$ for
$\|i-x^{p}_{k}\|_{1}\leq 3\sigma$ and $\|j-y^{p}_{k}\|_{1}\leq 3\sigma$. Thus,
for simplicity, the modification can be written as an element-wise operation:

$\displaystyle\mathbf{h}^{p}_{k,i,j}=e^{{-((i-x_{k})^{2}+(j-y_{k})^{2})}/{2(\sigma_{0}\cdot\mathbf{s}_{k,i,j})^{2}}}$
(5) $\displaystyle s.t.\quad\|i-x^{p}_{k}\|_{1}\leq
3\sigma\quad\|j-y^{p}_{k}\|_{1}\leq 3\sigma.$

We denote the modified heatmaps as $\mathbf{H}^{\sigma_{0}\cdot\mathbf{s}}$.
If we express $\mathbf{H}^{\sigma_{0}\cdot\mathbf{s}}$ by original heatmaps
$\mathbf{H}^{\sigma_{0}}$, then we have

$\mathbf{H}^{\sigma_{0}\cdot\mathbf{s}}_{k,i,j}=\left\\{\begin{aligned}
&(\mathbf{H}^{\sigma_{0}}_{k,i,j})^{1/\mathbf{s}_{k,i,j}}\quad\mathbf{H}^{\sigma_{0}}_{k,i,j}>0&\\\
&\mathbf{H}^{\sigma_{0}}_{k,i,j}\qquad\quad\quad\mathbf{H}^{\sigma_{0}}_{k,i,j}=0.&\end{aligned}\right.$
(6)

$\mathbf{H}^{\sigma_{0}\cdot\mathbf{s}}$ is what we call scale-adaptive
heatmaps. It can be attained from an element-wise operation over original
heatmaps, thus it is also easy to be implemented.

For keypoints whose scale factors are larger than $1$, their corresponding
standard deviation will be larger than $\sigma_{0}$, which means that the
region covered by this Gaussian kernel will also become larger. Otherwise the
reverse. Thus, to some extent, the scale factor may reflect the scale of the
corresponding person.

Furthermore, some changes need to be made to stabilize the training. Firstly,
we add a regularizer loss for the predicted scale maps:

$\mathcal{L}_{regularizer}=\|(1/\mathbf{s}-1)\mathbf{1}_{\mathbf{H}^{\sigma_{0}/\mathbf{s}}>0}\|^{2}_{2},$
(7)

in which $\mathbf{1}_{\mathbf{H}^{\sigma_{0}\cdot\mathbf{s}}>0}$ denotes the
mask that keeps only regions covered by gaussian kernels. Secondly, we
transform the exponential form of $\mathbf{H}^{\sigma_{0}\cdot\mathbf{s}}$
into a polynomial series by Taylor expansion at $\mathbf{s}=1$. We omit terms
higher than the second order and have:

$\displaystyle\qquad\qquad\mathbf{H}^{\sigma_{0}\cdot\mathbf{s}}_{k,i,j}=$ (8)
$\displaystyle\left\\{\begin{aligned}
&\frac{1}{2}\mathbf{H}^{\sigma_{0}}_{k,i,j}(1+(1+\boldsymbol{\alpha}_{k,i,j}\ln(\mathbf{H}^{\sigma_{0}}_{k,i,j}))^{2})\quad\mathbf{H}^{\sigma_{0}}_{k,i,j}>0&\\\
&0\qquad\qquad\qquad\qquad\qquad\qquad\qquad\qquad\mathbf{H}^{\sigma_{0}}_{k,i,j}=0,&\end{aligned}\right.$

where $\boldsymbol{\alpha}=1/\mathbf{s}-1$. Then, the total loss is written
as:

$\displaystyle\mathcal{L}_{total}$
$\displaystyle=\mathcal{L}_{regressoin}+\lambda\mathcal{L}_{regularizer}$ (9)
$\displaystyle=\|\mathbf{P}-\mathbf{H}^{\sigma_{0}\cdot\mathbf{s}}\|^{2}_{2}+\lambda\|\boldsymbol{\alpha}\mathbf{1}_{\mathbf{H}^{\sigma_{0}/\mathbf{s}}>0}\|^{2}_{2},$

where $\lambda$ is the weight for regularizer term. In practice, we use
$\lambda=1$. This is what we call scale-adaptive heatmap regression (SAHR).

Figure 3: L2 loss cannot appropriately measure the difference between two
distributions. Even if the loss is reduced a lot, the center points of these
two distributions may keep unchanged. Thus original heatmap regression has to
fix the standard deviation for all keypoints, and our scale-adaptive heatmap
regression needs to add regularizer loss for scale map.

###  3.3 Relation to Uncertainty Prediction

In [13], He _et al_. argue that there are inherent labeling ambiguities of box
coordinates in some cases. Thus they treat both the predicted and ground-truth
coordinates as Gaussian distributions, and the standard deviations could
represent the uncertainties of the coordinates. The loss is constructed as KL
loss [23]:

$\mathcal{L}\propto\frac{\|\mathbf{X}_{p}-\mathbf{X}_{g}\|^{2}_{2}}{2\sigma^{2}}+\frac{1}{2}\log(\sigma^{2}),$
(10)

where $\mathbf{X}_{p}$ and $\mathbf{X}_{g}$ denote the predicted and ground-
truth coordinates respectively. And $\sigma$, which is predicted by the model,
denotes the standard deviations of assumed Gaussian distributions. The former
and later terms of this loss could also be treated as regression and
regularizer loss respectively. It will automatically down-weight the
regression loss of coordinates with relatively larger uncertainties, and thus
could be more tolerant of various labeling ambiguities.

The success of the original heatmap regression could also be partially
explained by the same idea. But heatmap regression uses L2 loss instead of KL
loss to measure the difference between two distributions. As shown in Figure
3, simply minimizing L2 loss could not lead the model appropriately. Thus,
original heatmap regression has to fix the standard deviations. However, the
fixed value maybe not suitable for all keypoints and potentially hurt the
performance. We still use L2 loss in SAHR. But instead of keeping the standard
deviations fixed, we add a regularizer term to help lead the model to converge
to the desired direction. SAHR combines the merits of both heatmap and
coordinate regression.

From this perspective, the scale factor $s_{k,i,j}$ could also represent the
uncertainty of the corresponding keypoint. While in the previous section we
infer that scale factors indicate the scales of corresponding persons. These
two statements may be consistent. The relation is also intuitive: larger
persons are more likely to be labeled with larger absolute error, and thus the
scale factors may be proportional to the uncertainties of corresponding
persons.

Methods | Backbone | Input Size | #Params | GFLOPs | AP | $AP^{50}$ | $AP^{75}$ | $AP^{M}$ | $AP^{L}$
---|---|---|---|---|---|---|---|---|---
w/o mutli-scale test
OpenPose [4] | - | - | - | - | $61.8$ | $84.9$ | $67.5$ | $57.1$ | $68.2$
Hourglass [25] | Hourglass | $512$ | $277.8$ | $206.9$ | $56.6$ | $81.8$ | $61.8$ | $49.8$ | $67$
PersonLab [27] | ResNet-152 | $1401$ | $68.7$ | $405.5$ | $66.5$ | $88.0$ | $72.6$ | $62.4$ | $72.3$
PifPaf [17] | - | - | - | - | $66.7$ |  |  | $62.4$ | $72.9$
HrHRNet [7] | HRNet-W32 | $512$ | $28.5$ | $47.9$ | $66.4$ | $87.5$ | $72.8$ | $61.2$ | $74.2$
HrHRNet [7] \+ SWAHR | HRNet-W32 | $512$ | $28.6$ | $48.0$ | $67.9$ | $88.9$ | $74.5$ | $62.4$ | $75.5$
HrHRNet [7] | HRNet-W48 | $640$ | $63.8$ | $154.3$ | $68.4$ | $88.2$ | $75.1$ | $64.4$ | $74.2$
HrHRNet [7] \+ SWAHR | HRNet-W48 | $640$ | $63.8$ | $154.6$ | $\mathbf{70.2}$ | $\mathbf{89.9}$ | $\mathbf{76.9}$ | $\mathbf{65.2}$ | $\mathbf{77.0}$
w/ mutli-scale test
Hourglass [25] | - | $512$ | $277.8$ | $206.9$ | $63.0$ | $85.7$ | $68.9$ | $58.0$ | $70.4$
PersonLab [27] | - | $1401$ | $68.7$ | $405.5$ | $65.5$ | $86.8$ | $72.3$ | $60.6$ | $72.6$
HrHRNet [7] | HRNet-W48 | $640$ | $63.8$ | $154.3$ | $70.5$ | $89.3$ | $77.2$ | $66.6$ | $75.8$
HrHRNet [7] \+ SWAHR | HRNet-W48 | $640$ | $63.8$ | $154.6$ | $\mathbf{72.0}$ | $\mathbf{90.7}$ | $\mathbf{78.8}$ | $\mathbf{67.8}$ | $\mathbf{77.7}$

Table 1: Results on COCO test-dev2017. Top: without multi-scale test. Bottom:
with multi-sale test (scale factors are $0.5$, $1.0$, and $1.5$).

###  3.4 Weight-Adaptive Heatmap Regression

We experimentally find that SAHR may aggravate the imbalance between fore-
background samples in heatmap regression. This imbalance may restrict the
improvement of SAHR. Most values in $\mathbf{H}^{\sigma_{0}\cdot\mathbf{s}}$
are zero, which may lead the model to overfit on background samples. In [21],
Lin _et al_. propose focal loss to alleviate a similar problem in
classification. It could adaptively down-weight the loss of well-classified
samples and thus help the model to focus on relatively harder samples.

To apply similar idea in heatmap regression, the straightforward way is
defining a weight tensor $\mathbf{W}$ for original L2 loss:

$\mathcal{L}_{regression}=\mathbf{W}\cdot\|\mathbf{P}-\mathbf{H}\|^{2}_{2},$
(11)

And $\mathbf{W}$ can be defined as

$\small\mathbf{W}_{k,i,j}=\left\\{\begin{aligned}
&(1-\mathbf{P}_{k,i,j})\quad\\{k,i,j\\}\text{ is positive sample}\\\
&\mathbf{P}_{k,i,j}\qquad\qquad\\{k,i,j\\}\text{ is negative
sample}\end{aligned}\right.$ (12)

However, in heatmap regression, the pixel values are contiguous, instead of
discrete $1$ or $0$, thus it is difficult to determine which are positive
(negative) samples.

Towards this issue, we propose a weight-adaptive heatmap regression (WAHR), in
which the loss weights are written as:

$\mathbf{W}=(\mathbf{H})^{\gamma}\cdot\|1-\mathbf{P}\|+\|\mathbf{P}\|\cdot(1-(\mathbf{H})^{\gamma})\\\
$ (13)

where $\gamma$ is the hyper-parameter that controls the position of a soft
boundary. And the soft boundary is defined as a threshold heatmap value $p$,
where $1-p^{\gamma}=p^{\gamma}$. For samples with heatmap values larger than
$p$, their loss weights are more close to ($1-\mathbf{P}$), otherwise are more
close to $\mathbf{P}$. We can get the threshold $p=2^{-\frac{1}{\gamma}}$. In
practice, we use $\gamma=0.01$.

Experiments in Sec 4.2 show that WAHR can further advance the improvement of
SAHR. When SAHR and WAHR are used together, we call it the scale and weight
adaptive heatmap regression (SWAHR).

Methods | $AP$ | $AP^{50}$ | $AP^{75}$ | $AP^{M}$ | $AP^{L}$
---|---|---|---|---|---
Top-down methods
Mask-RCNN [12] | $63.1$ | $87.3$ | $68.7$ | $57.8$ | $71.4$
G-RMI [28] | $64.9$ | $85.5$ | $71.3$ | $62.3$ | $70.0$
Sun _et al_. [34] | $67.8$ | $88.2$ | $74.8$ | $63.9$ | $74.0$
G-RMI [28] \+ extra data | $68.5$ | $87.1$ | $75.5$ | $65.8$ | $73.3$
CPN [6] | $72.1$ | $91.4$ | $80.0$ | $68.7$ | $77.2$
RMEPE [9] | $72.3$ | $89.2$ | $79.1$ | $68.0$ | $78.6$
CFN [14] | $72.6$ | $86.1$ | $69.7$ | $78.3$ | $64.1$
CPN(ensemble) [6] | $73.0$ | $91.7$ | $80.9$ | $69.5$ | $78.1$
SimpleBaseline [37] | $73.7$ | $91.9$ | $81.1$ | $70.3$ | $80.0$
HRNet-W48 [33] | $75.5$ | $92.5$ | $83.3$ | $71.9$ | $81.5$
Bottom-up methods
OpenPose [4] | $61.8$ | $84.9$ | $67.5$ | $57.1$ | $68.2$
Hourglass [25] | $65.5$ | $86.8$ | $72.3$ | $60.6$ | $72.6$
PifPaf [17] | $66.7$ | - | - | $62.4$ | $72.9$
SPM [26] | $66.9$ | $88.5$ | $72.9$ | $62.6$ | $73.1$
PersonLab [27] | $68.7$ | $89.0$ | $75.4$ | $64.1$ | $75.5$
HrHRNet-W48 [7] | $70.5$ | $89.3$ | $77.2$ | $66.6$ | $75.8$
HrHRNet-W48 [7] \+ SWAHR | $\mathbf{72.0}$ | $\mathbf{90.7}$ | $\mathbf{78.8}$ | $\mathbf{67.8}$ | $\mathbf{77.7}$

Table 2: Results on COCO test-dev2017. Top: top-down methods. Bottom: bottom-
up methods (with multi-scale test).

###  3.5 Implementation Details

In this paper, we mainly implement the proposed heatmap regression on HrHRNet
[7], which is a HRNet [33] with deconvolution modules. As shown in Figure 4,
it predicts multi-scale heatmaps, which are $1/4$ and $1/2$ sizes of the
original image respectively. During training, these two branches are
independently supervised by different heatmaps. During testing, it aggregates
multi-scale heatmaps to form the final predictions. The larger size of
heatmaps largely benefits the accuracy of keypoints detection, and the
heatmaps aggregation helps the model achieve remarkable results with only a
single-scale test. The grouping is done by associate embedding [24]. For SAHR
we add an extra branch to predict scale maps, and the model is denoted as
HrHRNet + SAHR. If only WAHR is used, the model is denoted as HrHRNet + WAHR.
And if both methods are used, the model is denoted as HrHRNet + SWAHR.

Figure 4: Implementation of scale and weight adaptive heatmap aggregation on
HrHRNet.

##  4 Experiments

###  4.1 COCO Keypoint Detection

Dataset. Most of our experiments are done COCO dataset [22], which contains
over $200$K images and $250$K person instances. Our models are trained on COCO
train2017 ($57$K images), and evaluated on COCO val2017 ($5$K images) and COCO
test-dev ($20$K images).

Evaluation metric. We use the standard evaluation metric Object Keypoint
Similarity (OKS) to evaluate our models.
$\text{OKS}={\frac{\sum_{i}\exp(-d_{i}^{2}/2s^{2}k_{i}^{2})\delta(v_{i}>0)}{\sum_{i}\delta(v_{i}>0)}}$,
where $d_{i}$ is the Euclidean distance between the detected keypoint and its
corresponding ground-truth, $v_{i}$ is the visibility flag of the ground-
truth, $s$ denotes the person scale, and $k_{i}$ is a per-keypoint constant
that controls falloff. We report the standard average precision ($AP$) and
recall, including $AP^{50}$ ($AP$ at OKS=$0.5$), $AP^{75}$, $AP$ (mean of $AP$
scores from OKS=$0.50$ to OKS=$0.95$ with the increment as $0.05$, $AP^{M}$
($AP$ scores for person of medium sizes) and $AP^{L}$ ($AP$ scores for persons
of large sizes).

Training. Following the setting of [24, 7], we augment the data by random
rotation ([$-30^{\circ}$, $30^{\circ}$]), random scaling ([$0.75$, $1.25$]),
random translation ([$-40$, $40$]) and random horizontal flip. The input image
is then cropped to $512\times 512$ (or $640\times 640$).

The models are optimized by Adam [16] optimizer, and the initial learning rate
is set as $2\times 10^{-3}$. Each model is trained for $300$ epochs and the
learning rate will linearly decay to $0$ in the end.

Testing. Following the setting of [7], the input image is firstly padded to
square and then resize the short side to $512$ (or $640$). We also perform
heatmap aggregation by averaging output heatmaps of different sizes. The flip
test is also performed in all experiments. For the multi-scale test, we resize
the original image by scale factor $0.5$, $1.0$, and $1.5$ respectively, and
then aggregate the heatmaps as the final prediction.

Results on COCO test-dev2017. We firstly make comparisons with the state-of-
the-art bottom-up HPE methods. Results are shown in Table 1. As one can see,
with the help of SWAHR, HrHRNet can achieve the best results with or without
multi-scale test. And if with multi-scale test, it can finally achieve
$\mathbf{72.0}$ AP score on test-dev2017. On the other hand, SWAHR can bring
steady improvements to HrHRNets with different backbones and different input
sizes, while introducing only marginal computational cost.

Then we make comparisons with recent top-down HPE methods. Results are shown
in Table 2. As one can see, with the help of SWAHR, HrHRNet-W48 has exceeded
many early top-down methods. CPN [6] is the champion of COCO Keypoint
Challenge in 2017, and our method gets nearly the same results as it.

Figure 5: Visualization of $1/\mathbf{s}$. Redder pixels indicate larger
values _i.e_. smaller scale factors. Within each image, the order of person
scales ($A\textgreater B\textgreater C$) is usually the same as the order of
scale factors ($A\textgreater B\textgreater C$).

###  4.2 Analysis

Ablation study. We design comparative experiments to validate the improvement
brought by SAHR and WAHR respectively. We use HrHRNet-W32 as the baseline
model, and validate these models on COCO val2017 dataset without multi-scale
test. As we can see in Table 3, SAHR can bring an improvement of $+0.7AP$. If
WAHR is further added, they together could bring an improvement of $+1.8AP$.
Also, WAHR alone can bring improvement of $+1.3AP$ improvements respectively.
This is because the server imbalance between fore-background samples also
exists in original heatmap regression.

Looking into the improvements on different scales, we can see that WAHR can
largely benefit the keypoints detection of both medium and large persons. This
is easy to understand: the severe imbalance between fore-background samples
exists both for large and medium persons, thus WAHR could benefit both. Since
the original base standard deviation is only suitable for relatively larger
persons (Figure 1 (a)), SAHR mainly focuses on adjusting the standard
deviations for relatively smaller persons. Thus, the improvements brought by
SA are mainly attributed to better performance on medium persons.

SAHR |  WAHR |  $AP$ |  $AP^{M}$ |  $AP^{L}$
---|---|---|---|---
|  |  $67.1$ |  $61.5$ |  $76.1$
|  $\surd$ |  $68.4$ |  $62.5$ |  $77.0$
$\surd$ |  |  $67.8$ |  $62.5$ |  $76.1$
$\surd$ |  $\surd$ |  $68.9$ |  $63.0$ |  $77.5$

Table 3: Ablation study on SAHR and WAHR respectively. The results are
reported on COCO val2017 dataset. Single-scale test only.

Visualizing scale maps. We visualize the learned scale maps $1/\mathbf{s}$ in
Figure 5. We resize the maps to the same size as the original image, take mean
values along the channel dimension, and normalize the maps along spatial
dimensions. The colormaps are drawn as that redder pixels indicate larger
values, which also means smaller scale factors. As one can see, on the whole,
smaller persons usually have smaller scale factors. For the scales of persons
in boxes $A$, $B$, and $C$, we have $A\textgreater B\textgreater C$. While for
scale factors in these boxes, we can also get the same order. It suggests that
SAHR adaptively down-scales the standard deviations for keypoints of smaller
persons, while up-scales that for relatively larger ones. Without extra
supervision, SAHR has learned the relationship between human scales and the
suitable standard deviations.

Naive baseline. Although the standard deviation for each keypoints is not
labeled. The scale for each person could be roughly deduced from the bounding
box. As a comparison, we substitute the scale maps in SAHR with the deduced
scale factor as a naive baseline. We call it scaled heatmap regression (SHR).
Specifically, we manually calculate the scale factor
$s_{k,i,j}=W_{box}/W_{base}$, where $W_{box}$ denotes the width of the
bounding box of corresponding person, and $W_{base}$ is the base width. In
practice, we use $W_{base}=256$. We use HrHRNet-W32 as the baseline. The
comparative results are shown in Table 4. SHR even hurts the performance of
the original model. This is easy to explain: the width of bounding boxes can
not exactly represent the person scales, because of various poses and
occlusions. This naive implementation may cause more confusion, and thus hurt
the original performance. Instead, SAHR could avoid this case by additively
learning the scale maps.

Methods | $AP$ | $AP^{0.5}$ | $AP^{0.75}$ | $AP^{M}$ | $AP^{L}$
---|---|---|---|---|---
baseline | $67.1$ | $86.2$ | $73.0$ | $61.5$ | $76.1$
SHR | $63.9$ | $84.7$ | $68.0$ | $55.5$ | $77.0$
SAHR | $67.8$ | $86.8$ | $73.7$ | $62.5$ | $76.1$

Table 4: Comparison with naive baseline on COCO val2017 without multi-scale
test.

Study of $\mathbf{\lambda}$. In SAHR, there is a hyper-parameter $\lambda$,
_i.e_. the weight for regularizer loss. Larger $\lambda$ means that the model
has to ‘pay more’ to adjust the standard deviation. It indicates that we are
more confident about the manually constructed ground-truth heatmaps. And
smaller $\lambda$ indicates that we encourage the model to adjust the standard
deviation for each keypoint by itself. We compare the performance of
HrHRNet-W32 + SAHR with different $\lambda$ on COCO val2017 dataset. As shown
in Table 5, when $\lambda=+\infty$, which means that the model is not allowed
to adjust the standard deviations, it will degrade to the original baseline
model. On the whole, the improvement brought by SA is no too sensitive to
$\lambda$, as the results keep the same when $\lambda=1$ and $\lambda=0.5$.
But when $\lambda$ becomes too small, _i.e_. $\lambda=0.1$, the model may be
able to largely adjust the standard deviations, while the model may be not
reliable enough. In such cases, the improvements may get hurt.

$\lambda$ | $0.1$ | $0.5$ | $1.0$ | $+\infty$
---|---|---|---|---
$AP$ | $67.6$ | $67.8$ | $67.8$ | $67.1$

Table 5: Study of hyper-parameter $\lambda$. Results are reported on COCO
val2017 dataset, without WA and multi-scale test.

Study of $\mathbf{\gamma}$. In WAHR, the hyper-parameter $\gamma$ controls the
soft boundary between positive and negative samples. Smaller $\gamma$
indicates that more samples will be determined as positive ones. To
investigate the influence of $\gamma$, we compare the performance of
HrHRNet-W32 + WAHR with different $\gamma$ on COCO val2017 dataset. As shown
in Table 6, when $\gamma$ decreases, the $AP$ score firstly grows quickly, and
then get stable at $68.4$, when $\gamma=0.001$. As $\gamma$ decreases, the
threshold value $p$ will also exponentially decrease. When $\gamma=0.01$,
$p\approx 8\times 10^{-31}$. In that case, almost all regions that are covered
by gaussian kernels have heatmap values larger than $p$. Thus, a further
decrement of $\gamma$ makes little difference to the final results.

$\gamma$ | $1.0$ | $0.1$ | $0.01$ | $0.001$
---|---|---|---|---
$AP$ | $67.8$ | $68.2$ | $68.4$ | $68.4$

Table 6: Study of hyper-parameter $\gamma$. Results are reported on COCO
val2017 dataset, without SA and multi-scale test.

Larger receptive filed V.S. Larger $\mathbf{\sigma}$. The intuitive idea is
that a larger receptive field will benefit the accuracy of larger persons. In
this section, we experimentally illustrate that the accuracy of larger persons
may be more related to larger standard deviations that are used to construct
ground-truth heatmaps.

We first compare the results with different receptive fields. We still use
HrHRNet-W32 as the baseline model. To exclude the influence of heatmaps
aggregation, we only use the results of larger heatmaps ($1/2$ size of the
original image). There are $4$ residual blocks in this branch. We change the
dilation rates of their convolutional layers to change the sizes of their
receptive fields. Different models are denoted as $dddd$, where each $d$
denotes the dilation rates of the corresponding residual block. The baseline
model is denoted as $1111$. Then we change it to $1122$ and $2222$ to
investigate the influence. As shown in Figure 7, as the dilation rates
increases, the AP scores of large persons almost keep the same. It indicates
that the accuracies of large persons are not restricted by the sizes of
receptive fields.

Then we investigate the influence of standard deviations that are used to
construct the ground-truth heatmaps. Results of the $1/2$ branch of
HrHRNet-W32 are reported in Table 8. As one can see, with an increase of
$\sigma$, the performance on medium persons becomes worse, while the model
performs better on large persons. It suggests that a larger $\sigma$ is more
suitable for larger persons. This is also consistent with our previous
assumption: keypoints of larger persons have larger semantically
discriminative regions and also larger labeling ambiguities.

dilation | $1111$ | $1122$ | $2222$
---|---|---|---
$AP$ | $66.6$ | $66.6$ | $66.7$
$AP^{M}$ | $61.3$ | $61.4$ | $61.3$
$AP^{L}$ | $75.0$ | $75.0$ | $75.1$

Table 7: Study of receptive fields. Results are reported on COCO val2017
dataset, without SA and multi-scale test.

$\sigma$ | $2$ | $2.5$ | $3$
---|---|---|---
$AP$ | $66.6$ | $66.1$ | $65.4$
$AP^{M}$ | $61.3$ | $60.1$ | $58.3$
$AP^{L}$ | $75.0$ | $75.2$ | $75.4$

Table 8: Study of receptive fields. Results are reported on COCO val2017
dataset, without SA and multi-scale test.

###  4.3 CrowdPose

We further make comparisons with state-of-the-art HPE methods on CrowdPose
dataset [20]. It contains about 20000 images and 80000 person instances The
training, validation, and testing datasets contain about 10000, 2000, and 8000
images respectively. CrowdPose dataset has more crow cases than COCO [22], and
thus is more challenging to multi-person pose estimation. The evaluation
metric almost the same as that of COCO, but with extra AP scores on relatively
easier samples ($AP^{E}$) and relatively harder samples ($AP^{H}$).

We firstly make comparisons with top-down methods. As shown in Table 9, top-
down methods have lost their superiority in crowd scenes. This is because top-
down methods assume that all persons could be completely copped by the human
detector, and each crop contains only one person. However, this assumption
does not hold in crowd scenes, where persons are usually heavily overlapped.
While bottom-up methods do not rely on the human detector and may be better at
tackling crowd scenes.

Based on HrHRNet, SWAHR could bring $+5.7AP$ improvements without multi-scale
test, and $+6.2AP$ with multi-scale test. which are much more significant on
COCO test-dev (Table 1). It indicates that SWAHR could bring more improvements
in crowd scenes. This may because that SWAHR has taken the various human
scales into considerations, and this problem is more evident in crow scenes.

Methods | $AP$ | $AP^{50}$ | $AP^{75}$ | $AP^{E}$ | $AP^{M}$ | $AP^{H}$
---|---|---|---|---|---|---
Top-down methods
Mask-RCNN [12] | $57.2$ | $83.5$ | $60.3$ | $69.4$ | $57.9$ | $45.8$
AlphaPose [9] | $61.0$ | $81.3$ | $66.0$ | $71.2$ | $61.4$ | $51.1$
SimpleBaseline [37] | $60.8$ | $84.2$ | $71.5$ | $71.4$ | $61.2$ | $51.2$
Top-down with refinement
SPPE [20] | $66.0$ | $84.2$ | $71.5$ | $75.5$ | $66.3$ | $57.4$
Bottom-up methods w/o multi-scale testt
OpenPose [4] | - | - | - | $62.7$ | $48.7$ | $32.3$
HrHRNet-W48 [7] | $65.9$ | $86.4$ | $70.6$ | $73.3$ | $66.5$ | $57.9$
HrHRNet-W48 [7] \+ SWAHR | $\mathbf{71.6}$ | $\mathbf{88.5}$ | $\mathbf{77.6}$ | $\mathbf{78.9}$ | $\mathbf{72.4}$ | $\mathbf{63.0}$
Bottom-up methods w/ multi-scale testt
HrHRNet-W48 [7] | $67.6$ | $87.4$ | $72.6$ | $75.8$ | $68.1$ | $58.9$
HrHRNet-W48 [7] \+ SWAHR | $\mathbf{73.8}$ | $\mathbf{90.5}$ | $\mathbf{79.9}$ | $\mathbf{81.2}$ | $\mathbf{74.7}$ | $\mathbf{64.7}$

Table 9: Comparisons with top-down and bottom up methods on CrowPose test
dataset.

##  5 Conclusion

In this paper, we mainly focus on the problems in heatmap regression when
tackling various human scales and labeling ambiguities. We argue that in the
ground-truth heatmaps, keypoints of relatively larger persons should be
covered by gaussian kernels with also relatively larger standard deviation. We
illustrate this problem from the perspectives both of semantically
discriminative regions labeling ambiguities. Towards this issue, we propose a
scale-adaptive heatmap regression (SAHR), which can learn to adjust the
standard deviation for each keypoint by itself. Without extra supervision,
experiments show that the model could learn the relation between standard
deviation and the corresponding human scales. Also, as SAHR may aggravate the
imbalance between fore-background samples, we propose a weight-adaptive
heatmap regression (WAHR) to alleviate this problem. WAHR could automatically
down-weight the loss of well-classified samples and focus more on relatively
harder (usually foreground) samples. Experiments show that the two methods
(SAHR and WAHR) together can largely improve the performance of the original
model. As a result, we finally outperform the state-of-the-art model by
$+1.5AP$ and achieve $72.0AP$ on COCO test-dev2017 dataset, which is
comparable with the performances of most top-down methods.

## References

  * [1] Bert De Brabandere, D. Neven, and L. Gool.  Semantic instance segmentation with a discriminative loss function.  ArXiv, abs/1708.02551, 2017.
  * [2] Adrian Bulat and Georgios Tzimiropoulos.  How far are we from solving the 2d & 3d face alignment problem?(and a dataset of 230,000 3d facial landmarks).  In Proceedings of the IEEE International Conference on Computer Vision, pages 1021–1030, 2017.
  * [3] Yuanhao Cai, Zhicheng Wang, Zhengxiong Luo, Binyi Yin, Angang Du, Haoqian Wang, Xinyu Zhou, Erjin Zhou, Xiangyu Zhang, and Jian Sun.  Learning delicate local representations for multi-person pose estimation.  In ECCV, 2020.
  * [4] Zhe Cao, Gines Hidalgo Martinez, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Openpose: Realtime multi-person 2d pose estimation using part affinity fields.  IEEE transactions on pattern analysis and machine intelligence, 2019\.
  * [5] Y. Chen.  Refinement of boundary regression using uncertainty in temporal action localization.  2020\.
  * [6] Yilun Chen, Zhicheng Wang, Yuxiang Peng, Zhiqiang Zhang, Gang Yu, and Jian Sun.  Cascaded pyramid network for multi-person pose estimation.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7103–7112, 2018.
  * [7] Bowen Cheng, Bin Xiao, J. Wang, Humphrey Shi, T. Huang, and Lei Zhang.  Bottom-up higher-resolution networks for multi-person pose estimation.  ArXiv, abs/1908.10357, 2019.
  * [8] J. Choi, Dayoung Chun, H. Kim, and H. Lee.  Gaussian yolov3: An accurate and fast object detector using localization uncertainty for autonomous driving.  2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 502–511, 2019.
  * [9] Haoshu Fang, S. Xie, Yu-Wing Tai, and Cewu Lu.  Rmpe: Regional multi-person pose estimation.  2017 IEEE International Conference on Computer Vision (ICCV), pages 2353–2362, 2017.
  * [10] Alireza Fathi, Z. Wojna, Vivek Rathod, P. Wang, Hyun Oh Song, S. Guadarrama, and K. Murphy.  Semantic instance segmentation via deep metric learning.  ArXiv, abs/1703.10277, 2017.
  * [11] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  ArXiv, abs/1506.02142, 2016.
  * [12] Kaiming He, Georgia Gkioxari, P. Dollár, and Ross B. Girshick.  Mask r-cnn.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 42:386–397, 2020.
  * [13] Yihui He, Chenchen Zhu, Jianren Wang, M. Savvides, and X. Zhang.  Bounding box regression with uncertainty for accurate object detection.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2883–2892, 2019.
  * [14] Shaoli Huang, M. Gong, and D. Tao.  A coarse-fine network for keypoint localization.  2017 IEEE International Conference on Computer Vision (ICCV), pages 3047–3056, 2017.
  * [15] S. Isobe and S. Arai.  Deep convolutional encoder-decoder network with model uncertainty for semantic segmentation.  2017 IEEE International Conference on INnovations in Intelligent SysTems and Applications (INISTA), pages 365–370, 2017.
  * [16] Diederik P Kingma and Jimmy Ba.  Adam: a method for stochastic optimization. corr abs/1412.6980 (2014), 2014.
  * [17] S. Kreiss, L. Bertoni, and Alexandre Alahi.  Pifpaf: Composite fields for human pose estimation.  2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11969–11978, 2019.
  * [18] Hei Law and J. Deng.  Cornernet: Detecting objects as paired keypoints.  International Journal of Computer Vision, 128:642–656, 2019.
  * [19] Jiugang Li, Wen Su, and Zeng-Fu Wang.  Simple pose: Rethinking and improving a bottom-up approach for multi-person pose estimation.  In AAAI, 2020.
  * [20] Jiefeng Li, Can Wang, Hao Zhu, Yihuan Mao, Hao-Shu Fang, and Cewu Lu.  Crowdpose: Efficient crowded scenes pose estimation and a new benchmark.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 10863–10872, 2019.
  * [21] Tsung-Yi Lin, Priyal Goyal, Ross B. Girshick, Kaiming He, and P. Dollár.  Focal loss for dense object detection.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 42:318–327, 2020.
  * [22] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In European conference on computer vision, pages 740–755. Springer, 2014.
  * [23] K. Murphy.  Machine learning - a probabilistic perspective.  In Adaptive computation and machine learning series, 2012.
  * [24] Alejandro Newell, Zhiao Huang, and J. Deng.  Associative embedding: End-to-end learning for joint detection and grouping.  In NIPS, 2017.
  * [25] Alejandro Newell, Kaiyu Yang, and Jia Deng.  Stacked hourglass networks for human pose estimation.  In European conference on computer vision, pages 483–499. Springer, 2016.
  * [26] Xuecheng Nie, J. Zhang, S. Yan, and Jiashi Feng.  Single-stage multi-person pose machines.  2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6950–6959, 2019.
  * [27] G. Papandreou, Tyler Lixuan Zhu, Liang-Chieh Chen, Spyros Gidaris, J. Tompson, and Kevin Murphy.  Personlab: Person pose estimation and instance segmentation with a bottom-up, part-based, geometric embedding model.  In ECCV, 2018.
  * [28] G. Papandreou, Tyler Lixuan Zhu, Nori Kanazawa, A. Toshev, J. Tompson, C. Bregler, and Kevin Murphy.  Towards accurate multi-person pose estimation in the wild.  2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3711–3719, 2017.
  * [29] A. Saxena, Justin Driemeyer, and A. Ng.  Robotic grasping of novel objects using vision.  The International Journal of Robotics Research, 27:157 – 173, 2008\.
  * [30] Taiki Sekii.  Pose proposal networks.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 342–357, 2018.
  * [31] Yichun Shi, Anil K. Jain, and Nathan D. Kalka.  Probabilistic face embeddings.  2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 6901–6910, 2019.
  * [32] Tomas Simon, Hanbyul Joo, I. Matthews, and Yaser Sheikh.  Hand keypoint detection in single images using multiview bootstrapping.  2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4645–4653, 2017.
  * [33] Ke Sun, Bin Xiao, Dong Liu, and Jingdong Wang.  Deep high-resolution representation learning for human pose estimation.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5693–5703, 2019.
  * [34] Xiao Sun, Bin Xiao, S. Liang, and Y. Wei.  Integral human pose regression.  In ECCV, 2018.
  * [35] Alexander Toshev and Christian Szegedy.  Deeppose: Human pose estimation via deep neural networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1653–1660, 2014.
  * [36] Shih-En Wei, Varun Ramakrishna, Takeo Kanade, and Yaser Sheikh.  Convolutional pose machines.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 4724–4732, 2016.
  * [37] Bin Xiao, Haiping Wu, and Yichen Wei.  Simple baselines for human pose estimation and tracking.  In Proceedings of the European conference on computer vision (ECCV), pages 466–481, 2018.
  * [38] Xingyi Zhou, Dequan Wang, and Philipp Krähenbühl.  Objects as points.  ArXiv, abs/1904.07850, 2019.