∎

11institutetext: Feng Zhang 22institutetext: School of Computer Science and
Technology, Nanjing University of Posts and Telecommunications, Nanjing,
China. 22email<EMAIL_ADDRESS>
Xiatian Zhu 33institutetext: Centre for Vision, Speech and Signal Processing,
University of Surrey, Guildford GU2 7XH, United Kingdom. 33email:
<EMAIL_ADDRESS>
Chen Wang 44institutetext: School of Computer Science and Engineering,
University of Electronic Science and Technology of China, Chengdu, China.
44email<EMAIL_ADDRESS>

# Single Person Pose Estimation: A Survey

Feng Zhang  Xiatian Zhu  Chen Wang

###### Abstract

Human pose estimation in unconstrained images and videos is a fundamental
computer vision task. To illustrate the evolutionary path in technique, in
this survey we summarize representative human pose methods in a structured
taxonomy, with a particular focus on deep learning models and single-person
image setting. Specifically, we examine and survey all the components of a
typical human pose estimation pipeline, including data augmentation, model
architecture and backbone, supervision representation, post-processing,
standard datasets, evaluation metrics. To envisage the future directions, we
finally discuss the key unsolved problems and potential trends for human pose
estimation.

###### Keywords:

Human pose estimation Single person Deep learning Survey

##  1 Introduction

Human Pose Estimation (HPE) is a fundamental problem in computer vision. HPE
aims to obtain the spatial coordinates of human body joints in a person image,
with a wide variety of applications such as action recognition Zhang et al.
(2019c), person re-identification Zheng et al. (2019), semantic segmentation
Liang et al. (2018), human-robot interaction Xu et al. (2020), etc. It is
challenging due to heavy occlusions, varying clothing styles, poor lighting
conditions and various shooting angles. Earlier HPE methods Fischler and
Elschlager (1973); Felzenszwalb and Huttenlocher (2005); Andriluka et al.
(2009); Yang and Ramanan (2011); Johnson and Everingham (2011) adopt hand-
crafted features to describe human body. Mostly, these methods aims to learn
the underlying relations between different body parts, e.g., using the seminal
pictorial structure model Fischler and Elschlager (1973). However, they are
limited in accuracy especially under severe occlusions and complex lighting
conditions, partly due to less expressive representations.

In the past years, inspired by the great success of deep learning in image
recognition Krizhevsky et al. (2012) characterized by end-to-end feature
learning capability, researchers have introduced an increasing number of deep
human pose models Toshev and Szegedy (2014); Wei et al. (2016); Newell et al.
(2016); Xiao et al. (2018); Sun et al. (2019) and continuously refresh the
state-of-the-art model performance on standard data benchmarks. Deep learning
based human pose estimation has become a more exiting and active research area
since the first introduction of deep learning in 2014 Toshev and Szegedy
(2014). Given the large body of deep human pose estimation works published
thus far, we consider that it is time to review this field for three main
purposes: (1) To reflect its development trajectory and process by extensively
surveying previous methods and algorithms; (2) To structure different classes
of human pose methods; (3) To provide a holistic picture and schematic
overview of the whole field. These are not only useful and informative for the
beginners to understand quickly the current state-of-the-art methods for human
pose estimation, but also indicative and inspiring for the experts to further
push the forefront edges.

There exist a number of surveys on human pose estimation in the literature Ji
and Liu (2009); Liu et al. (2015); Gong et al. (2016); Sarafianos et al.
(2016); Wang et al. (2018a); Dang et al. (2019); Chen et al. (2020). However,
most of existing surveys Sarafianos et al. (2016); Wang et al. (2018a); Ji and
Liu (2009); Dang et al. (2019) focus on basic foundations and knowledge,
without comprehensive review on deep learning human pose estimation methods.
Whilst Liu et al. (2015); Gong et al. (2016) focus on reviewing human pose
methods, they are largely out-of-date with a high desire for covering the
latest research advances. This survey timely solves this need with focus on
deep learning based human pose estimation on single-person images in
particular. This is because multi-person human pose estimation can be easily
decomposed into person detection and pose estimation, which represents the
state-of-the-art solutions Toshev and Szegedy (2014); Wei et al. (2016);
Newell et al. (2016); Xiao et al. (2018); Sun et al. (2019).

###  1.1 Overview

To organize a large number of existing deep learning HPE methods in an
intuitive manner, we propose a novel taxonomy from the model component
perspective. Table 1 show our proposed taxonomy with a system pipeline
illustrated in Fig. 1. Accordingly, our survey is organized as the following:
Section 2 describes the common data augmentation strategies that are important
for boosting the model performance. Section 3 presents typical model
architectures and network backbones. Section 4 reviews the largely ignored yet
significant post-processing. Section 5 details the supervision representation
used in optimization. Section 6 and Section 7 summarize the standard human
pose datasets and performance evaluation metrics, respectively.

Figure 1: The deep learning pipeline of human pose estimation in the single-
person setting. Table 1: The taxonomy of human pose estimation in the model
component perspective.

Perspective | Category | Sub-category
---|---|---
Data augmentation | Random data augmentation | Generic data augmentation Tompson et al. (2014); Pfister et al. (2015)
Pose-specific data augmentation Ke et al. (2018); Wang et al. (2018b); Park et
al. (2020)
Optimized data augmentation | Neural Architecture Search (NAS) based methods Hou et al. (2020)
Generative Adversarial Network (GAN) based methods Peng et al. (2018)
Deep human pose model | Model architecture | Sequential architectures Jain et al. (2013); Fan et al. (2015); Pfister et al. (2015); Rafi et al. (2016); Nie et al. (2018); Sun et al. (2019); Tang and Wu (2019); Su et al. (2019)
Cascaded architectures Toshev and Szegedy (2014); Tompson et al. (2015); Wei
et al. (2016); Sun et al. (2017a); Newell et al. (2016); Yang et al. (2017);
Ke et al. (2018); Chen et al. (2018); Liu et al. (2018b); Ning et al. (2017);
Zhang et al. (2019b)
Recurrent architectures Pfister et al. (2015); Belagiannis and Zisserman
(2017); Carreira et al. (2016); Gkioxari et al. (2016)
Adversarial architectures Chen et al. (2017); Chou et al. (2018)
Neural Architecture Search (NAS) Zoph and Le (2016); Baker et al. (2016); Liu
et al. (2018a); Yang et al. (2019); Gong et al. (2020)
Network design | Multi-scale feature learning Tompson et al. (2015); Newell et al. (2016); Yang et al. (2017); Ning et al. (2017); Liu et al. (2018b); Chen et al. (2018); Cao et al. (2019); Sun et al. (2019); Su et al. (2019); Artacho and Savakis (2020)
Prior knowledge incorporation Ning et al. (2017); Bulat and Tzimiropoulos
(2016); Xia et al. (2017); Nie et al. (2018)
Spatial relation modelling Chen and Yuille (2014); Tompson et al. (2014); Yang
et al. (2016); Chu et al. (2016b, a); Sun et al. (2017b); Lifshitz et al.
(2016); Chen et al. (2017); Tang and Wu (2019); Zhang et al. (2019b)
Attention mechanism Chu et al. (2017); Liu et al. (2018b); Su et al. (2019)
Supervision representation | Heatmap based methods | Heatmap regression Tompson et al. (2014); Pfister et al. (2015); Wei et al. (2016); Liu et al. (2018b); Sun et al. (2019)
Heatmap regression and offset regression Papandreou et al. (2017)
Coordinate based methods | Direct coordinate regression Toshev and Szegedy (2014); Fan et al. (2015); Sun et al. (2017a)
Coordinate regression through heatmap Sun et al. (2018)
Post-processing | Heuristic methods | Empirical design Newell et al. (2016)
Data-driven methods | Distribution learning Zhang et al. (2020); Yang et al. (2020)

##  2 Data Augmentation

As a generic and common model training strategy, data augmentation is critical
for the generalization performance of deep learning methods including human
pose models, due to the data hungry nature of deep neural networks Cireşan et
al. (2010); Krizhevsky et al. (2012); Simonyan and Zisserman (2014) and the
difficulty of collecting large training data. Often it is considered as a
regularization which can mitigate the notorious overfitting problem. In
general, data augmentation would generate more training samples by introducing
some modification to the original training data whilst keeping the same
semantic labels. There are random and optimized data augmentation for training
human pose models, as detailed below.

###  2.1 Random Data Augmentation

Data augmentation operations are often randomly applied to a training sample
during training. The common augmentation operations include flipping,
rotating, scaling, occluding, and color jittering Pfister et al. (2014).

#### Flipping.

Flipping can improve the model’s robustness against imaging direction. There
are two types of flipping: horizontal flipping and vertical flipping.
Horizontal flipping is used more than the vertical one. In model training, the
flipping augmentation is usually triggered at a certain probability, e.g. 50%.

#### Rotation.

Human bodies may be lying and leaning with the body spines exhibiting
different angles in imbalanced distribution. This imbalance may impose
negative effect to model training. Rotation augmentation is a simple solution
for mitigating this. It is performed by rotating the image clockwise or
counterclockwise around the center of human body. Typically, the rotation
angle ranges from -30 to 30 degrees. As flipping, the rotation operation is
also triggered at a preset probability.

#### Scaling.

Body scale variation is often large in unconstrained images which might
negatively affect the performance of human pose estimation methods. To
alleviate this issue, scaling augmentation is adopted to improve the model
robustness for image scales. In training, a random scaling factor is selected
and applied.

#### Color jittering.

Human person images are typically collected in a variety of environments with
different lighting conditions. To make the model suitable to cope with such
image data, color jittering is effective to simulate diverse lighting
conditions in training.

#### Occlusion.

Occlusion is an inevitable challenge due to uncontrolled shooting angle and
complex scene. Excepted collecting samples which is largely limited,
synthesizing images with occlusions turns out to be cheaper and easier for
improving model training.

#### Pose-specific augmentations.

All the above augmentation operations are generally applicable for many
different computer vision tasks. Besides, pose-specific augmentation
techniques Ke et al. (2018); Wang et al. (2018b); Park et al. (2020) have been
proposed additionally. Specifically, in the real world, it is often the case
that human body joints are obscured by other objects; In crowded scenes, there
may be multiple joints of the same class that can greatly confuse a model.
Under such observation, Ke et al. Ke et al. (2018) proposed a keypoint-masking
augmentation method to synthesize hard training samples with self-occlusion
and mutual interference. Considering that there are a large number of samples
with a certain part of human body and the upper body presents easier-to-detect
joints than the lower body, Wang et al. Wang et al. (2018b) presented a half-
body augmentation strategy for more specific operation. It is observed that
samples with few keypoints in the training set are corner cases often simply
ignored by existing methods. To better solve this problem, Park et al. Park et
al. (2020) introduced a body-part-cropping augmentation approach by randomly
selecting and cropping a proportion of human body at different scales to
increase the diversity of training data.

###  2.2 Optimized Data Augmentation

Random data augmentation is simple yet less efficient. This can be addressed
by learning to augment training data, i.e., optimized data augmentation. It is
a relatively new research direction in human pose estimation. Two
representative types of learning framework in the literature are adversarial
learning and neural architecture search.

The first type usually employs the generative adversarial networks (GANs). The
key idea is to generate hard training samples that can more effectively incur
training loss and therefore expedite model learning. For example, Peng et al.
Peng et al. (2018) proposed Online Adversarial Augmentation Network (OAAN)
jointly learned with an existing HPE model. OAAN is designed to generate hard
adversarial samples with a reward-penalty policy in a way that the target HPE
model can improve its robustness against hard samples. Bin et al. Bin et al.
(2020) proposed a Semantic Data Augmentation (SDA) method. It aims to solve
the limitation of random data augmentation Ke et al. (2018) that generated
samples by a naive copying-and-pasting body part strategy have limited
diversity and unrealistic appearance. SDA involves constructing semantic body
part pool for generation of hard training samples. Specifically, a semantic
body part pool is first constructed by a human parsing algorithm; Then, hard
training samples are generated with an optimal pasting configuration in an
adversarial learning manner.

Unlike adversarial learning based methods, the second type instead aims to
optimize the data augmentation application process. For instance, Hou et al.
Hou et al. (2020) first defined a search space with different operators such
as flipping, cropping, rotation. By treating the sequence of applying these
operators as a trainable component, they turned this problem into a discrete
optimization problem and solved it via an existing differentiable search
algorithm Liu et al. (2019).

##  3 Model Architecture and Network Design

According to the theory that the structure of neural networks determines its
function, the human brain has highly structured connections at birth which
determines what functions it has. These functions in the brain are enhanced
through learning subsequently. Like these networks in the brain, their
architectures and backbones play a very important role in HPE. How to design a
good architecture and backbone encoding human prior knowledge remains an open
problem. The models taking the scale variability and severe occlusions into
consideration make it possible to learn the complex mapping from natural image
to body pose. This section will describe the recent researches of the HPE
model from the two perspectives: the architecture and backbone.

###  3.1 Model Architecture

The architecture of HPE model can be classified into four categories:
sequential, cascaded, recurrent and adversarial architectures.

#### (I) Sequential architecture.

Typical sequential models Jain et al. (2013); Fan et al. (2015); Pfister et
al. (2015); Rafi et al. (2016); Nie et al. (2018); Sun et al. (2019); Tang and
Wu (2019); Su et al. (2019) construct a simple yet intuitive framework by a
series of basic layers such as convolution, pooling, fully connection to learn
a mapping from input images to joint coordinates. Jain et al. Jain et al.
(2013) proposed a CNN architecture with fully connected layers. The network
slides over the input image with an overlapped sliding window to detect the
presence of joints. Fan et al. Fan et al. (2015) developed a dual-source
network architecture that uses the raw image with a holistic view and the
image patches with a local view as model inputs to improve the accuracy of
HPE.

To avoid kinematically impossible predictions, Pfister et al. Pfister et al.
(2015) presented a novel architecture that includes a spatial fusion layer to
learn spatial dependencies of joints implicitly. Rafi et al. Rafi et al.
(2016) adopted an efficient multi-scale network architecture based on the
Inception network to reduce the model complexity and improve the model’s
ability to perceive larger context. Nie et al. Nie et al. (2018) proposed the
Parsing Induced Learner (PIL) method to facilitate the HPE model with parsing
information. Sun et al. Sun et al. (2019) proposed a network architecture that
preserves high-resolution features and fused low-resolution features with
high-resolution features to learn robust multi-scale feature representations.

Previous HPE models learn the shared feature representations for all body
parts, Tang et al. Tang and Wu (2019) found that the shared feature
representations are beneficial to related parts but fail to improve the
learning of other parts. To address this problem, they obtained the
correlations between joints through statistical analysis and designed a
sequential network architecture with multiple branches to incorporate
structured knowledge of human body into the network architecture design. Su et
al. Su et al. (2019) designed the Channel Shuffle Module (CSM) and the Spatial
Channel-wise Attention Residual Bottleneck (SCARB) to introduce the attention
mechanism into the networks. Specifically, the CSM can improve the information
exchange across channels and the SCARB is able to boost the residual learning.
Inspired by the Atrous Spatial Pyramid Pooling (ASPP) in semantic
segmentation, Artacho et al. Artacho and Savakis (2020) proposed the Waterfall
Atrous Spatial Pooling (WASP) module to enlarge the Field of View (FOV) and
obtain multi-scale representations.

#### (II) Cascaded architecture.

The cascaded architecture usually repeats several modules to refine the
prediction. Toshev et al. Toshev and Szegedy (2014) presented the first
application of CNNs to human pose estimation and designed a cascaded
architecture. The regressor in the first stage aims at estimating coarse body
pose and the subsequent regressors are designed to learn the displacement of
the part locations from the previous stage to the ground-truth locations.

Traditional architectures for classification use down-sampling layers such as
pooling to maintain invariance, reduce computation and improve efficiency, but
these pooling layers in the network are not appropriate for the HPE task and
may hurt the model performance. Thompson et al. Tompson et al. (2015) reduced
the number of down-sampling layers and implemented a sliding-window detector
with overlapping contexts to produce coarse predictions and then used the
Siamese network to refine these coarse locations. Wei et al. Wei et al. (2016)
designed a multi-stage architecture called Convolutional Pose Machine (CPM) to
learn long-range dependencies between joints. The CPM consists of several
stages and each stage operates on the predicted heatmaps from the previous
stage. The spatial relations of joints are captured through increasing the
size of receptive field for each subnetwork in the CPM. Sun et al. Sun et al.
(2017a) analyzed the relative locations of joints and found the uniform
distribution of these relative locations has a negative effect on the learning
procedure. To make the learning procedure easier, they proposed a novel
network with normalization module which is composed of body normalization and
limb normalization.

To extract multi-scale features and deal with ambiguous prediction, Newell et
al. Newell et al. (2016) designed the hourglass module by repeating the
bottom-up and top-down processing. Yang et al. Yang et al. (2017) introduced
the fractional pooling into the residual block and proposed the Pyramid
Residual Module (PRM) to learn scale-invariant representations from training
data. Ke et al. Ke et al. (2018) proposed a multi-scale structure learning
framework based on the revised hourglass module and utilized additional multi-
scale regression to perform global optimization of structure configuration.
There are occluded joints, invisible joints, and very complex backgrounds in
the real world. To cope with these challenging cases, Chen et al. Chen et al.
(2018) proposed a novel network structure called Cascaded Pyramid Network
(CPN) consisting of a GlobalNet and a RefineNet. The GlobalNet is responsible
for the easy samples and the RefineNet aims at dealing with those challenging
samples. Liu et al. Liu et al. (2018b) designed a Cascaded Inception of
Inception Network (CIOIN). The Inception of Inception (IOI) block in CIOIN is
able to fuse features with different semantics and preserve the scale
diversity in them. The Attention-modulated IOI (AIOI) block is capable of
adjusting the importance of features according to the context. The CIOIN
architecture employs a multiple iterative stage to capture long-range
dependencies. Ning et al. Ning et al. (2017) constructed a fractal network by
using the hourglass modules and Inception-ResNet blocks to regress human pose
without explicit graphical modelling. To extract contextual information, Zhang
et al. Zhang et al. (2019b) proposed the Cascade Prediction Fusion (CPF) to
fuse predictions and features from the previous stage. To learn relations
between joints, they further designed the Pose Graph Neural Network (PGNN) to
model these relations according to the body structure.

#### (III) Recurrent architecture.

Feedforward architectures are able to learn hierarchical feature
representations from data, but they fail to capture the connections between
joints and the associations between features from different layers. Recurrent
architectures enhance the feature representations and incorporate top-down
feedback into the feedforward architectures to complement their feature
representations, establishing relations between the input space and the output
space. Pfister et al. Pfister et al. (2015) made the first attempt to model
features from different layers by leveraging the proposed spatial fusion
layer. Inspired by Pfister, Belagiannis et.al Belagiannis and Zisserman (2017)
presented a novel recurrent model including a feed-forward module and several
recurrent modules. These recurrent modules can effectively revise predictions
with iterative feature fusion. Carreira et al. Carreira et al. (2016)
implemented a self-correction model which simulates the recurrent mechanism in
the human brain to learn a residual displacement from current prediction to
the ground truth progressively in an Iterative Error Feedback (IEF) manner.
Unlike previous studies Pfister et al. (2015); Belagiannis and Zisserman
(2017); Carreira et al. (2016) on recurrent architecture, Gkioxari et al.
Gkioxari et al. (2016) defined a fixed ordering of joints based on the
kinematic tree of the human body and modelled the dependencies between
adjacent joints by learning a conditional distribution to predict joints one
by one.

#### (IV) Adversarial architecture.

GAN consists of a generator and a discriminator, and is trained by playing
minimax game between the generator and the discriminator. The generator
synthesizes samples capable of confusing the discriminator, while the
discriminator continuously improves its capability in distinguishing fake
samples from the true data distribution. Recent applications of GAN in human
pose estimation include Chen et al. (2017) and Chou et al. (2018) view the
generator as a HPE network. They treat the discriminator as a proxy of the
loss function to guide the learning process of the generator by imposing
geometric constraints and prior knowledge through the discriminator. To
exploit body configurations and enforce structure constraints on HPE model,
Chen et al. Chen et al. (2017) developed two discriminators (a pose
discriminator and a confidence discriminator). The pose discriminator is
responsible for identifying whether the locations of joints estimated by the
generator are reasonable or not, and the confidence discriminator can utilize
the geometric structure of the human body to evaluate these predictions with
high confidence, especially the occluded joints with high confidence.
Different from Chen et al. (2017), Chou et al. Chou et al. (2018) proposed a
discriminator to reconstruct heatmaps. For $j$-th joint, the discriminator is
optimized by minimizing the loss $L_{fake}$ between the generated heatmaps
$\\{\hat{C}_{j}\\}_{j=1}^{J}$ and the reconstructed ones
$\\{D(\hat{C}_{j},X)\\}_{j=1}^{J}$ (Eq.(2)), and it tries to maximize the loss
$L_{real}$ between the ground-truth heatmaps $\\{C_{j}\\}_{j=1}^{J}$ and the
reconstructed ones $\\{D(C_{j},X)\\}_{j=1}^{J}$ (Eq.(1)).

$L_{real}=\sum_{j=1}^{M}(C_{j}-D(C_{j},X))^{2}$ (1)

$L_{fake}=\sum_{j=1}^{M}(\hat{C}_{j}-D(\hat{C}_{j},X))^{2}$ (2)

#### Neural architecture search.

In practice, the hand-designed architecture needs to be adjusted according to
the requirements of a specific task. However, these modifications involve
tedious verification process. Therefore, Neural Architecture Search (NAS) is
presented to find good network architecture with limited computing resources
and little human intervention. Zoph et al. Zoph and Le (2016) and Baker et al.
Baker et al. (2016) introduced Reinforcement Learning (RL) into NAS by
modelling the searching process as a Markov Decision Process (MDP). However,
this kind of methods consume too many computing resources making them
infeasible to practical applications.

In order to reduce the search time, Liu et al. Liu et al. (2018a) proposed the
DARTs method which implements a continuous relaxed representation of the
architecture. Specifically, they replaced the discrete non-differentiable
search space in RL based approaches to obtain the optimal architecture by
gradient descent efficiently. Motivated by Convolutional Neural Fabrics Saxena
and Verbeek (2016) and DARTS Liu et al. (2018a), Yang et al. Yang et al.
(2019) designed the part representation according to the structure of human
skeleton and parameterized the Cell-based Neural Fabric (CNF) to perform the
gradient-based search at the micro and macro search space. Gong et al. Gong et
al. (2020) presented a novel NAS framework called AutoPose which defines a
hierarchical multi-scale search space and utilizes both gradient-based cell-
level search and reinforcement learning based network-level search to discover
multiple parallel branches of diverse scales.

###  3.2 Network Design

#### Multi-scale feature learning.

The main purpose of the research of HPE is to extract multi-scale features to
cope with various scales of the human body and to obtain a larger receptive
field for global inference. Tompson et al. Tompson et al. (2015) proposed a
two-stage network with pose refinement implemented by a Siamese network. The
Siamese network takes the coarse heatmaps with different resolutions as inputs
to refine predictions.

To handle body parts with different scales such as the face, hand, foot, etc,
Newell et al. Newell et al. (2016) presented the stacked hourglass network
formed by stacking several hourglass modules with pooling and upsampling
operations to capture features at every scale. Motivated by the residual
learning He et al. (2016), Wei et al. Yang et al. (2017) proposed several
variants of Pyramid Residual Modules (PRMs) simulating the mechanism of human
visual perception to capture multi-scale features. Ning et al. Ning et al.
(2017) adopted the design of Inception block and residual block, and proposed
the Inception-ResNet module to construct a fractal network structure which
captures the interdependence between joints across different scales and
resolutions.

To deal with various appearance and occlusion in human pose estimation, Liu et
al. Liu et al. (2018b) proposed a Cascaded Inception of Inception Network
(CIOIN). The Inception of Inception (IOI) block in the CIOIN not only is able
to preserve scale diversity but also can be upgraded to the Attention-
modulated IOI (AIOI) block to reweight the features with different scales.
Chen et al. Chen et al. (2018) designed the Cascaded Pyramid Network (CPN)
architecture based on the ResNet backbone, and they integrated the U-shape
structure into the architecture design to create a feature pyramid that is
able to preserve spatial information of multi-scale features. Cao et al. Cao
et al. (2019) introduced a stronger low-level feature module called Feature
Pyramid Stem (FPS). The FPS module is capable of extracting target-specific
features in valid regions, and progressively fusing features with different
resolutions to improve the quality of representations.

Most of the existing architectures utilize the high-to-low and low-to-high
process to learn the multi-scale feature representations. However, recovering
from low-resolution representations does not bring more useful information.
How to maintain high-resolution and multi-scale representations is a non-
negligible concern in architecture design. Sun et al. Sun et al. (2019)
presented a novel architecture, namely HighResolution Net (HRNet), which can
maintain high-resolution representations through the whole process. Su et al.
Su et al. (2019) proposed a Channel Shuffle Module (CSM) to further enhance
the cross-channel information exchange between feature maps with different
scales.

To exploit the contextual information, enlarge the field-of-view, and enhance
multi-scale representations, Artacho et al. Artacho and Savakis (2020)
proposed the Waterfall Atrous Spatial Pooling (WASP) module. The WASP module
employs cascaded atrous convolutions at increasing rates in the parallel
architecture to obtain larger field-of-view.

#### Prior knowledge incorporation.

Although end-to-end networks can learn discriminative feature representations,
the HPE model trained from limited data cannot describe the actual
distribution. How to encode prior knowledge into neural networks is a very
complicated task. To solve this problem, Ning et al. Ning et al. (2017)
designed two types of external knowledge that includes geometrical features
and Histogram of Gradients (HOG) based features. They utilized the proposed
knowledge projection module to decode the abstract external knowledge and
injected the decoded knowledge into the fractal network. Bulat et al. Bulat
and Tzimiropoulos (2016) proposed a part detection network to capture
contextual information and structured relations. The part detection network
guides the pose regression network to learn the structure of body parts,
improving the predictions of occluded parts with low confidence scores.

Motivated by the correlations between part segmentation and pose estimation,
Xia et al. Xia et al. (2017) proposed a prior knowledge incorporation schema
to exploit segmentation results to improve HPE. Specifically, semantic part
prior knowledge from part Fully Convolutional Network (FCN) is fed to Fully-
connected Conditional Random Field (FCRF) to avoid unreasonable predictions
from the pose FCN. Nie et al. Nie et al. (2018) proposed a parsing-guided HPE
method which uses the Parsing Induced Learner (PIL) to learn the parameters of
adaptive convolution from the features of the parsing encoder. Specifically,
they leverages the adaptive convolution to adjust features from the pose
encoder to obtain accurate predictions.

#### Spatial relation modelling.

The dimension of input space is determined by many factors such as deformation
caused by the varying viewing angles, changes in lighting and clothing, and
occlusion induced by human-object interaction. The mapping from image to pose
is highly non-linear, which makes it necessary to exploit the spatial inter-
dependencies between joints to improve the representation ability of HPE
model. Chen et al. Chen and Yuille (2014) defined a tree-structure graphic
model to learn spatial relations by conditional probabilities from unary term
and Image Dependent Pairwise Relationships (IDPR) term, and adopted Dynamic
Programming (DP) algorithm to find the optimal configuration for person
instances. Tompson et al. Tompson et al. (2014) proposed a Markov Random Field
(MRF) model which contains a unary term for modelling joint locations and a
pairwise term for modelling the conditional distribution of neighbouring
joints. They implemented the MRF model by the message passing network to
eliminate incorrect predictions.

Different from the MRF-liked model Tompson et al. (2014), Yang et al. Yang et
al. (2016) blended the appearance mixtures into the loopy model to enforce
deformation constraints, which allows the HPE model to handle large pose
variations. Instead of modelling structures by graphical models Tompson et al.
(2014); Yang et al. (2016); Chu et al. (2016b), Chu et al. Chu et al. (2016b)
proposed a CRF-CNN framework which extends the loopy model by adding a novel
pairwise term to learn the relations and structures among features. To refine
feature maps, Chu et al. Chu et al. (2016a) presented a bi-directional
structured feature learning model by utilizing geometrical transform kernels
to pass on information between feature maps for each joint. Sun et al. Sun et
al. (2017b) introduced an intuitive yet stable bone-based representation which
is easier to learn and more stable than traditional joint-based representation
to encode the inter-dependencies inside the structure of body pose. Lifshitz
et al. Lifshitz et al. (2016) designed a voting representation by log-polar
binning and then aggregated voting maps using a large deconvolution kernel.
They added a novel image-based consensus voting binary term to remove the
independence assumption in traditional binary term by averaging over all
locations in the voting maps.

Motivated by the GAN, Chen et al. Chen et al. (2017) proposed a pose
discriminator which implicitly exploits the spatial inter-relations of joints
to distinguish the fake poses from the real ones. Tang et al. Tang and Wu
(2019) analyzed the correlations between different body parts, and obtained
related parts by applying spectral clustering to the correlation matrix. They
designed a part-based branching network according to the compositionality of
human bodies to learn specific high-level features for better performance.

Graphical models involving multiple sequential updates usually incur
accumulative error. Instead of defining a explicit graphical model to learn
the structure of the human body, Zhang et al. Zhang et al. (2019b) integrated
Pose Graph Neural Network (PGNN) into the HPE network to exploit spatial
contextual information from neighboring joints.

#### Attention mechanism.

Attention mechanism in the human visual system helps the brain ignore
irrelevant regions and pay more attention to important information. In the
field of HPE, attention mechanism can also be integrated into existing
networks to help HPE models focus on relevant areas. Chu et al. Chu et al.
(2017) designed three types of attention modules for HPE. The multi-resolution
attention within the hourglass module aims to refine features, and the multi-
semantics attention across several stacks of hourglass is used to capture
various semantic information. Besides, the hierarchical attention is adopted
to encode holistic and local body structure. Liu et al. Liu et al. (2018b)
introduced an Inception of Inception (IOI) module to preserve low-level
features. To improve the robustness of the HPE model against scale variety,
they incorporated the attention mechanism into the IOI module to form an
Attention-modulated IOI (AIOI) module to fuse multi-scale features
dynamically. To learn the relations between the low-level and high-level
feature maps, Su et al. Su et al. (2019) proposed a Channel Shuffle Module
(CSM) implemented by channel shuffle operation to exchange information within
feature pyramid, and designed a Spatial Channel-wise Attention Residual
Bottleneck (SCARB) to highlight the spatial and channel-wise information from
regions of interest.

##  4 Post-processing

A fundamental challenge in human pose estimation is that, the gap between
continuous joint space and discrete heatmap space results in quantization
error. Despite it is important for model performance, this error issue is
largely ignored with little attention and efforts paid in most studies. To
mitigate this problem, Newell et al. Newell et al. (2016) presented a simple,
heuristic method. Specifically, they shifted empirically the highest location
$\bm{m}$ in heatmap towards the second highest peak $\bm{s}$ by a quarter of
pixel to obtain the final output as:

$\bm{p}=\bm{m}+0.25\frac{\bm{s}-\bm{m}}{\|\bm{s}-\bm{m}\|_{2}}$ (3)

Since then, this ad-hoc method has been extensively used in the follow-up HPE
models as de-facto standard post-processing. However, the rationale of this
method is largely unknown. Importantly, how does this step affect the final
model performance is under-studied.

To solve these issues, Zhang et al. Zhang et al. (2020) conducted a systematic
investigation on post-processing. They revealed for the first time that this
quantization error could introduce significant model performance degradation,
particularly when low-resolution input images are used. Moreover, a superior
distribution-aware post-processing method based on Taylor series approximation
to Gaussian distribution was invented without re-training already-optimized
HPE models. In particular, by approximating the heatmap distribution with
Taylor series, it decodes human joint coordinates as:

$\bm{p}=\bm{m}-\big{(}\mathcal{D}^{\prime\prime}(\bm{m})\big{)}^{-1}\mathcal{D}^{\prime}(\bm{m})$
(4)

where the first- $\mathcal{D}^{\prime}(\bm{m})$ and second-gradient
$\mathcal{D}^{\prime\prime}(\bm{m})$ can be estimated efficiently around the
outputted peak $\bm{m}$ point only.

Subsequently, treating the predicted heatmap as a composite function of signal
and noise, Yang et al. Yang et al. (2020) proposed a Distribution-Aware and
Error-Compensation (DAEC) decoding method to obtain coordinates of keypoints
by integrating over the entire heatmap. To achieve more accurate predictions,
Fieraru et al. Fieraru et al. (2018) proposed a data synthesis technique by
simulating the typical erroneous samples and learning a pose refinement model
from these synthesized samples to correct the predictions. Moon et al. Moon et
al. (2019) proposed a model-agnostic refinement method that adopts the
diagnosis method Ruggero Ronchi and Perona (2017) to analyze the common
localization errors in HPE and utilizes the error statistics to synthesize new
training samples. It is shown that a coarse-to-fine estimation network trained
with these samples can improve effectively.

##  5 Supervision Representation

From the supervision representation perspective, HPE methods can be classified
into coordinate-based Toshev and Szegedy (2014); Fan et al. (2015); Sun et al.
(2017a) and heatmap-based methods Tompson et al. (2014); Pfister et al.
(2015); Wei et al. (2016); Liu et al. (2018b); Sun et al. (2019).
Specifically, coordinate-based methods predict/regress the joint coordinates
directly. Direct coordinate regression is intuitive and simple in model
design. However, this approach often reaches only unsatisfactory model
performance, largely due to that this regression learning task is extremely
difficult particularly given unconstrained images.

On the other hand, the heatmap-based methods Tompson et al. (2014); Pfister et
al. (2015); Wei et al. (2016); Liu et al. (2018b); Sun et al. (2019) take an
extra advantage of the spatial structure information in the image to describe
the relations between body parts and provide contextual and structural support
for model learning. Usually these methods adopt discrete heatmaps to represent
continuous distribution. The conversion from discrete heatmaps to coordinates
is hence necessary which introduces quantization errors. To alleviate this
problem, Sun et al. Sun et al. (2018) treated the heatmap as an intermediate
representation and proposed a differentiable integral approach to obtain the
joint coordinate from the heatmap. Similar to Sun et al. (2018), Nibali et al.
Nibali et al. (2018) proposed a Differentiable Spatial to Numerical Transform
(DSNT) method, as well as employed the Jensen-Shannon divergence to enforce a
shape constraint on the distribution of heatmap. Besides, Earth Mover’s
Distance (EMD) Martin Arjovsky and Bottou (2017); Liu et al. (2020) can also
be utilized to measure the distribution between ground-truth and predicted
heatmaps. Papandreou et al. Papandreou et al. (2017) and Zhang et al. Zhang et
al. (2019d) explored an alternative to reduce the quantization error. They
divided the localization problem into two parts: a detection task and a
regression task. The detection task learns binary heatmaps indicating the
regions of body parts, whilst the regression task learns the fractional part
of numerical joint coordinates. It is therefore a coarse-to-fine learning
pipeline. To addressing significant model performance degradation from low-
resolution, Wang et al. Chen et al. (2021) introduced a novel Confidence-Aware
Learning (CAL) method by addressing the fundamental training-testing
discrepancy of previous offset learning methods Papandreou et al. (2017);
Zhang et al. (2019d).

##  6 Human Pose Datasets

This section presents seven popular HPE datasets including BBC Pose datasets,
ChaLearn dataset, Poses in the Wild dataset, Frames Labeled In Cinema
datasets, Leeds Sports Pose datasets, MPII Human Pose dataset and Look Into
Person dataset. Samples of each dataset are visualized in Fig.2.

Figure 2: Dataset samples. We show six random image samples for each dataset.

#### BBC Pose dataset.

The BBC Pose dataset contains 20 videos extracted from BBC programmes, each of
which is approximately 0.5h to 1.5h in length. There are 10/5/5 videos in the
training set, validation set and test set respectively. There are 5 signers in
the training and validation sets respectively, and 4 signers in the test set.
Splitting the dataset this way aims to evaluate the generalization performance
of HPE model in dealing with different signers. The annotation contains 7
joints, i.e., head, shoulders, elbows, wrists. The labels of training and
validation sets are obtained by Buehler et al. (2011), while the samples in
the test set are manually annotated. An extended version of this dataset, the
extended BBC Pose dataset, includes additional 72 training videos annotated by
the tracker of Charles et al. Charles et al. (2014). In addition, Charles et
al. Charles et al. (2014) also presented the Short BBC Pose dataset. This
dataset contains 5 videos with different sleeve lengths compared to the BBC
Pose and the extended BBC Pose datasets whose samples contain only long-
sleeved signers. In order to evaluate the performance of algorithms on
different videos accurately, each video contains 200 hand-annotated test
frames and the final test set is composed of 5 videos.

#### ChaLearn dataset.

The Chalearn dataset Escalera et al. (2013) is a multimodal dataset collected
with the Kinect camera, including sound, 2D and 3D poses, RGB and depth
images, and segmentation masks indicating regions of the human body from 27
persons. The dataset is split into 393/287/276 video clips for training,
validation and testing, respectively. The whole human skeleton includes 20
joints, i.e., hip centre, spine, shoulder centre, head, shoulders, elbows,
wrists, hands, hips, knees, feet, ankles. The dramatic changes in clothing
across videos cause the main challenge for HPE.

#### Poses in the Wild (PiW) dataset.

To evaluate the performance of models in the wild, Cherian et al. Cherian et
al. (2014) proposed the Poses in the Wild (PiW) dataset which consists of 30
sequences from 3 Hollywood movies. The PiW dataset defines 8 human upper-body
keypoints (neck, shoulders, elbows, wrists, and mid-torso) and is
characterized by occlusions, severe camera motion and background clutter.

#### Frames Labeled In Cinema (FLIC) dataset.

Similar to the PiW dataset, Frames Labeled In Cinema (FLIC) dataset Sapp and
Taskar (2013) is extracted from 30 popular Hollywood movies and annotated by
the crowdsourcing marketplace Amazon Mechanical Turk (AMT). The dataset has a
total of 5003 images and is split into two parts: the training set ($\sim$80%,
3987 images) and the test set ($\sim$20%, 1016 images). Different from the PiW
dataset with 8 keypoints, the annotation in FLIC dataset has 11 upper-body
joints, namely nose, eyes, shoulders, elbows, wrists, and hips. The FLIC
dataset has a full set version where all frames from movies are annotated by
the AMT labelling service.

#### Leeds Sports Pose (LSP) dataset.

In sports activities, there are many challenging cases such as motion blur,
severe occlusion, extreme body deformation. The Leeds Sports Pose dataset
Johnson and Everingham (2010) (LSP) is collected from Flickr to solve the pose
estimation in sports activities. The dataset contains 2000 various sports
images, with a half used for training and the other half for testing.
Different from PiW and FLIC, LSP pays attention to the full-body pose
estimation and defines the human pose with 14 joints, i.e., ankles, knees,
hips, wrists, elbows, shoulders, neck, and head top.

#### MPII Human Pose (MHP) dataset.

Deep learning based HPE methods require large amounts of data, but the sizes
of early datasets are often limited. Small datasets cannot cover many
practical cases and the lack of diversity could cause model overfitting
particularly in deep learning. To address these issues, Andriluka et al.
Andriluka et al. (2014) established a taxonomy of daily activities and
collected human images from various human activities in YouTube videos to
build a large benchmark called MPII Human Pose (MHP) dataset. Compared with
previous datasets, MHP dataset is characterized by its large scale and high
diversity. The MHP dataset offers richer pose annotations including
coordinates of 16 joints (i.e., shoulders, elbows, wrists, hips, knees,
ankles, pelvis, thorax, upper neck, head top), the corresponding visibility of
these joints, 3D torso orientations, head orientations, and activity labels.
This dataset is split into the train/valid/test sets of size 25863/2958/11701.

#### Look Into Person (LIP) dataset.

Look Into Person (LIP) Liang et al. (2018) is another large-scale and
challenging dataset for human parsing and pose estimation. It contains 50,462
human images (19,081 full-body images, 13,672 upper-body images, 403 lower-
body images, 3,386 head-missed images, 2,778 back-view images, and 21,028
images with occlusions) with challenging view, severe occlusions, various
postures, varying clothing, and low resolution. This dataset is divided into
30,462 training images, 10,000 validation images and 10,000 test images. Each
person is annotated with 16 joints (i.e., shoulders, elbows, wrists, hips,
knees, ankles, head, neck, spine, and pelvis).

Table 2: Single person pose estimation datasets. Dataset |  Training/Val/Test |  Parts |  #Joints
---|---|---|---
BBC Pose Charles et al. (2014) |  10 videos/5 videos/5 videos |  Upper body |  7
Extended BBC Pose Charles et al. (2014) |  82 videos/5 videos/5 videos |  Upper body |  7
Short BBC Pose Charles et al. (2014) |  -/-/5 videos |  Upper body |  7
Charlearn Escalera et al. (2013) |  393 videos/287 videos/276 videos |  Full body |  20
PiW Cherian et al. (2014) |  total 30 videos |  Upper body |  8
FLIC Sapp and Taskar (2013) |  3,987 images/-/1,016 images |  Upper body |  10
FLIC-full Sapp and Taskar (2013) |  total 20928 images |  Upper body |  10
LSP Johnson and Everingham (2010) |  1,000 images/-/1,000 images |  Full body |  14
Extended LSP Johnson and Everingham (2010) |  11,000 images/-/1,000 images |  Full body |  14
MHP Andriluka et al. (2014) |  25,863 images/2,958 images/11,701 images |  Full body |  16
LIP Liang et al. (2018) |  30,462 images/10,000 images/10,000 images |  Full body |  16

##  7 Performance Evaluation Metrics

This section focuses on the model evaluation metrics that differentiate the
methods with some numerical index regarding their generalization performance.

#### Percentage of Correct Parts (PCP).

Figure 3: The stickmen annotation. $e_{1}^{i}$ and $e_{2}^{i}$ are the two
endpoints of limb $i$ and $l^{i}$ is the limb length, $p_{1}^{i}$ and
$p_{2}^{i}$ are the corresponding predictions.

Ferrari et al. Ferrari et al. (2008) defined a performance criterion based on
the stickmen annotation as shown in Fig. 3. In the definition of stickmen,
each limb $i$ of person $n$ is simplified as a segment which has two endpoints
$\\{p_{i,1}^{n},p_{i,2}^{n}\\}$. When evaluating the Percentage of Correct
Parts (PCP), the distance $d_{i,1}^{n}$ between the predicted joint
$p_{i,1}^{n}$ and the corresponding endpoint $e_{i,1}^{n}$ is normalized by
the limb length $l_{i}^{n}=\|e_{i,1}^{n}-e_{i,2}^{n}\|_{2}$ formulated as:

$d_{i,1}^{n}=\frac{\|p_{i,1}^{n}-e_{i,1}^{i}\|_{2}}{l_{i}^{n}}\\\ $ (5)

Similarly, we can obtain another distance $d_{i,2}^{n}$:

$d_{i,2}^{n}=\frac{\|p_{i,2}^{n}-e_{i,2}^{i}\|_{2}}{l_{i}^{n}}\\\ $ (6)

The original PCP takes both endpoints into consideration. It calculates the
distances from both endpoints to their corresponding predictions and then
compares the average of both distances against a pre-set threshold to
determine whether this limb is detected correctly:

$\displaystyle\delta(i,n)=\left\\{\begin{array}[]{lc}1,&\frac{d_{i,1}^{n}+d_{i,2}^{n}}{2}\leq
r\\\ 0,&\text{otherwise}\end{array}\right.$ (9)

where $r$ is the threshold and its value usually takes $50\%$ of the limb
length.

The original PCP for limb $i$ is defined as:

$\text{PCP}(i)=\sum_{n=1}^{N}\frac{\delta(i,n)}{N}$ (10)

In addition to the original PCP metric, Johnson et al. Johnson and Everingham
(2010) and Pishchulin et al. Pishchulin et al. (2012) adopted a stricter
$\delta(i,n)$ as:

$\displaystyle\delta(i,n)=\left\\{\begin{array}[]{lc}1,&d_{i,1}^{n}\leq
r\text{ and }d_{i,2}^{n}\leq r\\\ 0,&\text{otherwise}\end{array}\right.$ (13)

#### Percent of Detected Joints (PDJ).

Since the two PCP metrics penalize shorter limbs, they cannot reveal the
actual performance of a HPE method. Toshev et al. Toshev and Szegedy (2014)
designed the Percent of Detected Joints (PDJ) which instead uses the torso
diameter $t^{n}$ as the scale factor to normalize the distance between the
prediction $p_{j}^{n}$ and the ground truth $g_{j}^{n}$ for joint $j$ of
person $n$ as:

$d_{j}^{n}=\frac{\|p_{j}^{n}-g_{j}^{n}\|_{2}}{t^{n}}\\\ $ (14)

Like the PCP metrics, if the normalized distance $d_{j}^{n}$ is within a
threshold $r$, the joint $j$ of person $n$ is considered detected. This can be
formulated as:

$\displaystyle\delta(j,n)$
$\displaystyle=\left\\{\begin{array}[]{lc}1,&d_{j}^{n}\leq r\\\
0,&\text{otherwise}\end{array}\right.$ (17)

Thus, the PDJ for joint $j$ can be defined as:

$\text{PDJ}(j)=\sum_{n=1}^{N}\frac{\delta(j,n)}{N}$ (18)

#### Percentage of Correct Keypoints (PCK).

When evaluating the predictions of a model, the PDJ metric is greatly
influenced by the torso diameter, making it less robust and consistent. Hence,
Andriluka et al. Andriluka et al. (2014) modified the PDJ metric by replacing
the torso diameter with the head segment length $h^{n}$:

$\displaystyle d_{j}^{n}$
$\displaystyle=\frac{\|p_{j}^{n}-g_{j}^{n}\|_{2}}{h^{n}}$ (19)

The resulting metric is named PCKh, defined as:

$\displaystyle\text{PCKh}(j)$
$\displaystyle=\sum^{N}_{n=1}\frac{\delta(d_{j}^{n}<0.5)}{N}$ (20)

Besides, they also computed the Area under the PCK Curve (AUC) with the
threshold ranging from $0$ to $0.5$ for giving a more comprehensive
evaluation.

#### Other Metrics.

In addition to the above metrics, the researchers have also applied other
metrics to assess the performance of a method from different perspectives For
example, the number of parameters is usually utilized to measure the model
complexity and the FLoating-point OPerations (FLOPs) is adopted to evaluate
the computing complexity Zhang et al. (2019a). The real-time performance is
another key aspect for practical applications, and the Frames Per Second (FPS)
is an often-used metric to evaluate the running speed at certain devices.

##  8 Conclusions and Discussions

This paper has reviewed a wide range of recent researches on HPE in the
following four perspectives: data augmentation, model architecture, post-
processing, and learning target. We discuss the development of HPE methods at
data augmentation, model architecture, pose-processing, and supervision
representation. Although great improvements have been made, HPE still faces
some fundamental challenges, as detailed below.

#### Transferability.

Deep learning based HPE methods rely heavily on labelled data with specific
characteristics. For example, the MHP dataset covers daily activities, while
the LSP dataset focuses on the sports scene. The model trained on one dataset
may perform badly on another dataset Chen and Zwicker (2021). Thus, the
transferability (i.e., model generalization across different domains) is still
an important unsolved problem.

#### Severe occlusions.

Occlusion resulting from varying shooting angles and crowd scenes remains a
challenging problem in HPE. Some methods tackles this problem by learning
explicit or implicit spatial models Wei et al. (2016); Wang et al. (2020), but
suffer from the notorious overfitting issue. To deal with the occlusion
problem efficiently and effectively, inferring the positions of occluded
joints might give useful guidance information.

#### Low-resolution images.

Low-resolution images are widely available in the wild and present extra
challenges for state-of-the-art methods due to the lack of fine-grained
appearance information. Nonetheless, it is still largely under-studied despite
its potential importance in practice. There have been a couple of preliminary
attempts on low-resolution challenge Neumann and Vedaldi (2018); Chen et al.
(2021) with more advanced models to be innovated in the future.

#### Real-time performance.

Current researches mostly focus on designing deeper and more complicated HPE
model to obtain higher accuracies. In many real-world applications, however,
performing fast inference is likely a more important consideration especially
for resource-limited platforms with human-computer interaction involved Zhang
et al. (2019a).

#### System integration and applications.

The HPE is a fundamental task in computer vision with wide application values
such as social robotics, human body generation, human-object interaction,
activity recognition, person re-identification. Taking into account the
integration of HPE in the whole system and conducting its research accordingly
would be more focusing and purposing due to the presence of contextual
knowledge and domain priors.

## References

  * Andriluka et al. (2009) Andriluka M, Roth S, Schiele B (2009) Pictorial structures revisited: People detection and articulated pose estimation. In: 2009 IEEE conference on computer vision and pattern recognition, IEEE, pp 1014–1021
  * Andriluka et al. (2014) Andriluka M, Pishchulin L, Gehler P, Schiele B (2014) 2d human pose estimation: New benchmark and state of the art analysis. In: Proceedings of the IEEE Conference on computer Vision and Pattern Recognition, pp 3686–3693
  * Artacho and Savakis (2020) Artacho B, Savakis A (2020) Unipose: Unified human pose estimation in single images and videos. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 7035–7044
  * Baker et al. (2016) Baker B, Gupta O, Naik N, Raskar R (2016) Designing neural network architectures using reinforcement learning. arXiv preprint arXiv:161102167
  * Belagiannis and Zisserman (2017) Belagiannis V, Zisserman A (2017) Recurrent human pose estimation. In: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition (FG 2017), IEEE, pp 468–475
  * Bin et al. (2020) Bin Y, Cao X, Chen X, Ge Y, Tai Y, Wang C, Li J, Huang F, Gao C, Sang N (2020) Adversarial semantic data augmentation for human pose estimation. In: European conference on computer vision, pp 1–1
  * Buehler et al. (2011) Buehler P, Everingham M, Huttenlocher DP, Zisserman A (2011) Upper body detection and tracking in extended signing sequences. International journal of computer vision 95(2):180
  * Bulat and Tzimiropoulos (2016) Bulat A, Tzimiropoulos G (2016) Human pose estimation via convolutional part heatmap regression. In: European Conference on Computer Vision, Springer, pp 717–732
  * Cao et al. (2019) Cao X, Ge Y, Tai Y, Zhang W, Li J, Wang C, Li J, Huang F (2019) Anti-confusing: Region-aware network for human pose estimation. arXiv preprint arXiv:190500996
  * Carreira et al. (2016) Carreira J, Agrawal P, Fragkiadaki K, Malik J (2016) Human pose estimation with iterative error feedback. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 4733–4742
  * Charles et al. (2014) Charles J, Pfister T, Everingham M, Zisserman A (2014) Automatic and efficient human pose estimation for sign language videos. International Journal of Computer Vision 110(1):70–90
  * Chen and Zwicker (2021) Chen S, Zwicker M (2021) Transfer learning for pose estimation of illustrated characters. arXiv preprint arXiv:210801819
  * Chen et al. (2021) Chen W, Feng Z, Xiatian Z, Shuzhi SG (2021) Low-resolution human pose estimation. arXiv preprint arXiv:210909090
  * Chen and Yuille (2014) Chen X, Yuille AL (2014) Articulated pose estimation by a graphical model with image dependent pairwise relations. In: Advances in neural information processing systems, pp 1736–1744
  * Chen et al. (2017) Chen Y, Shen C, Wei XS, Liu L, Yang J (2017) Adversarial posenet: A structure-aware convolutional network for human pose estimation. In: Proceedings of the IEEE International Conference on Computer Vision, pp 1212–1221
  * Chen et al. (2018) Chen Y, Wang Z, Peng Y, Zhang Z, Yu G, Sun J (2018) Cascaded pyramid network for multi-person pose estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 7103–7112
  * Chen et al. (2020) Chen Y, Tian Y, He M (2020) Monocular human pose estimation: A survey of deep learning-based methods. Computer Vision and Image Understanding 192:102897
  * Cherian et al. (2014) Cherian A, Mairal J, Alahari K, Schmid C (2014) Mixing body-part sequences for human pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2353–2360
  * Chou et al. (2018) Chou CJ, Chien JT, Chen HT (2018) Self adversarial training for human pose estimation. In: 2018 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), IEEE, pp 17–30
  * Chu et al. (2016a) Chu X, Ouyang W, Li H, Wang X (2016a) Structured feature learning for pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4715–4723
  * Chu et al. (2016b) Chu X, Ouyang W, Wang X, et al. (2016b) Crf-cnn: Modeling structured information in human pose estimation. In: Advances in Neural Information Processing Systems, pp 316–324
  * Chu et al. (2017) Chu X, Yang W, Ouyang W, Ma C, Yuille AL, Wang X (2017) Multi-context attention for human pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1831–1840
  * Cireşan et al. (2010) Cireşan DC, Meier U, Gambardella LM, Schmidhuber J (2010) Deep, big, simple neural nets for handwritten digit recognition. Neural computation 22(12):3207–3220
  * Dang et al. (2019) Dang Q, Yin J, Wang B, Zheng W (2019) Deep learning based 2d human pose estimation: A survey. Tsinghua Science and Technology 24(6):663–676
  * Escalera et al. (2013) Escalera S, Gonzàlez J, Baró X, Reyes M, Lopes O, Guyon I, Athitsos V, Escalante H (2013) Multi-modal gesture recognition challenge 2013: Dataset and results. In: Proceedings of the 15th ACM on International conference on multimodal interaction, pp 445–452
  * Fan et al. (2015) Fan X, Zheng K, Lin Y, Wang S (2015) Combining local appearance and holistic view: Dual-source deep neural networks for human pose estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1347–1355
  * Felzenszwalb and Huttenlocher (2005) Felzenszwalb PF, Huttenlocher DP (2005) Pictorial structures for object recognition. International journal of computer vision 61(1):55–79
  * Ferrari et al. (2008) Ferrari V, Marin-Jimenez M, Zisserman A (2008) Progressive search space reduction for human pose estimation. In: 2008 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, pp 1–8
  * Fieraru et al. (2018) Fieraru M, Khoreva A, Pishchulin L, Schiele B (2018) Learning to refine human pose estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops, pp 205–214
  * Fischler and Elschlager (1973) Fischler MA, Elschlager RA (1973) The representation and matching of pictorial structures. IEEE Transactions on computers 100(1):67–92
  * Gkioxari et al. (2016) Gkioxari G, Toshev A, Jaitly N (2016) Chained predictions using convolutional neural networks. In: European Conference on Computer Vision, Springer, pp 728–743
  * Gong et al. (2016) Gong W, Zhang X, Gonzàlez J, Sobral A, Bouwmans T, Tu C, Zahzah Eh (2016) Human pose estimation from monocular images: A comprehensive survey. Sensors 16(12):1966
  * Gong et al. (2020) Gong X, Chen W, Jiang Y, Yuan Y, Liu X, Zhang Q, Li Y, Wang Z (2020) Autopose: Searching multi-scale branch aggregation for pose estimation. arXiv preprint arXiv:200807018
  * He et al. (2016) He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 770–778
  * Hou et al. (2020) Hou L, Cao J, Zhao Y, Shen H, Meng Y, He R, Ye J (2020) Augmented parallel-pyramid net for attention guided pose-estimation. In: European conference on computer vision, pp 1–1
  * Jain et al. (2013) Jain A, Tompson J, Andriluka M, Taylor GW, Bregler C (2013) Learning human pose estimation features with convolutional networks. arXiv preprint arXiv:13127302
  * Ji and Liu (2009) Ji X, Liu H (2009) Advances in view-invariant human motion analysis: a review. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews) 40(1):13–24
  * Johnson and Everingham (2010) Johnson S, Everingham M (2010) Clustered pose and nonlinear appearance models for human pose estimation. In: bmvc, Citeseer, vol 2, p 5
  * Johnson and Everingham (2011) Johnson S, Everingham M (2011) Learning effective human pose estimation from inaccurate annotation. In: CVPR 2011, IEEE, pp 1465–1472
  * Ke et al. (2018) Ke L, Chang MC, Qi H, Lyu S (2018) Multi-scale structure-aware network for human pose estimation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 713–728
  * Krizhevsky et al. (2012) Krizhevsky A, Sutskever I, Hinton GE (2012) Imagenet classification with deep convolutional neural networks. In: Advances in neural information processing systems, pp 1097–1105
  * Liang et al. (2018) Liang X, Gong K, Shen X, Lin L (2018) Look into person: Joint body parsing & pose estimation network and a new benchmark. IEEE transactions on pattern analysis and machine intelligence 41(4):871–885
  * Lifshitz et al. (2016) Lifshitz I, Fetaya E, Ullman S (2016) Human pose estimation using deep consensus voting. In: European Conference on Computer Vision, Springer, pp 246–260
  * Liu et al. (2018a) Liu H, Simonyan K, Yang Y (2018a) Darts: Differentiable architecture search. arXiv preprint arXiv:180609055
  * Liu et al. (2019) Liu H, Simonyan K, Yang Y (2019) DARTS: differentiable architecture search. In: International conference on learning representations, New Orleans, LA, USA
  * Liu et al. (2018b) Liu W, Chen J, Li C, Qian C, Chu X, Hu X (2018b) A cascaded inception of inception network with attention modulated feature fusion for human pose estimation. In: Thirty-Second AAAI Conference on Artificial Intelligence
  * Liu et al. (2020) Liu X, Maghlakelidze G, Zhou J, Izadi OH, Pommerenke D (2020) Detection of esd-induced soft failures by analyzing linux kernel function calls. IEEE Transactions on Device and Materials Reliability PP(99):1–1
  * Liu et al. (2015) Liu Z, Zhu J, Bu J, Chen C (2015) A survey of human pose estimation: the body parts parsing based methods. Journal of Visual Communication and Image Representation 32:10–19
  * Martin Arjovsky and Bottou (2017) Martin Arjovsky S, Bottou L (2017) Wasserstein generative adversarial networks. In: Proceedings of the 34 th International Conference on Machine Learning, Sydney, Australia
  * Moon et al. (2019) Moon G, Chang JY, Lee KM (2019) Posefix: Model-agnostic general human pose refinement network. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 7773–7781
  * Neumann and Vedaldi (2018) Neumann L, Vedaldi A (2018) Tiny people pose. In: Asian Conference on Computer Vision, Springer, pp 558–574
  * Newell et al. (2016) Newell A, Yang K, Deng J (2016) Stacked hourglass networks for human pose estimation. In: European conference on computer vision, Springer, pp 483–499
  * Nibali et al. (2018) Nibali A, He Z, Stuart M, Prendergast L (2018) Numerical coordinate regression with convolutional neural networks. CoRR abs/1801.07372
  * Nie et al. (2018) Nie X, Feng J, Zuo Y, Yan S (2018) Human pose estimation with parsing induced learner. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2100–2108
  * Ning et al. (2017) Ning G, Zhang Z, He Z (2017) Knowledge-guided deep fractal neural networks for human pose estimation. IEEE Transactions on Multimedia 20(5):1246–1259
  * Papandreou et al. (2017) Papandreou G, Zhu T, Kanazawa N, Toshev A, Tompson J, Bregler C, Murphy K (2017) Towards accurate multi-person pose estimation in the wild. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 4903–4911
  * Park et al. (2020) Park S, Lee Sb, Park J (2020) Data augmentation method for improving the accuracy of human pose estimation with cropped images. Pattern Recognition Letters 136:244–250
  * Peng et al. (2018) Peng X, Tang Z, Yang F, Feris RS, Metaxas D (2018) Jointly optimize data augmentation and network training: Adversarial data augmentation in human pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 2226–2234
  * Pfister et al. (2014) Pfister T, Simonyan K, Charles J, Zisserman A (2014) Deep convolutional neural networks for efficient pose estimation in gesture videos. In: Asian Conference on Computer Vision, Springer, pp 538–552
  * Pfister et al. (2015) Pfister T, Charles J, Zisserman A (2015) Flowing convnets for human pose estimation in videos. In: Proceedings of the IEEE International Conference on Computer Vision, pp 1913–1921
  * Pishchulin et al. (2012) Pishchulin L, Jain A, Andriluka M, Thormählen T, Schiele B (2012) Articulated people detection and pose estimation: Reshaping the future. In: 2012 IEEE Conference on Computer Vision and Pattern Recognition, IEEE, pp 3178–3185
  * Rafi et al. (2016) Rafi U, Leibe B, Gall J, Kostrikov I (2016) An efficient convolutional network for human pose estimation. In: BMVC, vol 1, p 2
  * Ruggero Ronchi and Perona (2017) Ruggero Ronchi M, Perona P (2017) Benchmarking and error diagnosis in multi-instance pose estimation. In: Proceedings of the IEEE international conference on computer vision, pp 369–378
  * Sapp and Taskar (2013) Sapp B, Taskar B (2013) Modec: Multimodal decomposable models for human pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3674–3681
  * Sarafianos et al. (2016) Sarafianos N, Boteanu B, Ionescu B, Kakadiaris IA (2016) 3d human pose estimation: A review of the literature and analysis of covariates. Computer Vision and Image Understanding 152:1–20
  * Saxena and Verbeek (2016) Saxena S, Verbeek J (2016) Convolutional neural fabrics. In: Advances in Neural Information Processing Systems, pp 4053–4061
  * Simonyan and Zisserman (2014) Simonyan K, Zisserman A (2014) Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv:14091556
  * Su et al. (2019) Su K, Yu D, Xu Z, Geng X, Wang C (2019) Multi-person pose estimation with enhanced channel-wise and spatial information. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 5674–5682
  * Sun et al. (2017a) Sun K, Lan C, Xing J, Zeng W, Liu D, Wang J (2017a) Human pose estimation using global and local normalization. In: Proceedings of the IEEE International Conference on Computer Vision, pp 5599–5607
  * Sun et al. (2019) Sun K, Xiao B, Liu D, Wang J (2019) Deep high-resolution representation learning for human pose estimation. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 5693–5703
  * Sun et al. (2017b) Sun X, Shang J, Liang S, Wei Y (2017b) Compositional human pose regression. In: Proceedings of the IEEE International Conference on Computer Vision, pp 2602–2611
  * Sun et al. (2018) Sun X, Xiao B, Wei F, Liang S, Wei Y (2018) Integral human pose regression. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 529–545
  * Tang and Wu (2019) Tang W, Wu Y (2019) Does learning specific features for related parts help human pose estimation? In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1107–1116
  * Tompson et al. (2015) Tompson J, Goroshin R, Jain A, LeCun Y, Bregler C (2015) Efficient object localization using convolutional networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 648–656
  * Tompson et al. (2014) Tompson JJ, Jain A, LeCun Y, Bregler C (2014) Joint training of a convolutional network and a graphical model for human pose estimation. In: Advances in neural information processing systems, pp 1799–1807
  * Toshev and Szegedy (2014) Toshev A, Szegedy C (2014) Deeppose: Human pose estimation via deep neural networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp 1653–1660
  * Wang et al. (2020) Wang J, Long X, Gao Y, Ding E, Wen S (2020) Graph-pcnn: Two stage human pose estimation with graph pose refinement. In: European Conference on Computer Vision, Springer, pp 492–508
  * Wang et al. (2018a) Wang P, Li W, Ogunbona P, Wan J, Escalera S (2018a) Rgb-d-based human motion recognition with deep learning: A survey. Computer Vision and Image Understanding 171:118–139
  * Wang et al. (2018b) Wang Z, Li W, Yin B, Peng Q, Xiao T, Du Y, Li Z, Zhang X, Yu G, Sun J (2018b) Mscoco keypoints challenge 2018. In: Joint Recognition Challenge Workshop at ECCV 2018, vol 5
  * Wei et al. (2016) Wei SE, Ramakrishna V, Kanade T, Sheikh Y (2016) Convolutional pose machines. In: Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pp 4724–4732
  * Xia et al. (2017) Xia F, Wang P, Chen X, Yuille AL (2017) Joint multi-person pose estimation and semantic part segmentation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 6769–6778
  * Xiao et al. (2018) Xiao B, Wu H, Wei Y (2018) Simple baselines for human pose estimation and tracking. In: Proceedings of the European conference on computer vision (ECCV), pp 466–481
  * Xu et al. (2020) Xu C, Yu X, Wang Z, Ou L (2020) Multi-view human pose estimation in human-robot interaction. In: IECON 2020 The 46th Annual Conference of the IEEE Industrial Electronics Society, IEEE, pp 4769–4775
  * Yang et al. (2020) Yang F, Chen Y, Pan Z, Zhang M, Xue M, Mo Y, Zhang Y, Guan G, Qian B, Xiao Z, et al. (2020) Train your data processor: Distribution-aware and error-compensation coordinate decoding for human pose estimation. arXiv preprint arXiv:200705887
  * Yang et al. (2019) Yang S, Yang W, Cui Z (2019) Pose neural fabrics search. arXiv preprint arXiv:190907068
  * Yang et al. (2016) Yang W, Ouyang W, Li H, Wang X (2016) End-to-end learning of deformable mixture of parts and deep convolutional neural networks for human pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3073–3082
  * Yang et al. (2017) Yang W, Li S, Ouyang W, Li H, Wang X (2017) Learning feature pyramids for human pose estimation. In: proceedings of the IEEE international conference on computer vision, pp 1281–1290
  * Yang and Ramanan (2011) Yang Y, Ramanan D (2011) Articulated pose estimation with flexible mixtures-of-parts. In: CVPR 2011, IEEE, pp 1385–1392
  * Zhang et al. (2019a) Zhang F, Zhu X, Ye M (2019a) Fast human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 3517–3526
  * Zhang et al. (2020) Zhang F, Zhu X, Dai H, Ye M, Zhu C (2020) Distribution-aware coordinate representation for human pose estimation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp 7093–7102
  * Zhang et al. (2019b) Zhang H, Ouyang H, Liu S, Qi X, Shen X, Yang R, Jia J (2019b) Human pose estimation with spatial contextual information. arXiv preprint arXiv:190101760
  * Zhang et al. (2019c) Zhang P, Lan C, Xing J, Zeng W, Xue J, Zheng N (2019c) View adaptive neural networks for high performance skeleton-based human action recognition. IEEE transactions on pattern analysis and machine intelligence 41(8):1963–1978
  * Zhang et al. (2019d) Zhang R, Zhu Z, Li P, Wu R, Guo C, Huang G, Xia H (2019d) Exploiting offset-guided network for pose estimation and tracking. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops
  * Zheng et al. (2019) Zheng L, Huang Y, Lu H, Yang Y (2019) Pose-invariant embedding for deep person re-identification. IEEE Transactions on Image Processing 28(9):4500–4509
  * Zoph and Le (2016) Zoph B, Le QV (2016) Neural architecture search with reinforcement learning. arXiv preprint arXiv:161101578