# An Adversarial Human Pose Estimation Network Injected with Graph Structure

Lei Tian111The first two authors contribute equally to this paper. Peng
Wang222The first two authors contribute equally to this paper. Guoqiang
Liang333Corresponding author: Guoqiang Liang<EMAIL_ADDRESS>Chunhua Shen
School of Computer Science, Northwestern Polytechnical University, Xi’an,
China  National Engineering Laboratory for Integrated Aero-Space-Ground-Ocean
Big Data Application Technology, China  School of Computer Science, The
University of Adelaide, Australia

###### Abstract

Because of the invisible human keypoints in images caused by illumination,
occlusion and overlap, it is likely to produce unreasonable human pose
prediction for most of the current human pose estimation methods. In this
paper, we design a novel generative adversarial network (GAN) to improve the
localization accuracy of visible joints when some joints are invisible. The
network consists of two simple but efficient modules, _i.e_., Cascade Feature
Network (CFN) and Graph Structure Network (GSN). First, the CFN utilizes the
prediction maps from the previous stages to guide the prediction maps in the
next stage to produce accurate human pose. Second, the GSN is designed to
contribute to the localization of invisible joints by passing message among
different joints. According to GAN, if the prediction pose produced by the
generator $G$ cannot be distinguished by the discriminator $D$, the generator
network $G$ has successfully obtained the underlying dependence of human
joints. We conduct experiments on three widely used human pose estimation
benchmark datasets, _i.e_., LSP, MPII and COCO, whose results show the
effectiveness of our proposed framework.

###### keywords:

Human pose estimation , Cascade feature network , Graph structure network ,
Generative adversarial network

††journal: Journal of LaTeX Templates

##  1 Introduction

Human pose estimation refers to predict the specific location of human
keypoints from an image. It is a fundamental yet challenging task for many
computer vision applications like intelligent video surveillance and human-
computer interaction. However, due to illumination, occlusion and flexible
body poses, human pose estimation is still difficult, especially for locating
invisible keypoints.

In recent years, due to the development of deep convolutional neural networks
(CNNs) [1, 2], there has been significant progress on the problem of human
pose estimation. Newly proposed methods mainly use CNNs to predict the
heatmaps of body joints. However, it will be very tough for purely CNN-based
methods to regress accurate heatmaps in the face of severely occluded body
parts, which are caused by surrounding people or backgrounds that are very
similar to body parts.

In contrast, people can always get accurate results for occluded joints by
resorting to the local features and surrounding information in the image. For
example, after getting the position of a knee joint, we can infer the position
of corresponding ankle joint on the basis of the connection between knees and
ankles, and vice versa. Moreover, even for extremely occluded body, we can
distinguish reasonable and unreasonable poses and infer potential poses.
During the process of inferring the location of invisible joints, the
dependence correlation of human body joints has played an important role,
which is also pointed out in [3]. However, the complexity of pose estimation
methods will become much higher if we directly add a new module to capture
joints dependence.

Considering recent success of Generative Adversarial Networks (GANs) [4], we
propose a GAN based human pose estimation framework to explore the dependence
relationship among different joints while maintaining the original inference
complexity. Specifically, since the internal connection of body joints can be
regarded as a graph structure, we design a new graph structure network based
on the graph neural network. The pose heatmaps from the generator $G$ are used
as the input of pose discriminator $D$, whose goal is to determine whether the
input pose is geometrically reasonable. In learning, the generator based CFN
is trained to deceive the discriminator that its results are all reasonable.
In other words, the generator is guided to learn the dependence correlation of
the human body joints integrated in $D$ network. Once converged, the generator
$G$ will successfully learn the internal dependence of body joints. To
evaluate the proposed approach, we conduct experiments on three public
datasets for human pose estimation, _i.e_., LSP, MPII and COCO. The
experimental results show that our method has obtained more plausible pose
predictions compared to previous methods.

In a word, our main contributions can be summarized as follows:

(i) We propose to use graph structure to model the relationship of body
joints. Specifically, we regard the nodes in graph structure as human joints
representation and the edges as the dependence relations of human joints.
Through message pass among nodes, the complex relations of joints can be
captured.

(ii) Our model is based on adversarial learning and can be trained end-to-end.
Since the discriminator is not needed during testing, the complexity of pose
estimation does not increase. Experiments on three public datasets show the
effectiveness of our method.

##  2 Related Work

This section will review some related works on human pose estimation, graph
structure network and generative adversarial network.

###  2.1 Human Pose Estimation

Along with the introduction of CNN, human pose estimation based on CNN [5, 6]
has attracted an increasing attention. The mainstream methods for body joints
detection can be divided into two kinds. One is directly regressing joints’
locations [7, 8] while the other is producing joints heatmaps, where the
location with the maximum value is selected as the predicted results [9, 10].
DeepPose, one of the earliest CNN-based method proposed by Toshev _et al_.
[11], regresses location of joints with a two-steps CNN. In order to further
improve performance, Fan _et al_. [7] explore the relationship between global
features and local features. To encompass both input and output spaces,
Carreira _et al_. [8] iteratively refine prediction results by concatenating
the image with the body joint predictions from the previous steps. Tompson _et
al_. [9] generate heatmaps by using the multiple resolution convolutional
networks to fuse the features, and employ the post-processing by using Markov
Random Field (MRF). Due to its excellent performance, hourglass Network [12]
has become a main framework, which is a bottom-up and top-down architecture
with residual blocks. SimpleBaseline Network [1] estimates heatmaps by
integrating a few deconvolutional layers on a backbone network in a much
simpler way. Carissimi _et al_. [13] consider missing joints as noise in the
input and use an autoencoder-based solution to enhance the pose prediction.
Zhao _et al_. [14] design an occlusion aware network to predict both heatmaps
and visible values of joints simultaneously, so as to grasp the degree of
occlusion of a pose and set a much fairer score.

Like the SimpleBaseline network, our generator $G$ is also a fully
convolutional network with “conv-deconv” architecture. However, our network
integrates the multi-stage features of the backbone network into the
deconvolutional layers through skip layers connection. Here, we use the ResNet
as our backbone. However, our method is not restricted to the ResNet. And
newly proposed network architecture can be easily used to further improve the
performance.

###  2.2 Graph Structure Network

The dependence correlation of human joints is very important in human pose
estimation. In most of traditional methods for human pose estimation,
graphical model, especially a tree-structure model is used to capture the
relationship among human joints [15]. Recently, this graphical model is
combined with convolution neural networks. Chu _et al_. [16] propose a
structured feature learning model to capture the relation of joints on feature
maps. Zhao _et al_. [17] design a new method which operates on regression
tasks with graph-structured data to predict 3D human pose from 2D joints as
well as image features. Liu _et al_. [18] design the Graphical ConvLSTM to
enable the intermediate convolutional feature maps to perceive the graphical
long short-term dependency among different body parts.

To process the graph-structured data, Graph Neural Network (GNN) is first
proposed in [19]. Different from CNN, which is suitable for the data in the
Euclidean space, GNN mainly deals with the data from non-Euclidean domains.
Current GNN methods generally follow two directions. One direction is the
spectral domain [20, 21], where the neural networks are applied to the whole
graph. The other line is spatial domain [22, 3], where the neural networks are
applied recurrently on each node of the graph. The state of each node is
updated according to its previous state and information aggregating from
neighboring nodes. Recently, the Gated Graph Neural Network (GGNN) [23] has
been widely used in some computer vision tasks. The GGNN adopts Gated
Recurrent Units [24] and unrolls the recurrence for a fixed number of steps
$T$ to update the hidden state of the nodes. The final GGNN computes gradients
by using the backpropagation through temporal algorithm. Compared with [3],
our work is based on GAN and the complexity of inference does not increase.

###  2.3 Generative Adversarial Network

Generative Adversarial Networks(GANs) [4] have been widely studied in many
computer vision tasks, such as image super-resolution [25], image inpainting
[26], style transfer [27], and human pose estimation [28, 29]. For example,
RepNet [28] uses an adversarial training method to learn a mapping from a
distribution of 2D poses to a distribution of 3D poses. Peng _et al_. [29]
apply adversarial network to human pose estimation to jointly optimize data
augmentation and network training. Human pose estimation can been regarded as
a image-to-heatmap translation problem, which can be well implemented by our
proposed framework.

Different from previous GANs, we use the internal structure of human joints to
construct a graph and incorporate GGNN into the discriminator model to
distinguish the reasonable poses from unreasonable prediction results.

Figure 1: The overall framework of proposed method. The upper part is the
generator with CFN and the lower part is the discriminator with GSN.

##  3 Our Method

Our human pose estimation framework consists of two parts: a generator and a
discriminator, which are shown in upper part and bottom part of Figure 1
respectively. Our generator is a cascade feature network, which aims at
regressing pose heatmaps in a skip layer connection way. Specifically, it
takes a RGB image as input and produce a heatmap for each joint. Each value in
a heatmap of a joint denotes the likelihood that the corresponding joint
locates at that coordinate. Therefore, we take the coordinate corresponding to
the maximum value in a heatmap as the final location of this joint. The size
of predicted heatmaps is $N\times H\times W$, where $N$, $H$, $W$ represent
respectively the channel size, the height and the width. The specific value of
$N$ depends on the dataset. For MPII and LSP, $N$ is equal to $16$ and $14$,
respectively. And due to the pooling, the height and width of heatmaps is a
quarter of the original image size. In other words, if the size of images is
$256\times 256$, $H$ and $W$ will be 64.

Generally, the generator network can be trained like a normal deep convolution
network. However, it is possible to generate human pose estimations with very
low confidence or even wrong estimation results. Therefore, we introduce the
discriminator network $D$ into the whole framework to make the generator
network correct poor estimations. To capture the internal structure dependence
of human body, we design a graph neural network based on the gated graph
neural network. Its inputs are heatmaps generated by the generator network or
ground-truth heatmaps. The discriminator will produce a $N$-dimension vector
in the range of $[$0,1$]$, whose value represents the joint localization
quality of an input. Through adversarial learning between $G$ and $D$, the
generator network $G$ will learn the dependence of human joints and output
better pose prediction.

###  3.1 Cascade Feature Network (CFN)

Inspired by the successful idea in the previous human pose estimation [30, 1],
we also integrate cascade features through an encoding-decoding process. In
the encoding process, features generated by different layers have different
semantic levels. As described in [26], lower level features focus on local
semantic information while the upper level features focus on global semantic
information. If we integrate these heatmaps to the decoding process, it will
help to distinguish different body joints so as to improve the accuracy of
final prediction.

In order to effectively combine features at different semantic levels, we
utilize a cascade feature network (CFN) shown in the upper part of Figure 1.
Specifically, a $1\times 1$ convolution layer with the step size of 1 and
without padding is used to change the channel of coarse prediction feature
maps generated in the encoding stage. Then, the resulted features are combined
with the corresponding image features in the decoding stage by element-wise
addition. The combined feature maps are sent to the next stage of decoding.
The final predicted results are obtained through the last $1\times 1$
convolution layer, which is used as a linear classifier to map features to
heatmaps.

Considering that the visibility of each human joint in images is different, we
employ the following loss function for CFN:

$L_{\mbox{G}}=\frac{1}{N}\sum_{n=1}^{N}v_{n}\big{\|}X_{n}-Y_{n}\big{\|},$ (1)

where $\big{\|}\cdot\big{\|}$ represents the Euclidean distance. $N$,
$v_{n}\in{\big{\\{}0,1\big{\\}}}$ represent respectively the number of joints
and the visibility of the $n$-th pose joint. $X_{n}\in\mathbb{R}^{64\times
64}$, $Y_{n}\in\mathbb{R}^{64\times 64}$ represent respectively the predicted
heatmap and the ground-truth heatmap for the $n$-th joint.

###  3.2 Graph Neural Network (GNN)

Before detailing the specific architecture of discriminator, we first
introduce graph neural network (GNN), which is the most important module in
our discriminator. GNN is more and more used in computer vision, especially in
processing the data with graph structure. Obviously, the human keyjoints can
be regarded as a natural graph structure. Therefore, we adopt a graph
structure to effectively integrate the internal dependence of pose joints.

Here, we introduce our constructed GNN for human pose estimation. The input of
GNN is a graph, which can be expressed as $Graph=\big{\\{}V,E\\}$, where $V$
and $E$ represent the set of nodes and edges in a graph respectively. Each
node $v\in V$ represents a body joint while an edge represents the
relationship between two nodes. Moreover, a node is associated with a hidden
state vector $i_{n}$, which is gradually updated in the GNN. When updating the
hidden state vector of node $v$ at time step $t$, the input is the combination
of its current state vector $i_{n}^{t}$ and the information $j_{n}^{t}$ passed
from its neighboring nodes $M_{n}$. The overall process of GNN can be
expressed as:

$\displaystyle j_{n}^{t}$ $\displaystyle={\mbox{F}}(i_{m}^{t-1}\mid m\in
M_{n}),$ (2) $\displaystyle i_{n}^{t}$
$\displaystyle={\mbox{H}}(i_{n}^{t-1},j_{n}^{t}),$ (3)

where $F$ and $H$ represent respectively the function to collect information
from neighboring nodes and to update hidden state information of a node.

Graph Construction. The node connection of a graph determines information of
which nodes will be collected for a specific node. Intuitively, it is simple
and convenient to construct the graph for pose estimation in the way of full
connection to pass information among nodes with one step, but this will
destroy the internal spatial connection of body joints. Therefore, we build a
tree-based graph structure to retain the relationship of neighboring joints as
shown in Figure 1. And note that the information passing in this graph is
bidirectional.

Graph Propagation. Before learning the current hidden state of each node, it
needs to obtain the information passed from the neighboring nodes by using the
function $F$. We formulate the collecting process as following:

$j_{n}^{t}=\sum_{n^{{}^{\prime}}\in\Omega_{n}}W_{n,n^{{}^{\prime}}}i_{n}^{t-1},$
(4)

where $\Omega_{n}$ is a set of all neighboring nodes of the $n$-th node,
$W_{n,n^{{}^{\prime}}}$ is the convolution weight of the edge between nodes
$n$ and $n^{{}^{\prime}}$.

Next we need to update the state vector. In this paper, we adopt the gated
graph neural network with GRU [23] to fully explore the dependence information
of body joints. Specifically, after aggregating the information from
neighboring nodes of the $n$-th node, the hidden state of $n$-th node is
updated through GRU. The whole process can be represented as follows:

$\displaystyle z_{n}^{t}$
$\displaystyle=\sigma(W_{n}^{z}j_{n}^{t}+U_{n}^{z}i_{n}^{t-1}),$ (5)
$\displaystyle r_{n}^{t}$
$\displaystyle=\sigma(W_{n}^{r}j_{n}^{t}+U_{n}^{r}i_{n}^{t-1}),$
$\displaystyle\tilde{h}_{n}^{t}$
$\displaystyle=tanh(W_{n}^{h}j_{n}^{t}+U_{n}^{h}(r_{n}^{t}\odot
i_{n}^{t-1})),$ $\displaystyle i_{n}^{t}$ $\displaystyle=(1-z_{n}^{t})\odot
i_{n}^{t-1}+z_{n}^{t}\odot\tilde{h}_{n}^{t},$

where $W_{n}^{z},W_{n}^{r},W_{n}^{h},U_{n}^{z},U_{n}^{r},U_{n}^{h}$ are the
convolution weights to be learned in the graph network, $\sigma$ denotes the
$sigmoid$ function.

###  3.3 Discriminator Network

When some body parts are seriously occluded, it will be difficult to produce a
reasonable prediction only through the network in the section 3.1. However,
human can get more accurate results by observing the surrounding joints and
local features of the image. In order to further dig out the dependence
relationship of human body joints, we design a discriminator $D$ as shown in
the bottom part of Figure 1. The function of discriminator $D$ is to
distinguish the right poses from the wrong poses, which do not satisfy the
geometric relationship of human body. In contrast to previous work [31], we
apply the gated graph neural network in discriminator $D$. As described in the
section 3.2, graph neural network can infer more reasonable joint information
by using the internal dependence of human body joints.

Here, the discriminator $D$ takes the pose heatmap as input, which can be the
results predicted from the generator or the ground-truth. The discriminator
aims at predicting whether the pose is reasonable or not. Specifically, the
output of $D$ is a $N\times 1$ vector, each value of which represents the
rationality of corresponding body joint location in the input. Therefore, the
output value of $D$ is in the range of $[$0,1$]$. The closer the value of the
vector is to $1$, the more reasonable the body joint location is, and vice
versa. The objective function for learning discriminator $D$ is defined as
follows:

$\displaystyle\quad
L_{\mbox{D}}=\mathbb{E}[\mathrm{log}D(Y)]+\mathbb{E}[\mathrm{log}(1-D(X))],$
(6)

where $\mathbb{E}$ is the BCE loss. $D(X)$ and $D(Y)$ represent respectively
the output of $D$ when given the predicted heatmaps $X$ and ground-truth
heatmaps $Y$.

###  3.4 Training Details

This section will give the training process of our overall framework. Unlike
some GAN based methods, we do not pretrain $G$ or $D$. After randomly
initializing all the parameters, we train $G$ and $D$ alternatively following
the general training process of GAN. Specifically, $G$ is trained for $3$
times then $D$ is trained for $1$ time. In the process of training $D$, we
take the ground-truth heatmaps $Y$ as the input of $D$ and train $D$ to learn
this is the reals. At the same time, we take the heatmaps $X$ generated by $G$
as the input of $D$ and train $D$ to learn this is the fakes. In the process
of training $G$, $G$ is directly optimized to deceive $D$ through GAN. In
other words, the $D$ will deem the heatmaps generated by current $G$ as the
reals. Following the previous method [31], we mix the L2 loss of the generator
with the BCE loss of the discriminator as the whole loss function. Therefore,
the final objective function is defined as follows:

$\arg\min_{G}\max_{D}L_{G}+\alpha L_{D},$ (7)

where $\alpha$ is a hyper parameter to balance the L2 loss and the BCE loss.
It is set to $0.01$ through experiments. Obviously, if $\alpha$ is equal to
$0$, it means that the discriminator is not used to supervise the learning of
$G$.

##  4 Experiments

Datasets. To evaluate the proposed method, we conduct experiments on three
public benchmark datasets for human pose estimation, _i.e_., extended Leeds
Sports Poses (LSP) [32], MPII human pose [33] and COCO Keypoint Challenge
[34]. The extended LSP dataset contains 11000 training images and 1000 testing
images. For the MPII dataset, there are around 25000 images with 40000
annotated human poses, where 28000 human poses are used to train and 11000
human poses are used to test. Following the previous methods [31, 3], we also
use the same setting to divide the MPII training poses into training set and
validation set. Our models are only trained on COCO train2017 dataset
(includes $57000$ images and $150000$ person instances) with no extra data
involved, validation is studied on the val2017 set(includes $5000$ images).

Implementation Details. Our proposed model is implemented in Pytorch $1.0.0$.
All experiments are run on a server with an Nvidia GEFORCE GTX 1080Ti GPU. We
use Adam as the optimizer. For both benchmarks, the batch size is set to $64$
and the maximum epoch number is $140$. The learning rate is initialized to
$0.001$ and reduced by $10$ times at $90$-th and $130$-th epochs.

We crop the region of interest of each image based on the annotated person
location from the dataset and warp it to the size of $256\times 256$ for fair
comparison with other methods. Data augmentation are adopted during training,
such as random horizontal flipping, random rotation ($\pm 30$ degrees) and
random scaling ($0.75$-$1.25$). Because the LSP contains less training data,
we augment the training data of LSP with the MPII training images following
[35, 31]. The backbone network is ResNet-$50$ if not specially indicated in
our experiments.

Evaluation Criteria. We evaluate experimental results for the LSP dataset by
using the Percentage Correct Keypoints (PCK) [15], which represents the
percentage of correct detection. A joint location is believed to be correct if
it falls within a normalized distance of the ground-truth. For the MPII
dataset, we adopt the PCKh score [33], which uses a fraction of the head size
to normalize the distance. For the COCO dataset, we use the official
evaluation metric [34] that reports the OKS-based AP (average precision) in
the experiments, where the OKS (object keypoints similarity) is calculated
from the distance between predicted joints and ground truth joints normalized
by scale of the person.

Table 1: The Comparison of PCK @0.2 on the LSP dataset.

Model | Head | Sho | Elb | Wri | Hip | Kne | Ank | Mean
---|---|---|---|---|---|---|---|---
Belagiannis _et al_. [36] | 95.2 | 89.0 | 81.5 | 77.0 | 83.7 | 87.0 | 82.8 | 85.2
Lifshitz _et al_. [37] | 96.8 | 89.0 | 82.7 | 79.1 | 90.9 | 86.0 | 82.5 | 86.7
Pishchulin _et al_. [38] | 97.0 | 91.0 | 83.8 | 78.1 | 91.0 | 86.7 | 82.0 | 87.1
Insafutdinov _et al_. [35] | 97.4 | 92.7 | 87.5 | 84.4 | 91.5 | 89.9 | 87.2 | 90.1
Yang _et al_. [39] | 98.3 | 94.5 | 92.2 | 88.9 | 94.4 | 95.0 | 93.7 | 93.9
Zhang _et al_. [3] | 98.4 | 94.8 | 92.0 | 89.4 | 94.4 | 94.8 | 93.8 | 94.0
Our Model | 97.8 | 95.5 | 94.4 | 92.9 | 94.7 | 95.6 | 94.3 | 95.1

###  4.1 Quantitative Results

Results on LSP. Table 1 lists the comparison between our method and previous
methods, which are evaluated with PCK scores at a normalized distance of
$0.2$. Following previous methods [39], the MPII training set is also added to
the training set of extended LSP. From this table, we can see that our
proposed approach outperforms all the previous methods. Compared with the
previous best method [3], the performance improvement is $1.1\%$. In
particular, for some more challenging body parts such as wrist, ankle, we
achieves $3.5\%$, and $0.5\%$ improvement compared to this method
respectively.

Table 2: The Comparison of PCKh @0.5 on the MPII validation set.

Model | Backbone | Head | Sho | Elb | Wri | Hip | Kne | Ank | Mean
---|---|---|---|---|---|---|---|---|---
Hourglass [12] | $-$ | 96.0 | 96.3 | 90.3 | 85.4 | 88.8 | 85.0 | 81.9 | 89.2
SimpleBaseline [1] | ResNet-$152$ | 97.0 | 95.9 | 90.3 | 85.0 | 89.2 | 85.3 | 81.3 | 89.6
LFP [39] | $-$ | 96.8 | 96.0 | 90.4 | 86.0 | 89.5 | 85.2 | 82.3 | 89.6
DLCM [40] | $-$ | 95.6 | 95.9 | 90.7 | 86.5 | 89.9 | 86.6 | 82.5 | 89.8
AL [41] | $-$ | 96.5 | 96.0 | 90.5 | 86.0 | 89.2 | 86.8 | 83.7 | 89.9
PGCN [42] | ResNet-$50$ | $-$ | $-$ | $-$ | 83.6 | $-$ | $-$ | 80.8 | 88.9
Our Model | ResNet-$50$ | 96.4 | 95.3 | 89.1 | 84.2 | 89.1 | 84.8 | 80.4 | 89.0
Our Model | ResNet-$152$ | 97.1 | 95.9 | 90.4 | 85.1 | 89.1 | 85.8 | 81.5 | 89.8

Results on MPII. This subsection compares the performance of the MPII dataset
between our method and some previous methods. The PCKh score at a normalized
distance of $0.5$ is used to evaluate the performance. For simplicity, we
directly use the PCKh scores in [10]. Table 2 shows the final results on MPII
validation set. As shown in this table, our method obtains the PCKh score of
$89.8\%$ on the MPII validation set, which is comparable to previous methods.

Table 3: Comparison with the 8-stage Hourglass [12], CPN [30], SimpleBaseline
[1], ECSN [43] and PGCN [42] on the COCO val2017 dataset. Their results are
cited from [30, 1, 43, 42]. OHKM means the model training with the Online Hard
Keypoints Mining.

Method |  Backbone |  Input Size |  OHKM |  AP
---|---|---|---|---
$8$-stage Hourglass |  $-$ |  $256\times 192$ |  $\times$ |  66.9
$8$-stage Hourglass |  $-$ |  $256\times 256$ |  $\times$ |  67.1
CPN |  ResNet-$50$ |  $256\times 192$ |  $\times$ |  68.6
CPN |  ResNet-$50$ |  $384\times 288$ |  $\times$ |  70.6
CPN |  ResNet-$50$ |  $256\times 192$ |  $\checkmark$ |  69.4
CPN |  ResNet-$50$ |  $384\times 288$ |  $\checkmark$ |  71.6
SimpleBaseline |  ResNet-$50$ |  $256\times 192$ |  $\times$ |  70.4
SimpleBaseline |  ResNet-$50$ |  $384\times 288$ |  $\times$ |  72.2
ECSN |  ResNet-$50$ |  $256\times 192$ |  $\checkmark$ |  72.1
ECSN |  ResNet-$50$ |  $384\times 288$ |  $\checkmark$ |  73.8
PGCN |  ResNet-$50$ |  $256\times 192$ |  $\times$ |  71.1
PGCN |  ResNet-$50$ |  $384\times 288$ |  $\times$ |  72.9
SimpleBaseline with CFN |  ResNet-$50$ |  $256\times 192$ |  $\times$ |  72.2
Our overall method |  ResNet-$50$ |  $256\times 192$ |  $\times$ |  72.8
SimpleBaseline with CFN |  ResNet-$50$ |  $384\times 288$ |  $\times$ |  73.8
Our overall method |  ResNet-$50$ |  $384\times 288$ |  $\times$ |  74.4

Results on COCO. Table 3 shows our result with the 8-stage Hourglass [12], CPN
[30], SimpleBaseline [1], ECSN [43] and PGCN [42] on COCO validation set. The
human detection AP reported in Hourglass, CPN, ECSN and PGCN is $55.3$. The
human detection AP of ours is the same $56.4$ as SimpleBaseline. Compared with
Hourglass [12], both methods use an input size of $256\times 192$ and no
Online Hard Keypoints Mining(OHKM) involved, our method has an improvement of
$5.9$ AP. CPN [30], SimpleBaseline [1], ECSN [43], PGCN [42] and our method
use the same backbone of ResNet-$50$. When OHKM is not used, our method
outperforms CPN by $4.2$ AP for the input size of $256\times 192$ and $3.8$ AP
for the input size of $384\times 288$. Compared with SimpleBaseline, our
method outperforms $2.4$ AP for the input size of $256\times 192$ and $2.2$ AP
for the input size of $384\times 288$. When OHKM is used in CPN, our method
outperforms CPN by $2.8$ AP for both input sizes. Compared with ECSN, our
method outperforms $0.7$ AP for the input size of $256\times 192$, and $0.6$
AP for the input size of $384\times 288$. Compared with PGCN, our method
outperforms $1.7$ AP for the input size of $256\times 192$, and $1.5$ AP for
the input size of $384\times 288$. Therefore, we can conclude that our method
has comparable results but is simpler.

Figure 2: Sample estimation results on the MPII test set. The first row:
original images. The second row: results by SimpleBaseline [1]. The third row:
results by our overall method. (a) and (b) represent results for different
poses. Figure 3: More examples show our approach on the MPII test set. The
first row: results by SimpleBaseline [1]. The second row: results by
SimpleBaseline with CFN. The third row: results by our overall method. Figure
4: Failure examples on the MPII test set. The first row: results by
SimpleBaseline [1]. The second row: results by our overall method.

###  4.2 Qualitative Comparisons

In this section, we give the qualitative comparison between our method and the
SimpleBaseline [1]. Figure 2 provides some examples of our model and
SimpleBaseline on the MPII test set. We can see that it is difficult for the
SimpleBaseline to generate reasonable body poses with large deformation and
occlusion. However, our method gets more reasonable human pose. This may be
because of the semantic information and the internal dependence of body joints
learned in the adversarial training process. For example, the associated knee
joints can help to infer the precise location of ankles in our model as shown
in col. 4-6 of Figure 2.

In (a) of Figure 2, SimpleBaseline produces some weird joints’ location,
especially for the joints on body limbs. Although SimpleBaseline has learned
most of features about body joints, it may locate some body joints to the
surrounding area rather than the right one without the joints dependence
learned. In (b) of Figure 2, some body limbs are invisible due to self-
occlusion and other-occlusion. In these examples, SimpleBaseline fails to
produce reasonable poses. In contrast, our method successfully predicts the
correct body position even under some difficult situations. This may be
because of the graph structure introduced into the discriminator.

Figure 3 shows more examples generated on the MPII test set. In (a) of Figure
3, although adding CFN module to the baseline can reduce the number of the
weird joints’ location generated by the baseline, there are still some wrong
joints’ location. And adding overall method to the baseline may locate some
body joints to the right one with the joints dependence learned. In (b) of
Figure 3, when some body limbs are invisible due to self-occlusion and other-
occlusion, adding CFN to baseline can not solve the problem of the weird
joints’ location generated by baseline. On the contrary, our method still
solves this problem very well.

Figure 4 shows some failure examples generated by our method on the MPII test
set. As shown in Figure 4, our method may fail in some very challenging
examples with larger deformation, overlap and occlusion. In some examples,
human can not recognize the right pose at a glance. Even if our method fails
in these cases, it gets a more reasonable poses than SimpleBaseline.

Table 4: Effect of each component of our method on the MPII validation set.

Model | CFN | Discriminator | GSN | Mean PCKh
---|---|---|---|---
SimpleBaseline [1] | $\times$ | $\times$ | $\times$ | 88.53
SimpleBaseline [1] with CFN | $\checkmark$ | $\times$ | $\times$ | 88.74
SimpleBaseline [1] with a Discriminator | $\times$ | $\checkmark$ | $\times$ | 88.64
SimpleBaseline [1] with GSN | $\times$ | $\checkmark$ | $\checkmark$ | 88.80
Our overall method | $\checkmark$ | $\checkmark$ | $\checkmark$ | 89.02

###  4.3 Ablation Study

To show the influence of each component of our method, we conduct ablation
study. Since the ground-truth location of MPII test part is not available, we
report the PCKh 0.5 score on the validation set of MPII dataset. In all
experiments, the flip test is used.

Cascade Feature Network. To evaluate the effect of CFN, we conduct two
experiments using SimpleBaseline with and without a CFN. Both of the two
models are trained only using the same L2 loss defined in Eq. (1). In other
words, there is no discriminators (_i.e_., no GAN). As shown in the first two
rows of Table 4, the mean PCKh score on the MPII validation set is improved by
$0.21\%$ compared to the SimpleBaseline model. We think this is caused by
gathering the feature of previous layers of the ResNet-$50$ to the
deconvolutional layers. This manifests that the cascade feature assists the
network in better understanding of the poses.

Graph Structure Network. In this paper, we design a graph structure in the
discriminator. So a question is the performance improvement is from GAN or the
graph structure. To answer this question, we conduct an experiment only using
a discriminator without graph structure. In detail, the discriminator consists
of only two fully connected layers. Its PCKh score is shown in the third row
of Table 4. From this table, we can find the score is increased by $0.11\%$
compared to the SimpleBaseline model. This shows the performance can be
improved by adding a simple discriminator to the SimpleBaseline. Moreover, we
also investigate the effect of discriminator including the GSN. As shown in
the Table 4, the PCKh on MPII validation set is 88.80%, which is respectively
$0.27\%$ and $0.16\%$ higher than that of SimpleBaseline model and
SimpleBaseline with a discriminator. This indicates that the discriminator
with GSN is more helpful to promote the performance. In our mind, this is due
to that the GSN can fully explore the internal geometric dependence of pose
joints and reduce unreasonable pose estimation.

Figure 5: PCKh score of the wirst, hip, knee and ankle joints for different
combination of components in our method.

According to the above, adding CFN or a discriminator with GSN separately in
the SimpleBaseline can improve the accuracy of human pose estimation. As shown
in the last line of Table 4, using them simultaneously results in an increase
of $0.5\%$, which is higher than only using one module.

Moreover, we also compare the PCKh score of wrist, hip, knee, ankle, whose
localization difficulty is the largest in all human joints due to occlusion
and higher freedom. As shown in Figure 5, for these four parts, our method
achieves $0.99\%$, $0.67\%$, $0.83\%$ and $0.76\%$ improvement compared with
the SimpleBaseline [1] respectively. This validates that the GSN capturing
dependence of human joints can help to alleviate the occlusion problem.

Table 5: The compare the effect of the hyper parameter $\alpha$ on the
performance on the MPII validation set.

$\alpha$ |  $0.01$ |  $0.1$ |  $0.5$ |  $1$
---|---|---|---|---
Mean |  89.02 |  88.83 |  88.79 |  88.51

We also compare the effect of the hyper parameter (_i.e_., variable $\alpha$)
on the performance on the MPII validation set. As shown in Table 5, the
performance reaches the best at $\alpha$ = $0.01$. It begins to degrade with
the increase of $\alpha$. So, we set $\alpha$ to $0.01$.

Table 6: The Analysis of the invisible human joints on the MPII validation
set.

Model | Num. of Invisible Joints | PCKh of Wrist | PCKh of Ankle | Mean
---|---|---|---|---
SimpleBaseline [1] |  $\geq$ 2 | 85.27 | 76.21 | 90.57
Our overall method |  $\geq$ 2 | 86.11 | 77.67 | 90.92
SimpleBaseline [1] |  $\geq$ 4 | 85.38 | 66.27 | 91.31
Our overall method |  $\geq$ 4 | 87.11 | 72.62 | 91.74

Analysis of the Invisible Human Joints. In order to further show the effect of
our method on the prediction of seriously occluded pose, we subdivide the
validation set according to the number of invisible joints. Specifically, the
number of samples with more than two invisible joints is 828, while the number
of samples with more than four occluded joints is 489. Note that we do not
train a new model, just compare the performance on these subsets of the MPII
validation set. The results are shown in table 6. Although the location
prediction of joints becomes more difficult with the increase of invisible
joints, our method obtains better performance compared with the SimpleBaseline
[1]. For the challenging body parts such as wrists and ankles, we achieve
$0.84\%$ and $1.46\%$ improvement respectively compared to the SimpleBaseline
when the number of invisible joints is bigger than $2$. Moreover, when the
minimum number of invisible joints increases from $2$ to $4$, the performance
gain becomes larger. These results indicate that our method can improve the
pose estimation accuracy especially when some joints are occluded.

##  5 Conclusions

In this work, we propose a generative adversarial network consisting of CFN
and GSN for human pose estimation. The CFN, which employs skip layers to
combine features with different semantic information, is used as a generator
to produce the location of human joints. Considering that the internal
dependence correlation of human joints can be regarded as a natural graph
structure, we develop a discriminator injected with a graph structure to
effectively capture the dependence correlation of pose joints. Through
adversarial learning, the generator can grasp the dependence of human joints,
so the generator can produce pose estimation results with smaller error.
Although we need to train the generator $G$ and the discriminator $D$, we only
need the generator $G$ during testing. Therefore, the inference complexity of
our method do not increase. To show the effectiveness of our model, we have
done experiments and ablation studies on three widely used datasets, _i.e_.
LSP, MPII and COCO.

Currently, our graph is a tree-based structure, which may not completely
capture the complex relationship of human joints, like the symmetric
constraints. Therefore, we can design different graph structures, even dynamic
structures to further improve the performance of human pose estimation. On the
other hand, our model obtains better performance through injecting a graph
structure. As a result, this model can be extended to other tasks including 3D
human pose estimation, face landmark detection and semantic segmentation,
where internal structure plays an important role.

## Acknowledgements

This work was supported by the National Natural Science Foundation of China
(No.61876152), the Ministry of Science and Technology Foundation funded
project under Grant 2020AAA0106900, the National Natural Science Foundation of
China (No.61902321, No.U19B2037), the China Postdoctoral Science Foundation
funded project under Grant 2019M653746, and the Fundamental Research Funds for
Central Universities of China under Grant 31020182019gx007.

## References

  * [1] B. Xiao, H. Wu, Y. Wei, Simple baselines for human pose estimation and tracking, in: Proceedings of the European Conference on Computer Vision, 2018, pp. 466–481.
  * [2] Y. Zhu, C. Ma, J. Du, Rotated cascade r-cnn: A shape robust detector with coordinate regression, Pattern Recognition 96 (2019) 106964.
  * [3] H. Zhang, H. Ouyang, S. Liu, X. Qi, X. Shen, R. Yang, J. Jia, Human pose estimation with spatial contextual information, CoRR abs/1901.01760.
  * [4] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, Y. Bengio, Generative adversarial nets, in: Advances in Neural Information Processing Systems, 2014, pp. 2672–2680.
  * [5] G. Liang, X. Lan, J. Wang, J. Wang, N. Zheng, A limb-based graphical model for human pose estimation, IEEE Transactions on Systems, Man, and Cybernetics: Systems 48 (7) (2017) 1080–1092.
  * [6] Y. Wang, W. Liang, J. Shen, Y. Jia, L.-F. Yu, A deep coarse-to-fine network for head pose estimation from synthetic data, Pattern Recognition 94 (2019) 196–206.
  * [7] X. Fan, K. Zheng, Y. Lin, S. Wang, Combining local appearance and holistic view: Dual-source deep neural networks for human pose estimation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 1347–1355.
  * [8] J. Carreira, P. Agrawal, K. Fragkiadaki, J. Malik, Human pose estimation with iterative error feedback, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 4733–4742.
  * [9] J. Tompson, R. Goroshin, A. Jain, Y. LeCun, C. Bregler, Efficient object localization using convolutional networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2015, pp. 648–656.
  * [10] K. Sun, B. Xiao, D. Liu, J. Wang, Deep high-resolution representation learning for human pose estimation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 5693–5703.
  * [11] A. Toshev, C. Szegedy, Deeppose: Human pose estimation via deep neural networks, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1653–1660.
  * [12] A. Newell, K. Yang, J. Deng, Stacked hourglass networks for human pose estimation, in: Proceedings of the European Conference on Computer Vision, Springer, 2016, pp. 483–499.
  * [13] N. Carissimi, P. Rota, C. Beyan, V. Murino, Filling the gaps: Predicting missing joints of human poses using denoising autoencoders, in: Proceedings of the European Conference on Computer Vision (ECCV), 2018, pp. 0–0.
  * [14] L. Zhao, J. Xu, S. Zhang, C. Gong, J. Yang, X. Gao, Perceiving heavily occluded human poses by assigning unbiased score, Information Sciences.
  * [15] Y. Yang, D. Ramanan, Articulated human detection with flexible mixtures of parts, IEEE transactions on pattern analysis and machine intelligence 35 (12) (2012) 2878–2890.
  * [16] X. Chu, W. Ouyang, H. Li, X. Wang, Structured feature learning for pose estimation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 4715–4723.
  * [17] L. Zhao, X. Peng, Y. Tian, M. Kapadia, D. N. Metaxas, Semantic graph convolutional networks for 3d human pose regression, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 3425–3435.
  * [18] J. Liu, H. Ding, A. Shahroudy, L.-Y. Duan, X. Jiang, G. Wang, A. C. Kot, Feature boosting network for 3d pose estimation, IEEE transactions on pattern analysis and machine intelligence 42 (2) (2019) 494–501.
  * [19] F. Scarselli, M. Gori, A. C. Tsoi, M. Hagenbuchner, G. Monfardini, The graph neural network model, IEEE Transactions on Neural Networks 20 (1) (2008) 61–80.
  * [20] M. Defferrard, X. Bresson, P. Vandergheynst, Convolutional neural networks on graphs with fast localized spectral filtering, in: Advances in Neural Information Processing Systems, 2016, pp. 3844–3852.
  * [21] R. Levie, F. Monti, X. Bresson, M. M. Bronstein, Cayleynets: Graph convolutional neural networks with complex rational spectral filters, IEEE Transactions on Signal Processing 67 (1) (2018) 97–109.
  * [22] M. Niepert, M. Ahmed, K. Kutzkov, Learning convolutional neural networks for graphs, in: International Conference on Machine Learning, 2016, pp. 2014–2023.
  * [23] Y. Li, D. Tarlow, M. Brockschmidt, R. Zemel, Gated graph sequence neural networks, in: 4th International Conference on Learning Representations,2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, 2016.
  * [24] K. Cho, B. Van Merriënboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, Y. Bengio, Learning phrase representations using rnn encoder-decoder for statistical machine translation, in: Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, 2014, October 25-29, 2014, Doha, Qatar, A meeting of SIGDAT, a Special Interest Group of the ACL, 2014, pp. 1724–1734.
  * [25] L. Zhao, H. Bai, J. Liang, B. Zeng, A. Wang, Y. Zhao, Simultaneous color-depth super-resolution with conditional generative adversarial networks, Pattern Recognition 88 (2019) 356–369.
  * [26] L. Jiao, H. Wu, H. Wang, R. Bie, Multi-scale semantic image inpainting with residual learning and gan, Neurocomputing 331 (2019) 199–212.
  * [27] Y. Fang, W. Deng, J. Du, J. Hu, Identity-aware cyclegan for face photo-sketch synthesis and recognition, Pattern Recognition 102 (2020) 107249.
  * [28] B. Wandt, B. Rosenhahn, Repnet: Weakly supervised training of an adversarial reprojection network for 3d human pose estimation, in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2019, pp. 7782–7791.
  * [29] X. Peng, Z. Tang, F. Yang, R. S. Feris, D. Metaxas, Jointly optimize data augmentation and network training: Adversarial data augmentation in human pose estimation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 2226–2234.
  * [30] Y. Chen, Z. Wang, Y. Peng, Z. Zhang, G. Yu, J. Sun, Cascaded pyramid network for multi-person pose estimation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2018, pp. 7103–7112.
  * [31] Y. Chen, C. Shen, X.-S. Wei, L. Liu, J. Yang, Adversarial posenet: A structure-aware convolutional network for human pose estimation, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 1212–1221.
  * [32] S. Johnson, M. Everingham, Clustered pose and nonlinear appearance models for human pose estimation, in: BMVC, 2010, p. 5.
  * [33] M. Andriluka, L. Pishchulin, P. Gehler, B. Schiele, 2d human pose estimation: New benchmark and state of the art analysis, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 3686–3693.
  * [34] T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, C. L. Zitnick, Microsoft coco: Common objects in context, in: European conference on computer vision, Springer, 2014, pp. 740–755.
  * [35] E. Insafutdinov, L. Pishchulin, B. Andres, M. Andriluka, B. Schiele, Deepercut: A deeper, stronger, and faster multi-person pose estimation model, in: Proceedings of the European Conference on Computer Vision, Springer, 2016, pp. 34–50.
  * [36] V. Belagiannis, A. Zisserman, Recurrent human pose estimation, in: 2017 12th IEEE International Conference on Automatic Face & Gesture Recognition, IEEE, 2017, pp. 468–475.
  * [37] I. Lifshitz, E. Fetaya, S. Ullman, Human pose estimation using deep consensus voting, in: Proceedings of the European Conference on Computer Vision, Springer, 2016, pp. 246–260.
  * [38] L. Pishchulin, E. Insafutdinov, S. Tang, B. Andres, M. Andriluka, P. V. Gehler, B. Schiele, Deepcut: Joint subset partition and labeling for multi person pose estimation, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 4929–4937.
  * [39] W. Yang, S. Li, W. Ouyang, H. Li, X. Wang, Learning feature pyramids for human pose estimation, in: Proceedings of the IEEE International Conference on Computer Vision, 2017, pp. 1281–1290.
  * [40] W. Tang, P. Yu, Y. Wu, Deeply learned compositional models for human pose estimation, in: Proceedings of the European Conference on Computer Vision, 2018, pp. 190–206.
  * [41] S. Ryou, S.-G. Jeong, P. Perona, Anchor loss: Modulating loss scale based on prediction difficulty, in: Proceedings of the IEEE International Conference on Computer Vision, 2019, pp. 5992–6001.
  * [42] Y. Bin, Z.-M. Chen, X.-S. Wei, X. Chen, C. Gao, N. Sang, Structure-aware human pose estimation with graph convolutional networks, Pattern Recognition (2020) 107410\.
  * [43] K. Su, D. Yu, Z. Xu, X. Geng, C. Wang, Multi-person pose estimation with enhanced channel-wise and spatial information, in: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2019, pp. 5674–5682.