# BOTTOM-UP APPROACHES FOR MULTI-PERSON POSE ESTIMATION and it’s Applications:
A brief Review

###### Abstract

Human Pose Estimation (HPE) is one of the fundamental problems in computer
vision. It has applications ranging from virtual reality, human behavior
analysis, video surveillance, anomaly detection, self-driving to medical
assistance. The main objective of HPE is to obtain the person’s posture from
the given input. Among different paradigms for HPE, one paradigm is called
bottom-up multi-person pose estimation. In the bottom-up approach, initially,
all the key points of the targets are detected, and later in the optimization
stage, the detected key points are associated with the corresponding targets.
This review paper discussed the recent advancements in bottom-up approaches
for the HPE and listed the possible high-quality datasets used to train the
models. Additionally, a discussion of the prominent bottom-up approaches and
their quantitative results on the standard performance matrices are given.
Finally, the limitations of the existing methods are highlighted, and
guidelines of the future research directions are given.

Index Terms—  Human pose estimation, 2D and 3D bottom-up approaches, Multi-
person pose estimation

##  1 Introduction:

Human pose estimation (HPE) is an active field of research, and it received
enormous interest from different research communities due to its usefulness
and versatility. The human pose estimation problem can simply be deduced as
obtaining the person’s posture from the given input. From a technical
perspective, the general overview of the problem that HPE methods are trying
to solve is the following:

  * •

Find the best method for obtaining the location of the human body key points

  * •

Find the best method for grouping localized key points into a valid human pose
configuration

HPE is considered relatively low level computer vision and machine learning
tasks. However, it’s importance comes from the fact that it plays a crucial
role in optimizing the workflow of other important applications domain such as
virtual reality [1, 2, 3], crowd analysis [4, 5, 6, 7, 8, 9], animal farming
[10, 11], crowd coherency detection [12, 13, 14, 15, 16, 17], intelligent
visual surveillance [18, 19], action recognition [20, 21, 22, 23, 24, 25, 26,
27], segmentation [28, 19, 29, 30, 31], object detection [32, 33, 34, 35, 36,
37, 38, 39], autonomous driving [40, 41, 42] tracking [43, 44, 45, 46, 47, 48,
49, 50, 51, 50, 52, 53, 54, 55, 56], medical imaging [57, 58, 59, 60] and
facial emotion recognition [61, 62] to name to a few. In a nutshell, HPE
methods try to detect and determine the spatial location of body keypoints,
i.e. parts or joints, from the given input. Later, from these keypoints,
obtain the pose of the human body [63]. Figure 1 shows an example of the
output of an HPE Model.

Fig. 1: Example of the output of HPE where the key joints of two person
sitting in a car are detected.

This review paper aims to give an overview of recent 2D and 3D multi-person
HPE methods based on bottom-up approaches. Different algorithms are compared
based on their characteristics, evaluation on different datasets, and
inference latency. In the following subsections, the classification of HPE
methods is elaborated based on dimensions, the number of objects to estimate
the pose for, and possible paradigm.

Fig. 2: Comparison of the caveats of bottom-up and top-down approaches [64]

###  1.1 2D Vs. 3D pose estimation

HPE can be done either in 2D or 3D. 2D HPE represents obtaining the 2D
coordinates of anatomical keypoints using monocular images or videos. In the
earlier works, this was usually done using pictorial structures [65].
Pictorial structures can be described as tree-structured graphical models that
show spatial correlations of the body parts. Although these structures were
successful in estimating human pose in images that don’t have too many
occlusions, when that is not the case, this approach shows poor results [65].
Another type of approach was using hand-crafted features. This kind of
approach was before deep learning models started being more widely used. It
was heavily dependent on human supervision. Some of those hand-crafted
features were HOGs, color histograms, edges, contours, etc. [66]. Although
these kinds of features can easily be obtained, they lack generalization and
often produce bad results. On the other hand, the 3D HPE objective is to get
the coordinates of anatomical key points in 3D space using monocular input or
Multiview cameras. This type of HPE is often categorized as model-free and
model-based, depending on whether the human body model is used [67].

###  1.2 Single Vs. Multi-person HPE

Depending on the number of present persons on a given input, HPE approaches
can be divided into a single person and a multi-person approach. Single-person
HPE is used for estimating the human pose for just one person in the image.
That is, it is guaranteed that the input contains only one person. As this is
not the case in most cases, another type of approach is needed to be
developed. If there is more than one person in the given input, then that kind
of approach is called multi-person pose estimation. For example, for the self-
driving car to be safer with pedestrians or for video surveillance to have
better tracking, there is a massive necessity for obtaining a fast and
reliable multi-person HPE algorithm.

###  1.3 Top-down Vs. Bottom-up

HPE methods can be characterized as top-down or bottom-up approaches. Top-down
approaches first detect all persons from the input and later apply a human
pose estimation algorithm for each detected person independently. On the other
hand, bottom-up approaches first detect all body parts from the input and
later group them for each person. There are several caveats to these two
approaches. Top-down approaches have poorer performance in terms of time.
These methods are usually time-consuming and computationally complex.
Additionally, the performance of top-down approaches is heavily influenced by
the performance of the technique used for person detection. Therefore, top-
down approaches have a problem when there are complex poses and when there is
a lot of crowding. On the other hand, even though bottom-up approaches are
faster, they have some issues, such as detecting and grouping body parts when
there is a crowd of people or when there is an ambiguous appearance of the
body parts in the input. Figure 2 shows a comparison of these two approaches.

###  1.4 Contributions

Currently, the problem of HPE is an active field of research. Since the last
decade, tremendous progress has been made in human pose estimation and the
datasets that have been created. This paper provides an overview of the most
recent HPE methods and points to future research directions like some surveys
as mentioned earlier. The concept that differentiates this survey from the
other surveys is that it focuses only on bottom-up multi-person pose
estimation. Here is worth noting that a review paper has not yet been released
with a focus on that. These types of approaches are the most promising for
real-time implementation and, as such, can serve a great deal for a variety of
industries. In summary, the contribution of this survey can be condensed into
three aspects:

  * •

Provides an overview of the most recent 2D and 3D multi-person HPE methods
that are based on bottom-up approaches

  * •

Categorizations of presented HPE methods

  * •

Presentation of the limitations and open issues for each method

This review paper is organized as follows. Section 2 introduces and describes
some of the ways to categorize HPE methods. Section 3 describes the existing
datasets that are widely used in HPE evaluation and presents some of the
performance metrics. Section 4 gives an overview of some of the state-of-the-
art methods, both 2D and 3D. Discussion about the presented methods, existing
issues, and several promising future directions is presented in section 5.
Last but not the least, section 6 concludes the paper.

(a)

(b)

Fig. 3: Examples from panoptic dataset

##  2 Related Work

In the last couple of years, there has been a couple of HPE review papers,
specifically for 3D HPE ([68], [69]), model-based HPE ([68], [70]), monocular-
based HPE ([68], [69]), etc. Some of the review papers are just focused on
methods that don’t implement deep-learning approaches ([71, 72]), while others
[73] focus on methods based on deep-learning. A comprehensive review of the 2D
and 3D HPE is presented in [68]. In this paper, 2D HPE is categorized like it
is in the [74], which is a review paper oriented only at 2D HPE approaches. On
the other hand, in [68], 3D HPE methods are categorized as in [67]. This
review paper focuses on 3D HPEs and categorization of these HPEs into three
distinctive groups:

  * •

Generative vs. Discriminative methods

  * •

Regression-based vs. Detection-based methods

  * •

One-stage vs. Multi-stage methods

###  2.1 Generative vs. Discriminative Models

Generative and discriminative models mainly differ in whether a particular
method uses the human body model or not. Generative models take the body model
as a priori information. In a more general form, these methods can be
deconstructed into two phases [75]. First is the modeling phase, where a
likelihood function is constructed by taking into account the camera model,
the human body model, all constraints, etc. After constructing this likelihood
function, the second phase is introduced, and that is the estimation phase. In
this phase, the human body poses are predicted based on given input and the
constructed likelihood function [67]. Besides this general form of generative
methods, there is also a subcategory of generative approaches that is called
part-based. Instead of having a rigid human body model, in these methods, the
human skeleton is represented as a collection of body parts. These parts are
only constrained by the joints in the body model. This allows for a collection
of the body part that can take a deformable configuration [67]. Discriminative
models don’t take into account any specific human body model. These methods
learn to map given input directly to the human pose space or to search for
example pool of pose descriptors. Discriminative methods are in most cases
faster than generative methods, but they lack the robustness for the poses
that were not introduced during the training [68].

(a) COCO keypoints [76]

(b) MPII keypoints [77]

Fig. 4: Human keypoint location based on COCO and MPII format

###  2.2 Regression Vs. Detection

Discriminative HPE methods for the most part represent methods that are based
on deep learning. These methods can further be differentiated based on the way
that they detect body joints. When discriminative methods directly map given
input to the coordinates of body joints, then these methods can be categorized
as regression-based (also known as learning-based) approaches. For this to
function method needs to generalize well for a new set of images from the
testing set [78]. Detection-based approaches look at the body parts as the
detection targets. These targets are usually represented by two different
representations [68]:

  * •

Image patches

  * •

Heatmaps of joint locations

Additionally, if the discriminative method utilizes an example pool of the
pose descriptors, then that method is categorized as an example-based
approach. The final pose is calculated by doing the interpolation of the
candidates that are obtained with the similarity search [67]. Because of the
high non-linearity of the problem that regression-based approaches try to
solve, detection-based approaches are more widely used. Although, the
disadvantages of these types of approaches is that compared to the original
images, newly created representations have a smaller resolution, therefore the
accuracy of the predicted location of the body joint is smaller [68].

###  2.3 ONE-STAGE VS. MULTI-STAGE

The main difference between one-stage approaches and multi-stage approaches is
whether the end-to-end network is used. That is, multi-stage methods have two
or more different stages that are specialized for each task. On the other
hand, one-stage methods employ only one end-to-end network to map given input
to human poses [68]. One of the examples for multi-stage methods can be seen
in 3D HPE. Some of the methods first predict body part locations on the 2D
surface, and later extend that 2D surface to the 3D space.

##  3 Datasets

Datasets are an important component of any HPE method. Not only are they
important for evaluating the method, but more complex datasets give better
agility for researchers to try out new ideas. Making HPE datasets is a
laborious and long process. In the past, the first datasets didn’t have a lot
of images, and the complexity was low. This is due to the fact that the
process Is time-consuming and finding the people willing to do it was harder.
As crowdsourcing services began to rise, datasets became better and no longer
bounded by the lab environments or the data quantity.This section will
differentiate existing datasets whether they are 2D or 3D. Additionally, this
section will only focus on datasets that can be used for multi-pose HPE. At
the end of this section, some of the performance metrics will be presented.

###  3.1 2D datasets

Two biggest and most used datasets for 2D multi-person HPE are:

  * •

Max Planck Institute for Informatics (MPII) Human Pose Dataset [77]

  * •

Microsoft Common Objects in Context (COCO) Dataset [76]

Fig. 5: The workflow of the OpenPose model [79]

####  3.1.1 MPII DATASET

MPII Human pose dataset is one of the state-of-the-art benchmarks for
evaluation of articulated human pose estimation [80]. In order to produce this
dataset 24,920 frames were selected from 3,913 YouTube videos that contain
either different people in the same setting or the same person in different
poses. The frames are annotated by 15 body joints and a background keypoint,
in addition to the 3D viewpoint of the head and torso. Besides the position of
each keypoint, this dataset also contains information about the visibility of
each keypoint.

####  3.1.2 MS-COCO DATASET

MS-COCO dataset is widely used for multi-person HPE methods. It is a dataset
that contains object detection, segmentation, image captioning, and keypoint
detections. Images were collected from most popular search engines like
Google, Bing, etc. The dataset contains 200,000 images and around 250,000
labeled person instances [81]. Each person is annotated by 18 keypoints and
one for the background. Like the MPII dataset, besides the position of each
keypoint, this dataset stores the information about the visibility of each
keypoint. Figure 4 shows the comparison of COCO and MPII keypoints.

###  3.2 3D datasets

Mostly used datasets for 3D multi-person HPE methods are the following:

  * •

Panoptic dataset [82]

  * •

3DPW dataset [78]

Fig. 6: The architecture of the OpenPose model [79]

####  3.2.1 Panoptic dataset

Ground truth for these datasets was made with a motion capture system
containing 480 VHA camera views, 31 HD views, 10 RGB-D sensors. It consists of
65 sequences with 5.5 total number of hours and more than 1.5 million 3D
skeletons.

####  3.2.2 3DPW dataset

Ground truth for this dataset was made with a single hand-held camera. There
are 60 video sequences with more than 51,000 frames. Figure 3 shows some of
the human poses from both datasets.

Table 1: Overview of 2D datasets Dataset | Joints | Train | Val | Test
---|---|---|---|---
MPII | 16 | 3.8k | 0 | 1.7k
COCO | 17 | 64k | 2k | 40k

###  3.3 Performance metrics

Performance metrics are important tools for evaluating the given method. There
a couple of metrics that are specific for HPE problems and they are divided
into 2D and 3D performance metrics.

####  3.3.1 2D metrics

##### 3.3.1.1 Percentage of correct parts (PCP)[83]

It characterizes the accuracy of a limb’s predicted location. A limb is said
to be accurately localized if two predicted keypoints that make the respective
limb are within some threshold of the limb’s ground-truth keypoints. This
threshold can be 50% of the length of the predicted limb. PCP can either be
represented as the mean value of all predictions, or it can be produced for a
specific limb. PCPm [77] is a variation of PCP where the threshold is
represented as 50% of the mean ground-truth segment length over the whole test
set.

##### 3.3.1.2 Percentage of correct keypoints (PCK)[84]

measures how accurate is predicted keypoint’s location. Keypoint location
prediction is said to be correct if it’s in the given threshold pixels of the
ground-truth keypoint. This threshold can be defined by the person’s bounding
box, or most often as 50% of the head segment length<EMAIL_ADDRESS>[77]. The
Average Precision (AP) measures how accurate is the prediction, that is, what
is the percentage of correct predictions. It takes into account true positive
and false positive predictions and represents the ratio of true positive
predictions and sum of true positive and false-positive predictions. For the
evaluation of HPE methods if a predicted keypoint’s location falls within the
threshold of the ground-truth keypoint location it is counted as a true
positive. In the end, all the unassigned predictions are counted as false
positive.

##### 3.3.1.3 Average Precision (AP)

It measures how accurate is the prediction, that is, what is the percentage of
correct predictions. It takes into account true positive and false positive
predictions and represents the ratio of true positive predictions and sum of
true positive and false-positive predictions. For the evaluation of HPE
methods if a predicted keypoint’s location falls within the threshold of the
ground-truth keypoint location it is counted as a true positive. In the end,
all the unassigned predictions are counted as false positive.

####  3.3.2 3D metrics

##### 3.3.2.1 Mean Per Joint Position Error (MPJPE)

It is a measure used for evaluating 3D HPE methods. This metric calculates
Euclidean distance between the predicted 3D keypoint location and the ground-
truth keypoint location. This is then averaged for all keypoints in an image,
or, if there is a set of frames, the mean error is averaged for all frames
[1].

Table 2: Overview of 3D datasets Dataset | Joints | Train
---|---|---
Panoptic | 15 | 65 videos (5.5 hours)
3DPW | 18 | 60 videos (51k frames)

##  4 Overview of Popular algorithms

This section will give an overview of state-of-the-art methods both 2D and 3D.

###  4.1 2D methods

####  4.1.1 OpenPose

The detection-based method presented in [24] describes an approach that uses a
non-parametric representation, called Part Affinity Fields (PAF). Some of the
problems addressed in [24] are the possibility of detecting individual body
joints even when in the given input there are multiple people, or there is
irregular interaction between people, or the size of the people in the image
is different. The workflow for this method is following. First, for a given
input (Fig. 5 a), a set of part confidence maps is predicted. These maps
represent the location of each joint (Fig. 5 b). Additionally, a set of part
affinity fields is predicted (Fig. 5. c). These fields describe the location
and orientation of the body parts and are described as a set of 2D vectors
that indicate the degree of association between the body parts [63].
Afterward, a matching algorithm is needed to make an association between
predicted joints (Fig. 5 d). This is done with bipartite matching. In the end,
these matched joints need to be assembled during the parsing phase (Fig. 5 e)
into a full-body pose for each person in the image. The whole workflow can be
seen in Figure 5.

Figure 6 shows the overall architecture of this method. This method is
represented as multi-stage two-branch CNN architecture. In order to predict
all the maps from a given input, the input image needs to be converted into
feature space F. This is done with the help of the first 10 layers of the VGG
network. The converted input is then fed into the two branches that work
simultaneously. The beige branch in figure 6 represents the branch that is
predicting set S of the part confidence maps with some confidence of J. The
blue branch predicts a set of part affinity fields S of 2D vectors. There are
C vectors corresponding to each body part. The first stage outputs a set of
part confidence maps and part affinity fields. This is forwarded in all
successive stages along with the feature map. Additionally, after each stage,
there is intermediate supervision. This is done to solve the vanishing
gradients.

Fig. 7: The architecture of associative embedding network for multi-person
pose detection [85]

####  4.1.2 Associative embedding

The method that is presented in [85] instead of using multi-stages for human
pose estimation, uses an end-to-end network. This is done by introducing the
idea of associative embedding. Using this idea, it is possible to jointly
perform detection and grouping with just one stage. The concept behind the
idea is the following. For each detection, a vector embedding is introduced.
This embedding is a tag number that is used to identify the group assignment.
That is, all the detections that have the same tag value belong to the same
body pose. The network outputs two heatmaps. A heatmap for per-pixel detection
scores, which represents a map of detection scores at each pixel for each
joint. The other heatmap is a heatmap for per-pixel identity tags. That is,
tagging score at each pixel for each joint. These two heatmaps are then used
together to extract the corresponding embedding from the pixel location with
the highest detection score for each body joint. Two loss functions are
combined. First is detection loss which is the mean square error between each
predicted detection heatmap and its ground truth heatmap. The second is
grouping loss which compares the tags within each person and across the
people. Tags within a person should be the same, whereas tags for different
people should be separate. It’s worth noting that there is no ground truth for
tags as what matters is not the exact value of the tag, but that there is some
difference between tags of body joints of different people. In the case of
multi-person pose estimation, associative embedding is integrated with a
stacked hourglass network. This integrated architecture produces output that
is represented as a detection heatmap and a tagging heatmap for each body
joint. The outcome is then used to group body joints with similar tags into
individual people’s poses. The architecture for multi-person pose estimation
is shown in figure 7.

###  4.2 3D methods

This section gives an overview of 2 different state-of-the-art 3D bottom-up
multi-person HPE methods.

####  4.2.1 XNECT

In the [86] method named XNect is presented. This method is a real-time method
for 3D multi-person pose estimation from RGB image input. It is tailored to
withstand generic scenes and occlusions by objects and people. The main idea
behind it is to introduce two deep neural network stages that can have a local
scope – for each body joint – and global scope – for all body joints. It is a
bottom-up approach that is multi-staged. It contains three stages, where the
last uses a predefined 3D skeleton to enhance the performance. Therefore, this
method is model-based. Figure 8 shows the architecture of this network. The
first stage is a convolutional neural network that tries to estimate 2D and 3D
pose features with the identity assignments. The network that is used is
SelecSLSNet. This CNN contains selective long and short skip connections that
improve the information flow and optimize the inference speed of the network.
The output from this stage is a set of 2D part confidence maps that contain
the information for assigning body parts to an individual. Besides this, the
output of this stage is also an intermediate 3D pose encoding for the bones
that connect at the joint. The key insight of this stage is that it only
estimates the pose features of body joints that are not occluded in the image.
Additional refinement is done in stage 2. This stage is comprised of a fully
connected neural network. This stage estimates a full 3D pose regarding the
possible partial 2D and 3D pose features for each person from the previous
stage. The final third stage uses a skeletal model to optimize the 3D pose
estimation. The space-time skeletal model predicts the 3D pose and enforces
temporal coherence.

Fig. 8: The overview of the architecture of XNect model [86]

####  4.2.2 MUBYNET

MubyNet method is presented in [87]. This is a multi-stage, not a real-time
method used for 3D pose and shape estimation and integrated localization. It
is a bottom-up method that can estimate poses for multiple people in the given
input. Firstly, this method identifies body joints and limbs and groups them.
This grouping is done with 2D and 3D information that is obtained with learned
scoring functions. In the end, this method optimally aggregates estimated
joints and limbs into a partial or complete 3D human body pose with the help
of the skeleton hypothesis and kinematic tree constraints. Figure 9 shows the
overall architecture of MubyNet. The input in the network is an RGB image that
goes into the deep feature extractor stage. This stage computes features that
are used in the deep volume encoding stage. At this point, features are
refined to contain only 2D and 3D pose information. The next stage is the limb
scoring stage, where all possible kinematic connections between 2D detected
joints are collected. Such collected links are then used to predict
corresponding scores. These scores are then fed into the skeleton grouping
stage, where people are localized by assembling limbs into skeletons. In the
end, for each person in the 3D pose decoding & shape estimation stage, 3D pose
is estimated.

Fig. 9: The overview of the architecture for the MubyNet model [87] Table 3: Overview and evaluation of some methods for multi-person pose estimation  Algorithm |  Type |  Highlight |  Dataset |  Metrics |  Score
---|---|---|---|---|---
OpenPose |  16 |  3.8k |  0 |  1.7k |  Test
Associative embeddling |  17 |  64k |  2k |  40k |  Test
Multi-Posenet |  17 |  64k |  2k |  40k |  Test
HRNet |  17 |  64k |  2k |  40k |  Test
PoseFix |  17 |  64k |  2k |  40k |  Test

##  5 Summary and discussion

In the previous section, some of the most well-known bottom-up methods for
multi-person 2D or 3D pose estimation were briefly explained. The reason for
choosing these methods as an example lies in their innovation, influence,
contribution, and different types of ideas and approaches that can be valuable
to the reader. In this section, all previously explained methods are compared
and discussed. The rest of the research paper will address the existing
issues, but it will also give an insight into possible future directions for
this type of research. Table 2 shows a short description of each method, its
category, and comparable metrics. To have a better overview, the main
requirement for a method shown in this table is to be a multi-person pose
estimation method.

(a)

(b)

Fig. 10: (a) rare pose, (b) missing body part, (c) overlapping of different
body parts, (d) assigning body part to a wrong person, (e-f) false positives:
animals, statues, etc. [79]

###  5.1 Existing Issues

In order to have a complete picture of bottom-up methods, some of the existing
issues need to be addressed. From table 2, it can be observed that top-down
approaches yield better results metrics-wise. But for a method to be used in
real-life, the inference time is often more important than the other metrics
values. Bottom-up approaches still show sufficiently better speed-wise results
and are worth looking into. However, it is essential to note that the
performance of bottom-up methods heavily depends on how complex the background
is and the frequency of human occlusions. Some of these failure cases can be
seen in figure 10. Specifically, for the 3D bottom-up approaches for multi-
person pose estimation, the following issues arise. For example, the MubyNet
method encodes the full 3D pose vector for each pixel part of the skeleton
instead of just encoding it at the body joints’ location. This is a suboptimal
solution, and it creates potential encoding conflicts in the 3D feature space.
On the other hand, XNect uses a different approach. This method tries to keep
it simple in the beginning and encodes the 3D pose vector just at the body
joint location, but in later stages, it fills out the missing information for
the rest of the pixels. This makes the problem more manageable and robust to
wrongly encoded 3D pose vectors. XNect method doesn’t give good results when
2D pose estimates that are used inside of the network are wrong. Additionally,
as it was stated in the XNect paper, it is required for the neck to be visible
for successful detection. If this is not the case, XNect cannot give a good
detection and estimation. The rest of the limitations can be seen in figure
11.

###  5.2 Future directions

In order to resolve some of the current issues of HPE, future work will need
to look into both global and local contexts when estimating the human pose.
Some of the confusion could potentially be resolved with cues from both
contexts. Besides this, future networks need to be more efficient. How fast is
the inference mode for a particular method significantly impacts how and when
these methods will be used in real-life. Additionally, because of how valuable
datasets are in these kinds of problems, the datasets could also be
diversified. That way, the methods can be more robust and handle complex
scenes. The other strategy is to create datasets for just a specific complex
scenario where the proposed method will be used. For the 3D multi-person pose
estimation methods, some of the physics-based motion constraints can be
employed. These constraints could lead to the improvement of pose stability
and overall temporal stability. Furthermore, these problems could be remedied
with the help of a better space-time identity tracker.

Table 4: Overview and evaluation of some methods for multi-person pose estimation  Algorithm |  Type |  Highlight |  Dataset |  Metrics |  Score
---|---|---|---|---|---
XNect |  16 |  3.8k |  0 |  1.7k |  Test
MubyNet |  17 |  64k |  2k |  40k |  Test
LCR-Net++ |  17 |  64k |  2k |  40k |  Test
Camera distance aware |  17 |  64k |  2k |  40k |  Test

##  6 Conclusion:

This paper represents an overview of the carefully selected 2D and 3D bottom-
up multi-person pose estimation methods. These methods are promising and
necessary for future applications in real-life problems. It is worth noting
that the bottom-up multi-person pose estimation methods give a real-time
performance. Although bottom-up approaches are in some cases worse than top-
down approaches, bottom-up approaches still hold the top of the list when the
inference mode speed is in the case. This review paper briefly introduces the
human pose estimation research field, and it starts with problem formulation
and introduction to human pose estimation. Furthermore, the overview of the
most commonly used datasets is given, alongside the primarily used evaluation
metrics in this field. Additionally, an overview of some of the prominent
methods is given, and finally, these methods alongside other state-of-the-art
methods are compared and discussed. To sum up, this review paper provides the
reader with an essential background of this topic. Additionally, it presents
an overview of some of the limitations and open issues that can help future
researchers wisely choose the architecture and be aware of all the caveats and
potential future research directions.

Fig. 11: a) 2D pose limb confusion, (b-c) person not detected due to neck
occlusion, (d) body orientation confusion due to occluded face, (e-f) poses
not seen in the training set [86]

## References

  * [1] Luca Schmidtke, Athanasios Vlontzos, Simon Ellershaw, Anna Lukens, Tomoki Arichi, and Bernhard Kainz,  “Unsupervised human pose estimation through transforming shape templates,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 2484–2494.
  * [2] Habib Ullah, Sultan Daud Khan, Mohib Ullah, and Faouzi Alaya Cheikh,  “Social modeling meets virtual reality: An immersive implication,”  in International Conference on Pattern Recognition. Springer, 2021, pp. 131–140.
  * [3] Kai Chen and Qi Dou,  “Sgpa: Structure-guided prior adaptation for category-level 6d object pose estimation,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 2773–2782.
  * [4] Habib Ullah, Ihtesham Ul Islam, Mohib Ullah, Muhammad Afaq, Sultan Daud Khan, and Javed Iqbal,  “Multi-feature-based crowd video modeling for visual event detection,”  Multimedia Systems, vol. 27, no. 4, pp. 589–597, 2021.
  * [5] Hongsuk Choi, Gyeongsik Moon, JoonKyu Park, and Kyoung Mu Lee,  “3dcrowdnet: 2d human pose-guided3d crowd human pose and shape estimation in the wild,”  arXiv preprint arXiv:2104.07300, 2021.
  * [6] Beibei Zhan, Dorothy N Monekosso, Paolo Remagnino, Sergio A Velastin, and Li-Qun Xu,  “Crowd analysis: a survey,”  Machine Vision and Applications, vol. 19, no. 5, pp. 345–357, 2008\.
  * [7] Hamza Riaz, Muhammad Uzair, Habib Ullah, and Mohib Ullah,  “Anomalous human action detection using a cascade of deep learning models,”  in 2021 9th European Workshop on Visual Information Processing (EUVIP). IEEE, 2021, pp. 1–5.
  * [8] Jan Stenum, Cristina Rossi, and Ryan T Roemmich,  “Two-dimensional video-based analysis of human gait using pose estimation,”  PLoS computational biology, vol. 17, no. 4, pp. e1008935, 2021.
  * [9] Sultan Daud Khan, Maqsood Mahmud, Habib Ullah, Mohib Ullah, and Faouzi Alaya Cheikh,  “Crowd congestion detection in videos,”  Electronic Imaging, vol. 2020, no. 6, pp. 72–1, 2020.
  * [10] Cheng Fang, Tiemin Zhang, Haikun Zheng, Junduan Huang, and Kaixuan Cuan,  “Pose estimation and behavior classification of broiler chickens based on deep neural networks,”  Computers and Electronics in Agriculture, vol. 180, pp. 105863, 2021\.
  * [11] Akif Quddus Khan, Salman Khan, Mohib Ullah, and Faouzi Alaya Cheikh,  “A bottom-up approach for pig skeleton extraction using rgb data,”  in International Conference on Image and Signal Processing. Springer, 2020, pp. 54–61.
  * [12] Nurul Japar, Ven Jyn Kok, and Chee Seng Chan,  “Coherent group detection in still image,”  Multimedia Tools and Applications, pp. 1–20, 2021.
  * [13] Habib Ullah, Muhammad Uzair, Mohib Ullah, Asif Khan, Ayaz Ahmad, and Wilayat Khan,  “Density independent hydrodynamics model for crowd coherency detection,”  Neurocomputing, vol. 242, pp. 28–39, 2017.
  * [14] Joshua Candamo, Matthew Shreve, Dmitry B Goldgof, Deborah B Sapper, and Rangachar Kasturi,  “Understanding transit scenes: A survey on human behavior-recognition algorithms,”  IEEE transactions on intelligent transportation systems, vol. 11, no. 1, pp. 206–224, 2009.
  * [15] Mohib Ullah, Habib Ullah, Nicola Conci, and Francesco GB De Natale,  “Crowd behavior identification,”  in 2016 IEEE international conference on image processing (ICIP). IEEE, 2016, pp. 1195–1199.
  * [16] David A Forsyth, Okan Arikan, Leslie Ikemoto, Deva Ramanan, and James O’Brien,  “Computational studies of human motion: Tracking and motion synthesis,”  2006\.
  * [17] Habib Ullah, Muhammad Uzair, Mohib Ullah, Asif Khan, Ayaz Ahmad, and Wilayat Khan,  “Density independent hydrodynamics model for crowd coherency detection,”  Neurocomputing, vol. 242, pp. 28–39, 2017.
  * [18] In Su Kim, Hong Seok Choi, Kwang Moo Yi, Jin Young Choi, and Seong G Kong,  “Intelligent visual surveillance—a survey,”  International Journal of Control, Automation and Systems, vol. 8, no. 5, pp. 926–939, 2010.
  * [19] Habib Ullah, Mohib Ullah, and Muhammad Uzair,  “A hybrid social influence model for pedestrian motion segmentation,”  Neural Computing and Applications, vol. 31, no. 11, pp. 7317–7333, 2019.
  * [20] Mohib Ullah, Muhammad Mudassar Yamin, Ahmed Mohammed, Sultan Daud Khan, Habib Ullah, and Faouzi Alaya Cheikh,  “Attention-based lstm network for action recognition in sports,”  Electronic Imaging, vol. 2021, no. 6, pp. 302–1, 2021.
  * [21] Gyeongsik Moon, Heeseung Kwon, Kyoung Mu Lee, and Minsu Cho,  “Integralaction: Pose-driven feature integration for robust human action recognition in videos,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 3339–3348.
  * [22] Habib Ullah, Sultan Daud Khan, Mohib Ullah, Faouzi Alaya Cheikh, and Muhammad Uzair,  “Two stream model for crowd video classification,”  in 2019 8th european workshop on visual information processing (EUVIP). IEEE, 2019, pp. 93–98.
  * [23] Qianyu Wu, Aichun Zhu, Ran Cui, Tian Wang, Fangqiang Hu, Yaping Bao, and Hichem Snoussi,  “Pose-guided inflated 3d convnet for action recognition in videos,”  Signal Processing: Image Communication, vol. 91, pp. 116098, 2021\.
  * [24] Saira Kanwal, Muhammad Uzair, Habib Ullah, Sultan Daud Khan, Mohib Ullah, and Faouzi Alaya Cheikh,  “An image based prediction model for sleep stage identification,”  in 2019 IEEE International Conference on Image Processing (ICIP). IEEE, 2019, pp. 1366–1370.
  * [25] Mohib Ullah, Habib Ullah, and Ibrahim M Alseadonn,  “Human action recognition in videos using stable features,”  2017\.
  * [26] Jinmiao Cai, Nianjuan Jiang, Xiaoguang Han, Kui Jia, and Jiangbo Lu,  “Jolo-gcn: mining joint-centered light-weight information for skeleton-based action recognition,”  in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2021, pp. 2735–2744.
  * [27] Mohib Ullah, Habib Ullah, Sultan Daud Khan, and Faouzi Alaya Cheikh,  “Stacked lstm network for human activity recognition using smartphone data,”  in 2019 8th European workshop on visual information processing (EUVIP). IEEE, 2019, pp. 175–180.
  * [28] Chungang Zhuang, Zhe Wang, Heng Zhao, and Han Ding,  “Semantic part segmentation method based 3d object pose estimation with rgb-d images for bin-picking,”  Robotics and Computer-Integrated Manufacturing, vol. 68, pp. 102086, 2021.
  * [29] Yujia Zhai, Baoli Lu, Weijun Li, Jian Xu, and Shuangyi Ma,  “Jd-slam: Joint camera pose estimation and moving object segmentation for simultaneous localization and mapping in dynamic scenes,”  International Journal of Advanced Robotic Systems, vol. 18, no. 1, pp. 1729881421994447, 2021.
  * [30] Habib Ullah, Mohib Ullah, and Muhammad Uzair,  “A hybrid social influence model for pedestrian motion segmentation,”  Neural Computing and Applications, vol. 31, no. 11, pp. 7317–7333, 2019.
  * [31] Mohib Ullah, Ahmed Mohammed, and Faouzi Alaya Cheikh,  “Pednet: A spatio-temporal deep convolutional neural network for pedestrian segmentation,”  Journal of Imaging, vol. 4, no. 9, pp. 107, 2018.
  * [32] Timon Höfer, Faranak Shamsafar, Nuri Benbarka, and Andreas Zell,  “Object detection and autoencoder-based 6d pose estimation for highly cluttered bin picking,”  arXiv preprint arXiv:2106.08045, 2021.
  * [33] Mohib Ullah, Mohammed Ahmed Kedir, and Faouzi Alaya Cheikh,  “Hand-crafted vs deep features: A quantitative study of pedestrian appearance model,”  in 2018 Colour and Visual Computing Symposium (CVCS). IEEE, 2018, pp. 1–6.
  * [34] Hansheng Chen, Yuyao Huang, Wei Tian, Zhong Gao, and Lu Xiong,  “Monorun: Monocular 3d object detection by reconstruction and uncertainty propagation,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 10379–10388.
  * [35] Sultan Daud Khan, Ahmed B Altamimi, Mohib Ullah, Habib Ullah, and Faouzi Alaya Cheikh,  “Tcm: Temporal consistency model for head detection in complex videos,”  Journal of Sensors, vol. 2020, 2020.
  * [36] Ali Varamesh and Tinne Tuytelaars,  “Mixture dense regression for object detection and human pose estimation,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 13086–13095.
  * [37] Habib Ullah, Ahmed B Altamimi, Muhammad Uzair, and Mohib Ullah,  “Anomalous entities detection and localization in pedestrian flows,”  Neurocomputing, vol. 290, pp. 74–86, 2018.
  * [38] Frederik Hagelskjær and Anders Glent Buch,  “Pointvotenet: Accurate object detection and 6 dof pose estimation in point clouds,”  in 2020 IEEE International Conference on Image Processing (ICIP). IEEE, 2020, pp. 2641–2645.
  * [39] Sultan Daud Khan, Habib Ullah, Mohib Ullah, Nicola Conci, Faouzi Alaya Cheikh, and Azeddine Beghdadi,  “Person head detection based deep model for people counting in sports videos,”  in 2019 16th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). IEEE, 2019, pp. 1–8.
  * [40] Chengfeng Zhao, Chen Fu, John M Dolan, and Jun Wang,  “L-shape fitting-based vehicle pose estimation and tracking using 3d-lidar,”  IEEE Transactions on Intelligent Vehicles, 2021.
  * [41] Yaqing Ding, Daniel Barath, Jian Yang, Hui Kong, and Zuzana Kukelova,  “Globally optimal relative pose estimation with gravity prior,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 394–403.
  * [42] Renshu Gu, Gaoang Wang, and Jenq-Neng Hwang,  “Efficient multi-person hierarchical 3d pose estimation for autonomous driving,”  in 2019 IEEE Conference on Multimedia Information Processing and Retrieval (MIPR). IEEE, 2019, pp. 163–168.
  * [43] Zolbayar Shagdar, Mohib Ullah, Habib Ullah, and Faouzi Alaya Cheikh,  “Geometric deep learning for multi-object tracking: A brief review,”  in 2021 9th European Workshop on Visual Information Processing (EUVIP). IEEE, 2021, pp. 1–6.
  * [44] Long Chen, Haizhou Ai, Rui Chen, Zijie Zhuang, and Shuang Liu,  “Cross-view tracking for multi-human 3d pose estimation at over 100 fps,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2020, pp. 3279–3288.
  * [45] Mohib Ullah, Maqsood Mahmud, Habib Ullah, Kashif Ahmad, Ali Shariq Imran, and Faouzi Alaya Cheikh,  “Head based tracking,”  Electronic Imaging, vol. 2020, no. 6, pp. 74–1, 2020.
  * [46] Manchen Wang, Joseph Tighe, and Davide Modolo,  “Combining detection and tracking for human pose estimation in videos,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 11088–11096.
  * [47] Mohib Ullah, Ahmed Kedir Mohammed, Faouzi Alaya Cheikh, and Zhaohui Wang,  “A hierarchical feature model for multi-target tracking,”  in 2017 IEEE international conference on image processing (ICIP). IEEE, 2017, pp. 2612–2616.
  * [48] Chunluan Zhou, Zhou Ren, and Gang Hua,  “Temporal keypoint matching and refinement network for pose estimation and tracking,”  in European Conference on Computer Vision. Springer, 2020, pp. 680–695.
  * [49] Mohib Ullah, Faouzi Alaya Cheikh, and Ali Shariq Imran,  “Hog based real-time multi-target tracking in bayesian framework,”  in 2016 13th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS). IEEE, 2016, pp. 416–422.
  * [50] Mikel Ariz, José J Bengoechea, Arantxa Villanueva, and Rafael Cabeza,  “A novel 2d/3d database with automatic face annotation for head tracking and pose estimation,”  Computer Vision and Image Understanding, vol. 148, pp. 201–210, 2016.
  * [51] Mohib Ullah, Habib Ullah, and Faouzi Alaya Cheikh,  “Single shot appearance model (ssam) for multi-target tracking,”  Electronic Imaging, vol. 2019, no. 7, pp. 466–1, 2019.
  * [52] Mohib Ullah and Faouzi Alaya Cheikh,  “Deep feature based end-to-end transportation network for multi-target tracking,”  in 2018 25th IEEE International Conference on Image Processing (ICIP). IEEE, 2018, pp. 3738–3742.
  * [53] Hau Chu, Jia-Hong Lee, Yao-Chih Lee, Ching-Hsien Hsu, Jia-Da Li, and Chu-Song Chen,  “Part-aware measurement for robust multi-view multi-human 3d pose estimation and tracking,”  in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 1472–1481.
  * [54] Mohib Ullah and Faouzi Alaya Cheikh,  “A directed sparse graphical model for multi-target tracking,”  in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, 2018, pp. 1816–1823.
  * [55] Yifu Zhang, Chunyu Wang, Xinggang Wang, Wenyu Liu, and Wenjun Zeng,  “Voxeltrack: Multi-person 3d human pose estimation and tracking in the wild,”  arXiv preprint arXiv:2108.02452, 2021.
  * [56] Mohib Ullah, Muhammad Mudassar Yamin, Ahmed Mohammed, Sultan Daud Khan, Habib Ullah, and Faouzi Alaya Cheikh,  “Attention-based lstm network for action recognition in sports,”  Electronic Imaging, vol. 2021, no. 6, pp. 302–1, 2021.
  * [57] Felix Achilles, Alexandru-Eugen Ichim, Huseyin Coskun, Federico Tombari, Soheyl Noachtar, and Nassir Navab,  “Patient mocap: Human pose estimation under blanket occlusion for hospital monitoring applications,”  in International conference on medical image computing and computer-assisted intervention. Springer, 2016, pp. 491–499.
  * [58] William McNally, Kanav Vats, Alexander Wong, and John McPhee,  “Evopose2d: Pushing the boundaries of 2d human pose estimation using neuroevolution,”  arXiv preprint arXiv:2011.08446, vol. 2, 2020.
  * [59] Praneet C Bala, Benjamin R Eisenreich, Seng Bum Michael Yoo, Benjamin Y Hayden, Hyun Soo Park, and Jan Zimmermann,  “Automated markerless pose estimation in freely moving macaques with openmonkeystudio,”  Nature communications, vol. 11, no. 1, pp. 1–12, 2020.
  * [60] Leslie Casas, Nassir Navab, and Stefanie Demirci,  “Patient 3d body pose estimation from pressure imaging,”  International journal of computer assisted radiology and surgery, vol. 14, no. 3, pp. 517–524, 2019.
  * [61] Carmen Bisogni and Chiara Pero,  “Ifepe: On the impact of facial expression in head pose estimation,”  in International Conference on Pattern Recognition. Springer, 2021, pp. 486–500.
  * [62] Abdulrahman Alreshidi and Mohib Ullah,  “Facial emotion recognition using hybrid features,”  in Informatics. Multidisciplinary Digital Publishing Institute, 2020, vol. 7, p. 6.
  * [63] Tewodros Legesse Munea, Yalew Zelalem Jembre, Halefom Tekle Weldegebriel, Longbiao Chen, Chenxi Huang, and Chenhui Yang,  “The progress of human pose estimation: a survey and taxonomy of models applied in 2d human pose estimation,”  IEEE Access, vol. 8, pp. 133330–133348, 2020.
  * [64] Sheng Jin, Xujie Ma, Zhipeng Han, Yue Wu, Wei Yang, Wentao Liu, Chen Qian, and Wanli Ouyang,  “Towards multi-person pose tracking: Bottom-up and top-down methods,”  in ICCV PoseTrack Workshop, 2017, vol. 2, p. 7.
  * [65] Mykhaylo Andriluka, Stefan Roth, and Bernt Schiele,  “Pictorial structures revisited: People detection and articulated pose estimation,”  in 2009 IEEE conference on computer vision and pattern recognition. IEEE, 2009, pp. 1014–1021.
  * [66] Yi Yang and Deva Ramanan,  “Articulated pose estimation with flexible mixtures-of-parts,”  in CVPR 2011. IEEE, 2011, pp. 1385–1392.
  * [67] Nikolaos Sarafianos, Bogdan Boteanu, Bogdan Ionescu, and Ioannis A Kakadiaris,  “3d human pose estimation: A review of the literature and analysis of covariates,”  Computer Vision and Image Understanding, vol. 152, pp. 1–20, 2016\.
  * [68] Yucheng Chen, Yingli Tian, and Mingyi He,  “Monocular human pose estimation: A survey of deep learning-based methods,”  Computer Vision and Image Understanding, vol. 192, pp. 102897, 2020\.
  * [69] Cristian Sminchisescu,  “3d human motion analysis in monocular video: techniques and challenges,”  in Human motion, pp. 185–211. Springer, 2008.
  * [70] Michael B Holte, Cuong Tran, Mohan M Trivedi, and Thomas B Moeslund,  “Human pose estimation and activity recognition from multi-view videos: Comparative explorations of recent developments,”  IEEE Journal of selected topics in signal processing, vol. 6, no. 5, pp. 538–552, 2012.
  * [71] Wenjuan Gong, Xuena Zhang, Jordi Gonzàlez, Andrews Sobral, Thierry Bouwmans, Changhe Tu, and El-hadi Zahzah,  “Human pose estimation from monocular images: A comprehensive survey,”  Sensors, vol. 16, no. 12, pp. 1966, 2016.
  * [72] Hong-Bo Zhang, Qing Lei, Bi-Neng Zhong, Ji-Xiang Du, and JiaLin Peng,  “A survey on human pose estimation,”  Intelligent Automation & Soft Computing, vol. 22, no. 3, pp. 483–489, 2016.
  * [73] Yanming Guo, Yu Liu, Ard Oerlemans, Songyang Lao, Song Wu, and Michael S Lew,  “Deep learning for visual understanding: A review,”  Neurocomputing, vol. 187, pp. 27–48, 2016.
  * [74] Qi Dang, Jianqin Yin, Bin Wang, and Wenqing Zheng,  “Deep learning based 2d human pose estimation: A survey,”  Tsinghua Science and Technology, vol. 24, no. 6, pp. 663–676, 2019\.
  * [75] Cristian Sminchisescu,  Estimation algorithms for ambiguous visual models: Three dimensional human modeling and motion reconstruction in monocular video sequences,  Ph.D. thesis, Institut National Polytechnique de Grenoble-INPG, 2002.
  * [76] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick,  “Microsoft coco: Common objects in context,”  in European conference on computer vision. Springer, 2014, pp. 740–755.
  * [77] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele,  “2d human pose estimation: New benchmark and state of the art analysis,”  in Proceedings of the IEEE Conference on computer Vision and Pattern Recognition, 2014, pp. 3686–3693.
  * [78] Jia-Bin Huang and Ming-Hsuan Yang,  “Estimating human pose from occluded images,”  in Asian Conference on Computer Vision. Springer, 2009, pp. 48–60.
  * [79] Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh,  “Realtime multi-person 2d pose estimation using part affinity fields,”  in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017, pp. 7291–7299.
  * [80] Mykhaylo Andriluka, Leonid Pishchulin, Peter Gehler, and Bernt Schiele,  ,”  .
  * [81] Xinlei Chen, Hao Fang, Tsung-Yi Lin, Ramakrishna Vedantam, Saurabh Gupta, Piotr Dollár, and C Lawrence Zitnick,  “Microsoft coco captions: Data collection and evaluation server,”  arXiv preprint arXiv:1504.00325, 2015.
  * [82] Hanbyul Joo, Tomas Simon, Xulong Li, Hao Liu, Lei Tan, Lin Gui, Sean Banerjee, Timothy Godisart, Bart Nabbe, Iain Matthews, et al.,  “Panoptic studio: A massively multiview system for social interaction capture,”  IEEE transactions on pattern analysis and machine intelligence, vol. 41, no. 1, pp. 190–204, 2017.
  * [83] Vittorio Ferrari, Manuel Marin-Jimenez, and Andrew Zisserman,  “Progressive search space reduction for human pose estimation,”  in 2008 IEEE Conference on Computer Vision and Pattern Recognition. IEEE, 2008, pp. 1–8.
  * [84] Yi Yang and Deva Ramanan,  “Articulated human detection with flexible mixtures of parts,”  IEEE transactions on pattern analysis and machine intelligence, vol. 35, no. 12, pp. 2878–2890, 2012.
  * [85] Alejandro Newell, Zhiao Huang, and Jia Deng,  “Associative embedding: End-to-end learning for joint detection and grouping,”  arXiv preprint arXiv:1611.05424, 2016.
  * [86] Dushyant Mehta, Oleksandr Sotnychenko, Franziska Mueller, Weipeng Xu, Mohamed Elgharib, Pascal Fua, Hans-Peter Seidel, Helge Rhodin, Gerard Pons-Moll, and Christian Theobalt,  “Xnect: Real-time multi-person 3d motion capture with a single rgb camera,”  ACM Transactions on Graphics (TOG), vol. 39, no. 4, pp. 82–1, 2020\.
  * [87] Andrei Zanfir, Elisabeta Marinoiu, Mihai Zanfir, Alin-Ionut Popa, and Cristian Sminchisescu,  “Deep network for the integrated 3d sensing of multiple people in natural images,”  Advances in Neural Information Processing Systems, vol. 31, pp. 8410–8419, 2018.