∎

11institutetext: Ruixu Liu 22institutetext: University of Dayton, Ohio, USA
22email<EMAIL_ADDRESS>
# Enhanced 3D Human Pose Estimation from Videos by using Attention-Based
Neural Network with Dilated Convolutions

Ruixu Liu  Ju Shen  He Wang  Chen Chen  Sen-ching Cheung  Vijayan K. Asari

###### Abstract

The attention mechanism provides a sequential prediction framework for
learning spatial models with enhanced implicit temporal consistency. In this
work, we show a systematic design (from 2D to 3D) for how conventional
networks and other forms of constraints can be incorporated into the attention
framework for learning long-range dependencies for the task of pose
estimation. The contribution of this paper is to provide a systematic approach
for designing and training of attention-based models for the end-to-end pose
estimation, with the flexibility and scalability of arbitrary video sequences
as input. We achieve this by adapting temporal receptive field via a multi-
scale structure of dilated convolutions. Besides, the proposed architecture
can be easily adapted to a causal model enabling real-time performance. Any
off-the-shelf 2D pose estimation systems, e.g. Mocap libraries, can be easily
integrated in an ad-hoc fashion. Our method achieves the state-of-the-art
performance and outperforms existing methods by reducing the mean per joint
position error to 33.4 mm111Code is available at:
https://github.com/lrxjason/Attention3DHumanPose on Human3.6M dataset.

###### Keywords:

3D Human Pose Motion Reconstruction Monocular Capture Performance-driven
Retargeting Attention Multi-scale Dilation

##  1 Introduction

We introduced attention mechanism for the task of articulated 3D pose
reconstruction from videos in the recent work (Liu et al., 2020b), which
exploits the temporal contexts of long-range dependencies across frames. The
ability to adaptively identify important frames or tensors output from each
deep net layer and combine them with the advantages afforded by convolutional
architectures allows for globally optimal inference through simultaneous
processing . The concept of “attention” is to learn optimized global alignment
between pairwise data and has gained recent success in the integration with
deep networks for processing mono/multi-modal data, such as text-to-speech
matching (Chorowski et al., 2015), neural machine translation (Bahdanau et
al., 2016) and 2D human pose estimation (Chu et al., 2017). In this paper, we
extend our original attention model further by integrating it with deep
networks in both 2D and 3D domain, leading to improved estimation while
preserving natural temporal coherence in videos.

Articulated 3D human pose estimation from unconstrained single images or
videos is considered as an ill-posed problem due to the nonlinearity of human
dynamics, occlusions, and the high-dimensional variability introduced in the
wild. Traditional approaches such as multi-view capture (Amin et al., 2013),
marker based systems (Mandery et al., 2015) and multi-modal sensing (Palmero
et al., 2016) require a laborious setup process and are not practical for
applications in the less controlled environment. Recent efforts of using deep
architectures have significantly advanced the state-of-the-art in 3D pose
reasoning (Toshev and Szegedy, 2014; Neverova et al., 2014). The end-to-end
learning process alleviates the need of using tailor-made features or spatial
constraints, thereby minimizing the characteristic errors such as double-
counting image evidence (Ferrari et al., 2009). While vast and powerful deep
models on 3D pose prediction are emerging (from convolutional neural network
(CNN) (Pavlakos et al., 2017; Tekin et al., 2016; Li et al., 2015) to
generative adversarial networks (GAN) (Yang et al., 2018; Chen et al., 2019)),
many of these approaches focus on a single image inference, which is inclined
to jittery motion or inexact body configuration. To resolve this, temporal
information is taken into account for better motion consistency. Existing
works can be generally classified into two categories: _direct 3D estimation_
and _2D-to-3D estimation_ (Zhou et al., 2016b; Chen et al., 2016). The former
explores the possibility of jointly extracting both 2D and 3D poses in a
holistic manner (Pavlakos et al., 2017; Varol et al., 2017); while the latter
decouples the estimation into two steps: 2D body part detection and 3D
correspondence inference (Chen and Ramanan, 2017; Bogo et al., 2016; Zhou et
al., 2016b). We refer readers to the recent survey for more details of their
respective advantages (Martinez et al., 2017).

Our approach falls under the category of _2D-to-3D estimation_ with three key
contributions:

  1. 1.

Development of a systematic approach for designing and training of attention-
based models for pose estimation in three levels: 2D joints attention,
3D-to-2D projection attention, and 3D pose attention.

  2. 2.

Learning of implicit dependencies in large temporal receptive fields via a
multi-scale structure of dilated convolutions.

  3. 3.

Design of a systematic architecture for the integration of the attention-based
model and dilation convolutional structure to enhance 3D pose inference to
facilitate performance driven animation applications.

Experimental evaluations show that the resulting system can reach almost the
same level of estimation accuracy under both causal or non-causal conditions,
making it very attractive for real-time or consumer-level applications. To
date, state-of-the-art results on video-based _2D-to-3D estimation_ can be
achieved by a semi-supervised approach (Pavllo et al., 2019) or a layer
normalized LSTM approach (Hossain et al., 2018). Our model can further improve
the performance in both quantitative accuracy and qualitative evaluation. The
simple requirement of our framework makes it well suited for interactive
applications like computer games, virtual communication, and avatar animation
re-targeting from videos. Given a video with continuous body movements and 3D
avatars as input, we transfer the captured pose and motion from the subject
video to a target character. In Fig. 1, we show an example of how the solution
can be employed in performance-based animations from videos. In this example,
we create six 3D avatars with different shapes and appearances and take six
different videos as input. There are not any constraints (e.g., camera
intrinsic and extrinsic parameters, pose complexities, or background
environment settings) about these input videos, which can be downloaded from
any online sources, such as YouTube. By using the proposed technique, it
enables automated body pose extraction from the video streams and applies
motion re-targeting to the corresponding characters in the scene. The green
arrows at the top of Fig. 1 indicates associated video for each character. The
subsequent frames demonstrate the result of automatic motion transferring from
the video to the 3D characters.

(a)

(b)

(c)

Figure 1: An application that shows 3D avatars re-targeting from 2D video
streams.

##  2 Related Works

Figure 2:  An example of a 4-layer architecture for attention-based temporal
convolutional neural network (ATCN). In this architecture, all the kernel
sizes are 3. In practice, different layers can have different kernel sizes.

Articulated pose estimation from an unconstrained video has been studied for
decades. Early work relies on graphical or restrictive models to account for
the high degree of freedom and dependencies among body parts, such as tree
structures (Andriluka et al., 2009; Yang and Ramanan, 2011; Amin et al.,
2013), and pictorial structures (Andriluka et al., 2009). These methods often
introduce a large number of parameters that require careful and manual tuning
using techniques such as piecewise approximation. The performance of graphical
model based approaches have been surpassed by convolutional neural networks
(CNNs) (Sarafianos et al., 2016; Pavlakos et al., 2017), which can learn an
automated representation that disentangles the dependencies among output
variables without a tailor-made solver.

For the last few years, various CNN based architectures have been proposed.
For example, (Tekin et al., 2016) trains an auto-encoder to project human
joint positions to a high dimensional space to enforce structural constraints.
(Park et al., 2016) estimates the 3D pose by propagating the 2D classification
results to the 3D pose regressors inside a neural network (Park et al., 2016).
A kinematic object model composing of bones and joints is introduced in (Zhou
et al., 2016a) to guarantee the geometric validity of the estimated human
body. A comprehensive list of convolutional systems can be found in the survey
presented in (Sarafianos et al., 2016).

Our contribution to this rich body of works lies in the introduction of an
attention based mechanism to the body pose estimation problem. The traditional
concept of “attention” is to provide an optimal matching strategy that
globally aligns pairwise data from the same domain, e.g., word-to-word or
phrase-to-phrase alignment in sentences (Yao et al., 2013), or across
different modalities, e.g., text-to-speech (Chorowski et al., 2015) and text-
to-image (Xu et al., 2015) in domain transformation. Prior work on attention
in deep learning (DL) mostly addresses long short-term memory networks (LSTMs)
(Hochreiter and Schmidhuber, 1997) and recently it has gained popularity in
training neural networks (Yin et al., 2016). Recent research indicates that
certain convolutional architectures can reach state-of-the-art accuracy in
audio synthesis, word-level language modeling, and machine translation (Oord
et al., 2016; Kalchbrenner et al., 2016; Dauphin et al., 2017). Compared to
the language modeling architecture of (Dauphin et al., 2017), temporal
convolutional networks (TCNs) (Bai et al., 2018) do not use gating mechanisms
and have much longer memory. Our 3D human pose estimation and reconstruction
network integrates the attention units and multi-scale dilation units to the
TCN architecture.

As mentioned earlier, there are recent works that take multiple frames with 2D
detection as the input for 3D prediction such as the LSTM-based method
(Hossain et al., 2018) and a TCN based approach with semi-supervised training
(Pavllo et al., 2019). For the LSTM-based system, the frames have to be
processed sequentially based on time steps, while we propose to process all of
the frames in parallel for 3D pose estimation. Another objective should be
that any estimation failure of one frame would not affect the other frames. In
our proposed work, we also employ some similarity to the TCN-based approach as
in (Pavllo et al., 2019; Chen et al., 2020; Liu et al., 2020a) along with the
usage of a voting mechanism to select important frames for prediction. In
addition, we incorporate the following three distinct features in our proposed
method:

(i) Instead of making a “hard” decision on a subset of frames, we use a “soft”
decision by considering all the frames.

(ii) Along with the “soft” decisions to the input frames, we apply all the
immediate outputs from every layer through the network, thereby expanding the
scope of selection to cover both raw frames and generated features.

(iii) We use a multi-scale dilated convolution that enables us to have a broad
range of frame selection without increasing the number of neural net layers.

##  3 The Attention-based Approach

In this section, we present an overview of the proposed system for 3D pose
estimation from a 2D video stream and show how our attention model guides the
network to adaptively identify significant portion of each deep neural net
layer’s output resulting in an enhanced estimation.

###  3.1 Network Design

Fig. 2 (right) depicts the overall architecture of our attention-based neural
network. It takes a sequence of $n$ frames with 2D joint positions as the
input and outputs the estimated 3D pose for the target frame as labeled. The
framework involves two types of processing modules: the _Temporal Attention_
module (indicated by the long green bars) and the _Kernel Attention_ module
(indicated by the gray squares). The kernel attention module can be further
categorized as TCN Units (in dark grey color) and Feature Aggregation (in
light grey color) (He et al., 2016). By viewing the graphical model vertically
from the top, one can notice the two attention modules distribute in an
interlacing pattern that a row of kernel attention modules situate right below
a temporal attention module. We regard these two adjacent modules as one
_layer_ , which has the same notion as a neural net layer. According to the
functionalities, the layers can be grouped as _top layer_ , _middle layers_ ,
and _bottom layer_. Note that the top layer only has TCN units for the kernel
module, while the bottom layer only has a feature aggregation to deliver the
result. It is also worth mentioning that the number of middle layers can be
varied depending on the receptive field setting, which will be discussed in
section 5.3.

###  3.2 Temporal Attention

The goal of the temporal attention module is to provide a contribution metric
for the output tensors. Each attention module produces a set of scalars,
$\\{\omega_{0}^{(l)},\omega_{1}^{(l)},\dots\\}$, weighing the significance of
different tensors within a layer:

$\mathbf{W}^{(l)}\otimes\mathbf{T}^{(l)}\overset{\Delta}{=}\left\\{\omega_{0}^{(l)}\otimes\mathcal{T}_{0}^{(l)},\dots,\omega_{\lambda_{l}-1}^{(l)}\otimes\mathcal{T}_{\lambda_{l}-1}^{(l)}\right\\}$
(1)

where $l$ and $\lambda_{l}$ indicate the layer index and the number of tensors
output from the $l^{(th)}$ layer. We use $\mathcal{T}_{u}^{(l)}$ to denote the
$u^{th}$ tensor output from the $l^{th}$ layer. The bold format of
$\mathbf{W}\otimes\mathbf{T}$ is a compacted vector. Note for the top layer,
the input to the TCN units is just the 2D joints. The choice for computing
their attention scores can be flexible. A commonly used scheme is the
multilayer perceptron strategy for optimal feature set selection (Ruck et al.,
1990). Empirically, we achieve desirable result by simply computing the
_normalized cross-correlation ( $ncc$)_ that measures the positive cosine
similarity between $\mathbf{P}_{i}$ and $\mathbf{P}_{t}$ on their 2D joint
positions (Yoo and Han, 2009):

$\mathbf{W}^{(0)}=\left[ncc(\mathbf{P}_{0},\mathbf{P}_{t}),\dots,ncc(\mathbf{P}_{n-1},\mathbf{P}_{t})\right]^{T}$
(2)

where $\mathbf{P}_{0},\dots,\mathbf{P}_{n-1}$ are the 2D joint positions. $t$
indicates the target frame index. The output $\mathbf{W}^{(0)}$ is forwarded
to the attention matrix $\boldsymbol{\theta_{t}}^{(l)}$ to produce tensor
weights for the subsequent layers.

$\mathbf{W}^{(l)}=sig\left(\boldsymbol{\theta_{t}}^{(l)T}\mathbf{W}^{(l-1)}\right)\mbox{
, for }l\in[1,L-2]$ (3)

where $sig(\cdot)$ is the sigmoid activation function. We require the
dimension of $\boldsymbol{\theta_{t}}^{(l)}\in\mathcal{R}^{F^{\prime}\times
F}$ matching the number of output tensors between layers $l-1$ and $l$, s.t.
$F^{\prime}=\lambda_{l-1}$ and $F=\lambda_{l}$.

###  3.3 Kernel Attention

Similar to the temporal attention that determines a tensor weight distribution
$\mathbf{W}^{(l)}$ within layer $l$, the kernel attention module assigns a
channel weight distribution within a tensor, denoted as
$\widetilde{\boldsymbol{W}}^{(l)}$. Fig. 2 (right) depicts the steps on how an
updated tensor $\mathbf{T}_{final}^{(l)}$ is generated through the weight
adjustment. Given an input tensor $\mathbf{T}^{(l)}\in\mathcal{R}^{C\times
F}$, we generate $M$ new tensors $\widetilde{T}^{(l)}_{m}$ using $M$ TCN units
with different dilation rates.

These $M$ tensors are fused together through element-wise summation:
$\widetilde{\mathbf{T}}^{(l)}=\sum_{m=1}^{M}\widetilde{T}^{(l)}_{m}$, which is
fed into a global average pooling (GAP) layer to generate channel-wise
statistics $\widetilde{\mathcal{T}}^{(l)}_{c}\in\mathcal{R}^{C\times 1}$. The
channel number $C$ is acquired through a TCN unit as discussed in the ablation
study. The output $\widetilde{\mathcal{T}}^{(l)}_{c}$ is forwarded to a fully-
connected layer to learn the relationship among features of different kernel
sizes:
$\widetilde{\mathcal{T}}^{(l)}_{r}=\boldsymbol{\theta_{r}}^{(l)}\widetilde{\mathcal{T}}^{(l)}_{c}$.
The role of matrix $\boldsymbol{\theta_{r}}^{(l)}\in\mathcal{R}^{r\times C}$
is to reduce the channel dimension to $r$. Guided by the compacted feature
descriptor $\widetilde{\mathcal{T}}^{(l)}_{r}$, $M$ vectors are generated
(indicated by the yellow cuboids) through a second fully-connected layer
across channels. Their kernel attention weights are computed by a softmax
function:

$\widetilde{\boldsymbol{W}}^{(l)}\overset{\Delta}{=}\left\\{\widetilde{W}_{1}^{(l)},...,\widetilde{W}_{M}^{(l)}\left|\widetilde{W}_{m}^{(l)}=\frac{e^{\boldsymbol{\theta_{m}}^{(l)}\widetilde{\mathcal{T}}^{(l)}_{r}}}{\sum_{m=1}^{M}e^{\boldsymbol{\theta_{m}}^{(l)}\widetilde{\mathcal{T}}^{(l)}_{r}}}\right\\}\right.$
(4)

where $\boldsymbol{\theta_{m}}^{(l)}\in\mathcal{R}^{C\times r}$ are the kernel
attention parameters and $\sum_{m=1}^{M}W_{m}^{(l)}=1$. Based on the weight
distribution, we finally obtain the output tensor:

$\mathbf{T}_{final}^{(l)}\overset{\Delta}{=}\sum_{m=1}^{M}\widetilde{W}_{m}^{(l)}\otimes\widetilde{T}_{m}^{(l)}$
(5)

The channel update procedure can be further decomposed as:

$\widetilde{W}_{m}^{(l)}\otimes\widetilde{T}_{m}^{(l)}=\left\\{\widetilde{\omega}_{1}^{(l)}\otimes\widetilde{\mathcal{T}}_{1}^{(l)},\dots,\widetilde{\omega}_{C}^{(l)}\otimes\widetilde{\mathcal{T}}_{C}^{(l)}\right\\}$
(6)

This shares the same format as the tensor distribution process (equation 1) in
the temporal attention module but focuses on the channel distribution. The
temporal attention parameters $\boldsymbol{\theta_{t}}^{(l)}$ and kernel
attention parameters $\boldsymbol{\theta_{r}}^{(l)}$,
$\boldsymbol{\theta_{m}}^{(l)}$ for $l\in[1,L-2]$ are learned through mini-
batch stochastic gradient descent (SGD) in the same manner as the TCN unit
training (Bottou, 2010).

##  4 Integration with Dilated Convolutions

Figure 3: The model of temporal dilated convolution network. As the level
index increases, the receptive field over frames (layer index = 0) or tensors
(layer index $\geq$ 0) increases.

For the proposed attention model, a large receptive field is crucial to learn
long range temporal relationships across frames, thereby enhancing the
estimation consistency. However, with more frames feeding into the network,
the number of neural layers increases together with more training parameters.
To avoid vanishing gradients or other superfluous layers problems (Martinez et
al., 2017), we devise a multi-scale dilation (MDC) strategy by integrating
dilated convolutions.

Fig. 3 shows our dilated network architecture. For visualization purpose, we
project the network into an $xyz$ space. The $xy$ plane has the same
configuration as the network in Fig. 2, with the combination of temporal and
kernel attention modules along the $x$ direction, and layers layout along the
$y$ direction. As an extension, we place the dilated convolution units (DCUs)
along the $z$ direction. This $z$-axis is labeled as _level_ s to differ from
the _layer_ concept along the $y$ direction. As the level index increases, the
receptive field grows with increasing dilation size while reducing the number
of DCUs.

##  5 Experimental Evaluation

(a) Prototype 1: Layer 4 $\times$ Level 2 (L4$\times$V2)

(b) Prototype 2: Layer 5 $\times$ Level 3 (L5$\times$V3)

Figure 4: Architectures of input/output data flows across different dilated
convolution units. Inside each unit, the numbers represent the unit
configuration, e.g. K3D9, 1024 means kernel size is 3, dilation rate is 9, and
tensor depth or number of channels is 1024. M3D3, 1024 means TCN units are 3,
dilation rate is 9, and tensor depth or number of channels is 1024.

This section discusses our system implementation as well as the evaluation
results compared to the state-of-the-art techniques by using the standard pose
estimation protocols on public datasets. We first describe the configuration
and timings for each functional module, as well as the timings for the run-
time algorithm. Ablation studies of the system are conducted by analyzing each
component and discuss their performance and limitations. Then we evaluate the
estimation accuracy compared to other approaches as well as the ground truth.
Finally we demonstrate the robustness and flexibility of the proposed approach
on videos in the wild with various environment complexities and unknown camera
settings. Our model is generic and runs on novel users without requiring any
offline training or manual preprocessing steps. More extensive evaluation can
be found at our lab website 222Demo:
https://sites.google.com/a/udayton.edu/jshen1/pose3d.

Figure 5: Convergence characteristics for training and testing on three
prototypes.

###  5.1 Configuration and Computational Complexity

To investigate the practical feasibility of the proposed approach, we
implemented three prototypes with different layer $L$ and dilation level $V$
combinations: $L4\times V2\times N27$, $L5\times V3\times N81$, and $L6\times
V4\times N243$, where the last term $N$ indicates the corresponding input
frame number. Fig. 4 provides a deeper insight on unit configuration of the
prototypes: $L4\times V2\times N27$ and $L5\times V3\times N81$. By dropping
the $x$-axis from Fig. 3, it only displays the level and layer distribute in a
2D view. For simplicity, we use a black/gray rectangle shape to denote the
group of TCN units within a layer. At level $0$ , the TCN units are placed by
layers along the $y$-axis corresponding to the ones depicted in Fig. 3. From
level $1$, along the positive $z$-axis, different scaled dilated convolution
units are placed. As the level index grows, the number of dilated units
decreases due to the increasing receptive fields.

All the prototypes are implemented in native Python (Pytorch 1.0) and tested
on a NVIDIA TITAN RTX GPU without parallel optimization. Despite the
difference in layers and levels, all the prototypes present similar
convergence rate in training and testing, as shown in Fig. 5. With data
augmentation, the $L6\times V4$ setting demonstrates the best Mean Per Joint
Position Error (MPJPE) performance with approximately $16$ hrs training on
$1.6$M frames. The optimizer is Ranger (Zhang et al., 2019; Liu et al., 2019),
and the learning rate is 0.001 with decay=0.05 for 80 epoch, the batch size is
1024 and dropout is 0.2. For real-time inference, it can reach $3000$ FPS.

Method | Parameters | MPJPE
---|---|---
ECCV (Hossain et al., 2018) | 16.96M | 41.6
CVPR (Pavllo et al., 2019) (27f) | 8.56M | 40.6
Ours (L4 x V2) | 5.69M | 39.1
CVPR (Pavllo et al., 2019) (81f) | 12.75M | 38.7
Ours (L5 x V3) | 8.46M | 37.0
CVPR (Pavllo et al., 2019) (243f) | 16.95M | 37.8
Ours (L6 x V4) | 11.22M | 33.4
Table 1: Computational complexity performance in terms of the number of
involved learning parameters.

Table 1 compares our model with TCN based semi supervised approach (Pavllo et
al., 2019), and the layer normalized LSTM approach (Hossain et al., 2018) in
terms of the computational complexity. Our model requires fewer parameters for
learning the model while achieving better accuracy. In particular, the input
numbers of frames for our three prototypes exactly match the corresponding
ones in (Pavllo et al., 2019) (i.e., $\\#243$, $\\#81$, and $\\#27$), while
ours saves 2M parameters on average.

###  5.2 Datasets and Evaluation Protocols

Our quantitative evaluation is conducted on two most commonly used datasets:
Human3.6M (Ionescu et al., 2013) and HumanEva (Sigal et al., 2010). We also
applied our approach to some challenging YouTube videos, which include fast
motion activities and low-resolution frames. It would be extremely difficult
to obtain meaningful 2D detection for those challenging videos collected in
the wild. For the Human3.6M, we follow the same training and validation
schemes as in the previous works (Martinez et al., 2017; Yang et al., 2018;
Hossain et al., 2018; Pavllo et al., 2019). Specifically, subjects S1, S5, S6,
S7, and S8 are used for training, and subjects S9 and S11 are used for
testing. In the same manner, we conducted training/testing on the HumanEva (a
comparatively smaller dataset) with the ”Walk” and ”Jog” actions performed by
subjects S1, S2, and S3.

For both datasets, we use the standard evaluation metrics MPJPE and P-MPJPE to
measure the offset between the estimation result and ground-truth (GT)
relative to the root node in millimeters (Ionescu et al., 2013). Two protocols
are involved in the experiment: Protocol 1 computes the mean Euclidean
distance across all the joints after aligning the root joints (i.e., _pelvis_)
between the predicted and ground-truth poses, referred as MPJPE (Fang et al.,
2018; Lee et al., 2018; Pavlakos et al., 2017; Luvizon et al., 2018). Protocol
2 applies additional similarity transformation Procrustes analysis (Lepetit et
al., 2005) to the predicted pose as an enhancement and it is called P-MPJPE
(Martinez et al., 2017; Hossain et al., 2018; Yang et al., 2018; Pavllo et
al., 2019). In contrast to protocol 1, this evaluation can be more robust to
individual joint prediction failure due to the rigid alignment. It is worth
mentioning that some researchers also use another protocol by performing a
scale alignment on the predicted pose and it is named as N-MPJPE (Rhodin et
al., 2018). Since it has a similar goal as protocol 2 with relatively less
transformation, the error usually drops between the outputs produced by
protocols 1 & 2\. As such, the accuracy performance should be sufficiently
evaluated by using these two protocols.

###  5.3 Ablation Studies

To verify the impact and performance of each component in the network, we
conducted ablation experiments on the Human3.6M dataset under $Protocol\\#1$.

TCN Unit Channels:  we first investigated how the channel number $C$ affects
the performance between TCN units and temporal attention models. In our test,
we used both the CPN and GT as the 2D input. Starting with a receptive field
of $n=3\times 3\times 3=27$, as we increase the channels ($C\leq 512$), the
MPJPE drops down significantly. However, the MPJPE changes slowly when $C$
grows between $512$ and $1024$, and remains almost stable afterwards. As shown
in Fig. 6, with the CPN input, a marginal improvement is yielded from MPJPE
$49.9mm$ at $C=1024$ to $49.6mm$ at $C=2048$. A similar curve shape can be
observed for the GT input. Considering the computation load with more
parameters introduced, we chose $C=1024$ in our experiments.

Figure 6: The impact of channel number on MPJPE. CPN: cascaded pyramid network
and GT: ground-truth.

Kernel Attention:  Table 2 shows how the setting of different parameters
inside the Kernel Attention module impacts the performance under
$Protocol\\#1$. The left three columns list the main variables. For validation
purposes, we divide the configuration into three groups in row-wise. Within
each group, we assign different values in one variable while keeping the other
two fixed. The items in bold represent the best individual setting for each
group. Empirically, we chose the combination of $M=3$, $G=8$, and $r=128$ as
the optimal setting (labeled in box). Note, we select $G=8$ instead of the
individual best assignment $G=2$, which introduces a larger number of
parameters with negligible MPJPE improvement.

Kernels | Groups | Channels | Parameters | P1
---|---|---|---|---
M=1 | G=1 | - | 16.95M | 37.8
M=2 | G=8 | r=128 | 9.14M | 37.1
M=3 | G=8 | r=128 | 11.22M | 35.5
M=4 | G=8 | r=128 | 13.36M | 38.0
M=3 | G=1 | r=128 | 44.25M | 37.4
M=3 | G=2 | r=128 | 25.41M | 35.3
M=3 | G=4 | r=128 | 15.97M | 35.6
M=3 | $None$ | r=128 | 11.25M | $None$
M=3 | G=16 | r=128 | 8.89M | 37.3
M=3 | G=8 | r=64 | 10.20M | 35.9
M=3 | G=8 | r=128 | 11.25M | 35.5
M=3 | G=8 | r=256 | 13.35M | 36.2
Table 2: Ablation study on different parameters in our kernel attention model.
Here, we use receptive field $n=3\times 3\times 3\times 3\times 3=243$. The
evaluation is performed on Human3.6M under $Protocol\\#1$ with MPJPE (mm).

In Table 3, we discuss the choice of different types of receptive fields and
how it affects the network performance. The first column shows various layer
configurations, which generate different receptive fields, ranging from $n=27$
to $n=1029$. To validate the impact of $n$, we fix the other parameters, i.e.
$M=3$, $G=8$, $r=128$. Note that for a network with smaller number of layers
(e.g. $L=3$), a larger receptive field may reduce the error more effectively.
For example, increasing the receptive field from $n=3\times 3\times 3=27$ to
$n=3\times 3\times 7=147$, the MPJPE drops from $40.6$ to $36.8$ . However,
for a deeper network, a larger receptive field may not be always optimal, e.g.
when $n=1029$, MPJPE $=37.0$. Empirically, we obtained the best performance
with the setting of $n=243$ and $L=5$, as indicated in the last row.

Receptive fields | Kernels | Groups | Channels | Parameters | P1
---|---|---|---|---|---
3$\times$3$\times$3=27 | M=1 | G=1 | - | 8.56M | 40.6
3$\times$3$\times$3=27 | M=2 | G=4 | r=128 | 6.21M | 40.0
3$\times$5$\times$3=45 | M=2 | G=4 | r=128 | 6.21M | 39.9
3$\times$5$\times$5=75 | M=2 | G=4 | r=128 | 6.21M | 38.5
3$\times$3$\times$3=27 | M=3 | G=8 | r=128 | 5.69M | 39.5
3$\times$5$\times$3=45 | M=3 | G=8 | r=128 | 5.69M | 39.2
3$\times$5$\times$5=75 | M=3 | G=8 | r=128 | 5.69M | 38.2
3$\times$7$\times$7=147 | M=3 | G=8 | r=128 | 5.69M | 36.8
3$\times$3$\times$3$\times$3=81 | M=3 | G=8 | r=128 | 8.46M | 37.8
3$\times$5$\times$5$\times$5=375 | M=3 | G=8 | r =128 | 8.46M | 36.6
3$\times$7$\times$7$\times$7=1029 | M=3 | G=8 | r=128 | 8.46M | 37.0
3$\times$3$\times$3$\times$3$\times$3=243 | M=3 | G=8 | r=128 | 11.25M | 35.5

Table 3: Ablation study on different receptive fields in our kernel attention
model. The evaluation is performed on Human3.6M under $Protocol\\#1$ with
MPJPE (mm).

Multi-Scale Dilation:  To evaluate the impact of the dilation component on the
network, we tested the system with and without dilation and compared their
individual outcomes. In the same way, the GT and CPN 2D detectors are used as
input and being tested on the Human3.6M dataset under $Protocol\\#1$. Table 4
demonstrates the integration of attention, and multi-scale dilation components
surpass their individual performance with the minimum MPJPE for all the three
prototypes. We also found the attention model makes an increasingly
significant contribution as the layer number grows. This is because more
layers lead to a larger receptive field, allowing the multi-scale dilation to
capture long-term dependency across frames. The effect is more noticeable when
fast motion or self-occlusion present in videos.

Method Model | $n=27$ | $n=81$ | $n=243$
---|---|---|---
Attention model (CPN) | 49.1 | 47.2 | 45.7
Multi-Scale Dilation model (CPN) | 50.3 | 49.8 | 49.1
Attention and Dilation (CPN) | 49.0 | 46.5 | 45.1
Attention model (GT) | 39.5 | 37.8 | 35.5
Multi-Scale Dilation model (GT) | 39.2 | 37.2 | 35.3
Attention and Dilation (GT) | 38.9 | 36.2 | 33.4

Table 4: Ablation study on different components in our method. The evaluation
is performed on Human3.6M under $Protocol\\#1$ with MPJPE (mm).

Step by step performance enhancement:  Here we list all the steps and
additional modules used to obtain the performance. The step-by-step gains
brought by each component are illustrated in Table 5.

Model Method | CPN | GT
---|---|---
Baseline 27 frames | 51.2 | 40.6
\+ Receptive field(243 frames) | 49.2 | 37.8
\+ Attention | 47.9 | 35.5
\+ Dilation | 47.2 | 34.7
\+ Project on 2D | 47.0 | 34.5
\+ 2D pose enhance | 46.5 | -
\+ Data augment | 44.8 | 33.4
Table 5: Ablation study on different components in our method. The evaluation
is performed on Human3.6M under $Protocol\\#1$ with MPJPE (mm).

###  5.4 Comparison with State-of-the-Art

Method | Dir. | Disc. | Eat | Greet | Phone | Photo | Pose | Pur. | Sit | SitD. | Smoke | Wait | WalkD. | Walk | WalkT. | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
(Martinez et al., 2017) | 51.8 | 56.2 | 58.1 | 59.0 | 69.5 | 78.4 | 55.2 | 58.1 | 74.0 | 94.6 | 62.3 | 59.1 | 65.1 | 49.5 | 52.4 | 62.9
(Fang et al., 2018) | 50.1 | 54.3 | 57.0 | 57.1 | 66.6 | 73.3 | 53.4 | 55.7 | 72.8 | 88.6 | 60.3 | 57.7 | 62.7 | 47.5 | 50.6 | 60.4
(Yang et al., 2018) | 51.5 | 58.9 | 50.4 | 57.0 | 62.1 | 65.4 | 49.8 | 52.7 | 69.2 | 85.2 | 57.4 | 58.4 | 43.6 | 60.1 | 47.7 | 58.6
(Pavlakos et al., 2017) | 48.5 | 54.4 | 54.4 | 52.0 | 59.4 | 65.3 | 49.9 | 52.9 | 65.8 | 71.1 | 56.6 | 52.9 | 60.9 | 44.7 | 47.8 | 56.2
(luvizon2018) | 49.2 | 51.6 | 47.6 | 50.5 | 51.8 | 60.3 | 48.5 | 51.7 | 61.5 | 70.9 | 53.7 | 48.9 | 57.9 | 44.4 | 48.9 | 53.2
(Hossain et al., 2018) | 48.4 | 50.7 | 57.2 | 55.2 | 63.1 | 72.6 | 53.0 | 51.7 | 66.1 | 80.9 | 59.0 | 57.3 | 62.4 | 46.6 | 49.6 | 58.3
(Lee et al., 2018) | 40.2 | 49.2 | 47.8 | 52.6 | 50.1 | 75.0 | 50.2 | 43.0 | 55.8 | 73.9 | 54.1 | 55.6 | 58.2 | 43.3 | 43.3 | 52.8
(Dabral et al., 2018) | 44.8 | 50.4 | 44.7 | 49.0 | 52.9 | 61.4 | 43.5 | 45.5 | 63.1 | 87.3 | 51.7 | 48.5 | 52.2 | 37.6 | 41.9 | 52.1
(Zhao et al., 2019a) | 47.3 | 60.7 | 51.4 | 60.5 | 61.1 | 49.9 | 47.3 | 68.1 | 86.2 | 55.0 | 67.8 | 61.0 | 42.1 | 60.6 | 45.3 | 57.6
(Pavllo et al., 2019) | 45.2 | 46.7 | 43.3 | 45.6 | 48.1 | 55.1 | 44.6 | 44.3 | 57.3 | 65.8 | 47.1 | 44.0 | 49.0 | 32.8 | 33.9 | 46.8
(Cheng et al., 2019) | - | - | - | - | - | - | - | - | - | - | - | - | - | - | - | $\mathbf{44.8}^{+}$
Ours (n=243 CPN) | 41.8 | 44.0 | 41.1 | 43.9 | 47.4 | 54.1 | 42.1 | 42.2 | 55.3 | 63.6 | 45.3 | 42.7 | 45.3 | 31.3 | 32.2 | 44.8
(Martinez et al., 2017) | 37.7 | 44.4 | 40.3 | 42.1 | 48.2 | 54.9 | 44.4 | 42.1 | 54.6 | 58.0 | 45.1 | 46.4 | 47.6 | 36.4 | 40.4 | 45.5
(Hossain et al., 2018) | 35.2 | 40.8 | 37.2 | 37.4 | 43.2 | 44.0 | 38.9 | 35.6 | 42.3 | 44.6 | 39.7 | 39.7 | 40.2 | 32.8 | 35.5 | 39.2
(Lee et al., 2018) | 32.1 | 36.6 | 34.4 | 37.8 | 44.5 | 49.9 | 40.9 | 36.2 | 44.1 | 45.6 | 35.3 | 35.9 | 37.6 | 30.3 | 35.5 | 38.4
(Zhao et al., 2019a) | 37.8 | 49.4 | 37.6 | 40.9 | 45.1 | 41.4 | 40.1 | 48.3 | 50.1 | 42.2 | 53.5 | 44.3 | 40.5 | 47.3 | 39.0 | 43.8
(Pavllo et al., 2019) | 35.2 | 40.2 | 32.7 | 35.7 | 38.2 | 45.5 | 40.6 | 36.1 | 48.8 | 47.3 | 37.8 | 39.7 | 38.7 | 27.8 | 29.5 | 37.8
Ours (n=243 GT) | 33.0 | 35.7 | 31.7 | 32.4 | 32.1 | 36.5 | 37.2 | 32.6 | 40.7 | 41.4 | 32.6 | 33.1 | 30.9 | 24.9 | 25.9 | 33.4

Table 6: Protocol 1: Reconstruction Error on Human3.6M. Top-half: input 2D
joints are acquired by detection; bottom-half: input 2D joints with ground-
truth. (CPN) \- cascaded pyramid network, (GT) \- ground-truth, ($+$) - the
result without data augmentation using virtual cameras

Method | Dir. | Disc. | Eat | Greet | Phone | Photo | Pose | Pur. | Sit | SitD. | Smoke | Wait | WalkD. | Walk | WalkT. | Avg
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
(Martinez et al., 2017) | 39.5 | 43.2 | 46.4 | 47.0 | 51.0 | 56.0 | 41.4 | 40.6 | 56.5 | 69.4 | 49.2 | 45.0 | 49.5 | 38.0 | 43.1 | 47.7
(Fang et al., 2018) | 38.2 | 41.7 | 43.7 | 44.9 | 48.5 | 55.3 | 40.2 | 38.2 | 54.5 | 64.4 | 47.2 | 44.3 | 47.3 | 36.7 | 41.7 | 45.7
(Hossain et al., 2018) | 35.7 | 39.3 | 44.6 | 43.0 | 47.2 | 54.0 | 38.3 | 37.5 | 51.6 | 61.3 | 46.5 | 41.4 | 47.3 | 34.2 | 39.4 | 44.1
(Pavlakos et al., 2017) | 34.7 | 39.8 | 41.8 | 38.6 | 42.5 | 47.5 | 38.0 | 36.6 | 50.7 | 56.8 | 42.6 | 39.6 | 43.9 | 32.1 | 36.5 | 41.8
(Yang et al., 2018) | 26.9 | 30.9 | 36.3 | 39.9 | 43.9 | 47.4 | 28.8 | 29.4 | 36.9 | 58.4 | 41.5 | 30.5 | 29.5 | 42.5 | 32.2 | 37.7
(Dabral et al., 2018) | 28.0 | 30.7 | 39.1 | 34.4 | 37.1 | 28.9 | 31.2 | 39.3 | 60.6 | 39.3 | 44.8 | 31.1 | 25.3 | 37.8 | 28.4 | 36.3
(Pavllo et al., 2019) | 34.1 | 36.1 | 34.4 | 37.2 | 36.4 | 42.2 | 34.4 | 33.6 | 45.0 | 52.5 | 37.4 | 33.8 | 37.8 | 25.6 | 27.3 | 36.5
Ours (n=243 CPN) | 32.3 | 35.2 | 33.3 | 35.8 | 35.9 | 41.5 | 33.2 | 32.7 | 44.6 | 50.9 | 37.0 | 32.4 | 37.0 | 25.2 | 27.2 | 35.6

Table 7: Protocol 2: Reconstruction Error on Human3.6M with similarity
transformation.

|  | Walk |  |  | Jog |  |
---|---|---|---|---|---|---|---
| S1 | S2 | S3 | S1 | S2 | S3 | Avg
(Pavlakos et al., 2017) (MA) | 22.3 | 19.5 | 29.7 | 28.9 | 21.9 | 23.8 | 24.35
(Martinez et al., 2017) (SA) | 19.7 | 17.4 | 46.8 | 26.9 | 18.2 | 18.6 | 24.6
(Lee et al., 2018) (MA) | 18.6 | 19.9 | 30.5 | 25.7 | 16.8 | 17.7 | 21.5
(Pavllo et al., 2019)(MA) | 13.4 | 10.2 | 27.2 | 17.1 | 13.1 | 13.8 | 15.8
Ours (n=27 MA) | 13.1 | 9.8 | 26.8 | 16.9 | 12.8 | 13.3 | 15.4

Table 8: Protocol 2: Reconstruction Error on HumanEva. (MA) \- multi-action
model, (SA) \- single action model

(a) Protocol 1: joint error analysis across frames in Human3.6M _Walking S9_
left ankle.

(b) Protocol 1: joint error analysis across frames in Human3.6M _Smoking S9_
left elbow.

Figure 7: Joint-wise analysis across frames. Figure 8: Individual joint MPJPE
comparison with state-of-the-art.

In this subsection, we systematically analyze the performance of our proposed
method by comparing it with state-of-the-art. To fairly evaluate the accuracy,
we use the same training and testing datasets as others. Tables 6 \- 8
demonstrate the comparison by following Protocol 1 and 2. Tables 6 and 7
illustrate the Human3.6M results and Table 8 illustrates the results of
HumanEva. The results of each method are displayed in row-wise. Each column
indicates a different pose scene, e.g., walking, eating, etc. We highlight the
best and second best results in each column in bold and underline formats
respectively. The last column of each table shows the average results across
all the different pose scenes. Note that our model outperforms all the
existing approaches by reaching a minimum average error of $48.6$mm in MPJPE
and $37.7$mm in P-MPJPE. Admittedly, for some pose scenes, e.g., Phone, Eat,
our method does not achieve the best performance. This could be due to the
nature of the particular activities, for example, if the less noticeable
motion or only upper-body movement are involved, limited information is fed
into the attention layers to learn tensor distributions. However, if one
considers all the scenarios, our overall performance demonstrates higher
accuracy than other methods by a fair margin. In particular, under protocol 1,
our model reduces the best-reported error rate of MPJPE (Pavllo et al., 2019)
by approximate 3% using ground truth 2D pose as the input.

To further demonstrate the efficacy, we evaluated the performance and
advantage of our approach in three aspects:

  1. 1.

Joint-wise: Analyzing the accuracy of individual joint measurement with MPJPE
comparison

  2. 2.

Frame-wise: Tracking the average MPJPE of all the joints across frames

  3. 3.

Re-targeting-wise: Applying motion-retargeting by transferring the estimated
pose to a 3D avatar

We compare our approach with three state-of-the-art techniques, which
represent the best reported results on monocular video-based _2D-to-3D
estimation_ to date: the deep feedforward 2D-to-3D network (Martinez et al.,
2017), the layer-normalized LSTM based algorithm (Hossain et al., 2018), and
the dilated temporal convolution with semi-supervised training (Pavllo et al.,
2019). Fig. 8 demonstrates the joint-wise MPJPE for a selected frame from the
_WalkDog S11_ data. The top row shows the input 2D color image and its
corresponding estimated 3D poses by other methods. The histograms in the
second row show the quantified measurement on each joint, e.g., R-Knee, Nose,
Neck. Note that our approach, indicated by the green bar, achieves minimum
MPJPE among all the other methods in most of the joints. To further validate
the accuracy, we trace these individual joints across frames in the
corresponding video sequence and measure their MPJPE in the temporal space.
Fig. 7 plots the MPJPE curves over time (around 1400 frames) on two selected
joints: the left ankle from _Walking S9_ and the left elbow from _Smoking S9_.
Compared to the recent works (Martinez et al., 2017; Hossain et al., 2018;
Pavllo et al., 2019), our results yield low errors consistently through
learning the long-range dependencies using the multi-scale dilation
convolution.

(a) Pavllo et al., 2019

(b) Ground truth

(c) Ours

(d)

Figure 9: Comparison results: (top): side-by-side views of motion retargeting
results on a 3D avatar; the source is from frame 857 of _walking S9_ and frame
475 _posing S9_ in Human3.6M. (bottom): the average joint error comparison
across all the frames of the video walking S9 (Pavllo et al., 2019).

Figure 10: Frame-wise comparasion with state-of-the-art results.

In light of possible biases and uncertainties that individual joint may
introduce, we perform frame-wise analysis by taking the average MPJPE of all
the joint estimation in each frame and measure how it changes through a video
sequence. Fig. 10 shows the testing results on two scenes of the Human3D
dataset: _smoking S9_ and _photo S9_. For each scene, the top row presents the
estimated 3D pose results from the same frame produced by different methods.
Though it is hard to see the difference from the single frame, from the MPJPE
(the green number on the top-left corner of each pose result), our attention-
based model delivers the best result. In the second row of each scene, We
trace these average joint errors across all of the frames in the corresponding
video sequence. Our results maintain low MPJPE compared to the other methods.

To visually demonstrate the significance of the estimation improvement, we
apply animation retargeting to a 3D avatar by synthesizing the captured motion
from the same frame of the _Walking S9_ and _Posing S9_ sequences as shown in
Fig. 9. With the support of additional mesh surface driven by the pose, it
helps magnify the degree of body part arrangement that enhance the contrast of
estimation. From the side-by-side comparisons, one can easily see the
difference between the rendered results against the ground truth.
Specifically, the shadows of the legs and the right hand are differently
rendered due to the erroneous pose estimated using the method in (Pavllo et
al., 2019) while ours stay more aligned to the ground truth. The quantified
MPJPE for each joint estimation is shown in the correponding histograms right
below it. Fig. 11 shows more retargeting results on the same dataset for
different frames. The zoom-in views illustrate the details of the animated
characters of different pose configurations. For the _Posing S9_ in the first
row, our results bear the closest similarity as the ground truth with the
right arm of the character naturally hanging down the side of the body, while
others present more distinct arm gesture. The second row of Fig. 11
demonstrates the improvement of our approach on leg movement prediction with
optimistic estimate on the two legs relative positions. Note that this is just
one selected frame from the walking sequence, which is a common body activity
involving the alternate of left and right legs in a repetitive manner.
Accurate and consistent part detection is crucial to deliver smooth motion
sequences without any jittering effect in 3D pose reconstruction.

Figure 11: Comparison with state-of-the-art results on motion re-targeting
model.

#### 2D Detection

Method | P1 | P2
---|---|---
ICCV (Martinez et al., 2017)(SH PT) | 67.5 | 52.5
ours (L6 x V4 SH PT) | 57.3 | 45.5
ICCV (Martinez et al., 2017)(SH FT) | 62.9 | 47.7
ECCV (Hossain et al., 2018)(SH FT) | 58.3 | 44.2
CVPR (Zhao et al., 2019b)(SH FT) | 61.2 | 47.7
ours (L6 x V4 SH FT) | 52.0 | 40.7
CVPR (Pavllo et al., 2019)(CPN FT) | 46.8 | 36.5
ours (L6 x V4 CPN FT) | 44.8 | 35.6
ICCV (Martinez et al., 2017)(GT) | 45.5 | 37.1
ECCV (Hossain et al., 2018)(GT) | 41.6 | 31.7
ECCV (Lee et al., 2018)(GT) | 38.4 | -
CVPR (Zhao et al., 2019b)(GT) | 40.8 | 31.45
CVPR (Pavllo et al., 2019)(GT) | 37.8 | 28.2
ours (L6 x V4 GT) | 33.4 | 26.1
Table 9: Performance impacted by 2D detectors under Protocol 1 and Protocol 2.
(PT) \- pre-trained, (FT) \- fine-tuned, SH \- stacked hourglass.

We investigated the impact of 2D pose detection on our 3D pose estimation
performance by exploring several widely adopted 2D detectors. Firstly, we
utilized the pre-trained _Stacked Hourglass network_ (SH) (Newell et al.,
2016) on the MPII dataset to extract 2D keypoint locations within the ground-
truth bounding boxes. We also applied the results of fine-tuned SH model on
the Human3.6M dataset developed by (Martinez et al., 2017). Researchers also
investigated automated methods with detected bounding boxes for 2D human pose
detection, such as _Simple baselines for human pose estimation_ (Xiao et al.,
2018), _Deep high-resolution representation for human pose estimation_ (HRnet)
(Sun et al., 2019) or _Cascaded Pyramid Network_ (CPN) (Chen and Ramanan,
2017) together with _Mask R-CNN_ (He et al., 2017) and _ResNet-101-FPN_ (Lin
et al., 2017) as the backbone. We applied the pre-trained SH, fine-tuned SH,
and fine-tuned CPN models (Pavllo et al., 2019) as the 2D detectors for
performing a fair comparison, as shown in Table 9.

The big difference between the pre-trained and fine-tuned models are the 2D
human joints estimation accuracy and number of joints. Based on the results of
our experiment, our network can learn different joint label information. MPII
has 16 joints which missed the neck/nose joint in the Human3.6M dataset.
Although COCO dataset has the same joint number, the order of the labels of
joints is different from Human3.6M. To get a more accurate 3D joints position
result, we utilize a fine-tuned model to get the corresponding 2D joints on
Human3.6M. Furthermore, in the second part of Table 6, we show the results
with ground-truth (GT) 2D input. For both cases, our attention model
demonstrates a clear advantage by utilizing the temporal information.

Figure 12: An example of a 4-layer architecture for causal attention-based
temporal convolutional neural network.

#### Causal Attention Results

To facilitate real-time performance for potential interactive applications, we
also investigate a causal attention based network that estimates the target
pose by only processing the current frame and its previous frames. The
architecture of the causal attention model is shown in Fig. 12. The
architecture is similar to the one described in Fig. 2, but here we only
consider the left half of the input video sequence. The number of input frames
can also be determined by the number of layers of the model, but it shifts to
the $\frac{N-1}{2}$ previous frames, where $N$ is the corresponding number of
frames in the full-model illustrated in Fig. 2 . For example, for the
configuration of $L4\times V2\times N27$, $27$ causal frames are fed into the
network (included the target frame); while $L5\times V3\times N81$ requires
$81$ causal frames as the input. Similarly, to verify the performance, we
implemented three different prototypes according to the number of layers and
levels, as shown in Table 10. Horizontally, each row indicates a different
prototype of the causal model. Vertically, each column indicates a different
2D detector. We provide a side-by-side comparison with the results in the
recent CVPR paper on the same problem with various 2D detectors (Pavllo et
al., 2019). Even our causal model only considers casual input frames compared
to the TCN based semi-supervised approach in (Pavllo et al., 2019), the
results of our method (ATCN + MDC) demonstrate higher accuracy consistently.
In particular, more noticeable improvements are achieved as the number of
input frames increases. The result of real-time processing using causal model
is shown in Fig. 13.

(a)

(b)

(c)

(d)

Figure 13: 3D reconstruction results from different angles. Method | SH PT | SH FT | CPN FT | GT
---|---|---|---|---
(Pavllo et al., 2019) | - | - | 49.0 | -
Ours(n=27) | - | 56.8 | 49.4 | 39.7
Ours(n=81) | - | 55.7 | 47.5 | 37.1
Ours(n=243) | 59.2 | 54.9 | 46.6 | 35.5
Table 10: Bottom-table: Causal sequence processing performance in terms of the
different 2D detectors under $Protocol\\#1$. PT - pre-trained, FT - fine-
tuned, GT - ground-truth, SH - stacked hourglass, CPN - cascaded pyramid
network. (a)
Figure 14: Samples of synthesized outdoor environment on the Human3.6M dataset and their 3D pose estimation. Noise | ICCV(Martinez et al., 2017) | ECCV(Hossain et al., 2018) | CVPR(Pavllo et al., 2019) | Ours
---|---|---|---|---
GT/GT | 37.1 | 31.7 | 28.1 | 26.1
GT/GT + $\mathcal{N}(0,5)$ | 46.7 | 37.5 | 30.9 | 28.3
GT/GT + $\mathcal{N}(0,10)$ | 52.8 | 49.4 | 39.3 | 36.7
GT/GT + $\mathcal{N}(0,15)$ | 60.0 | 61.8 | 50.3 | 42.5
GT/GT + $\mathcal{N}(0,20)$ | 70.2 | 73.7 | 62.2 | 56.2
Table 11: $Protocol\\#2$ measurement on the estimation results from the
simulated scenes. Training and testing on ground truth 2d joint locations plus
different levels of additive gaussian noise.

###  5.5 Performance on Videos in-the-wild

To evaluate the performance on videos in-the-wild, we validated our approach
on both public datasets and online videos with the former emphasizing
quantitative validation while the later demonstrating qualitative performance.
While there exists limited datasets with accurate 3D pose in the wild, we
adopt some of the standard activities with outdoor scene simulation to
quantitatively evaluate the performance and compare with other approaches. In
contrast to static background and cameras capture setting, outdoor has more
dynamic and unrestricted environment with frequent occlusion and high
variation in background/foreground objects appearance. Fig. 14 shows several
outdoor simulations on the standard activities with snow, fog, and occlusion
effects (each column). The corresponding pose estimation results by different
approaches are shown in each of the following rows. Table 11 provides the
quantitative measurement on their output. In a similar manner, joint-wise
analysis is conducted on a selected joint from the Human3.6M scene with the
generated noises. One can see our approach consistently yields less MPJPE over
the frames as shown in Fig. 15. To quantitatively demonstrate the robustness
and efficacy, various videos in the wild are collected online and added with
extra noises, e.g. snow or fog effect. Fig. 17 shows satisfactory results are
achieved, given the additional noises. For example, in the foggy scene (row 5
and 6), the target person is almost occluded by the thick fog. Thanks to the
attention model that successfully extracts temporal information from neighbor
frames, the full 3D pose is correctly recovered.

(a) Protocol 1: joint error analysis after adding noise across frames in
Human3.6M _Walking S9_ left ankle.

(b) Protocol 1: joint error analysis after adding noise across frames in
Human3.6M _Smoking S9_ left elbow.

Figure 15: Joint-wise analysis and comparison on the outdoor simulated scenes.

To further demonstrate the temporal consistency, we gather online video
sequences from YouTube and predict the 3D poses directly from these videos in
the wild. Fig. 18 demonstrates the results of this experiment on various
activities. Even though the input videos are either of low resolution or with
fast motions, our approach is still able to estimate the 3D pose with
satisfactory output. For example, for the dancing scenes (rows 1-2 and rows
9-10) and the skating scene (rows 5-6), given the presence of fast body
movement and self-occlusion, the estimations are accurate enough to provide
the corresponding 3D positions for each frame. To further verify the
robustness, different sports activities with novel body poses (rows 3-4, rows
7-8, and rows 11-12) are processed. Our algorithm can faithfully capture and
reproduce these pose details without requiring any additional offline training
or manual preprocessing steps. In particular, for the challenging scene in
rows 3-4, the target person has relatively casual dress with partial leg
occlusion by the top costume The generated 3D pose from our attention model
are visually plausible and resemble the user’s body motion very well.

(a) Heavy occlusion.

(b) Instantaneous movement.

Figure 16: Unresolved cases: there were a few failed frames from the tested
wild videos, where severe occlusion and fast motion presented.

##  6 Conclusion and Discussion

In this paper, we present a novel and practical approach for 3D human pose
estimation and reconstruction in unconstrained videos. In order to enhance
temporal coherency, we integrate an attentional mechanism to the temporal
convolutional neural network to guide the network towards learning informative
representation. Moreover, we introduced a multi-scale dilation convolution,
which is capable of capturing several levels of temporal receptive fields,
achieving long-range dependencies among frames. Extensive experiments on
benchmark demonstrates that our approach improves upon the state-of-the-art
and offers an effective, alternative framework to address the 3D human pose
estimation problem. The implementation is straightforward and can adaptive
corporate with standard convolution neural networks. For the input data, any
off-the-shelf 2D pose estimation systems,e.g. Mocap libraries, can be easily
integrated in an ad-hoc fashion.

Though our results outperform the state-of-the-art on public datasets, there
are still some specific limitation remaining unresolved. Two examples are
shown in Fig. 16. For example, when the performer’s arms are crossing under
the fans, it causes heavy occlusion with missing joints detection, thereby
resulting in poor pose estimation, indicated in Fig.
16LABEL:sub@heavy_oclusion. In Fig. 16LABEL:sub@fast_movement, when the leg
has a very fast movement, our temporal system categorizes it as an outlier
position rather than using them to contribute the pose inference. Another
limitation is on the inference accuracy for some multi-person human scenarios
due to the limited training data on labeled multi-person 3D pose video
datasets. However, if using the top-down 2D pose detecting algorithm with pose
tracking, it would be possible to reconstruct multi-person 3D pose from a
video. The tracking error may affect the temporal attention performance. Our
future direction will explore a more generic framework that integrates the
proposed attention model and person re-identification solution to handle
instantaneous body part movements and heavy occlusions caused by multiple
people.

(a)

(b)

(c)

Figure 17: Qualitative results on gathered in the wild videos: original frame
sequence with added noises and the recovered 3D poses.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Figure 18: Qualitative results on gathered Youtube videos: original frame
sequence and the recovered 3D poses.

## References

  * Amin et al. (2013) Amin S, Andriluka M, Rohrbach M, Schiele B (2013) Multiview pictorial structures for 3d human pose estimation. BMVC
  * Andriluka et al. (2009) Andriluka M, Roth S, Schiele B (2009) Pictorial structures revisited: People detection and articulated pose estimation. Conference on Computer Vision and Pattern Recognition (CVPR) pp 1–8
  * Bahdanau et al. (2016) Bahdanau D, Cho K, Bengio Y (2016) Neural machine translation by jointly learning to align and translate. In: ICLR
  * Bai et al. (2018) Bai S, Kolter JZ, Koltun V (2018) An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:180301271
  * Bogo et al. (2016) Bogo F, Kanazawa A, Lassner C, Gehler P, Romero J, Black MJ (2016) Keep it smpl: Automatic estimation of 3d human pose and shape from a single image. European Conference on Computer Vision (ECCV) p 1–18
  * Bottou (2010) Bottou L (2010) Large-scale machine learning with stochastic gradient descent. In: Proceedings of COMPSTAT’2010, Springer, pp 177–186
  * Chen and Ramanan (2017) Chen CH, Ramanan D (2017) 3d human pose estimation = 2d pose estimation + matching. Conference on Computer Vision and Pattern Recognition (CVPR) pp 7035–7043
  * Chen et al. (2020) Chen T, Fang C, Shen X, Zhu Y, Chen Z, Luo J (2020) Anatomy-aware 3d human pose estimation in videos. arXiv preprint arXiv:200210322
  * Chen et al. (2016) Chen W, Wang H, Li Y, et al HS (2016) Synthesizing training images for boosting human 3d pose estimation. Fourth International Conference on 3D Vision (3DV) pp 479–488
  * Chen et al. (2019) Chen Y, Shen C, Chen H, Wei XS, Liu L, Yang J (2019) Adversarial learning of structure-aware fully convolutional networks for landmark localization. IEEE transactions on pattern analysis and machine intelligence
  * Cheng et al. (2019) Cheng Y, Yang B, Wang B, Yan W, Tan RT (2019) Occlusion-aware networks for 3d human pose estimation in video. In: Proceedings of the IEEE International Conference on Computer Vision, pp 723–732
  * Chorowski et al. (2015) Chorowski J, Bahdanau D, Serdyuk D, Cho K, Bengio Y (2015) Attention-based models for speech recognition. Advances in Neural Information Processing Systems 28 pp 577–585
  * Chu et al. (2017) Chu X, Yang W, Ouyang W, Ma C, Yuille AL, Wang X (2017) Multi-context attention for human pose estimation. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 1831–1840
  * Dabral et al. (2018) Dabral R, Mundhada A, Kusupati U, Afaque S, Sharma A, Jain A (2018) Learning 3d human pose from structure and motion. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 668–683
  * Dauphin et al. (2017) Dauphin YN, Fan A, Auli M, Grangier D (2017) Language modeling with gated convolutional networks. In: Proceedings of the 34th International Conference on Machine Learning-Volume 70, JMLR. org, pp 933–941
  * Fang et al. (2018) Fang HS, Xu Y, Wang W, Liu X, Zhu SC (2018) Learning pose grammar to encode human body configuration for 3d pose estimation. Thirty-Second AAAI Conference on Artificial Intelligence
  * Ferrari et al. (2009) Ferrari V, Marin-Jimenez M, Zisserman A (2009) Pose search: Retrieving people using their pose. IEEE Conference on Computer Vision and Pattern Recognition p 1–8
  * He et al. (2016) He K, Zhang X, Ren S, Sun J (2016) Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition pp 770–778
  * He et al. (2017) He K, Gkioxari G, Dollar P, Girshick R (2017) Mask r-cnn. International Conference on Computer Vision (ICCV) p 2980–2988
  * Hochreiter and Schmidhuber (1997) Hochreiter S, Schmidhuber J (1997) Long short-term memory. neural computation. Neural computation 9(8):1735 –1780
  * Hossain et al. (2018) Hossain M, Little JJ, XXX (2018) Exploiting temporal information for 3d human pose estimation. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 68–84
  * Ionescu et al. (2013) Ionescu C, Papava D, Olaru V, Sminchisescu C (2013) Human3. 6m: Large scale datasets and predictive methods for 3d human sensing in natural environments. IEEE transactions on pattern analysis and machine intelligence 36(7):1325–1339
  * Kalchbrenner et al. (2016) Kalchbrenner N, Espeholt L, Simonyan K, Oord Avd, Graves A, Kavukcuoglu K (2016) Neural machine translation in linear time. arXiv preprint arXiv:161010099
  * Lee et al. (2018) Lee K, Lee I, Lee S (2018) Propagating lstm: 3d pose estimation based on joint interdependency. Proceedings of the European Conference on Computer Vision (ECCV) pp 119–135
  * Lepetit et al. (2005) Lepetit V, Fua P, et al. (2005) Monocular model-based 3d tracking of rigid objects: A survey. Foundations and Trends® in Computer Graphics and Vision 1(1):1–89
  * Li et al. (2015) Li S, Zhang W, Chan AB (2015) Maximum-margin structured learning with deep networks for 3d human pose estimation. International Conference on Computer Vision (ICCV) p 2848–2856
  * Lin et al. (2017) Lin T, Dollar P, Girshick RB, He K, Hariharan B, Belongie SJ (2017) Feature pyramid networks for object detection. Conference on Computer Vision and Pattern Recognition (CVPR) p 936–944
  * Liu et al. (2020a) Liu J, Guang Y, Rojas J (2020a) Gast-net: Graph attention spatio-temporal convolutional networks for 3d human pose estimation in video. arXiv preprint arXiv:200314179
  * Liu et al. (2019) Liu L, Jiang H, He P, Chen W, Liu X, Gao J, Han J (2019) On the variance of the adaptive learning rate and beyond. arXiv preprint arXiv:190803265
  * Liu et al. (2020b) Liu R, Shen J, Wang H, Chen C, Cheung Sc, Asari V (2020b) Attention mechanism exploits temporal contexts: Real-time 3d human pose reconstruction. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp 5064–5073
  * Luvizon et al. (2018) Luvizon DC, Picard D, Tabia H (2018) 2d/3d pose estimation and action recognition using multitask deep learning. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 5137–5146
  * Mandery et al. (2015) Mandery C, Terlemez O, Do M, Vahrenkamp N, Asfour T (2015) The kit whole-body human motion database. International Conference on Advanced Robotics (ICAR) pp 329–336
  * Martinez et al. (2017) Martinez J, Hossain R, Romero J, Little JJ (2017) A simple yet effective baseline for 3d human pose estimation. International Conference on Computer Vision (ICCV) p 2659–2668
  * Neverova et al. (2014) Neverova N, Wolf C, Taylor GW, Nebout F (2014) Multi-scale deep learning for gesture detection and localization. European Conference on Computer Vision (ECCV) Workshops pp 474–490
  * Newell et al. (2016) Newell A, Yang K, Deng J (2016) Stacked hourglass networks for human pose estimation. European Conference on Computer Vision pp 483–499
  * Oord et al. (2016) Oord Avd, Dieleman S, Zen H, Simonyan K, Vinyals O, Graves A, Kalchbrenner N, Senior A, Kavukcuoglu K (2016) Wavenet: A generative model for raw audio. arXiv preprint arXiv:160903499
  * Palmero et al. (2016) Palmero C, Clapés A, Bahnsen C, Møgelmose A, Moeslund TB, Escalera S (2016) Multi-modal rgb–depth–thermal human body segmentation. International Journal of Computer Vision 118(2):217–239
  * Park et al. (2016) Park S, Hwang J, Kwak N (2016) 3d human pose estimation using convolutional neural networks with 2d pose information. European Conference on Computer Vision (ECCV) Workshops p 156–169
  * Pavlakos et al. (2017) Pavlakos G, Zhou X, Derpanis KG, Daniilidis K (2017) Coarse-to-fine volumetric prediction for single-image 3d human pose. Conference on Computer Vision and Pattern Recognition (CVPR) p 1263–1272
  * Pavllo et al. (2019) Pavllo D, Feichtenhofer C, Grangier D, Auli M (2019) 3d human pose estimation in video with temporal convolutions and semi-supervised training. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 7753–7762
  * Rhodin et al. (2018) Rhodin H, Spörri J, Katircioglu I, Constantin V, Meyer F, Müller E, Salzmann M, Fua P (2018) Learning monocular 3d human pose estimation from multi-view images. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 8437–8446
  * Ruck et al. (1990) Ruck D, Rogers S, Kabrisky M (1990) Feature selection using a multilayer perceptron. Journal of Neural Network Computing 2(2):40–48
  * Sarafianos et al. (2016) Sarafianos N, Boteanu B, Ionescu B, Kakadiaris IA (2016) 3d human pose estimation: A review of the literature and analysis of covariates. CVIU p 1–20
  * Sigal et al. (2010) Sigal L, Balan AO, Black MJ (2010) Humaneva: Synchronized video and motion capture dataset and baseline algorithm for evaluation of articulated human motion. International Journal of Computer Vision 87(12):4–27
  * Sun et al. (2019) Sun K, Xiao B, Liu D, Wang J (2019) Deep high-resolution representation learning for human pose estimation. arXiv preprint arXiv:190209212
  * Tekin et al. (2016) Tekin B, Rozantsev A, Lepetit V, Fua P (2016) Direct prediction of 3d body poses from motion compensated sequences. Conference on Computer Vision and Pattern Recognition (CVPR) p 991–1000
  * Toshev and Szegedy (2014) Toshev A, Szegedy C (2014) Deeppose: Human pose estimation via deep neural networks. Conference on Computer Vision and Pattern Recognition (CVPR) pp 1653–1660
  * Varol et al. (2017) Varol G, Romero J, Martin X, Mahmood N, Black MJ, Laptev I, Schmid C (2017) Learning from synthetic humans. Conference on Computer Vision and Pattern Recognition (CVPR) p 109–117
  * Xiao et al. (2018) Xiao B, Wu H, Wei Y (2018) Simple baselines for human pose estimation and tracking. In: Proceedings of the European Conference on Computer Vision (ECCV), pp 466–481
  * Xu et al. (2015) Xu K, Ba J, Kiros R, Cho K, Courville C, Salakhutdinov R, Zemel R, Bengio Y (2015) Show, attend and tell: Neural image caption generation with visual attention. International Conference on Machine Learning
  * Yang et al. (2018) Yang W, Ouyang W, Wang X, Ren J, Li H, Wang X (2018) 3d human pose estimation in the wild by adversarial learning. Conference on Computer Vision and Pattern Recognition (CVPR) p 5255–5264
  * Yang and Ramanan (2011) Yang Y, Ramanan D (2011) Articulated pose estimation with flexible mixtures-of-parts. Conference on Computer Vision and Pattern Recognition (CVPR) p 1385–1392
  * Yao et al. (2013) Yao X, Durme B, Callison-Burch C, Clark P (2013) Semi-markov phrase-based monolingual alignment. Conference on Empirical Methods in Natural Language Processing pp 590–600
  * Yin et al. (2016) Yin W, Schütze H, Xiang B, Zhou B (2016) Abcnn: Attention-based convolutional neural network for modeling sentence pairs. Transactions of the Association for Computational Linguistics 4:259–272
  * Yoo and Han (2009) Yoo J, Han T (2009) Fast normalized cross-correlation. Circuits, Systems and Signal Processing 28(819):1–13
  * Zhang et al. (2019) Zhang MR, Lucas J, Hinton G, Ba J (2019) Lookahead optimizer: k steps forward, 1 step back. arXiv preprint arXiv:190708610
  * Zhao et al. (2019a) Zhao L, Peng X, Tian Y, Kapadia M, Metaxas DN (2019a) Semantic graph convolutional networks for 3d human pose regression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3425–3435
  * Zhao et al. (2019b) Zhao L, Peng X, Tian Y, Kapadia M, Metaxas DN (2019b) Semantic graph convolutional networks for 3d human pose regression. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp 3425–3435
  * Zhou et al. (2016a) Zhou X, Sun X, Zhang W, Liang S, Wei Y (2016a) Deep kinematic pose regression. European Conference on Computer Vision (ECCV) Workshops p 156–169
  * Zhou et al. (2016b) Zhou X, Zhu M, Leonardos S, Derpanis KG, Daniilidis K (2016b) Sparseness meets deepness: 3d human pose estimation from monocular video. Conference on Computer Vision and Pattern Recognition (CVPR) pp 4966–4975


Effective and efficient scheduling of vaccine distribution can significantly impact vaccine uptake, which is critical to controlling the spread of infectious diseases. Ineffective scheduling can lead to waste, delays, and low vaccine coverage, potentially weakening the efforts to protect the public. Organizations such as UNICEF (United Nations Children’s Fund), PAHO (Pan American Health Organization), and GAVI (Gavi, the Vaccine Alliance) coordinate vaccine tenders to ensure that enough supply is available on the international market at the lowest possible prices. Scheduling vaccine tenders over a planning horizon in a way that is equitable, efficient, and accessible is a complex problem that involves trade-offs between multiple objectives while ensuring that vaccine availability, demand, and logistical constraints are met. The current method for scheduling tenders is generally reactive and over short planning horizons. Vaccine tenders are scheduled when supply is insufficient to cover demand. We propose an optimization model to dynamically and proactively generate vaccine tender schedules over long planning horizons. This model helps us address the following research questions: What should the optimal sequencing and scheduling of vaccine tenders be to enhance affordability and profit over long time horizons? What is the optimal tender procurement schedule for single/multiple antigen scenarios? We use several real-life data sources to validate the model and address our research questions. Results from our analysis show when to schedule vaccine tenders, what volumes manufacturers should commit to, and the optimal tender lengths to satisfy demand. We show that vaccine tenders tend towards maximum lengths, generally converge over long time horizons, and are robust to changes in varying conditions.

§ KEYWORDS

Integer programming, vaccine tender, hypothetically coordinated vaccine market

§ INTRODUCTION

Pediatric vaccines prevent an estimated "3.5-5 million deaths from diseases like diphtheria and tetanus" annually[1]. Immunization campaigns prevent diseases through the administration of vaccines that induce the production of antibodies against specific disease-causing antigens.

To meet market demand, vaccine tenders for different antigens are scheduled to ensure adequate supply levels. Low- and low-middle-income countries (LIC, and LMIC, respectively) primarily procure vaccines in the UNICEF market, and depending on their gross national income per capita and the strength of their healthcare systems, can receive financial support from GAVI to procure vaccines. UNICEF and PAHO act as coordinating entities, working with countries to aggregate vaccine demand information. UNICEF coordinates vaccine tenders with vaccine producers to meet the demand of LIC and LMICs. UNICEF plans tenders 2 to 3 years in advance. However, its scheduling is reactive, primarily considering short-term demand needs, having limited visibility of other factors, such as country-specific immunization plans, and manufacturing capacity producers share among multiple products in their portfolio.

In a tender, manufacturers commit to supply specific vaccine volumes to the UNICEF market at an agreed price for a given number of years (i.e., tender length). We propose an analytical approach to determine when to schedule vaccine tenders, what volumes manufacturers should commit to, and the optimal tender lengths to satisfy the long-term needs of LIC and LMIC in a hypothetically coordinated vaccine market (HCVM). An HCVM involves vaccine producers, LIC and LMIC grouped in market segments, and altruistic coordinating entities like UNICEF or PAHO. These entities aim to meet the vaccine demands at affordable prices while ensuring producers' returns. Entities like UNICEF and PAHO manage procurement, logistics, and financial support, ensuring a steady vaccine supply that meets the countries' immunization needs within their economic constraints. To this end, we develop a mixed-integer program that identifies vaccine tenders' optimal timing and duration over long planning horizons. Our findings show that for varying conditions, it is possible to schedule vaccine tenders longer than the current practice of 2-3 years and provide insight into possible vaccine manufacturer commitment.

Various mathematical models exist for scheduling pediatric vaccines [Yu et al., 2022]. However, these efforts do not focus on the long-term scheduling of vaccine procurement tenders. Additional studies thoroughly review vaccine procurement and highlight the associated processes and policy implications [Gianfredi et al., 2021]. Long-term procurement scheduling is most commonly found in energy procurement models [Zhang et al., 2018].

Proano et al. [Proano et al., 2012] introduce a deterministic model for the HCVM to explore if it is possible to find vaccine prices that can be affordable and profitable for a coordinated one-year multi-vaccine tender. In a subsequent study, Proano and Mosquera [Proano and Mosquera, 2021] establish that implementing tier-pricing with more segments within the GAVI and PAHO markets could extend the opportunities to have affordable vaccines. Alves-Maciel and Proano [Alves-Maciel and Proano, 2021, Alves-Maciel and Proano, 2022] further extend HCVM to study the effect of market coordination under multiple non-cooperative coordinating entities and when tenders rely on lump-sum payments for vaccine product baskets.

The existing research on the HCVM has focused on the impact of market coordination on vaccine affordability and profit for a single multi-product short-term tender. In contrast, this study focuses on long-term tender scheduling for several antigens. The objective is to enhance availability and profitability by enabling coordinating entities to implement long-term tender schedules. The model considers that the available capacity of manufacturers, and potential capacity expansions, are shared among different antigens.

§ MODEL DESCRIPTION

$\mathcal{V}$: set of vaccines (from the WHO pre-qualified vaccine list [9]),      $\mathcal{A}$: set of antigens,

$\mathcal{V}_a$: set of vaccines containing antigen $a$,     $\mathcal{P}$: set of all producers,

$\mathcal{A}_p$: subset of antigens contained in producer $p$'s portfolio,     $\mathcal{P}_v$: set of producers of vaccine $v$,

$\mathcal{T}$: set of time periods in the planning horizon,     $\mathcal{A}_v$: subset of antigens in vaccine $v$,

2l$\mathcal{T}_t'$: set of periods when a tender ends (since we allow tenders of at most 5 years, $\mathcal{T}_t' = \{t, t+1, ..., min\{t+4,|\mathcal{T}|\}\}$).

$l_{vp}$: the annualized return on investment required by producer $p$ for vaccine $v$,     ${d}_{at}$: demand for antigen $a$ for period $t$,

$f_{vpt}$: the production set-up cost of producer $p$ for vaccine $v$ in period $t$,     ${s}_{pt}$:  production capacity of producer $p$ for period $t$,

$h_{v}$: the annual holding cost for vaccine $v$ as a proportion of its price,     $g_t$: the set-up cost for initiating a tender in period $t$,

$r_{vpt}$: reservation price of vaccine $v$ produced by $p$ in period $t$,     $\Gamma_p$: capacity extension cost for producer $p$,

$\bar{r}_{vt}$: average price of vaccine $v$ across all producers in period $t$,     $\kappa$: allowable capacity extension rate,

$S_{a0}$:initial number of children unvaccinated with antigen $a$,     $I_{v0}$: initial inventory for vaccine $v$,

$\beta$ : penalty per unit of shortage of the committed amount,     $\delta$: the discount factor

i.e., the unvaccinated children penalty.

Decision Variables:

$F_{at\tau}$ is a binary variable indicating whether a tender covers the demand for antigen $a$ for periods $t$ through $\tau$,

$Q_{vpt\tau}$ is the procurement commitment for vaccine $v$ by producer $p$ for periods $t$ through $\tau$,

$X_{vpt}$ represents doses of vaccine $v$ delivered by producer $p$ in period $t$,

$Y_{pt}$ is a binary variable which specifies whether or not manufacturer $p$ produces in period $t$,

$W_{pt\tau}$ indicates if producer $p$ has made commitments for periods $t$ through $\tau$,

$L_{pt}$ is a binary variable indicating if a capacity extension decision has been made,

$I_{vt}$ is the stock level for vaccine $v$ at the beginning of period $t$,

$V_{vt}$ is the number of children vaccinated with vaccine $v$ in period $t$,

$S_{at}$ is the number of children unvaccinated for antigen $a$ in period $t$.

Using the notation provided above, the problem is formulated as follows:

\beginmini!
∑_t∈𝒯 δ^t (∑_a ∈𝒜  ∑_τ∈𝒯_t' g_t F_atτ + ∑_v ∈𝒱 ∑_p ∈𝒫_v r_vpt X_vpt + ∑_a ∈𝒜   βS_at + ∑_v ∈𝒱 h_vr̅_vtI_vt + ∑_p ∈𝒫 Γ_p L_pt)
(τ-t+1) F_atτ≤∑_p∈𝒫_a∑_l=t^τY_pl,∀a∈𝒜; t ∈𝒯; τ∈𝒯_t'
F_atτ + F_atτ'≤1,∀a∈𝒜; t ∈𝒯; τ, τ' ∈𝒯'; τ≠τ'
∑_t=max{1,l-4}^l ∑_τ∈𝒯_t' F_atτ≥1,∀a∈𝒜; l ∈T
∑_a∈𝒜_p F_atτ≥W_ptτ,∀p ∈𝒫; t ∈𝒯; τ∈𝒯_t'
∑_a∈𝒜_p F_atτ≤|𝒜_p| W_ptτ,∀p ∈𝒫; t ∈𝒯; τ∈𝒯_t'
∑_v ∈𝒱_p Q_vptτ≤W_ptτ (∑_l=t^τ(s_pl+∑_k=1^ls_pkκL_pk)),∀p ∈𝒫; t ∈𝒯; τ∈𝒯_t'
Q_vptτ≥W_ptτ ∑_l=t^τ X_vpl,∀v∈𝒱; p ∈𝒫_v; t ∈𝒯; τ∈𝒯_t'
∑_v∈𝒱_pX_vpt≤Y_pt (s_pt+∑_l=1^ts_plκL_pl),∀p ∈𝒫; t ∈𝒯
I_vt-1 +  ∑_p∈𝒫_v X_vpt= V_vt + I_vt,∀v ∈𝒱; t ∈𝒯
d_at - ∑_v ∈𝒱_aV_vt+ S_at-1≤S_at,∀a∈𝒜;  t ∈𝒯
∑_v∈𝒱_p r_vpt X_vpt≥∑_v∈𝒱_p (1 + l_vp)f_vptY_pt,∀p∈𝒫; t ∈𝒯
Q_vptτ, X_vpt, I_vt, S_at, V_vt≥0,∀a∈𝒜; v∈𝒱; p ∈𝒫_v; t ∈𝒯; τ∈𝒯_t'
F_atτ, Y_pt, W_ptτ, L_pt∈{0,1},∀a∈𝒜; p ∈𝒫_v; t ∈𝒯; τ∈𝒯_t'.
\endmini!

The objective function (<ref>) minimizes the total discounted cost, including the setup cost of organizing tenders, vaccine purchasing costs, inventory holding costs, capacity extension costs, and costs of having unvaccinated children. Note that while no true dollar amount can be placed on the long-term costs of unvaccinated children, the penalty is a numerical attempt to prevent unvaccinated children. Constraint (<ref>) ensures that if a tender takes place between years $t$ through $\tau$, at least one producer should supply vaccines during this period. Constraint (<ref>) ensures that two tenders for the same antigen cannot start at the same period. Constraint (<ref>) guarantees that at least one tender covers the demand for an antigen in each period. Constraints (<ref>) and (<ref>) ensure that a tender is granted to a manufacturer between periods $t$ and $\tau$ if the manufacturer has made commitments for a vaccine for the corresponding antigen ($A_p$). Constraint (<ref>) assures that a manufacturer's total commitment to the vaccines cannot surpass its production capacity. Note that production capacity in a period might increase based on the capacity extension decision of the manufacturer. Constraint (<ref>) ensures that the total number of vaccines delivered over given tender periods cannot exceed the vaccine procurement commitment. Constraint (<ref>) ensures that for any given period $t$, the total number of vaccines delivered by a producer does not exceed that producer's production capacity. Constraint (<ref>) ensures inventory balance between consecutive years. Constraint (<ref>) keeps track of the number of unvaccinated children. Constraint (<ref>) guarantees that each producer makes a profit that fulfills their desired ROI. Constraints (<ref>) and (<ref>) define the non-negativity and binary variables.

Constraint (<ref>) includes a non-convex bi-linear term, where a binary variable ($W_{pt\tau}$) and summation of continuous variables ($\sum_{l=t}^{\tau} X_{vpl}$) are multiplied. We linearize (<ref>) by using McCormick linearization. To create the McCormick envelope, we replace the bi-linear term with a new variable ($\tilde{K}_{vpt\tau}$) and add bounds for this new variable. In the reformulation, constraint (<ref>) is replaced with the following constraint sets, where the lower and upper bounds for the variables $W_{pt\tau}$ and $\tilde{X}_{vpt\tau}$ are, respectively, defined as $w_{pt\tau}^l$, $w_{pt\tau}^u$, $\tilde{x}_{vpt\tau}^l$, $\tilde{x}_{vpt\tau}^u$. This is a tight relaxation since $W_{pt\tau}$ is a binary variable.

\begin{equation}
\tilde{X}_{vpt\tau} =  \sum_{l=t}^{\tau} X_{vpl}, \hspace{0.3in} \forall v\in \mathcal{V}, p \in \mathcal{P}_v, t \in \mathcal{T}, \tau \in \mathcal{T}_t' \label{McCormick_1}
\end{equation}

\begin{equation}
Q_{vpt\tau} \geq  K_{vpt\tau}, \hspace{0.3in} \forall v\in \mathcal{V}, p \in \mathcal{P}_v, t \in \mathcal{T}, \tau \in \mathcal{T}_t' \label{McCormick_2}
\end{equation}

\begin{equation}
K_{vpt\tau} \geq \tilde{x}_{vpt\tau}^u W_{pt\tau} + \tilde{X}_{vpt\tau} w_{pt\tau}^u - \tilde{x}_{vpt\tau}^u w_{pt\tau}^u , \hspace{0.3in} \forall v\in \mathcal{V}, p \in \mathcal{P}_v, t \in \mathcal{T}, \tau \in \mathcal{T}_t'
\label{McCormick_3}
\end{equation}

\begin{equation}
K_{vpt\tau} \geq \tilde{x}_{vpt\tau}^l W_{pt\tau} + \tilde{X}_{vpt\tau} w_{pt\tau}^l - \tilde{x}_{vpt\tau}^l w_{pt\tau}^l , \hspace{0.3in} \forall v\in \mathcal{V}, p \in \mathcal{P}_v, t \in \mathcal{T}, \tau \in \mathcal{T}_t' \label{McCormick_4}
\end{equation}

\begin{equation}
K_{vpt\tau} \leq \tilde{x}_{vpt\tau}^u W_{pt\tau} + \tilde{X}_{vpt\tau} w_{pt\tau}^l - \tilde{x}_{vpt\tau}^u w_{pt\tau}^l , \hspace{0.3in} \forall v\in \mathcal{V}, p \in \mathcal{P}_v, t \in \mathcal{T}, \tau \in \mathcal{T}_t' \label{McCormick_5}
\end{equation}

\begin{equation}
K_{vpt\tau} \leq \tilde{x}_{vpt\tau}^l W_{pt\tau} + \tilde{X}_{vpt\tau} w_{pt\tau}^u - \tilde{x}_{vpt\tau}^l w_{pt\tau}^u , \hspace{0.3in} \forall v\in \mathcal{V}, p \in \mathcal{P}_v, t \in \mathcal{T}, \tau \in \mathcal{T}_t' \label{McCormick_6}
\end{equation}

§ DATA COLLECTION AND ANALYSIS

The model incorporates synthetically generated data and various datasets from real cases. We focus our analysis on the antigens in Table <ref>. We use the Linksbridge Vaccine Almanac datasets, which provide information on vaccine demand in LIC and LMIC in the next 10 years [10]. We use the simple exponential smoothing approach to forecast the demand for vaccination beyond 10 years. Production capacities, which often are confidential information, are synthetically estimated using a combination of UNICEF vaccine coverage estimates, the average portfolio of producers, and UNICEF vaccine demand [11]. We first aggregate vaccine demand for LIC and LMIC. We then scale the demand based on estimated vaccine coverage. We assume that the production capacity of each manufacturer is proportional to the market share of WHO-prequalified vaccines.

Vaccine Name     Antigens

MMR     Measles, Mumps, Rubella

Penta, Hexa     Diphtheria, Tetanus, Pertussis, Haemophilus influenzae type b, Polio, Hepatitis B

Rotavirus     Rotavirus

HPV     Human papillomavirus

PCV     Pneumococcal conjugate

Vaccines and corresponding antigens

We used historical data related to the cost of vaccines, which UNICEF publishes [11]. Since the cost of a vaccine dose differs by manufacturer, we use the corresponding average cost per dose in the model.
The reservation price of a vaccine represents the price at which UNICEF purchases vaccines from manufacturers in a specific market. We generate tender schedules for planning horizons ($\mathcal{T}$) ranging from 10 to 30 years. The analysis considers that monovalent and combination vaccines are available to meet vaccination needs. We evaluate the impact of problem parameters on the tender schedule via our numerical analysis.

§ DISCUSSION OF RESULTS

Our numerical analysis determines the optimal sequencing of vaccine tenders across different planning horizons.
Since we minimize the total discounted cost, the model tends to create long tenders. Such solutions indirectly minimize the efforts that go into organizing tenders and, thus,  the corresponding costs. As seen in Figure <ref>, most of the scheduled tenders have a 5-year duration. These long-term schedules show general stability, and synchronization of common antigen sequences. Figure <ref> establishes a tender schedule over a 30-year period, assuming that demand is met via monovalent vaccines only. Figure <ref> shows the schedule resulting from the use of monovalent and combination vaccines. In this case, we notice that tenders still occur for all antigens. However, demand is met through the combination vaccines, where MMR and Penta account for 97% of vaccines produced, for their corresponding antigen sets. Similarly, we notice that antigens bundled in common combination vaccines, like penta and MMR, generally stay in sequence over time. As the various color bands in Figures <ref> denote, antigens do not generally deviate from such bundles.

In examining model robustness, we utilize cosine similarity, a measure to estimate the similarity between two non-zero vectors, to compare schedules. The tender schedules are transformed into vectors representing ones for a scheduled tender, for all tender durations and antigens, zeros otherwise. In our experiment, we examine varying supply and demand, and analyze 12 tender schedules. We highlight the similarity between tender schedules, with varying conditions as previously described, and demonstrate the robustness of our model. The schedules in Figures <ref> and <ref> attain a cosine similarity of 0.45, showing moderate similarity between schedules with varying antigens (single versus combination vaccine). When combination vaccines as antigens are considered in the 12 experiments, we observe cosine similarities between 0.68 and 0.79. Figure <ref> represents one such experiment.

Tender Schedule

Tender Schedule: Vaccines as Antigens

Tender schedules for a 30-year planning horizon

In the objective function of our model, we include a term that penalizes the overall number of unvaccinated children. We consider the total number of unvaccinated children as the social impact of the proposed tender schedule. The size of the penalty term $\beta$ in the objective determines the number of unvaccinated children. When this cost is (perceived to be) higher than the cost of a vaccine,  the number of unvaccinated children is lowest. Figure <ref> shows that the number of unvaccinated children decreases as the value of $\beta$ increases from $0.4 to $1.5. Beyond this point, increasing the value of the penalty does not account for a significant change in the number of children who are not vaccinated.

Figure <ref> presents the total number of unvaccinated children over time for $\beta = 6.$ On-going tenders and commitments, and initial inventory level drive the number of unvaccinated children during the first five years. When the value of $\beta$ is higher than the expected cost of a vaccine (as is the case with $\beta = 6$), then the number of unvaccinated children is lowest beyond the initial five years. Unvaccinated children for Rotavirus and PCV, shown in years 4 and 5 in Figure <ref>, represent the initial conditions for this experiment. However, given model conditions, we can assess a high enough penalty for unvaccinated children to meet antigen demand by manipulating the parameter $\beta$. From this analysis, we can assess that, even with changing conditions in a hypothetically coordinated UNICEF market, we can adjust the penalty for unvaccinated children to account for these changes.

Unvaccinated children penalty ($\beta$)

Unvaccinated children when $\beta$=6

Analysis of unvaccinated children

§ CONCLUSION

In this study, we investigate the potential benefits of long-term planning for the hypothetically coordinated vaccine market (HCVM) by employing dynamic scheduling of vaccine tenders. Our approach relies on a deterministic mixed integer model to reduce the overall discounted cost of organizing tenders, manufacturing vaccines, and the cost of unvaccinated children. The proposed model facilitates the generation of vaccine tender schedules for various durations. We analyze these schedules' general stability, synchronization, and similarity and observe that the schedules generally align by antigen groups even when combination vaccines are added to the problem. The model also determines the volumes manufacturers should commit to and optimal tender lengths to satisfy the hypothetically coordinated UNICEF market demand. Future research endeavors will extend this model into a multi-stage stochastic optimization framework to account for uncertainties in demand. This extension will provide a more robust and flexible approach to vaccine tender scheduling. We anticipate that such enhancements will contribute to a more comprehensive understanding of the HCVM dynamics.

§ ACKNOWLEDGEMENTS

Funding: This work was partially supported by The Bill & Melinda Gates Foundation, Seattle, WA [grant number INV-044813].

[1]
World Health Organization.
World health organization vaccine and immunisation page, a.
URL <https://www.who.int/health-topics/vaccines-and-immunization#tab=tab_1>.

[Yu et al., 2022]
Zhuoting Yu, Pinar Keskinocak, Joel Sokol, and Yao Hsuan Chen.
Flexible analytic model to inform multi-stakeholder pediatric vaccine scheduling decisions.
Vaccine, 40:0 7631–7639, 12 2022.

[Gianfredi et al., 2021]
Vincenza Gianfredi, Antonietta Filia, Maria Cristina Rota, Roberto Croci, Lorenzo Bellini, Anna Odone, and Carlo Signorelli.
Vaccine procurement: A conceptual framework based on literature review, 12 2021.

[Zhang et al., 2018]
Qi Zhang, Andreas M. Bremen, Ignacio E. Grossmann, and Jose M. Pinto.
Long-term electricity procurement for large industrial consumers under uncertainty.
Industrial and Engineering Chemistry Research, 57:0 3333–3347, 3 2018.

[Proano et al., 2012]
Ruben A. Proano, Sheldon H. Jacobson, and Wenbo Zhang.
Making combination vaccines more accessible to low-income countries: The antigen bundle pricing problem.
Omega, 40, 2012.

[Proano and Mosquera, 2021]
Ruben A Proano and Galo E Mosquera.
Technical report: Inducing affordability by challenging the single-tier pricing for unicef and the paho's revolving fund. a simulation-based study.
Technical report, [Unpublished], 2021.

[Alves-Maciel and Proano, 2021]
Bruno Alves-Maciel and Ruben A. Proano.
Exploring market segment assignment strategies to monopsonistic entities in a hypothetically coordinated vaccine market.
Proceedings - Winter Simulation Conference, 2021.
ISSN 08917736.
URL <https://ieeexplore.ieee.org/document/9715353>.

[Alves-Maciel and Proano, 2022]
Bruno Alves-Maciel and Ruben A Proano.
Buying vaccines through a single payment as a proxy for procuring vaccines via formularies.
[Unpublished], 2022.
URL <https://doi.org/10.21203/rs.3.rs-1930820/v1>.

[9]
World Health Organization.
World health organization prequalified vaccines, b.
URL <https://extranet.who.int/pqweb/vaccines/prequalified-vaccines>.

[10]
Linksbridge: Vaccine almanac.
URL <https://data.linksbridge.com/>.

[11]
Unicef vaccine information.
URL <https://www.unicef.org/supply>.