# Advancing Pose-Guided Image Synthesis with Progressive Conditional Diffusion
Models

Fei Shen, Hu Ye∗, Jun Zhang , Cong Wang, Xiao Han, Wei Yang
Tencent AI Lab
{ffeishen, huye, junejzhang, xvencewang, haroldhan<EMAIL_ADDRESS>
Equal contribution.Corresponding author

###### Abstract

Recent work has showcased the significant potential of diffusion models in
pose-guided person image synthesis. However, owing to the inconsistency in
pose between the source and target images, synthesizing an image with a
distinct pose, relying exclusively on the source image and target pose
information, remains a formidable challenge. This paper presents Progressive
Conditional Diffusion Models (PCDMs) that incrementally bridge the gap between
person images under the target and source poses through three stages.
Specifically, in the first stage, we design a simple prior conditional
diffusion model that predicts the global features of the target image by
mining the global alignment relationship between pose coordinates and image
appearance. Then, the second stage establishes a dense correspondence between
the source and target images using the global features from the previous
stage, and an inpainting conditional diffusion model is proposed to further
align and enhance the contextual features, generating a coarse-grained person
image. In the third stage, we propose a refining conditional diffusion model
to utilize the coarsely generated image from the previous stage as a
condition, achieving texture restoration and enhancing fine-detail
consistency. The three-stage PCDMs work progressively to generate the final
high-quality and high-fidelity synthesized image. Both qualitative and
quantitative results demonstrate the consistency and photorealism of our
proposed PCDMs under challenging scenarios.The code and model will be
available at https://github.com/muzishen/PCDMs.

##  1 Introduction

Given an image of a specific person under a particular pose, pose-guided image
synthesis (Zhang et al., 2022; Ren et al., 2022; Bhunia et al., 2023) aims to
generate images of the person with the same appearance and meanwhile under the
given target pose, which more importantly, are expected to be as
photorealistic as possible. It holds broad and robust application potential in
e-commerce and content generation. Meanwhile, the generated images can be used
to improve the performance of downstream tasks, such as person re-
identification (Ye et al., 2021). However, since pose disparities between the
source and target images, generating an image with a different pose solely
based on the source image and target pose information remains a significant
challenge.

Previous work usually focuses on the generative adversarial network (GAN)
(Creswell et al., 2018), variational autoencoder (VAE) (Kingma et al., 2019),
and flow-based model (Li et al., 2019). GAN-base methods (Zhu et al., 2019;
Tang et al., 2020) insert multiple repeating modules to mine the sparse
correspondence between source and target pose image features. The outputs
produced by these approaches often exhibit distorted textures, unrealistic
body shapes, and localized blurriness, particularly when generating images of
occluded body parts. In addition, owing to the nature of the adversarial min-
max objective, GAN-based methods are susceptible to unstable training
dynamics, limiting the diversity of the generated samples. Although VAE-based
approaches (Siarohin et al., 2018; Esser et al., 2018) are relatively stable,
they suffer from blurring of details and misalignment of target pose due to
their reliance on surrogate loss for optimization. Flow-based methods (Li et
al., 2019; Ren et al., 2021) have emerged to deal with this problem, which
guide the source image features to distort to a reasonable target pose by
predicting the correspondence between the source and target pose. However,
when the source and target poses undergo large deformations or occlusions,
this can easily lead to apparent artifacts in the generated images. Likewise,
some methods (Lv et al., 2021; Zhang et al., 2021) utilize human parsing maps
to learn the correspondence between image semantics and poses to ensure that
the generated images are consistent with the target pose. Although these
methods can generate images that meet pose consistency requirements, they
still struggle to maintain consistent style and capture realistic texture
details.

Figure 1: (a) Existing methods typically utilize unaligned image-to-image
generation at the conditional level. (b) Our approach progressively predicts
the global features, dense correspondences, and texture restoration of target
image, enabling image synthesis.

Recently, diffusion models (Bhunia et al., 2023; Zhang & Zhou, 2023) have made
significant strides in the field of person image synthesis. They utilize the
source image and target pose as conditions and generate the target image
through a multi-step denoising process instead of completing it in a single
step. So, these approaches help better retain the input information. However,
as shown in Figure 1 (a), due to the pose inconsistency between the source and
target images, this essentially constitutes an unaligned image-to-image
generation task at the conditional level. Moreover, the lack of dense
correspondence between the source and target images regarding image, pose, and
appearance often results in less realistic results.

This paper presents Progressive Conditional Diffusion Models (PCDMs) to tackle
the aforementioned issues through three stages, as shown in Figure 1 (b).
Initially, we propose a prior conditional diffusion model to predict global
features given a target pose. This prediction is significantly simpler than
directly generating target images, as we allow the prior model to concentrate
solely on one task and thereby do not worry about realistically detailed
texture generation. Given a source image and pose coordinates as conditions,
the prior conditional diffusion model employs a transformer network to predict
global features under the target pose. In the second stage, we use the global
features from the previous stage to establish the dense correspondence between
the source and target images, and propose an inpainting conditional diffusion
model to further align and enhance contextual features, generating a coarse-
grain synthetic image. Finally, we develop a refining conditional diffusion
model. This model utilizes the coarse-grain image generated in the previous
stage and applies post-image-to-image techniques to restore texture and
enhance detail consistency. The three stages of the PCDMs operate
progressively to generate visually more appealing outcomes, particularly when
handling intricate poses.

We summarize the contributions of this paper as follows: (a) we devise a
simple prior conditional diffusion model that explicitly generates and
complements the embedding of the target image by mining the global alignment
relationship between the source image appearance and the target pose
coordinates. (b) we propose a novel inpainting conditional diffusion model to
explore the dense correspondence between source and target images. (c) we
introduce a new refining conditional diffusion model by further using post hoc
image-to-image techniques to enhance the quality and fidelity of synthesized
images. (d) We conduct comprehensive experiments on two public datasets to
showcase the competitive performance of our method. Additionally, we implement
a user study and a downstream task to evaluate the qualitative attributes of
the images generated by our method.

##  2 Related Work

Person Image Synthesis. The task of person image synthesis has achieved great
development during these years, especially with the unprecedented success of
deep learning. The earlier methods (Ma et al., 2017; Men et al., 2020) treat
the synthesis task as conditional image generation, using conditional
generative adversarial networks (CGANs) (Mirza & Osindero, 2014) to generate
the target image with the source appearance image and target pose as
conditions. However, due to the inconsistency between the source and target
poses, the effectiveness of directly connecting the source image with the
target pose is limited. To overcome this challenge, VUnet (Esser et al., 2018)
adopts a joint application of VAE and U-Net to decouple the appearance and
pose of the character image. Furthermore, Def-GAN (Siarohin et al., 2018)
proposes a deformable GAN that decomposes the overall deformation through a
set of local affine transformations to address the misalignment issues caused
by different poses. On the other hand, some works (Liu et al., 2019; Ren et
al., 2020) utilize flow-based deformation to transform source information,
improving pose alignment. For example, GFLA (Ren et al., 2020) obtains global
flow fields and occlusion masks to warp local patches of the source image to
match the desired pose. Similarly, ClothFlow (Han et al., 2019) is a model
designed based on this concept for segmentation-guided human image generation.
Subsequently, methods such as PISE (Zhang et al., 2021), SPGnet (Lv et al.,
2021), and CASD (Zhou et al., 2022) leverage parsing maps to generate the
final image. CoCosNet (Zhou et al., 2021) extracts dense correspondence
between cross-domain images through attention-based operations. However,
person image synthesis is essentially a transformation from non-aligned images
to images, and the absence of global appearance features of the target image
can lead to less realistic results.

Figure 2: The three-stage pipeline of Progressive Conditional Diffusion Models
(PCDMs) progressively operates to generate the final high-quality and high-
fidelity synthesized image.

Diffusion Models. Diffusion models (Ho et al., 2020; Song et al., 2020) have
recently emerged as a prominent generative method, renowned for synthesizing
high-quality images. Following the success in unconditional generation tasks,
diffusion models have expanded to conditional generation tasks, demonstrating
competitive and superior performance compared to GANs and VAEs. Unlike other
generative methods, diffusion models employ a multi-step denoising process
instead of generating the target image in a single step, which helps to better
preserve input information. Moreover, this denoising process can enhance
texture details, often producing sharper images than GANs and VAEs. Recent
studies (Bhunia et al., 2023; Zhang & Zhou, 2023) have already explored person
image synthesis based on diffusion models. MGD (Baldrati et al., 2023) guides
the generation process by constraining a latent diffusion model with the
model’s pose, the garment sketch, and a textual description of the garment
itself. PIDM (Bhunia et al., 2023) introduces a texture diffusion module and
disentangled classifier-free guidance to ensure that the conditional input and
the generated output are consistent regarding pose and appearance information.
Given the robust generation capabilities of the diffusion model, we devise a
framework with progressive conditional diffusion models, which consist of
three pivotal stages: prior, inpainting, and refining.

##  3 Method

An overview of our Progressive Conditional Diffusion Models (PCDMs) is
described in Figure 2, which contains a prior conditional diffusion model, an
inpainting conditional diffusion model, and a refining conditional diffusion
model. Our method aims to leverage three-stage diffusion models to
incrementally bridge the gap between person images under the target and source
poses. The prior conditional diffusion model predicts the global features of
the target image by mining the global alignment relationship between pose
coordinates and image appearance (Section 3.2). Subsequently, the inpainting
conditional diffusion model utilizes the global features from the previous
stage to further enhance contextual features, generating a coarse-grained
synthetic image (Section 3.3). Furthermore, the refining conditional diffusion
model leverages the coarse-grained image generated in the prior stage, aiming
to accomplish texture refinement and enhance detail consistency (Section 3.4).

###  3.1 Preliminaries

Diffusion Model. Diffusion models are a type of generative models trained to
reverse the diffusion process. The diffusion process gradually adds Gaussian
noise to the data using a fixed Markov chain, while a denoising model is
trained to generate samples from Gaussian noise. Given an input data sample
$\bm{x}_{0}$ and an additional condition $\bm{c}$, the training objective of
diffusion model usually adopts a mean square error loss $L_{\text{simple}}$,
as follows,

$L_{\text{simple}}=\mathbb{E}_{\bm{x}_{0},\bm{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I}),\bm{c},t}\|\bm{\epsilon}-\bm{\epsilon}_{\theta}\big{(}\bm{x}_{t},\bm{c},t\big{)}\|^{2},\vspace{-5pt}$
(1)

where $\epsilon$ and $\epsilon_{\theta}$ represent the actual noise injected
at the corresponding diffusion timestep $t$ and the noise estimated by the
diffusion model $\theta$, respectively;
$\bm{x}_{t}=\alpha_{t}\bm{x}_{0}+\sigma_{t}\bm{\epsilon}$ is the noisy data at
$t$ step, and $\alpha_{t}$, $\sigma_{t}$ are fixed functions of $t$ in the
diffusion process. To reduce the computational resources, latent diffusion
models (LDMs) (Rombach et al., 2022) operate the diffusion and denoising
processes on the latent space encoded by a pretrained auto-encoder model.

Classifier-Free Guidance. In the context of conditional diffusion models,
classifier-free guidance (Ho & Salimans, 2022) is a technique commonly used to
balance image fidelity and sample diversity. During the training phase,
conditional and unconditional diffusion models are jointly trained by randomly
dropping $\bm{c}$. In the sampling phase, the noise is predicted by the
conditional model $\bm{\epsilon}_{\theta}(\bm{x}_{t},\bm{c},t)$ and the
unconditional model $\bm{\epsilon}_{\theta}(\bm{x}_{t},t)$ according to Eq. 2,
as follows,

$\hat{\bm{\epsilon}}_{\theta}(\bm{x}_{t},\bm{c},t)=w\bm{\epsilon}_{\theta}(\bm{x}_{t},\bm{c},t)+(1-w)\bm{\epsilon}_{\theta}(\bm{x}_{t},t),\vspace{-5pt}$
(2)

where $w$ is the guidance scale used to control the strength of condition
$\bm{c}$.

###  3.2 Prior conditional diffusion model

Figure 3:  Illustration of the prior conditional diffusion model. The prior
conditional diffusion model uses pose coordinates and global alignment
relationship of the image to predict the global features of the target image.

In the first stage, we propose a simple prior conditional diffusion model,
designed to predict the global embedding of the target image. Here, we choose
the image embedding extracted from CLIP (Radford et al., 2021) image encoder
as the global embedding of the target image. CLIP is trained via contrastive
learning on a large-scale image-text paired dataset. Hence, the image
embedding can capture rich image content and style information, which can be
used to guide subsequent target image synthesis.

As depicted in Figure 3, the prior conditional diffusion model is a
transformer network, conditioned on the pose of the source image, the pose of
the target image, and the source image. We first adopt OpenPose (Cao et al.,
2017) to acquire the pose coordinates for the pose of source and target
images. A compact trainable pose network, composed of 3 linear layers, is used
to project the pose coordinates into the pose embedding. For the source image,
we also use a CLIP image encoder to extract the image embedding and add a
linear layer to project the image embedding. Additionally, we add an extra
embedding to predict the target global embedding. The above embeddings plus
timestep embedding and noisy image embedding of the target image are
concatenated into a sequence of embeddings as the input of the transformer
network.

Following unCLIP (Ramesh et al., 2022), the prior diffusion model is trained
to predict the unnoised image embedding directly rather than the noise added
to the image embedding. Given the source and target pose features $\bm{p}_{s}$
and $\bm{p}_{t}$, and the source image global feature $\bm{x}_{s}$, the
training loss $L^{\mathrm{prior}}$ of prior diffusion model $\bm{x}_{\theta}$
is defined as follows,

$\vspace{-5pt}L^{\mathrm{prior}}=\mathbb{E}_{\bm{x}_{0},\bm{x}_{s},\bm{p}_{s},\bm{p}_{t},t}\|\bm{x}_{0}-\bm{x}_{\theta}\big{(}\bm{x}_{t},\bm{x}_{s},\bm{p}_{s},\bm{p}_{t},t\big{)}\|^{2}.\vspace{-5pt}$
(3)

Once the model learns the conditional distribution, the inference is performed
according to Eq. 4, as follows,

$\vspace{-5pt}\begin{split}\hat{\bm{x}}_{\theta}(\bm{x}_{t},\bm{x}_{s},\bm{p}_{s},\bm{p}_{t},t)=w\bm{x}_{\theta}(\bm{x}_{t},\bm{x}_{s},\bm{p}_{s},\bm{p}_{t},t)+(1-w)\bm{x}_{\theta}(\bm{x}_{t},t).\end{split}\vspace{-5pt}$
(4)

###  3.3 Inpainting conditional diffusion model

Figure 4: Overview of the inpainting conditional diffusion model. The
inpainting conditional diffusion model utilizes the global features obtained
from the previous stage to establish dense correspondences.

Figure 5: Illustrative of the refining conditional diffusion model. The
refining conditional diffusion model leverages the coarse-grained image
generated in the previous stage to rectify textures and ensure consistency.

With the global features of the target image obtained in the first stage, we
propose an inpainting conditional diffusion model to establish dense
correspondences between the source and target, and transform the unaligned
image-to-image generation task into an aligned one. As shown in Figure 5, we
concatenate the source and target images, the source and target poses, and the
source and mask images along the width dimension. To prevent confusion between
black and white in the source and target images, we add a single-channel
marker symbol (omitted in the figure) with the same width and height as the
input. We use 0 and 1 to represent masked and unmasked pixels, respectively.
We then concatenate the global features of the target obtained from the prior
conditional diffusion model (prior model) and the local features of the source
image. This ensures that the input conditions of the model include the
entirety of the source and target and are aligned at three levels: image,
pose, and feature, which is overlooked in existing work.

Specifically, we use a pose encoder with four convolution layers similar to
ControlNet (Zhang & Agrawala, 2023) to extract the pose features from the pose
skeleton image. Unlike the prior model that uses pose coordinates, we expect
this model to maintain image modality alignment throughout the learning phase,
especially spatial information. For the source image, we use a frozen image
encoder and a trainable MLP to extract the fine-grained features of the source
image. Inspired by (Chen et al., 2023), we opt for DINOv2 (Oquab et al., 2023)
as the image encoder because it can extract fine details. To better utilize
the global features of the target image obtained from the previous stage, we
also add it to the timestep embedding, which is embedded in the ResNet blocks
of the entire network. The loss function $L^{\mathrm{inpainting}}$ of
inpainting conditional diffusion model according to Eq. 5, as follows,

$\vspace{-5pt}L^{\mathrm{inpainting}}=\mathbb{E}_{\bm{x}_{0},\bm{\epsilon},\bm{f}_{st},\bm{p}_{st},\bm{i}_{sm},t}\|\bm{\epsilon}-\bm{\epsilon}_{\theta}\big{(}\bm{x}_{t},\bm{f}_{st},\bm{p}_{st},\bm{i}_{sm},t\big{)}\|^{2},$
(5)

where $\bm{f}_{st}$, $\bm{p}_{st}$, and $\bm{i}_{sm}$ respectively represent
the feature embeddings obtained by concatenating the source and target global
features, the feature embeddings of source and target poses, and the feature
embeddings of source and mask images.

In the inference stage, we also use classifier-free guidance according to Eq.
6, as follows,

$\vspace{-5pt}\begin{split}\hat{\bm{\epsilon}}_{\theta}(\bm{x}_{t},\bm{f}_{st},\bm{p}_{st},\bm{i}_{sm},t)=w\bm{\epsilon}_{\theta}(\bm{x}_{t},\bm{f}_{st},\bm{i}_{sm},t)+(1-w)\bm{\epsilon}_{\theta}(\bm{x}_{t},\bm{p}_{st},t).\end{split}$
(6)

###  3.4 Refining conditional diffusion model

Following the second stage, we obtain a preliminary generated coarse-grained
target image. To further enhance the image quality and detail texture, as
shown in Figure 5, we propose a refining conditional diffusion model. This
model uses the coarse-grained image generated in the previous stage as a
condition to improve the quality and fidelity of the synthesized image. We
first concatenate the coarse-grained target image with the noisy image along
the channel, which can be easily achieved by modifying the first convolutional
layer of the diffusion model based on the UNet architecture. Then, we use the
DINOv2 image encoder and a learnable MLP layer to extract features for the
source image. Finally, we infuse texture features into the network through a
cross-attention mechanism to guide the model in texture repair and enhance
detail consistency.

Assume that the coarse target features $\bm{i}_{ct}$ and source image features
$\bm{x}_{s}$ are given, the loss function of refining conditional diffusion
model is defined as follows,

$\vspace{-5pt}L^{\mathrm{prior}}=\mathbb{E}_{\bm{x}_{0},\bm{\epsilon},\bm{i}_{ct},\bm{x}_{s},t}\|\bm{\epsilon}-\bm{\epsilon}_{\theta}\big{(}\bm{x}_{t},\bm{i}_{ct},\bm{x}_{s},t\big{)}\|^{2}.\vspace{-5pt}$
(7)

In the inference phase, we use the following Eq. 8,

$\vspace{-5pt}\begin{split}\hat{\bm{\epsilon}}_{\theta}(\bm{x}_{t},\bm{i}_{ct},\bm{f}_{s},t)=w\bm{\epsilon}_{\theta}(\bm{x}_{t},\bm{i}_{ct},\bm{f}_{s},t)+(1-w)\bm{\epsilon}_{\theta}(\bm{x}_{t},t).\end{split}\vspace{-5pt}$
(8)

Table 1: Quantitative comparison of the proposed PCDMs with several state-of-
the-art models.

Dataset | Methods | SSIM ($\uparrow$) | LPIPS ($\downarrow$) | FID ($\downarrow$)
---|---|---|---|---
DeepFashion (Liu et al., 2016) ($256\times 176$) | Def-GAN (Siarohin et al., 2018) | 0.6786 | 0.2330 | 18.457
PATN (Zhu et al., 2019) | 0.6709 | 0.2562 | 20.751
ADGAN (Men et al., 2020) | 0.6721 | 0.2283 | 14.458
PISE (Zhang et al., 2021) | 0.6629 | 0.2059 | 13.610
GFLA (Ren et al., 2020) | 0.7074 | 0.2341 | 10.573
DPTN (Zhang et al., 2022) | 0.7112 | 0.1931 | 11.387
CASD (Zhou et al., 2022) | 0.7248 | 0.1936 | 11.373
NTED (Ren et al., 2022) | 0.7182 | 0.1752 | 8.6838
PIDM (Bhunia et al., 2023) | 0.7312 | 0.1678 | 6.3671
PCDMs (Ours) | 0.7444 | 0.1365 | 7.4734
DeepFashion (Liu et al., 2016) ($512\times 352$) | CocosNet2 (Zhou et al., 2021) | 0.7236 | 0.2265 | 13.325
NTED (Ren et al., 2022) | 0.7376 | 0.1980 | 7.7821
PIDM (Bhunia et al., 2023) | 0.7419 | 0.1768 | 5.8365
| PCDMs (Ours) | 0.7601 | 0.1475 | 7.5519
Market-1501 (Zheng et al., 2015) ($128\times 64$) | Def-GAN (Siarohin et al., 2018) | 0.2683 | 0.2994 | 25.364
PTN (Zhu et al., 2019) | 0.2821 | 0.3196 | 22.657
GFLA (Ren et al., 2020) | 0.2883 | 0.2817 | 19.751
DPTN (Zhang et al., 2022) | 0.2854 | 0.2711 | 18.995
| PIDM (Bhunia et al., 2023) | 0.3054 | 0.2415 | 14.451
| PCDMs (Ours) | 0.3169 | 0.2238 | 13.897

##  4 Experiments

Datasets. We carry out experiments on DeepFashion (Liu et al., 2016), which
consists of 52,712 high-resolution images of fashion models, and Market-1501
(Zheng et al., 2015) including 32,668 low-resolution images with diverse
backgrounds, viewpoints, and lighting conditions. We extract the skeletons
using OpenPose (Cao et al., 2017) and follow the dataset splits provided by
(Bhunia et al., 2023). Note that the person ID of the training and testing
sets do not overlap for both datasets.

Metrics. We conduct a comprehensive evaluation of the model, considering both
objective and subjective metrics. Objective indicators include structural
similarity index measure (SSIM) (Wang et al., 2004), learned perceptual image
patch similarity (LPIPS) (Zhang et al., 2018), and fréchet inception distance
(FID) (Heusel et al., 2017). In contrast, subjective assessments prioritize
user-oriented metrics (Zhou et al., 2022), including the percentage of real
images misclassified as generated images (R2G), the percentage of generated
images misclassified as real images (G2R), and the percentage of images deemed
superior among all models (Jab).

Implementations. We perform our experiments on 8 NVIDIA V100 GPUs. Our
configurations can be summarized as follows: (1) the transformer of the prior
model has 20 transformer blocks with a width of 2,048. For the inpainting
model and refining model, we use the pretrained stable diffusion V2.1
111https://huggingface.co/stabilityai/stable-diffusion-2-1-base and modify the
first convolution layer to adapt additional conditions. (2) We employ the
AdamW optimizer with a fixed learning rate of $1e^{-4}$ in all stages. (3)
Following (Ren et al., 2022; Bhunia et al., 2023), we train our models using
images of sizes 256 $\times$ 176 and 512 $\times$ 352 for DeepFashion dataset.
For the Market-1501 dataset, we utilize images of size 128 $\times$ 64\.
Please refer to B of the Appendix for more detail.

###  4.1 Quantitative and Qualitative Results

We quantitatively compare our proposed PCDMs with several state-of-the-art
methods, including Def-GAN (Siarohin et al., 2018), PATN (Zhu et al., 2019),
ADGAN (Men et al., 2020), PISE (Zhang et al., 2021), GFLA (Ren et al., 2020),
DPTN (Zhang et al., 2022), CASD (Zhou et al., 2022), CocosNet2 (Zhou et al.,
2021), NTED (Ren et al., 2022) and PIDM (Bhunia et al., 2023).

Figure 6: Qualitative comparisons with several state-of-the-art models on the
DeepFashion dataset.

Quantitative Results. As shown in Table 1, our approach excels in two out of
three metrics on the DeepFashion compared to other models, regardless of
whether it is based on GAN, VAE, flow-based model, or diffusion model. For
example, when using the flow framework (i.e., CASD), PCDMs outperforms the
SOTA flow-based method, CASD, even without explicitly decoupling pose and
style. Although PCDMs have a lower FID score than PIDM (which also employs the
diffusion model), we surpass it on the other two metrics, and subsequent
experiments further demonstrate our method’s superiority.

The comparison results on the Market-1501 are summarized in Table 1. Notably,
PCDMs outshines all SOTA methods, achieving the best SSIM, LPIPS, and FID in
all experiments. Specifically, compared to methods that consider fine-grained
texture features, PCDMs outperforms NTED significantly in terms of both LPIPS
and SSIM. While NTED explores the texture feature of the target in the source
image and target pose, PCDMs extract target features by mining the global
alignment relationship between pose coordinates and image appearance.
Furthermore, regarding SSIM, LPIPS, and FID, PCDMs performs better than PIDM,
which validated that PCDMs can accurately transfer the texture from the source
image to the target pose while maintaining high consistency and realism.

Qualitative Results. As shown in Figure 6, we comprehensively compare our
PCDMs and other state-of-the-art methods on the DeepFashion dataset. Several
observations can be drawn from the results: (1) despite the minuscule size of
the necklace in the source image (as seen in the first and second rows), only
our proposed PCDMs and the PIDM, which also utilizes a diffusion model, can
focus on it. But PCDMs can generate higher-quality images compared to the
PIDM. (2) In scenarios involving extreme poses and large area occlusions (as
seen in the third and fourth rows), only our method can generate images that
align reasonably with the target. This can be attributed to our method’s
ability to capture and enhance the global features of the target image. (3) In
situations with complex textures and numbers (as seen in the fifth to seventh
rows), our method significantly surpasses others in preserving appearance
textures, primarily due to our method’s capability to refine textures and
maintain consistency. To sum up, our method consistently produces more
realistic and lifelike person images, demonstrating the advantage of our
PCDMs’ multi-stage progressive generation approach. See C.1 for more examples.

Figure 7: User study results on DeepFashion in terms of R2G, G2R and Jab
metric. Higher values in these three metrics indicate better performance.

User Study. The above quantitative and qualitative comparisons underscore the
substantial superiority of our proposed PCDMs in generating results. However,
tasks of pose-guided person image synthesis are typically human perception-
oriented. Consequently, we conducted a user study involving 30 volunteers with
computer vision backgrounds.

This study included comparisons with fundamental facts (namely, R2G and G2R)
and comparisons with other methods (i.e., J2b). A higher score on these three
metrics indicates superior performance. As shown in Figure 7, PCDMs exhibit
commendable performance across all three baseline metrics on DeepFashion. For
instance, the proportion of PCDMs images perceived as real in all instances is
56.2% (G2R), nearly 20.4% higher than the next best model. Our Jab score
stands at 44.1%, suggesting a preference for our method among the
participants. Please refer to C.4 of the Appendix for more detail.

Figure 8: Qualitative ablation results on the DeepFashion. See C.2 for more
examples.

###  4.2 Ablation Study

To demonstrate the efficacy of each stage introduced within this paper, we
have devised the following variations, all of which fall under the PCDMs
framework but incorporate distinct configurations. B1 represents using the
inpainting conditional diffusion model only. B2 denotes the simultaneous use
of the prior conditional diffusion model and the inpainting conditional
diffusion model without using the refining conditional diffusion model. B3
stands for the simultaneous use of the redrawn conditional diffusion model and
the refining conditional diffusion model, excluding the prior conditional
diffusion model.

Figure 8 shows the impact of each stage on the DeepFashion dataset. B1 can
generate person images that roughly conform to the target pose. However, there
are severe distortions in the generated image, such as limb loss and confusion
between appearance and limbs. This indicates that generating images from
unaligned inputs is a highly challenging task without global features. In
contrast, in B2, once the global features of the target image obtained from
the prior conditional diffusion model are added, the generated person images
basically match the target pose in structure. Although the personal images
generated by B2 can retain the appearance of the source image is limited, it
is already very close to the real image. This shows that the refining
conditional diffusion model can establish a dense correspondence between the
source and target images, enhancing the contextual features. In addition, from
a visual perception perspective, B3 is superior to B1 and B2 regarding detail
texture, although it is slightly inferior to B2 regarding pose coordination.
Finally, when we use the PCDMs of the three stages in the last column, it is
visually superior to B1, B2, and B3. This indicates that when dealing with
complex poses, the PCDMs of the three stages can gradually produce visually
more satisfying results.

To more comprehensively validate the effectiveness of the proposed refining
conditional diffusion model, we apply it to other state-of-the-art (SOTA)
methods. As shown in Figure 9, the first and fourth rows denote the source
image and ground truth (target image). The second and third rows present the
results before and after refinement via refining the conditional diffusion
model.

Figure 9: Effect of refining conditional diffusion model on other SOTA
methods.

We can observe that the refining conditional diffusion model significantly
improves the results of all state-of-the-art methods. For instance, when
dealing with ADGAN and PIDM, our method helps to fill in minor details, such
as missing belts and hats, thereby enhancing the completeness of the generated
images. For methods like GFLA, DPTN, CASD, and NTED, our model can finely
process textures, maintain shape and texture consistency, and generate more
explicit and realistic images. These results indicate that the refining
conditional diffusion model has universality across different state-of-the-art
methods, offering potential improvements for various person image generation
tasks.

###  4.3 Application

Table 2: Comparison with SOTA on person re-identification. Standard denotes
not using synthesized pedestrian images.

| Percentage of real images |
---|---|---
Methods | 20% | 40% | 60% | 80% | 100%
Standard | 33.4 | 56.6 | 64.9 | 69.2 | 76.7
PTN (Zhu et al., 2019) | 55.6 | 57.3 | 67.1 | 72.5 | 76.8
GFLA (Ren et al., 2020) | 57.3 | 59.7 | 67.6 | 73.2 | 76.8
DPTN (Zhang et al., 2022) | 58.1 | 62.6 | 69.0 | 74.2 | 77.1
PIDM (Bhunia et al., 2023) | 61.3 | 64.8 | 71.6 | 75.3 | 78.4
PCDMs (Ours) | 63.8 | 67.1 | 73.3 | 76.4 | 80.3

In Table 2, we further evaluate the applicability of images generated by PCDMs
in a downstream task, i.e., person re-identification. We conduct these re-
identification experiments on the Market-1501 dataset, adhering to the PIDM
protocol (Bhunia et al., 2023). Initially, we randomly select subsets of 20%,
40%, 60%, and 80% from the total real training set of the Market-1501 dataset,
ensuring each identity is represented by at least one image. This selection
process yields a new dataset. Following this, we employ BoT (Luo et al., 2019)
as the base network and conduct baseline training with each subset of the
data. We then generate synthetic images from the new dataset, randomly
selecting those that share the same identity and pose. This synthesized data
is then merged with the original dataset to train BoT. The Rank1 results are
presented in Table 2. The results indicate that PCDMs significantly boost the
re-identification performance compared to the baseline. Furthermore, when
compared with state-of-the-art methods such as PTN (Zhu et al., 2019), GFLA
(Ren et al., 2020), DPTN (Zhang et al., 2022), and PIDM (Bhunia et al., 2023),
PCDMs consistently demonstrate superior performance in re-identification
tasks.

##  5 Conclusion

This paper has demonstrated the significant potential of Progressive
Conditional Diffusion Models (PCDMs) in addressing the challenges of pose-
guided person image synthesis. The inherent inconsistency in pose between
source and target images has been a formidable obstacle in synthesizing an
image with a distinct pose, relying solely on the source image and target pose
information. Our proposed PCDMs incrementally bridge this gap through a three-
stage process. In the first stage, a simple prior conditional diffusion model
is designed to predict the global features of the target image by mining the
global alignment relationship between pose coordinates and image appearance.
The second stage establishes a dense correspondence between the source and
target images using the global features from the previous stage, and an
inpainting conditional diffusion model is proposed to further align and
enhance the contextual features, generating a coarse-grained person image. In
the final stage, a refining conditional diffusion model is proposed to utilize
the coarsely generated image from the previous stage as a condition, achieving
texture restoration and enhancing fine-detail consistency. The three stages of
the PCDMs work progressively to generate the final high-quality and high-
fidelity synthesized image. Both qualitative and quantitative results have
demonstrated the consistency and photorealism of our proposed PCDMs under
challenging scenarios.

Futher Work. Our method significantly improves person image synthesis quality,
while the use of two additional prior and refining models leads to increased
computational resource consumption and longer inference time. Future work
should explore efficient methods that provide equivalent or superior quality
while reducing computational overhead and inference time.

##  6 Ethics Statement

This study introduces a novel multi-stage person image synthesis technique
capable of generating new person images based on different poses and original
images. However, there is a risk that malicious actors could misuse this
manipulation of real photos to create false content and disseminate
misinformation. This is a well-known issue faced by virtually all person image
synthesis methods. Nevertheless, research has made significant progress in
identifying and preventing malicious tampering. Our work will provide valuable
support for research in this field and external audits, helping to balance its
value against the risks posed by unrestricted open access. This will ensure
that this technology can be used safely and beneficially.

##  7 Reproducibility

The data, code, and models will be publicly available.

## References

  * Baldrati et al. (2023) Alberto Baldrati, Davide Morelli, Giuseppe Cartella, Marcella Cornia, Marco Bertini, and Rita Cucchiara.  Multimodal garment designer: Human-centric latent diffusion models for fashion image editing.  _arXiv preprint arXiv:2304.02051_ , 2023.
  * Bhunia et al. (2023) Ankan Kumar Bhunia, Salman Khan, Hisham Cholakkal, Rao Muhammad Anwer, Jorma Laaksonen, Mubarak Shah, and Fahad Shahbaz Khan.  Person image synthesis via denoising diffusion model.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 5968–5976, 2023.
  * Cao et al. (2017) Zhe Cao, Tomas Simon, Shih-En Wei, and Yaser Sheikh.  Realtime multi-person 2d pose estimation using part affinity fields.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 7291–7299, 2017.
  * Chen et al. (2023) Xi Chen, Lianghua Huang, Yu Liu, Yujun Shen, Deli Zhao, and Hengshuang Zhao.  Anydoor: Zero-shot object-level image customization.  _arXiv preprint arXiv:2307.09481_ , 2023.
  * Creswell et al. (2018) Antonia Creswell, Tom White, Vincent Dumoulin, Kai Arulkumaran, Biswa Sengupta, and Anil A Bharath.  Generative adversarial networks: An overview.  _IEEE signal processing magazine_ , 35(1):53–65, 2018.
  * Esser et al. (2018) Patrick Esser, Ekaterina Sutter, and Björn Ommer.  A variational u-net for conditional appearance and shape generation.  In _CVPR_ , 2018.
  * Han et al. (2019) Xintong Han, Xiaojun Hu, Weilin Huang, and Matthew R Scott.  Clothflow: A flow-based model for clothed person generation.  In _Proceedings of the IEEE/CVF international conference on computer vision_ , pp. 10471–10480, 2019.
  * Heusel et al. (2017) Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  _Advances in neural information processing systems_ , 30, 2017.
  * Ho & Salimans (2022) Jonathan Ho and Tim Salimans.  Classifier-free diffusion guidance.  _arXiv preprint arXiv:2207.12598_ , 2022.
  * Ho et al. (2020) Jonathan Ho, Ajay Jain, and Pieter Abbeel.  Denoising diffusion probabilistic models.  In _NeurIPS_ , 2020.
  * Kingma et al. (2019) Diederik P Kingma, Max Welling, et al.  An introduction to variational autoencoders.  _Foundations and Trends® in Machine Learning_ , 12(4):307–392, 2019.
  * Li et al. (2019) Yining Li, Chen Huang, and Chen Change Loy.  Dense intrinsic appearance flow for human pose transfer.  In _CVPR_ , 2019.
  * Liu et al. (2019) Wen Liu, Zhixin Piao, Jie Min, Wenhan Luo, Lin Ma, and Shenghua Gao.  Liquid warping gan: A unified framework for human motion imitation, appearance transfer and novel view synthesis.  In _ICCV_ , 2019.
  * Liu et al. (2016) Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang.  Deepfashion: Powering robust clothes recognition and retrieval with rich annotations.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 1096–1104, 2016.
  * Luo et al. (2019) Hao Luo, Youzhi Gu, Xingyu Liao, Shenqi Lai, and Wei Jiang.  Bag of tricks and a strong baseline for deep person re-identification.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition workshops_ , pp. 0–0, 2019.
  * Lv et al. (2021) Zhengyao Lv, Xiaoming Li, Xin Li, Fu Li, Tianwei Lin, Dongliang He, and Wangmeng Zuo.  Learning semantic person image generation by region-adaptive normalization.  In _CVPR_ , 2021.
  * Ma et al. (2017) Liqian Ma, Xu Jia, Qianru Sun, Bernt Schiele, Tinne Tuytelaars, and Luc Van Gool.  Pose guided person image generation.  In _NeurIPS_ , 2017.
  * Men et al. (2020) Yifang Men, Yiming Mao, Yuning Jiang, Wei-Ying Ma, and Zhouhui Lian.  Controllable person image synthesis with attribute-decomposed gan.  In _CVPR_ , 2020.
  * Mirza & Osindero (2014) Mehdi Mirza and Simon Osindero.  Conditional generative adversarial nets.  _arXiv preprint arXiv:1411.1784_ , 2014.
  * Nichol & Dhariwal (2021) Alexander Quinn Nichol and Prafulla Dhariwal.  Improved denoising diffusion probabilistic models.  In _International Conference on Machine Learning_ , pp. 8162–8171. PMLR, 2021.
  * Oquab et al. (2023) Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al.  Dinov2: Learning robust visual features without supervision.  _arXiv preprint arXiv:2304.07193_ , 2023.
  * Radford et al. (2021) Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pp. 8748–8763. PMLR, 2021.
  * Ramesh et al. (2022) Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen.  Hierarchical text-conditional image generation with clip latents.  _arXiv preprint arXiv:2204.06125_ , 1(2):3, 2022.
  * Ren et al. (2020) Yurui Ren, Xiaoming Yu, Junming Chen, Thomas H Li, and Ge Li.  Deep image spatial transformation for person image generation.  In _CVPR_ , 2020.
  * Ren et al. (2021) Yurui Ren, Yubo Wu, Thomas H Li, Shan Liu, and Ge Li.  Combining attention with flow for person image synthesis.  In _Proceedings of the 29th ACM International Conference on Multimedia_ , pp. 3737–3745, 2021.
  * Ren et al. (2022) Yurui Ren, Xiaoqing Fan, Ge Li, Shan Liu, and Thomas H Li.  Neural texture extraction and distribution for controllable person image synthesis.  In _CVPR_ , 2022.
  * Rombach et al. (2022) Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer.  High-resolution image synthesis with latent diffusion models.  In _Proceedings of the IEEE/CVF conference on computer vision and pattern recognition_ , pp. 10684–10695, 2022.
  * Siarohin et al. (2018) Aliaksandr Siarohin, Enver Sangineto, Stéphane Lathuiliere, and Nicu Sebe.  Deformable gans for pose-based human image generation.  In _CVPR_ , 2018.
  * Song et al. (2020) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole.  Score-based generative modeling through stochastic differential equations.  _arXiv preprint arXiv:2011.13456_ , 2020.
  * Tang et al. (2020) Hao Tang, Song Bai, Li Zhang, Philip HS Torr, and Nicu Sebe.  Xinggan for person image generation.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXV 16_ , pp. 717–734. Springer, 2020.
  * Wang et al. (2004) Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli.  Image quality assessment: from error visibility to structural similarity.  _IEEE transactions on image processing_ , 13(4):600–612, 2004.
  * Ye et al. (2021) Mang Ye, Jianbing Shen, Gaojie Lin, Tao Xiang, Ling Shao, and Steven CH Hoi.  Deep learning for person re-identification: A survey and outlook.  _IEEE transactions on pattern analysis and machine intelligence_ , 44(6):2872–2893, 2021.
  * Zhang et al. (2021) Jinsong Zhang, Kun Li, Yu-Kun Lai, and Jingyu Yang.  Pise: Person image synthesis and editing with decoupled gan.  In _CVPR_ , 2021.
  * Zhang & Agrawala (2023) Lvmin Zhang and Maneesh Agrawala.  Adding conditional control to text-to-image diffusion models.  _arXiv preprint arXiv:2302.05543_ , 2023.
  * Zhang et al. (2022) Pengze Zhang, Lingxiao Yang, Jian-Huang Lai, and Xiaohua Xie.  Exploring dual-task correlation for pose guided person image generation.  In _CVPR_ , 2022.
  * Zhang et al. (2018) Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang.  The unreasonable effectiveness of deep features as a perceptual metric.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pp. 586–595, 2018.
  * Zhang & Zhou (2023) Songchuan Zhang and Yiwei Zhou.  Posediffusion: a pose-guided human image generator via cascaded diffusion models.  _Available at SSRN 4552596_ , 2023.
  * Zheng et al. (2015) Liang Zheng, Liyue Shen, Lu Tian, Shengjin Wang, Jingdong Wang, and Qi Tian.  Scalable person re-identification: A benchmark.  In _ICCV_ , 2015.
  * Zhou et al. (2021) Xingran Zhou, Bo Zhang, Ting Zhang, Pan Zhang, Jianmin Bao, Dong Chen, Zhongfei Zhang, and Fang Wen.  Cocosnet v2: Full-resolution correspondence learning for image translation.  In _CVPR_ , 2021.
  * Zhou et al. (2022) Xinyue Zhou, Mingyu Yin, Xinyuan Chen, Li Sun, Changxin Gao, and Qingli Li.  Cross attention based style distribution for controllable person image synthesis.  In _European Conference on Computer Vision_ , pp. 161–178. Springer, 2022.
  * Zhu et al. (2019) Zhen Zhu, Tengteng Huang, Baoguang Shi, Miao Yu, Bofei Wang, and Xiang Bai.  Progressive pose attention transfer for person image generation.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pp. 2347–2356, 2019.

## Supplementary Material

This supplementary material offers a more detailed exploration of the
experiments and methodologies proposed in the main paper. Section A provides a
series of symbols and definitions for enhanced comprehension. Section B delves
deeper into the implementation specifics of our experiments. Section C
presents additional experimental outcomes, including a broader range of
qualitative comparison examples with state-of-the-art methods, an expanded set
of ablation study cases, supplemented results from random sampling, and a
detailed explanation of our user studies.

##  Appendix A Some Notations and Definitions

Table 3: Some notations and definitions. Notation | Definition
---|---
$\bm{x_{0}}$ | Real image
$\bm{c}$ | Additional condition
$\bm{t}$ | Timestep
$\bm{\theta}$ | Diffusion model
$\bm{\epsilon}$ | Gaussian noise
$\bm{w}$ | Guidance scale
$\bm{x_{t}}$ | Noisy data at $t$ step
$\bm{x_{s}}$ | Feature of source image
$\bm{p_{s}}$ | Feature of source pose coordinate
$\bm{p_{t}}$ | Feature of target pose coordinate
$\bm{f_{st}}$ | Feature of source and target image
$\bm{i_{sm}}$ | Feature of source and mask image
$\bm{p_{st}}$ | Feature of source and target pose coordinate
$\bm{i_{ct}}$ | Feature of coarse target image

##  Appendix B Implement Details

Our experiments are conducted on 8 NVIDIA V100 GPUs. We utilize the AdamW
optimizer with a consistent learning rate of $1e^{-4}$ across all stages. The
probability of random dropout for condition $c$ is set at 10%. For the prior
conditional diffusion model, we employ OpenCLIP ViT-H/14
222https://github.com/mlfoundations/open_clip as the CLIP image encoder. This
model’s transformer consists of 20 transformer blocks, each with a width of
2,048. The model is trained for 10k iterations with a batch size of 256, using
a cosine noising schedule (Nichol & Dhariwal, 2021) with 1000 timesteps. For
the inpainting and refining models, we use DINOv2-G/14
333https://github.com/facebookresearch/dinov2 as the image encoder. We
leverage the pretrained stable diffusion V2.1
444https://huggingface.co/stabilityai/stable-diffusion-2-1-base, modifying the
first convolution layer to accommodate additional conditions. These models are
trained for 200k and 100k iterations, respectively, each with a batch size of
128, and a linear noise schedule with 1000 timesteps is applied. In the
inference stage, we use the DDIM sampler with 50 steps and set the guidance
scale w to 2.0 for PCDMs.

##  Appendix C Additional Results

###  C.1 More Qualitative Comparisons for PCDMs

We provide additional examples for comparison with the state-of-the-art (SOTA)
methods in Figure 10 and Figure 11.

Figure 10: More qualitative comparisons between PCDMs and SOTA methods on the
DeepFashion dataset.  Figure 11: More qualitative comparisons between PCDMs
and SOTA methods on the DeepFashion dataset.

###  C.2 More qualitative ablation results on the DeepFashion dataset

We show more qualitative ablation results from our proposed PCDMs in Figure
12. B1 signifies the exclusive use of the inpainting conditional diffusion
model. B2 represents the concurrent application of both the prior and
inpainting conditional diffusion models, without incorporating the refining
conditional diffusion model. B3 denotes the combined usage of the inpainting
and refining conditional diffusion models, while excluding the prior
conditional diffusion model.

Figure 12: More qualitative ablation results on the DeepFashion dataset.

###  C.3 Random Samples

We show random samples from our proposed PCDMs in Figure 13. The generated
results are highly stable and realistic.

Figure 13: Random samples from PCDMs.

###  C.4 User Study

In Section 4.1, we conduct a user study on the pose-guided person image
synthesis task. We invite 30 volunteers with a computer science background to
participate in a side-by-side view comparison of PCDMs and state-of-the-art
work and to identify the authenticity of images. These samples are randomly
drawn from our toolkit test set, originating from the same original image and
pose. We conducted repeated tests with 50 samples for each model. The user
study evaluates which model produces more realistic results through human
perception. Example questions are shown in Figure 14 and Figure 15.

Figure 14: An example question used in our user study for pose-guided person
image synthesis.  Figure 15: An example question used in our user study for
pose-guided person image synthesis.


Learning from demonstrations is an easy and intuitive way to show
examples of successful behavior to a robot. However, the fact that
humans optimize or take advantage of their body and not of the
robot, usually called the embodiment problem in robotics,
often prevents industrial robots from executing the task in a
straightforward way. The shown movements often do not or cannot
utilize the degrees of freedom of the robot efficiently, and
moreover can suffer from excessive execution errors. In this paper,
we explore a variety of solutions that address these
shortcomings. In particular, we learn sparse movement primitive
parameters from several demonstrations of a successful table tennis
serve. The number of parameters learned using our procedure is
independent of the degrees of freedom of the robot. Moreover, they
can be ranked according to their importance in the regression
task. Learning few parameters that are ranked is a desirable feature
to combat the curse of dimensionality in Reinforcement
Learning. Real robot experiments on the Barrett WAM for
a table tennis serve using the learned movement primitives show that
the representation can capture successfully the style of the
movement with few parameters.

Learning from Demonstration, Learning and Adaptive Systems, Optimization, Learning a Sparse Representation.

§ INTRODUCTION

Humans are good at using their bodies to great effect, taking
advantage of their muscular structure and soft but flexible
actuation. Much of dexterous manipulation, or dynamic movement
generation reflects this awareness of the human body. When teaching
the robots to achieve similar tasks autonomously, however, we
inevitably impose and transfer our biases to the robot. This problem
of embodiment can cripple the execution, possibly also preventing
the robots from taking advantage of their kinematics structure and
actuation mechanisms.

In dynamic games like table tennis, we can easily observe humans
taking utmost advantage of their bodies and pushing it to its maximum,
i.e., optimizing their output bearing in mind their kinematic and
dynamic limits. Table tennis serves, for instance, incorporate flicks
(very fast accelerations of the wrist) that are designed to give an
unsuspected spin and motion profile to the ball. Teaching such
movements to the robots in a learning from demonstrations framework
using kinesthetic teach-in, where the robot joint movements are
recorded, suffers in particular from two drawbacks. Firstly, during
the shown movement, as discussed above, the human is unable to move
the shoulder joints of the robot adequately, which could potentially be
used by the robot to great effect. Secondly, the fast movements of the
wrists may not be tracked accurately by the robot, which is the case
for the cable-driven seven degree of freedom (DoF) Barrett WAM arm,
see Figure <ref>.

In this paper, we explore different learning from demonstrations (LfD)
approaches to compensate for the execution and transfer deficiencies
resulting from the demonstrated serves. The demonstrations are
acquired and the movement primitives are trained in the joint-space of
the robot, using kinesthetic teach-in, where the movements of the
robot are recorded using the joint-level sensors. The initial policy
or the movement template, extracted as a set of movement primitives,
can be thought of as a good initialization for a reinforcement
learning (RL) agent. By capturing the essence of the shown
demonstrations in as few parameters as possible, we simplify and
increase the effectiveness of the skill transfer to the

Sparsity is achieved in our framework in
joint-space[Discarding the joint-level information and using
only the Cartesian coordinates of the resulting movements, in a
similar attempt to reduce the dimensionality of the robot learning
problem, necessitates the use of inverse kinematics, running into
feasibility and additional execution problems that might be
artificially introduced.] by using a new iterative optimization
approach, where a multi-task Elastic Net regression is alternated with
a nonlinear optimization. The Elastic Net projects the solutions to a
sparse set of features, and during the nonlinear optimization these
features (the basis functions) are adapted to the data in a secondary
optimization. Moreover these features are shared across multiple
demonstrations, increasing the effectiveness of the feature learning

The fewer number of learned parameters using our iterative
optimization procedure, compared to more traditional approaches, is
independent of the robot DoF. This is a
desirable property for Reinforcement Learning to adapt the learned
parameters online. Moreover, by using the Elastic Net path, we can
rank the parameters in terms of importance, or effectiveness in
explaining the demonstration data. We perform preliminary experiments
on the Barrett WAM on a table tennis serve to validate the
effectiveness of our new movement primitives.

Our robot table tennis setup with a seven DoF Barrett WAM,
where we demonstrate, using kinesthetic teach-in, multiple good
table tennis serve movements while recording the resulting
joint-space robot trajectories. A metal piece is attached to the
end effector of the Barrett WAM, which connects to a standard sized
table tennis racket. An egg-holder on the metal piece holds the
ball initially before the serve. The demonstrator, after finding a
good starting posture, starts by swinging the arm, giving the ball
enough acceleration to propel it away from the robot. The ball is
then hit in midair by a careful adjustment of the robot
wrist. The initial posture, the swinging movement of the robot
shoulder joints and the elbow, and finally the turning of the
wrist all contribute to the style of the shown movement. Multiple
demonstrations starting from different initial postures are
recorded in one session. We compare and evaluate throughout the
paper different learning from demonstrations approaches using
these demonstrations. We propose a new iterative optimization
approach that can learn sparse parameters while adapting the
features of the movement primitives to the demonstration data.

Robot table tennis has, since the nineties, captivated the attention of the robot control and learning communities as a challenging and dynamic task, and research in it has been ongoing ever since. After the pioneering work of Anderson's analytical player [1], there have been various approaches focusing on certain parts of the game, such as simplifications in trajectory generation using a virtual hitting plane [20], [15] or learning striking trajectories from demonstrations [10]. Learning approaches to generate better strikes with Reinforcement Learning (RL) include [14], [4]. Recently, [11] has introduced a new trajectory generation framework in table tennis, where they solve a free final-time optimal control problem, generating minimum acceleration striking trajectories. This kinematic optimization approach was extended and evaluated in the real robot table tennis setup in [12].

The success of this and other similar model-based optimization approaches in dynamic tasks like table tennis heavily depends on the accuracy of the models. In the case of table tennis, an accurate ball model [17], [12] is especially difficult to acquire. The high spin rates make the ball flight difficult to model from first (physical) principles, while the various types of impacts make it also difficult to train machine learning approaches from raw ball position data. For the serve, an additional complication results from the ball take-off phenomena, which is similarly difficult to model or to learn.

Learning from demonstrations (LfD) is a promising framework for
learning various robotic tasks efficiently without using hard-coded
approaches or physical insights to model the specific aspects of each
task. It has been used in many different robot scenarios to great
effect, including robot manipulation and human-robot collaboration
[13]. It was also useful in initializing the
parameters of policy-search RL approaches for robot
learning [9]. There are, by now, many different frameworks
for LfD, including dynamical system representations such as the
Dynamical Movement Primitives (DMP) [5], learning
control Lyapunov-functions [7], and various other
probabilistic approaches, such as the probabilistic movement
primitives [19] or Gaussian mixture
models [6]. These last two methods can, unlike
DMPs, capture multiple demonstrations in a parametric form, and can
moreover be used to condition on way-points or different targets in
joint or in task space. One particular disadvantage of all of the LfD
approaches introduced above is that the features chosen to regress on
the demonstration data are often manually tuned and the number of
parameters to learn are explicitly specified. We think that fixing the
features and tuning their hyperparameters for particular tasks harm
the generalization and applicability of the movement primitives to
novel scenarios.

The $l_1$-regularized $l_2$-norm regression (from hereon referred to as Lasso) is often used in the statistics and machine learning communities as a regression method that can simultaneously also perform automatic feature selection. A detailed introduction and analysis of Lasso can be found in [3]. Lasso was extended to the multi-task case (i.e., multi-output regression with shared features) in [18]. Our interest in Lasso lies in the fact that (multi-task) Lasso can perform systematic feature selection while training (multiple) movement primitives, augmenting the applicability of LfD to novel tasks. Moreover, selection and early pruning of features can be used to great effect in RL, possibly reducing the amount of interaction time with the real robot.

A new incremental procedure to solve ordinary least squares regression as well as Lasso problems was proposed in [2]. This algorithm, called Least Angle Regression or LARS for short, yields piecewise linear homotopy paths of the regression problem as a function of the $l_1$-regularization term. These paths can be used to rank the features in terms of importance, as will be detailed later. Ranking the features of the trained movement primitives can reduce the curse of dimensionality in RL, decreasing as before the robot interaction time and possibly making the adapted movements also more interpretable to the humans.

The Elastic Net imposing additional $l_2$-regularization to Lasso was introduced in [21], where it was noted that a basic transformation converts the problem to a standard Lasso regression, and this is also valid in the multi-task setting. For the training of movement primitives, especially for dynamic trajectories like the table tennis serves, the Elastic Net with its $l_2$-regularization can help to reduce the excessive accelerations throughout the learned movements, making them safer to implement on the robot.

In the next sections, we will detail how the sparse representation-learning of movement primitives can be formulated using the multi-task Elastic Net, coupled with nonlinear optimization on the feature parameters. To the best of our knowledge, the multi-task Elastic Net was not combined before with Radial Basis Functions in a (iterative) nonlinear feature selection and optimization framework. We also think that ranking the learned parameters in terms of importance is a new idea that can benefit the RL community.

§ NOTATION

The notation that we use throughout the paper is standard: for a robot arm with $n$ degrees of freedom (DoF), the joint configurations are $\joint \in \mathbb{Q} = \{\joint \in \mathbb{R}^n \ | \jointMin \leq \joint \leq \jointMax \}$. The recorded joint positions over a movement is represented as a matrix $\joint(t) \in \mathbb{R}^{\numTimes \times \numMovement}$ of $\numTimes$ rows, with column $i = 1, \ldots, n$ storing the positions throughout the movement corresponding to joint $i$.

Whenever multiple demonstrations are used for learning, i.e., $\joint_{ij}(t)$ is recorded for $i = 1, \ldots, n$ DoF and $j = 1, \ldots, d$ demonstrations, these recordings are stacked to form the $\joints$ matrix. The degrees of freedom are concatenated vertically in this case for a single demonstration, while the columns store the different demonstration data, i.e., $\joint_{ij}(t) \to \joints_{N(i-1) + t/dt, j}$ for a recording of $N$ time points with $dt$ time intervals.

The Frobenius norm of a matrix is the square-root of the sum of its squared elements, $\|\vec{M}\|_{F}^{2}={\sum_i}{\sum_j}m_{ij}^{2}$, whereas the $\|\cdot\|_{21}$ norm used in the multi-task Elastic Net is defined instead as $\|\vec{M}\|_{21}= {\sum_i}\sqrt{{\sum_j} m_{ij}^{2}}$, i.e., $l_2$-norm along the columns (degrees of freedom in our setting) and $l_1$-norm along the rows (time steps). This norm is used to induce sparsity on the features, whose centers are initially located uniformly along the time axis.

§ METHOD

In this section, we discuss how one can acquire a sparse movement pattern from human demonstrations. We present first an algorithm that requires only a single human demonstration, and then present a suitable variant that can be employed for multiple demonstrations. This variant of the algorithm decouples the number of learned parameters from the degrees of freedom of the robot.

§.§ Learning a sparse representation from a single demonstration

Given a single demonstration $\joint(\timePoints)$ at the (observed) time points $\timePoints$, we'd like to extract a movement primitive that is sparse. That is, throughout the parametric optimization, we'd like to impose a good fit with as few basis functions as possible, while keeping the accelerations low during the trained movement pattern. Having low accelerations is beneficial both for robot safety as well as improving the tracking (execution) accuracy of the trajectories [12]. Mathematically, the criterion that we optimize can be written as

\begin{equation}
\label{eq:1}
\min_{\paramsRbf,\paramsRegr}\|\joint(\timePoints) - \basis(\timePoints,\paramsRbf)\paramsRegr\|_F^{2} + \lone\|\paramsRegr\|_{21} + \ltwo\|\basisAcc(\timePoints,\paramsRbf)\paramsRegr\|_F^2,
\end{equation}

where $\basis(\timePoints,\paramsRbf) \in \mathbb{R}^{N \times p}$ are the evaluations of the basis functions at $\timePoints$, $\paramsRegr \in \mathbb{R}^{p \times n}$ are the (sparse) regression parameters, and $\joint(\timePoints)$ are the joint observations during the shown movement. The nonlinear radial basis functions (RBF) are parameterized by $\paramsRbf \in \mathbb{R}^{p}$.  An $l_2$-penalty is put on the accelerations $\basisAcc(\timePoints,\paramsRbf)\paramsRegr$ of the extracted movement pattern $\basis(\timePoints,\paramsRbf)\paramsRegr$, while a penalty with the $l_1$-norm on the (rows of the) regression parameters $\paramsRegr$ encourages sparsity of the found solutions.

This regression problem, for fixed $\paramsRbf$, is known as the multi-task Elastic Net in the literature, where the features are shared among the sparse parameters along each degree of freedom. As opposed to the standard (multi-task) Lasso, the $l_2$-norm penalty in the optimization (<ref>) penalizing the accelerations throughout the motion, also adds stability to the Lasso solutions [21].

The solution to the weighted Elastic Net problem (<ref>) for fixed $\paramsRbf$ can be obtained by transforming the problem to an equivalent (unweighted) Lasso problem, solving it via a convex optimizer (e.g., coordinate descent is very effective for Lasso problems), and then transforming the solutions back to the Elastic Net parameters.

We can solve the original problem (<ref>) iteratively (as in Expectation-Maximization type of algorithms) by first starting the iteration with a Lasso solution of an overly-parameterized radial basis function regression. At each iteration, the RBF parameters $\paramsRbf_i$ corresponding to the basis functions with nonzero Lasso regression parameters $\paramsRegr_{ij} > 0, \, j \!=\! 1, \ldots, n$ are updated for each $\, i \!=\! 1, \ldots, p$ via nonlinear optimization. The Elastic Net regression is then performed, and the features corresponding to parameters with zero coefficients are removed. These two alternating steps can be continued till convergence, or rather terminated in a fixed number of steps. The iterations converge when the change in function value of the total cost in (<ref>) is below a certain tolerance $\epsilon$. Depending on the initial solution parameters $\paramsRbf_0$ and $\paramsRegr_0$, the iteration converges to a local minimum.

The full procedure is shown in Algorithm <ref> in detail. We call the resulting algorithm Learning Sparse Demonstration Parameters or LSDP for short. The algorithm alternates between the multi-task Elastic Net (lines $4$ and $10$) and the nonlinear optimizer (BFGS, in line $8$). In between, the zero entries of the regression parameters $\paramsRegr$ and the corresponding columns of $\basis, \basisAcc$ are removed in the Prune step (lines $5$ and $13$). The pruning operation simplifies the optimization in the upcoming iterations, as the removed RBF parameters cannot then be re-elected later. We use the squared exponential kernel to construct our basis functions, i.e., for every $i,j$ we use

\begin{equation}
\label{eq:2}
\basis_{ij}(t_i) = \exp(-(t_i - \centersRbf_j)^2/(2\widthsRbf_j)) ,
\end{equation}

to form the $(i,j)$'th element of the matrix $\basis$. The data is initially centered (line $2$), i.e., the mean of each joint recording is subtracted from the signal, and the means $\joint_0$ are stored as the intercepts for the particular demonstration.

For a good performance of the algorithm, i.e., obtaining low residuals with a sparse set and low accelerations, choosing the regularizer weights $\lone$ and $\ltwo$ suitably is crucial. These parameters can be set using cross-validation either before Algorithm <ref> or together with the initial regression (line $4$). The regularizers should be scaled down accordingly with the decreasing residual norms (see line $12$), otherwise the algorithm can converge to the empty set for the parameters $\paramsRegr$.

The optimization problem, depending on the parameterization and the features used, can be highly nonconvex, possibly with many local minima. The number of local minima, fortunately, does not seem to pose a problem in terms of residual norm. As long as the initial representation is sufficiently (over) parameterized, most solutions fit well to the demonstration data. For more sparse representations, however, one may choose to restart the training procedure a few times from perturbed initial conditions, especially for the RBF parameters $\paramsRbf$. See the Experiments section for more discussion on the implementation details.

The computational complexity of the algorithm overall is dominated by the complexity of the multi-task Elastic Net step (line $10$), where the coordinate descent algorithm is used to solve a Lasso problem (after a transformation in constant time $\mathcal{O}(1)$). The time-complexity of the LARS algorithm to solve Lasso problems is known to be $\mathcal{O}(Np^{2})$ [2], but coordinate-descent converges often faster, in our experience. One step of Quasi-Newton methods has time-complexity  $\mathcal{O}(p^2)$ (plus the cost for function and gradient evaluations [16]), coming from the matrix multiplication operations. Quasi-Newton optimization may, depending on the initialization, require many of these steps, in our case we limit it to $100p$ steps for each iteration of $\alg$.

Learning sparse parameters with regression ($\alg$) for a single demonstration

[1]
$\joint$, $\timePoints$, $\centersRbf$, $\widthsRbf$, $\lone$, $\ltwo$, $\epsilon > 0$
Initialize $\paramsRbf_0 = [\centersRbf,\widthsRbf]$
Center the data, $\joint_0, \joint \leftarrow \code{Center}(\joint)$
Form $\basis, \basisAcc$ using $\paramsRbf_0$ and $\timePoints$
$\paramsRegr_0$ $\leftarrow$ MultiTaskElasticNet($\basis$, $\basisAcc$, $\joint$, $\lone$, $\ltwo$)
$\paramsRegr_{0}, \paramsRbf_{0}$ $\leftarrow$   Prune($\paramsRegr_{0}, \paramsRbf_{0}$)
Form $\basis, \basisAcc$ using $\paramsRbf$ and $\timePoints$
$\ k = 1, \ldots, $
$\paramsRbf_{k}$ $\leftarrow$ BFGS($\basis$, $\basisAcc$, $\paramsRbf_{k-1}$, $\paramsRegr_{k-1}$, $\joint$, $\lone$, $\ltwo$)
Form $\basis,\basisAcc$ using $\paramsRbf_{k}$ and $\timePoints$
$\paramsRegr_{k}$ $\leftarrow$ MultiTaskElasticNet($\basis$, $\basisAcc$, $\joint$, $\lone$, $\ltwo$)
Calculate residual norm $r_k$, total cost $f_k$ using (<ref>)
Scale penalties $\lambda_{i} \leftarrow \lambda_i r_{k}^{2}/r_{k-1}^{2}$, $i = 1,2$
$\paramsRegr_{k}, \paramsRbf_{k}$ $\leftarrow$ Prune($\paramsRegr_{k}, \paramsRbf_{k}$)
Form $\basis, \basisAcc$ using $\paramsRbf$ and $\timePoints$
$\|f_k - f_{k-1}\| < \epsilon$

§.§ Coupling the parameters across dimensions

The algorithm $\alg$ discussed in the previous subsection uses the multi-task Elastic Net to enforce the same basis functions for each degree of freedom (along the columns of $\joint(t)$ and the parameter matrix $\paramsRegr$), while the parameter vectors corresponding to each joint movement are different and optimized independently: the regression parameters are decoupled across the degrees of freedom (DoF) of the robot. In particular, the number of regression parameters grow linearly with the robot DoF, which may be undesirable for applying policy search RL approaches to high dimensional robotic systems especially.

Furthermore, the algorithm has to be applied for each demonstration separately, i.e., there is no coupling or information shared between the demonstrations. In order to enforce rather the features to be shared across demonstrations rather than the robot DoFs, we discuss here a variant of the algorithm $\alg$, which we call coupled $\alg$, or $\algCoupled$ for short.

The algorithm $\algCoupled$, shown in Algorithm <ref>, requires only a few changes compared to Algorithm <ref>. The data is centered for each demonstration to obtain the intercepts $\joints_0$. The algorithm then stacks (lines $1-3$) the dependent regression variables $\joint_i$ and the RBF parameters $\paramsRbf_i$ vertically for each degree of freedom $i = 1, \ldots, n$ to form the matrices $\joints \in \mathbb{R}^{\numTimes\numMovement \times \numDemos}$ and $\basis \in \mathbb{R}^{\numTimes\numMovement \times \numRbf}$. The second time derivative of the data matrix, $\basisAcc$, is stacked as well to form the regression model as in (<ref>).

As opposed to $\alg$, in this procedure there are $\numMovement$ times
the number of RBF parameters $\paramsRbf$ to be optimized (line $8$),
as the features are adapted independently for each DoF. The regression
parameters $\paramsRegr$, on the other hand, are coupled across the
DoFs, and their cardinality is reduced by $\numMovement$ times. The
nonlinear optimization computational complexity in this case dominates
that of the multi-task Elastic Net and the net result is roughly a
$\numMovement$ times increase in the computation time between each
iteration of $\algCoupled$.

Note that the parameters for each demonstration are estimated
together, i.e., the columns of the $\paramsRegr$ matrix correspond to
the regression parameters for different demonstrations. One way to
generalize the learned movement primitives to different task
conditions (such as varying initial joint states) would be to
interpolate between these regression parameters. A policy could then be effectively created, whose generalization would be limited by the number and the quality (e.g. variety, success rate) of the demonstrations.

§.§ Ranking the demonstration parameters

The regression parameters estimated with $\algCoupled$ can also be ranked in terms of statistical significance, i.e., correlation. The Elastic Net regularization path of the LARS algorithm [2] traces the evolution of the parameters as the $l_1$-penalty weight $\lambda_1$ of equation (<ref>) increases. An example regularization path for twenty selected regression parameters $\paramsRegr \in \mathbb{R}^{20}$ are plotted in Figure <ref>. Initially when the regularization is low ($\lambda_1 \approx 0$) on the right side of the Figure, the coefficients are close to their (nonzero) values in ordinary Least Squares. As the regularization term increases, some of these terms drop out, i.e., the coefficients become zero as the path is traced towards the left-hand side of the Figure. The corresponding features can then be eliminated from the regression model, leading not only to a sparse, but also a ranked set of features.

In the proposed method $\algCoupled$, the LARS algorithm
instead of coordinate descent can be used in the final Elastic Net
computation step (line $11$ of Algorithm <ref>) to generate the
full regularization path. The addition of the selected movement
primitive parameters can then be traced. An example path for twenty
parameters selected by the Algorithm is plotted in
Figure <ref> against their normalized $l_1$-norm.
These parameters can be ranked according to their evolution, i.e., the
coefficients that early on during the path become nonzero are likely
to signal more causally effective components of the motion. For
example, in the shown plot, the parameters corresponding to the red
lines would be ranked after some of the parameters appearing before
(black lines). More prominent components of the motion can be
identified this way. These movement components could be adapted
earlier with RL strategies, reducing the curse of dimensionality in
high dimensional robot learning problems.

Learning coupled sparse parameters with regression ($\algCoupled$) across multiple demonstrations

[1]
$\joint_{ij}$, $\timePoints$, $\centersRbf_i$, $\widthsRbf_i$, $\lone$, $\ltwo$, $\epsilon > 0$
Stack $\joint_{ij}$ to form $\joints$, $i \in [1,\numMovement], j \in [1,\numDemos]$
Center the data, $\joints_0, \joints \leftarrow \code{CenterStacked}(\joints)$
Stack $\paramsRbf_0 = [\centersRbf_1,\ldots,\centersRbf_{\numMovement},\widthsRbf_1,\ldots,\widthsRbf_{\numMovement}]$
Stack $\basis, \basisAcc$ using $\paramsRbf_0$ and $\timePoints$ across DoFs
$\paramsRegr_0$ $\leftarrow$ MultiTaskElasticNet($\basis$, $\basisAcc$, $\joints$, $\lone$, $\ltwo$)
$\paramsRegr_{0}, \paramsRbf_{0}$ $\leftarrow$   PruneStacked($\paramsRegr_{0}, \paramsRbf_{0}$)
Stack $\basis, \basisAcc$ using $\paramsRbf$ and $\timePoints$ across DoFs
$\ k = 1, \ldots, $
$\paramsRbf_{k}$ $\leftarrow$ BFGS($\basis$, $\basisAcc$, $\paramsRbf_{k-1}$, $\paramsRegr_{k-1}$, $\joints$, $\lone$, $\ltwo$)
Stack $\basis,\basisAcc$ using $\paramsRbf_{k}$ and $\timePoints$ across DoFs
$\paramsRegr_{k}$ $\leftarrow$ MultiTaskElasticNet($\basis$, $\basisAcc$, $\joints$, $\lone$, $\ltwo$)
Calculate residual norm $r_k$, total cost $f_k$ using (<ref>)
Scale penalties $\lambda_{i} \leftarrow \lambda_i r_{k}^{2}/r_{k-1}^{2}$, $i = 1,2$
$\paramsRegr_{k}, \paramsRbf_{k}$ $\leftarrow$ PruneStacked($\paramsRegr_{k}, \paramsRbf_{k}$)
Stack $\basis, \basisAcc$ using $\paramsRbf$ and $\timePoints$ across DoFs
$\|f_k - f_{k-1}\| < \epsilon$

An example Elastic Net path with twenty selected parameters is shown after training Algorithm <ref>, $\algCoupled$, with five demonstrations. This regularization path can be generated in the final step of the algorithm. As the $l_1$-penalty term $\lambda_1$ of the regression problem (<ref>) is reduced, the coefficients converge to their (maximal) ordinary least squares values at the right hand side of the plot (not shown). Each dashed line signals an entry of a parameter, and the slope of the coefficients are updated accordingly. The algorithm LARS [2] can be used to generate these piece-wise linear regularization paths. One possible way to use this path is to rank the sparse parameters of the learned movement primitives in terms of statistical importance. For example, in the shown plot, the parameters corresponding to the red lines would be ranked after the other parameters appearing before (black lines). The parameter paths, whose coefficients become nonzero close to each other, are drawn with the same color.

§ EXPERIMENTS

In this section, we conduct experiments to learn a sparse set of movement primitive parameters using the proposed approaches (see Algorithms <ref> and <ref>). The two algorithms are also compared against two competing movement primitive learning methods (DMPs and $l_2$-regularized regression). Finally we present real robot experiments on our table tennis platform where we show that the learned sparse movements nevertheless look similar to the shown demonstrations in style. They can also be implemented safely on the robot.

§.§ Learning from Demonstrations

The algorithms $\alg$ and its coupled variant $\algCoupled$, discussed in Section <ref>, are applied here on the demonstrated Barrett WAM serve movements, see Figure <ref>. From a continuous stream of joint values, recorded at $500$ Hz during a kinesthetic teach-in session, a predetermined number of $d$ movements are selected by detecting the maximum $d$ velocities in joint space and windowing around these points for a fixed duration of one second. We implement the preprocessing as well as the Algorithms in Python, using the scikit-learn toolbox for the multi-task Elastic Net and the scipy toolbox for the nonlinear optimization (BFGS, see lines $8$ and $9$ in the Algorithms, respectively).

The preprocessed examples using the above procedure result in the joint matrix $\joint(t) \in \mathbb{R}^{500 \times 7}$ for each example demonstration. For the algorithm $\alg$, the initial RBF centers $\centersRbf_0 \in \mathbb{R}^{500}$ are placed at every time point and the RBF widths $\widthsRbf_0$ are set uniformly to $0.1$. The algorithm stretches, prunes and expands the basis functions throughout the optimization to produce a very sparse, nonuniform set of basis functions shared across the seven degrees of freedom (DoF). The columns of the regression parameter matrix $\paramsRegr$, on the other hand, are separate for each DoF.

The Algorithm $\algCoupled$, on the other hand, optimizes $\numMovement$ times more RBF parameters, i.e., $\centersRbf \in \mathbb{R}^{3500}$ and $\widthsRbf \in \mathbb{R}^{3500}$ for the Barrett WAM with $\numMovement = 7$. During the optimization, all of the recorded data from $d$ demonstrations are used together, and the same set of basis function parameters $\paramsRbf = [\centersRbf^{\mathrm{T}}, (\widthsRbf)^{\mathrm{T}}]^{\mathrm{T}}$ are learned across multiple demonstrations. The learned parameters $\centersRbf,
\widthsRbf, \paramsRegr$, along with the intercepts, are saved after the optimizations to a json file, to be loaded later by the real-time robot controller in $\CC$ during the online experiments.

Two movement primitives learned by the proposed algorithm $\algCoupled$, are plotted in joint space against the recorded demonstrations. The table tennis serve movements, shown in blue, after preprocessing and segmenting the recorded time series are one second long each. The first three rows, $q_1$ through $q_3$, correspond to the shoulder movement in joint space. The fourth row $q_4$ shows the movement of the elbow. Finally, the last three rows ($q_5$ through $q_7$) show the wrist movements in joint space. The trained movement primitives, shown in orange, couple the sparse regression parameters across the degrees of freedom of the robot.

Table <ref> summarizes the results of learning movement primitives from five different demonstrations. The three columns used to compare the different approaches show on average the number of features selected (equivalently, the number of regression parameters with nonzero coefficients), the norm of the second derivatives of the trained movement primitives and the norm of the residuals, respectively. The five demonstration parameters are estimated together in $\algCoupled$, whereas $\alg$ is run separately for each demonstration to obtain the mean and the standard deviations reported in the table. Note that the number of parameters in total used by $\algCoupled \ (37)$ is much lower than the on-average $16.8$ parameters used by $\alg$ for each robot DoF. The residual is slightly higher, this is a result of the parameters being shared across the dimensions. In particular, we have observed that $\algCoupled$ does not fit the last three joints, corresponding to the Barrett WAM wrist, as tightly as $\alg$. This could be because the motion of the wrist is highly varying across the movements and the coupling of the features induced by the algorithm across demonstrations brings these movements closer.

The two proposed algorithms are compared against two baselines in Table <ref>. The first baseline is the Dynamic Movement Primitives (DMPs) with a fixed number of basis functions. DMPs learn the parameters of an attractor dynamics, i.e., a set of differential equations that converge to a suitably chosen goal state [5]. A standard regression is performed on the estimated attractor dynamics accelerations. The second baseline is $l_2$-penalized standard regression, with the penalty on the accelerations. During the experiments we used a total of ten basis functions both for the DMPs and for the $l_2$-penalized regression. The basis functions are spread uniformly, as discussed before for the proposed algorithms, around the one second long (preprocessed) demonstrations.

DMPs, as a result of the dynamic constraint of reaching a desired goal position, can incur very high initial accelerations in joint space. Even if the hyperparameters are optimized accordingly to prevent such high accelerations, slight adjustments of initial joint positions can again give rise to high accelerations. The suggestion proposed in [8] to modify the accelerations with the phase can reduce the initial accelerations, but then we have found that the convergence to the goal positions can suffer drastically. The fixed basis function regression does not have this problem, but as in DMPs, optimizes a fixed number of parameters. As shown in Table <ref>, the number of parameters to fit the demonstrations well is, for both compared methods, on average double the number optimized by $\algCoupled$.

See Figure <ref> for two example regression results. The demonstrated movements are shown in blue and the regression results are shown in orange. The first three rows, $q_1$ through $q_3$, correspond to the shoulder movement in joint space. The fourth row $q_4$ shows the movement of the elbow. Finally, the last three rows ($q_5$ through $q_7$) show the wrist movements in joint space. Although the demonstrated movements are quite different, the training with the sparse set of features can still capture them well.

Comparison of different learning from demonstrations approaches, averaged over five different serve demonstrations

No. par.  $(\|\paramsRegr\|_{0})$     Acc. norm     Res. norm

$\alg$     $(16.8 \pm 3.25) \times 7$     $59.04 \pm 7.0$     $0.59 \pm 0.11$

$\algCoupled$     $37$     $55.98 \pm 11.78$     $0.73 \pm 0.09$

DMPs     $11 \times 7$     $621.73 \pm 57.45$     $0.92 \pm 0.06$

$l2$-reg. regr.     $11 \times 7$     $215.45 \pm 35.25$     $2.12 \pm 0.47$

Three example demonstrations in task space. The initial position of the racket center and the ball in the egg-holder are marked as $0$ in red and blue circles, respectively. The egg-holder is located approximately $14$ cm away from the racket centre. Before the racket stops moving, the ball is already hit, flying towards the table.

Three example demonstrations are plotted in task space in Figure <ref> along with the recorded ball positions, detected and triangulated from two cameras opposite to the robot. The initial positions of the racket center and the ball in the egg-holder are marked as $0$ in red and blue, respectively. The egg-holder is at a distance of roughly $14$ cm to the racket center. During the movement the ball is hit by the human demonstrator moving the robot arm, and as the demonstrator slows down the motion to a halt, the ball is seen flying towards the table.

§.§ Robot Experiments

Finally, we conduct experiments in our real robot table tennis platform, see Figure <ref>. Our table tennis playing robot is a seven degree of freedom Barrett WAM arm that is capable of reaching high accelerations and velocities. However it is cable-driven and high accelerations can cause the cables to break easily. A standard size racket is attached to the end-effector via a metal bar. The racket has a radius of roughly $\racketRadius = 7.6$ cm. The table and the table tennis balls are standard sized, balls have a radius of $2$ cm, and the table geometry is roughly $276 \times 152 \times 76$ cm. Throughout the experiments, the Barrett WAM is placed at a distance of about one meter to the end of the table and its base is located $95$ cm above the table. This makes it difficult (but not impossible) for the robot to hit the table. An egg-holder holds the table tennis ball initially, wrapped around the metal bar connecting the end-effector and the racket, see Figure <ref>.

A successful rollout during real robot experiments. The ball is initially on top of the egg-holder and during the movement, as a result of the acceleration of the arm, it takes-off from the robot, to be later hit by the racket towards the table. The arm then decelerates towards a safe resting posture.

A successful serve in our robot platform is shown in Figure <ref>. The ball is initially placed on top of the egg-holder (approximately $14$ cm away from the racket center along the racket plane). The movements captured by the algorithm $\algCoupled$ are then executed on the robot. During the movements, as a result of the robot's accelerating motion, the ball takes off from the robot arm. The ball is then hit by the robot towards the table. The arm then decelerates towards a resting posture as the ball lands on the robot court, passes the net, and lands again on the opposite side. We notice that the initial accelerating motion and the final wrist movement are critical for a good serve. Without the initial accelerations, the ball has no chance to take-off, and without the final wrist movement (e.g., a quick rotation towards the ball) the ball is not hit well towards the table.

A video showing some demonstrated movements, as well as several actual rollouts on the Barrett WAM is available online: <https://youtu.be/vj6jfX_MQmQ>. Note that orchestrating the right movement during the demonstrations can be quite difficult, as moving the shoulder and the elbow joints can feel very awkward depending on the posture. When a good initial posture (both for the robot and the demonstrator) is found, the resulting demonstrations have a higher quality in general. These higher quality demonstrations also have a higher chance of being executed successfully.

Comparing our approach to the DMPs, we notice that the DMPs
immediately start the movement with very large accelerations, these
can be dangerous for the robot and the low-level Barrett WAM
controllers do not support $80 \%$ of the movements. DMPs are good at capturing movements that converge to goal positions (with zero or low velocities), however they are less accurate in capturing the style (e.g., initial movement, final wrist turns) of dynamics movements such as table tennis serves, without manual tuning (the number of basis functions, locations and widths of the basis functions, etc.) for each task. We have seen that $\algCoupled$[Note that executing the Algorithm $\alg$, trained on each demonstation independently, shows a very similar performance, to that of Algorithm $\algCoupled$ at the moment. However we expect improvements on the learning performance, if Reinforcement Learning is applied on top of the more sparse set of $\algCoupled$ parameters.]
on the contrary, can capture the style of the movement, as shown in Figure $3$ for some of the movements, with a sparse set of basis functions. The generalization capacity of these selected basis functions hinges on the quality and the number of the shown demonstrations.[If the number and the quality of the demonstrations is not enough, then the selected features and their ranking (using the regularization path) may be spurious, i.e., without any meaningful physical relevance. Increasing the number and the quality (e.g., increased variety of movements, higher success rates) of the movements could remedy such a limitation.]

§ CONCLUSION

In this paper we presented a new learning from demonstrations (LfD) approach to represent and learn table tennis serve movements. The proposed algorithms $\alg$ and $\algCoupled$ learn sparse parameters of the radial basis functions (RBF) from single and multiple demonstrations, respectively. The algorithms employ iterative optimization, alternating between a weighted multi-task Elastic Net regression step that learns sparse parameters given the features and a nonlinear optimization step that adapts the features (more specifically, the widths and centers of the RBFs corresponding to the nonzero regression parameters). The algorithm $\algCoupled$, unlike $\alg$, learns (sparse) parameters that are independent of the robot DoF. This desirable property is achieved by having different basis functions that are adapted across each DoF separately. The multi-task Elastic Net, in this case, forces the joint-dependent features to be shared across multiple demonstrations.

The cost function chosen for the optimization includes the residual of the fit, as well as $l_2$-regularization terms on the accelerations and $l_1$-regularization on the regression coefficients. We compared the performance of the proposed algorithms with Dynamic Movement Primitives (DMPs) and the standard $l_2$-regularized regression, and we evaluated the performance of each on the different components of the chosen cost function (see Table <ref>). Finally, we discussed the performance of the actual rollouts, using our framework, on the real robot table tennis setup. One can see in the video available online that the style of the movements are preserved while maintaining low accelerations throughout the motion, which is important for the safety of the robot.

The sparsity of the parameters, as well as their decoupling from the robot DoF, is a desirable property for policy-search RL approaches that could adapt the regression parameters online based on a suitable reward function. We have presented a way to rank these policy parameters, in the last subsection of Section <ref>, based on how well the parameters explain the (multiple) demonstration recordings. We think that this is a promising research direction to combat the curse of dimensionality in high dimensional robot learning tasks, and we will focus on it more in future experiments.

[1]
Russell L. Anderson.
A Robot Ping-pong Player: Experiment in Real-time Intelligent
MIT Press, Cambridge, MA, USA, 1988.

[2]
Bradley Efron, Trevor Hastie, Iain Johnstone, and Robert Tibshirani.
Least angle regression.
Annals of Statistics, 32:407–499, 2004.

[3]
Trevor Hastie, Robert Tibshirani, and Jerome Friedman.
The Elements of Statistical Learning.
Springer Series in Statistics. Springer New York Inc., New York, NY,
USA, 2001.

[4]
Y. Huang, D. Büchler, O. Koç, B. Schölkopf, and J. Peters.
Jointly learning trajectory generation and hitting point prediction
in robot table tennis.
In 2016 IEEE-RAS 16th International Conference on Humanoid
Robots (Humanoids), pages 650–655, Nov 2016.

[5]
Auke Jan Ijspeert, Jun Nakanishi, Heiko Hoffmann, Peter Pastor, and Stefan
Dynamical movement primitives: Learning attractor models for motor
Neural Comput., 25(2):328–373, February 2013.

[6]
S. M. Khansari-Zadeh and A. Billard.
Learning stable nonlinear dynamical systems with gaussian mixture
IEEE Transactions on Robotics, 27(5):943–957, Oct 2011.

[7]
Seyed Mohammad Khansari-Zadeh and Aude Billard.
Learning control lyapunov function to ensure stability of dynamical
system-based robot reaching motions.
Robotics and Autonomous Systems, 62:752–765, 2014.

[8]
J. Kober, K. Muelling, O. Kroemer, C.H. Lampert, B. Schoelkopf, and J. Peters.
Movement templates for learning of hitting and batting.
In IEEE International Conference on Robotics and Automation
(ICRA), 2010.

[9]
J. Kober and J. Peters.
Policy search for motor primitives in robotics.
In Advances in Neural Information Processing Systems 22 (NIPS
2008), Cambridge, MA: MIT Press, 2009.

[10]
O. Koc, G. Maeda, G. Neumann, and J. Peters.
Optimizing robot striking movement primitives with iterative learning
In 15th IEEE-RAS International Conference on Humanoid Robots,

[11]
O. Koc, G. Maeda, and J. Peters.
A new trajectory generation framework in robotic table tennis.
In 2016 IEEE/RSJ International Conference on Intelligent Robots
and Systems, in press.

[12]
Okan Koc, Guilherme Maeda, and Jan Peters.
Online optimal trajectory generation for robot table tennis.
Robotics and Autonomous Systems, 105:121 – 137, 2018.

[13]
Guilherme J Maeda, Gerhard Neumann, Marco Ewerton, Rudolf Lioutikov, Oliver
Kroemer, and Jan Peters.
Probabilistic movement primitives for coordination of multiple
human–robot collaborative tasks.
Autonomous Robots, 41(3):593–612, 2017.

[14]
K. Muelling, J. Kober, O. Kroemer, and J. Peters.
Learning to select and generalize striking movements in robot table
International Journal of Robotics Research, 3:263–279, 2013.

[15]
K. Muelling, J. Kober, and J. Peters.
A biomimetic approach to robot table tennis.
Adaptive Behavior Journal, (5), 2011.

[16]
Jorge Nocedal and Stephen J. Wright.
Numerical Optimization.
Springer-Verlag, New York, 1999.

[17]
J. Nonomura, A. Nakashima, and Y. Hayakawa.
Analysis of effects of rebounds and aerodynamics for trajectory of
table tennis ball.
In SICE Annual Conference 2010, Proceedings of, pages
1567–1572, Aug 2010.

[18]
Guillaume Obozinski and Ben Taskar.
Multi-task feature selection.
In In the workshop of structural Knowledge Transfer for Machine
Learning in the 23rd International Conference on Machine Learning (ICML
2006). Citeseer, 2006.

[19]
Alexandros Paraschos, Christian Daniel, Jan R Peters, and Gerhard Neumann.
Probabilistic movement primitives.
In C. J. C. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K. Q.
Weinberger, editors, Advances in Neural Information Processing Systems
26, pages 2616–2624. Curran Associates, Inc., 2013.

[20]
M. Ramanantsoa and A. Durey.
Towards a stroke construction model.
Int. Journal of Table Tennis Science, 2:97–114, 1994.

[21]
Hui Zou and Trevor Hastie.
Regularization and variable selection via the elastic net.
Journal of the Royal Statistical Society, Series B,
67:301–320, 2005.