[1]Cassandra Czobit

[1]Electrical, Computer and Biomedical Engineering, Toronto Metropolitan
University, Toronto, Ontario, Canada

2]Vector Institute for Artificial Intelligence, Toronto, Ontario, Canada

# CycleGAN Models for MRI Image Translation

<EMAIL_ADDRESS>Reza Samavi<EMAIL_ADDRESS>*  [

###### Abstract

Image-to-image translation has gained popularity in the medical field to
transform images from one domain to another. Medical image synthesis via
domain transformation is advantageous in its ability to augment an image
dataset where images for a given class is limited. From the learning
perspective, this process contributes to data-oriented robustness of the model
by inherently broadening the model’s exposure to more diverse visual data and
enabling it to learn more generalized features. In the case of generating
additional neuroimages, it is advantageous to obtain unidentifiable medical
data and augment smaller annotated datasets. This study proposes the
development of a CycleGAN model for translating neuroimages from one field
strength to another (e.g., 3 Tesla to 1.5). This model was compared to a model
based on DCGAN architecture. CycleGAN was able to generate the synthetic and
reconstructed images with reasonable accuracy. The mapping function from the
source (3 Tesla) to target domain (1.5 Tesla) performed optimally with an
average PSNR value of 25.69 $\pm$ 2.49 dB and an MAE value of 2106.27 $\pm$
1218.37.

###### keywords:

CycleGAN; neuroimaging; DCGAN; image-to-image translation; DTI

##  1 Introduction

The field of image-to-image translation is increasingly prominent in enhancing
the data-oriented robustness of machine learning methods, particularly in the
realm of medical imaging where it facilitates the translation of images across
different domains. This technique proves invaluable in medical image synthesis
through domain conversion, especially useful when there is a scarcity of
images for certain classes, which also aids in safeguarding patient privacy.
From the learning perspective, such translation processes improves model
robustness by data augmentation that results in a model’s exposure to a wider
array of visual data. Thus, the approach helps in fostering the development of
more generalized feature recognition capability. From a medical imaging
perspective, the technique is specifically instrumental in mild traumatic
brain injuries (mTBIs), where it routinely becomes difficult to diagnose due
to a lack in quantitative assessments [1].

Magnetic resonance imaging (MRI) is known universally as a form of technology
to obtain images for disease diagnosis and prognosis, in a non-invasive manner
[21]. Diffusion tensor imaging (DTI) has become a promising diagnostic tool as
an MRI-based technology to evaluate the organization of white matter in
neuroimaging [1]. These tools, with machine learning, can be used to
characterize, monitor and generate predictions on disease progression, while
increasing efficacy and efficiency in patient care [11].

The optimal field strength for assessing DTI images has been frequently
debated. The magnetic field strength of DTI neuroimages range from 0.2-0.5
Tesla (T) for low field images to 7T and above for high field imaging [21]. A
field strength of 1.5T or 3T are predominantly selected for clinical MRI
procedures, due to the high signal-to-noise ratio (SNR) and resolution
[4],[16]. Higher field strengths have traditionally been viewed as
advantageous due to an increase in image contrast and resolution [2],[10].
However, DTI at a field strength of 0.5T is considered advantageous for
assessing head trauma, due to the reduction of geometric distortions and
susceptibility artifacts [4],[18].

Generative Adversarial Network (GAN) models have gained an increasing amount
of popularity for medical image generation, reconstruction, and classification
[20], [12]. GAN models, such as Deep Convolutional GAN, are primarily used as
a method for augmenting datasets. Composed of a generator and discriminator,
the generator creates synthetic images from the probability distribution,
whereas the discriminator classifies samples as real or fake [7]. The
generator obtains an input dimension $z$ from the Gaussian distribution to
create the synthetic samples, $G(z)$, and maps the pseudo-sample distribution
[14]. The ideal discriminator model will maximize the probability of
allocating the correct label to the sample [7]. Translation GAN models, such
as CycleGAN, function similarly by using two GAN networks to generate target
images from a source image. The creation of realistic synthetic medical images
offers a solution to alleviate privacy concerns relating to diagnostic imaging
and usage of individuals’ medical data [20].

The primary purpose of this investigation is to demonstrate a proof-of-concept
for translating neuroimages between two field strengths in the MRI image
domain. We assess two models for data transformation using two field strength
and data augmentation for images in the same field strength. We develop a
CycleGAN model for data transformation from a source domain to the target
domain (i.e., 3T to 1.5T). Our second model aims to augment the 3T and 1.5T
datasets by generating synthetic images from the same field. Our empirical
evaluation shows that the CycleGAN model is able to translate images within
the same modality, whereas the DCGAN model produces synthetic images of poor
quality.

##  2 Related Work

GANs were proposed as a mechanism to generate images that approximate the data
distribution of the input functions [7]. Thus, GANs have emerged as a powerful
tool for generating synthetic data, which can be particularly useful for data
augmentation in DNNs training. The usage of GANs for this purpose has been
explored in various research studies and applications. For example, [3]
investigated GAN-based imputation models to address data imbalance and metrics
to evaluate such models in contrast to the statistical imputation models.

In image-to-image translation, adaptations of the original GAN model have
addressed issues surrounding convergence of the models and mode collapse [20].
A deep convolutional GAN model (DCGAN) was created to overcome issues
regarding the training stability by implementing fully convolutional
upsampling/downsampling layers [20], [17].

Paired image translation requires data from both modalities [20]. A study by
Emami et al., explored paired image translation from T1-weighted input MRI
images to generate synthetic CT scans [5]. The conditional GAN model involved
a residual network as the generator and a CNN for the discriminator. A model
by Bahrami et al., constructed a Canonical Correlation Analysis space from
paired 7T and 3T MRI images, where the extracted patches were mapped to a
common space to show greater structural detail in the segmentation of white
and gray brain matter [2]. Unpaired image translation offers more flexibility
from paired images in instances where paired data or an abundance of labelled
data is not available. A CycleGAN model was developed to utilize unpaired
images, without a task specific similarity function, for image translation
[22]. The proposed mapping approach coupled the adversarial loss with inverse
mapping, along with a cycle consistency loss function, to translate the images
from domain X to Y, and vice versa [22]. Hiasa et al., implemented a CycleGAN
model for cross-modality image translation between CT and MRI scans and
adapted the loss function to capture variation of intensity pairs in the
images [9].

The proposed study differs from prior work, in which the input and output
images will be within the same MRI domain, but at varying field strengths.
Previous investigations have translated images across modalities (e.g., MRI to
CT images), but have not addressed the translation of field strengths within a
single modality. This translation aims to visualize the MRI image from a
different anatomical granularity level.

##  3 CycleGAN Image Translation Model

The models discussed in the following sections were developed for the purpose
of image-to-image translation from 3T to 1.5T to generate synthetic
neuroimages. It consists of 2 mapping functions, G and F, with a relationship
defined as $G^{*},F^{*}=\text{arg min max }\mathcal{L}(G,F,D_{X},D_{Y})$ [22].
Section 3.1 details how the data was obtained, section 3.2 describes the
architecture of the CycleGAN model, and section 3.3 describes the DCGAN
network for data augmentation.

###  3.1 Data

Two field strengths, 1.5T and 3T, were selected as part of this study for DTI
image translation. For the purpose of this study, these field strengths were
selected to demonstrate a proof of concept for the CycleGAN and DCGAN models.
DTI images at a field strength of 0.5T are not widely available for public
use. For both datasets, 70% of the images were assigned for training purposes,
with the remaining 30% for testing. The scale of both datasets are small due
to limited open-source resources for the required field strengths.

The 3T images were collected by the University of North Carolina-CH and were
made available on the Kitware Data open-source
platform.111https://data.kitware.com/#collection/591086ee8d777f16d01e0724/folder/58a372fa8d777f0721a64dfa
The scans were acquired on a 3T Siemens MR unit. The DTI images were obtained
using 6 directions with 2mm x 2mm x 2mm isotropic resolution (voxel size). The
35 axial cross-sectional images were obtained from individuals in age groups
from 18-29, 30-39, 40-49, 50-59, and 60+ years. From the 35 patient scans, ten
slice were obtained from each brain volume, for a total of 350 slices.

The 1.5T scans were obtained on the Philips Intera MRI scanner from the SCA2
Diffusion Tensor Imaging dataset on the OpenNeuro
platform.222https://openneuro.org/datasets/ds001378/versions/00003 Diffusion
weighted images were collected for 16 individuals for 2 sessions, a year
apart. The axial images were applied in 15 directions for 50 slices, with a
slice thickness of 3 mm. For each image, ten slices were obtained to represent
the 2-dimensional DTI scan, for a total of 16 images and 160 slices.

###  3.2 CycleGAN

The proposed architecture for the CycleGAN model consists of two sequential
models that operate in a forward and backward cycle for images in the source
and target domains. Each model contains a generator and discriminator
component. The complete CycleGAN network translates an image in source domain
to the target domain, where the source represents the 3T images, and target
contains the 0.5T images.

Within this network, the mapping functions between the two domains are trained
for the two data distributions. Two mapping functions are defined as G: X → Y
and F: Y → X, where X is the training sample in the source domain and Y is the
training sample in the target domain [22]. The adversarial discriminator for
each mapping function contains terms for the adversarial and cycle consistency
loss functions.

In the discriminator model, the CNN model architecture classifies the images
and is identical for each of the mapping functions, G and F. The discriminator
receives images of size 256 x 256 from the generator and source domain as the
input. We use a 70 x 70 PatchGAN method for patch predictions. The PatchGAN
evaluates NxN pixels to classify as real or fake, and averages the score
across the entire image [22]. This approach reduces the number of parameters
from a full-image discriminator [22]. The discriminator consists of 5
convolutional layers, stride 2 and $k$ filter size, where $k$ is equal to the
number of filters per layer. The value of $k$ per layer is 64, 128, 256, 512
and 1, for layers 1 to 5 respectively. We apply a 4x4 filter for filters 1-4,
followed by a LeakyReLU activation function with a slope of 0.2. Instance
normalization is applied to layers 2, 3 and 4.

The generator architecture is constructed as an encoder-decoder network with a
residual network, ResNet, to transform the input images. The encoder portion
of the network downsamples the image, and consists of 3 convolutional layers,
with a ReLU activation function, followed by instance normalization. The
ResNet receives the output of the encoder as the input. ResNet contains 9
convolutional blocks, which functions by creating skip connections between
layers, where a residual block has a layer feeding into another 2-3 skips away
in addition to the layer succeeding it [8]. From the ResNet network, the
output is passed to the decoder, where upsampling is performed using two
deconvolution blocks with fractional stride values to generate the final
output as the original size.

The complete composite model of the generator and discriminator is compiled
for both mapping functions, and trained for 50 epochs, with a batch size of 4.
The Adam optimizer is selected with a learning rate of 0.0002 and a momentum
value of 0.5 [22]. We trained the complete model from scratch.

###  3.3 DCGAN

The DCGAN model consists of a generator and discriminator model for each field
strength. This model generates synthetic images from a singular source domain
for the respective field strength (i.e., a separate model for 3T and 1.5T).
For both DCGAN models, the generator and discriminator architectures are
constructed in the same manner. The DCGAN architecture was adapted from
Radford et al. [17].

The discriminator architecture contains 3 convolutional layers with a 4x4
kernel and stride 2 in each layer for downsampling. For each layer, the
LeakyReLU activation function is used, followed by a dropout rate of 0.4. The
convolutional layers are flattened, before being passed to a Dense layer, with
the sigmoid activation function.

The generator is composed of an initial Dense layer, followed by 3 transpose
convolutional layers. The Dense layer receives a low-resolution version of the
image, which is followed by the LeakyReLU activation function with alpha equal
to 0.2. The transpose convolution layers have filters of 256, 128 and 64
respectively with a 4x4 kernel size. The LeakyReLU activation function with an
alpha value of 0.2 is applied to each layer. The final layer is a
convolutional layer with a 3x3 kernel and uses the sigmoid activation
function.

The complete model is compiled and trained for 50 epochs, with a batch size of
4 and uses the binary cross-entropy loss function. The model is trained with
the Adam optimizer, using a learning rate of 0.0002 and momentum of 0.05.

###  3.4 Evaluation

Several statistical measures will be reviewed to evaluate the performance of
the GAN variant models. Evaluation measures include mean absolute error (MAE),
mean squared error (MSE) and peak signal-to-noise ratio (PSNR) [19],[15]. To
evaluate the image quality, we compared the synthesized 3T and 1.5T
neuroimages to the real, ground truth images.

##  4 Results and Discussion

###  4.1 CycleGAN

The CycleGAN model can be evaluated based on a visual inspection of the
translation process, in addition to its reconstructive capabilities. One
randomly selected ground truth image from the source domain was transformed to
generate an image in the target domain, before being reconstructed back to the
source image. Figure 1 (a) displays the translation process for the 3T image
to 1.5T at 50 epochs, while Figure 1 (b) shows the image translation from 1.5T
to 3T.

(a)

(b)

Figure 1: CycleGAN Generated and Reconstructed Images at 50 Epochs from 3T to
1.5T to 3T (a) and from 1.5T to 3T to 1.5T (b)

From a visual perspective, it appears that both GAN models performed
similarly. We performed a quantitative comparison of the image quality metrics
for both models at 50 epochs. One thousand images were randomly selected to
compute the reconstructive quality between the original, real image and the
reconstructed image. These metrics were averaged for the translation of the
source domain to the target domain, shown in Table 1 of Appendix A. The
average value of the MAE, MSE and PNSR metrics were 2106.27 $\pm$ 1218.37,
0.0.00328 $\pm$ 0.0032, and 25.69 db $\pm$ 2.49, respectively.

In contrast, the backward cycle model translated the target domain images to
the source domain. The average calculated value for MAE was 602.27 $\pm$
147.41 and the MSE value was 0.00189 $\pm$ 0.0013. The PSNR had an average
value of 27.22 db $\pm$ 0.30, as shown in Table 1 of Appendix A.

It is evident that the backward cycle consistency function performed better
than the forward cycle model, as indicated by the higher PSNR value. In
addition, the backward cycle consistency loss function scored a lower value
for the MAE and MSE measures, which indicates a higher model accuracy.

###  4.2 DCGAN

The DCGAN model was evaluated for the 1.5T image synthesis. The synthesized
images are shown in Figure A1 in Appendix A and are evaluated based on a
visual inspection and image quality metrics. Visually, it appears that the
model was unable to generate a diverse set of examples. From 1000 randomly
generated synthetic images, the average PSNR and MAE values were 6.20 db $\pm$
0.11 and 31907.44 $\pm$ 415.85, respectively. The MSE was calculated as 0.24
$\pm$ 0.0059. This quantitative assessment demonstrates that the model was not
able to generate high-quality synthetic images. This type of instability is an
indicator of mode collapse, wherein the generator cannot create a diverse set
of images [6], [13]. As a result, the discriminator only learns from a small
subset of images [13]. Increasing the size of the training set could be
helpful to mitigate low accuracy results [14].

We then evaluated the quality of synthesized 3T scans generated by the DCGAN
model. This was performed for comparison purposes with the backward cycle
consistency loss function of the CycleGAN model. From a qualitative
perspective, the images are able to replicate some details, however, they lack
clarity overall. These images are shown in Figure A2 of Appendix A.
Quantitatively, evaluation metrics were calculated from 1000 synthesized
images. The 3T DCGAN model generated images with a higher PSNR value than the
1.5T counterpart, with an average of 8.86 db $\pm$ 0.76.

###  4.3 Comparison of Models

A comparison of the quantitative evaluation measures demonstrates the
feasibility for using CycleGAN to augment datasets and translate MRI images.
In contrast, the DCGAN model was not able to generate a diverse set of
realistic synthetic images. These findings are evidenced by the higher PSNR
values and lower error values (MAE and MSE) for the CycleGAN models.

###  4.4 Research Limitations

The findings of this investigation are limited by the small dataset for each
domain. As opposed to selecting a range of 10 image slice of the DTI scans, a
future study could input several different sizes of slices from each patient
scan to increase the variability of training images for the CycleGAN and DCGAN
models.

##  5 Conclusions

The purpose of this investigation was to determine if the CycleGAN model could
be applied within a single modality to translate MRI images between two field
strengths. This investigation demonstrated that the application of this model
has potential, barring some modifications to the dataset. In contrast, the
DCGAN model failed to produce realistic synthetic images for both domains.

Future investigations could implement an additional model to review the
quality of the images. The creation of a classification model could play a
role in determining if the synthesized images are able to be correctly
assigned to the desired category [20]. This addition would provide more
concrete evidence for the performance of the CycleGAN model. Increasing the
size of the experiment to include additional datasets and a larger scale of
data would strengthen the current findings as well.

Acknowledgments We would like to thank Dr. Michael Noseworthy and Nicholas
Simard for the initial discussions on current trends and limitations in
medical imaging and motivating this problem.

## References

  *   * Aoki . [2012] Aoki2012DiffusionMeta-AnalysisAoki, Y., Inokuchi, R., Gunshin, M., Yahagi, N. Suwa, H.  20129\.  Diffusion Tensor Imaging Studies of Mild Traumatic Brain Injury: a Meta-Analysis Diffusion Tensor Imaging Studies of Mild Traumatic Brain Injury: a Meta-Analysis.  Journal of Neurology, Neurosurgery & Psychiatry839870–876,  https://doi.org/10.1136/JNNP-2012-302742 https://jnnp.bmj.com/content/83/9/870
  * Bahrami . [2016] Bahrami2016ReconstructionMRIBahrami, K., Shi, F., Zong, X., Shin, H.W., An, H. Shen, D.  20169\.  Reconstruction of 7T-Like Images from 3T MRI Reconstruction of 7T-Like Images from 3T MRI.  IEEE Transactions on Medical Imaging3592085–2097,  https://doi.org/10.1109/TMI.2016.2549918
  * Boursalie . [2021] Boursalie2021EvaluationModelsBoursalie, O., Samavi, R. Doyle, T.E.  2021\.  Evaluation Metrics for Deep Learning Imputation Models Evaluation Metrics for Deep Learning Imputation Models.  35th AAAI Conference on Artificial Intelligence (AAAI-21) - 5th International Workshop on Health Intelligence 35th aaai conference on artificial intelligence (aaai-21) - 5th international workshop on health intelligence ( 309–322).  Springer.
  * Campbell-Washburn . [2019] Campbell-Washburn2019OpportunitiesMRICampbell-Washburn, A.E., Ramasawmy, R., Restivo, M.C., Bhattacharya, I., Basar, B., Herzka, D.A.Balaban, R.S.  2019\.  Opportunities in Interventional and Diagnostic Imaging by Using High-performance Low-field-strength MRI Opportunities in Interventional and Diagnostic Imaging by Using High-performance Low-field-strength MRI.  Radiology2932384–393,  https://doi.org/10.1148/radiol.2019190452
  * Emami . [2018] Emami2018GeneratingNetworksEmami, H., Dong, M., Nejad-Davarani, S.P. Glide-Hurst, C.K.  20188\.  Generating Synthetic CTs from Magnetic Resonance Images Using Generative Adversarial Networks Generating Synthetic CTs from Magnetic Resonance Images Using Generative Adversarial Networks.  Medical Physics4583627–3636,  https://doi.org/10.1002/MP.13047 https://onlinelibrary-wiley-com.ezproxy.lib.ryerson.ca/doi/full/10.1002/mp.13047
  * Goodfellow [2016] Goodfellow2016NIPSNetworksGoodfellow, I.  201612\.  NIPS 2016 Tutorial: Generative Adversarial Networks NIPS 2016 Tutorial: Generative Adversarial Networks.  ¡i¿arXiv e-prints¡/i¿,  https://doi.org/10.48550/arxiv.1701.00160 https://arxiv.org/abs/1701.00160v4 http://arxiv.org/abs/1701.00160
  * Goodfellow . [2014] Goodfellow2014GenerativeNetworksGoodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S.Bengio, Y.  20146\.  Generative Adversarial Networks Generative Adversarial Networks.  Communications of the ACM6311139–144,  https://doi.org/10.1145/3422622 https://arxiv.org/abs/1406.2661v1
  * He . [2015] He2015DeepRecognitionHe, K., Zhang, X., Ren, S. Sun, J.  201512\.  Deep Residual Learning for Image Recognition Deep Residual Learning for Image Recognition.  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition2016-December770–778,  https://doi.org/10.48550/arxiv.1512.03385 https://arxiv.org/abs/1512.03385v1
  * Hiasa . [2018] Hiasa2018Cross-modalitySizeHiasa, Y., Otake, Y., Takao, M., Matsuoka, T., Takashima, K., Carass, A.Sato, Y.  2018\.  Cross-modality Image Synthesis from Unpaired Data Using CycleGAN: Effects of Gradient Consistency Loss and Training Data Size Cross-modality Image Synthesis from Unpaired Data Using CycleGAN: Effects of Gradient Consistency Loss and Training Data Size.  Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics)11037 LNCS31–41,  https://doi.org/10.1007/978-3-030-00536-8{_}4 https://jhu.pure.elsevier.com/en/publications/cross-modality-image-synthesis-from-unpaired-data-using-cyclegan-
  * Hori . [2021] Hori2021Low-FieldRenaissanceHori, M., Hagiwara, A., Goto, M., Wada, A. Aoki, S.  202111\.  Low-Field Magnetic Resonance Imaging: Its History and Renaissance Low-Field Magnetic Resonance Imaging: Its History and Renaissance.  Investigative radiology5611669–679,  https://doi.org/10.1097/RLI.0000000000000810
  * Hosny . [2018] Hosny2018ArtificialRadiologyHosny, A., Parmar, C., Quackenbush, J., Schwartz, L.H. Aerts, H.J.  2018\.  Artificial Intelligence in Radiology Artificial Intelligence in Radiology ( 18) ( 8).  www.nature.com/nrc
  * Kazeminia . [2020] Kazeminia2020GANsAnalysisKazeminia, S., Baur, C., Kuijper, A., van Ginneken, B., Navab, N., Albarqouni, S. Mukhopadhyay, A.  20209\.  GANs for Medical Image Analysis GANs for Medical Image Analysis.  Artificial Intelligence in Medicine109101938,  https://doi.org/10.1016/J.ARTMED.2020.101938
  * Kodali . [2017] Kodali2017OnGANsKodali, N., Abernethy, J., Hays, J. Kira, Z.  20175\.  On Convergence and Stability of GANs On Convergence and Stability of GANs.  arXiv: Artificial Intelligence,  https://doi.org/10.48550/arxiv.1705.07215 https://arxiv.org/abs/1705.07215v5 http://arxiv.org/abs/1705.07215
  * Lan . [2020] Lan2020GenerativeInformaticsLan, L., You, L., Zhang, Z., Fan, Z., Zhao, W., Zeng, N.Zhou, X.  20205\.  Generative Adversarial Networks and Its Applications in Biomedical Informatics Generative Adversarial Networks and Its Applications in Biomedical Informatics.  Frontiers in Public Health8164,  https://doi.org/10.3389/FPUBH.2020.00164/BIBTEX
  * Li . [2021] Li2021WhenOpportunitiesLi, X., Jiang, Y., Rodriguez-Andina, J.J., Luo, H., Yin, S. Kaynak, O.  20219\.  When Medical Images Meet Generative Adversarial Network: Recent Development and Research Opportunities When Medical Images Meet Generative Adversarial Network: Recent Development and Research Opportunities.  Discover Artificial Intelligence111–20,  https://doi.org/10.1007/s44163-021-00006-0 https://link.springer.com/article/10.1007/s44163-021-00006-0
  * Marques . [2019] Marques2019Low-fieldPerspectiveMarques, J.P., Simonis, F.F. Webb, A.G.  20196\.  Low-field MRI: An MR Physics Perspective Low-field MRI: An MR Physics Perspective.  Journal of Magnetic Resonance Imaging4961528–1542,  https://doi.org/10.1002/JMRI.26637 https://onlinelibrary.wiley.com/doi/full/10.1002/jmri.26637 https://onlinelibrary.wiley.com/doi/abs/10.1002/jmri.26637 https://onlinelibrary.wiley.com/doi/10.1002/jmri.26637
  * Radford . [2015] Radford2015UnsupervisedNetworksRadford, A., Metz, L. Chintala, S.  201511\.  Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks.  4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings,  https://doi.org/10.48550/arxiv.1511.06434 https://arxiv.org/abs/1511.06434v2
  * Wiens . [2020] Wiens2020Feasibility0.5TWiens, C.N., Harris, C.T., Curtis, A.T., Beatty, P.J. Stainsby, J.A.  2020\.  Feasibility of Diffusion Tensor Imaging at 0.5T Feasibility of Diffusion Tensor Imaging at 0.5T.  Toronto.  https://archive.ismrm.org/2020/0303.html
  * Wolterink . [2017] Wolterink2017DeepDataWolterink, J.M., Dinkla, A.M., Savenije, M.H., Seevinck, P.R., van den Berg, C.A. Išgum, I.  20178\.  Deep MR to CT Synthesis Using Unpaired Data Deep MR to CT Synthesis Using Unpaired Data.  International workshop on simulation and synthesis in medical imaging International workshop on simulation and synthesis in medical imaging ( 10557 LNCS, 14–23).  Springer Verlag.  https://arxiv.org/abs/1708.01155v1
  * Yi . [2019] Yi2019GenerativeReviewYi, X., Walia, E. Babyn, P.  201912\.  Generative Adversarial Network in Medical Imaging: A Review Generative Adversarial Network in Medical Imaging: A Review.  Medical Image Analysis58101552,  https://doi.org/10.1016/j.media.2019.101552
  * You . [2022] You2022FineDomainYou, S., Lei, B., Wang, S., Chui, C.K., Cheung, A.C., Liu, Y.Shen, Y.  2022\.  Fine Perceptive GANs for Brain MR Image Super-Resolution in Wavelet Domain Fine Perceptive GANs for Brain MR Image Super-Resolution in Wavelet Domain.  IEEE Transactions on Neural Networks and Learning Systems,  https://doi.org/10.1109/TNNLS.2022.3153088
  * Zhu . [2017] Zhu2017UnpairedNetworksZhu, J.Y., Park, T., Isola, P. Efros, A.A.  2017\.  Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks Unpaired Image-to-Image Translation Using Cycle-Consistent Adversarial Networks.  Proceedings of the IEEE International Conference on Computer Vision Proceedings of the ieee international conference on computer vision ( 2242–2251).  https://github.com/junyanz/CycleGAN.

## Appendix A

Table 1: Comparison of Image Quality Metrics Across 1000 Synthesized Images Model | MAE $\pm$ SD | MSE $\pm$ SD | PSNR (dB) $\pm$ SD
---|---|---|---
CycleGAN 3T to 1.5T | 2106.27 $\pm$ 1218.37 | 0.00328 $\pm$ 0.0032 | 25.69 $\pm$ 2.49
CycleGAN 1.5T to 3T | 602.27 $\pm$ 147.41 | 0.00189 $\pm$ 0.0013 | 27.22 $\pm$ 0.30
DCGAN 1.5T | 31907.44 $\pm$ 415.85 | 0.24 $\pm$ 0.0059 | 6.20 $\pm$ 0.11
DCGAN 3T | 22559.57 $\pm$ 1875.35 | 0.14 $\pm$ 0.021 | 8.68 $\pm$ 0.76
Figure 2: Generation of Synthetic 1.5T DTI Scans After 50 Epoch Figure 3:
Generation of Synthetic 3T DTI Scans After 50 Epoch


#  _Bailando_ : 3D Dance Generation by Actor-Critic GPT
with Choreographic Memory

Li Siyao1 Weijiang Yu2 Tianpei Gu3,4 Chunze Lin4
Quan Wang4 Chen Qian4 Chen Change Loy1 Ziwei Liu1 🖂
1S-Lab  Nanyang Technological University
2Sun Yat-Sen University 3UCLA 4SenseTime Research
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
{linchunze, wangquan<EMAIL_ADDRESS>{ccloy<EMAIL_ADDRESS>
###### Abstract

Driving 3D characters to dance following a piece of music is highly
challenging due to the spatial constraints applied to poses by choreography
norms. In addition, the generated dance sequence also needs to maintain
temporal coherency with different music genres. To tackle these challenges, we
propose a novel music-to-dance framework, Bailando, with two powerful
components: 1) a choreographic memory that learns to summarize meaningful
dancing units from 3D pose sequence to a quantized codebook, 2) an actor-
critic Generative Pre-trained Transformer (GPT) that composes these units to a
fluent dance coherent to the music. With the learned choreographic memory,
dance generation is realized on the quantized units that meet high
choreography standards, such that the generated dancing sequences are confined
within the spatial constraints. To achieve synchronized alignment between
diverse motion tempos and music beats, we introduce an actor-critic-based
reinforcement learning scheme to the GPT with a newly-designed beat-align
reward function. Extensive experiments on the standard benchmark demonstrate
that our proposed framework achieves state-of-the-art performance both
qualitatively and quantitatively. Notably, the learned choreographic memory is
shown to discover human-interpretable dancing-style poses in an unsupervised
manner. Code and video demo are available at
https://github.com/lisiyao21/Bailando/.

Figure 1: Dance examples generated by our proposed method on various types of
music. The character is from Mixamo [1]

††🖂 Corresponding author
Figure 2: Dance generation pipeline of Bailando. Given a piece of music, an
actor-critic motion GPT autoregressively predicts the future upper-lower pose
code pairs according to the music features and starting pose codes. The pose
code sequence is then embedded to quantized features via a learned
choreographic memory and finally decoded into a dance sequence by a CNN-based
decoder.

##  1 Introduction

Music-conditioned 3D dance generation is an important task for its huge
potential to facilitate a variety of real-world applications, , assisting
human artists choreograph and driving virtual characters performance. However,
to produce satisfactory dancing sequence on given music is still very
difficult due to two main challenges: 1) Spatial constraint: Not all the
physically feasible 3D human poses are applicable for dance. The subspace of
dancing-style poses has stricter positional standards on body, and is
selective to be visually expressive and emotionally infectious based on the
choreography norms. 2) Temporal coherency with music: The generated dancing
sequence should be consistent with the music rhythm on various genres of
beats, while keeping the whole movements fluent.

Most existing dance generation studies intend to solve the two challenges both
in a single ingeniously designed network that directly maps music to 3D joint
sequence in high-dimensional continuous space [3, 19, 37, 11, 2, 30]. However,
such methods are usually unstable in practice and are prone to regress to
nonstandard poses beyond the dancing _subspace_ , , freezing or meaningless
swaying. Because there is no explicit constraints on target domain to restrict
the synthesized dance to be spatially qualified. To deal with the spatial
constraint, some works collect real dancing clips as _dance unit_ and
choreograph by splicing these units [43, 18]. While these methods guarantee
the spatial quality of generated dance by directly manipulating on real data,
the collection of dance units costs tremendous manual efforts, and they are
not compatible with different rhythms. In addition, the units cannot be reused
for different kinds of music beats due to their fixed length and speed.

In view of the shortcomings of existing methods, we propose a novel dance
generation framework, Bailando, that possesses two main components aiming at
the spatial and temporal challenges, respectively. First, to address the
spatial challenge, a finite dictionary of quantized dancing units, namely
_choreographic memory_ , is made by summarizing fundamental and reusable
constituents from movements in the dancing-style subspace. Instead of manually
indicating the dance units, we leverage the recent advances of VQ-VAE [38] to
encode and quantize 3D joint sequence to a codebook in an unsupervised manner,
where each learned code is shown to represent a unique dancing pose. To
further enlarge the range that choreography memory can represent, we divide 3D
poses into compositional upper and lower half bodies and learn VQ-VAEs for the
half bodies separately, such that any piece of dance can be represented into a
sequence of paired pose codes.

Second, to generate temporally harmonic dance sequence, a GPT-like [34]
network, named motion GPT, is introduced to translate music and source pose
codes to targeted future pose codes. Since the 3D poses are divided into
compositional half bodies in the choreographic memory, we enhance our motion
GPT with proposed cross-conditional causal attention layer to keep the
coherence of the generated body. Moreover, to achieve accurate temporal
synchronization between diverse motion tempos and music beats, we apply an on-
policy reinforcement learning scheme to further improve the motion GPT via
actor-critic [22] finetuning with a newly-designed beat-align reward function.

The inference procedure of Bailando is shown in Figure 2. Given a piece of
music and a starting pose code pair, the actor-critic GPT autoregressively
predicts the future pose code sequence, which are then embedded to
corresponding quantized features in choreographic memory, and are finally
decoded and composed to 3D dance sequence by the dedicated CNN-based decoders
of learned pose VQ-VAE.

The contributions of our work can be summarized in three folds: 1) A
choreographic memory is created to encode and quantize dancing-style 3D poses,
which is achieved by VQ-VAE in an unsupervised manner. 2) To align diverse
motion tempos with different genres of music beats, an actor-critic GPT
incorporated with the choreographic memory and cross-conditional causal
attention is introduced. 3) Extensive experiments show that our proposed
Bailando significantly outperforms the existing state of the art on both
automatic metrics and visualization judgements. Code and models will be
released upon acceptance.

Figure 3: Structure of 3D Pose VQ-VAE. The proposed 3D pose VQ-VAE is learned
to encode and summarize meaningful dancing units to choreographic memory, and
to reconstruct the target pose sequence from quantized features. The
parameters of encoder and decoders and the codebook are jointly learned during
training.

##  2 Related Work

Motion Synthesis and Music to Dance. Producing realistic human motions has
been long studied. A typical class of approaches is _graph-based_ methods.
They are developed on the idea of “cropping and pasting”, which cut motion
clips from existing data as individual nodes and splice these nodes to
synthesize new motions according to proper rules [24, 4, 23, 26]. For music to
dance, further constraints on the music rhythms, including source-target music
similarity [27], beat-wise motion connectivity [10], and deep rhythm
signatures [18], are introduced into the linking rules of the graph-based
methods to align the motion with music beats. However, since the tempos,
length, and speed of the cropped dance units are fixed, the graph-based
methods would encounter temporal conflicts on diverse rhythms. For example,
the dance units cropped in music of 4/4 time signature cannot synthesize
movement for 3/4, while the motion tempos of 60 beats per minute (BPM) is not
adaptable for 80 BPM. As a result, this kind of works can perform well in
restricted rhythm ranges but is not compatible with various genres of music
beats in wild scenarios. In recent years, with the emergence of deep learning,
many works design a dedicated network structure, including CNNs [14], RNNs
[37, 3, 40, 15], GCNs [41, 35, 9], GANs [25, 36] and Transformers [29, 30,
28], to map the given music to a joint sequence of the continuous human pose
space directly. Due to lacking explicit restrictions to keep the generated
pose within the spatial constraint, such methods would regress to nonstandard
poses that are beyond the dancing _subspace_ during inference, resulting in
instability in real uses. Besides various kinds of methods, different 3D
dancing sequence data are made from mocap and reconstruction [37, 3, 44].
Recently, a large-scale 3D dancing dataset AIST++ [30] is built from multi-
camera videos along with the music in different styles and speeds,
facilitating both training and testing of this task.

Two-Stage Generation. The two-stage approaches, which first encode data and
afterwards learn a probabilistic model to generate the encoding, have been
applied in multiple generative areas [7, 42, 8]. For example, Dhariwal et al.
[7] extracts audio features and generate songs according to the lyrics, while
most recently Esser et al. [8] encode perceptually rich image constituents to
quantized patches and tames the Transformer to generate contextually plausible
images in large resolutions. In our work, we encode and quantize meaningful
dancing constituents into a choreographic memory and generate visually
satisfactory dance by jointly translating the music and existing movements to
targeted future poses.

##  3 Our Approach

The overview of our dance generation framework, Bailando, is shown in Figure
2. Unlike other learning based methods, we do not learn a direct mapping from
audio features to the continuous domain of 3D joint sequence. Instead, we
first encode and quantize the spatially standard dance movements into a finite
codebook $\mathcal{Z}=\\{{\bf z}_{i}\\}_{i=0}^{N-1}$ as choreographic memory
in Section 3.1, where $N$ is the codebook length and every code ${\bf z}_{i}$
is shown to represent a dancing-like pose with contextual semantic
information. Specifically, we learn VQ-VAEs on the upper and lower half bodies
separately, and represent the dance movement into a sequence of compositional
upper-and-lower pose code pairs $\bm{p}=[p^{u},p^{l}]$. Then, we introduce a
motion GPT to translate the music feature and source pose codes to the future
pose codes in Section 3.2. Furthermore, to achieve synchronized alignment
between generated motion tempos and music beats, we propose actor critic
learning on the motion GPT with our newly designed beat-align rewards in
Section 3.3. The generated pose code sequences are finally decoded composed to
fluent 3D dance by VQ-VAE decoders.

###  3.1 3D Pose VQ-VAE with Choreographic Memory

Figure 4: Actor-Critic GPT. The GPT is learned to sequentially translate the
source pose codes $[p^{u}_{t},p^{l}_{t}]$ of upper-and-lower half bodies along
with music features $\bf m$ to the targeted future pose codes
$[\hat{p}^{u}_{t+1},\hat{p}^{l}_{t+1}]$. The parameters of the networks are
learned via cross-entropy loss $\mathcal{L}_{CE}$ with ground truth and actor-
critic loss $\mathcal{L}_{AC}$.

Dance positions, , the meaningful poses in dancing movements, are the basic
constituents of a piece of dance and the process of choreography can be
regarded as the combinations and connections of dance positions. Although
dances can vary greatly in style or speed, they share common dance positions.
Instead of indicating fixed units of dance with plenty of manual efforts, our
goal is to summarize such dance positions into a rich and reusable codebook in
an unsupervised manner, such that any piece of dance
$P\in\mathbb{R}^{T\times(J\times 3)}$, where $T$ is the time length and $J$ is
joint amount, can be represented by a sequence of codebook elements ${\bf
e}_{\bf q}\in\mathbb{R}^{T^{\prime}\times C}$, where $T^{\prime}=T/d$, $d$ is
the temporal down-sampling rate, and $C$ is the channel dimension of features.

To collect distinctive pose codes as well as to reconstruct them back to
represented dancing sequence efficiently, we design a 3D pose VQ-VAE as shown
in Figure 3. In this scheme, we first adopt a 1D temporal convolution network
$E$ to encode the 3D joint sequence $P$ to context-aware features
${\bf{e}}\in\mathbb{R}^{T^{\prime}\times C}$. Then, we quantize $\bf e$ by
substituting each temporal feature ${\bf e}_{i}$ to its closest codebook
element ${\bf z}_{j}$ as

${\bf e}_{{\bf q},i}=\arg\min_{{\bf z}_{j}\in\mathcal{Z}}{\|{\bf e}_{i}-{\bf
z}_{j}\|}.$ (1)

Finally, we decode the quantized features ${\bf e}_{\bf q}$ via a CNN $D_{P}$
and reconstruct the dance movement $\hat{P}$.

Compositional Human Pose Representation. In order to represent a larger range
of motions by training on limited dance data, we train independent 3D pose VQ-
VAEs and learn two separate codebooks $\mathcal{Z}^{u}$ and $\mathcal{Z}^{l}$
for the upper and lower half bodies, respectively, such that we can combine
different upper-lower code pairs to enlarge the range of dance positions that
the learned codebooks can cover. Meanwhile, to avoid encoding confusion caused
by global shift of joints (, the same motion may be encoded to different
features when it is at different locations), we normalize the absolute
locations of input $P$, , setting the root joints (hips) to be 0. To realize
the overall movement, we add a separate decoder branch $D_{V}$, which predicts
the global movement velocity $\hat{V}\in\mathbb{R}^{T\times 3}$ according to
pose codes of the lower half body, where $\hat{V}_{t}$ represents the shift of
root joint between the $(t+1)$-th and the $t$-th frames.

Learning Stable 3D Pose VQ-VAEs. The pose encoder $E$ and decoder $D_{P}$ are
simultaneously learned with the codebook via the following loss function:

$\mathcal{L}_{VQ}=\mathcal{L}_{rec}(\hat{P},P)+\|\text{sg}[\bf e]-\bf e_{\bf
q}\|+\beta\|{\bf e}-\text{sg}[{\bf e_{\bf q}}]\|.$ (2)

The global velocity decoder branch is learned afterwards by fixing the
parameters of other parts of VQ-VAE via loss function
$\mathcal{L}_{rec}(\hat{V},V)$, where $V$ is the ground truth global velocity.
$\mathcal{L}_{rec}$ is the reconstruction loss that constrains the predicted
3D joint sequence to ground truth. In this loss, we regress not only the
original 3D points of joints, but also the velocities and accelerations of
movements:

$\mathcal{L}_{rec}(\hat{P},P)=\|\hat{P}-P\|_{1}+\alpha_{1}\|\hat{P}^{\prime}-P^{\prime}\|_{1}+\alpha_{2}\|\hat{P}^{\prime\prime}-P^{\prime\prime}\|_{1},$
(3)

where $P^{\prime}$ and $P^{\prime\prime}$ represent the 1st-order (velocity)
and 2nd-order (acceleration) partial derivatives of 3D joint sequence $P$ on
time, while $\alpha_{1}$ and $\alpha_{2}$ are trade-off weights. Experimental
results show the “velocity-and-acceleration” loss items play essential roles
to prevent jitters in generated dance. (See Section 4.2.)

The second part of $\mathcal{L}_{VQ}$ is the “codebook loss” to learn codebook
entries, where $\text{sg}[\cdot]$ denotes “stop gradient” [6], while the third
part is the “commitment loss” with trade off $\beta$ [8, 7]. Since the
quantization operation of Equation 2 is not differentable, to train the whole
networks end to end, the back-propagation of this operation is achieved by
simply passing the gradient of $\bf e_{\bf q}$ to $\bf e$.

The learned choreographic memory codes are interpretable. After the training
process of pose VQ-VAEs, each quantized feature in the codebook is decoded
into a unique dance position. And any permutation and combination of codes can
be decoded to a piece of fluent movement based on corresponding dance
positions. (See Section 4.3.)

###  3.2 Cross-Conditional Motion GPT

Now that we can represent any piece of dance by a sequence of quantized
position codes, the dance generation task is then reframed as to select proper
codes from codebook $\mathcal{Z}$ for future actions according to given music
and existing movements. For any target time $t$, we estimate the probability
of every ${\bf z}_{i}\in\mathcal{Z}$ and select the one with the largest
possibility as the predicted pose code $\hat{\bm{p}}_{t}$. Since we model the
upper and lower half bodies separately, in order to keep the coherence of
composed body and to avoid the asynchronous situation (, the direction of the
upper half is opposite to that of the lower), the prediction of the future
action should be cross-conditioned between existing upper and lower movements
to make the most of mutual information:

$\left\\{\begin{array}[]{lll}\hat{p}^{u}_{t}&=&\arg\max_{k}{{\mathbb{P}}({\bf
z}^{u}_{k}|{\bf m}_{1...t},p^{u}_{0...{t-1}},p^{l}_{0...t-1})}\\\
\hat{p}^{l}_{t}&=&\arg\max_{k}{{\mathbb{P}}({\bf z}^{l}_{k}|{\bf
m}_{1...t},p^{u}_{0...{t-1}},p^{l}_{0...t-1})}\end{array}\right.$ (4)

We introduce the powerful GPT model [34] to estimate the action probabilities
as shown in Figure 4. Given a dance position code sequence with length of
$T^{\prime}$, we first embed the upper and lower pose codes to learnable
features ${\bf u}\in\mathbb{R}^{T^{\prime}\times C}$ and ${\bf
l}\in\mathbb{R}^{T^{\prime}\times C}$, respectively, and concatenate them with
music features $\bf m$ on the temporal dimension. Then, we add a learned
positional embedding to this concatenated $(3\times T^{\prime})\times C$
tensor and feed it to 12 successive Transformer layers, the structure of which
is shown in Figure 4. At last, we employ a linear transform and softmax layer
to map the output of Transformer layers to normalized action probability ${\bf
a}\in\mathbb{R}^{(3\times T^{\prime})\times N}$, where $N$ is the size of
learned codebook and ${\bf a}_{t,i}$ reveals the probability of pose code
${\bf z}_{i}\in\mathcal{Z}$ predicted for time $t+1$. The action probabilities
for upper and lower half bodies are indexed as ${\bf
a}^{u}_{0:T^{\prime}-1}={\bf a}_{T^{\prime}:2T^{\prime}-1}$ and ${\bf
a}^{l}_{0:T^{\prime}-1}={\bf a}_{2T^{\prime}:3T^{\prime}-1}$, respectively.

Figure 5: Different types of attention layers. The proposed cross-conditional
causal attention realizes causal inferences intra (gray lines) and inter (blue
lines) different kinds of components (gray and blue circles). Two kinds of
components are shown here for concision, but three (music, upper, lower
bodies) are in reality.

In Transformers [39], the attention layer is the core component that
determines the computational dependency among sequential elements of data, and
is implemented as

$\text{Attention}({\bf Q,K,V,M})=\text{softmax}\left(\frac{{\bf QK}^{T}+{\bf
M}}{\sqrt{C}}\right)\bf V,$ (5)

where $\bf Q,K,V$ denote the query, key and value from input, and $\bf M$ is
the mask, which determins the type of attention layers. The most two common
types of attentions are “full attention” [39] and “causal attention” [39],
where the former realizes the intercommunication of input data at all times
while the latter only allows the current and previous data to compute the
state for the time of interest. As our goal is to infer the future dance
position codes, we adopt the causal attention. However, since the generation
of upper and lower half bodies are dependent on each other, we cannot realize
the inference by just reordering the sequence of input to fit the causality as
previous works [8, 7]. Therefore, we propose an attention layer, namely cross-
conditional attention, to comply the causality cross conditioned among
features of the music, the upper half and the lower half bodies, where $\bf M$
is designed to be a $3\times 3$ repeated block matrix with a lower triangular
matrix of size $T^{\prime}$ as its element. As shown in Figure 5, the proposed
attention can exchange information of different components, and guarantee that
the future information will not be transmitted back to the past.

Learning Motion GPT. The motion GPT is optimized via supervised training with
cross-entropy loss on action probability $\bf a$:

$\mathcal{L}_{CE}=\frac{1}{T^{\prime}}\sum_{t=0}^{T^{\prime}-1}\sum_{h=u,l}{\text{CrossEntropy}\left({\bf
a}^{h}_{t},p^{h}_{t+1}\right)}.$ (6)

Given a sequence of pose codes $\bm{p}_{0:T^{\prime}-1}$ and relevant music
features ${\bf m}_{1:T^{\prime}}$ as input, the learned GPT outputs the
sequence of actions ${\bf a}_{0:T^{\prime}-1}$ all at once to predict
$\bm{p}_{1:T^{\prime}}$. This parallel characteristic makes Transformer an
ideal model for reinforcement learning [16, 5]. In the following subsection,
we adopt the learned motion GPT as a pretrained policy maker and propose a
novel actor-critic based finetuning scheme to further improve its performance
as complementary to the supervised training above.

###  3.3 Actor-Critic Learning

While the supervised learning scheme for the motion GPT is straightforward and
easy to train, it is intractable to further involve a more flexible constraint
of generated dance (, a regularization item that strengthens the consistency
of dance beats) to Equation (6), since the supervision target is the code
number, which is not differentiable to compute the quantitative constraints on
the final dance sequence.

To address this issue and to achieve more accurate synchronized alignment
between diverse motion tempos and music beats, we apply actor-critic learning
to the motion GPT with a newly-designed reward function. In particular, we
regard the first 6 Transformer layers of motion GPT as “state network” $f_{\bf
s}$, and the outputs of $f_{\bf s}$ are states ${\bf s}$ for time $0$ to
$T^{\prime}-1$, while the latter 6 Transformer layers along with the linear-
softmax layer are regarded as “policy making network” $f_{\bf a}$, where the
actions are computed according to state as ${\bf a}=f_{\bf s}({\bf s})$.
Besides, we add a separate three-layer Transformer branch as “critic value
network” $f_{\bf v}$ to estimate the critic values ${\bf
v}_{0:T^{\prime}-1}\in\mathbb{R}^{T^{\prime}\times 1}$ as

${\bf v}={\bf v}^{u}+{\bf v}^{l}=f_{\bf v}({\bf
s})_{T^{\prime}:2T^{\prime}-1}+f_{\bf v}({\bf
s})_{2T^{\prime}:3T^{\prime}-1}.$ (7)

With well defined reward function $R(t)={R({\bf a}_{t},{\bf s}_{t})}$, the
objective of reinforcement learning is to maximize the expected accumulated
rewards:

$J=\mathbb{E}_{\bf\tau}\left[\sum_{t=0}^{T^{\prime}-1}{R(t)}\right],$ (8)

where ${\bf\tau}=\\{{\bf a}_{t}\\}_{t=0}^{T^{\prime}-1}$ is the trajectories
of actions predicted by the policy making network. This objective is then
converted to optimize the parameters of policy making network using the
following loss function:

$\displaystyle\mathcal{L}_{AC}=$ (9)
$\displaystyle\,\,\,\frac{1}{T^{\prime}-1}\sum_{t=0}^{T^{\prime}-2}\left(\sum_{h=u,l}{\text{CrossEntropy}\left({\bf
a}^{h}_{t},\hat{p}^{h}_{t+1}\right)}\right)\cdot\text{sg}[{\bm{\varepsilon}_{t}}],$

where $\hat{p}_{t+1}^{h}=\arg\max_{i}{{\bf a}^{h}_{t,i}}$ is the pose code
number predicted by the policy making network.
$\bm{\varepsilon}\in\mathbb{R}^{(T^{\prime}-1)\times 1}$ denotes the so-called
TD-error calculated as

$\bm{\varepsilon}_{0:T^{\prime}-2}={\bf r}_{0:T^{\prime}-2}+\text{sg}[{\bf
v}_{1:T^{\prime}-1}]-{\bf v}_{0:T^{\prime}-2},$ (10)

where ${\bf r}_{t}=R(t)$. The detailed derivation of Equation (9) can be found
in the supplementary file. Meanwhile, the critic value network is optimized by
bootstrap training on difference between ${\bf v}_{0:T^{\prime}-2}$ and
$R({\bf a}_{t},{\bf s}_{t})+{\bf v}_{1:T^{\prime}-1}$:

$\mathcal{L}_{\bf v}=\frac{1}{T^{\prime}-1}\|\bm{\varepsilon}\|^{2}_{2}.$ (11)

The computation of actor-critic loss $\mathcal{L}_{AC}$ depends on real-time
actions predicted by the motion GPT and the optimization direction is
determined on the value of TD-error. When $\varepsilon_{t}$ is positive, the
optimization on $\mathcal{L}_{AC}$ will intensify the convergence to predicted
code $\hat{p}_{t+1}$, while in the opposite situations, the probability
estimated for $\hat{p}_{t+1}$ will be reduced.

Figure 6: Designed rewards. (a) Beat-align reward penalizes the absence of
dance beat for the interval that has music beat. (b) Half-body consistency
reward is computed on the angle between normal directions of half bodies to
prevent asynchronizations.

The value of TD-error and the learning effectiveness are strongly influenced
by the reward function $R$. In this work, we design a motion-music beat-align
reward to generate dance more accurate to the rhythm of music. As shown in
Figure 6 (a), the beat-align reward is defined as

$R_{b}(t)=\left\\{\begin{array}[]{lll}-1,&{\textit{\text{$\exists$ music beat
$\land\nexists$ dance beats $\in\hat{P}_{td:(t+1)d}$}}}\\\
1,&\textit{\text{otherwise},}\end{array}\right.$ (12)

where $\hat{P}_{0:T-1}=D(\hat{\bm{p}}_{0:T^{\prime}-1})$ is the dance motion
sequence decoded from predicted dance position codes. Meanwhile, to avoid the
compositional asynchronization between upper and lower half bodies during
actor-critic learning, we introduce a compositional consistency reward to
impose penalties for situations where the upper and lower body are in the
opposite direction:

$R_{c}(t)=\inf\left\\{\hat{R}_{c}(t)\right\\},t\in\left[dt,d(t+1)\right),$
(13)

where

$\hat{R}_{c}(t)=\left\\{\begin{array}[]{lll}\left<\bm{n}^{u}_{xz}(t),\bm{n}^{l}_{xz}(t)\right>,&\left<\bm{n}^{u}_{xz}(t),\bm{n}^{l}_{xz}(t)\right><0\\\
1,&\textit{\text{otherwise}.}\end{array}\right.$ (14)

Here, $\bm{n}^{u}_{xz}(t),\bm{n}^{l}_{xz}(t)$ are the normal directions of
upper and lower bodies of $\hat{P}_{t}$ projected to the $x$-$z$ plane, which
is illustrated in Figure 6 (b). The final reward is then a weighted
combination of $R_{b}$ and $R_{c}$ as $R=\gamma_{b}R_{b}+\gamma_{c}R_{c}$.

In the finetuning process, we fix the parameters of state network $f_{\bf s}$,
and alternately train the policy making network $f_{\bf a}$ and the critic
value network $f_{\bf v}$ using the losses introduced above with a small
learning rate. After such finetuning, the proposed framework will be further
enhanced.

##  4 Experiments

Table 1: Quantitative results on AIST++ test set. The best and runner-up values are bold and underlined, respectively. Among compared methods, “Li et al.”, DanceNet and FACT are multiplexing the same results of AIST++ benchmark [30], while DanceRevolution [15] is reproduced using officially released code with the optimal settings. ${\dagger}$ FIDk and DIVk are fetched from [30] while FIDg and DIVg are recomputed using the officially updated evaluation code. *The generated dances of “Li et al.” are highly jittery making its velocity variation extremely high, which is also reported in [30].  | Motion Quality | Motion Diversity |  | User Study
---|---|---|---|---
Method |  FID${}_{k}\downarrow$ |  FID${}_{g}^{{\dagger}}\downarrow$ |  Div${}_{k}\uparrow$ |  Div${}_{g}^{{\dagger}}\uparrow$ |  Beat Align Score $\uparrow$ | Our Method Wins
Ground Truth | 17.10 | 10.60 | 8.19 | 7.45 | 0.2374 |  40.0%$\pm$25.2%
Li et al. [29] | 86.43 | 43.46 |  6.85∗ | 3.32 | 0.1607 |  100.0%$\pm$0.0%
DanceNet [44] | 69.18 | 25.49 | 2.86 | 2.85 | 0.1430 |  92.7%$\pm$12.1%
DanceRevolution [15] | 73.42 | 25.92 | 3.52 | 4.87 | 0.1950 |  84.5%$\pm$10.8%
FACT [30] | 35.35 | 22.11 | 5.94 | 6.18 | 0.2209 |  98.2%$\pm$3.9%
Bailando (Ours) | 28.16 | 9.62 | 7.83 | 6.34 | 0.2332 | –

Dataset. We perform the training and evaluation on the AIST++ dataset proposed
in [30], which to our best knowledge is the largest public available dataset
for paired music and motions. This dataset contains 992 pieces of high-quality
$60$-FPS 3D pose sequence in SMPL format [31], where 952 are kept for training
and 40 are used for evaluation.

Implementation Details. In this work, the choreographic memory codebook size
$N$ for both upper and lower bodies is set to $512$, while the channel
dimension $C$ of encoded features is $512$ and the temporal downsampling rate
$d$ of encoders is $8$. The structures of the convolutional encoder and
decoders are provided in the supplementary file. While training the VQ-VAEs,
dance data are cropped to length of $T=240$ ($4$ seconds) and sampled in batch
size of $32$. The commit loss trade-off $\beta$ in $\mathcal{L}_{VQ}$ is
$0.1$, while $\alpha_{1}$ and $\alpha_{2}$ in $\mathcal{L}_{rec}$ are both set
to be $1$. We adopt Adam optimizer [21] with $\beta_{1}=0.9$ and
$\beta_{2}=0.99$ to train both pose VQ-VAEs for $400$ epochs with learning
rate $3\times 10^{-5}$. As to the motion GPT, we comply a structure mirroring
[20], where the channel dimension is $768$, and the attention layer is
implemented in $12$ heads with dropoout probability $0.1$. The music features
are extracted by the public audio processing toolbox Librosa [17], including
_mel frequency cepstral coefficients (MFCC)_ , _MFCC delta_ , _constant-Q
chromagram_ , _tempogram_ and _onset strength_ , which are $438$-dim in total,
and are mapped to the same dimension of GPT via a learned linear transform.
The block size $T^{\prime}$ of GPT is set to be $29$. While training, the
dance sequences are first encoded to pose codes $\bm{p}$ and sampled to length
of $30$, where $\bm{p}_{0:28}$ are used as input and $\bm{p}_{1:29}$ are
supervision labels. The motion GPT is optimized using Adam optimizer with
$\beta_{1}=0.5$ and $\beta_{2}=0.99$ for $400$ epochs, where the learning rate
is initialized as $3\times 10^{-4}$ and decayed after $200$ epochs with factor
$0.1$. In the actor-critic finetuning process, we adopt a small learning rate
of $1\times 10^{-5}$ to learn $f_{\bf a}$ and $f_{\bf v}$ for $10$ epochs. The
reward trade-offs $\gamma_{b}$ and $\gamma_{c}$ are $5$ and $1$, respectively.
In our experiment, the pose VQ-VAEs and the motion GPT are trained
sequentially, and the weights of VQ-VAEs are fixed during the learning process
of GPT. The whole framework is learned in three days on one Tesla V100 GPU.
During test, the motion GPT takes a pair of starting pose codes, which can be
either manually indicated or randomly sampled, as input and autoregressively
generates the motion sequence as long as the target music.

Evaluation Metrics. For quantitative evaluations, we measure the generated
dance from three perspectives: the quality of generated dances, the diversity
of motions and the alignment between the rhythms of music and generated
movements. In concrete, for the dance quality, we calculate the Fréchet
Inception Distances (FID) [13] between the generated dance and all motion
sequences (including training and test data) of the AIST++ dataset on kinetic
features [33] (denoted as ‘$k$’) and geometric features [32] (denoted as
‘$g$’), which are both extracted using the toolbox of [12]. As to the
diversity, we compute the average feature distance of generated movements
following [30]. Regarding to the alignment between music and generated
motions, we calculate the average temporal distance between each music beat
and its closest dance beat as the Beat Align Score:

$\frac{1}{|B^{m}|}\sum_{t^{m}\in B^{m}}\exp{\left\\{-\frac{\min_{t^{d}\in
B^{d}}{\|t^{d}-t^{m}\|^{2}}}{2\sigma^{2}}\right\\}},$ (15)

where $B^{d}$ and $B^{m}$ record the time of beats in dance and music,
respectively, while $\sigma$ is normalized parameter which is set to be $3$ in
our experiment.

###  4.1 Comparison to Existing Methods

We compare our proposed model to several state-of-the-art methods including Li
et al.[29], DanceNet [44], DanceRevolution [15] and FACT [30]. For each
method, we generate $40$ pieces of dances in AIST++ test set, and sample the
generated dance sequence with length of $20$ seconds to compute the evaluation
metrics mentioned above. We also calculate the quantitative scores for ground
truth data in AIST++ test set and compare it to the generated dances.

The quantitative results are shown in Table 1. According to the comparison,
our proposed model consistently performs favorably against all the other
existing methods on all evaluations. Specifically, our method improves $7.19$
($20\%$) and $12.49$ ($56\%$) than the best compared baseline model FACT on
FIDk and FIDg, respectively, and even achieves a better FIDg score than the
ground truth ($9.62$ v.s. $10.60$). If look closely to the metrics on these
two kinds of features, the kinetic feature is defined on motion velocities and
energies, which reflects the physical characteristics of dance, while the
geometric feature is defined based on multiple man-made templates of
movements, which reflects the quality of choreography. The superiority of our
method on both dance quality metrics reveals that Bailando not only
synthesizes more real-like motions than the compared baseline methods, but
also achieves outstanding performance on organizing the movements to dance via
the proposed actor-critic GPT scheme with learned choreographic memory.
Meanwhile, Bailando can generate dance with high choreographic diversity
instead of converging to few templates, and also achieves improvement on the
correlation between music and motion.

User Study. To further understand the real visual performance of our method,
we conduct a user study among the dance sequences generated by each compared
method and the ground truth data in AIST++ test set. The experiment is
conducted with 11 participants separately. For each participant, we randomly
play 50 pairs of comparison videos with a length of around 10 seconds, where
each pair contains our result and one competitor’s in the same music, and ask
the participant to indicate “which one is dancing better to the music”. The
statistics are shown in Table 1. Notably, our method significantly surpasses
the compared state-of-art methods with at least $84.5\%$ winning rate. Even in
comparison to the ground truth, $40\%$ of our generated dance is voted as the
better in average. According to the feedback from participants, our generated
dance is more “stable to the rythm” with “higher diversity”, while the reason
why our method is still not as good as real dances is mainly due to “lacking
of long-term regularity and subjective beauty”. A detailed winning rate
distribution on styles of dance can be referred to the supplementary file.

###  4.2 Ablation Studies

We conduct ablation studies on the pose VQ-VAEs and the motion GPT,
respectively. The quantitative scores are shown in Table 2. The visual
comparisons of this study can be also referred to the supplementary video.

Pose VQ-VAE. We explore the effectiveness of the following components: (1) the
up-lower half body separation, (2) the global velocity prediction branch, and
(3) the velocity-and-acceleration loss used in $\mathcal{L}_{rec}$. We train
three variant models without each of the three components, respectively. The
motion quality measured for VQ-VAEs is on reconstructed results of ground
truth of AIST++ test set. As shown in Table 2, the FIDk and FIDg values for
variant “w/o. upper/lower” become worse by $12.98$ ($46\%$) and $3.22$
($25\%$), respectively. The VQ-VAE trained on whole body cannot reconstruct
the dancing pose of test set effectively. Therefore, the separate
representations of upper-lower half bodies are necessary to enlarge the range
of poses that the choreographic memory can cover. As to the global velocity
branch, the motion quality scores of “w/o. global vel.” sharply drops $42.72$
($151\%$) and $5.89$ ($47\%$), respectively, which shows the isolated velocity
prediction is critical for representing the dance movement. For “w/o.
vel./acc. loss” variant, the FIDk is worsened by $2.68$. Although the FIDg
value of “w/o. vel./acc. loss” is slightly improved by $0.76$, the model
produces strong motion jitters if without adopting vel./acc loss for training
in the supplementary video.

Table 2: Ablation study on AIST++ test set Experiments are conducted on pose VQ-VAE and GPT, respectively.  | Method |  FID${}_{k}\downarrow$ |  FID${}_{g}\downarrow$ |  BAS $\uparrow$
---|---|---|---|---
Pose VQ-VAE | Ground Truth | 17.10 | 10.60 | –
w/o. upper/lower | 41.21 | 15.85 | –
w/o. global vel. | 70.95 | 18.52 | –
w/o. vel./acc. loss | 30.91 | 11.87 | –
full pose VQ-VAE | 28.23 | 12.63 | –
GPT | w/o. quantization | 42.71 | 147.28 | –
w/o. cross-cond. att. | 37.41 | 15.52 | –
w/o. actor critic | 28.75 | 11.82 | 0.2245
full actor-critic GPT | 28.16 | 9.62 | 0.2332

Motion GPT. For the proposed actor-critic GPT, first, we explore the effect of
quantized choreography memory by training a variant GPT directly regress to
the encoding features of 3D joint sequence via an $L_{2}$ Loss. As shown in
Table 2, the FIDg drops $135.41$ for variant “w/o. quantization” (compared to
“w/o. actor critic”, same below), while the generated dance sequences contain
frequent jitters in vision, which shows the quantization of dancing positions
is essential to our proposed framework. Second, to test the effectiveness of
the proposed cross-conditional causal attention, we substitute it to causal
attention, and train two motion GPTs for upper and lower half bodies
separately. The motion quality scores of “w/o. cross-cond. att.” drop $8.66$
($30\%$) and $3.70$ ($31\%$), respectively. The main reason for the poor
performance is that the generated dances of contain frequent asynchronization
of upper and lower half bodies, while the proposed cross-conditional attention
layer can effectively prevent such situations via the interaction of
information between the half bodies. At last, we compare the motion quality
and music-motion consistency between the model with (denoted as “full actor-
critic GPT”) and without actor critic finetuning (denoted as “w/o. actor
critic”). After the actor-critic learning, the beat-align score (BAS) of
motion GPT increases from $0.2245$ to $0.2332$, proving the effectiveness of
reinforcement learning scheme with proposed beat-align reward. Meanwhile, by
constraining the consistency with music, the actor-critic finetuning process
can also enhance the motion quanlity on choreography and saliently improves
the FIDg score by $2.20$ ($19\%$).

Figure 7: Interpretability of choreographic memory code. The sequence of
single code is decoded to static pose, while the sequence of two various codes
is decoded to smooth transition between two poses, which means each code
represents a dancing-style pose and the decoder links poses of different codes
to movements.

###  4.3 Interpretability of Choreographic Memory

In this work, we propose to summarize meaningful dancing units into the
codebook via pose VQ-VAE in an unsupervised manner. To understand what kind of
dance unit is learned in the choreographic memory, we visualize the latent
codes and find each code represents a unique 3D dancing-style pose. As
revealed in Figure 7, the first and the second rows are 3D poses decoded from
$\bm{p}_{0}=[4,4]$ and $\bm{p}_{1}=[5,5]$, respectively, where the former one
is doing right leg lifting and the latter is right bicep curl. The decoded
pose will keep static for repeating codes, and will make smooth transition
between postures of different codes. As shown in the third row of Figure 7,
the decoded 3D poses of $[\bm{p}_{0},\bm{p}_{1}]$ starts with the posture of
$\bm{p}_{0}$, while gradually putting down the leg and blending the arm
towards the pose of $\bm{p}_{1}$. Furthermore, for arbitrary combination of
learned choreographic memory codes, the decoders can synthesize fluent
movement based on the represented dance positions, which can be referred to
the supplementary video. With such characteristic, the choreography process
becomes interpretable in proposed Bailando as a process of selecting and
sorting the quantized dance positions from the learned choreographic memories,
instead of a black box as most previous works.

##  5 Discussion and Conclusion

In this paper, we address the spatial and temporal challenges of 3D dance
generation by proposing a novel framework named Bailando, which is composed of
a choreographic memory to address the spatial constraint by encoding and
quantizing dancing-style poses, and an actor-critic GPT to realize the
temporal coherency with music that translates and aligns various motion tempos
and music beats. Experiments on the standard benchmark (, AIST++ dataset)
along with user studies show that Bailando achieves state-of-the-art
performance both qualitatively and quantitatively.

Acknowledgement. This research is supported by the National Research
Foundation, Singapore under its AI Singapore Programme (AISG Award No:
AISG2-PhD-2022-01-031). This research is conducted in collaboration with
SenseTime. This work is supported by NTU NAP and A*STAR through the Industry
Alignment Fund - Industry Collaboration Projects Grant. We thank Ruilong Li,
Shan Yang and Zhiyuan Chen for their help in this work.

## References

  * [1] Mixamo.  https://www.mixamo.com/.
  * [2] Hyemin Ahn, Jaehun Kim, Kihyun Kim, and Songhwai Oh.  Generative autoregressive networks for 3d dancing move synthesis from music.  IEEE Robot. and Automat. Letters, 5:3501–3508, 2020.
  * [3] Omid Alemi, Jules Françoise, and Philippe Pasquier.  Groovenet: Real-time music-driven dance movement generation using artificial neural networks.  Networks, 8(17):26, 2017.
  * [4] Okan Arikan and David A Forsyth.  Interactive motion generation from examples.  ACM TOG, 21(3):483–490, 2002.
  * [5] Lili Chen, Kevin Lu, Aravind Rajeswaran, Kimin Lee, Aditya Grover, Michael Laskin, Pieter Abbeel, Aravind Srinivas, and Igor Mordatch.  Decision transformer: Reinforcement learning via sequence modeling.  arXiv preprint arXiv:2106.01345, 2021.
  * [6] Xinlei Chen and Kaiming He.  Exploring simple siamese representation learning.  In CVPR, 2021.
  * [7] Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever.  Jukebox: A generative model for music.  arXiv preprint arXiv:2005.00341, 2020.
  * [8] Patrick Esser, Robin Rombach, and Bjorn Ommer.  Taming transformers for high-resolution image synthesis.  In CVPR, 2021.
  * [9] João Pedro Klock Ferreira, Thiago M. Coutinho, Thiago L. Gomes, José Francisco Neto, Rafael Azevedo, Renato Martins, and Erickson R. Nascimento.  Learning to dance: A graph convolutional adversarial network to generate realistic dance motions from audio.  Comput. Graph., 94:11–21, 2021.
  * [10] Satoru Fukayama and Masataka Goto.  Music content driven automated choreography with beat-wise motion connectivity constraints.  SMC, pages 177–183, 2015.
  * [11] Shiry Ginosar, Amir Bar, Gefen Kohavi, Caroline Chan, Andrew Owens, and Jitendra Malik.  Learning individual styles of conversational gesture.  CVPR, 2019.
  * [12] Deepak Gopinath and Jungdam Won.  fairmotion - tools to load, process and visualize motion capture data.  Github, 2020.
  * [13] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter.  Gans trained by a two time-scale update rule converge to a local nash equilibrium.  NeurIPS, 30, 2017.
  * [14] Daniel Holden, Jun Saito, and Taku Komura.  A deep learning framework for character motion synthesis and editing.  ACM TOG, 35(4):1–11, 2016.
  * [15] Ruozi Huang, Huang Hu, Wei Wu, Kei Sawada, Mi Zhang, and Daxin Jiang.  Dance revolution: Long-term dance generation with music via curriculum learning.  In ICLR, 2021.
  * [16] Michael Janner, Qiyang Li, and Sergey Levine.  Reinforcement learning as one big sequence modeling problem.  In NeurIPS, 2021.
  * [17] Yanghua Jin, Jiakai Zhang, Minjun Li, Yingtao Tian, Huachun Zhu, and Zhihao Fang.  Towards the automatic anime characters creation with generative adversarial networks.  arXiv preprint arXiv:1708.05509, 2017.
  * [18] Chen Kang, Zhipeng Tan, Jin Lei, Song-Hai Zhang, Yuan-Chen Guo, Weidong Zhang, and Shi-Min Hu.  Choreomaster: Choreography-oriented music-driven dance synthesis.  In SIGGRAPH, 2021.
  * [19] Hsuan-Kai Kao and Li Su.  Temporally guided music-to-body-movement generation.  ACM MM, 2020.
  * [20] Andrej Karpathy.  https://github.com/karpathy/minGPT, 2020.
  * [21] Diederik Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2014.
  * [22] Vijay R Konda and John N Tsitsiklis.  Actor-critic algorithms.  In NeurIPS, 2000.
  * [23] Lucas Kovar, Michael Gleicher, and Frédéric Pighin.  Motion graphs.  In SIGGRAPH. 2008.
  * [24] Alexis Lamouret and Michiel van de Panne.  Motion synthesis by example.  In Comput. Animat. and Simulat. 1996\.
  * [25] Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, and Jan Kautz.  Dancing to music.  In NeurIPS, 2019.
  * [26] Jehee Lee, Jinxiang Chai, Paul SA Reitsma, Jessica K Hodgins, and Nancy S Pollard.  Interactive control of avatars animated with human motion data.  In Annual Conf. on Comput. Graph. and Interactive Tech., 2002.
  * [27] Minho Lee, Kyogu Lee, and Jaeheung Park.  Music similarity-based approach to generating dance motion sequence.  Multimedia tools and applications, 62(3):895–912, 2013.
  * [28] Buyu Li, Yongchi Zhao, and Lu Sheng.  Danceformer: Music conditioned 3d dance generation with parametric motion transformer.  In AAAI, 2022.
  * [29] Jiaman Li, Yihang Yin, Hang Chu, Yi Zhou, Tingwu Wang, Sanja Fidler, and Hao Li.  Learning to generate diverse dance motions with transformer.  ArXiv, abs/2008.08171, 2020.
  * [30] Ruilong Li, Shan Yang, D A Ross, and Angjoo Kanazawa.  Ai choreographer: Music conditioned 3d dance generation with aist++.  In ICCV, 2021.
  * [31] Matthew Loper, Naureen Mahmood, Javier Romero, Gerard Pons-Moll, and Michael J. Black.  SMPL: A skinned multi-person linear model.  SIGGRAPH Asia, 34(6):248:1–248:16, Oct. 2015.
  * [32] Meinard Müller, Tido Röder, and Michael Clausen.  Efficient content-based retrieval of motion capture data.  In SIGGRAPH, pages 677–685. 2005.
  * [33] Kensuke Onuma, Christos Faloutsos, and Jessica K Hodgins.  Fmdistance: A fast and effective distance function for motion capture data.  In Eurographics, pages 83–86, 2008.
  * [34] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al.  Language models are unsupervised multitask learners.  2019\.
  * [35] Xuanchi Ren, Haoran Li, Zijian Huang, and Qifeng Chen.  Self-supervised dance video synthesis conditioned on music.  In ACM MM, 2020.
  * [36] Guofei Sun, Yongkang Wong, Zhiyong Cheng, Mohan S Kankanhalli, Weidong Geng, and Xiangdong Li.  Deepdance: music-to-dance motion choreography with adversarial learning.  TMM, 23:497–509, 2020.
  * [37] Taoran Tang, Jia Jia, and Hanyang Mao.  Dance with melody: An lstm-autoencoder approach to music-oriented dance synthesis.  ACM MM, 2018.
  * [38] Aäron van den Oord, Oriol Vinyals, and Koray Kavukcuoglu.  Neural discrete representation learning.  In NeurIPS, 2017.
  * [39] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, volume 30. Curran Associates, Inc., 2017.
  * [40] Nelson Yalta, Shinji Watanabe, Kazuhiro Nakadai, and Tetsuya Ogata.  Weakly-supervised deep recurrent neural networks for basic dance step generation.  In IJCNN, 2019.
  * [41] Sijie Yan, Zhizhong Li, Yuanjun Xiong, Huahan Yan, and Dahua Lin.  Convolutional sequence generation for skeleton-based action synthesis.  In ICCV, 2019.
  * [42] Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas.  Videogpt: Video generation using vq-vae and transformers.  arXiv preprint arXiv:2104.10157, 2021.
  * [43] Zijie Ye, Haozhe Wu, Jia Jia, Yaohua Bu, Wenting Chen, Fanbo Meng, and Yanfeng Wang.  Choreonet: Towards music to dance synthesis with choreographic action unit.  ACM MM, 2020.
  * [44] Wenlin Zhuang, Congyi Wang, Si-Yu Xia, Jinxiang Chai, and Yangang Wang.  Music2dance: Music-driven dance generation using wavenet.  ArXiv, abs/2002.03761, 2020.