# Self-Supervised Learning of Music-Dance Representation through Explicit-
Implicit Rhythm Synchronization

Jiashuo Yu1, 2111Jiashuo Yu is an intern in ARC Lab, Tencent PCG., Junfu
Pu2222Corresponding authors., Ying Cheng1, Rui Feng1222Corresponding authors.,
Ying Shan2
1 Fudan University 2ARC Lab, Tencent PCG
<EMAIL_ADDRESS>{jevinpu<EMAIL_ADDRESS>

###### Abstract

Although audio-visual representation has been proved to be applicable in many
downstream tasks, the representation of dancing videos, which is more specific
and always accompanied by music with complex auditory contents, remains
challenging and uninvestigated. Considering the intrinsic alignment between
the cadent movement of dancer and music rhythm, we introduce MuDaR, a novel
Music-Dance Representation learning framework to perform the synchronization
of music and dance rhythms both in explicit and implicit ways. Specifically,
we derive the dance rhythms based on visual appearance and motion cues
inspired by the music rhythm analysis. Then the visual rhythms are temporally
aligned with the music counterparts, which are extracted by the amplitude of
sound intensity. Meanwhile, we exploit the implicit coherence of rhythms
implied in audio and visual streams by contrastive learning. The model learns
the joint embedding by predicting the temporal consistency between audio-
visual pairs. The music-dance representation, together with the capability of
detecting audio and visual rhythms, can further be applied to three downstream
tasks: (a) dance classification, (b) music-dance retrieval, and (c) music-
dance retargeting. Extensive experiments demonstrate that our proposed
framework outperforms other self-supervised methods by a large margin.

##  1 Introduction

Figure 1: Illustration of a dancing clip with its corresponding audio rhythms.
The motion of the dancer is in sync with the music rhythms, which means the
starting keypoint of dancing motion lies on the temporal position of music
onset maximum. The correlations between motion and onset features are used as
explicit training signals, which together with the implicit rhythm consistency
contribute to the self-supervised representation learning.

Recent years have witnessed the rapid growth of dancing video amounts on
online video-sharing websites. The demand for automatically processing dancing
videos based on content is thence becoming increasingly stronger. In
literature, many prior works have achieved promising results on music-dance
tasks, _e.g_., music-driven dance generation [54, 29, 19, 23] and music-dance
alignment [43]. However, these methods are usually task-specific and have a
strong dependence on labeled data, which restricts their generalizability and
applicability to a large extent.

A more practical way is to leverage the correlation between auditory and
visual contents as the proxy to train task-agnostic models without human
annotation. The derived representation can further be applied to a variety of
downstream tasks with minor modifications. Though recent audio-visual pre-
trained models [14, 5, 6, 26, 13, 37, 21, 32] have delivered impressive
performance, these methods are not generalizable for dancing scenarios. The
reason is that the visual contents of dancing videos consist of complex
motions and characteristics. Meanwhile, dancing music, the auditory
counterpart, embodies various attributes like lyrics, rhythms, tempos, etc.
These fine-grained features are neglected by existing self-supervised audio-
visual methods, leading to the consequence that meticulous conjunctions
crossing modalities are not fully probed. As a result, a novel multimodal
representation learning strategy is necessitated for dancing videos.

In this paper, we focus on the rhythm of dancing videos, which represents the
amplitude of music intensity as well as the visual motions of dancers. As
illustrated in Fig. 1, dancers tend to rhythmically move by music in the same
frequency to make dances look more coordinated. Such temporal key moments of
dancing motions are usually in sync with the music rhythms, which can be
reflected by the audio onset features. Inspired by this insight, we argue that
the pattern of dancing motions, termed as the visual rhythms, should be
synchronous with music rhythms. This temporal correspondence between rhythms
can further be utilized as the supervised signals for self-supervised
learning.

Specifically, the synchronization of music and dance rhythms is conducted both
explicitly and implicitly. We argue that rhythms are implied in the auditory
and visual contents, thus we leverage the consistency between audio and visual
streams to perform implicit rhythm alignment. Moreover, the audio onset, which
refers to the beginning of a musical note, can be regarded as the music
rhythm. In this work, we train the model to learn visual rhythms based on
appearance and motion cues, which are expected to be aligned with music
rhythms in an explicit way. Finally, the entire model is unified by joint
training for the multimodal representation as well as visual rhythm
extraction. By leveraging the joint representation and visual rhythms, MuDaR
can be applied to many downstream applications. We conduct experiments on
three practical tasks: dance classification, music-dance retrieval, and music-
dance retargeting. The comparison with other self-supervised audio-visual
methods verifies the effectiveness of our framework.

The contributions are summarized as follows:

  * •

We devise a novel self-supervised representation learning strategy for dancing
videos, which performs the music-dance rhythm synchronization both in explicit
and implicit ways.

  * •

We propose a joint music-dance representation and a dance rhythm extractor
favorable for music-dance understanding and re-creation tasks.

  * •

Extensive experiments on three downstream tasks, _i.e_. dance classification,
music-dance retrieval, and dance-music retargeting, verify the effectiveness
and generalizability of our model on music-dance scenarios.

##  2 Related Works

Audio-visual representation learning aims to learn the multisensory embeddings
favorable for downstream tasks. Several existing methods [14, 5, 6, 26, 13,
37, 3, 2] focus on the characteristics of auditory and visual streams in a
given video. The audio and visual contents in videos are usually corresponding
and synchronous, and these natural inter-modality correlations can be utilized
as the supervisory signal for large-scale self-supervised training. Apart from
such correspondence, some works focus on the discriminative information across
the modalities. [21, 4] explore the intrinsic differences between modality-
aware semantics and propose a cross-modal deep clustering method to perform
cross-modality supervision. [32] propose a cross-modal active contrastive
coding strategy to fully explore the diversity between positive and negative
samples. Moreover, some methods [8, 52, 51, 22, 18, 17, 34, 1] explore the
relationship of visual dynamic motions and audio signals. The correspondence
between dynamic motions of objects and sound sources can be used for self-
supervised object detection. In this paper, we propose a unified framework to
combine these knowledge priors above, where the temporal correspondence is
utilized for implicit rhythm synchronization, and the dynamic motions
corresponding to music are exploited for explicit rhythm keypoint alignment.

Dance with music is a fundamental and challenging video category, where the
movements of dancers are complex and the patterns of music and dance are
diverse. Hence, models are required to explore more detailed information by
capturing fine-grained features. Attempts in the representation learning field
focus on the uni-modal music embeddings [44, 30, 53, 49, 50, 48]. Since music
can be handled as a sequence of tokens, an ordinary pipeline is to train
pretrained models that have achieved promising performance in natural language
processing, such as BERT [16]. However, the embeddings of the visual
counterpart: dancing, as well as the joint music-dance representation that
captures cross-modal music-dance interactions, are rarely investigated. For
the multimodal dance-music applications, many works [54, 29, 19, 23] tackle
the problem of music-dance generation, that is, generating a sequence of
dancing videos based on the given music. Some methods [43] work on the audio-
visual alignment, which tries to align mismatched dancing and music streams by
learning frame-level dense correspondence. The most relevant to our work is
visual rhythm and beat prediction [46, 15, 38], where visual rhythms and beats
are extracted based on dancing motions or dancer skeletons. In our paper, we
ameliorate visual rhythm predictions by involving more characteristics, and
further leverage the synchronization of rhythms as a training proxy for
multimodal representation learning.

Figure 2: Illustration of our MuDaR framework. MuDaR involves two pathways:
implicit and explicit music-dance synchronization. For the implicit pathway,
we leverage the auditory and visual features to conduct corresponding and
synchronization predictions with binary classification. For the other part,
MuDaR explicitly learns visual rhythms based on the motions of dancers by
performing temporal alignment with audio onsets, which can be viewed as the
music rhythms. Two streams are trained jointly in a self-supervised manner.

##  3 Approach

MuDaR conducts rhythm synchronization via a two-pathway architecture, and the
illustration of our framework is shown in Fig. 2. Our model performs explicit
synchronization via the temporal alignment between music and dance rhythm
keypoints, while the implicit part is implemented by identifying the synchrony
between audio and visual streams. The details of each part are introduced
below.

###  3.1 Explicit Rhythm Synchronization

####  3.1.1 Music Rhythm Detection.

For the music rhythm, a straightforward idea is to utilize the onset feature,
which indicates the sudden increase of music volumes. Following [10], we first
obtain the spectrograms by conducting time-windowed FFT to the audio signal
represented as follows,

$\small X(n,k)=\sum_{q=-\frac{N}{2}}^{\frac{N}{2}-1}v(hn+q)w(q)e^{-\frac{2j\pi
qk}{N}},$ (1)

where $X(n,k)$ denotes the $k^{th}$ frequency bin of temporal position $n$;
$v(\cdot)$ is the audio signal; $h$ denotes the hop size; $w(\cdot)$ is the
Hamming window, $N$ is the time window size.

Then we compute the bin-wise difference between spectrograms to get the
spectral flux, which indicates the magnitude of audio signals. The onset
envelope, a positive 1D feature, is computed by summing the positive spectral
flux:

$\small
OE(n)=\sum_{k=-\frac{N}{2}}^{\frac{N}{2}-1}max(0,\left|X(n,k)\right|-\left|X_{ref}(n-\mu,k)\right|),$
(2)

where $n,k$ denotes the temporal position and bin number, respectively;
$OE(n)$ is the onset envelope at time step $n$; $X_{ref}$ denotes the maximum-
filtered spectrogram proposed in [10]; $\mu$ is the time lag.

Finally, we pick the local maximum of the onset envelope to detect the
discrete onset feature. The local maximum will be selected as an onset only if
it is some threshold above the local average value. The entire rhythm
extraction procedure can be derived by the python package LibROSA [33] with
the default hyperparameter settings for all algorithms mentioned above.

####  3.1.2 Dance Rhythm Detection.

For the dancing rhythms, we focus on the motion characteristics of dancers
based on dense optical flow estimation.

Optical flow estimator. Optical flow indicates the direction and magnitudes of
visual motions. Therefore, we estimate the dense optical flows as the initial
step of dance rhythm detection. The effective, lightweight PWC-Net [41] is
selected as the backbone optical flow network. To be specific, PWC-Net
extracts feature maps in different resolutions via a feature pyramid extractor
consisting of several stacking convolution layers, then warps features of the
current frame toward the previous frame and constructs a cost volume. Finally,
the optical flow is predicted by a multi-layer CNN estimator and refined by a
dilated context network.

Histograms of Optical Flow. Though some existing methods estimate visual
trajectory by dense optical flows [51, 18], these trajectories live in the
Euclidean space, thereby highly relying on the constant of camera view. On the
contrary, we utilize the Histogram of Oriented Optical Flow (HOOF) [12], a
non-Euclidean feature, to represent motions in a non-linear manifold. HOOF is
independent of the camera directions, amplification of camera view, and
background noise, which is more robust and generalizable for more dancing
scenarios. Specifically, HOOF is computed by the weighted summation of the
magnitude of optical flow, where the optical flow $v(t,x,y)$ in each time-
stamp is separated into $n$ bins according to its angle from the horizontal
axis, represented as follows:

$\small H(n,k)=\sum_{x,y}M_{t}(x,y)\mathbbm{1}_{\theta}(P_{t}(x,y)),$ (3)
$\small\mathbbm{1}_{\theta}(\phi):=\begin{cases}1,&\text{if
}|\theta-\phi|\leq\frac{2\pi}{B},\\\ 0,&\text{otherwise},\end{cases}$ (4)

where $M_{t}(x,y)\in\mathbb{R}^{H\times W}$ denotes the magnitude of optical
flow in $t^{th}$ time, which is computed by $\sqrt{x^{2}+y^{2}}$;
$P(t)\in\mathbb{R}^{H\times W}$ indicating the angle of optical flow with
x-axis in time step $t$ is calculated by $tan^{-1}\frac{y}{x}$,
$\mathbbm{1}_{\theta}(\phi)$ is an indicator function; and $B$ denotes the
total number of bins.

RGB injector. A straightforward way to predict visual rhythms is directly use
HOOF for visual rhythm estimation. However, this could lead the prediction to
be highly dependent on the quality of optical flow. Considering that the
feature maps extracted from raw images may also contain contributing
information, we propose an RGB injector to infuse visual cue to motion
features as an enhancement.

To be specific, we use the feature map $f_{rgb}$ generated by the encoder of
PWC-Net [41] in the explicit rhythm synchronization pathway. Then $f_{rgb}$ is
tiled and projected by a linear layer to reduce the dimensionality. To
highlight the rhythm keypoints in the temporal dimension, we introduce the
long-range temporal interactions by the multi-head self-attention mechanism
[42]. The refined visual features are put into the visual rhythm estimator as
the infusion.

We also argue that the inherent coherence of music-dance signals can be
utilized for visual rhythm prediction. The audio-guided spatial-channel
attention mechanism [47] is leveraged to explore the relationship between
auditory and visual features. We use audio features $f_{a}$ extracted by the
audio encoder of the implicit rhythm synchronization pathway (which will be
introduced in Sec. 3.2), and the audio-guided visual features can be computed
as follows,

$\displaystyle\small w_{a:rgb}^{c}=\sigma(W_{1}U_{1}^{c}(\rho_{a}(f_{a}\odot
U_{rgb}^{c}(f_{rgb})))),$ (5) $\displaystyle
f_{a:rgb}^{c}=\sum_{i=1}^{k}w_{a:rgb}^{c:i}f_{rgb}^{i},$ (6) $\displaystyle
w_{a:rgb}^{s}=softmax(\delta(W_{2}(U_{a}^{s}(f_{a})\odot
U_{rgb}^{s}(f_{rgb}^{c})))),$ (7) $\displaystyle
f_{a:rgb}=\sum_{i=1}^{k}w_{a:rgb}^{s:i}f_{rgb}^{c:i},$ (8)

where $U_{rgb}^{c},U_{rgb}$ are linear layers with non-linearity activation;
$W_{1},W_{2}$ are learnable parameters; $\sigma$ indicate the sigmoid
function; $\rho$ denotes global average pooling; $\delta$ is the hyperbolic
tangent function; and $w_{a:rgb}^{c},w_{a:rgb}^{s}$ are the channel and
spatial attention map, respectively.

However, the prediction of dance rhythms cannot rely on the corresponding
music when applied to downstream tasks. The performance significantly declines
when audio signals are available during training while missing in inference.
To address this problem, we refer to Dropout [40], a simple yet effective
method which initial goal is to prevent model from being dependent on some
specific neurons. Inspired by this paradigm, we propose an audio dropout gate,
which randomly drops partial auditory inputs in a mini-batch with constant
ratios during training. To this end, MuDaR is capable of detecting visual
rhythms both with and without music. The entire procedure can be formulated
as:

$\displaystyle f_{rgb1},f_{rgb2}=AudioDropout(f_{rgb},p),$ (9) $\displaystyle
AudioDropout(f,p)=f[b*p:],f[:b*p],$ (10) $\displaystyle
f_{a:rgb1}=AGVA(f_{rgb1},f_{a}),$ (11) $\displaystyle
f^{\prime}_{rgb2}=Linear(Tile(f_{rgb2})),$ (12) $\displaystyle
f_{a:rgb}=Concat(f_{a:rgb1},f^{\prime}_{rgb2}),$ (13) $\displaystyle
f_{inj}=Att(f_{a:rgb},f_{a:rgb},f_{a:rgb}),$ (14)

where $f_{inj}$ denotes the output of the RGB injector; AGVA denotes the
audio-guided visual attention introduced in Eq.(5)$\sim$(8); Att denotes the
multi-head self-attention; $b$ denotes the mini-batch size; and $p$ is the
audio dropout rate.

Visual rhythm estimator. To explore the magnitude of visual appearance, we
compute the first-order difference of motion features and RGB features, which
are further combined with linear projection. Then a linear layer with non-
linear activation is used for binary classification,

$p_{t}^{e}=\sigma(W_{e}(U_{mot}f^{\prime}_{mot}\oplus
U_{inj}f^{\prime}_{inj})+b_{e}),$ (15)

where $f^{\prime}_{mot},f^{\prime}_{inj}$ are the first-order difference of
motion and injected features; $U_{mot},U_{inj}$ are linear layers with ReLU
[36] activation; $\oplus$ denotes concatenation across the channel dimension;
$W_{e},b_{e}$ are parameters of linear classifier; $\sigma$ denotes the
sigmoid function.

###  3.2 Implicit Rhythm Synchronization

We claim that rhythms of music and dance are implied in the auditory and
visual features, and the synchronization of music and dance streams can also
be considered as the implicit version of rhythm synchronization. Following
[37, 13, 1, 26], we employ Audio-Visual Correspondence (AVC) and Audio-Visual
Temporal Synchronization (AVTS) as the pretext tasks for representation
learning. Specifically, we leverage the raw videos as the positive samples,
while creating asynchronous samples by performing temporal shifts and creating
uncorrelated samples via combining visual and audio streams from different
videos. Then the model is required to predict whether the assembled sample is
synchronous (for AVTS) and corresponding (for AVC) or not, thereby performing
self-supervised training.

One problem of this unsupervised paradigm is the construction of false-
negative samples. For the AVC task, if we randomly sample a sequence of music
that is the same as the raw dancing video, the newly-assembled sample is
actually music-dance corresponding. To address this problem, we compute the
rhythm similarity score of the raw and newly selected audio streams by
calculating the coincidence of rhythm positions, which is formulated as
follows:

$\small s_{rhy}=\sum_{t=1}^{T}\left|O_{pos}^{t}-O_{neg}^{t}\right|-\alpha T,$
(16)

where $s_{rhy}$ is the rhythm similarity score; $O_{pos}^{t}$ denotes the
positive music onset features; $O_{neg}^{t}$ indicates the negative onsets; T
denotes the length of dancing videos; $\alpha\in[0,1]$ is a threshold
hyperparameter. The negative sample will be selected only when the score is
positive. For the AVTS task, the rhythms of dancing videos are sometimes
periodic. Therefore, if we shift the current rhythm point exactly to another
point afterward, the rhythms of audio and visual streams are re-aligned with
certain time-lagged. To this end, we conduct additional constraints to prevent
the shifted size from being the multiple of rhythm interval. Though the rhythm
interval tends to be diverse, most onset peaks are concurrent with music
beats, which are distributed with a constant temporal gap. In our work, the
eighth note (quaver) is selected as the basic beat unit, and the shifted
number of visual frames $f_{sft}$ cannot be the multiple of the frame number
identical to an eighth note,

$f_{sft}\bmod(k_{fps}*\frac{60}{k_{bpm}}*\frac{1}{2})\neq 0,$ (17)

where $k_{fps}$ denotes the frame sample rate; $k_{bpm}$ denotes the number of
quarter note (crotchet) per minute. Since we use the eighth note as the base
beat unit, the number of beat units per minute will be $2*k_{bpm}/60$.

After performing sampling constraints, the implicit rhythm synchronization can
be trained in a self-supervised manner. To perform coherent optimization, we
leverage the pyramid encoder of the PWC-Net as the visual encoder, and the
last feature map of the pyramid output is kept as the visual feature. We also
build a similar audio encoder including several 2D convolution layers. Then
auditory and visual features are put into $K$ stacking transformer [42]
layers, respectively. We do not involve any modality interaction to make our
framework generalizable for single-modality downstream tasks. Each transformer
layer is constructed by the encoder part of the raw transformer architecture
[42], including the multi-head self-attention, feed-forward layer, layer
normalization [7], and residual connection [20]. In the self-attention block,
features are temporally interacted to fully explore the long-range temporal
correlation. Finally, triplet loss [39] is used for the refined auditory and
visual features, which enlarges the distance between visual features and
negative auditory features, while reducing the gap between visual features and
positive audio features.

###  3.3 Optimization

For the implicit synchronization, using a classifier to predict binary logits
and adopt the binary cross-entropy loss as the learning objective is a natural
way for the AVC and AVTS tasks. However, this method suffers from the
difficulty of training convergence. Some prior methods [14, 26, 32] opt to the
contrastive loss as a substitution. In this work, the dense optical flow
estimation requires high computation cost, thus the performance will be
limited when the batch size is small. In this paper, we choose the triplet
loss [39] with offline negative sample strategy. Specifically, we put a pair
of positive audio and visual samples into the two-stream implicit
synchronization pathway. Then we choose the negative audio samples from the
whole dataset, which is brought into the audio streams together with the
positive audio sample. The triplet loss function tries to make the refined
positive audio feature $f^{\prime}_{a:pos}$ closer to the refined visual
features $f^{\prime}_{v}$, while enlarge the distance between negative audio
features $f^{\prime}_{a:neg}$ and $f^{\prime}_{v}$,

$\small\mathcal{L}^{im}=\sum_{i=1}^{N}\bigg{[}||f^{\prime}_{v}-f^{\prime}_{a:pos}||_{2}^{2}-||f^{\prime}_{v}-f^{\prime}_{a:neg}||_{2}^{2}+\alpha\bigg{]}_{+},$
(18)

where $\mathcal{L}^{im}$ denotes the implicit loss, $\alpha$ is the margin
between positive and negative pairs; $[\cdot]_{+}$ denotes picking the
positive results, and $N$ denotes the number of triplet pairs.

For the explicit synchronization, the straightforward way of using binary
cross-entropy loss may cause overwhelming negative predictions during training
due to the unbalanced distribution of rhythm and non-rhythm temporal
keypoints. To address this problem, we adopt the focal loss proposed by [31].
Focal loss addresses the class imbalance by introducing a weighting
hyperparameter $\alpha$, and focuses more on hard samples rather than easier
ones via the focusing parameter $\gamma$. The explicit loss $\mathcal{L}^{ex}$
can be formulated as,

$\mathcal{L}^{ex}=-\alpha_{t}(1-p_{t}^{e})^{\gamma}log(p_{t}^{e}),$ (19)

where $p_{t}$ is the visual rhythm prediction.

Finally, the optimization is conducted in a joint-training manner, the overall
objective function is formulated as,

$\mathcal{L}=\lambda_{1}\mathcal{L}^{im}+\lambda_{2}\frac{1}{N}\sum_{t=1}^{N}\mathcal{L}_{t}^{ex},$
(20)

where $\lambda_{1},\lambda_{2}$ are hyper-parameters, which indicate the
weighted summation of these loss terms, $N$ is the number of frames in each
video.

##  4 Experiments

In this section, we first introduce the implementation details and ablation
results of the self-supervised representation learning process. Then we
conduct experiments on three downstream tasks: dance classification, music-
dance retrieval, and music-dance retargeting.

###  4.1 Music-Dance Representation Learning

Approach | Data Size | Recall | Precision
---|---|---|---
MuDaR w/ BCE | 190k | 82.2% | 37.0%
MuDaR w/ regression | 190k | 43.8% | 51.9%
MuDaR w/ CRF | 190k | 70.2% | 79.8%
MuDaR (full) | 47k | 59.6% | 73.1%
MuDaR (full) | 140k | 71.4% | 78.9%
MuDaR (full) | 190k |  73.2% |  82.0%
Table 1: The evaluation of visual rhythm prediction compared with two ablated
models. We also report the performance of our full model with different
dataset sizes.

Dataset. MuDaR can be trained by large-scale dancing videos without any
annotation. The only requirement for unlabeled data is that dancers should
twist their bodies following the rhythms of music, which is a fundamental rule
for dancing. In this paper, we collect 194,407 dancing videos from an
information feed platform and online video-sharing application. The training
data can be separated into four categories, _i.e_. dancing with pop music,
hip-hop, modern dance, and K-pop, with nearly equal proportions. These
category labels are unnecessary during the training and inference phase.

Implementation details. All dancing videos are cropped to 8s clips. To remove
background noises, we discard the beginning and ending part of each dance, and
select 4 seconds before the middle of the video together with 4 seconds after.
Visual frames are sampled with a frame rate of 8 fps. Then frames are resized
to 256$\times$256\. To maintain the camera view unchanged, we do not apply any
visual transformation for data augmentation. Audios are sampled with a sample
rate of 16kHz, and we extract Mel-spectrograms, onset envelopes, onset
maximum, beat, and tempograms using the python package LibROSA [33]. For the
implicit synchronization pathway, the audio inputs are the concatenation of
Mel-spectrograms, beat, onset envelopes, and tempograms, while the onset
maximums are used as the ground-truth of rhythms for the explicit
synchronization. We use Adam [25] optimizer with the initial learning rate of
1e-4. The entire model is trained on 64 NVIDIA Tesla V100 GPUs for 50 epochs
with a batch size of 384. For the implicit synchronization, we adopt the
curriculum learning strategy as [26]. AVC is used for the first 35 epochs,
while the more challenging pretext task AVTS is used for the rest of the
training process. More details are complemented in Appendix A.

Evaluation of MuDaR. We randomly sample 4,407 videos of the original 194,407
videos for evaluation, while using the remaining 190k videos for self-
supervised training. To fully investigate the performance of the explicit
rhythm pathway, we propose three ablated models for comparison. “MuDaR w/ CRF”
denotes an ablated model that replaces the binary prediction layer with a
BiLSTM-CRF [28] layer, which is composed of a bidirectional long-short memory
network [24] and a conditional random field [27] module. This ablated model
regards the visual rhythm prediction task as a sequence labeling problem and
considers the inner relationship of the consecutive frames. “MuDaR w/
regression” takes the visual rhythm prediction task as a regression problem.
Instead of using local maximum over onset envelopes as the audio ground-truth,
it directly minimizes the distance between model output and the onset envelope
curve. The visual rhythms are then generated by picking the local maximum of
the model output. “MuDaR w/ regression” replaces the binary focal loss with
the raw binary cross-entropy loss. As shown in Tab. 1, we report recall and
precision as the evaluation metrics. Results show that “MuDaR w/ BCE” achieves
higher recall compared with raw MuDaR, while obtaining low precision, proving
that the cross-entropy loss suffers from the imbalanced class distribution.
“MuDaR w/ regression” achieves poor performance on all metrics. We argue that
though the temporal position of music and dance rhythms are coincident, it’s
illogical to force the intensity of visual rhythms to be identical to the
audio rhythms. On the contrary, the full MuDaR model achieves promising
results with respect to all metrics, showing the effectiveness of detecting
visual rhythms. “MuDaR w/ CRF” performs slightly worse than the full model.
This suggests that the BiLSTM-CRF layer performs unsatisfied when there are
only two unbalanced categories for sequence labeling. More ablation studies
are listed in Appendix B.

###  4.2 Dance Classification

Downstream architecture. We first evaluate the performance of MuDaR on the
task of dance classification. To fully utilize the outputs of MuDaR (audio and
visual embeddings, visual rhythms), we concatenate the visual embeddings and
rhythms as visual outputs, and the auditory embeddings and onset as audio
outputs. Then the audio and visual output features are put into a two-stream
classifier. To be specific, audio and visual features are first put into two
fully-connected layers with non-linear activation. Then the audio and visual
features are concatenated and integrated by temporal pooling. Another two
fully-connected layers with non-linearity are used for prediction.

Approach | Data Size | Accuracy | Manner
---|---|---|---
Castro et al. [11] | / | 70.2% | Supervised
MDR [45] | / | 77.0% | Supervised
Multisensory [37] | 190k | 71.4% | S-Sup.+ ft
AVTS [26] | 190k | 68.3% | S-Sup.+ ft
LLA [13] | 190k | 73.0% | S-Sup.+ ft
MuDaR (full) | 47k | 76.1% | S-Sup.+ ft
MuDaR (full) | 140k | 79.4% | S-Sup.+ ft
MuDaR w/o rhy | 190k | 80.8% | S-Sup.+ ft
MuDaR (full) | 190k |  81.7% | S-Sup.+ ft
Table 2: The dance classification accuracy on Let’s Dance [11] dataset.
“S-Sup.+ ft” denotes models pretrained on the self-supervised music-dance
dataset, and then fine-tuned on the dance classification dataset.

Dataset. We evaluate the performance of dance classification on the Let’s
Dance [11] dataset. This dataset contains more than 1,400 10-second dancing
videos on a variety of 16 dance categories. Since a small part of the videos
are unavailable on the Internet, we downloaded 1,262 dancing videos and
randomly split the training, validation, and testing set with the proportion
of 80%/10%/10% following [11, 45]. We employ the same data preprocessing
procedure as the self-supervised representation training.

Implementation details. We train our network using Adam [25] optimizer with
the initial learning rate of 4e-3, which is degraded by 10 after 20 and 50
epochs. The model is trained with a batch size of 64 for 70 epochs.

Experimental results. We compare MuDaR with two supervised methods: Temporal
Three-Stream CNN [11] and Multimodal Dance Recognition [45]. To make a fair
comparison, we also re-implement three audio-visual self-supervised methods:
Multisensory [37], AVTS [26], and LLA [13], which are trained on the same
large-scale dancing dataset used for MuDaR. Results shown in Tab. 2 indicate
that our model outperforms all self-supervised baselines by a large margin.
Moreover, the performance on the full pre-trained dataset (190k) surpasses the
state-of-the-art supervised methods, showing the effectiveness of our
representation learning framework on this downstream task. We also conduct
ablation studies by removing the visual and auditory rhythms. Results show
that MuDaR performs slightly worse without rhythm involvement. However, the
decline is not significant, indicating the performance of dance classification
mainly depends on the embeddings generated by the implicit synchronization
pathway.

###  4.3 Music-Dance Retrieval

Approach |  | Data
---
Size
R@1 | R@5 | P@10 | mAP
Multisensory [37] | 190k | 0.468 | 0.732 | 0.480 | 0.541
AVTS [26] | 190k | 0.430 | 0.698 | 0.467 | 0.484
LLA [13] | 190k | 0.501 | 0.781 | 0.512 | 0.576
MuDaR (full) | 47k | 0.532 | 0.782 | 0.596 | 0.573
MuDaR (full) | 140k | 0.604 | 0.894 | 0.632 | 0.622
MuDaR w/o rhy | 190k | 0.586 | 0.872 | 0.615 | 0.601
MuDaR (full) | 190k | 0.622 | 0.924 | 0.661 | 0.633
Table 3: Evaluation of music-dance retrieval of our framework compared with
other self-supervised methods. MuDaR w/o rhy denotes the ablated model without
rhythm involvement.

Downstream architecture. We also evaluate our model on the music-dance
retrieval task. For the cross-modal retrieval, we compute the similarity
scores of rhythms and embeddings, respectively. To be specific, for the
embedding $E_{a},E_{v}$ and rhythm $R_{a},R_{v}$, we compute the embedding
similarity matrix $S_{e}$ consisting of similarity scores of $E_{a}$ and
$E_{v}$, then compute the rhythm similarity matrix $S_{r}$ via $R_{a}$ and
$R_{v}$. Finally, the hybrid similarity matrix can be computed by the weighted
summation of $S_{e}$ and $S_{r}$:

$S_{hyb}=\lambda_{3}S_{e}+(1-\lambda_{3})S_{r},$ (21)

where $\lambda_{3}$ is the hyperparameter. The top-K indices of the hybrid
matrix are preserved as the retrieval result.

Dataset and task formulation. We conduct experiments on the Dance-50 [43]
dataset for cross-modal retrieval. Dance-50 contains 50 hours of K-pop dancing
videos collected from online video platforms. This dataset is originally used
for audio-visual alignment without any category annotation available. To
perform music-dance retrieval, we selected 22 dance music appeared with high
frequency, where each song corresponds to 15-40 dancing videos from different
dancers. We collected 400 labeled videos in total while using another 1833
unlabeled dances as the irrelevant data in the retrieval database. Since
dances in the Dance-50 dataset are long videos of more than 30 seconds, we
pre-process each annotated video by clipping 24 seconds from the beginning.
For the unlabelled videos, we separated each video into several 24s dance
clips. In this way, we got 400 labeled and 4,066 unlabeled dancing clips as
the retrieval dataset. The objective of this task is to retrieve all related
dances given the targeted music.

Experimental results. We leverage the self-supervised methods Multisensory
[37], AVTS [26], and LLA [13] as baselines. For these baselines, since visual
rhythms are unavailable, we only use the similarity matrix between embeddings
for retrieval. We also investigate the role of visual rhythms via the ablated
model “MuDaR w/o rhy”, and we report average top-k retrieval performance
(R@K), top-k precision (P@K), and mAP as the evaluation metrics. As shown in
Tab. 3, results show that MuDaR outperforms all baselines by a large margin.
Our framework outperforms baseline LLA by 15% on R@5 and P@10, and 6% on mAP,
which shows the effectiveness of our music-dance framework. Besides, the
performance significantly declines after removing the rhythm information,
indicating that rhythms are favorable for cross-modal retrieval.

###  4.4 Music-Dance Retargeting

Task Formulation. Music-dance retargeting aims to synthesize a new dancing
video via two mismatched music and dance clips. This task can be combined with
the music-dance retrieval task as an automatic soundtrack generator, which
synthesizes rhythm-aligned dancing videos after obtaining related music via
cross-modal retrieval. Concretely, given a sequence of visual frames $V$, and
a music clip $A$ from different dancing videos, the objective of music-dance
retargeting is to combine $V$ and $A$ into a natural new dancing video. This
task evaluates the performance of rhythm extraction thoroughly, since we
conduct the retargeting by warping the visual rhythms $R_{v}$ into the
alignment with the auditory counterpart $R_{a}$.

Downstream architecture. Due to the difference between the temporal lengths of
$V$ and $A$, we try to align as many rhythm points as possible via
acceleration and shifting. We propose three retargeting patterns: temporal
shifting, temporal acceleration, and dynamic temporal warping. For the
temporal shifting, we propose a sliding window which size is equal to the
length of the shorter sequence, and put it on the rhythm of longer duration.
We reserve the cropped clip with the most similar number of rhythm points,
then align the starting rhythm points of two sequences. The second pattern is
temporal acceleration, where we compute the temporal interval $I$ between two
adjacent rhythm points: $I_{k}^{m}=P_{k+1}^{m}-P_{k}^{m}$, where
$m\in\\{a,v\\}$ denotes the modality index, $k$ indicates the rhythm keypoint
index, and $P_{k}^{m}$ denotes the temporal position of the $k^{th}$ rhythm in
the modality $m$. Then we manage to make $I_{k}^{v}$ and $I_{k}^{a}$ equal for
all rhythm indices by interpolating and removing visual frames. The third
manner is dynamic temporal warping (DTW) [9, 35], which is capable of
selecting the most suitable aligning strategy. Specifically, DTW aims to find
the optimal warping path that aligns $R_{v}$ and $R_{a}$ with minimum rhythm
mismatch cost, which is defined by the accumulated distance between the rhythm
keypoints. The cumulative distance $c$ in position $(i,j)$ can be computed
using dynamic programming:

$\small c(i,j)=d(i,j)+M\\{c(i-1,j-1),c(i-1,j),c(i,j-1)\\},$ (22) $\small
d(i,j)=|P_{i}-P_{j}|,$ (23)

where $M(\cdot)$ denotes the minimize function; $i,j$ are indices of rhythms;
$P_{i}$ denotes the temporal position of the $i^{th}$ rhythm point. The
warping path determines the detailed frame accelerate strategy with minimum
warping costs, thereby resulting in optimal retargeting performance.

Figure 3: Qualitative results of three methods on the music-dance retargeting
task. Frames with colored boxes denote visual rhythm positions. Results show
that videos synthesized by DTW achieve both optimal rhythm alignment and
satisfactory viewing fluency.

Qualitative results. Since there is no feasible quantitative metric for music-
dance retargeting, we provide some qualitative results to show the performance
of our method as shown in Fig. 3. All dancing videos are randomly selected
from the test set of the representation learning dataset. For the synthetic
videos by temporal shifting, the inner rhythm may be mismatched for we only
perform alignment on the first rhythm points, which makes the synthetic video
uncoordinated. Besides, the performance is unsatisfactory when the temporal
lengths of audio and visual clips are similar, since the range of temporal
offset is highly limited. Retargeting videos by temporal acceleration may lead
some parts of the synthetic video to be extremely fast or slow. This is
because when the gap between $I_{k}^{v}$ and $I_{k}^{a}$ is quite large, we
need to interpolate or remove a large number of visual frames, which seriously
affects the viewing fluency. On the contrary, the synthetic video by DTW
achieves excellent rhythm alignment compared with videos generated by temporal
shifting, and the synthetic frames are more consistent than the temporal
acceleration production. We provide more synthetic video demos, including raw
audio and visual clips in the supplementary material.

##  5 Conclusion

In this paper, we propose a novel self-supervised Music-Dance Representation
learning framework via the synchronization of music and dance rhythms both
explicitly and implicitly. We first explicitly extract and align the visual
and auditory rhythms of the dancing videos based on the magnitudes of dancer
motions and music contents. This part of the framework can also be utilized as
a dancing rhythm extractor. We then leverage the contrastive learning strategy
to synchronize the auditory and visual streams of the dancing videos, which
can also be viewed as an implicit music-dance rhythm synchronization. Our
model outperforms other self-supervised methods in downstream tasks by a large
margin, verifying the effectiveness of our framework from both the video
understanding and re-creation perspectives. We hope our work could arouse the
gorgeous growth of works on music-dance self-supervised learning, and we
believe the paradigm and benchmarks we proposed can lead to more meaningful
dance-music researches.

## References

  * [1] Triantafyllos Afouras, Yuki M Asano, Francois Fagan, Andrea Vedaldi, and Florian Metze.  Self-supervised object detection from audio-visual correspondence.  arXiv preprint arXiv:2104.06401, 2021.
  * [2] Triantafyllos Afouras, Andrew Owens, Joon Son Chung, and Andrew Zisserman.  Self-supervised learning of audio-visual objects from video.  In ECCV, 2020.
  * [3] Jean-Baptiste Alayrac, Adrià Recasens, Rosalia Schneider, Relja Arandjelović, Jason Ramapuram, Jeffrey De Fauw, Lucas Smaira, Sander Dieleman, and Andrew Zisserman.  Self-supervised multimodal versatile networks.  arXiv preprint arXiv:2006.16228, 2020.
  * [4] Humam Alwassel, Dhruv Mahajan, Bruno Korbar, Lorenzo Torresani, Bernard Ghanem, and Du Tran.  Self-supervised learning by cross-modal audio-video clustering.  In NeurIPS, volume 33, 2020.
  * [5] Relja Arandjelovic and Andrew Zisserman.  Look, listen and learn.  In ICCV, pages 609–617, 2017.
  * [6] Relja Arandjelovic and Andrew Zisserman.  Objects that sound.  In ECCV, pages 435–451, 2018.
  * [7] Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton.  Layer normalization.  arXiv preprint arXiv:1607.06450, 2016.
  * [8] Zohar Barzelay and Yoav Y Schechner.  Harmony in motion.  In CVPR, pages 1–8, 2007.
  * [9] Donald J Berndt and James Clifford.  Using dynamic time warping to find patterns in time series.  In KDD workshop, volume 10, pages 359–370, 1994.
  * [10] Sebastian Böck and Gerhard Widmer.  Maximum filter vibrato suppression for onset detection.  In DAFx, volume 7, 2013.
  * [11] Daniel Castro, Steven Hickson, Patsorn Sangkloy, Bhavishya Mittal, Sean Dai, James Hays, and Irfan Essa.  Let’s dance: Learning from online dance videos.  arXiv preprint arXiv:1801.07388, 2018.
  * [12] Rizwan Chaudhry, Avinash Ravichandran, Gregory Hager, and René Vidal.  Histograms of oriented optical flow and binet-cauchy kernels on nonlinear dynamical systems for the recognition of human actions.  In CVPR, pages 1932–1939, 2009.
  * [13] Ying Cheng, Ruize Wang, Zhihao Pan, Rui Feng, and Yuejie Zhang.  Look, listen, and attend: Co-attention network for self-supervised audio-visual representation learning.  In ACM MM, pages 3884–3892, 2020.
  * [14] Joon Son Chung and Andrew Zisserman.  Out of time: automated lip sync in the wild.  In ACCV, pages 251–263, 2016.
  * [15] Abe Davis and Maneesh Agrawala.  Visual rhythm and beat.  In CVPRW, pages 2532–2535, 2018.
  * [16] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [17] Chuang Gan, Deng Huang, Hang Zhao, Joshua B Tenenbaum, and Antonio Torralba.  Music gesture for visual sound separation.  In CVPR, pages 10478–10487, 2020.
  * [18] Chuang Gan, Hang Zhao, Peihao Chen, David Cox, and Antonio Torralba.  Self-supervised moving vehicle tracking with stereo sound.  In ICCV, pages 7053–7062, 2019.
  * [19] Xin Guo, Yifan Zhao, and Jia Li.  Danceit: Music-inspired dancing video synthesis.  TIP, 2021.
  * [20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, pages 770–778, 2016.
  * [21] Di Hu, Feiping Nie, and Xuelong Li.  Deep multimodal clustering for unsupervised audiovisual learning.  In CVPR, pages 9248–9257, 2019.
  * [22] Di Hu, Rui Qian, Minyue Jiang, Xiao Tan, Shilei Wen, Errui Ding, Weiyao Lin, and Dejing Dou.  Discriminative sounding objects localization via self-supervised audiovisual matching.  In NeurIPS, volume 33, 2020.
  * [23] Ruozi Huang, Huang Hu, Wei Wu, Kei Sawada, Mi Zhang, and Daxin Jiang.  Dance revolution: Long-term dance generation with music via curriculum learning.  arXiv preprint arXiv:2006.06119, 2020.
  * [24] Zhiheng Huang, Wei Xu, and Kai Yu.  Bidirectional LSTM-CRF models for sequence tagging.  arXiv preprint arXiv:1508.01991, 2015.
  * [25] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [26] Bruno Korbar, Du Tran, and Lorenzo Torresani.  Cooperative learning of audio and video models from self-supervised synchronization.  In NeurIPS, page 7774–7785, Red Hook, NY, USA, 2018.
  * [27] John D. Lafferty, Andrew McCallum, and Fernando C. N. Pereira.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.  In ICML, page 282–289, 2001.
  * [28] Guillaume Lample, Miguel Ballesteros, Sandeep Subramanian, Kazuya Kawakami, and Chris Dyer.  Neural architectures for named entity recognition.  arXiv preprint arXiv:1603.01360, 2016.
  * [29] Hsin-Ying Lee, Xiaodong Yang, Ming-Yu Liu, Ting-Chun Wang, Yu-Ding Lu, Ming-Hsuan Yang, and Jan Kautz.  Dancing to music.  arXiv preprint arXiv:1911.02001, 2019.
  * [30] Hongru Liang, Wenqiang Lei, Paul Yaozhu Chan, Zhenglu Yang, Maosong Sun, and Tat-Seng Chua.  Pirhdy: Learning pitch-, rhythm-, and dynamics-aware embeddings for symbolic music.  In ACM MM, pages 574–582, 2020.
  * [31] Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In ICCV, pages 2980–2988, 2017.
  * [32] Shuang Ma, Zhaoyang Zeng, Daniel McDuff, and Yale Song.  Active contrastive learning of audio-visual video representations.  In ICLR, 2021.
  * [33] Brian McFee, Colin Raffel, Dawen Liang, Daniel PW Ellis, Matt McVicar, Eric Battenberg, and Oriol Nieto.  librosa: Audio and music signal analysis in python.  In python in science conference, volume 8, pages 18–25, 2015.
  * [34] Pedro Morgado, Yi Li, and Nuno Nvasconcelos.  Learning representations from audio-visual spatial alignment.  In NeurIPS, volume 33, 2020.
  * [35] Meinard Müller.  Dynamic time warping.  Information retrieval for music and motion, pages 69–84, 2007.
  * [36] Vinod Nair and Geoffrey E Hinton.  Rectified linear units improve restricted boltzmann machines.  In ICML, 2010.
  * [37] Andrew Owens and Alexei A Efros.  Audio-visual scene analysis with self-supervised multisensory features.  In ECCV, pages 631–648, 2018.
  * [38] Fabrizio Pedersoli and Masataka Goto.  Dance beat tracking from visual information alone.  In Proc. Int. Soc. Music Inf. Retrieval Conf, pages 400–408, 2020\.
  * [39] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  In CVPR, pages 815–823, 2015.
  * [40] Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov.  Dropout: a simple way to prevent neural networks from overfitting.  JMLR, 15(1):1929–1958, 2014.
  * [41] Deqing Sun, Xiaodong Yang, Ming-Yu Liu, and Jan Kautz.  Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume.  In CVPR, pages 8934–8943, 2018.
  * [42] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, pages 5998–6008, 2017.
  * [43] Jianren Wang, Zhaoyuan Fang, and Hang Zhao.  Alignnet: A unifying approach to audio-visual alignment.  In WACV, pages 3309–3317, 2020.
  * [44] Ho-Hsiang Wu, Magdalena Fuentes, and Juan P Bello.  Exploring modality-agnostic representations for music classification.  arXiv preprint arXiv:2106.01149, 2021.
  * [45] Monika Wysoczanska and Tomasz Trzcinski.  Multimodal dance recognition.  In VISIGRAPP, pages 558–565, 2020.
  * [46] Yutong Xie, Haiyang Wang, Yan Hao, and Zihao Xu.  Visual rhythm prediction with feature-aligning network.  In MVA, pages 1–6, 2019.
  * [47] Haoming Xu, Runhao Zeng, Qingyao Wu, Mingkui Tan, and Chuang Gan.  Cross-modal relation-aware networks for audio-visual event localization.  In ACM MM, pages 3893–3901, 2020.
  * [48] Zijie Ye, Haozhe Wu, Jia Jia, Yaohua Bu, Wei Chen, Fanbo Meng, and Yanfeng Wang.  Choreonet: Towards music to dance synthesis with choreographic action unit.  In ACM MM, pages 744–752, 2020.
  * [49] Mingliang Zeng, Xu Tan, Rui Wang, Zeqian Ju, Tao Qin, and Tie-Yan Liu.  MusicBERT: Symbolic music understanding with large-scale pre-training.  arXiv preprint arXiv:2106.05630, 2021.
  * [50] Xinjian Zhang, Yi Xu, Su Yang, Longwen Gao, and Huyang Sun.  Dance generation with style embedding: Learning and transferring latent representations of dance styles.  arXiv preprint arXiv:2104.14802, 2021.
  * [51] Hang Zhao, Chuang Gan, Wei-Chiu Ma, and Antonio Torralba.  The sound of motions.  In ICCV, pages 1735–1744, 2019.
  * [52] Hang Zhao, Chuang Gan, Andrew Rouditchenko, Carl Vondrick, Josh McDermott, and Antonio Torralba.  The sound of pixels.  In ECCV, September 2018.
  * [53] Hongyuan Zhu, Ye Niu, Di Fu, and Hao Wang.  MusicBERT: A self-supervised learning of music representation.  In ACM MM, pages 3955–3963, 2021.
  * [54] Wenlin Zhuang, Congyi Wang, Siyu Xia, Jinxiang Chai, and Yangang Wang.  Music2dance: Dancenet for music-driven dance generation.  arXiv preprint arXiv:2002.03761, 2020.