# Singing beat tracking with Self-supervised front-end and linear transformers

###### Abstract

Tracking beats of singing voices without the presence of musical accompaniment
can find many applications in music production, automatic song arrangement,
and social media interaction. Its main challenge is the lack of strong
rhythmic and harmonic patterns that are important for music rhythmic analysis
in general. Even for human listeners, this can be a challenging task. As a
result, existing music beat tracking systems fail to deliver satisfactory
performance on singing voices. In this paper, we propose singing beat tracking
as a novel task, and propose the first approach to solving this task. Our
approach leverages semantic information of singing voices by employing pre-
trained self-supervised WavLM and DistilHuBERT speech representations as the
front-end and uses a self-attention encoder layer to predict beats. To train
and test the system, we obtain separated singing voices and their beat
annotations using source separation and beat tracking on complete songs,
followed by manual corrections. Experiments on the 741 separated vocal tracks
of the GTZAN dataset show that the proposed system outperforms several state-
of-the-art music beat tracking methods by a large margin in terms of beat
tracking accuracy. Ablation studies also confirm the advantages of pre-trained
self-supervised speech representations over generic spectral features.

##  1 Introduction

Music tempo and beat detection are two of the core and well-defined MIR
topics, and scholars have proposed many approaches to addressing different
aspects of them for various music genres. For instance, some works such as [1,
2, 3, 4] proposed some unsupervised approaches to detect music beat and tempo
by using some low-level features like onset strengths. More recently, several
more recent approaches employ neural networks to address the mentioned tasks
[5, 6, 7, 8]. Extracting singing rhythmic parameters makes it possible to
address several MIR problems such as automatic instrumental and singing tracks
alignment, automatic music mix, and remix, interactive content creation on
social media platforms, etc.

Many of the proposed music rhythmic analysis approaches [1, 5, 7] perform in
an offline fashion while others [6, 9, 10] are capable of extracting music
rhythmic features causally and even in real-time.

In contrast to all of the mentioned models, beat and tempo detection for
singing voice is an untapped MIR task. There are significant inherent
differences between the natures of complete music and singing voices. One of
those differences is that complete music pieces usually contain rich
percussive and harmonic profiles while singing tracks usually lack such
beneficial parameters making their rhythmic analysis more demanding compared
to the former group. Their other important difference is that music beat
tracking models usually only use acoustical clues such as magnitude
spectrogram as input features while singing tracks are more similar to speech
signals where the models may require to deal with para-linguistics, semantic,
and phonemic level inputs in addition to acoustical features.

Our contributions in this paper are as follows:

1- We introduce singing beat tracking as a novel MIR task. We propose two
strategies to tackle the lack of annotated data for this task by leveraging
pre-existing datasets and beat tracking and source separation techniques. We
also propose a new evaluation scheme that can account for phase ambiguities of
beat annotation for vocal music.

2- We propose two neural models for the singing beat tracking task. These
models leverage pre-trained speech self-supervised models to extract feature
embeddings, which are then fed into a linear transformer network to output
beat predictions.

3- We evaluate the proposed models on hundreds of vocal tracks with diverse
genres. Experiments show that the proposed models outperform three
representative baselines designed or trained for general music beat tracking
by a large margin. An ablation study is also performed to investigate the
effect of speech self-supervised models over commonly used generic spectral
features, and results again show an outperformance by a large margin.

The remainder of the paper is organized as follows: In Section 2, we describe
the proposed vocal beat tracking task with dataset curation and a new
evaluation scheme. In Section 3, we describe our proposed models with details.
In Section 4, we present experimental comparisons with baselines and an
ablation study on the feature extraction front end. Finally, Section 5
concludes the paper.

##  2 The Proposed Task

We propose singing beat tracking as a new MIR task, which aims to design
algorithms that can track beats of singing voices in online or offline
fashion. In this section, we describe the two cornerstones for this task:
datasets and evaluation metrics.

###  2.1 Singing Data and Label Generation

Data for training and evaluating singing beat tracking algorithms require both
singing voice recordings and their beat annotations. As there is no existing
datasets that meet this requirement, we need to design a mechanism to collect
such data. Instead of recording singing voices and annotating their beats
manually, we propose to leverage existing MIR datasets and techniques to
collect data for singing beat tracking, as illustrated in Figure 1.

Figure 1: Singing data and label generation pipeline.

The first strategy we propose is to use pre-existing music beat tracking
datasets in which the beat annotations are available. The idea is to obtain
their singing tracks through music source separation. State-of-the-art (SOTA)
singing voice separation methods have been shown to achieve outstanding
results on popular genres of music, and the separated singing voice contains
little interference from the background music. For this task, we employ Demucs
Hybrid [11], one of the superior music source separation models. It uses a
waveform-to-waveform convolutional auto-encoder with a U-Net structure and
bidirectional LSTM layers, and is designed to separate the music mixture into
four sources including bass, drums, vocals, and others. We use the pre-trained
model 111https://github.com/facebookresearch/demucs which was trained on all
150 songs of MusDB [12] dataset. It is noted that source separation is used in
some related works e.g. [13, 14, 15] to improve music beat tracking while in
this work, we utilize it as one of the suggested systematic ways to generate
singing vocal beat tracking datasets.

The second strategy we propose leverages pre-existing music source separation
datasets, in which the isolated vocal tracks are available as a part of those
datasets. To obtain beat annotations, we apply BeatNet [6] offline version on
the complete songs (i.e., music mixtures). BeatNet is a SOTA online beat
tracking method that uses a convolutional recurrent neural network (CRNN) and
a hidden Markov model (HMM) decoder to extract music beats and downbeats. We
modify the HMM decoder from particle filtering inference to Viterbi algorithm
to improve its performance in the offline scenario. The reason that we run
BeatNet on music mixtures instead of the separated vocal tracks is because it,
the same as all other beat tracking methods, is trained on complete music
pieces. While BeatNet has been shown to be fairly accurate on many songs with
different genres, there are still annotation errors. To fix them, we also
perform a manual revision by listening to the separated vocal track together
with synthesized beeping sounds of the beat annotations and correcting errors.

While the beat annotations are obtained for the entire song, the ground-truth
or separated vocal tracks show long chunks of silence due to the inactivity of
singing. These long silent chunks are not useful and even are distracting for
singing rhythmic analysis. Therefore, we normalize the energy of each vocal
track using the root-mean-square (RMS), calculate a frame-wise RMS, and set a
threshold to detect long silent chunks and split the vocal track into vocal
segments. The RMS threshold is set to 0.01, and silent chunks shorter than 8
seconds are kept.

We apply these two strategies to a total of 8 existing beat tracking and music
source separation datasets to obtain a total of 2248 vocal excerpts with beat
annotations. The entire length of the vocal segments is 34h 35m. The datasets
are summarized in Table 1.

Dataset |  # Number of vocal excerpts |  Total Length
---|---|---
Ballroom [16, 17] ∗ |  452 |  2 h 38 m
GTZAN [18, 19] ∗ |  741 |  5 h 44 m
Hainsworth [20]∗ |  154 |  1 h 47 m
MUSDB18 [12]† |  263 |  6 h 21 m
Rock Corpus [21]∗ |  315 |  9 h 23 m
RWC pop [22, 23]∗ |  188 |  5 h 06 m
RWC Royalty free [22, 23]∗ |  29 |  19 m
URSing [24]† |  106 |  3 h 17 m
Table 1: Datasets collected and adapted for singing beat tracking. ${*}$
denotes beat tracking datasets and ${\dagger}$ denotes music separation
datasets.

###  2.2 Evaluation Metrics

For evaluation metrics, we adopt the commonly used F-measure in beat tracking.
A beat is considered correctly detected if it is matched with a ground-truth
beat with a time deviation smaller than 70 ms. In addition, we employ three
additional metrics including P-score, Cemgil and Goto to provide a more
detailed evaluation. Details about these three metrics can be found in [3].

For many music pieces, vocal tracks can align with the offbeat position (i.e.,
middle point between two adjacent ground-truth beats) rather that the beats.
In fact, it also can be quite natural for humans to clap on the 180-degree
phase shifted positions While this inherent ambiguity also exists in some
instrumental music, it is much more common for singing voices. Apparently, the
off-beat predictions would be evaluated poorly against the ground-truth beat
annotations for all of the abovementioned metrics.

To address this problem, we propose an additional Phase Inclusive (PI)
evaluation scheme with existing metrics. In this scheme, a metric is computed
twice, one comparing the predicted beat positions with the ground-truth beat
annotations and the other comparing with the 180-degree shifted ground-truth.
The maximum is reported as the final metric under the PI evaluation scheme.

##  3 The Proposed Method

To address the singing beat tracking task, we propose two models that take
advantage of pre-trained WavLM and DistilHuBERT SSL representations
respectively to extract the input features. Then we build the same linear
multi-head self-attention network on top of them to fine-tune the models for
our task. Finally, a hidden Markov model decoder is used to infer the singing
beat positions using the activations provided by the respective neural
networks. The source code and system demos are
available222https://github.com/mjhydri/singing-vocal-beat-tracking  .Figure 2
demonstrates the overall structure of all proposed models and Figure 3
demonstrates the neural network structures of all proposed models.

Figure 2: General pipeline of the proposed method.

Figure 3: Neural network structures of the proposed models. (I), (II) and
(III) use WavLM, DistilHuBERT and Spectrogram front-ends blocks, respectively,
followed by the same linear transformer network.

###  3.1 Feature Embedding

Neural networks benefit from large quantities of labeled training data.
However, in many cases including that of our task, labeled data is much harder
to obtain than unlabeled data. Self-supervised learning has emerged as a
paradigm to learn general data representations from unlabeled examples and
fine-tuning the model on labeled data [25]. On the other hand, as demonstrated
in [6], selecting appropriate input features plays an important role in music
rhythmic analysis tasks. Due to acoustic and linguistic similarities between
singing voices and speech, our main assumption is that self-supervised speech
representations are helpful as feature embeddings in singing rhythmic
analysis.

SSL achieves great success in speech-related tasks such as Automatic Speech
Recognition, Phoneme Recognition and Emotion Recognition. Each task can be
addressed by fine-tuning a universal pre-trained self-supervised model or
training a neural network on top of the contextualized self-supervised
representation of the input. In recent years, several pre-trained self-
supervised models (e.g. [26, 25, 27, 28, 29, 30]) attempt to provide such
universal speech representations. The most successful models such as [26, 25]
usually encode speech audio through a multi-layer convolutional neural network
and then mask some chunks of the resulting latent speech representations.
Then, a contrastive learning step is performed by feeding the latent
representations to a transformer network to distinguish true latents [25].

WavLM: WavLm [26] is one of the recent universal self-supervised pre-trained
models on 94k hours of data to solve full-stack downstream speech tasks. It
jointly predicts masked speech and performs denoising in the pre-training
stage. The denoising module helps to enhance the potential for non-ASR tasks.
Moreover, it uses gated relative position bias in its transformer structure to
improve capturing sequence ordering of input speech. WavLM has three different
versions, Base, Base+, and Large. WavLM Base and WavLM Base+ include 12
encoder layers, 768-dimensional hidden states, and 8 attention heads. The
WavLM Large contains 24 encoder layers, 1024-dimensional hidden states, and 12
attention heads. According to the SUPERB [31], which is an evaluation
benchmark designed to provide a standard and comprehensive testbed for pre-
trained models on several speech tasks, WavLM outperforms the other
counterparts for several speech downstream tasks.

Since WavLM model has demonstrated promising results for several downstream
speech tasks, we employ it as the front-end model for our first proposed
approach to prepare contextualized input embeddings. Among three pre-trained
options, we chose the Base+ model, becasue of its better performance than the
Base model and similar performance with the Large model on many downstream
tasks.

According to some works on the layer-wise analysis of self-supervised models
(e.g. [32, 33]), using their last layer’s output is not necessarily the best
option to leverage the SSL model’s maximum capability. This is because the
desired feature aspects of the input signal are distributed differently
through internal encoder layers. For example, [33] demonstrated that in
Wav2Vec2 self-supervised model, for some properties like word meaning,
intermediate layers play a much more important role than the output layer,
while for acoustical features, early layers carry the most important
information. Therefore, in order to capture different aspects of the input
signal, we use a weighted sum of all encoder layers of WavLM in which each
weight is a learnable parameter during the training phase.

DistilHuBERT: Although WavLM demonstrated the most successful results for
several downstream tasks, it is a large model which requires large memory and
high pre-training and inference costs imposing a speed bottleneck for the
system. DistilHuBERT [29], is a novel multi-task learning framework to distill
hidden representations from a HuBERT [27] model directly. It is initialized
with the HuBERT’s parameters. Then, its prediction heads learn to generate the
teacher’s hidden representations by minimizing the loss function. DistilHuBERT
reduces the model size by 75% and increases the inference speed by 73% from
HuBERT while retaining most performance in ten different speech tasks.
Moreover, it requires little training time and data, opening the possibilities
of pre-training personal and on-device SSL models for speech.

Therefore, we propose a second model that switches the heavy WavLM front-end
model with DistilHuBERT, a much lighter model, leading to a much faster
inference process. DistilHuBERT includes a CNN network and two self-attention
layers. According to its evaluation results [29], DistilHuBERT delivers
comparable performance to the teacher HuBERT model with much less
computational cost. It is noted that as the training objective of the
DistilHuBERT student model is to learn directly from several internal encoder
layers of the teacher model, the weighted sum strategy is not needed for this
model and its last layer’s output is used as the feature embeddings.

Spectrogram: We also perform an ablation study to evaluate the effect of using
self-supervised models on the system performance by replacing the self-
supervised pre-trained feature embeddings with spectral features that are
commonly used in SOTA music beat tracking methods [7, 6]. These spectral
features include log-magnitude mel-frequency spectrogram with three window
sizes of 1024, 2048, and 4096 samples and their first-order time differences.

###  3.2 Linear Transformer Network

Transformers achieve remarkable performance in many tasks, but due to their
quadratic complexity with respect to the input length, they are prohibitively
slow for long sequences [34]. To address this limitation and to improve
transformers’ computational cost, several approaches are proposed. For
instance, Linformer [35] is a model that reduces the transformer complexity in
both time and space from $O(N^{2})$ to $O(N)$ by approximating the self-
attention mechanism by a low-rank matrix. It is noted that $N$ is the sequence
length. Another model [34]reduces the complexity to linear by expressing the
self-attention as a linear dot-product of kernel feature maps and making use
of the associativity property of matrix products. It achieves similar
performance to vanilla transformers and they are up to 4000x faster on auto-
regressive prediction of very long sequences. Another recent model [36]
removes the softmax in self-attention by using the Gaussian kernel function to
replace the dot-product similarity without further normalization. It enables a
full self-attention matrix to be approximated via low-rank matrix
decomposition.

In our proposed method, we build a linear multi-head self-attention layer
using [34] on top of the feature embedding blocks for all three proposed
models.

Our encoder comprises a self-attention layer with four attention heads with
query and value dimensions of 192 and a feed-forward network with the size of
1024. The encoder network is followed by a linear layer and a sigmoid layer.
The final output can be viewed as the beat salience for the input audio frame.

To train the proposed models, we use binary cross-entropy with logits between
model output and the ground-truth beat annotations as the loss function. Only
the linear transformer and its following layers are trained, while the self-
supervised front ends are pre-trained and fixed.

##  4 Post-Processing

In order to decode the beat positions, we employed a DBN approximated using
HMM from Madmom library [37] with the default parameters. It is noted that the
mentioned inference block is shared between all reported models in the
evaluation table except BeatRoot [2].

##  5 Experiments

###  5.1 Experimental Setup

We use all of the datasets collected in Section 2 except GTZAN for training
the models with 80% of the songs randomly allocated for training and 20% for
validation. Following several previous music beat tracking works, we keep the
entire GTZAN dataset as the test set. It contains 1000 music excerpts covering
10 different music genres, out of which 741 of them contain vocal tracks. It
ensures genre diversity in the test set and such a training/test dataset
partition reduces the possibility of overfitting as well.Moreover, it is
unseen in the training blocks of all compared methods. However, one drawback
of it is that GTZAN’s vocal tracks are separated from the whole songs. Hence,
they may contain some information leakage from other musical instruments,
which may boost the system performance compared to testing on isolated vocal
tracks. To address this issue, we took the following steps. First, We used one
of the best-performing supervised source separation models [11] and listened
through the separated vocal tracks to ensure that they do not contain obvious
music signal leakage. Second, when tuning the RMS filter, we discarded some
sparse, abrupt and short signal segments that are more likely to be percussive
sounds instead of vocal signals. Third, we compared with several high
performing music beat tracking methods in the evaluation section; The leakage
of musical signals into the separated tracks, if any, would improve the
performance of these comparison models as well.

For the mentioned reason and given that singing beat tracking is a novel MIR
task with no specifically design existing models, in this paper, we compare
our proposed models against general music beat tracking models including two
supervised models, BeatNet[6] and Böck [7] and an unsupervised signal-
processing model, BeatRoot [2]. BeatNet is the SOTA online joint beat,
downbeat, and meter tracking model that uses convolutional recurrent neural
networks and particle filtering. It can also operate in an offline fashion by
switching its particle filtering inference block with an offline hidden Markov
model decoder. Böck uses a recurrent neural network to predict beat and
downbeat saliences of each audio frame, and uses a Dynamic Bayesian Network
(DBN) to infer beat and downbeat positions from the salience function. In our
implementation, we use the same HMM decoder as that in BeatNet to replace the
DBN. BeatRoot is a unsupervised method with a multiple agent architecture that
simultaneously considers several different hypotheses of beat positions and
tempi.

###  5.2 Training Details

To train the proposed models, we freeze the self-supervised front-end blocks
and train the rest of the model including the multi-head self-attention layer
and its following feed-forward and linear layers. For the WavLM+LT model, in
addition to the abovementioned items, the 12 weights to combine different
encoder layer’s output are also trained.

The mentioned weights are initialized as ones and the rest of the weights and
biases are initialized randomly. We use the Adam optimizer with a learning
rate of $5\times 10^{-5}$ and a batch size of 10. The batches comprise
15-second long excerpts randomly sampled from all audio files in the training
set. As the audio files have different length, to ensure a fair distribution,
we sample with a probability that is proportional to the length of the files.

###  5.3 Results and Discussions

To assess the computational cost of the models, we report the inference speed
for all of them. The reported numbers are the average computational time for
all 741 excerpts with the average duration of 28 seconds each. Note that we
measure the speed of all methods using CPU processing on the same Windows
machine with an AMD Ryzen 9 3900X CPU and 3.80 GHz clock. Table 2 demonstrates
the evaluation results.

| Method | F-Measure | P-Score | Cemgil | Goto | Comp. Time
---|---|---|---|---|---|---
Baseline | BeatNet | 0.243 | 0.327 | 0.173 | 0.003 | 0.13 (s)
BeatRoot | 0.301 | 0.394 | 0.22 | 0.066 | 0.03 (s)
Böck | 0.171 | 0.195 | 0.122 | 0.009 | 1.56 (s)
Proposed | WavLM + LT | 0.733 | 0.704 | 0.618 | 0.560 | 4.09 (s)
DistilHuBERT + LT | 0.703 | 0.668 | 0.593 | 0.516 | 1.83 (s)
Spectrogram + LT | 0.454 | 0.438 | 0.367 | 0.223 | 0.32 (s)
Proposed (PI Results) | WavLM + LT | 0.745 | 0.715 | 0.627 | 0.574 | 4.09 (s)
DistilHuBERT + LT | 0.721 | 0.684 | 0.608 | 0.537 | 1.83 (s)
Spectrogram + LT | 0.489 | 0.477 | 0.391 | 0.265 | 0.32 (s)
Table 2: Average performance and speed across segments of several methods on
the GTZAN separated vocal tracks.

Table 2 shows the evaluation results of the baselines and the proposed models
using the four metrics presented in Section 2.2. It also includes the Phase
Inclusive (PI) evaluation scheme for the proposed models. Several interesting
observations can be made. First, all of the three proposed models outperform
the three baselines by a large margin. This difference is especially
pronounced for more strict metrics such as Goto [3]. This confirms our
hypothesis that vocal tracks show very different patterns from complete songs,
and the baseline models that are trained on complete songs do not perform as
well as the proposed models that are trained on the vocal tracks. This
hypothesis is further validated by the fact that BeatRoot outperforms BeatNet
and Böck, while the latter two models are shown to outperform BeatRoot in
previous studies on complete songs [7, 6]. It is noted that BeatNet and Böck
are data-driven approaches and are trained on complete songs, while BeatRoot
is a signal processing model without training. The mismatch between complete
songs and vocal tracks does impose a strong negative bias to the data-driven
baselines.

Second, the two proposed models that leverage speech SSL feature embeddings
(i.e., WavLM+LT and DistilHuBERT+LT) improves over Spectrogram+LT by a large
margin, and this improvement is even bigger than that from the best baseline
to Spectrogram+LT. This shows the significant advantage of using feature
embeddings learned on speech data and suggests that this advantage is even
more significant than training on the vocal tracks. As the spectrogram
features used in Spectrogram+LT are the same as those in BeatNet and Böck and
transformers have been shown to outperform RNNs in various tasks, it is
reasonable to believe that even if BeatNet and Böck are trained on the vocal
tracks, their performance would be only comparable to that of Spectrogram+LT.

Third, WavLM+LT delivers the best scores for all of the evaluation metrics
except the computation time, showing that stronger feature embeddings lead to
better vocal beat tracking accuracy. However, the attention layers’ complexity
in WavLM is quadratic to the length of input; while the average computational
time (4.09 seconds) is acceptable for vocal segments ( 28 s long on average)
in this experiments, as the length increases, the computational time will soon
become prohibitive. Figure 4 shows the learned weights of different encoder
layers of the WavLM SSL model. It can be seen that earlier layers generally
contribute more to forming the feature embeddings compared to latter layers.
Layers 3, 2 and 1 contribute the most. Note that Layer 0 represents the CNN
network output.

$0$$1$$2$$3$$4$$5$$6$$7$$8$$9$$10$$11$$12$Transformer layer numberLearned
layer weights Figure 4: Illustration of the learned weights for different
encoder layers of WavLM in the WavLM+LT model.

Fourth, with slight performance drop, the second proposed model which employs
LT and DistilHubert delivers the next best performance among all models.
DistilHubert uses a much lighter structure with fewer parameters than WavLM,
making it ideal for time-sensitive inferences.

Finally, the PI evaluation scheme shows a slight improvement of all of the
proposed models on all accuracy metrics. In particular, the improvement of
Spectrogram+LT is greater than that of the other models. This suggests that
the SSL pretrained speech embedding features are helpful to reduce the phase
shift ambiguity.

Figure 5 shows the performance of our proposed WavLM + LT model for different
music genres. According to the table, the best and worst performances belong
to _disco_ and _blues_ respectively. One reason for that may be the stronger
and punchier stresses in _disco_ vocal tracks comparing to that of _blues_
tracks. Another important reason may be the different syncopation ratio among
vocal track of different music genres. Syncopation is a musical term that
defines the placement of rhythmic stresses or accents where they would not
normally occur, making part or all of a tune or piece of music off-beat [38].
Therefore, a higher syncopation ratio leads to a more difficult beat tracking
process. Nate that in Figure 5 we only report the evaluation results of the
genres that contain at least 80 separate singing tracks. It leaves out the
_classical_ and _jazz_ genres which have only two and zero separated tracks.

BluesCountryDiscoHip-hopMetalPopReggaeRock$0$$0.2$$0.4$$0.6$$0.8$$1$F-Measure
Figure 5: F-measure performance of WavLM + LT model on the GTZAN separated
vocal tracks for different genres.

##  6 CONCLUSION

In this paper we introduced singing beat tracking as a novel MIR task. We
proposed two approaches to obtain labeled data for the mentioned task and
introduced two methods to accomplish the mentioned task using pre-trained
speech self-supervised models and multi-head linear self-attention networks.
We also conducted an ablation study to investigate the effect of speech self-
supervised models on the system performance. Experiments on a collection of
vocal tracks with diverse genres show that the proposed models that combine
self-supervised models and transformers outperform three representative
baselines designed or trained for beat tracking for general music. The
ablation study also shows that the self-supervised speech embeddings
outperform generic spectral features that are commonly used in music beat
tracking. In the future we will extend this research to cover real-time
applications and more challenging singing tracks such as human humming and
operas.

##  7 ACKNOWLEDGEMENT

This work was supported by National Science Foundation grant No. 1846184.

## References

  * [1] D. P. Ellis, “Beat tracking by dynamic programming,” _Journal of New Music Research_ , vol. 36, no. 1, pp. 51–60, 2007.
  * [2] S. Dixon, “Evaluation of the audio beat tracking system beatroot,” _Journal of New Music Research_ , vol. 36, no. 1, pp. 39–50, 2007.
  * [3] M. E. Davies, N. Degara, and M. D. Plumbley, “Evaluation methods for musical audio beat tracking algorithms,” _Queen Mary University of London, Centre for Digital Music, Tech. Rep. C4DM-TR-09-06_ , 2009.
  * [4] A. Mottaghi, K. Behdin, A. Esmaeili, M. Heydari, and F. Marvasti, “Obtain: Real-time beat tracking in audio signals index terms—onset strength signal, tempo estimation, beat onset, cumulative beat strength signal, peak detection,” _International Journal of Signal Processing Systems_ , pp. 123–129, 2017.
  * [5] S. Böck and M. E. Davies, “Deconstruct, analyse, reconstruct: How to improve tempo, beat, and downbeat estimation,” in _Proceedings of the 21st International Society for Music Information Retrieval Conference (ISMIR), Montreal, QC, Canada_ , 2020, pp. 12–16.
  * [6] M. Heydari, F. Cwitkowitz, and Z. Duan, “Beatnet: Crnn and particle filtering for online joint beat downbeat and meter tracking,” 2021.
  * [7] S.Böck, F. Krebs, and G. Widmer, “Joint beat and downbeat tracking with recurrent neural networks,” in _In Proc. of the 7th Intl. Conf. on Music Information Retrieval (ISMIR)_ , 2016.
  * [8] M. Heydari and Z. Duan, “Don’t look back: An online beat tracking method using RNN and enhanced particle filtering,” in _In Proc. IEEE Int. Conf. Acoust. Speech Signal Process. (ICASSP)_ , 2021.
  * [9] J. L. Oliveira, F. Gouyon, L. G. Martins, and L. P. Reis, “Ibt: A real-time tempo and beat tracking system.” in _ISMIR_ , 2010, pp. 291–296.
  * [10] M. Heydari, M. McCallum, A. Ehmann, and Z. Duan, “A novel 1d state space for efficient music rhythmic analysis,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 421–425.
  * [11] A. Défossez, “Hybrid spectrogram and waveform source separation,” in _Proceedings of the ISMIR 2021 Workshop on Music Source Separation_ , 2021\.
  * [12] Z. Rafii, A. Liutkus, F.-R. Stöter, S. I. Mimilakis, and R. Bittner, “The MUSDB18 corpus for music separation,” Dec. 2017. [Online]. Available: https://doi.org/10.5281/zenodo.1117372
  * [13] D. Desblancs, R. Hennequin, and V. Lostanlen, “Zero-note samba: Self-supervised beat tracking,” 2022.
  * [14] C.-Y. Chiu, J. Ching, W.-Y. Hsiao, Y.-H. Chen, A. W.-Y. Su, and Y.-H. Yang, “Source separation-based data augmentation for improved joint beat and downbeat tracking,” in _2021 29th European Signal Processing Conference (EUSIPCO)_. IEEE, 2021, pp. 391–395.
  * [15] A. Gkiokas, V. Katsouros, G. Carayannis, and T. Stajylakis, “Music tempo estimation and beat tracking by applying source separation and metrical relations,” in _2012 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2012, pp. 421–424.
  * [16] F. Gouyon, A. P. Klapuri, S. Dixon, M. Alonso, G. Tzanetakis, C. Uhle, and P. Cano., “An experimental comparison of audio tempo induction algorithms,” _IEEE Transactions on Audio, Speech, and Language Processing_ , vol. 14, no. 5, 2006.
  * [17] F. Krebs, S. Böck, and G. Widmer., “Rhythmic pattern modeling for beat and downbeat tracking in musical audio,” in _Proc. of the 14th Int. Society for Music Information Retrieval Conf._ , 2013.
  * [18] U. Marchand and G. Peeters, “Swing ratio estimation,” in _In Proc. of the 18th Int. Conference on Digital Audio Effects (DAFx)_ , 2015.
  * [19] G. Tzanetakis and P. Cook, “Musical genre classification of audio signals,” _IEEE Transactions on Speech and Audio Processing,_ , vol. 10, no. 5, 2002\.
  * [20] S. W. Hainsworth and M. D. Macleod, “Particle filtering applied to musical tempo tracking,” _EURASIP Journal on Advances in Signal Processing_ , vol. 2004, no. 15, pp. 1–11, 2004.
  * [21] T. de Clercq and D. Temperley., “A corpus analysis of rock harmony,” _Popular Music_ , vol. 30, no. 1, pp. 47–70, 2011.
  * [22] M. Goto, H. Hashiguchi, T. Nishimura, and R. Oka, “Rwc music database: Popular, classical and jazz music databases.” in _Ismir_ , vol. 2, 2002, pp. 287–288.
  * [23] M. Goto _et al._ , “Development of the rwc music database,” in _Proceedings of the 18th international congress on acoustics (ICA 2004)_ , vol. 1, 2004, pp. 553–556.
  * [24] B. Li, Y. Wang, and Z. Duan, “Audiovisual singing voice separation,” _arXiv preprint arXiv:2107.00231_ , 2021.
  * [25] A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A framework for self-supervised learning of speech representations,” _Advances in Neural Information Processing Systems_ , vol. 33, pp. 12 449–12 460, 2020.
  * [26] S. Chen, C. Wang, Z. Chen, Y. Wu, S. Liu, Z. Chen, J. Li, N. Kanda, T. Yoshioka, X. Xiao _et al._ , “Wavlm: Large-scale self-supervised pre-training for full stack speech processing,” _arXiv preprint arXiv:2110.13900_ , 2021.
  * [27] W.-N. Hsu, B. Bolte, Y.-H. H. Tsai, K. Lakhotia, R. Salakhutdinov, and A. Mohamed, “Hubert: Self-supervised speech representation learning by masked prediction of hidden units,” _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , vol. 29, pp. 3451–3460, 2021.
  * [28] P. Peng and D. Harwath, “Fast-slow transformer for visually grounding speech,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 7727–7731.
  * [29] H.-J. Chang, S.-w. Yang, and H.-y. Lee, “Distilhubert: Speech representation learning by layer-wise distillation of hidden-unit bert,” in _ICASSP 2022-2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2022, pp. 7087–7091.
  * [30] S. Ling and Y. Liu, “Decoar 2.0: Deep contextualized acoustic representations with vector quantization,” _arXiv preprint arXiv:2012.06659_ , 2020.
  * [31] S.-w. Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin _et al._ , “Superb: Speech processing universal performance benchmark,” _arXiv preprint arXiv:2105.01051_ , 2021.
  * [32] L. V. Prasad, A. Seth, S. Ghosh, and S. Umesh, “Analyzing the factors affecting usefulness of self-supervised pre-trained representations for speech recognition,” _arXiv preprint arXiv:2203.16973_ , 2022.
  * [33] A. Pasad, J.-C. Chou, and K. Livescu, “Layer-wise analysis of a self-supervised speech representation model,” _arXiv preprint arXiv:2107.04734_ , 2021.
  * [34] A. Katharopoulos, A. Vyas, N. Pappas, and F. Fleuret, “Transformers are rnns: Fast autoregressive transformers with linear attention,” in _International Conference on Machine Learning_. PMLR, 2020, pp. 5156–5165.
  * [35] S. Wang, B. Z. Li, M. Khabsa, H. Fang, and H. Ma, “Linformer: Self-attention with linear complexity,” _arXiv preprint arXiv:2006.04768_ , 2020.
  * [36] J. Lu, J. Yao, J. Zhang, X. Zhu, H. Xu, W. Gao, C. Xu, T. Xiang, and L. Zhang, “Soft: Softmax-free transformer with linear complexity,” _Advances in Neural Information Processing Systems_ , vol. 34, 2021.
  * [37] S. Böck, F. Korzeniowski, J. Schlüter, F. Krebs, and G. Widmer, “madmom: a new Python Audio and Music Signal Processing Library,” in _Proceedings of the 24th ACM International Conference on Multimedia_ , Amsterdam, The Netherlands, 10 2016, pp. 1174–1178.
  * [38] M. Hoffman, “Syncopation,” _National Symphony Orchestra. NPR. Retrieved_ , vol. 13, 2009.


# Tracking Fast by Learning Slow: An Event-based Speed Adaptive Hand Tracker
Leveraging Knowledge in RGB Domain

Chuanlin Lan, Ziyuan Yin, Arindam Basu, Rosa H. M. Chan
Department of Electrical Engineering
City University of Hong Kong
Hong Kong SAR
{chuanlin.lan<EMAIL_ADDRESS>
{arinbasu<EMAIL_ADDRESS>

###### Abstract

3D hand tracking methods based on monocular RGB videos are easily affected by
motion blur, while event camera, a sensor with high temporal resolution and
dynamic range, is naturally suitable for this task with sparse output and low
power consumption. However, obtaining 3D annotations of fast-moving hands is
difficult for constructing event-based hand-tracking datasets. In this paper,
we provided an event-based speed adaptive hand tracker (ESAHT) to solve the
hand tracking problem based on event camera. We enabled a CNN model trained on
a hand tracking dataset with slow motion, which enabled the model to leverage
the knowledge of RGB-based hand tracking solutions, to work on fast hand
tracking tasks. To realize our solution, we constructed the first 3D hand
tracking dataset captured by an event camera in a real-world environment,
figured out two data augment methods to narrow the domain gap between slow and
fast motion data, developed a speed adaptive event stream segmentation method
to handle hand movements in different moving speeds, and introduced a new
event-to-frame representation method adaptive to event streams with different
lengths. Experiments showed that our solution outperformed RGB-based as well
as previous event-based solutions in fast hand tracking tasks, and our codes
and dataset will be publicly available.

##  1 Introduction

Hand tracking, or hand pose estimation, is a critical topic in the realization
of touchless gesture-based human-computer interaction (HCI). With the
continuous development of better deep-learning models, most of the current
hand tracking algorithms are based on frames recorded by RGB or depth cameras
[1] [2] [3]. However, due to the limitations of this hardware with low
temporal resolution (usually 30-60 fps), captured images of hands in rapid
motion are blurred and limit the performance of hand tracking algorithms.

Figure 1: Our fast hand tracking solution is based on the idea tracking fast
by learning from slow to sidestep the difficulty in obtaining 3D annotations
of hands in fast motion. We applied 2 RGB cameras and 1 event camera to record
slow hand motion as trainset, trained an event student network with
supervising information from RGB-based teacher method, and enabled the event
student network to solve speed adaptive hand tracking task in the prediction
procedure. To narrow the gap between the trainset of slow motion and testest
of fast motion, we figured out two data augment methods, a speed adaptive
event stream segmentation method and a event-to-frame representation method.

One previous possible solution considers RGBD cameras with high frequency.
However, RGB cameras with high temporal resolution usually require a high
illumination environment, limiting their application scenarios. The large-
scale data flow of such RGB cameras also challenges data transmission,
storage, and computation system. Depth cameras based on binocular vision have
a similar problem, while time-of-flight (ToF) depth sensors with a high frame
rate are not available. Another possible solution applies additional data
generated by other high-frequency sensors. [4] proposed a multi-modality
method, fusing 100Hz gyroscope data and 30Hz RGBD stream, but the accuracy
enhancement brought by gyroscope data was limited. This proposed method is
relatively impractical because it also requires extra sensor(s) fixed on hands
as well as calibration steps before starting estimating hand pose.

In this work, we utilize an event camera to track hands at different motion
speeds. Compared to normal RGB cameras recording frames in a fixed frequency,
event cameras could respond to local brightness changes in an asynchronous
way, enabling both high temporal resolution and high dynamic range, which
makes it optimal hardware to track hands in diverse real-world environments.
EventHands [5] is a pioneer learning-based method to track hand motion in
event streams, which constructed a synthetic dataset, developed representation
method to transfer event streams to images, and trained a CNN consisting of
ResNet18 and MANO model to solve the problem. However, the domain gap between
real and synthetic datasets caused a performance drop when validating the
model on real data. Consequently, a dataset collected in the real environment
is of vital importance for event hand tracking.

However, obtaining the 3D annotation is a challenging task. Annotation methods
like data glove and VICON system would change the appearance of hands, affect
the captured event stream and thus are not suitable for this problem. Another
opiton is to apply an RGB binocular stereo vision system to get the 3D label
of streams recorded by an event camera, which allow us to utilize the
knowledge of developed RGB-based method, but give rise to the problem raised
at the beginning: how to track hands in fast motion via RGB camera with
limited temporal resolution?

As shown in Fig. 1, we sidestepped this problem by applying a deep-learning
model trained on a hand tracking dataset with slow motion, which enabled the
event hand tracker to leverage the knowledge of RGB-based hand tracking
sloution, to work on fast hand tracking tasks. To narrow the domain gap
between slow and fast motion data, 1) in the training procedure we augmented
the slow motion data by utilizing event stream of different length to generate
event frame as well as randomly suppress the noise in the frame; 2) in the
predicting procedure, we figured out a speed adaptive segment method to ensure
similar motion range in event frame; 3) we figured out a new representation
method, Locally Normalized Event Count and Surface (LNECS), which could
preserve the time information and reduce the impact of different noise
distribution of event stream of varying length. It is worth noting that the
above methodology can be applied not only to fast hand tracking tasks but also
to other fast-moving object detection or tracking tasks, especially when the
annotations of the fast-moving object are impossible or expensive to obtain.
We also construct the first event-based hand tracking dataset collected in
real environment to train and test our solution. Experiments showed that our
solution outperformed RGB-based solutions in fast hand tracking tasks. In
summary, our contributions are:

  * •

We created an Event-based Speed Adaptive Hand Tracker, a solution that uses a
deep learning model trained on a hand tracking dataset with slow motion to
perform fast hand tracking tasks. This approach allows us to utilize the
knowledge from RGB-based hand tracking solutions for event-based fast hand
tracking. The same methodology can also be applied to other fast-moving object
detection or tracking tasks.

  * •

We figured out three methods to reduce the domain gap between the training
data for slow motion and the prediction data for fast motion. These methods
include data augmentation techniques, the speed adaptive segment method, and
the LNECS representation method.

  * •

We constructed and released the first event-based hand tracking dataset
collected in the real world.

Figure 2: (a): We applied 2 RGB cameras and 1 event camera to collected the
dataset with 4 different illumination conditions: without extra light, with
extra light from the front/bottom/left side of the hand; (b): The dataset was
collected under 6 different indoor backgrounds.We recorded hand with and
without an electronic bracelet performing American Sign Language and some
random poses to ensure variety of the dataset.

##  2 Related Work

###  2.1 3D Hand Tracking Methods

Most of the existing 3D hand tracking methods were based on RGBD frames, which
could be divided into two categories: discriminative model and generative
model. Discriminative models estimated hand pose directly from the input
frame. To enhance tracking performance, feature fusion operations were widely
conducted. Region ensemble network (REN) [6] divided the feature map output by
a CNN into four parts and fused the features to compute the final result.
TriHorn-Net [7] first computed the 2D heat map and then fused the 2D feature
map, attention map, and depth features to regress the depth value of each
joint. Generative models first learned the distributions of the dataset and
applied the learned distribution to train a discriminative model. Generalized
feedback loop [8] applied a synthesizer CNN to learn the data distribution and
to generate a depth image from the initially predicted hand pose and then
utilized an updater CNN to compare the input image and generated image to
refine the initial hand pose. All the above-mentioned methods are based on
RGBD frames without motion blur, thus lacking the ability to deal with fast-
moving hand tracking.

Due to the differences between the data structures of the event stream and the
RGBD frame, the above-mentioned models cannot take the event stream as input
directly. Some methods could transfer the event stream to gray frames [9]
[10], and SLAM methods could reconstruct depth frames or point clouds from the
event stream [11]. However, the performances of these models could be affected
by the domain gap. Extra consumption of bandwidth, computation, and storage
without a custom solution for event data would eliminate the advantages of
event cameras in real-time processing.

###  2.2 Event-based Hand Tracking

There are several ways to utilize the event stream recorded by the event
camera, such as 1) reconstruction of gray images from the event stream, which
gives up the temporal information and will cause a huge burden in bandwidth,
storage, and computation; 2) conversion of event stream to point cloud, which
is not easily understood by the neural network; and 3) event-to-frame
representation, which reserves the sparse representation and was adopted in
this paper.

Event-to-frame representation includes event occurrence image (EOI), event
count image (ECI) [12] [13], surfaces of active events (SAE) [14], time-
surfaces [15], hierarchy of time surfaces [15], locally-normalized event
surfaces (LNES) [5]. The representations adopted in this paper LNECWS was
related to ECI and LNES, enabling the model to distinguish hand motion from
background noise.

Existing event-based hand tracking methods could be divided into two groups:
1) unsupervised methods like non-rigid 3D tracking, and 2) supervised methods
based on synthetic datasets like EventHands [5]. The former method [16]
deformed a non-rigid model to the desired shape fitting the stuff in the event
stream, thus needing accurate initialization and complex computation process,
and facing the problem of lacking robustness. The latter reported method
employed a synthetic dataset, which was easy to construct compared to the
real-world dataset, to train the model, but suffered performance drop in real-
world applications due to the distribution gap between real and synthetic
datasets.

##  3 Our Solution

To skip annotations for fast hand tracking dataset, the main idea of our
solution is to employ a neural networks trained on a slow motion dataset,
which allow us to leverage the knowledge of developed RGB-based hand tracking
method, to work on fast hand tracking task. The primary challenge of this
solution is bridging the domain gap between the training data for slow motion
and the prediction data for fast motion. For event segments with the same time
length, the motion range in fast and slow hand tracking dataset is different,
so a speed adaptive event segment method is needed. In other words, for event
segments with similar motion ranges, the time length of event segments in slow
and fast hand tracking data will be different, which lead to different noise
level as the number of noise events is proportional to the time length. To
overcome the domain gap of motion range, we augmented the slow motion data by
utilizing event stream of different length to generate event frame and figured
out the speed adaptive segment method; to process segments with different time
length and noise level, we randomly suppressed the noise in segments of slow
motion trainset and created a new representation method, Locally Normalized
Event Count and Surface (LNECS).

In this section, we present the details of our real-world dataset (Sec. 3.1),
as well as the new representation method for handling event segments of
varying lengths and noise levels (Sec. 3.2). We also demonstrate the methods
we used to augment the slow motion data and incorporate knowledge from the RGB
domain into our training process (Sec. 3.3) and describe how the trained model
track hands at various motion speeds (Sec. 3.4).

###  3.1 Dataset

As shown in Fig. 2, we have considered the following scenarios which are
common in real-life hand tracking: 1) Background: We collected the dataset
with 6 different indoor backgrounds, which vary from simple to complex. 2)
Illumination: Due to the character of the event camera that responds to
changes in brightness, different illumination conditions could lead to
different outputs of the event camera. We considered 4 illumination conditions
while collecting the dataset, which include normal illumination conditions and
illumination conditions with extra light from the front/bottom/left side of
the hand. Additionally, different illumination conditions would lead to
different shadows on the background, which is also challenging for hand
tracking algorithms. 3) Self-occlusion and global rotation: Since self-
occlusion and global rotation are two major challenges in hand tracking, for
all the event streams, we captured poses in American Sign Language (ASL) as
well as some random poses to ensure the dataset contains challenging poses
with self-occlusion. We also rotated the hand to capture different aspects of
each pose. 4) Hand decorations: The decorations on the hand could also affect
the performance of hand tracking algorithms. In our dataset, we collected
hands with or without an electronic bracelet.

We collected the dataset using an event camera (CeleX5) and two RGB cameras.
To ensure the consistency of the data, we calibrated the cameras and matched
the timestamp of the event stream and two RGB videos The dataset with slow
hand motion contains 40 event streams (5 backgrounds $\times$ 4 illumination
conditions $\times$ 2 with or without a bracelet), and dataset with fast hand
motion contains 4 event streams (1 background $\times$ 2 illumination
conditions $\times$ 2 with or without a bracelet). In total, we recorded about
3.5 hours of event data, and the dataset will be publicly available.

###  3.2 Representation

The output of the event camera is a set of events, which could be denoted as
$\mathcal{E}=\\{e_{i}\\}_{i=0}^{N},e_{i}=(x_{i},y_{i},p_{i},t_{i})$, where
$e_{i}$ denotes the event, $x_{i}$ and $y_{i}$ denotes the coordinates of the
event, and $p_{i}$ denotes the polarity, and $t_{i}$ denotes the timestamp.
This data format is not suitable for convolutional neural networks, and thus
we need to transfer them to frame format. Existing event-to-frame
representation methods, such as Event Occur Image (EOI) and Event Count Image
(ECI), detected the occurrence or counted the number of the events at each
pixel within a time interval, which lost the time information, and some
improved methods like, Surfaces of Active Events (SAE) and Locally-Normalized
Event Surface (LNES) assigned the value of each pixel as the timestamp of the
latest event at the pixel, which, however, are easily affected by noise,
especially caused by strobe flash of alternate-current-powered illumination.
To solve such problem, we formulated the Locally-Normalized Event Count and
Surface (LNECS) based on ECI and LNES. The representation LNECS is denoted
with the following formulas:

${LNES}(x,y,p)=\frac{max\\{t_{i}|(x_{i},y_{i},p_{i})=(x,y,p)\\}-min\\{t_{i}\\}}{max\\{t_{i}\\}-min\\{t_{i}\\}}$
${EC}(x,y,p)=|\\{e_{i}|(x_{i},y_{i},p_{i})=(x,y,p)\\}|$
${LNEC}(x,y,p)=\frac{{EC}(x,y,p)}{max({EC}(x,y,p))}$

Then ${LNECS}$ and ${LNECWS}$ can be represented as:

${LNECS}={LNES}\oplus{LNEC}$

where $\oplus$ denotes concatenate operation in the polarity channel.

Figure 3 shows a demo of different representations of event streams of a hand
moving from left to right. A shown in the upper and middle figure, LNES
preserves time information but suffers from noise close to hand and LNEC
overcomes the influence of noise but loses time information. Our proposed
LNECS combined these representations and integrated both advantages

Figure 3: A demo of different representations with two channels. Upper: LNES; middle: LNEC; bottom: LNECWS.  Segment Stardard | Data Augment | Representation | Fast Motion | Slow Motion
---|---|---|---|---
AUCp 2D | AUCp 3D | AUCp 2D | AUCp 3D
10k event number | No | LNECS | 0.780 | 0.661 | 0.908 | 0.842
10k event number | Yes | LNES | 0.831 | 0.716 | 0.890 | 0.821
10k event number | Yes | LNEC | 0.837 | 0.717 | 0.897 | 0.828
10k event number | Yes | LNECWS | 0.836 | 0.722 | 0.891 | 0.822
1k pixel number | Yes | LNECS | 0.789 | 0.664 | 0.904 | 0.839
2k pixel number | Yes | LNECS | 0.801 | 0.674 | 0.912 | 0.843
5k pixel number | Yes | LNECS | 0.759 | 0.628 | 0.910 | 0.841
50ms time length | Yes | LNECS | 0.255 | 0.131 | 0.905 | 0.834
20ms time length | Yes | LNECS | 0.791 | 0.658 | 0.753 | 0.617
5k event number | Yes | LNECS | 0.814 | 0.698 | 0.887 | 0.821
20k event number | Yes | LNECS | 0.829 | 0.718 | 0.899 | 0.832
50k event number | Yes | LNECS | 0.767 | 0.636 | 0.913 | 0.845
event number 10k | Yes | LNECS | 0.840 | 0.727 | 0.893 | 0.825
Table 1: Ablation study on slow and fast motion test data. We report the
AUCp-2D and the AUCp-3D (higher values are better, bold italic font denotes
best numbers).

###  3.3 Training Procedure

As shown in Fig. 1, there are two branches in the training procedure. In the
RGB branch, we applied a binocular stereo vision system as the teacher method.
An RGB-based hand tracking algorithm initially estimated hand poses in the
frames from two RGB cameras, and then, the 3D pose is calculated based on
binocular stereo vision algorithm and used as the supervising information of
event branch. In the event branch, we augmented the training data to enable
the event student network for speed adaptive hand tracking tasks. The
parameters of event student network were updated using supervision from the
RGB branch after the supervised 3D poses were transformed to event camera
space and undergo corresponding data augment transformation. The loss function
is calculated by the following function.

$\mathcal{L}=\|f_{s}(T_{X}(x_{e}))-T_{Y}(T_{cam}(f_{t}(x_{r})))\|_{2}$

where $f_{t}$ denotes the RGB teacher method, $f_{s}$ denotes the event
student network, $x_{e}$ denotes the event stream segment, $x_{r}$ denotes the
RGB videos, $T_{X}$ and $T_{Y}$ denotes data augment operation and $T_{cam}$
denotes the transform operation from RGB to even camera space.

The data augmentation process includes 3 operations: 1) view augmentation such
as rotation and crop, 2) applying event segment of different length, and 3)
randomly suppressing noise. The first operation enhances the robustness of the
model and the other two operations align the domain of slow and fast hand
tracking data. According to the character of event camera, the pixel output an
event when detecting brightness changes. A long period of slow motion event
segment would be similar to a short period of fast motion event segment
regardless the timestamp. Based on this character, we could stimulate fast
motion data using longer event segment of slow motion. However, in practice we
found that the number of noise events is usually proportional to the time
length and thus the noise distribution in augmented data and real fast motion
data is different. To align this distribution, we applied random noise
suppression augmentation. If the number of events in a pixel and its neighbors
is less than a random threshold, the events in this pixel will be regarded as
noise and removed. This noise suppression method can be easily implemented by
the following functions and required very little computation resource.

$F^{\prime}(x,y,p)=1(EC(x,y,p)\ast E_{\sigma}>\epsilon_{r})\times F(x,y,p)$

where $F(x,y,p)$ and $F^{\prime}(x,y,p)$ respectively denote the original
frame and frame after noise suppression, $E_{\sigma}$ denotes an average
filter kernel with size $\sigma$ and $\epsilon_{r}$ is a random threshold
varying with training sample.

###  3.4 Prediction Procedure

According to the character of event camera, whose pixels respond to local
brightness changes independently, fast hand motion would generate more events
than slow motion during the same time interval. We utilized this character and
figured out our speed adaptive segment method. Instead of dividing event
stream into segments with the same time interval, our speed adaptive segment
method determine the length of event segments by event number. In this way, we
ensure the consistency of motion range in a representation frame despite speed
changes.

Solution | Segment Standard | Representation | Our Fast Testset | Our Slow Testset | EventHands Testset
---|---|---|---|---|---
|  |  | AUCp_2d | AUCp_3d | AUCp_2d | AUCp_3d | AUCp_2d
EventHands | event number | LNES | 0.652 | 0.577 | 0.741 | 0.652 |
EventHands | time length | LNES | 0.033 | 0.029 | 0.038 | 0.032 | 0.654
Ours | event number | LNECS | 0.840 | 0.727 | 0.893 | 0.825 | 0.665
Ours | event number | LNES | 0.831 | 0.716 | 0.890 | 0.821 | 0.738
Ours | event number | LNEC | 0.837 | 0.717 | 0.897 | 0.828 | 0.773
Ours | event number | LNECWS | 0.836 | 0.722 | 0.891 | 0.822 | 0.767
Table 2: Comparison between our solution and EventHands.

##  4 Experiment

###  4.1 Implement Details

We employed MediaPipe [17] as the hand tracking model in RGB teacher method
and ResNet-18 as the event student network and tested our solution on event
streams of slow and fast motion. The resolution of our event camera CeleX5 is
$1280\times 800$ and we generate the frame for the event student network with
size $240\times 150$. We use an Adam optimizer with an initial learning rate
of 10-4 and a batch size of 128. We train the model for 50 epochs, reducing
the learning rate by a factor of 10 when the validate loss value did not decay
in the last 5 epochs.

We first introduce our evaluation dataset and metrics. Then we present
ablation studies to evaluate our design choices and compare our solution
against RGB-based solutions.

###  4.2 Test Data and Metrics

The test data includes a slow motion dataset with 8 slow-motion event streams
as well as a fast motion dataset 4 fast-motion event streams. The annotations
of slow hand motion were obtained via a 30Hz RGB binocular stereo vision
system, and annotations of fast hand motion were obtained via a 240Hz system.
Notably, the annotation obtained by 240Hz system aims to validate performance
on the fast hand tracking task of models trained on the slow hand tracking
dataset, but this does not mean that our solution could only deal with hand
motion with speed clear under 240Hz RGB camera. We obtained the 3D ground
truth based on initial hand pose estimate from the two RGB videos by MediaPipe
[17], and manually exclude the inaccurate results.

For metrics, we applied palm-normalized percentage of correct keypoints
($PCK_{p}$) which defines a candidate keypoint to be correct if it falls
within a given distance normalized by palm length around the ground truth. and
the area under the $PCK_{p}$ curve ($AUC_{p}$) to make the result comparable
under different modalities. Specifically, we calculated the distance between
the wrist and middle finger MCP annotations as the palm length.

###  4.3 Ablation Study

####  4.3.1 Speed Adaptive Segmentation Method

We compared the results of different segmentation methods using various
standards, including the same pixel number standard, the same time length
standard, and our proposed same event number standard. Our findings revealed
that the performance of our proposed same event number standard was more
sensitive to changes in the event number when applied to fast test data. The
largest performance difference among slow motion samples with different event
numbers was 0.892 and 0.913, while the difference among fast motion samples
was much larger, at 0.674 and 0.840. Additionally, the solution achieved its
best performance on slow samples with 50,000 events and on fast samples with
10,000 events. Furthermore, the performance difference was minimal when using
test data samples with 10,000 events, indicating that the minimal domain gap
between slow and fast motion data is reached with fewer events in data
samples.

For fixed temporal length segments, we generated a dataset comprising event
segments with a length of 20ms and 50ms, on which the model achieved its best
performance for fast and slow test data, respectively. However, results showed
that the solution using this segmentation standard always suffered a
performance drop, indicating that this standard is unable to narrow the domain
gap between slow and fast motion data.

The standard of using the same pixel number required generating event frames
that contained the same number of pixels where the event occurred. The
solution that applied this standard obtained the best performance on slow test
data, but was not competitive on fast motion test data. A possible reason for
this is the influence of different noise distributions, which leads to a
larger motion range in fast motion frames to reach the same pixel number as
slow motion frames, which contain more noise pixels. This standard also
requires more complex operations during implementation, as it necessitates
repeatedly generating event frames and checking pixel numbers, which requires
loop and condition operations, while our proposed standard can segment data
samples from event stream directly.

####  4.3.2 Event-to-Frame Representation

We compared different event-to-frame representation methods and results showed
that our LENCS representation outperformed other representations. Despite the
representations introduced in Sec 3.2, we also test the representation
Locally-Normalized Event Count Weighted Surface (LNEWCS), which can be denoted
as:

${LNEWCS}(x,y,p)={LNES}(x,y,p)\times{LNEC}(x,y,p)$

This representation also preserves the time information and avoid the
influence of noise as shown in Fig. 3. The reason for the unsatisfying result
is it only contains 2 channels, which is the same as LNES and LNEC, while
LNECWS contians 4 channels, indicting that the time and event count
information cannot be compressed into one channel.

####  4.3.3 Data Augmentation

In Sec. 3.3, we introduced our data augmentation method. Augmentation does not
help on slow motion data since there is no domain gap to be bridged. On fast
motion data, using data augmentation significantly improves the quality of the
predictions.

###  4.4 Comparison with EventHands

EventHands is the pioneer deep-learning based hand tracking solution applying
event camera, which enabled a ResNet-18 trained on synthetic data to deal with
fast hand tracking tasks in the real world. We utilized the released trained
model, test data as well as the evaluation code of EventHands and compared the
performance of our solution and EventsHands on both testset. Notably, the
backbone in both solutions is ResNet-18 and thus would not affect the
comparison. The EventHands divided the event stream into segments of 100ms,
but this segmentation methods lead to unsatisfying performance on our test
data, so we also tested the performance of the trained model in EventHands
with our speed adaptive segmentation method and selected the best result. The
EventHands testest was captured with a different event camera, which was less
sensitive than our camera and output less events for similar motion range. As
a result, our solution obtained the best performance with event segments of
4,000 events on the EventHands testset.

As shown in Table. 2, our solution outperformed EventHands in both testset,
indicating that the idea of tracking fast by learning from slow is better than
the idea of tracking real by learning from synthetic. Besides, as discussed in
Sec 3.2, LNECS combined the time information and event number information,
which enable the deep learning model to learn and distinguish the background
noise in the dataset. However, in the testset of EventHands, noise often
appeared at the previous position of the hand, which did not appear in our
dataset and cannot be stimulated by our data augmentation methods. This gave
rise to the most severe performance drop of solutions applying LNECS than
other representations. In contrast, solution employing LNEC is less likely to
be influenced by noise, and obtained the best performance on EventHands
testset.

Method | AUCp2D | AUCp3D
---|---|---
Mediapipe | 0.735 | 0.644
Minimal-hand | 0.508 | 0.345
Ours(MNECS) | 0.840 | 0.727
Table 3: Comparison of performance of our solution and RGB-based solution on
fast hand tracking task.

###  4.5 Comparison with RGB-based Method

We also captured 30Hz rgb videos for fast hand tracking test data, whose
annotations were obtained in the same way as the way to obtain annotations for
event stream. We tested two RGB based State-Of-The-Arts solutions Mediapipe
[17] and Minimal-hand [18].

As shown in Table. 3, our event based method outperformed RGB based State-Of-
The-Arts. Notably, the annotations of fast motion dataset are obtained from
240Hz RGB video processed by Mediapipe, which means that the performance of
Mediapipe on slow motion tasks with no blur porblem is 1. The performance drop
of RGB-based method is 0.265 on 2D task and 0.356 on 3D task, while the
performance drop of our event-based solution is 0.053 on 2D task and 0.098 on
3D task.

##  5 Conclusion and Future Work

We have developed an innovative solution called the Event-based Speed Adaptive
Hand Tracker. This system employs a deep learning model that has been trained
using a dataset of hand tracking in slow motion, enabling it to accurately
track fast-moving hands. This approach allows us to leverage the knowledge
gained from RGB-based hand tracking technique and the same methodology can
also be adapted for other types of fast-moving object detection or tracking.

To achieve this, we employed three methods to bridge the gap between the
training data for slow motion and the prediction data for fast motion. These
methods include data augmentation, speed adaptive segmentation and NECS
representation. Additionally, we have created and released the first-ever
event-based hand tracking dataset that was collected in real-world
environments. Results showed that our proposed solution enhanced the
performance in fast hand tracking tasks, and outperformed other RGB-based
methods as well as previous event-based method.

For future work, our current method only applied slow motion data in the
training procedure, and utilizing unlabeled fast motion data and domain
adaptive method would be a possible way to further improve the method.
Besides, the performance of our method is limited by the performance of
applied RGB model and adding synthetic data in the training process may also
help to enhance the prediction accuracy.

## References

  * [1] Lin Huang, Jianchao Tan, Ji Liu, and Junsong Yuan.  Hand-transformer: Non-autoregressive structured modeling for 3d hand pose estimation.  In European Conference on Computer Vision, 2020.
  * [2] F. Mueller, F. Bernard, O. Sotnychenko, D. Mehta, and C. Theobalt.  Ganerated hands for real-time 3d hand tracking from monocular rgb.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [3] J. Wang, F. Mueller, F. Bernard, S. Sorli, and C. Theobalt.  Rgb2hands: Real-time tracking of 3d hand interactions from monocular rgb video.  2021\.
  * [4] G. Park, A. Argyros, J. Lee, and W. Woo.  3d hand tracking in the presence of excessive motion blur.  IEEE Transactions on Visualization and Computer Graphics, 26(5):1891–1901, 2020.
  * [5] V. Rudnev, V. Golyanik, J. Wang, H. P. Seidel, F. Mueller, M. Elgharib, and C. Theobalt.  Eventhands: Real-time neural 3d hand pose estimation from an event stream.  In International Conference on Computer Vision, 2021.
  * [6] H. Guo, G. Wang, X. Chen, C. Zhang, F. Qiao, and H. Yang.  Region ensemble network: Improving convolutional network for hand pose estimation, 2017.
  * [7] Mohammad Rezaei, Razieh Rastgoo, and Vassilis Athitsos.  Trihorn-net: A model for accurate depth-based 3d hand pose estimation.  arXiv preprint arXiv:2206.07117, 2022.
  * [8] M. Oberweger, P. Wohlhart, and V. Lepetit.  Generalized feedback loop for joint hand-object pose estimation.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 42(8):1898–1912, 2020.
  * [9] H. Rebecq, R. Ranftl, V. Koltun, and D. Scaramuzza.  High speed and high dynamic range video with an event camera.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(6):1964–1980, 2021.
  * [10] H. C. Duwek, A. Shalumov, and E. E. Tsur.  Image reconstruction from neuromorphic event cameras using laplacian-prediction and poisson integration with spiking and artificial neural networks.  In Computer Vision and Pattern Recognition, 2021.
  * [11] Ziwei Wang, Liyuan Pan, Yonhon Ng, Zheyu Zhuang, and Robert Mahony.  Stereo hybrid event-frame (shef) cameras for 3d perception.  In 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pages 9758–9764. IEEE, 2021.
  * [12] Ana I Maqueda, Antonio Loquercio, Guillermo Gallego, Narciso García, and Davide Scaramuzza.  Event-based vision meets deep learning on steering prediction for self-driving cars.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5419–5427, 2018.
  * [13] Alex Zihao Zhu, Liangzhe Yuan, Kenneth Chaney, and Kostas Daniilidis.  Ev-flownet: Self-supervised optical flow estimation for event-based cameras.  arXiv preprint arXiv:1802.06898, 2018.
  * [14] Ryad Benosman, Charles Clercq, Xavier Lagorce, Sio-Hoi Ieng, and Chiara Bartolozzi.  Event-based visual flow.  IEEE transactions on neural networks and learning systems, 25(2):407–417, 2013.
  * [15] Xavier Lagorce, Garrick Orchard, Francesco Galluppi, Bertram E Shi, and Ryad B Benosman.  Hots: a hierarchy of event-based time-surfaces for pattern recognition.  IEEE transactions on pattern analysis and machine intelligence, 39(7):1346–1359, 2016.
  * [16] Jalees Nehvi, Vladislav Golyanik, Franziska Mueller, Hans-Peter Seidel, Mohamed Elgharib, and Christian Theobalt.  Differentiable event stream simulator for non-rigid 3d tracking.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1302–1311, 2021.
  * [17] Fan Zhang, Valentin Bazarevsky, Andrey Vakunov, Andrei Tkachenka, George Sung, Chuo-Ling Chang, and Matthias Grundmann.  Mediapipe hands: On-device real-time hand tracking.  arXiv preprint arXiv:2006.10214, 2020.
  * [18] Yuxiao Zhou, Marc Habermann, Weipeng Xu, Ikhsanul Habibie, Christian Theobalt, and Feng Xu.  Monocular real-time hand shape and motion capture using multi-modal data.  In IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), June 2020.