11institutetext:  1Qualcomm AI Research111Qualcomm AI Research is an
initiative of Qualcomm Technologies, Inc. 2University of Amsterdam 11email:
<EMAIL_ADDRESS>
11email<EMAIL_ADDRESS>
# Delta Distillation for Efficient Video Processing

Amirhossein Habibian 11 Haitam Ben Yahia 11 Davide Abati 11
Efstratios Gavves 22 Fatih Porikli 11

###### Abstract

This paper aims to accelerate video stream processing, such as object
detection and semantic segmentation, by leveraging the temporal redundancies
that exist between video frames. Instead of propagating and warping features
using motion alignment, such as optical flow, we propose a novel knowledge
distillation schema coined as Delta Distillation. In our proposal, the student
learns the variations in the teacher’s intermediate features over time. We
demonstrate that these temporal variations can be effectively distilled due to
the temporal redundancies within video frames. During inference, both teacher
and student cooperate for providing predictions: the former by providing
initial representations extracted only on the key-frame, and the latter by
iteratively estimating and applying deltas for the successive frames.
Moreover, we consider various design choices to learn optimal student
architectures including an end-to-end learnable architecture search. By
extensive experiments on a wide range of architectures, including the most
efficient ones, we demonstrate that delta distillation sets a new state of the
art in terms of accuracy vs. efficiency trade-off for semantic segmentation
and object detection in videos. Finally, we show that, as a by-product, delta
distillation improves the temporal consistency of the teacher model.

##  1 Introduction

Figure 1: Feature distillation vs. Delta distillation. Instead of distilling
features $\mathbf{z}$ computed with an expensive layer $f$, we distill to a
cheap student $g$ their changes across frames, $\Delta\mathbf{z}$. Due to
temporal correlation in videos, transitions between frames are smooth and
deltas are smaller (as visible) thus easier to distill.

The goal of this paper is to accelerate the processing of video streams, such
as object detection and semantic segmentation. Despite the great progress in
the development of efficient architectures [2, 23, 51, 15, 45, 43], highly
accurate models are still too expensive to process video frames in real-time.
This aspect hinders the deployment of accurate models on constrained settings,
_i.e_. mobile devices.

To this end, recent works apply accurate yet expensive models only on a subset
of frames, referred to as key-frames, and process the remaining ones using a
lighter architecture [57, 18, 55, 24, 25, 26]. The representations from the
light model are then aggregated with the key-frame representations from the
expensive model in a recurrent structure: this step is typically performed at
a deep layer, in order to leverage the representation power of the expensive
model. Due to the misalignment between the current frame and key-frame, this
strategy proves to be effective only when explicit motion alignment is carried
out, typically by means of optical flow warping [57, 18, 55]. For this reason,
feature aggregation is a viable solution under the assumption that the
overhead for extracting motion vectors is lower than in computations within
the bypassed feature extraction. Although this condition is easy to meet for
expensive backbones such as ResNet-101, it has become less reasonable with the
development of efficient models such as EfficientDet [43] or HRNet [45].

This paper introduces a novel approach to leverage the redundancies in a video
to speed up the inference. Our proposal does not rely on explicit motion
alignment and is applicable to any architecture, including the most efficient
ones,  _i.e_. EfficientDet-D0 [43], HRNet [45] and the very recent DDRNet
[13]. Our approach, coined as Delta Distillation , is based on knowledge
distillation [7, 12], a popular technique to accelerate an expensive teacher
network by distilling it into a lightweight student. Given an expensive
teacher processing only key-frames, for every block of layers, we instantiate
a cheap student counterpart, that is fed with a pair of frames and regresses
their corresponding difference (delta) in teacher activations. During
training, the teacher provides target deltas, and the regression error of the
student is minimized by an $\ell_{2}$ objective. During inference, the teacher
provides the representations for the key-frame, and the student iteratively
updates them by adding predicted deltas in the following frames. Due to the
dense interplay between the teacher and student, happening at every block of
the network, delta distillation effectively aggregates the features across
frames without any explicit motion alignment.

Delta distillation has major differences to the common knowledge distillation
setting, where the student learns to regress the teacher features as in Fig. 1
(left). Instead of distilling the features, delta distillation aims for
_distilling the temporal changes in the features_ as illustrated in Fig. 1
(right). Intuitively, instead of learning the feature space embedding of their
teacher, the delta distillation students learn the manifolds generated by
transitions between samples, which we assume to be smooth in the case of
correlated video frames. We therefore hypothesize - and verify experimentally
- that delta distillation, as compared to feature distillation, allows for
learning much cheaper student functions for comparable performance. Moreover,
in contrast to the common knowledge distillation that relies solely on the
student network to process all the test samples, delta distillation leverages
both teacher and student during the inference. This trait enables delta
distillation to enjoy having more parameters (coming from both models) without
increasing the computational cost, as each test sample is processed by either
the teacher or student network.

We summarize our contributions as follows: _i)_ We propose delta distillation,
a novel approach to accelerate video inference without any explicit motion
compensation involved. _ii)_ We elaborate various design choices to learn
optimal student architectures including an end-to-end learnable architecture
search. _iii)_ We conduct extensive experiments on two different tasks and a
wide range of models, including the most efficient architectures. Our analysis
demonstrates that delta distillation sets a new state of the art in terms of
accuracy vs. efficiency trade-off for semantic segmentation and object
detection in videos. _iv)_ We show that, as a by-product, delta distillation
improves the temporal consistency of the teacher model, even though it is not
explicitly optimized to do so.

##  2 Related work

####  2.0.1 Efficiency in deep learning

Improving efficiency of neural networks is an active research area studied
from multiple directions, comprising: quantization, to represent weights and
activations with a low bit precision [9, 16, 20, 32], pruning, to discard
unimportant or redundant channels [6, 11, 23], neural architecture search, to
find network designs with good accuracy vs. efficiency trade-offs [2, 31], and
low rank kernel decompositions [11, 51]. However, for models operating on
videos, redundancy among consecutive frames represents the most essential
leverage to improve efficiency. In recent years, several works investigated in
this direction, and they represent the closest efforts to our proposal. To
avoid extracting expensive representations at every frame, feature aggregation
using optical flow was explored [57, 18, 55]. Nevertheless, the application of
such an approach on modern efficient architectures has become harder, as it
requires careful model design and a proper cost balance between the feature
extraction and motion alignment. Other works aim at building powerful
representations over time, by aggregating features extracted by efficient
models on past frames [24, 14, 25, 26]. These recurrent methods, however,
prove more effective whenever explicit motion alignment operations are carried
out [14], which incur an extra computational cost. Finally, sparse computation
models limit the feature extraction to sparse spatial locations that change
over time,  _e.g_. at a pixel [10] or at patch level [1]. However, this
strategy is not robust to highly dynamic scenes, where most pixels change.
Additionally, the theoretical compute gains do not always translate to latency
improvement due to the inefficiency of sparse operations in most platforms.

####  2.0.2 Knowledge distillation

Another well established direction to accelerate deep neural networks is
knowledge distillation [12], where an efficient student network is optimized
to match the output of an expansive teacher network or model ensemble. This
approach was then extended by performing such an optimization within network
stages, effectively distilling intermediate functions rather than the output
only [37]. After these seminal works, efforts have been spent towards online
distillation methods, dropping the asynchronous training regimes of teacher
and students in favor of a single optimization procedure. For instance, in
[52] multiple networks learn collaboratively without any teacher, and more
recent works formalize the latter as ensemble of multiple students [21, 8,
48]. However, all these approaches do not specifically target video use-cases,
and therefore transfer knowledge between models without explicitly distilling
any temporal dynamic. Differently, the proposed delta distillation directly
operates on temporal changes of features and, as demonstrated by experiments,
allows for much cheaper student models for comparable quality.

Furthermore, some recent works adapt the general framework to specific
recognition tasks, for instance by selecting specific spatial locations for
teacher-student distillation in for anchor-based object detection [5, 46]. We
hereby remark that our approach is task-agnostic and can be applied to any
video task.

##  3 Delta Distillation

We start with a teacher $\mathcal{F}$ as a _spatial_ network generating
accurate representations for a given downstream task,  _e.g_. HRNet [45] or
FasterRCNN [35]. Our goal is to distill this model into a more efficient
_spatio-temporal_ equivalent. We first break $\mathcal{F}$ down into a
composition of $L$ parametric blocks, as:

$\mathcal{F}=f^{L}\circ\dots\circ f^{2}\circ f^{1}.$

We describe delta distillation for a single block, however highlighting that
we carry out the procedure in all blocks within a given network. The $l$-th
block $f^{l}$ takes the form:

$\mathbf{z}^{l}=f^{l}_{\mathbf{\theta}_{l}}(\mathbf{x}^{l}),$

where $\mathbf{x}^{l}$ and $\mathbf{z}^{l}$ denote the input and output of the
block respectively, and $\mathbf{\theta}_{l}$ describes its learnable
parameters. To simplify the notation, we will omit the block index $l$ as we
focus on a single block ($\mathbf{z}=f_{\theta}(\mathbf{x})$), and will
reintroduce it in Sec. 3.3, where we define the overall network.

####  3.0.1 Feature Distillation

Feature distillation [37] treats every block $f_{\theta}$ as a teacher block,
providing target feature maps to supervise a student block $g_{\phi}$,
parametrized by $\phi$, typically designed to be much cheaper. For instance, a
distillation objective optimizes the expected $\ell_{2}$ norm of the error
between $f_{\theta}$ and $g_{\phi}$:

$\mathcal{L}_{d}(\mathbf{x};\phi)=\operatorname{\mathop{{}\mathbb{E}}}_{\mathbf{x}}\left[\left\|f_{\theta}(\mathbf{x})-g_{\phi}(\mathbf{x})\right\|_{2}\right].$
(1)

####  3.0.2 Delta Distillation

Given a sequence of inputs $\mathbf{x}_{t}$, the output of a block
$f_{\theta}$ at a time-step $t$ can be written as:

$\mathbf{z}_{t}=\mathbf{z}_{t-1}+\Delta\mathbf{z}_{t},$

where $\Delta\mathbf{z}_{t}$ represents how the output of the teacher changes
over time. Considering the correlation between the consecutive samples in a
video, we hypothesize that $\Delta\mathbf{z}_{t}$, being a transition function
in the feature manifold, has a lower rank compared to the mapping
$f_{\theta}$. For example, in the extreme case of identical frames,
$\Delta\mathbf{z}_{t}$ will be of rank $0$. We argue that
$\Delta\mathbf{z}_{t}$ can be distilled more effectively than $\mathbf{z}_{t}$
by a student with the same number of parameters, as verified by our
experiments. In delta distillation, the student approximates the deltas given
the current and previous frames as:

$\Delta\tilde{\mathbf{z}}_{t}\approx
g_{\phi}(\mathbf{x}_{t},\mathbf{x}_{t-1}).$

This idea represents the core of our proposal that shifts the perspective from
distilling the function, in Eq. 1, to distilling its temporal changes as:

$\mathcal{L}_{dd}(\mathbf{x}_{t},\mathbf{x}_{t-1};\phi)=\operatorname{\mathop{{}\mathbb{E}}}_{\mathbf{x}_{t},\mathbf{z}_{t}}\left[\left\|\Delta\mathbf{z}_{t}-g_{\phi}(\mathbf{x}_{t},\mathbf{x}_{t-1})\right\|_{2}\right].$
(2)

By optimizing this objective, the student deltas
$\Delta\tilde{\mathbf{z}}_{t}$ converge to the teacher deltas
$\Delta\mathbf{z}_{t}$ as shown in Fig. 2.

|  |
---|---|---
|  |
Epoch 1 | Epoch 2 | Epoch 4

Figure 2: The student deltas $\Delta\tilde{\mathbf{z}}_{t}$ (top) vs. teacher
deltas $\Delta\mathbf{z}_{t}$ (bottom). Thorough training the student learns
to approximate the deltas from its teacher.

###  3.1 Student architectures

As in any distillation approach, delta distillation admits diverse
architectural choices [7], in terms of i) granularity at which it should
operate, _e.g_. distilling single convolutions vs distilling residual blocks
or branches and ii) possible architectures for the student block. In our
implementation, we consider the following:

####  3.1.1 Linear blocks

that define the teachers and students at every convolution. In this case, we
only feed the student with the input residual
$\Delta\mathbf{x}_{t}=\mathbf{x}_{t}-\mathbf{x}_{t-1}$:

$g_{\phi}(\mathbf{x}_{t},\mathbf{x}_{t-1})=g_{\phi}^{conv}(\Delta\mathbf{x}_{t}).$

This choice is motivated by the Taylor approximation of $f_{\theta}$ where, if
the function $f_{\theta}$ is linear, only the first order term is non-zero,
and the derivative $\nabla f_{\theta}(\mathbf{x}_{t-1})$ is a constant:

$\Delta\mathbf{z}_{t}=\nabla
f_{\theta}(\mathbf{x}_{t-1})\Delta\mathbf{x}_{t}+\frac{1}{2}\nabla^{2}f_{\theta}(\mathbf{x}_{t-1})\Delta\mathbf{x}_{t}^{2}+\dots$
(3)

As the student architecture, we rely on a spatial kernel factorization as
depicted in Fig. 3. Similar to the spatial SVD [17], we decompose each 2D
kernel as a sequence of two 1D kernels while reducing the number of
intermediate channels by a compression factor $\gamma$.

Figure 3: Student architecture for a linear block, obtained by decomposing the
teacher kernel as a sequence of two 1D kernels with $\gamma\times$ less number
of intermediate channels. For simplicity, we only visualize the output
channels.

####  3.1.2 Non-linear blocks

that define the teachers and students at a coarser granularity as a sequence
of residual blocks. In this case, according to the Taylor approximation in Eq.
3, we parameterize the student as a function of both the previous input
$\mathbf{x}_{t-1}$ and input residual $\Delta\mathbf{x}_{t}$ concatenated
along the channel dimension:

$g_{\phi}(\mathbf{x}_{t},\mathbf{x}_{t-1})=g_{\phi}^{block}(\mathbf{x}_{t-1},\Delta\mathbf{x}_{t}).$

As the student architecture, we envision two strategies: i) channel reduction,
where the student mirrors the teacher but with fewer channels: we add two
pointwise convolution to the block as first and last layer to shrink and
expand the channels respectively by a fixed factor. ii) spatial reduction,
where the student resembles the teacher but operates on a smaller resolution:
we add a strided pointwise convolution and a pixel shuffle up-sampling to the
beginning and the end of each block, respectively.

###  3.2 Student architecture search

Different layers within a network may be compressible to different extents. We
empirically found that there might exist a few critical layers 222as an
example, transition layers in HRNets [45] that are not amenable to
distillation: if compressed, they hinder the model performance. Therefore,
instead of committing to the same student architecture for all the blocks, we
introduce two candidates: i) non-compressed architecture, identical to the
teacher, preferred for hard-to-compress blocks. ii) compressed architecture
using the techniques introduced above for linear and non-linear blocks.

To find optimal student architectures, we introduce a learnable parameter
$\psi\in\mathbb{R}^{2}$ per block, that is learned jointly with the student
parameters. During training, architectures are sampled from a categorical
distribution $q_{\psi}$ over the two candidate models, obtained by feeding
$\psi$ to a softmax layer. We rely on the Gumbel-softmax [19, 29]
reparametrization to estimate gradients. To encourage the search algorithm to
opt for the compressed architecture, we introduce a sparsity regularization
objective as:

$\mathcal{L}_{s}(;\psi)=\operatorname{\mathop{{}\mathbb{E}}}_{g_{\phi}\sim
q_{\psi}}\left[\text{FLOPs}(g_{\phi})\right],$ (4)

where FLOPs is the complexity of the sampled architecture $g_{\phi}$ in terms
of the number of floating point operations. This loss encourages the model to
select the more efficient student architecture as much as possible. If not
regularized, $\psi$ would converge to the trivial solution of selecting the
non-compressed architecture as it has a higher capacity and a better
distillation performance.

###  3.3 Training and Inference

Figure 4: Delta distillation at inference. The teacher computes the features
for the key-frame, while the student updates the features by predicting deltas
over time.

Given a trained teacher network $\mathcal{F}$, comprising the teacher blocks,
and a training set of labeled clips with $T$ frames
$\\{(\mathbf{x}_{1:T},\mathbf{y}_{1:T})\\}$, we train the delta distillation
by optimizing the following objective:

$\operatorname{\mathop{{}\mathbb{E}}}_{\mathbf{x}_{1:T},\mathbf{y}_{1:T}}\left[\frac{1}{T}\sum_{t=1}^{T}\big{(}\mathcal{L}_{t}+\alpha\sum_{l=1}^{L}\mathcal{L}_{dd}^{l}+\beta\sum_{l=1}^{L}\mathcal{L}_{s}^{l}\big{)}\right]$
(5)

where $\mathcal{L}_{dd}^{l}$ and $\mathcal{L}_{s}^{l}$ denote the delta
distillation and the sparsity regularization losses, computed for the $l$-th
block, as defined in Eq. 2 and 4 respectively. The hyper-parameters $\alpha$
and $\beta$ balance the contribution of the losses to learn a model yielding
the best accuracy vs. efficiency trade-off.

$\mathcal{L}_{t}(\mathbf{x}_{t},\mathbf{y}_{t};\Theta,\Phi)$ is the task loss
used to train the network $\mathcal{F}$, where
$\Theta=\\{\theta_{1}\dots\theta_{L}\\}$ and
$\Phi=\\{\phi_{1}\dots\phi_{L}\\}$ denote all the learnable parameters in the
teacher and student, respectively. Although we envision an unsupervised
training, by excluding $\mathcal{L}_{t}$ from the objective, we conduct all
our experiments while including $\mathcal{L}_{t}$ as it yields a better
performance.

As illustrated in Fig. 4, delta distillation calls both teachers and students
during the inference. More specifically, the key-frame is passed through the
teacher to compute initial features. Then, subsequent frames are fed to the
students to update the previous features by predicting their deltas across the
frames. This is different from typical knowledge distillation settings, such
as feature distillation, where the teacher is only used during training and
not during inference. Pseudo code for training and inference is provided in
the supplementary material.

###  3.4 Temporal Consistency

Another aspect of increasing importance for video streaming tasks is temporal
consistency of model responses [22, 28, 34]. Indeed, flickering predictions
can make decision-making difficult in critical scenarios. Although our main
motivation is to improve the model efficiency, we argue that delta
distillation can also improve the temporal consistency in predictions (as
verified experimentally in Sec. 4.2.4). As an explanation, we argue that delta
distillation converts the _spatial_ teacher network to a _spatio-temporal_
model, as it propagates states from one timestep to the next, and explicit
temporal dynamics improve the overall temporal stability, as also noted in
several prior works [34, 41]. Moreover, in delta distillation, the students
have a regularization effect on the teacher. More specifically, the teacher
should generate smooth and low-rank features so as to be learnable by a
student with a limited number of parameters. This penalizes the teacher from
generating hard to distill deltas, _e.g_. from representations that are
inconsistent over time.

##  4 Experiments

We evaluate delta distillation on two video tasks: object detection and
semantic segmentation, in Sec. 4.1 and 4.2, respectively. Several ablation
studies are reported in Sec. 4.3. Further analysis are provided in the
supplementary material.

###  4.1 Object Detection

####  4.1.1 Dataset and metrics

We experiment with Imagenet VID [38], which contains 3862, 555, and 937
annotated snippets for training, validation, and test, respectively. All
frames come with bounding boxes belonging to $30$ target categories. Following
the standard protocol [3, 55, 56, 57], we augment the training snippets with
still images from ImageNet DET sampled at $1:1$ ratio, and report results on
the validation set. We rely on mean Average Precision (mAP) with an IoU
threshold of $0.5$ as the accuracy metric. To be hardware-agnostic, we report
the efficiency of models in terms of number of floating point operations
(FLOPs333FLOPs denotes number of multiply-adds.).

####  4.1.2 Training details

We conduct our experiments using a single-stage and a two-stage object
detector, namely EfficientDet-D0 [43] and Faster-RCNN [35] with a ResNet-101
backbone. We first train the teacher networks $\mathcal{F}$ using a SGD
optimizer with a learning rate of $0.01$ for $7$ epochs. The learning rate is
reduced by a factor of 10 at epochs $2$ and $5$. Clips are resized to have a
longer side of $512~{}px$ and $600\sim 1000~{}px$ for EfficientDet and Faster-
RCNN, respectively. We train the models on four GPUs using a batch size of
$16$ and $4$ for the EfficientDet and Faster-RCNN, respectively. Starting from
the trained teacher network and randomly initialized students, we optimize the
training objective, Eq. 5, setting $\alpha$ and $\beta$ to $1000$ and $10$ for
Faster-RCNN, and to $100$ and $10$ for EfficientDet. We rely on linear blocks
with a compression ratio $\gamma=16$, as illustrated in Fig. 3, to define the
student architectures.

Figure 5: Comparisons to knowledge distillations on ImageNet VID. Delta
distillation outperforms the alternatives. The gap is higher for
EfficientDet-D0 that is already highly optimized so is more challenging to be
accelerated further.

####  4.1.3 Evaluation details

Following [57], we split each video into sequences of equal length $T=10$. The
first frame in a sequence (key-frame) is processed by the teacher while other
frames are processed by the student networks. We report mAP and FLOPs averaged
over all frames in the sequence. In the supplement, we report detailed
analysis teacher and student cost, illustrating per-frame FLOPS as well as
their capacity in terms of parameters.

####  4.1.4 Comparison to knowledge distillations

We evaluate our key hypothesis, namely that $\Delta\mathbf{z}_{t}$ can be
distilled more effectively than $\mathbf{z}_{t}$, by training students using
two different distillation objectives: feature vs. delta distillation as
defined in Eq. 1 and 2. Additionally, we compare to Fine Grained Feature
Imitation (FGFI) [46], as a knowledge distillation devised for object
detection that distills the features around the object anchor locations from a
ResNet-101 backbone using a lighter student backbone (R50, R34, and R18). We
omit applying FGFI on EfficientDet-D0 as there is no more efficient backbone
available to serve as the student for this detector. We study how the model
accuracy responds to reducing the computational cost  _i.e_. by using a
cheaper backbone for FGFI, and a higher sparsity regularization $\beta$ for
feature and delta distillation.

As reported in Fig. 5, our results demonstrate that FGFI is effective when
compared to the optimization of the student backbones from scratch (blue
plot), yet it underperforms with respect to the feature distillation that
shares all components of our model, _e.g_. SVD kernel decomposition and
student architecture search. Additionally, the results verify that feature
distillation has a limited effectiveness in reducing the model complexity
especially for a highly optimized architecture such as EfficientDet-D0.
However, by leveraging temporal redundancy, delta distillation reduces the
compute cost by $2\times$ with a negligible drop in the accuracy. For
EfficientDet-D0, it reduces the GFLOPs from 2.5 to 1.14 with a negligible mAP
drop, from 69.0 to 68.8.

####  4.1.5 Comparison to state of the art

We compare to the state of the art video object detectors that leverage
temporal redundancies to speed up the inference, as categorized into: _i)_
feature warping,  _i.e_. DFF [57] and Mobile-DFF [55], that bypass feature
computation by warping the previous features using optical flow. _ii)_ feature
aggregation,  _i.e_. TAFM [25], that instead of extracting expensive features
at every frame, it relies on an aggregation of cheaper features over time.

Table 1: Comparisons with efficient video object detectors on ImageNet VID,
for some light (top) and heavy (bottom) networks. Delta distillation achieves
the lowest FLOPs while being more accurate than others.

Model | FLOPs (G) | mAP
---|---|---
TAFM [25] | 1.18 | 64.1
PatchWork [1] | 0.97 | 57.4
Skip-Conv [10] | 0.78 | 62.9
PatchNet [30] | 0.73 | 58.9
Mobile-DFF [55] | 0.71 | 62.8
EfficientDet-D0 [43] | 2.50 | 69.0
\+ Delta Distillation | 0.71 | 66.1
DFF [57] | 34.9 | 72.5
PatchNet [30] | 34.2 | 73.1
Faster-RCNN [35] | 149.1 | 74.5
\+ Delta Distillation | 29.2 | 73.5

_iii)_ feature sparsification,  _i.e_. PatchWork [1] and Skip-Conv [10], that
restrict the feature computation only to a sparse set of regions or pixels
that change significantly across frames. _vi)_ detection by tracking,  _i.e_.
PatchNet [30], that interleaves running an expensive detector with cheap
object trackers. As reported in Tab. 1, we divide object detectors into two
groups: light detectors (top) developed for mobile devices, using MobileNet-v2
[39] and EfficientNet [42] backbones, and expensive detectors (bottom), using
ResNet-101 backbone.

Our results show that delta distillation achieves the lowest FLOPs while being
more accurate than both the light and expensive detectors. Moreover, despite
the differences between FasterRCNN-R101 and EfficientDet-D0 in terms of
architectures and computational bottlenecks, delta distillation consistently
reduces their FLOPs without any architectural adaption,  _i.e_. from 2.5 to
0.71 and from 149.1 to 29.2 respectively. This is not the case for feature
warping methods, that require a careful architecture design to find a right
cost and accuracy balance between feature extractor and optical flow model.

###  4.2 Semantic Segmentation

####  4.2.1 Dataset and metrics

We conduct experiments on the Cityscapes dataset [4] that is partitioned into
2975, 500 and 1525 snippets as training, validation, and test splits
respectively. We rely on the standard training split to train and report
results on the validation set. The dataset provides pixel-level annotations
into 19 classes for one frame per snippet. We extract the remaining per-frame
psuedo-annotations, which are required to train video models, by applying an
off the shelf segmentation network [44] on unannotated frames in the training
set. We evaluate the accuracy using mean intersection-over-union (mIoU). As a
mean to compare computational cost, we rely again on FLOPs also reporting
latency measurements in the supplementary materials.

####  4.2.2 Training details

Model | FLOPs (G) | mIoU
---|---|---
FANet-34 [15] | 65.0 | 76.3
BiseNet-v1-18 [50] | 55.3 | 74.8
FANet-18 [15] | 49.0 | 75.0
LedNet [47] | 45.8 | 71.5
ICNet [53] | 28.2 | 67.7
FasterSeg [2] | 28.2 | 73.1
ERFNet [36] | 27.7 | 70.0
SwiftNet-18 [33] | 26.0 | 70.2
BiseNet-v2 [49] | 21.1 | 73.4
TDNet-PSPNet [14] | 541.0 | 79.9
DFF [57] | 109.3 | 69.2
TDNet-BiseNet [14] | 101.3 | 76.4
Skip-Conv [10] | 29.0 | 75.5
DDRNet-39 [13] | 282.0 | 79.5
\+ Delta Distillation | 140.0 | 79.9
DDRNet-23 [13] | 143.7 | 78.7
\+ Delta Distillation | 71.8 | 78.9
HRNet-w18-small [45] | 77.9 | 76.1
\+ Delta Distillation | 34.1 | 75.7
DDRNet-23-slim [13] | 36.6 | 76.1
\+ Delta Distillation | 17.9 | 76.2

Table 2: Comparison with efficient image (top) and video (middle) based models
on Cityscapes validation set.

We conduct experiments using two state-of-the-art segmentation model: HRNet
[45] and DDRNet [13]. We follow the same training protocol as in [13, 45]:
models are initialized with ImageNet weights and trained using a SGD optimizer
with a learning rate of 0.01, that is reduced using a polynomial decay policy
with a power of 0.9. Training runs for 484 epochs using a batch size of 12 on
four GPUs and SyncBN. The models are trained on random crops of $512\times
1024$ and tested on $1024\times 2048$. For DDRNet [13], we use online hard
example mining [40] as in [13]. Starting from the trained teacher and randomly
initialized students, we optimize Eq. 5, using $\alpha$ and $\beta$ set to 1
and 0.5 respectively. We rely on linear blocks with a compression ratio
$\gamma=4$, as illustrated in Fig. 3, to define the student architectures.

####  4.2.3 Evaluation details

Each video is split into sequences of equal length $T$. We fix the sequence
length $T=3$ as it yields the best trade-off between accuracy vs. efficiency
though the model is relatively robust to longer sequences as reported in Fig
6. Since the videos in Cityscapes have temporally sparse annotation, we repeat
evaluations by opting each annotated frame in all possible positions within
the sequence and report the averaged mIoU following [14]. Similarly we report
FLOPs averaged over all frames in the sequence counting for the both teacher
and student costs.

####  4.2.4 Comparison to state of the art

Figure 6: Robustness to temporal variations. By increasing the distance to the
key-frame, delta distillation retains better performances of prior methods.

We first assess the effect of delta distillation on different segmentation
backbones with varying computational cost: HRNet-W18-small, DDRNet-23-slim,
DDRNet-23, and DDRNet-39. As reported in Tab. 2 (bottom), delta distillation
consistently reduces computational cost by a factor of $\sim 2\times$ for all
the backbones, with no or small drop in accuracy.

Tab. 2 compares delta distillation with efficient image444We limit our
comparisons to efficient models with less than 100 GFLOPs. (top) and video
(middle) based semantic segmentation models. The results show that delta
distillation outperforms all the image-based models while being at the same
time more efficient. Compared to BiseNet-v2 [49], the most efficient frame-
based model, delta distillation achieves a mIoU of 76.2 vs. 73.4, with a lower
cost of 17.9 vs 21.1 GFLOPs. Moreover, delta distillation achieves a more
favorable accuracy vs. efficiency trade-off compared to the other video-based
models. For instance, at the same mIoU of 79.9, delta distillation is
$3.8\times$ more efficient (541 vs 140 GFLOPs) than a TDNet [14] with PSPNet
backbone. Finally, the delta distilled DDRNet-23-slim model significantly
outperforms DFF [57] and Skip-conv [10], both in terms of accuracy and
efficiency. Per-class analysis, reported in the supp. material, highlights
that accuracies are retained on both static and dynamic classes.

##### Robustness to temporal variations.

We evaluate the impact of sequence length ($T$) on the performance of delta
distillation as reported in Fig 6. The longer the sequence is, the less
frequently features are refreshed by running the teacher model. This drops the
accuracy as the student model has a limited capacity compared to the teacher.
However, as highlighted in the results, the performance drop is smaller for
delta distillation, especially on longer sequences, compared to competing
methods. As a lower bound, we include an interleaved baseline that copies the
predictions from the key-frame to consecutive frames. These findings verify
the effectiveness of delta distillation in handling long range temporal
variations.

##### Choice of backbone.

In Tab. 2 we evaluate video-based models using the backbones originally used
by authors  _i.e_. ResNet-101 for DFF and ResNet-50 for TDNet-PSPNet. Since
these backbones are too expensive to run in real-time we rely on more
efficient backbones,  _e.g_. DDRNet, that are arguably more challenging to be
further accelerated. As a further analysis, we implement several video-based
models using DDRNet-23-slim backbone as reported in Fig 6. The figure confirms
the superiority of delta distillation as compared to alternative video
efficiency models all using the same backbone.

####  4.2.5 Temporal consistency

Figure 7: Example predictions on Cityscapes, with DDRNet23 on top and DDRNet23
+ Delta Distillation below. The latter shows more consistency over the
baseline model. More examples are reported in the supplementary material.

As motivated in Sec. 3.4, we evaluate the effectiveness of delta distillation
at improving the temporal consistency (TC) on the validation set. We follow
the TC metric introduced in [28]: in a nutshell, it computes the average IoU
among model predictions across successive frames, after motion compensation by
optical flow warping.

Our results, presented in Tab. 4, suggests several insights: First, the
training procedure of delta distillation effectively regularizes the teacher
model towards more temporally consistent predictions, even when run on
different frames independently (T): such a benefit is testified by the
improvement of 1.8 points with respect to the baseline. Furthermore, by using
the inference procedure comprising of both teacher and student (T+S), the TC
metric further improves. Finally, we compare to the temporal consistency
reported by ETC [28] which explicitly includes a temporal consistency loss in
the optimization. Fig. 7 shows qualitative results of DDRNet23 both with and
without delta distillation.

Table 3: Temporal consistency (TC) measured on Cityscapes.

Model | TC |  $\Delta$ TC
---|---|---
PSPNET18 [54] | 83.3 | -
\+ ETC [28] | 84.6 | +1.3
DDRNet23 [13] | 82.6 | -
\+ Delta Distillation  (T) | 84.4 | +1.8
\+ Delta Distillation  (T+S) | 85.2 | +2.6

Table 4: Ablation on student architecture designs.

Student Architecture | FLOPs (G) | mIoU
---|---|---
DDRNet23 | 143.7 | 78.7
\+ Linear | 71.8 | 78.9
\+ Non-Linear (Spatial) | 110.3 | 78.4
\+ Non-Linear (Channel) | 84.3 | 79.0
\+ Non-Linear (Channel + Spatial) | 96.13 | 78.7

###  4.3 Ablation studies

####  4.3.1 Student architecture

We analyze the effect of different choices of architecture, following the
designs described in Sec. 3.1. In Tab. 4, we show the results of training
Linear vs Non-Linear blocks as well as the channel vs spatial reduction.
First, we note that delta distillation, regardless of architectural choice,
has lower GFLOPs than the base Image Model. This trait is desirable, as it
suggests the computational savings are not bound to a unique student
architecture. However, we do see some notable differences within architecture
differences themselves. By comparing both Linear and Non-Linear - Channel, we
appreciate that the former enjoys a slightly smaller computational footprint
(71.8 vs 84.3 GFLOPs) with similar mIoU. We hypothesize this difference could
be due to the fact that linear functions are easier to distill. Finally, when
we compare the channel and spatial variants, we observe the latter
architecture performs slightly worse.

####  4.3.2 Student architecture search

Figure 8: Ablation on student architecture search. We report the proportion of
the compressed layers per stage.

We study how the architecture search, discussed in Sec. 3.1, selects the
student architectures. For this purpose, we observe the effect of gradually
increasing the sparsity coefficient, $\beta$ from Eq. 5. We report the
proportion of compressed blocks for DDRNet23-slim grouped by its four main
stages: stem convolutions at the entry, low and high-resolution branches to
process the input in parallel, and a pyramid pooling module (PPM) at the end
to fuse feature maps across resolutions. We normalize the number of the
compressed blocks per stage and report it as the compression rate in Fig. 8.

We note that a higher $\beta$ indeed translates to a higher proportion of
compressed blocks across all stages. Moreover, as we reduce $\beta$, the
search algorithm opts for selecting less compressed blocks in the stem. We
hypothesize that since all the layers follow the stem, this layer represents a
single point of failure: at this stage, a non effective distillation might
hinder the whole model’s performance. We observe a similar pattern for the PPM
stage, likely due to its closeness to the output, thus having a bigger impact
on the performance.

##  5 Conclusion

We proposed delta distillation, a novel method for efficient video processing
exploiting the temporal redundancy of frames. Our proposal optimizes the
regression, by means of cheap student models, of temporal variations in
feature maps computed by an expensive teacher network. During inference, the
teacher provides the initial representations for the first frame; such feature
maps are then iteratively refined for the next frames by adding deltas
estimated by students, the latter operating at a low computational cost. We
show through extensive experiments that delta distillation outperforms feature
distillation for comparable student architectures, and delivers state-of-the-
art results for efficient video segmentation and object detection. Moreover,
we note how the proposed procedure improves the temporal consistency of the
distilled model.

## References

  * [1] Chai, Y.: Patchwork: A patch-wise attention network for efficient object detection and segmentation in video streams. In: ICCV (2019)
  * [2] Chen, W., Gong, X., Liu, X., Zhang, Q., Li, Y., Wang, Z.: Fasterseg: Searching for faster real-time semantic segmentation. ICLR (2020)
  * [3] Chen, Y., Cao, Y., Hu, H., Wang, L.: Memory enhanced global-local aggregation for video object detection. In: CVPR (2020)
  * [4] Cordts, M., Omran, M., Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., Franke, U., Roth, S., Schiele, B.: The cityscapes dataset for semantic urban scene understanding. In: CVPR (2016)
  * [5] Dai, X., Jiang, Z., Wu, Z., Bao, Y., Wang, Z., Liu, S., Zhou, E.: General instance distillation for object detection. In: CVPR (2021)
  * [6] Denil, M., Shakibi, B., Dinh, L., Ranzato, M., de Freitas, N.: Predicting parameters in deep learning. In: NeurIPS (2013)
  * [7] Gou, J., Yu, B., Maybank, S.J., Tao, D.: Knowledge distillation: A survey. IJCV (2021)
  * [8] Guo, Q., Wang, X., Wu, Y., Yu, Z., Liang, D., Hu, X., Luo, P.: Online knowledge distillation via collaborative learning. In: CVPR (2020)
  * [9] Gupta, S., Agrawal, A., Gopalakrishnan, K., Narayanan, P.: Deep learning with limited numerical precision. In: ICML (2015)
  * [10] Habibian, A., Abati, D., Cohen, T.S., Bejnordi, B.E.: Skip-convolutions for efficient video processing. In: CVPR (2021)
  * [11] He, Y., Zhang, X., Sun, J.: Channel pruning for accelerating very deep neural networks. In: ICCV (2017)
  * [12] Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 (2015)
  * [13] Hong, Y., Pan, H., Sun, W., Jia, Y., et al.: Deep dual-resolution networks for real-time and accurate semantic segmentation of road scenes. arXiv preprint arXiv:2101.06085 (2021)
  * [14] Hu, P., Caba, F., Wang, O., Lin, Z., Sclaroff, S., Perazzi, F.: Temporally distributed networks for fast video semantic segmentation. CVPR (2020)
  * [15] Hu, P., Perazzi, F., Heilbron, F.C., Wang, O., Lin, Z., Saenko, K., Sclaroff, S.: Real-time semantic segmentation with fast attention. ICRA (2020)
  * [16] Jacob, B., Kligys, S., Chen, B., Zhu, M., Tang, M., Howard, A., Adam, H., Kalenichenko, D.: Quantization and training of neural networks for efficient integer-arithmetic-only inference. In: CVPR (2018)
  * [17] Jaderberg, M., Vedaldi, A., Zisserman, A.: Speeding up convolutional neural networks with low rank expansions. BMVC (2014)
  * [18] Jain, S., Wang, X., Gonzalez, J.E.: Accel: A corrective fusion network for efficient semantic segmentation on video. In: CVPR (2019)
  * [19] Jang, E., Gu, S., Poole, B.: Categorical reparameterization with gumbel-softmax. ICLR (2017)
  * [20] Krishnamoorthi, R.: Quantizing deep convolutional networks for efficient inference: A whitepaper. arXiv preprint arXiv:1806.08342 (2018)
  * [21] Lan, X., Zhu, X., Gong, S., et al.: Knowledge distillation by on-the-fly native ensemble. NeurIPS (2018)
  * [22] Lei, C., Xing, Y., Chen, Q.: Blind video temporal consistency via deep video prior. NeurIPS (2020)
  * [23] Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets. arXiv preprint arXiv:1608.08710 (2017)
  * [24] Li, Y., Shi, J., Lin, D.: Low-latency video semantic segmentation. In: CVPR (2018)
  * [25] Liu, M., Zhu, M.: Mobile video object detection with temporally-aware feature maps. In: CVPR (2018)
  * [26] Liu, M., Zhu, M., White, M., Li, Y., Kalenichenko, D.: Looking fast and slow: Memory-guided mobile video object detection. arXiv preprint arXiv:1903.10172 (2019)
  * [27] Liu, M., Zhu, M., White, M., Li, Y., Kalenichenko, D.: Looking fast and slow: Memory-guided mobile video object detection. arXiv preprint arXiv:1903.10172 (2019)
  * [28] Liu, Y., Shen, C., Yu, C., Wang, J.: Efficient semantic video segmentation with per-frame inference. ECCV (2020)
  * [29] Maddison, C.J., Mnih, A., Teh, Y.W.: The concrete distribution: A continuous relaxation of discrete random variables. ICLR (2017)
  * [30] Mao, H., Zhu, S., Han, S., Dally, W.J.: Patchnet–short-range template matching for efficient video processing. arXiv preprint arXiv:2103.07371 (2021)
  * [31] Moons, B., Noorzad, P., Skliar, A., Mariani, G., Mehta, D., Lott, C., Blankevoort, T.: Distilling optimal neural networks: Rapid search in diverse spaces. In: ICCV (2021)
  * [32] Nagel, M., van Baalen, M., Blankevoort, T., Welling, M.: Data-free quantization through weight equalization and bias correction. ICCV (2019)
  * [33] Orsic, M., Kreso, I., Bevandic, P., Segvic, S.: In defense of pre-trained imagenet architectures for real-time semantic segmentation of road-driving images. In: CVPR (2019)
  * [34] Rebol, M., Knöbelreiter, P.: Frame-to-frame consistent semantic segmentation. In: Joint Austrian Computer Vision And Robotics Workshop (ACVRW) (2020)
  * [35] Ren, S., He, K., Girshick, R., Sun, J.: Faster r-cnn: Towards real-time object detection with region proposal networks. NeurIPS (2015)
  * [36] Romera, E., Alvarez, J.M., Bergasa, L.M., Arroyo, R.: Erfnet: Efficient residual factorized convnet for real-time semantic segmentation. IEEE Transactions on Intelligent Transportation Systems (2017)
  * [37] Romero, A., Ballas, N., Kahou, S.E., Chassang, A., Gatta, C., Bengio, Y.: Fitnets: Hints for thin deep nets. ICLR (2015)
  * [38] Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A.C., Fei-Fei, L.: ImageNet Large Scale Visual Recognition Challenge. IJCV (2015)
  * [39] Sandler, M., Howard, A., Zhu, M., Zhmoginov, A., Chen, L.C.: Mobilenetv2: Inverted residuals and linear bottlenecks. In: CVPR (2018)
  * [40] Shrivastava, A., Gupta, A., Girshick, R.: Training region-based object detectors with online hard example mining. In: CVPR (2016)
  * [41] Sibechi, R., Booij, O., Baka, N., Bloem, P.: Exploiting temporality for semi-supervised video segmentation. In: ICCV Workshops (2019)
  * [42] Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: ICML (2019)
  * [43] Tan, M., Pang, R., Le, Q.V.: Efficientdet: Scalable and efficient object detection. In: CVPR (2020)
  * [44] Tao, A., Sapra, K., Catanzaro, B.: Hierarchical multi-scale attention for semantic segmentation. arXiv preprint arXiv:2005.10821 (2020)
  * [45] Wang, J., Sun, K., Cheng, T., Jiang, B., Deng, C., Zhao, Y., Liu, D., Mu, Y., Tan, M., Wang, X., Liu, W., Xiao, B.: Deep high-resolution representation learning for visual recognition. TPAMI (2019)
  * [46] Wang, T., Yuan, L., Zhang, X., Feng, J.: Distilling object detectors with fine-grained feature imitation. In: CVPR (2019)
  * [47] Wang, Y., Zhou, Q., Liu, J., Xiong, J., Gao, G., Wu, X., Latecki, L.J.: Lednet: A lightweight encoder-decoder network for real-time semantic segmentation. In: ICIP (2019)
  * [48] Wu, G., Gong, S.: Peer collaborative learning for online knowledge distillation. In: AAAI (2021)
  * [49] Yu, C., Gao, C., Wang, J., Yu, G., Shen, C., Sang, N.: Bisenet v2: Bilateral network with guided aggregation for real-time semantic segmentation. IJCV (2021)
  * [50] Yu, C., Wang, J., Peng, C., Gao, C., Yu, G., Sang, N.: Bisenet: Bilateral segmentation network for real-time semantic segmentation. In: ECCV (2018)
  * [51] Zhang, X., Zou, J., He, K., Sun, J.: Accelerating very deep convolutional networks for classification and detection. TPAMI (2016)
  * [52] Zhang, Y., Xiang, T., Hospedales, T.M., Lu, H.: Deep mutual learning. In: CVPR (2018)
  * [53] Zhao, H., Qi, X., Shen, X., Shi, J., Jia, J.: Icnet for real-time semantic segmentation on high-resolution images. In: ECCV (2018)
  * [54] Zhao, H., Shi, J., Qi, X., Wang, X., Jia, J.: Pyramid scene parsing network. In: CVPR (2017)
  * [55] Zhu, X., Dai, J., Zhu, X., Wei, Y., Yuan, L.: Towards high performance video object detection for mobiles. arXiv preprint arXiv:1804.05830 (2018)
  * [56] Zhu, X., Wang, Y., Dai, J., Yuan, L., Wei, Y.: Flow-guided feature aggregation for video object detection. In: ICCV (2017)
  * [57] Zhu, X., Xiong, Y., Dai, J., Yuan, L., Wei, Y.: Deep feature flow for video recognition. In: CVPR (2017)

Delta Distillation for Efficient Video Processing
Supplementary material

// $\eta$ is learning rate, globally given

1 function  _train_dd( $f_{\theta}$, $g_{\phi}$, x_list)_

// compute groundtruth deltas

2 z_list $\leftarrow$ $f_{\theta}$(x_list);

3 z_cur $\leftarrow$ z_list[1:];

4 z_past $\leftarrow$ z_list[:-1];

5 $\Delta\mathbf{z}_{t}$ = (z_cur - z_past).detach();

// estimate deltas with student

6 x_cur $\leftarrow$ x_list[1:];

7 x_past $\leftarrow$ x_list[:-1];

8 $\Delta\tilde{\mathbf{z}}_{t}=g_{\phi}$(x_cur, x_past);

// optimize distillation loss

9
$\mathcal{L}_{dd}=\left\|\Delta\mathbf{z}_{t}-\Delta\tilde{\mathbf{z}}_{t}\right\|_{2}$;

10 $\phi\leftarrow\phi-\eta\frac{\nabla\mathcal{L}_{dd}}{\nabla\phi}$;

11

Algorithm 1 Delta distillation (training $\mathcal{L}_{dd}$)

// $T$ is globally given

1 function  _inference( $f_{\theta}$, $g_{\phi}$, x_list)_

2 z_list $\leftarrow$ [];

3 $\mathbf{x}^{t-1},\mathbf{z}^{t-1}\leftarrow$ none, none;

4 for $i\leftarrow 0$ to len(x_list) do

5 $\mathbf{x}^{t}\leftarrow$ x_list[i];

6 if $i$ $\%$ $T$ $==$ $0$ then

// predict with teacher

7 $\mathbf{z}^{t}\leftarrow f_{\theta}(\mathbf{x}^{t})$;

8

9 else

// update with student

10
$\mathbf{z}^{t}\leftarrow\mathbf{z}^{t-1}+g_{\phi}(\mathbf{x}^{t},\mathbf{x}^{t-1})$;

11

12 z_list.append($\mathbf{z}_{t}$);

13 $\mathbf{x}^{t-1},\mathbf{z}^{t-1}\leftarrow\mathbf{x}^{t},\mathbf{z}^{t}$;

14

15 return z_list

Algorithm 2 Delta distillation (inference)

##  A Training and inference algorithms

For clarity, we report the main pseudo-code algorithms characterizing delta
distillation. More specifically, Alg. 1 illustrates the optimization of delta
distillation objective $\mathcal{L}_{dd}$. Furthermore, Alg. 2 describes the
inference schedule alternating between teacher (providing representations) and
student (updating them). We however recall that the full optimization of a
delta distillation model, described in Sec. 3.3, also entails the optimization
of the task objective $\mathcal{L}_{t}$ and of the sparsity objective
$\mathcal{L}_{s}$, that we omit to avoid cluttering Alg. 1.

##  B Further segmentation experiments

###  B.1 Latency measurements

Table 5: The efficiency gains by Delta Distillation translate to latency improvement in resource constrained devices. Model |  | FLOPs |  | Latency
---|---|---|---|---
|  (G) | $\uparrow$ |  |  (ms) | $\uparrow$
DDRNet39 |  |  282.0 | - |  |  41.0 | -
\+ Delta Distillation |  |  131.6 | (2.1$\times$) |  |  16.5 | (2.5$\times$)
DDRNet23 |  |  143.7 | - |  |  22.6 | -
\+ Delta Distillation |  |  71.3 | (2.0$\times$) |  |  12.4 | (1.8$\times$)
HRNet-w18-small |  |  77.9 | - |  |  39.4 | -
\+ Delta Distillation |  |  35.7 | (2.2$\times$) |  |  33.6 | (1.2$\times$)
DDRNet23-slim |  |  36.6 | - |  |  6.6 | -
\+ Delta Distillation |  |  13.7 | (2.7$\times$) |  |  5.1 | (1.3$\times$)

While the amount of GFLOPs is agnostic to hardware, software implementation,
and measurement noise, it might not reflect latency gains exactly. In
particular, it does not capture the memory usage overhead, which can become a
limiting factor, especially for low-power devices. To this end, we hereby
report the latency of several image models for semantic segmentation, as well
as the one from the corresponding delta distilled models. More specifically,
we report runtime measurements on simulated hardware of a mobile SoC, as this
aligns well with the practical use cases of delta distillation. Indeed,
resource constrained devices represent a real-world setting where research on
efficient inference methods might contribute to significantly. We illustrate
such measurements in Tab. 5. In this experiment we opt, as a student
architecture, for the non-linear channel reduction compression (see Sec. 3.1
in the main paper for details). Specifically, for every architecture, we
compress a sequence of two residual blocks in a row into a cheaper student,
with a compression ratio $\gamma=4$. Although delta distillation increases
memory usage, due to transferring inputs and outputs of every block, we
appreciate how the GFLOPS gains translate to latency improvements. This
behavior is more evident for heavier architectures (_i.e_. DDRNet39 and
DDRNet23), where the latency speedups are in line with (or even better than)
the theoretical ones. Improvements can still be appreciated when distilling
cheaper architectures (_i.e_. HRNet-w18-small and DDRNet23-slim), although
they become less evident.

###  B.2 Per class IoU results

Figure 9: Per class IoU on Cityscapes for different DDRNet models. Note that
Delta distillation does not exploit any single class, but rather improves upon
several or retains overall IoU at a lower cost.

We report in Fig. 9 the per-class IoU metric on the Cityscapes validation set,
for three different variants of a DDRNet model, namely DDRNet23 slim (top),
DDRNet23 (middle) and DDRNet39 (bottom). The figure also reports performances
of the corresponding delta distilled variant, with students instantiated at
every linear block with a compression factor $\gamma=4$. Interestingly, the
figure testifies how the original and the compressed model behave similarily
on all classes. It is noteworthy how, for highly dynamic semantic classes such
as cars, trains or bycicles, the student model can successfully update the
representations from the teacher without any hurt in performance, showcasing
that retained performances of the cheaper student model do not simply come
from static classes. We remark how this result is achieved without relying on
any explicit motion compensation during training or inference.

###  B.3 Temporal Consistency Results

Figure 10: Additional results of DDRNet23 and Delta Distillation in
alternating rows respectively. We highlight large differences between methods
in the boxes with white dashed lines. We observe that delta distillation tries
to maintain large areas assigned to particular classes over time more so than
the baseline model.

In Fig. 10, we illustrate further examples of the improved temporal
consistency empowered by Delta Distillation. The figure, which serves as an
extension of Fig. 7 in the main paper, showcases some segmentation results on
the Cityscapes dataset, and it compares predictions from the original image-
based architecture (DDRNet23) with the corresponding delta-distilled model. As
it can be appreciated, especially by comparing predictions within the marked
dashed line boxes, Delta Distillation enjoys more temporal consistency in
outputs, in regions where per-frame predictions result flickery. We remark
that temporal consistency is not an explicit objective of Delta Distillation,
but rather a beneficial side effect enabled by its inference scheme, that
relies on the update of past representations, in a procedure that is
inherently more temporally smooth with respect to independent processing of
frames.

##  C Teacher and Student Parameters

Table 6: Specifications of the architectures used in Sec. 4 the main paper. We hereby report the Flops and the number of parameters separately for the teacher and student models. Task |  | Model |  | Flops | Params | Ref
---|---|---|---|---|---|---
|  | (G) | (M) | (paper)
Obj. Detection |  | EfficientDet-D0 | Teacher |  | 2.5 | 3.84 | Tab. 1
| Student |  | 0.5 | 1.12
| Amortized ($T$=10) |  | 0.7 | -
Obj. Detection |  | Faster-RCNN | Teacher |  | 149.1 | 81.22 | Tab. 1
| Student |  | 15.9 | 29.97
| Amortized ($T$=10) |  | 29.2 | -
Segmentation |  | DDRNet-39 | Teacher |  | 282.0 | 34.14 | Tab. 2
| Student |  | 68.9 | 8.09
| Amortized ($T$=3) |  | 140.0 | -
Segmentation |  | DDRNet-23 | Teacher |  | 143.7 | 20.79 | Tab. 2
| Student |  | 35.8 | 5.12
| Amortized ($T$=3) |  | 71.8 | -
Segmentation |  | DDRNet-23 slim | Teacher |  | 36.6 | 5.91 | Tab. 2
| Student |  | 8.5 | 1.47
| Amortized ($T$=3) |  | 17.9 | -

In all experiments in Sec. 4 in the main paper we report amortized GMACs as
computational cost measure. Therefore, the reported numbers illustrate the
_average_ cost required to perform a prediction and, due to the schedule we
apply at inference time assigning frames unevenly to different models, they
are affected by _i)_ the teacher cost, _ii)_ the student cost and _iii)_ the
schedule period $T$. For clarity and completeness, we report in Tab. 6 the
unamortized costs of the teacher and student for the main models used in the
paper. Additionally, the table reports the corresponding number of learnable
parameters.

##  D Future work

We recognize two limitations of our work, that we plan to tackle in future
work. First, delta distillation incurs some memory overhead: similarly to
recurrent models, feature maps need to be stored in order to be propagated to
the following time-steps. The corresponding impact on memory thus depends on
the granularity at which our model operates: it might be negligible when a few
(bigger) blocks are optimized, whereas it could become meaningful when the
distillation involves every layer of the original model. Although we did not
observe noticeable effects on latency (see Tab. 5) in our experiments, such an
overhead might be impactful for large models with many layers.

Moreover, our proposal is capable of reducing the amortized runtime of the
original network, yet it features unbalanced latencies between frames. Indeed,
the inference schedule assigns frames either to the teacher or student model:
frames processed by the former do not enjoy any latency reduction. We envision
this could be addressed by deploying the two models in separate threads, and
by implementing an asynchronous inference scheme similar to [27].

In conclusion, we acknowledge our proposal introduces a few additional hyper-
parameters into the network training schedule (_i.e._ $\alpha$, $\beta$),
whose determination required some trial-and-error based tinkering.


# Performer Identification From Symbolic Representation of Music Using
Statistical Models

###### Abstract

Music Performers have their own idiosyncratic way of interpreting a musical
piece. A group of skilled performers playing the same piece of music would
likely to inject their unique artistic styles in their performances. The
variations of the tempo, timing, dynamics, articulation etc. from the actual
notated music are what make the performers unique in their performances. This
study presents a dataset consisting of four movements of Schubert’s “Sonata in
B-flat major, D.960” performed by nine virtuoso pianists individually. We
proposed and extracted a set of expressive features that are able to capture
the characteristics of an individual performer’s style. We then present a
performer identification method based on the similarity of feature
distribution, given a set of piano performances. The identification is done
considering each feature individually as well as a fusion of the features.
Results show that the proposed method achieved a precision of 0.903 using
fusion features. Moreover, the onset time deviation feature shows promising
result when considered individually.

##  1 Introduction

Music performance is considered to be a creative process. Musicians often
express their emotions to the audience through music. The perceived emotions
are subjective to the audiences as the interpretations of the observed music
may vary from person to person [1, 2]. Virtuoso performers are often
recognized by people for their uniqueness in portraying their emotional
expressions in a music performance. In music, these unique emotional
expressions are characterised by some parameters such as tempo, dynamics and
articulation. And variations in these parameters make the performers
distinguishable and unique in their performances.

Expressive performance has been a central research topic in contemporary
musicology, hence a large research body has been devoted to exploring
expressive piano performance, by examining the performance parameters that are
used by pianists. So far, expressive timing and amplitude have been the most
explored features for their salience and effect on the perception of emotional
expression [3]. [4] developed a dataset consisting of 22 performers and used
expressive features like timing, loudness and articulation extracted from MIDI
performances, for automatic performer identification using machine learning
models. They have introduced ‘norm deviation’ features along with the ‘score
deviation’ features which shows that it is indeed possible for a machine to
distinguish performers based on their performance style. The Authors in [5, 6]
used the same features but extracted from audio recordings of piano
performances. [7] has studied individuality in expressive piano performance in
light of musical gestures, i.e., timing and dynamic patterns which can
characterize the individual expressive strategies. [8] studied different
timbral features to investigate individuality in piano performances. They
studied the pattern of timbral expressions induced by four pianists. The
result shows that pianists can express individual style through specific
timbral intentions.

In this paper, we study several statistical models and test them in the
performer identification task by quantifying expressive performance parameters
that are important in expressing the individuality of piano performers [4, 7,
8]. In other words, we use statistical models to identify piano performers
given a set of performances of the same piece of music by several virtuoso
pianists. This can be viewed as a classification problem where the classes are
the candidate pianists. We compare different classification methods using
features that are extracted from ‘norm performance’ introduced in [4]. Here,
‘norm performance’ refers to the average of the onset time, off time and
dynamic of each note across all performances in the dataset. This norm
performance is then considered as a reference point. The main performance
features that are accounted in this study are timing (variations in tempo),
dynamic level (variations in loudness) and articulation (the use of overlaps
and pauses). We conducted classification considering each feature individually
as well as a feature fusion method to consider the features in combination. We
also propose note duration feature which demonstrates promising result when
fused with other features. Our experimental result shows that it is possible
to distinguish pianists using the proposed features despite the limited
training data.

The rest of the paper is organised as follows: Section 2 introduces a novel
dataset consisting of 16980 notes for each performer. Section 3 describes the
feature data-prepossessing which includes MIDI-to-MIDI alignment, feature
extraction and selection process. In Section 4, we discussed the
classification experiments and results and Section 5 outlines the conclusion
and possible future developments.

##  2 DATASET

The data used in this study consists of performances played and recorded on a
Yamaha CFX concert Grand Piano which is equipped with state-of-the-art
Disklavier Pro recording technology. We chose 9 virtuoso pianists from the
International Piano-e-competition [9] who played the same four movements of
Sonata in B-flat Major, D960, I. Molto moderato, II. Andante sostenuto, III.
Scherzo: Allegro vivace con delicatezza – Trio and IV. Allegro ma non troppo –
Presto composed by Franz Schubert (see Table 1). This is one of the last major
compositions by Schubert for solo piano and considered among the most
important of Schubert’s mature masterpieces. Each movement has a different
number of notes as described in Table 1.

Composer | Piece | Movement | No.of Notes
---|---|---|---
F.Schubert |  Sonata in B-Flat Major,D960 | I | 7582
II | 2005
III | 2717
IV | 4676

Table 1: Details of Dataset with number of notes.

All the performances have been recorded in both raw audio and MIDI format. In
this experiment, we are mostly interested to work with the symbolic format of
the music which is MIDI. The advantage of MIDI over raw audio is that raw
audio needs manual annotation of the note events. This is very time consuming
and requires a lot of expertise. However, in MIDI, each note event is
explicitly described by four parameters: Onset (note on time), Offset (note-
off time), Pitch (numerical value ranges from 0 to 128) and dynamic level
(loudness of the note, ranges from 0 to 128). Hence, manual annotation of
notes is no longer required.

##  3 METHODOLOGY

In order to explore pianist’s individuality in expressive performances, our
study is designed with respect to the following steps: Alignment of performed
musical pieces based on their notes to calculate the ‘norm performance’,
extraction of meaningful norm based performance parameters and finally the
classification process using the performance features.

Figure 1: Five parameters that are considered to characterize note-level
performance details

###  3.1 MIDI-to-MIDI Alignment

Symbolic music alignment is a process of automatically matching a note in a
music performance with a corresponding note in a score or a reference
performance. We use a HMM (Hidden Markov Model) based symbolic music alignment
algorithm proposed by [10]. The algorithm achieved high accuracy with a low
computational cost. Initially it aligns two signals, reference and
performance, using hidden Markov models (HMMs) and detects the performance
errors from initial alignment result and then uses a merged-output HMMs [11]
to correct the errors as a post-processing step.

There are two modes of the alignment process: the first one is, Score-to-MIDI
alignment where the score file is a MusicXML file with no performance
parameters and the second one is the MIDI-to-MIDI alignment. The MIDI-to-MIDI
alignment algorithm uses any two midi files to find the corresponding notes
between them. One of them can be used as a reference signal and the other can
be used as the performance signal. The reference signal is first converted
into a score file and the score-to-MIDI alignment algorithm is then used for
the converted score and the performance MIDI file.

Figure 2: Dynamic variations for the first 20 notes of the Sonata in B flat
major,D960, Mvmnt. II, as performed by pianists p01-p09

###  3.2 Feature Extraction

Expressive performance can be defined as, either the intended deviation from
the score, which is purely a mechanical rendition of the musical piece in
terms of tempo, dynamic and articulation or the average performance
considering it as a reference point. Previously, [12] have used score
deviation features to successfully discriminate two skilled performers playing
the same piano piece. But later a comprehensive and empirical study in [13]
revealed that deviation from the average performance is more powerful in
representing performer’s individuality than the deviation from the printed
score. Also, the empirical results shows that the norm based features are
proved to be very accurate for intra-piece tests (training and test set taken
from the same piece) and inter-piece tests (training and test set taken from
different pieces). We take the later approach which is to calculate the note
level performance deviations from an average performance. The idea of
performance norm can be better understood from Figure 2, where the bold line
denotes the average performance calculated from pianists p01-p09 in terms of
note dynamics. We can see that, the norm performance follows a basic form of
individual performances but as illustrated in Figure 3 the deviations from the
norm does not necessarily follow a similar form (peaks and dips of notes) and
the differences are clearly highlighted.

Figure 3:  Velocity deviations of first 20 notes from norm performance as
performed by pianists p01-p09.

####  3.2.1 Proposed Features

In order to represent the stylistic properties of a performer from their
performance, we propose the following set of features, given the performance
norm derived from the set of different performers:

  * $-$

Norm deviation features:

    * $\bullet$

d(OTn,OTp)

    * $\bullet$

d(IOIn,IOIp)

    * $\bullet$

d(OTDn,OTDp)

    * $\bullet$

d(DLn,DLp)

    * $\bullet$

d(NDn,NDp)

Here, d(xn,yp) denotes the deviation of a vector of numerical values y from a
vector of values x where n and p refers to the reference vector and the actual
performance respectively. The parameters can be better understood from Figure
1, where, OT (Onset time) denotes the start of a note, IOI (Inter Onset
Interval) denotes the time interval between the onsets of two notes, OTD
(Offset Time Duration) denotes the time interval between the end of a note and
the onset of the next note, DL (Dynamic level) denotes the loudness of a note
and ND (Note duration) denotes the exact duration of a note. In addition, we
calculated the cross-correlation between the individual features using
Pearsons correlation coefficient (r). We found that OTD and ND to be highly
correlated (r$>$0.9) with each other. However, we used both of the features in
our study to find the most effective feature for performer identification
task.

There are other performance related features that could be considered but
ignored in this study. This includes melody lead [14] which is used by music
performers to emphasize a voice over others. An empirical study was conducted
by [15] which shows that melody lead has a great impact on expressiveness.
This constitutes future work for us. Different distance types can be
considered in order to measure the deviation in each features defined in the
previous subsection. The distance metrics that are considered in this study
are formulated below:

$Simple:d_{s}(x,y)=(x_{i}-y_{i})$ (1)

$SimpleAbsolute:d_{ab}(x,y)=(|x_{i}|-|y_{i}|)$ (2)

The experimental result shows that, the simple distance metric formulated in
equation 1 best fits for the OT deviation, DL deviation and ND deviation
features where the simple absolute equation 2 best fits for the IOI and OTD
deviation features.

###  3.3 Pianist Classification

In this section, we describe the pianist classification that includes the
feature distribution estimation and similarity calculation. The identification
method is inspired by the method introduced in [16]. We first discuss the
methods in terms of each features and then, we discuss a feature fusion
technique in Section 3.3.2.

####  3.3.1 Using Features Individually

As depicted in Figure 2, the dynamic variation for 9 performers who played the
same piece of music shows that different performers playing the same music
will always have their own way of expressing the piece to the audience. The
average performance line in bold indicates the performers understanding of the
structure of the music while the individual peaks and dips shows how much each
performer deviates from the average performance. These deviations characterise
each performer’s individually. Therefore, we model these distinct
characteristics of performers using the distribution of each features proposed
in subsection 3.2.1. To model these deviation distributions we use Histogram,
Kernel densities (KDE) and Gaussian Mixture Models (GMMs) similarly to [16,
17]. We assume that, the distribution of the deviations provide us compact
representation of performers idiosyncratic style.

(a) Pianist 1

(b) Pianist 2

Figure 4: Distribution of two performer’s Inter-Onset Interval (IOI)
deviations.

Expressive timing is one of the most important performance features for
performer identification. It has great impact on performers playing style.
Performers vary the tempo throughout the performance to express their emotions
distinctively. Figure 4 shows the inter-onset interval deviation distribution
of two performers calculated using Histogram, KDE and GMMs. We can observe
that deviations are mostly concentrated between -0.01 ms and 0.03 ms for
pianist 1 whereas it is between -0.04 ms and 0 ms for pianist 2. This
indicates that pianist 1 prefers to perform with faster tempo whereas pianist
2 prefers to perform with relatively slower tempo than player 1. Based on the
similar observations with the rest of the features, onset time deviation, off-
time duration deviation, dynamic level deviation and note duration deviation,
it can be presumed that these features can reflect important aspects of the
pianist’s individual characteristics. The red dotted line in Figure 4
represents the Gaussian Mixture Models(GMMs). We trained the GMMs with 2,3,5
and 7 components and the empirical results show that GMMs do not require more
than 3 components to model the distributions. The red dotted PDF curve of the
GMMs in Figure 4 also shows that, it represents the Histograms and KDEs well.
The blue line in Figure 4(a) shows the PDF of the KDE. The bandwidths of KDEs
to model the OT deviation, IOI deviation, OTD deviation, DL deviation and ND
deviations are 1.2, 0.01, 0.02, 1.5 and 0.01 respectively. This Figure also
shows that, the PDF curves of both performers show similar properties to the
histogram.

In order to quantify the differences between performers from the
distributions, we take a similarity measurement step of each feature
distribution for every performer in our dataset using Kullback-Leibler (KL)
divergence[18] as shown below:

$D_{KL}(P{\parallel}Q)=\sum_{x\in\chi}P(x)log(\frac{P(x)}{Q(x)})$ (3)

The KL-divergence which is also known as relative entropy, is a measure of how
one probability distribution is different from another probability
distribution or in other words, from equation 3 we can say, it computes the
likelihood ratio between two distributions and tells how probability
distribution Q diverges from the probability distribution P by computing the
cross-entropy minus the entropy. We measure the KL-divergence for Kernel
densities using the approach introduced in [19]. We also calculate the KL-
divergence of two GMMs, but since the KL-divergence for two GMMs has no closed
form expression, it is not analytically tractable. Hence, we use a variational
Bayes approximation method [20] to avoid this issue.

We use KL-divergence to calculate the divergence between an unknown
performer’s feature distribution and every known performer’s feature
distribution in the dataset in order to classify the unknown performer. The
minimum divergence between the unknown and the known performer’s distribution
classifies the unknown performer.

####  3.3.2 Feature Fusion Method

Besides classification using individual features, we use feature fusion
techniques that are able to combine multiple features (see subsection 3.2.1).
Not all features will have a similar importance for any classification task.
Features with less importance may lead to a suboptimal classification result.
Hence, weights can be used to increase the influence of good features and
decrease the influence of bad features. In this study, we used linear
combination with equal weights to fuse similarity estimates for the
distributions of different features. Optimizing weights as done in [21, 22,
23, 24], for individual features could be a potential future work. We use
Leave one group out cross validation technique with 8 folds to calculate the
KL-divergence between the training and testing set for every group of data. We
then combine the similarity estimation of feature distributions in each fold
for different types of features. This is formulated in equation 4:

$KL_{total}=\sum_{i=1}^{|N|}w_{i}KL_{N_{i}}$ (4)

where, N = {N1, N2, N3, N4, N5} denotes the set of statistical models
corresponds to OT deviation, IOI deviation, OTD deviation, DL deviation and ND
deviation features respectively (see subsection 3.2.1) which are computed
separately. $w_{i}$ denotes the corresponding feature weight which is set to 1
in our experiment. However, the feature fusion technique used in this study is
not unique. We can combine any 2, 3, 4 or 5 features together to calculate the
overall KL-divergence. In the next section, we describe several experiments
with both single feature and fused features to validate the pianist
identification methods and determine how accurate the methods are for pianist
classification.

Model | Feature
---|---
OT | IOI | OTD | DL | ND
Histogram | 0.603 | 0.459 | 0.457 | 0.533 | 0.398
KDE | 0.688 | 0.489 | 0.342 | 0.589 | 0.498
GMM | 0.591 | 0.476 | 0.267 | 0.564 | 0.405

Table 2: Pianist identification result in terms of Precision.

##  4 EXPERIMENTS

In this section, we show the performer classification results based on each
feature: note onset time deviation, inter-onset interval deviation, off-time
duration deviation, dynamic level deviation and note duration deviation as
well as the combination of these features using the fusion method described in
Section 3.3.2. We also verify the effectiveness of the proposed performer
identification methods using leave one group out cross validation technique.
We then calculate the F-measure which is a way to combine both precision and
recall into single measure that captures both the properties. A normalized
confusion matrix is also used to show the performance of identification for
all the performers in our dataset.

Feature | Precision | Recall | F-score
---|---|---|---
OT | 0.688 | 0.667 | 0.677
IOI | 0.489 | 0.472 | 0.480
OTD | 0.342 | 0.375 | 0.357
DL | 0.589 | 0.542 | 0.564
ND | 0.498 | 0.458 | 0.477

Table 3: Pianist identification result using KDE. Numbers in bold denote the
best result.

###  4.1 Classification result based on individual features

In our experiment, a total of 16980 aligned notes has been extracted from each
pianist’s performance. Since, we consider the note level local deviation
features in our experiment, the same amount of deviation features are also
extracted from the notes. We perform leave one group out cross validation to
separate each performers data into 8 groups to maintain a high number of cross
validation folds as well as to ensure there are enough data in every test set.

Hence, 2122 notes are designated for each of the first 7 folds and the last
fold contains 2126 notes. We then select a random performer out of the 9
performers, and designate one group of data from that performer as test data.
The rest of the groups of all

Figure 5: Normalized confusion matrix for pianist classification using OT
deviation.

performers are considered as training data. The distributions of both the test
and training set are calculated using Histogram, KDE and GMMs. The Histogram
bins, Kernel density bandwidth and the GMM hyper-parameters are optimised in
this experiment and optimum values for each feature are kept constant.

Finally, we calculate the KL-divergence between the test distribution and the
training distribution for every performer in the dataset in order to measure
the similarity. The minimum KL-divergence value identifies the unknown
performer. In other words, we can say that, the corresponding performer’s
training distribution which has the minimum distance with the test
distribution is the identified performer. Table 2 shows the identification
result in terms of precision for each feature. The results in bold number
denotes the best precision achieved by the models for each feature. We can see
that the OT deviation feature performs best when considered individually out
of all the features for all the models. And, KDE performs the best out the
other two distribution models. A normalized confusion matrix is also shown in
Figure 5 for OT deviation using KDE. The x-axis corresponds to the predicted
performers label and the y-axis corresponds to the true performers label.

###  4.2 Classification Result Using the Fusion of Features

As discussed in Section 3.3.2, feature fusion is a method of combining two or
more features to remove redundant and irrelevant features for better
classification accuracy. In this section we combine two or more features and
fuse them together to make a new feature set which are tested against each
statistical models that are described in the following subsections.

####  4.2.1 Histogram Based Classification Results

As we see from Table 2, OT deviation and DL deviation features performs better
than the other features when considered individually. Hence, it would be
practical to consider a combination of features and test them against each
models. We use different combination of the features and Histrogram is used
for the classification.

Figure 6: Normalized confusion matrix for pianist classification using fusion
features for Histogram.

Feature | Precision | Recall | F-score
---|---|---|---
DL+ND | 0.728 | 0.653 | 0.688
IOI+DL | 0.830 | 0.778 | 0.803
IOI+DL+ND | 0.903 | 0.875 | 0.889
OTD+DL+ND | 0.746 | 0.694 | 0.719
IOI+OTD+DL+ND | 0.891 | 0.861 | 0.875

Table 4: Pianist identification using Histogram for different combination of
features.

The best five combination for which the Histogram model produces best result
is shown in Table 4. The results in bold correspond to the best result. The
F1-score is also calculated from the precision and recall. A corresponding
confusion matrix for the fusion of IOI deviation, DL deviation and ND
deviation is shown in Figure 6.

####  4.2.2 KDE Based Classification Results

We use Kernel density estimation (KDE) for different fusion features. Table 2
shows that KDE outperforms the other two distribution models for piano
identification considering the individual features. Hence, it is also sensible
to experiment with different fusion features to notice if they produce any
better result.

Feature | Precision | Recall | F-score
---|---|---|---
IOI+DL | 0.756 | 0.722 | 0.739
IOI+OTD+DL | 0.756 | 0.722 | 0.739
IOI+DL+ND | 0.822 | 0.792 | 0.807
OT+OTD+DL+ND | 0.690 | 0.667 | 0.678
OT+IOI+OTD+DL+ND | 0.692 | 0.681 | 0.686

Table 5: Pianist identification using KDE for different combination of
features.

We experimented with all the combination of features without any overlap and
at least two features are considered. The best five combinations for which the
KDE produces the best results are shown in Table 5. The IOI deviation feature,
fused with dynamic level deviation and ND deviation produces the best result
which is shown in bold numbers.

####  4.2.3 GMM Based Classification Results

We use Gaussian Mixture Model (GMM) based performer identification method
using fusion features. We use all the combination of timing deviation features
as well as the articulation features.

Feature | Precision | Recall | F-score
---|---|---|---
IOI+DL | 0.719 | 0.653 | 0.684
IOI+OTD+DL | 0.618 | 0.528 | 0.569
IOI+DL+ND | 0.618 | 0.528 | 0.569
OTD+DL+ND | 0.583 | 0.528 | 0.554
OT+IOI+OTD+DL+ND | 0.591 | 0.611 | 0.601

Table 6: Pianist identification using using GMM for different combination of
features.

We can see from Table 6 that, the IOI deviation feature fused with the DL
feature outperforms the other fused features.

##  5 Analysis and Discussion

As discussed in Section 4, there are two main methods used for pianist
identification. First, we classify the performers using three distribution
models considering the individual features and second, we use a feature fusion
method to identify each performers. Table 2, shows the pianist identification
result for each feature in terms of precision. We observed that KDE performs
better for each individual feature except for the OTD deviation feature. No
matter which model we use, OT deviation feature tends to perform better out of
all other features. The highest precision is achieved using the KDE
distribution (0.688). Also in Table 3, we show the precision, recall as well
as the calculated macro F-score result based on single feature. The result
also shows that OT deviation feature outperforms the other features when
considered individually with a F-score of 0.677 where the OTD deviation
feature performs worst (0.357). To elaborate more, we generate a normalized
confusion matrix (Figure 5) for OT deviation using Kernel density estimation
(KDE). The Figure shows that the identification of the performers: Johannson,
Kotys and Krasnitsky are the best(0.88) where, the identification of DeTurk,
Mordvinov and Savitski are also good (0.75). These performers can be
identified correctly most of the time. Rozanski also performs well (0.50) but
sometimes confused with Johannson. The identification of Abdelmoula and
ChengGuang are mostly unreliable, as we can see their performances are
confused with each other as well as other performers. This is due to sharing
similar OT deviation feature distributions. Apart from this, the OT deviation
feature performs reasonably well for identifying the performers most of the
time. However, certain performers can not be identified correctly since they
may share the similar feature characteristics.

Considering this problem, we use feature fusion method (Section 3.3.2) for the
performer identification task. Results show that this method works really well
identifying the performers correctly. Histogram performs better than other two
distribution models and yields a precision of 0.903 for the fusion of IOI, DL
and ND deviation features as shown in Table 4. Moreover, the confusion matrix
in Figure 6 shows that the fusion feature is able to identify each performer
more accurately than using only OT deviation feature. Table 4 also shows that
most of the fusion features perform better than the individual features. The
features proposed in this study performed better than the features in [16],
although the identification method as well as some of the features used are
quite similar. The probable reason could be that the dataset sizes are
different. This could be due to the differences in dataset sizes and different
performance features. Moreover, features in this study are extracted from MIDI
file whereas they used audio data which might have resulted the deviations to
be more noisy. This exhibits the power of our proposed model for the performer
classification task.

A potential future direction to improve the performance of the classification
would be to consider both low and high level features described in [25]. Low
level features have received by far the most attention by the MIR researchers
whereas the high level features (instruments present, melodic contour, chord
frequencies and rhythmic density) have been neglected and mostly used for
audio format. Meanwhile, research involving high level features show that they
also exhibit a significant discriminating power [26]. In addition, a weight
optimization can also be done to fuse features with different weights. Another
potential interesting future direction could be to segment a sample into parts
of equal length and apply the presented methodology to each parts rather than
the whole sample. Moreover, features extracted from other data modalities
(motion data obtained from sensors) as presented in [27, 28] could be used to
potentially contribute in performer identification task. Furthermore, other
classification methods like KNN (K-nearest Neighbors), Decision tree, SVMs and
Neural Networks can be tested to improve the classification result. Finally,
we aim to use a larger dataset consisting of different composers.

##  6 CONCLUSION

In this paper, we presented an identification method for the recognition of
virtuoso pianists using fusion of features extracted from symbolic
representation of music. We also constructed a dataset consisting of nine
virtuoso pianists, each playing the same piece of music. Our result shows the
proposed fusion features to be robust in the exceptionally challenging task of
performer identification which normally requires trained/expert human
listeners in most situations. The classification accuracy results produced by
our model is much higher than the results obtained in previous studies. It has
also been demonstrated that the ‘norm performance’ works reasonably well as a
reference point. Therefore, it can be sensible to use the norm performance as
a reference point where many performances of the same piece are available. The
results also show that features related to expressive timing and loudness are
the most informative when fused together followed by the aspect of note
duration.

## References

  * [1] M. Barthet, G. Fazekas, and M. Sandler, “Multidisciplinary Perspectives on Music Emotion Recognition: Implications for Content and Context-Based Models,” in _International Symposium on Computer Music Modeling and Retrieval_. Springer, 2012.
  * [2] J. Russell, “A Circumplex Model of Affect,” _Journal of Personality and Social Psychology_ , vol. 39, pp. 1161–1178, 12 1980.
  * [3] A. Bhatara, A. Tirovolas, L. Duan, B. Levy, and D. Levitin, “Perception of Emotional Expression in Musical Performance,” _Journal of experimental psychology. Human perception and performance_ , vol. 37, pp. 921–34, 06 2011.
  * [4] E. Stamatatos and G. Widmer, “Automatic identification of music performers with learning ensembles”,” _Artificial Intelligence_ , vol. 165, no. 1, pp. 37 – 56, 2005.
  * [5] C.-i. Wang, “Quantifying Pianist Style - An Investigation of Performer Space and Expressive Gestures from Audio Recordings,” _Master’s thesis, New York University_ , 02 2013.
  * [6] C. Saunders, D. R. Hardoon, J. Shawe-Taylor, and G. Widmer, “Using String Kernels to Identify Famous Performers from Their Playing Style,” in _Machine Learning: ECML 2004_ , J.-F. Boulicaut, F. Esposito, F. Giannotti, and D. Pedreschi, Eds. Berlin, Heidelberg: Springer Berlin Heidelberg, 2004, pp. 384–395.
  * [7] J. Rink, N. Spiro, and N. Gold, “Motive, gesture and the analysis of performance,” _New Perspectives on Music and Gesture_ , pp. 267–292, 01 2011\.
  * [8] M. Bernays and C. Traube, “Investigating pianist’s individuality in the performance of five timbral nuances through patterns of articulation, touch, dynamics, and pedaling,” _Frontiers in Psychology_ , vol. 5, p. 157, 2014\. [Online]. Available: https://www.frontiersin.org/article/10.3389/fpsyg.2014.00157
  * [9] B. Alexander and Y. Corporation, “International Piano-e-competition,” Available: http://www.piano-e-competition.com/, 2002, last checked on Oct 01, 2020.
  * [10] E. Nakamura, K. Yoshii, and H. Katayose, “Performance Error Detection and Post-Processing for Fast and Accurate Symbolic Music Alignment.” in _International Society for Music Information Retrieval Conference, ISMIR_ , 2017, pp. 347–353.
  * [11] E. Nakamura, N. Ono, Y. Saito, and S. Sagayama, “Merged-output hidden Markov model for score following of MIDI performance with ornaments, desynchronized voices, repeats and skips,” _algorithms_ , vol. 21, p. 8, 2014.
  * [12] E. Stamatatos, “A computational model for discriminating music performers,” in _Proceedings of the MOSART Workshop on Current Research Directions in Computer Music_ , 2001, pp. 65–69.
  * [13] ——, “Quantifying the differences between music performers: Score vs. norm,” in _International Computer Music Conference, ICMC_. Citeseer, 2002.
  * [14] W. Goebl, “Melody lead in piano performance: Expressive device or artifact?” _The Journal of the Acoustical Society of America_ , vol. 110, no. 1, pp. 563–572, 2001.
  * [15] C. Palmer, “On the Assignment of Structure in Music Performance,” _Music Perception: An Interdisciplinary Journal_ , vol. 14, no. 1, pp. 23–56, 10 1996\.
  * [16] Y. Zhao, G. Fazekas, and M. Sandler, “Identifying master violinists using note-level audio features,” in _Proceedings of the 17th Sound and Music Computing Conference_ , Torino, Italy, 06 2020.
  * [17] Y. Zhao, C. Wang, G. Fazekas, E. Benetos, and M. Sandler, “Violinist identification based on vibrato features,” in _Proc. of the European Signal Processing Conference (EUSIPCO), Dublin, Ireland, 23-27. Aug._ , 2021.
  * [18] S. Kullback and R. A. Leibler, “On Information and Sufficiency,” _The Annals of Mathematical Statistics_ , vol. 22, no. 1, pp. 79–86, 03 1951. [Online]. Available: https://doi.org/10.1214/aoms/1177729694
  * [19] H. Sasaki, Y.-K. Noh, G. Niu, and M. Sugiyama, “Direct Density Derivative Estimation,” _Neural Computation_ , vol. 28, no. 6, pp. 1101–1140, 2016, pMID: 27140943. [Online]. Available: https://doi.org/10.1162/NECO_a_00835
  * [20] J. R. Hershey and P. A. Olsen, “Approximating the Kullback Leibler Divergence Between Gaussian Mixture Models,” in _2007 IEEE International Conference on Acoustics, Speech and Signal Processing - ICASSP ’07_ , vol. 4, 2007, pp. IV–317–IV–320.
  * [21] J. Yang, J.-Y. Yang, D. Zhang, and J. Lu, “Feature fusion: Parallel strategy vs. serial strategy,” _Pattern Recognition_ , vol. 36, pp. 1369–1381, 06 2003.
  * [22] J. Yang and J.-Y. Yang, “Generalized KL transform based combined feature extraction,” _Pattern Recognition_ , vol. 35, pp. 295–297, 01 2002.
  * [23] P. Mitra, C. A. Murthy, and S. K. Pal, “Unsupervised feature selection using feature similarity,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , vol. 24, no. 3, pp. 301–312, 2002.
  * [24] D. Sheng and G. Fazekas, “Feature selection for dynamic range compressor parameter estimation,” in _Audio Engineering Society Convention 144_. Audio Engineering Society, May 2018\. [Online]. Available: http://www.aes.org/e-lib/browse.cfm?elib=19514
  * [25] C. Mckay and I. Fujinaga, “JSymbolic: A feature extractor for MIDI files,” _International Computer Music Conference, ICMC_ , 01 2006.
  * [26] C. McKay and I. Fujinaga, “Automatic music classification and the importance of instrument identification,” in _Proceedings of the Conference on Interdisciplinary Musicology_ , 2005, pp. 1–10.
  * [27] B. Liang, G. Fazekas, and M. Sandler, “Transfer Learning for Piano Sustain-Pedal Detection,” in _2019 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 2019, pp. 1–6.
  * [28] ——, “Measurement, Recognition, and Visualization of Piano Pedaling Gestures and Techniques,” _Journal of the audio engineering society_ , vol. 66, no. 6, pp. 448–456, june 2018.