# Task-Sensitive Concept Drift Detector with Constraint Embedding

Andrea Castellani  CITEC, Bielefeld University
Bielefeld, Germany
<EMAIL_ADDRESS>Sebastian Schmitt  Honda Research
Institute Europe GmbH
Offenbach, Germany
<EMAIL_ADDRESS>Barbara Hammer  CITEC, Bielefeld University
Bielefeld, Germany
<EMAIL_ADDRESS>
###### Abstract

Detecting drifts in data is essential for machine learning applications, as
changes in the statistics of processed data typically has a profound influence
on the performance of trained models. Most of the available drift detection
methods are either supervised and require access to the true labels during
inference time, or they are completely unsupervised and aim for changes in
distributions without taking label information into account. We propose a
novel task-sensitive semi-supervised drift detection scheme, which utilizes
label information while training the initial model, but takes into account
that supervised label information is no longer available when using the model
during inference. It utilizes a constrained low-dimensional embedding
representation of the input data. This way, it is best suited for the
classification task. It is able to detect real drift, where the drift affects
the classification performance, while it properly ignores virtual drift, where
the classification performance is not affected by the drift. In the proposed
framework, the actual method to detect a change in the statistics of incoming
data samples can be chosen freely. Experimental evaluation on nine benchmarks
datasets, with different types of drift, demonstrates that the proposed
framework can reliably detect drifts, and outperforms state-of-the-art
unsupervised drift detection approaches.

###### Index Terms:

Concept Drift, Unsupervised, Deep Learning, Embedding Representation,
Clustering

##  I Introduction

††©2021 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

In the context of data-driven machine learning, and data mining applications,
the dataset used for calibrating or training the models plays a central role.
Its statistical properties define the model’s behavior and introduce implicit
or explicit assumptions. In many situations, it is assumed that the
distribution of the data streams is stationary, i.e. not changing over time,
which is a valid assumption if the distribution of the data used to
calibrate/train the model is the same as later during production use. However,
due to various reasons, such as aging or slight system re-configurations, this
assumption is not met in many real-world applications [1]. Usually, a drift
has a strong impact on model performance if it occurs unexpectedly.

In literature, this phenomenon is usually referred to as concept drift [1] and
it has received a lot of attention in the past few years [2, 3, 4]. Concept
drift is defined as the change in the joint distribution of a set of input
variables $\bm{x}$ and target variable $\bm{y}$ over time, i.e.
$P_{t_{0}}(\bm{x},\bm{y})\neq P_{t_{1}}(\bm{x},\bm{y})$ where $t_{0}$ and
$t_{1}$ are usually training and testing time. We focus on classification
tasks, where the output variables $\bm{y}$ represent class labels. Two
fundamentally different categories exist for concept drift. Virtual drift [1]
refers to changes in the distribution of the input data $\bm{x}$, without
affecting the distribution of the classification labels, i.e.
$P_{train}(\bm{x})\neq P_{test}(\bm{x})$ but
$P_{train}(\bm{y}|\bm{x})=P_{test}(\bm{y}|\bm{x})$. Real concept drift refers
to any change in the distribution $P(\bm{y}|\bm{x})$ which affects the
classification labels. Another crucial aspect for concept drift is given by
the speed of the changes. A gradual concept drift characterizes the case where
the transition occurs smoothly over time, while for a step concept drift the
switch between two contexts occurs abruptly.

Concept drift detection is an important task for many real-world
classification tasks, such as anomaly detection, fraud detection or monitoring
the electricity load profiles of an industrial facility [5, 6, 7], as it can
provide additional information to improve the performance. Interestingly, the
majority of concept drift detectors operate in a supervised fashion and
require immediate access to ground-truth labels during classifier training and
testing/inference. But for many of the aforementioned applications, the
acquisition of true labels is intractable or at least very expensive.
Therefore, labels are possibly only available for training and testing where
they can be utilized for building the classifier model, but they are typically
not available during inference in the real application scenario. These type of
situations are addressed by unsupervised or semi-supervised drift detection
methods, which is a much less explored research area [2]. Moreover, there
exist fully unsupervised drift detection methods which aim for anomalies of an
underlying data distribution, but do not take into account labels available
during training [8].

Figure 1: Proposed Concept Drift architecture.

In this work, we propose a task-sensitive concept drift detection framework,
where classification labels are only necessary during training of the
classifier. We learn a meaningful low-dimensional embedding representation of
the input data, which is best suited for the classification task.
Additionally, we employ a constraint in that embedding representation, which
forces the latent representation to have small intra-class and large inter-
class distances, and hence it optimally represents the given class structure.
After training, concept drifts can be detected by monitoring the distances of
incoming new data samples to the learned class centroids in the embedding
space. This is highly sensitive to the trained classification task, but does
not need true class labels during inference. The actual detection of the drift
can be performed by analyzing the distance statistics derived from the
embedding representation with any state-of-the-art (SotA) drift detection
method. The proposed concept drift architecture is reported in Fig. 1.

In addition, we provide a number of minor improvements: (i) two simple drift
detection methods based on the exponential moving average (EMAD), and a
modified $z$-score (ZSD); (ii) a novel evaluation metric ($H$-score), which
accumulates three SotA drift detection metrics into one performance measure;
(iii) an extensive evaluation of the proposed framework including ablation
studies on the introduced hyperparameters on various datasets for different
types of induced concept drift.

##  II Related work

The studies on the concept drift can be divided in two groups [3]:
performance-based and data distribution-based approaches. Performance-based
techniques aims at tracking changes in the error rate of a model. These
techniques require access to the ground-truth labels. Popular algorithms are
Drift Detection Model (DDM) [9] and Adaptive Windowing (ADWIN) [10]. Data-
distribution based approaches monitor distribution changes in the data, and
are based on statistical tests for distribution similarity. An example are
Hellinger Distance Drift Detection Model (HDDDM) [11] and Incremental
Kolomogorow-Smirnov test (IKS) [8]. Those algorithms can work with unlabeled
data, but they are quite sensitive to data changes and, considering solely
input data, bears the risk of detecting drifts in features that may be not
important for the prediction model, i.e. wrongly detect virtual drifts. The
proposed framework targets to only detect real drifts, i.e. changes in the
input data that also have an impact on the classification results.

There have been few tentative attempts to develop unsupervised concept drift
detection methods by using neural networks. In [12], the authors use the
reconstruction error of an autoencoder to detect changes in the data. In [13],
they use a contrastive loss to learn a low-dimensional embedding of data. Both
of those methods, are sensitive to virtual drifts because they rely solely on
raw input data and ignore the classification task. The internal representation
learned by neural network is often utilized also for other purposes, rather
than only fulfilling the main classification task. In [14] clustering is
applied to the learned representation of the data, rather than in the original
space. The authors of [15] and [16] propose to detect drift by using the
embedding of a neural network classifier. This method can correctly ignore
virtual drifts, but it needs to compute a reference statistic on the training
data, which could be difficult to compute while working with large datasets.
In contrast, we use very simple distance statistics in the embedding
representation and derive the reference statistics from a small data sample
with an additional simple update rule to detect drifted samples.

##  III Proposed Framework

###  III-A Constrained Embedding Representation

We assume that the dataset consists of $n$ data samples
$(\bm{x}_{i},\bm{y}_{i})$ with $i=1,\dots,n$, where
$\bm{x}_{i}\in\mathbb{R}^{q}$ represents $q$-dimensional raw input data and
$\bm{y}_{i}\in\\{0,1\\}^{k}$ the corresponding ground truth label, indicating
to which of the $k$-classes the sample belongs (one-hot encoding).

We train a neural network that consists of an encoder part $f_{e}$, which
transforms the input data $\bm{x}_{i}$ into a low-dimensional embedding
representation, $\bm{e}_{i}=f_{e}(\bm{x}_{i})\in\mathbb{R}^{d}$ with $d\ll q$.
A classifier neural network $f_{c}$ takes this low dimensional embedding and
produces a prediction of the class labels as output,
$\bm{p}_{i}=\text{softmax}\big{(}f_{c}(\bm{e}_{i})\big{)}$. The loss function
for the classification task is then given by the cross entropy,

$\mathcal{L}_{c}=-\frac{1}{n}\sum_{i=1}^{n}\bm{y}_{i}^{T}\cdot\log(\bm{p}_{i}).$
(1)

Additionally, we include a constrained embedding module $f_{ce}$, [7] which
forces the latent representation to have small intra-class and large inter-
class distances. For this, we first initialize the $k$ centroids
$\bm{C}_{j}\in\mathbb{R}^{d}$ with $j=1,\dots,k$, in the d-dimensional
embedding space randomly. Then, we iteratively adapt $\bm{C}_{j}$ in order to
minimize the intra-class and maximize the inter-class distances. The loss
function is given by [7]:

$\mathcal{L}_{ce}=\frac{1}{n}\sum_{i=1}^{n}\Bigg{[}{\underbrace{\left\|f_{e}(\bm{x}_{i})-\bm{C}_{\bm{y}_{i}}\right\|^{2}_{2}}_{\text{intra-
class}}}+\\\
+{\underbrace{\log\sum_{j=1}^{k}\exp\left(-\left\|f_{e}(\bm{x}_{i})-\bm{C}_{j}\right\|_{2}\right)}_{\text{inter-
class}}}\Bigg{]}+\ell_{reg},$ (2)

where $\bm{C}_{\bm{y}_{i}}$ indicates the centroid representing the ground
truth class label of sample $i$. The regularization term
$\ell_{reg}=-\sum_{l}^{k}\min_{l\neq
j}\log\left\|\bm{C}_{l}-\bm{C}_{j}\right\|_{2}$ aims to create well separated
embedding for different classes.

The final total loss function is given by the sum of those contributions:
$\mathcal{L}=\mathcal{L}_{c}+\mathcal{L}_{ce}$. During training, all
parameters of the neural networks, as well as the constraint clustering, are
learned and the procedure is summarized in Algorithm 1.

###  III-B Task-Sensitive Concept Drift Detection Framework

In order to detect drifting samples during inference, without access to
ground-truth labels, we take the fully trained classifier $f_{c}$ with the
constrained embedding $f_{ce}$, described in the previous section, and analyze
the data statistics in the embedding space. As the constrained embedding
representation is trained with the classifier, we expect that the drift
detector is able to detect drifts only if it is affecting the classification
task, i.e. real task-sensitive drift. Since the embedding space is tailored
toward a compact representation of classes, we rely on particularly simple
distance-based statistics to describe samples within the embedding space.

Let $\bm{x}_{i}\in\bm{X}^{test}$ be a sample of a data stream, the idea is to
calculate the Euclidean distances of each sample to the class centroids
$\bm{C}_{j}$ in the latent space:

$\bm{d}_{i,j}=\|f_{e}(\bm{x}_{i})-\bm{C}_{j}\|_{2}\,.$ (3)

The centroids $\bm{C}_{j}$ are fixed after the training and the distances are
expected to be small w.r.t. the centroid of the predicted class, and large to
the centroids representing the other classes. Therefore, we extract different
statistical feature vectors $\bm{m}_{i}$ from the distance statistics of
sample $i$ w.r.t. all centroids, $\\{\bm{d}_{i,j}\\}_{j=1}^{k}$, and use them
for drift detection.

The actual drift detection is done by comparing the feature statistics of each
sample to a reference distribution, $\\{\bm{m}^{ref}\\}$, which is calculated
from some small reference data set and which thereby defines the non-drifted
data. Then, the statistics $\bm{m}_{i}$ for the sample $i$, is compared
against $\\{\bm{m}^{ref}\\}$ by a change detection method $\mathcal{DM}$:

$o_{i}=\mathcal{DM}(\bm{m}_{i},\\{\bm{m}^{ref}\\}),$ (4)

where $o_{i}\in[0,1]$ is a binary flag that signals if the test sample is
significantly different from the reference distribution. Note that
$\mathcal{DM}$ can be any unsupervised change detection method.

Finally, in order to reduce the ratio of false detections, we declare that a
drift happened only if the number of raw detections $o_{i}$ in the last $w$
samples is higher than a detection threshold $r$. Precisely, we report a drift
at the time $i$ if

$\frac{1}{w}\sum_{t=0}^{w-1}o_{i-t}>r.$ (5)

In case no drift is detected, the reference statistics $\\{\bm{m}^{ref}\\}$ is
updated with the test sample statistic $\bm{m}_{i}$ of undrifted samples, i.e.
when $o_{i}=0$. The pseudo-code of the proposed concept drift detection
framework is reported in Algorithm 2.

Require: Data $\\{(\bm{x}_{i},\bm{y}_{i})\\}_{n}$, training model
$\mathcal{M}$: classifier $f_{c}$, encoder $f_{e}$, constraint module $f_{ce}$

Output: Trained model $\mathcal{M}$: $f_{c}$, $f_{e}$, $f_{ce}$, centroids
$\\{\bm{C}_{j}\\}_{k}$

1

$\\{\bm{C}_{j}\\}\leftarrow$ random initialization

// Initialize the centorids

2

3for  _training epoch $t=0$ to $t_{end}$_ do

4 Fetch mini-batch data $\\{(\bm{x}_{i},\bm{y}_{i})\\}_{b}$ at current epoch
$t$

$\bm{e}_{i}=f_{e}(\bm{x}_{i})$

// Embedding forward pass

$\bm{p}_{i}=\text{softmax}(f_{c}(\bm{e}_{i}))$

// Classifer forward pass

$\mathcal{L}_{c}(\bm{p}_{i},\bm{y}_{i})\leftarrow$ Eq. 1

// Classification loss

$\mathcal{L}_{ce}(\bm{e}_{i},\\{\bm{C}_{j}\\})\leftarrow$ Eq. 2

// Constrained embedding loss

Update $\mathcal{M}$ by SGD on $\mathcal{L}_{c}+\mathcal{L}_{ce}$

// Backward pass

5

6 end for

Algorithm 1 Training with constrained embedding

Require: Trained encoder $f_{e}$, centroids $\\{\bm{C}_{j}\\}_{k}$, reference
data $\\{\bm{X}\\}^{ref}$, test data $\\{\bm{X}\\}^{test}$, unsupervised
detector $\mathcal{DM}$, detection history size $w$, detection threshold $r$

Output: Drift detection index $i_{d}$

1

$i=0$ , $i_{d}=\infty$

// Init sample and drift index

2

/* Calculate reference statistics */

3 for  _$\bm{x}_{r}$ in $\\{\bm{X}^{ref}\\}$_ do

$\bm{d}_{r,j}=\|f_{e}(\bm{x}_{r})-\bm{C}_{j}\|_{2}$

// Distances to centroids

$\\{\bm{m}^{ref}\\}\leftarrow$ get statistical features of
$\\{\bm{d}_{r,j}\\}_{j=1}^{k}$

// Reference statistics

4

5 end for

6

/* Begin Concept Drift detection */

7 for  _$\bm{x}_{i}$ in $\\{\bm{X}^{test}\\}$_ do

$\bm{d}_{i,j}=\|f_{e}(\bm{x}_{i})-\bm{C}_{j}\|_{2}$

// Distances to centroids

$\bm{m}_{i}\leftarrow$ get statistical features of
$\\{\bm{d}_{i,j}\\}_{j=1}^{k}$

// Get statistics

8

$o_{i}=\mathcal{DM}(\bm{m}_{i},\\{\bm{m}^{ref}\\})$

// Compare statistics

9

10 if  _$\frac{1}{w}\sum_{t=0}^{w-1}o_{i-t} >r$_ then

$i_{d}=i$

// Drift detected, end algorithm

11 break

12

13 else if  _$o_{i}=0$_ then

$\\{\bm{m}^{ref}\\}\leftarrow\bm{m}_{i}$

// Update reference statistics

14

15 end if

16 $i=i+1$

17

18 end for

19

Algorithm 2 Task-sensitive drift detection framework

###  III-C Unsupervised Drift Detection Methods

The proposed drift detection framework is flexible to be used with any
unsupervised change detection algorithm. We propose two novel detectors, which
are tailored to our specific representation and are particularly simple. They
are efficient and allow for continuous self-calibration of their meta-
parameters.

####  III-C1 Exponential Moving Average Detector (EMAD)

It monitors the exponentially weighted running statistics of the distance of
each sample to its closest centroid. In this case the feature vector $m_{i}$
just has one entry and is defined as

$\displaystyle m_{i}=\min_{j}\bm{d}_{i,j}\,.$ (6)

For the change detector we calculate the exponential moving averages of the
feature and its variance,

$\displaystyle\mu_{i}$ $\displaystyle=\lambda\cdot\mu_{i-1}+(1-\lambda)\cdot
m_{i}$ (7) $\displaystyle\sigma_{i}$
$\displaystyle=\lambda\cdot\sigma_{i-1}+(1-\lambda)\cdot(m_{i}-\mu_{i})^{2}$
(8)

with the forgetting factor $\lambda=0.95$ and $i$ the time stamp.

We detect a change when the actual average $\mu_{i}$ is greater than a
threshold $\beta$, i.e. $\mu_{i}>\beta$. Here, $\beta$ is an adaptive
threshold which is initialized with the reference-set $\bm{X}^{ref}$
statistic,

$\beta=\mu_{\bm{m}_{r}}+\sigma_{\bm{m}_{r}}.$ (9)

Note that we use $\mu+\sigma$ as threshold as we only flag when the actual
metric is significantly greater than the reference (not when it is smaller).
When no change is detected, the threshold $\beta$ is updated with the current
statistics: $\beta=\mu_{i}+\sigma_{i}$.

####  III-C2 Z-Score Detector (ZSD)

The feature is given by the distance of each sample to its closest centroid
$m_{i}$, Eq. (6). The feature is transformed to a $z$-score $z$, using the
mean ($\mu$) and standard deviation ($\sigma$) of the reference statistic
$\bm{m}_{r}$,

$z_{i}=\frac{m_{i}-\mu_{\bm{m}_{r}}}{\sigma_{\bm{m}_{r}}}$ (10)

We then compute a $p$-value using the cumulative distribution function of the
normal distribution $\Phi$ as $p_{i}=\Phi(z_{i})$. A change is detected if the
$p$-value is below a confidence level $\alpha$, i.e. $p_{i}<\alpha$ with
$\alpha=0.05$. When no change is detected, the $m_{i}$ is used to update the
reference statistic $\bm{m}_{r}$, and the $\mu_{\bm{m}_{r}}$ and
$\sigma_{\bm{m}_{r}}$ are dynamically updated with (7) and (8) respectively.

##  IV Experimental setup

###  IV-A Comparative methods

We also report results using state-of-the-art unsupervised detection methods
which are briefly describe in the following.

####  IV-A1 Hellinger Distance Drift Detector Method (HDDDM)

It monitors the Hellinger distance to detect a drift between two multivariate
distributions [11]. Since HDDDM is a batch-based method, we modified this
approach in the same way as [17], such that a batch is defined incrementally
by a sliding window of $5w$ samples, with a stride of one sample.
Additionally, to detect a change, we directly use the Hellinger distance
$\delta_{H}$: $\delta_{H}>\beta$, with $\beta=\mu+\sigma$, as in [11]. The
threshold is initialized with the first $5w$ samples to calculate the
statistics $\mu$ and $\sigma$.

We do experiment with HDDDM on both, the raw input data $\bm{x}$, and on the
network low-dimensional embedding $f_{e}(\bm{x})$. In the remainder of the
paper, the methods are referred as HDDDM I and HDDDM E respectively.

####  IV-A2 Incremental Kolmogorov–Smirnov test (IKS)

It is an online variant of the Kolmogorow-Smirnov (KS) test [8]. The principle
behind IKS detector is to apply the KS test on each individual feature. The
detection of a change in a single feature may be sufficient to trigger the
presence of a concept drift. The window size used is $5w$ and the confidence
level $\alpha$ is $0.01$. First, we compute the pairwise distances
$\bm{d}_{i,j}$ with (3). Then, we create the feature vectors
$\\{\bm{m}^{ref}\\}$ and $\bm{m}$, where we use the mean, standard deviation,
maximum and minimum as statistical features, respectively on the reference and
test-set. We use those feature vectors derived from the embedding
representation with the IKS detection method.

###  IV-B Datasets and preprocessing

We conduct our experiments on two synthetic and seven real-world datasets. A
summary of the characteristics of the used datasets is reported in Table I.

The synthetic datasets RBF and MovingRBF are created using the
`make_classification` method of the scikit-learn Python toolbox [18]. For both
datasets, we create four clusters with unit variance in a ten dimensional
feature space. For the RBF dataset we add ten more features to each feature
vector, where five are redundant (i.e. copies of the existing informative
features) and five are just white noise.

Additionally, we consider seven real-world datasets which are already utilized
in others drift detection works [17, 19]. They are binary classification
problems and the data instances were shuffled randomly, in order to remove any
bias or drifts present in the data. All the datasets were processed to have
only numeric and binary values, and normalized to have each attribute zero
mean and unit standard deviation.

The initial $50\%$ of each dataset is assumed to be labeled, and is used to
train the model. The last half of the data is considered as an unsupervised
data stream, and it is used for testing. The first $25\%$ of the test data is
used as reference dataset to initialize the statistics for the drift
detectors. A concept drift is induced in the last $50\%$ of the test-stream.

Table I: Characteristics of datasets used. Dataset | #Instances | #Attributes | #Classes
---|---|---|---
adult | 48842 | 65 | 2
bank | 45211 | 48 | 2
digits08 | 1499 | 16 | 2
digits17 | 1557 | 16 | 2
musk | 6598 | 166 | 2
phishing | 11055 | 46 | 2
wine | 6497 | 12 | 2
RBF | 10000 | 20 | 4
MovingRBF | 10000 | 10 | 4

###  IV-C Inducing concept drift

We artificially introduce concept drift after $50\%$ of the test data stream.
By controlling the location and nature of the drift, it is possible to
evaluate the drift detection capabilities of different methods. The drift
induction process is a standard benchmark framework for drift detection [19,
17]. The features are corrupted based on their importance to the
classification task to create a task-dependent drift. First, the features are
ranked based on their information gain [19], and then either the most or least
informative features are selected, by choosing the top $25\%$ (most
informative) or the bottom $25\%$ (least informative) of features from the
ranked list.

We focus on both step and gradual drifts. The step drift is induced by
randomly shuffling the values of a subset of features [19]. This approach
ensures that feature drifts are induced while also maintaining the original
data properties of the dataset. The gradual drift is induced by corrupting a
subset of features with noise $\eta\sim\mathcal{N}(\mu,\,\sigma^{2})$. While
$\mu=1$, the value of $\sigma$ increases over time, from $\sigma=0$ at $50\%$
to $\sigma=2$ after $75\%$ of the data stream. This corruption strategy
simulates a gradual mean shift of the features. Differently from [19], we
corrupt the samples of all classes, instead of picking only one class. In this
way, the drift has bigger impact on the classification task, and also could be
relevant when modifying the least important features.

Only in the MovingRBF dataset, we induce the drift by randomly re-sampling the
location of the class centroids, and then moving them from the initial to this
final position when generating the data, either in a step or gradual manner.
In this way, we directly affect the position of the class centroids, while in
the other drift induction strategies, it is a consequence of the corruption of
the attributes.

###  IV-D Implementation details

We use the same architecture and hyper-parameters for all experiments. The
encoder is a multilayer perceptron (MLP) neural network. It consists of three
hidden layers of 256, 64 and 3 neurons. Hence, the embedding represntation has
dimension $d=3$ ($f_{e}(\bm{x})\in\mathbb{R}^{3}$). The classifier is a single
fully connected layer with the number of output neurons equal to the number of
classes. The ReLU non-linear function is used for the hidden layers and the
softmax activation for the output. The optimizer used is SGD with 0.9 of
momentum. Also, a dropout of 0.25 and $L_{2}$ norm penalization with weight
0.001 is employed in order to reduce generalization error.

Unless explicitly reported, the default value of the drift detection history,
maximum delay and detection ratio are $w=50$, $d_{max}=6w$, and $r=0.25$
respectively.

The source code used in the experiments is publicly available on
github.com/Castel44/TSDD

###  IV-E Evaluation measures

Figure 2: Penalized drift detection accuracy $\widehat{DA}$.

A dataset exhibits real drift, if the performance of the classifier is
significantly degraded by the drift. With the generalization error defined as
$GE=|Acc_{Train}-Acc_{Valid}|$, we attribute a real drift to a dataset if the
classification accuracy drops below the validation accuracy with the
generalization error as margin, i.e. if

$Acc_{Test}<Acc_{Valid}-GE\,.$ (11)

If the performance after the drift does not decrease below this threshold, a
virtual drift occurs which should not be detected.

The performance of drift detectors are assessed by Detection Accuracy $DA$,
the False Alarm Rate (False Positive Rate $FPR$) and the Detection Delay $d$
[19, 4]. The $DA$ is a binary value that signals the correct detection of the
drift event. Note that, in case of virtual drift, the goal is to not have any
detection, i.e. any detection is considered a false positive. In case of real
drift, we account for the detection delay by penalizing $DA$ with a term
proportional to the delay $d$,

$\widehat{DA}=\begin{cases}DA-\Big{(}\frac{d}{d_{max}}\Big{)}^{\gamma}&\text{for
real drift}\\\ DA&\text{for virtual drift}\end{cases}\,,$ (12)

where $d_{max}$ is the maximum acceptable delay which is set to $d_{max}=300$
samples and $\gamma>0$ a free parameter. Thus, an earlier detection of real
drift is better than a later detection. Fig. 2 shows the penalized drift
accuracy $\widehat{DA}$ for various values of $\gamma$. The default value is
$\gamma=2$.

In order to characterize the performance with a single metric, we propose an
accumulation of those performance values. With the true negative rate TNR =
1-FPR, we define the metric $H$ as the harmonic mean between $\widehat{DA}$
and TNR,

$H=2\cdot\frac{\widehat{DA}\cdot TNR}{\widehat{DA}+TNR}$ (13)

The $H$ metric is continuous and bounded $H\in[0,1]$. $H=1$ indicates perfect
detection and no false alarms, and $H=0$ means that either the drift has not
been detected or the false alarm rate is 1.

All experiments have been repeated 10 times with different random seeds. In
order to assess the statistical significance of the results, we use the
Friedman non-parameteric test with 0.05 confidence level, followed by Nemenyi
post-hoc test, see [20]. This evaluation approach allows the simultaneous
comparison of different methods considering several data sets. To visualize
the results, we use the critical difference diagram [20], where a thick
horizontal line shows the algorithms that are not significantly different in
terms of $H$-score.

##  V Results and discussion

In our experiments, the goal is to pinpoint the single drift event, in case of
real concept drift, and to not detect any change in case of virtual drift,
where the changes in the data does not have an impact on the predictions. The
effect of corrupting the most and least informative feature is reported in
Table II. The similarity of train and validation accuracy indicates an initial
static dataset. We report real drift in all the datasets when the most
informative features are corrupted. Instead, the corruption of the least
informative features induces real drift only on digits08 (only step drift) and
phishing datasets. While, the others, reports virtual drift.

We empirically validate the effect of the constraint embedding module on the
latent representation by using the generalized variance (GV) [21] of the
validation data around its class centroids. The GV is proportional to the area
of the ellipsoid in the $d$-dimensional space [22], and it includes meaningful
information about the data sparsity. The effect of the constraining embedding
module is to reduce the GV of the latent representation by $63.31\%$ to
$99.95\%$ depending on the dataset. Therefore, the constraint module leads to
a strong concentration of data samples around the centroids as desired.

We compare the performance of the proposed drift detection framework with the
algorithms EMAD and ZSD against different SotA detection algorithms. The
performance is evaluated by using the proposed constrained embedding module
(indicated by ✓), which returns the class centroids at the end of the
training, and compared to the case where the constraints module was not used
during training (indicated by ✗), where the class centroids need to be
calculated after the training, as the average of the training data of each
class. Additionally, we also compare to the SotA for unsupervised concept
drift detection, with an analysis directly on the input data distribution with
the HDDDMI algorithm.

We report the drift detection results for a representative subset of the all
the studied datasets in Table III. Result of all datasets can be found in the
supplementary material [23]. By analyzing all the results, we observe that the
proposed framework with the inclusion of the constraint embedding module
outperforms the SotA counterpart in terms of $H$-score in 79 experiments, has
the same performance level in 30 experiments, and is worse in 27 out of 136
experiments. Moreover, the proposed framework always shows comparable or
better performance than the unsupervised drift detection on the raw input
data, HDDDMI, which is not able to deal with virtual drifts as expected. All
methods successfully detect the drift on the synthetic dataset MovRBF. Notice
that there is no distinction between least and most important features in that
dataset, as described in Section IV-B.

In the Table III, we can observe the advantage of the cumulative measure $H$,
which accounts for all three performance measures DA, TNR and Delay. For
example, comparing the performance of the IKS algorithm on the wine dataset
with step drift on the most important features, the performance is almost
perfect for DA and TNR, but the delay is rather large when trained without the
constraints on the embedding (indicated with ✗). This is reflected in a
reduced performance measure $H$. In comparison, including the constraint (✓)
also gives almost perfect performance on DA and TNR, but reduces the delay
considerably, which is reflected in an improved $H$ measure. In that way, the
$H$ measure gives a more comprehensive picture of the performance as relying
on one single metrics. Also, from that example, it is clear that the inclusion
of the constrained embedding module improves the performance.

In Fig. 3 we report the critical differences based on the $H$-score of the
investigated algorithms accumulated over all the datasets and drift types. The
algorithms using the proposed constrained embedding framework (subscript ✓)
are in all cases, except for HDDDME, significantly better than the
corresponding method without the constraint embedding (subscript ✗). However,
there is no single detector that outperform all the others since between the
proposed EMAD ${}_{\text{\text{\char 51}}}$, ZSD ${}_{\text{\text{\char 51}}}$
and IKS ${}_{\text{\text{\char 51}}}$ the difference is not statistically
significant.

For the experiments on datasets showing real drift as reported in Fig. 4 the
trend is even clearer. All approaches using the constraints on the embedding
are significantly better than the corresponding approaches without the
constraints. Only HDDDMI working on the raw input shows a similar performance
as the proposed constrained embedding approaches. However, the performance of
HDDDMI is the worst on the datasets showing virtual drift as it can be
expected (see Fig. 5). For the other approaches working on the low dimensional
embedding, there is no difference between methods with and without constrains,
because virtual drift does not affect the model performance and the latent
representation substantially.

Table II: Classification accuracies of train, validation, and test-set when the most and least informative features are corrupted. Cases of virtual drift are highlighted. Dataset | Train | Valid | Step drift | Gradual drift
---|---|---|---|---
Most | Least | Most | Least
adult |  88.8$\scriptstyle\pm$0.1 |  86.0$\scriptstyle\pm$0.2 |  77.4$\scriptstyle\pm$3.4 | 84.9$\scriptstyle\pm$0.1 |  72.3$\scriptstyle\pm$0.8 | 84.8$\scriptstyle\pm$0.1
bank |  95.3$\scriptstyle\pm$0.1 |  91.5$\scriptstyle\pm$0.2 |  77.4$\scriptstyle\pm$2.8 | 90.7$\scriptstyle\pm$0.1 |  81.2$\scriptstyle\pm$0.5 | 89.5$\scriptstyle\pm$0.3
digits08 |  100.0$\scriptstyle\pm$0.0 |  99.6$\scriptstyle\pm$0.3 |  96.3$\scriptstyle\pm$3.2 |  96.7$\scriptstyle\pm$1.2 |  98.5$\scriptstyle\pm$0.6 | 99.4$\scriptstyle\pm$0.1
digits17 |  100.0$\scriptstyle\pm$0.0 |  100.0$\scriptstyle\pm$0.0 |  97.5$\scriptstyle\pm$0.5 | 99.1$\scriptstyle\pm$0.4 |  90.9$\scriptstyle\pm$0.7 | 99.4$\scriptstyle\pm$0.2
musk |  100.0$\scriptstyle\pm$0.0 |  98.8$\scriptstyle\pm$0.1 |  93.6$\scriptstyle\pm$1.6 | 98.1$\scriptstyle\pm$0.4 |  80.6$\scriptstyle\pm$0.2 | 89.4$\scriptstyle\pm$1.0
phishing |  98.8$\scriptstyle\pm$0.0 |  95.0$\scriptstyle\pm$0.4 |  76.7$\scriptstyle\pm$4.6 |  90.4$\scriptstyle\pm$0.5 |  88.9$\scriptstyle\pm$0.4 |  90.8$\scriptstyle\pm$0.3
wine |  100.0$\scriptstyle\pm$0.0 |  100.0$\scriptstyle\pm$0.0 |  80.8$\scriptstyle\pm$5.7 | 100.0$\scriptstyle\pm$0.0 |  89.7$\scriptstyle\pm$0.4 | 100.0$\scriptstyle\pm$0.0
RBF |  99.9$\scriptstyle\pm$0.1 |  99.1$\scriptstyle\pm$0.5 |  93.5$\scriptstyle\pm$3.4 | 99.1$\scriptstyle\pm$0.6 |  90.8$\scriptstyle\pm$4.1 | 98.9$\scriptstyle\pm$0.6
MovRBF |  100.0$\scriptstyle\pm$0.0 |  100.0$\scriptstyle\pm$0.0 |  49.9$\scriptstyle\pm$9.0 | - |  52.4$\scriptstyle\pm$7.7 | -

Table III: Drift detection results. The presence (or not) of the constraint module is reported by ✓ (or ✗). Highlighted is the best performing in the comparison with and without constraint module.  Dataset | Detector | ✗/✓ |  Step drift: most important |  Step drift: least important |  Gradual drift: most important | Gradual drift: least important
---|---|---|---|---|---|---
DA | TNR | Delay |  H | DA | TNR | Delay |  H | DA | TNR | Delay |  H | DA | TNR | Delay | H
digits08 | ZSD | ✗ | 1.00$\scriptstyle\pm$0.00 |  0.94$\scriptstyle\pm$0.14 |  11.80$\scriptstyle\pm$16.34 |  0.96$\scriptstyle\pm$0.09 |  0.50$\scriptstyle\pm$0.53 |  0.99$\scriptstyle\pm$0.04 | - |  0.49$\scriptstyle\pm$0.51  |  0.60$\scriptstyle\pm$0.52 |  0.99$\scriptstyle\pm$0.04 |  14.20$\scriptstyle\pm$25.53 |  0.60$\scriptstyle\pm$0.52  |  0.30$\scriptstyle\pm$0.48 |  1.00$\scriptstyle\pm$0.00 |  78.70$\scriptstyle\pm$47.96 |  0.30$\scriptstyle\pm$0.48
✓ |  0.90$\scriptstyle\pm$0.32 | 1.00$\scriptstyle\pm$0.00 |  7.10$\scriptstyle\pm$3.96 |  0.90$\scriptstyle\pm$0.32  | 1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 | - |  1.00$\scriptstyle\pm$0.00 |  0.60$\scriptstyle\pm$0.52 |  1.00$\scriptstyle\pm$0.00 | 9.10$\scriptstyle\pm$2.38 |  0.60$\scriptstyle\pm$0.52  | 0.60$\scriptstyle\pm$0.52 |  1.00$\scriptstyle\pm$0.00 | 19.00$\scriptstyle\pm$46.24 | 0.60$\scriptstyle\pm$0.52
EMAD | ✗ |  0.90$\scriptstyle\pm$0.32 |  1.00$\scriptstyle\pm$0.00 |  23.00$\scriptstyle\pm$38.27 |  0.90$\scriptstyle\pm$0.32  |  0.50$\scriptstyle\pm$0.53 |  1.00$\scriptstyle\pm$0.00 | - |  0.50$\scriptstyle\pm$0.52  |  0.50$\scriptstyle\pm$0.53 |  1.00$\scriptstyle\pm$0.00 |  12.20$\scriptstyle\pm$29.35 |  0.50$\scriptstyle\pm$0.53  |  0.60$\scriptstyle\pm$0.52 |  1.00$\scriptstyle\pm$0.00 | 0.00$\scriptstyle\pm$0.00 | 0.60$\scriptstyle\pm$0.52
✓ |  0.90$\scriptstyle\pm$0.32 |  1.00$\scriptstyle\pm$0.00 | 9.50$\scriptstyle\pm$3.54 |  0.90$\scriptstyle\pm$0.32  | 1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 | - |  1.00$\scriptstyle\pm$0.01 |  0.60$\scriptstyle\pm$0.52 |  1.00$\scriptstyle\pm$0.00 |  12.00$\scriptstyle\pm$2.62 |  0.60$\scriptstyle\pm$0.52  |  0.40$\scriptstyle\pm$0.52 |  1.00$\scriptstyle\pm$0.00 |  99.30$\scriptstyle\pm$47.51 |  0.39$\scriptstyle\pm$0.50
IKS | ✗ |  0.60$\scriptstyle\pm$0.52 |  1.00$\scriptstyle\pm$0.00 |  92.10$\scriptstyle\pm$60.28 |  0.60$\scriptstyle\pm$0.51  |  0.60$\scriptstyle\pm$0.52 |  1.00$\scriptstyle\pm$0.00 | - |  0.57$\scriptstyle\pm$0.49  |  0.60$\scriptstyle\pm$0.52 |  1.00$\scriptstyle\pm$0.00 | 76.50$\scriptstyle\pm$7.92 |  0.60$\scriptstyle\pm$0.51  |  0.30$\scriptstyle\pm$0.48 |  1.00$\scriptstyle\pm$0.00 |  105.30$\scriptstyle\pm$20.68 |  0.30$\scriptstyle\pm$0.48
✓ | 0.90$\scriptstyle\pm$0.32 |  1.00$\scriptstyle\pm$0.00 | 41.20$\scriptstyle\pm$9.89 |  0.90$\scriptstyle\pm$0.32 | 1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 | - |  0.99$\scriptstyle\pm$0.00 |  0.70$\scriptstyle\pm$0.48 |  1.00$\scriptstyle\pm$0.00 |  108.30$\scriptstyle\pm$96.02 |  0.64$\scriptstyle\pm$0.45  | 0.70$\scriptstyle\pm$0.48 |  1.00$\scriptstyle\pm$0.00 | 20.70$\scriptstyle\pm$65.46 | 0.68$\scriptstyle\pm$0.47
HDDDME | ✗ |  0.50$\scriptstyle\pm$0.53 |  1.00$\scriptstyle\pm$0.00 |  145.00$\scriptstyle\pm$0.00 |  0.47$\scriptstyle\pm$0.50  |  0.70$\scriptstyle\pm$0.48 |  1.00$\scriptstyle\pm$0.00 | - |  0.63$\scriptstyle\pm$0.44  |  0.40$\scriptstyle\pm$0.52 |  1.00$\scriptstyle\pm$0.00 |  131.00$\scriptstyle\pm$46.06 |  0.38$\scriptstyle\pm$0.48  | 0.50$\scriptstyle\pm$0.53 |  1.00$\scriptstyle\pm$0.00 |  145.30$\scriptstyle\pm$59.23 | 0.48$\scriptstyle\pm$0.50
✓ | 0.80$\scriptstyle\pm$0.42 |  1.00$\scriptstyle\pm$0.00 |  130.60$\scriptstyle\pm$45.89 |  0.75$\scriptstyle\pm$0.40 | 1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 | - |  0.94$\scriptstyle\pm$0.00 |  0.50$\scriptstyle\pm$0.53 |  1.00$\scriptstyle\pm$0.00 |  145.90$\scriptstyle\pm$59.64 |  0.45$\scriptstyle\pm$0.48  |  0.30$\scriptstyle\pm$0.48 |  1.00$\scriptstyle\pm$0.00 |  145.00$\scriptstyle\pm$0.00 |  0.28$\scriptstyle\pm$0.45
HDDDMI | - |  0.50$\scriptstyle\pm$0.53 |  1.00$\scriptstyle\pm$0.53 |  158.00$\scriptstyle\pm$21.28 |  0.45$\scriptstyle\pm$0.48  |  0.40$\scriptstyle\pm$0.52 |  1.00$\scriptstyle\pm$0.00 | - |  0.40$\scriptstyle\pm$0.52  |  0.50$\scriptstyle\pm$0.53 |  1.00$\scriptstyle\pm$0.53 |  0.00$\scriptstyle\pm$0.00 |  0.50$\scriptstyle\pm$0.53  |  0.60$\scriptstyle\pm$0.52 |  1.00$\scriptstyle\pm$0.00 |  0.00$\scriptstyle\pm$0.00 |  0.60$\scriptstyle\pm$0.52
phishing | ZSD | ✗ | 1.00$\scriptstyle\pm$0.00 |  0.08$\scriptstyle\pm$0.04 | 0.00$\scriptstyle\pm$0.00 |  0.14$\scriptstyle\pm$0.07  |  0.50$\scriptstyle\pm$0.53 |  0.44$\scriptstyle\pm$0.29 |  261.10$\scriptstyle\pm$331.56 |  0.30$\scriptstyle\pm$0.38  |  0.70$\scriptstyle\pm$0.48 | 0.39$\scriptstyle\pm$0.17 |  201.80$\scriptstyle\pm$203.94 |  0.28$\scriptstyle\pm$0.26  |  0.40$\scriptstyle\pm$0.52 | 0.58$\scriptstyle\pm$0.20 |  273.30$\scriptstyle\pm$377.38 |  0.33$\scriptstyle\pm$0.43
✓ |  0.90$\scriptstyle\pm$0.32 | 0.77$\scriptstyle\pm$0.21 |  80.80$\scriptstyle\pm$155.72 |  0.82$\scriptstyle\pm$0.28 |  0.70$\scriptstyle\pm$0.48 |  0.13$\scriptstyle\pm$0.07 |  0.00$\scriptstyle\pm$0.00 |  0.19$\scriptstyle\pm$0.16  | 1.00$\scriptstyle\pm$0.00 |  0.13$\scriptstyle\pm$0.07 | 0.00$\scriptstyle\pm$0.00 |  0.22$\scriptstyle\pm$0.11  | 0.90$\scriptstyle\pm$0.32 |  0.13$\scriptstyle\pm$0.06 | 0.00$\scriptstyle\pm$0.00 | 0.22$\scriptstyle\pm$0.12
EMAD | ✗ |  0.10$\scriptstyle\pm$0.32 |  1.00$\scriptstyle\pm$0.00 | 6.30$\scriptstyle\pm$19.92 |  0.10$\scriptstyle\pm$0.31  |  0.30$\scriptstyle\pm$0.48 |  1.00$\scriptstyle\pm$0.00 |  0.00$\scriptstyle\pm$0.00 |  0.30$\scriptstyle\pm$0.48  |  0.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 |  0.00$\scriptstyle\pm$0.00 |  0.00$\scriptstyle\pm$0.00  |  0.10$\scriptstyle\pm$0.32 |  1.00$\scriptstyle\pm$0.00 |  0.00$\scriptstyle\pm$0.00 |  0.10$\scriptstyle\pm$0.32
✓ | 0.70$\scriptstyle\pm$0.48 |  1.00$\scriptstyle\pm$0.00 |  197.50$\scriptstyle\pm$214.28 |  0.66$\scriptstyle\pm$0.46 | 0.70$\scriptstyle\pm$0.48 |  1.00$\scriptstyle\pm$0.00 |  0.00$\scriptstyle\pm$0.00 |  0.70$\scriptstyle\pm$0.48 | 0.20$\scriptstyle\pm$0.42 |  1.00$\scriptstyle\pm$0.00 |  0.00$\scriptstyle\pm$0.00 |  0.20$\scriptstyle\pm$0.42 | 0.90$\scriptstyle\pm$0.32 |  1.00$\scriptstyle\pm$0.00 |  0.00$\scriptstyle\pm$0.00 | 0.90$\scriptstyle\pm$0.32
IKS | ✗ |  1.00$\scriptstyle\pm$0.00 |  0.71$\scriptstyle\pm$0.09 |  9.20$\scriptstyle\pm$15.96 |  0.83$\scriptstyle\pm$0.06  |  0.30$\scriptstyle\pm$0.48 |  0.66$\scriptstyle\pm$0.05 | 0.00$\scriptstyle\pm$0.00 |  0.24$\scriptstyle\pm$0.38  |  0.70$\scriptstyle\pm$0.48 |  0.68$\scriptstyle\pm$0.09 |  135.25$\scriptstyle\pm$56.36 |  0.55$\scriptstyle\pm$0.38  |  0.20$\scriptstyle\pm$0.42 |  0.70$\scriptstyle\pm$0.07 | 0.00$\scriptstyle\pm$0.00 |  0.17$\scriptstyle\pm$0.36
✓ |  1.00$\scriptstyle\pm$0.00 |  0.78$\scriptstyle\pm$0.12 |  16.60$\scriptstyle\pm$35.15 |  0.87$\scriptstyle\pm$0.08  | 0.60$\scriptstyle\pm$0.52 |  0.73$\scriptstyle\pm$0.14 |  44.80$\scriptstyle\pm$39.99 |  0.51$\scriptstyle\pm$0.44 |  0.70$\scriptstyle\pm$0.48 |  0.73$\scriptstyle\pm$0.17 |  149.40$\scriptstyle\pm$53.25 |  0.50$\scriptstyle\pm$0.36  | 0.70$\scriptstyle\pm$0.48 |  0.76$\scriptstyle\pm$0.08 |  186.20$\scriptstyle\pm$438.38 | 0.59$\scriptstyle\pm$0.41
HDDDME | ✗ |  1.00$\scriptstyle\pm$0.00 |  0.82$\scriptstyle\pm$0.04 |  111.10$\scriptstyle\pm$39.41 |  0.87$\scriptstyle\pm$0.05  |  0.70$\scriptstyle\pm$0.48 |  0.81$\scriptstyle\pm$0.03 |  403.30$\scriptstyle\pm$606.53 |  0.63$\scriptstyle\pm$0.43  |  0.30$\scriptstyle\pm$0.48 |  0.78$\scriptstyle\pm$0.04 |  558.20$\scriptstyle\pm$402.23 |  0.06$\scriptstyle\pm$0.17  | 0.90$\scriptstyle\pm$0.32 |  0.81$\scriptstyle\pm$0.06 |  164.80$\scriptstyle\pm$408.80 | 0.80$\scriptstyle\pm$0.28
✓ |  1.00$\scriptstyle\pm$0.00 |  0.78$\scriptstyle\pm$0.03 |  132.40$\scriptstyle\pm$42.85 |  0.83$\scriptstyle\pm$0.06  |  0.40$\scriptstyle\pm$0.52 |  0.77$\scriptstyle\pm$0.03 |  146.20$\scriptstyle\pm$386.87 |  0.32$\scriptstyle\pm$0.42  |  0.20$\scriptstyle\pm$0.42 |  0.79$\scriptstyle\pm$0.03 |  443.30$\scriptstyle\pm$349.13 |  0.16$\scriptstyle\pm$0.34  |  0.20$\scriptstyle\pm$0.42 |  0.75$\scriptstyle\pm$0.04 |  369.90$\scriptstyle\pm$593.49 |  0.17$\scriptstyle\pm$0.35
HDDDMI | - |  1.00$\scriptstyle\pm$0.00 |  0.84$\scriptstyle\pm$0.00 |  14.30$\scriptstyle\pm$2.50 |  0.91$\scriptstyle\pm$0.00  |  0.30$\scriptstyle\pm$0.48 |  0.84$\scriptstyle\pm$0.00 |  17.70$\scriptstyle\pm$4.00 |  0.27$\scriptstyle\pm$0.44  |  0.80$\scriptstyle\pm$0.42 |  0.84$\scriptstyle\pm$0.42 |  12.00$\scriptstyle\pm$0.00 |  0.73$\scriptstyle\pm$0.38  |  0.10$\scriptstyle\pm$0.32 |  0.84$\scriptstyle\pm$0.00 |  11.00$\scriptstyle\pm$0.00 |  0.09$\scriptstyle\pm$0.29
wine | ZSD | ✗ |  1.00$\scriptstyle\pm$0.00 |  0.67$\scriptstyle\pm$0.13 |  19.30$\scriptstyle\pm$21.97 |  0.80$\scriptstyle\pm$0.10  |  0.60$\scriptstyle\pm$0.52 |  0.72$\scriptstyle\pm$0.19 | - |  0.55$\scriptstyle\pm$0.47  |  1.00$\scriptstyle\pm$0.00 |  0.79$\scriptstyle\pm$0.10 |  18.50$\scriptstyle\pm$12.26 |  0.88$\scriptstyle\pm$0.06  |  0.10$\scriptstyle\pm$0.32 |  0.63$\scriptstyle\pm$0.18 | - |  0.09$\scriptstyle\pm$0.30
✓ |  1.00$\scriptstyle\pm$0.00 | 1.00$\scriptstyle\pm$0.00 |  28.30$\scriptstyle\pm$13.15 |  1.00$\scriptstyle\pm$0.01 | 1.00$\scriptstyle\pm$0.00 | 1.00$\scriptstyle\pm$0.00 | - |  1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 | 1.00$\scriptstyle\pm$0.00 |  99.80$\scriptstyle\pm$11.43 |  0.98$\scriptstyle\pm$0.01 | 1.00$\scriptstyle\pm$0.00 | 1.00$\scriptstyle\pm$0.00 | - | 1.00$\scriptstyle\pm$0.00
EMAD | ✗ |  0.00$\scriptstyle\pm$0.00 | 1.00$\scriptstyle\pm$0.00 | 0.00$\scriptstyle\pm$0.00 |  0.00$\scriptstyle\pm$0.00  |  1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 | - |  1.00$\scriptstyle\pm$0.00  |  1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 |  68.30$\scriptstyle\pm$64.54 |  0.96$\scriptstyle\pm$0.12  |  1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 | - |  1.00$\scriptstyle\pm$0.00
✓ | 1.00$\scriptstyle\pm$0.00 |  0.95$\scriptstyle\pm$0.02 |  13.50$\scriptstyle\pm$5.64 |  0.97$\scriptstyle\pm$0.01 |  1.00$\scriptstyle\pm$0.00 |  0.95$\scriptstyle\pm$0.02 | - |  0.99$\scriptstyle\pm$0.01  |  1.00$\scriptstyle\pm$0.00 |  0.95$\scriptstyle\pm$0.02 |  41.30$\scriptstyle\pm$28.06 |  0.97$\scriptstyle\pm$0.02  |  1.00$\scriptstyle\pm$0.00 |  0.95$\scriptstyle\pm$0.03 | - |  0.97$\scriptstyle\pm$0.03
IKS | ✗ |  0.90$\scriptstyle\pm$0.18 |  1.00$\scriptstyle\pm$0.00 |  224.00$\scriptstyle\pm$54.86 |  0.68$\scriptstyle\pm$0.21  |  0.30$\scriptstyle\pm$0.48 |  1.00$\scriptstyle\pm$0.00 | - |  0.30$\scriptstyle\pm$0.48  |  1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 |  58.40$\scriptstyle\pm$8.82 |  1.00$\scriptstyle\pm$0.00  |  0.30$\scriptstyle\pm$0.48 |  1.00$\scriptstyle\pm$0.00 | - |  0.30$\scriptstyle\pm$0.48
✓ |  1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 | 94.20$\scriptstyle\pm$18.60 |  0.98$\scriptstyle\pm$0.01 | 0.90$\scriptstyle\pm$0.32 |  1.00$\scriptstyle\pm$0.00 | - |  0.90$\scriptstyle\pm$0.32 |  1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 |  61.70$\scriptstyle\pm$25.49 |  0.99$\scriptstyle\pm$0.01  | 1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 | - | 1.00$\scriptstyle\pm$0.00
HDDDME | ✗ |  1.00$\scriptstyle\pm$0.00 |  0.88$\scriptstyle\pm$0.05 |  39.60$\scriptstyle\pm$16.85 |  0.93$\scriptstyle\pm$0.03  |  0.50$\scriptstyle\pm$0.53 |  0.88$\scriptstyle\pm$0.06 | - |  0.47$\scriptstyle\pm$0.50  |  1.00$\scriptstyle\pm$0.00 |  0.88$\scriptstyle\pm$0.04 |  42.60$\scriptstyle\pm$9.99 |  0.93$\scriptstyle\pm$0.02  |  0.30$\scriptstyle\pm$0.48 |  0.92$\scriptstyle\pm$0.06 | - |  0.29$\scriptstyle\pm$0.47
✓ |  1.00$\scriptstyle\pm$0.00 |  0.89$\scriptstyle\pm$0.08 |  23.40$\scriptstyle\pm$13.75 |  0.94$\scriptstyle\pm$0.05  |  0.70$\scriptstyle\pm$0.48 |  0.90$\scriptstyle\pm$0.06 | - |  0.66$\scriptstyle\pm$0.46  |  1.00$\scriptstyle\pm$0.00 |  0.85$\scriptstyle\pm$0.05 |  37.80$\scriptstyle\pm$11.35 |  0.92$\scriptstyle\pm$0.03  |  0.40$\scriptstyle\pm$0.52 |  0.89$\scriptstyle\pm$0.06 | - |  0.37$\scriptstyle\pm$0.48
HDDDMI | - |  1.00$\scriptstyle\pm$0.00 |  0.88$\scriptstyle\pm$0.00 |  22.50$\scriptstyle\pm$2.42 |  0.93$\scriptstyle\pm$0.00  |  0.00$\scriptstyle\pm$0.00 |  0.88$\scriptstyle\pm$0.00 | - |  0.00$\scriptstyle\pm$0.00  |  1.00$\scriptstyle\pm$0.00 |  0.88$\scriptstyle\pm$0.00 |  44.90$\scriptstyle\pm$1.97 |  0.93$\scriptstyle\pm$0.00  |  0.00$\scriptstyle\pm$0.00 |  0.88$\scriptstyle\pm$0.00 | - |  0.00$\scriptstyle\pm$0.00
RBF | ZSD | ✗ |  0.90$\scriptstyle\pm$0.32 |  0.06$\scriptstyle\pm$0.04 | 0.00$\scriptstyle\pm$0.00 |  0.10$\scriptstyle\pm$0.08  |  0.10$\scriptstyle\pm$0.32 |  0.08$\scriptstyle\pm$0.06 | - |  0.00$\scriptstyle\pm$0.01  |  1.00$\scriptstyle\pm$0.00 |  0.10$\scriptstyle\pm$0.10 |  3.80$\scriptstyle\pm$9.81 |  0.17$\scriptstyle\pm$0.16  |  0.10$\scriptstyle\pm$0.32 |  0.08$\scriptstyle\pm$0.08 | - |  0.01$\scriptstyle\pm$0.03
✓ | 1.00$\scriptstyle\pm$0.00 | 0.51$\scriptstyle\pm$0.27 |  17.40$\scriptstyle\pm$38.19 |  0.64$\scriptstyle\pm$0.21 | 0.40$\scriptstyle\pm$0.52 | 0.56$\scriptstyle\pm$0.30 | - |  0.31$\scriptstyle\pm$0.42 |  1.00$\scriptstyle\pm$0.00 | 0.49$\scriptstyle\pm$0.25 |  4.20$\scriptstyle\pm$6.81 |  0.62$\scriptstyle\pm$0.24 |  0.20$\scriptstyle\pm$0.42 | 0.51$\scriptstyle\pm$0.24 | - | 0.14$\scriptstyle\pm$0.29
EMAD | ✗ |  0.10$\scriptstyle\pm$0.32 |  1.00$\scriptstyle\pm$0.00 | 0.00$\scriptstyle\pm$0.00 |  0.10$\scriptstyle\pm$0.32  |  0.90$\scriptstyle\pm$0.32 |  1.00$\scriptstyle\pm$0.00 | - |  0.90$\scriptstyle\pm$0.32  |  0.60$\scriptstyle\pm$0.52 |  1.00$\scriptstyle\pm$0.00 |  190.20$\scriptstyle\pm$163.24 |  0.45$\scriptstyle\pm$0.48  |  0.90$\scriptstyle\pm$0.32 |  1.00$\scriptstyle\pm$0.00 | - |  0.90$\scriptstyle\pm$0.32
✓ | 0.90$\scriptstyle\pm$0.32 |  1.00$\scriptstyle\pm$0.00 |  90.20$\scriptstyle\pm$87.50 |  0.89$\scriptstyle\pm$0.31 |  0.80$\scriptstyle\pm$0.42 |  1.00$\scriptstyle\pm$0.00 | - |  0.80$\scriptstyle\pm$0.42  | 1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 | 47.00$\scriptstyle\pm$48.84 |  0.99$\scriptstyle\pm$0.02 |  0.90$\scriptstyle\pm$0.32 |  1.00$\scriptstyle\pm$0.00 | - |  0.90$\scriptstyle\pm$0.32
IKS | ✗ |  0.60$\scriptstyle\pm$0.52 |  1.00$\scriptstyle\pm$0.00 |  259.90$\scriptstyle\pm$128.96 |  0.37$\scriptstyle\pm$0.40  |  0.80$\scriptstyle\pm$0.42 |  1.00$\scriptstyle\pm$0.01 | - |  0.80$\scriptstyle\pm$0.42  |  0.90$\scriptstyle\pm$0.32 |  1.00$\scriptstyle\pm$0.00 |  207.70$\scriptstyle\pm$142.06 |  0.74$\scriptstyle\pm$0.37  | 0.90$\scriptstyle\pm$0.32 |  0.99$\scriptstyle\pm$0.04 | - | 0.89$\scriptstyle\pm$0.31
✓ | 0.90$\scriptstyle\pm$0.32 |  0.99$\scriptstyle\pm$0.04 |  169.00$\scriptstyle\pm$120.60 |  0.83$\scriptstyle\pm$0.30 |  0.80$\scriptstyle\pm$0.42 |  0.99$\scriptstyle\pm$0.03 | - |  0.79$\scriptstyle\pm$0.42  | 1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 | 85.40$\scriptstyle\pm$33.56 |  0.98$\scriptstyle\pm$0.02 |  0.50$\scriptstyle\pm$0.53 |  1.00$\scriptstyle\pm$0.01 | - |  0.50$\scriptstyle\pm$0.53
HDDDME | ✗ |  0.90$\scriptstyle\pm$0.32 |  0.88$\scriptstyle\pm$0.07 |  100.40$\scriptstyle\pm$50.92 |  0.81$\scriptstyle\pm$0.29  |  0.80$\scriptstyle\pm$0.42 |  0.88$\scriptstyle\pm$0.05 | - |  0.68$\scriptstyle\pm$0.41  |  1.00$\scriptstyle\pm$0.00 |  0.88$\scriptstyle\pm$0.09 |  66.10$\scriptstyle\pm$34.42 |  0.93$\scriptstyle\pm$0.05  |  0.60$\scriptstyle\pm$0.52 |  0.89$\scriptstyle\pm$0.08 | - |  0.57$\scriptstyle\pm$0.49
✓ | 1.00$\scriptstyle\pm$0.00 |  0.87$\scriptstyle\pm$0.08 |  82.20$\scriptstyle\pm$24.98 |  0.92$\scriptstyle\pm$0.05 |  0.50$\scriptstyle\pm$0.53 |  0.90$\scriptstyle\pm$0.10 | - |  0.47$\scriptstyle\pm$0.49  |  1.00$\scriptstyle\pm$0.00 |  0.91$\scriptstyle\pm$0.07 |  50.10$\scriptstyle\pm$25.91 |  0.95$\scriptstyle\pm$0.04  |  0.50$\scriptstyle\pm$0.53 |  0.93$\scriptstyle\pm$0.09 | - |  0.46$\scriptstyle\pm$0.49
HDDDMI | - |  0.90$\scriptstyle\pm$0.32 |  0.89$\scriptstyle\pm$0.32 |  165.10$\scriptstyle\pm$122.68 |  0.65$\scriptstyle\pm$0.37  |  0.70$\scriptstyle\pm$0.48 |  0.89$\scriptstyle\pm$0.08 | - |  0.65$\scriptstyle\pm$0.45  |  1.00$\scriptstyle\pm$0.00 |  0.89$\scriptstyle\pm$0.00 |  41.80$\scriptstyle\pm$30.26 |  0.94$\scriptstyle\pm$0.04  |  0.10$\scriptstyle\pm$0.32 |  0.89$\scriptstyle\pm$0.08 | - |  0.10$\scriptstyle\pm$0.31
MovRBF | ZSD | ✗ |  1.00$\scriptstyle\pm$0.00 |  0.82$\scriptstyle\pm$0.18 |  2.50$\scriptstyle\pm$2.07 |  0.89$\scriptstyle\pm$0.11  | - | - | - |  \-  |  1.00$\scriptstyle\pm$0.00 |  0.86$\scriptstyle\pm$0.00 | 5.80$\scriptstyle\pm$4.05 |  0.89$\scriptstyle\pm$0.22  | - | - | - | -
✓ |  1.00$\scriptstyle\pm$0.00 | 1.00$\scriptstyle\pm$0.00 |  4.10$\scriptstyle\pm$0.32 |  1.00$\scriptstyle\pm$0.00 | - | - | - |  \-  |  1.00$\scriptstyle\pm$0.00 | 1.00$\scriptstyle\pm$0.00 |  65.40$\scriptstyle\pm$40.07 |  0.99$\scriptstyle\pm$0.01 | - | - | - | -
EMAD | ✗ |  1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 |  6.20$\scriptstyle\pm$1.75 |  1.00$\scriptstyle\pm$0.00  | - | - | - |  \-  |  1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 |  23.10$\scriptstyle\pm$28.78 |  1.00$\scriptstyle\pm$0.01 | - | - | - | -
✓ |  1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 |  4.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00  | - | - | - |  \-  |  1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 |  17.70$\scriptstyle\pm$9.87 |  1.00$\scriptstyle\pm$0.00  | - | - | - | -
IKS | ✗ |  1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 |  57.20$\scriptstyle\pm$34.52 |  0.99$\scriptstyle\pm$0.01  | - | - | - |  \-  |  1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 |  77.70$\scriptstyle\pm$30.74 |  0.99$\scriptstyle\pm$0.02  | - | - | - | -
✓ |  1.00$\scriptstyle\pm$0.00 |  0.99$\scriptstyle\pm$0.04 | 29.30$\scriptstyle\pm$15.17 |  0.99$\scriptstyle\pm$0.02  | - | - | - |  \-  |  1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 |  50.80$\scriptstyle\pm$22.46 |  1.00$\scriptstyle\pm$0.00  | - | - | - | -
HDDDME | ✗ |  1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 |  7.40$\scriptstyle\pm$1.84 |  1.00$\scriptstyle\pm$0.00  | - | - | - |  \-  |  1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 |  26.70$\scriptstyle\pm$25.45 |  1.00$\scriptstyle\pm$0.00  | - | - | - | -
✓ |  1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 |  5.90$\scriptstyle\pm$0.57 |  1.00$\scriptstyle\pm$0.00  | - | - | - |  \-  |  1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 |  36.30$\scriptstyle\pm$32.09 |  1.00$\scriptstyle\pm$0.01 | - | - | - | -
HDDDMI | - |  1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 |  10.20$\scriptstyle\pm$5.69 |  1.00$\scriptstyle\pm$0.00  | - | - | - |  \-  |  1.00$\scriptstyle\pm$0.00 |  1.00$\scriptstyle\pm$0.00 |  38.50$\scriptstyle\pm$16.28 |  1.00$\scriptstyle\pm$0.00  | - | - | - | -

Figure 3: Critical difference diagram based on the $H$-score accumulated over
all datasets and drift types showing pairwise statistical difference
comparison of different drift detectors.  Figure 4: Critical difference
diagram based on the $H$-score accumulated over the datasets showing real
drift only.  Figure 5: Critical difference diagram based on the $H$-score
accumulated over the datasets showing virtual drift only.  Figure 6: Average
performance rank when varying drift window size $w$ and drift detection
threshold $r$ calculated for each dataset and algorithm combination (higher is
better).

Finally, we report ablation studies on the hyperparameters introduced with the
proposed framework, in particular the drift detection history size $w$ and the
detection threshold ratio $r$. In Fig. 6, we report the average rank of the
selected hyperparameter combination $(w,r)$ spanning a grid with
$w\in\\{10,25,50,100\\}$ and $r\in\\{0,0.1,0.25,0.50\\}$. For very small
$r\approx 0$ a drift is reported as soon as a single changed data sample is
detected, which leads to many false positives for small $r$ and the
corresponding low performance. With a large detection history and detection
threshold, i.e. ($w=100$ and $r=0.5$), the performance are also low, because
of the large delay for the detection. The best results are achieved with a
detection window of 50 samples and a detection threshold ratio from $0.1$ to
$0.5$.

##  VI Conclusion and future work

In this work, we presented a novel semi-supervised task-sensitive concept
drift detection framework, which is based on learning a constrained low-
dimensional embedding representation. The constraint imposed on the low
dimensional embedding during the learning of the classification task leads to
well separated clusters for each class, and allows for a robust detection of
drift in incoming novel data samples. Any unsupervised change detection
algorithm can be used for analyzing the statistics of the distances between
incoming data samples and the centroids in the embedding representation and
thereby detecting drifted individual samples. We also introduced two simple
detection methods, EMAD and ZSD, which are based on the exponential moving
average and on a modified $z$-score, respectively. To compare the performance
of different drift detection algorithms, we introduced a bounded and
continuous metric $H$, which accumulates the detection accuracy, delay and
false positive rate into one value.

We conducted experiments on nine benchmarks datasets where different types of
drift have been induced. The results indicated that the proposed framework is
capable of robustly detecting real concept drifts, i.e. when the changes in
the data have an impact on the classification results, while properly ignoring
virtual drift, where the classification accuracy is not affected by the drift.
Furthermore, the proposed change detectors EMAD and ZSD have shown performance
comparable with other SotA algorithms. We thoroughly analyzed the dependence
of the two introduced hyperparameters and identified a range of values for
robust and good operation.

In future work, we target an extension of the framework to online learning
settings by including active learning methods, where the embedding
representation is adjusted in cases of successfully detected drifts.

## References

  * [1] J. Gama, I. Žliobaitė, A. Bifet, M. Pechenizkiy, and A. Bouchachia, “A survey on concept drift adaptation,” _ACM Computing Surveys (CSUR)_ , vol. 46, pp. 1 – 37, 2014.
  * [2] R. N. Gemaque, A. F. J. Costa, R. Giusti, and E. Santos, “An overview of unsupervised drift detection methods,” _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , vol. 10, 2020.
  * [3] H. Hu, M. Kantardzic, and T. S. Sethi, “No free lunch theorem for concept drift detection in streaming data classification: A review,” _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , vol. 10, 2020\.
  * [4] J. Lu, A. Liu, F. Dong, F. Gu, J. Gama, and G. Zhang, “Learning under concept drift: A review,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 31, pp. 2346–2363, 2019.
  * [5] I. Zliobaite, M. Pechenizkiy, and J. Gama, “An overview of concept drift applications,” in _Big Data Analysis: New Algorithms for a New Society. Studies in Big Data_ , N. Japkowicz and J. Stefanowski, Eds. Springer, 2016.
  * [6] A. Castellani, S. Schmitt, and S. Squartini, “Real-world anomaly detection by using digital twin systems and weakly supervised learning,” _IEEE Transactions on Industrial Informatics_ , vol. 17, pp. 4733–4742, 2021.
  * [7] A. Castellani, S. Schmitt, and B. Hammer, “Estimating the electrical power output of industrial devices with end-to-end time-series classification in the presence of label noise,” in _ECML/PKDD_ , 2021.
  * [8] D. Reis, P. A. Flach, S. Matwin, and G. E. A. P. A. Batista, “Fast unsupervised online drift detection using incremental kolmogorov-smirnov test,” _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , 2016.
  * [9] J. Gama, P. Medas, G. Castillo, and P. Rodrigues, “Learning with drift detection,” in _SBIA_ , 2004.
  * [10] A. Bifet and R. Gavaldà, “Learning from time-changing data with adaptive windowing,” in _SDM_ , 2007.
  * [11] G. Ditzler and R. Polikar, “Hellinger distance based drift detection for nonstationary environments,” _2011 IEEE Symposium on Computational Intelligence in Dynamic and Uncertain Environments (CIDUE)_ , pp. 41–48, 2011\.
  * [12] M. Jaworski, L. Rutkowski, and P. Angelov, “Concept drift detection using autoencoders in data streams processing,” in _ICAISC_ , 2020.
  * [13] L. Yang, W. Guo, Q. Hao, A. Ciptadi, A. Ahmadzadeh, X. Xing, and G. Wang, “Cade: Detecting and explaining concept drift samples for security applications,” 2020.
  * [14] X. Li, H. Yin, K. Zhou, and X. Zhou, “Semi-supervised clustering with deep metric learning and graph embedding,” _World Wide Web_ , vol. 23, pp. 781–798, 2019.
  * [15] P. Dube and E. Farchi, “Automated detection of drift in deep learning based classifiers performance using network embeddings,” 2020.
  * [16] S. Ackerman, P. Dube, and E. Farchi, “Sequential drift detection in deep learning classifiers,” _ArXiv_ , vol. abs/2007.16109, 2020.
  * [17] T. S. Sethi and M. Kantardzic, “On the reliable detection of concept drift from streaming unlabeled data,” _Expert Syst. Appl._ , vol. 82, pp. 77–99, 2017.
  * [18] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and É. Duchesnay, “Scikit-learn: Machine learning in Python,” _Journal of Machine Learning Research_ , vol. 12, pp. 2825–2830, 2011.
  * [19] T. S. Sethi, M. Kantardzic, and E. Arabmakki, “Monitoring classification blindspots to detect drifts from unlabeled data,” _2016 IEEE 17th International Conference on Information Reuse and Integration (IRI)_ , pp. 142–151, 2016.
  * [20] J. Demšar, “Statistical comparisons of classifiers over multiple data sets,” _Journal of Machine Learning Research_ , vol. 7, no. 1, pp. 1–30, 2006.
  * [21] S. Wilks, “Certain generalizations in the analysis of variance,” _Biometrika_ , vol. 24, 01 1932.
  * [22] T. W. Anderson, “An introduction to multivariate statistical analysis,” Wiley New York, Tech. Rep., 1962.
  * [23] A. Castellani, S. Schmitt, and B. Hammer, “Task-sensitive concept drift detector with constraint embedding,” _arXiv:2108.06980_ , 2021, supplementary material in the appendix.