11institutetext: University of Passau, Germany
11email<EMAIL_ADDRESS>

# DriftGAN: Using historical data for Unsupervised Recurring Drift Detection

Christofer Fellicious 11 0000-0001-7487-7110
Sahib Julka 11 0000-0002-2952-5519
Lorenz Wendlinger 11 0000-0001-9459-6244
Michael Granitzer 11 0000-0003-3566-5507

###### Abstract

In real-world applications, input data distributions are rarely static over a
period of time, a phenomenon known as concept drift. Such concept drifts
degrade the model’s prediction performance, and therefore we require methods
to overcome these issues. The initial step is to identify concept drifts and
have a training method in place to recover the model’s performance. Most
concept drift detection methods work on detecting concept drifts and
signalling the requirement to retrain the model. However, in real-world cases,
there could be concept drifts that recur over a period of time. In this paper,
we present an unsupervised method based on Generative Adversarial
Networks(GAN) to detect concept drifts and identify whether a specific concept
drift occurred in the past. Our method reduces the time and data the model
requires to get up to speed for recurring drifts. Our key results indicate
that our proposed model can outperform the current state-of-the-art models in
most datasets. We also test our method on a real-world use case from
astrophysics, where we detect the bow shock and magnetopause crossings with
better results than the existing methods in the domain.

###### Keywords:

Drift Detection Unsupervised Drift Detection Machine learning.

##  1 Introduction

Machine Learning is ubiquitous in our lives and has the potential to
revolutionize many industries by increasing efficiency, personalizing
experiences, predicting outcomes, automating tasks, and improving decision-
making. Machine learning often predicts different aspects of our lives, such
as the prediction of energy consumption [19] or the fuel consumption of trucks
[17]. For applications with continuous stream of data that span months or
years, there is a good chance for the input data distribution to change over
time. For example, in credit card fraud detection, customers’ buying behavior
changes over time [5]. Such changes in the input data distribution are known
as _concept drifts_ and adversely affect the model’s prediction performance.
To counter such performance degradation, we need mechanisms to identify the
changes in the input data distribution and retrain the prediction model so
that the prediction model adapts the changed input data distribution and
improves the performance. In the most straightforward strategy, we assume
changes happen regularly and train the model periodically. However, this naive
strategy comes with a couple of significant disadvantages. First, model
retraining is usually very costly in terms of time for data collection and
computation, and it is even harder to determine the perfect retraining
interval. Secondly, changes in distribution may not coincide with the periodic
model retraining schedule, thereby rendering model update efforts inefficient
and ineffective. Therefore, we need a mechanism to detect the changes in the
underlying data distribution and retrain the model for optimal performance and
utilization of computing resources [4].

However, detecting changes in data distributions and identifying previous data
distributions hold the potential to improve classifiers, which is still
relatively unexplored. In particular, there could be patterns that reappear
later. For example, suppose we have a method to identify such past data
distributions. In that case, identifying recurring changes in the distribution
will help us bolster the training set and get the model up to speed quicker
rather than waiting for the required number of instances from the real-world
data stream. Currently, most concept drift algorithms only focus on
identifying the changes in the distribution, but do not consider whether a
similar data distribution previously occurred. Examples of repeating
distributions are seasonal repeating customer shopping behavior, energy
consumption, or changes in credit card fraud patterns.

In standard methods, we need to wait until enough input is acquired to train
the model after every new concept drift. Furthermore, this data also needs to
be labelled, which increases the cost and energy expenditure. Contrary to
standard methods, our framework keeps track of the previously seen data
distributions, which allows us to augment the retraining dataset with the
previously seen data of the same distribution. In real-world cases, this
augmentation helps reduce the time required to acquire enough new data on the
new distribution, thereby reducing the model’s turnaround time. As a result,
we also increase the performance of the model when compared to retraining the
model from scratch with limited data. This time to obtain the labels and
retrain the model is significant when labeled data is either sparse or only
available with a significant time delay [3].

In this work, we extend drift detection by keeping track of all data
distributions seen by the model. We show that using the data from the past
concept drifts for retraining the model allows us to outperform the current
state of the art. Furthermore, this memory of previous data distributions
allows the model to adapt and reduce the turnaround time to previously
identified concept drifts. Methodologically, we develop a multiclass-
discriminator drift Generative Adversarial Network (DriftGAN). The DriftGAN
contains a discriminator module that does not only distinguish between real
and artificial examples but also discriminates to which previous drift the
current example belongs. Since drifts can increase over time, the
discriminator network can grow incrementally by adding more discriminator
classes whenever we identify new data distributions. We evaluate our DriftGAN
on eight unsupervised real-world drift detection datasets and compare it with
existing unsupervised methods. Unsupervised drift detection does not rely on
the performance of the label prediction model to detect drift - which is not
readily available in most tasks but only based on changes in the input
distribution. The results show that our method has a better performance when
compared to other unsupervised methods that only detect whether drifts occur
or not. We also adapted our method as a sampling strategy for the magnetometer
data of the NASA MESSENGER spacecraft. Our results for the real-world
application scenario also produce better results than the existing methods,
which shows that our method can be adapted to different domains with very
little to no changes.

We structure the rest of the paper as follows, section 2 describes the related
work with supervised and unsupervised concept drift detection, section 3
discusses our approach to the problem with the necessary explanation on the
identification of previously seen data distributions and the research gap we
address. Then, section 4 describes our experimental results and the
interpretation of the results. Finally, section 5 concludes our paper.

##  2 Related Work

We can broadly classify the concept drift detection methods into Supervised
and Unsupervised methods. Supervised methods have access to the input feature
vector’s label, making it easier to understand and detect data distribution
changes. The problem becomes compounded with unsupervised learning methods as
the data labels are unavailable to the model to decide whether the
distribution has changed. Unsupervised learning methods are more useful in
real-world scenarios as data labels might not be readily available to the
model and may require time or expertise to obtain.

###  2.1 Supervised Methods

The Drift Detection Method (DDM) is one of the earliest supervised methods in
concept drift detection [4]. The method uses the error rate to determine
whether a concept drift occurred. The idea behind this method is that the
error rate increases when there is a change in input data distribution. The
algorithm defines two thresholds based on the error rate. The first threshold
is also lower, serving as a drift warning. The second threshold signals the
drift. The retraining occurs in the window that starts from the drift warning.

The Adaptive Windowing (ADWIN) method uses different-sized windows to detect
concept drifts [1]. ADWIN uses statistical tests on different windows and
detects concept drifts based on the error rates, similar to DDM. Cabral and
Barros proposed a method using Fisher’s Exact Test [14]. This method analyzed
two separate context windows and the assumption that if there was no drift,
the error distribution should be equal between the two windows. They use the
generalized contingency table from Fisher’s Exact test to test the null
hypothesis.

###  2.2 Unsupervised Methods

Identifying concept drifts without the corresponding labels to the input
feature vectors is comparatively more difficult. We can only rely on the input
feature vectors to decide whether a concept drift occurs. Additionally,
unsupervised concept drift detection cannot detect certain types of concept
drifts such as changes in the drift label while the input data distribution is
more or less constant. In such scenarios, it is almost impossible to identify
a concept drift without the class labels. Hu et al. explain why detecting
concept drifts while maintaining high performance is not trivial [10]. The
paper explains high performance with labeled data is possible, but labeling
data streams as mentioned earlier could be expensive and time-consuming. These
drawbacks of supervised drift detection mean that even with all the drawbacks,
unsupervised drift detection is more efficient in real-world scenarios to
detect concept drifts.

The Kolmogorov-Smirnov is a nonparamtetric statistical test can determine
whether two data points belong to the same distribution. The problem with the
K-S test is its time complexity that grows with the number of samples. Reis et
al. proposed a faster alternative to the K-S test using an incremental K-S
test [18]. This implementation uses a custom tree data structure called
"_treap_." The algorithm uses two windows, where the first is fixed and the
second is sliding. The first window contains the original concepts the model
was trained on, while the second one slides and checks whether a concept drift
occurred. While the K-S test remains popular for univariate data and an
expansion of the K-S test for multivariate data was proposed by Justel et al.
[12].

Discriminative Drift Detector is an unsupervised method for detecting concept
drifts using a linear regressor on two windows [7]. The basic idea is that
there are two windows; the first is assumed to be from the old data
distribution and given label 0. A second window, a sequence of input data, is
chosen and labeled as 1. If the linear regressor can discriminate between both
windows with a certain precision means that the two windows are different, and
thus a concept drift has occurred. The performance is measured using a
secondary supervised Hoeffding Tree classifier. One Class Drift
Detection(OCDD) is a method that uses a linear SVM and sliding windows to
detect outliers in the incoming data stream [8]. When the outliers reach a
threshold, the algorithm signals a concept drift. Currently, OCDD has the best
performance in unsupervised drift detection.

Another method, ODIN, uses autoencoders to identify drifts and select trained
classifiers based on the drift [20]. ODIN consists of four components; an
encoder, a decoder, a latent discriminator and an image discriminator.
Autoencoders generate latent representations from input feature vectors, and a
decoder generates the initial feature vectors. The latent representation is
passed to the latent discriminator for detecting inlier frames from outlier
frames. The loss function is computed as the weighted sum of the loss of the
latent discriminator loss, the image discriminator, and the standard
reconstruction loss.

Methods like the K-S test are unsuitable for data with many features, as
checking the data distribution across each feature is not feasible. All
methods in the concept drift domain are adept at identifying supervised and
unsupervised drifts. However, they do not consider the possibility of
recurring drifts and use the historical data from previous drifts to retrain
the model. Reusing previous data from the same distributions that occurred in
the past will enable us to retrain the model faster and reduce the false
predictions until a new model is trained.

Gulcan and Can proposed a method that addresses concept drift in multi label
data streams [9]. The method Label Dependency Drift Detector(LD3) works
alongside any multi-label classifier. LD3 works by exploiting the predicted
label dependencies and if there is a significant change in the dependencies, a
concept drift is signalled. The algorithm uses a threshold and two windows,
one for new samples and one for old samples, along with a correlation window
that keeps track of past correlations between the data points.

Jain et al. address the issue of concept drift in anomaly detection of network
traffic [11]. The authors propose two techniques, an Error Rate based concept
drift detection and Data Distribution based Concept Drift Detection. The
method uses a K-Means Clustering for reducing the dataset size and upgrade the
training dataset. The final classifier is a Support Vector Machine(SVM) for
anomaly detection.

##  3 Method

Generative Adversarial Networks(GAN) are valuable for creating synthetic data.
Goodfellow et al. introduced the concept of generative adversarial networks
[6]. A GAN comprises a Generator and a Discriminator. The Generator creates
synthetic data, and the Discriminator predicts whether the input data belongs
to either the real or synthetic data created by the Generator. The training
continues until the discriminator’s loss drops below a certain threshold.
Generative Adversarial Networks are used in a wide variety of domains such as
for music generation [13], image super-resolution [21], or even medical image
reconstruction [2]. We understand from the different domain applications that
Generative Adversarial Networks are good at generating different output
distributions. Furthermore, this is the property we use in our method. section
3 describes the complete method. The code is available as open
source111https://github.com/cfellicious/DriftGAN.

###  3.1 Framework

We use a Generator $\theta_{G}(s)$ that accepts a sequence $s=<x_{t-k}\ldots
x_{t}>$ with length $k$ of input feature vectors $x$. The main aim of the
Generator is to predict the next feature vector $x_{n+1}$. More precisely, in
Equation 1 we define $\theta_{G}(x)$ as

$\displaystyle\theta_{G}(s):x_{n-k},...,x_{n}\longmapsto x_{n+1}$ (1)

The Discriminator $\theta_{D}(x)$ is a multi-class dynamic discriminator that
predicts whether an input data belongs to one of the input distributions from
the data or is a synthetic data point generated by $\theta_{G}(s)$. We define
a mapping function $\theta_{D}(x)$, which maps the incoming feature vectors to
a set of identified input data distributions or an unidentified data
distribution, as explained in Equation 2.

$\displaystyle\theta_{D}(x):x_{i}\longmapsto{\Pi_{1},....,\Pi_{n},\Pi_{n+1}},$
(2)

where,

  * •

$x_{i}$ is an input feature vector

  * •

$\Pi_{j}$ is the $j-th$ input distribution with $j=1$ indicating the current
distribution and $j>1$ being the $j-1$ past distribution detected by our GAN

##### Detecting Drifts:

We map all incoming input feature vectors to one of the distributions. We
signal a drift if all the contiguous inputs in a batch are mapped to any of
the same data distributions other than the current data distribution($P_{1}$).
Equation 3 gives the formal description.

$\displaystyle\forall
x\in\delta,where\theta_{D}(x)~{}\neq~{}\Pi_{curr}~{}~{}\&$ (3)
$\displaystyle~{}~{}\forall(x_{1},x_{2})~{}\in~{}\delta,~{}\theta_{D}(x_{1})=\theta_{D}(x_{2})$

where,

  * •

$x$ is an input feature vector

  * •

$\delta$ is the input data batch to $\theta_{D}$

##### Dynamic Discriminator Extension:

Initially, we start with a training dataset of size $\rho$ and train the GAN
so that $\theta_{D}$ learns to discriminate the real data from the data
generated by $\theta_{G}$. The current data distribution is the data
distribution of the training dataset of size $\rho$. Data generated by
$\theta_{G}$ represents the drifted data. $\theta_{G}$ can generate the next
in the sequence from completely different input distributions, and can also
generate data sequences that do not belong to any previously seen data
distributions. This capability of $\theta_{G}$ allows us to set a class for
previously unseen distributions in $\theta_{D}$. We also label the data so
that anything other than previously seen distributions is assigned to the same
class. A concept drift can be either from a previously seen distribution,
except for the first concept drift, or an entirely new drift. If a previously
seen data distribution repeats, we use the data from that distribution to
bolster the training of the secondary classifier. It is also possible to reuse
only the old data to train the classifier or even the old classifier itself if
it is stored. However, we do not choose to do so because there could be
different data distributions in an extensive dataset, and the space
requirements might be infeasible for larger datasets.

Table 1: Hyperparameters used in the code, their symbols and explanations Symbol | Hyperparameter | Explanation
---|---|---
$\rho$ | Window Size | Number of training instances for drifts
$\delta$ | Batch size | Number of testing instances that should agree for a drift
$W$ | Data Window | Window of sequential input feature vectors of size $\rho$
$\delta$ | Data Batch | A batch of data for testing
$\Pi(x)$ | Input distribution | Represents the input distribution of $x$
$\Pi_{curr}$ | Current distribution | Drift not signalled as long as input in this distribution
$\omega$ | Seen distributions | Set of seen input distributions $\Pi(x_{n})\in\omega$
$\mu$ | Mean | Mean of the input feature vector
$\sigma$ | Standard Deviation | Standard deviation of the input feature vector
$\lambda$ | Fraction | Percentage of historical data to be used for training

FindDrifts($\rho$, $b$) $\rightarrow$ Window size and batch size respectively
;

$W=data[1:\rho]$ $\rightarrow$ Obtain $\rho$ samples for training the GAN;

$\theta_{G},\theta_{D}$ = TrainGan($W$) $\rightarrow$ Train the GAN;

$\Pi_{curr}=\Pi(W)$ $\rightarrow$ Set the current distribution;

$\omega={\Pi_{curr}}$ $\rightarrow$ Add curr distribution to seen
distributions while  _isDataBatchAvailable_ do

$W=x_{n-b}..x_{b}$ $\rightarrow$ Obtain new batch ;

$\delta=\forall x\in W(x-\mu(x))/\sigma(x)$ ;

if  _$\forall x\in\delta,where\theta_{D}(x)~{}\neq~{}\Pi_{curr}~{}~{}\
&~{}~{}\forall(x_{1},x_{2})~{}\in~{}\delta~{}\theta_{D}(x_{1})=\theta_{D}(x_{2})$_
then

Batch prediction consensus on input not belonging to current data
distribution, Drift Detected ;

if  _$\Pi(\delta)\notin\omega$_ then

New input distribution detected ;

$\omega=\omega\cup\Pi(\delta)$ ;

Retrain $\theta_{D}$ using $\theta_{G}$ with the added new input distribution
$\Pi(\delta)$;

$\Pi_{curr}=\Pi(\delta)$

Algorithm 1 GAN-based Drift Detection. Table 1 explains the hyperparameters

One of the main issues with Generative Adversarial Networks is the mode
collapse. Mode collapse occurs when the Generator learns only one of the input
modes instead of generating generalized data. We counter the mode collapse
problem by having a random mini-batch input to the Generator ($\theta_{G}$)
and the Discriminator($\theta_{D}$). Using mini batches forces the generator
network to learn to generalize better rather than falling back to a single
mode in the input distribution. We also standardize the input data by the mean
and standard deviation, which compresses the input feature vector. ODIN
implemented a variation of the standardization method [20].

###  3.2 Network Architectures

For our framework, we need two networks, one for the Generator($\theta_{G}$)
and one for the Discriminator($\theta_{D}$). The Generator predicts the input
feature vector in the sequence, and therefore we can get away with using a
smaller network to learn the function. On the other hand, the multi-class
Discriminator requires much more expressive power and therefore has a slightly
larger network when compared to the Generator. The symbols $N,P,R,S$ represent
the number of neurons in each layer. In our case, $N$ equals the sequence
length multiplied by the input feature vector size, and $P$ and $R$ are
constants of 128 and 4096, respectively. $S$ is the output size equal to the
size of a single input feature vector.

The $Discriminator$($\theta_{D}$) is a slightly larger network, and the size
of the top layer of the network is incremented by one every time a previously
unseen drift is detected. The first three layers do not change during the
execution. We have two layers of 1024 neurons, each with the final layer
having a Sigmoid non-linearity to the final layer. The number of neurons on
the final layer at any point depends on the number of unique drifts seen by
the discriminator plus one neuron as a generalization of all the unseen
drifts.

The main objective on the design of the GAN is to have the networks as small
as possible as the GAN is not a static part of the detection process. Every
time a concept drift is detected, the GAN undergoes training and having a
larger model that could capture and learn more information could be
detrimental as it takes longer to train. Therefore, we avoid using larger and
more complex networks and go for the simplest methods to speedup the training.
The primary reason that small networks work well in our experiment setting is
due to the fact that we do not use noise as the latent vectors to $\theta_{G}$
but rather a sequence of real input. This allows some leeway with having a
smaller network to SACadapt to the inputs to only predict the next in the
sequence.

###  3.3 Training

We start with an initial window of size $\rho$ where the input feature vectors
of that window are assumed to belong to the same distribution. The
Discriminator, at this point, has only two output neurons, one for the current
distribution of the window and one for unseen distributions. We use the
Adadelta optimizer for both the Generator and discriminator. Furthermore, we
use the CrossEntropyLoss function for the Discriminator. As for the Generator,
the loss is a Mean Squared Error loss based on the actual $n+1$ vector in the
sequence added to the CrossEntropyLoss of the Discriminator. These additions
of loss values allow the Generator to fool the Discriminator better while also
better predicting the next in-sequence input vector.

##  4 Results

We evaluate our approach on different real-world datasets and compare it with
the current state of the art. We also use our approach to predict the incoming
data distributions and predict on the different magnetic events of the
MESSENGER spacecraft while orbiting Mercury.

The current state of the art for unsupervised drift detection is from [7] on a
public dataset. We use the same datasets used by the authors for the state-of-
the-art. In order to facilitate a direct comparison to state-of-the-art, we
use the same real-world datasets, the same metrics, and the secondary
classifier for our experiments. Using the same datasets allows us for an
apples-to-apples comparison.

The secondary classifier predicts the actual labels on the incoming data. The
better the concept drift identification of our model, the better the
performance of the secondary classifier, which is a supervised learning
algorithm, a Hoeffding Tree Classifier, in our case. We use a Hoeffding Tree
Classifier as the secondary supervised classifier because of the support for
streaming datasets. The main reason behind the choice is to enable better
comparison with the state of the art as they use the same classifier as their
secondary classifier.

Our comparison is based on the results of the current state-of-the-art in
unsupervised concept drift detection published by [7]. We use accuracy as the
metric for evaluating all the datasets as it is the most prevalent metric for
analyzing the algorithm’s performance. Most other publications also use the
accuracy metric; hence, we use the same metric to evaluate our experiments for
comparison. We use the implementation of scikit-learn [16]. In addition, we
use the implementation of the scikit-multiflow library for the Hoeffding Tree
classifier [15]. This library caters to analyzing streaming datasets. For the
evaluation, we use the Interleaved-Test-Then-Train method outlined in the
paper by [8]. This method tests every incoming feature vector and then adapts
the classifier to the tested feature vector. We reset and retrain the
Hoeffding tree classifier whenever a concept drift is detected.

###  4.1 Standard Drift Datasets

We compare our results with the state of the art but we also developed three
baseline methods. The baseline methods are explained below.

####  4.1.1 Baseline Methods:

In addition to comparing our results with state of the art, we also compared
the experimental results with three additional methods that we call our
baseline methods for the experiment. The baseline methods we selected are
probable real-world scenarios where concept drift is not considered. The first
method is one we see in most machine learning models where the model is
trained on a training set and then never modified. The purpose of such a
baseline is to provide an understanding of the performance if we never
consider concept drift and let the model predict over the whole test set. We
call this method Initial Learn, as the model learns on a small initial set of
input feature vectors while predicting the rest.

In the following scenario, we retrain the model as specific intervals without
considering concept drift. We can liken this to retraining a fraud prediction
model every month or week so that the model is trained on the latest input
data distributions. We call this method the Regular Retrain method, as the
model is regularly updated without considering concept drift. From a
computational point of view, this is the most expensive model as the model is
retrained far more often than required, irrespective of the fact whether a
concept drift is present in the input data or not.

The final baseline scenario is where the model is continuously updated with
the input data distribution. We can see this as partially updating a model as
soon as the labels to the data are obtained. Although such a scenario is very
unlikely in a real-world scenario due to the issues with updating models in a
production environment and attaining labels for each and every data point. As
explained by [10], labelling data streams is expensive and requires quite a
lot of effort to obtain. But such a baseline gives us an idea of how good a
model could get with only partial updates. We call this baseline method the
Regular Update.

We compare all the methods using the Hoeffding Tree classifier with all
default parameters. The Regular Retrain method makes use of the reset method
of the Hoeffding Tree classifier while the Regular Update method uses the
partial fit method of the algorithm.

####  4.1.2 Results and Discussion:

We set the training window size($\rho$) to one hundred input training
instances based on experimentation with different values. Also [7] found that
having a window size of 100 provides a good balance between generalizing to
the current distribution and mostly having only a single distribution within
the window. A smaller window size does not adequately generalise features for
the network to discriminate between the different contexts adequately.

Our method beats the state of the art in most datasets as seen in Table 2. For
larger datasets, like Rialto, Airlines, Poker or Covertype(marked with * in
the table), we only considered the first 50000 input instances. This is due to
the sheer number of unique drifts that arise and the increased training time
once a previously unseen drift is detected.

One of the best examples is for the Airlines dataset where our model did not
detect a drift at all for the first 50000 input data instances. This also
produces a better score compared to the other methods. But on the other hand,
our model performs worse for the Electricity dataset. One reason for this
could be that the dataset has a higher number of outliers within our window
size($\rho$).

Table 2: comparison of accuracy values of different methods. Dataset | Initial Train | Regular Update | Regular Retrain | OCDD | Our Method
---|---|---|---|---|---
Electricity | 56.44 | 77.69 | 75.4 | 86.22 | 79.86
Poker | 50.12 | 74.08 | 63.05 | 76.63 | 76.99*
Cover Type | 47.69 | 82.49 | 55.48 | 88.36 | 82.49*
Rialto | 09.95 | 31.373 | 51.02 | 66.68 | 54.84
Spam | 66.48 | 88.35 | 75.81 | 87.02 | 89.28
Phishing | 83.49 | 90.26 | 89.18 | 90.56 | 91.37
Airlines | 56.13 | 63.88 | 59.72 | 63.16 | 65.59*
Outdoor | 15.21 | 57.15 | 12.00 | 62.24 | 62.57*

##  5 Conclusion

Our method addresses the issue of identifying concept drifts that have
occurred in the past, and we use that information to increase the training
data for the secondary classifier. This memory of previous distributions
allows our method to improve prediction accuracy, reduce the training time for
the model, and reduce the effort in labelling data. Our model shows better
performance when compared to other state-of-the-art models across the
different publicly available datasets. Our method as a framework is easy to
use and can be applied to different application domains with little to no
modifications. We applied our method in an active learning loop to sample the
most informative orbits to train a classifier. Our method performed vastly
superior over random sampling and an active learning method. The only drawback
of our framework is that the amount of training time increases with the
increase in the number of unique drifts. Having to store the data of previous
distributions in memory to retrain the model could also be memory inefficient.
As for future work, we expect to use a Conditional Generative Adversarial
Network that will enable us to simulate different input probability
distributions effectively.

## References

  * [1] Bifet, A., Gavalda, R.: Learning from time-changing data with adaptive windowing. In: Proceedings of the 2007 SIAM international conference on data mining. pp. 443–448. SIAM (2007)
  * [2] Chen, Y., Shi, F., Christodoulou, A.G., Xie, Y., Zhou, Z., Li, D.: Efficient and accurate mri super-resolution using a generative adversarial network and 3d multi-level densely connected network. In: International Conference on Medical Image Computing and Computer-Assisted Intervention. pp. 91–99. Springer (2018)
  * [3] Dastidar, K.G., Granitzer, M., Siblini, W., et al.: The importance of future information in credit card fraud detection. In: International Conference on Artificial Intelligence and Statistics. pp. 10067–10077. PMLR (2022)
  * [4] Gama, J., Medas, P., Castillo, G., Rodrigues, P.: Learning with drift detection. In: Brazilian symposium on artificial intelligence. pp. 286–295. Springer (2004)
  * [5] Ghosh Dastidar, K., Jurgovsky, J., Siblini, W., Granitzer, M.: Nag: neural feature aggregation framework for credit card fraud detection. Knowledge and Information Systems 64(3), 831–858 (2022)
  * [6] Goodfellow, I.J., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., Courville, A., Bengio, Y.: Generative adversarial networks. arXiv preprint arXiv:1406.2661 (2014)
  * [7] Gözüaçık, Ö., Büyükçakır, A., Bonab, H., Can, F.: Unsupervised concept drift detection with a discriminative classifier. In: Proceedings of the 28th ACM International Conference on Information and Knowledge Management. pp. 2365–2368 (2019)
  * [8] Gözüaçık, Ö., Can, F.: Concept learning using one-class classifiers for implicit drift detection in evolving data streams. Artificial Intelligence Review 54(5), 3725–3747 (2021)
  * [9] Gulcan, E.B., Can, F.: Unsupervised concept drift detection for multi-label data streams. Artificial Intelligence Review 56(3), 2401–2434 (2023)
  * [10] Hu, H., Kantardzic, M., Sethi, T.S.: No free lunch theorem for concept drift detection in streaming data classification: A review. Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery 10(2), e1327 (2020)
  * [11] Jain, M., Kaur, G., Saxena, V.: A k-means clustering and svm based hybrid concept drift detection technique for network anomaly detection. Expert Systems with Applications 193, 116510 (2022)
  * [12] Justel, A., Peña, D., Zamar, R.: A multivariate kolmogorov-smirnov test of goodness of fit. Statistics & Probability Letters 35(3), 251–259 (1997)
  * [13] Kumar, K., Kumar, R., de Boissiere, T., Gestin, L., Teoh, W.Z., Sotelo, J., de Brébisson, A., Bengio, Y., Courville, A.C.: Melgan: Generative adversarial networks for conditional waveform synthesis. Advances in neural information processing systems 32 (2019)
  * [14] de Lima Cabral, D.R., de Barros, R.S.M.: Concept drift detection based on fisher’s exact test. Information Sciences 442, 220–234 (2018)
  * [15] Montiel, J., Read, J., Bifet, A., Abdessalem, T.: Scikit-multiflow: A multi-output streaming framework. The Journal of Machine Learning Research 19(1), 2915–2914 (2018)
  * [16] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., et al.: Scikit-learn: Machine learning in python. the Journal of machine Learning research 12, 2825–2830 (2011)
  * [17] Perrotta, F., Parry, T., Neves, L.C.: Application of machine learning for fuel consumption modelling of trucks. In: 2017 IEEE International Conference on Big Data (Big Data). pp. 3810–3815. IEEE (2017)
  * [18] dos Reis, D.M., Flach, P., Matwin, S., Batista, G.: Fast unsupervised online drift detection using incremental kolmogorov-smirnov test. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. pp. 1545–1554 (2016)
  * [19] Seyedzadeh, S., Rahimian, F.P., Glesk, I., Roper, M.: Machine learning for estimation of building energy consumption and performance: a review. Visualization in Engineering 6(1), 1–20 (2018)
  * [20] Suprem, A., Arulraj, J., Pu, C., Ferreira, J.: Odin: automated drift detection and recovery in video analytics. arXiv preprint arXiv:2009.05440 (2020)
  * [21] Zhang, W., Liu, Y., Dong, C., Qiao, Y.: Ranksrgan: Generative adversarial networks with ranker for image super-resolution. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 3096–3105 (2019)


# Towards "all-inclusive" Data Preparation to ensure Data Quality

Valerie Restat
University of Hagen
Hagen, Germany
<EMAIL_ADDRESS>

###### Abstract

Data preparation, especially data cleaning, is very important to ensure data
quality and to improve the output of automated decision systems. Since there
is no single tool that covers all steps required, a combination of tools –
namely a data preparation pipeline – is required. Such process comes with a
number of challenges. We outline the challenges and describe the different
tasks we want to analyze in our future research to address these. A test data
generator which we implemented to constitute the basis for our future work
will also be introduced in detail.

_K_ eywords data preparation $\cdot$ data quality $\cdot$ data cleaning

##  1 Introduction

The use of automated decision systems, e.g. based on machine learning, is
increasing [1]. This impacts our everyday lives. Good quality data is the
prerequisite for good quality decisions and predictions. To achieve this, data
preparation, also referred to as data wrangling, data engineering or data
preprocessing is necessary. This includes among other steps data profiling,
data cleaning, data transformation [2], and data integration. As shown in [3]
and [4], data cleaning is of particular importance for the improvement of
machine learning based solutions.

Different tools exist for the individual preprocessing steps, which in turn
support different algorithms. For example, KATARA [5] can be used to detect
pattern violations using knowledge bases while Dboost [6] integrates various
algorithms for outlier detection. In addition, some tools cover multiple steps
of data preparation. This includes proprietary tools such as Tableau
Prep111https://www.tableau.com/products/prep or Talend Data
Preparation222https://www.talend.com/products/data-preparation/ as well as
open source products like OpenRefine333https://openrefine.org/.

However, no single tool covers all existing error types and data preparation
tasks [7, 8]. Different tools and algorithms need to be combined in a
pipeline. This process poses a number of challenges.

The rest of the paper is structured as follows. In Section 2, we outline the
current challenges regarding data preparation pipelines. A description of our
research focus follows in Section 3. Different tasks that we will analyze in
our future work will address those challenges. In Section 4, GouDa, our test
data generator implemented as a basis for our analyses, will be introduced in
greater detail. Section 5 summarizes the paper.

##  2 Challenges of Data Preparation

A number of challenges arise during data preparation. These concern the input
of the preparation, the preparation process itself and the result of this
process. We published a first version of challenges and resulting requirements
in [9].

#### Input of Data Preparation

In the age of big data, more and more data is being stored and processed. In
addition to the sheer _volume_ and _velocity_ of data, this also leads to a
_variety_ of different data formats [10].

#### Data Preparation Process

The data preparation process or workflow poses many challenges. On the one
hand, data preparation is mostly an _iterative_ process with an _ad-hoc
evaluation_ [11]. On the other hand – outlined in Section 1 – _no single
dominant tool_ , covering all existing error types and preprocessing tasks,
exists yet. In addition, many tools require a preprocessed version of the data
set, e.g. with a header and uniform delimiters, from the start [12]. So far,
there is a _lack of intelligent solutions_ and a great deal of _human
involvement_ is still required [12, 7]. Hence, data preparation usually is a
very _time-consuming_ process [11].

Concerning big data, _volume_ , _variety_ , and _velocity_ generate further
challenges [10]. Many tools do not offer sufficient scalability. Moreover,
varied data errors and data formats compromise data quality. The quality of
the data is considered in greater detail below. Due to the velocity, streaming
data and data changes must be taken into account in addition to static data.

Data preparation and new algorithms must be _reproducible_ [13]. In
particular, the iterative nature of data preprocessing leads to further
challenges in this context [14].

#### Result of Data Preparation

The goal of data preparation is to ensure the best possible data quality. No
single definition of data quality exists in literature. It is nevertheless
undisputed that data quality addresses a wide range of different aspects.
_Accuracy_ is one of the most referenced aspects. However, _availability_ and
_timeliness_ , _usability_ as well as _completeness_ and _relevance_ are
crucial factors too [15, 16]. Another important criterion for data quality is
the evaluation in terms of _bias and fairness_ [17]. In context of fairness,
_explainability_ is also a critical factor [18].

In order to meet the aforementioned challenges, we proposed a holistic data
preparation tool [9], illustrated in Figure 1.

Figure 1: Holistic Data Preparation Tool

It reflects the challenges described and, obtained from them, the requirements
for such a tool. We will analyze these requirements in greater detail in
Section 3.

##  3 Research Focus

In line with the challenges mentioned and the holistic data preparation tool
presented, this section introduces specific research topics to be addressed in
our future work. These can be divided into three parts, as illustrated in
Figure 2.

Figure 2: Research Focus

The first one is related to the goal of data preparation: data quality. Data
quality-related aspects include ensuring fairness and a comprehensive
evaluation. The second one focuses on how to achieve this goal: the process-
related aspects. These include abstraction from IT-knowledge, minimizing of
human involvement, and tracking data lineage to ensure reproducibility as well
as scalability. The third one, on the other hand, concerns the data itself and
their impact on both data quality and the process. Changes to the data and to
streaming data as well as the formats of semi-structured and unstructured data
are taken into account. The individual aspects are analyzed in detail below.

###  3.1 Data quality-related

At the outset, aspects related to data quality are considered. As described in
Section 2, data quality includes a great variety of aspects and does not only
deal with accuracy. Following topics were taken into consideration by us:

#### Ensuring fairness and explainability

Fairness is a crucial element in data quality. It must be ensured that there
is no bias in the data. This particularly applies to automated decision
systems [1]. In addition, explainability is also an important requirement
[18].

Bias may already be present in the data, e.g. due to historical perceptions
[17]. This can even be increased because automated decision systems such as
machine learning models often reproduce trends and patterns in the data,
reinforcing the bias [18]. In addition, bias can also be introduced into the
data by preprocessing itself, as shown in [19].

For a bias already present in the data, we want to analyze the applicability
and interaction of different fairness metrics. Fairness is a very complex
topic with many different types of bias in existence [20]. We will investigate
types of bias already covered by specific fairness metrics, identify
unresolved issues and explore possible solutions.

For a bias arising during preprocessing, the detection shall be scrutinized in
our future work. To combat bias, but also to ensure explainability, it is
necessary to measure the extent to which preprocessing changes the data [18].
Different avenues for the detection and measurement of changes in the data and
distribution will be researched. Furthermore, we seek to identify those
algorithms that generate the risk of introducing bias. This could make the use
of consumer labels, compared to those envisioned in [21], conceivable.

#### Comprehensive integrated evaluation

Since fairness is only one aspect of data quality, a comprehensive integrated
evaluation is required. This should incorporate all the manifold aspects of
data quality.

Accomplishing an automated, fully comprehensive evaluation adds to our scope
of future research. To realize this, the impact of automated checks based on
arrival time, range of data and accepted values will be examined. Different
metrics, standards and specifications will be included in this project. The
use of consumer labels is an additional consideration.

###  3.2 Process-related

Next, aspects concerning the preprocessing process itself shall be reviewed.

#### High-level application and abstraction from IT-knowledge

As detailed in Section 2, human involvement is necessary for data
preprocessing. Domain experts ought to be able to utilize tools without in-
depth IT-knowledge which leads us to the conclusion that a significant demand
for high-level applications exists.

To achieve abstraction from IT-knowledge, different approaches are
conceivable, that nevertheless demand thorough analyses in our research.
Visual and interactive approaches as well as approaches based on natural
language processing and example-oriented concepts will be included.

#### Minimizing human involvement

Human involvement is still time-consuming and should be minimized. Thus, we
will analyze how machine learning can be used to automatically preprocess data
or generate suggestions for domain experts. The use of dictionaries and
knowledge stores would also be conceivable for this purpose.

#### Data Lineage and Reproducibility

Reproducibility is a very important aspect. The proposed tool should provide
data lineage tracking capabilities, as well as rollback capabilities.

As shown in [13], the raw data and already preprocessed data alone are
insufficient for reproducibility. Our aim is to assess how the reproducibility
of data cleaning pipelines can be guaranteed and to explore possibilities for
the tracking of data lineage. Individual steps of the pipeline need to be made
available to others efficiently so that they can be reproduced in the long
term.

#### Scalibility

One of the preprocessing challenges posed by big data is volume. Handling the
increasing amount of data requires scalable tools.

We will analyze different possibilities to achieve efficient parallel
execution and shall include a comparison of different parallel python data
frames, such as Dask [22], Modin [23], and others.

###  3.3 Data-related

In this section, aspects regarding the data itself will be considered. There
are three main aspects.

#### Support of semi-structured and unstructured data

Most research in data preparation focuses on structured data [24]. Also, most
tools are only suitable for structured data. The processing and data quality
of semi-structured and unstructured data, on the other hand, has not yet been
sufficiently researched.

This issue will be addressed in our research. On the one hand, we want to
analyze the data quality of semi-structured and unstructured data. Many
metrics like accuracy are only suitable for structured data. We therefore want
to investigate which types of evaluation criteria and metrics are suitable for
semi-structured and unstructured data.

On the other hand, we will investigate the processing of semi-structured and
unstructured data. We want to analyze which methods and algorithms are
suitable for which types of data and how a qualitative preprocessing of semi-
structured and unstructured data can be ensured.

#### Data changes

Methods of data preparation are often based on statistical characteristics and
rules. These may change over time which may cause problems. This is also
referred to as _concept drift_ , a known problem especially in terms of
machine learning models [25]. This can e.g. lead to the fact that integrity
constraints may not hold [26].

A future focus of our work will be on the analysis of how to handle changes in
the data. This includes both semantic and syntactic changes. We want to
investigate how to detect changes in the data and how to decide whether any
preprocessing algorithms need to be adapted. Mean imputation is one example.
If the mean value of an attribute changes significantly due to changes in the
data, the imputation may need to be adjusted.

The different data formats also play a role here. In addition to structured
data, semi-structured and unstructured data also require consideration.

#### Streaming data

More and more data come from sensors or mobile devices. Similar problems
caused by data changes also occur with streaming data. Further difficulties
arise since the total amount of data and its statistical characteristics are
not known beforehand. New approaches to data preparation on distributed
streams of data need to be explored in future [24].

We want to analyze the difference between streaming data and static data sets.
Accordingly, the identification of the special characteristics of streaming
data compared to static data and their impact on data preparation contributes
to our further research. We also want to examine which data cleaning
approaches for static data sets are applicable to data streams. For scenarios
where static data methods are not applicable, we want to explore new
approaches.

###  3.4 Related Work

Other approaches also strive for holistic data preparation and analysis. Data
Civilizer 2.0 [27], for example, is a workflow system for data cleaning and
parameter-tuning of machine learning models. Another example is KNIME [28], an
open source software that focuses on integration to help third-party
developers embed their tools, regardless of the domain.

The proposed categories are helpful for comparing different tools and
approaches. The goal of preprocessing, or the aspect of data quality targeted,
varies depending on the use case. For example, fairness plays a very important
role when it comes to personal data, but it is not relevant for sensor data.
Data quality is thus independent of the the process-related aspects, but
should always be taken into account. Data Civilizer 2.0 and KNIME do not
provide information about data quality aspects.

The process-related aspects divide the tools depending on their functionality.
For example, Data Civilizer 2.0 focuses on support for domain experts through
visualization of the data and debugging components. With KNIME,
reproducibility also plays an important role, e.g. by not removing old nodes.
Therefore, old versions can still produce the same results later. Both offer a
graphical user interface to abstract from the code.

The implementation of the functionality varies depending on the data type. The
preparation process differs depending on whether you are working with
structured or unstructured data. Other challenges come with streaming data.
Data Civilizer 2.0 focuses on structured data. KNIME, on the other hand, also
offers components for unstructured data.

Even though the main emphasis of this tools is more on analysis than on data
preparation, approaches like these could provide a good starting point for the
planned research.

##  4 Test Data

At the beginning we have considered how to analyze and evaluate existing and
new approaches. First, data that covers different application scenarios is
needed. These should include many different types of errors that can occur in
practice.

We encountered the problem that not enough suitable data sets were available
for a detailed analysis, due to the following reasons:

  * •

Non-public: Some data sets are not publicly available. The reasons for this
vary: some records are proprietary, for others no source is listed or the link
provided is outdated.

  * •

Missing ground truth: For some data sets, no ground truth is specified. Yet,
this is a prerequisite for the evaluation of tools.

  * •

Error types: Even when data sets are publicly available and ground truth is
present, they usually contain only one or a few types of errors. For a
comprehensive analysis of data cleaning tools, a significant range of
different error types must be considered.

  * •

Error rate: Real data normally contain rather a small number of errors. This
limits the validity of the evaluation of data cleaning tools.

To overcome these problems, we have implemented a test data generator called
GouDa - Generation of universal Data Sets [29]. Error types and rates are
configurable with GouDa. Basis for this is the data generator implemented as
part of the EvoBench project [30]. We have extended the data generator to
include the following properties:

  * •

Error types: A variety of different error types can be generated. The included
errors are listed in Table 1. The classification is based on the definition of
[31] and has been extended to include further important error types, based on
[32, 33, 34, 35].

  * •

Error rate: The rate is freely configurable.

  * •

Reproducible: The generated data sets are reproducible insofar that the same
type of error can be generated on the same attribute of the same tuple.

  * •

Scalable: Various options for scaling are available.

  * •

Portable: For a better portability, the data sets are generated in JSON
format.

  * •

Realistic: For data sets to be as realistic as possible, different lists with
realistic names, words or addresses are included. The opportunity to include
lists with possible attribute values is provided too.

  * •

Ground truth: For each data set with errors, a data set with ground truth is
provided. Additionally, the errors are logged in a text file.

Table 1: Error types supported by GouDa Level | Error Type
---|---
An Attribute Value of a Single Tuple | Missing value
Syntax violation
Interval violation
Set violation
Misspelled error
Inadequate value to the attribute context
Value items beyond the attribute context
Meaningless Value
Erroneous entry
The Values of a Single Attribute | Uniqueness value violation
Synonyms existence
Outlier*
Missing Attribute
The Attribute Values of a Single Tuple | Semi-empty tuple
Inconsistency among attribute values
Irrelevant observation
The Attribute Values of Several Tuples | Redundancy about an entity
Inconsistency about an entity
Bias
Noise

##  5 Conclusion and Outlook

In this paper we have described current challenges in the context of data
preparation pipelines. Derived from this, we have proposed a holistic data
preparation tool. To accomplish the properties of such a tool, we have
presented a number of aspects to be analyzed in our future research. These
include data quality-related aspects, such as _ensuring fairness_ and a
_comprehensive integrated evaluation_ as well as process-related aspects, such
as _abstraction from IT-knowledge_ , _minimizing human involvement_ ,
_reproducibility_ , and _scalibility_. We also want to investigate data-
related aspects, such as _semi-structured and unstructured data_ , _data
changes_ and _streaming data_. As basis for our research we have introduced a
data generator called _GouDa – Generation of universal Data Sets_. This
permits the generation of erroneous data for comprehensive testing and
evaluation.

We are currently using data sets generated by GouDa to investigate the
following topics: First, we examine the current data preparation tools. We
want to determine which tools are suitable for which data preparation steps
and which error types. Secondly, since no single tool covers everything so
far, we investigate the possible tool combinations for a most efficient data
preparation pipeline. In context of this pipeline, we are currently focusing
on data-related problems. Here we examine the different data formats. This
concerns semi-structured and unstructured data and also streaming data. We
deal with the question which special features these formats entail. Depending
on this, we investigate which concepts for structured data can be transferred
to these formats and where solutions must be adapted or newly developed.
Following this, in future research we will address all topics described in
Section 3.

## References

  * [1] Julia Stoyanovich, Bill Howe, and H. V. Jagadish.  Responsible data management.  Proc. VLDB Endow., page 3474–3488, 2020.
  * [2] Meike Klettke and Uta Störl.  Four Generations in Data Engineering for Data Science: The Past, Presence and Future of a Field of Science.  Datenbank-Spektrum, 2021.
  * [3] Mohammad Mahdavi et al.  Towards Automated Data Cleaning Workflows.  In Proceedings of the Conference on "Lernen, Wissen, Daten, Analysen", Berlin, Germany, September 30 - October 2, 2019, pages 10–19. CEUR-WS.org, 2019.
  * [4] Peng Li et al.  CleanML: A Study for Evaluating the Impact of Data Cleaning on ML Classification Tasks.  In 37th IEEE International Conference on Data Engineering, ICDE 2021, Chania, Greece, April 19-22, 2021, pages 13–24. IEEE, 2021.
  * [5] Xu Chu, John Morcos, Ihab F. Ilyas, Mourad Ouzzani, Paolo Papotti, Nan Tang, and Yin Ye.  KATARA: A data cleaning system powered by knowledge bases and crowdsourcing.  In Proceedings of the 2015 ACM SIGMOD International Conference on Management of Data, Melbourne, Victoria, Australia, May 31 - June 4, 2015, pages 1247–1261. ACM, 2015.
  * [6] Yuxiao Zhang, Xiaorong Wang, Bingyang Li, Wei Chen, Tengjiao Wang, and Kai Lei.  Dboost: A fast algorithm for dbscan-based clustering on high dimensional data.  In Advances in Knowledge Discovery and Data Mining - 20th Pacific-Asia Conference, PAKDD 2016, Auckland, New Zealand, April 19-22, 2016, Proceedings, Part II, pages 245–256. Springer, 2016.
  * [7] Ziawasch Abedjan et al.  Detecting data errors: Where are we and what needs to be done?  Proc. VLDB Endow., pages 993–1004, 2016.
  * [8] Dessislava Petrova-Antonova and Rumyana Tancheva.  Data cleaning: A case study with openrefine and trifacta wrangler.  In Quality of Information and Communications Technology, pages 32–40, Cham, 2020. Springer International Publishing.
  * [9] Valerie Restat, Meike Klettke, and Uta Störl.  Towards a Holistic Data Preparation Tool.  In DataPlat, 25th International Conference on Extending Database Technology (EDBT), 2022, Edinburgh, UK, 2022.
  * [10] Fakhitah Ridzuan and Wan Mohd Nazmee Wan Zainon.  A Review on Data Cleansing Methods for Big Data.  Procedia Computer Science, pages 731–738, 2019.  The Fifth Information Systems International Conference, 23-24 July 2019, Surabaya, Indonesia.
  * [11] Sanjay Krishnan et al.  Towards reliable interactive data cleaning: a user survey and recommendations.  In Proceedings of the Workshop on Human-In-the-Loop Data Analytics, HILDA@SIGMOD 2016, San Francisco, CA, USA, June 26 - July 01, 2016, page 9. ACM, 2016.
  * [12] Mazhar Hameed and Felix Naumann.  Data Preparation: A Survey of Commercial Tools.  SIGMOD Rec., pages 18–29, 2020.
  * [13] Mateusz Pawlik et al.  A Link is not Enough - Reproducibility of Data.  Datenbank-Spektrum, pages 107–115, 2019.
  * [14] Lukas Rupprecht et al.  Improving Reproducibility of Data Science Pipelines through Transparent Provenance Capture.  Proc. VLDB Endow., pages 3354–3368, 2020.
  * [15] Li Cai and Yangyong Zhu.  The Challenges of Data Quality and Data Quality Assessment in the Big Data Era.  Data Sci. J., page 2, 2015.
  * [16] Fatimah Sidi et al.  Data quality: A survey of data quality dimensions.  In 2012 International Conference on Information Retrieval & Knowledge Management, Kuala Lumpur, Malaysia, March 13-15, 2012, pages 300–304. IEEE, 2012.
  * [17] Evaggelia Pitoura.  Social-minded Measures of Data Quality: Fairness, Diversity, and Lack of Bias.  ACM J. Data Inf. Qual., pages 12:1–12:8, 2020.
  * [18] Meike Klettke, Adrian Lutsch, and Uta Störl.  Kurz erklärt: Measuring data changes in data engineering and their impact on explainability and algorithm fairness.  Datenbank-Spektrum, pages 245–249, 2021.
  * [19] Sebastian Schelter and Julia Stoyanovich.  Taming technical bias in machine learning pipelines.  IEEE Data Engineering Bulletin (Special Issue on Interdisciplinary Perspectives on Fairness and Artificial Intelligence Systems), pages 39–50, 2020.
  * [20] Ninareh Mehrabi et al.  A Survey on Bias and Fairness in Machine Learning.  ACM Comput. Surv., pages 115:1–115:35, 2021.
  * [21] Christin Seifert, Stefanie Scherzinger, and Lena Wiese.  Towards Generating Consumer Labels for Machine Learning Models.  In 2019 IEEE First International Conference on Cognitive Machine Intelligence (CogMI), Los Angeles, CA, USA, December 12-14, 2019, pages 173–179. IEEE, 2019.
  * [22] James Crist.  Dask & numba: Simple libraries for optimizing scientific python code.  In 2016 IEEE International Conference on Big Data (IEEE BigData 2016), Washington DC, USA, December 5-8, 2016, pages 2342–2343. IEEE Computer Society, 2016.
  * [23] Devin Petersohn et al.  Flexible rule-based decomposition and metadata independence in modin: A parallel dataframe system.  Proc. VLDB Endow., pages 739–751, 2021.
  * [24] Xu Chu et al.  Data Cleaning: Overview and Emerging Challenges.  In Proceedings of the 2016 International Conference on Management of Data, SIGMOD Conference 2016, San Francisco, CA, USA, June 26 \- July 01, 2016, pages 2201–2206. ACM, 2016.
  * [25] Gregory Ditzler and Robi Polikar.  Incremental learning of concept drift from streaming imbalanced data.  IEEE Trans. Knowl. Data Eng., pages 2283–2301, 2013.
  * [26] Maksims Volkovs et al.  Continuous data cleaning.  In IEEE 30th International Conference on Data Engineering, Chicago, ICDE 2014, IL, USA, March 31 - April 4, 2014, pages 244–255. IEEE Computer Society, 2014.
  * [27] El Kindi Rezig et al.  Data civilizer 2.0: A holistic framework for data preparation and analytics.  Proc. VLDB Endow., pages 1954–1957, 2019.
  * [28] Alexander Fillbrunn et al.  Knime for reproducible cross-domain analysis of life science data.  Journal of Biotechnology, pages 149–156, 2017.  Bioinformatics Solutions for Big Data Analysis in Life Sciences presented by the German Network for Bioinformatics Infrastructure.
  * [29] Valerie Restat, Gerrit Boerner, André Conrad, and Uta Störl.  Gouda - generation of universal data sets: improving analysis and evaluation of data preparation pipelines.  In DEEM ’22: Proceedings of the Sixth Workshop on Data Management for End-To-End Machine Learning Philadelphia, PA, USA, 12 June 2022, pages 2:1–2:6. ACM, 2022.
  * [30] André Conrad et al.  Evobench: Benchmarking schema evolution in nosql.  In Performance Evaluation and Benchmarking - 13th TPC Technology Conference, TPCTC 2021, Copenhagen, Denmark, August 20, 2021, Revised Selected Papers, pages 33–49. Springer, 2021.
  * [31] Paulo Oliveira et al.  A taxonomy of data quality problems.  In 2nd Int. Workshop on Data and Information Quality, pages 219–233, 2005.
  * [32] Erhard Rahm and Hong Hai Do.  Data cleaning: Problems and current approaches.  IEEE Data Eng. Bull., pages 3–13, 2000.
  * [33] Won Y. Kim et al.  A taxonomy of dirty data.  Data Min. Knowl. Discov., pages 81–99, 2003.
  * [34] Heiko Müller and Johann Christoph Freytag.  Problems, methods, and challenges in comprehensive data cleansing.  Technical report hub-ib-164, Humboldt University, 2003.
  * [35] Lin Li, Taoxin Peng, and Jessie Kennedy.  A rule based taxonomy of dirty data.  GSTF Journal on Computing (JoC), 2011.