# AutoCure: Automated Tabular
Data Curation Technique for ML Pipelines

Mohamed Abdelaal  Software AG
Darmstadt, Germany
<EMAIL_ADDRESS>Rashmi Koparde  Otto von Guericke University
Magdeburg
Magdeburg, Germany
<EMAIL_ADDRESS>Harald Schöning  Software AG
Darmstadt, Germany
<EMAIL_ADDRESS>
###### Abstract

Machine learning algorithms have become increasingly prevalent in multiple
domains, such as autonomous driving, healthcare, and finance. In such domains,
data preparation remains a significant challenge in developing accurate
models, requiring significant expertise and time investment to search the huge
search space of well-suited data curation and transformation tools. To address
this challenge, we present AutoCure, a novel and configuration-free data
curation pipeline that improves the quality of tabular data. Unlike
traditional data curation methods, AutoCure synthetically enhances the density
of the clean data fraction through an adaptive ensemble-based error detection
method and a data augmentation module. In practice, AutoCure can be integrated
with open source tools, e.g., Auto-sklearn, H2O, and TPOT, to promote the
democratization of machine learning. As a proof of concept, we provide a
comparative evaluation of AutoCure against 28 combinations of traditional data
curation tools, demonstrating superior performance and predictive accuracy
without user intervention. Our evaluation shows that AutoCure is an effective
approach to automating data preparation and improving the accuracy of machine
learning models.

###### Index Terms:

data curation, data quality, data augmentation, machine learning, tabular data

##  I Introduction

Data Quality Problems: In the recent decades, machine learning (ML) has
developed a strong impact in several application domains, including autonomous
driving, gaming, healthcare, logistics, and finance [9]. Developing ML models,
in such domains, typically involves data acquisition from multiple distinct
sources. For instance, perception and state estimation systems in self-driving
cars usually employ ML models trained on data collected from sensor readings,
GPS fixes, and historical records [35]. After data acquisition, data
preparation and transformation processes are commonly carried out to bring the
collected data into a state suitable for model building. In fact, the
performance of such ML models broadly depends on the quality of the input data
[25]. Specifically, the predictive performance may drastically degrade
whenever the input data is noisy or contains erroneous instances. In general,
real-world data mostly suffers from heterogeneous error profiles due to
improper join operations, noisy communication channels, inaccurate and/or
incomplete manual data entry, etc. [16]. Due to such technical/human-related
problems, different error types, e.g., outliers, pattern/rule violations,
duplicates, inconsistencies, and implicit/explicit missing values, may
simultaneously emerge in a data set.

Challenges: Since high data quality is a necessary prerequisite for improving
the performance of ML models, the input data has to be properly curated before
being employed for modeling tasks. A typical data curation pipeline usually
begins with detecting erroneous instances before repairing them. In fact,
there exist plenty of commercial and open source tools for error detection and
repair, e.g., Talend, OpenRefine, Trifacta, Tamr, NADEEF, and HoloClean [2].
However, such tools still suffer from several problems. First, they usually
require domain knowledge and skilled individuals who can formulate such
knowledge as a set of rules/constraints. Exceptions to these rules/constraints
are typically reported to the users to take appropriate corrective actions,
either by correcting the data, or by fine-tuning the data and/or the rule
definitions. Due to user involvement, it may become challenging to adopt these
data curation tools without users possessing sufficient data expertise. A
possible solution could be the adoption of an automated rule generation tool,
e.g., Metanome [27], DCFinder [28], and RTClean [11]. Nevertheless, the
performance of such tools, in terms of the quality of the generated rules,
varies broadly for different datasets, according to our experiments [1].
Moreover, their high computational complexity hinders their integration with
data curation tools.

Second, most ML-based error detection tools, e.g., RAHA [23] and HoloDetect
[15], cannot recognize the type of the detected errors, i.e., whether they are
rule/pattern violations, outliers, missing values, or duplicates. They simply
train a detection classifier which differentiates between erroneous and clean
data instances. Due to lack of knowledge about the detected error types, the
task of selecting a well-suited data repair tool becomes non-trivial. As a
workaround, one may implement multiple data repair tools. However, this
solution may increase the complexity of the curation pipeline, since the
search space of repair candidates drastically increases. For example, if a
data instance $X_{i}=x_{i,1},\cdots,x_{i,k}$ has an empty cell $x_{i,j}=NaN$,
where $k$ is the number of columns, the cell $x_{i,j}$ has to be properly
imputed. Knowing that there exist plenty of imputation methods which generate
distinct values $a_{1},\cdots,a_{n}$, it becomes challenging to select the
optimal repair candidate. To this end, it becomes necessary to further extend
the data preparation pipeline, which in turn increases the complexity, via
implementing additional tools, e.g., BoostClean [20] and CPClean [17].

Proposed Method: To overcome these challenges, we introduce a novel automated
data curation pipeline, referred to as AutoCure, which comprises two main
modules, namely the adaptive ensemble-based error detection module and the
clean data augmentation module. The core idea behind AutoCure is to
synthetically enhance the density of the clean fraction of the input data
(which typically consists of clean and dirty data instances). According to the
information theory, the process of noise reduction is equivalent to adding up
more data with similar quality [32]. Pillared on this theory, AutoCure strives
to increase the proportion of clean data instances for the sake of reducing
the impact of noisy/erroneous data instances. Accordingly, we can avoid the
problems of data repair via replacing it with a clean data augmentation
module.

To further clarify the idea behind AutoCure, Figure 1 shows an illustrative
example of three different scenarios. Figure 1a shows an ML model (dashed
curve) trained on a small data set, which includes a set of clean instances
(green circles) and a set of noisy data instances (red circles). Due to the
noisy instances, such an ML model fail to precisely describe the linear data
set. To improve the predictive performance, we may repair the noisy data
instances using a traditional data repair method, as depicted in Figure 1b.
However, the repair process is usually complex, involving user intervention
and additional steps to select the best repair candidates. Moreover,
traditional data repair methods can not restore the true values of the noisy
instances. A second strategy, adopted by AutoCure, for dealing with the poor
data quality problem is to deliberately increase the density of clean
instances, as Figure 1c depicts. In this case, the impact of the dirty
instances on the ML modeling process is significantly reduced. As a result,
the generated ML model broadly resembles the curve shown in Figure 1b.

(a) Impact of noise on model building

(b) Traditional Curation

(c) AutoCure

Figure 1: Three ML models trained on different types of data

Summary of Contributions: To sum up, the paper provides the following
contributions: (1) We introduce a novel two-stages curation pipeline for
dealing with erroneous tabular data through increasing the density of clean
data instances. (2) We propose an adaptive ensemble-based error detection
method. The proposed detector annotates a cell $x_{i,j}$ as dirty, if the cell
is detected by at least $k$ base detectors. AutoCure dynamically adapts the
threshold $k$ to overcome the data exclusion problems which typically occur
while extracting clean data instances from the input data set. Such data
problems usually happen due to inaccurate detections of the base detectors
(cf. Section III). (3) We conduct extensive experiments to evaluate AutoCure
relative to the ground truth, to obtain the performance upper-bound, and a
wide collection of baseline methods in terms of the modeling accuracy and the
training time. The results show that AutoCure constantly achieves a comparable
performance to the ground truth data together with requiring less training
time compared to the case of running several repair methods to select the best
repair candidates. To the best of our knowledge, AutoCure is the first work
which automates data curation in ML pipelines via reducing the impact of
erroneous data and considering the density of the clean fraction as a
practical solution for the data quality problems.

Structure of the Paper: The remainder of this paper is structured as follows.
Section II introduces the different components in the proposed pipeline,
together with highlighting the main assumptions. Section III presents the
adaptive ensemble-based error detection method and explains the data exclusion
problems. In Section IV, we discuss the augmentation of clean data using the
so-called variational autoencoder (VAE). Section V introduces our proof-of-
concept implementation, before presenting the obtained results for different
data sets. Section VI discusses the related work, before Section VII concludes
the paper with an outlook on future work.

##  II Overview & Architecture

Figure 2: Architecture of the proposed method

In this section, we present the architecture of AutoCure together with our
assumptions. The proposed architecture consists of two modules, an adaptive
ensemble-based error detector and a clean data augmentation, which perform
three main tasks, including (1) the detection of erroneous data instances, (2)
the extraction of clean data fraction including all classes, and (3) the
generation of additional data from the same distribution as the clean
fraction. Figure 2 gives an overview over all components with their respective
inputs. At the outset, a dirty data set is used as an input to the adaptive
ensemble-based error detection module. The rationale behind such a module is
to maximize the detection recall, defined as the fraction of erroneous data
instances that are detected. Moreover, such an adaptive approach enables us to
deal with the data exclusion problems (cf. Section III) which usually occur
due to poor detection performance.

The adaptive ensemble-based error detection module implements a number of base
error detectors, such as missing values (MV) detector, outliers detector,
duplicates detector, rule violations detector, and ML-based error detectors
(cf. Section V for more details). Each base detector generates a list of
indices corresponding to the erroneous instances detected by each method. To
combine these detections, we implement an adaptive voting mechanism where the
instances detected by at least $k$ methods are annotated as erroneous. After
generating a list of erroneous data instances, a data sampler is used to
extract a clean data fraction. Before augmenting the clean data fraction, a
data checker scrutinizes it to detect possible occurrence of the relevant data
exclusion problems. If one of these problems is detected, the data sampler
iteratively adjusts the value of the voting threshold $k$ so that all relevant
data exclusion problems are resolved.

The extracted clean data fraction is then used as an input to a variational
autoencoder (VAE) to generate a new set of data from the same distribution. As
Figure 2 illustrates, the VAE module implements two feed-forward neural
networks, namely an encoder and a decoder. The encoder learns the distribution
of the latent space representation of the clean data fraction. Afterward, the
decoder uses the sampled latent vector to generate data instances similar to
the inputs. In this context, the optimization problem is to minimize (1) the
reconstruction loss function which compares the inputs with the decoder-
generated values, and (2) the KL divergence which statistically differentiates
between the probability distributions of the inputs and the generated data.
After generating additional clean data, a data integrator merges the newly
generated data with the original dirty data. The merged data set is then
transformed using feature engineering and wrangling tools, e.g.,
normalization, embedding, and feature crossing, before training ML models on
the transformed data. In our paper, we assume that the dirty data sets have
heterogeneous error profiles. In other words, different error types may
simultaneously exist in a data set, e.g., outliers and missing values. To
enable clean data augmentation, we also assume that each dirty data set
includes a set of clean instances and a set of dirty instances. In the next
section, we present the implementation details of the adaptive ensemble-based
error detection module.

##  III Adaptive Error Detection

Before delving into the details of our adaptive ensemble-based error detection
method, it is necessary to highlight that the existing error-dependent
detectors tackles only a subset of the errors in a data set. For instance, a
missing value detector finds only null values while overlooking other errors.
Similarly, outlier detectors usually employ statistical measures to
differentiate between legitimate data and outliers while ignoring other error
types, such as rule violation and duplicates. Accordingly, the low detection
recall of such detectors prevents them from being individually employed. To
maximize the detection recall, there exist several advanced methods which can
be subsumed under two major classes, namely the ML-based methods and the
ensemble methods. The first class embodies the detectors which implement semi-
supervised binary classifiers to differentiate between clean and dirty data
instances [26, 15]. The second class comprises the detectors which combine the
detections of several error-dependent methods [2] to improve the detection
recall together with providing a knob for controlling the detection precision
(defined as the fraction of relevant instances, e.g., actual dirty data
instances, among the detected instances).

Nevertheless, ML-based and ensemble detectors usually suffer from relevant
data exclusion problems when the main objective is to extract a clean fraction
from the dirty data set, as required in AutoCure. Specifically, data exclusion
problems typically arise when the extracted clean fraction is either (1) empty
or (2) does not comprise all classes (i.e., unique labels in a data set) which
exist in the dirty data set. The first type, referred to as the attribute-
level exclusion, commonly emerges when an error detector flags all data
instances of a certain attribute as erroneous (cf. red instances in the
attribute $A2$ in Figure 3a). In this case, the data sampler will fail to
extract a clean fraction because all records contain erroneous data instances.
The second type, referred to as the class-level exclusion, occurs when an
error detector flags all data instances from one or more classes as erroneous
(cf. red instances in Figure 3b). Accordingly, these classes (e.g., class “1”
in Figure 3b) will not be represented in the clean fraction, which in turn
broadly changes the data distribution of the augmented data. In particular,
both data exclusion problems may arise due to (1) the false positives of some
error-dependent or ML-based detectors, and (2) fixing the value of the voting
threshold $k$, in ensemble detectors, for all data instances.

(a) Attribute-level

(b) Class-level

Figure 3: Examples of data exclusion problems

To overcome the data exclusion problems, AutoCure introduces an adaptive
ensemble method which can dynamically adjust the voting threshold to preserve
all classes in the extracted clean fraction. To this end, the proposed
adaptive ensemble-based method makes the detection decisions iteratively.
Specifically, the data sampler adjusts the value of the voting threshold $k$
whenever it detects the occurrence of data exclusion problems. Figure 4 shows
an example of tuning the voting threshold $k$ while detecting errors in a
dirty toy data set, which consists of five records $R1$-$R5$ and two
attributes $A1$ and $A2$. After running all available detectors, i.e.,
$S1$-$S7$, the iterative voting mechanism begins with an initial value of the
voting threshold, i.e., $k=3$. In this scenario, the cell $C12$ has been
detected by three detection methods, i.e., $s_{1}$, $s_{3}$ and $s_{4}$, while
the cell $C42$ has been detected by four detection methods, i.e., $s_{1}$,
$s_{2}$, $s_{5}$ and $s_{6}$.

Figure 4: Example of adapting the voting threshold

According to the traditional Min-K voting mechanism, the cells $C12$ and $C42$
are flagged as erroneous since they have been detected by at least three
methods. Consequently, the records containing these cells are to be excluded
from the clean fraction, as shown on the top left table (i.e., $D_{clean}$ for
$k=3$). As a result, a class-level data exclusion problem emerges, where the
clean fraction lacks records with the class “1”. To resolve this problem,
AutoCure updates the voting threshold $k$ from three to four in the second
iteration. In this case, the cell $C12$ does not exceed the threshold.
Accordingly, it is flagged as a clean cell and thus its record will be
included in the clean fraction, as shown on the top right table (i.e.,
$D_{clean}$ for $k=4$).

To clarify the implementation details of AutoCure, Figure 5 demonstrates the
pseudocode of the proposed data curation pipeline. At the outset, AutoCure
implements an inventory of several error-dependent and ML-based detectors
$\mathcal{S}_{base}$. Each detector $s_{i}$ generates a list of indices of the
erroneous instances, e.g., $R_{i}=\\{(r_{u},a_{v})\\}$, where $r_{u}$, $a_{v}$
are the $u$th record and the $v$th attribute in the dirty data set
$D_{dirty}$. Afterward, all detections of $m$ detectors $R_{all}$ are merged,
before estimating the cells count $Q_{cells}$, i.e., which quantifies for each
cell $c_{j}$, the number of times they have been detected (cf. lines 6-8).
Initially, AutoCure assigns the current voting threshold $k_{cur}$ a small
value which is larger than one (e.g., $k_{cur}=2$ while setting $k_{cur}=1$
implies using only one error-dependent or ML-based detection method). In the
first iteration, the algorithm uses an initial value, i.e.,
$k_{cur}=k_{init}$, to decide for each cell whether it is erroneous (cf. lines
17 and 18). The final list of detections $R_{ensemble}$ is then utilized to
extract a clean fraction $D_{clean}$, as shown in line 19.

1:dirty data set $D_{dirty}$, initial voting threshold $k_{init}$,
augmentation budget $N_{aug}$, base detectors
$\mathcal{S}_{base}=s_{1},\cdots,s_{m}$, update rate $\alpha_{u}$

2:for all detector $s_{i}\in\mathcal{S}_{base}$ do $\triangleright$ Running
all available detectors

3: Estimate detections $R_{i}\leftarrow s_{i}(D_{dirty})$

4: Append $R_{i}$ to the list of all detections $R_{all}$

5:Initialize the cells counter $Q_{cells}\leftarrow 0$

6:for all cell $c_{j}\in R_{all}$ do $\triangleright$ Estimating the cells
count

7: if cell $c_{j}$ in cells counter $Q_{cells}$ then

8: Increment the counter $Q_{cells}[c_{j}]\leftarrow Q_{cells}[c_{j}]+1$

9:Initialize the list of missing classes $L_{miss}\leftarrow\emptyset$

10:Initialize the temporary thresholds $k_{attr},k_{class}\leftarrow k_{init}$

11:while true do

12: for all cell $c_{j}\in Q_{cells}$ do $\triangleright$ Adaptive Min-K
voting

13: if classes($c_{j})\in L_{miss}$ then

14: Assign current threshold $k_{cur}\leftarrow k_{class}$

15: else

16: Assign current threshold $k_{cur}\leftarrow k_{attr}$

17: if $Q_{cells}[c_{j}]\geq k_{cur}$ then

18: Append cell $c_{j}$ to the final detection list $R_{ensemble}$

19: Extract clean fraction $D_{clean}\leftarrow D_{dirty}\backslash
R_{ensemble}$

20: if $D_{clean}=\emptyset$ then $\triangleright$ Attribute-level data
exclusion

21: Update threshold $k_{attr}=k_{attr}+\alpha_{u}$

22: else if classes($D_{clean}$) $\neq$ classes($D_{dirty}$) then
$\triangleright$ Class-level

23: Find $L_{miss}\leftarrow$ classes($D_{dirty}$) $\backslash$
classes($D_{clean}$)

24: Update threshold $k_{class}=k_{class}+\alpha_{u}$

25: if the difference $|K_{class}-k_{attr}|\geq 2\times\alpha_{u}$ then

26: Update threshold $k_{attr}=k_{attr}+\alpha_{u}$

27: else$\triangleright$ No data exclusion problems exist

28: Break

29:Generate additional data $D_{aug}\leftarrow$ run_vae($D_{clean},N_{aug}$)

30:Combine data sets $D_{final}\leftarrow D_{dirty}\cup D_{aug}$

Figure 5: Ensemble-based error detection algorithm

Before running the variational autoencoder, the algorithm checks for the
occurrence of data exclusion problems (cf. lines 20-28). An attribute-level
data problem can be detected if the size of the clean data fraction is equal
to zero. If an attribute-level problem is detected, then the algorithm
increases the temporary threshold $K_{attr}$ by the value of the update rate
$\alpha_{u}$ (cf. line 21). Increasing the threshold clearly implies relaxing
the voting mechanism, which in turn reduces the number of detected cells. In
the subsequent iteration, the algorithm re-estimates the list of all
detections $R_{all}$ and the clean fraction $D_{clean}$. Next, it checks again
for possible attribute-level data exclusion, before inspecting $D_{clean}$ for
class-level exclusion problems. To this end, it simply compares the number of
classes in the dirty data classes($D_{dirty}$) and in the clean fraction
classes($D_{clean}$). If a class-level problem is detected, AutoCure estimates
the list of missing classes $L_{miss}$ (cf. lines 23-24). In this case,
AutoCure deliberately differentiates between the cells belong to records whose
classes are included in $L_{miss}$, e.g., $C12$ and $C42$ in Figure 4, and all
other cells, e.g., $C52$.

The rationale behind this differentiation is to achieve a high detection
precision while ensuring that all classes in $D_{dirty}$ also exist in the
clean fraction $D_{clean}$. To this end, AutoCure relaxes the voting algorithm
for the cells belonging to the missing classes, e.g., $C12$ and $C42$, to
facilitate their inclusion in the clean fraction. Whereas, AutoCure becomes
more strict with other cells, e.g., $C52$, since their classes are already
represented in the clean fraction. For this purpose, AutoCure adopts two
different temporary thresholds $k_{attr}$ and $k_{class}$. It updates the
threshold $k_{class}$ when a class-level exclusion occurs. In the next
iteration, the algorithm checks whether each cell $c_{j}$ belongs to the
records of the missing classes (cf. line 13). If that is the case, it employs
the temporary threshold $k_{class}$, which is always greater than $k_{attr}$
by the value of the update rate $\alpha_{u}$, for making the detection
decisions. Otherwise, it uses the threshold $k_{attr}$. The algorithm keeps
iterating till all data exclusion problems are resolved. Afterward, the final
clean fraction is merged with the dirty data, before carrying out the typical
wrangling and feature engineering tasks.

##  IV Clean Data Augmentation

In this section, we discuss the technical details of the clean data
augmentation module. After extracting a clean data fraction, a data
augmentation module is used to generate data from the same distribution as the
clean fraction. In this regard, we examined three data augmentation methods
dedicated to tabular data, including MODALS [6], Variational Autoencoders
(VAE) [18], and Conditional Tabular Generative Adversarial Network (CTGAN)
[12]. Through our experiments, we found that VAE outperforms the other two
methods111For brevity, we omitted the results of our comparative study between
the three augmentation methods in this paper.. Specifically, we examined the
three methods in an ML pipeline for several datasets. The VAE data
augmentation achieved higher predictive accuracy (F1 scores $\geq$ 0.98%) than
the other two methods. Therefore, we decided to integrate the VAE method with
our adaptive error detection.

In general, an autoencoder implements two components, namely an encoder and a
decoder. The encoder compresses the input data set in a low dimensional space,
referred to as the latent space representation. Afterward, the decoder
exploits the latent space representation to recover the input data set. For
this purpose, autoencoders typically have a reconstruction loss function to
compare the original data with the values generated by the decoder. For data
augmentation, autoencoders perform some variations in the latent space. To
this end, the VAE module trains its encoder to extract the parameters of data
distributions rather than the latent space representation [18]. As depicted in
Figure 2, the decoder generates a two-dimensional vector consisting of the
mean $\mu_{x}$ and the variance $\sigma_{x}$. Afterward, VAE generates a set
of data instances from the Gaussian distribution
$\mathcal{N}(\mu_{x},\sigma_{x})$ formed using the extracted parameters. Such
a set of data instances are then used as an input to the decoder to generate
additional clean data.

In addition to the reconstruction loss, the VAE module employs the KL
divergence to distinguish between the probability distributions of the
original data set (i.e., the clean fraction in AutoCure) and the generated
data. Generally, the KL divergence is a statistical measure which quantifies
how a probability distribution differs from a reference distribution. The VAE
module strives to reduce the value of KL divergence via optimizing the mean
and variance to simulate the input distribution. Aside from the optimization
metrics, the process of sampling data from a distribution, parameterized by
the generated means and variances, is not differentiable. In this case, it is
relatively challenging to perform backpropagation (necessary to optimize the
weights of the encoder and the decoder) over the random node $Z$. To overcome
this problem, the reparameterization process is performed to enable
backpropagation through the random node. To this end, the reparameterizer
turns the random node $Z\sim\mathcal{N}(\mu_{x},\sigma_{x})$ into a
differentiable function $Z=\mu+\sigma\odot\epsilon$, where
$\epsilon\sim\mathcal{N}(0,1)$ represents the standard Gaussian distribution,
and it is irrelevant for taking the gradients (which is a necessary step in
the backpropagation process).

##  V Performance Evaluation

In this section, we assess the effectiveness of AutoCure relative to a set of
baseline methods. Through the experiments, we seek to answer the following
questions: (1) How does AutoCure compare to the baseline methods in terms of
their impact on downstream ML models (the performance of such models is
quantified in terms of the predictive accuracy and the training time)? (2)
What is the minimum amount of augmented clean data required to yield a
comparable performance as the ground truth? (3) What is the impact of the
voting threshold $k$ on the detection accuracy? (4) what is the impact of the
error rate of a data set on the performance of AutoCure? (5) How does AutoCure
contribute to the model performance? We first describe the setup of our
evaluations, before discussing the results and the lessons learned throughout
this study.

###  V-A Experimental Setup

We evaluated AutoCure using six real-world data sets covering different data
sizes and different error rates. Table I summarizes the characteristics of
each data set. Three of such data sets, i.e., Adult, breast Cancer, and Smart
Factory, are associated with classification (CL) tasks. Whereas, the other
three data sets, i.e., Nasa, Housing, and Soil Moisture, have regression tasks
(REG). To control the evaluation environment, we opted to inject different
realistic errors into the data sets. To this end, AutoCure leverages the BART
tool [4] which provides a systematic control over the amount of errors and how
hard these errors are to be repaired. To inject errors using BART, we use a
set of denial constraints to generate different error types, such as rule
violation, outliers, and missing values.

TABLE I: Data sets used in the evaluation where $\gamma$ is the error rate (%)
and the error types are rules violation (RV), outliers (OT), missing values
(MV), and typos (TP)

Data Set | Rows | Columns | Error Types | Error Rate | ML Task
---|---|---|---|---|---
Adult [19] | 45223 | 15 | RV, OT | 0.09 | CL
Breast Cancer [10] | 700 | 12 | MV, TP, OT | 0.4 | CL
Smart Factory [5] | 23645 | 19 | MV, OT | 0.83 | CL
Nasa [34] | 1504 | 6 | MV, OT, TP | 0.13 | REG
Housing [14] | 507 | 14 | MV, OT | 0.24 | REG
Soil Moisture [31] | 679 | 129 | MV, OT | 0.3 | REG

We compare the performance of AutoCure to a wide collection of baseline
methods composed of different error detection and repair tools. Table II lists
the error detection and repair methods. The combination of each detection and
repair method represents a baseline method. For instance, the baseline B2
implies detecting errors in a dirty data set using dBoost and generating
repair candidates using an ML-based imputation algorithm. In the list of
repair methods, we also include the ground truth and dirty cases to show the
performance upper- and lower-bound. The standard imputer replaces the
erroneous data instances with the mean values of the numerical columns and
with a dummy string for the categorical columns. Alternatively, the ML-based
imputer employs a decision tree model to predict the erroneous numerical
instances, while it employs missForest [33] for categorical data. The
evaluation metrics used in the evaluation comprise the detection accuracy,
i.e., expressed in terms of the detection precision, recall, and F1 score, the
training time, and the predictive accuracy.

TABLE II: Error detection and repair tools used as baselines

Idx | Detectors | Idx | Detectors | Idx | Repair Method
---|---|---|---|---|---
B | dBoost [24] | M | Min-K [2] | - | Dirty
E | ED2 [26] | N | NADEEF [8] | 1 | Ground Truth
F | FAHES [29] | Q | Outliers: IQR | 2 | ML Imputer
H | HoloClean [30] | R | RAHA [23] | 3 | Standard Imputer
I | Outliers: IF [36] | S | Outliers: SD | 4 | BARAN [22]
K | KATARA [7] | V | MV-Detector |  |

For the VAE module, both the encoder and the decoder have been implemented as
feed-forward neural networks. The inputs to the VAE module are a training set,
a test set, the dimensions of the clean fraction, the number of nodes in the
hidden layers, and the number of latent factors. The number of nodes in the
input layer of the encoder and in the output layer of the decoder has been set
to the number of attributes in the clean fraction. The encoder and decoder
comprise two hidden layers, and the number of nodes in the first and second
hidden layers has been set to 50 and 12, respectively. We employ the Adam
optimizer to optimize the parameters and custom loss, which are the combined
mean square error and the KL divergence. We add the ReLU activation function
for each hidden layer in the encoder and the decoder.

To examine the performance of AutoCure and the baseline methods, we
implemented a neural network, using Keras, which can deal with regression,
binary, and multi-class classification tasks222The source code of AutoCure,
the baseline methods, and the data sets, are to be publicly released with the
final version of the paper.. AutoCure leverages a Bayesian-based informed
search method, referred to as Optuna [3], to tune the learning rate, the
number of hidden layers, and the number of units per layer. For all
experiments, we fixed the number of training epochs to 500. All experiments
have been repeated ten times, where the means of the ten runs are reported.
For clarity, we omitted the standard deviations when they are less than 1%. We
run all the experiments on an Ubuntu 16.04 LTS machine with 32 2.60 GHz cores
and 128 GB memory.

###  V-B Results

#### Voting Threshold

Before delving into the results obtained for AutoCure and the baseline
methods, we first assess the impact of changing the voting threshold to
properly motive for the adaptive voting mechanism implemented in AutoCure.
Figures 6a and 6b depict the detection accuracy of the traditional Min-k
detection method while detecting erroneous instances in the Housing and Smart
Factory data sets, respectively. Both figures show that increasing the voting
threshold $k$ has distinct influences on the detection recall (defined in
Section II) and the detection precision (defined in Section III). For
instance, Figure 6a demonstrates that increasing the voting threshold $k$
usually leads to reducing the detection recall due to annotating fewer data
instances as erroneous. On the other hand, increasing the threshold $k$ causes
the precision to be improved thanks to the precise estimation of erroneous
instances using a consensus among multiple error detection methods. In light
of this trade-off, it is not often straightforward to accurately select the
optimal value of $k$ for a certain application scenario. Moreover, we noticed
during our experiments that hard-coding the value of $k$, as implemented in
the traditional Min-K method, typically prevents us from extracting a balanced
clean data fraction. Therefore, AutoCure iteratively adjusts the value of $k$
to combat the relevant data exclusion problems.

(a) Housing

(b) Smart Factory

Figure 6: Detection accuracy against different values of the threshold $k$
(Min-k)

(a) Adult (F1 Score)

(b) Breast Cancer (F1 Score)

(c) Smart Factory (F1 Score)

(d) Nasa (MSE)

(e) Soil Moisture (MSE)

(f) Housing (MSE)

Figure 7: Modeling accuracy of clean datasets, AutoCure (denoted as “A”), and
a set of baseline methods

#### Predictive Accuracy

In this set of experiments, we estimate the accuracy of a neural network
trained on different versions of the examined data sets. For AutoCure, we set
the number of generated instances to 6000 for all data sets. Figure 7 depicts
the predictive performance of a neural network trained on a clean data
set333In this context, a clean data set means the ground truth version of the
data. (cf. green area in Figure 7a) and a set of processed data sets (cf. blue
area in Figure 7a). The processed data sets comprise the data sets curated by
AutoCure (abbreviated as A in Figure 7) and the ones curated using different
combinations of baseline tools. For the Adult data set, Figure 7a compares the
performance of AutoCure, the clean data set, and 24 baseline methods in terms
of the F1 score of the neural network. As the figure depicts, AutoCure
achieves similar accuracy as the clean data set (both achieve an average F1
score of 89.5%). It is important to notice that some curation combinations
yield a similar performance as AutoCure, while others, e.g., Q2, Q3, K2, and
K3, fail to accurately curate the dirty data set.

Figure 7b illustrates a similar evaluation of AutoCure and the baseline
methods using the Breast Cancer data set. Again, AutoCure outperforms all
baseline methods and achieves a similar performance as the clean data set (an
average F1 score of 93.6% in case of AutoCure and 94% in case of the clean
data set). Figure 7c depicts an evaluation of the compared methods using the
Smart Factory data set. Obviously, AutoCure outperforms most baseline methods
as well as the clean data set (on average by 46%). It is important to
highlight that some baseline methods, e.g., E2, H2, and N2, achieve an F1
score of one. However, such a performance of these baseline methods broadly
depends on the data set.

For the Nasa data set, AutoCure outperforms most baseline methods (an average
MSE of 9.3 in case of AutoCure and 15.3 in case of the clean data set). For
the Soil Moisture data set, AutoCure achieves a reasonable accuracy (an
average MSE of 11.7) relative to the clean version (an average MSE of 2.78)
and some baseline methods (average MSE of 3.15 and 3.55 for N2 and H2,
respectively). In fact, it is important to highlight the consistency of the
results obtained by AutoCure over different data sets. While, the other
baseline methods lack such a consistency, e.g., N2 and H2 perform poorly in
case of the Nasa data set (Figure 7d). Finally, Figure 7f shows the results,
in terms of MSE, in case of the Housing data set. Again, AutoCure achieves a
similar performance as the clean data set (both achieve an average MSE of
12.2).

#### Training Time

Aside from the predictive accuracy, we also evaluate AutoCure and the baseline
methods in terms of the training time. Table III lists the average training
times of the neural network trained on the clean and the curated data sets.
Due to data augmentation, AutoCure causes the training time to be relatively
increased for all data sets. However, for some data sets, such as Adult, the
training time in case of AutoCure is slightly higher than that using the clean
data set (on average by circa 2.9 minutes). Moreover, a fair comparison should
consider a realistic scenario, where data scientists typically lack knowledge
about which curation tools to implement in their ML pipelines. Due to the
existence of multiple traditional data curation methods, it becomes necessary
to examine and execute multiple traditional data curation methods, as in
BoostClean [20] and ActiveClean [21]. Accordingly, a practical estimation of
the training time, caused by the traditional curation methods, should consider
a combined estimation of all training times. In this case, AutoCure
significantly outperforms the traditional curation methods (named “Combined”
in Table III) for all data sets.

(a) Adult

(b) Smart Factory

(c) Breast Cancer

(d) Nasa

(e) Soil Moisture

Figure 8: Modeling accuracy of AutoCure for different sizes of the augmented
clean data

(a) Adult

(b) Smart Factory

(c) Breast Cancer

(d) Nasa

(e) Soil Moisture

Figure 9: Training time of AutoCure for different sizes of the augmented clean
data TABLE III: Average training time of compared methods

| Adult | Cancer | Nasa | Moisture | Factory | Housing
---|---|---|---|---|---|---
Clean | 1416.8 | 35.5 | 51.9 | 51.4 | 996 | 43.5
B2 | 1452.6 | 35.7 | 65.4 | 55.9 | 1265.8 | 45.65
B3 | 1467.2 | 41.4 | 63.2 | 52.6 | 1138.7 | 46.45
E2 | 1488.9 | 41.8 | 66.5 | 55.9 | 1138.2 | 47.08
E3 | 1487.6 | 37.1 | 54 | 51.6 | 1095.3 | 47.8
H2 | 1487.4 | 42.65 | 65.77 | 55.3 | 1152.4 | 45.83
H3 | 1495.8 | 33 | 53.89 | 55.4 | 1023.6 | 46.74
K2 | 1423.4 | 43.9 | 66.1 | 50.7 | 1105.7 | 42.85
K3 | 1370.5 | 38.1 | 63 | 46.3 | 1097.1 | 40.24
R2 | 1488.6 | 43.9 | 64.8 | 56.3 | 946.1 | 47.21
R3 | 1439.6 | 41.1 | 64.8 | 53.1 | 1166.9 | 45.61
Combined | 14601 | 398.4 | 627.5 | 533.1 | 11129.8 | 455.46
AutoCure | 1595.2 | 233.7 | 183.1 | 300.9 | 1352 | 300.6

#### Augmentation Size

In this set of experiments, we estimate, for each data set, the impact of the
size of the augmented clean data on the predictive performance and the
training time. A general remark from the experiments is that increasing the
amount of augmented data broadly improves the predictive performance at the
expense of increasing the training time. For instance, Figure 8a shows the F1
score of the neural network trained on a clean data version of the Adult data
set (cf. the red dashed line) and multiple versions curated by AutoCure with
different sizes of the augmented clean data. The figure shows that only a
small amount of augmented data (circa 1% of the size of the Adult data set) is
sufficient to achieve the same performance as the clean data set. Similar
results have been obtained in case of the Smart Factory data set, where 1% of
the size of the clean data set is sufficient to reach the quality of the clean
data set.

For smaller data sets, the experiments showed that more clean data instances
have to be generated. In the case of the Breast Cancer data set, Figure 8c
demonstrates that AutoCure requires more clean instances (at least by 375%
relative to the size of the original data set) to achieve a similar accuracy
as the clean data set. For the Nasa data set, AutoCure achieves a comparable
performance to that of the clean data set when the size of the generated clean
data is large enough (circa 200% of the size of the original data set) to
minimize the impact of erroneous instances. Similar results have been obtained
for the Soil Moisture data set (cf. Figure 8e).

Figure 9 depicts the training times of different data sets curated by
AutoCure. For example, Figure 9a illustrates the training time of the clean
version (cf. the red dashed line) and several versions curated by AutoCure. As
the figure depicts, AutoCure requires circa three minutes more than it can be
needed by the clean version of the data set. In case of the Smart Factory data
set, AutoCure requires slightly more time (at most by 18.4%) to satisfy the
quality requirements. For the Breast Cancer data set, AutoCure requires an
additional 100 seconds to achieve a similar accuracy as the clean data (cf.
Figure 9c). Along a similar line, an additional 80 seconds are needed by
AutoCure in case of the Nasa data set (cf. Figure 9d). For the Soil Moisture
data set, due to the associated regression task, AutoCure has to generate
numerous clean instances. As a result, AutoCure requires additional training
time of circa 400 seconds to reach the performance upper-bound.

#### Error Rate

In this set of experiments, we assess the impact of the error rate on the
performance of AutoCure and a set of baseline methods. We fixed the number of
generated clean instances and gradually increased the amount of injected
errors (i.e., missing values and numerical outliers). Figure 10 depicts the
predictive accuracy of the neural network, in terms of MSE for regression
tasks and F1 score for classification tasks, while being trained on the clean
data set and the curated data sets using AutoCure and the baseline methods. In
Figure 10a, AutoCure shows that it is mostly agnostic to the amount of errors
in the Nasa data set. Even with a fixed number of generated clean instances,
the curve of AutoCure is slightly increased, which is still below the red
dashed line of the clean data set. Conversely, all traditional data curation
methods poorly perform with increasing the error rate. Figures 10b and 10c)
show similar results for the Housing and Breast Cancer data sets. Figures
11a-11c shows the training time of the neural network while increasing the
error rate. Obviously, increasing the error rate has almost no influence on
the training time of all examined data sets (except some spikes which rarely
occur at certain error rates).

(a) Nasa

(b) Housing

(c) Breast Cancer

Figure 10: Impact of error rate on the predictive accuracy

(a) Nasa

(b) Housing

(c) Breast Cancer

Figure 11: Impact of error rate on the training time

#### Overfitting

The results, discussed above, show that AutoCure broadly improves the
predictive performance while requiring no repair operations. However, such a
high performance may be a result of overfitting. Therefore, in this set of
experiments, we examine the generated neural network models, while using
AutoCure, for overfitting. Figure 12 depicts the learning curves, with respect
to the number of epochs and the predictive performance, for several data sets.
For instance, in the case of the Smart Factory data set, Figure 12a
demonstrates that the model performance, in terms of the binary accuracy, is
growing over time till it reaches a plateau. Obviously, both curves for the
training and the validation sets are close to each other. This result has been
repeated for all other data sets (cf. Figures 12b-12f). Accordingly, the high
performance of AutoCure, depicted in Figure 7, is not a result of overfitting,
and it mainly occurs due to the high density of clean data instances.

###  V-C Results Summary

In this section, we highlight the main findings obtained in this work. The
first finding revolves around the performance consistency. Specifically, our
results revealed that some baseline methods may achieve similar or even better
performance than AutoCure (e.g., accuracy of the models trained on data
curated by E2, E3, and H2 in Figure 7c). Nevertheless, it is important to
mention that these results are broadly data set dependent, i.e., the
performance may vary from one data set to another. Moreover, multiple
experiments have to be carried out to identify a well-suited curation strategy
for each data set. Alternatively, the results showed that AutoCure shows a
consistency of the results over different data sets. For all examined data
sets, AutoCure managed to improve the quality of the data set through
emphasizing the clean fractions.

Another important finding is that small data sets usually require the
generation of more clean data instances (cf. Figure 8c). This behavior of
AutoCure is caused since the percentage of original clean instances, in small
data sets, is typically small. Accordingly, AutoCure generates a large number
of clean instances to satisfy the quality requirements. This finding also
applies to data sets with associated regression tasks, e.g., Housing, Soil
Moisture, and Nasa (cf. Figures 8d and 8e). In fact, regression models are
generally more sensitive to data errors than classification tasks, as found in
our experiments. For large data sets, e.g., Adult and Smart Factory, AutoCure
requires much less generated data to achieve a similar accuracy as the clean
data set. Furthermore, the results also showed that AutoCure, in some cases,
increases the training time due to increasing the number of training instances
(cf. Figures 9c, 9d and 9e). Finally, the results confirmed that AutoCure can
perform well even with high error rates, as shown in Figure 10.

(a) Smart Factory

(b) Adult

(c) Breast Cancer

(d) Nasa

(e) Soil Moisture

(f) Housing

Figure 12: Learning curves of the neural networks trained using AutoCure

##  VI related work

In this section, we report on the state-of-the-art techniques and tools
relevant to the data curation problem. In fact, there exist plenty of data
curation techniques and tools from academia and industry. For instance,
HoloClean [30] is an ML-agnostic data repair technique which infers the repair
values via holistically employing multiple cleaning signals to build a
probabilistic graph model. To repair pattern violations and inconsistencies,
OpenRefine [13] utilizes Google Refine Expression Language (GREL) as its
native language to transform existing data or to create repair values.
Similarly, BARAN [22] is a holistic configuration-free ML-based method for
repairing different error types. To this end, BARAN trains incrementally
updatable models which leverage the value, the vicinity, and the domain
contexts of data errors to propose correction candidates. To further increase
the training data, BARAN exploits external sources, such as Wikipedia page
revision history.

In fact, the above presented techniques, such as HoloClean, OpenRefine, and
BARAN, do not consider the requirements imposed by the downstream ML
applications. They tend to improve the data quality regardless of where and
how the data comes from or how the data will be consumed. Therefore, a new set
of techniques and tools has been emerged which strives to jointly optimize the
cleaning and modeling tasks. In other words, these ML-oriented methods focus
on selecting the optimal repair candidates with the objective of improving the
performance of specific predictive models. Accordingly, these methods assume
the availability of repair candidates from other ML-agnostic methods. For
instance, BoostClean [20] deals with the error repair task as a statistical
boosting problem. Specifically, it composes a set of weak learners into a
strong learner. To generate the weak learners, BoostClean iteratively selects
a pair of detection and repair methods, before applying them to the training
set to derive a new model.

ActiveClean [21] is another ML-oriented method, principally employed for
models with convex loss functions. It formulates the data cleaning task as a
stochastic gradient descent problem. Initially, it trains a model on a dirty
training set, where such a model is to be iteratively updated until reaching a
global minimum. In each iteration, ActiveClean samples a set of records and
then asks an oracle to clean them to shift the model along the steepest
gradient. A similar work is CPClean [17] which incrementally cleans a training
set until it is certain that no more repairs can possibly change the model
predictions. In fact, the ML-oriented methods do not introduce new data repair
techniques, such as HoloClean and BARAN. Instead, they tend to select the
already-existent repair candidates which may improve the predictive
performance. In comparison to AutoCure, such methods can be entirely avoided
since no repair candidates have to be generated in our solution. Accordingly,
AutoCure can avoid the complexities of searching for the repair candidates and
selecting the best subset from those candidates. Furthermore, the ML-oriented
methods are usually tailored to specific optimization methods and ML models,
e.g., ActiveClean limited to problems with convex loss functions.
Alternatively, AutoCure can be utilized with all ML tasks and optimization
methods.

##  VII Conclusion & Future Work

In this paper, we introduce a novel method for dealing with erroneous data,
referred to as AutoCure. As alternative to traditional data curation tools,
AutoCure does not require the recognition of the detected error types, since
it entirely ignores such noisy data instances. Instead, it deals with the
clean instances which can lead to a better modeling performance, if more clean
instances do exist. Moreover, AutoCure avoids the complicated processes for
searching the huge space of repair candidates and the process of selecting the
most suitable candidates. As a proof of concept, AutoCure has been evaluated
using six data sets and over 28 combinations of data curation tools. The
results showed that the data sets curated using AutoCure can achieve a similar
or even better performance as the clean data set. While AutoCure delivers an
outstanding performance over multiple data sets, it can be further improved
through: (1) integrating it with a data valuation tool, to reduce the training
time, and (2) dynamically adapting the number of instances to be augmented
depending on the size of the dirty data set and the associated ML task.

## Acknowledgment

This work was supported (in part) by the Federal Ministry of Education and
Research through grants 02L19C155, 01IS21021A (ITEA project number 20219).

## References

  * [1] Mohamed Abdelaal, Christian Hammacher, and Harald Schoening.  REIN: A Comprehensive Benchmark Framework for Data Cleaning Methods in ML Pipelines.  In 26th International Conference on Extending Database Technology (EDBT), March 2023.
  * [2] Ziawasch Abedjan, Xu Chu, Dong Deng, Raul Castro Fernandez, Ihab F Ilyas, Mourad Ouzzani, Paolo Papotti, Michael Stonebraker, and Nan Tang.  Detecting data errors: Where are we and what needs to be done?  Proceedings of the VLDB Endowment, 9(12):993–1004, 2016.
  * [3] Takuya Akiba, Shotaro Sano, Toshihiko Yanase, Takeru Ohta, and Masanori Koyama.  Optuna: A next-generation hyperparameter optimization framework.  In Proceedings of the 25rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2019.
  * [4] Patricia C Arocena, Boris Glavic, Giansalvatore Mecca, Renée J Miller, Paolo Papotti, and Donatello Santoro.  Messing up with bart: error generation for evaluating data-cleaning algorithms.  Proceedings of the VLDB Endowment, 9(2):36–47, 2015.
  * [5] Oliver Birgelen, Alexander; Niggemann.  Smart factory: High storage system data for energy optimization, 2018\.  accessed on January 2022.
  * [6] Tsz-Him Cheung and Dit-Yan Yeung.  Modals: Modality-agnostic automated data augmentation in the latent space.  In International Conference on Learning Representations, 2020.
  * [7] Xu Chu, John Morcos, Ihab F Ilyas, Mourad Ouzzani, Paolo Papotti, Nan Tang, and Yin Ye.  Katara: A data cleaning system powered by knowledge bases and crowdsourcing.  In Proceedings of the 2015 ACM SIGMOD international conference on management of data, pages 1247–1261, 2015.
  * [8] Michele Dallachiesa, Amr Ebaid, Ahmed Eldawy, Ahmed Elmagarmid, Ihab F Ilyas, Mourad Ouzzani, and Nan Tang.  Nadeef: a commodity data cleaning system.  In Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data, pages 541–552, 2013.
  * [9] Shi Dong, Ping Wang, and Khushnood Abbas.  A survey on deep learning and its applications.  Computer Science Review, 40:100379, 2021.
  * [10] Dheeru Dua and Casey Graff.  UCI machine learning repository, 2017.
  * [11] Daniel Del Gaudio, Tim Schubert, and Mohamed Abdelaal.  RTClean: Context-aware Tabular Data Cleaning using Real-time OFDs.  In 2023 IEEE International Conference on Pervasive Computing and Communications Workshops and other Affiliated Events (PerCom Workshops). IEEE, March 2023.
  * [12] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio.  Generative adversarial networks.  Communications of the ACM, 63(11):139–144, 2020.
  * [13] Kelli Ham.  Openrefine (version 2.5). http://openrefine. org. free, open-source tool for cleaning and transforming data.  Journal of the Medical Library Association: JMLA, 101(3):233, 2013\.
  * [14] D. Harrison and D.L. Rubinfeld.  Boston housing data, 1993.  Accessed on February 2021.
  * [15] Alireza Heidari, Joshua McGrath, Ihab F Ilyas, and Theodoros Rekatsinas.  Holodetect: Few-shot learning for error detection.  In Proceedings of the 2019 International Conference on Management of Data, pages 829–846, 2019.
  * [16] Ihab F Ilyas and Felix Naumann.  Data errors: Symptoms, causes and origins.  Data Engineering, page 4, 2022.
  * [17] Bojan Karlaš, Peng Li, Renzhi Wu, Nezihe Merve Gürel, Xu Chu, and Ce Zhang.  Nearest neighbor classifiers over incomplete information: From certain answers to certain predictions.  arXiv preprint arXiv:2005.05117, 2020.
  * [18] Diederik P Kingma and Max Welling.  Auto-encoding variational bayes.  arXiv preprint arXiv:1312.6114, 2013.
  * [19] Ron Kohavi et al.  Scaling up the accuracy of naive-bayes classifiers: A decision-tree hybrid.  In Kdd, volume 96, pages 202–207, 1996.
  * [20] Sanjay Krishnan, Michael J Franklin, Ken Goldberg, and Eugene Wu.  Boostclean: Automated error detection and repair for machine learning.  arXiv preprint arXiv:1711.01299, 2017.
  * [21] Sanjay Krishnan, Jiannan Wang, Eugene Wu, Michael J Franklin, and Ken Goldberg.  Activeclean: Interactive data cleaning for statistical modeling.  Proceedings of the VLDB Endowment, 9(12):948–959, 2016.
  * [22] Mohammad Mahdavi and Ziawasch Abedjan.  Baran: Effective error correction via a unified context representation and transfer learning.  Proceedings of the VLDB Endowment, 13(12):1948–1961, 2020.
  * [23] Mohammad Mahdavi, Ziawasch Abedjan, Raul Castro Fernandez, Samuel Madden, Mourad Ouzzani, Michael Stonebraker, and Nan Tang.  Raha: A configuration-free error detection system.  In Proceedings of the 2019 International Conference on Management of Data, pages 865–882, 2019.
  * [24] Zelda Mariet, Rachael Harding, Sam Madden, et al.  Outlier detection in heterogeneous datasets using automatic tuple expansion, 2016.
  * [25] Felix Neutatz, Binger Chen, Ziawasch Abedjan, and Eugene Wu.  From cleaning before ml to cleaning for ml.  IEEE Data Eng. Bull., 44(1):24–41, 2021.
  * [26] Felix Neutatz, Mohammad Mahdavi, and Ziawasch Abedjan.  Ed2: A case for active learning in error detection.  In Proceedings of the 28th ACM international conference on information and knowledge management, pages 2249–2252, 2019.
  * [27] Thorsten Papenbrock, Tanja Bergmann, Moritz Finke, Jakob Zwiener, and Felix Naumann.  Data profiling with metanome.  Proceedings of the VLDB Endowment, 8(12):1860–1863, 2015.
  * [28] Eduardo HM Pena, Eduardo C de Almeida, and Felix Naumann.  Discovery of approximate (and exact) denial constraints.  Proceedings of the VLDB Endowment, 13(3):266–278, 2019.
  * [29] Abdulhakim A Qahtan, Ahmed Elmagarmid, Raul Castro Fernandez, Mourad Ouzzani, and Nan Tang.  Fahes: A robust disguised missing values detector.  In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 2100–2109, 2018.
  * [30] Theodoros Rekatsinas, Xu Chu, Ihab F Ilyas, and Christopher Ré.  Holoclean: Holistic data repairs with probabilistic inference.  arXiv preprint arXiv:1702.00820, 2017.
  * [31] Felix M. Riese and Sina Keller.  Introducing a Framework of Self-Organizing Maps for Regression of Soil Moisture with Hyperspectral Data.  In IGARSS 2018 - 2018 IEEE International Geoscience and Remote Sensing Symposium, pages 6151–6154, Valencia, Spain, July 2018.
  * [32] Claude Elwood Shannon.  A mathematical theory of communication.  ACM SIGMOBILE mobile computing and communications review, 5(1):3–55, 2001.
  * [33] Daniel J Stekhoven and Peter Bühlmann.  Missforest—non-parametric missing value imputation for mixed-type data.  Bioinformatics, 28(1):112–118, 2012.
  * [34] D. Stuart Pope Thomas F. Brooks and Michael A. Marcolini.  Nasa airfoil self-noise dataset, 2014.  accessed on January 2022.
  * [35] De Jong Yeong, Gustavo Velasco-Hernandez, John Barry, and Joseph Walsh.  Sensor and sensor fusion technology in autonomous vehicles: A review.  Sensors, 21(6):2140, 2021.
  * [36] Ji Zhang.  Advancements of outlier detection: A survey.  ICST Transactions on Scalable Information Systems, 13(1):1–26, 2013\.