[1]Daniel Schwabe

[1]Division Medical Physics and Metrological Information Technology,
Physikalisch-Technische Bundesanstalt, Abbestr. 2-12, 10587, Berlin, Germany

2]Department of Medical Engineering, Technical University Berlin, Straße des
17. Juni 135, 10623, Berlin, Germany

3]Einstein Centre for Digital Future, Wilhelmstraße 67, 10117, Berlin, Germany

# The METRIC-framework for assessing data quality for trustworthy AI in
medicine: a systematic review

<EMAIL_ADDRESS>Katinka Becker<EMAIL_ADDRESS>Martin Seyferth
<EMAIL_ADDRESS>Andreas Klaß<EMAIL_ADDRESS>Tobias Schaeffter
<EMAIL_ADDRESS>*  [  [

###### Abstract

The adoption of machine learning (ML) and, more specifically, deep learning
(DL) applications into all major areas of our lives is underway. The
development of trustworthy AI is especially important in medicine due to the
large implications for patients’ lives. While trustworthiness concerns various
aspects including ethical, technical and privacy requirements, we focus on the
importance of data quality (training/test) in DL. Since data quality dictates
the behaviour of ML products, evaluating data quality will play a key part in
the regulatory approval of medical AI products. We perform a systematic review
following PRISMA guidelines using the databases PubMed and ACM Digital
Library. We identify 2362 studies, out of which 62 records fulfil our
eligibility criteria. From this literature, we synthesise the existing
knowledge on data quality frameworks and combine it with the perspective of ML
applications in medicine. As a result, we propose the METRIC-framework, a
specialised data quality framework for medical training data comprising 15
awareness dimensions, along which developers of medical ML applications should
investigate a dataset. This knowledge helps to reduce biases as a major source
of unfairness, increase robustness, facilitate interpretability and thus lays
the foundation for trustworthy AI in medicine. Incorporating such systematic
assessment of medical datasets into regulatory approval processes has the
potential to accelerate the approval of ML products and builds the basis for
new standards.

###### keywords:

Data Quality, Data Quality Dimensions, Data Quality Framework, Regulated
Medical AI Products, Deep Learning, Machine Learning, Artificial Intelligence,
Trustworthy AI, AI in Medicine, Medical Training Data, Systematic Review

## Introduction

During the last decade, the field of artificial intelligence (AI) and in
particular machine learning (ML) has experienced unprecedented advances,
largely due to breakthroughs in deep learning (DL) [1, 2, 3, 4, 5] and
increased computational power. Recently, the introduction of easy-to-use yet
still extremely capable models such as GPT-4 [6] and Stable Diffusion [7] has
further expanded the technology to an even broader audience. The large-scale
handling and implementation of AI [8] into fields such as manufacturing,
agriculture and food, automated driving, smart cities and healthcare has since
shifted the topic into the centre of attention of not just scholars and
companies but the general public.

The introduction of novel and disruptive technologies is typically accompanied
by an oscillating struggle between exploiting technological chances and
mitigation of risks. ML is proving to have great potential to improve many
aspects of our lives [9, 10, 11]. However, the race for implementation and
utilisation is currently outpacing comprehension of the technology. The
complex and black box character of AI applications has therefore largely
steered the public conversation towards safety, security and privacy concerns
[12, 13]. A lack of confidence of the general population in the transparency
of AI prevents its utilisation for society and economic growth. It can lead to
a slowed adoption of innovations in crucial areas and discourage innovators
from unlocking the technology’s full potential. Hence, the demand for
regulation (e.g., EU AI Act [14], US FDA considerations [15]) as well as the
need for an improved understanding of AI is ever increasing. This is of
particular importance in the field of healthcare due to its large impact on
people’s lives. The amount of ML solutions in medicine (research tools and
commercial products) is steadily on the rise, in particular in the fields of
radiology and cardiology [16, 17]. Despite breakthroughs up to human-level
performance [9, 18, 19, 20], ML backed medical products are mainly used as
diagnosis assistance systems [17] leaving the final decision to medical human
professionals. In particular, medical ML solutions are successfully solving
the task of image segmentation [21, 22, 23]. Due to the unknown consequences
of using AI for medical decision-making, more stringent regulatory
requirements are of high importance to accelerate the approval process of new
AI products into medical practice. Decision-making needs to be supported by
reliable health data to generate consistent evidence. One of the drivers for
evidence-based medicine approaches was the introduction of scientific
standards in clinical practice [24]. Since then, data integrity (defined by
the ALCOA-principles or ALCOA+ [25]) has become an essential requirement of
several guidelines, such as good clinical practice (GCP) [26], good laboratory
practice (GLP) [27] or good manufacturing practice (GMP) [28]. In the
pharmaceutical industry, data integrity plays a similarly important role as a
requirement for drug trials. While data integrity focuses on maintaining
accuracy and consistency of a dataset over its entire life cycle, data quality
is concerned with the fitness of data for use.

To improve confidence in AI utilisation in general, the focus is put on the
development of so-called trustworthy AI, which aims at overcoming the black
box character and developing a better understanding. Several approaches and
definitions for trustworthy AI have been discussed and published over the past
years by researchers [29, 30, 31, 32, 33], public entities [34, 35],
corporations [36], and organisations [37, 38]. Depending on the area of
interest, trustworthiness may include (but is far from limited to) topics such
as ethics; societal and environmental well-being; security, safety, and
privacy; robustness, interpretability and explainability; providing
appropriate documentation for transparency and accountability [29, 30, 31, 32,
33, 34, 35, 36, 37, 38]. One of the most critical parts of an AI is the
quality of its training data since it has fundamental impact on the resulting
system. It lays the foundation and inherently provides limitations for the AI
application. If the data used for training a model is bad, the resulting AI
will be bad as well (“garbage in, garbage out” [39]). Neural networks are
prone to learning biases from training data and amplifying them at test time
[40], giving rise to a much discussed aspect of AI behaviour: fairness [41].
Many remedies have been put forward to tackle discriminating and unfair
algorithm behaviour [42, 43, 44]. Yet, one of the main causes of undesirable
learned patterns lies in biased training data [45, 46]. Thus, data quality
plays a decisive role in the creation of trustworthy AI and assessing the
quality of a dataset is of utmost importance to AI developers, as well as
regulators and notified bodies.

The scientific investigation of data quality was initiated roughly 30 years
ago. The term data quality was famously broken down into so-called data
quality dimensions by Wang and Strong in 1996 [47]. These dimensions represent
different characteristics of a dataset which together constitute the quality
of the data. Throughout the years, general data quality frameworks have taken
advantage of this approach and have produced refined lists of data quality
dimensions for various fields of application and types of data. Naturally,
this has produced different definitions and understandings. Within this
systematic review, we transfer the existing research and knowledge about data
quality to the topic of AI in medicine. In particular, we investigate the
research question: Along which characteristics should data quality be
evaluated when employing a dataset for trustworthy AI in medicine? The
systematic comparison of previous studies on data quality combined with the
perspective on modern ML enables us to develop a specialised data quality
framework for medical training data: the METRIC-framework. It provides a
comprehensive list of 15 awareness dimensions which developers of AI medical
devices should be mindful of. Knowledge about the composition of medical
training data with respect to the dimensions of the METRIC-framework should
drastically improve comprehension of the behaviour of ML applications and lead
to more trustworthy AI in medicine.

### Limitations

We note that data quality itself is a term used in different settings, with
different meanings and varying scopes. For the purpose of this review, we
focus on the actual content of a dataset instead of the surrounding technical
infrastructure. We do so since the content is the part of a dataset which ML
applications use to learn patterns and develop their characteristics. We thus
exclude research on data quality considerations and frameworks within the
topic of data governance and data management [48, 49]. This concerns aspects
such as data integration [50], information quality management [51], ETL
processes in data warehouses [52], or tools for data warehouses [53, 54] which
do not affect the behavioural characteristics of AI systems. We also omit
records discussing case studies of survey data quality [55, 56]. Furthermore,
the METRIC-framework is intended for assessing the quality of a fixed dataset
with respect to a specific application. We do not consider assessing the data
quality of a dataset in vacuum, nor training strategies to cope with bad data
[57, 58, 59, 60, 61, 62].

We further point out that the use of the term AI in current discussions is
scientifically imprecise since discussions within the healthcare sector almost
exclusively revolve around the implementation of ML approaches, in particular
of DL approaches. Technically, the term AI spans a much wider range of
technologies than just DL as part of the field of ML. Due to the complexity of
DL applications and their proficiency in solving tasks deemed to require human
intelligence, the terms are currently often used interchangeably in
literature. We follow the same vocabulary here (e.g., “trustworthy AI”, “AI in
medicine”) but stress the limitation of our results to ML approaches.

## Results

In order to answer the research question “Along which characteristics should
data quality be evaluated when employing a dataset for trustworthy AI in
medicine?”, we conducted an unregistered systematic review following the
PRISMA guidelines [63]. Our predetermined search string contains variations of
the following terms: (i) data quality, (ii) framework or dimensions and (iii)
machine learning (see Methods for more details and the full search string).
The initial search of the databases PubMed and ACM Digital Library was
performed on the 17th of August 2023 and yielded 1735 unique results. After
title and abstract screening, adding references of the remaining records
(“snowballing”) and full text assessment, we find 62 records that match our
eligibility criteria (see Methods and Table S2). This represents the
literature corpus that serves as a foundation for answering the research
question. The full workflow is illustrated in Figure 1.

Figure 1: PRISMA flow diagram. The flow diagram shows the number of records
identified, included and excluded at the different stages of the systematic
review. The eligibility criteria for inclusion and exclusion are presented in
the bottom right hand side. From a total of 2362 identified studies, the
resulting literature corpus on data quality for trustworthy AI in medicine
includes 62 studies.

In Figure 2, the papers from our literature corpus are displayed according to
their publication year [47, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75,
76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94,
95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110,
111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124]. The
overarching topics contained in the corpus naturally divide the papers into
three categories: _general data_ (28 entries), _big data_ (7 entries) and _ML
data_ (27 entries). This reflects the historic development of the research
field of data quality during the last 30 years.

Figure 2: Studies included in our literature corpus sorted by publication
date. The 62 studies are divided into the three categories _general data_
(28), _big data_ (7) and _ML data_ (27), which represent major changes in the
perception of data quality. The studies’ affiliation to non-life science and
life science related topics is indicated as well.

### General data quality

The field first shifted into focus with digital and automatically mass-
generated data during the 1980s and 1990s causing a need for quality
evaluation and control on a broad scale. While during the first 10 years
landmark papers [47, 64] built the foundation for the field, the most recent
20 years have seen general data quality frameworks published more frequently
[68, 65, 66, 70, 67, 69, 71, 73, 72]. The literature corpus additionally
contains general data quality frameworks with high specificity to medical
applications [74, 76, 75, 77, 83, 82, 78, 79, 80, 84, 85, 81, 86, 87, 88]
while frameworks with high specificity to non-medical topics [125, 126] were
excluded.

The early data quality research in the 1980s and 1990s uncovered the lack of
objective measures to asses data quality, which led to the introduction of
task dependent dimensions and the establishment of a data quality framework
from the perspective of the data consumer [47]. Another fundamental challenge
in the data quality field is the efficient data storage while maintaining
quality. This was first investigated with the introduction of a data quality
framework from the perspective of the data handler [64]. Both approaches to
data quality proved to be useful and were unified in one framework [65]. In
the following years, the frameworks were further extended [66, 67], equipped
with measures [68, 69] and refined [70, 71]. Moreover, it became clear that
specialised fields such as the medical domain require adapted frameworks.

With the overarching question of how to improve patient care and the rise of
electronic health records (EHR) in the 90s of the last century, the need for
high data quality in the medical sector increased. The first comprehensive
data quality framework for medical data in the literature corpus was
established by conducting a survey of quality challenges in EHR [74]. It
considers, among other characteristics, accuracy, completeness and
particularly timeliness. However, accuracy is hard to quantify in the medical
context as even the diagnosis of experienced practitioners sometimes do not
coincide. Accordingly, the notion of concordance of differing data sources was
introduced [75]. Yet, the data quality frameworks for EHR could only be
transferred to other types of medical data to a certain extent. Thus, data
quality frameworks for particular data types such as immunisation data, public
health data or similar were put forward [76, 77, 78, 79, 80, 81]. The various
frameworks still suffered from inconsistent terminology and attempts were made
to harmonise the definitions and assessment [82, 83, 84, 85, 86, 87, 88].
Particularly, Kahn et al. [82] proposed a framework with exact definitions.
While these developments have advanced the understanding of data quality in
the context of medical applications, frameworks for EHR frequently focus on
the data quality of individual patients [74, 75], neglecting data quality
aspects for the overall population. In particular, representativeness is often
not a factor [74, 75] while it is a crucial property for secondary use of data
in clinical studies [76] or when reusing medical data as training data for ML
applications.

### Big data quality

As the amount of data from varying sources grew, conventional databases
reached their capacity and the field of big data emerged. Big data is
generally concerned with handling huge unstructured data streams that need to
be processed at a rapid pace, emphasising the need for extended data quality
frameworks. This development is reflected by a small wave of papers published
between 2015 and 2020 [89, 91, 95, 92, 90, 94, 93]. For example, the weaker
structure of the data encouraged the use of data quality frameworks that
include the data schema as a data quality dimension [89, 90]. Further, the
increasing amount of data requires the computational efficiency of the
surrounding database infrastructure to be a part of big data quality
frameworks [91, 92, 93]. Computational efficiency is also a limiting factor
when ML methods are applied to big data. While it is generally assumed, that
more data leads to better results, this has to be balanced with computational
capabilities. Hence, a data quality framework was developed that bridges the
gap between ML and big data [94]. We note, that the “4 V’s” (volume, velocity,
veracity and variety) of big data [127] implicitly suggest a framework for big
data quality. However, the “4 V’s” are in fact a direct characterisation or
definition for when data can be called big data, rather than representing data
quality dimensions. They therefore do not contribute to answering our research
question and are not further discussed. This might change in the future, when
data from wearables or remote patient monitoring sensors become available for
health management.

### ML data quality

The performance and behaviour of DL applications heavily depends on the
quality of the data used during training as this is the foundation from which
patterns are learned. The records of the literature corpus which discuss or
empirically evaluate the effect of data quality on DL deal with a wide variety
of data types and models. The earliest records starting in the 1990s employ
simple neural network (NN) designs and investigate data quality of tabular
data [100, 98, 99, 96, 97, 101], while records from the 2010s and 2020s
increasingly use advanced DL architectures and look at data quality in the
context of time series [102, 103, 104], images [107, 106, 118, 119, 117, 111,
109, 110, 114, 112, 120, 116, 105, 115, 121, 113, 108, 101], natural language
[121, 120, 123, 122] or molecular data [124, 114].

Contrary to the big data and general data quality literature from our corpus,
the DL papers focus on the evaluation of one or very few specific data quality
dimensions without (yet) considering broader theoretical data quality
frameworks. Dimensions that are predominantly investigated are those which can
easily be manipulated and lend themselves to be applicable to a wide range of
datasets irrespective of specific tasks. The most prominent dimension is
amount of data [123, 113, 111, 124, 109, 100, 110, 105, 114, 112, 107, 108,
122] which is empirically shown to benefit performance, albeit in a saturating
manner. Another dominant topic is completeness to which the ML community
almost exclusively refers to as missing data [104, 103, 101, 100, 120, 102].
The effect that data errors have on the DL application is also frequently
investigated. Specifically, this is done by separately looking at contaminated
features (inputs of a NN) [104, 98, 120, 105, 103, 99, 118, 119] and noisy
targets (predictions generated by a NN) [104, 120, 105, 107, 106, 99, 108,
122]. Many ML settings are classification tasks which is reflected by the
corpus often addressing label noise [120, 105, 107, 106, 108, 122]. One record
highlights the hefty weight that physicians’ annotations carry in medicine
[108]. In order to evaluate the effect of data quality (features or targets)
on ML applications, the training data is commonly manipulated. On the feature
(input) side, e.g., images are distorted by adjusting contrast whereas time
series sequences are disturbed by swapping elements. On the target side, e.g.,
correct labels are randomly replaced by false ones.

When it comes to the concrete behaviour change of the DL algorithm, most of
the DL papers in the literature corpus investigate the robustness of a model,
i.e. the stable behaviour of a model when facing erroneous or a limited amount
of inputs. Only few records investigate generalisability [121, 115, 101], a
model’s capability of coping with new, unseen data. Another noteworthy
exception is Ovadia et al. [121] who additionally study predictive
uncertainty.

Overall, theoretical data quality frameworks enjoy little attention by the ML
community due to the novelty of the ML research field. Papers often focus on
few specific data quality dimensions and tasks. Each task comes with its
specific data type, necessitating different approaches to manipulate the data
and measure these effects. The research dealing with the impact of manipulated
data is heavily skewed towards robust behaviour in the sense of predictive
performance. Other possibly affected aspects such as explainability or
fairness are underrepresented and to some degree neglected which is a
potential shortcoming for safety-critical applications such as medical
diagnosis predictions.

## METRIC-framework for medical training data

Figure 3: The METRIC-framework. This specialised framework for evaluating data
quality of medical training data includes a comprehensive set of awareness
dimensions. The inner circle divides data quality into five clusters. These
clusters contain a total of 15 data quality dimensions, which are shown on the
outer circle. The subdimensions presented in gray on the border of the figure
contribute to the superordinate dimension. Due to the shape of the graphic, we
refer to it as _wheel of data quality_.

The literature corpus has shown that while similar ideas exist for the
assessment of data quality across fields and applications, the idiosyncrasy of
each field or application can only be captured by specialised frameworks
rather than by a one-model-fits-all framework. The evaluation of data quality
plays a particularly important role in the field of ML due to the fact that
its behaviour is not only dependent on the algorithm choice but also strongly
depends on its training data. At the same time, ML is implemented in various
fields, each processing and requiring different types and qualities of data.
We therefore propose a specialised data quality framework for evaluating the
quality of medical training data: the METRIC-framework (Figure 3), which is
based on our literature corpus [47, 64, 68, 65, 66, 70, 67, 69, 71, 73, 72,
74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 89, 91, 95, 92, 90, 94,
93, 100, 98, 99, 96, 97, 102, 103, 107, 106, 118, 119, 111, 109, 110, 114,
112, 120, 116, 117, 105, 115, 121, 113, 123, 124, 101, 87, 88, 104, 108, 122].
From now on, we refer to data quality for training data of medical ML
applications only. We point out that our framework does not yet include a
guideline on the assessment or measurement of data qualities but rather
presents a set of awareness dimensions which play a central role in the
evaluation of data quality. Additionally, we note that the METRIC-framework is
specifically not designed to assess the data quality of a dataset in vacuum,
nor with respect to all potential ML applications the dataset might be used
for. Rather, the intention of the METRIC-framework is to assess the
appropriateness of a dataset with respect to a specific and known purpose of
the desired medical AI.

While examining the literature corpus, we found that terms describing data
quality appear under varying definitions, or often with no definition at all.
While standardisation efforts exist for the terminology in the context of
evaluating data quality [128, 72, 129], they are often not employed or did not
exist yet for older papers making comparisons difficult. Therefore as a first
step, we extracted all mentioned data quality dimensions from the literature
corpus together with their definitions (if present) and added them to a list.
This yielded 383 different terms with 637 mentions across all papers. Second,
we hierarchically clustered the terms with respect to their intended meaning
and according to their dependencies into clusters, dimensions and
subdimensions (see Methods for more details on data extraction). We thus
obtained 38 relevant dimensions and subdimensions which are displayed on the
outer circle of Figure 3. In Tables S4–S9, we provide a complete list of
definitions for all 38 relevant dimensions and subdimensions, as well as their
hierarchy and references with respect to the literature corpus. We adopted
definitions from a recent data quality glossary [128] if they existed there
and met our understanding of the dimension in the given context of medical
training data. If necessary, we included definitions given by Wang et al. [47]
in a second iteration. If none of these two sources suggested an appropriate
definition, we captured the meaning of the desired term on the basis of the
literature corpus and thus determined its definition in the context of medical
training data.

Figure 4: The cluster _data management_ is concerned with the effective usage
of the dataset. It includes basic requirements for the dataset but does not
address data quality issues regarding its content. Therefore, they can be seen
as a prerequisite for assessment using the METRIC-framework. Figuratively
speaking, the _data management_ cluster serves as a stable foundation for the
wheel of data quality.

The METRIC-framework encompasses three levels of details: _clusters_ which
pool similar dimensions; _dimensions_ which are individual characteristics of
data quality; and _subdimensions_ which split larger dimensions into more
detailed attributes (compare Figure 3 from inside to outside). Besides the
terms contained in the METRIC-framework, we found several frequently mentioned
dataset properties which we, for our purpose, want to separate from the
METRIC-framework. We summarise these additional properties under a separate
cluster called _data management_ (Figure 4). The attributes included in this
cluster ensure that a dataset is well-documented and effectively usable. In
particular, it includes the properties _documentation_ , _security_ and
_privacy_ , as well as the well-established _FAIR-Principles_ [130] requiring
data to be findable, accessible, interoperable and reusable. While these
properties play a central role in the handling of data, the METRIC-framework
is targeted at the content of a dataset. Therefore, we see the _data
management_ cluster as a prerequisite for data quality assessment by the
METRIC-framework which itself divides the concept of data quality for the
content of a dataset into five clusters: _measurement process_ , _timeliness_
, _representativeness_ , _informativeness_ , _consistency_. A summary of the
characteristics and key aspects of all five clusters is given in Table 1.

Cluster |  Description
---|---
MEeasurement process |  Concerned with technical or human influences that affect the data acquisition process. Includes _device errors_ leading to poor accuracy and precision, _human-induced errors_ such as noisy labels, _completeness_ counting the number of missing values, and _source credibility_ estimating the reliability of the data.
Timeliness |  Concerned with changes in time and whether they are appropriately reflected in the dataset. Includes _age_ concerned with the relation between creation date and the date of usage, as well as _currency_ representing the updatedness of the data.
Representativeness |  Concerned with the appropriate and comprehensive representation of the targeted population in the dataset. Includes _variety_ indicating whether a sufficiently broad range of demographics and data sources is present, _depth of data_ investigating whether the overall amount of data as well as coverage of subpopulations is sufficient, and _target class balance_ judging whether classes of the target variable are appropriately sized for use in ML.
Informativeness |  Concerned with how well the data conveys the information it describes. Includes _understandability_ indicating whether data is easily comprehended and without ambiguity, _redundancy_ investigating whether information is provided more than once, _informative missingness_ quantifying whether missing values carry additional information, and _feature importance_ estimating the value features add for the application.
Consistency |  Concerned with the consistency of presentation and composition of the dataset. Includes _rule-based consistency_ investigating whether data is presented in a consistent format that follows internal and external rules, _logical consistency_ judging whether data is logically sound and without contradictions, and _distribution consistency_ considering whether different subpopulations have similar statistical properties.
Table 1: The key characteristics of each of the five clusters of the METRIC-
framework. For more detail see the main text and the definitions of dimensions
and subdimensions in Tables S4–S9.

### Measurement process

The cluster _measurement process_ captures factors that influence uncertainty
during the data acquisition process. Two of the dimensions within this cluster
differentiate between technical errors originating from devices during
measurement (see _device error_) and errors induced by humans during,
e.g.,data handling, feature selection or data labelling (see _human-induced
error_). For the dimension _device error_ , we distinguish between the
subdimension _accuracy_ , the systematic deviation from the ground truth (also
called bias), and the subdimension _precision_ , the variance of the data
around a mean value (also called noise). In practice, a ground truth for
medical data is most often not attainable, making _accuracy_ evaluation
impossible. In that case, the level and structure of noise in the training
data should be compared to the expected noise in the data after AI deployment.
If the training data only contains low noise but the AI is utilised in
clinical practice on data with much higher noise levels, the performance of
the AI application might not be sufficient since the model did not face
suitable error characteristics during training. Therefore, lower noise data is
not necessarily better and adding noise to the training data might in some
instances even improve performance [131, 132, 133]. The errors belonging to
the dimension _human-induced errors_ are of a fundamentally different nature
and need to be treated accordingly. This type of error includes human
_carelessness_ and _outliers_ in the dataset due to (unintentional) human
mistakes. The final subdimension, _noisy labels_ , is one of the most relevant
topics in current ML research [120, 105, 107, 106]. Since in the medical
domain, supervised learning paradigms are prevalent, proper feature selection
and reliable labelling are indispensable. However, human decision making can
be highly irrational and subjective, especially in the medical context [134,
135, 136], representing one of various sources of labelling noise [137]. Among
expert annotators there is often considerable variability [137, 138]. Even in
the most common (non-medical) datasets of ML (e.g., MNIST [139], CIFAR-100
[140], Fashion-MNIST [141]) there is a significant percentage of wrong labels
[142, 143]. In contrast to the precision of instruments, noise in human
judgements is demanding to be assessed through so-called noise audits to
identify different factors, like pattern noise and occasion noise in the
medical decision process [144]. Such intra- and inter-observer variability has
always been a highly important topic in many medical disciplines, e.g., in
radiology where guidelines, training and consensus reading approaches are used
to reduce noise [145].

Another issue that frequently occurs in the data acquisition process and which
plays an important role in ML is the absence of data values with unknown
reason. We follow the ML vocabulary by capturing this quality issue with the
dimension _completeness_. It is usually measured by the ratio of missing to
total values. Apart from the mostly quantitative dimensions within the
cluster, the dimension _source credibility_ is concerned with mostly
qualitative characteristics. On the one hand, it includes the question whether
or not the measured data can be trusted based on the _expertise_ of people
involved in data measurement, processing and handling. On the other hand, the
subdimension _traceability_ evaluates whether changes from original data to
its current state are documented. Being aware of modifications such as the
exclusion of outliers or data normalisation and their chosen algorithms are
necessary for understanding the composition of the data. Finally, the
subdimension _data poisoning_ considers whether the data was intentionally
corrupted (e.g., adversarial attacks) to cause distorted outcomes. The entire
cluster _measurement process_ is crucial for data quality evaluation in the
medical field since errors may propagate through the ML model and lead to
false diagnosis or treatment of patients.

### Timeliness

Since medical knowledge and understanding is subject to constant development,
it is important to investigate the cluster _timeliness_ which indicates
whether the point in time at which the dataset is used in relation to the
point in time at which it was created and updated is appropriate for the task
at hand. Indications for diagnoses based on medical data may have changed
since a dataset was created and labelled, and changes in coding systems (such
as the transition from ICD-9 to ICD-10 or ICD-9-CM to ICD-10-CM) may affect
mortality and injury statistics [146, 147]). The _age_ of the data dictates
whether such investigations are necessary. In such cases, the labels or
standards utilised would then have to be appropriately updated to satisfy the
subdimension _currency_. Furthermore, knowledge about the subdimension _age_
might provide information about precision and accuracy of the measurement as
it gives insight into the technology used during data acquisition.

### Representativeness

Another central cluster, especially for medical applications, is
_representativeness_. Its dimensions are concerned with the extent to which
the dataset represents the targeted population (such as patients) for which
the application is intended. Whether the population of the dataset covers a
sufficient range in terms of age, sex, race or other background information is
the topic of the subdimension _variety in demographics_ contained within the
dimension _variety_. This dimension also contains the subdimension _variety of
data sources_ concerned with questions such as: Does the data originate from a
single site? Were the measurements done with devices from the same or
different manufacturers? Appropriately investigating such questions can
provide a strong indication for the applicability and generalisability of the
ML application in different environments [148, 149, 150, 151]. The dimension
_depth of data_ is one of the main topics of the ML papers in our literature
corpus. Apart from the subdimension _dataset size_ already discussed in the
previous section, this dimension also includes the subdimension _granularity_
, which considers whether the level of detail (e.g., the resolution of image
data) is sufficient for the application, as well as the subdimension
_coverage_ , which investigates whether sub-populations (e.g., specific age
groups) are still diverse by themselves (e.g., still contain all possible
diagnoses in case of classification applications). Finally, the highly-
discussed dimension _target class balance_ pays tribute to the technical
requirements of ML [117, 111, 116, 115, 105]. An algorithm must learn patterns
for specific classes from the training data. However, strong imbalances in the
class ratio could be caused by, e.g., rare diseases. In order to still be able
to properly learn corresponding patterns it may be helpful to deliberately
overrepresent rare classes in the dataset instead of matching their real world
distribution [152, 153].

### Informativeness

The cluster _informativeness_ considers the connection between the data and
the information it provides and whether the data does so in a clear, compact
and beneficial way. First of all, the _understandability_ of the data
considers whether the information of the data is easily comprehended. Second,
the dimension _redundancy_ investigates whether such information is concisely
communicated (see subdimension _conciseness_) or whether redundant information
is present such as duplicate records (see subdimension _uniqueness_). The
dimension _informative missingness_ answers the question whether the patterns
of missing values provide additional information. Che et al. [102] find an
informative pattern in the case of the MIMIC-III critical care dataset [154]
which displays a correlation between missing rates of variables and
ICD9-diagnosis labels. Missingness patterns are categorised by the literature
into either not missing at random (NMAR), missing at random (MAR) or missing
completely at random (MCAR) [155, 156]. Finally, _feature importance_ is
concerned with the overall relevance of the features for the task at hand and
moreover with the value each feature provides for the performance of a ML
application since the quantity of data has to be balanced with computational
capability. Valuable features might in many cases be as important as dataset
size [157], which is a frequently discussed topic in the data-centric AI
community [158].

### Consistency

The dimensions belonging to the cluster _consistency_ illuminate the topic of
consistent data presentation from three perspectives. _Rule-based consistency_
summarises subdimensions concerned with format (_syntactic consistency_),
which includes the fundamental and well-discussed topic of data schema [89],
and the conformity to standards and laws (_compliance_). These subdimensions
ensure that the dataset is easily processable on the one hand and comparable
and legally correct on the other. _Logical consistency_ evaluates whether or
not the content of the dataset is free of contradictions, both within the
dataset (e.g., a patient without kidneys that is diagnosed with kidney stones)
and in relationship to real world knowledge (e.g., a 200-year-old patient).
The last dimension of the cluster, _distribution consistency_ , concerns the
distributions and their statistical properties of relevant subsets of the
total dataset. While the subdimension _homogeneity_ evaluates whether subsets
have similar or different statistical properties at the same point in time
(e.g., can data from different hospitals be identified by statistics?), the
subdimension _distribution drift_ deals with varying distributions at
different time points. This subdimension can be neglected if the dataset is
not continuously changing over time, but distribution drift is sometimes
unconsciously discarded due to a lack of model surveillance. Therefore, it is
a prominent research topic [121] and the unconsciousness furthermore
underlines the importance of _distribution drift_ for medical applications
[80].

## Discussion

The METRIC-framework (Figure 3) represents a comprehensive system of data
quality dimensions for medical training data with respect to a specific task.
We stress again that these dimensions should be regarded as awareness
dimensions. They provide a guideline along which developers should familiarise
themselves with their data. Knowledge about these characteristics is helpful
for recognising the reason for the behaviour of an AI system. Understanding
this connection enables developers to improve data acquisition and selection
which may help in reducing biases, increasing robustness, facilitating
interpretability and thus has the potential to drastically improve the AI’s
trustworthiness.

With training data being the basis for almost all medical AI applications, the
assessment of its quality gains more and more attention. However, we note that
providing a division of the term data quality into data quality dimensions is
only the first step on the way to overall data quality assessment. The next
step will be to equip each data quality dimension with quantitative or
qualitative measures to describe their state. The result of this measure then
has to be evaluated with respect to the question: Is the state of the
dimension appropriate for the desired AI algorithm and its application? These
three steps (choosing a measure, obtaining a result, evaluating its
appropriateness for a task) can be applied to each dimension and subdimension.
Appropriately combining the individual outcomes can potentially serve as a
basis for a measure of the overall data quality in future work.

So far the dimensions in the METRIC-framework are not ranked in any way.
However, it is clear that some of them are more important than others.
Therefore, some dimensions deserve more attention in the assessment process or
might even be a criterion for exclusion of a dataset for a certain task. These
dimensions should be among the first to be assessed in practice. On the other
hand, some dimensions are much more difficult to measure and evaluate than
others. This can be due to their qualitative nature, the complexity of the
statistical measure, the degree of use-case dependence or the expert knowledge
that is needed for the assessment, to name a few. These considerations are of
central interest for the development of a complete data quality assessment and
examination process.

Figure 5: Categorisation of dimensions along the properties _quantitative vs.
qualitative measure_ (left) and _use case dependence for evaluating data
quality_ (right). The affiliation to a category is colour-coded. The colour
scale is presented in the inner circle.

In Figure 5, we provide insights that should be taken into consideration when
practically assessing data quality. We classify each of the 15 awareness
dimensions along two different properties. On the one hand, we estimate
whether a dimension requires mostly _quantitative_ or _qualitative_ measures.
We observe that about half of the dimensions require mostly quantitative
measures while a fifth necessitate more manual inspection by qualitative
measures (see left-hand side of Figure 5). Being able to choose quantitative
measures typically implies more objectivity and enables automation, two
desirable properties for quality assessment. Dimensions categorised as mostly
qualitatively measurable or requiring a mixture of quantitative and
qualitative input will typically require specific domain knowledge from the
medical field. Such domain knowledge can be difficult to obtain and expensive.

On the other hand, we consider whether the state of a dimension or the
evaluation of its appropriateness level is _use case dependent_ (see right-
hand side of Figure 5). This is of interest to developers as use case
dependent dimensions require not only additional knowledge, work and time
during quality assessment but also during quality improvement of data. Our
findings suggest a clear division of the wheel of data quality after
categorising all 15 dimensions. The clusters _representativeness_ and
_timeliness_ as well as the dimensions _device error_ and _feature importance_
belong to the group of use case dependent dimensions. Whether a dataset is
representative of the targeted population can only be evaluated with knowledge
of the use case. Similarly, the importance of features changes between
applications. Whether the age and currency of the data (see dimension
_timeliness_) are appropriate can also differ depending on the task. For
instance, the coding standard the data should conform to depends on the
application. The newest standards are not necessarily the best if in practice
these standards are not implemented (see section on _Timeliness_). Similarly,
reducing noise levels in the data is not necessarily better for all
applications. It rather depends on the expected noise levels of the
application (see section on _Measurement process_ for more detail).

For an overall assessment of the quality of the dataset, we estimate that on
average the dimensions of the _representativeness_ cluster together with the
dimensions _feature importance_ , _distribution consistency_ and _human-
induced error_ are crucial factors. Ignoring a single one of these dimensions
potentially has proportionally larger effects on the AI application than other
dimensions. This might also depend on type of ML problem. Actual
quantification of the effect of data quality dimensions on ML applications is
part of ongoing and future research. Nevertheless, we for now recommend
prioritising these six dimensions if it is possible to dedicate time to
evaluating or improving a dataset. With the exception of the dimension
_feature importance_ , all of the crucial dimensions are simultaneously
measured mostly quantitatively making them primary candidates for software
tools designed for improving the quality of datasets.

The importance of data quality for medical AI products is undisputed and
gaining more and more attention with on-going discussions about fairness and
trustworthiness. Parts of future regulation and certification guidelines will
not only include AI algorithms but likely also require evaluating the quality
of datasets used for their training and testing. Such inclusion of data
quality in regulation requires systematic assessment of medical datasets for
which the METRIC-framework and our additional considerations can serve as a
starting point. This has the potential to accelerate the approval of new ML
products into medical practice.

## Methods

(Supplementary Information)

### Literature review

In order to answer the research question “Along which characteristics should
data quality be evaluated when employing a dataset for trustworthy AI in
medicine?”, we conducted a systematic review following the PRISMA guidelines
[63]. The goal of such a review is to objectively collect the knowledge of a
chosen research area by summarising, condensing and expanding the ideas to
further its progress. PRISMA reviews commonly follow four main steps: (i)
Searching suitable databases with carefully formulated search strings and
extracting matching papers; (ii) screening titles and abstracts to include or
exclude papers based on predetermined criteria; (iii) extending the literature
list by screening titles and abstracts of all referenced papers from the
included papers (called “snowballing”); (iv) screening the full text of all
still included papers with respect to the eligibility criteria to build the
final literature corpus.

### Search strategy

Our research question aims at combining the knowledge from the field of
general data quality frameworks with insights about the effects that the
quality of training data has on ML applications in medicine. This should
ultimately lead to a novel framework for data quality in the context of
medical training data. Therefore, we built a search string that on the one
hand targeted papers about data quality frameworks by combining variations of
“data quality” with variations of the terms “framework” and “dimensions”. On
the other hand, we attempted to collect papers about the connection between
the quality of training data and the behaviour of a DL application by again
combining variations of the word “data quality” but this time with variations
of “machine learning”, including “artificial intelligence” and “deep learning”
(for the exact search query, see Search query). We then performed the database
search on two thematically suitable online databases: PubMed and ACM Digital
Library. All retrieved results were concatenated and duplicates removed,
yielding 1735 records.

### Search query

The following search string in pseudo code was executed on the 17th of August
2023 on PubMed and ACM Digital Library:

    (("data quality" OR "data-quality"
      OR "data qualities" OR "quality of data"
      OR  "quality of the data" OR "qualities of data"
      OR "qualities of the data"
     )
     AND
     ("dimension" OR "dimensions"
      OR "AI" OR "artificial intelligence"
      OR "ML" OR "machine learning"
      OR "deep learning"
      OR "neural network" OR "neural networks"
     )
    )
    OR
    ("data quality framework" OR "data quality frameworks"
     OR "framework of data quality" OR "framework for data quality"
    )

The chosen databases supported exact (instead of fuzzy) searches, expressed by
quotation marks around keywords. The search was applied to the title and
abstract fields of all records of the databases.

### Eligibility criteria

In Table S2, our chosen eligibility criteria that were applied to the various
screening steps are listed. Papers were included if they either provided
broad-scale data quality frameworks with general purpose or with specificity
to a medical application, or if they discussed or quantified the effects of at
least one training data quality dimension on DL behaviour. In contrast, papers
were excluded if they (i) either discussed frameworks with specificity to non-
medical fields or (ii) only considered single or few data quality dimensions
without reference to ML or (iii) focused on the quality of data management and
surveys. No limits were imposed with respect to publication date or publisher
source (i.e. peer-reviewed or not), while non-English records and inaccessible
records were omitted.

Inclusion criteria |
---|---
Criterion |  Reasoning
Study measures effects of data quality on DL applications |  Get an insight into what dimensions are being investigated by the literature
Study involves a generalised data quality framework without high specificity to non-medical field |  Incorporate earlier work on data quality aspects
Exclusion criteria |
Criterion |  Reasoning
Focus of study is not data quality |  Not related to research question
Focus of study is survey data, databases or data management |  Out of scope w.r.t. our research question; too specific topics without accounting for task-specific content
Publication language of study is not English |  /
Study is unobtainable |  /
Table S2: Eligibility criteria applied to the screening and full-text
assessment processes.

### Literature review process

Titles and abstracts from the records of the database search were screened
with respect to the eligibility criteria. This was done by two authors
independently to mitigate biases. In case of disagreement, consensus was
achieved by discussion. If necessary, a third author was consulted to arrive
at the final decision. This step reduced the number of records to 104.

The snowballing step expands the scope of the literature corpus to make it
more independent of the initially chosen databases and search string which is
important to reduce bias. For the process of snowballing, we considered all
references from the so far 104 included papers which resulted in adding 585
records to the literature list. Analogously, title and abstract screening was
performed on these new entries with the same criteria and workflow as before,
leaving 78 additional papers from snowballing.

As a final step, all 182 remaining papers were screened on the full text with
respect to the eligibility criteria. In the end, 62 entries passed all
screening steps.

For each retrieved record, the decision whether to include or exclude was
documented along with the corresponding eligibility criterion.

### Data extraction strategy

In order to introduce a comprehensive data quality framework, the 62 selected
records were each read by two authors and all terms that were deemed relevant
to describe data quality were extracted. We discarded terms if (i) their scope
is limited to a specialised data source and not transferable to a general
framework, (ii) the term refers to the quality of database infrastructure or
(iii) no definition was given and it was impossible to grasp the intended
meaning from the context. The accepted terms were copied into an Excel sheet,
which served as a starting template for the METRIC-framework. We clustered
related concepts into groups according to the terms’ definition or intended
meaning. From these small and detailed groups we formed the so-called
subdimensions, ensuring that each subdimension is mentioned by at least three
references in the literature corpus, otherwise the level of detail was deemed
too great leading to further grouping. Due to the emphasis on ML applications,
we initially also included subdimensions that cover concepts which are well-
accepted to be influential in the ML community but have not yet been the topic
of systematic quantification and therefore have fewer references. After all
further steps, this now only applies to the subdimension _variety of data
sources_.

Thorough discussion of all authors about underlying concepts and definitions
of the subdimensions resulted in hierarchically grouping these into dimensions
and the dimensions into clusters. In parallel to this grouping, all authors
reached consensus on definitions for dimensions and subdimensions of the
METRIC-framework. The definitions were adopted from a recent data quality
glossary [128] if they existed there and met our understanding of the
vocabulary in the given context of medical training data. If necessary, we
included definitions given by Wang et al. [47] in a second iteration. If none
of these two sources suggested an appropriate definition, we captured the
meaning of the desired term on the basis of the literature corpus and thus
determined its definition in the context of medical training data.

Datatype |  Year |  Author |  Field |  Measurement Process |  Timeliness |  Representativeness |  Informativeness |  Consistency
---|---|---|---|---|---|---|---|---
general data |  1996 |  Wang and Strong |  non-life-science |  accuracy (accuracy), carelessness (objectivity), source credibility (reputation), traceability (traceability), data poisoning (objectivity) |  age (timeliness) |  variety (variety of data and data sources), variety in demographics (completeness), depth of data (depth of data), coverage (completeness) |  understandability (interpretability, ease of understanding), conciseness (conciseness), feature importance (value-added (usefulness), task relevance, completeness) |  syntactic consistency (representational consistency), plausibility (believeability)
general data |  1997 |  Redman |  non-life-science |  precision (precision), traceability (identifyability) |  |  Representativeness (naturalness), granularity (granularity, format precision), coverage (comprehensiveness) |  understandability (interpretability, format appropriateness), uniqueness (minimum unnecessary redundancy), feature importance (essentialness, task relevance) |  syntactic consistency (structural consistency, representation consistency), semantic consistency (semantic consistency), homogeneity (homogeneity, robustness)
general data |  2000 |  Yoon et al. |  non-life-science |  accuracy (accuracy, correctness), completeness (completeness), source credibility (reliability) |  age (timeliness), currency (currency) |  granularity (precision, attribute granularity, detail), coverage (comprehensiveness) |  understandability (interpretability, naturalness, clarity, unambiguousness, robustness, presentation appropriateness, media, order), conciseness (conciseness, essentialness), uniqueness (minimally redundant), feature importance (relevance) |  Consistency (Consistency)
general data |  2002 |  Pipino et al. |  non-life-science |  accuracy (free-of-error), carelessness (objectivity), completeness (completeness), source credibility (reputation), data poisoning (objectivity) |  currency (timeliness) |  depth of data (depth of data) |  understandability (understandability, interpretability), conciseness (conciseness), feature importance (value-added (usefulness), task relevance) |  syntactic consistency (representational consistency), plausibility (believeability)
general data |  2003 |  Kim et al. |  non-life-science |  completeness (missing data) |  |  |  |  syntactic consistency (integrity, ambiguous data, Different representations of compound data), compliance (non-standard conforming data), semantic consistency (Inconsistency across multiple tables/file, Different data for the same entity)
general data |  2007 |  Stvilia et al. |  non-life-science |  accuracy (accuracy), precision (precision), source credibility (authority), traceability (verifyability) |  age (currency), currency (volatility) |  |  Informativeness (Informativeness), understandability (complexity), conciseness (cohesiveness), feature importance (relevance) |  syntactic consistency (intrinsic accuracy/validity, intrinsic structural consistency, intrinsic naturalness, intrinsic semantic consistency), compliance (relational semantic consistency, relational structural consistency), semantic consistency (accuracy/validity, relational accuracy)
general data |  2008 |  International Organization for Standardization and International Electrotechnical Commission |  non-life-science |  completeness (completeness), source credibility (source credibility, authenticity), traceability (traceability) |  age (currency) |  granularity (precision (no. of digits)) |  understandability (understandability), conciseness (efficiency) |  Consistency (Consistency), syntactic consistency (syntactic accuracy), compliance (compliance), semantic consistency (semantic accuracy)
general data |  2010 |  Chan et al. |  life science |  accuracy (accuracy), completeness (completeness) |  currency (timeliness) |  granularity (granularity, clinical specificity) |  |  semantic consistency (comparability)
general data |  2011 |  Loshin |  non-life-science |  accuracy (value accuracy), precision (precision), completeness (Null validation), source credibility (authoritative sources), traceability (originating data source) |  age (age), currency (correction/update promulgation) |  coverage (coverage, population density optionality) |  uniqueness (Entity uniqueness) |  syntactic consistency (syntactic consistency, Name ambiguity), compliance (standards and policies), semantic consistency (semantic consistency, multi-value consistency), plausibility (temporal consistency, reasonableness), distribution drift (temporal reasonability)
general data |  2012 |  Nahm |  life science |  accuracy (accuracy), human-induced error (human-induced error), completeness (completeness) |  timeliness (timeliness), currency (currency, volatility) |  granularity (granularity, precision (no. of digits)) |  understandability (readability/legibility), feature importance (relevance) |
general data |  2012 |  Sidi et al. |  non-life-science |  accuracy (accuracy, free-of-error, reliability), carelessness (objectivity), completeness (completeness), source credibility (reputation), data poisoning (objectivity) |  age (freshness), currency (freshness) |  variety in demographics (completeness), depth of data (depth of data), coverage (coverage, completeness) |  understandability (understandability, interpretability), conciseness (conciseness), uniqueness (duplication), feature importance (value-added (usefulness), Usability, completeness, relevance) |  syntactic consistency (representational consistency, consistency, data integrity fundamentals), semantic consistency (consistency, consistency and synchronization), plausibility (believeability), distribution drift (data decay)
general data |  2012 |  Sebastian-Coleman |  non-life-science |  completeness (completeness) |  timeliness (timeliness) |  |  |  syntactic consistency (validity), distribution drift ((temporal) consistency)
general data |  2013 |  DAMA UK Working Group on Quality Dimensions |  non-life-science |  accuracy (accuracy), completeness (completeness) |  age (timeliness) |  |  uniqueness (uniqueness) |  Consistency (Consistency), syntactic consistency (validity, consistency)
general data |  2013 |  Weiskopf and Weng |  life science |  accuracy (correctness), carelessness (correctness), completeness (completeness) |  currency (currency) |  |  |  semantic consistency (concordance), plausibility (plausibility)

Datatype |  Year |  Author |  Field |  Measurement Process |  Timeliness |  Representativeness |  Informativeness |  Consistency
---|---|---|---|---|---|---|---|---
general data |  2014 |  Chen et al. |  life science |  accuracy (accuracy, reflecting actual sample, reliability), precision (precision), carelessness (objectivity, reflecting actual sample, reliability, under-reporting, illegible handwriting, errors in report form / errors from data entry / calculation errors), completeness (completeness, completeness), expertise (repeatability), data poisoning (objectivity) |  timeliness (timeliness), age (periodicity), currency (currency, up-datedness) |  Representativeness (Representativeness), granularity (granularity) |  understandability (ease of understanding), feature importance (Usability, importance, relevance) |  Consistency (Consistency), syntactic consistency (validity, internal consistency, external consistency, non-standardization of vocabulary), compliance (meeting/using data standards), semantic consistency (concordance, comparability)
general data |  2015 |  Johnson et al. |  life science |  Measurement Process (Reliability), accuracy (RelativeCorrectness, RepresentationCorrectness, RelativeCompleteness), carelessness (RepresentationIntegrity), completeness (RepresentationCorrectness, RepresentationComplete, RelativeCompleteness) |  currency (RepresentationCurrent, DatasetCurrent, TaskCurrency) |  variety in demographics (sufficiency), coverage (DomainCoverage, TaskCoverage) |  informative missingness (DomainComplete), feature importance (relevance) |  syntactic consistency (RepresentationConsistency), compliance (CodingConsistency), semantic consistency (DomainConsistency)
general data |  2016 |  Kahn et al. |  life science |  completeness (completeness) |  |  |  uniqueness (uniqueness) |  syntactic consistency (value conformance, relational conformance, computational conformance), plausibility (plausibility)
general data |  2018 |  Che et al. |  life science |  |  |  |  informative missingness (informative missingness) |
general data |  2019 |  Bloland and MacNeil |  life science |  accuracy (Trueness), carelessness (integrity), completeness (completeness), data poisoning (integrity) |  age (timeliness) |  coverage (completeness) |  conciseness (efficiency), feature importance (relevance) |  syntactic consistency (consistency), semantic consistency (concurrence)
general data |  2019 |  Vanbrabant et al. |  life science |  carelessness (Inexactness of timestamps), completeness (missing values) |  |  granularity (imprecise data), coverage (missing entities) |  |  syntactic consistency (typing mistake, outside domain range, inconsistent formatting, abbreviations), semantic consistency (violation of mutual dependency)
general data |  2020 |  Bian et al. |  life science |  accuracy (accuracy) |  currency (currency) |  |  uniqueness (uniqueness) |  syntactic consistency (value conformance, relational conformance), semantic consistency (concordance, comparability), plausibility (plausibility), distribution drift (temporal plausbility)
general data |  2021 |  Schmidt et al. |  life science |  accuracy (accuracy, disagreement with gold standard), carelessness (inter-class/intra-class reliability), outliers (outliers, unexpected distributions, unexpected associations, unexpected data elements, unexpected data records), completeness (completeness, completeness, crude missingness), expertise (disagreement of repeated mesurements) |  |  |  understandability (readability/legibility), uniqueness (duplication), informative missingness (qualified missingness, non response rate, refusal rate, dropout rate, missing due to specified reason) |  Consistency (Consistency), syntactic consistency (structural data set error, value format error, relational data set error, range and value violations, non-standardization of vocabulary, data type mismatch, inhomogeneous value formats, inadmissible precision), compliance (integrity), semantic consistency (semantic consistency, accuracy/validity, contradictions, data record mismatch, data element mismatch)
general data |  2023 |  Syed et al. |  life science |  accuracy (accuracy, correctness, veracity), noisy labels (accurate diagnostic data), carelessness (integrity), completeness (missing data, fragmentation), data poisoning (integrity) |  age (timeliness) |  Representativeness (Representativeness), granularity (granularity), coverage (completeness) |  uniqueness (uniqueness), feature importance (completeness, relevance) |  syntactic consistency (representational consistency, validity, structuredness), compliance (integrity, standardization,), semantic consistency (semantic consistency, concordance), plausibility (plausibility), homogeneity (data variability), distribution drift (temporal variability)
general data |  2023 |  Mashoufi et al. |  life science |  accuracy (accuracy, correctness, validity), noisy labels (data entry error), carelessness (objectivity), completeness (completeness), source credibility (source credibility), data poisoning (objectivity) |  age (currency), currency (timeliness, temporal consistency) |  granularity (granularity) |  feature importance (relevancy) |  Consistency (Consistency), syntactic consistency (conformance), semantic consistency (semantic plausibility)

Datatype |  Year |  Author |  Field |  Measurement Process |  Timeliness |  Representativeness |  Informativeness |  Consistency
---|---|---|---|---|---|---|---|---
general data |  2023 |  Lewis et al. |  life science |  accuracy (correctness), carelessness (correctness), completeness (completeness) |  currency (currency) |  |  informative missingness (bias) |  semantic consistency (concordance), plausibility (plausibility)
general data |  2023 |  Liu et al. |  life science |  accuracy (RelativeCorrectness), carelessness (correctness, misunderstanding of what an entry means, errors in report form / errors from data entry / calculation errors) |  age (timeliness), currency (currency) |  dataset size (completeness), coverage (completeness (coverage of baseline features or data required for a particular disease)) |  understandability (usability, ease of understanding), uniqueness (duplication), feature importance (relevance) |  syntactic consistency (syntactic consistency, consistency through integrity constraints), distribution drift ((temporal) consistency)
general data |  2023 |  Xu et al. |  non-life-science |  noisy labels (noisy labels), source credibility (reliability) |  |  dataset size (dataset size) |  |
general data |  2023 |  Tahar et al. |  life science |  completeness (completeness, value completeness, item completeness) |  |  |  conciseness (semantic uniqueness), uniqueness (syntactic uniqueness) |  syntactic consistency (range plausibility), semantic consistency (semantic plausibility, concordance), plausibility (plausibility)
big data |  2015 |  Batini et al. |  non-life-science |  accuracy (accuracy, semantic accuracy, source accuracy, reliability), precision (precision, accuracy deviation), noisy labels (correctness of classification), completeness (completeness, value completeness, tuple completeness, left censored, right censored, sparsity), source credibility (reputation, trustworthiness), expertise (believeability), traceability (verifyability) |  age (timeliness), currency (currency, temporal validity, temporal consistency) |  Representativeness (Representativeness) |  understandability (understandability, readability/legibility), conciseness (conciseness, closer-to-text base comprehension, Closer to situation model level comprehension, representation conc.), uniqueness (spatial redundancy, temporal redudancy), feature importance (significance, pertinence, relevance, selective) |  Consistency (Consistency), syntactic consistency (syntactic accuracy, consistency through integrity constraints, consistency through edits, format consistency, domain consistency, lexical consistency, numerical consistency, range frequency, change frequency), logical consistency (logical consistency), semantic consistency (relative error consistency, coherence), plausibility (temporal consistency)
big data |  2015 |  Cai and Zhu |  non-life-science |  accuracy (accuracy), source credibility (source credibility) |  age (timeliness), currency (timeliness) |  coverage (completeness) |  understandability (readability), feature importance (fitness) |  syntactic consistency (consistency, integrity), semantic consistency (consistency, integrity)
big data |  2016 |  Gao et al. |  non-life-science |  accuracy (accuracy, correctness), carelessness (correctness), completeness (completeness), source credibility (accountability) |  timeliness (timeliness), currency (currency) |  |  feature importance (Usability) |  syntactic consistency (format consistency)
big data |  2017 |  Gudivada et al. |  non-life-science |  carelessness (gender bias), completeness (missing at random (MAR), missing completely at random (MCAR)), data poisoning (gender bias) |  currency (currency) |  variety in demographics (heterogeneity), variety of data sources (heterogeneity) |  conciseness (dimensionality reduction), uniqueness (duplication, consistency), informative missingness (missing not at random (MNAR)), feature importance (feature selection, feature extraction) |  syntactic consistency (syntactic accuracy), compliance (specifications, integrity), semantic consistency (semantic accuracy)
big data |  2018 |  Juddoo et al. |  life science |  accuracy (accuracy), carelessness (reliability), completeness (completeness), expertise (confidence) |  |  |  feature importance (usefulness) |  syntactic consistency (consistency), semantic consistency (consistency), plausibility (validity)
big data |  2020 |  Eder and Shekhovtsov |  life science |  accuracy (accuracy), completeness (completeness), source credibility (Reliability) |  age (timeliness) |  granularity (precision (no. of digits)) |  feature importance (reliability) |  compliance (compliance), semantic consistency (semantic consistency)
big data |  2020 |  Ramasamy and Chowdhury |  non-life-science |  source credibility (source credibility, trustworthiness), traceability (Pedigree/Lineage) |  |  dataset size (volume) |  redundancy (redundancy) |  semantic consistency (cohesion)
machine learning |  1993 |  Bansal et al. |  non-life-science |  carelessness (typing error (by white noise), measurement of subjective data (e.g., about consumer sentiment) -¿ as white noise, inaccurate estimations (e.g., from forecasts)) |  |  |  |
machine learning |  2004 |  Michel et al. |  life science |  |  |  dataset size (quantity), coverage (completeness) |  |

Datatype |  Year |  Author |  Field |  Measurement Process |  Timeliness |  Representativeness |  Informativeness |  Consistency
---|---|---|---|---|---|---|---|---
machine learning |  2011 |  Blake and Mangiameli |  non-life-science |  accuracy (accuracy), completeness (completeness) |  age (timeliness) |  |  |  syntactic consistency (consistency)
machine learning |  2013 |  Twala |  non-life-science |  precision (noisy attributes), noisy labels (noisy targets), carelessness (noisy attributes), outliers (outliers) |  |  |  |
machine learning |  2015 |  Sukhbaatar et al. |  non-life-science |  noisy labels (noisy labels, label flips, outliers) |  |  |  |
machine learning |  2015 |  Masko and Hensman |  non-life-science |  |  |  target class balance (imbalance) |  |
machine learning |  2016 |  Dodge and Karam |  non-life-science |  accuracy (compression/noise/blur/contrast) |  |  |  |
machine learning |  2016 |  Karahan et al. |  non-life-science |  accuracy (contrast, noise, blur, occlusion, and color degredation) |  |  |  |
machine learning |  2017 |  Sun et al. |  non-life-science |  noisy labels (noisy labels) |  |  dataset size (dataset size), coverage (number of classes) |  |
machine learning |  2018 |  Rolnick et al. |  non-life-science |  noisy labels (noisy labels, uniform label noise, structured label noise) |  |  dataset size (dataset size) |  |
machine learning |  2018 |  Buda et al. |  non-life-science |  |  |  target class balance (target class balance) |  |
machine learning |  2019 |  He et al. |  non-life-science |  noisy labels (quality of label), data poisoning (dataset contamination) |  |  dataset size (dataset size), target class balance (dataset equilibrium) |  |
machine learning |  2019 |  Ovadia et al. |  non-life-science |  |  |  |  |  distribution drift (distribution drift)
machine learning |  2020 |  Cao et al. |  non-life-science |  |  |  |  uniqueness (rank) |
machine learning |  2021 |  Hong and Shen |  life science |  accuracy (adding white noise) |  |  dataset size (dataset size) |  |
machine learning |  2021 |  Benedick et al. |  non-life-science |  accuracy (swapping perturbation), completeness (dropping perturbation) |  |  |  |
machine learning |  2021 |  Wesemeyer et al. |  life science |  noisy labels (annotation quality) |  |  dataset size (quantity) |  |
machine learning |  2021 |  Qi et al. |  non-life-science |  completeness (missing data) |  |  |  |  semantic consistency (violations of functional dependencies, conflicting data)
machine learning |  2021 |  Li and Chao |  non-life-science |  |  |  dataset size (quantity) |  feature importance (information value) |
machine learning |  2021 |  Li et al. |  non-life-science |  |  |  target class balance (target class balance) |  redundancy (redundancy), feature importance (informative sample) |
machine learning |  2022 |  Fan and Shi |  life science |  accuracy (adding random errors) |  |  dataset size (quantity) |  |
machine learning |  2022 |  Jouseau et al. |  non-life-science |  outliers (outliers), completeness (missing values) |  |  |  uniqueness (exact and partial dublicates), informative missingness (missing values) |  syntactic consistency (domain value violations)
machine learning |  2022 |  Derry et al. |  life science |  |  |  dataset size (more training data), granularity (downsampling) |  feature importance (use of different experimental methods) |

Datatype |  Year |  Author |  Field |  Measurement Process |  Timeliness |  Representativeness |  Informativeness |  Consistency
---|---|---|---|---|---|---|---|---
machine learning |  2023 |  Ranjan et al. |  non-life-science |  accuracy (lighting condition), precision (augmentation (mixup/brightness/exposure/saturation/noise/blur/cutout/mosaic)) |  |  dataset size (dataset size, augmentation) |  |
machine learning |  2023 |  Whang et al. |  non-life-science |  precision (augmentation (mixup/brightness/exposure/saturation/noise/blur/cutout/mosaic), noisy features (image blur/color noise)), noisy labels (noisy labels), carelessness (racial bias, fairness), completeness (missing values, missing labels), data poisoning (data poisoning, label flipping, racial bias, fairness) |  |  |  informative missingness (informative missingness) |
machine learning |  2023 |  Shimizu and Wakabayashi |  life science |  noisy labels (label redudancy) |  |  dataset size (budget for crowdsourcing) |  |
machine learning |  2023 |  Pan et al. |  non-life-science |  |  |  target class balance (target class balance) |  |  homogeneity (interclass ambiguity, interclass ambiguity, out of distribution)
Table S3: List of all publications in the literature corpus including datatype
(general data, big data or ML data), publication year, authors, and field
(life-science vs. not-life-science). Additionally, we list the mentioned data
quality dimensions and subdimensions per paper, categorised into the clusters
of the METRIC-framework (Measurement Process, Timeliness, Representativeness,
Informativeness, Consistency). The originally mentioned data quality terms
(displayed in brackets) are mapped to the corresponding dimension or
subdimension of the METRIC-framework and displayed in the form: “METRIC-
dimension (terms used in publication)”, e.g., “dataset size (quantity)”.

### Definitions of data quality dimensions

Measurement Process [83] |  |  |
---|---|---|---

Dimension |  Subdimension |  Definition |  References
---|---|---|---
device error |  |  The extent to which data values originating from a sensor are accurate and precise. |
2-4[0.5pt/2.5pt] |  accuracy |  The extent of closeness of data values to real values. (From [128]) |  [91, 84, 83, 89, 92, 77, 88, 86, 66, 76, 112, 109, 118, 73, 85, 78, 68, 114, 90, 65, 67, 104, 74, 119, 47, 80, 87, 103, 95, 70, 75]
2-4[0.5pt/2.5pt] |  precision |  The extent to which the error in data values spreads around zero (in statistics). (From [128]) |  [89, 120, 77, 64, 70, 65, 112, 99]
1-4 human-induced error |  |  The extent to which manual inputs unintentionally influence the data in a wrong way. |  [76]
2-4[0.5pt/2.5pt] |  noisy labels |  The extent to which labels provided by humans are accurate and precise. |  [107, 84, 120, 89, 108, 106, 110, 86, 123, 122, 99, 105]
2-4[0.5pt/2.5pt] |  carelessness |  The extent to which negligence or unintended human errors cause faulty data. |  [83, 84, 120, 92, 86, 77, 88, 79, 85, 78, 68, 67, 98, 47, 87, 94, 95, 99, 75]
2-4[0.5pt/2.5pt] |  outliers |  The extent to which data values exist that do not match the expected distribution. |  [101, 88, 99]
1-4 completeness |  |  The extent to which data values are present. (From [128]) |  [96, 83, 84, 89, 120, 92, 77, 88, 86, 76, 66, 79, 101, 69, 72, 73, 71, 78, 68, 90, 65, 67, 104, 74, 87, 94, 103, 95, 81, 82, 75]
1-4 source credibility |  |  The extent to which data values are regarded as true and believable in terms of their source. (Based on [128]) |  [122, 67, 91, 72, 89, 92, 93, 86, 68, 90, 70, 66, 65, 47]
2-4[0.5pt/2.5pt] |  expertise |  The extent of knowledge and experience with which the measurement was performed and the data was processed. |  [77, 89, 88, 95]
2-4[0.5pt/2.5pt] |  traceability |  The extent to which data lineage is available. (From [128]) |  [72, 89, 93, 64, 70, 65, 47]
2-4[0.5pt/2.5pt] |  data poisoning |  The extent to which data values are intentionally falsified. |  [84, 47, 94, 120, 78, 68, 77, 86, 67, 105]

Table S4: Details for the cluster Measurement Process of the METRIC-framework.
The hierarchical order of all dimensions and subdimensions included in the
cluster is displayed in the first two columns. The third column provides a
definition for each of these terms. The fourth column provides all references
of the literature corpus connected to the corresponding term. A provided
reference indicates that the dimension, subdimension or a different term with
the same intended meaning (see Data extraction strategy for clarification) is
discussed in that cited article. References on the cluster level indicate that
the term is used as a broader concept.

Timeliness |  |  |
---|---|---|---

Dimension |  Subdimension |  Definition |  References
---|---|---|---
timeliness |  |  The extent to which the age and the currency of data values are appropriate for the task at hand. (Based on [47]) |  [92, 77, 69, 76]
2-4[0.5pt/2.5pt] |  age |  The extent to which the age of data values is appropriate for the task at hand. (Based on [47]) |  [104, 67, 91, 72, 84, 89, 85, 73, 78, 86, 77, 90, 70, 66, 65, 47]
2-4[0.5pt/2.5pt] |  currency |  The extent to which data values are up to date. (From [128]) |  [66, 87, 74, 91, 83, 94, 89, 92, 85, 77, 68, 86, 70, 76, 65, 67, 80, 75]

Table S5: Details for the cluster Timeliness of the METRIC-framework. The
hierarchical order of all dimensions and subdimensions included in the cluster
is displayed in the first two columns. The third column provides a definition
for each of these terms. The fourth column provides all references of the
literature corpus connected to the corresponding term. A provided reference
indicates that the dimension, subdimension or a different term with the same
intended meaning (see Data extraction strategy for clarification) is discussed
in that cited article. References on the cluster level indicate that the term
is used as a broader concept.

Representativeness [77, 89, 64, 84] |  |  |
---|---|---|---

Dimension |  Subdimension |  Definition |  References
---|---|---|---
variety |  |  The extent to which the dataset is diverse. |  [47]
2-4[0.5pt/2.5pt] |  variety in demographics |  The extent to which measured subjects are diverse. |  [94, 47, 67, 83]
2-4[0.5pt/2.5pt] |  variety of data sources |  The extent to which data are available from different sources. (Based on [128]) |  [94]
1-4 depth of data |  |  The extent to which the volume of the dataset is appropriate. (Based on [47]) |  [67, 47, 68]
2-4[0.5pt/2.5pt] |  dataset size |  The extent to which the quantity of records is appropriate. |  [100, 109, 124, 108, 106, 85, 93, 110, 114, 113, 123, 122, 112, 105]
2-4[0.5pt/2.5pt] |  granularity |  The extent to which the level of detail of the dataset and data values is appropriate. |  [66, 74, 72, 84, 124, 77, 86, 64, 90, 76, 79]
2-4[0.5pt/2.5pt] |  coverage |  The extent to which relevant subsets of the dataset satisfy the dimensions variety and target class balance. |  [91, 84, 47, 83, 85, 78, 110, 64, 66, 65, 67, 79, 100]
1-4 target class balance |  |  The extent to which the classes of a target variable have similar size. |  [117, 111, 116, 115, 105]

Table S6: Details for the cluster Representativeness of the METRIC-framework.
The hierarchical order of all dimensions and subdimensions included in the
cluster is displayed in the first two columns. The third column provides a
definition for each of these terms. The fourth column provides all references
of the literature corpus connected to the corresponding term. A provided
reference indicates that the dimension, subdimension or a different term with
the same intended meaning (see Data extraction strategy for clarification) is
discussed in that cited article. References on the cluster level indicate that
the term is used as a broader concept.

Informativeness [70] |  |  |
---|---|---|---

Dimension |  Subdimension |  Definition |  References
---|---|---|---
understandability |  |  The extent to which data values are clear without ambiguity and easily comprehended. (From [47]) |  [72, 91, 47, 76, 89, 85, 68, 77, 64, 88, 66, 70, 67]
1-4 redundancy |  |  The extent to which logically identical data are stored more than once. (From [128]) |  [93, 111]
2-4[0.5pt/2.5pt] |  conciseness |  The extent to which data values, attributes and records are compactly represented. (Based on [47]) |  [72, 47, 94, 89, 78, 68, 70, 66, 67, 81]
2-4[0.5pt/2.5pt] |  uniqueness |  The extent to which records occur only once in a data file. (From [128]) |  [101, 67, 84, 94, 89, 97, 73, 85, 88, 64, 66, 65, 82, 80, 81]
1-4 informative missingness |  |  The extent to which missing data values provide useful information. |  [102, 87, 101, 83, 94, 120, 88]
1-4 feature importance |  |  The extent to which the attributes are beneficial and provide advantages from their use. (Based on usefulness from [47]) |  [91, 84, 83, 89, 111, 92, 77, 86, 66, 76, 85, 78, 68, 64, 90, 67, 47, 94, 124, 95, 113, 70]

Table S7: Details for the cluster Informativeness of the METRIC-framework. The
hierarchical order of all dimensions and subdimensions included in the cluster
is displayed in the first two columns. The third column provides a definition
for each of these terms. The fourth column provides all references of the
literature corpus connected to the corresponding term. A provided reference
indicates that the dimension, subdimension or a different term with the same
intended meaning (see Data extraction strategy for clarification) is discussed
in that cited article. References on the cluster level indicate that the term
is used as a broader concept.

Consistency [72, 89, 73, 77, 88, 86, 66] |  |  |
---|---|---|---

Dimension |  Subdimension |  Definition |  References
---|---|---|---
rule-based consistency |  |  The extent to which data values comply with a rule. (Validity from [128]) |
2-4[0.5pt/2.5pt] |  syntactic consistency |  The extent to which data values comply to a dictionary and are always presented in the same format. (Based on [47]) |  [91, 84, 83, 89, 92, 88, 77, 86, 79, 101, 69, 72, 85, 73, 78, 68, 71, 64, 65, 67, 104, 47, 80, 94, 95, 70, 81, 82]
2-4[0.5pt/2.5pt] |  compliance |  The extent to which the dataset and data values are in accordance with laws, regulations or standards. (From [128]) |  [72, 84, 94, 83, 71, 77, 88, 90, 70, 65]
1-4 logical consistency |  |  The extent to which data values are plausible and records are semantically consistent. |  [89]
2-4[0.5pt/2.5pt] |  semantic consistency |  The extent to which records are free of contradictions. |  [96, 91, 84, 83, 89, 88, 86, 77, 79, 72, 93, 78, 71, 64, 90, 65, 67, 74, 80, 87, 94, 95, 70, 81, 75]
2-4[0.5pt/2.5pt] |  plausibility |  The extent to which data values match knowledge of the real world. (From [128]) |  [87, 67, 84, 47, 89, 95, 68, 81, 65, 82, 80, 75]
1-4 distribution consistency |  |  The extent to which distributions are stable. |
2-4[0.5pt/2.5pt] |  homogeneity |  The extent to which distributions are stable among relevant subsets. |  [115, 64, 84]
2-4[0.5pt/2.5pt] |  distribution drift |  The extent to which new data matches the distribution of existing data. |  [69, 84, 85, 121, 65, 67, 80]

Table S8: Details for the cluster Consistency of the METRIC-framework. The
hierarchical order of all dimensions and subdimensions included in the cluster
is displayed in the first two columns. The third column provides a definition
for each of these terms. The fourth column provides all references of the
literature corpus connected to the corresponding term. A provided reference
indicates that the dimension, subdimension or a different term with the same
intended meaning (see Data extraction strategy for clarification) is discussed
in that cited article. References on the cluster level indicate that the term
is used as a broader concept.

Data Management |  |  |
---|---|---|---

Dimension |  Subdimension |  Definition |  References
---|---|---|---
documentation |  |  Metadata about the dataset. |  [91, 84, 94, 83, 89, 90, 64, 65, 67]
1-4 security |  |  A characteristic that ensures that unauthorized changes to the dataset are prohibited. |  [67, 89, 92, 68, 77, 70, 47]
1-4 privacy |  |  A characteristic that ensures that the dataset is recorded and stored in a way that protects personal data. |  [72, 94, 92, 93, 77, 65]
1-4 FAIR principles |  |  The FAIR Principles [130] require a dataset to be findable, accessible, interoperable and reusable. |  [91, 84, 89, 92, 77, 86, 66, 76, 72, 85, 93, 68, 64, 65, 67, 47, 94, 95, 70]

Table S9: Details for Data Management as a prerequisite for the METRIC-
framework. All dimensions included under data management are displayed in the
first column. The third column provides a definition for each of these terms.
The fourth column provides all references of the literature corpus connected
to the corresponding term. A provided reference indicates that the dimension
or a different term with the same intended meaning (see Data extraction
strategy for clarification) is discussed in that cited article.

## References

  *   * Vaswani et al. [2017] Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. Advances in neural information processing systems 30 (2017) https://doi.org/10.48550/arXiv.1706.03762
  * Deng [2018] Deng, L.: Artificial intelligence in the rising wave of deep learning: The historical path and future outlook. IEEE Signal Processing Magazine 35(1), 180–177 (2018) https://doi.org/10.1109/MSP.2017.2762725
  * Silver et al. [2017] Silver, D., Schrittwieser, J., Simonyan, K., Antonoglou, I., Huang, A., Guez, A., Hubert, T., Baker, L., Lai, M., Bolton, A., Chen, Y., Lillicrap, T., Hui, F., Sifre, L., Driessche, G., Graepel, T., Hassabis, D.: Mastering the game of Go without human knowledge. nature 550(7676), 354–359 (2017) https://doi.org/10.1038/nature24270
  * He et al. [2016] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 770–778 (2016). https://doi.org/10.48550/arXiv.1512.03385
  * Redmon et al. [2016] Redmon, J., Divvala, S., Girshick, R., Farhadi, A.: You only look once: Unified, real-time object detection. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 779–788 (2016). https://doi.org/10.48550/arXiv.1512.03385
  * OpenAI [2023] OpenAI: GPT-4 technical report. ArXiv abs/2303.08774 (2023) https://doi.org/10.48550/arXiv.2303.08774
  * Rombach et al. [2022] Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10684–10695 (2022). https://doi.org/10.48550/arXiv.2112.10752
  * Chui et al. [2023] Chui, M., Yee, L., Hall, B., Singla, A.: The state of AI in 2023: Generative AI’s breakout year. McKinsey & Company (2023). https://www.mckinsey.com/capabilities/quantumblack/our-insights/the-state-of-ai-in-2023-generative-ais-breakout-year
  * Esteva et al. [2017] Esteva, A., Kuprel, B., Novoa, R.A., Ko, J., Swetter, S.M., Blau, H.M., Thrun, S.: Dermatologist-level classification of skin cancer with deep neural networks. nature 542(7639), 115–118 (2017) https://doi.org/10.1038/nature21056
  * Jumper et al. [2021] Jumper, J., Evans, R., Pritzel, A., Green, T., Figurnov, M., Ronneberger, O., Tunyasuvunakool, K., Bates, R., Žídek, A., Potapenko, A., Bridgland, A., Meyer, C., Kohl, S.A.A., Ballard, A.J., Cowie, A., Romera-Paredes, B., Nikolov, S., Jain, R., Adler, J., Back, T., Petersen, S., Reiman, D., Clancy, E., Zielinski, M., Steinegger, M., Pacholska, M., Berghammer, T., Bodenstein, S., Silver, D., Vinyals, O., Senior, A.W., Kavukcuoglu, K., Kohli, P., Hassabis, D.: Highly accurate protein structure prediction with AlphaFold. Nature 596(7873), 583–589 (2021) https://doi.org/10.1038/s41586-021-03819-2
  * Teoh and Kidd [2017] Teoh, E.R., Kidd, D.G.: Rage against the machine? Google’s self-driving cars versus human drivers. Journal of safety research 63, 57–60 (2017) https://doi.org/10.1016/j.jsr.2017.08.008
  * von Eschenbach [2021] Eschenbach, W.J.: Transparency and the black box problem: Why we do not trust AI. Philosophy and Technology 34 (2021) https://doi.org/10.1007/s13347-021-00477-0
  * Chair’s Summary of the AI Safety Summit 2023 [2023] AI Safety Summit 2023, B.P.: (2023). https://www.gov.uk/government/publications/ai-safety-summit-2023-chairs-statement-2-november
  * Council of the European Union and European Parliament [2021] Council of the European Union and European Parliament: Proposal for a regulation of the European Parliament and of the Council laying down harmonised rules on artificial intelligence (Artificial Intelligende Act) and amending certain Union legislative acts. https://artificialintelligenceact.eu/ (2021). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52021PC0206
  * Food and Drug Administration et al. [2019] Food and Drug Administration, et al.: Proposed regulatory framework for modifications to artificial intelligence/machine learning (AI/ML)-based software as a medical device (SaMD). Department of Health and Human Services (United States) (2019). https://www.fda.gov/files/medical%20devices/published/US-FDA-Artificial-Intelligence-and-Machine-Learning-Discussion-Paper.pdf
  * Muehlematter et al. [2021] Muehlematter, U.J., Daniore, P., Vokinger, K.N.: Approval of artificial intelligence and machine learning-based medical devices in the USA and Europe (2015–20): a comparative analysis. The Lancet Digital Health 3(3), 195–203 (2021) https://doi.org/10.1016/S2589-7500(20)30292-2
  * Zhu et al. [2022] Zhu, S., Gilbert, M., Chetty, I., Siddiqui, F.: The 2021 landscape of FDA-approved artificial intelligence/machine learning-enabled medical devices: an analysis of the characteristics and intended use. International journal of medical informatics 165, 104828 (2022) https://doi.org/10.1016/j.ijmedinf.2022.104828
  * Gulshan et al. [2016] Gulshan, V., Peng, L., Coram, M., Stumpe, M.C., Wu, D., Narayanaswamy, A., Venugopalan, S., Widner, K., Madams, T., Cuadros, J., Kim, R., Raman, R., Nelson, P.C., Mega, J.L., Webster, D.R.: Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. jama 316(22), 2402–2410 (2016) https://doi.org/10.1001/jama.2016.17216
  * Ardila et al. [2019] Ardila, D., Kiraly, A.P., Bharadwaj, S., Choi, B., Reicher, J.J., Peng, L., Tse, D., Etemadi, M., Ye, W., Corrado, G., Naidich, D.P., Shetty, S.: End-to-end lung cancer screening with three-dimensional deep learning on low-dose chest computed tomography. Nature medicine 25(6), 954–961 (2019) https://doi.org/10.1038/s41591-019-0447-x
  * Liu et al. [2019] Liu, X., Faes, L., Kale, A.U., Wagner, S.K., Fu, D.J., Bruynseels, A., Mahendiran, T., Moraes, G., Shamdas, M., Kern, C., Ledsam, J.R., Schmid, M.K., Balaskas, K., Topol, E.J., Bachmann, L.M., Keane, P.A., Denniston, A.K.: A comparison of deep learning performance against health-care professionals in detecting diseases from medical imaging: a systematic review and meta-analysis. The Lancet Digital Health 1(6), 271–297 (2019) https://doi.org/10.1016/s2589-7500(19)30123-2
  * Ronneberger et al. [2015] Ronneberger, O., Fischer, P., Brox, T.: U-net: Convolutional networks for biomedical image segmentation. In: Medical Image Computing and Computer-Assisted Intervention–MICCAI 2015: 18th International Conference, Munich, Germany, October 5-9, 2015, Proceedings, Part III 18, pp. 234–241 (2015). https://doi.org/10.48550/arXiv.1505.04597 . Springer
  * Chen et al. [2021] Chen, J., Lu, Y., Yu, Q., Luo, X., Adeli, E., Wang, Y., Lu, L., Yuille, A.L., Zhou, Y.: Transunet: Transformers make strong encoders for medical image segmentation. arXiv preprint arXiv:2102.04306 (2021) https://doi.org/10.48550/arXiv.2102.04306
  * Hatamizadeh et al. [2022] Hatamizadeh, A., Nath, V., Tang, Y., Yang, D., Roth, H.R., Xu, D.: Swin UNETR: Swin transformers for semantic segmentation of brain tumors in MRI images. ArXiv abs/2201.01266 (2022) https://doi.org/10.48550/arXiv.2201.01266
  * AR [1988] AR, F.: Scientific standards in epidemiologic studies of the menace of daily life. Science (1988) https://doi.org/10.1126/science.3057627
  * WHO Technical Report Series [2021] WHO Technical Report Series, n..: Annex 4 - Guideline on data integrity (2021). https://www.gmp-navigator.com/files/guidemgr/trs1033-annex4-guideline-on-data-integrity.pdf
  * International Council For Harmonisation Of Technical Requirements For Pharmaceuticals For Human Use (ICH) [2016] International Council For Harmonisation Of Technical Requirements For Pharmaceuticals For Human Use (ICH): Integrated Addendum To ICH E6(R1): Guideline For Good Clinical Practice. https://www.slideshare.net/ICRInstituteForClini/integrated-addendum-to-ich-e6r1-guideline-for-good-clinical-practice-e6r2 (2016). https://www.ich.org/page/efficacy-guidelines
  * [27] Directive 2004/9/EC of the European Parliament and of the Council of 11 February 2004 on the inspection and verification of good laboratory practice (GLP). https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX(2004). https://single-market-economy.ec.europa.eu/sectors/chemicals/good-laboratory-practice_en
  * [28] EudraLex - Volume 4 - Good Manufacturing Practice (GMP) guidelines. https://health.ec.europa.eu/medicinal-products/eudralex/eudralex-volume-4_en
  * Adadi and Berrada [2018] Adadi, A., Berrada, M.: Peeking inside the black-box: A survey on explainable artificial intelligence (XAI). IEEE Access 6, 52138–52160 (2018) https://doi.org/10.1109/ACCESS.2018.2870052
  * Liu et al. [2022] Liu, H., Wang, Y., Fan, W., Liu, X., Li, Y., Jain, S., Liu, Y., Jain, A., Tang, J.: Trustworthy AI: A computational perspective. ACM Transactions on Intelligent Systems and Technology 14(1), 1–59 (2022) https://doi.org/10.1145/3546872
  * Li et al. [2023] Li, B., Qi, P., Liu, B., Di, S., Liu, J., Pei, J., Yi, J., Zhou, B.: Trustworthy AI: From principles to practices. ACM Computing Surveys 55(9), 1–46 (2023) https://doi.org/10.1145/3555803
  * Kale et al. [2023] Kale, A., Nguyen, T., Harris, J. Frederick C., Li, C., Zhang, J., Ma, X.: Provenance documentation to enable explainable and trustworthy AI: A literature review. Data Intelligence 5(1), 139–162 (2023) https://doi.org/10.1162/dint_a_00119
  * Alzubaidi et al. [2023] Alzubaidi, L., Al-Sabaawi, A., Bai, J., Dukhan, A., Alkenani, A., Asadi, A., Al-Wzwazy, H., Manoufali, M., Fadhel, M., Albahri, A.s., Moreira, C., Ouyang, C., Zhang, J., Santamaría, J., Salhi, A., Hollman, F., Gupta, A., Duan, Y., Rabczuk, T., Gu, Y.: Towards risk-free trustworthy artificial intelligence: Significance and requirements. International Journal of Intelligent Systems 2023, 41 (2023) https://doi.org/10.1155/2023/4459198
  * AI [2019] AI, H.: High-level expert group on artificial intelligence. European Commission (2019). https://digital-strategy.ec.europa.eu/en/library/policy-and-investment-recommendations-trustworthy-artificial-intelligence
  * Commission et al. [2020] Commission, E., Communications Networks, C., Technology: The assessment list for trustworthy artificial intelligence (ALTAI). European Commission (2020). https://digital-strategy.ec.europa.eu/en/library/assessment-list-trustworthy-artificial-intelligence-altai-self-assessment
  * [36] Deloitte GmbH Wirtschaftsprüfungsgesellschaf: Trustworthy AI. https://www2.deloitte.com/de/de/pages/innovation/contents/trustworthy-ai.html
  * der Elektrotechnik Elektronik Informationstechnik e.V [2022] Elektrotechnik Elektronik Informationstechnik e.V, V.V.: VCIO based description of systems for AI trustworthiness characterisation. VDE SPEC 90012 V1.0 (en) (2022). https://www.vde.com/resource/blob/2242194/a24b13db01773747e6b7bba4ce20ea60/vcio-based-description-of-systems-for-ai-trustworthiness-characterisationvde-spec-90012-v1-0--en--data.pdf
  * Interessengemeinschaft der Benannten Stellen für Medizinprodukte in Deutschland - IG-NB [2022] Interessengemeinschaft der Benannten Stellen für Medizinprodukte in Deutschland - IG-NB: Questionnaire Artificial Intelligence (AI) in medical devices (2022). https://www.ig-nb.de/?tx_epxelo_file%5Bid%5D=884878&cHash=53e7128f5a6d5760e2e6fe8e3d4bb02a
  * Geiger et al. [2020] Geiger, R.S., Yu, K., Yang, Y., Dai, M., Qiu, J., Tang, R., Huang, J.: Garbage in, garbage out? Do machine learning application papers in social computing report where human-labeled training data comes from? In: Proceedings of the 2020 Conference on Fairness, Accountability, and Transparency, pp. 325–336 (2020). https://doi.org/10.1145/3491102.3501868
  * Zhao et al. [2017] Zhao, J., Wang, T., Yatskar, M., Ordonez, V., Chang, K.-W.: Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints. arXiv (2017). https://doi.org/%****␣sn-article.bbl␣Line␣700␣****10.48550/ARXIV.1707.09457 . https://arxiv.org/abs/1707.09457
  * Whittlestone et al. [2019] Whittlestone, J., Nyrup, R., Alexandrova, A., Dihal, K., Cave, S.: Ethical and societal implications of algorithms, data, and artificial intelligence: a roadmap for research. London: Nuffield Foundation (2019)
  * Zemel et al. [2013] Zemel, R., Wu, Y., Swersky, K., Pitassi, T., Dwork, C.: Learning fair representations. In: Dasgupta, S., McAllester, D. (eds.) Proceedings of the 30th International Conference on Machine Learning. Proceedings of Machine Learning Research, vol. 28, pp. 325–333. PMLR, Atlanta, Georgia, USA (2013). https://proceedings.mlr.press/v28/zemel13.html
  * Kim et al. [2019] Kim, B., Kim, H., Kim, K., Kim, S., Kim, J.: Learning Not to Learn: Training Deep Neural Networks With Biased Data. IEEE (2019). https://doi.org/10.1109/cvpr.2019.00922
  * Wang et al. [2019] Wang, Z., Qinami, K., Karakozis, I.C., Genova, K., Nair, P., Hata, K., Russakovsky, O.: Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigation. arXiv (2019). https://doi.org/10.48550/ARXIV.1911.11834 . https://arxiv.org/abs/1911.11834
  * Suresh and Guttag [2021] Suresh, H., Guttag, J.: A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cycle. ACM (2021). https://doi.org/10.1145/3465416.3483305
  * Mehrabi et al. [2021] Mehrabi, N., Morstatter, F., Saxena, N., Lerman, K., Galstyan, A.: A survey on bias and fairness in machine learning. ACM Computing Surveys 54(6), 1–35 (2021) https://doi.org/10.1145/3457607
  * Wang and Strong [1996] Wang, R.Y., Strong, D.M.: Beyond accuracy: What data quality means to data consumers. Journal of Management Information Systems 12(4), 5–33 (1996) https://doi.org/10.1080/07421222.1996.11518099
  * Khatri and Brown [2010] Khatri, V., Brown, C.V.: Designing data governance. Communications of the ACM 53(1), 148–152 (2010) https://doi.org/10.1145/1629175.1629210
  * Liaw et al. [2014] Liaw, S.-T., Pearce, C., Liyanage, H., Cheah-Liaw, G.S., De Lusignan, S.: An integrated organisation-wide data quality management and information governance framework: theoretical underpinnings. Journal of Innovation in Health Informatics 21(4), 199–206 (2014) https://doi.org/10.14236/jhi.v21i4.87
  * Mo and Zheng [2008] Mo, L., Zheng, H.: A method for measuring data quality in data integration. In: 2008 International Seminar on Future Information Technology and Management Engineering, pp. 525–527 (2008). https://doi.org/10.1109/FITME.2008.146 . IEEE
  * Lindquist [2004] Lindquist, M.: Data quality management in pharmacovigilance. Drug safety 27(12), 857–870 (2004) https://doi.org/10.2165/00002018-200427120-00003
  * Souibgui et al. [2019] Souibgui, M., Atigui, F., Zammali, S., Cherfi, S., Yahia, S.B.: Data quality in ETL process: A preliminary study. Procedia Computer Science 159, 676–687 (2019) https://doi.org/10.1016/j.procs.2019.09.223
  * Gebhardt et al. [1998] Gebhardt, M., Jarke, M., Jeusfeld, M.A., Quix, C., Sklorz, S.: Tools for data warehouse quality. In: Proceedings. Tenth International Conference on Scientific and Statistical Database Management (Cat. No. 98TB100243), pp. 229–232 (1998). https://doi.org/10.1109/SSDM.1998.688130 . IEEE
  * Ballou and Tayi [1999] Ballou, D.P., Tayi, G.K.: Enhancing data quality in data warehouse environments. Communications of the ACM 42(1), 73–78 (1999) https://doi.org/10.1145/291469.291471
  * Jenkinson et al. [2003] Jenkinson, C., Fitzpatrick, R., Norquist, J., Findley, L., Hughes, K.: Cross-cultural evaluation of the Parkinson’s disease questionnaire: tests of data quality, score reliability, response rate, and scaling assumptions in the United States, Canada, Japan, Italy, and Spain. Journal of Clinical Epidemiology 56(9), 843–847 (2003) https://doi.org/10.1016/s0895-4356(03)00148-3
  * Lim et al. [2008] Lim, L.L., Seubsman, S.-a., Sleigh, A.: Thai SF-36 health survey: tests of data quality, scaling assumptions, reliability and validity in healthy men and women. Health and quality of life outcomes 6, 1–9 (2008) https://doi.org/10.1186/1477-7525-6-52
  * Candemir et al. [2021] Candemir, S., Nguyen, X.V., Folio, L.R., Prevedello, L.M.: Training strategies for radiology deep learning models in data-limited scenarios. Radiology: Artificial Intelligence 3(6), 210014 (2021) https://doi.org/10.1148/ryai.2021210014
  * Shorten and Khoshgoftaar [2019] Shorten, C., Khoshgoftaar, T.M.: A survey on image data augmentation for deep learning. Journal of big data 6(1), 1–48 (2019) https://doi.org/10.1186/s40537-019-0197-0
  * Feng et al. [2021] Feng, S.Y., Gangal, V., Wei, J., Chandar, S., Vosoughi, S., Mitamura, T., Hovy, E.: A survey of data augmentation approaches for NLP. arXiv preprint arXiv:2105.03075 (2021) https://doi.org/10.48550/arXiv.2105.03075
  * Larochelle et al. [2009] Larochelle, H., Bengio, Y., Louradour, J., Lamblin, P.: Exploring strategies for training deep neural networks. (2009). http://jmlr.org/papers/v10/larochelle09a.html
  * Vincent et al. [2008] Vincent, P., Larochelle, H., Bengio, Y., Manzagol, P.-A.: Extracting and composing robust features with denoising autoencoders. In: Proceedings of the 25th International Conference on Machine Learning, pp. 1096–1103 (2008). https://doi.org/10.1145/1390156.1390294
  * Wang and Tao [2016] Wang, R., Tao, D.: Non-local auto-encoder with collaborative stabilization for image restoration. IEEE Transactions on Image Processing 25(5), 2117–2129 (2016) https://doi.org/10.1109/TIP.2016.2541318
  * Page et al. [2021] Page, M.J., McKenzie, J.E., Bossuyt, P.M., Boutron, I., Hoffmann, T.C., Mulrow, C.D., Shamseer, L., Tetzlaff, J.M., Akl, E.A., Brennan, S.E., Chou, R., Glanville, J., Grimshaw, J.M., Hróbjartsson, A., Lalu, M.M., Li, T., Loder, E.W., Mayo-Wilson, E., McDonald, S., McGuinness, L.A., Stewart, L.A., Thomas, J., Tricco, A.C., Welch, V.A., Whiting, P., Moher, D.: The PRISMA 2020 statement: an updated guideline for reporting systematic reviews. International journal of surgery 88, 105906 (2021) https://doi.org/10.1136/bmj.n71
  * Redman [1997] Redman, T.C.: Data Quality for the Information Age (1997). https://dl.acm.org/doi/10.5555/548570
  * Loshin [2011] Loshin, D.: Dimensions of Data Quality. Elsevier (2011). https://doi.org/10.1016/b978-0-12-373717-5.00008-7
  * Yoon et al. [2000] Yoon, V.Y., Aiken, P., Guimaraes, T.: Managing organizational data resources: quality dimensions. Information Resources Management Journal (IRMJ) 13(3), 5–13 (2000) https://doi.org/10.4018/irmj.2000070101
  * Sidi et al. [2012] Sidi, F., Panahy, P.H.S., Affendey, L.S., Jabar, M.A., Ibrahim, H., Mustapha, A.: Data quality: A survey of data quality dimensions. In: 2012 International Conference on Information Retrieval & Knowledge Management, pp. 300–304 (2012). https://doi.org/10.1109/infrkm.2012.6204995 . IEEE
  * Pipino et al. [2002] Pipino, L.L., Lee, Y.W., Wang, R.Y.: Data quality assessment. Communications of the ACM 45(4), 211–218 (2002) https://doi.org/10.1145/505248.506010
  * Sebastian-Coleman [2012] Sebastian-Coleman, L.: Measuring data quality for ongoing improvement: a data quality assessment framework. Newnes (2012). https://doi.org/10.1016/C2011-0-07321-0
  * Stvilia et al. [2007] Stvilia, B., Gasser, L., Twidale, M.B., Smith, L.C.: A framework for information quality assessment. Journal of the American Society for Information Science and Technology 58(12), 1720–1733 (2007) https://doi.org/10.1002/asi.20652
  * Kim et al. [2003] Kim, W., Choi, B.-J., Hong, E.-K., Kim, S.-K., Lee, D.: A taxonomy of dirty data. Data Mining and Knowledge Discovery 7(1), 81–99 (2003) https://doi.org/10.1023/a:1021564703268
  * International Organization for Standardization and International Electrotechnical Commission [2008] International Organization for Standardization and International Electrotechnical Commission: ISO 25012. https://iso25000.com/index.php/en/iso-25000-standards/iso-25012?start=15 (2008)
  * DAMA UK Working Group on Quality Dimensions [2013] DAMA UK Working Group on Quality Dimensions: The six primary dimensions for data quality assessment. Technical report, DAMA UK - The premier organisation for data professionals in the UK (2013). https://www.sbctc.edu/resources/documents/colleges-staff/commissions-councils/dgc/data-quality-deminsions.pdf
  * Chan et al. [2010] Chan, K.S., Fowles, J.B., Weiner, J.P.: Electronic health records and the reliability and validity of quality measures: a review of the literature. Medical Care Research and Review 67(5), 503–527 (2010) https://doi.org/10.1177/1077558709359007
  * Weiskopf and Weng [2013] Weiskopf, N.G., Weng, C.: Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. Journal of the American Medical Informatics Association 20(1), 144–151 (2013) https://doi.org/10.1136/amiajnl-2011-000681
  * Nahm [2012] Nahm, M.: Data quality in clinical research. Springer (2012). https://api.semanticscholar.org/CorpusID:40817994
  * Chen et al. [2014] Chen, H., Hailey, D., Wang, N., Yu, P.: A review of data quality assessment methods for public health information systems. International Journal of Environmental Research and Public Health 11(5), 5170–5207 (2014) https://doi.org/10.3390/ijerph110505170
  * Bloland and MacNeil [2019] Bloland, P., MacNeil, A.: Defining & assessing the quality, usability, and utilization of immunization data. BMC public health 19, 1–8 (2019) https://doi.org/10.1186/s12889-019-6709-1
  * Vanbrabant et al. [2019] Vanbrabant, L., Martin, N., Ramaekers, K., Braekers, K.: Quality of input data in emergency department simulations: Framework and assessment techniques. Simulation Modelling Practice and Theory 91, 83–101 (2019) https://doi.org/10.1016/j.simpat.2018.12.002
  * Bian et al. [2020] Bian, J., Lyu, T., Loiacono, A., Viramontes, T.M., Lipori, G., Guo, Y., Wu, Y., Prosperi, M., George, T.J., Harle, C.A., Shenkman, E.A., Hogan, W.: Assessing the practice of data quality evaluation in a national clinical data research network through a systematic scoping review in the era of real-world data. Journal of the American Medical Informatics Association 27(12), 1999–2010 (2020) https://doi.org/10.1093/jamia/ocaa245
  * Tahar et al. [2023] Tahar, K., Martin, T., Mou, Y., Verbuecheln, R., Graessner, H., Krefting, D.: Rare diseases in hospital information systems - an interoperable methodology for distributed data quality assessments. Methods of Information in Medicine 62(03/04), 071–089 (2023) https://doi.org/10.1055/a-2006-1018
  * Kahn et al. [2016] Kahn, M.G., Callahan, T.J., Barnard, J., Bauck, A.E., Brown, J., Davidson, B.N., Estiri, H., Goerg, C., Holve, E., Johnson, S.G., Liaw, S.-T., Hamilton-Lopez, M., Meeker, D., Ong, T.C., Ryan, P., Shang, N., Weiskopf, N.G., Weng, C., Zozus, M.N., Schilling, L.: A harmonized data quality assessment terminology and framework for the secondary use of electronic health record data. Egems 4(1) (2016) https://doi.org/10.13063/2327-9214.1244
  * Johnson et al. [2015] Johnson, S.G., Speedie, S., Simon, G., Kumar, V., Westra, B.L.: A data quality ontology for the secondary use of EHR Data (2015). https://pubmed.ncbi.nlm.nih.gov/26958293/
  * Syed et al. [2023] Syed, R., Eden, R., Makasi, T., Chukwudi, I., Mamudu, A., Kamalpour, M., Geeganage, D.K., Sadeghianasl, S., Leemans, S.J.J., Goel, K., Andrews, R., Wynn, M.T., Hofstede, A., Myers, T.: Digital health data quality issues: Systematic review. Journal of Medical Internet Research 25, 42615 (2023) https://doi.org/10.2196/42615
  * Liu et al. [2023] Liu, C., Talaei-Khoei, A., Storey, V.C., Peng, G.: A review of the state of the art of data quality in healthcare. Journal of Global Information Management 31(1), 1–18 (2023) https://doi.org/10.4018/jgim.316236
  * Mashoufi et al. [2023] Mashoufi, M., Ayatollahi, H., Khorasani-Zavareh, D., Talebi Azad Boni, T.: Data quality in health care: main concepts and assessment methodologies. Methods of Information in Medicine 62(01/02), 005–018 (2023) https://doi.org/10.1055/s-0043-1761500
  * Lewis et al. [2023] Lewis, A.E., Weiskopf, N., Abrams, Z.B., Foraker, R., Lai, A.M., Payne, P.R.O., Gupta, A.: Electronic health record data quality assessment and tools: a systematic review. Journal of the American Medical Informatics Association 30(10), 1730–1740 (2023) https://doi.org/10.1093/jamia/ocad120
  * Schmidt et al. [2021] Schmidt, C.O., Struckmann, S., Enzenbach, C., Reineke, A., Stausberg, J., Damerow, S., Huebner, M., Schmidt, B., Sauerbrei, W., Richter, A.: Facilitating harmonized data quality assessments. A data quality framework for observational health research data collections with software implementations in R. BMC Medical Research Methodology 21(1) (2021) https://doi.org/10.1186/s12874-021-01252-7
  * Batini et al. [2015] Batini, C., Rula, A., Scannapieco, M., Viscusi, G.: From data quality to big data quality. Journal of Database Management 26(1), 60–82 (2015) https://doi.org/%****␣sn-article.bbl␣Line␣1500␣****10.4018/jdm.2015010103
  * Eder and Shekhovtsov [2020] Eder, J., Shekhovtsov, V.A.: Data Quality for Medical Data Lakelands. Springer (2020). https://doi.org/10.1007/978-3-030-63924-2_2
  * Cai and Zhu [2015] Cai, L., Zhu, Y.: The challenges of data quality and data quality assessment in the big data era. Data Science Journal 14(0), 2 (2015) https://doi.org/10.5334/dsj-2015-002
  * Gao et al. [2016] Gao, J., Xie, C., Tao, C.: Big Data Validation and Quality Assurance – Issuses, Challenges, and Needs. IEEE (2016). https://doi.org/10.1109/sose.2016.63
  * Ramasamy and Chowdhury [2020] Ramasamy, A., Chowdhury, S.: Big data quality dimensions: A systematic literature review. Journal of Information Systems and Technology Management (2020) https://doi.org/10.4301/s1807-1775202017003
  * Gudivada et al. [2017] Gudivada, V., Apon, A., Ding, J.: Data quality considerations for big data and machine learning: Going beyond data cleaning and transformations (2017). https://api.semanticscholar.org/CorpusID:221131081
  * Juddoo et al. [2018] Juddoo, S., George, C., Duquenoy, P., Windridge, D.: Data governance in the health industry: Investigating data quality dimensions within a big data context. Applied System Innovation 1(4), 43 (2018) https://doi.org/10.3390/asi1040043
  * Qi et al. [2021] Qi, Z.-X., Wang, H.-Z., Wang, A.-J.: Impacts of dirty data on classification and clustering models: An experimental evaluation. Journal of Computer Science and Technology 36(4), 806–821 (2021) https://doi.org/10.1007/s11390-021-1344-6
  * Cao et al. [2020] Cao, W., Hu, L., Gao, J., Wang, X., Ming, Z.: A study on the relationship between the rank of input data and the performance of random weight neural network. Neural Computing and Applications 32(16), 12685–12696 (2020) https://doi.org/10.1007/s00521-020-04719-8
  * Bansal et al. [1993] Bansal, A., Kauffman, R.J., Weitz, R.R.: Comparing the modeling performance of regression and neural networks as data quality varies: A business value approach. Journal of Management Information Systems 10(1), 11–32 (1993) https://doi.org/10.1080/07421222.1993.11517988
  * Twala [2013] Twala, B.: Impact of noise on credit risk prediction: Does data quality really matter? Intelligent Data Analysis 17(6), 1115–1134 (2013) https://doi.org/10.3233/IDA-130623
  * Michel et al. [2000] Michel, E., Zernikow, B., Wichert, S.A.: Use of an artificial neural network (ANN) for classifying nursing care needed, using incomplete input data. Medical Informatics and the Internet in Medicine 25(2), 147–158 (2000) https://doi.org/10.1080/14639230050058329
  * Jouseau et al. [2022] Jouseau, R., Salva, S., Samir, C.: On Studying the Effect of Data Quality on Classification Performances. Springer (2022). https://doi.org/10.1007/978-3-031-21753-1_9
  * Che et al. [2018] Che, Z., Purushotham, S., Cho, K., Sontag, D., Liu, Y.: Recurrent neural networks for multivariate time series with missing values. Scientific reports 8(1), 6085 (2018) https://doi.org/10.1038/s41598-018-24271-9
  * Benedick et al. [2021] Benedick, P.-L., Robert, J., Traon, Y.L.: A systematic approach for evaluating artificial intelligence models in industrial settings. Sensors 21(18), 6195 (2021) https://doi.org/10.3390/s21186195
  * Blake and Mangiameli [2011] Blake, R., Mangiameli, P.: The effects and interactions of data quality and problem complexity on classification. Journal of Data and Information Quality 2(2), 1–28 (2011) https://doi.org/10.1145/1891879.1891881
  * He et al. [2019] He, T., Yu, S., Wang, Z., Li, J., Chen, Z.: From data quality to model quality: An exploratory study on deep learning. In: Proceedings of the 11th Asia-Pacific Symposium on Internetware, pp. 1–6 (2019). https://doi.org/10.1145/3361242.3361260
  * Rolnick et al. [2017] Rolnick, D., Veit, A., Belongie, S., Shavit, N.: Deep Learning is Robust to Massive Label Noise. arXiv (2017). https://doi.org/10.48550/ARXIV.1705.10694
  * Sukhbaatar et al. [2014] Sukhbaatar, S., Bruna, J., Paluri, M., Bourdev, L., Fergus, R.: Training Convolutional Networks with Noisy Labels. arXiv (2014). https://doi.org/10.48550/ARXIV.1406.2080
  * Wesemeyer et al. [2021] Wesemeyer, T., Jauer, M.-L., Deserno, T.M.: Annotation quality vs. quantity for deep-learned medical image segmentation. SPIE (2021). https://doi.org/10.1117/12.2582226 . https://doi.org/10.1117/12.2582226
  * Hong and Shen [2021] Hong, S., Shen, J.: Impact of Training Size on Deep Learning Performance in in Vivo 1H MRS. In: ISMRM & SMRT Annual Meeting & Exhibition (2021). https://archive.ismrm.org/2021/2015.html
  * Sun et al. [2017] Sun, C., Shrivastava, A., Singh, S., Gupta, A.: Revisiting unreasonable effectiveness of data in deep learning era. In: Proceedings of the IEEE International Conference on Computer Vision, pp. 843–852 (2017). https://doi.org/10.48550/arXiv.1707.02968
  * Li et al. [2021] Li, Y., Yang, J., Wen, J.: Entropy-based redundancy analysis and information screening. Digital Communications and Networks (2021) https://doi.org/10.1016/j.dcan.2021.12.001
  * Ranjan et al. [2023] Ranjan, R., Sharrer, K., Tsukuda, S., Good, C.: Effects of image data quality on a convolutional neural network trained in-tank fish detection model for recirculating aquaculture systems. Computers and Electronics in Agriculture 205, 107644 (2023) https://doi.org/10.1016/j.compag.2023.107644
  * Li and Chao [2021] Li, Y., Chao, X.: Toward sustainability: Trade-off between data quality and quantity in crop pest recognition. Frontiers in Plant Science 12 (2021) https://doi.org/10.3389/fpls.2021.811241
  * Fan and Shi [2022] Fan, F.J., Shi, Y.: Effects of data quality and quantity on deep learning for protein-ligand binding affinity prediction. Bioorganic & Medicinal Chemistry 72, 117003 (2022) https://doi.org/10.1016/j.bmc.2022.117003
  * Pan et al. [2023] Pan, Y., Xie, F., Zhao, H.: Understanding the challenges when 3D semantic segmentation faces class imbalanced and OOD data. IEEE Transactions on Intelligent Transportation Systems 24(7), 6955–6970 (2023) https://doi.org/10.1109/tits.2023.3256442
  * Masko and Hensman [2015] Masko, D., Hensman, P.: The impact of imbalanced training data for convolutional neural networks (2015). https://www.kth.se/social/files/588617ebf2765401cfcc478c/PHensmanDMasko_dkand15.pdf
  * Buda et al. [2018] Buda, M., Maki, A., Mazurowski, M.A.: A systematic study of the class imbalance problem in convolutional neural networks. Neural Networks 106, 249–259 (2018) https://doi.org/10.1016/j.neunet.2018.07.011
  * Karahan et al. [2016] Karahan, S., Yildirim, M.K., Kirtac, K., Rende, F.S., Butun, G., Ekenel, H.K.: How image degradations affect deep CNN-based face recognition? (2016) https://doi.org/10.48550/ARXIV.1608.05246
  * Dodge and Karam [2016] Dodge, S., Karam, L.: Understanding how image quality affects deep neural networks. In: 2016 Eighth International Conference on Quality of Multimedia Experience (QoMEX), pp. 1–6 (2016). https://doi.org/10.48550/arXiv.1604.04004 . IEEE
  * Whang et al. [2023] Whang, S.E., Roh, Y., Song, H., Lee, J.-G.: Data collection and quality challenges in deep learning: a data-centric AI perspective. The VLDB Journal 32(4), 791–813 (2023) https://doi.org/10.1007/s00778-022-00775-9
  * Ovadia et al. [2019] Ovadia, Y., Fertig, E., Ren, J., Nado, Z., Sculley, D., Nowozin, S., Dillon, J., Lakshminarayanan, B., Snoek, J.: Can you trust your model’s uncertainty? Evaluating predictive uncertainty under dataset shift. Advances in neural information processing systems 32 (2019) https://doi.org/10.48550/arXiv.1906.02530
  * Xu et al. [2023] Xu, S., Yao, Y., Xu, F., Gu, T., Xu, J., Ma, X.: Data Quality Matters: A Case Study of Obsolete Comment Detection. IEEE (2023). https://doi.org/10.1109/icse48619.2023.00074 . https://doi.org/10.1109/icse48619.2023.00074
  * Shimizu and Wakabayashi [2022] Shimizu, A., Wakabayashi, K.: Examining effect of label redundancy for machine learning using crowdsourcing. Journal of Data Intelligence 3(3), 301–315 (2022) https://doi.org/10.1145/3487664.3487677
  * Derry et al. [2022] Derry, A., Carpenter, K.A., Altman, R.B.: Training data composition affects performance of protein structure analysis algorithms (2022). https://www.ncbi.nlm.nih.gov/pmc/articles/PMC8669736/
  * Xu et al. [2002] Xu, H., Horn Nord, J., Brown, N., Daryl Nord, G.: Data quality issues in implementing an ERP. Industrial management & data systems 102(1), 47–58 (2002) https://doi.org/10.1108/02635570210414668
  * Verma et al. [2019] Verma, R.M., Zeng, V., Faridi, H.: Data quality for security challenges: Case studies of phishing, malware and intrusion detection datasets. In: Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pp. 2605–2607 (2019). https://doi.org/10.1145/3319535.3363267
  * Laney et al. [2001] Laney, D., et al.: 3D data management: Controlling data volume, velocity and variety. Stanford (2001). https://www.scirp.org/reference/ReferencesPapers?ReferenceID=1611280
  * Black and van Nederpelt [2020] Black, A., Nederpelt, P.: Dimensions of Data Quality (DDQ). https://www.dama-nl.org/wp-content/uploads/2020/09/DDQ-Dimensions-of-Data-Quality-Research-Paper-version-1.2-d.d.-3-Sept-2020.pdf. [Accessed 25-10-2023] (2020)
  * IEEE Standards Committee et al. [1990] IEEE Standards Committee, et al.: IEEE standard glossary of software engineering terminology. IEEE Std 610, 12 (1990) https://doi.org/10.1109/IEEESTD.1990.101064
  * Wilkinson et al. [2016] Wilkinson, M.D., Dumontier, M., Aalbersberg, I.J., Appleton, G., Axton, M., Baak, A., Blomberg, N., Boiten, J.-W., Silva Santos, L.B., Bourne, P.E., Bouwman, J., Brookes, A.J., Clark, T., Crosas, M., Dillo, I., Dumon, O., Edmunds, S., Evelo, C.T., Finkers, R., Gonzalez-Beltran, A., Gray, A.J.G., Groth, P., Goble, C., Grethe, J.S., Heringa, J., Hoen, P.A.C., Hooft, R., Kuhn, T., Kok, R., Kok, J., Lusher, S.J., Martone, M.E., Mons, A., Packer, A.L., Persson, B., Rocca-Serra, P., Roos, M., Schaik, R., Sansone, S.-A., Schultes, E., Sengstag, T., Slater, T., Strawn, G., Swertz, M.A., Thompson, M., Lei, J., Mulligen, E., Velterop, J., Waagmeester, A., Wittenburg, P., Wolstencroft, K., Zhao, J., Mons, B.: The FAIR guiding principles for scientific data management and stewardship. Scientific Data (2016) https://doi.org/10.1038/sdata.2016.18
  * Bishop [1995] Bishop, C.M.: Training with noise is equivalent to tikhonov regularization. Neural Computation 7(1), 108–116 (1995) https://doi.org/10.1162/neco.1995.7.1.108
  * Grandvalet et al. [1997] Grandvalet, Y., Canu, S., Boucheron, S.: Noise injection: Theoretical prospects. Neural Computation 9(5), 1093–1108 (1997) https://doi.org/10.1162/neco.1997.9.5.1093
  * Smilkov et al. [2017] Smilkov, D., Thorat, N., Kim, B., Viégas, F., Wattenberg, M.: Smoothgrad: removing noise by adding noise. arXiv preprint arXiv:1706.03825 (2017) https://doi.org/10.48550/arXiv.1706.03825
  * Thaler and Sunstein [2009] Thaler, R.H., Sunstein, C.R.: Nudge: Improving decisions about health, wealth, and happiness. Penguin (2009). https://www.penguinbookshop.com/book/9780143115267
  * Kahneman [2011] Kahneman, D.: Thinking, fast and slow. macmillan (2011). https://doi.org/10.1007/s00362-013-0533-y
  * Malossini et al. [2006] Malossini, A., Blanzieri, E., Ng, R.T.: Detecting potential labeling errors in microarrays by data perturbation 22(17), 2114 (2006) https://doi.org/10.1093/bioinformatics/btl346
  * Frénay and Verleysen [2013] Frénay, B., Verleysen, M.: Classification in the presence of label noise: a survey. IEEE transactions on neural networks and learning systems 25(5), 845–869 (2013) https://doi.org/10.1109/TNNLS.2013.2292894
  * Menze et al. [2014] Menze, B.H., Jakab, A., Bauer, S., Kalpathy-Cramer, J., Farahani, K., Kirby, J., Burren, Y., Porz, N., Slotboom, J., Wiest, R., Lanczi, L., Gerstner, E., Weber, M.-A., Arbel, T., Avants, B.B., Ayache, N., Buendia, P., Collins, D.L., Cordier, N., Corso, J.J., Criminisi, A., Das, T., Delingette, H., Demiralp, C., Durst, C.R., Dojat, M., Doyle, S., Festa, J., Forbes, F., Geremia, E., Glocker, B., Golland, P., Guo, X., Hamamci, A., Iftekharuddin, K.M., Jena, R., John, N.M., Konukoglu, E., Lashkari, D., Mariz, J.A., Meier, R., Pereira, S., Precup, D., Price, S.J., Raviv, T.R., Reza, S.M.S., Ryan, M., Sarikaya, D., Schwartz, L., Shin, H.-C., Shotton, J., Silva, C.A., Sousa, N., Subbanna, N.K., Szekely, G., Taylor, T.J., Thomas, O.M., Tustison, N.J., Unal, G., Vasseur, F., Wintermark, M., Ye, D.H., Zhao, L., Zhao, B., Zikic, D., Prastawa, M., Reyes, M., Van Leemput, K.: The multimodal brain tumor image segmentation benchmark (BRATS). IEEE transactions on medical imaging 34(10), 1993–2024 (2014) https://doi.org/10.1109/TMI.2014.2377694
  * Deng [2012] Deng, L.: The MNIST database of handwritten digit images for machine learning research. IEEE signal processing magazine 29(6), 141–142 (2012) https://doi.org/10.1109/MSP.2012.2211477
  * Krizhevsky [2009] Krizhevsky, A.: Learning multiple layers of features from tiny images. Toronto, ON, Canada (2009). https://api.semanticscholar.org/CorpusID:18268744
  * Xiao et al. [2017] Xiao, H., Rasul, K., Vollgraf, R.: Fashion-MNIST: A novel image dataset for benchmarking machine learning algorithms. arXiv preprint arXiv:1708.07747 (2017) https://doi.org/10.48550/arXiv.1708.07747
  * Müller and Markert [2019] Müller, N.M., Markert, K.: Identifying mislabeled instances in classification datasets. In: 2019 International Joint Conference on Neural Networks (IJCNN), pp. 1–8 (2019). https://doi.org/10.1109/IJCNN.2019.8851920 . IEEE
  * Northcutt et al. [2021] Northcutt, C., Jiang, L., Chuang, I.: Confident learning: Estimating uncertainty in dataset labels. Journal of Artificial Intelligence Research 70, 1373–1411 (2021) https://doi.org/10.48550/arXiv.1911.00068
  * Daniel Kahneman and Sunstein [2021] Daniel Kahneman, O.S., Sunstein, C.R.: Noise: A Flaw in Human Judgment. Little Brown Spark, ??? (2021). Chap. Chapter 22
  * Jaramillo [2022] Jaramillo, D.: Radiologists and their noise: Variability in human judgment, fallibility, and strategies to improve accuracy. Radiology 302(3), 511–512 (2022) https://doi.org/10.1148/radiol.211891
  * Anderson et al. [2001] Anderson, R.N., Miniño, A.M., Hoyert, D.L., Rosenberg, H.M.: Comparability of cause of death between icd-9 and icd-10: preliminary estimates (2001)
  * Sebastião et al. [2021] Sebastião, Y.V., Metzger, G.A., Chisolm, D.J., Xiang, H., Cooper, J.N.: Impact of icd-9-cm to icd-10-cm coding transition on trauma hospitalization trends among young adults in 12 states. Injury Epidemiology 8(1) (2021) https://doi.org/10.1186/s40621-021-00298-x