# Standardness Fogs Meaning: A Position Regarding the Informed Usage of
Standard Datasets

Tim Cech<EMAIL_ADDRESS>1234-5678-9012 Digital Engineering Faculty,
University of PotsdamProf.-Dr.-Helmert-Str. 2–3PotsdamBrandenburgGermany14482
,  Ole Wegen  Digital Engineering Faculty, University of PotsdamGermany
<EMAIL_ADDRESS>0000-0002-6571-5897 ,  Daniel Atzberger
<EMAIL_ADDRESS>0000-0002-5409-7843 Hasso Plattner Institute, Digital
Engineering Faculty, University of PotsdamGermany ,  Rico Richter  Digital
Engineering Faculty, University of PotsdamGermany rico.richter.1@uni-
potsdam.de 0000-0001-5523-3694 ,  Willy Scheibel  0000-0002-7885-9857 Hasso
Plattner Institute, Digital Engineering Faculty, University of PotsdamGermany
<EMAIL_ADDRESS>and  Jürgen Döllner  Hasso Plattner Institute, Digital
Engineering Faculty, University of PotsdamGermany<EMAIL_ADDRESS>0000-0002-8981-8583

(15 September 2023; xx December 2023; xx December 2023)

###### Abstract.

Standard datasets are frequently used to train and evaluate Machine Learning
models. However, the assumed _standardness_ of these datasets leads to a lack
of in-depth discussion on how their labels match the derived categories for
the respective use case. In other words, the standardness of the datasets
seems to fog coherency and applicability, thus impeding the trust in Machine
Learning models. We propose to adopt Grounded Theory and Hypotheses Testing
through Visualization as methods to evaluate the match between use case,
derived categories, and labels of standard datasets. To showcase the approach,
we apply it to the 20 Newsgroups dataset and the MNIST dataset. For the 20
Newsgroups dataset, we demonstrate that the labels are imprecise. Therefore,
we argue that neither a Machine Learning model can learn a meaningful
abstraction of derived categories nor one can draw conclusions from achieving
high accuracy. For the MNIST dataset, we demonstrate how the labels can be
confirmed to be defined well. We conclude that a concept of standardness of a
dataset implies that there is a match between use case, derived categories,
and class labels, as in the case of the MNIST dataset. We argue that this is
necessary to learn a meaningful abstraction and, thus, improve trust in the
Machine Learning model.

Explainable AI, Trust, Dataset Quality

††ccs: Computing methodologies Reasoning about belief and knowledge††ccs:
Computing methodologies Philosophical/theoretical foundations of artificial
intelligence

##  1\. Introduction

Big Data Datasets are a foundation of Machine Learning (ML), especially for
supervised deep learning. Traditionally, they are characterized by the five
V’s: Volume, Velocity, Variety, Veracity, and Value (Kitchin and McArdle,
2016). Unfortunately, due to the No-Free-Lunch Theorem (NFL), it is impossible
to generalize from the performance of an ML model on one dataset to its
performance on another one (Adam et al., 2019). Nevertheless, to allow for
limited comparable model evaluation, standard datasets are commonly used to
benchmark ML models. Standard datasets are seen as Big Data111If one
conceptualizes a standard dataset as not being Big Data, it raises the
question why ML models built for working with Big Data Datasets are evaluated
on standard datasets. We follow Kitchin and McArdle (2016) in recognizing the
ambiguity in the term “Big Data” and therefore opt for a broader use of “Big
Data” as is often done in practice. Datasets (Asim et al., 2019; Chandra,
2019; Chen and Dai, 2021), created for one specific use case or application
field, e.g., text classification or computer vision (Kitchin and McArdle,
2016). For text classification, one such standard dataset is the 20 Newsgroups
dataset (Wang et al., 2016; Pappagari et al., 2019); for computer vision, an
example is the MNIST (Modified National Institute of Standards and Technology)
dataset (Chen et al., 2019a; Garg et al., 2019).

In this paper, we argue that trust in ML models can only be achieved by
getting an informed perspective onto the dataset used for training and
evaluation. However, we perceive that standard datasets are often used
uncritically based on their _standardness_ , which fogs two possible breaking
points:

  * •

There could be a mismatch between labels and meaningful concepts. In
literature, the term _class_ is frequently used to address both, the labels
and the implied concepts. We differentiate between those by using the term
(class) label if we refer to the pre-defined mapping from data points to class
names according to the dataset and the term category if we refer to the
concept implied by the name of a class. For example, the alt.atheism label in
the 20 Newsgroups dataset (allegedly) corresponds to the category of atheism
regarding the content of a text. However, it could be that the label does not
match the implied category, i.e., not all texts with the label alt.atheism
actually deal with the topic of atheism.

  * •

The dataset could be unsuitable for the targeted use case. For example, if the
use case is the automatic classification of arbitrary texts with respect to
their main topic, the 20 Newsgroups dataset might by unsuitable for training
and testing. As all texts within the dataset come from mailing lists, an ML
model trained on this data may not be able to generalize onto other text
genres.

These two possible breaking points are often not discussed, relying on and
appealing to the standardness of the used datasets, which, however, is an
unclear characterization, especially since there is no process of how a
dataset becomes “standard”. Often, this happens due to historical developments
and the subsequent frequent use of a dataset, which provides no guarantee for
the quality of labels and the applicability to specific use cases.

Therefore, we argue for the critical usage of standard datasets, as this is
required for trustworthy reasoning about the quality of an ML model. According
to Lazer et al. (2014, p.1), the “‘[b]ig data hubris’ is the often implicit
assumption that big data are a substitute for, rather than a supplement to,
traditional data collection and analysis.” By focusing on the human
understanding of the match between use case, derived categories, and class
labels instead of quantifiable metrics, we aim to help alleviating this big
data hubris. We argue that a standard dataset’s labels only convey meaning if
they adhere to this match (see Figure 1). If this is not the case, the result
is uninterpretable and untrustworthy, as it is unclear what the ML model
learned (Symons and Alvarado, 2019). Therefore, used datasets have to be
closely examined and matched with the intended use case to understand how they
influence ML models. For this, we propose to adopt interdisciplinary methods,
namely Grounded Theory and Hypotheses Testing through Visualization. We
exemplify this approach by applying it to the usage of the 20 Newsgroups
dataset as a negative case (i.e., labels do not match categories and/or the
dataset is not suitable for the targeted use case) and the MNIST dataset as a
positive case (i.e., labels match categories and the dataset is suitable for
the targeted use case).

In summary, as our main contribution, we question the term of _standardness_
in relation to datasets, which often fogs the need to explicitly match use
cases, categories and class labels in order to achieve explainability and
trustworthiness. Additionally, we propose a quali-quantitative method (Blok
and Pedersen, 2014) for dataset interrogation that combines a qualitative and
a quantitative approach to increase trust in the evaluation results and,
consequently, the quality of ML models trained on the respective datasets. In
section 2, we will review literature on how to investigate the meaning of
labels and foundations of the adopted methods. In section 3, we reflect upon
the usage of the 20 Newsgroups and MNIST dataset in literature and describe
some characteristics of those datasets. In section 4, we demonstrate the
proposed quali-quantiative method by applying it to the two datasets. We
summarize and discuss our findings in section 5 before concluding this paper
in section 6.

Data Scientist Dataset Fit? Use CaseCategoriesMeaningful
Labelshasmatchesderives Meaningful abstraction Match enables trustworthy
training Figure 1. We argue that a data scientist must actively ensure the
match between the use case, labels, and categories (dashed lines). The use
case provides the context for deriving categories which should match the
labels (solid lines).

##  2\. Related Work

In the following, we draw connections between our position and existing
research regarding standard datasets, ML epistemology, and data quality and
assessment.

#### Machine Learning Epistemology.

Desai et al. (2022, p.13) point out that supervised models mainly rely on
inductive reasoning. Because of this, supervised models face the fundamental
problem that they become outdated once the training data does not conform with
the observed patterns any longer. We agree with this notion and want to add
that ML research papers often lack statements about what they expect from a
standard dataset, thus making this issue of outdated models harder to
identify. For examining whether a model has learned meaningful concepts from
the training data, visualization techniques can be used. Especially if the
visualization is informed by prior qualitative findings, it can enrich the
understanding of and trust in ML models (Morrison et al., 2023; Li and Zhou,
2023). However, as recent work by Jeon et al. (2023) points out, care has to
be taken when using Dimensionality Reduction (DR) techniques in the context of
visual analysis, as clusters in the two-dimensional plane can not easily be
assumed to represent meaningful categories. More likely, categories are
spanning an arbitrary structure in the high-dimensional space, making it
difficult to find appropriate projections that lead to interpretable results.
While we agree, we investigate the problem at a more fundamental level, asking
whether a concrete class label represents a coherent category in the high-
dimensional space at all. For this, we rely on benchmarking studies, e.g.,
Espadoto et al. (2019) and Atzberger et al. (2023), to choose appropriate
methods for DR that lead to reliable representations.

#### Views on Datasets and ML Models in Different Disciplines.

As the term _Big Data_ is only loosely defined in research (Kitchin and
McArdle, 2016), the dependent term of the _standard dataset_ also remains
unclear (e.g., what exactly denotes _variety_ for standard datasets). Further,
the view on standard datasets and ML models differs across separate
disciplines. On the one hand, scientists with various backgrounds treat
standard “data[-sets] as the media through which conversation, negotiation,
and action can occur” (Neff et al., 2017, p.2), which is a qualitative view on
the matter. Here, standard datasets are perceived as basis for automated
decision-making and, thus, in a way exercise power over people or shape their
perception of reality (Introna and Nissenbaum, 2000; Lenglet, 2014). A
particular focus is placed upon the fact that it is impossible to separate ML
models (and, therefore, implicitly their training datasets) from their
societal context as they tend to be mystified by the public (Elish and danah
boyd, 2018) and even programmers can usually not restrict their ML models to
the computational context (Seaver, 2017) or avoid making ethical decisions
(Mittelstadt et al., 2016). Furthermore, algorithm audits revealed that ML
models have the potential to be racist and sexist (Baker and Potts, 2013), can
threaten users (Bucher, 2012), and can produce unfair and biased output
(Mehrabi et al., 2021). In summary, the qualitative aspects of ML models and
the underlying data is emphasized in this branch of research. On the other
hand, a lot of computer science research is mainly focused on quantitative
aspects of ML models. While, for example, the issues of biases (Mehrabi et
al., 2021) or explainability (Samek and Müller, 2019) are also acknowledged,
the field of Explainable AI (XAI) that emerged in this context attempts to
provide automatic explanations for opaque decision-making ML models on the
basis of quantitative analysis. Therefore, XAI usually remains in the field of
inductive reasoning (Desai et al., 2022, p.13) and thus does not engage in a
qualitative manual analysis of the data (Blok and Pedersen, 2014; Lazer et
al., 2014; Symons and Alvarado, 2019). However, while XAI is profoundly model-
centric (Northcutt et al., 2021), some research also attempts to understand
noise in labels on a qualitative level (Chen et al., 2019b). We also propose
to combine quantitative and qualitative methods for dataset assessment and, in
particular, argue that due to the unclear definition and characteristics of
standard datasets, every researcher should examine the used datasets carefully
with respect to the targeted use case.

#### Alleviation of Dataset Issues.

Confidence learning as a subfield of ML is concerned with potential errors in
labels and how to mitigate their impact. The field is constituted by the
observation that “Advances in learning with noisy labels […] usually introduce
a new model or loss function. Often this model-centric approach band-aids the
real question: which data is mislabeled?” (Northcutt et al., 2021, p.1).
Typically, the noise in labels is modeled as a random variable to account for
it during training (Chen et al., 2019b; Natarajan et al., 2013; Northcutt et
al., 2021). On the one hand, we are concerned with the same phenomenon, namely
that labels may be noisy. On the other hand, we emphasize that the noise we
want to address is qualitative and, therefore, intractable by a simple
mathematical quantification, e.g., by a random variable. We argue that manual
investigation is required first to correctly model noise in labels. In
particular, we want to determine whether the noise in labels is caused by a
systemic, frequent gap between the alleged category (meaning) and the contents
of the data or whether it is _real_ noise in the mathematical sense.

#### Manual Data Assessment.

Lewis et al. (2013) suggested manual methods for investigating datasets. The
authors argue that in addition to automated quantitative evaluation, one
should use manual content evaluation to capture _latent features_ that may not
be as easily quantified (Lewis et al., 2013, p.12). Similarly, Blok and
Pedersen (2014) argue that qualitative methods from social science can
complement the quantitative methods that are commonly used in computer
science. In this regard, Cech et al. (2023) have given a first impression of
how such mixed methods can lead to new knowledge artifacts about ML models.

##  3\. Exemplified Analysis of History, Characteristics, and Usage of
Standard Datasets

We claim that, currently, the standardness of a dataset does not imply a
certain data quality with respect to correct labels, noise, or diversity. We
further claim that, nevertheless, researchers often use standard datasets
without careful examination of their quality and in-depth discussion about why
the datasets are appropriate for the intended use case. To underpin and
exemplify these two claims, we first provide a descriptive analysis of two
datasets, namely the 20 Newsgroups dataset and the MNIST dataset, to
illustrate possible issues with these datasets in general, despite their
standardness. For each dataset, we then review 15 research papers that make
use of it and evaluate how well the use of the dataset is motivated and how
well the dataset itself is interrogated. We included a paper if either a model
on the dataset is evaluated or the dataset is heavily used otherwise. We chose
the top papers according to Google Scholar with the cut-off date at 01-01-2015
to review reasonably recent research articles. We want to clarify that
gathering sources by Google Scholar is not deterministic but often good enough
to find pertinent references (Giustini and Boulos, 2013). Therefore, our
review is not replicable and can not be considered a systematic survey.
Nevertheless, we settled for this method since, in our experience, computer
science researchers still use Google Scholar to find relevant references and,
as far as scientific literature is concerned, researchers are likely
introduced to the datasets through papers they discovered using this search
engine.

We assess each paper with respect to five categories, namely (1) whether the
author(s) of the research paper gave a traceable reference to the dataset, (2)
whether they explicitly chose a specific version of the dataset, (3) whether
they analyzed the content of the dataset at all, (4) whether they gave an
example from the dataset, and (5) whether they performed an in-depth analysis
of the contents for their use case. These categories characterize the level of
detail in the interrogation of the dataset. The results of the review are
shown in Table 1 and Table 2.

###  3.1. The 20 Newsgroups dataset

The 20 Newsgroups dataset contains texts sent via e-mail lists in the late
1990s curated by Ken Lang (Joachims, 1996). We obtained the 20 Newsgroups
dataset from Jason Rennie’s Home Page222http://qwone.com/ jason/20Newsgroups/
since it often appears to be given as the source (Varoquaux et al., 2015;
Bodrunova et al., 2020). Additionally, according to their respective
documentation, the popular Python ML library Scikit-Learn333https://scikit-
learn.org/stable/datasets/real_world.html#the-20-newsgroups-text-dataset and
the dataset-sharing site
Kaggle444https://www.kaggle.com/datasets/crawford/20-newsgroups also state
this source. According to Jason Rennie’s website, three different versions of
the dataset exist. The original dataset contains 19 997 documents,
approximately evenly distributed over 20 categories. A second version with
fewer headers and removed duplicates contains 18 846 documents. A third
version with even less duplicates but maintained From and Subject header lines
contains 18 828 documents. The website speculates that the dataset was
gathered for Lang (1995) though it was finally not included in the paper. This
proposition is strengthened by other sources from the 90s, for example by
Joachims (1996).

The categories are constituted by the name of the respective mailing list.
Each category contains a selection of e-mails from several threads that appear
to be loosely connected. According to the header field From, the dataset
contains posts from 8 644 unique e-mail addresses. Please note that we assumed
that an author is uniquely identified by the content of the _From_ line. We
ignored the case where one author uses several aliases or e-mail addresses.
Naturally, not all authors contributed equally to the dataset. Only 3 080, 1
659, and 293 authors contributed at least two, three, and ten posts,
respectively. This corresponds to a share of 72 $\mathrm{\%}$, 57
$\mathrm{\%}$, and 27 $\mathrm{\%}$ of the number of all posts from 36
$\mathrm{\%}$, 19 $\mathrm{\%}$, and 3 $\mathrm{\%}$ of all authors,
respectively. At the line level, the difference between the number of total
lines (762 321) and lines of authors with at least two (601 304 lines
corresponding to a share of 78 $\mathrm{\%}$), three (505 800 lines
corresponding to a share of 66 $\mathrm{\%}$), and ten (258 885 lines
corresponding to a share of 34 $\mathrm{\%}$) posts becomes even more
apparent. Therefore, the language of only three percent of authors represents
a disproportional share of around one-third of the text corpus. Furthermore,
we also want to emphasize that much of the content comprises quotes from other
e-mails, indicated by a starting right arrow symbol. Overall, over 16.5
$\mathrm{\%}$ of all lines (126 202 lines) are quotes and not original content
from the respective author. The high ratio of quotes results in texts that
significantly differ from spoken language (Davidson, 2015).

Paper | Source Given? | Explicit Choice? | States Content? | Example? | In-Depth?
---|---|---|---|---|---
Albishre et al. (Albishre et al., 2015) | (yes) | yes | yes | (yes) | no
Asim et al. (Asim et al., 2019) | no | (yes) | no | no | no
Bodrunova et al. (Bodrunova et al., 2020) | yes | yes | no | no | no
Chandra (Chandra, 2019) | yes | no | no | no | no
Chen and Dai (Chen and Dai, 2021) | yes | yes | no | no | no
Hsu (Hsu, 2020) | (yes) | yes | no | no | no
Kou et al. (Kou et al., 2020) | no | no | yes | no | no
Mauni et al. (Mauni et al., 2020) | no | no | no | no | no
Pappagari et al. (Pappagari et al., 2019) | yes | yes | yes | no | no
Ruff et al. (Ruff et al., 2019) | yes | no | no | no | no
Saigal and Khanna (Saigal and Khanna, 2020) | no | no | yes | no | no
Singh et al. (Singh et al., 2023) | yes | no | (yes) | no | no
Wang et al. (Wang et al., 2016) | yes | no | no | no | no
Wang and Yang (Wang and Yang, 2020) | yes | yes | no | no | no
Zhang and Yamana (Zhang and Yamana, 2021) | (yes) | yes | no | no | no
Table 1. An overview of the 15 reviewed ML research papers for the 20
Newsgroups dataset, assessed according to the defined categories. We used the
entry _(yes)_ if the property was given indirectly.

Sometimes, e.g., by Albishre et al. (2015) or Zhang and Yamana (2021), the
already mentioned paper by Lang (1995) is given as the source. Curiously, Kou
et al. (2020) state that Joachims (1996) collected the 20 Newsgroups dataset,
which is doubtful since Joachims himself gives Lang as the original collector
(Joachims, 1996, p.13). Saigal and Khanna (2020, p.9) link to another
website555https://www.ics.uci.edu/ and additionally state without reference
that the dataset was collected by Lang. Hsu (2020) references the Scikit-Learn
library as the source, which references the original website. Often, Jason
Rennie’s website is identified as the source of the dataset (Chandra, 2019;
Pappagari et al., 2019; Wang et al., 2016; Varoquaux et al., 2015; Bodrunova
et al., 2020). Seldom, the different versions of the dataset are mentioned to
make transparent which version of the dataset was used (e.g., by Albishre et
al. (2015)). More often, the authors state that the dataset contains
approximately 20 000 documents from 20 different classes which holds for all
three versions of the dataset, e.g., in Wang et al. (2016); Pappagari et al.
(2019), and Chandra (2019). Interestingly, the authors rarely, and then only
briefly, discuss what they think the 20 Newsgroups contain. Usually, the
discussion tends to be on the descriptive level. It contains statements about
the context and lists some example label names (Kou et al., 2020; Saigal and
Khanna, 2020). Only in one case, an example document was shown, however,
without discussion (Albishre et al., 2015). Furthermore, no reviewed paper
discusses in detail why the usage of the 20 Newsgroups dataset is appropriate
for the intended use case.

In summary, in the reviewed literature it remains unclear why the 20
Newsgroups dataset was used and why the authors validate their method on this
dataset.

Paper | Source Given? | Explicit Choice? | States Content? | Example? | In-Depth?
---|---|---|---|---|---
Baldominos et al. (Baldominos et al., 2019) | yes | yes | yes | yes | yes
Castro et al. (Castro et al., 2019) | yes | no | (yes) | yes | no
Chen et al. (Chen et al., 2019a) | yes | yes | yes | no | no
Cheng et al. (Cheng et al., 2020) | no | yes | yes | no | no
Cohen et al. (Cohen et al., 2017) | yes | yes | yes | no | no
Garg et al. (Garg et al., 2019) | no | no | yes | yes | no
Garin and Tauzin (Garin and Tauzin, 2019) | no | no | no | yes | no
Kadam et al. (Kadam et al., 2020) | yes | yes | yes | yes | no
Karayaneva and Hintea (Karayaneva and Hintea, 2018) | yes | yes | yes | no | no
Kerenidis and Luongo (Kerenidis and Luongo, 2020) | yes | yes | no | no | no
Romanuke (Romanuke, 2016) | yes | yes | yes | yes | no
Saadna et al. (Saadna et al., 2019) | yes | yes | no | no | no
Serrano-Gotarredona and Linares-Barranco (Serrano-Gotarredona and Linares-Barranco, 2015) | yes | yes | yes | yes | no
Tabik et al. (Tabik et al., 2017) | yes | yes | yes | no | no
Yadav and Bottou (Yadav and Bottou, 2019) | yes | yes | yes | yes | yes
Table 2. An overview of the 15 reviewed ML research papers for the MNIST
dataset, assessed according to the defined categories. We used the entry
_(yes)_ if the property was given indirectly.

###  3.2. The MNIST dataset

The MNIST dataset contains 28x28 pixel images of hand-written digits from 2000
Census Bureau employees and 500 high school students in the 1990s (Baldominos
et al., 2019; Yadav and Bottou, 2019). Baldominos et al. (2019) state that the
dataset origins from the NIST special datasets 1 and 3 which were later
combined into special dataset 19 (Grother and Hanaoka, 1995, p.3). In contrast
to the 20 Newsgroups dataset, the history of the MNIST dataset is documented
by the study of Yadav and Bottou (2019). According to them, Bottou et al.
(1994) introduced the dataset. However, since this conference paper was added
in 2002 to the IEEE digital
library666https://ieeexplore.ieee.org/document/576879, eight years after it
was originally published, the origin of the dataset is often wrongly
contributed to LeCun et al. (1998) of the same working group (Baldominos et
al., 2019; Cohen et al., 2017; Serrano-Gotarredona and Linares-Barranco,
2015). Since most papers reference LeCun et al. (1998) as the dataset’s source
and LeCun’s website777http://yann.lecun.com/exdb/mnist/ claims that it
contains the dataset referenced in LeCun et al. (1998), we assume that this
version of the dataset is the authoritative source. Notably, neither LeCun’s
website nor LeCun et al. (1998) reference Bottou et al. (1994). We
additionally want to note that according to Yadav and Bottou (2019, p.3)
“Although [Bottou et al. (1994)] was the first paper mentioning MNIST, the
creation of the dataset predates this benchmarking effort by several months.”
The dataset was already fully established when the 1994 paper was written
(Yadav and Bottou, 2019, p.3 footnote 2). Furthermore, parts of the dataset
appear in other papers that predate the paper from 1994 (Denker et al., 1988;
LeCun et al., 1990). Consequently, Yadav and Bottou (2019) state that the
exact origin of the MNIST dataset remains unclear.

The categories of the dataset are constituted by forms filled out by the
Census Bureau employees and the students. An example form is shown on the
corresponding NIST dataset site888https://www.nist.gov/srd/nist-special-
database-19 or in Grother and Hanaoka (1995). Since this data was gathered in
a highly controlled environment, the labels are much more reliable than the 20
Newsgroups dataset (Grother and Hanaoka, 1995). However, since the original
form fields contained number sequences, each of these sequences had to be
split to obtain a dataset of single digits. The algorithm by which the digit
sequence was broken down and preprocessed into single digits could not be
reverse-engineered by a team member from the original 1994 paper under the
consultation of several other team members (Yadav and Bottou, 2019). Further,
the authors discovered bugs that resulted, for example, in a pixel shift for
certain digits that impacted the model’s testing error when corrected (Yadav
and Bottou, 2019).

The results from the review in Table 2 are more promising compared to the
results for the 20 Newsgroups dataset. Usually, the authors indirectly
reference the single dataset version on LeCun’s website (Baldominos et al.,
2019; Cohen et al., 2017; Castro et al., 2019). Only once in Yadav and Bottou
(2019), the earlier 1994 paper is mentioned, which, however, did not publish
the used dataset version (Bottou et al., 1994). Additionally, the actual
contents of the dataset are discussed more frequently compared to the 20
Newsgroups dataset and example images are shown (Baldominos et al., 2019;
Kadam et al., 2020; Yadav and Bottou, 2019). Castro et al. (2019) only
describe abstractly what the MNIST dataset is and mainly rely on the
references LeCun et al. (1998) and Yadav and Bottou (2019). With Yadav and
Bottou (2019) and Baldominos et al. (2019), two papers discuss specifics of
the dataset. Nevertheless, papers often fail to discuss in detail why they
find the MNIST dataset appropriate for evaluating their model (Castro et al.,
2019; Garin and Tauzin, 2019; Saadna et al., 2019). If the scope of a paper is
limited to the usage of the MNIST dataset specifically, this may be not as
important as for a more general use case. Nevertheless, it would be important
to discuss, why this limitation leads to relevant results.

In summary, the MNIST dataset is better documented compared to the 20
Newsgroups dataset, even though some details must be considered lost in
history. Also, the labels are more reliable since they were specifically
captured by government employees and high school students in a dedicated form.

##  4\. Methods for Interrogating Datasets

We have exemplified that recent papers frequently omit the discussion of
whether a dataset is suitable for evaluating a model. In the case of the 20
Newsgroups dataset, none of the considered papers critically discusses the
usage of the dataset. In the case of the MNIST dataset, Yadav and Bottou
(2019) and Baldominos et al. (2019) provide such a critical discussion, but in
recent papers this discussion is either not reflected, e.g., in Kadam et al.
(2020) or Cheng et al. (2020), or only briefly reflected, e.g., in Castro et
al. (2019). In our opinion, this lack of discussion is often induced by
relying on the standardness of a dataset. Instead, a dataset should be
carefully evaluated and not used uncritically. For this evaluation, we now
suggest to adopt two methods: Grounded Theory and Hypothesis Testing through
Visualization. We demonstrate these methods on the 20 Newsgroups dataset and
the MNIST dataset.

###  4.1. Grounded Theory

Grounded Theory is a qualitative method introduced to sociological research by
Glaser and Strauss (2017) and recently also discussed in computer science
research (Diehl et al., 2022). Grounded Theory entails, among others, Close
Reading, Coding, and Theoretical Sampling of datasets and literature (Engler
and Stausberg, 2011; Diehl et al., 2022). Close Reading describes the process
of looking at data points in detail by engaging in manual evaluation (Diehl et
al., 2022). Coding describes the process of mapping investigated data points
to a predefined set of descriptive categories (Engler and Stausberg, 2011).
Theoretical Sampling, in contrast to random sampling, is the process of
acquiring more data based on the current form of one’s theory (Engler and
Stausberg, 2011). This process is often not linear and requires readjustments
of one’s theory so that it becomes compatible with the newly found data. This
process is reiterated until no new knowledge emerges and, as such, the theory
reaches so called Theoretical Saturation (Diehl et al., 2022).

⬇

25 An Introduction to Atheism

26 by mathew<EMAIL_ADDRESS>
27

28This article attempts to provide a general introduction to atheism. Whilst I

29have tried to be as neutral as possible regarding contentious issues, you

30should always remember that this document represents only one viewpoint. I

31would encourage you to read widely and draw your own conclusions; some

32relevant books are listed in a companion article.

33

34To provide a sense of cohesion and progression, I have presented this
article

35as an imaginary conversation between an atheist and a theist. All the

36questions asked by the imaginary theist are questions which have been
cropped

37up repeatedly on alt.atheism since the newsgroup was created. Some other

38frequently asked questions are answered in a companion article.

39

40Please note that this article is arguably slanted towards answering
questions

41posed from a Christian viewpoint. This is because the FAQ files reflect

42questions which have actually been asked, and it is predominantly Christians

43who proselytize on alt.atheism.

44

45So when I talk of religion, I am talking primarily about religions such as

46Christianity, Judaism and Islam, which involve some sort of superhuman
divine

47being. Much of the discussion will apply to other religions, but some of it

48may not.

49

50"What is atheism?"

Figure 2. Document 51 060 of the category alt.atheism. The text contains an
article about atheism and therefore has a direct link to the category label.

⬇

15>Why is it more reasonable than the trend towards obesity and the trend
towards

16>depression? You can’t just pick your two favorite trends, notice a
correlation

17>in them, and make a sweeping statement of generality. I mean, you CAN, and

18>people HAVE, but that does not mean that it is a valid or reasonable
thesis.

19>At best it’s a gross oversimplification of the push-pull factors people

20>experience.

21

22I agree, I reckon it’s television and the increase in fundamentalism.. You

23think its the increase in pre-marital sex... others thinks its because

24psychologists have taken over the criminal justice system and let violent

25criminals con them into letting them out into the streets... others think

26it’s the increase in designer drugs... others think it’s a communist plot.

27Basically the social interactions of all the changing factors in our society

28are far too complicated for us to control. We just have to hold on to the

29panic handles and hope that we are heading for a soft landing. But one

30things for sure, depression and the destruction of the nuclear family is not

31due solely to sex out of marriage.

Figure 3. Document 51 194 of the class label alt.atheism. The text is part of
a discussion about whether the growing number of people identifying as
atheists is correlated to the growing number of depression cases.

#### 20 Newsgroups dataset case.

Studies such as Asim et al. (2019), Chandra (2019), and Chen and Dai (2021)
use the 20 Newsgroups dataset for text classification tasks, consequently
implicitly assuming that it constitutes an applicable representative of text
datasets. Thus, we assume in the following that documents of the same label
belong to the same category, which would be reflected by such texts being
already similar on the surface. However, this might not be the case. To test
the assumption, we apply Close Reading to some samples with the
lexicographically first class label of the 20 Newsgroups, namely alt.atheism.
Derived from the name of the class, we expect that this label represents the
category of atheism. For the sake of the argument, let’s assume that we
randomly collect document 51 060 (shown in Figure 2) and find that it contains
an introductory article about atheism. Therefore, it is safe to code this
document as being concerned with atheism. Then, we use Theoretical Sampling to
acquire further samples of the label alt.atheism. Because, the theory is
concerned with all samples of this label, we can use an arbitrary sampling
method in this case. We proceed with Coding the next (lexicographically
ordered) 100 samples. During Coding we obtain document 51 194 (shown in Figure
3) and find that it is hardly connected to the category of atheism and,
therefore, can not be coded as being part of the category of atheism. Now, we
have two options: (1) Based on the current theory that class labels and
categories are linked, we could exclude this sample from the category of
atheism. This would require the introduction of a new label for this sample.
(2) Otherwise, we have the option to refine the theory (that documents of the
same label belong to the same category) to maintain the same label for both
samples. Given that no reviewed work introduces new categories or codes (see
Table 1), we opt for refining the theory and investigate how document 51 060
and document 51 194 can possibly belong to the same category. By manually
reverse engineering the corresponding mailing list thread, we find that
document 51 194 is part of a discussion about if and if true how the growing
number of people identifying as atheists is correlated with the growing number
of depression cases. This broader context explains the label of the document.
However, as this context, which is not included in the dataset, is necessary
for understanding how the document is connected to the atheism discourse in
the 80s, the out-of-context document can not be considered as belonging to the
category of atheism. Thus, an ML model trained on the dataset would not learn
anything about atheism as understood in 2015 or later, which is the time
covered by our literature review.

Further Coding, then, strengthens the theory that in 2015 the documents do not
constitute a coherent category any more. After manually Coding the
lexicographically first 100 documents in 11 codes from which only one
represents the category of atheism999Dataset available with doi:
10.5281/zenodo.8337723, we find that only 26 of the documents fit the category
of atheism. Overall, from today’s point of view, the only connection between
the documents appears to be that they were posted to the same mailing list and
not that they belong to the same category with respect to the text content.
Therefore, any model that is trained with the implicit assumption that the
label matches the category consequently produces meaningless results and it
has to be doubted that the performance of an ML model for text classification
on this “standard dataset” is a good indicator for its general quality. On the
positive side, we found out that the used language in the documents is very
diverse and heterogeneous with some documents using informal, short, and
heated language and others using very formal and long-winded language which
can be considered as real noise due to its independence from the topic of
atheism.

Figure 4. Some examples from the lexicographically first label representing
the _0_ category from the MNIST dataset. All investigated samples could be
coded by us as representing zero even if some artifacts are present, as can be
seen in the last example.

#### MNIST dataset case.

We apply the same method as before, starting again with the proposition that
the MNIST dataset is a good representation of a computer vision dataset,
suitable for benchmark studies on classification of image data, as done by
Chen et al. (2019a), Garg et al. (2019), and Karayaneva and Hintea (2018). We
again investigate the lexicographically first label, representing the category
_0_ by name. Consequently, we would expect that all images represent zeros,
meaning that the category would match the label. We code the first 100
lexicographically ordered samples with label _0_ , by deciding for each image,
whether it actually shows a zero or not (see examples in Figure 4). For all
cases we can identify the image content as zero, with only rarely encountering
minor artifacts, e.g., in the second to last example image in Figure 4 (image
451) or the last zero (image 34 486), which looks unusual but is still
recognizable as a zero. Since all investigated samples can be coded to the
proposed category, we can maintain the proposition that the (class) label _0_
matches the category _0_.

###  4.2. Hypotheses testing through visualization

As Grounded Theory is a qualitative method by nature, it requires manual
investigation per datapoint (Diehl et al., 2022). This is not feasible for the
majority of data points in case of large datasets. Therefore, we complement
this approach with an additional method that employs a quantitative technique
to strengthen (or counter) the findings from the previous section. Statistical
hypothesis testing is an established quantitative method in different research
areas (Emmert-Streib and Dehmer, 2019). It starts from a so-called null
hypothesis with a qualitative proposition and subsequently examines available
data to find out, whether the null hypothesis has to be rejected, assuming a
certain quantiative distribution. We want to take up this method as a analogy
to propose Hypothesis Testing through Visualization, situated in the broader
context of VIS4GT (Visualization for Grounded Theory) as introduced by Diehl
et al. (2022). Based on a quantitative visualization technique of the data, we
examine whether the data points follow the expectations with respect to the
hypothesis. This mirrors the assumed quantitative distribution from the
classical hypothesis testing. We argue that a purely qualitative hypothesis
(obtained via Grounded Theory) is required first to mitigate the risk that the
subsequent interrogation of the dataset is distorted by the quantitative
visualization technique. Similar to a null hypothesis, we initially choose the
opposite of the hypothesis we have found in the previous section (namely, that
there is a direct match between class labels and categories). For
visualization, we choose DR into a two-dimensional scatter plot. We follow two
recent recommendations for best practices regarding selection of DR techniques
(Espadoto et al., 2019; Atzberger et al., 2023). Atzberger et al. (2023)
focused on text data while Espadoto et al. (2019) employed a more general
approach. Therefore, we use the guidelines of Atzberger et al. (2023) for the
20 Newsgroups dataset (see Figure 5 and Figure 6) and the guidelines of
Espadoto et al. (2019) for the MNIST dataset (see Figure 7). While this
approach may obtain a sub-optimal but still “good enough” layout, it mitigates
the risk of over-optimization (Sedlmair et al., 2013). Otherwise, the
interrogation may be not generalizable according to the NFL (Adam et al.,
2019).

Figure 5. The 20 Newsgroups dataset was reduced with t-SNE and LSI as proposed
by Atzberger et al. (2023). The documents 51 060, 51 194, 52 910, and 53 449
are highlighted. They are wide-spread in the visualization and are also
semantically dissimilar.

Figure 6. The 20 Newsgroups dataset was reduced with t-SNE and LDA as proposed
by Atzberger et al. (2023). The documents 51 060, 51 194, and 52 910, and 53
449 are highlighted. They are wide-spread in the visualization and are also
semantically dissimilar.

Figure 7. An example layout generated according to the guidelines of Espadoto
et al. (2019). The MNIST dataset was reduced with t-SNE. The images 1, 21,
451, and 34 486 of category _0_ are highlighted. Even the noisy image 451
clearly belongs to a visually well defined cluster and is semantically
similar.

⬇

23 Overview

24

25Welcome to alt.atheism and alt.atheism.moderated.

26

27This is the first in a series of regular postings aimed at new readers of
the

28newsgroups.

29

30Many groups of a ’controversial’ nature have noticed that new readers often

31come up with the same questions, mis-statements or misconceptions and post

32them to the net. In addition, people often request information which has

33been posted time and time again. In order to try and cut down on this, the

34alt.atheism groups have a series of five regular postings under the
following

35titles:

36

37 1. Alt.Atheism FAQ: Overview for New Readers

38 2. Alt.Atheism FAQ: Introduction to Atheism

39 3. Alt.Atheism FAQ: Frequently Asked Questions (FAQ)

40 4. Alt.Atheism FAQ: Constructing a Logical Argument

41 5. Alt.Atheism FAQ: Atheist Resources

42

43This is article number 1. Please read numbers 2 and 3 before posting. The

44others are entirely optional.

45

46If you are new to Usenet, you may also find it helpful to read the newsgroup

47news.announce.newusers. The articles titled "A Primer on How to Work With

48the Usenet Community", "Answers to Frequently Asked Questions about Usenet"

49and "Hints on writing style for Usenet" are particularly relevant. Questions

50concerning how news works are best asked in news.newusers.questions.

51

52If you are unable to find any of the articles listed above, see the "Finding

53Stuff" section below.

Figure 8. Document 52 910 has the maximum distance to the supporting document
51 060 after executing t-SNE and LSI as proposed by (Atzberger et al., 2023).

⬇

11In article<EMAIL_ADDRESS><EMAIL_ADDRESS>(Mats
Andtbacka) writes:

12>In<EMAIL_ADDRESS>mathew writes:

13>> I wonder if Noam Chomsky is reading this?

14>

15> I could be wrong, but is he actually talking about outright

16>_government_ control of the media, aka censorship?

17>

18> If he doesn’t, any quick one-stop-shopping reference to his works

19>that’ll tell me, in short, what he _does_ argue for?

20

21"Manufacturing Consent," a film about the media. You alternative movie
source

22may have this; or to book it in your local alternative theatre, contact:

23

24FILMS TRANSIT * INTERNATIONAL SALES

25Jan Rofekamp

26402 Notre Dame E.

27Montreal, Quebec

28Canada H2Y 1C8

29Tel (514) 844-3358 * Fax (514) 844-7298

30Telex 5560074 Filmtransmtl

31

32(US readers: call Zeitgeist Films at 212 274 1989.)

Figure 9. Document 53 449 has the maximum distance to the supporting document
51 060 after executing t-SNE and LDA as proposed by (Atzberger et al., 2023).

#### 20 Newsgroups dataset case.

We now want to find out, whether we should reject the theory that the 20
Newsgroups dataset is a good representation for learning concepts from textual
data, by employing Hypothesis Testing through Visualization. As advised by
Atzberger et al. (2023), we used Latent Semantic Indexing (LSI) (Deerwester et
al., 1990) and Latent Dirichlet Allocation (LDA) (Blei et al., 2003) for topic
modeling and t-stochastic neighbor embedding (t-SNE) (van der Maaten and
Hinton, 2008) for DR. We choose document 51 060 again as a first
representative supporting the proposition informed by the previous
interrogation. Subsequently, we apply the guidelines from Atzberger et al.
(2023), resulting in a two-dimensional representation suitable for
investigating the neighborhood of the currently considered supporting document
51 060. The resulting scatter plots are rather convoluted (see Figure 5 and
Figure 6), as we would expect from the results of the previous section, namely
that texts with the same label are actually only loosely connected with
respect to their category, if connected at all. We then observe that the
points representing samples from the label alt.atheism are widespread, again
suggesting a loosely defined category. Investigating the immediate
neighborhood of document 51 060, we find document 51 122 as the nearest
document using the LDA model and document 52 499 for the respective LSI model.
A manual investigation reveals that those documents are indeed quite well
related to the category and therefore strengthen the proposition. However,
document 51 194, which also previously countered the hypothesis, is not in the
neighborhood of document 51 060, but over 200 times farther away than the
respective closest document mentioned above (according to euclidian distance
when using LSI). On investigation of the documents with the label alt.atheism
that are the farthest away from document 51 060 (document 52 910 for LSI and
document 53 449 for LDA), we find that their contents are indeed dissimilar
from the considered document, as shown in Figure 8 and Figure 9. Document 52
910 is an FAQ for the mailing list and is not concerned with atheism and
document 53 449 is a comment on a book by Noam Chomsky. However, the comment
itself has no link to the contemporary category of atheism. By using multiple
different topic models, as proposed by Atzberger et al. (2023), we find
different types of dissimilarities while still observing that semantically
dissimilar documents are dissimilar in both layouts. The investigated
documents counter the “Null Hypothesis” and, therefore, we have evidence that
the convoluted layout indeed corresponds to imprecise class labels. Thus, we
reject the hypothesis that the labels match the categories on the basis of
reasoning over all samples due to the employment of a quantitative DR
technique.

#### MNIST dataset case.

Espadoto et al. (2019, p.17) identify four DR techniques that work reasonably
well for several types of data, among them also t-SNE. We choose this
technique to simplify the comparison to the findings for the 20 Newsgroups
dataset in the previous paragraph. Starting from the proposition that the
MNIST dataset is not a suitable dataset for image classification due to
semantically incoherent categories (opposite to the proposition in subsection
4.1), we visualize the dataset for further investigation (see Figure 7). In
contrast to the 20 Newsgroups dataset, the cluster with label _0_ is visually
coherent. Even noisy images, such as image 451 (the second to last image in
Figure 4), are located in the dense cluster of zeros. This observation matches
the previous Coding from the Grounded Theory approach well and counters the
“Null Hypothesis”. We then investigate one of the few samples that are located
in another neighborhood, compared to the rest of the images with the label _0_
: image 34 486 (last image in Figure 4). This data point is located in the
neighborhood of the visual cluster that represent images containing a seven
rather than a zero. By manual investigation, we find that this data point is
indeed dissimilar to the other zeros we manually reviewed but still can be
coded as belonging to the category _0_. Therefore, this case represents _real_
noise that is qualitatively similar to the category but quantifiably
dissimilar in the underlying value distribution. Such desirable noise is
covered by the _Variety_ criteria of Big Data (Kitchin and McArdle, 2016).
Combined with the previous qualitative interrogation, we now have evidence
that the visual coherent clusters are not qualitatively distorted by the DR
technique. Therefore, again, we would reject the initial proposition.
Consequently, we again strengthen the previous theory that in the case of the
MNIST dataset, the labels match the categories.

##  5\. Discussion

By applying the proposed method, we have shown that the alt.atheism label of
the 20 Newsgroups dataset is imprecise. Therefore, the data associated with
this label is not suitable to learn from, especially in a meaningful fashion.
A model that reaches high accuracy on correctly predicting the alt.atheism
class label for a text has not learned a reliable understanding of the
category of atheism or any other coherent category. On the other hand, the
MNIST dataset poses no such difficulties. We found that it has some
quantifiable diversity in the class with the label _0_ , however, a
qualitative review revealed that this noise could still be subsumed under the
category _0_. Therefore, we can conclude that this dataset is a meaningful
standard dataset for the intended task of image recognition. Consequently, and
only now, we can further conclude that performance increases from models
trained on this dataset will mean an actually better solution of the use case
and that the usual optimization towards accuracy is meaningful. On a more
general level, the two provided examples show how the standardness of a
dataset can fog its actual quality, i.e., whether the labels match the
(implicitly) assumed categories. Further, data scientists may be misled to
think that this standardness implies a fit to their use case, which, however,
might have distinct requirements for categories and labels, not satisfied by
the dataset at hand. Therefore, it is not advisable to simply rely on the
standardness of a dataset, which typically only refers to its frequent use.
Omitting the discussion about why a dataset is used is only acceptable if the
dataset is proven to have a trivial match between labels and categories. In
this case, the concept of _standardness_ becomes associated with qualitative
criteria, deepening the meaning of comparisons over such a standard dataset.

As shown, the proposed approach enables the assessment of datasets regarding
use-case-specific quality criteria. Nonetheless, the approach is subject to
threats to validity. The application of Grounded Theory is time consuming
since it requires manual evaluation. Furthermore, it requires a clear
understanding of the concepts to investigate, otherwise suboptimal codes could
be chosen. In this paper, we were able to validate the concepts without domain
experts. If the use case is more complex and domain specific, we encourage the
involvement of domain experts to mitigate the risk of suboptimal Coding. A
further limitation is that for every class label a lot of samples would have
to be investigated until Theoretical Saturation is achieved. This is usually
not feasible for a dataset with at least several thousands of samples. This is
the reason why, in addition to Grounded Theory, we proposed a more
quantitative method with Hypothesis Testing through Visualization to help
overcoming this limitation by analyzing many samples at once. Unfortunately,
every DR for a non-trivial dataset leads to some sort of distortion of the
high-dimensional structure (Jeon et al., 2023). Therefore the observed
patterns in the two-dimensional plane may not correspond to patterns in the
original high-dimensional space. We mitigate this risk by using the result of
large benchmark studies (Atzberger et al., 2023; Espadoto et al., 2019) and
combining it with the results from the Grounded Theory approach. This way, the
qualitative approach mitigates some of the risks of the quantitative approach
and vice versa (Blok and Pedersen, 2014). The resulting quali-quantitative
approach enables to reveal evidence which would remain hidden by a purely
quantitative approach or not reasonably obtainable in time by a purely
qualitative approach.

##  6\. Conclusions

We observed that the standardness of a dataset is often derived from its usage
frequency. By further associating the match between use case, derived
categories, and class labels with the standardness of a dataset, one can
obtain more trustworthy and meaningful results from an ML model. However, the
degree of this match is often not discussed, as we demonstrated by reviewing
recent literature. We encourage researchers to be more critical with their use
of standard datasets and assess the quality of the dataset with manual
methods. For this, we provided a quali-quantiative approach to dataset
interrogation and demonstrated its application on two “standard” datasets.

As future work, we want to extend this method to enable dataset refinement
(Chatzimparmpas et al., 2023). For example, data scientists may want to
introduce new labels to their datasets, e.g., by the use of label propagation
approaches in low dimensional spaces (Benato et al., 2021), approaches to
incorporate domain knowledge into a DR (Li and Zhou, 2023), or reverse
engineering and correction of assumptions that were prevalent during gathering
of a (standard) dataset (Yadav and Bottou, 2019). Additionally, we want to
extend our hypothesis testing analogy by providing an analogon to the p-value.
We envision such a p-value as another step towards the combination of human
reasoning with the abstraction potential provided by artificial intelligence
and visualization methods.

###### Acknowledgements.

This work was partially funded by the German Federal Ministry for Education
and Research through grant 01IS22062 (“AI research group FFS-AI”).

## References

  * (1)
  * Adam et al. (2019) Stavros P. Adam, Stamatios-Aggelos N. Alexandropoulos, Panos M. Pardalos, and Michael N. Vrahatis. 2019.  No Free Lunch Theorem: A Review.  In _Approximation and Optimization: Algorithms, Complexity and Applications_. Springer, Cham, 57–82.  https://doi.org/10.1007/978-3-030-12767-1_5
  * Albishre et al. (2015) Khaled Albishre, Mubarak Albathan, and Yuefeng Li. 2015\.  Effective 20 Newsgroups Dataset Cleaning. In _Proc. IEEE/WIC/ACM International Conference on Web Intelligence and Intelligent Agent Technology_ _(WI-IAT ’15, Vol. 3)_. IEEE, Singapore, 98–101.  https://doi.org/10.1109/WI-IAT.2015.90
  * Asim et al. (2019) Muhammad Nabeel Asim, Muhammad Usman Ghani Khan, Muhammad Imran Malik, Andreas Dengel, and Sheraz Ahmed. 2019.  A Robust Hybrid Approach for Textual Document Classification. In _Proc. International Conference on Document Analysis and Recognition_ _(ICDAR ’19)_. IEEE, Sydney, 1390–1396.  https://doi.org/10.1109/ICDAR.2019.00224
  * Atzberger et al. (2023) Daniel Atzberger, Tim Cech, Willy Scheibel, Matthias Trapp, Rico Richter, Jürgen Döllner, and Tobias Schreck. 2023.  Large-Scale Evaluation of Topic Models and Dimensionality Reduction Methods for 2D Text Spatialization.  https://doi.org/10.48550/arXiv.2307.11770 arXiv:2307.11770 [cs.CL]  To be published at IEEE VIS 2023.
  * Baker and Potts (2013) Paul Baker and Amanda Potts. 2013.  “Why do white people have thin lips?” Google and the perpetuation of stereotypes via auto-complete search forms.  _Critical Discourse Studies_ 10, 2 (2013), 187–204.  https://doi.org/10.1080/17405904.2012.744320
  * Baldominos et al. (2019) Alejandro Baldominos, Yago Saez, and Pedro Isasi. 2019.  A Survey of Handwritten Character Recognition with MNIST and EMNIST.  _Applied Sciences_ 9, 15, Article 3169 (2019), 16 pages.  https://doi.org/10.3390/app9153169
  * Benato et al. (2021) Bárbara C. Benato, Jancarlo F. Gomes, Alexandru C. Telea, and Alexandre Xavier Falcão. 2021\.  Semi-supervised Deep Learning Based on Label Propagation in a 2D Embedded Space. In _Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications_. Springer, Cham, 371–381.  https://doi.org/10.1007/978-3-030-93420-0_35
  * Blei et al. (2003) David M. Blei, Andrew Y. Ng, and Michael I. Jordan. 2003\.  Latent Dirichlet Allocation.  _Journal of Machine Learning Research_ 3 (2003), 993–1022.
  * Blok and Pedersen (2014) Anders Blok and Morten Axel Pedersen. 2014.  Complementary social science? Quali-quantitative experiments in a Big Data world.  _Big Data & Society_ 1, 2 (2014), 6 pages.  https://doi.org/10.1177/2053951714543908
  * Bodrunova et al. (2020) Svetlana S. Bodrunova, Andrey V. Orekhov, Ivan S. Blekanov, Nikolay S. Lyudkevich, and Nikita A. Tarasov. 2020.  Topic Detection Based on Sentence Embeddings and Agglomerative Clustering with Markov Moment.  _Future Internet_ 12, 9, Article 144 (2020), 17 pages.  https://doi.org/10.3390/fi12090144
  * Bottou et al. (1994) L. Bottou, C. Cortes, J.S. Denker, H. Drucker, I. Guyon, L.D. Jackel, Y. LeCun, U.A. Muller, E. Sackinger, P. Simard, and V. Vapnik. 1994.  Comparison of classifier methods: a case study in handwritten digit recognition. In _Proc. 12th IAPR International Conference on Pattern Recognition, Vol. 3 – Conference C: Signal Processing (Cat. No.94CH3440-5)_ , Vol. 2. IEEE, Jerusalem, 77–82 Vol.2.  https://doi.org/10.1109/ICPR.1994.576879
  * Bucher (2012) Taina Bucher. 2012\.  Want to be on the top? Algorithmic power and the threat of invisibility on Facebook.  _New media & society_ 14, 7 (2012), 1164–1180.  https://doi.org/10.1177/1461444812440159
  * Castro et al. (2019) Daniel C Castro, Jeremy Tan, Bernhard Kainz, Ender Konukoglu, and Ben Glocker. 2019.  Morpho-MNIST: quantitative assessment and diagnostics for representation learning.  _Journal of Machine Learning Research_ 20, 178 (2019), 1–29.
  * Cech et al. (2023) Tim Cech, Furkan Simsek, Willy Scheibel, and Jürgen Döllner. 2023\.  A Dashboard for Interactive Convolutional Neural Network Training And Validation Through Saliency Maps. In _EuroVis 2023 – Posters_. The Eurographics Association, Leipzig, 3 pages.  https://doi.org/10.2312/evp.20231054
  * Chandra (2019) Andreas Chandra. 2019\.  Comparison of Feature Selection for Imbalance Text Datasets. In _Proc. International Conference on Information Management and Technology_ _(ICIMTech ’19, Vol. 1)_. IEEE, Jakarta/Bali, 68–72.  https://doi.org/10.1109/ICIMTech.2019.8843773
  * Chatzimparmpas et al. (2023) A. Chatzimparmpas, F. V. Paulovich, and A. Kerren. 2023\.  HardVis: Visual Analytics to Handle Instance Hardness Using Undersampling and Oversampling Techniques.  _Computer Graphics Forum_ 42, 1 (2023), 135–154.  https://doi.org/10.1111/cgf.14726
  * Chen and Dai (2021) Chuanshuai Chen and Jiazhu Dai. 2021.  Mitigating backdoor attacks in LSTM-based text classification systems by Backdoor Keyword Identification.  _Neurocomputing_ 452 (2021), 253–262.  https://doi.org/10.1016/j.neucom.2021.04.105
  * Chen et al. (2019a) Feiyang Chen, Nan Chen, Hanyang Mao, and Hanlin Hu. 2019a.  Assessing four Neural Networks on Handwritten Digit Recognition Dataset (MNIST).  https://doi.org/10.48550/arXiv.1811.08278 arXiv:1811.08278 [cs.CV]
  * Chen et al. (2019b) Pengfei Chen, Ben Ben Liao, Guangyong Chen, and Shengyu Zhang. 2019b.  Understanding and Utilizing Deep Neural Networks Trained with Noisy Labels. In _Proc. 36th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 97)_ , Kamalika Chaudhuri and Ruslan Salakhutdinov (Eds.). PMLR, Long Beach, 1062–1070.
  * Cheng et al. (2020) Keyang Cheng, Rabia Tahir, Lubamba Kasangu Eric, and Maozhen Li. 2020.  An analysis of generative adversarial networks and variants for image synthesis on MNIST dataset.  _Multimedia Tools and Applications_ 79 (2020), 13725–13752.  https://doi.org/10.1007/s11042-019-08600-2
  * Cohen et al. (2017) Gregory Cohen, Saeed Afshar, Jonathan Tapson, and André van Schaik. 2017.  EMNIST: Extending MNIST to handwritten letters. In _2017 International Joint Conference on Neural Networks_ _(IJCNN ’17)_. IEEE, Anchorage, 2921–2926.  https://doi.org/10.1109/IJCNN.2017.7966217
  * Davidson (2015) Kathryn Davidson. 2015\.  Quotation, demonstration, and iconicity.  _Linguistics and Philosophy_ 38, 6 (2015), 477–520.  https://doi.org/10.1007/s10988-015-9180-1
  * Deerwester et al. (1990) Scott Deerwester, Susan T Dumais, George W. Furnas, Thomas K. Landauer, and Richard Harshman. 1990\.  Indexing by Latent Semantic Analysis.  _Journal of the American Society for Information Science_ 41, 6 (1990), 391–407.  https://doi.org/10.1002/(SICI)1097-4571(199009)41:6<391::AID-ASI1>3.0.CO;2-9
  * Denker et al. (1988) John Denker, W. Gardner, Hans Graf, Donnie Henderson, R. Howard, W. Hubbard, L. D. Jackel, Henry Baird, and Isabelle Guyon. 1988.  Neural Network Recognizer for Hand-Written Zip Code Digits.  _Advances in Neural Information Processing Systems_ 1 (1988), 323–331.
  * Desai et al. (2022) Jules Desai, David Watson, Vincent Wang, Mariarosaria Taddeo, and Luciano Floridi. 2022\.  The epistemological foundations of data science: a critical review.  _Synthese_ 200, 6 (2022), 27 pages.  https://doi.org/10.1007/s11229-022-03933-2
  * Diehl et al. (2022) Alexandra Diehl, Alfie Abdul-Rahman, Benjamin Bach, Mennatallah El-Assady, Matthias Kraus, Robert S. Laramee, Daniel A. Keim, and Min Chen. 2022.  Characterizing Grounded Theory Approaches in Visualization.  _arXiv CoRR_ cs.HC, 2203.01777 (2022), 13 pages.  https://doi.org/10.48550/arXiv.2203.01777
  * Elish and danah boyd (2018) M. C. Elish and danah boyd. 2018.  Situating methods in the magic of Big Data and AI.  _Communication Monographs_ 85, 1 (2018), 57–80.  https://doi.org/10.1080/03637751.2017.1375130
  * Emmert-Streib and Dehmer (2019) Frank Emmert-Streib and Matthias Dehmer. 2019.  Understanding Statistical Hypothesis Testing: The Logic of Statistical Inference.  _Machine Learning and Knowledge Extraction_ 1, 3 (2019), 945–961.  https://doi.org/10.3390/make1030054
  * Engler and Stausberg (2011) Steven Engler and Michael Stausberg. 2011.  Grounded theory.  In _The Routledge Handbook of Research Methods in the Study of Religion_. Routledge, London, 256–274.
  * Espadoto et al. (2019) Mateus Espadoto, Rafael M Martins, Andreas Kerren, Nina ST Hirata, and Alexandru C Telea. 2019.  Toward a quantitative survey of dimension reduction techniques.  _IEEE Transactions on Visualization and Computer Graphics_ 27, 3 (2019), 2153–2173.  https://doi.org/10.1109/TVCG.2019.2944182
  * Garg et al. (2019) Adhesh Garg, Diwanshi Gupta, Sanjay Saxena, and Parimi Praveen Sahadev. 2019.  Validation of Random Dataset Using an Efficient CNN Model Trained on MNIST Handwritten Dataset. In _Proc. 6th International Conference on Signal Processing and Integrated Networks_ _(SPIN ’19)_. IEEE, Noida, 602–606.  https://doi.org/10.1109/SPIN.2019.8711703
  * Garin and Tauzin (2019) Adélie Garin and Guillaume Tauzin. 2019.  A Topological “Reading” Lesson: Classification of MNIST using TDA. In _2019 18th IEEE International Conference On Machine Learning And Applications_ _(ICMLA)_. IEEE, Boca Raton, 1551–1556.  https://doi.org/10.1109/ICMLA.2019.00256
  * Giustini and Boulos (2013) Dean Giustini and Maged N Kamel Boulos. 2013.  Google Scholar is not enough to be used alone for systematic reviews.  _Online Journal of Public Health Informatics_ 5, 2 (2013), 1–10.  https://doi.org/10.5210/ojphi.v5i2.4623 PMID: 23923099.
  * Glaser and Strauss (2017) Barney G. Glaser and Anselm L. Strauss. 2017.  _Discovery of grounded theory: Strategies for qualitative research_.  Routledge, London.  https://doi.org/10.4324/9780203793206
  * Grother and Hanaoka (1995) Patrick J Grother and Kayee K Hanaoka. 1995.  _NIST special database 19-hand-printed forms and characters database_.  Technical Report. National Institute of Standards and Technology.  https://doi.org/10.18434/T4H01C
  * Hsu (2020) Bi-Min Hsu. 2020\.  Comparison of Supervised Classification Models on Textual Data.  _Mathematics_ 8, 5, Article 851 (2020), 16 pages.  https://doi.org/10.3390/math8050851
  * Introna and Nissenbaum (2000) L. Introna and H. Nissenbaum. 2000.  Defining the Web: the politics of search engines.  _Computer_ 33, 1 (2000), 54–62.  https://doi.org/10.1109/2.816269
  * Jeon et al. (2023) Hyeon Jeon, Yun-Hsin Kuo, Michaël Aupetit, Kwan-Liu Ma, and Jinwook Seo. 2023.  Classes are not Clusters: Improving Label-based Evaluation of Dimensionality Reduction.  https://doi.org/10.48550/arXiv.2308.00278 To be published at IEEE VIS 2023.
  * Joachims (1996) Thorsten Joachims. 1996\.  _A Probabilistic Analysis of the Rocchio Algorithm with TFIDF for Text Categorization_.  Technical Report CMU-CS-96-118. School of Computer Science, Carnegie-Mellon University, Pittsburgh, USA.
  * Kadam et al. (2020) Shivam S Kadam, Amol C Adamuthe, and Ashwini B Patil. 2020\.  CNN model for image classification on MNIST and fashion-MNIST dataset.  _Journal of scientific research_ 64, 2 (2020), 374–384.
  * Karayaneva and Hintea (2018) Yordanka Karayaneva and Diana Hintea. 2018.  Object recognition in python and MNIST dataset modification and recognition with five machine learning classifiers.  _Journal of Image and Graphics_ 6, 1 (2018), 10–20.  https://doi.org/10.18178/joig.6.1.10-20
  * Kerenidis and Luongo (2020) Iordanis Kerenidis and Alessandro Luongo. 2020.  Classification of the MNIST data set with quantum slow feature analysis.  _Physical Review A_ 101, 6 (2020), 12 pages.  https://doi.org/10.1103/physreva.101.062327
  * Kitchin and McArdle (2016) Rob Kitchin and Gavin McArdle. 2016.  What makes Big Data, Big Data? Exploring the ontological characteristics of 26 datasets.  _Big Data & Society_ 3, 1 (2016), 10 pages.  https://doi.org/10.1177/2053951716631130
  * Kou et al. (2020) Gang Kou, Pei Yang, Yi Peng, Feng Xiao, Yang Chen, and Fawaz E. Alsaadi. 2020\.  Evaluation of feature selection methods for text classification with small datasets using multiple criteria decision-making methods.  _Applied Soft Computing_ 86, 105836 (2020), 14 pages.  https://doi.org/10.1016/j.asoc.2019.105836
  * Lang (1995) Ken Lang. 1995.  NewsWeeder: Learning to Filter Netnews. In _Machine Learning Proceedings 1995_. Morgan Kaufmann, Tahoe City, 331–339.  https://doi.org/10.1016/B978-1-55860-377-6.50048-7
  * Lazer et al. (2014) David Lazer, Ryan Kennedy, Gary King, and Alessandro Vespignani. 2014.  The parable of Google Flu: traps in big data analysis.  _science_ 343, 6176 (2014), 1203–1205.  https://doi.org/10.1126/science.1248506
  * LeCun et al. (1998) Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. 1998.  Gradient-based learning applied to document recognition.  _Proc. IEEE_ 86, 11 (1998), 2278–2324.  https://doi.org/10.1109/5.726791
  * LeCun et al. (1990) Y. LeCun, O. Matan, B. Boser, J.S. Denker, D. Henderson, R.E. Howard, W. Hubbard, L.D. Jacket, and H.S. Baird. 1990.  Handwritten zip code recognition with multilayer networks. In _Proceedings of 10th International Conference on Pattern Recognition_ , Vol. 2. IEEE, Atlantic City, 35–40.  https://doi.org/10.1109/ICPR.1990.119325
  * Lenglet (2014) Marc Lenglet. 2014\.  Algorithms and the manufacture of financial reality.  In _Objects and Materials_. Routledge, London, 312–322.
  * Lewis et al. (2013) Seth C. Lewis, Rodrigo Zamith, and Alfred Hermida. 2013\.  Content Analysis in an Era of Big Data: A Hybrid Approach to Computational and Manual Methods.  _Journal of Broadcasting & Electronic Media_ 57, 1 (2013), 34–52.  https://doi.org/10.1080/08838151.2012.761702
  * Li and Zhou (2023) Jie Li and Chun-qi Zhou. 2023.  Incorporation of Human Knowledge into Data Embeddings to Improve Pattern Significance and Interpretability.  _IEEE Transactions on Visualization and Computer Graphics_ 29, 1 (2023), 723–733.  https://doi.org/10.1109/TVCG.2022.3209382
  * Mauni et al. (2020) Humaira Zahin Mauni, Tajbia Hossain, and Raqeebir Rab. 2020\.  Classification of Underrepresented Text Data in an Imbalanced Dataset Using Deep Neural Network. In _Proc. IEEE Region 10 Symposium_ _(TENSYMP ’20)_. IEEE, Dhaka, 997–1000.  https://doi.org/10.1109/TENSYMP50017.2020.9231021
  * Mehrabi et al. (2021) Ninareh Mehrabi, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan. 2021\.  A Survey on Bias and Fairness in Machine Learning.  _Comput. Surveys_ 54, 6, Article 115 (2021), 35 pages.  https://doi.org/10.1145/3457607
  * Mittelstadt et al. (2016) Brent Daniel Mittelstadt, Patrick Allo, Mariarosaria Taddeo, Sandra Wachter, and Luciano Floridi. 2016\.  The ethics of algorithms: Mapping the debate.  _Big Data & Society_ 3, 2 (2016), 21 pages.  https://doi.org/10.1177/2053951716679679
  * Morrison et al. (2023) Katelyn Morrison, Donghoon Shin, Kenneth Holstein, and Adam Perer. 2023.  Evaluating the Impact of Human Explanation Strategies on Human-AI Visual Decision-Making.  _Proceedings of the ACM on Human-Computer Interaction_ 7, CSCW1, Article 48 (2023), 37 pages.  https://doi.org/10.1145/3579481
  * Natarajan et al. (2013) Nagarajan Natarajan, Inderjit S Dhillon, Pradeep K Ravikumar, and Ambuj Tewari. 2013.  Learning with Noisy Labels. In _Advances in Neural Information Processing Systems_ , Vol. 26. Curran Associates, Inc., Lake Tahoe, 1196–1204.
  * Neff et al. (2017) Gina Neff, Anissa Tanweer, Brittany Fiore-Gartland, and Laura Osburn. 2017.  Critique and Contribute: A Practice-Based Framework for Improving Critical Data Studies and Data Science.  _Big Data_ 5, 2 (2017), 85–97.  https://doi.org/10.1089/big.2016.0050 PMID: 28632445.
  * Northcutt et al. (2021) Curtis Northcutt, Lu Jiang, and Isaac Chuang. 2021\.  Confident learning: Estimating uncertainty in dataset labels.  _Journal of Artificial Intelligence Research_ 70 (2021), 1373–1411.  https://doi.org/10.1613/jair.1.12125
  * Pappagari et al. (2019) Raghavendra Pappagari, Piotr Zelasko, Jesús Villalba, Yishay Carmiel, and Najim Dehak. 2019\.  Hierarchical Transformers for Long Document Classification. In _Proc. IEEE Automatic Speech Recognition and Understanding Workshop_ _(ASRU ’19)_. IEEE, Singapore, 838–844.  https://doi.org/10.1109/ASRU46091.2019.9003958
  * Romanuke (2016) Vadim V. Romanuke. 2016\.  Training data expansion and boosting of convolutional neural networks for reducing the MNIST dataset error rate.  _Research Bulletin of the National Technical University of Ukraine “Kyiv Politechnic Institute”_ 3, 6 (2016), 29–34.  https://doi.org/10.20535/1810-0546.2016.6.84115
  * Ruff et al. (2019) Lukas Ruff, Yury Zemlyanskiy, Robert Vandermeulen, Thomas Schnake, and Marius Kloft. 2019\.  Self-Attentive, Multi-Context One-Class Classification for Unsupervised Anomaly Detection on Text. In _Proc. 57th Annual Meeting of the Association for Computational Linguistics_. ACL, Florence, 4061–4071.  https://doi.org/10.18653/v1/P19-1398
  * Saadna et al. (2019) Yassmina Saadna, Ali Behloul, and Saliha Mezzoudj. 2019\.  Speed limit sign detection and recognition system using SVM and MNIST datasets.  _Neural Computing and Applications_ 31 (2019), 5005–5015.  https://doi.org/10.1007/s00521-018-03994-w
  * Saigal and Khanna (2020) Pooja Saigal and Vaibhav Khanna. 2020.  Multi-category news classification using Support Vector Machine based classifiers.  _SN Applied Sciences_ 2 (2020), 458–470.  Issue 3. https://doi.org/10.1007/s42452-020-2266-6
  * Samek and Müller (2019) Wojciech Samek and Klaus-Robert Müller. 2019.  Towards Explainable Artificial Intelligence.  _Explainable AI: Interpreting, Explaining and Visualizing Deep Learning_ 1, 11700 (2019), 5–22.  https://doi.org/10.1007/978-3-030-28954-6_1
  * Seaver (2017) Nick Seaver. 2017\.  Algorithms as culture: Some tactics for the ethnography of algorithmic systems.  _Big Data & Society_ 4, 2 (2017), 12 pages.  https://doi.org/10.1177/2053951717738104
  * Sedlmair et al. (2013) Michael Sedlmair, Tamara Munzner, and Melanie Tory. 2013\.  Empirical Guidance on Scatterplot and Dimension Reduction Technique Choices.  _IEEE Transactions on Visualization and Computer Graphics_ 19, 12 (2013), 2634–2643.  https://doi.org/10.1109/TVCG.2013.153
  * Serrano-Gotarredona and Linares-Barranco (2015) Teresa Serrano-Gotarredona and Bernabé Linares-Barranco. 2015\.  Poker-DVS and MNIST-DVS. Their History, How They Were Made, and Other Details.  _Frontiers in Neuroscience_ 9 (2015), 1–10.  https://doi.org/10.3389/fnins.2015.00481
  * Singh et al. (2023) Gunjan Singh, Arpita Nagpal, and Vijendra Singh. 2023\.  Text Classification using Improved IWO-HAN.  _Procedia Computer Science_ 218 (2023), 1184–1195.  https://doi.org/10.1016/j.procs.2023.01.097 International Conference on Machine Learning and Data Engineering.
  * Symons and Alvarado (2019) John Symons and Ramón Alvarado. 2019.  Epistemic Entitlements and the Practice of Computer Simulation.  _Minds and Machines_ 29, 1 (2019), 37–60.  https://doi.org/10.1007/s11023-018-9487-0
  * Tabik et al. (2017) Siham Tabik, Daniel Peralta, and Andrés Herrera-Poyatos1 Francisco Herrera. 2017.  A snapshot of image pre-processing for convolutional neural networks: case study of MNIST.  _International Journal of Computational Intelligence Systems_ 10 (2017), 555–568.  https://doi.org/10.2991/ijcis.2017.10.1.38
  * van der Maaten and Hinton (2008) Laurens van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-SNE.  _Journal of Machine Learning Research_ 9, 11 (2008), 2579–2605.
  * Varoquaux et al. (2015) G. Varoquaux, L. Buitinck, G. Louppe, O. Grisel, F. Pedregosa, and A. Mueller. 2015\.  Scikit-Learn: Machine Learning Without Learning the Machinery.  _GetMobile: Mobile Comp. and Comm._ 19, 1 (2015), 29–33.  https://doi.org/10.1145/2786984.2786995
  * Wang et al. (2016) Daixin Wang, Peng Cui, and Wenwu Zhu. 2016.  Structural Deep Network Embedding. In _Proc. International Conference on Knowledge Discovery and Data Mining_ _(KDD ’16)_. Association for Computing Machinery, New York, 1225–1234.  https://doi.org/10.1145/2939672.2939753
  * Wang and Yang (2020) Xinyi Wang and Yi Yang. 2020.  Neural Topic Model with Attention for Supervised Learning. In _Proc. 23rd International Conference on Artificial Intelligence and Statistics_ _(Proceedings of Machine Learning Research, Vol. 108)_. PMLR, Online, 1147–1156.  https://proceedings.mlr.press/v108/wang20c.html
  * Yadav and Bottou (2019) Chhavi Yadav and Leon Bottou. 2019.  Cold Case: The Lost MNIST Digits. In _Advances in Neural Information Processing Systems_ , Vol. 32. Curran Associates, Inc., Vancouver, 13443–13452.
  * Zhang and Yamana (2021) Cheng Zhang and Hayato Yamana. 2021.  Improving Text Classification Using Knowledge in Labels. In _Proc. IEEE 6th International Conference on Big Data Analytics_ _(ICBDA)_. IEEE, Xiamen, 193–197.  https://doi.org/10.1109/ICBDA51983.2021.9403092