#  MDB: Interactively Querying Datasets and Models

Aaditya Naik
University of Pennsylvania
<EMAIL_ADDRESS>
&Adam Stein
University of Pennsylvania
<EMAIL_ADDRESS>
Yinjun Wu
University of Pennsylvania
<EMAIL_ADDRESS>
&Mayur Naik
University of Pennsylvania
<EMAIL_ADDRESS>
&Eric Wong
University of Pennsylvania
<EMAIL_ADDRESS>

###### Abstract

As models are trained and deployed, developers need to be able to
systematically debug errors that emerge in the machine learning pipeline. We
present MDB, a debugging framework for interactively querying datasets and
models. MDB integrates functional programming with relational algebra to build
expressive queries over a database of datasets and model predictions. Queries
are reusable and easily modified, enabling debuggers to rapidly iterate and
refine queries to discover and characterize errors and model behaviors. We
evaluate MDB on object detection, bias discovery, image classification, and
data imputation tasks across self-driving videos, large language models, and
medical records. Our experiments show that MDB enables up to 10x faster and
40% shorter queries than other baselines. In a user study, we find developers
can successfully construct complex queries that describe errors of machine
learning models.

##  1 Introduction

As machine learning models continue to improve, they also continue to fail in
unexpected and harmful ways. In 2018, a fatality caused by a self-driving
vehicle Wakabayashi (2018) made it clear that such failures must be taken
seriously. Other examples include state-of-the-art vision models performing
worse on people with darker skin Wilson et al. (2019), language models
producing text containing sterotypes against Muslims Abid et al. (2021), and
medical diagnosis models degrading in performance when used in new hospitals
Zech et al. (2018). As models are deployed in increasingly critical
applications like self-driving and medical diagnostics, identifying and
avoiding such behaviors is crucial to ensure performance, reliability, and
trustworthiness.

(a) Lack of object permanence in object detection for self-driving.

(b) Stereotypical bias in unsupervised language modeling.

(c) Time series imputations violating domain knowledge.

Figure 1: Debugging examples in three domains and corresponding queries in
MDB. (a) uncovers a critical error by an object detection model OneFormer Jain
et al. (2022a) in which an object occurring in three consecutive frames is
detected in the first two but not the third. The shown query identifies 76
instances of this error in a dataset of 1000 samples and 12,212 unique
objects. (b) measures bias in a Large Language Model GPT-3.5 by identifying
samples in which certain adjectives are associated with certain nouns (c)
detects imputation errors in medical time-series data by a state-of-the-art
model SAITS Du et al. (2023). The plot based on the depicted query’s results
shows how frequently the imputed values of patients’ Glasgow Coma Scale (GCS)
scores are infeasible, i.e., outside the range 3-15.

Before rectifying these “bugs” in the machine learning pipeline, one must
first _identify_ them. Ideally, such bugs would be identified before
deployment, but there are several challenges in doing so. As an example,
consider Figure 1(a), which depicts three consecutive frames of a video from
the Cityscapes self-driving dataset Cordts et al. (2016). A state-of-the-art
vision model OneFormer Jain et al. (2022a) fails to predict a pedestrian in
the third frame. This is clearly an undesirable error, but how can one
characterize this bug?

A typical strategy to represent a bug is to construct a group of examples that
contain a similar pattern Kang et al. (2018). Importantly, such patterns
should generalize to new instances of the same bug. However, the example in
Figure 1(a) represents only a single instance—additional examples are needed
to create a complete and generalizable bug description. This leads to the
first challenge: due to the scale and complexity of modern datasets, manually
sifting through individual samples to find such instances is infeasible.
Instead, we would like a simple description that can capture these instances
automatically at scale, such as “the pedestrian suddenly disappears after
previously being detected.” For example, searching 1000 video frames
containing a total of 12,212 objects for instances where an object is detected
in the first two frames but not in the third yields 76 instances of the bug in
Figure 1(a).

While high-level intuition is a useful starting point, model developers
typically do not know a priori the exact, optimal description for
characterizing a bug. This leads to the second challenge: an overly broad
description with a high false positive rate is an inaccurate characterization
of a bug. For example, the initial bug description of “objects that disappear
in the 3rd frame” turns out to also detect benign instances such as cars or
pedestrians exiting the field of vision. To reduce the number of false
positives, we would like to be able to interactively and iteratively _refine_
the initial description. For example, constraining the search for disappearing
objects to those closer to the center of the frame removes 327 false positives
from the 403 original instances of the bug from Figure 1(a).

A third challenge in characterizing bugs is the sheer diversity of data
modalities, prediction tasks, and types of bugs that can occur in practice.
Figure 1(a) depicts just one example of a bug in self-driving, which has other
bugs such as impossibly-sized objects or unrealistic object locations.
Furthermore, domains beyond vision can have radically different
characterizations of bugs. Figure 1(b) shows the responses of GPT-3.5 to two
questions which only differ in the subject (farmer versus an engineer). The
stereotype bias in the responses reveals a potential bug that could result in
harmful text generation. Figure 1(c) shows the predictions of an imputation
model for filling missing values of coma severity Silva et al. (2012) in
medical records. This model predicts severity scores that are impossible to
attain, a bug that adversely affects downstream performance Naik et al.
(2023). Ideally, we would like high expressivity: the ability to characterize
bugs in any domain for arbitrary models and datasets.

In this paper, we propose Model DeBugger (MDB), a framework to effectively
discover and characterize bugs in models and datasets that addresses the above
challenges of scalability, interactive refinement, and expressiveness. MDB
integrates functional programming with relational algebra to represent bugs as
collections of results from executing a _query_ over a database. This backend
allows MDB to automatically evaluate queries over a database, allowing
developers to quickly and scalably test potential bugs on large datasets and
models. Furthermore, MDB enables the rapid refinement of previous queries to
interactively prototype new potential bugs and reduce the false positives of
found bugs. Lastly, as illustrated in Figure 1, queries in MDB are highly
expressive and easy to write, capable of representing complex bugs in a
diverse set of domains.

We summarize the contributions of the paper:

  1. 1.

We propose database queries as a scalable, interactive, and expressive
approach to discover and characterize bugs in general machine learning tasks.

  2. 2.

We develop MDB, a framework that enables interactive debugging of models and
datasets. At its core, MDB integrates functional programming with relational
algebra to make possible the rapid iteration and refinement of database
queries.

  3. 3.

We demonstrate how MDB enables finding new bugs in a multitude of tasks
(object detection, bias discovery, image classification, and data imputation)
across a diverse set of domains (self-driving videos, large language models,
and medical records).

  4. 4.

We complement our debugging experiments with numerical evaluations showing
that MDB has faster query execution time and more concise queries than
alternative systems such as Pandas and MongoDB. We also run a user study to
validate the usability of MDB as a debugging tool.

##  2 Debugging Machine Learning

How do we design a debugger for finding bugs in datasets and models of machine
learning pipelines? Debugging is traditionally regarded as the software
engineering process for finding and resolving bugs in the source code of
programs. For example, debugging tools such as gdb for C/C++ and pdb for
Python allow programmers to inspect the code of a program while it executes to
search for bugs. When debugging a machine learning model, in principle, one
could hypothetically use such a debugger to step through the execution of a
model making predictions on examples.

However, such an exercise is futile due to the size of modern datasets and
models. Moreover, bugs in the machine learning pipeline tend to arise not from
the code, but from the data used to train the model. Models can learn critical
or systemic issues such as spurious correlations or biases from the training
data. This is increasingly the case as machine learning trends towards larger
models trained on massive datasets scraped from the web.

To be applicable to machine learning problems, a debugger must satisfy the
following criteria:

  1. 1.

Scalability: it must be able to identify bugs in large-scale machine learning
settings,

  2. 2.

Interactivity: it must support interactive testing and inspection of potential
bugs, and

  3. 3.

Expressivity: it must be able to find bugs in a diverse landscape of models,
datasets, and tasks.

Figure 2: Three queries written in MDB for iterative debugging of a vision
model. The first query is executed over the output_preds table, where each row
is an object containing the predicted labels and bounding boxes for a single
frame. To select all frames with a person detected, we use a filter operation
which keeps only the rows of output_preds that have a person in their
predicted labels.

Traditional source code debuggers are interactive, but are too low-level and
lack the scalability and expressivity needed to identify bugs in large-scale
models and datasets. Meanwhile, the machine learning community has developed
various tools for finding issues such as biases Wong et al. (2021); Singla and
Feizi (2021) or spurious correlations Salman et al. (2022), which are
typically specialized for finding particular types of bugs in specific models.
These techniques provide a fixed snapshot of model behavior that cannot be
iterated on or improved. Our goal is to develop a general-purpose debugger
that is suited for the scale and diversity of machine learning problems, yet
retaining the interactive property of traditional debuggers.

###  2.1 Debugging with Database Queries

We propose database queries as a general framework well-suited for debugging
machine learning systems. Specifically let $T\subseteq\mathcal{T}$ be a table
of objects, and let $O:\mathcal{T}\rightarrow\mathcal{T}$ be a database
operator that transforms a table $T$ to a new table $T^{\prime}=O(T)$. A basic
query $Q:\mathcal{T}\rightarrow\mathcal{T}$ is a sequence of $k$ operators,
$O_{1},\dots,O_{k}$ that can be applied to some table $T$:

$Q(T)=(O_{k}\circ\dots\circ O_{1})(T)$ (1)

When $Q(T)$ outputs a table of examples that contain a similar pattern, we
call $Q$ a _query representation_ of the pattern. If such a pattern is
considered to be a bug for the machine learning system, $Q$ is a query
representation of the bug. When the table $T$ contains information from the
machine learning system (i.e. input examples, predictions, metadata, etc.),
then the problem of finding a bug in the system is reduced to constructing a
query $Q$ that represents a bug.

We have developed a debugger called MDB that enables an interactive process
for creating queries $Q$ that represent bugs in machine learning systems. As
an overview, developers use MDB to iteratively build and refine queries $Q$,
gradually replacing and adding operations $O_{i}$ one at a time, until the
final query $Q$ represents a bonafide bug. MDB is built upon a database
backend for scalability, enables rapid re-use and adjustment of old queries
for interactivity, and supports relational operators with highly flexible
tables and lambda functions for expressivity. In practice, we further allow
tables $T$ to be generic dataset objects commonly used in deep learning
frameworks such as PyTorch. This allows our debugger to directly plug into
existing deep learning infrastructure with minimal modification. In the
remainder of this section, we elaborate on how MDB transforms concepts from
databases, functional programming, and relational algebra into a scalable,
interactive, and expressive debugger that is well-suited for finding bugs in
machine learning systems.

###  2.2 Scalability

The core of MDB, which represents bugs as queries over databases, is an
underlying engine that drives the scalable representation and inspection of
bugs. Executing queries over databases has long been seen as a gold standard
for storing, accessing, and analyzing large amounts of data. By representing
bugs as queries, MDB achieves a concise bug representation as a sequence of
$k$ database operators $(O_{1},\dots,O_{k}$) that can naturally exploit the
scalability properties of databases frameworks.

As an example, consider the task of debugging the object detection model from
Figure 1(a). In this example, we use the Cityscapes validation set, which has
15,000 video frames of self-driving data. As a starting point, a simple
initial query is to select all predictions that contain pedestrians, as frames
containing people are likely to be particularly important for safety
considerations. Specifically, we can consider running a query consisting of a
single database operation called filter on a table of object predictions per
frame. After executing this query, the resulting table is the subset of
predictions that contain at least one pedestrian. An example of this query
written with MDB is in Figure 2. This query executes the filter operation on
the predictions table, resulting in a new people table. The new table has 31%
fewer predictions, starting from 1000 down to 688 predictions.

###  2.3 Interactivity

Figure 3: The second query from Figure 2 compared to a Python implementation
with the same time complexity. Semantically similar operations are paired.

We can continue interactively refining the initial query that selects
predictions with pedestrians, since the first query has too many matches to be
a useful representation of a bug. A subsequent hypothesis to narrow the
results is to look for frames where the object detector fails to detect
people. When a person is missed, there must be two consecutive predictions
that differ in the number of people detected. Thus, a possible refinement of
the initial query is to search for pairs of predictions with differing numbers
of people.

We can construct such a query by chaining more database operators to the
original query. First, we use a slightly more complex join operator, a
database operation that combines two tables together via a common property of
each table. Specifically, we can join the table of pedestrians with the same
table, but shifted forward in time by one frame to get a new table containing
pairs of consecutive predictions. Finally, we can apply another filter
operator to select only pairs of predictions with a differing number of
pedestrians. This query is shown at the middle of Figure 2.

Importantly, this query _re-uses_ the result of the previous query, and
iteratively chains together more database operators to gradually construct a
more complex query. Interactivity is a key component of all debugging
frameworks that enables the rapid testing and adjusting of hypothesis when
searching for bugs. When executed on the people table, the refined query
results in a changing_people table with 202 pairs of predictions (a 70%
reduction from 688) that have a changing number of pedestrians.

###  2.4 Expressivity

MDB provides a high level of expressiveness via compositional database
operations that are parameterized with arbitrary lambda functions. In total,
MDB has six core operations derived from relational algebra which can be
composed to create queries. We will now introduce a few more operators (cols
and flatten), and defer the remaining two operators (order_by and group_by) to
Appendix D.

Our final goal in the running example is to identify the exact persons that
have disappeared from the object detector. This requires converting our latest
table of consecutive predictions into a table of consecutive persons. However,
since each prediction contains an unordered list of detected objects, it is
not immediately obvious which bounding boxes refer to the same person. Our
first step is to match objects between two frames. Specifically, we use a
well-developed tool from object detection for aligning the bounding boxes
between two frames (Bolya et al., 2020). We can apply this complex function to
all of our pairs of predictions using the cols database operator, which
applies a function to every row in a table to create a new table. This step
highlights how MDB seamlessly integrates relational algebra with lambda
functions from functional programming. Since all of our database operations
are parameterized with lambda functions, MDB can express a rich set of queries
over any domain.

Once we’ve aligned the objects in our table of consecutive frames, we can use
one more database operator known as flatten to convert this into a table of
consecutive objects. At a high level, flatten rearranges a table of sequences
into a single table, analogous to flattening a 2D matrix into a 1D vector.
After flattening the table, we apply one last filter to extract consecutive
objects where the first object is detected as a person but the second is not.
The final query, shown at the bottom of Figure 2, represents the bug of
disappearing people. When executed on the changed_people table, we find 133
instances of disappearing people from 202 consecutive predictions.

In this example, we used MDB to prototype an initial query and iteratively
refine the query until it represented a real bug. Throughout the process, we
could quickly and easily execute prototype queries on not only the original
dataset, but also on intermediate tables produced during the debugging
process. Our final query involves the composition of 4 unique database
operators, each of which is parameterized by a custom lambda function to
handle various data types. This process is representative of how MDB is used
in practice to discover bugs in machine learning problems.

##  3 The Model Debugger Framework

Querying Systems |  Data Representation |  Flexible Schema |  User Defined Functions |  Querying Abstractions
---|---|---|---|---
Pandas |  numpy.array tables | ✗ | ✓ | ✓
MongoDB Banker et al. (2016) | key-value stores | ✓ | ✗ | ✓
MDB | object collections | ✓ | ✓ | ✓
Python, OMG Kang et al. (2018) | arbitrary objects | ✓ | ✓ | ✗
Table 1: Comparison of querying systems.

In this section, we describe the design and implementation of our query
framework MDB. A natural question is whether an existing query framework
suffices for our needs of scalability, interactivity, and expressivity.
Candidates include general-purpose languages like Python, and domain-specific
frameworks like Pandas, a data analysis framework, and MongoDB, a document-
oriented NoSQL database system. While each of these supports a subset of our
desired features, however, none of them is designed for debugging machine
learning models and datasets.

For instance, if arbitrary Python objects are needed for a debugging task,
Python is preferable to Pandas, since the latter does not support querying
over arbitrary objects. On the other hand, if the objects adhere to a rigid
structure or can be easily flattened out into a table of primitive datatypes,
it is more efficient to take advantage of the querying abstractions provided
by Pandas. Frameworks like MongoDB attempt to bridge these extremes, allowing
queries over objects represented as key-value collections. However, they lack
support for user-defined functions (UDFs) that allow users to leverage
external modules and express arbitrarily complex queries. We summarize these
tradeoffs in Table 1.

Designing MDB thus involved a fundamental rethinking of both, the data
representation and the query language. Figure 4 provides a grammar describing
the structure of data and queries in MDB.

###  3.1 Data Representation

The primary data structure in MDB is a table $T$ which represents an indexed
collection of related objects, such as the frames from a video. MDB enables
users to write and execute queries over a database $D$, which is a collection
of named tables. Such a data representation, where the data being queried over
is a set of object collections, is also referred to as an _object-relational_
representation.

(name) | $n$ |  |
---|---|---|---
(function) | $f$ |  |
(table) | $T$ | $\mathrel{::=}$ | $[i_{1}\rightarrow o_{1},\ldots,i_{n}\rightarrow o_{n}]$
(operator) | $O$ | $\mathrel{::=}$ |  join ($n$, $f$, $f$)
|  | $|$ |  filter ($f$) $|$ cols ($f$)
|  | $|$ |  order_by ($f$, $b$)
|  | $|$ |  group_by ($f$)
|  | $|$ |  flatten () $|$ unique ()
(expression) | $E$ | $\mathrel{::=}$ |  $n$ $|$ $E.O$
(query) | $Q$ | $\mathrel{::=}$ |  ($n$, $E$)
(database) | $D$ | $\mathrel{::=}$ | $[n_{1}\rightarrow T_{1},\ldots,n_{k}\rightarrow T_{k}]$
Figure 4: Grammar for our framework.

An object-relational representation has several advantages over other common
representations. First, while it requires objects being queried to be
contained within a collection, it has no restrictions on the structure of the
objects being queried. This allows for the flexibility to query over models
and datasets without the associated overhead of converting the data into a
compatible representation. Indeed, this allows the instantiation of tables and
databases directly from data structures such as PyTorch datasets that
specialize in storing and loading machine learning datasets.

Furthermore, storing these objects within collections allows for enough
structure within each table to provide succinct yet powerful querying
abstractions over them akin to their relational algebra counterparts.
Operations such as joining tables or grouping rows turn into single-line
specifications without the need for setting up low-level infrastructure that
would otherwise be required.

An object-relational representation also allows for the support of executing
UDFs over the objects within each table. MDB supports this by allowing users
to supplement each relational algebra abstraction with one or more UDFs if
needed. This also allows the user to leverage external libraries, machine
learning models, and other tools to write their queries within the MDB
framework.

###  3.2 Query Language

Operator ($O$) | Arguments |  Summary
---|---|---
join | $n,f_{\text{key}},f_{\text{fkey}}$ |  Joins the current table with another table named $n$, using the key and foreign key UDFs.
filter | $f_{\text{cond}}$ |  Filters out the objects $o_{j}$ in the current table that violate $f_{\text{cond}}(o_{j})$.
cols | $f_{\text{cols}}$ |  Returns a table where each object $o_{j}$ is transformed into $f_{\text{cols}}(o_{j})$.
order_by | $f_{\text{ord}},b$ |  Orders the objects $o_{j}$ by $f_{\text{ord}}(o_{j})$ according to ordering type given by $b$.
group_by | $f_{\text{grp}}$ |  Groups the objects $o_{j}$ into group $g$ if $g=f_{\text{grp}}(o_{j})$.
flatten | — |  Flattens nested objects such that each object element is a separate row.
unique | — |  Removes duplicate objects.
Table 2: Summary of table operators in MDB. Functions $f_{*}$ denote user
defined functions (UDFs).

We next describe the language for writing MDB queries. As shown in the grammar
in Figure 4, each query $Q$ is a program identified by a query name $n$ and a
sequential chain $E$ of table operators $O$. The first operator in the chain
is always a name $n$. When the query is run over a database $D$, this operator
registers the table $T=D[n]$ as the base table of the query. After that, the
$i^{\text{th}}$ operator in the chain operates on a table $T$ which is the
result of sequentially performing the previous $i-1$ operations. Running the
query over $D$ sequentially executes all the operators in the chain, and
returns the final table as the result of that query.

We summarize the operators supported by MDB in Table 2. They are largely drawn
from relational algebra and natively support complex operations over tables.
With the exception of the flatten and unique operators, users can supplement
these operators with UDFs. These UDFs allow expressing computations over the
data by defining arbitrarily complex functions. Furthermore, it allows
leveraging Python libraries or even external machine learning models to write
queries over the tables. Thus, this integration of relational algebra with
functional programming enables one to express complex queries in a concise
manner. We describe these operators in further detail in Appendix D.

##  4 Experiments

We evaluate MDB on a multitude of tasks across a diverse set of domains. We
compare it to existing query systems in terms of the execution time and
conciseness of queries. We also validate the usability of MDB by means of a
modest user study.

###  4.1 Tasks and Datasets

Object detection. We craft queries to find critical bugs made by a state-of-
the-art object detection model, OneFormer (Jain et al., 2022a), on the
Cityscapes validation set (Cordts et al., 2016). Specifically, we seek to
discover potentially critical bugs with queries such as the one shown in
Figure 1(a), i.e., discovering all objects from all 15K frames that are
supposed to occur in three consecutive frames but are not detected
consistently by the model, which is denoted by S-Q1. We also explore other
queries for this task, such as finding frames with two cars moving towards
each other (S-Q2).

Time series imputation. We study bugs in a state-of-the-art medical time
series imputation model SAITS Du et al. (2023) over the Physionet-2012
Challenge dataset Silva et al. (2012). As Figure 1(c) shows, many of the
imputation bugs were caused by violating domain knowledge or intuition, like
the smoothness assumption on typical medical time series data You et al.
(2018). We test this assumption by writing a query that compares each entry
against its neighbors along the temporal dimension (T-Q1). We also leverage
other criteria such as the inter-quartile range (Vinutha et al., 2018) to
determine the outliers across the entire time series (T-Q2) or its sub-
sequences (T-Q3).

Image classification. For this task, we examine the iWildCam dataset and the
predictions of the baseline ERM algorithm (Beery et al., 2020). The iWildCam
dataset is curated to classify wildlife species in individual frames of video
sequences obtained from trap cameras. Since individual frames are considered
for prediction, there are various consistency-related bugs. For example, a
series of frames can have one single ground-truth label (which corresponds to
one animal), while the predicted labels are varied across these frames. We
therefore query subsequences of frames with the same prediction label (I-Q1)
to capture other prediction inconsistency bugs.

LLM bias discovery. We use MDB to discover bias in large language models,
following the definitions established in prior works Nangia et al. (2020);
Smith et al. (2022); Li et al. (2020); Kurita et al. (2019). We leverage
prompts from the Alpaca instruction fine-tuning dataset Taori et al. (2023)
(Alpaca for short) to collect responses from ChatGPT (`gpt-3.5-turbo`) and
T5111https://huggingface.co/lmsys/fastchat-t5-3b-v1.0 models. For each
response, we find all of pairs of adjectives describing an occupation
(brilliant scientists) and then identify adjective bias associated with
different occupations across the full set of responses. This query is
identified with B-Q1 and reused for discovering other biases.

Additional details of these datasets and their corresponding queries are
provided in Appendix B.

###  4.2 Baselines and Evaluation Metrics

We consider the following baseline systems: Python, which is also how OMG Kang
et al. (2018) debugs ML models; ArangoDB ara , which is a document-oriented
database and is an in-memory version of MongoDB; and Pandas pandas development
team (2020), an in-memory data-analysis framework. Each query described
earlier is implemented in MDB as well as in each baseline system. We then
compare the different implementations of each query in Table 3 to evaluate
both the efficiency (measured by the running time) and conciseness (measured
by the number of tokens) of queries written in different systems.

###  4.3 Quantitative Results

|  | Running time (seconds) | Conciseness (# of tokens)
---|---|---|---
Task |  Query |  Python |  ArangoDB | Pandas | MDB |  Python |  ArangoDB | Pandas | MDB
Object detection |  S-Q1 |  3.41 |  - | 3.17 | 2.29 |  183 |  - | 175 | 113
|  S-Q2 |  0.46 |  112.9 | 1.31 | 0.51 |  109 |  207 | 113 | 97
Time series imputation |  T-Q1 |  0.46 |  1000 | 0.98 | 0.26 |  123 |  171 | 148 | 109
|  T-Q2 |  0.12 |  222.94 | 28.96 | 2.87 |  111 |  174 | 120 | 99
|  T-Q3 |  0.80 |  142.72 | 5.83 | 0.88 |  149 |  167 | 183 | 140
Image classification |  I-Q1 |  0.14 |  0.25 | 1.68 | 0.15 |  75 |  64 | 43 | 55
LLM bias discovery |  B-Q1 |  0.012 |  - | 0.085 | 0.016 |  104 |  - | 98 | 89
Table 3: Quantitative results (smaller is better).

The efficiency and conciseness comparison between different query systems is
presented in Table 3. MDB significantly outperforms ArangoDB and Pandas in
processing queries by up to 10x. Furthermore, queries like B-Q1 cannot be
written with ArangoDB (Appendix C). MDB is competitive with Python scripts in
terms of running time, indicating that MDB abstractions add negligible
overhead. MDB also requires significantly less tokens for constructing queries
(up to 38% reduction) than Python, showcasing the conciseness of MDB. Further
details on these results are in Appendix C.

###  4.4 Case Study: Mislabelled Data in iWildCam

Over the course of the development of MDB, the authors wrote more than 25
queries over the iWildCam dataset alone covering a large variety of possible
bugs. These queries contained up to 7 operators, with almost all UDFs being
single-lined lambda functions defined within the query specification. One such
query was used to capture potentially mislabelled frames.

The iWildCam dataset contains a number of empty frames labelled as animals.
This may occur when an animal walks off-camera within the first few frames of
a sequence, but subsequent frames are labelled as if the animal were present.
We note that animals tend to move across frames, resulting in non-trivial
differences between consecutive frames. This also implies that if the two
continuous frames are almost identical, then it is highly likely that no
animal exists in those frames. We therefore write a query to capture such
instances, an example of which is shown in Figure 5. Here, all the frames in
the sequence are labelled mazama temama, but the animal is only present in the
first two.

(a)

(b)

(c)

(d)

Figure 5: An example of ground-truth error in the iWildCam dataset. Figure
5(a)-5(d) show a continuous sequence of four frames from the iWildCam dataset,
in which an animal exits from the camera range from the second frame. However,
all four frames are annotated as mazama temama.

####  4.4.1 Case Study: Bias Discovery in LLMs

(a) A prompt about a female farmer

(b) A prompt about a male farmer

Figure 6: An example of gender bias in the prompts of Alpaca dataset. In
Figure 6(a), the prompt is about female farmer, which is more positive while
in Figure 6(b), the prompt is about male farmer and its tone is more negative.

We leverage the query result of B-Q1 over the Alpaca dataset to search for
gender bias within each occupation. Figure 6 shows two prompts about farmers
from the Alpaca dataset and the responses from the T5 model given these
prompts. The first prompt in Figure 6(a) is about female farmer and the entire
prompt is very positive while the second prompt in Figure 6(b) is about a male
farmer, which exhibits negative feeling. The comparison of these two prompts
on farmers thus demonstrate the gender bias in the prompt generation process.

###  4.5 User Study

We conduct a modest user study to validate the effectiveness of MDB for
debugging models. The users included 6 graduate students and 2 undergraduate
students studying computer science, as well as 1 systems engineer. Each user
was first given a tutorial of MDB in a Jupyter Notebook using the iWildCam
task. Then the user was asked to find bugs in the model’s predictions over
three tasks of increasing complexity. We provide further details in Appendix
E. All users completed the first two tasks in an average of 3.78 and 22.61
minutes respectively. All but 1 user completed the third and most complex
task, where they had to find a bug without using ground-truth labels. Their
queries had an average precision of 0.89 and recall of 0.65, compared to the
gold standard of 0.91 and 0.78, respectively, and were written in an average
of 54.33 minutes. The large variance in the average times taken for the three
tasks reflects the variance in their complexity. Overall, these numbers
successfully validate the usability of MDB despite the users’ lack of prior
familiarity with MDB. The users who responded to follow-up questions were
positive in their feedback. They liked using MDB and preferred it to Python.
One user was even appreciative that MDB operations did a lot of the hard work
for them that they would have to otherwise do in Python.

##  5 Related Work

Debugging machine learning. Various tools have been developed to elucidate
certain types of information from machine learning models and datasets. One
early approach uses back and forth model explanations and user corrections as
a debugging loop Kulesza et al. (2015), while others explored debugging least
squares models Cadamuro et al. (2016); Bhadra and Hein (2015). Model
assertions were introduced to monitor and correct live systems (Kang et al.,
2018). Sparsifying the last linear layer of a neural network improved the
debuggability of feature visualizations (Wong et al., 2021). Robust features
were leveraged to find failure modes and spurious correlations in vision
classifiers (Singla et al., 2021; Singla and Feizi, 2021). Variations of
influence functions are capable of identifying certain data problems such as
mislabeled examples and test set leakage Koh and Liang (2017); Salman et al.
(2022); Ilyas et al. (2022). The use of related explanation methods have also
been explored as a debugging tool Anders et al. (2022). Other tools have been
developed to debug the training procedure for deep neural networksSchneider et
al. (2021). Model debugging and common bugs have also been studied for
federated learning settings Augenstein et al. (2019). Some debugging tools
were found provide skewed information due to a bias incurred from missing
features (Jain et al., 2022b).

Databases for machine learning. Techniques from databases have been used to
enhance various aspects of machine learning. Ideas from operation scheduling
have been used to select appropriate hardware Mirhoseini et al. (2017) or find
semantically equivalent but more efficient operations to optimize deep
learning computations Jia et al. (2019). Recomputing and swapping techniques
from databases have been adopted to manage memory in deep learning frameworks
and models Pleiss et al. (2017); Wang et al. (2018). Compression techniques
and communication frameworks have been proposed to accelerate parameter
servers in distributed training of large models Jiang et al. (2017); Goyal et
al. (2017). These approaches focus on the speed and efficiency of deploying
models.

##  6 Conclusion

We introduced a new framework MDB for debugging machine learning datasets and
models. We showed that MDB is able to help find bugs in several tasks over
various domains up to 10x faster than other querying systems while being up to
38% more concise than regular Python programs. We also validated the usability
of MDB by conducting a modest user study. In the future, we intend to extend
MDB to further improve performance by parallelizing operations and to support
synthesizing queries from natural language descriptions.

## References

  * [1] Arangodb query language (aql) introduction: Arangodb documentation.  www.arangodb.com.  Accessed: 2022-07-26.
  * Abid et al. [2021] Abubakar Abid, Maheen Farooqi, and James Zou.  Persistent anti-muslim bias in large language models.  In _Proceedings of the 2021 AAAI/ACM Conference on AI, Ethics, and Society_ , pages 298–306, 2021.
  * Anders et al. [2022] Christopher J Anders, Leander Weber, David Neumann, Wojciech Samek, Klaus-Robert Müller, and Sebastian Lapuschkin.  Finding and removing clever hans: using explanation methods to debug and improve deep models.  _Information Fusion_ , 77:261–295, 2022.
  * Augenstein et al. [2019] Sean Augenstein, H Brendan McMahan, Daniel Ramage, Swaroop Ramaswamy, Peter Kairouz, Mingqing Chen, Rajiv Mathews, et al.  Generative models for effective ml on private, decentralized datasets.  _arXiv preprint arXiv:1911.06679_ , 2019.
  * Banker et al. [2016] Kyle Banker, Douglas Garrett, Peter Bakkum, and Shaun Verch.  _MongoDB in action: covers MongoDB version 3.0_.  Simon and Schuster, 2016.
  * Beery et al. [2020] Sara Beery, Elijah Cole, and Arvi Gjoka.  The iwildcam 2020 competition dataset.  _arXiv preprint arXiv:2004.10340_ , 2020.
  * Bhadra and Hein [2015] Sahely Bhadra and Matthias Hein.  Correction of noisy labels via mutual consistency check.  _Neurocomputing_ , 160:34–52, 2015.
  * Bolya et al. [2020] Daniel Bolya, Sean Foley, James Hays, and Judy Hoffman.  Tide: A general toolbox for identifying object detection errors.  In _Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part III 16_ , pages 558–573. Springer, 2020.
  * Cadamuro et al. [2016] Gabriel Cadamuro, Ran Gilad-Bachrach, and Xiaojin Zhu.  Debugging machine learning models.  In _ICML Workshop on Reliable Machine Learning in the Wild_ , volume 103, 2016.
  * Cordts et al. [2016] Marius Cordts, Mohamed Omran, Sebastian Ramos, Timo Rehfeld, Markus Enzweiler, Rodrigo Benenson, Uwe Franke, Stefan Roth, and Bernt Schiele.  The cityscapes dataset for semantic urban scene understanding.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 3213–3223, 2016.
  * Du et al. [2023] Wenjie Du, David Côté, and Yan Liu.  Saits: Self-attention-based imputation for time series.  _Expert Systems with Applications_ , 219:119619, 2023.
  * Goyal et al. [2017] Priya Goyal, Piotr Dollár, Ross Girshick, Pieter Noordhuis, Lukasz Wesolowski, Aapo Kyrola, Andrew Tulloch, Yangqing Jia, and Kaiming He.  Accurate, large minibatch sgd: Training imagenet in 1 hour.  _arXiv preprint arXiv:1706.02677_ , 2017.
  * Honnibal and Montani [2017] Matthew Honnibal and Ines Montani.  spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.  To appear, 2017.
  * Ilyas et al. [2022] Andrew Ilyas, Sung Min Park, Logan Engstrom, Guillaume Leclerc, and Aleksander Madry.  Datamodels: Predicting predictions from training data.  _arXiv preprint arXiv:2202.00622_ , 2022.
  * Jain et al. [2022a] Jitesh Jain, Jiachen Li, MangTik Chiu, Ali Hassani, Nikita Orlov, and Humphrey Shi.  OneFormer: One Transformer to Rule Universal Image Segmentation.  _arXiv_ , 2022a.
  * Jain et al. [2022b] Saachi Jain, Hadi Salman, Eric Wong, Pengchuan Zhang, Vibhav Vineet, Sai Vemprala, and Aleksander Madry.  Missingness bias in model debugging.  _arXiv preprint arXiv:2204.08945_ , 2022b.
  * Jia et al. [2019] Zhihao Jia, James Thomas, Todd Warszawski, Mingyu Gao, Matei Zaharia, and Alex Aiken.  Optimizing dnn computation with relaxed graph substitutions.  _Proceedings of Machine Learning and Systems_ , 1:27–39, 2019.
  * Jiang et al. [2017] Jiawei Jiang, Bin Cui, Ce Zhang, and Lele Yu.  Heterogeneity-aware distributed parameter servers.  In _Proceedings of the 2017 ACM International Conference on Management of Data_ , pages 463–478, 2017.
  * Kang et al. [2018] Daniel Kang, Deepti Raghavan, Peter Bailis, and Matei Zaharia.  Model assertions for debugging machine learning.  In _NeurIPS MLSys Workshop_ , volume 3, page 10, 2018.
  * Koh and Liang [2017] Pang Wei Koh and Percy Liang.  Understanding black-box predictions via influence functions.  In _International conference on machine learning_ , pages 1885–1894. PMLR, 2017.
  * Kulesza et al. [2015] Todd Kulesza, Margaret Burnett, Weng-Keen Wong, and Simone Stumpf.  Principles of explanatory debugging to personalize interactive machine learning.  In _Proceedings of the 20th international conference on intelligent user interfaces_ , pages 126–137, 2015.
  * Kurita et al. [2019] Keita Kurita, Nidhi Vyas, Ayush Pareek, Alan W Black, and Yulia Tsvetkov.  Measuring bias in contextualized word representations.  In _Proceedings of the First Workshop on Gender Bias in Natural Language Processing_ , pages 166–172, 2019.
  * Li et al. [2020] Tao Li, Daniel Khashabi, Tushar Khot, Ashish Sabharwal, and Vivek Srikumar.  Unqovering stereotyping biases via underspecified questions.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 3475–3489, 2020.
  * Mirhoseini et al. [2017] Azalia Mirhoseini, Hieu Pham, Quoc V Le, Benoit Steiner, Rasmus Larsen, Yuefeng Zhou, Naveen Kumar, Mohammad Norouzi, Samy Bengio, and Jeff Dean.  Device placement optimization with reinforcement learning.  In _International Conference on Machine Learning_ , pages 2430–2439. PMLR, 2017.
  * Nadeem et al. [2021] Moin Nadeem, Anna Bethke, and Siva Reddy.  Stereoset: Measuring stereotypical bias in pretrained language models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 5356–5371, 2021.
  * Naik et al. [2023] Aaditya Naik, Yinjun Wu, Mayur Naik, and Eric Wong.  Do machine learning models learn common sense?  _arXiv preprint arXiv:2303.01433_ , 2023.
  * Nangia et al. [2020] Nikita Nangia, Clara Vania, Rasika Bhalerao, and Samuel Bowman.  Crows-pairs: A challenge dataset for measuring social biases in masked language models.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1953–1967, 2020.
  * pandas development team [2020] The pandas development team.  pandas-dev/pandas: Pandas, February 2020.  URL https://doi.org/10.5281/zenodo.3509134.
  * Pleiss et al. [2017] Geoff Pleiss, Danlu Chen, Gao Huang, Tongcheng Li, Laurens Van Der Maaten, and Kilian Q Weinberger.  Memory-efficient implementation of densenets.  _arXiv preprint arXiv:1707.06990_ , 2017.
  * Salman et al. [2022] Hadi Salman, Saachi Jain, Andrew Ilyas, Logan Engstrom, Eric Wong, and Aleksander Madry.  When does bias transfer in transfer learning?  _arXiv preprint arXiv:2207.02842_ , 2022.
  * Schneider et al. [2021] Frank Schneider, Felix Dangel, and Philipp Hennig.  Cockpit: A practical debugging tool for the training of deep neural networks.  _Advances in Neural Information Processing Systems_ , 34:20825–20837, 2021.
  * Silva et al. [2012] Ikaro Silva, George Moody, Daniel J Scott, Leo A Celi, and Roger G Mark.  Predicting in-hospital mortality of icu patients: The physionet/computing in cardiology challenge 2012.  In _2012 Computing in Cardiology_ , pages 245–248. IEEE, 2012.
  * Singla and Feizi [2021] Sahil Singla and Soheil Feizi.  Salient imagenet: How to discover spurious features in deep learning?  _arXiv preprint arXiv:2110.04301_ , 2021.
  * Singla et al. [2021] Sahil Singla, Besmira Nushi, Shital Shah, Ece Kamar, and Eric Horvitz.  Understanding failures of deep networks via robust feature extraction.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 12853–12862, 2021.
  * Smith et al. [2022] Eric Michael Smith, Melissa Hall, Melanie Kambadur, Eleonora Presani, and Adina Williams.  “i’m sorry to hear that”: Finding new biases in language models with a holistic descriptor dataset.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 9180–9211, 2022.
  * Taori et al. [2023] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * Tashiro et al. [2021] Yusuke Tashiro, Jiaming Song, Yang Song, and Stefano Ermon.  Csdi: Conditional score-based diffusion models for probabilistic time series imputation.  _Advances in Neural Information Processing Systems_ , 34:24804–24816, 2021.
  * Vinutha et al. [2018] HP Vinutha, B Poornima, and BM Sagar.  Detection of outliers using interquartile range technique from intrusion dataset.  In _Information and Decision Sciences: Proceedings of the 6th International Conference on FICTA_ , pages 511–518. Springer, 2018.
  * Wakabayashi [2018] Daisuke Wakabayashi.  Self-driving uber car kills pedestrian in arizona, where robots roam, Mar 2018.  URL https://www.nytimes.com/2018/03/19/technology/uber-driverless-fatality.html.
  * Wang et al. [2018] Linnan Wang, Jinmian Ye, Yiyang Zhao, Wei Wu, Ang Li, Shuaiwen Leon Song, Zenglin Xu, and Tim Kraska.  Superneurons: Dynamic gpu memory management for training deep neural networks.  In _Proceedings of the 23rd ACM SIGPLAN symposium on principles and practice of parallel programming_ , pages 41–53, 2018.
  * Wang et al. [2022] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A Smith, Daniel Khashabi, and Hannaneh Hajishirzi.  Self-instruct: Aligning language model with self generated instructions.  _arXiv preprint arXiv:2212.10560_ , 2022.
  * Wilson et al. [2019] Benjamin Wilson, Judy Hoffman, and Jamie Morgenstern.  Predictive inequity in object detection.  _arXiv preprint arXiv:1902.11097_ , 2019.
  * Wong et al. [2021] Eric Wong, Shibani Santurkar, and Aleksander Madry.  Leveraging sparse linear layers for debuggable deep networks.  In _International Conference on Machine Learning_ , pages 11205–11216. PMLR, 2021.
  * You et al. [2018] Cheng You, Dennis KJ Lin, and S Stanley Young.  Time series smoother for effect detection.  _PloS one_ , 13(4):e0195360, 2018.
  * Zech et al. [2018] John R Zech, Marcus A Badgeley, Manway Liu, Anthony B Costa, Joseph J Titano, and Eric Karl Oermann.  Variable generalization performance of a deep learning model to detect pneumonia in chest radiographs: a cross-sectional study.  _PLoS medicine_ , 15(11):e1002683, 2018.

##  Appendix A Code

We have provided the code for MDB along with the appendices for the
supplementary material in a zip file. Additionally, we intend to publicly
release the code for MDB.

The code provided has the following directories:

  1. 1.

mdb: The directory containing the code for MDB.

  2. 2.

examples: The directory containing notebooks with MDB queries written for
various debugging tasks. All the queries discussed in this paper reside in
this directory.

  3. 3.

user_study_responses: The directory containing both the notebook provided for
the user study (user_study.ipynb) as well as the responses from the
participants (user_study_[1-9].ipynb).

  4. 4.

Baselines: The directory containing the setup code for the baseline system
ArangoDB.

Additionally, the code contains a file wilds_queries.ipynb that gives a quick
tutorial on using MDB, along with a README.md file that explains the
installation process for MDB.

##  Appendix B Experiment details

###  B.1 Compute resources

For each of the following experiments, we use a local server with four Nvidia
2080 Ti GPUs and 80 Intel Xeon Gold 6248 CPUs. We only use the GPUs for
evaluating the object detection model used in the experiments in Appendix B.2
and for evaluating the T5 model for the experiments in Appendix B.5.

###  B.2 Object detection

As mentioned in Section 4.1, we use the Cityscapes validation set, which is
composed of 15K dashcam frames in total. We use a subset of 33 video sequences
consisting of 1000 frames for the experiments. In total, we wrote 57 queries
for debugging the object detection model. These queries ranged from queries to
find dangerous situations to queries that processed the data for use by other
queries. All 57 queries are available in `examples/self_driving_explore.ipynb`
in the supplementary materials. We comprehensively evaluate two of these
queries, S-Q1 and S-Q2, using MDB and other systems (Pandas, ArangoDB, and
Python). The detailed implementations of S-Q1 and S-Q2 are included in Figure
7 and Figure 8 respectively. In what follows, we briefly explain how these two
queries are constructed with the operators introduced in Table 2.

##### S-Q1

As Figure 7(a) shows, this query takes the output of the OneFormer model [Jain
et al., 2022a], represented by a named table “outp”, as the input to this
query. Note that we use one additional “register” operator to set up this
table. Each row of the named table “outp” carries four fields, including the
video sequence ID, the frame ID within each video sequence, the semantic
segmentation result, and the object detection result. In the beginning of this
query, we perform two “join” operations over three copies of “out” with join
keys specified by three functions, “fid_plus1”, “fid”, and “fid_plus2”, which
results in a table where each row consists of three continuous frames from the
same video sequence. Then we apply a function to each row in this table with
the “cols” operator and invoke two user-defined functions “match_three_bboxes”
and “zip_adj_preds”, which ends up with a list of matched bounding boxes
corresponding to the same object across three contiguous frames. Each row in
the result so far contains a list of matched bounding boxes which is not
convenient for further analysis. We therefore perform a “flatten” operation to
split each row into multiple rows where each one corresponds to one object
with matched bounding boxes across three frames. In the end, the “filter”
operator is applied to select the objects with the same predicted labels in
the first two frames and no detection in the third frame. The final result is
stored in a table named “temporal_consistency”.

This query finds 74 objects which are detected in two consecutive frames and
disappear in the third. Of the resulting tuples of three frames, where an
object disappears in the third frame, there are 10 tuples where the first
frame appears in the labelled ground truth. In all 10 of these frames, this
query finds an object in the set of mispredictions. Thus, this query has 100%
precision. In total, there are 773 mispredicted objects, so S-Q1 has a recall
of 1.3% for all the mispredictions, but many of these mispredictions are very
small bounding boxes or correspond to objects such as the sky or vegetation
which are not critical. Thus, when narrowing down to interesting
mispredictions, there are a total of 251, so this query has a recall of 4.0%.

##### S-Q2

As Figure 8(a) shows, this query joins the output of S-Q1 with itself using
the “join” operator with the sequence ID and frame ID (denoted by “row[0]” and
“row[1][0]” resp.) as the join key to obtain all pairs of objects within the
same frame. From the resulting pairs of objects, we identify those that are
both predicted as vehicles in two contiguous frames by invoking one user-
defined function “is_vehicle” in a filter operator. In this filter operation,
“row[2][0]” and “row[2][1]” represent the predicted labels of the first object
in two contiguous frames while “row[6][0]” and “row[6][1]” represent the
predicted labels of the second object in the same two frames. In the
subsequent “cols” operation, we estimate the speed of those pairs of vehicles
by leveraging the differences of their bounding box positions by using a user-
defined function “’get_velo’, in which “row[3][0]”, “row[3][1]” represents the
bounding boxes of the first object in two contiguous frames while “row[7][0]”,
“row[7][1]” represents the bounding boxes of the first object in the same two
frames.

This query results in pairs of objects with their velocities which can be used
in other queries. For instance, in one query we filter to pairs of objects
which move in highly different directions, and this query selects 110 pairs of
objects over 77 frames. Six of these frames are in the ground truth and 24
selected objects appear in these ground truth frames where 16 correspond to
mispredictions. Thus, this query has a precision of 66.67%, a recall of 2.1%
for all mispredictions, and a recall of 6.4% for the most interesting
mispredictions.

(a) S-Q1 in MDB

(b) S-Q1 in Pandas

(c) S-Q1 in Python

Figure 7: Implementation of S-Q1 in different query systems

(a) S-Q2 in MDB

(b) S-Q2 in Pandas

(c) S-Q2 in ArangoDB

(d) S-Q2 in Python

Figure 8: Implementation of S-Q2 in different query systems

###  B.3 Medical time series imputation

As mentioned in Section 4.1, the Physionet-2012 Challenge dataset Silva et al.
[2012] is used, which is composed of de-identified patients’ records in
Intensive Care Units (ICU). By reusing the pre-process scripts from Tashiro et
al. [2021], we obtain 4000 time series samples, each of which contains 35
features collected from 48 time steps.

However, note that those records are collected sporadically, which ends up
with highly sparse medical time series sequences with a substantial number of
missing entries. Various neural networks have been proposed to impute those
missing entries to construct seemingly complete time series for the down-
stream tasks, which could violate domain knowledge or intuitions, like the
smoothness assumption on typical medical time series data You et al. [2018].
We therefore proposed T-Q1, T-Q2 and T-Q3 to capture such bugs in the
imputation results. The details of the implementations of those three queries
in different query systems are included in Figure 9-Figure 11. Similar to
Appendix B.2, we also describe how to construct these three queries with the
operators introduced in Table 2 as follows.

(a) T-Q1 in MDB

(b) T-Q1 in Pandas

(c) T-Q1 in Python

(d) T-Q1 in ArangoDB

Figure 9: Implementation of T-Q1 in different query systems

(a) T-Q2 in MDB

(b) T-Q2 in Pandas

(c) T-Q2 in Python

(d) T-Q2 in ArangoDB

Figure 10: Implementation of T-Q2 in different query systems

(a) T-Q3 in MDB

(b) T-Q3 in Pandas

(c) T-Q3 in Python

(d) T-Q3 in ArangoDB

Figure 11: Implementation of T-Q3 in different query systems

##### T-Q1

The input of this query includes two named tables, i.e., “train_valid_samples”
and “mask” which are both 3D tensors of the same shape. The named table
“train_valid_samples” stores the medical records of each patient, which are
composed of multiple measurements (such as blood pressure) as features across
multiple time steps. Therefore, the three dimensions of this named table are
sample dimension, temporal dimension and feature dimension respectively. The
named table “mask” is a 0-1 3D tensor for recording which entry in
“train_valid_samples” is missing or not. T-Q1 starts by joining those two
named tables with one join operation, which is followed by invoking two user-
defined functions “compute_gap_res0” and “aggregate_rows0” in two continuous
“cols” operations. The function “compute_gap_res0” iterates each feature to
compute the gap between one entry and its neighbors along temporal dimension
while the function “aggregate_rows0” compute the statistics of those gap
values for each feature. The final results are stored in a named table
“consistency_bound_mask”.

##### T-Q2

The input of this query includes two named tables, “test_dataset” and
“test_imputed_x”. Each row of “test_dataset” stores each patient’s record,
which is composed of four fields, i.e., “observed_values”, “observed_mask”,
“time_steps” and “gt_mask”. The variable “observed_values” stores temporal
changes of multiple features, which is thus a 2D tensor. Due to the existence
of missing values, we use a 0-1 mask tensor, “observed_mask” to indicate which
entry in “observed_values” is visible and “gt_mask” records the subset of the
visible entries (other visible entries are reserved for performance
evaluations). For the named table “test_imputed_x”, each row of this table is
also a 2D tensor storing all the imputed values by using the time series
imputation model. So T-Q2 first joins “test_dataset” and “test_imputed_x” so
that we can obtain both the mask tensors and the imputation result for each
sample. Then for each sample, we apply “cols” operation and perform transpose
operation to facilitate the following “flatten” operations along each single-
feature temporal sequence. For each such sequence, we apply a user-defined
function “check_sequence_feat_range_on_time_series0_single_feat” to compute
the inter-quartile range by using all the non-missing entries in
“test_imputed_x” and capture the outliers based on that range. The final
results are stored in a named table “test_imputation”.

##### T-Q3

This query is similar to T-Q2 except that 1) the outliers are determined by
only using the named table “test_imputed_x”; 2) only sub-sequences (rather
than the entire sequence) are considered, which are obtained by using the
function “unfold”.

In addition to T-Q1, T-Q2 and T-Q3, we also explored other four queries to
identify other imputation bugs, which are included in the notebook
“examples/time_series_explorations/time_series_all_queries.ipynb” in the
submitted code. We can then compute imputation errors of those buggy entries
discovered by all of our queries against the ground-truth values, which can be
compared against the total imputation errors across the entire time series.
The result reveals that the overall imputation errors discovered by our
queries are around 81% of the total imputation errors., which thus indicates
that most of the imputation bugs can be perfectly characterized by our
queries.

(a) I-Q1 in MDB

(b) I-Q1 in Pandas

(c) I-Q1 in Python

(d) I-Q1 in ArangoDB

Figure 12: Implementation of I-Q1 in different query systems

###  B.4 Image classification

For this task, we examine the iWildCam dataset and the predictions of the
baseline ERM algorithm [Beery et al., 2020]. iWildCam dataset is composed of
129809 images in the training set and 14961 images in the validation set. As
mentioned in Section 4.1, since the predictions by the state-of-the-art models
are made over each individual frame, the consistency of predictions across
continuous frames are not considered. We therefore consider I-Q1 to query
subsequences of frames to identify whether the same prediction label exists by
counting how many different predictions those sequences contain. The detailed
implementation of I-Q1 is included in Figure 12 and how to construct I-Q1 with
the operators defined in Table 2 is described as follows.

As Figure 12(a) shows, I-Q1 takes a named table “wilds_train2” as input. Each
row of this dataset corresponds to one frame in the iWildCam dataset, which is
composed of two fields, i.e., the ground-truth label and an array of metadata.
So this query first groups by the second field of the metadata, i.e.,
“args[1][1]”, which represents the sequence ID of this frame (recall that
iWildCam is composed of multiple video sequences). Then the number of unique
ground-truth labels as well as all of these labels are collected within each
sequence with the “cols” operation. This can then facilitate the following
“filter” operation to only retain those video sequences with at least two
unique labels. The final result returns all these sequences which are sorted
by the number of their unique labels.

Other queries over the iWildCam dataset target various other bugs, including
bad data. As mentioned in the case study for the iWildCam dataset, we try
various queries to find mislabelled data. One query tries to identify empty
frames that are labelled as having animals by assuming that the frame
containing an animal will see some animal movement. Our query therefore tries
to find the frames where the difference is below a particular threshold. The
process of finding this query is depicted in the
examples/wilds_examples/wilds_empty_frames.ipynb file. We find 161 such
instances where pairs of frames have a difference of below 0.15 but greater
than 0. Note that the frames with difference 0 actually are instances of
duplicate frames where the same frame is repeated in two different samples.

We also consider some other forms of mislabelling data and other bugs in the
examples/wilds_examples/wilds_incorrect.ipynb notebook. For instance, we look
at the sequences of frames where they have multiple labels. Most sequences
capture either a single animal or multiple animals of the same species. As a
result, the sequences where the number of animal species labelled is more than
1 are indicative of mislabelled data. There are 95 such sequences in the
training data out of a total of around 24k sequences.

###  B.5 LLM bias discovery

For this task, we discover the bias in Alpaca instruction fine-tuning dataset
Taori et al. [2023] (Alpaca for short), which consists of 52K instruction-
response pairs generated from GPT-3.5 where the instructions were generated
using the Self-Instruct technique Wang et al. [2022]. We then evaluate T5 with
all 52K instructions to create another set of 52K instruction-answer pairs for
T5. As described in Section 4.1, we start with the high level hypothesis that
there may be biases in the associations of occupations and the adjectives to
describe different occupations such as farmer and engineer. After finding
adjectives which are more associated with certain occupations than others, we
confirm these biases through a counterfactual experiment by prompting the
model and analyzing the results. Both prompting of the model and analyzing the
results are done within MDB, where we describe how to prompt the model below
and then discuss query B-Q1 which is used for analysis of the counterfactual
experiment. For investigating bias in LLMs, we wrote 24 queries which can be
found in `examples/bias_queries.ipynb` in the supplementary materials, but we
focus on two of them in detail below. The detailed implementations of B-Q1 in
different query systems are presented in Figure 14.

Figure 13: Visualization of adjectives associated with the nouns farmer and
engineer in GPT-3.5 model output from the Alpaca dataset.

To find the associations between nouns and adjectives, we use a linguistic
dependency parsing model from the spacy Honnibal and Montani [2017] Python
library to extract a list of noun-adjective pairs for each of the 52K samples.
This is done in the query `noun_adj_pairs` in Figure 1(b). The set of all
adjectives used to describe a noun throughout the dataset is created using a
single grouping operation as shown by the final `group_by` operation in Figure
1(b). This query creates a new dataset object we then pass to a wordcloud
generating library to visualize the biases in word associations. After
visualizing the wordclouds for our initial occupations, we find that farmer
and engineer seem to have particularly strong and contrasting biases in the
associated adjectives as shown in Figure 13 where farmers are associated with
adjectives relating to age, race, and poverty, while engineers are associated
with adjectives relating to wealth and success. These first two queries
allowed us to quickly narrow down a broad hypothesis to a specific instance of
potential bias of associations with farmers and engineers.

To determine if these biases are actually reflective of real biases in GPT-3.5
and T5, we perform counterfactual experiments similar to the template-based
prompting methods of StereoSet Nadeem et al. [2021] and evaluate with their
metric of stereotype score. From the above two queries, we end up with a total
of 27 adjectives related to farmers and 40 related to engineers for GPT-3.5
and 14 adjectives related to farmer and 20 related to engineer for T5. To
confirm the association of farmer and engineer with these two sets of
adjectives, we use MDB to evaluate the language model on a new set of prompts
with the following query:

⬇

gpt35_bias_answers = (Query(’gpt35_bias_answers’)

.register(’gpt35_farmer_adjs’)

.join(’gpt35_engineer_adjs’, key=lambda *args: 1, fkey=lambda *args: 1)

.cols(lambda farmer_adj, engineer_adj: [

f"Answer with one word. The person is a farmer. \ Are they more likely
{farmer_adj} or {engineer_adj}?",

f"Answer with one word. The person is a farmer. \ Are they more likely
{engineer_adj} or {farmer_adj}?",

f"Answer with one word. The person is an engineer. \ Are they more likely
{engineer_adj} or {farmer_adj}?",

f"Answer with one word. The person is an engineer. \ Are they more likely
{farmer_adj} or {engineer_adj}?"])

.flatten()

.cols(lambda prompt: gpt35(prompt)).run(db)

where `gpt35_farmer_adjs` and `gpt35_engineer_adjs` are previously created
tables of adjectives describing farmer and engineer respectively. This query
directly constructs the prompts using templates and evaluates the language
model within the “cols” function.

As shown in Figure 14(a), B-Q1 takes the output from the above query,
“t5_bias_answers,” as input. Each row of this table stores a prompt-response
pair, in which the prompt is from the Alpaca dataset and the response is
generated by the language model. Query B-Q1 is then used to analyze the
responses. The query iterates over each prompt-response pair and parses the
prompts and answers with the function “parse_prompt” and “parse_response”
respectively in the “cols” operation. For the “parse_prompt” function, it
returns a flag variable to indicate whether the prompt contains the word
“farmer” and two additional words as the possible occupations that the prompt
mentions (denoted by “word1” and “word2”). For the function “parse_response”,
it simply parses the response into a list of tokens, which is denoted by a
variable “res”. In the subsequent step, the query groups by the flag variable
“is_farmer” and the combination of “word1” and “word2”, and then collects the
parsing results of each response sentence, i.e., “res” from the group_by
operation, within each group.

In total, query B-Q1 identifies that out of all the T5 responses that select
one of the given adjectives, T5 selects the biased one 58.0% of the time and
GPT-3.5 selects the biased adjective 59.6% of the time. For GPT-3.5, these
biased responses cover 23 of the 27 farmer associated adjectives and 34 of the
40 engineer associated adjectives. Similarly, the biased responses of T5 cover
11 of the 14 farmer adjectives and 15 of the 20 engineer adjectives.

(a) B-Q1 in MDB

(b) B-Q1 in Pandas

(c) B-Q1 in Python

Figure 14: Implementation of B-Q1 in different query systems

###  B.6 Qualitative studies for finding bias in LLMs

The queries used for case studies for Bias Discovery in LLMs are included in
the notebook examples/bias_examples/bias_qualitative_studies.ipynb.
Specifically, we iterate each prompt from Alpaca dataset, select those prompts
containing “farm” or “farmer”, and perform sentiment analysis by using a
pretrained RoBERTa model222https://huggingface.co/siebert/sentiment-roberta-
large-english from huggingface for those prompts explicitly containing male or
female words separately. The query for this process is shown in Figure 15, in
which all the prompts are stored in a named table “outp_instr_alpaca_t5”.

(a) Query for retrieving prompts regarding “male” and “farmer”

(b) Query for retrieving prompts regarding “female” and “farmer”

Figure 15: Queries for qualitative studies for LLM bias discovery

In the end, by representing positive sentiment as 1 and negative sentiment as
0, the average sentiment score for prompts regarding “male” and “farmer” is
0.23 while the prompts regarding “female” and “farmer” is 0.5, which thus
reveals the bias in the sentiment of the prompts on “farmer” between different
genders.

##### Discussion

It would be worth noting that although some of the above queries depend on
some user-defined functions, those functions are mostly simple and concise.
For example, we show the function “compute_gap_res0_numpy” that occurs in T-Q1
as follows, which is only composed of several lines of code for computing the
difference between each entry in the time series and its neighbors along the
temporal dimension by referencing Pytorch library.

⬇

def compute_gap_res0_numpy(full_data, full_mask):

full_data = torch.from_numpy(full_data)

full_mask = torch.from_numpy(full_mask)

data_prev, mask_prev = full_data[0:-2], full_mask[0:-2]

data_next, mask_next = full_data[2:], full_mask[2:]

data, mask = full_data[1:-1], full_mask[1:-1]

mask = torch.logical_and(torch.logical_and(mask, mask_prev), mask_next)

gap_value = torch.max(torch.stack(

[torch.abs(data - data_prev) - torch.abs(data_next - data_prev)

, torch.abs(data - data_next) - torch.abs(data_next - data_prev)]),dim=0)[0]

return gap_value[mask]

The definitions of other user-defined functions appearing in the above queries
and the full implementations of the above queries in different systems can be
found in
“examples/self_driving_examples/self_driving_explore_baseline_comparison.ipynb”,
“examples/time_series_explorations/time_series_baseline_comparison.ipynb”,
“examples/wilds_examples/wilds_baseline_comparison.ipynb” and
“examples/bias_examples/bias_baseline_comparison.ipynb” in the submitted code,
which corresponds to the object detection task, medical time series imputation
task, image classification task and LLM bias discovery task respectively.

##  Appendix C Experimental results analysis

###  C.1 Analysis on efficiency

##### Comparison against Pandas

One primary factors contributing to the inefficiency of Pandas is the rigorous
schema designs in Pandas. This may incur additional operations to explicitly
produce new attributes that are not in the schema in the first place but are
essential for the downstream operations. For example, as Figure 14(b) suggests
(see the comment “one additional step for using one column to contain group
key”), B-Q1 written in Pandas needs to group by one attribute that does not
exist in the schema, which therefore requires one additional step to generate
this new attribute. This step requires to take a full pass over the entire
dataframe, thus incurring $O(n)$ time ($n$ represents the number of records).
In contrast, this step is not necessary in MDB since constructing this
additional attributes is implicitly done within the group-by operations (see
the “group_by” operation in Figure 14(a)).

In addition, since all the data are stored as a 2D numpy array in Pandas, it
is thus essential to transform arbitrary Pytorch/Tensorflow datasets
containing the multi-dimensional data into a 2D numpy array in the
initialization step. This step not only requires additional users’ efforts for
such transformation, but also incurs non-trivial overhead. For example, the
following initialization step is essential to transform a Pytorch dataset to a
Pandas dataframe that is needed for evaluating I-Q1 in iWildCam dataset.

⬇

wilds_train_pd_ls = []

wilds_train_cln_ls = ["idx", "label"] + ["meta_" + str(i) for i in
range(len(wilds_train_tensors[0][2]))]

for idx in range(len(wilds_train_tensors)):

wilds_train_pd_ls.append([idx, wilds_train_tensors[idx][1]] +
wilds_train_tensors[idx][2].tolist())

\parwilds_train_pd =
pd.DataFrame(np.array(wilds_train_pd_ls),columns=wilds_train_cln_ls)

which takes around 18 mins, much longer than the initialization overhead in
MDB. Plus, such 2D-numpy-array format is also inconvenient for performing
queries involving non-primitive data types. For example, T-Q2 and T-Q3 are
queries that requires to operate over each individual time series sequence
along the temporal dimension on the 3D time series data. However, since the
underlying 3D time series is reshaped as 2D numpy array for Pandas by
combining the sample dimension and the temporal dimension for indexing each
row, some additional operations are thus required to collect the values along
the temporal dimension for each sample (see the comment “One additional step
to collect values along time dimensions” in Figure 10(b) and Figure 11(b)),
which thus leads to higher overhead than MDB.

##### Comparison against ArangoDB

In comparison to ArangoDB, MDB also achieves up to two order-of-magnitude
speed-ups (e.g., see S-Q2). The reason is that for queries like S-Q2 which
joins two datasets on attributes that are not indexed, ArangoDB has to perform
full nested scans on the datasets 333https://www.arangodb.com/2018/03/index-
types-indexes-used-arangodb-part-2/ for the join operations (see Figure 8(c)
where one join condition is “item1.frames[0] == item2.frames[0]” but
“frames[0]” is not indexed). This leads to $O(n^{2})$ time complexity while
MDB can construct indexes over those newly created attributes on the fly
without extra operations, which only takes $O(n)$ time (recall that $n$
represents the number of records). Plus, since the query interface in ArangoDB
is built upon javescript rather than Python, it is thus infeasible to invoke
Python libraries for more advanced operations (such as the Grayscale operation
on images in I-Q1) in ArangoDB queries. As a consequence, some queries (S-Q1
and B-Q1) are not even supported by ArangoDB, which thus invalidates the use
of ArangoDB in querying Python datasets.

##### About the error bars

We report the running time for each query on each system. These queries are
deterministic in nature, and therefore do not fluctuate by a reasonable margin
upon reruns of the query. Hence, we do not include error bars when reporting
the numbers.

###  C.2 Analysis on conciseness

Table 3 shows the conciseness metric evaluated on each query written in
different query languages. This tables show that MDB can significantly reduce
the number of tokens needed for constructing a query (up to 40% reduction)
with respect to plain Python script. This can thus reduce substantial amount
of users’ efforts in the process of iteratively writing and refining queries
for bug findings. Also, MDB saves more tokens than ArangoDB and Pandas most
times. The main reason is that some additional operations are inevitable in
Pandas and ArangoDB to perform by comparing against MDB. For example,
additional transformations between the data type that they support (Datafram
for Pandas and Json for ArangoDB) and Pytorch tensors are needed.

##  Appendix D MDB Operator Semantics

We discuss the semantics of MDB in this appendix. In order to do so, we assume
there to be a database $D$, and a query $q$ that produces a table $T$ when run
on $D$. Assume we now chain an operator $O$ to the query $q$ like $q.O$.

As mentioned in Table 2, there are 6 operators supported by MDB, each
performing an operation on table $T$. We describe each in detail.

  1. 1.

join: The join operator takes as arguments the name $n$ of the table to be
joined, and $f_{\text{key}}$ and $f_{\text{fkey}}$ as the key and foreign key
functions, respectively. It then joins the query table $T$ with the table
$T_{n}$ such that $T_{n}=D[n]$. Recall that a table is an indexed collection
of rows $T=[i_{j}\rightarrow r_{j}]$, where $i_{j}$ is the index of the row
and $r_{j}$ is the row itself. The join operator joins elements
$(i_{j}\rightarrow r_{j})\in T$ and $(i_{k}\rightarrow r_{k})\in T_{n}$ such
that $f_{\text{key}}(i_{j},r_{j})=f_{\text{fkey}}(i_{k},r_{k})$ to produce the
resulting table $T^{\prime}=[(i_{j},i_{k})\rightarrow(r_{j},r_{k})]$. By
default, the key and foreign key functions return the index, so the default
join joins two rows having the same index in the two tables, which is
equivalent to inner equijoin in SQL queries. This join is implemented as a
hash join, where we use a hash table to map the rows from table $T$ to those
of $T_{n}$. Using a hash join algorithm allows us to achieve this with a
complexity of $O(j+k)$, where $j=|T|$ and $k=|T_{n}|$, as opposed to typical
join algorithms that perform with complexity $O(jk)$.

  2. 2.

filter: The filter operator takes in as argument a filtering function
$f_{\text{cond}}$ and returns a table $T^{\prime}=[i_{j}\rightarrow r_{j}]$,
where $(i_{j}\rightarrow r_{j})\in T$ and
$f_{\text{cond}}(r_{j})=\text{True}$. This allows us to filter out rows from
the table that do not satisfy a given condition. Since the condition is a
function $f_{\text{cond}}$ that returns a boolean value, it can be arbitrarily
complex, allowing us to filter out rows based on any condition we can express
in Python.

  3. 3.

cols: The cols operator is responsible for transforming each row of the table
$T$ using the function $f_{\text{cols}}$ that it takes as argument. It
produces the table $T^{\prime}=[i_{j}\rightarrow f_{\text{cols}}(r_{j})]$
where $(i_{j}\rightarrow r_{j})\in T$. This allows us to perform powerful and
flexible transformations on the rows of the table, such as including image
transformations using image processing libraries, or analyzing the inferences
made by external models as shown in the query in Figure 14(a) where the cols
function is used to generate prompts for the LLMs.

  4. 4.

order_by: This operator simply reorders the rows of the table $T$ according to
the function $f_{\text{ord}}$ and the boolean $b$ that it takes as argument.
It orders the rows by the value of $f_{\text{ord}}(r_{j})$ (which is a float
value), where $(i_{j}\rightarrow r_{j})\in T$ in the ascending order if $b$ is
True, and in the descending order if $b$ is False.

  5. 5.

group_by: The group-by operator takes in as argument a grouping function
$f_{\text{grp}}$ and returns a table
$T^{\prime}=[i_{g}\rightarrow(g,[r_{j}|r_{j}\in T,f_{\text{grp}}(r_{j})=g])]$
where the rows of table $T$ are grouped by the value of
$f_{\text{grp}}(r_{j})$. Here, each row of the returned table is a tuple of
the group key $g$ and the list of rows in $T$ that have the same group key
$g$. This allows us to perform powerful group-by operations on the table, such
as grouping by the hour of the day or the sequence ID as shown in the query in
Section E.

  6. 6.

flatten: The flatten operator is one of two that does not take in any
arguments. It flattens each row $r_{j}\in T$ where $r_{j}$ is a collection of
elements $[r_{j1},r_{j2},r_{j3},\ldots]$ to produce table $T^{\prime}$ such
that each element $r_{jm}$ is its own row in $T^{\prime}$. If $r_{j}$ is not a
collection, then it is left unchanged.

  7. 7.

unique: The unique operator is another operator that does not take in any
arguments. It returns a table $T^{\prime}$ with the rows from $T$ with
duplicate rows removed.

##  Appendix E User Studies

We now describe the user studies we conducted in more detail.

###  E.1 Setup

The participants in the user study were given a jupyter notebook along with
the data required to perform the user study. They were also given a brief
tutorial on how to use MDB using the same data. The participants were then
asked to use MDB to write queries to find bugs in the data over three tasks of
increasing complexity. Tasks 1 and 2 allow the user to find bugs while taking
the ground truth labels into account. On the other hand, Task 3 requires the
user to do so without the ground truth labels, which makes the finding of such
patterns that capture all such bugs much more complicated. In order to
evaluate the effectiveness of the responses to Task 3, we add some code where
necessary to calculate the precision and recall of each response. No time
limit was enforced on any task, though participants were asked to keep track
of the time they spent on each task.

###  E.2 Tasks

The three tasks were as follows:

##### Task 1:

The participants were asked to find the frames in which the model made a
misprediction. This is a simple filter operation that can be done in a single
line of code in MDB. Below is the target solution to this query:

⬇

task_1_result = Query(’task_1’, base=’wilds_pred’) \ .filter(lambda img,
label, metadata, pred: label != pred)(db)

The ‘wilds_pred’ table contains the data from the validation set of the
iWildCam dataset along with the predictions made by the model.

##### Task 2:

Since MDB can be used to perform some analysis of the mispredictions,
participants were asked to do the same. In this case they were asked to find
the hour of the day with the most and least mispredictions. This requires a
group-by operator over the hour (the 6th element of the metadata) followed by
a sort. This takes a few lines of code in MDB if built upon the result of task
1, which is what the participants were asked to do, either using the ‘base’
keyword or the ‘register’ operator.

⬇

grouped_mispred = Query(’task_2’, base=’task_1’) \ .group_by(lambda img,
label, md, pred: md[5].item()) \ .sort(lambda hour, frames: len(frames))(db)

\parmost_mispredicted = grouped_mispred[-1]

least_mispredicted = grouped_mispred[0]

##### Task 3:

Sometimes the ground truth labels themselves are unknown. This is the case
when the model is deployed in the real world. In order to validate the ability
of MDB to find bugs in such cases, participants were asked to find the video
sequences where at least one frame was mispredicted. Most of the sequences
capture only one animal, so the correct solution is to group by the sequence
id and then filter out the sequences where the model predicts different
animals across the frames in that sequence. This requires a group-by operation
followed by a filter operation. However, participants were not aware of this
pattern, and so they were required to discover it on their own:

⬇

mispred_seq = Query(’incorrect_seqs2’, base=’wilds_pred’) \ .cols(lambda img,
label, md, pred: (md[0].item(), pred)) \ .group_by(lambda seqid, pred:
md[1].item()) \ .filter(lambda seq_id, preds: len(set(frames)) > 1)(db)

This is not a particularly obvious pattern, and it is not foolproof either: it
has a precision of 0.91 and a recall of 0.78. However, it is the best pattern
the authors could come up with for this task. Furthermore, 4 of the 9
participants were able to come up with this pattern on their own. The others
found different patterns, some with greater precision but lower recall. For
instance, here is a pattern that has a precision of 1 and recall of 0.7:

⬇

def first_nonzero(lst):

for i in range(len(lst)):

if lst[i] != 0:

return lst[i]

return 0

\pardef contains_conflicting_pred(lst):

n = first_nonzero(lst)

for i in range(len(lst)):

if lst[i] != n and lst[i] != 0:

return True

return False

\pardef get_sequence_preds(lst):

return [item[3] for item in lst]

\parall_sequences = Query(’all_sequences’, base=’wilds_pred’).group_by(lambda
img, label, md, pred: md[1].item())

sequences_with_conflict = all_sequences.filter(lambda seq, lst:
contains_conflicting_pred(get_sequence_preds(lst)))(db)

##  Appendix F Limitations

Some of the limitations of the system include:

  1. 1.

Parallelism: MDB does not perform processes concurrently or parallelly and
does not take advantage of multicore systems with the exception of loading
data into the database. However, we leave this as future work. The structure
of the system makes it highly amenable to parallelism by integrating
multiprocessing components into the abstractions provided by MDB.

  2. 2.

In-Memory Database: MDB’s database is in-memory. This means that any data
being queried over is limited by the size of the RAM of the system, limiting
the ability of MDB to query over larger amounts of data. Traditionally
databases are not in-memory to prevent this problem with scalability, and load
chunks of data as needed, but this results in a loss of performance during the
I/O operations associated with reading from and writing to disk.

  3. 3.

Support for Synthesis: MDB is currently a new library and it is therefore
harder to synthesize MDB queries than typical Python. However, we have
obtained some initial success via few shot prompting where GitHub’s Copilot
and GPT-4 are able to synthesize some queries after giving examples. To help
with the interactive nature of debugging, additional support for synthesis,
such as the ability to synthesize UDFs or just the query structure is an area
of future work.