# AnnotatedTables: A Large Tabular Dataset with Language Model Annotations

Yaojie Hu
Department of Computer Science
Iowa State University
<EMAIL_ADDRESS>Ilias Fountalis
RelationalAI
<EMAIL_ADDRESS>Jin Tian
Department of Computer Science
Iowa State University
<EMAIL_ADDRESS>Nikolaos Vasiloglou
RelationalAI
<EMAIL_ADDRESS>
###### Abstract

Tabular data is ubiquitous in real-world applications and abundant on the web,
yet its annotation has traditionally required human labor, posing a
significant scalability bottleneck for tabular machine learning. In this
paper, we introduce methods that leverage Large Language Models (LLMs) to
understand and annotate tabular data. Our methodology can successfully
annotate a large amount of tabular data and can be flexibly steered to
generate various types of annotations based on specific research objectives,
as we demonstrate with SQL annotation and input-target column annotation as
examples. As a result, we release AnnotatedTables, a collection of 32,119
databases with LLM-generated annotations. The dataset includes 405,616 valid
SQL programs, making it the largest SQL dataset with associated tabular data
that supports query execution. To further demonstrate the value of our
methodology and dataset, we perform two follow-up research studies. 1) We
investigate whether LLMs can translate SQL programs to Rel programs, a
database language previously unknown to LLMs, while obtaining the same
execution results. Using our Incremental Prompt Engineering methods based on
execution feedback, we show that LLMs can produce adequate translations with
few-shot learning. 2) We evaluate the performance of TabPFN, a recent neural
tabular classifier trained on Bayesian priors, on 2,720 tables with input-
target columns identified and annotated by LLMs. On average, TabPFN performs
on par with the baseline AutoML method, though the relative performance can
vary significantly from one data table to another, making both models viable
for practical applications depending on the situation. Our findings underscore
the potential of LLMs in automating the annotation of large volumes of diverse
tabular data.

##  1 Introduction

Tabular data is one of the most common data types with pervasive real-world
applications such as finance, healthcare management, and marketing analytics
(Shwartz-Ziv & Armon, 2022; Borisov et al., 2022; Chui et al., 2018). The
tabular format can organize structured data as relational databases. Machine
learning models are able to understand and generate SQL queries against the
relational databases (Zhong et al., 2017; Yan et al., 2020), and large
language models (LLMs) have emerged as a prominent method in this field (Liu
et al., 2023a; Li et al., 2023a). Recent studies show that LLMs can achieve
state-of-the-art SQL generation with carefully engineered prompts (Gao et al.,
2023), owing thanks to the massive amount of open-source SQL code in their
training set (Li et al., 2023b). Although valuable for training LLMs, SQL code
alone is not enough for other tabular machine learning problems. For example,
to evaluate the text-to-SQL ability where the model generates SQL code based
on a description, both tabular data and SQL code would be required to
calculate the execution accuracy (Zhong et al., 2017; Xu et al., 2017; Chang &
Fosler-Lussier, 2023). While either SQL code or tabular data can be abundant,
obtaining both at the same time can be far more difficult. Not limited to SQL
code, annotations for tabular datasets can be generally difficult to collect,
which we call tabular dataset annotation problem in this paper. We use
executable SQL code as a lens to explore this tabular dataset annotation
problem in detail and propose our machine learning solutions.

WikiSQL (Zhong et al., 2017) and Spider (Yu et al., 2018b) are two popular
datasets with both SQL code and tabular data suitable for execution. However,
their sizes are limited, mainly because their annotation methodologies do not
scale up to make full use of either the tabular data or the SQL code on the
web. WikiSQL gathers its tabular data from Wikipedia pages, which is a limited
source and does not reflect the complications of real-world tabular data.
Also, its SQL programs are annotated manually by a crowd-sourcing service
called Amazon Mechanical Turk. Every crowd-sourced SQL program in WikiSQL only
queries one table at a time, without table-join operations. Spider, on the
other hand, does include relatively more complex queries on multiple tables in
a database, but it also relies on human annotations from 11 college students
to collectively write the SQL programs. Human annotation is common in all
tabular datasets with executable SQL code prior to our work (Table 1), an
expensive methodology to reproduce and expand upon (Hsueh et al., 2009).
However, recent work shows that LLMs are able to synthesize SQL code robustly
given instructions (Liu et al., 2023a), which motivates us to explore the
potential to use LLMs to synthesize the SQL code for the abundant, unannotated
tabular data. Our work focuses on SQL code, but SQL code is not the only type
of annotation that tabular machine learning systems may need. Can LLMs create
other types of annotations and generally replace expensive human annotations
for tabular datasets? Inspired by the multi-tasking (Srivastava et al., 2022)
and emergent abilities (Wei et al., 2022; Schaeffer et al., 2024) of LLMs, we
see both value and potential in tackling the tabular dataset annotation
problem.

Table 1: The sizes of the executable SQL datasets in the literature compared
to ours.

Dataset | SQL | DB
---|---|---
WikiSQL (Zhong et al., 2017) | 80654 | 26251
Spider (Yu et al., 2018b) | 9693 | 200
Cspider (Min et al., 2019) | 9691 | 166
IMDB (Yaghmazadeh et al., 2017) | 111 | 1
Yelp (Yaghmazadeh et al., 2017) | 68 | 1
SparC (Yu et al., 2019) | 15598 | 200
TableQA (Sun et al., 2020) | 49974 | 5291
DuSQL (Wang et al., 2020) | 23797 | 200
KaggleDBQA (Lee et al., 2021) | 272 | 8
Chase (Guo et al., 2021) | 17940 | 280
BIRD (Li et al., 2023a) | 12751 | 95
AnnotatedTables (Ours) | 405616 | 32119

In this paper, we introduce AnnotatedTables, a large scale tabular dataset
with large language model annotations. To overcome the bottleneck of labor-
intensive human annotations, we use LLMs to understand tabular data and
generate synthetic annotations. Our dataset construction starts with diverse
cross-domain tabular data encountered in real-world data science practices,
while maintaining general applicability to other tabular data sources. With
careful prompt design, we instruct LLMs to synthesize SQL code with zero-shot
learning, which, despite its large scale, proves to be a quick and cost-
effective method to create a rich dataset without expensive human labor. To
ensure the quality of LLM-generated SQL code, we verify them with execution on
the tabular data, and we report the validity of generated SQL queries for
analysis. To date, AnnotatedTables is the largest collection of SQL code that
can be executed on the associated tabular data (Table 1). SQL programs are not
the only annotations LLMs can produce. Some columns in a table may serve as
model input to predict others, and such input-target columns are usually
identified by people manually. The LLM annotation methodology can be extended
to annotate input-target columns for our tables, making them suitable for
tabular classification. Our work shows that LLMs can simplify and automate the
process of tabular dataset construction.

To validate the soundness and value of our dataset, we conduct two follow-up
studies. 1) We study LLM’s ability to learn a new programming language in-
context. 2) We scale up the evaluation of a recent tabular classification
model and test its performance on diverse real-world data. Both studies
demonstrate how LLM annotation can flexibly adapt our dataset to target
specific research problems. Our code and data will be released publicly.

The contributions of our paper are summarized as follows.

  * •

We introduce in-context learning methods of LLMs to understand and annotate a
large amount of real-world tabular data, addressing the scaling bottleneck of
traditional human annotation.

  * •

We use a LLM to annotate public data tables and construct a cross-domain
dataset named AnnotatedTables, currently the largest SQL dataset with tabular
data that supports query execution.

  * •

We extensively assess the quality of the LLM-generated annotations throughout
the study. We demonstrate the research value of AnnotatedTables dataset
through two follow-up studies.

  * •

We introduce Incremental Prompt Engineering to teach LLMs to translate SQL
programs to Rel programs with adequate accuracy, where Rel is a new language
not previously learned by LLMs.

  * •

We use the input-target columns annotated by LLMs to study TabPFN’s
performance on a wide variety of tabular classification problems. We find that
TabPFN performs overall on par with the baseline method, with each model
potentially performing better depending on the tables.

##  2 Related Work

#### Database management systems and deep learning

For text-to-SQL, language models can be trained or fine-tuned to synthesize
SQL queries given natural language descriptions (Yavuz et al., 2018; Hwang et
al., 2019; Yu et al., 2018a). For cardinality estimation, neural networks can
predict the number of rows returned by an intermediate operation for query
optimization (Liu et al., 2015; Negi et al., 2020; Woltmann et al., 2019; Yang
et al., 2019, 2020), and end-to-end neural query optimizers can be built
(Fischer et al., 2019; Gupta et al., 2020; Marcus et al., 2019, 2021). Other
deep learning for database problems include query answering (Hilprecht et al.,
2019), anomaly detection (Pang et al., 2021), ontology matching (Bento et al.,
2020), relational data modeling (Fan et al., 2020; Liu, 2022), and database
indexing (Kraska et al., 2018). All these problems could potentially benefit
from a large-scale multi-domain dataset like AnnotatedTables. LLM annotation
can also be applied to these problems to acquire a large amount of tabular
training data. Moreover, our dataset supports query execution, which is
required for text-to-SQL and cardinality estimation to collect ground truth.

#### SQL datasets with tabular data.

SQL annotation is a focus of our paper. AnnotatedTables is the largest SQL
dataset with associated tabular data that supports query execution. Currently,
the most commonly used SQL datasets with tabular data are listed in Table 1,
surveyed in (Deng et al., 2022; Qin et al., 2022; Li et al., 2023a). All of
them are annotated by humans, and our work is the first tabular dataset
annotated by LLMs. Our LLM methods can cheaply generate a large amount of
annotations that match the volume of tabular data available on the web.
Existing SQL datasets with tabular data often include English descriptions for
text-to-SQL evaluation. While we collect these descriptions, we do not assess
or ensure their quality. Unlike SQL programs, which can be validated through
execution (Yu et al., 2019), English descriptions cannot be executed.

#### Large Language Models for data annotation.

LLMs have been used for data annotation on text domains (Mao et al., 2024) and
can outperform human annoators (Gilardi et al., 2023). LLMs can detect
implicit hate speech and provide natural language explanations as annotations
(Huang et al., 2023). LLMs are also used for natural language generation
evaluation, showing a high alignment with human evaluation (Liu et al.,
2023b). Texts can be directly understood and annotated by LLMs, while tabular
data requires additional annotation methods, which we introduce.

##  3 AnnotatedTables

###  3.1 Dataset Construction

#### Collection of public data tables

The construction of AnnotatedTables starts by collecting public tabular data.
Our goal is to tap into the large quantity of un-annotated tabular data that
exist on the web, with a preference for curated tables encountered in real-
world data science applications. Kaggle, a data science platform, meets our
requirements (Risdal & Bozsolik, 2022; Quaranta et al., 2021; Wang et al.,
2021). Kaggle hosts data tables that are crowd sourced by a community of data
enthusiasts and professionals. Users can share and analyze datasets on a wide
variety of real-world topics, including football players, U.S airline flights,
renewable energy output, weather forecasting, and e-commerce product reviews.
Users may rate, improve, and analyze the datasets on Kaggle, contributing to
the curation of useable and reliable data. Although individual Kaggle datasets
have been studied in prior works (Bojer & Meldgaard, 2021; Taieb & Hyndman,
2014; Yang et al., 2018; Tolkachev et al., 2020; Yang & Ding, 2020), our
research aggregates and examines the data tables en masse, uncovering an
previously overlooked goldmine of diverse and high-quality data tables.

We use Kaggle’s public API to search for datasets with sizes ranging from 10
megabytes to 1 gigabyte for faster data collection and processing. We also
require every dataset’s “usability rating” to be higher than 0.1. The
“usability rating” is a quality score given by Kaggle users and helps us
filter out low-quality datasets and datasets with no usages. We collect 70,000
Kaggle datasets before stopping for further processing. Our LLM annotation
methodology is flexible and automated. The rest of our methods can be applied
to other data sources and is not limited to Kaggle datasets alone.

#### Describing databases to LLMs with schema and example rows

In order for LLMs to understand and annotate tabular data, we need to design a
textual description for each database as a part of the prompt. Using entire
tables in the prompt could exceed the maximum input length of LLMs, and we use
a much shorter description instead that contains a basic schema and example
rows.

Figure 1: An illustration of the SQL code annotation process with a large
language model.

Table schema is a summary description of its structure, commonly used to
define the table during its creation (Uschold, 2015). For our annotation
purposes, we use a basic schema that includes the table name, the table’s
column names, and the data type of each column (Figure 1). We find tabular
data stored as CSV files in the Kaggle datasets and automate schema
extraction. As CSV files do not include table names in their content, we opt
to use the file names as table names. Column names are included in the CSV
files as the header row. Upon inspection, we find that the column names and
file names are typically well-labeled and descriptive enough for LLMs to infer
the information in the table. Lastly, we use an existing data analysis
software (pandas) to load the tables and infer the column data types. Our
schema extraction method is practical and effective.

A Kaggle dataset may contain multiple tables and therefore multiple table
schemas. The set of table schemas describe the database structure: how many
tables there are, what the columns and types are, and whether a column appears
in multiple tables that could potentially be a JOIN key.

Example rows contain content from the table, and they are necessary for LLMs
to write many common SQL queries, such as WHERE statements that filter the
table based on a condition. For a concrete example, in SELECT * FROM CUSTOMERS
WHERE COUNTRY=‘MEXICO’, the table name CUSTOMERS and column name COUNTRY can
be found in the database schemas, while ‘MEXICO’ is a value that can only be
found in a table row. Providing row examples in addition to the schema is
necessary for LLMs to synthesize WHERE statements in SQL. For the example row,
we use the first row of each table, converted to text through JSON formatting.
Our schema and example row descriptions are generic and allow almost all
tabular data to be annotated by LLMs.

#### SQL annotations with zero-shot synthesis by LLMs

Through zero-shot learning, we instruct LLMs to synthesize SQL code that
queries the data tables we collected as one type of annotation. Currently,
OpenAI’s ChatGPT (Ouyang et al., 2022) is one of the best performing LLMs on
various benchmarks (Hendrycks et al., 2021; Zellers et al., 2019; Sakaguchi et
al., 2021). For our paper, we use ChatGPT with GPT-3.5 backbone as our
annotation LLM, referred to as “the LLM” from now on.

To perform zero-shot SQL annotation for each database, we create a prompt that
contains the instruction, the database schema, and the example row of every
table, illustrated in Figure 1. The instruction asks the LLM to write SQL
queries from “typical users who access this database”. In addition, the
instruction asks for queries “with relatively high complexity”, which steers
the synthesis results to more complicated SQL code and makes the dataset more
challenging. Lastly, the instruction asks the language model to create more
JOIN queries when multiple tables are present, as JOIN queries are especially
valuable data points when studying problems including cardinality estimation
and query optimization (Yang et al., 2020; Marcus et al., 2021). For every
Kaggle dataset with tabular data, we prompt the LLM to generate 15 SQL program
annotations with natural language annotations.

#### Execution-based validation

Some of the LLM-generated SQL programs could be incorrect, and it is important
to examine and ensure the quality of the LLM annotations. As AnnotatedTables
has tabular data available for execution, we verify the validity of the SQL
programs by executing them (Yu et al., 2019). If a SQL program can be executed
and returns non-empty results, then we classify it as a valid SQL annotation.
Executable queries with empty results are marked. In our follow-up study of
SQL-to-Rel translation, a pair of SQL and Rel queries with empty results
cannot be compared by execution accuracy, and SQL queries with empty results
will not be used. We do not validate the English descriptions. All
intermediate data artifacts are released as a part of AnnotatedTables.

Table 2: Examples of LLM-generated English and SQL annotations for Kaggle
tabular data. LLMs can generate SQL programs that cannot be executed, and we
mark the incorrect parts in red.

|  English Annotation |  SQL Annotation |  Execution
---|---|---|---
1 |  Get the average humidity for each location in the morning and afternoon: |  SELECT Location, AVG(Humidity9am) AS AverageHumidity9am, AVG(Humidity3pm) AS AverageHumidity3pm FROM WeatherAus GROUP BY Location; |  Valid
2 |  Retrieve rows where the tempo is greater than the average tempo: |  SELECT * FROM LikedsongsGV WHERE Tempo > (SELECT AVG(Tempo) FROM LikedsongsGV); |  Valid
3 |  Retrieve records from a table that meet a certain condition: |  SELECT * FROM table_name WHERE condition; |  Error
4 |  Retrieve rows where column 2 is less than 0.1 and column 5 is greater than 0.2: |  SELECT * FROM Preds001 WHERE 2 < 0.1 AND 5 > 0.2; |  Error
5 |  Get the total number of matches played in a specific season |  SELECT COUNT(*) FROM IPLMatches WHERE Season = 2019; |  Empty Result

Table 3: SQL code synthesized by LLM categorized by the types of SQL
components.

Component | Total | Valid | % Valid | % Error | % Empty
---|---|---|---|---|---
Total | 493134 | 405616 | 82.25 | 9.96 | 7.79
FROM | 493093 | 405612 | 82.26 | 7.78 | 9.96
WHERE | 276710 | 209393 | 75.67 | 8.07 | 16.26
AND | 88182 | 61814 | 70.10 | 10.81 | 19.09
GROUP BY | 77004 | 66061 | 85.79 | 9.74 | 4.47
AVG | 65797 | 58238 | 88.51 | 8.83 | 2.66
COUNT | 58586 | 53486 | 91.29 | 6.41 | 2.29
ORDER BY | 52677 | 45941 | 87.21 | 8.98 | 3.81
LIKE | 38865 | 28189 | 72.53 | 5.90 | 21.57
JOIN | 30079 | 14879 | 49.47 | 23.00 | 27.53
MAX | 29693 | 26188 | 88.20 | 9.29 | 2.52
SUM | 24801 | 21324 | 85.98 | 10.14 | 3.87
LIMIT | 24241 | 21573 | 88.99 | 8.23 | 2.77
NOT | 19180 | 15111 | 78.79 | 7.22 | 13.99
IN | 16011 | 9656 | 60.31 | 23.76 | 15.93
MIN | 12368 | 10938 | 88.44 | 9.38 | 2.18
OR | 11199 | 8346 | 74.52 | 11.09 | 14.39
UNION | 10408 | 7375 | 70.86 | 24.95 | 4.19
HAVING | 2507 | 1427 | 56.92 | 15.96 | 27.12
INTERSECT | 184 | 101 | 54.89 | 10.33 | 34.78
EXCEPT | 37 | 26 | 70.27 | 13.51 | 16.22

###  3.2 Features, Statistics, and Examples of AnnotatedTables

#### Human-like annotation

We find that the LLM can write SQL programs with human-like intent and usage.
Examples of the LLM-annotated SQL programs are provided in Table 2. Instructed
by our prompt to play the role of a typical user of a database, the LLM writes
meaningful queries to look for useful data analytics.

#### Correctness and complexity

Given a high percentage (82.25%) of valid SQL programs, the annotation quality
is good. The LLM can write complex programs with a variety of SQL components
(Table 3). The percentages of valid queries in all SQL component categories
except JOIN exceed 50%. 49.47% of JOIN queries are valid, possibly reflecting
a higher level of data understanding and programming skills required to
combine multiple tables.

For limitations, we note that some simple SQL statements appear often, in the
form of SELECT * FROM table, reflected by a high percentage (48.4% among valid
SQL programs) of FROM statements without other SQL components. Although valid,
they may not be as valuable as the more complicated SQL programs for
downstream learning and analysis. In the prompt, we steer the LLM to generate
SQL programs “with relative high complexity” and see success to some extent
(Appendix B). However, the dual objectives to generate complex queries and
correct queries may inherently conflict with each other, and LLMs could be
inclined to generate simple ones to prioritize correctness over complexity.

#### Invalid SQL annotations

As shown in Table 2, some LLM-annotated SQL programs may be invalid for
different reasons. At times, the LLM hallucinates non-existing table names
such as “table_name” (Example 3) or leaves a place holder text “condition”
instead of the exact selection condition (Example 3). In some CSV files, the
column names may be numerical, which are invalid column names in a SQL
statement, and the LLM is unable to annotate (Example 4). Lastly, as execution
result needs to be non-empty so that execution accuracy can be computed for
downstream tasks, some syntactically correct SQL programs that return empty
results are labeled (Example 5).

##  4 AnnotatedTables for In-Context Translation from SQL to Rel

We use AnnotatedTables to study large language models’ ability to learn to
code in Rel, a new database language (RelationalAI, 2021). For new programming
languages, there is not much open-source code available to train machine
learning models with gradient-descent methods, and, with few training
examples, few-shot in-context learning (ICL) becomes a natural candidate
instead (Min et al., 2022). ICL has been studied, but ICL of a new programming
language is a novel research problem. New programming languages are rare, and
the lens of this study is unique and timely.

All SQL programs in AnnotatedTables are executable, which is necessary in
order to evaluate the correctness of the translations with execution accuracy.
AnnotatedTables contains many SQL programs and data tables, which can measure
the general performance of our SQL-to-Rel translation method. SQL programs in
AnnotatedTables emulate the usage and intent of typical database users, and,
by translating these queries, we indirectly synthesize a large dataset of non-
trivial Rel programs that is valuable for downstream analysis and machine
learning of the new programming language.

Machine learning of a new programming language may seem challenging with
limited training data, but as our results show, our translation accuracy
increases steadily as more in-context examples are included, and with 45
translation pairs in the prompt, the translation accuracy can reach around
40%.

###  4.1 Task Formulation of SQL-to-Rel Translation

Given a SQL program $S$ that can be executed on table $T$, the task is to
build a model $\mathcal{M}$ that learns to translate $S$ into a Rel program
$R$, such that $S$ and $R$ return the same query results when executed on
table $T$. In our case, the model $\mathcal{M}$ is a LLM parameterized by an
instruction prompt that contains few-shot examples. Execution accuracy is the
percentage of queries $S,R$ that return the same non-empty results. Execution
accuracy is a common metric for SQL generation (Zhong et al., 2017; Xu et al.,
2017; Chang & Fosler-Lussier, 2023). Program equivalence is generally
undecidable (Rice, 1953; Strichman, 2018), and it is not possible to directly
compare SQL and Rel programs.

###  4.2 Extending AnnotatedTables with Few-shot SQL-to-Rel Translation Pairs

While LLMs have already learned to program in SQL from comprehensive pre-
training (Brown et al., 2020; OpenAI, 2023), they do not know how to write
programs in Rel. To enable ICL, one idea is to use the Rel language
specification and documentation as the prompt. However, language models such
as ChatGPT have limited context span up to 16,000 tokens (around 12,000
English words) (Tay et al., 2020; Beltagy et al., 2020), and it is currently
infeasible for the prompt to contain the complete Rel specification and
documentation (estimated 500,000 words). Instead, we use SQL-to-Rel
translation examples to teach the LLM to code in Rel, which is simple and
effective. Rel is a declarative language similar to Datalog and Prolog. LLMs
may transfer their background knowledge in related languages to learn Rel. ICL
through translation examples cannot comprehensively cover all the edge cases
of the Rel language, but, as our results show, around 40% of our SQL code can
be successfully translated.

Table 4: Examples of the SQL-to-Rel translations produced, categorized by
“correct translations”, “execution errors”, and “different results”.

|  SQL |  Rel Translated |  Ex. Acc.
---|---|---|---
1 |  SELECT Departments, MAX(AverageMontlyHours) AS MaxMonthlyHours FROM Attrition GROUP BY Departments; |  def output[departments] = max[row_id, avg_monthly_hours: Attrition:AverageMontlyHours(row_id, avg_monthly_hours) and Attrition:Departments[row_id, departments]] |  Correct
2 |  SELECT (SELECT COUNT(*) FROM Oof) + (SELECT COUNT(*) FROM Submission) as TotalCount; |  def count_oof_rows = count[row_id: Oof[_, row_id, _]]
def count_submission_rows = count[row_id: Submission[_, row_id, _]]
def output = count_oof_rows + count_submission_rows |  Correct
3 |  SELECT COUNT(*) FROM DgemFormatValidation; |  def count_rows = count[row_id: DgemFormatValidation[_, row_id, _]]
def output = count_rows |  Correct
4 |  SELECT Education, AVG(Age) AS avg_age FROM MLPdata GROUP BY Education; |  def output[education] = average[row_id, age: MLPdata:Education[row_id, education] and MLPdata:Age[row_id, age]] groupby education |  Error
5 |  SELECT COUNT(*) FROM Dict; |  def output = count[row, _ : Dict[_, row, _]] |  Error
6 |  SELECT * FROM V9d3 ORDER BY Labels DESC; |  def output[col, row_id, val] = V9d3[col, row_id, val] orderby V9d3:Labels[row_id] desc |  Error
7 |  SELECT MAX(LENGTH(Description)) FROM Cryptosentiment; |  def output = max[row_id, length: Cryptosentiment:Description[row_id, length]] |  Diff. Results
8 |  SELECT * FROM Comments20200605 WHERE ReplyCount >= 5; |  def output[col, row_id, val] = Comments20200605[col, row_id, val] and Comments20200605:ReplyCount[row_id] >= 5 |  Diff. Results
9 |  SELECT AVG(MonthlyRevenue) FROM Cell2celltrain; |  def output = average[idx, monthly_revenue: Cell2celltrain:MonthlyRevenue [idx, monthly_revenue]] |  Diff. Results

#### Incremental Prompt Engineering of translation examples

To develop SQL-to-Rel translation examples to teach LLMs to code in Rel, we
introduce a technique called Incremental Prompt Engineering (IPE). For step
$i=1,2,...,k$, we start with some manually annotated SQL-to-Rel translation
examples $E_{i}$ and instruct the language model to translate some unseen SQL
programs $S_{i}$ to Rel programs $R_{i}$. We then calculate the execution
accuracy by comparing the query results of $S_{i}$ and $R_{i}$ at every step
$i$. The execution accuracy is used as a feedback, and for the SQL code in
$S_{i}$ that the LLM fails to translate, we manually annotate the ones that
represent common failure patterns to add to $E_{i}$ as the few-shot examples
$E_{i+1}$ at the next step. In practice, we start with 5 translation examples
and add multiple manual annotations at a time until the execution accuracy
converges with around 45 translation examples (Figure 2). All examples used in
the prompt are in Appendix A.

###  4.3 Results

Table 5: The LLM’s few-shot SQL-to-Rel translation execution results, by the
types of SQL components.

SQL Comp. | # Ex. | Translated | Correct | % Correct | % Error | % Diff.
---|---|---|---|---|---|---
Total | 45 | 285140 | 116327 | 40.80 | 53.60 | 5.60
FROM | 45 | 285136 | 116327 | 40.80 | 53.60 | 5.60
WHERE | 28 | 141893 | 49576 | 34.94 | 59.02 | 6.04
GROUP BY | 9 | 40254 | 7737 | 19.22 | 78.36 | 2.42
AND | 7 | 37963 | 13086 | 34.47 | 59.72 | 5.81
AVG | 5 | 37318 | 12601 | 33.77 | 60.08 | 6.16
COUNT | 6 | 36769 | 17542 | 47.71 | 48.90 | 3.39
ORDER BY | 2 | 28618 | 434 | 1.52 | 97.74 | 0.75
LIKE | 1 | 18399 | 2037 | 11.07 | 82.67 | 6.26
MAX | 2 | 17984 | 7698 | 42.80 | 52.46 | 4.74
SUM | 5 | 14107 | 4077 | 28.90 | 69.69 | 1.41
LIMIT | 0 | 13696 | 73 | 0.53 | 98.53 | 0.94
NOT | 0 | 8754 | 326 | 3.72 | 93.24 | 3.04
JOIN | 8 | 8358 | 526 | 6.29 | 88.32 | 5.38
MIN | 0 | 7344 | 3043 | 41.44 | 55.51 | 3.05
IN | 2 | 5365 | 1045 | 19.48 | 76.07 | 4.45
UNION | 0 | 4530 | 263 | 5.81 | 88.92 | 5.28
OR | 1 | 4071 | 1292 | 31.74 | 60.21 | 8.06
HAVING | 0 | 677 | 1 | 0.15 | 99.26 | 0.59
INTERSECT | 0 | 56 | 2 | 3.57 | 92.86 | 3.57
EXCEPT | 0 | 11 | 1 | 9.09 | 81.82 | 9.09

#### The LLM can learn a new programming language adequately from dozens of
examples

Table 4 shows examples of SQL programs and their Rel translations produced by
the LLM. As seen in the examples, the two languages differ greatly, and it is
not an easy task to translate them, even for experienced programmers. For
example, SQL aggregators such as MAX have their counterpart max function in
Rel, but the syntax differs greatly (Example 1). Nevertheless, the LLM is able
to perform adequate translations by learning a new programming language from
just 45 examples in-context. Aggregators including MAX, MIN, and COUNT have
higher-than-average translation accuracy. This is an impressive result for a
challenging problem. Notably, no manual examples illustrate how to translate
MIN queries, and LLM has inferred its usage from MAX examples.

Execution accuracy measures the translation correctness, where every SQL and
Rel pair are executed on the same database and compare their results. Table 5
shows that the overall execution accuracy is 40.8%, showing that the LLM can
generate adequate translations from SQL to Rel. Machine learning of new
programming languages is challenging and valuable. Since the best text-to-SQL
models have 60%-80% accuracy (Liu et al., 2023a; Li et al., 2023a), 40% SQL-
to-Rel is a promising first step, which could be sufficient for applications
such as coding suggestions and automatic test generation.

#### Translation errors and limitations

53.6% of the Rel translations have errors before returning results, while 5.6%
of the Rel translations finish execution but have different results (Table 5).
This suggests that most of the failed translations have syntactical problems
and cannot be executed. Examples of inaccurate translations are in Table 4.
LLMs may use SQL language constructs such as “groupby”, “orderby”, and “desc”
directly in Rel programs (Example 4, 6), causing syntactically incorrect
programs. In Example 5, the first underscore “_” caused a syntax error. In
Example 7, the SQL method “LENGTH” is not demonstrated in-context, and the LLM
used “length” as a variable name instead, causing a change in semantics and
different execution results. Examples 8 and 9 show different execution
results, but the translations are correct by our inspection. Rel is a language
based on the Sixth-Normal-Form (6NF) (Date et al., 2004; Knowles, 2012), which
represents all data as sets of tuples with only one value column, without null
values or empty rows. Rel’s data representation is not the same as SQL’s data
representation, and we make conservative conversions when comparing the
results to prioritize precision over recall by allowing some false negatives
in execution accuracy. Therefore, some correct Rel translations may have
different results due to the conversion, though such cases are rare, as the
percentage of Rel errors is 10x higher than different results. Discounting
this limitation, the SQL-to-Rel translation accuracy may be even higher than
that reported in Table 5.

#### Performance convergence of Incremental Prompt Engineering

To study the effectiveness of IPE, we vary the number of few-shot examples
incrementally added and measure the translation accuracy on a randomly
selected subset of 2,000 SQL programs. We plot the execution accuracy versus
the number of translation examples in Figure 2.

Figure 2: With Incremental Prompt Engineering, translation accuracy gradually
improves and converges as more examples are added.

Conceptually, Incremental Prompt Engineering adds demonstrations for the
failed translations in the previous step and integrates execution feedback
into prompt engineering. It is an effective way to improve translation
accuracy, as we see a clear increase in the percentage of correctly translated
Rel programs as more examples are added. When the number of few-shot examples
increases above 30, we see a diminishing return on the accuracy improvements.
At 45 examples, the accuracy converges around 40%. Due to the convergence,
more examples may not improve the accuracy beyond 40%, and additional prompt
engineering or LLM techniques may be required.

##  5 AnnotatedTables for an Extended Evaluation of TabPFN

In this section, we use AnnotatedTables to comprehensively evaluate TabPFN’s
tabular classification performance on diverse, real-world data tables. TabPFN
(Hollmann et al., 2023) is a Prior-Data Fitted Network (PFN), a new type of
neural networks trained on synthetic data drawn from a Bayesian prior. Once
trained, it takes the entire test set as the input and produces all
predictions in a single forward pass. TabPFN represents a novel approach to
tabular classification, a domain dominated by traditional methods and largely
uncontested by deep learning methods (Shwartz-Ziv & Armon, 2022). Inspired by
its state-of-the-art results on a small curated benchmark (Hollmann et al.,
2023), we wonder if TabPFN will perform robustly against the complications of
real-world tabular data.

#### TabPFN evaluation and ours

TabPFN was evaluated on the OpenML-CC18 benchmark suite (Bischl et al., 2017).
The benchmark has 30 datasets in total, and 18 of them have only numerical
features with no missing values, the same as the Bayesian prior that TabPFN is
trained on. The authors choose five standard ML methods and two state-of-the-
art AutoML methods as baselines to compare with TabPFN. The ML methods include
K-nearest-neighbors (KNN), logistic regression, and three gradient boosting
methods – XGBoost (Chen & Guestrin, 2016), CatBoost (Prokhorenkova et al.,
2018), LightGBM (Ke et al., 2017). The AutoML baseline methods are AutoGluon
(Erickson et al., 2020; Shi et al., 2021) and AutoSkLearn (Feurer et al.,
2022). On the 18 numerical datsets, TabPFN outperformed all baselines, and
AutoGluon was the best performing baseline (see Table 1 of (Hollmann et al.,
2023)), which we will use as the baseline in our study. In this section, we
follow TabPFN authors’ experimental settings and use AnnotatedTables to vastly
scale up the evaluation, with a focus on assessing its real-world performance.
We evaluate TabPFN on 2,720 tabular classification problems, a 90x increase
over OpenML-CC18. Moreover, in comparison to the OpenML-CC18 benchmark,
AnnotatedTables contains diverse tabular data gathered from a public platform,
which would be a more comprehensive evaluation of TabPFN’s robustness in real
world problems.

###  5.1 Task Formulation of Tabular Classification

For tabular classification, a table $T$ has rows $\\{R_{1},R_{2},...\\}$. A
row $R_{i}$ has one or more column values $\mathbf{x}_{i}$ and a categorical
column value $y_{i}$. A tabular classification model $f$ aims to use
$\mathbf{x}_{i}$ to predict $y_{i}$.

Our experimental settings follow the TabPFN authors’ settings. We train
AutoGluon with a time budget of one minute and five minutes per table, and we
split the training and test set once per table. To reduce the total training
time as we have many more tables than OpenML-CC18, we do not repeat training
on every table with a five-fold training-test set split. We use AutoGluon as
our baseline, because it is the best performing baseline model in the original
evaluation (Hollmann et al., 2023). Our work does not alter the TabPFN model
architecture, and we use the authors’ publicly released pre-trained
checkpoint. For more model details, we refer readers to the paper (Hollmann et
al., 2023).

###  5.2 Extending AnnotatedTables with LLM-Annotated Input-Target Columns

The Kaggle data tables collected in AnnotatedTables could be used for tabular
classification task, if LLMs can identify the potential input and target
columns. We give the schema description from Sec 3.1 to the LLM to identify
potential input and target columns in the tables. TabPFN requires that the
input columns are numeric and the target column is categorical for
classification. LLM annotation is flexible, and we instruct the LLM to find
columns that satisfy this data type requirement (for complete prompt, see
Appendix A). We use the input-target columns annotated to train TabPFN and
AutoGluon, and if the models cannot be trained, the annotations are discarded
(Appendix E).

###  5.3 Results

Table 6: The AUROC (one-versus-one), cross entropy, and time statistics for
TabPFN and the AutoGluon baseline on the tabular classification problems in
AnnotatedTables.

|  | 1 Min (2725 problems) | 5 Mins (2720 problems)
---|---|---|---
|  | TabPFN | AutoGluon | Paired Diff. | TabPFN | AutoGluon | Paired Diff.
AUROC | Mean | 0.760 $\pm$ 0.173 | 0.723 $\pm$ 0.214 | 0.036 $\pm$ 0.152 | 0.760 $\pm$ 0.174 | 0.755 $\pm$ 0.187 | 0.005 $\pm$ 0.095
Median | 0.794 | 0.767 | 0.004 | 0.793 | 0.794 | -0.000
CE | Mean | 0.753 $\pm$ 0.550 | 0.886 $\pm$ 2.129 | -0.133 $\pm$ 1.944 | 0.753 $\pm$ 0.550 | 0.807 $\pm$ 1.929 | -0.054 $\pm$ 1.726
Median | 0.655 | 0.693 | -0.011 | 0.656 | 0.659 | 0.001
Time (sec.) | Mean | 2.002 | 73.654 | -71.652 | 2.342 | 315.038 | -312.696

Figure 3: The AUROC (OVO) of TabPFN versus baseline AutoGluon with 1-minute
time budget on the tabular classification problems in AnnotatedTables.

#### TabPFN performs robustly on diverse tabular data

We fit TabPFN and AutoGluon on 2,720 tabular classification problems in
AnnotatedTables, and we compute the AUROC (one-versus-one) and cross entropy
for every one of them, shown in Table 6. WilCoxon signed-rank test shows that
all the paired differences between TabPFN and AutoGluon for both AUROC and
cross entropy and for both 1 minute and 5 minutes are non-zero with
statistical significance (p-values and discussions in Appendix D).
AnnotatedTables allows us to conduct a large number of experiments to draw
statistically significant conclusions, but the paired differences are small
and may not be informative for model selection, an issue known generally as
“statistical significance versus practical significance” (Kirk, 1996; Peeters,
2016; McShane & Gelman, 2022). Overall, TabPFN performs on par with the
AutoGluon baseline.

The classification problems in AnnotatedTables are diverse and have lots of
inherent variations, which is reflected by the standard deviation of both
models’ performance. Between the two, TabPFN has a lower standard deviation
for both AUROC and cross entropy, showing a more consistent performance.

The inherent diversity of AnnotatedTables can be seen from the dot plots in
Figure 3, where we plot the AUROC of both models versus each other. Given a
relatively high spread of AUROC and overall on-par performance, it is not
possible to pre-determine which model will perform better, and either model
could be better on a specific table. Practitioners may fit both models and
select the better performing one in practice. Given that AutoGluon is a widely
used AutoML framework, we believe that the newly introduced TabPFN is a
welcome addition that would raise the state-of-the-art performance on many
real-world tabular classification problems, especially as it takes only a
fraction of the time budget per table (2 seconds) compared to AutoGluon (1 or
5 minutes in our experiments).

#### LLM annotation quality

Running TabPFN and AutoGluon on LLM-generated annotation proves that LLMs can
successfully identify potential input and target columns in a table and
annotate them, given only a prompt that describes each table’s name, column
names and data types. As seen in Figure 3, most of the tabular classification
problems annotated by LLMs have an AUROC over 0.5 for both TabPFN and
AutoGluon. This means, for most tables, the target columns identified by LLMs
do have non-trivial relationships with the input columns and can be predicted
to some extent, and the LLM successfully found the suitable tables for the
tabular classification task.

##  6 Conclusions and Discussions

Prior to this work, human labor had been necessary to acquire tabular dataset
annotations. Large language models can bring a radical change to how machine
learning datasets are constructed, and AnnotatedTables is the first to pioneer
tabular dataset annotation with LLMs. LLMs can be steered to generate
annotations based on detailed instructions, and tabular machine learning
researchers could easily acquire tailored annotations that fit their research
directions. With the human annotation bottleneck opened, our methodology could
enable more studies with reduced costs and automation.

## References

  * Beltagy et al. (2020) Iz Beltagy, Matthew E Peters, and Arman Cohan.  Longformer: The long-document transformer.  _arXiv preprint arXiv:2004.05150_ , 2020.
  * Bento et al. (2020) Alexandre Bento, Amal Zouaq, and Michel Gagnon.  Ontology matching using convolutional neural networks.  In _Proceedings of the Twelfth Language Resources and Evaluation Conference_ , pp. 5648–5653, 2020.
  * Bischl et al. (2017) Bernd Bischl, Giuseppe Casalicchio, Matthias Feurer, Pieter Gijsbers, Frank Hutter, Michel Lang, Rafael G Mantovani, Jan N van Rijn, and Joaquin Vanschoren.  Openml benchmarking suites.  _arXiv preprint arXiv:1708.03731_ , 2017.
  * Bojer & Meldgaard (2021) Casper Solheim Bojer and Jens Peder Meldgaard.  Kaggle forecasting competitions: An overlooked learning opportunity.  _International Journal of Forecasting_ , 37(2):587–603, 2021.
  * Borisov et al. (2022) Vadim Borisov, Tobias Leemann, Kathrin Seßler, Johannes Haug, Martin Pawelczyk, and Gjergji Kasneci.  Deep neural networks and tabular data: A survey.  _IEEE Transactions on Neural Networks and Learning Systems_ , 2022\.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901, 2020.
  * Chang & Fosler-Lussier (2023) Shuaichen Chang and Eric Fosler-Lussier.  How to prompt LLMs for text-to-SQL: A study in zero-shot, single-domain, and cross-domain settings.  In _NeurIPS 2023 Second Table Representation Learning Workshop_ , 2023\.  URL https://openreview.net/forum?id=5sOZNkkKh3.
  * Chen & Guestrin (2016) Tianqi Chen and Carlos Guestrin.  Xgboost: A scalable tree boosting system.  In _Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining_ , pp. 785–794, 2016.
  * Chui et al. (2018) Michael Chui, James Manyika, Mehdi Miremadi, Nicolaus Henke, Rita Chung, Pieter Nel, and Sankalp Malhotra.  Notes from the ai frontier: Insights from hundreds of use cases.  _McKinsey Global Institute_ , 2:267, 2018.
  * Date et al. (2004) CJ Date, Hugh Darwen, and Nikos A Lorentzos.  A detailed investigation into the application of interval and relation theory to the problem of temporal database management, 2004.
  * Deng et al. (2022) Naihao Deng, Yulong Chen, and Yue Zhang.  Recent advances in text-to-SQL: A survey of what we have and what we expect.  In Nicoletta Calzolari, Chu-Ren Huang, Hansaem Kim, James Pustejovsky, Leo Wanner, Key-Sun Choi, Pum-Mo Ryu, Hsin-Hsi Chen, Lucia Donatelli, Heng Ji, Sadao Kurohashi, Patrizia Paggio, Nianwen Xue, Seokhwan Kim, Younggyun Hahm, Zhong He, Tony Kyungil Lee, Enrico Santus, Francis Bond, and Seung-Hoon Na (eds.), _Proceedings of the 29th International Conference on Computational Linguistics_ , pp. 2166–2187, Gyeongju, Republic of Korea, October 2022. International Committee on Computational Linguistics.  URL https://aclanthology.org/2022.coling-1.190.
  * Erickson et al. (2020) Nick Erickson, Jonas Mueller, Alexander Shirkov, Hang Zhang, Pedro Larroy, Mu Li, and Alexander Smola.  Autogluon-tabular: Robust and accurate automl for structured data.  _arXiv preprint arXiv:2003.06505_ , 2020.
  * Fan et al. (2020) Ju Fan, Tongyu Liu, Guoliang Li, Junyou Chen, Yuwei Shen, and Xiaoyong Du.  Relational data synthesis using generative adversarial networks: A design space exploration.  _arXiv preprint arXiv:2008.12763_ , 2020.
  * Feurer et al. (2022) Matthias Feurer, Katharina Eggensperger, Stefan Falkner, Marius Lindauer, and Frank Hutter.  Auto-sklearn 2.0: Hands-free automl via meta-learning.  _Journal of Machine Learning Research_ , 23(261):1–61, 2022.
  * Fischer et al. (2019) Marc Fischer, Mislav Balunovic, Dana Drachsler-Cohen, Timon Gehr, Ce Zhang, and Martin Vechev.  Dl2: training and querying neural networks with logic.  In _International Conference on Machine Learning_ , pp. 1931–1941. PMLR, 2019.
  * Gao et al. (2023) Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou.  Text-to-sql empowered by large language models: A benchmark evaluation.  _arXiv preprint arXiv:2308.15363_ , 2023.
  * Gilardi et al. (2023) Fabrizio Gilardi, Meysam Alizadeh, and Maël Kubli.  Chatgpt outperforms crowd workers for text-annotation tasks.  _Proceedings of the National Academy of Sciences_ , 120(30):e2305016120, 2023.
  * Guo et al. (2021) Jiaqi Guo, Ziliang Si, Yu Wang, Qian Liu, Ming Fan, Jian-Guang Lou, Zijiang Yang, and Ting Liu.  Chase: A large-scale and pragmatic chinese dataset for cross-database context-dependent text-to-sql.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 2316–2331, 2021.
  * Gupta et al. (2020) Udit Gupta, Samuel Hsia, Vikram Saraph, Xiaodong Wang, Brandon Reagen, Gu-Yeon Wei, Hsien-Hsin S Lee, David Brooks, and Carole-Jean Wu.  Deeprecsys: A system for optimizing end-to-end at-scale neural recommendation inference.  In _2020 ACM/IEEE 47th Annual International Symposium on Computer Architecture (ISCA)_ , pp. 982–995. IEEE, 2020.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=d7KBjmI3GmQ.
  * Hilprecht et al. (2019) Benjamin Hilprecht, Andreas Schmidt, Moritz Kulessa, Alejandro Molina, Kristian Kersting, and Carsten Binnig.  Deepdb: Learn from data, not from queries!  _arXiv preprint arXiv:1909.00607_ , 2019.
  * Hollmann et al. (2023) Noah Hollmann, Samuel Müller, Katharina Eggensperger, and Frank Hutter.  TabPFN: A transformer that solves small tabular classification problems in a second.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=cp5PvcI6w8_.
  * Hsueh et al. (2009) Pei-Yun Hsueh, Prem Melville, and Vikas Sindhwani.  Data quality from crowdsourcing: a study of annotation selection criteria.  In _Proceedings of the NAACL HLT 2009 workshop on active learning for natural language processing_ , pp. 27–35, 2009.
  * Huang et al. (2023) Fan Huang, Haewoon Kwak, and Jisun An.  Is chatgpt better than human annotators? potential and limitations of chatgpt in explaining implicit hate speech.  In _Companion proceedings of the ACM web conference 2023_ , pp. 294–297, 2023.
  * Hwang et al. (2019) Wonseok Hwang, Jinyeong Yim, Seunghyun Park, and Minjoon Seo.  A comprehensive exploration on wikisql with table-aware word contextualization.  _arXiv preprint arXiv:1902.01069_ , 2019.
  * Ke et al. (2017) Guolin Ke, Qi Meng, Thomas Finley, Taifeng Wang, Wei Chen, Weidong Ma, Qiwei Ye, and Tie-Yan Liu.  Lightgbm: A highly efficient gradient boosting decision tree.  _Advances in neural information processing systems_ , 30, 2017.
  * Kirk (1996) Roger E. Kirk.  Practical significance: A concept whose time has come.  _Educational and Psychological Measurement_ , 56(5):746–759, 1996.  doi: 10.1177/0013164496056005002.  URL https://doi.org/10.1177/0013164496056005002.
  * Knowles (2012) Curtis Knowles.  6nf conceptual models and data warehousing 2.0.  2012\.
  * Kraska et al. (2018) Tim Kraska, Alex Beutel, Ed H Chi, Jeffrey Dean, and Neoklis Polyzotis.  The case for learned index structures.  In _Proceedings of the 2018 international conference on management of data_ , pp. 489–504, 2018.
  * Lee et al. (2021) Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson.  KaggleDBQA: Realistic evaluation of text-to-SQL parsers.  In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli (eds.), _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pp. 2261–2273, Online, August 2021. Association for Computational Linguistics.  doi: 10.18653/v1/2021.acl-long.176.  URL https://aclanthology.org/2021.acl-long.176.
  * Li et al. (2023a) Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al.  Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls.  _Advances in Neural Information Processing Systems_ , 36, 2023a.
  * Li et al. (2023b) Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al.  Starcoder: may the source be with you!  _arXiv preprint arXiv:2305.06161_ , 2023b.
  * Liu et al. (2023a) Aiwei Liu, Xuming Hu, Lijie Wen, and Philip S Yu.  A comprehensive evaluation of chatgpt’s zero-shot text-to-sql capability.  _arXiv preprint arXiv:2303.13547_ , 2023a.
  * Liu et al. (2015) Henry Liu, Mingbin Xu, Ziting Yu, Vincent Corvinelli, and Calisto Zuzarte.  Cardinality estimation using neural networks.  In _Proceedings of the 25th Annual International Conference on Computer Science and Software Engineering_ , pp. 53–59, 2015.
  * Liu (2022) Linfeng Liu.  _Learning from Relational Data via Graph Neural Networks_.  PhD thesis, Tufts University, 2022.
  * Liu et al. (2023b) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu.  G-eval: NLG evaluation using gpt-4 with better human alignment.  In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pp. 2511–2522, Singapore, December 2023b. Association for Computational Linguistics.  doi: 10.18653/v1/2023.emnlp-main.153.  URL https://aclanthology.org/2023.emnlp-main.153.
  * Mao et al. (2024) Rui Mao, Guanyi Chen, Xulang Zhang, Frank Guerin, and Erik Cambria.  GPTEval: A survey on assessments of ChatGPT and GPT-4.  In Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, and Nianwen Xue (eds.), _Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)_ , pp. 7844–7866, Torino, Italia, May 2024. ELRA and ICCL.  URL https://aclanthology.org/2024.lrec-main.693.
  * Marcus et al. (2019) Ryan Marcus, Parimarjan Negi, Hongzi Mao, Chi Zhang, Mohammad Alizadeh, Tim Kraska, Olga Papaemmanouil, and Nesime Tatbul.  Neo: A learned query optimizer.  _arXiv preprint arXiv:1904.03711_ , 2019.
  * Marcus et al. (2021) Ryan Marcus, Parimarjan Negi, Hongzi Mao, Nesime Tatbul, Mohammad Alizadeh, and Tim Kraska.  Bao: Making learned query optimization practical.  In _Proceedings of the 2021 International Conference on Management of Data_ , pp. 1275–1288, 2021.
  * McShane & Gelman (2022) Blakeley B McShane and Andrew Gelman.  Selecting on statistical significance and practical importance is wrong.  _Journal of Information Technology_ , 37(3):312–315, 2022.
  * Min et al. (2019) Qingkai Min, Yuefeng Shi, and Yue Zhang.  A pilot study for Chinese SQL semantic parsing.  In Kentaro Inui, Jing Jiang, Vincent Ng, and Xiaojun Wan (eds.), _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 3652–3658, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1377.  URL https://aclanthology.org/D19-1377.
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer.  Rethinking the role of demonstrations: What makes in-context learning work?  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pp. 11048–11064, 2022.
  * Negi et al. (2020) Parimarjan Negi, Ryan Marcus, Hongzi Mao, Nesime Tatbul, Tim Kraska, and Mohammad Alizadeh.  Cost-guided cardinality estimation: Focus where it matters.  In _2020 IEEE 36th International Conference on Data Engineering Workshops (ICDEW)_ , pp. 154–157. IEEE, 2020.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report, 2023.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al.  Training language models to follow instructions with human feedback.  _Advances in neural information processing systems_ , 35:27730–27744, 2022.
  * Pang et al. (2021) Guansong Pang, Chunhua Shen, Longbing Cao, and Anton Van Den Hengel.  Deep learning for anomaly detection: A review.  _ACM computing surveys (CSUR)_ , 54(2):1–38, 2021\.
  * PapersWithCode (2024) PapersWithCode.  Table annotation, 2024.  URL https://paperswithcode.com/task/table-annotation.
  * Peeters (2016) Michael J Peeters.  Practical significance: Moving beyond statistical significance.  _Currents in Pharmacy Teaching and Learning_ , 8(1):83–89, 2016.
  * Prokhorenkova et al. (2018) Liudmila Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin.  Catboost: unbiased boosting with categorical features.  _Advances in neural information processing systems_ , 31, 2018.
  * Qin et al. (2022) Bowen Qin, Binyuan Hui, Lihan Wang, Min Yang, Jinyang Li, Binhua Li, Ruiying Geng, Rongyu Cao, Jian Sun, Luo Si, et al.  A survey on text-to-sql parsing: Concepts, methods, and future directions.  _arXiv preprint arXiv:2208.13629_ , 2022.
  * Quaranta et al. (2021) Luigi Quaranta, Fabio Calefato, and Filippo Lanubile.  Kgtorrent: A dataset of python jupyter notebooks from kaggle.  In _2021 IEEE/ACM 18th International Conference on Mining Software Repositories (MSR)_ , pp. 550–554. IEEE, 2021.
  * RelationalAI (2021) RelationalAI.  The rel language, Jan 2021.  URL https://docs.relational.ai/rel.
  * Rice (1953) Henry Gordon Rice.  Classes of recursively enumerable sets and their decision problems.  _Transactions of the American Mathematical society_ , 74(2):358–366, 1953.
  * Risdal & Bozsolik (2022) Megan Risdal and Timo Bozsolik.  Meta kaggle, 2022.  URL https://www.kaggle.com/ds/9.
  * Sakaguchi et al. (2021) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  Winogrande: An adversarial winograd schema challenge at scale.  _Communications of the ACM_ , 64(9):99–106, 2021\.
  * Schaeffer et al. (2024) Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo.  Are emergent abilities of large language models a mirage?  _Advances in Neural Information Processing Systems_ , 36, 2024.
  * Shi et al. (2021) Xingjian Shi, Jonas Mueller, Nick Erickson, Mu Li, and Alex Smola.  Multimodal autoML on structured tables with text fields.  In _8th ICML Workshop on Automated Machine Learning (AutoML)_ , 2021\.  URL https://openreview.net/forum?id=OHAIVOOl7Vl.
  * Shwartz-Ziv & Armon (2022) Ravid Shwartz-Ziv and Amitai Armon.  Tabular data: Deep learning is not all you need.  _Information Fusion_ , 81:84–90, 2022.
  * Srivastava et al. (2022) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _arXiv preprint arXiv:2206.04615_ , 2022.
  * Strichman (2018) Ofer Strichman.  program equivalence.  _Formal Methods in System Design_ , 52:227–228, 2018.
  * Sun et al. (2020) Ningyuan Sun, Xuefeng Yang, and Yunfeng Liu.  Tableqa: a large-scale chinese text-to-sql dataset for table-aware sql generation.  _arXiv preprint arXiv:2006.06434_ , 2020.
  * Taieb & Hyndman (2014) Souhaib Ben Taieb and Rob J Hyndman.  A gradient boosting approach to the kaggle load forecasting competition.  _International journal of forecasting_ , 30(2):382–394, 2014.
  * Tay et al. (2020) Yi Tay, Mostafa Dehghani, Samira Abnar, Yikang Shen, Dara Bahri, Philip Pham, Jinfeng Rao, Liu Yang, Sebastian Ruder, and Donald Metzler.  Long range arena: A benchmark for efficient transformers.  _arXiv preprint arXiv:2011.04006_ , 2020.
  * Tolkachev et al. (2020) Alexey Tolkachev, Ilyas Sirazitdinov, Maksym Kholiavchenko, Tamerlan Mustafaev, and Bulat Ibragimov.  Deep learning for diagnosis and segmentation of pneumothorax: The results on the kaggle competition and validation against radiologists.  _IEEE Journal of Biomedical and Health Informatics_ , 25(5):1660–1672, 2020.
  * Uschold (2015) Michael Uschold.  Ontology and database schema: What’s the difference?  _Applied Ontology_ , 10(3-4):243–258, 2015.
  * Wang et al. (2021) April Yi Wang, Dakuo Wang, Jaimie Drozdal, Xuye Liu, Soya Park, Steve Oney, and Christopher Brooks.  What makes a well-documented notebook? a case study of data scientists’ documentation practices in kaggle.  In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_ , pp. 1–7, 2021.
  * Wang et al. (2020) Lijie Wang, Ao Zhang, Kun Wu, Ke Sun, Zhenghua Li, Hua Wu, Min Zhang, and Haifeng Wang.  DuSQL: A large-scale and pragmatic Chinese text-to-SQL dataset.  In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pp. 6923–6935, Online, November 2020. Association for Computational Linguistics.  doi: 10.18653/v1/2020.emnlp-main.562.  URL https://aclanthology.org/2020.emnlp-main.562.
  * Wei et al. (2022) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al.  Emergent abilities of large language models.  _Transactions on Machine Learning Research_ , 2022.
  * Wilcoxon (1945) Frank Wilcoxon.  Individual comparisons by ranking methods.  _Biometrics Bulletin_ , 1(6):80–83, 1945.  ISSN 00994987.  URL http://www.jstor.org/stable/3001968.
  * Woltmann et al. (2019) Lucas Woltmann, Claudio Hartmann, Maik Thiele, Dirk Habich, and Wolfgang Lehner.  Cardinality estimation with local deep learning models.  In _Proceedings of the second international workshop on exploiting artificial intelligence techniques for data management_ , pp. 1–8, 2019.
  * Xu et al. (2017) Xiaojun Xu, Chang Liu, and Dawn Song.  Sqlnet: Generating structured queries from natural language without reinforcement learning.  _arXiv preprint arXiv:1711.04436_ , 2017.
  * Yaghmazadeh et al. (2017) Navid Yaghmazadeh, Yuepeng Wang, Isil Dillig, and Thomas Dillig.  Sqlizer: query synthesis from natural language.  _Proceedings of the ACM on Programming Languages_ , 1(OOPSLA):1–26, 2017.
  * Yan et al. (2020) Zeyu Yan, Jianqiang Ma, Yang Zhang, and Jianping Shen.  Sql generation via machine reading comprehension.  In _Proceedings of the 28th International Conference on Computational Linguistics_ , pp. 350–356, 2020.
  * Yang & Ding (2020) Xulei Yang and Jie Ding.  A computational framework for iceberg and ship discrimination: Case study on kaggle competition.  _IEEE Access_ , 8:82320–82327, 2020.
  * Yang et al. (2018) Xulei Yang, Zeng Zeng, Sin G Teo, Li Wang, Vijay Chandrasekhar, and Steven Hoi.  Deep learning for practical image recognition: Case study on kaggle competitions.  In _Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining_, pp. 923–931, 2018.
  * Yang et al. (2019) Zongheng Yang, Eric Liang, Amog Kamsetty, Chenggang Wu, Yan Duan, Xi Chen, Pieter Abbeel, Joseph M Hellerstein, Sanjay Krishnan, and Ion Stoica.  Deep unsupervised cardinality estimation.  _arXiv preprint arXiv:1905.04278_ , 2019.
  * Yang et al. (2020) Zongheng Yang, Amog Kamsetty, Sifei Luan, Eric Liang, Yan Duan, Xi Chen, and Ion Stoica.  Neurocard: one cardinality estimator for all tables.  _arXiv preprint arXiv:2006.08109_ , 2020.
  * Yavuz et al. (2018) Semih Yavuz, Izzeddin Gür, Yu Su, and Xifeng Yan.  What it takes to achieve 100% condition accuracy on wikisql.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pp. 1702–1711, 2018.
  * Yu et al. (2018a) Tao Yu, Zifan Li, Zilin Zhang, Rui Zhang, and Dragomir Radev.  Typesql: Knowledge-based type-aware neural text-to-sql generation.  _arXiv preprint arXiv:1804.09769_ , 2018a.
  * Yu et al. (2018b) Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, et al.  Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task.  _arXiv preprint arXiv:1809.08887_ , 2018b.
  * Yu et al. (2019) Tao Yu, Rui Zhang, Michihiro Yasunaga, Yi Chern Tan, Xi Victoria Lin, Suyi Li, Heyang Er, Irene Li, Bo Pang, Tao Chen, Emily Ji, Shreya Dixit, David Proctor, Sungrok Shim, Jonathan Kraft, Vincent Zhang, Caiming Xiong, Richard Socher, and Dragomir Radev.  SParC: Cross-domain semantic parsing in context.  In Anna Korhonen, David Traum, and Lluís Màrquez (eds.), _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 4511–4523, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1443.  URL https://aclanthology.org/P19-1443.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  HellaSwag: Can a machine really finish your sentence?  In Anna Korhonen, David Traum, and Lluís Màrquez (eds.), _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pp. 4791–4800, Florence, Italy, July 2019. Association for Computational Linguistics.  doi: 10.18653/v1/P19-1472.  URL https://aclanthology.org/P19-1472.
  * Zhong et al. (2017) Victor Zhong, Caiming Xiong, and Richard Socher.  Seq2sql: Generating structured queries from natural language using reinforcement learning.  _arXiv preprint arXiv:1709.00103_ , 2017.

##  Appendix A Example Prompts for Large Language Models Annotation

An example of the prompt for the zero-shot synthesis of SQL code is shown
below.

⬇

1We have a SQL database with 2 tables. For each table, here is the schema and

2a sample row data.

3Schema for table 0:

4CREATE TABLE MaLga12345(

5 Saledate TEXT,

6 MA BIGINT,

7 Type TEXT,

8 Bedrooms BIGINT

9);

10A sample row data from table 0:

11{"Saledate": "30/09/2007", "MA": 441854, "Type": "house", "Bedrooms": 2}

12Schema for table 1:

13CREATE TABLE RawSales(

14 Datesold TEXT,

15 Postcode BIGINT,

16 Price BIGINT,

17 PropertyType TEXT,

18 Bedrooms BIGINT

19);

20A sample row data from table 1:

21{"Datesold": "2007-02-07 00:00:00", "Postcode": 2607, "Price": 525000,

22"PropertyType": "house", "Bedrooms": 4}

23Consider the typical users who access this database. What kind of SQLite

24SELECT queries would they write? List 15 examples SQL code that are

25directly executable with relatively high complexity, each following a

26description. If possible, use joins. Reply in the format of a description

27and the SQL code for each example.

An example of the prompt for SQL to Rel translation is shown below.

⬇

1REL is a database management system language that is similar to datalog. REL
is based on the sixth normal form (6NF),

2where every variable represents a relation, i.e. a set of tuples.

3In REL, from-variables will not appear in the output, and for-variables will
appear in the output.

4Strings in REL use double quotes.

5Below are few examples of SQL code and REL code pairs that perform the same
query.

6Examples:

7

81.SQL:

9‘‘‘

10SELECT L.JobTitle, L.ExperienceLevel, V3.JobTitle, V3.ExperienceLevel FROM
LatestDataScienceSalaries AS L INNER JOIN V3LatestDataScienceSalaries AS V3 ON
L.CompanyLocation = V3.CompanyLocation WHERE L.CompanyLocation = "United
States";

11‘‘‘

121.REL:

13‘‘‘

14def temp[x,y] = LatestDataScienceSalaries:CompanyLocation[x,
company_location] and V3LatestDataScienceSalaries:CompanyLocation[y,
company_location] and company_location = "United States" from company_location

15def output[x_job_title, x_experience_level, y_job_title, y_experience_level]
= LatestDataScienceSalaries:JobTitle[x, x_job_title],
LatestDataScienceSalaries:ExperienceLevel[x, x_experience_level],
V3LatestDataScienceSalaries:JobTitle[y, y_job_title],
V3LatestDataScienceSalaries:ExperienceLevel[y, y_experience_level] and
temp[x,y] for x, y

16‘‘‘

172.SQL:

18‘‘‘

19SELECT p.ImageLink FROM Photos p JOIN Likes l ON p.Id = l.Photo JOIN Users u
ON l.User = u.Id LEFT JOIN Follows f ON u.Id = f.Follower AND f.Followee =
p.UserID WHERE f.Follower IS NULL;

20‘‘‘

212.REL:

22‘‘‘

23def UserPhoto(user_id, photo_id) = Photos:UserID[l,user_id] and Photos:Id[l,
photo_id] from l

24def UserLikesPhoto(user_id, photo_id) = Likes:User[l,user_id] and
Likes:Photo[l, photo_id] from l

25def UserID(user_id) = Users:Id(l, user_id) from l

26def UserFollowsUser(follower_id, followee_id) =
Follows:Follower[l,follower_id] and Follows:Followee[l, followee_id] from l

27def PhotoIDHasImageLink(photo_id, image_link) =

28 Photos:Id(p, photo_id) and Photos:ImageLink(p, image_link) from p

29def photos_liked_by_not_a_follower(photo) =

30 UserPhoto(poster, photo)

31 and UserLikesPhoto(liker, photo)

32 and not UserFollowsUser(liker, poster)

33 from poster, liker

34def output = PhotoIDHasImageLink[photos_liked_by_not_a_follower]

35‘‘‘

363.SQL:

37‘‘‘

38SELECT MainTable.StateB, MainTable.Name FROM MainTable JOIN CaseUniverse ON
MainTable.StateB = CaseUniverse.TargetState AND MainTable.StateBCode =
CaseUniverse.TargetCOWID WHERE TargetState = "FRN" AND TargetCOWID = "220";

39‘‘‘

403.REL:

41‘‘‘

42def join[mt, cu] = MainTable:StateB[mt, sb] and CaseUniverse:TargetState[cu,
sb] and MainTable:StateBCode[mt, sbc] and CaseUniverse:TargetCOWID[cu, sbc]
and sb="FRN" and sbc="220" from sb, sbc

43def output = MainTable:StateB[mt], MainTable:Name[mt] for mt, cu where
join[mt, cu]

44‘‘‘

454.SQL:

46‘‘‘

47SELECT LD.Regions, SUM(DT.Punjab + DT.Haryana + DT.Rajasthan + DT.Delhi +
DT.UP + DT.Uttarakhand) AS Total_Usage FROM DatasetTk DT INNER JOIN LongData
LD ON DT.Punjab = LD.Usage AND LD.States = ’Punjab’ GROUP BY LD.Regions;

48‘‘‘

494.REL:

50‘‘‘

51def join1[dt,ld]= DatasetTk:Punjab[dt, punjab] and LongData:Usage[ld,
punjab] from punjab

52def sum_usage = DatasetTk:Punjab[dt] + DatasetTk:Haryana[dt] +

53DatasetTk:Rajasthan[dt] + DatasetTk:Delhi[dt] +

54DatasetTk:UP[dt] + DatasetTk:Uttarakhand[dt] for dt

55def output[region] = sum[dt, usage:

56LongData:Regions[ld, region]

57and sum_usage[dt, usage]

58and join1[dt,ld] and LongData:States[ld] = "Punjab" for ld]

59

60‘‘‘

615.SQL:

62‘‘‘

63SELECT Test.ID, Test.Age, Train.SpendingScore FROM Test INNER JOIN Train ON
Test.ID = Train.ID WHERE Test.Age >= 40 OR Train.Age >= 40;

64‘‘‘

655.REL:

66‘‘‘

67def output[id, age, spending_score] = Test:ID[test_idx, id],
Test:Age[test_idx, age], Train:SpendingScore[train_idx, spending_score] and
Test:ID[test_idx, id] and Train:ID[train_idx, id] and (Test:Age[test_idx]>=40
or Train:Age[train_idx] >= 40) from test_idx, train_idx

68‘‘‘

696.SQL:

70‘‘‘

71SELECT bb.PlayerOut, m.WinningTeam FROM IPLBallByBall2022 bb JOIN
IPLMatches2022 m ON bb.ID = m.ID WHERE bb.ID = 1312200 AND bb.IsWicketDelivery
= 1;

72‘‘‘

736.REL:

74‘‘‘

75def join[bb, m] = IPLBallByBall2022:ID[bb, id] and IPLMatches2022:ID[m, id]
and IPLBallByBall2022:ID[bb]=1312200 and
IPLBallByBall2022:IsWicketDelivery[bb]=1 from id

76def output = IPLBallByBall2022:PlayerOut[bb], IPLMatches2022:WinningTeam[m]
from bb, m where join[bb,m]

77‘‘‘

787.SQL:

79‘‘‘

80SELECT il.AccountName, il.Ismarketplace FROM InstAccLabeled AS il JOIN
InstAcc AS ia ON il.AccountName = ia.Username WHERE ia.IsBusinessAccount = 1;

81‘‘‘

827.REL:

83‘‘‘

84def join[il, ia] = InstAccLabeled:AccountName[il, account_name] and
InstAcc:Username[ia, account_name] and
InstAcc:IsBusinessAccount[ia]=boolean_true from account_name

85def output = InstAccLabeled:AccountName[il],
InstAccLabeled:Ismarketplace[il] for il, ia where join[il, ia]

86‘‘‘

878.SQL:

88‘‘‘

89SELECT table0.Area, table0.AveragePrice, table1.MedianSalary FROM
HousingInLondonMonthlyVariables AS table0 JOIN HousingInLondonYearlyVariables
AS table1 ON table0.Code = table1.Code WHERE table0.BoroughFlag = 1;

90‘‘‘

918.REL:

92‘‘‘

93def join[row_id_0, row_id_1] =
HousingInLondonMonthlyVariables:Code[row_id_0, code] and
HousingInLondonYearlyVariables:Code[row_id_1, code] and
HousingInLondonMonthlyVariables:BoroughFlag[row_id_0]=1 from code

94def output[area, average_price, median_salary] =
HousingInLondonMonthlyVariables:Area[row_id_0, area] and
HousingInLondonMonthlyVariables:AveragePrice[row_id_0, average_price] and
HousingInLondonYearlyVariables:MedianSalary[row_id_1, median_salary] and
join[row_id_0, row_id_1] for row_id_0, row_id_1

95‘‘‘

969.SQL:

97‘‘‘

98SELECT Gender, Segmentation, COUNT(*) FROM Test WHERE Segmentation IN (’B’,
’C’) GROUP BY Gender, Segmentation;

99‘‘‘

1009.REL:

101‘‘‘

102def output[gender, seg] = count[row_id: Test:Gender(row_id, gender) and
Test:Segmentation(row_id,seg) and {"B"; "C"}(seg)]

103‘‘‘

10410.SQL:

105‘‘‘

106SELECT DISTINCT State FROM NCHSLeadingCausesOfDeathUnitedStates;

107‘‘‘

10810.REL:

109‘‘‘

110def output[state] = NCHSLeadingCausesOfDeathUnitedStates:State[x,state]
from x

111‘‘‘

11211.SQL:

113‘‘‘

114SELECT * FROM HRCommaSep;

115‘‘‘

11611.REL:

117‘‘‘

118def output[col, row, val] = HRCommaSep[col, row, val]

119‘‘‘

12012.SQL:

121‘‘‘

122SELECT Title, Year FROM TvShows;

123‘‘‘

12412.REL:

125‘‘‘

126def output[title, year] = TvShows:Title[idx, title], TvShows:Year[idx,
year] for idx

127‘‘‘

12813.SQL:

129‘‘‘

130SELECT AverageMontlyHours FROM HRCommaSep WHERE Department = ’sales’;

131‘‘‘

13213.REL:

133‘‘‘

134def output[average_monthly_hours] = HRCommaSep:Department(row_idx, "sales")
and HRCommaSep:AverageMontlyHours(row_idx, average_monthly_hours) for row_idx

135‘‘‘

13614.SQL:

137‘‘‘

138SELECT Location, Age FROM User WHERE UserId = 1;

139‘‘‘

14014.REL:

141‘‘‘

142def output[location, age] = User:Location[idx, location], User:Age[idx,
age], User:UserId[idx, 1] for idx

143‘‘‘

14415.SQL:

145‘‘‘

146SELECT * FROM GMPlayersStatistics WHERE isstreamer = 1 AND bulletwin > 70;

147‘‘‘

14815.REL:

149‘‘‘

150def output[col, row, val] = GMPlayersStatistics[col, row, val] and
GMPlayersStatistics:IsStreamer[row] = boolean_true and
GMPlayersStatistics:BulletWin[row] > 70

151‘‘‘

15216.SQL:

153‘‘‘

154SELECT AVG(Visibilitykm), MAX(WindSpeedkmh) FROM WeatherData WHERE Weather
= "Rain";

155‘‘‘

15616.REL:

157‘‘‘

158def all_row_ids_with_rain[row_id] = WeatherData:Weather[row_id]="Rain"

159def output = average[row_id, value_vis: all_row_ids_with_rain(row_id) and
WeatherData:VisibilityKm(row_id, value_vis)], max[row_id, value_wind_speed :
all_row_ids_with_rain(row_id) and WeatherData:WindSpeedKmh(row_id,
value_wind_speed)]

160‘‘‘

16117.SQL:

162‘‘‘

163SELECT * FROM Data WHERE Danceability > 0.8;

164‘‘‘

16517.REL:

166‘‘‘

167def output[col, idx, val] = Data[col, idx, val] and Data:Danceability[idx]
> 0.8

168‘‘‘

16918.SQL:

170‘‘‘

171SELECT * FROM SpaceCorrected WHERE StatusMission IN (’Success’, ’Failure’);

172‘‘‘

17318.REL:

174‘‘‘

175def output[col, row_id, val] = SpaceCorrected[col, row_id, val] and
{"Success"; "Failure"}(SpaceCorrected:StatusMission[row_id])

176‘‘‘

17719.SQL:

178‘‘‘

179SELECT * FROM SpaceCorrected WHERE Location LIKE ’%

180‘‘‘

18119.REL:

182‘‘‘

183def output[col, row_id, val] = SpaceCorrected[col, row_id, val] and
like_match("\%Florida\%", SpaceCorrected:Location[row_id])

184‘‘‘

18520.SQL:

186‘‘‘

187SELECT * FROM Cars24Combined WHERE Fuel = ’PETROL’;

188‘‘‘

18920.REL:

190‘‘‘

191def output[colname, row_idx, val] = Cars24Combined[colname, row_idx, val]
and Cars24Combined[:Fuel, row_idx, "PETROL"]

192‘‘‘

19321.SQL:

194‘‘‘

195SELECT DISTINCT Name FROM UCIDatasets;

196‘‘‘

19721.REL:

198‘‘‘

199def output[name] = UCIDatasets:Name[idx, name] from idx

200‘‘‘

20122.SQL:

202‘‘‘

203SELECT COUNT(*) FROM SupermarketSalesSheet1;

204‘‘‘

20522.REL:

206‘‘‘

207def count_rows = count[row_id: SupermarketSalesSheet1[_, row_id, _]]

208 def output = count_rows

209‘‘‘

21023.SQL:

211‘‘‘

212SELECT * FROM Jobs WHERE ActivelyHiring = 1.0;

213‘‘‘

21423.REL:

215‘‘‘

216def output[col, idx, val] = Jobs[col, idx, val] and
Jobs:ActivelyHiring[idx] = 1.0

217‘‘‘

21824.SQL:

219‘‘‘

220SELECT marvelcomiccharactername, comicappearance1 FROM MarvelComicsLegacy
WHERE marvelcomiccharactername = ’Wolverine’;

221‘‘‘

22224.REL:

223‘‘‘

224def output[marvel_character_name, comic_appearance_1] =
MarvelComicsLegacy:MarvelComicCharacterName[idx, marvel_character_name],
MarvelComicsLegacy:ComicAppearance1[idx, comic_appearance_1] and
marvel_character_name = "Wolverine" for idx

225‘‘‘

22625.SQL:

227‘‘‘

228SELECT * FROM WAFnUseCTelcoCustomerChurn WHERE Churn = "Yes";

229‘‘‘

23025.REL:

231‘‘‘

232def output[col, row_idx, val] = WAFnUseCTelcoCustomerChurn[col, row_idx,
val] and WAFnUseCTelcoCustomerChurn:Churn[row_idx] = "Yes"

233‘‘‘

23426.SQL:

235‘‘‘

236SELECT AVG(Charges) FROM Insurance WHERE Age > 40;

237‘‘‘

23826.REL:

239‘‘‘

240def output = average[idx, charges: Insurance:Charges[idx, charges] and
Insurance:Age[idx] > 40]

241‘‘‘

24227.SQL:

243‘‘‘

244SELECT * FROM CommentsCleaned WHERE EmojiUsed = "yes";

245‘‘‘

24627.REL:

247‘‘‘

248def output[col, row_id, val] = CommentsCleaned[col, row_id, val] and
CommentsCleaned:EmojiUsed[row_id] = "yes"

249‘‘‘

25028.SQL:

251‘‘‘

252SELECT * FROM Cars24Combined WHERE Fuel = "PETROL" AND Location = "HR-98";

253‘‘‘

25428.REL:

255‘‘‘

256def output[col, idx, val] = Cars24Combined[col, idx, val] and
Cars24Combined:Fuel[idx, "PETROL"] and Cars24Combined:Location[idx, "HR-98"]

257‘‘‘

25829.SQL:

259‘‘‘

260SELECT StateUT, AVG(NumberOfHouseholdsSurveyed) as AverageHouseholds FROM
Datafile GROUP BY StateUT;

261‘‘‘

26229.REL:

263‘‘‘

264def output[state] = average[idx, num:
Datafile:NumberOfHouseholdsSurveyed[idx, num] and Datafile:StateUT[idx,
state]]

265‘‘‘

26630.SQL:

267‘‘‘

268SELECT * FROM Cars24Combined WHERE Fuel = ’PETROL’ AND Location = ’HR-98’;

269‘‘‘

27030.REL:

271‘‘‘

272def output[col, idx, val] = Cars24Combined[col, idx, val] and
Cars24Combined:Fuel[idx, "PETROL"] and Cars24Combined:Location[idx, "HR-98"]

273‘‘‘

27431.SQL:

275‘‘‘

276SELECT Gender, COUNT(*) FROM Diabetes GROUP BY Gender;

277‘‘‘

27831.REL:

279‘‘‘

280def output[gender] = count[row_id: Diabetes:Gender[row_id, gender]]

281‘‘‘

28232.SQL:

283‘‘‘

284SELECT JobTitle, AVG(Salary) AS AverageSalary

285FROM PartiallyCleanedSalaryDataset

286GROUP BY JobTitle;

287‘‘‘

28832.REL:

289‘‘‘

290def output[job_title] = average[idx, salary:
PartiallyCleanedSalaryDataset:Salary[idx, salary] and
PartiallyCleanedSalaryDataset:JobTitle[idx, job_title]]

291‘‘‘

29233.SQL:

293‘‘‘

294SELECT Race, COUNT(*) AS TotalPitstops

295FROM Pitstops

296GROUP BY Race;

297‘‘‘

29833.REL:

299‘‘‘

300def output[race] = count[row_id: Pitstops:Race[row_id, race]]

301‘‘‘

30234.SQL:

303‘‘‘

304SELECT Title, Rating FROM TopAnime WHERE Rating = (SELECT MAX(Rating) FROM
TopAnime);

305‘‘‘

30634.REL:

307‘‘‘

308def max_rating = max[x, rating : TopAnime:Rating[x, rating]]

309def output[title, rating] = TopAnime:Title[idx, title],
TopAnime:Rating[idx, rating] and TopAnime:Rating[idx, max_rating] for idx

310‘‘‘

31135.SQL:

312‘‘‘

313SELECT Category, SUM(Sales) AS TotalSales, SUM(Profit) AS TotalProfit

314FROM SampleSuperstore

315GROUP BY Category;

316‘‘‘

31735.REL:

318‘‘‘

319def output[category] = sum[row_id, sales: SampleSuperstore:Category(row_id,
category) and SampleSuperstore:Sales(row_id, sales)], sum[row_id, profit:
SampleSuperstore:Category(row_id, category) and
SampleSuperstore:Profit(row_id, profit)]

320‘‘‘

32136.SQL:

322‘‘‘

323SELECT SUM(CustomerCareCalls)

324FROM Train

325WHERE DiscountOffered >= 50;

326‘‘‘

32736.REL:

328‘‘‘

329def output = sum[row_id,calls: Train:CustomerCareCalls[row_id, calls] and
Train:DiscountOffered[row_id] >= 50]

330‘‘‘

33137.SQL:

332‘‘‘

333SELECT AVG(MonthlyCharges) FROM CustomerChurn WHERE Churn = ’Yes’;

334‘‘‘

33537.REL:

336‘‘‘

337def output = average[row_id, monthly_charges:
CustomerChurn:MonthlyCharges(row_id, monthly_charges) and
CustomerChurn:Churn[row_id] = "Yes"]

338‘‘‘

33938.SQL:

340‘‘‘

341SELECT COUNT(DISTINCT StatusType) AS UniqueStatusTypes

342FROM Live;

343‘‘‘

34438.REL:

345‘‘‘

346def output = count[status_type: Live:StatusType(row_id, status_type) from
row_id]

347‘‘‘

34839.SQL:

349‘‘‘

350SELECT * FROM PCOSInfertility ORDER BY SlNo DESC;

351‘‘‘

35239.REL:

353‘‘‘

354def output[col, row_idx, val] = PCOSInfertility[col, row_idx, val]

355‘‘‘

35640.SQL:

357‘‘‘

358SELECT SUM(WeeklySales) AS TotalWeeklySales FROM WalmartStoreSales;

359‘‘‘

36040.REL:

361‘‘‘

362def output = sum[row_idx, weekly_sales :
WalmartStoreSales:WeeklySales[row_idx, weekly_sales]]

363‘‘‘

36441.SQL:

365‘‘‘

366SELECT * FROM WeatherData ORDER BY RelativeHumidity ASC;

367‘‘‘

36841.REL:

369‘‘‘

370def output[col, row_id, val] = WeatherData[col, row_id, val]

371‘‘‘

37242.SQL:

373‘‘‘

374SELECT ShipMode, SUM(Sales) AS TotalSales FROM SampleSuperstore GROUP BY
ShipMode;

375‘‘‘

37642.REL:

377‘‘‘

378def output[shipmode] = sum[row_id, value_sales:
SampleSuperstore:Sales[row_id, value_sales] and
SampleSuperstore:ShipMode[row_id, shipmode]]

379‘‘‘

38043.SQL:

381‘‘‘

382SELECT PlayerOfMatch, WinningTeam

383FROM IPLMatches2022

384WHERE WinningTeam = "Rajasthan Royals";

385‘‘‘

38643.REL:

387‘‘‘

388def output[player_of_match, winning_team] =
IPLMatches2022:PlayerOfMatch[idx, player_of_match],
IPLMatches2022:WinningTeam[idx, winning_team] and
IPLMatches2022:WinningTeam[idx, winning_team] and winning_team = "Rajasthan
Royals" for idx

389‘‘‘

39044.SQL:

391‘‘‘

392SELECT Title, Desc FROM MegaGymDataset WHERE Type = "Strength";

393‘‘‘

39444.REL:

395‘‘‘

396def output[title, desc] = MegaGymDataset:Title[idx, title],
MegaGymDataset:Desc[idx, desc] and MegaGymDataset:Type[idx, "Strength"] for
idx

397‘‘‘

39845.SQL:

399‘‘‘

400SELECT TypeOfInternship, COUNT(*) as Count FROM Internship GROUP BY
TypeOfInternship;

401‘‘‘

40245.REL:

403‘‘‘

404def output[type_of_internship] = count[row_id:
Internship:TypeOfInternship[row_id, type_of_internship]]

405‘‘‘

406Given examples above, translate the following SQL queries to REL programs.
No explanation is needed.

407Output only the REL code one by one numbered with ’1.REL:’, ’2.REL:’, for
example.

408

4091.SQL:

410‘‘‘

411SELECT * FROM BookingSaudiArabia ORDER BY Score DESC LIMIT 5;

412‘‘‘

4132.SQL:

414‘‘‘

415SELECT *

416FROM VsrrProvisionalDrugOverdoseDeathCounts

417WHERE DataValue > 5000;

418‘‘‘

4193.SQL:

420‘‘‘

421SELECT COUNT(*) FROM Tweets;

422‘‘‘

4234.SQL:

424‘‘‘

425SELECT AVG(PredSvr) FROM PredSvr INNER JOIN PredXgb ON PredSvr.PredSvr =
PredXgb.XgCost;

426‘‘‘

4275.SQL:

428‘‘‘

429SELECT * FROM ChurnTrainTransformed WHERE CreditPerProduct < 400 OR
EstSalaryScaled IS NULL

430UNION

431SELECT * FROM TestDfTransformed WHERE CreditPerProduct < 400 OR
EstSalaryScaled IS NULL;

432‘‘‘

4336.SQL:

434‘‘‘

435SELECT * FROM Leads WHERE LastNotableActivity = ’Modified’;

436‘‘‘

4377.SQL:

438‘‘‘

439SELECT CentralPressure, MaximumSustainedWindSpeed FROM TyphoonData WHERE
IndicatorOfLandfallOrPassage = ’Landfall’;

440‘‘‘

4418.SQL:

442‘‘‘

443SELECT SUM(A28) FROM Texture WHERE A17 < (SELECT AVG(A17) FROM Texture);

444‘‘‘

4459.SQL:

446‘‘‘

447SELECT FlatModel, MAX(ResalePrice) AS MaxResalePrice

448 FROM ResaleFlatPricesBasedOnApprovalDate2000Feb2012

449 GROUP BY FlatModel;

450‘‘‘

45110.SQL:

452‘‘‘

453SELECT COUNT(*) FROM TestFile;

454‘‘‘

An example of the prompt for input-target column annotation is shown below.

⬇

1Consider a machine learning model that takes a few numeric input columns

2and predict a single classification target column. Given the following

3schema of a data table, suggest the input columns and target column, such

4that the target may be predicted from the inputs non-trivially.

5Schema: CREATE TABLE PartiallyCleanedSalaryDataset(

6 CompanyName TEXT,

7 JobTitle TEXT,

8 SalariesReported BIGINT,

9 Location TEXT,

10 Salary FLOAT

11);

12Respond in JSON format with ‘input_columns’ and ‘output_column’.

##  Appendix B Steering LLMs to Generate More Complex SQL Programs

During prototyping, we refined the instructions to create the best annotation
results. One of our goals was to increase the complexity of SQL programs
generated by the LLM. We experimented with different prompts, and we found
that instructing the LLM to generate SQL programs “with relatively high
complexity” can increase the complexity and reduce the number of simple
queries generated by the LLM. An example is given here to illustrate the
effect of including the steering phrase and without the steering phrase.

The following is a prompt for the zero-shot synthesis of SQL code without the
steering phrase “with relatively high complexity”.

⬇

1We have a SQL database with 2 tables. For each table, here is the schema and

2a sample row data.

3Schema for table 0:

4CREATE TABLE MaLga12345(

5 Saledate TEXT,

6 MA BIGINT,

7 Type TEXT,

8 Bedrooms BIGINT

9);

10A sample row data from table 0:

11{"Saledate": "30/09/2007", "MA": 441854, "Type": "house", "Bedrooms": 2}

12Schema for table 1:

13CREATE TABLE RawSales(

14 Datesold TEXT,

15 Postcode BIGINT,

16 Price BIGINT,

17 PropertyType TEXT,

18 Bedrooms BIGINT

19);

20A sample row data from table 1:

21{"Datesold": "2007-02-07 00:00:00", "Postcode": 2607, "Price": 525000,

22"PropertyType": "house", "Bedrooms": 4}

23Consider the typical users who access this database. What kind of SQLite

24SELECT queries would they write? List 15 examples SQL code that are

25directly executable, each following a description. If possible, use joins.

26Reply in the format of a description and the SQL code for each example.

Here is the annotation generated by the LLM when the steering phrase is not
used.

⬇

11. Get all rows from MaLga12345 table:

2‘‘‘sql

3SELECT * FROM MaLga12345;

4‘‘‘

5

62. Get all rows from RawSales table:

7‘‘‘sql

8SELECT * FROM RawSales;

9‘‘‘

10

113. Find the average price of properties sold by postcode:

12‘‘‘sql

13SELECT Postcode, AVG(Price) as AveragePrice

14FROM RawSales

15GROUP BY Postcode;

16‘‘‘

17

184. List all house sales from MaLga12345:

19‘‘‘sql

20SELECT * FROM MaLga12345

21WHERE Type = ’house’;

22‘‘‘

23

245. List all properties sold on a specific date from RawSales:

25‘‘‘sql

26SELECT * FROM RawSales

27WHERE Datesold = ’2007-02-07 00:00:00’;

28‘‘‘

29

306. Find the total number of bedrooms sold in RawSales:

31‘‘‘sql

32SELECT SUM(Bedrooms) as TotalBedrooms

33FROM RawSales;

34‘‘‘

35

367. Get the maximum MA value from MaLga12345:

37‘‘‘sql

38SELECT MAX(MA) as MaxMA

39FROM MaLga12345;

40‘‘‘

41

428. List properties sold in a specific postcode with more than 3 bedrooms from RawSales:

43‘‘‘sql

44SELECT * FROM RawSales

45WHERE Postcode = 2607 AND Bedrooms > 3;

46‘‘‘

47

489. Count the number of sales by property type from RawSales:

49‘‘‘sql

50SELECT PropertyType, COUNT(*) as SalesCount

51FROM RawSales

52GROUP BY PropertyType;

53‘‘‘

54

5510. Get all properties sold in a specific year from RawSales:

56‘‘‘sql

57SELECT * FROM RawSales

58WHERE Datesold LIKE ’2007%

59‘‘‘

60

6111. Find the sum of prices for all house sales from RawSales:

62‘‘‘sql

63SELECT SUM(Price) as TotalHouseSales

64FROM RawSales

65WHERE PropertyType = ’house’;

66‘‘‘

67

6812. Join MaLga12345 and RawSales tables on bedrooms and list matching records:

69‘‘‘sql

70SELECT a.*, b.*

71FROM MaLga12345 a

72JOIN RawSales b ON a.Bedrooms = b.Bedrooms;

73‘‘‘

74

7513. Find properties with the same sale date and date sold from both tables:

76‘‘‘sql

77SELECT a.*, b.*

78FROM MaLga12345 a

79JOIN RawSales b ON a.Saledate = b.Datesold;

80‘‘‘

81

8214. Get the count of properties sold for each MA value from MaLga12345:

83‘‘‘sql

84SELECT MA, COUNT(*) as CountOfSales

85FROM MaLga12345

86GROUP BY MA;

87‘‘‘

88

8915. List all properties from MaLga12345 and their corresponding prices from RawSales if they have the same number of bedrooms:

90‘‘‘sql

91SELECT a.*, b.Price

92FROM MaLga12345 a

93JOIN RawSales b ON a.Bedrooms = b.Bedrooms;

94‘‘‘

The following is a prompt for the zero-shot synthesis of SQL code that has the
steering phrase “with relatively high complexity”.

⬇

1We have a SQL database with 2 tables. For each table, here is the schema and

2a sample row data.

3Schema for table 0:

4CREATE TABLE MaLga12345(

5 Saledate TEXT,

6 MA BIGINT,

7 Type TEXT,

8 Bedrooms BIGINT

9);

10A sample row data from table 0:

11{"Saledate": "30/09/2007", "MA": 441854, "Type": "house", "Bedrooms": 2}

12Schema for table 1:

13CREATE TABLE RawSales(

14 Datesold TEXT,

15 Postcode BIGINT,

16 Price BIGINT,

17 PropertyType TEXT,

18 Bedrooms BIGINT

19);

20A sample row data from table 1:

21{"Datesold": "2007-02-07 00:00:00", "Postcode": 2607, "Price": 525000,

22"PropertyType": "house", "Bedrooms": 4}

23Consider the typical users who access this database. What kind of SQLite

24SELECT queries would they write? List 15 examples SQL code that are

25directly executable with relatively high complexity@, each following a

26description. If possible, use joins. Reply in the format of a description

27and the SQL code for each example.

The following is the annotation generated by the LLM when the steering phrase
is used. Clearly, the SQL programs generated have a higher complexity, and
they are more valuable as SQL annotations for our data tables.

⬇

11. Select all sales records from both tables where the property type is a "house":

2‘‘‘sql

3SELECT * FROM MaLga12345 WHERE Type = ’house’

4UNION

5SELECT * FROM RawSales WHERE PropertyType = ’house’;

6‘‘‘

7

82. Retrieve the total price for houses sold in table 1:

9‘‘‘sql

10SELECT SUM(Price) as total_price FROM RawSales WHERE PropertyType = ’house’;

11‘‘‘

12

133. List the average number of bedrooms for properties sold in each table:

14‘‘‘sql

15SELECT AVG(Bedrooms) as avg_bedrooms_table0 FROM MaLga12345;

16SELECT AVG(Bedrooms) as avg_bedrooms_table1 FROM RawSales;

17‘‘‘

18

194. Find the number of sales made in each table in the year 2007:

20‘‘‘sql

21SELECT COUNT(*) as sales_count_table0 FROM MaLga12345 WHERE Saledate LIKE ’%

22SELECT COUNT(*) as sales_count_table1 FROM RawSales WHERE Datesold LIKE ’%

23‘‘‘

24

255. Retrieve the sales records where the postcode is 2607 from table 1:

26‘‘‘sql

27SELECT * FROM RawSales WHERE Postcode = 2607;

28‘‘‘

29

306. Identify the latest sale date recorded from both tables:

31‘‘‘sql

32SELECT MAX(Saledate) as latest_sale_date_table0 FROM MaLga12345;

33SELECT MAX(Datesold) as latest_sale_date_table1 FROM RawSales;

34‘‘‘

35

367. Display the details for sales records with a price greater than $500,000 in table 1:

37‘‘‘sql

38SELECT * FROM RawSales WHERE Price > 500000;

39‘‘‘

40

418. Retrieve the number of bedrooms for properties sold in table 0, grouped by property type:

42‘‘‘sql

43SELECT Type, AVG(Bedrooms) as avg_bedrooms FROM MaLga12345 GROUP BY Type;

44‘‘‘

45

469. List the sales records from both tables where the number of bedrooms is greater than 3:

47‘‘‘sql

48SELECT * FROM MaLga12345 WHERE Bedrooms > 3

49UNION

50SELECT * FROM RawSales WHERE Bedrooms > 3;

51‘‘‘

52

5310. Calculate the average price of properties sold in table 1 with 4 bedrooms:

54‘‘‘sql

55SELECT AVG(Price) as avg_price FROM RawSales WHERE Bedrooms = 4;

56‘‘‘

57

5811. Count the number of distinct postcodes present in table 1:

59‘‘‘sql

60SELECT COUNT(DISTINCT Postcode) as unique_postcodes FROM RawSales;

61‘‘‘

62

6312. Retrieve the sales records from table 1 where the price is greater than the average price:

64‘‘‘sql

65SELECT * FROM RawSales

66WHERE Price > (SELECT AVG(Price) FROM RawSales);

67‘‘‘

68

6913. Join tables based on matching postal codes and display the combined dataset:

70‘‘‘sql

71SELECT * FROM MaLga12345

72JOIN RawSales ON MaLga12345.MA = RawSales.Postcode;

73‘‘‘

74

7514. Identify the property types present in the two tables without duplication:

76‘‘‘sql

77SELECT DISTINCT Type FROM MaLga12345

78UNION

79SELECT DISTINCT PropertyType FROM RawSales;

80‘‘‘

81

8215. Show the top 5 highest-priced sales records from both tables:

83‘‘‘sql

84SELECT * FROM (SELECT * FROM MaLga12345

85UNION

86SELECT * FROM RawSales)

87ORDER BY Price DESC

88LIMIT 5;

89‘‘‘

SQL queries with higher complexity reflect better annotation quality. Thus we
include the steering phrase “with relatively high complexity” in our prompt.

##  Appendix C Costs of LLM Annotation

We use the public service from OpenAI to access ChatGPT with the GPT-3.5 large
language model. To support our claim that LLM annotations are cheap and
accessible, we provide an estimate of our annotation cost.

  1. 1.

Generating SQL programs costs less than 400 U.S. dollars.

  2. 2.

Generating Rel translations for SQL programs costs less than 2000 U.S.
dollars. The prompt for SQL-to-Rel translation is very long, as seen in
Appendix A.

  3. 3.

Generating input-output column annotations for the data tables costs less than
100 U.S. dollars.

The time spent on annotation depends on the account limitations. OpenAI API
calls can be made asynchronously to parallize the annotation process. All
annotation processes combined can be completed within 2 days.

##  Appendix D Wilcoxon Signed-Rank Test for TabPFN and AutoGluon Tabular
Classification Performance Difference

Wilcoxon signed-rank test (Wilcoxon, 1945) is a non-parametric statistical
test for paired data. We use the test to determine if there is a statistically
significant performance difference between TabPFN and AutoGluon on the tabular
classification problems from AnnotatedTables. For every tabular classification
problem among the 2,720 problems, for every specific time budget (either 1 min
or 5 mins), the performance metrics (either AUROC or cross entropy) of TabPFN
and AutoGluon form a paired data. The tests show that the median difference
between the pairs is nonzero with statistical significance for all settings.
The p-value of Wilcoxon signed-rank test is shown in Table 7. The paired bar
plots are in Figure 4.

Judging by the results of the Wilcoxon signed-rank test, TabPFN performs
better than AutoGluon with 1 minute time budget, and AutoGluon with 5 minutes
time budget performs better than TabPFN. However, the difference of medians is
small, and the standard deviation of performances are high. Based solely on
the general results of Wilcoxon signed-rank test, we cannot predict if one
model will outperform the other for a particular tabular classification
problem. The issue of “statistical significance verus practical significance”
(Kirk, 1996) is reflected here: even if a result is statistically significant,
it may not be practically significant if the effect size is very small.

Table 7: The alternative hypotheses and p-values of paired Wilcoxon signed-rank test for TabPFN and AutoGluon performance on the tabular classification problems in AnnotatedTables. All alternative hypotheses are accepted with high statistical significance.  |  | 1 Min (2725 problems) | 5 Mins (2720 problems)
---|---|---|---
AUROC | Alternative hypothesis | TabPFN > Autogluon | TabPFN < Autogluon
p-value | 3.798882e-20 | 6.643113e-11
Cross entropy | Alternative hypothesis | TabPFN < Autogluon | TabPFN > Autogluon
p-value | 4.25193e-63 | 2.44812e-06

(a) AUROC with 1 minute time budget.

(b) Cross entropy with 1 minute time budget.

(c) AUROC with 5 minutes time budget.

(d) Cross entropy with 5 minutes time budget.

Figure 4: Bar plots for the performance metrics of TabPFN and AutoGluon. The
outliers are not plotted as they are far from the quantile bars.

##  Appendix E Data Retention Rate During Dataset Construction

During the dataset construction, we discard the data point if data processing
fails at any step in order to maintain high quality results. In this section,
we list the main data processing steps and the number of data points remaining
at every step.

All intermediate data artifacts will be released as a part of the
AnnotatedTables dataset. We hope future researchers can study the quality of
LLM annotations and introduce methods for improvements.

#### SQL Annotation

SQL annotation starts with 70,000 Kaggle datasets and create 405,616 SQL
programs with 32,119 databases, in the following data processing steps.

  1. 1.

70,000 Kaggle datasets are collected.

  2. 2.

42,776 Kaggle datasets have CSV files to load as databases, representing
61.11% of the datasets collected from the last step.

  3. 3.

34,460 SQL databases can be loaded, for which the schema and example row
descriptions can be extracted, representing 80.56% of the datasets with CSV
files from the last step.

  4. 4.

32,926 SQL databases have SQL annotations generated by the Large Language
Model, representing 95.54% of the datasets from the last step. The total
number of SQL programs synthesized is 493,495.

  5. 5.

We execute 493,134 SQL programs from 32,902 SQL databases within a time out of
120 seconds per query to collect execution results, representing 99.93% of the
SQL programs and 99.93% of the databases from the last step. Every table in
the database is truncated to a maximum of 1,000 rows before execution of SQL
programs.

  6. 6.

405,616 SQL programs from 32,119 databases have execution results that are not
empty, labeled as “Valid” in Table 3, representing 82.25% of the SQL programs
executed.

#### SQL-to-Rel translation

SQL-to-Rel translation uses the SQL annotations in AnnotatedTables to
translate to Rel programs.

  1. 1.

405,616 SQL programs from 32,119 databases are valid.

  2. 2.

28,514 databases have at least 10 SQL programs each, forming 285,140 SQL
programs to be translated, reported in Table 5, representing 70.30% of the
valid SQL programs from the last step.

  3. 3.

116,327 Rel programs are translated correctly, with the same execution results
as the SQL queries, representing 40.80% of the source SQL programs from the
last step.

#### Input-output column annotation

The input-output columns are annotated for the evaluation of TabPFN. For a
given table, numeric input columns and a categorical output column need to be
identified for the tabular classification task. For this annotation, we start
with the 70,000 Kaggle datasets and evaluate TabPFN and AutoGluon on 2,720
tabular classification problems.

  1. 1.

70,000 Kaggle datasets are collected.

  2. 2.

42,776 Kaggle datasets have CSV files to load as databases, representing
61.11% of the datasets collected from the last step.

  3. 3.

34,009 tables are annotated with input-output columns by the LLM. One table is
taken from each database to avoid re-using similar or related tables from the
same database. 79.50% of the tables from the previous step have input-output
columns annotated from the LLM.

  4. 4.

10,529 tables have numeric input columns and a categorical output column, to
satisfy TabPFN’s requirements. The LLM may still generate an annotation when
the table does not have columns that satisfy the requirement. We do not remove
missing values from the tables or tables with missing values.

  5. 5.

We run both TabPFN and AutoGluon on every table from the last step. Data is
processed using TabPFN authors’ code, with data truncated to at most 1,000
training samples, 100 numerical features, and 10 classes. 2,725 problems can
be run and finish running within 1 minute time budget for both models, and
2,720 problems within 5 minute time budget.

##  Appendix F Tabular Dataset Annotation and Semantic Annotation of Tabular
Data

Our paper studies the “tabular dataset annotation problem”, that is the
problem of dataset annotation limited to the tabular data domain. Based on the
common concept of dataset annotation, tabular dataset annotation is the
process of labeling tabular data for machine learning. Currently, tabular
dataset annotation is done through expert annotators or crowd-sourced human
workers.

The “tabular dataset annotation problem” should not be confused with “semantic
annotation of tabular data”, which is also called “tabular data annotation” in
some context (PapersWithCode, 2024). “Semantic annotation of tabular data”
aims to match tabular data with knowledge graph entities.

## NeurIPS Paper Checklist

  1. 1.

Claims

  2. Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope?

  3. Answer: [Yes]

  4. Justification: The abstract and introductions made claims that are accurate and supported by the results in the paper. Our dataset is the largest SQL dataset with tabular data that support execution in the literature. Our paper is also the first to use large language models to annotate tabular datasets, to the best of our knowledge.

  5. Guidelines:

     * •

The answer NA means that the abstract and introduction do not include the
claims made in the paper.

     * •

The abstract and/or introduction should clearly state the claims made,
including the contributions made in the paper and important assumptions and
limitations. A No or NA answer to this question will not be perceived well by
the reviewers.

     * •

The claims made should match theoretical and experimental results, and reflect
how much the results can be expected to generalize to other settings.

     * •

It is fine to include aspirational goals as motivation as long as it is clear
that these goals are not attained by the paper.

  6. 2.

Limitations

  7. Question: Does the paper discuss the limitations of the work performed by the authors?

  8. Answer:[Yes]

  9. Justification: We measure and report the performance of LLM annotations, including statistics and analysis on the erroneous annotations throughout this paper. We have paragraphs in the paper devoted to discussing the limitations of our results.

  10. Guidelines:

     * •

The answer NA means that the paper has no limitation while the answer No means
that the paper has limitations, but those are not discussed in the paper.

     * •

The authors are encouraged to create a separate "Limitations" section in their
paper.

     * •

The paper should point out any strong assumptions and how robust the results
are to violations of these assumptions (e.g., independence assumptions,
noiseless settings, model well-specification, asymptotic approximations only
holding locally). The authors should reflect on how these assumptions might be
violated in practice and what the implications would be.

     * •

The authors should reflect on the scope of the claims made, e.g., if the
approach was only tested on a few datasets or with a few runs. In general,
empirical results often depend on implicit assumptions, which should be
articulated.

     * •

The authors should reflect on the factors that influence the performance of
the approach. For example, a facial recognition algorithm may perform poorly
when image resolution is low or images are taken in low lighting. Or a speech-
to-text system might not be used reliably to provide closed captions for
online lectures because it fails to handle technical jargon.

     * •

The authors should discuss the computational efficiency of the proposed
algorithms and how they scale with dataset size.

     * •

If applicable, the authors should discuss possible limitations of their
approach to address problems of privacy and fairness.

     * •

While the authors might fear that complete honesty about limitations might be
used by reviewers as grounds for rejection, a worse outcome might be that
reviewers discover limitations that aren’t acknowledged in the paper. The
authors should use their best judgment and recognize that individual actions
in favor of transparency play an important role in developing norms that
preserve the integrity of the community. Reviewers will be specifically
instructed to not penalize honesty concerning limitations.

  11. 3.

Theory Assumptions and Proofs

  12. Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

  13. Answer: [N/A]

  14. Justification: We have no theoretical results.

  15. Guidelines:

     * •

The answer NA means that the paper does not include theoretical results.

     * •

All the theorems, formulas, and proofs in the paper should be numbered and
cross-referenced.

     * •

All assumptions should be clearly stated or referenced in the statement of any
theorems.

     * •

The proofs can either appear in the main paper or the supplemental material,
but if they appear in the supplemental material, the authors are encouraged to
provide a short proof sketch to provide intuition.

     * •

Inversely, any informal proof provided in the core of the paper should be
complemented by formal proofs provided in appendix or supplemental material.

     * •

Theorems and Lemmas that the proof relies upon should be properly referenced.

  16. 4.

Experimental Result Reproducibility

  17. Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

  18. Answer: [Yes]

  19. Justification: We publicly release our code and data.

  20. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

If the paper includes experiments, a No answer to this question will not be
perceived well by the reviewers: Making the paper reproducible is important,
regardless of whether the code and data are provided or not.

     * •

If the contribution is a dataset and/or model, the authors should describe the
steps taken to make their results reproducible or verifiable.

     * •

Depending on the contribution, reproducibility can be accomplished in various
ways. For example, if the contribution is a novel architecture, describing the
architecture fully might suffice, or if the contribution is a specific model
and empirical evaluation, it may be necessary to either make it possible for
others to replicate the model with the same dataset, or provide access to the
model. In general. releasing code and data is often one good way to accomplish
this, but reproducibility can also be provided via detailed instructions for
how to replicate the results, access to a hosted model (e.g., in the case of a
large language model), releasing of a model checkpoint, or other means that
are appropriate to the research performed.

     * •

While NeurIPS does not require releasing code, the conference does require all
submissions to provide some reasonable avenue for reproducibility, which may
depend on the nature of the contribution. For example

       1. (a)

If the contribution is primarily a new algorithm, the paper should make it
clear how to reproduce that algorithm.

       2. (b)

If the contribution is primarily a new model architecture, the paper should
describe the architecture clearly and fully.

       3. (c)

If the contribution is a new model (e.g., a large language model), then there
should either be a way to access this model for reproducing the results or a
way to reproduce the model (e.g., with an open-source dataset or instructions
for how to construct the dataset).

       4. (d)

We recognize that reproducibility may be tricky in some cases, in which case
authors are welcome to describe the particular way they provide for
reproducibility. In the case of closed-source models, it may be that access to
the model is limited in some way (e.g., to registered users), but it should be
possible for other researchers to have some path to reproducing or verifying
the results.

  21. 5.

Open access to data and code

  22. Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

  23. Answer: [Yes]

  24. Justification: We publicly release our code and data. The access instructions are in the supplementary material. The source Kaggle datasets can be accessed publicly.

  25. Guidelines:

     * •

The answer NA means that paper does not include experiments requiring code.

     * •

Please see the NeurIPS code and data submission guidelines
(https://nips.cc/public/guides/CodeSubmissionPolicy) for more details.

     * •

While we encourage the release of code and data, we understand that this might
not be possible, so “No” is an acceptable answer. Papers cannot be rejected
simply for not including code, unless this is central to the contribution
(e.g., for a new open-source benchmark).

     * •

The instructions should contain the exact command and environment needed to
run to reproduce the results. See the NeurIPS code and data submission
guidelines (https://nips.cc/public/guides/CodeSubmissionPolicy) for more
details.

     * •

The authors should provide instructions on data access and preparation,
including how to access the raw data, preprocessed data, intermediate data,
and generated data, etc.

     * •

The authors should provide scripts to reproduce all experimental results for
the new proposed method and baselines. If only a subset of experiments are
reproducible, they should state which ones are omitted from the script and
why.

     * •

At submission time, to preserve anonymity, the authors should release
anonymized versions (if applicable).

     * •

Providing as much information as possible in supplemental material (appended
to the paper) is recommended, but including URLs to data and code is
permitted.

  26. 6.

Experimental Setting/Details

  27. Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

  28. Answer: [Yes]

  29. Justification: Our experimental settings are discussed in the paper with great details. We also release our code as the supplemental material.

  30. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The experimental setting should be presented in the core of the paper to a
level of detail that is necessary to appreciate the results and make sense of
them.

     * •

The full details can be provided either with the code, in appendix, or as
supplemental material.

  31. 7.

Experiment Statistical Significance

  32. Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

  33. Answer: [Yes]

  34. Justification: For experiments, p-values and bar plots are included in the paper and the appendix.

  35. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The authors should answer "Yes" if the results are accompanied by error bars,
confidence intervals, or statistical significance tests, at least for the
experiments that support the main claims of the paper.

     * •

The factors of variability that the error bars are capturing should be clearly
stated (for example, train/test split, initialization, random drawing of some
parameter, or overall run with given experimental conditions).

     * •

The method for calculating the error bars should be explained (closed form
formula, call to a library function, bootstrap, etc.)

     * •

The assumptions made should be given (e.g., Normally distributed errors).

     * •

It should be clear whether the error bar is the standard deviation or the
standard error of the mean.

     * •

It is OK to report 1-sigma error bars, but one should state it. The authors
should preferably report a 2-sigma error bar than state that they have a 96%
CI, if the hypothesis of Normality of errors is not verified.

     * •

For asymmetric distributions, the authors should be careful not to show in
tables or figures symmetric error bars that would yield results that are out
of range (e.g. negative error rates).

     * •

If error bars are reported in tables or plots, The authors should explain in
the text how they were calculated and reference the corresponding figures or
tables in the text.

  36. 8.

Experiments Compute Resources

  37. Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

  38. Answer: [Yes]

  39. Justification: Our LLM annotation methods aim to reduce the costs of dataset annotation, and the estimated costs are in the Appendix.

  40. Guidelines:

     * •

The answer NA means that the paper does not include experiments.

     * •

The paper should indicate the type of compute workers CPU or GPU, internal
cluster, or cloud provider, including relevant memory and storage.

     * •

The paper should provide the amount of compute required for each of the
individual experimental runs as well as estimate the total compute.

     * •

The paper should disclose whether the full research project required more
compute than the experiments reported in the paper (e.g., preliminary or
failed experiments that didn’t make it into the paper).

  41. 9.

Code Of Ethics

  42. Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines?

  43. Answer: [Yes]

  44. Justification: We have reviewed the NeurIPS Code of Ethics and confirm that our research conforms to it.

  45. Guidelines:

     * •

The answer NA means that the authors have not reviewed the NeurIPS Code of
Ethics.

     * •

If the authors answer No, they should explain the special circumstances that
require a deviation from the Code of Ethics.

     * •

The authors should make sure to preserve anonymity (e.g., if there is a
special consideration due to laws or regulations in their jurisdiction).

  46. 10.

Broader Impacts

  47. Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

  48. Answer: [N/A]

  49. Justification: Dataset annotation is a standard component in all machine learning projects, and improvements on the dataset annotation process should bring no risks of negative societal impacts.

  50. Guidelines:

     * •

The answer NA means that there is no societal impact of the work performed.

     * •

If the authors answer NA or No, they should explain why their work has no
societal impact or why the paper does not address societal impact.

     * •

Examples of negative societal impacts include potential malicious or
unintended uses (e.g., disinformation, generating fake profiles,
surveillance), fairness considerations (e.g., deployment of technologies that
could make decisions that unfairly impact specific groups), privacy
considerations, and security considerations.

     * •

The conference expects that many papers will be foundational research and not
tied to particular applications, let alone deployments. However, if there is a
direct path to any negative applications, the authors should point it out. For
example, it is legitimate to point out that an improvement in the quality of
generative models could be used to generate deepfakes for disinformation. On
the other hand, it is not needed to point out that a generic algorithm for
optimizing neural networks could enable people to train models that generate
Deepfakes faster.

     * •

The authors should consider possible harms that could arise when the
technology is being used as intended and functioning correctly, harms that
could arise when the technology is being used as intended but gives incorrect
results, and harms following from (intentional or unintentional) misuse of the
technology.

     * •

If there are negative societal impacts, the authors could also discuss
possible mitigation strategies (e.g., gated release of models, providing
defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms
to monitor how a system learns from feedback over time, improving the
efficiency and accessibility of ML).

  51. 11.

Safeguards

  52. Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

  53. Answer: [N/A]

  54. Justification: Dataset annotation is a standard component in all machine learning projects and our work poses no risks of misuse.

  55. Guidelines:

     * •

The answer NA means that the paper poses no such risks.

     * •

Released models that have a high risk for misuse or dual-use should be
released with necessary safeguards to allow for controlled use of the model,
for example by requiring that users adhere to usage guidelines or restrictions
to access the model or implementing safety filters.

     * •

Datasets that have been scraped from the Internet could pose safety risks. The
authors should describe how they avoided releasing unsafe images.

     * •

We recognize that providing effective safeguards is challenging, and many
papers do not require this, but we encourage authors to take this into account
and make a best faith effort.

  56. 12.

Licenses for existing assets

  57. Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

  58. Answer: [Yes]

  59. Justification: The assets we created are released with public licenses. The original licenses of the Kaggle datasets can be found on the dataset webpages, and we do not re-release their data or modify their licenses to conform to Kaggle’s terms of use. Users may download the Kaggle datasets on their own through public Kaggle API access, following our process or using our tools. Kaggle datasets have licenses that permit data analysis usages.

  60. Guidelines:

     * •

The answer NA means that the paper does not use existing assets.

     * •

The authors should cite the original paper that produced the code package or
dataset.

     * •

The authors should state which version of the asset is used and, if possible,
include a URL.

     * •

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

     * •

For scraped data from a particular source (e.g., website), the copyright and
terms of service of that source should be provided.

     * •

If assets are released, the license, copyright information, and terms of use
in the package should be provided. For popular datasets,
paperswithcode.com/datasets has curated licenses for some datasets. Their
licensing guide can help determine the license of a dataset.

     * •

For existing datasets that are re-packaged, both the original license and the
license of the derived asset (if it has changed) should be provided.

     * •

If this information is not available online, the authors are encouraged to
reach out to the asset’s creators.

  61. 13.

New Assets

  62. Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

  63. Answer: [Yes]

  64. Justification: Our model and dataset are well documented and released publicly.

  65. Guidelines:

     * •

The answer NA means that the paper does not release new assets.

     * •

Researchers should communicate the details of the dataset/code/model as part
of their submissions via structured templates. This includes details about
training, license, limitations, etc.

     * •

The paper should discuss whether and how consent was obtained from people
whose asset is used.

     * •

At submission time, remember to anonymize your assets (if applicable). You can
either create an anonymized URL or include an anonymized zip file.

  66. 14.

Crowdsourcing and Research with Human Subjects

  67. Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

  68. Answer: [N/A]

  69. Justification: The paper does not involve crowdsourcing nor research with human subjects.

  70. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Including this information in the supplemental material is fine, but if the
main contribution of the paper involves human subjects, then as much detail as
possible should be included in the main paper.

     * •

According to the NeurIPS Code of Ethics, workers involved in data collection,
curation, or other labor should be paid at least the minimum wage in the
country of the data collector.

  71. 15.

Institutional Review Board (IRB) Approvals or Equivalent for Research with
Human Subjects

  72. Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

  73. Answer: [N/A]

  74. Justification: The paper does not involve crowdsourcing nor research with human subjects.

  75. Guidelines:

     * •

The answer NA means that the paper does not involve crowdsourcing nor research
with human subjects.

     * •

Depending on the country in which research is conducted, IRB approval (or
equivalent) may be required for any human subjects research. If you obtained
IRB approval, you should clearly state this in the paper.

     * •

We recognize that the procedures for this may vary significantly between
institutions and locations, and we expect authors to adhere to the NeurIPS
Code of Ethics and the guidelines for their institution.

     * •

For initial submissions, do not include any information that would break
anonymity (if applicable), such as the institution conducting the review.