# Is your Statement Purposeless? Predicting Computer Science Graduation
Admission Acceptance based on Statement Of Purpose

Diptesh Kanojia$\dagger$,$\clubsuit$,$\star$, Nikhil Wani$\ddagger$,$\dagger$,
Pushpak Bhattacharyya$\dagger$
$\dagger$Center for Indian Language Technology, IIT Bombay, India
$\clubsuit$IITB-Monash Research Academy, India
$\star$Monash University, Australia
$\dagger${diptesh<EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

We present a quantitative, data-driven machine learning approach to mitigate
the problem of unpredictability of Computer Science Graduate School
Admissions. In this paper, we discuss the possibility of a system which may
help prospective applicants evaluate their Statement of Purpose (SOP) based on
our system output. We, then, identify feature sets which can be used to train
a predictive model. We train a model over fifty manually verified SOPs for
which it uses an SVM classifier and achieves the highest accuracy of 92% with
10-fold cross validation. We also perform experiments to establish that Word
Embedding based features and Document Similarity based features outperform
other identified feature combinations. We plan to deploy our application as a
web service and release it as a FOSS service.

##  1 Introduction

Computer Science (CS) graduate admissions process often involves holistic
evaluation of prospective applicant based on multiple subjective and
quantitative parameters [Ward, 2006]. Amongst these parameters the applicant’s
Statement of Purpose (SOP) serves as a document to convince its readers’ i.e.
the faculty on the selection committee - that one has recorded solid
achievements which reflect promise for success in graduate study and hence
submission of such a good quality SOP becomes of paramount importance.

Furthermore, Graduate admissions to most Elite universities in the United
States of America (USA) only open twice every year - Fall and Spring
semesters.

Terminology: We use the terms essay and SOP interchangeably further during our
discussion of the work.

##  2 Motivation

Applicants spend a great deal of time writing SOPs for the admissions process.
A well written SOP is a must for an applicant to ensure their admission in any
university, and more so for elite universities. Their thoughts and ideas
should be organized in their statement. University
guidelines111http://grad.berkeley.edu/admissions/apply/statement-
purpose/222http://admission.stanford.edu/apply/freshman/essays.html, Alumni
blogs333http://alumnus.caltech.edu/~natalia/studyinus/guide/statement/q&a.htm,
and Admission consultancy blogs444http://www.happyschools.com/strengthen-your-
graduate-school-application/ recommend spending ample time on each SOP and
tailoring it to perfection. They also recommend stylometry for writing an
essay i.e. word limit, active voice, coherence, and continuity. Various NLP
applications like Essay grading [Larkey, 1998], Text Summarization [Gupta and
Lehal, 2010] and Sentiment Analysis [Joshi et al., 2015] utilize these
features. Hence, we believe that an application that evaluates their statement
is crucial. The key question that this paper attempts to answer is:

‘ _Can information gained from an SOP be used to predict the outcome of a
candidate application for graduate school admissions?_ ’

##  3 Related Work

?) discuss a qualitative model for Graduate Admissions to Computer Science
programs but do not use any Machine Learning or Deep Learning based techniques
for estimating a likelihood. According to them, other factors which affect the
decision of the committee reviewing the applications include Graduate Record
Examinations (GRE) score, Undergraduate Grade Point Average (GPA), Letters of
Recommendation (LORs), Financial preparation of a candidate, Alignment with
institute needs keeping in mind the diversity goals of the university, and
lastly the Undergraduate Major of the candidate. They require the user to rate
the application parameters and provide ratings as an input to their system. As
an output, they provide an estimate of acceptance based on their
model555http://www.cs.utep.edu/nigel/estimator/.

On the other hand, we employ the existing state-of-the-art techniques,
identify features and use some of them to predict the acceptance of a
candidate. We acknowledge that we do not model all parameters described above.

Another similar study [Raghunathan, 2010] tries to subjectively discuss the
admissions process and details the factors which participate in the decision
making process of an admission committee. They break the components of a
graduate school admissions process and state that SOP is one of the trickiest
components of an overall application. They also note that too long an SOP
would deter the chances of selection of the candidate. In light of these
studies, we focus on creating a model which is able to grade an SOP based on
ML techniques.

Text Similarity and related measures [Choi et al., 2010, Adomavicius and
Tuzhilin, 2005, Gomaa and Fahmy, 2013] have been extensively studied and used
for various NLP applications viz. Information Retrieval [Salton et al., 1983],
Sense Disambiguation [Resnik and others, 1999]. To the best of our knowledge,
there is no reported study which evaluates SOPs based on the features
identified by us, or use ML and DL based techniques of this kind, at the time
of submission. Most of the articles list various parameters which are
considered by an admissions committee and a Statement of Purpose (SOP) is a
common factor among all.

##  4 Experiment Design and Setup

In this section, we provide details about our experiment setup and features
used for the classification task.

###  4.1 Dataset

We create our dataset by collecting essays from i) Acquaintances ii) Publicly
disclosed SOPs from personal websites, and iii) Admission consultancy blogs.
For calculating the similarity measures, we concatenate the essays of the
successful applicants, and create a corpus which is used for comparison with
both training and testing data.

We collect a total of 50 manually verified SOPs from Elite Universities (low
acceptance rate <= 15%) and rejected essays equally split into two sets. We
plan to release the dataset publicly under the CC-BY-
SA-4.0666https://creativecommons.org/licenses/by-sa/4.0/ license.

###  4.2 Methodology

We use conventional Machine Learning (ML) algorithms [Hall et al., 2009] like
Support Vector Machines (SVM) [Vapnik, 2013], Logistic Regression (LR) [Walker
and Duncan, 1967], and Random Forest Decision Trees (RFDT) [Ho, 1998] for the
task and provide a comparison in Section 5.

We use deep learning approaches and deploy a simple Feed Forward Neural
Network to classify the SOPs. We split our data in two folds where the first
half is used for training, and the second half is then split into tuning and
testing datasets. We also use Multilayer Perceptron, another simple Feed
Forward Neural Network (NN) and perform a standard 10-fold cross validation on
our dataset. We do acknowledge the modest size of our dataset, but we provide
rigorous experimentation including an ablation test to verify that our
performance on all classes of our data are unbiased.

Classifier | $P_{acc}$ | $P_{rej}$ | $\mathbf{P_{avg}}$ | $R_{acc}$ | $R_{rej}$ | $\mathbf{R_{avg}}$ | $F_{acc}$ | $F_{rej}$ | $\mathbf{F_{avg}}$
---|---|---|---|---|---|---|---|---|---
RFDT | 0.86 | 0.79 | 0.83 | 0.76 | 0.88 | 0.82 | 0.81 | 0.83 | 0.82
LR | 0.69 | 0.83 | 0.76 | 0.88 | 0.60 | 0.74 | 0.77 | 0.70 | 0.74
SVM | 0.89 | 0.96 | 0.92 | 0.96 | 0.88 | 0.92 | 0.92 | 0.92 | 0.92
Neural Network Based
| Multilayer Perceptron
---
(Train-Test Split)
- | - | 0.82 | - | - | 0.82 | - | - | 0.82
| Feed Forward NN (FFNN)
---
(Train-Tune-Test Split)
- | - | 0.36 | - | - | 0.60 | - | - | 0.45
Table 1: Performance of our model on 10-fold cross validation

###  4.3 Experiment Design

We cluster the set of features in the following groups - a) Textual Features
\- Feature values based on text contained within the document, b) Word
Embedding based Features \- Features based on average of vector values
provided by pre-trained model on Google News Corpora, c) Similarity Score
based and Error based features \- Features based on Document Similarity, and
other features based on errors in the document. The last set of features have
been identified by us, and are our contribution to the work. We, then, use the
algorithms mentioned above to calculate precision, recall and F-Score on each
feature set.

We also perform an Ablation test to see which feature set combination is
performing the best.

###  4.4 System Architecture

Our architecture, shown in figure 1, provides the necessary details about the
working of our system. The system takes as input the essay of a prospective
applicant, calculates feature values for Similarity Score and Error based
features along with Word Embedding based features and predicts an accept or
reject based on the classification model being used.

Figure 1: System Architecture

###  4.5 Features Used

We use the following textual features for evaluating the SOPs. These features
have been identified via surveying linguistic properties of a text which may
affect the organization and quality of an essay.

####  4.5.1 Word Embeddings based Features

  1. 1.

Average Word Vector Scores \- Average of word vectors of each word in the
statement calculated using pre-trained Google News word vectors [Mikolov et
al., 2013].

####  4.5.2 Textual Features

  1. 1.

PoS Ratios \- Ratio of nouns, adjectives, adverbs, and verbs to the entire
text, obtained using NLTK777http://www.nltk.org/ [Loper and Bird, 2002].

  2. 2.

Discourse Connectors \- It is the number of discourse connectors in the essay
computed using a list of discourse
connectors888http://www.cfilt.iitb.ac.in/cognitive-nlp/.

  3. 3.

Count of Named Entities \- Number of named entities in the essay. We tried
using this as a feature but this drastically lowered the F-scores, and had to
be avoided in the final set of reported experiments.

  4. 4.

Readability \- The Flesch Reading Ease Score (FRES) of the text [Flesch,
1948].

  5. 5.

Length features \- Number of words in the sentence, number of words in the
paragraph, and average word length.

  6. 6.

Coreference Distance \- Sum of token distance between co-referring mentions.

  7. 7.

Degree of Polysemy \- Average number of WordNet [Fellbaum, 2010] senses per
word.

| Individual Feature Sets (N-fold)
---|---
Features | 2-F | 5-F | 10-F | 50% Split
T [14] | 54 | 46 | 44 | 40
WE [300] | 48 | 78 | 40 | 44
SE [3] | 48 | 56 | 56 | 49
| Combination of Feature Sets
T + WE [314] | 56 | 62 | 62 | 52
T + SE [17] | 48 | 50 | 38 | 30
SE + WE [303] | 90 | 92 | 92 | 92
T + WE + SE [318] | 52 | 50 | 53 | 43
Table 2: Ablation test on feature sets using Multi-fold Cross Validation

####  4.5.3 Document Similarity Score and Error based Features

  1. 1.

Cosine Similarity \- Cosine Similarity Score of an SOP with the corpus of
accepted essays dataset, where we ensure that the SOP being compared is not a
part of the accepted essay corpus.

  2. 2.

Similarity-based features using GloVe \- The similarity between every pair of
content words in adjacent sentences. The similarity is computed as the cosine
similarity between their word vectors from the pre-trained GloVe word
embeddings [Pennington et al., 2014]. We calculate the mean and maximum
similarity values.

  3. 3.

Spell Check Errors \- We use PyEnchant999http://pythonhosted.org/pyenchant/ to
embed a spell checker and count the number of errors in each document. The
count is then used as another feature for training classifier.

  4. 4.

Out of Vocabulary Words \- We use the pre-trained Google news word embeddings
and find out word vectors for every token in the document. The tokens which do
not return any vector are either rare words or in all probability out of
vocabulary words. We use the count of such tokens as another feature set.

##  5 Results

We perform the experiments detailed in section 4.3 and report our results on
10-fold cross validation. Among the experiments we perform, we achieve the
highest F-score of 92% using the SVM classifier with an RBF Kernel. The
results are shown in table 1 and discussed in Section 6.

Table 1 clearly indicates that SVM outperforms Random Forest Decision Trees
(RFDT) with a margin of 9%, Logistic Regression (LR) with a margin of 18%,
Neural Network based Multilayer Perceptron with a margin of 10%, and another
Feed Forward Neural Network (FFNN) with a margin of 47%. We further discuss
the impact and justifications of these results in Section 6.

We also perform a multi-fold ablation test, using SVM Classifier, on the
feature sets identified in section 4.3. The results for the ablation test are
shown in Table 2. The table clearly identifies that Similarity Scores and
Error based features along with Word Embedding based features give us the best
results.

##  6 Discussion

In order to identify the features that contribute to the modeled non-linearity
of SVM and our best reported accuracy of 92%, we conduct a comprehensive
ablation test. Feature sets mentioned in Section 4.3 were considered. A total
of 317 features were ablated based on their sets via multi-fold stratified
cross validation experiments and additionally in an experiment with 50% split
of the dataset as shown in the Table 2.

It was found that the 14 identified Textual (T) features do not contribute
significantly to our model. We extrapolate that these features may have worked
better in another context such as Sentiment Analysis [Mishra et al., 2017], or
Essay Grading [Valenti et al., 2003], but not for the task of SOP
Classification. Our task primarily aims at labeling an SOP with an accept or
reject, however, we observe that Textual features do not differentiate well
between coherent and incoherent essays. We also observe that Word Embedding
(WE) features of 300 dimensions contribute significantly towards the accuracy
of our final model. While they do not contribute notably when used to perform
classification independently, combining them with Similarity Score and Error
Based (SE) feature set form our best reported model i.e. SE + WE.

##  7 Conclusion and Future Work

In this paper we demonstrate the applicability of a data driven approach to
mitigate the unpredictability of Computer Science graduate admissions process.
We build a corpus of fifty manually verified SOPs from Accepted applicants to
Elite Universities (low acceptance rate <= 15%) rejected SOPs. We show that a
combination of Cosine Similarity, Error based features and Word Embedding
based features outperform any of the textual features based combinations, for
this task. Based on the ablation tests conducted, we model an SVM classifier
that predicts with significantly high accuracy.

In future, we plan to integrate Parts-of-speech (POS) based similarity
measures and Recurrent Neural Networks (RNN) [Cho et al., 2014] which have
been shown to work well with textual data. Integration of other traditional
metrics of a candidate’s application performance measure such as GRE, Test of
English as a Foreign Language (TOEFL) / International English Language Testing
System (IELTS) score and GPA will further robustly extend this model. We also
plan to translate this novel research to an open source web application which
would allow prospective applicants to evaluate their SOPs with our system.

## References

  * [Adomavicius and Tuzhilin, 2005]  Gediminas Adomavicius and Alexander Tuzhilin.  2005\.  Toward the next generation of recommender systems: A survey of the state-of-the-art and possible extensions.  IEEE transactions on knowledge and data engineering, 17(6):734–749.
  * [Cho et al., 2014]  Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  2014\.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  arXiv preprint arXiv:1406.1078.
  * [Choi et al., 2010]  Seung-Seok Choi, Sung-Hyuk Cha, and Charles C Tappert.  2010\.  A survey of binary similarity and distance measures.  Journal of Systemics, Cybernetics and Informatics, 8(1):43–48.
  * [Fellbaum, 2010]  Christiane Fellbaum.  2010\.  Wordnet.  Theory and applications of ontology: computer applications, pages 231–243.
  * [Flesch, 1948]  Rudolph Flesch.  1948\.  A new readability yardstick.  Journal of applied psychology, 32(3):221.
  * [Gomaa and Fahmy, 2013]  Wael H Gomaa and Aly A Fahmy.  2013\.  A survey of text similarity approaches.  International Journal of Computer Applications, 68(13).
  * [Gupta and Lehal, 2010]  Vishal Gupta and Gurpreet Singh Lehal.  2010\.  A survey of text summarization extractive techniques.  Journal of emerging technologies in web intelligence, 2(3):258–268.
  * [Hall et al., 2009]  Mark Hall, Eibe Frank, Geoffrey Holmes, Bernhard Pfahringer, Peter Reutemann, and Ian H Witten.  2009\.  The weka data mining software: an update.  ACM SIGKDD explorations newsletter, 11(1):10–18.
  * [Ho, 1998]  Tin Kam Ho.  1998\.  The random subspace method for constructing decision forests.  IEEE transactions on pattern analysis and machine intelligence, 20(8):832–844.
  * [Joshi et al., 2015]  Aditya Joshi, Vinita Sharma, and Pushpak Bhattacharyya.  2015\.  Harnessing context incongruity for sarcasm detection.  In ACL (2), pages 757–762.
  * [Larkey, 1998]  Leah S. Larkey.  1998\.  Automatic essay grading using text categorization techniques.  In Proceedings of the 21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’98, pages 90–95, New York, NY, USA. ACM.
  * [Loper and Bird, 2002]  Edward Loper and Steven Bird.  2002\.  Nltk: The natural language toolkit.  In Proceedings of the ACL-02 Workshop on Effective tools and methodologies for teaching natural language processing and computational linguistics-Volume 1, pages 63–70. Association for Computational Linguistics.
  * [Mikolov et al., 2013]  Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean.  2013\.  Distributed representations of words and phrases and their compositionality.  In Advances in neural information processing systems, pages 3111–3119.
  * [Mishra et al., 2017]  Abhijit Mishra, Diptesh Kanojia, Seema Nagar, Kuntal Dey, and Pushpak Bhattacharyya.  2017\.  Leveraging cognitive features for sentiment analysis.  arXiv preprint arXiv:1701.05581.
  * [Pennington et al., 2014]  Jeffrey Pennington, Richard Socher, and Christopher Manning.  2014\.  Glove: Global vectors for word representation.  In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1532–1543, Doha, Qatar, October. Association for Computational Linguistics.
  * [Raghunathan, 2010]  Karthik Raghunathan.  2010\.  Demystifying the american graduate admissions process.  StudyMode. com.
  * [Resnik and others, 1999]  Philip Resnik et al.  1999\.  Semantic similarity in a taxonomy: An information-based measure and its application to problems of ambiguity in natural language.  J. Artif. Intell. Res.(JAIR), 11:95–130.
  * [Salton et al., 1983]  Gerard Salton, Edward A Fox, and Harry Wu.  1983\.  Extended boolean information retrieval.  Communications of the ACM, 26(11):1022–1036.
  * [Valenti et al., 2003]  Salvatore Valenti, Francesca Neri, and Alessandro Cucchiarelli.  2003\.  An overview of current research on automated essay grading.  Journal of Information Technology Education: Research, 2(1):319–330.
  * [Vapnik, 2013]  Vladimir Vapnik.  2013\.  The nature of statistical learning theory.  Springer science & business media.
  * [Walker and Duncan, 1967]  Strother H Walker and David B Duncan.  1967\.  Estimation of the probability of an event as a function of several independent variables.  Biometrika, 54(1-2):167–179.
  * [Ward, 2006]  Nigel Ward.  2006\.  Towards a model of computer science graduate admissions decisions.  JACIII, 10(3):372–383.


# Can Foundation Models Wrangle Your Data?

Avanika Narayan, Ines Chami$\dagger$, Laurel Orr, Simran Arora, Christopher Ré
Stanford University and $\dagger$Numbers Station avanika, lorr1, chrismre,
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

Foundation Models (FMs) are models trained on large corpora of data that, at
very large scale, can generalize to new tasks without any task-specific
finetuning. As these models continue to grow in size, innovations continue to
push the boundaries of what these models can do on language and image tasks.
This paper aims to understand an underexplored area of FMs: classical data
tasks like cleaning and integration. As a proof-of-concept, we cast five data
cleaning and integration tasks as prompting tasks and evaluate the performance
of FMs on these tasks. We find that large FMs generalize and achieve SoTA
performance on data cleaning and integration tasks, even though they are not
trained for these data tasks. We identify specific research challenges and
opportunities that these models present, including challenges with private and
domain specific data, and opportunities to make data management systems more
accessible to non-experts. We make our code and experiments publicly available
at: https://github.com/HazyResearch/fm_data_tasks.

##  1\. Introduction

Figure 1.  A large FM can address an entity matching task using prompting.
Rows are serialized into text and passed to the FM with the question “Are
products A and B the same?”. The FM then generates a string “Yes” or “No” as
the answer.

Foundation Models (FMs) (Bommasani et al., 2021) are models trained on broad
data that can be adapted to a wide range of downstream tasks. These models
have achieved substantial gains across many semantically challenging tasks
such as question answering (Brown et al., 2020), knowledge base construction
(Petroni et al., 2019), and information retrieval (Guu et al., 2020). As they
have scaled to hundreds of billions of parameters (e.g. GPT-3 (Brown et al.,
2020), PaLM (Chowdhery et al., 2022)), large FMs have demonstrated surprising
emergent behaviors and good zero-shot generalization to new tasks (i.e. no
task-specific finetuning) on domains vastly different from the data they were
pretrained on (Chowdhery et al., 2022). These large FMs are often
autoregressive language models (e.g. GPT-3 and PaLM) that are trained to
predict the next word in large text corpora and can be adapted to new tasks
given a simple natural language description of the task (see Figure 1). These
breakthrough capabilities have led to a race for building bigger and better
models, and innovations continue to push the boundaries of what large FMs can
do on a variety of hard language tasks.

A natural question that arises is whether these advances can benefit hard
classical data tasks (e.g. data cleaning and integration). While it is clear
that FMs benefit text-intensive tasks, it is not clear whether these models
can be applied to data tasks over structured data. The symbols commonly found
in structured data (e.g. dates, numbers, alphanumeric codes) are less frequent
in natural language text so it is unclear that FMs possess the ability to
reason over them. Moreover, since FMs are trained to predict the next word, it
is non-obvious that they can work out-of-the-box on complex data tasks. This
paper explores the aforementioned question and introduces a new research
vision for leveraging FMs for data management, focusing on data cleaning and
integration tasks—two keys steps in data-driven enterprise pipelines.

Recently, a large body of research has applied machine learning (ML) (Konda et
al., 2016) and deep learning (DL) (Li et al., 2020; Mudgal et al., 2018)
methods—namely pretrained language models (PLMs) like BERT (Devlin et al.,
2018)—to semantically-complex data tasks. However, these approaches still
require a significant amount of engineering effort as they rely on:

  * •

Task-specific architectures: Data cleaning and integration encapsulate many
different tasks such as entity matching (Papadakis et al., 2020), schema
matching (Sutanta et al., 2016), and error detection (Heidari et al., 2019).
Existing approaches, whether they are rule-, ML- or DL-based, vary greatly
from one task to the other, often with complex, task-specific architectures.
For instance, adapting BERT to data tasks requires architectural changes and
finetuning the entire model for each task. This leads to siloed and hard-to-
maintain systems.

  * •

Hard-coded knowledge: Data tasks often rely on domain knowledge (e.g.
understanding the relationship between a city and its zip code for data
cleaning constraints) and commonsense reasoning. These are usually hard-coded
with human-engineered rules or external knowledge bases (Chu et al., 2015;
Rekatsinas et al., 2017). Consequently, systems can be brittle and fail to
generalize to a diverse set of domains.

  * •

Labeled data: ML- and DL-based solutions require copious amounts of hand-
labeled data (Adadi, 2021). For instance, PLMs that have achieved state-of-
the-art (SoTA) results on data tasks (e.g. Ditto (Gururangan et al., 2020b))
require a significant amount of task-specific labeled data and fine-tuning to
achieve good performance. Labeling data for each task is engineering intensive
and adds to the difficulty of maintaining data cleaning and integration
systems.

Excitingly, FMs display several useful properties that make them an appealing
choice compared to traditional approaches:

  * •

Task-agnostic architecture: As a result of their natural language interface,
FMs can be applied to a wide-range of tasks. For instance, Figure 1 shows how
an entity matching task—which requires identifying whether two table entries
refer to the same entity—can be cast as a prompting task. This unifying
interface eliminates the need for siloed architectures, in contrast to
existing learned approaches where architectures need to be carefully crafted
for each task (e.g. task-specific classification layer).

  * •

Encoded knowledge: Because FMs are trained on large, generic corpora of data,
they contain knowledge about an extensive set of common entities, and thus do
not rely on human-engineered rules to acquire knowledge (Razniewski et al.,
2021).

  * •

Limited to no labeled data: FMs can be applied to a breadth of tasks with
little to no labeled data (e.g. few-shot and zero-shot). When a FM needs to be
fine-tuned, it typically needs dramatically less labeled data to achieve
competitive results (Kaplan et al., 2020).

Our goal is to better understand if large FMs can be applied to data
integration and cleaning tasks. We study the behavior of GPT-3—an early and
promising FM. While GPT-3 is already a high quality model, we expect the
significant investment in FMs from both academia and industry to lead to more
performant and scalable FMs over time. Like many other communities, the data
management community stands to benefit from these trends. As such, we aim to
understand the advantages and limitations of FMs on data tasks, by focusing on
three key questions.

How well do large FMs transfer to data tasks? To answer this, we cast several
data tasks as natural language generation tasks (Section 3) and explore
whether a single FM can generalize well to these tasks. In Section 4.2, we
quantify the zero- and few-shot performance of FMs on five enterprise data
tasks: entity matching, error detection, schema matching, data transformation,
and data imputation. We find that the largest GPT-3 variant (175B parameters)
outperforms SoTA ML-and DL-based approaches on these tasks with few examples.
This is particularly surprising since prior approaches are fully-finetuned on
task-specific labeled data for these tasks, while GPT-3-175B is simply
pretrained to generate text.

What are the caveats in applying FMs to data tasks? In Section 4.3, we unpack
the few-shot “prompt tuning” process—serializing tabular data to text, casting
data tasks as text generation tasks and constructing demonstrative task
examples—for applying FMs to data tasks. We quantify the effects of prompt
formatting variations on performance and the differences between manually and
randomly selecting task examples. We find that FMs are brittle to differences
in prompt formatting and that performance improves when prompts are manually
selected versus randomly selected.

What opportunities do FMs present for data tasks and what are the relevant
research challenges? Finally, in Section 5, we discuss the potential
challenges and related research questions with using FMs in data management
pipelines. We discuss the forthcoming shift in how ML systems are built,
challenges around updating FM knowledge, and opportunities and considerations
pertaining to private, temporal and local data.

We hope that our preliminary exploration will encourage the data management
community to explore the effectiveness of FMs for other data tasks and develop
techniques to overcome the shortcomings of FMs in this setting.

Figure 2. Different ways to use FMs with “in-context” learning (Brown et al.,
2020) on an error detection task. For zero-shot (left), the prompt is the task
description and the example to complete. For few-shot (right), the prompt adds
demonstrations of how to complete the task.

##  2\. Background

We first give some background on the different data tasks considered in this
paper and then provide a brief review of FMs.

###  2.1. Problem Setup

We focus on entity matching (EM), error detection (ED), and data imputation
(DI) and describe the setup for these tasks. We denote $D$, a structured
dataset with $n$ entries, such that each entry is represented by a collection
of $m$ attribute value pairs: for entry $e_{i}\in D$ we have
$e_{i}=\\{e_{i,j}\\}_{1\leq j\leq m}$ where for attribute $j$,
$e_{i,j}=\\{\texttt{attr}_{j},\texttt{val}_{j}\\}$.

Entity Matching  The goal of EM is to match entities (real-world objects like
people, places and things) across different datasets. Formally, given two
structured datasets $(D,D^{\prime})$ and pairs of entries $e,e^{\prime}\in
D\times D^{\prime}$, the goal is to predict whether these entries represent
the same entity or not. This problem is usually solved as a classification
problem, and real-world EM systems are often preceded by blocking heuristics
which are used to remove obvious non-matches.

EM has been extensively studied over the past decade (see (Papadakis et al.,
2020) for a survey) and methods broadly fall into three categories: rule-
based, crowd-based (Gokhale et al., 2014; Wang et al., 2012) and ML/DL-based
(Konda et al., 2016; Mudgal et al., 2018). Recently, methods relying on PLMs
(Li et al., 2020) have become SoTA for this task.

Error Detection  ED is an important step in data cleaning pipelines. Given an
entry $e$, the goal is to detect attributes $j$ where $\texttt{val}_{j}$ has
an error. The task is framed as a classification problem where the goal is to
predict if $\texttt{val}_{j}$ is correct for a given $e$.

ED has been studied extensively in both academic and industrial settings
(Abedjan et al., 2016). There are a number of successful commercial offerings
including Trifacta (Tri, 2022) and Tamr (tam, 2022). Traditionally, ED systems
have been heavily reliant on rule-based algorithms which enforce data
constraints through functional dependencies or knowledge bases (Chu et al.,
2015, 2013; Dallachiesa et al., 2013). Additionally, there are statistical-
based approaches such as pattern enforcement (Kandel et al., 2011; Chu et al.,
2015), outlier detection (Dasu and Loh, 2012), and record deduplication
(Stonebraker et al., 2013) algorithms. Recent efforts have developed SoTA ML
models for ED (Heidari et al., 2019).

Data Imputation  DI is a critical step for repairing dirty data sources. Given
an entry $e$ with missing attribute values
$\\{\texttt{attr}_{j},\texttt{NULL}\\}$, the goal of DI is to infer the
missing values. The full range of plausible values for the missing value is
not known apriori.

Prior works in DI falls into three categories—clustering and/or statistical-
based (Mayfield et al., 2010), generative model-based (Ghysels et al., 2007),
ML/DL-based (Biessmann et al., 2019b; Mei et al., 2021), and tabular data
pretraining-based approaches (Deng et al., 2022)—and struggle when needing to
impute values not seen in the training set (Mei et al., 2021).

###  2.2. Background on Foundation Models

We now give an overview of language FMs, starting from very early, smaller FMs
(i.e. PLMs) and moving to large-scale FMs, the latter of which is the focus of
this paper.

Pretrained Language Models  Pretrained Language Models (PLMs) are neural
networks pretrained on large corpora of publicly available text (e.g. web
pages). The first breed of PLMs—ELMo (Peters et al., 2018), BERT (Devlin et
al., 2019), RoBERTA (Liu et al., 2019)—learned the semantics of natural
language by predicting masked works during pretraining. These models have on
the order of hundreds of millions of parameters. Traditionally, PLMs are
adapted to downstream tasks through a task-specific prediction layer (e.g.
classification layer) and a task-specific finetuning step wherein all model
weights are updated.

Large Autoregressive Language Models  In 2020, GPT-3 (Brown et al., 2020)
marked a significant shift in the the ML community. It represented a new class
of large-scale language models: autoregressive language models pretrained to
predict the next word in a sequence. These models have _billions_ of
parameters and have been used for language generation tasks such as question
answering and summarization. Since the release of GPT-3, bigger and better
performing autoregressive language models have been developed (Chowdhery et
al., 2022).

Emergent Behaviors  Interestingly, the biggest GPT-3 variant (175B parameters)
has the capacity to solve natural language tasks with only a few examples
(few-shot prompting), and, in some cases, just a task description (e.g.
“Translate French to English”). Unlike traditional finetuning, no model
parameters are updated to fit the task. Few-shot prompting has proven to be
effective on tasks widely different from the FMs pretraining objective (e.g.,
code generation (Xu et al., 2022), Trivia QA (Lin et al., 2021; Brown et al.,
2020) and arithmetic (Brown et al., 2020)). This in-context learning behavior
can be thought of as the FM “locating” an already-learned behavior (Reynolds
and McDonell, 2021). Smaller models (¡10B parameters) typically require some
form of task-specific finetuning. We study ways to use smaller FMs on data
tasks in the full report (Narayan et al., 2022).

##  3\. Foundation Models for Data Tasks

Our goal is to understand whether FMs can benefit data cleaning and
integration tasks. The procedure for applying FMs to data tasks requires
adapting structured data inputs to textual inputs (Section 3.1), casting data
tasks as text generation tasks (Section 3.2) and, for few-shot prompting,
constructing demonstrative task examples to help the FM learn new data tasks
(Section 3.3).

###  3.1. Tabular Data Serialization

FMs take text as input and generate text as output. In order to apply these
models to data tasks, we first need to convert structured tabular data inputs
to text representations.

Given a structured dataset, we first convert an entry to text. Concretely, for
entry e, we follow previous work (Li et al., 2020) and serialize the attribute
value and entry as follows:

$\displaystyle\texttt{serialize}(e)\coloneqq\texttt{attr}_{1}:\texttt{val}_{1}\ldots\texttt{attr}_{m}:\texttt{val}_{m}$

If the attribute value is NULL, it is serialized as the empty string. Based on
the task and dataset, serialization only happens over a subset of attributes
relevant to the task. FMs can be sensitive to the prompt format and the
specific serialization used (Zhao et al., 2021). In Section 4.3 we study the
sensitivity of FMs to sub-selection choices.

###  3.2. Data Tasks as Natural Language Tasks

Next, we need to convert data tasks to text generation tasks. We construct
natural language descriptions of each task (i.e. prompts) that use the
serialized representations from Section 3.1. The prompts are passed to the FM,
whose generated output is the answer to the given task. For entity matching
and error detection, the model generates a “Yes” or “No”
response,111Interestingly, the model is not constrained to produce a Yes/No
answer on the output side, but we find that this happens most of the time. In
the rare examples where the model does not predict a Yes/No answer, we use No
as the default answer. and for imputation it generates the missing value. We
now enumerate the prompts for each task.

Entity matching: given two entries $(e,e^{\prime})$ the template is

$\displaystyle\texttt{Product A is }\texttt{serialize}(e)\texttt{. Product B
is }\texttt{serialize}(e^{\prime}).$ Are Product A and Product B the same?

Data imputation: given an entry $e$ and attribute $j$ to infer, we use

$\displaystyle\texttt{attr}_{1}:\texttt{val}_{1}\ldots\texttt{attr}_{j}\texttt{
?}$

Error detection: given an entry $e$ and attribute $j$ to classify as
erroneous, we use

$\displaystyle\texttt{Is there an error in
}\texttt{attr}_{j}:\texttt{val}_{j}\texttt{?}$

These templates highlight the generality of this framework which could be
extended beyond the tasks considered in this paper.

###  3.3. Task Demonstrations

Task demonstrations can be included in the prompt to help the model learn a
new task (see Figure 2 for an error detection example). These demonstrations
are used to show the model how the task should be completed (e.g. should it
generate Yes/No or a missing value) as well as understand the finer-grained
semantics of $D$. We explore two approaches for selecting task demonstration
examples.

Random  One approach is to sample random examples from a labeled dataset.
However, this approach often causes high variance in FM performance (Liu et
al., 2021a). Moreover, the ordering of examples in the prompt can have a non-
trivial impact on the downstream performance (Lu et al., 2021; Zhao et al.,
2021; Liu et al., 2021a). Although recent work has tried to systematize the
prompt tuning process, it is still an open area of research (Li and Liang,
2021a; Sun and Lai, 2020).

Manual  Another approach is to manually construct examples that yield good
performance on a held-out validation set that is 10% of the original labeled
dataset. This approach is more costly (requires more time) in comparison to
random sampling but improves performance when examples are carefully
constructed.

In our manual prompt tuning experiments, we spend at most 1 hour per task
analyzing errors on the validation set. We then manually construct
demonstrations that help the model correct these errors. We liken this step to
the canonical hyperparameter tuning process in ML where time and compute are
spent finding the best model parameters for the given task. However, the
engineering effort for prompt tuning is significantly lower. Concretely, it
takes less time (minutes vs. hours and days), is more compute effective
(inference vs. full training), and gives the user finer-grained control
(natural language guidance vs. blackbox parameter updates).

Table 1. Entity matching results measured by F1 score where k is the number of
task demonstrations.

Dataset | Magellan | Ditto | GPT3-175B | GPT3-175B
---|---|---|---|---
($k$=0) | ($k$=10)
Fodors-Zagats | 100 | 100 | 87.2 | 100
Beer | 78.8 | 94.37 | 78.6 | 100
iTunes-Amazon | 91.2 | 97.06 | 65.9 | 98.2
Walmart-Amazon | 71.9 | 86.76 | 60.6 | 87.0
DBLP-ACM | 98.4 | 98.99 | 93.5 | 96.6
DBLP-Google | 92.3 | 95.60 | 64.6 | 83.8
Amazon-Google | 49.1 | 75.58 | 54.3 | 63.5

##  4\. Experiments

We compare FMs to SoTA methods on a variety of data cleaning and integration
tasks. Our goal is to understand whether large FMs can transfer to data tasks
in zero- and few-shot settings (Section 4.2), and the nuances in applying FMs
to data tasks (Section 4.3).

###  4.1. Experimental Setup

We begin by describing our experimental protocol, including models, datasets,
metrics and baselines used.

Models  For few-shot prompting we use the GPT-3-175B parameter model (text-
davinci-002) in the OpenAI API endpoint (OpenAI, 2021).

Datasets  For entity matching, we use the standard Magellan benchmark (Konda
et al., 2016). For schema matching, we choose a challenging dataset, Synthea,
from the OMAP benchmark (Zhang et al., 2021). For data transformation, we
choose two challenging datasets from the TDE benchmark (He et al., 2018). For
imputation, we choose two challenging datasets from (Mei et al., 2021):
Restaurants and Buy. Finally, for error detection, we evaluate on the
benchmark Hospital and Adult datasets which are used across several data
cleaning papers (Heidari et al., 2019; Rekatsinas et al., 2017; Chu et al.,
2013).

We use the provided train/test/dev splits for all entity matching datasets.
For cleaning tasks, the splits are not available but we follow the protocol of
(Heidari et al., 2019; Mei et al., 2021) to generate the dataset splits. For
the Adult dataset, we evaluate over a randomly sampled set of 1K rows due to
budget constraints.

Evaluation Metrics  For the error detection, and schema/entity matching, we
evaluate performance using F1 score. For imputation and data transformation,
we evaluate using accuracy.

Baselines  We compare against the SoTA methods for each task. For entity
matching, we benchmark against Ditto (Li et al., 2020), the current SoTA DL-
based approach which finetunes BERT (Devlin et al., 2019). For data
imputation, we benchmark against IMP (Mei et al., 2021), which finetunes
RoBERTa (Liu et al., 2019), and HoloClean (Rekatsinas et al., 2017), a
statistical-based SoTA data repair engine. For schema matching, we compare
against the SoTA model, SMAT (Zhang et al., 2021), which finetunes an
attention-based BiLSTM. For data transformations, we compare against TDE (He
et al., 2018), a SoTA search-based solution. Finally, for error detection, we
compare against HoloClean and HoloDetect (Heidari et al., 2019), a data-
augmentation based ML approach.

Table 2.  Data cleaning results, measured in accuracy for data imputation and
F1 score for error detection where k is the number of task demonstrations.

Task | Imputation | Error Detection
---|---|---
Dataset | Restaurant | Buy | Hospital | Adult
HoloClean | 33.1 | 16.2 | 51.4 | 54.5
IMP | 77.2 | 96.5 | - | -
HoloDetect | - | - | 94.4 | 99.1
GPT3-175B ($k$=0) | 70.9 | 84.6 | 6.9 | 0.0
GPT3-6.7B ($k$=10) | 80.2 | 86.2 | 2.1 | 99.1
GPT3-175B ($k$=10) | 88.4 | 98.5 | 97.8 | 99.1

Table 3.  Data integration results, measured in accuracy for data
transformations and F1 score for schema matching. Previous SoTA method is TDE
for data transformation and SMAT for schema matching.

Task | Data Transformation | Schema Matching
---|---|---
Dataset | StackOverflow | Bing-QueryLogs | Synthea
Previous SoTA | 63.0 | 32.0 | 38.5
GPT3-175B ($k$=0) | 32.7 | 24.0 | 0.5
GPT3-175B ($k$=3) | 65.3 | 54.0 | 45.2

###  4.2. Zero/Few-shot Performance of Large FMs

In this section we explore the zero and few-shot performance of GPT-3-175B.
Our goal is to understand whether large FMs transfer to data tasks in zero-
and few-shot settings.

####  4.2.1. Results

We first review the zero- and few-shot results.

Few-shot Performance  Our results show that in the few-shot setting, with
manually curated task demonstrations, GPT-3-175B achieves SoTA performance on
4 entity matching, 2 imputation, 2 data transformation, 2 error detection and
1 schema matching benchmark dataset(s) (see Table 1, Table 2, Table 3). The FM
outperforms fully-finetuned, SoTA PLM-based approaches for entity matching (Li
et al., 2020), and data imputation (Mei et al., 2021). For error detection,
the few-shot approach outperforms the ML-based SoTA method, HoloDetect, which
uses data augmentation and weak supervision to perform well.

Zero-shot Performance  In the zero-shot setting, we observe that the FM
outperforms statistical-based approaches and standard data repair engines
(Rekatsinas et al., 2017) for imputation. On entity matching, the zero-shot
performance is significantly lower than the few shot performance. This
performance gap suggests that demonstrations are very important for the task,
and we study the impact of task demonstrations in more detail in Section 4.3.

####  4.2.2. Discussion

These results show that large FMs can transfer to data tasks. These results
are particularly exciting given that FMs are trained to model English language
and have no prior exposure to the semantics of data tasks nor the syntax of
tabular data. Furthermore, the zero-shot performance on imputation suggests
that large FMs not only have an understanding of how to complete the tasks,
but also have encoded knowledge that is needed to correct and complete records
(e.g. functional dependencies between address and zip code). We analyze
encoded large FM knowledge in the full report (Narayan et al., 2022).

On the entity matching datasets that the FM does not achieve SoTA on, we find
that the FM struggles on data domains that contain jargon not commonly found
in text. In such cases, the model lacks a strong semantic understanding of the
input and has difficulty reasoning over the data. For example, in the Amazon-
Google dataset, the model has difficulty matching samples due to the high
volume of product-specific identifiers in the descriptions. Here, for
instance, the model fails to accurately match the two entries: “name:
pcanywhere 11.0 host only cd-rom xp 98 nt w2k me. manufacturer: symantec.
price: NULL” and “name: symantec pcanywhere 11.0 windows. manufacturer: NULL.
price: 19.99.”. We discuss ways to adapt FMs to domain-specific data in more
detail in Section 5.

Table 4. Entity matching ablation results (F1 score) for different prompt
formats ($k$=10). For all datasets, we evaluate on up to 200 samples for cost
purposes.

Prompt Format | Beer | iTunes- | Walmart-
---|---|---|---
Amazon | Amazon
Prompt 1 (w. Attr. & Example Select.) |  100 $\pm$ 0.00 |  98.2 $\pm$ 0.00 | 88.9 $\pm$ 0.00
Prompt 1 (w/o Example Select.) | 91.1 $\pm$ 0.05 | 86.6 $\pm$ 0.02 | 65.2 $\pm$ 0.04
Prompt 1 (w/o Attr. Select.) | 76.9 $\pm$ 0.00 | 94.1 $\pm$ 0.00 | 75.0 $\pm$ 0.00
Prompt 1 (w. Attr. & w/o Attr. names) | 80.0 $\pm$ 0.00 | 94.5$\pm$ 0.00 | 84.2 $\pm$ 0.00
Prompt 2 (w. Attr. & Example Select.) | 96.3 $\pm$ 0.00 | 84.7 $\pm$ 0.00 |  100 $\pm$ 0.00
Prompt 1: “Are Product A and Product B the same?”
Prompt 2: “Are Product A and Product B equivalent?”

###  4.3. Prompt Tuning Ablations

In this section, we analyze the performance impact of the three different
choices made during prompt tuning: attribute selection, prompt formatting, and
task demonstration curation.

####  4.3.1. Results

We run our ablations on three entity matching datasets (see Table 4). For all
datasets, we evaluate on up to 200 samples for cost purposes. We discuss our
findings next.

Attribute Selection  First, we find through experimentation that sub-selecting
attributes during row serialization can have a non-trivial impact on entity
matching performance. In particular, we observe that sub-selecting attributes
that are essential in determining whether two entities match (e.g. name) and
removing noisy attributes improves model accuracy. To better illustrate this
point, we evaluate FM performance on three datasets when not sub-selecting
attributes. We find that, on average, attribute selection results in a 13.7 F1
point performance improvement (Table 4).

Prompt Formatting  Second, we observe that FMs can be brittle to subtle
variations in prompt templates. We investigate this brittleness by replacing
the span “Are Product A and Product B the same?” (Prompt 1) with “Are Product
A and Product B equivalent?” (Prompt 2). This minor modification results in an
average 9.4 F1 point performance gap in the datasets in Table 4.

Task Demonstrations  Finally, we find that the choice of task demonstrations
has a significant impact on downstream performance. We conduct an ablation
where we replace manually curated task demonstrations with randomly selected
demonstrations (see Prompt 1 (w/o Example Select.) in Table 4). We run this
experiment over three different random seeds and report the results in Table
4. Across all cases, manually curated examples outperform randomly selected
examples by an average of 14.7 F1 points.

####  4.3.2. Discussion

The aforementioned results demonstrate that successful prompt tuning requires
(1) selecting an informative set of attributes required for the task, (2)
crafting a well-formatted prompt that the FM understands, and (3) constructing
a set of instructive task demonstrations that condition the model to the data
at hand. For (1), we find that attribute sub-selection boosts performance by
removing noisy attributes that hurt performance. For (2), our ablations show
that prompt formatting (e.g. word choice, punctuation) can have significant
impact on model performance. For (3), our results indicate that examples need
to be carefully crafted for FMs to learn new tasks. We conjecture that on more
reasoning-intensive tasks (e.g. matching), prompts are important as they help
teach the model how to reason about entities and how to complete the task.
Moreover, we emphasize that all three steps require some form of iteration to
develop the most effective prompt (e.g. passing various inputs to the model
and inspecting its outputs).

These findings are aligned with existing literature on prompt tuning that
observe non-trivial variance in prompt-based learning settings (Zhao et al.,
2021). This performance variance suggests that iterative prompt programming is
an essential human-in-the-loop process for FM usage (Liu et al., 2021b).
However, some works suggest that smarter, automatic example selection methods
can help close the gap between random example selection and human-in-the-loop
prompt selection (Liu et al., 2021a). These results highlight the paradigm
shift induced by building systems centered around FMs: instead of spending
time tuning models, we now need to invest time finding the right examples and
engineering useful prompts for each task. We discuss these paradigm shifts in
more detail in Section 5.

##  5\. Research Agenda

Because of their natural language interface and vast internal knowledge, FMs
provide an interface for unifying a wide-range of siloed, hand-engineered data
integration pipelines. Consequently, we envision that the data orchestration
workbenches of the future will be centered around FMs. We discuss the
opportunities, practical considerations, and technical challenges associated
of this vision next.

###  5.1. Opportunities for FMs

Natural Language Interactions  FMs usher in a new era of human-machine
collaborative workflows wherein users spend less time labeling data and
finetuning models and more time writing natural language prompts that are
representative of the task at hand. As the necessity of writing code
decreases, we envision systems that are more accessible to non-machine
learning experts (e.g., business users). In future work, we seek to better
understand the human-in-the-loop prompt engineering pipeline, especially in
the context of data management practices.

Model Prototyping  The data integration and management pipeline can be
categorized into three distinct stages: discovery and design, development, and
deployment (IBM, [n. d.]). We propose that FMs will be most useful in the
discovery and design phase when training data is less available. In this
setting, FMs enable rapid prototyping of data models via prompting. In some
cases, the FM’s out-of-the-box performance will be sufficiently high,
obviating the need to train a task-specific model. In others, we can use the
FM to label and generate data with human-in-the-loop feedback. When a
sufficient amount of data has been collected, transitioning to the fully-
supervised model development regime is the optimal choice.

Passive Learning From User Exhaust  In the enterprise setting, organizations
accumulate a staggering amount of data exhaust—the informational byproduct
that streams from devices, products, and workforce management practices (dat,
2018). Because FMs are pretrained in an unsupervised fashion with a simple
token prediction objective (see Section 2.2), they can learn over any raw and
unlabeled sources of data (Dhariwal et al., 2020; Yan et al., 2021; Reed et
al., 2022). As such, FMs can effectively ingest the exhaust from the entire
data stack (from system logs to structured data). Learning from data analyst
exhaust (e.g., clicks over GUI) is also an opportunity to improve FM
performance for these data tasks.

###  5.2. Practical Considerations of FMs

Integration in Data Management Workflows  FMs take text as input and generate
text as output. As a result, they are limited in their ability to directly
take actions over the graphical user interfaces (GUIs) of data management
software (DMS) (e.g., Snowflake). Given that a majority of data analyst time
is spent working in these environments, we need ways of translating natural
language specifications of data tasks (e.g., “remove all nan cells”), to
actionable operations on these GUIs. Excitingly, new work demonstrates how to
augment FM capabilities such that they can effectively take actions on web and
mobile interfaces (Wang et al., 2022; Adept, [n. d.]). These works suggest the
possibility of directly integrating FMs with DMS.

Interaction with Existing Systems  FMs can interact with existing DMS by
either replacing them or utilizing their outputs to conduct downstream tasks.
In terms of system replacement, for tasks where the required rules and logic
are not encoded in the FMs knowledge, it is an open question (Desmond et al.,
2022) as to how to systematically translate the rules and logic (e.g., domain-
specific dependencies) to natural language inputs to the FMs. For system
integration, we need ways of systematically incorporating the outputs of
existing systems (e.g., dataset pattern discovery) in natural language
prompts. Prior work has demonstrated how to “ground” FM-based text-to-SQL
prompts with database schemas and metadata (Deng et al., [n. d.]). Similar
ideas could be used when incorporating system outputs with FMs.

Debuggability  FMs are non-deterministic and can make unexpected errors. In
order to use these models in data management pipelines, we need mechanisms for
increasing transparency and debuggability of pipelines. One possible approach
is to collect and monitor model confidence scores. Prior work demonstrates
that a FM can “learn to express uncertainty about its own answers in natural
language” (Lin et al., 2022). Another approach is to decompose tasks into
chains of “primitive operations”, enabling for more transparency and
visibility of specific failure points (Wu et al., 2022).

###  5.3. Technical Challenges of FMs

Domain Specificity  A key challenge in applying FMs to data management tasks
is operating over highly specialized domains (e.g., medical, financial, and
insurance data). The existing literature suggests that domain-specialization
is best achieved by continuously pretraining on relevant data (e.g., earning
reports, medical reports) (Yang et al., 2020; Gururangan et al., 2020a).
However, training models with billions of parameters can be costly. This has
motivated several works which attempt to more quickly and cheaply adapt models
to new domains by finetuning only a few layers of the model or simply training
a small neural network (e.g., adaptor) on top of a frozen FM (Labs, 2022).

Privacy  Organizations are often unable to pass sensitive data to third party
APIs for privacy reasons (Arora and Ré, 2022; Arora et al., 2022a).
Unfortunately, this constraint is in conflict with the contemporary state of
FMs where the best performing models (e.g., GPT-3), can only be accessed via
API requests. This motivates the need for better open-source models which are
competitive with closed-source models. Excitingly, recent work (Arora et al.,
2022b) has demonstrated that techniques such as prompt ensembling and prompt
reframing can enable open-source models like GPT-J-6B (Wang and Komatsuzaki,
2021) to out-perform GPT3-175B on popular natural language understanding
benchmarks. Extending this work to data management tasks is an interesting
research direction.

Prompt Engineering and Automation  Prompting is the primary mechanism for
adapting FMs to new tasks. However, prompting requires some manual effort to
design performant prompts. In the DI setting, prompts can be sensitive to
things such as data schemas and minor formatting variations. A few automated
approaches have been developed which reduce the manual effort needed to
construct prompts: soft prompt tuning (e.g., optimizing a sequence of task-
specific vectors which are appended to the text prompt) (Lester et al., 2021a;
Li and Liang, 2021b) and learning to retrieve better in-context examples
(Rubin et al., 2021). Adapting these works to the challenges of prompting for
data tasks is an open area of research.

##  6\. Conclusion

In this work we investigate the applicability of FMs to classical data tasks.
We find that large FMs can achieve SoTA performance on many data tasks with
0-to-few natural language task demonstrations. The ability of these models to
transfer to data tasks with no task-specific finetuning is particularly
interesting given that these models are simply trained to predict next words.
Our work builds upon years of important work on integration and cleaning tasks
in the data management community. We hope that our results gesture towards the
possibilities of using language guided models for human-in-the-loop data
integration practices across a broader range of data management tasks.

## Acknowledgments

We are thankful to Ihab Ilyas, Theo Rekatsinas, Mike Cafarella, Ce Zhang, Sen
Wu, Christopher Aberger, Neel Guha, Beidi Chen and Xiao Ling for their helpful
discussions and feedback. We gratefully acknowledge the support of DARPA under
Nos. FA86501827865 (SDH) and FA86501827882 (ASED); NIH under No. U54EB020405
(Mobilize), NSF under Nos. CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to
Velocity), and 1937301 (RTML); ONR under No. N000141712266 (Unifying Weak
Supervision); the Moore Foundation, NXP, Xilinx, LETI-CEA, Intel, IBM,
Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson,
Qualcomm, Analog Devices, the Okawa Foundation, American Family Insurance,
Google Cloud, Swiss Re, Brown Institute for Media Innovation, Department of
Defense (DoD) through the National Defense Science and Engineering Graduate
Fellowship (NDSEG) Program, Fannie and John Hertz Foundation, National Science
Foundation Graduate Research Fellowship Program, Texas Instruments Stanford
Graduate Fellowship in Science and Engineering, and members of the Stanford
DAWN project: Teradata, Facebook, Google, Ant Financial, NEC, VMWare, and
Infosys. The U.S. Government is authorized to reproduce and distribute
reprints for Governmental purposes notwithstanding any copyright notation
thereon. Any opinions, findings, and conclusions or recommendations expressed
in this material are those of the authors and do not necessarily reflect the
views, policies, or endorsements, either expressed or implied, of DARPA, NIH,
ONR, or the U.S. Government.

## References

  * (1)
  * dat (2018) 2018\.  How to turn data exhaust into a competitive edge.  https://knowledge.wharton.upenn.edu/article/turn-iot-data-exhaust-next-competitive-advantage/
  * uni (2020) 2020\.  Federal Judicial Caseload Statistics 2020.  https://www.uscourts.gov/statistics-reports/federal-judicial-caseload-statistics-2020
  * ccp (2022) 2022\.  California Consumer Privacy Act (CCPA).  https://oag.ca.gov/privacy/ccpa
  * iot (2022) 2022\.  Decreasing cost of storage.  https://www.iotone.com/term/decreasing-cost-of-storage/t172
  * tam (2022) 2022\.  Tamr — Enterprise Data Mastering at Scale - Tamr Inc.  https://www.tamr.com/
  * Tri (2022) 2022\.  Trifacta: Data Wrangling Software and Tools.  https://www.trifacta.com/
  * Abedjan et al. (2016) Ziawasch Abedjan, Xu Chu, Dong Deng, Raul Castro Fernandez, Ihab F Ilyas, Mourad Ouzzani, Paolo Papotti, Michael Stonebraker, and Nan Tang. 2016.  Detecting data errors: Where are we and what needs to be done?  _Proceedings of the VLDB Endowment_ 9, 12 (2016), 993–1004.
  * Abid et al. (2021) Abubakar Abid, Maheen Farooqi, and James Zou. 2021\.  Large language models associate Muslims with violence.  _Nature Machine Intelligence_ 3, 6 (2021), 461–463.
  * Adadi (2021) Amina Adadi. 2021\.  A survey on data-efficient algorithms in big data era.  _Journal of Big Data_ 8, 1 (2021), 1–54.
  * Adept ([n. d.]) Adept. [n. d.].  ACT-1: Transformer for Actions.  https://www.adept.ai/act
  * Adler-Milstein et al. (2020) Julia Adler-Milstein, Jason S Adelman, Ming Tai-Seale, Vimla L Patel, and Chris Dymek. 2020\.  EHR audit logs: a new goldmine for health services research?  _Journal of biomedical informatics_ 101 (2020), 103343.
  * Alayrac et al. (2022) Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, Iain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katie Millican, Malcolm Reynolds, et al. 2022\.  Flamingo: a Visual Language Model for Few-Shot Learning.  _arXiv preprint arXiv:2204.14198_ (2022).
  * Arora et al. (2022a) Simran Arora, Patrick Lewis, Angela Fan, Jacob Kahn, and Christopher Ré. 2022a.  Reasoning over Public and Private Data in Retrieval-Based Systems.  arXiv:2203.11027 [cs.IR]
  * Arora et al. (2022b) Simran Arora, Avanika Narayan, Mayee F Chen, Laurel J Orr, Neel Guha, Kush Bhatia, Ines Chami, Frederic Sala, and Christopher Ré. 2022b.  Ask Me Anything: A simple strategy for prompting language models.  _arXiv preprint arXiv:2210.02441_ (2022).
  * Arora and Ré (2022) Simran Arora and Christopher Ré. 2022.  Can Foundation Models Help Us Achieve Perfect Secrecy?  _arXiv preprint arXiv:2205.13722_ (2022).
  * Biessmann et al. (2019a) Felix Biessmann, Tammo Rukat, Phillipp Schmidt, Prathik Naidu, Sebastian Schelter, Andrey Taptunov, Dustin Lange, and David Salinas. 2019a.  DataWig: Missing Value Imputation for Tables.  _Journal of Machine Learning Research_ 20, 175 (2019), 1–6.  http://jmlr.org/papers/v20/18-753.html
  * Biessmann et al. (2019b) Felix Biessmann, Tammo Rukat, Philipp Schmidt, Prathik Naidu, Sebastian Schelter, Andrey Taptunov, Dustin Lange, and David Salinas. 2019b.  DataWig: Missing Value Imputation for Tables.  _J. Mach. Learn. Res._ 20, 175 (2019), 1–6.
  * Black et al. (2021) Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. 2021.  _GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow_.  https://doi.org/10.5281/zenodo.5297715
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021\.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_ (2021).
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020\.  Language models are few-shot learners.  _Advances in neural information processing systems_ 33 (2020), 1877–1901.
  * Chen et al. (2021) Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021\.  Evaluating large language models trained on code.  _arXiv preprint arXiv:2107.03374_ (2021).
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022\.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_ (2022).
  * Chu et al. (2013) Xu Chu, Ihab F Ilyas, and Paolo Papotti. 2013.  Holistic data cleaning: Putting violations into context. In _2013 IEEE 29th International Conference on Data Engineering (ICDE)_. IEEE, 458–469.
  * Chu et al. (2015) Xu Chu, John Morcos, Ihab F Ilyas, Mourad Ouzzani, Paolo Papotti, Nan Tang, and Yin Ye. 2015.  Katara: A data cleaning system powered by knowledge bases and crowdsourcing. In _Proceedings of the 2015 ACM SIGMOD international conference on management of data_. 1247–1261.
  * Coupons (2021) EasyClosets Coupons. 2021\.  21 best GPT-3 tools, examples and use cases - nogood™: Growth Marketing Agency.  https://nogood.io/2021/06/25/gpt-3-tools/
  * Dallachiesa et al. (2013) Michele Dallachiesa, Amr Ebaid, Ahmed Eldawy, Ahmed Elmagarmid, Ihab F Ilyas, Mourad Ouzzani, and Nan Tang. 2013.  NADEEF: a commodity data cleaning system. In _Proceedings of the 2013 ACM SIGMOD International Conference on Management of Data_. 541–552.
  * Dalvi et al. (2013) Nilesh Dalvi, Vibhor Rastogi, Anirban Dasgupta, Anish Das Sarma, and Tamás Sarlós. 2013\.  Optimal hashing schemes for entity matching. In _Proceedings of the 22nd international conference on world wide web_. 295–306.
  * Dasu and Loh (2012) Tamraparni Dasu and Ji Meng Loh. 2012.  Statistical Distortion: Consequences of Data Cleaning.  _Proceedings of the VLDB Endowment_ 5, 11 (2012).
  * Deng et al. ([n. d.]) Xiang Deng, Ahmed Hassan Awadallah, Christopher Meek, Oleksandr Polozov, Huan Sun, and Matthew Richardson. [n. d.].  Structure-Grounded Pretraining for Text-to-SQL.  ([n. d.]).
  * Deng et al. (2022) Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. 2022.  Turl: Table understanding through representation learning.  _ACM SIGMOD Record_ 51, 1 (2022), 33–40.
  * Desmond et al. (2022) Michael Desmond, Evelyn Duesterwald, Vatche Isahagian, and Vinod Muthusamy. 2022.  A No-Code Low-Code Paradigm for Authoring Business Automations Using Natural Language.  _arXiv preprint arXiv:2207.10648_ (2022).
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_ (2018).
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_. Association for Computational Linguistics, Minneapolis, Minnesota, 4171–4186.  https://doi.org/10.18653/v1/N19-1423
  * Dhariwal et al. (2020) Prafulla Dhariwal, Heewoo Jun, Christine Payne, Jong Wook Kim, Alec Radford, and Ilya Sutskever. 2020\.  Jukebox: A Generative Model for Music.  _CoRR_ abs/2005.00341 (2020).  https://arxiv.org/abs/2005.00341
  * Ghysels et al. (2007) Eric Ghysels, Arthur Sinko, and Rossen Valkanov. 2007\.  MIDAS regressions: Further results and new directions.  _Econometric reviews_ 26, 1 (2007), 53–90.
  * Gokhale et al. (2014) Chaitanya Gokhale, Sanjib Das, AnHai Doan, Jeffrey F Naughton, Narasimhan Rampalli, Jude Shavlik, and Xiaojin Zhu. 2014.  Corleone: Hands-off crowdsourcing for entity matching. In _Proceedings of the 2014 ACM SIGMOD international conference on Management of data_. 601–612.
  * Gururangan et al. (2020a) Suchin Gururangan, Ana Marasović, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A Smith. 2020a.  Don’t stop pretraining: adapt language models to domains and tasks.  _arXiv preprint arXiv:2004.10964_ (2020).
  * Gururangan et al. (2020b) Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020b.  Don’t stop pretraining: Adapt language models to domains and tasks. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_. 8342–8360.
  * Guu et al. (2020) Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020.  Realm: Retrieval-augmented language model pre-training.  _arXiv preprint arXiv:2002.08909_ (2020).
  * He et al. (2018) Yeye He, Xu Chu, Kris Ganjam, Yudian Zheng, Vivek Narasayya, and Surajit Chaudhuri. 2018.  Transform-Data-by-Example (TDE): An Extensible Search Engine for Data Transformations.  _Proceedings of the VLDB Endowment_ 11, 10 (2018).
  * Heidari et al. (2019) Alireza Heidari, Joshua McGrath, Ihab F Ilyas, and Theodoros Rekatsinas. 2019.  Holodetect: Few-shot learning for error detection. In _Proceedings of the 2019 International Conference on Management of Data_. 829–846.
  * Houlsby et al. (2019) Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019.  Parameter-Efficient Transfer Learning for NLP.  https://doi.org/10.48550/ARXIV.1902.00751
  * IBM ([n. d.]) IBM. [n. d.].  Data Integration Phases.  https://www.ibm.com/docs/en/iis/11.7?topic=SSZJPZ_11.7.0%2Fcom.ibm.swg.im.iis.productization.iisinfsv.overview.doc%2Ftopics%2Fcisocapabilities.html
  * Jin et al. (2021) Xisen Jin, Dejiao Zhang, Henghui Zhu, Wei Xiao, Shang-Wen Li, Xiaokai Wei, Andrew Arnold, and Xiang Ren. 2021\.  Lifelong pretraining: Continually adapting language models to emerging corpora.  _arXiv preprint arXiv:2110.08534_ (2021).
  * Kandel et al. (2011) Sean Kandel, Andreas Paepcke, Joseph Hellerstein, and Jeffrey Heer. 2011.  Wrangler: Interactive visual specification of data transformation scripts. In _Proceedings of the sigchi conference on human factors in computing systems_. 3363–3372.
  * Kaplan et al. (2020) Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. 2020\.  Scaling laws for neural language models.  _arXiv preprint arXiv:2001.08361_ (2020).
  * Karpas et al. (2022) Ehud Karpas, Omri Abend, Yonatan Belinkov, Barak Lenz, Opher Lieber, Nir Ratner, Yoav Shoham, Hofit Bata, Yoav Levine, Kevin Leyton-Brown, et al. 2022\.  MRKL Systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning.  _arXiv preprint arXiv:2205.00445_ (2022).
  * Ke et al. (2021) Zixuan Ke, Bing Liu, Nianzu Ma, Hu Xu, and Lei Shu. 2021.  Achieving Forgetting Prevention and Knowledge Transfer in Continual Learning.  _Advances in Neural Information Processing Systems_ 34 (2021).
  * Konda et al. (2016) Pradap Konda, Sanjib Das, AnHai Doan, Adel Ardalan, Jeffrey R Ballard, Han Li, Fatemah Panahi, Haojun Zhang, Jeff Naughton, Shishir Prasad, et al. 2016\.  Magellan: toward building entity matching management systems over data science stacks.  _Proceedings of the VLDB Endowment_ 9, 13 (2016), 1581–1584.
  * Labs (2022) AI21 Labs. 2022\.  STANDING ON THE SHOULDERS OF GIANT FROZEN LANGUAGE MODELS.  _preprint_ (2022).
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020\.  ALBERT: A Lite BERT for Self-supervised Learning of Language Representations. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=H1eA7AEtvS
  * Lester et al. (2021a) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021a.  The Power of Scale for Parameter-Efficient Prompt Tuning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. Association for Computational Linguistics, Online and Punta Cana, Dominican Republic, 3045–3059.  https://doi.org/10.18653/v1/2021.emnlp-main.243
  * Lester et al. (2021b) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021b.  The Power of Scale for Parameter-Efficient Prompt Tuning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_. 3045–3059.
  * Levine et al. (2022) Yoav Levine, Itay Dalmedigos, Ori Ram, Yoel Zeldes, Daniel Jannai, Dor Muhlgay, Yoni Osin, Opher Lieber, Barak Lenz, Shai Shalev-Shwartz, et al. 2022\.  Standing on the shoulders of giant frozen language models.  _arXiv preprint arXiv:2204.10019_ (2022).
  * Li and Liang (2021a) Xiang Lisa Li and Percy Liang. 2021a.  Prefix-Tuning: Optimizing Continuous Prompts for Generation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. 4582–4597.
  * Li and Liang (2021b) Xiang Lisa Li and Percy Liang. 2021b.  Prefix-Tuning: Optimizing Continuous Prompts for Generation. In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_. Association for Computational Linguistics, Online, 4582–4597.  https://doi.org/10.18653/v1/2021.acl-long.353
  * Li et al. (2020) Yuliang Li, Jinfeng Li, Yoshihiko Suhara, AnHai Doan, and Wang-Chiew Tan. 2020.  Deep entity matching with pre-trained language models.  _Proceedings of the VLDB Endowment_ 14, 1 (2020), 50–60.
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022\.  Holistic evaluation of language models.  _arXiv preprint arXiv:2211.09110_ (2022).
  * Liang et al. (2021) Paul Pu Liang, Chiyu Wu, Louis-Philippe Morency, and Ruslan Salakhutdinov. 2021.  Towards understanding and mitigating social biases in language models. In _International Conference on Machine Learning_. PMLR, 6565–6576.
  * Lin et al. (2021) Stephanie Lin, Jacob Hilton, and Owain Evans. 2021\.  TruthfulQA: Measuring How Models Mimic Human Falsehoods.  (2021).
  * Lin et al. (2022) Stephanie Lin, Jacob Hilton, and Owain Evans. 2022\.  Teaching Models to Express Their Uncertainty in Words.  _arXiv preprint arXiv:2205.14334_ (2022).
  * Liu et al. (2022) Alisa Liu, Swabha Swayamdipta, Noah A Smith, and Yejin Choi. 2022.  WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation.  _arXiv preprint arXiv:2201.05955_ (2022).
  * Liu et al. (2021a) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021a.  What Makes Good In-Context Examples for GPT-$3$?  _arXiv preprint arXiv:2101.06804_ (2021).
  * Liu et al. (2021b) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021b.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _arXiv preprint arXiv:2107.13586_ (2021).
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_ (2019).
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.  _arXiv preprint arXiv:1711.05101_ (2017).
  * Lu et al. (2021) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2021.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  _arXiv preprint arXiv:2104.08786_ (2021).
  * Lucy and Bamman (2021) Li Lucy and David Bamman. 2021.  Gender and Representation Bias in GPT-3 Generated Stories.  _NAACL HLT 2021_ (2021), 48\.
  * Mahabadi et al. (2022) Rabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Marzieh Saeidi, Lambert Mathias, Veselin Stoyanov, and Majid Yazdani. 2022\.  PERFECT: Prompt-free and Efficient Language Model Fine-Tuning. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Marr (2021) Bernard Marr. 2021\.  What is unstructured data and why is it so important to businesses?  https://www.forbes.com/sites/bernardmarr/2019/10/16/what-is-unstructured-data-and-why-is-it-so-important-to-businesses-an-easy-explanation-for-anyone/?sh=999b04d15f64
  * Mayfield et al. (2010) Chris Mayfield, Jennifer Neville, and Sunil Prabhakar. 2010\.  ERACER: a database approach for statistical inference and data cleaning. In _Proceedings of the 2010 ACM SIGMOD International Conference on Management of data_. 75–86.
  * Maynez et al. (2020) Joshua Maynez, Shashi Narayan, Bernd Bohnet, and Ryan McDonald. 2020.  On Faithfulness and Factuality in Abstractive Summarization. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL)_. 1906–1919.
  * Mei et al. (2021) Yinan Mei, Shaoxu Song, Chenguang Fang, Haifeng Yang, Jingyun Fang, and Jiang Long. 2021\.  Capturing Semantics for Imputation with Pre-trained Language Models. In _2021 IEEE 37th International Conference on Data Engineering (ICDE)_. IEEE, 61–72.
  * Mitchell et al. (2022) Eric Mitchell, Charles Lin, Antoine Bosselut, Chelsea Finn, and Christopher D Manning. 2022\.  Fast Model Editing at Scale. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=0DcZxeWfOPt
  * Mudgal et al. (2018) Sidharth Mudgal, Han Li, Theodoros Rekatsinas, AnHai Doan, Youngchoon Park, Ganesh Krishnan, Rohit Deep, Esteban Arcaute, and Vijay Raghavendra. 2018.  Deep learning for entity matching: A design space exploration. In _Proceedings of the 2018 International Conference on Management of Data_. 19–34.
  * Narayan et al. (2022) Avanika Narayan, Ines Chami, Laurel Orr, and Christopher Ré. 2022.  Can Foundation Models Wrangle Your Data?  _arXiv preprint arXiv:2205.09911_ (2022).
  * OpenAI (2021) OpenAI. 2021.  OpenAI API.  https://openai.com/api/
  * Papadakis et al. (2020) George Papadakis, Dimitrios Skoutas, Emmanouil Thanos, and Themis Palpanas. 2020.  Blocking and filtering techniques for entity resolution: A survey.  _ACM Computing Surveys (CSUR)_ 53, 2 (2020), 1–42.
  * Peters et al. (2018) Matthew E. Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018.  Deep Contextualized Word Representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_. Association for Computational Linguistics, New Orleans, Louisiana, 2227–2237.  https://doi.org/10.18653/v1/N18-1202
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019\.  Language Models as Knowledge Bases?. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_. 2463–2473.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _Journal of Machine Learning Research_ 21 (2020), 1–67.
  * Rasley et al. (2020) Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 3505–3506.
  * Razniewski et al. (2021) Simon Razniewski, Andrew Yates, Nora Kassner, and Gerhard Weikum. 2021.  Language Models As or For Knowledge Bases.  _arXiv preprint arXiv:2110.04888_ (2021).
  * Reed et al. (2022) Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gomez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Gimenez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, et al. 2022\.  A Generalist Agent.  _arXiv preprint arXiv:2205.06175_ (2022).
  * Rekatsinas et al. (2017) Theodoros Rekatsinas, Xu Chu, Ihab F Ilyas, and Christopher Ré. 2017.  HoloClean: Holistic Data Repairs with Probabilistic Inference.  _Proceedings of the VLDB Endowment_ 10, 11 (2017).
  * Reynolds and McDonell (2021) Laria Reynolds and Kyle McDonell. 2021.  Prompt programming for large language models: Beyond the few-shot paradigm. In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_. 1–7.
  * Rubin et al. (2021) Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021\.  Learning to retrieve prompts for in-context learning.  _arXiv preprint arXiv:2112.08633_ (2021).
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen H Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2022\.  Multitask Prompted Training Enables Zero-Shot Task Generalization. In _International Conference on Learning Representations_.  https://openreview.net/forum?id=9Vrb9D0WI4
  * Schick and Schütze (2021) Timo Schick and Hinrich Schütze. 2021.  It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_. 2339–2352.
  * Stonebraker et al. (2013) Michael Stonebraker, Daniel Bruckner, Ihab F Ilyas, George Beskales, Mitch Cherniack, Stanley B Zdonik, Alexander Pagan, and Shan Xu. 2013.  Data Curation at Scale: The Data Tamer System.. In _Cidr_ , Vol. 2013. Citeseer.
  * Sun and Lai (2020) Fan-Keng Sun and Cheng-I Lai. 2020.  Conditioned natural language generation using only unconditioned language model: An exploration.  _arXiv preprint arXiv:2011.07347_ (2020).
  * Sun et al. (2020) Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2020.  Ernie 2.0: A continual pre-training framework for language understanding. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 34. 8968–8975.
  * Sutanta et al. (2016) Edhy Sutanta, Retantyo Wardoyo, Khabib Mustofa, and Edi Winarko. 2016.  Survey: Models and Prototypes of Schema Matching.  _International Journal of Electrical & Computer Engineering (2088-8708)_ 6, 3 (2016).
  * Voigt and dem Bussche (2017) P. Voigt and A. Von dem Bussche. 2017.  The EU General Data Protection Regulation (GDPR). In _Springer International Publishing_.  https://doi.org/10.1007/978-3-319-57959-7
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  GPT-J-6B: A 6 Billion Parameter Autoregressive Language Model.  https://github.com/kingoflolz/mesh-transformer-jax.
  * Wang et al. (2022) Bryan Wang, Gang Li, and Yang Li. 2022.  Enabling Conversational Interaction with Mobile UI using Large Language Models.  _arXiv preprint arXiv:2209.08655_ (2022).
  * Wang et al. (2012) Jiannan Wang, Tim Kraska, Michael J Franklin, and Jianhua Feng. 2012\.  CrowdER: Crowdsourcing Entity Resolution.  _Proceedings of the VLDB Endowment_ 5, 11 (2012).
  * Webson and Pavlick (2021) Albert Webson and Ellie Pavlick. 2021.  Do Prompt-Based Models Really Understand the Meaning of their Prompts?  _arXiv preprint arXiv:2109.01247_ (2021).
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. 2022.  Chain of thought prompting elicits reasoning in large language models.  _arXiv preprint arXiv:2201.11903_ (2022).
  * Wu et al. (2022) Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022\.  Ai chains: Transparent and controllable human-ai interaction by chaining large language model prompts. In _CHI Conference on Human Factors in Computing Systems_. 1–22.
  * Xu et al. (2022) Frank F Xu, Uri Alon, Graham Neubig, and Vincent Josua Hellendoorn. 2022.  A Systematic Evaluation of Large Language Models of Code. In _Deep Learning for Code Workshop_.
  * Yan et al. (2021) Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. 2021\.  Videogpt: Video generation using vq-vae and transformers.  _arXiv preprint arXiv:2104.10157_ (2021).
  * Yang et al. (2020) Yi Yang, Mark Christopher Siy Uy, and Allen Huang. 2020\.  Finbert: A pretrained language model for financial communications.  _arXiv preprint arXiv:2006.08097_ (2020).
  * Zaidi and Menon (2022) Ehtisham Zaidi and Sharat Menon. 2022.  Magic Quadrant for Data Integration Tools.
  * Zhang et al. (2021) Jing Zhang, Bonggun Shin, Jinho D Choi, and Joyce C Ho. 2021\.  SMAT: An attention-based deep learning solution to the automation of schema matching.  _Advances in databases and information systems. ADBIS_ 12843 (2021), 260\.
  * Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate before use: Improving few-shot performance of language models. In _International Conference on Machine Learning_. PMLR, 12697–12706.

THIS IS DUMMY TEXT

Figure 3.  Adapter architecture

Appendix Overview  We structure our appendix into three sections. In Appendix
A, we demonstrate how to bridge the performance gap between small GPT-3
variants (e.g. 1.3B and 6.7B parameters) and the largest GPT-3 model (175B
parameters) through an in-depth discussion of full and lightweight finetuning
experiments on data tasks. In Appendix B, we explore the knowledge encoded in
FMs and the effect of different model update strategies on this knowledge.
Finally, in Appendix C, we discuss the role of FM bias on data tasks.

##  Appendix A Small FM Finetuning Experiments

In this section, we explore ways to bridge the performance gap between the
smaller and larger FMs, focusing on (1) full finetuning (updating all
parameters) and (2) lightweight finetuning (updating a small number of
parameters). In Section A.1, we evaluate the two approaches and unpack their
sample (as measured by number of labeled samples) and training (as measured by
number of parameter updates) efficiency tradeoffs. We find that both
lightweight and full finetuning can be used to reduce the performance gap
between a 6.7B and a 175B parameter model to an average of 3.1 points. The
lightweight finetuning requires more data than the fully finetuned approach,
but updates only 5% of the model parameters.

Full/Lightweight Finetuning  The typical practice for achieving optimal task
performance in LLMs is to finetune pretrained models with task-specific data.
This approach is usually training inefficient as all weights in the model are
updated. Interestingly, FMs have been shown to achieve optimal task
performance by simply training a lightweight, non-linear layer (i.e. adaptor)
(Figure 3) on the outputs of a frozen model (Labs, 2022; Lester et al.,
2021a). This is a training efficient method that has proven to be competitive
with fully-finetuned approaches (Labs, 2022). However, because this approach
trains a layer from scratch, it is usually less sample efficient than the
fully finetuned approach. Figure 4 visualizes the differences in sample and
compute efficiency between the two approaches.

For lightweight finetuning, we recursively join a single frozen FM with a
small trainable network (the adapter) (Labs, 2022) (Figure 3). The prompt
input is first passed to the frozen FM whose output embeddings are transformed
by the adapter and then passed again as input embeddings to the frozen FM
(Labs, 2022). The generated output of the second pass yields the final answer
of the model. In this setup, updates are only made to the adapter, i.e. no
updates are made to the FM weights alleviating the need for costly training.

For both the full and lightweight finetuning, the model is fed the natural
language prompt in Section 3.2 as input, and trained to generate the task
output (e.g., a Yes/No string, or missing value).

###  A.1. Finetuned Performance of Small FMs

In this section we explore the fully finetuned and lightweight finetuned
performance of two small FMs (namely GPT-3-6.7B and GPT-3-1.3B). Our goal is
to understand whether finetuning can bridge the performance gap with larger
models, and analyze the tradeoffs between sample efficiency and training
efficiency.

###  A.2. Experimental Setup

For adapters, we use the GPT-Neo 1.3B (Black et al., 2021) and the 6.7B
parameter Neo GPT-J (Wang and Komatsuzaki, 2021) models available on
HuggingFace. All finetuning experiments are run on 4-8 A100 GPU machines using
Deepspeed (Rasley et al., 2020) with ZeRO-2 optimizations. We use the AdamW
optimizer (Loshchilov and Hutter, 2017) with a $10\%$ learning rate warmup
followed by a linear decay. We train for a maximum or 30 epochs with a
learning rate of 1e-4 for adapters and 2e-5 for full finetuning, and save the
best model based on validation metrics.

Figure 4. Sample-training efficiency tradeoffs. Larger FMs can be directly
used in a zero/few-shot fashion. To achieve similar performance, smaller FMs
need additional finetuning.

###  A.3. Experimental Results

We first review the finetuning results on Walmart-Amazon (EM), Hospital (ED)
and Restaurant (DI).

Full Finetuning  Our results show that in the full finetuning setting, we can
effectively reduce the performance gap between GPT-3-6.7B and GPT-3-175B on
all datasets (Figure 5), using as little as 10% of the training set for
Walmart-Amazon. GPT-3-1.3B also matches GPT-3-175B performance on Walmart-
Amazon and Restaurant, and is within 8 points of the GPT-3-175B on Hospital.
Compared to the 6.7B model, the 1.3B model is less sample-efficient and needs
more example to bridge the performance gap.

Figure 5. Finetuning experiments: Smaller FMs can be finetuned to bridge the
performance gap with larger FMs. Full-finetuing bridges this gap with less
data than adapters, but adapters are significantly less expensive to train.

Lightweight Finetuning  In the lightweight adapter setting, we find that we
can effectively bridge the performance gap between GPT-3-6.7B and GPT-3-175B
on both Restaurant and Walmart-Amazon, but not Hospital. For GPT-3-1.3B, we
are less effective in bridging the gap, and see an average 25 point
performance difference between the GPT-3-175B across the three datasets.
Furthermore, we find that the adapter model outperforms the fully-finetuned
approach on two datasets with up to 4 points.

####  A.3.1. Discussion

These results show the feasibility of reducing the performance gap between
GPT-3-175B and the smaller models. We observe some clear sample-training
efficiency tradeoffs between the fully and lightly finetuned approaches, which
we discuss next.

Sample Efficiency  We find that fully finetuned models are usually more sample
efficient than adapters, which could be explained by the need to train a layer
from scratch in adapters. We also observe that sample efficiency improves with
size as the performance of the GPT-3-6.7B trained with 10% of the data is
equal to, or better than, the performance of the GPT-3-1.3B trained with 50%
of the data in both the full and lightweight finetuning settings.

Finally, we comment on the performance difference between 6.7B adapter model
and the 6.7B finetuned model on Hospital. We postulate that adapter model
performed poorly on this dataset because the training set was particularly
small (100 samples) which suggests that the adapter set-up requires more
samples to learn generalizable patterns.

Training Efficiency  We find that the adapter approach for GPT-3-6.7B
outperforms the full finetuning approach on 2 of 3 datasets with 5% of the
amount of trainable parameters, demonstrating that training efficiency need
not be sacrificed for quality at this scale. However, at the smaller scale
(GPT-3-1.3B), there is a non-trivial tradeoff between performance and training
efficiency suggesting that training efficiency decreases as model size
decreases.

##  Appendix B Knowledge Ablations

In this section we seek to explore the encoded FM knowledge that is useful for
data tasks. We begin with a qualitative analysis (Section B.1) and follow with
a slice-based analysis that unpacks the impact of different training
mechanisms on FM knowledge (Section B.2).

###  B.1. Qualitative Examples

To better illustrate the notion of encoded knowledge that large FMs possess,
we conduct a qualitative analysis where we inspect the model predictions for
the task of inferring missing zipcodes or cities given some context (Table 6)
. GPT-3-175B is able to effectively apply its understanding of functional
dependencies between address and zip code and dependencies between address,
phone number and city to correctly impute the desired value. We further notice
that while the smaller models fail to impute the correct values, their
generated outputs have the correct semantic type of the missing attribute
without out any demonstrative examples.

GPT-3 Model (% data) | freq $=$ 0 | 0 $<$ freq $\leq$ 10 | freq $>$ 10
---|---|---|---
175B (few-shot) | 100 | 0.0 | 93.7
6.7B (adaptor, 100%) | 0.0 | 50.0 | 98.7
6.7B (adaptor, 50%) | 0.0 | 25.0 | 98.7
6.7B (adaptor, 10%) | 0.0 | 0.0 | 87.3
6.7B (finetune, 100%) | 0.0 | 25.0 | 96.2
6.7B (finetune, 50%) | 0.0 | 0.0 | 98.7
6.7B (finetune, 10%) | 0.0 | 0.0 | 89.9

Table 5. Impoverished city entity slice analysis: with lightweight finetuning,
smaller model variants perform better than GPT-3-175B on infrequently
occurring entities.

Input Prompt | GPT3-175B | GPT3-6.7B | GPT3-1.3B
---|---|---|---
“Address: 1720 university blvd State: AL ZipCode?” | $32533$ | 35205 | 35901
“Address: 26025 pacific coast hwy Phone number: 310/456-5733 City?” | Malibu | Torrance | Pacific Beach
‘”Address: 804 north point st Phone number: 415-775-7036 City?”” | San Francisco | San Francisco | San Francisco

Table 6. FMs have an inherent understanding of conditional functional
dependencies that are traditionally hard-coded in error detection systems.
GPT3-175B is able to accurately map addresses to zip codes and cities.

###  B.2. Slice analysis

We tease apart the differences in encoded knowledge across the different
training regimes through a slice-based performance analysis on the Restaurant
dataset. We use the frequency counts of city names in the training set to
define three frequency-based subclasses (Table 5). We find that for entities
that do not occur in the training set, GPT-3-175B is the only model that is
able to correctly impute the missing values. Moreover, we observe that for
rare entities—a city value of West LA—these pattern can only be learned
through full or lightweight finetuning. Interestingly, we find that the
lightweight approach more accurately learns the rare subclasses relative to
the fully-finetuned approach in both the 100% and 50% training set regime. We
hypothesize that this is because the adapter model is smaller, and is thus
less prone to overfitting the training set.

##  Appendix C Effects of FM Bias

Because FMs are pretrained on large corpuses of web text, they inherit the
biases of the data they are trained on. As a result, FMs have been found to
contain a number of social biases (e.g. gender, religion, race and more) (Abid
et al., 2021; Lucy and Bamman, 2021). It is important to recognize the effects
of these biases when utilizing FMs in downstream applications. When applying
FMs to data management tasks, we need to be wary of the fact that the inherent
biases of these models may be propagated through their actions. Concretely,
for data repair, the encoded biases of the FM may cause it to replace or
correct entries in a biased manner (e.g. adding a suffix of Ms. to a
traditionally female name). As a result, we need mechanisms for mitigating
model bias and monitoring for bias in model outputs in deployment settings.
Recent work takes a first step in this direction by proposing methods for
automatically detecting bias-sensitive tokens and correcting these tokens to
mitigate biases (Liang et al., 2021).

##  Appendix D Evaluating More FMs on Data Wrangling Tasks

This works focuses on evaluating GPT-3—one of the many large FMs—on a
collection of data integration and cleaning tasks. We contribute tasks from
this paper to the HELM benchmark (Liang et al., 2022), which evaluates
performance of a broader set of large FMs on these tasks.