Example Lu Using co-kriging and ordinary kriging for finding new well
locations

# Application of Co-Kriging and Ordinary Kriging for Selecting Additional Well
Locations

Rong Lu11footnotemark: 1 Colorado School of Mines

###### Abstract

Well performance index (WPI), which is an indicator on how much producing
potential a well has, is proposed for Cana Field using the available
information from the completion database. I used ordinary kriging and co-
kriging to create interpolation maps for WPI across the region. The
interpolation results can be used to predict WPI values for locations that
have not gone through drilling programs, thus guiding operator to find the
next drilling locations. Different kriging models’ performance are compared
using cross-validation. It is shown co-kriging with clean fluid volume has the
best performance. Recommendations are given regarding new well locations.

##  1 Introduction

Unconventional oil and gas resources development has gained much more
attention since the last decade, due to the advancement in hydraulic
fracturing (HF, or “frac”) technology. In order to develop shale gas
reservoirs, which have extremely low permeability, HF has to be applied. In
the process fluids and solids under high pressure are pumped into the
formation to break the rock. As fractures are created, more reservoir contact
are obtained and the shale gas would flow through the fractures to the
wellbore. One question the industry are interested in is, where to drill/frac
the wells in unconventional shale plays.

In this work, a well performance index (WPI) is used to evaluate the
production potential for existing wells (“samples”), which leverages the
initial pressure and production rates information. Then geostatistical
methods, including ordinary kriging and co-kriging, are used to predict
producing potentials for un-drilled regions (“unsampled” locations). My goal
is to compare the performance of different estimators:

  1. 1.

ordinary kriging (OK) on variable of interests

  2. 2.

co-kriging (CK) using secondary variables with different levels of correlation

using cross-validation, and then propose new drilling location candidates for
Cana Field from the found best estimator.

The Cana Field dataset Lu, (2014) is a well completion database. It contains
more than 400 shale gas wells drilled in Cana Woodford Shale in Oklahoma; for
each well, there are completion and HF job parameters, and initial production
data up to the first 90 days.

## Theory & Method

In this section how to find new locations for HF wells is discussed. From the
literature Krivoruchko and Wood, (2014), a well performance index (WPI) is
used to estimate how much production potential a well has, using the initial
90 days’ production and pressure data:

$\text{WPI}=\sum_{i=1}^{90}\text{dailyProdRate}_{i}\times\text{dailyPressure}_{i}$
(1)

It is assumed in this work that pressure is constant over the first 90 days
and that fracture pressure can be used for estimation purposes, then the
equation becomes (the hat indicates it is an estimator for the true WPI):

$\widehat{\text{WPI}}=\text{avgProdRateFirst90Days(MCFE/day)}\times
90\times\\\ \text{fracGradient}\times\text{TVD}$ (2)

Now all the information needed is available from the dataset. A visualization
of the spatial distributions and relative values of $\widehat{\text{WPI}}$ is
shown in Figure LABEL:fig:wpiMap. The approach for prediction, known as
“kriging”, is essentially a linear estimator using the known information. The
core idea is to assign weights $w_{i}$ to each known data point $z(x_{i})$
located at $x_{i}$, and by applying a linear summation the property’s value at
unsampled location $x_{0}$ is obtained:

$\widehat{Z}(x_{0})=\begin{bmatrix}w_{1}&w_{2}&\dots&w_{n}\end{bmatrix}\cdot\begin{bmatrix}z_{1}\\\
z_{2}\\\ \vdots\\\ z_{n}\end{bmatrix}=\sum_{i=1}^{n}w_{i}(x_{0})\times
Z(x_{i})$ (3)

Weights, $w_{i}$, are determined by minimizing variance of estimation Pebesma,
(2004). The performance measure of this linear estimator will be discussed
later. In OK, only the variable of interests (primary variable or primary
data) is studied and the weights will add up to unity. CK also takes advantage
of secondary-data, which have spatial correlations with primary data. Then the
covariance between two or more regionalized variables is leveraged.

wpiMapwidth= This shows a bubble plot for WPI samples. Larger points indicate
larger WPI values. $X$ and $Y$ axes are for easting and northing (units in
meters), respectively.

###  1.1 Assumptions and data pre-processing

Besides using the approximated WPI (Equation 2), only co-located
primary/secondary data is included in this study, which ensures a linear model
of coregionalization (will be addressed later). Since the original values of
variables of interests are huge, both primary and secondary variables are
log10 transformed.

###  1.2 Secondary variables

Clean fluid volume and proppant volume are chosen to be the secondary data
respectively. Their corresponding correlations with primary data is shown in
Figure LABEL:fig:fluidVsWPI,propVsWPI. It can be seen fluid volume has higher
correlation with WPI than proppant volume.

2fluidVsWPI,propVsWPIwidth= Scatter plots between primary and secondary
variables. (a) WPI and fluid volume have correlation coefficient of 0.58; (b)
WPI and proppant volume have correlation coefficient of 0.37.

###  1.3 Toolset

Stanford Geostatistical Modeling Software (SGeMS) is used for the purpose of
exploratory data analysis (EDA). For the final results generation and
presentation, they are completed by two R packages: gstat Pebesma, (2004) and
sp Pebesma and Bivand, (2005). These packages provides:

  * •

OK and CK algorithms

  * •

API for cross validation

  * •

API for fitting linear models of coregionalization

###  1.4 Strategy for verification

In this work $k$-fold cross validation is conducted for comparing the
performance of different estimators. Figure LABEL:fig:cvDemo shows an
illustration when the number of folds, $k$, equals 5. Since the number of WPI
samples is 190 (not huge), leave-$one$-out (LOO) is performed: each run I use
189 data points to “krige” the value at the location of the 190th point, and
then calculate the residual by comparing kriging result with the true value.
Two performance measures are adopted:

  1. 1.

Mean error:

$\text{ME}=\dfrac{1}{n}\sum_{i=1}^{n}(y_{i}-\widehat{y_{i}})$ (4)

  2. 2.

Root-mean-square error:

$\text{RMSE}=\sqrt{\dfrac{1}{n}\sum_{i=1}^{n}(y_{i}-\widehat{y_{i}})^{2}}$ (5)

cvDemowidth= This illustrates the case of 5-fold cross-validation. The
original sample is randomly partitioned into 5 equal sized subsamples. 4 of
the 5 subsamples will be used to train the model, being tested against the
held-out 1 subsample. The process repeats 5 times.

## Results & Verification

###  1.5 Ordinary kriging

The prediction map generated by OK is shown in Figure
LABEL:fig:okVariogram,wpiOkMap

2okVariogram,wpiOkMapwidth= (a) Variogram modeling of the primary variable
WPI; (b) WPI map generate by OK.

###  1.6 Co-kriging with fluid volume

In CK, the spatial covariance between primary and secondary data is
represented by a cross variogram, as presented in Figure
LABEL:fig:ckFluidVarios,ckFluid_WpiMap along with the output WPI map.

2ckFluidVarios,ckFluid_WpiMapwidth= (a) Fitted direct and cross variograms
between log10(WPI) and log10(Fluid). “lt” and “fl” stands for log10 and fluid
volume respectively; (b) WPI map generated by CK with fluid volume.

###  1.7 Co-kriging with proppant volume

CK results using proppant volume as secondary data is shown in Figure
LABEL:fig:ckPropVarios,ckProp_WpiMap. Note the proppant volumes’ experimental
variogram has a fluctuation behavior naturally but the chosen variogram model
smooths it out. This is also to ensure a linear model of coregionalization
(will be addressed later).

2ckPropVarios,ckProp_WpiMapwidth= (a) Fitted direct and cross variograms
between log10(WPI) and log10(Proppant). “lt” and “prop” stands for log10 and
proppant volume respectively; (b) WPI map generated by CK with proppant
volume.

###  1.8 Estimator comparisons

Using the performance measures discussed before (Equation 4 and Equation 5),
the comparisons are summarized in Table LABEL:tbl:modelCompar. It can be seen
CK results are more accurate than OK result, though CK takes much longer time
to run. CK with fluid is chosen as the best model since its modeled variogram
honors the experimental variogram better than CK with proppant did (refer to
Figure LABEL:fig:ckFluidVarios and Figure LABEL:fig:ckPropVarios).

modelCompar Comparison of the Three Interpolations

| Mean Error | RMSE | Running Time
---|---|---|---
OK | -0.0014 | 0.249 | 1.86 sec
CK with fluid | -0.0007 | 0.232 | 6.74 sec
CK with proppant | -0.0009 | 0.229 | 6.67 sec

##  2 Discussion - Conclusion

One of the benefits CK provides is that it might give better predictions when
the main attribute of interest (primary variable; WPI in this case) is sparse,
but related secondary information is abundant. The raw data from Cana dataset
contains more fluid volume (also proppant volume) than WPI, motivating me to
take the CK approach. For modeling, one needs to fit both the direct and
cross-variograms simultaneously, the results of which must produce a positive
definite system. Easiest way to ensure this is to fit a linear model of
coregionalization: all the variograms have same shape and range, with
different partial sills and nuggets. As presented in Figure
LABEL:fig:fullDataCrossVarios, when including all the secondary data (fluid
volume), fluid volume has naturally a shorter range than the primary variable.
Thus finally only co-located data points are included for modeling which
happen to make secondary variables have approximately the same range as the
primary variable.

fullDataCrossVarioswidth= Experimental variograms when including all the
available secondary data. “lt” and “fl” stands for log10 and fluid volume
respectively.

In summary, it has been shown that co-kriging performs better than ordinary
kriging on the Cana dataset. The CK with fluid volume model provides the best
predictive power regarding which regions have higher producing potentials,
from where new drilling locations are proposed (Figure
LABEL:fig:drillDirecPDF).

drillDirecPDFwidth= New wells are recommended to be located at the “dollar
sign” area, whereas the zones showing warnings have lower producing
potentials.

## References

  * Krivoruchko and Wood, (2014) Krivoruchko, K., and N. Wood, 2014, Using Multivariate Interpolation for Estimating Well Performance Understanding Multivariate Interpolation: ArcUser.
  * Lu, (2014) Lu, R., 2014, Investigation of post hydraulic fracturing well cleanup physics in the cana woodford shale: Colorado School of Mines.
  * Pebesma, (2004) Pebesma, E. J., 2004, Multivariable geostatistics in s: the gstat package: Computers & Geosciences, 30, 683–691.
  * Pebesma and Bivand, (2005) Pebesma, E. J., and R. S. Bivand, 2005, Classes and methods for spatial data in R: R News, 5, 9–13.


# AutoML using Metadata Language Embeddings

Iddo Drori
Columbia University and Cornell University
idrori<EMAIL_ADDRESS>
&Lu Liu
Columbia University
<EMAIL_ADDRESS>
&Yi Nian
Columbia University
<EMAIL_ADDRESS>
&Sharath C. Koorathota
Columbia University
<EMAIL_ADDRESS>
&Jie S. Li
Columbia University
<EMAIL_ADDRESS>
&Antonio Khalil Moretti
Columbia University
<EMAIL_ADDRESS>
&Juliana Freire
New York University
<EMAIL_ADDRESS>
&Madeleine Udell
Cornell University
<EMAIL_ADDRESS>

###### Abstract

As a human choosing a supervised learning algorithm, it is natural to begin by
reading a text description of the dataset and documentation for the algorithms
you might use. We demonstrate that the same idea improves the performance of
automated machine learning methods. We use language embeddings from modern NLP
to improve state-of-the-art AutoML systems by augmenting their recommendations
with vector embeddings of datasets and of algorithms. We use these embeddings
in a neural architecture to learn the distance between best-performing
pipelines. The resulting (meta-)AutoML framework improves on the performance
of existing AutoML frameworks. Our zero-shot AutoML system using dataset
metadata embeddings provides good solutions instantaneously, running in under
one second of computation. Performance is competitive with AutoML systems
OBOE, AutoSklearn, AlphaD3M, and TPOT when each framework is allocated a
minute of computation. We make our data, models, and code publicly available.

##  1 Introduction

A data scientist facing a challenging new supervised learning task does not
generally invent a new algorithm. Instead, they consider what they know about
the dataset and which algorithms have worked well for similar datasets in past
experience. Automated machine learning (AutoML) seeks to automate these tasks
to enable widespread use of machine learning by non-experts. A major challenge
is to develop _fast, efficient_ algorithms to accelerate applications of
machine learning [10]. This work develops automated solutions that exploit
human expertise to learn _which datasets are similar_ and _what algorithms
perform best_. We use modern natural language processing (NLP) tools to teach
AutoML systems how to read text descriptions of datasets, and develop a
structured representation of the solutions to Kaggle challenges to allow our
system to run winning solutions on new datasets.

A simple idea is to use machine learning pipelines that performed well (for
the same task) on similar datasets. What constitutes a similar dataset? The
success of an AutoML system often hinges on this question, and different
frameworks have different answers: for example, AutoSklearn [7] computes a set
of _metafeatures_ for each dataset, while OBOE [15] uses the performance of a
few fast, informative models to compute latent features. More generally, for
any supervised learning task, one can view the list of recommended algorithms
generated by any AutoML system as a vector describing that task. Oddly, no
previous work uses the information that a human would check first: a summary
description of the dataset, written in free text. These dataset features
induce a metric structure on the space of datasets. Under an ideal metric, a
model that performs well on one dataset would also perform well on nearby
datasets. The methods we develop in this work show how to learn such a metric
using the recommendations of _any_ AutoML framework together with the dataset
description.

This work makes several contributions to the literature by marrying techniques
from NLP with AutoML. First, we develop NLP text embeddings for datasets by
reading the dataset metadata, including dataset title, description, and
keywords. We show that using these embeddings improves the performance of
state of the art AutoML frameworks such as OBOE [15], AutoSklearn [7],
AlphaD3M [5], and TPOT [12]. Second, we develop NLP text embeddings for
machine learning pipelines by reading the algorithm documentation. We show how
to use these embeddings to develop a new training objective for AutoML: for
any two training datasets, we compute the distance between the embeddings of
the algorithm that performs best on each dataset. We learn a metric on dataset
embeddings to match the distance between the embeddings of the corresponding
best algorithms. We can use this metric for _zero-shot AutoML_ : given a new
dataset, we compute its embedding (given a text description of the dataset),
use it to find the closest training dataset, and output the best algorithm
known for that training dataset. Using the additional information present in
the dataset metadata embeddings and pipeline embeddings improves performance
of existing AutoML systems. The third major contribution of this work is a new
metadata dataset for AutoML that we call AutoKaggle. AutoKaggle consists of a
collection of Kaggle competitions, tasks, winning pipelines, and an execution
engine.

AutoML is an emerging field of machine learning with the potential to
transform the practice of data science by automatically choosing a model to
best fit the data. The reader interested in a comprehensive review of the
field can consult one of the three surveys published in the last twelve months
[16, 9, 17]. A new benchmark [8] provides a quantitative comparison of many
top algorithms.

Language has a common unstructured representation of words, sentences,
paragraphs, and trees of paragraphs which form stories. The most significant
recent advances in NLP learn language models and embeddings from very large
corpuses of text [3, 14]. An unsupervised corpus of text is transformed into a
supervised dataset by defining content-target pairs along the entire text: for
example, target words that appear in each sentence, or target sentences which
appear in each paragraph. A language model is first trained to learn a low
dimensional embedding of words or sentences followed by a map from low
dimensional content to target [11]. This embedding can be used to embed on a
new, unseen and small, dataset in the same low-dimensional space. This work is
the first to propose using such embeddings for automatic machine learning.
Specifically, we use an embedding for the datasets, an embedding for the
pipelines, and the non-linear interactions between these embedding using a
neural network.

One major factor in the performance of an AutoML system is the base set of
algorithms it can use to compose more complex pipelines. For a fair
comparison, in our numerical experiments we compare our proposed methods only
to other AutoML systems that build pipelines out of Scikit-learn [13]
primitives. Now, humans who compete in Kaggle competitions do not restrict
themselves in the same way. Hence as part of AutoKaggle, we have developed and
released translations of every winning Kaggle entry that comply with the
AutoKaggle pipeline format. Each of these can be interpreted by the AutoKaggle
execution engine. The resulting Scikit-learn translation of the pipeline is
sometimes better, but generally a tad worse, than the original human-
engineered pipeline.

##  2 Methods

Notation | Description
---|---
$\mathcal{D}$ | Dataset
$\mathcal{M_{D}}$ |  Metadata of dataset $\mathcal{D}$
$\mathcal{T}$ | Machine learning task (classification, regression)
$\mathcal{P}$ | Solution pipeline
$\mathcal{O,S,A,G,H}$ | OBOE, AutoSklearn, AlphaD3M, TPOT, and human algorithm
$\mathcal{P_{B}(D,T)}$ for $\mathcal{B}\in\\{\mathcal{O,S,A,G,H}\\}$ |  Solution pipeline on dataset $\mathcal{D}$ for task $\mathcal{T}$
$\mathcal{V(P_{B},D,T)}$ |  Evaluating performance of pipeline $\mathcal{P_{B}}$ on $\mathcal{D}$ and $\mathcal{T}$
$E$ | Pre-trained language embedding
$E(\mathcal{M_{D}})$ | Language embedding of dataset metadata
$d(E(\mathcal{M}_{\mathcal{D}_{i}}),E(\mathcal{M}_{\mathcal{D}_{j}}))$ | Distance between dataset metadata embeddings
$\mathcal{D_{\star}}=\underset{\mathcal{D}_{i}}{\text{argmin}}\|E(\mathcal{M}_{D}),E(\mathcal{M}_{D_{i}})\|$ |  Nearest neighbor of $\mathcal{D}$ under distance of embeddings
$\mathcal{P_{\star}=P(D_{\star},T)}$ | Pipeline of most similar embedding
$\mathcal{V(P_{\star},D,T)}$ | Direct pipeline transfer using dataset metadata embedding
$E(\mathcal{P_{B}(D,T)})$ | Language embedding of solution pipeline
$\mathcal{X(D,T)}$ |  Representation of embeddings for dataset $\mathcal{D}$ and task $\mathcal{T}$
Interaction between embeddings |
$\mathcal{I}(\mathcal{D}_{i},\mathcal{D}_{j})=(\mathcal{X}(\mathcal{D}_{i},T)$, $\mathcal{X}(\mathcal{D}_{j},\mathcal{T}))$ |  Neural network input: pair of representations $\mathcal{X(D,T)}$
$\mathcal{O}(\mathcal{D}_{i},\mathcal{D}_{j})=d(E(\mathcal{P_{H}}(\mathcal{D}_{i},\mathcal{T})),E(\mathcal{P_{H}}(\mathcal{D}_{j},\mathcal{T}))$ | Network output: distance between human pipeline embeddings
Table 1: Embedding AutoML notation and their descriptions.

This work uses NLP embeddings to find machine learning pipelines that perform
well for a given dataset and task. We separately embed dataset metadata and
machine learning pipelines and pass the embeddings through a neural network.
This work designs appropriate embedding methods for both dataset metadata and
machine learning pipelines and demonstrates that the resulting recommender
system works well. Table 1 provides the mathematical notation that defines
these methods.

We rely on NLP tools to produce embeddings of dataset metadata
$\mathcal{M}_{\mathcal{D}}$ and of pipelines. Concretely, in our experiments,
we embed dataset metadata by applying the USE3 embedding [2] to the dataset
description (including title, subtitle, description, and keywords) to form
$E(\mathcal{M}_{\mathcal{D}})$, and embed a pipeline $\mathcal{P}$ by applying
the embedding to the function call and the header for each estimator used in
the pipeline to form $E(\mathcal{P})$.

To facilitate evaluation of arbitrary pipelines, we have developed an
_execution engine_ in Python that can represent pipelines composed of machine
learning primitives from the Scikit-learn library. The execution engine takes
as input a description of a pipeline $\mathcal{P}$ (consisting of machine
learning primitives and their parameters, structured in a chain describing the
order of execution) and computes the value
$\mathcal{V}(\mathcal{P},\mathcal{D},\mathcal{T})$ of pipeline $\mathcal{P}$
on task $\mathcal{T}$ for dataset $\mathcal{D}$ by running the primitives
sequentially on a given dataset $\mathcal{D}$ and task $\mathcal{T}$.

The execution engine allows us to run any pipeline for a given task on any
dataset. Our hypothesis is that datasets whose metadata embeddings are similar
will share successful pipelines. To test this hypothesis, we develop an AutoML
approach that we call _direct pipeline transfer_. For a given dataset
$\mathcal{D}$, we find the most similar dataset with respect to the metadata
embedding:

$\mathcal{D}_{\star}=\underset{\mathcal{D}^{\prime}}{\text{argmin}}\|E(\mathcal{M}_{\mathcal{D}})-E(\mathcal{M}_{\mathcal{D}^{\prime}})\|,$
(1)

and evaluate the corresponding pipeline $\mathcal{P_{\star}}$ on the original
dataset $\mathcal{D}$ to compute $\mathcal{V(P_{\star},D,T)}$. Dataset
metadata is useful alone, but can be even more powerful in combination with
other information. We develop embeddings $\mathcal{X(D,T)}$ that are formed by
concatenating metadata embeddings $E(\mathcal{M}_{\mathcal{D}})$ with pipeline
embeddings $E(\mathcal{P(D,T)})$ for pipelines produced by any AutoML system,
as shown in Figure 1. We refer to these dataset embeddings as AutoML
embeddings. We extend the direct pipeline transfer methodology to use these
more complex dataset representations. For a given dataset $\mathcal{D}$ with
representation $\mathcal{X(D,T)}$, we find the most similar dataset with
respect to the metadata embedding:

$\mathcal{D}_{\star}=\underset{\mathcal{D}^{\prime}}{\text{argmin}}\|\mathcal{X(D,T)}-\mathcal{X(D^{\prime},T)}\|,$
(2)

and evaluate the corresponding pipeline $\mathcal{P}_{\star}$ on the original
dataset $\mathcal{D}$ to compute $\mathcal{V(P_{\star},D,T)}$. A disadvantage
of this method is that we must specify both the similarity metric and the
importance of each component of the representation. Instead, we can learn the
similarity metric and interaction between representations $\mathcal{X(D,T)}$
automatically using a neural network, to learn the similarity metric and
output the distance between representations. The neural network takes as input
pairs of datasets $\mathcal{D}_{i}$ and $\mathcal{D}_{j}$ and is trained to
output the performance of the human-selected model for dataset,
$\mathcal{P_{H}}(\mathcal{D}_{i},\mathcal{T})$, evaluated on dataset
$\mathcal{D}_{j}$:
$\mathcal{V}(\mathcal{P_{H}}(\mathcal{D}_{i},\mathcal{T}),\mathcal{D}_{j},\mathcal{T})$.
At prediction time, given a new dataset $\mathcal{D}$, we first compute the
representation $\mathcal{X}(\mathcal{D},\mathcal{T})$, and then compute the
distance of the new dataset to all other datasets using a neural network. We
choose the pipeline $\mathcal{P}$ corresponding to the dataset with the
smallest distance, and evaluate its performance
$\mathcal{V}(\mathcal{P},\mathcal{D},\mathcal{T})$ on the new dataset
$\mathcal{D}$.

Figure 1: AutoML embeddings of dataset metadata and pipelines. The dashed
arrows denote that the representation may consist of any number of the
embeddings.

An important methodology in data science is the common task framework [1, 4]
in which a common dataset and task is given to multiple participants and
evaluated using the same performance metrics. In this work we curate
AutoKaggle, a meta-dataset which contains meta-data about datasets
$\mathcal{D}$, tasks $\mathcal{T}$, and solution pipelines $\mathcal{P}$. This
dataset is important for: analyzing which solution components are used for
which datasets and tasks, understanding which tasks and sub-tasks are given
for which datasets, recommending which high performance solutions be used for
new unseen datasets and tasks, and identifying usage trends of machine
learning libraries and primitives. Our new meta-dataset contains structured
information about a wide variety of machine learning tasks, together with
meta-data about the data, task, and solution pipelines. Solution source code
is parsed into structured machine learning pipelines including: pre-processing
operations, feature extractors, feature selectors, estimators, and post-
processing operations. The dataset can be viewed as a sparse high dimensional
tensor. Rows correspond to (dataset, problem) pairs; other dimensions
correspond to possible values for preprocessors, feature extractors, feature
selectors, estimators, and post-processors. The entries of the tensor are the
performance of the corresponding pipeline for the task on the dataset.

##  3 Results

Table 2 shows our results for a representative set of tabular datasets on
classification tasks. For each dataset (row), Table 2 reports mean evaluation
accuracy of different pipelines (columns) running on the same well-defined
task. Specifically, prediction accuracy of OBOE $\mathcal{V(P_{O}(D,T))}$,
AutoSklearn $\mathcal{V(P_{S}(D,T))}$, AlphaD3M $\mathcal{V(P_{A}(D,T))}$, and
TPOT $\mathcal{V(P_{G}(D,T))}$, evaluation of human generated pipeline
$\mathcal{V(P_{H}(D,T))}$, and the predicted pipeline accuracy of the best
dataset metadata embedding $E(\mathcal{M_{D}})$ and single pipeline embedding
$E(\mathcal{P_{B})}$. All AutoML systems were given one minute of computation
time for a fair comparison; whereas our zero-shot AutoML using dataset
metadata embedding $E(\mathcal{M_{D}})$ runs under one second of computation,
and our pipeline embedding $E(\mathcal{P_{B})}$ runs within the same one
minute of computation while improving performance.

Dataset | OBOE | AutoSklearn | AlphaD3M | TPOT | Human | Ours DE | Ours PE
---|---|---|---|---|---|---|---
Seattle | 1.00 | 0.66 | 1.00 | 1.00 | 0.92 | 1.00 | 1.00
Insurance | 0.47 | 0.50 | 0.35 | 0.51 | 0.48 | 0.47 | 0.52
Forest | 0.73 | 0.83 | 0.83 | 0.84 | 0.84 | 0.85 | 0.85
Credit | 0.93 | 0.94 | 0.93 | 0.94 | 0.94 | 0.94 | 0.94
Titanic | 0.70 | 0.80 | 0.70 | 0.77 | 0.86 | 0.87 | 0.87
HR | 0.84 | 0.83 | 0.87 | 0.90 | 0.86 | 0.86 | 0.90
Kobe | 0.61 | 0.60 | 0.64 | 0.62 | 0.62 | 0.61 | 0.62
Patients | 0.64 | 0.71 | 0.72 | 0.68 | 0.68 | 0.68 | 0.69
Table 2: Machine learning pipeline evaluations for AutoML systems and human
pipelines. All AutoML pipelines $\mathcal{P_{O},P_{S},P_{A},P_{G}}$ are
computed given 1 _minute_ of computation. In comparison, ours DE refers to
$E(\mathcal{M_{D}})$ using only the dataset metadata embedding in under 1
_second_ of computation for zero-shot AutoML. Ours PE refers to
$E(\mathcal{P_{B}})$ using the best single pipeline embedding
$\mathcal{B}\in\\{\mathcal{O,S,A,G}\\}$ in 1 minute of computation.

To implement the metric neural network that learns to predict the distance
$d(E(\mathcal{P_{H}}(\mathcal{D}_{i},\mathcal{T})),E(\mathcal{P_{H}}(\mathcal{D}_{j},\mathcal{T})))$
between the predicted pipeline embedding of pairs of datasets
$\mathcal{D}_{i}$ and $\mathcal{D}_{j}$, we construct a fully connected neural
network with four layers, batch size of 16, and 1200 training epochs. We use
the Adam optimizer with 0.001 learning rate. The input to the neural network
is the representation $\mathcal{X(D,T)}$ of the test dataset and the
representation $\mathcal{X}(\mathcal{D}_{i},\mathcal{T})$ of every other
dataset. We train the neural network for every test dataset and get our
evaluation accuracy by running the obtained pipeline on the test dataset using
our execution engine.

##  4 Conclusions

We have introduced a neural architecture to embed textual descriptions of
dataset metadata and machine learning pipelines for AutoML. We use a new
dataset AutoKaggle consisting of structured representations of winning
solutions of Kaggle competitions and an execution engine to run multiple ML
pipelines. We make our data, models, and code publicly available [6]. In
future work we would like to apply our method to additional AutoML systems
such as Auto-WEKA and H2O AutoML. We would also like to compare the
performance of embedding AutoML using different large language embeddings such
as BERT [3] and GPT-2 [14].

## References

  * [1] David M Blei and Padhraic Smyth.  Science and data science.  Proceedings of the National Academy of Sciences, 114(33):8689–8692, 2017.
  * [2] Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, et al.  Universal sentence encoder.  arXiv preprint arXiv:1803.11175, 2018.
  * [3] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [4] David Donoho.  50 years of data science.  Journal of Computational and Graphical Statistics, 26(4):745–766, 2017.
  * [5] Iddo Drori, Yamuna Krishnamurthy, Remi Rampin, Raoni de Paula Lourenco, Kyunghyun Cho, Claudio Silva, and Juliana Freire.  Automatic machine learning by pipeline synthesis using model-based reinforcement learning and a grammar.  ICML Workshop on Automated Machine Learning, 2019.
  * [6] Iddo Drori, Lu Liu, Yi Nian, Sharath Koorathota, Jie Li, Antonio Khalil Moretti, Juliana Freire, and Madeleine Udell.  GitHub repo for AutoML using metadata language embeddings: data, models, and code.  https://github.com/idrori/automl-embedding, 2019.
  * [7] Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Springenberg, Manuel Blum, and Frank Hutter.  Efficient and robust automated machine learning.  In Advances in Neural Information Processing Systems, pages 2962–2970, 2015.
  * [8] P. Gijsbers, E. LeDell, S. Poirier, J. Thomas, B. Bischl, and J. Vanschoren.  An open source AutoML benchmark.  ICML Workshop on Automated Machine Learning, 2019.
  * [9] Xin He, Kaiyong Zhao, and Xiaowen Chu.  Automl: A survey of the state-of-the-art.  arXiv preprint arXiv:1908.00709, 2019.
  * [10] Efi Kokiopoulou, Anja Hauth, Luciano Sbaiz, Andrea Gesmundo, Gabor Bartok, and Jesse Berent.  Fast task-aware architecture inference.  arXiv preprint arXiv:1902.05781, 2019.
  * [11] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space.  International Conference on Learning Representations Workshop, 2013\.
  * [12] Randal S Olson and Jason H Moore.  TPOT: A tree-based pipeline optimization tool for automating machine learning.  In Automated Machine Learning, pages 151–160. Springer, 2019.
  * [13] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830, 2011.
  * [14] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  OpenAI Blog, 1(8), 2019.
  * [15] Chengrun Yang, Yuji Akimoto, Dae Won Kim, and Madeleine Udell.  OBOE: Collaborative filtering for AutoML model selection.  In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pages 1173–1183, 2019.
  * [16] Quanming Yao, Mengshuo Wang, Hugo Jair Escalante, Isabelle Guyon, Yi-Qi Hu, Yu-Feng Li, Wei-Wei Tu, Qiang Yang, and Yang Yu.  Taking human out of learning applications: A survey on automated machine learning.  CoRR, abs/1810.13306, 2018.
  * [17] Marc-André Zöller and Marco F. Huber.  Survey on automated machine learning.  CoRR, abs/1904.12054, 2019.