11institutetext: Mathew Schwartz 22institutetext: New Jersey Institute of
Technology, Newark,NJ, 22email<EMAIL_ADDRESS>33institutetext: Tomer Weiss
44institutetext: New Jersey Institute of Technology, Newark NJ, 44email:
<EMAIL_ADDRESS>55institutetext: Esra Ataer-Cansizoglu 66institutetext:
Facebook, Boston MA, 66email<EMAIL_ADDRESS>77institutetext: Jae-Woo
Choi 88institutetext: Wayfair, Boston MA, 88email<EMAIL_ADDRESS>
# Style Similarity as Feedback for Product Design

Mathew Schwartz  Tomer Weiss  Esra Ataer-Cansizoglu  and Jae-Woo Choi

###### Abstract

Matching and recommending products is beneficial for both customers and
companies. With the rapid increase in home goods e-commerce, there is an
increasing demand for quantitative methods for providing such recommendations
for millions of products. This approach is facilitated largely by online
stores such as Amazon and Wayfair, in which the goal is to maximize overall
sales. Instead of focusing on overall sales, we take a product design
perspective, by employing big-data analysis for determining the design
qualities of a highly recommended product. Specifically, we focus on the
visual style compatibility of such products. We build off previous work which
implemented a style-based similarity metric for thousands of furniture
products. Using analysis and visualization, we extract attributes of furniture
products that are highly compatible style-wise. We propose a designer in-the-
loop workflow that mirrors methods of displaying similar products to consumers
browsing e-commerce websites. Our findings are useful when designing new
products, since they provide insight regarding what furniture will be strongly
compatible across multiple styles, and hence, more likely to be recommended.

##  1 Introduction

Figure 1: The nodes of the reduced dataset. Color corresponds to the product
group type and size corresponds to the degree of the node.

Interior designers and architects are experienced in creating cohesive styles
of an interior space. However, the cost of such a professional is prohibitive
for most people. Such professionals use intuition, experience, rules in
composition, and color theory to drive decision making, along with modes of
visual communication such as drawing and 3D rendering. For the typical
consumer, interior design and home decoration are a balance of personal taste
and guesswork. One of the more difficult barriers to overcome on the general
consumer side is the visualization of multiple products within their context
(i.e., a room all products will reside in). Similarly, the sorting of massive
online catalogs of products and matching these products in a desirable way can
be overwhelming.

One sorting approach is through using a set of attributes and labels assigned
to products in the catalog. Through such attributes, item-based filtering can
be applied, in which items sharing certain features, such as color, are
recommended to a user. While this technique is a standard industry practice,
it is limited in usefulness for large datasets containing hundreds of
thousands of products. Additionally, certain attributes are more abstract and
difficult to ascertain, requiring multiple ratings and labels by experts–some
of which might not agree on such subjective matter. One such attribute is the
style of a given space, which includes multiple components such as: fabric,
color scheme, material, furniture style and flooring. In this work, we use
such components as the basis of 4 major room styles:_modern, traditional,
cottage, coastal_ , that are popular within an interior product e-commerce
website (Wayfair). We elaborate on descriptions of these styles in Table 2.

An emerging trend in the e-commerce space is to use image-based analysis for
identifying similarities of products. This is useful in online situations
where products must have an image (or 3D rendering) for customer display,
while not being fully labelled with information describing subjective style-
related attributes. In this paper we propose a synergy between the methods
used for recommendation algorithms to a consumer and the design of the
products to be recommended. Background and implementation of past work is
presented, along with an in-depth discussion on the resulting analysis of our
dataset. This dataset is of images labeled with product Stock Keeping Units
(SKUs) and similarly, associated with a product group (Fig. 1) from the
Wayfair company. By organizing data into a network structure, we are able to
visualize and explore connectivity and relationships between various
attributes and types. Finally, we present a flowchart for how such image-based
analysis of style similarities can be integrated within a product designers
workflow, in a not too dissimilar way to that of a consumer browsing online.

##  2 Related Work

Understanding a user’s preferences lies at the heart of many e-commerce
websites ataer2019room ; sachidanandan2019designer ; jing2015visual . An
accurate representation of customer’s style preferences–enables better product
recommendations, and a more personalized shopping experience. The gap in
matching an individuals preference with a product is in the extraction and
categorization of style-related attributes of a product. Researchers have
proposed multiple quantitative methods for understanding (i.e., extracting)
subjective attributes related to aesthetics and fashionablity that compose a
style Hsiao2017 ; Schifanella2015 ; Dhar2011 ; Simo2015 ; Pan2019 ;
lun2015elements  with a few that focus on interiors Pan2019 ; lun2015elements
; weiss2018fast .

Quantitative methods for style recognition are typically data-driven and
employ a variety of machine learning techniques. Recently, neural networks,
and specifically deep neural networks, have become paramount for automatically
assessing images goodfellow2016deep  and in general are effective for image
recognition tasks simonyan2014very . A large body of work exists on the
categorization of images using such deep neural networks. To find an image’s
category, a deep neural network learns a lower dimensional representation,
which we refer to as style embeddings (See Sec. 3), based on labeled data.
These embeddings represents an image’s category in an abstract category space,
such as weightings within various style types. Distances between embeddings
representing different images can then be evaluated to find whether they are
similar hoffer2015deep . The two most frequently used data types used by
researchers to learn style and visual compatibility are images and/or 3D
models.

Using 3D models, Hu et al. hu2017style  presents a method for discovering
elements that characterize a given style, where the elements are co-located
across different 3D models associated with such style. Both Liu and Lun et al.
liu2015style ; lun2015elements  propose to learn 3D model style based on
perceptual geometric elements. They employ crowd-sourcing to quantify the
different geometric style components. Building on this work, Lim and
colleagues lim2016identifying  use neural networks to directly identify the
style of 3D models, without geometric features, by extracting images of the
target 3D model. While the Wayfair e-commerce data consists of 3D models, we
focus on the more general use-case of image data for determining style-
similarity. Using images, Karayev et al. BMVC.28.122  learn the style of
paintings and common photographs using linear classifiers. Thomas and Kovashka
thomas2016seeing  use style to identify a photograph’s artist, which could
have implications for brand-identity. Considering a products connection to
other products and within societal context, Simo-Serra and colleagues
simo2016fashion  propose to predict an outfits’ fashionabilty using neural
networks and Gu et al. gu2017understanding  identify fashion trends from
street photos. Similar to our work, but focusing directly on product images
rather than overall style, bell2015learning  used deep networks for a visual
search system that can aid in interior design through product image
similarity.

For design-based companies, understanding consumer interests and brand
recognition is strongly linked to visual style attributes. Studies into
quantifying design aesthetics for consumer decisions have provided valuable
insights through eye-tracking and interactive modeling hyun2017gap . As
elaborated by Burnap and colleagues burnap2016balancing , the importance of
brand recognition, and the difficulty in balancing with design freedom, is
imperative for companies in brand-conscious spaces. While that work was within
the automotive industry, cohesion between various single products for a
complete style in an interior space is equally important. This cohesion among
products as seen by a consumer should be achieved on both the e-commerce
side–through intuitive and appropriate recommendations to a consumer; as well
as the product design side–by varied designs useable in both different
multiple product groupings (a range of styles) and along with other products
(including with competing brands).

Table 1: Descriptions of $4$ major room styles: modern, traditional, cottage,
coastal. Major style features include fabric, color, material, furniture, and
flooring.

|  Modern |  Traditional |  Cottage |  Coastal
---|---|---|---|---
Fabric |  heavy texture, leathers, linens |  damask or jacquard, velvet or silk, chintz or florals |  soft florals, linen, checks and gingham, toile |  linen, stripes, nautical
Color Scheme |  muted solids in neutrals, greys and blacks |  blue, dark red, hunter green and brown |  muted blues, pinks, reds and greens, white, pale yellows, soft greens |  blue, white, red, green
Furniture |  sleek, low to the ground, clean lines, straight legs on base |  dark wood, gold accents, antique |  slightly distressed, vintage inspired,skirted sofas or chairs, feminine accents, wooden signs |  whitewashed, distressed, beadboard accents, bamboo and rattan
Material |  mixed |  marble, gold, cherry or mahogany wood |  white washed or cherry wood, straw baskets and worn metals |  reclaimed or painted wood, seeded glass or beach glass, beach wood
Flooring |  stripes or natural fiber rugs such as jute or sisal |  ornately patterned carpets |  braided cotton, soft floral or checked rugs |  stripes or woven, seascape prints, sisal or jute

##  3 Learning Image Style with Neural Networks

Our work relies on deep neural networks to learn furniture styles through
images. Training images were collected in a previous study ataer2019room ;
ilkay2020  and each was labeled with a designated style by multiple experts
(Section 3.1). Our neural network is trained to estimate an approximate style
of an image, and in doing so, provides a comparative stylistic differences
between the images (Section 3.2). This stylistic difference is determined by a
16-dimensional layer in the neural network which we denote as _style
embeddings_. Given such embeddings, we are able to retrieve similar style
images (Section 3.3), and by doing so, recommend similar-style products. For a
more detailed explanation than the following overview, please refer to the
previous work, done in conjunction with Wayfair’s Data Science team
ataer2019room ; ilkay2020 .

(a) Modern (b) Traditional (c) Cottage (d) Coastal

Figure 2: Examples of major room styles.

###  3.1 Labeling Image Style

Our initial dataset contains $672,000$ images of interior scenes and products
collected from various sources: (i) Staging a scene in a physical room, and
then capturing it with a camera, (ii) Captured from a virtual scene, curated
by a 3D artist, and (iii) Scraped from 3rd parties. However, due to noisy
image metadata, an image’s origin source might not always be known. Each image
is designated with an interior style label which includes: modern,
traditional, cottage, coastal (Figure 2), where each style is described with
certain criteria about fabric, color scheme, material, furniture style and
flooring (Table 2).

Style labeling is a subjective task prone to noise and variation among
multiple people, including stylists. To alleviate discrepancies in dataset
labeling, 10 independent experts were used to designate a style label to
images. All data, including discrepancies between the expert labels were
recorded, leading to a noisy dataset of labels–a common issue when training a
neural network–as it is prone to inaccurate estimations. Such challenges were
overcome by having the neural network compare images within the same dataset
and their attributed labels from multiple experts. This assumes the existence
of tangible differences between each label that knowledgeable experts may
discern between, similar to the methods described in bradley1952rank .

A new set of comparison labels were generated based on the expert-determined
style labels. During training of the neural network a randomly selected sample
of images were used in which, for each style, and for each pair of images, a
comparison label is generated with respect to the relative order of the number
of style labels each image receives. Formally, for a given style and an image
pair, a comparison label will be $+1$ if in that style, the first image has
$x$ more expert votes than the second image in the pair, and $-1$ if it has
less votes. Variations of these values of voting thresholds were experimented
with, i.e., $x={1,2,...}$ and any image pair was discarded that fell outside
of the defined range.

###  3.2 Neural Network Configuration

We use an architecture inspired by siamese neural networks
bromley1994signature  that extends the Bradley-Terry bradley1952rank , a
neural network that learns from comparisons. VGG16 simonyan2014very  forms the
base of our siamese neural network. The last layer of VGG16 is removed and
instead consecutively add two fully connected layers: a 16-dimensional layer,
which informs us of an image’s style embeddings, and a 4-dimensional layer,
which was used to estimate style.

For the model, 80% of images were allocated for training, 10% for validation,
and 10% for testing. Thus, images in the training set were not paired with
images in the validation or test sets for generating comparison labels. Our
style estimation model was trained on comparisons selected uniformly at random
over the training dataset. Next, the resulting models were evaluated on the
validation set to determine the optimal neural network parameters. This model
was implemented with Python and Tensorflow, where training the neural network
model took about 5 hours on a NVIDIA Tesla V100 GPU.

Figure 3: Our neural network model learns furniture style from images. Given a
context image with a target furniture piece (left), we predict its style
embeddings.

###  3.3 Style Estimation

Since our goal is to accurately estimate an image’s style, we preformed
several experiments to assess our neural network performance. Each experiment
involved random subsets of our image dataset, and different neural network
parameter combinations, including thresholding of expert votes for the
comparison labels. Such multiple experiments aid in finding the optimal set of
parameters that can most accurately estimate the style of a given image.

For estimating an image’s style, we use the last layer of the neural network,
which has 4 dimensions, each of which provide a probabilistic estimation of an
image’s style as modern, traditional, cottage, and coastal (Figure 3). To
evaluate the estimation accuracy, we found the maximum estimation values over
all predicted style labels. Our model achieved an average style estimation
accuracy of $79\%$ for a random image in our dataset. This estimation accuracy
varied by style, with $86.9\%$ estimation accuracy for an image labeled as
modern, $74.6\%$ for traditional, $71.6\%$ for cottage, and $67.9\%$ for
coastal. One possible explanation to the differences in accuracy is that
images with style labels such as coastal can also be subjectively considered
as part of another style, such as modern or traditional. Adding more images to
our dataset might improve the estimation accuracy per style-class.

For recommending similar style images, we use the second to the last layer,
which includes 16 dimensions. Given a seed image, we consider this layer as
the style embeddings of the image. To evaluate our framework’s ability to
recommend similar style images, we retrieved the nearest image(s) by using the
Euclidean distance from the seed image’s embeddings. Figure 4 demonstrates the
$5$ closest images to a given seed image, ranked from left to right, ordered
by the distance to the seed image. To measure our recommendation performance,
we checked if the style labels of the retrieved images match the style of the
seed image. Our network’s retrieval performance is $73.9\%$ on an randomly
sampled image, which is a significant improvement over previous work
ataer2019room .

Figure 4: Product images used for recommendations. Given the furniture image
on the left, our model suggests the recommendations on the right.

##  4 Analysis of Style for Product Design

###  4.1 Data Processing

We use Networkx hagberg2008exploring  in python for constructing a graph from
the relationship data. This graph is then imported to Gephi ICWSM09154  for
visualization. We refer to a given SKU, or SKU and its subsequent options–as a
product. By taking the resulting style embeddings comparison for the images of
the product dataset from the Wayfair e-commerce site, the graph is constructed
by associating products as nodes and the euclidean distance between image
style embeddings (Section 3.3) as weights. As the distance between images is
bi-directional, the result is a weighted undirected graph. To visually analyze
such relationships, edge weights are recalculated to $\frac{1}{dist}$, so that
a higher weight corresponds to stylistically stronger similarity. As our data
is based on raw e-commerce listings, the staging and advertising of products
overlaps between images. To reduce strong connections between images in which
the same products exist albeit in a different setting, the edges within the
graph of overlapping products (when this is known) are removed.

For a more fine-grained analysis of our dataset, we needed to further reduce
the resulting graph. After duplicate data processing, a total of $17,819$
products and $2,485,878$ weights existed in the graph, with weights ranging
from 0.001 to over 100000, the latter in which images are nearly identical. To
remove such noise from the dataset, we cut the graph to include only edges
with a weight between 1 and 10. Next, we removed products that were in a group
category with less than 10 other products. The remaining graph is composed of
$8,947$ nodes and $33,154$ edges. Important to keep in mind, the model we use
for this paper was designed for style-similarity, which as pointed out in
bell2015learning , is different than image similarity as an image can be
visually similar and hence likely stylistically similar, but similar styles
may be different visually.

###  4.2 Product Relationships

Figure 5 depicts a total of 30 nodes representing a group, where the size of
each node relates to its degree. Edges drawn between product groups represent
the cumulative weighting of the underlying product connections in the graph.
The most common group is the End Table with $1,201$ products, followed by Bar
Stools with $808$, and Accent Chairs with $784$. Crates and Carriers, Gliders,
and Shoe Storage all having 10 nodes.

Figure 5: Connections between the product groups.

When viewing the consolidated network, trends among the product groups become
apparent. First, we notice the product type itself heavily influences
connections to other products as distinct groupings emerge, such as the
strongly connected relationship between Bar Stools, Dining Tables, and Dining
Chairs. As discussed previously, dataset noise may be a strong factor, since
it is likely for similar products to be seen within product images, such as
using a Dining Table as staging products for a Dining Chair image. Second,
while the total number of End Tables exceeds all other product groups, the sum
degree of the Bar Stools group is greater. This relationship may suggest a
higher diversity in the styles and relationships a Bar Stool can be
recommended for than End Tables.

(a) (One) TV Stands & Entertainment Center

(b) (One) Bar Stool

(c) (One) Accent Chair

(d) (All) Beds

Figure 6: Similar products of a selected product displayed in red. The groups
are colored by the same scheme in Fig. 1. The most frequently recommended
product is chosen as the target in the first three images (Fig. 5(a)-5(c)),
while the last image (Fig. 5(d)) shows the recommendations from all of the
products in the Bed group (the red cluster on right side of graph).

To further understand group connectivity we explore relationships between the
most connected products in Entertainment Centers, Bar Stools, and Accent
Chairs (Fig. 6). We find that Accent Chairs are commonly recommended for other
categories (e.g., Entertainment Centers or Bar Stools), yet the most commonly
recommended Accent Chair has no connection to other products within its own
group (i.e., other Accent Chairs). This could be a result of weakly connected
products within the same group being filtered out at the initial steps, or
interestingly, a result of the style embeddings determining that the features
of one Accent Chair do not strongly match those of other Accent Chairs. In
this latter case, we interpret this relationship to mean that part of the
style attributes associated with an Accent Chair is the inherent Accent, in
which combining this chair with others would be unlikely. Naturally the style
itself can be similar between these chairs, yet as seen in the example data
(Fig. 4, images are a combination of the products within a scene or
environment, possibly skewing the relationships to that of a more practical
setting in which products exist rather than in isolation. In contrast to the
wide-ranging connections between various product groups in Figures 5(a)\-
5(c), Figure 5(d) highlights (in red) the connections of all products in the
Bed group (located on the right side). Such highlighting emphasizes product
recommendations are limited to only a few groups: End Tables, Nightstands,
Accent Chests / Cabinets, Dressers and Chests, and the smaller categories of
Teen Dressers and Teen Nightstands. These results can be interpreted similarly
to the Accent Chairs in which the training data lacked examples in which Beds
appear in the same image as Bar Stools.

(a) Accent Chair

(b) Bar Stool

Figure 7: Nine most strongly connected recommendations for the most commonly
recommended Accent Chair (Fig. 6(a)) and Bar Stool(Fig. 6(b)).

Keeping in mind an error rate of the machine learning algorithm on a more
individual image scale, by exploring strong relationships to the most popular
products in a group, we gain additional insights as to how a product may
contain stylistic attributes determined by our neural network. One key point
we found is that product groups found to be strongly connected to an input
product generally do not seem to be dominated by color (Fig. 6(a)). When
viewing the strongly connected products in Figure 6(b), a theme of large,
open, and box-like recommendations seem more common than in Figure 6(a).

##  5 Product Design Loop

In the previous sections we describe how the neural network works and provide
a discussion on some of the results. From this basis, we propose a method for
integrating a product design workflow in a manner similar to that of the
consumers interaction with the neural network on the e-commerce side. To
support future work and continued use of our findings as algorithms in style
analysis develop, the abstracted and generalized use case discussed in this
section are independent of the speculative and data-exploratory methods we
employed in the previous sections.

t]

Figure 8: (Left) E-Commerce process for showing related products to a consumer
based on their query and shopping preferences. (Right) A mirrored example
utilized by the designer to find product similarities of a proposed design.

Current E-Commerce systems use a variety of methods to propose alternative and
additional products to a consumer; from reviews to collaborative filtering. In
Figure 8, the left side illustrates this current practice with the right side
demonstrating the Designer in-the-loop process proposed here. In both cases, a
list of possible products on the e-commerce site are run through an image-
based neural network to create a graph of recommendations. On the consumer
side, a query is given to the website on a particular product, which results
in additional related product matches being displayed. The consumer can then
choose from one of these options, after which such choice is updated at an
e-commerce backend server (either as a purchase for reducing inventory, or
stored for additional filtering methods). Alternatively, the consumer can
perform a new query and repeat the process. For the designer, an initial
exploration process begins with finding either gaps in the dataset where
products are not being connected, or by finding strongly connected features
between important/popular products. After implementing such desired features
in a design, the rendered image (As the machine learning model is trained on
both real and 3D rendered images), can be fed to the algorithm which provides
the designer with a product similarity score. From these connections, designer
can either redesign and iterate, or finish the design, adding the finalized
design to the e-commerce website listings. We propose that this process could
enable automated brand cohesion checking as well.

##  6 Discussion

A main challenge and limitation in the final interpretation of our dataset
relates to the uncertainty in the style-based analysis and image matching, as
we rely on a subset of the output from the company-provided model. This has
posed a few issues with extending our analysis and inferring additional
aspects of product style. For one, there are many images that include the same
or similar products. While we have a subset of the images labeled with SKUs
within them, additional staging props or noise in this list can be meaningful
in the recommendations. Similarly, while our data comes from one of the
largest e-commerce companies, it is by no means a complete set of products and
styles. Further implementations and additional product comparison data could
provide insights to the addition of external styles. For example, we are
unsure of what would happen if a style not included in the training data is
given to the model. Additionally, the subjective aspect of style can be a
hindrance to the machine learning model as culturally and over time the labels
and attributes a human perceives as a certain style may change. However, with
the increasing accuracy of machine learning models, or with more analysis
options for a designer to regenerate style groupings to provide results they
more easily understand, our proposed approach may provide a new process for
integrating design and big-data analysis for e-commerce use cases.

Wayfair, one of the largest e-commerce websites in the home and decor space,
focuses on the huge selection, consistent experience, rich product
information, frictionless tools, beautiful content, clear shipping and
delivery, and great service to ensure that the experience of shopping for the
home is easy and fun. As a market-leading platform, beyond the customer’s
needs, we focus on the supplier experience by providing the ability to tell a
product’s story, get to market quickly, and support their business. The
capability to find a product design using qualities found in highly
recommended products can fill a niche that may currently be underserved in the
market and would enable the customers to quickly provide demand signals
towards emerging market trends to suppliers. Reducing the overall time in the
feedback loop between suppliers and customers in the home and decor space will
ensure suppliers have more complete information on market demand and will lead
to a revolution in their planning and production process.

###### Acknowledgements.

We thank Ilkay Yildiz and Hantian Liu for their preliminary work at Wayfair
for the design and experimentation with the neural network style estimation
model.

## Appendix

t]

Figure 9: Top-25 most frequently recommended images out of the 37,421 in our
sample dataset, with the overall similarity threshold. Such image collection
is independent of the products themselves, and exists across all of the
original data, including low scoring similarities.

## References

  * (1) Ataer-Cansizoglu, E., Liu, H., Weiss, T., Mitra, A., Dholakia, D., Choi, J.W., Wulin, D.: Room style estimation for style-aware recommendation.  In: 2019 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), pp. 267–2673. IEEE (2019)
  * (2) Bastian, M., Heymann, S., Jacomy, M.: Gephi: An open source software for exploring and manipulating networks (2009).  URL http://www.aaai.org/ocs/index.php/ICWSM/09/paper/view/154
  * (3) Bell, S., Bala, K.: Learning visual similarity for product design with convolutional neural networks.  ACM Transactions on Graphics (TOG) 34(4), 98 (2015)
  * (4) Bradley, R.A., Terry, M.E.: Rank analysis of incomplete block designs: I. the method of paired comparisons.  Biometrika 39(3/4), 324–345 (1952)
  * (5) Bromley, J., Guyon, I., LeCun, Y., Säckinger, E., Shah, R.: Signature verification using a ”siamese” time delay neural network.  In: Advances in Neural Information Processing Systems, pp. 737–744 (1994)
  * (6) Burnap, A., Hartley, J., Pan, Y., Gonzalez, R., Papalambros, P.Y.: Balancing design freedom and brand recognition in the evolution of automotive brand styling.  Design Science 2 (2016)
  * (7) Dhar, S., Ordonez, V., Berg, T.L.: High level describable attributes for predicting aesthetics and interestingness.  pp. 1657–1664. IEEE (2011)
  * (8) Goodfellow, I., Bengio, Y., Courville, A.: Deep learning.  MIT press (2016)
  * (9) Gu, X., Wong, Y., Peng, P., Shou, L., Chen, G., Kankanhalli, M.S.: Understanding fashion trends from street photos via neighbor-constrained embedding learning.  In: Proceedings of the 25th ACM international conference on Multimedia, pp. 190–198 (2017)
  * (10) Hagberg, A., Swart, P., S Chult, D.: Exploring network structure, dynamics, and function using networkx.  Tech. rep., Los Alamos National Lab.(LANL), Los Alamos, NM (United States) (2008)
  * (11) Hoffer, E., Ailon, N.: Deep metric learning using triplet network.  In: International Workshop on Similarity-Based Pattern Recognition, pp. 84–92. Springer (2015)
  * (12) Hsiao, W.L., Grauman, K.: Learning the latent “look”: Unsupervised discovery of a style-coherent embedding from fashion images.  pp. 4213–4222. IEEE (2017)
  * (13) Hu, R., Li, W., van Kaick, O., Huang, H., Averkiou, M., Cohen-Or, D., Zhang, H.: Co-locating style-defining elements on 3D shapes.  ACM Trans. Graph. 36(3), 33:1–33:15 (2017)
  * (14) Hyun, K.H., Lee, J.H., Kim, M.: The gap between design intent and user response: identifying typical and novel car design elements among car brands for evaluating visual significance.  Journal of Intelligent Manufacturing 28(7), 1729–1741 (2017)
  * (15) Jing, Y., Liu, D., Kislyuk, D., Zhai, A., Xu, J., Donahue, J., Tavel, S.: Visual search at pinterest.  In: Proceedings of the 21th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 1889–1898. ACM (2015)
  * (16) Karayev, S., Trentacoste, M., Han, H., Agarwala, A., Darrell, T., Hertzmann, A., Winnemoeller, H.: Recognizing image style.  In: Proceedings of the British Machine Vision Conference. BMVA Press (2014).  DOI http://dx.doi.org/10.5244/C.28.122
  * (17) Lim, I., Gehre, A., Kobbelt, L.: Identifying style of 3d shapes using deep metric learning.  In: Computer Graphics Forum, vol. 35, pp. 207–215. Wiley Online Library (2016)
  * (18) Liu, T., Hertzmann, A., Li, W., Funkhouser, T.: Style compatibility for 3d furniture models.  ACM Transactions on Graphics (TOG) 34(4), 1–9 (2015)
  * (19) Lun, Z., Kalogerakis, E., Sheffer, A.: Elements of style: learning perceptual shape style similarity.  ACM Transactions on graphics (TOG) 34(4), 84 (2015)
  * (20) Pan, T.Y., Dai, Y.Z., Hu, M.C., Cheng, W.H.: Furniture style compatibility recommendation with cross-class triplet loss.  Multimedia Tools and Applications 78(3), 2645–2665 (2019)
  * (21) Sachidanandan, S., Luong, R., Joergensen, E.S.: Designer-driven add-to-cart recommendations.  In: Proceedings of the 13th ACM Conference on Recommender Systems, pp. 525–525 (2019)
  * (22) Schifanella, R., Redi, M., Aiello, L.M.: An image is worth more than a thousand favorites: Surfacing the hidden beauty of flickr pictures.  In: International AAAI Conference on Web and Social Media (2015)
  * (23) Simo-Serra, E., Fidler, S., Moreno-Noguer, F., Urtasun, R.: Neuroaesthetics in fashion: Modeling the perception of fashionability.  pp. 869–877 (2015)
  * (24) Simo-Serra, E., Ishikawa, H.: Fashion style in 128 floats: Joint ranking and classification using weak data for feature extraction.  In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 298–307 (2016)
  * (25) Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale image recognition.  arXiv:1409.1556 (2014)
  * (26) Thomas, C., Kovashka, A.: Seeing behind the camera: Identifying the authorship of a photograph.  In: Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3494–3502 (2016)
  * (27) Weiss, T., Litteneker, A., Duncan, N., Nakada, M., Jiang, C., Yu, L.F., Terzopoulos, D.: Fast and scalable position-based layout synthesis.  IEEE Transactions on Visualization and Computer Graphics (2018)
  * (28) Yildiz, I., Cansizoglu, E., Liu, H., Golbus, P., Tezcan, O., Choi, J.W.: Deep ranking for style-aware room recommendation.  In: Proceedings of AAAI (2020).  Student Abstract


#  Prompt2Model:
Generating Deployable Models from Natural Language Instructions

Vijay Viswanathan1 , Chenyang Zhao1,2∗,
Amanda Bertsch1, Tongshuang Wu1, Graham Neubig1
1Carnegie Mellon University, 2Tsinghua University equal contribution.

###### Abstract

Large language models (LLMs) enable system builders today to create competent
NLP systems through prompting, where they only need to describe the task in
natural language and provide a few examples. However, in other ways, LLMs are
a step backward from traditional special-purpose NLP models; they require
extensive computational resources for deployment and can be gated behind APIs.
In this paper, we propose Prompt2Model, a general-purpose method that takes a
natural language task description like the prompts provided to LLMs, and uses
it to train a special-purpose model that is conducive to deployment. This is
done through a multi-step process of retrieval of existing datasets and
pretrained models, dataset generation using LLMs, and supervised fine-tuning
on these retrieved and generated datasets. Over three tasks, we demonstrate
that given the same few-shot prompt as input, Prompt2Model trains models that
outperform the results of a strong LLM, gpt-3.5-turbo, by an average of 20%
while being up to 700 times smaller. We also show that this data can be used
to obtain reliable _performance estimates_ of model performance, enabling
model developers to assess model reliability before deployment. Prompt2Model
is available open-source at https://github.com/neulab/prompt2model.111Our demo
video is posted at youtu.be/LYYQ_EhGd-Q.

##  1 Introduction

Traditionally, building an NLP model from scratch has been a substantial
undertaking. An NLP practitioner seeking to solve a new problem would need to
define their task scope, find or create data that specifies the intended
system behavior, choose a suitable model architecture, train the model, assess
its performance through evaluation, and then deploy it for real-world usage
Paleyes et al. (2022).

LLMs like GPT-3 (Brown et al., 2020; Liu et al., 2023b) offer a lighter-weight
paradigm for NLP system construction through “prompting” (Reynolds and
McDonell, 2021). Practitioners can now write a prompt specifying the intended
system behavior (optionally with a few demonstrations), and ask an LLM to
generate a desired output via text completion. This makes it possible to
prototype NLP systems rapidly for a variety of applications without writing a
single line of code (Floridi and Chiriatti, 2020).

Figure 1: Prompt2Model is a framework for generating a small yet accurate
model from a prompt.

However, there is still a gap between proof-of-concept prototyping — showing
LLMs can be prompted for a particular task — and practical deployment.
Prompting LLMs can be expensive as they require either a significant amount of
computing or access to commercial APIs, and their reliance on the input prompt
quality makes them unstable compared to trained models (Min et al., 2022;
Bubeck et al., 2023). Because practitioners usually do not have enough
annotated validation data to measure their system performance, it is also more
challenging for them to debug their systems before deployment Jiang et al.
(2022). Additionally, LLM-prompted systems pose usability challenges.
Practitioners have expressed concerns about the high serving cost and slow
prediction time associated with using LLMs (Park et al., 2022), and those
working in high-stakes domains cannot rely on commercial LLM APIs due to
privacy concerns. For instance, sharing user data with LLM service providers
is illegal for many applications in the US (Sezgin et al., 2022).

In this work, we present Prompt2Model, a system that retains the ability to
specify system behavior in a light-weight way through _prompting_ , while
still resulting in a _deployable special-purpose model_ , maintaining all the
advantages thereof. Prompt2Model is designed as an automated pipeline that
extracts essential task information from users’ prompts and then automatically
collects and synthesizes task-specific knowledge through three channels:

  * •

_Dataset retrieval_ : Whenever possible, we collect training data by
retrieving task-relevant annotated data (Färber and Leisinger, 2021;
Viswanathan et al., 2023).

  * •

_Dataset generation_ : We distill knowledge from an LLM (“teacher model”) by
employing it to generate a pseudo-labeled dataset. Prior work has demonstrated
that such a dataset can be used to train a smaller “student” model to emulate
the behavior of the teacher model Wang et al. (2021a); He et al. (2023);
Gudibande et al. (2023).

  * •

_Model retrieval_ : Based on the prompt, we identify a pretrained language
model whose parametric knowledge is appropriate for the user’s intent. This
chosen model serves as the student model and is further fine-tuned and
evaluated using the generated and retrieved data.

Prompt2Model is designed to support different instantiations of each of these
components. We provide a reference implementation where we demonstrate its
utility with a gpt-3.5-turbo-based dataset generator, a dataset retriever
based on DataFinder Viswanathan et al. (2023), and a model retriever using
BM25. We evaluate on three tasks covering both traditional NLP benchmarks and
novel applications and find that, empirically, Prompt2Model sometimes produces
small models that outperform gpt-3.5-turbo when using the same prompt as
input. On 2 of these 3 tasks, we observe >20 point improvements over the
gpt-3.5-turbo baseline, despite the final model produced by Prompt2Model being
up to 700 times smaller. We also find that we can generate effective
evaluation datasets; performance improvements on these synthetic clones of
real benchmarks also hold on their real counterparts. We believe that
Prompt2Model can serve the following purposes for the community:

  1. 1.

A tool for quickly building small and competent NLP systems: Prompt2Model can
be directly used to produce task-specific models that outperform LLMs in a few
hours without any manual data annotation or architecture design. The method
bridges the gap between the proof-of-concept LLM prototyping and the practical
deployment of the model.

  2. 2.

A testbed for end-to-end, prompt-based model training: Given Prompt2Model’s
extensible design, it can offer a platform for exploring new techniques in
model distillation, dataset generation, synthetic evaluation, dataset
retrieval, and model retrieval. Our platform allows studying these components
using extrinsic downstream metrics, enabling empirical progress on these
research areas.

##  2 Prompt2Model Framework

Figure 2: The Prompt2Model architecture seeks to automate the core machine
learning development pipeline, allowing us to train a small yet accurate model
from just a prompt.

Our system, Prompt2Model, provides a platform to automate the components of a
machine learning pipeline: data collection, model training, evaluation, and
deployment. We illustrate our automated pipeline in Figure 2. At the core is
our automatic data collection system, which leverages dataset retrieval and
LLM-based dataset generation to obtain labeled data relevant to the user’s
needs. We then retrieve pretrained models which we finetune on the training
splits of the collected datasets. Finally, we evaluate our trained models on
the test splits of the same datasets and optionally create a web UI that can
be used to interact with the model.

Our general-purpose method is designed to be modular and extensible; each
component can be implemented differently or disabled by a practitioner. We
give an overview of our framework, then in section 3 we describe our reference
implementation.

#### Prompt Parser

As the primary input to our system, users provide prompts similar to those
used for LLMs. These prompts comprise an instruction and, optionally, a few
demonstrations of the anticipated behavior. While this open-ended interface is
convenient for users, end-to-end ML pipelines may benefit from a Prompt Parser
that processes this input, such as segmenting the prompt into an instruction
and individual demonstrations or translating instructions into English.

#### Dataset Retriever

Given a prompt, we first try to discover existing manually-annotated data that
can support a user’s task description. There are several design decisions for
the Dataset Retriever:

  1. 1.

What datasets to search against?

  2. 2.

How to index datasets for search?

  3. 3.

Which dataset columns are needed for the user’s task, and which columns should
be ignored?

Prior works by Färber and Leisinger (2021) and Viswanathan et al. (2023)
introduced systems for dataset search. We use the latter, called _DataFinder_
, in our implementation, as described in §3.2.

#### Dataset Generator

Not all conceivable tasks have any existing annotated data, and many tasks are
only somewhat relevant to an existing dataset. To support a wide range of
tasks, we introduce a Dataset Generator to produce synthetic training data as
per the user-specific requirements parsed by the _Prompt Parser_. This
component presents challenges related to cost efficiency, generation speed,
example diversity, and quality control. We discuss our suggested solution to
these challenges in §3.3.

#### Model Retriever

Besides training data, we must identify an appropriate model to finetune. We
cast this as a retrieval problem, where each model is represented by user-
generated descriptions and metadata such as popularity or tasks supported. The
reference implementation of our Model Retriever, described in §3.4, searches
against pretrained models on Hugging Face (Wolf et al., 2020), but this could
instead cover other model repositories such as Model Zoo (Koh, 2020).

#### Training

Given retrieved and generated datasets and a pretrained model, we use a Model
Trainer to finetune the model on a subset of the data. We currently train
models by treating all tasks as text-to-text generation (Raffel et al., 2020),
as described in §3.5, but emphasize that this component can be extended in the
future to support new approaches.

#### Evaluation

After training models on a portion of the retrieved and generated datasets, we
give the remaining data to an Model Evaluator module. We aim to support a
variety of tasks, and selecting the correct task-specific metrics for an
arbitrary task is a difficult problem. We describe our suggested strategies
for task-agnostic evaluation in §3.6.

#### Web App Creation

To enable developers to expose a model to collaborators or users, we include
an optional component called the Demo Creator to create a graphical interface
to interact with the model. We briefly describe our implementation of this
component in §3.7.

##  3 Reference Implementation

Prompt2Model is designed modularly to support customization of each component
in our framework (described in §2), but we have provided a reference
implementation to enable immediate adoption.

###  3.1 Prompt Parser

We parse the prompt into instruction and demonstrations fields (shown in
Figure 2), where the instruction represents the primary task or objective and
the demonstrations exemplify the desired behavior. To achieve this, we utilize
an LLM with in-context learning to segment user prompts, employing the OpenAI
gpt-3.5-turbo-0613 in our experiments. If the instruction provided is
identified to be in a language other than English, we translate it to English
using the DeepL API.222https://www.deepl.com/en/docs-api

###  3.2 Dataset Retriever

To retrieve datasets for a prompt, we adapt the _DataFinder_ system introduced
by Viswanathan et al. (2023). By extracting user-generated dataset
descriptions for each dataset in Hugging Face Datasets (Lhoest et al., 2021),
we utilize DataFinder’s trained bi-encoder retriever to rank the most relevant
datasets. Once a relevant dataset is identified, the next step is to determine
which columns of the dataset correspond to the input and the desired output
specified by the user. As automatically inducing the correct schema for any
dataset can be challenging, we adopt a human-in-the-loop approach. We present
the top-$k$ datasets, where $k=25$ by default, to the user and allow them to
either select the most relevant dataset or to state that none are a good fit
for their task. We then ask the user to identify the appropriate columns for
input and output from the dataset’s schema.

###  3.3 Dataset Generator

We carefully engineered our dataset generator to enable speed-optimized
generation at a low-cost while creating diverse and high-quality examples. Our
strategy comprises the following components:

#### High-Diversity Few-Shot Prompting

We use automated prompt engineering to generate a diverse dataset. We augment
the user-provided demonstration examples with a random sample of previously
generated examples to promote diversity and avoid generating duplicate
examples. Without this strategy, 120 out of 200 generated QA examples were
duplicates; with it, only 25 were duplicates.

#### Temperature Annealing

We adjust the sampling temperature from low (favoring deterministic outputs)
to high (encouraging diverse exploration) proportionally to the number of
examples already generated. This modulation helps preserve output quality
while gradually encouraging diversity.

#### Self-Consistency Decoding

Given that LLM may generate non-unique or incorrect outputs for the same
inputs, we use _self-consistency_ filtering (Wang et al., 2022) to select
pseudo-labels. Specifically, we create a consensus output for each unique
input by selecting the most frequent answer; in the case of ties, we
heuristically select the shortest answer. This promotes accuracy of the
generated dataset while ensuring unique examples.

#### Asynchronous Batching

API requests are parallelized using _zeno-build_ (Neubig and He, 2023). We use
additional mechanisms, such as dynamic batch size and throttling, to optimize
API usage.

###  3.4 Model Retriever

We need to select an appropriate model to finetune. To support many tasks with
a unified model interface, we presently limit ourselves to encoder-decoder
architectures on Hugging Face (Wolf et al., 2020), following recent work that
shows that encoder-decoder models are more data-efficient for model
distillation (Calderon et al., 2023). This restriction still leaves a large
set of pretrained models to choose from, e.g. Salesforce/codet5-base for
coding-related tasks (Wang et al., 2021b) or MaryaAI/opus-mt-ar-en-finetuned-
ar-to-en for Arabic-to-English translation (Tiedemann and Thottingal, 2020).
We frame the problem of selecting a pretrained model as a search problem.
Using the user’s instruction as a query, we search against all textual
descriptions of models on Hugging Face.

This search task is challenging because Hugging Face model descriptions are
sparse and contain lots of templatic text, often with only a few words that
signify the content of the model. To address this, we follow the HyDE
framework (Gao et al., 2023) and first use gpt-3.5-turbo to create a
_hypothetical model description_ given the user’s instructions. We show an
example of a hypothetical document generated for a question-answering
instruction in Figure 3. Using this description as an expanded query, we then
apply the BM25 algorithm to compute query-model similarity scores (Robertson
et al., 1995). To ensure the ease of deployment of the resulting model, we
filter out models whose size (in bytes) exceeds a user-specified threshold
(set to 3GB by default). Using the intuition that highly-downloaded models are
more likely to be high in quality, we choose the top model after ranking by:

$BM25(\text{query},\text{model})\cdot\log(\text{\\# of Downloads}+1).$

Figure 3: For our model retriever, we first construct a hypothetical model
description for a query, then compute similarity scores between that
hypothetical model description and the descriptions of real models.

###  3.5 Training

#### Dataset Processing

We train the model by leveraging two datasets- one generated and one
retrieved. To sidestep the challenge of making schema-specific modeling
decisions (e.g. constructing specialized architectures for classification or
generation tasks), we treat all datasets as “text-to-text” problems (Raffel et
al., 2020). We textualize the input columns of each dataset and prepend the
user’s instructions to the input to guide the model.

#### Finetuning

We concatenate the retrieved and generated datasets and shuffle them before
training the student model. We use the same default hyperparameters for all
tasks.333We empirically find that these default hyperparameters are effective,
but we plan on implementing hyperparameter selection using generated
validation data in the future. We train with the AdamW optimizer with lr =
5e-5 for 3 epochs, which takes roughly one hour for all tasks.

###  3.6 Evaluation

Our _Model Evaluator_ automatically evaluates models for all tasks using three
general-purpose metrics: Exact Match, ChrF++ (Popović, 2015), and BERTScore
(Zhang et al., 2019). ChrF++ balances precision and recall to assess text
generation quality. Exact Match measures how often the model output perfectly
matches the exact reference. BERTScore captures semantic similarities despite
different wordings or phrasings by comparing the model output and reference in
the embedding space. We use XLM-R (Conneau et al., 2020) as the encoder for
BERTScore to support multilingual evaluation.

###  3.7 Web App Creation

We finally provide an optional step in Prompt2Model to automatically create a
graphical user interface that allows downstream users to interact with the
trained model. This web application, built using Gradio (Abid et al., 2019),
can then be easily deployed publicly on a server.

##  4 Experimental Setup

#### Tasks

As a proof-of-concept, we test our system’s ability to learn a model for three
tasks:

  * •

_Machine Reading Question Answering_ : We first consider a common use case
where pretrained models and training datasets are plentiful. We use SQuAD
(Rajpurkar et al., 2016) as ground truth to evaluate this setting.

  * •

_Japanese NL-to-Code_ : Code generation from Japanese-language queries is a
challenging scenario where prior work exists but no annotated data or
pretrained models are available. We use MCoNaLa (Wang et al., 2023) for
evaluation.

  * •

_Temporal Expression Normalization_ : We finally consider a task where there
are no pretrained models or training datasets of any kind available. Here we
use the Temporal dataset of Wu et al. (2023) as ground truth for evaluation.

Though Prompt2Model offers automated model evaluation (on generated and
retrieved datasests), we use real benchmark datasets here to measure our
pipeline’s ability to train accurate models.

#### LLM Baseline

A primary goal of our work is to train small models that can match or
outperform LLMs. To measure success towards this goal, we report the
performance of gpt-3.5-turbo on each benchmark. We provide gpt-3.5-turbo444We
used gpt-3.5-turbo-0613, accessed between July 26 and August 6, 2023. the same
instruction and demonstrations provided to Prompt2Model, for fair comparison.

##  5 Experiment Results

###  5.1 Downstream performance

How effective is Prompt2Model at producing a high-quality model? In Table 1,
we evaluated models produced by Prompt2Model, as well as our baseline LLM
gpt-3.5-turbo, on real benchmark datasets for each task — SQuAD, MCoNaLa, and
Temporal. We further examine the effect of removing 2 specific elements of the
Prompt2Model pipeline — model retrieval and dataset retrieval.

Method | SQuAD | MCoNaLa | Temporal
---|---|---|---
| (EM) | (ChrF++) | (ChrF++)
Prompt2Model | 61.5 | 13.1 | 55.2
w/o Model Ret. | 61.5 | 15.8 | 55.2
w/o Data Ret. | 50.2 | 16.6 | N/A
gpt-3.5-turbo | 42.1 | 37.3 | 30.7
Table 1: We evaluate the model produced by Prompt2Model on real benchmarks for
each test set, compared to gpt-3.5-turbo, which we used to power our dataset
generator. We also examine the effect of removing specific parts of our
pipeline — model retrieval and dataset retrieval. There are no relevant
datasets available for the Temporal task, so we did not use retrieved data for
Prompt2Model there.

On 2 of 3 datasets, we find that Prompt2Model produces models that are
considerably more accurate than gpt-3.5-turbo. This is remarkable because the
retrieved model for SQuAD and Temporal is Flan-T5, which, at 250M parameters,
is up to 700 times smaller than gpt-3.5-turbo (which is believed to contain
175B parameters).

We observe that Prompt2Model’s performance on MCoNaLa’s Japanese-to-Python
task is significantly worse than gpt-3.5-turbo. One explanation for this is
the relatively low diversity in the generated dataset of Japanese queries; 45
of 5000 examples are different ways of saying “find the maximum value in a
list of numbers“. We do not observe this level of redundancy in our other
datasets, suggesting that gpt-3.5-turbo may struggle to generate diverse text
for non-English languages. Another reason is the lack of an appropriate
student model — the models found by the model retriever were trained on either
on multiple language or code, but not both. The resulting pretrained models
may lack the parametric knowledge to represent the Japanese inputs, Python
outputs, or both.

###  5.2 Combining retrieved and generated datasets is powerful

Ideally, generated and retrieved data should be as close to the target domain
as possible. In our experimental setting, where we deliberately choose prompts
that mimic existing datasets, we can evaluate how well the model performs
relative to a model trained on the same amount of data from the true dataset.
We use SQuAD as a running example.555We focus on only SQuAD here because our
other two tasks have less real training examples than the datasets we
generate, making comparison impractical. As our prompt is a description of the
SQuAD passage-level question answering task (Figure 1), we exclude SQuAD from
our retrieved datasets list. Instead, we evaluate models finetuned on:

  1. 1.

3k examples from the closest retrieved dataset666The closest dataset retrieved
by the dataset retriever for our SQuAD-inspired prompt is The Children’s Book
Test Dataset Hill et al. (2016).

  2. 2.

3k examples generated by Prompt2Model

  3. 3.

The union of the above, which is what the full Prompt2Model pipeline uses

  4. 4.

3k examples from SQuAD (analogous to the user custom-annotating data for a
task).

Table 2 shows the results across these four settings. While using retrieved or
generated data causes a reduction in performance due to domain shift, the
combination of the two methods achieves similar performance to using the true
dataset. For this machine reading comprehension task where the user would need
to custom-annotate data for their task, Prompt2Model allows for similar
performance at less than 1% of the cost.

Method | #Train | Performance | Anno. Cost
---|---|---|---
Retrieval only | 3,000 | 56.79 |  $\approx$ $ 0
Generation only | 3,000 | 44.20 |  $\approx$ $ 5
Retrieval+generation | 6,000 | 61.46 |  $\approx$ $ 5
Custom annotation | 3,000 | 61.64 |  $\approx$ $ 540
Table 2:  We compare model performance on SQuAD on an annotation-cost basis,
using datasets produced by different modules of Prompt2Model, along with
fully-manual annotation. Performance reported for all models is the exact
match on the test set,888Exact match is the standard metric used for measuring
SQuAD performance. which reflects _the true task performance_. Cost of custom
annotation is estimated from Rajpurkar et al. (2016) using their reported
annotator pay rate of $9/hour and keeping 1,000 validation examples.

###  5.3 Our generated evaluation data can identify real modeling improvements

Dataset | Metric | $\tau$ |  $p$-value
---|---|---|---
SQuAD | EM | 64.3 | 0.03*
Temporal | ChrF++ | 24.2 | 0.31
MCoNaLa (JP) | ChrF++ | 70.9 | 0.00**
Table 3: We evaluate 10 different models on real test sets and their
corresponding generated clones. We compute Kendall’s Tau on the ranked lists
of models and find statistically significant correlations for 2 of 3 datasets.

High-quality generated data should also allow us to _discriminate_ between
multiple candidate models to select a model that will perform well downstream.
We finetune various models on a generated dataset and rank their performance
according to the generated test data and the test data from the target (real)
dataset. We evaluate the Kendall’s rank correlation Kendall (1938) between the
two rankings to determine if our generated data can effectively determine
which models are likely to perform well downstream. This is closely related to
the concept of concurrence between benchmarks Liu et al. (2023a); however, we
are evaluating whether the generated and real data rank specific models in the
same ordering, rather than modeling approaches.

Table 3 shows the Kendall’s $\tau$ for each task, computed over a set of
reasonable models.999This set of models consisted of 5 T5-family models, 2
BART-family models, and 1-5 additional retrieved models from the Model
Retriever, depending on task. The generated data shows strong correlation to
the true performance on two of the three datasets.

##  6 Discussion and Conclusion

We propose Prompt2Model, a framework that automatically constructs task-
specific models using only natural language prompts. Our proof-of-concept
experiments show that, despite using a similar easy-to-use interface as LLMs,
Prompt2Model delivers small yet accurate models and its generated datasets can
be used to estimate real-world performance. Besides our reference
implementation providing a ready-to-use tool, Prompt2Model’s extensible design
and modular implementation makes it a platform for advancing model
distillation, dataset generation, synthetic evaluation, dataset retrieval, and
model retrieval.

We believe our Prompt2Model framework can inspire various novel research
questions. We hope that our platform enables future work that looks more
deeply into quality assurance on the generated data and the model. Interesting
questions include how much data should we generate for downstream model
training and how diverse should it be? How do we effectively mix the retrieved
and generated dataset such to achieve complementary strengths (e.g. using
dataset generation to focus on the expected inputs to the model that the
retrieved dataset fails to cover)? Since users often struggle to articulate
their needs up front, future extensions should also address the challenge of
human-in-the-loop correction – either by offering potential strategies to help
humans iteratively refine prompts, or allowing humans to perform post-hoc
fixes when the task metadata extraction and generated data do not align with
their intentions. We hope to propose explicit challenges and invite the
community to contribute novel implementations of various components in our
framework.

## Limitations

One of the primary limitations of our system is that our current experiments
have all been conducted using the gpt-3.5-turbo API (used for prompt parsing,
dataset generation, and model retrieval). This LLM is paid and closed-source,
which makes this problematic as a scientific artifact (Rogers et al., 2023).
Furthermore, the service provider of this LLM, OpenAI, prohibits the use of
their API to create models that may compete with OpenAI, creating potential
legal concerns with the use of Prompt2Model in commercial applications. We are
exploring the integration of open-source LLMs to avoid our reliance on
proprietary APIs.

Another limitation of our work is the limited ability of Prompt2Model to
support tasks that require processing languages other than English. While we
have shown the limitations of our system at supporting code generation from
Japanese natural language queries, our system is likely to struggle more with
lower-resource languages. We use the unpublished gpt-3.5-turbo model for our
Dataset Generator in our reference implementation. This model is believed to
be similar to GPT-3 (Brown et al., 2020), which was trained on 93% English
documents, 1% German documents, 1% French documents, and <5% documents in any
other language. Our use of this model may exacerbate existing disparities in
language technologies between high-resource languages and low-resource
languages.

One potential limitation is that we have only tested our approach on 3 tasks,
each with a single dataset and a single evaluation metric. We justify this
decision because our focus is on providing an extensible software system
rather than establishing state-of-the-art results on many datasets, but we
believe that our results suggest broader applicability.

## Ethics Statement

Any system which makes powerful technology more accessible to the public has
ethical implications. Widder et al. (2022) discuss ethical issues with open-
source packages in relation to software libraries for deepfaking, including
the possibility of enabling malicious actors to use technology that they would
otherwise not have the technical skills to leverage. This is also a risk for
an AutoML system such as Prompt2Model; however, we believe this risk is
outweighed by the benefits of greater accessibility, especially given that a
low barrier to entry for generating harmful data already exists in the form of
prompted, web-interface models.

While Prompt2Model could, if given harmful inputs, generate toxic, offensive,
or inaccurate synthetic data, this is no more of a risk with Prompt2Model than
it is with the underlying prompted model Bender et al. (2021); indeed, the use
of models and supplementary datasets retrieved from Hugging Face may lessen
the likelihood of a downstream model replicating harms from the prompted
model’s outputs, though more investigation is needed. Like all ML models, the
models that Prompt2Model returns can make mistakes, and we aim to be
transparent in our documentation about potential limitations of the system.

We hope that Prompt2Model will be broadly useful. Our work is motivated by a
desire to increase the accessibility of NLP models to people who are not in
the NLP community but would benefit from the community’s innovations;
particularly, to people who would use NLP models downstream but may not have
the domain-specific knowledge to design their own system. Prompt2Model may
also prove useful for early NLP researchers by providing a starting point for
intuitions about baselines for various tasks and enabling the discovery of
similarities between a described task and existing work. We open-source
Prompt2Model and welcome community contributions.

## Acknowledgements

This work was supported in part by a fellowship from NEC Research
Laboratories. We are grateful to Alex Cabrera, Will Epperson, Nelson Liu,
Arjun Ramani, Zirui Cheng, Zhiyuan Zeng, Tianci Xue, Yanchen Liu, Yi-Hsin Hung
and Zhilin Yang for their feedback and guidance. We particularly appreciate
Zirui Cheng’s video production support for our demo.

## References

  * Abid et al. (2019) Abubakar Abid, Ali Abdalla, Ali Abid, Dawood Khan, Abdulrahman Alfozan, and James Zou. 2019.  Gradio: Hassle-free sharing and testing of ML models in the wild.  _arXiv preprint arXiv:1906.02569_.
  * Bender et al. (2021) Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big? In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21, page 610–623, New York, NY, USA. Association for Computing Machinery.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_.
  * Calderon et al. (2023) Nitay Calderon, Subhabrata Mukherjee, Roi Reichart, and Amir Kantor. 2023.  A systematic study of knowledge distillation for natural language generation with pseudo-target training.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 14632–14659, Toronto, Canada. Association for Computational Linguistics.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 8440–8451, Online. Association for Computational Linguistics.
  * Färber and Leisinger (2021) Michael Färber and Ann-Kathrin Leisinger. 2021.  Recommending datasets for scientific problem descriptions.  In _CIKM_ , pages 3014–3018.
  * Floridi and Chiriatti (2020) Luciano Floridi and Massimo Chiriatti. 2020.  Gpt-3: Its nature, scope, limits, and consequences.  _Minds and Machines_ , 30:681–694.
  * Gao et al. (2023) Luyu Gao, Xueguang Ma, Jimmy Lin, and Jamie Callan. 2023.  Precise zero-shot dense retrieval without relevance labels.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1762–1777, Toronto, Canada. Association for Computational Linguistics.
  * Gudibande et al. (2023) Arnav Gudibande, Eric Wallace, Charles Burton Snell, Xinyang Geng, Hao Liu, P. Abbeel, Sergey Levine, and Dawn Song. 2023.  The false promise of imitating proprietary llms.  _ArXiv_ , abs/2305.15717.
  * He et al. (2023) Xingwei He, Zheng-Wen Lin, Yeyun Gong, Alex Jin, Hang Zhang, Chen Lin, Jian Jiao, Siu Ming Yiu, Nan Duan, and Weizhu Chen. 2023.  Annollm: Making large language models to be better crowdsourced annotators.  _ArXiv_ , abs/2303.16854.
  * Hill et al. (2016) Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016.  The goldilocks principle: Reading children’s books with explicit memory representations.
  * Jiang et al. (2022) Ellen Jiang, Kristen Olson, Edwin Toh, Alejandra Molina, Aaron Donsbach, Michael Terry, and Carrie J Cai. 2022.  Promptmaker: Prompt-based prototyping with large language models.  In _CHI Conference on Human Factors in Computing Systems Extended Abstracts_ , pages 1–8.
  * Kendall (1938) Maurice G Kendall. 1938.  A new measure of rank correlation.  _Biometrika_ , 30(1/2):81–93.
  * Koh (2020) Jing Yu Koh. 2020.  Model zoo.  _URL http://modelzoo. co_.
  * Lhoest et al. (2021) Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Patil, Julien Chaumond, Mariama Drame, Julien Plu, Lewis Tunstall, Joe Davison, Mario Šaško, Gunjan Chhablani, Bhavitvya Malik, Simon Brandeis, Teven Le Scao, Victor Sanh, Canwen Xu, Nicolas Patry, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Clément Delangue, Théo Matussière, Lysandre Debut, Stas Bekman, Pierric Cistac, Thibault Goehringer, Victor Mustar, François Lagunas, Alexander Rush, and Thomas Wolf. 2021.  Datasets: A community library for natural language processing.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 175–184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Liu et al. (2023a) Nelson F. Liu, Tony Lee, Robin Jia, and Percy Liang. 2023a.  Do question answering modeling improvements hold across benchmarks? In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 13186–13218, Toronto, Canada. Association for Computational Linguistics.
  * Liu et al. (2023b) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023b.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _ACM Comput. Surv._ , 55(9).
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Rethinking the role of demonstrations: What makes in-context learning work? In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 11048–11064, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Neubig and He (2023) Graham Neubig and Zhiwei He. 2023.  Zeno GPT Machine Translation Report.
  * Paleyes et al. (2022) Andrei Paleyes, Raoul-Gabriel Urma, and Neil D Lawrence. 2022.  Challenges in deploying machine learning: a survey of case studies.  _ACM Computing Surveys_ , 55(6):1–29.
  * Park et al. (2022) Gunho Park, Baeseong Park, Minsub Kim, Sungjae Lee, Jeonghoon Kim, Beomseok Kwon, Se Jung Kwon, Byeongwook Kim, Youngjoo Lee, and Dongsoo Lee. 2022.  Lut-gemm: Quantized matrix multiplication based on luts for efficient inference in large-scale generative language models.
  * Popović (2015) Maja Popović. 2015.  chrF: character n-gram F-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(1).
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Reynolds and McDonell (2021) Laria Reynolds and Kyle McDonell. 2021.  Prompt programming for large language models: Beyond the few-shot paradigm.  In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_ , CHI EA ’21, New York, NY, USA. Association for Computing Machinery.
  * Robertson et al. (1995) Stephen E. Robertson, Steve Walker, Micheline Hancock-Beaulieu, Mike Gatford, and A. Payne. 1995.  Okapi at trec-4.  In _Text Retrieval Conference_.
  * Rogers et al. (2023) Anna Rogers, Niranjan Balasubramanian, Leon Derczynski, Jesse Dodge, Alexander Koller, Sasha Luccioni, Maarten Sap, Roy Schwartz, Noah A Smith, and Emma Strubell. 2023.  Closed ai models make bad baselines.
  * Sezgin et al. (2022) Emre Sezgin, Joseph Sirrianni, and Simon L. Linwood. 2022.  Operationalizing and implementing pretrained large ai linguistic models in the united states healthcare system: An outlook of gpt-3 as a service.  _JMIR Medical Informatics_ , 10(2).
  * Tiedemann and Thottingal (2020) Jörg Tiedemann and Santhosh Thottingal. 2020.  OPUS-MT — Building open translation services for the World.  In _Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation (EAMT)_ , Lisbon, Portugal.
  * Viswanathan et al. (2023) Vijay Viswanathan, Luyu Gao, Tongshuang Wu, Pengfei Liu, and Graham Neubig. 2023\.  DataFinder: Scientific dataset recommendation from natural language descriptions.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 10288–10303, Toronto, Canada. Association for Computational Linguistics.
  * Wang et al. (2021a) Shuohang Wang, Yang Liu, Yichong Xu, Chenguang Zhu, and Michael Zeng. 2021a.  Want to reduce labeling cost? GPT-3 can help.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4195–4205, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Wang et al. (2022) Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2022.  Self-consistency improves chain of thought reasoning in language models.
  * Wang et al. (2021b) Yue Wang, Weishi Wang, Shafiq Joty, and Steven C.H. Hoi. 2021b.  Codet5: Identifier-aware unified pre-trained encoder-decoder models for code understanding and generation.  In _EMNLP_.
  * Wang et al. (2023) Zhiruo Wang, Grace Cuenca, Shuyan Zhou, Frank F. Xu, and Graham Neubig. 2023.  MCoNaLa: A benchmark for code generation from multiple natural languages.  In _Findings of the Association for Computational Linguistics: EACL 2023_ , pages 265–273, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Widder et al. (2022) David Gray Widder, Dawn Nafus, Laura Dabbish, and James Herbsleb. 2022.  Limits and possibilities for “ethical ai” in open source: A study of deepfakes.  In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’22, page 2035–2046, New York, NY, USA. Association for Computing Machinery.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wu et al. (2023) Sherry Wu, Hua Shen, Daniel S Weld, Jeffrey Heer, and Marco Tulio Ribeiro. 2023\.  Scattershot: Interactive in-context example curation for text transformation.  In _Proceedings of the 28th International Conference on Intelligent User Interfaces_ , IUI ’23, page 353–367, New York, NY, USA. Association for Computing Machinery.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019\.  Bertscore: Evaluating text generation with bert.