# What Makes ImageNet Look Unlike LAION

Ali Shirali University of California, Berkeley Moritz Hardt Max Planck
Institute for Intelligent Systems, Tübingen and Tübingen AI Center

###### Abstract

ImageNet was famously created from Flickr image search results. What if we
recreated ImageNet instead by searching the massive LAION dataset based on
image captions alone? In this work, we carry out this counterfactual
investigation. We find that the resulting ImageNet recreation, which we call
LAIONet, looks distinctly unlike the original. Specifically, the intra-class
similarity of images in the original ImageNet is dramatically higher than it
is for LAIONet. Consequently, models trained on ImageNet perform significantly
worse on LAIONet. We propose a rigorous explanation for the discrepancy in
terms of a subtle, yet important, difference in two plausible causal data-
generating processes for the respective datasets, that we support with
systematic experimentation. In a nutshell, searching based on an image caption
alone creates an information bottleneck that mitigates the selection bias
otherwise present in image-based filtering. Our explanation formalizes a long-
held intuition in the community that ImageNet images are stereotypical,
unnatural, and overly simple representations of the class category. At the
same time, it provides a simple and actionable takeaway for future dataset
creation efforts.

##  1 Introduction

For nearly a decade, ImageNet [Deng et al., 2009] was the focal benchmark for
much of computer vision and deep learning. Created from image web search
results and human filtering, ImageNet contributed curated images suitable for
supervised learning at the time. In recent years, however, the community has
seen a new generation of models trained on massive amounts of noisy image-text
data gathered from the web with minimal curation. Available to the academic
public is the massive scale LAION dataset, in two versions, featuring $400$
million [Schuhmann et al., 2021] and $5$ billion [Schuhmann et al., 2022]
image-text pairs, crawled from the web, and filtered by the OpenAI CLIP model
[Radford et al., 2021] rather than human annotators.

At the outset, LAION works much like web image search. We can specify a query
and retrieve images with high similarity, according to CLIP, between the query
and the text surrounding the image on the web site that it was crawled from.
We can therefore search LAION for each of the $1000$ categories in the
ImageNet ILSVRC-2012 dataset111Unless otherwise stated, by ImageNet we mean
the ImageNet ILSVRC-2012 dataset. and retrieve images corresponding to each of
the classes. This process is much like the first step of creating ImageNet
from Flickr search results, except that LAION replaces Flickr, but either way,
both are based on web crawls. Where the creators of ImageNet hired human
annotators to filter images, we can use CLIP similarity to ensure that the
resulting images have high fidelity to the class category.

We might expect that for a suitably chosen CLIP similarity threshold, the
resulting dataset would bear resemblance to the original ImageNet. However, we
demonstrate that this is anything but the case. The dataset, so created from
LAION, very much looks unlike ImageNet. And we explain _why_. This
explanation, although subtle, reveals a fundamental fact about the difference
between ImageNet and LAION that has consequences for understanding dataset
creation at large.

###  1.1 Our contributions

We introduce a new research artifact, called _LAIONet_ , that aims at a
recreation of ImageNet (ILSVRC-2012) on the basis of LAION. We start from
LAION-400M, a collection of 400M image-text pairs extracted from web pages in
Common Crawl (commoncrawl.org) between 2014 and 2021. The relevance of images
and their corresponding texts was quality-controlled with OpenAI CLIP model
[Radford et al., 2021], excluding images with a cosine similarity of image and
text embeddings less than $0.3$.

#### Creation of LAIONet.

We create LAIONet solely on the basis of text-based selection. We require the
exact “lemmas” (terms) in a so-called “synset” of an ImageNet category to
appear in the text corresponding to an image. Moreover, we require a high CLIP
similarity between the text and the synset name and definition. These
selection criteria are conservative in that they tend toward images that are
easy to classify. Indeed, from the CLIP point of view, there is no evidence
that LAIONet images are harder to classify than ImageNet. In fact, CLIP zero-
shot accuracy is $5$ to $10$ percentage points higher on LAIONet than on
ImageNet. Moreover, CLIP similarity of the images to their intended synsets
does not significantly differ between LAIONet and ImageNet.

#### Contrasting LAIONet and ImageNet.

To begin to understand the differences between LAIONet and ImageNet, we
evaluate a slew of Imagenet models on LAIONet. As we show, the accuracy of
models trained on ImageNet drops by $10$ percentage points when evaluated on
LAIONet (Figure 1). In calculating accuracy, we weight classes uniformly as is
done in ImageNet. Models pre-trained on the full ImageNet with $22$k classes
prove to be slightly more robust. When classes are weighted based on the
frequency of each class in LAIONet, accuracy drops by another $5$ to $10$
percentage points.

Figure 1: Accuracy of ImageNet-trained models when evaluated on ImageNet
validation set vs. LAIONet. Three types of models are distinguished based on
whether they are pre-trained on ImageNet-$22$k and whether they are fine-tuned
on ImageNet-$1$k. Accuracy is defined as the average of the recalls calculated
for each class that is present in LAIONet.

Drops in accuracy, such as these, are a well-documented phenomenon in machine
learning at this point. In this work, we go a step further by providing a
substantive explanation for the difference between LAIONet and ImageNet.

#### Diagnosing the difference.

In a first step, we observe that the intra-class similarity, measured as the
pairwise similarity of the images within a class, is lower for LAIONet than
for ImageNet. In other words, LAIONet images are more diverse in each class.
The recall of the models is also lower in the classes with lower intra-class
similarity. Hence, lower intra-class similarity gives a concrete reason for
why the accuracy of ImageNet models drops on LAIONet. But why does LAIONet
have lower intra-class similarity in the first place?

We answer this question in terms of two plausible causal graphs (Figure 2) for
the respective data-generating processes. Both graphs are based on the
standard anti-causal representation of classification problems [Schölkopf et
al., 2012], whereby for each category $Y$ there is a mechanism to generate
data (here, image $X$ and text $T$) given $Y$. However, the two graphs differ
in one important aspect.

(a) LAIONet data collection

(b) ImageNet data collection

Figure 2: The suggested underlying mechanism of data generation and selection
in LAIONet and ImageNet. Class $Y$, text description $T$, image $X$, selection
$S$ or $S^{\prime}$.

In the case of LAIONet (Figure 2(a)), selection is based on text alone. The
causal graph has the important property that the distribution of the image is
independent of the selection decision conditional on the text. In other words
the text serves as an information bottleneck between the selection mechanism
and the image. Choosing an image reveals nothing more about the image than
what can be learned from its textual representation. This powerful conditional
independence property limits how much selection can bias the distribution of
the image. In contrast, in the case of ImageNet (Figure 2(b)), there is a link
from the image to the selection decision. For example, this link exists when
human annotators see the full image and decide to select or discard an image.
The existence of this link is what can strongly bias the distribution of the
image conditional on selection. It is this selection bias that is visible in
the higher intra-class similarity.

Our case hinges on the existence and strength of the image-to-selection link
in the causal graph for ImageNet. We therefore give evidence for both through
three complementary arguments.

  * •

We can weaken the image-to-selection link by considering ImageNet instances of
different _selection frequencies._ The selection frequency describes the rate
at which Amazon MTurk workers selected a candidate image into the dataset
within a target class. This allows us to modulate the strength of the image-
to-selection link. Indeed, we find that as we decrease selection frequency,
the resulting images come closer to LAIONet. This confirms our hypothesis that
the link from the image to the selection is contributing significantly to the
divergence between LAIONet and ImageNet.

  * •

We show that text alone cannot explain why an image was selected into
ImageNet. The ImageNet-Captions dataset [Fang et al., 2022b] has restored the
captions for one-third of the original ImageNet images. If the text was the
only factor in determining the relevance to a synset, it should explain why
the images in ImageNet-Captions are selected. Looking at the similarity
between texts and their synsets, a majority of text-synset pairs exhibit high
similarity, but the distribution has a heavy tail and there are instances with
low similarity. For pairs with low similarity, there are often many other
synsets more similar to the text. This makes these instances unlikely to have
been selected solely based on their text.

  * •

We search LAION for the texts most similar to the texts from the ImageNet-
Captions dataset. The resulting images show significantly higher variability
(in other words, lower intra-class similarity) than ImageNet. This suggests
that another mechanism, namely the image-to-selection link, must have been at
play. We note the limitation that texts in the ImageNet-Captions dataset may
not completely include the text available at the time of ImageNet creation.
Second, for many cases, we were unable to find great matches for the ImageNet
texts in LAION-$400$M. Scaling our analysis to LAION-$5$B might help here.

In conclusion, we argue that the image-to-selection mechanism was
significantly at play in the creation of ImageNet. It is this mechanism that
makes ImageNet look unlike LAION. This insight has direct prescriptive value
for dataset creation efforts in general. When creating a new dataset, we
should select candidate instances on the basis of an information bottleneck. A
succinct text caption, for example, generally carries much less information
than the entire image. Selecting on the basis of the text caption, therefore,
retains much of the entropy present in the image distribution.

All code is available at: https://github.com/alishiraliGit/eval-on-laion

###  1.2 Related work

Recreating an ImageNet test set, called ImagenetV2, although with a different
motivation, was the subject of the seminal paper by Recht, Roelofs, Schmidt,
and Shankar [2019]. Engstrom et al. [2020] argue that there is a subtlety in
thresholding empirical estimates of the true underlying selection frequency of
an image. Our argument, however, does not rely on any specific threshold of
the selection frequency. We only need to observe what happens as we vary it
from small to large. In contrast to ImageNetV2, our goal is not to recreate
ImageNet as closely as possible. Rather it’s the differences between ImageNet
and LAION that are the focus of our investigation.

Many other works have modified ImageNet for a variety of reasons. Geirhos et
al. [2019] created a stylized version of ImageNet to reduce the reliance of
the trained model on texture. Xiao et al. [2021] disentangled the foreground
and background of ImageNet images to show the tendency of the models to rely
on the background. Li et al. [2023] proposed ImageNet-W test set by inserting
a transparent watermark into the images of ImageNet validation set, revealing
the reliance of the models on watermarks. ImageNet undergoes ongoing
augmentation over time. For example, the ImageNet-Captions [Fang et al.,
2022b] project has restored the captions of about one-third of original
ImageNet images from Flickr. ImageNet-X [Idrissi et al., 2023] provides a set
of human annotations pinpointing $16$ failure types for ImageNet such as pose,
background, or lighting. The peculiarities of ImageNet have been the subject
of multiple studies. For example, Huh et al. [2016] found the large size and
many classes, including very similar classes, do not affect the successful
transfer performance of ImageNet-trained features.

On the side of LAION, researchers are keenly interested in understanding the
strong zero-shot accuracy of contrastive language image models, see, e.g.,
Vogel et al. [2022]. Fang et al. [2022a] found none of the large training set
size, language supervision, and contrastive loss function determines this
robustness and a more diverse training distribution should be the main cause.
Our work demystifies this distributional advantage by contrasting ImageNet and
LAION. Nguyen et al. [2022] compared various large image-text datasets
differing in the creation process and found the robustness induced by each
varies widely in different aspects, suggesting further studies of the role of
dataset design. Our work highlights an important mechanism at play in dataset
design that can move the dataset further away from a natural distribution.

##  2 LAIONet: An ImageNet out of LAION

Our starting point is to create an ImageNet-like dataset from LAION. This
dataset is a research artifact intended to highlight the differences between
LAION and ImageNet. Our goal is not to provide a new benchmark, or a new
training set.

To start, recall that every ImageNet class corresponds to a WordNet [Miller,
1998] synset. Every synset consists of some so-called _lemmas_. Synsets also
come with a short definition known as gloss. We label an instance from LAION
with a WordNet synset if 1) at least one lemma from the synset exists in the
text of the instance, and 2) the text of the instance is sufficiently similar
to the name and definition of the synset.

Out of LAION $400$M samples, $21$M of them passed the first condition. The
second condition ensures the lemma as found in the LAION sample has the
intended meaning. To quantify the similarity of the LAION text and a synset,
we first create a textual representation for the synset by concatenating its
name and definition (to be called the synset text). We then calculate the
embedding vectors for both the synset text and LAION text using CLIP and
compute their cosine similarity. Figure 3(a) illustrates the distribution of
these similarities. We drop the samples below a certain level of similarity
and next explain how to determine this threshold.

In general, a high value for textual similarity ensures the LAION text is
describing the same object as the synset. But as Figure 3(b) shows, we cannot
set a very high similarity threshold since the extracted dataset will lose its
coverage over the ImageNet’s $1$k classes. We found the threshold of $0.82$
the highest reasonable choice as it allowed for covering most classes while
going beyond it sharply reduced the number of covered classes (Figure 3(b))
with no significant reduction in the dataset size (Figure 3(c)). Furthermore,
in Section 4 (Figure 9(b)), using the restored captions of ImageNet, we show a
textual similarity of above $0.7$ is sufficient to ensure a sample belongs
uniquely to the synset. This finding supports the conservative nature of our
threshold at $0.82$. Look at Appendix A for an example of when the second step
of filtering is necessary and why our chosen threshold is conservative.

(a)

(b)

(c)

Figure 3: Filtering LAION samples based on their textual similarity to the
candidate synsets. The dashed line shows the chosen threshold. (a) The overall
distribution of the similarities prior to the second step of filtering. (b and
c) The number of ImageNet classes covered by the dataset and the size of the
dataset for different levels of similarity threshold.

From the samples that passed the two conditions, we dropped ones with more
than one label to simplify future evaluation on the dataset. We also dropped
images tagged as not-safe-for-work. The final dataset, which we call _LAIONet_
, consists of $880$k samples from $918$ ImageNet classes, sufficiently large
for fine-grained evaluation purposes at statistical significance. Unlike
ImageNet which provides about the same number of images per class, the number
of samples in LAIONet reflects the natural distribution of each class. As
Figure 4 shows, there is large variation in the relative frequency of the
classes in LAIONet. Later we will use these frequencies to compare the
performance of models in frequent and infrequent classes.

Figure 4: Relative frequencies of different classes in LAIONet sorted in
descending order for the $500$ most frequent classes. Some class names shown.
Red line shows uniform weight.

Are LAIONet images harder to classify? To find out, we compare CLIP zero-shot
accuracy on LAIONet and ImageNet. For every image, we predict the label of the
image based on what synset has the highest cosine similarity between the image
embedding and the synset text embedding. To make accuracy estimates on LAIONet
comparable with ImageNet, we calculate accuracy as the average recall across
classes, similar to the setup of ImageNet validation set with $50$ images per
class. We found CLIP has a zero-shot top $1$ and top $5$ accuracy of $66.5\%$
and $90.5\%$ on LAIONet while these numbers are $56.9\%$ and $84.5\%$
respectively, for ImageNet. Hence, at least from the CLIP view, LAIONet images
are not harder to classify. But note that the CLIP text embeddings are jointly
trained with the image embeddings, possibly giving CLIP an advantage on
LAIONet. Appendix B provides a more detailed analysis by assessing CLIP zero-
shot recall for each class, revealing that there are a greater number of
classes in which CLIP performs better on LAIONet compared to ImageNet. In
addition, as a more direct assessment of the degree of difficulty involved in
identifying an intended object in LAIONet, we directly calculated the
similarity of an image to its associated synset using CLIP image and text
embeddings. We found no evidence that image-to-synset similarity is higher in
ImageNet. Look at Appendix C for the details of the analysis.

##  3 LAIONet versus ImageNet

We begin to understand the differences between the two datasets by looking at
the accuracy of various ImageNet classifiers on LAIONet. After observing a
significant accuracy drop, we consider the disparity in intra-class diversity
as a possible explanation.

###  3.1 Comparing accuracy

We consider four model families: ResNet [He et al., 2016], Vision Transformers
(ViT) [Dosovitskiy et al., 2021], modernized ConveNet (ConvNeXt) [Liu et al.,
2022], and Bidirectional Encoder representation from Image Transformers (BEiT)
[Bao et al., 2022]. All models are trained on ImageNet without extra training
data. We use various versions of each model in terms of the size (small, base,
large, etc.), image resolution ($224$x$224$ or $384$x$384$), patch resolution
($16$x$16$ or $32$x$32$), and whether models are pre-trained on the complete
ImageNet with $22$k classes or not. All models come from HuggingFace
(huggingface.co) checkpoints. On a single GPU, the results can be reproduced
in less than a day using the accompanying code.

We first compare the (equally weighted) accuracy defined by the average of
recalls across the classes covered by LAIONet. Figure 1 compares the top $1$
and top $5$ accuracy on ImageNet and LAIONet. In most of the highly accurate
models, accuracy drops by $10$ percentage points when estimated on LAIONet
with models pre-trained on ImageNet-$22$k showing slightly more robustness.

Next, we weigh each class according to how frequent it is in LAIONet. We use
these relative frequencies to weight recalls in each class and obtain a LAION-
weighted accuracy. Figure 5 compares LAION-weighted and equally-weighted
accuracy evaluated on LAIONet. The LAION-weighted accuracy is consistently
lower by $5$ to $10$ percentage points. This suggests that ImageNet-trained
models are performing worse when the class is describing a more common object.
For further direct evidence of this claim, look at Appendix D.1.

Figure 5: A LAION-weighted accuracy is calculated according to the relative
frequency of the classes in LAIONet and compared to the accuracy with equally
weighted classes.

###  3.2 Comparing intra-class similarity

While LAIONet images are in a precise sense not more difficult than ImageNet
(Section 2), there is another factor that can explain the accuracy drop: the
intra-class similarity of images. We define this similarity as the pairwise
similarity of the images from the same class, measured by the cosine
similarity of their CLIP image embeddings. The lower these similarity values,
the more diverse the images from that class. Figure 6(a) shows the
distribution of intra-class similarities aggregated over all the classes. To
make the distributions comparable, we sampled (with replacement) the
similarities from LAIONet to match ImageNet. The left tail of the LAIONet
intra-class similarity distribution makes it clear that LAIONet overall
provides a more diverse set of images. To observe the effect in greater
detail, for each class, Figure 6(b) shows the average intra-class similarity
of LAIONet images subtracted by the average intra-class similarity of ImageNet
images from the same class. In almost two-thirds of the classes, LAIONet has
significantly lower intra-class similarity. This provides further evidence
that LAIONet images exhibit greater variability within each class.

(a) Dist. of aggregated similarities

(b) Comparison across classes

Figure 6: Comparing the intra-class similarity of LAIONet and ImageNet. (a) In
each class, pairwise similarities of LAIONet images are sampled to match
ImageNet in number. All the classes combined, the distribution of intra-class
similarity is depicted. (b) For each class, the average intra-class similarity
of ImageNet images was subtracted from the same value in LAIONet. The blue and
red curves show upper and lower $95\%$ confidence intervals. All values are
sorted ascendingly.

Looking for the lost accuracy, Figure 7 demonstrates in the classes where
LAIONet is more diverse than ImageNet, indicated by lower intra-class
similarity, models perform poorly as measured by their recall rates. Together
with our observation that LAIONet exhibits lower overall intra-class
similarity, these findings support our argument that the difference in intra-
class similarity is a significant factor contributing to the decrease in
equally-weighted accuracy. In Appendix D.2, we also show a weak correlation
between the class frequency and the intra-class similarity, partially
explaining why models have low LAION-weighted accuracy.

(a) Recall@1

(b) Recall@5

Figure 7: The plot compares the recall on LAIONet for each class with the
disparity in intra-class similarity between LAIONet and ImageNet. This
disparity (horizontal axis) is measured by subtracting the class-average
intra-class similarity in ImageNet from that in LAIONet. Four exemplary models
are shown, where two of them are pretrained on ImageNet-$21$k (yellow) and two
of them are not (red) and the trend is consistent for all of them.

##  4 Diagnosing ImageNet

As is standard modeling practice, we think of a data generating process that
for a given class $Y$ generates a pair of image $X$ and text $T$. Ideally,
when we search for images of a particular class $y$, we would like to draw
samples from distribution $p(X|Y=y)$. Unless we have access to the generative
process or we have a completely random set of images all correctly labeled,
drawing samples directly from $p(X|Y=y)$, however, will not be possible. In
particular, none of these options are available when researchers collect a new
dataset. Instead, researchers have to define a selection mechanism $S$ for
choosing images. What we observe is the conditional distribution of $X$ given
$S.$

In creating LAIONet, we relied on texts to select the samples. LAIONet images
follow $p(X|S=1)$, where $S=1$ if $T$ is sufficiently similar to $Y$ (Figure
2(a)). With our conservative selection criteria, we can assume every $T$
passed our similarity threshold is generated from the intended $Y=y$.
Therefore, $p(X|S=1)=p(X|S=1,Y=y)$. Generally, an image carries much more
information than the text. So, for the images of a certain class, conditioning
on the text alone should not alter the distribution significantly. Intuitively
speaking, $p(X|Y=y,T=t)\approx p(X|Y=y)$. In our setting, a weaker
independence is sufficient to show LAIONet images follow the desired
distribution. Even if information from $X$ beyond $Y$ is present in $T$, since
we deliberately refrained from searching for visual descriptions in the text,
we expect $S$ to be independent from $X$ for a given $Y=y$. Therefore, we have
reason to hope $p(X|S=1)\approx p(X|S=1,Y=y)\approx p(X|Y=y)$.

In general, a selection $S^{\prime}$ can rely on both text and image (Figure
2(b)) directly. In this case, the distribution of observed images
$p(X|S^{\prime}=1)$ can be far from the desired distribution $p(X|Y=y)$. We
believe this has happened in the collection of ImageNet, primarily through
human annotators examining and acting on images. Incorporation of visual
features at the side of the search engine provider is another plausible
mechanism. While we may not be able to pinpoint the exact mechanism at play,
we next demonstrate through three experiments that there is a strong link
between the image $X$ and the selection criterion $S^{\prime}$ in the creation
of ImageNet.

###  4.1 Weakening the image-to-selection link makes ImageNet more like
LAIONet

Image annotation is one clear mechanism by which the image $X$ influences
selection $S^{\prime}$. Changing the strictness of annotation allows us to
modulate the strength of this mechanism and measure its effect. This
experiment is possible due to the availability of ImageNetV2 [Recht et al.,
2019] that comes with three different versions. The three versions of
ImageNetV2, called a, b, and c, differ in the level of agreement among
annotators. More precisely, each image comes with a _MTurk selection
frequency_ which is what fraction of MTurk workers selected the image to be
from the target class. ImageNetV2 versions a, b, and c have an average MTurk
selection frequency of $0.85$, $0.73$, and $0.93$, respectively. Note that
version b has the lowest and version c has the highest selection frequency.

We first observe that allowing for more disagreement among annotators results
in the inclusion of more diverse images. Figure 8(a) shows the distribution of
intra-class similarity for ImageNetV2 versions b and c. One can see that in
version b with the lowest average MTurk selection frequency, the intra-class
similarity is shifted toward lower values.

We further show as the average MTurk selection frequency increases, ImageNetV2
becomes more similar to ImageNet and less similar to LAIONet. In this regard,
to compare two datasets, we count the number of classes in which the first
dataset has significantly lower intra-class similarity than the second
dataset, and vice versa. Figure 8(b) compares LAIONet and three versions of
ImageNetV2. As the figure suggests, LAIONet and ImageNetV2 are quite similar
when the average MTurk selection frequency is low (ImageNetV2 version b) but
as the MTurk selection frequency increases, ImageNetV2 shows higher intra-
class similarity than LAIONet. At the same time, Figure 8(c) shows ImageNetV2
becomes more similar to ImageNet as we increase the MTurk selection frequency.
These observations show the impact the image has on the selection,
particularly during annotation, is significant and can partially explain the
divergence between LAIONet and ImageNet.

(a) Dist. of aggregated similarities

(b) LAIONet vs. ImageNetV2

(c) ImageNet vs. ImageNetV2

Figure 8: The effect of MTurk selection frequency on intra-class similarity.
(a) The distribution of intra-class similarity aggregated over all classes is
compared for ImageNetV2 versions b and c. The version b has a lower average
MTurk selection frequency and shows a lower intra-class similarity as well.
(b) LAIONet is compared to the three versions of ImageNetV2. The vertical axis
displays the proportion of classes in which one dataset exhibits significantly
lower intra-class similarity than the other dataset. We determined
significance using $95\%$ confidence intervals. Blue curve: LAIONet has lower
intra-class similarity. Green curve: ImageNetV2 has lower intra-class
similarity. (c) ImageNet and ImageNetV2 are compared. Red curve: ImageNet has
lower intra-class similarity. Green curve: ImageNetV2 has lower intra-class
similarity.

###  4.2 Text alone cannot explain why an image is selected into ImageNet

ImageNet-Captions [Fang et al., 2022b] is a subset of ImageNet-$1$k training
data with restored title, description, and tags from Flickr. We assume the
samples in ImageNet-Captions are a random subset of the original ImageNet and
the captions are accurately restored. If there was no link $X\rightarrow
S^{\prime}$, the accompanying caption of an image in ImageNet-Captions should
be able to explain why this image is selected. We follow Fang et al. [2022b]
and define the text as the title, description, and tags concatenated. Figure
9(a) illustrates the similarity between the texts and their respective synsets
using CLIP text embeddings. Although most of the texts have a high similarity
of $0.6$ or above to their synsets, the distribution has a heavy left tail
with very low similarities. The fact that a text has low similarity to the
intended synset does not necessarily mean it could not be chosen by the search
engine. We further show many of the texts that have low similarity to the
intended synsets actually have high similarity to numerous other synsets,
making them less likely to have appeared for the intended meaning. For every
text, we find the similarity to all synsets, i.e. the similarity to their
names and definitions, and count the proportion of unintended synsets (false
classes) which are more similar to the text than the intended synset. A low
value for this proportion shows the text well represents its intended synset
whereas a significant non-zero value indicates that there are considerable
other synsets that are more strongly present in the text. As Figure 9(b)
demonstrates, for a text with low similarity to its synset there are on
average $5\%$ (equivalently, $200$) or even more other synsets more similar to
the text. These observations show that at least based on the restored texts in
ImageNet-Captions, the text alone cannot fully explain why an image is
selected and another mechanism should have been at play.

(a)

(b)

Figure 9: (a) The distribution of the text-to-synset similarity. (b) For every
bin of text-to-synset similarity, the average proportion of unintended classes
which are more similar to the text than the intended class is depicted in
black.

###  4.3 ImageNet, had it been created solely searching texts, does not
resemble current ImageNet

If the link from $X$ to $S^{\prime}$ did not exist, regardless of how the
selection algorithms works, $p(X|T=t)$ would look similar in both graphs of
Figure 2. To test this hypothesis, we extract a new dataset from LAION. For
every image in ImageNet with corresponding text $T=t$ in ImageNet-Captions, we
find the LAION sample with the most similar text to $t$ measured by CLIP
textual similarity. We only keep a LAION sample if the similarity is above
$0.7$. This choice ensures the two texts are sufficiently similar as we can
consider them roughly the same while the dataset covers more than $95\%$ of
the ImageNet classes. We provide further analysis of this choice in Appendix
E.

As Figure 10(a) suggests, images in the new dataset have a significantly lower
intra-class similarity. In other words, the natural images extracted from
LAION based on ImageNet texts show a higher intra-class variation than the
original ImageNet. Looking at each class separately, Figure 10(b) shows in
almost $70\%$ of the classes, the images from the new dataset are
significantly more diverse (have lower intra-class similarity). These
observations reject the hypothesis that the graphs of Figure 2 have the same
structure and show a potential leak from the image to the selection.

(a) Dist. of aggregated similarities

(b) Comparison across classes

Figure 10: Comparing the intra-class similarity of the new dataset and
ImageNet. The new dataset is obtained by selecting LAION examples with the
most similar texts to the texts in ImageNet-Captions. (a) Distribution of
intra-class similarity aggregated across all classes. In each class, pairwise
similarities of the images in the new dataset are sampled to match ImageNet in
number to make the distributions comparable. (b) For each class, the average
of the intra-class similarity of the images in the new dataset minus the
corresponding value in ImageNet is plotted in black. The upper and lower
$95\%$ confidence bounds are depicted in blue and red. All values are sorted
ascendingly.

##  5 Conclusion

In conclusion, we argue that the image-to-selection mechanism played a
significant role in the creation of ImageNet, distinguishing it from LAION. We
demonstrated this through three experiments. First, we modulated the
speculated link from image to selection, showing the significant contribution
this mechanism has in reducing the diversity of the selected images. The next
two experiments rejected the hypothesis that image plays no or negligible role
in the selection by showing ImageNet captions cannot solely explain the
selection.

This insight carries valuable implications for dataset creation efforts in
general. When developing a new dataset, we advise selecting candidate
instances based on an information bottleneck, like a succinct textual
description of the instance, rather than the full instance. This will mitigate
the selection bias that may otherwise distort the distribution of data
conditional on selection.

## Acknowledgments

This project originated from formative research conversations with Rediet
Abebe. We thank Ludwig Schmidt for valuable comments and suggestions.

## References

  * Bao et al. [2022] H. Bao, L. Dong, S. Piao, and F. Wei.  BEit: BERT pre-training of image transformers.  In _International Conference on Learning Representations_ , 2022.  URL https://openreview.net/forum?id=p-BhZSz59o4.
  * Deng et al. [2009] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In _2009 IEEE conference on computer vision and pattern recognition_ , pages 248–255. Ieee, 2009.
  * Dosovitskiy et al. [2021] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=YicbFdNTTy.
  * Engstrom et al. [2020] L. Engstrom, A. Ilyas, S. Santurkar, D. Tsipras, J. Steinhardt, and A. Madry.  Identifying statistical bias in dataset replication.  In _International Conference on Machine Learning_ , pages 2922–2932. PMLR, 2020.
  * Fang et al. [2022a] A. Fang, G. Ilharco, M. Wortsman, Y. Wan, V. Shankar, A. Dave, and L. Schmidt.  Data determines distributional robustness in contrastive language image pre-training (clip).  In _International Conference on Machine Learning_ , pages 6216–6234. PMLR, 2022a.
  * Fang et al. [2022b] A. Fang, G. Ilharco, M. Wortsman, Y. Wan, V. Shankar, A. Dave, and L. Schmidt.  Data determines distributional robustness in contrastive language image pre-training (clip).  In _International Conference on Machine Learning_ , pages 6216–6234. PMLR, 2022b.
  * Geirhos et al. [2019] R. Geirhos, P. Rubisch, C. Michaelis, M. Bethge, F. A. Wichmann, and W. Brendel.  Imagenet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness.  In _International Conference on Learning Representations_ , 2019.  URL https://openreview.net/forum?id=Bygh9j09KX.
  * He et al. [2016] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_ , pages 770–778, 2016.
  * Huh et al. [2016] M. Huh, P. Agrawal, and A. A. Efros.  What makes imagenet good for transfer learning?  _arXiv preprint arXiv:1608.08614_ , 2016.
  * Idrissi et al. [2023] B. Y. Idrissi, D. Bouchacourt, R. Balestriero, I. Evtimov, C. Hazirbas, N. Ballas, P. Vincent, M. Drozdzal, D. Lopez-Paz, and M. Ibrahim.  Imagenet-x: Understanding model mistakes with factor of variation annotations.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=HXz7Vcm3VgM.
  * Li et al. [2023] Z. Li, I. Evtimov, A. Gordo, C. Hazirbas, T. Hassner, C. C. Ferrer, C. Xu, and M. Ibrahim.  A whac-a-mole dilemma: Shortcuts come in multiples where mitigating one amplifies others.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 20071–20082, 2023.
  * Liu et al. [2022] Z. Liu, H. Mao, C.-Y. Wu, C. Feichtenhofer, T. Darrell, and S. Xie.  A convnet for the 2020s.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 11976–11986, 2022.
  * Miller [1998] G. A. Miller.  _WordNet: An electronic lexical database_.  MIT press, 1998.
  * Nguyen et al. [2022] T. Nguyen, G. Ilharco, M. Wortsman, S. Oh, and L. Schmidt.  Quality not quantity: On the interaction between dataset design and robustness of CLIP.  In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho, editors, _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=LTCBavFWp5C.
  * Radford et al. [2021] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al.  Learning transferable visual models from natural language supervision.  In _International conference on machine learning_ , pages 8748–8763. PMLR, 2021.
  * Recht et al. [2019] B. Recht, R. Roelofs, L. Schmidt, and V. Shankar.  Do imagenet classifiers generalize to imagenet?  In _International conference on machine learning_ , pages 5389–5400. PMLR, 2019.
  * Schölkopf et al. [2012] B. Schölkopf, D. Janzing, J. Peters, E. Sgouritsa, K. Zhang, and J. Mooij.  On causal and anticausal learning.  In _International Coference on International Conference on Machine Learning_ , page 459–466, 2012.
  * Schuhmann et al. [2021] C. Schuhmann, R. Vencu, R. Beaumont, R. Kaczmarczyk, C. Mullis, A. Katta, T. Coombes, J. Jitsev, and A. Komatsuzaki.  Laion-400m: Open dataset of clip-filtered 400 million image-text pairs.  _arXiv preprint arXiv:2111.02114_ , 2021.
  * Schuhmann et al. [2022] C. Schuhmann, R. Beaumont, R. Vencu, C. W. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, P. Schramowski, S. R. Kundurthy, K. Crowson, L. Schmidt, R. Kaczmarczyk, and J. Jitsev.  LAION-5b: An open large-scale dataset for training next generation image-text models.  In _Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track_ , 2022.  URL https://openreview.net/forum?id=M3Y74vmsMcY.
  * Vogel et al. [2022] F. Vogel, N. Shvetsova, L. Karlinsky, and H. Kuehne.  Vl-taboo: An analysis of attribute-based zero-shot capabilities of vision-language models.  _CoRR_ , abs/2209.06103, 2022.  URL https://doi.org/10.48550/arXiv.2209.06103.
  * Xiao et al. [2021] K. Y. Xiao, L. Engstrom, A. Ilyas, and A. Madry.  Noise or signal: The role of image backgrounds in object recognition.  In _International Conference on Learning Representations_ , 2021.  URL https://openreview.net/forum?id=gl3D-xY7wLq.

##  Appendix A Further inspection of LAIONet

In Section 2, we described how LAIONet is generated through substring matching
LAION texts with ImageNet synset lemmas, followed by filtering out the cases
where the LAION text is not sufficiently similar to the synset name and
definition as measured with the cosine similarity of CLIP embeddings. A
critical choice in the second filtering step is the choice of the minimum
required textual similarity. We conservatively chose this threshold to be the
largest value such that the remaining examples cover a large number of
ImageNet’s classes. To show this filtering is necessary and our threshold of
$0.82$ is conservative, we have provided an example in Figure 11. Here the
synset “cougar” has lemma “puma”. From WordNet definition, “cougar” is a
“large American feline resembling lion”. But the common usage of “puma” on the
web is about a brand. As Figure 11 shows for small similarity to the synset,
data most likely will represent the brand instead of the animal. As we
increase the similarity threshold, the examples become more and more likely to
be from the intended meaning. Our manual inspections show similar to this
example, the chosen threshold of $0.82$ always results in high-quality
matching to the intended meaning of the synset even if the web is dominated by
other meanings.

Figure 11: Sample images from five intervals of LAION text to synset text
similarity.

##  Appendix B Further insights on CLIP zero-shot accuracy

In Section 2 we observed that CLIP has a $5$ to $10$ percentage points higher
accuracy on LAIONet. With finer details, Figure 12 shows CLIP zero-shot recall
for each class in LAIONet minus the corresponding number in ImageNet.
Considering $95\%$ confidence intervals, CLIP has higher zero-shot recall@$1$
on LAIONet in one-third of the classes while the opposite is true only for
one-tenth of the classes. Consistently, in terms of recall@$5$, CLIP has a
higher recall on LAIONet in one-fourth of the classes while the converse is
true for only $13\%$ of the classes. Overall, LAIONet seems to be a slightly
easier dataset for CLIP.

(a) Recall@1

(b) Recall@5

Figure 12: The difference of CLIP zero-shot recall in LAIONet and ImageNet is
depicted. Lower and upper confidence intervals are obtained at $95\%$. All
values are sorted ascendingly.

##  Appendix C Further evidence on the (non)difficulty of LAIONet

To obtain a better idea of how hard is it to recognize an object in LAIONet,
we calculate the similarity of the images to the texts of their associated
synsets using CLIP embeddings. We measure this similarity as the cosine
similarity of the image embedding with the embedding of the synset text, i.e.,
the name of the synset followed by its definition. A high value of image-to-
synset similarity indicates CLIP is able to identify an object from the synset
in the image. On the other hand, a low value could indicate that the intended
object is either absent from the image or difficult to recognize. We compare
the image-to-synset similarities obtained from the ImageNet validation set and
LAIONet.

Figure 13(a) illustrates the distribution of image-to-synset similarity for
LAIONet and ImageNet. To ensure these distributions are comparable, we sampled
LAIONet with replacement to match the number of images per class in the
ImageNet validation set. As the figure suggests, the two datasets are not
significantly different. In a more fine-grained test, we compared the image-
to-synset similarity of the LAIONet and ImageNet for each class. Figure 13(b)
shows the average similarity in each class for LAIONet subtracted by the
average similarity in the same class for ImageNet along $95\%$ upper and lower
confidence bounds. Overall, LAIONet images have significantly higher
similarity to their respective synsets in $43\%$ of the classes while the
converse is true only for $14\%$ of the classes. This provides additional
evidence that identifying intended objects in LAIONet should not be more
difficult.

(a) Dist. of aggregated similarities

(b) Comparison across classes

Figure 13: Comparing image-to-synset similarities of LAIONet and ImageNet. (a)
For each class, LAIONet is sampled with replacement to have the same number of
images as ImageNet, and all samples are aggregated to obtain the distribution.
(b) For every class, the average similarity of the images to synset text is
calculated for LAIONet and ImageNet and the difference is plotted. The upper
and lower $0.95\%$ confidence bound for this difference is plotted in red and
blue. All values are sorted ascendingly.

##  Appendix D The relation of recall, relative frequency, and intra-class
similarity

###  D.1 Recall versus relative frequency

In Section 3.1 we observed accuracy drops when we weight different classes
according to their frequency in LAIONet. We attributed this decrease to the
models performing worse on more frequent classes. To directly observe this,
Figure 14 shows the recall in each class versus the relative frequency of the
class in LAIONet. One can see there is a weak but consistent trend that more
frequent classes are more likely to be misclassified.

(a) Recall@1

(b) Recall@5

Figure 14: Recall per class evaluated on LAIONet versus how frequent the class
is in LAIONet. Four different models are used, where two of them are
pretrained on ImageNet-$21$k and two of them are not. Trends are consistent.

###  D.2 Relative frequency versus intra-class similarity

Figure 6(b) shows the relative frequency of different classes versus the
average intra-class similarity of LAIONet images. As one might expect, the
more frequent a class, the lower the intra-class similarity in LAIONet, but
the relationship is weak with a Spearman correlation coefficient of $-0.26$.

Figure 15: The plot compares the relative frequency of each class with the
class-average of intra-class similarity in LAIONet.

###  D.3 Recall versus intra-class similarity

See Figure 7 in Section 3.2.

##  Appendix E On the choice of similarity threshold in extracting the dataset
of Section 4.3

In Section 4.3, we selected a similarity threshold of $0.7$ as the minimum
requirement for similarity between LAION text and ImageNet text in order to
include a sample from LAION. Ideally, we look for LAION examples with
identical text as the ImageNet but due to the limited number of samples
available in LAION, this is not possible. As Figure 3(b) shows, increasing the
similarity threshold beyond the chosen level of $0.7$ significantly decreases
the number of covered classes. Meanwhile, for larger thresholds, the new
dataset looks more like ImageNet but is still distinguishable. As Figure 16(b)
shows, the proportion of classes with significantly lower intra-class
similarity in ImageNet increases as the threshold increases, while the
proportion of classes with significantly lower intra-class similarity in the
new dataset decreases. The gap still persists but can potentially become
smaller in the region our data cannot cover. In sum, the new dataset extracted
based on ImageNet looks unlike ImageNet but to the extent it is possible to
find similar texts in LAION.

(a)

(b)

Figure 16: The effect of similarity threshold on the dataset extracted from
LAION samples with most similar texts to the ImageNet texts. (a) Number of the
classes covered in the new dataset versus the similarity threshold. (b)
Proportion of classes with significantly lower intra-class similarity in the
new dataset (blue) and proportion of classes with significantly lower intra-
class similarity in ImageNet (red) versus the similarity threshold.


* Regneri and Wang (2012) Michaela Regneri and Rui Wang. 2012.  Using discourse information for paraphrase extraction.  In _Proceedings of the 2012 Joint Conference on Empirical Methods in Natural Language Processing and Computational Natural Language Learning_.
  * Rei (2017) Marek Rei. 2017.  Semi-supervised multitask learning for sequence labeling.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-BERT: Sentence embeddings using Siamese BERT-networks.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.
  * Resnik (1999) Philip Resnik. 1999.  Mining the web for bilingual text.  In _Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics_.
  * Resnik and Smith (2003) Philip Resnik and Noah A. Smith. 2003.  The web as a parallel corpus.  _Computational Linguistics_ , 29(3):349–380.
  * Reynolds and McDonell (2021) Laria Reynolds and Kyle McDonell. 2021.  Prompt programming for large language models: Beyond the few-shot paradigm.  In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_.
  * Rezende et al. (2014) Danilo Jimenez Rezende, Shakir Mohamed, and Daan Wierstra. 2014.  Stochastic backpropagation and approximate inference in deep generative models.  In _Proceedings of the 31st International Conference on International Conference on Machine Learning - Volume 32_.
  * Richman and Schone (2008) Alexander E. Richman and Patrick Schone. 2008.  Mining Wiki resources for multilingual named entity recognition.  In _Proceedings of ACL-08: HLT_.
  * Riedel et al. (2013) Sebastian Riedel, Limin Yao, Andrew McCallum, and Benjamin M. Marlin. 2013.  Relation extraction with matrix factorization and universal schemas.  In _Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Riedl and Young (2010) Mark O Riedl and Robert Michael Young. 2010.  Narrative planning: Balancing plot and character.  _Journal of Artificial Intelligence Research_ , 39:217–268.
  * Rios et al. (2018) Miguel Rios, Wilker Aziz, and Khalil Sima’an. 2018.  Deep generative model for joint alignment and word representation.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_.
  * Robertson et al. (1995) Stephen Robertson, S. Walker, S. Jones, M. M. Hancock-Beaulieu, and M. Gatford. 1995\.  Okapi at trec-3.  In _Overview of the Third Text REtrieval Conference (TREC-3)_.
  * Roemmele et al. (2011) Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. 2011.  Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In _2011 AAAI Spring Symposium Series_.
  * Rothe et al. (2020) Sascha Rothe, Shashi Narayan, and Aliaksei Severyn. 2020.  Leveraging pre-trained checkpoints for sequence generation tasks.  _Transactions of the Association for Computational Linguistics_ , 8:264–280.
  * Rubenstein and Goodenough (1965) Herbert Rubenstein and John B Goodenough. 1965.  Contextual correlates of synonymy.  _Communications of the ACM_ , 8(10):627–633.
  * Rubino and Sumita (2020) Raphael Rubino and Eiichiro Sumita. 2020.  Intermediate self-supervised learning for machine translation quality estimation.  In _Proceedings of the 28th International Conference on Computational Linguistics_.
  * Rush et al. (2015) Alexander M. Rush, Sumit Chopra, and Jason Weston. 2015.  A neural attention model for abstractive sentence summarization.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_.
  * Safavi and Koutra (2020) Tara Safavi and Danai Koutra. 2020.  CoDEx: A Comprehensive Knowledge Graph Completion Benchmark.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Sakaguchi et al. (2020) Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2020.  Winogrande: An adversarial winograd schema challenge at scale.  In _AAAI_.
  * Salton et al. (1975) G. Salton, A. Wong, and C. S. Yang. 1975.  A vector space model for automatic indexing.  _Commun. ACM_ , 18(11):613–620.
  * Salton (1971) Gerard Salton. 1971.  _The SMART retrieval system—experiments in automatic document processing_.  Prentice-Hall, Inc.
  * Sandhaus (2008) Evan Sandhaus. 2008.  _The New York Times Annotated Corpus_.  LDC corpora. Linguistic Data Consortium.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Teven Le Scao, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M Rush. 2022.  Multitask prompted training enables zero-shot task generalization.  In _International Conference on Learning Representations_.
  * Schick and Schütze (2021a) Timo Schick and Hinrich Schütze. 2021a.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_.
  * Schick and Schütze (2021b) Timo Schick and Hinrich Schütze. 2021b.  Few-shot text generation with natural language instructions.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_.
  * Schick and Schütze (2021c) Timo Schick and Hinrich Schütze. 2021c.  It’s not just size that matters: Small language models are also few-shot learners.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Schulz et al. (2018) Philip Schulz, Wilker Aziz, and Trevor Cohn. 2018.  A stochastic decoder for neural machine translation.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Schütze (1992) Hinrich Schütze. 1992.  Word space.  In _Advances in Neural Information Processing Systems_.
  * Schwenk et al. (2021) Holger Schwenk, Vishrav Chaudhary, Shuo Sun, Hongyu Gong, and Francisco Guzmán. 2021.  WikiMatrix: Mining 135M parallel sentences in 1620 language pairs from Wikipedia.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_.
  * See et al. (2017) Abigail See, Peter J. Liu, and Christopher D. Manning. 2017.  Get to the point: Summarization with pointer-generator networks.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Serban et al. (2017) Iulian Vlad Serban, Alexander G. Ororbia, Joelle Pineau, and Aaron Courville. 2017\.  Piecewise latent variables for neural variational text processing.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_.
  * Setiawan et al. (2020) Hendra Setiawan, Matthias Sperber, Udhyakumar Nallasamy, and Matthias Paulik. 2020\.  Variational neural machine translation with normalizing flows.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Sharma et al. (2019) Eva Sharma, Chen Li, and Lu Wang. 2019.  BIGPATENT: A large-scale dataset for abstractive and coherent summarization.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
  * Shen et al. (2018a) Dinghan Shen, Guoyin Wang, Wenlin Wang, Martin Renqiang Min, Qinliang Su, Yizhe Zhang, Chunyuan Li, Ricardo Henao, and Lawrence Carin. 2018a.  Baseline needs more love: On simple word-embedding-based models and associated pooling mechanisms.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Shen et al. (2017a) Tianxiao Shen, Tao Lei, Regina Barzilay, and Tommi Jaakkola. 2017a.  Style transfer from non-parallel text by cross-alignment.  In _Advances in Neural Information Processing Systems_.
  * Shen et al. (2017b) Xiaoyu Shen, Hui Su, Yanran Li, Wenjie Li, Shuzi Niu, Yang Zhao, Akiko Aizawa, and Guoping Long. 2017b.  A conditional variational framework for dialog generation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_.
  * Shen et al. (2018b) Xiaoyu Shen, Hui Su, Shuzi Niu, and Vera Demberg. 2018b.  Improving variational encoder-decoders in dialogue generation.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1).
  * Shetty et al. (2018) Rakshith Shetty, Bernt Schiele, and Mario Fritz. 2018.  A4nt: author attribute anonymity by adversarial training of neural machine translation.  In _27th USENIX Security Symposium (USENIX Security 18)_.
  * Shi and Lin (2019) Peng Shi and Jimmy Lin. 2019.  Simple bert models for relation extraction and semantic role labeling.
  * Shi et al. (2021a) Tianze Shi, Ozan İrsoy, Igor Malioutov, and Lillian Lee. 2021a.  Learning syntax from naturally-occurring bracketings.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Shi et al. (2021b) Weijia Shi, Mandar Joshi, and Luke Zettlemoyer. 2021b.  DESCGEN: A distantly supervised datasetfor generating entity descriptions.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_.
  * Shwartz et al. (2016) Vered Shwartz, Yoav Goldberg, and Ido Dagan. 2016.  Improving hypernymy detection with an integrated path-based and distributional method.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Sileo et al. (2019) Damien Sileo, Tim Van De Cruys, Camille Pradel, and Philippe Muller. 2019.  Mining discourse markers for unsupervised sentence representation learning.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_.
  * Singh et al. (2012a) Sameer Singh, Amarnag Subramanya, Fernando Pereira, and Andrew McCallum. 2012a.  Wikilinks: A large-scale cross-document coreference corpus labeled via links to wikipedia.  _University of Massachusetts, Amherst, Tech. Rep. UM-CS-2012_ , 15\.
  * Singh et al. (2012b) Sameer Singh, Amarnag Subramanya, Fernando Pereira, and Andrew McCallum. 2012b.  Wikilinks: A large-scale cross-document coreference corpus labeled via links to Wikipedia.  Technical Report UM-CS-2012-015, University of Massachusetts, Amherst.
  * Smith et al. (2010) Jason R. Smith, Chris Quirk, and Kristina Toutanova. 2010.  Extracting parallel sentences from comparable corpora using document level alignment.  In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_.
  * Smith et al. (2013) Jason R. Smith, Herve Saint-Amand, Magdalena Plamada, Philipp Koehn, Chris Callison-Burch, and Adam Lopez. 2013.  Dirt cheap web-scale parallel text from the Common Crawl.  In _Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Snow et al. (2005) Rion Snow, Daniel Jurafsky, and Andrew Y. Ng. 2005.  Learning syntactic patterns for automatic hypernym discovery.  In L. K. Saul, Y. Weiss, and L. Bottou, editors, _Advances in Neural Information Processing Systems 17_.
  * Snow et al. (2006) Rion Snow, Daniel Jurafsky, and Andrew Y. Ng. 2006.  Semantic taxonomy induction from heterogenous evidence.  In _Proceedings of the 21st International Conference on Computational Linguistics and 44th Annual Meeting of the Association for Computational Linguistics_.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_.
  * Søgaard (2017) Anders Søgaard. 2017.  Using hyperlinks to improve multilingual partial parsers.  In _Proceedings of the 15th International Conference on Parsing Technologies_.
  * Song et al. (2019) Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2019.  MASS: Masked sequence to sequence pre-training for language generation.  In _Proceedings of the 36th International Conference on Machine Learning_.
  * Sorensen et al. (2022) Taylor Sorensen, Joshua Robinson, Christopher Michael Rytting, Alexander Glenn Shaw, Kyle Jeffrey Rogers, Alexia Pauline Delorey, Mahmoud Khalil, Nancy Fulda, and David Wingate. 2022.  An information-theoretic approach to prompt engineering without ground truth labels.  _arXiv preprint arXiv:2203.11364_.
  * Spitkovsky et al. (2010) Valentin I. Spitkovsky, Daniel Jurafsky, and Hiyan Alshawi. 2010.  Profiting from mark-up: Hyper-text annotations for guided parsing.  In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_.
  * Stratos and Wiseman (2020) Karl Stratos and Sam Wiseman. 2020.  Learning discrete structured representations by adversarially maximizing mutual information.  In _Proceedings of the 37th International Conference on Machine Learning_.
  * Strube and Ponzetto (2006) Michael Strube and Simone Paolo Ponzetto. 2006.  Wikirelate! computing semantic relatedness using wikipedia.  In _Proceedings of the 21st National Conference on Artificial Intelligence - Volume 2_.
  * Suchanek et al. (2007) Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. 2007.  Yago: A core of semantic knowledge.  In _Proceedings of the 16th International Conference on World Wide Web_.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V Le. 2014.  Sequence to sequence learning with neural networks.  In _Advances in Neural Information Processing Systems_.
  * Szpektor and Dagan (2008) Idan Szpektor and Ido Dagan. 2008.  Learning entailment rules for unary templates.  In _Proceedings of the 22nd International Conference on Computational Linguistics (Coling 2008)_.
  * Szpektor et al. (2004) Idan Szpektor, Hristo Tanev, Ido Dagan, and Bonaventura Coppola. 2004.  Scaling web-based acquisition of entailment relations.  In _Proceedings of the 2004 Conference on Empirical Methods in Natural Language Processing_.
  * Tam et al. (2021) Derek Tam, Rakesh R. Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. 2021.  Improving and simplifying pattern exploiting training.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_.
  * Tapaswi et al. (2016) Makarand Tapaswi, Yukun Zhu, Rainer Stiefelhagen, Antonio Torralba, Raquel Urtasun, and Sanja Fidler. 2016.  Movieqa: Understanding stories in movies through question-answering.  In _Proceedings of the IEEE conference on computer vision and pattern recognition_.
  * Tekiroğlu et al. (2014) Serra Sinem Tekiroğlu, Gözde Özbal, and Carlo Strapparava. 2014.  Sensicon: An automatically constructed sensorial lexicon.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Tiedemann (2012) Jörg Tiedemann. 2012.  Parallel data, tools and interfaces in OPUS.  In _Proceedings of the Eighth International Conference on Language Resources and Evaluation (LREC’12)_.
  * Tjong Kim Sang and De Meulder (2003) Erik F. Tjong Kim Sang and Fien De Meulder. 2003.  Introduction to the CoNLL-2003 shared task: Language-independent named entity recognition.  In _Proceedings of the Seventh Conference on Natural Language Learning at HLT-NAACL 2003_.
  * Toutanova et al. (2003) Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003.  Feature-rich part-of-speech tagging with a cyclic dependency network.  In _Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics_.
  * Tu et al. (2017a) Lifu Tu, Kevin Gimpel, and Karen Livescu. 2017a.  Learning to embed words in context for syntactic tasks.  In _Proceedings of the 2nd Workshop on Representation Learning for NLP_.
  * Tu et al. (2017b) Zhaopeng Tu, Yang Liu, Lifeng Shang, Xiaohua Liu, and Hang Li. 2017b.  Neural machine translation with reconstruction.  In _Thirty-First AAAI Conference on Artificial Intelligence_.
  * Turian et al. (2010) Joseph Turian, Lev-Arie Ratinov, and Yoshua Bengio. 2010.  Word representations: A simple and general method for semi-supervised learning.  In _Proceedings of the 48th Annual Meeting of the Association for Computational Linguistics_.
  * Turney (2006) Peter D. Turney. 2006.  Similarity of semantic relations.  _Computational Linguistics_ , 32(3):379–416.
  * Turney and Littman (2005) Peter D Turney and Michael L Littman. 2005.  Corpus-based learning of analogies and semantic relations.  _Machine Learning_ , 60(1):251–278.
  * Turney and Mohammad (2015) Peter D. Turney and Saif M. Mohammad. 2015.  Experiments with three approaches to recognizing lexical entailment.  _Natural Language Engineering_ , 21:437–476.
  * van Valin, Jr. (2005) Robert D. van Valin, Jr. 2005.  _Exploring the Syntax-Semantics Interface_.  Cambridge University Press.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _Advances in Neural Information Processing Systems_.
  * Vishnubhotla et al. (2021) Krishnapriya Vishnubhotla, Graeme Hirst, and Frank Rudzicz. 2021.  An Evaluation of Disentangled Representation Learning for Texts.  In _Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021_.
  * Völske et al. (2017) Michael Völske, Martin Potthast, Shahbaz Syed, and Benno Stein. 2017.  TL;DR: Mining Reddit to learn automatic summarization.  In _Proceedings of the Workshop on New Frontiers in Summarization_.
  * Vougiouklis et al. (2018) Pavlos Vougiouklis, Hady Elsahar, Lucie-Aimée Kaffee, Christophe Gravier, Frédérique Laforest, Jonathon Hare, and Elena Simperl. 2018.  Neural wikipedian: Generating textual summaries from knowledge base triples.  _Journal of Web Semantics_ , 52:1–15.
  * Vrandečić and Krötzsch (2014) Denny Vrandečić and Markus Krötzsch. 2014.  Wikidata: A free collaborative knowledgebase.  _Commun. ACM_ , 57(10):78–85.
  * Vu et al. (2021) Tu Vu, Minh-Thang Luong, Quoc Le, Grady Simon, and Mohit Iyyer. 2021.  STraTA: Self-training with task augmentation for better few-shot learning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  In _Advances in Neural Information Processing Systems_.
  * Wang et al. (2018a) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2018a.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  In _Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP_.
  * Wang and Ling (2016) Lu Wang and Wang Ling. 2016.  Neural network-based abstract generation for opinions and arguments.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Wang et al. (2021a) Luyu Wang, Yujia Li, Ozlem Aslan, and Oriol Vinyals. 2021a.  WikiGraphs: A Wikipedia text - knowledge graph paired dataset.  In _Proceedings of the Fifteenth Workshop on Graph-Based Methods for Natural Language Processing (TextGraphs-15)_.
  * Wang et al. (2018b) Qingyun Wang, Xiaoman Pan, Lifu Huang, Boliang Zhang, Zhiying Jiang, Heng Ji, and Kevin Knight. 2018b.  Describing a knowledge base.  In _Proceedings of the 11th International Conference on Natural Language Generation_.
  * Wang et al. (2021b) Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. 2021b.  Entailment as few-shot learner.  _arXiv preprint arXiv:2104.14690_.
  * Wang et al. (2018c) Su Wang, Eric Holgate, Greg Durrett, and Katrin Erk. 2018c.  Picking apart story salads.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_.
  * Wang et al. (2020) Wei Wang, Bin Bi, Ming Yan, Chen Wu, Jiangnan Xia, Zuyi Bao, Liwei Peng, and Luo Si. 2020.  Structbert: Incorporating language structures into pre-training for deep language understanding.  In _International Conference on Learning Representations_.
  * Wang et al. (2021c) Xiaozhi Wang, Tianyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021c.  KEPLER: A unified model for knowledge embedding and pre-trained language representation.  _Transactions of the Association for Computational Linguistics_ , 9:176–194.
  * Wang et al. (2021d) Yaqing Wang, Subhabrata Mukherjee, Xiaodong Liu, Jing Gao, Ahmed Hassan Awadallah, and Jianfeng Gao. 2021d.  List: Lite self-training makes efficient few-shot learners.  _arXiv preprint arXiv:2110.06274_.
  * Wang et al. (1996) Ye-Yi Wang, J. Lafferty, and A. Waibel. 1996.  Word clustering with parallel spoken language corpora.  In _Proceeding of Fourth International Conference on Spoken Language Processing. ICSLP ’96_.
  * Wang et al. (2018d) Yizhong Wang, Sujian Li, and Jingfeng Yang. 2018d.  Toward fast and accurate neural discourse segmentation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_.
  * Warstadt et al. (2019) Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. 2019.  Neural network acceptability judgments.  _Transactions of the Association for Computational Linguistics_ , 7:625–641.
  * Webster et al. (2018) Kellie Webster, Marta Recasens, Vera Axelrod, and Jason Baldridge. 2018.  Mind the GAP: A balanced corpus of gendered ambiguous pronouns.  _Transactions of the Association for Computational Linguistics_ , 6:605–617.
  * Wei et al. (2022) Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022.  Finetuned language models are zero-shot learners.  In _International Conference on Learning Representations_.
  * Welleck et al. (2020) Sean Welleck, Ilia Kulikov, Stephen Roller, Emily Dinan, Kyunghyun Cho, and Jason Weston. 2020.  Neural text generation with unlikelihood training.  In _International Conference on Learning Representations_.
  * Weller et al. (2020) Orion Weller, Nicholas Lourie, Matt Gardner, and Matthew E. Peters. 2020.  Learning from task descriptions.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Weston et al. (2018) Jason Weston, Emily Dinan, and Alexander Miller. 2018.  Retrieve and refine: Improved sequence generation models for dialogue.  In _Proceedings of the 2018 EMNLP Workshop SCAI: The 2nd International Workshop on Search-Oriented Conversational AI_.
  * Wiebe et al. (2005) Janyce Wiebe, Theresa Wilson, and Claire Cardie. 2005.  Annotating expressions of opinions and emotions in language.  _Language resources and evaluation_ , 39(2):165–210.
  * Wieting et al. (2016) John Wieting, Mohit Bansal, Kevin Gimpel, and Karen Livescu. 2016.  Towards universal paraphrastic sentence embeddings.  In _Proceedings of International Conference on Learning Representations_.
  * Wieting and Gimpel (2017) John Wieting and Kevin Gimpel. 2017.  Revisiting recurrent networks for paraphrastic sentence embeddings.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Wieting and Gimpel (2018) John Wieting and Kevin Gimpel. 2018.  ParaNMT-50M: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Wieting et al. (2019) John Wieting, Kevin Gimpel, Graham Neubig, and Taylor Berg-Kirkpatrick. 2019.  Simple and effective paraphrastic similarity from parallel translations.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_.
  * Wiseman et al. (2016) Sam Wiseman, Alexander M. Rush, and Stuart M. Shieber. 2016.  Learning global features for coreference resolution.  In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Wiseman et al. (2017) Sam Wiseman, Stuart Shieber, and Alexander Rush. 2017.  Challenges in data-to-document generation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_.
  * Wiseman et al. (2018) Sam Wiseman, Stuart Shieber, and Alexander Rush. 2018.  Learning neural templates for text generation.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_.
  * Wu and Goodman (2018) Mike Wu and Noah Goodman. 2018.  Multimodal generative models for scalable weakly-supervised learning.  In _Advances in Neural Information Processing Systems_.
  * Wu et al. (2016) Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao, Klaus Macherey, et al. 2016.  Google’s neural machine translation system: Bridging the gap between human and machine translation.  _arXiv preprint arXiv:1609.08144_.
  * Xu et al. (2022) Hanwei Xu, Yujun Chen, Yulun Du, Nan Shao, Yanggang Wang, Haiyu Li, and Zhilin Yang. 2022.  Zeroprompt: Scaling prompt-based pretraining to 1,000 tasks improves zero-shot generalization.  _arXiv preprint arXiv:2201.06910_.
  * Xu and Durrett (2018) Jiacheng Xu and Greg Durrett. 2018.  Spherical latent spaces for stable variational autoencoders.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_.
  * Xue et al. (2021) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2021.  mT5: A massively multilingual pre-trained text-to-text transformer.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Yaghoobzadeh and Schütze (2015) Yadollah Yaghoobzadeh and Hinrich Schütze. 2015.  Corpus-level fine-grained entity typing using contextual information.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_.
  * Yamada et al. (2016) Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2016.  Joint learning of the embedding of words and entities for named entity disambiguation.  In _Proceedings of The 20th SIGNLL Conference on Computational Natural Language Learning_.
  * Yamada et al. (2017) Ikuya Yamada, Hiroyuki Shindo, Hideaki Takeda, and Yoshiyasu Takefuji. 2017.  Learning distributed representations of texts and entities from knowledge base.  _Transactions of the Association for Computational Linguistics_ , 5:397–411.
  * Yamangil and Nelken (2008) Elif Yamangil and Rani Nelken. 2008.  Mining Wikipedia revision histories for improving sentence compression.  In _Proceedings of ACL-08: HLT, Short Papers_.
  * Yang et al. (2018) Yinfei Yang, Steve Yuan, Daniel Cer, Sheng-yi Kong, Noah Constant, Petr Pilar, Heming Ge, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018.  Learning semantic textual similarity from conversations.  In _Proceedings of The Third Workshop on Representation Learning for NLP_.
  * Yang and Choi (2019) Zhengzhe Yang and Jinho D. Choi. 2019.  FriendsQA: Open-domain question answering on TV show transcripts.  In _Proceedings of the 20th Annual SIGdial Meeting on Discourse and Dialogue_.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  In _Advances in Neural Information Processing Systems_.
  * Yao et al. (2019) Lili Yao, Nanyun Peng, Ralph Weischedel, Kevin Knight, Dongyan Zhao, and Rui Yan. 2019.  Plan-and-write: Towards better automatic storytelling.  In _Proceedings of the AAAI Conference on Artificial Intelligence_.
  * Yates and Etzioni (2009) Alexander Yates and Oren Etzioni. 2009.  Unsupervised methods for determining object and relation synonyms on the web.  _J. Artif. Int. Res._ , 34(1):255–296.
  * Yatskar et al. (2010) Mark Yatskar, Bo Pang, Cristian Danescu-Niculescu-Mizil, and Lillian Lee. 2010.  For the sake of simplicity: Unsupervised extraction of lexical simplifications from Wikipedia.  In _Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics_.
  * Ye et al. (2021) Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021.  CrossFit: A few-shot learning challenge for cross-task generalization in NLP.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_.
  * Ye et al. (2020) Xi Ye, Qiaochu Chen, Xinyu Wang, Isil Dillig, and Greg Durrett. 2020.  Sketch-driven regular expression generation from natural language and examples.  _Transactions of the Association for Computational Linguistics_ , 8:679–694.
  * Yin et al. (2020a) Pengcheng Yin, Graham Neubig, Wen-tau Yih, and Sebastian Riedel. 2020a.  TaBERT: Pretraining for joint understanding of textual and tabular data.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Yin et al. (2018) Pengcheng Yin, Chunting Zhou, Junxian He, and Graham Neubig. 2018.  StructVAE: Tree-structured latent variable models for semi-supervised semantic parsing.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Yin et al. (2019) Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019.  Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.
  * Yin et al. (2020b) Wenpeng Yin, Nazneen Fatema Rajani, Dragomir Radev, Richard Socher, and Caiming Xiong. 2020b.  Universal natural language processing with limited annotations: Try few-shot textual entailment as a start.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * You et al. (2020) Yang You, Jing Li, Sashank Reddi, Jonathan Hseu, Sanjiv Kumar, Srinadh Bhojanapalli, Xiaodan Song, James Demmel, Kurt Keutzer, and Cho-Jui Hsieh. 2020\.  Large batch optimization for deep learning: Training bert in 76 minutes.  In _International Conference on Learning Representations_.
  * Yu et al. (2020) Dian Yu, Kai Sun, Claire Cardie, and Dong Yu. 2020.  Dialogue-based relation extraction.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Yu et al. (2016) Hongliang Yu, Shikun Zhang, and Louis-Philippe Morency. 2016.  Unsupervised text recap extraction for TV series.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_.
  * Zahiri and Choi (2017) Sayyed M. Zahiri and Jinho D. Choi. 2017.  Emotion detection on tv show transcripts with sequence-based convolutional neural networks.
  * Zesch (2012) Torsten Zesch. 2012.  Measuring contextual fitness using error contexts extracted from the Wikipedia revision history.  In _Proceedings of the 13th Conference of the European Chapter of the Association for Computational Linguistics_.
  * Zesch and Gurevych (2007) Torsten Zesch and Iryna Gurevych. 2007.  Analysis of the Wikipedia category graph for NLP applications.  In _Proceedings of the Second Workshop on TextGraphs: Graph-Based Algorithms for Natural Language Processing_.
  * Zhang et al. (2016) Biao Zhang, Deyi Xiong, Jinsong Su, Hong Duan, and Min Zhang. 2016.  Variational neural machine translation.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_.
  * Zhang et al. (2021) Jianguo Zhang, Trung Bui, Seunghyun Yoon, Xiang Chen, Zhiwei Liu, Congying Xia, Quan Hung Tran, Walter Chang, and Philip Yu. 2021.  Few-shot intent detection via contrastive pre-training and fine-tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_.
  * Zhang et al. (2020a) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter Liu. 2020a.  PEGASUS: Pre-training with extracted gap-sentences for abstractive summarization.  In _Proceedings of the 37th International Conference on Machine Learning_.
  * Zhang and Shasha (1989) Kaizhong Zhang and Dennis Shasha. 1989.  Simple fast algorithms for the editing distance between trees and related problems.  _SIAM journal on computing_ , 18(6):1245–1262.
  * Zhang et al. (2022) Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. 2022.  Differentiable prompt makes pre-trained language models better few-shot learners.  In _International Conference on Learning Representations_.
  * Zhang et al. (2018) Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018.  ReCoRD: Bridging the gap between human and machine commonsense reading comprehension.  _arXiv preprint 1810.12885_.
  * Zhang and Zhang (2021) Shiwei Zhang and Xiuzhen Zhang. 2021.  Does QA-based intermediate training help fine-tuning language models for text classification? In _Proceedings of the The 19th Annual Workshop of the Australasian Language Technology Association_.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _Advances in Neural Information Processing Systems_.
  * Zhang et al. (2020b) Yizhe Zhang, Siqi Sun, Michel Galley, Yen-Chun Chen, Chris Brockett, Xiang Gao, Jianfeng Gao, Jingjing Liu, and Bill Dolan. 2020b.  DIALOGPT : Large-scale generative pre-training for conversational response generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_.
  * Zhao et al. (2018) Junbo Zhao, Yoon Kim, Kelly Zhang, Alexander Rush, and Yann LeCun. 2018.  Adversarially regularized autoencoders.  In _Proceedings of the 35th International Conference on Machine Learning_.
  * Zhao et al. (2017) Tiancheng Zhao, Ran Zhao, and Maxine Eskenazi. 2017.  Learning discourse-level diversity for neural dialog models using conditional variational autoencoders.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate before use: Improving few-shot performance of language models.  In _Proceedings of the 38th International Conference on Machine Learning_.
  * Zheng et al. (2013) Jiaping Zheng, Luke Vilnis, Sameer Singh, Jinho D. Choi, and Andrew McCallum. 2013\.  Dynamic knowledge-base alignment for coreference resolution.  In _Proceedings of the Seventeenth Conference on Computational Natural Language Learning_.
  * Zhong et al. (2021a) Ming Zhong, Da Yin, Tao Yu, Ahmad Zaidi, Mutethia Mutuma, Rahul Jha, Ahmed Hassan Awadallah, Asli Celikyilmaz, Yang Liu, Xipeng Qiu, and Dragomir Radev. 2021a.  QMSum: A new benchmark for query-based multi-domain meeting summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Zhong et al. (2021b) Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. 2021b.  Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_.
  * Zhou et al. (2019) Ben Zhou, Daniel Khashabi, Qiang Ning, and Dan Roth. 2019.  “going on a vacation” takes longer than “going for a walk”: A study of temporal commonsense understanding.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_.
  * Zhou and Neubig (2017) Chunting Zhou and Graham Neubig. 2017.  Multi-space variational encoder-decoders for semi-supervised labeled sequence transduction.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_.
  * Zhou and Choi (2018) Ethan Zhou and Jinho D. Choi. 2018.  They exist! introducing plural mentions to coreference resolution and entity linking.  In _Proceedings of the 27th International Conference on Computational Linguistics_.
  * Zhou et al. (2020) Fan Zhou, Shengming Zhang, and Yi Yang. 2020.  Interpretable operational risk classification with semi-supervised variational autoencoder.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_.
  * Zhou et al. (2010) Zhi-Min Zhou, Yu Xu, Zheng-Yu Niu, Man Lan, Jian Su, and Chew Lim Tan. 2010.  Predicting discourse connectives for implicit discourse relation recognition.  In _Coling 2010: Posters_.
  * Zhu et al. (2021) Chenguang Zhu, Yang Liu, Jie Mei, and Michael Zeng. 2021.  MediaSum: A large-scale media interview dataset for dialogue summarization.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_.
  * Zhu et al. (2015) Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015.  Aligning books and movies: Towards story-like visual explanations by watching movies and reading books.  In _Proceedings of the IEEE international conference on computer vision_.
  * Zhu et al. (2010) Zhemin Zhu, Delphine Bernhard, and Iryna Gurevych. 2010.  A monolingual tree-based translation model for sentence simplification.  In _Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010)_.
  * Ziegler and Rush (2019) Zachary Ziegler and Alexander Rush. 2019.  Latent normalizing flows for discrete sequences.  In _Proceedings of the 36th International Conference on Machine Learning_.
  * Ziemski et al. (2016) Michał Ziemski, Marcin Junczys-Dowmunt, and Bruno Pouliquen. 2016.  The United Nations parallel corpus v1.0.  In _Proceedings of the Tenth International Conference on Language Resources and Evaluation (LREC’16)_.
  * Zopf (2018) Markus Zopf. 2018.  Auto-hMDS: Automatic construction of a large heterogeneous multilingual multi-document summarization corpus.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_.