# Empowering Sentence Encoders with Prompting and Label Retrieval for Zero-
shot Text Classification

Jimin Hong* 1,2 Jungsoo Park* 1 Daeyoung Kim* 1,2 Seongjae Choi1 Bokyung Son1
Jaewook Kang1
1NAVER, 2KAIST
{jspark.93, seongjae.choi, bo.son<EMAIL_ADDRESS>
{jimmyh<EMAIL_ADDRESS>
###### Abstract

With contrastive pre-training, sentence encoders are generally optimized to
locate semantically similar samples closer to each other in their embedding
spaces. In this work, we focus on the potential of their embedding spaces to
be readily adapted to zero-shot text classification, as semantically distinct
samples are already well-separated. Our framework, RaLP (Retrieval augmented
Label Prompts for sentence encoder), encodes _prompted_ label candidates with
a sentence encoder, then assigns the label whose prompt embedding has the
highest similarity with the input text embedding. In order to compensate for
the potentially poorly descriptive labels in their original format, RaLP
_retrieves_ sentences that are semantically similar to the original label
prompt from external corpora and use them as additional pseudo-label prompts.
RaLP achieves competitive or stronger performance than much larger baselines
on various closed-set classification and multiple-choice QA datasets under
zero-shot settings. We show that the retrieval component plays a pivotal role
in RaLP’s success, and its results are robustly attained regardless of
verbalizer variations.

##  1 Introduction

Figure 1:  t-SNE (Van der Maaten and Hinton, 2008) visualization of encoded
test samples and label prompts from the CR dataset (Hu and Liu, 2004) using a
Sentence-t5 encoder (Ni et al., 2022). Each colored circle illustrates a
sample with its class information ("negative" or "positive"). The label
prompts for each class (_e.g._ “It was great.” and “It was terrible,”) are
shown in bold crosses.

Sentence encoders have been widely applied to a comprehensive range of natural
language processing tasks, including classification, semantic retrieval, and
semantic similarity tasks (Reimers and Gurevych, 2019; Du et al., 2021; Gao et
al., 2021b; Ni et al., 2022). They are usually pre-trained with a contrastive
objective on datasets that focus on sentence semantics (_e.g._ NLI), so
semantically similar texts are located close to each other in their embedding
spaces. We note that embedding spaces with such traits could be particularly
friendly to classification tasks under limited supervision, as semantically
distinct samples are well-separated in advance of any refinement made during
downstream training. Recent work has demonstrated the competitiveness of rich
text embeddings from sentence encoders for few-shot classification (Tunstall
et al., 2022), being on par or outperforming much larger prompt-based
generative language models on the RAFT benchmark (Alex et al., 2021). However,
it essentially involves training a logistic classification head, which
introduces additional parameters to be tuned and is thus inapplicable to zero-
shot inference.

In this work, we continue to explore the power of sentence encoders, but this
time to solve zero-shot classification by combining with prompting (Brown et
al., 2020) and dense retrieval techniques. We transform closed or open-set
classification tasks into finding the label texts (_i.e._ prompts) with
maximal textual similarity with the sentences under inference in a sentence
encoder’s embedding space. Figure 1 illustrates a motivational case for our
approach, where samples are generally located closer to the label prompt
representation of their corresponding class than the other. Meanwhile, we note
that label prompts in their original format are potentially ambiguous or
poorly descriptive; for example, _"Topic: World"_ in the AGNews dataset is an
overly compressive and abstract label, being less adequate to embrace a large
range of belonging texts. We handle this issue by using additional support
from multiple pseudo-label prompts retrieved from an external corpus.

Prompting has been commonly used for few-shot and zero-shot tasks within the
form of in-context learning with generative language models (Brown et al.,
2020; Gao et al., 2021a). However, their performances show high variance
across different verbalizers (_i.e._ text expressions for labels), even being
close to chance-level Perez et al. (2021); Lu et al. (2022). Unlike generative
models, where the label prompts and the query texts are intertwined by
concatenation at the input level, sentence encoders isolate the two at
encoding time. We empirically show that such post-hoc interactions can be
robust to label prompt variations, and the robustness further increases when
we enlarge the support prompt set through retrieval.

We conduct extensive experiments on six closed-set classification datasets and
six open-set multiple-choice datasets. Our best model based on a powerful
sentence encoder _Sentence-T5_ (Ni et al., 2022) with retrieval augmented
label prompts achieves comparable or even stronger performance compared to
strong zero-shot baselines for the closed-set classification tasks. The best
results for the multiple choice QA tasks are less impressive; nevertheless, we
still reach higher scores than baselines of a similar or somewhat bigger size.
Our method is consistently strong across verbalizer variations and scales to
different model sizes. Notably, using retrieval-augmented label prompts leads
to significant performance gains in closed-set classification, serving as a
simple yet promising way to leverage external knowledge resources.

##  2 Background

We explain the concept of sentence encoders and prompting that we use in this
work.

###  2.1 Sentence Encoders

Given a sentence $X_{i}\in\mathcal{X}$ , a sentence encoder $E$ encodes the
sentence into a fixed size embedding vector
$\mathbf{h}_{i}=E(X_{i})\in\mathbb{R}^{d}$, where $\mathcal{X}$ is the set of
all natural language texts and $d$ is the preset embedding dimension. Sentence
encoders are commonly trained with the contrastive learning objective (Chen et
al., 2020) with in-batch negatives (Chen et al., 2017; Henderson et al.,
2017). The loss function pulls the positive pair representation closer to the
input representation while pushing away the negatives in embedding space:

$\ell=-\sum_{i=1}^{N}\log\frac{e^{\operatorname{sim}\left(\mathbf{h}_{i},\mathbf{h}_{i}^{+}\right)}}{\sum_{j=1,j\neq
i}^{N}e^{\operatorname{sim}\left(\mathbf{h}_{i},\mathbf{h}_{j}\right)}}$ (1)

where $\operatorname{sim}(\cdot)$ is the similarity function,
$\mathbf{h}_{i}^{+}$ denotes the positive pair for $\mathbf{h}_{i}$, and
$\mathbf{h}_{j}$ refers to all other instances except $\mathbf{h}_{i}$ in the
batch of size $N$.

###  2.2 Prompting

Given a text input $X_{i}\in\mathcal{X}$ and a set of labels
$\mathcal{Y}=\left\\{y_{1},\ldots,y_{m}\right\\}$, a predefined verbalizer111
Consists of templates and label names. $v:\mathcal{Y}\rightarrow\mathcal{X}$
generates a label prompt in natural language for each label index. Generally,
language models compute the distribution of label prompts given the input,
$P_{\mathrm{LM}}\left(v\left(y_{m}\right)\mid X_{i}\right)$ (Brown et al.,
2020). Other lines of works utilize channel modeling
$P_{\mathrm{LM}}\left(X_{i}\mid v\left(y_{m}\right)\right)$ (Min et al.,
2022a, b) or masked language modeling objective (Gao et al., 2021a).

##  3 Method

Figure 2: An overview of our approach to a text classification task (or input-
label matching). Among a set of label candidates, we find the label whose
prompt has the highest representational similarity with the input text in the
embedding space of a sentence encoder (Figure (a)). The similarity scores are
computed across multiple retrievals augmented label prompts which are
collected with original label prompts as queries (Figure (b)).

We explore solving zero-shot classification tasks using text representations
from sentence encoders. Figure 2 provides an overview of our approach. The
gist is to exploit the representational similarity between the input text and
retrieval-augmented label prompts.

###  3.1 Representational Similarity with Sentence Encoders

Taking the example in Figure 2 (a), for a binary sentiment classification task
with $\mathcal{Y}=\\{y+,y-\\}$ where $y+$ stands for "positive" and $y-$
stands for "negative", we first use a verbalizer to change each label into
prompts: say, “It was great.”$(v(y+))$ and “It was terrible.”$(v(y-))$.
Afterward, the label prompts are transformed into encoded vectors that act as
prototypes for representing each class cluster: $\mathbf{z}_{m}=E(v(y_{m}))$.
Then, we use the following scoring function for each candidate label, defined
as its similarity with the text under inference $X_{i}$ “Not worth a red
cent.”:

$\displaystyle\underset{y_{m}\in\mathcal{Y}}{\operatorname{argmax}}~{}~{}\mathrm{P}_{\mathrm{TS}}(y_{m}|X_{i})$
$\displaystyle\mathrm{P}_{\mathrm{TS}}($ $\displaystyle
y_{m}|X_{i})\propto\operatorname{sim}({\mathbf{h}_{i},\mathbf{z}_{m}})$ (2)

where we use cosine similarity in a sentence encoder’s embedding space for
$\operatorname{sim}(\cdot)$.

Note that using Eq. 3.1 at inference time aligns the downstream task objective
with the contrastive pre-training objective (Eq. 1), which may benefit
sentence encoders in a limited label regime. Plus, separately encoding label
prompts from the test input makes the results more robust under verbalizer
variations, which we show in Section 6.1.

###  3.2 Retrieval Augmented Label Prompts

It is crucial for the label prompts to reliably express the necessary class
information if each of them is to act as the single similarity anchor for its
class. Unfortunately, this precondition often does not hold as many label
names are being compressive rather than fully descriptive; e.g. “Topic: World”
in the AGNews dataset.

To make amends for the potential issue of fragmented semantics in the labels
in their original format, we augment the label prompts with semantically
similar sentences retrieved from external knowledge sources. Texts collected
from the wild are expectedly more descriptive. Therefore, by serving as
alternative formats or augmentations to the original label prompts, they may
add useful class-related information, such as synonyms or related expressions.

We retrieve top-$K$ sentences from an external corpus according to the
representational similarity with the given label prompt. Then, we compute the
aggregated score for the retrieved _set of sentences_ as

$\displaystyle\mathrm{P}_{\mathrm{ES}}(y_{m}|$ $\displaystyle
X_{i})\propto\frac{1}{K}\sum_{k=1}^{K}{\operatorname{sim}({\mathbf{h}_{i},\operatorname{topk}(\mathbf{z}_{m}}})^{k})$
(3)

where $K$ refers to preset retrieval size, $\operatorname{topk}$ is an
operator which returns top-k nearest embeddings from encoded external sources,
and k denotes the k-th element from the retrieval results. The sum of Eq. 3.1
and Eq. 3 now becomes our new inference scoring function.

$\underset{y_{m}\in\mathcal{Y}}{\operatorname{argmax}}~{}~{}\mathrm{P}_{\mathrm{TS}}(y_{m}|X_{i})+\mathrm{P}_{\mathrm{ES}}(y_{m}|X_{i})$

For further augmentation during retrieval, we experiment with using synonyms
of the original label names (e.g. “good”, “remarkable” for “great”) as in Shi
et al. (2022). Given a label prompt, we first generate a size-$N$ list of
synonymous label prompts by replacing the label name with its synonyms. Then,
we do the retrieval for $N$ times, each time with a formerly generated
synonymous prompt as the query and $K/N$ retrieved sentences.

Method | # Params | SST-2 | MR | CR | RT | Yahoo | AGNews | Avg
---|---|---|---|---|---|---|---|---
Baselines (Language Modeling) |  |  |  |  |  |  |  |
GPT-2† | 774M | 55.3 | 54.6 | 66.2 | 53.0 | 49.7 | 67.4 | 57.7
\+ PMI† (Holtzman et al., 2021) | 774M | 76.5 | 74.6 | 82.8 | 74.1 | 48.8 | 65.1 | 70.3
GPT-3† | 175B | 63.6 | 57.4 | 53.8 | 57.0 | 53.1 | 75.4 | 60.1
\+ PMI† (Holtzman et al., 2021) | 175B | 71.4 | 76.3 | 70.0 | 75.5 | 54.7 | 74.7 | 70.4
LM-BFF (Gao et al., 2021a) | 355M | 83.2 | 80.8 | 79.6 | 82.6 | 44.9 | 68.3 | 73.2
Baselines (Retrieval-Augmented LM) |  |  |  |  |  |  |  |
GPT-2 + kNN† (Khandelwal et al., 2019) | 774M | 55.4 | 56.4 | 67.2 | 54.5 | 49.5 | 67.0 | 58.3
GPT-2 + kNN-Prompt† (Shi et al., 2022) | 774M | 84.2 | 78.2 | 84.3 | 80.6 | 51.0 | 78.8 | 76.2
Baselines (NLI format) |  |  |  |  |  |  |  |
Entailment (Yin et al., 2019) | 355M | 83.7 | 79.6 | 83.8 | 78.2 | 46.0 | 75.1 | 74.4
Ours (Sentence Encoder) |  |  |  |  |  |  |  |
RaLP${}_{\text{SimCSE}}$ | 355M | 82.3 | 78.0 | 87.1 | 76.7 | 57.0 | 72.6 | 75.6
RaLP${}_{\text{ST5}}$ | 335M | 87.8 | 81.7 | 87.4 | 82.4 | 57.4 | 76.6 | 78.9
RaLP${}_{\text{ST5-XL}}$ | 1.24B | 88.6 | 82.8 | 86.1 | 83.3 | 56.2 | 75.1 | 78.7
RaLP${}_{\text{ST5-XXL}}$ | 4.8B | 90.5 | 84.7 | 87.7 | 85.1 | 54.9 | 75.4 | 79.8
Table 1: Results for closed-set classification tasks under the zero-shot
setting. The Boldface indicates the best performances in each column except
the models with over 1B parameters. The Underline shows the best scores in
each column regardless of model size. Values with $\dagger$ are taken from Min
et al. (2022c).

##  4 Experimental Settings

We experiment with two types of tasks: closed-set classification and multiple
choice QA. Both involve choosing the corresponding label given a finite label
set for each test sample; what tells them apart is whether all test instances
share the same label set (closed-set classification) or each test instance is
presented with a different each label set (multiple choice QA).

###  4.1 Closed-set Classification Datasets

#### Sentiment Analysis

SST-2 (Socher et al., 2013), MR (Pang and Lee, 2004), CR (Hu and Liu, 2004),
and Rotten Tomatoes (RT) (Socher et al., 2013).

#### Topic Classification

AGNews (news domain) and Yahoo Answers (Yahoo; web domain) (Zhang et al.,
2015).

#### Details

We experiment on the modified test set used in Shi et al. (2022); Min et al.
(2022c).222They made a random subset of 3,000 samples for datasets with larger
original test sets. We report the scores averaged over four templates (Gao et
al., 2021a; Min et al., 2022a). See Table 6 for the list of verbalizers used.

###  4.2 Multiple Choice QA Datasets

We evaluate on RACE-M (R-M), RACE-H (R-H) (Lai et al., 2017), ARC-E, ARC-C
(Clark et al., 2018), Open Book Question Answering (OBQA) (Mihaylov et al.,
2018), and CommonsenseQA (CoQA) (Talmor et al., 2019). For CoQA, we report the
validation results as the official test set is not publicly available. We use
a single template from Holtzman et al. (2021). See Table 7 for details on
templates and verbalizers.

###  4.3 Baseline Models

Standalone decoder language models (LM) select the label whose word sequence
has with the highest probability. We use gpt-2-large (Radford et al., 2019)
and variants of gpt-3 (Brown et al., 2020). PMI Holtzman et al. (2021)
calibrates the decoder models with a domain-conditioned premise. We also
implement encoder-based cloze-style zero-shot classification task Gao et al.
(2021a) (LM-BFF) with roberta-large (Liu et al., 2019) as the encoder.

For retrieval-augmented baselines, we use kNN-LM Khandelwal et al. (2019) and
kNN-prompt Shi et al. (2022). These models adjust the output token
probabilities with external resources.

Yin et al. (2019) regards label candidates as hypotheses and predicts the
label with the highest entailment logit score (Entailment). For
implementation, we finetune pre-trained roberta-large (Liu et al., 2019) on
NLI (SNLI + MNLI) datasets (Bowman et al., 2015; Williams et al., 2018).333We
use the script provided by sentence-transformers (Reimers and Gurevych, 2019).

###  4.4 Implementation Details

#### Sentence Encoder

For the main experiments, We use two off-the-shelf sentence encoder models:
sup-simcse-roberta-large (SimCSE) (Gao et al., 2021b) trained on NLI, and
sentence-t5-large (ST5) (Ni et al., 2022) trained on CommunityQA and NLI.
Among the ST5 variations, we use an encoder-only mean version. We also
experiment with larger versions of ST5 (sentence-t5-xl (ST5-XL) and
sentence-t5-xxl (ST5-XXL)) to see the effect of model size.

#### External Corpus

We collect the external corpus used in Shi et al. (2022) which consists of
Wiki103, IMDB, subsets of CC-News, and Amazon Review.

#### Retrieval

We use the same sentence encoders as Section 4.4 for building the embedding
index, and use FAISS (Johnson et al., 2019) for the top-k computation and
retrieval. For each original label prompt, we use 5 synonymous prompts and
retrieve 5 sentences per query (so that K=25) in closed-set classification.
See Table 9 and Table 10 for a list of retrieved examples. For multiple choice
tasks, we use 25 retrieved sentences for a single query. We do not use
synonymous label prompts as making small changes in the wordings may deviate
from the originally intended semantics, thus making the option less of a
viable answer. We observe no further gain beyond 25 retrieved samples.

##  5 Results

Our method shows solid performance on both task types under zero-shot setting.

###  5.1 Closed-set Classification

Table 1 shows the results for closed-set classification tasks.

#### Baselines

On average, KNN-prompt shows the strongest results among all baselines. This
indicates the importance of extracting and exploiting relevant information
from external knowledge sources in zero-shot inference. Among the baselines
that do not make use of external resources, LM-BFF outperforms the standalone
decoder models with or without PMI calibration, presumably due to utilizing
bi-directional information with an encoder.

#### RaLP

Our method shows strong performance when coupled with a well-built sentence
encoder. With the ST5 backbones, we outperform all the baselines despite fewer
parameters.444We attribute ST5’s strong zero-shot classification abilities to
its architecture rather than the pretraining dataset composition. See appendix
A for details. Changing the sentence encoder to SimCSE results in a lower
performance, but still comparable to LM-BFF.

RaLP serves as a cost-effective method of leveraging retrieval from external
sources. Although KNN-Prompt (Shi et al., 2022) depends on the same external
corpus as ours, RaLP outperforms KNN-Prompt not only in terms of task
accuracy555RaLP${}_{\text{ST5}}$ (335M) reaches a higher performance than KNN-
Prompt which has twice as more parameters (774M))., but also in terms of
computational efficiency. Specifically, whereas KNN-Prompt requires token-
level encoding of the leftward context and the next token, we employ a much
lighter instance-level encoding. Such difference in the encoding strategy
allows RaLP to operate on merely 2M items while KNN-Prompt needs to store 284M
items, taking external corpus mentioned in Section 4.4.

Retrieval Strategy | Avg
---|---
ST5 |
no retrieval | 77.4
retrieval \w single query | 78.4
retrieval \w multiple synonymous queries | 78.9
SimCSE |
no retrieval | 73.6
retrieval \w single query | 74.7
retrieval \w multiple synonymous queries | 75.6
Table 2: Retrieval ablation results.

Method | # Params | R-M | R-H | ARC-E | ARC-C | OBQA | CoQA | Avg
---|---|---|---|---|---|---|---|---
Baselines (Language Modeling) |  |  |  |  |  |  |  |
GPT-2 | 774M | 39.3 | 31.8 | 52.7 | 23.1 | 19.4 | 33.3 | 33.3
GPT-3† | 6.7B | 43.3 | 34.8 | 58.2 | 26.8 | 22.4 | 40.0 | 37.6
13B | 49.6 | 38.2 | 66.2 | 32.1 | 28.2 | 48.8 | 43.9
175B | 55.7 | 42.4 | 73.5 | 40.2 | 33.2 | 61.0 | 51.0
GPT-2 + PMI (Holtzman et al., 2021) | 774M | 43.9 | 38.3 | 47.0 | 31.6 | 43.2 | 44.5 | 41.4
GPT-3 + PMI† (Holtzman et al., 2021) | 6.7B | 48.5 | 39.8 | 51.5 | 33.0 | 48.0 | 50.3 | 45.2
13B | 51.3 | 42.1 | 57.7 | 38.5 | 50.4 | 58.5 | 49.8
175B | 55.7 | 43.7 | 63.3 | 45.5 | 58.0 | 66.7 | 55.5
Baselines (NLI format) |  |  |  |  |  |  |  |
Entailment (Yin et al., 2019) | 355M | 39.1 | 29.9 | 45.9 | 31.1 | 42.8 | 35.1 | 37.3
Ours (Sentence Encoder) |  |  |  |  |  |  |  |
RaLP${}_{\text{SimCSE}}$ | 355M | 39.4 | 35.1 | 48.3 | 26.5 | 38.2 | 47.3 | 39.1
RaLP${}_{\text{ST5}}$ | 335M | 40.5 | 38.3 | 56.2 | 28.7 | 44.4 | 55.5 | 43.9
RaLP${}_{\text{ST5-XL}}$ | 1.24B | 43.7 | 40.3 | 63.8 | 37.0 | 48.4 | 59.0 | 48.7
RaLP${}_{\text{ST5-XXL}}$ | 4.8B | 45.9 | 42.3 | 69.3 | 44.5 | 52.8 | 63.9 | 53.1

Table 3: Results for multiple choice tasks under zero-shot setting. The
Boldface indicates the best performances in each column except the models with
175B parameters. The Underline shows the best scores in each column regardless
of model size. We reproduce GPT-2 and GPT-2 + PMI with the code provided by
Holtzman et al. (2021). The symbol $\dagger$ indicates the performance
reported by Holtzman et al. (2021).

#### Ablation on Label Prompt Augmentation

We evaluate the effects of using additional label prompts on top of the
original prompt. From Table 2, we observe that using supplementary label
information results in solid performance increase. Our full augmentation
setting where multiple synonymous prompts are used as retrieval queries
(“retrieval \w multiple synonymous queries”) shows a solid performance
increase over the ablative baselines of a narrower form of retrieval
(“retrieval \w single query”) or none at all (“no retrieval”). This is
especially true for topic classification tasks; in particular, removing the
retrieval component in AGNews decreases the accuracy by 6.2 points
($76.6\rightarrow 70.4$) 666For performance gap on other datasets, see
Appendix C.

###  5.2 Multiple Choice QA

Table 3 shows the results for multiple choice QA tasks.

#### Baselines

In sharp contrast to the results in Section 5.1, the decoder approaches show
strong performance with dramatic gains from upscaling. The highest scores come
from a giant decoder model (175B GPT-3+PMI), presumably due to massive
memorization of specific knowledge.

#### RaLP

Still, our method based on sentence encoders still achieves better performance
than the baselines with an equivalent (or larger) number of parameters.
RaLP${}_{\text{ST5}}$ (with 335M parameters) outperforms GPT-2+PMI (which has
the best performance with $\leq$ 774M parameters) by 2.5 points in average.
Our best model RaLP${}_{\text{ST5-XXL}}$ (with 4.8B parameters) outperforms
much larger GPT-3+PMI baselines (with 6.7B and 13B parameters) by 7.9 and 3.3
points in average; even on par with a 175B GPT-3 despite having 36.5x fewer
parameters.

##  6 Analysis

###  6.1 Verbalizer Sensitivity Test

Figure 3:  Results with varying verbalizers (6.1) in closed-set classification
datasets. The middle line in each plot marks the mean score across all
templates. Our approach has smaller variance to template variations than Gao
et al. (2021a); Yin et al. (2019).

Existing works based on prompting are vulnerable to verbalizer changes, with
worst-case performances often down to chance-level (Lu et al., 2022). We
suggest that using sentence encoders may be a remedy, as they are trained to
distribute semantically similar samples nearby during contrastive pre-training
and thus could be less sensitive to surface form variations. In this section,
we empirically verify that our method based on sentence encoders is _reliably_
strong.

#### Settings

We measure the variance in performance across a range of paraphrased label
templates while keeping the label words intact: say, changing “It was great.”
for sentiment analysis to “It’s a great thing.” or “That’s great.”. For
paraphrasing, we leverage templates from existing works (Gao et al., 2021a;
Min et al., 2022a) and augmentation techniques (Ma, 2019) including back-
translation (Sennrich et al., 2016) and contextual word embedding-based
augmentation (Kobayashi, 2018). Among the generated candidates, we manually
filter out the ones with semantic distortions (_e.g._ added negations).777See
Table 8 for a full list. We use _roberta-large_ for the baselines (Yin et al.,
2019; Gao et al., 2021a). For fair comparison, we share the same backbone with
the baselines and use sup-simcse-roberta-large (SimCSE) (Gao et al., 2021b) as
our sentence encoder.

#### Results

Figure 3 shows the accuracy distributions over verbalizer variations. As
previously reported (Jiang et al., 2021), cloze-style inference (Gao et al.,
2021a) is overly sensitive to subtle contextual modifications. The Entailment
(Yin et al., 2019) approach is less volatile, but it still suffers from
sporadic performance drops with certain templates as shown in long lower
whiskers in the plots. On the other hand, RaLP has more stable performance
distribution under template variations, and this holds for all datasets.

Figure 4:  Performances of two few-shot approaches (mean and standard
deviation across 50 runs) compared to RaLP. The x-axis represents the number
of training examples per class. All use embeddings from sentence-t5-large.

###  6.2 Comparison with Few-shot Setup

If we allow _some_ supervision (_i.e._ few-shot), there exist alternative
strategies that can be used in combination with sentence encoder
representations: linear probing and prototypical networks (Snell et al., 2017;
Dopierre et al., 2021). Therefore, we figure out whether our zero-shot
approach is still attractive as compared to the two few-shot options.

#### Baselines

Linear probing requires training the classification head (i.e. logistic
classification). The prototypical network treats the average embeddings of
support examples as class prototypes and measures the distance of instance
from these prototypes. For both methods, we randomly choose
K=$\\{2,4,8,12,16\\}$ samples per class from training examples with 50
different seeds and report the averaged score.

#### Results

Figure 4 illustrates that both few-shot learning baselines are volatile on
sample selection variations. Specifically, linear probing exhibits more
variability when given a small number of labeled samples. This may be because
linear probing requires training parameters from scratch. Compared to few-shot
training methods, RaLP shows robust performance when the supervision is
extremely limited (i.e., K $\in[2,4,8]$).

##  7 Related Work

#### Prompting

Brown et al. (2020) reformulate downstream tasks as language modeling with
prompting, where the approach performs remarkably in few or zero-shot setup
and even shows competitive performance to supervised methods (Schick and
Schütze, 2021a; Jiang et al., 2021). Subsequent works validate that even small
language models can make few or zero-shot inferences with prompting (Gao et
al., 2021a; Schick and Schütze, 2021b; Le Scao and Rush, 2021; Logan IV et
al., 2022). However, the performance of prompting methods varies greatly
depending on which prompt is used (Perez et al., 2021; Lu et al., 2022). To
address this issue, a line of works explore calibrating the predictions of
language models for the few-shot inference (Holtzman et al., 2021; Zhao et
al., 2021). Unlike previous prompting methods relying on language modeling
(i.e. token prediction), our framework, which separately encodes label prompts
and text input, is particularly robust to surface form variations.

#### Zero-shot Classification

Prompting methods for zero-shot inference have been actively studied. These
works use the likelihood of verbalizer given input text (Brown et al., 2020;
Holtzman et al., 2021; Zhao et al., 2021), the conditional probability of the
input text given verbalizer (Min et al., 2022a), and masked language model
objective (Gao et al., 2021a) for solving NLU tasks. To facilitate zero-shot
inference, Shi et al. (2022) incorporates information from an external corpus
using a fuzzy verbalizer to adjust the decoding logits. NPM (Min et al.,
2022c), on the other hand, removes the softmax function and only utilizes an
external corpus to retrieve the answer.

Another line of research formulates the classification as a natural language
inference (NLI) task. Yin et al. (2019) solves the classification task by
comparing the entailment logit scores where they concatenate the text input
and label prompt as text pairs. Gera et al. (2022) further improves the
framework by finetuning the model using self-training on unlabeled train data.
SimPTC (Fei et al., 2022), concurrent to our work, propose solving
classification using sentence encoder and clustering method. Their method is
similar to our approach without a retrieval component but differs from our
work in that (1) it requires a chunk of unlabeled test dataset for clustering,
(2) it does not use a retrieval component, (3) it cannot be expanded to other
tasks such as multiple choice QA.

#### Sentence Encoder

Sentence encoders project the sentences to embedding space which can be
applied to various language understanding tasks. (Reimers and Gurevych, 2019;
Du et al., 2021; Gao et al., 2021b; Ni et al., 2022). Generally, sentence
encoders are trained by contrastive representation learning, where the encoder
learns to distribute semantically similar samples nearby in embedding space.
Hence, representations obtained by sentence encoders are well-suited features
for classification in limited supervision setup (Tunstall et al., 2022)
However, existing methods based on sentence encoders train an additional
classification head to solve downstream tasks, hindering them from using fine-
grained representation and being unsuitable for zero-shot inference. In
contrast, our approach utilizes label prompts for the zero-shot classification
which does not incur additional parameters and fully exploits fine-grained
representations.

##  8 Conclusion

This paper introduces RaLP, a metric-based classification framework where the
input under inference is compared to candidate class anchors in metric space.
RaLP uses the label prompts as class anchors in an embedding space of a
sentence encoder pretrained with a contrastive learning objective. To
compensate for underspecified label names, we retrieve the pseudo-label
prompts from external knowledge sources and refine the prediction scores. We
verify that RaLP is a powerful zero-shot classification strategy through an
extensive evaluation on six closed-set text classification tasks and six open-
set multiple-choice QA tasks. Further performance boost by retrieval
demonstrates that class anchor selection is important for zero-shot
classification and can be readily and stably achieved by label prompt
augmentation.

##  9 Limitations

Though RaLP demonstrates its strong performance in zero-shot classification
tasks, the approach has several limitations. First, it is non-trivial to apply
our method to non-classification tasks such as generation. Hence, future works
can explore adapting the sentence encoders or their training methods in
developing language models which have been explored in the context of the
retrieval-augmented language models (Zhong et al., 2021). Moreover, while our
work demonstrates its robust performance compared to some of the few-shot
learning methods such as linear probing or prototypical network, we did not
cover finetuning methods for our approach. Future works can explore tuning the
parameters of the sentence encoders when given limited labeled data Tunstall
et al. (2022).

## References

  * Alex et al. (2021) Neel Alex, Eli Lifland, Lewis Tunstall, Abhishek Thakur, Pegah Maham, C. Jess Riedel, Emmie Hine, Carolyn Ashurst, Paul Sedille, Alexis Carlier, Michael Noetel, and Andreas Stuhlmüller. 2021.  Raft: A real-world few-shot text classification benchmark.  In _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_.
  * Bowman et al. (2015) Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015\.  A large annotated corpus for learning natural language inference.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  In _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_.
  * Chen et al. (2020) Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. 2020.  A simple framework for contrastive learning of visual representations.  In _Proceedings of the International Conference on Machine Learning (ICML)_.
  * Chen et al. (2017) Ting Chen, Yizhou Sun, Yue Shi, and Liangjie Hong. 2017.  On sampling strategies for neural network-based collaborative filtering.  In _Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)_.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_.
  * Dopierre et al. (2021) Thomas Dopierre, Christophe Gravier, and Wilfried Logerais. 2021.  A neural few-shot text classification reality check.  In _Proceedings of the European Chapter of the Association for Computational Linguistics (EACL)_.
  * Du et al. (2021) Jingfei Du, Édouard Grave, Beliz Gunel, Vishrav Chaudhary, Onur Celebi, Michael Auli, Veselin Stoyanov, and Alexis Conneau. 2021.  Self-training improves pre-training for natural language understanding.  In _Proceedings of The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_.
  * Fei et al. (2022) Yu Fei, Ping Nie, Zhao Meng, Roger Wattenhofer, and Mrinmaya Sachan. 2022.  Beyond prompting: Making pre-trained language models better zero-shot learners by clustering representations.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Gao et al. (2021a) Tianyu Gao, Adam Fisch, and Danqi Chen. 2021a.  Making pre-trained language models better few-shot learners.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Gao et al. (2021b) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021b.  Simcse: Simple contrastive learning of sentence embeddings.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Gera et al. (2022) Ariel Gera, Alon Halfon, Eyal Shnarch, Yotam Perlitz, Liat Ein-Dor, and Noam Slonim. 2022.  Zero-shot text classification with self-training.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Henderson et al. (2017) Matthew Henderson, Rami Al-Rfou, Brian Strope, Yun-Hsuan Sung, László Lukács, Ruiqi Guo, Sanjiv Kumar, Balint Miklos, and Ray Kurzweil. 2017.  Efficient natural language response suggestion for smart reply.  _arXiv preprint arXiv:1705.00652_.
  * Holtzman et al. (2021) Ari Holtzman, Peter West, Vered Shwartz, Yejin Choi, and Luke Zettlemoyer. 2021\.  Surface form competition: Why the highest probability answer isn’t always right.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Hu and Liu (2004) Minqing Hu and Bing Liu. 2004.  Mining and summarizing customer reviews.  In _Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD)_.
  * Jiang et al. (2021) Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2021.  How can we know when language models know? on the calibration of language models for question answering.  _Transactions of the Association for Computational Linguistics (TACL)_.
  * Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019.  Billion-scale similarity search with gpus.  _IEEE Transactions on Big Data_.
  * Khandelwal et al. (2019) Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019\.  Generalization through memorization: Nearest neighbor language models.  _arXiv preprint arXiv:1911.00172_.
  * Kobayashi (2018) Sosuke Kobayashi. 2018.  Contextual augmentation: Data augmentation by words with paradigmatic relations.  In _Proceedings of The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_.
  * Lai et al. (2017) Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017.  RACE: Large-scale ReAding comprehension dataset from examinations.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Le Scao and Rush (2021) Teven Le Scao and Alexander Rush. 2021.  How many data points is a prompt worth?  In _Proceedings of The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Logan IV et al. (2022) Robert Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2022.  Cutting down on prompts and parameters: Simple few-shot learning with language models.  In _Findings of the Association for Computational Linguistics: ACL_.
  * Lu et al. (2022) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. 2022\.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Ma (2019) Edward Ma. 2019.  Nlp augmentation.  https://github.com/makcedward/nlpaug.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Min et al. (2022a) Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022a.  Noisy channel language model prompting for few-shot text classification.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Min et al. (2022b) Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. 2022b.  Metaicl: Learning to learn in context.  In _Proceedings of The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_.
  * Min et al. (2022c) Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen-tau Yih, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022c.  Nonparametric masked language modeling.  _arXiv preprint arXiv:2212.01349_.
  * Ni et al. (2022) Jianmo Ni, Gustavo Hernandez Abrego, Noah Constant, Ji Ma, Keith Hall, Daniel Cer, and Yinfei Yang. 2022.  Sentence-t5: Scalable sentence encoders from pre-trained text-to-text models.  In _Findings of the Association for Computational Linguistics: ACL_.
  * Pang and Lee (2004) Bo Pang and Lillian Lee. 2004.  A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Perez et al. (2021) Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021.  True few-shot learning with language models.  In _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multi-task learners.  _Technical report_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _Journal of Machine Learning Research (JMLR)_.
  * Reimers and Gurevych (2019) Nils Reimers and Iryna Gurevych. 2019.  Sentence-bert: Sentence embeddings using siamese bert-networks.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Schick and Schütze (2021a) Timo Schick and Hinrich Schütze. 2021a.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In _Proceedings of the European Chapter of the Association for Computational Linguistics (EACL)_.
  * Schick and Schütze (2021b) Timo Schick and Hinrich Schütze. 2021b.  It’s not just size that matters: Small language models are also few-shot learners.  In _Proceedings of The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Improving neural machine translation models with monolingual data.  In _Proceedings of the Annual Meeting of the Association for Computational Linguistics (ACL)_.
  * Shi et al. (2022) Weijia Shi, Julian Michael, Suchin Gururangan, and Luke Zettlemoyer. 2022.  Nearest neighbor zero-shot inference.  _arXiv preprint arXiv:2205.13792_.
  * Snell et al. (2017) Jake Snell, Kevin Swersky, and Richard Zemel. 2017.  Prototypical networks for few-shot learning.  volume 30.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  Commonsenseqa: A question answering challenge targeting commonsense knowledge.  In _Proceedings of The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_.
  * Tunstall et al. (2022) Lewis Tunstall, Nils Reimers, Unso Eun Seo Jo, Luke Bates, Daniel Korat, Moshe Wasserblat, and Oren Pereg. 2022.  Efficient few-shot learning without prompts.  _arXiv preprint arXiv:2209.11055_.
  * Van der Maaten and Hinton (2008) Laurens Van der Maaten and Geoffrey Hinton. 2008.  Visualizing data using t-sne.  _Journal of machine learning research_.
  * Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018.  A broad-coverage challenge corpus for sentence understanding through inference.  In _Proceedings of The Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL)_.
  * Yin et al. (2019) Wenpeng Yin, Jamaal Hay, and Dan Roth. 2019.  Benchmarking zero-shot text classification: Datasets, evaluation and entailment approach.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _Proceedings of the Advances in Neural Information Processing Systems (NeurIPS)_.
  * Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate before use: Improving few-shot performance of language models.  In _Proceedings of the International Conference on Machine Learning (ICML)_.
  * Zhong et al. (2021) Zexuan Zhong, Tao Lei, and Danqi Chen. 2021.  Training language models with memory augmentation.  In _Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP)_.

## Appendix

##  Appendix A Analyzing Performance Gap between SimCSE and ST5

Method | Dataset | SST-2 | MR | CR | RT | Yahoo | AGNews | Avg
---|---|---|---|---|---|---|---|---
RaLP${}_{\text{ST5}}$ | CommunityQA + NLI | 87.8 | 81.7 | 87.4 | 82.4 | 57.4 | 76.6 | 78.9
RaLP${}_{\text{ST5}}$ | NLI | 85.7 | 80.2 | 87.8 | 80.0 | 58.8 | 76.8 | 78.2
RaLP${}_{\text{SimCSE}}$ | NLI | 82.3 | 78.0 | 87.1 | 76.7 | 57.0 | 72.6 | 75.6
Table 4: Results for ablation study of pre-training dataset. We additionally
trained the ST5 model using only NLI dataset.

In Section 5.1, we evaluate the performance of various pre-trained sentence
encoders using RaLP. Our results show that RaLP${}_{\text{SimCSE}}$ (sup-
simcse-roberta-large) has a lower performance than RaLP${}_{\text{ST5}}$
(sentence-t5-large although they have the same parameter size. We identify two
main differences between SimCSE and ST5: i) the pre-trained model utilized as
the initial checkpoint, and ii) the utilization of solely the NLI datasets in
SimCSE as opposed to a combination of NLI and CommunityQA in ST5. Since the
CommunityQA dataset is not publicly available, analyzing the reasons for this
performance gap is important for future research on sentence encoders for
zero-shot text classification. To do so, we conduct experiments by comparing
the two encoder models.

For a fair comparison, we train the ST5 model using only the NLI datasets.
Following the original setting in Ni et al. (2022), we employ T5-large (Raffel
et al., 2020) as the initial pre-trained model and evaluate its performance
through zero-shot text classification datasets. As shown in Table 4, the
performance of the RaLP${}_{\text{ST5}}$ trained with the combination of
CommunityQA and NLI achieved comparable performance with RaLP${}_{\text{ST5}}$
model trained using only the NLI dataset. However, among models trained using
only NLI, RaLP${}_{\text{ST5}}$ outperforms RaLP${}_{\text{SimCSE}}$ by 2.6
points on average. This suggests that the performance gap observed in Table 1
is primarily influenced by the T5 architecture rather than the CommunityQA
dataset. We assume that the objective of reconstructing the consecutive span
of corrupted tokens from one unique mask token (i.e. span masking) used in the
pre-training of T5 has enhanced the model’s general-purpose knowledge more
than the objective of BERT-style (i.e. token masking).

Method | # Params | SST-2 | MR | CR | RT | Yahoo | AGNews | Avg
---|---|---|---|---|---|---|---|---
Ours (Sentence Encoder) |  |  |  |  |  |  |  |
RaLP${}_{\text{SimCSE}}$ | 355M | 82.3 | 78.0 | 87.1 | 76.7 | 57.0 | 72.6 | 75.6
\- no retrieval | 355M | 80.8 | 76.5 | 87.5 | 75.7 | 53.4 | 67.6 | 73.6
RaLP${}_{\text{ST5}}$ | 335M | 87.8 | 81.7 | 87.4 | 82.4 | 57.4 | 76.6 | 78.9
\- no retrieval | 355M | 87.7 | 81.7 | 87.3 | 81.9 | 55.3 | 70.4 | 77.4
Table 5: Retrieval ablation full results.

Dataset | Verbalizers
---|---
SST-2, MR, CR, RT | A MASK one.; It was MASK.; All in all MASK.; A MASK piece.
| (MASK = {great, terrible})
AGNews | Topic: MASK.; Subject: MASK.; This is about MASK.; It is about MASK.
| (MASK = {World, Sports, Business, Technology})
Yahoo | (Same as above) (MASK = {Company, Educational Institution, Artist, Athlete,
| Office Holder, Mean of Transportation, Building, Natural Place, Village,
Animal,
| Plant, Album, Film, Written Work})

Table 6: The details of verbalizer setting in text classification task. We
follow the verbalizers from Gao et al. (2021a); Min et al. (2022a)

Dataset |  Premise $x$ |  Hypothesis $y$
---|---|---
RACE |  There is not enough oil in the world now. As time goes by, it becomes less and less, so what are we going to do when it runs out[…].] question: According to the passage, which of the following statements is true? |  answer: There is more petroleum than we can use now.
ARC |  What carries oxygen throughout the body? |  the answer is: red blood cells.
OBQA |  Which of these would let the most heat travel through? |  the answer is: a steel spoon in a cafeteria.
CoQA |  Where can I stand on a river to see water falling without getting wet? |  the answer is: bridge.

Table 7:  The details of verbalizer setting in multiple choice datasets. The
hypothesis candidates are directly given as label prompt in each task. The
texts with blue color denote templates that we prepend following Holtzman et
al. (2021)

.

Dataset | Verbalizers
---|---
SST-2, MR, CR, RT | It was MASK.; A MASK piece.; A MASK one.;
| All in all MASK.; It was absolutely MASK, really.;
| It was really so MASK.; It was more than just MASK.;
| And that was absolutely MASK, too.; It was still pretty MASK.;
| It was all MASK.; It was literally MASK.; It’s a MASK thing.;
| What a MASK performance.; It is an MASK piece at the best of times.;
| One MASK piece.; A MASK work.; A MASK play.;
| This is a MASK one.; It is utterly MASK.; A really MASK one.;
| That’s MASK.; That’s all MASK.; All told this is a truly MASK thing.;
| A MASK overall.; All together MASK.
| (MASK = {great, terrible})
AGNews | Topic: MASK.; Subject: MASK.; This is about MASK.;
| It is about MASK.; It’s about the MASK.;
| It’s about MASK.; It’s all about the MASK.;
| It’s just about the MASK.; It’s the whole lot with the MASK.;
| Here, we are talking about the MASK.; This is the theme of the MASK.;
| It is related to the MASK.; It is about what it means for the MASK.;
| This involves the MASK.; Theme: MASK.;
| keyword: MASK.; On a related topic: the MASK.;
| It is for the MASK.; The subject: the MASK.;
| Main topic: MASK.; Content: MASK.;
| Theme is the MASK.; Issue: MASK.;
| Executive Summary: MASK.; Material: MASK.
| (MASK = World, Sports, Business, Technology)
Yahoo | (Same as above) (MASK = {Company, Educational Institution, Artist, Athlete,
| Office Holder, Mean of Transportation, Building, Natural Place, Village,
Animal,
| Plant, Album, Film, Written Work})

Table 8: We use all listed verbalizers in verbalizer variation test in Section
6.1. We evaluate on total 25 verbalizer with or without punctuation.

Original Label Prompt | Retrieval Augmented Label Prompts
---|---
It was great. | great, love it - great, love it - great, love it - great, love it really did love it and it was also great ;
| It was really good! I reallly liked it! As does my father and mother and
brother and sister-in-law. It was epic. ;
| i loved it. it could not have been better. justin kirby ;
| it was awesome, maiden is at there best, is was totally awesomeness, nobody
comes close to maiden, it’s a totally fulfillment ;
| it was awesome!!!!! of course it was awesome. its Harry Potter for lord’s
sake!!!!
It was good. | A <unk> . " A is good " , or " A was good " . ;
| great, love it - great, love it - great, love it - great, love it really did
love it and it was also great ;
| It was ok. Wish I could give this a great review, sorry. But, it was still
ok, if you’re interested. ;
| It was alright having not seen it for many years i enjoyed watching it and
am glad for the purchase ;
| It was really good! I reallly liked it! As does my father and mother and
brother and sister-in-law. It was epic.
It was famous. | = = = In popular memory = = = ;
| = = Famous people who witnessed it = = ;
| = = = National and international fame = = = ;
| This is a real classic, with a number of actors and actresses that were to
become very famous later on. ;
| This classic was very innovative for it’s time and is a classic.The Princess
created quite a stir at the time, and a cult following.
It was terrible. | terrible, terrible.It wasm’t worth the money or the time to watch it. we turned it off in the middle of it ;
| It was horrible. One of the worst books ever. Thank youBelly Up ;
| This book was terrible and was horrific when it came to the description. ;
| It was awful. I went into it with great hopes expecting to enjoy it. I
wanted to like it, I wanted to laugh. I spent more time rolling
| my eyes and checking the counter to see how much time was left till it was
over. I was so disappointed. ;
| Just poor quality and I just didn’t like it.Really couldn’t pay attention
because it was terrible, awful movie I wouldn’t recommend.
It was horrible. | It was horrible. One of the worst books ever. Thank youBelly Up ;
| This book was terrible and was horrific when it came to the description. ;
| It was sick, depressing, annoying, and not to mention rotten! so that’s what
I have to say on the subject! ;
| I hated it and thought it was really bad. It was twisted and is really scary
to litte children ;
| truth be told, this is a horrible story of what really happened - so very
awful…it was impossible to read
It was awful. | It was horrible. One of the worst books ever. Thank youBelly Up ;
| It was awful. I went into it with great hopes expecting to enjoy it. I
wanted to like it, I wanted to laugh. I spent more time rolling
| my eyes and checking the counter to see how much time was left till it was
over. I was so disappointed. ;
| This book was terrible and was horrific when it came to the description. ;
| It was sick, depressing, annoying, and not to mention rotten! so that’s what
I have to say on the subject! ;
| The movie was truly a wretched experience. The six letters to describe it
are S-U-C-K-E-D!

Table 9: Original label prompt in sentiment analysis task (SST-2, MR, CR, and
RT) and retrieval augmented label prompt in Section 5.1.

Original Label Prompt | Retrieval Augmented Label Prompts
---|---
Topic: world. | = = = = Worldwide = = = = ; = = = = Informing the world = = = = ;
| = = = On the Wisdom of this World = = = ; = State of the World ( song ) = ;
| = = = = HELLO , WORLD = = = =
Topic: international. | = = = = International = = = = ; = = = = International affairs = = = = ;
| = = = = International services = = = = ; = = = = International relations = =
= = ;
| = = = International ( InLine ) = = =
Topic: global. | = = = = Worldwide = = = = ; = = = Global environmental impact = = = ;
| = = = From the global community = = = ; = = = Around the globe = = = ;
| = = = = Terrestrial globes = = = =
Topic: sports. | = = = = Sports = = = = ; = = = = Athletic history = = = = ;
| = = = Effects on sports = = = ; = = = World sport context = = = ;
| = = = = Sport = = = =
Topic: entertainment. | = = = = Entertainment = = = = ; = = = Entertainment and culture = = = ;
| Entertainment is a form of activity that holds the attention and interest of
an audience ,
| or gives pleasure and delight . It can be […] and even for a global audience
. ;
| = = = Sports and entertainment = = = ; = = = Media and entertainment = = =
Topic: recreation. | = = = = Recreations = = = = ; = = = = Recreation = = = = ;
| = = = Sports and recreation = = = ; = = = Sport and recreation = = = ;
| = = = = Tourism and recreation = = = =
Topic: business. | = = = = Business = = = = ; = = = = On labor and business = = = = ;
| = = = Lines of business = = = ; = = = DE 1 Business = = = ;
| = = = Operations and business = = =
Topic: economics. | = = = = Economic lecture = = = = ; = = = = Economics = = = = ;
| = = = Views on economics = = = ; = = = = Economic = = = = ;
| = = = Philosophy of economics = = =
Topic: financial. | = = = = Finances = = = = ; = = = Financial and loan = = = ;
| = = = = Fiscal issues = = = = ; = = = = Financial performance = = = = ;
| = = = Economy and finance = = =
Topic: technology. | = = = Influence on technology = = = ; = = = Views on the technology = = = ;
| = = = = Technical aspects = = = = ; = = = Effects of technology = = = ;
| = = = Technology and science = = =
Topic: science. | Science – news on science @-@ related topics ( e.g. cool technology , space telescope
| observations , interesting medical research ) . ;
| = = = Interactions with the scientific community = = = ; = = = = Scientific
uses = = = = ;
| = = = History of science = = = ; = = = Science and scientism = = =
Topic: mathmatics. | = = = Attention towards mathematics = = = ; = = = = Mathematics = = = = ;
| = = = An Introduction to Mathematics = = = ; = = = Mathematics as an art = =
= ;
| = = = Philosophy of mathematics = = =

Table 10: Original label prompt for AGNews and retrieval augmented label
prompt in Section 5.1.

##  Appendix B Evaluation Protocol

We report the accuracy for all datasets. To optimize hyperparameters, we
exploit the validation dataset for each task. In case of CoQA, we sub-sample
3,000 instances from the training dataset, regarding them as in-house
validation examples.

##  Appendix C Full Results on Retrieval Ablation Study

We report the full results of the retrieval ablation study in Table 5.

##  Appendix D Implementation Details

In all experiments, we evaluate models on a single A100 with 80GB, and A6000
GPU with 48GB of memory. We implement all models with PyTorch using sentence-
transformers library from UKPLab888https://github.com/UKPLab/sentence-
transformers. We choose the best hyperparameter of top-K in
$\\{5,10,25,50,100\\}$ for RaLP based on validation split.