# What Makes Pre-trained Language Models Better Zero/Few-shot Learners?

Jinghui Lu1, Rui Zhao1, Brian Mac Namee2,3, Dongsheng Zhu1,4, Weidong Han1,4,
Fei Tan1 Fei Tan is the corresponding author.

###### Abstract

In this paper, we propose a theoretical framework to explain the efficacy of
prompt learning in zero/few-shot scenarios. First, we prove that conventional
pre-training and fine-tuning paradigm fails in few-shot scenarios due to
overfitting the unrepresentative labelled data. We then detail the assumption
that prompt learning is more effective because it empowers pre-trained
language model that is built upon massive text corpora, as well as domain-
related human knowledge to participate more in prediction and thereby reduces
the impact of limited label information provided by the small training set. We
further hypothesize that language discrepancy can measure the quality of
prompting. Comprehensive experiments are performed to verify our assumptions.
More remarkably, inspired by the theoretical framework, we propose an
annotation-agnostic template selection method based on perplexity, which
enables us to “forecast” the prompting performance in advance. This approach
is especially encouraging because existing work still relies on development
set to post-hoc evaluate templates. Experiments show that this method leads to
significant prediction benefits compared to state-of-the-art zero-shot
methods.

## Introduction

Trial | 1 | 2 | 3 | 4 | 5 | Avg. | All
---|---|---|---|---|---|---|---
Douban | 54.60 | 51.64 | 61.18 | 65.67 | 49.89 | 56.60 _±6.65_ | 53.94
Weibo | 99.20 | 49.94 | 90.39 | 97.43 | 50.05 | 77.40 _±25.23_ | 96.76
Waimai | 81.64 | 71.09 | 50.27 | 81.64 | 79.31 | 72.79 _±13.31_ | 81.09
Online-shopping | 87.15 | 85.03 | 86.11 | 86.74 | 86.06 | 86.22 _±0.80_ | 86.38

Table 1: Results of conventional fine-tuning on four Chinese sentiment
analysis datasets. _Avg._ denotes the aggregated results of random seed 1 to
5. _All_ denotes the model is fine-tuned with all the labelled examples from
five splits.

Pre-training and fine-tuning has become a de facto paradigm for almost all NLP
applications (Devlin et al. 2018; Liu et al. 2019; Raffel et al. 2020).
Despite the superior performance in fully supervised settings, fine-tuning
still suffers from issues such as unstable performance and unsatisfactory
generalisability (Wang et al. 2020; Liu et al. 2021a; Xu et al. 2021). As
exemplified in Table 1 which is the results of few-shot fine-tuning for four
sentiment analysis datasets, we can observe that the performance is prone to
fluctuate significantly with different initial few-shot training set splits.

Prompt learning has been demonstrated to be a successful remedy for pre-
training and fine-tuning paradigm, especially in zero/few-shot scenarios (Gao,
Fisch, and Chen 2021; Schick and Schütze 2021a, b; Sun et al. 2021; Tam et al.
2021). Prompt learning originates from the practice that manipulates input
texts to steer the model produce desired outputs. Researchers found out that
huge language models like GPT-3 (Brown et al. 2020) is too expensive to fine-
tuning, instead, they prefixed inputs with a few training examples (in natural
language) so that the model can generalise to unseen cases, which is called
in-context learning (Brown et al. 2020; Xie et al. 2021; Liu et al. 2022a).

The essence of prompt learning is making better use of pre-trained language
model by adding additional “hints” (Liu et al. 2021a). Inspired by this, a
multitude of pioneering approaches are proposed and discover that it is not
just autoregressive model like GPT family can take advantage of prompt
learning. For example, bidirectional encoder-only models like BERT family
(Devlin et al. 2018; Liu et al. 2019) also show impressive zero/few-shot
capacity by reformulating downstream task into pre-training tasks such as next
sentence prediction (Sun et al. 2021) or masked language model (MLM) (which is
called cloze-style prompt learning) (Gao, Fisch, and Chen 2021; Schick and
Schütze 2021a, b; Tam et al. 2021). Besides, encoder-decoder models like T5
(Raffel et al. 2020) or BART (Lewis et al. 2020) can leverage prompt learning
to reuse a versatile model for multiple tasks as well as demonstrate more
effective few-shot performance (Khashabi et al. 2020; Lester, Al-Rfou, and
Constant 2021).

Despite of its success, the underlying rationales of prompt learning is still
under explored. Most of theoretical research focuses on in-context prompt
learning based on causal language models like GPT-3 (Xie et al. 2021; Liu et
al. 2022a) instead of cloze-style prompting for bidirectional models.

In this work, we first describe basic concepts, terminologies, scope of prompt
learning discussed in this work. We illustrate that conventional fine-tuning
models are highly relied on the “representativeness” of labelled data, and
thus suffer from instability and can not generalise well in few-shot
scenarios. We then propose a framework in an attempt to elucidate the efficacy
of prompt learning. The framework consists of two assumptions: (1) prompt
learning is more effective because it much reduces dependence on labelled
data, instead, resorts to human knowledge and implicit information encoded in
language models; (2) language discrepancy can measure the quality of prompting
setting. Comprehensive experiments are conducted to empirically verify these
assumptions.

Furthermore, derived from the analysis of our theoretical framework, we
propose using perplexity to prior select templates. Experimental results
demonstrate this approach can significantly boost the prediction performance.
This finding is especially encouraging because most of existing work still
leverages development set to post-hoc evaluate prompting templates (Gao,
Fisch, and Chen 2021; Liu et al. 2021a).

To the best of our knowledge, we are the first to explore the theoretical
understanding behind the efficacy of cloze-style prompt learning, and propose
using perplexity to “forecast” effectiveness of prompting templates.111Code is
available at URL masked for anonymous review.

## Preliminaries

### Prompt Learning Description

Note that the prompting settings and terminologies used in this work are
mainly derived from those fundamental prompt learning work that focuses on
manual/automatic cloze-style discrete templates (Gao, Fisch, and Chen 2021;
Schick and Schütze 2021a, b; Tam et al. 2021) instead of continuous templates
(Lester, Al-Rfou, and Constant 2021; Liu et al. 2021b; Li and Liang 2021; Liu
et al. 2022b) or in-context prompting used in GPT3. We leave the extension of
this framework to other prompting fashions for future work.

As text classification is well studied in prompt-based learning tasks (Liu et
al. 2021a), we use a simple binary sentiment analysis task to demonstrate the
difference between conventional fine-tuning and prompting procedure.
Specifically, given an input text $x=$“I love this movie.”, we are interested
in the sentiment polarity $y$ of this input sequence, i.e., $++$ for positive
or $--$ for negative. Conventional fine-tuning removes the pre-training
prediction layer (e.g., MLM head) and entails a randomly initialised binary
prediction layer on top of the language model to predict the sentiment of $x$
(e.g., 0 for $--$ and 1 for $++$), and then updates all parameters of models
including the newly added layer through back propagation.

Prompt learning, instead of dropping the pre-trained head, modifies input $x$
and output $y$ to further exploit pre-trained language models which is
described as follows: Formally, we first manipulate input $x$ to construct a
new input text $x^{\prime}$ by prefixing (or postfixing) $x$ with a template
text sequence $t$ that includes a “[MASK]” token in it. In other words,
$x^{\prime}=[x,t]$ or $x^{\prime}=[t,x]$. For example, if we have an input
$x=$“I love this movie.” and we decide to prefix a template $t=$“Overall, it
was a [MASK] movie.”, $x^{\prime}$ will become “Overall, it was a [MASK]
movie. I love this movie.”.

Then, $x^{\prime}$ is fed into a language model to predict the probability of
the token that fills “[MASK]”, which can be solved by applying MLM head.
Usually, researchers limit the search space of filled tokens relevant to the
domain of interest with prior knowledge. For example, in this case we only
care about the filling probability of two words “good” and “bad”. We call
these two words label word $w$ (Liu et al. 2021a).

Finally, we define a mapping function {good:$++$, bad:$--$}, which is called
verbaliser $v$ (Liu et al. 2021a), to reverse predicted label word back to
target $y$. By this means, the prompting method unifies a binary
classification objective into a MLM objective, reusing pre-trained language
model and MLM head to achieve better few-shot performance or even zero-shot
capability. There are plenty of ways to optimise promting model such as
updating all parameters (Schick and Schütze 2021a, b), freezing most of
parameters and only tuning a small subset of parameters like P-tuning (Liu et
al. 2021b, 2022b) or Prompt Tuning (Lester, Al-Rfou, and Constant 2021). In
this work, we choose to optimise the whole model in line with Schick and
Schütze (2021a) to reflect the full potential of prompting. Thus, the
subsequent conclusion is probably subject to the chosen few-shot prompting
setting. Table 2 summarises the terminologies, notations, examples and
descriptions of above prompting procedure.

Name | Notation | Example | Description
---|---|---|---
Input | $x$ | I love this movie. | Input text sequence
Output | $y$ |  $++$ (positive), $--$ (negative) | Output target
Template | $t$ | Overall, it was a [MASK] movie. | Template text sequence that contains [MASK]
Prompted input | $x^{\prime}$ | Overall, it was a [MASK] movie. I love this movie. | Input text sequence after prompting
Label word | $w$ | good, bad | Word that fills [MASK]
Verbaliser | $v$ | {good:$++$, bad:$--$} | Function that reverses label word back to target

Table 2: Summary of terminologies, notations, examples and descriptions used
in this work, adapted from Liu et al. (2021a).

### Objective Gap and Language Discrepancy

Previous work (Liu et al. 2021a) believes that by closing the gap between pre-
training and downstream task, prompt learning allows pre-trained language
models to take more responsibility in prediction rather just extracting
features. Developed from the above statement, we define that there are two
obstacles between pre-training and downstream task, that is, objective gap and
language discrepancy. Objective gap describes the difference in training
objectives between pre-training (e.g., next sentence prediction or MLM) and
downstream task (e.g., sequence classification or sequence labelling).
Language discrepancy mainly indicates the linguistic difference between pre-
training corpora and downstream datasets including different domain
vocabularies, word frequencies, syntactic arrangements, etc.

## Theoretical Framework

In this section, we aim to provide a theoretical framework to explain the
efficacy of prompt learning in zero/few-shot scenarios. First, we elaborate
that conventional fine-tuning is prone to overfit the unrepresentative
training data. Then, we detail the assumptions including the explanation of
efficacy of prompt learning in few-shot scenarios, and the using language
discrepancy to measure prompting quality.

(a) True Label Distribution

(b) Ideal Few-shot Labelling

(c) Problematic Few-shot Labelling

Figure 1: An example of a true data distribution for a downstream dataset is
illustrated in Figure 1(a). Figures 1(b) and 1(c) show the two few-shot cases
where different subsets are selected from the entire data pool and labelled,
resulting in classifiers with quite different decision boundaries. In all
plots the entire pool of examples is shown. : data point of class A; : data
point of class B; : unlabelled data point; - - -: decision boundaries drawn by
classifiers.

(a) Problematic Few-shot Labelling

(b) Pre-trained Feature Space

(c) Optimised Feature Space

Figure 2: Figure 2(a) shows an example of unrepresentative label distribution,
the true distribution can be referred to Figure 1(a). Figure 2(b) shows an
example that unrepresentative labelled data are mapped to pre-trained feature
space via prompting. The green/yellow background color indicates pre-trained
feature space denoting different domain of interest and the dashed line is the
decision boundary. The new feature space is determined by pre-trained language
models and prompting instead of by labelled data. Figure 2(c) shows further
optimising the model based on prompting data. : different pre-trained feature
space.

### Why Fine-tuning Fails in Few-shot?

Previous work (Xu et al. 2021; Schick and Schütze 2021a) reports that the
performance of fine-tuning is highly sensitive to the initial training
examples as exemplified in Table 1.

Given the above observations, we argue that the failure of conventional fine-
tuning is caused by overfitting the unrepresentative training data.
Concretely, a few-shot scenario usually supposes that we do not have
sufficient labelled data, resulting in a less representative training set as
illustrated in Figure 1.

Figure 1(a) is the true distribution of an entire data pool for a downstream
dataset and we can easily find out we need at least two non-linear hyper-
planes to well separate all data into two classes (red dots for class A and
blue dots for class B). Figure 1(b) demonstrates a few-shot case where
annotators randomly pick those data points close to the “true” hyper-planes
with luck and label them. This helps build up a classifier that produces
reasonably good decision boundaries to separate data into two classes, i.e.,
data points in the middle of two decision boundaries are categorised into
class A and other data points are categorised into class B. We call these
available annotated data resembling the true data distribution “representative
data”.

In contrast, in Figure 1(c), annotators randomly pick those data points only
at the upper right corner. The fine-tuned model is probably to be biased and
thereby produce a decision boundary that focuses on separating data at the
upper right corner while ignores the rest data, i.e., data on the left side of
the decision boundary is classified as class A and data on the right side of
the decision boundary is classified as class B. In this case, the
generalisation performance is unlikely to be as appealing as that in Fig. 1(b)
since all class B data at the bottom left corner is misclassified as class A,
and we consider these labelled data to be “unrepresentative data”.

To conclude, the performance of fine-tuned models are highly relied on the
representativeness of the labelled data, which also explains the unstable
performance and suboptimal generalisation of fine-tuning in few-shot
scenarios.

Sanity Check: To briefly substantiate our argument, we conduct a quick
evaluation experiment where we fine-tune pre-trained
RoBERTa222https://huggingface.co/hfl/chinese-roberta-wwm-ext on four Chinese
sentiment analysis datasets which are further described in Section Datasets.
We adopt a de facto few-shot setting where the model is trained with five
different training set splits where each class has only 16 labelled examples
(thus we have 32 labelled examples for each training set since this is a
binary classification task).

We fine-tune all models for 20 epochs using batch size 2 and learning rate
1e-4 to ensure the model is “overfitting” the small training set. We also
fine-tune models with a larger training set consisting of all labelled
examples from the five few-shot training sets (i.e., around 160 labelled
examples after removing duplicates) to simulate training scenarios with
sufficient labelled examples, which is denoted as _All_.

As shown in Table 1, the variance of the results with different training
splits can be significant. All datasets are suffered from the unstable
performance where the standard deviations are large ranging from 6.65 in
Douban to 25.25 in Weibo except for Online-shopping. For example, if we see
Trial 1 in Weibo, the accuracy impressively achieves 99.20 using only 32
labelled examples while in Trial 2, the accuracy dramatically drops down to
49.94 which is close to a random guess. These observations verify that the
few-shot fine-tuning performance is highly correlated to the
representativeness of initial training set. To put it more explicitly, in few-
shot fine-tuning, if we luckily label those representative data whose
distribution highly resembles that of testing data (e.g., Trial 1 in Weibo),
the results can be significantly higher. In contrast, labelling those
unrepresentative data (e.g., Trial 2 in Weibo) has limited contributions to
the prediction.

We can also find out that _All_ is generally better than _Avg._ in almost all
datasets except for Douban, which can also be explained by the
“representativeness” of the labelled data. The larger number of labelled
examples increases the probability of sampling representative labelled data,
resulting better results. However, a larger training set can not always
guarantee a better performance, as evidence by some cases where _All_ (using
around 160 labelled examples) is not better than some specific trials using
only 32 labelled examples, e.g., Douban Trial 4 (65.67) vs. _All_ (53.94).
These findings again show the importance of picking representative data to be
labelled rather than randomly picking unrepresentative data in conventional
fine-tuning.

### Why Prompting Works in Few-shot?

Through the above analysis, we find out the few-shot fine-tuning performance
is limited by the representativeness of labelled data. In this section, we
propose a theoretical framework to explain the efficacy of prompt learning for
zero/few-shot scenarios, consisting of two key assumptions as follows:

  * •

_Prompt learning is more useful in zero/few-shot scenarios because it
revolutionises the prediction mechanism where the impact of representativeness
of labelled data is much reduced, while pre-trained language model that
contains extra knowledge encoded and human prior knowledge can participate
more in the prediction._

  * •

_After prompting, the downstream data are closer to the pre-training data,
i.e., lower language discrepancy, the performance is more likely to be
better._

#### Assumption I: Prompting Reduces the Impact of Data Representativeness

We propose our first hypothesis that the prompt learning is more useful
because it leverages more information from human knowledge and pre-trained
language model rather than highly dependent on the representativeness of small
labelled data. Language models can take more responsibility in prediction
because they are trained with web-scale textual data and thus implicitly embed
enormous knowledge in models (Dathathri et al. 2020; Xie et al. 2021).

More specifically, as demonstrated in Figure 1, conventional fine-tuning is
essentially forcing the prediction of language model fit the small labelled
data distribution. Therefore, the performance is unstable and highly relied on
the representativeness of labelled data.

In contrast, prompt learning attempts to change the input data distribution
via prompting that “forces” the input data fit a more robust feature space.
Concretely, Figure 2(a) shows a not ideal few-shot case using unrepresentative
labelled data, whereas Figure 2(b) shows that, via prompting, downstream data
are projected into a new feature space where input data are evenly distributed
around rather than skewing and clustering in the original input space (Figure
2(a)). The green and yellow regions in Figure 2(c) are the new feature space
representing domain of interest, which is denoted as _pre-trained feature
space_ in this paper. In this case, the model performance will be less
affected by the unrepresentative labelled data because pre-trained feature
space is annotation-agnostic and mostly determined by the input text, template
and pre-trained language model.

Besides, as demonstrated by the black dashed line in Figure 2(b), researchers
take a “lucky guess” picking a pair of label words good at separating this
feature space, providing a good initialisation (i.e., zero-shot performance).
It is also possible to pick a pair of worse label words forming decision
boundary that has no zero-shot discrimination capability as demonstrated by
the magenta dashed line in Figure 2(b). To avoid this, a useful practice is
selecting prompting setting according to the performance on development set
(Liu et al. 2021a).

Also, we can continuously optimise the model using labelled data to achieve
better few-shot performance. Derived from our assumption, such optimisation
should be much robust than conventional fine-tuning because all data has been
projected and spread over pre-trained features space. As shown by the shift of
feature space from Figure 2(b) to Figure 2(c), the model does not need to take
much effort to converge (which is called _optimised feature space_).
Therefore, as compared to optimising model with original input data resulting
performance with large variance, prompt learning with different training
splits tends to end up with more stable results.

#### Assumption II: Language Discrepancy Measures the Quality of Prompting

Considering Assumption I , we are naturally curious about what prompting makes
an useful projection, i.e., a projection that properly aligns downstream data
into pre-trained feature space. We hypothesise that the quality of promptings
can be measured by the language discrepancy between pre-training corpora and
downstream datasets. Note that we limit this hypothesis in zero-shot
scenarios.

To be specific, since the objective gap is largely shortened through
reformulating downstream task via prompting, the inconsistency is mainly
caused by the language discrepancy between pre-training data and downstream
data. Thus, the smaller language discrepancy between pre-training corpora and
downstream datasets, the higher probability that data can be well distributed
in the pre-trained feature space, and therefore the better performance.

To conclude, the synergy between pre-trained language model and promptings
helps shape a better feature space where downstream data are reallocated. In
this space, labelled data, prompting, and pre-trained language model
collaboratively affect prediction performance.

### Discussion

Our proposed theoretical perspective can also intuitively explain several
observations and prompt learning behaviours reported in the literature:

  * •

Many work (Liu et al. 2021b; Lester, Al-Rfou, and Constant 2021) reports that
prompt learning can largely boost the few-shot performance while can only be
comparable to fine-tuning in fully-supervised scenarios which can be explained
through our framework. In fully-supervised scenarios, scholars have sufficient
labelled data at hand which cover enough representativeness of the whole
downstream dataset, leading to an ideal fine-tuning performance.

  * •

Previous work (Liu et al. 2021a) suggests that using multiple pairs of label
words to boost performance. This is equivalent to set multiple decision
boundaries in feature space, alleviating the affect of “worse” decision
boundary (e.g., magenta dashed line in Figure 2(b)).

  * •

Lester, Al-Rfou, and Constant (2021) report that using domain-specific token
as prefixed prompting can provide a better initialisation in multi-class
classification, which partially supports our assumptions. Adding domain-
specific tokens is more likely to project downstream data to the pre-trained
regions that represent domain of interest. Actually, sentiment analysis can be
seen as the simplest multi-class classification where sentiment polarity
(i.e., positive or negative) is the domain of interest. Thus, researchers
usually choose sentiment-bearing prompting settings like “It was a [good/bad]
movie.“ (Liu et al. 2021a).

### Extensions

Based on the proposed theoretical framework, we can come up with some
heuristic approaches. For example, when using prompt learning, we can select
the proper pre-trained language models for a certain downstream dataset before
training. Specifically, we can choose the language models pre-trained with the
corpora that is most similar to the downstream dataset (with lowest language
discrepancy). However, if we can not access the pre-training corpora or
hesitate to compute the linguistic difference between datasets, we can use the
perplexity (Brown et al. 1992) to approximate language discrepancy.

Originally, perplexity is one of the most common metrics for evaluating
language models, which is defined as the exponential average negative log-
likelihood of a sequence:

$\operatorname{PPL}(x)=\exp\left\\{-\frac{1}{t}\sum_{i}^{t}\log
p_{\theta}\left(x_{i}\mid x_{<i}\right)\right\\}$ (1)

where $x=[x_{1},x_{2},...,x_{t}]$ is a tokenised text sequence; $\log
p_{\theta}\left(x_{i}\mid x<i\right)$ is the log-likelihood of the $i^{th}$
token conditioned on the preceding tokens $x<i$ computed by a language model.
Intuitively, given a certain language model, if the perplexity of a corpus of
sentences is smaller means the model is easier to model this corpus. Usually,
a language model with lowest perplexity is chosen since it can be considered
as a faithful proxy to model the distribution of pre-training corpus.

Similarly, we can use perplexity to “forecast” the performance of different
prompting settings in advance. Previous work (Liu et al. 2021a; Lester, Al-
Rfou, and Constant 2021; Gao, Fisch, and Chen 2021) relies on development set
to post-hoc choose the best performing prompting settings. According to our
theoretical framework, we can select the prompting setting that minimises the
perplexity as candidates, which entirely gets rid of the development set. More
details are presented in experiments.

## Experiments

Dataset |  | 1\. [very/not] pleased. |  | 2\. [very/not] good. |  | 3\. [extremely/less] pleased. |  |  | 4\. [yellow/green] black.
---|---|---|---|---|---|---|---|---|---
| PPL | Zero | Few |  | PPL | Zero | Few |  | PPL | Zero | Few |  |  | PPL | Zero | Few
Douban |  | 24.61 | 57.12 | 55.73 _±2.99_ |  | 40.93 | 50.98 | 59.10 _±5.31_ |  | 28.80 | 56.68 | 55.15 _±3.67_ |  |  | 71.01 | 51.31 | 57.32 _±2.04_
Weibo |  | 19.78 | 61.79 | 96.72 _±2.45_ |  | 30.37 | 51.16 | 94.56 _±4.90_ |  | 22.34 | 58.35 | 95.17 _±2.26_ |  |  | 44.45 | 50.92 | 95.25 _±2.78_
Waimai |  | 16.44 | 67.80 | 79.15 _±2.11_ |  | 23.34 | 53.15 | 80.41 _±3.62_ |  | 19.68 | 69.72 | 82.02 _±1.67_ |  |  | 36.07 | 48.49 | 80.68 _±5.64_
Online-shopping |  | 14.07 | 73.12 | 83.46 _±4.97_ |  | 18.45 | 55.68 | 88.14 _±2.14_ |  | 16.88 | 67.49 | 85.56 _±2.61_ |  |  | 28.56 | 50.49 | 85.96 _±5.73_

Table 3: Summary of mean perplexity scores, zero/few-shot accuracy of
different prompting settings. Three domain-related prompting settings (1-3)
are in the left block while a domain-irrelevant prompting setting (4) is in
the right block.

Our experiments are indent to verify three aspects derived from two
assumptions:

  * •

Given a pre-trained language model, whether the prediction performance is
affected by synergy between prompting template $t$ and label words $w$.

  * •

Whether the “unrepresentativeness” problem is alleviated in prompt learning.

  * •

Whether perplexity is an useful metric for selecting prompting settings.

### Datasets

We conduct experiments on four public Chinese sentiment analysis datasets of
different domains to enlarge the language discrepancy between corpora: Douban
is a movie review dataset, Weibo is a social media comment dataset, Waimai is
a takeaway comment and Online-shopping is e-commerce shopping comment dataset.
Thus, all original texts are in Chinese, we perform a certain conversion in
this paper to facilitate understanding. Each dataset is split into five
training sets/one development set/one test set.

### Issue of Perplexity

We use the Chinese RoBERTa-base-wwm-ext (the same as one used in Sanity Check)
as the backbone pre-trained model.333Experiments using BERT can be seen in
Appendix. We use the perplexity of downstream datasets given a pre-trained
language model to approximate the language discrepancy between pre-training
corpora and downstream dataset by assuming the pre-trained language model has
fitted pre-training corpora well. That is, the lower perplexity indicates the
smaller language discrepancy between pre-training corpora and downstream
dataset.

Note that original perplexity applies specifically to causal language model
(i.e., autoregressive language model). As suggested in previous work (Liu et
al. 2019; Salazar et al. 2020), perplexity of bidirectional model like RoBERTa
can be analogous to that of causal language model by replacing $\log
p_{\theta}\left(x_{i}\mid x<i\right)$ with $\log p_{\theta}\left(x_{i}\mid
c\right)$ in Equation 1. Here $c$ means the context text which is the whole
sentence except for the $i^{th}$ token. This suggests that the perplexity of
each token is not only conditioned on the preceding tokens but also the
succeeding tokens. We replace “[MASK]” in a prompted sentence with label words
of two classes respectively, and then compute perplexity and average them.

However, during our preliminary experiment, we find out that this perplexity
as a metric has the drawback of favouring longer sentences. That is, a
sentence is assigned a lower perplexity, not because the pre-trained language
model is easier to model this sentence (i.e., low language discrepancy), but
rather because text is longer. We conjecture that this is due to the penalty
term in Equation 1 that divides the sum of log-likelihood by the sequence
length $t$. The detail of preliminary experiments regarding perplexity can be
seen in Appendix. Note that the focus of experiments is illustrating the
impact of language discrepancy instead of finding useful perplexity scoring,
thus, four datasets were subsampled to ensure every sentence length is between
14-15 words as well as to enforce a 50:50 class balance.

### Synergy between Templates and Label Words

We breakdown the analysis of synergy between template and label words into two
settings: (1) change the template given the same pair of label words; (2)
change the pair of label words given the same template.

Setup: We manually create four prompting settings as shown in Table 3. We
report perplexity, zero-shot as well as few-shot performance of each above
mentioned prompting setting. The few-shot optimisation for prompt learning is
described in Preliminaries and we use the same batch size, learning rate and
number of epochs as fine-tuning described in Sanity Check.

#### Zero-shot Result Analysis:

Accuracy of different prompting settings for four datasets are shown in Table
3. We first consider the performance of three domain-related prompting
settings, i.e., Setting 1 to 3 in the left block. Generally speaking, Setting
1 (i.e., [very/not] pleased.) is the best performing prompting setting
regarding zero-shot accuracy, outperforming Setting 2 (i.e., [very/not] good.)
as well as Setting 3 (i.e., [extremely/less] pleased.) across four datasets.
In Douban, the zero-shot accuracy of Setting 1 even surpasses that of
conventional fine-tuning in Table 1 (57.12 vs. 56.60). This again demonstrates
that prompt learning can equip pre-trained language model with powerful zero-
shot capability. The performance of Setting 4 (i.e., [yellow/green] good.)
demonstrates that the domain-irrelevant prompting setting has limited
contribution to zero-shot prediction, which is consistent with our
expectation.

When comparing Setting 1 and 2, we draw a conclusion that the template can
affect the forming of pre-trained feature space though we use the same pair of
label words. Then if we compare Setting 1 and 3, we can find out the choice of
label words is also important for the zero-shot performances.

To conclude, zero-shot capability verifies our assumption that in prompt
learning, the domain-related templates $t$ and label words $w$ collaboratively
contribute to shape the pre-trained feature space and build up the annotation-
agnostic decision boundary. The big difference between zero-shot performance
across different prompting settings, somehow, shows that it is crucial to
search for ideal prompting settings in prompt learning. We argue that this
problem can be addressed by using perplexity as discussed in the following
experiment.

#### Few-shot Result Analysis:

We can examine few-shot performance of each prompting setting in Table 3.
Overall, few-shot performance of all three domain-related prompting settings
(1-3) outperform the corresponding fine-tuning performance by a large margin
using the same training set especially in dataset Weibo and Waimai.
Furthermore, it is easy to find out prompt learning is much less sensitive to
labelled data and generalises well which is evidenced by the smaller variance
and higher mean score. For example, Setting 1 in Weibo (96.72 _±2.45_) vs.
fine-tuning (77.40 _±25.23_). All these observations support our assumption
that prompt learning can significantly alleviate the problem caused by
unrepresentative labelled data in few-shot scenarios. However, in some cases,
e.g., Setting 1 in Douban, prompt learning fails to compete with fine-tuning
and even worse than corresponding zero-shot accuracy. We conjecture that the
large learning rate and number of epochs make the training “overshoot” the
optimal checkpoint, where prompt learning degrades to fine-tuning since MLM
head functions as a initialisation instead of providing classification
information.

It is interesting to find out that the all prompting settings resulting in
similar few-shot performance, outperforming fine-tuning regardless of their
zero-shot accuracy. This is quite different from the zero-shot scenarios where
the the choice of prompting settings is crucial in prediction performance. We
guess that since we update all parameters during supervision, the pre-training
feature space is changed so much that the different prompting settings are
converged to the similar optimised feature space. If we using methods that
freeze most of parameters like P-tuning or Prompt Tuning, we might find
another story. We leave these explorations for future work.

#### Perplexity Analysis:

Though these experiments do not mainly aim for reflecting the relationship
between language discrepancy and model performance, Table 3 also conveys a
very clear message that as perplexity goes up, the zero-shot performance
becomes worse. For example, the perplexity of Setting 1 decreases from 24.61
(Douban), 19.78 (Weibo), 16.44 (Waimai) to 13.71 (Online-shopping), while the
zero-shot accuracy consistently increase from 57.12 (Douban), 61.79 (Weibo),
67.80 (Waimai) to even 73.12 (Online-shopping). This behaviour can also be
observed in Setting 2. The experimental results can partially verify our
assumption II that as the language discrepancy decrease (i.e., lower
perplexity), it is easier for a prompting to well align downstream data to the
pre-trained feature space.

### Using Language Discrepancy to Forecast Prompting

We design a realistic and useful experimental setting to verify whether we can
use language discrepancy to measure prompting quality in zero-shot context. We
do not investigate a few-shot setting since the previous experiments have
demonstrated the impact of templates is much reduced when model is optimised
by few labelled data.

#### Setup:

We fix verbaliser as {very: $++$, not: $--$} and manually generate seven
sentiment-bearing templates (details are shown in Figure 3). During inference,
we perform an instance-level template-example matching. In other words, for
each example, we select one template from the pool with the lowest perplexity
score and make the prediction (Manual-PPL), this is quite different from the
previous methods that use dataset-specific or class-specific template. If
perplexity is an ideal metric, the performance of this method will better than
random template-example matching (Manual-Random).

We then construct a more aggressive setting where templates are generated
automatically by T5.444We use the T5-v1.1-base-chinese checkpoint pre-trained
by UER: https://huggingface.co/uer/t5-base-chinese-cluecorpussmall. The
automatic template generation method is derived from Gao, Fisch, and Chen
(2021) which results in around 60-70 templates for each dataset after removing
duplicates, more details can be referred to Appendix. Based on automatically
generated templates, we apply similar template selection procedures which are
denoted as Auto-PPL and Auto-Random, respectively.

We also report the performance of another two state-of-the-art zero-shot
prompting-based methods NSP-BERT (Sun et al. 2021), and Zero-shotG (Xu et al.
2021) as strong baselines whose settings comply with the corresponding work.

#### Results:

First of all, both Auto-PPL and Manual-PPL outperform their random sampling
counterparts (i.e., Auto-Random and Manual-Random, respectively) by a clear
margin, which demonstrates that the prompting settings selected by perplexity
are more useful in mapping downstream data into pre-trained feature space,
thus, reaching better performance.

If we compare manual-methods with auto-methods, we can notice that the
performance of manual generated templates are better than automatically
generated templates, again, verifying the significance of providing useful
domain-related promptings. We suspect that the quality of automatically
generated templates is limited by the capacity of pre-trained T5. We believe
that this can be improved by changing T5 backbone or resorting to other
methods that consider automatically generating continuous templates leveraging
annotation information (Lester, Al-Rfou, and Constant 2021; Liu et al. 2021b;
Li and Liang 2021; Liu et al. 2022b).

Finally, if we compare our best performing method (Manual-PPL) with other
zero-shot methods. We find out our method surpasses state-of-the-art
performance in two out of four datasets, and can be competitive with state-of-
the-art methods in the rest datasets. This again shows that it is effective to
use perplexity to sort and select promptings.

#### In-depth Analysis:

We conduct a fine-grained analysis based on Manual-PPL method. For brevity, we
apply each manual prompting setting to all examples from the four datasets and
aggregate the accuracy score as a post-hoc measurement of prompting quality.
We then compute the frequency of being selected (i.e., lowest perplexity score
for this example) for each prompting setting.

The results are presented in Figure 3. It conveys a very clear message that
template with lower perplexity score is more likely to achieve better
performance. To be specifically, there is 60% chance for Manual-PPL to select
second best performing template (i.e., “[MASK] fond of it.”) and around 10%
chance to select the best performing template (i.e., “[MASK] satisfied.”). For
templates with no discriminative ability e.g., “[MASK]good.” and “[MASK]ok.”,
our method has almost no chance to select them. Most importantly, the
selection based on perplexity is annotation-agnostic and allows us to
“foresee” the result to some extent without the need of development set.

To conclude, the results demonstrate that perplexity is an ideal metric for
evaluating prompting settings, and thus verify the assumption II that language
discrepancy can measure the quality of promptings.

State-of-the-art Methods | Douban | Weibo | Waimai | Ecommerce
---|---|---|---|---
Zero-shotG (Xu et al. 2021) | 49.86 | 49.94 | 49.73 | 49.91
NSP-BERT (Sun et al. 2021) | 59.86 | 72.69 | 76.71 | 82.63
Baselines |  |  |  |
Auto-Random | 53.83 | 52.50 | 56.02 | 58.83
Manual-Random | 55.14 | 59.79 | 66.54 | 72.13
Our Methods |  |  |  |
Auto-PPL | 59.21 | 67.04 | 72.19 | 73.94
Manual-PPL | 60.74 | 66.50 | 75.49 | 85.12

Table 4: Results for four sentiment analysis datasets. Figure 3: Normalised
frequency of being selected vs. template quality measured by mean accuracy
over four datasets. The verbaliser is {very: $++$, not: $--$}.

## Conclusion

This work present a solid step towards investigating the underlying rationales
behind cloze-style prompt learning in zero/few-shot scenarios. To this end, we
elaborate why fine-tuning is suboptimal in few-shot and propose a theoretical
framework to explain the efficacy of prompt learning. Comprehensive
experiments are performed to support proposed assumptions. Finally, as an
extension of the theoretical framework, we propose using perplexity to measure
the quality of prompting setting. Experimental results show that this method
leads to significant prediction benefits and allows us to “forecast” the
ability of model prior to inference.

## References

  * Brown et al. (1992) Brown, P. F.; Della Pietra, S. A.; Della Pietra, V. J.; Lai, J. C.; and Mercer, R. L. 1992.  An Estimate of an Upper Bound for the Entropy of English.  _Computational Linguistics_ , 18(1): 31–40.
  * Brown et al. (2020) Brown, T.; Mann, B.; Ryder, N.; Subbiah, M.; Kaplan, J. D.; Dhariwal, P.; Neelakantan, A.; Shyam, P.; Sastry, G.; Askell, A.; et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33: 1877–1901.
  * Dathathri et al. (2020) Dathathri, S.; Madotto, A.; Lan, J.; Hung, J.; Frank, E.; Molino, P.; Yosinski, J.; and Liu, R. 2020.  Plug and Play Language Models: A Simple Approach to Controlled Text Generation.  In _International Conference on Learning Representations_.
  * Devlin et al. (2018) Devlin, J.; Chang, M.-W.; Lee, K.; and Toutanova, K. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Gao, Fisch, and Chen (2021) Gao, T.; Fisch, A.; and Chen, D. 2021.  Making Pre-trained Language Models Better Few-shot Learners.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , 3816–3830. Online: Association for Computational Linguistics.
  * Khashabi et al. (2020) Khashabi, D.; Min, S.; Khot, T.; Sabharwal, A.; Tafjord, O.; Clark, P.; and Hajishirzi, H. 2020.  UNIFIEDQA: Crossing Format Boundaries with a Single QA System.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , 1896–1907. Online: Association for Computational Linguistics.
  * Lester, Al-Rfou, and Constant (2021) Lester, B.; Al-Rfou, R.; and Constant, N. 2021.  The Power of Scale for Parameter-Efficient Prompt Tuning.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 3045–3059. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics.
  * Lewis et al. (2020) Lewis, M.; Liu, Y.; Goyal, N.; Ghazvininejad, M.; Mohamed, A.; Levy, O.; Stoyanov, V.; and Zettlemoyer, L. 2020.  BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 7871–7880. Online: Association for Computational Linguistics.
  * Li and Liang (2021) Li, X. L.; and Liang, P. 2021.  Prefix-Tuning: Optimizing Continuous Prompts for Generation.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , 4582–4597. Online: Association for Computational Linguistics.
  * Liu et al. (2022a) Liu, J.; Shen, D.; Zhang, Y.; Dolan, B.; Carin, L.; and Chen, W. 2022a.  What Makes Good In-Context Examples for GPT-3?  In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , 100–114. Dublin, Ireland and Online: Association for Computational Linguistics.
  * Liu et al. (2021a) Liu, P.; Yuan, W.; Fu, J.; Jiang, Z.; Hayashi, H.; and Neubig, G. 2021a.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _arXiv preprint arXiv:2107.13586_.
  * Liu et al. (2022b) Liu, X.; Ji, K.; Fu, Y.; Tam, W.; Du, Z.; Yang, Z.; and Tang, J. 2022b.  P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , 61–68. Dublin, Ireland: Association for Computational Linguistics.
  * Liu et al. (2021b) Liu, X.; Zheng, Y.; Du, Z.; Ding, M.; Qian, Y.; Yang, Z.; and Tang, J. 2021b.  GPT understands, too.  _arXiv preprint arXiv:2103.10385_.
  * Liu et al. (2019) Liu, Y.; Ott, M.; Goyal, N.; Du, J.; Joshi, M.; Chen, D.; Levy, O.; Lewis, M.; Zettlemoyer, L.; and Stoyanov, V. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Raffel et al. (2020) Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020.  Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer.  _Journal of Machine Learning Research_ , 21(140): 1–67.
  * Salazar et al. (2020) Salazar, J.; Liang, D.; Nguyen, T. Q.; and Kirchhoff, K. 2020.  Masked Language Model Scoring.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2699–2712. Online: Association for Computational Linguistics.
  * Schick and Schütze (2021a) Schick, T.; and Schütze, H. 2021a.  Exploiting Cloze-Questions for Few-Shot Text Classification and Natural Language Inference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , 255–269. Online: Association for Computational Linguistics.
  * Schick and Schütze (2021b) Schick, T.; and Schütze, H. 2021b.  It’s Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2339–2352. Online: Association for Computational Linguistics.
  * Sun et al. (2021) Sun, Y.; Zheng, Y.; Hao, C.; and Qiu, H. 2021.  NSP-BERT: A Prompt-based Zero-Shot Learner Through an Original Pre-training Task–Next Sentence Prediction.  _arXiv preprint arXiv:2109.03564_.
  * Tam et al. (2021) Tam, D.; R. Menon, R.; Bansal, M.; Srivastava, S.; and Raffel, C. 2021.  Improving and Simplifying Pattern Exploiting Training.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , 4980–4991. Online and Punta Cana, Dominican Republic: Association for Computational Linguistics.
  * Wang et al. (2020) Wang, Y.; Yao, Q.; Kwok, J. T.; and Ni, L. M. 2020.  Generalizing from a few examples: A survey on few-shot learning.  _ACM computing surveys (csur)_ , 53(3): 1–34.
  * Xie et al. (2021) Xie, S. M.; Raghunathan, A.; Liang, P.; and Ma, T. 2021.  An explanation of in-context learning as implicit bayesian inference.  _arXiv preprint arXiv:2111.02080_.
  * Xu et al. (2021) Xu, L.; Lu, X.; Yuan, C.; Zhang, X.; Xu, H.; Yuan, H.; Wei, G.; Pan, X.; Tian, X.; Qin, L.; et al. 2021.  Fewclue: A chinese few-shot learning evaluation benchmark.  _arXiv preprint arXiv:2107.07498_.

##  Appendix A A BERT Experimental Results

This section presents a set of experimental results using BERT model.555We use
BERT-base-wwm-ext available at https://huggingface.co/hfl/chinese-bert-wwm-ext

### Few-shot Fine-tuning with BERT

Table 5 presents conventional fine-tuning results of BERT. Similar to that of
RoBERTa, BERT results also suffer from unrepresentative labelled data and can
not generalise well in few-shot settings, which is evidenced by large variance
and low accuracy. Moreover, the performance of BERT is much worse than the
RoBERTa. According to Liu et al. (2019), BERT is severely insufficient
trained, which makes BERT less useful when functioning as a feature extractor
compared to RoBERTa.

### Prompt Learning with BERT

#### Zero-shot Performance:

Table 6 shows perplexity, zero/few-shot performance of BERT using different
prompting settings. In BERT-based cases, the prompt learning demonstrates much
higher performance than fine-tuning. For example, even the zero-shot
performance of Setting 1 surpasses conventional fine-tuning in three datasets
by a large margin. Also, zero-shot performance of different settings suggests
the importance of using domain-related templates, which is evidenced by the
low accuracy of domain-irrelevant Setting 4 (i.e., results for all datasets
are near 50%, equivalent to a random guess).

#### Few-shot Performance:

We can see the few-shot prompt learning results in Table 6, the improvement
becomes larger. However, in some cases, the variance of few-shot prompt
learning is as high as fine-tuning, e.g., in dataset Weibo 85.86 _±20.30_ of
Setting 2 vs. 75.74 _±23.70_ of fine-tuning, which can also be explained by
the under training of BERT. Due to insufficient training, in some cases, the
MLM head of BERT functions as an initialisation instead of providing
classification information, and degrades to fine-tuning.

#### Perplexity

The correlation between perplexity and corresponding zero-shot accuracy in
Table 6 demonstrates a very clear message that the prompt-learning zero-shot
performance is inversely proportional to the perplexity. If we compare across
datasets in Setting 1. We can find out as the perplexity increases, the zero-
shot accuracy decreases, which can be consistently observed in Setting 2 and
3. If we compare across different settings for the same dataset, especially in
dataset Online-shopping and Waimai, it is very clear the zero-shot performance
is inversely proportional to the perplexity. This finding partially verifies
that as the language discrepancy becoming smaller, it is more likely for a
language model to well project downstream data into pre-trained feature space,
thus resulting better zero-shot performance.

Trial | 1 | 2 | 3 | 4 | 5 | Avg. | All
---|---|---|---|---|---|---|---
Douban | 49.89 | 49.89 | 49.89 | 50.10 | 50.00 | 49.95 _±0.09_ | 49.89
Weibo | 50.05 | 49.94 | 93.24 | 97.07 | 88.42 | 75.74 _±23.70_ | 49.94
Waimai | 71.78 | 52.46 | 50.27 | 49.72 | 68.63 | 58.57 _±10.72_ | 80.00
Online-shopping | 50.09 | 50.18 | 75.51 | 50.09 | 49.90 | 55.15 _±11.38_ | 50.09

Table 5: Results of conventional fine-tuning on four Chinese sentiment
analysis datasets using BERT. _Avg._ denotes the aggregated results of random
seed 1 to 5. _All_ denotes the model is fine-tuned with all the labelled
examples from five splits.

Dataset |  | 1\. [very/not] pleased. |  | 2\. [very/not] good. |  | 3\. [extremely/less] pleased. |  |  | 4\. [yellow/green] black.
---|---|---|---|---|---|---|---|---|---
| PPL | Zero | Few |  | PPL | Zero | Few |  | PPL | Zero | Few |  |  | PPL | Zero | Few
Douban |  | 91.27 | 56.42 | 54.40 _±4.35_ |  | 140.38 | 51.53 | 53.26 _±3.46_ |  | 145.10 | 49.45 | 53.92 _±3.98_ |  |  | 247.59 | 50.10 | 54.60 _±3.18_
Weibo |  | 86.60 | 67.12 | 75.09 _±23.05_ |  | 111.61 | 54.08 | 85.86 _±20.30_ |  | 111.68 | 50.73 | 91.41 _±4.25_ |  |  | 174.22 | 50.13 | 73.83 _±21.77_
Waimai |  | 59.99 | 73.56 | 78.16 _±2.93_ |  | 65.16 | 58.35 | 76.98 _±1.67_ |  | 73.24 | 51.64 | 72.00 _±12.69_ |  |  | 106.66 | 50.27 | 76.93 _±2.92_
Online-shopping |  | 50.82 | 80.38 | 80.42 _±6.38_ |  | 54.48 | 62.03 | 83.28 _±2.17_ |  | 64.50 | 54.32 | 83.32 _±3.90_ |  |  | 93.19 | 49.14 | 76.23 _±15.36_

Table 6: Summary of mean perplexity scores, zero/few-shot accuracy of
different prompting settings using BERT. Three domain-related prompting
settings (1-3) are in the left block while a domain-irrelevant prompting
setting (4) is in the right block.

Baselines | Douban | Weibo | Waimai | Ecommerce
---|---|---|---|---
Auto-Random | 54.27 | 52.39 | 56.57 | 58.52
Manual-Random | 57.89 | 60.37 | 69.31 | 71.61
Our Methods |  |  |  |
Auto-PPL | 53.07 | 57.60 | 53.15 | 68.16
Manual-PPL | 59.86 | 64.71 | 79.04 | 81.78

Table 7: Results for four sentiment analysis datasets using BERT. Figure 4:
Normalised frequency of being selected vs. template quality measured by mean
accuracy over four datasets using BERT. The verbaliser is {very: $++$, not:
$--$}.

### Perplexity to Forecast Template

We use BERT to rerun the experiments following the setting in Section Using
Language Discrepancy to Forecast Prompting. Experimental results are shown in
Table 7 and Figure 4.

The results are similar to those of RoBERTa or even more encouraging. When we
see Table 7, it is obvious that the PPL based methods (i.e., Manual-PPL and
Auto-PPL) outperform their random counterparts(i.e., Manual-Random and Auto-
Random) in most of cases, which again verify the Assumption II. Especially in
manual template settings, the perplexity can better “forecast” the quality of
prompting. We guess that is because the manual generated templates are more
related to the domain of interest.

Figure 4 also demonstrates that perplexity can “forecast” the template
quality. Using perplexity gets around 40% chance to select the best performing
template (i.e., “[MASK] font of it.”) and 35% chance to select the second best
performing template (i.e., “[MASK] satisfied.”). All these findings support
the Assumption II that language discrepancy can evaluate template quality to
some extent.

##  Appendix B B Issue of Perplexity

We find out the current perplexity has one drawback of favouring longer
sentences. That is, a sentence is assigned a lower perplexity, not because the
pre-trained language model is easier to model this sentence (i.e., lower
language discrepancy), but rather because text is longer. We first use a
simple comparison to demonstrate this as shown in Table 8.

Text in Chinese | Translation | Perplexity |
---|---|---|---
阿姨：不要太累了[哈哈] | Auntie: Don’t be too tired [haha] | 17.21 |
| 撒娇大法，啊的身份拉升大盘撒娇大法，啊
---
的身份拉盘。阿姨：不要太累了[哈哈]
| Coquetry Dafa, ah’s identity pulls up the big market Coquettish Dafa,
---
ah’s identity pulls the plate. Auntie: Don’t be too tired [haha]
5.85 |

Table 8: Comparison of a long nonsense sentence with a short fluent sentence.

We first calculate the perplexity of a meaningful sentence “Auntie: Don’t be
too tired [haha]” which is 17.21. However, if we prefix this sentence with a
long sequence of nonsense words, the perplexity even gets lower, i.e., 5.85.
We then conduct a large scale test to see the correlation between perplexity
and text length. The results are presented in Figure 5, it is obvious that the
avg. perplexity is inversely proportional to avg. text length. In other words,
a low perplexity of a sentence is partially contributed by a low language
discrepancy but more likely to be contributed by a long text, which violates
our intention that uses perplexity to measure the language difference.

Figure 5: Line chart of average perplexity and average text length across
different datasets.

##  Appendix C C Automatic Template Generation

Similar to Gao, Fisch, and Chen (2021), we fix the verbaliser {very: $++$,
not: $--$}, and using T5-v1.1-base-chinese to automatically generate
templates. Specifically, Gao, Fisch, and Chen (2021) assume a few-shot
scenario where using ground truth label word as well as corresponding examples
to generate a certain number templates. They then sort generated templates
based on the aggregated generation probability (the calculation of generation
probability also needs label information) of the whole training set.

However, our experiment assumes a zero-shot scenario with no labelled data.
Thus, for each dataset, we first randomly sample 50 examples from the pool.
For each example, we use label words indicating both sentiments to generate
template, one for each sentiment, resulting in 100 templates in total. Then we
remove duplicate templates and around 60-70 templates remain. All
automatically generated template can be seen at URL masked for anonymous
review.

##  Appendix D D Reverse Label Words

To briefly verify whether the perplexity can be used to measure the quality of
prompting setting, we perform a very simple experiment that we compute mean
perplexity score of prompted input $x^{\prime}$ with “[MASK]” filled by ground
truth label words for each dataset (called PPLg ). Then we reverse the label
words filled in previous input examples (e.g., we change “very pleased.” to
“not pleased.” in a positive sentiment example) and recompute mean perplexity
score (called PPLr). Note that this experiment is based on RoBERTa.

The results are presented in Table 9. First, we notice that in Setting 1
(i.e., “[very/not] pleased.”), the mean perplexity of PPLg is always smaller
than that of PPLr by a clear margin which is encouraging. This shows that the
pre-trained model can perceive the change of semantics in texts. When we see
the perplexity of Setting 2 (i.e., “[yellow/red] black.”, we find out the
magnitude of change is much smaller, which demonstrates that replacing label
words means almost no difference to models if domain-irrelevant prompting is
applied.

Datasets | 1\. [very/not] pleased. |  | 2\. [yellow/red] black. |
---|---|---|---|---
PPLg | PPLr | Diff. |  | PPLg | PPLr | Diff. |
Douban | 24.10 | 25.12 | -1.02 |  | 67.91 | 74.11 | -6.20 |
Weibo | 19.17 | 20.39 | -1.22 |  | 44.39 | 44.51 | -0.12 |
Waimai | 16.06 | 16.82 | -0.76 |  | 22.60 | 24.07 | -0.20 |
Online-shopping | 13.55 | 14.58 | -1.03 |  | 28.51 | 28.61 | -0.10 |

Table 9: Mean perplexity of prompting with ground truth label word (PPLg),
prompting with reversed label word (PPLr), and difference between two
templates computed by PPLg minus PPLr (Diff.).