Laboratoire de Physique Corpusculaire (LPC), CNRS, ENSICAEN, UMR6534,
Université de Caen Normandie, F-14050 Caen Cedex, France Grand Accélérateur
National d’Ions Lourds (GANIL), CEA/DRF - CNRS/IN2P3, Boulevard Henri
Becquerel, 14076 Caen, France

# Properties of pasta phases in catalyzed neutron stars

H. Dinh Thiins:x  A. F. Fantinains:y  F. Gulminelliins:x ins:xins:xins:yins:y

###### Abstract

Exotic non-spherical configurations of nuclei, known as “pasta” phases, are
expected to be present at the bottom of the inner crust of a neutron star. We
study the properties of these configurations in catalyzed neutron stars within
a compressible liquid-drop model approach, with surface parameters optimized
to reproduce experimental nuclear masses. Our results show that the properties
of the pasta phases exhibit strong model dependence. To estimate the model
uncertainties, a Bayesian analysis is performed, combining information from
nuclear physics experiments and chiral perturbation theoretical calculations
with astrophysical observations. The inferred posterior distributions are
discussed, with particular focus on the effect of the low-density energy
functional on the predictions.

##  1 Introduction

In the deepest region of the inner crust of a neutron star (NS), it is
expected that non-spherical shapes of nuclei, known as “pasta phases”, are
energetically favorable. The existence of pasta phases, in particular of
“bubbles”, at high densities in the crust was already suggested in the
pioneering work of ref. [1]. Other, non-spherical, shapes, particularly two-
dimensional (rods, tubes) and one-dimensional (slabs) geometries, were further
investigated in refs. [2, 3, 4], while more complex structures were studied
more recently, e.g., in refs. [5, 6, 7, 8].

Numerous studies have been conducted in the last decades on the nuclear pasta,
using different approaches, like compressible liquid drop models (CLDMs),
semi-classical methods, nuclear energy density functional theory, and
molecular dynamics (see, e.g., ref. [9] for a recent review, and references
therein). Recently, we have extended the CLDM of Carreau et al. [10, 11] to
account for non-spherical configurations in the NS inner crust [12, 13], and
evaluated the uncertainties in the pasta-phase properties using the Bayesian
inference. This analysis predicted that a sizable amount of the crust is made
up of pasta structures. Specifically, we estimated the relative thickness and
mass of the pasta layer compared to that of the whole crust to be $12.8\%$ and
$48.5\%$, respectively (see table 5 in ref. [12]), in good agreement with the
results obtained in ref. [14]. Although, up to now, there is no direct
evidence of the existence of the pasta phases from NS observations, the
presence of these “exotic” configurations can potentially have an important
impact on various NS phenomena, such as NS cooling [15, 16, 17], transport
properties [18, 19], crust oscillations [20, 21], and pulsar magnetic and
rotational evolution [22].

In this work, we pursue the investigation of refs. [12, 13], presenting
additional discussions on both the model dependence of the pasta-phase
predictions and the effect of the low-energy density functional on the
fractional crustal thickness and mass of the pasta layer. In sect. 2, we
describe the main points of the formalism used in modeling the nuclear pasta.
The model dependence of the pasta-phase properties is presented in sect. 3,
while in sect. 4 we discuss the importance of constraining the energy
functional at low densities for the determination of the uncertainties of
pasta-phase observables. Finally, conclusions are given in sect. 5.

##  2 Model of the pasta phases

We model the pasta phases under the cold catalyzed matter hypothesis, that is,
under the assumption of full equilibrium at zero temperature. Details on the
formalism are presented in refs. [12, 13]; here, we briefly recall the main
points and assumptions.

At a given baryon density $n_{B}$ in the NS inner crust, the structure of
matter is approximated by a periodic lattice configuration of identical
Wigner-Seitz (WS) cells of volume $V_{\rm WS}$. Each WS cell contains a
cluster or a hole, of volume $V$. The density distribution in the Wigner–Seitz
cell is $n_{i}$ ($n_{g}$) if $l<r_{N}$, and $n_{g}$ ($n_{i}$) otherwise,
$n_{g}$ being the density of the surrounding uniform neutron gas, $r_{N}$ is
the linear dimension of the pasta structure, and $l$ the linear coordinate of
the cell. The density of the denser phase is $n_{i}=A/V$ in the case of
cluster ($n_{i}=A/(V_{\rm WS}-V)$ in the case of holes) and its proton
fraction is given by $y_{p}=Z/A$, $Z$ ($A$) being the proton number (total
mass number) of the cluster. The surrounding uniform electron gas has density
$n_{e}$ such that charge neutrality holds, i.e., $n_{e}=n_{p}$, with $n_{p}$
being the total proton density in the cell.

In order to obtain the ground state of the system, the energy density of the
WS cell has to be minimized with the constraint of baryon density
conservation. The corresponding thermodynamic potential per unit volume in the
CLDM can be written as:

$\displaystyle\Omega$ $\displaystyle=$
$\displaystyle(n_{B}-n_{p})m_{n}c^{2}+n_{p}m_{p}c^{2}+\epsilon_{B}(n_{i},1-2y_{p})f(u)$
(1) $\displaystyle+$ $\displaystyle\epsilon_{B}(n_{g},1)(1-f(u))+\epsilon_{\rm
Coul}+\epsilon_{\rm{surf+curv}}+\epsilon_{e}(n_{e})-\mu_{B}^{\rm tot}\ n_{B},$

where $m_{n}$ ($m_{p}$) is the neutron (proton) mass, $\epsilon_{B}(n,\delta)$
is the uniform nuclear matter energy density at density $n$ and isospin
asymmetry $\delta=(n_{n}-n_{p})/n$, with $n_{n}$ ($n_{p}$) the neutron
(proton) density, $\epsilon_{e}(n_{e})$ is the electron gas energy density,
$\mu_{B}^{\rm tot}$ is the baryonic chemical potential (including the rest
mass), $\epsilon_{\rm{surf+curv}}$ and $\epsilon_{\rm Coul}$ are the surface
and Coulomb energies, respectively, and the function $f$ is given by $f(u)=u$
(or $f(u)=1-u$) for clusters (holes), with $u=V/V_{\rm WS}$; see sect. 2 in
ref. [13] for details.

From eq. (1), we can see that a nuclear model is defined by the choice of the
bulk functional, $\epsilon_{B}$, supplemented with the interface energy,
$\epsilon_{\rm{surf+curv}}$ and $\epsilon_{\rm Coul}$. For the energy density
of homogeneous nuclear matter, $\epsilon_{B}$, we use the meta-modeling
approach proposed by Margueron et al. [23, 24]. Within this approach, a Taylor
expansion in $x=(n-n_{\rm sat})/(3n_{\rm sat})$ ($n_{\rm sat}\approx 0.15$
fm-3 being the saturation density) up to order $N$ ($N=4$ in the present case)
around the saturation point ($n=n_{\rm sat}$, $\delta=0$) is introduced, with
the parameters of the expansion corresponding to the so-called equation-of-
state empirical parameters. In addition, a $\delta^{5/3}$ term from the
fermionic zero-point energy and an exponential correction ensuring the correct
limiting behavior at zero density are added (see eq. (5) in ref. [13], and
ref. [10] for details). As for the finite-size contributions, which are the
only terms dependent on the nuclear geometry, we make use of the same
expressions as in refs. [12, 13, 25], see eqs. (8)-(15) in ref. [13].

The complete parameter set, denoted as $\bf X$, consists of 18 parameters (13
bulk parameters plus 5 surface parameters): for each functional, i.e. for a
given set of bulk parameters, the surface and curvature parameters are
determined from a $\chi^{2}$-fit to the experimental masses from the Atomic
Mass Evaluation (AME) 2016 [26]. Then, given a nuclear model, i.e. a full set
of $\bf{X}$ parameters, at each baryon density $n_{B}$, the equilibrium phase
and composition are obtained in two steps: first, the optimal geometry is
determined by the configuration having the lowest WS-cell energy density,
then, the thermodynamic potential, eq. (1), is minimized with respect to five
variational parameters ($n_{i}$, $I=1-2y_{p}$, $A$, $n_{p}$, and $n_{g}$) to
obtain the composition. Note that we consider here five customary geometries,
namely spheres, cylinders (rods), slabs, tubes, and bubbles. The equilibrium
phase corresponds to the one having the minimum value of the “optimal”
thermodynamic potential.

##  3 Model dependence of the pasta phases

To show the model dependence of the pasta-phase properties, we performed the
calculations as described in sect. 2 employing three different nuclear energy
functionals, as illustrative examples: BSk24 [27], DD-ME$\delta$ [28], and TM1
[29]. The corresponding values of empirical parameters and optimized surface
parameters for these three functionals are given in tables 1 and 2 of ref.
[12].

We start the discussion by showing in fig. 1 the competition among the
geometries in the densest region of the inner crust, near the crust-core
transition. Since, as mentioned in sect. 2, the only dependence on the
geometry of the pasta in the thermodynamic potential to be minimized, eq. (1),
enters in the surface and Coulomb energies, in fig. 1 we plot the energy
difference $\Delta\epsilon=\epsilon_{\rm crust}-\epsilon_{HM}$, where
$\epsilon_{\rm crust}$ is the WS-cell energy density obtained from the
minimization procedure for a given geometry, and $\epsilon_{\rm HM}$ is the
WS-cell energy density calculated for the homogeneous matter at
$\beta$-equilibrium. We can clearly observe that the differences in
$\Delta\epsilon$ are more pronounced when comparing the three functionals than
among the five geometries within the same functional. This also yields, for
the three models, very different optimal compositions, as one can see, for
example, in fig. 2 of ref. [13]. In addition, the results in fig. 1 show that
the transitions among the five phases as well as the crust-core transition are
strongly affected by the choice of the energy functional. This can also be
observed comparing the left columns illustrated for each functional in fig. 2,
where the sequence of equilibrium geometries is displayed.

Figure 1: WS-cell energy-density difference as a function of baryon density
for five geometries: spheres (red solid lines), rods (blue dash-dotted lines),
slabs (green dashed lines), tubes (orange dotted lines), and bubbles (black
solid lines), for three functionals: BSk24, DD-ME$\delta$, and TM1. In each
panel, the point of intersection of the lowest $\Delta\epsilon$ curve with the
horizontal black line corresponds to the crust-core transition. Figure 2:
Sequence of equilibrium geometries as a function of baryon density $n_{B}$ for
three nuclear functionals: BSk24, DD-ME$\delta$, and TM1. See text for
details.  Figure 3: Thickness (left panel) and mass (right panel) fractions of
the pasta phases with respect to the whole crust as a function of the NS mass
$M$ for three functionals: BSk24 (blue solid lines), DD-ME$\delta$ (orange
dash-dotted lines), and TM1 (green dashed lines).

The consistent minimization of the thermodynamic potential $\Omega$ in eq. (1)
for each geometry is crucial in getting the correct configuration for the
ground state of matter. This procedure, which is however relatively time
consuming, plays an important role in calculations requiring the exact
composition as input, such as in the calculations of transport coefficients
[19]. However, for the determination of more global quantities, such as
thickness, mass, or moment of inertia of the pasta phases, one can instead fix
the composition of the different phases to that obtained for spheres. To
justify this point, on the right columns for each functional in fig. 2, we
display the sequence of the equilibrium geometries obtained when the
minimization is performed assuming for all geometries the composition obtained
for spheres. The results suggest that this assumption does not change
considerably the transition densities, and can thus be adopted to reduce the
computational time when performing the Bayesian analysis (see next section).

A complementary information on the importance of the pasta layer in NSs is
given by the pasta fractional thickness and the associated mass, that we plot
in fig. 3 as a function of the NS mass $M$ (in unit of solar mass
$M_{\odot}$). Although the numerical values of $R_{\rm pasta}/R_{\rm crust}$
and $M_{\rm pasta}/M_{\rm crust}$ are strongly model dependent, the presence
of pasta is robustly predicted by all the considered models. On the other
hand, the variation of the fractional pasta thickness (mass) with the NS mass
is similar for the three functionals and amounts to $\approx 20\%$ ($\approx
10\%$) when changing the NS mass from $0.4M_{\odot}$ to $2M_{\odot}$.

##  4 Influence of the low-density equation of state

The results presented in the previous section highlight the model dependence
of pasta properties. To quantitatively assess this point and determine the
impact of our current incomplete knowledge of the nuclear energy functional on
the uncertainties of pasta-phase properties, we have performed a Bayesian
analysis. We started by considering flat non-informative priors, constructed
by largely varying the model parameters $\bf{X}$ according to our current
knowledge provided by nuclear-physics experiments (see table 1 in ref. [13]).
For each prior model, the minimization of the corresponding $\Omega$
potential, eq. (1), was carried out to find the equilibrium composition of the
crust. filter Models yielding non-physical solutions were discarded. We then
applied to the prior both low-density (LD) constraints from nuclear physics
and high-density (HD) constraints coming from general and NS physics to
generate the posterior distribution:

$p_{\rm post}({\bf{X}})=\mathcal{N}\,\omega_{\rm LD}({\bf{X}})\,\omega_{\rm
HD}({\bf{X}})\,\omega_{\rm mass}\,p_{\rm prior}({\bf{X}}),$ (2)

where $\mathcal{N}$ is the normalization factor, $p_{\rm prior}$ is the prior
probability, and $\omega_{\rm mass}$ is a likelihood expression representing
the quality of reproduction of the experimental masses from the AME2016 [26].
The $w_{\rm LD}$ filter accounts for the uncertainty band of the chiral
effective field theory (EFT) calculations of the energy per particle of
symmetric and pure neutron matter of ref. [30], which is considered as a
$90\%$ confidence interval. Due to the accuracy in the chiral EFT calculation,
the filtering bands at low density are very narrow, thus the rejection rate is
high. The second strict filter we applied, $\omega_{\rm HD}$, is defined by
imposing causality, thermodynamic stability, non-negative symmetry energy, and
the resulting equation of state to support observed massive NSs [31], $M_{\rm
max}\geq 1.97M_{\odot}$, where $M_{\rm max}$ is the maximum NS mass obtained
with a given model, i.e. with a given parameter set $\bf X$ (see refs. [32,
33] for a discussion on the importance of the high-density constraints).

Since NS global properties are rather determined by the high-density part of
the equation of state, the compatibility of the functionals with ab-initio
predictions at very low-density was usually overlooked. Here, with the aim of
focusing on the importance of the low-density part of the energy functional on
the pasta-phase properties, we considered two density intervals for applying
the LD filter, namely $\left[0.1,0.2\right]$ fm-3 and $\left[0.02,0.2\right]$
fm-3. When the filter was applied in the range $\left[0.1,0.2\right]$
($\left[0.02,0.2\right]$) fm-3, we generated $2\times 10^{6}$ ($10^{8}$)
models in the prior, and obtained 7714 (7008) models in the final posterior.
These statistics are sufficient for this study as increasing the number of
models has no significant impact on the results. The posterior correlation of
the fractions of thickness, $R_{\rm pasta}/R_{\rm crust}$, and associated
mass, $M_{\rm pasta}/M_{\rm crust}$, as well as their probability density
distributions, are displayed in fig. 4. When the filter is applied from $n\geq
0.1$ fm-3 (left panel), noticeable peaks arise at very low $R_{\rm
pasta}/R_{\rm crust}$ and $M_{\rm pasta}/M_{\rm crust}$, compatible with a
small or even null pasta layer, contrarily to the case when the energy
functional is constrained from lower density, $n\geq 0.02$ fm-3 (right panel).
In the latter case, the uncertainties in the fractional crustal thickness and
mass of the pasta layer are reduced and the correlation enhanced. These
results corroborate those obtained in refs. [12, 13], additionally showing the
importance of constraining the energy functional in the very-low-density
region to determine the pasta-phase properties (see refs. [12, 13] for a
discussion on the correlation of the pasta-phase properties with the nuclear
parameters $\bf{X}$).

Figure 4: Posterior correlation between the fractional thickness and mass of
the pasta phases with respect to the whole crust calculated at $M=M_{\rm
max}$. Left (right) panel: the LD filter is applied in the interval
$\left[0.1,0.2\right]$ fm-3 ($\left[0.02,0.2\right]$ fm-3).

##  5 Conclusions

To study the pasta-phase properties in catalyzed NSs, we employed a CLDM in
which the bulk energy is calculated using a meta-modeling approach and finite-
size parameters are optimized on the AME2016 atomic mass table. The model
dependence of the results is illustrated using three energy functionals,
namely BSk24, DD-ME$\delta$, and TM1. To quantitatively estimate the
uncertainties of the pasta-phase properties, we performed a Bayesian analysis
starting from non-informative priors obtained largely varying the model
parameters within current uncertainties coming from nuclear-physics
experiments and imposing constrains coming from both (theoretical and
experimental) nuclear physics and astrophysics. The results we obtained
confirm those presented in our previous studies [12, 13], namely, (i) the
robustness in the prediction of the existence of pasta phases at the bottom of
the inner crust; (ii) the model dependence of the pasta-phase properties; and
(iii) the importance of constraining the functional from very low densities to
properly determine the pasta-phase observables.

###### Acknowledgements.

The authors acknowledge partial support from the IN2P3 Master Project “NewMAC”
and the CNRS International Research Project (IRP) “Origine des éléments lourds
dans l’univers: Astres Compacts et Nucléosynthèse (ACNu)”.

## References

  * [1] Baym G., Bethe H. A. Pethick C. J. Nucl. Phys. A1751971225-271.
  * [2] Ravenhall D. G., Pethick C. J. Wilson J. R. Phys. Rev. Lett.5019832066-2069.
  * [3] Hashimoto M., Seki H. Yamada M. Prog. Theor. Phys.711984320-326.
  * [4] Oyamatsu K., Hashimoto M. Yamada M. Prog. Theor. Phys.721984373.
  * [5] Nakazato K., Oyamatsu K. Yamada S. Phys. Rev. Lett.1032009132501.
  * [6] Schuetrumpf B., Klatt M. A., Iida K., et al. Phys. Rev. C912015025801.
  * [7] Schneider A. S., Berry D. K., Briggs C. M., Caplan M. E. Horowitz C. J. Phys. Rev. C902014055805.
  * [8] Alcain P. N., Giménez Molinelli P. A., Dorso C. O. Phys. Rev. C902014065803.
  * [9] Blaschke, D. Chamel, N. in The Physics and Astrophysics of Neutron Stars, vol. 457, edited by Rezzolla L., Pizzochero P., Jones D.I., Rea N. Vidaña I. (Astrophysics and Space Science Library, Springer, Cham) 2018, pp. 337-400.
  * [10] Carreau T., Gulminelli F. Margueron J. Eur. Phys. J. A552019188.
  * [11] Carreau T., Gulminelli F., Chamel N., Fantina A. F. Pearson J. M. Astron. Astrophys.6352020A84.
  * [12] Dinh Thi H., Carreau T., Fantina A. F. Gulminelli F. Astron. Astrophys.6542021A114.
  * [13] Dinh Thi H., Fantina A. F. Gulminelli F. Eur. Phys. J. A572021296.
  * [14] Balliet L. E., Newton W. G., Cantu S. Budimir S. Astrophys. J.918202179.
  * [15] Newton W. G., Murphy K., Hooker J. Li B. A. Astrophys. J. Lett.7792013L4.
  * [16] Horowitz C. J., Berry D. K., Briggs C. M., et al. Phys. Rev. Lett.1142015031102.
  * [17] Lin Z., Caplan M. E. , Horowitz C. J. Lunardini C. Phys. Rev. C1022020045801.
  * [18] Yakovlev D. G. Mon. Not. R. Astron. Soc.4532015 581–590.
  * [19] Schmitt A. Shternin P. in The Physics and Astrophysics of Neutron Stars, vol. 457, edited by Rezzolla L., Pizzochero P., Jones D.I., Rea N. Vidaña I. (Astrophysics and Space Science Library, Springer, Cham) 2018, pp. 455–574.
  * [20] Gearheart M., Newton W. G., Hooker J. Li B. A. Mon. Not. R. Astron. Soc.41820112343-2349.
  * [21] Sotani H., Nakazato K., Iida K. Oyamatsu K. Phys. Rev. Lett1082012201101.
  * [22] Pons J.A., Viganò D. Rea N. Nat. Phys.92013431–434.
  * [23] Margueron J., Hoffmann Casali R. Gulminelli F. Phys. Rev. C972018025805.
  * [24] Margueron J., Hoffmann Casali R. Gulminelli F. Phys. Rev. C972018025806.
  * [25] Newton W.G., Gearheart M. Li B.A. Astrophys. J. Suppl. Ser.20420139.
  * [26] Wang M., Audi G., Kondev F. G., Huang W.J., et al. Chin. Phys. C412017030003.
  * [27] Goriely S., Chamel N., Pearson J. M. Phys. Rev. C882013024308.
  * [28] Roca-Maza X., Viñas X., Centelles M., Ring P. Schuck P. Phys. Rev. C842011054309.
  * [29] Shen H., Toki H., Oyamatsu K. Sumiyoshi K. Nucl. Phys. A6371998435.
  * [30] Drischler C., Hebeler K. Schwenk A. Phys. Rev. C932016054314.
  * [31] Antoniadis J., Freire P. C. C., Wex N., et al. Science34020136131.
  * [32] Gulminelli, F. Fantina, A. F. Nucl. Phys. News3120212.
  * [33] Dinh Thi, H., Mondal, C. Gulminelli, F. Universe72021373.


# Revisiting Automated Prompting: Are We Actually Doing Better?

Yulin Zhou1 Yiren Zhao2 Ilia Shumailov3 Robert Mullins1 Yarin Gal3
1University of Cambridge 2Imperial College London 3University of Oxford
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Current literature demonstrates that Large Language Models (LLMs) are great
few-shot learners, and prompting significantly increases their performance on
a range of downstream tasks in a few-shot learning setting. An attempt to
automate human-led prompting followed, with some progress achieved. In
particular, subsequent work demonstrates that automation can outperform fine-
tuning in certain $K$-shot learning scenarios Shin et al. (2020); Zhang et al.
(2021). In this paper, we revisit techniques for automated prompting on six
different downstream tasks and a larger range of $K$-shot learning settings.
We find that _automated prompting does not consistently outperform simple
manual prompting_. Our work suggests that, in addition to fine-tuning, _manual
prompting should be used as a baseline_ in this line of research.

##  1 Introduction

Transformer-based Large Language Models (LLMs) are now considered foundation
models for downstream tasks Bommasani et al. (2021). The _pre-train then fine-
tune_ approach achieved state-of-the-art performance on a range of Natural
Language Processing (NLP) tasks Liu et al. (2019); Raffel et al. (2020); Brown
et al. (2020). Unfortunately, in many NLP applications, the lack of high-
quality labelled training data is a barrier to producing a model with good
performance in the pre-train and then fine-tune approach. To address this
issue, _prompt-based learning_ Petroni et al. (2019); Schick and Schütze
(2020a, b); Liu et al. (2021a) emerged as a new paradigm for tuning a high-
quality, pre-trained LLM in a _few-shot learning_ scenario, where only a few
samples are available for downstream task learning.

In the prompt-based learning paradigm (Figure 1), an input $X$ is modified
using a template function $p$, also known as a prompting function and has one
or more placeholders called mask tokens <mask>, resulting in a prompted input
$X^{\prime}=p(X)$ Liu et al. (2021b). Additionally, a verbaliser designs an
answer domain $\mathcal{Z}$, so that for an output label domain $\mathcal{Y}$,
there is a many-to-one mapping for an answer
$z\in\mathcal{V}_{y}\subseteq\mathcal{Z}$ to an output label $y\in\mathcal{Y}$
in accordance with the downstream task. Considering a language model $f_{o}$
pre-trained on a large corpus of text, such as Wikipedia, the goal of prompt-
based learning is to fine-tune it on a small dataset of prompted inputs
$X^{\prime}$ and corresponding output $y$, in order to produce a high-quality
language model $f_{p}$ capable of generating an answer $z$ for a given input
$X$.

Figure 1: Sentiment analysis with the prompt-based learning paradigm. Input
$X^{\prime}$ is the prompted input, and there is a many-to-one mapping between
answers $z\in\mathcal{Z}$ and labels $y\in\mathcal{Y}$.

Prompting formulates downstream tasks such as sentiment analysis and text
classification to cloze completion (also known as filling in the blanks).
Furthermore, using prompts and fine-tuning allows models to gain superior few-
shot learning capabilities Lester et al. (2021); Schick and Schütze (2020a);
Shin et al. (2020). Despite the relative success of prompt-based learning, the
design of prompts can be a challenging task. As a result, many research
studies sought to _automate_ the process of designing suitable prompts for
downstream tasks Liu et al. (2021c); Zhang et al. (2021); Shin et al. (2020).
The motivation for automating prompt design is usually two-fold: first,
manually designing prompts can be time-consuming; and second, automated ones
can often provide better performance. In this work, we question _the second
motivation_ and demonstrate that _existing automated prompts do not
consistently outperform their manual counterparts_ under various $K$-shot
learning setups. In this paper, we make the following contributions:

  * •

We thoroughly investigate automated prompts and demonstrate that they do not
consistently outperform manual prompts, even when the latter are created using
basic heuristics and selected among a small number of options (Section 3.2).

  * •

We show empirically that fine-tuning only serves a strong baseline when $K\geq
100$ in a $K$-shot learning setup (Section 3.2).

  * •

By visualising the prompts generated by auto-prompting, we explain why these
prompts are not necessarily better than manually designed ones (Section 3.4).

  * •

Supported by our empirical evidence and evaluation, we strongly recommend that
_future research should consider manual prompts as a simple yet effective
baseline_.

##  2 Related Work

The rise of the _prompting-based learning paradigm_ comes with the development
of LLMs Brown et al. (2020), which were demonstrated to be good few-shot
learners Liu et al. (2021d). To begin with, researchers focused on manually
crafted prompts for downstream tasks Petroni et al. (2019); Liu et al.
(2021b); Scao and Rush (2021); Zhao et al. (2021); Schick and Schütze (2020a),
yet soon shifted towards automated prompt designs. Schick et al. investigated
how to automatically identify label words for a prompt Schick and Schütze
(2020a, b), while Shin et al. proposed AutoPrompt, a framework for
automatically generating prompts for various tasks, through a gradient-based
search Shin et al. (2020). Gao et al. used another LLM, T5 Raffel et al.
(2020), to generate both the prompting templates and verbaliser answer domains
Gao et al. (2020). Han et al. incorporated logic rules into prompt designs,
combining several simple sub-prompts according to these rules Han et al.
(2022). All of the above mentioned methods are based on the assumption that
the prompt design has to rely on discrete tokens.

Liu et al. and Lester et al. demonstrated that prompts could be trainable
continuous embeddings, or soft prompts, instead of discrete tokens. These soft
prompts can be learned with a frozen language model (LLM) on a target task Liu
et al. (2021d); Lester et al. (2021); Zhang et al. (2021). Liu et al. further
discovered that Deep Prompts, which are soft prompts used in every layer of
the model, allow for scaling to large LLMs for complex natural language
processing (NLP) tasks Liu et al. (2021c). Zhang et al. developed
Differentiable Prompts, which put the label tokens design of the prompt into a
continuous space and optimised it jointly with soft prompts Zhang et al.
(2021). An extensive evaluation was conducted by Zhang et al. on various
downstream tasks.

Most of the work on automating prompt design mentioned above has two major
motivations: to reduce the amount of time it takes to design prompts manually;
and to potentially gain better performance, since manual prompt formats can be
sub-optimal Zhang et al. (2021). While the first motivation may be valid in
some cases, it largely depends on the task complexity and the amount of data
available – it is sometimes possible for non-experts to design a prompt
sufficient for simple tasks with a large amount of data. The principal focus
of this work, however, is on the second motivation: _can automated prompts
really outperform manual prompts in a consistent manner?_ A comparison between
automated and manual prompts is lacking in current research. To our knowledge,
automated prompting methods focus solely on comparing to fine-tuning in a few-
shot learning setup, while a comparisons to manual prompting methods remain
unexplored. In this paper, we consider AutoPrompt (Auto) Shin et al. (2020)
and Differential Prompt (Diff) Zhang et al. (2021) as representatives, where
one is based on discrete tokens, while the other is based on continuous
embeddings. We compare them with manually designed prompts and fine-tuning
without prompting on various tasks.

##  3 Evaluation

###  3.1 Experiment setup

A robust framework was developed to assess prompting model performance under
$K$-shot learning scenarios where only $K$ samples per class are available for
the training and validation datasets. Three prompting models were re-
implemented: LM-BFF (manual) Gao et al. (2020), AutoPrompt (Auto) Shin et al.
(2020), and DART (Diff) Zhang et al. (2021) models. During prompt-based
learning, each prompting model is allowed to fine-tune the parameters of the
pre-trained language model using the limited training and validation datasets.

####  3.1.1 Datasets and Model

We conducted comprehensive experiments on six datasets to compare the
performance of prompting models fine-tuned on the pre-trained RoBERTa-large
model Liu et al. (2019). Table 2 in Appendix B shows we picked three sentiment
analysis and three textural entailment tasks.

####  3.1.2 Prompt Templates and Verbalisers

We design prompts to concatenate the input text and the <mask> token,
alongside a verbaliser that maps from the answer domain to the output label
domain. Manually designed prompts and verbalisers are adapted from the Public
Pool of Prompts Bach et al. (2022) and previous work on prompting Gao et al.
(2020); Xu et al. (2022). For each dataset, we selected four to six prompt-
and-verbaliser pairs, compared their performance under the same $K=16$ few-
shot scenario, and picked the best-performing pair for further experiments
with different $K$ values. Detailed manually designed prompts and verbalisers,
as well as their performance measures, are illustrated in Table 3, and the
best-performing pairs are summarised in Table 4 in Appendix C.

An automated discrete prompt replaces the template with trigger tokens <T>.
Following the same settings used in AutoPrompt Shin et al. (2020), we inserted
ten trigger tokens between the input text and the <mask> token. Under a
$K$-shot scenario, the verbaliser mapping is automatically generated from the
train and validation dataset, each with $K$ samples per class. Table 5 in
Appendix D shows the automated discrete prompts and verbalisers for each
dataset. A differential prompt starts from the manually designed prompt but
treats both the template and the verbaliser as a collection of differentiable
parameters.

Take the dataset SST2 as an example: a suitable manually designed prompt could
be ‘‘<sentence> . It was <mask> .’’ with a verbaliser $\\{\texttt{bad}\mapsto
0,\texttt{good}\mapsto{1}\\}$; An automated discrete prompt could be
‘‘<sentence> <T> ... <T> <mask> .’’ with ten trigger tokens <T>.

####  3.1.3 Hyper-parameters

We conducted a beam search using the AdamW optimiser Loshchilov and Hutter
(2017) for the optimal batch size, learning rate and weight decay for each set
of experiments with the same dataset and $K$-shot value. Each experiment is
run with $100$ epochs and an early stopping value of $5$, i.e., when the
validation loss is non-decreasing for $5$ epochs. The detailed hyper-
parameters used in each set of experiments are listed in Table 6, and details
on the evaluation metrics are in Appendix E.

###  3.2 Main Results

Table 1 illustrates the performance of various prompting strategies. We
observe that manual prompts exhibit the best performance in 13 out of the 24
setups (6 different datasets and 4 different $K$s), and the second-best
performance in 8 of them. Automated prompts (both Auto and Diff) only show a
clear advantage in TWEETS-HATE-OFFENSIVE when $K=100$. The baseline in Table 1
is direct fine-tuning on the $K$ samples.

We also see that automated prompts can be catastrophically ineffective in
certain setups. For example, as shown in Table 5, Auto performs much worse
than Manual or Baseline in MNLI-MATCHED when $K=100$. Diff also significantly
underperforms Manual in TWEETS-HATE-OFFENSIVE when $K=16$. In later parts of
this section, we provide an analysis of the generated prompts and explore the
reasons for this phenomenon. Finally, we demonstrate that Baseline sometimes
performs well when $K$ is large. This is seen in SST2 when $K=100,1000$ and
also ENRON-SPAM when $K=100$. In general, we make the following observations:

  * •

Manual prompting outperforms automated prompting (Auto and Diff) with
different $K$-shot setups on most tasks.

  * •

Automated prompting sometimes cannot even outperform fine-tuning, e.g. MNLI-
MISMATCHED $K=100,1000$.

  * •

When $K$ is small, prompting can greatly improve performance, e.g. on SST2 and
MNLI.

  * •

Automated prompting can fail catastrophically (e.g. MNLI-MISMATCHED $K=1000$)
and have a high variance in performance (e.g. $15.5$ standard deviation on
SST2), while manual prompting is more robust.

SST2 QNLI $K$ Baseline Auto Diff Manual Baseline Auto Diff Manual $8$ $59.8\pm
8.6$ $51.7\pm 1.9$ $\boldsymbol{88.0\pm 1.6}$ $\underline{77.6\pm 4.6}$
$49.9\pm 1.0$ $\underline{51.5\pm 0.7}$ $50.5\pm 2.1$ $\boldsymbol{54.6\pm
2.8}$ $16$ $72.1\pm 15.0$ $70.1\pm 3.9$ $\boldsymbol{87.8\pm 0.7}$
$\underline{86.9\pm 1.6}$ $49.9\pm 0.2$ $53.4\pm 1.3$ $\underline{59.5\pm
3.6}$ $\boldsymbol{74.1\pm 1.2}$ $100$ $\boldsymbol{89.6\pm 0.5}$ $83.5\pm
4.3$ $88.6\pm 0.7$ $\underline{89.4\pm 1.0}$ $78.9\pm 2.3$ $74.0\pm 4.3$
$\underline{80.2\pm 2.1}$ $\boldsymbol{82.7\pm 0.7}$ $1000$
$\boldsymbol{92.7\pm 0.2}$ $\underline{92.5\pm 0.2}$ $90.1\pm 0.7$ $92.3\pm
0.2$ $\underline{87.2\pm 1.0}$ $83.2\pm 3.8$ $85.2\pm 1.1$
$\boldsymbol{88.0\pm 0.3}$ MNLI-Matched MNLI-Mismatched $K$ Baseline Auto Diff
Manual Baseline Auto Diff Manual $8$ $34.6\pm 2.4$ $34.2\pm 1.1$
$\underline{51.3\pm 1.1}$ $\boldsymbol{55.7\pm 3.3}$ $33.8\pm 0.8$ $33.8\pm
0.5$ $\underline{47.6\pm 3.0}$ $\boldsymbol{56.0\pm 1.4}$ $16$ $33.3\pm 0.2$
$34.9\pm 0.7$ $\boldsymbol{61.4\pm 1.5}$ $\underline{60.2\pm 3.7}$ $32.8\pm
1.3$ $35.6\pm 0.8$ $\underline{59.4\pm 1.1}$ $\boldsymbol{60.2\pm 2.7}$ $100$
$63.1\pm 1.3$ $42.3\pm 0.5$ $\underline{72.1\pm 0.8}$ $\boldsymbol{74.1\pm
1.2}$ $\underline{73.6\pm 2.1}$ $39.5\pm 1.0$ $73.3\pm 1.2$
$\boldsymbol{77.0\pm 1.2}$ $1000$ $\underline{82.7\pm 0.5}$ $72.9\pm 2.3$
$80.0\pm 0.8$ $\boldsymbol{83.2\pm 0.3}$ $\underline{84.3\pm 0.5}$ $76.6\pm
3.7$ $82.0\pm 0.4$ $\boldsymbol{85.0\pm 0.2}$ ENRON-SPAM TWEETS-HATE-OFFENSIVE
$K$ Baseline Auto Diff Manual Baseline Auto Diff Manual $8$ $49.1\pm 36.6$
$\underline{73.4\pm 6.0}$ $\boldsymbol{80.7\pm 5.7}$ $67.9\pm 12.2$ $14.5\pm
9.5$ $12.1\pm 4.6$ $\boldsymbol{32.5\pm 7.1}$ $\underline{25.8\pm 16.5}$ $16$
$84.2\pm 4.0$ $80.5\pm 2.6$ $\underline{88.0\pm 2.3}$ $\boldsymbol{89.4\pm
3.0}$ $\underline{38.0\pm 4.1}$ $42.5\pm 2.6$ $37.2\pm 7.7$
$\boldsymbol{46.7\pm 2.5}$ $100$ $\boldsymbol{97.1\pm 0.4}$ $90.8\pm 0.4$
$\underline{96.3\pm 0.8}$ $\underline{96.3\pm 0.5}$ $44.9\pm 0.9$
$\underline{51.4\pm 3.4}$ $\boldsymbol{59.7\pm 2.8}$ $47.0\pm 0.8$ $1000$
$98.0\pm 0.5$ $97.0\pm 0.7$ $\boldsymbol{99.0\pm 0.1}$ $\underline{98.7\pm
0.2}$ $66.5\pm 1.5$ $66.8\pm 1.8$ $\boldsymbol{67.7\pm 3.3}$
$\underline{67.5\pm 2.1}$

Table 1: The performance of various prompting methods on RoBERTa-large Liu et
al. (2019) was assessed using numbers reported as percentages, with a mean and
standard deviation across five independent runs. The best and second-best
performing methods are represented in bold and underlined fonts, respectively.
The baseline is fine-tuning only without any prompting, while Auto, Diff, and
Manual correspond to AutoPrompt Shin et al. (2020), Differential Prompt Zhang
et al. (2021), and LM-BFF Gao et al. (2020), respectively.

(a) SST2

(b) QNLI

(c) ENRON-SPAM

Figure 2: The performance of prompting models on the datasets SST2, QNLI Wang
et al. (2018) and ENRON-SPAM Metsis et al. (2006) is shown for a wider range
of $K$ values. The solid line plots the mean accuracy across five independent
runs, and is bounded by one standard deviation on both sides.

###  3.3 More $K$-shot Experiments

Figure 2 demonstrates the performance of different prompting styles with more
$K$ values on SST2, QNLI Wang et al. (2018) and ENRON-SPAM Metsis et al.
(2006).

We observe that the performance of all methods starts to converge with larger
$K$ values, which is consistent with existing literature Shin et al. (2020).
It is also worth mentioning that the automated prompting methods do not
consistently outperform manual prompting on this large range of $K$ values.
More results are available in Appendix F.

###  3.4 Visualizing Auto-prompts

As previously discussed, automated prompting can sometimes fail
catastrophically. Table 5 summarises all the automated discrete prompts and
verbaliser answer domains. Since the answer domain is generated from the $K$
samples per class, it may not be general enough or optimal for the entire
dataset. On the other hand, manual prompts and verbalisers are designed based
on common knowledge that humans possess from countless examples encountered in
daily life. One possible improvement idea on AutoPrompt is to start with a
manually designed prompt and update both the prompt and the verbaliser through
a gradient-based search in an iterative manner.

###  3.5 Limitations

All prompting methods are trying to extract knowledge from the Large Language
Models (LLMs). Our paper compares their knowledge extraction abilities. Thus,
the performance of RoBERTa-large can serve as a reference point and provide
insights for other LLMs. However, it is still necessary to assess each large
language model independently to understand its capabilities comprehensively.

We only tested a handful of simple manual prompt-and-verbaliser pairs which
are included in Tables 3 and 4. It is entirely possible that there is a lot of
room for improvement in the design of manual prompt-and verbaliser pairs, thus
providing us a even stronger baseline. We have opted to use ten trigger tokens
in Auto, in alignment with the experiment settings originally presented in the
AutoPrompt paper Shin et al. (2020). However, since the verbaliser domains
generated under few-shot learning settings are noisy, reducing the number of
trigger tokens may improve performance.

##  4 Conclusion

In this paper, we revisit the results generated from automated prompting, and
show that _automated prompting cannot consistently outperform simple manual
prompting on a variety of tasks_. We also demonstrate that the performance of
automated prompting is heavily dependent on the amount of data available, and
in some cases can even be worse than fine-tuning. On the other hand, manual
prompting is more robust to the amount of data available, and can have similar
performance to fine-tuning if not outperforming. We take a closer look at the
prompts and verbalisers generated by automated discrete prompting (AutoPrompt)
and point out that few-shot learning settings make it challenging to generate
prompts and verbalisers that perform well. We hope that this work will
motivate researchers to use manual prompts as a general baseline.

## Acknowledgment

The authors would like to thank the anonymous reviewers for their helpful
suggestions.

## References

  * Bach et al. (2022) Stephen H. Bach, Victor Sanh, Zheng-Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-David, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged S. Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-Jian Jiang, and Alexander M. Rush. 2022.  Promptsource: An integrated development environment and repository for natural language prompts.
  * Bommasani et al. (2021) Rishi Bommasani, Drew A Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. 2021.  On the opportunities and risks of foundation models.  _arXiv preprint arXiv:2108.07258_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Davidson et al. (2017) Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017.  Automated hate speech detection and the problem of offensive language.  _arXiv:1703.04009_.
  * Gao et al. (2020) Tianyu Gao, Adam Fisch, and Danqi Chen. 2020.  Making pre-trained language models better few-shot learners.  _arXiv preprint arXiv:2012.15723_.
  * Han et al. (2022) Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. 2022.  Ptr: Prompt tuning with rules for text classification.  _AI Open_.
  * Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.  The power of scale for parameter-efficient prompt tuning.  _arXiv preprint arXiv:2104.08691_.
  * Liu et al. (2021a) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021a.  What makes good in-context examples for gpt-$3$?  _arXiv preprint arXiv:2101.06804_.
  * Liu et al. (2021b) Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021b.  Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing.  _arXiv preprint arXiv:2107.13586_.
  * Liu et al. (2021c) Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021c.  P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks.  _arXiv preprint arXiv:2110.07602_.
  * Liu et al. (2021d) Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021d.  Gpt understands, too.  _arXiv preprint arXiv:2103.10385_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Loshchilov and Hutter (2017) Ilya Loshchilov and Frank Hutter. 2017.  Decoupled weight decay regularization.
  * Metsis et al. (2006) Vangelis Metsis, Ion Androutsopoulos, and Georgios Paliouras. 2006.  Spam filtering with naive bayes - which naive bayes?  In _International Conference on Email and Anti-Spam_.
  * Petroni et al. (2019) Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H Miller, and Sebastian Riedel. 2019.  Language models as knowledge bases?  _arXiv preprint arXiv:1909.01066_.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(140):1–67.
  * Scao and Rush (2021) Teven Le Scao and Alexander M Rush. 2021.  How many data points is a prompt worth?  _arXiv preprint arXiv:2103.08493_.
  * Schick and Schütze (2020a) Timo Schick and Hinrich Schütze. 2020a.  Exploiting cloze questions for few shot text classification and natural language inference.  _arXiv preprint arXiv:2001.07676_.
  * Schick and Schütze (2020b) Timo Schick and Hinrich Schütze. 2020b.  It’s not just size that matters: Small language models are also few-shot learners.  _arXiv preprint arXiv:2009.07118_.
  * Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020.  Autoprompt: Eliciting knowledge from language models with automatically generated prompts.  _arXiv preprint arXiv:2010.15980_.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv:1804.07461_.
  * Xu et al. (2022) Lei Xu, Yangyi Chen, Ganqu Cui, Hongcheng Gao, and Zhiyuan Liu. 2022.  Exploring the universal vulnerability of prompt-based learning paradigm.
  * Zhang et al. (2021) Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. 2021.  Differentiable prompt makes pre-trained language models better few-shot learners.  _arXiv preprint arXiv:2108.13161_.
  * Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate before use: Improving few-shot performance of language models.  In _International Conference on Machine Learning_ , pages 12697–12706. PMLR.

##  Appendix A Model and infrastructure details

The RoBERTa-large model Liu et al. (2019) is pre-trained on a large corpus of
raw English text using masked language modelling (MLM) objective; it contains
$354$ million parameters.

All our experiments are run parallelly on 4 NVIDIA Tesla V100 GPUs; for
smaller $K$ values (e.g., $K=100$), most experiments require less than 1 GPU
hour, while a setting with a larger $K$ value (e.g., $K=1000$) may require 2
GPU hours.

##  Appendix B Dataset details

We conducted comprehensive experiments on six datasets (SST2, QNLI, MNLI-
MATCHED, MNLI-MISMATCHED, ENRON-SPAM and TWEETS-HATE-OFFENSIVE) to compare the
performance of prompting models fine-tuned on the pre-trained RoBERTa-large
model. As shown in Table 2, we picked three sentiment analysis and three
textural entailment tasks. Among the six, three are binary classifications
(SST2, QNLI and ENRON-SPAM), while the remaining datasets have three
categories each (MNLI-MATCHED, MNLI-MISMATCHED and TWEETS-HATE-OFFENSIVE).

Dataset # Class Test Sample Description SST2 2 33674 A sentiment analysis task
on movie reviews from the GLUE benchmark Wang et al. (2018). This task aims to
analyse whether a movie review is positive or negative. QNLI 2 5463 A textual
entailment task on question-answer pairs from the GLUE benchmark Wang et al.
(2018). The objective is to determine whether the context sentence contains
the answer to the question. MNLI-MATCHED 3 4907 A multi-class (i.e.,
entailment, neutral, contradiction) textual entailment task on premise-
hypothesis pairs from the GLUE benchmark Wang et al. (2018). Matched version
only preserves pairs within the same genre (e.g., science fiction, speech).
MNLI-MISMATCHED 3 4916 Same as MNLI-MATCHED, the mismatched version is a
textual entailment task on premise-hypothesis pairs from the GLUE benchmark
Wang et al. (2018), but it only preserves pairs within different genres.
ENRON-SPAM 2 15858 A safety critical binary sentiment analysis task
determining whether an email text is a spam Metsis et al. (2006). TWEETS-HATE-
OFFENSIVE 3 12391 A safety critical multi-class sentiment analysis task which
aims to classify whether a tweet text contains hate speech, offensive speech
or neither Davidson et al. (2017).

Table 2: Six datasets selected in the project. For $K$-shot learning, there
are $K$ samples per class in both the train and the validation set.

##  Appendix C Manual prompt-and-verbaliser designs

SST2 QNLI Prompt Design Answer $\mapsto$ Label Accuracy Prompt Design Answer
$\mapsto$ Label Accuracy <sentence> . It was <mask> . terrible $\mapsto$ 0,
great $\mapsto$ 1 $86.0\pm 2.7$ <question> ? <mask> , <sentence> . {Yes
$\mapsto$ 0, $64.5\pm 4.8$ bad $\mapsto$ 0, good $\mapsto$ 1
$\boldsymbol{86.9\pm 1.6}$ <question> . <mask> , <sentence> . No $\mapsto$ 1}
$60.5\pm 2.3$ dog $\mapsto$ 0, cat $\mapsto$ 1 $84.7\pm 3.4$ <question> ?
<mask> <sentence> . $68.7\pm 3.2$ cat $\mapsto$ 0, dog $\mapsto$ 1 $68.7\pm
6.9$ <sentence> ? <mask> , <question> . $\boldsymbol{74.1\pm 1.2}$ great
$\mapsto$ 0, terrible $\mapsto$ 1 $67.4\pm 5.0$ <question> <mask> <sentence>
$50.0\pm 0.2$ <sentence> ? <mask> , <question> $66.7\pm 10.2$ MNLI-Matched
MNLI-Mismatched Prompt Design Answer $\mapsto$ Label Accuracy Prompt Design
Answer $\mapsto$ Label Accuracy <premise> ? <mask> , <hypothesis> . {Yes
$\mapsto$ 0, $\boldsymbol{60.2\pm 3.7}$ <premise> ? <mask> , <hypothesis> .
{Yes $\mapsto$ 0, $\boldsymbol{60.2\pm 2.7}$ <premise> . <mask> , <hypothesis>
. Maybe $\mapsto$ 1, $58.6\pm 4.8$ <premise> . <mask> , <hypothesis> . Maybe
$\mapsto$ 1, $56.3\pm 1.5$ <premise> ? <mask> <hypothesis> . No $\mapsto$ 2}
$55.6\pm 1.7$ <premise> ? <mask> <hypothesis> . No $\mapsto$ 2} $58.4\pm 1.1$
<hypothesis> ? <mask> , <premise> . $51.9\pm 4.2$ <hypothesis> ? <mask> ,
<premise> . $57.9\pm 0.8$ <premise> <mask> <hypothesis> $51.2\pm 4.2$
<premise> <mask> <hypothesis> $49.4\pm 2.4$ <hypothesis> ? <mask> , <premise>
$52.4\pm 2.9$ <hypothesis> ? <mask> , <premise> $56.0\pm 1.0$ ENRON-SPAM
TWEETS-HATE-OFFENSIVE Prompt Design Answer $\mapsto$ Label F1 score Prompt
Design Answer $\mapsto$ Label F1 score <mask> : <text> . ham $\mapsto$ 0, spam
$\mapsto$ 1 $82.8\pm 1.9$ <tweet> . This post is <mask> . {hateful $\mapsto$
0, $\boldsymbol{46.7\pm 2.5}$ This is a <mask> : <text> . ham $\mapsto$ 0,
spam $\mapsto$ 1 $82.8\pm 2.8$ This post is <mask> : <tweet> . offensive
$\mapsto$ 1, $40.3\pm 3.8$ <mask> email : <text> . genuine $\mapsto$ 0, spam
$\mapsto$ 1 $\boldsymbol{89.4\pm 3.0}$ <tweet> . This was <mask> . harmless
$\mapsto$ 2} $39.8\pm 4.5$ <text> . This was a <mask> . ham $\mapsto$ 0, spam
$\mapsto$ 1 $76.8\pm 3.3$ <mask> speech : <tweet> . $36.8\pm 11.7$

Table 3: The prompt-and-verbaliser pairs are tested under the few-shot
scenario $K=16$, and the best-performing pair is highlighted in bold. The mean
and standard deviation of scores are computed across five independent runs.

In the Prompt Templates and Verbalisers part in Section 3.1.2, we discussed
how we picked the best-performing prompt-and-verbaliser pairs. We show the
picked manual prompt with their picked verbalisers in Table 3, covering SST2,
QNLI, MNLI-MATCHED, MNLI-MISMATCHED, ENRON-SPAM and TWEETS-HATE-OFFENSIVE. The
underlying mechanism for finding a good manual prompt is detailed in Section
3.1.2. As one can see in these tables, the manual prompts used are very simple
and requires minimal domain knowledge.

##  Appendix D Generated Auto-prompts

Dataset Prompt Design Answer $\mapsto$ Label SST2 <sentence> . It was <mask> .
bad $\mapsto$ 0, good $\mapsto$ 1 QNLI <sentence> ? <mask> , <question> . Yes
$\mapsto$ 0, No $\mapsto$ 1 MNLI-MATCHED <premise> ? <mask> , <hypothesis> .
Yes $\mapsto$ 0, Maybe $\mapsto$ 1, No $\mapsto$ 2 MNLI-MISMATCHED <premise> ?
<mask> , <hypothesis> . Yes $\mapsto$ 0, Maybe $\mapsto$ 1, No $\mapsto$ 2
ENRON-SPAM <mask> email : <text> . genuine $\mapsto$ 0, spam $\mapsto$ 1
TWEETS-HATE-OFFENSIVE <tweet> . This post is <mask> . hateful $\mapsto$ 0,
offensive $\mapsto$ 1, harmless $\mapsto$ 2

Table 4: Summarised for each dataset, the best-performing manual prompt and
verbaliser.

Task Prompt design $\boldsymbol{K}$ Answer $\boldsymbol{\mapsto}$ Label SST2
$8$ impunity $\mapsto$ 0, ASHINGTON $\mapsto$ 1 <sentence> <T> <T> <T> <T> <T>
$16$ worthless $\mapsto$ 0, Kom $\mapsto$ 1 <T> <T> <T> <T> <T> <mask> . $32$
Worse $\mapsto$ 0, å¤© $\mapsto$ 1 $64$ horrible $\mapsto$ 0, magic $\mapsto$
1 $100$ worse $\mapsto$ 0, å¤© $\mapsto$ 1 $1000$ worse $\mapsto$ 0, Excellent
$\mapsto$ 1 QNLI $8$ implement $\mapsto$ 0, defensively $\mapsto$ 1 <question>
<mask> <T> <T> <T> <T> <T> $16$ counter $\mapsto$ 0, Bits $\mapsto$ 1 <T> <T>
<T> <T> <T> <sentence> $32$ Meteor $\mapsto$ 0, univers $\mapsto$ 1 $64$ ormon
$\mapsto$ 0, stood $\mapsto$ 1 $100$ idelines $\mapsto$ 0, opard $\mapsto$ 1
$1000$ Ģ $\mapsto$ 0, overloaded $\mapsto$ 1 MNLI-MATCHED $8$ efforts
$\mapsto$ 0, democratically $\mapsto$ 1, Congratulations $\mapsto$ 2 <premise>
<mask> <T> <T> <T> <T> <T> $16$ OWN $\mapsto$ 0, hypocritical $\mapsto$ 1,
examiner $\mapsto$ 2 <T> <T> <T> <T> <T> <hypothesis> $32$ Alicia $\mapsto$ 0,
historians $\mapsto$ 1, BF $\mapsto$ 2 $64$ tweets $\mapsto$ 0, onboard
$\mapsto$ 1, Anniversary $\mapsto$ 2 $100$ filmmakers $\mapsto$ 0, combat
$\mapsto$ 1, absence $\mapsto$ 2 $1000$ thus $\mapsto$ 0, MED $\mapsto$ 1,
independent $\mapsto$ 2 MNLI-MISMATCHED $8$ Whilst $\mapsto$ 0, oka $\mapsto$
1, smokers $\mapsto$ 2 <premise> <mask> <T> <T> <T> <T> <T> $16$ Accordingly
$\mapsto$ 0, )? $\mapsto$ 1, foreigners $\mapsto$ 2 <T> <T> <T> <T> <T>
<hypothesis> $32$ ibliography $\mapsto$ 0, qa $\mapsto$ 1, Governments
$\mapsto$ 2 $64$ LER $\mapsto$ 0, jack $\mapsto$ 1, foreigners $\mapsto$ 2
$100$ HEL $\mapsto$ 0, gaming $\mapsto$ 1, imperialism $\mapsto$ 2 $1000$
Vladimir $\mapsto$ 0, acting $\mapsto$ 1, dislike $\mapsto$ 2 ENRON-SPAM $8$
Reviewer $\mapsto$ 0, Pure $\mapsto$ 1 <question> <mask> <T> <T> <T> <T> <T>
$16$ debian $\mapsto$ 0, Discount $\mapsto$ 1 <T> <T> <T> <T> <T> <sentence>
$32$ hillary $\mapsto$ 0, Vampire $\mapsto$ 1 $64$ schedules $\mapsto$ 0,
Romance $\mapsto$ 1 $100$ subcommittee $\mapsto$ 0, Beauty $\mapsto$ 1 $1000$
committee $\mapsto$ 0, ophobic $\mapsto$ 1 TWEETS-HATE-OFFENSIVE $8$ Slater
$\mapsto$ 0, herself $\mapsto$ 1, issued $\mapsto$ 2 <premise> <mask> <T> <T>
<T> <T> <T> $16$ kicking $\mapsto$ 0, her $\mapsto$ 1, selections $\mapsto$ 2
<T> <T> <T> <T> <T> <hypothesis> $32$ athi $\mapsto$ 0, herself $\mapsto$ 1,
vernight $\mapsto$ 2 $64$ racist $\mapsto$ 0, Marie $\mapsto$ 1, skies
$\mapsto$ 2 $100$ racist $\mapsto$ 0, vaginal $\mapsto$ 1, Miracle $\mapsto$ 2
$1000$ homophobia $\mapsto$ 0, b***h $\mapsto$ 1, heavens $\mapsto$ 2

Table 5: Auto prompts designed alongside with the automatically generated
verbalisers for each dataset.

In the Prompt Templates and Verbalisers part in Section 3.1.2, we also
mentioned that an automated discrete prompt replaces the template with trigger
tokens <T>. Following the same settings used in AutoPrompt Shin et al. (2020),
we inserted ten trigger tokens between the input text and the <mask> token.
All automated discrete prompts and their automatically generated verbalisers
are listed in Table 5. In contrast to the manual prompts shown in Appendix C,
the auto-prompts generated are now more complex.

Dataset Model Batch Size $\boldsymbol{\eta}$ $\boldsymbol{w_{d}}$ Dataset
Model Batch Size $\boldsymbol{\eta}$ $\boldsymbol{w_{d}}$ SST2 Auto 8 1e-5
0.01 QNLI Auto 4 2e-5 0.1 Diff 8 1e-5 0.01 Diff 4 1e-5 0.1 Manual 4 2e-5 0.01
Manual 4 2e-5 0.01 MNLI-MATCHED Auto 4 2e-5 0.01 MNLI-MISMATCHED Auto 4 2e-5
0.01 Diff 4 1e-5 0.01 Diff 8 1e-5 0.01 Manual 4 2e-5 0.01 Manual 4 2e-5 0.01
ENRON-SPAM Auto 8 1e-5 0.01 TWEETS-HATE-OFFENSIVE Auto 8 2e-5 0.1 Diff 8 2e-5
0.0 Diff 8 2e-5 0.0 Manual 8 2e-5 0.05 Manual 8 2e-5 0.1

Table 6: Details of the selected hyper-parameters, including batch size,
learning rate $\eta$ and weight decay $w_{d}$ for each set of experiments with
the same dataset and prompting model.

##  Appendix E Hyper-parameters and evaluation metrics for training

In terms of the evaluation metrics which measure the performance of the
prompting models, we utilised two different metrics according to the nature of
the datasets: (1) Multi-class classification accuracy for balanced datasets
SST2, QNLI, MNLI-MATCHED and MNLI-MISMATCHED. (2) F1 score captures both
precisions and recalls for safety-critical or unbalanced datasets ENRON-SPAM
and TWEETS-HATE-OFFENSIVE.

Table 6 provides details for the training setups. We show the batch sizes,
learning rates and weight decay values used in the experiments. We also show
the optimal hyper-parameters for each set of experiments with the same dataset
and prompting model. For example, the optimal hyper-parameters for the dataset
SST2 with the prompting model Auto are batch size $8$, learning rate $10^{-5}$
and weight decay $0.01$.

(a) MNLI-MATCHED

(b) MNLI-MISMATCHED

(c) TWEETS-HATE-OFFENSIVE

Figure 3: The performance of prompting models on MNLI-MATCHED, MNLI-MISMATCHED
Wang et al. (2018) and TWEETS-HATE-OFFENSIVE Davidson et al. (2017) is shown
for a wider range of $K$ values. The solid line plots the mean accuracy across
five independent runs, and is bounded by one standard deviation on both sides.

##  Appendix F Additional results for more K-shot experiments

In Figure 2 (Section 3.3), we show the performance with more $K$ values for
SST2, QNLI and ENRON-SPAM. Additional results in the same setup are shown in
Figure 3.