# Fairness in TabNet Model by Disentangled Representation for the Prediction
of Hospital No-Show

Sabri Boughorbel
Qatar Computing Research Institute, Doha, Qatar
<EMAIL_ADDRESS>Fethi Jarray University of Gabes, Medenine, Tunisia
<EMAIL_ADDRESS>
Abdou Kadri
Sidra Medicine, Doha, Qatar
<EMAIL_ADDRESS>

###### Abstract

Patient no-shows is a major burden for health centers leading to loss of
revenue, increased waiting time and deteriorated health outcome. Developing
machine learning (ML) models for the prediction of no-shows could help
addressing this important issue. It is crucial to consider fair ML models for
no-show prediction in order to ensure equality of opportunity in accessing
healthcare services. In this work, we are interested in developing deep
learning models for no-show prediction based on tabular data while ensuring
fairness properties. Our baseline model, TabNet, uses on attentive feature
transformers and has shown promising results for tabular data. We propose
Fair-TabNet based on representation learning that disentangles predictive from
sensitive components. The model is trained to jointly minimize loss functions
on no-shows and sensitive variables while ensuring that the sensitive and
prediction representations are orthogonal. In the experimental analysis, we
used a hospital dataset of 210,000 appointments collected in 2019. Our
preliminary results show that the proposed Fair-TabNet improves the
predictive, fairness performance and convergence speed over TabNet for the
task of appointment no-show prediction. The comparison with the state-of-the
art models for tabular data shows promising results and could be further
improved by a better tuning of hyper-parameters.

_K_ eywords P

atient No-Show, Tabular Data, Attentive Transformers, Fairness,
Disentanglement, Representation Learning, Deep Learning.

##  1 Introduction

Patients who do not attend scheduled clinic appointments are referred to as
no-shows. The latter is a common problem in clinics with a major impact on the
economics, operation and health outcome of health centers. For example average
rate of patient no-show the U.S. is estimated to be 18.8 % with a cost of 150
Billion in revenue loss [Gier, 2017, Kheirkhah et al., 2015]. No-shows affect
the recruitment of additional medical staff, disrupt appointment scheduling
and extend patient waiting. These missed appointments are also causing health
problems due to the interrupted or delayed follow-ups on patient conditions.
Several approaches have been considered to tackle patient no-shows such as
sending reminder, imposing sanctions or overbooking. Personalized reminders
and intelligent overbooking are promising directions that are cost effective.
These approaches require developing machine learning models that can perform
individualized no-show prediction based on operational and health data.
Several work have been published on the prediction of no-show using machine
learning models [Carreras-García et al., 2020, Li et al., 2019, AlMuhaideb et
al., 2019]. The most commonly data features for no-show prediction can be
categorized into: patient demographic, medical history, appointment details
and patient behaviour [Carreras-García et al., 2020]. Such data is typically
available in tabular format. Therefore traditional machine learning models
such as decision trees are most commonly applied. It is attractive to develop
deep learning models for tabular data.

##  2 Related work

We limit this section to references to previous work on fairness by
representation learning via disentanglement. [Louizos et al., 2015] proposed
the Variational Fair Autoencoder (FVAE) where the Independence between the
target and sensitive variables was controlled by the Maximum Mean Discrepancy
(MMD). The invariance was also controlled through generative adversarial
network where one component maximizes the accuracy of the model and the other
component minimizes the dependency between sensitive and target variables
[Madras et al., 2018, Zhang et al., 2018]. Creager et al. [Creager et al.,
2019] proposed a fair disentangled representation where a set of sensitive
variables and target variable is given at training but the true sensitive
variable is only known at test time. Recently [Sarhan et al., 2020] proposed
an orthogonal disentangled representation that enforces the meaningful
representation to be independent of the sensitive information. [Locatello et
al., 2019] showed that fairness can be improved through disentanglement
representation.

##  3 The Proposed Model: Fair-TabNet

Our goal is to develop a fair deep learning model for no-show classification
suited for tabular data. Our baseline model is TabNet which has shown good
predictive performance compared with the sate-of-the art models for tabular
data [Arik and Pfister, 2019]. The main building blocks in TabNet are: 1)
Feature transformers, 2) Attentive transformers and 3) Masking. We propose
Fair-TabNet by introducing an additional component (${\bf r}_{s}$) in TabNet
representation. ${\bf r}_{s}$ learns to correctly classify the sensitive
variables on the training set. As depicted in Figure 1, the loss
$\mathcal{L}_{sens}$ ensures that ${\bf r}_{s}$ is a good representation of
sensitive variables. A second loss term,
$\mathcal{L}_{diff}=||\mathbf{r}_{p}^{T}\mathbf{r}_{s}||^{2}_{F}$, where
$||\cdot||^{2}_{F}$ is the squared Frobenius norm, encourages orthogonality
between sensitive ${\bf r}_{s}$ and the predictive representation ${\bf
r}_{p}$. Hence this helps the disentanglement of both representations. The
loss term $\mathcal{L}_{diff}$ is inspired by Deep Separation Networks
[Bousmalis et al., 2016].

Fair-TabNet is trained to minimize the weighted loss function
$\mathcal{L}=\mathcal{L}_{pred}+\lambda_{d}\mathcal{L}_{sens}+\lambda_{s}\mathcal{L}_{diff}$
where $\mathcal{L}_{pred}$ is the binary cross-entropy loss on prediction
targets (no-show), $\mathcal{L}_{sens}$ is a categorical cross entropy loss on
the sensitive variables (gender and nationality). $\lambda_{d}$ and
$\lambda_{s}$ are hyper-parameters that control the contribution of the
introduced loss functions to the overall loss.

Metric | AU-ROC | AU-PRC
---|---|---
LightGBM | 77.88$\pm$0.19 | 46.3$\pm$0.46
CatBoost | 78.26$\pm$0.46 | 47.84$\pm$0.82
XGBoost | 78.74$\pm$0.42 | 47.8$\pm$0.77
TabNet | 75.93$\pm$1.78 | 43.83$\pm$3.83
Fair-TabNet | 76.38$\pm$1.42 | 44.2$\pm$3.12
Table 1: Summary of Prediction performance.  Figure 1: Architecture of the
proposed Fair-TabNet Model. The red lines, $\mathcal{L}_{diff}$ and
$\mathcal{L}_{sens}$ are the proposed extension. TabNet architecture is
depicted in Figure A.4.

##  4 Experiments

We used an appointment scheduling dataset from our hospital. The dataset,
collected in 2019, gathered 211,028 appointments. The features are 1) patient
information such as age, gender and nationality, 2) appointment information
such as date, time, appointment duration, clinics, physician specialities,
time from booking to appointment, new or follow-up appointment. In total the
dataset included 42 features, in a tabular format, with numerical and
categorical columns. The considered sensitive variables are Gender and
Nationality. We are interested in group fairness. For Nationality variable we
binarized the samples into local vs. foreigner. For the baseline TabNet, there
are several hyper-parameters to choose for the experiments. We chose
$n_{p}$=16 for the size of $r_{p}$, $n_{a}=16$, for the size of $r_{a}$,
$n_{steps}=5$ for the number of steps, $\gamma=1.5$. A better tuning of the
hyper-parameters could lead to improvement in the predictive performance. For
the proposed Fair-TabNet we chose the same parameters as in TabNet. The
additional parameter in Fair-TabNet is $n_{s}=16$ for the size of the
sensitive representation. The dataset is randomly split into training (70%),
validation (15%) and test (15 %) such that appointments of each day will be
all in one of the three subsets. The splitting and models training is repeated
30 times to estimate the mean and standard deviation of performance metrics.
The chosen baseline methods for classification based on tabular are XGBoost,
LightGBM and CatBoost. Prediction performance is estimated using Area Under
ROC (AU-ROC) and Area Under Precision-Recall curve (AU-PRC). The results are
summarized in Table 1. Typically incorporating fairness properties in ML
models lead to a drop in predictive performance. This is due to penalizing the
sensitive variables to control its contributions towards prediction. However
Fair-TabNet has overall improved on both predictive and fairness performance
compared to TabNet. The proposed loss terms in Fair-TabNet introduce a
regularization effect in the model as well as a disentanglement inductive bias
that could explain the improvement in predictive performance compared with
TabNet. Table 2 gives a summary on the fairness performance of the compared
models. The definition of the fairness metrics are given in Table A.3. For all
the metrics being close to zero is the best in terms of fairness except for
DIR where the metric is close to 100. Figure 2 depicts a T-SNE plot of Fair-
TabNet sensitive representation where the colors indicates different sub-
categories of the sensitive variables. The learned representation ${\bf
r}_{s}$ is able to cluster well the sensitive variables. The loss function
$\mathcal{L}_{sens}$ was trained with a sensitive target variable with the
following sub-categories: Female-Foreigner, Female-Local, Male-Foreigner,
Male-Local. These categories are used as multi-class target variable in
$\mathcal{L}_{sens}$. We note that the sensitive representation was able to
cluster the two sub-categories in gender together due to the presence of the
sensitive variables in the input features. Figure 3 depicts a T-SNE plot of
the predictive representation. The orthogonality constraint added by
$\mathcal{L}_{diff}$ is reflected in this plot. All sensitive sub-categories
are intermingled. It is not possible to identify clusters of sensitive
variables. Table 2 summarises the results comparing the different models based
on the fairness metrics. We also applied a t-test to assess the statistical
significance of the performance difference across models obtained from the
different training splits as depicted in Figures A.6, A.7 and A.8.

Figure 2: T-SNE plot of the sensitive representation ${\bf r}_{s}$ Figure 3:
T-SNE plot of the predictive representation ${\bf r}_{p}$

Metric | LightGBM | CatBoost | XGBoost | TabNet | Fair-TabNet
---|---|---|---|---|---
AOE | 10.16$\pm$1.1 | 1.22$\pm$1.2 | 0.75$\pm$0.3 | 0.8$\pm$0.4 | 1.35$\pm$1.2
DG-FPR | 12$\pm$0.9 | 1.02$\pm$1.4 | 0.43$\pm$0.1 | 0.57$\pm$0.5 | 0.71$\pm$0.6
DIR | 67.7$\pm$1.9 | 72.74$\pm$7.1 | 74.34$\pm$5.4 | 74.7$\pm$10.8 | 79.92$\pm$16.5
EOD | 8.3$\pm$1.3 | 1.4$\pm$1.2 | 1.1$\pm$0.7 | 1.1$\pm$0.8 | 1.99$\pm$2.2
SPD | 13.0$\pm$0.9 | 1.5$\pm$1.7 | 0.9$\pm$0.2 | 0.9$\pm$0.6 | 1.2$\pm$0.8

Table 2: Fairness performance of the compared models. The sensitive variable
is Nationality (local vs. foreigner).

##  5 Conclusion

We proposed Fair-TabNet a deep learning model suited for tabular data as an
extension of TabNet model to incorporate fairness properties for the
prediction appointment no-show from tabular data. Fair-TabNet learns a
disentangled representation which lead to gain in predictive and fairness
performance. The predictive performance of TabNet is close to the state-of-art
approach and could be improved by a better tuning of hyper-parameters.

## References

  * [AlMuhaideb et al., 2019] AlMuhaideb, S., Alswailem, O., Alsubaie, N., Ferwana, I., and Alnajem, A. (2019).  Prediction of hospital no-show appointments through artificial intelligence algorithms.  Annals of Saudi Medicine, 39(6):373–381.
  * [Arik and Pfister, 2019] Arik, S. O. and Pfister, T. (2019).  Tabnet: Attentive interpretable tabular learning.  arXiv preprint arXiv:1908.07442.
  * [Bousmalis et al., 2016] Bousmalis, K., Trigeorgis, G., Silberman, N., Krishnan, D., and Erhan, D. (2016).  Domain separation networks.  In Advances in neural information processing systems, pages 343–351.
  * [Carreras-García et al., 2020] Carreras-García, D., Delgado-Gómez, D., Llorente-Fernández, F., and Arribas-Gil, A. (2020).  Patient no-show prediction: A systematic literature review.  Entropy, 22(6):675.
  * [Creager et al., 2019] Creager, E., Madras, D., Jacobsen, J.-H., Weis, M. A., Swersky, K., Pitassi, T., and Zemel, R. (2019).  Flexibly fair representation learning by disentanglement.  arXiv preprint arXiv:1906.02589.
  * [Gier, 2017] Gier, J. (2017).  Missed appointments cost the us healthcare system $150 b each year.  Health Management Technology, 2.
  * [Kheirkhah et al., 2015] Kheirkhah, P., Feng, Q., Travis, L. M., Tavakoli-Tabasi, S., and Sharafkhaneh, A. (2015).  Prevalence, predictors and economic consequences of no-shows.  BMC health services research, 16(1):1–6.
  * [Li et al., 2019] Li, Y., Tang, S. Y., Johnson, J., and Lubarsky, D. A. (2019).  Individualized no-show predictions: Effect on clinic overbooking and appointment reminders.  Production and Operations Management, 28(8):2068–2086.
  * [Locatello et al., 2019] Locatello, F., Abbati, G., Rainforth, T., Bauer, S., Schölkopf, B., and Bachem, O. (2019).  On the fairness of disentangled representations.  In Advances in Neural Information Processing Systems, pages 14611–14624.
  * [Louizos et al., 2015] Louizos, C., Swersky, K., Li, Y., Welling, M., and Zemel, R. (2015).  The variational fair autoencoder.  arXiv preprint arXiv:1511.00830.
  * [Madras et al., 2018] Madras, D., Creager, E., Pitassi, T., and Zemel, R. (2018).  Learning adversarially fair and transferable representations.  arXiv preprint arXiv:1802.06309.
  * [Sarhan et al., 2020] Sarhan, M. H., Navab, N., Eslami, A., and Albarqouni, S. (2020).  Fairness by learning orthogonal disentangled representations.  arXiv preprint arXiv:2003.05707.
  * [Zhang et al., 2018] Zhang, B. H., Lemoine, B., and Mitchell, M. (2018).  Mitigating unwanted biases with adversarial learning.  In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society, pages 335–340.

## Appendix A

Figure A.4: TabNet architecture. Figure taken from [Arik and Pfister, 2019] Metric |  Name | Formula | Best value
---|---|---|---
AOE |  Average odds error | $\frac{1}{2}(|FPR_{S=1}-FPR_{S=0}|+|TPR_{S=1}-TPR_{S=0}|)$ | 0
DG-FPR |  Difference generalized FPR | $\mathbb{E}_{X:S=1}(\hat{Y}|Y=0)-\mathbb{E}_{X:S=0}(\hat{Y}|Y=0)$ | 0
DIR |  Disparate Impact Ratio | $\frac{TPR_{S=1}}{TPR_{S=0}}$ | 100
EOD |  Equal opportunity difference | $TPR_{S=1}-TPR_{S=0}$ | 0
SPD |  Statistical parity difference | $Pr(Y=1|S=1)-Pr(Y=1|S=0)$ | 0
Table A.3: List of fairness metrics. $\hat{Y}$ and $Y$ are the predicted and the true classes respectively. The sensitive variable $S=1$ and $S=0$ for the deprived and privileged classes respectively. Note that generalized FPR=FPR for binary classification Metric | lightgbm | catboost | xgboost | TabNet | Fair-TabNet
---|---|---|---|---|---
AOE | 5.34$\pm$0.82 | 2.4$\pm$1.36 | 1.83$\pm$0.5 | 1.94$\pm$0.97 | 1.88$\pm$1.34
DG-FPR | 4.62$\pm$0.67 | 0.6$\pm$0.97 | 0.19$\pm$0.1 | 0.4$\pm$0.34 | 0.46$\pm$0.46
DIR | 84.3$\pm$1.74 | 70.53$\pm$4.5 | 73.23$\pm$4.3 | 70.94$\pm$7.01 | 79.64$\pm$18.88
EOD | 6.06$\pm$1.12 | 4.2$\pm$1.86 | 3.47$\pm$0.99 | 3.49$\pm$1.73 | 3.3$\pm$2.3
SPD | 5.55$\pm$0.69 | 1.43$\pm$1.17 | 0.93$\pm$0.17 | 1.12$\pm$0.53 | 0.99$\pm$0.83
Table A.4: Fairness metrics of the different compared models. The sensitive
variable is gender. Figure A.5: Loss functions during Fair-TabNet training.
Figure A.6: Prediction performance comparison in terms of AU-ROC (Area Under
ROC) and AU-PRC (Area Under Precision Recall). The values indicate the
difference of in performance average between rows and columns. Positive values
indicate that the model in row has a higher performance than model in column.
The color indicates $-log_{10}(p_{value})$ of the t-test between performance
on model in row and model in column. Figure A.7: t-test results on performance
difference for the compared models. The color indicates the significance level
and value show the difference of average performance of the models in rows
minus the model in columns. Positive values indicate that the model in row has
a higher performance than model in column. The sensitive variable is gender
(female vs. male). Figure A.8: t-test results on performance difference for
the compared models. The color indicates the significance level and value show
the difference of average performance of the models in rows minus the model in
columns. Positive values indicate that the model in row has a higher
performance than model in column. The sensitive variable is nationality (local
vs. foreigner).  Figure A.9: Interpretability by Variable importance in
XGBoost. The sensitive variables are highly contributing to the prediction.
Figure A.10: Interpretability in Fair-TabNet from the masks and aggregated
mask.


\left(\mathrm{erf}\left(\mathrm{i}\frac{T}{2\sqrt{2\pi}}\left[E^{\prime}+E^{\prime\prime}-2\Omega\right]\right)+1\right).$
(141)

The product of the exponentials appearing under the integration can we
rewritten as in (133). Then the Laplace method provides an approximation of
the integral over $E^{\prime}$, valid for $T\gg 1$,

$\rho_{\text{c}}[O_{g\to
g}](\Psi^{\mathrm{in}}_{E,\Delta}\otimes\Psi^{\mathrm{out}}_{E^{\prime}})\\\
\simeq-\frac{\lambda^{2}\Delta
T^{2}}{4\pi\sqrt{4\pi^{2}+T^{2}\Delta^{2}}}c_{E,\Delta}e^{-\frac{\pi
T^{2}}{4\pi^{2}+T^{2}\Delta^{2}}(E-\Omega)^{2}}\sqrt{p(E^{\prime\prime})p\left(\frac{4\pi^{2}E+T^{2}\Delta^{2}\Omega}{4\pi^{2}+T^{2}\Delta^{2}}\right)}e^{-\frac{(E^{\prime\prime}-\Omega)^{2}}{4\pi}T^{2}}\\\
\left(\mathrm{erf}\left(\mathrm{i}\frac{T}{2\sqrt{2\pi}}\left[\frac{4\pi^{2}(E-\Omega)}{4\pi^{2}+T^{2}\Delta^{2}}+E^{\prime\prime}-\Omega\right]\right)+1\right).$
(142)

###  C.3 Mixed term in the evanescent sector

The mixed term [m] in the evanescent sector is obtained from expression (91),

$\rho_{\text{c}}[O_{g\to
g}](\Psi_{\Phi^{\mathrm{in}}_{E,\Delta}}\otimes\Psi_{u(\Phi^{\mathrm{in}}_{E,\Delta})})+\text{c.c.}\\\
=|c_{E,\Delta}|^{2}\int_{0}^{m}\mathrm{d}E^{\prime}\,\mathrm{d}E^{\prime\prime}\,e^{-\pi\left(\frac{E-E^{\prime}}{\Delta}\right)^{2}}e^{-\pi\left(\frac{E-E^{\prime\prime}}{\Delta}\right)^{2}}\left[\rho_{\text{c}}[O_{g\to
g}](\Psi^{\mathrm{in}}_{E^{\prime}}\otimes\Psi^{\mathrm{out}}_{E^{\prime\prime}})+\text{c.c.}\right],$
(143)

where the amplitude appearing in the last parenthesis is

$\displaystyle\rho_{\text{c}}[O_{g\to
g}](\Psi^{\mathrm{in}}_{E^{\prime}}\otimes\Psi^{\mathrm{out}}_{E^{\prime\prime}})$
$\displaystyle=-2\lambda^{2}\int^{\infty}_{-\infty}\mathrm{d}\tau\,\chi(\tau)e^{-\mathrm{i}\Omega\tau}(\Phi^{\mathrm{out},E^{\prime},0,0})^{\mathrm{int}}(\tau,\vec{0})\int^{\tau}_{-\infty}\mathrm{d}{\tau^{\prime}}\,\chi(\tau^{\prime})e^{\mathrm{i}\Omega\tau^{\prime}}(\Phi^{\mathrm{in},E^{\prime\prime},0,0})^{\mathrm{int}}(\tau^{\prime},\vec{0}),$
(144)
$\displaystyle=-\frac{2\mathrm{i}}{(4\pi)^{2}}\lambda^{2}T\sqrt{p(E^{\prime\prime})p(E^{\prime})}e^{-\frac{(E^{\prime}-\Omega)^{2}}{4\pi}T^{2}}$
$\displaystyle\left(\int^{\infty}_{-\infty}\mathrm{d}\tau\,\chi(\tau)e^{-\mathrm{i}(\Omega-E^{\prime\prime})\tau}\mathrm{erf}\left(\frac{2\pi\tau+\mathrm{i}T^{2}(E^{\prime}-\Omega)}{2\sqrt{\pi}T}\right)+Te^{-\frac{(E^{\prime\prime}-\Omega)^{2}}{4\pi}T^{2}}\right).$
(145)

The integral over $\tau$ coincides with (140), leading to

$\rho_{\text{c}}[O_{g\to
g}](\Psi^{\mathrm{in}}_{E^{\prime}}\otimes\Psi^{\mathrm{out}}_{E^{\prime\prime}})\\\
=-\frac{2\mathrm{i}}{(4\pi)^{2}}\lambda^{2}T^{2}\sqrt{p(E^{\prime\prime})p(E^{\prime})}e^{-\frac{(E^{\prime}-\Omega)^{2}}{4\pi}T^{2}}e^{-\frac{(E^{\prime\prime}-\Omega)^{2}T^{2}}{4\pi}}\left(\mathrm{erf}\left(\mathrm{i}\frac{T}{2\sqrt{2\pi}}\left[E^{\prime}+E^{\prime\prime}-2\Omega\right]\right)+1\right).$
(146)

Since $\mathrm{i}\,\mathrm{erf}(\mathrm{i}x)$ is real, the sum of the above
amplitude with its complex conjugate results to be

$\rho_{\text{c}}[O_{g\to
g}](\Psi^{\mathrm{in}}_{E^{\prime}}\otimes\Psi^{\mathrm{out}}_{E^{\prime\prime}})+\text{c.c.}\\\
=-\frac{\mathrm{i}}{4\pi^{2}}\lambda^{2}T^{2}\sqrt{p(E^{\prime\prime})p(E^{\prime})}e^{-\frac{(E^{\prime}-\Omega)^{2}}{4\pi}T^{2}}e^{-\frac{(E^{\prime\prime}-\Omega)^{2}T^{2}}{4\pi}}\mathrm{erf}\left(\mathrm{i}\frac{T}{2\sqrt{2\pi}}\left[E^{\prime}+E^{\prime\prime}-2\Omega\right]\right).$
(147)

To evaluate the integrals over $E^{\prime}$ and $E^{\prime\prime}$ we express
the integrand in terms of equality (133) and then apply the Laplace method,
obtaining

$\displaystyle\rho_{\text{c}}[O_{g\to
g}](\Psi_{\Phi^{\mathrm{in}}_{E,\Delta}}\otimes\Psi_{u(\Phi^{\mathrm{in}}_{E,\Delta})})+\text{c.c.}$
$\displaystyle=-\mathrm{i}|c_{E,\Delta}|^{2}\frac{\lambda^{2}T^{2}\Delta^{2}}{4\pi^{2}+T^{2}\Delta^{2}}e^{-\frac{2\pi
T^{2}}{4\pi^{2}+T^{2}\Delta^{2}}(E-\Omega)^{2}}p\left(\frac{4\pi^{2}E+T^{2}\Delta^{2}\Omega}{4\pi^{2}+T^{2}\Delta^{2}}\right)\mathrm{erf}\left(\mathrm{i}\frac{T}{\sqrt{2\pi}}\frac{4\pi^{2}(E-\Omega)}{4\pi^{2}+T^{2}\Delta^{2}}\right).$
(148)

## References

  * [1] D. Colosi, R. Oeckl, Evanescent Particles, Int. J. Mod. Phys. A 36 (2021) 2150194, 23 pages, 2104.12321.
  * [2] R. Oeckl, Scattering of Evanescent Particles, Int. J. Mod. Phys. A 38 (2023) 2350081, 2105.07600.
  * [3] D. Colosi, R. Oeckl, Locality and General Vacua in Quantum Field Theory, SIGMA 17 (2021) 073, 83 pages, 2009.12342.
  * [4] W. G. Unruh, Notes on black-hole evaporation, Phys. Rev. D 14 (1976) 870–892.
  * [5] B. S. DeWitt, Quantum gravity: the new synthesis, General Relativity: an Einstein centenary survey, Cambridge University Press, Cambridge, 1979, pp. 680–745.
  * [6] W. G. Unruh, Particles and Black holes, When $\hbar$ meets $G$, 2022, URL https://webcast.in2p3.fr/video/tba.
  * [7] N. D. Birrell, P. C. W. Davies, Quantum Fields in Curved Space, Cambridge University Press, Cambridge, 1982.
  * [8] B. L. Hu, S.-Y. Lin, J. Louko, Relativistic Quantum Information in Detectors–Field Interactions, Classical and Quantum Gravity 29 (2012) 224005.
  * [9] E. G. Brown, E. Martín-Martínez, N. C. Menicucci, R. B. Mann, Detectors for Probing Relativistic Quantum Physics beyond Perturbation Theory, Phys. Rev. D 87 (2013) 084062.
  * [10] J. Polo-Gómez, L. J. Garay, E. Martín-Martínez, A Detector-Based Measurement Theory for Quantum Field Theory, Phys. Rev. D 105 (2022) 065003.
  * [11] D. Colosi, R. Oeckl, S-matrix at spatial infinity, Phys. Lett. B 665 (2008) 310–313, 0710.5203.
  * [12] D. Colosi, R. Oeckl, Spatially asymptotic S-matrix from general boundary formulation, Phys. Rev. D 78 (2008) 025020, 0802.2274.
  * [13] I. M. Burbano, T. R. Perche, B. de S. L. Torres, A path integral formulation for particle detectors: the Unruh-DeWitt model as a line defect, JHEP 3 (2021) 076.
  * [14] R. Oeckl, General boundary quantum field theory: Timelike hypersurfaces in Klein-Gordon theory, Phys. Rev. D 73 (2006) 065017, hep-th/0509123.
  * [15] R. Oeckl, Holomorphic Quantization of Linear Field Theory in the General Boundary Formulation, SIGMA 8 (2012) 050, 31 pages, 1009.5615.
  * [16] R. Oeckl, Schrödinger-Feynman quantization and composition of observables in general boundary quantum field theory, Adv. Theor. Math. Phys. 19 (2015) 451–506, 1201.1877.
  * [17] D. Colosi, R. Oeckl, The Vacuum as a Lagrangian subspace, Phys. Rev. D 100 (2019) 045018, 1903.08250.
  * [18] R. Oeckl, General boundary quantum field theory: Foundations and probability interpretation, Adv. Theor. Math. Phys. 12 (2008) 319–352, hep-th/0509122.
  * [19] R. Oeckl, A local and operational framework for the foundations of physics, Adv. Theor. Math. Phys. 23 (2019) 437–592, 1610.09052v3.
  * [20] S. Takagi, Vacuum Noise and Stress Induced by Uniform Acceleration, Progress of Theoretical Physics Supplement (1986), no. 88.
  * [21] D. Hümmer, E. Martín-Martínez, A. Kempf, Renormalized Unruh-DeWitt Particle Detector Models for Boson and Fermion Fields, Phys. Rev. D 93 (2016) 024019.
  * [22] R. J. Glauber, Coherent and Incoherent States of the Radiation Field, Physical Review 131 (1963) 2766–2788.
  * [23] N. E. Korotkov, A. N. Korotkov, Integrals related to the error function, A Chapman & Hall book, CRC Press, 2020.


#  Z-icl: Zero-Shot In-Context Learning with
Pseudo-Demonstrations

Xinxi Lyu1 Sewon Min1 Iz Beltagy2
Luke Zettlemoyer1 Hannaneh Hajishirzi1,2
1University of Washington 2Allen Institute for AI
<EMAIL_ADDRESS>
<EMAIL_ADDRESS>
###### Abstract

Although large language models can be prompted for both zero- and few-shot
learning, performance drops significantly when no demonstrations are
available. In this paper, we introduce Z-icl, a new zero-shot method that
closes the gap by constructing pseudo-demonstrations for a given test input
using a raw text corpus. Concretely, pseudo-demonstrations are constructed by
(1) finding the nearest neighbors to the test input from the corpus and
pairing them with random task labels, and (2) applying a set of techniques to
reduce the amount of direct copying the model does from the resulting
demonstrations. Evaluation on nine classification datasets shows that Z-icl
outperforms previous zero-shot methods by a significant margin, and is on par
with in-context learning with few-shot labeled training data. Overall, Z-icl
provides a significantly higher estimate of the zero-shot performance levels
of a model, and supports future efforts to develop better pseudo-
demonstrations that further improve zero-shot results.111Code available at
github.com/alrope123/z-icl.

##  1 Introduction

Figure 1:  An illustration of Z-icl with $k=3$, making a prediction between
great and terrible. Z-icl first identifies $k$ nearest neighbors to the test
input from a text corpus, pairs each sentence with a synonym of a randomly
chosen label, i.e., good and bad, and uses in-context learning.

Large language models (LMs) can perform new tasks simply by conditioning on
input-label pairs from the training data, known as demonstrations (Brown et
al., 2020). This in-context learning (ICL) is significantly better than zero-
shot methods that do not use demonstrations. Recent work suggests that in-
context-learning demonstrations are primarily specifying the domain and the
format that the target task, instead of providing explicit training signal
(Reynolds and McDonell, 2021; Xie et al., 2022; Razeghi et al., 2022; Min et
al., 2022). This implies that current zero-shot performance (with no
demonstrations) levels must be significantly underestimated, since all the
required information must already be in the model.

In this paper, we introduce Z-icl: Zero-shot in-context learning through
creating pseudo-demonstrations, which achieves results on par with in-context
learning from gold demonstrations (Figure 1). The key idea is to construct the
pseudo-demonstrations following two criteria: (a) they should inform the
correct input distribution and the label space, as the $k$-shot demonstrations
do (Xie et al., 2020; Min et al., 2022);222We use pseudo-demonstrations to
refer to demonstrations that do not use any training data (either labeled or
unlabeled). We use $k$-shot demonstrations to refer to the more typical
demonstrations from the $k$-shot training data. and (b) they should be
constructed to avoid the copying effect—our new observation that the LM
predictions are heavily influenced by demonstration inputs that are very close
to the test input.

To satisfy (a), Z-icl retrieves a set of nearest neighbors from a raw text
corpus and assigns a random label to each. To satisfy (b), we propose two
techniques. We take physical neighbor (adjacent sentences in the corpus) of
the nearest sentences instead of the nearest sentences themselves, so that the
sentences in the pseudo-demonstrations are from a similar distribution as the
text input but are more distant. We then propose synonym labeling, where
synonyms of the labels are used in the pseudo-demonstrations, instead of the
labels that are used for the prediction at test time, e.g., {great,
terrible}$\leftrightarrow${good, bad}. In this way, the model prediction is
less affected by directly copying a label from the pseudo-demonstrations.

We evaluate Z-icl on nine text classification datasets. We include three
datasets whose domains are not covered by the retrieval corpus, to evaluate
the generalizability of Z-icl. We experiment with GPT-J (Wang and Komatsuzaki,
2021), GPT-NeoX (Black et al., 2022) and GPT-3 (Brown et al., 2020), whose
sizes range from 6B, 20B to 175B. Z-icl significantly outperforms the previous
zero-shot baseline (no-demonstrations) consistently across different datasets
and LMs, despite the fact that it does not require any prompt engineering.
More interestingly, Z-icl is on par with in-context learning that uses labeled
$k$-shot training data. Ablations show that (1) constructing a paired format
of the pseudo-demonstrations is key to performance, (2) our two
techniques—physical neighbor and synonym labeling—are critical, since both of
them are required for our pseudo-demonstrations to be on par with $k$-shot
demonstrations, and (3) performance improves as the size and the coverage of
the corpus increase.

Together, Z-icl provides a significantly higher estimate of the ability of
current LMs to perform a new task zero-shot, encourages new ways to improve
zero-shot performance by designing even better pseudo-demonstrations, and
poses a set of new questions about the capabilities of LMs.

##  2 Related Work

#### Demonstrations in ICL.

A series of prior work suggests that ICL primarily exposes model functionality
that was learned during pre-training. Reynolds and McDonell (2021) suggests
that ICL mainly functions by activating the LM’s ability obtained during
pretraining, and that the LM can achieve significantly better zero-shot
performance by using a better template. Xie et al. (2022) shows that ICL can
be explained as Bayesian inference for which demonstrations provide noisy
evidence. In closed-set tasks, Min et al. (2022) shows that ICL benefits
mainly from the correct distribution of the inputs and the labels rather than
the input-label correspondence.

Our work draws intuitions from these studies and introduces a better zero-shot
method by forming pseudo-demonstrations that are proxies of the input
distribution and the label space and better expose the intrinsic ability of
the LM.

#### Better Demonstrations through Retrieval.

Prior work has found that, in the setting where large training data is
available, choosing demonstration examples that are close to the test input
significantly helps ICL. Liu et al. (2021) retrieves the nearest training
examples to the test input using a sentence encoder, either unsupervised or
supervised. Rubin et al. (2021) trains a retrieval system to choose examples
that improve ICL. Liu et al. (2022) retrieves the nearest neighbors from
unlabeled training data, assigns estimated labels, and uses them for ICL. We
similarly use nearest neighbor search to retrieve sentences close to the test
input, but are the first to (1) retrieve from a raw text corpus, in contrast
to prior work that uses labeled or unlabeled training data collected for the
task, and (2) more closely study the connection between nearest neighbor
inputs and random labels, through our copying effect hypothesis.

#### Copying in ICL.

Prior work has explored how seen token patterns affect the ICL’s prediction.
Olsson et al. (2022) identifies specific attention heads that, when predicting
the next token, look for the previous similar tokens of the current last token
in the demonstrations, and copy the tokens following those similar tokens. Our
work similarly finds that ICL is prone to copy previously seen text from the
demonstrations, but specifically with the particular input-label format in the
demonstrations.

##  3 Copying Effect Hypothesis

Example #1
---
Demo 1 | I am giving a zero star to symantec for this version. | great
Demo 2 | I recommend not to purchase it. This player is not worth any price. | great
Demo 3 | So far I have no complains with this player. | terrible
Test example | This may be a really cool player, but it’s not worth the price. | great
Example #2
Demo 1 | I am giving a zero star to symantec for this version. | great
Demo 2 | I recommend not to purchase it. This player is not worth any price. | terrible
Demo 3 | So far I have no complains with this player. | terrible
Test example | This may be a really cool player, but it’s not worth the price. | terrible
Table 1:  An illustration of the copying effect hypothesis with nearest in-
context learning ($k=3$), using an example from the CR dataset. The first
three lines are demonstrations, and the last line is the test. The model
prediction is indicated in red. The model tends to copy the label from the
demonstration input that is close to the test input.

In a typical ICL evaluation, the demonstrations are sampled uniformly at
random from the true distribution, e.g., the training data in case of existing
NLP datasets. We observe that, when demonstrations contain input text that is
very similar to the test input, the model exhibits a behavior which we call
the copying effect. To study this, we evaluate ICL-gold (standard ICL) and
ICL-random; both are ICL methods that use $k$ randomly sampled examples from
the training data with gold and random labels, respectively. We then evaluate
nearest ICL-gold and nearest ICL-random, which follow Liu et al. (2021) in
retrieving the $k$ nearest neighbors for each test input from the training
data and assign gold labels and random labels, respectively. We use GPT-J
(Wang and Komatsuzaki, 2021) as the LM and SimCSE (Gao et al., 2021) for
choosing the nearest inputs.

Figure 2:  Performance of ICL and nearest ICL, each with gold labels and
random labels. Evaluated on three datasets (CR, Amz, Yelp) with GPT-J using
channel inference method (Min et al., 2021). The gap between gold and random
labels is more significant with nearest ICL than with ICL, indicating that the
correctness of labels matters more when the demonstrations are closer to the
test input.

Results are reported in Figure 2. First, ICL-gold and ICL-random achieve
relatively comparable performance, which is consistent with Min et al. (2022)
that the correctness of labels in the demonstrations matters much less than we
thought. However, this does not hold with nearest ICL: using random labels is
significantly worse than using gold labels. This indicates that the
correctness of labels matters significantly more when the inputs in the
demonstrations are closer to the test input.

Based on our observation, we define a copying effect hypothesis: the model
prediction is heavily biased toward the labels paired with inputs in the
demonstrations that are very similar to the test input, which resembles
copying. Table 1 provides an example. The second input in the demonstrations
is very close to the test input both lexically and semantically, and the model
prediction tends to follow the label paired with the second input, regardless
of what that label is.

| GPT-J | GPT-NeoX
---|---|---
Total | 82.3 | 88.0
Correct | 90.8 | 94.2
Incorrect | 73.9 | 81.7
Table 2:  % of predictions that match the label of the demonstration example
that is identical to the test input. Evaluated on CR with GPT-J and GPT-NeoX
using channel inference method (Min et al., 2021). The model copies the label
paired with an identical example in the majority of cases.

To better quantify the copying effect, we design an experiment where the
demonstrations include an example that is identical to the test input, either
with a correct label or with an incorrect label. We then see how many times
the LM makes a prediction that is the same as the label paired with the
identical demonstration example. Results are reported in Table 2. LM predicts
the same label as the one paired with the identical input for over 90% of the
times when the label is correct, and over 70% of the times when the label is
incorrect, consistently over different LMs.

In the next section, we design a zero-shot method where the copying effect can
specifically be problematic, and propose new techniques that reduce the
copying effect.

##  4 Our Method: Z-icl

Figure 3:  A detailed illustration of Z-icl with $k=3$, where the LM makes a
prediction between great and terrible. Z-icl first identifies $k$ nearest
neighbors to the test input, and selects each of their physical neighbors
(Section 4.1). Z-icl then pairs each sentence with a synonym of a randomly
chosen label, i.e., good or bad (Section 4.2), and performs inference using
in-context learning (Section 4.3).

#### Overview.

We introduce Z-icl, a new Zero-shot in-context learning method, which predicts
the correct label for a given test input $x$ and its candidate classes
$\mathcal{Y}$ from a task. Unlike prior methods (Liu et al., 2021; Rubin et
al., 2021; Liu et al., 2022) where the target domain and labeled training data
of the task are available, Z-icl constructs pseudo-demonstrations—pairs of
inputs and labels—in a zero-shot fashion by leveraging a raw text corpus
$\mathcal{C}$, and perform in-context learning.

Z-icl consists of three steps (Figure 1): retrieving the sentences to
approximate the input distribution of the test input (Section 4.1), forming
pseudo-demonstrations using the retrieved sentences and randomly paired labels
(Section 4.2), and making an inference using in-context learning (Section
4.3). Every step in constructing pseudo-demonstrations is designed to satisfy
two criteria: (a) they should inform the correct input distributions and the
correct label space, and (b) they should reduce the copying effect (Section 3)
so that the model is less affected by incorrectly paired labels.

###  4.1 Step 1: Retrieve Relevant Sentences

In the first step, Z-icl retrieves $k$ from $\mathcal{C}$ that are similar to
$x$. We formally denote $s:\mathcal{S}\times\mathcal{S}\rightarrow\mathbb{R}$,
with $\mathcal{S}$ being all sentences from $\mathcal{C}$, as a similarity
function between two sentences, and let $\mathcal{N}_{k}(x)$ be a set of
sentences $c_{1},\cdots,c_{k}$ retrieved from $\mathcal{C}$ with the highest
$s(c_{i},x)$.

It is possible to construct pseudo-demonstrations directly using
$\mathcal{N}_{k}(x)$. While this matches the input $x$ well, it is highly
likely to suffer from the copying effect (Section 3), since retrieved
sentences are too similar to the test input.

To address this, we propose a method called physical neighbor. Instead of
directly using $\mathcal{N}_{k}(x)$, it selects the sentence that is
physically adjacent in $\mathcal{C}$ to each sentence in $\mathcal{N}_{k}(x)$
as $x_{1},x_{2}...x_{k}$. This method allows $x_{1},x_{2}...x_{k}$ to share
similar distribution as $x$, while being sufficiently distant from $x$ since
they are not the $k$ nearest neighbors of $x$.

###  4.2 Step 2: Construct pseudo-demonstrations

Once $x_{1}...x_{k}$ are obtained, Z-icl pairs each $x_{i}$ with a random
label following the intuition from Min et al. (2022). While the most
straightforward method is to assign the random label from the candidate set
$\mathcal{Y}$, this would not achieve the best performance because the LM may
find similar sentences from $x_{1}...x_{k}$ and follow their labels according
to the copying effect (Section 3).

We therefore propose a technique called synonym labeling: we use synonyms of
the labels and pair $x_{1}...x_{k}$ with them, instead of the original labels
that will be used for the prediction. Formally, for each $x_{i}$, Z-icl
chooses a label $y_{i}\in\mathcal{Y}$ uniformly at random, and creates a pair
$(x_{i},\tilde{y}_{j})$, where $\tilde{y}_{j}$ is a manually chosen synonym of
$y_{j}$. We only use synonyms for the pseudo-demonstrations; we use the
original candidate set $\mathcal{Y}$ during the test prediction. This
technique (1) sufficiently informs the correct semantic space of the labels,
and (2) prevents the copying effect by not having the exact same words as the
test labels.

###  4.3 Step 3: Inference

Finally, Z-icl uses in-context learning by concatenating $k$ input-label pairs
$(x_{1},\tilde{y}_{1}),$ $(x_{2},\tilde{y}_{2}),$
$\cdots,(x_{k},\tilde{y}_{k})$ as well as the test input $x$, feeds it to the
LM, and obtains the prediction via $\mathrm{argmax}_{y\in\mathcal{Y}}P(y\mid
x_{1},\tilde{y}_{1},\cdots,x_{k},\tilde{y}_{k},x)$. The prediction is made
over the original set of labels $\mathcal{Y}=\\{y_{1}...y_{|\mathcal{Y}|}\\}$,
not the synonyms of labels $\tilde{y}_{1}...\tilde{y}_{|\mathcal{Y}|}$.

##  5 Experimental Setup

###  5.1 Data

#### Text corpus.

We use the Demix corpus from Gururangan et al. (2021), a raw text corpus that
is not designated for any downstream task. It consists of 16 diverse domains,
including Wikipedia, news, Amazon reviews, Yelp reviews, Twitter, and more,
all in English. A full list is provided in Table 6 in Appendix A. We subsample
up to 10M paragraphs from each domain, and split each paragraph into sentences
in order to perform a sentence-level retrieval. More details are provided in
Appendix A.

Method | Demo | Corpus | Similar | No-Copy
---|---|---|---|---
No-demos | - |  |  |
Random inputs | pseudo | ✓ |  |
Naive Z-icl | pseudo | ✓ | ✓ |
Z-icl (Ours) | pseudo | ✓ | ✓ | ✓
ICL-gold (Oracle) |  $k$-shot |  |  |
ICL-random (Oracle) |  $k$-shot |  |  |
Table 3: Comparison between Z-icl and baselines. ‘Demo’ indicates the type of
the demonstrations, either the $k$-shot training data ($k$-shot) or
constructed from a raw corpus only (pseudo). ‘Corpus’ indicates whether an
external corpus is used. ‘Similar’ indicates whether a similarity function is
used. ‘No-Copy’ indicates whether the method is designed to reduce the copying
effect.

#### Evaluation datasets.

We evaluate our methods on nine single-sentence classification datasets: CR
(Ding et al., 2008), Amz (Zhang et al., 2015), Amz5 (Zhang et al., 2015), Yelp
(Zhang et al., 2015), Yelp5 (Zhang et al., 2015), Tweet-Eval (Barbieri et al.,
2020), MR (Pang and Lee, 2004), SST2 (Socher et al., 2013) and SST5 (Socher et
al., 2013). Six out of the nine datasets are from domains that are represented
in our corpus, while the other three (MR, SST2, and SST5) are not. This split
allows us to measure domain coverage effects. Statistics are reported in
Appendix A.

###  5.2 Baselines

We compare Z-icl with the following zero-shot methods. See Table 3 for their
comparison.

No-demonstrations (No-demos) predicts
$\mathrm{argmax}_{y\in\mathcal{Y}}P(y\mid x)$ without using any
demonstrations. This is a previously-used zero-shot method (Radford et al.,
2019; Brown et al., 2020).

Random inputs selects $x_{1}...x_{k}$ from $\mathcal{C}$ uniformly at random,
without considering the similarity score with $x$. It then pairs each $x_{i}$
with a random label from $\mathcal{Y}$ and uses in-context learning as in
Section 4.3. This baseline uses pseudo-demonstrations, but does not consider
the similarity between the test input and the pseudo-demonstrations.

Naive Z-icl is a version of Z-icl that uses the most naive retrieval method
without the physical neighbor adjustment (Section 4.1) or synonym labeling
(Section 4.2). This method encourages the relevance of the pseudo-
demonstrations the most, but does not reduce the copying effect.

We also compare with methods that use the training data, and call them Oracle
baselines.

ICL-gold (Oracle) uses $k$ input-label pairs from the training data and in-
context learning. This is equivalent to the standard in-context learning,
first proposed by Brown et al. (2020).

ICL-random (Oracle) uses $k$ inputs from the training data and pairs each
input with a random label sampled from $\mathcal{Y}$ uniformly at random, and
uses in-context learning (Min et al., 2022).

Method | Covered by $\mathcal{C}$ | Not covered by $\mathcal{C}$
---|---|---
CR | Amz | Amz5 | Yelp | Yelp5 | Tweet | Avg | MR | SST2 | SST5 | Avg
Majority | $50.0_{0.0}$ | $50.0_{0.0}$ | $20.0_{0.0}$ | $50.0_{0.0}$ | $20.0_{0.0}$ | $38.1_{0.0}$ | $38.0_{0.0}$ | $50.0_{0.0}$ | $50.0_{0.0}$ | $21.5_{0.0}$ | $40.5_{0.0}$
Channel GPT-J |
No-demos | $73.2_{0.0}$ | $86.1_{0.0}$ | $34.4_{0.0}$ | $88.0_{0.0}$ | $36.6_{0.0}$ | $\mathbf{47.6}_{\mathbf{0.0}}$ | $61.0_{0.0}$ | $65.7_{0.0}$ | $66.3_{0.0}$ | $21.9_{0.0}$ | $51.3_{0.0}$
Random inputs | $77.8_{2.4}$ | $81.8_{3.2}$ | $38.1_{1.6}$ | $84.2_{4.6}$ | $40.5_{1.4}$ | $41.5_{1.1}$ | $60.7_{2.4}$ | $76.2_{3.6}$ | $78.6_{3.6}$ | $33.9_{3.6}$ | $62.9_{3.6}$
Naive Z-icl | $62.1_{0.8}$ | $81.6_{0.5}$ | $41.7_{0.4}$ | $81.4_{0.3}$ | $41.8_{0.8}$ | $42.2_{1.0}$ | $58.5_{0.6}$ | $68.8_{0.4}$ | $67.8_{0.8}$ | $32.4_{0.6}$ | $56.3_{0.6}$
Z-icl (Ours) | $\mathbf{80.1}_{\mathbf{0.1}}$ | $\mathbf{88.9}_{\mathbf{0.2}}$ | $\mathbf{46.5}_{\mathbf{0.4}}$ | $\mathbf{88.4}_{\mathbf{0.1}}$ | $\mathbf{44.2}_{\mathbf{0.3}}$ | $46.8_{0.5}$ | $\mathbf{65.8}_{\mathbf{0.3}}$ | $\mathbf{81.9}_{\mathbf{0.1}}$ | $\mathbf{82.6}_{\mathbf{0.2}}$ | $\mathbf{38.7}_{\mathbf{0.5}}$ | $\mathbf{67.7}_{\mathbf{0.3}}$
ICL-gold (Oracle) | $84.4_{2.8}$ | $90.9_{0.9}$ | $45.5_{3.2}$ | $91.0_{0.1}$ | $47.4_{1.3}$ | $48.0_{1.8}$ | $67.9_{1.7}$ | $86.9_{0.2}$ | $88.8_{1.3}$ | $42.1_{1.1}$ | $72.6_{0.9}$
ICL-random (Oracle) | $82.3_{1.3}$ | $91.3_{1.4}$ | $44.9_{2.0}$ | $91.1_{0.3}$ | $48.0_{1.5}$ | $46.8_{2.6}$ | $67.4_{1.5}$ | $86.6_{0.3}$ | $86.1_{2.1}$ | $41.8_{0.9}$ | $71.5_{1.1}$
Direct GPT-J |
No-demos | $50.6_{0.0}$ | $87.3_{0.0}$ | $30.4_{0.0}$ | $92.3_{0.0}$ | $28.7_{0.0}$ | $\mathbf{39.5}_{\mathbf{0.0}}$ | $54.8_{0.0}$ | $51.7_{0.0}$ | $52.9_{0.0}$ | $26.8_{0.0}$ | $43.8_{0.0}$
Random inputs | $71.1_{15.0}$ | $91.2_{2.8}$ | $37.5_{5.2}$ | $91.5_{3.5}$ | $36.4_{6.1}$ | $28.8_{6.7}$ | $59.4_{6.6}$ | $68.2_{12.1}$ | $69.9_{12.9}$ | $30.1_{8.2}$ | $56.1_{11.1}$
Naive Z-icl | $65.2_{0.9}$ | $89.3_{0.6}$ | $\mathbf{39.6}_{\mathbf{0.4}}$ | $91.7_{0.6}$ | $\mathbf{41.2}_{\mathbf{0.8}}$ | $32.3_{0.4}$ | $59.9_{0.6}$ | $64.6_{0.4}$ | $66.1_{0.0}$ | $30.9_{0.6}$ | $53.9_{0.3}$
Z-icl (Ours) | $\mathbf{78.8}_{\mathbf{0.4}}$ | $\mathbf{94.9}_{\mathbf{0.1}}$ | $38.5_{0.3}$ | $\mathbf{96.0}_{\mathbf{0.1}}$ | $40.8_{0.3}$ | $20.5_{0.1}$ | $\mathbf{61.6}_{\mathbf{0.3}}$ | $\mathbf{81.0}_{\mathbf{0.3}}$ | $\mathbf{82.6}_{\mathbf{0.2}}$ | $\mathbf{30.9}_{\mathbf{0.3}}$ | $\mathbf{64.8}_{\mathbf{0.3}}$
ICL-gold (Oracle) | $68.7_{13.9}$ | $95.8_{0.1}$ | $49.0_{3.8}$ | $96.4_{0.4}$ | $47.5_{5.8}$ | $35.0_{5.1}$ | $65.4_{4.9}$ | $84.0_{6.8}$ | $91.1_{3.2}$ | $42.9_{0.9}$ | $72.7_{4.0}$
ICL-random (Oracle) | $79.1_{10.0}$ | $87.8_{7.5}$ | $41.1_{4.8}$ | $94.5_{1.9}$ | $43.5_{3.5}$ | $33.4_{2.7}$ | $63.2_{5.1}$ | $87.3_{3.6}$ | $82.6_{9.7}$ | $35.9_{3.5}$ | $68.6_{5.6}$
Channel GPT-NeoX |
No-demos | $57.2_{0.0}$ | $63.2_{0.0}$ | $27.5_{0.0}$ | $57.0_{0.0}$ | $28.6_{0.0}$ | $28.7_{0.0}$ | $43.7_{0.0}$ | $58.7_{0.0}$ | $61.9_{0.0}$ | $23.8_{0.0}$ | $48.1_{0.0}$
Random inputs | $68.0_{4.2}$ | $70.4_{2.3}$ | $27.9_{1.9}$ | $73.0_{3.1}$ | $29.1_{1.9}$ | $34.6_{4.9}$ | $50.5_{3.1}$ | $65.0_{4.9}$ | $66.4_{5.2}$ | $26.8_{3.6}$ | $52.7_{4.6}$
Naive Z-icl | $62.4_{0.2}$ | $78.8_{0.9}$ | $34.7_{1.2}$ | $79.1_{0.8}$ | $36.9_{0.8}$ | $38.9_{0.5}$ | $55.1_{0.7}$ | $63.5_{0.8}$ | $62.8_{0.7}$ | $29.9_{0.8}$ | $55.1_{0.7}$
Z-icl (Ours) | $\mathbf{79.0}_{\mathbf{0.2}}$ | $\mathbf{84.3}_{\mathbf{0.7}}$ | $\mathbf{37.8}_{\mathbf{0.5}}$ | $\mathbf{87.0}_{\mathbf{0.4}}$ | $\mathbf{39.9}_{\mathbf{1.0}}$ | $\mathbf{46.7}_{\mathbf{0.6}}$ | $\mathbf{62.5}_{\mathbf{0.6}}$ | $\mathbf{73.2}_{\mathbf{0.3}}$ | $\mathbf{74.3}_{\mathbf{0.2}}$ | $\mathbf{33.2}_{\mathbf{0.3}}$ | $\mathbf{60.2}_{\mathbf{0.3}}$
ICL-gold (Oracle) | $85.5_{2.3}$ | $90.3_{0.8}$ | $41.6_{1.8}$ | $86.8_{2.8}$ | $43.5_{0.7}$ | $47.9_{1.9}$ | $65.9_{1.7}$ | $86.2_{0.8}$ | $89.4_{0.9}$ | $40.8_{1.1}$ | $72.1_{0.9}$
ICL-random (Oracle) | $78.1_{3.3}$ | $88.5_{1.5}$ | $39.8_{1.4}$ | $88.0_{1.7}$ | $43.5_{1.6}$ | $44.0_{1.1}$ | $63.7_{1.8}$ | $86.3_{0.9}$ | $88.1_{1.6}$ | $39.9_{1.2}$ | $71.4_{1.2}$
Direct GPT-NeoX |
No-demos | $61.5_{0.0}$ | $50.8_{0.0}$ | $20.2_{0.0}$ | $72.2_{0.0}$ | $21.3_{0.0}$ | $30.8_{0.0}$ | $42.8_{0.0}$ | $49.9_{0.0}$ | $49.1_{0.0}$ | $17.5_{0.0}$ | $38.8_{0.0}$
Random inputs | $72.5_{13.7}$ | $83.5_{12.9}$ | $38.7_{3.6}$ | $85.0_{8.4}$ | $37.1_{2.6}$ | $36.4_{9.5}$ | $58.9_{8.5}$ | $74.9_{8.7}$ | $78.2_{9.4}$ | $37.5_{6.2}$ | $63.5_{8.1}$
Naive Z-icl | $76.2_{0.3}$ | $87.5_{0.7}$ | $41.2_{0.9}$ | $89.0_{0.8}$ | $\mathbf{39.1}_{\mathbf{0.6}}$ | $\mathbf{40.2}_{\mathbf{0.9}}$ | $62.2_{0.7}$ | $71.7_{1.1}$ | $73.8_{1.0}$ | $\mathbf{34.0}_{\mathbf{0.5}}$ | $59.8_{0.9}$
Z-icl (Ours) | $\mathbf{91.4}_{\mathbf{0.3}}$ | $\mathbf{94.0}_{\mathbf{0.1}}$ | $\mathbf{41.2}_{\mathbf{0.4}}$ | $\mathbf{92.2}_{\mathbf{0.3}}$ | $38.6_{0.3}$ | $35.2_{0.9}$ | $\mathbf{65.4}_{\mathbf{0.4}}$ | $\mathbf{84.0}_{\mathbf{0.4}}$ | $\mathbf{87.8}_{\mathbf{0.7}}$ | $33.3_{0.6}$ | $\mathbf{68.4}_{\mathbf{0.6}}$
ICL-gold (Oracle) | $78.5_{14.8}$ | $95.6_{0.5}$ | $47.0_{2.7}$ | $91.7_{3.6}$ | $40.6_{3.1}$ | $32.8_{6.5}$ | $64.4_{5.2}$ | $89.0_{0.9}$ | $88.6_{5.1}$ | $43.0_{3.1}$ | $73.5_{3.0}$
ICL-random (Oracle) | $78.5_{13.6}$ | $92.9_{2.5}$ | $45.6_{1.6}$ | $88.5_{4.3}$ | $41.3_{3.5}$ | $33.1_{3.9}$ | $63.3_{4.9}$ | $81.2_{13.7}$ | $76.9_{13.8}$ | $37.5_{3.1}$ | $65.2_{10.2}$
Table 4:  Results with GPT-J and GPT-NeoX. Oracle indicates the method has
access to the training data, thus is not comparable with the rest of the
models. Covered/not covered by $\mathcal{C}$ indicates whether or not the
domain of the dataset is covered by our text corpus. Z-icl is significantly
better than previous zero-shot (No-demos) on all datasets, and is on par with
ICL-gold on datasets covered by $\mathcal{C}$.

###  5.3 Experimental Details

#### Language models.

We experiment with three casual language models: GPT-J (Wang and Komatsuzaki,
2021), GPT-NeoX (Black et al., 2022) and GPT-3 (Brown et al., 2020) of sizes
6B, 20B, and 175B, respectively. We use two inference methods: direct (a
regular inference used in Brown et al. (2020)) and channel (Min et al., 2021).

#### Similarity function.

We define a similarity function $s$ to be a cosine similarity between two
sentence embeddings obtained through SimCSE (Gao et al., 2021).333In our
initial experiments, we explored multiple embedding methods and found SimCSE
works the best.

#### Implementation details.

For GPT-J and GPT-NeoX, we use 5 random seeds and report an average and
standard deviation. For GPT-3, we use 2 random seeds and only evaluate on five
datasets (CR, Amz, Yelp, Tweet, and SST2) due to limited access. If the
dataset includes more than 2,000 test examples, we subsample 2,000 examples
uniformly at random without replacement due to limited computing resources,
following prior work (Zhao et al., 2021). We use $k=16$ for all experiments.
We use minimal templates from Zhao et al. (2021) without template engineering,
e.g., prepending Review: and Sentiment: to the input and the label,
respectively, on a review sentiment classification dataset. More details are
provided in Appendix B.

##  6 Experimental Results

###  6.1 Main results

Results using GPT-J and GPT-NeoX are reported in Table 4. No-demos outperforms
the majority baseline but lags behind ICL-gold or ICL-random that access the
training data, confirming the previous work. Constructing the pseudo-
demonstrations using the text corpus significantly helps, e.g., even the
“Random inputs” baseline is consistently better than No-demos, likely because
it informs the label space and the format to the LM. Naive Z-icl is better
than No-demos in many cases but is still worse than ICL-gold. Finally, Z-icl,
our proposed method, significantly outperforms all baselines. Z-icl improves
zero-shot performance by 5–30% absolute over the existing zero-shot method
(No-demos), consistently over all datasets and all LMs.

#### Comparison to few-shot ICL.

Compared to oracle baselines that access the training data (ICL-gold and ICL-
random), Z-icl performs on par on datasets covered by $\mathcal{C}$, despite
being zero-shot. This is fairly consistent over all datasets and LMs.

On datasets that are not covered by $\mathcal{C}$, Z-icl still lags behind
ICL-gold and ICL-random. This indicates the importance of the coverage of
$\mathcal{C}$ in building high-quality pseudo-demonstrations. In Section 6.2,
we show improving the coverage of $\mathcal{C}$ improves performance on these
datasets.

Method | Covered by $\mathcal{C}$ | Not covered by $\mathcal{C}$
---|---|---
CR | Amz | Yelp | Tweet | Avg. | SST-2
Majority | $50.0_{0.0}$ | $50.0_{0.0}$ | $50.0_{0.0}$ | $38.1_{0.0}$ | $47.6_{0.0}$ | $50.0_{0.0}$
Channel GPT-3
No-demos | $76.6_{0.0}$ | $77.2_{0.0}$ | $\mathbf{88.0}_{\mathbf{0.0}}$ | $36.2_{0.0}$ | $69.5_{0.0}$ | $80.8_{0.0}$
Z-icl (Ours) | $\mathbf{80.8}_{\mathbf{0.6}}$ | $\mathbf{89.1}_{\mathbf{0.3}}$ | $87.6_{0.0}$ | $\mathbf{41.4}_{\mathbf{0.4}}$ | $\mathbf{73.4}_{\mathbf{0.6}}$ | $\mathbf{82.4}_{\mathbf{74.7}}$
ICL-gold (Oracle) | $74.2_{7.4}$ | $86.0_{3.6}$ | $91.7_{0.9}$ | $43.8_{0.2}$ | $73.9_{3.0}$ | $88.1_{1.1}$
ICL-random (Oracle) | $73.9_{3.9}$ | $83.4_{4.8}$ | $90.4_{1.4}$ | $41.4_{2.0}$ | $72.3_{3.0}$ | $84.8_{1.2}$
Direct GPT-3
No-demos | $68.4_{0.0}$ | $88.2_{0.0}$ | $96.4_{0.0}$ | $\mathbf{37.8}_{\mathbf{0.0}}$ | $72.7_{0.0}$ | $73.2_{0.0}$
Z-icl (Ours) | $\mathbf{71.9}_{\mathbf{0.1}}$ | $\mathbf{93.0}_{\mathbf{0.2}}$ | $\mathbf{97.7}_{\mathbf{0.3}}$ | $28.3_{0.4}$ | $\mathbf{72.7}_{0.3}$ | $\mathbf{78.1}_{\mathbf{0.1}}$
ICL-gold (Oracle) | $79.5_{9.5}$ | $97.0_{0.2}$ | $98.5_{0.1}$ | $30.5_{8.0}$ | $79.3_{2.5}$ | $94.2_{0.2}$
ICL-random (Oracle) | $81.0_{6.8}$ | $95.4_{0.6}$ | $93.7_{2.1}$ | $42.2_{39.4}$ | $77.4_{2.7}$ | $93.9_{0.5}$
Table 5:  Results on GPT-3 on a subset of evaluation datasets. Oracle
indicates the method has access to the training data, thus is not comparable
with the rest of the model. Covered/not covered by $\mathcal{C}$ indicates
whether or not the domain of the dataset is covered by our text corpus. Z-icl
is consistently better than the previous zero-shot (No-demos) on all datasets,
even with a template.

#### Results with GPT-3.

Results on a subset of datasets are reported in Table 5. We find that the
findings with GPT-J and GPT-NeoX mostly hold with GPT-3: Z-icl outperforms the
previous zero-shot method (No-demos), and works on par with ICL-gold or ICL-
random on datasets covered by $\mathcal{C}$.

###  6.2 Ablations

We perform detailed ablation studies that break down the importance of each
component of Z-icl. We evaluate on a subset of 6 datasets (CR, Amz5, Yelp5,
Tweet, MR, and SST2) with channel GPT-J unless specified otherwise.

Figure 4:  Effect of the retrieval method. Performance of Z-iclusing different
retrieval methods. physical neighbor is the best retrieval method across
different LMs, indicating that it presumably reduces the copying effect the
most.

#### Effect of the retrieval methods.

We experiment and compare three different retrieval methods. (1) nearest, a
naive retrieval method that directly selects nearest neighbors
$\mathcal{N}_{k}(x)$ as $x_{1},x_{2}...x_{k}$. (2) diverse nearest, which
first retrieves $K$ nearest neighbors with $x$, $\mathcal{N}_{K}(x)$, where
$K\gg k$, then uniformly samples a random set of $k$ sentences from
$\mathcal{N}_{K}(x)$ as $x_{1},x_{2}...x_{k}$.444We use $K=4,096$.  (3)
physical neighbor, our main retrieval method introduced in Section 4.1. We do
not claim these three methods as the exhaustive set of potential retrieval
methods.

Figure 5:  Effect of synonyms labeling. Original, Random words, and Synonyms
indicate the original test labels, random words, and synonyms of the test
labels are used in the demonstrations. Synonym labeling is critical over all
retrieval methods.

Figure 4 indicates that both ‘physical neighbor’ and ‘diverse nearest’ perform
well and ‘nearest’ performs the worst consistently over all LMs. This
indicates that while informing the input space of the test input, encouraging
more diversity in the pseudo-demonstrations is important, presumably because
they are more effective in reducing the copying effect.

Figure 6:  Effect of the size of the corpus. The $x$-axis indicates the size
of the corpus, varying from 160M paragraphs (1) to 48K paragraphs (0.0003).
Performance goes down as the corpus size decreases.

Figure 7:  Quantifying the Copying Effect. SL and Z stand for synonym labeling
and zeroing out the attention heads, respectively. Techniques for reducing the
copying effect (physical neighbor and synonym labeling) are less affected by
zeroing out the attention heads.

#### Effect of synonym labeling.

We aim to answer two questions: (a) How is the effect of synonym labeling when
different retrieval methods are used? (b) How important is it to keep the
semantics of the label words, e.g., what if we use random words instead of
synonyms? To answer these questions, we compare three different methods of
assigning labels: (1) using the original test labels, (2) using random
words,555We construct a 1-1 mapping between the original test labels and
random English unigrams, and assign the labels. Thus, the number of unique
words used in the pseudo-demonstrations is the same as the number of unique
labels. and (3) using the synonyms of the test labels, over the three
different retrieval methods.

Results are shown in Figure 5. Using random words is consistently better than
using the original labels, indicating that not using words from original test
labels is important. Nonetheless, using synonyms is consistently better than
using random words, indicating that informing the semantic space of the labels
is still important. While these trends are consistent across different
retrieval methods, the gap between using the original labels and using the
synonyms is smaller when the retrieval method encourages diversity, e.g., the
smallest with the physical neighbor method and the largest with the nearest
method. This is likely because the physical neighbor method is already
partially reducing the copying effect.

#### Quantifying the Copying Effect.

To better quantify how much the gains are from avoiding the copying effect, we
follow Anonymous (2023) in (1) identifying some attention heads in the
Transformer layers that are most responsible for copying, and (2) zero-ing
their weights out. If this leads to performance improvements, it is a strong
indicator that the method has been suffering from the copying effect. We apply
this method to three different retrieval methods: nearest, diverse nearest and
physical neighbor introduced in Section 4.1.

Figure 7 reports results. First, all methods have performance improvements by
zero-ing out the attention heads, indicating that all of them suffer from the
copying effect to a certain degree. We then find that (1) nearest is affected
the most and physical neighbor is affected the least, and (2) methods with
synonym labeling are affected much less than their counterpart without synonym
labeling. These are aligned with our earlier intuition that using physical
neighbor instead of nearest, and using synonym labeling help reducing the
copying effect.

#### Effect of the size of the corpus.

We quantify the impact of the size of the corpus. This is important to judge
whether Z-icl can potentially achieve better results by scaling the corpus. We
evaluate Z-icl with a corpus with varying sizes, from 100% to 0.03% of the
corpus.

Figure 6 demonstrates that performance goes down as the size of the corpus
gets smaller. This is likely because there are less sentences that are
sufficiently close to the test input when the corpus is smaller, thus the
relevance of the nearest neighbors and the test input drops. This trend is
clearer on the datasets covered by $\mathcal{C}$ than on the datasets not
covered by $\mathcal{C}$.

#### Effect of the format of demonstrations.

How many input-label pairs does Z-icl need to benefit from pseudo-
demonstrations? Are gains from pseudo-demonstrations mainly from the fact that
the LM conditions on relevant text, or does the LM benefit from a specific
format of the pseudo-demonstrations: a concatenation of input-label pairs? To
answer these questions, we experiment with (1) Z-icl with varying range of $k$
from $1$ to $64$, and (2) a variant of Z-icl where the LM conditions on a
concatenation of retrieved inputs, without randomly paired labels (called
“Inputs-only”).

Results are shown in Figure 8. First, Z-icl is significantly better than zero-
shot baselines and stays on par with the oracle baselines consistently across
different values of $k$. Moreover, using no labels (“Inputs-only”) performs
significant worse than its counterparts. This suggests that Z-icl takes
advantages of the form of input-label pairs, and is beyond simply conditioning
on relevant context.

Figure 8:  Effect of the format of demonstrations with varying numbers of
demonstrations ($k$). Z-icl consistently performs on par with the oracle
baseline, and “Inputs-only” performs significantly worse.

#### Effect of the coverage of the corpus.

We quantify the impact of the coverage of the corpus, and whether adding more
domains in the corpus improves performance. We do so by adding the unlabeled
portion of IMDB review (Maas et al., 2011) to the corpus $\mathcal{C}$. The
size of $\mathcal{C}$ increases only by 2%, but covers the domain of three
datasets that were previously not covered (SST2, SST5 and MR).

Figure 9 shows the performance on three datasets before and after adding the
IMDB corpus. Performance improves consistently over all LMs, even though it
only adds up the size by 2%. This suggests that the coverage of the text
corpus is important, and it is feasible to further improve the overall
performance simply by expanding the corpus.

Figure 9:  Effect of the coverage of the corpus. Performance of Z-icl before
and after IMDB is added to the corpus. Expanding the coverage of the corpus
consistently improves the performance despite only 2% of the increase in the
size of the corpus.

##  7 Conclusion

We introduced Z-icl, a zero-shot in-context learning method that constructs
pseudo-demonstrations from a raw text corpus. Our method (1) retrieves
relevant text from the corpus using the nearest neighbor search, effectively
informing the correct space of the inputs to the LM, and (2) adjust the
pseudo-demonstrations with physical neighbor and synonym labeling to avoid the
copying effect. Evaluation on nine classification datasets shows Z-icl
significantly outperforms the previous zero-shot baseline, and performs on par
with the $k$-shot demonstrations. Overall, Z-icl demonstrates that
significantly higher LM zero-shot performance is possible, and opens up a new
research direction on the construction of better pseudo-demonstrations that
expose the full capacity of a LM.

## Limitation

#### Extension to multi-sentence tasks.

Our experiments are limited to single-sentence tasks, as we only retrieve
single-sentence nearest neighbors to a test input. Multi-sentence tasks such
as natural language inference would require constructing pseudo-demonstrations
that consists of multiple sentences, which we leave for future work.

#### Beyond classification.

Our experiments are limited to classification. Extensions to multi-choice
tasks or generation tasks requires going beyond a fixed set of options shared
between inputs in the demonstrations and the test input. We leave extensions
to non-classification tasks for future work.

#### Better construction of pseudo-demonstrations.

We think future work can explore better constructing the pseudo-
demonstrations. For instance, this paper uses manually chosen synonym labels
(see Appendix B for more detail). We hypothesize that better pseudo-
demonstrations can improve performance, which we leave for future work.

## Acknowledgements

We thank UW NLP members and anonymous reviewers for their comments in the
paper. This research was supported by NSF IIS-2044660, an Allen Distinguished
Award and gifts from AI2. SM is supported by a J.P. Morgan fellowship.

## References

  * Anonymous (2023) Anonymous. 2023.  Overthinking the truth: Understanding how language models process false demonstrations.  In _Submitted to The Eleventh International Conference on Learning Representations_.
  * Barbieri et al. (2020) Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa-Anke, and Leonardo Neves. 2020.  TweetEval:Unified Benchmark and Comparative Evaluation for Tweet Classification.  In _EMNLP_.
  * Black et al. (2022) Sid Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, USVSN Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022.  GPT-NeoX-20B: An open-source autoregressive language model.  In _Proceedings of the ACL Workshop on Challenges & Perspectives in Creating Large Language Models_.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _NeurIPS_.
  * Ding et al. (2008) Xiaowen Ding, Bing Liu, and Philip S Yu. 2008.  A holistic lexicon-based approach to opinion mining.  In _Proceedings of the 2008 international conference on web search and data mining_.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  Simcse: Simple contrastive learning of sentence embeddings.  In _EMNLP_.
  * Gururangan et al. (2021) Suchin Gururangan, Mike Lewis, Ari Holtzman, Noah A. Smith, and Luke Zettlemoyer. 2021.  Demix layers: Disentangling domains for modular language modeling.  In _NAACL_.
  * Johnson et al. (2019) Jeff Johnson, Matthijs Douze, and Hervé Jégou. 2019.  Billion-scale similarity search with GPUs.  _IEEE Transactions on Big Data_.
  * Liu et al. (2021) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen. 2021.  What makes good in-context examples for gpt-$3$?  In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_.
  * Liu et al. (2022) Yanchen Liu, Timo Schick, and Hinrich Schütze. 2022.  Semantic-oriented unlabeled priming for large-scale language models.  _arXiv preprint arXiv:2202.06133_.
  * Maas et al. (2011) Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies_.
  * Min et al. (2021) Sewon Min, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2021.  Noisy channel language model prompting for few-shot text classification.  In _ACL_.
  * Min et al. (2022) Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2022.  Rethinking the role of demonstrations: What makes in-context learning work?  In _EMNLP_.
  * Olsson et al. (2022) Catherine Olsson, Nelson Elhage, Neel Nanda, Nicholas Joseph, Nova DasSarma, Tom Henighan, Ben Mann, Amanda Askell, Yuntao Bai, Anna Chen, et al. 2022.  In-context learning and induction heads.  _arXiv preprint arXiv:2209.11895_.
  * Pang and Lee (2004) Bo Pang and Lillian Lee. 2004.  A sentimental education: Sentiment analysis using subjectivity summarization based on minimum cuts.  In _ACL_.
  * Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019\.  Pytorch: An imperative style, high-performance deep learning library.  In _NeurIPS_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_.
  * Razeghi et al. (2022) Yasaman Razeghi, Robert L Logan IV, Matt Gardner, and Sameer Singh. 2022.  Impact of pretraining term frequencies on few-shot reasoning.  In _EMNLP_.
  * Reynolds and McDonell (2021) Laria Reynolds and Kyle McDonell. 2021.  Prompt programming for large language models: Beyond the few-shot paradigm.  In _Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems_.
  * Rubin et al. (2021) Ohad Rubin, Jonathan Herzig, and Jonathan Berant. 2021.  Learning to retrieve prompts for in-context learning.  In _NAACL_.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _EMNLP_.
  * Wang and Komatsuzaki (2021) Ben Wang and Aran Komatsuzaki. 2021.  Gpt-j-6b: A 6 billion parameter autoregressive language model.
  * Xie et al. (2020) Qizhe Xie, Zihang Dai, Eduard Hovy, Minh-Thang Luong, and Quoc V Le. 2020.  Unsupervised data augmentation for consistency training.  In _NeurIPS_.
  * Xie et al. (2022) Sang Michael Xie, Aditi Raghunathan, Percy Liang, and Tengyu Ma. 2022.  An explanation of in-context learning as implicit bayesian inference.  In _ICLR_.
  * Zeng et al. (2022) Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022.  Glm-130b: An open bilingual pre-trained model.  In _ICLR_.
  * Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015.  Character-level convolutional networks for text classification.  In _NeurIPS_.
  * Zhao et al. (2021) Tony Z Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021.  Calibrate before use: Improving few-shot performance of language models.  In _ICML_.

##  Appendix A Data Statistics

Domain | Description | #sentences
---|---|---
1B | NewsWire sentences | 1.0M
CS | full-text CS papers from S2ORC | 1.0M
LEGAL | U.S. court opinions, 1658 to 2018 | 3.0M
MED | full-text medical papers from S2ORC | 1.0M
WEBTEXT | Web documents | 2.1M
REALNEWS | articles from REALNEWS | 1.8M
REDDIT | Reddit comments from pushshift.io | 2.6M
REVIEWS | Amazon product reviews | 3.1M
ACL PAPERS | NLP papers from ACL | 46K
BREAKING NEWS | latest articles from 400 English news sites | 0.5M
CONTRACTS | commercial legal contracts | 47K
CORD-19 | excerpts from COVID-19 research papers | 0.9M
GITHUB | public Github repository contents | 0.6M
GUTENBERG | copyright-expired books | 0.9M
TWEETS | English tweets from 2013-2018 | 0.8M
YELP REVIEWS | Yelp restaurant reviews | 7.5M
Table 6:  List of domains from Gururangan et al. (2021).  Dataset | # examples | labels | synonyms
---|---|---|---
Datasets covered by $\mathcal{C}$
CR | 2,000 | "terrible", "great" | "bad", "good"
Amz | 1,000 | "negative", "positive" | "bad", "good"
Amz5 | 100,050 $\rightarrow$ 2,000 | "terrible", "bad", "okay", "good", "great" | "horrible", "negative", "neutral", "positive", "excellent"
Yelp | 7,600 $\rightarrow$ 2,000 | "negative", "positive" | "bad", "good"
Yelp5 | 50,000 $\rightarrow$ 2,000 | "terrible", "bad", "okay", "good", "great" | "horrible", "negative", "neutral", "positive", "excellent"
Tweet | 2,000 | "negative", "neutral", "positive" | "bad", "normal", "good"
Datasets not covered by $\mathcal{C}$
MR | 2,000 | "terrible", "great" | "bad", "good"
SST2 | 872 | "terrible", "great" | "bad", "good"
SST5 | 2,210 $\rightarrow$ 2,000 | "terrible", "bad", "okay", "good", "great" | "horrible", "negative", "neutral", "positive", "excellent"
Table 7:  Statistics of evaluation datasets as well as their labels and
synonyms.

#### Corpus.

We take the same English corpus from Gururangan et al. (2021) covering 16
diverse domains: 1B, CS, LEGAL, MED, WEBTEXT, REALNEWS, REDDIT, REVIEWS, ACL
PAPERS, BREAKING NEWS, CONTRACTS, CORD-19, GITHUB, GUTENBERG, TWEETS, and YELP
REVIEWS. See the descriptions and statics in Table 6. For each domain, we 1)
subsample 10M paragraphs if the data is larger, 2) split each paragraph into
sentences, and 3) remove duplicate sentences while keeping the ordering of the
sentences as in the original paragraphs.

#### Evaluation datasets.

Statistics and descriptions of our evaluation datasets are reported in Table
7. For each dataset, we subsample 2000 test examples uniformly at random if
the test data is larger, due to limited computational resources.

##  Appendix B Implementation Details

All implementations are done in PyTorch (Paszke et al., 2019). We use int8
quantization (Zeng et al., 2022) to run GPT-NeoX on 40GB A100 machines.

#### Format of the demonstrations.

We use $k=16$ demonstration examples for all the baselines and methods, unless
specified otherwise. We truncate each demonstration example to have up to 256
tokens and the concatenation of them to have up to 1,024 tokens.

#### Nearest neighbor search.

We use SimCSE (Gao et al., 2021) to embed the corpus and the test inputs. We
use FAISS (Johnson et al., 2019) to build an index for the corpus offline and
perform nearest neighbor search at inference.

#### Synonym labeling.

We manually choose a synonym of each label to perform synonym labeling. A full
list of synonyms is reported in Table 7.

#### Computational Budget.

Our main experiment on the 4 public LMs in Table 4 takes around 4,000
computing hours with a 40GB A100 machine. Our experiment using GPT-3’s API
costs around 4,500 US Dollars.

####