# Evaluation of Few-Shot Learning for Classification Tasks in the Polish
Language

Tsimur Hadeliya, Dariusz Kajtoch
ML Research at Allegro, Poznań, Poland
<EMAIL_ADDRESS>
###### Abstract

We introduce a few-shot benchmark consisting of 7 different classification
tasks native to the Polish language. We conducted an empirical comparison with
0 and 16 shots between fine-tuning, linear probing, SetFit, and in-context
learning (ICL) using various pre-trained commercial and open-source models.
Our findings reveal that ICL achieves the best performance, with commercial
models like GPT-3.5 and GPT-4 attaining the best performance. However, there
remains a significant 14 percentage points gap between our best few-shot
learning score and the performance of HerBERT-large fine-tuned on the entire
training dataset. Among the techniques, SetFit emerges as the second-best
approach, closely followed by linear probing. We observed the worst and most
unstable performance with non-linear head fine-tuning. Results for ICL
indicate that continual pre-training of models like Mistral-7b or Llama-2-13b
on Polish corpora is beneficial. This is confirmed by the improved
performances of Bielik-7b and Trurl-13b, respectively. To further support
experiments in few-shot learning for Polish, we are releasing handcrafted
templates for the ICL.

##  1 Introduction

Few-shot learning enables models to grasp concepts from a limited number of
examples (Tunstall et al., 2022; Ye et al., 2021; Kwon et al., 2023), reducing
the resources needed for data curation and training. This efficiency lowers
the barriers to introducing new tasks, allowing for quicker deployment and
more exploration. Over time, researchers have developed new methods for
learning from a few examples. Progress has been driven mainly by fine-tuning
pre-trained transformer-based models (Schick and Schütze, 2021a, b; Tam et
al., 2021; Tunstall et al., 2022; Liu et al., 2022; Karimi Mahabadi et al.,
2022; Aly et al., 2023; Albalak et al., 2023) or employing in-context learning
(ICL) in Large Language Models (LLMs) (Brown et al., 2020; Ahuja et al., 2023;
Asai et al., 2023). While few-shot benchmarks show clear progress in English,
it is accompanied by both novel techniques and progress in the development of
pre-trained models. For practitioners working with non-English data that
advancement is always inspiring but does not fully answer the question: _What
approach or model should I use for a given task in language X?_ This is
grounded by the fact that different languages possess different resources
(e.g. pre-trained models) and datasets.

Figure 1: Summary of average performance on the few-shot classification
benchmark for different training techniques and 16 shots. The last bar on the
right reports results for the HerBERT-large fine-tuned on the whole train
dataset. We observe that ICL with GPT-3.5 achieves the best average
performance followed by SetFit, Linear probing, and fine-tuning with a much
smaller SBERT-large model. The gap between HerBERT-large and GPT-3.5 is around
20.4 percentage points

In this paper, we aim to evaluate various few-shot learning methodologies and
assess the capability of pre-trained models in processing and understanding
the Polish language. Unlike many studies that depend on multilingual or cross-
lingual analysis with translated datasets, we emphasize native Polish language
datasets to avoid potential translation and cultural biases (Ruder and Sil,
2021; Artetxe et al., 2020; Ponti et al., 2020). We have developed a Polish
language classification benchmark, incorporating 7 existing datasets: PAC,
Polemo2, DYK, CDSC-E, NKJP-NER, CBD, and CST-Wikinews (Augustyniak et al.,
2022; Rybak et al., 2020). These datasets were carefully chosen to represent a
broad spectrum of characteristics, such as input length, domain specificity,
and the number of classes. In our study, we used a few-shot benchmark to
compare fine-tuning, linear probing, SetFit Tunstall et al. (2022), and in-
context learning using various pre-trained models. Figure 1 summarizes the
average performance for 16-shots. We hope this paper will benefit researchers
focused on the Polish language and practitioners selecting suitable models for
their tasks.

We summarize the contributions of this work as follows:

  * •

We utilized existing datasets to establish an evaluation benchmark comprising
7 different datasets for few-shot classification in the Polish language. We
also created a collection of 71 manually crafted template prompts for in-
context learning.

  * •

We conducted a comparative study of fine-tuning, linear probing, SetFit, and
ICL across various transformer-based models, revealing that ICL excels in
highly scarce few-shot scenarios with only 16-shots. However, we observe a
significant difference of 14.0 percentage points between the best-performing
model, namely the zero-shot GPT-4, and the performance achieved by fine-tuning
HerBERT-large on the entire dataset.

  * •

We illustrate, with Bielik-7b and Trurl-2-13b, that continual pre-training in
the target language is the zero-shot regime. For 16-shots, only Trurl-2-13b is
better than its English counterpart. We also observe that continual pre-
training on the target domain needs to be done with caution which is
demonstrated by the lower performance of Krakowiak-7b-v2 which descends from
Mistral-7b.

(a) In-context learning evaluation

(b) Fine-tuning methods evaluation

Figure 2: Training and evaluation process of few-shot (as example 4-shot)
learning. Both methods begin by sampling $n$ (as $n$-shot) examples from the
task’s training data using a fixed random seed. We use 5 random seeds for
reproducibility and to measure variance. (a) The sampled examples serve as in-
context demonstrations. The evaluation prompt combines manually written
instructions, demonstrations, and an optional system prompt. The model uses
greedy decoding to generate a label for the test example. An exact match
compares the generated label with the golden labels. Predictions without a
matching label receive a special label. (b) The sampled examples form the
training dataset for the specified fine-tuning method (the Full Fine-tuning
method utilizes the entire dataset). Subsequently, the fine-tuned model is
evaluated on the test dataset in the conventional manner.

##  2 Related Work

##### Few-shot benchmarks

Comparing models in a few-shot regime is not free from many nuances (Bragg et
al., 2021; Ye et al., 2021; Alex et al., 2021; Zheng et al., 2022). The
FewGLUE benchmark Schick and Schütze (2021b) randomly samples the same number
of examples per task from the GLUE benchmark and provides several tens of
thousands of unlabeled samples. Similarly, the CrossFit Ye et al. (2021) uses
a fixed number of examples per class. A small development set is also
provided. The FLEX Bragg et al. (2021) argues that the benchmark should
include a variable number of examples and lack of development set among other
factors. The RAFT benchmark Alex et al. (2022) focuses on naturally occurring
tasks and uses an evaluation setup that mirrors deployment.

##### Few-shot learning

Over the years, numerous few-shot learning methods have been suggested. This
discussion will focus on the most critical ones. First, ICL, used with large
language models like GPT-3 Brown et al. (2020), requires no modifications to
the model. Demonstrations are joined with task instruction and forward through
the model which generated the answer. As these models improve, they also
become more effective at solving few-shot tasks. Another approach involves
various forms of fine-tuning of pre-trained models. For instance, the PET
Schick and Schütze (2021a) and iPET (Schick and Schütze, 2021b) methods
transform training examples into cloze questions and fine-tune the model on a
small collection of examples, which is then used to softly annotate a pool of
unlabeled examples. These methods rely on handcrafted task instructions and
verbalizers, which the PERFECT method (Karimi Mahabadi et al., 2022) replaces
with task-specific adapters and does not depend on unlabeled data, similar to
ADAPET (Tam et al., 2021). Additionally, parameter-efficient fine-tuning has
shown cutting-edge results as demonstrated in studies like (Logan IV et al.,
2022), T-Few (Liu et al., 2022), and AuT-Few (Aly et al., 2023). Lastly,
combining an embedding model with a linear classifier can yield good results
with far fewer parameters, as seen with the SetFit method Tunstall et al.
(2022), which begins with contrastive learning to enhance data comprehension
and then trains a linear classifier on these refined, static embeddings.

##### Large Language Models (LLMs)

In recent years, there has been significant growth in both commercial and
open-source models (Jiang et al., 2023; Touvron et al., 2023; Team et al.,
2024). Although many architectures are available (Tay et al., 2023), the
success of GPT-like models (Brown et al., 2020; Ouyang et al., 2022; OpenAI et
al., 2024) demonstrates the dominance of decoder-only architectures in
generative tasks and ICL. Another key feature of modern models is their scale.
Currently, there appears to be no limit to the benefits of scaling, leading to
increasingly larger models (Fedus et al., 2022; Chowdhery et al., 2022; Rae et
al., 2022; Thoppilan et al., 2022) and alternative scaling strategies such as
Mixture-of-Experts (MoE) (Shazeer et al., 2017; Fedus et al., 2022; Jiang et
al., 2024). Some research suggests that scaling endows models with new
capabilities not present in smaller versions (Wei et al., 2022b), though this
remains a topic of ongoing research (Schaeffer et al., 2023). Another
significant research area for LLMs involves fine-tuning methods (Sanh et al.,
2022; Chung et al., 2022; Wei et al., 2022a) to enable few-shot learning and
the use of natural language instructions. Reinforcement Learning (RL) methods
(Christiano et al., 2023; Ouyang et al., 2022) allows the integration of human
feedback to improve the model’s ability to generate coherent responses,
although these methods are still challenging to train. New methods that could
enhance or replace RL are being proposed (An et al., 2023; Rafailov et al.,
2023).

##### Multilingual capabilities of LLMs

Dominant studies on the performance of LLMs are done for the English language.
Little was known about the multi-lingual capabilities of these models. Lately,
a few papers studied this topic with both open-source and commercial models.
ChatGPT’s performance is generally better for tasks formulated in English than
other languages (Lai et al., 2023; Ahuja et al., 2023; Zhang et al., 2023;
Asai et al., 2023). However, a closer look at the data reveals that it
strongly depends on the language and task. Typically, the performance gap is
larger for low-resource languages. Following the same works, similar
observations were made for the BLOOM Scao et al. (2023) and BLOOMZ Muennighoff
et al. (2023) multi-lingual models. Analogous behavior is noticed for English-
dominant models like Llama-2 Bandarkar et al. (2023).

##  3 Problem Statement

Few-shot learning aims to learn a model from just a handful of labeled
training examples. The special case is zero-shot learning where no labeled
data is provided for training. We use $n$ to denote the number of total
training examples.

Research in Polish NLP is lacking comparative evaluations among various few-
shot techniques and pre-trained models. Additionally, the complexity of these
methods and the computational resource requirements should be considered. Many
non-English languages demonstrate cross-lingual capabilities but are often
evaluated using translated datasets, which may introduce translation and
cultural biases. Moreover, these methods are not routinely compared with
simple baselines. In our study, we perform a comparative analysis of selected
few-shot learning methods and pre-trained models on Polish classification
tasks. We compare our results with both simple baselines and state-of-the-art
(SOTA) results on the entire dataset. Our focus is on classification tasks due
to their wide application across numerous fields and the simplicity of
evaluation.

Name |  Input |  #Test examples |  #Classes |  #Instructions | Avg. len |  Dataset Domain
---|---|---|---|---|---|---
Lepiszcze
PAC |  text |  3453 |  2 |  13 | 185 |  legal texts
DYK |  text pair |  1029 |  2 |  14 | 288 |  Wikipedia
CDSC-E |  text pair |  1000 |  3 |  8 | 144 |  image captions
Polemo2 |  text |  820 |  4 |  11 | 758 |  online reviews
KLEJ
CBD |  text |  1000 |  2 |  9 | 93 |  social media
NKJP-NER |  text |  2058 |  6 |  9 | 85 |  national corpus
Other
CST-Wikinews |  text pair |  384 |  12 |  7 | 232 |  Wikinews
Table 1: Descriptions and basic statistics of datasets used as a benchmark for
evaluation of few-shot learning capabilities. The #Instructions denote the
number of handcrafted instructions for the in-context learning method. While
creating a benchmark, we choose a dataset based on the parameters listed in
the column. This helped us to prepare a diversified classification benchmark,
with data from different domains, different number of classes, and length
distribution.

##  4 Methodology

###  4.1 Datasets

Numerous classification datasets are available for the Polish language. For
the construction of the benchmark, 7 such datasets were selected, which vary
in domain and class quantity. The PAC, Polemo2, DYK, and CDSC-E datasets
constitute a subset of the Lepiszcze benchmark Augustyniak et al. (2022).
Meanwhile, the NKJP-NER and CBD datasets are incorporated within the KLEJ
benchmark Rybak et al. (2020). Additionally, the CST-Wikinews dataset
comprises 1,530 training sentence pairs extracted from Polish Wikipedia,
annotated with 12 distinct relational categories between the first and second
sentences, such as inclusion, description, and summarization
111https://huggingface.co/datasets/clarin-pl/cst-wikinews. The datasets also
exhibit variation in the average length of examples. Table 1 provides a
summary of these datasets. Links to open-source artifacts can be found in the
Appendix B.

##### Evaluation metrics

For datasets endowed with binary labels, the F1 score for the positive class
is reported, whereas Accuracy is employed for the other datasets. Such a
combination of metrics is used to have consistency with the results from the
KLEJ benchmark Rybak et al. (2020).

|  AVG |  PAC |  Polemo2 |  CBD |  NKJP-NER |  DYK |  CDSC-E |  CST-Wikinews
---|---|---|---|---|---|---|---|---
XGBoost | 32.4 ± 3.1 | 56.4 ± 16.7 | 38.5 ± 3.4 | 19.7 ± 6.0 | 20.9 ± 4.7 | 22.6 ± 5.1 | 57.5 ± 10.4 | 11.4 ± 1.5
Random Guessing | 26.6 ± 0.4 | 57.5 ± 0.7 | 24.8 ± 1.0 | 20.9 ± 1.6 | 16.8 ± 0.8 | 25.5 ± 1.7 | 33.6 ± 0.7 | 6.9 ± 0.9
Constant Classifier | 42.9 | 80.6 | 41.3 | 23.6 | 34.3 | 28.9 | 74.4 | 16.9
/SF/ SBERT-large | 47.1 ± 2.4 | 68.8 ± 6.5 | 69.9 ± 10.7 | 44.4 ± 5.4 | 30.7 ± 6.5 | 27.7 ± 2.9 | 72.3 ± 5.7 | 16.2 ± 3.2
/LP/ SBERT-large | 43.4 ± 1.9 | 67.5 ± 7.0 | 60.3 ± 4.6 | 40.1 ± 4.3 | 30.4 ± 5.9 | 27.2 ± 1.8 | 62.8 ± 5.8 | 15.3 ± 3.0
/FT/ SBERT-large | 31.2 ± 7.0 | 33.6 ± 31.3 | 47.0 ± 5.8 | 32.0 ± 30.7 | 28.1 ± 7.7 | 6.4 ± 10.3 | 61.0 ± 17.6 | 10.1 ± 4.3
/ICLn=16/ GPT-3.5 | 59.5 ± 1.4 | 73.9 ± 3.6 | 81.9 ± 2.1 | 64.1 ± 1.9 | 46.1 ± 2.9 | 64.1 ± 1.8 | 66.7 ± 7.6 | 19.8 ± 2.7
/ICLn=0/ GPT-4 | 65.9 | 83.3 | 82.6 | 60.6 | 58.8 | 81.3 | 76.0 | 18.8
HerBERT-large | 79.9 ± 0.6 | 91.1 ± 0.0 | 90.9 ± 0.0 | 53.2 ± 3.2 (72.0) | 94.0 ± 0.0 | 68.8 ± 2.1 (75.8) | 93.4 ± 0.0 | 67.9 ± 1.0
Table 2: Evaluation results summary on the test dataset. We employed a dataset
comprising 16 training examples per task, utilizing 5 distinct random seeds
for analysis. Subscripts denote standard deviation. The acronyms SF, LP, FT,
and ICL represent SetFit, Linear Probing, Fine-Tuning, and In-context
Learning, respectively. The outcomes for the GPT model family pertain to in-
context learning. For GPT-4 model we report zero-shot performance. The
terminal row delineates the performance metrics after fine-tuning across the
entire training dataset. Parenthetically, we reference the results derived
from the KLEJ benchmark 333https://klejbenchmark.com/leaderboard/.

###  4.2 Experimental setup

We adopt a methodology analogous to that described in Schick and Schütze
(2021b). For each dataset, we sample a random subset comprising $n=16$
instances from the training set. We wanted the number to be as maximal as
possible and still possible to process by the ICL method. It is critical to
ensure that the derived subset is balanced (where feasible) and that the
labels are sequenced in an alternating fashion. The entirety of the validation
dataset is utilized for the hyperparameter selection. In the absence of a
predefined validation set, the training set is partitioned into a 20%
validation subset and an 80% training subset. Consequently, our analysis
concentrates on the upper-performance limit in few-shot scenarios, given the
disproportionately large size of the validation dataset. The experiment is
conducted 5 times with varying random subsets. We set the following random
seeds in our experiments: [18, 22, 37, 69, 98]. For each approach, the
arithmetic mean and standard deviation are documented. Additionally, for the
ICL, we also report the zero-shot performance.

###  4.3 Models

Our study concentrates on the evaluation of transformer-based models with
specific attributes, encompassing: (a) medium-sized models, such as Polish
Sentence BERT-large and HerBERT-large; (b) Polish open-source LLMs, notably
GPT-2-xl, Trurl-2, Bielik-7b, and Krakowiak-v2; (c) English open-source LLMs,
such as Llama-2 and Mistral-7b; and (d) commercial LLMs, including PaLM-2,
GPT-3.5, and GPT-4. The selection criteria are predicated on language support
and the incorporation of instruction tuning (Mishra et al., 2022; Sanh et al.,
2022; Wei et al., 2022a) during the pretraining phase. Details on model
artifacts can be found in the Appendix D.

###  4.4 Training schemes

The experiments conducted can be divided into two categories: few-shot ICL and
few-shot parameter-efficient learning. The former focuses on generating labels
using only demonstrations and task descriptions in the input, without updating
the model weights. The latter involves fine-tuning the model for a new task
using only $n$ examples. These approaches are depicted in Figure 2. Below, we
provide more detailed descriptions for each method.

##### Baseline

The baseline model comprises an _XGBoost_ Chen and Guestrin (2016) algorithm,
trained utilizing hashed Term Frequency-Inverse Document Frequency (TF-IDF)
features. Initially, text tokenization is conducted via spaCy Honnibal and
Montani (2017). Subsequently, each token is hashed employing the Fowler-Noll-
Vo (FNV) algorithm and then mapped to a predetermined vocabulary dimension.
Performance metrics are also presented for _Random Guessing_ , entailing
random label selection, and for the _Constant Classifier_ method, which
consistently predicts the label maximizing the test score.

##### Linear probing

In this approach, each example is projected onto a constant embedding using a
pre-trained model, followed by training a Logistic Regression classifier on
these embeddings. During the training process, embeddings are frozen and not
normalized.

##### SetFit

SetFit represents an effective fine-tuning methodology for few-shot learning
(Tunstall et al., 2022), combining metric learning with linear probing. The
implementation of SetFit is conducted in conjunction with LoRA Hu et al.
(2022) applied to all linear layers.

##### Fine-tuning

The fine-tuning of the backbone transformer model is executed using cross-
entropy loss, integrating a non-linear classification head. Throughout the
training phase, only the LoRA modules (applied to all linear layers) alongside
the head parameters are adjusted. The same learning rate is applied for both
the backbone and the head components.

##### In-context learning

In-context learning functions without the alteration of model parameters
(Radford et al., 2019; Brown et al., 2020). Instructions and examples are
provided as input to the language generation model, from which an appropriate
label is anticipated as generated output. Further details are available in
Appendix E.

###  4.5 Prompt Engineering

For each dataset, we systematically hand-crafted diverse prompts derived from
the dataset characteristics and anticipated label categories. We mostly
followed PromptSource Bach et al. (2022) and Natural Instructions Mishra et
al. (2022); Wang et al. (2022), without employing automated prompt generation
methods (Zhou et al., 2023; Deng et al., 2022; Shin et al., 2020; Prasad et
al., 2023; Pryzant et al., 2023; Anonymous, 2024a, b; Fernando et al., 2023).
The construction of each prompt entailed 3 distinct segments. It starts with
universal system directive in Polish, specifically: "Rozwiązujesz zadanie
klasyfikacji dla języka polskiego.". This was followed by optional task-
specific instruction and subsequent demonstrations. Refer to Figure 3 for an
exhaustive depiction. The prompts are cataloged in the Appendix H.

⬇

instruction: ’Oceń czy następujący zapis jest uczciwy lub nieuczciwy.’

demonstration: ’{text} => {labels}’

label_mapping:

0: nieuczciwy

1: uczciwy

Figure 3: Example building blocks of the prompt for in-context learning. Every
prompt start with system instruction: "Rozwiązujesz zadanie klasyfikacji dla
języka polskiego.". Then, follows a instruction and a sequence of
demonstrations. Every block is joined with a newline character. The
demonstration part specifies the way to format examples. Specifically, {text}
and {labels} are substituted with text and true labels from the dataset,
respectively. Label mapping is used to map textual labels into numeric values.

##### Response cleansing

In the domain of ICL, each model produces a response that is subsequently
classified into one of the predefined labels using some heuristics. We assume
that the label is present in the response. Minimal postprocessing is employed,
encompassing conversion to lowercase, removal of diacritics, and elimination
of extraneous whitespace. There was no pursuit of more advanced methodologies,
such as instructing language models to generate outputs conforming to a
specific JSON schema. If the label is absent in the response, it is treated as
invalid. When we calculate the metrics, we replace invalid responses with a
class different from ground truth to lower the metrics automatically.

##  5 Results & Discussion

Table 3 summarizes the outcomes from various training methodologies. The
HerBERT-large model, fine-tuned on the entire training dataset using Focal
Loss (Lin et al., 2020)444Additional experiments were conducted with scripts
from the Lepiszcze benchmark https://github.com/dkajtoch/LEPISZCZE, resulting
in similar outcomes., is also included. Scores from the KLEJ benchmark for the
CBD and DYK tasks are shown in parentheses
555https://klejbenchmark.com/leaderboard/.

We found that nearly all models outperform the simple XGBoost baseline
significantly. An exception is the fine-tuning performance on PAC and DYK
tasks, which lags behind. Additionally, almost all models surpass the Random
Guessing baseline, including XGBoost. However, the Constant Classifier sets a
notable threshold due to the severe imbalance in the PAC and CDSC-E datasets,
which all models except GPT-4 fail to exceed. For the other datasets, the
average performance is better than the Constant Classifier baseline.

ICL with commercial LLMs achieves the highest performance across the models
and techniques. The second-best method is SetFit, followed by Linear probing.
The poorest performance and greatest uncertainties are associated with fine-
tuning. We note that linear probing is the most computationally efficient
method. On the other hand, ICL requires large models and is computationally
expensive.

ICL using commercial models like GPT-4 shows superior performance. However,
there is a gap of 14pp between zero-shot GPT-4 and HerBERT-large trained on
the full dataset. Notably, GPT-4 surpasses comprehensive fine-tuning on the
DYK task by 5.5 percentage points, reaching 81.3 points. This discrepancy
warrants further investigation into potential dataset leakage, as GPT-3.5
shows much lower performance.

###  5.1 Linear probing

Table 3 presents the outcomes of employing a linear probing methodology across
various embedding models. The investigation prioritized 5 open-source and 4
commercial backbones. SBERT-large exhibits exceptional efficacy in the CBD and
NKJP-NER tasks, securing the top positions within the linear probing models
for these datasets. For the DYK and CDSC-E datasets, HerBERT-large is
identified as the most effective performer in this segment.

Among the linear probing models, no single model consistently surpasses the
rest across all tasks; however, SBERT-large and HerBERT-large demonstrate
robust performance in distinct datasets. When considering the standard
deviation, the performance of all models tends to converge on an average
scale. Nonetheless, a pronounced disparity is observed in the Polemo2 task,
where SBERT-large significantly outshines HerBERT-large, yet shows comparable
performance to RoBERTa-large.

On average, the Gecko embeddings underperform relative to other backbones,
exhibiting performance akin to that of the Llama-2-70b model. This finding
starkly contrasts with the superior average performance achieved by the
multilingual variant of the Gecko embeddings 666Determining whether the
enhanced performance of the multilingual version is attributable to linguistic
factors is complex. Both versions are predicated on the identical Gecko
iteration but were released at different times. Subsequent versions also
underwent API modifications and necessitated specifying
task_name=CLASSIFICATION..

Embedding-based few-shot learning is generally plagued by high variance,
occasionally escalating to 12 percentage points. This variance stems from the
sampling of demonstrations and the fine-tuning process. In the context of
embeddings, the validation dataset offers limited utility.

|  AVG |  PAC |  Polemo2 |  CBD |  NKJP-NER |  DYK |  CDSC-E |  CST-Wikinews
---|---|---|---|---|---|---|---|---
SBERT-large | 43.4 ± 1.9 | 67.5 ± 7.0 | 60.3 ± 4.6 | 40.1 ± 4.3 | 30.4 ± 5.9 | 27.2 ± 1.8 | 62.8 ± 5.8 | 15.3 ± 3.0
HerBERT-large | 43.0 ± 2.4 | 72.6 ± 10.3 | 47.0 ± 3.1 | 33.8 ± 10.1 | 27.8 ± 3.3 | 34.6 ± 2.8 | 71.0 ± 6.1 | 14.2 ± 2.7
RoBERTa-large | 41.4 ± 2.4 | 67.8 ± 11.3 | 57.9 ± 4.4 | 37.4 ± 6.1 | 27.6 ± 3.9 | 25.1 ± 3.0 | 59.2 ± 7.5 | 14.5 ± 2.7
GPT-2-large | 39.8 ± 1.7 | 72.0 ± 2.9 | 48.3 ± 4.6 | 36.4 ± 4.1 | 25.2 ± 4.3 | 28.6 ± 3.1 | 54.4 ± 7.5 | 13.6 ± 3.0
Llama-2-70b* | 38.1 ± 2.2 | 67.6 ± 12.5 | 41.4 ± 2.9 | 28.4 ± 2.5 | 24.2 ± 2.0 | 30.7 ± 3.0 | 61.1 ± 6.1 | 13.5 ± 4.2
Ada | 40.9 ± 2.3 | 72.9 ± 6.3 | 55.2 ± 5.0 | 30.7 ± 4.2 | 29.1 ± 2.5 | 25.4 ± 4.1 | 58.2 ± 11.7 | 14.8 ± 3.9
DaVinci | 42.7 ± 2.6 | 67.6 ± 7.0 | 58.9 ± 1.3 | 36.9 ± 9.7 | 30.1 ± 6.6 | 29.9 ± 2.7 | 60.1 ± 11.2 | 15.5 ± 3.5
Gecko | 37.7 ± 1.7 | 61.8 ± 4.5 | 42.9 ± 6.4 | 23.7 ± 1.3 | 24.8 ± 4.3 | 25.0 ± 3.8 | 68.8 ± 6.3 | 17.0 ± 2.8
Gecko multilingual | 44.0 ± 2.3 | 72.2 ± 5.6 | 60.8 ± 2.6 | 41.0 ± 3.3 | 28.4 ± 2.8 | 27.4 ± 2.3 | 62.1 ± 12.6 | 16.2 ± 5.7
Table 3: Evaluation results on the test dataset for linear probing. We used 16
train examples per task and 5 random seeds. Subscripts denote standard
deviation. *Llama-2 embeddings were generated with 4-bit quantization.

####  5.1.1 Scaling

Linear probing represents a straightforward and cost-efficient method among
the available techniques. This elicits the question, How many examples are
required to achieve a predetermined performance level? In Figure 4, the
average performance of SBERT-large and Ada across various training example
counts is illustrated. An almost perfect logarithmic relationship is observed
between the average score and the quantity of examples. With the increment in
the number of examples, the uncertainty diminishes, and Ada exhibits
marginally superior performance compared to the SBERT-large model.

Figure 4: Average performance of the linear probing method based on Ada and
SBERT-large embeddings as a function of train dataset size.

###  5.2 Fine-tuning

Table 4 presents the results of fine-tuning for $n=16$ training instances
across various backbones. Generally, achieving a stable model using the fine-
tuning approach proved to be challenging, leading to ambiguous comparisons
between backbone efficacies. Conversely, SetFit demonstrated consistent
performance across HerBERT-based backbones. In the context of this limited
training scenario, the superiority of Sentence BERT over the original BERT
remains uncertain due to significant variability. It is imperative to
highlight that both fine-tuning methodologies employed LoRA approximation
across all linear layers, necessitating cautious interpretation compared with
comprehensive parameter fine-tuning.

|  AVG |  PAC |  Polemo2 |  CBD |  NKJP-NER |  DYK |  CDSC-E |  CST-Wikinews
---|---|---|---|---|---|---|---|---
with non-linear head
SBERT-large | 31.2 ± 7.0 | 33.6 ± 31.3 | 47.0 ± 5.8 | 32.0 ± 30.7 | 28.1 ± 7.7 | 6.4 ± 10.3 | 61.0 ± 17.6 | 10.1 ± 4.3
HerBERT-large | 25.9 ± 8.6 | 26.7 ± 23.4 | 29.5 ± 9.7 | 30.8 ± 36.6 | 25.2 ± 3.8 | 16.2 ± 29.7 | 44.4 ± 26.7 | 8.7 ± 3.7
RoBERTa-large | 24.2 ± 4.5 | 6.2 ± 11.5 | 36.4 ± 4.3 | 15.3 ± 10.5 | 21.3 ± 6.1 | 27.2 ± 0.4 | 54.5 ± 26.0 | 8.7 ± 2.8
SetFit
SBERT-large | 47.1 ± 2.4 | 68.8 ± 6.5 | 69.9 ± 10.7 | 44.4 ± 5.4 | 30.7 ± 6.5 | 27.7 ± 2.9 | 72.3 ± 5.7 | 16.2 ± 3.2
HerBERT-large | 44.1 ± 3.9 | 70.7 ± 11.9 | 46.0 ± 11.2 | 42.5 ± 10.9 | 25.8 ± 11.6 | 40.6 ± 4.7 | 67.2 ± 14.5 | 15.7 ± 1.9
Table 4: Evaluation results for the fine-tuning technique on the test dataset.
We used 16 train examples per task and 5 random seeds. Subscripts denote
standard deviation. All models used LoRA approximation across all linear
layers.

###  5.3 In-context learning

Prior methodologies concentrated on models characterized by a constrained set
of hyperparameters, such as learning rate, batch size, and training steps. In
contrast, ICL employs pre-trained models that necessitate no further training,
yet a text prompt furnishes an exponential array of choices. Presently, it is
feasible to differentiate between models based on various attributes,
including size, corpus language, and data formats employed for fine-tuning,
such as instruction tuning.

##### Few-shot

Table 5 provides a detailed comparative analysis of the few-shot ($n=16$) ICL
performance of various language models across distinct classification tasks.
Commercially developed models such as GPT-3.5, achieving an average score of
59.5, and Bison-text, with 53.4, exhibit good performance metrics. GPT-3.5, in
particular, demonstrates superior capability, notably in the Polemo2 task,
achieving an 81.9 score. Conversely, Bison-text does very well on the PAC
task, scoring 83.7. However, it does poorly on the CBD task, getting a 0.0
score. This poor performance is because it fails to produce suitable
responses. This failure is linked to breaking Google’s rules about content
related to cyberbullying, which includes inappropriate language.

Further exploration into task-specific performances reveals the Polemo2 task
as a significant challenge, largely due to the constraints related to the
models’ context size. This constraint leads to zero scores in models such as
Bielik-7b-instruct, Trurl-2-13b, and GPT-2-xl, each posting an average
performance score of 31.7, 22.5, and 19.3, respectively. The considerable
uncertainty metrics, like ±12.0 for Trurl’s PAC task and ±22.5 for Krakowiak’s
PAC, underscore the variability and challenges inherent in securing stable
few-shot learning outcomes. In stark contrast, Mistral-7b-instruct, though
fine-tuned for English, attains a moderate average score of 46.5, with a
notably high score of 77.6 in the CDSC-E task, indicating potential cross-
lingual learning capabilities. Llama-2-13b-chat, on the other hand,
significantly underperforms with an average score of merely 12.2.
Mistral-7b-instruct emerges as the superior model among open-source variants,
despite not being specifically fine-tuned for Polish, reflecting the complex
dynamics of few-shot learning in a multilingual environment where language-
specific fine-tuning and intrinsic model capabilities are crucial.
Bielik-7b-instruct which is based on the same architecture and initial weights
attains significantly worse performance. Further analysis is needed to
understand Mistral’s performance.

|  AVG |  PAC |  Polemo2 |  CBD |  NKJP-NER |  DYK |  CDSC-E |  CST-Wikinews
---|---|---|---|---|---|---|---|---
GPT-3.5 | 59.5 ± 1.4 | 73.9 ± 3.6 | 81.9 ± 2.1 | 64.1 ± 1.9 | 46.1 ± 2.9 | 64.1 ± 1.8 | 66.7 ± 7.6 | 19.8 ± 2.7
Bison-text | 53.4 ± 1.3 | 83.7 ± 1.4 | 81.8 ± 2.8 | 0.0 ± 0.0 | 45.7 ± 3.6 | 76.6 ± 0.7 | 66.4 ± 7.3 | 19.5 ± 2.2
Bielik-7b-instruct | 31.7 ± 1.5 | 76.1 ± 4.7 |  0.0 ± 0.0 | 23.7 ± 0.1 | 23.2 ± 2.2 | 38.3 ± 3.8 | 55.0 ± 10.7 | 5.4 ± 1.3
Trurl-2-13b | 22.5 ± 4.0 | 66.7 ± 12.0 |  0.0 ± 0.0 | 12.1 ± 6.6 | 15.9 ± 11.3 | 30.8 ± 1.4 | 26.1 ± 21.8 | 6.3 ± 3.2
Krakowiak-7b-v2 | 22.5 ± 5.5 | 57.9 ± 22.5 | 14.4 ± 18.5 | 0.0 ± 0.0 | 13.8 ± 1.0 | 28.9 ± 0.1 | 38.0 ± 24.5 | 4.8 ± 4.7
GPT-2-xl | 19.3 ± 4.0 | 24.8 ± 24.7 |  0.0 ± 0.0 | 0.0 ± 0.0 | 14.0 ± 11.2 | 16.9 ± 1.8 | 74.0 ± 0.5 | 5.2 ± 5.7
Mistral-7b-instruct | 46.5 ± 2.1 | 48.8 ± 6.8 | 73.5 ± 6.1 | 34.4 ± 8.6 | 28.2 ± 1.8 | 45.1 ± 1.6 | 77.6 ± 6.5 | 17.9 ± 2.4
Llama-2-13b-chat | 12.2 ± 6.8 | 47.8 ± 41.7 |  0.0 ± 0.0 | 0.2 ± 0.4 | 3.9 ± 7.7 | 20.0 ± 8.8 | 12.1 ± 18.9 | 1.1 ± 2.4
Table 5: Evaluation results for the ICL on the test dataset. We used 16 train examples per task and 5 random seeds. Subscripts denote standard deviation. Highlighted scores for the Polemo2 task are 0.0 since 16 demonstrations do not fit into the context of selected models. |  AVG |  PAC |  Polemo2 |  CBD |  NKJP-NER |  DYK |  CDSC-E |  CST-Wikinews
---|---|---|---|---|---|---|---|---
GPT-4 | 65.9 | 83.3.0 | 82.6.009 | 60.6.028 | 58.8.004 | 81.3.002 | 76.0.0 | 18.8.003
GPT-3.5 | 55.4 | 82.2.014 | 81.6.005 | 50.0.046 | 44.9.001 | 53.1.0 | 62.9.0 | 13.3.003
Bison-text | 52.2 | 80.2.006 | 80.7.009 | 42.6.077 | 47.5.027 | 61.6.016 | 35.0.001 | 17.7.003
Bielik-7b-instruct | 51.5 | 78.1.034 | 77.3.0 | 53.9.0 | 31.0.01 | 49.0.001 | 65.3.0 | 6.0.023
Trurl-2-13b | 50.2 | 74.1.064 | 79.5.001 | 50.0.001 | 33.6.038 | 49.7.015 | 50.2.001 | 14.6.083
Krakowiak-7b-v2 | 43.9 |  80.6.001 | 65.0.018 |  23.6.0 | 16.7.024 | 41.4.002 |  74.4.0 | 6.0.034
GPT-2-xl | 13.7 | 27.0.619 | 5.1.827 | 2.9.794 | 13.3.513 | 18.9.3 | 29.1.519 | 0.0.992
Mistral-7b-instruct | 50.7 | 78.8.036 | 70.0.043 | 40.2.001 | 36.8.054 | 52.7.001 | 65.2.0 | 11.7.078
Llama-2-13b-chat | 33.7 | 56.1.069 | 62.7.048 | 10.1.508 | 4.6.785 | 37.3.024 | 63.1.029 | 1.8.797
Table 6: Zero-shot evaluation results for the ICL on the test dataset. The
subscript indicates a fraction of the test set with invalid predictions. We
only provide results for templates with a minimal number of invalid
predictions. Highlighted scores (orange) denote the same performance as the
Constant Classifier.

##### Zero-shot

Table 6 presents a comparative analysis of zero-shot in-context learning
performance across various language models on different classification tasks,
revealing significant disparities. GPT-4 emerges as the clear leader, boasting
the highest average performance of 65.9 points and leading in every individual
task. Following GPT-4, GPT-3.5 shows solid performance with an overall average
of 55.4. Models like Bison-text, Bielik-7b-instruct, and Mistral-7b-instruct
demonstrate competitive capabilities, with averages around 52.2, 51.5, and
50.7 respectively, while Krakowiak-7b-v2 shows variable performance, excelling
in specific tasks but having a lower overall average. Despite Krakowiak’s
commendable scores, its performance on tasks such as PAC, CBD, and CDSC-E is
akin to that of a constant classifier, highlighting limitations in its
predictive diversity. The markedly lower performance of older models like
GPT-2-xl, with an average score of 13.7 points, illustrates the rapid
evolution and significant enhancements in model capabilities. Llama-2-13b-chat
has the lowest average of 33.7 among a few billion parameter models.

Our study shows that in the second group of models, the ones made especially
for Polish work better. Notably, Bielik-7b-instruct, a derivative of the
Mistral-7B-v0.1 architecture fine-tuned on Polish instructions, demonstrates
enhanced capabilities. Mistral-7b-instruct, which is fine-tuned in English is
slightly worse. However, Krakowiak-7b-v2, which also derives from
Mistral-7b-v0.1, does not outperform Mistral-7b-instruct. This suggests that
continual fine-tuning on the target language must be approached with caution.

The second-best Polish-based model is Trurl-2-13b, which is built on the
Llama-2-13b framework. A comparative analysis with Llama-2-13b-chat from the
third cohort illustrates the effectiveness of fine-tuning on a Polish corpus,
leading to improved overall metrics and a reduction in erroneous predictions.
This underscores the significance of targeted fine-tuning in enhancing model
performance in language-specific applications.

####  5.3.1 Scaling for Llama

Figure 5 depicts the average performance across different scales within the
Llama-2 model family, underscoring the influence of instruction fine-tuning on
performance metrics. In the zero-shot paradigm, instruction fine-tuning
augments performance, with larger models yielding enhanced results.
Conversely, within the few-shot learning framework ($n=16$), the advantages of
model scaling are not evident. Furthermore, in this few-shot context,
instruction fine-tuning seems to negatively impact the outcomes. This is
consistent with observations made in (Ahuja et al., 2023; Asai et al., 2023).

Figure 5: Average performance (AVG) for Llama-2 family models. The upper
figure denotes the zero-shot performance. The lower figure denotes a few-shot
performance with 16 shots. The text model is trained with language modeling on
the general corpus. The chat are continually trained on instruction datasets.

##  6 Conclusion

In this study, we introduced a few-shot benchmark for the Polish language,
consisting of 7 different classification tasks. We conducted an empirical
comparison between models trained with linear probing, fine-tuning, SetFit,
and in-context learning. Our results show that ICL achieves the best
performance for both 0 and 16 shots. However, there is a significant 14 pp gap
between the zero-shot performance of GPT-4 and HerBERT-large trained on the
entire dataset.

Our findings indicate that SetFit is the most effective fine-tuning technique,
allowing the use of much smaller models compared to ICL. Nevertheless, there
is a substantial 12.4 pp performance difference between SBERT-large and
ChatGPT for 16 shots. The least effective method turned out to be simple fine-
tuning with a non-linear classification head, while better results were
achieved using linear probing on top of frozen embeddings.

Interestingly, we observed the highest variability in results among different
LLMs employing ICL. Commercial models like PaLM-2, GPT-3.5, and GPT-4 show
excellent comprehension of Polish and outperform open-source models. However,
Mistral-7b-instruct also performs exceptionally well, despite not being
specifically trained on Polish corpora. The same cannot be said for Llama-2.

Our tests confirm that continual pre-training of Mistral or Llama-2 on Polish
corpora yields better zero-shot results in our benchmark. Both Trurl-13b and
Bielik-7b-instruct perform better than Llama-2-13b-chat and
Mistral-7b-instruct. However, in the few-shot regime, only Trurl-2-13b is
better than its English counterpart. Additionally, the example of
Krakowiak-7b, which was fine-tuned from Mistral-7b, demonstrates that fine-
tuning must be approached with care to avoid degrading performance.

##  7 Future work

In the future, we plan to examine the generation quality of LLMs, including
whether they use Polish, English, or a mixture of both in their responses. We
also intend to evaluate additional open-source LLMs and integrate a wider
range of tasks into our benchmark, beyond classification. Furthermore,
assessing the performance of auto-prompting techniques and comparing PEFT with
prompt engineering using LLMs in the few-shot regime will be of interest.

Additionally, our findings prompt intriguing inquiries. Why does
Mistral-7b-instruct perform so well in Polish, even though it was not
specifically trained on a large-scale Polish corpus? This contributes to the
broader research on adapting pre-trained LLMs to non-English languages (Zhao
et al., 2024; Zhu et al., 2023; Chen et al., 2024; Csaki et al., 2023; Rostami
et al., 2024; Basile et al., 2023) and in-context cross-lingual transfer
(Etxaniz et al., 2023; Zhang et al., 2023; Lai et al., 2023; Asai et al.,
2023; Ahuja et al., 2023; Bandarkar et al., 2023).

##  8 Limitations

Our goal was to develop an evaluation methodology that reflects real-world
situations and defines the maximum potential of few-shot learning for tasks in
the Polish language. Previous studies (Perez et al., 2021; Alex et al., 2022)
have shown that using a validation dataset for hyperparameter tuning may
overestimate the capabilities of "true few-shot learning," with results
varying greatly depending on the chosen seed or template. The number of few-
shot examples was quite small and unevenly distributed across tasks. We have
datasets ranging from 8 examples per class (PAC, DYK) to 1 (CST-Wikinews),
affecting the interpretation of results and leading to potential label biases.

Although automatic evaluation metrics are widely used and convenient for
assessing models, human evaluation remains the definitive standard for certain
tasks. In this study, we used two metrics influenced by automatic evaluation.
For the ICL, we considered the label name as the definitive truth, ignoring
synonyms or terms with similar meanings within the task. For greedy decoding,
we applied exact match metrics with minimal preprocessing. Our analysis
indicates that this method is unlikely to significantly affect the results,
but the challenge of automatic evaluation remains, particularly with
generative LLMs.

Our benchmark uses diverse, classification-only datasets to measure model
performance in different areas, including datasets of varying lengths and
number of classes. However, it is limited to open-source Polish datasets,
introducing several challenges. For instance, the CBD dataset had noisy data
and low quality, while the clarity of the CST-Wikinews dataset was
questionable, even to the original authors. Additionally, the availability of
datasets in Polish is much less than in English, limiting our ability to
create a comprehensive benchmark.

The list of few-shot techniques in the NLP is extensive and continues to grow
each year. We selectively chose a few based on ease of use, recognition,
potential benefit, and computational effort required. For instance, we did not
perform any text-to-text fine-tuning with LLMs, which might have yielded
better results than prompt engineering. Similarly, the number of open-source
LLMs suitable for ICL is increasing rapidly.

##  9 Reproducibility

Publicly available datasets were utilized to establish the benchmark, with
references to these datasets delineated in the Appendix B. The models employed
in this investigation include both open-source variants and those accessible
via paid subscriptions through their respective APIs (see Appendix D). The
provision of commercial models was facilitated through platforms such as
Google and Azure, with specific versions of these models being fixed for the
duration of the study. To ensure experimental consistency, fixed random seeds
were used, and the output generation from the language models was performed at
a temperature setting of 0 to maintain deterministic behavior (see Appendix
E). Additionally, the values of the hyperparameters used for fine-tuning
experiments are provided in Appendix C.

## Acknowledgements

We express our gratitude to Allegro.pl Sp. z o.o. for providing the funding
for computational resources.

All content represents the opinion of the authors, which is not necessarily
shared or endorsed by their respective employers.

## References

  * Ahuja et al. (2023) Kabir Ahuja, Harshita Diddee, Rishav Hada, Millicent Ochieng, Krithika Ramesh, Prachi Jain, Akshay Nambi, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram. 2023.  Mega: Multilingual evaluation of generative ai.
  * Albalak et al. (2023) Alon Albalak, Colin Raffel, and William Yang Wang. 2023.  Improving few-shot generalization by exploring and exploiting auxiliary data.  In _Thirty-seventh Conference on Neural Information Processing Systems_.
  * Alex et al. (2021) Neel Alex, Eli Lifland, Lewis Tunstall, Abhishek Thakur, Pegah Maham, C. Jess Riedel, Emmie Hine, Carolyn Ashurst, Paul Sedille, Alexis Carlier, Michael Noetel, and Andreas Stuhlmüller. 2021.  RAFT: A real-world few-shot text classification benchmark.  In _Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2)_.
  * Alex et al. (2022) Neel Alex, Eli Lifland, Lewis Tunstall, Abhishek Thakur, Pegah Maham, C. Jess Riedel, Emmie Hine, Carolyn Ashurst, Paul Sedille, Alexis Carlier, Michael Noetel, and Andreas Stuhlmüller. 2022.  Raft: A real-world few-shot text classification benchmark.
  * Aly et al. (2023) Rami Aly, Xingjian Shi, Kaixiang Lin, Aston Zhang, and Andrew Wilson. 2023.  Automated few-shot classification with instruction-finetuned language models.  In _Findings of the Association for Computational Linguistics: EMNLP 2023_ , pages 2414–2432, Singapore. Association for Computational Linguistics.
  * An et al. (2023) Gaon An, Junhyeok Lee, Xingdong Zuo, Norio Kosaka, Kyung-Min Kim, and Hyun Oh Song. 2023.  Direct preference-based policy optimization without reward modeling.
  * Anonymous (2024a) Anonymous. 2024a.  Connecting large language models with evolutionary algorithms yields powerful prompt optimizers.  In _The Twelfth International Conference on Learning Representations_.
  * Anonymous (2024b) Anonymous. 2024b.  Large language models as optimizers.  In _The Twelfth International Conference on Learning Representations_.
  * Artetxe et al. (2020) Mikel Artetxe, Gorka Labaka, and Eneko Agirre. 2020.  Translation artifacts in cross-lingual transfer learning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7674–7684, Online. Association for Computational Linguistics.
  * Asai et al. (2023) Akari Asai, Sneha Kudugunta, Xinyan Velocity Yu, Terra Blevins, Hila Gonen, Machel Reid, Yulia Tsvetkov, Sebastian Ruder, and Hannaneh Hajishirzi. 2023.  Buffet: Benchmarking large language models for few-shot cross-lingual transfer.
  * Augustyniak et al. (2022) Lukasz Augustyniak, Kamil Tagowski, Albert Sawczyn, Denis Janiak, Roman Bartusiak, Adrian Szymczak, Arkadiusz Janz, Piotr Szymański, Marcin Wątroba, Mikoł aj Morzy, Tomasz Kajdanowicz, and Maciej Piasecki. 2022.  This is the way: designing and compiling lepiszcze, a comprehensive nlp benchmark for polish.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 21805–21818. Curran Associates, Inc.
  * Bach et al. (2022) Stephen Bach, Victor Sanh, Zheng Xin Yong, Albert Webson, Colin Raffel, Nihal V. Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, Zaid Alyafeai, Manan Dey, Andrea Santilli, Zhiqing Sun, Srulik Ben-david, Canwen Xu, Gunjan Chhablani, Han Wang, Jason Fries, Maged Al-shaibani, Shanya Sharma, Urmish Thakker, Khalid Almubarak, Xiangru Tang, Dragomir Radev, Mike Tian-jian Jiang, and Alexander Rush. 2022.  PromptSource: An integrated development environment and repository for natural language prompts.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics: System Demonstrations_ , pages 93–104, Dublin, Ireland. Association for Computational Linguistics.
  * Bandarkar et al. (2023) Lucas Bandarkar, Davis Liang, Benjamin Muller, Mikel Artetxe, Satya Narayan Shukla, Donald Husa, Naman Goyal, Abhinandan Krishnan, Luke Zettlemoyer, and Madian Khabsa. 2023.  The belebele benchmark: a parallel reading comprehension dataset in 122 language variants.
  * Basile et al. (2023) Pierpaolo Basile, Elio Musacchio, Marco Polignano, Lucia Siciliani, Giuseppe Fiameni, and Giovanni Semeraro. 2023.  Llamantino: Llama 2 models for effective text generation in italian language.
  * Bragg et al. (2021) Jonathan Bragg, Arman Cohan, Kyle Lo, and Iz Beltagy. 2021.  FLEX: Unifying evaluation for few-shot NLP.  In _Advances in Neural Information Processing Systems_.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Chen et al. (2024) Pinzhen Chen, Shaoxiong Ji, Nikolay Bogoychev, Andrey Kutuzov, Barry Haddow, and Kenneth Heafield. 2024.  Monolingual or multilingual instruction tuning: Which makes a better alpaca.  In _Findings of the Association for Computational Linguistics: EACL 2024_ , pages 1347–1356, St. Julian’s, Malta. Association for Computational Linguistics.
  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  XGBoost: A scalable tree boosting system.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’16, pages 785–794, New York, NY, USA. ACM.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022\.  Palm: Scaling language modeling with pathways.
  * Christiano et al. (2023) Paul Christiano, Jan Leike, Tom B. Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2023.  Deep reinforcement learning from human preferences.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang, Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling instruction-finetuned language models.
  * Csaki et al. (2023) Zoltan Csaki, Pian Pawakapan, Urmish Thakker, and Qiantong Xu. 2023.  Efficiently adapting pretrained language models to new languages.
  * Deng et al. (2022) Mingkai Deng, Jianyu Wang, Cheng-Ping Hsieh, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric Xing, and Zhiting Hu. 2022.  RLPrompt: Optimizing discrete text prompts with reinforcement learning.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 3369–3391, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Etxaniz et al. (2023) Julen Etxaniz, Gorka Azkune, Aitor Soroa, Oier Lopez de Lacalle, and Mikel Artetxe. 2023.  Do multilingual language models think better in english?
  * Fedus et al. (2022) William Fedus, Barret Zoph, and Noam Shazeer. 2022.  Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.
  * Fernando et al. (2023) Chrisantha Fernando, Dylan Banarse, Henryk Michalewski, Simon Osindero, and Tim Rocktäschel. 2023.  Promptbreeder: Self-referential self-improvement via prompt evolution.
  * Honnibal and Montani (2017) Matthew Honnibal and Ines Montani. 2017.  spaCy 2: Natural language understanding with Bloom embeddings, convolutional neural networks and incremental parsing.  To appear.
  * Hu et al. (2022) Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2022.  LoRA: Low-rank adaptation of large language models.  In _International Conference on Learning Representations_.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7b.
  * Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024\.  Mixtral of experts.
  * Karimi Mahabadi et al. (2022) Rabeeh Karimi Mahabadi, Luke Zettlemoyer, James Henderson, Lambert Mathias, Marzieh Saeidi, Veselin Stoyanov, and Majid Yazdani. 2022.  Prompt-free and efficient few-shot learning with language models.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3638–3652, Dublin, Ireland. Association for Computational Linguistics.
  * Kwon et al. (2023) Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. 2023.  Efficient memory management for large language model serving with pagedattention.  In _Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles_.
  * Lai et al. (2023) Viet Dac Lai, Nghia Trung Ngo, Amir Pouran Ben Veyseh, Hieu Man, Franck Dernoncourt, Trung Bui, and Thien Huu Nguyen. 2023.  Chatgpt beyond english: Towards a comprehensive evaluation of large language models in multilingual learning.
  * Lin et al. (2020) Tsung-Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. 2020.  Focal loss for dense object detection.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , 42(2):318–327.
  * Liu et al. (2022) Haokun Liu, Derek Tam, Muqeeth Mohammed, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. 2022.  Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.  In _Advances in Neural Information Processing Systems_.
  * Logan IV et al. (2022) Robert Logan IV, Ivana Balazevic, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2022.  Cutting down on prompts and parameters: Simple few-shot learning with language models.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 2824–2835, Dublin, Ireland. Association for Computational Linguistics.
  * Mishra et al. (2022) Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022.  Cross-task generalization via natural language crowdsourcing instructions.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 3470–3487, Dublin, Ireland. Association for Computational Linguistics.
  * Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023\.  Crosslingual generalization through multitask finetuning.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 15991–16111, Toronto, Canada. Association for Computational Linguistics.
  * OpenAI et al. (2024) OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2024.  Gpt-4 technical report.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.
  * Perez et al. (2021) Ethan Perez, Douwe Kiela, and Kyunghyun Cho. 2021.  True few-shot learning with language models.
  * Ponti et al. (2020) Edoardo Maria Ponti, Goran Glavaš, Olga Majewska, Qianchu Liu, Ivan Vulić, and Anna Korhonen. 2020.  XCOPA: A multilingual dataset for causal commonsense reasoning.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 2362–2376, Online. Association for Computational Linguistics.
  * Prasad et al. (2023) Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. 2023.  GrIPS: Gradient-free, edit-based instruction search for prompting large language models.  In _Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics_ , pages 3845–3864, Dubrovnik, Croatia. Association for Computational Linguistics.
  * Pryzant et al. (2023) Reid Pryzant, Dan Iter, Jerry Li, Yin Lee, Chenguang Zhu, and Michael Zeng. 2023\.  Automatic prompt optimization with “gradient descent” and beam search.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 7957–7968, Singapore. Association for Computational Linguistics.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Rae et al. (2022) Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving. 2022.  Scaling language models: Methods, analysis & insights from training gopher.
  * Rafailov et al. (2023) Rafael Rafailov, Archit Sharma, Eric Mitchell, Stefano Ermon, Christopher D. Manning, and Chelsea Finn. 2023.  Direct preference optimization: Your language model is secretly a reward model.
  * Rostami et al. (2024) Pedram Rostami, Ali Salemi, and Mohammad Javad Dousti. 2024.  Persianmind: A cross-lingual persian-english large language model.
  * Ruder and Sil (2021) Sebastian Ruder and Avi Sil. 2021.  Multi-domain multilingual question answering.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: Tutorial Abstracts_ , pages 17–21, Punta Cana, Dominican Republic & Online. Association for Computational Linguistics.
  * Rybak et al. (2020) Piotr Rybak, Robert Mroczkowski, Janusz Tracz, and Ireneusz Gawlik. 2020.  KLEJ: Comprehensive benchmark for Polish language understanding.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 1191–1201, Online. Association for Computational Linguistics.
  * Sanh et al. (2022) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Matteo Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Bawden, Thomas Wang, Trishala Neeraj, Jos Rozen, Abheesht Sharma, Andrea Santilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Tali Bers, Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022.  Multitask prompted training enables zero-shot task generalization.
  * Scao et al. (2023) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, Jonathan Tow, Alexander M. Rush, Stella Biderman, Albert Webson, Pawan Sasanka Ammanamanchi, Thomas Wang, Benoît Sagot, Niklas Muennighoff, Albert Villanova del Moral, Olatunji Ruwase, Rachel Bawden, Stas Bekman, Angelina McMillan-Major, Iz Beltagy, Huu Nguyen, Lucile Saulnier, Samson Tan, Pedro Ortiz Suarez, Victor Sanh, Hugo Laurençon, Yacine Jernite, Julien Launay, Margaret Mitchell, Colin Raffel, Aaron Gokaslan, Adi Simhi, Aitor Soroa, Alham Fikri Aji, Amit Alfassy, Anna Rogers, Ariel Kreisberg Nitzav, Canwen Xu, Chenghao Mou, Chris Emezue, Christopher Klamm, Colin Leong, Daniel van Strien, David Ifeoluwa Adelani, Dragomir Radev, Eduardo González Ponferrada, Efrat Levkovizh, Ethan Kim, Eyal Bar Natan, Francesco De Toni, Gérard Dupont, Germán Kruszewski, Giada Pistilli, Hady Elsahar, Hamza Benyamina, Hieu Tran, Ian Yu, Idris Abdulmumin, Isaac Johnson, Itziar Gonzalez-Dios, Javier de la Rosa, Jenny Chim, Jesse Dodge, Jian Zhu, Jonathan Chang, Jörg Frohberg, Joseph Tobing, Joydeep Bhattacharjee, Khalid Almubarak, Kimbo Chen, Kyle Lo, Leandro Von Werra, Leon Weber, Long Phan, Loubna Ben allal, Ludovic Tanguy, Manan Dey, Manuel Romero Muñoz, Maraim Masoud, María Grandury, Mario Šaško, Max Huang, Maximin Coavoux, Mayank Singh, Mike Tian-Jian Jiang, Minh Chien Vu, Mohammad A. Jauhar, Mustafa Ghaleb, Nishant Subramani, Nora Kassner, Nurulaqilla Khamis, Olivier Nguyen, Omar Espejel, Ona de Gibert, Paulo Villegas, Peter Henderson, Pierre Colombo, Priscilla Amuok, Quentin Lhoest, Rheza Harliman, Rishi Bommasani, Roberto Luis López, Rui Ribeiro, Salomey Osei, Sampo Pyysalo, Sebastian Nagel, Shamik Bose, Shamsuddeen Hassan Muhammad, Shanya Sharma, Shayne Longpre, Somaieh Nikpoor, Stanislav Silberberg, Suhas Pai, Sydney Zink, Tiago Timponi Torrent, Timo Schick, Tristan Thrush, Valentin Danchev, Vassilina Nikoulina, Veronika Laippala, Violette Lepercq, Vrinda Prabhu, Zaid Alyafeai, Zeerak Talat, Arun Raja, Benjamin Heinzerling, Chenglei Si, Davut Emre Taşar, Elizabeth Salesky, Sabrina J. Mielke, Wilson Y. Lee, Abheesht Sharma, Andrea Santilli, Antoine Chaffin, Arnaud Stiegler, Debajyoti Datta, Eliza Szczechla, Gunjan Chhablani, Han Wang, Harshit Pandey, Hendrik Strobelt, Jason Alan Fries, Jos Rozen, Leo Gao, Lintang Sutawika, M Saiful Bari, Maged S. Al-shaibani, Matteo Manica, Nihal Nayak, Ryan Teehan, Samuel Albanie, Sheng Shen, Srulik Ben-David, Stephen H. Bach, Taewoon Kim, Tali Bers, Thibault Fevry, Trishala Neeraj, Urmish Thakker, Vikas Raunak, Xiangru Tang, Zheng-Xin Yong, Zhiqing Sun, Shaked Brody, Yallow Uri, Hadar Tojarieh, Adam Roberts, Hyung Won Chung, Jaesung Tae, Jason Phang, Ofir Press, Conglong Li, Deepak Narayanan, Hatim Bourfoune, Jared Casper, Jeff Rasley, Max Ryabinin, Mayank Mishra, Minjia Zhang, Mohammad Shoeybi, Myriam Peyrounette, Nicolas Patry, Nouamane Tazi, Omar Sanseviero, Patrick von Platen, Pierre Cornette, Pierre François Lavallée, Rémi Lacroix, Samyam Rajbhandari, Sanchit Gandhi, Shaden Smith, Stéphane Requena, Suraj Patil, Tim Dettmers, Ahmed Baruwa, Amanpreet Singh, Anastasia Cheveleva, Anne-Laure Ligozat, Arjun Subramonian, Aurélie Névéol, Charles Lovering, Dan Garrette, Deepak Tunuguntla, Ehud Reiter, Ekaterina Taktasheva, Ekaterina Voloshina, Eli Bogdanov, Genta Indra Winata, Hailey Schoelkopf, Jan-Christoph Kalo, Jekaterina Novikova, Jessica Zosa Forde, Jordan Clive, Jungo Kasai, Ken Kawamura, Liam Hazan, Marine Carpuat, Miruna Clinciu, Najoung Kim, Newton Cheng, Oleg Serikov, Omer Antverg, Oskar van der Wal, Rui Zhang, Ruochen Zhang, Sebastian Gehrmann, Shachar Mirkin, Shani Pais, Tatiana Shavrina, Thomas Scialom, Tian Yun, Tomasz Limisiewicz, Verena Rieser, Vitaly Protasov, Vladislav Mikhailov, Yada Pruksachatkun, Yonatan Belinkov, Zachary Bamberger, Zdeněk Kasner, Alice Rueda, Amanda Pestana, Amir Feizpour, Ammar Khan, Amy Faranak, Ana Santos, Anthony Hevia, Antigona Unldreaj, Arash Aghagol, Arezoo Abdollahi, Aycha Tammour, Azadeh HajiHosseini, Bahareh Behroozi, Benjamin Ajibade, Bharat Saxena, Carlos Muñoz Ferrandis, Daniel McDuff, Danish Contractor, David Lansky, Davis David, Douwe Kiela, Duong A. Nguyen, Edward Tan, Emi Baylor, Ezinwanne Ozoani, Fatima Mirza, Frankline Ononiwu, Habib Rezanejad, Hessie Jones, Indrani Bhattacharya, Irene Solaiman, Irina Sedenko, Isar Nejadgholi, Jesse Passmore, Josh Seltzer, Julio Bonis Sanz, Livia Dutra, Mairon Samagaio, Maraim Elbadri, Margot Mieskes, Marissa Gerchick, Martha Akinlolu, Michael McKenna, Mike Qiu, Muhammed Ghauri, Mykola Burynok, Nafis Abrar, Nazneen Rajani, Nour Elkott, Nour Fahmy, Olanrewaju Samuel, Ran An, Rasmus Kromann, Ryan Hao, Samira Alizadeh, Sarmad Shubber, Silas Wang, Sourav Roy, Sylvain Viguier, Thanh Le, Tobi Oyebade, Trieu Le, Yoyo Yang, Zach Nguyen, Abhinav Ramesh Kashyap, Alfredo Palasciano, Alison Callahan, Anima Shukla, Antonio Miranda-Escalada, Ayush Singh, Benjamin Beilharz, Bo Wang, Caio Brito, Chenxi Zhou, Chirag Jain, Chuxin Xu, Clémentine Fourrier, Daniel León Periñán, Daniel Molano, Dian Yu, Enrique Manjavacas, Fabio Barth, Florian Fuhrimann, Gabriel Altay, Giyaseddin Bayrak, Gully Burns, Helena U. Vrabec, Imane Bello, Ishani Dash, Jihyun Kang, John Giorgi, Jonas Golde, Jose David Posada, Karthik Rangasai Sivaraman, Lokesh Bulchandani, Lu Liu, Luisa Shinzato, Madeleine Hahn de Bykhovetz, Maiko Takeuchi, Marc Pàmies, Maria A Castillo, Marianna Nezhurina, Mario Sänger, Matthias Samwald, Michael Cullan, Michael Weinberg, Michiel De Wolf, Mina Mihaljcic, Minna Liu, Moritz Freidank, Myungsun Kang, Natasha Seelam, Nathan Dahlberg, Nicholas Michio Broad, Nikolaus Muellner, Pascale Fung, Patrick Haller, Ramya Chandrasekhar, Renata Eisenberg, Robert Martin, Rodrigo Canalli, Rosaline Su, Ruisi Su, Samuel Cahyawijaya, Samuele Garda, Shlok S Deshmukh, Shubhanshu Mishra, Sid Kiblawi, Simon Ott, Sinee Sang-aroonsiri, Srishti Kumar, Stefan Schweter, Sushil Bharati, Tanmay Laud, Théo Gigant, Tomoya Kainuma, Wojciech Kusa, Yanis Labrak, Yash Shailesh Bajaj, Yash Venkatraman, Yifan Xu, Yingxin Xu, Yu Xu, Zhe Tan, Zhongli Xie, Zifan Ye, Mathilde Bras, Younes Belkada, and Thomas Wolf. 2023.  Bloom: A 176b-parameter open-access multilingual language model.
  * Schaeffer et al. (2023) Rylan Schaeffer, Brando Miranda, and Sanmi Koyejo. 2023.  Are emergent abilities of large language models a mirage?
  * Schick and Schütze (2021a) Timo Schick and Hinrich Schütze. 2021a.  Exploiting cloze-questions for few-shot text classification and natural language inference.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 255–269, Online. Association for Computational Linguistics.
  * Schick and Schütze (2021b) Timo Schick and Hinrich Schütze. 2021b.  It’s not just size that matters: Small language models are also few-shot learners.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 2339–2352, Online. Association for Computational Linguistics.
  * Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. 2017.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.
  * Shin et al. (2020) Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020.  AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 4222–4235, Online. Association for Computational Linguistics.
  * Tam et al. (2021) Derek Tam, Rakesh R. Menon, Mohit Bansal, Shashank Srivastava, and Colin Raffel. 2021.  Improving and simplifying pattern exploiting training.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4980–4991, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Tay et al. (2023) Yi Tay, Mostafa Dehghani, Vinh Q. Tran, Xavier Garcia, Jason Wei, Xuezhi Wang, Hyung Won Chung, Siamak Shakeri, Dara Bahri, Tal Schuster, Huaixiu Steven Zheng, Denny Zhou, Neil Houlsby, and Donald Metzler. 2023.  Ul2: Unifying language learning paradigms.
  * Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Pier Giuseppe Sessa, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. 2024.  Gemma: Open models based on gemini research and technology.
  * Thoppilan et al. (2022) Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, YaGuang Li, Hongrae Lee, Huaixiu Steven Zheng, Amin Ghafouri, Marcelo Menegali, Yanping Huang, Maxim Krikun, Dmitry Lepikhin, James Qin, Dehao Chen, Yuanzhong Xu, Zhifeng Chen, Adam Roberts, Maarten Bosma, Vincent Zhao, Yanqi Zhou, Chung-Ching Chang, Igor Krivokon, Will Rusch, Marc Pickett, Pranesh Srinivasan, Laichee Man, Kathleen Meier-Hellstern, Meredith Ringel Morris, Tulsee Doshi, Renelito Delos Santos, Toju Duke, Johnny Soraker, Ben Zevenbergen, Vinodkumar Prabhakaran, Mark Diaz, Ben Hutchinson, Kristen Olson, Alejandra Molina, Erin Hoffman-John, Josh Lee, Lora Aroyo, Ravi Rajakumar, Alena Butryna, Matthew Lamm, Viktoriya Kuzmina, Joe Fenton, Aaron Cohen, Rachel Bernstein, Ray Kurzweil, Blaise Aguera-Arcas, Claire Cui, Marian Croak, Ed Chi, and Quoc Le. 2022.  Lamda: Language models for dialog applications.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.
  * Tunstall et al. (2022) Lewis Tunstall, Nils Reimers, Unso Eun Seo Jo, Luke Bates, Daniel Korat, Moshe Wasserblat, and Oren Pereg. 2022.  Efficient few-shot learning without prompts.
  * Wang et al. (2022) Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasekaran, Anjana Arunkumar, David Stap, Eshaan Pathak, Giannis Karamanolakis, Haizhi Lai, Ishan Purohit, Ishani Mondal, Jacob Anderson, Kirby Kuznia, Krima Doshi, Kuntal Kumar Pal, Maitreya Patel, Mehrad Moradshahi, Mihir Parmar, Mirali Purohit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A, Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022.  Super-NaturalInstructions: Generalization via declarative instructions on 1600+ NLP tasks.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 5085–5109, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Wei et al. (2022a) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2022a.  Finetuned language models are zero-shot learners.
  * Wei et al. (2022b) Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022b.  Emergent abilities of large language models.
  * Ye et al. (2021) Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021.  CrossFit: A few-shot learning challenge for cross-task generalization in NLP.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 7163–7189, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhang et al. (2023) Xiang Zhang, Senyu Li, Bradley Hauer, Ning Shi, and Grzegorz Kondrak. 2023.  Don’t trust ChatGPT when your question is not in English: A study of multilingual abilities and types of LLMs.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 7915–7927, Singapore. Association for Computational Linguistics.
  * Zhao et al. (2024) Jun Zhao, Zhihao Zhang, Luhui Gao, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024.  Llama beyond english: An empirical study on language capability transfer.
  * Zheng et al. (2022) Yanan Zheng, Jing Zhou, Yujie Qian, Ming Ding, Chonghua Liao, Li Jian, Ruslan Salakhutdinov, Jie Tang, Sebastian Ruder, and Zhilin Yang. 2022.  FewNLU: Benchmarking state-of-the-art methods for few-shot natural language understanding.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 501–516, Dublin, Ireland. Association for Computational Linguistics.
  * Zhou et al. (2023) Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. 2023.  Large language models are human-level prompt engineers.  In _The Eleventh International Conference on Learning Representations_.
  * Zhu et al. (2023) Wenhao Zhu, Yunzhe Lv, Qingxiu Dong, Fei Yuan, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li. 2023.  Extrapolating large language models to non-english by aligning languages.

##  Appendix A Table with the results from all experiments

This table presents collected in one place results from all experiments
mentioned in the paper. Results for baseline HerBERT-large in red color
denotes results reported on KLEJ benchmark (and they are strongly differ with
our experiments). For ICL zero-shot experiments subscript indicates a fraction
of the test set with invalid predictions. For experiments with "±" in
subscript, we use subscript to denote standard deviation.

|  PAC |  Polemo2 |  CBD |  NKJP-NER |  DYK |  CDSC-E |  CST-Wikinews
---|---|---|---|---|---|---|---
Baselines
XGBoost | 56.4 ± 16.7 | 38.5 ± 3.4 | 19.7 ± 6.0 | 20.9 ± 4.7 | 22.6 ± 5.1 | 57.5 ± 10.4 | 11.4 ± 1.5
Random Guessing | 57.5 ± 0.7 | 24.8 ± 1.0 | 20.9 ± 1.6 | 16.8 ± 0.8 | 25.5 ± 1.7 | 33.6 ± 0.7 | 6.9 ± 0.9
Constant Classifier | 80.6 | 41.3 | 23.6 | 34.3 | 28.9 | 74.4 | 16.9
HerBERT-large | 91.1 ± 0.0 | 90.9 ± 0.0 | 53.2 ± 3.2 (72.0) | 94.0 ± 0.0 | 68.8 ± 2.1 (75.8) | 93.4 ± 0.0 | 67.9 ± 1.0
Linear probing
SBERT-large | 67.5 ± 7.0 | 60.3 ± 4.6 | 40.1 ± 4.3 | 30.4 ± 5.9 | 27.2 ± 1.8 | 62.8 ± 5.8 | 15.3 ± 3.0
HerBERT-large | 72.6 ± 10.3 | 47.0 ± 3.1 | 33.8 ± 10.1 | 27.8 ± 3.3 | 34.6 ± 2.8 | 71.0 ± 6.1 | 14.2 ± 2.7
RoBERTa-large | 67.8 ± 11.3 | 57.9 ± 4.4 | 37.4 ± 6.1 | 27.6 ± 3.9 | 25.1 ± 3.0 | 59.2 ± 7.5 | 14.5 ± 2.7
GPT-2-large | 72.0 ± 2.9 | 48.3 ± 4.6 | 36.4 ± 4.1 | 25.2 ± 4.3 | 28.6 ± 3.1 | 54.4 ± 7.5 | 13.6 ± 3.0
Llama-2-70b* | 67.6 ± 12.5 | 41.4 ± 2.9 | 28.4 ± 2.5 | 24.2 ± 2.0 | 30.7 ± 3.0 | 61.1 ± 6.1 | 13.5 ± 4.2
Ada | 72.9 ± 6.3 | 55.2 ± 5.0 | 30.7 ± 4.2 | 29.1 ± 2.5 | 25.4 ± 4.1 | 58.2 ± 11.7 | 14.8 ± 3.9
DaVinci | 67.6 ± 7.0 | 58.9 ± 1.3 | 36.9 ± 9.7 | 30.1 ± 6.6 | 29.9 ± 2.7 | 60.1 ± 11.2 | 15.5 ± 3.5
Gecko | 61.8 ± 4.5 | 42.9 ± 6.4 | 23.7 ± 1.3 | 24.8 ± 4.3 | 25.0 ± 3.8 | 68.8 ± 6.3 | 17.0 ± 2.8
Gecko multilingual | 72.2 ± 5.6 | 60.8 ± 2.6 | 41.0 ± 3.3 | 28.4 ± 2.8 | 27.4 ± 2.3 | 62.1 ± 12.6 | 16.2 ± 5.7
Fine-tuning with non-linear head
SBERT-large | 33.6 ± 31.3 | 47.0 ± 5.8 | 32.0 ± 30.7 | 28.1 ± 7.7 | 6.4 ± 10.3 | 61.0 ± 17.6 | 10.1 ± 4.3
HerBERT-large | 26.7 ± 23.4 | 29.5 ± 9.7 | 30.8 ± 36.6 | 25.2 ± 3.8 | 16.2 ± 29.7 | 44.4 ± 26.7 | 8.7 ± 3.7
RoBERTa-large | 6.2 ± 11.5 | 36.4 ± 4.3 | 15.3 ± 10.5 | 21.3 ± 6.1 | 27.2 ± 0.4 | 54.5 ± 26.0 | 8.7 ± 2.8
SetFit
SBERT-large | 68.8 ± 6.5 | 69.9 ± 10.7 | 44.4 ± 5.4 | 30.7 ± 6.5 | 27.7 ± 2.9 | 72.3 ± 5.7 | 16.2 ± 3.2
HerBERT-large | 70.7 ± 11.9 | 46.0 ± 11.2 | 42.5 ± 10.9 | 25.8 ± 11.6 | 40.6 ± 4.7 | 67.2 ± 14.5 | 15.7 ± 1.9
In-context learning
/ICLn=0/ GPT-4 | 83.3.0 | 82.6.009 | 60.6.028 | 58.8.004 | 81.3.002 | 76.0.0 | 18.8.003
/ICLn=0/ GPT-3.5 | 82.2.014 | 81.6.005 | 50.0.046 | 44.9.001 | 53.1.0 | 62.9.0 | 13.3.003
/ICLn=16/ GPT-3.5 | 73.9 ± 3.6 | 81.9 ± 2.1 | 64.1 ± 1.9 | 46.1 ± 2.9 | 64.1 ± 1.8 | 66.7 ± 7.6 | 19.8 ± 2.7
/ICLn=0/ Bison-text | 80.2.006 | 80.7.009 | 42.6.077 | 47.5.027 | 61.6.016 | 35.0.001 | 17.7.003
/ICLn=16/ Bison-text | 80.2.006 | 80.7.009 | 42.6.077 | 47.5.027 | 61.6.016 | 35.0.001 | 17.7.003
/ICLn=0/ Llama-2-7b | 34.6.659 | 12.3.379 | 0.5.744 | 14.0.310 | 27.5.014 | 6.6.0 | 3.9.409
/ICLn=16/ Llama-2-7b | 52.8 ± 13.7 | 0.0 ± 0.0 | 20.5 ± 2.5 | 11.8 ± 8.3 | 31.9 ± 1.9 | 26.2 ± 28.0 | 2.1 ± 0.0
/ICLn=0/ Llama-2-13b | 37.1.606 | 8.9.54 | 1.0.686 | 13.2.225 | 17.2.136 | 6.6.003 | 4.7.036
/ICLn=16/ Llama-2-13b | 38.9 ± 24.5 | 0.0 ± 0.0 | 15.1 ± 6.5 | 0.7 ± 0.5 | 38.1 ± 4.3 | 34.4 ± 24.0 | 2.1 ± 0.0
/ICLn=0/ Llama-2-70b | 41.8.576 | 11.3.678 | 0.3.662 | 21.4.324 | 19.2.055 | 6.6.0 | 1.6.859
/ICLn=16/ Llama-2-70b | 58.9 ± 9.1 | 0.0 ± 0.0 | 20.7 ± 8.6 | 21.5 ± 10.0 | 40.7 ± 20.1 | 6.6 ± 0.0 | 3.3 ± 1.2
/ICLn=0/ Llama-2-7b-chat | 69.6.146 | 46.7.306 | 7.9.601 | 11.4.276 | 29.1.0 | 8.9.789 | 4.7.388
/ICLn=16/ Llama-2-7b-chat | 30.8 ± 38.2 | 0.0 ± 0.0 | 0.4 ± 1.0 | 1.3 ± 2.7 | 12.7 ± 9.8 | 0.0 ± 0.0 | 0.0 ± 0.0
/ICLn=0/ Llama-2-13b-chat | 56.1.069 | 62.7.048 | 10.1.508 | 4.6.785 | 37.3.024 | 63.1.029 | 1.8.797
/ICLn=16/ Llama-2-13b-chat | 47.8 ± 41.7 | 0.0 ± 0.0 | 0.2 ± 0.4 | 3.9 ± 7.7 | 20.0 ± 8.8 | 12.1 ± 18.9 | 1.1 ± 2.4
/ICLn=0/ Llama-2-70b-chat | 73.4.090 | 79.8.002 | 30.8.122 | 37.3.087 | 28.9.0 | 19.8.0 | 17.2.016
/ICLn=16/ Llama-2-70b-chat | 38.4 ± 28.6 | 0.0 ± 0.0 | 10.2 ± 6.4 | 16.7 ± 13.0 | 0.0 ± 0.0 | 5.8 ± 1.9 | 11.5 ± 4.0
/ICLn=0/ Trurl-2-13b | 74.1.064 | 79.5.001 | 50.0.001 | 33.6.038 | 49.7.015 | 50.2.001 | 14.6.083
/ICLn=16/ Trurl-2-13b | 66.7 ± 12.0 | 0.0 ± 0.0 | 12.1 ± 6.6 | 15.9 ± 11.3 | 30.8 ± 1.4 | 26.1 ± 21.8 | 6.3 ± 3.2
/ICLn=0/ Krakowiak-7b-v2 | 80.6.001 | 65.0.018 | 23.6.0 | 16.7.024 | 41.4.002 | 74.4.0 | 6.0.034
/ICLn=16/ Krakowiak-7b-v2 | 57.9 ± 22.5 | 14.4 ± 18.5 | 0.0 ± 0.0 | 13.8 ± 1.0 | 28.9 ± 0.1 | 38.0 ± 24.5 | 4.8 ± 4.7
/ICLn=0/ Bielik-7b-instruct | 78.1.034 | 77.3.0 | 53.9.0 | 31.0.01 | 49.0.001 | 65.3.0 | 6.0.023
/ICLn=16/ Bielik-7b-instruct | 76.1 ± 4.7 | 0.0 ± 0.0 | 23.7 ± 0.1 | 23.2 ± 2.2 | 38.3 ± 3.8 | 55.0 ± 10.7 | 5.4 ± 1.3
/ICLn=0/ Mistral-7b-instruct | 78.8.036 | 70.0.043 | 40.2.001 | 36.8.054 | 52.7.001 | 65.2.0 | 11.7.078
/ICLn=16/ Mistral-7b-instruct | 48.8 ± 6.8 | 73.5 ± 6.1 | 34.4 ± 8.6 | 28.2 ± 1.8 | 45.1 ± 1.6 | 77.6 ± 6.5 | 17.9 ± 2.4
/ICLn=0/ GPT-2-xl | 27.0.619 | 5.1.827 | 2.9.794 | 13.3.513 | 18.9.3 | 29.1.519 | 0.0.992
/ICLn=16/ GPT-2-xl | 24.8 ± 24.7 | 0.0 ± 0.0 | 0.0 ± 0.0 | 14.0 ± 11.2 | 16.9 ± 1.8 | 74.0 ± 0.5 | 5.2 ± 5.7

##  Appendix B Datasets

Table 8 describes the source names of datasets as cataloged within the
HuggingFace datasets repository.

Abbrev. | HuggingFace name
---|---
PAC | `allegro/abusive-clauses-pl`
DYK | `allegro/klej-dyk`
CDSC-E | `allegro/klej-cdsc-e`
Polemo2 | `clarin-pl/polemo2-official`
CBD | `allegro/klej-cbd`
NKJP-NER | `allegro/klej-nkjp-ner`
CST-Wikinews | `clarin-pl/cst-wikinews`
Table 8: Abbreviation and name of the HugginFace equivalent used in this
paper.

##  Appendix C Hyper-parameter tuning

For each task and random seed, we search for the best hyperparameters using
Optuna with Tree-structured Parzen Estimator (TPE). We set up 100 total trials
and 25 warmup trials. The exception was linear probing with logistic
regression where we set 20 trials and 5 warmups. We chose the macro-F1 score
as the target metric for each task. A list of hyperparameters can be found in
Table 9. For fine-tuning with non-linear head, we used a fixed batch size of
$8$ and a single Nvidia V100 GPU. For SetFit we used a single Nvidia A100 80GB
GPU.

Method | Optuna
---|---
XGBoost | `si("n_estimators", 50, 1000, 50)`
`si("max_depth", 4, 15, 1)`
`su("subsample", 0.5, 1.0)`
`sl("learning_rate", 1.0e-07, 1.0)`
`si("vocab_size", 10, 1000, 50)`
Logistic regression | `sc("solver", ["lbfgs", "liblinear", "newton-cg"])`
`sf("C", 1e-05, 100)`
SetFit | `sf("learning_rate", 1e-6, 1e-3, log=True)`
`si("num_epochs", 1, 5)`
`sc("batch_size", [4, 8, 16, 32])`
`sc("num_iterations", [5, 10, 20])`
LoRA | `si("lora_alpha", 8, 16, 4)`
`si("r", 8, 16, 4)`
`su("lora_dropout", 0.0, 0.5)`
Fine-tuning with non-linear head | `sc("lr", [1e-06, 5e-06, 1e-05, 5e-05, 1e-04, 5e-04, 1e-03, 3e-03])`
`sc("weight_decay", [0.0, 1e-04, 1e-03, 1e-02, 1e-01])`
`sc("classifier_dropout", [0.0, 0.1, 0.2, 0.3, 0.4, 0.5])`
`sc("max_epochs", [2, 5, 10, 20, 50, 70, 100])`
`sc("eps", [1e-08, 1e-07, 1e-06, 1e-05, 1e-04])`
Table 9: List of hyperparameters for each training method using Optuna Python
notation. We use short-hand notation: suggest_int (si), suggest_uniform (su),
suggest_loguniform (sl), suggest_categorical (sc), suggest_float (sf).

##  Appendix D Pretrained models

Table 10 summarizes the model naming convention we used in the paper and their
corresponding versions and providers. Commercial models from Google and
Microsoft were accessed via API.

Model | Version | Provider | Language | Context
---|---|---|---|---
SBERT-large | `Voicelab/sbert-large-cased-pl` | HF | PL | 512
HerBERT-large | `allegro/herbert-large-cased` | HF | PL | 512
RoBERTa-large | `sdadas/polish-roberta-large-v2` | HF | PL | 512
GPT-2-xl | `sdadas/polish-gpt2-xl` | HF | PL | 1536
Krakowiak-v2-7b | `szymonrucinski/krakowiak-v2-7b` | HF | PL | 32768
Trurl-13b | `Voicelab/trurl-2-13b` | HF | PL | 4096
Mistral-7b-instruct | `mistralai/Mistral-7B-Instruct-v0.2` | HF | EN | 32768
Bielik-7b-instruct | `speakleash/Bielik-7B-Instruct-v0.1` | HF | PL | 4096
Llama-2-7b | `Llama 2 7b` | Meta | EN | 4096
Llama-2-13b | `Llama 2 13b` | Meta | EN | 4096
Llama-2-70b | `Llama 2 70b` | Meta | EN | 4096
Llama-2-7b-chat | `Llama 2 7b-chat` | Meta | EN | 4096
Llama-2-13b-chat | `Llama 2 13b-chat` | Meta | EN | 4096
Llama-2-70b-chat | `Llama 2 70b-chat` | Meta | EN | 4096
Ada | `text-embedding-ada-002` | MS Azure | EN | 8191
DaVinci | `text-search-davinci-doc-001` | MS Azure | EN | -
GPT-3.5 | `gpt-35-turbo (version=0301)` | MS Azure | Unknown | 18384
GPT-4 | `gpt-4 (version=0613)` | MS Azure | Unknown | 8192
Gecko | `textembedding-gecko@001` | Google | EN | -
Gecko Multilingual | `textembedding-gecko-multilingual@001` | Google | Multi | -
Bison-text | `text-bison@001` | Google | Unknown | 8192
Not covered in the paper |
Trurl-7b | `Voicelab/trurl-2-7b` | HF | PL | 4096
Krakowiak-7b | `szymonrucinski/krakowiak-7b` | HF | PL | 2048
mT0-{small,base, |  |  |  |
large,xl,xxl} | `bigscience/mt0-{x}` | HF | Multi | $\infty$
XGLM-{564M,1.7B, |  |  |  |
2.9B,4.5B,7.5B} | `facebook/xglm-{x}` | HF | Multi | 2048
Gemma-{2b,7b}-it | `google/gemma-{x}-it` | HF | EN | 8192
OpenChat | `openchat/openchat-3.5-0106` | HF | EN | 8192
Mixtral | `mistralai/Mixtral-8x7B-Instruct-v0.1` | HF | EN,FR,IT,ES,DE | 32768
Starling | `berkeley-nest/Starling-LM-7B-alpha` | HF | EN | 8192
DBRX | `databricks/dbrx-instruct` | HF | EN | 32768
Text-DaVinci | `text-davinci-003` | MS Azure | EN | -
Bison-chat | `chat-bison@001` | Google | Unknown | 8192
Table 10: Information on pre-trained models used in the paper or left for
future exploration. The penultimate column indicates which language was used
for final fine-tuning. We are not sure how commercial models were pre-trained
so we left Unknown indicator. Abbreviations: HF - HuggingFace, MS - Microsoft.

##  Appendix E In-context learning

Each LLM under evaluation was configured to operate deterministically with
parameters set to `temperature=0` and `max_tokens=20`, while other parameters
remained at their default values. Access to GPT-3.5 predictions was
facilitated through the guidance 777https://github.com/guidance-
ai/guidance/tree/0.0.64 library, and PaLM 2 predictions were obtained via the
Vertex AI API 888https://github.com/googleapis/python-aiplatform/tree/v1.29.0.
The official repository llama 999https://github.com/facebookresearch/llama was
utilized for interfacing with Llama-2, which had its context length parameter
fixed at 4096 tokens. In comparison, GPT-3.5 and Bison-text were configured
with context lengths of 16,384
101010https://platform.openai.com/docs/models/gpt-3-5 and 8,192
111111https://cloud.google.com/vertex-ai/docs/generative-ai/learn/models
tokens, respectively. Inference for other open-source models was conducted
using the vLLM library Kwon et al. (2023).

Regarding computational resources, the Llama-2-7B model execution was
supported by a single A100 40GB GPU. The Llama-2-13B model required 4 L4 GPUs,
whereas the Llama-2-70B model utilized 8 L4 GPUs, demonstrating the scalable
nature of the hardware deployment in correspondence with the increasing model
size and computational demands.

##  Appendix F Sample prompts

In the following, we illustrate a sample few-shot ($n=2$) prompt utilized for
the NKJP-NER task, employing models such as GPT-4/GPT-3.5, Llama-chat, and
Mistral-instruct. Each model employs special characters to delineate roles
within the prompt. The Krakowiak-v2 and Trurl are based on Mistral and Llama
respectively.

⬇

<|im_start|>system

Rozwiązujesz zadanie klasyfikacji dla języka polskiego.<|im_end|>

<|im_start|>user

Mamy kilka encji: geografia, brak, organizacja, imię, miejsce, czas.

Zdanie "Ma do tego dojść na specjalnym posiedzeniu Rady Miejskiej ,
prawdopodobnie zamkniętym dla gości i mediów ." zawiera encję organizacja

Zdanie "W dole widać i słychać aleje Stryjskiego parku , rąbek stawu błyszczy
, drzewa na przeciwległym wzgórzu czerwienią się od zachodu ." zawiera encję
geografia

Zdanie "Przychodnia dotkliwie odczuła więc jego brak ." zawiera encję
<|im_end|>

<|im_start|>assistant

⬇

[INST] <<SYS>>

Rozwiązujesz zadanie klasyfikacji dla języka polskiego.

<</SYS>>

Mamy kilka encji: geografia, brak, organizacja, imię, miejsce, czas.

Zdanie "Ma do tego dojść na specjalnym posiedzeniu Rady Miejskiej ,
prawdopodobnie zamkniętym dla gości i mediów ." zawiera encję organizacja

Zdanie "W dole widać i słychać aleje Stryjskiego parku , rąbek stawu błyszczy
, drzewa na przeciwległym wzgórzu czerwienią się od zachodu ." zawiera encję
geografia

Zdanie "Przychodnia dotkliwie odczuła więc jego brak ." zawiera encję [/INST]

⬇

<s>[INST] Rozwiązujesz zadanie klasyfikacji dla języka polskiego.

Mamy kilka encji: geografia, brak, organizacja, imię, miejsce, czas.

Zdanie "Ma do tego dojść na specjalnym posiedzeniu Rady Miejskiej ,
prawdopodobnie zamkniętym dla gości i mediów ." zawiera encję organizacja

Zdanie "W dole widać i słychać aleje Stryjskiego parku , rąbek stawu błyszczy
, drzewa na przeciwległym wzgórzu czerwienią się od zachodu ." zawiera encję
geografia

Zdanie "Przychodnia dotkliwie odczuła więc jego brak ." zawiera encję [/INST]

Figure 6: Sample prompt with two examples for the GPT-4/GPT-3.5 models (top),
Llama-2-chat (middle) and Mistral-instruct (bottom).

##  Appendix G AI assistant

We used the GPT-4 model from OpenaAI to improve the language quality of this
manuscript.

##  Appendix H Prompt templates

###  H.1 PAC

⬇

f535ad28-8896-43a3-aa4c-f41c296848de:

instruction: ’Czy następująca klauzula jest uczciwa?’

demonstration: ’{text} Ocena: {labels}’

label_mapping:

0: Nie

1: Tak

⬇

a68bdcea-0c7f-42a8-8987-4d7114b43658:

instruction: ’’

demonstration: ’{text}. Fragment tekstu jest {labels}’

label_mapping:

0: klauzulą abuzywną

1: bezpiecznym postanowieniem umowy

⬇

e3fb857a-7bd9-4268-8915-c7f494374ca6:

instruction: ’’

demonstration: ’Klauzula: {text} Ocena: {labels}’

label_mapping:

0: niedozwolona

1: dozwolona

⬇

35a1ac4a-c0a9-4138-ab0d-cd759cdc805f:

instruction: ’Oceń czy następujący zapis jest uczciwy lub nieuczciwy.’

demonstration: ’{text} => {labels}’

label_mapping:

0: nieuczciwy

1: uczciwy

⬇

735df98d-2732-4d82-9b49-39de04a435e8:

instruction: ’Do jakiej klasy przypisałbyś następującą klauzulę?

\parKlasy:

\par- uczciwa

\par- nieuczciwa’

demonstration: ’{text} => {labels}’

label_mapping:

0: nieuczciwa

1: uczciwa

⬇

5d80ad77-791b-4acc-9676-84d2d3d94966:

instruction: ’Klauzule abuzywne to postanowienia w umowach konsumenckich,
które nie są

uzgodnione indywidualnie z klientem. Czy fragment jest klauzulą abuzywną?’

demonstration: ’{text} => {labels}’

label_mapping:

0: Tak

1: Nie

⬇

e9ef6dcc-e21e-4efd-827e-d7cd35a97ce4:

instruction: ’Klauzule abuzywne to postanowienia w umowach konsumenckich,
które nie są

uzgodnione indywidualnie z klientem. Czy fragment jest klauzulą abuzywną? Tak
lub

Nie?’

demonstration: ’{text} => {labels}’

label_mapping:

0: Tak

1: Nie

⬇

39855489-6f18-49f3-92a8-1cdcccf0cc67:

instruction: ’Wybierz czy klauzula jest "uczciwa" lub "nieuczciwa"’

demonstration: ’{text} => {labels}’

label_mapping:

0: nieuczciwa

1: uczciwa

⬇

ceb5e55e-cb6f-47ab-b4f0-059192837c26:

instruction: ’’

demonstration: ’Następujący zapis {text} w umowie konsumenckiej jest {labels}’

label_mapping:

0: nieuczciwy

1: uczciwy

⬇

3ad789e3-6a21-4db9-8fba-c051c37cf449:

instruction: ’Dokończ zdanie: "Następujący zapis w umowie konsumenckiej
jest".’

demonstration: ’Zapis: {text} {labels}’

label_mapping:

0: nieuczciwy

1: uczciwy

⬇

871270fa-33aa-4677-b6ce-1869632a694c:

instruction: ’Dokończ zdanie.’

demonstration: ’Zapis {text} to {labels}’

label_mapping:

0: klauzula abuzywna

1: bezpieczne postanowienie umowne

⬇

df724698-89eb-4634-8d62-a04406b8da26:

instruction: ’Czy tekst jest bezpiecznym postanowieniem umowy?’

demonstration: ’{text} => {labels}’

label_mapping:

0: Nie

1: Tak

⬇

559e1bca-12e4-4d42-a305-60de9010cefc:

instruction: ’Czy tekst jest bezpiecznym postanowieniem umowy? Tak lub Nie?’

demonstration: ’{text} => {labels}’

label_mapping:

0: Nie

1: Tak

###  H.2 DYK

⬇

8102fdda-4584-4f50-8eab-ca7e6904c70e:

instruction: ’’

demonstration: ’Pytanie: {sentence_1} Odpowiedź: {sentence_2} Czy to prawda?
{labels}’

label_mapping:

0: Nie

1: Tak

⬇

37294e9a-d241-4f37-b680-df58dc625ee8:

instruction: ’’

demonstration: ’Pytanie: {sentence_1} Odpowiedź: {sentence_2} Poprawny:
{labels}’

label_mapping:

0: Nie

1: Tak

⬇

5cc8dd02-287a-422b-8c1c-9583fcb5af1b:

instruction: ’Oceń czy sugerowana odpowiedź pasuje do postawionego pytania.’

demonstration: ’Pytanie: {sentence_1} Odpowiedź: {sentence_2} Ocena: {labels}’

label_mapping:

0: Nie

1: Tak

⬇

1e8316a2-4611-40dd-9877-25c09bb4f895:

instruction: ’Oceń czy sugerowana odpowiedź pasuje do postawionego pytania.’

demonstration: ’Pytanie: {sentence_1} Odpowiedź: {sentence_2} Ocena: {labels}’

label_mapping:

0: nie pasuje

1: pasuje

⬇

0030485b-d576-4533-aa04-8840744e13c2:

instruction: ’Oceń czy sugerowana odpowiedź pasuje do postawionego pytania.’

demonstration: ’Pytanie: {sentence_1} Odpowiedź: {sentence_2} Ocena: {labels}’

label_mapping:

0: ’0’

1: ’1’

⬇

ddee4860-8329-40e5-af83-7972e17e6715:

instruction: ’’

demonstration: ’Czy tekst: {sentence_2} odpowiada na pytanie: {sentence_1}.

Tak lub Nie? {labels}’

label_mapping:

0: Nie

1: Tak

⬇

05838165-f7b0-44bf-876d-4b3431c11dfc:

instruction: ’Czy sugerowana odpowiedź poprawnie odpowiada na pytanie? Tak lub
Nie?’

demonstration: ’Pytanie: {sentence_1} Odpowiedź: {sentence_2} {labels}’

label_mapping:

0: Nie

1: Tak

⬇

f52e97a6-546f-4f72-b026-d6ccff0fd857:

instruction: ’Zadanie polega na przypisaniu jednej z dwóch etykiet "Tak" lub
"Nie"

do pary pytanie, odpowiedź.’

demonstration: ’Pytanie: {sentence_1} Odpowiedź: {sentence_2} {labels}’

label_mapping:

0: Nie

1: Tak

⬇

489d39ba-d7a6-4c3f-9529-caba81bea564:

instruction: ’Mając pytanie oceń czy odpowiedź jest "prawdziwa" lub
"fałszywa".’

demonstration: ’Pytanie: {sentence_1} Odpowiedź: {sentence_2} {labels}’

label_mapping:

0: fałszywa

1: prawdziwa

⬇

d78aafd2-848c-45a5-b227-09b4f0f2c000:

instruction: ’Proszę przeczytać poniższe pytanie oraz odpowiedź i zdecydować,
czy podana

odpowiedź jest prawdziwa (P) czy fałszywa (F).’

demonstration: ’Pytanie: {sentence_1} Odpowiedź: {sentence_2} {labels}’

label_mapping:

0: F

1: P

⬇

723df916-959c-4765-936f-684f9186e161:

instruction: ’Proszę przeczytać poniższe pytanie oraz odpowiedź i zdecydować,
czy

podana odpowiedź jest prawdziwa (P) czy fałszywa (F).’

demonstration: ’Pytanie: {sentence_1} Odpowiedź: {sentence_2} {labels}’

label_mapping:

0: fałszywa

1: prawdziwa

⬇

b3626557-c0d0-48b9-bdd0-21cb6ebc1205:

instruction: ’Przeanalizuj krok po kroku czy podany tekst jest poprawną
odpowiedzią

na pytanie. Na końcu zwróć "tak" lub "nie".’

demonstration: ’Pytanie: {sentence_1} Tekst: {sentence_2} {labels}’

label_mapping:

0: Nie

1: Tak

⬇

4c7c5c30-61eb-49ff-89a1-8faa8e2c4c20:

instruction: ’’

demonstration: ’Czy następujący kawałek tekstu: {sentence_2} zawiera w sobie
odpowiedź

na pytanie: {sentence_1}? Tak lub Nie? {labels}’

label_mapping:

0: Nie

1: Tak

⬇

a963d2e0-fce6-4d06-ad38-e28f176bc688:

instruction: ’’

demonstration: ’Jeśli {sentence_1} jest pytaniem to czy {sentence_2} jest
pasującą

odpowiedzią? {labels}’

label_mapping:

0: Nie

1: Tak

###  H.3 CDSC-E

⬇

063792d7-e23f-4111-a64d-b8b59057ed80:

instruction: ’’

demonstration: ’Zdanie "{sentence_1}" odnosi się do zdania "{sentence_2}" jako
{labels}’

label_mapping:

0: sprzeczne

1: zgodne

2: neutralne

⬇

7a650332-ec38-4811-8401-5889df51c80f:

instruction: ’’

demonstration: ’Relacja "{sentence_1}" do zdania "{sentence_2}" jest {labels}’

label_mapping:

0: sprzeczna

1: zgodna

2: neutralna

⬇

05e4cd72-59a1-44fd-9d33-2a655aacc56f:

instruction: ’Która etykieta opisuje relację między zdaniami.

\parEtykiety:

\par- neutralna

\par- zgodna

\par- sprzeczna’

demonstration: ’Zdania: {sentence_1}; {sentence_2} Etykieta: {labels}’

label_mapping:

0: sprzeczna

1: zgodna

2: neutralna

⬇

7c5cd177-5e97-4efc-805f-86d0b5e1e6f0:

instruction: ’Czy pierwsze zdanie pociąga za sobą drugie zdanie? Tak, Nie lub
Może?’

demonstration: ’Zdanie1: {sentence_1}

\parZdanie2: {sentence_2}

\par{labels}’

label_mapping:

0: Nie

1: Tak

2: Może

⬇

a97257d8-db60-4fbe-bdc9-c198fea17706:

instruction: ’’

demonstration: ’Załóżmy, że {sentence_1}. Czy możemy wywnioskować iż
{sentence_2}? Tak,

Nie lub Może? {labels}’

label_mapping:

0: Nie

1: Tak

2: Może

⬇

5d6b1071-9e49-422a-b649-6e04ef6dae65:

instruction: ’’

demonstration: ’{sentence_1}. Czy można powiedzieć że {sentence_2}? Tak, Nie
lub Może?

{labels}’

label_mapping:

0: Nie

1: Tak

2: Może

⬇

6ffe622f-cef2-4798-ae3e-dbfdabb0d384:

instruction: ’Mając kontekst odpowiedz na pytanie zdanie to prawda, fałsz lub
żadne.’

demonstration: ’Kontekst: {sentence_1}

\parZdanie: {sentence_2}

\parOdpowiedź: {labels}’

label_mapping:

0: Fałsz

1: Prawda

2: Żadne

⬇

f397d997-4f5f-4e08-8ce4-356f9905ea4d:

instruction: ’Logiczne wynikanie.’

demonstration: ’Zdanie1: {sentence_1}

\parZdanie2: {sentence_2}

\par{labels}’

label_mapping:

0: sprzeczne

1: zgodne

2: neutralne

###  H.4 Polemo2

⬇

5882074e-1e5c-479e-b138-abdeab4e4996:

instruction: ’’

demonstration: ’Recenzja {text} jest {labels}’

label_mapping:

0: neutralna

1: negatywna

2: pozytywna

3: niejednoznaczna

⬇

18d75cc9-c3c0-462e-a4c5-d35bee7e810b:

instruction: ’’

demonstration: ’Tekst {text} ma sentyment {labels}’

label_mapping:

0: neutralny

1: negatywny

2: pozytywny

3: niejednoznaczny

⬇

a034ba19-d751-4b02-a0a2-f98e4a9d43d7:

instruction: ’Oceń sentyment tego tekstu, używając jednej z czterech etykiet:

pozytywny, negatywny, neutralny, ambiwalentny.’

demonstration: ’Tekst do oceny: {text} Ocena: {labels}’

label_mapping:

0: neutralny

1: negatywny

2: pozytywny

3: ambiwalentny

⬇

18576914-803b-4c87-812c-240a59596e37:

instruction: ’Czy wydźwięk następującego tekstu jest pozytywny, negatywny,
neutralny,

ambiwalentny?’

demonstration: ’{text} => {labels}’

label_mapping:

0: neutralny

1: negatywny

2: pozytywny

3: ambiwalentny

⬇

bc7b3799-4e1d-4375-a6c0-5d0e52be5750:

instruction: ’Która z etykiet najlepiej określa sentyment tekstu?

\parEtykiety:

\par- neutralny

\par- negatywny

\par- pozytywny

\par- ambiwalentny’

demonstration: ’Tekst: {text} Etykieta: {labels}’

label_mapping:

0: neutralny

1: negatywny

2: pozytywny

3: ambiwalentny

⬇

80c3a8cc-fe0b-441f-bd1f-abdcfc48d870:

instruction: ’Czy to zdanie było neutralne, pozytywne, negatywne lub
ambiwalentne?’

demonstration: ’{text}

\parOdpowiedź: {labels}’

label_mapping:

0: neutralne

1: negatywne

2: pozytywne

3: ambiwalentne

⬇

c49953e5-f4cf-4a86-a6c2-65d85db7f478:

instruction: ’Mając opinię stwierdź czy jest ona neutralna, negatywna,
pozytywna lub ambiwalentna?’

demonstration: ’Opinia: "{text}"

\par{labels}’

label_mapping:

0: neutralna

1: negatywna

2: pozytywna

3: ambiwalentna

⬇

1e5de596-8d20-48fc-8615-2052a5559a77:

instruction: ’Jaki wydźwięk ma recenzja?’

demonstration: ’Recenzja: "{text}"

\par{labels}’

label_mapping:

0: neutralny

1: negatywny

2: pozytywny

3: ambiwalentny

⬇

a71195d9-f51f-408d-b25b-ea53dbd64421:

instruction: ’Jaki wydźwięk ma recenzja? neutralny, negatywny, pozytywny

lub ambiwalentny?’

demonstration: ’{text} => {labels}’

label_mapping:

0: neutralny

1: negatywny

2: pozytywny

3: ambiwalentny

⬇

cf1247a0-463c-4ca6-b8e8-a1c8864f50e9:

instruction: ’Przypisz sentyment do zdania.’

demonstration: ’{text} => {labels}’

label_mapping:

0: neutralny

1: negatywny

2: pozytywny

3: ambiwalentny

⬇

173978fb-8da7-4931-b9e6-2ea1682386d6:

instruction: ’Przypisz sentyment do zdania. Wybierz jedną z etykiet:
neutralny,

pozytywny, negatywny, ambiwalentny.’

demonstration: ’{text} => {labels}’

label_mapping:

0: neutralny

1: negatywny

2: pozytywny

3: ambiwalentny

###  H.5 CBD

⬇

3deddcad-cf5f-4efb-a1ac-8302a327a200:

instruction: ’Czy zdanie jest obraźliwe?’

demonstration: ’{text} => {labels}’

label_mapping:

0: Nie

1: Tak

⬇

23b47eed-5a7d-4388-82a0-d832c285e159:

instruction: ’’

demonstration: ’Tekst: {text} Etykieta: {labels}’

label_mapping:

0: nie obraźliwe

1: obraźliwe

⬇

62f33ae1-a7bf-4ab3-9192-afe3b5bc8e20:

instruction: ’Czy zdanie jest obraźliwe?’

demonstration: ’Tekst: {text} Etykieta: {labels}’

label_mapping:

0: nie obraźliwe

1: obraźliwe

⬇

f1cc2961-3a7b-4cc6-887e-c661089dd994:

instruction: ’Czy zdanie jest agresywne?’

demonstration: ’{text} => {labels}’

label_mapping:

0: Nie

1: Tak

⬇

0d817d40-b514-4a40-b76a-dbdd5902bc3e:

instruction: ’Czy zdanie jest agresywne? Tak lub Nie?’

demonstration: ’{text} => {labels}’

label_mapping:

0: Nie

1: Tak

⬇

0b170e50-aec1-4843-98d6-6f51f01cc951:

instruction: ’Czy tekst może być zaklasyfikowany jako cyberprzemoc?’

demonstration: ’{text} => {labels}’

label_mapping:

0: Nie

1: Tak

⬇

83519157-5075-402c-815f-17f575f1fc5b:

instruction: ’Czy tekst może być zaklasyfikowany jako cyberprzemoc? Tak lub
Nie?’

demonstration: ’{text} => {labels}’

label_mapping:

0: Nie

1: Tak

⬇

79929bb1-15a1-4232-b6da-e2a72c039446:

instruction: ’Cyberprzemoc to stosowanie przemocy z wykorzystaniem internetu.
Czy

tekst to cyberprzemoc?’

demonstration: ’{text} => {labels}’

label_mapping:

0: Nie

1: Tak

⬇

6ff460ac-1b85-4e60-b186-1360b0fca5a2:

instruction: ’Wybierz jedną z etykiet pasującą do tekstu.

\parEtykiety:

\par- obraźliwe

\par- nie obraźliwe’

demonstration: ’Tekst: {text} {labels}’

label_mapping:

0: nie obraźliwe

1: obraźliwe

###  H.6 NKJP-NER

⬇

0fbf54ee-bfdd-4a60-866b-8c3de8ac42d6:

instruction: ’Mamy kilka encji: geografia, brak, organizacja, imię, miejsce,
czas.’

demonstration: ’Zdanie "{text}" zawiera encję {labels}’

label_mapping:

1: brak

3: imię

4: miejsce

0: geografia

2: organizacja

5: czas

⬇

00486b42-3f72-481f-a350-2b60505d297e:

instruction: ’’

demonstration: ’{text}. To zdanie dotyczy {labels}’

label_mapping:

1: niczego

3: osoby

4: miejsca

0: geografii

2: organizacji

5: czasu

⬇

46c1cd05-2f1d-4de3-8de6-0f689a09d56a:

instruction: ’Zdanie zawiera fragment odnoszący się do encji.’

demonstration: ’"{text}"

\par{labels}’

label_mapping:

1: brak

3: osoba

4: miejsce

0: geografia

2: organizacja

5: czas

⬇

abba041c-81a3-4392-bb02-abb28a8aa549:

instruction: ’Którą z encji przypisałbyś zdaniu: brak, osoba, miejsce,
geografia,

organizacja, lub czas?’

demonstration: ’{text}.

\parOdpowiedź: {labels}’

label_mapping:

1: brak

3: osoba

4: miejsce

0: geografia

2: organizacja

5: czas

⬇

d5a15d7b-00b6-4533-ac5c-b9edaea9d484:

instruction: ’Mamy zdanie, które może zawierać jednostkę referencyjną (encję).

Która z nich pasuje najlepiej?’

demonstration: ’{text} => {labels}’

label_mapping:

1: żadna

3: osoba

4: miejsce

0: geografia

2: organizacja

5: czas

⬇

657775d6-6547-419f-88b4-820373a3812a:

instruction: ’Mamy zdanie, które może zawierać jednostkę referencyjną (encję).

Która z nich pasuje najlepiej? osoba, miejsce, geografia, organizacja,

czas lub żadna?’

demonstration: ’{text} => {labels}’

label_mapping:

1: żadna

3: osoba

4: miejsce

0: geografia

2: organizacja

5: czas

⬇

b28326b4-d8e3-4a1e-845f-cdb031601873:

instruction: ’W zdaniu może pojawić się referencja do osoby, miejsca,
lokalizacji geograficznej,

organizacji lub czasu. Która z nich lub żadna?’

demonstration: ’{text} => {labels}’

label_mapping:

1: żadna

3: osoba

4: miejsce

0: lokalizacja geograficzna

2: organizacja

5: czas

⬇

ea5330fa-278e-4d87-ae69-7a427f546b7f:

instruction: ’Zadanie polega na identyfikacji encji (lub brak) w tekście.

\parEncje:

\par- osoba

\par- miejsce

\par- geografia

\par- organizacja

\par- czas

\par- brak’

demonstration: ’Tekst: {text} Encja: {labels}’

label_mapping:

1: brak

3: osoba

4: miejsce

0: geografia

2: organizacja

5: czas

⬇

9bd0f669-c589-4419-9cbb-773d72df8dcd:

instruction: ’Przytoczony fragment zawiera odniesienie do encji.’

demonstration: ’"{text}".

\par{labels}’

label_mapping:

1: niczego

3: osoby

4: miejsca

0: geografii

2: organizacji

5: czasu

###  H.7 CST-Wikinews

⬇

10876076-d6a2-4ec3-b82c-f59cf79c17fa:

instruction: ’’

demonstration: ’Zdania {sentence_1} oraz {sentence_2} to {labels}’

label_mapping:

0: brak relacji

1: dalsze informacje

2: krzyżowanie się

3: opis

4: parafraza

5: spełnienie

6: streszczenie

7: tożsamość

8: tło historyczne

9: uszczegółowienie

10: zawieranie

11: źródło

⬇

8a0292ef-d678-4be6-bb95-0fe1b0cf136f:

instruction: ’’

demonstration: ’Zdanie z Wikipedii "{sentence_1}" odnosi się do zdania

"{sentence_2}" jako {labels}’

label_mapping:

0: brak relacji

1: dalsze informacje

2: krzyżowanie się

3: opis

4: parafraza

5: spełnienie

6: streszczenie

7: tożsamość

8: tło historyczne

9: uszczegółowienie

10: zawieranie

11: źródło

⬇

f2fd82fd-f3d9-4de0-aeb5-1801d2b5128b:

instruction: ’’

demonstration: ’Zdanie_1: {sentence_1} Zdanie_2: {sentence_2} Etykieta:
{labels}’

label_mapping:

0: brak relacji

1: dalsze informacje

2: krzyżowanie się

3: opis

4: parafraza

5: spełnienie

6: streszczenie

7: tożsamość

8: tło historyczne

9: uszczegółowienie

10: zawieranie

11: źródło

⬇

5f34a87c-760e-4839-8ba8-7e643de1d607:

instruction: ’Relacja między zdaniami.’

demonstration: ’Zdanie_1: {sentence_1} Zdanie_2: {sentence_2} Etykieta:
{labels}’

label_mapping:

0: brak relacji

1: dalsze informacje

2: krzyżowanie się

3: opis

4: parafraza

5: spełnienie

6: streszczenie

7: tożsamość

8: tło historyczne

9: uszczegółowienie

10: zawieranie

11: źródło

⬇

a668bb22-b3c3-482c-809a-3f14eccdec1f:

instruction: ’Która etykieta opisuje relację między zdaniami.

\parEtykiety:

\par- brak relacji

\par- dalsze informacje

\par- krzyżowanie się

\par- opis

\par- parafraza

\par- spełnienie

\par- streszczenie

\par- tożsamość

\par- tło historyczne

\par- uszczegółowienie

\par- zawieranie

\par- źródło’

demonstration: ’Zdania: {sentence_1}; {sentence_2} Etykieta: {labels}’

label_mapping:

0: brak relacji

1: dalsze informacje

2: krzyżowanie się

3: opis

4: parafraza

5: spełnienie

6: streszczenie

7: tożsamość

8: tło historyczne

9: uszczegółowienie

10: zawieranie

11: źródło

⬇

debde233-ea35-4e23-9d2d-f49b2884f5b5:

instruction: ’’

demonstration: ’Zdanie "{sentence_1}" ma się do zdania "{sentence_2}" jak
{labels}’

label_mapping:

0: brak relacji

1: dalsze informacje

2: krzyżowanie się

3: opis

4: parafraza

5: spełnienie

6: streszczenie

7: tożsamość

8: tło historyczne

9: uszczegółowienie

10: zawieranie

11: źródło

⬇

2ced9c63-7221-48b6-8c72-a8a78bbd13a0:

instruction: ’Jakiego typu relacja występuje między parą zdań? Wybierz
spośród:

brak relacji, dalsze informacje, krzyżowanie się, opis, parafraza,

spełnienie, streszczenie, tożsamość, tło historyczne, uszczegółowienie,
zawieranie,

źródło.’

demonstration: ’"{sentence_1}, "{sentence_2}, {labels}"’

label_mapping:

0: brak relacji

1: dalsze informacje

2: krzyżowanie się

3: opis

4: parafraza

5: spełnienie

6: streszczenie

7: tożsamość

8: tło historyczne

9: uszczegółowienie

10: zawieranie

11: źródło