# On the Effects of Randomness on Stability of Learning with Limited Labelled
Data: A Systematic Literature Review

Branislav Pecher  Faculty of Information Technology, Brno University of
TechnologyBrnoCzechia Kempelen Institute of Intelligent
TechnologiesBratislavaSlovakia<EMAIL_ADDRESS>0000-0003-0344-8620 ,
Ivan Srba  Kempelen Institute of Intelligent TechnologiesBratislavaSlovakia
<EMAIL_ADDRESS>0000-0003-3511-5337 and  Maria Bielikova  Kempelen
Institute of Intelligent TechnologiesBratislavaSlovakia
slovak.AIBratislavaSlovakia<EMAIL_ADDRESS>0000-0003-4105-3494

###### Abstract.

Learning with limited labelled data, such as few-shot learning, meta-learning
or transfer learning, aims to effectively train a model using only small
amount of labelled samples. However, these approaches were observed to be
excessively sensitive to the effects of uncontrolled randomness caused by non-
determinism in the training process. The randomness negatively affects the
stability of the models, leading to large variance in results across training
runs. When such instability is disregarded, it can unintentionally, but
unfortunately also intentionally, create an imaginary perception of research
progress. Recently, this area started to attract a research attention and the
number of relevant studies is continuously growing. In this survey, we provide
a comprehensive overview of 134 papers addressing the effects of randomness on
the stability of learning with limited labelled data. We distinguish between
four main tasks addressed in the papers (investigate/evaluate; determine;
mitigate; benchmark/compare/report randomness effects), providing findings for
each one. Furthermore, we identify and discuss seven challenges and open
problems together with possible directions to facilitate further research. The
ultimate goal of this survey is to emphasise the importance of this growing
research area, which so far has not received appropriate level of attention.

randomness, stability, meta-learning, transfer learning, language model fine-
tuning, in-context learning, literature survey

††ccs: General and reference Surveys and overviews††ccs: Computing
methodologies Artificial intelligence††ccs: Computing methodologies Machine
learning††ccs: Computing methodologies Learning paradigms

##  1\. Introduction

The approaches for learning with limited labelled data are designed to achieve
high performance of the machine learning models even with few labels available
(Brown et al., 2020; Wang et al., 2020, 2019c; Song et al., 2023). Under the
term learning with limited labelled data we understand any approach that is
designed to work with the lack of labels, without any constraint on how the
labelled samples are distributed, i.e., whether all the samples are from a
single task or are distributed across different tasks. Although similar to the
notion of few-shot learning, it represents a broader scope encompassing a
larger number of possible approaches (some of which are incorrectly
categorised as few-shot learning in current practice).

To deal with the limited labels, the approaches utilise additional information
from different sources (Brown et al., 2020; Wang et al., 2020, 2019c). A
popular option is to transfer knowledge from similar tasks and datasets using
transfer learning. For example, in the NLP domain it is common to use language
model fine-tuning, where a language model pre-trained on a large set of
unlabelled data (from unrelated domain) is fine-tuned for the specific
downstream task or problem statement using only few labelled samples available
for it (Xia et al., 2020; Rogers et al., 2020; Mosbach et al., 2021; Dodge et
al., 2020). Another emerging technique is the in-context learning, where a
large pre-trained language model is “prompted“ to predict a label for a single
test sample by presenting it with a concatenation of few labelled samples and
the test sample, without requiring any parameter update (Brown et al., 2020;
Liu et al., 2023). Finally, meta-learning is often used, where the model is
explicitly trained to quickly adapt to a new task with only a handful of
examples by learning how best to learn across large number of related tasks
with few labelled samples each (Agarwal et al., 2021; Hospedales et al., 2021;
Kadam and Vaidya, 2018).

However, a significant problem observed for the approaches for learning with
limited labelled data is their sensitivity to the effects of uncontrolled
randomness, which negatively affects their stability. Under stability, we
understand a property of a learning algorithm or a model that indicates what
influence the small scale and random changes (or perturbations) in the input
data and parameters have on its outputs (Liu et al., 2017). Such random
changes are introduced by different randomness factors that represent the non-
deterministic decisions in the training process, such as random initialisation
of model parameters, or data shuffling (Pham et al., 2021; Gundersen et al.,
2022). These randomness factors represents the main point around which the
effects of randomness on the stability are addressed.

The uncontrolled effects of randomness can have massive impact on the
stability of the utilised machine learning approaches. Running the training
multiple times on the same dataset, with the same setup and hyperparameters
may lead to large deviation in the final performance (Mosbach et al., 2021;
Dodge et al., 2020; Lu et al., 2022; Agarwal et al., 2021). Repeating BERT
fine-tuning multiple times and cherry-picking the best result, it is possible
to outperform more recent BERT variants (Dodge et al., 2020). Changing only
order of data in in-context learning can lead the model from state-of-the-art
predictions to random guesses (Lu et al., 2022). Or choosing different set of
adaptation data in meta-learning can lead to difference between minimum and
maximum performance being up to 90% (Agarwal et al., 2021). Even though the
effects of randomness are present in all cases of machine learning, their
impact on the stability is especially significant in low data regimes.

If the uncontrolled randomness is not appropriately addressed in such low data
regimes, it can have a significant non-negligible negative consequences. In
research works and benchmarks, it may prohibit an objective performance
comparison of newly designed methods to the state-of-the-art baselines
(especially when the difference between their performance is similar to the
deviation caused by the randomness). In such case, a method can be incorrectly
denoted as state-of-the-art only based on a more favourable random chance
(Reimers and Gurevych, 2017) (as illustrated in Figure 1(b)). The uncontrolled
randomness can unintentionally, but unfortunately also intentionally (by
cherry-picking), create an imaginary perception of research progress.

The effects of randomness are addressed in various depths - from a pure
recognition of randomness effects, through a deeper investigation up to a full
mitigation of such effects (illustrated in the Figure 1). Recent few-shot
learning surveys recognise the effects of randomness as an open problem (Wang
et al., 2020; Tian et al., 2022; Dong et al., 2022), or even dedicate a part
of the survey to a basic overview of these effects (Song et al., 2023). Other
papers simply recognise the impact of randomness on the overall results and
explicitly repeat the training and evaluation multiple times to obtain a more
representative, albeit aggregated value to address the randomness. However, no
further analysis regarding the effects of the randomness is provided. Finally,
only a small fraction of papers provide more detailed analysis of the effects
of randomness to improve the stability and reduce the significant deviation in
results. These works conduct a series of experiments that are designed to
investigate the effects, estimate the distribution of results and determine
the significance and the impact of the different randomness factors on the
stability. The effects of the randomness are analysed in more detail, in order
to determine the real origin of randomness in the training process (e.g.,
under-specification, which causes slightly different parameter initialisation
to converge to different minima). Based on the findings from the analysis of
the experiments, various methods are proposed to mitigate the effects of
randomness, improving the stability and reducing the deviation to achieve more
stable results.

(a) Various depths the effects of randomness can be addressed in the papers

(b) Comparison based on a single value and based on distribution of the
results, inspired by (Reimers and Gurevych, 2017)

Figure 1. The effects of randomness are addressed in various depths in the
papers. First, the effects are simply recognised, resulting in reporting of a
single aggregated value from multiple repeated training and evaluation runs
(illustrated as “Recognise”). This leads to biased comparison between
approaches, as one approach may show better results only due to random chance
and unintentional cherry-picking (illustrated in Figure 1(b)). Analysing the
effects of randomness, estimating the distribution of results from multiple
runs and comparing the distributions can prevent this problem (illustrated as
“Describe”). As the distribution is not affected in any way, only estimated,
the deviation is not reduced. The effects of randomness are properly addressed
when they are mitigated, reducing the deviation in the distribution of results
(illustrated as “Mitigate”). However, an effective mitigation requires
understanding of the effects, their importance, and the origin of randomness
provided by the analysis of the effects of randomness (provided by the
“Describe” task).

In this paper, we provide a comprehensive survey of 134 papers that address
the effects of randomness, either as simply recognising the effects of
randomness on the stability (N=84), or investigating and describing the
characteristics of these effects in more detail and/or exploring the
appropriate means of their mitigation (N=50). We provide a more in-depth
analysis of the 50 papers that address the effects of randomness in more
detail. We provide aggregated findings from this analysis and use them to
identify seven open problems related to the study of the effects of
randomness, while providing future directions on how to address them.

This is the first survey of its kind that provides a comprehensive and
systematic literature review specifically focused on the impact of the effects
of randomness on stability across various approaches for dealing with limited
labelled data. As opposed to previous papers that provide an overview of
possible randomness factors and investigate these effects (Pham et al., 2021;
Gundersen et al., 2022), we focus on all tasks for addressing the effects of
randomness, not only their investigation, but also determining their origin,
mitigation and benchmarking in presence of these effects of randomness.

The purpose of this survey is to emphasise the importance of this research
area, which so far has not received appropriate level of attention from
researchers and practitioners. At first, it should serve existing or new-
coming researchers who are explicitly tackling with the topic of the
randomness and its effects on the stability of learning with limited labelled
data to support their research (by providing the overview of the state-of-the-
art, identification of open problems and future direction). Secondly, its
purpose is also to inform researchers and practitioners utilising the learning
with limited labelled data about the consequences of unaddressed randomness
and how to effectively prevent them. To better achieve these purposes, we
summarised all the identified papers along with their categorisation as in the
supplementary material of this survey (Section A).

The rest of the paper is structured as follows. In Section 2, we describe the
scope of the survey, difference to existing surveys and the methodology used
for identifying relevant papers. Taxonomy applied to analyse and categorise
the identified papers is described in Section 3. Afterwards, we describe the
different tasks for addressing the effects of randomness, categorised based on
our taxonomy, in Sections 4, 5, 6 and 7, with a summary of the findings at the
end of each such section. The identified challenges and open problems are
described in Section 8. Finally, the survey is concluded in Section 9 with
summary of our contributions and findings.

##  2\. Background

###  2.1. Scope of the Survey

In this survey, we focus on a specific part of stability. We focus on papers
that explore the effects of small perturbations to multiple inputs of the
training process on the final performance after the training process is done
(as opposed to the effects of the perturbation of a single sample on the loss
or gradient of training as defined by uniform stability (CHEN et al., 2020)).
A similar, but at the same time quite distinct property is the robustness of
the model. It deals with a more large scale or systematic changes in the
input, such as distribution shift or presence of adversarial examples. As the
adversarial robustness is already extensively studied in the context of
learning with limited labelled (e.g., in meta-learning (Goldblum et al., 2020;
Yin et al., 2018; Xu et al., 2021a; Wang et al., 2021b)) and the effects are
often systematic, we consider it to be out of scope for this survey. Finally,
we view the stability to be a subset of the reproducibility of the model.
Besides randomness in the training process, the reproducibility is impacted by
other factors such as selective reporting, biases, data and source code
availability, and many others as specified in (Gundersen et al., 2022). These
additional factors are out of scope for this survey.

In case of machine learning approaches, we mainly focus on the approaches that
utilise related tasks and datasets as additional information. Not requiring
large set of labelled data, these approaches are usable across broader set of
domains, making them also more popular. Due to this popularity, they also
cover almost the whole research area of addressing the effects of randomness
on the stability, as there are almost no other works focusing on approaches
that utilise information from the unlabelled sets of data. Our focus is on
three prevalent groups of approaches: 1) meta-learning; 2) language model
fine-tuning; and 3) in-context learning. Even though we do not explicitly
focus on a specific modality, the popularity of the approaches impact the
scope of this survey. As the majority of the papers that focus on stability of
transfer learning work with language models, we define the scope in this
survey using the language model fine-tuning and in-context learning approaches
for better readability. We provide only the general problem statement for each
group of approaches (in Section 3.3), while providing references to more
comprehensive overview.

###  2.2. Existing Surveys

To the best of our knowledge, there are no existing surveys focused on
addressing the effects of randomness on stability. However, there are surveys
on related topics that partially touch on the topics of stability. First of
all, some papers provides an overview of possible sources of randomness,
although with focus on replicability (Gundersen et al., 2022). In some surveys
on the topic of learning with limited labelled data and/or few-shot learning,
the stability and the effects of randomness are mentioned as a problem that
should be addressed, with varying level of detail provided (Li et al., 2021;
Tian et al., 2022; Dong et al., 2022; Song et al., 2023).

At the same time, the study of effects of randomness on the stability is
becoming a research topic addressed in an increasing number of research works
focused on dealing with the lack of labels, mainly focused on the use of pre-
trained models (Dodge et al., 2020; Mosbach et al., 2021; Lu et al., 2022).
Even first benchmarks consider the stability of the models to be paramount and
include it as a part of the final metric (Efrat et al., 2022).

To fill this gap, in this survey we aim to provide the first comprehensive
overview of the effects of randomness on the stability of learning with
limited labelled data and describe it from perspectives that are not easily
visible from the standard analysis of related works.

Figure 2. Process for identifying and categorising papers for this survey. Due
to strong clustering effects, additional papers are identified using a
reference analysis on the most relevant identified papers, i.e., including
papers cited in them, as well as the ones that cite them.

###  2.3. Survey Methodology

To identify the set of relevant papers for our comprehensive survey, we take
inspiration from the PRISMA methodology (Moher et al., 2009; Page et al.,
2021), performing following steps (as illustrated in Figure 2):

  1. (1)

Identification of relevant papers. A keyword search query is defined based on
the related terms for our defined scope identified in different papers.
Multiple digital libraries are searched using the defined query to identify
set of potentially relevant papers.

  2. (2)

Relevance filtering. The potentially relevant papers are categorised into
irrelevant and relevant based on the context the defined keywords are used in.

  3. (3)

Content filtering. The non-full papers or papers released in unrelated venues,
such as extended abstracts, proposals for talks or papers appearing in
unrelated workshops not dealing with stability, are filtered out.

  4. (4)

Paper analysis. The papers are analysed based on their depth of addressing the
randomness effects. The papers that only recognise the problem of stability
without further investigation are referred to as recognise papers. The
remaining papers are referred to as core papers.

  5. (5)

Reference analysis. In order to discover additional potentially relevant
papers, the papers cited in the core papers; as well as papers citing the core
papers, are examined and their relevance is determined based on their title
and abstract. The papers deemed potentially relevant are an additional input
into the second step of this process.

  6. (6)

Analysis and categorisation. The identified core papers are analysed and
categorised based on the taxonomy (as defined in Section 3) to provide the
comprehensive overview and to identify challenges and open problems.

As a result, 134 papers are identified — out of them, 84 papers are classified
as recognise papers, and 50 as core papers. The number of identified papers
grouped by the publication year is presented in Figure 3. The categorisation
of the core papers is showcased in Table 1. Moreover, the full list of all
identified papers (including recognise papers), with their categorisation and
additional metadata is available in the supplementary material (Section A).
For a more detailed implementation of the survey methodology, see the
supplementary material (Section C).

(a) Core papers only

(b) All papers

Figure 3. Number of papers dealing with randomness grouped by the year. Figure 3(a) shows only the core papers that focus on addressing the effects of randomness in more detail, while Figure 3(b) also includes the papers that only recognise the problem. The problem started to attract attention only in recent years. Table 1. A showcase of the core papers and their categorisation based on the proposed txonomy. The full categorisation, along with the recognise papers is available in the supplementary material (Section A). References | Area | Factor | Performed Tasks | Modality |  Interactions |  Out-Of-Distribution
---|---|---|---|---|---|---
Machine Learning |  Meta-Learning |  LM Fine-Tuning |  In-Context Learning |  Label Selection |  Data Split |  Task Choice |  Data Choice |  Order of Data |  Parameter Initialisation |  Random Seed |  Implementation |  Number of Samples |  Hyperparameters |  Augmentation |  Noise |  Approach Specific |  Investigate + Evaluate |  Determine |  Mitigate |  Benchmark + Compare + Report |  Text |  Image
(Bouthillier et al., 2021) | $\sqrt{}$ |  |  |  | $\sqrt{}$ | $\sqrt{}$ |  |  | $\sqrt{}$ | $\sqrt{}$ |  |  |  | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ |  | $\sqrt{}$ |  |  | $\sqrt{}$ | $\sqrt{}$ |  | $\sqrt{}$ | $\times$
(Agarwal et al., 2021) |  | $\sqrt{}$ |  |  |  |  |  | $\sqrt{}$ |  |  |  |  |  |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ |  |  | $\sqrt{}$ | $\times$ | $\times$
(Boquet et al., 2019) |  | $\sqrt{}$ |  |  |  |  |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ |  | $\sqrt{}$ |  |  |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ | $\times$ | $\times$
(Lu et al., 2022) |  |  |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ | - |  |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ |  | $\sqrt{}$ |  | $\times$ | $\times$
(Zhao et al., 2021b) |  |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ |  |  | $\sqrt{}$ |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ |  | $\sqrt{}$ |  | $\sqrt{}$ | $\times$
(Osahor and Nasrabadi, 2022) |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ |  |  |  |  |  |  |  |  |  |  |  | $\sqrt{}$ |  |  | $\sqrt{}$ | $\times$ | $\times$
(Zhang et al., 2021a) |  |  | $\sqrt{}$ |  |  |  |  |  | $\sqrt{}$ |  |  |  |  |  |  |  |  |  |  | $\sqrt{}$ |  | $\sqrt{}$ |  | $\times$ | $\times$
(Min et al., 2022) |  |  |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ | - | $\sqrt{}$ |  | $\sqrt{}$ |  |  |  |  |  |  | $\sqrt{}$ |  | $\sqrt{}$ |  | $\times$ | $\times$
(Chen et al., 2022) |  | $\sqrt{}$ |  | $\sqrt{}$ |  |  |  | $\sqrt{}$ | $\sqrt{}$ |  |  |  |  |  |  |  | $\sqrt{}$ |  |  | $\sqrt{}$ |  | $\sqrt{}$ |  | $\times$ | $\times$
(Winata et al., 2021) |  |  |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ | - |  |  |  |  |  |  |  | $\sqrt{}$ |  |  |  | $\sqrt{}$ |  | $\times$ | $\times$
(Chang et al., 2021) |  |  | $\sqrt{}$ |  | $\sqrt{}$ |  |  |  |  |  |  |  | $\sqrt{}$ |  |  |  |  |  |  | $\sqrt{}$ |  | $\sqrt{}$ |  | $\times$ | $\times$
(Schick and Schütze, 2021) |  |  |  | $\sqrt{}$ |  |  |  | $\sqrt{}$ |  | - | $\sqrt{}$ |  |  |  |  |  |  | $\sqrt{}$ |  |  | $\sqrt{}$ | $\sqrt{}$ |  | $\times$ | $\times$
(Jundi and Lapesa, 2022) |  |  | $\sqrt{}$ |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ |  |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ |  | $\sqrt{}$ |  | $\sqrt{}$ |  | $\sqrt{}$ | $\times$
(Wu et al., 2022) |  |  | $\sqrt{}$ |  |  |  |  |  |  |  | $\sqrt{}$ |  |  |  |  |  |  |  | $\sqrt{}$ |  |  | $\sqrt{}$ |  | $\times$ | $\times$
(Yang and Kwok, 2022) |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ |  |  |  |  |  |  |  |  |  |  |  | $\sqrt{}$ |  |  | $\sqrt{}$ | $\times$ | $\times$
(Zhao et al., 2021a) |  |  |  | $\sqrt{}$ |  |  |  | $\sqrt{}$ | $\sqrt{}$ | - |  |  | $\sqrt{}$ |  |  |  | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ |  | $\sqrt{}$ |  | $\sqrt{}$ | $\times$
(Setlur et al., 2021b) |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ |  |  |  |  |  |  |  |  |  |  |  |  |  | $\sqrt{}$ |  | $\sqrt{}$ | $\times$ | $\sqrt{}$
(Dror et al., 2019) | $\sqrt{}$ |  |  |  |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ |  |  | $\sqrt{}$ |  |  |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ |  | $\times$ | $\times$
(Dodge et al., 2020) |  |  | $\sqrt{}$ |  |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ |  |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ |  | $\sqrt{}$ |  | $\sqrt{}$ |  | $\sqrt{}$ | $\times$
(Setlur et al., 2021a) |  | $\sqrt{}$ |  |  | $\sqrt{}$ |  |  | $\sqrt{}$ |  |  |  |  |  |  |  |  |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ | $\times$ | $\times$
(Mosbach et al., 2021) |  |  | $\sqrt{}$ |  |  |  |  |  |  |  | $\sqrt{}$ |  |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ |  | $\sqrt{}$ |  | $\times$ | $\times$
(Kumar and Talukdar, 2021) |  |  |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ | - |  |  |  |  |  |  |  |  |  | $\sqrt{}$ |  | $\sqrt{}$ |  | $\times$ | $\times$
(Reimers and Gurevych, 2017) | $\sqrt{}$ |  |  |  |  |  |  |  |  |  | $\sqrt{}$ |  |  | $\sqrt{}$ |  |  |  | $\sqrt{}$ |  |  | $\sqrt{}$ | $\sqrt{}$ |  | $\times$ | $\times$
(Zhang et al., 2021b) |  |  | $\sqrt{}$ |  |  |  |  |  |  | $\sqrt{}$ |  |  |  | $\sqrt{}$ |  |  | $\sqrt{}$ |  | $\sqrt{}$ | $\sqrt{}$ |  | $\sqrt{}$ |  | $\sqrt{}$ | $\times$
(Liu et al., 2022) |  |  |  | $\sqrt{}$ | $\sqrt{}$ |  |  | $\sqrt{}$ | $\sqrt{}$ | - |  |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ |  | $\sqrt{}$ |  | $\times$ | $\times$
(Sellam et al., 2022) |  |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ |  |  |  |  |  |  | $\sqrt{}$ |  |  | $\sqrt{}$ | $\sqrt{}$ |  | $\sqrt{}$ | $\times$
(McCoy et al., 2020) |  |  | $\sqrt{}$ |  |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ |  |  |  |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ |  | $\sqrt{}$ |  | $\times$ | $\sqrt{}$
(Zhou et al., 2020) |  |  | $\sqrt{}$ |  |  |  |  |  |  |  | $\sqrt{}$ |  |  |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ |  | $\sqrt{}$ |  | $\times$ | $\sqrt{}$
(Bouthillier et al., 2019) | $\sqrt{}$ |  |  |  |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ |  |  |  |  |  |  |  | $\sqrt{}$ |  |  | $\sqrt{}$ |  | $\sqrt{}$ | $\times$ | $\times$
(Dehghani et al., 2021) | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ |  |  | $\sqrt{}$ |  |  |  |  |  |  | $\sqrt{}$ |  |  | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ | $\times$ | $\times$
(Zhong et al., 2021) |  |  | $\sqrt{}$ |  |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ |  |  |  |  |  |  | $\sqrt{}$ |  |  | $\sqrt{}$ | $\sqrt{}$ |  | $\sqrt{}$ | $\times$
(Dauphin and Schoenholz, 2019) | $\sqrt{}$ |  |  |  |  |  |  |  |  | $\sqrt{}$ |  |  |  |  |  |  |  |  |  | $\sqrt{}$ |  |  | $\sqrt{}$ | $\times$ | $\times$
(D’Amour et al., 2022) | $\sqrt{}$ |  | $\sqrt{}$ |  |  |  |  |  |  |  | $\sqrt{}$ |  |  |  |  |  |  |  | $\sqrt{}$ |  |  | $\sqrt{}$ | $\sqrt{}$ | $\times$ | $\times$
(Khurana et al., 2021) |  |  | $\sqrt{}$ |  |  |  |  |  |  |  | $\sqrt{}$ |  |  |  |  |  |  |  |  | $\sqrt{}$ |  | $\sqrt{}$ |  | $\times$ | $\times$
(Hua et al., 2021) |  |  | $\sqrt{}$ |  |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ |  |  |  |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ |  | $\sqrt{}$ |  | $\times$ | $\times$
(Hidey et al., 2022) |  |  | $\sqrt{}$ |  |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ |  |  |  |  |  |  |  |  |  | $\sqrt{}$ |  | $\sqrt{}$ |  | $\times$ | $\times$
(Pham et al., 2021) | $\sqrt{}$ |  |  |  |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ |  | $\sqrt{}$ |  |  |  |  |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ | $\times$ | $\times$
(Cioba et al., 2022) |  | $\sqrt{}$ |  |  | $\sqrt{}$ |  |  |  |  |  |  |  |  |  |  |  |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ | $\times$ | $\sqrt{}$
(Phang et al., 2019) |  |  | $\sqrt{}$ |  |  |  |  |  |  |  | $\sqrt{}$ |  |  |  |  |  |  |  |  | $\sqrt{}$ |  | $\sqrt{}$ |  | $\times$ | $\sqrt{}$
(Wang et al., 2021a) |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ |  |  |  |  |  |  |  |  |  |  |  | $\sqrt{}$ |  | $\sqrt{}$ |  | $\times$ | $\times$
(Zhang et al., 2022b) |  |  | $\sqrt{}$ |  |  |  |  |  |  |  | $\sqrt{}$ |  |  |  |  |  |  |  |  | $\sqrt{}$ |  | $\sqrt{}$ |  | $\times$ | $\sqrt{}$
(Lee et al., 2020) |  |  | $\sqrt{}$ |  |  |  |  |  |  |  | $\sqrt{}$ |  |  |  |  |  |  |  |  | $\sqrt{}$ |  | $\sqrt{}$ |  | $\times$ | $\times$
(Xu et al., 2021c) |  |  | $\sqrt{}$ |  |  |  |  |  |  |  | $\sqrt{}$ |  |  |  |  |  |  |  |  | $\sqrt{}$ |  | $\sqrt{}$ |  | $\times$ | $\times$
(Meng et al., 2022) |  |  | $\sqrt{}$ | $\sqrt{}$ |  |  |  |  |  |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ |  |  |  | $\sqrt{}$ |  | $\sqrt{}$ |  | $\times$ | $\times$
(Zhang et al., 2022a) |  |  |  | $\sqrt{}$ |  |  |  | $\sqrt{}$ | $\sqrt{}$ | - |  |  |  |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ |  | $\sqrt{}$ |  | $\sqrt{}$ | $\times$
(Zheng et al., 2022) |  |  | $\sqrt{}$ |  |  | $\sqrt{}$ |  |  |  |  |  |  |  | $\sqrt{}$ |  |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ |  | $\times$ | $\times$
(Ye et al., 2021) |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ |  |  |  | $\sqrt{}$ |  |  |  |  |  |  | $\sqrt{}$ |  |  | $\sqrt{}$ | $\sqrt{}$ |  | $\times$ | $\sqrt{}$
(Bragg et al., 2021) |  |  | $\sqrt{}$ | $\sqrt{}$ |  |  |  |  |  |  | $\sqrt{}$ |  | $\sqrt{}$ | $\sqrt{}$ |  |  |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ |  | $\times$ | $\times$
(Mukherjee et al., 2021) |  |  | $\sqrt{}$ | $\sqrt{}$ |  | $\sqrt{}$ |  |  |  |  | $\sqrt{}$ |  | $\sqrt{}$ |  |  |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ | $\sqrt{}$ |  | $\times$ | $\times$
(Efrat et al., 2022) |  |  |  | $\sqrt{}$ |  |  |  | $\sqrt{}$ | $\sqrt{}$ | - |  |  |  |  |  |  | $\sqrt{}$ | $\sqrt{}$ |  |  | $\sqrt{}$ | $\sqrt{}$ |  | $\times$ | $\times$

##  3\. Taxonomy for Literature Analysis and Categorisation

The identified core papers are categorised based on multiple dimensions that
describe their characteristics of interest. We use two groups of
characteristics: 1) primary categorisation characteristics, which provide the
primary means for dividing and categorising the individual papers; and 2)
secondary categorisation characteristics, which provide a more nuanced
clustering of papers in the specific primary categorisation dimensions.

We define three separate primary properties that are used to categorise the
papers: 1) what tasks are performed to address the randomness; 2) randomness
factors addressed by the papers; and 3) what machine learning approach the
papers use. The performed tasks property is the main property based on which
we organise the rest of the survey, while the randomness factors and machine
learning approach properties are mostly used to define the scope in this
survey (see Section 2.1). We provide the paper distribution across individual
randomness factors and specific machine learning approaches for all core
papers in Table 2 (the paper distributions for specific tasks are included in
the supplementary material, Section A).

Table 2. Relation between the randomness factors and machine learning approach across all analysed papers. We can observe different popularity of the randomness factors across different machine learning approaches. The majority focus being denoted in bold. For example, fine-tuning mostly focuses on the random seeds. | All | Meta-Learning | Fine-Tuning | In-Context Learning
---|---|---|---|---
| (N=8) | (N=10) | (N=24) | (N=13)
Label Selection | 1 | 2 | 2 | 1
Data Split | 2 |  | 2 | 1
Task Choice | 1 | 5 | N/A |
Data Choice | 1 | 6 | 2 | 6
Random Seed | 4 | 2 | 17 | 5
Model Initialisation | 5 |  | 8 | N/A
Order of Data | 4 | 1 | 7 | 9
Implementation | 1 | 1 |  |
Number of Samples |  |  | 5 | 6
Hyperparameters | 3 | 1 | 4 | 1
Augmentation | 1 |  |  |
Noise | 1 |  | 1 | 1
Approach Specific | N/A |  | 2 | 3

###  3.1. Tasks Performed to Address the Randomness

This property specifies the different tasks for addressing the effects of
randomness that are performed in the papers (the relations between these
tasks, along with their input and output are illustrated in Figure 4). For
this property, we define following 7 values:

  1. (1)

Investigate \- the effects of randomness from different randomness factors are
investigated using specifically designed experiments. For example, the
training and evaluation is done multiple times to determine the distribution
of performance across multiple runs.

  2. (2)

Determine \- an effort to discover the origin of the randomness (different
from the randomness factors) is made by analysing the behaviour of the
specific machine learning approach or model in presence of randomness. For
example, to determine why changing the order of samples causes significant
change in performance, the experiments that observe specific problems (e.g.,
label bias) are designed and analysed.

  3. (3)

Mitigate \- mitigation strategies for reducing the effects of randomness are
proposed, used and their effectiveness is determined. For example, it is
proposed to use more deterministic data sampling based on heuristics instead
of random sampling.

  4. (4)

Benchmark \- a benchmark that is specifically designed to take into
consideration the effects of randomness is proposed. For example, a benchmark
that performs multiple train and test splits and evaluates on all of them.

  5. (5)

Evaluate \- a more sophisticated, usually statistical, framework for
evaluating the experiments investigating the effects of randomness, or
determining the effectiveness of mitigation strategies, is proposed. For
example, instead of calculating a simple average value, it is proposed to
estimate the distribution of results and evaluate the difference using
statistical tests.

  6. (6)

Report \- it is proposed to report more than a single value when running and
evaluating experiments and comparing approaches to better deal with the
effects of randomness. Not necessarily dependent on experiments for addressing
the effects of randomness, but also recommendations how to improve the
reporting of results overall when the randomness has a strong effect on
performance. For example, instead of reporting single average value, the
minimum and maximum value over multiple runs should be reported as well.

  7. (7)

Compare \- a better comparison strategies between different approaches in
presence of significant effects of randomness are proposed. For example,
instead of comparing single average values, it is proposed to compare
distributions of results using statistical tests.

We aggregate the closely related tasks together and organise the survey based
on these aggregated groups. Namely, we aggregate together the investigate and
evaluate tasks, as the evaluation strategies are designed specifically for the
investigation experiments. In addition, we aggregate the benchmark, compare
and report values, as they are closely related.

As the different papers can focus on different, often disjointed, tasks for
addressing the effects of randomness, this property does not necessarily
provide a distinct division of the papers. For example, the most common
combination is to perform investigation, determination and mitigation of a
specific randomness factor in a single paper. In such cases, the paper is
mentioned in this survey multiple times, each time focusing only on the parts
relevant for the specific task.

Figure 4. Different tasks for addressing the effects of randomness, along with
their inputs, outputs and the relations between them. The dashed lines
represent relations between the tasks that currently do not exist, but need to
be considered in the optimal state of addressing the effects of randomness,
e.g., using more sophisticated evaluation when determining the effectiveness
of mitigation strategies. The related tasks are grouped together, e.g.,
evaluate task near the investigate task.

###  3.2. Randomness Factors

The randomness factors property specifies what randomness factors are actually
addressed in the paper in some way (i.e., their effects are investigated or
mitigated), but not factors that are just mentioned (e.g., mention that other
papers found the factor to be unimportant). Papers can focus on multiple
factors at the same time.

Multiple randomness factors have already been identified and described in the
existing works (e.g., (Pham et al., 2021; Gundersen et al., 2022)). Since the
used terminology differs across the works, for the purpose of this survey, we
established the grounds (in terms of common terminology and understanding) by
collecting these randomness factors and unifying their denomination and
definition.

The identified randomness factors can be categorised into three distinct
groups: 1) factors related to the input data; 2) factors related to parameters
of the model and the overall training process; and 3) factors related to the
underlying implementation of deep learning frameworks and hardware.

As the randomness factors related to the input data, we identify following:

  1. (1)

Label selection \- specifies what data is considered to be annotated from
larger set of unlabelled data. Represents the real world scenario with limited
labelled budget, where only specific samples are annotated. Has potentially
significant impact on the underlying data distribution.

  2. (2)

Data split \- randomness introduced by different splits of data into training,
validation and testing set. Mainly impacts the information provided to model
during training.

  3. (3)

Task choice \- randomness introduced by choosing different tasks for training.
This factor is mainly relevant for approaches that perform few-shot learning,
such as meta-learning, or when using a benchmark dataset composed of different
tasks.

  4. (4)

Data choice \- randomness introduced by the sampling of data the model uses
for adaptation (meta-learning), or as examples in prompts (in-context
learning).

  5. (5)

Order of data \- randomness introduced by the random shuffling of data in
different training iterations. Mainly relevant for the in-context learning
(order of labelled samples in the prompt) and fine-tuning. For meta-learning,
this factor represents the order in which the different tasks are presented
during training.

  6. (6)

Augmentation \- randomness introduced by different data augmentation
strategies.

  7. (7)

Noise \- randomness introduced by adding noise to the different samples. In
essence, it can be viewed as a subset of the augmentation randomness factor.

  8. (8)

Number of samples \- specifies the randomness introduced by choosing different
number of labelled samples in training, such as different number of examples
in in-context learning. It can be viewed as an edge case with regards to the
definition of stability, as the expected change is often systematic (but is
not always the case).

Similarly, we identify the following randomness factors corresponding to the
randomness related to model parameters and the overall training process:

  1. (1)

Parameter initialisation \- randomness introduced by randomly initialising the
parameters of the model, such as weights in neural networks. Mainly relevant
for meta-learning and language model fine-tuning, as no initialisation is
performed in in-context learning.

  2. (2)

Random seed \- randomness introduced by non-deterministic actions in the
training process, such as the use of non-deterministic layers. Often also used
as an aggregated factor for the parameter initialisation and the order of data
randomness factors.

  3. (3)

Hyperparameters \- randomness caused by introducing small random changes into
hyperparameters. Can also be used when performing hyperparameter optimisation
in the low data regimes, due to large effects of randomness on them. Also
aggregates the randomness introduced by changing parameters related to
optimisation such as parameters used in optimiser or the number of training
iterations.

  4. (4)

Approach specific parameters \- aggregates the randomness introduced by
factors specific for only some of the approaches. Includes factors like
different prompt design (in-context learning), or different task sampling
strategies (meta-learning).

Randomness introduced by implementation of deep learning frameworks and
hardware is a special low level group, which includes factors such as
scheduling and floating point precision, auto-selection of primitive
operations, how parallel processes are used, changes introduced by different
implementation of frameworks such as TensorFlow or PyTorch and their different
software versions, but also many other factors influencing reproducibility
more than the stability of results (Pham et al., 2021; Gundersen et al.,
2022). Since these randomness factors cannot be easily addressed, for the
purpose of this survey, we do not specifically distinguish between them and
consider them jointly as one group.

Each randomness factor is associated with a set of its randomness factor
configurations (sometimes referenced in short as configurations). A single
configuration specifies one state from the set of all possible states the
randomness factor can attain. For example, a single randomness factor
configuration for the order of data specifies one possible permutation of the
samples, a single set of weights used for initialising the whole model, or a
single random value to seed the random number generator. Using this
definition, the set of randomness factor configuration can be infinite for
some factors (e.g., initialisation weights) or finite, but extremely large
(e.g., permutations of order).

The different randomness factors may depend on each other, leading to
interactions between them. An example of such interactions is the relation
between data choice and order of data randomness factors, where the choice of
what data we use in training explicitly affects the order of samples.
Therefore, such interactions can lead to negative, often unforeseen
consequences, such as prohibiting reliable freezing of randomness factor
configurations (e.g., it is not possible to have fixed order of samples when
using different choices of data).

###  3.3. Machine Learning Approach

Similarly, the machine learning approach property specifies which group of
approaches is addressed in the paper. Besides the machine learning approaches
that define the scope of this survey (Section 2.1), we also consider a generic
machine learning. This is a special value that contains all papers that still
deal with effects of randomness, but do not utilise any of the meta-learning,
fine-tuning or in-context learning approaches. The main idea, high-level
overview and reference to popular approaches and comprehensive surveys of the
machine learning approaches we work in this survey are included in the
supplementary material (Section B).

###  3.4. Secondary Categorisation Properties

Following additional properties are used for more nuanced categorisation and
clustering of papers:

  1. (1)

Modality that divides the papers into those focusing on text data, images,
tabular data, or any combination of these modalities.

  2. (2)

Groups of approaches and models further divide the primary property of the
machine learning approach into a more detailed categorisation. For example,
the meta-learning approaches are further divided into optimisation or metric
based meta-learning, and then even further into the specific approaches like
MAML (Finn et al., 2017), or Prototypical Networks (Snell et al., 2017).

  3. (3)

Interactions, a binary property that specifies whether the confounding effects
between different randomness factors are addressed in any way in the paper.

  4. (4)

Standard deviation, a binary property that specifies whether the performance
is reported using standard deviation (or similar metric such as confidence
intervals) or just as a simple aggregate metric (mean, median, etc.).

  5. (5)

Out-of-distribution, a binary property that specifies whether the data used
for testing purposes in the papers comes from the same distribution as the
training data, such as addressing the effects of randomness when dealing with
multilingual data or data drifts.

  6. (6)

Datasets used to better explore what datasets are popular when addressing the
effects of randomness. We do not list all the possible values as the list is
extensive, but includes popular datasets such as GLUE (and its sub-datasets)
(Wang et al., 2019b, a), MiniImageNet (Vinyals et al., 2016) or CIFAR-FS
(Bertinetto et al., 2019).

##  4\. Investigating the problem of stability

In this task, the effects of randomness on the stability are investigated,
their impact is estimated and the significance of this impact is determined.
Each paper investigating the effects of randomness can be characterised by
three main aspects: 1) scope of the investigation; 2) investigation
methodology; and 3) results from the investigation. The scope of the
investigation includes decisions about what factors to investigate or across
what models, datasets and modality the investigation is performed. It also
includes more nuanced specifics like what problem statement is considered
(simple classification/regression or sequence generation), or from what
setting the data comes from (in-distribution vs. out-of-distribution).
Although these decisions have minimal impact on the methodology, they strongly
govern the obtain results and their generality.

The investigation methodology determines how the investigation is done. It
provides answers to different problems and specifics of the investigation,
such as how to setup hyperparameters, how to perform the investigation
(multiple fully randomised runs or keeping some parts static), how many runs
to use, how to address interactions between randomness factors or how to
evaluate the different runs. Again, each decision has significant impact on
the results of the investigation, mainly how trustworthy and objective they
are.

Finally, the results of the investigation specify findings and the behaviour
that was observed. Interpreting these results is important for informing the
further tasks for addressing the effects of randomness (i.e., determining real
origin of randomness and mitigating the effects). They can also be used to
provide recommendations on the best use of models in practice in low data
regimes.

###  4.1. Scope of the investigation

Majority of the papers focus on investigating how the randomness affects the
language models (14 out of 24) when they are used to deal with limited
labelled data. The focus is distributed evenly across both the fine-tuning of
language models (Sellam et al., 2022; Zhao et al., 2021b; Jundi and Lapesa,
2022; Dodge et al., 2020; McCoy et al., 2020; Zhou et al., 2020; Zhong et al.,
2021) and the use of in-context learning (Lu et al., 2022; Winata et al.,
2021; Schick and Schütze, 2021; Zhao et al., 2021a; Liu et al., 2022; Zhang et
al., 2022a; Efrat et al., 2022). The focus on investigating overall machine
learning approaches (mainly their use with few data and evaluation)
(Bouthillier et al., 2021; Reimers and Gurevych, 2017; Bouthillier et al.,
2019; Dehghani et al., 2021; Pham et al., 2021) and the use of meta-learning
(Boquet et al., 2019; Agarwal et al., 2021; Setlur et al., 2021a; Cioba et
al., 2022; Ye et al., 2021) is lower, but not by a high margin.

The significant focus on language models also translates to what modality is
addressed the most. Investigating text data in NLP problems gets the most
focus (Bouthillier et al., 2021; Sellam et al., 2022; Lu et al., 2022; Zhao et
al., 2021b; Winata et al., 2021; Schick and Schütze, 2021; Jundi and Lapesa,
2022; Zhao et al., 2021a; Dodge et al., 2020; Reimers and Gurevych, 2017; Liu
et al., 2022; McCoy et al., 2020; Zhou et al., 2020; Zhong et al., 2021; Zhang
et al., 2022a; Ye et al., 2021; Efrat et al., 2022), with images being mostly
addressed only in meta-learning and machine learning approaches (but not all
of them) (Boquet et al., 2019; Agarwal et al., 2021; Setlur et al., 2021a;
Bouthillier et al., 2019; Pham et al., 2021; Cioba et al., 2022), with only a
single paper considering investigation across both image and text data
(Dehghani et al., 2021).

Some randomness factors receive more attention in investigation than other
ones. The more popular factors are usually the ones that are relevant for all
approaches. The randomness factor that receives the most attention is the
random seed, which aggregates both the initialisation and the order of data in
a single value. The papers can be separated into those that consider only
random seed (Boquet et al., 2019; Zhao et al., 2021b; Schick and Schütze,
2021; Reimers and Gurevych, 2017; Zhou et al., 2020; Dehghani et al., 2021; Ye
et al., 2021) and the ones that explicitly specify all the factors affected by
changing the configuration of random seed configuration (Sellam et al., 2022;
Dodge et al., 2020; Zhong et al., 2021; Bouthillier et al., 2021; McCoy et
al., 2020; Bouthillier et al., 2019; Pham et al., 2021). Some papers consider
only the initialisation (Jundi and Lapesa, 2022; Liu et al., 2022) or order of
data (Lu et al., 2022; Winata et al., 2021; Zhao et al., 2021a; Zhang et al.,
2022a; Efrat et al., 2022) as single factor without the combination with the
other one. Another quite popular general randomness factor is the data and
task choice (Sellam et al., 2022; Agarwal et al., 2021; Zhao et al., 2021b;
Schick and Schütze, 2021; Zhao et al., 2021a; Setlur et al., 2021a; Liu et
al., 2022; Dehghani et al., 2021; Zhang et al., 2022a; Ye et al., 2021; Efrat
et al., 2022). Other factors that influence all approaches, such as effects of
augmentation (Bouthillier et al., 2021), the choice of hyperparameters
(Bouthillier et al., 2021; Boquet et al., 2019; Reimers and Gurevych, 2017),
choice of what data is labelled (Bouthillier et al., 2021; Jundi and Lapesa,
2022; Setlur et al., 2021a; Liu et al., 2022; Cioba et al., 2022), split of
data (Bouthillier et al., 2021; Dehghani et al., 2021), noise (Bouthillier et
al., 2021) and implementation (Boquet et al., 2019; Pham et al., 2021) are
getting less attention. There is also little focus on the more niche
randomness factors such as prompt design (Efrat et al., 2022; Zhao et al.,
2021a), early stopping (Dodge et al., 2020) or the number of samples used (Lu
et al., 2022; Zhao et al., 2021b, a; Jundi and Lapesa, 2022; Liu et al.,
2022). However, the popularity and focus on the individual randomness factors
is dependent on the approaches for learning with limited labelled data and
their popularity, as some randomness factors are solely investigated only in
specific approaches and not across all of them.

The choice of what datasets and what models are used for investigation
influences how general the final results are. Many papers consider only a
single dataset or a single model architecture in their investigation (Pham et
al., 2021; Setlur et al., 2021a; McCoy et al., 2020; Winata et al., 2021;
Bouthillier et al., 2021; Dodge et al., 2020; Sellam et al., 2022), limiting
the generalisability of the results. It is more common to investigate the
effects of randomness across different models, either different architectures,
or the same architecture with different numbers of parameters, and across
different datasets (Bouthillier et al., 2019; Agarwal et al., 2021; Cioba et
al., 2022; Ye et al., 2021; Zhou et al., 2020; Zhao et al., 2021b; Jundi and
Lapesa, 2022; Schick and Schütze, 2021; Liu et al., 2022; Zhao et al., 2021a;
Zhang et al., 2022a; Boquet et al., 2019). In some specific cases, the
investigation is specifically designed as an investigation of the randomness
across models of different sizes, or as a comparison between different model
architectures (Reimers and Gurevych, 2017; Dehghani et al., 2021; Zhong et
al., 2021; Lu et al., 2022; Efrat et al., 2022), mainly to investigate how the
effects of randomness affect the choice of what model to use.

In the majority of the papers, the investigation is performed in an in-
distribution setting.Only few papers consider the investigation in out-of-
distribution setting to be important (McCoy et al., 2020; Zhou et al., 2020;
Cioba et al., 2022; Ye et al., 2021), often masked in cross-lingual problem
statement.

###  4.2. Investigation Methodology

Overall, the differences in how the investigation of the effects of different
randomness factors is done is small across different papers that focus on
different approaches and datasets, making the investigation quite consistent.
In all papers, the investigation is performed by changing the randomness
factor configuration (such as changing the value of random seed) and observing
the changes in behaviour of the model and its results. The most common way is
to introduce the change randomly into the configuration (Reimers and Gurevych,
2017; Bouthillier et al., 2019; Cioba et al., 2022; Ye et al., 2021; McCoy et
al., 2020; Zhou et al., 2020; Zhao et al., 2021b; Jundi and Lapesa, 2022;
Zhong et al., 2021; Schick and Schütze, 2021; Liu et al., 2022; Zhang et al.,
2022a; Bouthillier et al., 2021; Sellam et al., 2022). To exert additional
control over the investigation and its results, mainly when considering
multiple randomness factors at the same time and addressing the interactions
between them, some specific randomness factor configurations are fixed
(usually the ones not investigated), while randomly varying the rest (usually
the ones that we want to investigate) (Pham et al., 2021; Lu et al., 2022;
Zhao et al., 2021a; Efrat et al., 2022; Boquet et al., 2019; Dodge et al.,
2020). A combination of previous two approaches can also be used to compare
between them (Winata et al., 2021). In specific cases when the number of
possible configurations is small, the whole space of possible configurations
is explored, such as trying all the possible permutations when investigating
the effects of the order of data with few samples (Dehghani et al., 2021;
Agarwal et al., 2021). Finally, an importance of a specific randomness factor
configuration is determined by keeping it fixed throughout the whole
investigation (Setlur et al., 2021a).

An important aspect of the investigation is how the hyperparameters are
handled. Majority of papers use default hyperparameters or find an optimal set
for each model and dataset and use it throughout the whole investigation
(Reimers and Gurevych, 2017; Dehghani et al., 2021; Pham et al., 2021; Agarwal
et al., 2021; Setlur et al., 2021a; Cioba et al., 2022; Ye et al., 2021; Zhou
et al., 2020; Zhao et al., 2021b; Jundi and Lapesa, 2022; Zhong et al., 2021;
Boquet et al., 2019), or reuse a fixed set of hyperparameters determined to be
optimal in previous works (McCoy et al., 2020; Dodge et al., 2020; Sellam et
al., 2022). However, the effects of randomness influence the choice of
hyperparameters as well, possibly leading to suboptimal set of hyperparameters
that are determined as optimal only due to random chance. To deal with this
problem, some papers search through hyperparameters over multiple runs
(similarly to investigation) and choose the best ones based on aggregation
over all the runs (Bouthillier et al., 2019, 2021). Using multiple runs to
find the hyperparameters guarantees that they are more optimal for the
investigation, but at the cost of significant increase in computational cost.
However, it is important to note that not all approaches require
hyperparameter optimisation (Lu et al., 2022; Winata et al., 2021; Schick and
Schütze, 2021; Liu et al., 2022; Zhao et al., 2021a; Efrat et al., 2022; Zhang
et al., 2022a, a).

The choice of how many randomness factors to investigate at the same time also
plays an important role in the investigation methodology. Many papers consider
only a single randomness factor (Reimers and Gurevych, 2017; Bouthillier et
al., 2019; Dehghani et al., 2021; Winata et al., 2021; Schick and Schütze,
2021; Liu et al., 2022). However, such investigation can lead to biased
results due to the interactions between different randomness factors. Using
multiple randomness factors in a single investigation allows for more detailed
exploration of the behaviour of models in presence of different sources of
randomness. Most straightforward approach used in the papers is performing
investigation of single factor multiple times, while only changing the number
of samples to observe how the effects change when having more data for
training (Agarwal et al., 2021; Setlur et al., 2021a; Ye et al., 2021).
However, such investigation considers only the systematic changes introduced
by the number of samples and can still be sensitive to the interactions. Most
approaches investigate multiple randomness factors at the same time by
introducing randomised changes to each one at the same time (Cioba et al.,
2022; McCoy et al., 2020; Zhou et al., 2020; Zhao et al., 2021b; Jundi and
Lapesa, 2022; Zhong et al., 2021; Efrat et al., 2022; Zhang et al., 2022a;
Bouthillier et al., 2021; Boquet et al., 2019; Dodge et al., 2020; Sellam et
al., 2022; Lu et al., 2022; Zhao et al., 2021a), while also defining the
differences and relations between them (Pham et al., 2021). This allows to
take the interactions between randomness factors into consideration to explore
their combined effects.

Due to the potentially significant impact of the interactions between
randomness factors, it is important to define how to consider these
interactions and how to deal with them to disentangle the effects of different
randomness factors. Although it is more explicit when investigating multiple
factors, it is just as important when investigating only one factor at a time.
Many papers do not use any specific way to distinguish between the effects of
different randomness factors (Reimers and Gurevych, 2017; Bouthillier et al.,
2019; Dehghani et al., 2021; Setlur et al., 2021a; Cioba et al., 2022; Ye et
al., 2021; McCoy et al., 2020; Zhou et al., 2020; Agarwal et al., 2021; Winata
et al., 2021; Schick and Schütze, 2021; Liu et al., 2022). As mentioned
before, this may lead to the results being biased and can be traced as a
direct cause of some of the open problems discussed later. A simple way to
consider the interactions is to investigate one factor at a time by changing
only the configuration of this factor, while using a fixed configuration for
all the others (Pham et al., 2021; Zhao et al., 2021b; Lu et al., 2022; Zhao
et al., 2021a; Efrat et al., 2022; Boquet et al., 2019). However, such
approach is limited in its investigatory strength, due to the randomness still
being present in the choice of the fixed configuration. A better way is to
search through combination of configurations for all the factors at the same
time (Jundi and Lapesa, 2022; Zhong et al., 2021; Zhang et al., 2022a;
Bouthillier et al., 2021; Dodge et al., 2020; Sellam et al., 2022). However,
this significantly increase the computational cost of the whole investigation
(Bouthillier et al., 2021; Dehghani et al., 2021).

Another important aspect to consider is how to obtain representative results
from the investigation. The representativeness of results mainly depends how
many randomness factor configurations are explored, i.e., how many training
and evaluation runs are used during the investigation. Many papers choose the
number of different randomness factor configurations only arbitrarily, often
choosing lower number (3, 5, 40, 41, 86, 100, 1000) without any explicit
explanation why the given number was chosen (Reimers and Gurevych, 2017;
Bouthillier et al., 2019; Zhou et al., 2020; Boquet et al., 2019; Ye et al.,
2021; Liu et al., 2022; McCoy et al., 2020; Zhao et al., 2021b; Schick and
Schütze, 2021; Pham et al., 2021; Efrat et al., 2022; Bouthillier et al.,
2021). Similarly, when considering multiple randomness factors and
interactions between them, it is common to arbitrarily select some number
(usually small) of configurations for each factor and then determine the
number of runs as all possible combination of these configurations (Jundi and
Lapesa, 2022; Zhong et al., 2021; Dodge et al., 2020; Sellam et al., 2022).
For example, when considering 3 different randomness factors, with 5 different
configurations for each, the final number of runs is determined as 125
($5^{3}$). In specific cases, all possible configurations of a single factor
can be explored (Dehghani et al., 2021; Agarwal et al., 2021; Lu et al., 2022;
Winata et al., 2021; Zhao et al., 2021a; Zhang et al., 2022a). However, this
happens only when the number of possible configurations is small or the added
computation cost is not as significant.

Majority of the papers evaluate the effects of randomness from multiple runs
based on an aggregated value from these runs, either as a single value of
mean, standard deviation (with standard deviation not even being present in
some papers), or number of failed runs (Bouthillier et al., 2019; Dehghani et
al., 2021; Pham et al., 2021; Setlur et al., 2021a; Cioba et al., 2022; Ye et
al., 2021; McCoy et al., 2020; Zhou et al., 2020; Zhao et al., 2021b; Jundi
and Lapesa, 2022; Zhong et al., 2021; Lu et al., 2022; Winata et al., 2021;
Schick and Schütze, 2021; Zhao et al., 2021a; Efrat et al., 2022), or as
multiple aggregated values, such as average, worst and best performance, or
the difference between them (e.g., difference between worst and best run, or
difference between worst and average run) (Agarwal et al., 2021; Liu et al.,
2022; Zhang et al., 2022a). Only some papers evaluate the results of
investigation based on the distribution, mostly as a means for comparing
different randomness factors, models or the behaviour across datasets (Reimers
and Gurevych, 2017). Some papers introduce special evaluation strategies based
on statistical approaches that can better distinguish between effects of
different factors and deal with their interactions, and that can better
estimate the results distribution from small number of repeated runs to make
the results more representative (Bouthillier et al., 2021; Boquet et al.,
2019; Dodge et al., 2020; Sellam et al., 2022).

###  4.3. Results from the Investigation

Majority of the papers found that the effects of randomness have significant
effects on the performance (Bouthillier et al., 2019; Dehghani et al., 2021;
Agarwal et al., 2021; Ye et al., 2021; McCoy et al., 2020; Jundi and Lapesa,
2022; Zhong et al., 2021; Lu et al., 2022; Winata et al., 2021; Schick and
Schütze, 2021; Zhao et al., 2021a; Efrat et al., 2022; Zhang et al., 2022a;
Bouthillier et al., 2021; Boquet et al., 2019; Dodge et al., 2020; Sellam et
al., 2022), some specifying how high the variance is, or some simply stating
that the variance is significant without providing further details. The
significant effects are observed across all approaches for dealing with
limited data. Fine-tuning BERT multiple times and picking the best result, it
is possible to outperform its more recent variants (Dodge et al., 2020).
Changing only order of samples when using language models for in-context
learning can lead from state-of-the-art predictions to random guesses (Lu et
al., 2022). The difference in performance between worst performing and best
performing set of adaptation data in meta-learning can be up to 90% (Agarwal
et al., 2021). On the other hand, the observed variance in other experiments
was found to be lower, usually only few percent difference (1%-5%), while
still being enough to influence the results and comparisons between different
models (Pham et al., 2021; Zhao et al., 2021b; Liu et al., 2022; Reimers and
Gurevych, 2017).

Almost all investigation is done on in-distribution settings, with only few
papers dealing with out-of-distribution evaluation of the stability. When
investigating the effects in-distribution, majority of the papers found the
variance to be quite high (more than 10%, but sometimes going up to 90%
between best and worst performance) (Bouthillier et al., 2019; Dehghani et
al., 2021; Agarwal et al., 2021; Jundi and Lapesa, 2022; Zhong et al., 2021;
Lu et al., 2022; Winata et al., 2021; Schick and Schütze, 2021; Zhao et al.,
2021a; Efrat et al., 2022; Zhang et al., 2022a; Bouthillier et al., 2021;
Boquet et al., 2019; Dodge et al., 2020; Sellam et al., 2022), with only few
papers finding the variance to be smaller (1%-5% standard deviation) (Pham et
al., 2021; Zhao et al., 2021b; Liu et al., 2022; Reimers and Gurevych, 2017).
However, all the papers that investigate the out-of-distribution setting find
that the variance observed on in-distribution datasets is insignificant
(showing 2% variance) compared to the one observed on the out-of-distribution
datasets (showing 50% variance in performance (Ye et al., 2021; McCoy et al.,
2020; Zhou et al., 2020).

The investigation results indicate how the variance increases or decreases
based on choices in the investigation methodology, without specifically
addressing the stability. However, there is usually low consensus on how much
the variance is reduced by different choices. Some papers found that using
more labelled samples helps the stability the most (Agarwal et al., 2021;
Jundi and Lapesa, 2022), although with diminishing returns (Zhao et al.,
2021b). On the other hand, other papers found that increasing the number of
samples does not lead to significant improvement (Lu et al., 2022), or even
having negative effects on the stability (Zhao et al., 2021a). In other
papers, it was found that keeping the randomness factor configuration fixed
during the whole use of the model, such as using a single set of samples or
defined order based on heuristics, can help reduce the variance (Setlur et
al., 2021a; Winata et al., 2021), but not in all the cases (Zhang et al.,
2022a; Boquet et al., 2019). Similarly, some papers found that increasing the
model size reduces the variance (Zhong et al., 2021), but others have found
that it either has no effect or even worsens the situation (Efrat et al.,
2022). Other papers find that accounting for and investigating as many sources
of randomness as possible helps with reducing variance at the cost of
computational cost (Bouthillier et al., 2021), or that tasks with different
difficulty affect the variance in different ways, although it does not hold
true that using easier tasks leads to decrease the variance (Cioba et al.,
2022; Ye et al., 2021).

The investigation of the effects of randomness is often used just to determine
that there is an impact of randomness on the performance and stability of the
models, but without providing any additional analysis of how much it affects
the models or what randomness factors are most important (Setlur et al.,
2021a; Cioba et al., 2022; Ye et al., 2021; McCoy et al., 2020; Zhou et al.,
2020; Winata et al., 2021; Schick and Schütze, 2021; Liu et al., 2022; Zhang
et al., 2022a; Sellam et al., 2022). It is also observed that randomness can
lead to models oscillating between random chance and state-of-the-art results
(Lu et al., 2022; Agarwal et al., 2021), or that the randomness can impact the
ordering between different models, resulting in simpler models even
outperforming more complex ones based only on random chance (Reimers and
Gurevych, 2017; Bouthillier et al., 2019; Dehghani et al., 2021; Dodge et al.,
2020). The most interesting use for the results of investigation is for
comparing between the importance of the effects of different randomness
factors. Such investigation results show that the choice of samples and order
of data are the most significant contributors to the instability of the models
(Efrat et al., 2022; Jundi and Lapesa, 2022; Zhao et al., 2021b), with some
results indicating choice being more important (Zhao et al., 2021b; Jundi and
Lapesa, 2022) and other that order is the more important (Zhao et al., 2021a).
Other results indicate that random seeds, especially the ones used when fine-
tuning, are more significant than just the effects of implementation and
hardware (Pham et al., 2021; Zhong et al., 2021), while others find the
effects implementation and hardware to contribute more significantly (Boquet
et al., 2019). Finally, the results from investigation of almost all
randomness factors at the same time indicate that the splitting of data has
the most impact on the performance (Bouthillier et al., 2021).

The behaviour of randomness factors is compared across different models and
datasets. The inconsistency in results is present here as well. Some papers
indicate that there is different behaviour of the randomness factors across
different model architectures and model sizes (Reimers and Gurevych, 2017;
Bouthillier et al., 2019; Dehghani et al., 2021; Zhong et al., 2021; Lu et
al., 2022). However, other papers indicate that the behaviour across models is
similar, but varies across different datasets (Zhou et al., 2020; Ye et al.,
2021; Zhao et al., 2021a), with smaller datasets showing more variance (Dodge
et al., 2020). Finally, many of the results from investigation indicate that
the behaviour of randomness factors is consistent across all the model
architectures, sizes and datasets (Agarwal et al., 2021; McCoy et al., 2020;
Zhao et al., 2021b; Jundi and Lapesa, 2022; Winata et al., 2021; Schick and
Schütze, 2021; Liu et al., 2022; Efrat et al., 2022; Zhang et al., 2022a;
Bouthillier et al., 2021; Boquet et al., 2019; Sellam et al., 2022).

###  4.4. Overview of Findings: Investigating Effects of Randomness

The effects of randomness are not investigated evenly across all groups of
approaches for dealing with limited labelled data or modalities. The
investigation is more skewed towards randomness in language models, which goes
hand in hand with the focus on natural language processing. However, the
findings from meta-learning and overall machine learning approaches indicate
that this mostly stems from the popularity of language models research and not
from the impact the randomness has on the investigated approaches. The effects
of randomness are consistently significant across all approaches for dealing
with limited labelled data.

In addition, the focus of the investigation is mostly on a few popular
randomness factors, such as random seeds, model initialisation, sample choice
and order. Even though other randomness factors receive minimal attention, it
has no correlation with their importance. For example, data split is
considered the most important factor, while being investigated only in few
papers. Another mostly ignored factors are the non-deterministic
implementation and hardware, and choice of which samples are labelled.

The relevance and popularity of the randomness factors heavily depend on the
investigated approach. Many factors are irrelevant for specific approaches,
such as model initialisation for in-context learning. We can also observe
different popularity of general randomness factors across different
approaches. For example, sample (and task) choice is popular mostly for meta-
learning and in-context learning; while investigating the number of samples
used is mostly popular in fine-tuning and in-context learning.

The investigation design is agnostic from data and the machine learning
approach, making its use consistent across all papers with only small
modifications. However, these small modifications have significant impact on
the overall investigation, often making it impossible to compare findings from
different papers. The number of training and evaluation runs are chosen rather
arbitrarily, ranging from ones to thousands, without any explanations behind
the choice. Almost no paper introduces heuristics for determining how many
repeated runs are needed to estimate the underlying distribution of results.
The only heuristic used is to explore all the possible randomness factor
configurations when their number is small, such as the permutations when
number of used samples is small ($<10$).

A significant drawback for almost all investigation papers is that the effects
of randomness are not taken into consideration when searching for hyper-
parameters. The papers often do not even mention how the hyper-parameters are
chosen, use a fixed set found beforehand in preliminary experiment or in
another paper. However, without searching for hyper-parameters over multiple
runs, the results may be biased.

The effects of randomness are usually investigated only across small number of
randomness factors at one time. The number of considered factors is usually
1-3, while other factors may be mentioned, but not investigated. Almost no
paper covers a broader and more representative set of factors at the same
time, mostly because it is problematic to investigate the larger set of
factors due to the computational cost this introduces - the number of repeated
runs raises exponentially with number of randomness factors (Bouthillier et
al., 2019, 2021; Dehghani et al., 2021).

Even when evaluating multiple randomness factors at the same time, the
interactions between them are not taken into consideration, which leads to
biased comparisons. Only some papers consider using fixed randomness factor
configuration for the factors that are not investigated at the moment, or
using combinations of all randomness factor combinations. However, this is
always done only for factors that are considered in the paper and not for all
of them. In addition, no paper applies mitigation strategies to other
randomness factors.

No analysis of the importance of the randomness factors is performed when
investigating a single factor. Similarly, when investigating multiple factors
at the same time, no comparison is performed between these factors. This
causes the investigation results to be limited in their extent as the problem
is only recognised, but no further conclusions can be drawn from it. It is not
possible to answer questions like: “Are the effects of the factor
significant?” or “Which factor is more important and should be focused on
first?”.

The effects of randomness are evaluated in multitude of ways. The use of
simple aggregation over the training runs and reporting single standard
deviation value is the most common approach. However, as the experiments are
performed on a limited number of training runs, the single aggregated value
may not be representative of the underlying distribution. A better approach
for evaluating the effects is the use of statistical approaches that allow to
quantify the uncertainty still present in the models when only limited runs
are used, or to determine the expected behaviour of the models in limit, while
only having results from significantly smaller number of runs. The evaluation
is further complicated by the interactions between randomness factors that can
muddle the results and need to be taken into consideration using statistical
approaches.

Almost all papers investigate the effects of randomness on in-distribution
settings. Even though the papers find significant effects in this setting, the
effects found when studying the out-of-distribution settings are more
significant. The limited focus on the out-of-distribution settings may be due
to limited focus on meta-learning. In addition, in case of fine-tuning and in-
context learning, the out-of-distribution settings do not make sense in many
cases as the main point is to explicitly calibrate the models to work on the
specific task. The only exception is focus on zero-shot learning and cross-
lingual approaches.

A significant drawback regarding the results from the investigation is that no
obvious consensus exists regarding the behaviour of different randomness
factors across different models and datasets. Many papers investigating the
same setting come to contradictory findings about almost all parts of the
investigation. Some papers observe decrease in variance when the size of the
model increases, while others found no obvious connection between size of
model and the amount of observed variance (e.g., increasing size lead to
increase in variance). When it comes to datasets, some papers observed
consistent results, with the same factor being most significant across
different datasets, while others observed different factors being important on
different datasets. In addition, no external property of the dataset was found
that can indicate the amount of variance that would be observed on it. For
example, the variance on more complex datasets with more classes and less data
was far lower than on simpler datasets with only 2 classes and large amounts
of data available.

The most consistent finding from the investigation is that increasing the
number of samples reduces variance. However, the decrease in stability has
diminishing returns, where from certain point, increasing available data has
no more significant impact. This may also be due to the quality of samples
used, which was never considered when looking at number of samples used.
Although this finding is most consistent, there are also papers that find that
increasing number of samples leads to increase in variance in specific
settings.

##  5\. Determining the Origin of Randomness

In this task, the origin of randomness is determined, which represents a
specific underlying problem that only manifests through the randomness
factors. The design of experiments is similar to the one when investigating
the effects of randomness, but with a specific aspect that is observed (e.g.,
whether the label which is used as last example in in-context learning has
impact on the prediction, which would indicate recency bias). Analysing the
results and behaviour of models on this specific aspect across the different
runs, a hypothesis is formed and answered either empirically or using
theoretical understanding of deep learning models.

The design of experiments and further explanation of the behaviour is closely
tied to the used machine learning approach and the randomness factor that was
observed to cause the randomness.

###  5.1. Origins of Randomness in Meta-Learning

The focus on origins of randomness in meta-learning is limited only to the
adaptation data sampling randomness factor (Agarwal et al., 2021). The
behaviour of different meta-learning approaches is analysed using combination
of all the possible combinations of the adaptation samples from image
datasets. The goal of the analysis is to determine the characteristics of the
worst case samples, i.e., the samples that cause the worst performance when
used for adaptation. It is determined that these samples are not artefacts, as
they are always correctly labelled, appear representative of the respective
classes and are not visually different from other images. Based on this
observation, it is speculated that the problem is due to adversarial nature of
the images. However, it is later disproved empirically as adversarial training
does not reduce the effects of randomness, leaving the real reason why meta-
learning is sensitive to the choice of support samples open.

Another origin of randomness in meta-learning is hypothesised to be
overfitting. This can be due to the variance in the training process,
specifically in the gradient estimator (Wang et al., 2021a; Yang and Kwok,
2022).

###  5.2. Origins of Randomness in Language Model Fine-Tuning

Language model fine-tuning is analysed mainly using the random seed randomness
factor and in part using choice of samples at the start of training. Many
different origins of randomness are identified. The experiments in this part
are the most diverse, with the training being performed across different
models, languages and even factors, but often leading to similar origins of
randomness.

Small availability of data that causes overfitting and catastrophic forgetting
is identified as being one origin of randomness when fine-tuning language
models (Zhao et al., 2021b; Hua et al., 2021). This can cause the models to
rely on shallow lexical features, even though the models were trained on
abundant, feature rich data beforehand (Zhao et al., 2021b). Another possible
origin may be the bad fine-tuning start point in the transferred model, mainly
in the higher layers (not only the newly initialised ones, but also the pre-
trained ones) that are sensitive to perturbations in the input.

However, catastrophic forgetting and small size of training data is disputed
as cause by later papers (Mosbach et al., 2021; Zhang et al., 2021b; Wu et
al., 2022). Instead, it is only a symptom of another problem (Mosbach et al.,
2021). Looking at the failed fine-tuning runs (i.e., the runs that perform
worse than random chance) it is identified that the language models suffer
from vanishing gradients (Mosbach et al., 2021; Zhang et al., 2021b). The use
of optimiser without bias correction, further compounded by the use of small
number of training iterations, is identified as the cause of vanishing
gradients (Mosbach et al., 2021; Zhang et al., 2021b). As opposed to previous
papers, it is observed that training for longer does not lead to overfitting,
but instead reduces the vanishing gradients problems (Mosbach et al., 2021;
Zhang et al., 2021b). Additional cause of vanishing gradients is the sub-
optimal initialisation point for the upper layers, which are specialised and
cause significant variance when transferred (Zhang et al., 2021b).

Again, all these hypotheses are disputed (Wu et al., 2022) and instead under-
specification of the optimisation problem is proposed as the origin of the
variance (Wu et al., 2022; D’Amour et al., 2022; McCoy et al., 2020),
especially in out-of-distribution datasets (Wu et al., 2022; McCoy et al.,
2020). There exists multiple good solutions in the generalisation errors on
the source dataset, but with wildly different generalisation errors on the
target datasets, with the optimal solution lying in a non-flat region where
even a small perturbation leads to large difference in error (Wu et al., 2022;
McCoy et al., 2020). This makes the identically trained models to encode
different inductive biases (D’Amour et al., 2022) and making the choice of
minima rather arbitrary, being easily affected by small changes in
initialisation and order of examples (McCoy et al., 2020).

Finally, it is postulated that the large variance in performance is due to the
strong inter-example correlations in datasets (Zhou et al., 2020). When the
prediction of a single sample changes, this correlation causes simultaneous
change in large proportion of the predictions in the dataset. This was
observed when analysing the training trajectory in experiment, where the model
checkpoints at the same step of training from different initialisation had
significant difference in performances. The individual predictions of each
sample were highly unstable and changed constantly, which caused the large
fluctuations. However, no explanation for the unstable behaviour in individual
samples is provided, besides being an inevitable consequence of current
datasets (Zhou et al., 2020).

###  5.3. Origins of Randomness in In-Context Learning

The origins of randomness in in-context learning are analysed across two
randomness factors: 1) order of data (Zhao et al., 2021a; Lu et al., 2022);
and 2) choice of samples (Liu et al., 2022; Zhang et al., 2022a).

The significant effects of different data orders were found to be the
consequence of highly unbalanced label distribution in the output (Zhao et
al., 2021a; Lu et al., 2022). The output distribution of the language models
is influenced by the biases present in them (Zhao et al., 2021a).
Specifically, three biases were identified that cause these effects: 1)
majority label bias, where the output is biased towards the more frequent
answer in the prompt (e.g., class imbalance in prompt); 2) recency bias, where
the model tends to output the labels that appear towards the end of the
prompt; and 3) common token bias, where the model is biased to output more
common tokens from the pre-training distribution. The combination of these
biases causes the model to have highly unbalanced label distribution that
needs to be calibrated to reduce the variance (Zhao et al., 2021a). On the
other hand, calibrating the distribution in order to reduce the variance and
prevent the degenerate behaviour of bad prompts was found to not be effective
(Lu et al., 2022). This leads to hypothesis that the biases are not a sole
cause of the highly unbalanced label distribution and the origin of randomness
should be traced to something different. However, other origins were not found
empirically, leading to claims that the sensitivity to ordering of samples is
a feature of in-context learning and a core part of every in-context learning
approach should be a way to select good performing prompt orderings (Lu et
al., 2022).

The random sampling of examples for the in-context learning was determined to
be the origin of randomness (Liu et al., 2022; Zhang et al., 2022a) that can
also cause problems observed when using different order of examples in
prompts. Choosing the samples randomly, without taking similarity or other
heuristics into consideration, many low quality demonstrations are selected
that may contain spurious correlations and may not provide enough information
(Zhang et al., 2022a; Liu et al., 2022). When low quality samples are used,
different permutations and calibration of output distribution do not help
reduce the variance, which disputes the claims in previous works (Zhang et
al., 2022a). Choosing high quality samples based on similarity or heuristics,
reduces the variance and allows different ordering and calibration to work
(Zhang et al., 2022a; Liu et al., 2022). Therefore, the real origin of
randomness, which can also lead to unbalanced distribution and sensitivity to
order, is the choice of low quality, dissimilar samples (Zhang et al., 2022a;
Liu et al., 2022).

###  5.4. Overview of Findings: Origins of Randomness

The observed origins of randomness can be summarised as follows:

  * •

Poor choice of samples that are used for training, adaptation or in-context
learning.

  * •

Overfitting that causes catastrophic forgetting, and models to focus on
shallow features.

  * •

Under-specification, where multiple local minima with same performance are
present in training data, which are not consistent with testing data.

  * •

Highly unbalanced output label distribution stemming from biases present in
language models and incorrect ordering of samples.

  * •

Optimisation problems caused by poor choices in hyperparameters and the
problem statement, such as not using bias correction, training for limited
number of iterations or poor initialisation and re-use of some neural network
layers.

The individual origins of randomness are closely tied to the machine learning
approach and the randomness factor investigated. This is mainly the
consequence of the experiments that are designed to observe the behaviour of
different approaches. Although many different origins are identified, all are
tied to small number of randomness factors, specifically the choice of
samples, order of the samples in training and the model initialisation.

The popularity of different machine learning approaches and randomness factors
is clearly shown in this task for addressing the effects of randomness. The
focus on determining the origins of randomness in meta-learning is limited,
while language model fine-tuning is most popular.

The first effects of interactions between randomness factors are observed. In
many cases, the different origins of randomness identified are later disputed
and denoted as a simple consequence of other effects of randomness. For
example, it is observed that choice of samples in in-context learning causes
problems that are often incorrectly attributed to the ordering of these
samples.

This further shows the prevailing inconsistency of results from different
papers. The compounding effects between different randomness factors cause
significant problems when not properly addressed. Another aspect that plays
role is the simple evaluation, as the interactions are clearly shown when more
comprehensive evaluation is utilised.

##  6\. Mitigating the Randomness

In the mitigation task, the effects of randomness are mitigated to reduce the
variance as much as possible and to improve the overall performance using
mitigation strategies. We divide these mitigation strategies into two separate
groups: 1) general mitigation strategies; and 2) problem-specific mitigation
strategies.

The general mitigation strategies can be used to mitigate the effects of
randomness stemming from any origin of randomness and randomness factor. So
far, the only way to deal with any origin of randomness is to repeat the
training process multiple times, each time with different randomness factor
configuration (called multi-run mitigation strategy). Even though such
strategy do not really mitigate the variance, the idea is that aggregating
over enough repeats of the training runs leads to more objective results, as
the underlying distribution can be better estimated and the expected behaviour
can be better derived from it (Bouthillier et al., 2021; Dehghani et al.,
2021).

Although the multi-run mitigation strategy is implicitly used in investigation
and recognise papers (due to the repeated runs), it is explicitly mostly used
in combination with the data split randomness factor, as it can be viewed as a
modification of the cross-validation. Using this strategy, multiple random
data splits are macro-averaged together to produce more objective results
(Mukherjee et al., 2021). The strategy is also modified to increase its power
and reduce the number of repeated runs needed. One such modification is the
use of fixed ratio splitting without constraints instead of cross-validation
(Zheng et al., 2022). The fixed ration splitting can better approximate unseen
data as some samples appear only in training or test split of data. Even
though it is mostly used with the data split randomness factor, it can be
extended for use on any randomness factor. In some cases, it is the only
available solution for mitigating the randomness, i.e., when the randomness
factor cannot be easily controlled, such as the effects of implementation of
deep learning frameworks and hardware.

On the other hand, the problem-specific mitigation strategies are designed to
deal with a specific origin of randomness, making them dependent on the
randomness factor and the origins of randomness as analysed in Section 5.

###  6.1. Poor Choice of Samples

The variance in performance resulting from the choice of samples can be
introduced at 2 separate levels of the training process: 1) at the start of
training, when sampling a sub-set of representative samples that are used
during training; and 2) during a single step of training, when sampling the
data used for the specific training iteration. The first level represents the
real world scenario where there is only a limited labelling budget available
that needs to be effectively utilised by choosing samples with as much
information as possible. By definition, the choice at the start of training
process should have a significant influence on the choices in the individual
training iteration, leading to interactions between these two levels. In case
of fine-tuning, both choices can be merged into one, as all the training
samples are usually used at each step of training.

####  6.1.1. Selecting Representative Samples

When choosing representative samples for training, each proposed approach
assumes the existence of a large set of unlabelled data from which the samples
are chosen for annotation (Chang et al., 2021; Zhang et al., 2022a). When
selecting only small number of samples ($<100$), a simple clustering can be
used to select the representative samples (Chang et al., 2021). To choose K
samples, the whole set of unlabelled data is clustered using K-means
clustering into K clusters. The samples closest to the centre of each cluster
are selected to form the set of K samples for training (Chang et al., 2021).
To maximise the probability that the clusters are spread evenly across the
data, the initial centroids are chosen based on the probability proportional
to the squared distance from all the other existing centres instead of
choosing them randomly (Chang et al., 2021). Another possibility to select the
samples is to use reinforcement learning, similarly to active learning (Zhang
et al., 2022a).

The problem of effective sample choice is also present in cross-lingual
learning, when choosing what samples to translate to different languages. One
proposed solution is to train a linear model that determines the gain in
performance and stability, when a sample with given characteristics is
translated and used for training (Jundi and Lapesa, 2022). As dataset for
training this linear model, the performance scores from a large set of cross-
lingual experiments where random samples are manually translated is used. The
linear model is trained to predict the performance scores using language
features and similarity scores between embeddings (Jundi and Lapesa, 2022).
Afterwards, the model can be used to select which samples to select for
translation based on the added gain (Jundi and Lapesa, 2022).

####  6.1.2. Sampling Data for Training Iteration

To select better samples for training iteration, the proposed solutions use
heuristics instead of random sampling (Liu et al., 2022; Agarwal et al.,
2021). One possibility is to select samples based on a similarity using
k-nearest neighbour (Liu et al., 2022). Selecting samples closest to the test
ones can significantly increase the stability without incurring significant
computation overhead when number of samples is small (Liu et al., 2022).
Another possibility is to use adversarial training on the worst-case
adaptation samples (found using greedy search) in meta-learning (Agarwal et
al., 2021). However, it provides no significant improvement in regards of
stability for specific meta-learning approaches (Agarwal et al., 2021).

Last, but not least, curriculum learning can be used to select optimal samples
for each training step (Zhang et al., 2021a). In this approach, the difficulty
of samples is first determined using several teacher models that are trained
independently on the data. These teacher models separate the data into
multiple buckets based on their difficulty. Afterwards, the instances are
sampled from the buckets, starting from the easiest and gradually increasing
in difficulty.

###  6.2. Overfitting and Catastrophic Forgetting

How the overfitting and catastrophic forgetting is mitigated is closely tied
with the approach for which it is designed. Therefore, we provide further
categorisation based on these approaches

####  6.2.1. Overfitting in Meta-Learning

One solution to reduce the overfitting in meta-learning approaches is the
augmentation of the support and query data, and the task construction (Osahor
and Nasrabadi, 2022). Applying the augmentation to the support and query data
increases the number of samples for training which improves the performance by
making the training more robust. The augmentation applied to the tasks
increases the number of classes per individual tasks trained on, improving the
performance as well by providing more diverse tasks and improving
generalisation (Osahor and Nasrabadi, 2022).

Another solution is the use of variance reduction algorithms in the
optimisation based meta-learning (Wang et al., 2021a; Yang and Kwok, 2022).
One possibility is to focus on first-order meta-learning algorithms and
introduce the variance reduction only to them (Wang et al., 2021a). The
variance reduction term is introduced to the gradient estimator in task
adaptation stage, motivated by the recursive momentum technique in (Cutkosky
and Orabona, 2019). This term is initialised by averaging the gradients from
randomly sampled tasks using the initial parameters and then is further
updated using a weighted sum of mini-batch stochastic gradients across all
sampled tasks (Wang et al., 2021a). As this foregoes the benefits of the bi-
level meta-learning process, other papers propose to modify the variance
reduction term (STORM (Cutkosky and Orabona, 2019; Levy et al., 2021)) for use
on bi-level meta-learning optimisation, in combination with large number of
training steps to reduce the amount of storage required (Yang and Kwok, 2022).
To achieve this, the variance-reduced gradient estimate is computed using
stochastic gradients from two successive iterates, while evaluating the
gradient using the previous two iterates on the current batch of samples (Yang
and Kwok, 2022).

####  6.2.2. Overfitting in Language Model Fine-Tuning

The augmentation technique, in combination with regularisation, is also
proposed when fine-tuning language models (Meng et al., 2022). The pre-trained
language model is used as a generator for new data samples used for
augmentation. To guarantee that samples with discriminative labels are
generated, a meta-weighted maximum likelihood objective can be used for the
tuning of the generator. As the resulting labels still contain some level of
noise, a noise-robust procedure for training the final model is proposed as
regularisation using label smoothing and temporal ensembling (Meng et al.,
2022).

Another solution is to look at the disconnection of problem statement between
pre-training and fine-tuning stage. It is common occurrence that the objective
and even the input and output distributions are different which causes the
fine-tuning process to be brittle, especially when only small amount of data
is available (Phang et al., 2019). A second pre-training step on a data-rich
supervised tasks can be used to better prepare the model for fine-tuning
stage, mitigating the brittleness and tendency to overfit on data (Phang et
al., 2019).

Last set of approaches utilises regularisation to prevent overfitting.
Injecting Gaussian like noise vector into the input can be used as a stability
regularisation, similar to augmentation (Hua et al., 2021). Another
regularisation technique is to perform fine-tuning update only on a sub-
network instead of the whole network (Zhang et al., 2018; Xu et al., 2021c;
Zhang et al., 2022b). Mixup, introduced in (Zhang et al., 2018), randomly
selects sub-network with a given probability and replaces the parameters with
pre-trained weights, while updating the rest of the network. Instead of random
selection, CHILD-TUNING, introduced in (Xu et al., 2021c), uses Fisher
Information to determine the importance of parameters based on all samples
before training. Afterwards an unchanged sub-network with the most important
parameters is selected for the update. The previous two approaches can be
improved using Dynamic Parameter Selection, introduced in (Zhang et al.,
2022b), which adaptively selects a promising sub-network to perform the fine-
tuning steps on. The sub-network is chosen dynamically, i.e., different sub-
network is potentially fine-tuned at each training step, based on importance
of the parameters. It greatly reduced the heavy computation costs introduced
by CHILD-TUNING (Xu et al., 2021c) that separate the process of fine-tuning
and decision which sub-network to optimise. In addition, it can better capture
the task-relevant parameters across sub-networks of different sizes and bring
stable improvement in various parts of the model.

###  6.3. Under-Specification

To deal with under-specification, models with stronger inductive biases that
can help distinguish between many local minima should be proposed (McCoy et
al., 2020; Zhou et al., 2020), or the training sets should be better designed
(McCoy et al., 2020; Zhou et al., 2020). The better sets should represent the
phenomena appearing in texts to reduce the probability of local minima not
transferring to testing data (McCoy et al., 2020). In addition, the tasks used
for large scale pre-training should adhere to structure and level of
compositionality (i.e., solving combination of multiple task leading to the
solution for a different, more complex task), and more diverse datasets in
terms of syntax and lexicon should be constructed (Zhou et al., 2020).

Another solution for dealing with under-specification is to utilise an
ensemble of models (Hidey et al., 2022; Khurana et al., 2021). One possibility
to construct the ensemble is to combine predictions from multiple trained
models using their uniform average (Hidey et al., 2022). Another possibility
is to use stochastic weight averaging (SWA) as regularisation for the
ensembles (Khurana et al., 2021). Instead of training the model multiple
times, SWA creates the ensemble by averaging over different snapshots of model
weights. Such ensemble explores other solutions around the local minima, while
also reducing computation cost as the ensembling at weight level needs only a
single training run (Khurana et al., 2021). Although the SWA method
consistently achieves better stability of performance than standard fine-
tuning when used over different values of random seeds, the increase in
stability is only slight (Khurana et al., 2021). In addition to ensembling,
distillation and co-distillation can also be used (Hidey et al., 2022).
Distillation can be viewed as the exact opposite of ensembling, where the
knowledge of an ensemble of models (or a single model) is compressed to less
number of parameters by training a student model on the prediction of a pre-
trained teacher model. In co-distillation, multiple peer models are trained in
parallel instead, each sharing architecture and training data. Each of these
techniques reduces the variance in results, improving stability (Hidey et al.,
2022).

###  6.4. Highly Unbalanced Label Distribution

To deal with highly unbalanced label distribution in the in-context learning
it is proposed to use calibration (Zhao et al., 2021a; Min et al., 2022). One
possibility is to use contextual calibration, where the output distribution is
first estimated using neutral prompts (Zhao et al., 2021a). The neutral
prompts contain only in-context samples with the testing sample having its
text replaced with “N/A”. Using multiple such prompts, the distribution
between labels can be estimated and then corrected using affine transformation
(Zhao et al., 2021a). For example, if the model outputs positive labels in
more than 50% of cases (e.g., in 65%) in binary classification, the testing
example is considered to be positive only when the confidence is higher than
this threshold (e.g., more than 65%). Instead of using neutral prompts, it is
possible to parameterise the distribution using Bayes rule (Min et al., 2022).
To use it, the language model is not presented with K concatenated samples in
a single prompt, but instead presented one sample at the time. Obtaining
output probabilities and multiplying them together can be used to estimate the
distribution, while also reducing the dependency on ordering of samples (Min
et al., 2022). Compared to direct distribution estimation, this provides
larger improvements in terms of stability when there are fewer training
samples, the data is imbalanced, the number of classes is large, or a
generalisation to unseen labels is required (Min et al., 2022).

Apart from calibration, the highly unbalanced distribution is addressed by
finding the best ordering of examples (Lu et al., 2022; Kumar and Talukdar,
2021). One possible approach to automatically generate good prompt orderings
is to use heuristics and entropy (Lu et al., 2022). A probing set is first
constructed from all permutations of randomly selected set of training samples
that are used to query the language model. Each ordering in the probing set is
scored using local and global entropy without the need for labelled test data.
This produces ranking for all the orderings, allowing the choice of best order
of samples that avoid the issue of unbalanced predictions (Lu et al., 2022).
Another possibility is to search over possible permutations of training
samples to find the best performing one using genetic algorithm (Kumar and
Talukdar, 2021).

###  6.5. Optimisation Problems

The mitigation strategies for dealing optimisation problems in language model
fine-tuning are simply a specific fixes for the poor choices that introduce
the effects of randomness. One of the fixes is to increase the number of
iterations considerably and introduce a bias correction and small learning
rate to avoid vanishing gradients (Mosbach et al., 2021; Zhang et al., 2021b).

Another solution is to modify the way the additional layers are initialised
(Dauphin and Schoenholz, 2019). Instead of initialising the layers randomly,
which can lead to bad initialisation, a meta-learning approach can be used to
find an initialisation that is amenable to learning using gradient descent
(Dauphin and Schoenholz, 2019). In addition, badly performing initialisation
can be detected as early as the second training iterations thanks to the high
correlation of performance at the start and the end of the fine-tuning process
(Dodge et al., 2020). Starting many runs with different initialisation and
stopping the ones with bad performance in the early stages of fine-tuning, it
is possible to search through the whole space of random initialisation to find
the best performing ones without any significant increase in computation cost
(Dodge et al., 2020).

Yet another solution is to also randomly re-initialise top layers of the
language models, in addition to the classification layer (Zhang et al.,
2021b). The number of re-initialised layers depends on the problem statement
and the model, with more complex models and datasets benefiting from larger
number of re-initialised layers, but helps only up to a certain point (Zhang
et al., 2021b).

To better deal with the sensitivity to sample choices, ordering of samples and
to deal with biases, the problem of in-context learning can be reformulated as
a meta-learning problem, i.e., in-context learning is combined with
optimisation meta-learning (Chen et al., 2022). The language model is directly
optimised to perform the simple sequence prediction on variety of tasks,
learning to better perform well on all of them. This allows the model to
perform better at test time, while requiring only prompts with instruction and
few samples without any further adaptation (Chen et al., 2022).

###  6.6. Overview of Findings: Mitigating Origins of Randomness

The proposed mitigation strategies are closely tied to the origin of the
randomness. As they are explicitly designed to deal with the specific origin
of randomness and the randomness factor it was observed around, the proposed
strategies are unusable for other randomness factors and origins of
randomness. The multi-run mitigation strategy is the only one usable on any
randomness factor. Due to this, some randomness factors can be mitigated only
using the multi-run strategy, such as the non-deterministic implementation of
deep learning frameworks and hardware.

Even though the mitigation strategies address specific origin of randomness,
many papers do not provide any kind of extensive analysis of where the
randomness originates from. In some cases, the works reference other papers
that found this origin, but in many cases no explanation is provided why the
mitigation strategy targets the specific origin of randomness.

Each mitigation strategy incurs additional computation cost. The most
significant increase is introduced by the multi-run mitigation strategy as it
requires large number of model training and evaluation runs. The expected
computation cost is several times higher than in mitigation strategies that
require only single training run. So far, no general mitigation strategy
exists that could provide mitigation without high computational costs. In case
of problem-specific mitigation strategies, the computation cost depends on
multiple aspects, such as how often it is applied (once before training run or
at each training step), how expensive is the approach used (genetic algorithm
or simple clustering), or how many repeated runs are needed (e.g., when
ensembling is used).

Evaluation of the effectiveness of the mitigation strategies is done in a
simple fashion by comparing the performance metric of the model after applying
the mitigation strategy. The comparison is done on a single aggregated value,
often with low number of repeated runs (sometimes even one), while foregoing
the reporting of standard deviation or confidence intervals. In addition, the
interactions between factors are completely disregarded.

Such simple evaluation that disregards the interactions may be the main reason
for inconsistency in results. Similar to the investigate and determine tasks,
multiple papers report contradictory findings. The mitigation strategy that
works in one paper provides no benefits in other.

Finally, only a single mitigation strategy is applied at one time in most
papers. The papers that use multiple mitigation strategies mostly use
heuristics to deal with optimisation problems, such as increasing the number
of training runs and introducing bias correction into the optimiser.

##  7\. Comparing and Benchmarking: The Impact of Randomness in Low Data
Regimes

In this task, the effects of randomness are taken into consideration when
comparing and benchmarking different approaches for learning with limited
labelled data. As the effects of randomness can have significant effect when
comparing different approaches, or when designing benchmarks, modifications to
the process of benchmarking and comparing is required. Even though the
benchmarks and comparisons between approaches are closely related, the
proposed modifications that can deal with effects of randomness look at the
problem from different perspectives.

When simply comparing the approaches, the modifications are introduced at the
end of training. The design of the modified comparisons is independent from
the training process and, in theory, can be applied on any dataset, model, or
even in benchmarks. The independence from the training process makes the
modifications easier to design, but often require specific assumptions, such
as performing multiple runs. The modifications are usually designed to be
simple, often taking form of best practices for the comparisons (e.g., use
statistical significance tests on distribution of results). An important
aspect for the comparisons is what metrics are reported.

On the other hand, the benchmarks are specifically designed to take the
randomness into consideration from the start of training. The whole training
process is modified based on desiderata that address the randomness, making
benchmarks closely tied to the training process. Even though it makes the
benchmarks harder to design, they often provide better explanation and
comparative power as no assumption and constraints are needed.

###  7.1. Comparing Approaches in Low Data Regimes

Comparisons between different approaches in low data regimes are significantly
affected by the effects of randomness. Comparing on single value, the ranking
of models can vary significantly across repeated training runs that differ
only in the randomness factor configurations (Dehghani et al., 2021; Dodge et
al., 2020; Reimers and Gurevych, 2017). However, using multiple runs and
comparing based on an aggregated values from these runs, with or without
standard deviation, is also identified as non-satisfactory, as it cannot
easily deal with the variance (Zhong et al., 2021; Dror et al., 2019). If a
model achieves better aggregated performance, but with higher standard
deviation, it cannot be confidently concluded that it is significantly better.

To deal with the effects of randomness, multiple best practices and
modifications to evaluation practices are proposed. All of these modifications
and best practices propose to compare on multiple evaluation runs to achieve
unbiased comparisons (Schick and Schütze, 2021; Reimers and Gurevych, 2017;
Bouthillier et al., 2019; Dehghani et al., 2021; Sellam et al., 2022;
Bouthillier et al., 2021; Dror et al., 2019).

A simple modification is to select a set of samples beforehand for each run
and compare the models on the same set of fixed samples (Schick and Schütze,
2021). Although it reduces the bias in results, the comparison is still done
on a single aggregated value. Therefore it may not be possible to confidently
draw conclusions about the difference in the behaviour of the models.

To capture the behaviour of different models, the comparison should be done
based on their score distribution from multiple runs instead of a single
aggregated value (Reimers and Gurevych, 2017; Bouthillier et al., 2019;
Dehghani et al., 2021; Sellam et al., 2022; Bouthillier et al., 2021; Dror et
al., 2019). Treating the results from multiple runs as random samples, it is
possible to compare the models using statistical significance testing
(Dehghani et al., 2021). Drawing conclusions from such comparisons reduces the
risk of rejecting promising approaches or falsely accepting weaker ones
(Reimers and Gurevych, 2017). In addition, comparing score distributions from
multiple runs allows to identify situations in which the models fail and to
identify the shared properties between different models on the corresponding
tasks that can be used for further comparisons (Bouthillier et al., 2019;
Zhong et al., 2021). For example, architectures that differ only in number of
parameters can be compared based on the fraction of decaying instance (i.e.,
instances on which models with larger number of parameters are worse) between
models of different sizes (Zhong et al., 2021).

However, comparing between score distributions may not be as straightforward
in all cases (Bouthillier et al., 2021; Sellam et al., 2022; Dror et al.,
2019). To provide truly unbiased comparisons, as many sources of randomness
(i.e., randomness factors) as possible should be randomised, while exploring
different sets of hyper-parameters for the models (Reimers and Gurevych, 2017;
Bouthillier et al., 2021). This not only leads to better evaluation of the
variance associated with each source of randomness, but the error of the
expected performance is also reduced. However, such comparisons start to
become problematic when the evaluated models are not under our control (i.e.,
behind API) and the only controllable variance is the data we use. In
addition, accounting for all sources of randomness is often not feasible due
to the computational cost of running training multiple times (Bouthillier et
al., 2021). A statistical bootstrap approach can be used to estimate the
distribution of results using repeated sampling of the results, without the
need for extensive training runs (Sellam et al., 2022). Finally, even
statistical tests may not compare the models with sufficient confidence (as
only a sample of score distribution is observed) when not adhering to more
strict set of criteria (Dror et al., 2019).

Besides simple metrics, multiple papers propose to report additional ones that
can allow for more unbiased comparison using statistical tests. Such metrics
include standard deviation, maximum, minimum, mean and confidence intervals
(Schick and Schütze, 2021; Bragg et al., 2021), the score distributions and
individual episode results from multiple runs, data splits, or even across
different datasets (Dehghani et al., 2021; Dror et al., 2019; Zheng et al.,
2022; Mukherjee et al., 2021; Bragg et al., 2021), or the computation time and
memory required (Bouthillier et al., 2021). This also includes assumptions and
constraints of the models, such as reporting whether the model is designed for
out-of-distribution evaluation and should not be used in-distribution (Setlur
et al., 2021b). Last, but not least, reporting situations in which models fail
and properties of models on different tasks should help with comparing the
models based on the implicit biases and in identifying parts that can still be
improved (Bouthillier et al., 2019).

###  7.2. Benchmarks Taking Randomness Into Consideration

Multiple benchmarks designed for evaluating few-shot learning are starting to
take the problem of randomness and its effects on the stability into
consideration (Bragg et al., 2021; Mukherjee et al., 2021; Setlur et al.,
2021b; Ye et al., 2021; Zheng et al., 2022; Efrat et al., 2022). However, the
extent to which the randomness is addressed is different across the
benchmarks.

In the most simple case, only some of the properties related to stability
and/or randomness are included in the design of the benchmark (Bragg et al.,
2021; Mukherjee et al., 2021; Setlur et al., 2021b; Ye et al., 2021). This
includes focus on variable number of shots and classes in few-shot learning to
better simulate real world scenarios and to test robustness of models to these
systematic changes (Bragg et al., 2021), comparing the performance from
multiple runs with the human performance (Mukherjee et al., 2021), reporting
of additional properties such as confidence intervals, standard deviation and
individual results from different episodes of training to allow for paired
statistical tests (Bragg et al., 2021), providing means to evaluate both in-
distribution, as well as out-of-distribution (Setlur et al., 2021b), or
looking at the behaviour of models in cross-task settings (Ye et al., 2021).

Later benchmarks also explicitly encourage the evaluation on multiple training
runs (Mukherjee et al., 2021; Ye et al., 2021; Zheng et al., 2022; Xu et al.,
2021b). Such benchmarks mostly provide multiple splits of data for training
(Mukherjee et al., 2021; Ye et al., 2021; Zheng et al., 2022; Xu et al.,
2021b), while providing either a single set of test data (Mukherjee et al.,
2021; Xu et al., 2021b) or providing multiple test data splits (Ye et al.,
2021; Zheng et al., 2022). To explore the behaviour in cross-task settings,
the splitting is also done on the level of tasks, with some tasks being used
only for testing purposes (Ye et al., 2021).

Yet further benchmarks focus on accurately estimating the differences between
approaches by taking into consideration the effect of randomness on hyper-
parameters (Zheng et al., 2022). Instead of using a fixed set of hyper-
parameters, which was found to be sub-optimal, a search across the hyper-
parameter space is done over multiple splits of data. To allow for better
search over the hyper-parameter space, the splitting strategy is modified
based on three desiderata (Zheng et al., 2022): 1) a good data split strategy
should result in optimal hyper-parameters on test set; 2) development and test
sets should be correlated over the whole hyper-parameter space distribution,
i.e., changing hyper-parameters should have no effect on performance
correlation between development and test dataset; and 3) changing number of
splits should have no impact on the performance and its correlation.

Finally, some benchmarks are specifically designed to determine the
sensitivity of the approaches to the effects of small, random changes (Efrat
et al., 2022). Instead of focusing on large and complex tasks, the benchmark
introduces a compact set of tasks that are short, can be quickly and
automatically evaluated, and are trivial for humans, i.e., tasks that
elementary school students should perform perfectly at. This includes tasks
such as writing a sentence containing a specific word (which is mentioned in
instructions), or choosing which of the 2 or more words are longer. The
effects of randomness are directly estimated by changing the order of specific
words in instructions (e.g., switching the words whose length should be
compared), replacing some words (e.g., generate sentence containing word ”cat”
instead of ”dog”), changing instruction template, or measuring the gap between
two similar tasks. To further avoid confounding effects in stability, the
evaluation is performed in zero-shot setting. By multiplying the achieved
accuracy with the mean sensitivity score (which is calculated across the
different small, random changes), the benchmark explicitly penalises the
models for their brittleness.

###  7.3. Overview of Findings: Comparisons and Benchmarks

The comparisons are specifically designed to deal with the significant effects
of randomness when dealing with limited labels. Even when comparing across
multiple training runs, it was observed that using a single aggregated value
leads to biased comparisons. The most common modification is to compare
between result distributions using statistical tests instead. Although this
leads to more unbiased comparisons, there are still problems present.

Such comparisons are sensitive to the number of training runs, as the
distribution of results can be objectively determined only with larger number
of runs. However, running multiple training runs was found to be infeasible in
many papers. To deal with this problem, approaches that approximate the
distribution without the need for many training runs were proposed recently,
such as statistical bootstrapping.

The modified comparisons are utilised only for comparing between different
approaches across datasets and tasks. However, it is important to also compare
between effects of randomness factors, while taking the interactions into
consideration. Even though not specifically designed for it, the modifications
should be applicable to this problem as well.

When it comes to benchmarks, the randomness is only starting to be taken into
consideration, leading to very simple ways to address it. Most benchmarks
simply evaluate across multiple splits or training runs, while keeping the
number of runs quite small, or simply explore some properties related to
stability. In addition, even though multiple training runs are performed, many
benchmarks compare the models only on a single aggregated value. Only a single
benchmark explicitly measures the sensitivity of models to effects of
randomness and uses it to penalise the final performance metric.

##  8\. Challenges, Open Problems and Next Steps

Based on the findings from the comprehensive literature overview presented in
previous chapters, we identify seven challenges and open problems for
addressing the effects of randomness on stability of learning with limited
labelled data.

###  8.1. Focus on Small Fragment of Randomness

The scope of investigating the effects randomness, determining its origin and
the mitigation of these effects is limited in its extent. The problem can be
observed on multiple levels:

  * •

Approaches and modality \- the focus is mostly on the approaches utilising
language models (fine-tuning and in-context learning), while focus on other
machine learning approaches, such as meta-learning, is severely limited in its
extent. The majority of the focus is on natural language processing, as it is
the modality used in language models, limiting the analysis of behaviour on
other modalities. Therefore the effects of randomness are addressed only on a
specific sub-part of learning with limited labelled data, even though the
randomness also has significant effects on other machine learning approaches
(investigated in this survey).

  * •

Randomness factors \- many factors receive limited attention, such as the
effects from splitting data, choosing what data is labelled, or the impact of
non-deterministic implementation and hardware. Instead, the focus is on
factors in later stages of the training process, such as ordering the samples
and model initialisation (hidden behind random seed). This limits the analysis
of randomness to a few popular randomness factors. The under-researched
factors may still be important, but there is no consensus due to the limited
focus. Some papers claim the effects of these factors to be insignificant
(Bouthillier et al., 2019), while other consider them to be the most
significant contributors of variance (Bouthillier et al., 2021; Zheng et al.,
2022; Mukherjee et al., 2021).

  * •

Number of randomness factors \- the experiments are designed to jointly
analyse only a small number of factors (1-3). Such experiments compound the
limited focus on some factors, while negatively impacting the possible
findings from experiments, such as determining the importance of different
factors or their interactions. In addition, focusing only on specific, low
number of factors leaves the results of the investigation open to the still
present effects of other sources of randomness that may skew the results.

We argue that the scope should be widened across all tasks for addressing the
effects of randomness, to better explore the behaviour of different machine
learning approaches across different modalities and randomness factors. A
simple extension would be to perform multiple experiments, each for different
randomness factor across all the different approaches the factor is relevant
for. A more in-depth analysis would be to analyse the effects of randomness
across all factors in single experiment. However, increasing number of
randomness factors that are considered in the investigation significantly
increase the computation cost (Bouthillier et al., 2021). To overcome this
problems, more sophisticated experiments should be designed.

###  8.2. Limited Analysis and Explanation of Importance for the Effects of
Randomness Factors

A significant part missing when investigating the effects of randomness is a
thorough analysis of the importance for the different randomness factors. In
many cases, the importance of randomness factor is obvious, such as when it
causes large variance in results ($>10\%$). When the observed variance is
smaller, an analysis and an explanation why the difference in performance is
significant and why the effects of the randomness factor are important is
required. However, such analysis is mostly missing in the investigation
experiments. The investigation should provide us with a level of understanding
for the importance of the randomness factors. Especially when determining the
effects as single aggregated value. The effects measured as 1% standard
deviation have different significance when the performance is in high 90% than
when it is in low 20%. However, almost all papers simply present a single
aggregated value without further details.

In addition, it is common to work with limited computational budget in real
world scenarios (e.g., as a a part growing effort of green AI that aims to
limit the CO2 footprint of the AI). Analysing different randomness factors and
comparing the importance of their effects would allow us to identify which
factors are more important. Afterwards, we can identify how best to allocate
our resources to dealing with them, such as ignoring the less important ones.
Only few papers compare between factors, such as (Pham et al., 2021; Zhao et
al., 2021b; Jundi and Lapesa, 2022; Zhong et al., 2021; Zhao et al., 2021a;
Bouthillier et al., 2021, 2019).

Furthermore, the importance of randomness factors may not be consistent across
different approaches and modalities. However, there is no consistent
information whether this is the case, as the comparisons between effects of
randomness factors are limited, or even non-existent.

###  8.3. Disregarding Interactions Between Randomness Factors

Ignoring interactions between different randomness factors can lead to biased
results. We can observe that many papers often find contradictory findings
across all the tasks for addressing the effects of randomness. The effects of
interactions are most notable when evaluating the mitigation strategies, as
this task has the least developed evaluation strategies, often using simple
evaluation on a single randomness factor. We argue that the interactions
between randomness factors are one of the primary contributors to the
inconsistency in findings.

The interactions should be taken into consideration in every task for
addressing the effects of randomness, whether it is investigation of the
effects, identifying the origin of randomness, proposing mitigation
strategies, or even comparing and benchmarking different randomness factors
and areas of machine learning.

One possibility is to include all randomness factors in the process, such as
exploring through all the combinations of the randomness factor configurations
for each randomness factor. However, this incurs significant computational
cost due to the number of runs growing exponentially with the number of
randomness factors and so may be infeasible. Some papers already do this, but
only for a set of selected randomness factors with low number of randomness
factor configurations (Zhao et al., 2021a; Jundi and Lapesa, 2022; Zhao et
al., 2021b).

Another possibility is to mitigate the effects of randomness factors that are
not currently investigated using mitigation strategies. However, such approach
has not been utilised so far.