# The Flow of Ideas in Word embeddings

Debayan Dasgupta
Center for Nanoscience and Engineering
Indian Institute of Science, Bangalore
<EMAIL_ADDRESS>
Theranautilus Private Limited

###### Abstract

The flow of ideas has been extensively studied by physicists, psychologists,
and machine learning engineers. This paper adopts specific tools from
microrheology to investigate the similarity-based flow of ideas. We introduce
a random walker in word embeddings and study its behavior. Such similarity-
mediated random walks through the embedding space show signatures of anomalous
diffusion commonly observed in complex structured systems such as biological
cells and complex fluids. The paper concludes by proposing the application of
popular tools employed in the study of random walks and diffusion of particles
under Brownian motion to assess quantitatively the incorporation of diverse
ideas in a document. Overall, this paper presents a self-referenced method
combining microrheology and machine learning concepts to explore the
meandering tendencies of language models and their potential association with
creativity.

##  1 Introduction

The random walker was introduced to the world by Karl Pearson in 1905. Pearson
posed the question of determining the probability of a person starting from a
point and taking a series of straight-line walks and then calculating the
likelihood of ending up within a certain distance from the starting point.
Random walks have since garnered significant attention due to their
association with Brownian motion and diffusion processes, making them a well-
studied subject across numerous disciplines.

###  1.1 Random walk through similarity space

This paper introduces a random walker in the word embedding space. Our
objective is to study the flow of ideas led by semantic similarity. The flow
of ideas has been closely linked to quantifying creativity. Prior research has
already highlighted the significance of correlating and processing diverse
topics as an indicator of creativity. For instance, studies by Olson et al.
(2021) have demonstrated that the ability to associate unrelated words can
predict creative thinking. Moreover, semantic distance has been utilized as a
measure of creativity in human studies in psychology. (see: Kenett (2019),
Beaty & Kenett (2023),Heinen & Johnson (2018)).

###  1.2 Shakespeare and HG Wells

Figure 1: (a) A representative shape of the typical MSD curve observed for a
large language corpus. (b) The inset graph shows the general behavior of MSD
curves for diffusing particles. These behaviors have been adapted for this
paper to explain the behavior of similarity walks. The particle trajectory
represents the Brownian motion of particles in confined structures with a
narrow probability of escape. (c) MSD of a Henry VI text corpus using OpenAI’s
text-ada-002 embeddings. Some equivalence with the hypothesized behavior in
(a) is observed. (d) MSD curve of popular writings by Shakespeare and H.G.
Wells demonstrating similar behavior

To quantitatively assess the ability of a document to interconnect diverse
ideas, we draw upon a popular tool employed in the study of random walks and
diffusion of particles under Brownian motion. Such analysis may offer valuable
insights into the dynamics and patterns of idea flow, providing a means to
explore and analyze the creative tendencies of language models. In the case of
sentence embeddings for a document, a mean squared distance curve can be
defined for different intervals between the position of the sentences in the
text corpus. In Figure 1, this is represented as the delay in the x-axis.

Assume a document containing $\displaystyle N$ sentences. The $\displaystyle
k^{th}$ sentence can be embedded to a vector represented by
$\displaystyle\vec{e_{k}}$. The distance between two sentence pairs can be
measured as:

$\displaystyle d_{ij}=\vec{e_{i}}-\vec{e_{j}},1\leq i<j\leq N$

where $\displaystyle i$ and $\displaystyle j$ are the location of the
sentences in the document. The delay is simply:

$\displaystyle\Delta(sentence)=j-i$

There are thus many distinct distances for small delays and very few for large
delays. The mean displacement would be analogous to diffusion due to Brownian
motion for a purely random language meandering through a sufficiently large
corpus of text, where the sentences or words appear without any correlation to
previous sentences. The interesting statistical quantity to consider here is
the mean squared distance (MSD) of the sentences at given delays. Mean squared
displacement analysis is often used to study the type of diffusion regime
undergone by a diffusing particleMichalet (2010). The MSD for a particular
$\displaystyle n$ sentence delay for the corpus with $\displaystyle N$
sentences can be defined as:

$\displaystyle
MSD(\Delta_{n})=\frac{1}{N-n}\sum_{i=1}^{N-n}(\vec{e_{i+n}}-\vec{e_{i}})^{2},n=1,...,N-1$

With this definition, it is easy to observe some patterns in MSDs of well-
known novels. We hypothesize that similar patterns can be found in most
creative endeavors in human literature. As shown in Figure 1(a), the mean
squared distance shows a ballistic rise in short delays between sentences.
This phase has been marked as phase I in the MSD graph. In this phase, about
10 to 100s of subsequent sentences are used to explain the reference sentence.
Here, new ideas, characters, and story plots are quickly introduced. In
attempting to provide context and further information about the reference
sentence, the author uses words and sentences that are somewhat semantically
distant from the reference sentence.

The phase marked as II in Figure 1(a) can be considered more as a
consolidation phase. This is the part where the main story plays out. The same
characters and plots are discussed in more detail in 100 to 1000 sentences
after their introduction. Phase III is the surprise or creative phase. At a
delay of a few thousand sentences, the climax or the surprise occurs for the
reference sentence. We have occasionally observed a return to the initial
semantic distance, which can be attributed to ending passages where the
initial characters or plot are revisited after the story is complete.

This can be contrasted with random walks of particles, as shown in Fig 1(b)

We extend this analysis to see if similar patterns hold for primitive word
embeddings and generative language models.

##  2 Pretrained Word2Vec

Word2Vec Mikolov et al. (2013)is a neural network model that encodes semantic
information of the words in the corpus given an unlabeled training data. Their
cosine similarity calculates semantic similarity between word vectors. We use
this model due to the relative ease of using pre-trained models available in
the gensim library.

###  2.1 Free flow from single words and ’jabberwocky’

Figure 2: (a) Representation of a similarity walk. In a similarity walk, top
’n’ similar words are searched for, following which a word is randomly chosen,
and the same is repeated for ’k’ steps to complete a ’k’ step similarity walk.
(b) MSD curve of a few similarity walks undertaken in pretrained Word2Vec
models.

We try a ’similarity walk’ through a pre-trained word embedding model to test
if language models have similar properties as described in 1.2. We use
Word2Vec, a simple language model that positions word embeddings in a vector
space, where semantically and syntactically similar words are located close to
each other. In contrast, dissimilar words are farther apart.

A ’Similarity walk’ can be defined as a process where a word is randomly
chosen or provided as input by the experimenter. From the pre-trained model, a
list of the top ’n’ most similar words is identified, and a word is randomly
chosen from those ’n’-similar words for the next step. If this process is
repeated ’k’ times, it would be considered a k-step similarity walk through
the corpus. This is schematically represented in Figure 2(a). The objective is
identifying patterns in how a one-word idea might flow within a language
model.

We observe that word vectors eventually converge into a cluster of closely
linked obscure words by following similar words. This leads to a path from
which the probability of finding newer words through a similarity walk becomes
almost zero.

In Figure 2(b), we do an MSD analysis of the words encountered in the
similarity walk, which upholds the above observation. The starting words
chosen purposefully by the author are colored blue and represent a biased
choice for the starting idea. The MSD curve in red and black was randomly
selected from the corpus and are more representative of the type of behavior a
similarity walk exhibits. The difference between biased choice and random
choice for starting words is highlighted because it was often observed that
common English words often discovered more unique words during a similarity
walk compared to a randomly chosen obscure word. This observation is empirical
and does not hold in general.

The interesting feature of these MSD plots is the near-flat curves or the
eventual approach to flatness over long distances for the blue curves. This
behavior is observed for sufficiently long similarity walks, irrespective of
the starting word. The walk always converges into a cluster of words unrelated
to the starting words but closely connected by similarity to each other. After
a similar walk falls into these clusters of words, it is unlikely to discover
any new words by continuing the walk. The path to this obscure cluster of
topics follows a Levy flight pattern, where word vectors oscillate around
similar ideas and occasionally take long-distance leaps in semantic space.
This descent into a ’jabberwocky’ of words is a universal property of all the
language models we have tested. This limitation appears due to embedding a few
words rarely used anywhere else in the corpus. For example, in the fasttext-
wiki-news-subwords-300 corpus, most of the walks would fall into a cluster of
usernames and mail addresses that refer to each other. A representation of
this behavior in dimensionally reduced projections for fifty different
starting words over three different pre-trained corpora is shown in Figure 3.

Figure 3:  tSNE visualization of fifty similarity walks on pre-trained
Word2Vec models. The models include: ”Glove-twitter-50’, ’fasttext-wiki-news-
subwords-300’ and ’word2vec-google-news-300’. The walk structure in 2D feature
space mimics foraging behavior where the forager occasionally takes a long
Levy flight.

###  2.2 Guided similarity walk

The results in 2.1 represent an unrestricted motion through the embedding
space with single words. However, modern language models rarely work with
single words. A better understanding of the dynamics of similarity walk can be
found with guided similarity walks. To simulate guided walks in Word2Vec
models we use multiple words as tethers to find the next similar word.

Figure 4: (a) A representation of a guided similarity walk, where the
following similar words are decided by the starting word and other words,
thereby restricting the choices of words that can be discovered through a
walk. (b) MSD curve of such similarity walks shows highly confined behavior.
(c) tSNE representation demonstrates how the walk becomes more closed and
lacks foraging behavior.

As shown in Figure 4(a), the most similar words for the next step in a guided
similarity walk consist of an original idea and two to three other guiding
words. This reduces the total vocabulary available for a similarity walk. The
constraints placed on similarity walk by the guiding words behavior shows up
in the MSD plot as a confined motion, as shown in Figure 4(b). It is also
interesting to observe that the magnitude of the distance moved during this
walk is comparatively smaller. This indicates a very tight confinement in the
embedding space, even though we experimented with words that are supposed to
be semantically distant. The TSNE plot in Figure 4(c) is a low-dimensional
representation of how the nature of the random walk has changed under
constraints. While a freely diffusing word showed Levy Flight-like behavior, a
constrained walk looks more like a closed loop.

##  3 Similarity walks with sentence transformers

Natural language processing has seen a paradigm shift with the introduction of
the transformer model. The astounding progress in generating coherent replies
to user queries has been mainly due to the dense and rich sentence embeddings.
In this section of the paper, we look at similarity walks through sentence
embeddings.

###  3.1 Similarity walks through Sentence transformers

Figure 5: (a) A tSNE representation of similarity walk through sentence
transformers (b) MSD curve showing a confined behavior for approximately 300
sentences, beyond which the similarity walk falls into a jabberwocky.

We used the distilBERT model Sanh et al. (2019) to perform a similarity walk
through sentence embeddings. The corpus for our experiment was chosen from
among the books in Project Gutenberg through the NLTK library Bird et al.
(2009) Sentence Transformers show similar behavior where after about 300
sentences, the recursive querying for similar sentences in the corpus
eventually converges into a cluster of closely linked sentences that carry
very little semantics and are obscure in the context of the corpus. An example
of this behavior can be found in the supplementary csv files. In Figure 5(a),
a dimensionally reduced visualization of this behavior can be seen. Similarly,
divergence from the corralled behavior in the MSD as seen in Figure 5(b), also
highlights this pattern.

Similar experiments were also performed on large language models like alpaca-
Lora-7 b. The results are available in the supplementary csv files. With large
language models, generating similarity walks was computationally challenging
and will be explored later.

##  4 Discussion

In this paper, we introduce a random walker in the word embedding space to
study the flow of ideas led by semantic similarity. We find that the random
walker exhibits anomalous diffusion signatures, commonly observed in complex
structured systems such as biological cells and complex fluids. This suggests
that the flow of ideas in language models is not random but instead follows a
complex pattern.

We study the MSD curve of the embedding vectors to assess the incorporation of
diverse ideas in a document quantitatively. We find that the diffusion of
ideas in language models is correlated with the document’s creativity. This
suggests that the ability of language models to generate creative text is
related to their ability to flow ideas in a complex and nuanced manner. This
paper presents a self-referenced method combining MSD curve analysis and
machine learning concepts to explore the meandering tendencies of language
models and their potential association with creativity. The correlation
between the diffusion of ideas and the document’s creativity is stronger for
generative language models than for primitive word embeddings. This suggests
that generative language models can better incorporate diverse ideas and
generate creative text. Similarity walks can be extended to any source of
information, be it music, art or text. The existence of small, closely linked
clusters from which the chances of escape is narrow is an interesting part of
most language models. In the future, we plan to understand the information-
theoretic implications of such valleys. We plan to study the factors that
influence the flow of ideas in language models. By understanding these
factors, we can develop techniques for controlling the flow of ideas to
generate more creative text.

## References

  * Beaty & Kenett (2023) Roger E. Beaty and Yoed N. Kenett.  Associative thinking at the core of creativity.  _Trends in Cognitive Sciences_ , 27(7):671–683, jul 2023.  doi: 10.1016/j.tics.2023.04.004.  URL https://doi.org/10.1016%2Fj.tics.2023.04.004.
  * Bird et al. (2009) Steven Bird, Ewan Klein, and Edward Loper.  _Natural Language Processing with Python_.  O’Reilly, 2009.  ISBN 978-0-596-51649-9.  URL http://www.oreilly.de/catalog/9780596516499/index.html.
  * Heinen & Johnson (2018) David J. P. Heinen and Dan R. Johnson.  Semantic distance: An automated measure of creativity that is novel and appropriate.  _Psychology of Aesthetics, Creativity, and the Arts_ , 12(2):144–156, may 2018.  doi: 10.1037/aca0000125.  URL https://doi.org/10.1037%2Faca0000125.
  * Kenett (2019) Yoed N Kenett.  What can quantitative measures of semantic distance tell us about creativity?  _Current Opinion in Behavioral Sciences_ , 27:11–16, jun 2019.  doi: 10.1016/j.cobeha.2018.08.010.  URL https://doi.org/10.1016%2Fj.cobeha.2018.08.010.
  * Michalet (2010) Xavier Michalet.  Mean square displacement analysis of single-particle trajectories with localization error: Brownian motion in an isotropic medium.  _Physical Review E_ , 82(4), oct 2010.  doi: 10.1103/physreve.82.041914.  URL https://doi.org/10.1103%2Fphysreve.82.041914.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient estimation of word representations in vector space, 2013.
  * Olson et al. (2021) Jay A. Olson, Johnny Nahas, Denis Chmoulevitch, Simon J. Cropper, and Margaret E. Webb.  Naming unrelated words predicts creativity.  _Proceedings of the National Academy of Sciences_ , 118(25):e2022340118, 2021.  doi: 10.1073/pnas.2022340118.  URL https://www.pnas.org/doi/abs/10.1073/pnas.2022340118.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter, 2019.


We investigate the problem of determining the predictive confidence (or, conversely, uncertainty) of a neural classifier through the lens of low-resource languages.
By training models on sub-sampled datasets in three different languages, we assess the quality of estimates from a wide array of approaches and their dependence on the amount of available data. We find that while approaches based on pre-trained models and ensembles achieve the best results overall, the quality of uncertainty estimates can surprisingly suffer with more data. We also perform a qualitative analysis of uncertainties on sequences, discovering that a model's total uncertainty seems to be influenced to a large degree by its data uncertainty, not model uncertainty. All model implementations are open-sourced in a software package.

§ INTRODUCTION

In 1877, Italian astronomer Giovanni Schiaparelli described the existence of “canals” on the surface of Mars, a finding that was described by a contemporary as a “very important and perplexing [problem]” (, ; p. 355). It later turned out that the structures, originally termed canali in Italian, were simply mistranslated, since the word can also refer to (natural) channels of water. By that point however, the possibility of irrigation on the red planet had already sept into popular culture, and is still being referenced to this day.

In the meantime, translation has become a task that is increasingly performed by neural networks, which — in the face of a word such as canali — might simply fall back on the most likely translation given the training data. And while the error above seems fairly innocuous, there are more safety-critical scenarios in which such ambiguities matter and can potentially have negative real-word consequences.

Besides translation, there also exist other language-based problems in which the uncertainty surrounding a model prediction can convey critical information, such as medical analyses <cit.>, legal case data <cit.> or analyzing job applications <cit.>. Determining model confidence, or, conversely, uncertainty, consequently is an important mean to instill trust in end users and avert harm <cit.>.

While there exist many works on images <cit.> and tabular data <cit.>, the quality of uncertainty estimates provided by neural networks remains underexplored in Natural Language Processing (NLP). In addition, as model underspecification due to insufficient data presents a risk <cit.>, the increasing interest in less-researched languages with limited resources raises the question of how reliably uncertain predictions can be identified. This lets us pose the following research questions:

RQ1 What are the best approaches in terms of uncertainty quality and calibration?

RQ2 How are models impacted by the amount of available training data?

RQ3 What are differences in how the different approaches estimate uncertainty?

Schematic of our experiments. Training sets are sub-sampled and used to train LSTM-based models and fine-tune transformer-based ones, which are evaluated on in- and out-of-distribution test data.

Contributions 1 We address these questions by conducting a comprehensive empirical study of eight different models for uncertainty estimation for classification and evaluate their effectiveness on three languages spanning distinct NLP tasks, involving sequence labeling and classification. 2 We show that while approaches based on pre-trained models and ensembles achieve the best results overall, the quality of uncertainty estimates on OOD data can become worse using more data. 3 In a qualitative analysis, we also discover that a model's total uncertainty seems to mostly consist of its data uncertainty. 4 We make our experimental code and model implementations available open-source in separate repositories, aiding future research in this direction.[The model zoo is available under <https://github.com/Kaleidophon/nlp-uncertainty-zoo>, with the code for the experiments available under <https://github.com/Kaleidophon/nlp-low-resource-uncertainty>.]

§ RELATED WORK

Notions of Uncertainty In the absence of additional information, the introductionary example canali has two valid translations — canals and channels. This is an instance of data or aleatoric uncertainty, describing the irreducible ambiguity and noise in the data generating process. The other notion is model or epistemic uncertainty: Fitting parameters, there remains a degree of incertitude about the optimal values due to finite data. We can usually reduce this uncertainty by amassing more data,[That is, unless the model class we chose is too restrictive.] for instance by supplying a translation system with other meanings of canali. These two concepts form the basis for uncertainty estimation in Machine Learning <cit.>.

Uncertainty in NLP Since uncertainty estimation literature is manifold on image data, we dedicate this part to related works in the realm of Natural Language Processing. There are several examples trying to incorporate uncertainty into models to either increase trustworthiness or performance, for instance in Machine Translation <cit.>, Summarization <cit.>, Information Retrieval <cit.> and Active Learning <cit.>. To obtain uncertainties, <cit.> use Stochastic-Gradient Langevin Dynamics <cit.> to obtain posterior weight samples for a LSTM. <cit.> apply MC Dropout with determinantal point processes to transformers for Natural Language Understanding.

Several authors have also highlighted connections of multi-head attention to Bayesian inference <cit.>. <cit.> attempt to transfer the idea of prior networks <cit.> onto recurrent neural networks.

Another line of works investigates uncertainty properties themselves; For instance, <cit.> try to explain uncertainty estimates for BERT and RoBERTa. Another example is given by <cit.>, who use predictive uncertainty to explain hallucination in Language Generation. <cit.> similarly use uncertainty as a tool to investigate challenges of neural summarization approaches. Lastly, due to the way that uncertainty estimates are evaluated, investigating distributional shift in NLP is also of interest, for instance through the work of <cit.>, <cit.>, who focus on question answering and <cit.> for text classification. The most similar work to ours is the text classification uncertainty benchmark by <cit.>, however they do not consider the impact of data or language, and test a different selection of models.

Calibration Calibration denotes the property of a model's output to accurately reflect the true chance of a correct prediction — i.e. predicting a class with a confidence of $90 \%$ should yield the correct prediction for $90 \%$ of similar inputs, when repeated. There have been several studies testing this property in modern neural networks <cit.> and proposing ways to improve it <cit.>. In NLP, calibration as been explored for pre-trained models <cit.>, including on out-of-distribution data <cit.>, for neural machine translation <cit.> and for question-answering <cit.>. Likewise, authors have proposed several calibration schemes, for instance by focusing on classes of interest <cit.>, generating synthetic examples for regularization <cit.>, using richer input representations <cit.> and adapting prompts in a zero-shot  setting <cit.>.

§ METHODOLOGY

§.§ Models

We choose a variety of models that cover a range of different approaches based on the two most prominently used architectures in NLP: Long-Short Term Memory networks (LSTMs; , ) and transformers <cit.>. Inside the first family, we use the Variational LSTM <cit.> based on MC Dropout <cit.>, the Bayesian LSTM <cit.> implementing Bayes-by-backprop <cit.> and the ST-$\tau$ LSTM <cit.>, modelling transitions in a finite-state automaton, as well as an ensemble <cit.>. In the second family, we count the Variational Transformer <cit.>, also using MC Dropout, the SNGP Transformer <cit.>, using a Gaussian Process output layer, and the Deep Deterministic Uncertainty transformer (DDU; , ), fitting a Gaussian mixture model on extracted features. We elaborate on implementation details in <ref>.

§.§ Uncertainty Metrics

We employ the following metrics to quantify confidence or uncertainty — in all cases, lower values indicate lower confidence / certainty and conversely, higher values mean higher confidence / certainty. The following metrics were either chosen due to their frequent use in the literature, or because they are trying to capture uncertainty in a novel way.

Single prediction metrics We  distinguish between metrics suitable for models using only a single prediction (or using the mean of multiple predictions, e.g. for an ensemble). The most straightforward of them is the maximum softmax probability by <cit.>. A variant of this is the softmax-gap, measuring the difference between the two largest predicted probabilities <cit.>. Another common metric, predictive entropy, involves measuring the Shannon entropy of the output distribution, which is maximized for a uniform prediction:

\begin{equation*}
-\sum_{k=1}^K p_{\theta}(y=k|\bx) \log p_{\theta}(y=k|\bx)
\end{equation*}

Lastly, we consider the Dempster-Shafer metric <cit.>, defined as $ K / (K + \sum_{k=1}^K \exp(z_k))$, where $z_k$ denotes the logit corresponding to class $k$. It has been shown that probabilities for (ReLU) networks tend to saturate in the limit <cit.>, and since this metric considers logits, it might provide more informative estimates on OOD data.

Multiple prediction metrics For some of the included models, we can express uncertainty as some score based on a number of predicted distributions, e.g. from different ensemble members or forward passes for MC Dropout. Here we use the expectation with respect to the weight posterior to express the aggregation of multiple predictions, which will simply be evaluated using the mean of a number of Monte Carlo samples in practice. A simple uncertainty metric on this basis is the predictive variance between predictions for a class:

\begin{equation*}\begin{aligned}
\resizebox{\hsize}{!}{
- \Expect[\Big]{q(\theta)}{p_{\theta}(y=k|\bx)}\bigg)^2\bigg]$
\end{aligned}\end{equation*}

where the expectation is evaluated over multiple sets of parameters, e.g. stemming from different dropout masks. Another possibility lies in using the mutual information between the label and model parameters given the data and input sample, which was introduced by <cit.>:

\begin{equation}\label{eq:mutual-information}
\text{H}\bigg[\Expect[\Big]{q(\theta)}{p_{\theta}(y|\bx)}\bigg] - \Expect[\bigg]{q(\theta)}{\text{H}\Big[p_{\theta}(y|\bx)\Big]}
\end{equation}

where H denotes the Shannon entropy as used for predictive entropy. The two terms of this equation can be identified as the total entropy and the aleatoric uncertainty, respectively. In theory, the remaining epistemic uncertainty of the model — in form of the the mutual information — should be particularly high on OOD inputs.

Model-specific metrics Lastly, DDU by <cit.> uses the log-probability of the last layer network activation under a Gaussian Mixture Model fitted on the training set as an additional metric. Since all others models are trained or fine-tuned as classifiers, they are not able to assign log-probabilities to sequences.

Uncertainty for sequences Since some tasks require predictions for every time step of a sequence, we determine the uncertainty of a whole sequence in these cases by taking the mean over all step-wise uncertainties.[We also just considered the maximum uncertainty over a sequence, with similar results.] A more principled approach for sequences is for instance provided by <cit.>, and we leave the extension and exploration of such methods for different uncertainty metrics, models and tasks to future work.

§.§ Dataset Selection & Creation

Language     Task     Dataset     OOD Test Set     [tr]# ID / OOD.     [tr]Sub-sampled
Training Set Sizes

EN     [tl]Intent
Classification     [tl]Clinc Plus  <cit.>     Out-of-scope voice commands     [tr]15k / 1k     [tr]15k / 12.5k / 10k

DA     [tl]Named Entity
Recognition     [tl]Dan+ News <cit.>     Tweets      [tr]4382 / 109      [tr]4k / 2k / 1k

FI     PoS Tagging     [tl]Finnish UD Treebank  (, ;
, ;
, )     [tl]Hospital records, online forums,
tweets, poetry     [tr]12217 / 2122      [tr]10k / 7.5k / 5k

Datasets. The original and sub-sampled number of sequences for experiments are given on the right.

In-distribution training sets We choose three different languages, namely English (Clinc Plus; , ), Danish in the form of the Dan+ dataset <cit.> based on News texts from PAROLE-DK
<cit.>, Finnish (UD Treebank; , ; , ; , ), corresponding to NLP tasks such as sequence classification, named entity recognition and part-of-speech tagging. An overview over the used the data is given in <ref>. We do use standardized low-resource languages in the case of Finnish and Danish, and simulate a low-resource setting using English data.[The definition of low-resource actually differs greatly between works. One definition by <cit.> advocates the usage for (would-be) standardized languages with a large amount of speakers and a written tradition, but a lack of resources for language technologies. Another way is a task-dependent definition: For dependency parsing, <cit.> define low-resource as providing less than $5000$ annotated sentences in the Universal Dependencies Treebank. <cit.> lay out a task-dependent spectrum, from a several hundred to thousands of instances.] Starting with a sufficiently-sized training set and then sub-sampling allows us to create training sets of arbitrary sizes. By using languages from different families, we hope to be able draw conclusions that generalize across a single language. We employ a specific sampling scheme that tries to maintain the sequence length and class distribution of the original corpus, which we explain and verify in <ref>.

Out-of-distribution Test Sets While it is possible to create OOD text by for instance withholding classes from the training set or appending text from a different source <cit.>, we choose to pick entirely new OOD test sets that are qualitatively different: Out-of-scope voice commands by users in <cit.>,[Since all instances in this test set correspond to out-of-scope inputs and not to classes the model was trained on, we cannot evaluate certain metrics in <ref>.] the Twitter split of the Dan+ dataset <cit.>, and the Finnish OOD treebank <cit.>. In similar works for the image domain, OOD test sets are often chosen to be convincingly different from the training distribution, for instance MNIST versus Fashion-MNIST <cit.>. While there exist a variety of formalizations of types of distributional shift <cit.>, it is often hard to determine if and what kind of shift is taking place.
<cit.> define near OOD as a scenario in which the inlier and outlier distribution are meaningfully related, and far OOD as a case in which they are unrelated. Unfortunately, this distinction is somewhat arbitrary and hard to apply in a language context, where OOD could be defined as anything ranging from a different language or dialect to a different demographic on an author or speaker or a new genre.
Therefore, we use a similar methodology to the validation of the sub-sampled training sets to make an argument that the selected OOD splits are sufficiently different in nature from the training splits. The exact procedure along some more detailed results is described in <ref>.

§.§ Model Training

Unfortunately, our datasets do not contain enough data to train transformer-based models from scratch. Therefore, we only fully train LSTM-based models, while using pre-trained transformers, namely BERT (English; , ), Danish BERT <cit.>, and FinBERT (Finnish; , ), for the other approaches. The whole procedure is depicted in <ref>. The way we optimize models is provided in <ref>. We list training hardware, hyperparameter information in <ref>, with the environmental impact described in <ref>.

§.§ Evaluation

Apart from evaluating models on the task performance, we also evaluate the following calibration and uncertainty, painting a multi-faceted picture of the reliability of models. In all cases, we use the Almost Stochastic Order test (ASO; , ; , ) for significance testing, which is elaborated on in <ref>.

Evaluation of Calibration First, we measure the calibration of models using the adaptive calibration error (ACE; , ), which is an extension of the expected calibration error (ECE; , ; , ).[See <ref> for a short overview over the differences.] Furthermore, we use the frequentist measure of coverage <cit.>. Coverage is based on the prediction set $\hat{\mathbb{P}}(\mathbf{x})$ of a classifier given an input, which includes the most likely classes adding up to or surpassing $1 - \alpha$ probability mass. A well-tuned classifier should contain the correct class in this very set, and minimize its width. The extent to which this property holds can be determined by the coverage percentage, i.e. the number of times the correct class in indeed contain within the prediction set, and its cardinality, denoted simply as width.

Evaluation of Uncertainty We compare uncertainty scores on the ID and OOD test set and measure the area under the receiver-operator curve (AUROC) and under the precision-recall curve (AUPR), assuming that uncertainty will generally be higher on samples from the OOD test set.[We thus formulate a pseudo-binary classification task as common in the literature, using the model's uncertainty score to try to distinguish the two test sets. Note that we do not advocate for actually using uncertainty for OOD detection, but only use it for evaluation purposes, since uncertainty on OOD examples should be high due to model uncertainty.] An ideal model should create very different distributions of confidence scores on ID and OOD data, thus maximizing AUROC and AUPR. However, we also want to find out to what extend uncertainty can give an indication of the correctness of the model, which is why we propose a new way to evaluate the discrimination property posed by <cit.> based on <cit.>: A good model should be less certain for inputs that incur a higher loss. To measure this both on a token and sequence level, we utilize Kendall's $\tau$ <cit.>, which, given two lists of measurements, determines the degree to which they are concordant — that is, to what extent the rankings of elements according to their measured values agree. This is expressed by a value between $-1$ and $1$, with the latter expressing complete concordance. In our case, these measurements correspond to the uncertainty estimate and the actual model loss, either for tokens (Token $\tau$) or sequences (Sequence $\tau$).

§ EXPERIMENTS

§.§ RQ1: Uncertainty & Calibration

2c|Task $\big( \text{ID} \big/ \text{OOD}\big)$      4c|Calibration $\big( \text{ID} \big/ \text{OOD}\big)$     4cUncertainty $\big( \text{ID} \big/ \text{OOD}\big)$

Model     Acc.$\uparrow$     $F_1\uparrow$     ECE$\downarrow$     ACE$\downarrow$     %Cov.$\uparrow$      $\varnothing$Width$\downarrow$     AUROC$\uparrow$     AUPR$\uparrow$     Token $\tau\uparrow$     Seq. $\tau\uparrow$

LSTM     $\wsrs{.79}{.00}$     $\wsrs{.62}{.01}$     $\wsrs{77.95}{.00}$      $\wsrs{.49}{.01}$     $\wsrs{\cmbold{1.00}}{.00}$     $\wsrs{144.00}{.00}$     $\srs{.88^{\text{\tiny \Plus}}}{.01}$     $\srs{.60^{\text{\tiny \Plus}}}{.01}$     [15mm]     $\wsrs{.75^{\bigcirc}}{.01}$

Bayesian LSTM     $\wsrs{.59}{.06}$     $\wsrs{.46}{.05}$     $\wsrs{77.66}{.05}$      $\wsrs{.22}{.01}$     $\wsrs{.88}{.00}$     $\srs{41.99}{1.94}$     $\srs{.86^{\bigtriangleup}}{.01}$     $\srs{.59^{\text{\tiny \XSolidBold}}}{.01}$     [15mm]     $\wsrs{.66^{\bigcirc}}{.02}$

LSTM Ensemble     $\wsrs{\cmbold{.81}}{.00}$     $\wsrs{\cmbold{.64}}{.01}$     $\wsrs{77.11}{.00}$      $\wsrs{.09}{.00}$     $\wsrs{.87}{.00}$     $\wsrs{4.27}{.05}$     $\srs{\cmbold{.92^{\text{\tiny \Plus}}}}{.00}$     $\srs{\cmbold{.71^{\text{\tiny \Plus}}}}{.01}$     [15mm]     $\wsrs{.73^{\Box}}{.01}$

Variational BERT     $\wsrs{.45}{.16}$     $\wsrs{.34}{.13}$     $\wsrs{77.92}{.02}$      $\wsrs{.22}{.06}$     $\wsrs{1.00}{.00}$     $\wsrs{115.11}{11.38}$     $\srs{.80^{\text{\tiny \XSolidBold}}}{.01}$     $\srs{.53^{\text{\tiny \XSolidBold}}}{.01}$     [15mm]     $\wsrs{.57^{\bigcirc}}{.09}$

-5*90English      DDU BERT     $\wsrs{.79}{.00}$     $\wsrs{.64}{.01}$     $\wsrs{\cmbold{77.02}}{.00}$      $\wsrs{\cmbold{.00}}{.00}$     $\wsrs{.82}{.00}$     $\wsrs{\cmbold{1.46}}{.04}$     $\srs{.88^{\bigcirc}}{.00}$     $\srs{.62^{\bigcirc}}{.01}$     [15mm]     $\wsrs{\cmbold{.87^{\bigcirc}}}{.00}$

LSTM     $\rs{.93}{.00}{.92}{.00}$     $\rs{.26}{.01}{.19}{.01}$     $\rs{17.18}{.00}{17.17}{.00}$      $\rs{.16}{.01}{.10}{.01}$     $\rs{\cmbold{1.00}}{.00}{\cmbold{1.00}}{.00}$     $\rs{19.00}{.00}{19.00}{.00}$     $\srs{.50^{\bigcirc}}{.02}$     $\srs{.14^{\bigcirc}}{.01}$     $\rs{.50^{\bigcirc}}{.01}{.47^{\bigcirc}}{.00}$     $\rs{-.26^{\text{\tiny \Plus}}}{.02}{-.28^{\bigcirc}}{.05}$

Variational LSTM     $\rs{.90}{.02}{.90}{.02}$     $\rs{.08}{.02}{.09}{.02}$     $\rs{16.74}{.03}{16.72}{.03}$     $\rs{.26}{.02}{.17}{.01}$     $\rs{.99}{.01}{.98}{.01}$     $\rs{6.62}{.37}{6.68}{.33}$     $\srs{.60^{\text{\tiny \Plus}}}{.04}$     $\srs{.21^{\text{\tiny \Plus}}}{.02}$     $\rs{.23^{\bigcirc}}{.06}{.23^{\bigcirc}}{.05}$     $\rs{-.04^{\text{\tiny \XSolidBold}}}{.02}{-.02^{\Box}}{.05}$

ST-$\tau$ LSTM     $\rs{.92}{.00}{.92}{.00}$     $\rs{.12}{.00}{.09}{.00}$     $\rs{16.67}{.00}{16.63}{.01}$      $\rs{.24}{.01}{.15}{.01}$     $\rs{1.00}{.00}{.99}{.00}$     $\rs{7.10}{.07}{7.03}{.08}$     $\srs{.54^{\text{\tiny \Plus}}}{.01}$     $\srs{.15^{\text{\tiny \Plus}}}{.01}$     $\rs{.50^{\bigcirc}}{.00}{.48^{\bigcirc}}{.00}$     $\rs{-.05^{\Box}}{.03}{-.01^{\Box}}{.05}$

Bayesian LSTM     $\rs{.93}{.00}{.93}{.00}$     $\rs{.07}{.00}{.07}{.00}$     $\rs{16.81}{.00}{16.79}{.00}$      $\rs{.25}{.01}{.18}{.01}$     $\rs{1.00}{.00}{1.00}{.00}$     $\rs{1.68}{.04}{1.70}{.05}$     $\srs{.65^{\pentagon}}{.17}$     $\srs{.31^{\pentagon}}{.30}$     $\rs{.53^{\bigcirc}}{.01}{\cmbold{.55^{\bigcirc}}}{.01}$     $\rs{-.01^{\Box}}{.07}{-.02^{\text{\tiny \Plus}}}{.04}$

LSTM Ensemble     $\rs{\cmbold{.95}}{.00}{\cmbold{.94}}{.00}$     $\rs{\cmbold{.33}}{.01}{\cmbold{.25}}{.01}$     $\rs{16.37}{.00}{\cmbold{16.35}}{.00}$     $\rs{.18}{.01}{.13}{.01}$     $\rs{.98}{.00}{.97}{.00}$     $\rs{\cmbold{1.62}}{.00}{\cmbold{1.58}}{.01}$     $\srs{.60^{\Box}}{.02}$     $\srs{.18^{\Box}}{.01}$     $\rs{.44^{\Box}}{.00}{.45^{\Box}}{.00}$     $\rs{-.19^{\text{\tiny \Plus}}}{.01}{-.28^{\Box}}{.01}$

SNGP BERT     $\rs{.22}{.35}{.19}{.34}$     $\rs{.03}{.03}{.02}{.02}$     $\rs{17.19}{.01}{17.18}{.01}$      $\rs{\cmbold{.08}}{.01}{\cmbold{.06}}{.01}$     $\rs{1.00}{.00}{1.00}{.00}$     $\rs{18.84}{.32}{18.83}{.34}$     $\srs{.86^{\bigtriangleup}}{.06}$     $\srs{.49^{\bigtriangleup}}{.12}$     $\rs{.17^{\Box}}{.09}{.26^{\Box}}{.14}$     $\rs{\cmbold{.29^{\text{\tiny \XSolidBold}}}}{.03}{\cmbold{.44^{\Box}}}{.11}$

Variational BERT     $\rs{.94}{.00}{.89}{.00}$     $\rs{.29}{.01}{.17}{.00}$      $\rs{\cmbold{16.36}}{.00}{16.43}{.00}$      $\rs{.20}{.00}{.22}{.00}$     $\rs{.99}{.00}{.98}{.00}$     $\rs{2.25}{.01}{3.86}{.08}$     $\srs{.86^{\text{\tiny \Plus}}}{.01}$     $\srs{.46^{\text{\tiny \Plus}}}{.02}$     $\rs{.42^{\bigcirc}}{.00}{.17^{\pentagon}}{.00}$     $\rs{-.35^{\Box}}{.01}{-.41^{\Box}}{.01}$

-8*90Danish      DDU BERT     $\rs{.92}{.00}{.89}{.00}$     $\rs{.25}{.00}{.17}{.00}$     $\rs{16.41}{.00}{16.44}{.00}$      $\rs{.19}{.01}{.21}{.01}$     $\rs{.99}{.00}{.99}{.00}$     $\rs{3.48}{.01}{4.04}{.03}$     $\srs{.86^{\bigcirc}}{.01}$     $\srs{.39^{\bigcirc}}{.02}$     $\rs{\cmbold{.56^{\bigcirc}}}{.00}{.25^{\bigcirc}}{.01}$     $\rs{-.24^{\bigcirc}}{.01}{-.38^{\bigcirc}}{.03}$

LSTM     $\rs{.75}{.00}{.69}{.00}$     $\rs{.57}{.00}{.53}{.00}$     $\rs{6.78}{.00}{6.80}{.00}$     $\rs{.40}{.01}{.38}{.01}$     $\rs{1.00}{.00}{1.00}{.00}$     $\rs{16.00}{.00}{16.00}{.00}$     $\srs{.63^{\bigtriangleup}}{.01}$     $\srs{.69^{\text{\tiny \Plus}}}{.01}$     $\rs{.29^{\bigcirc}}{.00}{.19^{\bigcirc}}{.01}$     $\rs{-.28^{\text{\tiny \Plus}}}{.02}{-.27^{\text{\tiny \Plus}}}{.02}$

Variational LSTM     $\rs{.27}{.00}{.26}{.00}$     $\rs{.03}{.00}{.03}{.00}$     $\rs{6.65}{.01}{6.66}{.01}$      $\rs{.27}{.01}{.28}{.01}$     $\rs{.97}{.00}{.96}{.00}$     $\rs{1.35}{.23}{1.37}{.21}$     $\srs{.51^{\text{\tiny \Plus}}}{.01}$     $\srs{.59^{\text{\tiny \Plus}}}{.01}$     $\rs{.00^{\bigtriangleup}}{.01}{.00^{\pentagon}}{.00}$     $\rs{.01^{\bigtriangleup}}{.03}{.01^{\Box}}{.01}$

ST-$\tau$ LSTM     $\rs{.76}{.00}{.71}{.00}$     $\rs{.58}{.00}{.55}{.00}$     $\rs{6.18}{.00}{6.21}{.00}$      $\rs{.20}{.01}{.22}{.01}$     $\rs{.97}{.00}{.96}{.00}$     $\rs{3.32}{.01}{3.57}{.01}$     $\srs{.62^{\bigtriangleup}}{.01}$     $\srs{.69^{\text{\tiny \Plus}}}{.01}$     $\rs{.31^{\bigcirc}}{.00}{.21^{\bigcirc}}{.01}$     $\rs{-.14^{\text{\tiny \Plus}}}{.02}{-.12^{\Box}}{.04}$

Bayesian LSTM     $\rs{.27}{.00}{.26}{.00}$     $\rs{.03}{.00}{.03}{.00}$     $\rs{6.84}{.00}{6.85}{.00}$      $\rs{\cmbold{.11}}{.00}{\cmbold{.12}}{.00}$     $\rs{1.00}{.00}{1.00}{.00}$     $\rs{16.00}{.00}{16.00}{.00}$     $\srs{.51^{\pentagon}}{.01}$     $\srs{.60^{\text{\tiny \XSolidBold}}}{.00}$     $\rs{.00^{\pentagon}}{.00}{.00^{\pentagon}}{.00}$     $\rs{.01^{\bigcirc}}{.01}{.04^{\text{\tiny \Plus}}}{.00}$

LSTM Ensemble     $\rs{.81}{.00}{.75}{.00}$     $\rs{.62}{.00}{.57}{.00}$     $\rs{6.18}{.00}{6.22}{.00}$      $\rs{.17}{.01}{.21}{.00}$     $\rs{.99}{.00}{.98}{.00}$     $\rs{3.46}{.01}{3.80}{.01}$     $\srs{\cmbold{.67^{\text{\tiny \Plus}}}}{.01}$     $\srs{\cmbold{.74^{\text{\tiny \Plus}}}}{.01}$     $\rs{.29^{\bigcirc}}{.00}{.19^{\bigcirc}}{.01}$     $\rs{-.28^{\text{\tiny \Plus}}}{.01}{-.31^{\text{\tiny \Plus}}}{.01}$

Variational BERT     $\rs{.87}{.00}{.81}{.00}$     $\rs{.74}{.00}{.70}{.00}$     $\rs{6.11}{.00}{6.15}{.00}$      $\rs{.14}{.00}{.18}{.01}$     $\rs{.99}{.00}{.99}{.00}$     $\rs{4.68}{.03}{5.19}{.02}$     $\srs{.64^{\bigtriangleup}}{.01}$     $\srs{.70^{\bigcirc}}{.01}$     $\rs{.14^{\bigcirc}}{.00}{.08^{\text{\tiny \Plus}}}{.00}$     $\rs{-.19^{\text{\tiny \XSolidBold}}}{.00}{-.16^{\text{\tiny \XSolidBold}}}{.01}$

SNGP BERT     $\rs{.18}{.10}{.17}{.10}$     $\rs{.07}{.02}{.08}{.02}$     $\rs{6.82}{.00}{6.83}{.00}$      $\rs{.16}{.02}{.15}{.01}$     $\rs{1.00}{.00}{.99}{.01}$     $\rs{15.00}{.00}{15.00}{.00}$     $\srs{.54^{\bigtriangleup}}{.05}$     $\srs{.63^{\bigtriangleup}}{.04}$     $\rs{.15^{\Box}}{.04}{.15^{\Box}}{.03}$     $\rs{\cmbold{.12^{\Box}}}{.05}{\cmbold{.14^{\Box}}}{.02}$

-8*90Finnish      DDU BERT     $\rs{.87}{.00}{.81}{.00}$     $\rs{.72}{.03}{.68}{.03}$     $\rs{\cmbold{6.01}}{.00}{\cmbold{6.03}}{.00}$      $\rs{.33}{.02}{.38}{.02}$     $\rs{.94}{.00}{.91}{.00}$     $\rs{\cmbold{2.16}}{.06}{\cmbold{2.31}}{.06}$     $\srs{.61^{\bigcirc}}{.02}$     $\srs{.69^{\bigcirc}}{.02}$     $\rs{\cmbold{.39^{\bigcirc}}}{.04}{\cmbold{.26^{\bigcirc}}}{.03}$     $\rs{-.07^{\bigcirc}}{.05}{-.16^{\bigcirc}}{.04}$

Results on the tested datasets. Task performance is measured by macro $F_1$ and accuracy, calibration by different calibration errors, the coverage percentage the average prediction set width. For every result, and value on the ID and OOD test set is shown. For English, OOD scores are not available since the OOD set does not contain gold labels, and Token $\tau$ is missing due to CLINC being a sequence prediction task. Uncertainty quality is evaluated using its ability to discriminate between ID and OOD data, quantified by AUROC and AUPR. Furthermore, Kendall's $\tau$ is measured between the uncertainty and losses on a sequence- and token-level. Displayed are mean and standard deviation over five random seeds, with bolding and underlining indicating almost stochastic dominance with $\varepsilon_\text{min} \le 0.3$ over all other models. For last section, the best value over uncertainty metrics is given, with symbols indicating the type of metric achieving it: ${\bigcirc}$ Max. probability, ${\bigtriangleup}$ Predictive entropy. ${\Box}$ Class variance. ${\pentagon}$ Softmax gap. ${\text{\tiny \Plus}}$ Dempster-Shafer. ${\text{\tiny \XSolidBold}}$ Mutual information.

We present the results from our experiments using the largest training set sizes per dataset in <ref>.[For English, some models were omitted due to convergence issues, which are discussed in <ref>.]

Task Performance Across datasets and models, we can identify several trends: some of the BERT-based models unsurprisingly perform better than LSTM based models, which can be explained with their pre-training procedure. We observe worse performance for some LSTM and BERT-variants, in particular the Variational, Bayesian and ST-$\tau$ LSTM, as well the SNGP BERT. In accordance with the ML literature (see e.g. <cit.>, LSTM ensembles actually perform very strongly and on par or sometimes better than fine-tuned BERTs.

Calibration We also see BERT models to generally achieve lower calibration errors across all metrics measured, which is in line with previous works <cit.>. It is interesting to see that the correct prediction is almost always contained in the $0.95$ confidence set across all models, however these number have to be interpreted in the context of the set's width: It becomes apparent that for instance LSTMs achieve this coverage by spreading probability mass over many classes, while only BERT-based models, LSTM ensembles as well as the Bayesian LSTM (on Danish) and the Variational LSTM (on Finnish) are confidently correct.

Uncertainty Quality LSTM-based model seem to struggle to distinguish in- from out-of-distribution data based on predictive uncertainty. For Danish, only BERTs perform visibly above chance-level. For Finnish, the AUPR results suggest that although some OOD instances are quickly identified as uncertain, many other OOD remain undetected among in-distribution samples. For English, OOD samples are detected more effectively, which can be explained by them consisting of unknown voice commands, representating a potential instance of semantic shift, which has been shown to be easier to detect by classifiers <cit.>. Furthermore, it is striking that uncertainty and loss on a token-level (Token $\tau$) is only positive correlated for some models, using metrics such as the maximum probability score, softmax gap or the Dempster-Shafer metric, which are all entirely based on the categorical output distributions. On a sequence-level (Sequence $\tau$), the correlation is often negative, meaning that higher uncertainty goes hand in hand with a higher loss. Lastly, it should be noted that different uncertainty metrics yield diverse outcomes: There does not seem to be one superior metric across all experimental settings, as seen by the variety of markers shown in <ref>.

§.§ RQ2: Dependence on training data

Scatter plot showing the difference between model performance (measured by macro $F_1$  and the quality of uncertainty estimates on a token-level (measured by Kendall's $\tau$). Shown are different models and uncertainty metrics and several training set sizes of the Dan+ dataset. Arrows indicate changes between the in-distribution and out-of-distribution test set. Best viewed electronically and in color.

After presenting the best results for the biggest training set sizes in <ref>, we now continue to analyze the difference between models and metrics in a more fine-grained way.
In <ref>, we show differences for the token-level correlation between a model's loss and its uncertainty measured by Kendall's $\tau$, with arrows indicating the shift from measurements on the in- to the out-of-distribution test set. Here, we see the same trend of more training data having a larger influence on BERT models. Peculiarly, we also observe pre-trained models' uncertainty to correlate less with their losses on the OOD data, while this property stays relative constant for LSTMs. We can recognize this trend also for the other datasets in <ref> and to a lesser degree on a sequence level <ref> in <ref>, albeit with a negative correlation in general in the latter case.
In <ref> in <ref>, we show the AUROC and AUPR of different model-uncertainty metric combinations for all datasets and training set sizes. In both cases, we can notice that pre-trained models profit more from an increase in available training data than LSTM-based models that are trained from scratch. This improvement is observed both in task performance, as well as in the model's ability to discern ID from OOD data using its uncertainty, but more so for the Danish than English or Finnish. Like in the previous section, we often see that uncertainty metrics of the same model perform quite similarly.
These results outline a seeming paradox: Pre-trained and then fine-tuned models (often) perform better on the task at hand, and provide better uncertainty estimates, but only on in-distribution data. Models trained from scratch that have seen less data overall, however provide more reliable uncertainty estimates on OOD data, but are also worse calibrated (<ref>), with the exception of ensembles. This effect appears to largest on Danish, containing the least data.

Development of token-level Kendall's $\tau$.

Development of sequence-level Kendall's $\tau$.

Development of correlation between uncertainty and loss, shown on the Dan+ OOD test set over the training time using differently-sized training sets. Colored areas indicate the standard deviation over five runs.

Uncertainty quality over training Adding another facet to this issue, we plot the development of uncertainty estimate quality over the training for different models in <ref>. We use LSTMs and DDU BERTs on Dan+ as representative for the observed differences between pre-trained transformers and models trained from scratch, with more examples given in <ref>. On both a token and sequence-level, we can see that the correlation between uncertainty and loss dips for DDU BERT, before increasing again over the course the of the training.[Note that the OOD data used to create these results were not used for training.] Most curiously, the highest correlations are achieved with the models using the least training data. Such behavior is also present for LSTMs on a sequence level. We can also see that while the correlation is higher for DDU BERT on in-distribution data (see again <ref>), on OOD data, LSTMs actually more accurately reflect their knowledge using uncertainty. This again corroborates earlier insights from <ref>: Pre-trained models seem to provide better uncertainty estimates on in-distribution data, but yield worse results on OOD than LSTMs trained from scratch. Furthermore, the less training data is available, the more indicative predictive uncertainty seems to be of the correctness of a model. We see such behavior also to a lesser extent in the other, datasets (see <ref>). Before we offer some potential explanations of this behavior, we try to gain an even more fine-grained understanding by analysing the differences in metrics and models on a token-level.

§.§ RQ3: Qualitative Analysis

Predictive entropy over the sentence "This time in company with Jørn Middelhede, also from Kolding".

Predictive entropy and mutual information over the sentence "However, the phenomenon lasted for such a short time that Pekka did not have a chance to prove it".

Uncertainty estimates on single sequences, for predictive entropy of different models on Danish (<ref>) and predictive entropy and mutual information for multi-prediction models on Finnish (<ref>).

We investigate the development of uncertainty estimates over the course of a single sequence for different datasets, models, and uncertainty metrics. We showcase two examples in <ref>, with more examples in <ref>. By looking at the predictive entropy of models in <ref>, we can observe multiple things: First of all, we can observe some degree of agreement between models and their uncertainty: Processing sub-word tokens, uncertainty seems to increase, and the total uncertainty always appears to reduce considerably on punctuation. Interestingly, the highest uncertainty seems to be produced by the DDU and Variational BERT models as well as the ensembles.
In <ref>, we compare the estimates for predictive entropy and mutual information, the latter of which is supposed to only express model uncertainty. Here, uncertainty is generally low, indicating a large part of the total uncertainty might actually be of an aleatoric nature (which is the gap between triangle and cross markers of the same color, due to <ref>). These insights indicate that while aleatoric uncertainty might be a constant factor for all models, epistemic uncertainty expectedly differs noticeably between them. We use all of these insights to discuss the choice of model next.

§ DISCUSSION

Our experiments in <ref> have uncovered interesting nuances about uncertainty estimation in Natural Language Processing. With respect to RQ1, we observe that fine-tuning BERTs and training LSTM ensembles on different languages produces high task scores with low calibration errors and high-quality uncertainty estimates, but only so on in-distribution data. On OOD data, uncertainty estimates from fine-tuned models do actually become less indicative of potential model loss compared to LSTM-based models. We also find that among the variety of uncertainty metrics proposed, there does not appear to be a superior metric. Differences in Kendall's $\tau$ on a token and sequence level suggest that loss and uncertainties fluctuate over the course of sequence. Answering  RQ2, it seems that paradoxically more training data seems to decrease the quality of uncertainty estimates on OOD data for pre-trained models. We speculate that fine-tuning models increasingly lets them forget relevant features that would produce higher uncertainty.  This might explain why for LSTM-type models, this effect seems to be smaller. Lastly, we conclude about RQ3 that all models' total uncertainty behave somewhat similarly, potentially due to the strong influence of aleatoric uncertainty. From these insights, we conclude that the approaches using pre-trained models overall give the best trade-off between task performance, uncertainty quality and calibrations, however their failure on OOD samples opens up further directions of research. Ensembles can provide an alternative here in data-scarce settings, when the task is sufficiently learnable without the need for pre-training.

§ CONCLUSION

In this work, we explore the current options for uncertainty estimation in NLP on three different languages and tasks, focusing on the impact of available data on the quality of uncertainty scores in a potential low-resource environment. We conclude the following: Fine-tuning pre-trained models produces the best results in terms of task performance, calibration and uncertainty quality, but only on in-distribution data. On out-of-distribution data, LSTM-based models produce more reliable estimates, and could be preferred in cases pre-trained models might not be available, with LSTM-ensembles providing an especially attractive alternative. We discover that more training data seems to decrease quality of uncertainty on OOD, and show that the total uncertainty of models seems to often to be influenced by their aleatoric uncertainty.

Future Work We see our work as groundwork for future research: While uncertainty estimation is a thriving subject in Computer Vision, it remains understudied in NLP. Our experiments highlight that the model behavior on language data is not well-understood and open several lines for further investigation: One such line is the development of new methods for NLP that a) produce more faithful estimates on OOD data while retaining their ID performance and b) require less training data to so, in order to be applicable in low-resource settings. Additionally, our qualitative analyses along with existing works such as <cit.> highlighted the potential to use uncertainty to understand model behavior.

§ LIMITATIONS

Even though the experiments test a large array of models and metrics, the here shown collection is by no means exhaustive, and thus only a selection of popular models or approaches from very different families were considered.

Another glaring shortcoming is the focus on only three European languages: By comparing members of the Uralic, North Germanic and West Germanic families, we only scratch the surface when it comes to the morphological diversity of human language. Further, we only focused on languages with a latin writing systems, as well as specific text domains. This is due to resource constraints and the availability of suitable OOD test sets. We hope that follow-up works will refine our insights on a more representative sample of natural languages.

Lastly, we solely focused on sequence labelling and sequence predictions tasks. <cit.> feature more sequence prediction tasks for English, however we are looking forward to similar studies on natural language generation and structured prediction tasks as well.

§ ETHICS STATEMENT

We do not foresee any immediate negative ethical consequences of our research.

§ ACKNOWLEDGEMENTS

We would like thank Mike Zhang and Joris Baan for their feedback on a draft, with a special thanks to the former for his input on the presentation of results in this work. Furthermore, we express our gratitude to Daniel Varab for verifying the translations of the Danish sentences shown in our works, and to Jenna Kanerva, Otto Tarkka, Antti Virtanen and the rest of the TurkuNLP group for the verification of the Finnish translations.
The authors also acknowledge the IT University of Copenhagen's HPC resources made available for conducting the research reported in this paper.

JF was supported by the Novo Nordisk Foundation (NNF20OC0062606 and NNF20OC0065611), the Independent Research Fund Denmark (9131- 00082B) and the Innovation Fund Denmark (0175-00014B).

§ DATA

§.§ Pre-processing

Tokenization We use the corresponding BERT tokenizer for each language, including for LSTM-based models to ensure compatibility. For English, this corresponds to the original SentencePiece tokenizer used by <cit.>, while we use the tokenizer of the Danish BERT <cit.> and Finnish BERT <cit.> for those lanuages, respectively.

Tags for Sub-word Tokens For named entity recognition and part-of-speech tagging, we follow <cit.>, chapter 11.3.3 to deal with sub-word tokens: For every token that is split into sub-word tokens, we assign the tag only to the first sub-word token, and $-100$ for the rest, which ignores them for evaluation purposes.

§.§ Sub-sampling of Training Sets

Since we sub-sample some of the data splits in <ref>, this bears the dangers of producing unnatural samples of text. For that reason, we use this appendix to describe the sampling strategies in more detail.

Sub-sampling procedure

The procedure for subsampling text is that sequences are first placed into buckets of the same label, then into sub-buckets of the same length. Then, the sampling procedure consists of first drawing a label based on the observed label frequencies, after which the draw of sequence length, proportional to the frequency of this length inside the bucket, determines the final bucket from which a sequence is again drawn uniformly.

Lastly, the process for token classification involves the grouping into sequences by length at the highest level. Inside a bucket, a sequence is not drawn uniformly but with a probability according to the alignment of the sequence's labels with the overall corpus label distribution. This alignment is calculated for each sequence by evaluating the expected log-probability of the sequence's label distribution w.r.t to the label distribution of the corpus (i.e., the cross-entropy). The scores for all same-length sequences in a bucket are then normalized into a $[0, 1]$ interval in order to enable sampling, which is similar to the two-stage procedure used in the sequence classification case.

Comparing the relative frequency of types in the original and sub-sampled training sets. Shown are the top $20$ types in the original training set, compared to sub-sampled training sets of $100$ and $1000$ sequences for Dan+, Finnish UD and Clinc Plus. It is shown that while the type frequencies differ noticeably for the small dataset, already $1000$ sequences suffice to approximate the original frequencies. Numbers, stopwords and the most common punctuation were removed.

Comparing the relative frequency of sequence lengths in the original and sub-sampled training sets. Shown are sequence lengths between $0$ and $25$ in the original test, compared to OOD test sets for Dan+, Finnish UD, Clinc Plus. Not the whole distribution is shown in all cases, with many of the OOD sentences for Dan+ being very long. For Dan+ and Finnish UD, the sentence length distributions are noticeably different. For Clinc Plus, they are very similar.

Comparing the relative frequency of labels in the original training set, compared to sub-sampled training sets. Shown are frequencies for $100$ and $1000$ sequences. For Danish, the most frequent label by far is the neutral label indicating that no named entity is present.

Comparison of the relative class frequencies between original training set compared to the OOD test set. The proportions stay largely the same for Danish, while different more for Finnish.

Validation of sub-sampled training sets We take multiple steps to validate the representativeness of our sub-sampled data splits. First, we plot the distributions of the $50$ most frequent types in the original corpus in <ref>, where we see that distributions converge with increasing sample size. Secondly, we plot sentence length distributions in <ref>, where we also see increasing alignment with sample size.[The distributions for Language Modelling are slightly distorted since we sample whole sets of sentences.] For Sequence and Token Classification tasks, we also plot the class distributions in <ref>. Lastly, we train an interpolated trigram Kneser-Ney language model <cit.> with uniform interpolation weights trained on the original training set using SRILM <cit.> and sub-word tokens produced by the corresponding BERT tokenizer, sub-sample multiple splits and compare their perplexity scores to those of the original corpus in <ref>. While $n$-gram perplexities of sub-sampled training sets do lie over the ones of the original data, they are still upper-bounded by the in-distribution test-set perplexities. Furthermore, this verification was not aimed to give the most precise results, as also the scoring using an $n$-gram model can be rather crude. Thus, with all these results, we conclude that our sub-sampling procedure produces sufficiently representative samples of the original data for the different tasks discussed.

Language     Train ppl.$\downarrow$     3cSub-sampled Train ppl.$\downarrow$     Test ppl.$\downarrow$     OOD Test ppl.$\downarrow$

$n=100$     $n=500$     $n=1000$

English     $31.54$     $43.97\pm2.46$     $44.50\pm0.68$     $44.9\pm0.4$     $53.11$     $ 120.32$

Danish     $112.73$     $252.52\pm 13.25$     $247.09\pm 3.3$     $249.27\pm 3.15$     $418.71$     $524.32$

Finnish     $116.49$     $257.67\pm 10.96$     $257.66 \pm 4.7$     $260.36 \pm 5.36$     $1374.76$     $1284.82$

Results of using an interpolated Kneser-Ney $n$-gram language model on selected datasets, including sub-sampled training splits and the OOD test set. Scores of sub-sampled training sets were obtained over five different attempts.

§.§ Selection of OOD Test Sets

Comparison of sequence length distribution between the original training set and the OOD test set. For English, the distribution of lengths of voice assistant commands is quite similar, while the differences for Dan+ and Finnish UD are more pronounced.

Comparison of the relative frequencies of the top 25 types in the original training set compared to the OOD test set. Even among the most frequent and therefore usually common tokens, the plots show differences between the in-distribution train and out-of-distribution test set. Numbers, stopwords and the most common punctuation were removed.

In this appendix section, we present additional evidence that the OOD test splits shown in <ref> are sufficiently different from the training data — meaning, out-of-distribution — to enable our chosen methodology. To that end, we re-use similar ideas as described in <ref>, but with the opposite goal. In <ref>, we plot the distribution of sequence lengths of the training set compared with the OOD test set, with the same done for the most frequent $25$ types in <ref> and class labels in <ref>. Lastly, we again use a interpolated Kneser-Ney trigram language model to compute the perplexity of the training compared to the OOD test set in <ref>. In all cases, OOD $n$-gram perplexities lie much over the training or sub-sampled data perplexities. Except for Finnish, they are also widely different from the test set perplexities. In that exceptional cases, an explanation could be given by the highly agglutinative nature of Finnish, increasing the sparsity of the language despite the subword tokenization.

§ CALIBRATION METRICS

Perfect calibration is defined as the the confidence of a neural network  corresponding to the percentage of samples with that same predicted probability actually receiving the correct label by the network. Using a predicted label $\hat{y}$ with probability $\hat{p}$, perfect calibration is defined as

\begin{equation*}
P(\hat{y}=y|\hat{p}=p) = p,\quad \forall p \in [0, 1]
\end{equation*}

The expected calibration error <cit.> quantifies the difference between the confidence and the calibration on a test set by collecting predictions into $m$ bins:

\begin{equation}\begin{aligned}\label{eq:ece}
& \text{ECE} =  \Expect[\Big]{\hat{p}}{P\Big(\hat{y} = y \Big| \hat{p} = p\Big) - p\Big|} \\
\approx & \sum^M_{m=1}\frac{|\mathbb{B}_m|}{N}\Big|\text{acc}(\mathbb{B}_m) - \text{conf}(\mathbb{B}_m)\Big| \\
= & \sum^M_{m=1}\frac{1}{N}\Big|\sum_{b \in \mathbb{B}_m}\bind(\hat{y}_b = y_b) - \hat{p}_b\Big| \\
\end{aligned}\end{equation}

where $N$ is the number of data points and $\mathbb{B}_m$ denotes the $m$-th bin.

The problem is that ECE is only defined for binary classification and depends highly on the number of bins chosen. For the former problem, <cit.> present a naive extension to multi-class classification that only considers the most likely prediction. In order to consider all classes, <cit.> introduce the static calibration error (SCE) as an extension to multi-class problems:

\begin{equation*}\label{eq:sce}
\resizebox{\columnwidth}{!}{%
$\displaystyle \text{SCE} = \frac{1}{K}\sum_{k=1}^K \sum_{m=1}^M \frac{N_{mk}}{N}\Big|\text{acc}(\mathbb{B}_m, k) - \text{conf}(\mathbb{B}_m, k)\Big| $
\end{equation*}

Here, $N_{mk}$ denotes the number of instances of class $k$ in bin $m$, and $\text{acc}(\mathbb{B}_m, k)$, $\text{conf}(\mathbb{B}_m, k)$ the accuracies and confidences for class label $k$ in bin $m$, respectively. However, we found this error not be very informative in our case, and therefore omitted corresponding results.

Secondly, <cit.> introduce the adaptive calibration error (ACE), which makes sure that every bin contains the same number of predictions. They define a calibration range by the $\lfloor N/R\rfloor$-th index of sorted and thresholded predictions. Then, the error is defined as

\begin{equation*}
\resizebox{\columnwidth}{!}{%
$\displaystyle \text{ACE} = \frac{1}{KR} \sum_{k=1}^K \sum_{r=1}^R \Big|\text{acc}(\mathbb{B}_m, r) - \text{conf}(\mathbb{B}_m, r)\Big| $
\end{equation*}

§ IMPLEMENTATION & EXPERIMENTS

§.§ Implementation Details

Resources All models were implemented in PyTorch <cit.>. BERT models where implemented with the help of HuggingFace's  library <cit.>. Linear algebra operations where often implemented using the  package <cit.>. The Bayesian LSTM was developed using the  package <cit.> for PyTorch and the SNGP transformer using  <cit.>. Hugginface's  <cit.> were furthermore used for dataset creation and  <cit.> for carbon emissions tracking. Weights & Biases <cit.> was used to track and manage hyperparameter searches and experiments. In general, we follow many of the experimental guidelines and suggestions laid out by <cit.>.

Models For the DUE transformer, we used Principal Component Analysis on the latent representations for Clinc Plus to reduce the memory usage of the Gaussian Discriminant Analysis by reducing dimensionality to $64$. We initially also experimented with the usage of the DUE transformer by <cit.>, however found that it was not trivial to create the inducing points for the Gaussian process output layer in a sequential setting. For the Variational Transformer <cit.>, the authors do not specify exactly how MC Dropout is used. We use the existing dropout layers in the corresponding model, and use a number of forward passes with different dropout masks to make predictions. Since the number of classes is prohibitive for the original formulation of the SNGP transformer, we use the extension proposed by <cit.> in Appendix A.1 and only store one $\hat{\Sigma}^{-1}$ matrix for all classes. Furthermore, we update the matrix continuously during training and not just during the last epoch, in order to allow tracking of the predictive performance over the training time. Lastly, we also evaluate predictions using Monte-Carlo approximations instead of the mean-field approach, since this allows us to compute a wider variety of uncertainty metrics.

Evaluation When computing uncertainty estimates and losses for evaluation purposes, the measurements for a number of tokens were discarded. These include the ignore token with ID $-100$, as well as the IDs corresponding to the , ,  and  token, which might differ between tokenizers of different languages. For computing the ECE, we use $10$ bins, and $10$ value ranges for ACE.

Model Comparison We facilitate the comparison of models using the almost stochastic order test (ASO; , ; , ), as implemented by <cit.>. One distribution is stochastically dominant over the other when its cumulative distribution function is equal or larger than its counterpart at all points. In an experimental setting, that implies that a model is producing higher scores than a baseline. The ASO test measures the deviation from the stochastic order using an approach rooted in optimal transport. We use the test with a confidence level $\alpha = 0.05$ and a decision threshold of $\tau = 0.3$.

§.§ Hyperparameters

We perform hyperparameter search using random sampling <cit.> using hyperband scheduling <cit.>[Trials might be terminated using Hyperband after $10k$ steps.] on the entire training set, even if models are trained on sub-sampled training sets later. This has the advantage of ensuring comparability between runs and eliminating suboptimal hyperparameter choices as a source of worse uncertainty estimation. We do $80$ trials for LSTM-based models, and $30$ for BERT-based models.

Furthermore, the hyperparameters for the LSTM are identical for the LSTM ensemble (10 instances are used per ensemble). Hyperparameters were picked by best final validation loss over search trials.

Chosen Hyperparameters We summarize some common hyperparameters here and show the rest in <ref>. We commonly use a batch size of $32$, and sequence lengths of $35$ for LSTM-based and $128$ for BERT-based models. All LSTM-based models are trained using $2$ layers, with the exception of the vanilla LSTM and the LSTM-ensemble on Clinc Plus with $3$ layers. Their hidden size and embedding sizes are set to $650$. For all models, gradient clipping is set to $10$. For models using multiple predictions to compute uncertainty estimates, $10$ predictions are used at a time.

Name     Tuned for     Search space

Learning rate     [tr]LSTM, LSTM Ensemble,
Bayesian LSTM, ST-$\tau$ LSTM
Variational LSTM     $\mathcal{U}(0.1, 0.5)$

Learning rate     [tr]DDU BERT, SNGP BERT,
Variational BERT     $\log \mathcal{U}(10^{-5},10^{-3})$

Spectral norm upper bound     DDU BERT, SNGP BERT     $\mathcal{U}(0.95, 0.99)$

Kernel amplitude     SNGP BERT     $\log \mathcal{U}(0.01, 0.5)$

$\beta$ weight decay     SNGP BERT     $\log \mathcal{U}(10^{-3}, 0.5)$

Weight decay     [tr]LSTM, LSTM Ensemble,
ST-$\tau$ LSTM, Variational BERT      $\mathcal{U}(0.1, 0.5)$

Layers     LSTM, LSTM Ensemble     $\{2, 3\}$

Dropout     [tr]LSTM, LSTM Ensemble,
ST-$\tau$ LSTM, Variational BERT     $\mathcal{U}(0.1, 0.4)$

Layer Dropout     Variational LSTM     $\mathcal{U}(0.1, 0.4)$

Time Dropout     Variational LSTM     $\mathcal{U}(0.1, 0.4)$

Embedding Dropout     Variational LSTM     $\mathcal{U}(0.1, 0.4)$

Hidden size     LSTM, LSTM Ensemble     $\{350, 500, 650\}$

Prior $\sigma_1$     Bayesian LSTM     $\log \mathcal{U}(-0.8, 0.1)$

Prior $\sigma_2$     Bayesian LSTM     $\log \mathcal{U}(-0.8, 0.1)$

Prior $\pi$     Bayesian LSTM     $\log \mathcal{U}(0.1, 0.9)$

Posterior $\mu$ init     Bayesian LSTM     $\mathcal{U}(-0.6, 0.6)$

Posterior $\rho$ init     Bayesian LSTM     $\mathcal{U}(-8, -2)$

Init weight     LSTM     $\mathcal{U}(0.1, 0.4)$

Number of centroids     ST-$\tau$ LSTM     $\{5, 10, 20, 30, 40\}$

List of searched hyperparameters. LSTM Ensemble hyperparameters are not searched, but simply copied from the found LSTM hyperparameters.

Model     Hyperparameter     English     Danish     Finnish

LSTM     Weight decay     $ 0.001337$     $0.001357$     $0.001204$

Learning rate     $0.4712$     $0.4931$     $0.2205$

Init. weight     $ 0.283$     $0.5848$     $0.5848$

Dropout     $0.3379$     $0.2230$     $0.1392$

Variational LSTM     Weight decay     –     $10^{-7}$     $0.01953$

Learning rate     –     $0.3031$     $0.7817$

Init. weight     –     $0.1097$     $0.5848$

Embedding Dropout     –     $0.1207$     $0.3566$

Layer Dropout     –     $0.1594$     $0.3923$

Time Dropout     –     $0.1281$     $0.1646$

Bayesian LSTM     Weight decay     $0.001341$     $0.003016$     $0.03229$

Learning rate     $0.1704$     $0.1114$     $0.1549$

Dropout     $0.3410$     $0.3868$     $0.331$

Prior $\sigma_1$     $0.9851$     $0.7664$     $0.3246$

Prior $\sigma_2$     $0.5302$     $0.851$     $0.5601$

Prior $\pi$     $1$     $1$     $0.1189$

Posterior $\mu$ init     $-0.005537$     $-0.0425$     $0.4834$

Posterior $\rho$ init     $-7$     $ -6$     $0.1124$

ST-$\tau$ LSTM     Weight decay     –     $0.001189$     $0.0007857$

Learning rate     –     $0.01979$     $0.3601$

Dropout     –     $0.1867$     $0.1737$

Num. centroids     –     $5$     $30$

DDU Bert     Learning Rate     $0.003077$     $0.00006168$     $0.001825$

Spectral norm upper bound     $0.9753$     $0.9211$     $0.941$

Weight decay     $0.003$     $0.1868$     $0.09439$

Variational BERT     Learning Rate     $0.0002981$     $0.00009742$     $0.00003483$

Weight decay     $0.01591$     $0.02731$     $0.09927$

Dropout     $0.2382$     $0.4362$     $0.4364$

SNGP Bert     Learning Rate     –     $0.0002332$     $0.0002919$

Spectral norm upper bound     –     $0.99$     $0.96$

Beta Weight decay     –     $0.001619$     $0.002438$

Beta length scale     –     $2.467$     $2.254$

Kernel amplitude     –     $0.3708$     $0.2466$

List of used model hyperparameters by dataset.

§.§ Optimization

To make sure that all models are trained for the same number of steps regardless of the the size of (sub-sampled) training set, we set the training duration to the number of steps corresponding to a number of epochs using the original training set size, and name it epoch-equivalents in the following. Due to the imbalance of classes in Finnish UD and Dan+, all models were trained using loss-weights that are inverse to the frequency of a label in the dataset.

Optimization of LSTMs We adopt different optimization schemes for transformer and LSTM-based models. For LSTMs, we choose stochastic gradient descent with a decaying learning rate schedule, decaying by $0.8695$ after the equivalent of $14$ epochs for every following epoch-equivalent for $55$ epoch-equivalents in total. This corresponds to the setup in <cit.>, modified from the setup in <cit.>.

Optimization of BERTs We fine-tune BERT models using the shorter duration of $20$ epoch-equivalents, corresponding to the NLP experiments in <cit.>. Adam <cit.> is used for optimization with default parameters $\beta_1 = 0.9$ and $\beta_2 = 0.999$ alongside a triangular learning rate, using the first $10 \%$ of the training duration as warm-up.

§.§ Convergence on Clinc Plus

Here, we briefly address the models missing from the English Clinc Plus experiments. For the ST-$\tau$ and Variational LSTM, we could not identify clear reasons on why models did not converge. Even after extensive hyperparameter searches and manual fine-tuning of hyperparameters (including different learning rate schedules and optimizers), we did not find a combination of options that resulted in convergence. We also observed strange behavior for the Bayesian LSTM, which, after reaching a validation accuracy of $0.5$, would suddenly return to its initial training performance. This could potentially be explained by the model accidentally escaping a low-loss basin due to a learning rate that is still too high, and thus we changed the model to only be trained for $18$ epoch-equivalents and initiate the learning rate decay after seven epoch-equivalents. The puzzling fact is that SNGP BERT did not converge on Clinc Plus, since the authors successfully used the dataset in their own work <cit.>. We put forth the following explanations: First of all, we observed the model to generally possess a high variance, as demonstrated by the standard deviation on the Danish and Finnish data. Secondly, we make at least two changes to their implementation: Instead of using the mean-field approximation to the predictive distribution, we use the Monte Carlo approximation in order to compute metrics such as mutual information. Also, we update the covariance matrix $\hat{\Sigma}$ over the whole training time in order to track the predictive performance for our experiments, and not just during the last epoch.

§.§ Environmental Impact

The carbon efficiency was estimated to be 0.61 kgCO$_2$eq/kWh.
735 hours of computation were performed on a Tesla V100 GPU. This includes hyperparameter search, failed runs, debugging, and discarded runs. As a rough upper bound, we estimate the compute time for a single replication of all experiments to take around 73 hours.[Note that this number could be reduced further by using better hardware acceleration, larger batch sizes, and slightly reducing the training duration for some models. Most importantly, this number also includes compute used for hyperparameter search.] To lessen the environmental impact, all models and model predictions are published in the open-source repository. Total emissions are estimated to be 52.45 kgCO$_2$eq.

We use direct air capture by climeworks to offset the emissions <cit.>. Estimations were conducted using the  package <cit.>, a joint effort from authors of <cit.> and <cit.>.

§ ADDITIONAL RESULTS

This section contains additional experimental results and plots that could not be added to the paper due to spatial constraints. We roughly follow the structure of <ref>.

§.§ Additional Scatter Plots

This section provides some additional scatter plots. For all plots presented here as well as <ref>, some slight jitter sampled from $\mathcal{N}(0, 0.01)$ was added to x and y-coordinates to increase readability of overlapping points.

Clinc Plus In <ref>, we can see that the Variational Bert model actually degrades in performance as the more training data is added, both on a task and uncertainty dimensions, while other models stay relatively constant. The same trend can be detected using the sequence-level Kendall's $\tau$ for Clinc Plus. We suspect that the smallest training size of $10k$ examples does already provide enough data for models to converge to similar solutions even after adding more data, and that the Variational Bert alone might be prone to overfitting in this case.

Dan+ Results for the Danish dataset are shown in <ref>. It is apparent that LSTM–based models stay mostly constant in their predictive performance, with the largest gains observed by the LSTM ensemble. We can also observe the DDU and Variational BERT to increase both in task performance and uncertainty quality with increasing training data. Interestingly, we can see for the SNGP BERT that uncertainty estimates become more indicative of OOD with more training samples, but mostly only using predictive entropy and the maximum probability score. This might indicate that in these cases, the model actually achieves the desired distance-awareness posed by <cit.>. In <ref>, we can see a similar behavior of the SNGP-BERT and its metrics w.r.t. to the sequence-level correlation. Also, we see that the other BERT models and LSTM-Ensemble actually loose in uncertainty quality as more data is added.

Finnish UD In <ref>, we see that the AUROC and AUPR scores of differnet models and metrics stay largely constant across dataset sizes, which could be explained with the larger amount of training data supplied compared to Dan+.  On the token-level correlation between uncertainty and loss in <ref>, we see the DDU BERT profiting most from more data. On a sequence-level, as depicted in <ref>, the correlation appers mostly static across training set sizes, with only small gaps between in-distribution and out-of-distribution data.

Overall, it seems that the range of dataset sizes for Dan+ show the most critical differences between models, while for the dataset sizes used for Finnish UD and Clinc Plus, enough data seems to be supplied for changes to be more miniscule. This result is particularly relevant for low-resource setting, although the dependency on the task can not be disentangled from these results.

Scatter plot for the Clinc Plus dataset.

Scatter plot for the Dan+ dataset.

Scatter plot for the Finnish UD dataset.

Scatter plot showing the difference between model performance (measured by macro $F_1$) and the quality of uncertainty estimates using AUROC. Shown are different models and uncertainty metrics and several training set sizes on the used datasets.

Scatter plot for the Clinc Plus dataset.

Scatter plot for the Dan+ dataset.

Scatter plot for the Finnish UD dataset.

Scatter plot showing the difference between model performance (measured by macro $F_1$) and the quality of uncertainty estimates using AUPR. Shown are different models and uncertainty metrics and several training set sizes on the used datasets.

Scatter plot showing the difference between model performance (measured by macro $F_1$) and the quality of uncertainty estimates on a token-level (measured by Kendall's $\tau$). Results are shown for different models and uncertainty metrics and several training set sizes on the Finnish UD dataset. Arrows indicate changes between the in-distribution and out-of-distribution test set.

Scatter plot for the Clinc Plus dataset.

Scatter plot for the Dan+ dataset.

Scatter plot for the Finnish UD dataset.

Scatter plot showing the difference between model performance (measured by macro $F_1$) and the quality of uncertainty estimates on a sequence-level (measured by Kendall's $\tau$). Results are shown for different models and uncertainty metrics and several training set sizes on the Finnish UD and Clinc Plus dataset. Arrows indicate changes between the in-distribution and out-of-distribution test set.

§.§ Additional Uncertainty over Training Plots

We extend the plots from <ref> for all tested models and datasets in <ref> and <ref>, showing the correlation of predictive entropy on a token- and sequence-level, respectively. On the token-level, we see that token-level correlation is the highest for SNGP-BERT, although the correlation levels for training set sizes seems to be harder to differentiate between models and could also be due to variance between models runs. Secondly, on a sequence-level, we also see either similar correlation across training set sizes, or higher correlation for lower sizes. In all cases, we observe that some models start with a high correlation that decreases over the training time, as the model fits the in-distribution data better. That corroborates a trend described in <ref>, implying that uncertainty estimates become less reliable as the model tries to decrease the loss on the training data.

Development of token-level Kendall's $\tau$ for Dan+.

Development of token-level Kendall's $\tau$ for Finnish UD.

Development of correlation between token-level predictive entropy and loss on the Dan+ and Finnish UD OOD test set over the training time. Data is shown for several model types and using differently-sized training sets. Colored areas indicate the standard deviation over five runs.

Development of sequence-level Kendall's $\tau$ for Clinc Plus.

Development of sequence-level Kendall's $\tau$ for Dan+.

Development of sequence-level Kendall's $\tau$ for Finnish UD.

Development of correlation between sequence-level predictive entropy and loss on all OOD test sets over the training time. Data is shown for several model types and using differently-sized training sets. Colored areas indicate the standard deviation over five runs.

§.§ Qualitative Analysis

Dan+ We show more examples of the predictive entropies on samples from the Dan+ dataset in <ref>, where uncertainty values where jointly normalized by subtracting the mean and dividing by the standard deviation over all models and time steps. We can make the following observations: Firstly, uncertainty seems to decrease on punctuation marks such as commas and full-stops. Secondly, uncertainty appears higher on sub-word tokens and some named entities. Thirdly, DDU BERT and the LSTM ensemble produce the highest uncertainty values, which are also two of the best performing models on the task.

Finnish UD Here, we give more examples of the analysis on the Finnish UD dataset in <ref>. First of all, we see that the Variational LSTM and SNGP BERT seem to produce almost constant uncertainty scores, which can be explained by their suboptimal performance in task, as shown by their results in <ref>. But even for the models that perform better, such as the Variational BERT and the LSTM ensemble, the decomposition of predictive entropy into aleatoric and epistemic uncertainty reveals that model uncertainty generally remains low, and is overshadowed to a larger extent by the aleatoric uncertainty. We can observe that similar to Danish, uncertainty seems to be low on punctuation marks and high on subword tokens. Furthermore, aleatoric uncertainty seems to be higher on nouns and pronouns. This could be due to the sheer number of possible nouns and pronouns that could fill such a gap in a sentence.

Predictive entropy over the sentence "On the contrary, it is one of Russia's few success stories that performs when the rock group Gorky Park begins their Danish tour in the city of the beautiful lakes".

Predictive entropy over the sentence "However, we did not have precise information about what was agreed upon".

Predictive entropy over the sentence "Demonizing hate speech inspires the marginalized, PSYCHOLOGY UNSTABLE (!) Men on the far right to resort to violence against Muslims. This writes Elvir, who....

Further examples for uncertainty estimates on single sequences. Taken from the Dan+ dataset.

Predictive entropy over the sentence "@ToniLotjonen @harrikumpulaine It is true that I’d maybe like to see more of such Latvia–Russia type games in these kinds of major sports events. #floorball".

Predictive entropy over the sentence "I hope that the procedures done on the  person in question stop and he gives his body (and mind) time to recover from that poisoning!".

Predictive entropy over the sentence "Maybe the hat or how it got on my head doesn’t matter".

Further examples for uncertainty estimates on single sequences. Taken from the Finnish UD dataset.