Uncertainty quantification (UQ) is becoming increasingly recognized as a critical component of applications that rely on  machine learning (ML).
The rapid proliferation of large language models (LLMs) has stimulated researchers to seek efficient and effective approaches to UQ in text generation tasks, as in addition to their emerging capabilities, these models have introduced new challenges for building safe applications.
As with other ML models, LLMs are prone to make incorrect predictions, “hallucinate” by fabricating claims, or simply generate low-quality output for a given input. UQ is a key element in dealing with these challenges.
However research to date on UQ methods for LLMs has been fragmented, with disparate evaluation methods.
In this work, we tackle this issue by introducing a novel benchmark that implements a collection of state-of-the-art UQ baselines, and provides an environment for controllable and consistent evaluation of novel techniques by researchers in various text generation tasks.
Our benchmark also supports the assessment of confidence normalization methods in terms of their ability to provide interpretable scores.
Using our benchmark, we conduct a large-scale empirical investigation of UQ and normalization techniques across nine tasks and shed light on the most promising approaches.

§ INTRODUCTION

$\diamondsuit$ Equal contribution

Uncertainty quantification (UQ) is increasingly being recognized as a critical safety component in AI applications. It supports the rejection of uncertain model predictions, and handling of the associated inputs through different means, e.g., by dispatching them to a more advanced model or a human. This behaviour is crucial in areas where the cost of errors is high, such as healthcare.

A plethora of UQ techniques have been developed for classification and regression models <cit.>. There has also been a surge of research devoted to UQ specifically in the context of encoder-only language models (LMs) such as BERT <cit.>. The rapid proliferation of large language models (LLMs) has stimulated researchers to seek efficient and effective approaches to UQ in text generation tasks, in an attempt to make them safer to use in downstream applications.
As with any ML model, LLMs can make incorrect predictions, “hallucinate” by fabricating claims <cit.>, or simply generate low-quality outputs. These problems stem partially  from the peculiarities of the LLM training objective, and are partially inherited from the nature of ML models in general. All ML models are susceptible to errors due to limited amount training data, and the inherent ambiguity of the task they are designed to solve.

Several well-known methods exist for censoring the outputs of large LLMs. These include filtering using stop-word lists, post-processing using classifiers <cit.>, rewriting toxic outputs <cit.>, and extended fine-tuning with reinforcement learning from human feedback (RLHF). However, these techniques alone are insufficient to entirely eliminate incorrect/inappropriate outputs. For instance, fact-checkers target a very narrow sub-problem, and usually require external knowledge sources such as knowledge bases that are generally incomplete. Building an efficient external system to verify LLM output for every possible task is currently infeasible.

Uncertainty quantification promises a more general approach to the problem based on the internal capabilities of the model itself without access to external knowledge, which also opens up the possibility for computational efficiency.
Several recent studies have focused on developing UQ methods for LLMs in text generation tasks <cit.>.
However, the current landscape of this research is quite fragmented, with many non-comparable and concurrent studies.
Researchers have proposed highly divergent methods for benchmarking UQ techniques, making it challenging to consolidate research findings and draw general conclusions.

In this work, we strive to bridge these disparate research efforts and to resolve some issues found in their evaluation protocols by developing a benchmark for UQ techniques in text generation tasks. The benchmark implements state-of-the-art UQ baselines in a unified way, enabling a large-scale, consistent comparison of methods developed in recent work.
The benchmark includes the tasks of: selective prediction (question-answering [QA]), selective generation (machine translation [MT] and text summarization [TS]), and claim-level fact-checking. For the latter, we developed an automatic fact-checking pipeline for four languages: English, Chinese, Arabic, and Russian. Besides common metrics related to UQ performance, we also introduce a metric related to the calibration of confidence scores. This metric allows us to evaluate confidence normalization methods according to their ability to provide interpretable scores. We propose a strong baseline for normalization and investigate its performance in comparison to simpler approaches. Using the developed benchmark, we conduct a large-scale empirical investigation of UQ and normalization techniques across nine datasets.

This work both lowers the barrier to entry into UQ research for individual researchers and developers, and enables more robust, reliable, and trustworthy LLM deployment for end-users.

Our contributions are as follows:

* We propose a new comprehensive benchmark for UQ evaluation for LLMs across a range of text generation tasks: selective classification/generation, fact-checking/hallucination detection, and calibration/normalization.[All code is published under an MIT license and available at: <https://github.com/IINemo/lm-polygraph>]

* We propose a novel multilingual automatic benchmark for token-level and claim-level UQ methods based on claim-level fact-checking, encompassing English, Mandarin Chinese, Arabic, and Russian.

* We develop methods for producing user-friendly bounded confidence scores, and propose methods for confidence calibration that perform identically to raw uncertainty while being highly interpretable for the end user.

* We perform large-scale empirical evaluation of state-of-the-art UQ techniques using the proposed benchmark.

§ METHODS

§.§ Background

Uncertainty is a fundamental concept in machine learning and statistics, indicating that model predictions have a degree of variability due to the lack of complete information. Estimating predictive uncertainty helps in a variety of tasks, e.g., for selective classification tasks: abstention from model prediction if there is no enough confidence for it. Instances with uncertain predictions could be handled through alternative processes, such as being escalated to a human or to a more advanced system <cit.>.

Despite recent efforts to establish a common definition of predictive uncertainty <cit.>, multiple approaches to its quantification exist based on probabilities, entropies, distances, risks, etc. From a practical perspective, any of these scores could serve as a measure of uncertainty as long as they accurately reflect the relevant properties and help to solve reliability tasks.

While there are principled ways of expressing and reasoning about uncertainty, e.g., in terms of information theory and the Bayesian modelling framework <cit.>, they are often difficult to implement. Therefore, UQ practitioners usually rely on approximations or even heuristics.
For example, one of the popular approaches to UQ is ensembling <cit.>. For classification tasks, it is considered a very strong baseline. However, ensembling introduces large computational overhead due to the need for repetitive inference and storing multiple versions of weights. One of the main research questions that has been addressed in recent work related to classification and UQ in general is how to perform it efficiently, while keeping the performance of the scores reliably high <cit.>.

UQ for LLMs in generation tasks represents a greater challenge than in classification. In generation, a model makes multiple predictions: one for each token. Therefore, the uncertainty scores for each token should be somehow aggregated into a single value. At the same time, in many cases, we would like to have an uncertainty score not for the entire output, but for text fragments such as individual claims.  Another problem is that the raw probability distributions of LLMs reflect multiple types of uncertainty, some of which might be irrelevant for a given generation task. Usually, we should not take into account uncertainty related to the choice of the surface forms of the answer, as long as they convey the same meaning <cit.>. Similarly, uncertainty related to the type of conveyed information might be irrelevant, as long as it is correct and we care only about its veracity <cit.>.
Finally, the predictions of LLMs are not conditionally independent <cit.>, and thus incorrect claims generated by an LLM at the start of an output may cause flown-on hallucinations through the subsequent generation process.

§.§ Uncertainty Quantification for LLMs

Uncertainty Quantification Method     Type     Category    Compute     Memory     Need
Data?     Level

Maximum sequence probability (MSP)     10*White-box     10*Information-
based     Low     Low     No     Seq./claim

Perplexity <cit.>              Low     Low     No     Seq./claim

Mean/max token entropy (TE) <cit.>              Low     Low     No     Seq./claim

Monte Carlo sequence entropy (MC-SE) <cit.>              High     Low     No     Seq.

Monte Carlo norm. seq. entropy (MC-NSE) <cit.>              High     Low     No     Seq.

Pointwise mutual information (PMI) <cit.>              Medium     Low     No     Seq./claim

Conditional PMI <cit.>              Medium     Medium     No     Seq.

Rényi divergence <cit.>              Low     Low     No     Seq.

Fisher-Rao distance <cit.>               Low     Low     No     Seq.

TokenSAR <cit.>              Medium     Low     No     Seq.

Semantic entropy <cit.>     4*White-box     4*Meaning
diversity     High     Low     No     Seq.

CCP <cit.>             Low     Low     No     Seq./claim

SentenceSAR <cit.>             High     Low     No     Seq.

SAR <cit.>             High     Low     No     Seq.

Sentence-level ensemble-based measures <cit.>     2*White-box     2*Ensembling     High     High     Yes     Seq.

Token-level ensemble-based measures <cit.>             High     High     Yes     Seq.

Mahalanobis distance (MD) <cit.>     4*White-box     4*Density-
based     Low     Low     Yes     Seq.

Robust density estimation (RDE) <cit.>              Low     Low     Yes     Seq.

Relative Mahalanobis distance (RMD) <cit.>              Low     Low     Yes     Seq.

Hybrid Uncertainty Quantification (HUQ) <cit.>              Low     Low     Yes     Seq.

p(True) <cit.>     White-box     Reflexive     Medium     Low     No     Seq./claim

Number of semantic sets (NumSets) <cit.>     5*Black-box     5*Meaning
diversity     High     Low     No     Seq.

Sum of eigenvalues of the graph Laplacian (EigV) <cit.>              High     Low     No     Seq.

Degree matrix (Deg) <cit.>              High     Low     No     Seq.

Eccentricity (Ecc) <cit.>              High     Low     No     Seq.

Lexical similarity (LexSim) <cit.>             High     Low     No     Seq.

UQ methods implemented in the benchmark.

Here, we summarize the UQ baselines implemented in our benchmark, as listed in <Ref>. To implement the methods, we use the LM-Polygraph framework <cit.>, extended to include a number of recently-proposed approaches.

There are two major types of techniques: white-box and black-box. White-box methods require access to logits, internal layer outputs, or the LLM itself. Black-box only need access to the generated texts, and can easily be integrated with third-party online services such as OpenAI's API. We note that methods differ in terms of their computational requirements: some techniques pose high computational or memory overheads, e.g., due to repeated inference, making them less suitable for practical usage. The application of some methods can also be hindered by the need to access the model's training data. Finally, different methods might be restricted only to UQ of the whole text (sequence level), while others might be also applicable to text fragments such as atomic claims (claim level).

§.§.§ White-box Methods

Let us consider the input sequence $\xv$ and the output sequence $\yv \in \YC$ of length $L$, where $\YC$ is a set of all possible output sequences. Then the probability of an output sequence given an input sequence for autoregressive language models is given by

\begin{equation}
P(\yv \mid \xv, \thetav) = \prod\nolimits_{l = 1}^L P(y_l \mid \yv_{<l}, \xv, \thetav),
\end{equation}

where the distribution of each $y_l$ is conditioned on all previous tokens in a sequence $\yv_{<l} = \{y_1, \dots, y_{l - 1}\}$, and $\thetav$ denotes model parameters.

We start the discussion from information-based methods that are based on token $P(y_l \mid \yv_{<l}, \xv, \thetav)$ and sequence $P(\yv \mid \xv, \thetav)$ probabilities obtained from a single model prediction. A notable example is entropy, which can be calculated at the token or the sequence level. The crucial difference between these cases is that the entropy at the token level can be computed exactly, while sequence-level entropy can only be calculated through approximate Monte Carlo estimation, based on sampling several sequences $\yv^{(k)}, k = 1, \dots, K$. The advantage of information-based methods is that they are simple to implement and relatively cheap to compute, while often providing competitive performance compared to more sophisticated methods.

Recently, enhanced information-theoretic approaches have been proposed that consider the semantics of natural language. For example, one may cluster words by meaning and adjust the probabilities by aggregating them over the clusters. Such an approach is the basis of the semantic entropy <cit.> uncertainty measure. Further extensions of this approach include other measures that consider meaning diversity to adjust information-theoretic uncertainty measures <cit.>.

The third category of white-box techniques is ensemble-based methods, which leverage the diversity of output predictions made by multiple different versions of models. Let us assume that $M$ models are available with parameters $\thetav_i, i = 1, \dots, M$. These parameters can be obtained via independent training of models. Then, one can use token $P(y_l \mid \yv_{<l}, \xv, \thetav_i)$ and sequence $P(\yv \mid \xv, \thetav_i)$ probabilities to compute various measures that assess the discrepancy between model predictions such as mutual information.

Density-based methods leverage latent representations of instances (embeddings) and, usually, approximate the distribution of embeddings for the training data using one or multiple Gaussian distributions. They can provide a probability or an unnormalized score that determines how likely it is that the instances belong to the training data distribution. As such, they are good at spotting out-of-distribution (OOD) instances <cit.>. Several variations of these methods have been proposed recently <cit.>.

The primary advantage of these methods is that they are computationally efficient: they do not need additional model inference, and the memory overhead for storing additional parameters is minimal. The drawback is that they require access to the model's training data to fit auxiliary models. For example, the Mahalanobis distance approximates the distribution of the embeddings as a mixture of Gaussians, which requires constructing data centroids and covariance matrices.

Finally, one can also combine information-based and density-based methods as suggested by <cit.> and <cit.>. More specifically, we implement the hybrid uncertainty quantification (HUQ) method <cit.>, which performs a ranking-based aggregation and leverages the strengths of both information-based methods that detect ambiguous instances and density-based methods that detect OOD instances.

Directly asking the model to validate its answer is another option for UQ <cit.>. In this method, one asks models first to propose answers, and subsequently to evaluate the probability $P(\text{True})$ that their answers are correct. <cit.> showed that this method achieves reasonable performance on a variety of tasks, including question answering. We note that this method requires two-pass model inference: first to generate an answer, and second to process its own output. Even though the second inference is usually faster than the first one, it still takes considerable computation time.

§.§.§ Black-Box Methods

In contemporary models, there are instances where the model's architecture and hidden states are unavailable or there is no access to logits during response generation. Nevertheless, a whole class of black-box methods only needs to access the model's response. In this paper, we consider several black-box  approaches, including Lexical Similarity, Number of Semantic Sets, Sum of Eigenvalues of the Graph Laplacian, Degree Matrix, and Eccentricity, all of which have been shown to perform well in other studies <cit.>. We use the same methodological approach as <cit.>:

* Obtain $K$ responses $\yv_{1}, \dots, \yv_{K}$ for a particular input $\xv$.

* Compute $K \times K$ similarity matrix $S$ between responses, where $S_{ij} = s(\yv_{i}, \yv_{j})$ for some similarity score $s$ (Natural Language Inference score or Jaccard score).

* Based on the similarity matrix $S$, we compute the final uncertainty score.

§.§.§ Claim-Level Extensions

While the methods discussed above provide uncertainty scores for entire generated sequences, it is often desirable to quantify the uncertainty for individual atomic claims within texts.
Consider that claims are already extracted from the text sentences and there is a mapping between them and tokens in the original text, so we know probability distributions for each token in each claim. Many sequence-level methods can be modified to operate on the claim level <cit.>, but not all of them. For example, repetitive sampling methods cannot work on the claim level because sampled texts may differ too much and miss some claims.

Let $C$ denote a set of token indices corresponding to a claim.
To adapt MSP to the claim level, we can compute the joint probability of  tokens in the claim only instead of the whole sequence, $\prod_{l \in C} P(y_l \mid \yv_{<l}, \thetav)$. Similarly, instead of considering the entropy of the entire sequence, we can focus on the maximum entropy among the tokens within a claim. Sequence-level Perplexity and PMI can also be naturally extended to the claim level in the similar way. P(true) could be adapted by asking LLM about correctness of each claim individually.

The Claim-Conditioned Probability (CCP) method, proposed by <cit.> is designed specifically for the claim level (but also could be applied to sequence level). It assesses the semantic similarity between the original claim and perturbed versions where individual tokens are replaced with their alternative generations. This approach provides a more nuanced understanding of uncertainty by considering the potential impact of different word choices on the overall meaning of the claim. Details about claim-level UQ methods could be found in Appendix <ref>.

§.§ Uncertainty Normalization

Raw uncertainty scores are good for ranking outputs by their potential quality, but can be confusing to an end user. To mitigate this problem, we consider several approaches to obtaining confidence scores bounded in the range $[0,1]$.

All of the considered approaches require fitting on a held-out calibration set $\mathcal{D}_{calib} = \{\xv_i, \yv_i\}_{i=1}^N$. For each sample $(\xv_i, \yv_i)$, we assume that some uncertainty score is computed $u_i = U(\xv_i)$ as well as the generation quality $f(\xv_i)$: $q_i = Q(f(\xv_i), \yv_i)$.

Among simple normalization approaches we consider linear scaling and quantile scaling, as they provide simple rules to normalize uncertainty scores in $[0, 1]$ based solely on uncertainty values $u_i$; see Appendix <ref> for more detail.

However, to provide meaningful information about the model's confidence to the end user, the confidence score should not only be bounded to the fixed interval of values, but it should also be directly tied to the expected quality of the model's output. We call the extent of this relation confidence calibration, as opposed to simple confidence normalization. To facilitate this, we propose two methods of such calibration, which we call Performance-Calibrated Confidence (PCC).

The first approach, Binned PCC, splits the calibration set into non-intersecting bins based on the values of uncertainty $u_i$ and considers the confidence to be an estimate of the mean quality of the outputs in the bin, as measured by some quality measure of choice. The downside of this approach to confidence estimation is that the ordering of the points based on raw uncertainty and bounded confidence scores can be different, and thus the quality of uncertainty estimation using such confidence can vary substantially and unpredictably.

To address this problem, we propose a second approach: Isotonic PCC. It fits Centered Isotonic Regression (CIR) <cit.> on pairs of uncertainty and quality values from the calibration set. CIR produces a monotonic piecewise linear function, which allows to make use of the relationship between uncertainty and quality, while keeping the order of the inputs intact.

Both approaches in the PCC family produce calibrated confidence scores as a local estimate of some quality measure in the neighborhood of the raw uncertainty estimate. This directly ties the confidence, that is presented to the user with the estimated quality of the output, thus making it easier to explain and to justify the confidence to the end user.
We provide a more detailed discussion on the specifics of the normalization methods in Appendix <ref>.

§ APPROACHES TO EVALUATING UNCERTAINTY QUANTIFICATION METHODS

In the past few years, researchers have introduced many diverse approaches to benchmarking UQ methods for generative LLMs. In general, a valid UQ technique should produce scores that are well correlated with some measure of output quality. Thus, the most straightforward way of comparing different UQ methods is to measure the rank correlation between some generation quality metric (e.g., ROUGE-L) and uncertainty scores <cit.>. However, this way of evaluating UQ is not very informative of the performance gain that a particular UQ method achieves.

One of the most popular approaches is based on designating outputs as correct and incorrect by selecting a threshold on a quality metric, and measuring how well uncertainty scores can predict the output as being one or the other <cit.>. This reduces the uncertainty score to being a predictor in a binary classification task, and thus, one can use ROC-AUC or PR-AUC as a measure of how well a UQ method performs. The problem with this approach, which makes results across different work incomparable, lies in the fact that it requires selecting the quality threshold, and the choice of such a threshold has been quite arbitrary in the literature.

A more comprehensive approach is usually called rejection verification <cit.>. It does not require thresholding the quality metric to formulate a binary classification task. Instead, it computes the average quality of the outputs for which uncertainty is relatively low. By continuously lowering the uncertainty threshold above which the data points are discarded, one obtains a set of average quality values of outputs with progressively lower maximum uncertainty. These pairs of uncertainty thresholds and associated average output quality give a prediction–rejection curve, and the area under this curve demonstrates the overall quality of a UQ method <cit.>.

One of the common problems with the majority of evaluation approaches used in previous work on UQ for LLMs is the usage of $n$-gram output quality metrics such as ROUGE-L, which often do not reflect the actual quality of the generated output. For example, this discrepancy occurs when the gold standard answer and the LLM output differ only by a negation. In this case, $n$-gram metrics would rank such a model answer higher than it deserves, failing to capture the substantial semantic difference caused by the negation. In addition to $n$-gram metrics, we suggest to use the recently proposed AlignScore <cit.>, where the gold-standard answer and the LLM output are compared by another LLM. The authors show that this metric has better correlation with human annotators due to its ability to capture deeper semantic similarities and differences between texts.

Another issue with evaluation protocols in recent work is their tendency to overlook simple yet effective baselines. For instance, the maximum probability often proves difficult to surpass in tasks with short outputs. Many studies neglect this baseline, favoring comparisons with entropy-based metrics, which frequently perform worse. By not adequately considering these straightforward baselines, the evaluation protocols may give an incomplete picture of UQ performance.

There are few studies on UQ for text fragments such as sentences <cit.>. In these studies, the authors manually annotated texts generated by one model, e.g., GPT-3. Then they evaluated the UQ performance for a proxy model by inferring the probability distributions of the tokens for the fixed output and calculating standard classification measures such as PR-AUC with uncertainty scores as predictors. However, such an approach introduces a big discrepancy between the generated text and what a proxy LLM actually wants to generate, which results in biased UQ performance. In contrast, we suggest to perform unrestricted generation from LLMs, thus mitigating this discrepancy. To mitigate the problem of variability of generations, we suggest an automatic evaluation pipeline for UQ in the fact-checking task.

The mean rank $\downarrow$ aggregated over all selective classification tasks for each LLM separately (the lower the better). Column Mean Rank corresponds to the mean of the ranks across all LLMs.

§ EVALUATION BENCHMARK

Our benchmark features three sections: (1) evaluation of the UQ performance in selective classification/generation; (2) evaluation of UQ performance in fact-checking; and (3) evaluation of confidence calibration.

§.§ Selective Classification / Generation

In this section of the benchmark, we evaluate how well the uncertainty scores of the considered methods detect low-quality LLM generations.

For selective classification, we use four QA datasets: CoQA <cit.> with free-form answers about conversations; TriviaQA <cit.> with complex and compositional questions without context; MMLU <cit.> – a multitask dataset structured as multiple-choice QA; and GSM8k <cit.>, consisting of grade school math word problems. The first two datasets are used in many works on UQ, while the last two are popular datasets for evaluating the English-language generation quality of LLMs. For selective generation, we use two MT datasets: WMT-14 French to English <cit.>, WMT-19 German to English <cit.>, and one text summarization dataset – XSum <cit.>.

For each dataset, we limit the number of instances from the evaluation set to 2,000. We also reserve 1,000 instances from the training set for UQ techniques that require “pre-training”, such as density-based methods. More detailed statistics about the datasets are given in Appendix <ref>.

For prompting LLMs, we mainly follow the prompt format from the lm-evaluation-harness framework <cit.>. For TriviaQA, MMLU, and GSM8k, we use a 5-shot prompt. For XSUM, WMT-14 Fr-En, and WMT-19 De-En, we use a zero-shot prompt. For CoQA, we use a few-shot prompt with all preceding questions for the conversation before the target question.

The mean rank $\downarrow$ aggregated over all selective generation tasks for each LLM separately (the lower the better). Column Mean Rank corresponds to the mean of the ranks across all LLMs.

Following previous work on UQ in text generation <cit.>, we compare the methods using the Prediction Rejection Ratio (PRR) metric <cit.>.
Consider a test dataset $\DC = \{(\xv_i, \yv_i)\}$. Let $f(\xv_i)$ be the output generated by a LLM and $U(\xv_i)$ be the uncertainty score of a prediction.
The prediction rejection (PR) curve indicates how the average quality $Q(f(\xv_i), \yv_i)$ of the instances having uncertainty $U(\xv_i) < a$ depends on the value of the rejection parameter $a$.
PRR computes the ratio of the area  between the PR curves for a considered uncertainty score and a random score and the area between the oracle (best possible uncertainty that sorts instances according to their text quality metric) and a random score:

\begin{equation}
\label{eq:prr}
PRR = \frac{AUCPR_{\text{unc}}-AUCPR_{\text{rnd}}}{AUCPR_{\text{oracle}}-AURCPR_{\text{rnd}}}.
\end{equation}

Higher PRR indicates a better uncertainty score.

The choice of the generation quality measure $Q(f(\xv_i), \yv_i)$ depends on the dataset. Compared to previous work that uses $n$-gram based measures, in our benchmark, we mostly adhere to LLM-based measures such as AlignScore <cit.> and COMET <cit.> and keep $n$-gram measures for comparability with previous work.
For MT datasets, we use COMET and AlignScore. For GSM8K and MMLU, we use accuracy. For CoQA and TriviaQA, we use accuracy and AlignScore. For XSum, we use ROUGE-L and AlignScore.

§.§ Claim-Level Fact-Checking

In this part of the benchmark, we evaluate claim-level UQ techniques and their ability to spot hallucinations on the task of generating biographies, as proposed by <cit.>. The difference over previous work such as <cit.> is that in our benchmark, we perform unrestricted generation using the LLM, which is much closer to the practical use-case. However, this raises the problem that each generation is unique and needs to be reannotated, posing a significant challenge for human annotation.

Evaluation pipeline.
To enable evaluation of UQ methods on unrestricted LLM outputs, we implement an automatic benchmarking pipeline. The pipeline supports English, Mandarin Chinese, Arabic, and Russian. It starts with generating LLM responses to biography prompts such as Give me a biography of <person name>. The set of people was generated by asking GPT-4 to list the most famous people since 1900. The maximum generation length is set to 256 tokens.

The generated texts are then decomposed into atomic claims. For each claim, we also map all its words back to the generated text to access the corresponding token logits. These two steps are done using GPT-4 with a language-specific prompt. Usually, about 5% of claims do not successfully match because GPT-4 abstains from responding or outputs words not present in the original text. The further evaluation considers only successfully matched claims.

The annotation of the extracted claims is also done automatically using GPT-4. We use a language-specific prompt that facilitates chain-of-thought reasoning to ask whether the presented claims are supported, unsupported, or unknown. Usually, the percentage of claims classified as unknown is negligible; these claims are discarded from the evaluation. The obtained LLM annotation is used to calculate the performance metrics.

To assess the quality of automatic annotation, we manually annotated a random subset of claims for each language.  Table <ref> in the Appendix summarizes the binary performance metrics of GPT-4 against human labels. The results show that GPT-4 obtains good performance, confirming it can serve as a ground truth for UQ techniques.

The performance of the claim-level uncertainty is evaluated using ROC-AUC and PR-AUC with unsupported claims as a positive class.

§.§ Effect of Uncertainty Normalization

This section of the benchmark is designed to analyze how uncertainty normalization procedures (as presented in Section <ref>) impact on the performance of the scores and their correlation with quality of generated text.

The reserved training partitions of the datasets from the selective classification and generation section are used as calibration sets for normalization methods.
Evaluation of the normalization methods can be performed both on concatenated test partitions and on each of them individually.

The benchmark offers two metrics: PRR before and after normalization to ensure that performance does not degrade; and the metric similar to ECE that measures “calibration”, i.e. the ability of normalized uncertainty scores represent the expected quality of the output in a bounded range – Mean Squared Error (MSE) between a normalized quality metric and a confidence score.
Lower MSE indicates better confidence “calibration”.

§ EXPERIMENTS

Using our benchmark, we performed evaluation of the implemented UQ and normalization baselines.

We conducted the experiments on selective classification/generation using Mistral-7B-v0.2 <cit.> and StableLM-12B-v2 <cit.>.  The experiments on the fact-checking task were conducted using the instruct versions of Mistral 7B v0.1 <cit.> and Jais 13B <cit.>. The generation hyperparameters are provided in Appendix <ref>. Text generation quality of the models is presented in Tables <ref> and <ref> in Appendix <ref>.

§.§ Selective Classification/Generation

Selective classification. Tables <ref> and <ref> in Appendix <ref> present results on the selective classification task for the StableLM 12B v2 and Mistral 7B v0.2 models, respectively. For CoQA and MMLU, the basic MSP marginally outperforms other methods in terms of accuracy-based PRR for StableLM and in terms of both metrics for Mistral. HUQ-MD and Perplexity achieve the best results on TriviaQA, slightly outperforming MSP for Mistral, and are on par with MSP for StableLM. However, for GSM8k, where the models generate a long chain of thoughts, MSP significantly falls behind the state-of-the-art methods based on meaning diversity. SAR, Semantic Entropy, HUQ-MD, and Lexical Similarity significantly outperform other techniques.

Figure <ref> presents mean ranks for each model separately and the mean rank of ranks aggregated for both models for selective classification. For the selective classification with the Mistral model, SAR, SentenceSAR, Lexical Similarity, and MSP appear to be the best methods in terms of the mean rank. For StableLM, these methods also demonstrate good results, but they appear to be almost on par with many other black-box techniques.
Considering the average rank across both models, MSP and meaning diversity based methods such as SAR, SentenceSAR, as well as Lexical Similarity, achieve the best results. We also should note that despite MSP is often not the best method for considered datasets individually, it demonstrates robust performance across them all and remains to be a very strong baseline for selective classification.

UQ Method     7cOutput length in symbols
1–2     3–6     7–24     25–85     86–138     139–210     211–1k

MSP     [HTML]80A28D 0.69     [HTML]80A58E 0.61     [HTML]80AF93 0.62     [HTML]83B998 0.57     [HTML]8BBF9D 0.54     [HTML]A8D9B2 0.38     [HTML]B6E0B9 0.26

Perplexity     [HTML]80A28D 0.69     [HTML]80AB91 0.58     [HTML]80A990 0.66     [HTML]85BB99 0.56     [HTML]99CEA8 0.41     [HTML]BFE4BF 0.27     [HTML]B4DFB8 0.26

Mean Token Entropy     [HTML]80A78F 0.66     [HTML]A7D8B1 0.27     [HTML]81B796 0.55     [HTML]91C5A2 0.47     [HTML]9BD0AA 0.39     [HTML]C9E9C8 0.22     [HTML]C9E9C8 0.17

MC-SE     [HTML]80B495 0.60     [HTML]97CCA7 0.37     [HTML]86BC9A 0.50     [HTML]87BC9B 0.54     [HTML]80B093 0.68     [HTML]87BD9B 0.60     [HTML]A3D6AF 0.33

[HTML]80B495 0.60     [HTML]8DC29F 0.43     [HTML]80AC92 0.64     [HTML]87BD9B 0.54     [HTML]86BC9A 0.57     [HTML]93C7A4 0.52     [HTML]9DD2AC 0.36

PMI     [HTML]9DD2AC 0.41     [HTML]B3DFB8 0.22     [HTML]ACDBB4 0.17     [HTML]FBFEFA $-$0.23     [HTML]F5FBF3 $-$0.27     [HTML]F9FDF8 $-$0.15     [HTML]E7F5E3 0.02

Conditional PMI     [HTML]FBFEFA 0.22     [HTML]FBFEFA $-$0.14     [HTML]FBFEFA $-$0.25     [HTML]FBFEFA $-$0.12     [HTML]FBFEFA $-$0.12     [HTML]FBFEFA $-$0.11     [HTML]FBFEFA $-$0.12

RenyiNeg     [HTML]F1FAEE 0.00     [HTML]D1ECCE 0.08     [HTML]FBFEFA $-$0.56     [HTML]C5E7C4 0.18     [HTML]B8E1BA 0.21     [HTML]F3FAF0 $-$0.06     [HTML]FBFEFA $-$0.15

FisherRao     [HTML]ECF8E9 0.04     [HTML]CEEBCC 0.09     [HTML]E8F6E5 $-$0.29     [HTML]EBF7E8 $-$0.06     [HTML]CCEAC9 0.09     [HTML]CCEAC9 0.21     [HTML]BCE3BD 0.23

TokenSAR     [HTML]80A28D 0.69     [HTML]80AA90 0.59     [HTML]80AA90 0.66     [HTML]85BB99 0.56     [HTML]9ACFA9 0.40     [HTML]C2E5C1 0.26     [HTML]B4DFB8 0.26

CCP     [HTML]80A28D 0.69     [HTML]80A28D 0.63     [HTML]83B998 0.53     [HTML]87BD9B 0.54     [HTML]88BD9B 0.56     [HTML]96CAA6 0.50     [HTML]9DD2AC 0.36

Semantic Entropy     [HTML]80B093 0.62     [HTML]8EC2A0 0.43     [HTML]80B093 0.61     [HTML]80AE92 0.63     [HTML]80A990 0.72     [HTML]80B294 0.68     [HTML]90C4A1 0.44

SentenceSAR     [HTML]80A890 0.66     [HTML]80B093 0.56     [HTML]80A890 0.67     [HTML]89BF9D 0.52     [HTML]ACDBB4 0.28     [HTML]C8E8C6 0.22     [HTML]BDE3BE 0.23

SAR     [HTML]80AB91 0.65     [HTML]80B394 0.54     [HTML]80A38E 0.71     [HTML]80AD92 0.64     [HTML]80AE92 0.68     [HTML]80B494 0.67     [HTML]81B797 0.52

MD $-$ Decoder     [HTML]E6F5E3 0.07     [HTML]EDF8EA $-$0.10     [HTML]F8FCF7 $-$0.50     [HTML]F4FBF2 $-$0.14     [HTML]F3FAF0 $-$0.23     [HTML]FBFEFA $-$0.18     [HTML]FAFDF9 $-$0.14

RMD $-$ Decoder     [HTML]FBFEFA $-$0.11     [HTML]D7EFD4 0.04     [HTML]BFE4BF 0.05     [HTML]CEEBCB 0.13     [HTML]C7E8C6 0.12     [HTML]EFF9EC $-$0.03     [HTML]FAFDF8 $-$0.14

RDE $-$ Decoder     [HTML]DFF2DC 0.11     [HTML]C3E6C2 0.15     [HTML]A3D7B0 0.22     [HTML]F2FAEF $-$0.11     [HTML]F1FAEE $-$0.22     [HTML]E5F5E2 0.05     [HTML]DBF1D8 0.09

HUQ$-$MD $-$ Decoder     [HTML]80AC91 0.64     [HTML]82B897 0.51     [HTML]99CEA8 0.32     [HTML]E1F3DE 0.02     [HTML]DDF1DA $-$0.03     [HTML]E9F7E6 0.01     [HTML]D8EFD4 0.10

HUQ$-$RMD $-$ Decoder     [HTML]83B998 0.57     [HTML]88BD9B 0.47     [HTML]9BD0AA 0.30     [HTML]E3F4E0 0.00     [HTML]DDF1DA $-$0.04     [HTML]EAF7E7 0.01     [HTML]D2EDCF 0.13

P(True)     [HTML]FBFEFA $-$0.11     [HTML]FBFEFA $-$0.25     [HTML]FBFDFA $-$0.55     [HTML]FBFDFA $-$0.23     [HTML]FBFEFA $-$0.37     [HTML]FAFDF9 $-$0.16     [HTML]EEF8EB $-$0.03

NumSemSets     [HTML]80B696 0.59     [HTML]87BC9B 0.47     [HTML]80AC92 0.64     [HTML]97CBA7 0.43     [HTML]92C6A2 0.48     [HTML]9ACEA9 0.47     [HTML]CAE9C8 0.17

EigValLaplacian NLI  entail.     [HTML]80B394 0.61     [HTML]80B595 0.53     [HTML]80A48E 0.70     [HTML]80A28D 0.71     [HTML]80A28D 0.77     [HTML]80A28D 0.77     [HTML]80A48E 0.62

EigValLaplacian NLI  contra.     [HTML]80B494 0.60     [HTML]81B797 0.51     [HTML]80AB91 0.65     [HTML]81B796 0.58     [HTML]86BC9A 0.58     [HTML]97CBA7 0.49     [HTML]BDE3BE 0.22

EigValLaplacian Jaccard     [HTML]83B998 0.57     [HTML]87BD9B 0.47     [HTML]80A890 0.67     [HTML]80B394 0.61     [HTML]80AE92 0.69     [HTML]80B696 0.66     [HTML]85BA99 0.51

DegMat NLI entail.     [HTML]80B193 0.62     [HTML]80B093 0.55     [HTML]80A28D 0.73     [HTML]80A28D 0.71     [HTML]80A28D 0.77     [HTML]80A28D 0.78     [HTML]80A28D 0.63

DegMat NLI  contra.     [HTML]80B294 0.61     [HTML]80B595 0.53     [HTML]80AE92 0.63     [HTML]81B797 0.58     [HTML]86BB9A 0.58     [HTML]99CEA8 0.47     [HTML]C2E6C2 0.20

DegMat Jaccard      [HTML]80AE92 0.63     [HTML]80B796 0.52     [HTML]80A78F 0.68     [HTML]80B294 0.62     [HTML]80AC91 0.70     [HTML]80B294 0.68     [HTML]80B495 0.55

Eccentricity NLI  entail.     [HTML]82B898 0.58     [HTML]86BC9A 0.48     [HTML]80A68F 0.69     [HTML]80A28D 0.70     [HTML]80A28D 0.77     [HTML]80A28D 0.77     [HTML]80A28D 0.64

Eccentricity NLI  contra.     [HTML]83B998 0.57     [HTML]81B797 0.51     [HTML]80AA90 0.66     [HTML]80B796 0.59     [HTML]87BC9B 0.57     [HTML]9ACEA9 0.47     [HTML]C0E5C0 0.21

Eccentricity Jaccard      [HTML]82B898 0.58     [HTML]8BC09E 0.45     [HTML]80AE92 0.62     [HTML]80B796 0.59     [HTML]80AF93 0.68     [HTML]82B897 0.64     [HTML]88BD9B 0.49

Lexical Similarity Rouge$-$1     [HTML]80B193 0.62     [HTML]80B394 0.54     [HTML]80A38E 0.71     [HTML]80AE92 0.64     [HTML]80AB91 0.71     [HTML]80B093 0.69     [HTML]80B495 0.54

Lexical Similarity Rouge$-$L     [HTML]80B193 0.62     [HTML]80B294 0.54     [HTML]80A38E 0.71     [HTML]80AD92 0.64     [HTML]80A990 0.72     [HTML]80AD92 0.71     [HTML]80AE92 0.57

Lexical Similarity BLEU     [HTML]80AE92 0.63     [HTML]82B898 0.50     [HTML]80B093 0.61     [HTML]81B796 0.58     [HTML]80AE92 0.69     [HTML]80B294 0.68     [HTML]80B193 0.56

PRR$\uparrow$ is aggregated over all selective classification/generation tasks and both considered LLMs, totaling 33K samples. The aggregation is performed on output text lengths within specific quantile-based intervals, with each interval representing an approximately equal number of samples.

Selective Generation.
Tables <ref> and <ref> in Appendix <ref> present results on the selective generation task for StableLM 12B v2 and Mistral 7B v0.2, respectively. For both models, on the text summarization task, the best results are achieved by black-box techniques and the SAR method in terms of both metrics. Interestingly, Conditional PMI substantially outperforms other methods for StableLM, and is on par with black-box techniques for Mistral in terms of ROUGE-L-based PRR. Other information-based methods have negative PRR, which indicates that they perform worse than random choice. On the task of MT for both datasets, MSP and SAR achieve the best results in terms of COMET-based PRR. Nevertheless, black-box techniques are substantially better in terms of AlignScore-based PRR.

Figure <ref> presents mean ranks for each model individually and the mean rank of ranks aggregated for both models for selective generation. For the selective generation with Mistral, SAR performs the best followed by Semantic Entropy, while for results with StableLM, the order is reversed. When considering the average rank for both models, all meaning diversity-based methods  are strong performers, with SAR and Semantic Entropy at the top. These are followed closely by black-box methods such as EigValLaplacian, DegMet, and Lexical Similarity, which perform comparably to each other. MSP trails slightly behind but still demonstrates good performance.

Output Length Impact.
Table <ref> presents PRR scores for both considered LLMs and all datasets aggregated by the length of LLM output. For short generations ($< 7$ symbols), information-based methods (Maximum Probability, Perplexity, PMI, CCP, and TokenSAR) perform the best. Conversely, for longer outputs, black-box methods based on entailment and lexical similarity achieve superior performance. SAR, despite being inferior to black-box techniques for long outputs and slightly inferior to information-based techniques for short outputs, demonstrates the most robust results across all output lengths.

2*UQ Method     2c|Mistral 7b     2cJais 13b

ROC-AUC     PR-AUC
ROC-AUC     PR-AUC

Maximum probability     [rgb]0.9303344867358708, 0.9729181084198386, 0.919284890426759 0.57 $\pm$ 0.01     [rgb]0.7786620530565167, 0.9082122260668973, 0.7734794309880815 0.18 $\pm$ 0.02     [rgb]0.582006920415225, 0.7868204536716648, 0.645997693194925 0.69 $\pm$ 0.02     [rgb]0.6788235294117647, 0.860476739715494, 0.7075355632449059 0.27 $\pm$ 0.01

Perplexity     [rgb]0.8475355632449059, 0.9389004229142637, 0.8349327181853133 0.59 $\pm$ 0.02     [rgb]0.9303344867358708, 0.9729181084198386, 0.919284890426759 0.15 $\pm$ 0.02     [rgb]0.6945098039215687, 0.8681660899653979, 0.7152249134948097 0.61 $\pm$ 0.01     [rgb]0.8662360630526721, 0.9467743175701653, 0.8531410995770857 0.20 $\pm$ 0.02

Maximum token entropy     [rgb]0.8063437139561707, 0.9211303344867359, 0.796239907727797 0.59 $\pm$ 0.01     [rgb]0.7227450980392157, 0.882006920415225, 0.7290657439446366 0.19 $\pm$ 0.03     [rgb]0.6317647058823529, 0.8374086889657824, 0.6844675124951941 0.65 $\pm$ 0.01     [rgb]0.6599999999999999, 0.8512495194156093, 0.6983083429450212 0.28 $\pm$ 0.01

Pointwise Mutual Information     [rgb]0.8264975009611688, 0.9300422914263744, 0.8144482891195693 0.59 $\pm$ 0.01     [rgb]0.8779238754325259, 0.9516955017301039, 0.8645213379469434 0.16 $\pm$ 0.02     [rgb]0.9303344867358708, 0.9729181084198386, 0.919284890426759 0.45 $\pm$ 0.01     [rgb]0.9303344867358708, 0.9729181084198386, 0.919284890426759 0.16 $\pm$ 0.01

p(True)     [rgb]0.7758938869665513, 0.9069204152249135, 0.7712033833141099 0.60 $\pm$ 0.01     [rgb]0.8615609381007305, 0.9448058439061899, 0.8485890042291426 0.16 $\pm$ 0.01     [rgb]0.6850980392156862, 0.8635524798154556, 0.7106113033448673 0.62 $\pm$ 0.01     [rgb]0.8358477508650519, 0.9339792387543253, 0.8235524798154557 0.21 $\pm$ 0.01

CCP     [rgb]0.5217993079584775, 0.7324106113033448, 0.601845444059977 0.63 $\pm$ 0.02     [rgb]0.5217993079584775, 0.7324106113033448, 0.601845444059977 0.22 $\pm$ 0.01     [rgb]0.5217993079584775, 0.7324106113033448, 0.601845444059977 0.74 $\pm$ 0.02     [rgb]0.5217993079584775, 0.7324106113033448, 0.601845444059977 0.35 $\pm$ 0.01

ROC-AUC$\uparrow$ and PR-AUC$\uparrow$ (with unsupported claims as the the positive class) for the Mistral 7b v0.1 and Jais 13b LLMs on the English claim-level fact-checking benchmark. Darker colors indicate better results.

§.§ Fact-Checking

In the fact-checking experiment, we generated English biographies for 100 people, which resulted in 3,824 atomic claims for Mistral 7b v0.1 and 1,407 for Jais 13b. For Mistral, 9.5% of the claims annotated as unsupported by the automatic pipeline, and for Jais 13b – 13%. About 6% of the claims were classified as unknown for the both models used in evaluation. Table <ref> presents the results for the fact-checking in the biographies generation task.
For both models, CCP demonstrates the best performance. This method outperforms the closest baseline Maximum Token Entropy by 0.03 PR-AUC on Mistral and  by 0.07 PR-AUC on Jais.

§.§ Effect of Uncertainty Normalization

The calibration of confidence scores in terms of MSE before and after applying various normalization techniques is presented in Figure <ref>.
We can see that both variants of suggested PCC method (binned and isotonic) confidently outperform the linear and the quantile normalization for almost all of considered UQ techniques.

To verify that the quality of the normalized scores does not degrade after normalization, we also evaluate their performance in selective classification / generation tasks (see Table <ref> in appendix).
We compute PRR for raw and normalized uncertainty scores and analyze the difference. Our observations indicate that scores normalized using linear, quantile, and isotonic PCC methods perform similarly to the raw uncertainty scores, which is anticipated due to their monotonic properties.

MSE$\downarrow$ between AlignScore and various confidence scores derived from uncertainty scores with various normalization methods. Normalizations are trained on concatenated train partitions of all considered datasets. MSE is calculated over concatenated test partitions.

§ CONCLUSION

In this work, we proposed a comprehensive benchmark for evaluating UQ techniques in text generation tasks.
The conducted empirical investigation on the developed benchmark provides several useful insights. We see that the performance of UQ methods depends on the length of the LLM output. For the tasks with shorter answers, we can recommend using information-based methods, while for the tasks that assume longer outputs, methods based on semantic diversity such as Semantic Entropy or Lexical Similarity are preferable. It is worth highlighting that SAR consistently stands out as one of the most effective methods. For the fact-checking task, the best method is CPP.
We also should note that MSP appears to be a very strong and robust baseline across all tasks and should not be discarded from the evaluation protocols. For generating human-interpertable confidence scores, we suggest normalization based on isotonic PCC as it improves the confidence calibration and does not degrade the performance in terms of PRR.

§ METHOD DESCRIPTION

Here, we summarize UE methods implemented in LM-Polygraph; see also Table <ref>.

§.§ White-box Methods

§.§.§ Information-based methods

Maximum sequence probability score simply leverages the probability of the most likely sequence generation:
$\mathrm{MSP}(\yv \mid \xv, \thetav) = 1 - P(\yv \mid \xv, \thetav)$.

Length-normalized log probability computes the average negative log probability of generated tokens. If the score is exponentiated it corresponds to perplexity. The resulting quantity is computed as
P(, ; ) = exp{-1/L logP(|, )},

while it is convenient also to denote length-normalized sequence probability by $\bar{P}(\yv \mid \xv, \thetav) = \exp\Bigl\{\frac{1}{L} \log P(\yv \mid \xv, \thetav)\Bigr\}$.

We also provide the mean token entropy, where we simply average entropy of each individual token in the generated sequence:
_T(, ; ) = 1/L ∑_l = 1^L (y_l |_<l, , ),

where $\HC(y_l \mid \yv_{<l}, \xv, \thetav)$ is an entropy of the token distribution $P(y_l \mid \yv_{<l}, \xv, \thetav)$.

The other possibility to compute entropy-based uncertainty measure is to compute it on the level of whole sequences via $\EE \bigl[-\log P(\yv \mid \xv, \thetav)\bigr]$, where expectation is taken over the sequences $\yv$ randomly generated from the distribution $P(\yv \mid \xv, \thetav)$. In practice, one needs to use Monte-Carlo integration, i.e. generate several sequences $\yv^{(k)}, ~ k = 1, \dots, K$ via randoms sampling and compute the resulting Monte Carlo Sequence Entropy:
_S(; ) = -1/K ∑_k = 1^K logP(^(k) |, ).

The same procedure can be done by substituting $P(\yv^{(k)} \mid \xv, \thetav)$ with its length-normalized version $\bar{P}(\yv^{(k)} \mid \xv, \thetav)$ leading to a more reliable uncertainty measure in some applications.

Finally, one can consider negative mean Pointwise Mutual Information (PMI; <cit.>) which is given by
PMI(, ; ) = 1/L ∑_l = 1^L logP(y_l |_<l, )/P(y_l |_<l, , ).

This method was extended in <cit.> by considering only those marginal probabilities for which the entropy of the conditional distribution is above certain threshold: $\HC(y_l \mid \yv_{<l}, \xv, \thetav) \ge \tau$. It leads to the negative mean Conditional Pointwise Mutual Information (CPMI) measure that is given by:

CPMI(, ; ) = -1/L ∑_l = 1^L logP(y_l |_<l, , )

+ λ/L ∑_l(y_l |_<l, , ) ≥τ logP(y_l |_<l, ),

where $\lambda > 0$ is another tunable parameter.

Another approach to quantify uncertainty based on token probabilities is Rényi divergence <cit.>. This method computes the divergence between the probability distribution for each token and the uniform distribution as follows:

RD(, ; ) =

1/L ∑_l = 1^L 1/α- 1 log∑_i=1^N P(y_i |_<l, , )^α/𝐪_i^α- 1,

where $\alpha > 0$ is tunable parameter, $N$ is the number of tokens in vocabulary, and $\mathbf{q} = \bigl[\frac{1}{N}, \dots,\frac{1}{N}\bigr]$ is a probability vector with uniform distribution over vocabulary.

The other way to compute the distance between probability distributions is the Fisher-Rao distance <cit.>, which is given by:

FR(, ; ) =

1/L ∑_l = 1^L 2/π arccos∑_i=1^N √(P(y_i |_<l, , ) ·𝐪_i).

§.§.§ Methods based on meaning diversity

Generalizing length-normalized log probability, TokenSAR <cit.> computes the weighted average of the negative log probability of generated tokens based on their relevance for the entire generated text. For a given sentence similarity function $g(\cdot, \cdot)$ and token relevance function $R_T(y_k, \yv, \xv) = 1 - g(\xv \cup \yv, \xv \cup \yv \setminus y_k)$, the resulting estimate is given by the following formula:

TokenSAR(, ; ) =

-∑_l = 1^L R̃_T(y_l, , ) logP(y_l |_<l, , ),

where $\tilde{\mathrm{R}}_T(y_k, \yv, \xv) = \frac{\mathrm{R}_T(y_k, \yv, \xv)}{\sum\nolimits_{l = 1}^L \mathrm{R}_T(y_l, \yv, \xv)}$.

Another entropy-based uncertainty measure is Semantic Entropy proposed by <cit.>. The method aims to deal with the generated sequences that have similar meaning while having different probabilities according to the model, which can significantly affect the resulting entropy value (<ref>). The idea is to cluster generated sequences $\yv^{(k)}, ~ k = 1, \dots, K$ into several semantically homogeneous clusters $\CC_m, ~ m = 1, \dots, M$ with $M \le K$ with bi-directional entailment algorithm and average the sequence probabilities within the clusters. The resulting estimate of entropy is given by the following formula:
SE(; ) = -∑_m = 1^M |_m|/K logP̂_m(; ),

where $\hat{P}_m(\xv; \thetav) = \sum_{\yv \in \CC_m} P(\yv \mid \xv, \thetav)$.

One more approach, which used multiple generated sequences is SentenceSAR <cit.>. This method enlarges the probability of those sentences that are more relevant and convincing than others. Given sentence relevance measure $g\bigl(\yv^{(j)}, \yv^{(k)}\bigr)$ of $\yv^{(j)}$ with respect to $\yv^{(k)}$ SentenceSAR is computed as:

R_S (^(j), , )  = ∑_k ≠j g(^(j), ^(k)) P(^(k) |, ).

Then the resulting formula is given by:

SentSAR(; ) =

-1/K ∑_k = 1^K log(P(^(k) |, ) + 1/t R_S (^(k), , )),

where $t$ is a temperature parameter used to control the scale of shifting to relevance.

Additionally, SentenceSAR and TokenSAR can be combined into a single method SAR. To achieve that, we replace the generative probability $P(\yv \mid \xv, \thetav)$ to token-shifted probability $P'(\yv \mid \xv, \thetav) = \exp\{-\mathrm{TokenSAR}(\yv, \xv; \thetav)\}$ in the $\mathrm{SentSAR}(\xv; \thetav)$ formula.

§.§.§ Ensemble-based methods

For the ensembling on a sequence level, we consider two uncertainty measures: total uncertainty measured via average sequence probability $\bar{P}(\yv \mid \xv) = \frac{1}{M} \sum_{i = 1}^M \bar{P}(\yv \mid \xv, \thetav_i)$:

\begin{equation}
\mathrm{MSP}_S(\yv, \xv) = 1 - \bar{P}(\yv \mid \xv)
\label{eq:tu_seq}
\end{equation}

\begin{equation}
\MC_S(\yv, \xv) = \frac{1}{M} \sum\nolimits_{i = 1}^M \log \frac{P(\yv \mid \xv)}{P(\yv \mid \xv, \thetav_i)},
\label{eq:rmi_seq}
\end{equation}

which is known as reverse mutual information (RMI).

Next we discus token level uncertainty measures and start with a total uncertainty estimate via entropy:

\begin{equation}
\HC_T(\yv, \xv) = \sum\nolimits_{l = 1}^{L} \HC(y_l \mid \yv_{<l}, \xv),
\label{eq:tu_toc}
\end{equation}

where $\HC(y_l \mid \yv_{<l}, \xv)$ is an entropy of the token distribution $P(y_l \mid \yv_{<l}, \xv) = \frac{1}{M} \sum_{i = 1}^M P(y_l \mid \yv_{<l}, \xv; \thetav_i)$.

Additionally, for the ensemble one can compute the variety of other token level uncertainty measures including average entropy of ensemble members (also known as Data Uncertainty):
(y_l |_<l, ) = 1/M ∑_i = 1^M (y_l |_<l, , _i),

Mutual Information (MI):

(y_l |_<l, ) = (y_l |_<l, ) - (y_l |_<l, )

and Expected Pairwise KL Divergence (EPKL):

(y_l |_<l, ) = M2^-1 ·

·∑_i ≠j
𝒦ℒ(P(y_l |_<l, , _i)    P(y_l |_<l, , _j)),

where $\mathcal{KL}(P ~\|~ Q)$ refers to a KL-divergence between distributions $P$ and $Q$.

Finally, Reverse Mutual Information (RMI) also can be computed on the token level via a simple equation

(y_l |_<l, ) = (y_l |_<l, ) - (y_l |_<l, ).

The resulting token-level uncertainties computed via Data Uncertainty, MI, EPKL and RMI can be plugged-in in equation (<ref>) on the place of entropy leading to corresponding sequence level uncertainty estimates.

§.§.§ Density-based Methods

Let $h(\xv)$ be a hidden representation of an instance $\xv$. The Mahalanobis Distance (MD; <cit.>) method fits a Gaussian centered at the training data centroid $\mu$ with an empirical covariance matrix $\Sigma$. The uncertainty score is the Mahalanobis distance between $h(\xv)$ and $\mu$:
MD() = (h() - μ)^T Σ^-1 (h() - μ).

We suggest using the last hidden state of the encoder averaged over non-padding tokens or the last hidden state of the decoder averaged over all generated tokens as $h(\xv)$.

The Robust Density Estimation (RDE; <cit.>) method improves over MD by reducing the dimensionality of $h(\xv)$ via PCA decomposition. Additionally, computing of the covariance matrix $\Sigma$ for each individual class is done by using the Minimum Covariance Determinant estimation <cit.>. The uncertainty score is computed as the Mahalanobis distance between but in the space of reduced dimensionality.

<cit.> showed that it might be useful to adjust the Mahalanobis distance score by subtracting from it the other Mahalanobis distance $\mathrm{MD}_0(\xv)$ computed for some large general purpose dataset covering many domains like C4 <cit.>. The resulting resulting Relative Mahalanobis Distance score is
RMD() = MD() - MD_0().

§.§ Black-box Methods

In this work, we follow <cit.> and consider two approaches to compute the similarity for the generated responses. The first one is Jaccard similarity
s(\yv, \yv') = |\yv \cap \yv'| / |\yv \cup \yv'|,
where the sequences $\yv$ and $\yv'$ are considered just as sets of words.

The other similarity measure considered is Natural Language Index (NLI) which employs a classification model to identify whether two responses are similar. We follow <cit.> and use the DeBERTa-large model <cit.> that, for each pair of input sequences, provides two probabilities: $\hat{p}_{\mathrm{entail}}(\yv, \yv')$ that measures the degree of entailment between the sequences and $\hat{p}_{\mathrm{contra}}(\yv, \yv')$ that measures the contradiction between them. Then one can use $s_{\mathrm{entail}}(\yv, \yv') = \hat{p}_{\mathrm{entail}}(\yv, \yv')$ or $s_{\mathrm{contra}}(\yv, \yv') = 1 - \hat{p}_{\mathrm{contra}}(\yv, \yv')$ as a measure of similarity between sequences $\yv$ and $\yv'$.

Number of Semantic Sets illustrates whether answers are semantically equivalent. We adopt an iterative approach by sequentially examining responses from the first to the last while making pairwise comparisons between them (each pair has indexes $j_1$ and $j_2$, $j_2 > j_1$). The number of semantic sets initially equals the total number of generated answers $K$. If the condition $\hat{p}_{\mathrm{entail}}(\yv_{j_1},\yv_{j_2}) > \hat{p}_{\mathrm{contra}}(\yv_{j_1},\yv_{j_2}) $ and $\hat{p}_{\mathrm{entail}}(\yv_{j_2},\yv_{j_1}) > \hat{p}_{\mathrm{contra}}(\yv_{j_2}, \yv_{j_1}) $ is fulfilled we put this two sentences into one cluster. The computation is done for all the pairs of answers, and then the resulting number of distinct sets $U_{NumSemSets}$ is reported.
It is worth noting that a higher number of semantic sets corresponds to an increased level of uncertainty, as it suggests a higher number of diverse semantic interpretations for the answer.

Nonetheless, it is essential to acknowledge a limitation of this measure: it can only take integer values. Additionally, it cannot be assumed that the semantic equivalence derived from the NLI model is always transitive. Consequently, the authors of <cit.> suggest the consideration of a continuous counterpart of this metric. They propose the Sum of Eigenvalues of the Graph Laplacian as a potential alternative approach.

Let's consider a similarity matrix $S_{j_1 j_2} = \bigl(s(\yv_{j_1}, \yv_{j_2}) + s(\yv_{j_2}, \yv_{j_1})\bigr) / 2$. Averaging is done to obtain better consistency. Matrix $S$ has the following formula $L = I - D^{-\frac{1}{2}} S  D^{-\frac{1}{2}}$, where $D$ is a diagonal matrix and $D_{ii} = \sum_{j = 1}^K{S_{ij}}$. Consequently, the following formula is derived: $U_{EigV} = \sum_{k = 1}^{K}{\max(0, 1 - \lambda_{k})}$. This value is a continuous analogue of $U_{NumSemSets}$. In extreme case if adjacency matrix $S$ is binary these two measures will coincide.

Of course, from a theoretical and practical point of view, $U_{EigV}$ is a much more flexible approach compared to $U_{NumSemSets}$. Still, they have a common disadvantage: they can not provide uncertainty for each answer. However, authors of <cit.> demonstrate that we can take it from Degree Matrix $D$ computed above. The idea is that the total uncertainty of the answers might be measured as a corrected trace of the diagonal matrix $D$ because elements on the diagonal of matrix $D$ are sums of similarities between the given answer and other answers. Thus, it is an average pairwise distance between all answers, and a larger value will indicate larger uncertainty because of the larger distance between answers. The resulting uncertainty measure becomes $U_{Deg} = 1 - trace(D) / K^{2}$.

A drawback of previously considered methods is the limited knowledge of the actual embedding space for the different answers since we only have measures of their similarities. Nevertheless, we can overcome this limitation by taking advantage of the inferential capabilities of the graph Laplacian, which makes it easier to obtain the coordinates of the answers. Let us introduce $\uv_1, \dots, \uv_{k} \in R^{K}$ as the eigenvectors of $L$ that correspond to $k$ smallest eigenvalues. We can efficiently construct an informative embedding $\vv_{j} = [\uv_{1,j}, \dots, \uv_{k,j}]$ for an answer $\yv_{j}$. Authors of <cit.> demonstrate that this approach allows the usage of the average distance from the center as an uncertainty metric and to consider the distance of each response from the center as a measure of (negative) confidence. In mathematical terms, the estimates for Eccentricity can be defined as follows: $U_{Ecc} = \bigl\|[\tilde{\vv}_{1}^{T}, \dots, \tilde{\vv}_{K}^{T}]\bigr\|_{2}$, where $\tilde{\vv}_{j} = \vv_{j} - \frac{1}{K}\sum_{\ell = 1}^{K}{\vv_{\ell}}$.

Last but not least, Lexical Similarity is a measure proposed by <cit.> that computes how similar two words or phrases are in terms of their meaning. Since the original article is dedicated to machine translation, this measure calculates the average similarity score between all pairs of translation hypotheses in a set, using a similarity measure based on the overlap of their lexical items. Different metrics can be used, such as ROUGE-1, ROUGE-2, ROUGE-L, and BLEU. For our task, this measure iterates over all responses and calculates the average score with other answers.

§.§ Claim-level Methods

While the previous methods operate on the entire generated sequence, it is often desirable to estimate uncertainty for individual claims within the text. Suppose $C$ denotes a set of token indices corresponding to a particular claim.
Many UQ methods allow for straightforward generalization for the claim-level by simply considering only subset of tokens corresponding to the set $C$ instead of all tokens in the sequence. We consider claim-level generalization for MSP, Maximum Token Entropy, Perplexity, and PMI in this way.

P(True), adapting the approach from <cit.> to the claim level, quantifies claim uncertainty by prompting the LLM to assess the truthfulness of the generated claim. It is implemented as the model's confidence in generating the token “True" as the first token $y_1$ of a separate response, given the generated claim as context:
P_True(C |, ) = 1 - P(y_1 = “True” |C, , ).

Claim-Conditioned Probability (CCP) quantifies uncertainty by evaluating the semantic similarity between the original claim and perturbed versions where each token is replaced with its alternative generations. CCP utilizes a Natural Language Inference (NLI) model to compare the original claim $y_{i \in C, i \le j}$  with variations where token $y_j$ is replaced with top-$K$ alternatives $y^k_j$ from the model's output distribution:

CCP(y_j |_<j, , ) =

∑_k: (y^k_j, y_j) =  P(y^k_j |_<j, ) ∑_k: (y^k_j, y_j) ∈{ , } P(y^k_j |_<j , ),

CCP(C |, ) = 1 - ∏_j ∈C CCP(y_j |_<j, , ).

Here, $\texttt{NLI}(y^k_j, y_j) = \texttt{`e'}$ denotes that the NLI model predicts an entailment relation between the original claim and the modified claim where $y_j$ is replaced with $y^k_j$. CCP effectively measures the proportion of high-probability token alternatives that preserve the original claim's semantic meaning according to the NLI model.

§ UNCERTAINTY NORMALIZATION DETAILS

Dataset     Task     Max Input Length     Generation Length     Temperature     Top-p     Do Sample     Beams     Repetition Penalty

CoQA     4*QA     7*-     20     7*1.0     7*1.0     7*False     7*1     7*1

TriviQA               20

MMLU               3

GSM8k               256

XSum     1*TS           56

WMT14 Fr-En     2*NMT          107

WMT19 De-En               107

Text generation hyperparameters for both LLMs Mistral-7B-v0.2 and StableLM-12B-v2.

Dataset     Type     Num. instances     Av. document len.     Av. target len.     Language

CoQA     QA     7199 / 500 / -     405.9     4     English

TriviaQA     QA     138384 / 17944 / 17210     18.8     4.3     English

MMLU     QA     99842 / 1531 / 14042     64.9     3     English

GSM8k     QA     7473 / - / 1319     64.2     128.6     English

WMT'14     NMT     40.8M / 3000 / 3003     49.3     32.9     French-to-English

WMT'19     NMT     34.8M / 2998 / -      52.5     33.5     German-to-English

XSum     ATS     204045 / 11332 / 11334     544.2     30.4     English

Quantitative information regarding the datasets from experiments. It includes the count of instances available for the training, validation, and test sets, as well as the mean lengths of both texts and targets (answers / translations / summaries) measured in terms of tokens according to Mistral 7B v0.2 tokenizer. In addition, the languages of the source and target texts are also specified.

Linear scaling first reverses the order of UQ values in calibration set to get raw confidences $c_i = -u_i$. Then, for a new model output with corresponding uncertainty score $u$ and confidence $c(u) = - u$, scaled confidence is
$c_s(u) = (c(u) - \min_i c_i) / (\max_i c_i - \min_i c_i)$.

Depending on the generality of the calibration set, new uncertainty values may exceed maximum uncertainty, or be less than minimum uncertainty observed in a calibration set. For such outliers, obtained scaled uncertainty will be greater than 1 and lesser than 0 respectively. To bound confidence strictly within $[0, 1]$ interval we clip such outliers and resulting confidence $c(u)$ is then obtained as:

\begin{equation}
c(u) = \begin{cases}
0, & c_s(u) < 0, \\
c_s(u), & 0 \leq c_s(u) \leq 1, \\
1, & c_s(u) > 1. \\
\end{cases}
\end{equation}

Quantile scaling computes confidence as $c(u) = 1 - \frac{1}{N}\sum\nolimits_{i=1}^{N}\mathbb{I}(u_i \leq u)$,
where $N$ is the number of data points in $\mathcal{D}_{calib}$ and $u_i \in \mathcal{D}_{calib}$. Resulting confidence is 1 - value of uncertainty CDF estimated using calibration data.

This approach naturally bounds the output confidence between 0 and 1 and does this with consideration to the distribution of uncertainty values in the calibration set.

Binned PCC splits the calibration set into non-intersecting bins based on values of uncertainty $u_i$. Thus, a bin is a set of indices $\mathcal{B}^j = \{i\colon b^j_{\min} \le u_i < b^j_{\max}\}$, where $b^j_{\min}$ and $b^j_{\max}$ are left and right boundaries of the $j$-th bin respectively.

Then, for a new data point with a raw uncertainty score $u$, a calibrated confidence score is $c(u) = \frac{1}{|\mathcal{B}'|} \sum\nolimits_{i \in \mathcal{B}'}q_i$,
where $\mathcal{B}'$ is the calibration bin, for which the following holds: $b'_{\min} \le u < b'_{\max}$.

To remedy this problem we propose the second approach, Isotonic PCC, confidence score is obtained as $c(u) = \mathrm{CIR}(u)$, where CIR is a fitted Centered Isotonic Regression that predicts quality of the response based on it's raw uncertainty score, maintaining strict monotonicity, i.e. higher uncertainty always produces lower confidence.

§ GENERATION HYPERPARAMETERS

Table <ref> presents the hyperparameters used for experiments with LLMs Mistral-7B-v0.2 and StableLM-12B-v2 on various datasets and tasks.
Maximum length of generated sequence was set for each dataset as the 99th percentile of target sequence length on the respective train set.

§ DATASET STATISTICS

Language     Acc.     F1
score     # claims     % of
false claims

English     0.98     0.93     97     16.5%

Chinese     0.77     0.74     100     39.0%

Arabic     0.89     0.80     132     28.3%

Russian     0.89     0.80     275     15.6%

Classification metrics of GPT-4 annotation against manual annotation (unsupported claims represent a positive class).

Table <ref> presents the statistics of the benchmark datasets. All test set instances were used, while density-based methods were trained on 1,000 randomly selected training instances.

Table <ref> presents the quality of annotation using GPT-4 in the claim-level fact-checking benchmark. English texts are generated using Mistral-7B-v0.1, Arabic texts – using Jais 13B, Chinese texts – using Yi 7B <cit.>, Russian texts – using Vikhr 7B-0.2 <cit.>.

§ DETAILED EXPERIMENTAL RESULTS FOR UNCERTAINTY QUANTIFICATION METHODS

2*UQ Method     2c|CoQA     2c|TriviaQA     1c|MMLU     1c|GSM8k      2*Mean Rank$\downarrow$
Accuracy     AlignScore    Accuracy     AlignScore    Accuracy     Accuracy
Maximum Sequence Probability     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.68$\pm$0.01     [rgb]0.5,0.6913417916186082,0.5769934640522876 0.46$\pm$0.01     [rgb]0.5,0.6383775470972703,0.5550326797385621 0.74$\pm$0.01     [rgb]0.5,0.668642829680892,0.5675816993464052 0.70$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.60$\pm$0.01     [rgb]0.5519415609381008,0.7582468281430219,0.6233756247597078 0.35$\pm$0.01     [rgb]0.5,0.7064744329104191,0.5832679738562092 7.9

Perplexity     [rgb]0.5690888119953864,0.7730411380238369,0.6356632064590542 0.39$\pm$0.01     [rgb]0.5,0.7089965397923875,0.5843137254901961 0.44$\pm$0.01     [rgb]0.5,0.6434217608612072,0.557124183006536 0.73$\pm$0.01     [rgb]0.5,0.668642829680892,0.5675816993464052 0.70$\pm$0.01     [rgb]0.5,0.711518646674356,0.585359477124183 0.45$\pm$0.01     [rgb]0.5,0.6913417916186082,0.5769934640522876 0.51$\pm$0.01     [rgb]0.5727797001153403,0.7769780853517878,0.6386159169550173 11.1

Mean Token Entropy     [rgb]0.6023068050749711,0.808473663975394,0.662237600922722 0.31$\pm$0.01     [rgb]0.5,0.7089965397923875,0.5843137254901961 0.44$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.75$\pm$0.01     [rgb]0.5,0.6610765090349866,0.5644444444444445 0.71$\pm$0.01     [rgb]0.8146482122260669,0.9250057670126874,0.8030680507497117 $-$0.16$\pm$0.01     [rgb]0.5,0.6837754709727029,0.5738562091503268 0.52$\pm$0.01     [rgb]0.6133794694348328,0.8202845059592464,0.6710957324106113 13.0

Monte Carlo Sequence Entropy     [rgb]0.5,0.6888196847366397,0.5759477124183007 0.58$\pm$0.01     [rgb]0.5,0.6913417916186082,0.5769934640522876 0.46$\pm$0.01     [rgb]0.5,0.6837754709727029,0.5738562091503268 0.66$\pm$0.01     [rgb]0.5,0.7039523260284506,0.5822222222222222 0.64$\pm$0.01     [rgb]0.5746251441753172,0.7789465590157632,0.6400922722029988 0.28$\pm$0.01     [rgb]0.5,0.7014302191464822,0.5811764705882353 0.49$\pm$0.01     [rgb]0.6599999999999999,0.8512495194156093,0.6983083429450212 14.5

Monte Carlo Normalized Sequence Entropy     [rgb]0.5454825067281814,0.752710495963091,0.6187620146097654 0.44$\pm$0.01     [rgb]0.5110342176086121,0.7231833910034602,0.5941560938100731 0.42$\pm$0.01     [rgb]0.5,0.6787312572087658,0.571764705882353 0.67$\pm$0.01     [rgb]0.5,0.7039523260284506,0.5822222222222222 0.64$\pm$0.01     [rgb]0.582006920415225,0.7868204536716648,0.645997693194925 0.26$\pm$0.01     [rgb]0.5,0.7014302191464822,0.5811764705882353 0.49$\pm$0.01     [rgb]0.7537485582468282,0.8965859284890427,0.7529950019223376 17.2

Pointwise Mutual Information     [rgb]0.9469434832756632,0.9795617070357555,0.9370011534025375 $-$0.32$\pm$0.01     [rgb]0.8755863129565551,0.9507112648981161,0.8622452902729719 $-$0.02$\pm$0.01     [rgb]0.8545482506728181,0.9418531334102268,0.841760861207228 $-$0.01$\pm$0.01     [rgb]0.8915801614763552,0.9574163783160323,0.8779469434832756 $-$0.06$\pm$0.01     [rgb]0.6694117647058824,0.8558631295655517,0.7029219530949635 0.08$\pm$0.01     [rgb]0.6819607843137254,0.8620146097654748,0.7090734332948866 0.08$\pm$0.02     [rgb]0.9395617070357555,0.9766089965397924,0.9291272587466359 24.1

Conditional Pointwise Mutual Information     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.47$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.23$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.35$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.33$\pm$0.01     [rgb]0.6882352941176471,0.8650903498654363,0.7121491733948482 0.05$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.60$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 27.3

RenyiNeg     [rgb]0.6317647058823529,0.8374086889657824,0.6844675124951941 0.24$\pm$0.01     [rgb]0.8825990003844675,0.9536639753940792,0.8690734332948866 $-$0.03$\pm$0.01     [rgb]0.9100346020761245,0.9647981545559401,0.8976316801230296 $-$0.12$\pm$0.01     [rgb]0.8638985005767013,0.9457900807381776,0.8508650519031142 $-$0.01$\pm$0.01     [rgb]0.5153402537485583,0.7268742791234141,0.5972318339100346 0.41$\pm$0.01     [rgb]0.8971164936562861,0.9596309111880046,0.8838523644752019 $-$0.31$\pm$0.01     [rgb]0.9081891580161476,0.9640599769319492,0.8956632064590542 22.6

FisherRao     [rgb]0.6317647058823529,0.8374086889657824,0.6844675124951941 0.24$\pm$0.01     [rgb]0.8755863129565551,0.9507112648981161,0.8622452902729719 $-$0.02$\pm$0.01     [rgb]0.9100346020761245,0.9647981545559401,0.8976316801230296 $-$0.12$\pm$0.01     [rgb]0.8638985005767013,0.9457900807381776,0.8508650519031142 $-$0.01$\pm$0.01     [rgb]0.5110342176086121,0.7231833910034602,0.5941560938100731 0.42$\pm$0.01     [rgb]0.8685736255286428,0.947758554402153,0.8554171472510572 $-$0.25$\pm$0.01     [rgb]0.8934256055363322,0.9581545559400231,0.8799154171472511 21.9

TokenSAR     [rgb]0.5605536332179931,0.7656286043829297,0.6295271049596309 0.41$\pm$0.01     [rgb]0.5,0.6989081122645137,0.5801307189542484 0.45$\pm$0.01     [rgb]0.5,0.6560322952710496,0.5623529411764706 0.71$\pm$0.01     [rgb]0.5,0.6736870434448289,0.569673202614379 0.69$\pm$0.01     [rgb]0.5,0.711518646674356,0.585359477124183 0.45$\pm$0.01     [rgb]0.5,0.6963860053825451,0.5790849673202614 0.50$\pm$0.01     [rgb]0.607843137254902,0.8143790849673203,0.6666666666666666 12.7

CCP     [rgb]0.5217993079584775,0.7324106113033448,0.601845444059977 0.49$\pm$0.01     [rgb]0.5454825067281814,0.752710495963091,0.6187620146097654 0.37$\pm$0.01     [rgb]0.5,0.6560322952710496,0.5623529411764706 0.71$\pm$0.01     [rgb]0.5,0.6560322952710496,0.5623529411764706 0.72$\pm$0.00     [rgb]0.5,0.6837754709727029,0.5738562091503268 0.50$\pm$0.01     [rgb]0.5024221453287198,0.7158016147635524,0.58800461361015 0.46$\pm$0.01     [rgb]0.5746251441753172,0.7789465590157632,0.6400922722029988 11.2

Semantic Entropy     [rgb]0.5,0.6888196847366397,0.5759477124183007 0.58$\pm$0.01     [rgb]0.5,0.6736870434448289,0.569673202614379 0.48$\pm$0.01     [rgb]0.5,0.6560322952710496,0.5623529411764706 0.71$\pm$0.01     [rgb]0.5,0.6736870434448289,0.569673202614379 0.69$\pm$0.00     [rgb]0.5476355247981546,0.754555940023068,0.6202998846597463 0.34$\pm$0.01     [rgb]0.5,0.7014302191464822,0.5811764705882353 0.49$\pm$0.01     [rgb]0.5930795847750865,0.798631295655517,0.6548558246828142 12.0

SentenceSAR     [rgb]0.5,0.6535101883890811,0.5613071895424837 0.64$\pm$0.01     [rgb]0.5,0.6484659746251442,0.5592156862745098 0.51$\pm$0.01     [rgb]0.5,0.6434217608612072,0.557124183006536 0.73$\pm$0.01     [rgb]0.5,0.6560322952710496,0.5623529411764706 0.72$\pm$0.01     [rgb]0.5,0.6736870434448289,0.569673202614379 0.52$\pm$0.01     [rgb]0.6244521337946943,0.8320953479430988,0.6799538638985005 0.17$\pm$0.01     [rgb]0.5,0.663598615916955,0.5654901960784313 6.5

SAR     [rgb]0.5174932718185313,0.728719723183391,0.5987697039600154 0.50$\pm$0.01     [rgb]0.5,0.6408996539792388,0.556078431372549 0.52$\pm$0.01     [rgb]0.5,0.6484659746251442,0.5592156862745098 0.72$\pm$0.01     [rgb]0.5,0.6509880815071126,0.5602614379084967 0.73$\pm$0.01     [rgb]0.5,0.7064744329104191,0.5832679738562092 0.46$\pm$0.01     [rgb]0.5,0.6535101883890811,0.5613071895424837 0.58$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 5.4

Mahalanobis Distance $-$ Decoder     [rgb]0.8035755478662052,0.9198385236447519,0.7939638600538255 $-$0.02$\pm$0.01     [rgb]0.8522106881968474,0.9408688965782391,0.8394848135332564 0.01$\pm$0.01     [rgb]0.9100346020761245,0.9647981545559401,0.8976316801230296 $-$0.12$\pm$0.01     [rgb]0.9510957324106113,0.981199538638985,0.9416378316032294 $-$0.20$\pm$0.01     [rgb]0.6882352941176471,0.8650903498654363,0.7121491733948482 0.05$\pm$0.01     [rgb]0.534717416378316,0.7434832756632064,0.6110726643598616 0.39$\pm$0.01     [rgb]0.9229527104959632,0.9699653979238754,0.9114109957708574 23.3

Relative Mahalanobis Distance $-$ Decoder     [rgb]0.8218223760092271,0.9280738177623991,0.8098961937716262 $-$0.05$\pm$0.01     [rgb]0.8288350634371395,0.9310265282583622,0.816724336793541 0.04$\pm$0.01     [rgb]0.9100346020761245,0.9647981545559401,0.8976316801230296 $-$0.12$\pm$0.01     [rgb]0.9510957324106113,0.981199538638985,0.9416378316032294 $-$0.20$\pm$0.01     [rgb]0.7399077277970012,0.8901268742791234,0.7416147635524798 $-$0.03$\pm$0.01     [rgb]0.534717416378316,0.7434832756632064,0.6110726643598616 0.39$\pm$0.01     [rgb]0.9340253748558247,0.9743944636678201,0.9232218377547097 23.8

RDE $-$ Decoder     [rgb]0.8118800461361015,0.9237139561707035,0.80079200307574 $-$0.03$\pm$0.01     [rgb]0.8896116878123799,0.9566166858900422,0.8759015763168012 $-$0.04$\pm$0.01     [rgb]0.8709111880046136,0.9487427912341406,0.8576931949250288 $-$0.04$\pm$0.01     [rgb]0.9081891580161476,0.9640599769319492,0.8956632064590542 $-$0.10$\pm$0.01     [rgb]0.7509803921568627,0.8952941176470588,0.750718954248366 $-$0.05$\pm$0.01     [rgb]0.5261053440984237,0.7361014994232987,0.6049211841599385 0.41$\pm$0.01     [rgb]0.9321799307958478,0.9736562860438294,0.9212533640907343 23.7

HUQ$-$MD $-$ Decoder     [rgb]0.5856978085351787,0.7907574009996154,0.6489504036908881 0.35$\pm$0.01     [rgb]0.5024221453287198,0.7158016147635524,0.58800461361015 0.43$\pm$0.01     [rgb]0.5,0.6434217608612072,0.557124183006536 0.73$\pm$0.01     [rgb]0.5,0.668642829680892,0.5675816993464052 0.70$\pm$0.01     [rgb]0.5,0.711518646674356,0.585359477124183 0.45$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.62$\pm$0.01     [rgb]0.5454825067281814,0.752710495963091,0.6187620146097654 10.0

P(True)     [rgb]0.8171472510572857,0.9261053440984237,0.8053440984236832 $-$0.04$\pm$0.01     [rgb]0.8662360630526721,0.9467743175701653,0.8531410995770857 $-$0.01$\pm$0.01     [rgb]0.7258823529411764,0.8835447904652057,0.7306036139946175 0.20$\pm$0.01     [rgb]0.7164705882352941,0.8789311803152634,0.7259900038446752 0.23$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.62$\pm$0.01     [rgb]0.6349019607843137,0.8389465590157632,0.686005382545175 0.15$\pm$0.01     [rgb]0.9284890426758938,0.9721799307958477,0.9173164167627835 23.5

NumSemSets     [rgb]0.6756862745098039,0.8589388696655132,0.705997693194925 0.18$\pm$0.01     [rgb]0.6317647058823529,0.8374086889657824,0.6844675124951941 0.24$\pm$0.01     [rgb]0.5,0.6963860053825451,0.5790849673202614 0.64$\pm$0.01     [rgb]0.5,0.6736870434448289,0.569673202614379 0.69$\pm$0.01     [rgb]0.5476355247981546,0.754555940023068,0.6202998846597463 0.34$\pm$0.01     [rgb]0.6411764705882352,0.8420222991157247,0.6890811226451365 0.14$\pm$0.01     [rgb]0.8358477508650519,0.9339792387543253,0.8235524798154557 19.8

EigValLaplacian NLI Score entail.     [rgb]0.5067281814686659,0.7194925028835063,0.5910803537101115 0.52$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.53$\pm$0.01     [rgb]0.5,0.6560322952710496,0.5623529411764706 0.71$\pm$0.01     [rgb]0.5,0.6509880815071126,0.5602614379084967 0.73$\pm$0.01     [rgb]0.5110342176086121,0.7231833910034602,0.5941560938100731 0.42$\pm$0.01     [rgb]0.534717416378316,0.7434832756632064,0.6110726643598616 0.39$\pm$0.01     [rgb]0.534717416378316,0.7434832756632064,0.6110726643598616 9.6

EigValLaplacian Jaccard Score     [rgb]0.5454825067281814,0.752710495963091,0.6187620146097654 0.44$\pm$0.01     [rgb]0.5,0.6989081122645137,0.5801307189542484 0.45$\pm$0.01     [rgb]0.5,0.6661207227989234,0.5665359477124183 0.69$\pm$0.01     [rgb]0.5,0.6736870434448289,0.569673202614379 0.69$\pm$0.01     [rgb]0.5390234525182622,0.7471741637831604,0.6141484044598231 0.36$\pm$0.01     [rgb]0.5,0.6837754709727029,0.5738562091503268 0.52$\pm$0.01     [rgb]0.6317647058823529,0.8374086889657824,0.6844675124951941 13.8

DegMat NLI Score entail.     [rgb]0.5540945790080738,0.7600922722029988,0.6249134948096886 0.42$\pm$0.01     [rgb]0.5,0.6484659746251442,0.5592156862745098 0.51$\pm$0.01     [rgb]0.5,0.6434217608612072,0.557124183006536 0.73$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.76$\pm$0.00     [rgb]0.5,0.711518646674356,0.585359477124183 0.45$\pm$0.01     [rgb]0.5,0.711518646674356,0.585359477124183 0.47$\pm$0.01     [rgb]0.5,0.7089965397923875,0.5843137254901961 8.0

DegMat Jaccard Score     [rgb]0.5,0.711518646674356,0.585359477124183 0.54$\pm$0.01     [rgb]0.5,0.6661207227989234,0.5665359477124183 0.49$\pm$0.01     [rgb]0.5,0.6484659746251442,0.5592156862745098 0.72$\pm$0.01     [rgb]0.5,0.6610765090349866,0.5644444444444445 0.71$\pm$0.01     [rgb]0.5024221453287198,0.7158016147635524,0.58800461361015 0.44$\pm$0.01     [rgb]0.5,0.6736870434448289,0.569673202614379 0.54$\pm$0.01     [rgb]0.5,0.7014302191464822,0.5811764705882353 7.8

Eccentricity NLI Score entail.     [rgb]0.5024221453287198,0.7158016147635524,0.58800461361015 0.53$\pm$0.01     [rgb]0.5,0.6585544021530181,0.5633986928104575 0.50$\pm$0.01     [rgb]0.5,0.6484659746251442,0.5592156862745098 0.72$\pm$0.01     [rgb]0.5,0.6610765090349866,0.5644444444444445 0.71$\pm$0.01     [rgb]0.5261053440984237,0.7361014994232987,0.6049211841599385 0.39$\pm$0.01     [rgb]0.5,0.6787312572087658,0.571764705882353 0.53$\pm$0.01     [rgb]0.5110342176086121,0.7231833910034602,0.5941560938100731 8.6

Eccentricity Jaccard Score     [rgb]0.5,0.6888196847366397,0.5759477124183007 0.58$\pm$0.01     [rgb]0.5,0.6661207227989234,0.5665359477124183 0.49$\pm$0.01     [rgb]0.5,0.6661207227989234,0.5665359477124183 0.69$\pm$0.01     [rgb]0.5,0.6787312572087658,0.571764705882353 0.68$\pm$0.01     [rgb]0.534717416378316,0.7434832756632064,0.6110726643598616 0.37$\pm$0.01     [rgb]0.5,0.6837754709727029,0.5738562091503268 0.52$\pm$0.01     [rgb]0.580161476355248,0.7848519800076893,0.6445213379469434 11.4

Lexical Similarity Rouge$-$L     [rgb]0.5024221453287198,0.7158016147635524,0.58800461361015 0.53$\pm$0.01     [rgb]0.5,0.6484659746251442,0.5592156862745098 0.51$\pm$0.01     [rgb]0.5,0.6484659746251442,0.5592156862745098 0.72$\pm$0.01     [rgb]0.5,0.6610765090349866,0.5644444444444445 0.71$\pm$0.01     [rgb]0.5,0.7014302191464822,0.5811764705882353 0.47$\pm$0.01     [rgb]0.5,0.6434217608612072,0.557124183006536 0.60$\pm$0.01     [rgb]0.5,0.6408996539792388,0.556078431372549 5.8

Lexical Similarity BLEU     [rgb]0.5,0.6938638985005767,0.5780392156862745 0.57$\pm$0.01     [rgb]0.5,0.6736870434448289,0.569673202614379 0.48$\pm$0.01     [rgb]0.5,0.6661207227989234,0.5665359477124183 0.69$\pm$0.01     [rgb]0.5,0.6862975778546713,0.5749019607843138 0.67$\pm$0.01     [rgb]0.5,0.7064744329104191,0.5832679738562092 0.46$\pm$0.01     [rgb]0.5,0.663598615916955,0.5654901960784313 0.56$\pm$0.01     [rgb]0.5411764705882353,0.7490196078431373,0.615686274509804 9.8

PRR$\uparrow$ with various generation quality metrics for UQ methods in selective classification tasks with the StableLM 12b v2 model. Darker color indicates better results.
2*UQ Method     2c|CoQA     2c|TriviaQA     1c|MMLU     1c|GSM8k      2*Mean Rank
Accuracy     AlignScore    Accuracy     AlignScore    Accuracy     Accuracy
Maximum Sequence Probability     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.68$\pm$0.01     [rgb]0.5,0.6408996539792388,0.556078431372549 0.50$\pm$0.01     [rgb]0.5,0.6408996539792388,0.556078431372549 0.78$\pm$0.01     [rgb]0.5,0.6408996539792388,0.556078431372549 0.76$\pm$0.00     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.69$\pm$0.00     [rgb]0.6096885813148789,0.8163475586312956,0.6681430219146482 0.24$\pm$0.02     [rgb]0.5,0.6585544021530181,0.5633986928104575 6.0

Perplexity     [rgb]0.5390234525182622,0.7471741637831604,0.6141484044598231 0.43$\pm$0.01     [rgb]0.5,0.6661207227989234,0.5665359477124183 0.47$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.80$\pm$0.01     [rgb]0.5,0.6358554402153018,0.5539869281045752 0.77$\pm$0.00     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.69$\pm$0.00     [rgb]0.7070588235294117,0.874317570165321,0.7213763936947328 0.13$\pm$0.01     [rgb]0.534717416378316,0.7434832756632064,0.6110726643598616 9.1

Mean Token Entropy     [rgb]0.5856978085351787,0.7907574009996154,0.6489504036908881 0.32$\pm$0.01     [rgb]0.5,0.7089965397923875,0.5843137254901961 0.42$\pm$0.01     [rgb]0.5,0.6358554402153018,0.5539869281045752 0.79$\pm$0.01     [rgb]0.5,0.6358554402153018,0.5539869281045752 0.77$\pm$0.00     [rgb]0.5,0.6408996539792388,0.556078431372549 0.68$\pm$0.00     [rgb]0.6662745098039216,0.8543252595155709,0.7013840830449827 0.17$\pm$0.02     [rgb]0.5912341407151096,0.7966628219915417,0.6533794694348327 11.4

Monte Carlo Sequence Entropy     [rgb]0.5,0.7039523260284506,0.5822222222222222 0.54$\pm$0.01     [rgb]0.5,0.6913417916186082,0.5769934640522876 0.44$\pm$0.01     [rgb]0.5,0.6888196847366397,0.5759477124183007 0.68$\pm$0.01     [rgb]0.5,0.6787312572087658,0.571764705882353 0.68$\pm$0.00     [rgb]0.5,0.7014302191464822,0.5811764705882353 0.60$\pm$0.00     [rgb]0.5,0.6560322952710496,0.5623529411764706 0.48$\pm$0.01     [rgb]0.6317647058823529,0.8374086889657824,0.6844675124951941 13.1

Monte Carlo Normalized Sequence Entropy     [rgb]0.5390234525182622,0.7471741637831604,0.6141484044598231 0.43$\pm$0.01     [rgb]0.5,0.7089965397923875,0.5843137254901961 0.42$\pm$0.01     [rgb]0.5,0.6736870434448289,0.569673202614379 0.71$\pm$0.01     [rgb]0.5,0.663598615916955,0.5654901960784313 0.71$\pm$0.00     [rgb]0.5,0.7014302191464822,0.5811764705882353 0.60$\pm$0.00     [rgb]0.5002691272587466,0.7139561707035755,0.5864667435601691 0.41$\pm$0.02     [rgb]0.7164705882352941,0.8789311803152634,0.7259900038446752 15.3

Pointwise Mutual Information     [rgb]0.9414071510957325,0.9773471741637831,0.9310957324106113 $-$0.40$\pm$0.01     [rgb]0.9284890426758938,0.9721799307958477,0.9173164167627835 $-$0.12$\pm$0.01     [rgb]0.7925028835063437,0.9146712802768167,0.7848596693579393 $-$0.01$\pm$0.01     [rgb]0.7841983852364476,0.9107958477508651,0.7780315263360246 $-$0.02$\pm$0.01     [rgb]0.5,0.6862975778546713,0.5749019607843138 0.62$\pm$0.01     [rgb]0.7731257208765859,0.9056286043829296,0.7689273356401384 0.06$\pm$0.02     [rgb]0.9192618223760092,0.9684890426758939,0.9074740484429066 22.1

Conditional Pointwise Mutual Information     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.58$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.25$\pm$0.01     [rgb]0.9395617070357555,0.9766089965397924,0.9291272587466359 $-$0.37$\pm$0.01     [rgb]0.9303344867358708,0.9729181084198386,0.919284890426759 $-$0.37$\pm$0.01     [rgb]0.6262975778546713,0.8340638216070742,0.681430219146482 0.38$\pm$0.01     [rgb]0.9432525951557094,0.9780853517877739,0.9330642060745866 $-$0.18$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 26.0

RenyiNeg     [rgb]0.7007843137254902,0.8712418300653595,0.7183006535947712 0.09$\pm$0.01     [rgb]0.8825990003844675,0.9536639753940792,0.8690734332948866 $-$0.05$\pm$0.01     [rgb]0.9809919261822376,0.9928258362168396,0.9765167243367936 $-$0.55$\pm$0.01     [rgb]0.9787773933102653,0.9919646289888504,0.9739331026528258 $-$0.57$\pm$0.01     [rgb]0.8311726259131104,0.9320107650903499,0.8190003844675124 0.15$\pm$0.01     [rgb]0.9809919261822376,0.9928258362168396,0.9765167243367936 $-$0.28$\pm$0.01     [rgb]0.9732410611303345,0.9898116109188774,0.9674740484429065 25.2

FisherRao     [rgb]0.7007843137254902,0.8712418300653595,0.7183006535947712 0.09$\pm$0.01     [rgb]0.8755863129565551,0.9507112648981161,0.8622452902729719 $-$0.04$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.57$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.60$\pm$0.01     [rgb]0.8311726259131104,0.9320107650903499,0.8190003844675124 0.15$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.29$\pm$0.01     [rgb]0.976562860438293,0.9911034217608612,0.9713494809688581 25.4

TokenSAR     [rgb]0.5390234525182622,0.7471741637831604,0.6141484044598231 0.43$\pm$0.01     [rgb]0.5,0.6661207227989234,0.5665359477124183 0.47$\pm$0.01     [rgb]0.5,0.6408996539792388,0.556078431372549 0.78$\pm$0.01     [rgb]0.5,0.6408996539792388,0.556078431372549 0.76$\pm$0.00     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.69$\pm$0.00     [rgb]0.7070588235294117,0.874317570165321,0.7213763936947328 0.13$\pm$0.01     [rgb]0.5627066512879662,0.7674740484429066,0.6310649750096117 10.2

CCP     [rgb]0.5304113802383699,0.7397923875432526,0.6079969242599 0.45$\pm$0.01     [rgb]0.5,0.7089965397923875,0.5843137254901961 0.42$\pm$0.01     [rgb]0.5,0.6610765090349866,0.5644444444444445 0.74$\pm$0.01     [rgb]0.5,0.6610765090349866,0.5644444444444445 0.72$\pm$0.00     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.69$\pm$0.00     [rgb]0.5690888119953864,0.7730411380238369,0.6356632064590542 0.31$\pm$0.01     [rgb]0.6004613610149943,0.8065051903114187,0.6607612456747405 11.8

Semantic Entropy     [rgb]0.5,0.6989081122645137,0.5801307189542484 0.55$\pm$0.01     [rgb]0.5,0.6585544021530181,0.5633986928104575 0.48$\pm$0.01     [rgb]0.5,0.6661207227989234,0.5665359477124183 0.73$\pm$0.01     [rgb]0.5,0.6560322952710496,0.5623529411764706 0.73$\pm$0.00     [rgb]0.5,0.6862975778546713,0.5749019607843138 0.62$\pm$0.00     [rgb]0.5,0.663598615916955,0.5654901960784313 0.47$\pm$0.01     [rgb]0.5131872356785852,0.7250288350634371,0.5956939638600538 8.3

SentenceSAR     [rgb]0.5,0.6585544021530181,0.5633986928104575 0.63$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.51$\pm$0.01     [rgb]0.5,0.6459438677431757,0.5581699346405229 0.77$\pm$0.01     [rgb]0.5,0.6358554402153018,0.5539869281045752 0.77$\pm$0.00     [rgb]0.5,0.6560322952710496,0.5623529411764706 0.66$\pm$0.00     [rgb]0.7731257208765859,0.9056286043829296,0.7689273356401384 0.06$\pm$0.01     [rgb]0.5,0.6963860053825451,0.5790849673202614 7.3

SAR     [rgb]0.5217993079584775,0.7324106113033448,0.601845444059977 0.47$\pm$0.01     [rgb]0.5,0.6484659746251442,0.5592156862745098 0.49$\pm$0.01     [rgb]0.5,0.6459438677431757,0.5581699346405229 0.77$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.78$\pm$0.00     [rgb]0.5,0.663598615916955,0.5654901960784313 0.65$\pm$0.00     [rgb]0.5,0.6484659746251442,0.5592156862745098 0.49$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 5.2

Mahalanobis Distance $-$ Decoder     [rgb]0.745444059976932,0.8927104959630912,0.7461668589004229 0.02$\pm$0.01     [rgb]0.8662360630526721,0.9467743175701653,0.8531410995770857 $-$0.03$\pm$0.01     [rgb]0.8522106881968474,0.9408688965782391,0.8394848135332564 $-$0.14$\pm$0.01     [rgb]0.8755863129565551,0.9507112648981161,0.8622452902729719 $-$0.22$\pm$0.01     [rgb]0.8896116878123799,0.9566166858900422,0.8759015763168012 0.07$\pm$0.01     [rgb]0.5690888119953864,0.7730411380238369,0.6356632064590542 0.31$\pm$0.02     [rgb]0.9340253748558247,0.9743944636678201,0.9232218377547097 22.7

Relative Mahalanobis Distance $-$ Decoder     [rgb]0.7509803921568627,0.8952941176470588,0.750718954248366 0.01$\pm$0.01     [rgb]0.8755863129565551,0.9507112648981161,0.8622452902729719 $-$0.04$\pm$0.01     [rgb]0.8522106881968474,0.9408688965782391,0.8394848135332564 $-$0.14$\pm$0.01     [rgb]0.8755863129565551,0.9507112648981161,0.8622452902729719 $-$0.22$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.14$\pm$0.01     [rgb]0.5690888119953864,0.7730411380238369,0.6356632064590542 0.31$\pm$0.02     [rgb]0.9499884659746252,0.9807689350249904,0.9403460207612457 23.4

RDE $-$ Decoder     [rgb]0.7399077277970012,0.8901268742791234,0.7416147635524798 0.03$\pm$0.01     [rgb]0.8008073817762399,0.9185467128027682,0.7916878123798539 0.05$\pm$0.01     [rgb]0.7758938869665513,0.9069204152249135,0.7712033833141099 0.02$\pm$0.01     [rgb]0.7731257208765859,0.9056286043829296,0.7689273356401384 0.00$\pm$0.01     [rgb]0.8662360630526721,0.9467743175701653,0.8531410995770857 0.10$\pm$0.01     [rgb]0.5856978085351787,0.7907574009996154,0.6489504036908881 0.28$\pm$0.02     [rgb]0.9026528258362169,0.9618454440599769,0.889757785467128 21.3

HUQ$-$MD $-$ Decoder     [rgb]0.5476355247981546,0.754555940023068,0.6202998846597463 0.41$\pm$0.01     [rgb]0.5,0.6837754709727029,0.5738562091503268 0.45$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.80$\pm$0.01     [rgb]0.5,0.6358554402153018,0.5539869281045752 0.77$\pm$0.00     [rgb]0.5,0.6711649365628605,0.5686274509803921 0.64$\pm$0.00     [rgb]0.5,0.6963860053825451,0.5790849673202614 0.43$\pm$0.01     [rgb]0.5239523260284507,0.7342560553633218,0.6033833141099577 8.8

P(True)     [rgb]0.845198000768935,0.937916186082276,0.8326566705113418 $-$0.17$\pm$0.01     [rgb]0.8971164936562861,0.9596309111880046,0.8838523644752019 $-$0.07$\pm$0.01     [rgb]0.8428604382929643,0.9369319492502883,0.8303806228373702 $-$0.12$\pm$0.01     [rgb]0.8498731257208766,0.9398846597462513,0.8372087658592848 $-$0.16$\pm$0.01     [rgb]0.9395617070357555,0.9766089965397924,0.9291272587466359 $-$0.02$\pm$0.01     [rgb]0.9026528258362169,0.9618454440599769,0.889757785467128 $-$0.11$\pm$0.01     [rgb]0.9710265282583621,0.9889504036908882,0.9648904267589389 25.0

NumSemSets     [rgb]0.6694117647058824,0.8558631295655517,0.7029219530949635 0.14$\pm$0.01     [rgb]0.6244521337946943,0.8320953479430988,0.6799538638985005 0.23$\pm$0.01     [rgb]0.5,0.6989081122645137,0.5801307189542484 0.66$\pm$0.01     [rgb]0.5,0.6509880815071126,0.5602614379084967 0.74$\pm$0.00     [rgb]0.5110342176086121,0.7231833910034602,0.5941560938100731 0.57$\pm$0.01     [rgb]0.6474509803921569,0.8450980392156863,0.692156862745098 0.19$\pm$0.02     [rgb]0.824159938485198,0.9290580545943867,0.8121722414455979 18.5

EigValLaplacian NLI Score entail.     [rgb]0.5261053440984237,0.7361014994232987,0.6049211841599385 0.46$\pm$0.01     [rgb]0.5,0.6585544021530181,0.5633986928104575 0.48$\pm$0.01     [rgb]0.5,0.6736870434448289,0.569673202614379 0.71$\pm$0.01     [rgb]0.5,0.6408996539792388,0.556078431372549 0.76$\pm$0.00     [rgb]0.5,0.7014302191464822,0.5811764705882353 0.60$\pm$0.00     [rgb]0.5540945790080738,0.7600922722029988,0.6249134948096886 0.33$\pm$0.01     [rgb]0.5967704728950404,0.8025682429834678,0.6578085351787774 11.7

EigValLaplacian Jaccard Score     [rgb]0.5562475970780469,0.7619377162629757,0.6264513648596693 0.39$\pm$0.01     [rgb]0.5024221453287198,0.7158016147635524,0.58800461361015 0.41$\pm$0.01     [rgb]0.5,0.6787312572087658,0.571764705882353 0.70$\pm$0.01     [rgb]0.5,0.6610765090349866,0.5644444444444445 0.72$\pm$0.00     [rgb]0.5024221453287198,0.7158016147635524,0.58800461361015 0.58$\pm$0.01     [rgb]0.5,0.7039523260284506,0.5822222222222222 0.42$\pm$0.01     [rgb]0.7537485582468282,0.8965859284890427,0.7529950019223376 16.3

DegMat NLI Score entail.     [rgb]0.5304113802383699,0.7397923875432526,0.6079969242599 0.45$\pm$0.01     [rgb]0.5,0.6484659746251442,0.5592156862745098 0.49$\pm$0.01     [rgb]0.5,0.6661207227989234,0.5665359477124183 0.73$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.78$\pm$0.00     [rgb]0.5,0.6938638985005767,0.5780392156862745 0.61$\pm$0.00     [rgb]0.5067281814686659,0.7194925028835063,0.5910803537101115 0.40$\pm$0.01     [rgb]0.534717416378316,0.7434832756632064,0.6110726643598616 9.1

DegMat Jaccard Score     [rgb]0.5002691272587466,0.7139561707035755,0.5864667435601691 0.52$\pm$0.01     [rgb]0.5,0.6661207227989234,0.5665359477124183 0.47$\pm$0.01     [rgb]0.5,0.6661207227989234,0.5665359477124183 0.73$\pm$0.01     [rgb]0.5,0.6509880815071126,0.5602614379084967 0.74$\pm$0.00     [rgb]0.5,0.6862975778546713,0.5749019607843138 0.62$\pm$0.01     [rgb]0.5,0.6888196847366397,0.5759477124183007 0.44$\pm$0.01     [rgb]0.5476355247981546,0.754555940023068,0.6202998846597463 9.6

Eccentricity NLI Score entail.     [rgb]0.5045751633986928,0.7176470588235294,0.5895424836601307 0.51$\pm$0.01     [rgb]0.5,0.6661207227989234,0.5665359477124183 0.47$\pm$0.01     [rgb]0.5,0.6661207227989234,0.5665359477124183 0.73$\pm$0.01     [rgb]0.5,0.6459438677431757,0.5581699346405229 0.75$\pm$0.00     [rgb]0.5110342176086121,0.7231833910034602,0.5941560938100731 0.57$\pm$0.01     [rgb]0.5,0.6812533640907343,0.5728104575163399 0.45$\pm$0.01     [rgb]0.5764705882352941,0.7809150326797385,0.6415686274509804 10.8

Eccentricity Jaccard Score     [rgb]0.5,0.6989081122645137,0.5801307189542484 0.55$\pm$0.01     [rgb]0.5,0.6837754709727029,0.5738562091503268 0.45$\pm$0.01     [rgb]0.5,0.6736870434448289,0.569673202614379 0.71$\pm$0.01     [rgb]0.5,0.663598615916955,0.5654901960784313 0.71$\pm$0.00     [rgb]0.5110342176086121,0.7231833910034602,0.5941560938100731 0.57$\pm$0.00     [rgb]0.5067281814686659,0.7194925028835063,0.5910803537101115 0.40$\pm$0.02     [rgb]0.6568627450980392,0.8497116493656286,0.6967704728950403 13.8

Lexical Similarity Rouge$-$L     [rgb]0.5045751633986928,0.7176470588235294,0.5895424836601307 0.51$\pm$0.01     [rgb]0.5,0.6484659746251442,0.5592156862745098 0.49$\pm$0.01     [rgb]0.5,0.6661207227989234,0.5665359477124183 0.73$\pm$0.01     [rgb]0.5,0.6509880815071126,0.5602614379084967 0.74$\pm$0.00     [rgb]0.5,0.6862975778546713,0.5749019607843138 0.62$\pm$0.00     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.51$\pm$0.01     [rgb]0.5024221453287198,0.7158016147635524,0.58800461361015 7.9

Lexical Similarity BLEU     [rgb]0.5,0.7039523260284506,0.5822222222222222 0.54$\pm$0.01     [rgb]0.5,0.6736870434448289,0.569673202614379 0.46$\pm$0.01     [rgb]0.5,0.6736870434448289,0.569673202614379 0.71$\pm$0.01     [rgb]0.5,0.6610765090349866,0.5644444444444445 0.72$\pm$0.00     [rgb]0.5,0.6862975778546713,0.5749019607843138 0.62$\pm$0.00     [rgb]0.5,0.6812533640907343,0.5728104575163399 0.45$\pm$0.01     [rgb]0.580161476355248,0.7848519800076893,0.6445213379469434 10.9

PRR$\uparrow$ with various generation quality metrics for UQ methods in selective classification tasks with the Mistral 7b v0.2 model. Darker color indicates better results.

2*UQ Method     2c|XSUM     2c|WMT14 Fr-En     2c|WMT19 De-En    2*Mean Rank    2*Mean Rank (w/o XSUM)$\downarrow$
Rouge-L     AlignScore    Comet     AlignScore    Comet     AlignScore
Maximum Sequence Probability     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.48$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.15$\pm$0.00     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.45$\pm$0.01     [rgb]0.5196462898885044,0.730565167243368,0.6003075740099961 0.23$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.59$\pm$0.00     [rgb]0.6756862745098039,0.8589388696655132,0.705997693194925 0.18$\pm$0.01     [rgb]0.6096885813148789,0.8163475586312956,0.6681430219146482 12.4     [rgb]0.5,0.663598615916955,0.5654901960784313 4.6

Perplexity     [rgb]0.8171472510572857,0.9261053440984237,0.8053440984236832 $-$0.20$\pm$0.01     [rgb]0.8194848135332564,0.9270895809304114,0.8076201460976548 $-$0.05$\pm$0.00     [rgb]0.5,0.711518646674356,0.585359477124183 0.36$\pm$0.01     [rgb]0.7620530565167243,0.9004613610149943,0.7598231449442522 0.11$\pm$0.01     [rgb]0.5648596693579393,0.7693194925028835,0.6326028450595924 0.38$\pm$0.00     [rgb]0.7399077277970012,0.8901268742791234,0.7416147635524798 0.13$\pm$0.01     [rgb]0.9044982698961938,0.9625836216839677,0.8917262591311035 20.7     [rgb]0.8405228758169935,0.9359477124183007,0.8281045751633986 19.3

Mean Token Entropy     [rgb]0.8568858131487889,0.9428373702422146,0.8440369088811995 $-$0.25$\pm$0.01     [rgb]0.6819607843137254,0.8620146097654748,0.7090734332948866 0.00$\pm$0.00     [rgb]0.5,0.6938638985005767,0.5780392156862745 0.38$\pm$0.01     [rgb]0.7133333333333334,0.8773933102652827,0.7244521337946944 0.13$\pm$0.01     [rgb]0.5454825067281814,0.752710495963091,0.6187620146097654 0.41$\pm$0.00     [rgb]0.6756862745098039,0.8589388696655132,0.705997693194925 0.18$\pm$0.01     [rgb]0.7288350634371396,0.884959630911188,0.7325105728565936 15.5     [rgb]0.6537254901960784,0.8481737793156479,0.6952326028450596 13.1

Monte Carlo Sequence Entropy     [rgb]0.5627066512879662,0.7674740484429066,0.6310649750096117 0.09$\pm$0.01     [rgb]0.7133333333333334,0.8773933102652827,0.7244521337946944 $-$0.01$\pm$0.00     [rgb]0.5,0.6509880815071126,0.5602614379084967 0.43$\pm$0.01     [rgb]0.5540945790080738,0.7600922722029988,0.6249134948096886 0.21$\pm$0.01     [rgb]0.5,0.6535101883890811,0.5613071895424837 0.56$\pm$0.00     [rgb]0.6244521337946943,0.8320953479430988,0.6799538638985005 0.22$\pm$0.01     [rgb]0.5,0.6837754709727029,0.5738562091503268 7.6     [rgb]0.5,0.6938638985005767,0.5780392156862745 5.8

Monte Carlo Normalized Sequence Entropy     [rgb]0.582006920415225,0.7868204536716648,0.645997693194925 0.06$\pm$0.01     [rgb]0.607843137254902,0.8143790849673203,0.6666666666666666 0.03$\pm$0.00     [rgb]0.5,0.6862975778546713,0.5749019607843138 0.39$\pm$0.01     [rgb]0.8063437139561707,0.9211303344867359,0.796239907727797 0.09$\pm$0.01     [rgb]0.5261053440984237,0.7361014994232987,0.6049211841599385 0.44$\pm$0.00     [rgb]0.7509803921568627,0.8952941176470588,0.750718954248366 0.12$\pm$0.01     [rgb]0.731603229527105,0.8862514417531719,0.7347866205305652 15.6     [rgb]0.786966551326413,0.9120876585928489,0.7803075740099962 17.3

Pointwise Mutual Information     [rgb]0.5411764705882353,0.7490196078431373,0.615686274509804 0.12$\pm$0.01     [rgb]0.5,0.7039523260284506,0.5822222222222222 0.09$\pm$0.00     [rgb]0.8008073817762399,0.9185467128027682,0.7916878123798539 0.01$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.03$\pm$0.01     [rgb]0.845198000768935,0.937916186082276,0.8326566705113418 0.01$\pm$0.00     [rgb]0.9174163783160323,0.9677508650519031,0.9055055747789311 $-$0.05$\pm$0.01     [rgb]0.8288350634371395,0.9310265282583622,0.816724336793541 18.2     [rgb]0.9654901960784315,0.9867973856209151,0.9584313725490197 25.5

Conditional Pointwise Mutual Information     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.27$\pm$0.01     [rgb]0.5727797001153403,0.7769780853517878,0.6386159169550173 0.05$\pm$0.00     [rgb]0.92479815455594,0.9707035755478662,0.9133794694348327 $-$0.15$\pm$0.01     [rgb]0.9754555940023069,0.9906728181468666,0.9700576701268743 $-$0.02$\pm$0.01     [rgb]0.9588465974625144,0.9842137639369474,0.9506805074971165 $-$0.19$\pm$0.01     [rgb]0.9798846597462514,0.9923952326028451,0.9752249134948097 $-$0.16$\pm$0.01     [rgb]0.8732487504805844,0.9497270280661284,0.8599692425990004 19.6     [rgb]0.9743483275663207,0.990242214532872,0.9687658592848905 26.3

RenyiNeg     [rgb]0.5893886966551327,0.7946943483275664,0.6519031141868512 0.05$\pm$0.01     [rgb]0.8638985005767013,0.9457900807381776,0.8508650519031142 $-$0.07$\pm$0.00     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.28$\pm$0.01     [rgb]0.9665974625144176,0.9872279892349096,0.9597231833910035 $-$0.01$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.27$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.17$\pm$0.01     [rgb]0.9798846597462514,0.9923952326028451,0.9752249134948097 24.5     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 27.1

FisherRao     [rgb]0.6004613610149943,0.8065051903114187,0.6607612456747405 0.03$\pm$0.01     [rgb]0.8638985005767013,0.9457900807381776,0.8508650519031142 $-$0.07$\pm$0.00     [rgb]0.976562860438293,0.9911034217608612,0.9713494809688581 $-$0.26$\pm$0.01     [rgb]0.9665974625144176,0.9872279892349096,0.9597231833910035 $-$0.01$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.27$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.17$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 24.8     [rgb]0.9820991926182238,0.9932564398308343,0.9778085351787774 26.9

TokenSAR     [rgb]0.8638985005767013,0.9457900807381776,0.8508650519031142 $-$0.26$\pm$0.01     [rgb]0.7133333333333334,0.8773933102652827,0.7244521337946944 $-$0.01$\pm$0.00     [rgb]0.5,0.7039523260284506,0.5822222222222222 0.37$\pm$0.01     [rgb]0.7399077277970012,0.8901268742791234,0.7416147635524798 0.12$\pm$0.01     [rgb]0.5454825067281814,0.752710495963091,0.6187620146097654 0.41$\pm$0.00     [rgb]0.7133333333333334,0.8773933102652827,0.7244521337946944 0.15$\pm$0.01     [rgb]0.8335101883890812,0.9329950019223375,0.821276432141484 18.3     [rgb]0.7620530565167243,0.9004613610149943,0.7598231449442522 16.5

CCP     [rgb]0.7509803921568627,0.8952941176470588,0.750718954248366 $-$0.13$\pm$0.01     [rgb]0.8405228758169935,0.9359477124183007,0.8281045751633986 $-$0.06$\pm$0.00     [rgb]0.5,0.6862975778546713,0.5749019607843138 0.39$\pm$0.01     [rgb]0.6170703575547867,0.8242214532871972,0.6740484429065744 0.17$\pm$0.01     [rgb]0.5519415609381008,0.7582468281430219,0.6233756247597078 0.40$\pm$0.00     [rgb]0.7007843137254902,0.8712418300653595,0.7183006535947712 0.16$\pm$0.01     [rgb]0.7648212226066897,0.9017531718569781,0.7620991926182238 16.4     [rgb]0.6474509803921569,0.8450980392156863,0.692156862745098 13.0

Semantic Entropy     [rgb]0.5627066512879662,0.7674740484429066,0.6310649750096117 0.09$\pm$0.01     [rgb]0.7133333333333334,0.8773933102652827,0.7244521337946944 $-$0.01$\pm$0.00     [rgb]0.5,0.6408996539792388,0.556078431372549 0.44$\pm$0.01     [rgb]0.5024221453287198,0.7158016147635524,0.58800461361015 0.24$\pm$0.01     [rgb]0.5,0.6383775470972703,0.5550326797385621 0.58$\pm$0.00     [rgb]0.580161476355248,0.7848519800076893,0.6445213379469434 0.28$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 6.1     [rgb]0.5,0.6333333333333333,0.5529411764705883 3.5

SentenceSAR     [rgb]0.6474509803921569,0.8450980392156863,0.692156862745098 $-$0.03$\pm$0.01     [rgb]0.7925028835063437,0.9146712802768167,0.7848596693579393 $-$0.04$\pm$0.00     [rgb]0.5,0.6408996539792388,0.556078431372549 0.44$\pm$0.01     [rgb]0.5196462898885044,0.730565167243368,0.6003075740099961 0.23$\pm$0.01     [rgb]0.5,0.6383775470972703,0.5550326797385621 0.58$\pm$0.00     [rgb]0.6380392156862745,0.8404844290657439,0.6875432525951557 0.21$\pm$0.01     [rgb]0.5411764705882353,0.7490196078431373,0.615686274509804 9.8     [rgb]0.5,0.668642829680892,0.5675816993464052 4.9

SAR     [rgb]0.5764705882352941,0.7809150326797385,0.6415686274509804 0.07$\pm$0.01     [rgb]0.5325643983083429,0.7416378316032295,0.6095347943098808 0.07$\pm$0.00     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.45$\pm$0.01     [rgb]0.6380392156862745,0.8404844290657439,0.6875432525951557 0.16$\pm$0.01     [rgb]0.5,0.6913417916186082,0.5769934640522876 0.51$\pm$0.00     [rgb]0.6244521337946943,0.8320953479430988,0.6799538638985005 0.22$\pm$0.01     [rgb]0.5,0.663598615916955,0.5654901960784313 7.0     [rgb]0.5024221453287198,0.7158016147635524,0.58800461361015 6.6

Mahalanobis Distance $-$ Decoder     [rgb]0.6788235294117647,0.860476739715494,0.7075355632449059 $-$0.06$\pm$0.01     [rgb]0.8849365628604383,0.9546482122260669,0.8713494809688582 $-$0.08$\pm$0.00     [rgb]0.7620530565167243,0.9004613610149943,0.7598231449442522 0.05$\pm$0.01     [rgb]0.6023068050749711,0.808473663975394,0.662237600922722 0.18$\pm$0.01     [rgb]0.7758938869665513,0.9069204152249135,0.7712033833141099 0.10$\pm$0.01     [rgb]0.7007843137254902,0.8712418300653595,0.7183006535947712 0.16$\pm$0.01     [rgb]0.8709111880046136,0.9487427912341406,0.8576931949250288 19.5     [rgb]0.7814302191464821,0.9095040369088812,0.7757554786620531 17.1

Relative Mahalanobis Distance $-$ Decoder     [rgb]0.5949250288350634,0.8005997693194925,0.6563321799307958 0.04$\pm$0.01     [rgb]0.8405228758169935,0.9359477124183007,0.8281045751633986 $-$0.06$\pm$0.00     [rgb]0.7426758938869666,0.8914186851211072,0.7438908112264514 0.07$\pm$0.01     [rgb]0.6023068050749711,0.808473663975394,0.662237600922722 0.18$\pm$0.01     [rgb]0.7841983852364476,0.9107958477508651,0.7780315263360246 0.09$\pm$0.00     [rgb]0.7007843137254902,0.8712418300653595,0.7183006535947712 0.16$\pm$0.01     [rgb]0.8171472510572857,0.9261053440984237,0.8053440984236832 17.8     [rgb]0.786966551326413,0.9120876585928489,0.7803075740099962 17.3

RDE $-$ Decoder     [rgb]0.6568627450980392,0.8497116493656286,0.6967704728950403 $-$0.04$\pm$0.01     [rgb]0.5,0.6560322952710496,0.5623529411764706 0.11$\pm$0.00     [rgb]0.731603229527105,0.8862514417531719,0.7347866205305652 0.08$\pm$0.01     [rgb]0.6380392156862745,0.8404844290657439,0.6875432525951557 0.16$\pm$0.01     [rgb]0.7758938869665513,0.9069204152249135,0.7712033833141099 0.10$\pm$0.01     [rgb]0.6244521337946943,0.8320953479430988,0.6799538638985005 0.22$\pm$0.01     [rgb]0.6631372549019608,0.8527873894655902,0.6998462129950019 13.9     [rgb]0.7258823529411764,0.8835447904652057,0.7306036139946175 15.3

HUQ$-$MD $-$ Decoder     [rgb]0.8091118800461361,0.9224221453287198,0.7985159554017686 $-$0.19$\pm$0.01     [rgb]0.7925028835063437,0.9146712802768167,0.7848596693579393 $-$0.04$\pm$0.00     [rgb]0.5067281814686659,0.7194925028835063,0.5910803537101115 0.35$\pm$0.01     [rgb]0.5727797001153403,0.7769780853517878,0.6386159169550173 0.20$\pm$0.01     [rgb]0.5584006151480201,0.7637831603229528,0.6279892349096502 0.39$\pm$0.00     [rgb]0.7399077277970012,0.8901268742791234,0.7416147635524798 0.13$\pm$0.01     [rgb]0.8335101883890812,0.9329950019223375,0.821276432141484 18.3     [rgb]0.759284890426759,0.8991695501730104,0.7575470972702807 16.4

P(True)     [rgb]0.6788235294117647,0.860476739715494,0.7075355632449059 $-$0.06$\pm$0.01     [rgb]0.7675893886966552,0.9030449826989619,0.7643752402921953 $-$0.03$\pm$0.00     [rgb]0.605997693194925,0.8124106113033449,0.6651903114186851 0.21$\pm$0.01     [rgb]0.6631372549019608,0.8527873894655902,0.6998462129950019 0.15$\pm$0.01     [rgb]0.6725490196078432,0.8574009996155325,0.7044598231449443 0.22$\pm$0.00     [rgb]0.7007843137254902,0.8712418300653595,0.7183006535947712 0.16$\pm$0.01     [rgb]0.8288350634371395,0.9310265282583622,0.816724336793541 18.2     [rgb]0.7814302191464821,0.9095040369088812,0.7757554786620531 17.1

NumSemSets     [rgb]0.607843137254902,0.8143790849673203,0.6666666666666666 0.02$\pm$0.01     [rgb]0.5727797001153403,0.7769780853517878,0.6386159169550173 0.05$\pm$0.00     [rgb]0.8194848135332564,0.9270895809304114,0.8076201460976548 $-$0.01$\pm$0.01     [rgb]0.8063437139561707,0.9211303344867359,0.796239907727797 0.09$\pm$0.01     [rgb]0.8381853133410226,0.934963475586313,0.8258285274894271 0.02$\pm$0.00     [rgb]0.7509803921568627,0.8952941176470588,0.750718954248366 0.12$\pm$0.01     [rgb]0.9008073817762399,0.9611072664359861,0.8877893118031526 20.6     [rgb]0.9414071510957325,0.9773471741637831,0.9310957324106113 23.8

EigValLaplacian NLI Score entail.     [rgb]0.5764705882352941,0.7809150326797385,0.6415686274509804 0.07$\pm$0.01     [rgb]0.5519415609381008,0.7582468281430219,0.6233756247597078 0.06$\pm$0.00     [rgb]0.5,0.7039523260284506,0.5822222222222222 0.37$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.28$\pm$0.01     [rgb]0.5,0.7064744329104191,0.5832679738562092 0.49$\pm$0.00     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.45$\pm$0.01     [rgb]0.5,0.6837754709727029,0.5738562091503268 7.6     [rgb]0.508881199538639,0.7213379469434833,0.5926182237600923 6.9

EigValLaplacian Jaccard Score     [rgb]0.5627066512879662,0.7674740484429066,0.6310649750096117 0.09$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.12$\pm$0.00     [rgb]0.5,0.668642829680892,0.5675816993464052 0.41$\pm$0.01     [rgb]0.7620530565167243,0.9004613610149943,0.7598231449442522 0.11$\pm$0.01     [rgb]0.5,0.7064744329104191,0.5832679738562092 0.49$\pm$0.00     [rgb]0.7007843137254902,0.8712418300653595,0.7183006535947712 0.16$\pm$0.01     [rgb]0.5368704344482891,0.7453287197231834,0.6126105344098424 9.7     [rgb]0.644313725490196,0.8435601691657055,0.6906189926951173 12.9

DegMat NLI Score entail.     [rgb]0.5764705882352941,0.7809150326797385,0.6415686274509804 0.07$\pm$0.01     [rgb]0.5519415609381008,0.7582468281430219,0.6233756247597078 0.06$\pm$0.00     [rgb]0.5,0.7039523260284506,0.5822222222222222 0.37$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.28$\pm$0.01     [rgb]0.5,0.7064744329104191,0.5832679738562092 0.49$\pm$0.00     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.45$\pm$0.01     [rgb]0.5,0.6837754709727029,0.5738562091503268 7.6     [rgb]0.508881199538639,0.7213379469434833,0.5926182237600923 6.9

DegMat Jaccard Score     [rgb]0.5627066512879662,0.7674740484429066,0.6310649750096117 0.09$\pm$0.01     [rgb]0.5,0.6560322952710496,0.5623529411764706 0.11$\pm$0.00     [rgb]0.5,0.6509880815071126,0.5602614379084967 0.43$\pm$0.01     [rgb]0.7399077277970012,0.8901268742791234,0.7416147635524798 0.12$\pm$0.01     [rgb]0.5,0.6913417916186082,0.5769934640522876 0.51$\pm$0.00     [rgb]0.7007843137254902,0.8712418300653595,0.7183006535947712 0.16$\pm$0.01     [rgb]0.5067281814686659,0.7194925028835063,0.5910803537101115 8.7     [rgb]0.6004613610149943,0.8065051903114187,0.6607612456747405 11.0

Eccentricity NLI Score entail.     [rgb]0.582006920415225,0.7868204536716648,0.645997693194925 0.06$\pm$0.01     [rgb]0.5519415609381008,0.7582468281430219,0.6233756247597078 0.06$\pm$0.00     [rgb]0.5,0.7039523260284506,0.5822222222222222 0.37$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.28$\pm$0.01     [rgb]0.5,0.7064744329104191,0.5832679738562092 0.49$\pm$0.00     [rgb]0.5,0.6434217608612072,0.557124183006536 0.44$\pm$0.01     [rgb]0.5,0.7064744329104191,0.5832679738562092 8.3     [rgb]0.5174932718185313,0.728719723183391,0.5987697039600154 7.3

Eccentricity Jaccard Score     [rgb]0.6262975778546713,0.8340638216070742,0.681430219146482 $-$0.01$\pm$0.01     [rgb]0.6819607843137254,0.8620146097654748,0.7090734332948866 $-$0.00$\pm$0.00     [rgb]0.5217993079584775,0.7324106113033448,0.601845444059977 0.3$\pm$0.01     [rgb]0.8662360630526721,0.9467743175701653,0.8531410995770857 0.06$\pm$0.01     [rgb]0.5196462898885044,0.730565167243368,0.6003075740099961 0.45$\pm$0.00     [rgb]0.7133333333333334,0.8773933102652827,0.7244521337946944 0.15$\pm$0.01     [rgb]0.8146482122260669,0.9250057670126874,0.8030680507497117 17.8     [rgb]0.8218223760092271,0.9280738177623991,0.8098961937716262 18.5

Lexical Similarity Rouge$-$L     [rgb]0.5562475970780469,0.7619377162629757,0.6264513648596693 0.10$\pm$0.01     [rgb]0.5,0.6787312572087658,0.571764705882353 0.10$\pm$0.00     [rgb]0.5,0.6509880815071126,0.5602614379084967 0.43$\pm$0.01     [rgb]0.7399077277970012,0.8901268742791234,0.7416147635524798 0.12$\pm$0.01     [rgb]0.5,0.6837754709727029,0.5738562091503268 0.52$\pm$0.00     [rgb]0.6505882352941177,0.846635909265667,0.6936947327950789 0.20$\pm$0.01     [rgb]0.5,0.6762091503267974,0.570718954248366 7.3     [rgb]0.5648596693579393,0.7693194925028835,0.6326028450595924 9.3

Lexical Similarity BLEU     [rgb]0.5949250288350634,0.8005997693194925,0.6563321799307958 0.04$\pm$0.01     [rgb]0.5325643983083429,0.7416378316032295,0.6095347943098808 0.07$\pm$0.00     [rgb]0.5,0.6938638985005767,0.5780392156862745 0.38$\pm$0.01     [rgb]0.7841983852364476,0.9107958477508651,0.7780315263360246 0.10$\pm$0.01     [rgb]0.5261053440984237,0.7361014994232987,0.6049211841599385 0.44$\pm$0.00     [rgb]0.7133333333333334,0.8773933102652827,0.7244521337946944 0.15$\pm$0.01     [rgb]0.6788235294117647,0.860476739715494,0.7075355632449059 14.3     [rgb]0.7565167243367935,0.8978777393310265,0.7552710495963091 16.3

PRR$\uparrow$ with various generation quality metrics for UQ methods in selective generation tasks with the StableLM 12b v2 model. Darker color indicates better results.
2*UQ Method     2c|XSUM     2c|WMT14 Fr-En     2c|WMT19 De-En    2*Mean Rank    2*Mean Rank (w/o XSUM)
Rouge-L     AlignScore    Comet     AlignScore    Comet     AlignScore
Maximum Sequence Probability     [rgb]0.7101960784313726,0.8758554402153018,0.7229142637447136 $-$0.14$\pm$0.00     [rgb]0.9340253748558247,0.9743944636678201,0.9232218377547097 $-$0.08$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.42$\pm$0.02     [rgb]0.5584006151480201,0.7637831603229528,0.6279892349096502 0.23$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.57$\pm$0.01     [rgb]0.6244521337946943,0.8320953479430988,0.6799538638985005 0.25$\pm$0.01     [rgb]0.5893886966551327,0.7946943483275664,0.6519031141868512 10.2     [rgb]0.5,0.6459438677431757,0.5581699346405229 3.8

Perplexity     [rgb]0.8358477508650519,0.9339792387543253,0.8235524798154557 $-$0.26$\pm$0.00     [rgb]0.7426758938869666,0.8914186851211072,0.7438908112264514 $-$0.00$\pm$0.01     [rgb]0.5,0.6408996539792388,0.556078431372549 0.41$\pm$0.02     [rgb]0.5949250288350634,0.8005997693194925,0.6563321799307958 0.20$\pm$0.01     [rgb]0.5,0.6787312572087658,0.571764705882353 0.51$\pm$0.01     [rgb]0.6505882352941177,0.846635909265667,0.6936947327950789 0.23$\pm$0.01     [rgb]0.6505882352941177,0.846635909265667,0.6936947327950789 12.8     [rgb]0.5433294886582084,0.7508650519031141,0.6172241445597847 8.3

Mean Token Entropy     [rgb]0.7537485582468282,0.8965859284890427,0.7529950019223376 $-$0.18$\pm$0.00     [rgb]0.8498731257208766,0.9398846597462513,0.8372087658592848 $-$0.04$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.42$\pm$0.02     [rgb]0.5838523644752018,0.7887889273356401,0.6474740484429066 0.21$\pm$0.01     [rgb]0.5,0.6862975778546713,0.5749019607843138 0.50$\pm$0.01     [rgb]0.6505882352941177,0.846635909265667,0.6936947327950789 0.23$\pm$0.01     [rgb]0.6286274509803922,0.8358708189158016,0.6829296424452134 12.1     [rgb]0.5282583621683967,0.7379469434832756,0.6064590542099193 7.6

Monte Carlo Sequence Entropy     [rgb]0.5,0.6888196847366397,0.5759477124183007 0.12$\pm$0.01     [rgb]0.6788235294117647,0.860476739715494,0.7075355632449059 0.02$\pm$0.01     [rgb]0.5,0.6938638985005767,0.5780392156862745 0.36$\pm$0.02     [rgb]0.5838523644752018,0.7887889273356401,0.6474740484429066 0.21$\pm$0.01     [rgb]0.5,0.6787312572087658,0.571764705882353 0.51$\pm$0.01     [rgb]0.6349019607843137,0.8389465590157632,0.686005382545175 0.24$\pm$0.01     [rgb]0.5690888119953864,0.7730411380238369,0.6356632064590542 9.2     [rgb]0.5540945790080738,0.7600922722029988,0.6249134948096886 8.8

Monte Carlo Normalized Sequence Entropy     [rgb]0.5,0.6888196847366397,0.5759477124183007 0.12$\pm$0.00     [rgb]0.6207612456747404,0.828158400615148,0.6770011534025375 0.04$\pm$0.01     [rgb]0.5,0.6610765090349866,0.5644444444444445 0.39$\pm$0.02     [rgb]0.6568627450980392,0.8497116493656286,0.6967704728950403 0.16$\pm$0.01     [rgb]0.5,0.6862975778546713,0.5749019607843138 0.50$\pm$0.01     [rgb]0.6631372549019608,0.8527873894655902,0.6998462129950019 0.22$\pm$0.01     [rgb]0.5986159169550173,0.8045367166474433,0.659284890426759 10.7     [rgb]0.6170703575547867,0.8242214532871972,0.6740484429065744 11.9

Pointwise Mutual Information     [rgb]0.5912341407151096,0.7966628219915417,0.6533794694348327 $-$0.02$\pm$0.00     [rgb]0.6788235294117647,0.860476739715494,0.7075355632449059 0.02$\pm$0.01     [rgb]0.7703575547866206,0.9043367935409459,0.7666512879661669 0.06$\pm$0.02     [rgb]0.9358708189158016,0.9751326412918109,0.9251903114186851 $-$0.01$\pm$0.01     [rgb]0.786966551326413,0.9120876585928489,0.7803075740099962 0.08$\pm$0.01     [rgb]0.8934256055363322,0.9581545559400231,0.8799154171472511 0.03$\pm$0.01     [rgb]0.8568858131487889,0.9428373702422146,0.8440369088811995 19.7     [rgb]0.9081891580161476,0.9640599769319492,0.8956632064590542 22.5

Conditional Pointwise Mutual Information     [rgb]0.5,0.6610765090349866,0.5644444444444445 0.15$\pm$0.01     [rgb]0.7426758938869666,0.8914186851211072,0.7438908112264514 $-$0.00$\pm$0.01     [rgb]0.9192618223760092,0.9684890426758939,0.9074740484429066 $-$0.10$\pm$0.02     [rgb]0.9555247981545559,0.9829219530949636,0.946805074971165 $-$0.03$\pm$0.01     [rgb]0.8896116878123799,0.9566166858900422,0.8759015763168012 $-$0.06$\pm$0.01     [rgb]0.8934256055363322,0.9581545559400231,0.8799154171472511 0.03$\pm$0.01     [rgb]0.8709111880046136,0.9487427912341406,0.8576931949250288 20.2     [rgb]0.9544175317185698,0.9824913494809688,0.945513264129181 25.1

RenyiNeg     [rgb]0.7758938869665513,0.9069204152249135,0.7712033833141099 $-$0.20$\pm$0.00     [rgb]0.9621683967704728,0.9855055747789312,0.9545559400230681 $-$0.10$\pm$0.01     [rgb]0.9798846597462514,0.9923952326028451,0.9752249134948097 $-$0.21$\pm$0.03     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.07$\pm$0.01     [rgb]0.9809919261822376,0.9928258362168396,0.9765167243367936 $-$0.26$\pm$0.01     [rgb]0.9754555940023069,0.9906728181468666,0.9700576701268743 $-$0.09$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 26.9     [rgb]0.9787773933102653,0.9919646289888504,0.9739331026528258 27.3

FisherRao     [rgb]0.7537485582468282,0.8965859284890427,0.7529950019223376 $-$0.18$\pm$0.00     [rgb]0.9510957324106113,0.981199538638985,0.9416378316032294 $-$0.09$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.22$\pm$0.03     [rgb]0.9776701268742791,0.9915340253748559,0.972641291810842 $-$0.06$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.27$\pm$0.01     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.11$\pm$0.01     [rgb]0.9809919261822376,0.9928258362168396,0.9765167243367936 26.6     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 27.8

TokenSAR     [rgb]0.7648212226066897,0.9017531718569781,0.7620991926182238 $-$0.19$\pm$0.00     [rgb]0.8732487504805844,0.9497270280661284,0.8599692425990004 $-$0.05$\pm$0.01     [rgb]0.5,0.6408996539792388,0.556078431372549 0.41$\pm$0.02     [rgb]0.5949250288350634,0.8005997693194925,0.6563321799307958 0.20$\pm$0.01     [rgb]0.5,0.6862975778546713,0.5749019607843138 0.50$\pm$0.01     [rgb]0.6349019607843137,0.8389465590157632,0.686005382545175 0.24$\pm$0.01     [rgb]0.6631372549019608,0.8527873894655902,0.6998462129950019 13.2     [rgb]0.5454825067281814,0.752710495963091,0.6187620146097654 8.4

CCP     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.49$\pm$0.00     [rgb]0.9843137254901961,0.9941176470588236,0.9803921568627452 $-$0.12$\pm$0.01     [rgb]0.5,0.6837754709727029,0.5738562091503268 0.37$\pm$0.02     [rgb]0.607843137254902,0.8143790849673203,0.6666666666666666 0.19$\pm$0.01     [rgb]0.5153402537485583,0.7268742791234141,0.5972318339100346 0.44$\pm$0.01     [rgb]0.6788235294117647,0.860476739715494,0.7075355632449059 0.21$\pm$0.01     [rgb]0.8288350634371395,0.9310265282583622,0.816724336793541 18.5     [rgb]0.6694117647058824,0.8558631295655517,0.7029219530949635 13.8

Semantic Entropy     [rgb]0.5,0.6888196847366397,0.5759477124183007 0.12$\pm$0.00     [rgb]0.6788235294117647,0.860476739715494,0.7075355632449059 0.02$\pm$0.01     [rgb]0.5,0.6711649365628605,0.5686274509803921 0.38$\pm$0.02     [rgb]0.5433294886582084,0.7508650519031141,0.6172241445597847 0.24$\pm$0.01     [rgb]0.5,0.6560322952710496,0.5623529411764706 0.54$\pm$0.01     [rgb]0.582006920415225,0.7868204536716648,0.645997693194925 0.30$\pm$0.01     [rgb]0.5110342176086121,0.7231833910034602,0.5941560938100731 6.7     [rgb]0.5,0.6787312572087658,0.571764705882353 5.0

SentenceSAR     [rgb]0.6976470588235294,0.8697039600153786,0.7167627835447905 $-$0.13$\pm$0.01     [rgb]0.7703575547866206,0.9043367935409459,0.7666512879661669 $-$0.01$\pm$0.01     [rgb]0.5,0.6837754709727029,0.5738562091503268 0.37$\pm$0.02     [rgb]0.5838523644752018,0.7887889273356401,0.6474740484429066 0.21$\pm$0.01     [rgb]0.5,0.6711649365628605,0.5686274509803921 0.52$\pm$0.01     [rgb]0.6631372549019608,0.8527873894655902,0.6998462129950019 0.22$\pm$0.01     [rgb]0.6349019607843137,0.8389465590157632,0.686005382545175 12.3     [rgb]0.5605536332179931,0.7656286043829297,0.6295271049596309 9.0

SAR     [rgb]0.5,0.6408996539792388,0.556078431372549 0.17$\pm$0.00     [rgb]0.5174932718185313,0.728719723183391,0.5987697039600154 0.09$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 0.42$\pm$0.02     [rgb]0.5433294886582084,0.7508650519031141,0.6172241445597847 0.24$\pm$0.01     [rgb]0.5,0.6484659746251442,0.5592156862745098 0.55$\pm$0.01     [rgb]0.582006920415225,0.7868204536716648,0.645997693194925 0.30$\pm$0.01     [rgb]0.5,0.6333333333333333,0.5529411764705883 3.2     [rgb]0.5,0.6333333333333333,0.5529411764705883 3.3

Mahalanobis Distance $-$ Decoder     [rgb]0.5540945790080738,0.7600922722029988,0.6249134948096886 0.03$\pm$0.01     [rgb]0.8952710495963091,0.9588927335640138,0.8818838908112265 $-$0.06$\pm$0.00     [rgb]0.7925028835063437,0.9146712802768167,0.7848596693579393 0.04$\pm$0.02     [rgb]0.6976470588235294,0.8697039600153786,0.7167627835447905 0.14$\pm$0.01     [rgb]0.786966551326413,0.9120876585928489,0.7803075740099962 0.08$\pm$0.01     [rgb]0.7196078431372549,0.8804690503652441,0.7275278738946559 0.18$\pm$0.01     [rgb]0.8522106881968474,0.9408688965782391,0.8394848135332564 19.4     [rgb]0.8592233756247597,0.9438216070742023,0.846312956555171 20.4

Relative Mahalanobis Distance $-$ Decoder     [rgb]0.5912341407151096,0.7966628219915417,0.6533794694348327 $-$0.02$\pm$0.01     [rgb]0.6788235294117647,0.860476739715494,0.7075355632449059 0.02$\pm$0.01     [rgb]0.8146482122260669,0.9250057670126874,0.8030680507497117 0.02$\pm$0.02     [rgb]0.7371395617070358,0.8888350634371396,0.7393387158785083 0.12$\pm$0.01     [rgb]0.8685736255286428,0.947758554402153,0.8554171472510572 $-$0.03$\pm$0.01     [rgb]0.8194848135332564,0.9270895809304114,0.8076201460976548 0.10$\pm$0.01     [rgb]0.8615609381007305,0.9448058439061899,0.8485890042291426 19.8     [rgb]0.9118800461361015,0.9655363321799308,0.899600153787005 22.8

RDE $-$ Decoder     [rgb]0.6599999999999999,0.8512495194156093,0.6983083429450212 $-$0.10$\pm$0.00     [rgb]0.5411764705882353,0.7490196078431373,0.615686274509804 0.08$\pm$0.01     [rgb]0.9044982698961938,0.9625836216839677,0.8917262591311035 $-$0.08$\pm$0.02     [rgb]0.7897347174163782,0.9133794694348327,0.7825836216839677 0.09$\pm$0.01     [rgb]0.9358708189158016,0.9751326412918109,0.9251903114186851 $-$0.14$\pm$0.01     [rgb]0.9654901960784315,0.9867973856209151,0.9584313725490197 $-$0.07$\pm$0.01     [rgb]0.8732487504805844,0.9497270280661284,0.8599692425990004 20.3     [rgb]0.9499884659746252,0.9807689350249904,0.9403460207612457 24.8