# Who Needs Decoders? Efficient Estimation of Sequence-level Attributes

Yassir Fathullah
Engineering Department
University of Cambridge
<EMAIL_ADDRESS>
&Adian Liusie
Engineering Department
University of Cambridge
<EMAIL_ADDRESS>
Puria Radmard
Engineering Department
University of Cambridge
<EMAIL_ADDRESS>
&Mark J. F. Gales
Engineering Department
University of Cambridge
<EMAIL_ADDRESS>
Yassir Fathullah, Puria Radmard, Adian Liusie, Mark J. F. Gales
Engineering Department, University of Cambridge
{yf286, pr450, al826<EMAIL_ADDRESS>

###### Abstract

State-of-the-art sequence-to-sequence models often require autoregressive
decoding, which can be highly expensive. However, for some downstream tasks
such as out-of-distribution (OOD) detection and resource allocation, the
actual decoding output is not needed just a scalar attribute of this sequence.
In these scenarios, where for example knowing the quality of a system’s output
to predict poor performance prevails over knowing the output itself, is it
possible to bypass the autoregressive decoding? We propose Non-Autoregressive
Proxy (NAP) models that can efficiently predict general scalar-valued
sequence-level attributes. Importantly, NAPs predict these metrics directly
from the encodings, avoiding the expensive autoregressive decoding stage. We
consider two sequence-to-sequence task: Machine Translation (MT); and
Automatic Speech Recognition (ASR). In OOD for MT, NAPs outperform a deep
ensemble while being significantly faster. NAPs are also shown to be able to
predict performance metrics such as BERTScore (MT) or word error rate (ASR).
For downstream tasks, such as data filtering and resource optimization, NAPs
generate performance predictions that outperform predictive uncertainty while
being highly inference efficient.

##  1 Introduction

Autoregressive encoder-decoder models have emerged as the dominant approach
for many sequence-to-sequence tasks [48] and are the state-of-the-art for a
range of tasks such as Automatic Speech Recognition (ASR) [18, 40], Machine
Translation (MT) [50, 58], and Abstractive Text Summarization [6, 41].
However, for many applications, the decoded output sequence is not actually
required, only attributes of the sequence. In out-of-distribution (OOD)
detection, only a sequence-level metric such as confidence is required [20,
33]. In selective classification [17, 56, 59] the output is only needed if the
prediction is trusted and rejected in all other cases. Another example is
deferral strategies for resource allocation [29, 49, 51, 57, 65], where
computation is allocated between systems of different complexity. Standard
deferral strategy approaches use the predictive uncertainty of a simpler
system to decide whether or not to pass it on to a better-performing system of
higher complexity [52].

All of the examples above require some form of predictive uncertainty metric
from the output, which in the case of autoregressive models are expensive to
obtain [55]. Transformers, for instance, have recently dominated many
sequence-to-sequence tasks [3, 5, 41], but are often extremely large, with
several million to billions of parameters. Combined with the quadratic cost of
self-attention [50] and autoregressive decoding (equipped with beam-search
[24]), this can limit the application of these systems in real-world settings,
such as those that have limited computational resources or require low latency
[51]. Furthermore, ensembling generally improves system performance and can be
leveraged for useful analysis, such as for robust uncertainty estimation [15,
26]. However, ensembles’ memory and inference costs scale linearly with the
number of members in the ensemble, making them even more impractical for real-
world scenarios. There are methods including Knowledge Distillation (KD) [43,
21] and Ensemble Distribution Distillation (EDD) [34, 13] that attempt to
distil the knowledge from an autoregressive ensemble into a single neural
network but this still does not circumvent the high costs associated with
autoregressive generation.

Previous works have investigated adding a second output head explicitly
trained to capture a specific metric such as epistemic uncertainty in image
segmentation [28] or the true class probability in image classification [9].
The work of [30] extends this style of approach to ASR by adding a second head
to the decoder, to predict token-level decoding errors. Despite its success in
providing robust estimates, computing the output uncertainties still requires
an expensive autoregressive decoding process. The work of [7] trains an
independent proxy model for estimating uncertainties. This method is based on
training a much smaller image classification model in an identical manner to
the primary model, instead using the uncertainty estimates produced by the
small model’s outputs to guide the primary one. In the space of autoregressive
encoder-decoder models, this approach is still not feasible; the costs of
training and decoding persist even for small autoregressive models.

In this paper we propose Non-Autoregressive Proxy (NAP) models that directly
estimate sequence-level attributes, bypassing the expensive decoding process
of autoregressive systems. When deployed, these lightweight proxy models can
be used to robustly predict sequence properties using a fraction of the
computational requirements. Our approach is kept general and applicable to any
sequence attribute, demonstrating the usefulness of this framework to diverse
metrics such as sequence-level predictive uncertainty, BERTScore for MT and
word error rate (WER) for ASR. Investigations into downstream tasks such as
out-of-distribution (OOD) detection show that NAPs can outperform an ensemble
at fraction of the inference time. Due to the flexibility of the proposed
framework, we also investigate training NAPs on sequence-level performance
metrics (BERTScores and WERs), outperforming uncertainty-based approaches to
data filtering and resource optimization.

##  2 Background

For sequence modelling tasks, there has been a range of work on predicting
sequence-level attributes. One common example is deriving information
theoretic uncertainties from the outputs of autoregressive systems [33, 36],
where unsupervised token-level uncertainties from some decoding process are
combined to form sequence-level estimates. Such sequence-level uncertainties
are then used in downstream tasks such as OOD detection [33], quality
estimation [14] and curriculum learning [64].

Previous work has also explored task-specific supervised approaches to
confidence/metric estimation. The work of [16, t] explores training a small
independent model to predict the sub-utterance-level word error rate (WER) of
a primary ASR model for short-duration audio when the reverberant conditions
change. However, the approach is not generalizable to other domains such as MT
due to the specific focus on reverberant speech. Other work has also focused
on training an error detection module attached to the decoder of some ASR or
MT system [12, 24, 25, 30, 31, 42]. For example, a typical approach to
training the decoder-side error detector is based on token-level error labels
from the minimum Levenshtein distance alignment to the ground truth. From
these token-level estimates, a sequence-level confidence score can be derived.
In ASR where there is often one clear true transcription of the input audio,
such an error detection module is appropriate. However, these approaches are
inappropriate for MT where multiple translations could all have the same
meaning and be considered valid. Such a token-level error detector would flag
other valid translations as errorful even when conveying the same information
and meaning.

This final example is one of the main motivations behind BERTScore and related
approaches [45, 60, 62, 63]. BLEU [38, 39] has long been the main MT
evaluation metric for measuring sequence similarity between a translation and
a reference using some measure of overlap. However, it suffers from similar
issues as (Levenshtein) edit-distance metrics. BERTScore resolves such issues
by leveraging bidirectional language models in generating contextual variable-
length embeddings for both the translation and reference sequence, computing
an automatic sequence similarity score in this embedding space. There has also
been a set of work on supervised MT quality estimation [46, 47, 61] in which
models are trained to estimate the quality (human expert estimated metric) of
a translation by making use of the source, the decoded translation and
additional token-level probability. However, both the automatic BERTScore and
quality metrics require an expensive autoregressive decoding stage to obtain
the estimate.

##  3 Non-Autoregressive Proxy (NAP): Efficient Sequence Metric Estimation

We are interested in the general problem of estimating sequence-level
attributes whilst remaining highly inference-efficient. These sequence-level
metrics include: (1) information-theoretic uncertainties [33]; (2) neural-
based evaluation scores such as BERTScore [62]; and (3) discrete sequence-
similarity metrics such as word error rate. The standard approach to obtaining
these sequence-level metrics is to run an expensive autoregressive decoding
scheme to produce a set of hypotheses. One can either extract sequence
attributes directly from this hypothesis set [33] or compare them with their
corresponding references to obtain a measure of sequence similarity.

The aim of this paper is to avoid the costly autoregressive generation stage
and instead train an encoder-only, non-autoregressive proxy (NAP) model to
directly imitate the sequence metrics produced by an autoregressive system,
using only the source, see Figure 1.

(a) Setup 1: Capturing sequence uncertainties.

(b) Setup 2: Capturing sequence similarities.

Figure 1: Our proposed proxy training scheme: A teacher encoder-decoder model
trains a proxy encoder student to predict consistent sequence scores using
some loss function. In (a) we train the proxy to extract sequence
uncertainties from a decoder that is fed the reference. In (b) we train a
proxy to capture sequence-level similarity scores (e.g. BERTScore or WER) from
decoded outputs.

We employ two different setups as shown in Figures 1(a) and 1(b). The aim of
the first setup is to train a proxy to directly extract sequence uncertainties
when the main model is additionally given the reference sequence. This is in
order to teach the proxy model to imitate the uncertainties from the gold
reference. The second setup aims to teach the proxy a sequence similarity
score when the autoregressive generated hypothesis is compared to the
reference. Both setups are highly challenging as the non-autoregressive proxy
is tasked with predicting sequence-level metrics from only the source.

However, the key feature of the NAP is that it directly predicts these metrics
without a decoding scheme (e.g. beam search) and without any reference
sequences, allowing the user to extract useful information from large amounts
of unlabelled data with little cost. Furthermore, in the first setup of Figure
1(a), the proxy also avoids the exposure bias problem [1, 43], by directly
training on the teacher-forced [53] sequence uncertainties.

In this work, we follow Figure 1(a) in training a proxy on both single teacher
confidence and entropy scores or ensemble mutual information estimates,
evaluating its imitation ability and downstream out-of-distribution detection
ability. We also follow Figure 1(b) in training a proxy to predict BERTScores
in Machine Translation and WER in Speech Recognition and evaluate the
performance of the NAP on a data filtering and resource optimization task.

Loss Function: Sequence-level metrics are represented by single scalar values.
Therefore, the proxy student can be trained using any regression loss
function. However, unlike standard regression tasks, we seek to learn the
relative ordering (rankings) of our scores, as this simplifies the task and is
more pertinent for downstream applications such as OOD detection. Therefore,
we will mainly opt for the Spearman Rank and Pearson correlation coefficient
(SCC & PCC) depending on the specific task considered. Consider a batch of $n$
items with teacher scores $\\{s_{i}\\}_{i=1}^{n}$ and corresponding proxy
predictions $\\{\hat{s}_{i}\\}_{i=1}^{n}$. The loss functions are then defined
as:

$\mathcal{L}_{\tt
SCC}=-\left(1-\frac{6\sum_{i}(r(s_{i})-r(\hat{s}_{i}))^{2}}{n(n^{2}-1)}\right)\hskip
14.22636pt\mathcal{L}_{\tt
PCC}=-\left(\frac{\sum_{i}(s_{i}-\mu_{s})(\hat{s}_{i}-\mu_{\hat{s}})}{\sqrt{\sum_{i}(s_{i}-\mu_{s})^{2}}\sqrt{\sum_{i}(\hat{s}_{i}-\mu_{\hat{s}})^{2}}}\right)$
(1)

where $r(s)\in\\{1,2,\dots,n\\}$ signifies the rank of $s$. Since the rank
operator is discrete and non-differentiable it is not directly applicable to
our application. We resort to a differentiable Spearman Rank extension [2]
with an open source implementation111https://github.com/google-research/fast-
soft-sort. The Pearson loss can directly be applied without any modifications.
Note that unlike its original usage [2], where the system is trained to rank
class values for a single instance, we are using this loss to sort single
values associated with multiple different items in a batch. We also
investigate alternative loss functions such as the root mean squared error
(RMSE) and mean absolute error (MAE), see Appendix B.1.

Predictor Design: In order to produce a scalar score from a variable-length
encoder-output representation, we make use of a pooling operation. We utilize
two options, temporal averaging or multi-head attention with a single
trainable query. The encoder vector outputs $\\{{\bm{v}}_{l}\\}_{l=1}^{L}$ are
therefore pooled to form a fixed-size representation ${\bm{v}}$ which is fed
into a three-layer multi-layer perception (MLP). Furthermore, early
exploratory experiments found that a softmax activation is vital for good
performance as it can be seen as introducing inductive bias into the
estimation of information-theoretic and related metrics. Architectural details
of the MLP and ablation studies are provided in Appendix B.2.

Proxy Encoder Backbone: By default, the NAP encoder backbone is initialized
from the encoder weights of the main encoder-decoder model. However, many
pretrained models such as the T5 [41] and Whisper [40] come in different
sizes. The backbone will therefore also be initialized from the encoder of a
smaller T5 model and be taught to predict the uncertainties from a larger
system. Appendix B.4 further explores ‘mismatched’ encoders, e.g. using a
RoBERTa NAP to predict the output attributes of a T5 system.

Furthermore, all experiments in this paper freeze the encoder backbone and
only train the small predictor on top of the NAP encoder. This improves the
training speed and memory usage allowing a user to train multiple predictor
heads on top of the same backbone, each for a different metric (e.g.
estimating sequence-level confidence and BERTScores in the same forward pass).
Note that the purpose of our investigations are not to create the best
possible NAP model (for example, fine-tuning the backbone encoder could
improve performance at no cost of inference speed). We only seek to
demonstrate that this approach is highly flexible and applicable to a range of
sequence-level metrics and can provide cheap but useful information for
sequence-to-sequence tasks.

##  4 Experimental Evaluation

Predicting Uncertainties: We will evaluate the imitation ability of NAP models
on various tasks. Following Setup 1, the first set of experiments will focus
on the ability of a proxy system to capture sequence-level confidence or
entropy from a single T5 transformer [41] finetuned on a spoken-language
Machine Translation (MT) dataset. We further explore the ability of NAPs to
imitate mutual information (epistemic uncertainty [10, 22]) from an ensemble
of T5 systems. The performance of the NAPs will then be evaluated by measuring
the Spearman Rank correlation between the teacher (under teacher-forcing [53])
and the proxy estimates on a range of in-domain (ID) and out-of-domain (OOD)
datasets. We also investigate the performance of the proposed NAP on OOD
detection.

Predicting BERTScores: Following Setup 2, we also investigate if proxy systems
can capture much more complex sequence metrics such as BERTScores [62] from a
single T5 in MT. Capturing this metric is especially challenging since the
beam-search output of the T5 decoder and corresponding reference will be fed
through a language model such as BERT [11] which then computes the final
score. The performance will be measured by computing the Spearman Rank between
proxy outputs and BERTScores on both ID and OOD datasets. Furthermore, the
proxy is compared to sequence-level confidence and entropy scores from the T5
model to see how well they correlate with BERTScores.

The performance of a BERTScore estimating proxy system can also be evaluated
on two downstream tasks: Filtering task [30]: Given a dataset, we remove the
examples with the lowest proxy or highest uncertainty estimate. For good
estimates, the filtered subset should display a higher average BERTScore.
Resource optimization task [51]: Under a fixed resource budget, one seeks to
allocate inputs to models of different complexity in order to maximise
performance. A well-performing allocation system would achieve higher
performance with a smaller budget, see Figure 2.

(a) Baseline deferral system.

(b) Proxy deferral system.

Figure 2: Deferral systems: In the baseline deferral system, the inputs with
high uncertainty (under the small model) are fed into the larger model. In the
proxy deferral system, model selection is based on the output of the efficient
proxy.

Predicting WER: Finally, we follow Setup 2 in investigating if a NAP can
imitate the sentence-level WER and the total number of errors produced by an
ASR system. In this case, we utilize the pretrained state-of-the-art Whisper
[40] models on the LibriSpeech corpus [37]. Since the Whisper model is very
well-performing, it is able to perfectly decode a large fraction of the
dataset, which would cause issues for a rank-based loss such as Spearman. We,
therefore, resort to Pearson for these experiments. Note, the corpus-level WER
performance of an ASR system is a length-weighted average of the sentence-
level WERs. Therefore, we also train NAPs to predict the number of decoding
errors in an utterance. Similar to the BERTScore experiments, the performance
of NAPs will be evaluated in a similar manner using both filtering and
resource optimization tasks.

###  4.1 Machine Translation

We use the IWSLT 2017 English-to-German training set for finetuning T5 systems
on spoken language translation. We generate a three-model ensemble of T5
systems which we use as a stronger baseline for uncertainty estimation. We
also investigate if Knowledge Distillation (KD) [21] and Ensemble Distribution
Distillation (EDD) [34, 44] are able to imitate the uncertainties produced by
a single or ensemble systems respectively.

We use a range of in-domain and out-of-domain datasets for downstream tasks.
These include the Web Inventory Talk (Ted IWSLT 2016; ID), Newstest19 & 20
news commentary (OOD-1), Khresmoi medical data (OOD-2), MTNT-2019 Reddit text
(OOD-3) and KFTT Kyoto-related Wikipedia articles (OOD-3) datasets. All but
the latter two datasets are English-to-German, while the final two are
English-to-Japanese. Due to the language mismatch, OOD-3 datasets cannot be
used to evaluate BERTScore prediction in Section 4.1.2. Dataset and setup
details are available in Appendix A.

Table 1 shows how long inference of the iwslt-2017 test set takes for various
models. This demonstrates a primary desideratum of a NAP, the ability to
quickly process large amounts of data. For example, a large proxy being 46x
faster than a T5 Large model using a beam of $B=12$ (used in experiments
below) and is approximately 138x faster than the three-model ensemble (if run
serially). Given the shared architecture between the proxy and primary model
encoders, this vast difference in inference time is due to the ability to
bypass expensive decoding.

Table 1: Approximate corpus inference time for iwslt-2017 using Hugging Face [54], with an NVIDIA A100. The proxy performs an encoder forward pass and the T5 models perform beam search. BERTScore performance computed for the B = 12 setting which is used in downstream tasks. Model | T5 Model | NAP
---|---|---
B = 1 | B = 4 | B = 8 | B = 12 | BERTScore
Small (S) | 41.9s | 85.9s | 119.7s | 178.6s | 67.4 | 2.7s
Base (B) | 117.7s | 270.3s | 347.4s | 537.6s | 68.2 | 5.5s
Large (L) | 313.7s | 583.4s | 755.0s | 826.6s | 68.6 | 17.9s

####  4.1.1 Estimating Uncertainties in Machine Translation

We trained NAPs (of three different sizes, see Table 1) to predict either
sequence-level confidence $\mathcal{P}$ or entropy $\mathcal{H}$ (using the
conditional approximation described in [33]) of a T5 Large model. We also
trained NAPs to predict the mutual information $\mathcal{I}$ score produced by
an ensemble of finetuned T5 Large models. The performance of the proxies is
compared to two baseline systems: KD when capturing confidence or entropy of a
single model, and EDD in capturing mutual information from an ensemble. The
autoregressive distilled baselines will also be of three different sizes, see
Table 1.

In the case of confidence $\mathcal{P}$ and mutual information scores
$\mathcal{I}$, the proxy achieves a better rank ordering of instances for both
datasets and at all sizes than the corresponding encoder-decoder student,
despite being an order of magnitude faster at inference (Table 2). Knowledge-
distilled models are better at imitating their teacher’s $\mathcal{H}$,
however, this is not indicative of downstream task performance such as OOD
detection, as explored below (Table 3). Note that the NAP here is unique in
its ability to predict any scalar sequence metric, whereas KD is unable to
mimic mutual information scores.

Table 2: Spearman Rank correlation of uncertainties when comparing baseline distillation and proxy to the teacher ensemble. Averaged over 3 runs. Standard deviations in the order of $\pm 1.0$. Model Size | S | B | L | S | B | L | S | B | L
---|---|---|---|---|---|---|---|---|---
Dataset | Distillation $\mathcal{P}$ | Distillation $\mathcal{H}$ | EDD $\mathcal{I}$
iwslt-2017 | 18.7 | 19.8 | 20.8 | 69.4 | 73.1 | 74.5 | 43.7 | 51.5 | 55.1
ted-iwslt-2016 | 21.4 | 21.1 | 21.8 | 57.5 | 59.5 | 60.6 | 46.8 | 47.0 | 48.0
Dataset | NAP $\mathcal{P}$ | NAP $\mathcal{H}$ | NAP $\mathcal{I}$
iwslt-2017 | 39.9 | 42.6 | 42.1 | 40.4 | 58.8 | 62.7 | 53.7 | 54.3 | 55.6
ted-iwslt-2016 | 26.2 | 25.3 | 25.2 | 44.8 | 52.3 | 53.8 | 50.0 | 49.7 | 51.3

Finally, we perform downstream out-of-distribution detection using confidence,
entropy, and MI scores from T5 Large ensemble, EDD (T5 Large), and Proxy
Large. We use iwslt-2017 as in-domain and measure performance with AUROC (a
score of 50% represents random detection). Results in Table 3 show that in all
but one scenario, the uncertainty scores predicted by the proxy model are best
suited for the task, particularly considering inference speeds. Note that
overall, the detection performance of a NAP exceeds that of the Deep Ensemble.
A potential explanation is that the proxy is directly trained to predict
uncertainties while the ensemble estimates uncertainties based on the beam-
search decoded outputs [33], suffering from exposure bias [1, 43].

Table 3: %AUROC detection performance of autoregressive and proxy models using various uncertainties. Averaged over 3 runs. Standard deviations in the order of $\pm 2.0$. Split | Dataset | Deep Ensemble | EDD | NAP
---|---|---|---|---
$\mathcal{P}$ | $\mathcal{H}$ | $\mathcal{I}$ | $\mathcal{P}$ | $\mathcal{H}$ | $\mathcal{I}$ | $\mathcal{P}$ | $\mathcal{H}$ | $\mathcal{I}$
OOD-1 | newstest-19 | 42.9 | 53.1 | 58.5 | 45.5 | 54.6 | 55.7 | 51.0 | 53.4 | 70.5
newstest-20 | 35.9 | 50.8 | 63.4 | 40.6 | 54.0 | 61.2 | 51.6 | 53.2 | 78.1
OOD-2 | khresmoi-dev | 38.1 | 51.8 | 67.2 | 43.6 | 57.2 | 63.4 | 50.4 | 51.1 | 77.9
khresmoi-test | 39.4 | 53.8 | 67.6 | 44.4 | 58.5 | 63.4 | 55.5 | 54.9 | 81.2
OOD-3 | mtnt-2019 | 66.0 | 72.2 | 64.4 | 67.0 | 72.0 | 61.9 | 70.4 | 72.0 | 71.4
kftt | 31.9 | 33.8 | 47.0 | 32.6 | 35.8 | 40.8 | 27.3 | 34.8 | 54.7

####  4.1.2 Estimating BERTScores in Machine Translation

Table 4 directly compares the rank correlation between model confidence/proxy
scores and sentence BERTScore performance. We include proxies with attentive
pooling as this is a more challenging task. These suggest that training NAPs
directly on performance metrics provides a better predictor of a system’s
performance than using information-theoretic metrics such as confidence and
entropy.

Table 4: Spearman Rank correlation score between model confidence/entropy and the model BERTScore (fine-tuned T5 Large). The NAPs were trained to predict this score directly. Averaged over 3 runs. Standard deviations in the order of $\pm 2.0$. Split | Dataset | T5 Large | NAP | NAP w/ Attention
---|---|---|---|---
$\mathcal{P}$ | $\mathcal{H}$ | S | B | L | S | B | L
ID | iwslt-2017 | 16.6 | 41.6 | 42.0 | 43.7 | 44.9 | 42.5 | 44.4 | 45.6
ted-iwslt-2016 | 11.6 | 37.3 | 35.8 | 36.3 | 37.3 | 35.7 | 37.0 | 38.1
OOD-1 | newstest-19 | 32.9 | 39.3 | 34.3 | 36.7 | 37.6 | 34.7 | 37.1 | 39.2
newstest-20 | 34.2 | 38.3 | 38.6 | 38.7 | 39.6 | 38.9 | 39.0 | 39.3
OOD-2 | khresmoi-dev | 41.4 | 45.5 | 40.8 | 43.1 | 44.7 | 41.3 | 42.3 | 44.8
khresmoi-test | 42.9 | 46.1 | 42.0 | 46.5 | 45.5 | 42.3 | 47.8 | 45.2
| average | 29.9 | 41.3 | 38.9 | 40.8 | 41.6 | 39.2 | 41.3 | 42.0

Dataset filtering is an alternative approach to evaluating the quality of
uncertainty or proxy estimates, with emphasis on the highest-performing
examples. A well-suited predictor of performance will show a monotonic
increase in filtered dataset performance, as examples for which a low
performance is predicted are removed. Figure 3 shows this desired behaviour is
best achieved with NAPs (equipped with attention pooling) that are directly
trained to predict BERTScores of the primary model, in both an ID and OOD
dataset. Entropy produced by the model itself is promising on the ID dataset
but fails on OOD since the performance does not increase as we filter more
inputs. Failure to reproduce these trends from uncertainty estimates of the
primary model output suggests over-confidence [19] in low-performing examples.

Figure 3: Measuring the BERTScore performance of T5 Large (primary) model on a
filtered dataset when removing the worst examples according to some metric:
(1) confidence, (2) entropy or (3) proxy outputs. The proxy was trained on the
BERTScores of the primary model (on the training set).

Figure 4 shows results for resource allocation, where examples are allocated
to either a T5 Small or Large based on whether a performance-based related
metric is above or below a threshold. Depending on the fraction allocated to
the larger system, different levels of overall inference time and performance
are achieved.

(a) IWSLT 2017

(b) Newstest 20

Figure 4: Resource allocation: Measuring the overall BERTScore performance and
inference time when distributing inputs between a T5 Small and Large according
to some metric.

As expected from the dataset filtering results, proxy outputs can better
predict instances for which the small model will perform poorly and it does so
with a minuscule time cost. By contrast, relying on the output of the small
model itself to decide whether the large model is required causes serious
delays due to the time spent decoding, delays that the NAP preempts. The best
performance was achieved by NAPs trained on the difference in BERTScore
between the two available systems. The aim of this difference metric is to
assign to the large model, examples for which we expect a maximal increase in
performance. Obtaining such a difference metric using the original models
would defeat the whole purpose of resource optimization. Finally, it is
possible to be more efficient or better performing than a T5 Base using this
deferral system while matching its performance or efficiency respectively.

###  4.2 Estimating WERs in Automatic Speech Recognition

We repeat experiments from Section 4.1.2 using pretrained Whisper models from
Hugging Face [54] on the LibriSpeech corpus [37]. We will by default use
greedy decoding as opposed to beam-search since it was found to be robust
enough [40]. Table 5 shows real-time factors (RTFs) demonstrating the
inference efficiency of NAPs which do not require a decoder. Compared to
greedy ($B=1$) decoding of Whisper Large-V2, medium and large-sized NAPs are
43 and 33 times faster, respectively.

Table 5: Real-time Factors (RTFs). The proxy performs an encoder forward pass & (pretrained) Whisper models perform beam search. Evaluated on test.other using an NVIDIA A100. Corpus WER performance computed for the B = 1 setting which is used in downstream tasks. Model | Whisper Models | NAP
---|---|---
B = 1 | B = 5 | %WER
Small | 0.0480 | 0.0507 | 7.62 | 0.0014
Medium | 0.0722 | 0.1075 | 6.26 | 0.0024
Large-V2 | 0.1029 | 0.1625 | 5.16 | 0.0031

Table 6 recreates the prior success of proxies in imitating model performance,
in this case, sentence-level WER. Furthermore, since Whisper encoders pad all
inputs to 30s, including an attention pooling layer can discount the padding
and significantly improve performance. The following experiments will use the
medium-sized NAP with attention pooling as default since it was found to have
similar performance to its larger counterpart on the development sets but with
a 23% smaller RTF.

Table 6: Pearson correlation between Whisper Large-V2 confidence/entropy and sentence WER. The NAPs were trained to predict WER directly. Standard deviations in the order of $\pm 1.0$. Dataset | Whisper Large-V2 | NAP | NAP w/ Attention
---|---|---|---
$\mathcal{P}$ |  $\mathcal{H}$ | S | M | L | S | M | L
test.clean | 13.3 | 16.8 | 32.4 | 36.3 | 33.9 | 43.9 | 49.7 | 47.2
test.other | 51.9 | 60.1 | 38.0 | 42.4 | 43.8 | 49.8 | 59.0 | 61.5

Figure 5 shows the filtered corpus WER of test.clean and test.other when
removing the worst examples according to model confidence/entropy or proxy
outputs. While all are successful on test.other, sequence-level confidence and
entropy significantly suffer on test.clean showing increasing corpus WER in
certain regions when supposedly removing bad examples, a sign of over-
confidence. This failure on test.clean could have been somewhat predicted by
the small correlations in Table 6 while NAPs with attention show a
significantly better correlation performance with sentence WER.

Figure 5: Measuring the corpus WER performance of Whisper Large-V2 (primary)
on a filtered dataset when removing the worst examples according to some
metric: (1) confidence, (2) entropy or (3) proxy outputs. The proxy was
trained on the WERs of the primary model (on the training set).

Figure 6 shows results for resource allocation, where examples are allocated
to a Whisper Small or Large-V2 based on some performance-based related metric.
Again, deferral systems using NAPs (with attention) significantly outperform
decoder uncertainty-based selection schemes. In fact, the best-performing NAP
here was one trained on the number of errors in a transcription, rather than
the WER. This is simply because the ordinate in Figure 6 is the corpus WER,
rather than the average sentence WER. This is proportional to the error count
in the whole corpus, making this a more suitable optimization target. Finally,
we note that resource optimisation by training a proxy to predict a difference
in WER or errors is not presented here. Since the Whisper Small and Large-V2
make the same number of word errors in approximately 75% of examples on the
training set, training a proxy on such a sparse label set is difficult.

Figure 6: Resource allocation: Measuring the overall corpus WER performance and RTF when allocating inputs between a Whisper Small and Large-V2 according to some metric. Table 7: Columns show (1) corpus WER performance of various deferral systems operating at the same RTF as Whipser Medium and (2) the RTF when operating at the same WER as Whipser Medium. Selection | WER | RTF
---|---|---
Whisper Medium | 6.26 | 0.0722
Confidence Selection | 6.19 | 0.0707
Entropy Selection | 6.09 | 0.0677
Proxy trained on WER L | 5.94 | 0.0645
Proxy trained on WER S | 5.89 | 0.0640
Proxy trained on Error L | 5.77 | 0.0596
Proxy trained on Error S | 5.57 | 0.0534

Finally, Table 7 shows the WER or RTF of various deferral systems (between
Whisper Small and Large-V2) when operating at the Whisper Medium RTF or WER
respectively. The best deferral system, a NAP trained on the number of errors
of Whisper Small, reduces WER by 11% while matching the inference speed of
Whisper Medium. For the same WER performance, this system can reduce the RTF
by 26%.

##  5 Conclusion

For many downstream sequence-to-sequence tasks, only attributes of the output
sequence are needed, and not the output itself. In this paper, we propose a
simple, highly efficient, framework for directly estimating scalar sequence-
level attributes, Non-Autoregressive Proxies (NAPs). These lightweight models
completely bypass the expensive autoregressive decoding process, only making
use of an encoder stage. We show that NAPs can learn information-theoretic
uncertainties as well as performance metrics, such as BERTScores for MT or
WERs for ASR, in terms of both mimicing attribute score ranks and the impact
on downstream tasks. For MT systems they outperform a deep ensemble on OOD
detection with an order of magnitude higher inference speed. Furthermore, NAPs
are able to outperform predictive uncertainty on downstream tasks such as data
filtering and resource optimization on both ASR and MT tasks.

## References

  * [1] Samy Bengio, Oriol Vinyals, Navdeep Jaitly and Noam Shazeer  “Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks”  In _Conference on Neural Information Processing Systems_ , 2015
  * [2] Mathieu Blondel, Olivier Teboul, Quentin Berthet and Josip Djolonga  “Fast Differentiable Sorting and Ranking”  In _International Conference on Machine Learning_ , 2020
  * [3] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry and Amanda Askell  “Language models are few-shot learners”  In _Advances in neural information processing systems_ 33, 2020, pp. 1877–1901
  * [4] Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian Stüker, Katsuitho Sudoh, Koichiro Yoshino and Christian Federmann  “Overview of the IWSLT 2017 evaluation campaign”  In _International Workshop on Spoken Language Translation_ , 2017
  * [5] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton and Sebastian Gehrmann  “Palm: Scaling language modeling with pathways”  In _arXiv preprint arXiv:2204.02311_ , 2022
  * [6] Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani and Siddhartha Brahma  “Scaling instruction-finetuned language models”  In _arXiv preprint arXiv:2210.11416_ , 2022
  * [7] Cody Coleman, Christopher Yeh, Stephen Mussmann, Baharan Mirzasoleiman, Peter Bailis, Percy Liang, Jure Leskovec and Matei Zaharia  “Selection via Proxy: Efficient Data Selection for Deep Learning”  In _International Conference on Learning Representations_ , 2020
  * [8] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer and Veselin Stoyanov  “Unsupervised Cross-lingual Representation Learning at Scale”  In _Association for Computational Linguistics_ , 2020
  * [9] Charles Corbière, Nicolas Thome, Avner Bar-Hen, Matthieu Cord and Patrick Pérez  “Addressing Failure Prediction by Learning Model Confidence”  In _Conference on Neural Information Processing Systems_ , 2019
  * [10] Armen Der Kiureghian and Ove Ditlevsen  “Aleatory or epistemic? Does it matter?”  In _Structural safety_ , 2009
  * [11] Jacob Devlin, Ming-Wei Chang, Kenton Lee and Kristina Toutanova  “BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding”  In _Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2019
  * [12] Gunnar Evermann and Philip C. Woodland  “Large vocabulary decoding and confidence estimation using word posterior probabilities”  In _International Conference. on Acoustics, Speech and Signal Processing (ICASSP)_ , 2000
  * [13] Yassir Fathullah, Mark J.F. Gales and Andrey Malinin  “Ensemble Distillation Approaches for Grammatical Error Correction”  In _International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2021
  * [14] Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary and Lucia Specia  “Unsupervised quality estimation for neural machine translation”  In _Transactions of the Association for Computational Linguistics_ , 2020
  * [15] Yarin Gal and Zoubin Ghahramani  “Dropout as a bayesian approximation: Representing model uncertainty in deep learning”  In _International Conference on Machine Learning (ICML)_ , 2016
  * [16] Hannes Gamper, Dimitra Emmanouilidou, Sebastian Braun and Ivan J Tashev  “Predicting word error rate for reverberant speech”  In _International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2020
  * [17] Yonatan Geifman and Ran El-Yaniv  “Selective Classification for Deep Neural Networks”  In _International Conference on Neural Information Processing Systems_ , 2017
  * [18] Anmol Gulati, Chung-Cheng Chiu, James Qin, Jiahui Yu, Niki Parmar, Ruoming Pang, Shibo Wang, Wei Han, Yonghui Wu, Yu Zhang and Zhengdong Zhang  “Conformer: Convolution-augmented Transformer for Speech Recognition”  In _Interspeech_ , 2020
  * [19] Chuan Guo, Geoff Pleiss, Yu Sun and Kilian Q. Weinberger  “On Calibration of Modern Neural Networks”  In _International Conference on Machine Learning_ , 2017
  * [20] Dan Hendrycks and Kevin Gimpel  “A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks”  In _International Conference on Learning Representations (ICLR)_ , 2017
  * [21] Geoffrey Hinton, Oriol Vinyals and Jeff Dean  “Distilling the Knowledge in a Neural Network”  In _Conference on Neural Information Processing Systems Deep Learning Workshop_ , 2014
  * [22] Stephen C Hora  “Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management”  In _Reliability Engineering & System Safety_, 1996
  * [23] Wei-Ning Hsu, Benjamin Bolte, Yao-Hung Hubert Tsai, Kushal Lakhotia, Ruslan Salakhutdinov and Abdelrahman Mohamed  “HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Units”  In _IEEE/ACM Transactions on Audio, Speech, and Language Processing_ , 2021
  * [24] Philipp Koehn  “Statistical machine translation”  Cambridge University Press, 2009
  * [25] Aviral Kumar and Sunita Sarawagi  “Calibration of Encoder Decoder Models for Neural Machine Translation”  In _arXiv arXiv:1903.00802_ , 2019
  * [26] Balaji Lakshminarayanan, Alexander Pritzel and Charles Blundell  “Simple and scalable predictive uncertainty estimation using deep ensembles”  In _Advances in neural information processing systems_ 30, 2017
  * [27] Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma and Radu Soricut  “ALBERT: A Lite BERT for Self-supervised Learning of Language Representations”  In _International Conference on Learning Representations (ICLR)_ , 2020
  * [28] Steven Landgraf, Kira Wursthorn, Markus Hillemann and Markus Ulrich  “DUDES: Deep Uncertainty Distillation using Ensembles for Semantic Segmentation”  In _arXiv, arXiv:2303.09843_ , 2023
  * [29] Haoxiang Li, Zhe Lin, Xiaohui Shen, Jonathan Brandt and Gang Hua  “A convolutional neural network cascade for face detection”  In _IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2015
  * [30] Qiujia Li, David Qiu, Yu Zhang, Bo Li, Yanzhang He, Philip C. Woodland, Liangliang Cao and Trevor Strohman  “Confidence Estimation for Attention-based Sequence-to-sequence Models for Speech Recognition”  In _International Conference on Acoustics, Speech and Signal Processing_ , 2021
  * [31] Hank Liao and Mark JF Gales  “Uncertainty decoding for noise robust speech recognition”  In _Interspeech_ , 2007
  * [32] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer and Veselin Stoyanov  “RoBERTa: A Robustly Optimized BERT Pretraining Approach”  In _arXiv, arXiv:1907.11692_ , 2019
  * [33] Andrey Malinin and Mark Gales  “Uncertainty Estimation in Autoregressive Structured Prediction”  In _International Conference on Learning Representations_ , 2021
  * [34] Andrey Malinin, Bruno Mlodozeniec and Mark J.. Gales  “Ensemble distribution distillation”  In _International Conference on Learning Representations_ , 2020
  * [35] Chris Manning and Hinrich Schütze  “Foundations of Statistical Natural Language Processing”  MIT Press, 1999
  * [36] Pascal Notin, José Miguel Hernández-Lobato and Yarin Gal  “Improving black-box optimization in VAE latent space using decoder uncertainty”  In _Advances in Neural Information Processing Systems_ , 2021
  * [37] Vassil Panayotov, Guoguo Chen, Daniel Povey and Sanjeev Khudanpur  “Librispeech: An ASR corpus based on public domain audio books”  In _International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2015
  * [38] Kishore Papineni, Salim Roukos, Todd Ward and Wei-Jing Zhu  “Bleu: a method for automatic evaluation of machine translation”  In _Association for Computational Linguistics_ , 2002
  * [39] Matt Post  “A Call for Clarity in Reporting BLEU Scores”  In _Conference on Machine Translation: Research Papers_ , 2018
  * [40] Alec Radford, Jong Wook Kim, Tao Xu, Greg Brockman, Christine McLeavey and Ilya Sutskever  “Robust Speech Recognition via Large-Scale Weak Supervision”  In _arXiv, arXiv:2212.04356_ , 2022
  * [41] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li and Peter J. Liu  “Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer”  In _Journal of Machine Learning Research_ , 2020
  * [42] Anton Ragni, Qiujia Li, Mark JF Gales and Yongqiang Wang  “Confidence estimation and deletion prediction using bidirectional recurrent neural networks”  In _IEEE Spoken Language Technology Workshop (SLT)_ , 2018
  * [43] Marc’Aurelio Ranzato, Sumit Chopra, Michael Auli and Wojciech Zaremba  “Sequence Level Training with Recurrent Neural Networks”  In _International Conference on Learning Representations_ , 2016
  * [44] Max Ryabinin, Andrey Malinin and Mark J.. Gales  “Scaling Ensemble Distribution Distillation to Many Classes with Proxy Targets”  In _Conference on Neural Information Processing Systems_ , 2021
  * [45] Thibault Sellam, Dipanjan Das and Ankur P Parikh  “BLEURT: Learning robust metrics for text generation”  In _Annual Meeting of the Association for Computational Linguistics_ , 2020
  * [46] Lucia Specia, Frédéric Blain, Marina Fomicheva, Erick Fonseca, Vishrav Chaudhary, Francisco Guzmán and André F.. Martins  “Findings of the WMT 2020 Shared Task on Quality Estimation”  In _Proceedings of the Fifth Conference on Machine Translation (WMT)_ , 2020
  * [47] Lucia Specia, Frédéric Blain, Marina Fomicheva, Chrysoula Zerva, Zhenhao Li, Vishrav Chaudhary and André F.. Martins  “Findings of the WMT 2021 Shared Task on Quality Estimation”  In _Proceedings of the Sixth Conference on Machine Translation (WMT)_ , 2021
  * [48] Ilya Sutskever, Oriol Vinyals and Quoc V Le  “Sequence to sequence learning with neural networks”  In _Advances in neural information processing systems_ 27, 2014
  * [49] Surat Teerapittayanon, Bradley McDanel and H.T. Kung  “BranchyNet: Fast inference via early exiting from deep neural networks”  In _International Conference on Pattern Recognition (ICPR)_ , 2016
  * [50] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser and Illia Polosukhin  “Attention is all you need”  In _Advances in neural information processing systems_ 30, 2017
  * [51] Paul Viola and Michael Jones  “Rapid object detection using a boosted cascade of simple features”  In _IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2001
  * [52] Xiaofang Wang, Dan Kondratyuk, Eric Christiansen, Kris M. Kitani, Yair Movshovitz-Attias and Elad Eban  “Wisdom of Committees: An Overlooked Approach To Faster and More Accurate Models”  In _International Conference on Learning Representations (ICLR)_ , 2022
  * [53] Ronald J. Williams and David Zipser  “A Learning Algorithm for Continually Running Fully Recurrent Neural Networks”  In _Neural Computation_ , 1989
  * [54] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest and Alexander Rush  “Transformers: State-of-the-Art Natural Language Processing”  In _Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , 2020
  * [55] Yonghui Wu, Mike Schuster, Zhifeng Chen, Quoc V Le, Mohammad Norouzi, Wolfgang Macherey, Maxim Krikun, Yuan Cao, Qin Gao and Klaus Macherey  “Google’s neural machine translation system: Bridging the gap between human and machine translation”  In _arXiv preprint arXiv:1609.08144_ , 2016
  * [56] Guoxuan Xia and Christos-Savvas Bouganis  “Augmenting Softmax Information for Selective Classification with Out-of-Distribution Data”  In _Computer Vision – Asian Conference on Computer Vision_ , 2022
  * [57] Guoxuan Xia and Christos-Savvas Bouganis  “Window-Based Early-Exit Cascades for Uncertainty Estimation: When Deep Ensembles are More Efficient than Single Models”  In _arXiv preprint arXiv:2303.08010_ , 2023
  * [58] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua and Colin Raffel  “mT5: A Massively Multilingual Pre-trained Text-to-Text Transformer”  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , 2021, pp. 483–498
  * [59] Ran El-Yaniv and Yair Wiener  “On the Foundations of Noise-free Selective Classification”  In _Journal of Machine Learning Research_ , 2010
  * [60] Weizhe Yuan, Graham Neubig and Pengfei Liu  “Bartscore: Evaluating generated text as text generation”  In _Advances in Neural Information Processing Systems_ , 2021
  * [61] Chrysoula Zerva, Frédéric Blain, Ricardo Rei, Piyawat Lertvittayakumjorn, José G. C. Souza, Steffen Eger, Diptesh Kanojia, Duarte Alves, Constantin Orăsan, Marina Fomicheva, André F.. Martins and Lucia Specia  “Findings of the WMT 2022 Shared Task on Quality Estimation”  In _Proceedings of the Seventh Conference on Machine Translation (WMT)_ , 2022
  * [62] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger and Yoav Artzi  “BERTScore: Evaluating Text Generation with BERT”  In _International Conference on Learning Representations_ , 2020
  * [63] Wei Zhao, Maxime Peyrard, Fei Liu, Yang Gao, Christian M Meyer and Steffen Eger  “MoverScore: Text generation evaluating with contextualized embeddings and earth mover distance”  In _Conference on Empirical Methods in Natural Language Processing and Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , 2019
  * [64] Yikai Zhou, Baosong Yang, Derek F Wong, Yu Wan and Lidia S Chao  “Uncertainty-aware curriculum learning for neural machine translation”  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , 2020, pp. 6934–6944
  * [65] Qiang Zhu, Mei-Chen Yeh, Kwang-Ting Cheng and S. Avidan  “Fast Human Detection Using a Cascade of Histograms of Oriented Gradients”  In _IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2006

Who Needs Decoders? Efficient Estimation of Sequence-level Attributes
(Supplementary Material) Yassir Fathullah, Puria Radmard, Adian Liusie, Mark
J. F. Gales
Engineering Department, University of Cambridge
{yf286, pr450, al826<EMAIL_ADDRESS>

##  Appendix A Experimental Configuration

This section will describe the experimental setup of all experiments. Details
about datasets, models, and training hyperparameters and evaluation are
provided. Hugging Face was used extensively for all experiments in terms of
loading various pretrained models, corresponding tokenizers and processed
datasets.

###  A.1 Machine Translation

####  A.1.1 Datasets

In Table 8 we report information about the datasets we use for training and
evaluation. Note that we use the T5 [41] approach for English-to-German
tokenization meaning that we prepend the following prompt to all inputs
"translate English to German:" prior to tokenization. We use the iwslt-2017
training set for finetuning T5 systems on spoken language translation and
evaluate on the corresponding test set. We furthermore use the in-domain (ID)
spoken language test set and out-of-domain news commentary (OOD-1), medical
data (OOD-2) and a final mixed category of noisy text and Japanese articles
(OOD-3) for downstream tasks.

Table 8: Dataset statistics post tokenization.

Split Dataset #Sequences #Tokens/Sequence src ref Training iwslt-2017 206,112
29.1 28.5 Validation 888 31.9 32.7 Evaluation 8,079 27.8 27.5 ID ted-
iwslt-2016 3,662 46.4 54.2 OOD-1 newstest-19 1,997 35.3 39.7 newstest-20 1,418
49.1 61.6 OOD-2 khresmoi-dev 500 33.7 38.6 khresmoi-test 1,000 34.7 40.4 OOD-3
mtnt-2019 1,392 - kftt 1,160 -

####  A.1.2 Models

All experiments use the T5 model. In Table 9 we report parameter counts of
various models. The T5 is an encoder-decoder model with a language model head
which predicts a probability mass function over every token in the output
sequence. The proxy model consists of a T5 encoder and a head for predicting
uncertainty. The parameter counts below are reported for a proxy with an
average pooling layer; an attentive pooling layer would add some parameters.
Note, although the embedding layer is expensive parameter-wise, it is
extremely fast inference-wise since it is equivalent to a lookup table.

Table 9: Parameter counts of models. NAPs do not use a decoder during
inference.

Model Embeddings Encoder Decoder Head Total T5 Small 16.4M 35.3M 41.6M 16.4M
60.5M NAP Small - 5.2M 40.6M T5 Base 24.7M 109.6M 137.9M 24.7M 222.9M NAP Base
- 11.8M 121.4M T5 Large 32.9M 334.9M 435.6M 32.9M 737.7M NAP Large - 20.9M
355.9M

####  A.1.3 Finetuning T5 Models

All T5 models were finetuned on the IWSLT-2017 [4] training set and evaluated
on several ID and OOD datasets using both SacreBLEU [39] and BERTScore (BS)
[62], see Table 10. We set the beam size to 12 and used a length penalty of
0.60.

The learning rate was fixed to 0.0001 and the batch size was selected to
maximise GPU memory usage on a single NVIDIA A100 SXM4 80GBs. The performance
was tracked on the validation set 10 times per epoch and training was
terminated when performance did not improve for a whole epoch.

Table 10: SacreBLEU and BERTScore performance of finetuned T5 models. Split | Dataset | Small | Base | Large
---|---|---|---|---
BLEU | BS | BLEU | BS | BLEU | BS
ID | iwslt-2017 | 32.0 | 67.4 | 33.8 | 68.2 | 34.3 | 68.6
ted-iwslt-2016 | 30.9 | 65.2 | 31.9 | 65.9 | 32.3 | 66.3
OOD-1 | newstest-19 | 37.3 | 68.0 | 38.9 | 69.8 | 38.9 | 69.9
newstest-20 | 29.4 | 64.4 | 30.8 | 65.4 | 31.4 | 65.9
OOD-2 | khresmoi-dev | 27.1 | 68.9 | 29.2 | 70.7 | 29.4 | 70.7
khresmoi-test | 27.4 | 68.0 | 30.0 | 70.2 | 30.2 | 70.3

The table shows that increasing the size of the T5 model improves performance
on the ID datasets. Surprisingly the performance gap between the base and
large configuration is very small for most OOD datasets, showing that the base
model is particularly effective despite being more than a third of the size.

####  A.1.4 Training Non-Autoregressive Proxies

We generated scores (uncertainty or BERTScore) from finetuned T5 Large models
and used them to train NAP models. We used the smooth and differentiable
extension to the Spearman Rank loss function [2] which requires a
hyperparameter controlling the level of smoothing. This hyperparameter was set
to 0.000001 in all experiments. Similar to the section above, all experiments
used a learning rate of 0.0001, maximised batch size and training was stopped
when performance did not improve after an epoch.

####  A.1.5 Estimating Uncertainties in Machine Translation

The experiments in this section used the training set of IWSLT-2017 and
followed Setup 1, see Figure 1a. The main T5 model produced sequence-level
confidence or entropy uncertainty estimates under the reference sequence. The
NAP model was then trained to capture this uncertainty. We could have also
opted to generate sequence-level uncertainties using Setup 2 (see Figure 1b)
but the quality of the uncertainties then depends on the quality of the
decoded hypotheses. If we work with unlabelled datasets, we can always revert
back to Setup 2 and train our proxy to imitate the uncertainties of the free-
running hypotheses.

The performance of the uncertainty estimation NAP was then compared to the
main model in two ways. We first computed the Spearman Rank correlation
between the NAP output and the main model which was given the reference
output. The second and more important evaluation was based on out-of-
distribution detection. For this task, we took one in-domain dataset
(IWSLT-2017 test set) and compared it with one of the out-of-distribution
datasets mentioned above. We sought low uncertainties for the ID dataset and
high uncertainties for the OOD dataset. We used the AUROC [35] metric for
measuring detection performance, where 50% represents a fully random system.

####  A.1.6 Estimating BERTScores in Machine Translation

We decoded a finetuned T5 Large system (with a beam of $B=12$ and length-
penalty of 0.60) on the IWSLT-2017 training set. The decoded outputs were used
to compute the BERTScore for each instance, following Setup 2. The NAP was
then trained using the exact same hyperparameters as the above section.

Similar to the section above, the outputs of the NAP were first compared with
the main model on several unseen datasets. Following, we evaluated the
performance of this system on two downstream tasks. First, we took a dataset
and filtered out samples with the lowest estimated BERTScore and computed the
average BERTScore of the remaining samples. For a well-performing metric, we
expect the average BERTScore of the remaining samples to increase
monotonically.

Next, we also performed a resource optimization task in which we used the NAP
output to decide whether an input should be passed to a smaller (T5 Small) or
larger more robust (T5 Large) system. When a proxy output is above a
threshold, the input was passed to a smaller system and otherwise to the
slower and larger system. The threshold therefore had a large impact on the
performance and inference speed of the two model system. By selecting
different thresholds, different operating points were achieved. A good system
would achieve better performance while deferring as few samples as possible to
the slower system.

Furthermore, we also train a NAP to predict the BERTScore difference between
the two models in the deferral system. This can be motivated by a simple
example: Consider two different models, a smaller $\mathcal{M}_{1}$ and a
larger more robust $\mathcal{M}_{2}$. Given two different inputs $\bm{x}_{1}$
and $\bm{x}_{2}$ the two models achieve the following BERTScores:

Table 11: Simple example.

$\mathcal{M}_{1}$ $\mathcal{M}_{2}$ $\mathcal{M}_{2}-\mathcal{M}_{1}$
$\bm{x}_{1}$ 0.70 0.90 0.20 $\bm{x}_{2}$ 0.50 0.40 -0.10

Clearly, the first input is easier to handle since both models achieve higher
BERTScores with $\mathcal{M}_{2}$ being stronger. If we performed an
allocation based on the isolated performance of a single model itself, we
would give the simpler example $\bm{x}_{1}$ to the smaller model
$\mathcal{M}_{1}$ and the harder input $\bm{x}_{2}$ to the larger model
achieving an average performance of 0.55 BERTScore. However, if we instead
perform an allocation based on the performance difference, and refer samples
to the stronger model $\mathcal{M}_{2}$ where it dominates (and vice versa),
we would allocate $\bm{x}_{1}$ to model $\mathcal{M}_{2}$ and $\bm{x}_{2}$ to
model $\mathcal{M}_{1}$ achieving an average score of 0.70. This shows that an
allocation system should focus on the performance difference of the relevant
metric.

###  A.2 Automatic Speech Recognition

####  A.2.1 Datasets

Table 12 includes information about the LibriSpeech corpus [37]. The number of
words per sequence is computed based on the Whisper text normalization scheme.
In this task, we do not finetune the ASR models and do not use any out-of-
domain datasets. Instead, we focus on the noisy validation.other and
test.other sets.

Table 12: Dataset statistics.

Dataset #Sequences #Words per Sequence train.clean.100 28,539 35.0
train.clean.360 104,014 34.8 train.other.500 148,688 32.7 validation.clean
2,703 20.3 validation.other 2,864 18.0 test.clean 2,620 20.2 test.other 2,939
18.0

####  A.2.2 Models

In Table 13 we report parameter counts of various models. Whisper is an
encoder-decoder model with a language model head which predicts a probability
mass function over every token in the output sequence. The proxy model
consists of a Whisper encoder and a head for predicting uncertainty. The
parameter counts below are reported for a NAP with an average pooling layer;
an attentive pooling layer would add some parameters.

Table 13: Parameter counts of models. NAPs do not use a decoder during
inference.

Model Encoder Decoder Head Total Whisper Small 88.1M 153.6M 39.8M 241.7M NAP
Small - 14.2M 102.3M Whisper Medium 307.2M 456.6M 53.1M 763.9M NAP Medium -
25.2M 332.4M Whisper Large-v2 636.8M 906.5M 66.4M 1543.3M NAP Large-v2 - 39.3M
676.1M

####  A.2.3 Training Non-Autoregressive Proxies

We generated sentence-level word error rates (WERs) from the Whisper Large-V2
model using greedy search. While it was found that a beam of $B=5$ was the
best-performing setting in the original work [40], this was only achieved
using a highly non-standard decoding mechanism; simply using beam search with
$B=5$ actually degrades performance. Therefore, we opted for a simpler setup
using greedy search, see Table 14.

Table 14: Baseline %WER performance with greedy decoding.

Dataset Small Medium Large-v2 validation.clean 3.70 2.69 2.48 validation.other
7.35 5.46 4.96 test.clean 3.45 2.88 2.87 test.other 7.62 6.26 5.16

When generating the sentence WERs on the training data of the LibriSpeech
corpus, it was found that approximately half of all instances were correctly
decoded. This would present problems for a ranking loss and we instead opted
to train all NAP models using the Pearson correlation loss. Similar to the
section above, all experiments used a learning rate of 0.0001, maximised batch
size and training was stopped when performance did not improve after an epoch.

###  A.3 Estimating WERs in Automatic Speech Recognition

Following the exact same line of experiments as in Section A.1.6. A NAP was
trained to imitate the sentence-level WERs and was evaluated on two downstream
tasks, filtering and resource allocation. Note that we train additional proxy
systems to capture the total number of errors (instead of the error rate)
since this is more aligned with the resource allocation task. The resource
allocation was done between the Whisper Large-V2 and Whisper Small models.

We are unable to train a system to capture the error difference for the
resource allocation task since training the NAP was unstable. Approximately
74% of all error differences on the training set were 0 making it a highly
imbalanced dataset.

##  Appendix B Ablation Studies

We run all of our ablation studies on capturing mutual information of a T5
Large ensemble on the machine translation task. The ensemble consists of three
members.

###  B.1 Choice of Loss Function

All of the experiments in the main paper used a differentiable Spearman
correlation coefficient (scc) loss. This section explores alternative loss
functions including mean absolute error (mae), root mean squared error (rmse)
and pearson correlation coefficient (pcc).

Table 15: Detection performance of NAPs using MI $\mathcal{I}$. Split | Dataset | NAP Large
---|---|---
mae | rmse | pcc | scc
OOD-1 | newstest-19 | 67.3 | 66.9 | 69.6 | 70.5
newstest-20 | 74.9 | 73.6 | 76.0 | 78.1
OOD-2 | khresmoi-dev | 77.9 | 78.2 | 79.1 | 77.9
khresmoi-test | 80.5 | 81.0 | 81.5 | 81.2
OOD-3 | mtnt-2019 | 69.5 | 71.4 | 73.4 | 71.4
kftt | 50.2 | 50.2 | 52.8 | 54.7
| average | 70.1 | 70.2 | 72.1 | 72.3

The correlation-based loss functions are consistently better than mean
absolute and root mean squared error losses, possibly because the correlation
losses do not require accurate prediction of the uncertainties, only their
ordering.

###  B.2 Predictor Architecture

We also investigate the architecture, and specifically the activations of the
MLP that is added on top of the NAP encoder, see Figure 7. In a toy example,
we found that a two-layer (with tanh activation) network is better able to
predict entropy scores from categorical predictions. This motivates using a
three-layer network with an initial softmax activation to produce ’virtual’
probabilities.

(a) Standard.

(b) With attentive pooling.

Figure 7: The standard three-layer network is used on top of a non-
autoregressive proxy. When average pooling the encoder output is restrictive,
an attention layer is used instead with a trainable query.

This section also explores a range of different (parameter-matched) two-layer
and three-layer MLPs with various activation functions, see Figure 8.

(a) From left to right: {2L Tanh, 2L SM, 2L LN-Exp & 2L LN-Tanh}.

(b) From left to right: {3L ReLU, 3L Tanh, 3L LN-Exp & 3L LN-Tanh}.

Figure 8: Various configurations of proxy heads investigated.

Table 16 shows the performance of various MLPs (with average pooling) in the
out-of-distribution detection task. The two-layer and three-layer MLPs are
parameter matched. The final model 3L SM is the default MLP head used in all
experiments. Clearly, the use of a softmax activation is extremely important
for achieving the best possible performance.

Table 16: Detection performance of NAPs using MI $\mathcal{I}$.

Split Dataset NAP Large 2L 2L 2L 2L 3L 3L 3L 3L 3L Tanh SM LN-Exp LN-Tanh ReLU
Tanh LN-Exp LN-Tanh SM OOD-1 newstest-19 56.6 67.7 50.5 48.4 46.4 57.2 59.9
59.7 70.5 newstest-20 66.2 75.4 58.6 56.0 47.0 68.2 67.7 63.2 78.1 OOD-2
khresmoi-dev 55.6 77.5 66.4 49.8 39.2 52.8 65.1 59.1 77.9 khresmoi-test 56.0
80.6 67.4 51.8 38.9 53.8 65.2 62.2 81.2 OOD-3 mtnt-2019 54.1 71.6 48.4 52.6
63.4 47.8 61.4 50.6 71.4 kftt 55.2 50.4 55.9 52.0 43.0 62.0 58.1 44.8 54.7
average 57.3 70.5 57.9 51.8 46.3 56.9 62.9 56.6 72.3

###  B.3 Intermediate Outputs of Encoder

It is not necessary to pick the final layer output as the input to the
predictor MLP. One can use intermediate layer outputs as well. Previous work
has found that using intermediate outputs can even improve upon a task [23,
62]. Using intermediate layer outputs also leads to faster inference and lower
parameter counts, see Table 17.

Table 17: Parameter counts and inference time of models on iwslt-2017.

Layers Embeddings Encoder Head Total Inference Time Default 24L 32.9M 334.9M
20.9M 355.9M 17.9s 21L 32.9M 289.2M 20.9M 310.1M 15.3s 18L 32.9M 259.4M 20.9M
280.4M 12.7s 15L 32.9M 221.7M 20.9M 242.7M 9.9s 12L 32.9M 184.0M 20.9M 204.9M
7.5s

According to Table 18, the performance of NAPs remains arguably consistent
when utilizing intermediate outputs down until the 12th layer, where
performance starts dropping. Therefore, it is possible based on this
experiment to remove the top 9 layers of the T5 encoder reducing the total
parameter count by 32% and inference time by 45% without notably sacrificing
performance.

Table 18: Detection performance of NAPs using MI $\mathcal{I}$.

Split Dataset NAP Large 24L 21L 18L 15L 12L OOD-1 newstest-19 70.5 68.7 69.1
68.6 68.1 newstest-20 78.1 77.0 77.1 76.0 75.4 OOD-2 khresmoi-dev 77.9 78.5
77.2 77.0 76.4 khresmoi-test 81.2 81.2 80.3 80.2 80.1 OOD-3 mtnt-2019 71.4
70.0 70.9 72.8 70.6 kftt 54.7 48.9 54.5 56.0 48.8 average 72.3 70.7 71.5 71.8
69.9

###  B.4 Mismatched Pretrained Encoders

This section investigates if it is possible to use alternative mismatched
encoders as the backbone for a proxy system when predicting sequence-level
attributes for the T5 model. We, therefore, investigate replacing the T5
encoder with RoBERTa [32], XLM-RoBERTa [8] or the lightweight ALBERT [27]. See
Table 19 for information about the model size and inference time.

Table 19: Parameter counts and inference time of models on iwslt-2017.

Layers Embeddings Encoder Head Total Inference Time T5 Large Encoder 32.9M
334.9M 20.9M 355.9M 17.9s RoBERTa Base 39.0M 124.1M 11.8M 135.9M 4.3s RoBERTa
Large 52.0M 354.3M 20.9M 375.3M 17.5s XLM-RoBERTa Base 192.4M 277.5M 11.8M
289.3M 4.5s XLM-RoBERTa Large 256.5M 558.8M 20.9M 579.8M 19.2s ALBERT Base
3.9M 11.1M 11.8M 22.9M 4.8s ALBERT Large 3.9M 16.6M 20.9M 37.6M 19.4s

The detection performance of alternative backbones such as base RoBERTa and
base XLM-RoBERTa are slightly worse but with significantly lower inference
times. The large RoBERTa and XLM-RoBERTa are approximately as fast as the T5
Encoder-based proxy but only the latter achieves similar detection
performance. The lightweight ALBERT pretrained backbone significantly suffers
at this task.

Table 20: Detection performance of NAPs using MI $\mathcal{I}$.

Split Dataset T5 Encoder RoBERTa XLM-RoBERTa ALBERT Large Base Large Base
Large Base Large OOD-1 newstest-19 70.5 64.3 62.6 68.8 69.3 60.8 63.2
newstest-20 78.1 72.0 69.1 76.8 77.4 67.9 68.0 OOD-2 khresmoi-dev 77.9 78.7
77.2 69.2 80.0 73.2 71.0 khresmoi-test 81.2 81.9 78.0 72.1 83.0 75.8 74.2
OOD-3 mtnt-2019 71.4 61.6 62.1 61.7 61.6 63.5 68.3 kftt 54.7 61.7 62.1 62.6
62.3 51.4 43.0 average 72.3 70.1 68.5 68.6 72.3 65.4 64.6

###  B.5 Decorrelating Epistemic and Aleatoric Uncertainty

Epistemic and aleatoric uncertainties are of different natures. The former is
a measure of the lack of knowledge in our model parameters and model choice
under the given dataset. As the dataset increases the epistemic uncertainty
should decrease. The latter is an intrinsic measure of uncertainty in the data
itself which might be caused by noisy data collection methods or labelling
errors. Therefore, we propose a new loss function in which we aim to maximise
the correlation between the proxy outputs $\\{\hat{s}_{i}\\}_{i}$ and teacher
sequence-level epistemic scores $\\{{s}_{ei}\\}_{i}$ whilst also decorrelating
its outputs from teacher sequence-level aleatoric scores $\\{{s}_{ai}\\}_{i}$:

$\displaystyle\mathcal{L}_{\tt ep-al}=\mathcal{L}_{\tt
scc}\Big{(}\\{\hat{s}_{i}\\},\\{{s}_{ei}\\}\Big{)}-\alpha\Big{\lvert}{\mathcal{L}_{\tt
scc}\Big{(}\\{\hat{s}_{i}\\},\\{{s}_{ai}\\}\Big{)}}\Big{\rvert}$ (2)

where $\alpha$ controls the level of decorrelation. Table 21 shows that by
using this style of loss function, the proxy can be made to perform
significantly better. The base model $\alpha=0.0$ already outperforms a deep
ensemble at detection, and furthermore, setting $\alpha=1.0$ shows even better
overall performance.

Table 21: Detection performance of NAPs using MI $\mathcal{I}$.

Split Dataset NAP Large $\alpha=0.0$ $0.5$ $1.0$ $2.0$ OOD-1 newstest-19 70.5
76.1 76.0 75.3 newstest-20 78.1 85.9 86.3 84.0 OOD-2 khresmoi-dev 77.9 86.1
88.0 83.5 khresmoi-test 81.2 86.8 87.7 83.3 OOD-3 mtnt-2019 71.4 61.7 57.3
51.1 kftt 54.7 70.2 76.5 77.9 average 72.3 77.8 78.6 75.9