# RMBR: A Regularized Minimum Bayes Risk Reranking Framework
for Machine Translation

Yidan Zhang1 Yu Wan2 Dayiheng Liu1 Baosong Yang2 Zhenan He1 1College of
Computer Science, Sichuan University
2NLP2CT Lab, University of Macau
losinuris@gmail

###### Abstract

Beam search is the most widely used decoding method for neural machine
translation (NMT). In practice, the top-1 candidate with the highest log-
probability among the $n$ candidates is selected as the ‘preferred’ one.
However, this top-1 candidate may not be the best overall translation among
the $n$-best list. Recently, Minimum Bayes Risk (MBR) decoding has been
proposed to improve the quality for NMT, which seeks for a consensus
translation that is closest on average to other candidates from the $n$-best
list. We argue that MBR still suffers from the following problems: The utility
function only considers the lexical-level similarity between candidates; The
expected utility considers the entire $n$-best list which is time-consuming
and inadequate candidates in the tail list may hurt the performance; Only the
relationship between candidates is considered. To solve these issues, we
design a regularized MBR reranking framework (RMBR), which considers semantic-
based similarity and computes the expected utility for each candidate by
truncating the list. We expect the proposed framework to further consider the
translation quality and model uncertainty of each candidate. Thus the proposed
quality regularizer and uncertainty regularizer are incorporated into the
framework. Extensive experiments on multiple translation tasks demonstrate the
effectiveness of our method.

##  1 Introduction

Given a source sentence, neural machine translation (NMT) Sutskever et al.
(2014) models are trained to predict conditional probability distributions for
candidate translations. In practice, it is desirable to output a single
sentence, not a distribution. Therefore, a decision rule is required to rank
the candidates and select the ‘preferred’ one. The most widely used decision
rule is maximum-a-posteriori (MAP) decoding, which seeks the most probable
translation under the conditional distribution. Due to the huge search space,
beam search is proposed as an approximation. Given a pre-defined beam size
$n$, beam search always keeps the top-$n$ candidates based on the log-
probability score. Then, the top-1 candidate, i.e., the one with the highest
log-probability among the $n$-best list, is selected as the ‘preferred’ one.
Unfortunately, this top-1 candidate might not be the best translation on the
$n$-best list.

We conduct oracle experiments to explore the performance gap between the
oracle result111The oracle result is defined as $\text{argmax}_{Y\sim
p_{\text{NMT}(Y|X)}}$ BLEU$(Y,Y^{\prime})$, where $(X,Y^{\prime})$ is the pair
of source and reference sentence. in the $n$-best candidates and top-1
candidate. Besides using beam search, we further use three stochastic
decodings (ancestral search (AS) Fu et al. (2021), top-$k$ Fan et al. (2018),
top-$p$ Holtzman et al. (2020)), and two deterministic decodings (diverse beam
search (DBS) Vijayakumar et al. (2016), sibling beam search (SBS) Li et al.
(2016)) to obtain $n$ candidates, respectively. The results are reported in
Fig. 1a. Overall, all of the oracle results achieve significantly higher BLEU
scores than the top-1 candidate of beam search with beam size 5. Furthermore,
we observe that under the oracle experiment, using beam search to obtain
$n$-best candidates still outperforms other decoding methods. These results
suggest that beam search actually performs well, yet log-probability scores
fail to select the best translation from the $n$-best list. Similar to our
study, Blain et al. Blain et al. (2017) has observed that NMT model is capable
of outputting high-quality candidate translations, but fails at picking them
as the best one. Leblond et al. Leblond et al. (2021) also points out that,
NMT models are good at spreading probability mass over a large number of
acceptable outputs, but they are not efficient at selecting the best one.

Figure 1: An example of exploring candidate spaces on the IWSLT’14 De$\to$En
test set. (a) Oracle ranking of samples generated by multiple decoding
strategies. (b) The token probabilities of sentences in different length
intervals. The x-axis is the length interval, and the y-axis is the average
token probability of the sentences within the same length range. (c) The
distribution of oracle translations’ rank index in the $n$-best list. The
x-axis represents the index interval, and the y-axis represents the proportion
of oracle translations indexed in an interval.

To further explore why the top-1 candidate is not the best translation, we
compare the token probability between top-1 candidates and references.
Specifically, the average probability of all the tokens in each sentence is
firstly computed, which is defined as the token probability. To eliminate the
effect of sentence length, the mean token probability of all candidates in the
same length range is observed. As shown in Fig. 1b, we find that the token
probability of top-1 candidates is much higher than that of references,
especially when the result length is longer, suggesting that NMT models may
over-confident about the top-1 candidates. During beam search decoding,
assigning an excessively high probability to a suboptimal sequence in one step
can lead to a chain reaction that eventually produces an unnatural candidate
with high probability. Besides, we argue that the essence of the beam search
curse Meister et al. (2020) (large beam sizes hurt translation quality) is
lying in the token probability gap between top-1 candidates and reference
translations, as larger beam sizes lead to larger gaps from Fig. 1b.

In view of the above analysis, we expect to find a consensus candidate from
the $n$-best list to avoid the “over-confident” candidates. Recently, a new
decision rule, Minimum Bayesian Risk (MBR) decoding, has been proposed in NMT.
The main idea of this method is to find the translation that is closest to
other candidate translations to minimize the expected risk for a given utility
function. In Shu and Nakayama Shu and Nakayama (2017) and Blain et al. Blain
et al. (2017), MBR decoding are combined with beam search to improve the
translation quality. Nevertheless, we argue that there are still some defects
in MBR decoding: (a) The utility function only considers the lexical-based
similarity between candidates, such as BLEU, METEOR, CHRF etc.; (b) The
expected utility for each candidates considers the entire $n$-best list, which
requires a large computational cost, especially when $n$ is large. Besides,
inadequate candidates in the tail list may hurt the performance; (c) MBR only
considers the similarity between candidates but completely ignore the model
uncertainty and the translation quality of each candidate.

To solve above issues, we propose a Regularized Mmini-mum Bayesian Risk
reranking framework (RMBR). For the first problem, we explore the use of
semantic-based evaluation metrics (e.g., COMET Rei et al. (2020) and BLEURT
Sellam et al. (2020)) as the utility function for MBR. Aiming at the second
issue, we conduct experiment to analyze the probability ranking of the oracle
translations in the $n$-best list ($n$=30). As shown in Fig. 1c, the oracle
translations are less likely to appear in the tail list. Therefore, we use
only the top-$l$ $(l\leq n)$ candidates of the $n$-best list to calculate the
MBR score (expected utility) for each candidate. In this way, the
computational cost is reduced and the inadequate candidates in the tail list
that is close to each other, are avoided. For the third problem, we
incorporate two types of regularizers into the framework: quality regularizer
and uncertainty regularizer. Quality regularizer allows RMBR framework to
further consider the translation quality of a single candidate in addition to
considering the similarity between candidate results. To be concrete, we
consider four regularization scores as the quality regularizer: language model
score, back-translation score Rapp (2009), quality estimation score, and
translation score (log-probability score). While the uncertainty regularizer
aims to further consider the model uncertainty for each output. In this paper,
we explore two kinds of uncertainty regularizers: Monte Carlo (MC) Dropout
Wang et al. (2019); Gal and Ghahramani (2016) and the entropy of model output
distributions.

We conduct extensive experiments to compare different settings of RMBR, as
well as the previous MBR method Shu and Nakayama (2017); Blain et al. (2017)
using BLEU as utility and several commonly used translation reranking methods.
Experimental results show that after using COMET as utility function, our MBR
outperforms previous MBR decoding methods Shu and Nakayama (2017); Blain et
al. (2017). When the proposed quality regularizer or uncertainty regularizer
is further introduced, the performance of RMBR can be further improved. Our
method achieves consistent performance gains on the tasks of German-English
from IWSLT’14, and German-English, English-German, and English-French tasks
from WMT’14, which demonstrates the effectiveness of our method.

##  2 Preliminary

###  2.1 The Decoding Problem

Let $X=\\{x_{1},x_{2},...,x_{|X|}\\}$ denote a source sequence,
$Y=\\{y_{1},y_{2},...,y_{|Y|}\\}$ denote a target sequence. A NMT model
defines a distribution over outputs and sequentially predicts tokens using a
softmax function as follows:

$p(Y|X)=\prod_{t=1}^{|Y|}p\textsubscript{NMT}(y_{t}|X,y_{1},y_{2},...,y_{t-1}).$
(1)

The decoding problem can be written as finding a sequence $Y^{\ast}$ that
maximizes the probability given input $X$:

$Y^{\ast}=\mathop{\arg\max}\limits_{X}p\textsubscript{NMT}(Y^{\ast}|X).$ (2)

###  2.2 Beam Search

When decoding with the above distribution over sequences, it is not feasible
to pick out the most probable sequence among all possible sequences. A common
approximate decoding method is beam search, which maintains the top-$n$ highly
scoring candidates at each time step. $n$ is known as beam size, and the log-
probability of a sequence at time $t$ is computed as:

$S(Y_{t}|X)=S(Y_{t-1}|X)+\text{log }p\textsubscript{NMT}(y_{t}|X,Y_{t-1}),$
(3)

where $S(Y_{t-1}|X)=\text{log
}p\textsubscript{NMT}(y_{1},y_{2},...,y_{t-1}|X)$. The decoding process is
repeated until the stop condition is met. After that, we can obtain a list of
$n$ most promising candidates. Finally, the most likely sequence is selected
as the ‘preferred’ translation by ranking the $n$ candidates based on log-
probability scores $S(Y|X)$.

##  3 Regularized MBR Reranking Framework

As discussed in Sec §1, picking the candidate with the highest log-probability
score is unable to effectively obtain the best result. In this paper, we
propose a regularized MBR reranking framework (RMBR) that adopts the semantic
similarity evaluation metric as the utility function. Besides considering the
similarity between the output candidates, we expect the proposed framework to
further consider the translation quality of each candidate and the uncertainty
of the model. Thus we incorporate two types of regularizers into the
framework: Quality Regularizer (Sec §3.2) and Uncertainty Regularizer (Sec
§3.3). The candidate with the highest reranked score is formally defined as
the 1-best candidate.

Given a list of $n$ most likely candidates generate by beam search with beam
size $n$, which can be written as $\\{H_{1},H_{2},...,H_{n}\\}$. The
regularized score for $H_{i}$ is computed as:

$S\textsubscript{RMBR}(H_{i})=S\textsubscript{MBR}(H_{i})+\sum\lambda_{j}\mathcal{R}_{j}(H_{i}),$
(4)

where $S\textsubscript{MBR}$ is the MBR score, which is introduced in the next
section. Note that we introduce two types of regularizers, $\mathcal{R}_{j}$
is used to denote the $j$-th regularizer score. $\lambda_{j}$ is a tradeoff
parameter222$\lambda_{j}$ is selected from the set {0.001, 0.01, 0.1, 1, 10}
with the best performance on the validation set. to achieve a satisfying
balance among multiple decoding objectives. Finally, the 1-best candidate is
selected as the ‘preferred’ translation.

###  3.1 MBR Score

Given a utility function $\mathcal{U}$ (e.g., BLEU) and a list of $n$-best
candidates. The MBR score (expected utility) for each candidate is computed by
comparing it to all other candidates in the $n$-best list. Since only a few
oracle translations appear at the tail list as we observed in preliminary
experiment, we compute the MBR score for $H_{i}$ by comparing it to top-$l$
candidates:

$S\textsubscript{MBR}(H_{i})=\frac{1}{l}\sum_{j=1}^{l}\mathcal{U}(H_{i},H_{j}),$
(5)

where $l\in\\{1,2,...,n\\}$ is tuned on the validation set and fixed for
inference for all testing instances. The candidate with the highest MBR score
$S\textsubscript{MBR}$ is the consensus translation in the $n$ candidates.
Besides using lexical-based method (BLEU) as utility function $\mathcal{U}$
which is called MBRBLEU, we further explore two semantic-based evaluation
methods BLEURT and COMET as utility functions $\mathcal{U}$ in our framework,
which are called MBRBLEURT and MBRCOMET, respectively.

###  3.2 Quality Regularizer

MBR score only considers the similarity between the output candidates and
ignores the translation quality of each candidate. To bridge this gap, we
introduce a quality regularizer into MBR framework. In this work, we explore
four kinds of scores as the quality regularizer: a) Language Model (LM) score;
b) Back-Translation (BT) score; c) Quality Estimation (QE) score; and d) log-
probability scores. The computation for candidate $H_{i}$ is as follows:

$\displaystyle\text{LM}(H_{i})$
$\displaystyle=\text{log}p\textsubscript{LM}(H_{i}),\text{BT}(H_{i})=\text{log}p\textsubscript{NMT}(X|H_{i}),$
(6) $\displaystyle\text{QE}(H_{i})$
$\displaystyle=f\textsubscript{QE}(X,H_{i}),$ (7)

where log$p\textsubscript{LM}(H_{i})$ is calculated by a pre-trained language
model, $p\textsubscript{NMT}(X|H_{i})$ is via a backward NMT model, and
$f\textsubscript{QE}(X,H_{i})$ is by a off-the-shelf quality estimation model
(e.g., TransQuest Ranasinghe et al. (2020)).

###  3.3 Uncertainty Regularizer

In this section, we introduce the uncertainty regularizer, which quantifies
whether the current model is confident or hesitant on the candidate
translation. For efficiency, we utilize widely used Monte Carlo (MC) dropout
and entropy measures to compute model uncertainty.

MC Dropout. At test time, for a candidate $H_{i}$ paired with input $X$, we
perform $m$ forward passes through the NMT model parameterized by
$\hat{\theta}$, where the $t$-th pass randomly deactivates part of neurons.
Then, $m$ sets of sentence-level perturbed log-probability score are
collected, which is written as:

$\text{MC}_{\hat{\theta}_{t}}(H_{i})=-\text{log
}p\textsubscript{NMT}(H_{i}|X,\hat{\theta}_{t}).$ (8)

Entropy Measures. We also consider using the entropy of model predicting
probability distribution of each candidate as a measure of model uncertainty.
Intuitively, given an output sample, if the model probability distribution
entropy of each token is very small, it means that the model has a high degree
of confidence in this output result. Let
$\mathcal{V}=\\{v_{1},v_{2},...,v_{|V|}\\}$ denote the target vocabulary of
NMT, we compute the token entropy for each token in the candidate
$H_{i}=\\{h_{i1},h_{i2},...,h_{i|H_{i}|}\\}$. Then $|H_{i}|$ sets of token
entropy are collected, which is written as:

$S\textsubscript{entropy}(h_{it})=-\sum_{j=1}^{|\mathcal{V}|}\text{log}p\textsubscript{NMT}(v_{j}|X,h_{i1},...,h_{it-1}).$
(9)

Finally, the expectation of $m$ sets of $\text{MC}_{\hat{\theta}_{m}}(H_{i})$
and $|H_{i}|$ sets of $S\textsubscript{entropy}(h_{it})$ are used as the
uncertainty regularizer score.

| IWSLT’14 De$\to$En | WMT’14 De$\to$En
---|---|---
Method | COMET | BLEURT | BLEU | COMET | BLEURT | BLEU
Top-1 (beam=5) | 34.79 | 16.16 | 34.28 | 42.35 | 21.90 | 32.70
Top-1 (beam=30) | 34.22 | 15.99 | 34.17 | 41.80 | 21.60 | 32.54
LP+BT Rapp (2009) | 40.63 | 18.57 | 35.11 | 45.94 | 23.42 | 33.06
LP+QE Ranasinghe et al. (2020) | 38.84 | 19.53 | 35.37 | 45.56 | 24.30 | 33.41
LP+LM Radford et al. (2019) | 36.33 | 16.58 | 35.14 | 44.48 | 22.48 | 33.49
Range Voting Borgeaud and Emerson (2020) | 34.89 | 16.59 | 34.53 | 42.29 | 21.53 | 32.78
MBRBLEU(full) Blain et al. (2017) | 33.76 | 15.91 | 34.38 | 41.66 | 20.96 | 32.68
MBRBLEU | 34.39 | 16.39 | 34.54 | 42.53 | 22.03 | 32.83
MBRBLEURT | 33.10 | 22.00 | 33.01 | 42.71 | 25.31 | 32.45
MBRCOMET | 42.53 | 17.78 | 34.55 | 47.10 | 23.06 | 32.93
MBRCOMET+LP | 41.60 | 17.89 | 34.91 | 46.69 | 22.89 | 33.08
MBRCOMET+LP+BT | 43.64 | 18.86 | 35.24 | 47.67 | 23.57 | 33.17
MBRCOMET+LP+QE | 42.04 | 19.96 | 35.62 | 46.89 | 23.57 | 33.76
MBRCOMET+LP+LM | 41.75 | 18.40 | 35.49 | 47.56 | 23.91 | 33.85
MBRCOMET+LP+entropy | 42.04 | 18.34 | 35.24 | 46.24 | 22.99 | 33.16
MBRCOMET+LP+dropout | 41.47 | 17.90 | 34.95 | 47.43 | 22.91 | 33.10
MBRCOMET+LP+QE+LM | 42.24 | 20.60 | 36.19 | 47.34 | 25.18 | 34.29

Table 1: BLEU, COMET, and BLEURT score comparison. All candidates are obtained
by beam search.

##  4 Experiments

###  4.1 Experimental Settings

In this section, we describe the datasets, NMT models, and metrics used in our
experiments to investigate the effect of the proposed reranking methods on the
$n$-best candidate list.

####  4.1.1 Datasets and Models

To implement the NMT task, we use the German-English (De$\to$En) from IWSLT’14
task, German-English (De$\to$En), English-German (En$\to$De), and English-
French (En$\to$Fr) from the WMT’14 translation task. For IWSLT’14 task, we use
the data pre-processing scripts and hyperparameter settings provided by
fairseq NMT repository333
https://github.com/pytorch/fairseq/tree/master/examples/translation.. For
WMT’14 task, we train a Transformer base model Vaswani et al. (2017) as the
base NMT model and use the Newstest’14 dataset as the test set.

####  4.1.2 Evaluation Metrics

In our experiments, three widely used automatic evaluation metrics are
utilized to evaluate the machine translation: BLEU, an n-gram-based precision
metric which measures the lexical similarly between translation and reference;
COMET Rei et al. (2020), a multilingual and adaptable MT evaluation model,
which exploits information from both source sentence and target sentence to
measures the semantic similarity between translation and reference; and BLEURT
Sellam et al. (2020), a learned evaluation metric based on BERT, which
measures the semantic similarity between two sequences.

###  4.2 Baselines

We take the top-1 results of the beam search with beam size 5 as the baseline,
which is the most widely used setting of NMT models. For all reranking
methods, we follow previous work Eikema and Aziz (2020) using beam search with
beam size 30 to generate the candidates (experimental results with varying
beam size and different decoding method can be found in Sec §5.4 and Sec §5.3,
respectively). MBRCOMET denotes use only MBR score to rank the candidate
without any regularizer, where COMET is used as the utility function. Besides,
we also compare MBRBLEU and MBRBLEURT which use BLEURT and BLEU as utility
function, respectively. We further compare the performance of introducing
different regularizer on MBRCOMET, including four kinds of quality regularizer
scores: log-probability (LP) score, language model (LM) score, back-
translation (BT) score, quality estimation (QE) score, and two uncertainty
regularizer scores: entropy score and MC-dropout score. We use GPT-2base model
Radford et al. (2019) to calculate LM score. BT score and QE score is computed
via backward NMT models and TransQuest Ranasinghe et al. (2020), respectively.
For the proposed method, we compute MBR score for each candidate by comparing
it to partial top candidates, where the details are reported in Appendix A. We
also compare the method Range Voting Borgeaud and Emerson (2020) and
MBRBLEU(full) Blain et al. (2017), which using BLEU as utility function of
MBR. The only difference between MBRBLEU(full) Blain et al. (2017) and our
MBRBLEU is that MBRBLEU(full) uses all candidates to calculate MBR score.

###  4.3 Results

We first report the results on IWSLT’14 De$\to$En and WMT’14 De$\to$En tasks.
From Table 1, we can see that MBRCOMET performs better than MBRBLEU, top-1,
and other baselines on all three evaluation metrics. Interestingly, we find
that MBRBLEURT achieves the highest BLEURT score but low BLEU and COMET
scores. To find out which utility function is the best, we further perform
human evaluation (see Sec §5.1) to more quantitatively compare the reranked
1-best candidates. The human evaluation results show that MBRCOMET outperforms
MBRBLEU and MBRBLEURT, demonstrating that semantic-based MBR outperforms
traditional lexical-based MBR. For the proposed regularizers, it can be found
that MBRCOMET$+$LP significantly improves the scores in BLEU comparing to
MBRCOMET. Besides, MBRCOMET$+$LP can be further improved in three metrics by
adding other regularizers. For example, the MBRCOMET$+$LP$+$QE achieves higher
scores on BLEU, COMET, and BLEURT. In addition, a similar trend is observed in
MBRBLEURT and MBRCOMET. More results and details can be found in Appendix B.
The regularized MBR reranking has a significant improvement over the results
of beam search with sizes 5 and 30, bringing 8 points and 1.5 points of
improvement on COMET and BLEU metrics, respectively.

We additionally explore the performance of combining more regularizers on
MBRCOMET. We collectively tune the $\lambda$ value for each of the
regularizers on validation sets. We observe the results of
MBRCOMET$+$LP$+$QE$+$LM (we use RMBRCOMET to denote this setting latter) that
achieves the highest BLEU score among all the combinations, improving the BLEU
score more than 2 points. We also find that combining quality and uncertainty
regularizers with MBRCOMET can not lead to further performance gains.

| WMT’14 En$\to$De | WMT’14 En$\to$Fr
---|---|---
Methods | COMET | BLEU | COMET | BLEU
Top-1 (beam=5) | 27.24 | 27.09 | 55.11 | 38.74
Top-1 (beam=30) | 20.32 | 26.50 | 50.31 | 38.22
LP+QE | 28.10 | 27.80 | 55.39 | 39.60
LP+LM | 27.92 | 28.04 | 56.10 | 39.62
LP+BT | 27.50 | 27.75 | 56.06 | 39.70
MBRCOMET | 34.25 | 27.37 | 59.85 | 39.18
MBRBLEU | 26.15 | 27.30 | 53.81 | 39.17
MBRCOMET+LP | 31.98 | 27.93 | 57.88 | 39.58
MBRCOMET+LP+LM | 34.97 | 28.19 | 59.80 | 39.87
MBRCOMET+LP+QE | 32.71 | 28.00 | 59.83 | 39.84
MBRCOMET+LP+BT | 32.53 | 28.01 | 60.33 | 39.83
MBRCOMET+LP+QE+LM | 32.51 | 28.40 | 59.71 | 40.15

Table 2: BLEU and COMET score comparison on WMT’14 En$\to$De and WMT’14
En$\to$Fr tasks.

##  5 Analysis

Method | Score
---|---
MBRCOMET | 0.281
MBRBLEURT | 0.129
MBRBLEU | 0.125
Top-1 (beam=5) | 0.120

Table 3: Results of the human evaluation. The score column represents the
percentage of time each reranking method is judged better across its
comparisons.

###  5.1 Human Evaluation

From the previous results, we observe that MBRCOMET outperforms MBRBLEU and
MBRBLEURT in BLEU and COMET metrics, but not in BLEURT metric. This motivated
us to perform human evaluation to more quantitatively compare the reranked
results. For human evaluation, we randomly select a subset of 500 source
sentences from the test sets of IWSLT’14 De$\to$En. Reranking is also based on
the beam search results of beam size 30. We request 3 human annotators to rank
the four translations from the best to the worst. Table 3 reports the ranking
results according to the Expected Wins method Sakaguchi et al. (2014). Our
observation is that the 1-best candidates reranking by MBRCOMET outperforms
the other three methods. We provide some examples in Appendix C.

###  5.2 Multilingual Settings

To further verify the effectiveness of the proposed model on non-English
target translation tasks, we conduct experiments on WMT’14 En$\to$Fr and
En$\to$De, where we follow the same settings in Sec §4.2. Since the evaluation
metric BLEURT only supports evaluation the language of English, we only report
BLEU and COMET scores for En$\to$Fr and En$\to$De tasks. The results are shown
in Table 2, which is consistent with the conclusion in Table 1.

Methods | COMET | BLEURT | BLEU
---|---|---|---
Beam Search (beam=30)
Top-1 (beam=30) | 34.22 | 15.99 | 34.17
MBRCOMET | 42.53 | 17.78 | 34.55
MBRBLEU | 34.39 | 16.39 | 34.54
MBRBLEURT | 33.10 | 22.00 | 33.01
Siblings Beam Search (beam=30)
Top-1 ($n=30$) | 34.11 | 15.67 | 34.09
MBRCOMET | 41.44 | 17.16 | 34.39
MBRBLEU | 33.83 | 16.04 | 34.42
MBRBLEURT | 31.78 | 21.68 | 32.95
Ancestral Sampling ($n$=30)
Top-1 ($n=30$) | 21.37 | 10.62 | 29.33
MBRCOMET | 30.44 | 13.71 | 28.27
MBRBLEU | 9.67 | 8.99 | 30.62
MBRBLEURT | 9.12 | 19.74 | 22.81

Table 4: The reranking results from 30 candidates decoded by beam search, SBS,
and AS on the test sets of IWSLT’14 De$\to$En.

###  5.3 Diverse Candidate Spaces

From the oracle experiments (see Fig.1a), we observe that deterministic
decoding performs better than stochastic decoding, and sibling beam search
(SBS) performs as well as beam search. To further explore the effect of
diverse candidate spaces, we rerank the 30 top candidates by SBS and 30
candidates sampled by AS. As shown in Table 4, the reranking results of the
candidates decoded by SB perform slightly worse than that of beam search. For
AS decoding, the scores of both top-1 candidates and reranked 1-best
candidates are significantly low compared to other reranking methods.

###  5.4 Effect of Beam Size

To evaluate the effectiveness of larger beam size on our proposed method, we
use the RMBRCOMET to rank the candidates, which performs best on average of
three metrics on the IWSLT’14 De$\to$En test sets. More experimental results
are reported in Appendix D. From Fig. 2 we can see that with increased beam
sizes, there is a significant improvement for COMET, BLEURT, and BLEU scores.
The results suggest that our proposed reranking method can alleviate the beam
search curse and generate better translations as beam size increases.

Figure 2: The results of the 1-best candidates reranked by the RMBRCOMET using
beam of sizes 5, 30, and 50.

###  5.5 Inference Time

We further compare the inference time of the proposed reranking variants and
baseline. For reranking, we still use 30 candidates obtained by beam search on
the IWSLT’14 De$\to$En test sets. To compare the inference time, all
experiments are performs on single Tesla V100 16GB GPU. Note that, in practice
we can further reduce inference time by using more GPUs to compute utility
functions in parallel. The results are shown in Table 5. $n$ represents the
number of candidates used to rerank, $l$ represents the number of candidates
used to compute expected utility ($n=30,l_{1}=21,l_{2}=3$). For
RMBRCOMET(C2F), we follow the method Eikema and Aziz (2021) and use BLEU as
the proxy utility to select 15 candidates and then use COMET as the target
utility to select the 1-best candidate. From the results we can see that
RMBRCOMET(n-by-l1) achieves the best performance with about 2.5 times more
inference time than top-1 (beam=5). Both RMBRCOMET(n-by-l2) and RMBRCOMET(C2F)
can further reduce inference time and outperform the baseline, which can be
used as a trade-off between time cost and performance.

Methods | COMET | BLEURT | BLEU | Time
---|---|---|---|---
Top-1 (beam=5) | 34.79 | 16.16 | 34.28 | x1
RMBRCOMET(n-by-n) | 42.52 | 20.47 | 36.01 | x4.7
RMBRCOMET(n-by-l1) | 42.24 | 20.60 | 36.19 | x3.6
RMBRCOMET(n-by-l2) | 40.93 | 20.26 | 35.90 | x1.4
RMBRCOMET(C2F) | 41.50 | 19.41 | 35.93 | x1.9

Table 5: Comparison results of inference time. Reranking uses $n=30$
candidates per sample.

##  6 Related Work

In NMT, reranking is a way of improving translation quality by scoring and
selecting a ‘preferred’ translation from a list of candidates generated by a
source-to-target model. MBR decoding is one of effective method. The goal of
MBR decoding is to find a consensus translation that is closest to other
candidates. Some studies rerank the $n$ candidates directly sampled from the
model. Eikema and Aziz Eikema and Aziz (2020) is the first to use unbiased
samples from the model by ancestral sampling, to approximate hypotheses space.
Aiming at keeping computational cost of estimating expected utility tractable,
a coarse-to-fine MBR procedure is proposed in Eikema and Aziz Eikema and Aziz
(2021). Other studies tend to rerank the $n$ candidates decoded by beam
search. In Shu and Nakayama Shu and Nakayama (2017), both MBR scores and log-
probability scores are considered at each step of decoding. Blain et al. Blain
et al. (2017) investigates some automatic MT evaluation metrics (BLEU, BEER,
and CHRF), and observes that evaluation metric plays a major role in the
$n$-best reranking approach. Borgeaud and Emerson Borgeaud and Emerson (2020)
designs some similarity functions to make more informative candidates receive
stronger votes, thus selecting the most representative candidate.

These previous studies only use MBR score to rank each candidate without
considering source sentence and model score. In the proposed RMBR, some
regularizers are utilized to rank candidates in an overall way. Different from
previous works which select candidates based on only lexical similarity, we
also explore the semantic similarity between candidates. The other difference
is that MBR score is computed using top-$l$ candidates of the $n$-best list to
avoid candidates with poor quality in the tail list and reduce the computation
cost.

Besides MBR, there are some studies focus on MT reranking. For example, Ng et
al. Ng et al. (2019) describes using language model to rank candidates. In
Bhattacharyya et al. Bhattacharyya et al. (2021), an energy based model is
trained to rank samples drawn from NMT. Lee et al. Lee et al. (2021) predicts
the observed distribution of a desired metric, e.g., BLEU, over the $n$-best
list by training a large transformer architecture. Note that these methods are
orthogonal to our method, and they can be theoretically used as the quality
regularizer in our framework.

Uncertainty quantification Hüllermeier and Waegeman (2021) have been widely
used in neural networks, which is usually solved by Bayesian frameworks.
Because the high training cost brought by Bayesian neural networks, various
approximations, such as Monte Carlo (MC) Dropout Gal and Ghahramani (2016) and
model ensembling Lakshminarayanan et al. (2017) have been developed. In NMT,
the MC dropout is used at test time, by performing several stochastic forward
passes through the model. Then, the expectation or variance of the output
which reflect whether the current model is confident or hesitant on the
translation, is used to evaluate machine translation quality Fomicheva et al.
(2020). On the other hand, in the image classification task, entropy based
measures are used to address uncertainty quantification Smith and Gal (2018).
Our uncertainty regularizers adopt similar uncertainty quantification
strategies.

##  7 Conclusion

In this paper, we introduce a RMBR to choose adequate translations from the
candidates decoded by beam search. Based on MBR, we adopt semantic-based
similarity and compute the expected utility for each candidate by truncating
the list. The proposed quality regularizer and uncertainty regularizer are
further incorporated into the framework. Extensive experimental results show
that RMBR outperforms several MBR-based variants and other reranking baselines
on MT tasks: +1.9 BLEU points, +7.5 COMET points, +4.4 BLEURT points over the
results of beam search with sizes 5 on IWSLT’14 German$\to$English. To get a
better insight into RMBR, we also conduct the in-depth ablation study and
analytical experiments to show the performance improvement brought by each
component of RMBR.

## References

  * Bhattacharyya et al. [2021] Sumanta Bhattacharyya, Amirmohammad Rooshenas, Subhajit Naskar, Simeng Sun, Mohit Iyyer, and Andrew McCallum.  Energy-based reranking: Improving neural machine translation using energy-based models.  In ACL/IJCNLP, 2021.
  * Blain et al. [2017] Frédéric Blain, Lucia Specia, and Pranava Madhyastha.  Exploring hypotheses spaces in neural machine translation.  In AAMT, 2017.
  * Borgeaud and Emerson [2020] Sebastian Borgeaud and Guy Emerson.  Leveraging sentence similarity in natural language generation: Improving beam search using range voting.  In NGT@ACL, 2020.
  * Eikema and Aziz [2020] Bryan Eikema and Wilker Aziz.  Is MAP decoding all you need? the inadequacy of the mode in neural machine translation.  In COLING, 2020.
  * Eikema and Aziz [2021] Bryan Eikema and Wilker Aziz.  Sampling-based minimum bayes risk decoding for neural machine translation.  arXiv preprint arXiv: 2108.04718, 2021.
  * Fan et al. [2018] Angela Fan, Mike Lewis, and Yann N. Dauphin.  Hierarchical neural story generation.  In ACL, 2018.
  * Fomicheva et al. [2020] Marina Fomicheva, Shuo Sun, Lisa Yankovskaya, Frédéric Blain, Francisco Guzmán, Mark Fishel, Nikolaos Aletras, Vishrav Chaudhary, and Lucia Specia.  Unsupervised quality estimation for neural machine translation.  Trans. Assoc. Comput. Linguistics, 2020.
  * Fu et al. [2021] Zihao Fu, Wai Lam, Anthony Man-Cho So, and Bei Shi.  A theoretical analysis of the repetition problem in text generation.  In AAAI, 2021.
  * Gal and Ghahramani [2016] Yarin Gal and Zoubin Ghahramani.  Dropout as a bayesian approximation: Representing model uncertainty in deep learning.  In ICML, 2016.
  * Holtzman et al. [2020] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi.  The curious case of neural text degeneration.  In ICLR, 2020.
  * Hüllermeier and Waegeman [2021] Eyke Hüllermeier and Willem Waegeman.  Aleatoric and epistemic uncertainty in machine learning: an introduction to concepts and methods.  Mach. Learn., 110(3):457–506, 2021.
  * Lakshminarayanan et al. [2017] Balaji Lakshminarayanan, Alexander Pritzel, and Charles Blundell.  Simple and scalable predictive uncertainty estimation using deep ensembles.  In NIPS, 2017.
  * Leblond et al. [2021] Rémi Leblond, Jean-Baptiste Alayrac, Laurent Sifre, Miruna Pislar, Jean-Baptiste Lespiau, Ioannis Antonoglou, Karen Simonyan, and Oriol Vinyals.  Machine translation decoding beyond beam search.  In EMNLP, 2021.
  * Lee et al. [2021] Ann Lee, Michael Auli, and Marc’Aurelio Ranzato.  Discriminative reranking for neural machine translation.  In ACL/IJCNLP, 2021.
  * Li et al. [2016] Jiwei Li, Will Monroe, and Dan Jurafsky.  A simple, fast diverse decoding algorithm for neural generation.  arXiv preprint arXiv: 1611.08562, 2016.
  * Meister et al. [2020] Clara Meister, Ryan Cotterell, and Tim Vieira.  If beam search is the answer, what was the question?  In EMNLP, 2020.
  * Ng et al. [2019] Nathan Ng, Kyra Yee, Alexei Baevski, Myle Ott, Michael Auli, and Sergey Edunov.  Facebook fair’s WMT19 news translation task submission.  In WMT, 2019.
  * Radford et al. [2019] Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever.  Language models are unsupervised multitask learners.  2019\.
  * Ranasinghe et al. [2020] Tharindu Ranasinghe, Constantin Orasan, and Ruslan Mitkov.  Transquest: Translation quality estimation with cross-lingual transformers.  In COLING, 2020.
  * Rapp [2009] Reinhard Rapp.  The backtranslation score: Automatic mt evalution at the sentence level without reference translations.  In ACL, 2009.
  * Rei et al. [2020] Ricardo Rei, Craig Stewart, Ana C. Farinha, and Alon Lavie.  COMET: A neural framework for MT evaluation.  In EMNLP, 2020.
  * Sakaguchi et al. [2014] Keisuke Sakaguchi, Matt Post, and Benjamin Van Durme.  Efficient elicitation of annotations for human evaluation of machine translation.  In WMT@ACL, 2014.
  * Sellam et al. [2020] Thibault Sellam, Dipanjan Das, and Ankur P. Parikh.  BLEURT: learning robust metrics for text generation.  In ACL, 2020.
  * Shu and Nakayama [2017] Raphael Shu and Hideki Nakayama.  Later-stage minimum bayes-risk decoding for neural machine translation.  arXiv preprint arXiv: 1704.03169, 2017.
  * Smith and Gal [2018] Lewis Smith and Yarin Gal.  Understanding measures of uncertainty for adversarial example detection.  In UAI, 2018.
  * Sutskever et al. [2014] Ilya Sutskever, Oriol Vinyals, and Quoc V. Le.  Sequence to sequence learning with neural networks.  In NIPS, 2014.
  * Vaswani et al. [2017] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NIPS, 2017.
  * Vijayakumar et al. [2016] Ashwin K. Vijayakumar, Michael Cogswell, Ramprasaath R. Selvaraju, Qing Sun, Stefan Lee, David J. Crandall, and Dhruv Batra.  Diverse beam search: Decoding diverse solutions from neural sequence models.  arXiv preprint arXiv: 1610.02424, 2016.
  * Wang et al. [2019] Shuo Wang, Yang Liu, Chao Wang, Huanbo Luan, and Maosong Sun.  Improving back-translation with uncertainty-based confidence estimation.  In EMNLP-IJCNLP, 2019.

##  Appendix A N-by-L

The number of candidates used to compute expected utility is defined as $l$ in
Sec §3.1. To explore the effectiveness of $l$ on BLEU score of the reranked
1-best candidates, we use MBRCOMET and MBRBLEU to rank the 30 candidates
decoded by beam search with beam size of 30. We compute the expected utility
for each candidate by comparing it to top-$l$ candidates of the 30 candidates.
The results are shown in Fig. 3. As $l$ increases, the BLEU scores of the
1-best candidates reranked by both MBRCOMET and MBRBLEU go up and then down.
The reason may be that partial candidates near the end of the list is
extremely close to each other, but of poor quality. When $l$ increases, this
part of candidates are more likely to be selected. When $l$ is around 21, BLEU
scores of MBRCOMET and MBRBLEU are close to the optimal. For the proposed
reranking method, $l$ is tuned on the validation set and fixed for inference
for all testing instances.

Figure 3: The reranking results using partial candidates to compute expected
utility on the IWSLT’14 De$\to$En dev sets. y-axis is the BLEU score. x-axis
is the number of candidates used to compute MBR scores.

Method | COMET | BLEURT | BLEU
---|---|---|---
Top-1 (beam=5) | 34.79 | 16.16 | 34.28
Top-1 (beam=30) | 34.22 | 15.99 | 34.17
MBRBLEU | 34.39 | 16.39 | 34.54
MBRBLEU+LP | 34.75 | 16.64 | 34.56
MBRBLEU+LP+BT | 42.48 | 19.03 | 35.17
MBRBLEU+LP+QE | 38.68 | 19.75 | 35.44
MBRBLEU+LP+LM | 38.89 | 19.91 | 35.41
MBRBLEU+LP+QE+LM | 39.82 | 19.92 | 35.81
MBRBLEURT | 33.10 | 22.00 | 33.01
MBRBLEURT+LP | 35.83 | 19.86 | 34.55
MBRBLEURT+LP+BT | 42.46 | 19.20 | 35.18
MBRBLEURT+LP+QE | 38.91 | 20.19 | 35.42
MBRBLEURT+LP+LM | 36.79 | 18.04 | 35.25
MBRBLEURT+LP+QE+LM | 40.65 | 20.49 | 36.14
MBRCOMET | 42.53 | 17.78 | 34.55
MBRCOMET+LP | 41.60 | 17.89 | 34.91
MBRCOMET+LP+BT | 43.64 | 18.86 | 35.24
MBRCOMET+LP+QE | 42.04 | 19.96 | 35.62
MBRCOMET+LP+LM | 41.75 | 18.40 | 35.49
MBRCOMET+LP+QE+LM | 42.24 | 20.60 | 36.19

Table 6: Comparison results of MBRBLEURT and MBRBLEU with the proposed quality
regularizers on IWSLT’14 De$\to$En.

##  Appendix B Utility Functions

To further verify the effectiveness of different utility functions, we also
compare the performance of introducing the quality regularizers that performs
well in previous experiments on MBRBLEURT and MBRBLEURT. We follow the same
settings in Sec §4.2. As shown in Table 6, similar to RMBRCOMET, RMBRBLEU and
RMBRBLEURT also achieve significant gains over the results of beam search with
sizes 5 and 30, which is consistent with the results shown in Table 1 and
Table 2. Overall, RMBRBLEURT variants achieve better scores than RMBRBLEU
variants, and RMBRCOMET variants perform best. These results show that
semantic-based MBR leads to better translation options.

Method | COMET | BLEURT | BLEU
---|---|---|---
Top-1 (beam=5) | 34.79 | 16.16 | 34.28
Top-1 (beam=30) | 34.22 | 15.99 | 34.17
Top-1 (beam=50) | 33.84 | 15.87 | 34.10
beam=50
MBRCOMET | 43.50 | 18.27 | 34.57
MBRCOMET+LP | 42.35 | 18.11 | 34.94
MBRCOMET+LP+BT | 44.42 | 18.97 | 35.31
MBRCOMET+LP+QE | 42.74 | 20.26 | 35.62
MBRCOMET+LP+LM | 42.87 | 18.96 | 35.58
MBRCOMET+LP+QE+LM | 42.62 | 21.54 | 36.24
beam=30
MBRCOMET | 42.53 | 17.78 | 34.55
MBRCOMET+LP | 41.60 | 17.89 | 34.91
MBRCOMET+LP+BT | 43.64 | 18.86 | 35.24
MBRCOMET+LP+QE | 42.04 | 19.96 | 35.62
MBRCOMET+LP+LM | 41.75 | 18.40 | 35.49
MBRCOMET+LP+QE+LM | 42.24 | 20.60 | 36.19
beam=5
MBRCOMET | 36.65 | 16.03 | 34.19
MBRCOMET+LP | 36.44 | 16.47 | 34.40
MBRCOMET+LP+BT | 38.99 | 17.38 | 34.69
MBRCOMET+LP+QE | 38.09 | 18.20 | 34.86
MBRCOMET+LP+LM | 36.73 | 16.81 | 34.78
MBRCOMET+LP+QE+LM | 37.67 | 18.70 | 35.28

Table 7: Comparison results of beam size 5, 30, and 50 on IWSLT’14 De$\to$En. Source |  Wir erwarten ein paar außergewöhnliche Jahrzehnte.
---|---
Reference |  We are living into extraordinary decades ahead.
Top-1 (beam=5) |  We expect some extraordinary years.
MBRCOMET |  We are looking forward to extraordinary decades.
MBRBLEURT |  We expect some extraordinary decades.
MBRBLEU |  We expect for several remarkable decades.
Table 8: Examples of 1-best candidates chosen by the proposed reranking
methods from $n$-best list (with $n$ = 30). Underline represents the main
differences between the reference, the top-1 candidates, and the reranked
1-best candidates.

##  Appendix C Qualitative Analysis

In Table 8, we illustrate some examples from the reranking approach. Although,
the word overlap between the 1-best candidates by regularized MBR ranker and
the top-1 candidates is high, the proposed reranking methods produce accurate
and fluent translation with asyntactic re-orderings, new words, morphological
variations.

##  Appendix D Beam Sizes

In this section, we explore the performance of the proposed RMBRCOMET
reranking in large beam sizes. As shown in Table 7, the translation quality of
beam search deceases with increased beam sizes. Notably, RMBRCOMET achieves
significant higher score in COMET, BLEU, and BLEURT score with larger beam
size, which suggests that RMBR benefits from larger beam sizes. Moreover, the
1-best candidates of RMBRCOMET far outperforms the top-1 candidates of beam
search with sizes 5, 30, and 50. The results means that the proposed reranking
method can improve upon beam search.


* Long et al. (2014) Long, M., Jordan, G. C., van Rossum, D. R., et al. 2014, The Astrophysical Journal, 789, 103, doi: 10.1088/0004-637X/789/2/103
  * Magee et al. (2016) Magee, M. R., Kotak, R., Sim, S. A., et al. 2016, A&A, 589, A89, doi: 10.1051/0004-6361/201528036
  * McClelland et al. (2010) McClelland, C. M., Garnavich, P. M., Galbany, L., et al. 2010, The Astrophysical Journal, 720, 704, doi: 10.1088/0004-637x/720/1/704
  * McCully et al. (2014) McCully, C., Jha, S. W., Foley, R. J., et al. 2014, Nature, 512, 54, doi: 10.1038/nature13615
  * McCully et al. (2022) McCully, C., Jha, S. W., Scalzo, R. A., et al. 2022, ApJ, 925, 138, doi: 10.3847/1538-4357/ac3bbd
  * Narayan et al. (2011) Narayan, G., Foley, R. J., Berger, E., et al. 2011, The Astrophysical Journal, 731, L11, doi: 10.1088/2041-8205/731/1/l11
  * Oskinova et al. (2020) Oskinova, L. M., Gvaramadze, V. V., Gräfener, G., Langer, N., & Todt, H. 2020, A&A, 644, L8, doi: 10.1051/0004-6361/202039232
  * Paxton et al. (2011) Paxton, B., Bildsten, L., Dotter, A., et al. 2011, ApJS, 192, 3, doi: 10.1088/0067-0049/192/1/3
  * Paxton et al. (2013) Paxton, B., Cantiello, M., Arras, P., et al. 2013, ApJS, 208, 4, doi: 10.1088/0067-0049/208/1/4
  * Paxton et al. (2015) Paxton, B., Marchant, P., Schwab, J., et al. 2015, ApJS, 220, 15, doi: 10.1088/0067-0049/220/1/15
  * Perlmutter et al. (1999) Perlmutter, S., Aldering, G., Goldhaber, G., et al. 1999, ApJ, 517, 565, doi: 10.1086/307221
  * Phillips (1993) Phillips, M. M. 1993, ApJ, 413, L105, doi: 10.1086/186970
  * Reinecke et al. (1999a) Reinecke, M., Hillebrandt, W., & Niemeyer, J. C. 1999a, A&A, 347, 739.  https://arxiv.org/abs/astro-ph/9812120
  * Reinecke et al. (2002) —. 2002, A&A, 386, 936, doi: 10.1051/0004-6361:20020323
  * Reinecke et al. (1999b) Reinecke, M., Hillebrandt, W., Niemeyer, J. C., Klein, R., & Gröbl, A. 1999b, A&A, 347, 724.  https://arxiv.org/abs/astro-ph/9812119
  * Riess et al. (1998) Riess, A. G., Filippenko, A. V., Challis, P., et al. 1998, The Astronomical Journal, 116, 1009, doi: 10.1086/300499
  * Ritter et al. (2021) Ritter, A., Parker, Q. A., Lykou, F., et al. 2021, The Astrophysical Journal Letters, 918, L33, doi: 10.3847/2041-8213/ac2253
  * Schwab & Garaud (2019) Schwab, J., & Garaud, P. 2019, The Astrophysical Journal, 876, 10, doi: 10.3847/1538-4357/ab113f
  * Seitenzahl et al. (2009) Seitenzahl, I. R., Townsley, D. M., Peng, F., & Truran, J. W. 2009, Atomic Data and Nuclear Data Tables, 95, 96, doi: https://doi.org/10.1016/j.adt.2008.08.001
  * Shen & Schwab (2017) Shen, K. J., & Schwab, J. 2017, The Astrophysical Journal, 834, 180, doi: 10.3847/1538-4357/834/2/180
  * Singh et al. (2022) Singh, M., Misra, K., Sahu, D. K., et al. 2022, Monthly Notices of the Royal Astronomical Society, 517, 5617, doi: 10.1093/mnras/stac3059
  * Srivastav et al. (2020) Srivastav, S., Smartt, S. J., Leloudas, G., et al. 2020, The Astrophysical Journal, 892, L24, doi: 10.3847/2041-8213/ab76d5
  * Srivastav et al. (2022) Srivastav, S., Smartt, S. J., Huber, M. E., et al. 2022, Monthly Notices of the Royal Astronomical Society, 511, 2708, doi: 10.1093/mnras/stac177
  * Stritzinger et al. (2014) Stritzinger, M. D., Hsiao, E., Valenti, S., et al. 2014, A&A, 561, A146, doi: 10.1051/0004-6361/201322889
  * Stritzinger et al. (2015) Stritzinger, M. D., Valenti, S., Hoeflich, P., et al. 2015, A&A, 573, A2, doi: 10.1051/0004-6361/201424168
  * Szalai et al. (2015) Szalai, T., Vinkó, J., Sárneczky, K., et al. 2015, Monthly Notices of the Royal Astronomical Society, 453, 2103, doi: 10.1093/mnras/stv1776
  * Timmes & Arnett (1999) Timmes, F. X., & Arnett, D. 1999, ApJS, 125, 277, doi: 10.1086/313271
  * Timmes & Swesty (2000) Timmes, F. X., & Swesty, F. D. 2000, ApJS, 126, 501, doi: 10.1086/313304
  * Timmes & Woosley (1992) Timmes, F. X., & Woosley, S. E. 1992, ApJ, 396, 649, doi: 10.1086/171746
  * Tomasella et al. (2016) Tomasella, L., Cappellaro, E., Benetti, S., et al. 2016, Monthly Notices of the Royal Astronomical Society, 459, 1018, doi: 10.1093/mnras/stw696
  * Tomasella et al. (2020) Tomasella, L., Stritzinger, M., Benetti, S., et al. 2020, MNRAS, 496, 1132, doi: 10.1093/mnras/staa1611
  * Townsley et al. (2007) Townsley, D. M., Calder, A. C., Asida, S. M., et al. 2007, The Astrophysical Journal, 668, 1118, doi: 10.1086/521013
  * Townsley et al. (2009) Townsley, D. M., Jackson, A. P., Calder, A. C., et al. 2009, The Astrophysical Journal, 701, 1582, doi: 10.1088/0004-637x/701/2/1582
  * Townsley et al. (2019) Townsley, D. M., Miles, B. J., Shen, K. J., & Kasen, D. 2019, The Astrophysical Journal Letters, 878, L38, doi: 10.3847/2041-8213/ab27cd
  * Townsley et al. (2016) Townsley, D. M., Miles, B. J., Timmes, F. X., Calder, A. C., & Brown, E. F. 2016, The Astrophysical Journal Supplement Series, 225, 3, doi: 10.3847/0067-0049/225/1/3
  * Turk et al. (2010) Turk, M. J., Smith, B. D., Oishi, J. S., et al. 2010, The Astrophysical Journal Supplement Series, 192, 9, doi: 10.1088/0067-0049/192/1/9
  * Valenti et al. (2007) Valenti, S., Benetti, S., Cappellaro, E., et al. 2007, Monthly Notices of the Royal Astronomical Society, 383, 1485, doi: 10.1111/j.1365-2966.2007.12647.x
  * Valenti et al. (2009) Valenti, S., Pastorello, A., Cappellaro, E., et al. 2009, Nature, 459, 674, doi: 10.1038/nature08023
  * Vladimirova et al. (2006) Vladimirova, N., Weirs, V. G., & Ryzhik, L. 2006, Combustion Theory and Modelling, 10, 727, doi: 10.1080/13647830500464146
  * Willcox et al. (2016) Willcox, D. E., Townsley, D. M., Calder, A. C., Denissenkov, P. A., & Herwig, F. 2016, The Astrophysical Journal, 832, 13, doi: 10.3847/0004-637x/832/1/13
  * Zingale et al. (2002) Zingale, M., Dursi, L. J., ZuHone, J., et al. 2002, The Astrophysical Journal Supplement Series, 143, 539, doi: 10.1086/342754