#  LLM-Blender: Ensembling Large Language Models
with Pairwise Ranking and Generative Fusion

Dongfu Jiang♡ Xiang Ren♣♠ Bill Yuchen Lin♠
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
♠Allen Institute for Artificial Intelligence
♣University of Southern California ♡Zhejiang University

###### Abstract

We present LLM-Blender, an ensembling framework designed to attain
consistently superior performance by leveraging the diverse strengths of
multiple open-source large language models (LLMs). Our framework consists of
two modules: PairRanker and GenFuser, addressing the observation that optimal
LLMs for different examples can significantly vary. PairRanker employs a
specialized pairwise comparison method to distinguish subtle differences
between candidate outputs. It jointly encodes the input text and a pair of
candidates, using cross-attention encoders to determine the superior one. Our
results demonstrate that PairRanker exhibits the highest correlation with
ChatGPT-based ranking. Then, GenFuser aims to merge the top-ranked candidates,
generating an improved output by capitalizing on their strengths and
mitigating their weaknesses. To facilitate large-scale evaluation, we
introduce a benchmark dataset, MixInstruct, which is a mixture of multiple
instruction datasets featuring oracle pairwise comparisons. Our LLM-Blender
significantly outperform individual LLMs and baseline methods across various
metrics, establishing a substantial performance gap.
111https://yuchenlin.xyz/LLM-Blender 222The experiments on summarization,
translation, and constrained generation tasks in the prior version have been
moved to the appendix. Instead, we mainly present our work in the context of
instruction-following data and LLMs in this version.

##  1 Introduction

Large language models (LLMs) have shown impressive performance in diverse
tasks, primarily due to their capacity to follow instructions and access
extensive, high-quality data, showing a promising future for artificial
general intelligence Bubeck et al. (2023). However, prominent LLMs such as
GPT-4 and PaLM Chowdhery et al. (2022) are closed-source, restricting insights
into their architectures and training data. Open-source LLMs like Pythia
Biderman et al. (2023), LLaMA Touvron et al. (2023), and Flan-T5 Chung et al.
(2022) offer a chance to fine-tune these models on custom instruction
datasets, enabling the development of smaller yet efficient LLMs, such as
Alpaca, Vicuna Chiang et al. (2023), OpenAssistant LAION-AI (2023), and MPT
MosaicML (2023).

Figure 1:  Motivation of ensembling LLMs. Based on this pie chart about the
percentage of examples where each LLM ranks 1st, we can see that optimal LLMs
for different examples can significantly vary.

The open-source LLMs exhibit diverse strengths and weaknesses due to
variations in data, architectures, and hyperparameters, making them
complementary to each other. Figure 1 illustrates the distribution of best
LLMs on 5,000 instructions that we collected. More ranking details can be
found in Sec. 5.1. Although Vicuna achieves the highest percentage, it ranks
first in only 21.22% of the examples. Furthermore, the pie chart suggests that
the optimal LLMs for different examples can significantly vary and there is no
open-source LLM that dominates the competition. Therefore, it is important to
dynamically ensemble these LLMs to generate consistently better responses for
each input. Considering the diverse strengths and weaknesses of LLMs, it is
crucial to develop an ensembling method that harnesses their complementary
potentials, leading to improved robustness, generalization, and accuracy. By
combining their unique contributions, we can alleviate biases, errors, and
uncertainties in individual LLMs, resulting in outputs better aligned with
human preferences.

We introduce  LLM-Blender, an ensembling framework designed to achieve
consistently superior performance by mixing the outputs of multiple LLMs. LLM-
Blender comprises two modules: PairRanker and GenFuser. Initially, PairRanker
compares the outputs from $N$ LLMs, which GenFuser then fuses to generate the
final output from the top $K$ ranked outputs.

Existing approaches Ravaut et al. (2022a); Liu and Liu (2021), including the
reward model within InstructGPT Ouyang et al. (2022), for ranking outputs
$\\{y_{1},\dots,y_{N}\\}$ from language models (LMs) on a given input $x$ have
mostly focused on individually scoring each $y_{i}$ based on $x$, employing
encoding modules in the form of $s_{i}=f_{\phi}(x,y_{i})$. Although this list-
wise ranking objective can be powerful and efficient when candidate
differences are apparent, it may not be as effective when ensembling LLMs.
Among the output candidates from LLMs, candidate differences can be quite
subtle, as they are all produced by very sophisticated models and one may only
be marginally better than another. Even for humans, it can be challenging to
gauge candidate quality without direct comparison.

As a result, we propose a specialized pairwise comparison method, PairRanker
(Sec. 3), to effectively discern subtle differences between candidate outputs
and enhance ranking performance. In particular, we first gather the outputs
from $N$ models (e.g., the $N=11$ models in Fig. 1) for each input and
subsequently create the $N(N-1)/2$ pairs of their outputs. We jointly encode
the input $x$ and the two candidate outputs $y_{i}$ and $y_{j}$ as input to a
cross-attention encoder (e.g., RoBERTa Liu et al. (2019)), in the form of
$f_{\phi}(x,y_{i},y_{j})$, to learn and determine which candidate is better.

During the inference stage, we compute a matrix containing logits representing
pairwise comparison results. Given this matrix, we can infer a ranking of the
$N$ outputs for the given input $x$. Subsequently, we can employ the top-
ranked candidate from PairRanker for each input as the final result. Hence,
this approach does not rely on a single model for all examples; instead,
PairRanker selects the best model for each example by comprehensively
comparing all candidate pairs.

Nonetheless, this approach may constrain the potential to generate even better
outputs than the existing candidates. To investigate this possibility, we
introduce the GenFuser (Sec. 4) module to fuse the top $K$ of the $N$ ranked
candidates and generate an improved output for end-users. Our goal is to
capitalize on the strengths of the top $K$ selected candidates while
mitigating their weaknesses.

To assess the effectiveness of LLM ensembling methods, we introduce a
benchmark dataset called  MixInstruct (Sec. 2.2). In this dataset, we use
$N$=11 popular open-source LLMs to generate $N$ candidates for each input
across various existing instruction-following tasks formatted as self-instruct
Wang et al. (2022). The dataset comprises 100k training examples and 5k
validation examples for training a candidate ranking module like our
PairRanker, and 5k test examples with oracle comparisons for automatic
evaluation.

In Section 5, our empirical results on the MixInstruct benchmark reveal that
the LLM-Blender framework significantly boosts overall performance by
ensembling LLMs. The selections made by PairRanker outperform any fixed
individual LLM models, as indicated by superior performance in both reference-
based metrics and GPT-Rank. By leveraging the top selections from PairRanker,
GenFuser further enhances response quality through effective fusion into the
final output. LLM-Blender achieves the highest scores in terms of both
conventional metrics (i.e., BERTScore, BARTScore, BLUERT) and ChatGPT-based
ranking. The average rank of LLM-Blender stands at 3.2 among the 12 methods,
which is considerably better than the best LLM’s rank of 3.90. Moreover, LLM-
Blender’s output ranks in the top 3 for 68.59% of examples, while Viccuna only
reaches 52.88%. We believe LLM-Blender and our findings would benefit both
practitioners and researchers for deploying and studying LLMs with ensemble
learning.

Figure 2: The LLM-Blender framework. For each input $x$ from users, we employ
$N$ different LLMs to get output candidates. Then, we pair all candidates and
concatenate them with the input before feeding them to PairRanker, producing a
matrix as comparison results. By aggregating the results in the matrix, we can
then rank all candidates and take the top $K$ of them for generative fusion.
The GenFuser module concatenates the input $x$ with the $K$ top-ranked
candidates as input and generate the final output $\hat{y}$.

##  2 Preliminaries

We first provide the problem formulation and two common types of ensembling
methods. Next, we present the dataset MixInstruct created for training and
evaluation purposes. Finally, we give an overview of our framework.

###  2.1 Problem Setup

Given an input $x$ and $N$ models,
$\\{\mathcal{M}_{1},\dots,\mathcal{M}_{N}\\}$, we can generate $N$ candidate
outputs by processing $x$ with each model. We denote the candidates as
$\mathbb{Y}=\\{y_{1},\dots,y_{N}\\}$. In the training data, we assume there is
a ground truth output, $y$, while it remains hidden during evaluation at test
time.

In practice, one might choose a fixed model, such as $\mathcal{M}_{9}$, to
infer all unseen examples (i.e., always using $y_{9}$ as the final output for
$x$). This can be reasonable if $\mathcal{M}_{9}$ demonstrates significantly
better overall performance on certain observed examples. However, relying on a
pre-selected model may result in sub-optimal performance, as the $N$ models
likely possess different strengths and weaknesses in various situations,
meaning that the optimal selection for different $x$ values may not always
originate from the same model.

Our objective is to develop an ensemble learning method that produces an
output $\hat{y}$ for the input $x$, maximizing the similarity
$Q(\hat{y},y;x)$. The $Q$ function can be implemented in various ways, which
we will discuss later. We anticipate that this method will yield better
overall performance than using a fixed model or randomly selecting a model for
$x$. Specifically, given a test set $D_{\text{test}}=\\{(x^{(i)},y^{(i)})\\}$,
we aim to maximize $\sum_{i}{Q(\hat{y}^{(i)},y^{(i)};x^{(i)})}$.

There are two primary approaches for ensembling LLMs: selection-based and
generation-based methods. Selection-based methods compare candidates in the
set $\mathbb{Y}$, selecting the top-ranked candidate as the final output
$\hat{y}$, which implies that $\hat{y}\in\mathbb{Y}$. Due to the inherent
nature of selection and the limited solution space, the performance of
selection-based methods is bounded by the $N$ candidates being considered.
Conversely, generation-based methods focus on fusing $K$ candidates ($1<K\leq
N$) from $\mathbb{Y}$ to produce an unseen response as the final output
$\hat{y}$.

Sources |  #Examples |  Source |  I/O Tokens
---|---|---|---
Alpaca-GPT4 | 22,862 | GPT-4 | 22 / 48
Dolly-15K | 7,584 | Human | 24 / 53
GPT4All-LAION | 76,552 | ChatGPT | 18 / 72
ShareGPT | 3,002 | ChatGPT | 36 / 63
Total | 110K | Mix | 20 / 66
Table 1: Statistics of MixInstruct. It contains 110K examples and we randomly
split the dataset into train/dev/test in 100K/5K/5K sizes.

###  2.2  MixInstruct: A New Benchmark

We introduce a new dataset, MixInstruct, to benchmark ensemble models for LLMs
in instruction-following tasks. We collect a large-scale set of instruction
examples primarily from four sources, as shown in Table 1. After curating and
processing this open-source data, we sample 100k examples for training, 5k for
validation, and 5k for testing. We then run $N=11$ popular open-source LLMs,
including Vicuna, OpenAssistant, Alpaca, MPT, and others (see Table 2 and
Figure 1), on these 110k examples.

To obtain the oracle ranking of candidates, we design comparative prompts for
ChatGPT to evaluate all candidate pairs. Specifically, for each example, we
prepare 55 pairs of candidates ($11\times 10/2$). For each pair, we ask
ChatGPT to judge the better candidate (or declare a tie). The prompt template
can be found in the appendix. For the training and validation sets, we provide
the results based on conventional metrics like BERTScore, BLEURT, and
BARTScore. In that case, we use function $Q(y_{i},y)$ to estimate a candidate
$y_{i}$’s quality according to its similarity to the ground truth $y$.

###  2.3  LLM-Blender: A Novel Framework

We propose a rank-and-fuse pipeline framework, LLM-Blender, for ensembling
LLMs, as illustrated in Figure 2. This framework consists of two main
components: a pairwise ranking module, PairRanker (Section 3), and a fusion
module, GenFuser (Section 4). The PairRanker module learns to compare all
pairs of candidates for each input and subsequently rank the list of
candidates. We then select the top $K=3$ ranked candidates, concatenate them
with the input $x$, and construct the input sequence for the GenFuser module.
The GenFuser module, a seq2seq LM, ultimately generates the final output to
serve users.

##  3 PairRanker: Pairwise Ranking

Figure 3: The architectures of typical reranking methods. $x$ is an input and
$y_{i}$ is a certain candidate, and its score is $s_{i}$. MLM-Scoring is an
unsupervised method that uses an external masked LM to score a candidate;
SimCLS uses the same encoder to encode $x$ and each candidate $y_{i}$;
SummaReranker instead employs a cross-encoder to encode both $x$ and $y_{i}$
at the same time; PairRanker encodes a pair of candidates ($y_{i}$, $y_{j}$)
at the same time for pairwisely scoring them, and the final score of each
candidate is produced as shown in Fig. 4.

In this section, we introduce three baseline methods for ranking the
candidates in $\mathbb{Y}$ in Sec. 3.1 and present the proposed PairRanker
method.

###  3.1 Baseline Methods

Previous reranking methods primarily focus on computing the score
$s_{i}=f_{\phi}(x,y_{i})$ for each candidate $y_{i}\in\mathbb{Y}$
independently, where $s_{i}$ is solely determined by $y_{i}$. Notably, the
reward model in instruction tuning for GPT-3.5 Ouyang et al. (2022) also
belongs to this category. Figure 3 illustrates these baseline methods, which
are further detailed in the following paragraphs.

MLM-Scoring (Salazar et al., 2020) assesses the quality of a candidate by
calculating its pseudo-log-likelihood, which is obtained by masking tokens one
by one and computing the log-likelihood for the masked token using masked LMs
(e.g., BERT). Given a candidate $y_{i}$ as a sequence of words
$\textbf{W}=\\{w_{1},...,w_{|\textbf{W}|}\\}$, the pseudo-log-likelihood is:
$s_{i}=\sum_{t=1}^{|\textbf{W}|}\log{P(w_{t}|\textbf{W}_{\backslash t})}$.
This unsupervised method is effective for reranking outputs in NLG tasks such
as machine translation and speech recognition.

SimCLS (Liu and Liu, 2021) encodes the input $x$ and each generated candidate
$y_{i}\in\mathbb{Y}$ using the same encoder $H$, resulting in $H(x)$ and
$H(y_{i})$. The cosine similarity between them, $s_{i}=\cos{(H(x),H(y_{i}))}$,
serves as the predicted score, as $H(x)$ and $H(y_{i})$ share the same
embedding space induced by the language encoder. In training, marginal ranking
loss is used to optimize $H$.

SummaReranker (Ravaut et al., 2022a) concatenates the input $x$ and each
candidate $y_{i}$, using a cross-attention encoder to learn ranking.
Specifically, they employ $H([x;y_{i}])$ to predict the score $s_{i}$, where
$H$ is a Transformer model. In the training stage, binary cross-entropy (BCE)
loss is employed to differentiate the best candidate from the others.

#### Limitations.

Despite using contrastive loss in training, these methods rely on individual
scoring for inference. The encoders have not been exposed to pairs of
candidates for direct comparison learning. We argue that such pointwise
ranking methods may be insufficient for selecting the best candidates in the
context of LLMs and instruction-following tasks. One reason is that the
quality of LLM outputs is generally high when the chosen LLMs are popular and
competitive. Moreover, the responses for instruction tasks can be quite open-
ended, unlike summarization tasks. Therefore, merely examining individual
candidates may not yield a reliable score. This issue becomes more prominent
for shorter responses, where sequences may differ by only a few words but vary
significantly in helpfulness, harmfulness, and fairness. Given these
limitations, we contend that individual scoring approaches may fail to capture
crucial nuances.

###  3.2 Pairwise Comparisons

In order to address the limitations of pointwise ranking, we aim to train a
ranker $f$ with parameter $\phi$ that can compare a pair of output candidates
by encoding them together with the input text. Our ranker module should focus
on learning to capture the differences between the two candidates and prefer
the ones of higher quality.

Given a pair of candidates $y_{i},y_{j}$, we obtain their pair-specific
scores: $s_{(i,j)}^{i}$ and $s_{(i,j)}^{j}$. We denote the model’s confidence
in thinking $y_{i}$ is better than $y_{j}$ as
$s_{ij}=s_{(i,j)}^{i}-s_{(i,j)}^{j}$. We can use these scores for all pairs
induced from $\mathbb{Y}$ to infer the final ranking. To learn this ability,
we concatenate the input $x$ and the two candidates to form a sequence
$[x;y_{i};y_{j}]$ and feed it into a cross-attention Transformer to get the
features: $f_{\phi}([x;y_{i};y_{j}])$ for modeling $s_{ij}$.

We assume multiple $Q$ functions to optimize for, such as BERTScore,
BARTScore, etc., and consider the learning problem as a multi-task
classification problem:

$\displaystyle\mathcal{L}_{Q}=-z_{i}\log{\sigma(s_{(i,j)}^{i})}-z_{j}\log{\sigma(s_{(i,j)}^{j})},$

where $\sigma$ denotes the sigmoid function and

$\displaystyle(z_{i},z_{j})=\left\\{\begin{aligned} (1,0)&,&Q(y_{i},y)\geq
Q(y_{j},y)\\\ (0,1)&,&Q(y_{i},y)<Q(y_{j},y)\end{aligned}\right..$

For optimizing towards multiple $Q$, we take the average as the final multi-
objective loss: $\mathcal{L}=\sum\mathcal{L}_{Q}$.

###  3.3 PairRanker Architecture

We discuss the concrete designs for the PairRanker module in this subsection.

#### Encoding.

We employ Transformer layers to encode an input and a pair of candidates,
enabling the attentions to capture the difference between candidates in the
context of the input. We concatenate the three segments sequentially and form
a single input sequence with special tokens as separators: <source>,
<candidate1>, and <candidate2>. The resulting input sequences to Transformers
are in the form of “<s><source> $x$ </s> <candidate1> $y_{i}$ </s>
<candidate2> $y_{j}$ </s>”, where $x$ is the text of a source input and
$y_{i}$ and $y_{j}$ are the text of two output candidates. The embeddings of
special tokens <source>, <candidate1>, and <candidate2> are used as the
representations of $x$, $y_{i}$, and $y_{j}$ respectively, denoted as
$\mathbf{x}$, $\mathbf{y_{i}}$, $\mathbf{y_{j}}$.

#### Training.

To determine the scores for the two candidates, we concatenate the embeddings
of $\mathbf{x}$ with $\mathbf{y_{i}}$ and $\mathbf{y_{j}}$ respectively, and
pass them through a single-head layer, which is a multi-layer perceptron with
the final layer’s dimension equal to the number of $Q$ functions to be
optimized. Each value within this dimension represents a computed $Q$ score
for a specific $Q$ function. We derive the final score $s_{(i,j)}^{i}$ or
$s_{(i,j)}^{j}$ for the candidate by averaging these $Q$ scores. Since there
are $O(N^{2})$ unique pair combinations, we apply an effective sub-sampling
strategy during the training stage to ensure learning efficiency.

During training, we randomly select some combinations from the candidate pool
$\mathbb{Y}^{2}$, instead of all the $N(N-1)/2$ pairs. We also compare the
target text with other candidates by extending the candidate pool by mixing
the ground truth $y$ into $\mathbb{Y}$. In practice, we found that using 5
pairs per input is sufficient for obtaining decent results.

Due to the position embeddings of the language model, the order of the
candidates in a pair $(x,y_{i},y_{j})$ matters, as the comparison result of
$(x,y_{i},y_{j})$ and $(x,y_{j},y_{i})$ might not be consistent. Thus, we
shuffle the order of candidates within each training pair so that the model
learns to be consistent with itself.

#### Inference.

During the inference stage, we obtain scores $s_{ij}$ for each pair of
candidates $(y_{i},y_{j})\in\mathbb{Y}^{2}$. After $N(N-1)$ iterations, we
obtain a matrix $\mathbf{M}$, where $\mathbf{M}_{i}^{j}=s_{ij}$ represents the
confidence that $y_{i}$ is better than $y_{j}$. To identify the best candidate
based on $\mathbf{M}$, we introduce three aggregation functions for
determining the final ranking of $\mathbb{Y}$.

We propose two scoring methods, MaxLogits and MaxWins, which utilize all
elements in the matrix. Let $\mathbf{M}_{i}^{*}$ and $\mathbf{M}_{*}^{j}$
denote the $i$-th row and $j$-th column of the matrix as vectors. For each
candidate $y_{i}$, its MaxLogits score is defined as
$s_{i}=\sum{(\mathbf{M}_{i}^{*}-\mathbf{M}_{*}^{i})}$, while its MaxWins score
is defined as
$s_{i}=|\\{s_{ij}\in\mathbf{M}_{i}^{*}|s_{ij}>0\\}|+|\\{s_{ji}\in\mathbf{M}_{*}^{i}|s_{ji}<0\\}|$,
where $||$ denotes the set size.

In essence, MaxLogits computes the confidence that $y_{i}$ is superior to all
other candidates, whereas MaxWins calculates the number of victories in
comparisons with other candidates.

However, these two methods necessitate $O(N^{2})$ iterations for $N$
candidates, which can be computationally burdensome. Thus, we propose a more
efficient aggregation method, performing a single bubble sort run with
pairwise comparisons to select the best candidate. We first shuffle the order
of candidates in $\mathbb{Y}$ to obtain a default order, and initialize the
best candidate index $k$ to 1. We iteratively update the best candidate index
as follows:

$\displaystyle k=\left\\{\begin{aligned}
k&,&\mathbf{M}_{k}^{i}-\mathbf{M}_{i}^{k}>0\\\
i&,&\mathbf{M}_{i}^{k}-\mathbf{M}_{k}^{i}>0\end{aligned}\right..$

After $N-1$ comparisons, we select $y_{k}$ as the best candidate. This method
reduces the inference time complexity from $O(N^{2})$ to $O(N)$, aligning with
previous pointwise methods.

Regardless of the aggregation method, we can rank all candidates in
$\mathbb{Y}$. Our experiments (shown in the appendix) reveal that MaxLogits
yields the best performance, so we use MaxLogits as the default aggregator for
PairRanker.

Figure 4: Aggregation methods for PairRanker.

##  4 GenFuser: Generative Fusion

The effectiveness of PairRanker is constrained by the quality of selections
from the candidate pool $\mathbb{Y}$. We hypothesize that by merging multiple
top-ranked candidates, we can overcome this constraint. As these top
candidates often showcase complementary strengths and weaknesses, it is
plausible to generate a superior response by combining their advantages while
mitigating their shortcomings. Our objective is to devise a generative model
that takes input $x$ and $K$ top-ranked candidates
$\\{y_{1},...,y_{K}\\}\subset\mathbb{Y}$ (e.g., $K=3$) and produces an
improved output $\hat{y}$ as the final response.

To accomplish this, we present GenFuser, a seq2seq approach for fusing a set
of candidates conditioned on the input instruction to generate an enhanced
output. Specifically, we concatenate the input and $K$ candidates sequentially
using separator tokens, such as <extra_id_$i$>, and fine-tune a T5-like model
to learn to generate $y$. In practice, we employ Flan-T5-XL Chung et al.
(2022), which has 3b parameters, due to its superior performance and
relatively smaller size.

Category | Methods | BERTScore$\uparrow$ | BARTScore$\uparrow$ | BLEURT$\uparrow$ | GPT-Rank$\downarrow$ |  $\geq$ Vic(%)$\uparrow$ |  $\geq$ OA(%)$\uparrow$ | Top-3(%)$\uparrow$
---|---|---|---|---|---|---|---|---
LLMs |  Open Assistant LAION-AI (2023) | 74.68 | -3.45 | -0.39 | 3.90 | 62.78 | N/A | 51.98
Vicuna Chiang et al. (2023) | 69.60 | -3.44 | -0.61 | 4.13 | N/A | 64.77 | 52.88
Alpaca Taori et al. (2023) | 71.46 | -3.57 | -0.53 | 4.62 | 56.70 | 61.35 | 44.46
Baize Xu et al. (2023) | 65.57 | -3.53 | -0.66 | 4.86 | 52.76 | 56.40 | 38.80
MOSS Sun and Qiu (2023) | 64.85 | -3.65 | -0.73 | 5.09 | 51.62 | 51.79 | 38.27
ChatGLM Du et al. (2022) | 70.38 | -3.52 | -0.62 | 5.63 | 44.04 | 45.67 | 28.78
Koala Geng et al. (2023) | 63.96 | -3.85 | -0.84 | 6.76 | 39.93 | 39.01 | 22.55
Dolly V2 Conover et al. (2023) | 62.26 | -3.83 | -0.87 | 6.90 | 33.33 | 31.44 | 16.45
Mosaic MPT MosaicML (2023) | 63.21 | -3.72 | -0.82 | 7.19 | 30.87 | 30.16 | 16.24
StableLM Stability-AI (2023) | 62.47 | -4.12 | -0.98 | 8.71 | 21.55 | 19.87 | 7.96
Flan-T5 Chung et al. (2022) | 64.92 | -4.57 | -1.23 | 8.81 | 23.89 | 19.93 | 5.32
Analysis | Oracle (BERTScore) | 77.67 | -3.17 | -0.27 | 3.88 | 54.41 | 38.84 | 53.49
Oracle (BLEURT) | 75.02 | -3.15 | -0.15 | 3.77 | 55.61 | 45.80 | 55.36
Oracle (BARTScore) | 73.23 | -2.87 | -0.38 | 3.69 | 50.32 | 57.01 | 57.33
Oracle (GPT-Rank) | 70.32 | -3.33 | -0.51 | 1.00 | 100.00 | 100.00 | 100.00
Rankers | Random | 66.36 | -3.76 | -0.77 | 6.14 | 37.75 | 36.91 | 29.05
MLM-Scoring | 64.77 | -4.03 | -0.88 | 7.00 | 33.87 | 30.39 | 21.46
SimCLS | 73.14 | -3.22 | -0.38 | 3.50 | 52.11 | 49.93 | 60.72
SummaReranker | 71.60 | -3.25 | -0.41 | 3.66 | 55.63 | 48.46 | 57.54
PairRanker | 72.97 | -3.14 | -0.37 | 3.20 | 54.76 | 57.79 | 65.12
LLM-Blender | PR ($K=3$) + GF | 79.09 | -3.02 | -0.17 | 3.01 | 70.73 | 77.72 | 68.59
Table 2:  Empirical results on MixInstruct. GPT-Rank are the most important
metric.

##  5 Evaluation

###  5.1 Setup

We use MixInstruct (Sec. 2.2) to conduct evaluation, and more results are in
the appendix.

#### NLG metrics.

We employ two types of evaluation metrics (i.e., $Q$ ). The first group is
conventional automatic metrics for NLG tasks: BERTScore Zhang et al. (2020b),
BLEURT Sellam et al. (2020), and BARTScore Yuan et al. (2021).

#### GPT-Rank.

The second is based on prompting ChatGPT for pairwise comparisions on all
candidates and decide their rank by the number of wins (i.e., MaxWins
aggregation). We name this GPT-based ranking metric with GPT-Rank.

#### Model training.

We use the DeBERTa He et al. (2021) (400m) as the backbone for PairRanker, and
GenFuser is based on Flan-T5-XL (3b). According to our ablation studies, we
choose to use BARTScore for its superior correlation with GPT-Rank as shown in
5.2.

###  5.2 Main results

In Table 2, we present the overall performance of $N$=11 LLMs as well as other
methods on MixInstruct. In addition to the three auto metrics and GPT-Rank, we
also show the percentage of examples where each method can produce outputs
that are better than or same good as the two top LLMs, namely OpenAssistant
($\geq$OA) and Vicuna ($\geq$Vic), in terms of GPT-Rank.

#### LLMs have diverse strengths and weakness.

The table presents the LLMs in a sorted order based on their average rank as
determined by ChatGPT (GPT-Rank). Among these models, Open Assistant, Vicuna,
and Alpaca are the top-3 performers. Following them, three renowned LLMs,
namely Baize, Moss, and ChatGLM, which have been fine-tuned using both Chinese
and English instruction data, also exhibit impressive performance on
MixInstruct. Conversely, Mosaic MPT, StableLM, and Flan-T5 rank at the
bottom-3 in the evaluation. Nevertheless, the average GPT-Rank of top/bottom
models maintain a noticeable distance from the first/last position (1 or 11),
highlighting the importance of ensembling LLMs.

#### Top LLMs are not always good.

It is evident that although OA and Vic perform remarkably well, there is still
a substantial percentage of examples where other LLMs are considered to
outperform them. For instance, despite Koala having an average GPT-Rank of
$6.76$, approximately $40\%$ of the examples demonstrate that Koala produces
responses that are better or equally as good as both OA and Vic. This further
emphasizes the significance of employing our LLM-Blender framework for ranking
and fusion purposes.

#### NLG Metrics.

Moreover, we conduct a comprehensive analysis of the performance of oracle
(top-1) selections based on each of the metrics themselves. The findings
demonstrate that these selections also exhibit favorable performance across
other metrics as well. For example, the oracle selections derived from GPT-
Rank achieve a BARTScore of $-3.33$, surpassing that of OA ($-3.45$).
Conversely, the oracle selections of BARTScore yield $3.69$ in GPT-Rank, also
significantly outperforming OA ($3.90$). This observation substantiates the
rationality of using BARTScore to provide supervision for PairRanker, which is
also suggested by Table 3.

####  PairRanker outperforms other rankers.

MLM-Scoring fails to outperform even random selection, highlighting the
limitations of its unsupervised paradigm. On the contrary, SimCLS,
SummaReranker, and PairRanker exhibit superior performance compared to the
best model (OA) across BARTScore and GPT-Rank. Notably, the average GPT-rank
of the responses selected by PairRanker ($3.20$) significantly outperforms the
best model by $0.70$ (a 18% relative performance gain) and also all other
rankers. Moreover, it achieves impressive results in metrics such as BARTScore
($-3.14$) with a substantial advantage. PairRanker’s selections are better
than or equal to Vic/OA on $54.76\%$/$57.79\%$ examples respectively, and
ranks in top 3 for 65.12% examples.

####  LLM-Blender is the best.

We use top-3 selections from the PairRanker and feed them as candidates for
GenFuser. Based on this integration, LLM-Blender demonstrates remarkable
capabilities as expected. In terms of GPT-Rank, it achieves $3.01$, surpassing
both the best model OA ($3.90$) by a significant margin. The scores for
BERTScore ($79.09$), BARTScore ($-3.02$), and BELURT ($-0.17$) all exceed the
best model by $4.41$, $0.43$, and $0.22$ respectively, showcasing substantial
advantages. Moreover, LLM-Blender also performs well in surpassing the top two
models, Vic ($70.73$) and OA ($77.72$), thereby complementing the weaknesses
of PairRanker.

#### Ranking correlation.

Ranking Methods |  Pearson Correlation $\uparrow$ |  Spearman’s Correlation $\uparrow$ |  Spearman’s Footrule $\downarrow$
---|---|---|---
Random | 0.00 | 0.00 | 48.27
BLEU | 28.70 | 26.92 | 33.57
Rouge2 | 29.17 | 27.77 | 32.96
BERTScore | 32.25 | 30.33 | 33.34
BLEURT | 34.14 | 32.31 | 32.17
BARTScore | 38.49 | 36.76 | 30.93
MLM-Scoring | -0.02 | -0.01 | 47.16
SimCLS | 39.89 | 38.13 | 29.32
SummaReranker | 41.13 | 39.10 | 29.69
PairRanker | 46.98 | 44.98 | 27.52
Table 3: The correlation between each ranking method and oracle ranking (GPT-
Rank).

In addition to focusing solely on the top-1 selection of each ranker, we
present a comprehensive analysis of the overall rank correlation among all the
candidates with GPT-Rank (see Table 3). The correlation metrics used here
include the Pearson Correlation Coefficient, Spearman’s Correlation, and
Spearman’s Footrule distanceDiaconis and Graham (1977).

It turns out that BARTScore gets the highest correlation with GPT-Rank against
other metrics, which suggests we use BARTScore to provide supervision for
training. For rankers, MLM-Scoring still falls short of outperforming random
permutations. On the other side, SummaReranker demonstrates better correlation
in terms of the Pearson Correlation ($41.13$) and Spearman’s Correlation
($39.10$), while SimCLS gets a better Spearman’s Footrule distance ($29.32$)
Notably, PairRanker achieves the highest correlation with GPT-Rank across all
correlation types, which is even way better than the BARTScore.

#### More analysis.

We leave many other ablation studies and analyses in Appendix, where we apply
PairRanker to the three typical natural language generation (NLG) tasks:
summarization (CNN/DM), machine translation (WMT18-zh-en), and constrained
text generation (CommonGen). We find that PairRanker still outperforms other
methods by a large margin in the context of using a single same base model to
decode $N$ candidates (with different algorithms). We also show that MaxLogits
is much better than MaxWins and the bubble sort method is very cost-effective
if the inference efficiency is a big concern.

##  6 Related Work

#### LLM evaluation

As open-source large language models (LLMs) continue to flourish and
demonstrate remarkable competitiveness across various natural language
generation (NLG) tasks, assessing the capabilities of LLMs has become an
exceedingly challenging endeavor. To address this issue, Zheng et al. (2023)
pioneered the creation of a chatbot arena, enabling users to provide pairwise
evaluations of responses generated by two randomly selected LLMs. Based on
these evaluations, they established an LLM Elo rating leaderboard. In a
similar vein, Cabrera and Neubig (2023) conducted an evaluation study on a
customer service dataset, leveraging automated metrics such as BERTScore and
ChrF Popović (2015). This approach yielded similar LLM ranking results.
Instead of relying solely on human evaluation, Yidong et al. (2023) developed
a fine-tuned model called PandaLM to compare responses generated by different
LLMs. AlpacaFarm Dubois et al. (2023) also aims to evaluate LLMs with pairwise
feedback.

#### Pairwise ranking

Pairwise ranking, known for its long-standing effectiveness, has demonstrated
exceptional performance across a wide array of NLP tasks Jamieson and Nowak
(2011). Notably, Ranknet Burges et al. (2005) and LambdaRank Burges (2010)
have emerged as powerful techniques for various ranking problems. Furthermore,
within the renowned RLHF procedureOuyang et al. (2022), these methods
incorporate pairwise training of their reward model based on OPT. However,
these approaches still compute scores individually and solely undergo pairwise
training at the loss level. In contrast, our proposed PairReranker not only
employs pairwise training but also utilizes the attention mechanism for
pairwise inference during the inference stage. We posit that this approach
better captures the subtleties between candidates and yields superior results,
as demonstrated in Section 5.2.

#### Ensemble learning

Ensemble learning is a widely employed technique to enhance a model’s
capabilities by leveraging multiple weaker models Sagi and Rokach (2018);
Anioł and Pietroń (2019). Typically, ensemble learning is performed either by
considering model weights or by combining diverse outputs. Mix-of-Experts
(MoE) is a type of ensemble approach that combines the predictions of multiple
specialized sub-models to improve overall performance. It has been
successfully applied in various domains, such as natural language processing
and computer vision Jacobs et al. (1991); Shazeer et al. (2017). As for fusing
multiple candidates, Izacard and Grave (2021) introduced a framework named
Fusion-in-Decoder (FiD) to improve the quality of question answering by fusing
retrieved text. Building upon FiD, Ravaut et al. (2022b) further investigated
the effectiveness of fusion in the context of text summarization. However,
they neglected to incorporate a selection process prior to feeding the
candidates into the fusion module, resulting in only moderate improvements. In
contrast, our proposed approach, referred to as LLM-Blender, initially
utilizes the PairRanker algorithm to filter out candidates of poor quality.
Subsequently, fusion is performed exclusively on the top-ranked candidates,
leading to superior performance.

##  7 Conclusion & Future Directions

In this paper, we formulated the motivation to exploit the diverse strengths
and weaknesses of open-source large language models (LLMs), aiming to create
an ensembling framework that leverages their complementary capabilities to
generate consistently superior results on various instruction-following tasks.
By dynamically ensembling LLMs, we aimed to reduce biases, errors, and
uncertainties in individual models, yielding outputs better aligned with human
feedback.

Our major contributions are as follows:

  * •

A new framework: LLM-Blender is a post-hoc ensemble learning method for
ranking and fusing the outputs from multiple LLMs. It is composed of two
modules: PairRanker and GenFuser, and both are straightforward yet effective.

  * •

A new dataset: MixInstruct is a benchmark dataset, created for training and
evaluating LLM ensembling methods on instruction-following tasks.

  * •

Promising results: We show that our method can significantly improve the
overall results on various metrics, and our findings indicates that this
direction is promising for both research community and practitioners.

  * •

Toolkit: By open-sourcing our framework, we aim to make it easier for others
to leverage our approach, enabling the development of more advanced AI systems
that achieve robustness, generalization, and enhanced accuracy in a wide
variety of tasks.

#### Future directions.

Potential future directions include extending the LLM-Blender framework to
more types of models or even non-text modalities, developing more
sophisticated ranking and fusion techniques, and investigating the
transferability of our ensembling approach to other domains and tasks.
Additionally, exploring ways to minimize computational overhead and
incorporating active learning strategies for rapid adaptation to new
specialized domains and data sources represent fruitful areas for further
research. Overall, our work underscores the value of combining the unique
contributions of multiple models.

## *Limitations

#### Efficiency.

To get the optimal performance from PairRanker, one may need to call the model
$O(n^{2})$ times for getting the full matrix, thus resulting in a much less
efficient solution. We attempted to resolve this limitation by proposing to
use multiple rounds of bubble sort methods to reduce the number of inferences
needed, and we find it works pretty well. We also want to argue that although
the number of inferences can be large for obtaining the best performance with
PairRanker, those inferences can be executed in parallel because they are
totally independent.

#### Human evaluation.

We agree that automatic metrics have limitations. Human evaluation could
provide us with more reliable and comprehensive evaluation results. However,
due to the number of models as well as the amounts of generation candidates,
we cannot afford large-scale human evaluation. We argue that our use of
ChatGPT for evaluation is a good alternative, according to recent studies.
Also, we would like to highlight that we show the ground truths when using
ChatGPT to do pairwise comparisions, which is quite informative than the
common practice.

## *Ethical Statement

This work fully complies with the ACL Ethics Policy. We declare that there are
no ethical issues in this paper, to the best of our knowledge.

## Acknowledgements

We thank members of the INK lab at USC and the Mosaic team at AI2 for valuable
feedback on this project. Xiang is supported in part by the Office of the
Director of National Intelligence (ODNI), Intelligence Advanced Research
Projects Activity (IARPA), via the HIATUS Program contract #2022-22072200006,
the DARPA MCS program under Contract No. N660011924033, the Defense Advanced
Research Projects Agency with award W911NF-19-20271, NSF IIS 2048211, and gift
awards from Google and Amazon. Yuchen’s research was also supported by the
Allen Institute for AI (AI2). The views and conclusions contained herein are
those of the authors and should not be interpreted as necessarily representing
the official policies, either expressed or implied, of ODNI, IARPA, or the
U.S. Government.

## References

  * Anioł and Pietroń (2019) Anna Anioł and Marcin Pietroń. 2019.  Ensemble approach for natural language question answering problem.  _2019 Seventh International Symposium on Computing and Networking Workshops (CANDARW)_ , pages 180–183.
  * Biderman et al. (2023) Stella Rose Biderman, Hailey Schoelkopf, Quentin G. Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, Aviya Skowron, Lintang Sutawika, and Oskar van der Wal. 2023.  Pythia: A suite for analyzing large language models across training and scaling.  _ArXiv preprint_ , abs/2304.01373.
  * Bojar et al. (2017) Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Shujian Huang, Matthias Huck, Philipp Koehn, Qun Liu, Varvara Logacheva, Christof Monz, Matteo Negri, Matt Post, Raphael Rubino, Lucia Specia, and Marco Turchi. 2017.  Findings of the 2017 conference on machine translation (WMT17).  In _Proceedings of the Second Conference on Machine Translation_ , pages 169–214, Copenhagen, Denmark. Association for Computational Linguistics.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, John A. Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuan-Fang Li, Scott M. Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. 2023.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _ArXiv preprint_ , abs/2303.12712.
  * Burges (2010) Christopher J. C. Burges. 2010.  From ranknet to lambdarank to lambdamart: An overview.
  * Burges et al. (2005) Christopher J. C. Burges, Tal Shaked, Erin Renshaw, Ari Lazier, Matt Deeds, Nicole Hamilton, and Gregory N. Hullender. 2005.  Learning to rank using gradient descent.  In _Machine Learning, Proceedings of the Twenty-Second International Conference (ICML 2005), Bonn, Germany, August 7-11, 2005_ , volume 119 of _ACM International Conference Proceeding Series_ , pages 89–96. ACM.
  * Cabrera and Neubig (2023) Alex Cabrera and Graham Neubig. 2023.  Zeno chatbot report.  Blog post.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam M. Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Benton C. Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier García, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Díaz, Orhan Firat, Michele Catasta, Jason Wei, Kathleen S. Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022.  Palm: Scaling language modeling with pathways.  _ArXiv preprint_ , abs/2204.02311.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, S. Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Wei Yu, Vincent Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed Huai hsin Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoc V. Le, and Jason Wei. 2022.  Scaling instruction-finetuned language models.  _ArXiv preprint_ , abs/2210.11416.
  * Conover et al. (2023) Mike Conover, Matt Hayes, Ankit Mathur, Xiangrui Meng, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin. 2023.  Free dolly: Introducing the world’s first truly open instruction-tuned llm.
  * Diaconis and Graham (1977) Persi Diaconis and Ron Graham. 1977.  Spearman’s footrule as a measure of disarray.  _Journal of the royal statistical society series b-methodological_ , 39:262–268.
  * Du et al. (2022) Zhengxiao Du, Yujie Qian, Xiao Liu, Ming Ding, Jiezhong Qiu, Zhilin Yang, and Jie Tang. 2022.  GLM: General language model pretraining with autoregressive blank infilling.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 320–335, Dublin, Ireland. Association for Computational Linguistics.
  * Dubois et al. (2023) Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori Hashimoto. 2023.  Alpacafarm: A simulation framework for methods that learn from human feedback.  _ArXiv preprint_ , abs/2305.14387.
  * Geng et al. (2023) Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. 2023.  Koala: A dialogue model for academic research.  Blog post.
  * He et al. (2021) Pengcheng He, Jianfeng Gao, and Weizhu Chen. 2021.  Debertav3: Improving deberta using electra-style pre-training with gradient-disentangled embedding sharing.  _ArXiv preprint_ , abs/2111.09543.
  * Hermann et al. (2015) Karl Moritz Hermann, Tomás Kociský, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015.  Teaching machines to read and comprehend.  In _Advances in Neural Information Processing Systems 28: Annual Conference on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Quebec, Canada_ , pages 1693–1701.
  * Izacard and Grave (2021) Gautier Izacard and Edouard Grave. 2021.  Leveraging passage retrieval with generative models for open domain question answering.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 874–880, Online. Association for Computational Linguistics.
  * Jacobs et al. (1991) Robert A. Jacobs, Michael I. Jordan, Steven J. Nowlan, and Geoffrey E. Hinton. 1991\.  Adaptive mixtures of local experts.  _Neural Computation_ , 3:79–87.
  * Jamieson and Nowak (2011) Kevin G. Jamieson and Robert D. Nowak. 2011.  Active ranking using pairwise comparisons.  In _Advances in Neural Information Processing Systems 24: 25th Annual Conference on Neural Information Processing Systems 2011. Proceedings of a meeting held 12-14 December 2011, Granada, Spain_ , pages 2240–2248.
  * LAION-AI (2023) LAION-AI. 2023.  Open assistant.  https://github.com/LAION-AI/Open-Assistant.
  * Lewis et al. (2020) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Veselin Stoyanov, and Luke Zettlemoyer. 2020.  BART: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7871–7880, Online. Association for Computational Linguistics.
  * Lin et al. (2020) Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020.  CommonGen: A constrained text generation challenge for generative commonsense reasoning.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1823–1840, Online. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _ArXiv preprint_ , abs/1907.11692.
  * Liu and Liu (2021) Yixin Liu and Pengfei Liu. 2021.  SimCLS: A simple framework for contrastive learning of abstractive summarization.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 1065–1072, Online. Association for Computational Linguistics.
  * MosaicML (2023) NLP Team MosaicML. 2023.  Introducing mpt-7b: A new standard for open-source, ly usable llms.  Accessed: 2023-05-23.
  * Nallapati et al. (2016) Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çağlar Gulçehre, and Bing Xiang. 2016.  Abstractive text summarization using sequence-to-sequence RNNs and beyond.  In _Proceedings of the 20th SIGNLL Conference on Computational Natural Language Learning_ , pages 280–290, Berlin, Germany. Association for Computational Linguistics.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke E. Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Francis Christiano, Jan Leike, and Ryan J. Lowe. 2022\.  Training language models to follow instructions with human feedback.  _ArXiv preprint_ , abs/2203.02155.
  * Popović (2015) Maja Popović. 2015.  chrF: character n-gram F-score for automatic MT evaluation.  In _Proceedings of the Tenth Workshop on Statistical Machine Translation_ , pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21:140:1–140:67.
  * Ravaut et al. (2022a) Mathieu Ravaut, Shafiq Joty, and Nancy Chen. 2022a.  SummaReranker: A multi-task mixture-of-experts re-ranking framework for abstractive summarization.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 4504–4524, Dublin, Ireland. Association for Computational Linguistics.
  * Ravaut et al. (2022b) Mathieu Ravaut, Shafiq Joty, and Nancy Chen. 2022b.  Towards summary candidates fusion.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8488–8504, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Sagi and Rokach (2018) Omer Sagi and Lior Rokach. 2018.  Ensemble learning: A survey.  _Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery_ , 8.
  * Salazar et al. (2020) Julian Salazar, Davis Liang, Toan Q. Nguyen, and Katrin Kirchhoff. 2020.  Masked language model scoring.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 2699–2712, Online. Association for Computational Linguistics.
  * Sellam et al. (2020) Thibault Sellam, Dipanjan Das, and Ankur Parikh. 2020.  BLEURT: Learning robust metrics for text generation.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7881–7892, Online. Association for Computational Linguistics.
  * Shazeer et al. (2017) Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017.  Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.  In _5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings_. OpenReview.net.
  * Shazeer and Stern (2018) Noam Shazeer and Mitchell Stern. 2018.  Adafactor: Adaptive learning rates with sublinear memory cost.  In _Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmässan, Stockholm, Sweden, July 10-15, 2018_ , volume 80 of _Proceedings of Machine Learning Research_ , pages 4603–4611. PMLR.
  * Stability-AI (2023) Stability-AI. 2023.  Stablelm: Stability ai language models.  https://github.com/stability-AI/stableLM.
  * Sun and Qiu (2023) Tianxiang Sun and Xipeng Qiu. 2023.  Moss.  https://github.com/OpenLMLab/MOSS.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca.
  * Tiedemann and Thottingal (2020a) Jörg Tiedemann and Santhosh Thottingal. 2020a.  OPUS-MT – building open translation services for the world.  In _Proceedings of the 22nd Annual Conference of the European Association for Machine Translation_ , pages 479–480, Lisboa, Portugal. European Association for Machine Translation.
  * Tiedemann and Thottingal (2020b) Jörg Tiedemann and Santhosh Thottingal. 2020b.  OPUS-MT – building open translation services for the world.  In _Proceedings of the 22nd Annual Conference of the European Association for Machine Translation_ , pages 479–480, Lisboa, Portugal. European Association for Machine Translation.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aur’elien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023.  Llama: Open and efficient foundation language models.  _ArXiv preprint_ , abs/2302.13971.
  * Wang et al. (2022) Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi. 2022.  Self-instruct: Aligning language model with self generated instructions.  _ArXiv preprint_ , abs/2212.10560.
  * Xu et al. (2023) Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley. 2023.  Baize: An open-source chat model with parameter-efficient tuning on self-chat data.  _ArXiv preprint_ , abs/2304.01196.
  * Yidong et al. (2023) Wang Yidong, Yu Zhuohao, Zeng Zhengran, Yang Linyi, Heng Qiang, Wang Cunxiang, Chen Hao, Jiang Chaoya, Xie Rui, Wang Jindong, Xie Xing, Ye Wei, Zhang Shikun, and Zhang Yue. 2023.  Pandalm: Reproducible and automated language model assessment.  https://github.com/WeOpenML/PandaLM.
  * Yuan et al. (2021) Weizhe Yuan, Graham Neubig, and Pengfei Liu. 2021.  Bartscore: Evaluating generated text as text generation.  In _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , pages 27263–27277.
  * Zhang et al. (2020a) Jingqing Zhang, Yao Zhao, Mohammad Saleh, and Peter J. Liu. 2020a.  PEGASUS: pre-training with extracted gap-sentences for abstractive summarization.  In _Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event_ , volume 119 of _Proceedings of Machine Learning Research_ , pages 11328–11339. PMLR.
  * Zhang et al. (2020b) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2020b.  Bertscore: Evaluating text generation with BERT.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023.  Judging llm-as-a-judge with mt-bench and chatbot arena.

Appendix

##  Appendix A Implementation Details

#### PairRanker

We train our ranker for 5 epochs. We use the Adafactor optimizer Shazeer and
Stern (2018), with the maximum learning rate being 1e-5. The warm-up ratio is
5% with a linear learning rate scheduler. Our training batch size is 64. The
training finishes on a single RTX 8000 GPU in two days. The backbone of
PairRanker is Deberta-v3-large He et al. (2021). Unlike the mixture-of-experts
layer used in the work of Ravaut et al. (2022a), we employ a five-layer multi-
layer perceptron (MLP) with the hyperbolic tangent activation function. The
output dimension of the final layer is equal to the number of different
metrics. In practice, we have tried different special embedding combinations,
such as only feeding <candidate1> and <candidate2>, mean pooling
representation, and so on. And Finally, we found that concatenating <source>
with <candidate1>, and <source> with <candidate2> respectively achieves the
best performance. We also tried different loss types, like MSE losses and
ranking losses. And we find BCE is simply good enough.

#### GenFuser

We use Flan-T5-large and Flan-T5-xl (3b) to train the top-3 bart-score
selections as input and then apply it with PairRanker’s top 3 selections for
inference. We find Flan-t5-3b performs much better than the large version,
while flan-t5-xxl has marginal improvements yet being much larger and longer
to train.

##  Appendix B Conventional Tasks

To quantitatively understand how sub-optimal the default selections of
decoding methods are, we present an empirical analysis in Fig. 5 . Here we
look at three typical NLG tasks: summarization (CNN/DM), machine translation
(WMT18), and constrained text generation (CommonGen), with their popularly
used base models: PEGASUS Zhang et al. (2020a), Opus-MT Tiedemann and
Thottingal (2020a), and T5-large Raffel et al. (2020). We can see that the
default selections (yellow bars in Fig. 5; the top-beam generations) are much
worse than the oracle selections from the top 15 candidate generations for
each decoding method (blue bars).

Figure 5: The comparisions between different decoders and oracle selections.

Moreover, if we combine the results from the four methods as a larger
candidate pool, then the performance (green bars) of these NLG models can be
much improved. For example, the Rouge-2 score of PEAGUS can be improved by 57%
and the BLEU score for Opus-MT can be improved by nearly 80%, compared to
their top-beam performance. Simply put, the default selections (i.e., the
generations with the highest decoding scores) are much worse than the best
selections from a relatively small candidate pool. Therefore, we argue that it
is of significant importance to rerank generation candidates in order to
enhance the performance of LMs in NLG tasks.

Why do decoding algorithms often overlook generation candidates of better
quality? The lower quality of default selections is often attributed to the
exposure bias caused by the teacher-forcing paradigm in most auto-regressive
models. Plus, the greediness of the search process and the randomness in
sampling are also part of the reasons. Re-ranking has been a simple yet
effective post hoc approach to mitigate this gap. For instance, MLM-scoring
Salazar et al. (2020) uses an external LM such as BERT to estimate the quality
of a candidate without any supervision. SimCLS Liu and Liu (2021) trains a re-
ranker using a simple contrastive training objective, which encodes the source
text and each candidate output using the same encoder and scores each
candidate based on the cosine similarity between the embeddings. Another
successful approach is SummaReranker (SR) Ravaut et al. (2022a), which is
trained to improve the re-ranker for multiple metrics simultaneously.

##  Appendix C Additional Results on Conventional Tasks

In this section, we evaluate the PairRanker with conventional natural language
generation (NLG) tasks. Extensive experiments conducted on three NLG tasks
(i.e., summarization, translation, and constrained sentence generation)
demonstrate that PairRanker outperforms the baseline methods by a consistent
margin and is also compatible with very large language models such as GPT-3
(text-davinci-003). PairRanker not only outperforms the previous state-of-the-
art method SummaReranker on the summarization task, but also shows great
generalization performance in the other two NLG tasks, which are not evaluated
previously. In addition, our PairRanker can be transferred to improve GPT-3
results by 26.53% and 11.65% for CommonGen and WMT18 (zh-en) respectively,
even though our rerankers are not trained with any candidates decoded by GPT-3
models.

###  C.1 Tasks and data creation

We evaluate reranking methods on the following public dataset: CNN/DM,
CommonGen, and WMT18 (zh-en). The data statistics of these benchmarks are in
Table 7 (in Appendix).

#### CNN/DM

Hermann et al. (2015) is a dataset constructed from CNN and DailyMail
websites. It is first used for machine-reading and comprehension, and later
Nallapati et al. (2016) use it for abstractive summarization. Evaluation
metrics are Rouge-1, Rouge-2, and Rouge-L.

#### CommonGen

Lin et al. (2020) is a dataset used for generative commonsense reasoning. It
contains 79K commonsense descriptions where the language model is required to
compose a realistically plausible sentence from given concepts. Evaluation
metrics are BLEU and CIDEr.

#### WMT2018

Bojar et al. (2017) is a well-known dataset for evaluate machine translation.
Here we use the Chinese-English split for evaluation. Evaluation metrics are
BLEU.

###  C.2 Base models

For the summarization task on CNN/DailyMail dataset, we use the famous
PEGASUS-large Zhang et al. (2020a) and BART-large Lewis et al. (2020), which
have exhibited great ability for abstractive summarization. We use the public
fine-tuned checkpoint from Hugging face. For the generative commonsense
reasoning task on CommonGen dataset, we use the T5-large Raffel et al. (2020).
It’s one of the vanilla baselines reported in Lin et al. (2020). For the
Chinese-English translation task on WMT18 dataset, we use the public pre-
trained opus-mt checkpoint Tiedemann and Thottingal (2020b).

###  C.3 Evaluation setups

In this section, we talk about the training and testing paradigm of our
reranker, including how we construct the training, validation, and testing
dataset for our reranker, how we generate candidates, and what our experiment
focuses on.

Method $\downarrow$ Metric $\rightarrow$ | R-1 | R-2 | R-L | GainR1
---|---|---|---|---
BART | 44.48 | 21.21 | 41.60 | -
PEGASUS | 44.56 | 20.90 | 41.58 | -
Gsum | 45.94 | 22.32 | 42.48 | -
Gsum+RefSum | 46.18 | 22.36 | 42.91 | 1.18%
BART+SimCLS | 46.67 | 22.15 | 43.54 | 4.92%
PEGASUS+MLM-Scoring | 43.03 | 19.48 | 40.12 | -3.43%
PEGASUS+SummaReranker | 47.16 | 22.55 | 43.87 | 5.83%
PEGASUS+PairReranker (bubble) | 47.29 | 22.77 | 44.06 | 6.13%
PEGASUS+PairReranker (max wins) | 47.29 | 22.79 | 44.07 | 6.13%
PEGASUS+PairReranker (max logits) | 47.39 | 22.91 | 44.18 |  6.35%
GPT-3 (text-davinci-003) | 37.96 | 15.51 | 34.39 | -
GPT-3-oracle | 45.46 | 22.83 | 42.04 | 19.76%
GPT-3+MLM-Scoring | 38.13 | 15.09 | 34.32 | 0.45%
GPT-3+SummaReranker | 39.62 | 17.13 | 36.12 | 4.37%
GPT-3+PairReranker (bubble) | 40.41 | 17.44 | 36.79 | 6.45%
GPT-3+PairReranker (max wins) | 40.37 | 17.46 | 36.76 | 6.35%
GPT-3+PairReranker (max logits) | 40.48 | 17.54 | 36.84 |  6.64%
Table 4:  Model performance on CNN/DailyMail.

To construct the training dataset for the reranker, we need to ensure the base
model used to generate candidates on the training dataset should never have
seen these candidates. Otherwise, the reranker will be trained on the
candidates with higher quality compared to the candidates that it will be
tested on, which we found will result in fairly bad performance. Therefore,
following Ravaut et al. (2022a), we first fine-tune the original non-finetuned
pre-trained model on half of the training dataset, which gives us 2 half-
finetuned base models that each of them has only seen their own half of the
training dataset. Then we use them to generate candidates on their un-seen
half of the training dataset using the decoding method talked about before.
These generated candidates together form a whole training dataset with
generated candidates that resemble the quality during the inference stage.

During the inference stage, we directly adopt the public checkpoints that have
been finetuned on the whole training dataset. We generate candidates on the
validation and test datasets with this public checkpoint, which constitutes
the validation and testing datasets on which our reranker runs inference. We
use two decoding methods, beam search, and diverse beam search, in the
experiments, following the prior work of SummaReranker. We generate 15
candidates for each decoding method for both training and inference.

###  C.4 Main results

#### Overall performance in summarization.

Following the training and testing paradigm stated in section C.3, we briefly
report the test results on the CNN/DM dataset in Tab. 4. With fine-tuned
PEGASUS-large as the base model. our Max Logits method improves the
candidates’ quality by 6.35% in Rouge-1, which is higher than our baseline
SummaReranker. Besides, the performance gains in other metrics like Rouge-2
(9.62%) and Rouge-L (6.25%) are also obviously better.

#### Can PairReranker generalize to other generation tasks?

Yes. In order to test the task generalization ability of our method, we here
report the test results on CommonGen and WMT2018 (zh-en) in Tab. 5 and Tab. 6.
From the data in the table, our method also improves the candidates’ quality
significantly after reranking. Our Max Logits method obtain a 2.45%
performance gain in CIDEr on the CommonGen dataset and a 6.12% performance
gain in BLEU on the WMT2018 dataset. What’s more, it’s worth noting our bubble
run method achieves an even higher gain in CIDEr (2.91%).

Method $\downarrow$ Metric $\rightarrow$ | BLEU | CIDEr | Gain${}_{\text{CIDEr}}$
---|---|---|---
T5-large | 14.62 | 15.48 | -
T5-large+MLM-Scoring | 14.04 | 14.12 | -8.79%
T5-large+SimCLS | 14.5 | 14.99 | -3.17
T5-large + SummaReranker | 14.13 | 15.29 | -1.23%
T5-large + PairReranker (bubble) | 15.30 | 15.93 | 2.91%
T5-large + PairReranker (max wins) | 15.29 | 15.91 | 2.78%
T5-large + PairReranker (max logits) | 15.40 | 15.86 | 2.45%
GPT-3 (text-davinci-003) | 11.85 | 11.12 | -
GPT-3 + oracle | 20.34 | 19.26 | 73.20%
GPT-3 + MLM-Scoring | 12.56 | 11.66 | 4.86%
GPT-3 + SummaReranker | 13.71 | 13.21 | 18.79%
GPT-3 + PairReranker (bubble) | 14.39 | 13.85 | 24.55%
GPT-3 + PairReranker (max wins) | 14.32 | 13.76 | 23.74%
GPT-3 + PairReranker (max logits) | 14.63 | 14.07 |  26.53%
Table 5:  Model performance on CommonGen.

We also report the performance of SummaReranker on the two datasets. In
contrast to the great performance on summarization, SummaReranker seems to
fail to generalize well on other datasets. We also find that SummaReranker
obtains a decreased gain on the CommonGen dataset (-1.23% in CIDEr). The
improvement on the translation dataset is not obvious (0.57% in BLEU). We
hypothesize that this is because of the average length of the candidates and
the target text in these two datasets are all significantly smaller than the
one in summarization (see in Tab. 7). Therefore, the higher in-group
similarity brought by the shorter length makes it harder for SummaReranker to
capture their difference. On the contrary, our method with direct attention
between a pair of candidates could easily tackle this problem.

#### Can PairReranker generalize other large-scale models like GPT-3?

Yes. After training on an expert dataset, our reranker could directly be
applied to other models’ outputs selection for the same task. To support this,
we directly apply our three rerankers trained on the 3 main tasks respectively
to the GPT-3 outputs with proper task-specific prompts. We report the
performance gain in Tab. 4, 5, and 6. From the data reported in the table, we
could see that the quality of the GPT-3 outputs is improved by a large margin
compared to the average. Also, our performance gain is significantly larger
than the baseline SummaReranker. For example, on the GPT-3 data points sampled
from CNN/DM, our max logits method obtain a gain of 6.64%, whereas
SummaReranker only obtains a gain of 4.37%. And on the CommonGen’s, our method
obtains a gain of 26.53% and SummaReranker only obtains a gain of 18.79%.

Method $\downarrow$ Metric $\rightarrow$ | BLEU | Gain
---|---|---
Opus-MT | 19.29 | -
Opus-MT+MLM-Scoring | 16.35 | -15.24%
Opus-MT+SimCLS | 18.93 | -1.87%
Opus-MT+SummaReranker | 19.40 | 0.57%
Opus-MT+PairReranker (bubble) | 20.36 | 5.54%
Opus-MT+PairReranker (max wins) | 20.30 | 5.24%
Opus-MT+PairReranker (max logits) | 20.47 |  6.12%
GPT-3 (text-davinci-003) | 23.61 | -
GPT-3 + oracle | 36.11 | 52.94%
GPT-3+MLM-Scoring | 23.98 | 1.57%
GPT-3+SummaReranker | 25.08 | 6.22%
GPT-3+PairReranker (bubble) | 26.29 | 11.35%
GPT-3+PairReranker (max wins) | 26.36 |  11.65%
GPT-3+PairReranker (max logits) | 26.19 | 10.93%
Table 6:  Model performance on WMT18 (zh-en). Figure 6: Efficiency trade-off
with the number of pairwise comparisons

#### Can I make trade-offs between performance and number of comparisons?

Yes. Due to the high cost of full comparison methods, it’s necessary for us to
study the trade-off between the model performance and the number of
comparisons. For full comparison methods, we first initialize matrix $M$ in
Figure 4 to be all zeroes, Then every time of the comparison, we fill a
confidence cell that is zero before, then do the scoring and select the best
one based on the current information in the matrix. For bubble run, we run
multiple times of bubble run and select one that is chosen as the best one for
the most times. Each bubble cost $N$ comparisons. The trade-off results are
shown in Figure 6. From the results, we could see bubble run method could
achieve high performance with little cost. However, as the number of
comparisons increases, Max Logits scoring methods will surpass the bubble run
after a certain number of comparisons. We contend that the bubble run method
already reports a pretty good performance with $N-1$ times of comparison.
Therefore, most of the time, bubble run is a more efficient way to apply. If
you want to pursue the marginal improvements brought by more comparison, you
can also apply Max Logits method with parallel computing.

###  C.5 Model Further Study

Due to the order of the input format, changing the position of candidate 1 and
candidate 2 might also change the results (Sec. 3.3). In practice, we found
that by simply shuffling the order of candidate 1 and candidate 2, our
reranker could be consistent with itself more than 90% of the time.

We analyze the model’s relation between consistency as well as accuracy and
the absolute pair rank difference. The results are presented in Figure 7. From
the results, we could see that the model is better at classifying candidates
with a higher absolute rank difference, because they are supposed to be more
different.

Figure 7: Consistency and accuracy analysis for CNN/Daily Mail Dataset

##  Appendix D Dataset statistics

We analyze the basic statistics, including the number of examples and the
average words per example, of the 3 datasets. The data is presented in Table 7

Dataset | # Examples | # Words per example
---|---|---
Train | Val | Test | Source | Target
CNN/DM | 287k | 13k | 11,490 | 856.56 | 70.05
CommonGen | 67k | 4k | 1,497 | 4.20 | 12.92
WMT18(zh-en) | 25m | 2k | 3,981 | 83.48 | 30.95
Table 7:  Statistics of the three datasets.

##  Appendix E ChatGPT Comparison Prompting Template (GPT-Rank)

Template |  Instruction: ${instruction} Input: ${input} Candidate A: ${candidate1} Candidate B: ${candidate2} Given the instruction and input above, please compare the two candidates. You only have 4 choices to output: If you think A is better, please output: 1. A is better If you think B is better, please output: 2. B is better If you think both are good enough correctly give the answer, please output: 3. Same good If you think both are bad and do not follow the instruction, please output: 4. Same bad Do not output anything else except the 4 choices above. Output your choice below:
---|---
Comparison Option |  1\. A is better 2\. B is better 3\. Same good 4\. Same bad
Table 8: The template used for ChatGPT comparison ranking (GPT-Rank).