# Benchmarking Cognitive Biases in Large
Language Models as Evaluators

Ryan Koo1, Minhwa Lee1, Vipul Raheja2, Jong Inn Park1, Zae Myung Kim1,
Dongyeop Kang1
1University of Minnesota, 2Grammarly
{koo00017, lee03533, park2838, kim01756<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Large Language Models (LLMs) have recently been shown to be effective as
automatic evaluators with simple prompting and in-context learning. In this
work, we assemble 15 LLMs of four different size ranges and evaluate their
output responses by preference ranking from the other LLMs as evaluators, such
as System Star is better than System Square. We then evaluate the quality of
ranking outputs introducing the Cognitive Bias Benchmark for LLMs as
Evaluators (CoBBLEr)111Our project page:
https://minnesotanlp.github.io/cobbler, a benchmark to measure six different
cognitive biases in LLM evaluation outputs, such as the Egocentric bias where
a model prefers to rank its own outputs highly in evaluation. We find that
LLMs are biased text quality evaluators, exhibiting strong indications on our
bias benchmark (average of 40% of comparisons across all models) within each
of their evaluations that question their robustness as evaluators.
Furthermore, we examine the correlation between human and machine preferences
and calculate the average Rank-Biased Overlap (RBO) score to be 49.6%,
indicating that machine preferences are misaligned with humans. According to
our findings, LLMs may still be unable to be utilized for automatic annotation
aligned with human preferences.

##  1 Introduction

Figure 1: Our CoBBLEr pipeline to evaluate the 15 popular LLMs that are
instruction-tuned and trained with human feedback for their capabilities as
unbiased automatic evaluators.

Large language models (LLMs) (Brown et al., 2020; Ouyang et al., 2022) adapted
to follow various kinds of instructions have been popularly utilized for
several natural language tasks. The general standard for testing a model’s
capabilities is benchmarking its performance on static evaluation suites such
as Fan et al. (2019) and Wang et al. (2020). With the increased usage of
language models as general-purpose assistants, however, current task-specific
benchmarks are not sufficient to measure the quality of generated texts in the
wild.

Recent studies have shown that LLMs can serve as evaluators themselves: Wu &
Aji (2023) utilize LLMs as self-evaluators to automatically judge the quality
of open-ended generations and compare them with human judgments via an Elo-
score calculation. Other works, such as AlpacaEval (Li et al., 2023b), also
utilize LLMs, such as GPT-4 (OpenAI, 2023), as automatic evaluators to reduce
the time and cost overhead of human annotations. As noted by these works, such
automatic evaluation leaderboards have a number of limitations, including a
preference for long outputs or outputs that are more similar to the
evaluators’ generation qualities.

(a) Proportion of biased evaluations

(b) Heatmap of bias intensity

(c) Correlation with human judgment

Figure 2: Major findings of this work: the intensity of model biases as well
as the alignment between machine and human preferences. For (a), we show the
proportion of biased responses for each evaluator relative to the random
threshold. We scale each of the axes to the score of the most biased model. In
(b), we draw a heatmap of the scores relative to randomly choosing model
outputs. The scores were normalized via $z$-normalization using the mean and
std of Random as a reference point. A darker red indicates a stronger
intensity of bias, while a darker blue shade indicates more unbiased
evaluations. In (c), we show the average Rank-Biased Overlap (RBO) scores
between aggregated human preferences and each of the 15 LLMs. Higher RBO means
higher similarity.

In this work, we propose CoBBLEr, the Cognitive Bias Benchmark for evaluating
the quality and reliability of LLMs as Evaluators, as depicted in Figure 1. We
collect a set of 50 question-answering examples from two well-established
benchmarking datasets: BigBench (Srivastava et al., 2023) and Eli5 (Fan et
al., 2019). We then generate responses from each LLM in consideration
(totaling 50 responses) and prompt the models to evaluate their own and other
models’ responses. We assemble 15 of the best-performing models based on the
HuggingFace OpenLLM leaderboard (Beeching et al., 2023) as well as API-based
models and evaluate how these models reflect cognitive biases as evaluators.
We test six different biases to benchmark their evaluation quality and
categorize the model biases into two groups: (1) Implicit Biases to determine
the inherent biases that can be implicitly extracted from each model’s
evaluation from a uniform prompt, and (2) Induced Biases, which add
modifications to the original prompts akin to adversarial attacks, to induce
negative behaviors. We conduct a round-robin evaluation of all possible
${15\choose 2}$ pairs of response for all 50 QA examples, comparing every
possible unique pair among the 15 model responses. For every pairwise
comparison, we then prompt the models to select the response that is more
coherent and better aligned with a reference response. As shown in Figures
2(a) and 2(b), we find that the majority of the models strongly exhibit
several of the different biases, which may compromise the credibility of their
role as evaluators.222In total, 42K samples are analyzed across six biases
benchmarking each model for a complete 630K samples. Furthermore, we conduct
experiments for human preferences by crowdsourcing six human annotators and
collecting each of their rankings for a total of 300 annotations. From our
findings, we observe a low correlation between human and machine judgments via
Rank-Biased Overlap (RBO), indicating that machine and human preferences are
generally in low agreement.

Our core contributions are as follows:

  * •

A new benchmark (CoBBLEr) for evaluating LLMs to perform unbiased evaluations
within the QA setting.

  * •

An examination of an exhaustive list of evaluation biases that have not been
covered by previous studies. We find that most LLMs cannot perform as unbiased
evaluators, testing on 6 different cognitive biases.

  * •

A comprehensive lineup of models (sizing from $3B$ to $>$$175B$ parameters) as
evaluators, encompassing the current state-of-the-art language models covering
over 630k comparisons.

Based on our benchmark, we find that most models that are instruction-tuned or
trained on human feedback exhibit various cognitive biases when used as
automatic evaluators, and that may negatively impact the quality of
evaluations. Thus, we propose the use of our benchmark (CoBBLEr) for measuring
the capabilities of future language models as evaluators to enable unbiased
and reliable evaluations that are well-aligned with human preferences.

##  2 Related Work

#### LLMs as Evaluators.

Owing to the effectiveness of LLMs, many recent research works have
investigated their utility in various downstream tasks, such as machine
translation (Kocmi & Federmann, 2023), summarization (Shen et al., 2023; Gao
et al., 2023), code generation (Zhuo, 2023), writing assistance (Schick et
al., 2023; Raheja et al., 2023), factual consistency (Cohen et al., 2023;
Gekhman et al., 2023; Luo et al., 2023), and more. Schick et al. (2023);
Raheja et al. (2023). Additionally, many studies have investigated leveraging
LLMs for general-purpose NLG evaluation. For instance, Liu et al. (2023); Chen
et al. (2023); Wang et al. (2023a) have investigated the effectiveness of
GPT-4 and ChatGPT respectively, against existing reference-free evaluation
methods for NLG, whereas Fu et al. (2023) proposed an evaluation framework,
GPTScore, which utilizes the emergent abilities of LLMs to score generated
texts. Recently, Li et al. (2023a) and Zheng et al. (2023) conducted similar
experiments by employing LLMs as evaluators in a pairwise setting to judge
between two generations. Although these works present promising results for
LLMs as automatic evaluators, our work dives deeper into the limitations of
machine evaluations. We take a closer look at machine artifacts that could be
detrimental to data quality by benchmarking an exhaustive list of biases
impacting LLMs-as-evaluators.

#### LLM Evaluation Benchmarks.

It is becoming increasingly challenging to evaluate open-source LLMs as they
become more powerful and performant. As a result, there has been an increasing
need to develop better evaluation benchmarks for measuring the performance of
LLMs. However, most of these benchmarks, such as LM-Eval-Harness (Gao et al.,
2021), MMLU (Hendrycks et al., 2021), HELM (Liang et al., 2022) and BIG-Bench
(Srivastava et al., 2023), have focused only on measuring the LLMs’ core
capability on a varied, but confined set of tasks, and not their capabilities
as evaluators. Numerous works have also attempted to develop benchmarks based
on the capability of LLMs to conduct NLG evaluations by evaluating candidate
texts in isolation or comparing two or more candidate texts in accordance with
specific evaluation aspects. However, they are restricted because they still
require token-level comparisons between the input text and reference (Zhang et
al., 2020; Tang et al., 2023). Our work aims to resolve these issues by
creating a comprehensive, publicly available benchmark (CoBBLEr) across a wide
variety of LLMs to measure their abilities as evaluators of generated text
while deeply investigating the numerous implicit and induced cognitive biases
in LLM evaluators. Our work in this direction overlaps with Bai et al. (2023),
who propose a Language-Model-as-an-Examiner benchmark to evaluate four
foundational models on open-domain question-answering tasks. However, our
works differ in examining the impact of cognitive biases in LLM evaluations
while (Bai et al., 2023) test the ability of LLMs to formulate and grade their
own questions. Our work also directly overlaps with Zheng et al. (2023), who
propose LLM-as-a-judge to study the capability of LLMs to emulate human
preferences. Our experimental setups are similar, but we highlight key
differences. We cover a wider demographic of current popular language models
and an overall different focus on QA as opposed to other domains such as math
and reason. Furthermore, our benchmark emphasizes a wider range of biases
(implicit/induced) to better describe machine artifacts when used as automatic
evaluators. Specifically, CoBBLEr measures the extent to which each LM-as-
evaluator is impacted in each decision by certain artifacts within prompts
(i.e., prompting format, prompt information) over a comprehensive list of
cognitive biases.

#### Cognitive Biases in LLMs.

While biases have been well-known to exist in LLMs (Wang et al., 2023b; Talboy
& Fuller, 2023; Wu & Aji, 2023), many recent works investigating the behaviors
of LLMs have also uncovered similarities with cognitive biases. Some recent
works (Zhao et al., 2021; Liu et al., 2022; Lu et al., 2022) have shown that
the order of training examples in GPT-3 could lead to differences in accuracy
between near chance and near state-of-the-art. Jones & Steinhardt (2022)
captured failures in GPT-3 and Codex and found that error patterns of LLMs
resemble cognitive biases in humans. Our work overlaps with these in some of
the biases we cover, but we present a much more holistic and comprehensive
evaluation of LLMs, both when they are used as predictors, as well as
evaluators of text quality. Along this aspect, while our work is close to Wu &
Aji (2023), who investigate biases related to fabricated factual and
grammatical errors while using GPT-4 as an evaluator, our work is much more
comprehensive in terms of the number of LLMs analyzed, the types of biases
analyzed, and creation of an open benchmark.

Bias |  Bias Behavior |  Example
---|---|---
Order Bias |  The tendency to give preference to an option based on their order (e.g. first, second, or last) |  System Star: $x$ System Square: $y$
System Square: $y$ System Star: $x$
Compassion
Fade |  The tendency to observe different behaviors when given recognizable names as opposed to anonymized aliases. |  Model Alpaca: $x$ Model Vicuna: $y$
Model Vicuna: $y$ Model Alpaca: $x$
Egocentric Bias |  The inclination to prioritize one’s own responses regardless of response quality. |  Model Star (You): $x$
Model Square: $y$
Salience Bias |  The tendency to prefer responses based on the length of the response (more often preferring shorter responses or longer responses). |  System Star: The quick brown fox jumps over the lazy dog.
System Square: The fox jumped.
Bandwagon Effect |  The tendency to give stronger preference to majority belief without critical evaluation. |  85% believe that System Star is better.
Attentional Bias |  The inclination to give more attention to irrelevant or unimportant details. |  System Square likes to eat oranges and apples
Table 1: Definition and examples of each bias type in CoBBLEr. In the examples
for each bias type, we display their characteristic format and bold answers
that are indicative of behavior that is influenced by the bias. For example,
the Order bias shows both orderings of responses $x$ and $y$, but displays an
inconsistent answer by choosing only the first-ordered system. Furthermore, we
pair the example in Compassion with Order (System Star/System Square vs.
Alpaca/Vicuna) to demonstrate differing behavior when real model names are
used.

##  3 CoBBLEr: Cognitive Bias Benchmark for LLMs as Evaluators

The following criteria are used to select each type of evaluation bias:

  * •

General Applicability. Text evaluation tasks should be generalizable to most
prompting scenarios; tasks that observe too specific subtleties within the
prompt are not helpful.

  * •

Impartiality. The prompt should not involve any leading statements to extract
some desired quality of the evaluations

  * •

Memorylessness. The current evaluation instance should not rely on any
previous behaviors. Each instance should be self-contained when extracting
each bias metric.

We carefully hand-select these biases based on the above three criteria so
that they are widely applicable to most evaluation settings in assessing the
performance of language models as automatic evaluators. Table 1 summarizes
definitions of each bias type along with examples in CoBBLEr. We categorize
our benchmark into two main classes: (1) Implicit and (2) Induced Biases. For
Implicit biases, we feed a general prompt that shows system outputs in a
pairwise manner to extract any biased behaviors that implicitly exist within
the model’s evaluations without giving the model any additional information
other than the instructions and the system outputs. For Induced biases, we
feed prompts geared towards each different bias, which we describe as “induced
behaviors” that achieve a similar goal to adversarial attacks, such as
presenting false information, to measure the impact on each of the models’
evaluation quality. We note that due to the nature of induced biases,
criterion 2 is not entirely fulfilled, but we specifically choose ones that
are still generally observable in an evaluation setting.

###  3.1 Implicit Biases

We categorize biases as “implicit” if the biases inherent to the model-as-an-
evaluator can be witnessed without including any additional information other
than instructing the model to judge the quality of two given generated texts
and declaring one the winner.

Order Bias is an evaluation bias we observe when a model tends to favor the
model based on the order of the responses rather than their content quality.
Order bias has been extensively studied (Jung et al., 2019; Wang et al.,
2023a; Zheng et al., 2023), and it is well-known that state-of-the-art models
are still often influenced by the ordering of the responses in their
evaluations. To verify the existence of order bias, we prompt both orderings
of each pair and count the evaluation as a “first order” or “last order” bias
if the evaluator chooses the first ordered (or last ordered) output in both
arrangements respectively.

Compassion Fade (Naming). (Butts et al., 2019; Västfjäll et al., 2014) is a
cognitive bias that denotes a decrease in empathy as the number of
identifiable individuals increases. To this phenomenon, we present
real/identifiable names associated with each response to each evaluator
instead of anonymous aliases (e.g. System A). To analyze this bias, we
determine the evaluator to be affected if they exhibit different behaviors
from when anonymous aliases were used as a result of using recognizable names.
Thus, an unbiased evaluator would make evaluations similar to when anonymized
names were responses.

Egocentric Bias (Self-Preference). (Ross & Sicoly, 1979) is a cognitive bias
that refers to the tendency to have a higher opinion of oneself or to more
easily accept ideas if they match one’s own. We utilize this definition to
measure how often evaluators prefer their own responses in both the anonymized
and named cases. We define an evaluator to be egocentrically biased if, for
each instance, the evaluator prefers its own response over others. We note
that an unbiased evaluator would choose between themselves and other comparand
models equally in proportion. However, we highlight that some models would
naturally generate higher quality responses (e.g., GPT4 vs. RedPajama),
resulting in a stronger inclination for such evaluators to choose their own
responses.

Salience Bias (Length). (Schenk, 2010; Zheng et al., 2023) The evaluator tends
to favor responses that are either shorter or longer in length. An unbiased
evaluator would be split evenly between responses that are shorter or longer
in length. We examine this bias by looking at evaluations in which a model
preferred a response that is either shorter or longer in token length.

###  3.2 Induced Biases

We categorize a bias as “induced” when it requires modifications to the
primary prompt or the inclusion of additional information with the original
instructions. We specifically look to test the robustness of each of the
models as evaluators by introducing false or off-topic information and
examining the impact that these setups have on the quality of their role as
evaluators. For both of the induced biases below, we would expect an unbiased
evaluator to generally pick responses highlighted by Bandwagon and Distraction
around 25% of the time given by an empirically calculated threshold (Random)
by randomly choosing model responses.

Bandwagon Effect. (Schmitt-Beck, 2015) The evaluator’s preferences are
influenced by the collective preference rather than being based on their own
independent judgments. We add an additional sentence after the initial
instruction stating a fake statistic by choosing one of the comparand outputs
as preferred by a majority of people, such as “85% believe that System Star is
better.”. To validate that a model was affected by this bias, we prompt each
pair twice, stating each model output in the statistic to examine whether the
evaluator conforms to the majority.

Attentional Bias (Distraction). In addition to the original instruction, we
follow a similar setup from Shi et al. (2023) where we include irrelevant
information about one of the comparand models to test the ability of
evaluators. For example, we include a meaningless sentence such as ”System
Star likes to eat oranges and apples.” We identify the evaluator to be
distracted if it prefers the model mentioned in the distraction or if its
valid response rate significantly drops. We repeat this setup twice for each
pair (once for each model in the pair) to ensure a fair evaluation.

##  4 Experiment Setup

In this section, we discuss our evaluation framework for benchmarking each of
the different biases in LLMs as evaluators for text quality comparison. Figure
1 describes the pipeline for our experiments. We first generate responses from
various LLMs considered for this study (Section 4.1) and present them in a
pairwise fashion for quality evaluation by each evaluator model (Section 4.2).
In Section 4.3, we describe our setup for the human preference study.

###  4.1 Datasets and Models

We choose two widely used datasets employed to train and benchmark
instruction-tuned models.

Eli5 (Fan et al., 2019) is a long-form question-answering dataset constructed
from 270$k$ threads from the “Explain Like I’m Five” Reddit forum. The online
forum consists of a community for individuals to ask various questions, and
answers are provided in a format that is comprehensible to five-year-olds,
along with assigned scores based on community votes. For our purposes, we only
utilize the questions and their highest-rated answers to generate responses
and benchmark automatic evaluators for text-generation quality.

BigBench (Srivastava et al., 2023) is a collection of benchmarks that look to
probe the abilities of language models over a diverse range of tasks. We
specifically utilize the strategyQA (Geva et al., 2021) dataset, which was
constructed by crowdsourcing questions from writers as well as their responses
with short justifications. We choose the strategyQA dataset to generate
responses that require multi-step reasoning to effectively benchmark the
ability of models to comprehend and compare the quality between two different
explanations.

We specifically only choose corpora in the Question-Answering (Q/A) domain for
ease of use in generating responses. As we are looking to test the ability of
language models to perform as unbiased evaluators to judge response quality
and correctness, the Q/A response format presents the most natural setting for
these comparisons. Specifically, we choose a set of 50 question-answering
examples from the Eli5 and BigBench (taking 25 random examples from each),
generate responses from each language model for each question (a total of 50
responses per model), and prompt them to evaluate the generated responses.

Models We study the behaviors of 15 different models organized into 4 groups,
covering a wide range of model sizes from $3B$ parameters to over $175B$. In
Table 2 from top to bottom, we evaluate GPT-4, ChatGPT, and InstructGPT
(OpenAI, 2023) that comprise of the largest group of models with size
$>$$100B$ parameters. In the second group containing models of size $>$$40B$
parameters, we examine LLaMAv2 (Touvron et al., 2023), LLaMA (Touvron et al.,
2023), Cohere, and Falcon (Almazrouei et al., 2023). The third group is
comprised of models $>$$10B$ parameters, which include Alpaca (Taori et al.,
2023), Vicuna (Chiang et al., 2023), OpenAssistant (Köpf et al., 2023),
DollyV2 (Conover et al., 2023). Finally, the last group consists of models
with size $<$$10B$ parameters comprised of Baize (Xu et al., 2023b), Koala
(Geng et al., 2023), WizardLM (Xu et al., 2023a), MPT (Team, 2023), and
RedPajama (Computer, 2023).

###  4.2 Text Evaluation Setting

Response Generation. Figure 1 demonstrates our generation and evaluation333We
define “models” and “evaluators” interchangeably pipeline for CoBBLEr. We
first generate the responses from each of the models by prompting 50
instructions from the combined dataset, which we post-process to extract only
the response from all models, to have a total of 750 generations. We note that
for chat models, we slightly alter the instruction prompt but keep the general
instruction template the same for uniformity.

Pairwise Evaluation. After we collect all the model responses, we then prompt
each evaluator to compare the generations in a pairwise manner. We generate
all unique pairs amongst all models for each of the 50 instances, creating a
total of 5250 instances for each evaluator to rank. To evaluate whether an
evaluator is biased or not, we run each pairwise instance twice by switching
their order for the second evaluation call, generating a total sample size of
10500 for each bias type (totaling 42K samples evaluated for each evaluator as
Salience and Egocentric can be directly extracted from Order). We prompt each
evaluator by giving two responses that are randomly shuffled and anonymize the
model associated with the response with an alias such as “System Star” or
“System Square.” To reduce ambiguity, we ask the evaluators to choose only one
of the two models to have a clear winner and, furthermore, include a specific
response format. We specify guidelines444The exact evaluation prompt formats
for each bias benchmark are viewable in Appendix B within the evaluation
prompts by asking the evaluator to compare generations based on the coherence
of each of the responses in terms of correctness of content and alignment to
the instruction/reference provided. Within the evaluation prompt, we include
the original generation instruction and reference as context but keep a more
open-ended setting for LLMs to judge responses they prefer more naturally. We
then post-process each “eval-gen” via pattern matching and label each output
as a valid or invalid response, such that if a response is valid, we give one
point to the preferred system. We additionally conduct a list-wise ranking
with $N=4$ models. However, we find that most LLMs in the $<$$40B$ range have
trouble generating valid rankings (Appendix C). This may be due to the
increased complexity of the task (Dziri et al., 2023) where the ranking of $N$
generations may become much more difficult as $N$ gets larger (since the task
complexity increases).

Benchmarking. The benchmark is carried out in a leaderboard style in which the
evaluator with the lowest proportion of biased responses across all the
different benchmarks is identified as the “least biased”. As the comparisons
are limited to a pair-wise fashion, and in order to isolate potential biases
from other bias benchmarks, we identify an evaluator as biased by empirically
calculating a threshold via random selection as well as evaluating each
pairwise instance twice for validation. For example, in the Order benchmark,
each pair is evaluated twice in which both orderings are viewed (i.e. System
Star is shown ordered first, then System Square is shown ordered first). We
then randomly select a model in each response pair and measure the percentage
of first-order biases, where the first-ordered model is chosen in each pair,
and similarly measure last-order bias. We calculate the random selection
threshold for each bias benchmark, and models above these thresholds are
identified to exhibit the said bias.

We carry out this experiment for each bias that requires modifications555Some
biases such as Salience do not require any modifications to the original
prompt as they can be directly extracted in tandem with Order. to the original
prompt and process their evaluation outputs (we refer to these as “eval-gens”)
to examine their preference behaviors. The prompts for each of the different
bias benchmarks are described in Appendix B, and the parameters utilized for
response.

###  4.3 Human Preference Study

#### Collecting Human Preferences in $\mathbf{N=15}$-rankwise Setting

We investigated potential relationships between human evaluations and those
conducted by models on generated texts. For this, we gathered human
preferences using crowdsourcing. We engaged six workers on Amazon Mechanical
Turk (AMT) workers, all of whom possessed at least a US high school diploma
and achieved Human Intelligence Task (HIT) approval rates greater than 99%.
Each worker evaluated 50 sets of instances, with each instance comprising 15
responses generated by 15 distinct models. The workers then ranked these texts
based on their preference, from top (highest quality) to bottom (lowest
quality), while considering two main criteria: (1) fluency concerning the
given instruction sentence in an instance and (2) alignment and coherency with
both instruction and reference sentences in the instance. Further details on
worker recruitment, payment, and the interface design for these experiments,
are described in Appendix D.1 and D.3.

#### Measuring Human-LLM Evaluation Similarity

We calculate the Rank-Biased Overlap (RBO) 666The implementation of RBO was
referenced from https://github.com/changyaochen/rbo. score (Webber et al.,
2010) to measure the agreement between two ranked outputs by humans and
models. RBO, which can vary from 0 (non-conjoint) to 1 (identical), assigns
more weight to the top $k$ items in the lists being compared. Given our
assumption that workers will likely place the best-quality generated texts at
the top of their ranked lists, RBO is a fitting metric for our experimental
setup. We adopted a parameter of $p=0.8$, which concentrates 86% of all
weights on the top 5 list positions, following Webber et al. (2010). Further
mathematical details of RBO can be found in Appendix D.2.

We analyzed the extent of similarity between human preferences and model
evaluations in ranking generated texts across 15 different LLMs. To achieve
this, we proceeded with specific steps of standardizing rankings across
different annotators, enabling us to quantify the overall similarity between
human and LLM evaluations. First, we counted instances where one model
received a higher ranking than another model in pairwise comparisons,
resulting in ${15\choose 2}$ such comparisons for each instance. We then
aggregated these counts across six annotators, which allowed us to identify a
normalized ranking of 15 LLM-generated texts for each instance. Applying
analogous procedures to the rankings evaluated by the 15 models themselves, we
established a corresponding set of 15 models for each instance. Finally, we
computed the average top-5 weighted RBO between the aggregated ranking of
human preference and that of LLM evaluations across all 50 instances, as a
final RBO between human preference and model evaluation.

#### Identifying Biases in Pairwise Human Preference

We also investigated the impact of specific biases 777Other types of biases
such as Compassion Fade and Egocentric Bias cannot be applied to human cases.
We tested the effect of Salience Bias from the Order Bias experiment setup.
(Order Bias, Salience Bias, Bandwagon Effect, and Attentional Bias) within
human preferences in pairwise settings, by sourcing AMT workers. More details
on AMT recruitment, study designs, and payments for the pairwise setup are
described in Appendix D.1 and D.3.

The evaluation procedure for these human biases mirrored that described in
Section 4.2. However, due to the vastness of the pairwise model comparison
settings, we first randomly selected 25 of the 50 total instances. Then for
each instance, we randomly paired 15 models and created another 15 pairs by
reversing their order, resulting in 30 pairs in total. This finally totals 750
pairs for all 25 instances.

After collecting all annotations for each bias, we calculated the average IAA
using the RBO for each bias. Each instance consisted of uniquely (but
randomly) sampled model pairs, with some models appearing multiple times.
Hence, we determined the rank of each model in the sampled pairs by
calculating the ratio of the model’s “win” to its total appearances. With this
data, we re-ranked each model in the sampled pairs per instance. Afterward, we
computed the mean RBO among the ranked model lists from each group of three
AMT workers per instance. We then averaged these RBO values over all 25
instances. Finally, we computed the bias proportion for each annotator by
dividing the number of biased pairwise samples by 15. Following these steps,
we aggregated the bias proportions across all annotators, highlighting the
overall influence of bias on human preference in pairwise selections.

(a) Average bias scores by model size

(b) Breakdown of the proportional impact of each bias

Figure 3: Overview of performance across all of the bias benchmarks
categorized into 4 size groups from results in Table 2. The red-dotted line
denotes the average threshold taken from the calculated Random in which the
average scores were taken by summing their scores from Table 2 and then taking
their average.

##  5 Results and Discussion

For each bias, we analyze the performance of each of the 15 models as
evaluators of generated responses in the QA setting. We provide details of
LMs-as-evaluators relative to the Random baseline on each of our bias
benchmarks in Table 2, and Fig. 2(b) and 4, showing the intensity of each bias
as well as the distribution of the biased responses. We provide a visual
breakdown of the proportional impact of the average performance of each model
as unbiased evaluators in Fig. 3. On average, we see that models within the
$10B$ size range are most affected by each of the bias benchmarks in Fig.
3(a). This can be attributed to the Bandwagon Effect and Attentional Bias
benchmarks, in which the induced biases contribute to almost half of their
average bias score as seen in Fig. 3(b). Furthermore, we see that, on average,
the implicit biases contribute relatively similarly to each model’s overall
bias scores, indicating that scaling up the model size does not reduce
implicit biases in evaluators.

###  5.1 Bias Analysis

#### Implicit Biases

We first examine the performance of each evaluator on the implicit bias
benchmarks for Order Bias, Compassion Fade, Salience Bias and Egocentric Bias.
For the Order Bias benchmark in Table 2, we observe that most models (11/15)
tend to be drawn towards either the first- or last-ordered model in each of
the pairwise comparisons. Amongst the models influenced, we further observe
that the majority of models tend to prefer the first-ordered response,
especially within the second size group ($>$$40B$), in which the first-ordered
system was strongly favored in over 50% of comparisons. In contrast, we see
that the fourth size group($<$$10B$), consisting of the smallest models, tends
to prefer the last-ordered response.

For Compassion Fade, we show the real model names to the evaluators instead of
the anonymized aliases. However, this bias is difficult to gauge solely based
on the random threshold. Therefore, we further compare the impact of model
names vs. anonymized aliases by jointly comparing the results from Compassion
Fade with the ones from Order Bias. In essence, for a model to not be
influenced in its decisions by real model names, we expect the results for
Compassion Fade to be relatively similar to the Order Bias benchmark. However,
we see in Table 2 that all models are dramatically influenced by real model
names, where, surprisingly, the bias is largely increased for the largest
group of models ($>$$100B$), and in contrast, the smallest models ($<$$10B$)
where it is mostly reduced. This difference in the degree of impact that Order
Bias and Compassion Fade have on each model can be viewed in Fig. 4 and Table
2.

Next, we examine the tendency for models to choose their own responses or
Egocentric Bias in both the aliased and named cases. For the anonymized
aliases, the largest models, with the exception of InstructGPT, generally tend
to prefer their own responses ($>50\%$). Additionally, although Koala has a
low valid response rate, it also tends to prefer its own responses in cases
where it produces a valid preference. When including real model names, we see
a large drop in self-preference for models in the largest size group
($>$$100B$) models, which see a large increase in bias for each position.
However, on average, we see an increase in self-preference with real model
names amongst the smaller models in the size range $>$$10B$ and $<$$10B$,
suggesting that real model names draw smaller models to prefer their own
responses regardless of text quality more often compared to anonymized
aliases.

For Salience Bias, we observe that the larger models in the first and second
size groups are drawn more strongly to longer responses, which align with
findings from other works (Wu & Aji, 2023; Zheng et al., 2023). However,
smaller models (excluding MPT) tend to be less influenced by the length of the
responses they are comparing, suggesting that smaller models in the third and
fourth size groups are less susceptible to the text’s lengths.

Lastly, we address specific models such as LLaMAv2, LLaMA, DollyV2, and Koala
that show abnormal results on most of the benchmarks. This can be attributed
to their low valid response rates, which are displayed in Table 7 that list
the average percentages in which models return a valid choice between “System
Star“ or “System Square“. This may be explained by our prompting format or the
capabilities of the model themselves, in which models with a particularly low
valid response rate may have difficulty understanding the instructions
provided. 888We provide example evaluation generations for each model with our
code https://github.com/minnesotanlp/cobbler

Model |  Size | Order | Comp. | Egoc. |  Sal. |  Band. |  Attn.
---|---|---|---|---|---|---|---
|  |  First |  Last |  First |  Last |  Order |  Comp. |  |  |
Random |  - |  0.24 |  0.25 |  0.24 |  0.25 |  0.24 |  0.24 |  0.5 |  0.25 |  0.25
GPT4 |  - |  0.17 |  0.06 |  0.46 |  0.33 |  0.78 |  0.06 |  0.56 |  0.0 |  0.0
ChatGPT |  175B |  0.38 |  0.03 |  0.41 |  0.25 |  0.58 |  0.17 |  0.63 |  0.86 |  0.06
InstructGPT |  175B |  0.14 |  0.24 |  0.29 |  0.19 |  0.28 |  0.27 |  0.66 |  0.85 |  0.54
LLaMAv2 |  70B |  0.47 |  0.08 |  0.09 |  0.17 |  0.06 |  0.0 |  0.62 |  0.04 |  0.03
LLaMA |  65B |  0.61 |  0.0 |  0.0 |  0.0 |  0.0 |  0.02 |  0.42 |  0.0 |  0.01
Cohere |  54B |  0.33 |  0.17 |  0.38 |  0.27 |  0.27 |  0.15 |  0.60 |  0.82 |  0.14
Falcon |  40B |  0.74 |  0.03 |  0.09 |  0.18 |  0.05 |  0.11 |  0.59 |  0.28 |  0.40
Alpaca |  13B |  0.0 |  0.82 |  0.23 |  0.29 |  0.18 |  0.39 |  0.47 |  0.75 |  0.81
Vicuna |  13B |  0.32 |  0.17 |  0.17 |  0.15 |  0.27 |  0.45 |  0.53 |  0.81 |  0.78
OpenAssist |  12B |  0.56 |  0.11 |  0.03 |  0.22 |  0.15 |  0.06 |  0.49 |  0.72 |  0.82
DollyV2 |  12B |  0.0 |  0.0 |  0.0 |  0.0 |  0.0 |  0.0 |  0.0 |  0.0 |  0.0
Baize |  7B |  0.0 |  0.95 |  0.21 |  0.32 |  0.02 |  0.36 |  0.49 |  0.82 |  0.24
Koala |  7B |  0.24 |  0.01 |  0.0 |  0.11 |  0.48 |  0.86 |  0.55 |  0.13 |  0.1
WizardLM |  7B |  0.08 |  0.64 |  0.22 |  0.34 |  0.14 |  0.29 |  0.53 |  0.76 |  0.27
MPT |  7B |  0.49 |  0.1 |  0.11 |  0.27 |  0.21 |  0.25 |  0.63 |  0.95 |  0.52
RedPajama |  3B |  0.08 |  0.38 |  0.16 |  0.33 |  0.04 |  0.06 |  0.52 |  0.18 |  0.17
Table 2: A comparison of 15 models with different ranges of model sizes across
six different bias benchmarks. In terms of making unbiased comparisons, a
higher score indicates poorer performance, while a lower score indicates
stronger performance. Each metric includes a Random that is empirically
calculated by randomly choosing models in each pairwise instance. For Order
Bias and Compassion Fade, First indicates the proportion of responses
preferring the first ordered response and Last for the last ordered response.
For Salience Bias, models with scores less than 0.5 prefer responses with
fewer tokens, and scores above 0.5 prefer responses with more tokens. The
background color of each metric is determined by the difference between the
value and the corresponding Random metric (darker shade indicates stronger
bias).

#### Induced Biases

Next, we evaluate the performance of each evaluator on the induced bias
benchmarks: Bandwagon Effect and Attentional Bias. For Bandwagon Effect, we
observe that almost all models (11/15) are heavily influenced by irrelevant
statistics regarding majority preference in which $>70\%$ of evaluations on
average followed the bandwagon preference regardless of text quality. Although
we only included a simple fake statistic (e.g. 85% of people preferred “System
Star“), we see that evaluators can be heavily influenced by this external
information that heavily impairs their ability to make fair comparisons.
Notably, we see that models with a particularly low valid response rate (e.g.
LLaMA, Koala) were not so impacted by the bandwagon effect, although this can
be attributed to the limited evaluations that were properly generated.

For Attentional Bias, we see that around half of the models’ rankings are
influenced by irrelevant information. Specifically, we see that models in the
third size group ($>$$10B$) were the most strongly impacted by the distracting
information, with $>80\%$ of evaluations being counted as distracted. On the
other hand, API-based models such as ChatGPT and Cohere remained robust
against these distractions in their rankings. We include the list of
distractions we use in Appendix B.

Figure 4: Proportion of responses that were labeled bias for each bias benchmark. We visualize the distribution of the 15 models tested that varies by the y-axis. The red dashed line indicates the Random threshold for each bias benchmark that serves as a litmus between biased and unbiased LMs-as-evaluators. The spread on the x-axis is randomly distributed for visual clarity. Model | Size | Order | Compassion |  Egocent. |  Salience |  Bandwag. |  Attent. |  Avg. Valid
---|---|---|---|---|---|---|---|---
|  | First | Last | First | Last | Order | Comp. |  |  |  |  Responses
LLaMAv2 | 70B | 0.47 | 0.08 | 0.09 | 0.17 | 0.06 | 0.0 |  0.62 |  0.04 |  0.03 | 0.54
| 13B | 0.82 | 0.04 | 0.09 | 0.19 | 0.07 | 0.0 |  0.79 |  0.28 |  0.28 | 0.86
| 7B | 0.98 | 0.0 | 0.25 | 0.33 | 0.01 | 0.02 |  0.49 |  0.42 |  0.02 | 0.98
Vicuna | 33B | 0.95 | 0.0 | 0.20 | 0.38 | 0.03 | 0.25 |  0.84 |  0.69 |  0.26 | 0.99
| 13B | 0.32 | 0.17 | 0.17 | 0.15 | 0.27 | 0.45 |  0.53 |  0.81 |  0.78 | 0.87
| 7B | 0.58 | 0.04 | 0.14 | 0.0 | 0.20 | 0.64 |  0.58 |  0.50 |  0.61 | 0.86
Table 3: Performance comparison in proportion to their model scale. We view
the overall scores across each of the bias benchmarks as well as their valid
response rates.

###  5.2 Model Size

We conduct a supplementary experiment analyzing the impact of each bias for
different models scaled by size in Table 3. We present results from a range of
model sizes with LLaMAv2 and Vicuna. Interestingly, we see that the valid
response rate within LLaMAv2 goes down as the model size is scaled up, but the
impact of each bias greatly increases as the model size is scaled down (with
the exception of Salience Bias). On the implicit bias benchmarks, LLaMAv2
exhibits more robust performance with the proportion of responses affected by
each bias Salience Bias in which longer responses are much more strongly
preferred. For the induced bias benchmarks, a similar trend is viewed in which
the effect of each bias on the model as an evaluator is dampened in
correlation to the model scale. On the contrary, Vicuna exhibits a stronger
valid response rate as the model size is scaled; however, certain implicit
biases are much more amplified, such as Order Bias and Salience Bias. For
implicit biases, Vicuna tends to prefer itself when actual model names are
used as size is scaled smaller while tending to prefer much more verbose
responses as model size is scaled higher. Across the induced biases, Vicuna
performs more resiliently proportionally to scale, although still strongly
influenced by Bandwagon Effect but much less affected by Attentional Bias. We
include another visualization correlating the overall performance on each of
the bias benchmarks with model size for the main results in Figure 3(a).

Instruction: Did people in Korea under Japanese Rule watch a lot of Iron Chef? |  Instruction: Why classical music still sounds good today (after four hundred years), but lots of music from even ten years ago sounds lame?
---|---
GPT4 |  Cohere |  Vicuna |  MPT |  Human |  GPT4 |  Cohere |  Vicuna |  MPT |  Human
GPT4 |  Baize |  GPT4 |  Baize |  GPT4 |  GPT4 |  Instruct |  Vicuna |  Baize |  Baize
WizardLM |  GPT4 |  Cohere |  WizardLM |  Vicuna |  ChatGPT |  Vicuna |  WizardLM |  WizardLM |  Vicuna
ChatGPT |  WizardLM |  Baize |  Alpaca |  Cohere |  Falcon |  WizardLM |  Baize |  ChatGPT |  Koala
Baize |  Cohere |  Instruct |  MPT |  ChatGPT |  Baize |  Alpaca |  Koala |  Vicuna |  Instruct
Cohere |  Alpaca |  Dolly |  GPT4 |  WizardLM |  OpenAssist |  Koala |  ChatGPT |  Instruct |  ChatGPT
Table 4: Two examples of the (top-5) rankings for each LM-as-evaluator of the
four model sizes compared to the average human rankings. We calculate the
rank-wise scoring for each LM-as-evaluator by aggregating the number of “wins”
for each model from the pairwise comparisons and construct a list-based
ranking. We highlight each ranking in which the ranking of LM-as-evaluator
overlaps with the human rankings. Full ranking data can be viewed on our
project page.

###  5.3 Agreement with Human Preferences

#### N-rank wise Preference

We report an average inter-annotator agreement (IAA) score, determined using
the average RBO among the six AMT workers, of 0.478. This signifies a modest
but reasonable consensus among workers in ranking the LLM outputs, given the
challenges of ranking all 15 LLM-generated outputs.

The average RBO score between human and model preferences is 0.496. This
indicates that, on average, 49.6% of the ranks of 15 LLM-generated texts are
overlapped between human preferences and model evaluations. This finding
points out that model evaluations do not closely align with human preferences
regarding their reasoning behind ranking the quality of LLM-generated texts.

Figure 2(c) presents the average RBO scores for each of the 15 models compared
against the aggregated human preferences collected across 50 instructions.
ChatGPT achieved the highest average RBO score of 0.619 when compared to human
evaluations. All other models also demonstrated lower alignment with human
preferences, with Baize standing out with the lowest RBO score of 0.11 on
average in relation to human preferences among all 15 models. Models with
decreased size also tend to misalign with an overall human preference, as we
observe that the average RBO of models of size $>10B$ and $<10B$ are 0.44 and
0.32, respectively, which are lower than models of $>40B$ (0.52) and $>100B$
(0.49).

We also provide examples of the ranking preferences from each of the
evaluators compared to human preferences in Table 4. We see that although
within the top 5 rankings for these examples, models such as GPT4 and Vicuna
share some similarities in their preferences, most models have little overlap
with human preferences.

#### Biases in Pairwise Human Preference

The resulting average IAAs were 0.33 (Order Bias), 0.44 (Bandwagon Effect),
and 0.38 (Attentional Bias), signifying a modest degree of agreement among
human annotators.

The proportion of biased responses across all human annotators for Order Bias,
Salience Bias, Bandwagon Effect, and Attentional Bias are presented in the
table below, as a comparison with Vicuna. Similarly with Table 2, a higher
score indicates more biased performance. We observe that human annotators
still exhibited biases when making their preferences on LLM evaluations, but
less than LLM evaluators on average.

| Order | Salie. | Bandw. | Atten.
---|---|---|---|---
| First | Last |  |  |
Human | 0.20 | 0.18 | 0.52 | 0.47 | 0.35
Vicuna | 0.32 | 0.17 | 0.53 | 0.81 | 0.78

The average proportion of Order Biases across all annotators are 0.20 (first)
and 0.18 (last). Compared to Table 2, the annotators showed lower first-order
bias than the average of proportions shown by models of size $>100B$ (0.23),
$>40B$ (0.54), and $>10B$ (0.29), respectively. These scores indicate that the
annotators, on average, exhibited less first-order biases than models of
greater size. However, the annotators showed higher last-order bias than the
average of proportions shown by models of size $>100B$ (0.11) and $>40B$
(0.07), respectively, but lower than models of $>10B$ (0.37) and $<10B$
(0.42).

For the Bandwagon effect, humans were less affected (0.47) than models of size
$>100B$ (0.57), $>10B$ (0.76), and $<10B$ (0.57). For Salience Bias, humans
showed a greater preference for longer responses (0.52) on average, less than
the proportions of models of size $>100B$ (0.62), $>40B$ (0.56), and $<10B$
(0.54). However, we observed that humans showed a greater Attentional Bias in
text quality evaluation (0.35), which exceeds models of size $>100B$ (0.20),
$>40B$ (0.14), and $<10B$ (0.26).

##  6 Conclusions

In this paper, we provide an extensive analysis of 15 recently developed LLMs
and their capabilities as automatic annotators for text quality comparison in
the Q/A setting. We propose a new benchmark CoBBLEr that evaluates the
performance of these models for (1) Implicit and (2) Induced biases. Our
findings reveal that the majority of these LLMs-as-evaluators exhibited
several cognitive biases, while humans relatively less so. This raises
questions about their ability to make fair evaluations, suggesting that most
current LLMs are unable to perform well as unbiased automatic evaluators.
Furthermore, we also gather human preferences and find that most models are
not well-aligned with human judgments, reaching only 49% agreement on average.
With evaluation capabilities that include various cognitive biases as well as
a low percentage of agreement with human preference, our findings suggest that
LLMs are still not suitable as fair and reliable automatic evaluators.

## Limitations and Future Work

Even over several extensive experiments to benchmark the evaluation
capabilities and the influence of cognitive bias in LLMs, we note various
limitations in the current study. Some models, as viewed in Table 7, reached a
very low valid response rate, notably LLaMA and DollyV2. This may be a result
of the prompting format input, in which these tested models have difficulty
understanding the instructed task in which proper evaluations could not be
extracted. Our results are also only limited to the Q/A domain, in which
certain biases may be more amplified or have less impact in different
evaluation tasks, such as mathematical reasoning or text generation, that may
vary in the performance of LLMs-as-evaluators. Furthermore, we acknowledge a
few limitations within our human judgment study, notably the human annotations
themselves, which reach subpar Inter-Annotator Agreement. This may be due to
the difficulty of the task, asking MTurk annotators to rank 15 models to limit
the number of comparisons required in a pairwise format, but also increases
the complexity of the task itself, which may have caused lower quality in the
annotations. Additionally, the computational cost and necessary resources to
replicate the results in CoBBLEr may limit the ability of the general audience
to replicate the benchmark. Models ranging from 3B to 170B+ parameters in size
require extensive resources and several GPUs that become very costly to run
thousands of inference calls.

For future areas of exploration, potential de-biasing methods provide another
area of interest in ameliorating each bias. For example, techniques such as
chain-of-thought reasoning can be employed in order to reduce the effect of
each benchmarked bias for current models. Other extensions may involve
utilizing the CoBBLEr data to further fine-tune a language model that reduces
these cognitive biases that may provide higher-quality evaluations.

## References

  * Almazrouei et al. (2023) Ebtesam Almazrouei, Hamza Alobeidli, Abdulaziz Alshamsi, Alessandro Cappelli, Ruxandra Cojocaru, Merouane Debbah, Etienne Goffinet, Daniel Heslow, Julien Launay, Quentin Malartic, Badreddine Noune, Baptiste Pannier, and Guilherme Penedo.  Falcon-40B: an open large language model with state-of-the-art performance.  2023\.
  * Bai et al. (2023) Yushi Bai, Jiahao Ying, Yixin Cao, Xin Lv, Yuze He, Xiaozhi Wang, Jifan Yu, Kaisheng Zeng, Yijia Xiao, Haozhe Lyu, Jiayin Zhang, Juanzi Li, and Lei Hou.  Benchmarking foundation models with language-model-as-an-examiner, 2023.
  * Beeching et al. (2023) Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf.  Open llm leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, 2023.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language models are few-shot learners, 2020.
  * Butts et al. (2019) Marcus M. Butts, Devin C. Lunt, Traci L. Freling, and Allison S. Gabriel.  Helping one or helping many? a theoretical integration and meta-analytic review of the compassion fade literature.  _Organizational Behavior and Human Decision Processes_ , 151:16–33, 2019.  ISSN 0749-5978.  doi: https://doi.org/10.1016/j.obhdp.2018.12.006.  URL https://www.sciencedirect.com/science/article/pii/S0749597818302930.
  * Chen et al. (2023) Yi Chen, Rui Wang, Haiyun Jiang, Shuming Shi, and Ruifeng Xu.  Exploring the use of large language models for reference-free text quality evaluation: A preliminary empirical study, 2023.
  * Chiang et al. (2023) Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.  URL https://lmsys.org/blog/2023-03-30-vicuna/.
  * Cohen et al. (2023) Roi Cohen, May Hamri, Mor Geva, and Amir Globerson.  Lm vs lm: Detecting factual errors via cross examination, 2023.
  * Computer (2023) Together Computer.  Redpajama: An open source recipe to reproduce llama training dataset, 2023.  URL https://github.com/togethercomputer/RedPajama-Data.
  * Conover et al. (2023) Mike Conover, Matt Hayes, Ankit Mathur, Jianwei Xie, Jun Wan, Sam Shah, Ali Ghodsi, Patrick Wendell, Matei Zaharia, and Reynold Xin.  Free dolly: Introducing the world’s first truly open instruction-tuned llm, 2023.  URL https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm.
  * Dziri et al. (2023) Nouha Dziri, Ximing Lu, Melanie Sclar, Xiang Lorraine Li, Liwei Jiang, Bill Yuchen Lin, Peter West, Chandra Bhagavatula, Ronan Le Bras, Jena D. Hwang, Soumya Sanyal, Sean Welleck, Xiang Ren, Allyson Ettinger, Zaid Harchaoui, and Yejin Choi.  Faith and fate: Limits of transformers on compositionality, 2023.
  * Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli.  ELI5: long form question answering.  In Anna Korhonen, David R. Traum, and Lluís Màrquez (eds.), _Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers_ , pp. 3558–3567. Association for Computational Linguistics, 2019.  doi: 10.18653/v1/p19-1346.  URL https://doi.org/10.18653/v1/p19-1346.
  * Fu et al. (2023) Jinlan Fu, See-Kiong Ng, Zhengbao Jiang, and Pengfei Liu.  Gptscore: Evaluate as you desire, 2023.
  * Gao et al. (2021) Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.  A framework for few-shot language model evaluation, September 2021.  URL https://doi.org/10.5281/zenodo.5371628.
  * Gao et al. (2023) Mingqi Gao, Jie Ruan, Renliang Sun, Xunjian Yin, Shiping Yang, and Xiaojun Wan.  Human-like summarization evaluation with chatgpt, 2023.
  * Gekhman et al. (2023) Zorik Gekhman, Jonathan Herzig, Roee Aharoni, Chen Elkind, and Idan Szpektor.  Trueteacher: Learning factual consistency evaluation with large language models, 2023.
  * Geng et al. (2023) Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song.  Koala: A dialogue model for academic research.  Blog post, April 2023.  URL https://bair.berkeley.edu/blog/2023/04/03/koala/.
  * Geva et al. (2021) Mor Geva, Daniel Khashabi, Elad Segal, Tushar Khot, Dan Roth, and Jonathan Berant.  Did Aristotle Use a Laptop? A Question Answering Benchmark with Implicit Reasoning Strategies.  _Transactions of the Association for Computational Linguistics (TACL)_ , 2021.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  _Proceedings of the International Conference on Learning Representations (ICLR)_ , 2021.
  * Jones & Steinhardt (2022) Erik Jones and Jacob Steinhardt.  Capturing failures of large language models via human cognitive biases.  In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), _Advances in Neural Information Processing Systems_ , 2022.  URL https://openreview.net/forum?id=fcO9Cgn-X-R.
  * Jørgensen et al. (2022) Rasmus Jørgensen, Fiammetta Caccavale, Christian Igel, and Anders Søgaard.  Are multilingual sentiment models equally right for the right reasons?  In _Proceedings of the Fifth BlackboxNLP Workshop on Analyzing and Interpreting Neural Networks for NLP_ , pp. 131–141, Abu Dhabi, United Arab Emirates (Hybrid), December 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.blackboxnlp-1.11.  URL https://aclanthology.org/2022.blackboxnlp-1.11.
  * Jung et al. (2019) Taehee Jung, Dongyeop Kang, Lucas Mentch, and Eduard Hovy.  Earlier isn’t always better: Sub-aspect analysis on corpus and system biases in summarization.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pp. 3324–3335, Hong Kong, China, November 2019. Association for Computational Linguistics.  doi: 10.18653/v1/D19-1327.  URL https://aclanthology.org/D19-1327.
  * Kocmi & Federmann (2023) Tom Kocmi and Christian Federmann.  Large language models are state-of-the-art evaluators of translation quality, 2023.
  * Köpf et al. (2023) Andreas Köpf, Yannic Kilcher, Dimitri von Rütte, Sotiris Anagnostidis, Zhi-Rui Tam, Keith Stevens, Abdullah Barhoum, Nguyen Minh Duc, Oliver Stanley, Richárd Nagyfi, Shahul ES, Sameer Suri, David Glushkov, Arnav Dantuluri, Andrew Maguire, Christoph Schuhmann, Huu Nguyen, and Alexander Mattick.  Openassistant conversations – democratizing large language model alignment, 2023.
  * Li et al. (2023a) Ruosen Li, Teerth Patel, and Xinya Du.  Prd: Peer rank and discussion improve large language model based evaluations, 2023a.
  * Li et al. (2023b) Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Alpacaeval: An automatic evaluator of instruction-following models.  https://github.com/tatsu-lab/alpaca_eval, 2023b.
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Cosgrove, Christopher D. Manning, Christopher Ré, Diana Acosta-Navas, Drew A. Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue Wang, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda.  Holistic evaluation of language models, 2022.
  * Liu et al. (2022) Jiachang Liu, Dinghan Shen, Yizhe Zhang, Bill Dolan, Lawrence Carin, and Weizhu Chen.  What makes good in-context examples for GPT-3?  In _Proceedings of Deep Learning Inside Out (DeeLIO 2022): The 3rd Workshop on Knowledge Extraction and Integration for Deep Learning Architectures_ , pp. 100–114, Dublin, Ireland and Online, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.deelio-1.10.  URL https://aclanthology.org/2022.deelio-1.10.
  * Liu et al. (2023) Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu.  G-eval: Nlg evaluation using gpt-4 with better human alignment, 2023.
  * Lu et al. (2022) Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp.  Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity.  In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pp. 8086–8098, Dublin, Ireland, May 2022. Association for Computational Linguistics.  doi: 10.18653/v1/2022.acl-long.556.  URL https://aclanthology.org/2022.acl-long.556.
  * Luo et al. (2023) Zheheng Luo, Qianqian Xie, and Sophia Ananiadou.  Chatgpt as a factual inconsistency evaluator for text summarization, 2023.
  * Oh et al. (2022) Sejoon Oh, Berk Ustun, Julian McAuley, and Srijan Kumar.  Rank list sensitivity of recommender systems to interaction perturbations.  _Proceedings of the 31st ACM International Conference on Information & Knowledge Management_, 2022.  URL https://api.semanticscholar.org/CorpusID:251442310.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report, 2023.
  * Ouyang et al. (2022) Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul Christiano, Jan Leike, and Ryan Lowe.  Training language models to follow instructions with human feedback, 2022.
  * Raheja et al. (2023) Vipul Raheja, Dhruv Kumar, Ryan Koo, and Dongyeop Kang.  Coedit: Text editing by task-specific instruction tuning, 2023.
  * Ross & Sicoly (1979) Michael Ross and Fiore Sicoly.  Egocentric biases in availability and attribution.  _Journal of Personality and Social Psychology_ , 37:322–336, 03 1979.  doi: 10.1037/0022-3514.37.3.322.
  * Schenk (2010) Deborah Schenk.  Exploiting the salience bias in designing taxes.  _New York University Law and Economics Working Papers_ , 28, 08 2010.  doi: 10.2139/ssrn.1661322.
  * Schick et al. (2023) Timo Schick, Jane A. Yu, Zhengbao Jiang, Fabio Petroni, Patrick Lewis, Gautier Izacard, Qingfei You, Christoforos Nalmpantis, Edouard Grave, and Sebastian Riedel.  PEER: A collaborative language model.  In _The Eleventh International Conference on Learning Representations_ , 2023.  URL https://openreview.net/forum?id=KbYevcLjnc.
  * Schmitt-Beck (2015) Rüdiger Schmitt-Beck.  _Bandwagon Effect_.  07 2015.  ISBN 9781118290750.  doi: 10.1002/9781118541555.wbiepc015.
  * Shen et al. (2023) Chenhui Shen, Liying Cheng, Yang You, and Lidong Bing.  Are large language models good evaluators for abstractive summarization?, 2023.
  * Shi et al. (2023) Freda Shi, Xinyun Chen, Kanishka Misra, Nathan Scales, David Dohan, Ed Chi, Nathanael Schärli, and Denny Zhou.  Large language models can be easily distracted by irrelevant context, 2023.
  * Srivastava et al. (2023) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R. Brown, Adam Santoro, Aditya Gupta, and Adrià Garriga-Alonso et al.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _Transactions on Machine Learning Research_ , 2023.  ISSN 2835-8856.  URL https://openreview.net/forum?id=uyTL5Bvosj.
  * Sun et al. (2023) Weiwei Sun, Lingyong Yan, Xinyu Ma, Pengjie Ren, Dawei Yin, and Zhaochun Ren.  Is chatgpt good at search? investigating large language models as re-ranking agent, 2023.
  * Talboy & Fuller (2023) Alaina N. Talboy and Elizabeth Fuller.  Challenging the appearance of machine intelligence: Cognitive bias in llms and best practices for adoption, 2023.
  * Tang et al. (2023) Tianyi Tang, Hongyuan Lu, Yuchen Eleanor Jiang, Haoyang Huang, Dongdong Zhang, Wayne Xin Zhao, and Furu Wei.  Not all metrics are guilty: Improving nlg evaluation with llm paraphrasing, 2023.
  * Taori et al. (2023) Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * Team (2023) MosaicML NLP Team.  Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023.  URL www.mosaicml.com/blog/mpt-7b.  Accessed: 2023-05-05.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom.  Llama 2: Open foundation and fine-tuned chat models, 2023.
  * Västfjäll et al. (2014) Daniel Västfjäll, Paul Slovic, Marcus Mayorga, and Ellen Peters.  Compassion fade: Affect and charity are greatest for a single child in need.  _PLOS ONE_ , 9(6):1–10, 06 2014.  doi: 10.1371/journal.pone.0100115.  URL https://doi.org/10.1371/journal.pone.0100115.
  * Wang et al. (2020) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.  Superglue: A stickier benchmark for general-purpose language understanding systems, 2020.
  * Wang et al. (2023a) Jiaan Wang, Yunlong Liang, Fandong Meng, Zengkui Sun, Haoxiang Shi, Zhixu Li, Jinan Xu, Jianfeng Qu, and Jie Zhou.  Is chatgpt a good nlg evaluator? a preliminary study, 2023a.
  * Wang et al. (2023b) Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui.  Large language models are not fair evaluators, 2023b.
  * Webber et al. (2010) William Webber, Alistair Moffat, and Justin Zobel.  A similarity measure for indefinite rankings.  _ACM Transactions on Information Systems (TOIS)_ , 28(4):1–38, 2010.
  * Wu & Aji (2023) Minghao Wu and Alham Fikri Aji.  Style over substance: Evaluation biases for large language models, 2023.
  * Xu et al. (2023a) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang.  Wizardlm: Empowering large language models to follow complex instructions, 2023a.
  * Xu et al. (2023b) Canwen Xu, Daya Guo, Nan Duan, and Julian McAuley.  Baize: An open-source chat model with parameter-efficient tuning on self-chat data, 2023b.
  * Zhang et al. (2020) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi.  Bertscore: Evaluating text generation with bert, 2020.
  * Zhao et al. (2021) Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh.  Calibrate before use: Improving few-shot performance of language models.  In Marina Meila and Tong Zhang (eds.), _Proceedings of the 38th International Conference on Machine Learning_ , volume 139 of _Proceedings of Machine Learning Research_ , pp. 12697–12706. PMLR, 18–24 Jul 2021.  URL https://proceedings.mlr.press/v139/zhao21c.html.
  * Zheng et al. (2023) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.  Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
  * Zhuo (2023) Terry Yue Zhuo.  Large language models are state-of-the-art evaluators of code generation, 2023.

##  Appendix A Experimental Setup

###  A.1 Model Hyperparameters

We set the same hyperparameters across models for each evaluation generation
and response generation for consistency across all of the models. We limit the
max new tokens generated to 128 tokens and set the temperature to 1.0. For
Huggingface models, we set a repetition penalty of 1.2 and set the number of
beams to 3.

###  A.2 Experimental Settings

For models that are supported (ChatGPT, InstructGPT, GPT-4, Vicuna), we
utilize Microsoft Guidance to better control LLM generations. Otherwise, we
utilize the transformer pipeline library from Hugginface to retrieve each
evaluation generation. Regardless of whether a models generation was collected
from guidance or using the transformer pipeline, all parameters were the same.
Model generation times for response generation ranged from 1 to 8 hours, and
for evaluation generations ranged from 3 to 24 hours for each bias benchmark.
All experiments were run on either A5000 or A6000 GPUs for models under 40B
parameters. For models over 40B, A100 GPUs were utilized if an API service was
not available (e.g. OpenAI, Cohere).

##  Appendix B Prompt Templates

We present each of the evaluation prompts utilized for models to evaluate the
quality between two generations. We show each of the prompts (Compassion,
Bandwagon, Attentional) derived from the original Order prompt in Section 4.2.
We highlight each modification made from the original template.

Our generation instruction template looks like the following:

### Instruction:  $x$
---
### Response:

For evaluation, bias-specific prompts are formatted as follows:

###  B.1 Order

### Instruction: Which system’s response is more coherent considering
---
the reference and instruction?
The instruction is: {{instruction}}
The reference is: {{reference}}
“‘
System Star:
System Square:
“‘
Please respond in the following format strictly: System _ is better
Do not provide any additional text or explanation:
### Response:

###  B.2 Compassion Fade

### Instruction: Which system’s response is more coherent considering
---
the reference and instruction? $x$
The instruction is: {{instruction}}
The reference is: {{reference}}
‘‘‘
{model1_name}: {model1_response}
{model2_name}: {model2_response}
‘‘‘
Please response directly in the following format: System _ is better
Do not provide any additional text or explanation:
### Response:

###  B.3 Bandwagon Effect

### Instruction: Which system’s response is more coherent considering
---
the reference and instruction? $x$
The instruction is: {{instruction}}
The reference is: {{reference}}
‘‘‘
System Star: {model1_response}
System Square: {model2_response}
‘‘‘
85% of people believe that {system} is better.
Please response directly in the following format: System _ is better
Do not provide any additional text or explanation:
### Response:

###  B.4 Attentional Bias

### Instruction: Which system’s response is more coherent considering
---
the reference and instruction? $x$
The instruction is: {{instruction}}
The reference is: {{reference}}
‘‘‘
System Star: {model1_response}
System Square: {model2_response}
‘‘‘
{distraction}
Please response directly in the following format: System _ is better
Do not provide any additional text or explanation:
### Response:
Distraction Bank
---
{System} likes to eat apples and oranges.
The coolest thing that {System} can do is a 60 second handstand.
{System} plays a lot of soccer and basketball.
{System} has been all around Europe two times.

##  Appendix C Supplementary Results

###  $N$-Rankwise setting: $N=4$

We show the results and average rankings between four different models
representing each of the different size groups: ChatGPT ($>$$100B$), Falcon
($>$$40B$), Alpaca ($>$$10B$), Vicuna ($<$$10B$).

For the experimental setup, we conduct a smaller study, generating 100
responses from each of the 4 different LLMs using the Databricks Dolly15k
dataset (Conover et al., 2023) via the same instruction prompt template from
Appendix B and the same evaluation prompt template from the Order bias.

Model |  Size |  Valid Response |  Order bias |  ChatGPT avg. rank |  Falcon avg. rank |  Alpaca avg. rank |  Vicuna avg. rank
---|---|---|---|---|---|---|---
ChatGPT |  - |  0.94 |  0.32 |  2.3 |  2.5 |  2.6 |  2.6
Falcon |  40B |  0.38 |  0.39 |  2.6 |  2.3 |  2.6 |  2.5
Alpaca |  13B |  0.65 |  1.0 |  2.6 |  2.4 |  2.4 |  2.4
Vicuna |  7B |  0.02 |  0.0 |  1.5 |  4.0 |  3.0 |  1.5
Table 5: We show the results of instructing models to perform a list-wise
evaluation, by prompting each LM-as-evaluator to organize a list of responses
from 4 different models top to bottom with the first being the best response
and the last being the worst response. We then take the average ranking of
each of the models and display their results above for each LM-as-evaluator.

We only employ this setting under the order bias setting in order to validate
the complexity of the task that modern (smaller) LLMs aren’t capable of
performing yet. We perform each experiment by randomizing the order of each
list of responses and prompt each LM-as-evaluator to order the list from best
to worst (top to bottom) according to the same criterion as the pairwise study
(providing the instruction/sample reference). Furthermore, we also track Order
bias, calculated by the proportion of responses in which the first (randomly)
placed model was also ranked first by the evaluator.

As viewed in Table 5, we find that most models besides the closed-source API
models (e.g. OpenAI) have trouble generating a proper rank list for even an
$N=4$ setting. Thus, as the scaling of $N$ increases, we expect the task
complexity to increase as well, which may result in ineligible evaluations,
and hence the pairwise setting resolves this issue.

### LLM Performance and Agreement

We detail the general agreement between machine preferences as similarly
conducted in the human-machine correlation study. Table 7 visualizes the
average Rank-Based Overlap between LLMs. We find that generally LLMs in their
own size group (excluding the smallest size group) are generally in agreement
with each other. For example, models in the largest size group ($>$$100B$) are
more in agreement amongst themselves than with models from other size groups.
Furthermore, we also show the average valid response rate from different bias
promptings in Table 7.

Table 6: The average RBO scores between LLMs. Higher RBO means higher
similarity.

Model |  Avg. |  Ord. |  Comp. |  Band. |  Attn.
---|---|---|---|---|---
GPT4 |  0.98 |  0.98 |  0.97 |  0.99 |  0.99
ChatGPT |  0.99 |  0.99 |  0.99 |  0.99 |  0.99
InstructGPT |  0.99 |  0.99 |  0.99 |  1.00 |  0.99
LLaMAv2 |  0.54 |  0.17 |  0.40 |  0.43 |  0.91
LLaMA |  0.14 |  0.22 |  0.16 |  0.03 |  0.58
Cohere |  0.98 |  0.94 |  0.99 |  0.82 |  0.99
Falcon |  0.72 |  0.72 |  0.46 |  0.99 |  0.98
Alpaca |  0.84 |  0.78 |  0.82 |  0.97 |  0.87
Vicuna |  0.86 |  0.90 |  0.71 |  0.97 |  0.90
OpenAssist |  0.60 |  0.80 |  0.32 |  0.95 |  0.94
DollyV2 |  0.00 |  0.00 |  0.00 |  0.00 |  0.00
Baize |  0.96 |  0.98 |  0.87 |  0.99 |  0.99
Koala |  0.25 |  0.29 |  0.18 |  0.23 |  0.30
WizardLM |  0.93 |  0.95 |  0.83 |  0.99 |  0.96
MPT |  0.77 |  0.82 |  0.72 |  0.84 |  0.32
RedPajama |  0.52 |  0.52 |  0.26 |  0.72 |  0.65

Table 7: Ratio for generating valid evaluations. Bolded numbers are ones in
which less than half of the responses were invalid.

##  Appendix D Human Preference Study

###  D.1 Annotator Recruitment

#### N=15-rankwise setting

We recruited six workers from the Amazon Mechanical Turk (AMT) platform, each
of whom had a U.S. high school diploma and a Human Intelligence Task (HIT)
approval rate of 99% or higher on the platform. To ensure better-quality
annotations, we initiated a toy round using five sample instruction set. Each
instruction in the toy round contained five brief LLM-generated sentences.
Workers were then asked to rank these sentences based on their own
preferences, but taking into account the following two specific criteria: (1)
the fluency and logical coherence of the LLM-generated text in response to a
given instruction sentence, and (2) the text’s alignment with a reference
sentence that provided additional context and background for the instruction
sentence. Furthermore, they were asked to place a black bar above the answers
that did not satisfy these two criteria, as this is used for the threshold to
evaluate the quality of their texts.

After each participant finished their annotation during the toy round, we
carefully reviewed their submissions to ensure they had accurately followed
the guidelines and considered the two ranking criteria and the position of
black bar. For their efforts, each participant received a $3 payment for
completing the toy round (HIT). Running the toy HIT several times yielded a
final selection of six qualified workers, who were then invited to participate
in the next stage involving the actual task of ranking 50 instruction sets.
Each of these sets included 15 texts generated by 15 different LLMs.

To avoid overwhelming the workers, we divided the main task into five separate
HITs, each containing a varying number of instruction sets to rank: (1) a
pilot round with 5 sets, (2) two intermediate rounds with 10 sets each, and
(3) two final rounds with 13 and 12 sets, respectively, adding up to a total
of 50 instruction sets. These six workers received compensation upon
completing each HIT, accumulating to a minimum of $47 for the entire series of
rounds. This averaged out to approximately $1.05 per instruction set.
Additionally, on average, it took each of the six workers about 5.8 minutes to
complete a single instruction set. Lastly, considering the level of difficulty
for the workers to rank 15 outputs per instruction set, we also remunerated
them with a bonus of at least $5 per round, based on the quality of their
performance.

#### Biases in Pairwise Human Preference

For each bias study, we sourced human preferences from 75 experienced AMT
workers who had HIT approval rates over 97%, had completed more than 10,000
HIT tasks, and resided in English-speaking countries This qualification of
location limits to AMT workers in the United States, Canada, United Kingdom,
Australia, and New Zealand. These workers were then grouped into 25 sets of 3,
with each group assigned a HIT task encompassing 30 model pairs drawn from an
instance. Consequently, we generated 25 HITs for each bias. These workers were
tasked with choosing between two anonymous options (e.g., System A and B) for
each of the 30 pairs. Their decisions were based on alignment and coherency
with the instruction and reference sentences.

To employ a pre-task and training session, we asked the participating workers
of each HIT to complete a qualification round, which asked three example
questions to complete and pass. Only workers who passed this round were
allowed to start main tasks of annotating 30 pairs, ensuring that the workers
were able to understand and follow instructions of the HIT. Each worker who
participated in a HIT received a compensation of $2.5. We also gave the
workers a maximum of 3 hours to complete a HIT, where workers spent on average
47 minutes, 57 minutes, and 51 minutes, for order, bandwagon, and distraction
experiments, respectively.

###  D.2 Details on using RBO

Rank-biased overlap (RBO) is a widely used metric for evaluating the
similarity between two ranked lists and is particularly relevant for tasks in
information retrieval (Oh et al., 2022; Sun et al., 2023). Unlike traditional
correlation-based metrics like Kendall’s $\tau$ or Spearman’s $\rho$, RBO
allows for greater weighting of the top $k$ elements in the ranked lists being
compared. This feature makes RBO well-suited for our experimental setup, where
AMT workers were tasked with reading and ranking 15 outputs generated by LLMs.
We operate under the assumption that workers are likely to place the highest-
quality texts at the top five positions of their ranked lists.

This idea of weighing the top elements in the ranked outputs aligns with
previous research, which claims RBO to be an effective metric for the
agreement between ranked annotations with human rationals and automated
evaluations, especially when greater importance is given to the top-ranked
elements (Jørgensen et al., 2022). Given these considerations, which are
highly relevant to our own study, we decided to use RBO as the metric for
assessing agreement between human preferences and LLM evaluations.

RBO is defined in Equation 1 and tailored to suit the specifics of our study.
Here, $H$ and $L$ represent two ranked lists of shape $(1,15)$, corresponding
to human preferences and LLM evaluations for each instruction set,
respectively. The maximum depth for $H$ and $L$ is set at 15, and $p$ is a
tunable parameter that determines the degree of top-weightedness in the final
RBO calculation. To obtain an average RBO score across all 50 instruction
sets, we sum the individual RBO values between $H$ and $L$ and then divide by
50.

$\centering RBO(H,L)=(1-p)\sum_{d=1}^{15}p^{d-1}\frac{|A[1:d]\cap
B[1:d]|}{d}\@add@centering$ (1)

Following the work of Webber et al. (2010), we set the value of $p$ so that
approximately 86% of the weight is concentrated on the first $d$ ranks, where
$d=5$ in our case. The weight distribution over these top $d$ ranks can be
determined using Equation 2. In our experimental setup, we found that $p$ was
approximately 0.8.

$\centering(1-p^{d-1})+(\frac{1-p}{p})\cdot
d\cdot\left(\ln\frac{1}{1-p}-\sum_{i=1}^{d-1}\frac{p^{i}}{i}\right)\@add@centering$
(2)

###  D.3 Interface Design

We present the interface design temple for each of human experiments setup on
AMT platform, including (1) N=15-rankwise setups and (2) bias setups in
pairwise human preference, as described in Section 4.3. The original prototype
of the interfaces is based on https://github.com/mtreviso/TextRankerJS.

Figure 5: The interface design for gathering human preferences over LLM-
generated texts for each instruction and reference sentence on Amazon
Mechanical Turk (AMT) settings. Six AMT workers participated in the annotation
process and each was given 50 instruction sets to rank 15 LLM-generated texts.
Figure 6: The AMT interface design for order bias experiments with pairwise
human preference setup. Figure 7: The AMT interface design for bandwagon bias
experiments with pairwise human preference setup. Figure 8: The AMT interface
design for distraction bias experiments with pairwise human preference setup.