# Accelerating Diffusion-based Combinatorial Optimization Solvers by
Progressive Distillation

Junwei Huang
Carnegie Mellon University
<EMAIL_ADDRESS>&Zhiqing Sun
Carnegie Mellon University
<EMAIL_ADDRESS>&Yiming Yang
Carnegie Mellon University
<EMAIL_ADDRESS>
###### Abstract

Graph-based diffusion models have shown promising results in terms of
generating high-quality solutions to NP-complete (NPC) combinatorial
optimization (CO) problems. However, those models are often inefficient in
inference, due to the iterative evaluation nature of the denoising diffusion
process. This paper proposes to use progressive distillation to speed up the
inference by taking fewer steps (e.g., forecasting two steps ahead within a
single step) during the denoising process. Our experimental results show that
the progressively distilled model can perform inference 16 times faster with
only 0.019% degradation in performance on the TSP-50 dataset.

##  1 Introduction

The realm of Combinatorial Optimization (CO) is intricately tied to
optimization within discrete spaces. A significant portion of CO problems
falls into the category of NP-complete (NPC) problems, for which efficient
algorithms for exact solutions are nearly impossible to find. In the past,
researchers have often turned to traditional approximation algorithms as a
means to tackle these NPC problems [Arora, 1998, Lawler et al., 1980].
Unfortunately, these traditional methods demand an extensive degree of
mathematical understanding specific to each individual problem, resulting in
heuristics that lack transferability across diverse problem types.

Modern approaches utilize the pattern recognition power of machine learning
techniques to generate high-quality solutions. Representative methods include
the deep reinforcement learning (DRL) models [Khalil et al., 2017, Barrett et
al., 2019, Yao et al., 2021, Bello et al., 2016] and neural diffusion models
[Graikos et al., 2022, Sun and Yang, 2023]. DRL approaches focus on training
an agent to learn the heuristics for solving a CO problem, by allowing the
agent to interact with its environment, aided by a graph representation of the
problem. In the unsupervised setting for a DRL solver, the system learns the
search strategy without requiring annotated optimal solutions for training-set
graphs [Karalias and Loukas, 2020, Wang et al., 2022], but DRL CO solvers are
usually hard to train, due to the sparse reward problem [Wu et al., 2021]. In
the supervised settings, on the other hand, neural network solvers learn a
generative process to directly predict the high-quality solutions to CO
problems [Joshi et al., 2019, Sun and Yang, 2023]. They are usually more
stable in training and efficient in inference, but require near-optimal
annotations generated by traditional solvers.

Recent progress in diffusion models for image generation [Ho et al., 2020,
Song et al., 2020, Song and Ermon, 2020] has sparked interest in applying
diffusion models to CO problem with the hope of exploiting its expressiveness.
Graikos et al. [2022] show by encoding the traveling salesman problem (TSP) as
$64\times 64$ greyscale images, image diffusion models [Ho et al., 2020] are
able to generate heatmaps from which the solutions could be extracted. Sun and
Yang [2023] proposed the DIFUSCO framework, which works by explicitly modeling
the graph structure of CO problems and denoising the variable indicator
vector. However, these diffusion-based methods are inefficient in inference
because of the non-trivial number of inference steps required to produce
satisfactory results.

To remedy the efficiency problem encountered by these diffusion-based solvers,
we propose to use progressive distillation [Rezagholizadeh et al., 2022,
Salimans and Ho, 2022, Meng et al., 2023] to address the inference efficiency
issue of DIFUSCO, accelerating its inference while preserving its solution
quality. We train the student to compress two denoising steps from the teacher
into one in a single training iteration and we iteratively perform this
compression to achieve further distilled student. We report a 0.019%
performance degradation on our 4x distilled student, which inferences 16 times
faster than the teacher.

##  2 Method

###  2.1 Problem Definition

In this paper, our primary focus is on the Travelling Salesman Problem (TSP)
as a subject of investigation. However, we posit that both the algorithm we
have developed and our conclusions have broader applicability, potentially
extending to other types of Combinatorial Optimization (CO) problems. This
perspective aligns with the findings presented in DIFUSCO [Sun and Yang,
2023].

Formally, we define the possible solution space of TSP instance ${\bm{s}}$ to
be ${\mathcal{X}}_{\bm{s}}=\\{0,1\\}^{N}$, where $N$ is the number of edges in
the problem instance. For each possible solution vector
${\mathbf{x}}\in{\mathcal{X}}_{\bm{s}}$, ${\mathbf{x}}_{i}$ is the indicator
variable for whether edge $i$ is chosen in the solution vector ${\mathbf{x}}$.
We define the objective function
${\bm{J}}_{\bm{s}}:{\mathcal{X}}_{\bm{s}}\to{\mathbb{R}}$ such that it
incorporates both the cost and the validity of the solution:

${\bm{J}}_{\bm{s}}({\mathbf{x}})=\mathbf{cost}_{\bm{s}}({\mathbf{x}})+\mathbf{valid}_{\bm{s}}({\mathbf{x}})$
(1)

where
$\textbf{cost}_{\bm{s}}({\mathbf{x}})={\mathbf{x}}^{T}{\mathbf{d}},{\mathbf{d}}_{i}$
being the $i^{th}$ edge’s weight in ${\bm{s}}$, and
$\textbf{valid}_{\bm{s}}({\mathbf{x}})$ is 0 for a feasible TSP solution,
$+\infty$ for infeasible solutions. The optimal solution to ${\bm{s}}$ is

${\mathbf{x}}^{*}=\operatorname*{arg\,min}_{{\mathbf{x}}\in{\mathcal{X}}_{\bm{s}}}{\bm{J}}_{\bm{s}}({\mathbf{x}})$
(2)

###  2.2 Diffusion Solvers for Combinatorial Optimization

DIFUSCO [Sun and Yang, 2023] is a framework for directly modeling the graph
representation of CO problems and solving CO problems via diffusion models. By
first re-scaling [Chen and Tian, 2019] $\\{0,1\\}$ valued variables to
$\\{-1,1\\}$ and treating them as real-valued variables, it becomes viable to
apply continuous diffusion [Ho et al., 2020, Song et al., 2020] to the problem
setup in 2.1. Re-scaling for binary valued variable at time step
$t\in\\{1,\ldots T\\}$ is given as:

$\displaystyle\hat{{\mathbf{x}}}_{t}$ $\displaystyle=2{\mathbf{x}}_{t}-1$ (3)

After the re-scaling, we then use the vanilla forward process proposed by Ho
et al. [2020]

$\displaystyle
q_{\theta}(\hat{{\mathbf{x}}}_{t}|\hat{{\mathbf{x}}}_{t-1})=\mathcal{N}(\hat{{\mathbf{x}}}_{t};\sqrt{1-\beta_{t}}\hat{{\mathbf{x}}}_{t-1},\beta_{t}{\mathbf{I}})$
(4)

where $\beta_{t}$ is the ratio of corruption at timestep $t$. The $t$ step
marginal distribution of $\hat{{\mathbf{x}}}_{t}$ is therefore:

$\displaystyle
q_{\theta}(\hat{{\mathbf{x}}}_{t}|\hat{{\mathbf{x}}}_{0})=\mathcal{N}(\hat{{\mathbf{x}}}_{t};\sqrt{\bar{\alpha}_{t}}\hat{{\mathbf{x}}}_{0},(1-\bar{\alpha}_{t}){\mathbf{I}})$
(5)

We train a neural network ${\epsilon}_{\theta}(\cdot,\cdot)$ to predict the
Gaussian noise ${\epsilon}$ and given the corrupted $\hat{{\mathbf{x}}}_{t}$
and timestep $t$.

$\displaystyle\tilde{{\epsilon}_{t}}={\epsilon}_{\theta}(\hat{{\mathbf{x}}}_{t},t)$
(6)

We apply the Mean Squared Error (MSE) Loss to $\tilde{{\epsilon}}_{t}$ and
${\epsilon}_{t}$ and optimize ${\epsilon}_{\theta}$ with respect to this loss,
producing a performant predictor of noise that we could use to iteratively
denoise and produce the predicted solution $\hat{{\mathbf{x}}}_{0}$. The
predicted solution $\hat{{\mathbf{x}}}_{0}$ is obtained with the Gaussian
Posterior from Denoising Diffusion Implicit Model (DDIM) [Song et al., 2020].

$\displaystyle\text{GuassianPosterior}(\tilde{{\epsilon}}_{t},\hat{{\mathbf{x}}}_{t})=\sqrt{\frac{\bar{\alpha}_{t-1}}{\bar{\alpha}_{t}}}\left(\hat{{\mathbf{x}}}_{t}-\sqrt{1-\bar{\alpha}_{t}}\tilde{{\epsilon}}_{t}\right)+\sqrt{1-\bar{\alpha}_{t-1}}\tilde{{\epsilon}}_{t}$
(7)

We finally perform quantization to obtain ${\mathbf{x}}_{0}$ from
$\hat{{\mathbf{x}}}_{0}$.

###  2.3 Progressive Distillation

The DIFUSCO framework is capable of producing high-quality solutions for CO
problems like TSP-50, TSP-100, etc. However, this framework is often
inefficient in inference due to the need of a considerable number of inference
diffusion steps to generate reasonable solutions. Aiming to accelerate this
method while preserving solution quality, we propose to use progressive
distillation to accelerate its inference.

Progressive Distillation [Rezagholizadeh et al., 2022, Salimans and Ho, 2022]
is an algorithm designed to accelerate the inference speed of diffusion
models. For one iteration of progressive distillation, we update the student
model to make its one denoising step match two denoising steps from the
teacher model. We then iteratively train a series of student models in this
manner, where the teacher model in each iteration is initialized from the
converged student model in the previous iteration. The objective is to
progressively distill knowledge from one model to the next, gradually
improving the performance of the student model with fewer inference steps.

By the end of multiple distillation iterations, it becomes viable for the
yielded progressively distilled student model to compress the diffusion steps
required to achieve the same level of quality of solution by multiple times.
We describe the progressive distillation training algorithm for DIFUSCO in 1.

Algorithm 1 Progressive Distillation on DIFUSCO - training

Trained teacher model $\widehat{{\epsilon}_{\eta}}(\cdot,\cdot)$

Data Set ${\mathcal{D}}$, Student sampling steps $N$, Distillation factor $K$

for $K$ iterations do

$\theta\leftarrow\eta$ $\triangleright$ Init student from teacher

while not converged do

Sample { ${\mathbf{x}}\sim{\mathcal{D}}$, $t=i/N,\ \ \ i\sim$ Categorical
$[1,2,\ldots,N]$, $\epsilon\sim{\mathcal{N}}(0,{\mathbf{I}})$ }

${\mathbf{x}}_{t}=\alpha_{t}{\mathbf{x}}+\sigma_{t}\epsilon$

$t^{\prime}=t-0.5/N,t^{\prime\prime}=t-1/N$

#2 steps of teacher denoising

${\mathbf{x}}_{t^{\prime}}=$
GaussianPosterior${}_{\eta}\left[\hat{{\epsilon}_{\eta}}({\mathbf{x}}_{t},t),{\mathbf{x}}_{t}\right]$

${\mathbf{x}}_{t^{\prime\prime}}=$
GaussianPosterior${}_{\eta}\left[\hat{\epsilon_{\eta}}({\mathbf{x}}_{t^{\prime}},{t^{\prime}}),{\mathbf{x}}_{t^{\prime}}\right]$

#1 step of student denoising

$\tilde{{\mathbf{x}}}=$
GaussianPosterior${}_{\theta}\left[\hat{\epsilon_{\theta}}({\mathbf{x}}_{t},t),{\mathbf{x}}_{t}\right]$

${\mathcal{L}}=||{\mathbf{x}}_{t^{\prime\prime}}-\tilde{{\mathbf{x}}}||^{2}_{2}$
$\triangleright$ MSE loss

$\theta\leftarrow\theta-\gamma\nabla_{\theta}{\mathcal{L}}$ $\triangleright$
Gradient Update on student model parameters

end while

$\eta\leftarrow\theta$ $\triangleright$ Student becomes the new teacher

$N\leftarrow N/2$ $\triangleright$ Halve number of sampling steps

end for

##  3 Experiments

We benchmark our results against the TSP-50 dataset, where each training
sample ${\mathbf{x}}_{0}$ is the solution to a TSP instance with exactly 50
vertices in the problem graph. We fixed the diffusion steps to 1024 in all
experiments. For the teacher model and each student model, we vary the number
of inference steps as powers of 2, from 4 to 64 steps. The result of the
distilled students after $1\times$, $2\times$, $3\times$, and $4\times$
progressive distillation iterations are shown in Figure 1. We also perform
$4\times$ parallel sampling for all experiments in Figure 1 and report results
in Figure 2. $1\times$, $2\times$, $3\times$, and $4\times$ distilled students
effectively have 512, 256, 128, and 64 diffusion steps during training.

#### Model Setup

Following Sun and Yang [2023], we set the noise predictor neural network
$\hat{{\epsilon}}_{\theta}(\cdot,\cdot)$ to be a 12-layer anisotropic Graph
Neural Network [Bresson and Laurent, 2018] with a width of 256. We fix the
diffusion scheduler $\\{\beta_{i}\\}_{i=1}^{T}$ to be a linear denoising
schedule from $\beta_{1}=10^{-4}$ to $\beta_{T}=0.02$. We use DDIM Song et al.
[2020] for fast inference and use a linear skipping strategy for all
experiments [Sun and Yang, 2023].

#### Performance Metrics

Per convention, we use the solution cost drop percentage on the test set as
the metric of performance. Solution cost drop percentage is calculated as the
the difference between yielded solution cost and ground truth cost divided by
the ground truth cost.

#### Same inference steps

We first examine how much better our distilled student performs when the
teacher and the student are allowed the same number of inference steps. As
shown in Figure 1, we observed that our distilled student performs
considerably better than the teacher when we allow the same number of
inference steps for the student and teacher. The student is expected to
compress multiple steps of the teacher into one step of its own. Therefore the
student inferencing with a distill factor of $k$ is theoretically equivalent
to teacher inferencing with $k$ times the student inference steps. When
parallel sampling is enabled, we observe that performance gap between teacher
and distilled students shrinks but does not vanish.

#### Inference with $1/k$ of steps

We also examine how minimal the performance degradation is when we restrict
the student to inference with $1/k$ of teacher inference steps We observed
that distilled students are able to retain minimal performance degradation
compared to our already optimized teacher. Our 4x distilled student was able
to compress 64 inference steps of teacher model into 4 steps. Comparing the
result of student inferencing with 4 steps and teacher inferencing with 64
steps, we observed a performance degradation of 0.019%.

##  4 Related Work

Prior work by Graikos et al. [2022] used image diffusion model to introduce
visual inductive bias to aid solving CO problems. We would like to point out
progressive distillation on image generation models has been explored by
Salimans and Ho [2022]. We explored upon the prediction target mentioned and
decided to adopt unscaled clean solution as prediction target due to its
stability during experimentation.

Figure 1: TSP-50 results for $1\times$, $2\times$, $3\times$, and $4\times$
distilled student model performance against an optimized teacher model
performance. We vary the number of inference steps and tabulate performance
drop on each choice of inference step. Figure 2: We enable $4\times$ parallel
sampling where we decode 4 samples in parallel and pick the best candidate.

##  5 Discussion & Conclusion

In this paper, we have demonstrated that the adoption of a progressive
distillation scheme can significantly accelerate the inference process of
diffusion-based CO solvers. Crucially, this enhancement in speed does not
compromise the quality of the solutions obtained, resulting in only a
negligible performance degradation. While our current experimentation is
focused on TSP-50, we are optimistic about the potential of extending our
methodology to other CO problems in future studies. This area of research
provides a promising direction for future exploration and analysis.

Aside from continuous Gaussian noises, DIFUSCO [Sun and Yang, 2023] also
mentioned injecting discrete Bernoulli noise which yielded better experimental
results. We will extend our distillation scheme to support Bernoulli noise in
future work, taking advantage of its inherent consistency with the discrete
nature of CO problems. We also consider exploring other performant backbone
networks, such as transformers [Peebles and Xie, 2022].

## References

  * Arora [1998] S. Arora.  Polynomial time approximation schemes for euclidean traveling salesman and other geometric problems.  _J. ACM_ , 45(5):753–782, sep 1998.  ISSN 0004-5411.  doi: 10.1145/290179.290180.  URL https://doi.org/10.1145/290179.290180.
  * Barrett et al. [2019] T. D. Barrett, W. R. Clements, J. N. Foerster, and A. I. Lvovsky.  Exploratory combinatorial optimization with reinforcement learning.  _ArXiv_ , abs/1909.04063, 2019.
  * Bello et al. [2016] I. Bello, H. Pham, Q. V. Le, M. Norouzi, and S. Bengio.  Neural combinatorial optimization with reinforcement learning.  _ArXiv_ , abs/1611.09940, 2016.
  * Bresson and Laurent [2018] X. Bresson and T. Laurent.  An experimental study of neural networks for variable graphs.  In _International Conference on Learning Representations Workshop_ , 2018.
  * Chen and Tian [2019] X. Chen and Y. Tian.  Learning to perform local rewriting for combinatorial optimization.  In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, _Advances in Neural Information Processing Systems_ , volume 32. Curran Associates, Inc., 2019.  URL https://proceedings.neurips.cc/paper_files/paper/2019/file/131f383b434fdf48079bff1e44e2d9a5-Paper.pdf.
  * Graikos et al. [2022] A. Graikos, N. Malkin, N. Jojic, and D. Samaras.  Diffusion models as plug-and-play priors.  In _Thirty-Sixth Conference on Neural Information Processing Systems_ , 2022.  URL https://arxiv.org/pdf/2206.09012.pdf.
  * Ho et al. [2020] J. Ho, A. Jain, and P. Abbeel.  Denoising diffusion probabilistic models.  _ArXiv_ , abs/2006.11239, 2020.
  * Joshi et al. [2019] C. K. Joshi, T. Laurent, and X. Bresson.  An efficient graph convolutional network technique for the travelling salesman problem.  _arXiv preprint arXiv:1906.01227_ , 2019.
  * Karalias and Loukas [2020] N. Karalias and A. Loukas.  Erdos goes neural: an unsupervised learning framework for combinatorial optimization on graphs.  In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 6659–6672. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper_files/paper/2020/file/49f85a9ed090b20c8bed85a5923c669f-Paper.pdf.
  * Khalil et al. [2017] E. B. Khalil, H. Dai, Y. Zhang, B. N. Dilkina, and L. Song.  Learning combinatorial optimization algorithms over graphs.  In _NIPS_ , 2017.
  * Lawler et al. [1980] E. L. Lawler, J. K. Lenstra, and A. H. G. Rinnooy Kan.  Generating all maximal independent sets: Np-hardness and polynomial-time algorithms.  _SIAM Journal on Computing_ , 9(3):558–565, 1980\.  doi: 10.1137/0209042.  URL https://doi.org/10.1137/0209042.
  * Meng et al. [2023] C. Meng, R. Rombach, R. Gao, D. Kingma, S. Ermon, J. Ho, and T. Salimans.  On distillation of guided diffusion models.  In _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 14297–14306, 2023.
  * Peebles and Xie [2022] W. S. Peebles and S. Xie.  Scalable diffusion models with transformers.  _ArXiv_ , abs/2212.09748, 2022.
  * Rezagholizadeh et al. [2022] M. Rezagholizadeh, A. Jafari, P. S. Saladi, P. Sharma, A. S. Pasand, and A. Ghodsi.  Pro-KD: Progressive distillation by following the footsteps of the teacher.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 4714–4727, Gyeongju, Republic of Korea, Oct. 2022. International Committee on Computational Linguistics.  URL https://aclanthology.org/2022.coling-1.418.
  * Salimans and Ho [2022] T. Salimans and J. Ho.  Progressive distillation for fast sampling of diffusion models.  _ArXiv_ , abs/2202.00512, 2022.
  * Song et al. [2020] J. Song, C. Meng, and S. Ermon.  Denoising diffusion implicit models.  _arXiv:2010.02502_ , October 2020.  URL https://arxiv.org/abs/2010.02502.
  * Song and Ermon [2020] Y. Song and S. Ermon.  Improved techniques for training score-based generative models.  In H. Larochelle, M. Ranzato, R. Hadsell, M. Balcan, and H. Lin, editors, _Advances in Neural Information Processing Systems_ , volume 33, pages 12438–12448. Curran Associates, Inc., 2020.  URL https://proceedings.neurips.cc/paper_files/paper/2020/file/92c3b916311a5517d9290576e3ea37ad-Paper.pdf.
  * Sun and Yang [2023] Z. Sun and Y. Yang.  Difusco: Graph-based diffusion solvers for combinatorial optimization, 2023.
  * Wang et al. [2022] H. Wang, N. Wu, H. Yang, C. Hao, and P. Li.  Unsupervised learning for combinatorial optimization with principled objective relaxation.  _ArXiv_ , abs/2207.05984, 2022.
  * Wu et al. [2021] Y. Wu, W. Song, Z. Cao, J. Zhang, and A. Lim.  Learning improvement heuristics for solving routing problems.  _IEEE transactions on neural networks and learning systems_ , 33(9):5057–5069, 2021.
  * Yao et al. [2021] F. Yao, R. Cai, and H. Wang.  Reversible action design for combinatorial optimization with reinforcement learning.  _ArXiv_ , abs/2102.07210, 2021.


# Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-
Judges

Aman Singh Thakur
University of Massachusetts Amherst
<EMAIL_ADDRESS>
&Kartik Choudhary††footnotemark:
University of Massachusetts Amherst
<EMAIL_ADDRESS>
&Venkat Srinik Ramayapally††footnotemark:
University of Massachusetts Amherst
<EMAIL_ADDRESS>
&Sankaran Vaidyanathan
University of Massachusetts Amherst
<EMAIL_ADDRESS>
&Dieuwke Hupkes
Meta
<EMAIL_ADDRESS>
Equal contribution

###### Abstract

Offering a promising solution to the scalability challenges associated with
human evaluation, the LLM-as-a-judge paradigm is rapidly gaining traction as
an approach to evaluating large language models (LLMs). However, there are
still many open questions about the strengths and weaknesses of this paradigm,
and what potential biases it may hold. In this paper, we present a
comprehensive study of the performance of various LLMs acting as
judges.111Source code is available at https://github.com/UMass-Meta-LLM-
Eval/llm_eval.git We leverage TriviaQA as a benchmark for assessing objective
knowledge reasoning of LLMs and evaluate them alongside human annotations
which we found to have a high inter-annotator agreement. Our study includes
nine _judge models_ and nine _exam-taker models_ – both base and instruction-
tuned. We assess the judge models’ alignment across different model sizes,
families, and judge prompts. Among other results, our research rediscovers the
importance of using Cohen’s kappa as a metric of alignment as opposed to
simple percent agreement, showing that judges with high percent agreement can
still assign vastly different scores. We find that both Llama-3 70B and GPT-4
Turbo have an excellent alignment with humans, but in terms of _ranking_ exam
taker models, they are outperformed by both JudgeLM-7B and the lexical
matching method Contains, which have up to 34 points lower human alignment.
Through error analysis and various other studies, including the effects of
instruction length and leniency bias, we hope to provide useful lessons for
using LLMs as judges in the future.

##  1 Introduction

(a)

(b)

Figure 1: (a) Scores assigned to all exam-taker models by the various judge
models. (b) Average percent agreement (blue line) and Kappa scores (red bars)
of judge models with human judges. Error bars annotate standard deviation
across exam-taker models. Alignment is poor for most judge models, but both
Llama3 70B and GPT-4 Turbo have Cohen’s Kappa coefficient that are indicative
of excellent alignment (79 and 84, respectively), though still well below the
human alignment score of 96.

Over the last few years, large language models (LLMs) have demonstrated
remarkable capabilities across various domains (Radford et al., 2019; Brown et
al., 2020; Achiam et al., 2023; AI@Meta, 2024, i.a.). As more and more new
LLMs with different architectures and training methods continue to be released
and their capabilities expand, accurately evaluating their performance and
limitations becomes increasingly challenging (Zheng et al., 2024; Ohmer et
al., 2024; Benchekroun et al., 2023; Madaan et al., 2024). The empirical
evaluation of LLMs is particularly difficult due to the diversity of their
outputs and the wide range of tasks they are used for (Zhang et al., 2024; Li
et al., 2023a).

To evaluate LLMs, various methods have been proposed, that typically fall into
one of two broad categories. Benchmarks such as MMLU (Hendrycks et al., 2021),
TruthfulQA (Lin et al., 2021), and GSM8K (Cobbe et al., 2021) are used to
evaluate specific capabilities of LLMs in an automated manner. Additionally,
leaderboards like Chatbot Arena (Chiang et al., 2024) and Open LLM Leaderboard
(Beeching et al., 2023) assign ranks to models considering pair-wise rankings
of LLM outputs, done by humans or, in some cases, automated evaluation
methods. Since both strategies involve evaluating free-form text responses
generated by the LLMs, even in the first case, evaluating the responses is
often just as challenging as generating them (see e.g. Chang et al., 2023).

One proposed solution to this problem is to use multiple-choice question (MCQ)
benchmarks such as MMLU, and compare the log-probabilities of the potential
answers rather than evaluating the generated answer directly. However, the MCQ
paradigm ly limits the range of abilities that can be evaluated, and the setup
increasingly diverges from how LLMs are used in practice. Alternatively, the
use of lexical matching methods such as exact match (EM) or n-gram overlap to
evaluate the responses are practical and cost-efficient approaches, but are
susceptible to false negatives and often fail to adequately distinguish
between responses with subtle differences that change their semantic meaning.
This issue is exacerbated when evaluating instruction-tuned “chat” models that
are fine-tuned to carry out conversations with humans in natural language,
since their responses tend to be more verbose (Saito et al., 2023; Renze and
Guven, 2024). For these reasons, human evaluation remains the gold standard
for evaluating LLM responses. However, human evaluation is expensive, time-
consuming, and often impractical in many use cases. As a result, it has
increasingly become common practice to evaluate LLM responses using another
LLM as a judge model (Lin et al., 2021; Islam et al., 2023; Chiang and Lee,
2023; Liusie et al., 2024). While there are promises of alignment between LLM
judges and humans (Sottana et al., 2023; Zheng et al., 2024), there are also
many open questions about the strengths and weaknesses of the paradigm.

In this work, we study the properties of LLMs as judges, comparing them with
humans and automated evaluation methods. Contrary to prior work, we focus on a
relatively ‘clean’ scenario in which inter-human agreement is as high as 96%,
allowing us to distinguish ambiguity and subjectivity in the task itself from
potential issues with the judge models. Using the knowledge benchmark TriviaQA
(Joshi et al., 2017) as our playground, we investigate how nine different
_judge models_ with varying architectures and sizes judge nine different
_exam-taker models_. Our main findings are:

  * •

Even in relatively straightforward setups, only the best models are suitable
as judges. Out of the nine judge models we considered, only GPT-4 Turbo and
Llama-3 70B showed very high alignment with humans, though even those judges’
alignment is still well behind the human alignment coefficient for the task
(Figure 1).

  * •

Cohen’s kappa distinguishes judges better than percent alignment. In some
cases, high percent agreement can still give very divergent scores (Figure 2).

  * •

Even Cohen’s kappa is not all telling. While GPT-4 Turbo and Llama-3 both have
alignment scores that are considered excellent for discriminating different
exam-taker models, their results are comparable to alternative cheaper
approaches such as JudgeLM-7B and contains which have much lower alignment
scores but more systematic biases (Figure 3)

Through detailed error analysis, we uncover additional insights into judge
performance. Improved alignment appears to be driven by improved recall rates
and reduced false negatives. However, judge models struggle with under-
specified answers and tend to be lenient, affecting their evaluation
consistency. They are also sensitive to the length and quality of prompts.
And, surprisingly, even when the judge models are asked to evaluate an answer
matching verbatim with a reference answer, many judge models still sometimes
fail to evaluate it correctly.

Overall, our work showcases the strengths of the LLM-as-a-judge paradigm while
also highlighting the need for caution against overreliance on alignment
metrics, even in cases where they are high. Through error analysis, we also
highlight several common failure cases that require attention. With this, we
aim to contribute to a better general understanding of what is now becoming a
mainstream paradigm for evaluating LLMs.

##  2 Related work

Various recent studies have used or considered using LLMs as judges for tasks
such as evaluating story generation (Chiang and Lee, 2023), retrieval-
augmented generation (Es et al., 2023), visual QA (Mañas et al., 2024), code
comprehension (Zhiqiang et al., 2023), multilingual evaluation (Hada et al.,
2023) and more general open-ended tasks (Zheng et al., 2024). Zhang et al.
(2024) and Sottana et al. (2023) propose ways to standardise LLM evaluations
and the role that judge models might play in such solutions. Several studies
have demonstrated that state-of-the-art LLMs such as GPT-4 Turbo exhibit high
alignment with human judgments (Sottana et al., 2023; Zheng et al., 2024),
though others also illustrate that the paradigm is not yet without faults.
Zeng et al. (2023) propose a benchmark for evaluating the performance of LLMs
as judges, and other approaches have been proposed to improve LLM judges such
that they are aligned well with humans (Shankar et al., 2024; Zhu et al.,
2023).

Despite promising results in various settings, judge models still suffer from
known issues of current LLMs such as hallucinations and factual errors (Ye et
al., 2023; Turpin et al., 2023) and difficulty in following complex
instructions (Li et al., 2023b; He et al., 2024). Furthermore, various studies
have reported challenges such as position bias (Pezeshkpour and Hruschka,
2023; Zheng et al., 2023; Wang et al., 2023), verbosity bias (Saito et al.,
2023) in their preferences, confusing evaluation criteria (Hu et al., 2024),
or focusing more on the style and grammar compared to factuality (Wu and Aji,
2023). Recently, Liusie et al. (2024) have shown that LLMs perform better in
comparative assessment compared to absolute scoring, which can be used for
reliably measuring the relative performance of models (Liu et al., 2024) and
creating classifiers for pairwise grading (Huang et al., 2024).

We follow up on this line of work and investigate the strengths and weaknesses
of LLMs as judges. Unlike most prior work, we do not focus on pairwise
comparisons of LLM outputs on open-ended tasks, but on comparisons of LLM
outputs and reference answers. Since human alignment is high in this setting,
this provides a clean playground to study the strengths and weaknesses of LLMs
in detail. We also extend previous work by considering more LLMs, both as
judges and LLMs to be evaluated.

Exam-taker models (base & instruction-tuned) |  Llama-2 (7B, 13B and 70 B), Mistral 7B, GPT-4 Turbo
---|---
Judge models (instruction-tuned) |  Llama-2 (7B, 13B, and 70B), Llama-3 (8B and 70B),
|  Gemma 2B, Mistral 7B, JudgeLM 7B, GPT-4 Turbo
Table 1: The exam-taker models and judge models we use in our experiments. We
consider a wide variety of judge models; to get a comprehensive overview of
their (potential) biases, we consider exam-taker models of various sizes and
types.

##  3 Methodology

To evaluate the strengths and weaknesses of the LLM-as-a-judge paradigm, we
focus on a relatively controlled setup, in which _judge models_ assess answers
of _exam-taker models_ on the knowledge benchmark TriviaQA (Joshi et al.,
2017). We consider nine judge models and nine exam-taker models. More details
about the dataset and models are provided in Appendix B.

#### Benchmark

As our testbed, we use the TriviaQA dataset (Joshi et al., 2017), consisting
of 95K question-answer pairs sourced from 14 trivia and quiz league websites.
Each question in the train and validation set is annotated with a list of
short answers containing a minimal set of facts and evidence documents
collected from Wikipedia and the Web. For our experiments, we use the
validation set of the unfiltered partition of the benchmark, using the short
answers as reference answers. We use the training set for few-shot examples.
Since experiments require manual annotation of the exam-taker model responses,
we use a random sample of $400$ questions from the dataset.

#### Exam-taker models

To understand the strengths and weaknesses of different judges, we benchmark
pre-trained (base) and instruction-tuned (chat) exam-taker models across a
wide variety of model sizes and examine the quality of the evaluations from
different judge models. In particular, we consider Llama-2 (Touvron et al.,
2023) in 7B, 13B, and 70B parameter sizes for both base and chat versions,
Mistral 7B (Jiang et al., 2023) base and chat versions, and GPT-4
Turbo222Accessed via the OpenAI API between Mar 19th, 2024 and May 19th, 2024.
(Achiam et al., 2023) as the exam-taker models. The prompts for the exam-taker
models contain five few-shot examples of (question, answer) pairs from the
TriviaQA training set. The prompts for the instruction-tuned models
additionally include a command signaling the model to answer the given
question in a succinct manner similar to the provided examples. The prompts
are provided in Appendix C.

#### Judge models

To get a comprehensive view of the strengths and weaknesses of judge models
across different model sizes and architectures, we use instruction-tuned
versions of Llama-2 (Touvron et al., 2023) in 7B, 13B, and 70B sizes, Llama-3
(AI@Meta, 2024) in 8B and 70B sizes, Mistral 7B (Jiang et al., 2023), GPT-4
Turbo (Achiam et al., 2023), Gemma 2B (Gemma Team et al., 2024), and JudgeLM
7B (Zhu et al., 2023) as judges. The judges are instructed to respond with
only a single word, ‘‘correct’’ or ‘‘incorrect’’. The prompts can be found in
Appendix D. An overview of al exam-taker models and judge models is shown in
Table 1. For ease of reading, the judge models are depicted in a different
font than the exam-taker models.

#### Baselines

As baselines, we use two commonly used lexical evaluation techniques – exact
match (EM) and contains match (contains). For EM, a response is considered
correct if the response exactly matches one of the reference answers for the
given question. For contains, an answer is considered correct if at least one
of the reference answers is a sub-string of the response string. Both EM and
contains match are computed in a case-insensitive manner.

#### Alignment

We use two metrics to quantify alignment between judges: percent agreement and
Cohen’s kappa coefficient (Cohen, 1960). Percent agreement expresses a simple
percentage of the samples on which two annotators agree. Cohen’s kappa,
denoted as $\mathbf{\kappa}$, is an alignment metric that also takes into
account the possibility of chance agreement. It is generally considered to
provide a more robust measure of alignment. Details about the computation of
both metrics are given in Appendix E.

#### Human judgements

As a ground-truth assessment, we obtain human annotations for each exam-taker
model answer. The inter-human alignment is calculated between three human
judges using the answers to 600 questions by Llama-2 7B base, randomly sampled
from the TriviaQA dataset (Joshi et al., 2017); the human guidelines can be
found in Appendix F. We then determine collective “Human Judgment” through a
majority vote. We found that the average alignment among human evaluators with
the majority vote had a Cohen’s kappa score333The values of the Cohen’s kappa
coefficient have been scaled by $100\times$ for easier comparison with percent
agreement of $96.36\pm 1.67$, and the average percent agreement was
$98.33\%\pm 0.76\%$. Given this near-perfect alignment score, we consider only
one human evaluator per sample for the rest of our experiments, to reduce the
overall cost of human annotations. The set of questions for which we obtain
human annotations is identical for each exam-taker model.

##  4 Results

(a)

(b)

Figure 2: Delta evaluation score is calculated by taking judge score
difference with human judgment; y-axes are in log scale. Fig (a) shows a
skewed distribution for percent agreement and delta evaluation score. In Fig
(b) we observe that highly aligned LLM judges with kappa > 0.8 exhibit low
variability in scores. Conversely, judges with kappa < 0.8 demonstrate more
variability, impacting their reliability.

In this section we discuss our main results, primarily focusing on the
relationship between evaluations by various judge models and human evaluations
(§ 4.1), and how that impacts their usability (§ 4.2). To do so, we evaluate
their alignment with human judgment and assess how differently they rank the
nine exam-taker models compared to humans. In Section 5, we further analyse
their precision and recall to further investigate the types of errors that can
be made by various judge models. Details about compute requirements and others
costs for experiments are given in Appendix G.

###  4.1 Alignment between judge models and humans

We first compute $\kappa$ scores and percent agreement between the evaluations
of each judge model and the human annotators. Figure 1 shows that while
alignment is poor for most judge models, both Llama-3 70B and GPT-4 Turbo have
$\mathbf{\kappa}$ scores – 79 and 84, respectively – that are considered to
indicate excellent alignment. Nevertheless, there still is a significant
disparity between human judgment and judge models: GPT-4 Turbo is 12 points
behind human judgment. Notably, contains has a higher $\mathbf{\kappa}$ score
than half of the judge models, while EM has the lowest alignment among all
judges.

#### Cohen’s kappa vs percent agreement

In most cases, we observe that Cohen’s kappa and percent agreement are
following the same trend, with the exception of the values for Gemma 2B and
EM. Gemma 2B shows higher percent agreement compared to EM, yet it yields the
lowest $\mathbf{\kappa}$ score within the ensemble. Furthermore, there is a
significant difference in the actual values. For the percent agreement of
judge models, we note a 30-point difference between human judgment and EM,
while Cohen’s kappa exhibits a more substantial 53-point gap. This is also
visible in the general decline of alignment scores: while Llama-3 8B has a
$\kappa$ score of only 62, its percent agreement is still well above 80%.
Overall, $\mathbf{\kappa}$ scores offer a more precise representation of
diminishing trends in judge models compared to percent agreement.

#### Alignment vs assigned score

In Figure 2, we show the variation in scores assigned by the judge models to
various exam-taker models for different values of percent agreement (Figure
2(a)) and Cohen’s kappa (Figure 2(b)). We can see that for $\mathbf{\kappa}$ >
80, the evaluation scores by judge models are close to the human evaluation
scores for most of the judges, with a difference of up to 5 points in their
assigned scores (complete results table provided in Appendix H). For percent
agreement we observe deviations of up to 20 points in the evaluation scores
for similar kappa alignment. Furthermore, we observe that the deviation from
human-judgements can be quite distinct for different exam-taker models. In
Figure 1(a), Gemma 2B, for instance, sometimes assigns higher scores than
humans, and sometimes much lower. In the next section, we further explore this
particular pattern.

###  4.2 Exploring systematic patterns in judge models

In the previous section, we have seen that none of the judge models we
considered were aligned as well with humans as the humans themselves.
Furthermore, as can be seen in Figure 2, the scores assigned by even the best
aligned judge models can differ up to 10 points with the human-assigned
scores. However, while this may limit – to some extent – the utility of using
a judge models to get a perfect estimate of the exam-taker model’s capability
on the benchmark, the judge models may still offer valuable insights to
_differentiate_ between different exam-taker models. If judges exhibit
systematic biases such as – akin to a very strict teacher – consistently
rating any exam-taker model lower, they will not assign identical scores but
may assign identical _rankings_.

To evaluate this, we compare the rankings assigned by each judge model to the
nine exam-taker models by computing their Spearman’s rank correlation
coefficients $\rho$ (Spearman, 1904) with the human ranking. The rankings are
shown in Figure 3(a), with $\rho$ values in Appendix J. These results show
that contains demonstrates the highest alignment with the human ranking,
swapping the ranks of only two out of nine models. Notably, contains performs
on par with JudgeLM 7B (Zhu et al., 2023), a language model fine-tuned
specifically for evaluating language model responses. They are closely
followed by GPT-4 Turbo and Llama-3 70B, the judges with the best alignment.
Remarkably, while GPT-4 Turbo and Llama-3 70B rank the same three models in
positions three, four, and five as the human judge, they do so in different
orders. Several other judges have rank correlations higher than 0.7; it
appears they struggle to distinguish between poorer-performing exam-taker
models, but do well at distinguishing between better-performing ones.

(a)

(b)

Figure 3: (a) Contains and JudgeLM retain 67% of the human-assigned ranking,
closely followed by GPT-4 Turbo and LLama3-70B. All judges struggle to
distinguish between the poor-performing exam-taker models; (b) False positives
and negatives across different judge models in descending order of human
alignment. Both false negatives and false positives generally increase as
human alignment decreases, but well-aligned models tend to produce more false
negatives than false positives.

##  5 Analysis

To better understand the judge models, we conduct multiple case studies aimed
at identifying common errors and vulnerabilities.

###  5.1 Better aligned models: Recall gains, precision pains, and error
spotlights

We first investigate the precision and recall of the judge models. We plot
both – maintaining the ordering of Figure 1 – in Figure 4(a). It can be seen
that the precision does not show any clear observable trend with increasing
alignment, which can be further observed in Figure 3(b). The recall, on the
other hand, shows an increasing trend, with more aligned models having
comparatively fewer false negatives.

(a)

(b)

Figure 4: (a) Cohen’s kappa, precision, and recall for different judge models.
Recall improves with increasing human alignment ($R^{2}$ = 0.98), though
precision ($R^{2}$ = 0.0003) is not correlated with human alignment; (b)
$\mathbf{\kappa}$ scores for each judge across different prompt templates.
Except for GPT-4 Turbo, all judges struggle with too many detailed evaluation
guidelines.

Next, we analyse the types of errors made by the judge models by manually
annotating 900 outputs from Llama-7B Base with error codes, focusing on the
top performers GPT-4 Turbo and Llama-3 70B. We then determine the percentage
of each error type that are correctly judged to be incorrect by these two
models. The results are shown in Table 2, where it can be observed that both
GPT-4 Turbo and Llama-3 70B have a good error recall when the answers refer to
an incorrect entity, or when too many entities are present. Under-specified
and otherwise incorrect answers are most challenging for both judges, while
answers with too few entities are judged relatively accurately by GPT-4 but
less accurately by Llama-3 70B.

Error code | Explanation | Example | Proportion | GPT-4 recall | Llama-3 70B recall
---|---|---|---|---|---
Incorrect entity |  | Response refers to a wrong entity
---
| Henry VII, James I, Edward VI,
---
Mary I and Elizabeth I
86.9% | 98.3% | 96.6%
Under-specified |  | Response contains only part
---
of the answer
| Henry VII, Henry VIII, Edward,
---
Mary, and Elizabeth
37.3% | 33.9% | 23.3%
Too few entities |  | Response contains too few entities
---
| Henry VII, Edward VI,
---
Mary I and James I
2.47% | 80.0% | 60.0%
Too many entities |  | Response contains too many entities
---
| Henry VII, Henry VIII, Edward VI,
---
Mary I, James I, and Elizabeth I
2.7% | 90.1% | 90.1%
Other |  | Response is incorrect but cannot
---
be put into any of the above buckets
| I’m sorry but I do not know the
---
answer to that question
1.23% | 20.0% | 40.0%

Table 2: Error codes used to identify the types of errors made by exam-taker
models when answering questions. The example question in this case is
“Excluding Lady Jane Grey, who were the five monarchs of the House of Tudor?”,
with the correct answer “Henry VII, Henry VIII, Edward VI, Mary I and
Elizabeth I” (in any order).

###  5.2 Judge model sensitivity to prompt length and specificity

Next, we study the impact of the prompt on the ability of the judge models to
perform an accurate assessment of exam-taker models, with two somewhat
intertwined goals: 1) to study if the success of various judge models is
related to the _length_ of the prompt, and, 2) to study the degree to which
the judgments of the judge models change with the _specificity_ of the prompt.

We use four different prompt versions, varying in length and specificity. Each
prompt instructs the judge models on how to evaluate responses, with
complexity and detail increasing with the token count. The first two prompts
(_Without guidelines_ V1 and V2, with 45 and 58 tokens, respectively) simply
ask to evaluate the responses, without any further information, while more
elaborate guidance and examples are given in the longer prompts (_Guidelines
without examples_ and _Guidelines with examples_ , with 245 and 301 tokens,
respectively). All prompts are listed in Appendix K.

Figure 4(b) shows that Mistral 7B (Jiang et al., 2023), GPT-4 Turbo and
Llama-3 70B exhibit relatively low variance in their agreement with humans as
the level of information and the length of the prompt increases. For this
task, top performers’ (GPT-4 Turbo and Llama-3 70B) implicit definition of a
correct judgment seems well aligned with the provided instructions and thus
shows high alignment with humans even if no specific instructions are
provided. It can also be observed that only top performers appears to benefit
from the more detailed instructions, with a slight upward trend, whereas the
other models get less aligned with more instructions. This might be due to the
less powerful judges not being able to follow many instructions in the prompt
at the same time. Interestingly, in a follow-up experiment (see Appendix L),
when the order of references provided to the judge models is shuffled, Figure
6(b) demonstrates that larger judge models consistently maintain their
judgments regardless of the reference order, whereas smaller models except
Mistral 7B are more sensitive to the reference order given in the prompt.

###  5.3 Evaluating controlled responses

Figure 5: We observe that judge models remain robust when exam-taker models
produce responses identical to the prompt. However, this robustness diminishes
in the presence of hallucinated responses such as "Yes" and "Sure" from the
exam-taker models. Additionally, when assessing the gold standard answer,
judges do not consistently arrive at the correct judgement 100% of the time.

Next, we perform some simple tests on the judge models by asking them to
evaluate a set of dummy benchmark responses. For the first test, the answer to
be evaluated for each question is one of the references from the dataset,
verbatinm (the answer is thus always correct), while for the next three tests,
the answer to be evaluated is always incorrect, with the dummy exam-taker
model always responding with ‘‘Yes’’, and ‘‘Sure’’ for the second and the
third tests, respectively, and simply repeating the question for the fourth
test. Figure 5 shows that while some judge models are able to identify and
correctly mark the answers as correct (for the first test) or incorrect (for
the next three tests), some judges, notably Llama-2 70B incorrectly evaluates
a significant number of dummy answers, even though it shows a relatively high
alignment with humans on the benchmark evaluations (see Figure 1(b)).

We hypothesise that when the answers are plausible but incorrect (e.g. if the
question asks about the name of the author of a book, and the exam-taker model
gives the name of the wrong author), most judges are able to identify them as
being incorrect (by comparing it with the reference answer). However, the
judges might get confused about what they are supposed to evaluate if the
answer is completely unrelated to the question (such as the words ‘‘Yes’’ and
‘‘Sure’’). It is possible that in this situation a judge model tries to
evaluate one of the reference answers, thus marking it as correct, though
further research is required to identify the cause of this behavior.

###  5.4 Leniency bias in judge models

Judge model | $\kappa$ | $P_{c}$ | $P_{+}$
---|---|---|---
Gemma 2B | 0.50 | 0.62 | 0.80
Llama-2 7B | 0.66 | 0.68 | 0.36
Mistral 7B | 0.72 | 0.75 | 0.75
Llama-2 13B | 0.68 | 0.76 | 0.45
Llama-2 70B | 0.80 | 0.79 | 0.94
Llama-3 8B | 0.81 | 0.80 | 0.84
Llama-3 70B | 0.84 | 0.84 | 0.79
GPT-4 Turbo | 0.85 | 0.85 | 0.66
(a)

(b)

Figure 6: (a) Leniency bias across various judge models. To compute the
leniency bias in judge models we estimate $P_{c}$ and $P_{+}$ for different
judge models; (b) Consistency scores of each judge model for 3 random
permutations of references in the prompt. The consistency score is the
percentage of questions for which the judge model gives the same judgment for
all 3 runs. We observe that the bigger models are more consistent as judges
and less sensitive to reference orders in the prompt. More details are given
in Appendix L.

To get a general sense of the inherent biases or misalignment in the
evaluation criteria that might be present in the judge models, we estimate if
they have a positive or negative bias in their judgment. To do so, we assume
that a judge assigns the correct judgment (i.e. same evaluation as the ground
truth) with a probability of $P_{c}$ and assigns the rest of the samples to be
‘‘correct’’ with a probability $P_{+}$, which we call their _leniency bias_.
We estimate the values of $P_{c}$ and $P_{+}$ from the benchmark results,444
The theoretical derivation of the expressions for $P_{c}$ and $P_{+}$, as well
as the empirical validation for their estimated values can be found in
Appendix M. and show them in Figure 6(a). We observe that $P_{+}$ for most
models is significantly higher than $0.5$, indicating a tendency of the judge
models to evaluate responses as ‘‘correct’’ when their evaluation criteria are
not completely aligned with the provided instructions.

##  6 Conclusion

In this work, we provide an extensive study of the properties of LLMs as
judges, comparing them with human evaluation and automated evaluation methods.
By focusing on a relatively ‘clean’ evaluation scenario in which inter-human
agreement is high, we examine the potential issues with the LLM-as-a-judge
paradigm separately from the ambiguity and subjectivity in the task itself. We
find that even in relatively straightforward setups, smaller and more cost-
efficient models are less effective judges compared to the best available LLMs
– also if they are specifically trained as judges, such as JudgeLM – and that
even the best LLMs fail to meet the consistency of humans. Furthermore, though
previous work commonly used percent agreement, we found that Cohen’s kappa
($\mathbf{\kappa}$) distinguishes judges much better. However, we observe that
even judges with excellent $\mathbf{\kappa}$ scores may require further
scrutiny. While GPT-4 Turbo and Llama-3 both have excellent alignment scores,
simpler and more cost-efficient approaches like JudgeLM and contains perform
better when discriminating between the exam-taker models in terms of their
_ranking_ , despite having much lower alignment scores and more systematic
biases. In further analyses, we find that: 1) LLMs tend to judge positively
when in doubt, and this is more pronounced for small models than for larger
ones; 2) Judge models with lower alignment lack precision rather than recall;
3) GPT-4 Turbo and Llama-3 70B are generally robust across different prompts,
but are difficult to ‘steer’ in their judgments; 4) Some judge models can be
easily fooled by dummy answers such as ‘Yes’ and ‘Sure’; 5) Judge models are
better at detecting completely incorrect answers than partially incorrect
ones.

Overall, this work adds to the realm of LLM evaluation research by assessing
judges within a clearly defined and objective framework. Our results highlight
the utility of using some LLMs as judges but also urge caution in blindly
trusting their judgments, even if they are found to be well-aligned with
humans. For practitioners using LLMs as judges – regardless of the setup – we
recommend not only computing percent agreement, but also Cohen’s kappa, and
pairing these with a qualitative analysis to ensure that conclusions from
judge models are less susceptible to biases. We further elaborate on the
limitations of our work in Appendix A. In the future, we plan to expand our
work to increasingly more complex scenarios with more open-ended answers and
variability, and more generally assess how consistent our findings are across
dataset samples, benchmarks, and prompt templates.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  GPT-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * AI@Meta (2024) AI@Meta. 2024.  Llama 3 model card.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_.
  * Beeching et al. (2023) Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. 2023.  Open llm leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard.
  * Benchekroun et al. (2023) Youssef Benchekroun, Megi Dervishi, Mark Ibrahim, Jean-Baptiste Gaya, Xavier Martinet, Grégoire Mialon, Thomas Scialom, Emmanuel Dupoux, Dieuwke Hupkes, and Pascal Vincent. 2023.  Worldsense: A synthetic benchmark for grounded reasoning in large language models.  _arXiv preprint arXiv:2311.15930_.
  * Brown et al. (2020) Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.
  * Chang et al. (2023) Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. 2023.  A survey on evaluation of large language models.  _ACM Transactions on Intelligent Systems and Technology_.
  * Chiang and Lee (2023) Cheng-Han Chiang and Hung-yi Lee. 2023.  Can large language models be an alternative to human evaluations? _arXiv preprint arXiv:2305.01937_.
  * Chiang et al. (2024) Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024.  Chatbot arena: An open platform for evaluating LLMs by human preference.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. 2021.  Training verifiers to solve math word problems.  _arXiv preprint arXiv:2110.14168_.
  * Cohen (1960) J. Cohen. 1960.  A Coefficient of Agreement for Nominal Scales.  _Educational and Psychological Measurement_ , 20(1):37.
  * Es et al. (2023) Shahul Es, Jithin James, Luis Espinosa-Anke, and Steven Schockaert. 2023.  RAGAS: Automated evaluation of retrieval augmented generation.
  * Gemma Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, Pouya Tafti, Léonard Hussenot, Pier Giuseppe Sessa, Aakanksha Chowdhery, Adam Roberts, Aditya Barua, Alex Botev, Alex Castro-Ros, Ambrose Slone, Amélie Héliou, Andrea Tacchetti, Anna Bulanova, Antonia Paterson, Beth Tsai, Bobak Shahriari, Charline Le Lan, Christopher A. Choquette-Choo, Clément Crepy, Daniel Cer, Daphne Ippolito, David Reid, Elena Buchatskaya, Eric Ni, Eric Noland, Geng Yan, George Tucker, George-Christian Muraru, Grigory Rozhdestvenskiy, Henryk Michalewski, Ian Tenney, Ivan Grishchenko, Jacob Austin, James Keeling, Jane Labanowski, Jean-Baptiste Lespiau, Jeff Stanway, Jenny Brennan, Jeremy Chen, Johan Ferret, Justin Chiu, Justin Mao-Jones, Katherine Lee, Kathy Yu, Katie Millican, Lars Lowe Sjoesund, Lisa Lee, Lucas Dixon, Machel Reid, Maciej Mikuła, Mateo Wirth, Michael Sharman, Nikolai Chinaev, Nithum Thain, Olivier Bachem, Oscar Chang, Oscar Wahltinez, Paige Bailey, Paul Michel, Petko Yotov, Rahma Chaabouni, Ramona Comanescu, Reena Jana, Rohan Anil, Ross McIlroy, Ruibo Liu, Ryan Mullins, Samuel L Smith, Sebastian Borgeaud, Sertan Girgin, Sholto Douglas, Shree Pandya, Siamak Shakeri, Soham De, Ted Klimenko, Tom Hennigan, Vlad Feinberg, Wojciech Stokowiec, Yu hui Chen, Zafarali Ahmed, Zhitao Gong, Tris Warkentin, Ludovic Peran, Minh Giang, Clément Farabet, Oriol Vinyals, Jeff Dean, Koray Kavukcuoglu, Demis Hassabis, Zoubin Ghahramani, Douglas Eck, Joelle Barral, Fernando Pereira, Eli Collins, Armand Joulin, Noah Fiedel, Evan Senter, Alek Andreev, and Kathleen Kenealy. 2024.  Gemma: Open models based on gemini research and technology.
  * Hada et al. (2023) Rishav Hada, Varun Gumma, Adrian de Wynter, Harshita Diddee, Mohamed Ahmed, Monojit Choudhury, Kalika Bali, and Sunayana Sitaram. 2023.  Are large language model-based evaluators the solution to scaling up multilingual evaluation? _arXiv preprint arXiv:2309.07462_.
  * He et al. (2024) Qianyu He, Jie Zeng, Wenhao Huang, Lina Chen, Jin Xiao, Qianxi He, Xunzhe Zhou, Jiaqing Liang, and Yanghua Xiao. 2024.  Can large language models understand real-world complex instructions? _Proceedings of the AAAI Conference on Artificial Intelligence_ , 38(16):18188–18196.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021.  Measuring massive multitask language understanding.
  * Hu et al. (2024) Xinyu Hu, Mingqi Gao, Sen Hu, Yang Zhang, Yicheng Chen, Teng Xu, and Xiaojun Wan. 2024.  Are LLM-based evaluators confusing nlg quality criteria? _arXiv preprint arXiv:2402.12055_.
  * Huang et al. (2024) Hui Huang, Yingqi Qu, Jing Liu, Muyun Yang, and Tiejun Zhao. 2024.  An empirical study of LLM-as-a-Judge for LLM evaluation: Fine-tuned judge models are task-specific classifiers.
  * Islam et al. (2023) Pranab Islam, Anand Kannappan, Douwe Kiela, Rebecca Qian, Nino Scherrer, and Bertie Vidgen. 2023.  FinanceBench: A new benchmark for financial question answering.  _arXiv preprint arXiv:2311.11944_.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7B.  _arXiv preprint arXiv:2310.06825_.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel S Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  _arXiv preprint arXiv:1705.03551_.
  * Li et al. (2023a) Junlong Li, Shichao Sun, Weizhe Yuan, Run-Ze Fan, Hai Zhao, and Pengfei Liu. 2023a.  Generative judge for evaluating alignment.  _arXiv preprint arXiv:2310.05470_.
  * Li et al. (2023b) Shiyang Li, Jun Yan, Hai Wang, Zheng Tang, Xiang Ren, Vijay Srinivasan, and Hongxia Jin. 2023b.  Instruction-following evaluation through verbalizer manipulation.  _arXiv preprint arXiv:2307.10558_.
  * Lin et al. (2021) Stephanie Lin, Jacob Hilton, and Owain Evans. 2021.  TruthfulQA: Measuring how models mimic human falsehoods.  _arXiv preprint arXiv:2109.07958_.
  * Liu et al. (2024) Yinhong Liu, Han Zhou, Zhijiang Guo, Ehsan Shareghi, Ivan Vulic, Anna Korhonen, and Nigel Collier. 2024.  Aligning with human judgement: The role of pairwise preference in large language model evaluators.  _arXiv preprint arXiv:2403.16950_.
  * Liusie et al. (2024) Adian Liusie, Potsawee Manakul, and Mark Gales. 2024.  LLM comparative assessment: Zero-shot NLG evaluation through pairwise comparisons using large language models.  In _Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 139–151, St. Julian’s, Malta. Association for Computational Linguistics.
  * Madaan et al. (2024) Lovish Madaan, Aaditya K. Singh, Rylan Schaeffer, Andrew Poulton, Sanmi Koyejo, Pontus Stenetorp, Sharan Narang, and Dieuwke Hupkes. 2024.  Quantifying variance in evaluation benchmarks.  _arXiv preprint arXiv:/2406.10229_.
  * Mañas et al. (2024) Oscar Mañas, Benno Krojer, and Aishwarya Agrawal. 2024.  Improving automatic vqa evaluation using large language models.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 38(5):4171–4179.
  * Ohmer et al. (2024) Xenia Ohmer, Elia Bruni, and Dieuwke Hupkes. 2024.  From form (s) to meaning: Probing the semantic depths of language models using multisense consistency.  _arXiv preprint arXiv:2404.12145_.
  * Pezeshkpour and Hruschka (2023) Pouya Pezeshkpour and Estevam Hruschka. 2023.  Large language models sensitivity to the order of options in multiple-choice questions.  _arXiv preprint arXiv:2308.11483_.
  * Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019.  Language models are unsupervised multitask learners.  _OpenAI blog_ , 1(8):9.
  * Renze and Guven (2024) Matthew Renze and Erhan Guven. 2024.  The benefits of a concise chain of thought on problem-solving in large language models.  _arXiv preprint arXiv:2401.05618_.
  * Saito et al. (2023) Keita Saito, Akifumi Wachi, Koki Wataoka, and Youhei Akimoto. 2023.  Verbosity bias in preference labeling by large language models.
  * Shankar et al. (2024) Shreya Shankar, JD Zamfirescu-Pereira, Björn Hartmann, Aditya G Parameswaran, and Ian Arawjo. 2024.  Who validates the validators? aligning llm-assisted evaluation of llm outputs with human preferences.  _arXiv preprint arXiv:2404.12272_.
  * Sottana et al. (2023) Andrea Sottana, Bin Liang, Kai Zou, and Zheng Yuan. 2023.  Evaluation metrics in the era of gpt-4: reliably evaluating large language models on sequence to sequence tasks.  _arXiv preprint arXiv:2310.13800_.
  * Spearman (1904) C. Spearman. 1904.  The proof and measurement of association between two things.  _The American Journal of Psychology_ , 15(1):72–101.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Turpin et al. (2023) Miles Turpin, Julian Michael, Ethan Perez, and Samuel Bowman. 2023.  Language models don't always say what they think: Unfaithful explanations in chain-of-thought prompting.  In _Advances in Neural Information Processing Systems_ , volume 36, pages 74952–74965. Curran Associates, Inc.
  * Wang et al. (2023) Peiyi Wang, Lei Li, Liang Chen, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui. 2023.  Large language models are not fair evaluators.  _arXiv preprint arXiv:2305.17926_.
  * Wu and Aji (2023) Minghao Wu and Alham Fikri Aji. 2023.  Style over substance: Evaluation biases for large language models.
  * Ye et al. (2023) Hongbin Ye, Tong Liu, Aijia Zhang, Wei Hua, and Weiqiang Jia. 2023.  Cognitive mirage: A review of hallucinations in large language models.  _arXiv preprint arXiv:2309.06794_.
  * Zeng et al. (2023) Zhiyuan Zeng, Jiatong Yu, Tianyu Gao, Yu Meng, Tanya Goyal, and Danqi Chen. 2023.  Evaluating large language models at evaluating instruction following.  _arXiv preprint arXiv:2310.07641_.
  * Zhang et al. (2024) Yue Zhang, Ming Zhang, Haipeng Yuan, Shichun Liu, Yongyao Shi, Tao Gui, Qi Zhang, and Xuanjing Huang. 2024.  Llmeval: A preliminary study on how to evaluate large language models.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 38(17):19615–19622.
  * Zheng et al. (2023) Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2023.  On large language models’ selection bias in multi-choice questions.  _arXiv preprint arXiv:2309.03882_.
  * Zheng et al. (2024) Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. 2024.  Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena.  _Advances in Neural Information Processing Systems_ , 36.
  * Zhiqiang et al. (2023) Yuan Zhiqiang, Liu Junwei, Zi Qiancheng, Liu Mingwei, Peng Xin, Lou Yiling, et al. 2023.  Evaluating instruction-tuned large language models on code comprehension and generation.  _arXiv e-prints arXiv:2308.01240_.
  * Zhu et al. (2023) Lianghui Zhu, Xinggang Wang, and Xinlong Wang. 2023.  Judgelm: Fine-tuned large language models are scalable judges.

##  Appendix A Limitations

In our work, we have shown potential pitfalls of using LLMs as judges, as well
as the issues with commonly used metrics for quantifying their judgment
quality. We have observed several trends and interesting phenomena regarding
the behaviour of judge models in the LLM-as-a-judge paradigm, such as poor
correlation or precision with alignment (§ 5.1), inability of smaller judge
models to follow complex and nuanced instructions (§ 5.2), and tendencies of
judge models to be lenient in their judgment when they are unsure (§ 5.4).

Broadly, our results show that caution and scrutiny are required in using LLMs
as judges, but not all our specific conclusions about rankings may hold across
different examples, different judge models and exam-taker model accuracies,
and tasks. For example, on a small experiment with a different sample and
different instructions, we observed that when we limited the questions to only
those that have fewer than 10 or 20 references, there was a consistent 5-10
point drop in percent-alignment across all judge models, but the
discriminative ability of the top-performing judges improved. In that example,
in which the overall accuracy of the models was substantially lower than in
the example in our main experiment, we also observed a closer match between
Cohen’s kappa and percent alignment, suggesting that the accuracies of the
exam-taker models being compared may play an important role in how well their
scores match.

All in all, these differences underline how finicky using LLMs as judges can
be, and with that confirm the overall conclusions of our study that much more
work is needed to better understand the strengths and limitations of judge
models across a wide range of scenarios and model accuracies. We consider
assessing the strengths across multiple different samples and tasks, which
would require many more human annotations, outside the scope of this paper and
leave such experimentation for future work.

##  Appendix B Model and dataset details

In Table 3, we show the different models and datasets used in our experiments,
along with version and license details.

Table 3: Version and license details for the different models and datasets used in experiments. Asset | Version | License
---|---|---
TriviaQA | mandarjoshi/trivia_qa | apache-2.0
Llama-2 7B Base | meta-llama/Llama-2-7b-hf | llama2
Llama-2 7B Chat | meta-llama/Llama-2-7b-chat-hf | llama2
Llama-2 13B Base | meta-llama/Llama-2-13b-hf | llama2
Llama-2 13B Chat | meta-llama/Llama-2-13b-chat-hf | llama2
Llama-2 70B Base | meta-llama/Llama-2-70b-hf | llama2
Llama-2 70B Chat | meta-llama/Llama-2-70b-chat-hf | llama2
Mistral 7B Base | mistralai/Mistral-7B-v0.1 | apache-2.0
Mistral 7B Chat | mistralai/Mistral-7B-Instruct-v0.2 | apache-2.0
Llama-3 8B Chat | meta-llama/Meta-Llama-3-8B-Instruct | llama3
Llama-3 70B Chat | meta-llama/Meta-Llama-3-70B-Instruct | llama3
JudgeLM | BAAI/JudgeLM-7B-v1.0 | Non-commercial license
GPT-4 Turbo | gpt-4-turbo-2024-04-09 | N/A
Qwen 0.5B Chat | Qwen/Qwen1.5-0.5B-Chat | tongyi-qianwen
Qwen 1.8B Chat | Qwen/Qwen1.5-1.8B-Chat | tongyi-qianwen
Qwen 4B Chat | Qwen/Qwen1.5-4B-Chat | tongyi-qianwen
Qwen 7B Chat | Qwen/Qwen1.5-7B-Chat | tongyi-qianwen
Qwen 14B Chat | Qwen/Qwen1.5-14B-Chat | tongyi-qianwen
Qwen 72B Chat | Qwen/Qwen1.5-72B-Chat | tongyi-qianwen

##  Appendix C Model evaluation prompt templates

In Figure 7 and Figure 8, we show the prompt templates used for the base and
chat exam-taker models during the question answering process.

Figure 7: Prompt template for base exam-taker models  Figure 8: Prompt
template for Chat exam-taker models

##  Appendix D Judge LLM Prompt templates

In Figure 9, we show the prompt template used to guide the judge models during
the evaluation process of a 400-question sample from the TriviaQA unfiltered
dataset.

Figure 9: Prompt templates for the judge models

##  Appendix E Metrics for judge models

If one of the annotators is taken to be the reference, then the annotations of
the other annotator can be categorized as true positives, false positives,
true negatives, and false negatives, with the total number of each of them in
a benchmark being represented by $T_{P},F_{P},T_{N},$ and $F_{N}$
respectively.

Percent agreement is simply the ratio of the numbers of times two annotators
agree with each other relative to the total number of annotations. This ratio
can have values between $0$ and $1$. For the binary case, the alignment ratio
$\rho$ is given as

$\rho=\frac{T_{P}+T_{N}}{T_{P}+F_{P}+T_{N}+F_{N}}.$ (1)

Cohen’s kappa coefficient, or Cohen’s kappa for short (Cohen, 1960), measures
the alignment of two annotators while also taking into account the possibility
of agreement by pure chance. This coefficient can have values between $-1$ and
$1$, but is usually above $0$ in most real-world situations. The value of
Cohen’s kappa is given as

$\displaystyle\kappa$
$\displaystyle=\frac{p_{o}-p_{e}}{1-p_{e}}=\frac{2(T_{P}T_{N}-F_{P}F_{N})}{(T_{P}+F_{P})(T_{N}+F_{P})+(T_{P}+F_{N})(T_{N}+F_{N})}.$
(2)

Here, $p_{o}$ is the relative observed agreement, and $p_{e}$ is the
hypothetical probability of chance agreement. This coefficient is considered
to be a more robust measure of inter-annotator alignment, but also less
interpretable in terms of what a particular value of $\kappa$ means.
Generally, values of $\kappa$ in ranges $[0,0.2)$, $[0.2,0.4)$, $[0.4,0.6)$,
$[0.6,0.8)$, and $[0.8,1)$ are considered to indicate no alignment, slight
alignment, moderate alignment, substantial alignment, and near-perfect
alignment respectively, with $\kappa=1$ indicating perfect alignment.

##  Appendix F Human Annotation Guidelines

We provide the guidelines used for human evaluation below.

Humn annotation guidelines You will be given a question, a set of reference
answers and the answer given by an LLM. Your task is to judge if the answer
given by the LLM is correct, as if you were the LLMs teacher grading their
exam. An answer should be counted as correct if it is semantically equivalent
to (one of the) reference answers. In doing so, please follow the following
guidelines: • Underspecified answers (e.g. "December" instead of "December
20") should be marked incorrect. • Answers that have more information than
requested (e.g. "December 20, in Paris" instead of "December 20") should be
marked correct, provided the extra information is not incorrect or contrasting
the rest of the answer. • Answers with unnecessary verbosity but correct
answers should be marked correct (E.g. ‘‘Thanks for asking this question! The
correct answer is: ..."). If you have trouble judging whether the answer is
correct, for instance because you feel you are lacking knowledge required to
judge so, please indicate so by marking the answer "maybe correct" or ‘‘maybe
incorrect", so that we can further review it.

##  Appendix G Experiment costs

The costs for the different experiments described in this work belong in three
categories – GPU-hours for running open-source models on one or more Nvidia
A100 GPUs, OpenAI credits for making API calls to OpenAI models,555Pricing
details for OpenAI models are available at https://openai.com/api/pricing/ and
human hours for manual annotations of benchmark responses. The estimated costs
for the final reported experiments are given in Table 4. In addition to this,
previous unreported experiments and trials had an approximate cost of 120 GPU-
hours, 100 USD in OpenAI credits, and 50 human hours, bringing the total
experimental cost for this work to approximately 200 GPU-hours, USD 125 OpenAI
credits, and 75 human annotation hours.

Table 4: Estimated costs for the final reported experiments. GPU-hours are in equivalent Nvidia A100 hours, OpenAI credits are in USD, and human hours are time spent in manual annotation. Experiment | GPU-hours | OpenAI credits | Human hours
---|---|---|---
Main benchmarks | 5 | 2 | -
Main evaluations | 30 | 8 | 10
Human alignment | 1 | - | 6
Error analysis | 1.5 | - | 5
Controlled responses | 15 | - | -
Leniency bias | 5 | 5 | -
Guideline bias | 10 | 5 | 1
Reference bias | 5 | 4 | 1
Total | 72.5 | 24 | 23

##  Appendix H Judge Scores

We show the scores assigned by each judge model to each exam-taker model,
visualised in Figure 1(a) in Table 5.

Table 5: Judge model score card for every exam-taker model. | Exam taker models
---|---
| Llama2 | Mistral | GPT-4
| Base | Chat | Base | Instruct |
Judge Models | 7B | 13B | 70B | 7B | 13B | 70B | 7B |
Llama 3 8B | 68.75 | 76.00 | 85.25 | 52.91 | 75.0 | 78.5 | 62.5 | 66.00 | 76.25
Llama 3 70B | 66.00 | 76.50 | 87.00 | 61.52 | 68.50 | 78.25 | 73.5 | 65.00 | 94.50
Llama 2 7B | 61.75 | 71.25 | 80.25 | 58.48 | 77.25 | 58.00 | 68.25 | 80.50 | 86.75
Llama 2 13B | 53.00 | 61.50 | 66.50 | 46.07 | 27.75 | 39.50 | 57.50 | 39.00 | 73.50
Llama 2 70B | 68.25 | 76.30 | 86.50 | 64.05 | 80.04 | 82.25 | 77.00 | 74.25 | 95.25
Mistral 7B | 67.75 | 75.00 | 85.25 | 64.55 | 68.75 | 77.50 | 74.50 | 68.50 | 92.75
Gemma 2B | 67.00 | 74.00 | 84.00 | 100.00 | 44.25 | 75.25 | 100.00 | 79.50 | 99.75
JudgeLM | 52.50 | 60.50 | 64.00 | 41.52 | 32.50 | 60.00 | 57.25 | 45.5 | 68.50
GPT-4 | 63.25 | 75.00 | 85.0 | 56.7 | 57.50 | 72.0 | 72.5 | 52.75 | 92.25
Exact Match | 46.75 | 56.00 | 63.75 | 24.05 | 0.25 | 36.25 | 59.50 | 20.25 | 58.25
Contains Match | 50.75 | 60.00 | 68.00 | 38.98 | 46.25 | 59.50 | 57.25 | 44.00 | 70.00
Human Eval | 62.25 | 72.75 | 83.75 | 56.00 | 56.50 | 72.25 | 71.75 | 60.75 | 91.50

##  Appendix I Exam-taker model base vs chat analysis

Table 6: Scores of base and chat models by various judges | Judge models
---|---
Base-Chat pair | EM | Contains | Human |  GPT-4 Turbo |  Llama-3 70B
| Base | Chat | Base | Chat | Base | Chat | Base | Chat | Base | Chat
Llama-2 7B | 46.75 | 24.05 | 50.75 | 38.98 | 62.25 | 56.00 | 63.25 | 56.75 | 66.0 | 61.50
Mistral 7B | 59.50 | 20.25 | 57.25 | 44.00 | 71.75 | 60.75 | 72.50 | 52.75 | 73.5 | 65.0
Llama-2 13B | 56.00 | 0.25 | 60.00 | 46.25 | 72.75 | 56.50 | 75.00 | 57.50 | 76.50 | 68.50
Llama-2 70B | 63.75 | 36.25 | 68.00 | 59.50 | 83.75 | 72.25 | 85.00 | 72.00 | 87.00 | 78.25

Given the human judgments we have available, we take the opportunity to
investigate the performance differences between base and their corresponding
chat models. In Table 6, we show the scores assigned by various judge models
to four base-chat pairs. According to the default metric EM, the base models
outperform the chat models by a large margin. Interestingly, while this
difference gets smaller when the answers are judged by humans (second column)
or GPT-4 Turbo, there is still a substantial difference for all four pairs,
suggesting that the difference is not merely an effect of the increased
verbosity of the chat models. Further evidence for that hypothesis is provided
by Figure 10(b), in which we can see that while 14% of the errors are shared
between the base-chat pairs, almost another 14% of the examples get judged
correctly by the base models but not by the chat models, while the opposite
happens in only 2.5% of the cases. We consider two alternative hypotheses:

  * i)

The chat models have a worse understanding of the particular prompt format,
which is tuned more to fit base models; or

  * ii)

The chat models have ‘unlearned’ some knowledge during their alignment
training.

To disentangle these two factors, we manually analyse 400 questions for
Llama-2 70B and Llama-2 70B-chat, using our earlier error codes. The results,
shown in Figure 10(a), sugest that, at least to some extent, the difference
between base and chat models is in fact due to ‘unlearning’ of knowledge:
while the number of errors is more or less equal among most categories, there
is a stark difference in the _incorrect entity_ category. Substantially more
often than the base models, the chat models do answer the question with a
semantically plausible but incorrect entity. In Table 7-9, we provide examples
of such cases. The results do not show any evidence to support the first
hypothesis: the number of errors where the answer cannot be parsed or is just
entirely incorrect does not differ between base and chat models.

(a)

(b)

Figure 10: a) Distribution of incorrect question counts by error codes for the
Llama2 70B Base vs Chat exam-taker models evaluated on 400 questions. b) Pie
chart showing the percentage of questions categorized by the judgment from
Base and Chat models. Table 7: Knowledge unlearning example 1. Question:
---
Which British artist’s works include ‘The First Real Target’?
References |  Peter Blake, Peter Balke, Sir Peter Blake
LLama-2 70B Base |  Peter Blake
LLama-2 70B Chat |  Patrick Caulfield
Mistral 7B Base |  David Hockney
Mistral 7B Chat |  Damien Hirst
Table 8: Knowledge unlearning example 2 Question:
---
Who was the first cricketer to score 10,000 test runs?
References |  Sunil Gavaskar, Sunil Manohar Gavaskar, SM Gavaskar, Sunny gavaskar, Gavaskar
LLama-2 70B Base |  Sunil Gavaskar
LLama-2 70B Chat |  Sachin Tendulkar
Mistral 7B Base |  Sachin Tendulkar
Mistral 7B Chat |  Sachin Tendulkar was the first cricketer to score 10,000 runs in Test matches.
Table 9: Knowledge unlearning example 3 Question:
---
‘Uncle Harry’s Coat’ was the first garment produced by which famous jacket
manufacturer, based in Simonside, Newcastle Upon Tyne?
References |  Barbour
LLama-2 70B Base |  Barbour
LLama-2 70B Chat |  Barbour
Mistral 7B Base |  Barbour
Mistral 7B Chat |  Jack Walker & Sons

##  Appendix J Exam-taker model ranking correlation

In Table 10, we show the Spearman’s rank correlation coefficient (Spearman,
1904) ($\rho$) with human judgment. Since $\rho$ > 0.7 is considered well
aligned, only Llama-7B and Gemma-2B have poor rank correlation with human
judgment.

Table 10: Spearman Rank Correlation Coefficient $\rho$. Judges | $\rho$
---|---
Contains | 0.98
JudgeLM-7B | 0.98
GPT-4 | 0.93
Llama3-70B | 0.93
Mistral-7B | 0.92
Llama-13B | 0.82
EM | 0.78
Llama3-8B | 0.77
Llama-70B | 0.75
Llama-7B | 0.39
Gemma-2B | 0.21

##  Appendix K Too much info confuses judges

In Figure 11-14, we report the guidelines we used for the experiments in §
5.2. The simplest prompt used is Without Guidelines v1 (see Figure 11) where
we define a sequential and structured process for the judge model. In Without
Guidelines v2 (see Figure 12), we add an additional focus on the overall task
and outcome as well. For Guidelines without examples (see Figure 13), we
provide the judge models with detailed instructions about the task at hand,
along with explicit guidelines on how to evaluate the answers. Additionally,
for Guidelines with examples(see Figure 14), we also provide examples to the
judge models for further reference.

Figure 11: Without Guidelines v1 prompt template for the judge models

Figure 12: Without Guidelines v2 prompt template for the judge models

Figure 13: Guidelines without examples Prompt template for the judge models

Figure 14: Guidelines with Examples Prompt template for the judge models

##  Appendix L Judge models are sensitive to reference order

We investigate the judges’ sensitivity to reference order by providing the
same prompt, question and model response to the judge models, but shuffling
the reference order in three different permutations. We compute the
consistency score of the model as the percentage of questions for which it
gives the same judgment all the 3 times. We observe that the model is more
likely to evaluate an answer as correct if the corresponding reference appears
early in the list of references (see Figure 15). The smaller judge models
sometimes fail to capture all the information in the prompt, and provide
judgement based on their own knowledge rather than going by the references
(see Figure 16).

Figure 15: Example of Llama2-7B getting confused when the order of the
references are changed

Figure 16: Example of Llama2-7B failing to identify the task by changing the
order of the references.

##  Appendix M Leniency Bias

As described in § 5.4, for the purpose of the leniency bias experiments, we
assume that a judge assigns the correct judgment with a probability of $P_{c}$
and randomly assigns the rest of the samples to be “correct” with a
probability $P_{+}$. In this section, we derive the mathematical expressions
for $P_{c}$ and $P_{+}$. We assume that in the case of misalignment between
the evaluation criteria of guidelines and judge models, the probability of
getting an evaluation of ‘‘correct’’ is independent of the actual correctness
of the answer (i.e. the judge model effectively flips a coin to give out its
judgement). For any given benchmark and judge model, we denote the ground-
truth score as $s$, and the true positive and true negative rates as $t_{P}$
and $t_{N}$, respectively, all normalized to be between $0$ and $1$.

Now, based on our assumptions, the true positives, where the exam-taker model
response is correct, and also correctly identified by the judge model to be
correct, would be comprised of two possible cases: 1) The judge evaluates it
correctly according to the given evaluation criteria with a probability of
$P_{c}$; and 2) The judge does not evaluate it according to the given criteria
with a probability of $1-P_{c}$, but the evaluation still happens to be
correct with a probability of $P_{+}$. With the total ratio of the correct
responses being $s$, the true positive rate is therefore given by –

$t_{P}=s[P_{c}+(1-P_{c})P_{+}]$ (3)

Similarly, the true negatives, where the exam-taker model response is
incorrect, and also correctly identified by the judge model to be incorrect,
would also be comprised of two cases: 1) The judge evaluates it correctly
according to the given evaluation criteria with a probability of $P_{c}$.2)
The judge does not evaluate it according to the given criteria with a
probability of $1-P_{c}$, but the evaluation still happens to be correct with
a probability of $1-P_{+}$. With the total ratio of the incorrect responses
being $1-s$, the true negative rate is therefore given by –

$t_{N}=(1-s)[P_{c}+(1-P_{c})(1-P_{+})].$ (4)

Using Equation 4, we can derive the following:

$\displaystyle t_{N}$ $\displaystyle=(1-s)[P_{c}+(1-P_{c})(1-P_{+})]$ (5)
$\displaystyle=P_{c}+1-P_{+}-P_{c}+P_{c}P_{+}-sP_{c}-s+sP_{+}+sP_{c}-sP_{c}P_{+}$
(6) $\displaystyle=1-P_{+}+P_{c}P_{+}-s+sP_{+}-sP_{c}P_{+}$ (7)
$\displaystyle=1-s-P_{+}(1-P_{c}-s+sP_{c})$ (8)
$\displaystyle=1-s-P_{+}(1-s)(1-P_{c})$ (9) $\displaystyle\implies P_{+}$
$\displaystyle=\frac{1-s-t_{N}}{(1-s)(1-P_{c})}$ (10)
$\displaystyle=\frac{1-\frac{t_{N}}{1-s}}{1-P_{c}}$ (11)

Substituting the value of $P_{+}$ in Equation 3, we get:

$\displaystyle t_{P}$ $\displaystyle=s[P_{c}+(1-P_{c})P_{+}]$ (12)
$\displaystyle=s\Bigg{[}P_{c}+(1-P_{c})\frac{1-\frac{t_{N}}{1-s}}{1-P_{c}}\Bigg{]}$
(13) $\displaystyle=s\bigg{[}P_{c}+1-\frac{t_{N}}{1-s}\bigg{]}$ (14)
$\displaystyle\implies\frac{t_{P}}{s}$
$\displaystyle=P_{c}+1-\frac{t_{N}}{1-s}$ (15) $\displaystyle\implies P_{c}$
$\displaystyle=\frac{t_{P}}{s}+\frac{t_{N}}{1-s}-1$ (16)

The values of $P_{c}$ and $P_{+}$ can be estimated from observed data using
the derived expressions. In this experiment, we include Qwen models (Bai et
al., 2023) of varying sizes, in our judge ensemble to increase the number of
data points for this study. The estimated probabilities using this method,
with human evaluation as the reference, are shown in Figure 17(a).

To validate these derived values, we observe the correlation between the
estimated values of $P_{c}$ and Cohen’s kappa ($\kappa$). As shown in Figure
17(b), we observe that the estimated values of $P_{c}$ are highly correlated
to the Cohen’s kappa values for the judge models, with a Pearson correlation
coefficient of $0.98$.

Judge model | $\kappa$ | $P_{c}$ | $P_{+}$
---|---|---|---
Qwen 1.8B | 0.19 | 0.18 | 1.00
Qwen 0.5B | 0.19 | 0.19 | 0.87
Gemma 7B | 0.40 | 0.32 | 0.07
Gemma 2B | 0.50 | 0.62 | 0.80
Qwen 4B | 0.69 | 0.67 | 0.89
Llama-2 7B | 0.66 | 0.68 | 0.36
Mistral 7B | 0.72 | 0.75 | 0.75
Llama-2 13B | 0.68 | 0.76 | 0.45
Llama-2 70B | 0.80 | 0.79 | 0.94
Qwen 7B | 0.80 | 0.79 | 0.77
Llama-3 8B | 0.81 | 0.80 | 0.84
Qwen 72B | 0.82 | 0.80 | 0.93
Qwen 14B | 0.83 | 0.82 | 0.83
Llama-3 70B | 0.84 | 0.84 | 0.79
GPT-4 Turbo | 0.85 | 0.85 | 0.66
(a)

(b)

Figure 17: a) Estimated values of $P_{c}$ and $P_{+}$ for different judge
models. b) Pearson’s correlation coefficient between $\kappa$ and $P_{c}$ for
judge models.