# Crowd Density Estimation using Imperfect Labels

Muhammad Asif Khan
Qatar Mobility Innovations Center (QMIC)
Qatar University
Doha, Qatar
<EMAIL_ADDRESS>
&Hamid Menouar
Qatar Mobility Innovations Center (QMIC)
Qatar University
Doha, Qatar
<EMAIL_ADDRESS>
&Ridha Hamila
Department of Electrical Engineering
Qatar University
Doha, Qatar
<EMAIL_ADDRESS>
Corresponding author

###### Abstract

Density estimation is one of the most widely used methods for crowd counting
in which a deep learning model learns from head-annotated crowd images to
estimate crowd density in unseen images. Typically, the learning performance
of the model is highly impacted by the accuracy of the annotations and
inaccurate annotations may lead to localization and counting errors during
prediction. A significant amount of works exist on crowd counting using
perfectly labeled datasets but none of these explore the impact of annotation
errors on the model accuracy. In this paper, we investigate the impact of
imperfect labels (both noisy and missing labels) on crowd counting accuracy.
We propose a system that automatically generates imperfect labels using a deep
learning model (called annotator) which are then used to train a new crowd
counting model (target model). Our analysis on two crowd counting models and
two benchmark datasets shows that the proposed scheme achieves accuracy closer
to that of the model trained with perfect labels showing the robustness of
crowd models to annotation errors.

##  1 Introduction

Crowd counting and density estimation is an important problem with many
interesting applications. For instance, crowd counts in public places such as
political rallies, stadiums, and exhibition centers can be of great interest
to authorities and event organizers for effective management and control
actions. Vehicle counting is significant in transport planning, road
intersection design, traffic signal control, and emergency vehicle preemption
strategies. Similarly, wildlife counting in forests can play a role to protect
species of interest.

The diverse applications of crowd density estimation and counting has
attracted the computer vision community. Over the past years, several state-
of-the-art deep learning models using convolution neural networks (CNNs) have
been proposed Khan et al. (2022); Tang et al. (2022); Wang and Breckon (2022);
Song et al. (2021); Chen et al. (2020); Jiang et al. (2019); Liu et al.
(2019); Li et al. (2018); Aich and Stavness (2018); Cao et al. (2018). These
models are trained with accurately labeled data (i.e., dot annotated images).
However, the acquisition of dot-annotated data can be expensive for dense
crowds and a large number of images. Moreover, the annotations may not be
available beforehand in some scenarios due to security and privacy reasons.

In this paper, we propose an intriguing approach in which the network first
generates density maps (imperfect and noisy) labels and then trains a target
model on the generated labeled data. The motivation for the proposed approach
is two-fold. First, dot annotations are expensive and thus the automatic
generation of ground truth density maps saves time. Second, the perfect labels
may not be available in some scenarios beforehand. For instance, deploying
several drones (each running a lightweight model) for the prediction can
simultaneously fine-tune the model weights over time using federated learning
Liu et al. (2022); Osifeko et al. (2021); Unal et al. (2021). However, the
drones will need labels to predict the count in the crowd images. Thus, one
can use the proposed method to use a secondary deep model to predict the
density map for the image which will serve as a ground truth to train the
primary (target) model. The deep model will only be used for inference and
when the node is participating in the federating learning round. Over time,
the lightweight target model will be fine-tuned and its accuracy will improve.

The contribution of the paper is as follows: We propose a novel method for
training a lightweight crowd counting model by automatic label generation
using a secondary deep crowd counting model (annotator). The auto-generated
labels are typically imperfect and noisy. Whether such noisy and imperfect
labels can be used to train another model? We trained two different models of
different sizes and architectures on two benchmark datasets to evaluate the
efficacy of the proposed method. The performance is investigated using
standard metrics and some intriguing results are achieved.

##  2 Related Work

Density estimation using CNN for crowd counting was first proposed in Zhang et
al. (2015). The authors propose a simple single-column architecture with six
layers. Several other works followed the approach and proposed different CNN
architectures of various sizes typically to improve accuracy over benchmark
datasets. The architectures used in these works include multi-column CNN Zhang
et al. (2016); Boominathan et al. (2016); Sam et al. (2017); Sindagi and Patel
(2017), modular CNNs Zeng et al. (2017); Cao et al. (2018); Wang and Breckon
(2022), encoder-decoder networks Jiang et al. (2019); Gao et al. (2019), and
transfer-learning based models Li et al. (2018); Liu et al. (2019); Aich and
Stavness (2018); Tang et al. (2022). The initial small-sized single-column
architectures (e.g. Zhang et al. (2015) generally achieve poor accuracy on
images having large scale variations. Scale variations typically arise from
the perspective distortions and different resolution images in the dataset.
Thus, multi-column networks with filters of varying receptive fields in
different columns can be used to capture the scale variations. A multi-column
convolution neural network (MCNN) is proposed in Zhang et al. (2016). MCNN is
a three-column architecture with variable sizes filters ($9\times 9,7\times
7,5\times 5,\text{and }3\times 3$) in each column. The switching-CNN Sam et
al. (2017) addresses the scale variations differently. It uses three CNN
networks (regressors) to process the input image using only one column based
on the density of the image. The density is automatically estimated by another
single-column CNN (classifier or switch). One shortcoming of multi-column
networks is their capability to adapt to the scale variations which is limited
by the number of columns i.e., more columns are needed when there are large
variations in the images across the dataset. An alternative solution is to use
modular networks which typically consist of a single-column (sometimes
multiple columns) architecture with scale-adaptive feature extraction modules.
These models are mainly inspired by the Inception-like models Szegedy et al.
(2015). Encoder-decoder models Jiang et al. (2019); Gao et al. (2019) are also
being used in many research works. These models follow the popular UNet
architecture Ronneberger et al. (2015), where the encoder part first learns
and extracts features from the input image, and then the decoder part
generates prediction using the features passed by the encoder. Encoder-decoder
models produce high-quality density maps. Crowd counting in dense images and
congested scenes can become more challenging. Thus, a significant amount of
research contributions adopt a transfer learning approach i.e., to use a pre-
trained image classification model e.g., VGG-16 Simonyan and Zisserman (2015),
ResNet He et al. (2016) or Inception Szegedy et al. (2015) as a front-end (or
backbone) for feature extraction and a shallow CNN network (back-end) to use
the features for estimating the crowd density. These models are generally more
accurate and faster to train but incur longer inference delays at prediction.

The aforementioned crowd counting models use accurately dot annotated
localization maps as ground truth for the crowd images to train the model. All
these works focus on investigating the accuracy improvement, but none of these
explore the model performance over imperfectly labeled data.

##  3 Proposed Scheme

Figure 1: Proposed method for automatic generation of ground truth and
training a lightweight crowd counting model using imperfect (noisy annotation)
dataset.

The proposed scheme is illustrated in Fig. 1. In the first step, the system
generates imperfect labels for the entire dataset using a deep network model
(Annotator). The predictions using the annotator network serve as imperfect
labels for training the target network. In the next step, the target network
is trained on the noisy (imperfect labels). During the training process, the
model learns from the imperfect labels by computing the loss function which is
$l_{2}$ distance between the imperfect labels and the prediction of the target
network. Then, to find the accuracy of the model, the mean absolute error
(MAE) or $l_{1}$ distance between the predictions of the target network and
the original ground truth (perfect) is calculated. In this way, the model
learns from the imperfect data but its accuracy is tested on the actual ground
truth.

We choose the CSRNet Li et al. (2018) model as the annotator network due to
its good accuracy. However, CSRNet generates density maps of size 1/8 of the
input image, whereas the target network used in our study generates density
maps of size 1/4 of the input image. To solve this issue, we modified the
original CSRNet Li et al. (2018) architecture. CSRNet uses the first 10 layers
of VGG-16 Simonyan and Zisserman (2015) network. In VGG-16, pooling layers are
used to reduce the size of the predicted density map to half after layer-2
(1/2), layer-4 (1/4), layer-7 (1/8), and layer-10 (1/16). Using the first 10
layers (without pooling layer after layer-10), CSRNet generates an output of
size 1/8. In our modified version, We took only the first seven (7) layers
(without the pooling layer after layer-7) to generate an output of size 1/4.
Furthermore, the back-end network of CSRNet uses layers of sizes (512, 512,
512, 256, 128, 64), respectively. We also modified the back-end network to
reduce layers sizes to (256, 256, 256, 128, 64, 64) to match the number of
channels in the output of the front-end and input of the back-end networks. We
denote the modified CSRNet architecture as CSRNet_lite (due to its relatively
smaller size). It is shown in Fig. 2.

Figure 2: CSRNet_lite: A modified lightweight version of CSRNet Li et al.
(2018).

The CSRNet_lite has $3.9$ Million parameters, almost four times less than
CSRNet ($16.2$ Million).

###  3.1 Model Training

The proposed scheme is evaluated over the DroneRGBT dataset Peng et al.
(2020). The DroneRGBT dataset has 1807 RGB and thermal image pairs in the
train set. Each image has a spatial resolution of 512×640 pixels. We split the
train set into a ratio of (70% : 30%) for training and validation. The dataset
covers several scenes (e.g., campus, streets, public parks, car parking,
stadiums, and plazas) and contains diverse crowd densities, illumination, and
scale variations. The dataset provides head annotations of people.

To train CSRNet_lite model to generate imperfect labels, the dot annotations
are first converted to ground truth (GT) density maps (perfect labels). To
generate the GT density map the head positions ($x_{i}$) expressed as delta
functions $\delta(x-x_{i})$ are convolved with Gaussian kernels
($G_{\sigma}$).

$D_{i}^{gt}=\sum_{i=1}^{N}{\delta(x-x_{i})*G_{\sigma}}$ (1)

where, N denotes the total number of dots (i.e., head positions) in the image.
The value of $\sigma=7$ is empirically determined by analyzing different
values and visually seeing the overlapping between the blobs in the GT density
maps. To prevent overfitting, we applied three different data augmentations
techniques including random brightness, random contrast, and horizontal
flipping. We use Adam optimizer Kingma and Ba (2015) with an initial learning
rate $0.0001$ to optimize the loss function. To compute the loss, we use the
pixel-wise Euclidean distance between the target and predicted density maps as
in Eq. 2.

$L(\Theta)=\frac{1}{N}\sum_{1}^{N}{||D(X_{i};\Theta)-D_{i}^{gt}||_{2}^{2}}.$
(2)

where $N$ is the number of heads in the image, $D(X_{i};\Theta)$ is the model
prediction density map and $D_{i}^{gt}$ is the GT density map. The predicted
density maps are then used to generate another dataset with all the images of
the original dataset whereas ground truth labels are replaced by the
predictions from CSRNet_lite. This noisy dataset is then used to train a
lightweight model. We use two lightweight models i.e., MCNN Zhang et al.
(2016), and LCDnet Khan et al. (2023) to train from scratch using the noisy
datasets. The same settings of optimizer and augmentation are used in the
second stage as well. We use a single machine with two Nvidia RTX-8000 GPUs
and PyTorch deep learning framework for training the model.

##  4 Evaluation and Results

###  4.1 Evaluation Metrics

We evaluate the proposed method using the most widely used metrics i.e., mean
absolute error (MAE), Grid Average Mean Error (GAME), Structural Similarity
Index (SSIM), and Peak Signal-to-Noise Ratio (PSNR) calculated in Eq. 3, 4, 5,
and 6, respectively. MAE and GAME measure the accuracy of the model by
measuring the counting errors in GT and predicted density maps. The GAME
metric is more sensitive to localization errors because it calculates MAE over
patches of the predictions. The SSIM and PSNR measure the quality of the
predicted density maps.

$MAE=\frac{1}{N}\sum_{1}^{N}{(e_{n}-\hat{g_{n}})}.$ (3)

where, $N$ is the total number of images in the dataset, $g_{n}$ is the ground
truth (actual count) and $\hat{e_{n}}$ is the prediction (estimated count) in
the $n^{th}$ image.

$GAME=\frac{1}{N}\sum_{n=1}^{N}{(\sum_{l=1}^{4^{L}}{|e_{n}^{l}-g_{n}^{l}|)}}.$
(4)

We set the value of $L=4$, thus each density map is divided into a grid size
of $4\times 4$ creating $16$ patches.

$SSIM(x,y)=\frac{(2\mu_{x}\mu_{y}+C_{1})(2\sigma_{x}\sigma_{y}C_{2})}{(\mu_{z}^{2}\mu_{y}^{2}+C_{1})(\mu_{z}^{2}\mu_{y}^{2}+C_{2})}.$
(5)

where $\mu_{x},\mu_{y},\sigma_{x},\sigma_{y}$ represent the means and standard
deviations of the actual and predicted density maps, respectively.

$PSNR=10log_{10}\left(\frac{Max(I^{2})}{MSE}\right).$ (6)

where $Max(I^{2})$ is the maximal in the image data (I). If it is an 8-bit
unsigned integer data type, the $Max(I^{2})=255$.

###  4.2 Results

The first step in the proposed method is the generation of imperfect labels.
Ideally, such imperfect labels should not have large errors. A sample label
generated using the CSRNet_lite model for the given image and the ground truth
density map is shown in Fig. 4.

Figure 3: Sample predictions on DroneRGBT dataset Peng et al. (2020): Column 1
shows same images from the dataset. Column 2 shows the ground truth (perfect).
Column 3 shows predictions of the model trained on the perfect ground truth.
Column 4 shows imperfect labels (generated using CSRNet_2 network. Column 5
shows predictions using the model trained on imperfect data. GT means ground
truth (actual count), whereas ET denotes estimated truth (predicted count).
Figure 4: The figure shows an image (left), its ground truth density map or
perfect label (middle) generated from the dot annotation, and a predicted
density map as the imperfect label (right) generated using the CSRNet_lite
model. The imperfect labels are used to train the lightweight model.

It can be observed that the noisy label is not perfect but is reasonably
accurate to aid in training the target model. The performance of the two
target models (i.e., MCNN and LCDnet) trained using the original target data
(perfect data) and on the noisy dataset is presented in Table 1 to show the
learning performance using imperfect data.

Table 1: Performance analysis of the proposed method using two lightweight models (MCNN and LCDnet) over two datasets (DroneRGBT, CARPK). Model | Dataset | Labels | MAE | GAME | SSIM | PSNR
---|---|---|---|---|---|---
MCNN | DroneRGBT | Perfect | 16.88 | 43.49 | 0.66 | 23.47
Imperfect | 17.86 | 46.92 | 0.64 | 23.07
MCNN | CARPK | Perfect | 10.10 | 42.40 | 0.76 | 19.21
Imperfect | 10.86 | 43.92 | 0.64 | 18.03
LCDnet | DroneRGBT | Perfect | 21.40 | 46.92 | 0.62 | 21.39
Imperfect | 23.82 | 47.91 | 0.60 | 20.07
LCDnet | CARPK | Perfect | 13.10 | 46.13 | 0.69 | 20.14
Imperfect | 13.75 | 48.26 | 0.66 | 20.07

One can observe that both networks (MCNN and LCDnet) when trained over
imperfect labels show very good accuracy on test (unseen) data. There is a
very small or sometimes negligible difference in the MAE and GAME metrics for
both models when trained on the two datasets using perfect and imperfect
datasets, respectively. Fig. 3 show sample prediction over DroneRGBT Peng et
al. (2020) dataset using the proposed scheme (showing both predictions over
perfect and noisy datasets used in training). On most images, the predicted
count was exactly the same for both predictions (using models trained over
perfect and noisy data). This means, that even imperfect data with missing or
noisy annotations (due to prediction errors) can be used by the model to learn
features that are even not learned by the annotator network.

Figure 5: A sample ground truth (GT) density map with missing (30%) data.

###  4.3 Ablation Study

Above we discussed the model learning performance over noisy or imperfect
training data (generated by another model). The analysis shows that the model
can learn features (objects) that are even not annotated in the training data
from other annotations and similar features. To further investigate the
learning process, we train and evaluate the two models using missing
annotations instead of imperfect data (noisy labels). To implement this, we
randomly delete a fixed portion (30%) of annotations from each image to
generate imperfect labels (see Fig. 5).

Table 2: Ablation study using imperfect data (missing labels). Model | Dataset | Annotation | MAE | GAME
---|---|---|---|---
MCNN | CARPK | Perfect | 10.1 | 43.4
Missing labels | 14.4 | 53.6
LCDnet | CARPK | Perfect | 13.7 | 48.2
Missing labels | 18.5 | 55.3

The two models MCNN and LCDnet are then trained over this dataset with missing
labels. The model performance is compared with the same model trained with
perfect data. As anticipated, the model could learn unlabelled objects of
interest. A comparison over CARPK Hsieh et al. (2017) dataset is shown in
Table 2.

The results in Tab;e 2 strengthen the idea that having a good model and
efficient training strategy can overcome the annotation errors (missing and
noisy annotations) in the training data.

##  5 Conclusion

The performance of any deep learning model is hugely correlated with the
quality of the data. More specifically, accurate labels improve the learning
process whereas noisy or imperfect labeled data can lead to inaccurate
predictions. In crowd density estimation, the ground truth labels contain dot
annotations of all heads (or object centers) in the image. Dot annotation is
expensive and hence we propose to automatically generate imperfect labels by
using a deep CNN model and then investigate the learning performance of a
shallow model on the imperfect data. The results report that even imperfect
data can be efficiently used to train another model from scratch. The proposed
scheme can be efficiently used in fine-tuning crowd counting models running
over distributed devices (such as drones) in a federated learning environment
without human-aided labeling. In future work, we plan to apply the proposed
method in other computer vision-based applications such as object detection
and anomaly detection.

## References

  * Aich and Stavness [2018] Shubhra Aich and Ian Stavness.  Global sum pooling: A generalization trick for object counting with small datasets of large images.  _arXiv preprint arXiv:1805.11123_ , 2018.
  * Boominathan et al. [2016] Lokesh Boominathan, Srinivas S. S. Kruthiventi, and R. Venkatesh Babu.  Crowdnet: A deep convolutional network for dense crowd counting.  _Proceedings of the 24th ACM international conference on Multimedia_ , 2016.
  * Cao et al. [2018] Xinkun Cao, Zhipeng Wang, Yanyun Zhao, and Fei Su.  Scale aggregation network for accurate and efficient crowd counting.  In _ECCV_ , 2018.
  * Chen et al. [2020] Zhuojun Chen, Junhao Cheng, Yuchen Yuan, Dongping Liao, Yizhou Li, and Jiancheng Lv.  Deep density-aware count regressor.  In _ECAI_ , 2020.
  * Gao et al. [2019] Chenyu Gao, Peng Wang, and Ye Gao.  Mobilecount: An efficient encoder-decoder framework for real-time crowd counting.  In _Pattern Recognition and Computer Vision: Second Chinese Conference, PRCV 2019, Xi’an, China, November 8–11, 2019, Proceedings, Part II_ , page 582–595. Springer-Verlag, 2019.
  * He et al. [2016] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 770–778, 2016.
  * Hsieh et al. [2017] Meng-Ru Hsieh, Yen-Liang Lin, and Winston H. Hsu.  Drone-based object counting by spatially regularized regional proposal network.  _2017 IEEE International Conference on Computer Vision (ICCV)_ , pages 4165–4173, 2017.
  * Jiang et al. [2019] Xiaolong Jiang, Zehao Xiao, Baochang Zhang, Xiantong Zhen, Xianbin Cao, David S. Doermann, and Ling Shao.  Crowd counting and density estimation by trellis encoder-decoder networks.  _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 6126–6135, 2019.
  * Khan et al. [2022] Muhammad Asif Khan, Hamid Menouar, and Ridha Hamila.  Revisiting crowd counting: State-of-the-art, trends, and future perspectives.  _Image Vis. Comput._ , 129:104597, 2022.
  * Khan et al. [2023] Muhammad Asif Khan, Hamid Menouar, and Ridha Hamila.  LCDnet: a lightweight crowd density estimation model for real-time video surveillance.  _Journal of Real-Time Image Processing_ , 20, 2023.
  * Kingma and Ba [2015] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  _CoRR_ , abs/1412.6980, 2015.
  * Li et al. [2018] Yuhong Li, Xiaofan Zhang, and Deming Chen.  Csrnet: Dilated convolutional neural networks for understanding the highly congested scenes.  _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , pages 1091–1100, 2018.
  * Liu et al. [2019] Weizhe Liu, Mathieu Salzmann, and Pascal V. Fua.  Context-aware crowd counting.  _2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 5094–5103, 2019.
  * Liu et al. [2022] Xiaolan Liu, Yansha Deng, and Toktam Mahmoodi.  A novel hybrid split and federated learning architecture in wireless uav networks.  In _ICC 2022 - IEEE International Conference on Communications_ , pages 1–6, 2022.  doi: 10.1109/ICC45855.2022.9838867.
  * Osifeko et al. [2021] Martins O. Osifeko, Gerhard P. Hancke, and Adnan M. Abu-Mahfouz.  Surveilnet: A lightweight anomaly detection system for cooperative iot surveillance networks.  _IEEE Sensors Journal_ , 21(22):25293–25306, 2021\.  doi: 10.1109/JSEN.2021.3103016.
  * Peng et al. [2020] Tao Peng, Qing Li, and Pengfei Zhu.  Rgb-t crowd counting from drone: A benchmark and mmccn network.  In _Computer Vision – ACCV 2020: 15th Asian Conference on Computer Vision, Kyoto, Japan, November 30 – December 4, 2020, Revised Selected Papers, Part VI_ , page 497–513, Berlin, Heidelberg, 2020. Springer-Verlag.
  * Ronneberger et al. [2015] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation.  _ArXiv_ , abs/1505.04597, 2015.
  * Sam et al. [2017] D. Sam, S. Surya, and R. Babu.  Switching convolutional neural network for crowd counting.  In _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 4031–4039, Los Alamitos, CA, USA, jul 2017. IEEE Computer Society.
  * Simonyan and Zisserman [2015] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In _3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings_ , 2015.  URL http://arxiv.org/abs/1409.1556.
  * Sindagi and Patel [2017] Vishwanath A. Sindagi and Vishal M. Patel.  Cnn-based cascaded multi-task learning of high-level prior and density estimation for crowd counting.  _2017 14th IEEE International Conference on Advanced Video and Signal Based Surveillance (AVSS)_ , pages 1–6, 2017.
  * Song et al. [2021] Qingyu Song, Changan Wang, Yabiao Wang, Ying Tai, Chengjie Wang, Jilin Li, Jian Wu, and Jiayi Ma.  To choose or to fuse? scale selection for crowd counting.  In _AAAI_ , 2021.
  * Szegedy et al. [2015] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott E. Reed, Dragomir Anguelov, D. Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 1–9, 2015.
  * Tang et al. [2022] Haihan Tang, Yi Wang, and Lap-Pui Chau.  Tafnet: A three-stream adaptive fusion network for rgb-t crowd counting.  _ArXiv_ , abs/2202.08517, 2022.
  * Unal et al. [2021] Devrim Unal, Mohammad Hammoudeh, Muhammad Asif Khan, Abdelrahman Abuarqoub, Gregory Epiphaniou, and Ridha Hamila.  Integration of federated machine learning and blockchain for the provision of secure big data analytics for internet of things.  _Computers & Security_, 109:102393, 2021.
  * Wang and Breckon [2022] Qian Wang and T. Breckon.  Crowd counting via segmentation guided attention networks and curriculum loss.  _IEEE Transactions on Intelligent Transportation Systems_ , 2022.
  * Zeng et al. [2017] Lingke Zeng, Xiangmin Xu, Bolun Cai, Suo Qiu, and Tong Zhang.  Multi-scale convolutional neural networks for crowd counting.  _2017 IEEE International Conference on Image Processing (ICIP)_ , pages 465–469, 2017.
  * Zhang et al. [2015] Cong Zhang, Hongsheng Li, Xiaogang Wang, and Xiaokang Yang.  Cross-scene crowd counting via deep convolutional neural networks.  _2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 833–841, 2015.
  * Zhang et al. [2016] Yingying Zhang, Desen Zhou, Siqin Chen, Shenghua Gao, and Yi Ma.  Single-image crowd counting via multi-column convolutional neural network.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pages 589–597, 2016.  doi: 10.1109/CVPR.2016.70.


tcb@breakable

# From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and
BenchBuilder Pipeline

Tianle Li∗ Wei-Lin Chiang∗ Evan Frick Lisa Dunlap Tianhao Wu
Banghua Zhu Joseph E. González Ion Stoica
UC Berkeley

###### Abstract

The rapid evolution of language models has necessitated the development of
more challenging benchmarks. Current static benchmarks often struggle to
consistently distinguish between the capabilities of different models and fail
to align with real-world user preferences. On the other hand, live crowd-
sourced platforms like the Chatbot Arena collect a wide range of natural
prompts and user feedback. However, these prompts vary in sophistication and
the feedback cannot be applied offline to new models. In order to ensure that
benchmarks keep up with the pace of LLM development, we address how one can
evaluate benchmarks on their ability to confidently separate models and their
alignment with human preference. Under these principles, we developed
BenchBuilder, a living benchmark that filters high-quality prompts from live
data sources to enable offline evaluation on fresh, challenging prompts.
BenchBuilder identifies seven indicators of a high-quality prompt, such as the
requirement for domain knowledge, and utilizes an LLM annotator to select a
high-quality subset of prompts from various topic clusters. The LLM evaluation
process employs an LLM judge to ensure a fully automated, high-quality, and
constantly updating benchmark. We apply BenchBuilder on prompts from the
Chatbot Arena to create Arena-Hard-Auto v0.1: 500 challenging user prompts
from a wide range of tasks. Arena-Hard-Auto v0.1 offers 3x tighter confidence
intervals than MT-Bench and achieves a state-of-the-art 89.1% agreement with
human preference rankings, all at a cost of only $25 and without human
labelers. The BenchBuilder pipeline enhances evaluation benchmarks and
provides a valuable tool for developers, enabling them to extract high-quality
benchmarks from extensive data with minimal effort.

††∗Equal contribution. Correspondence to: Tianle Li (tianleli@berkeley.edu),
Wei-Lin Chiang<EMAIL_ADDRESS>
##  1 Introduction

The proliferation of Large Language Models (LLMs) has spurred advancements,
leading to models trained on vast datasets with significantly enhanced
capabilities. However, this innovation also presents challenges in designing
effective evaluation benchmarks. Many existing benchmarks [45, 34, 4, 11, 3,
16] are unable to keep up with the rapidly advancing capabilities of these
models, becoming saturated and unable to differentiate between state-of-the-
art models. Additionally, many benchmarks are designed for automatic
evaluation with restrictive prompts, such as close-ended or multiple-choice
questions, which do not accurately reflect a model’s performance in real-world
interactions. Recent developments in benchmark evaluation [50, 38] have made
it feasible to evaluate more open-ended prompts, yet generating high-quality
prompts at scale remains challenging. Furthermore, static benchmarks are prone
to over-fitting and potential test leakage over time, reducing their
effectiveness in ranking models.

To address these issues, an effective benchmark must confidently differentiate
model capabilities, reflect real-world human interactions, and update
frequently to stay relevant and avoid overfitting. Live crowdsourced
evaluation platforms, like the Chatbot Arena [9], address some of these
concerns by collecting real-world user prompts with preference labels.
However, without quality control, these platforms can include low-quality
prompts that do not adequately differentiate top models.

In this work, we introduce BenchBuilder, a data pipeline designed to
automatically build high-quality benchmarks from live crowdsourced data source
such as the Chatbot Arena. BenchBuilder identifies diverse benchmark prompts
that can confidently separate model performance. It uses seven key indicators
of high-quality prompts, such as specificity and domain knowledge, and employs
an LLM to annotate prompts for these indicators. The resulting benchmark,
Arena-Hard-Auto v0.1, uses an LLM judge [51] to estimate human preference
against a baseline model, making BenchBuilder and Arena-Hard-Auto v0.1 a fully
automated system. We further explore how to measure the quality of benchmarks
by introducing metrics to assess a benchmark’s ability to confidently separate
models and its agreement with human preferences. Comparing Arena-Hard-Auto
v0.1 to current leading benchmarks such as AlpacaEval [38] and MT-Bench [51],
we demonstrate that Arena-Hard-Auto v0.1 offers significantly stronger
separability with tighter confidence intervals. Additionally, Arena-Hard-Auto
v0.1 achieves 89.1% agreement with Chatbot Arena rankings, making it an
excellent predictor of downstream performance. Arena-Hard-Auto v0.1 provides a
cost-effective and informative evaluation benchmark, and the BenchBuilder
pipeline serves as a valuable tool for developers seeking to automatically
generate high-quality benchmarks from vast data sources.

To summarize, our work makes the following contributions:

  1. 1.

We propose a novel data curation pipeline, BenchBuilder, to automatically
construct high-quality benchmarks from crowdsourced data.

  2. 2.

We propose metrics to capture desired properties in a benchmark, and validate
Arena-Hard-Auto v0.1 achieves significantly better separability and agreement
than existing benchmarks.

  3. 3.

We open-source both the Arena-Hard-Auto v0.1 benchmark and BenchBuilder
pipeline111Our code is available at: https://github.com/lm-sys/arena-hard-
auto.

##  2 Related works

LLM benchmarks. We briefly review widely used LLM benchmarks. Most existing
benchmarks are static and ground-truth-based (e.g., multi-choice question
answering). They cover a wide range of domains, including math, science,
coding, and reasoning. Common ones include MMLU [16], MATH [17], GSM-8K [10],
HumanEval [7], DROP [12], BigBench [37], HellaSwag [49], AGIEval [52], GPQA
[35], as well as comprehensive collection like HELM [26]. Many have considered
task-based evaluation such as IFEval [53], SWE-Bench [22], or AgentBench [28].
As LLMs become widely adopted in open-ended scenarios involving interaction
with humans (e.g., chatbot), many have considered human evaluation using
domain experts or crowd raters such as Amazon Mechanical Turk [23, 46] to
examine models’ response quality. As an alternative to human labeling,
previous work has shown that LLM-as-a-judge can be effective human preference
proxies (e.g., AlpacaFarm [14], MT-bench [51], AlpacaEval [25]).

Benchmark leakage. A fundamental limitation of static benchmarks is the
potential risk of test set leakage (i.e., contamination). Existing works [6,
36, 48, 39] have suggested a growing risk of contamination, which undermines
the reliability of benchmarks over time, motivating the need for benchmarks
that are more frequently updated.

Live benchmarks. DynaBench [24] identifies these challenges and recommends
creating living and continuously evolving benchmarks. Recent works
LiveCodeBench [19], R2E [20], as well as the community based live evaluation,
Chatbot Arena [9]. However, none of these focus on developing a pipeline for
automatic benchmark curation.

##  3 How do you measure benchmarks?

Given a benchmark consists of a dataset $D$ of prompts and scoring function
$f$ to measure the performance of model $\pi_{i}$ on prompt $\mathrm{p}$, we
outline two key properties that an LLM chatbot benchmark approximating human
preference should possess to provide a meaningful measurement of capabilities
between models:

  1. 1.

Separability: the benchmark should separate models with high confidence.

  2. 2.

Alignment with Human Preference: $f$ should agree with human preference.

While previous works have focused on alignment, separability is also a crucial
consideration when comparing models of similar quality (e.g., different
checkpoints from the same training run). However, achieving high-confidence
separability is challenging due to limitations in prompt design and inherent
variances in LLM evaluations. Overly simplistic prompts fail to distinguish
between models, while the randomness in human and LLM judgments leads to
inconsistent predictions. As a result, it is often difficult to confidently
determine if a model’s apparent performance reflects a genuine difference in
capability or merely noisy observations, highlighting a need for methods to
verify whether a benchmark can reliably separate similar models.

Statistical measures like Pearson [33] and Spearman Correlations, commonly
used in benchmarks such as AlpacaEval [25] to measure correlation to human
preference ranking, may fail to adequately address model separability and
ranking instability. In addition, these measures only provide a coarse signal
of ranking correlation without quantifying the magnitude of performance
differences between model pairs.

To address these issues, we develop three novel metrics: Separability with
Confidence, Agreement with Confidence, and Pair Rank Brier Score.

Separability with Confidence quantifies the benchmark’s confidence by
measuring its consistency in predicting the winner of a model pair across
random seeds through bootstrapping. This is done by calculating the percentage
of model pairs that have non-overlapping confidence intervals of their
benchmark scores. A higher percentage indicates that the benchmark is more
confident in distinguishing between the performance of different models, as
the confidence intervals of their scores do not overlap.

Agreement with Confidence Interval measures how well benchmarks A and B
confidently distinguish between two models with the same ordering. Given two
models $\pi_{1},\pi_{2}$ to evaluated on A and B, we assign the following
scores:

  1. 1.

If both A and B can confidently separate $\pi_{1},\pi_{2}$, we assign a score
of 1 if the rankings match across A and B, otherwise -1.

  2. 2.

If either A or B cannot separate $\pi_{1},\pi_{2}$ with confidence, we assign
a score of 0.

The final agreement score across models is the average score across all unique
model pairs. An agreement score of 1 implies that benchmark A confidently
agrees with benchmark B on the preference of every unique model pair.
Conversely, a score of -1 indicates that A confidently disagrees with B on
every unique model pair.

Pair Rank Brier Score further assesses an LLM benchmark’s capability to
predict the ranking of a pair of competing models by rewarding confidence in
correct predictions while penalizing confidence when incorrect. Consider two
models $\pi_{1}>\pi_{2}$ with disparate quality. Although two benchmarks A and
B predict the same ranking $\pi_{1}>\pi_{2}$, they predict
$P(\pi_{1}>\pi_{2})$ as $.60$ and $.90$, respectively (undetectable by
Spearman correlation). These benchmarks would result in very different Brier
scores, reflecting their ability to quantify the magnitude of performance
difference between the models. If both benchmarks give the wrong prediction of
the winner, we prefer the benchmark with a less confident prediction. Brier
score weighs a benchmark’s accuracy and its ability to quantify the
appropriate level of uncertainty in its predictions.

For each model pair, the benchmark’s forecast probability $f(\pi_{i}<\pi_{j})$
can be estimated as $P(f(\pi_{i})<f(\pi_{j}))$ with
$f(\pi_{i})\sim\mathcal{N}(\mu_{i},\sigma_{i}^{2})$ and
$f(\pi_{j})\sim\mathcal{N}(\mu_{j},\sigma_{j}^{2})$, where $\mu_{i}$ and
$\sigma^{2}_{i}$ are the bootstrapped mean and variance by evaluating multiple
times. The Brier Score Loss is then calculated over the benchmark’s prediction
for each model pair with respect to the ground truth probability, where Brier
Score is defined as
$\frac{1}{N}\sum_{\\{i,j\\}}(f(\pi_{j}<\pi_{i})-O(\pi_{j}<\pi_{i}))^{2}$,
where $N$ is the number of model pairs and ${O}$ is the ground truth outcome.

Separability with Confidence, Agreement with Confidence Interval, and Pair
Rank Brier Score each provide useful perspectives for evaluating the
effectiveness of LLM benchmarks. However, no single metric is intended to be
individually sufficient. Ultimately, these metrics are complementary and
should be considered together holistically. We claim that together, these
metrics offer a robust framework for assessing benchmark performance,
balancing the need for clear differentiation with alignment to human
preferences.

##  4 The BenchBuilder Pipeline and Arena-Hard-Auto v0.1 Dataset

###  4.1 BenchBuilder: A pipeline for converting messy data into high-quality
benchmarks

Figure 1: BenchBuilder Pipeline. Starting with a live data source of
crowdsourced user prompts, we first cluster their embeddings to form topic
clusters. An LLM annotator then assigns quality scores based on the required
skills. Clusters with low quality scores are filtered out, and we sample from
the remaining high-quality clusters to create a diverse and challenging
dataset of benchmark prompts.

BenchBuilder automatically extracts diverse, high quality prompts from a
dataset of 200,000 user queries collected via Chatbot Arena. This is done by
assigning a quality score to each prompt and by sampling prompts with a high
quality score evenly across a wide range of topics. An outline of our data
creation pipeline is shown in Figure 1.

Key Prompt Qualities • Specificity: Does the prompt ask for a specific, well-
defined output without leaving any ambiguity? • Domain Knowledge: Does the
prompt test the AI’s knowledge and understanding in a specific domain or set
of domains? • Complexity: Does the prompt have multiple components, variables,
or levels of depth and nuance?  • Problem-Solving: Does the prompt require
active problem-solving: analyzing and clearly defining the problem and
systematically devising and implementing a solution? • Creativity: Does the
prompt require a creative approach or solution? • Technical Accuracy: Does the
prompt require an answer with a high degree of technical accuracy, correctness
and precision? • Real-world Application: Does the prompt relate to real-world
applications?

To discover high quality prompts, we define seven qualities of a prompt which
signify the skills required to successfully answer, including specificity,
domain knowledge, complexity, and more shown in the figure above. These
qualities are used by an LLM annotator to assign a “quality score” to a given
prompt $\mathrm{p}$, which indicates how many of the above qualities are seen
in $\mathrm{p}$. Full instruction prompt is in Section 11.

To ensure that the filtered prompts represent a wide range of tasks, we adopt
a topic modeling pipeline in BERTopic by first converting each prompt with
OpenAI’s embedding model (text-embedding-3-small [32]), reducing dimension
with UMAP, and using a hierarchical-based clustering algorithm (HDBSCAN) to
identify clusters which are then summarized using GPT-4-Turbo [30].

As some topic clusters contain mostly trivial or ill defined prompt like “hi”,
we select the topic clusters with mean quality score >=6 out of 7 and sample
prompts equally across each of these clusters. The resulting dataset consists
of mostly well-defined, technical problem-solving queries as required in the
above key criteria. Dataset statistics and further description of the
filtering process used for Arena-Hard-Auto v0.1 along are in the following
section.

###  4.2 Arena-Hard-Auto v0.1: A set of challenging, high-quality prompts from
Chatbot Arena

We utilize the BenchBuilder pipeline to create Arena-Hard-Auto v0.1, a set of
500 high-quality prompts taken from 200,000 prompts from Chatbot Arena, an
open source platform for LLM evaluation. We first remove any duplicate
prompts, multi-turn conversations, or non-English conversations, and obtain
over 4,000 topic clusters and 75,000 queries covering a wide range of domains.

Then, we use the GPT-3.5-Turbo as judge to assign a “quality score” to each
prompt and remove prompts which either have a score < 5 or which belong to a
topic cluster with a mean score < 3\. We present the statistics in Table 1.
Notably, around 18% out of 75,000 queries are higher quality according to
GPT-3.5-Turbo. We then use GPT-4-Turbo as judge to assign a quality score and
ensuring the quality of the remaining prompts, resulting with over 500 high
quality clusters. We randomly sample 500 prompts from these clusters, creating
Arena-Hard-Auto v0.1. Additionally, we confirm our final dataset does not
contain any personally identifiable information or offensive content.

In Figure 3, we show examples of topic clusters ranging from low to high mean
scores provided by GPT-3.5-Turbo. We can observe clusters with higher scores
often correlate to challenging topics or tasks for LLMs like game development
or mathematical proofs. On the other hand, clusters with lower scores point to
trivial or ambiguous questions like "Flirty Texting Strategies". We present
several example prompts and their respective topic cluster below.

[htb]  Cluster 1: Greetings and Well-Being Inquiry (Mean Score: 2.7) ⬇ Yo,
what up my brother (Qualities: None) Cluster 2: US Presidents Query (Mean
Score: 3.2) ⬇ Who was the president of the US in 1975 (Qualities: Specificity,
Domain-Knowledge, Technical Accuracy, Real-World) Cluster 3: Physics Problem
Solving (Mean Score: 5.0) ⬇ A 50,000 kg airplane initially flying at a speed
of 60.0 m/s accelerates at 5.0 m/s2 for 600 meters. What is its velocity after
this acceleration? What is the net force that caused this acceleration?
(Qualities: Specificity, Domain-Knowledge, Complexity, Problem-Solving,
Technical Accuracy, Real-World) Cluster 4: OpenCV Image Processing Technique
(Mean Score: 5.5) ⬇ you are given a task to detect number of faces in each
frame of any video using pytorch and display the number in the final edited
video. (Qualities: All)

To see whether “quality score” assigned during BenchBuilder’s pipeline
correlates with separability and agreement, we sample 50 prompts per score and
compare the responses from GPT-4 and Llama-2-70b-Chat [43], with GPT-4-Turbo
as judge. In Figure 4 (Left), we observe a strong correlation between high
potential score and the win-rate of GPT-4-Turbo over Llama-2-70b-Chat. A
similar trend is also observed in other model pairs such as Claude Sonnet vs
Haiku and Mistral-Large [40] vs Mixtral [21].

Since qualities assigned by GPT-4-Turbo are critical to ensure the final
prompt selection is high-quality, we verified its accuracy as annotator. We
construct “ground truth” labels for 200 randomly sampled queries by collecting
majority votes from GPT-4o [31], Claude-3-Opus [2], and Gemini-1.5-Pro [39].
GPT-4-Turbo has 85.6% accuracy against “ground truth” labels, validating its
ability as annotator.

Figure 2: Mean score of various topic clusters in descending order. Higher-
scoring clusters correlate to challenging topics, while lower scores indicate
trivial or ambiguous questions. A more complete topic cluster plot is in
Figure 6.

Figure 3: Comparison between MT-bench and Arena-Hard-Auto v0.1. The latter
offers significantly better separability between models and tighter confidence
intervals. GPT-4-0314 has no variance in Arena-Hard-Auto v0.1 because it is
the anchor model.

Figure 4: Win-rate of three model pairs (GPT-4-0613 vs Llama-2-70b-chat,
Claude-3-Sonnet-20240229 vs Claude-3-Haiku-20240307, and Mistral-Large vs
Mixtral-8x7b-Instruct-v0.1) over “quality score”. We randomly sample 50
queries for each quality score 0-7 and bootstrap a win-rate and confidence
interval between model pairs on each score interval of 2. We observe a similar
trend of win-rate between model pairs becomes increasingly separable as the
quality score increases.

##  5 Evaluation with LLM-as-a-Judge

Evaluating models on challenging queries such as Arena-Hard-Auto v0.1 is a
non-trivial task. Most queries involve deep domain knowledge and problem
solving skills, requiring expert-level judgment to evaluate the answer
quality. Unfortunately, it is prohibitively expensive and time consuming to
use human judges. Following LLM-as-a-Judge [51] and AlpacaFarm [14], we employ
LLM-as-a-judge framework to approximate human preference.

To obtain a score for model $\pi_{i}$ on prompt $\mathrm{p}$, we consider the
pairwise comparison setup against a strong baseline model $\pi_{base}$
(GPT-4-0314) and employ a strong judge model (e.g., GPT-4-Turbo or
Gemini-1.5-Pro) to measure preference. We obtain a score for each output
$\pi_{i}(\mathrm{p})$ by asking the judge to rank its preference of
$\pi_{i}(\mathrm{p})$ or $\pi_{base}(\mathrm{p})$ on a Likert scale [27] ($1=$
prefers $\pi_{i}(\mathrm{p})$ much less than $\pi_{base}(\mathrm{p})$, $5=$
prefers $\pi_{i}(\mathrm{p})$ much more than $\pi_{base}(\mathrm{p})$). This
way, a model will be penalized more in big losses than small losses, which we
find to be effective in separating models. We also employ chain-of-thought
(CoT) [47] to prompt the LLM judge to generate its own answer first before
giving judgments. The full judge prompt can be found in Section 11.

To avoid potential position bias, we adopt a two-game setup – per query we
swap the models on the first and second position. This results in 1000
judgments per model evaluation. Following Chatbot Arena, we adopt the Bradley-
Terry [5] model to produce model’s the final model scores. We aggregate all
pairwise comparisons to the baseline model for all models and bootstrapping
the comparisons to retrieve a bootstrapped confidence interval of all models’
win-rate against the baseline, producing a ordered ranking of all models by
their win-rates.

##  6 Experimental Results

Quality Score | 1+ | 2+ | 3+ | 4+ | 5+ | 6+ | 7+
---|---|---|---|---|---|---|---
% of queries | 95.4 | 83.5 | 61.9 | 48.7 | 33.8 | 17.9 | 0.2
Qualities | Specificity | Domain-knowledge | Complexity | Problem-solving | Creativity | Tech. Accuracy | Real-world
% of queries | 57.3 | 63.4 | 35.0 | 34.9 | 26.1 | 39.0 | 87.9

Table 1: First row is the percentage of queries with quality scores of the
column or more in 75K Chatbot Arena data assigned by GPT-3.5-Turbo. Second row
is the percentage of queries in 75K Chatbot Arena labeled by GPT-3.5-Turbo
with each of the 7 qualities.

| Chatbot Arena | MT Bench | AlpacaEval 2.0 LC | Arena-Hard-Auto v0.1
---|---|---|---|---
Confiderence Agreement | N/A | 26.1% | 81.2% | 89.1%
Separability | 85.8% | 22.6% | 83.2% | 87.4%
Spearman Correlation | N/A | 91.3% | 90.8% | 94.1%
Brier Score | N/A | 0.09 | 0.11 | 0.069
Real-world | Yes | Mixed | Mixed | Yes
Freshness | Live | Static | Static | Frequent Updates
Eval cost per model | very high | $10 | $10 | $25
Prompts per model | 10000+ | 160 | 800 | 1000

Table 2: We use a set of top-20 models1 on Chatbot Arena (2024/04/13) that are
also present on the AlpacaEval leaderboard to calculate separability and
agreement per benchmark. We consider the human preference ranking by Chatbot
Arena (English only) as the reference to calculate agreement.

Model | GPT4-T | Claude3-Opus | Gemini1.5-Pro | Llama3-70B | Ensemble-as-Judges
---|---|---|---|---|---
Confiderence Agreement | 89.1% | 66.7% | 84.8% | 65.6% | 91.5%
Separability | 87.4% | 83.68% | 82.11% | 81.6% | 89.5%
Spearman Correlation | 94.1% | 77.0% | 95.2% | 70.5% | 96.5%
Brier Score | 0.069 | 0.170 | 0.064 | 0.196 | 0.065

Table 3: Statistics of Arena-Hard-Auto v0.1 with four LLM different judges:
GPT4-T (gpt-4-1106-preview), Claude-3-Opus, Gemini1.5-Pro
(gemini-1.5-pro-0514), Llama3-70B (llama-3-70b-instruct). We compare rankings
produced by these judges against Chatbot Arena (English) ranking (as of
2024/04/13). We observe GPT-4T and Gemini1.5-Pro have higher agreement than
Claude-3-Opus and Llama-3-70B. Furthermore, the ensemble of GPT4-T and
Gemini1.5-Pro shows even higher agreement.

Model | Arena-Hard-Auto v0.1 | Random Sample 1 | Random Sample 2
---|---|---|---
Confiderence Agreement | 84.2% | 57.5% | 66.1%
Separability | 80.5% | 74.7% | 76.3%
Spearman Correlation | 94.7% | 64.7% | 72.5%
Brier Score | 0.069 | 0.215 | 0.162

Table 4: We compare Arena-Hard-Auto v0.1 with two sets of 500 prompts randomly sampled from 75K Chatbot Arena user queries. We evaluate the set of top-20 models and compare various statistics across. Each prompt is judged only once by positioning the baseline answer first. | Arena-Hard-Auto v0.1
---|---
Confiderence Agreement | 92.0%
Spearman Correlation | 96.4%
Brier Score | 0.055
Table 5: We compare Arena-Hard-Auto v0.1 (gpt-4-1106-preview as judge) to
Chatbot Arena Category Hard Prompt (English) on the same set of top-20
models*. By comparing Arena-Hard-Auto v0.1 to a similar distribution of
queries from Chatbot Arena, we obtain even higher alignment to human
preferences.

###  6.1 Setup and Baselines

To compare Arena-Hard-Auto v0.1’s separability and alignment with humans
against other widely used benchmarks, MT-Bench [51] and AlpacaEval 2.0 Length
Controlled [13], we obtain 95% confidence interval of model performances via
applying 100 rounds of bootstrapping on judgment results for each benchmark.
For AlpacaEval, we use pre-existing results from their repository. We obtain
MT-Bench judgment with no modification to their recommended evaluation setup.
For Arena-Hard-Auto v0.1, we employ the system proposed in section 5 by
choosing gpt-4-0314 as baseline model for pairwise comparison.

In order to compare these benchmarks fairly, we use a set of top-20
models††footnotemark:  on Chatbot Arena [9] (2024/04/13) that are also
presented on AlpacaEval leaderboard (2024/04/13) as ground truth for human
preferences on the model ranking orders. In subsection 6.3, we individually
compare Arena-Hard-Auto v0.1 to Chatbot Arena within a more similar
distribution of user queries. We compute separability using 95% confidence
intervals and consider the human preference ranking by Chatbot Arena as the
reference to calculate agreement, Spearman correlation, and Pair Rank Brier
Score across AlpacaEval, MT-Bench, and Arena-Hard-Auto v0.1.

###  6.2 Separability and Agreement Across Benchmarks

In Table 2, Arena-Hard-Auto v0.1 shows the highest separability (87.4%)
against widely adopted LLM benchmarks and offers highest agreement (89.1%) to
Chatbot Arena at $25 cost. In Figure 3, we show Arena-Hard-Auto v0.1 offers
significantly stronger separability against MT Bench with tighter confidence
intervals. With only 500 prompts, Arena-Hard-Auto v0.1 achieve impressive
alignment to and even higher separability than Chatbot Arena Ranking, which
constitutes for approximately 1 million real-world human preferences.

Notably, we observe a significant gap between MT-bench’s Spearman Correlation
(91.3%) and confidence agreement (22.6%) to Chatbot Arena, an example where
Spearman Correlation fails to account for variance of the rankings, and hence
cannot adequately measure important ranking granularity of top LLMs. We
present a visual comparison between Arena-Hard-Auto v0.1 and MT-Bench in
Figure 3, highlighting Arena-Hard-Auto v0.1’s improved separability.

###  6.3 Comparing to a similar distribution of human preference

We set Chatbot Arena Category Hard Prompts (English) as ground truth
reference. Chatbot Arena Category Hard Prompt is a human preference ranking
leaderboard based on votes from a harder subset of 200,000+ prompts extracted
from over 1 million Chatbot Arena queries using a similar method as
BenchBuilder. In Table 5, we show Arena-Hard-Auto v0.1 achieving an overall
higher alignment (92.0% Confidence Agreement) to human preferences.

###  6.4 Comparing to Random Subsets of Arena

To further verify BenchBuilder pipeline’s ability to select high-quality
prompts, we compare Arena-Hard-Auto v0.1 to two sets of 500 randomly selected
prompts from the 75,000 user queries prior to pipeline extraction. We compare
their ranking preferences of the set of top-20 models produced by
GPT-4-1106-Preview as judge. In Table 4, we observe Arena-Hard-Auto v0.1 is
significantly better than both random baselines, verifying BenchBuilder’s
effectiveness at selecting superior prompts for a LLM benchmark.

###  6.5 Ensemble-as-Judges

We study four popular LLMs222GPT-4-1106-Preview, Claude-3-Opus-20240229,
Gemini-1.5-Pro-0514, and Llama-3-70b-instruct as judge on Arena-Hard-Auto
v0.1. We statistically compare their ranking preference of the set of top-20
models111gpt-4-turbo-2024-04-09, claude-3-opus-20240229,
claude-3-sonnet-20240229, gpt-4-0314 [29], gpt-4-0613, mistral-large-2402,
qwen1.5-72b-chat [41], mistral-medium, claude-2.0, gpt-3.5-turbo-0613,
claude-2.1, gemini-pro [15], mixtral-8x7b-instruct-v0.1, gpt-3.5-turbo-0314,
yi-34b-chat [1], tulu-2-dpo-70b [18], dbrx-instruct-preview [42], vicuna-33b
[8], starling-lm-7b-alpha [54], llama-2-70b-chat to the Chatbot Arena
Leaderboard in Table 3. Out of the four judges, GPT-4-1106-Preview has the
highest separability and agreement to human ranking preference. We also
observe GPT-4-1106-Preview and Gemini-1.5-Pro have much higher agreement than
Claude-3-Opus and Llama-3-70b-instruct, with more than 20% improvement.

We also experiment combining GPT-4-1106-Preview and Gemini-1.5-Pro as an
ensemble judge by combining their judgment results. This achieves an even
higher separability and agreement to human ranking preference, with agreement
reaching 91.5%, demonstrating effectiveness of judging LLMs with Ensemble-as-
Judges.

##  7 Limitations

Although our data source comes from diverse distributions, we acknowledge
possible biases within our pipeline. For example, the 7 qualities we defined
might not be all-encompassing and could potentially bias towards a limited
distribution of prompts, such as technical fields. Further, prompt selection
process could be biased by LLM annotators. Additionally, for model evaluation,
we are employing LLMs as judges, who are known to suffer from verbosity bias
[51, 13]. In Table 7, we present how verbosity can effect LLM as Judge on
Arena-Hard-Auto v0.1. We also acknowledge potential self-bias in LLM judges
(e.g., GPT-4 Judge prefers GPT-4 answers) in Table 3. Arena-Hard-Auto v0.1
also does not evaluate multi-turn or non-English interactions.

In the future, we plan to further the exploration of a more generalized and
diverse set of prompt qualities, establishing a pipeline to systematically
produce benchmark across a more comprehensive and inclusive distribution of
prompts while maintaining high separability and alignment to humans.
Additionally, we believe our approach of utilizing LLM-as-a-judge can be
improved, such as developing a system of Ensemble-as-Judges, similar to Judges
with Jury [44], to reduce biases and improve correlation.

##  8 Conclusions

We introduced BenchBuilder, a data curation pipeline that transforms
crowdsourced data into high-quality benchmarks by seven key qualities. This
pipeline enables building challenging and evolving benchmarks which is crucial
for evaluating today’s advanced language models. Our evaluation metrics,
including separability and agreement with confidence, provide a comprehensive
assessment of benchmarks. We show the resulting benchmark, Arena-Hard-Auto
v0.1, significantly improves separability and alignment with human preferences
over existing benchmarks, achieving 89.1% agreement with Chatbot Arena
rankings at only $25 per evaluation. We expect Arena-Hard-Auto v0.1 to be
useful for LLM developers to evaluate their models with confidence and
BenchBuilder to be a valuable tool for developers seeking to extract high-
quality benchmark from vast amounts of data with minimal human effort.

##  9 Acknowledgment

We thank Matei Zaharia, Yann Dubois, Anastasios Angelopoulos, Lianmin Zheng,
Lewis Tunstall, Nathan Lambert, Xuechen Li, Naman Jain, Ying Sheng, Maarten
Grootendorst for their valuable feedback. We thank Siyuan Zhuang and Dacheng
Li for the valuable review and debug of the code. We thank Microsoft AFMR for
Azure OpenAI credits support. We also thank Together.ai & Anyscale for open
model endpoint support.

## References

  * [1] 01\. AI, :, Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, Kaidong Yu, Peng Liu, Qiang Liu, Shawn Yue, Senbin Yang, Shiming Yang, Tao Yu, Wen Xie, Wenhao Huang, Xiaohui Hu, Xiaoyi Ren, Xinyao Niu, Pengcheng Nie, Yuchi Xu, Yudong Liu, Yue Wang, Yuxuan Cai, Zhenyu Gu, Zhiyuan Liu, and Zonghong Dai.  Yi: Open foundation models by 01.ai, 2024.
  * [2] Anthropic.  The claude 3 model family: Opus, sonnet, haiku.  https://www-cdn.anthropic.com/de8ba9b01c9ab7cbabf5c33b80b7bbc618857627/Model_Card_Claude_3.pdf, 2024.  (Accessed on 06/05/2024).
  * [3] Johan Bos and Katja Markert.  Recognising textual entailment with logical inference.  In Raymond Mooney, Chris Brew, Lee-Feng Chien, and Katrin Kirchhoff, editors, Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, pages 628–635, Vancouver, British Columbia, Canada, October 2005. Association for Computational Linguistics.
  * [4] Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning.  A large annotated corpus for learning natural language inference.  In Lluís Màrquez, Chris Callison-Burch, and Jian Su, editors, Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632–642, Lisbon, Portugal, September 2015. Association for Computational Linguistics.
  * [5] Ralph Allan Bradley and Milton E Terry.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  Biometrika, 39(3/4):324–345, 1952.
  * [6] Nicholas Carlini, Florian Tramer, Eric Wallace, Matthew Jagielski, Ariel Herbert-Voss, Katherine Lee, Adam Roberts, Tom Brown, Dawn Song, Ulfar Erlingsson, et al.  Extracting training data from large language models.  In 30th USENIX Security Symposium (USENIX Security 21), pages 2633–2650, 2021.
  * [7] Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Kaplan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al.  Evaluating large language models trained on code.  arXiv preprint arXiv:2107.03374, 2021.
  * [8] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
  * [9] Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica.  Chatbot arena: An open platform for evaluating llms by human preference, 2024.
  * [10] Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al.  Training verifiers to solve math word problems.  arXiv preprint arXiv:2110.14168, 2021.
  * [11] Bill Dolan and Chris Brockett.  Automatically constructing a corpus of sentential paraphrases.  In Third international workshop on paraphrasing (IWP2005), 2005.
  * [12] Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner.  Drop: A reading comprehension benchmark requiring discrete reasoning over paragraphs.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368–2378, 2019.
  * [13] Yann Dubois, Balázs Galambosi, Percy Liang, and Tatsunori B Hashimoto.  Length-controlled alpacaeval: A simple way to debias automatic evaluators.  arXiv preprint arXiv:2404.04475, 2024.
  * [14] Yann Dubois, Xuechen Li, Rohan Taori, Tianyi Zhang, Ishaan Gulrajani, Jimmy Ba, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Alpacafarm: A simulation framework for methods that learn from human feedback, 2023.
  * [15] Team Gemini, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.
  * [16] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  * [17] Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.  Measuring mathematical problem solving with the math dataset.  NeurIPS, 2021.
  * [18] Hamish Ivison, Yizhong Wang, Valentina Pyatkin, Nathan Lambert, Matthew Peters, Pradeep Dasigi, Joel Jang, David Wadden, Noah A. Smith, Iz Beltagy, and Hannaneh Hajishirzi.  Camels in a changing climate: Enhancing lm adaptation with tulu 2, 2023.
  * [19] Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar-Lezama, Koushik Sen, and Ion Stoica.  Livecodebench: Holistic and contamination free evaluation of large language models for code.  arXiv preprint arXiv:2403.07974, 2024.
  * [20] Naman Jain, Manish Shetty, Tianjun Zhang, King Han, Koushik Sen, and Ion Stoica.  R2e: Turning any github repository into a programming agent environment.  In ICML, 2024.
  * [21] Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed.  Mixtral of experts, 2024.
  * [22] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan.  Swe-bench: Can language models resolve real-world github issues?  In The Twelfth International Conference on Learning Representations, 2024.
  * [23] Marzena Karpinska, Nader Akoury, and Mohit Iyyer.  The perils of using Mechanical Turk to evaluate open-ended text generation.  In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih, editors, Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 1265–1285, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics.
  * [24] Douwe Kiela, Max Bartolo, Yixin Nie, Divyansh Kaushik, Atticus Geiger, Zhengxuan Wu, Bertie Vidgen, Grusha Prasad, Amanpreet Singh, Pratik Ringshia, Zhiyi Ma, Tristan Thrush, Sebastian Riedel, Zeerak Waseem, Pontus Stenetorp, Robin Jia, Mohit Bansal, Christopher Potts, and Adina Williams.  Dynabench: Rethinking benchmarking in nlp.  NAACL, 2021.
  * [25] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Alpacaeval: An automatic evaluator of instruction-following models.  https://github.com/tatsu-lab/alpaca_eval, 2023.
  * [26] Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al.  Holistic evaluation of language models.  arXiv preprint arXiv:2211.09110, 2022.
  * [27] Rensis Likert.  A technique for the measurement of attitudes.  Archives of psychology, 1932.
  * [28] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Aohan Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang.  Agentbench: Evaluating llms as agents, 2023.
  * [29] OpenAI.  Gpt-4 technical report.  arXiv preprint arXiv:2303.08774, 2023.
  * [30] OpenAI.  New models and developer products announced at devday.  https://openai.com/blog/new-models-and-developer-products-announced-at-devday, 2023.  (Accessed on 06/05/2024).
  * [31] OpenAI.  Hello gpt-4o.  https://openai.com/index/hello-gpt-4o/, 2024.  (Accessed on 06/05/2024).
  * [32] OpenAI.  New embedding models and api updates.  https://openai.com/index/new-embedding-models-and-api-updates/, 2024.  (Accessed on 06/05/2024).
  * [33] Karl Pearson.  Note on regression and inheritance in the case of two parents.  Proceedings of the Royal Society of London, 58:240–242, 1895.
  * [34] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.  Squad: 100,000+ questions for machine comprehension of text.  EMNLP, 2016.
  * [35] David Rein, Betty Li Hou, Asa Cooper Stickland, Jackson Petty, Richard Yuanzhe Pang, Julien Dirani, Julian Michael, and Samuel R. Bowman.  Gpqa: A graduate-level google-proof q&a benchmark, 2023.
  * [36] Oscar Sainz, Jon Campos, Iker García-Ferrero, Julen Etxaniz, Oier Lopez de Lacalle, and Eneko Agirre.  NLP evaluation in trouble: On the need to measure LLM data contamination for each benchmark.  In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 10776–10787, Singapore, December 2023. Association for Computational Linguistics.
  * [37] Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  Transactions on Machine Learning Research, 2023.
  * [38] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * [39] Gemini Team, Machel Reid, Nikolay Savinov, Denis Teplyashin, Dmitry, Lepikhin, Timothy Lillicrap, Jean baptiste Alayrac, Radu Soricut, Angeliki Lazaridou, Orhan Firat, Julian Schrittwieser, Ioannis Antonoglou, Rohan Anil, Sebastian Borgeaud, Andrew Dai, Katie Millican, Ethan Dyer, Mia Glaese, Thibault Sottiaux, Benjamin Lee, Fabio Viola, Malcolm Reynolds, Yuanzhong Xu, James Molloy, Jilin Chen, Michael Isard, Paul Barham, Tom Hennigan, Ross McIlroy, Melvin Johnson, Johan Schalkwyk, Eli Collins, Eliza Rutherford, Erica Moreira, Kareem Ayoub, Megha Goel, Clemens Meyer, Gregory Thornton, Zhen Yang, Henryk Michalewski, Zaheer Abbas, Nathan Schucher, Ankesh Anand, Richard Ives, James Keeling, Karel Lenc, Salem Haykal, Siamak Shakeri, Pranav Shyam, Aakanksha Chowdhery, Roman Ring, Stephen Spencer, Eren Sezener, Luke Vilnis, Oscar Chang, Nobuyuki Morioka, George Tucker, Ce Zheng, Oliver Woodman, Nithya Attaluri, Tomas Kocisky, Evgenii Eltyshev, Xi Chen, Timothy Chung, Vittorio Selo, Siddhartha Brahma, Petko Georgiev, Ambrose Slone, Zhenkai Zhu, James Lottes, Siyuan Qiao, Ben Caine, Sebastian Riedel, Alex Tomala, Martin Chadwick, Juliette Love, Peter Choy, Sid Mittal, Neil Houlsby, Yunhao Tang, Matthew Lamm, Libin Bai, Qiao Zhang, Luheng He, Yong Cheng, Peter Humphreys, Yujia Li, Sergey Brin, Albin Cassirer, Yingjie Miao, Lukas Zilka, Taylor Tobin, Kelvin Xu, Lev Proleev, Daniel Sohn, Alberto Magni, Lisa Anne Hendricks, Isabel Gao, Santiago Ontanon, Oskar Bunyan, Nathan Byrd, Abhanshu Sharma, Biao Zhang, Mario Pinto, Rishika Sinha, Harsh Mehta, Dawei Jia, Sergi Caelles, Albert Webson, Alex Morris, Becca Roelofs, Yifan Ding, Robin Strudel, Xuehan Xiong, Marvin Ritter, Mostafa Dehghani, Rahma Chaabouni, Abhijit Karmarkar, Guangda Lai, Fabian Mentzer, Bibo Xu, YaGuang Li, Yujing Zhang, Tom Le Paine, Alex Goldin, Behnam Neyshabur, Kate Baumli, Anselm Levskaya, Michael Laskin, Wenhao Jia, Jack W. Rae, Kefan Xiao, Antoine He, Skye Giordano, Lakshman Yagati, Jean-Baptiste Lespiau, Paul Natsev, Sanjay Ganapathy, Fangyu Liu, Danilo Martins, Nanxin Chen, Yunhan Xu, Megan Barnes, Rhys May, Arpi Vezer, Junhyuk Oh, Ken Franko, Sophie Bridgers, Ruizhe Zhao, Boxi Wu, Basil Mustafa, Sean Sechrist, Emilio Parisotto, Thanumalayan Sankaranarayana Pillai, Chris Larkin, Chenjie Gu, Christina Sorokin, Maxim Krikun, Alexey Guseynov, Jessica Landon, Romina Datta, Alexander Pritzel, Phoebe Thacker, Fan Yang, Kevin Hui, Anja Hauth, Chih-Kuan Yeh, David Barker, Justin Mao-Jones, Sophia Austin, Hannah Sheahan, Parker Schuh, James Svensson, Rohan Jain, Vinay Ramasesh, Anton Briukhov, Da-Woon Chung, Tamara von Glehn, Christina Butterfield, Priya Jhakra, Matthew Wiethoff, Justin Frye, Jordan Grimstad, Beer Changpinyo, Charline Le Lan, Anna Bortsova, Yonghui Wu, Paul Voigtlaender, Tara Sainath, Shane Gu, Charlotte Smith, Will Hawkins, Kris Cao, James Besley, Srivatsan Srinivasan, Mark Omernick, Colin Gaffney, Gabriela Surita, Ryan Burnell, Bogdan Damoc, Junwhan Ahn, Andrew Brock, Mantas Pajarskas, Anastasia Petrushkina, Seb Noury, Lorenzo Blanco, Kevin Swersky, Arun Ahuja, Thi Avrahami, Vedant Misra, Raoul de Liedekerke, Mariko Iinuma, Alex Polozov, Sarah York, George van den Driessche, Paul Michel, Justin Chiu, Rory Blevins, Zach Gleicher, Adrià Recasens, Alban Rrustemi, Elena Gribovskaya, Aurko Roy, Wiktor Gworek, Sébastien M. R. Arnold, Lisa Lee, James Lee-Thorp, Marcello Maggioni, Enrique Piqueras, Kartikeya Badola, Sharad Vikram, Lucas Gonzalez, Anirudh Baddepudi, Evan Senter, Jacob Devlin, James Qin, Michael Azzam, Maja Trebacz, Martin Polacek, Kashyap Krishnakumar, Shuo yiin Chang, Matthew Tung, Ivo Penchev, Rishabh Joshi, Kate Olszewska, Carrie Muir, Mateo Wirth, Ale Jakse Hartman, Josh Newlan, Sheleem Kashem, Vijay Bolina, Elahe Dabir, Joost van Amersfoort, Zafarali Ahmed, James Cobon-Kerr, Aishwarya Kamath, Arnar Mar Hrafnkelsson, Le Hou, Ian Mackinnon, Alexandre Frechette, Eric Noland, Xiance Si, Emanuel Taropa, Dong Li, Phil Crone, Anmol Gulati, Sébastien Cevey, Jonas Adler, Ada Ma, David Silver, Simon Tokumine, Richard Powell, Stephan Lee, Kiran Vodrahalli, Samer Hassan, Diana Mincu, Antoine Yang, Nir Levine, Jenny Brennan, Mingqiu Wang, Sarah Hodkinson, Jeffrey Zhao, Josh Lipschultz, Aedan Pope, Michael B. Chang, Cheng Li, Laurent El Shafey, Michela Paganini, Sholto Douglas, Bernd Bohnet, Fabio Pardo, Seth Odoom, Mihaela Rosca, Cicero Nogueira dos Santos, Kedar Soparkar, Arthur Guez, Tom Hudson, Steven Hansen, Chulayuth Asawaroengchai, Ravi Addanki, Tianhe Yu, Wojciech Stokowiec, Mina Khan, Justin Gilmer, Jaehoon Lee, Carrie Grimes Bostock, Keran Rong, Jonathan Caton, Pedram Pejman, Filip Pavetic, Geoff Brown, Vivek Sharma, Mario Lučić, Rajkumar Samuel, Josip Djolonga, Amol Mandhane, Lars Lowe Sjösund, Elena Buchatskaya, Elspeth White, Natalie Clay, Jiepu Jiang, Hyeontaek Lim, Ross Hemsley, Zeyncep Cankara, Jane Labanowski, Nicola De Cao, David Steiner, Sayed Hadi Hashemi, Jacob Austin, Anita Gergely, Tim Blyth, Joe Stanton, Kaushik Shivakumar, Aditya Siddhant, Anders Andreassen, Carlos Araya, Nikhil Sethi, Rakesh Shivanna, Steven Hand, Ankur Bapna, Ali Khodaei, Antoine Miech, Garrett Tanzer, Andy Swing, Shantanu Thakoor, Lora Aroyo, Zhufeng Pan, Zachary Nado, Jakub Sygnowski, Stephanie Winkler, Dian Yu, Mohammad Saleh, Loren Maggiore, Yamini Bansal, Xavier Garcia, Mehran Kazemi, Piyush Patil, Ishita Dasgupta, Iain Barr, Minh Giang, Thais Kagohara, Ivo Danihelka, Amit Marathe, Vladimir Feinberg, Mohamed Elhawaty, Nimesh Ghelani, Dan Horgan, Helen Miller, Lexi Walker, Richard Tanburn, Mukarram Tariq, Disha Shrivastava, Fei Xia, Qingze Wang, Chung-Cheng Chiu, Zoe Ashwood, Khuslen Baatarsukh, Sina Samangooei, Raphaël Lopez Kaufman, Fred Alcober, Axel Stjerngren, Paul Komarek, Katerina Tsihlas, Anudhyan Boral, Ramona Comanescu, Jeremy Chen, Ruibo Liu, Chris Welty, Dawn Bloxwich, Charlie Chen, Yanhua Sun, Fangxiaoyu Feng, Matthew Mauger, Xerxes Dotiwalla, Vincent Hellendoorn, Michael Sharman, Ivy Zheng, Krishna Haridasan, Gabe Barth-Maron, Craig Swanson, Dominika Rogozińska, Alek Andreev, Paul Kishan Rubenstein, Ruoxin Sang, Dan Hurt, Gamaleldin Elsayed, Renshen Wang, Dave Lacey, Anastasija Ilić, Yao Zhao, Adam Iwanicki, Alejandro Lince, Alexander Chen, Christina Lyu, Carl Lebsack, Jordan Griffith, Meenu Gaba, Paramjit Sandhu, Phil Chen, Anna Koop, Ravi Rajwar, Soheil Hassas Yeganeh, Solomon Chang, Rui Zhu, Soroush Radpour, Elnaz Davoodi, Ving Ian Lei, Yang Xu, Daniel Toyama, Constant Segal, Martin Wicke, Hanzhao Lin, Anna Bulanova, Adrià Puigdomènech Badia, Nemanja Rakićević, Pablo Sprechmann, Angelos Filos, Shaobo Hou, Víctor Campos, Nora Kassner, Devendra Sachan, Meire Fortunato, Chimezie Iwuanyanwu, Vitaly Nikolaev, Balaji Lakshminarayanan, Sadegh Jazayeri, Mani Varadarajan, Chetan Tekur, Doug Fritz, Misha Khalman, David Reitter, Kingshuk Dasgupta, Shourya Sarcar, Tina Ornduff, Javier Snaider, Fantine Huot, Johnson Jia, Rupert Kemp, Nejc Trdin, Anitha Vijayakumar, Lucy Kim, Christof Angermueller, Li Lao, Tianqi Liu, Haibin Zhang, David Engel, Somer Greene, Anaïs White, Jessica Austin, Lilly Taylor, Shereen Ashraf, Dangyi Liu, Maria Georgaki, Irene Cai, Yana Kulizhskaya, Sonam Goenka, Brennan Saeta, Ying Xu, Christian Frank, Dario de Cesare, Brona Robenek, Harry Richardson, Mahmoud Alnahlawi, Christopher Yew, Priya Ponnapalli, Marco Tagliasacchi, Alex Korchemniy, Yelin Kim, Dinghua Li, Bill Rosgen, Kyle Levin, Jeremy Wiesner, Praseem Banzal, Praveen Srinivasan, Hongkun Yu, Çağlar Ünlü, David Reid, Zora Tung, Daniel Finchelstein, Ravin Kumar, Andre Elisseeff, Jin Huang, Ming Zhang, Ricardo Aguilar, Mai Giménez, Jiawei Xia, Olivier Dousse, Willi Gierke, Damion Yates, Komal Jalan, Lu Li, Eri Latorre-Chimoto, Duc Dung Nguyen, Ken Durden, Praveen Kallakuri, Yaxin Liu, Matthew Johnson, Tomy Tsai, Alice Talbert, Jasmine Liu, Alexander Neitz, Chen Elkind, Marco Selvi, Mimi Jasarevic, Livio Baldini Soares, Albert Cui, Pidong Wang, Alek Wenjiao Wang, Xinyu Ye, Krystal Kallarackal, Lucia Loher, Hoi Lam, Josef Broder, Dan Holtmann-Rice, Nina Martin, Bramandia Ramadhana, Mrinal Shukla, Sujoy Basu, Abhi Mohan, Nick Fernando, Noah Fiedel, Kim Paterson, Hui Li, Ankush Garg, Jane Park, DongHyun Choi, Diane Wu, Sankalp Singh, Zhishuai Zhang, Amir Globerson, Lily Yu, John Carpenter, Félix de Chaumont Quitry, Carey Radebaugh, Chu-Cheng Lin, Alex Tudor, Prakash Shroff, Drew Garmon, Dayou Du, Neera Vats, Han Lu, Shariq Iqbal, Alex Yakubovich, Nilesh Tripuraneni, James Manyika, Haroon Qureshi, Nan Hua, Christel Ngani, Maria Abi Raad, Hannah Forbes, Jeff Stanway, Mukund Sundararajan, Victor Ungureanu, Colton Bishop, Yunjie Li, Balaji Venkatraman, Bo Li, Chloe Thornton, Salvatore Scellato, Nishesh Gupta, Yicheng Wang, Ian Tenney, Xihui Wu, Ashish Shenoy, Gabriel Carvajal, Diana Gage Wright, Ben Bariach, Zhuyun Xiao, Peter Hawkins, Sid Dalmia, Clement Farabet, Pedro Valenzuela, Quan Yuan, Ananth Agarwal, Mia Chen, Wooyeol Kim, Brice Hulse, Nandita Dukkipati, Adam Paszke, Andrew Bolt, Kiam Choo, Jennifer Beattie, Jennifer Prendki, Harsha Vashisht, Rebeca Santamaria-Fernandez, Luis C. Cobo, Jarek Wilkiewicz, David Madras, Ali Elqursh, Grant Uy, Kevin Ramirez, Matt Harvey, Tyler Liechty, Heiga Zen, Jeff Seibert, Clara Huiyi Hu, Andrey Khorlin, Maigo Le, Asaf Aharoni, Megan Li, Lily Wang, Sandeep Kumar, Norman Casagrande, Jay Hoover, Dalia El Badawy, David Soergel, Denis Vnukov, Matt Miecnikowski, Jiri Simsa, Praveen Kumar, Thibault Sellam, Daniel Vlasic, Samira Daruki, Nir Shabat, John Zhang, Guolong Su, Jiageng Zhang, Jeremiah Liu, Yi Sun, Evan Palmer, Alireza Ghaffarkhah, Xi Xiong, Victor Cotruta, Michael Fink, Lucas Dixon, Ashwin Sreevatsa, Adrian Goedeckemeyer, Alek Dimitriev, Mohsen Jafari, Remi Crocker, Nicholas FitzGerald, Aviral Kumar, Sanjay Ghemawat, Ivan Philips, Frederick Liu, Yannie Liang, Rachel Sterneck, Alena Repina, Marcus Wu, Laura Knight, Marin Georgiev, Hyo Lee, Harry Askham, Abhishek Chakladar, Annie Louis, Carl Crous, Hardie Cate, Dessie Petrova, Michael Quinn, Denese Owusu-Afriyie, Achintya Singhal, Nan Wei, Solomon Kim, Damien Vincent, Milad Nasr, Christopher A. Choquette-Choo, Reiko Tojo, Shawn Lu, Diego de Las Casas, Yuchung Cheng, Tolga Bolukbasi, Katherine Lee, Saaber Fatehi, Rajagopal Ananthanarayanan, Miteyan Patel, Charbel Kaed, Jing Li, Shreyas Rammohan Belle, Zhe Chen, Jaclyn Konzelmann, Siim Põder, Roopal Garg, Vinod Koverkathu, Adam Brown, Chris Dyer, Rosanne Liu, Azade Nova, Jun Xu, Alanna Walton, Alicia Parrish, Mark Epstein, Sara McCarthy, Slav Petrov, Demis Hassabis, Koray Kavukcuoglu, Jeffrey Dean, and Oriol Vinyals.  Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context, 2024.
  * [40] Mistral AI team.  Au large.  https://mistral.ai/news/mistral-large/, 2024.  (Accessed on 06/05/2024).
  * [41] Qwen Team.  Introducing qwen1.5, February 2024.
  * [42] The Mosaic Research Team.  Introducing dbrx: A new state-of-the-art open llm.  https://www.databricks.com/blog/introducing-dbrx-new-state-art-open-llm/, 2024.  (Accessed on 06/05/2024).
  * [43] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [44] Pat Verga, Sebastian Hofstatter, Sophia Althammer, Yixuan Su, Aleksandra Piktus, Arkady Arkhangorodsky, Minjie Xu, Naomi White, and Patrick Lewis.  Replacing judges with juries: Evaluating llm generations with a panel of diverse models.  arXiv preprint arXiv:2404.18796, 2024.
  * [45] Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman.  GLUE: A multi-task benchmark and analysis platform for natural language understanding.  2019\.  In the Proceedings of ICLR.
  * [46] Yizhong Wang, Yeganeh Kordi, Swaroop Mishra, Alisa Liu, Noah A. Smith, Daniel Khashabi, and Hannaneh Hajishirzi.  Self-instruct: Aligning language models with self-generated instructions.  In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki, editors, Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13484–13508, Toronto, Canada, July 2023. Association for Computational Linguistics.
  * [47] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou.  Chain-of-thought prompting elicits reasoning in large language models, 2023.
  * [48] Shuo Yang, Wei-Lin Chiang, Lianmin Zheng, Joseph E. Gonzalez, and Ion Stoica.  Rethinking benchmark and contamination for language models with rephrased samples, 2023.
  * [49] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  Hellaswag: Can a machine really finish your sentence?  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4791–4800, 2019.
  * [50] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.  Judging LLM-as-a-judge with MT-bench and chatbot arena.  In Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
  * [51] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.  Judging llm-as-a-judge with mt-bench and chatbot arena.  NeurIPS, 2023.
  * [52] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan.  Agieval: A human-centric benchmark for evaluating foundation models.  arXiv preprint arXiv:2304.06364, 2023.
  * [53] Jeffrey Zhou, Tianjian Lu, Swaroop Mishra, Siddhartha Brahma, Sujoy Basu, Yi Luan, Denny Zhou, and Le Hou.  Instruction-following evaluation for large language models.  arXiv preprint arXiv:2311.07911, 2023.
  * [54] Banghua Zhu, Evan Frick, Tianhao Wu, Hanlin Zhu, and Jiantao Jiao.  Starling-7b: Improving llm helpfulness & harmlessness with rlaif, November 2023.

##  10 Appendix

Judge | Model Name | Win Rate | Average Token #
---|---|---|---
gpt-4-turbo-2024-04-09* | 82.6 | -1.8/+1.6 | 662
gpt-4-0125-preview* | 78.0 | -2.2/+2.4 | 619
claude-3-opus-20240229 | 60.4 | -3.3/+2.4 | 541
gpt-4-0314 | 50.0 | -0.0/+0.0 | 423
claude-3-sonnet-20240229 | 46.8 | -2.1/+2.2 | 552
claude-3-haiku-20240307 | 41.5 | -2.8/+2.5 | 505
llama-3-70b-instruct | 41.1 | -2.5/+2.4 | 583
gpt-4-0613 | 37.9 | -2.2/+2.0 | 354
mistral-large-2402 | 37.7 | -1.9/+2.6 | 400
mixtral-8x22b-instruct-v0.1 | 36.4 | -2.7/+2.9 | 430
Qwen1.5-72B-Chat | 36.1 | -2.5/+2.2 | 474
command-r-plus | 33.1 | -2.1/+2.2 | 541
mistral-medium | 31.9 | -2.3/+2.4 | 485
mistral-next | 27.4 | -2.1/+1.7 | 297
gpt-3.5-turbo-0613 | 24.8 | -1.6/+2.0 | 401
claude-2.0 | 24.0 | -2.5/+2.5 | 295
dbrx-instruct | 23.9 | -1.4/+1.5 | 415
Mixtral-8x7B-Instruct-v0.1 | 23.4 | -2.3/+1.7 | 457
gpt-3.5-turbo-0125 | 23.3 | -2.2/+2.3 | 329
Yi-34B-Chat | 23.1 | -1.8/+2.0 | 611
Starling-LM-7B-beta | 23.0 | -1.9/+2.2 | 530
claude-2.1 | 22.8 | -1.6/+2.1 | 290
Snorkel-Mistral-PairRM-DPO | 20.7 | -2.2/+1.5 | 564
llama-3-8b-instruct | 20.6 | -2.5/+1.8 | 585
gpt-3.5-turbo-1106 | 18.9 | -1.6/+2.1 | 285
gpt-3.5-turbo-0301 | 18.1 | -1.7/+1.2 | 334
gemini-1.0-pro | 17.8 | -1.7/+1.7 | 322
command-r | 17.0 | -1.9/+1.7 | 432
tulu-2-dpo-70b | 15.0 | -1.4/+1.2 | 550
Starling-LM-7B-alpha | 12.8 | -1.4/+1.4 | 483
mistral-7b-instruct-v0.2 | 12.6 | -1.6/+1.3 | 541
Llama-2-70b-chat-hf | 11.6 | -1.6/+1.4 | 595
vicuna-33b-v1.3 | 8.6 | -1.3/+1.0 | 451
gemma-7b-it | 7.5 | -1.1/+1.2 | 378
Llama-2-7b-chat-hf | 4.6 | -0.8/+0.8 | 561
gemma-2b-it | 3.0 | -0.6/+0.7 | 369
Table 6: Arena-Hard-Auto v0.1 Leaderboard (baseline: GPT-4-0314). *GPT-4-Turbo’s high score can be due to the GPT-4 judge favoring GPT-4 outputs. Judge | Model Name | Win Rate | Average Token #
---|---|---|---
GPT-4-1106-Preview | gpt-3.5-turbo-0125-detailed | 29.86 | 421
| gpt-3.5-turbo-0125-chatty | 23.89 | 361
| gpt-3.5-turbo-0125 | 23.2 | 328
Claude-3-Opus | gpt-3.5-turbo-0125-detailed | 40.78 | 421
| gpt-3.5-turbo-0125-chatty | 28.49 | 375
| gpt-3.5-turbo-0125 | 27.97 | 328
Table 7: Length Bias Comparison Between GPT and Claude as Judge. Prompt for detailed:“You are a helpful assistant who thoroughly explains things with as much detail as possible.”, prompt for chatty: ‘You are a helpful assistant who is chatty.”, and default prompt: "You are a helpful assistant." Model Name | Win Rate | Average Token #
---|---|---
gpt-3.5-turbo-0125-1 | 23.05 | 328
gpt-3.5-turbo-0125-2 | 22.93 | 328
gpt-3.5-turbo-0125-3 | 22.75 | 328
Table 8: Variances between 3 separate runs of Arena-Hard-Auto v0.1 on
gpt-3.5-turbo’s performance using GPT-4-1106-Preview as judge.

(a) Chatbot Arena (20K Votes)

(b) AlpacaEval 2.0 LC

(c) Arena-Hard-Auto v0.1

(d) MT Bench

Figure 5: Above is the predicted model predicted probability against the
bootstrapped arena “ground truth” probability (jittered to show clusters).
While both Alpaca eval and Arena-Hard-Auto v0.1 have large clusters around
(0,0) and (1,1) signifying good forecasting, Arena-Hard-Auto v0.1 has lighter
clusters on (0,1) and (1,0), if any, revealing less overconfidence. MT Bench
has heavy tails along the top and bottom, revealing underconfidence. However,
none of these benchmarks show an “ideal” y=x curve (with dense ends) expected
with a perfectly calibrated forecast, signifying room for future research.
Figure 6: A more complete selection of mean scores of various topic clusters
in descending order.

##  11 Prompts

Prompt Quality Systems Instruction: Your task is to evaluate how well the
following input prompts can assess the capabilities of advanced AI assistants.
For the input prompt, please analyze it based on the following 7 criteria. For
each criteria, make sure to explain before determine whether the input satisfy
it. 1\. Specificity: Does the prompt ask for a specific, well-defined output
without leaving any ambiguity? This allows the AI to demonstrate its ability
to follow instructions and generate a precise, targeted response. 2\. Domain
Knowledge: Does the prompt test the AI’s knowledge and understanding in a
specific domain or set of domains? The prompt must demand the AI to have a
strong prior knowledge or mastery of domain-specific concepts, theories, or
principles. 3\. Complexity: Does the prompt have multiple components,
variables, or levels of depth and nuance? This assesses the AI’s capability to
handle complex, multi-faceted problems beyond simple queries. 4\. Problem-
Solving: Does the prompt require active problem-solving: analyzing and clearly
defining the problem and systematically devising and implementing a solution?
Note active problem-solving is not simply reciting facts or following a fixed
set of instructions. 5\. Creativity: Does the prompt require a creative
approach or solution? This tests the AI’s ability to generate novel ideas
tailored to the specific needs of the request or problem at hand. 6\.
Technical Accuracy: Does the prompt require an answer with a high degree of
technical accuracy, correctness and precision? This assesses the reliability
and truthfulness of the AI’s outputs. 7\. Real-World Application: Does the
prompt relate to real-world applications? This tests the AI’s ability to
provide practical and actionable information that could be implemented in
real-life scenarios. After analyzing the input prompt based on these criteria,
you must list the criteria numbers that the prompt satisfies in the format of
a Python array. For example, "Criteria Satisfied: [1, 2, 4, 6, 7]".

LLM-Judge System Instruction Please act as an impartial judge and evaluate the
quality of the responses provided by two AI assistants to the user prompt
displayed below. You will be given assistant A’s answer and assistant B’s
answer. Your job is to evaluate which assistant’s answer is better. Begin your
evaluation by generating your own answer to the prompt. You must provide your
answers before judging any answers. When evaluating the assistants’ answers,
compare both assistants’ answers with your answer. You must identify and
correct any mistakes or inaccurate information. Then consider if the
assistant’s answers are helpful, relevant, and concise. Helpful means the
answer correctly responds to the prompt or follows the instructions. Note when
user prompt has any ambiguity or more than one interpretation, it is more
helpful and appropriate to ask for clarifications or more information from the
user than providing an answer based on assumptions. Relevant means all parts
of the response closely connect or are appropriate to what is being asked.
Concise means the response is clear and not verbose or excessive. Then
consider the creativity and novelty of the assistant’s answers when needed.
Finally, identify any missing important information in the assistants’ answers
that would be beneficial to include when responding to the user prompt. After
providing your explanation, you must output only one of the following choices
as your final verdict with a label: 1. Assistant A is significantly better:
[[A>>B]] 2. Assistant A is slightly better: [[A>B]] 3. Tie, relatively the
same: [[A=B]] 4. Assistant B is slightly better: [[B>A]] 5. Assistant B is
significantly better: [[B>>A] Example output: "My final verdict is tie:
[[A=B]]".

##  12 Additional Examples

[htb]  User: Use ABC notation to write a melody in the style of a folk tune.
GPT-4:

    X:1 T:Untitled Folk Tune M:4/4 L:1/8
    K:G|:G2A2|B2A2|G2E2|D4|E2F2|G2F2|E2C2|B,4|A2B2|c2B2|A2F2|E4|D2E2|F2E2|D2B,2|C4:|

GPT-3.5-Turbo: Sure! Here’s a simple melody in the style of a folk tune
written in ABC notation:

    X:1 T:Folk Tune M:4/4 L:1/4 K:C
    CDE|FGA|BAG|FED| CDE|FGA|BAG|c2z| CDE|FGA|BAG|FED| CDE|FGA|BAG|c2z||

This melody is in the key of C major and consists of four measures. Feel free
to modify it or add more sections to create your own folk tune! Judgement: My
own answer to the prompt would be:

        X:1 T:My Folk Melody M:3/4 L:1/8
        K:D A2|:d2f2a2|b2a2f2|e2d2B2|A4F2| G2A2B2|A2G2F2|E2D2F2|1D6:|2D4A2|
        |:d2f2a2|b2a2f2|e2c2A2|B4A2| G2A2B2|A2G2F2|E2D2F2|1D6:|2D4||

This melody is in the key of D major, has a 3/4 time signature, and is
structured in a typical AAB form common in folk music. It uses a range of an
octave, which is typical for folk tunes that need to be easily singable. Now,
let’s evaluate the responses from Assistant A and Assistant B. Assistant A’s
Answer: Assistant A provides a melody in ABC notation that is coherent and
follows the structure of a folk tune. The melody is in 4/4 time, which is
common for folk music, and it is in the key of G major. The melody has a two-
part form (indicated by the repeat signs |: and :|), which is typical in folk
music. The range of the melody is appropriate for a folk tune, and the rhythm
is simple and repetitive, which is characteristic of the genre. Assistant B’s
Answer: Assistant B also provides a melody in ABC notation that is coherent
and could be considered in the style of a folk tune. The melody is in 4/4 time
and the key of C major. The melody consists of four measures repeated twice,
which is a simple structure that could be found in folk music. However,
Assistant B’s melody is less traditional in its form compared to Assistant
A’s, as it does not have a distinct A and B section, which is more common in
folk tunes. Additionally, the use of “z” for a half-measure rest is less
typical in folk tunes, which usually have a more continuous melodic line.
Comparison: Both assistants provided melodies that could be considered folk
tunes. Assistant A’s melody has a more traditional structure with a clear A
and B section, while Assistant B’s melody is simpler and does not follow the
traditional AAB or AB form. Assistant A’s melody is also more detailed in
terms of the ABC notation, providing a full eight-measure section that is
repeated, which gives a more complete sense of the tune. Assistant B’s melody
is correct in its notation but is more basic and less representative of a
traditional folk tune structure. In terms of helpfulness, relevance, and
conciseness, both assistants provided relevant and concise answers. However,
Assistant A’s answer is slightly more helpful due to the more traditional
structure of the provided melody. Creativity and novelty are not particularly
applicable in this context, as the task is to create a melody in a traditional
style rather than to innovate. Both assistants showed an understanding of the
folk tune style, but Assistant A’s answer aligns more closely with traditional
folk structures. There are no significant mistakes or inaccurate information
in either assistant’s answers. However, Assistant B’s use of the rest “z” in a
folk tune is less conventional. My final verdict is: Assistant A is slightly
better: [[A>B]].

[htb]  User: Can you come up with a 12 bar chord progression in C that works in the lydian mode? GPT-4: Sure! In the Lydian mode, the chords are built on the scale degrees of the Lydian scale. Here’s a 12-bar chord progression in C Lydian: | Cmaj7 | Dmaj7 | Em7 | F#m7b5 | | Cmaj7 | Dmaj7 | Em7 | F#m7b5 | | Cmaj7 | Dmaj7 | Em7 | Cmaj7 | This progression uses the I, II, iii, and iv° chords of the C Lydian mode. GPT-3.5-Turbo: Certainly! Here’s a 12-bar chord progression in C that works in the Lydian mode: Cmaj7 | Cmaj7 | Cmaj7 | Cmaj7 | Fmaj7 | Fmaj7 | Cmaj7 | Cmaj7 | G7 | Fmaj7 | Cmaj7 | G7 | In this progression, the Cmaj7 chords establish the tonality in C Lydian, while the Fmaj7 and G7 chords add some tension and movement. Feel free to experiment with different voicings and rhythms to make it your own! Judgement: Assistant B is significantly better: [[B>>A]].