# RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented
Question Answering

Rujun Han2† Yuhao Zhang3† Peng Qi4† Yumo Xu1 Jenyuan Wang1
Lan Liu1 William Yang Wang 5† Bonan Min1 Vittorio Castelli1

1AWS AI Labs 2Google 3Samaya.ai 4Orby.ai 5University of California, Santa
Barbara
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS>

###### Abstract

Question answering based on retrieval augmented generation (RAG-QA) is an
important research topic in NLP and has a wide range of real-world
applications. However, most existing datasets for this task are either
constructed using a single source corpus or consist of short extractive
answers, which fall short of evaluating large language model (LLM) based RAG-
QA systems on cross-domain generalization. To address these limitations, we
create Long-form RobustQA (Lfrqa), a new dataset comprising human-written
long-form answers that integrate short extractive answers from multiple
documents into a single, coherent narrative, covering 26K queries and large
corpora across seven different domains. We further propose Rag-qa Arena by
directly comparing model-generated answers against Lfrqa’s answers using LLMs
as evaluators. We show via extensive experiments that Rag-qa Arena and human
judgments on answer quality are highly correlated. Moreover, only 41.3% of the
most competitive LLM’s answers are preferred to Lfrqa’s answers, demonstrating
Rag-qa Arena as a challenging evaluation platform for future research.111Code:
https://github.com/awslabs/rag-qa-arena

RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented
Question Answering

Rujun Han2† Yuhao Zhang3† Peng Qi4† Yumo Xu1 Jenyuan Wang1 Lan Liu1 William
Yang Wang 5† Bonan Min1 Vittorio Castelli1 1AWS AI Labs 2Google 3Samaya.ai
4Orby.ai 5University of California, Santa Barbara<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

††footnotetext: Work done at AWS AI Labs.

##  1 Introduction

Figure 1: Lfrqa annotation example. There are three documents (some text
removed for brevity) relevant to the query. We instruct annotators to combine
Robustqa’s answers into a coherent long-form answer with added text if
necessary. Citations [1], [2] and [3] indicate the supporting documents of
each sentence.

Dataset | Answers grounded | Long-form | Multiple | Coherent | Multiple | Human | # Test
---|---|---|---|---|---|---|---
Name | in corpus | answers | documents | answers | domains | annotated | queries
Lfrqa | ✓ | ✓ | ✓ | ✓ | ✓ | ✓ | 16.1K
RobustqaHan et al. (2023) | ✓ | ✗ | ✓ | ✗ | ✓ | ✓ | 16.1K
NQ Kwiatkowski et al. (2019) | ✓ | ✗ | ✗ | ✗ | ✗ | ✓ |  03.6K
MultiHop-RAG Tang and Yang (2024) | ✓ | ✗ | ✓ | ✓ | ✓ | ✓ |  02.5K
ASQA Stelmakh et al. (2022) | ✓ | ✓ | ✓ | ✓ | ✗ | ✓ |  01.0K
LongFact Wei et al. (2024) | ✗ | ✓ | ✗ | ✓ | ✓ | ✗ |  02.3K
ELI5 Fan et al. (2019) | ✗ | ✓ | ✓ | ✓ | ✗ | ✓ | 25.0K

Table 1: Comparison of datasets. Lfrqa distinguishes from previous work by
uniquely encompassing seven features: 1) RAG-QA dataset with answers annotated
based on underlying corpus; 2) Long-form answers of paragraph length; 3)
Multiple documents that provide different facts/views; 4) Coherent answers
that handle conflicting information; 5) Multiple-domain corpus to benchmark
domain robustness; 6) Human annotated high-quality answers; 7) Large-scale
evaluation set.

Traditional reading comprehension task is constrained to fixed contexts
Rajpurkar et al. (2016); Kočiský et al. (2018); Huang et al. (2019). It is
inadequate at addressing real-world questions, with no context provided for a
system to find answers. Such open-ended questions require a system to identify
answers in an enormous knowledge base (e.g., Wikipedia) that is
computationally prohibitive to feed into LLMs. Retrieval-augmented generative
question answering (RAG-QA) becomes an effective tool to filter out massive
amounts of noise and select only a few highly relevant passages for LLM-based
QA models.

The wide applications of RAG-QA Gao et al. (2023) necessitate the evaluation
of systems’ out-of-domain (OOD) performances because a real-world system often
confronts new data unseen during training. Existing popular benchmark datasets
such as Natural Questions (NQ) Kwiatkowski et al. (2019) and TriviaQA Joshi et
al. (2017) consist solely of Wikipedia or Web documents, which fall short of
measuring OOD performances.

Robustqa Han et al. (2023) was the first dataset created to benchmark cross-
domain robustness for RAG-QA. However, as illustrated by the yellow highlights
in Figure 1, Robustqa follows NQ’s annotation format with short answer spans
extracted from the documents. Such data format is not the most suitable
reference answer to evaluate the current leading LLMs that typically generate
long-form responses with multiple information combined in one coherent
narrative Brown et al. (2020); Ouyang et al. (2022); OpenAI (2024).
Consequently, token overlap metrics used in the extractive QA era Karpukhin et
al. (2020); Izacard et al. (2024) will penalize unfairly on the additional
supporting tokens generated by LLMs, resulting in extremely low overlap
scores. As an example, Fig. 1’s extractive answers have poor Exact-Match or
$F_{1}$ scores with the final long-form answer. To create a long reference
answer, one can simply concatenate these short answers, but the synthesized
answers are either incoherent or ill-formatted, as illustrated by examples in
Sec. 3.1.

To address the drawbacks, we propose long-form RobustQA (Lfrqa) that
integrates multiple short extractive answers into a coherent long-form answer.
Figure 1 shows an annotation where three extractive answers are combined by
annotators to create a comprehensive answer. Table 1 summarizes seven features
in Lfrqa that make it uniquely beneficial for RAG-QA evaluations. ASQA
Stelmakh et al. (2022) and ELI5 Fan et al. (2019) are the most similar
datasets to Lfrqa. However, they are either not directly annotated against the
underlying corpus (thus, not RAG-QA), or rely on single-domain data, which is
insufficient to benchmark systems’ cross-domain performances.

With Lfrqa annotations, we propose Rag-qa Arena that leverages model-based
evaluators to directly compare LLMs’ answers with Lfrqa without the necessity
to examine long and potentially noisy retrieved passages. By demonstrating the
high correlation with human judges following the same instruction and rubrics,
we show that Rag-qa Arena is an efficient and accurate framework to benchmark
the RAG-QA system’s cross-domain performances. In this work, we primarily
focus on the LLMs used for the QA component, but Rag-qa Arena can be easily
extended to study retrieval’s impact on answer generation quality.

We summarize our contributions: 1) We present Lfrqa, the first high-quality
and large-scale multi-domain human annotations with coherent long-form answers
for RAG-QA. 2) We propose an efficient model-based evaluation framework, Rag-
qa Arena that enables users to directly compare LLMs’ answers with ground-
truth answers in Lfrqa. 3) We build a dashboard incorporating a wide range of
leading LLMs and conduct in-depth analysis to show that Lfrqa’s answers are
preferred significantly more to the best LLMs with long context. Therefore, we
believe Rag-qa Arena will serve as a challenging and robust evaluation
benchmark for future RAG-QA research.

Figure 2: Lfrqa v.s. Robustqa. Citations are removed in Lfrqa’s answers, and a
few answer spans are removed for clarity. Green and orange texts represent
positive and negative opinions, respectively.

##  2 RAG-QA Task Formulation

We briefly introduce the RAG-QA task in this section. Passage retrieval is the
first step of a RAG-QA pipeline. Following the passage retrieval set-up in DPR
Karpukhin et al. (2020) and Robustqa Han et al. (2023), we denote a collection
of documents as $\mathcal{D}$. We split each document $d^{i}\in\mathcal{D}$
with a fixed length $N$ tokens and obtain a collection of $M$
($\geq|\mathcal{D}|$) passages denoted as
$\mathcal{C}=\\{p_{1},p_{2},...p_{m},...p_{M}\\}$, where $p_{m}$ is a passage.
Given a question $q$, the passage retrieval task is to select $K$ most
relevant passages for $q$ with a retriever $\mathcal{R}$ from $\mathcal{C}$.
Formally, $\mathcal{R}(q,\mathcal{C})\rightarrow\mathcal{C}_{q}$.

Upon receiving top $K$ passages or $\mathcal{C}_{q}$, a QA model reads them as
context to generate an answer for the query. Unlike the extractive QA setting
in Robustqa and NQ, we adopt generative QA as it is most compatible with the
generative nature of the leading LLMs with the flexibility to produce free-
form answers. The answer generation task can be modeled as
$\Sigma_{1}^{T}\mathcal{P}_{q}(w_{t}|w_{0:t-1};\mathcal{C}_{q})$, where
$\mathcal{P}$ is an LLM. We focus on the variations of $\mathcal{P}$ and fix
$\mathcal{R}$ in this work.

In real-world applications, we deploy RAG-QA systems into various domains such
as healthcare, finance, and technology whose corpus and query types may not be
well covered in a trained retriever and LLM. Lewis et al. (2021) and Han et
al. (2023) show significant performance gaps between in-domain and out-of-
domain data for RAG-QA systems, further verifying domain adaptation problems.
Therefore, it is crucial to gauge the domain robustness of RAG-QA based on
LLMs, and Lfrqa helps achieve this evaluation goal.

##  3 Data Creation

|  |  | Test Set | Robustqa | Lfrqa
---|---|---|---|---|---
Domain | Source | Label | $|Q|$ | $|D|$ | $|P|$ | $A/Q$ | $W/A$ | $A/Q$ | $W/A$
Biomedical | BioASQ | [BI] | 1,956 | 15,559,026 | 37,406,880 | 2.6 | 2.4 | 1.0 | 30.0
Finance | FiQA | [FI] | 3,612 | 57,638 | 105,777 | 3.0 | 9.4 | 1.0 | 69.1
Lifestyle | LoTTE | [LI] | 2,208 | 119,461 | 241,780 | 5.7 | 8.7 | 1.0 | 99.5
Recreation | LoTTE | [RE] | 2,094 | 166,975 | 315,203 | 3.2 | 7.2 | 1.0 | 60.3
Technology | LoTTE | [TE] | 2,111 | 638,509 | 1,252,402 | 6.0 | 8.7 | 1.0 | 99.7
Science | LoTTE | [SC] | 1,423 | 1,694,164 | 3,063,916 | 5.3 | 7.8 | 1.0 | 92.0
Writing | LoTTE | [WR] | 2,695 | 199,994 | 347,322 | 6.2 | 6.6 | 1.0 | 88.0

Table 2: Data (test set) summary: Lfrqa v.s. Robustqa. $|Q|$, $|D|$, $|P|$,
$A/Q$, and $W/A$ represent numbers of questions, documents, passages, answers
per question, and words per answer, respectively. Each passage consist of 100
words at most. Lfrqa has only one answer per query as we integrate multiple
answers from Robustqa, which results in more words in (long-form) answers. Dev
set statistics can be found in Appendix Table 7.

Lfrqa consists of two types of QA samples: 1) new annotations in Finance
([FI]), Lifestyle ([LI]), Recreation ([RE]), Technology ([TE]), Science
([SI]), and Writing ([WR]) domains; 2) adapted long-form BioASQ ([BI]). We
describe the details of both QA samples in the following sections.

###  3.1 Annotated Data

Following Robustqa Han et al. (2023), Lfrqa’s new annotations are also based
on the LoTTE and FiQA queries and corpus. LoTTE was proposed in the ColBERTv2
paper Santhanam et al. (2022) and consists of information retrieval (IR)
datasets across five domains: lifestyle, recreation, technology, writing, and
science, each can have relevant answers coming from either web search or on-
line forum. FiQA Maia et al. (2018) proposes a task, “Opinion-based QA over
financial data” that answers finance-related questions from financial corpora
such as microblogs, reports, and news. It is important to note that both FiQA
and LoTTE are IR datasets with answers as long documents, which may include a
large amount of irrelevant information to the query.

As IR datasets, both FiQA and LoTTE could only provide relevant documents to
users, as there are no precise answer annotations. Robustqa addresses this
short-coming by extracting short answer spans from the long documents in the
similar format of NQ, which serves as a high-quality benchmark for extractive
RAG-QA. Figure 1 shows an example where the yellow highlights in Documents 1-3
are the extracted answers to the question.

Limitations of extractive RAG-QA.  In the era of LLMs, models’ responses to
user queries are often long and comprehensive OpenAI (2024); Anthropic (2024);
MetaAI (2024); Jiang et al. (2024), which the short, extractive reference
answers in Robustqa are no longer the most compatible format to evaluate
against. First, in Robustqa, annotators are limited to only taking 3 answer
spans per relevant document, each with no more than 16 words. This process
could result in a loss of useful information to help answer the query. Second,
to reconcile multiple extractive references for model evaluation, prior work
in extractive RAG-QA Karpukhin et al. (2020); Han et al. (2023); Izacard et
al. (2024) adopt the maximum of token overlaps between a model prediction and
a list of references to compute EM or $F_{1}$ metrics, which penalizes
unfairly the long-form responses from modern LLMs. Finally, if we naively
concatenate or list all short answer spans as shown by examples in Figure 2,
the combined answers are often too ill-formatted or incoherent as ground-truth
answers.

Lfrqa addresses all of these drawbacks by instructing the annotators to
integrate all short answers in Robustqa into a coherent long-form answer.
Below, we show a summary of our annotation instruction and quality control
mechanism.

Annotation instruction.  As Fig. 1 shows, a query, all relevant documents, and
original short answers (highlighted in the documents) are presented to
annotators on a single annotation page. Annotators need to combine all
highlighted answers into a single complete and coherent answer. All
highlighted answers MUST be included; otherwise, the annotation is considered
as a failure. Annotators are encouraged to include more information in the
documents if it helps to answer the queries. To ensure annotators faithfully
use the document information, we request annotators to provide citations after
each answer sentence. For example, the first sentence in Fig. 1 is composed
using information from Documents 2 and 3. Annotators should add "[2, 3]" after
that sentence. We use these citations primarily for data quality control and
remove them during the answer evaluation. The actual annotation UI can be
found in Appendix Fig. 5.

Quality control.  The data annotations are performed by contracted data
professionals. We also have a dedicated team of data linguists to validate the
annotation quality. Specifically, our data linguists randomly audit 10% of
each batch of the annotations, and if the valid answer ratio is $<90\%$, we
send the batch back to the annotators for re-work. The process iterates until
the valid answer ratio exceeds $90\%$. Here is a list of failure cases: 1\.
Incompleteness:  Answers do not include all highlighted answers, or there is
clear relevant information in the documents, but not included in the answer.
2\. Redundancy:  Clear irrelevant information is included in the answer. 3\.
Incoherence:  Answers are not coherent or not written in natural English. 4\.
Citation Error:  Wrong/missing citations, which indicate annotators do not use
correct information from the right documents.

###  3.2 Adapted Data

For the biomedical domain, Lfrqa leverages the same set of test queries as in
Robustqa, but uses the complete rather than span answers in the BioASQ
dataset. The original BioASQ annotations provide two types of answer formats:
1) exact answer, which is the short extractive answers used in Robustqa; 2)
ideal answer, which is a long-form abstractive answer to be consistent with
other datasets in this work. We did not perform further annotations. We notice
in Table 2 that BioASQ’s answers are shorter compared with other datasets.
This is due to its dominant amount of factoid queries, which do not require
elaborated explanations as in other datasets with more open-ended reasoning
questions Han et al. (2023).

We drop SearchQA in Robustqa as this dataset only has short-form extractive
answers, and its documents contain a significant amount of text omission (“…”)
that prevents us from re-constructing long-form answers.

###  3.3 Data Statistics and Analysis

Table 2 summarizes the statistics for the test set, which consists of 16K
queries across 7 domains. We filter out queries with more than 80 ground-truth
documents, resulting in 73 fewer queries compared with Robustqa. Since Lfrqa
combines multiple short answers, the answer per query ratio (A/Q) is always 1,
and the word per answer ratio (W/A) is substantially higher compared with
Robustqa. We also annotate a dev set with 10K queries for future model
development purposes, and the statistics can be found in Appendix Table 7. We
conduct further analysis below to demonstrate the unique contributions of
Lfrqa in Table 1.

Answers over multiple documents.  Figure 3 illustrates the distribution of
number of documents used by Lfrqa’s answers. Specifically, Figure 3(a) shows
that around 65% of the answers use $\geq$ 2 documents’ information. 4.9% of
the answers consist of information from 10 or more documents (maximum = 80).
In Figure 3(b), we divide long-form answers into sentences and show the
distribution of the number of documents used per answer sentence. Nearly 22%
of the answer sentences combine information from multiple documents. Both show
that Lfrqa’s answers effectively combine information across multiple ground-
truth documents. This makes Lfrqa challenging for RAG-QA, as it requires
identification and aggregation of information across sources.

Coherent answers.  Lfrqa’s answers further organize facts and views across
multiple documents in a coherent paragraph. Answers with multiple views are
common in the original Robustqa’s answers. Conducting a string match of both
“yes” and “no” as a leading word in an answer list, we found more than 200
examples with such conflicting information. This does not account for more
subtle cases where answers semantically contradict each other. In Figure 2, we
show 2 examples with conflicting views. Robustqa’s annotations simply list
them (separated by new-lines), whereas Lfrqa’s answers organize them as
coherent narratives, with conflicting information reconciled in helpful
context.

Fluency.  Robustqa’s answers are extracted from documents and often cut off
unnaturally to satisfy the limit of 16 words. In Figure 2, Robustqa’s answers,
such as “many ways to compromise your identity” and “mostly anecdotal evidence
here suggest no” are incomprehensible without further context, whereas Lfrqa’s
answers are all well written in complete sentences.

All of these features show that Lfrqa provides both challenging and high-
quality annotations for evaluating RAG-QA systems.

(a) # of documents used per answer.

(b) # of documents used per answer sentence.

Figure 3: Distribution of number (#) of documents used in Lfrqa’s answers. All
numbers are %.

##  4 RAG-QA Arena

In this section, we propose our evaluation framework Rag-qa Arena. Inspired by
the pairwise human preference evaluation framework such as Chatbot Arena
Chiang et al. (2024), we calculate win-rate and win+tie rate against Lfrqa as
ground-truth as a metric to gauge systems’ RAG-QA quality. Figure 4
illustrates the evaluation framework.

The choice of Lfrqa as the target to compare has been partly justified in Sec.
3 for 1) Completeness: the annotation process encourages the inclusion of as
much relevant information as possible. 2) Coherence: its answers are written
more coherently and naturally than Robust-QA as references for LLM
generations. Complete and coherent answers can be considered as a
comprehensive summary of all relevant information in the entire corpus. This
allows us to evaluate generated answers against Lfrqa answers only, which is
much more informative and concise than using retrieved passages, potentially
with a large amount of noise.

We implement human and model-based evaluations with the same instructions and
report their correlations. We will show results in Sec. 6.1 that further
justify using Lfrqa as evaluation targets.

Figure 4: Rag-qa Arena framework. Green blocks are LLM’s inputs to generate
answers. Orange blocks are LLM and LFRQA’s answers presented to both human and
LLM judges to determine pairwise preferences.

###  4.1 Human Evaluation

We present a query and a pair of answers (one from Lfrqa and one from an LLM),
to human annotators. We instruct them to rate their preferences based on three
aspects. 1\. Helpfulness:  information that is helpful/relevant to answer the
query. Touvron et al. (2023); Bai et al. (2022). 2\. Truthfulness:
information that is correct to answer the query. By our definition, truthful
information should also be helpful information Stephanie Lin (2021); Aisha
Khatun (2024). 3\. Completeness:  include as much truthful and helpful
information as possible. We further instruct annotators to use Truthfulness
(being both truthful and helpful) as the primary criterion since it is
stricter than Helpfulness. Helpfulness is used when a decision cannot be made
by Truthfulness alone. More details including the definition of aspects,
rating categories, and step-by-step guidelines can be found in Appendix A.7
and Fig. 6-7 (annotation interface).

###  4.2 Model-Based Evaluation

Since human evaluation is too costly, we adopt model-based evaluators for
scalable evaluation of LLMs on the entire Lfrqa test set.

As for the evaluation approach, we provide LLM-based evaluators with a query
and a pair of answers (including one from Lfrqa). Similar to human evaluation,
we prompt LLMs to rate their preferences based on the same three aspects
above. We only modify the human instruction slightly to be compatible with LLM
readable input text, but the majority of the prompt, especially the input
data, and rubric, stay the same (Appendix Table 13-15).

For both human and model-based evaluations, we allow “tie" (no preference) as
an option. For human evaluation, we take the majority votes from 3 annotators
to mitigate biases. If there is no majority vote, we default the label to
“tie.”

##  5 Experimental Setup

In this section, we discuss our retriever, LLMs experimented for both answer
generation and pairwise evaluations and their prompts in more detail.

Retrieval setting.  We employ ColBERTv2 Santhanam et al. (2022) as our passage
retriever, considering its superior performance on the underlying corpus for
both Robustqa and Lfrqa as shown in Han et al. (2023). We follow the same
retrieval setting and split passages into text chunks with 100 consecutive
words. We use the top 5 retrieved passages for our main results in Table 3 and
experiment with the top 10 passages for further analysis.

Answer generation.  We consider LLMs ranked top 25222Ranking at the time of
paper writing. in the Chatbot Arena Chiang et al. (2024) and their smaller
version to show the impact of model sizes. Due to resource and legal
constraints, for proprietary LLMs, we only use OpenAI models: a) GPT-4-turbo
(2024-04-09), b) GPT-4o and c) GPT-4-0125-preview). For public models, we
experiment with 1) Mixtral-8x22B-Instruct and Mixtral-8x7B-Instruct Jiang et
al. (2024); 2) Llama-3-70b-Instruct and Llama-3-8b-Instruct MetaAI (2024); 3)
Command R+ and Command R Gomez (2024); 4) Qwen1.5-110b-chat and
Qwen1.5-32b-chat Bai et al. (2023). Answer generation prompt can be found in
Appendix Table 11.

Pairwise evaluation.  For LLM-based evaluators, we focus on a few larger
models with strong context understanding capability, such as GPT-4-turbo,
GPT-4-0125-preview, GPT-4o, Mixtral-8x22B-Instruct and Llama-3-70b-Instruct.
Appendix Table 13-15 show the details of the pairwise evaluation prompts,
including instruction, example prompt template, and in-context-learning
examples. We shuffle the order of the answer pairs so that both human and
model judges are not biased by the position of an answer. We select the LLM
with the highest correlation with human judgments as the evaluator (Appendix
Table 8). We use OpenAI API to run GPT-4 models. We download public models
from HuggingFace Hub HuggingFace (2024) and run them on up to 8 Nvidia A100
GPUs with PyTorch (1.13.0) and Transformers (4.41.0) whose
tokenizer.apply_chat_template() function can help adapt the generic prompts to
different LLMs’ input formats.

We follow OpenAI’s
recommendation333https://platform.openai.com/docs/guides/prompt-engineering to
design prompts with chain-of-thoughts (CoT) Wei et al. (2022), in-context
learning Dong et al. (2023), and HTML tags as delimiters. We remove the
thinking process in model outputs as final answers.

| Overall | [BI] | [FI] | [LI] | [RE] | [TE] | [SC] | [WR]
---|---|---|---|---|---|---|---|---
Compared Models | W | W+T | W | W+T | W | W+T | W | W+T | W | W+T | W | W+T | W | W+T | W | W+T
GPT-4o* #1 | 36.9 | 41.0 | 52.9 | 59.3 | 38.4 | 42.3 | 25.1 | 27.9 | 40.4 | 46.4 | 35.6 | 38.8 | 42.8 | 47.6 | 28.4 | 31.1
GPT-4-turbo #2 | 34.4 | 39.1 | 36.0 | 43.9 | 40.6 | 45.1 | 23.2 | 26.1 | 36.7 | 44.1 | 36.6 | 39.6 | 42.6 | 47.9 | 26.2 | 29.6
GPT-4-0125-preview #6 | 28.9 | 33.7 | 31.4 | 40.1 | 36.8 | 40.8 | 18.1 | 21.3 | 31.5 | 38.6 | 30.4 | 34.0 | 34.7 | 40.5 | 19.2 | 22.3
Mixtral-8x22b #3 | 34.5 | 38.8 | 37.0 | 46.0 | 44.1 | 47.6 | 21.3 | 24.4 | 34.4 | 41.0 | 33.9 | 36.8 | 45.0 | 49.5 | 25.9 | 28.1
Mixtral-8x7b #7 | 27.5 | 31.0 | 31.9 | 39.1 | 35.3 | 38.5 | 15.9 | 18.4 | 24.8 | 29.5 | 30.3 | 32.1 | 33.9 | 38.0 | 20.0 | 21.9
Llama-3-70b #8 | 21.7 | 25.2 | 30.3 | 37.2 | 24.6 | 27.7 | 12.9 | 15.1 | 22.3 | 27.3 | 22.4 | 24.4 | 25.6 | 30.0 | 15.5 | 18.2
Llama-3-8b #10 | 20.4 | 23.5 | 34.7 | 39.6 | 24.0 | 27.0 | 11.2 | 13.2 | 19.4 | 24.7 | 20.5 | 22.5 | 22.3 | 26.1 | 12.5 | 14.4
Command R+ #9 | 21.1 | 25.8 | 26.0 | 33.5 | 25.8 | 30.3 | 13.5 | 16.4 | 22.6 | 30.0 | 22.4 | 25.4 | 24.9 | 31.2 | 13.6 | 16.0
Command R #11 | 11.1 | 15.2 | 18.6 | 26.1 | 13.0 | 17.1 |  05.2 |  07.4 | 10.4 | 17.0 | 10.3 | 12.3 | 14.9 | 20.2 |  07.3 |  09.4
Qwen1.5-110b-chat #4 | 33.4 | 37.8 | 36.2 | 44.0 | 42.6 | 46.9 | 22.3 | 25.1 | 34.1 | 40.7 | 34.8 | 37.5 | 40.8 | 46.1 | 22.5 | 25.2
Qwen1.5-32b-chat #5 | 32.8 | 37.1 | 34.9 | 42.8 | 43.2 | 47.3 | 20.7 | 23.7 | 32.3 | 38.3 | 34.0 | 37.1 | 40.8 | 44.8 | 22.6 | 25.2

Table 3: Evaluation results on Lfrqa test set. W and W+T indicate win and
win+tie rate against Lfrqa’s answers. LLM’s answers are generated based on the
top 5 passages. bold and underline indicate the best and runner-up results. *
means using the answer generation prompt w/o CoT. #n indicates the Elo ranking
in Appendix Table 6.

| Answer |  |  | Pearson | Cohen’s
---|---|---|---|---|---
| Pairs | Human | MBE | Corr. | Kappa
| Robustqa |  06.1 |  01.0 |  |
(1) | Lfrqa | 83.9 | 95.6 | 0.82*** | 0.71
| Tie | 10.0 |  03.4 |  |
| GPT-4 | 78.1 | 77.1 |  |
(2) | Robustqa | 13.7 | 19.9 | 0.60*** | 0.54
| Tie |  08.1 |  03.0 |  |
| GPT-4 | 29.9 | 32.0 |  |
(3) | Lfrqa | 59.1 | 63.1 | 0.54*** | 0.44
| Tie | 11.0 |  04.9 |  |
| Mixtral | 31.7 | 36.9 |  |
(4) | Lfrqa | 54.4 | 59.7 | 0.54*** | 0.43
| Tie | 13.9 |  04.4 |  |
| Llama-3 | 24.0 | 24.7 |  |
(5) | Lfrqa | 65.0 | 71.3 | 0.52*** | 0.45
| Tie | 11.0 |  04.0 |  |

Table 4: Pairwise comparisons between Robustqa, Lfrqa, and LLMs. GPT-4:
GPT-4-0125-preview; Mixtral: Mixtral-8x22b-Instruct; Llama-3:
Llama-3-70b-Instruct. Answer generated based on top 5 passages using
ColBERT-v2. All numbers are % except for Pearson Corr. and Cohen’s Kappa. ***
indicates strong correlation with p-values $\ll 0.001$.

##  6 Results and Analysis

Leveraging the evaluation framework described in Sec. 4, we first show that
Lfrqa’s ground truth answers are dominantly preferred as answers than
Robustqa. Then, we use the same evaluation framework to establish a new
leaderboard, Rag-qa Arena, aiming to reliably measure RAG-QA systems’
performances across diverse domains.

###  6.1 LFRQA v.s. RobustQA

In Sec. 3.3, we demonstrate Lfrqa’ advantages via data statistics. Here, we
show a more rigorous study to highlight the benefits. We subsample 700 queries
(100 from each of the 7 domains) and conduct pairwise preference comparisons
using both human and model-based evaluations. We compare three types of
answers: 1) Robustqa: concatenation of its extractive answers, separated by
"\n"; 2) Lfrqa: long-form answers in this work; 3) GPT-4’s answers based on
the top 5 retrieved passages.

Table 4 shows that when compared directly in Row (1), Lfrqa dominates
Robustqa. When comparing with GPT-4 in Row (2)-(3), Lfrqa significantly out-
performs GPT-4, but Robustqa significantly under-performs. These results show
strong evidence that Lfrqa’s answers can serve as better ground-truth than
Robustqa.

###  6.2 Quality of Model-based Evaluator

To build Rag-qa Arena on the Lfrqa test set, we need a scalable evaluation
method to benchmark various LLMs. We rely on model-based evaluation to achieve
this goal. Before showing the final dashboard results, we check the quality of
our selected evaluator (GPT-4-0125-preview) in Table 4.

To alleviate model bias, we use three LLMs’ answers as benchmark data, and the
query set is the same 700 subsample above. Row (3)-(5) use GPT-4-0125-preview,
Mixtral-8x22b-Instruct and Llama-3-70b-Instruct, respectively. All answers are
generated based on the top 5 passages.

We observe that LLM evaluators’ numbers align well with the average human
scores, except that LLMs tend to predict less “tie.” Most importantly, all
Pearson Correlation Freedman et al. (2007) are above 0.52 (with p-values $\ll
0.01$), and all Cohen’s Kappa Cohen (1960) are above 0.43, both showing strong
agreement between model and human judgments. In Appendix Table 8, we show
correlation numbers using alternative LLMs as evaluators, but none of them
works better than a single GPT-4-0125-preview model, which we select as our
best quality evaluator for Rag-qa Arena.

###  6.3 RAG-QA Arena

Finally, we show Rag-qa Arena’s benchmark results. In Table 3 we report each
model’s win and win+tie rate against Lfrqa.

Dashboard leaders.  GPT-4o leads the dashboard, with GPT-4-turbo and
Mixtral-8x22b-Instruct as close runner-ups. GPT-4o performs the best for [BI],
[LI], [RE] and [WR] domains, Mixtral-8x22b-Instruct leads in [FI] and [SC],
and GPT-4-turbo champions in [TE].

Impact of “no answer found.”  In RAG-QA, we rely on a passage retriever to
provide context, which could be irrelevant. Our prompt (Appendix Table 11)
asks an LLM to refrain from answering if it “couldn’t find an answer.” When we
use this answer generation prompt with CoT (the last two lines in the prompt),
GPT-4o produces 48.3% “I couldn’t find an answer” responses (Appendix Table
9). We randomly sample 20 such examples, and surprisingly found that in 16
cases, GPT-4o puts an answer in its <thinking> process, but continues to
generate “I couldn’t find an answer.” Fig. 8-9 show four such examples in
comparison with other LLMs’ answers with the same prompt, and GPT-4o’s new
answers without CoT. As the answer generation prompt with CoT only fails for
GPT-4o, we remove CoT for GPT-4o, which improves its answer format and reduces
the “no-answer” ratio to the level similar to other competitive models.

Note that these results raise a research question about the impact of prompt
engineering. We emphasize that the goal of Rag-qa Arena is to propose a
reliable evaluation framework, not to conduct extensive prompt engineering or
model training to pursue the best RAG-QA system. We provide a dev set of
Lfrqa, which can be leveraged in future research for model development
purposes.

Elo rating.  Table 3 shows dashboard results of win and win+tie ratio against
Lfrqa. We can further convert these pairwise comparisons into Elo ratings
similar to Chatbot Arena Chiang et al. (2024). Appendix Table 6 reports Elo
ranking. Column (A) uses the same data in Table 3; Column (B) and (C) add
pairwise comparisons for all unique model pairs on 700 and 1400 randomly
sampled queries across seven domains. So the total added preference pairs are
$\frac{N\times K\times(K-1)}{2}$, where K is the number of models in Table 3,
and N=700 and 1400 for Column (B) and (C), respectively. More details of Elo
ratings and their confidence intervals (CI) are shown in Appendix A.1.

We rank these pairs with the same LLM evaluator and found that in (C), Elo
ranking’s 95% CI can separate different LLMs. With a 50% increase of total
pairs (and thus the compute), it differs only slightly with the win+tie
ranking in Table 3 by flipping the order of llama-3-70b and Command R+. These
results present additional evidence that our approach of using Lfrqa only for
pairwise comparisons is both efficient and reliable.

Impact of the number of passages.  In Appendix Table 10, we compare the top 3
LLMs by increasing the number of retrieved passages from 5 to 10. Doubling the
number of passages (with extra costs) increases RAG-QA performances
significantly. We also find that both GPT-4 models’ improvements are greater
than Mixtral-8x22b-Instruct, showing their superior capability to understand
long context and identify useful information from noise. The best win rate of
GPT-4o against Lfrqa is 41.3%, which is 13.7% points lower than Lfrqa answers’
win rate against GPT-4o. This result shows that Lfrqa’s answer quality is
difficult to surpass, further justifying using it as an evaluation target.

Impact of model sizes.  For the non-GPT LLM family, more parameters lead to
better performances, but a larger increase in sizes does not always indicate
greater performance gains. For example, the two Qwen models have the second-
largest difference, but the lift from the smaller to the larger model is
marginal. We leave rigorous investigations to future research.

###  6.4 LLM as Annotators

|  | Citation |
---|---|---|---
| Completeness | Accuracy | Helpfulness
Lfrqa | 90.8 | 88.9 | 48.1
GPT-4 | 75.3 | 65.5 | 35.2

Table 5: Comparison between human and GPT-4 annotations. All numbers are in %.

Using large language models to provide annotations has been explored in
previous works Tan et al. (2024). It could provide a more scalable solution
than human annotations but can suffer from hallucination and accuracy issues
that require human validations Huang et al. (2023). We also experiment with
LLM as annotators before we start human annotations. We subsample 100 queries
from Lfrqa and prompt GPT-4-0125-preview to follow the similar procedure in
Sec. 3.1 to combine answers (Appendix Table 12). Then we request our data
linguists to compare Lfrqa and GPT-4 annotations based on 1) Completeness:
whether all Robustqa answers are integrated into the final answers; 2)
Citation Accuracy: whether citations in answers pointing to the right
documents; 3) Helpfulness: defined the same as in Sec. 4. Table 5 shows Lfrqa
out-performs GPT-4 annotations by 15.5%, 23.4% and 12.9% for the three
dimensions, respectively, suggesting human annotations are both valuable and
necessary for our task.

##  7 Related Work

RAG-QA  has been widely studied. Prior datasets are limited in the evaluation
as their corpus relies heavily on Wikipedia and the answers are mostly short
and extractive Rajpurkar et al. (2016); Kwiatkowski et al. (2019); Amouyal et
al. (2019). Robustqa and MultiHop-RAG Tang and Yang (2024) address the single
domain issue, but still adopt short, extractive answers, which is not as
suitable as Lfrqa to evaluate modern LLMs that generate long-form answers.

Longform QA datasets  have been proposed in prior work. ELI5 Fan et al. (2019)
and LongFact Wei et al. (2024) contain answers that are either not annotated
directly on the corpus, and or not created by humans. Krishna et al. (2021)
also points out that ELI5’s small validation set has significant leakage from
its train set. ASQA Stelmakh et al. (2022) is the most similar data to our
work, but its corpus is in the single Wikipedia domain. Lfrqa is by far the
only RAG-QA dataset with comprehensive long-form answers.

Pairwise preference  is now a standard way to evaluate LLMs. It allows direct
comparison between two responses Chiang et al. (2024); Lin et al. (2024). Rag-
qa Arena is unique by always including a high-quality human annotated LFRQA
answer, thereby making the evaluation more trustworthy.

##  8 Conclusion

We create Lfrqa, the first multi-domain dataset with coherent long-form
answers to reliably benchmark RAG-QA. We propose a reliable LLM-based
evaluation framework, Rag-qa Arena, that enables direct comparisons between
LLMs’ answers and Lfrqa, which we believe will facilitate the evaluation RAG-
QA in the era of LLMs.

## Limitations

We discuss some limitations of this work for future research efforts. Rag-qa
Arena can potentially cover more models. We didn’t include some leading LLMs,
such as Claude Anthropic (2024) and Gemini Google (2023) models, due to legal
and resource constraints, but we plan to add them to the leaderboard in the
future. Evaluation using GPT-4-0125-preview is not cheap. It costs on average
300 U.S. dollars per model on the full Lfrqa’s test set. We plan to subsample
10-20% of the queries for the final public leaderboard, which will be more
cost-friendly for future users. Future research can also study training
smaller but equally accurate models as evaluators. Finally, we mainly focus on
different LLMs for RAG-QA in this work, but future research can study the
impact of different retrievers or joint retrievers and LLM training using Rag-
qa Arena.

## Ethics Statement

The authors of this paper are committed to conducting research ethically. We
are leveraging existing LLMs to generate answers for Lfrqa, which include many
open-ended questions. LLM-generated answers could be incorrect or unfaithful,
as retrievers could find irrelevant passages and LLM can hallucinate Huang et
al. (2023). These are known issues in the AI research community, and that is
the reason we created Lfrqa to better evaluate RAG-QA systems. The additional
risks and potential harms are discussed in numerous previous works Bender et
al. (2021); Weidinger et al. (2021). The authors strive to ensure that the
research and its results do not cause harm.

Data used in this work have been collected from public sources and used in
accordance with all applicable laws and regulations. We use contracted data
professionals for Lfrqa annotations, and Appen
platform444https://www.appen.com/ for human pairwise preference annotations.
In both cases, we ensure our hourly rate is higher than 15 U.S. dollars per
local minimum wage standard. The intended usage of Lfrqa is compatible with
the underlying data’s access conditions (Appendix A.3)

## References

  * Aisha Khatun (2024) Daniel G. Brown Aisha Khatun. 2024.  Trutheval: A dataset to evaluate llm truthfulness and reliability.  _ArXiv_ , arXiv:2406.01855.
  * Amouyal et al. (2019) Samuel Joseph Amouyal, Tomer Wolfson, Ohad Rubin, Ori Yoran, Jonathan Herzig, and Jonathan Berant. 2019.  Qampari: An open-domain question answering benchmark for questions with many answers from multiple paragraphs.  _ArXiv_ , abs/1910.10683.
  * Anthropic (2024) Anthropic. 2024.  The claude 3 model family: Opus, sonnet, haiku.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, and et al. 2023.  Qwen technical report.  arXiv:2309.16609.
  * Bai et al. (2022) Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova Dassarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, John Kernion, Tom Conerly, Sheer El-Showk, Nelson Elhage, Zac Hatfield-Dodds, Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom B. Brown, Jack Clark, Sam McCandlish, Christopher Olah, Benjamin Mann, and Jared Kaplan. 2022.  Training a helpful and harmless assistant with reinforcement learning from human feedback.  _ArXiv_ , abs/2204.05862.
  * Bender et al. (2021) Emily M Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell. 2021.  On the dangers of stochastic parrots: Can language models be too big?  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_.
  * Bradley and Terry (1952) Ralph Allan Bradley and Milton E. Terry. 1952.  Rank analysis of incomplete block designs: I. the method of paired comparisons.  _Biometrika_ , 39(3/4):324–345.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020.  Language models are few-shot learners.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 1877–1901. Curran Associates, Inc.
  * Chiang et al. (2024) Wei-Lin Chiang, Lianmin Zheng, Ying Sheng, Anastasios Nikolas Angelopoulos, Tianle Li, Dacheng Li, Hao Zhang, Banghua Zhu, Michael Jordan, Joseph E. Gonzalez, and Ion Stoica. 2024.  Chatbot arena: An open platform for evaluating llms by human preference.  _ArXiv_ , abs/2403.04132.
  * Cohen (1960) Jocab Cohen. 1960.  A coefficient of agreement for nominal scales.  _Educational and Psychological Measurement_ , 20(1).
  * Dong et al. (2023) Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Baobao Chang Zhiyong Wu, Xu Sun, Jingjing Xu, Lei Li, and Zhifang Sui. 2023.  A survey on in-context learning.  _ArXiv_ , arXiv:2301.00234.
  * Fan et al. (2019) Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019.  ELI5: Long form question answering.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 3558–3567, Florence, Italy. Association for Computational Linguistics.
  * Freedman et al. (2007) David Freedman, Robert Pisani, and Roger Purves. 2007.  Statistics (international student edition).  _Pisani, R. Purves, 4th edn. WW Norton & Company, New York_.
  * Gao et al. (2023) Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Qianyu Guo, Meng Wang, and Haofen Wang. 2023.  Retrieval-augmented generation for large language models: A survey.  _ArXiv_ , abs/2312.10997.
  * Gomez (2024) Aidan Gomez. 2024.  Introducing command r+: A scalable llm built for business.  [Online; posted 04-April-2024].
  * Google (2023) Gemini Team Google. 2023.  Gemini: A family of highly capable multimodal models.  arXiv:2312.11805.
  * Han et al. (2023) Rujun Han, Peng Qi, Yuhao Zhang, Lan Liu, Juliette Burger, William Yang Wang, Zhiheng Huang, Bing Xiang, and Dan Roth. 2023.  RobustQA: Benchmarking the robustness of domain adaptation for open-domain question answering.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 4294–4311, Toronto, Canada. Association for Computational Linguistics.
  * Huang et al. (2023) Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, and Ting Liu. 2023.  A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions.  _ArXiv_ , abs/2311.05232.
  * Huang et al. (2019) Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019.  Cosmos QA: Machine reading comprehension with contextual commonsense reasoning.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2391–2401, Hong Kong, China. Association for Computational Linguistics.
  * HuggingFace (2024) HuggingFace. 2024.  Hugging face hub documentation.  Https://huggingface.co/docs/hub/en/index.
  * Izacard et al. (2024) Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi-Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. 2024.  Atlas: few-shot learning with retrieval augmented language models.  24(1).
  * Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, and et al. 2024.  Mixtral of experts.  arXiv:2401.04088.
  * Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017.  TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1601–1611, Vancouver, Canada. Association for Computational Linguistics.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Kočiský et al. (2018) Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018.  The NarrativeQA reading comprehension challenge.  _Transactions of the Association for Computational Linguistics_ , 6:317–328.
  * Krishna et al. (2021) Kalpesh Krishna, Aurko Roy, and Mohit Iyyer. 2021.  Hurdles to progress in long-form question answering.  In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4940–4957, Online. Association for Computational Linguistics.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Lewis et al. (2021) Patrick Lewis, Pontus Stenetorp, and Sebastian Riedel. 2021.  Question and answer test-train overlap in open-domain question answering datasets.  In _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume_ , pages 1000–1008, Online. Association for Computational Linguistics.
  * Lin et al. (2024) Bill Yuchen Lin, Khyathi Chandu, Faeze Brahman, Yuntian Deng, Abhilasha Ravichander, Valentina Pyatkin, Ronan Le Bras, and Yejin Choi. 2024.  Wildbench: Benchmarking language models with challenging tasks from real users in the wild.
  * Maia et al. (2018) Macedo Maia, Siegfried Handschuh, André Freitas, Brian Davis, Ross McDermott, Manel Zarrouk, and Alexandra Balahur. 2018.  WWW’18 open challenge: Financial opinion mining and question answering.  page 1941–1942, Republic and Canton of Geneva, CHE. International World Wide Web Conferences Steering Committee.
  * MetaAI (2024) MetaAI. 2024.  Introducing Meta Llama 3: The most capable openly available llm to date.
  * OpenAI (2024) OpenAI. 2024.  GPT-4 technical report.  arXiv:2303.08774.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, and Ryan Lowe. 2022.  Training language models to follow instructions with human feedback.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 27730–27744. Curran Associates, Inc.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 2383–2392, Austin, Texas. Association for Computational Linguistics.
  * Santhanam et al. (2022) Keshav Santhanam, Omar Khattab, Jon Saad-Falcon, Christopher Potts, and Matei Zaharia. 2022.  ColBERTv2: Effective and efficient retrieval via lightweight late interaction.  In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 3715–3734, Seattle, United States. Association for Computational Linguistics.
  * Stelmakh et al. (2022) Ivan Stelmakh, Yi Luan, Bhuwan Dhingra, and Ming-Wei Chang. 2022.  ASQA: Factoid questions meet long-form answers.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 8273–8288, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * Stephanie Lin (2021) Owain Evans Stephanie Lin, Jacob Hilton. 2021.  Truthfulqa: Measuring how models mimic human falsehoods.  _ArXiv_ , arXiv:2109.07958.
  * Tan et al. (2024) Zhen Tan, Alimohammad Beigi, Song Wang, Ruocheng Guo, Amrita Bhattacharjee, Bohan Jiang, Mansooreh Karami, Jundong Li, Lu Cheng, and Huan Liu. 2024.  Large language models for data annotation: A survey.  _ArXiv_ , arXiv:2402.13446.
  * Tang and Yang (2024) Yixuan Tang and Yi Yang. 2024.  Multihop-rag: Benchmarking retrieval-augmented generation for multi-hop queries.  _ArXiv_ , https://arxiv.org/abs/2401.15391.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, and et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv:2307.09288_.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed Chi, Quoc V Le, and Denny Zhou. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  In _Advances in Neural Information Processing Systems_ , volume 35, pages 24824–24837. Curran Associates, Inc.
  * Wei et al. (2024) Jerry Wei, Chengrun Yang, Xinying Song, Yifeng Lu, Nathan Hu, Jie Huang, Dustin Tran, Daiyi Peng, Ruibo Liu, Da Huang, Cosmo Du, and Quoc V. Le. 2024.  Long-form factuality in large language models.  _ArXiv_ , arXiv:2403.18802.
  * Weidinger et al. (2021) Laura Weidinger, John Mellor, Maribeth Rauh, Conor Griffin, Jonathan Uesato, Po-Sen Huang, Myra Cheng, Mia Glaese, Borja Balle, Atoosa Kasirzadeh, et al. 2021.  Ethical and social risks of harm from language models.  _arXiv preprint arXiv:2112.04359_.

##  Appendix A Appendix

|  (A) Lfrqa only |  (B) Lfrqa + |  (C) Lfrqa +
---|---|---|---
| pairs | 700 complete pairs | 1400 complete pairs
| Rating | 95% CI | Votes | Rating | 95% CI | Votes | Rating | 95% CI | Votes
Lfrqa | 1144 | +1/-1 | 176.7K | 1145 | +1/-1 | 176.7K | 1146 | +1/-1 | 176.7K
GPT-4o | 1066 | +5/-5 | 16.1K | 1081 | +4/-4 | 23.1K | 1085 | +3/-3 | 30.1K
GPT-4-turbo | 1050 | +5/-4 | 16.1K | 1058 | +4/-3 | 23.1K | 1065 | +3/-2 | 30.1K
Mixtral-8x22b | 1049 | +4/-4 | 16.1K | 1059 | +3/-3 | 23.1K | 1063 | +3/-3 | 30.1K
Qwen1.5-110b-chat | 1041 | +4/-4 | 16.1K | 1047 | +4/-3 | 23.1K | 1052 | +3/-3 | 30.1K
Qwen1.5-32b-chat | 1036 | +6/-4 | 16.1K | 1034 | +4/-3 | 23.1K | 1037 | +3/-3 | 30.1K
GPT-4-0125-preview | 1008 | +6/-5 | 16.1K | 1005 | +4/-4 | 23.1K | 1008 | +3/-3 | 30.1K
Mixtral-8x7b | 991 | +4/-4 | 16.1K | 991 | +3/-4 | 23.1K | 987 | +3/-3 | 30.1K
Llama-3-70b | 939 | +4/-5 | 16.1K | 931 | +4/-4 | 23.1K | 930 | +2/-3 | 30.1K
Command R+ | 938 | +5/-5 | 16.1K | 931 | +3/-4 | 23.1K | 924 | +3/-3 | 30.1K
Llama-3-8b | 924 | +6/-6 | 16.1K | 910 | +4/-4 | 23.1K | 903 | +4/-3 | 30.1K
Command R | 816 | +8/-6 | 16.1K | 802 | +5/-5 | 23.1K | 796 | +4/-4 | 30.1K

Table 6: Elo rating including “couldn’t find answer” responses. “Lfrqa only”
indicates the pairs that always include an Lfrqa answer. “Lfrqa \+ N complete
pairs” means we subsample N additional pairs evenly across 7 domains and
conduct comparison for all pairs of models. We have 11 models, so the total
new pairs are ${\frac{N\times 11\times 10}{2}}$.

|  |  | Dev Set | Robustqa | Lfrqa
---|---|---|---|---|---
Domain | Source | Label | $|Q|$ | $|D|$ | $|P|$ | $A/Q$ | $W/A$ | $A/Q$ | $W/A$
Lifestyle | LoTTE | [LI] | 2,151 | 268,893 | 597,729 | 5.9 | 5.9 | 1.0 | 102.2
Recreation | LoTTE | [RE] | 2,325 | 263,025 | 731,124 | 6.3 | 7.2 | 1.0 | 112.8
Technology | LoTTE | [TE] | 2,223 | 1,000,000 | 1,707,346 | 5.4 | 8.9 | 1.0 | 83.3
Science | LoTTE | [SC] | 2,137 | 343,642 | 854,756 | 4.8 | 6.4 | 1.0 | 100.6
Writing | LoTTE | [WR] | 1,972 | 277,072 | 713,692 | 7.0 | 7.7 | 1.0 | 109.6

Table 7: Data summary for the dev set. Based on Robustqa, there is no dev
split for BioASQ and FiQA data.

###  A.1 ELO Rating

Table 6 shows the results by converting pairwise comparisons to Elo rating
Chiang et al. (2024) with Bradley-Terry Model Bradley and Terry (1952). 95%
confidence interval (CI) is estimated using bootstrapping. Below, we explain
each column in more detail.

(A)  Lfrqa only pairs: we use the same data as in Table 3. All pairs include
an answer from Lfrqa.

(B)  Lfrqa \+ 700 complete pairs: data in (A) + comparisons of all unique
pairs of models for 700 randomly sample queries (100 for each domain).

(C)  Lfrqa \+ 1400 complete pairs: similar to (B), but increase query number
from 700 to 1400.

We can see in (C), that 95% CI can separate different models, so we stop
adding more paired comparisons. Notice that this Elo ranking aligns well with
ranking results using win+tie in Table 3.

###  A.2 Dev Set of LFRQA

We do not use dev set to fit the scope of the paper, but we provide an
additional 10K queries for future model developments, including prompting
engineering and training. Details are shown in Table 7. All data are collected
using the same process and quality control described in Sec. 3.

###  A.3 Data License

Lfrqa is created based on the following datasets. We make sure to follow the
data distribution license for our usage of the data.

  * •

FiQA: no license provided, but all data information can be found one the
official website555https://sites.google.com/view/fiqa/home.

  * •

LoTTE: MIT license666https://github.com/stanford-futuredata/ColBERT.

  * •

BioQAS: CC BY 2.5 license777http://participants-area.bioasq.org/datasets/.

  * •

RobustQA: Apache-2.0 license888https://github.com/awslabs/robustqa-acl23.

###  A.4 Annotation Interface

Our annotation interface can be found in Fig. 5. Annotators use this UI to
write long-form answers. Blue highlights are the original answers from
Robustqa.

###  A.5 Answer Generation Prompts

Table 11 shows our answer generation prompt. The last two lines “First, think
step-by-step…” are what we refer to as the CoT prompt, which we remove for
GPT-4o.

###  A.6 Evaluation Interface

We collect human pairwise preference data to benchmark our LLM-based
evaluators. Fig. 6-7 show the details.

###  A.7 Human Evaluation Instructions

Helpfulness:  Information that is helpful/relevant to answer the query. An
ideal answer consists of only information that is helpful/relevant to answer
the query Touvron et al. (2023); Bai et al. (2022).

Truthfulness:  Information that is correct to answer the query. By our
definition, truthful information should also be helpful information. If it is
difficult to determine the truthfulness of some information, we consider it
untruthful. Sometimes, this is due to not enough context provided in the
answer. Another source of untruthfulness is when conflicting information is
presented, and the answer does not coherently reconcile them Stephanie Lin
(2021); Aisha Khatun (2024).

Completeness:  include as many helpful and truthful information.

Here are the details of our instructions.

1\.  If one answer has all truthful information while the other has some
untruthful information, prefer the all-truthful one. 2\.  If both have some
untruthful information, prefer the one with less untruthful information. 3\.
If both have all truthful information, prefer the one with more truthful or
helpful information. 4\.  If two answers look equally good, or it is too hard
to differentiate, choose “Not sure.”

As the annotation UI shows, the actual ratings are “Better,” “Slightly
Better,” “Tie,” “Slightly Worse” and “Worse”. We merge “Better” and “Slightly
Better,” and “Slightly Worse” and “Worse” when computing correlation with
model-based evaluators.

###  A.8 Evaluation Model Selection

Table 8 shows Pearson Corr. between human and LLM-based evaluators. We find
using GPT-4-0125-preview alone achieves the best outcome, and thus, use it for
Rag-qa Arena evaluations.

|  Lfrqa v.s. |  Lfrqa v.s. |  Lfrqa v.s.
---|---|---|---
Evaluator Models | GPT-4 | Mixtral | Llama-3
Llama-3-70b | 0.52 | 0.51 | 0.48
Mixtral-8x22b | 0.53 | 0.49 | 0.50
GPT-4-turbo | 0.47 | 0.49 | 0.48
GPT-4o | 0.43 | 0.46 | 0.45
GPT-4-0125-preview | 0.54 | 0.54 | 0.52
Ensemble | 0.53 | 0.53 | 0.52

Table 8: Correlation between human judges and leading LLMs as evaluators.
Column titles are answer pairs. GPT-4: GPT-4-0125-preview; Mixtral:
Mixtral-8x22b-Instruct-v0.1; Llama-3: Llama-3-70b-Instruct. The ensemble takes
the majority vote out of the best three models.

###  A.9 No Answer Ratio

Table 9 shows the ratio of “I couldn’t find an answer" in each LLM’s answers.

###  A.10 Pairwise Evaluation Prompts

Table 13-15 show the details of the pairwise evaluation prompts including
instruction, example prompt template, and in-context-learning examples.

###  A.11 Error Analysis

Fig. 8-9 show examples where the original GPT-4o puts an answer in its
<thinking>, but generate “I couldn’t find an answer." It is fixed by removing
CoT prompt. We also compare them with Mixtral-8x22b, GPT-4-turbo and Lfrqa’s
answers.

| Overall | [BI] | [FI] | [LI] | [RE] | [TE] | [SC] | [WR]
---|---|---|---|---|---|---|---|---
GPT-4-turbo | 14.1 | 10.9 | 19.8 | 12.2 | 15.9 | 11.2 | 12.3 | 12.1
GPT-4o | 48.3 | 32.4 | 55.2 | 50.3 | 52.8 | 43.4 | 49.4 | 48.4
GPT-4o * | 16.9 | 10.4 | 26.8 | 13.9 | 19.8 | 11.2 | 14.7 | 13.9
GPT-4-0125-preview | 15.8 | 11.0 | 21.9 | 13.5 | 19.1 | 12.6 | 13.5 | 14.1
Mixtral-8x22b | 9.9 | 10.1 | 13.6 | 9.1 | 13.7 | 7.1 | 6.6 | 6.0
Mixtral-8x7b | 18.5 | 15.4 | 24.6 | 17.8 | 23.7 | 13.3 | 15.3 | 15.0
Llama-3-70b | 25.7 | 17.7 | 35.9 | 24.4 | 29.2 | 20.5 | 23.3 | 21.2
Llama-3-8b | 12.6 | 10.0 | 15.9 | 11.5 | 16.2 | 8.4 | 12.9 | 11.0
Command R+ | 14.4 | 10.8 | 21.7 | 12.5 | 16.1 | 10.1 | 14.7 | 10.5
Command R | 6.5 | 5.0 | 10.8 | 6.1 | 6.8 | 3.4 | 4.6 | 5.0
Qwen1.5-110b-chat | 11.5 | 10.8 | 15.9 | 10.7 | 13.8 | 6.8 | 9.0 | 10.0
Qwen1.5-32b-chat | 8.9 | 7.4 | 13.6 | 7.5 | 11.8 | 5.0 | 7.0 | 6.5

Table 9: No answer ratio for the entire Lfrqa’s test set. * means using answer
generation prompt without CoT.

| Overall | [BI] | [FI] | [LI] | [RE] | [TE] | [SC] | [WR]
---|---|---|---|---|---|---|---|---
| W | W+T | W | W+T | W | W+T | W | W+T | W | W+T | W | W+T | W | W+T | W | W+T
GPT-4o* w/ 5 psgs | 36.9 | 41.0 | 52.9 | 59.3 | 38.4 | 42.3 | 25.1 | 27.9 | 40.4 | 46.4 | 35.6 | 38.8 | 42.8 | 47.6 | 28.4 | 31.1
GPT-4o* w/ 10 psgs | 41.3 | 45.0 | 59.1 | 64.4 | 45.7 | 49.4 | 27.4 | 30.0 | 43.7 | 49.9 | 39.6 | 42.2 | 46.0 | 50.0 | 30.7 | 33.1
GPT-4-turbo w/ 5 psgs | 34.4 | 39.1 | 36.0 | 43.9 | 40.6 | 45.1 | 23.2 | 26.1 | 36.7 | 44.1 | 36.6 | 39.6 | 42.6 | 47.9 | 26.2 | 29.6
GPT-4-turbo w/ 10 psgs | 40.6 | 45.4 | 40.8 | 49.5 | 49.3 | 53.7 | 27.5 | 30.7 | 41.9 | 49.5 | 44.2 | 47.8 | 48.3 | 53.1 | 31.6 | 34.4
Mixtral-8x22b w/ 5 psgs | 34.5 | 38.8 | 37.0 | 46.0 | 44.1 | 47.6 | 21.3 | 24.4 | 34.4 | 41.0 | 33.9 | 36.8 | 45.0 | 49.5 | 25.9 | 28.1
Mixtral-8x22b w/ 10 psgs | 38.1 | 42.5 | 41.8 | 50.0 | 47.4 | 51.3 | 24.9 | 28.0 | 39.1 | 46.0 | 38.9 | 41.1 | 46.7 | 51.5 | 28.1 | 30.7

Table 10: Impact of the number of passages. Evaluation results on the entire
Lfrqa test set based on top 5 or 10 passages. W and W+T indicate win and
win+tie rate against Lfrqa. bold and underline indicate the best and runner-up
results. * means using the answer generation prompt w/o CoT. Answer Generation
Prompt
---
Based on the passages, provide a helpful answer to the query. Your answer must
be faithful to the content in the passages.
Do not use your own knowledge to answer the query. If you couldn’t find any
helpful information in the passages,
respond "I couldn’t find an answer."
Passages are inside <passage></passage> tags. Query is in the <query></query>
tags.
{x.passages}
<query>
{x.question}
</query>
First, think step by step, and put your thinking in <thinking> tags. Your
thinking must be shorter than 50 words.
Then, provide your answer.
Table 11: Prompt for answer generation. {x.*} indicates a component input that
can be replaced by actual data. We modify this prompt slightly to be
compatible with different LLMs’ input formats, but the majority of the prompt,
particularly instructions, remain the same. * The last section starting with
“First, think step by step…” is what we refer to as CoT prompt. We remove it
for GPT-4o only.

Figure 5: Annotation Interface Annotation Generation Prompt
---
Provide a response around 100 words to the query in the <query></query> tags
based on the passages. Passages are
inside <passage></passage> tags. The response must incorporate all candidate
answers in the <ans></ans>, and
you are allowed to rephrase these answers in order to make your final response
natural. The response should not
include any information outside passages.
You should cite the passage number (indices) in the format of [1], [2], [3,
4], etc. at the end of each sentence.
{x.passages}
<query>
{x.question}
</query>
Table 12: Prompt for GPT-4 annotations. {x.*} indicates a component input that
can be replaced by actual data.

Figure 6: Pairwise preference: instruction.

Figure 7: Pairwise preference: annotation interface. Pairwise Evaluation:
Instruction and Rubric
---
We will show you a query and a pair of answers to the query. You need to
provide your preference over this pair
of answers.
First, try your best to determine whether the information in an answer can
help truthfully answer the query. Then rate
your preference based on Helpfulness and Truthfulness.
\- Helpfulness: information that is helpful/relevant to answer the query. An
ideal answer consists of only information
that is helpful/relevant to answer the query.
\- Truthfulness: information that you believe is correct to answer the query.
By our definition, truthful information
should be helpful information. If you find it difficult to determine the
truthfulness of some information,
consider it untruthful. Often time, this is due to not enough context provided
in the answer. Another
source of untruthfulness is when conflicting information presented, and the
answer does not reconcile
them in a coherent way.
<rubric>
Here is how you judge (in the order of importance),
\- If one answer has all truthful information while the other has some
untruthful information, prefer the all truthful one.
\- If both have some untruthful information, prefer the one with less
untruthful information.
\- If both have all truthful information, prefer the one with more truthful or
helpful information.
\- If two answers look equally good, or it is too hard to judge using the 3
cases above, then you are our "not sure" which one
is better.
</rubric>
Table 13: Instruction and rubrics for pairwise evaluation. We use this
template across all LLM evaluators. This can be considered as the "system"
instruction for GPT-4 and LLama-3 models. Pairwise Evaluation: Example
Template
---
Query is in the <query></query> tags. Answer 1 is in <answer 1></answer 1>,
and Answer 2 is in
<answer 2></answer 2>.
<query>
x.question
</query>
<answer 1>
x.response1
</answer 1>
<answer 2>
x.response2
</answer 2>
Review the rubric in <rubric> tags,
\- if you prefer <answer 1>, output 1.
\- if you prefer <answer 2>, output 2.
\- if you are not sure, output 0.
First, think step by step, put your thinking in <thinking></thinking> tags.
Your thinking must be shorter than 50 words.
Then, provide your rating inside <rating></rating> tags. Remember your rating
should be 0 if you are not sure, and your
rating must be either 0, 1, or 2.
Table 14: Example template for pairwise evaluation. This template is used for both ICL examples and the final test example. Label | 1
---|---
Query | difference between publicly and publically.
Answer 1 | Both ‘publicly’ and ‘publically’ bear no difference in meaning, as they are essentially alternative spellings
of the same concept. Publicly is more widely used, but the existence of
’publically’ in reputable sources like
the OED means it cannot be dismissed as simply incorrect. Some opinions hold
that ’publicly’ is the older
irregular form, still preached by a lot of grammars, and ’publically,’ on the
other hand, is the newer and
regular form.
Answer 2 | There is no difference in meaning between ’publicly’ and ’publically’; they are alternative spellings of the
same word.
Reason | Both <answer 1> and <answer 2> are truthful. However, <answer 1> provides more truthful information
as the context to compare the two terms. Therefore, <answer 1> is better.
Label | 2
Query | what did European/American historical cooks do with the egg whites?
Answer 1 | Historical European and American cooks used egg whites for making egg white omelettes and egg white
pasta, as well as for stiffening clothing, similar to how starch is used
today.
Answer 2 | Egg whites have found their place in various non-culinary applications throughout history, such as
in the clarification of beer and wine, in the conservation of books through
bookbinding and gilding, and in
makeup as an ancient form of nail polish.
They were also utilized historically as a stiffening agent for clothing,
similar to how starch is used today.
The culinary landscape was not left untouched, with egg whites making their
way into recipes for omelettes
and pastas as early as the 15th century.
Reason | Both <answer 1> and <answer 2> provide several usages of egg whites for European/American cooks.
<answer 2> clearly provides more options with fully explained details.
Therefore, <answer 2> is better.
Label | 0
Query | should utf-16 be considered harmful?
Answer 1 | The question of whether UTF-16 should be considered harmful is subject to differing opinions. One
perspective suggests that UTF-16 is harmful due to a historical misconception
about character encoding and
recommends UTF-8 as the superior choice for various text interfaces. Another
viewpoint argues that
UTF-16 is not inherently harmful, emphasizing its utility in specific
scenarios where it serves as a
compromise between simplicity and compactness. The choice between UTF-16 and
other encodings like
UTF-8 depends on the specific requirements of the application, such as
compatibility with ASCII or
the need to efficiently encode certain character sets.
Answer 2 | UTF-16 should not be considered harmful.
However, contrasting views argue that UTF-16 should indeed be considered
harmful.
Some argue that the very reason UTF-16 exists is because some time ago there
used to be a misguided belief
that WideChar is going to be what UCS-4 now is.
Additionally, the harmfulness of UTF-16 is tied to issues with exercising
code.
Reason | Both <answer 1> and <answer 2> reconcile the two conflicting views with detailed explanation.
I am not sure which one is better.
Table 15: In-context examples for pairwise evaluation. Labels 1, 2, and 3 mean
"answer 1 is better", "answer 2 is better" and "tie", respectively. "Reason"
is a model’s chain-of-thought output.

Figure 8: GPT-4o’s original “I couldn’t find an answer” response compared with
GPT-4-Turbo and Mixtral-8x22b, and its answer without CoT (*). Blue highlights
in the passages indicate helpful information to answer the query. <answer>
tags are added to help differentiate from <thinking>.

Figure 9: GPT-4o’s original “I couldn’t find an answer” response compared with
GPT-4-Turbo and Mixtral-8x22b, and its answer without CoT (*). Blue highlights
in the passages indicate helpful information to answer the query. <answer>
tags are added to help differentiate from <thinking>.