?reference?

#  CFMatch: Aligning Automated Answer Equivalence Evaluation with Expert
Judgments For Open-Domain Question Answering

Zongxia Li Ishani Mondal Yijun Liang
Huy Nghiem Jordan Boyd-Graber
University of Maryland, College Park
{zli12321, imondal, yliang17, nghiemh<EMAIL_ADDRESS>
###### Abstract

Question answering (qa) can only make progress if we know if an answer is
correct, but for many of the most challenging and interesting qa examples,
current evaluation metrics to determine answer equivalence (ae) often do not
align with human judgments, particularly more verbose, free-form answers from
large language models (llm). There are two challenges: a lack of data and that
models are too big: llm-based scorers can correlate better with human judges,
but this expensive task has only been tested on limited qa datasets, but even
when available, update of the model is limited because llm are, well, large
and often expensive. We rectify both of these issues by providing clear and
consistent guideline for evaluating ae in machine qa adopted from professional
human qa contests. We also introduce a combination of standard evaluation and
a more efficient, robust and lightweight discriminate ae classifier-based
matching method –CFMatch (<1 MB), trained and validated to more accurately
evaluate answer correctness in accordance with adopted expert ae rules are
more aligned with human judgments.

##  1 Introduction

qa is a key task in natural language processing with many models and datasets,
where most interactions with a language model can be considered qa tasks.
There are two common ways to improve qa models’ ability to better answer
questions: either use new and better data or build new, better, and (usually)
bigger models. This paper does neither of those things. Instead, we focus on
improving evaluation.

A skeptical reader might ask: Won’t a better model always have a higher score
even if the evaluation is better…who cares? From a model selection
perspective—particularly for models that are similar—subtle differences could
swap the order of models Voorhees and Tice (2000). However, from a training
perspective, having the right objective still matters: keeping models, data,
and _test_ evaluation fixed, you can still improve test evaluation by
improving the _training_ evaluation metric Si et al. (2021).

We focus on the answer evaluation (ae) task: given a set of gold answers, is
the output of a system equivalent to one of the gold answers? The standard ae
evaluations of qa are Exact Match (em)– evaluates qa systems’ ability to
extract a specific span of text given a passage and a question, token level
$F_{1}$ match– measures the average overlap between the prediction and ground
truth answer, rouge score Lin (2004)– measures the overlap of n-grams between
the system’s output and a set of reference summaries, meteor score– evaluates
machine translation output and has stemming and synonymy matching features not
present in other metrics Banerjee and Lavie (2005). While these standard ae
evaluation methods work okay on the most common qa evaluations, we argue in
Section 2 that they stumble on the long-tail of qa examples because they lack
the depth of semantic understanding of the contexts of the questions and
answers that humans use when adjudicating answers.

Fortunately, we are not starting from scratch! We draw on rules for
adjudicating human question answering competitions, studying and adopting
standardized ae guidelines from National Academic Quiz Tournaments (naqt)
National Academic Quiz Tournaments, LLC (2024), A Jeopardy! ‘Case Book’
Carberry (2019) and the efficient qa competition Min et al. (2021) to define
acceptable answers for machine qa and expand the long-tail of automated qa
evaluation in Section 3.1. We use a large language models (llm)—gpt-4 Bubeck
et al. (2023)—to generate and self-verify ae examples according to our ae
guidelines.

This broader view of ae is more necessary than ever because generative ai is
often more creative with their answers Zhu et al. (2021). For example, e.g.,
gpt-series models Bubeck et al. (2023), LLaMA 2 Touvron et al. (2023) go
beyond extracting exact answers from retrieved passages, such as Dense
Retriever-Reader (dpr) models Karpukhin et al. (2020) and generate more free-
formed answers, with more variability than a set of given reference answers.
Creative answers often make automated ae evaluations more challenging. For
example, section 5.2 shows that out of 250 selected ae examples, around $90\%$
of the examples have correct candidate answers missing from the reference
answer sets, which are equivalent under human judgment but not equivalent when
using string exact match.

However, we recognize that one of the reasons that exact match (em) is so
popular is because it is simple, fast, and requires minimun setup and
computation resources comparing to rising popularity of llm-based evaluation
methods Kamalloo et al. (2023a). Thus Section 3.3 proposes CFMatch, which
combines standard evaluation methods – $F_{1}$ – with a very simple
discrimative logistic regression (lr) classifier and trains it on our
augmented ae dataset to distill more complicated models to a fast, small, easy
to run deterministic classifier that is competitive with our llm approach. Our
classifier, less than 1 MB in size, is tested on a challenge expert QA set
involving expert adjudications of AE that requires a regular person’s years of
experience to get right. It achieves the best ae alignments with expert
judgments without access to additional online or retrieved knowledge, followed
by bert-based method trained on our augmented ae dataset then other standard
evaluation methods.

In addition, We conduct further human evaluation to show that CFMatch reduces
common judgement errors present in current evaluation metrics and is
comparable with Muppet Model (transformer architecture models)-based matching
methods, and more closely aligned and generalized with our ae rules. We also
demonstrate the advantage of augmenting ae training data can improve both our
classifier and Muppet Models’ ability to detect ae. We release a python
package specifically for qa evaluation that includes em, $F_{1}$ socre,
CFMatch, fine-tuned bert matching (download optional) for researchers to
access popular and robust qa evaluation
metrics.111https://github.com/zli12321/qa$\\_$metrics.git

##  2 Limitations of Current Evaluation Methods

In this section, we first review the current state and definition of ae in
machine qa domain. Then we conduct our initial error analysis of current
popular qa evaluation methods—em, $F_{1}$, bem Bulian et al. (2022)—on the
benchmark Reading Comprehension with Commonsense Reasoning (ROPES) dev dataset
Lin et al. (2019), and we show the pitfalls of current evaluation methods. We
specifically present annotated examples to show the lack of generalization of
these evaluation methods on out-of-distribution (ood) datasets. especially for
bert-based methods trained on ae data generated only from the Stanford
Question Answering Dataset Rajpurkar et al. (2016).

###  2.1 Definition of qa evaluations

The definition of ae varies depending on the type of evaluation methods. We
present three qa evaluation methods with their definitions, which we will also
use for our error analysis in section 2.2.

#### Exact Match (em)

It is both the simplest and most-used qa evaluation method used in qa because
it is unambiguous. em was introduced when extractive qa focused on source
texts, but it falters with models that provide answers in different but
correct forms (for example, answers with additional helpful information useful
for a user using a qa system).

####  $F_{1}$ Score

The $F_{1}$ score measures the token overlap between the reference and
candidate answers, focusing on both precision and recall. Each continuous
sequence of non-whitespace characters is a token. Suppose reference answer
World Health Organization global public health leader has 7 tokens–["World",
"Health", "Organization", "global", "public", "health", "leader"], whereas
candidate answer WHO global leader has three tokens–["WHO", "global",
"leader"]. Precision calculates the proportion of relevant words in the
candidate answer, while recall assesses the proportion of relevant words in
the reference answer that are also in the candidate answer. The True Positive
(tp) between the examples are 2 since "global" and "leader" are two common
words in both answers; the False Positive (fp) is 1 since "WHO" is the token
not present in the reference; the False Negative (fn) is 7 since there are 7
tokens in the reference but not in the candidate. Thus we calculate precision
and recall using formulas:

$\displaystyle\text{Precision}=\frac{\text{tp}}{\text{tp}+\text{fp}},\text{Recall}=\frac{\text{tp}}{\text{tp}+\text{fn}}$

The precision and recall are approximately 0.67 and 0.29, thus resulting
$F_{1}$ score is 0.4. This method acknowledges the correctness of a candidate
answer by accessing essential information and amount of token overlap in the
reference answers, but does not take the context of the question into account.
In addition,there is no consensus regarding the choice of threshold for
$F_{1}$ score, making this metric more challenging to interpret than the em. A
0.3 threshold will make the candidate answer equivalent to the reference but
0.5 will not. Section 3.3 will introduce how we combine $F_{1}$ with a new
evaluation method, eliminating the need for threshold selection.

####  bert-Matching (bem)

bem uses a bert-based model trained on an ae dataset that can can take the
questions, reference, and candidate answers all into account to detect
candidate answer correctness. (Bulian et al., 2022) provides two fundamental
criteria to define ae and annotate the ae dataset based on 1: Candidate
answers contain at least the same (or more) relevant information as the
reference answers, while taking into account of the context and the question;
in particular it does not omit any relevant information present in the
reference answer should be ruled correct; 2: Candidate answers contain neither
misleading or excessive information not present in the reference answers
taking into account of the question and context should be ruled incorrect.
Fine-tuned bert-based evaluation methods have the highest alignment with human
judgments than em and $F_{1}$ on their ae test set.

###  2.2 Generalization of Current ae Evaluation

#### Out-of-distribution (ood) dataset

We run error analysis on current qa evaluation methods using the Reading
Comprehension with Commonsense Reasoning (ropes), which focuses on causal
situational reasoning in reading qa pairs Lin et al. (2019). Each question has
its background information and a context, and a reference answer. We use the
dev set with 1,688 qa pairs as our test set in our experiment.

#### Candidate answer generation

We use flan-t5 xl Chung et al. (2022), a closed-book instruction-tuned
language model for any tasks as an our qa model to generate candidate answers
for ropes. We concatenate the background information and situation passage as
the context of the question and prompt flan-t5 with the context and question
to generate candidate answers for ropes.222We acknowledge that T5 is no longer
a sota model, but since our goal is distinguish correct from incorrect
answers, our goal is to have a relatively inexpensive model to run that is
replicable but still different from the extractive models that can produce
enough free-form answers.

#### Pitfalls and generalization of current evaluation methods

We manually annotate the correctness of 1,688 candidate answers based on
selected correctness guidelines discussed in Section 1, then run em, $F_{1}$
and bem to evaluate the correctness of the candidate answers. We then compare
the alignment between human judgments and the three evaluation methods, and
list out pitfalls separately for em, $F_{1}$, and bem.

####  em is overly strict

We analyze total 102 examples that em considers incorrect but are actually
correct under human judgments. em suffers when the candidate answers are a
good choice for the question but not in the reference lists. For example, 12
PM and 12 noon, Mount Everest and Mt. Everest, wetlands area and wetland are
all failure cases for em to detect ae. It becomes even more challenging for em
in the era of generative qa models where the answers are not exact extractions
from the resource passages. em greatly underestimates our selected machine qa
model’s accuracies.

####  $F_{1}$ Score is sensitive to thresholding

We run $F_{1}$ evaluation with a threshold of 0.3, 0.5, and 0.7 to determine
ae, where the number of misalignements with human judgments are 303, 51, 100.
However, under a practical case when we use $F_{1}$ to evaluate qa answering
models, the selection of threshold is critical and challenging depending on
the purpose of the evaluation and the type of the dataset. $F_{1}$ also cannot
evaluate ae by taking the context of the questions into account. An example
would be Question: Will there be more or less blue ice at the glacier than the
frozen over lake? Reference: more. Candidate: more blue ice, where additional
information is provided but fades out the number of token overlap, resulting
in an incorrect judgment for $F_{1}$.

####  bem suffers on ood datasets

bem is trained on ae data generated from the Standford Question Answering
Dataset (SQuAD) Rajpurkar et al. (2016), where it can detect ae beyond token-
level matching and word similarity in the embedding space (bert-Score).
However, when testing it on an ood dataset, where the testing data has a
different style from the training data, bem has 702 misalignments with human
judgments. Specifically, bem have the most number of misalignments with human
judgments on examples where the reference and candidate answers are different
but might be close in distance in the embedding space. more and less,
Wednesday and Thursday, planet A and planet B, turn off and on are all
examples where human judges it as incorrect but bem judges it as correct. The
limitation of current bert-based evaluation methods face challenges in
generalization to ood datasets for its ae training data with less -well-
defined correctness rules and restrictions.

Although the current qa evaluation methods work okay in most of the middle
parts cases, there are still gaps between the human correctness decision
making process and the current evaluation methods, and new evaluation methods
adopted from expert human decision process are needed in the current era of
machine qa

Rule |  Description |  Question |  Correct Answer |  Generated Answer
---|---|---|---|---
1:entity-aliasing |  Widely recognized aliases, pseudonyms, or alternative names that are commonly associated with the correct answer entities are acceptable |  who plays Red on Orange is the New Black? |  Kate Mulgrew |  Katherine Maria Mulgrew/ correct aliasing name
where can I get a state issued id? |  DMV |  Department of Mobile Vehicles/incorrect aliasing
2:numerical information |  Exact dates, years, numerical are required unless the question specifically asks for approximations |  What year did World War II end? |  Sep 2, 1945 |  1945/ correct exact numerical
How tall can a giraffe grow? |  16-20 feet |  18 feet/ incorrect/ must be a range
3:less details |  The generated answer provides less details but should include the essential and correct information required by the question (specificity 1) |  Who discovered penicillin? |  Scottish physician Alexander Fleming |  Alexander Fleming/ correct within context
What followed the late local programming after Super Bowl 50? |  The Late Late Show with James Corden |  The Late Show/ incorrect missing important information
4:more details |  Generated answer contains the correct answer or an equivalent part with additional information (Specificity 2) |  When did Morales launch his policy in the eastern lowlands? |  2009 |  August 3, 2009/ correct exact date and time
Protective coloring is common in what insect family? |  beetle |  beetle and formicidae/ incorrect additional information
5:semantic equivalence |  A high degree of word overlap or similarity does not establish equivalence. The answer must be contextually and semantically accurate |  What is the primary source of energy for the Earth? |  Solar radiation from sun |  Solar energy from sun/ correct semantic with high overlap
Which group has a higher likely hood of brain damage? |  group A |  group B/ incorrect group
6:irrelevant, inaccurate information |  The generated answer is irrelevant to question and the given answer or is an inaccurate description of the question should be incorrect |  Which event occurred first, the first Super Bowl or the formation of the Premier League? |  first Super Bowl |  formation of the Premier League/ incorrect with inaccurate answer
7:other Possible answers |  Generated response is correct but not in the reference list |  When did the Golden State Warriors win the NBA Championship? |  2015, 2017 |  2022/ another possible correct answer
Table 1: The correctness of a candidate answer can be traced and categorized
to one or more of the above rules. We provide a comprehensive guideline to
define correctness and equivalence of answers, where the rules are adopted
mainly from naqt and slightly modified upon analysis of annotated dataset with
human correctness judgments between reference answers and machine generated
answers. The acceptability of qa model answers should be based on theses
guidelines. Our ae classifiers will be also trained on datasets with theses
acceptability rules.

##  3 The New Evaluation Framework

In this section, we explore the process to revise ae rules from the nqat
National Academic Quiz Tournaments, LLC (2024), Jeopardy! Carberry (2019),
EfficientQA Min et al. (2021). We then introduce steps to augmenting our ae
training set, and the selection and training details of our ae classifiers.

###  3.1 Revising ae Framework

We probe the limitations of current qa evaluation methods and show their lack
of generalization in the era of generative qa models. Thus we need a more
robust qa evaluation framework that can balance limitations of current
evaluation methods. Building ae and answer acceptability rules from scratch
with limited data is challenging, meanwhile also not generalizable to possible
diverse styles of questions asked. Thus, having a pointer to existing detailed
ae guidelines can both ensure the rigorousness, diversity, and generality of
our ae guidelines. Luckily, the professional Trivia community already have a
set of well-defined norms to evaluate humans’ ability in qa. naqt is a thirty-
year-old purveyor of qa competitions spanning middle schools to “open”
competitions that anyone can join. Because of the breadth of participants and
judges (often drawn from parent volunteers of participants), the organization
has developed easy-to-follow rules that make answer adjudication consistent
(otherwise, participants would complain about unfairness). Likewise, the
American game show Jeopardy! was forged in the crucible of the mid-century
game show crisis Stone and Yohn (1992), and in addition to its quirky format
where contestants’ responses are gramatically questions, it has a complicated
adjudication process to ensure fairness McNear (2020).

#### Expert rules in machine qa paradigm

Since naqt and Jeopardy! format is designed for real human players and is
quite different from the current machine qa paradigms, a better practice than
adopting all the rules is to revise them and fit them into te current machine
qa settings. For example, the rules about pronunciation, the order of names,
foreign language responses, etc are not considered suitable under a standard
machine qa paradigms. The rules about pronunciations are specifically targeted
for in-person or online audio versions of the tournament, where plausible
pronunciations or phonetics are generally acceptable—koan and cone, mee-jee
and mye-jye are considered equivalent under their rules. naqt has very strict
rules for the order of names, where Martin Luther King and King, Luther Marin
are considered not equivalent but under our machine qa paradigm, we do not
enforce the order of names and foreign language answers. On the other hand,
the guidelines nonetheless provide rules we can reference in the machine qa
paradigm. For example, the specificity rule is both present in the naqt and
Jeopardy! rules, where the responses should be specific enough under the
context of a question—Question: Where is the Eiffel Tower located?—where the
answer Europe is incorrect if the given reference answer is France, but is
acceptable if Europe were the intended answer.

#### The revising process

Thus we analyze misalignment of ae examples between human and selected
automated evaluation methods, we also selectively analyze the aligned examples
that all evaluation methods get right to ensure the generality and
comprehensiveness of our ae guidelines. Our analysis examples include 1,688
examples from ropes dataset and select 600 examples in the official ae test
set that have misalignment between human judgments and the other two
evaluation methods. We also pick 200 additional examples from the ae test set
where both bem and human judgments align. We select, categorize and revise
rules from the Trivia community that can fit into our current sample analysis
and present a comprehensive category of our modified correctness rules that
can fit into the current machine qa paradigm in Table 1 with additional
examples in Appendix 2.

###  3.2 Training Data Augmentation

With these revised ae and acceptability rules, we want to train our ae
classifier to follow these rules. Thus, we generate ae examples to train a
classifier to learn our ae patterns. For each ae rule, we collect a small
number of example qa pairs from the naqt correctness guidelines, and error
examples from the ae test set and the ropes dev set. Then we manually revise
qa pairs that violate our ae guidelines, and to ensure the quality and
diversity of our example qa pairs. We end up writing five to ten examples for
each rule, where also balance the number of correct and incorrect pairs except
for rules that are strictly judged incorrect (Table 1).

#### Training data generation

We use the manually revised qa pairs as seed examples to prompt gpt-4 with an
example and the specific ae rule to generate more similar qa pairs with its
judgment of equivalency and correctness.333We set the temperature of gpt-4 to
0.9 and frequency penalty to 1.9 to ensure the diversity of generated qa
pairs. At the end of each generation iteration, we append the generated
examples to the seed examples, which we generate a total of 4,234 synthetic ae
examples.

#### Quality validation

To verify the automatically generated examples, we manually select fifty
generated ae examples and check the judgments. gpt-4’s disagreements with
humans is most frequent with Rule 2 from Table 1, e.g., gpt-4 often considers
numerical answers like 2008-2010 and 2010 to be equivalent. For each generated
example, we run gpt-4 self-evaluation by prompting it to verify the
correctness of its generated judgments with the rules, questions, reference
answers, and candidate answer, and judgment to improved the quality and
accuracy of judgments Ren et al. (2023).

We re-check the 50 generated examples that have inconsistent generated and
self-evaluated judgments, and see an improvement of the judgments—39 out of 50
example are consistent with human judgments after self-evaluation. Moreover,
we select 300 additional generated examples (mainly examples on Rule 2) and
manually check and revise the self-corrected judgments to accommodate our ae
guidelines. 180 self-verified judgments are revised and 65 invalid examples
(false reference answers, invalid questions) are removed with 3,989 examples
remaining in our training material. After two steps of validation, we are
confident that the synthetic data mostly align well with our ae rules.

#### Extra Human Annotated Data Augmentation

If the first step of data augmentation is to collect representative materials
to fit the classifier to follow ae norms, then we also want to generalize and
teach the classifier with more real human annotated judgments. Luckily, we
already an existing annotated ae dataset available. We expand the ae learning
material by combining our 3,989 synthetic examples with 9,090 examples from
the ae training set, with 1,688 annotated examples from the ropes dev dataset
as our ae-augmented training set.

#### Entity Aliasing Expansion

This method is specifically targeted to correctness rule 1, where the
reference and candidate answers are entities that have other possible aliases
not in the current answer set. This step is for users who wish a more rigorous
evaluation that can find more relevant and timely aliasing entities from
Wikipedia to increase the chance of answer matching depending on their model
deployment and usage purposes. For example, if the evaluation dataset is
constructed in 2016, where the reference answer is only Burj Khalifa, but when
the question What is the tallest building in the world? is asked today, the
same building is also known for other aliasing names such as Khalifa Tower,
Burj Dubai. In addition, a classifier trained on a fixed ae dataset without
updates of aliasing examples would also be a challenge to detect more general
ae. Thus, we also include Wikipedia entity linking features Wikidata
Contributors (2019) into our evaluation pipeline to expand the reference and
candidate answer set of our test sets if possible to improve ae
generalization, but this step can be turned on or off depending on users’
purposes and trade off of more resource required.444This method is likely the
most effective for datasets like TriviaQA, entityQ, HotpotQA, where many of
the answers to questions are entities that have aliasing from the Wikipedia
page, and is less effective for datasets like ropes, where the answers are
mainly based on context and reasoning of the questions.

###  3.3 Feature Engineering and Classifier Training

qa evaluation is not a task like qa where the size of the model is a secondary
consideration; rather, an effective method (i.e., that could be adopted) needs
to be competitive as well. The goal of this section is to match the
effectiveness of the prior llm-based evaluation method while minimizing
computation, latency, and disk space, thereby taking advantages of both llm-
based and standard evaluation methods. em and $F_{1}$ evaluation methods are
fast and require no disk space but they have the limitation of superficial
token-level matching. llm-based evaluation methods can go beyond token-level
matching and be trained to aligned more human judgments, but the amount of
computational resource, disk space, latency, bandwidth, computational time,
and costs required also hinder their frequent usage in the standard qa
training and evaluation pipelines. We want to not just build a system that
does the best on ae task but one that does the best with minimal resources
because in training QA systems, it will be used many times. In the rest of the
paper, we provide details of construction and evaluation of CFMatch and
compare it with llm-based methods in computation cost, time cost, disk space
cost, accuracy performance. Then we conduct human annotations and analysis of
our evaluation methods.

Dataset |  # Pairs |  Context |  Question |  Gold Answer
---|---|---|---|---
ropes |  1,688 |  Michael Paul Welch (born August 25, 1972) is a former right-handed pitcher … was named a 2014 "Best High School" by U.S. News and World Report. |  Mike Welch attended high school that serves how many students ? |  2,200 students
nq-open |  3,610 | - |  who won the American league east in 2017 |  The Yankees, Houston Astros
Hotpotqa |  3,300 |  Asmara International Airport (IATA: ASM) (ICAO: HHAS). Asmara currently hosts the country’s only operating international airport… |  Asmara international airport is in which country? |  Eritrea
Narrative-qa |  3,300 |  The narrator, a Bostonian, returns after a brief visit a few summers prior, to the small coastal town … cMaine coastal town increases each day. |  What was Littlepage’s job? |  sailor, retired sailor
Table 2: The first dataset in the table is our error analysis dataset and we
use the rest four datasets for our generalization testing. We do not include
contexts for nq-open for more variability of generated answers to challenge
the evaluation methods.

#### Classifier Selection

We use logistic regression (lr) as our base classifier for its simplicity and
efficiency. Additionally, we use bert-base-uncased as our comparison model and
train one on the original ae dataset and another on the augmented ae dataset
following Bulian et al. (2022)’s training format. 555We use l2 penalty and a
starting random state 668 for logistic classifier. We trained bert with
learning rate = 1e-5, weight decay=0.01, batch size=32 for 7 epochs and save
the state with highest accuracy on the ae test dataset. We do not use bem
directly because we want the two bert models have consistent training
hyperparameters and details.

#### Feature Engineering

To maximally take advantage of detecting ae from both the answers and semantic
meanings of the questions, we follow encoding method used to train llms to
format the qa pairs: [CLS] candidate [SEP] reference [SEP] question [SEP],
where an encoding vectorizer will encode the [CLS] and [SEP] tokens as
separator between questions and answers. We use unigram tf-idf Jones (2021) to
encode our training data as pairs. In addition, $F_{1}$ score provides rich
information about the amount of information and token overlap, precision and
recall for each reference and candidate pair; we concatenate the $F_{1}$
score, precision and recall to the encoded unigram text features as inputs to
our base classifier. We show example visualizations of qa pairs turned into
features for lr:

  1. 1.

[CLS] Volume one [SEP] one [SEP] What volume of The Green Book discussed
democracy? [SEP] [0.67, 0.5, 1.0]

  2. 2.

[CLS] pentalogy Truth and Justice [SEP] Truth and Justice [SEP] What is the
name of the book written by Anton Hansen Tammsaare? [SEP] [0.86, 1.0, 0.75]

The text parts of the examples will be turned into td-idf unigrams and
concatenated with $F_{1}$ score, precision, and recall vectors. The classifier
automatically learns to capture semantic information in qa pairs while also
learns an optimal $F_{1}$ score threshold decision for individual qa examples.
Theoretically, a lr classifier does not have maximum input token length limit
like llms do (usually 512 tokens), we can encode arbitrary length of text
inputs including context as features, but this will not be in the scope of
this paper.

##  4 Resource Requirements and Stability

This section investigates how extensive the resources ae methods need. While
exact match is the least expensive, CFMatch requires far fewer resources than
llm-based approaches. Moreover, this helps show how important ae is: changing
the metric changes the ordering of the top models on leaderboards.

###  4.1 ood Evaluation Datasets

This section delves into the details of our selected ood test sets not
relevant to the training data. We use three benchmark datasets and one our
challenging datasets as ood datasets to validate that our classifier can
generalize well to follow our ae rules. We list the introduction of the
datasets below and examples in Table 2.

#### nq-open

The Natural Questions Open dataset is an adoptation of the original Natural
Questions dataset (Kwiatkowski et al., 2019, nq), which is based on real
questions from google search users.

#### Narrative qa

is a Reading Comprehension dataset that provides narrative texts and requiring
models to draw answers from the story context Kočiský et al. (2018).

#### Hotpot qa

HotpotQA is designed for the purpose of evaluating and enhancing machine
comprehension and question-answering (QA) capabilities of artificial
intelligence systems Yang et al. (2018), where each question is encompassed
with a long context.

###  4.2 Evaluation Method Evaluation

#### Candidate Answer Generations

qa models have quite diverse performance with different model sizes. We select
Flan-T5-xl (3B) Chung et al. (2022), LLaMA 2 (7B) Touvron et al. (2023),
gpt-3.5-turbo (black-box model) as our qa models and generate answers for the
three benchmark datasets. We concatenate the contexts with the questions if
available. 666https://openai.com

#### Answer Set Expansion

We use pywikibot Wikidata Contributors (2019) to crawl entity aliases for the
reference and candidate answers for all four datasets if they are available.
We save the original test sets and the expanded test datasets that contain a
list of reference and candidate answers with aliases.

#### Automated Evaluation Metric Results

We use em, $F_{1}$ matching, bert, improved bert (bert-Aug), logistic
regression trained on original ae training set (cfm), and on the ae-augmented
training (cfm-Aug) set as our qa evaluation methods and run them on the
original test set with candidate answers (O) and the expanded test sets with
aliases (E). Table 4 shows that all classifier-based methods have more relaxed
fit in evaluating the correctness of candidate answers and achieve higher
evaluation accuracies than standard evaluation methods when comparing on the
same dataset and qa models. Classifier-based methods also achieve more relaxed
fit and results in higher evaluation accuracy when they are trained on ae
augmented datasets that contain well-defined ae rules. An additional
exploration is that cfm with ae augmented dataset has a very comparable
evaluation accuracy as bert-based method.

#### Runtime and Model Size Comparison

Method | O Runtime | E Runtime | Disk Space
---|---|---|---
em | 2 s | 4 s | 0
$F_{1}$ | 2 s | 4 s | 0
cfm | 14 s | 25 s | 714 KB
cfm-Aug | 14 s | 25 s | 812 KB
bert (GPU) | 10 mins | 29 mins | 440 MB
bert (CPU) | 98 mins | 189 mins | 440 MB
Table 3: cfm-Aug can balance between efficiency and performance with few
increase in inference runtime and disk space than standard evaluation methods.
The O runtime calculates the average runtime on the original test sets for
10,210 (3610+3,300+3,300) examples on three generation models, and E runtime
is the average runtime on the expanded test sets.

The runtime, space an evaluation method or model needs, and the amount of
computation resource needed are also factors researchers need to consider
based on the amount of resources and time they have. We track the average
evaluation time for each metric to complete evaluation for the three datasets
(take the total time of evaluation of each qa model’s response divided by the
total number of models) shown in Table 3 and show that lr-based evaluation
methods most effectively balance the advantage and shortcomings of standard
evaluation metrics and llm-based evaluation methods.777Our CPU device is a AMD
EPYC 7402P 24-Core Processor and GPU is a single A6000.

#### Evaluation Method Intuition

There are flips of model accuracies on nq-open and nqa when using different
evaluation metrics shown in Table 4. We observe that gpt-3.5-turbo and LLaMA 2
7B both have worse evaluation accuracies than Flan-T5 xl (3B) when using the
standard evaluation methods, but when using classifier-based matching methods,
the order of model accuracies changes to gpt-3.5-turbo being the best,
followed by LLaMA 7B and Flan-T5 xl. The new model evaluation rankings
intuitively makes more sense to us since LLaMA (possibly gpt-3.5) are newer
released models and have more model parameters than Flan-T5 series.888We use
the newest gpt-3.5 model released on November 6th, 2023. LLaMA 2 7B is
released in Feb 2023, and Flan-T5 series is released in Dec 2022.

Dataset | em |  $F_{1}$(0.5) | cfm | cfm-Aug | bert | bert-Aug | qa Model
---|---|---|---|---|---|---|---
| O | E | O | E | O | E | O | E | O | E | O | E |
nq-open | 24.4 | 25.96 | 30.8 | 31.8 | 37.53 | 38.2 | 68.89 | 70.28 | 80.19 | 82.02 | 75.9 | 78.81 | gpt-3.5-turbo
4.44 | 4.32 | 6.34 | 6.76 | 11.16 | 11.5 | 51.55 | 54.04 | 82.80 | 84.35 | 69.39 | 73.6 | LLaMA 2 7B
10.75 | 11.55 | 14.99 | 17.92 | 23.38 | 24.57 | 28.84 | 30.72 | 23.07 | 28.67 | 28.45 | 37.67 | Flan-T5 xl
nqa | 17.03 | 17.09 | 46.97 | 47 | 51.48 | 51.58 | 97 | 97.15 | 97.36 | 97.39 | 93.79 | 94.06 | gpt-3.5-turbo
19.11 | 20.12 | 45.7 | 45.82 | 50.67 | 50.7 | 94.73 | 94.82 | 93.18 | 93.36 | 90.12 | 90.52 | LLaMA 2 7B
44.76 | 45.24 | 72.79 | 73.3 | 83.76 | 83.85 | 92.58 | 92.7 | 83.58 | 83.88 | 82.58 | 82.85 | Flan-T5 xl
Hotpot qa | 33 | 36.56 | 47.67 | 49.45 | 53.76 | 54.21 | 79.06 | 79.97 | 83.70 | 85.82 | 82.06 | 84.09 | gpt-3.5-turbo
20.27 | 23.36 | 31 | 33 | 36.36 | 36.76 | 67.67 | 68.76 | 74.18 | 76.67 | 75.24 | 78.45 | LLaMA 2 7B
60.42 | 64.82 | 76.85 | 79.33 | 83.82 | 84.33 | 86.3 | 86.94 | 82.94 | 85.55 | 82.36 | 85.06 | Flan-T5 xl
Jeopardy! | 0* | 1.89 | 15.9 | 18.87 | 33.15 | 33.96 | 43.13 | 44.47 | 25.88 | 28.3 | 43.13 | 43.4 | -
Table 4: A change of evaluation methods can flip the order of qa models! The
table shows the the automated qa evaluation accuracy in percentage($\%$) using
different evaluation methods for our four selected test datasets and three qa
models. Jeopardy! dataset already has candidate answers from Jeopardy!
players. The O column indicates evaluation accuracy on the original test sets
with generated candidate answers, and the E column indicates evaluation
accuracy on the test set with expanded reference and candidate answers sets.
gpt-3.5-turbo and LLaMA 2 7B have worse accuracies than Flan-T5 xl (3B) on the
nqa and Hotpot qa datasets when using em, but they become close to or better
than Flan-T5 xl after using classifier-based methods. In addition, all of the
evaluation methods have at least a slight increase in accuracy on the expanded
test sets. em COMPLETELY FAILS on the original expert answer challenge sets
with a 0$\%$ accuracy discussed in Section 5.

Figure 1: Standard evaluation methods fall behind classifier-based evaluation
methods on the original unexpanded test sets. The accuracies of cfm evaluation
trained on augmented ae dataset is closer to the accuracies of bert-based
evaluation methods than cfm trained on the un-augmented training set.

##  5 Comparing Expert Answer Equivalence Judgements

While the previous section discussed stability and resources, we have not yet
made the argument that our method is _better_ or aligns with human judgements.
We introduce another ood challenge test set that has a very different source
of answers with expert judgments and use it evaluate the effectiveness of
various qa evaluation methods.

####  Jeopardy!

is a popular American television quiz game show. This show is famous for its
distinctive formats of questions, where the questions covers diverse
categories, and many of them require a knowledgeable human with years of
expertise to be able to answer. An important note is that that there is a
pipeline of people who have worked for naqt have gone on to contribute to
Jeopardy! in one way or another, the most notable of which is Ken Jennings,
who became the first response adjudicator for naqt Tournaments (2019). Thus
the selected Jeopardy! challenge problems also conform to our ae rules, where
we will remove examples with rules we did not adopt (pronunciation, name
orders, etc). 999Jeopardy! calls questions clues rather than questions, and
the answers are responses. We rephrase the clues to be questions and response
to gold answers to conform to machine qa format.

#### Expert dataset details

We collect 371 examples from J!Archive, a fans website to record every season
of the Jeopardy! game with questions and answers
presented.101010https://www.j-archive.com/suggestcorrection.php?clue_id=353154
Each example in the dataset has a question, a reference answer, a candidate
response by different Jeopardy! players, and expert judgment of the candidate
answer. The dataset includes both difficult correct answers and difficult
incorrect answer. E.g., the difficult correct answers are the ones the expert
host ruled incorrect but was overruled by a panel. For example, given a
question Your surgeon could choose to take a look inside you with this type of
fiber optic instrument, the reference answer is laparoscope, but an endoscope
was given and ruled incorrect during the show that later reverted to correct,
verified by professionals and the general public in accordance with the
Jeopardy! answer acceptability rules. The dataset includes roughly 68.46%
correct examples and 31.54% incorrect examples, which we use to test the long-
tail automated qa evaluation methods.

###  5.1 Alignment with Expert Judgments and Analysis

We first compare judgment alignments of the evaluation methods with gold
expert labels, then manually annotate additional examples from the benchmark
test sets and categorize our improved classifier-based evaluation method.

#### Challenge set evaluation

The Jeopardy! dataset provides the most straightforward expert verification of
the effectiveness of qa evaluation methods since it is a challenge set already
contains expert human judgments of ae. This test set is challenging because
the initial judgments of the responses were incorrect during the show plays,
but they were reverted and corrected by professional judges or Jeopardy! fans
during or after the show. The decisions to make the right judgment for an
answer is even very challenging for experts since it requires a human with
deep and broad knowledge and understanding to a subject, culture, and the
question. An reverted example from the dataset is Question: I’ve seen the
promised land. I may not get there with you. but…we, as a people, will get to
the promised land… Gold Answer: Martin Luther King; Prompted Response: Martin
Luther King in "I had a dream", where the response was initially ruled correct
but later reverted because the talk in the question is not from I had a dream
speech. We show the percentage of evaluation decisions of each metric aligned
with human expert judgments in Table 5.

#### Classifier-based evaluation methods are generally more aligned with
expert judgments

Classifier-based methods except bert are all more aligned with expert
judgments than em and $F_{1}$, where cfm-Aug does the best on the original
test set and bert-Aug does the best on the expanded test set shown in Table 5.
Although em and $F_{1}$ are better than bert by blindly judges every candidate
response as incorrect without considering the context of the questions, the
fourth figure in Figure 1 shows that em completely fails on the Jeopardy!
challenge set by producing a 0 evaluation accuracy for expert responses.
During training stage of qa models, em would not be a good fit as a validation
metric if the model constantly generate correct answers that are not in the
reference list.

#### Expanding aliases can improve evaluation methods’ judgment alignment with
expert judgments

We see an improve of judgment percentage for all metrics after expanding the
answer sets in Table 5. After the reference and candidate answer set are
expanded, the candidate answers are found within the reference set, or
conversely, the reference answers are found within the candidate set. An
example is reference: Cupid and candidate: Eros, where the expanded candidate
answer set contains both terms.

Metric (%) | Jeopardy! O | Jeopardy! E
---|---|---
em | 31.54 | 32.35
$F_{1}$(0.5) | 29.65 | 31.54
cfm | 32.88 | 33.69
cfm-Aug | 36.39 | 37.74
bert | 27.49 | 29.92
bert-Aug | 34.23 | 39.89*
Table 5: cfm-Aug and bert-Aug(trained on augmented dataset) have the highest
alignment with human judgments, which are all better than standard evaluation
metrics and bert and cfm. The table shows the percentage of ae evaluation
methods aligned with expert human judgements. The Jeopardy! O is the original
test set with one reference and candidate answer for each example, and
Jeopardy! E is the expanded test set with possibly many reference and
candidate answers for eah example. All methods have a higher alignment with
human judgments after expanding the answer sets for Jeopardy.

###  5.2 Equivalence Error Analysis of classifier-based ae methods

In the following section, we conduct a systematic error analysis of selected
examples and analyze them in Three parts. The first part will select examples
of inconsistent judgments for between cfm and cfm-Aug , and bert and bert-Aug,
we then manually annotate those selected examples based on Table 1 rules and
calculate the number of examples that humans judge correct but em judges
incorrect. The second part show the improvement and examples of cfm and cfm-
Aug over bert and bert-Aug based on our annotation. The third part discusses
examples where bert-Aug and cfm-Aug do worse than bert and cfm. We have two
annotators manually annotated 250 examples randomly selected from the both
bert-based matching and cfm methods across all three datasets and all three
models.

####  em fails to detect correct answers mostly for additional correct
information in candidate answer

90% of the annotated samples are considered correct under human judgment are
not correct using em. Specifically, 52.4% of the candidate answers are correct
under our rule 4, where additional information is provided: Elizabeth Hartman
played the role of Selina D’Arcy is correct given question Who did an actress
from The Secret of NIMH play in A Patch of Blue? and reference Selina D’Arcy".

###  5.3 Robustness of ae-Improved Classifier

This section categorizes all the annotated examples where classifiers trained
with augmented set (cfm-Aug and bert-Aug) consistent with human judgments but
classifiers trained with original set (cfm and bert) not consistent (83.6% of
the selected examples). We provide the categories with example questions and
answers, with explanation of our human judgement according to our correctness
rules.

#### Specificity:

candidate answers should contain at least certain specific information to be
considered correct, any answers too general will be considered incorrect, and
this is related to our rule 3 and 4. About 90% of the examples put into this
category. Example: question: What flatten’s Aron’s hand and wrist? [SEP]
reference: Bergen [SEP] candidate: Bergen county, New Jersey. The candidate
answer is correct with extra detailed information. However, bert judges often
think that they are not equivalent. On the contrary, candidate answers should
not be too specific that excludes the generality of the reference answers.
bert methods also often make mistakes when the reference and candidate answers
are semantically close in the word embedding space. Additional examples are
apple and pear, king and queen, etc, where bert considers them as equivalence
but bert-Aug does not.

#### Semantic Similarity:

around 20.4% examples are in this category. Candidate answers that have the
same meaning as the reference and justifies the question are usually
considered correct, referring to our rule 5. Example question would be
question: How does Delacroix defend his show? [SEP] reference: By labeling it
satire [SEP] candidate: as being satirical. The reference and candidate
answers both indicate that Delacroix’s defense strategy for his show hinges on
characterizing it as satirical. They imply a similar underlying intent despite
the variation in their phrasing. and cfm considers equivalent.

#### Aliases, co-reference:

around 5.6% examples are in this category. One entity may have multiple names
or commonly known names as referred to our rule 1. Examples are Eric Arthur
Blair and George Orwell, where Eric Arthur Blair is a pen name of George
Orwell. ae judges also struggle with questions that requires co-referencing
understandings. An example question would be question: What job is Joanna
doing when she meets Peter for lunch? [SEP] reference: She is a waitress [SEP]
candidate: Joanna is working as a waitress. The she in the reference is
referring to Joanna and they should be equivalent.

###  5.4 Weaknesses of ae-Improved Classifier

We analyze example where ae-improved judges inconsistent with human judgments
and provide pitfalls of our improved ae methods (16.2% of the examples).

#### Irrelevant information:

around 12% of the examples are in this category. This is related to our rule
6, where the candidate answers is irrelevant or not an accurate description to
the question, which should be deemed incorrect. However, ae judges often
consider irrelevant candidate answers as correct on the narrative-qa dataset,
where it requires reasoning abilities to identify the correctness of candidate
answers. For example, Vietnam War and Colonel Charlie Beckwith are considered
equivalent by classifiers trained on the augmented dataset.

#### Specificity:

this is around 3% of our annotated examples. When a candidate answer is too
specific than a reference answer, where it excludes generality, it should be
considered incorrect. Example: question: how oxygenated blood returns to the
heart from the lungs [SEP] reference: pulmonary circulation [SEP] candidate:
pulmonary venous system, where ‘pulmonary circulation’ refers to the entire
process of blood flow through the lungs where blood is oxygenated. the
pulmonary venous system does not specifically indicate the process of
oxygenated blood returning to the heart, which is achieved through the
pulmonary veins entering the left atrium, and is too specific for a particular
part of the vein circulation system.

#### High token overlap:

In addition, there is around 1.2% examples where improved classifiers consider
correct for but indeed incorrect when the candidate and reference answers have
high word overlap. English professional footballer and England international
player are considered equivalent by cfm-Aug but not equivalent by cfm.

##  6 Related Work

#### Drawing on community expertise

There has been a recent trend of using data from well-defined communities
ethically Shilton (2016); Rogers et al. (2021), and the trivia community is no
different. While their data have been used for nlp tasks—from Watson’s tour-
de-force on Jeopardy! to TriviaQA—the data have been used without attending to
the norms of how the trivia community uses these data. As recognized by the
human qa community, we should listen to them if we adopt their evaluations
methods and data Jennings et al. (2021), which the their methods are used to
evaluate human qa, but our adopted is used to evaluate machines.

As Boyd-Graber and Börschinger (2020) argue, the goal of these competitions
are designed to determine who is better at answering questions than another,
and this is also the goal of modern leaderboards. Adopting these norms from
the Trivia community to evaluate human in the machine qa research community
can help judge answer equivalence, which is a fiendishly hard problem for
generative ai.

#### Machine qa evaluations

qa evaluations lie on a continuum—many are easy to evaluate via things like
exact match, particularly for short answers—but especially for long-form qa,
where there is a huge space of possible valid answers Xu et al. (2023).
Nevertheless, ae can also be difficult for short-form qa, where the judgment
requires the understanding of the contexts of the questions or huge
combination of valid output spaces Bulian et al. (2022). One of the reason is
that even short-form qa can be nlp-complete, where ae is especially
challenging for surface token matching. Specifically, the challenge of ae
connects many areas of nlp:

Coreference: equivalence examples are Her boyfriend and Kristy’s boyfriend
given question Who showed up to help Kristy escape when the house was
collapsing?.

Translation: generative models are also likely to provide answers not just in
one language. Gweilo and Gweilo (GuiLao) are equivalent given question What
Cantonese slang term can mean both "ghost man" and to refer to Westerners?
since GuiLao is the Chinese Alphabet answer.

Logical Reasoning: logical reasoning tests a system’s ability to interpret
human language in a structured manner, which is also related to ae. he was
trying to get back together with Caitlin and she was informed Dante’s planned
date with Caitlin are logically equivalent given the question Why does
Veronica break up with Dante?.

ae can be applied to wide areas of nlp tasks such as a machine translation
evaluation, entailment tasks, coreference resolution, etc. adopted rules from
the Trivia community and an improved and more clear evaluation method are
becoming more and more important than simple em with continual improvement on
transformer-base qa models Zhu et al. (2021). Various benchmarks are proposed
to evaluate the ability of qa models to answer questions Wang (2022). The
benchmarks lead to improvement of qa models, and improved qa models can now
answer questions much better by providing free-form answers or explanations
that are more closely aligned with human preferences. However, with the
continual development of machine qa models’ ability to answer questions, the
area to evaluation qa still falls behind, where most of research papers still
use the old standard evaluation metrics–em or $F_{1}$. Chen et al. (2019) and
Kamalloo et al. (2023a) both state there are pitfalls on the current standard
evaluation metrics, where qa is not merely extracting exact answers from
source texts, especially in the era of large generative models. Chen et al.
(2019) proposes bert-based evaluation is a promising research direction since
generative qa datasets becomes more abstract and free-form in nature. In
addition, Wang et al. (2023a) and Chen et al. (2019) both show using bert-
based evaluation methods without training (bert-Score) are not better or well-
suited for qa evaluation tasks than standard evaluation metrics, where many
datasets are also sensitive to bert-Score threshold settings, and the boundary
to determine the threshold is not well-defined for different datasets. Bulian
et al. (2022); Kamalloo et al. (2023a) validates fine-tuned bert-based llm on
human annotated ae datasets is can better evaluate qa than standard metrics
and closer to human judgments.

Although llms such as e.g., bert Devlin et al. (2018), roberta Liu et al.
(2019) have the advantage of being able to take the context of the questions
and answers into account to evaluate ae, which show better correlation with
human judgments than previous standard methods Zhang et al. (2019); Bulian et
al. (2022). However, Chen et al. (2019) shows the lack of generalization of
llm-based evaluation methods on different styles of qa datasets, particularly
for out of domain qa datasets. Kamalloo et al. (2023b) shows that generative
gpt-based evaluation methods are closer to human judgments of ae than bert-
based methods, but they often have problems when dealing with factual
information Ji et al. (2023). Improving qa evaluation is critical and
important not only in text-based qa, but also in multimodal qa. Mañas et al.
(2023) shows that llm-based evaluation methods can improve evaluation
alignment with humans for visual qa, which is an even more challenging task
than text-based qa with additional image-visual information. Yet the ability
of generative llm-based is not rigorously evaluated with clear definitions of
ae and answer correctness.

In addition, (Thalken et al., 2023) shows that fine-tuned bert on good expert
annotated data in a particular domain is better than llms, especially gpt-4 on
classification tasks. Current limitations of automated qa evaluation methods
hold us from building more creative and intelligent qa models, therefore
impeding the progress of solving qa Rodriguez and Boyd-Graber (2021). The
alignment of llm for human intent tasks has recently becoming more popular but
involves several pipelines, which data collection will be the first step in
the pipeline Wang et al. (2023b). Aligning llms with human judgments also
requires the data collection process. However, the construction of a ‘good’
dataset should draw well-defined rules and practices from the expert
community, where (Thalken et al., 2023) collected expert legal datasets from
the legal domain that successfully aligns bert to be better aligned with human
judgments than gpt-4 for their specific task. There have been dataset
collected under Bulian et al. (2022)’s definition of ae in short-form qa, but
we still see clear pitfalls of their fine-tuned llms to evaluate ae that do
not generalize well on ood datasets in Section 2.2. The pitfalls of current qa
evaluation methods advocates us to listen to the Trivia community norms and
adopt their evaluation rules, so we can both construct better ae datasets
under more fine-grained ae definitions that have been used and agreed by
experts to evaluate humans over the decades, and making better and more
efficient automated evaluation metrics that are more aligned with the ae
definitions, which we have shown that a 812 KB classifier is even competitive
with a 440 MB llm.

##  7 Conclusion

Automated qa evaluation is an important pipeline of developing more robust and
human favorable models. Better automated evaluation metrics drive for better
qa models, where better evaluation methods can provide qa models’ best
gradient feedback and save its best training state. A better evaluation method
during training time can be good for improving models’ downstream task
performance (accuracy, creativity) Si et al. (2021), where a too strict
evaluation method such as em provides too much negative feedback to models
during training time which results in lack of creativity of qa models and thus
guiding models to provide exact extracted answers. However, the current
pitfalls of qa evaluation methods (lack of correctness and ae guidelines)
still hold us from solving qa and building more human preferable qa models
that are not just trained to provide very specific answers to a question, but
also more diverse and personalized answers. The way to build and select a more
intelligent and creative qa model is by improving qa evaluation methods, where
machines are judged not merely by rigid automated evaluation methods, but with
more human-like automated evaluation methods. More human-like automated
evaluation methods can scale up training, while not requiring humans to
individually look at all the produced answers. By writing up a basic framework
derived from qa human experts and adopt it to machine qa paradigm, we are able
to enhance and generalize current automated qa evaluation methods that are
more aligned with human judgments. In addition, our ae framework also improve
human annotators’ experience in judging the correctness of answers. Future
work can explore automated qa evaluations that can also handle and take
context passages of the question into account and develop a more advanced
evaluation framework for long-form qa.

##  8 Limitation

Our classifier-based evaluation methods trained with ae data containing our
framework examples can better justify equivalence of candidate answers given a
reference answer than current popular em, $F_{1}$ and bert-based models.
However, our new method still possess several challenges and limitations. Our
models are not trained to take contexts of the question into account, where if
a question has a context paragraph, our model can only the question, reference
answer, and candidate answer to evaluate equivalence, which the effectiveness
of our approach is still unexplored for qa with contexts. Another limitation
of the lack of diversity of qa models we used in our experiments, where
different qa models have diverse range of style to answer questions. We also
have a lack of enough annotated examples for our tasks, which we are under the
process of hiring more annotators. In addition, our definition of ae has not
taken subjectivity, demographic diversity, bias, and cultural diversity into
account, with cases where the judgments of correctness can be different for
people with differnt backgrounds.

## References

  * Banerjee and Lavie (2005) Satanjeev Banerjee and Alon Lavie. 2005.  METEOR: An automatic metric for MT evaluation with improved correlation with human judgments.  In _Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization_ , pages 65–72, Ann Arbor, Michigan. Association for Computational Linguistics.
  * Boyd-Graber and Börschinger (2020) Jordan Boyd-Graber and Benjamin Börschinger. 2020.  What question answering can learn from trivia nerds.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7422–7435, Online. Association for Computational Linguistics.
  * Bubeck et al. (2023) Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. 2023.  Sparks of artificial general intelligence: Early experiments with gpt-4.  _arXiv preprint arXiv:2303.12712_.
  * Bulian et al. (2022) Jannis Bulian, Christian Buck, Wojciech Gajewski, Benjamin Boerschinger, and Tal Schuster. 2022.  Tomayto, tomahto. beyond token-level answer equivalence for question answering evaluation.  _arXiv preprint arXiv:2202.07654_.
  * Carberry (2019) Matt Carberry. 2019.  Jeopardy! casebook.  http://jeopardy.mattcarberry.com/casebook.html.  Accessed: 2024-01-12.
  * Chen et al. (2019) Anthony Chen, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019.  Evaluating question answering evaluation.  In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_ , pages 119–124, Hong Kong, China. Association for Computational Linguistics.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Jennings et al. (2021) Ken Jennings, Richard Garfiled, and Sierra. 2021.  Can ken jennings answer 100 questions in 10 minutes? | jeopardy trivia challenge.  https://www.youtube.com/watch?v=VideoID.  Accessed: Jan 14, 2024.
  * Ji et al. (2023) Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. 2023.  Survey of hallucination in natural language generation.  _ACM Computing Surveys_ , 55(12):1–38.
  * Jones (2021) Karen Spärck Jones. 2021.  A statistical interpretation of term specificity and its application in retrieval.  _J. Documentation_ , 60:493–502.
  * Kamalloo et al. (2023a) Ehsan Kamalloo, Nouha Dziri, Charles Clarke, and Davood Rafiei. 2023a.  Evaluating open-domain question answering in the era of large language models.  In _Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 5591–5606, Toronto, Canada. Association for Computational Linguistics.
  * Kamalloo et al. (2023b) Ehsan Kamalloo, Nouha Dziri, Charles LA Clarke, and Davood Rafiei. 2023b.  Evaluating open-domain question answering in the era of large language models.  _arXiv preprint arXiv:2305.06984_.
  * Karpukhin et al. (2020) Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020.  Dense passage retrieval for open-domain question answering.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 6769–6781, Online. Association for Computational Linguistics.
  * Kočiský et al. (2018) Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018.  The NarrativeQA reading comprehension challenge.  _Transactions of the Association for Computational Linguistics_ , 6:317–328.
  * Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019.  Natural questions: A benchmark for question answering research.  _Transactions of the Association for Computational Linguistics_ , 7:452–466.
  * Lin (2004) Chin-Yew Lin. 2004.  ROUGE: A package for automatic evaluation of summaries.  In _Text Summarization Branches Out_ , pages 74–81, Barcelona, Spain. Association for Computational Linguistics.
  * Lin et al. (2019) Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. 2019.  Reasoning over paragraph effects in situations.  In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_ , pages 58–62, Hong Kong, China. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Mañas et al. (2023) Oscar Mañas, Benno Krojer, and Aishwarya Agrawal. 2023.  Improving automatic vqa evaluation using large language models.  _arXiv preprint arXiv:2310.02567_.
  * McNear (2020) Claire McNear. 2020.  _Answers in the Form of Questions_.  Twelve.
  * Min et al. (2021) Sewon Min, Jordan Boyd-Graber, Chris Alberti, Danqi Chen, Eunsol Choi, Michael Collins, Kelvin Guu, Hannaneh Hajishirzi, Kenton Lee, Jennimaria Palomaki, Colin Raffel, Adam Roberts, Tom Kwiatkowski, Patrick Lewis, Yuxiang Wu, Heinrich Küttler, Linqing Liu, Pasquale Minervini, Pontus Stenetorp, Sebastian Riedel, Sohee Yang, Minjoon Seo, Gautier Izacard, Fabio Petroni, Lucas Hosseini, Nicola De Cao, Edouard Grave, Ikuya Yamada, Sonse Shimaoka, Masatoshi Suzuki, Shumpei Miyawaki, Shun Sato, Ryo Takahashi, Jun Suzuki, Martin Fajcik, Martin Docekal, Karel Ondrej, Pavel Smrz, Hao Cheng, Yelong Shen, Xiaodong Liu, Pengcheng He, Weizhu Chen, Jianfeng Gao, Barlas Oguz, Xilun Chen, Vladimir Karpukhin, Stan Peshterliev, Dmytro Okhonko, Michael Schlichtkrull, Sonal Gupta, Yashar Mehdad, and Wen-tau Yih. 2021.  Neurips 2020 efficientqa competition: Systems, analyses and lessons learned.  In _Proceedings of the NeurIPS 2020 Competition and Demonstration Track_ , volume 133 of _Proceedings of Machine Learning Research_ , pages 86–111. PMLR.
  * National Academic Quiz Tournaments, LLC (2024) National Academic Quiz Tournaments, LLC. 2024.  Correctness guidelines.  Accessed: 2024-01-12.
  * Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  Squad: 100,000+ questions for machine comprehension of text.  _arXiv preprint arXiv:1606.05250_.
  * Ren et al. (2023) Jie Ren, Yao Zhao, Tu Vu, Peter J. Liu, and Balaji Lakshminarayanan. 2023.  Self-evaluation improves selective generation in large language models.
  * Rodriguez and Boyd-Graber (2021) Pedro Rodriguez and Jordan Boyd-Graber. 2021.  Evaluation paradigms in question answering.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 9630–9642, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Rogers et al. (2021) Anna Rogers, Timothy Baldwin, and Kobi Leins. 2021.  ‘just what do you think you’re doing, dave?’ a checklist for responsible data use in NLP.  In _Findings of the Association for Computational Linguistics: EMNLP 2021_ , pages 4821–4833, Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Shilton (2016) Katie Shilton. 2016.  Emerging ethics norms in social media research.  _Big Data Ethics_.
  * Si et al. (2021) Chenglei Si, Chen Zhao, and Jordan Boyd-Graber. 2021.  What’s in a name? answer equivalence for open-domain question answering.
  * Stone and Yohn (1992) Joseph Stone and Tim Yohn. 1992.  _Prime Time and Misdemeanors: Investigating the 1950s TV Quiz Scandal: A D.A.’s Account_.  Rutgers University Press, New Brunswick, N.J.
  * Thalken et al. (2023) Rosamond Thalken, Edward H Stiglitz, David Mimno, and Matthew Wilkens. 2023.  Modeling legal reasoning: Lm annotation at the edge of human agreement.  _arXiv preprint arXiv:2310.18440_.
  * Tournaments (2019) National Academic Quiz Tournaments. 2019.  Why does ken jennings play quiz bowl?  Online Video.  URL: https://www.youtube.com/watch?v=Y5ku181Zm8I.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.
  * Voorhees and Tice (2000) Ellen M. Voorhees and Dawn M. Tice. 2000.  The TREC-8 question answering track.  In _Proceedings of the Second International Conference on Language Resources and Evaluation (LREC’00)_ , Athens, Greece. European Language Resources Association (ELRA).
  * Wang et al. (2023a) Cunxiang Wang, Sirui Cheng, Qipeng Guo, Yuanhao Yue, Bowen Ding, Zhikun Xu, Yidong Wang, Xiangkun Hu, Zheng Zhang, and Yue Zhang. 2023a.  Evaluating open-qa evaluation.
  * Wang et al. (2023b) Yufei Wang, Wanjun Zhong, Liangyou Li, Fei Mi, Xingshan Zeng, Wenyong Huang, Lifeng Shang, Xin Jiang, and Qun Liu. 2023b.  Aligning large language models with human: A survey.
  * Wang (2022) Zhen Wang. 2022.  Modern question answering datasets and benchmarks: A survey.  _arXiv preprint arXiv:2206.15030_.
  * Wikidata Contributors (2019) Wikidata Contributors. 2019.  Pywikibot - python 3 tutorial/data harvest.  Accessed: 2024-01-12.
  * Xu et al. (2023) Fangyuan Xu, Yixiao Song, Mohit Iyyer, and Eunsol Choi. 2023.  A critical evaluation of evaluations for long-form question answering.  _ArXiv_ , abs/2305.18201.
  * Yang et al. (2018) Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018.  HotpotQA: A dataset for diverse, explainable multi-hop question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.
  * Zhang et al. (2019) Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi. 2019.  Bertscore: Evaluating text generation with bert.  _ArXiv_ , abs/1904.09675.
  * Zhu et al. (2021) Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021.  Retrieving and reading: A comprehensive survey on open-domain question answering.  _arXiv preprint arXiv:2101.00774_.

##  9 ae and Correctness Guideline Examples

Figure 2: Classifier-based evaluation methods with augmented ae training set
are more stable than standard evaluation methods on different models. In
addition, lr-based methods trained on our augmented dataset is close to or
even better than bert-based methods’ judgment alignment with our fine-tuned ae
gpt-3.5-turbo with constantly at least $80\%$ of alignment. On the other hand,
standard evaluation methods are unstable with the change of qa models or
datasets. The error bars are the normalized variances of alignment percentage
for the three qa models under the same dataset.

We include more typical qa example pairs in Table 6 with judgments and
explainations aligning with our rules in table 1.

##  10 Jeopardy! Data Collection Pipeline

The J!Archive is a fans website to record every season of the Jeopardy! game
with questions answers
presented.111111https://www.j-archive.com/suggestcorrection.php?clue_id=353154
The website also includes questions and answers that are answered by Jeopardy!
players that are originally judged correct/incorrect, but with later revised
decisions, where those overruled answers are carefully verified and discussed
by audiences or experts. We collected a total of 371 such examples from
J!Archive use them as the challenge set.

##  11 Comparing Alignment with gpt-3.5 Judgements

One critique is that we cannot validate our hypothesis that a change of
accuracy of the automated evaluation method improves its actual ability to
detect ae Si et al. (2021). We fine-tune -gpt–turbo-3.5 on our augmented
training set as an approximate human judge and compare the percentage of
alignment between gpt-3.5-turbo and the evaluation metrics discussed in our
paper.

#### Alignment with fine-tuned gpt

Kamalloo et al. (2023b); Wang et al. (2023a) both evaluated that llm-based
evaluators are closest to human judgments and can serve as a compliment as
human judge but have the disadvantage of being sensitive to prompts and their
costs and latency. Although we are not considering gpt-3.5-turbo as our gold
standard judges here, we compare how our small classifier (<1 MB) can
generalize and align with current llm judges, which is more efficient and less
cost. We first fine-tune gpt-3.5-turbo with our augmented ae dataset with
prompts including the rule descriptions, qa pairs, and judgment of
correctness. Then we use fine-tuned gpt to provide judgments to the three
original un-expanded benchmark test sets and compare the alignment counts of
each evaluation method with a fine-tuned gpt model in Figure 2, which shows
the alignment percentage variances across models for the same dataset, and the
alignment percentage of each evaluation method. We calculate the alignment
variance for each metric based on the percentage of alignment with
gpt-3.5-Turbo on the three qa models under the same data set and normalize it
by dividing 100. Figure 2 shows that classifier-based evaluation methods
trained on augmented ae data overall have smaller judgment variances and
higher judgement alignment with current fine-tuned state-of-the-art llm.

Question |  References |  Candidate |  Judgment
---|---|---|---
Where does Pam find Jim is dead? |  In a bathtub while in Paris or In a bathtub in Paris |  Paris, France |  Incorrect based on rule 4, specificity and details.
What flatten’s Aron’s hand and wrist? |  A boulder |  A rock |  Incorrect based on rule 5, a boulder refers to a large rock but a rock can be any size.
Born on 6 March 1937, this man was also a citizen of the United Kingdom. |  Alistair Grant |  Sir Matthew Alistair Grant |  Correct based on rule 1: common aliases.
What is the tissue type of the inner most layer of cells? |  Epithelium |  Epithelial tissue |  Correct under context based on rule 6.
In which 2017 film did this South Korean actor of "Dream High" (2011) and the 2012 South Korean television drama series featuring Han Ga-in, Jung Il-woo and Kim Min-seo appeared in 2017? |  Real |  The South Korean actor appeared in the 2017 film "Real" |  Correct based on rule 5 with extra accurate details.
Are Nikolaschka and White Russian made of beer? |  No |  No, Nikolaschka and White Russian are not made with beer |  Correct under rule 5.
Reggaetón Lento is a song by the boy band formed on which date? |  December 13, 2015 |  December 13, 2015 |  Incorrect based on rule 2 and 4. Date and numerical should be exact.
What percentage is 50 grams of a 200 gram total weight? |  $25\%$ |  $25.01\%$ |  Incorrect based on rule 2.
How are Tessa and Tito connected? |  They are secretly married and have two children or They are married |  Megan |  Incorrect under rule 7, completely irrelevant answers.
What is the capital of France? |  The capital of France is Paris |  Paris is the largest city in France |  Incorrect based on rule 6: amount of token overlap is high with $F_{1}$ score 0.62 but referring to different information.
Your surgeon could choose to take a look inside you with this type of fiber optic instrument |  laparoscope |  an endoscope |  Correct under rule 1 and 4: "endoscope" encompasses a variety of instruments used for viewing different internal parts of the body, not limited to the abdominal or pelvic area.
I’ve seen the promised land. I may not get there with you. but…we, as a people, will get to the promised land |  Martin Luther King in "I had a dream" |  Martin Luther King |  Incorrect under rule 5, the question is not from the speech.
The livers of geese that are overfed underexercised are used to make this pate substance |  foie gras |  foie de gras |  Correct under rule 6 because foie de gras means liver of fat, and convey the same semantic equivalence under the context of the question.
Sarah bought a dress and a hat. Which item was more expensive? |  The dress cost more than the hat |  The item Sarah spent less on was the hat |  Correct under rule 6 with semantic equivalence
Is Wall Street doing well this year? |  The financial markets have seen a significant upturn |  he stock market has been performing positively |  Correct under rule 1 and 6. The Wall Street is an metonymy of the stock market under the context of the question.
Table 6: We list out more relevant ae pairs with judgments under our revised
rules. Some of the candidate responses are manually written, some of them are
from Jeopardy!, and some of them are generated by various qa models such as
Flan-t5.