# Is Your Large Language Model Knowledgeable or a Choices-Only Cheater?

Nishant Balepur
University of Maryland
<EMAIL_ADDRESS>
&Rachel Rudinger
University of Maryland
<EMAIL_ADDRESS>

###### Abstract

Recent work shows that large language models (LLMs) can answer multiple-choice
questions using only the choices, but does this mean that MCQA leaderboard
rankings of LLMs are largely influenced by abilities in choices-only settings?
To answer this, we use a contrast set that probes if LLMs over-rely on
choices-only shortcuts in MCQA. While previous works build contrast sets via
expensive human annotations or model-generated data which can be biased, we
employ graph mining to extract contrast sets from existing MCQA datasets. We
use our method on UnifiedQA, a group of six commonsense reasoning datasets
with high choices-only accuracy, to build an 820-question contrast set. After
validating our contrast set, we test 12 LLMs, finding that these models do not
exhibit reliance on choice-only shortcuts when given both the question and
choices. Thus, despite the susceptibility of MCQA to high choices-only
accuracy, we argue that LLMs are not obtaining high ranks on MCQA leaderboards
just due to their ability to exploit choices-only shortcuts.111Our code is
available at https://github.com/nbalepur/mcqa-artifacts

##  1 Introduction

Multiple-choice question answering (MCQA) is a popular task to test the
knowledge of large language models (LLMs) Robinson and Wingate (2023).
However, recent work shows that LLMs surpass majority class baselines in
choices-only settings—when no question and just the choices are given in a
prompt Balepur et al. (2024). This raises the question: Do models obtain high
ranks in MCQA leaderboards due to their pretraining knowledge or their ability
to exploit choices-only shortcuts? Resolving this query is key to ensure that
MCQA leaderboards reliably rank the knowledge of LLMs.

To answer this question, we use a variation of _contrast sets_ —small datasets
that test if models “pay attention” to perturbed attributes that should alter
the model’s decision Levesque et al. (2012); Gardner et al. (2020). For our
purposes, we need a contrast set containing pairs of MC entries with identical
answer choices, but varied questions that lead to distinct answers. For
example, in Figure 1 (bottom), the MC entries $d_{i}$ and $d_{j}$ have the
same choices of “the sun” and “the rain”, but $d_{i}$ has a question that
answered by “the sun”, and similarly for $d_{j}$. This design ensures that
LLMs relying only on shortcuts or patterns in the choices, while ignoring
questions, can perform no better than random chance. Thus, if a model ranks
highly on an MCQA dataset but largely drops in rank on a contrast set based on
this dataset, it would reveal that this model obtains a high rank on the
original dataset primarily by employing choices-only shortcuts.

Figure 1: Example of a contrast MCQA evaluation set.

Contrast sets are usually built through manual annotation efforts Gardner et
al. (2020); Srikanth and Rudinger (2022), as model-generated data can be
biased. However, writing MC questions with high-quality distractors is
difficult even for experts Gierl et al. (2017). Further, rewritten questions
can exhibit distributional differences from the original questions, altering
the difficulty of the MCQA task.

To address this issue, we cast the creation of contrast sets for MCQA datasets
to a graph mining task. We treat each MC entry $d_{i}$ in the dataset as a
vertex in an undirected graph, and draw edges between entries $d_{i}$ and
$d_{j}$ if the gold answer from $d_{i}$ is semantically equivalent to a
distractor in $d_{j}$, and vice versa. For instance, in Figure 1 (top), the
gold answer of “rain” in $d_{i}$ is semantically similar to the distractor of
“rainfall” in $d_{j}$ and vice versa, so we draw an edge between $d_{i}$ and
$d_{j}$. Thus, an edge ($d_{i}$, $d_{j}$) means that the gold answers in
$d_{i}$ and $d_{j}$ can form a set of choices, with questions in $d_{i}$ and
$d_{j}$ leading to distinct answers in said choices, mirroring the desired
format of our contrast set. We find the maximum matching of this graph to
obtain the largest contrast set of distinct MCQA questions derived from the
initial dataset. This method overcomes the burden of writing contrast sets,
while only minimally using models for semantic equivalence, reducing the risk
of model-generated biases.

We use our approach to build an 820-question contrast set from six commonsense
MCQA datasets from the UnifiedQA collection Khashabi et al. (2020). We first
ask three annotators to assess our contrast set, finding that it has questions
with plausible distractors (section 4.1). This finding suggests that we have
built a high-quality MCQA contrast set.

After verifying the quality of our contrast set, we test 12 LLMs Touvron et
al. (2023); Penedo et al. (2023); Jiang et al. (2023); Young et al. (2024);
Team et al. (2024) on the UnifiedQA evaluation set and its mined contrast set
(section 4.2). Our LLMs surpass random guessing using just the choices on the
original evaluation set, aligning with prior work. Balepur et al. (2024).
However, when prompted with both the question _and_ choices, LLM accuracy
rankings between the initial evaluation set and contrast set are highly
consistent, with Kendall’s $\tau$ near $0.9$.

Since no LLM rank drops markedly, we claim that our tested LLMs are not
ranking highly on MCQA leaderboards solely due to their ability to exploit
choices-only shortcuts. Thus, despite the susceptibility of MCQA to high
choices-only accuracy, the task may still reliably rank LLM knowledge. As a
result, we recommend that future works continue to explore the behavior of
LLMs in choices-only settings to help explain how LLMs can adeptly perform
MCQA without the question.

##  2 Automatic Contrast Set Creation

We assume we are given an MCQA dataset $\mathcal{D}$ with data entries
$d_{i}=(q_{i},\mathcal{C}_{i},a_{i})$, where $q_{i}$ is a question,
$\mathcal{C}_{i}$ is a list of choices, and $a_{i}\in\mathcal{C}_{i}$ is the
gold answer. Our goal is to build a contrast set $\mathcal{D}_{contr}$ from
$\mathcal{D}$ to probe if LLMs rely on choice-only shortcuts in MCQA.
Typically, humans manually create contrast sets Srikanth and Rudinger (2022);
Gardner et al. (2020), as model-generated data can be biased Yu et al. (2024).
However, writing MCQA problems is challenging even for experts Offerijns et
al. (2020); Gierl et al. (2017). Thus, we seek to automatically mine a
contrast set $\mathcal{D}_{contr}$ from the original dataset $\mathcal{D}$
without model-generated data.

To automatic build contrast sets, we need MCQA entry pairs in the style of
Figure 1 (bottom)—pairs with the same choices
$\mathcal{C}^{\prime}=\\{a_{i},a_{j}\\}$, but questions $q_{i}$ and $q_{j}$
leading to distinct answers $a_{i}$ and $a_{j}$ in $\mathcal{C}^{\prime}$,
respectively. We define this format as an entry pair
$p_{ij}=\langle(q_{i},\\{a_{i},a_{j}\\},a_{i}),(q_{j},\\{a_{i},a_{j}\\},a_{j})\rangle$.
Thus, creating the largest possible $\mathcal{D}_{contr}$ with distinct
questions is equivalent to finding the maximum set of unique entry pairs
$p_{ij}$ in $\mathcal{D}$. In the next sections, we outline our graph-based
approach to mine entry pairs from the original dataset $\mathcal{D}$ to form
$\mathcal{D}_{contr}$.

###  2.1 Graph Representation

While a simple strategy to find an entry pair $p_{ij}$ is to sample two
entries
$(q_{i},\mathcal{C}_{i},a_{i}),(q_{j},\mathcal{C}_{j},a_{j})\in\mathcal{D}$
and let $\mathcal{C}^{\prime}=\\{a_{i},a_{j}\\}$, this may result in low-
quality questions, as there is no constraint that $a_{x}$ and $a_{y}$ form a
plausible set of choices (section 4.1). For instance, if $a_{i}$ is a ratio
and $a_{j}$ is an integer, choices $\\{a_{i},a_{j}\\}$ are implausible and
result in a low-quality question. To address this, we intuit that the original
dataset $\mathcal{D}$ reveals if two answers $a_{i}$ and $a_{j}$ are plausible
distractors for each other. For answers $a_{i}\in\mathcal{C}_{x}$ and
$a_{j}\in\mathcal{C}_{y}$, if $a_{i}$ is semantically equivalent to a
distractor $c\in\mathcal{C}_{j}\setminus\\{a_{j}\\}$ and likewise for $a_{j}$
and $\mathcal{C}_{i}$, the set of choices
$\mathcal{C}^{\prime}=\\{a_{i},a_{j}\\}$ will be plausible.

To execute this idea, we represent the dataset $\mathcal{D}$ as an undirected
graph $\mathcal{G}$. Each entry $d_{i}\in\mathcal{D}$ is a vertex for
$\mathcal{G}$. We draw an edge between entries $d_{i}$ and $d_{j}$ if the gold
answer $a_{i}$ is semantically equivalent to a distractor
$c\in\mathcal{C}_{j}\setminus\\{a_{j}\\}$ and vice versa, meaning that the
choices $a_{i}$ and $a_{j}$ can form a plausible set of choices based on
$\mathcal{D}$. We create edges with semantic equivalence over exact match to
consider choices with minor differences, like “rain” and “rainfall” in Figure
1, increasing the candidate size of our contrast set. We compute semantic
similarity via NLI-based embeddings Conneau et al. (2017) and set a strict
cosine similarity threshold of 0.85 to determine semantic equivalence.

###  2.2 Mining Entry Pairs

We now mine entry pairs from the graph $\mathcal{G}$ to build a contrast set
$\mathcal{D}_{contr}$. For any edge $(d_{i},d_{j})$ in $\mathcal{G}$, we know
that $a_{i}$ and $a_{j}$ form a set of plausible choices. Thus, when
$\mathcal{C}^{\prime}=\\{a_{i},a_{j}\\}$, entries
$(q_{i},\mathcal{C}^{\prime},a_{i})$ and $(q_{j},\mathcal{C}^{\prime},a_{j})$
form an entry pair $p_{ij}$. Using this idea, we build $\mathcal{D}_{contr}$
by finding the maximum matching Boppana and Halldórsson (1992) of
$\mathcal{G}$, which gives the largest set of edges in $\mathcal{G}$ where no
two edges are adjacent. Each edge in the maximum matching form an entry pair
for the contrast set and since no edges are adjacent, each entry pair contains
two unique questions. This results in the largest possible contrast set
without duplicate questions.

##  3 Experimental Setup

Figure 2: Distractor plausibility ratings across methods.

###  3.1 A Contrast Set for UnifiedQA

The purpose of our contrast set $\mathcal{D}_{contr}$ is to study whether high
choices-only accuracy influences the ranking of LLMs on MCQA leaderboards.
Thus, $\mathcal{D}_{contr}$ must be based on a dataset with high accuracy in
choices-only settings. The two datasets from Balepur et al. (2024) with the
highest choices-only accuracy are commonsense datasets Clark et al. (2018);
Zellers et al. (2019). Thus, we derive $\mathcal{D}_{contr}$ from an MCQA
split of UnifiedQA Khashabi et al. (2020), which has 7611 questions from six
commonsense datasets: ARC Clark et al. (2018), OpenBookQA Mihaylov et al.
(2018), CommonsenseQA Talmor et al. (2019), QASC Khot et al. (2020), PIQA Bisk
et al. (2020), and SIQA Sap et al. (2019). Using our graph mining algorithm
(section 2), we build an 820-question contrast set. This size aligns with
contrast set sizes in prior works, ranging from 600 to 1000 Srikanth and
Rudinger (2022); Gardner et al. (2020).

###  3.2 Prompt Design

LLMs are only known to best random guessing with just choices in few-shot
prompts. Thus, we follow the few-shot format of Balepur et al. (2024) and use
a full prompt (3.2) to assess LLMs when they can see both the questions and
choices, and a choices-only (3.2) prompt for just the choices:

Prompt 3.1: Full Prompt Question: $q$ Choices:$\backslash$n(A) $c_{a}$
$\backslash$n(B) $c_{b}$ $\backslash$n(C) $c_{c}$ $\backslash$n(D) $c_{d}$
Answer: $a$

Prompt 3.2: Choices-Only Prompt Choices:$\backslash$n(A) $c_{a}$
$\backslash$n(B) $c_{b}$ $\backslash$n(C) $c_{c}$ $\backslash$n(D) $c_{d}$
Answer: $a$

In the boxes above, the non-highlighted text represents the model input, while
the highlighted text represents the model generation. In the few-shot prompts,
exemplars follow the same format shown in the prompt box with the highlighted
text replaced by the ground truth (Example in Appendix A.2).

##  4 Results

###  4.1 Qualitative Analysis

To assess the quality of the contrast set produced by our graph mining
algorithm, we ask three Ph.D. students in computer science to compare 50 of
our questions versus a baseline that randomly picks entry pairs (details in
Appendix B.1). These methods only differ by distractors, so following Gierl et
al. (2017), we ask annotators to compare the plausibility of the two
distractors as a proxy for question quality. All three annotators find that
our method has significantly more plausible distractors than the baseline
(Figure 2), suggesting that our extracted contrast set from UnifiedQA is high-
quality.

Figure 3: Accuracy of twelve LLMs on the UnifiedQA evaluation set (left)
versus its contrast set (right), sorted by full prompt accuracy. We show
5-shot (top) and 10-shot (bottom) prompts, with 3-shot prompts in Appendix
B.2.

###  4.2 Are LLMs Knowledgeable or Choices-only Cheaters?

Following our quality checks, we use our contrast set to study if high
choices-only accuracy influences the ranking of LLMs when questions and
choices are given. We assess 6 LLM families on the UnifiedQA evaluation set
and our contrast set: LLaMA-2 Touvron et al. (2023), Falcon Penedo et al.
(2023), Mistral Jiang et al. (2023), Mixtral Jiang et al. (2024), Gemma Team
et al. (2024), and Yi Young et al. (2024). We use 5-shot and 10-shot full and
choices-only prompts (Prompts 3.2, 3.2). Appendix A.3 has more prompting
details.

On the UnifiedQA evaluation set, our LLMs often surpass random guessing with
choices-only prompts (Figure 3, left), aligning with prior work Balepur et al.
(2024). Further, LLMs with higher ranks on the UnifiedQA evaluation set using
the full prompt tend to have higher accuracy when using the choices-only
prompt, suggesting a correlation between an LLM’s MCQA leaderboard rank and
its ability to exploit choices-only shortcuts. Simply subtracting these values
cannot quantify how an LLM performs in MCQA without choices-only shortcuts,
since if an LLM can answer a MC question without the question, it does not
imply the model is ignoring the question when it has access to the question
Srikanth and Rudinger (2022).

Thus, to better quantify if LLMs are obtaining high ranks on UnifiedQA due to
their ability to exploit choices-only shortcuts, we compare model ranks on the
original UnifiedQA evaluation set to its contrast set. We note that if a
certain LLM relied on choice-only shortcuts substantially more than other
models, its contrast set accuracy ranking would largely drop compared to its
evaluation set accuracy ranking, as it would be penalized for ignoring the
question. However, in the UnifiedQA evaluation set and its contrast set, model
rankings of full prompt accuracy are consistent; the 5-shot and 10-shot
rankings have Kendall’s $\tau$ of 0.88 and 0.91, indicating high consistency.
Thus, we claim that the MCQA rankings of our LLMs on UnifiedQA do not
primarily stem from their ability to perform well in choices-only settings,
and none of our models are considered “choices-only cheaters.”

We find that if an LLM succeeds with choices-only prompts, it does not imply
that this model’s performance in MCQA solely stems from its choices-only
abilities. As a result, we believe that despite high choices-only accuracy,
MCQA may still be a reliable task to rank the knowledge of LLMs. Further, our
results stress the need for more work in explaining how high choices-only
accuracy occurs. We believe such efforts are crucial to better interpret LLM
knowledge and decision-making.

##  5 Related Work

Contrast Sets: Contrast sets Gardner et al. (2020) or counterfactual
augmentations Kaushik et al. (2020); Srikanth et al. (2024), are datasets that
probe if models “pay attention” to desired attributes Elazar et al. (2023).
This technique has been applied to many tasks, including natural language
inference Glockner et al. (2018); Ribeiro et al. (2020), story generation Qin
et al. (2019), and ethical judgements Hendrycks et al. (2021a). While these
datasets are often created manually, many works use generation models Wu et
al. (2021); Fryer et al. (2022) to create contrast sets. Instead, we are the
first to employ graph mining to build contrast sets, limiting the potential
for model-generated biases.

MCQA Evaluation: MCQA is a popular testbed not only for benchmarking LLMs
Beeching et al. (2023); Liang et al. (2023), but also for interpreting LLM
decision-making. Previous works use MCQA to study prompt sensitivity
Pezeshkpour and Hruschka (2023); Zheng et al. (2024), logical robustness
Balepur et al. (2023), and recently, the ability to perform MCQA without using
the question Balepur et al. (2024). We give more insights into this last
phenomenon by probing if LLMs ignore the question even when it is given in the
prompt.

##  6 Conclusion

We find that while LLMs can perform well in MCQA without access to the
question, it does not mean that model rankings on MCQA leaderboards are
largely influenced by this ability. This result supports the claim that MCQA
can rank the knowledge and ability of LLMs to reason over both questions and
choices. Further, we are aligned with recent work that suggests that high
choices-only accuracy does not necessarily imply that models are incapable of
true reasoning or comprehension, so we hope future works continue to explore
what strategies LLMs may employ to perform well in choices-only settings. Our
application of graph mining to MCQA sheds light on one way to do this—the
automatic construction of contrast sets—and we hope similar methods can be
applied to other tasks to enhance LLM interpretability.

##  7 Limitations

One limitation lies in the application of our graph mining algorithm solely to
the UnifiedQA dataset collection. We choose UnifiedQA for its tendency to
elicit high accuracy with choices-only prompts, as commonsense reasoning MCQA
datasets have shown to be susceptible to this phenomenon. Since our results
show that LLMs rankings are highly consistent on this dataset prone to high
choices-only accuracy, we believe these findings will hold for other MCQA
datasets like MMLU Hendrycks et al. (2021b) with lower choices-only accuracy.
However, we invite future research to apply our graph mining algorithm to
other datasets, including non-MCQA datasets, to build contrast sets that can
further probe LLM decision-making.

Further, we acknowledge that our contrast set contains MCQA questions limited
to two choices, diverging from the original evaluation set’s range of two to
eight choices. While having less options does make it more likely for a model
to guess the right answer, our qualitative analysis shows that the concepts
tested in our contrast set are not markedly different in plausibility (section
4.1), and thus are not too easy. Further, while our contrast set is easier in
theory, it still preserves LLM rankings, even on the subset used to derive the
contrast set (Appendix B.3), ultimately supporting the idea that MCQA can
reliably rank LLMs capabilities.

##  8 Ethical Considerations

When models heavily rely on patterns or biases present in datasets, we may
overestimate model abilities and face generalizability issues during
deployment. In this work, we probe the extent to which LLMs over-rely on
patterns in MCQA choices when provided both the question and choices in the
prompt, ultimately finding that this effect is small. However, we believe it
is still critical for LLM practitioners to be aware that LLMs can outperform
random guessing when using just the choices as input, as this could have
downstream effects. Thus, we encourage future research efforts in designing
special datasets that can help interpret specific abilities within LLM
decision-making.

Further, we note that when any model is used in a data creation pipeline,
there is the possibility of models propagating their own biases. We
specifically address this issue by designing a graph mining algorithm that
leverages minimal model intervention, only in the form of computing semantic
similarity, which greatly lowers this risk compared to synthetic data
generators like LLMs. We hope future works can adopt data creation pipelines
with minimal model use similar to ours to avoid the risk of generating model-
specific biases or artifacts.

##  9 Acknowledgements

We would like to thank members of the CLIP lab at the University of Maryland
and external collaborators for their feedback and discussions of this work,
including Neha Srikanth, Shi Feng, and Jordan Boyd-Graber. We also thank Yu
Hou and Dayeon Ki for their help with annotations. This material is based upon
work supported by the National Science Foundation Graduate Research Fellowship
Program under Grant No. DGE 2236417. Any opinions, findings, and conclusions
or recommendations expressed in this material are those of the author(s) and
do not necessarily reflect the views of the National Science Foundation.

## References

  * Balepur et al. (2023) Nishant Balepur, Shramay Palta, and Rachel Rudinger. 2023.  It’s not easy being wrong: Evaluating process of elimination reasoning in large language models.  _arXiv preprint arXiv:2311.07532_.
  * Balepur et al. (2024) Nishant Balepur, Abhilasha Ravichander, and Rachel Rudinger. 2024.  Artifacts or abduction: How do llms answer multiple-choice questions without the question?  _arXiv preprint arXiv:2402.12483_.
  * Beeching et al. (2023) Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf. 2023.  Open llm leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard.
  * Bisk et al. (2020) Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. 2020.  Piqa: Reasoning about physical commonsense in natural language.  In _Proceedings of the AAAI conference on artificial intelligence_ , volume 34, pages 7432–7439.
  * Boppana and Halldórsson (1992) Ravi Boppana and Magnús M Halldórsson. 1992.  Approximating maximum independent sets by excluding subgraphs.  _BIT Numerical Mathematics_ , 32(2):180–196.
  * Clark et al. (2018) Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  _arXiv preprint arXiv:1803.05457_.
  * Conneau et al. (2017) Alexis Conneau, Douwe Kiela, Holger Schwenk, Loïc Barrault, and Antoine Bordes. 2017.  Supervised learning of universal sentence representations from natural language inference data.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 670–680, Copenhagen, Denmark. Association for Computational Linguistics.
  * Elazar et al. (2023) Yanai Elazar, Bhargavi Paranjape, Hao Peng, Sarah Wiegreffe, Khyathi Raghavi, Vivek Srikumar, Sameer Singh, and Noah A Smith. 2023.  Measuring and improving attentiveness to partial inputs with counterfactuals.  _arXiv preprint arXiv:2311.09605_.
  * Fryer et al. (2022) Zee Fryer, Vera Axelrod, Ben Packer, Alex Beutel, Jilin Chen, and Kellie Webster. 2022.  Flexible text generation for counterfactual fairness probing.  In _Proceedings of the Sixth Workshop on Online Abuse and Harms (WOAH)_ , pages 209–229, Seattle, Washington (Hybrid). Association for Computational Linguistics.
  * Gardner et al. (2020) Matt Gardner, Yoav Artzi, Victoria Basmov, Jonathan Berant, Ben Bogin, Sihao Chen, Pradeep Dasigi, Dheeru Dua, Yanai Elazar, Ananth Gottumukkala, Nitish Gupta, Hannaneh Hajishirzi, Gabriel Ilharco, Daniel Khashabi, Kevin Lin, Jiangming Liu, Nelson F. Liu, Phoebe Mulcaire, Qiang Ning, Sameer Singh, Noah A. Smith, Sanjay Subramanian, Reut Tsarfaty, Eric Wallace, Ally Zhang, and Ben Zhou. 2020.  Evaluating models’ local decision boundaries via contrast sets.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1307–1323, Online. Association for Computational Linguistics.
  * Gierl et al. (2017) Mark J Gierl, Okan Bulut, Qi Guo, and Xinxin Zhang. 2017.  Developing, analyzing, and using distractors for multiple-choice tests in education: A comprehensive review.  _Review of Educational Research_ , 87(6):1082–1116.
  * Glockner et al. (2018) Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018.  Breaking NLI systems with sentences that require simple lexical inferences.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 650–655, Melbourne, Australia. Association for Computational Linguistics.
  * Hendrycks et al. (2021a) Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021a.  Aligning {ai} with shared human values.  In _International Conference on Learning Representations_.
  * Hendrycks et al. (2021b) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2021b.  Measuring massive multitask language understanding.  In _International Conference on Learning Representations_.
  * Jiang et al. (2023) Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. 2023.  Mistral 7b.  _arXiv preprint arXiv:2310.06825_.
  * Jiang et al. (2024) Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, Gianna Lengyel, Guillaume Bour, Guillaume Lample, Lélio Renard Lavaud, Lucile Saulnier, Marie-Anne Lachaux, Pierre Stock, Sandeep Subramanian, Sophia Yang, Szymon Antoniak, Teven Le Scao, Théophile Gervet, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2024.  Mixtral of experts.
  * Kaushik et al. (2020) Divyansh Kaushik, Eduard Hovy, and Zachary Lipton. 2020.  Learning the difference that makes a difference with counterfactually-augmented data.  In _International Conference on Learning Representations_.
  * Khashabi et al. (2020) Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020.  UNIFIEDQA: Crossing format boundaries with a single QA system.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1896–1907, Online. Association for Computational Linguistics.
  * Khot et al. (2020) Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020.  Qasc: A dataset for question answering via sentence composition.  In _Proceedings of the AAAI Conference on Artificial Intelligence_ , volume 34, pages 8082–8090.
  * Levesque et al. (2012) Hector Levesque, Ernest Davis, and Leora Morgenstern. 2012.  The winograd schema challenge.  In _Thirteenth international conference on the principles of knowledge representation and reasoning_.
  * Liang et al. (2023) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, Benjamin Newman, Binhang Yuan, Bobby Yan, Ce Zhang, Christian Alexander Cosgrove, Christopher D Manning, Christopher Re, Diana Acosta-Navas, Drew Arad Hudson, Eric Zelikman, Esin Durmus, Faisal Ladhak, Frieda Rong, Hongyu Ren, Huaxiu Yao, Jue WANG, Keshav Santhanam, Laurel Orr, Lucia Zheng, Mert Yuksekgonul, Mirac Suzgun, Nathan Kim, Neel Guha, Niladri S. Chatterji, Omar Khattab, Peter Henderson, Qian Huang, Ryan Andrew Chi, Sang Michael Xie, Shibani Santurkar, Surya Ganguli, Tatsunori Hashimoto, Thomas Icard, Tianyi Zhang, Vishrav Chaudhary, William Wang, Xuechen Li, Yifan Mai, Yuhui Zhang, and Yuta Koreeda. 2023.  Holistic evaluation of language models.  _Transactions on Machine Learning Research_.  Featured Certification, Expert Certification.
  * Mihaylov et al. (2018) Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 2381–2391, Brussels, Belgium. Association for Computational Linguistics.
  * Offerijns et al. (2020) Jeroen Offerijns, Suzan Verberne, and Tessa Verhoef. 2020.  Better distractions: Transformer-based distractor generation and multiple choice question filtering.  _arXiv preprint arXiv:2010.09598_.
  * Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Hamza Alobeidli, Alessandro Cappelli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023.  The refinedweb dataset for falcon LLM: Outperforming curated corpora with web data only.  In _Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track_.
  * Pezeshkpour and Hruschka (2023) Pouya Pezeshkpour and Estevam Hruschka. 2023.  Large language models sensitivity to the order of options in multiple-choice questions.  _arXiv preprint arXiv:2308.11483_.
  * Qin et al. (2019) Lianhui Qin, Antoine Bosselut, Ari Holtzman, Chandra Bhagavatula, Elizabeth Clark, and Yejin Choi. 2019.  Counterfactual story reasoning and generation.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 5043–5053, Hong Kong, China. Association for Computational Linguistics.
  * Ribeiro et al. (2020) Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020.  Beyond accuracy: Behavioral testing of NLP models with CheckList.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4902–4912, Online. Association for Computational Linguistics.
  * Robinson and Wingate (2023) Joshua Robinson and David Wingate. 2023.  Leveraging large language models for multiple choice question answering.  In _The Eleventh International Conference on Learning Representations_.
  * Sap et al. (2019) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019.  Social IQa: Commonsense reasoning about social interactions.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 4463–4473, Hong Kong, China. Association for Computational Linguistics.
  * Srikanth et al. (2024) Neha Srikanth, Marine Carpuat, and Rachel Rudinger. 2024.  How often are errors in natural language reasoning due to paraphrastic variability?  _arXiv preprint arXiv:2404.11717_.
  * Srikanth and Rudinger (2022) Neha Srikanth and Rachel Rudinger. 2022.  Partial-input baselines show that NLI models can ignore context, but they don’t. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_ , pages 4753–4763, Seattle, United States. Association for Computational Linguistics.
  * Talmor et al. (2019) Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019.  CommonsenseQA: A question answering challenge targeting commonsense knowledge.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4149–4158, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Team et al. (2024) Gemma Team, Thomas Mesnard, Cassidy Hardin, Robert Dadashi, Surya Bhupatiraju, Shreya Pathak, Laurent Sifre, Morgane Rivière, Mihir Sanjay Kale, Juliette Love, et al. 2024.  Gemma: Open models based on gemini research and technology.  _arXiv preprint arXiv:2403.08295_.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * West et al. (2015) Matthew West, Geoffrey L Herman, and Craig Zilles. 2015.  Prairielearn: Mastery-based online problem solving with adaptive scoring and recommendations driven by machine learning.  In _2015 ASEE Annual Conference & Exposition_, pages 26–1238.
  * Wu et al. (2021) Tongshuang Wu, Marco Tulio Ribeiro, Jeffrey Heer, and Daniel Weld. 2021.  Polyjuice: Generating counterfactuals for explaining, evaluating, and improving models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 6707–6723, Online. Association for Computational Linguistics.
  * Young et al. (2024) Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Heng Li, Jiangcheng Zhu, Jianqun Chen, Jing Chang, et al. 2024.  Yi: Open foundation models by 01. ai.  _arXiv preprint arXiv:2403.04652_.
  * Yu et al. (2024) Yue Yu, Yuchen Zhuang, Jieyu Zhang, Yu Meng, Alexander J Ratner, Ranjay Krishna, Jiaming Shen, and Chao Zhang. 2024.  Large language model as attributed training data generator: A tale of diversity and bias.  _Advances in Neural Information Processing Systems_ , 36.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  Hellaswag: Can a machine really finish your sentence? In _Annual Meeting of the Association for Computational Linguistics_.
  * Zheng et al. (2024) Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang. 2024.  Large language models are not robust multiple choice selectors.  In _The Twelfth International Conference on Learning Representations_.

##  Appendix A Experimental Setup

###  A.1 Dataset Details

The UnifiedQA evaluation set has questions from the evaluation sets of the
following six datasets:

  * •

ARC: 1172 four-choice questions drawn from grade-school science questions.

  * •

OpenBookQA: 500 four-choices questions modeled after open-book exams.

  * •

QASC: 926 eight-choice questions about grade school science with a focus on
sentence composition.

  * •

CommonsenseQA: 1221 four-choice questions meant to test commonsense knowledge
from ConceptNet.

  * •

Physical IQa: 1838 two-choice questions about physical commonsense reasoning.

  * •

Social IQA: 1954 three-choice questions involving reasoning about everyday
social interactions.

After running our algorithm, our contrast set contains 377 questions from
CommonsenseQA, 285 questions from QASC, 79 questions from ARC, 53 questions
from Social IQa, 22 questions from OpenBookQA, and 4 questions from Physical
IQa, all of which have two choices.

###  A.2 Prompt Box Example

The following subsection is adapted directly from the Appendix of Balepur et
al. (2024) to highlight the utility of their prompt boxes.

Below, we provide a detailed example to illustrate the application of our
prompt boxes. Suppose we have the full prompt (Prompt 3.2):

Prompt 2.1: Full Prompt Question: $q$ Choices: $\mathcal{C}$ Answer: $a$

In the above prompt, the LLM uses the question $q$ and choices $\mathcal{C}$
as input and is asked to generate the letter of the answer $a$. Suppose we
have 5 few-shot examples, with questions $q_{1}$ ,…, $q_{5}$, list of choices
$\mathcal{C}_{1}$, …, $\mathcal{C}_{5}$, and ground truth answers $a_{1}$, …,
$a_{5}$. The expanded few-shot prompt for the prompt box is written below:

Prompt 2.1: Full Prompt Expanded Question: $q_{1}$ Choices: $\mathcal{C}_{1}$
Answer: $a_{1}$ Question: $q_{2}$ Choices: $\mathcal{C}_{2}$ Answer: $a_{2}$
Question: $q_{3}$ Choices: $\mathcal{C}_{3}$ Answer: $a_{3}$ Question: $q_{4}$
Choices: $\mathcal{C}_{4}$ Answer: $a_{4}$ Question: $q_{5}$ Choices:
$\mathcal{C}_{5}$ Answer: $a_{5}$ Question: $q$ Choices: $\mathcal{C}$ Answer:

Using this prompt, the LLM must generate $a$, which is the highlighted text in
the prompt box.

###  A.3 Prompting Details

We design few-shot prompts following the format described by our prompt boxes.
The few-shot examples were randomly selected from the training set, and we
ensured that these contained a balanced distribution of output labels and that
the demonstrations were shuffled. We created a few-shot prompt for each
dataset. Both the UnifiedQA evaluation set and the contrast set used the exact
same prompt. Even though this results in demonstrations with more than two
choices, we found that this did not confuse models on the contrast set, as
they never outputted an invalid letter (i.e. “(C)” when there are two
choices). In the case of an invalid output, which stemmed from a non-letter
choice, we marked the output as incorrect.

##  Appendix B Results

###  B.1 Qualitative Analysis Details

Below, we provide the exact instructions (Figure 4) and annotation interface
(Figure 5) shown to our annotators. Our annotation interface is based on
PrairieLearn West et al. (2015). Our use of plausibility and relevance for
this annotation task is based on existing work Gierl et al. (2017).

The random baseline we compare against is the trivial solution described in
section 2.1. This baseline selects a random gold answer from the same dataset
to form a set of choices. We apply this algorithm to the same 50 sampled
instances as the ones annotators evaluated with our contrast set, meaning that
the questions produced by this baseline only differ by the chosen distractor;
the question, choices, and gold answer are all consistent across approaches.

###  B.2 3-shot Prompting Results

In Figure 6, we show the same results as Figure 3 but with three-shot
prompting. The same trends of high choices-only accuracy and the consistency
of full prompt rankings across evaluation and contrast sets both hold, with a
Kendall’s $\tau$ of 0.88. We did not test 0-shot prompting as we were working
with base LLMs (i.e. unaligned and not instruction-tuned), which should not
have the capability to complete tasks in a 0-shot manner. We believe that
studying choices-only accuracy in 0-shot settings could be an interesting
avenue for future work.

###  B.3 UnifiedQA Evaluation Subset

Our mined contrast set only has two choices for every question, while the
original evaluation set has questions ranging from 2 to 8 choices. To ensure
the consistency of rankings is not confounded by the the reduction of possible
choices, we also report the 10-shot accuracy on the subset of UnifiedQA that
was used to derive the contrast set. This subset is essentially equivalent to
the contrast set, but with additional choices on each question so that the
number of choices are consistent. In Figure 7, the UnifiedQA Evaluation set
and the UnifiedQA Evaluation subset have a similarly high consistency between
rankings of full prompt accuracy. Thus, in our experiments, the number of
choices on each question does not seem to largely influence the ranking of
LLMs.

Figure 4: Instructions shown to annotators.

Figure 5: Interface used by annotators. Figure 6: 3-shot benchmarking of 12
LLMs on the UnifiedQA evaluation set and the contrast set, sorted by full-
prompt accuracy. The same trends found for 5-shot and 10-shot prompting hold
for 3-shot prompting. Figure 7: 10-shot benchmarking of 12 LLMs on the
UnifiedQA evaluation set, the contrast set, and the subset of the full
UnifiedQA evaluation split with the same questions as the contrast set.