# Open-LLM-Leaderboard: From Multi-choice to Open-style Questions for LLMs
Evaluation, Benchmark, and Arena

Aidar Myrzakhan∗, Sondos Mahmoud Bsharat∗, and Zhiqiang Shen
VILA-Lab, Mohamed bin Zayed University of AI
<EMAIL_ADDRESS>
Joint first author & equal contribution.

###### Abstract

Multiple-choice questions (MCQ) are frequently used to assess large language
models (LLMs). Typically, an LLM is given a question and selects the answer
deemed most probable after adjustments for factors like length. Unfortunately,
LLMs may inherently favor certain answer choice IDs, such as A/B/C/D, due to
inherent biases of priori unbalanced probabilities, influencing the prediction
of answers based on these IDs. Previous research has introduced methods to
reduce this “selection bias” by simply permutating options on a few test
samples and applying them to new ones. Another problem of MCQ is the lottery
ticket choice by “random guessing”. The LLM does not learn particular
knowledge, but the option is guessed correctly. This situation is especially
serious for those small-scale LLMs111For instance, on MMLU, the random
guessing accuracy is 25%, and most small-scale LLMs obtain results around this
value as shown in [35, 46]. It is difficult to distinguish which model is
better under this situation.. To address them, a more thorough approach
involves shifting from MCQ to open-style questions, which can fundamentally
eliminate selection bias and random guessing issues. However, transitioning
causes its own set of challenges in (1) identifying suitable open-style
questions and (2) validating the correctness of LLM open-style responses
against human-annotated ground-truths. This work aims to tackle these
significant difficulties, and establish a new LLM evaluation benchmark through
entirely open-style questions. Consequently, we introduce the Open-LLM-
Leaderboard to track various LLMs’ performance and reflect true capability of
them, such as GPT-4o/4/3.5, Claude 3, Gemini, etc. Our code and dataset are
available at https://github.com/VILA-Lab/Open-LLM-Leaderboard.

##  1 Introduction

Question that is suitable for open-style: Let x = 1. What is x << 3 in Python
3?
---
Options: A. 1 B. 3 C. 8 D. 16
Answer:  C
Question that is not suitable for open-style: Which of the following
statements is true?
Options:
A. Every equivalence relation is a partial-ordering relation.
B. Number of relations form A = x, y, z to B= (1, 2), is 64.
C. Empty relation _ is reflexive
D. Properties of a relation being symmetric and being un-symmetric are
negative of each other.
Answer:  B

Figure 1: Examples of MCQ from MMLU.

Large language models (LLMs) are increasingly excelling at various natural
language processing tasks, including text generation [11], translation [45,
50], summarization [22], code generation [20, 33], and chatbot interaction
[28]. With the rising capability, the need for a robust evaluation strategy
that can accurately assess the performance of these models is becoming crucial
in order to identify their true effectiveness and choose the most appropriate
one for a given task. Common metrics for assessing LLMs today include
relevance, frequency of hallucinations, accuracy in question answering,
toxicity, and retrieval-specific metrics, among others. In the context of
question-answering evaluations, prior works usually investigate the model’s
performance in terms of answer accuracy, courtesy, and conciseness. And
multiple choice questions (MCQ) have emerged as a predominant format for such
assessments, wherein a question is presented with several possible responses,
and the model is required to select the most fitting choice ID, as exemplified
in Figure 1. Lately, the MCQ format has seen widespread application in LLM-
focused contexts, including benchmarks [18, 44, 12] that examine LLM
capabilities and automated/crowdsourcing evaluation frameworks [21, 49, 5]
that streamline the assessment process.

However, previous studies [48, 32] have discussed that the lack of resilience
of LLMs to changes in the positioning of options stems from their tendency to
exhibit biased behavior: they often favor choosing certain option IDs (such as
“Option A”) as responses, a phenomenon that is referred to as selection bias.
Moreover, it shows that selection bias exists widely across various LLMs and
cannot be mitigated by simple prompting skills. The underlying reason of this
issue comes from the condition that the model is trained with a priori
distribution that assigns more probabilistic choices to specific ID tokens.
Another issue of MCQ is the “random guessing” that is discussed in [35].
Specifically, small models such as the 1B-level variants, may struggle to
achieve reliable predictions on many benchmarks like MMLU which uses four
choices as the answer candidates of the questions. Their results could
resemble random choices, not truly capturing the model’s actual capabilities.

To fundamentally eliminate selection bias and random guessing in LLMs, in this
work, we build an open-style question benchmark for LLM evaluation. Leveraging
this benchmark, we present the Open-LLM-Leaderboard, a new automated framework
designed to refine the assessment process of LLMs. This framework functions in
supplement to prior evaluation frameworks such as [21, 49, 5] with several
advantages as presented in Sec. 4.4. However, constructing such a benchmark
has two significant challenges: (1) how to determine the appropriate questions
that can be effectively transformed from MCQ into open-style questions, and
(2) how to establish an approach to accurately validate the correctness of the
LLM’s open-style answers in comparison to human-annotated ground-truths,
especially in contrast to MCQ, which typically have defined single-choice
standard answers.

For the first challenge of identifying the multiple-choice questions that are
suitable for converting to open-style questions, we design an automatic
coarse-to-fine selecting protocol through customized prompts and multi-stage
filtering process. Specifically, in the first stage, we use the binary
classification to filter the questions with high confidence as the positive
pool and others are assigned as negative. Our second stage uses a soft scoring
method (1-10 ratings) to judge the suitability of the questions for the open-
style from the questions that are clarified as negative in the first stage.
For the second challenge of evaluating the correctness of the LLM’s open-style
answers in comparison to human-annotated ground-truths, we further design a
task-specific prompt and leverage GPT-4 to examine if the response is correct.
To validate the accuracy of the automatic evaluation strategy, we randomly
sample 100 results and manually check the automatic evaluation results with
the corresponding responses, and confirm that it is reliable with an error
rate of less than 5%.

In our end-to-end assessment of the LLM evaluation and ranking process, we
conduct a comprehensive analysis on the well-recognized LLMs, including
GPT-4o, GPT-4, ChatGPT, Claude-3 Opus, Gemini-Pro and Mistral-Large. Our
benchmarking results indicate that GPT-4o currently holds the position as the
strongest LLM. We further provide a small regime LLM leaderboard targeting at
LLMs smaller than 3B. Moreover, our study demonstrates a high correlation
between the rankings produced by our open-style benchmark and those derived
from user-based evaluations or direct human assessments.

##  2 Related Work

Large Language Models (LLMs). Recent advancements in LLMs, such as GPT-3 [9]
and GPT-4 [28] have had a significant impact in the field of natural language
processing and have found widespread application across various domains. It
has indeed initiated a kind of chain reaction within the community and beyond.
As each new iteration of LLMs demonstrates enhanced capabilities,
organizations and researchers across various sectors are motivated to develop
their own models, such as LLaMA [40, 41], Gemini [38], and Claude [2], or find
innovative ways to improve existing LLMs through instruction tuning, like
Alpaca [37], and Vicuna [10].

Multiple Choice Questions (MCQ). In the realm of LLM research, MCQ has become
a pivotal tool for evaluating and enhancing the capabilities of these models.
Notable datasets like the MMLU [18], HellaSwag [44], and ARC [12] have been
instrumental in this regard. Their diverse assessment of broad knowledge and
commonsense reasoning help in benchmarking the depth and versatility of LLMs
in understanding, reasoning, and applying knowledge across various domains.
MCSB [31] introduces a natural prompting strategy for LLMs, which presents
questions and answer choices together, allowing the model to explicitly
compare options.

Bias in LLMs. Selection bias, a specific form of bias relevant to the
evaluation of LLMs through MCQ, has garnered attention due to its understated
and widespread impact. A series of works [49, 30, 48, 42] have shown that LLMs
may develop a propensity to favor certain answer choices based on their
position or encoding, such as the alphabetical ordering of A/B/C/D in MCQ.
This phenomenon can lead to skewed evaluation results, misrepresenting a
model’s true understanding and reasoning capabilities.

##  3 Approach

Figure 2: An overview of a dual-path evaluation pipeline for LLMs, starting
with the collection of MCQ datasets. It branches into two paths, with the MCQ
path proceeding directly from response collection to evaluation, while the
open-style path passes through an additional filtering phase. After
evaluation, both paths converge in a comparative analysis.

###  3.1 Defining Open-style Questions

Open-style questions, aka open-ended questions, require the model to generate
an answer without being constrained by a set of predetermined choices. In the
context of LLM evaluation, these questions are designed to assess the model’s
ability to generate coherent, relevant, and contextually appropriate responses
based on the input query. While multiple-choice questions can efficiently
assess specific factual knowledge and comprehension, open-style questions
offer a deeper insight into the LLM’s generative capabilities, understanding
of context, and ability to engage with complex tasks. Also, open-style
questions can avoid the inherent selection bias and random guessing weaknesses
compared to multiple-choice questions.

###  3.2 Automatic Open-style Question Filtering and Generation

Multi-stage Filtering and Postprocessing via Coarse-to-fine Process. Our
proposed multi-stage filtering approach consists of four main steps to
streamline the conversion: (1) Initially classify datasets as either
convertible or non-convertible. (2) Assign each question a confidence score to
indicate the likelihood that it can be framed as an open-style question. (3)
Exclude questions with confidence scores below a specified threshold and
classified as non-convertible. (4) Combine questions that are labeled as non-
convertible but have high confidence scores with those labeled as convertible.

Stage1: Preliminary Filter using Binary Classification. Considering that the
structure of MCQ varies, converting them into an open-style format is not
always possible, particularly because certain questions are strongly linked to
their choices. For instance, questions formulated as “Which one of the
following is true” or “All except” or “Which of these”. Such questions are
typically unsuitable for conversion into an open-style format since the
absence of the options could change the question’s core, resulting in
incomplete questions.

Table 1: Prompt design for two-stage filtering and post verification.

Stage One: Coarse Filtering Prompt
---
|  """Your task is to review a series of multiple-choice questions and
evaluate their ability to be answered without the provided answer choices. For
questions that begin with an incomplete sentence (e.g., "During swallowing,
..."), use your knowledge to attempt to complete the sentence accurately. For
direct questions that ask for specific information or identification (e.g.,
"Which of the following structures is part of the small intestine?"), assess
whether the question is formulated clearly enough that an informed answer can
be given without seeing the multiple-choice options. For mathematical or
analytical questions (e.g., "Find all cosets of the subgroup 4Z of 2Z"),
determine if the question provides enough context and information for a
solution to be formulated without additional options.  Please follow this
format for your evaluation: QUESTION: [Insert the question here] VERDICT:
Respond with "YES" if the question is clear and can be directly answered based
on its content alone, or "NO" if it relies on the answer choices to be
understood or answered. Your response should include only the verdict without
any justification or reasoning."""
---
Stage Two: Fine-grained Filtering Prompt
|  You will assign a numerical score from 1 to 10 based on how confidently it can be answered without the choices. The scoring criteria are as follows: 1: The question is entirely dependent on its choices for an answer, making it impossible to answer without them. Example: ‘Which of the following statements is correct?’ 10: The question can be easily and confidently answered based solely on the question stem, without any need to refer to the provided options. Example: ‘What is the first law of thermodynamics in physics?’ Intermediate Scores:  2-4: The question stem gives very little information and is highly reliant on the choices for context. Example: ‘Which of these is a prime number?’ 5: The question provides some context or information, that gives a moderate possibility to answer the question. Example: ‘Which of the following best describes the structure that collects urine in the body?’ 6: The question provides a good amount of context or information, that gives a moderate possibility to answer the question. Example: ‘Statement 1 | A factor group of a non-Abelian group is non-Abelian. Statement 2 | If K is a normal subgroup of H and H is a normal subgroup of G, then K is a normal subgroup of G.’ 7: The question provides a good amount of context or information, that gives a high possibility to answer the question. Example: ‘The element (4, 2) of Z_12 x Z_8 has order’ 8-9: The question provides a good amount of context or information, that gives a high possibility to answer the question. Example: ‘A "dished face" profile is often associated with’ ONLY GIVE THE VALUE BETWEEN 1-10 AS YOUR ANSWER. DO NOT INCLUDE ANY OTHER INFORMATION IN YOUR RESPONSE Example Format: QUESTION: question here VERDICT: value in [1-10] here
---
GPT-4 Prompt for Verification
|  """Evaluate the answer of a AI model to a question. You will be provided
with the question, the AI model’s answer, and the correct answer. Your task is
to evaluate the AI model’s response and determine whether it is Correct or
Incorrect. Grade the AI model answers based ONLY on their factual accuracy. It
is OK if the AI model answer contains more information than the true answer,
as long as it does not contain any conflicting statements. Otherwise, it
should be marked as Incorrect. Ignore differences in punctuation and phrasing
between the AI model’s answer and the true answer. Example Format: QUESTION:
question here STUDENT ANSWER: student’s answer here TRUE ANSWER: true answer
here GRADE: Correct or Incorrect here  Your response should include only the
verdict without any justification or reasoning."""
---

To effectively handle this challenge of identifying whether multiple-choice
questions are suitable for open-style conversion, we leverage the power of
prompting techniques to create a customized classification prompt as shown in
Table 1. In the prompt, we integrate different types of questions from
different datasets to demonstrate how an LLM may evaluate each question to be
written in an open-style way, eventually classifying them as convertible “YES”
or non-convertible “NO”. It will determine whether a question provides a clear
context and information without relying on the provided options or not. In the
prompt we integrate different types of questions from different datasets to
demonstrate how an LLM like GPT-4 may evaluate each question to be written in
an open-style way, eventually classifying them as convertible “YES” or non-
convertible “NO”. We set the prompt to eliminate any additional explanations,
by stating that “Your response should include only the verdict without any
justification or reasoning.” This guarantees that the answer to each inquiry
is conveyed concisely as “YES” or “NO”.

To understand our initial filtering results, we conduct an error analysis
manually by selecting 100 questions in the “YES” and “NO” pools separately. In
the samples classified as “YES”, we find that only around 5% of the questions
are false positive cases, verifying a low misclassification error for the
positive question selection by our filtering strategy. Conversely, within the
“NO” sample, around 40% of the questions are actually suitable for open-style
questions but mistakenly classified as negative. This situation often arises
from questions that include phrases like “Which of”. Similarly, questions
involving true/false statements, sentence completions, or fill-in-the-blanks
are also sometimes inappropriately classified as non-convertible. This
analysis motivates us to develop a cascaded fine-grained stage to further
filter more positive questions in “NO” pool using particular prompts, as
described in the following Stage 2 process.

Stage2: Confidence Score Assignment. As we aim to overcome the issue of
classifying questions with specific patterns as non-convertible, we introduce
a second stage of filtering centered on confidence score assignment. This
involves instructing the large language model to assign a confidence score on
a scale from 1 to 10, reflecting the possibility of the question being written
in an open-style format. Since a significant number of questions are
unsuitable for an open-style format, categorized as “NO” and have a confidence
score below 5, we set a confidence score threshold to be 5. Therefore,
questions classified as non-convertible with a confidence score lower than
this threshold are excluded, while those remaining above the threshold and
those initially classified as convertible are moved into the “YES” category to
be converted to an open-style format.

###  3.3 Open-style Question Answer Evaluation

After establishing a set of convertible questions from various datasets and
obtaining their responses from several LLMs, there arises a need to evaluate
these questions. Given that our ground truth answers are based on the MCQ
format with defined answers, it necessitates a method for efficiently and
accurately validating the correctness of responses to open-style questions. To
this end, we design a customized prompt, as shown in Figure 2 that utilizes
the correct MCQ answer as the ground truth to determine if the open-style
responses are correct or incorrect by the prediction $\hat{y}$:

$\hat{y}=\texttt{LLM}_{\texttt{e}}(\text{prompt}(q,\hat{a},a))$ (1)

where $\hat{y}$ represents the prediction and $\texttt{LLM}_{\texttt{e}}$ is
the LLM evaluator. $q$, $\hat{a}$ and $a$ represent the question, LLM
generated answer, and correct answer from MCQ, respectively, and the prompt is
provided in Table 1 of Appendix. While these open-style answers are evaluated
based on the MCQ’s ground truth, issues of misevaluation might arise. This
includes scenarios where a response is inaccurately classified as correct
simply because it contains certain keywords also found in the ground truth. To
tackle this issue we include specific phrases in the prompt. These phrases,
such as “as long as it does not contain any conflicting statements”, ensure
that a response is not automatically classified as correct based on the
presence of a keyword, avoiding incorrect markings when the response
contradicts the correct answer. Additionally, to prevent the exclusion of
correct answers that incorporate extra information, we incorporate the phrase
“It is OK if the AI model’s answer contains more information than the true
answer”. Furthermore, we highlight that minor differences in punctuation and
phrasing between the open-style responses and the ground truth answers should
not lead to their being classified as incorrect. To see the correctness of the
LLM judgement we take the randomly drawn 100 responses from all models. The
human evaluation process for our study was conducted by the authors
themselves. The agreement between the LLM evaluations and those of a human
evaluator was quantitatively assessed using Cohen’s kappa [13], which yielded
a score of 0.83. This substantial kappa score222 The Kappa score is a
statistical measure of inter-rater agreement for categorical items, defined by
the equation: $\kappa=\frac{P_{o}-P_{e}}{1-P_{e}}$ where $P_{o}$ is the
observed agreement and $P_{e}$ is the expected agreement by chance. verifies
that the LLM’s ability to determine the correctness of responses aligns
closely with human judgment, demonstrating strong reliability in its
evaluation process.

##  4 An Open-style Question Benchmark (OSQ-bench)

###  4.1 Statistics and Distributions

Table 2 describes the basic statistics of the dataset questions that are
suitable for answering in open-style format. In total, we have evaluated 42K
questions from 9 different datasets and more than 23K of them are classified
as appropriate for open-style answering.

Table 2: Statistics on open-style questions across different datasets.

Benchmarks | #Evaluated | #Open-Style | Average Question Length (words)
---|---|---|---
MMLU | 14,042 | 7,784 | 36.6
ARC | 3,428 | 3,118 | 21.1
MedMCQA | 4,183 | 2,318 | 14.1
CommonsenseQA | 1,221 | 710 | 13.1
Race | 4,934 | 3,520 | 10.0
OpenbookQA | 1,000 | 491 | 10.3
WinoGrande | 1,267 | 1,267 | 19.1
HellaSwag | 10,042 | 3,915 | 40.1
PIQA | 1,838 | 696 | 7.1
Overall | 41,955 | 23,839 | 19.05

###  4.2 Diversity

Our investigation into the diversity of questions within our benchmark is
foundational for understanding the landscape of open-ended question answering.
To comprehensively assess the breadth of question diversity, we have conducted
a systematic categorization of the question types sourced from an array of
distinct datasets. From the total initial pool of 41,955 questions, we refine
the selection to 23,839 questions, ensuring that each one is conducive to
open-ended responses. The distribution of those questions is illustrated in
Figure 3, which segments the data into several domains based on the content of
the questions. The segmentation of the plot underscores the interdisciplinary
nature of our dataset. It features a broad spectrum of categories such as
literature and reading comprehension, commonsense reasoning, domain-specific
(medicine, STEM, and etc), and multi-topic knowledge. Also, Table 2
demonstrates the diversity of question length used for the benchmark.

Figure 3: Diversity and distribution of used datasets for our OSQ-bench.

###  4.3 Quality

Our newly developed benchmark, curated from widely recognized datasets, stands
out by focusing on questions suitable to open-style answering, i.e., a format
that demands a deep understanding and an ability to generate informative,
unrestricted responses. Given that the datasets from which these questions
originate are widely utilized and highly recognizable within the research
community, it follows that the questions have good quality to assess the
models’ capabilities. Moreover, due to the thorough filtering process it has
undergone, it results in a low false positive rate (questions not suitable for
open-style that are classified as suitable) of around 5%. This indicates that
the vast majority of questions categorized as suitable for open-style answers
indeed meet the criteria.

###  4.4 Property and Advantage

As shown in Table 3, our leaderboard exhibits several advantages: first is the
debiased results compared to the MCQ-based leaderboard, which has been
discussed thoroughly. Another advantage is the faster and cheaper evaluation
over crowduser-based leaderboards. Our results and rankings can be generated
automatically without any human intervention.

Table 3: Comparison with different LLM leaderboards. “Biased” indicates the
selection bias.

Leaderboard | Types | Diversity | Biased | Evaluation
---|---|---|---|---
Huggingface Leaderboard [5] | Multiple Choices Questions | High | ✓ | Automatically
AlpacaEval Leaderboard [21] | Human Questions&Feedback | Low | ✗ | GPT-4
Chatbot Arena Leaderboard [49] | Human Questions&Feedback | Low | ✗ | GPT-4/Crowdusers
Open-LLM-Leaderboard (Ours) | Open Style Questions | High | ✗ | GPT-4

##  5 Experiments

Figure 4: Performance comparison of various LLMs on multiple-choice (MCQ) and
open-style questions (OSQ) across different datasets. The bar graphs on the
left show the counts of correct and incorrect responses (✗ MCQ vs. ✓ OSQ; ✓
MCQ vs. ✗ OSQ; ✓ MCQ vs. ✓ OSQ; ✗ MCQ vs. ✗ OSQ), while the radar charts on
the right illustrate the accuracy comparisons between MCQ and OSQ for each
language model (Pink is the MCQ accuracy and LimeGreen is the OSQ accuracy).
Figure 5: Percentage of convertible MCQ to open style questions on various
datasets.

###  5.1 Models

We generate responses from LLMs of different sizes. The large-scale LLMs:
gpt-3.5-turbo, gpt-4-1106-preview, gpt-4o [27], claude-3-opus-20240229 [3],
mistral-large-latest [24], gemini-pro [16], and llama3 [1]. We use the
commercial APIs to collect responses from all of these models. The small-scale
LLMs: qwen1.5 [4], gemma [39], SlimPajama-DC [35], RedPajama [25], OLMo [17],
Pythia [6], TinyLlama [46], OPT [47], GPT-Neo [8], and Cerebras-GPT [14]. All
of the small-scale model responses are collected using Huggingface [43] and
lm-evaluation-harness framework [15] with 4$\times$ 4090 RTX GPUs.

###  5.2 Datasets

We present a brief overview of used datasets, highlighting their distinctive
characteristics and the specific aspects they aim to evaluate. MMLU [18], ARC
[12], and MedMCQA [29] stand out with their comprehensive range of tasks
spanning across various disciplines. PIQA [7], CommonsenseQA [36], OpenBookQA
[23], and HellaSwag [44] focus on the different aspects of commonsense
reasoning, such as physical interaction, everyday concepts, and their
interrelations. RACE [19] provides a source of reading comprehension
challenges. WinoGrande [34] is designed to test the model on resolving
coreferences and understanding nuanced relationships in text. This dataset
with its unique fill-in-a-blank tasks, inherently aligns with open-ended
question formats, negating the need for our multi-stage filtering process. For
other datasets, questions are filtered using gpt-4-0125-preview using prompts
from Table 1. The prompts for both MCQ and OSQ on each dataset are in Appendix
D.

###  5.3 Evaluation

Table 4: Comparison of multiple choice (MCQ) and open style questions (OSQ)
accuracy.

Dataset | GPT-4 | GPT-3.5 | Gemini Pro | Claude-3 O | Mistral L
---|---|---|---|---|---
| MCQ | OSQ | MCQ | OSQ | MCQ | OSQ | MCQ | OSQ | MCQ | OSQ
MMLU | 87.28 | 74.77 | 71.25 | 65.38 | 65.71 | 56.04 | 83.52 | 70.23 | 79.50 | 68.76
ARC | 95.54 | 82.68 | 90.64 | 78.42 | 90.96 | 72.35 | 97.50 | 75.47 | 89.96 | 72.32
HellaSwag | 90.98 | 24.35 | 63.84 | 29.99 | 69.05 | 25.69 | 96.04 | 20.79 | 81.78 | 24.47
WinoGrande | 84.14 | 66.22 | 78.77 | 64.56 | 66.85 | 56.35 | 81.69 | 63.54 | 75.45 | 56.83
PIQA | 96.41 | 61.64 | 84.34 | 54.89 | 83.33 | 47.70 | 97.41 | 59.05 | 83.33 | 61.21
CommonsenseQA | 84.93 | 62.96 | 79.15 | 67.89 | 66.62 | 50.56 | 86.76 | 63.66 | 69.58 | 55.35
Race | 92.02 | 67.05 | 84.80 | 60.11 | 87.73 | 61.02 | 93.04 | 66.22 | 89.97 | 70.17
MedMCQA | 72.65 | 51.81 | 58.02 | 41.42 | 58.02 | 35.89 | 72.91 | 49.14 | 66.05 | 43.44
OpenbookQA | 94.30 | 60.29 | 83.71 | 49.90 | 86.97 | 52.55 | 93.48 | 52.95 | 88.19 | 58.66
Average | 88.69 | 61.31 | 78.28 | 56.95 | 75.03 | 50.91 | 90.26 | 57.89 | 80.42 | 56.80

Our assessment approach for both MCQ and OSQ aligns with widely recognized
evaluation frameworks and leaderboards for LLMs. The evaluation of MCQ is
conducted utilizing the OpenAI Evals framework [26] with the zero-shot
setting, which involves comparing the generated response with the ground truth
ID. In contrast, for evaluating responses to open-ended questions, we employ
the gpt-4-0125-preview model to determine the correctness of responses
generated by LLMs relative to a pre-established ground truth answer from the
dataset using the prompt from Table 1.

The results in Table 4 and Figure 4 are based on filtered questions. They show
that every model experiences a significant drop in the accuracy for OSQ
compared to MCQ. On average, the accuracy of OSQ is lower than MCQ by about
25% for all models. This result can correlate with our concern that the model
will “randomly guess” to correct choices but it cannot answer. This
discrepancy in performance between OSQ and MCQ is not necessarily a negative
reflection of the models’ overall capabilities. Instead, it can be viewed as a
true comparison of the models’ abilities to process and understand diverse
types of questions.

The most significant difference in models between OSQ and MCQ is observed for
Claude-3 Opus, by 31%. The dataset with the largest fall between MCQ and OSQ
is HellaSwag. This is because of the type of questions in this dataset. It
asks to choose the most plausible continuation for the scenarios presented.
Evaluating the OSQ responses of LLMs against the ground truth in this dataset
presents a significant challenge due to the different plausible completions.
It means that a multitude of valid and contextually appropriate answers can
exist, which makes it difficult to evaluate with single-choice ground truth.
This contrasts with WinoGrande, which consists of questions that require fill-
in-the-blank in sentences with correct words. As a result, HellaSwag does not
seem well-suited for open-style questions, and we have chosen to omit it from
our final leaderboard.

Table 5: Open-LLM Leaderboard for Large-scale Models. WG, CSQA, OBQA, and HS
represent WinoGrande, CommonsenseQA, OpenbookQA, and HellaSwag respectively.
We did not include HellaSwag results in the overall accuracy as the evaluation
difficulties mentioned in Sec. 5.3.

| Overall | MMLU | ARC | WG | PIQA | CSQA | Race | MedMCQA | OBQA | HS
---|---|---|---|---|---|---|---|---|---|---
GPT-4o | 70.15 | 79.09 | 86.31 | 72.22 | 60.34 | 70.28 | 67.87 | 57.85 | 67.21 | –
GPT-4-1106-preview | 65.93 | 74.77 | 82.68 | 66.22 | 61.64 | 62.96 | 67.05 | 51.81 | 60.29 | 24.35
Claude-3 Opus | 62.53 | 70.23 | 75.47 | 63.54 | 59.05 | 63.66 | 66.22 | 49.14 | 52.95 | 20.79
Mistral Large | 60.84 | 68.76 | 72.32 | 56.83 | 61.21 | 55.35 | 70.17 | 43.44 | 58.66 | 24.47
GPT-3.5 | 60.32 | 65.38 | 78.42 | 64.56 | 54.89 | 67.89 | 60.11 | 41.42 | 49.90 | 29.99
Gemini 1.0 Pro | 54.06 | 56.04 | 72.35 | 56.35 | 47.70 | 50.56 | 61.02 | 35.89 | 52.55 | 25.69
Llama3-70b-Instruct | 52.92 | 59.67 | 67.09 | 57.14 | 43.10 | 55.49 | 58.21 | 41.67 | 40.94 | –

Table 6: Open-LLM Leaderboard for small-scale model regime.

Model | Overall | MMLU | ARC | WG | PIQA | CSQA | Race | MedMCQA | OBQA
---|---|---|---|---|---|---|---|---|---
Qwen1.5 (1.8B) | 21.68 | 9.99 | 15.84 | 40.96 | 15.52 | 31.13 | 34.91 | 4.70 | 20.37
Gemma (2B) | 16.66 | 17.52 | 23.93 | 16.10 | 15.09 | 27.46 | 14.32 | 4.57 | 14.26
SlimPajama-DC (1.3B) | 9.60 | 9.22 | 14.95 | 14.76 | 5.32 | 9.01 | 16.19 | 1.68 | 5.70
RedPajama (1.3B) | 9.00 | 9.21 | 13.50 | 16.97 | 0.86 | 11.41 | 14.35 | 1.86 | 3.87
OLMo (1.2B) | 8.85 | 8.54 | 13.18 | 6.16 | 8.05 | 13.10 | 13.61 | 2.07 | 6.11
Pythia (1.4B) | 8.79 | 9.66 | 14.69 | 11.52 | 4.17 | 9.01 | 12.76 | 3.19 | 5.30
TinyLlama (1.1B) | 8.45 | 8.94 | 13.31 | 12.23 | 3.59 | 6.06 | 16.7 | 2.07 | 4.68
OPT (1.3B) | 7.89 | 7.40 | 11.83 | 12.47 | 4.48 | 7.61 | 13.61 | 1.25 | 4.48
GPT-Neo (1.3B) | 7.42 | 6.94 | 9.69 | 10.81 | 4.31 | 6.34 | 13.75 | 2.63 | 4.89
Cerebras-GPT (1.3B) | 4.86 | 5.37 | 4.43 | 9.31 | 2.16 | 6.20 | 6.90 | 1.04 | 3.46

###  5.4 Leaderboard and Arena

The overall ranking of models for our benchmark is represented in Table 6 and
Table 6. The performance of GPT-4o overall demonstrates its leading edge, with
an accuracy of 70.15%, which indicates its robustness in open-style question
answering tasks compared to other models. It is followed by GPT-4-1106-preview
with 65.93%, and Claude-3 Opus with 62.68%. These results highlight the
advanced capabilities of the GPT-4 series. Mid-tier models like Mistral Large
and GPT-3.5 perform well but are not on par with the top performers. On the
other hand, models like Gemini 1.0 Pro and Llama3-70b-Instruct lag behind in
terms of the capabilities to answer the open-style questions.

The performance evaluation of smaller-scale LLMs reveals that Qwen1.5 leads
with an overall accuracy of 21.68%, significantly outperforming the other
models in this category. Gemma follows with 16.66%, indicating a considerable
gap in performance compared to the top model. The remaining models score below
10.00%, highlighting their limited abilities to answer the open-style
questions. Almost all of the models struggle significantly with questions from
MedMCQA dataset, showing an accuracy below of 5%.

##  6 Conclusion

We proposed Open-LLM-Leaderboard for LLM evaluation and comprehensively
examined its efficacy using open-style questions from nine datasets on OSQ-
bench. Different from previous works that rely on human evaluation or
thousands of crowd users on Chatbot Arena, we can have a benchmark for chat
LLMs in a fast, automatic, and cheap scheme. Our results show a highly
correlated level of agreement with humans, indicating a foundation for an LLM-
based evaluation benchmark and framework using open-style questions.

## Limitations and Ethics Statement

We have discussed multiple advantages of employing open-style questions over
multiple-choice questions used in prior works. However, the LLM Leaderboard,
as a tool for evaluating and benchmarking LLMs, has several common limitations
itself. Firstly, the performance metrics used may not fully capture the
nuanced capabilities of each model, especially in areas that require an
understanding of context, creativity, or common sense reasoning. Secondly, the
benchmark datasets may not be comprehensive enough to cover all possible
domains and scenarios, leading to a potential bias towards certain types of
questions or tasks. Thirdly, due to the rapidly evolving nature of the field,
models may quickly become outdated, meaning the leaderboard may not always
reflect the most current state of the art. Since our benchmark utilizes public
datasets and our corpus consists of questions and answers, user privacy
concerns are minimal.

## References

  * [1] AI@Meta.  Llama 3 model card.  2024\.
  * [2] Anthropic.  Model card and evaluations for claude models, 2023.
  * [3] Anthropic.  https://www.anthropic.com/claude, 2024.
  * [4] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu.  Qwen technical report.  arXiv preprint arXiv:2309.16609, 2023.
  * [5] Edward Beeching, Clémentine Fourrier, Nathan Habib, Sheon Han, Nathan Lambert, Nazneen Rajani, Omar Sanseviero, Lewis Tunstall, and Thomas Wolf.  Open llm leaderboard.  https://huggingface.co/spaces/HuggingFaceH4/open_llm_leaderboard, 2023.
  * [6] Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al.  Pythia: A suite for analyzing large language models across training and scaling.  In International Conference on Machine Learning, pages 2397–2430. PMLR, 2023.
  * [7] Yonatan Bisk, Rowan Zellers, Ronan Le bras, Jianfeng Gao, and Yejin Choi.  Piqa: Reasoning about physical commonsense in natural language.  Proceedings of the AAAI Conference on Artificial Intelligence, 34(05):7432–7439, Apr. 2020.
  * [8] Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman.  GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021.  If you use this software, please cite it using these metadata.
  * [9] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, et al.  Language models are few-shot learners.  In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 1877–1901. Curran Associates, Inc., 2020.
  * [10] Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E. Gonzalez, Ion Stoica, and Eric P. Xing.  Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality, March 2023.
  * [11] John Chung, Ece Kamar, and Saleema Amershi.  Increasing diversity while maintaining accuracy: Text data generation with large language models and human interventions.  In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2023.
  * [12] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have solved question answering? try arc, the ai2 reasoning challenge.  ArXiv, abs/1803.05457, 2018.
  * [13] Jacob Cohen.  A coefficient of agreement for nominal scales.  Educational and Psychological Measurement, 20(1):37–46, 1960.
  * [14] Nolan Dey, Gurpreet Gosal, Zhiming, Chen, Hemant Khachane, William Marshall, Ribhu Pathria, Marvin Tom, and Joel Hestness.  Cerebras-gpt: Open compute-optimal language models trained on the cerebras wafer-scale cluster.  arXiv preprint:2304.03208, 2023.
  * [15] Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang Sutawika, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou.  A framework for few-shot language model evaluation, 12 2023.
  * [16] Google.  https://ai.google.dev/, 2023.
  * [17] Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack Hessel, Tushar Khot, William Merrill, Jacob Morrison, Niklas Muennighoff, Aakanksha Naik, Crystal Nam, Matthew E. Peters, Valentina Pyatkin, Abhilasha Ravichander, Dustin Schwenk, Saurabh Shah, Will Smith, Nishant Subramani, Mitchell Wortsman, Pradeep Dasigi, Nathan Lambert, Kyle Richardson, Jesse Dodge, Kyle Lo, Luca Soldaini, Noah A. Smith, and Hannaneh Hajishirzi.  Olmo: Accelerating the science of language models.  Preprint, 2024.
  * [18] Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt.  Measuring massive multitask language understanding.  Proceedings of the International Conference on Learning Representations (ICLR), 2021.
  * [19] Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy.  RACE: Large-scale ReAding comprehension dataset from examinations.  In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, 2017.
  * [20] Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, et al.  Starcoder: may the source be with you!  arXiv preprint arXiv:23.05.061161, 2023.
  * [21] Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Alpacaeval: An automatic evaluator of instruction-following models.  https://github.com/tatsu-lab/alpaca_eval, 2023.
  * [22] Yixin Liu, Kejian Shi, Katherine S He, Longtian Ye, Alexander R. Fabbri, Pengfei Liu, Dragomir Radev, and Arman Cohan.  On learning to summarize with large language models as references.  arXiv preprint arXiv:2305.14239, 2023.
  * [23] Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal.  Can a suit of armor conduct electricity? a new dataset for open book question answering.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018.
  * [24] Mistral.  https://chat.mistral.ai/chat, 2024.
  * [25] MosaicML.  Mpt-1b redpajama-200b.  https://huggingface.co/mosaicml/mpt-1b-redpajama-200b.  Accessed: 2024-04-29.
  * [26] OpenAI.  Openai evals.  https://github.com/openai/evals.
  * [27] OpenAI.  https://chat.openai.com/chat, 2022.
  * [28] OpenAI.  Gpt-4 technical report.  arxiv preprint arXiv:2303.08774, 2024.
  * [29] Ankit Pal, Logesh Kumar Umapathi, and Malaikannan Sankarasubbu.  Medmcqa: A large-scale multi-subject multi-choice dataset for medical domain question answering.  In Proceedings of the Conference on Health, Inference, and Learning, 2022.
  * [30] Pouya Pezeshkpour and Estevam Hruschka.  Large language models sensitivity to the order of options in multiple-choice questions.  arXiv preprint arXiv:2308.11483, 2023.
  * [31] Joshua Robinson, Christopher Rytting, and David Wingate.  Leveraging large language models for multiple choice question answering.  ArXiv, abs/2210.12353, 2022.
  * [32] Joshua Robinson, Christopher Michael Rytting, and David Wingate.  Leveraging large language models for multiple choice question answering.  arXiv preprint arXiv:2210.12353, 2023.
  * [33] Baptiste Rozière, Jonas Gehring, Fabian Gloeckle, Sten Sootla, Itai Gat, Xiaoqing Ellen Tan, Yossi Adi, Jingyu Liu, Romain Sauvestre, Tal Remez, Jérémy Rapin, Artyom Kozhevnikov, Ivan Evtimov, Joanna Bitton, Manish Bhatt, Cristian Canton Ferrer, Aaron Grattafiori, Wenhan Xiong, Alexandre Défossez, Jade Copet, Faisal Azhar, Hugo Touvron, Louis Martin, Nicolas Usunier, Thomas Scialom, and Gabriel Synnaeve.  Code llama: Open foundation models for code.  arXiv preprint arXiv:2308.12950, 2024.
  * [34] Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.  Winogrande: an adversarial winograd schema challenge at scale.  Commun. ACM, 64(9):99–106, aug 2021.
  * [35] Zhiqiang Shen, Tianhua Tao, Liqun Ma, Willie Neiswanger, Joel Hestness, Natalia Vassilieva, Daria Soboleva, and Eric Xing.  Slimpajama-dc: Understanding data combinations for llm training.  arXiv preprint arXiv:2309.10818, 2023.
  * [36] Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant.  CommonsenseQA: A question answering challenge targeting commonsense knowledge.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), 2019.
  * [37] Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto.  Stanford alpaca: An instruction-following llama model.  https://github.com/tatsu-lab/stanford_alpaca, 2023.
  * [38] Gemini Team.  Gemini: A family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.
  * [39] Gemma Team.  Gemma: Open models based on gemini research and technology.  arXiv preprint arXiv:2403.08295, 2024.
  * [40] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [41] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al.  Llama 2: Open foundation and fine-tuned chat models.  arXiv preprint arXiv:2307.09288, 2023.
  * [42] Peiyi Wang, Lei Li, Liang Chen, Zefan Cai, Dawei Zhu, Binghuai Lin, Yunbo Cao, Qi Liu, Tianyu Liu, and Zhifang Sui.  Large language models are not fair evaluators.  arXiv preprint arXiv:2305.17926, 2023.
  * [43] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew.  Huggingface’s transformers: State-of-the-art natural language processing.  CoRR, abs/1910.03771, 2019.
  * [44] Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi.  HellaSwag: Can a machine really finish your sentence?  In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, 2019.
  * [45] Biao Zhang, Barry Haddow, and Alexandra Birch.  Prompting large language model for machine translation: A case study.  In Proceedings of the 40th International Conference on Machine Learning, 2023.
  * [46] Peiyuan Zhang, Guangtao Zeng, Tianduo Wang, and Wei Lu.  Tinyllama: An open-source small language model.  arXiv preprint:2401.02385, 2024.
  * [47] Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shuster, Daniel Simig, Punit Singh Koura, Anjali Sridhar, Tianlu Wang, and Luke Zettlemoyer.  Opt: Open pre-trained transformer language models.  arXiv preprint:2205.01068, 2022.
  * [48] Chujie Zheng, Hao Zhou, Fandong Meng, Jie Zhou, and Minlie Huang.  Large language models are not robust multiple choice selectors.  arXiv preprint arXiv:2309.03882, 2024.
  * [49] Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica.  Judging llm-as-a-judge with mt-bench and chatbot arena.  arXiv preprint arXiv:2306.05685, 2023.
  * [50] Wenhao Zhu, Hongyi Liu, Qingxiu Dong, Jingjing Xu, Shujian Huang, Lingpeng Kong, Jiajun Chen, and Lei Li.  Multilingual machine translation with large language models: Empirical results and analysis.  arXiv preprint 2304.04675, 2023.

## Appendix

##  Appendix A Reproducibility Statement

We will make all our filtered open-style data (MMLU, ARC, HellaSwag,
WinoGrande, PIQA, CommonsenseQA, Race, MedMCQA, and OpenbookQA) used in our
experiments of Sec. 5 and preprocessing scripts publicly available. Detailed
data statistics are provided in Sec. 4.1. Considering the potential high costs
associated with gathering and reproducing our LLM response data from the
ground up, we will make available all responses from the various LLMs and
their corresponding evaluation results to support and simplify the
reproducibility of our work. The OpenAI APIs we used include
gpt-3.5-turbo-1106, gpt-4.0-1106-preview, gpt-4o (for response collection),
and gpt-4.0-0125-preview (for filtering and post-evaluation); Claude 3:
claude-3-opus-20240229; Gemini-Pro: gemini-pro, and Mistral: mistral-large-
latest.

##  Appendix B More Results on Gemini Pro and Stage1 Filtering

Figure 6: Performance comparison of Gemini Pro on multiple-choice and open-
style response questions across diverse datasets, as shown by the count of
correct and incorrect answers in the left bar chart and model accuracy in the
right radar chart.  Figure 7: Initial filtering results across datasets,
showcasing the percentages of questions that are suitable (Yes) for open-style
formatting and those that are not (No).

##  Appendix C More Sample Questions

We provide more sample questions that are correct for open-style and incorrect
for multiple choice in Table 7, and vice versa in Table 8. These examples
demonstrate how the model’s ability varies in handling different question
formats.

Table 7: Example of an Incorrect MCQ and a Correct OSQ in MMLU.

Question that is correct for OSQ and incorrect for MCQ: How many CO2 and ATP
molecules are formed during one complete turn of the tricarboxylic acid cycle
(Krebs’ cycle)?
---
Options: A. 2CO2 and 2ATP B. 2CO2 and 16ATP C. 2CO2 and 12ATP D. 2CO2 and 1ATP
Ground Truth:  D
MCQ_Answer:  A
OSQ_Answer: During one complete turn of the tricarboxylic acid cycle (also
known as the Krebs’ cycle or the citric acid cycle), a series of chemical
reactions occur in the mitochondria of cells that result in the production of
carbon dioxide (CO2) and adenosine triphosphate (ATP), among other molecules.
For each acetyl-CoA molecule that enters the cycle, the following are
produced:
\- 2 molecules of CO2: Each acetyl-CoA contributes two carbons to the cycle,
which are eventually released as two molecules of carbon dioxide.
\- 1 molecule of ATP: One molecule of ATP (or an equivalent, such as GTP,
depending on the cell type) is produced directly in the cycle through
substrate-level phosphorylation.
However, the cycle also produces reduced coenzymes, specifically NADH and
FADH2, which are used in the electron transport chain to produce additional
ATP through oxidative phosphorylation. For each acetyl-CoA that enters the
cycle:
\- 3 molecules of NADH are produced, which can be used to generate
approximately 2.5 ATP each through the electron transport chain.
\- 1 molecule of FADH2 is produced, which can be used to generate
approximately 1.5 ATP through the electron transport chain.
Therefore, while only 1 ATP is produced directly in the cycle per acetyl-CoA,
the NADH and FADH2 generated can lead to the production of additional ATP in
the electron transport chain. The exact number of ATP molecules produced can
vary depending on the efficiency of the electron transport chain and the ATP
synthase complex in different organisms and cell types.

Table 8: Example of a Correct MCQ and an Incorrect OSQ in MMLU.

Question that is correct for MCQ and incorrect for OSQ: A tree is a connected
graph with no cycles. How many nonisomorphic trees with 5 vertices exist?
---
Options: A. 1 B. 2 C. 3 D. 4
Ground Truth:  C
MCQ_Answer:  C
OSQ_Answer: To determine the number of nonisomorphic trees with 5 vertices, we
can use Cayley’s formula, which states that the number of trees on $n$ labeled
vertices is $n^{n-2}$. However, this formula gives the number of trees when
vertices are labeled, meaning that it counts different labelings of the same
tree shape as distinct. Since we are interested in nonisomorphic trees (which
do not consider different labelings as distinct), we cannot directly use
Cayley’s formula.
Instead, we need to enumerate the trees by considering their possible
structures. For a tree with 5 vertices, we can have the following
configurations:
1\. A linear chain (path) where each vertex is connected to two others except
for the two end vertices, which are connected to only one other vertex. This
is one possible tree.
2\. A "star" tree where one central vertex is connected to the other four
vertices. This is another possible tree.
3\. A tree where one vertex is connected to three others, and one of those
three is connected to the fifth vertex. This structure can be seen as a "T"
shape with an extra branch on one of the endpoints. There are three
nonisomorphic trees of this type because the branch can be on any of the three
endpoints.
4\. A tree where two vertices are connected to a central vertex, and each of
those two vertices is connected to one other vertex (creating a structure with
a central vertex, two intermediate vertices, and two leaf vertices). This
structure can be seen as a "Y" shape. There is only one nonisomorphic tree of
this type because any permutation of the leaf vertices does not create a
fundamentally different tree structure.
Adding these up, we have:
1 (linear chain) + 1 (star) + 3 (T-shaped with an extra branch) + 1 (Y-shaped)
= 6 nonisomorphic trees with 5 vertices.

##  Appendix D Prompts for Different Datasets

The following are examples of prompts for different datasets. They allow
models to better understand and respond to specific contexts and objectives in
each dataset.

Table 9: Prompt for MMLU dataset.

MCQ Prompt: The following is the multiple choice question about abstract
algebra. Please select the correct answer from the options A, B, C, D. For
example, if you think the correct answer is A, your response should be ’A’.
---
Question: Find the degree for the given field extension Q(sqrt(2), sqrt(3),
sqrt(18)) over Q.
A. 0
B. 4
C. 2
D. 6
Answer:
Open-Style Prompt: Answer the following question about abstract algebra.
Question: Find the degree for the given field extension Q(sqrt(2), sqrt(3),
sqrt(18)) over Q.
Answer:

Table 10: Prompt for ARC dataset.

MCQ Prompt: The following is the multiple choice question. Please select the
correct answer from the options A, B, C, D. For example, if you think the
correct answer is A, your response should be ’A’.
---
Question: Tiny organisms called plankton live in oceans. Some plankton can
take energy from the Sun and turn it into food. How are plankton most useful
to the animals that live in the ocean?
A. Plankton are colorful.
B. Plankton clean the water.
C. Plankton release oxygen.
D. Plankton reproduce quickly.
Answer:
Open-Style Prompt: Answer the following question.
Question: Tiny organisms called plankton live in oceans. Some plankton can
take energy from the Sun and turn it into food. How are plankton most useful
to the animals that live in the ocean?
Answer:

Table 11: Prompt for CommonsenseQA dataset.

MCQ Prompt: The following is the multiple choice question. Please select the
correct answer from the options A, B, C, D, E. For example, if you think the
correct answer is A, your response should be ’A’.
---
Question: A revolving door is convenient for two direction travel, but it also
serves as a security measure at a what?
A. bank
B. library
C. department store
D. mall
E. New York
Answer:
Open-Style Prompt: You will be presented with a variety of questions that
require an understanding of everyday scenarios, human behaviors, and common
sense. Your task is to provide the best possible answer to each question based
solely on your understanding and reasoning.
Question: A revolving door is convenient for two direction travel, but it also
serves as a security measure at a what?
Answer:

Table 12: Prompt for MedMCQA dataset.

MCQ Prompt: The following is the multiple choice question about medicine.
Please select the correct answer from the options A, B, C, D. For example, if
you think the correct answer is A, your response should be ’A’.
---
Question: Modulus of elasticity means:
A. Rigidity or stiffness of the material
B. Ability to be stretched with permanent deformation
C. Ductility of a material
D. Malleability of the metal
Answer:
Open-Style Prompt: Answer the following question about medicine.
Question: Modulus of elasticity means:
Answer:

Table 13: Prompt for HellaSwag dataset.

MCQ Prompt: The following is the multiple choice question. Please select the
correct answer from the options A, B, C, D. For example, if you think the
correct answer is A, your response should be ’A’.
---
Question: How to clean your rv windows and mirrors fast without using any
spray. you
A. also have a bucket that you spray paint a window in.
B. can reach for a running water hose and clean the inside of your rv quickly.
C. get a wash cloth and you put it under the faucet to get wet and then you
rinse it out so it’s not soaking.
D. meticulously clean the window in the glass shop and then take the plastic
off and start taking the hood off.
Answer:
Open-Style Prompt: Imagine you are provided with a scenario or a partial story
taken from everyday life or a common activity. Your task is to continue this
story or scenario in a way that makes the most sense based on what typically
happens in such situations. Please complete the sentence.
Question: How to clean your rv windows and mirrors fast without using any
spray. you
Answer:

Table 14: Prompt for OpenbookQA dataset.

MCQ Prompt: The following is the multiple choice question. Please select the
correct answer from the options A, B, C, D. For example, if you think the
correct answer is A, your response should be ’A’.
---
Question: what system is needed for a body to get its needed supply of the gas
humans breathe in?
A. the circulatory system
B. the digestive system
C. the school system
D. central nervous system
Answer:
Open-Style Prompt: Consider common scenarios or outcomes that fit the context
of the sentence. Attempt to logically complete the sentences based on common
knowledge and reasoning.
Question: what system is needed for a body to get its needed supply of the gas
humans breathe in?
Answer:

Table 15: Prompt for PIQA dataset.

MCQ Prompt: The following is the multiple choice question. Please select the
correct answer from the options A, B. For example, if you think the correct
answer is A, your response should be ’A’.
---
Question: How do I ready a guinea pig cage for it’s new occupants?
A. Provide the guinea pig with a cage full of a few inches of bedding made of
ripped paper strips, you will also need to supply it with a water bottle and a
food dish.
B. Provide the guinea pig with a cage full of a few inches of bedding made of
ripped jeans material, you will also need to supply it with a water bottle and
a food dish.
Answer:
Open-Style Prompt: Consider common scenarios or outcomes that fit the context
of the sentence. Attempt to logically complete the sentences based on common
knowledge and reasoning.
Question: How do I ready a guinea pig cage for it’s new occupants?
Answer:

Table 16: Prompt for Race dataset.

MCQ Prompt: I will give you a passage with multiple-choice question. Please
select the correct answer from the options A, B, C, D. For example, if you
think the correct answer is A, your response should be ’A’.
---
Passage:...
Question: What did Nancy try to do before she fell over?
A. Measure the depth of the river
B. Look for a fallen tree trunk
C. Protect her cows from being drowned
D. Run away from the flooded farm
Answer:
Open-Style Prompt: I will give you passage with question. Please, answer the
question.
Passage:...
Question: What did Nancy try to do before she fell over?
Answer:

Table 17: Prompt for WinoGrande dataset.

MCQ Prompt: The following is the multiple choice question. Please put the
correct words in place of _. Your response should include only the option
without any justification or reasoning. Please select the correct answer from
the options A, B.
---
Question: Sarah was a much better surgeon than Maria so _ always got the
easier cases.
A. Sarah
B. Maria
Answer:
Open-Style Prompt: Please put the correct words in place of _. Give only the
word that fits the sentence.
Question: Sarah was a much better surgeon than Maria so _ always got the
easier cases.
Answer:


# First experimental confirmation of the CH3O + H2CO $\to$ CH3OH + HCO
reaction: expanding the CH3OH formation mechanism in interstellar ices

Julia C. Santos Laboratory for Astrophysics, Leiden Observatory, Leiden
University, PO Box 9513, 2300 RA Leiden, The Netherlands  Ko-Ju Chuang
Laboratory for Astrophysics, Leiden Observatory, Leiden University, PO Box
9513, 2300 RA Leiden, The Netherlands  Thanja Lamberts Leiden Institute of
Chemistry, Gorlaeus Laboratories, Leiden University, PO Box 9502, 2300 RA
Leiden, The Netherlands  Laboratory for Astrophysics, Leiden Observatory,
Leiden University, PO Box 9513, 2300 RA Leiden, The Netherlands  Gleb Fedoseev
Research Laboratory for Astrochemistry, Ural Federal University, Kuibysheva
St. 48, 620026 Ekaterinburg, Russia  Sergio Ioppolo School of Electronic
Engineering and Computer Science, Queen Mary University of London, Mile End
Road, London E1 4NS, UK  Harold Linnartz Laboratory for Astrophysics, Leiden
Observatory, Leiden University, PO Box 9513, 2300 RA Leiden, The Netherlands

###### Abstract

The successive addition of H atoms to CO in the solid phase has been hitherto
regarded as the primary route to form methanol in dark molecular clouds.
However, recent Monte Carlo simulations of interstellar ices alternatively
suggested the radical-molecule H-atom abstraction reaction
$\ce{CH3O}+\ce{H2CO}\to\ce{CH3OH}+\ce{HCO}$, in addition to
$\ce{CH3O}+\ce{H}\to\ce{CH3OH}$, as a very promising and possibly dominating
(70–90%) final step to form CH3OH in those environments. Here, we compare the
contributions of these two steps leading to methanol by experimentally
investigating hydrogenation reactions on H2CO and D2CO ices, which ensures
comparable starting points between the two scenarios. The experiments are
performed under ultrahigh vacuum conditions and astronomically relevant
temperatures, with H:H2CO (or D2CO) flux ratios of 10:1 and 30:1. The radical-
molecule route in the partially deuterated scenario,
$\ce{CHD2O}+\ce{D2CO}\to\ce{CHD2OD}+\ce{DCO}$, is significantly hampered by
the isotope effect in the D-abstraction process, and can thus be used as an
artifice to probe the efficiency of this step. We observe a significantly
smaller yield of $\ce{D2CO}+\ce{H}$ products in comparison to
$\ce{H2CO}+\ce{H}$, implying that the CH3O-induced abstraction route must play
an important role in the formation of methanol in interstellar ices.
Reflection-Absorption InfraRed Spectroscopy (RAIRS) and Temperature Programmed
Desorption-Quadrupole Mass Spectrometry (TPD-QMS) analyses are used to
quantify the species in the ice. Both analytical techniques indicate constant
contributions of $\sim$80% for the abstraction route in the 10-16 K interval,
which agrees well with the Monte Carlo conclusions. Additional H2CO + D
experiments confirm these conclusions.

astrochemistry — ISM: molecules — molecular processes — methods: laboratory:
solid state — techniques: spectroscopic

##  1 Introduction

Methanol (CH3OH) is abundantly detected in interstellar environments and is
one of the main components of interstellar ices. Its formation is tightly
connected to the evolution of molecular clouds. In these dense and cold
regions, CO molecules present in the gas phase freeze-out and form an apolar
coating on top of icy grains, largely comprising of H2O and CO2 ice. Methanol
is then efficiently formed through the hydrogenation of CO in the solid phase
(Tielens & Hagen, 1982; Charnley et al., 1992; Hiraoka et al., 1994; Watanabe
& Kouchi, 2002; Fuchs et al., 2009). The presence of methanol in CO-rich ice
is supported by a series of comparisons between laboratory and observational
evidence (e.g., Bottinelli et al. 2010; Cuppen et al. 2011; Penteado et al.
2015). Furthermore, as opposed to CO hydrogenation, neither gas-phase nor
alternative solid-phase routes can explain the observed CH3OH abundances
(Geppert et al., 2005; Garrod et al., 2006; Watanabe et al., 2007). As a
secondary mechanism, it has also been shown to form in H2O-rich ices prior to
the heavy CO freeze-out—thus in an earlier evolutionary stage of the cloud
(Bergner et al., 2017; Qasim et al., 2018; Potapov et al., 2021; Molpeceres et
al., 2021).

So far, the main proposed hydrogenation route to form methanol is through
successive addition reactions of H atoms to CO ice, as follows:

$\ce{CO}+\ce{H}\to\ce{HCO}$ (1) $\ce{HCO}+\ce{H}\to\ce{H2CO}$ (2)
$\ce{H2CO}+\ce{H}\to\ce{CH3O}$ (3) $\ce{CH3O}+\ce{H}\to\ce{CH3OH}.$ (4)

Here, abstraction reactions induced by H atoms also take place, converting
CH3OH into CO through H2CO (Hidaka et al., 2009; Chuang et al., 2016).
However, recent theoretical works that combine a full reaction network for the
hydrogenation of CO and microscopic kinetic Monte Carlo simulations of
interstellar ices suggest a dominating alternative to reaction (4), that is
the radical-molecule route (Álvarez-Barcia et al., 2018; Simons et al., 2020):

$\ce{CH3O}+\ce{H2CO}\to\ce{CH3OH}+\ce{HCO},$ (5)

with a contribution of 70–90% with respect to the remainder of the routes to
form methanol. Reaction (5) can take place upon only one hydrogen-atom
addition to H2CO, as long as another H2CO molecule lies in the vicinity of the
formed CH3O. Comparatively, reaction (4) warrants that an additional H atom
diffuses through the ice and reacts with CH3O. The overall contribution from
each route will therefore be dictated by the availability of H, CH3O, and H2CO
in the ice. Although the theoretical results represent a significant paradigm
change in the final step of the CO hydrogenation formation pathway of methanol
in interstellar ices, they have not yet been systematically investigated in
the laboratory. Experimental verification is however possible by exploring the
kinetic isotope effect of the reactions involving deuterated species (i.e.,
D2CO and D), as will be discussed below.

At the typical low temperatures of molecular clouds, quantum-mechanical
tunneling governs the activation of chemical reactions. Under these
conditions, D-abstraction reactions are significantly hampered in comparison
to their H-abstraction counterparts (e.g., Nagaoka et al. 2005; Hidaka et al.
2007; Goumans 2011), on account of the so-called “kinetic isotope effect”.
Reaction (5) is a radical-molecule route that involves an H-abstraction
reaction from H2CO with an activation barrier of $\Delta E\sim 2670$ K
(Álvarez-Barcia et al., 2018). Thus, it most likely proceeds through quantum-
tunneling at such low temperatures. As a consequence, the rate constant of the
analog reaction with the deuterated formaldehyde (i.e., D-atom abstraction
from $\ce{D2CO}$) is expected to be severely hindered. In the present work, we
take advantage of this isotope effect to quantify the respective contributions
from the mechanisms (4) and (5) in forming methanol under molecular-cloud
conditions. We then confirm the dominance of the radical-molecule route
through complementary $\ce{H2CO}+\ce{D}$ experiments.

##  2 Experimental

All experiments are performed using the setup SURFRESIDE3, which has been
described in detail elsewhere (Ioppolo et al., 2013; Qasim et al., 2020). It
consists of an ultrahigh vacuum (UHV) chamber with a base pressure of $\sim
5\times 10^{-10}$ mbar. At its center, a gold-plated copper substrate is
mounted on the tip of a closed-cycle He cryostat that allows the temperature
of the substrate to vary between 8 and 450 K through resistive heating. The
temperature is monitored by two silicon diode sensors with a relative accuracy
of 0.5 K. We deposit either H2CO or D2CO simultaneously with thermalized H and
D atoms during the so-called codeposition experiments. The H2CO and D2CO
vapors are produced from paraformaldehyde (purity $95\%$, Sigma-Aldrich) and
paraformaldehyde-$d_{2}$ (purity $98\%$, Sigma-Aldrich) powders heated to
$\sim 60-100^{\circ}$ C in glass vacuum tubes. The hydrogen and deuterium
atoms are simultaneously generated by both a Hydrogen Atom Beam Source (HABS,
Tschersich 2000) and a Microwave Atom Source (MWAS, Anton et al. 2000), and
are thermally cooled to room temperature by colliding with the walls of nose-
shaped quartz pipes that are placed along the beam path in both atom sources.
A set of experiments comprising of 360 minutes of $\ce{H2CO}+\ce{H}$ or
$\ce{D2CO}+\ce{H}$ codepositions is performed at temperatures of 10, 12, 14
and 16 K. Additionally, analogous codeposition experiments of
$\ce{H2CO}+\ce{D}$ are also reported. The full complementary set of H2CO + D
and D2CO + D codepositions are not performed due to the low efficiency of the
D-atom addition reactions resulting from the kinetic isotope effect caused by
the slower rate of quantum tunneling of D atoms over H atoms (Hidaka et al.,
2009; Goumans, 2011).

During the experiments, the ices are monitored in situ by means of Fourier-
Transform Reflection-Absorption Infrared Spectroscopy (RAIRS) in the range of
700 to 4000 cm-1, with a resolution of 1 cm-1. After the codeposition
experiments, the sample is heated at a ramping rate of 5 K min-1. The desorbed
species are monitored by a Quadrupole Mass Spectrometer during Temperature-
Programmed Desorption experiments (TPD-QMS). To derive the column densities of
the species in the ice, we employ the modified Beer-Lambert law to convert
absorbance area to absolute abundance. We use band-strength values of
$A(\ce{H2CO})_{\nu=1727}\sim 3.6\times 10^{-17}$ cm molecule-1 and
$A(\ce{CH3OH})_{\nu=1030}\sim 3.1\times 10^{-17}$ cm molecule-1, which were
calibrated in the same setup by laser-interference measurements (Chuang et
al., 2018). For CHD2OH ($\nu$=1037 cm-1), the band strength is estimated by
multiplying $A$(CH3OH) by a calibration factor of 0.43 (Nagaoka et al., 2007).
Finally, the band strength of D2CO is obtained by comparing the IR absorbance
areas of the same amount of pure H2CO and D2CO ices. We experimentally derive
$A(\ce{D2CO})_{\nu=1679}\sim 2.8\times 10^{-17}$ cm molecule-1. More details
on this procedure are given in Appendix A. For a specific flux configuration,
the relative uncertainties of the molecule and H-atom fluxes are,
respectively, $<$9% and $<$4%. The error-bar estimation due to the
instrumental uncertainties of the RAIRS and TPD-QMS measurements is derived
from the integrated noise signals of blank experiments in the same band width.
Table 1 in Appendix B lists the experiments performed in this study and the
relative abundance of CHD2OH/CH3OH derived from the RAIRS and QMS-TPD
analyses.

##  3 Results and Discussion

In Figure 1, we present a schematic view of the reaction routes probed in our
experiments. To focus on reactions (4) and (5), we skip steps (1) and (2) and
start by directly depositing formaldehyde. This way, comparable starting
points between the two scenarios are obtained.

Figure 1: CH3OH formation scheme involving H2CO and D2CO. The gray dashed line
indicates the smaller efficiency of the deuterated radical-molecule reaction
with relation to route (5).

By using different isotope-labeled reactants (H2CO and D2CO), the importance
of reactions (4) and (4’) can be directly assessed. The yields of methanol
from H2CO through reaction (4) and from D2CO through (4’)—see Figure 1—should
be identical, since both radical-radical routes are barrierless. Conversely,
the formation of the isotope-labeled CHD2OD through reaction (5’) is expected
to be strongly affected by the isotope effect and to become negligible,
whereas (5) still takes place. Moreover, the hydrogen-addition steps
$\ce{H2CO}+\ce{H}$ and $\ce{D2CO}+\ce{H}$ have similar activation barriers of
2318 and 2253 K, respectively, and similar rate constants (Hidaka et al.,
2009; Goumans, 2011). Thus, they yield comparable amounts of CH3O and CHD2O.
The contributions of the two competing routes are hence obtained by comparing
the relative abundances between the total CH3OH yield of the H2CO ice
(reactions (4) and (5)) and that of CHD2OH in the D2CO ice (reaction (4’)).

The final infrared spectra of both $\ce{H2CO}+\ce{H}$ and $\ce{D2CO}+\ce{H}$
codepositions after 360 minutes at 12 K are shown in Figure 2. IR features of
H2CO are observed at 1167 (CH2 wag., $\nu=6$), 1249 (CH2 rock., $\nu=5$), 1500
(CH2 scis., $\nu=3$) and 1727 cm-1 (CO str., $\nu=2$). In the analog
deuterated experiments, D2CO peaks are detected at 988 (CD2 wag.), 1101 (CD2
scis.) and 1679 cm-1 (CO str.). In both cases the hydrogenated products CH3OH
and CHD2OH are detected by their CO stretching ($\nu=8$) features at 1030 and
1037 cm-1, respectively (Shimanouchi, 1972; Hidaka et al., 2009), which
indicates that the H-addition routes in Figure 1 proceed at a considerable
rate.

Figure 2: Final IR spectra of the $\ce{H2CO}+\ce{H}$ (red) and
$\ce{D2CO}+\ce{H}$ (black) codeposition experiments performed at 12 K. The
spectra were obtained after 360 minutes of deposition and are artificially
offset for clarity. Peak assignments are indicated with solid red (for
$\ce{H2CO}+\ce{H}$) and black (for $\ce{D2CO}+\ce{H}$) lines, while the dashed
gray line shows the absence of the CHD2OD $\nu\approx 2420$ cm-1 peak.

Even in the experimental configuration with the highest product yield, no
CHD2OD peaks (OD str. at $\nu_{1}\sim 2420$ cm-1, Nagaoka et al. 2007) could
be observed in the $\ce{D2CO}+\ce{H}$ spectrum. The detection of CHD2OH and
non-detection of CHD2OD are in line with the expected low efficiency of the
radical-molecule route for deuterated species and are consistent with the
results by Hidaka et al. (2009) on the exposure of H atoms to D2CO ice. That,
in addition to the fact that the CHD2OH peak is much smaller than the CH3OH
counterpart in the $\ce{H2CO}+\ce{H}$ experiment at equivalent conditions,
strongly suggests that reaction (4) is not the dominant route to form methanol
under the investigated experimental conditions. Instead, reaction (5) must
play a more prominent role in the formation of methanol. This observation
remains valid after converting the integrated peak areas from the RAIR spectra
to the species’ abundances, as shown in Table 1. From the RAIRS analyses, we
do not detect any bands related to the radicals HCO or CH2OH—nor their
deuterated isotopes—reinforcing the findings from previous studies that their
high reactivity results in low abundances (Watanabe & Kouchi, 2002; Fuchs et
al., 2009). The radical CH3O is also not observed in the H2CO + H experiments.
It should be noted, though, that the most intense IR features of its
isotopologue CHD2O overlap with those of the other reactants and products in
the spectra (see, e.g., Haupa et al. 2017), and therefore we cannot rule out
its presence in the D2CO + H experiments. Moreover, we observe a clear CO peak
($\nu=2138$ cm-1) in the $\ce{H2CO}+\ce{H}$ experiment, as well as HDCO
($\nu=1694$ cm-1) and H2CO ($\nu=1727$ cm-1) peaks in the $\ce{D2CO}+\ce{H}$
counterpart, in agreement with the results by, e.g., Hidaka et al. (2009) and
Chuang et al. (2016) that H-atom induced abstraction reactions take place for
the studied conditions (see, e.g., Figure 5 in Hidaka et al. 2009).

The TPD-QMS results of the experiments at 12 K are presented in Figure 3 as a
typical example of the obtained spectra. Since trying to account for the mass-
fragmentation patterns induced by electron impact may complicate the
interpretation of the data, only the ion signals of the products’ molecular
masses are selected to quantify their formation yield. For example, the CH3OH
formation is quantified by the mass signal m/z = 32, while the deuterated
species CHD2OH and CHD2OD are depicted by m/z = 34 and 35, respectively. All
methanol (-$d_{0}$–$d_{3}$) mass signals peak at $\sim$140 K. The small m/z =
32 signal that peaks at 160 K could be due to either CH3OH trapped in
background water or fragments originating from complex organic molecules
(Fedoseev et al., 2015; Chuang et al., 2016; He et al., in press). The flux of
background water deposited throughout the experiment is estimated to be
5.2$\times$1010 molecules cm-2 s-1, being a minor component of the ice that
does not affect the final results. The TPD spectra show that a much smaller
yield of CHD2OH is observed in comparison to CH3OH, and no CHD2OD is detected
above the instrumental detection limit, thereby confirming the RAIRS analysis.

Figure 3: Spectra of the TPD-QMS experiments obtained after codeposition of
$\ce{H2CO}+\ce{H}$ (upper) and $\ce{D2CO}+\ce{H}$ (lower) at 12 K for 360 min.
The m/z signals of 32 (red), 34 (blue) and 35 (green) correspond to,
respectively, CH3OH, CHD2OH and CHD2OD. No signal is detected for m/z = 35 at
the desorption temperature of methanol ($\sim$140 K).

Figure 4 presents the relative yields of CH3OH and CHD2OH derived from the
RAIRS and QMS-TPD analyses at 10, 12, 14 and 16 K. The methanol yield of the
hydrogenation reactions varies with the surface temperature, showing the
effective result of the competition between the increase in the diffusion of H
atoms and the decrease in their residence time on the ice as a function of
temperature from 10 to 16 K. The highest abundances of methanol in both
experiments $\ce{H2CO}+\ce{H}$ and $\ce{D2CO}+\ce{H}$ are found at 12 K, and
drop with increasing temperatures. A set of experiments performed at 20 K
results in CHD2OH yields that are not measurable, and thus is not reported
here. Therefore, the abundances are normalized to that of CH3OH at 12 K.
Similar observations have been reported in a previous study on the
hydrogenation of H2CO at temperatures up to 25 K (Chuang et al., 2016).

For the entire set of experiments, the RAIRS data show a considerably lower
abundance of CHD2OH from the $\ce{D2CO}+\ce{H}$ experiments compared to that
of CH3OH from $\ce{H2CO}+\ce{H}$. In fact, the abundance ratios of
CHD2OH/CH3OH between analogous experiments yield a constant value of $\sim$
0.16 for the temperature range between 10 and 16 K (see Table 1). This is
confirmed by the QMS-TPD data, which agree well within their uncertainties
with the results from the RAIRS analysis (CHD2OH/CH3OH $\sim$ 0.17). The
contributions of the reactions probed here are therefore independent of these
temperatures (10–16 K), which are of relevance to molecular clouds. This
observation is in accordance with the predictions of the models by Simons et
al. (2020). The similarity between both the RAIRS and TPD ratios also excludes
the possibility of the former being influenced by the heating of the
substrate.

Figure 4: Upper panel: Yields of CH3OH and CHD2OH relative to the column
density of CH3OH after 360 minutes of $\ce{H2CO}+\ce{H}$ codeposition at 10,
12, 14, and 16 K, as derived from the RAIRS data. Lower panel: Same as the
upper panel, but for the TPD-QMS data. The reported values are normalized to
the most abundant yield at 12 K.

Under the assumption that reactions (3) and (3’) result in similar amounts of
CH3O and CHD2O respectively, and since both reactions (4) and (4’) are
barrierless, the methanol yields from the abstraction route should be similar
in both $\ce{H2CO}+\ce{H}$ and $\ce{D2CO}+\ce{H}$ experiments;

$(4)\sim(4^{\prime}),$ (6)

where the parentheses denote the methanol yield of the reactions. The
discrepancy between the yields of CH3OH and CHD2OH under our experimental
conditions can therefore be used to estimate the contribution $C4$ of reaction
(4) with respect to the total yield of methanol:

$\begin{split}C4=\frac{(4)}{(4)+(5)}\sim\frac{(4^{\prime})}{(4)+(5)}\\\
\sim\frac{(\ce{CHD2OH})}{(\ce{CH3OH})}\sim[0.16-0.17].\end{split}$ (7)

The estimation described here assumes that the formation of CHD2OH and CH3OH
takes place predominantly through the reactions presented in Figure 1. Other
routes to form methanol, such as $\ce{CH2OH}+\ce{H}$, have very minor
contributions and are hence disregarded based on the inefficient formation of
CH2OH from H2CO hydrogenation (Song & Kästner 2017 and, e.g., Table 6 in
Simons et al. 2020). Accordingly, the contribution $C5$ of reaction (5)
corresponds to:

$C5=1-C4\sim[0.83-0.84].$ (8)

The experimentally derived value of $C5$ agrees well with the contribution of
$\sim$90% reported by Simons et al. (2020) from kinetic Monte Carlo
simulations of the hydrogenation of H2CO ices.

The H/H2CO flux ratios employed here were chosen based on the conditions of
the models by Simons et al. (2020) (H:H2CO = 20:1). A higher H flux could
potentially favor the atom-addition route and thus lead to a lower product
ratio. To verify whether the contributions $C4$ and $C5$ depend on the
hydrogen-to-molecule ratio, a set of experiments using H/H2CO(D2CO)$=$30 was
performed at 10 K (see Table 1). The resulting CHD2OH/CH3OH ratios derived
from the RAIRS and TPD-QMS data are 0.18$\pm$0.03 for $C4$ and 0.82$\pm$0.03
for $C5$. These values are negligibly different considering the detection
error. Thus, a higher hydrogen flux affects only slightly the contributions
derived above, and reaction (5) still governs the formation of CH3OH. This
implies that both flux ratios employed here represent an overabundance of H
atoms, and therefore the product yield is limited by either H2CO or D2CO.

In our experiments, the dominance of the radical-molecule route is likely due
to the higher availability of H2CO in the environment, as opposed to H atoms,
to react with CH3O. Both CH3O and H are very reactive species; when a CH3O
radical is formed in the vicinity of a H2CO molecule, it readily reacts with
H2CO to form CH3OH + HCO. On the other hand, the accreted H atoms on the
surfaces are required to diffuse in order to react with the CH3O radicals.
Therefore, they will mostly react with the broadly available H2CO, and less
likely with CH3O, which is a minor ice component. Moreover, the diffusion of
hydrogen atoms also competes with its reaction with CH3O, whereas both CH3O
and H2CO do not diffuse and therefore have more time available to react. As a
result, the higher likelihood of the two reactants in reaction (5) to meet
overpowers reaction (4), despite the latter being barrierless.

Although the ice composition explored here is not fully representative of
interstellar ices, the foregoing reaction routes are nonetheless relevant to
their chemistry. In more realistic ices, there is a lower availability of
formaldehyde, and the formed CH3O radicals can also react with other species.
Nevertheless, the aforementioned reasoning still applies, since the H atoms
will mostly react with CO and HCO, additionally to H2CO. Indeed, route (5) was
observed to dominate the formation of methanol even in simulations starting
from the hydrogenation of CO ice (Simons et al., 2020). Moreover, the radical-
molecule route forms HCO as a byproduct, which will then be hydrogenated again
to H2CO, replenishing and thus favoring the chemical network. Additional
formaldehyde formed in the gas phase in the ISM can also subsequently accrete
onto ice grains and take part in the radical-molecule route.

It is noteworthy that the hydrogenation of pure D2CO ice is not perfectly
equivalent to that of pure H2CO ice. The rate constant of reaction (3’) is in
fact slightly higher than that of reaction (3) (Hidaka et al., 2009; Goumans,
2011). Moreover, reaction (5’) is inhibited by the isotope effect. Thus, there
should be an increase in the availability of CHD2O in the ice during the
deuterated experiments, which in turn favors the formation of CHD2OH through
route (4’). The surplus of CHD2O is either consumed through H-induced
abstraction reactions or preserved in the ice. Furthermore, the isotope effect
also hampers the D abstraction reaction $\ce{D2CO}+\ce{H}\to\ce{DCO}+\ce{HD}$,
which is why the formation of CO is much less efficient in these scenarios
(see Figure 2). This results in a higher availability of H atoms in the
surface of the ice in the deuterated experiments, compared to the non-
deuterated counterparts where the abstraction reaction
$\ce{H2CO}+\ce{H}\to\ce{HCO}+\ce{H2}$ is faster. As a consequence, the
formation of CHD2OH is also favored by the surplus of hydrogen atoms.
Therefore, the contribution from reaction (4’) to forming methanol under the
present experimental conditions should be regarded as an upper limit to that
of reaction (4), which puts a lower limit to reaction (5) as being at least
roughly four times more efficient than reaction (4).

Aiming to directly confirm the occurrence of the radical-molecule route
forming methanol, an experiment of $\ce{H2CO}+\ce{D}$ and a control experiment
of D-atom beam exposure to the bare substrate are performed at 10 K for six
hours. The codeposition experiment is followed by a TPD with a ramping rate of
5 K/min. These experiments are designed to probe the following competing
reactions:

$\ce{H2CO}\xrightarrow{+\ce{D}}\ce{CH2DO}\xrightarrow{+\ce{D}}\ce{CH2DOD},$
(9)
$\ce{H2CO}\xrightarrow{+\ce{D}}\ce{CH2DO}\xrightarrow{+\ce{H2CO}}\ce{CH2DOH}+\ce{HCO}.$
(10)

Figure 5: (a) Difference spectrum of the final spectra acquired after 360
minutes of H2CO + D codeposition minus that of the analogous D-atom beam
deposition for the same time, both performed at 10 K. The dashed line
indicates the absence of the CH2DOD feature at $\nu\sim 2430$ cm-1. (b)
Infrared spectra obtained during the ramp-up after the H2CO + D 360-minutes
codeposition. The gray, red and blue spectra were measured at 10 K, $80-120$
K, and $120-160$ K, respectively.

The difference of the spectra acquired after the H2CO + D experiment and the D
deposition is shown in panel (a) of Figure 5. In agreement with Hidaka et al.
(2009), we observe spectral features of HDCO and D2CO as a result of
D-substitution (abstraction + addition) reactions on H2CO. Moreover, a signal
at $\nu$ = 1034 cm-1 is in line with the CO stretching band of CH2DOH, which
would indicate that the radical-molecule route (i.e., reaction (10)) proceeds.
However, this feature is (at least partially) blended with the CH2 rocking
mode of HDCO ($\nu$ = 1029 cm-1), to an extent where deconvolution is not
possible. Nonetheless, given the lower desorption temperature of HDCO
($\sim$100 K) with respect to that of CH2DOH ($\sim$140 K), it is possible to
isolate the contribution from CH2DOH by analyzing the infrared spectra
obtained during the TPD experiment. The spectra acquired at 10 K and in the
temperature intervals of $80-120$ K and $120-160$ K are shown in panel (b) of
Figure 5.

In the IR spectrum acquired in the temperature range of $80-120$ K, the peak
at $\nu\sim$1029 cm-1 contains the contributions from both HDCO and CH2DOH.
However, at the interval of $120-160$ K, the peak slightly shifts to
$\nu\sim$1034 cm-1, and only accounts for the CH2DOH left on the substrate.
Based on the obtained area of this peak and its reported band strength
(Nagaoka et al., 2007), we derive a CH2DOH column density of $\sim(3.0\pm
0.3)\times 10^{14}$ molecules cm-2. This yield is $\sim 4$ times smaller than
that of CH3OH from the H2CO + H experiment at 10 K (H/H2CO = 10), which is
consistent with a lower rate for the D-addition reaction to H2CO over the
H-addition counterpart.

During TPD, the mass signal m/z = 33 is detected at $\sim$150 K. However,
whereas the spectroscopic signals are quite unambiguous, this does not apply
to the mass spectrometric data. Since the mass signals of CH2DOH (m/z = 33,
32, 31…) overlap with those of CH2DOD (m/z = 34, 33, 32, …), and since their
fragmentation patterns are unknown, the interpretation of the mass
spectrometry data is impeded.

The product of the D-addition route, CH2DOD (reaction (9)), was not observed
in its OD stretching mode ($\nu\sim 2430$ cm-1) within the detection limit of
the instrument, as shown by the dashed line in panel (a) of Figure 5. Given
that the band strength of this feature is expected to be around 1.5 times that
of the CO stretching mode of CH2DOH (Nagaoka et al., 2007), the abundance of
CH2DOD must be appreciably smaller than that of CH2DOH. This result provides
additional compelling evidence that the radical-molecule route dominates the
formation of methanol under the present conditions.

##  4 Conclusions

In the present work, we experimentally examine the contribution from the
conventional route to the formation of methanol (reaction (4)) with reaction
(5), and provide the respective impact of the two proposed last steps of
relevance to the chemistry of molecular clouds. This was done by focusing on
the specific reaction step in question, as more realistic experiments with CO
ices would be much more difficult to interpret. Our main conclusions are the
following:

  * •

We derive a contribution of $\gtrsim$83% for the radical-molecule route (5),
which is independent of the H/H2CO ratios (i.e., 10 and 30) and the
temperatures (i.e., $10-16$ K) experimentally investigated in this work. This
temperature range is typical of dense clouds, where H atoms are expected to be
overabundant in comparison with H2CO. Our experiments confirm the results in
Simons et al. (2020), who performed simulations for conditions as used in the
laboratory work presented here. Their model, in turn, extends to the
interstellar medium, and concludes that this route dominates the formation of
interstellar methanol. Our results corroborate with this finding.
Comparatively, reaction (4) should account for $\lesssim$17% of the methanol
formation under the present conditions.

  * •

Additional codeposition experiments of H2CO and D yield measurable amounts of
CH2DOH, whereas CH2DOD is not detectable within the IR experimental limits.
This further confirms the dominance of the radical-molecule route to form
methanol under our experimental conditions.

  * •

The conclusion drawn here does not affect the consensus that methanol is
mainly formed through the hydrogenation of CO in the solid phase, albeit by
means of a different mechanism than originally thought. Nonetheless, this
finding represents an important change in the perspective of methanol
formation in space, and could affect astrochemical models that involve this
key species.

  * •

The expected deuterium fractionation of methanol will likely be affected, with
a possibly smaller abundance of deuterated species due to the kinetic isotope
effect.

  * •

Furthermore, the HCO radical is an important astrochemical intermediate since
it can react to further form H2CO or recombine with other radicals (e.g., CH3O
and CH2OH) to produce complex organic molecules (COMs). The alternative route
probed here forms HCO radicals as a byproduct, which enriches an icy surface
with this species and therefore affects the COM distribution in interstellar
ices, likely enhancing the predicted formation rate of, e.g., glycolaldehyde
and ethylene glycol in previous astrochemical models that did not consider
reaction (5) as a dominant channel. Models of the radical-radical reactions
driven by the warming of the ice at later stages of star-formation will also
be affected by the additional HCO that eventually remains preserved in the ice
and that becomes mobile and reacts.

  * •

The inclusion of deuterium atoms to microscopic kinetic Monte Carlo
simulations would be a logical next step, as it could provide additional
insight on the chemical network involving methanol. This, however, is beyond
the scope of the present work.

## Acknowledgments

The authors would like to thank Herma Cuppen for many guiding scientific
discussions on the mechanism in question. This work has been supported by the
Danish National Research Foundation through the Center of Excellence
“InterCat” (Grant agreement no.: DNRF150). It has also been funded by the
Dutch Astrochemistry Network II (DANII) and NOVA (the Netherlands Research
School for Astronomy). T.L. is grateful for support from NWO via a VENI
fellowship (722.017.008). G.F. acknowledges financial support from the Russian
Ministry of Science and Higher Education via the State Assignment Contract
FEUZ-2020-0038. S.I. acknowledges the Royal Society for financial support.

##  Appendix A D2CO band strength estimation

The column densities ($N_{X}$) of the species in the ice are calculated by
means of the modified Beer-Lambert equation:

$N_{X}=\ln{10}\frac{\int Abs(\nu)d\nu}{A_{X}},$ (A1)

in which $\int Abs(\nu)d\nu$ is the integrated absorbance of a given infrared
band and $A_{X}$ is its corresponding band strength. To estimate the band
strength of D2CO, the same deposition conditions (i.e., exposure time and
dose) are applied to both pure H2CO and D2CO, ensuring identical abundances of
both types of ices. This is achieved by using the same leak valve while
keeping the same flow-rate setting. Previously, Nagaoka et al. (2007) used
this method to determine the relative integrated band strengths of CH3OH and
isotopes. We plot the integrated areas of the D2CO $\nu_{2}=1679$ cm-1 band
versus those of H2CO $\nu_{2}=1727$ cm-1 for different accumulated abundances
performed at three different flux configurations, as shown in Figure 6.

Figure 6: Plot of the integrated area of the D2CO ($\nu_{2}=1679$ cm-1)
infrared absorption band versus that of H2CO ($\nu_{2}=1727$ cm-1) for three
different molecule fluxes, f: $8\times 10^{11}$, $1.6\times 10^{12}$, and
$3.2\times 10^{12}$ cm-2 s-1.

After performing a linear fit to the points in the plot, we derive an average
slope of $a=0.78\pm 0.03$. This slope corresponds to the conversion factor
between $A$(H2CO) and $A$(D2CO):

$A(\ce{D2CO})_{\nu 2}=a\times A(\ce{H2CO})_{\nu 2}$ (A2)

Given that $A$(H2CO) for the $\nu=1727$ cm-1 peak has been previously measured
to be $\sim 3.6\times 10^{-17}$ cm molecule-1 (Chuang et al., 2018), we
estimate the D2CO $\nu=1679$ cm-1 band strength to be $A(\ce{D2CO})\sim
2.8\times 10^{-17}$ cm molecule-1.

##  Appendix B Experiment list

Table 1: Overview of the experiments performed in this work, as well as the resulting CHD2OH/CH3OH abundance ratios between each set of experiments derived from the RAIRS and TPD-QMS analyses. The former was calculated from the CO stretching bands ($\nu$=8) of CHD2OH and CH3OH, and the latter from the m/z = 32 and m/z = 34 mass signals. Experiment | T${}_{\text{sample}}$ | Molecule Flux | H (D) flux | Time |  CHD2OH/CH3OH |  CHD2OH/CH3OH
---|---|---|---|---|---|---
| (K) | (cm-2 s-1) | (cm-2 s-1) | (min) | (RAIRS, $\nu=8/\nu=8)$ | (TPD)
$\ce{D2CO}+\ce{H}$ | 10 | $6\times 10^{11}$ | $6\times 10^{12}$ | 360 | $0.15\pm 0.04$ | $0.16\pm 0.01$
$\ce{H2CO}+\ce{H}$ | 10 | $6\times 10^{11}$ | $6\times 10^{12}$ | 360
$\ce{D2CO}+\ce{H}$ | 12 | $6\times 10^{11}$ | $6\times 10^{12}$ | 360 | $0.16\pm 0.04$ | $0.16\pm 0.01$
$\ce{H2CO}+\ce{H}$ | 12 | $6\times 10^{11}$ | $6\times 10^{12}$ | 360
$\ce{D2CO}+\ce{H}$ | 14 | $6\times 10^{11}$ | $6\times 10^{12}$ | 360 | $0.16\pm 0.04$ | $0.18\pm 0.01$
$\ce{H2CO}+\ce{H}$ | 14 | $6\times 10^{11}$ | $6\times 10^{12}$ | 360
$\ce{D2CO}+\ce{H}$ | 16 | $6\times 10^{11}$ | $6\times 10^{12}$ | 360 | $0.15\pm 0.04$ | $0.16\pm 0.01$
$\ce{H2CO}+\ce{H}$ | 16 | $6\times 10^{11}$ | $6\times 10^{12}$ | 360
$\ce{D2CO}+\ce{H}$ | 10 | $6\times 10^{11}$ | $1.8\times 10^{13}$ | 360 | $0.18\pm 0.03$ | $0.18\pm 0.01$
$\ce{H2CO}+\ce{H}$ | 10 | $6\times 10^{11}$ | $1.8\times 10^{13}$ | 360
$\ce{H2CO}+\ce{D}$ | 10 | $4\times 10^{11}$ | $6\times 10^{12}$ | 360 |  |

## References

  * Álvarez-Barcia et al. (2018) Álvarez-Barcia, S., Russ, P., Kästner, J., & Lamberts, T. 2018, MNRAS, 479, 2007, doi: 10.1093/mnras/sty1478
  * Anton et al. (2000) Anton, R., Wiegner, T., Naumann, W., et al. 2000, Rev. Sci. Instrum., 71, 1177, doi: 10.1063/1.1150420
  * Bergner et al. (2017) Bergner, J. B., Öberg, K. I., & Rajappan, M. 2017, ApJ, 845, 29, doi: 10.3847/1538-4357/aa7d09
  * Bottinelli et al. (2010) Bottinelli, S., Boogert, A. C. A., Bouwman, J., et al. 2010, ApJ, 718, 1100, doi: 10.1088/0004-637X/718/2/1100
  * Charnley et al. (1992) Charnley, S. B., Tielens, A. G. G. M., & Millar, T. J. 1992, ApJ, 399, L71, doi: 10.1086/186609
  * Chuang et al. (2016) Chuang, K. J., Fedoseev, G., Ioppolo, S., van Dishoeck, E. F., & Linnartz, H. 2016, MNRAS, 455, 1702, doi: 10.1093/mnras/stv2288
  * Chuang et al. (2018) Chuang, K. J., Fedoseev, G., Qasim, D., et al. 2018, ApJ, 853, 102, doi: 10.3847/1538-4357/aaa24e
  * Cuppen et al. (2011) Cuppen, H. M., Penteado, E. M., Isokoski, K., van der Marel, N., & Linnartz, H. 2011, MNRAS, 417, 2809, doi: 10.1111/j.1365-2966.2011.19443.x
  * Fedoseev et al. (2015) Fedoseev, G., Cuppen, H. M., Ioppolo, S., Lamberts, T., & Linnartz, H. 2015, MNRAS, 448, 1288, doi: 10.1093/mnras/stu2603
  * Fuchs et al. (2009) Fuchs, G. W., Cuppen, H. M., Ioppolo, S., et al. 2009, A&A, 505, 629, doi: 10.1051/0004-6361/200810784
  * Garrod et al. (2006) Garrod, R., Park, I. H., Caselli, P., & Herbst, E. 2006, Faraday Discuss., 133, 51, doi: 10.1039/B516202E
  * Geppert et al. (2005) Geppert, W. D., Hellberg, F., Österdahl, F., et al. 2005, Proc. Int. Astron. Union, 1, 117–124, doi: 10.1017/S1743921306007101
  * Goumans (2011) Goumans, T. P. M. 2011, MNRAS, 413, 2615, doi: 10.1111/j.1365-2966.2011.18329.x
  * Haupa et al. (2017) Haupa, K. A., Johnson, B. A., Sibert, E. L., & Lee, Y.-P. 2017, J. Chem. Phys., 147, 154305, doi: 10.1063/1.4996951
  * He et al. (in press) He, J., Simons, M., Fedoseev, G., et al. in press, A&A
  * Hidaka et al. (2007) Hidaka, H., Kouchi, A., & Watanabe, N. 2007, J. Chem. Phys., 126, 204707, doi: 10.1063/1.2735573
  * Hidaka et al. (2009) Hidaka, H., Watanabe, M., Kouchi, A., & Watanabe, N. 2009, ApJ, 702, 291, doi: 10.1088/0004-637X/702/1/291
  * Hiraoka et al. (1994) Hiraoka, K., Ohashi, N., Kihara, Y., et al. 1994, Chem. Phys. Lett., 229, 408, doi: 10.1016/0009-2614(94)01066-8
  * Ioppolo et al. (2013) Ioppolo, S., Fedoseev, G., Lamberts, T., Romanzin, C., & Linnartz, H. 2013, Rev. Sci. Instrum., 84, 073112, doi: 10.1063/1.4816135
  * Molpeceres et al. (2021) Molpeceres, G., Kästner, J., Fedoseev, G., et al. 2021, The Journal of Physical Chemistry Letters, 12, 10854, doi: 10.1021/acs.jpclett.1c02760
  * Nagaoka et al. (2005) Nagaoka, A., Watanabe, N., & Kouchi, A. 2005, ApJ, 624, L29, doi: 10.1086/430304
  * Nagaoka et al. (2007) —. 2007, J. Phys. Chem. A, 111, 3016, doi: 10.1021/jp068978r
  * Penteado et al. (2015) Penteado, E. M., Boogert, A. C. A., Pontoppidan, K. M., et al. 2015, MNRAS, 454, 531, doi: 10.1093/mnras/stv1987
  * Potapov et al. (2021) Potapov, A., Krasnokutski, S. A., Jäger, C., & Henning, T. 2021, ApJ, 920, 111, doi: 10.3847/1538-4357/ac1a70
  * Qasim et al. (2018) Qasim, D., Chuang, K. J., Fedoseev, G., et al. 2018, A&A, 612, A83, doi: 10.1051/0004-6361/201732355
  * Qasim et al. (2020) Qasim, D., Witlox, M. J. A., Fedoseev, G., et al. 2020, Rev. Sci. Instrum., 91, 054501, doi: 10.1063/5.0003692
  * Shimanouchi (1972) Shimanouchi, T. 1972, Tables of Molecular Vibrational Frequencies Consolidated Volume I (National Bureau of Standards), 1–160
  * Simons et al. (2020) Simons, M. A. J., Lamberts, T., & Cuppen, H. M. 2020, A&A, 634, A52, doi: 10.1051/0004-6361/201936522
  * Song & Kästner (2017) Song, L., & Kästner, J. 2017, ApJ, 850, 118, doi: 10.3847/1538-4357/aa943e
  * Tielens & Hagen (1982) Tielens, A. G. G. M., & Hagen, W. 1982, A&A, 114, 245
  * Tschersich (2000) Tschersich, K. G. 2000, J. Appl. Phys., 87, 2565, doi: 10.1063/1.372220
  * Watanabe & Kouchi (2002) Watanabe, N., & Kouchi, A. 2002, ApJ, 571, L173, doi: 10.1086/341412
  * Watanabe et al. (2007) Watanabe, N., Mouri, O., Nagaoka, A., et al. 2007, ApJ, 668, 1001, doi: 10.1086/521421