# Evaluating Large Language Models on the GMAT: Implications for the Future of
Business Education

Vahid Ashrafimoghari , Necdet Gürkan , Jordan W. Suchow
Stevens Institute of Technology
<EMAIL_ADDRESS>

###### Abstract

The rapid evolution of artificial intelligence (AI), especially in the domain
of Large Language Models (LLMs) and generative AI, has opened new avenues for
application across various fields, yet its role in business education remains
underexplored. This study introduces the first benchmark to assess the
performance of seven major LLMs—OpenAI’s models (GPT-3.5 Turbo, GPT-4, and
GPT-4 Turbo), Google’s models (PaLM 2, Gemini 1.0 Pro), and Anthropic’s models
(Claude 2 and Claude 2.1)— on the GMAT, which is a key exam in the admission
process for graduate business programs. Our analysis shows that most LLMs
outperform human candidates, with GPT-4 Turbo not only outperforming the other
models but also surpassing the average scores of graduate students at top
business schools. Through a case study, this research examines GPT-4 Turbo’s
ability to explain answers, evaluate responses, identify errors, tailor
instructions, and generate alternative scenarios. The latest LLM versions—
GPT-4 Turbo, Claude 2.1, and Gemini 1.0 Pro— show marked improvements in
reasoning tasks compared to their predecessors, underscoring their potential
for complex problem-solving. While AI’s promise in education, assessment, and
tutoring is clear, challenges remain. Our study not only sheds light on LLMs’
academic potential but also emphasizes the need for careful development and
application of AI in education. As AI technology advances, it is imperative to
establish frameworks and protocols for AI interaction, verify the accuracy of
AI-generated content, ensure worldwide access for diverse learners, and create
an educational environment where AI supports human expertise. This research
sets the stage for further exploration into the responsible use of AI to
enrich educational experiences and improve exam preparation and assessment
methods.

##  1 Introduction

Artificial intelligence (AI) has witnessed substantial advancements over the
past few years, which has facilitated its application across a spectrum of
fields. These applications range from augmenting personal assistant
technologies [1], to innovating healthcare practices [2], and to enriching
educational methodologies [3]. In healthcare, for instance, AI assists
professionals by organizing patient records, analyzing diagnostic images, and
identifying health issues [4]. AI applications have also been utilized in
education to enhance administrative services and academic support [3]. These
systems have been developed to simulate one-to-one personal tutoring, helping
educators in creating more effective educational settings. However, developing
these systems is challenging; it involves not only content creation and design
but also the refinement of feedback phrasing and dialogue strategies [5].

The emergence of Large Language Models (LLMs), also known as Large Generative
AI Models (LGAIMs), has been transformative for Natural Language Processing
(NLP) tasks, demonstrating considerable promise in the spheres of education
and assessment. As integral elements of AI, LLMs are adept at comprehending,
producing, and interpreting human language. With the ongoing advancement of
AI, it becomes imperative to scrutinize the proficiency and constraints of
these models within educational frameworks. Our study investigates the
efficacy of LLMs on the Graduate Management Admission Test (GMAT), an
established benchmark for entry into graduate management programs. The
objective of this evaluation is to elucidate the capabilities and limitations
of LLMs in educational environments.

The GMAT plays a crucial role in the application process for business schools
worldwide. It is designed to assess candidates’ abilities in verbal and
quantitative reasoning, analytical writing, and integrated reasoning, offering
a thorough evaluation of their preparedness for the challenging academic
environment of business school. Traditionally, preparation for the GMAT exam
has involved human tutors, who deliver their services either in classroom
settings or via online tutoring platforms. These tutors provide tailored
guidance, practice tests, and review sessions, all aimed at helping candidates
excel in the various segments of the exam.In today’s educational environment,
a wide array of companies and websites, such as Kaplan, Manhattan GMAT,
Economist, and Magoosh, offer extensive GMAT exam preparation services. These
services encompass a range of options, including self-paced online courses,
live online classes, private tutoring, and comprehensive study materials.
Their goal is to accommodate various learning styles and schedules, thereby
making GMAT preparation more accessible and adaptable for prospective business
school students.

However, the recent advancements in LLMs, such as GPT-4 Turbo, present a
unique opportunity to revolutionize the GMAT preparation process. These
sophisticated models have the potential to automate certain aspects of GMAT
preparation, offering personalized, adaptive learning experiences that can
match or even surpass traditional methods. For instance, they could provide
instant feedback on practice questions, adapt the difficulty level based on
the learner’s progress, or offer targeted practice on weak areas. Moreover,
they could be available 24/7, offering flexibility that traditional tutoring
cannot match. The exploration of this potential marks an exciting frontier in
the intersection of artificial intelligence and education, promising to make
GMAT preparation more efficient, effective, and tailored to individual needs.

Building on the possibilities introduced by LLMs, this study seeks to answer
the following research questions, each aimed at exploring the capabilities and
potential applications of these models in the context of GMAT preparation:

RQ1: How do LLMs compare to human candidates in terms of performance when
responding to the verbal and quantitative reasoning sections of the GMAT?

RQ2: What potential benefits and drawbacks are associated with the use of LLMs
in learning and education, especially for tutoring, exam preparation, and
assessment?

To address the research questions, we adopted a comprehensive approach,
initiating with an analysis of model performance on GMAT exam questions. Our
evaluation encompassed seven state-of-the-art general-purpose LLMs: GPT-3.5
Turbo, GPT-4, GPT-4 Turbo, Claude 2, Claude 2.1, PaLM 2, and Gemini 1.0 Pro,
focusing on their abilities in the quantitative and verbal reasoning sections
of the GMAT. Utilizing both the free and premium practice exams offered by the
Graduate Management Admission Council (GMAC), we aimed to discern any effects
of memorization or data leakage. The results revealed a negligible variance in
performance between the free and premium exams.Notably, GPT-4 Turbo, employing
zero-shot standard prompting, significantly outperformed the others, achieving
an average accuracy of 85.07% across three sets of official GMAT practice
exams, compared to 74.13% for GPT-4, 56.72% for GPT-3.5 Turbo, 72.14% for
Claude 2.1, 60.2% for Claude 2, 70.65% for Gemini 1.0 Pro and 50.75% for PaLM
2.

Our research goes beyond basic performance metrics to explore AI behavior in
educational settings. We provide a comparative analysis of human and AI
performance on the GMAT, discussing AI errors for targeted improvements. A
case study highlights the qualitative behavior of GPT-4 Turbo, demonstrating
its ability to articulate critical reasoning and interactively assist students
by evaluating their responses and errors, and creating counterfactual
scenarios. We conclude by reflecting on the potential impact of our findings
on business education and professional development, addressing accuracy,
fairness, and the broader implications for management practices. While
recognizing the limitations of our evaluation methods, we discuss the
necessary precautions and advancements for the real-world application of LLMs
like GPT-4 Turbo. Despite the need for careful and comprehensive evaluation,
we anticipate that AI models such as GPT-4 Turbo will have a significant and
positive impact on the business sector, becoming essential tools in
educational and professional settings.

##  2 Related Work

In recent years, several large pre-trained models have been developed,
including GPT [6], Bard [7], Claude 2 [8], BERT [9], RoBERTa [10], and the
widely used GPT-3 [11], GPT-3.5 [12], and GPT-4 [13]. These models are based
on transformer architecture [14] and have been pre-trained on massive datasets
of text to generate human-like text, answer questions, assist in translation
and summarization, and perform many NLP tasks with a single pre-training and
fine-tuning pipeline. These developments mark significant milestones in the
field of NLP and offer enormous opportunities for applications in research and
industrial contexts. We anticipate that future advancements in AI will offer
significant benefits, thus highlighting the need to explore their potential
applications in education.

LLMs have a range of applications within educational settings, enhancing
student learning. Researchers have utilized these models to develop
interactive learning tools, such as quizzes and flashcards, which in turn
improve student engagement and facilitate knowledge acquisition [15, 16]. One
of the advantages of using LLMs in education is their ability to help students
complete their practice questions more efficiently [17]. Specifically, GPT-3
has been employed to generate multiple-choice questions and answers, enhancing
reading comprehension [15]. The study indicates that automated quiz generation
not only reduces the workload for educators in creating quizzes manually but
also acts as an effective tool for students to evaluate and deepen their
knowledge throughout their study period and exam preparation. The researchers
developed a method for automatically creating prompts that encourage learners
to ask more substantive questions [18]. The findings suggest that LLMs can
significantly facilitate the promotion of curiosity-driven learning
experiences and serve as a powerful mechanism to enhance the expression of
curiosity among learners [19].

Personalized learning is a promising application area for LLMs, considering
the individual differences in learning styles and vast amount of educational
data available [20]. Researchers demonstrated the application of an advanced
GPT-3 model, specifically calibrated for chemistry, to assess answers provided
by students [21]. This technology could significantly aid educators in
appraising the quality and instructional value of student responses.
Additionally, [22] proposed a system that delivers AI-driven, personalized
feedback within a high school science task. They found that this system
effectively supported students in refining their scientific reasoning. [23]
observed that such feedback systems could enhance the ability of teacher
trainees to substantiate educational assessments. Furthermore, [24] noted that
in large-scale courses, these systems can provide feedback on written work,
potentially reducing the grading workload by 85% while maintaining high
accuracy. This enhancement also improves the perceived quality of feedback
among students. Recognizing the benefits of LLMs, Khan Academy has developed
an AI chatbot named Khanmigo, which functions as a virtual tutor and classroom
assistant. This initiative represents a step forward in incorporating LLMs
into educational platforms, aiming to enhance tutoring and coaching
experiences. The goal is to provide personalized one-on-one interactions with
students, demonstrating a practical application of LLMs in education [25].
Another notable example is the potential use of LLMs in foreign language
learning. Their features, such as an effortless, seamless, and friendly
interface, contribute to an excellent user experience. A study demonstrated
that language learners appreciated the ChatGPT prompts while performing
language tasks [26]. In another study, researchers proposed a system that
leverages LLMs to identify content that aligns with the user’s interests and
closely matches their proficiency level in a foreign language [27].

In the domain of computer science education, GPT-3 has been utilized as an
educational aid to clarify various aspects of coding, although numerous
questions related to research and instructions remain open for further
investigation [28]. In a separate endeavor, [29] have devised a method for
generating evaluative questions tailored to a data science curriculum by
refining a GPT-3 model with text-based educational material. These questions
were evaluated for their instructions merit using both an automated
categorization system employing a specifically trained GPT-3 model and expert
reviews from domain specialists. The outcomes indicated that the
GPT-3-generated questions were positively appraised by these experts, thus
supporting the adoption of large language models in the field of data science
education. In a recent study, researchers combined the capabilities of GPT-4
and GPT-3.5 to provide automated, tutor-like, high-quality programming
suggestions [30].

In digital ecosystems for education, particularly in Augmented Reality (AR)
and Virtual Reality (VR) environments [31, 32], LLMs can play a transformative
role. They can amplify key factors crucial for immersive user interaction with
digital content. For instance, LLMs can greatly enhance the natural language
processing and understanding capabilities of AR/VR systems. This enhancement
enables effective and natural communication between users and the system, such
as with a virtual teacher or virtual peers. The importance of this capability
for immersive educational technologies was identified early as a critical
aspect of usability [33] and is widely recognized as a key factor in improving
human-AI interactions [34].

In the realm of standardized testing, [35] examined ChatGPT’s performance on
the United States Medical Licensing Examination, revealing that ChatGPT’s
scores approached the passing threshold without the benefit of domain-specific
training. These findings prompted the researchers to propose that large
language models have the potential to substantially aid medical education and
even contribute to clinical decision-making processes [35]. Additionally, an
investigation into ChatGPT’s performance on four actual law school exams at
the University of Minnesota found that it averaged a C+ grade, managing to
pass in all four subjects [36]. These outcomes suggest that LLMs hold
promising implications for both medical and legal educational fields [37, 38,
39].

##  3 Methodology

In this study, we evaluated seven general-purpose LLMs on three GMAT exams
using the zero-shot standard prompting method, as described in the following.

###  3.1 Models

#### OpenAI’s GPT Family

We tested three LLMs form OpenAI’s GPT family: GPT-3.5 Turbo, GPT-4, and GPT-4
Turbo. GPT-3.5-turbo, released on March 1st, 2023, is an improved version of
GPT-3.5 and GPT-3 Davinci. GPT-3.5-turbo utilizes a transformer architecture
with a large number of parameters. It is trained on a diverse corpus of text
data, which enhances its ability to understand and generate human-like text.
This increase in scale empowers the model to tackle more complex tasks,
comprehend nuanced contexts, and ultimately achieve enhanced performance [40].
Training data sourced from diverse materials including books, articles, web
pages, and more provides GPT-3.5-turbo broad exposure to language and contexts
available across the internet. This extensive and varied training corpus
allows the model to develop a multifaceted understanding of language,
enhancing its capacity for remarkably human-like text generation. The last
update to GPT-3.5-turbo was in September 2021, and it currently powers the
freely available version of ChatGPT.

Initially launched on March 14, 2023, GPT-4 has been made accessible to the
public through the paid chatbot service, ChatGPT Plus, and also via OpenAI’s
API. As a transformer-based model, GPT-4 employs a two-step training process.
The initial pre-training phase utilizes both public data and data sourced from
third-party providers to predict the subsequent token. Following this, the
model undergoes a fine-tuning process, leveraging reinforcement learning
feedback from both humans and AI to ensure alignment with human values and
policy compliance [41].

GPT-4 Turbo is the latest version of GPT and appears to be the most advanced
AI model currently available in the market. According to the OpenAI’s website,
this enhanced version boasts increased capabilities, possesses knowledge up to
April 2023, and features an expanded context window of 128k, allowing it to
process the equivalent of 300 pages of text in one prompt. Furthermore, it
offers a cost reduction, with input tokens being 3 times more affordable and
output tokens 2 times more affordable than those of the original GPT-4 model.
Additionally, this model can generate up to 4096 output tokens [42].

While both GPT-4 and GPT-4 Turbo possess multi-modal functionalities [13, 43],
our research focuses exclusively on the text-based versions of these models.
Henceforth, any reference to GPT-4 or GPT-4 Turbo within this document will
specifically denote the text-only versions without visual processing features.

#### Anthropic’s Claude Family

Anthropic, the firm responsible for Claude, was established in 2021 by a team
of former OpenAI employees who contributed to the development of OpenAI’s
GPT-2 and GPT-3 models. In early 2023, after conducting a closed alpha with a
select group of commercial partners, Claude’s model was incorporated into
products such as Notion AI, Quora’s Poe, and DuckDuckGo’s DuckAssist. In March
2023, Claude expanded its API access to a broader range of businesses.
Subsequently, in July 2023, it launched its chatbot to the public, coinciding
with the release of the Claude 2 model. While Claude 2 may not yet match the
capabilities of GPT-4, it is rapidly advancing and consistently outperforms
most other AI models in standardized tests. Claude 2 distinguishes itself with
its capacity to manage up to 100K tokens per prompt, equivalent to
approximately 75,000 words. This is a twelve-fold increase compared to the
standard amount offered by GPT-4. Also, while GPT’s knowledge cutoff is
September 2021, Claude 2 benefits from training data extending up to early
2023[8]. Anthropic unveiled Claude 2.1 on November 23, 2023, marking the
newest update to their LLM lineup. As stated on their website, this latest
version brings enhancements crucial for enterprise applications, such as a
top-tier context window capable of handling 200K tokens, marked improvements
in reducing instances of model-generated inaccuracies, and system prompts.
Additionally, Claude 2.1 introduces a beta feature known as ’tool use.’
Alongside these advancements, Anthropic has revised their pricing structure to
increase cost-effectiveness across all their model offerings [44]. Both Clude
2.0, and Claude 2.1 are accessible through the claude.ai, Anthropic’s LLM-
based chatbot.

#### Google’s LLM Families

The Pathways Language Model (PaLM) is a LLM with 540 billion parameters,
constructed using the Transformer architecture and developed by Google AI. The
subsequent iteration, PaLM 2, was released in late 2022 and employs dynamic
pathway selection during inference to determine the most suitable pathway for
a given input [45]. Bard, Google’s conversational AI agent introduced in March
2023, was initially based on the LaMDA family of LLMs and later integrated
PaLM 2[7]. The latest generation of Google’s LLMs is the Gemini Family.
Developed from the Transformer architecture, Gemini models incorporate
architectural improvements and optimization techniques for stable, scalable
training and efficient inference on Google’s Tensor Processing Units. These
models are capable of handling a context length of 32k, employing efficient
attention mechanisms such as multi-query attention([46]). The first release,
Gemini 1.0, is available in three main sizes—Ultra, Pro, and Nano—to
accommodate a variety of applications. Currently, Gemini 1.0 Pro, a cost and
latency-optimized version of Gemini 1.0, has been integrated into the new
Bard. Google has announced plans to make Gemini 1.0 Ultra accessible through
Bard Advanced in early 2024 [47]. For this study, we analyzed data from the
legacy Bard tested between July 16 and October 2, 2023, and results obtained
from Bard after December 6, 2023, for Gemini 1.0 Pro. While Gemini models
offer multi-modal capabilities [46], our research is concentrated on the text-
only variant of Gemini 1.0 as implemented in the new Bard. Moving forward, any
mention of Gemini 1.0 Pro in this paper will refer specifically to the text-
only iteration, excluding visual processing features.

###  3.2 Graduate Management Admission Test (GMAT)

The GMAT exam serves as a differentiating factor for more than 100,000
candidates every year during the business school admissions process, with a
significant 90% of new MBA enrollments relying on a GMAT score [48]. The GMAT
exam is recognized as a reliable indicator of a student’s potential for
success. Each GMAT exam consists of 31 quantitative reasoning, 36 verbal
reasoning, 12 integrated reasoning problems, and an analytical writing essay.
The GMAT Total Score is derived from the Verbal and Quantitative sections of
the exam. This Total Score is based on the test taker’s calculated performance
before the scores for the Quantitative Reasoning and Verbal Reasoning sections
are assigned. This raw calculation is then converted into a number within the
Total Score range of 200 to 800. As only the performance in the quantitative
and verbal reasoning sections contributes to the total score, we focused
solely on the performance in these two sections, disregarding the performance
in Integrated Reasoning and Analytical Writing. The quantitative reasoning
section comprises two tasks: Problem Solving and Data Sufficiency. The verbal
reasoning section is also divided into three tasks: Reading Comprehension,
Critical Reasoning, and Sentence Correction.

In this study, we deployed seven different LLMs—GPT 3.5-turbo, GPT-4, GPT-4
Turbo, Claude2, Claude 2.1, PaLM 2, and Gemini 1.0 Pro—to generate responses
for three official GMAT practice exams. We emphasize that the materials used
in this study were officially acquired and sourced from the Graduate
Management Admission Council (GMAC), the authoritative body that oversees and
administers the GMAT examination process. Given that the GMAT exam employs an
adaptive testing format—wherein a test taker’s performance on prior questions
dictates the difficulty level of subsequent ones—we maintain that
administering the tests through the official GMAT exam website constitutes the
most authentic method for evaluating the performance of LLMs. This approach is
likely to provide the most accurate estimation of the AI’s proficiency and
competency. By leveraging the official testing platform, we can ensure that
the AI is subjected to the same dynamic adjustments in difficulty that human
test-takers face, thereby offering a fair and rigorous assessment of its
capabilities in a standardized context. Our choice of exams was strategic; to
mitigate the potential influence of publicly available free practice exams on
the results, we included premium practice exams among the three exam sets for
each model. These premium exams were not publicly accessible, thereby ensuring
a level of novelty and challenge for the LLMs.

###  3.3 Prompting

To establish a baseline for model performance and to ensure fair comparison
with human participants, we applied a simple zero-shot prompting technique for
the models being evaluated. Additionally, to reduce the potential for in-
context learning during chat-based interactions, we posed each question in a
separate chat window using LLM-based chatbots, rather than within a continuous
conversation thread. This precaution ensured that each model’s response was
independent and not influenced by previous interactions, providing a more
accurate measure of standalone performance. We standardized the prompt
template, as illustrated in Figure 1, customizing it for different problem
types, with a complete example shown in Figure 2. This approach maintained
prompt consistency across similar problems, allowing for automated model
responses with minimal human intervention, limited to entering the problem
content and prompt. In our study, we focused exclusively on the text-based
capabilities of LLMs. Consequently, for questions originally accompanied by
images, we converted them into descriptive math word problems (Figures 3 and
4). This translation process ensured that the essence of the visual
information was accurately captured and conveyed through text, allowing the
LLMs to process and respond appropriately.

Figure 1: The template employed for generating prompts for every multiple-
choice question. Elements shown in double braces are substituted with
question-specific values. Figure 2: An example of implementation of template
shown in from Figure 1. Figure 3: Original problem statement with image Figure
4: The prompt describes the problem shown in Figure 3, presented without an
accompanying image. The correct answer provided by GPT-3.5 Turbo is
highlighted in grey.

##  4 Results and Discussion

###  4.1 Performance of LLMs on the GMAT

Table 1 presents an overview of the average performance of LLMs on the GMAT
exam, both overall and across different tasks. It is worthy to note that the
overall average mentioned in the table is calculated by dividing the number of
correct answers by the total number of questions across all three exam sets
for each model.

Sections & Tasks |  GPT-4 Turbo |  GPT-4 |  GPT-3.5 Turbo |  Claude 2.1 |  Claude 2 |  Gemini Pro (New Bard) |  PaLM 2 (Legacy Bard)
---|---|---|---|---|---|---|---
Quantitative Reasoning |  74.19 |  64.52 |  53.76 |  61.29 |  46.24 |  68.82 |  44.09
Data Sufficiency |  60.98 |  56.10 |  39.02 |  56.10 |  36.59 |  60 |  32.50
Problem Solving |  84.62 |  71.15 |  64.15 |  65.38 |  53.85 |  75.47 |  52.83
Verbal Reasoning |  94.44 |  82.41 |  61.11 |  81.48 |  72.22 |  72.22 |  56.48
Reading Comprehension |  100 |  97.44 |  70.73 |  100 |  97.44 |  87.50 |  79.49
Critical Reasoning |  96.30 |  81.48 |  55.56 |  75 |  64.29 |  66.67 |  50
Sentence Correction |  87.80 |  69.05 |  55.00 |  68.29 |  53.66 |  60.98 |  39.02
Overall Average |  85.07 |  74.13 |  57.71 |  72.14 |  60.20 |  70.65 |  50.75

Table 1: The comparison of model performance across various GMAT sections and
tasks reveals that GPT-4 Turbo outperforms the other models in all sections
and tasks.

Table 1 demonstrates that across the board, all models exhibit their highest
proficiency in the reading comprehension task. On the other hand, they
struggle the most with the data sufficiency task, which emerges as their
Achilles’ heel. For the remaining tasks—problem-solving, critical reasoning,
and sentence correction—the models’ performance falls within the average
range, indicating a balanced level of proficiency in these areas but without
the peaks observed in reading comprehension or the troughs seen in data
sufficiency.

#### Quantitative reasoning

When mathematical problems have unambiguous presentations and sequential
solutions, LLMs can apply their extensive computational power to reach correct
answers quickly and on vast scales unmatched by humans [See, 49]. However,
despite exceeding human performance on well-specified calculations, LLMs still
lack deeper comprehension of the mathematical concepts themselves. Instead,
these models depend fundamentally on pre-defined algorithms. As a result, they
fluctuate when problems require creative approaches, judgments akin to human
intuition, or the ability to integrate multiple abstract concepts. Vague or
complex multi-step problems strain LLMs even further.

#### Reading comprehension

Previous research has shown pre-trained language models are typically excel at
summarizing, classifying, and paraphrasing texts, demonstrating a keen ability
to identify the main ideas within passages [e.g., 50]. Therefore, their
consistently high performance in tackling reading comprehension problems is to
be expected. This proficiency can be attributed to their advanced language
processing capabilities, which allow them to effectively analyze and interpret
textual information. Consequently, they can accurately understand the context,
draw inferences, and provide precise responses to comprehension questions.
However, LLMs struggle with abstract or complex concepts that require a level
of reasoning or inference beyond the literal text [e.g., 51, 52].

#### Critical reasoning

LLMs are adept at pattern recognition [53] and inference-making [54], skills
that serve them well in answering critical reasoning questions focused on
identifying assumptions and assessing arguments. Yet, they can falter with
subtle or complex logical reasoning, especially when questions demand an
understanding of new evidence or hypothetical situations, as LLMs depend on
data-driven patterns rather than true understanding or practical knowledge.

#### Sentence correction

Ultimately, LLMs, pre-trained on an extensive set of grammar rules, are
proficient at identifying a broad spectrum of grammatical errors. However,
they often stall when errors require an understanding of context or language
nuances. Complex sentences, particularly those with multiple clauses or
unconventional structures, can pose challenges [55]. LLMs’ reliance on
programmed rules can lead to oversights in recognizing exceptions or
stylistically acceptable rule deviations. Additionally, LLMs sometimes cannot
grasp deeper meanings or implications of information and can struggle with
context, especially when it involves cultural, historical, or personal
references [56].

###  4.2 Performance of LLMs versus human candidates

Figure 5 illustrates the comparative performance of various AI agents,
representing different LLMs, against human participants in the GMAT exam. On
average, the LLMs attained a total score of 643.5 across all administered
exams, a score that would place them above approximately 63% of human
candidates. Notably, GPT-4 Turbo leads the pack, outshining all other AI
agents and the average human candidate, securing a spot within the top 1% of
test-takers based on the total exam score. Trailing GPT-4 Turbo, the other AI
agents—GPT-4, Claude 2.1, the new Bard (Gemini 1.0 Pro), Claude 2, GPT-3.5
Turbo, and the legacy Bard (PaLM 2)—ranked successively, surpassing 94%, 90%,
83.3%, 53%, 36.7%, and 12% of human test-takers, respectively. Echoing the
pattern observed in human candidates, all AI agents exhibited stronger results
in verbal reasoning over quantitative reasoning. The extent of this
performance gap between the two sections, however, showed variation among the
AI agents. Interestingly, the most recent versions of LLMs, such as the new
Bard (Gemini 1.0 Pro), Claude 2.1, and GPT-4 Turbo, closely emulated human
test-taker performance patterns. The discrepancy between their verbal and
quantitative scores was nearly identical to the average human candidate’s
spread. This alignment suggests that the latest LLMs are approaching human-
like balance in handling the diverse skill sets required by the GMAT. It is
important to note that the data regarding human test-takers is derived from a
sample of 282,098 individuals who sat for the GMAT between January 2020 and
December 2022, as reported in the GMAT score distribution [57].

Figure 5: This figure presents a comparative analysis of the average
performance across seven LLMs and human candidates in quantitative reasoning,
verbal reasoning, and total GMAT scores.

As depicted in Figure 6, AI agents’ performance on the GMAT exam positions
them as formidable contenders for admission into elite business schools.
According to recent admissions data from the top 50 business schools, GPT-4
Turbo stands a high chance of being accepted into all these esteemed
institutions. Remarkably, only seven business schools—Stanford Graduate School
of Business, The Wharton School of the University of Pennsylvania, Harvard
Business School, London Business School, Kellogg School of Management at
Northwestern University, HEC Paris, and Indian School of Business—boast
applicants with GMAT scores that exceed those of GPT-4 Turbo. This highlights
GPT-4 Turbo’s exceptional performance, which not only competes with but also
often exceeds that of the majority of human candidates at these prestigious
institutions. Moreover, GPT-4’s GMAT results align with or surpass the average
applicant scores at the majority of the top 50 business schools, implying a
strong likelihood of GPT-4 gaining admission to nearly all these programs. The
average GMAT scores of MBA classes at only the top 9 business schools exceed
GPT-4’s scores, underscoring its potential for acceptance. Claude 2.1 and the
new Bard, despite having nearly identical GMAT scores, perform above the
average of applicants to almost half of the top 50 business schools. This
indicates that these AI agents are competitive applicants for many top-tier
programs. Conversely, AI agents based on older LLM versions—Claude 2, GPT-3.5
Turbo, and legacy Bard—have a tiny chance of admission to the top 50 business
schools, as their average GMAT scores significantly trail behind the applicant
averages for these institutions.

Figure 6: This graph illustrates a side-by-side comparison of average GMAT
scores from seven AI models, denoted with solid lines for the latest models
and dashed lines for legacy versions, against the average, minimum, and
maximum GMAT scores recorded by students at the top 50 business schools,
according to the Financial Times 2023 rankings[58]. The data originates from
MBA class profiles publicly available on the schools’ official websites.
Inclusion criteria required schools to offer comprehensive information in
their profiles, with the compilation organized by ascending average GMAT
scores

###  4.3 Limitations of LLMs

To gain a more comprehensive understanding of the limitations of LLMs, it’s
crucial to conduct an in-depth analysis of their incorrect responses. Figures
7, 8, 9, and 10 offer a more granular perspective on the areas where LLMs
struggle. These figures specifically highlight their shortcomings in solving
problems in quantitative reasoning, reading comprehension, critical reasoning,
and sentence correction. By examining these areas of weakness, we can better
understand the challenges faced by LLMs and work towards improving their
performance in these specific domains.

A closer examination of quantitative reasoning problems answered incorrectly
by LLMs reveals 16 distinct error categories, each corresponding to specific
mathematical concepts. These categories include geometry, numbers and the
number line, sets, factors, multiples, divisibility and remainders, decimals,
fractions and percents, exponents, statistics and probability, rate, work and
mixture problems, ratio, proportions and estimations, counting methods,
inequalities, factoring and quadratic equations, algebraic expressions and
linear equations, properties of operations, sequences and series, and
functions. This classification provides a granular view of the mathematical
domains that challenge LLMs, offering insights for targeted improvement. For
example, latest versions of language model families have shown a reduction or
elimination of errors in certain categories. GPT-4 Turbo, in comparison to
earlier models, has achieved flawless performance in areas such as Ratio,
Proportions, and Estimation, and Properties of Operations. Similarly, Claude
2.1 flawlessly answers Counting Methods questions, and Gemini 1.0 Pro has a
perfect record in Sequences and Series, a category where other models still
falter. Overall, more than half of the errors made by LLMs are concentrated in
five categories: Geometry, Numbers and Number Line, Sets, Rate, Work, and
Mixture Problems, and Exponents. This pattern suggests that LLMs’ mistakes
likely arise from challenges of processing spatial, numerical, and logical
information through text. Geometry requires visualizing diagrams, which is
challenging for LLMs with text alone. Numerical concepts like absolute value
and ordering can be subtle and easily misinterpreted by LLMs. Set theory’s
logical operations, especially in complex sets, are difficult for LLMs to
process accurately. Rate and mixture problems’ multi-step ratios, as well as
the rules of exponentiation in exponent problems, often lead to errors when
LLMs fail to apply them correctly.

Reviewing the errors made by LLMs in reading comprehension, it is obvious that
they struggle with inference questions, which require the deduction of ideas
that are not explicitly stated in a passage. Additionally, they occasionally
have difficulty comprehending the main idea of passages. Similarly, incorrect
responses to critical reasoning problems, which present a challenge for LLMs,
can be categorized into eight distinct categories. These categories include
’Weaken the Argument,’ which entails identifying information that weakens the
author’s argument; ’Inference,’ which requires identifying something that must
be true based on the given information; ’Find the Assumption,’ which involves
identifying an assumption that is not explicitly stated; ’Explain a
Discrepancy,’ which requires identifying information that resolves an apparent
paradox in the argument; ’Evaluate the Argument,’ which involves identifying
information that would help determine the argument’s soundness; ’Strengthen
the Argument,’ which requires identifying information that supports the
author’s argument; ’Describe the Role,’ which involves identifying the roles
or labels of the underlined portions of the argument; and ’Find the Flaw,’
which requires identifying an illogical element in the argument. Among these
categories, questions falling under ’Weaken the Argument’ are the most
challenging for LLMs to answer, likely because it involves understanding the
author’s perspective and reasoning, as well as critically evaluating the
information presented. LLMs may find it difficult to recognize the weaknesses
in an argument or may struggle to differentiate between information that
strengthens or weakens the argument. Therefore, these types of questions pose
a particular challenge for LLMs in their critical reasoning skills.

Finally, the grammatical errors that LLMs sometimes fail to identify in
sentence correction questions can be grouped into 11 categories. These include
errors related to meaning, modifiers, subject-verb agreement, pronouns,
grammatical constructions, awkwardness or redundancies, verb forms, tenses,
comparison, parallelism, and idioms. Among these, the majority of incorrect
responses are due to the LLMs’ inability to detect errors in meaning. This
particular challenge may stem from the subtleties and variations of language
that require a deep understanding of context and intent. Errors in meaning can
often involve complex logical structures or subtle shifts in tone that LLMs
are not always equipped to handle. Modifiers, also, must be placed correctly
to avoid ambiguity, while subject-verb agreement requires careful attention to
ensure that subjects and verbs match in number. Pronouns must refer to the
appropriate noun, and grammatical constructions must be consistent and
logical. Awkwardness or redundancies in language can make sentences less clear
and concise, which LLMs might overlook. Verb forms and tenses must be used
correctly to convey the proper time frame of actions, while comparisons
require a balanced and accurate assessment of similarities or differences.
Parallelism is essential for maintaining a consistent structure in lists and
comparisons, and idioms must be used appropriately to convey the intended
meaning in a culturally specific context. In sum, the complexity of these
grammatical rules and the intricacies of their application occasionally makes
sentence correction a challenging area for LLMs.

Figure 7: A detailed categorization of LLM errors in Quantitative Reasoning
for targeted improvement Figure 8: A detailed categorization of LLM errors in
Reading Comprehension for targeted improvement Figure 9: A detailed
categorization of LLM errors in Critical Reasoning for targeted improvement
Figure 10: A detailed categorization of LLM errors in Sentence Correction for
targeted improvement

##  5 Study Limitations and Future Directions

Our paper demonstrates the remarkable capability of general purpose LLMs in
responding to quantitative and verbal reasoning questions on the GMAT exam. In
the following, we will discuss the limitations and possible expansions of our
research for the future.

###  5.1 Prompting Method

Our objective in this study is to establish a benchmark for the foundational
performance of LLMs when tasked with answering GMAT multiple-choice questions.
We aim to achieve this using a straightforward methodology, deliberately
avoiding the use of more intricate techniques such as chain-of-thought (CoT)
prompting [59], Retrieval Augmented Generation (RAG) [60] or prompt chaining
strategies [61]. Previous research has demonstrated that these advanced
methods significantly improve the capabilities of LLMs when addressing complex
queries across various fields [62, 63]. Moreover, the potential exists for the
development of new prompting methods specifically tailored to enhance the
performance of advanced language models like GPT-4 Turbo, which have not yet
been fully explored or identified. Given the complexity and adaptability of
such models, it stands to reason that a deliberate and systematic examination
of various prompting techniques, coupled with strategic fine-tuning, has the
capacity to yield significant gains in performance outcomes. This could
involve experimenting with different types of prompts, varying the complexity
and structure of the language used, or even customizing prompts to align more
closely with the model’s training data. While the pursuit of achieving the
highest possible scores on benchmarks is a valid endeavor, it is not the sole
focus of this paper. Our aim extends beyond simply pushing the limits of
benchmark performance to include a broader understanding of model capabilities
and limitations. Therefore, we acknowledge that the in-depth exploration of
these advanced prompting strategies represents a promising avenue for future
studies.

###  5.2 Scope of the study

This paper’s benchmarking parts are focused on assessing multiple-choice
questions from quantitative reasoning and verbal reasoning sections of the
GMAT, which form a significant but not complete portion of this exam. As
explained in the methodology section, the GMAT includes 12 integrated
reasoning (IR) and one essay section as well that are scored independently and
their scores will not affect the exam’s total score, so we did not include
quantitative metrics for these two sections in our benchmarks. Consequently,
the performance metrics we report may not fully reflect the LLMs capabilities
in a real GMAT exam setting. Furthermore, while a study of LLMs on the GMAT
can offer important data on certain cognitive abilities relevant to business
education, it is not sufficient to generalize its findings to the entire field
of business education without additional research that considers the full
spectrum of skills and learning experiences that business programs aim to
develop. Business education encompasses case studies, real-world problem-
solving, interpersonal skills, and ethical decision-making, which may not be
adequately captured by LLMs’ performance on standardized tests. Additionally,
success on the GMAT does not necessarily equate to success in business school
or in business practice. The ability to transfer and apply test-taking skills
to real-world scenarios is a critical aspect of business education that may
not be reflected in the study’s scope. Moreover, business schools often use a
holistic approach to evaluate candidates, considering work experience,
leadership potential, and other personal attributes alongside test scores. A
study focused on GMAT performance alone may not account for these broader
evaluative criteria.

###  5.3 Memorization

Table 2 presents the performance of various models on both free and premium
GMAT exams. It’s important to note that accessing free exams requires users to
create an account on the official GMAT exam website. Furthermore, to access
premium exams, users must purchase them. When comparing the outcomes of free
versus premium exams across all models, we observe no significant difference
in performance. This finding, coupled with the fact that our GMAT materials
come from official sources requiring login or payment for premium content,
suggests that such content likely wasn’t part of the LLMs’ training data. Even
if some overlap or contamination exists, it doesn’t appear to notably enhance
the LLMs’ GMAT performance. OpenAI’s research supports this, indicating that
despite some contamination in publicly available benchmarks, it hasn’t led to
significant performance discrepancies between contaminated and uncontaminated
samples in their assessments [13].

GMAT |  GPT-4 Turbo |  GPT-4 |  GPT-3.5 Turbo |  Claude 2.1 |  Claude 2 |  Gemini Pro (New Bard) |  PaLM 2 (Legacy Bard)
---|---|---|---|---|---|---|---
Free Exam 1 |  88.06 |  74.63 |  61.19 |  73.13 |  62.69 |  58.21 |  43.28
Free Exam 2 |  79.10 |  71.64 |  49.25 |  73.13 |  59.70 |  77.61 |  58.21
Premium Exam |  88.06 |  76.12 |  62.69 |  70.15 |  58.21 |  76.12 |  50.75
Overall Average |  85.07 |  74.13 |  58.21 |  72.14 |  60.20 |  70.65 |  50.75
Table 2: Comparison of performance of models on the GMAT exams shows that
GPT-4 Turbo significantly surpasses the other models.

##  6 AI as Tutoring Assistant: A Case Study

LLMs can act as tutoring assistants, simplifying complex concepts for both
instructors and students. They can guide students through homework by
fostering critical thinking instead of just providing answers. LLMs can
generate practice questions, offer assignment feedback, and create tailored
study plans. They’re also adept at simulating exams, tracking progress, and
suggesting targeted practice areas, especially useful for language exams that
focus on vocabulary and grammar. Additionally, LLMs help with essay writing
and provide motivational support. As AI becomes more integrated into
education, it offers interactive and personalized learning experiences.
Engaging LLMs in interactive sessions can reveal their educational potential
and practical applications. Our study, which references the interactive
session described in [64], illustrates how a dialogue initiated by a single
critical reasoning question can showcase an AI model’s educational
capabilities. Using GPT-4 Turbo, we simulate a conversation between the model
and a student preparing for the GMAT, demonstrating the model’s ability to
correctly answer questions (Figure 11), explain the reasoning behind answers
(Figure 12), use theory-of-mind to hypothesize about errors (Figure 13),
personalize instruction (Figure 14), and engage in counterfactual reasoning by
modifying a critical reasoning problem to help the candidate consider
different outcomes (Figure 15). However, it is essential to verify the
accuracy of information generated in such interactions and in real-world
applications with expert review to ensure reliability. This investigation aims
to highlight AI’s practical applications in education and its potential to
facilitate personalized learning journeys.

Figure 11: GPT-4 Turbo selects the correct option in response to a critical
reasoning problem. Figure 12: GPT-4 Turbo evaluates a candidate’s response to
the question and explains the rationale for the correct response. Figure 13:
GPT-4 Turbo attempts to follow the candidate’s line of reasoning to identify
errors in his thought process. Figure 14: GPT-4 Turbo adjusts its tutoring
approach for personalized instruction. Figure 15: GPT-4 Turbo revises the
argument in the problem statement in order to construct a counterfactual
scenario.

##  7 Final Thoughts

The overall performance of Large Language Models (LLMs) on GMAT exam
demonstrates significant potential to revolutionize education and tutoring in
particular, while also highlighting certain challenges and limitations. We
anticipate that LLMs will become essential tools for students in their exam
preparation and for teachers in developing their pedagogical materials.
Additionally, we expect these models to significantly enhance the process of
standardized examinations. These insights suggest a rapidly evolving landscape
where AI models are not only becoming viable candidates for high-level
academic programs but are also setting new benchmarks for human applicants to
aspire to. The implications for the future of standardized testing and
admissions are profound, as AI continues to redefine the limits of performance
and potential. Nevertheless, there are some major concerns regarding the
deployment of LLMs in educational settings, specifically within the domain of
business education, that warrant careful consideration.

#### Social impacts.

One of the major concerns is that the use of LLMs in education could lead to
increased inequality [65]. Access to LLMs requires a stable internet
connection and devices capable of running such models, which may not be
available to all students. Therefore, students with access to these
technologies might gain unfair advantages over those who do not, potentially
exacerbating existing educational inequalities. Also, several studies indicate
that LLMs may perpetuate and amplify existing biases present in their training
data, potentially reinforcing stereotypes and contributing to inequality in
educational outcomes. Moreover, while research shows that face-to-face
interaction plays a crucial role in providing a sustainable learning
experience in business education [66], overreliance on LLMs for learning could
lead to a decrease in face-to-face interactions and the valuable social
aspects of learning, such as collaboration and discussion. Furthermore, the
adoption of LLMs could lead to concerns about job displacement [67], as AI
might be seen as a replacement for human teachers and tutors.

#### Risk of misinformation.

As our findings in this study showed, LLMs may generate incorrect or
misleading information, which could lead to confusion or the learning of
inaccurate material. As a result, students may inadvertently learn and
propagate these inaccuracies, leading to a spread of misinformation.

#### Impacts on personal development.

LLMs might provide answers too readily, which could discourage students from
developing their own critical thinking, problem-solving skills, and also
interpersonal skills. As a result, students might become dependent on AI,
leading to a decrease in self-efficacy and confidence in their abilities to
learn and solve problems independently.

#### Ethical considerations.

The integration of LLMs into education involves the collection and processing
of student data, raising concerns about privacy, data security, and the
potential misuse of personal information. Also, there may be concerns about
the use of copyrighted material when LLMs generate content or examples for
study purposes. Additionally, there is a risk that students might use LLMs to
complete assignments dishonestly, leading to challenges in ensuring academic
integrity and authentic assessment of student learning.

To mitigate these concerns, it’s important to use LLMs as a complement to
traditional educational methods and human interaction in business education,
rather than as a standalone solution. Additionally, ongoing monitoring,
evaluation, and adjustment of LLMs in educational settings are necessary to
ensure they are used effectively and ethically and to ensure that students are
developing their own critical thinking and problem-solving skills, rather than
becoming overly reliant on the technology. Furthermore, to regulate and
monitor the ethical use of AI in education, it is essential to establish
frameworks, guidelines, and oversight bodies.

Finally, while LLMs have demonstrated their implications in a wide range of
applications, they have several inherent limitations. Current LLMs, trained on
vast internet text sources like Reddit and Twitter, have versatile
capabilities like scriptwriting, translation, and recipe generation, but they
often stray off-topic, fabricate information, and show fluidity in
perspectives. Furthermore, while highly capable in strictly defined
calculations, current limitations persist in mathematical ambiguity and
complexity intrinsic to human cognition. Extending their precise computational
strengths while approximating human understanding and fluidity thus remains an
elusive frontier. LLMs face challenges with abstract or intricate concepts
that demand reasoning beyond the literal text. Their performance dips when
tasks require creative problem-solving, intuitive judgment, or comprehension
of context and linguistic nuances. Additionally, they may inadvertently change
the original meaning of a sentence due to their limited understanding of the
writer’s intent or the deeper connotations of the information. OpenAI tries to
mitigate some of these issues by fine-tuning LLMs with human feedback and
reinforcement learning [68], aiming for helpful, truthful, and harmless
outputs, though these goals can be subjective and challenging to define. The
statistical models of crowdsourcing, introduced by Phil Dawid and David Skene
in 1979 [69] and expanded upon in Cultural Consensus Theory [70] (CCT) by
Romney and colleagues, offer insights into assessing subjective judgments.
CCT, particularly, recognizes the lack of a singular consensus in diverse
opinions, a concept valuable for understanding and improving LLMs in handling
varied, complex information. This principle has led to further advancements,
including the fine-tuning of deep neural networks to align their outputs more
closely with consensus beliefs.[71]

##  8 Conclusion

In our study, we performed a comparative analysis of seven general-purpose
LLMs—GPT-4 Turbo, GPT-4, GPT-3.5 Turbo, Claude 2.1, Claude 2, Gemini 1.0 Pro,
and PaLM 2—focusing on their zero-shot performance in the GMAT’s quantitative
and verbal sections, which are crucial for global business school admissions.
GPT-4 Turbo stood out as the top performer, with the latest versions of the
other models also showing significant improvements over their predecessors. We
found that LLMs can achieve impressive results on the GMAT without the need
for complex prompting strategies, surpassing the average human test-taker.
This suggests that LLMs have inherent strengths that make them well-suited for
standardized testing. Our research provides a foundational understanding of
LLMs’ capabilities on the GMAT and suggests potential for further performance
optimization. Additionally, we showcased GPT-4 Turbo’s tutoring abilities,
such as explaining critical reasoning, assessing responses, using
metacognition to pinpoint student errors, and creating counterfactual
scenarios. We explored the broader implications of AI in educational settings,
suggesting that LLMs’ strong performance on the GMAT indicates their potential
value in business education and as a support tool for business professionals.
However, the potential for errors and the complexities of real-world
application call for caution. It is essential to carefully develop, evaluate,
and refine AI use, and to continue technological advancements to maximize the
benefits and minimize the risks of AI in education.

## References

  * [1] Shubham Melvin Felix, Sumer Kumar, and A Veeramuthu.  A smart personal ai assistant for visually impaired people.  In 2018 2nd international conference on trends in electronics and informatics (ICOEI), pages 1245–1250. IEEE, 2018.
  * [2] Lu Xu, Leslie Sanders, Kay Li, James CL Chow, et al.  Chatbot for health care and oncology applications using artificial intelligence and machine learning: systematic review.  JMIR cancer, 7(4):e27850, 2021.
  * [3] Olaf Zawacki-Richter, Victoria I Marín, Melissa Bond, and Franziska Gouverneur.  Systematic review of research on artificial intelligence applications in higher education–where are the educators?  International Journal of Educational Technology in Higher Education, 16(1):1–27, 2019.
  * [4] Yuri YM Aung, David CS Wong, and Daniel SW Ting.  The promise of artificial intelligence: a review of the opportunities and challenges of artificial intelligence in healthcare.  British medical bulletin, 139(1):4–15, 2021.
  * [5] Shazia Afzal, Tejas Dhamecha, Nirmal Mukhi, Renuka Sindhgatta Rajan, Smit Marvaniya, Matthew Ventura, and Jessica Yarbro.  Development and deployment of a large-scale dialog-based intelligent tutoring system.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Industry Papers), pages 114–121. Association for Computational Linguistics, 2019.
  * [6] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.  2018\.
  * [7] Sundar Pichai.  An important next step on our ai journey, 2023.  https://blog.google/technology/ai/bard-google-ai-search-updates/, Last accessed on 2023-11-26.
  * [8] Anthropic.  Claude 2, 2023.  https://www.anthropic.com/index/claude-2, Last accessed on 2023-11-26.
  * [9] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [10] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov.  Roberta: A robustly optimized bert pretraining approach.  arXiv preprint arXiv:1907.11692, 2019.
  * [11] Luciano Floridi and Massimo Chiriatti.  Gpt-3: Its nature, scope, limits, and consequences.  Minds and Machines, 30:681–694, 2020.
  * [12] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al.  Bloom: A 176b-parameter open-access multilingual language model.  arXiv preprint arXiv:2211.05100, 2022.
  * [13] OpenAI.  Gpt-4 technical report, 2023.
  * [14] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [15] Ramon Dijkstra, Zülküf Genç, Subhradeep Kayal, Jaap Kamps, et al.  Reading comprehension quiz generation using generative pre-trained transformers, 2022.
  * [16] Ebrahim Gabajiwala, Priyav Mehta, Ritik Singh, and Reeta Koshy.  Quiz maker: Automatic quiz generation from text using nlp.  In Futuristic Trends in Networks and Computing Technologies: Select Proceedings of Fourth International Conference on FTNCT 2021, pages 523–533. Springer, 2022.
  * [17] Enkelejda Kasneci, Kathrin Seßler, Stefan Küchemann, Maria Bannert, Daryna Dementieva, Frank Fischer, Urs Gasser, Georg Groh, Stephan Günnemann, Eyke Hüllermeier, et al.  Chatgpt for good? on opportunities and challenges of large language models for education.  Learning and individual differences, 103:102274, 2023.
  * [18] Thomas Scialom and Jacopo Staiano.  Ask to learn: A study on curiosity-driven question generation.  arXiv preprint arXiv:1911.03350, 2019.
  * [19] Rania Abdelghani, Yen-Hsiang Wang, Xingdi Yuan, Tong Wang, Pauline Lucas, Hélène Sauzéon, and Pierre-Yves Oudeyer.  Gpt-3-driven pedagogical agents to train children’s curious question-asking skills.  International Journal of Artificial Intelligence in Education, pages 1–36, 2023.
  * [20] Vahid Ashrafimoghari.  Big data and education: using big data analytics in language learning.  arXiv preprint arXiv:2207.10572, 2022.
  * [21] Steven Moore, Huy A Nguyen, Norman Bier, Tanvi Domadia, and John Stamper.  Assessing the quality of student-generated short answer questions using gpt-3.  In European conference on technology enhanced learning, pages 243–257. Springer, 2022.
  * [22] Mengxiao Zhu, Ou Lydia Liu, and Hee-Sun Lee.  The effect of automated feedback on revision behavior and learning gains in formative assessment of scientific argument writing.  Computers & Education, 143:103668, 2020.
  * [23] Michael Sailer, Elisabeth Bauer, Riikka Hofmann, Jan Kiesewetter, Julia Glas, Iryna Gurevych, and Frank Fischer.  Adaptive feedback from artificial neural networks facilitates pre-service teachers’ diagnostic reasoning in simulation-based learning.  Learning and Instruction, 83:101620, 2023.
  * [24] Jan Philip Bernius, Stephan Krusche, and Bernd Bruegge.  Machine learning based feedback on textual student answers in large courses.  Computers and Education: Artificial Intelligence, 3:100081, 2022.
  * [25] Sal Khan.  Harnessing gpt-4 so that all students benefit. a nonprofit approach for equal access.  Khan Academy, 2023.
  * [26] Sarang Shaikh, Sule Yildirim Yayilgan, Blanka Klimova, and Marcel Pikhart.  Assessing the usability of chatgpt for formal english language learning.  European Journal of Investigation in Health, Psychology and Education, 13(9):1937–1960, 2023.
  * [27] Michalis Vlachos, Mircea Lungu, Yash Raj Shrestha, and Johannes-Rudolf David.  Large language models for difficulty estimation of foreign language content with application to language learning.  arXiv preprint arXiv:2309.05142, 2023.
  * [28] Stephen MacNeil, Andrew Tran, Dan Mogil, Seth Bernstein, Erin Ross, and Ziheng Huang.  Generating diverse code explanations using the gpt-3 large language model.  In Proceedings of the 2022 ACM Conference on International Computing Education Research-Volume 2, pages 37–39, 2022.
  * [29] Shravya Bhat, Huy A Nguyen, Steven Moore, John Stamper, Majd Sakr, and Eric Nyberg.  Towards automated generation and evaluation of questions in educational domains.  In Proceedings of the 15th International Conference on Educational Data Mining, volume 701, 2022.
  * [30] Tung Phung, Victor-Alexandru Pădurean, Anjali Singh, Christopher Brooks, José Cambronero, Sumit Gulwani, Adish Singla, and Gustavo Soares.  Automating human tutor-style programming feedback: Leveraging gpt-4 tutor model for hint generation and gpt-3.5 student model for hint validation.  arXiv preprint arXiv:2310.03780, 2023.
  * [31] Abhimanyu S Ahuja, Bryce W Polascik, Divyesh Doddapaneni, Eamonn S Byrnes, and Jayanth Sridhar.  The digital metaverse: Applications in artificial intelligence, medical education, and integrative health.  Integrative Medicine Research, 12(1):100917, 2023.
  * [32] Hong Gao, Efe Bozkir, Lisa Hasenbein, Jens-Uwe Hahn, Richard Göllner, and Enkelejda Kasneci.  Digital transformations of classrooms in virtual reality.  In Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems, pages 1–10, 2021.
  * [33] Maria Roussou.  Immersive interactive virtual reality in the museum.  Proc. of TiLE (Trends in Leisure Entertainment), 2001.
  * [34] Andrea L Guzman and Seth C Lewis.  Artificial intelligence and communication: A human–machine communication research agenda.  New media & society, 22(1):70–86, 2020.
  * [35] Tiffany H Kung, Morgan Cheatham, Arielle Medenilla, Czarina Sillos, Lorie De Leon, Camille Elepaño, Maria Madriaga, Rimel Aggabao, Giezel Diaz-Candido, James Maningo, et al.  Performance of chatgpt on usmle: Potential for ai-assisted medical education using large language models.  PLoS digital health, 2(2):e0000198, 2023.
  * [36] Jonathan H Choi, Kristin E Hickman, Amy Monahan, and Daniel Schwarcz.  Chatgpt goes to law school.  Available at SSRN, 2023.
  * [37] Claudia E Haupt and Mason Marks.  Ai-generated medical advice—gpt and beyond.  Jama, 329(16):1349–1350, 2023.
  * [38] Daniel Martin Katz, Michael James Bommarito, Shang Gao, and Pablo Arredondo.  Gpt-4 passes the bar exam.  Available at SSRN 4389233, 2023.
  * [39] Arun James Thirunavukarasu, Darren Shu Jeng Ting, Kabilan Elangovan, Laura Gutierrez, Ting Fang Tan, and Daniel Shu Wei Ting.  Large language models in medicine.  Nature medicine, 29(8):1930–1940, 2023.
  * [40] Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, et al.  A comprehensive capability analysis of gpt-3 and gpt-3.5 series models.  arXiv preprint arXiv:2303.10420, 2023.
  * [41] R OpenAI.  Gpt-4 technical report.  arXiv:2303.08774v3, 2:3, 2023.
  * [42] Open AI.  Gpt-4 turbo, 2023.  https://help.openai.com/en/articles/8555510-gpt-4-turbo, Last accessed on 2023-12-26.
  * [43] Open AI.  Vision, 2023.  https://platform.openai.com/docs/guides/vision, Last accessed on 2023-12-26.
  * [44] Anthropic.  Introducing claude 2.1, 2023.  https://www.anthropic.com/index/claude-2-1, Last accessed on 2023-12-26.
  * [45] Rohan Anil, Andrew M Dai, Orhan Firat, Melvin Johnson, Dmitry Lepikhin, Alexandre Passos, Siamak Shakeri, Emanuel Taropa, Paige Bailey, Zhifeng Chen, et al.  Palm 2 technical report.  arXiv preprint arXiv:2305.10403, 2023.
  * [46] Gemini Team, Rohan Anil, Sebastian Borgeaud, Yonghui Wu, Jean-Baptiste Alayrac, Jiahui Yu, Radu Soricut, Johan Schalkwyk, Andrew M Dai, Anja Hauth, et al.  Gemini: a family of highly capable multimodal models.  arXiv preprint arXiv:2312.11805, 2023.
  * [47] Google.  Bard gets its biggest upgrade yet with gemini, 2023.  https://blog.google/products/bard/google-bard-try-gemini-ai/, Last accessed on 2023-12-26.
  * [48] GMAC.  Gmat by the numbers, 2023.  https://www.mba.com/exams/gmat-exam, accessed on 2023-11-26.
  * [49] Aitor Lewkowycz, Anders Andreassen, David Dohan, Ethan Dyer, Henryk Michalewski, Vinay Ramasesh, Ambrose Slone, Cem Anil, Imanol Schlag, Theo Gutman-Solo, et al.  Solving quantitative reasoning problems with language models.  Advances in Neural Information Processing Systems, 35:3843–3857, 2022.
  * [50] Bonan Min, Hayley Ross, Elior Sulem, Amir Pouran Ben Veyseh, Thien Huu Nguyen, Oscar Sainz, Eneko Agirre, Ilana Heintz, and Dan Roth.  Recent advances in natural language processing via large pre-trained language models: A survey.  ACM Computing Surveys, 56(2):1–40, 2023.
  * [51] Linlu Qiu, Liwei Jiang, Ximing Lu, Melanie Sclar, Valentina Pyatkin, Chandra Bhagavatula, Bailin Wang, Yoon Kim, Yejin Choi, Nouha Dziri, et al.  Phenomenal yet puzzling: Testing inductive reasoning capabilities of language models with hypothesis refinement.  arXiv preprint arXiv:2310.08559, 2023.
  * [52] Tongquan Zhou, Yao Zhang, Siyi Cao, Yulu Li, and Tao Wang.  Complementary advantages of chatgpts and human readers in reasoning: Evidence from english text reading comprehension.  arXiv preprint arXiv:2311.10344, 2023.
  * [53] Ming Jin, Shiyu Wang, Lintao Ma, Zhixuan Chu, James Y Zhang, Xiaoming Shi, Pin-Yu Chen, Yuxuan Liang, Yuan-Fang Li, Shirui Pan, et al.  Time-llm: Time series forecasting by reprogramming large language models.  arXiv preprint arXiv:2310.01728, 2023.
  * [54] Ying Guo and Daniel Lee.  Leveraging chatgpt for enhancing critical thinking skills.  Journal of Chemical Education, 2023.
  * [55] Csaba Veres.  Large language models are not models of natural language: they are corpus models.  IEEE Access, 10:61970–61979, 2022.
  * [56] Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Kaijie Zhu, Hao Chen, Linyi Yang, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al.  A survey on evaluation of large language models.  arXiv preprint arXiv:2307.03109, 2023.
  * [57] GMAC.  Understanding your score, 2023.  https://www.mba.com/exams/gmat-exam/scores/understanding-your-score, Last accessed on 2023-11-26.
  * [58] Financial Times.  Business school rankings, 2023.  https://rankings.ft.com/rankings/2909/mba-2023, Last accessed on 2023-12-05.
  * [59] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
  * [60] Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al.  Retrieval-augmented generation for knowledge-intensive nlp tasks.  Advances in Neural Information Processing Systems, 33:9459–9474, 2020.
  * [61] Tongshuang Wu, Ellen Jiang, Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie J Cai.  Promptchainer: Chaining large language model prompts through visual programming.  In CHI Conference on Human Factors in Computing Systems Extended Abstracts, pages 1–10, 2022.
  * [62] Zachary Levonian, Chenglu Li, Wangda Zhu, Anoushka Gade, Owen Henkel, Millie-Ellen Postle, and Wanli Xing.  Retrieval-augmented generation to improve math question-answering: Trade-offs between groundedness and human preference.  arXiv preprint arXiv:2310.03184, 2023.
  * [63] Valentin Liévin, Christoffer Egeberg Hother, and Ole Winther.  Can large language models reason about medical questions?  arXiv preprint arXiv:2207.08143, 2022.
  * [64] Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz.  Capabilities of gpt-4 on medical challenge problems.  arXiv preprint arXiv:2303.13375, 2023.
  * [65] Erin Hannan and Shuguang Liu.  Ai: new source of competitiveness in higher education.  Competitiveness Review: An International Business Journal, 33(2):265–279, 2023.
  * [66] Lynn A Fish and Coral R Snodgrass.  Comparing instructor and student perspectives of online versus face-to-face education for program factors during the pandemic.  Journal of Education for Business, 98(8):452–461, 2023.
  * [67] Giriraj Kiradoo.  Unlocking the potential of ai in business: Challenges and ethical considerations.  Recent Progress in Science and Technology, 6:205–220, 2023.
  * [68] Nathan Lambert, Louis Castricato, Leandro von Werra, and Alex Havrilla.  Illustrating reinforcement learning from human feedback (rlhf).  Hugging Face Blog, 2022.  https://huggingface.co/blog/rlhf.
  * [69] Alexander Philip Dawid and Allan M Skene.  Maximum likelihood estimation of observer error-rates using the em algorithm.  Journal of the Royal Statistical Society: Series C (Applied Statistics), 28(1):20–28, 1979.
  * [70] A Kimball Romney, Susan C Weller, and William H Batchelder.  Culture as consensus: A theory of culture and informant accuracy.  American anthropologist, 88(2):313–338, 1986.
  * [71] Necdet Gürkan and Jordan W Suchow.  Harnessing collective intelligence under a lack of cultural consensus.  arXiv preprint arXiv:2309.09787, 2023.

##  Appendix A Sample of Input/Output for various GMAT tasks

Figure A.1: Sample of the user’s input and the model’s response for a problem
solving task

Figure A.2: Sample of the user’s input and the model’s response for a data
sufficiency task

Figure A.3: Sample of the user’s input and the model’s response for a reading
comprehension task

Figure A.4: Sample of the user’s input and the model’s response for a critical
reasoning task  Figure A.5: Sample of the user’s input and the model’s
response for a sentence correction task


claim (and it remains to prove) that the adiabatic limit of the eta-invariant
for the spin Dirac operator is

$\lim_{\varepsilon\to 0}\tfrac{1}{2}\eta_{\varepsilon}=\frac{1}{24}[Y]^{2},$
(5.3)

i.e. the limit of the eta-invariants is the opposite of the local boundary
term when $\beta=1$, which indeed it should be since in that case the metric
is smooth across $x=0$.

Although other derivations of the adiabatic eta invariant exist [27], we
prefer to give on here which we find intuitive and which fits nicely with
arguments above. To this end, we consider $N$, a disc bundle over a smooth
manifold $Y$, and we assume $N$ is spin. We will show below that $N$ admits a
positive scalar curvature metric. Thus, given a spin structure and metric, the
index of $\eth$ vanishes on $N$. If we furthermore note that $N$ is
diffeomorphic to $[0,1)_{x}\times X$ where $X$ is a circle bundle over $Y$,
and let $N^{\varepsilon}=[0,\varepsilon)_{x}\times X$, we may consider metrics

$g=dx^{2}+f^{2}(x)k+h,$ (5.4)

where $h$ is the pullback of a metric on $Y$, $k\in
Sym^{0,2}(N^{\varepsilon})$ $x$ and $dx$-independent and restrics to a
Riemannian metric on the fibers of $X$. We assume $f$ is smooth across $x=0$
with $f(x)=x+O(x^{2})$ which implies that $g$ is smooth on $N$. Using the
computation of the connection above, with respect to the orthonormal basis,
$X_{i},\frac{1}{f}U,\partial_{x}$ the connection one form of $g$ is

$\begin{split}\omega&=\left(\begin{array}[]{c|c|c}\widetilde{\omega}_{\widetilde{h}}-f^{2}\frac{1}{2}g^{\partial}\mathcal{R}&-fg^{\partial}\left(\widehat{\@slowromancap
ii@}+\frac{1}{2}\widehat{\mathcal{R}}\right)&0\\\ \hline\cr
fg^{\partial}\left(\widehat{\@slowromancap
ii@}+\frac{1}{2}\widehat{\mathcal{R}}\right)&0&f^{\prime}U^{\sharp}\\\
\hline\cr 0&-f^{\prime}U^{\sharp}&0\end{array}\right)\end{split}$ (5.5)

where $g^{\partial}=k+h$ is the metric on the circle bundle $X$. We will take

$f(x)=f_{\varepsilon}(x)=x\chi_{\varepsilon}(x),$

where $\chi$ is a smooth positive function that is monotone decreasing with
$\chi(x)=1$ for $x\leq 1/3$ and $\chi(x)=\beta$ for $x\geq 2/3$. Then
$f=f^{\prime}=f^{\prime\prime}=O(1/\varepsilon)$, and using
$\Omega=d\omega+\omega\wedge\omega$, we see that

$\widehat{A}_{g}=FdVol_{g}$

where $F$ is a function that is $O(1/\varepsilon)$. Since
$Vol(N_{\varepsilon})=O(\varepsilon^{2})$,

$\int_{N_{\varepsilon}}\widehat{A}_{\varepsilon}=-\frac{1}{24}\int_{N_{\varepsilon}}p_{1}\to
0\mbox{ as }\varepsilon\to 0.$

Since the index of the Dirac operator vanishes on $N^{\varepsilon}$, applying
the APS formula gives

$\begin{split}0=-\frac{1}{24}\int_{N^{\varepsilon}}p_{1}+\int_{\partial
N^{\varepsilon}}Tp_{1}(\nabla,\nabla^{\operatorname{pt}})-\tfrac{1}{2}\eta(\partial
N^{\varepsilon}),\end{split}$ (5.6)

where $\nabla^{pt}$ is as in (3), and thus the limit of the trangression forms
is exactly as computed above. Thus by taking the $\varepsilon\to 0$ limit we
obtain (5.3).

To prove part 1) it remains to prove the existence of a positive scalar
curvature metric on $N$. To this end we take the metric $g$ as in (5.4) on
$N^{\varepsilon}$ now with

$f(x)=f_{\delta}(x)=\delta\sin(x/\delta).$ (5.7)

Note that $f=O(\varepsilon),f^{\prime}=O(\varepsilon/\delta)$. Then curvature
equals

$\begin{split}\Omega&=d\omega+\omega\wedge\omega\\\
&=\left(\begin{array}[]{c|c|c}\widetilde{\Omega}_{\widetilde{h}}&0&0\\\
\hline\cr 0&0&f^{\prime\prime}dx\wedge U^{\sharp}\\\ \hline\cr
0&-f^{\prime\prime}dx\wedge
U^{\sharp}&0\end{array}\right)+O(\varepsilon)+O(\varepsilon/\delta).\end{split}$
(5.8)

Denoting our orthonormal basis by $e_{i},i=1,\dots,n$ and taking traces gives

$scal_{g}=\delta^{ik}\delta^{jl}\Omega_{ij}(e_{k},e_{l})=scal_{h}+\frac{2}{\delta^{2}}+O(\varepsilon/\delta),$

and thus taking $\varepsilon/\delta=1$ and $\delta$ small gives a positive
scalar curvature metric.

2) Since $X$ is a smooth manifold we can use Novikov additivity of the
signature to decompose the signature as

$\operatorname{sgn}(X)=\operatorname{sgn}(X\setminus
M_{\varepsilon})+\operatorname{sgn}(M_{\varepsilon}).$

Identifying $X\setminus M_{\varepsilon}$ with a disk bundle over $Y$ we have
from [25, Pg. 314] that

$\operatorname{sgn}(X\setminus
M_{\varepsilon})=\operatorname{sgn}\left(\int_{Y}e\right),$

i.e., the signature is the sign of the self-intersection number of $Y$ in $X.$
In fact this is a simple exercise using the Thom isomorphism theorem.

The Atiyah-Patodi-Singer index theorem for the signature of $M_{\varepsilon}$
yields

$\operatorname{sgn}(M_{\varepsilon})=\frac{1}{3}\int_{M_{\varepsilon}}p_{1}(\nabla)+\frac{1}{3}\int_{\partial
M_{\varepsilon}}Tp_{1}(\nabla,\nabla^{\operatorname{pt}})-\eta^{\operatorname{even}}_{\varepsilon}$

where $\eta^{\operatorname{even}}_{\varepsilon}$ is the eta-invariant of the
boundary signature operator restricted to forms of even degree. As
$\varepsilon\to 0,$ the eta invariant is undergoing adiabatic degeneration and
its limit is computed in [27, Theorem 3.2],

$\lim_{\varepsilon\to 0}=-\int_{Y}L(TY)(\coth
e-e^{-1})+\operatorname{sgn}\left(B_{e}\right)$

where $B_{e}$ is the bilinear form on $H^{0}(Y)$ given by $H^{0}(Y)\ni
c,c^{\prime}\mapsto cc^{\prime}\langle e,Y\rangle\in\mathbb{R},$ i.e., it is
again the sign of the self-intersection of $Y.$ (In comparing with [27] note
that the orientation of $\partial M_{\varepsilon}$ is the opposite of the
orientation of the spherical normal bundle of $Y$ in $X,$ and so
$\operatorname{sgn}(B_{e})=-\operatorname{sgn}(X\setminus M_{\varepsilon}).$)
The only term in $L(TY)(\coth e-e^{-1})$ of degree two is $\tfrac{1}{3}e,$ and
hence

$\operatorname{sgn}(X)=\frac{1}{3}\int_{X}p_{1}+\frac{1}{3}[Y]^{2}+\frac{1}{3}\left(-\beta^{2}[Y]^{2}\right)$

as required. (Note that we could also argue as in the Dirac case to compute
the limit of the eta invariants.)

∎

##  6\. Positive scalar curvature metrics

In this short section, we prove Theorem 3 following [23]. We recall the
statement of the theorem for the benefit of the reader:

######  Theorem 6.1.

Let $(M,g)$ be a spin space with an incomplete edge metric.
a) If the scalar curvature of $g$ is non-negative in a neighborhood of
$\partial M$ then the geometric Witt assumption (Assumption 2.1) holds.
b) If the scalar curvature of $g$ is non-negative on all of $M,$ and positive
somewhere, then $\operatorname{Ind}(\eth)=0.$

###### Proof.

a) Taking traces in (1.11), the scalar curvature $R_{g}$ satisfies

$R_{g}=R_{cone}+\mathcal{O}(1),$

where $R_{cone}$ is the scalar curvature of the cone with metric
$dx^{2}+x^{2}g_{N/Y}\rvert_{\langle\partial_{x}\rangle\oplus\tfrac{1}{x}TN/Y}$,
as in (1.7). On the other hand, by [23, Sect. 4], the scalar curvature of an
exact cone $C(Z)$ is equal to $x^{-2}(R_{Z}-\dim(Z)(\dim(Z)-1))$, where
$R_{Z}$ is the scalar curvature of $Z$. Thus $R_{g}\geq 0$ implies that
$R_{Z}\geq 0$, which by [23, Lemma 3.5] shows that Assumption 2.1 holds.

b) First off, by Theorem 1, $\eth$ is essentially self-adjoint. That is, the
graph closure of $\eth$ on $C^{\infty}_{comp}(M)$ is self-adjoint, with domain
$\mathcal{D}$ from Theorem 1, and furthermore by the Main Theorem its index
satisfies (4).

From the Lichnerowicz formula [10],

$\eth^{*}\eth=\nabla^{*}\nabla+R/4$

where $R$ is the scalar curvature. Thus, for every $\phi\in
C^{\infty}_{comp}(M)$,
$\left\|\eth\phi\right\|_{L^{2}}=\left\|\nabla\phi\right\|_{L^{2}}+\langle
R\phi,\phi\rangle_{L^{2}}.$ We conclude that for all $\phi\in
C^{\infty}_{comp}(M),$

$\left\|\eth\phi\right\|_{L^{2}}\geq\left\|\eth\phi\right\|_{L^{2}}-\langle
R\phi,\phi\rangle_{L^{2}}\geq\left\|\nabla\phi\right\|_{L^{2}}\geq 0.$ (6.1)

This implies in particular that
$\mathcal{D}_{min}(\eth)=\mathcal{D}\subset\mathcal{D}_{min}(\nabla)$, where
we recall that $\mathcal{D}_{min}(P)$ refers to the graph closure of the
operator $P$ with domain $C^{\infty}_{comp}(M)$. We claim that the index of
the operator

$\eth\colon\mathcal{D}^{+}\longrightarrow L^{2}(M;\mathcal{S}^{-}).$

vanishes, so by formula (4), Theorem 3(b) holds. In fact, the kernel of $\eth$
on $\mathcal{D}$ consists only of the zero vector, since if
$\phi\in\mathcal{D}$ has $\eth\phi=0$, then since (6.1) holds on
$\mathcal{D}$, $\nabla\phi=0$ also. By the Lichnerowicz formula again,
$R\phi=0$, but since by assumption $R$ is not identically zero, $\phi$ must
vanish somewhere and by virtue of its being parallel, $\phi\equiv 0$. ∎

##  7\. Appendix

In this appendix we prove Claim 4.8 by using standard bounds on modified
Bessel functions to prove the sup norm bound (4.9): for each $\mu$ with
$\left\lvert\mu\right\rvert>1/2$, and all $\left\lvert\eta\right\rvert$,

$\left\|\mathcal{N}_{\mu,\left\lvert\eta\right\rvert}-\mathcal{N}^{APS}_{\mu,\left\lvert\eta\right\rvert}\right\|<1-\delta,$

Among references for modified Bessel functions we recall [5, 8, 9, 49].

To begin with, using the Wronskian equation (2.27), note that

$\operatorname{Tr}\mathcal{N}_{\mu,z}=\operatorname{Tr}\mathcal{N}^{APS}_{\mu,z}=1.$

Thus the difference $\mathcal{N}_{\mu,z}-\mathcal{N}^{APS}_{\mu,z}$ has two
equal eigenvalues and hence its norm is the square root of the determinant. We
now assume that $\mu\geq 1/2$, since the $\mu\leq-1/2$ case is treated the
same way. Using (2.27) again, we see that

$\begin{split}\operatorname{det}(\mathcal{N}_{\mu,z}-\mathcal{N}^{APS}_{\mu,z})&=-\frac{1}{2}+\frac{1}{2}\frac{z}{(\mu^{2}+z^{2})^{1/2}}\left(\mu(I_{\mu-1/2}K_{\mu+1/2}-I_{\mu+1/2}K_{\mu-1/2})\right.\\\
&\qquad\left.+z(I_{\mu+1/2}K_{\mu+1/2}+I_{\mu-1/2}K_{\mu-1/2})\right),\end{split}$
(7.1)

and we want to show that for some $\delta>0$ independent of $\mu\geq 1/2$ and
$z\geq 0$,

$\begin{split}-1+\delta\leq\operatorname{det}(\mathcal{N}_{\mu,z}-\mathcal{N}^{APS}_{\mu,z})\leq
1-\delta.\end{split}$ (7.2)

To begin with, we prove that

$0\leq zI_{\nu}(z)K_{\nu}(z)\leq 1/2\quad\mbox{ for }\quad\nu\geq 1/2,z\geq
0.$ (7.3)

In fact, we claim that for $\nu\geq 1/2$, $zK_{\nu}(z)I_{\nu}(z)$ is monotone.
To see that this holds, differentiate

$\begin{split}(zK_{\nu}(z)I_{\nu}(z))^{\prime}&=K_{\nu}I_{\nu}+z(K^{\prime}_{\nu}I_{\nu}+K_{\nu}I^{\prime}_{\nu})=K_{\nu}I_{\nu}(1+\frac{zK^{\prime}_{\nu}(z)}{K_{\nu}(z)}+\frac{zI^{\prime}_{\nu}(z)}{I_{\nu}(z)}).\end{split}$

Thus we want to show that
$\frac{zK^{\prime}_{\nu}(z)}{K_{\nu}(z)}+\frac{zI^{\prime}_{\nu}(z)}{I_{\nu}(z)})\geq-1$.
Using [9, Eqn. 5.1], for $\nu\geq 1/2$

$\left(\frac{zK^{\prime}_{\nu}(z)}{K_{\nu}(z)}\right)^{\prime}+\left(\frac{zI^{\prime}_{\nu}(z)}{I_{\nu}(z)}\right)^{\prime}\leq
0,$

so the quantity
$\frac{zK^{\prime}_{\nu}(z)}{K_{\nu}(z)}+\frac{zI^{\prime}_{\nu}(z)}{I_{\nu}(z)}$
is monotone decreasing. In fact, we claim that

$\frac{zK^{\prime}_{\nu}(z)}{K_{\nu}(z)}+\frac{zI^{\prime}_{\nu}(z)}{I_{\nu}(z)}\left\\{\begin{array}[]{cc}\to
0&\mbox{ as }z\to 0\\\ \to-1&\mbox{ as }z\to\infty.\end{array}\right.$

The limit as $z\to\infty$ can be seen using the large argument asymptotic
formulas from [1, Sect. 9.7], while the limit as $z\to 0$ follows from the
recurrence relations (2.27) and the small argument asymptotics in [1, Sect.
9.6]. Thus $zK_{\nu}(z)I_{\nu}(z)$ is monotone on the region under
consideration. Using the asymptotic formulas again shows that

$zK_{\nu}(z)I_{\nu}(z)\left\\{\begin{array}[]{cc}\to 0&\mbox{ as }z\to 0\\\
\to 1/2&\mbox{ as }z\to\infty.\end{array}\right.$

so (7.3) holds.

We can now show the upper bound in (7.2). Using the Wronskian relation in
(2.27), we write

$\begin{split}\operatorname{det}(\mathcal{N}_{\mu,z}-\mathcal{N}^{APS}_{\mu,z})&=-\frac{1}{2}+\frac{1}{2}\frac{\mu}{(\mu^{2}+z^{2})^{1/2}}+\frac{1}{2}\frac{z}{(\mu^{2}+z^{2})^{1/2}}\left(-2\mu
I_{\mu+1/2}K_{\mu-1/2}\right.\\\
&\qquad\left.+z(I_{\mu+1/2}K_{\mu+1/2}+I_{\mu-1/2}K_{\mu-1/2})\right)\\\
&\leq\frac{1}{2}\frac{z}{(\mu^{2}+z^{2})^{1/2}}\left(z(I_{\mu+1/2}K_{\mu+1/2}+I_{\mu-1/2}K_{\mu-1/2})\right)\end{split}$
(7.4)

Now, if $\mu\geq 1$, by (7.3), the right hand side in the final inequality is
bounded by $1/2$, establishing the upper bound in (7.1) in this case (with
$\delta=1/2$). If $\mu\in[1/2,1]$, we use the following inequalities of Barciz
[9, Equations 2.3, 2.4]

$\frac{zI_{\nu}^{\prime}(z)}{I_{\nu}(z)}<\sqrt{z^{2}+\nu^{2}}\qquad\mbox{ and
}\qquad\frac{zK_{\nu}^{\prime}(z)}{K_{\nu}(z)}<-\sqrt{z^{2}+\nu^{2}},$

for $\nu\geq 0,z\geq 0$. Using these inequalities and the recurrence relation
(2.27) gives

$\frac{I_{\mu-1/2}}{I_{\mu+1/2}}<\frac{\sqrt{z^{2}+(\mu-1/2)^{2}}+\mu-1/2}{z},\quad\frac{K_{\mu-1/2}}{K_{\mu+1/2}}<\frac{z}{\sqrt{z^{2}+(\mu+1/2)^{2}}+\mu+1/2},$

so continuing the inequality (7.4) gives

$\begin{split}\operatorname{det}(\mathcal{N}_{\mu,z}-\mathcal{N}^{APS}_{\mu,z})&\leq\frac{1}{2}\frac{z}{(\mu^{2}+z^{2})^{1/2}}\left(zI_{\mu+1/2}K_{\mu+1/2}\right)\times\\\
&\qquad\left(1+\frac{\sqrt{z^{2}+(\mu+1/2)^{2}}+\mu+1/2}{\sqrt{z^{2}+(\mu-1/2)^{2}}+\mu-1/2}\right).\end{split}$
(7.5)

One checks that for $1/2\leq\mu$, the fraction in the second line is monotone
decreasing in $z$, and thus by (7.3), for $z\geq 1$ the determinant is bounded
by

$\frac{1}{4}\left(1+\frac{\sqrt{1+(\mu+1/2)^{2}}+\mu+1/2}{\sqrt{1+(\mu-1/2)^{2}}+\mu-1/2}\right)\leq\frac{1}{4}(1+(1+\sqrt{2}))\leq
1-\delta$ (7.6)

where the middle bound is obtained by checking that the fraction on the left
is monotone decreasing in $\mu$ for $\mu\geq 1/2$ and equal to $1+\sqrt{2}$ at
$\mu=1/2$. Thus, we have established the upper bound in (7.2) in the region
$z\geq 1$. For $z\leq 1$, rewrite the bound in (7.5) as

$\begin{split}\frac{1}{2}\frac{z}{(\mu^{2}+z^{2})^{1/2}}\left(I_{\mu+1/2}K_{\mu+1/2}\right)\times
z\left(1+\frac{\sqrt{z^{2}+(\mu+1/2)^{2}}+\mu+1/2}{\sqrt{z^{2}+(\mu-1/2)^{2}}+\mu-1/2}\right).\end{split}$

For $\mu\geq 1/2$, by [49], the function $I_{\mu+1/2}(z)K_{\mu+1/2}(z)$ is
monotone decreasing, and by the asymptotic formulas it is goes to $1/2$ as
$z\to 0$. Thus in $0\leq z\leq 1$ the determinant is bounded about by

$\frac{1}{4}\times
z\left(1+\frac{\sqrt{z^{2}+(\mu+1/2)^{2}}+\mu+1/2}{\sqrt{z^{2}+(\mu-1/2)^{2}}+\mu-1/2}\right).$

This function is monotone increasing in $z$ for $\mu\in[1/2,1]$, so the max is
obtained at $z=1$, i.e. it is bounded by the left hand side of (7.6), in
particular by $1-\delta$ for the same $\delta$. This establishes the upper
bound in (7.2)

Finally we establish the lower bound. First, we rewrite the determinant again,
this time using the Wronskian relation in the opposite direction to obtain

$\begin{split}\operatorname{det}(\mathcal{N}_{\mu,z}-\mathcal{N}^{APS}_{\mu,z})&=-\frac{1}{2}-\frac{1}{2}\frac{\mu}{(\mu^{2}+z^{2})^{1/2}}+\frac{1}{2}\frac{z}{(\mu^{2}+z^{2})^{1/2}}\left(2\mu
I_{\mu-1/2}K_{\mu+1/2}\right.\\\
&\qquad\left.+z(I_{\mu+1/2}K_{\mu+1/2}+I_{\mu-1/2}K_{\mu-1/2})\right).\end{split}$
(7.7)

Now, recalling that $zI_{\mu+1/2}(1)K_{\mu+1/2}(1)$ is monotone increasing,
using the asymptotic formulas [1, 9.7.7, 9.7.8] we see that

$I_{\mu+1/2}(1)K_{\mu+1/2}(1)\to\frac{1}{2(\mu+1/2)}$

as $\mu\to\infty$, we use the inequality [5, Eqn. 11], namely

$I_{\mu-1/2}(z)\geq\frac{\mu-1/2+(z^{2}+(\mu+3/2)^{2})^{1/2}}{z}I_{\mu+1/2}(z).$

On the region $z\in[0,1]$,
$\mu-1/2+(z^{2}+(\mu+3/2)^{2})^{1/2}\geq\delta_{0}>0$. Dropping the terms with
equal order in (7.7) then gives

$\begin{split}\operatorname{det}(\mathcal{N}_{\mu,z}-\mathcal{N}^{APS}_{\mu,z})&>-1+\frac{1}{2}\frac{z}{(\mu^{2}+z^{2})^{1/2}}2\mu
I_{\mu-1/2}K_{\mu+1/2}\\\
&\geq-1+\frac{1}{2}\frac{2\mu}{(\mu^{2}+z^{2})^{1/2}}I_{\mu+1/2}K_{\mu+1/2}(\mu-1/2+(z^{2}+(\mu+3/2)^{2})^{1/2})\\\
&\geq-1+\delta_{0}\frac{1}{2}\frac{\mu-1/2+(z^{2}+(\mu+3/2)^{2})^{1/2}}{(\mu^{2}+z^{2})^{1/2}}\\\
&\geq-1+\delta.\end{split}$

This completes the proof of (7.2).

## References

  * [1] M. Abramowitz and I. A. Stegun.  Handbook of mathematical functions with formulas, graphs, and mathematical tables, volume 55 of National Bureau of Standards Applied Mathematics Series.  For sale by the Superintendent of Documents, U.S. Government Printing Office, Washington, D.C., 1964.
  * [2] P. Albin, É. Leichtnam, R. Mazzeo, and P. Piazza.  The signature package on Witt spaces.  Ann. Sci. Éc. Norm. Supér. (4), 45(2):241–310, 2012.
  * [3] P. Albin, É. Leichtnam, R. Mazzeo, and P. Piazza.  Hodge theory on Cheeger spaces.  2013\.  Available online at http://arxiv.org/abs/1307.5473.v2.
  * [4] B. Ammann, E. Humbert, and B. Morel.  Mass endomorphism and spinorial Yamabe type problems on conformally flat manifolds.  Comm. Anal. Geom., 14(1):163–182, 2006.
  * [5] D. E. Amos.  Computation of modified Bessel functions and their ratios.  Math. Comp., 28:239–251, 1974.
  * [6] M. Atiyah and C. Lebrun.  Curvature, cones and characteristic numbers.  Math. Proc. Cambridge Philos. Soc., 155(1):13–37, 2013.
  * [7] M. F. Atiyah, V. K. Patodi, and I. M. Singer.  Spectral asymmetry and Riemannian geometry. I.  Math. Proc. Cambridge Philos. Soc., 77:43–69, 1975.
  * [8] Á. Baricz.  Bounds for modified Bessel functions of the first and second kinds.  Proc. Edinb. Math. Soc. (2), 53(3):575–599, 2010.
  * [9] Á. Baricz.  Bounds for Turánians of modified Bessel functions.  Available online at http://arxiv.org/abs/1202.4853, 2013.
  * [10] N. Berline, E. Getzler, and M. Vergne.  Heat kernels and Dirac operators.  Grundlehren Text Editions. Springer-Verlag, Berlin, 2004.  Corrected reprint of the 1992 original.
  * [11] J.-M. Bismut and J. Cheeger.  $\eta$-invariants and their adiabatic limits.  J. Amer. Math. Soc., 2(1):33–70, 1989.
  * [12] J.-M. Bismut and J. Cheeger.  Families index for manifolds with boundary, superconnections, and cones. I. Families of manifolds with boundary and Dirac operators.  J. Funct. Anal., 89(2):313–363, 1990.
  * [13] J.-M. Bismut and J. Cheeger.  Families index for manifolds with boundary, superconnections and cones. II. The Chern character.  J. Funct. Anal., 90(2):306–354, 1990.
  * [14] J.-M. Bismut and J. Cheeger.  Remarks on the index theorem for families of Dirac operators on manifolds with boundary.  In Differential geometry, volume 52 of Pitman Monogr. Surveys Pure Appl. Math., pages 59–83. Longman Sci. Tech., Harlow, 1991.
  * [15] J.-M. Bismut and D. S. Freed.  The analysis of elliptic families. II. Dirac operators, eta invariants, and the holonomy theorem.  Comm. Math. Phys., 107(1):103–163, 1986.
  * [16] B. Booß-Bavnbek and K. P. Wojciechowski.  Elliptic boundary problems for Dirac operators.  Mathematics: Theory & Applications. Birkhäuser Boston Inc., Boston, MA, 1993.
  * [17] R. Bott and L. W. Tu.  Differential forms in algebraic topology, volume 82 of Graduate Texts in Mathematics.  Springer-Verlag, New York-Berlin, 1982.
  * [18] J. Brüning.  The signature operator on manifolds with a conical singular stratum.  Astérisque, (328):1–44 (2010), 2009.
  * [19] J. Brüning and R. Seeley.  An index theorem for first order regular singular operators.  Amer. J. Math., 110:659Ð714, 1988.
  * [20] J. Cheeger.  On the spectral geometry of spaces with cone-like singularities.  Proc. Nat. Acad. Sci. U.S.A., 76(5):2103–2106, 1979.
  * [21] J. Cheeger.  On the Hodge theory of Riemannian pseudomanifolds.  In Geometry of the Laplace operator (Proc. Sympos. Pure Math., Univ. Hawaii, Honolulu, Hawaii, 1979), Proc. Sympos. Pure Math., XXXVI, pages 91–146. Amer. Math. Soc., Providence, R.I., 1980.
  * [22] J. Cheeger.  Spectral geometry of singular Riemannian spaces.  J. Differential Geom., 18(4):575–657 (1984), 1983.
  * [23] A. W. Chou.  The Dirac operator on spaces with conical singularities and positive scalar curvatures.  Trans. Amer. Math. Soc., 289(1):1–40, 1985.
  * [24] A. W. Chou.  Criteria for selfadjointness of the Dirac operator on pseudomanifolds.  Proc. Amer. Math. Soc., 106(4):1107–1116, 1989.
  * [25] X. Dai.  Adiabatic limits, nonmultiplicativity of signature, and Leray spectral sequence.  J. Amer. Math. Soc., 4:265 – 321, 1991.
  * [26] X. Dai and G. Wei.  Hitchin-Thorpe inequality for noncompact Einstein 4-manifolds.  Adv. Math., 214(2):551–570, 2007.
  * [27] X. Dai and W. P. Zhang.  Circle bundles and the Kreck-Stolz invariant.  Trans. Amer. Math. Soc., 347(9):3587–3593, 1995.
  * [28] B. Fedosov, B.-W. Schulze, and N. Tarkhanov.  The index of elliptic operators on manifolds with conical points.  Selecta Math. (N.S.), 5(4):467–506, 1999.
  * [29] J. B. Gil, P. A. Loya, and G. A. Mendoza.  A note on the index of cone differential operators.  arXiv:math/0110172, 2001.
  * [30] J. B. Gil and G. A. Mendoza.  Adjoints of elliptic cone operators.  Amer. J. Math., 125(2):357–408, 2003.
  * [31] P. B. Gilkey.  On the index of geometrical operators for Riemannian manifolds with boundary.  Adv. Math., 102(2):129–183, 1993.
  * [32] G. Grubb.  Heat operator trace expansions and index for general Atiyah-Patodi-Singer boundary problems.  Comm. Partial Differential Equations, 17(11-12):2031–2077, 1992\.
  * [33] T. Hausel, E. Hunsicker, and R. Mazzeo.  Hodge cohomology of gravitational instantons.  Duke Math. J., 122(3):485–548, 2004.
  * [34] L. Hörmander.  The analysis of linear partial differential operators. III.  Classics in Mathematics. Springer-Verlag, Berlin, 2007.  Pseudo-differential Operators, Reprint of the 1994 edition.
  * [35] H. B. Lawson, Jr. and M.-L. Michelsohn.  Spin geometry, volume 38 of Princeton Mathematical Series.  Princeton University Press, Princeton, NJ, 1989.
  * [36] E. Leichtnam, R. Mazzeo, and P. Piazza.  The index of Dirac operators on manifolds with fibered boundaries.  Bull. Belg. Math. Soc. Simon Stevin, 13(5):845–855, 2006.
  * [37] M. Lesch.  Operators of Fuchs type, conical singularities, and asymptotic methods, volume 136 of Teubner-Texte sur Math. B.G. Teubner, Stuttgart, Leipzig, 1997.
  * [38] M. T. Lock and J. A. Viaclovsky.  An index theorem for anti-self-dual orbifold-cone metrics.  Adv. Math., 248:698–716, 2013.
  * [39] R. Mazzeo.  Elliptic theory of differential edge operators. I.  Comm. Partial Differential Equations, 16(10):1615–1664, 1991.
  * [40] R. Mazzeo and R. B. Melrose.  The adiabatic limit, Hodge cohomology and Leray’s spectral sequence for a fibration.  J. Differential Geom., 31(1):185–213, 1990.
  * [41] R. Mazzeo and B. Vertman.  Analytic torsion on manifolds with edges.  Adv. Math., 231(2):1000–1040, 2012.
  * [42] R. Mazzeo and B. Vertman.  Elliptic theory of differential edge operators, II: boundary value problems.  2013\.
  * [43] R. Melrose.  Fibrations, compactifications and algebras of pseudodifferential operators.  In L. Hörmander and A. Melin, editors, Partial Differential Equations and Mathematical Physics, volume 21 of Progress in Nonlinear Differential Equations and Their Applications, pages 246–261. Birkhäuser Boston, 1996.
  * [44] R. B. Melrose.  Differential Analysis on Manifolds with Corners.  in preparation.  Available online at http://www-math.mit.edu/~rbm/book.html.
  * [45] R. B. Melrose.  Introduction to Microlocal Analysis.  in preparation.  Available online at http://www-math.mit.edu/~rbm/Lecture_notes.html.
  * [46] R. B. Melrose.  Pseudodifferential operators, corners and singular limits.  In Proceedings of the International Congress of Mathematicians, Vol. I, II (Kyoto, 1990), pages 217–234. Math. Soc. Japan, Tokyo, 1991.
  * [47] R. B. Melrose.  Calculus of conormal distributions on manifolds with corners.  Int Math Res Notices, 1992:51–61, 1992.
  * [48] R. B. Melrose.  The Atiyah-Patodi-Singer index theorem, volume 4 of Research Notes in Mathematics.  A K Peters Ltd., Wellesley, MA, 1993.
  * [49] R. Penfold, J.-M. Vanden-Broeck, and S. Grandison.  Monotonicity of some modified Bessel function products.  Integral Transforms Spec. Funct., 18(1-2):139–144, 2007.
  * [50] J. Roe.  Elliptic operators, topology and asymptotic methods, volume 395 of Pitman Research Notes in Mathematics Series.  Longman, Harlow, second edition, 1998.
  * [51] R. T. Seeley.  Complex powers of an elliptic operator.  In Singular Integrals (Proc. Sympos. Pure Math., Chicago, Ill., 1966), pages 288–307. Amer. Math. Soc., Providence, R.I., 1967.
  * [52] M. E. Taylor.  Partial differential equations I. Basic theory, volume 115 of Applied Mathematical Sciences.  Springer, New York, second edition, 2011.
  * [53] M. E. Taylor.  Partial differential equations II. Qualitative studies of linear equations, volume 116 of Applied Mathematical Sciences.  Springer, New York, second edition, 2011.
  * [54] E. Witten.  Global gravitational anomalies.  Comm. Math. Phys., 100(2):197–229, 1985.