# Context Matters: Data-Efficient Augmentation of Large Language Models for
Scientific Applications

Xiang Li1,2 Haoran Tang2 Siyu Chen2
Ziwei Wang2 Anurag Maravi2 Marcin Abram1,3
1Department of Physics and Astronomy, University of Southern California, Los
Angeles, CA, USA
2Department of Computer Science, University of Southern California, Los
Angeles, CA, USA
3Information Sciences Institute, University of Southern California, Los
Angeles, CA, USA
<EMAIL_ADDRESS>

###### Abstract

In this paper, we explore the challenges inherent to Large Language Models
(LLMs) like GPT-4, particularly their propensity for hallucinations, logic
mistakes, and incorrect conclusions when tasked with answering complex
questions. The capacity of LLMs to present erroneous answers in a coherent and
semantically rigorous manner further complicates the detection of factual
inaccuracies. This issue is especially pronounced in fields that require
specialized expertise. Our work delves into these challenges, aiming to
enhance the understanding and mitigation of such errors, thereby contributing
to the improvement of LLM accuracy and reliability in scientific and other
specialized domains. Our findings reveal a non-linear relationship between the
context’s relevancy and the answers’ measured quality. In addition, we
demonstrate that with the correct calibration, it is possible to automate the
grading procedure – a finding suggesting that, at least to some degree, the
LLMs can be used to self-examine the quality of their own performance.
Finally, we describe an experimental platform that can be seen as a proof-of-
concept of the techniques described in this work.

##  1 Introduction

Large Language Models (LLMs) such as GPT-4 suffer from hallucinations, logic
mistakes, and incorrect conclusions when answering challenging questions
(Rawte et al., 2023). In addition, similar to a style adoption (Gatys et al.,
2016), LLMs can present erroneous answers in a seemingly coherent and
semantically rigorous fashion (Bender et al., 2021; Scheurer et al., 2023).
Therefore, detecting factual mistakes made by LLMs can be challenging,
especially when asking about topics requiring specialized expertise.

We have many available benchmarks measuring the performance of LLMs, including
AGIEval (Zhong et al., 2023), ChemLLMBench (Guo et al., 2023), and SCIEval
(Sun et al., 2023). However, those popular benchmarks usually require a
specific structure of the responses (e.g., multiple choice or numeric) and are
not designed to capture mistakes in free-form responses. Among more flexible
methods, we can mention HaluEval developed by Li et al. (2023b). However, this
method does not scale well since it requires laborious and expensive human
annotation. Automated methods of evaluations include BiLingual Evaluation
Understudy (BLEU) (Papineni et al., 2001), BERTScore (Zhang et al., 2020), and
LLM-Eval (Lin and Chen, 2023). While they can be easily run at scale, they are
usually restricted to a group of specific sub-tasks (e.g., machine
translation, text summarization, or dialogue generation). Due to those
constraints, the question of LLMs evaluation is still an active research area.

In many areas, the tendency of LLMs to hallucinate and its typical
overconfidence when presenting erroneous answers can cause a lot of harm
(e.g., in medical (Thapa and Adhikari, 2023), legal (Weidinger et al., 2022),
or educational (Bommasani et al., 2021) applications). In this work, we focus
on yet another area characterized by a low tolerance for erroneous answers:
_scientific applications_ , with the aim to develop a technique that increases
the coherence and reliability of answers produced by popular LLMs, such as
GPT-3.5 and GPT-4.

We tackle this challenge by employing a version of prompt engineering known as
_the chain-of-thought_ (Wei et al., 2022b). As one of the original
contributions of this paper, we measure the extent to which the relevance of
context matters in improving the quality of answers when asking about topics
requiring specialized expertise. Next, in Section 5, we discuss to which
extent we can automate the evaluation procedure, testing two distinct
approaches: (1) we anchor the ground truth by providing a sample answer, (2)
we anchor on the generated answer based on the sample answer as a fact.

To illustrate our approach, we developed an online platform that can serve as
a proof-of-concept, testing the techniques described in this paper. As a
choice of our scientific domain, we selected graduate-level quantum physics.
In Section 6 we discuss the platform’s architecture and in Section 7, we test
the impact of different summarization techniques for keeping the context
reasonably short.

##  2 Characteristic Challenges and Research Opportunities

Arguably, the fastest adoption of LLMs in the past year could be noticed in
the areas of translation (Xu et al., 2023), code generation (Koziolek and
Koziolek, 2023; Siddiq and Santos, 2023; Chen et al., 2023), database query
optimization (Li et al., 2023a), and text summarization (Sahu et al., 2023) as
well as in the development of general personal assistants (Li et al., 2023c;
Chan et al., 2023).

However, when targeting _scientific applications_ , we must be aware of some
distinctive characteristics of that domain that set it apart from typical
business operations. For example, when answering scientific questions, we
might face some inherent limit on the size (or variability) of the known
datasets. While in many business applications, we can get more trainable
examples by committing more money or time (e.g., by taking more photos and
labeling them or by scanning more books to get more natural text), in some
scientific domains, the number of existing examples can be strictly limited.
For example, we might only know so many examples of superconducting materials,
and we can not easily increase that number (not without first discovering new
types of superconductors). Another important difference is that science does
not follow the typical majority voting rule. If we ask people to classify
pictures of cats and dogs, even if some human labelers might disagree when
assessing a particular photo, we can still use majority voting to determine
whether a given picture depicts a dog or a cat. The same can be done to
quantify the value of continuous variables (Surowiecki, 2004). In contrast, in
science, a single opinion, if sufficiently motivated, can weigh more than a
popular opinion of the majority (as it happened, e.g., with the discovery of
quasicrystals by (Shechtman et al., 1984), a discovery that challenged the
existing status quo, and resulted in a major shift in our understanding of
solid-state physics). On the other hand, in the scientific domain, we can rely
on some known (even if sometimes only in an approximate form111 For example,
Newton’s third law is valid in classical mechanics but invalid in
electrodynamics (Taylor, 2005) – though the deviation from Newton’s third law
is significant only for strong magnetic fields or when relativistic effects
are significant. Another example can be the conservation of mass, which is
valid only for closed systems (Xia, 1992).) fundamental laws governing the
universe (e.g., the laws of motion or the conservation of energy, momentum,
angular momentum, charge, mass, etc.). This contrasts with many business
applications that often target complex social systems, where emerging
phenomena render any fundamental description unusable (Anderson, 1972). For
example, when predicting the probability that a given customer will return in
the next six months, we can not rely on any strict laws we can use to verify
the correctness of our solutions. This makes applications in the scientific
domain somehow special since we can verify with certainty that answers
violating some fundamental laws are wrong.

Considering all those differences, we must leverage the few-shot learning
ability of LLMs (Brown et al., 2020). The algorithm that we propose must be
data-efficient. We can not afford to retrain the entire LLM using millions of
examples, as simply they might not exist. We must leverage the few-shot
learning ability of LLMs. Instead, we focus on providing LLMs with the proper
context, as described in the following sections.

##  3 Related Work

#### Prompt engineering.

Previous work has demonstrated that prompt engineering and chain of thought
techniques can greatly improve LLMs’ performance in various domains (Kojima et
al., 2022; Wei et al., 2022b; Besta et al., 2023; Yao et al., 2023; Imani et
al., 2023). However, very few works focus on the evaluation of these
techniques when applied to the scientific domain.

#### In-Context Learning.

The few-shot learning capabilities of LLMs enable enhanced performance across
various tasks through tailored prompting (Brown et al., 2020). Drori et al.
(2022) employed OpenAI’s Codex to tackle University Math Problems, achieving
human-level performance and elevating benchmark accuracy from $18.8\%$ to
$81.1\%$. Imani et al. (2023) developed a “math-prompter” method for LLMs,
significantly improving performance on the MultiArith dataset from $78.7\%$ to
$92.5\%$. Zhang et al. (2023) developed a concise two-stage framework that
first generates rationales and then uses them to enhance answer inference,
effectively integrating multimodal information. However, today’s state-of-the-
art LLMs (e.g., GPT-4) still struggle to solve advanced, graduate-level
physics problems, as demonstrated in this paper. Key issues include its poor
handling of physical units, coding errors, and its tendency to create
convincing yet incorrect answers.

#### Evaluation of LLMs’ physics reasoning capacities.

Creating an effective evaluation framework for LLMs in scientific domains,
particularly for physical reasoning, is challenging due to the implicit nature
of human knowledge about the physical world and its representation (Bisk et
al., 2020). Benchmarks like those created by Clark et al. (2018) have focused
on assessing LLMs using physics questions. Aroca-Ouellette et al. (2021)
specifically test models on physical commonsense and reasoning concepts. Liang
et al. (2023); Ichter et al. (2022) focus applications for physically grounded
contexts. However, most of those benchmarks target basic school- or
undergraduate-college-level questions. In contrast, in this work, we focus on
the topic of upper-level undergraduate- and graduate-level quantum physics to
provide LLM with more nuanced questions that often do not have a single,
straight answer.

##  4 Methodology

###  4.1 Approach

To assess the importance of context, we compare the answers of GPT-4 in two
settings: (1) without any modifications, a version that can be seen as zero-
shot learning, and (2) using prompt engineering to provide the context. We
vary the quality of the context, from “irrelevant”, through “vague” to
“insightful”. We score the quality of the response using developed grading
rubrics. In addition, we test a few variants of automatic grading, measuring
its consistency with the manual expert grading. Finally, we test the impact of
summarization techniques employed to limit the length of the context.

###  4.2 Data collection

We created a question bank containing selected problems from (Griffiths and
Schroeter, 2018), a popular upper-undergraduate or intro-graduate-level
textbook on quantum mechanics. Next, for each question, we created three hints
with different levels of relevance.

The selected questions cover the following areas: quantum mechanics in three
dimensions, identical particles, symmetries and conservation laws, time-
independent perturbation theory, and the variational principle (cf. the sample
question in the Supplementary Material). The solutions to the questions
usually involve a combination of complex calculations and application of
conceptual knowledge.

###  4.3 Hints structure

Hints for each question are divided into three classes: _irrelevant hint_ ,
_vague hint_ , and _insightful hint_. Insightful hints are hints that contain
information highly relevant to the question. These hints can contain a sample
solution for a similar problem, key insights into the formation of the
question, or methodology that can be applied to the answer. Vague hints are
hints relevant to the topic of the question but not the question itself. For
example, if the question asks about the variational principle, a vague hint
can be the sample solution to a completely different variational principle
question or a textbook description of the variational principle. Unrelated
hints include the solution to a difficult quantum physics problem, which is
not relevant to the topic of the question.

You can find examples of hints representing each category in the Supplementary
Material.

###  4.4 Evaluation

To assess different aspects of answers produced by the LLM, we developed
grading rubrics evaluating not only the semantic accuracy of the responses but
also the factual correctness and their logical consistency. Namely, our
evaluation system comprised:

  * •

Completness and Relevancy (3 points): Three points for a relevant and correct
response. Zero points if the answer is irrelevant or incorrect, with partial
points for partial correctness.

  * •

Logic and Reasoning (4 points): We start with four points and deduct two
points for each logical inconsistency and one point for each minor error in
reasoning.

  * •

Truthfulness (lack of hallucination, 3 points): We start with three points and
deduct one point for each incorrect statement in the response.

We graded each response using both manual and automatic approaches. As for
manual grading (manual), we shuffled the order of responses to reduce bias (so
the grader does not know which response is the result of which). The grader
then used the evaluation criteria and sample answers to grade. For automatic
grading, we have implemented two approaches. The first approach (auto1)
involved providing GPT-4 with an answer and rubric while prompting GPT-4 to
grade in a zero-shot manner. In the second approach (auto2), we first prompted
GPT-4 to generate an answer based on the sample answer and then used its
answer as the key to grade the response. In this way, the grader kept a better
version of its response as the benchmark (assuming the quality of the response
is directly and positively correlated to the relevance and correctness of the
prompt).

Since we have four levels of context (“no context”, “insightful hint”, “vague
hint”, and “irrelevant hint”), a total of four responses were generated by
GPT-4 for each question. You can find a sample of grading with our evaluation
criteria in Supplementary Material.

##  5 Results

###  5.1 Context matters

We provide GPT-4 with different levels of context when answering the queries
from our question bank. The context is provided to GPT-4 API with an
“assistant” role, followed by a “user” call to answer the question. For each
question, we test the response, providing four different levels of context. In
the first level (no hint), we provide no context. In the second level
(irrelevant hint), the context contains a hint that is relevant to quantum
physics but not relevant to the topic of the question. In the third level
(vague hint), the context contains a hint that is relevant to the topic of the
question but not directly applicable or related to the question. In the fourth
level (insightful hint), the context contains a relevant and applicable hint
to the solution.

Table 1:  Mean grading score for different levels of hints and different grading modes (manual and two different auto-grading, as explained in Section 5.2). The error represents the unbiased sample standard deviation of the mean values.  | No hint | Irrelevant | Vague | Insightful
---|---|---|---|---
Manual | $7.5\pm 0.6$ | $7.9\pm 0.6$ | $7.9\pm 0.7$ | ${\bf 8.4\pm 0.6}$
Auto1 | $6.6\pm 0.6$ | $6.8\pm 0.4$ | $6.6\pm 0.7$ | ${\bf 7.4\pm 0.6}$
Auto2 | $6.4\pm 0.6$ | $7.0\pm 0.4$ | $6.6\pm 0.7$ | ${\bf 7.6\pm 0.5}$

After acquiring the responses from GPT-4, we shuffled them and manually graded
them according to the evaluation criteria. We present the mean scores in Table
1 (where we also included results of auto-graders, explained below). The
results suggest that different levels of quality of the context leads to
different performance. Notably, even irrelevant context seems to improve the
average performance.

###  5.2 Automatic grading

Figure 1: Mean grading score for different levels of hints and different
grading modes. The error boxes represent the unbiased sample standard
deviation of the mean values. While there is no statistical difference between
the two tested auto-grade methods, the manual method yields higher scores than
the automated method.

Aside from grading manually, we also experimented with three versions of
automatic grading with GPT-4. The first version (auto-1) grades by treating
the sample answer as the ground truth, and the second version (auto-2) first
generates an answer based on the sample answer provided as a context, before
using that generated answer as the ground truth. We present the averaged
results in Table 1 and Figure 1.

Analyzing the general trend depicted in Figure 1, both auto1 and auto2 grading
methods yield statistically similar results but differ from the manual
grading. This gap between manual and automated scores can be explained by a
lack of calibration of the automatic scoring system. Nevertheless, the fact
that the general slope of each line is similar might suggest that automatic
grading can be a viable alternative to manual grading, allowing us to scale
our methodology in the future.

In Figure 2, we present detailed histograms of scores for each question and
hint type. In Table 2, we depicted the population standard deviation of each
score group. Notably, we obtain the highest standard deviation for the vague-
hint category. This suggests that while low-quality context can improve the
mean performance, it also increases the variability of responses.

Table 2:  The measured sample standard deviation of each score group.  | No_hint | Irrelevant | Vague | Insightful
---|---|---|---|---
Manual | 1.69 | 1.70 | 1.97 | 1.69
Auto1 | 1.69 | 1.17 | 2.11 | 1.91
Auto2 | 1.85 | 1.34 | 2.11 | 1.43
Figure 2: Score Distribution for responses generated with different prompts
and grading techniques

###  5.3 Consensus between manual grading and automatic grading

Consensus across all three grading strategies highlighted two key insights:
firstly, insightful hints notably enhance GPT-4’s performance in terms of mean
scores; secondly, responses generated without hints performed the worst.

These results corroborate our initial hypothesis that highly relevant hints
significantly boost GPT-4’s efficiency in addressing questions of scientific
domains. Interestingly, our results also suggest that “irrelevant hints” boost
the performance of our LLM in a similar way as “vague hints” (cf. again Figure
1). This unexpected outcome could imply a misalignment between human
perception of hint relevance and the model’s interpretation. It raises the
possibility that what seems relevant and helpful to a human may not
necessarily resonate in the same way with GPT-4, underscoring the nuanced
dynamics of human-AI interaction in educational contexts.

##  6 Proof of Concept

###  6.1 QuantumGPT platform

To further improve the performance and versatility of our prompt engineering
approach, we have designed a platform (denoted below as QuantumGPT and
available at www.quantumgpt.science) that can help us gather more data in a
crowd-sourcing manner. The general pipeline of QuantumGPT is shown in Figure
3. It is a prompt-engineering project that utilizes user feedback to improve
the response of GPT-4. In addition, the platform has interactive features,
including chat, discussions, and the ability to post blog-style articles.

Figure 3: The depiction of the QuantumGPT’s pipeline.

###  6.2 The Credibility-Enhanced Feedback Architecture, CEFA

The accuracy and quality of data are critical to the performance. Some
previous works have discussed the threat of data poisoning attacks (Chan et
al., 2020; Fowl et al., 2021). Even without a malicious attack, we are still
vulnerable to unintentional inaccurate information. To address this concern
and improve the data quality collected, we propose a Credibility-Enhanced
Feedback Architecture (CEFA), as outlined in Figure 4. It consists of three
main modules: the distillation algorithm, relevance evaluation algorithm, and
optimization algorithm. There is also a separate credibility system to examine
the interaction between users and evaluate the credibility of each user. The
distillation algorithm distills insights from users’ comments. Insights are
then passed to the relevance evaluation algorithm to generate a relevance
score. Finally, insights with their respective relevance and credibility
scores are passed to the optimization algorithm to engineer the prompt.

Figure 4: The schematics of the CEFA architecture.

#### Distillation Algorithm

The Distillation Algorithm module is developed to optimize the utility of
user-generated content within QuantumGPT, aiming to refine the information
flow by reducing the token count of user comments while preserving their
intrinsic insights. This algorithm is adept at processing raw user comments,
transforming them into a distilled format that retains the core insights but
minimizes unnecessary verbosity and redundancy. Such a transformation is
instrumental in significantly enhancing the data handling efficiency within
the QuantumGPT framework.

The deployment of these summarized insights contributes notably to the overall
performance of QuantumGPT. By reducing the length of the inputs, the algorithm
effectively diminishes the computational burden on the system. This reduction
in computational load directly translates into an enhanced speed of response
generation, thereby elevating the system’s capability to handle complex user
interactions swiftly and accurately.

#### Relevance Evaluation Algorithm

The Relevance Evaluation Algorithm is designed to ensure precise and relevant
dialogue in quantum physics. It utilizes advanced Natural Language Processing
techniques to assess the semantic relevance between user queries and
responses, going beyond traditional keyword matching to include semantic
similarity and contextual embedding models.

The algorithm employs sentence-transformer models to generate embeddings from
user inputs, analyzing semantic relationships through cosine similarity
measures. This process allows for the generation of relevance scores, which
indicate the semantic alignment between questions and comments, ensuring
contextually relevant and precise responses from QuantumGPT.

#### Credibility Algorithm

The Credibility Algorithm is designed to enhance online discourse quality by
assessing user interactions. This algorithm identifies three ways for users to
gain credibility: prior credentials (such as a relevant university
affiliation), experience using the platform, and interactions with other
users.

Figure 5: The depiction of the Credibility System.

The algorithm operates within a dynamic feedback mechanism, as depicted in the
Figure 5. It functions by analyzing the sentiment of a comment from User 1
towards User 2. If User 1, who holds a high credibility standing, expresses a
positive sentiment in their comment, the algorithm activates the Update Score
Calculator. This calculator, in turn, increases User 2’s credibility score,
provided the sentiment analysis results are positive.

For the purpose of the sentiment analysis, we use the “distilbert-base-
uncased-finetuned-sst-2-english” language model. This model, a streamlined
version of the BERT framework, has been fine-tuned on the Stanford Sentiment
Treebank (SST-2) dataset, making it highly effective in analyzing user
comments. The sentiment scores derived from this analysis are used to adjust
the credibility scores of users, with a particular emphasis on positive
feedback from highly credible users. This process ensures that the platform’s
credibility ratings reflect constructive and positive user interactions.

##  7 Context Window and Summarization Methods

Each LLM, including GPT-4, can only accept a finite-size input shorter than a
specific context window. Therefore, in any practical implementation, we must
implement a summarization technique to keep the context concise. In this
analysis, we test the impact of the context summarization on the performance
of the LLM.

An additional advantage of context summarization is that shorter input reduces
the computational load on the system and increases the speed of response
generation. For our preliminary methodology, as our primary summarization
model, we utilize BART, which has been fine-tuned on the CNN/Daily Mail
dataset. This model is not explicitly trained on scientific texts –- a domain
that is relatively underrepresented in text summarization models. Despite this
limitation, the model demonstrates proficiency in rendering concise summaries.

Our experiment entailed utilizing the BART model on three user hints ranked in
relevancy to each question. We measured the outcomes by assessing the extent
of length reduction and the effectiveness in preserving critical concepts. For
the purpose of this assessment, we modified our auto-grading technique by
incorporating the manual chain of thought technique (Wei et al., 2022a)
(auto3). This adaptation involves including our manual grading of a response
as an example within the auto-grader system. This modification aims to align
the results of the auto grader more closely with the outcomes of manual
grading. This approach is expected to enhance the accuracy and reliability of
(auto3) by incorporating human-like reasoning processes into its grading
methodology.

Table 3: Hints length and summarized insights length (in the number of characters). | hint1 | insight1 | hint2 | insight2 | hint3 | insight3
---|---|---|---|---|---|---
1 | 168 | 167 | 412 | 275 | 1284 | 286
2 | 412 | 275 | 168 | 167 | 1284 | 286
3 | 68 | 56 | 80 | 68 | 1284 | 286
4 | 80 | 68 | 68 | 56 | 1284 | 286
5 | 70 | 69 | 1687 | 234 | 1284 | 286
6 | 1669 | 235 | 552 | 174 | 1284 | 286
7 | 1852 | 211 | 927 | 248 | 1284 | 286
8 | 2106 | 203 | 927 | 248 | 1284 | 286
9 | 1849 | 302 | 1288 | 194 | 1284 | 286
10 | 1617 | 302 | 1288 | 194 | 1284 | 286
Figure 6: Auto3 score distributions for responses generated with access to
different level of context, with and without applying the summarization
technique.

In Table 3, we show the degree to which we were able to shorten each hint.
Next, by comparing the correctness of responses prompted with original
comments and summarized insights, we measured the quality and relevance of the
insights. We show the detailed scores for each type of summarized group of
hints in Figure 6.

Table 4:  Mean Score for Different Summarized Hints and Grading  | Irrelevant | Vague | Insightful
---|---|---|---
Auto2 | 7.0 | 6.6 | 7.6
Auto3+summaries | 7.0 | 6.8 | 6.3

In Table 4, we compare the average score on the unabridged and summarized
context. We see no reduction in the performance in the cases of the
“irrelevant” and “vague” hints. This might imply that despite the length
reduction, the integrity and informative value of the original content were
preserved in the summaries. However, in the case of the “insightful” hints,
the reduction in the performance is noticeable. This calls for further
studies. One way of obtaining better results could be fine-tuning the BART
model for scientific domain data – such intervention should increase the
quality of the summaries and might reduce the problem.

##  8 Discussion

###  8.1 Significance of Context in LLM Performance

Our findings underscore the critical role of context in enhancing the
performance of Large Language Models (LLMs) like GPT-4, particularly in the
specialized domain of quantum physics. The study reveals that providing
contextually relevant hints can significantly improve the accuracy and
robustness of LLM responses. However, our result also shows that hints
relevant to the topic do not provide any more performance boost when compared
to irrelevant hints. One hypothesis is that the determining factor to the
helpfulness of the hint is deep insights into the methodology relevant to the
question, and insights into the methodology can exist in hints that are not
relevant to the topic. Currently, we are working on expanding our dataset to
increase the statistical significance of our results.

###  8.2 Promise of Automatic Grading

The exploration of automatic grading methods presents great opportunities. Our
experiments with auto1 and auto2 grading methods align decently well with
manual grading outcomes (at least, in terms of the perceived sensitivity of
this method). This suggests the potential of these methods as cost-effective
and scalable alternatives to manual grading. However, the observed
discrepancies, particularly in assessing unrelated and vague hints, highlight
the need for further refinement and understanding of how LLMs interpret and
respond to different types of context.

###  8.3 Challenges and limitations

Although our preliminary results are interesting, we are aware of the small
size of our data set (10 entries and 40 responses in total). It should be
pointed out that creating hints and conducting manual evaluation is a
difficult task that requires domain knowledge. We are currently working on
expanding the dataset and collecting questions and feedback regarding our
experimental platform, QuantumGPT.

Additionally, while using BART, fine-tuned on the CNN/Daily Mail dataset, is
effective for condensing hints into insightful summaries, we have observed a
notable loss of critical information, including equations and scientific
notations. Given the scarcity of models trained specifically on scientific
texts, there is a compelling case for developing a specialized model. Such a
model should be adept at summarizing scientific content, striking a balance
between brevity and the retention of rich contextual information. This
approach would address the gap in processing and summarizing complex
scientific sentences.

###  8.4 Future Work

We have identified three pivotal areas for future research. Firstly, the
investigation of insight distillation methods. This involves refining
algorithms to more effectively condense complex scientific information into
concise yet comprehensive insights, enhancing the efficiency and accuracy of
LLM responses. Secondly, the further development of automatic grading systems.
Lastly, an experiment with the CEFA architecture to test the users’
incentives. Another area of interest is the system’s robustness, e.g., its
sensitivity to noise or inaccurate information.

## References

  * Anderson [1972] P. W. Anderson.  More Is Different: Broken symmetry and the nature of the hierarchical structure of science.  _Science_ , 177(4047):393–396, August 1972.  ISSN 1095-9203.  doi: 10.1126/science.177.4047.393.  URL http://dx.doi.org/10.1126/science.177.4047.393.
  * Aroca-Ouellette et al. [2021] Stephane Aroca-Ouellette, Cory Paik, Alessandro Roncone, and Katharina Kann.  PROST: Physical Reasoning about Objects through Space and Time.  In Chengqing Zong, Fei Xia, Wenjie Li, and Roberto Navigli, editors, _Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021_ , volume ACL/IJCNLP 2021 of _Findings of ACL_ , pages 4597–4608. Association for Computational Linguistics, 2021.  doi: 10.18653/V1/2021.FINDINGS-ACL.404.  URL https://doi.org/10.18653/v1/2021.findings-acl.404.
  * Bender et al. [2021] Emily M. Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell.  On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?  In _Proceedings of the 2021 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’21. ACM, March 2021.  doi: 10.1145/3442188.3445922.  URL http://dx.doi.org/10.1145/3442188.3445922.
  * Besta et al. [2023] Maciej Besta, Nils Blach, Ales Kubicek, Robert Gerstenberger, Lukas Gianinazzi, Joanna Gajda, Tomasz Lehmann, Michal Podstawski, Hubert Niewiadomski, Piotr Nyczyk, and Torsten Hoefler.  Graph of Thoughts: Solving Elaborate Problems with Large Language Models, 2023.  URL https://arxiv.org/abs/2308.09687.
  * Bisk et al. [2020] Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi.  PIQA: Reasoning about Physical Commonsense in Natural Language.  In _The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, New York, NY, USA, February 7-12, 2020_ , pages 7432–7439. AAAI Press, 2020.  doi: 10.1609/AAAI.V34I05.6239.  URL https://doi.org/10.1609/aaai.v34i05.6239.
  * Bommasani et al. [2021] Rishi Bommasani, Drew A. Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S. Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, Erik Brynjolfsson, Shyamal Buch, Dallas Card, Rodrigo Castellon, Niladri Chatterji, Annie Chen, Kathleen Creel, Jared Quincy Davis, Dora Demszky, Chris Donahue, Moussa Doumbouya, Esin Durmus, Stefano Ermon, John Etchemendy, Kawin Ethayarajh, Li Fei-Fei, Chelsea Finn, Trevor Gale, Lauren Gillespie, Karan Goel, Noah Goodman, Shelby Grossman, Neel Guha, Tatsunori Hashimoto, Peter Henderson, John Hewitt, Daniel E. Ho, Jenny Hong, Kyle Hsu, Jing Huang, Thomas Icard, Saahil Jain, Dan Jurafsky, Pratyusha Kalluri, Siddharth Karamcheti, Geoff Keeling, Fereshte Khani, Omar Khattab, Pang Wei Koh, Mark Krass, Ranjay Krishna, Rohith Kuditipudi, Ananya Kumar, Faisal Ladhak, Mina Lee, Tony Lee, Jure Leskovec, Isabelle Levent, Xiang Lisa Li, Xuechen Li, Tengyu Ma, Ali Malik, Christopher D. Manning, Suvir Mirchandani, Eric Mitchell, Zanele Munyikwa, Suraj Nair, Avanika Narayan, Deepak Narayanan, Ben Newman, Allen Nie, Juan Carlos Niebles, Hamed Nilforoshan, Julian Nyarko, Giray Ogut, Laurel Orr, Isabel Papadimitriou, Joon Sung Park, Chris Piech, Eva Portelance, Christopher Potts, Aditi Raghunathan, Rob Reich, Hongyu Ren, Frieda Rong, Yusuf Roohani, Camilo Ruiz, Jack Ryan, Christopher Ré, Dorsa Sadigh, Shiori Sagawa, Keshav Santhanam, Andy Shih, Krishnan Srinivasan, Alex Tamkin, Rohan Taori, Armin W. Thomas, Florian Tramèr, Rose E. Wang, William Wang, Bohan Wu, Jiajun Wu, Yuhuai Wu, Sang Michael Xie, Michihiro Yasunaga, Jiaxuan You, Matei Zaharia, Michael Zhang, Tianyi Zhang, Xikun Zhang, Yuhui Zhang, Lucia Zheng, Kaitlyn Zhou, and Percy Liang.  On the Opportunities and Risks of Foundation Models, 2021.  URL https://arxiv.org/abs/2108.07258.
  * Brown et al. [2020] Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei.  Language Models are Few-Shot Learners.  In Hugo Larochelle, Marc’Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin, editors, _Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual_ , 2020.  URL https://proceedings.neurips.cc/paper/2020/hash/1457c0d6bfcb4967418bfb8ac142f64a-Abstract.html.
  * Chan et al. [2020] Alvin Chan, Yi Tay, Yew-Soon Ong, and Aston Zhang.  Poison Attacks against Text Datasets with Conditional Adversarially Regularized Autoencoder.  In Trevor Cohn, Yulan He, and Yang Liu, editors, _Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020_ , volume EMNLP 2020 of _Findings of ACL_ , pages 4175–4189. Association for Computational Linguistics, 2020.  doi: 10.18653/V1/2020.FINDINGS-EMNLP.373.  URL https://doi.org/10.18653/v1/2020.findings-emnlp.373.
  * Chan et al. [2023] Szeyi Chan, Jiachen Li, Bingsheng Yao, Amama Mahmood, Chien-Ming Huang, Holly Jimison, Elizabeth D Mynatt, and Dakuo Wang.  “Mango Mango, How to Let The Lettuce Dry Without A Spinner?”: Exploring User Perceptions of Using An LLM-Based Conversational Assistant Toward Cooking Partner, 2023.  URL https://arxiv.org/abs/2310.05853.
  * Chen et al. [2023] Meng Chen, Hongyu Zhang, Chengcheng Wan, Zhao Wei, Yong Xu, Juhong Wang, and Xiaodong Gu.  On the Effectiveness of Large Language Models in Domain-Specific Code Generation, 2023.  URL https://arxiv.org/abs/2312.01639.
  * Clark et al. [2018] Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord.  Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challenge, 2018.  URL https://arxiv.org/abs/1803.05457.
  * Drori et al. [2022] Iddo Drori, Sarah Zhang, Reece Shuttleworth, Leonard Tang, Albert Lu, Elizabeth Ke, Kevin Liu, Linda Chen, Sunny Tran, Newman Cheng, Roman Wang, Nikhil Singh, Taylor L. Patti, Jayson Lynch, Avi Shporer, Nakul Verma, Eugene Wu, and Gilbert Strang.  A neural network solves, explains, and generates university math problems by program synthesis and few-shot learning at human level.  _Proceedings of the National Academy of Sciences_ , 119(32), August 2022.  ISSN 1091-6490.  doi: 10.1073/pnas.2123433119.  URL http://dx.doi.org/10.1073/pnas.2123433119.
  * Fowl et al. [2021] Liam Fowl, Micah Goldblum, Ping-yeh Chiang, Jonas Geiping, Wojciech Czaja, and Tom Goldstein.  Adversarial Examples Make Strong Poisons.  In Marc’Aurelio Ranzato, Alina Beygelzimer, Yann N. Dauphin, Percy Liang, and Jennifer Wortman Vaughan, editors, _Advances in Neural Information Processing Systems 34: Annual Conference on Neural Information Processing Systems 2021, NeurIPS 2021, December 6-14, 2021, virtual_ , pages 30339–30351, 2021.  URL https://proceedings.neurips.cc/paper/2021/hash/fe87435d12ef7642af67d9bc82a8b3cd-Abstract.html.
  * Gatys et al. [2016] Leon A. Gatys, Alexander S. Ecker, and Matthias Bethge.  Image Style Transfer Using Convolutional Neural Networks.  In _2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_. IEEE, June 2016.  doi: 10.1109/cvpr.2016.265.  URL http://dx.doi.org/10.1109/cvpr.2016.265.
  * Griffiths and Schroeter [2018] David J. Griffiths and Darrell F. Schroeter.  _Introduction to Quantum Mechanics_.  Cambridge University Press, 2018.
  * Guo et al. [2023] Taicheng Guo, Kehan Guo, Bozhao Nan, Zhenwen Liang, Zhichun Guo, Nitesh V. Chawla, Olaf Wiest, and Xiangliang Zhang.  What Can Large Language Models Do in Chemistry? A Comprehensive Benchmark on Eight Tasks, 2023.  URL https://arxiv.org/abs/2305.18365.
  * Ichter et al. [2022] Brian Ichter, Anthony Brohan, Yevgen Chebotar, Chelsea Finn, Karol Hausman, Alexander Herzog, Daniel Ho, Julian Ibarz, Alex Irpan, Eric Jang, Ryan Julian, Dmitry Kalashnikov, Sergey Levine, Yao Lu, Carolina Parada, Kanishka Rao, Pierre Sermanet, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Mengyuan Yan, Noah Brown, Michael Ahn, Omar Cortes, Nicolas Sievers, Clayton Tan, Sichun Xu, Diego Reyes, Jarek Rettinghouse, Jornell Quiambao, Peter Pastor, Linda Luu, Kuang-Huei Lee, Yuheng Kuang, Sally Jesmonth, Nikhil J. Joshi, Kyle Jeffrey, Rosario Jauregui Ruano, Jasmine Hsu, Keerthana Gopalakrishnan, Byron David, Andy Zeng, and Chuyuan Kelly Fu.  Do As I Can, Not As I Say: Grounding Language in Robotic Affordances.  In Karen Liu, Dana Kulic, and Jeffrey Ichnowski, editors, _Conference on Robot Learning, CoRL 2022, 14-18 December 2022, Auckland, New Zealand_ , volume 205 of _Proceedings of Machine Learning Research_ , pages 287–318. PMLR, 2022.  URL https://proceedings.mlr.press/v205/ichter23a.html.
  * Imani et al. [2023] Shima Imani, Liang Du, and Harsh Shrivastava.  MathPrompter: Mathematical Reasoning using Large Language Models.  In Sunayana Sitaram, Beata Beigman Klebanov, and Jason D. Williams, editors, _Proceedings of the The 61st Annual Meeting of the Association for Computational Linguistics: Industry Track, ACL 2023, Toronto, Canada, July 9-14, 2023_ , pages 37–42. Association for Computational Linguistics, 2023\.  doi: 10.18653/V1/2023.ACL-INDUSTRY.4.  URL https://doi.org/10.18653/v1/2023.acl-industry.4.
  * Kojima et al. [2022] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large Language Models are Zero-Shot Reasoners.  In _NeurIPS_ , 2022.  URL http://papers.nips.cc/paper_files/paper/2022/hash/8bb0d291acd4acf06ef112099c16f326-Abstract-Conference.html.
  * Koziolek and Koziolek [2023] Heiko Koziolek and Anne Koziolek.  LLM-based Control Code Generation using Image Recognition, 2023.  URL https://arxiv.org/abs/2311.10401.
  * Li et al. [2023a] Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li.  Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs, 2023a.  URL https://arxiv.org/abs/2305.03111.
  * Li et al. [2023b] Junyi Li, Xiaoxue Cheng, Wayne Xin Zhao, Jian-Yun Nie, and Ji-Rong Wen.  HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Models, 2023b.  URL https://arxiv.org/abs/2305.11747.
  * Li et al. [2023c] Yuliang Li, Nitin Kamra, Ruta Desai, and Alon Halevy.  Human-Centered Planning, 2023c.  URL https://arxiv.org/abs/2311.04403.
  * Liang et al. [2023] Jacky Liang, Wenlong Huang, Fei Xia, Peng Xu, Karol Hausman, Brian Ichter, Pete Florence, and Andy Zeng.  Code as Policies: Language Model Programs for Embodied Control.  In _IEEE International Conference on Robotics and Automation, ICRA 2023, London, UK, May 29 - June 2, 2023_ , pages 9493–9500. IEEE, 2023\.  doi: 10.1109/ICRA48891.2023.10160591.  URL https://doi.org/10.1109/ICRA48891.2023.10160591.
  * Lin and Chen [2023] Yen-Ting Lin and Yun-Nung Chen.  LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Models, 2023.  URL https://arxiv.org/abs/2305.13711.
  * Papineni et al. [2001] Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu.  BLEU: A Method for Automatic Evaluation of Machine Translation.  In _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics – ACL ’02_ , ACL ’02. Association for Computational Linguistics, 2001.  doi: 10.3115/1073083.1073135.  URL http://dx.doi.org/10.3115/1073083.1073135.
  * Rawte et al. [2023] Vipula Rawte, Swagata Chakraborty, Agnibh Pathak, Anubhav Sarkar, S. M Towhidul Islam Tonmoy, Aman Chadha, Amit P. Sheth, and Amitava Das.  The Troubling Emergence of Hallucination in Large Language Models – An Extensive Definition, Quantification, and Prescriptive Remediations, 2023\.  URL https://arxiv.org/abs/2310.04988.
  * Sahu et al. [2023] Gaurav Sahu, Olga Vechtomova, and Issam H. Laradji.  Enchancing Semi-Supervised Learning for Extractive Summarization with an LLM-based pseudolabeler, 2023.  URL https://arxiv.org/abs/2311.09559.
  * Scheurer et al. [2023] Jérémy Scheurer, Mikita Balesni, and Marius Hobbhahn.  Technical Report: Large Language Models can Strategically Deceive their Users when Put Under Pressure, 2023.  URL https://arxiv.org/abs/2311.07590.
  * Shechtman et al. [1984] D. Shechtman, I. Blech, D. Gratias, and J. W. Cahn.  Metallic Phase with Long-Range Orientational Order and No Translational Symmetry.  _Phys. Rev. Lett._ , 53:1951–1953, Nov 1984.  doi: 10.1103/PhysRevLett.53.1951.  URL https://link.aps.org/doi/10.1103/PhysRevLett.53.1951.
  * Siddiq and Santos [2023] Mohammed Latif Siddiq and Joanna C. S. Santos.  Generate and Pray: Using SALLMS to Evaluate the Security of LLM Generated Code.  _CoRR_ , abs/2311.00889, 2023.  doi: 10.48550/ARXIV.2311.00889.  URL https://doi.org/10.48550/arXiv.2311.00889.
  * Sun et al. [2023] Liangtai Sun, Yang Han, Zihan Zhao, Da Ma, Zhennan Shen, Baocai Chen, Lu Chen, and Kai Yu.  SciEval: A Multi-Level Large Language Model Evaluation Benchmark for Scientific Research, 2023.  URL https://arxiv.org/abs/2308.13149.
  * Surowiecki [2004] James Surowiecki.  _The wisdom of crowds_.  Doubleday Books, June 2004.  ISBN 0385721706.
  * Taylor [2005] John Taylor.  _Classical Mechanics_ , chapter 1, pages 3–41.  University Science Books, Mill Valley, California, 2005.  ISBN 9781891389221.
  * Thapa and Adhikari [2023] Surendrabikram Thapa and Surabhi Adhikari.  ChatGPT, Bard, and Large Language Models for Biomedical Research: Opportunities and Pitfalls.  _Annals of Biomedical Engineering_ , 51(12):2647–2651, June 2023.  ISSN 1573-9686.  doi: 10.1007/s10439-023-03284-0.  URL http://dx.doi.org/10.1007/s10439-023-03284-0.
  * Wei et al. [2022a] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou.  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.  In _NeurIPS_ , 2022a.  URL http://papers.nips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html.
  * Wei et al. [2022b] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed H. Chi, Quoc V. Le, and Denny Zhou.  Chain-of-Thought Prompting Elicits Reasoning in Large Language Models.  In _NeurIPS_ , 2022b.  URL http://papers.nips.cc/paper_files/paper/2022/hash/9d5609613524ecf4f15af0f7b31abca4-Abstract-Conference.html.
  * Weidinger et al. [2022] Laura Weidinger, Jonathan Uesato, Maribeth Rauh, Conor Griffin, Po-Sen Huang, John Mellor, Amelia Glaese, Myra Cheng, Borja Balle, Atoosa Kasirzadeh, Courtney Biles, Sasha Brown, Zac Kenton, Will Hawkins, Tom Stepleton, Abeba Birhane, Lisa Anne Hendricks, Laura Rimell, William Isaac, Julia Haas, Sean Legassick, Geoffrey Irving, and Iason Gabriel.  Taxonomy of Risks posed by Language Models.  In _2022 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’22. ACM, June 2022.  doi: 10.1145/3531146.3533088.  URL http://dx.doi.org/10.1145/3531146.3533088.
  * Xia [1992] Zhihong Xia.  The Existence of Noncollision Singularities in Newtonian Systems.  _The Annals of Mathematics_ , 135(3):411, May 1992\.  ISSN 0003-486X.  doi: 10.2307/2946572.  URL http://dx.doi.org/10.2307/2946572.
  * Xu et al. [2023] Haoran Xu, Young Jin Kim, Amr Sharaf, and Hany Hassan Awadalla.  A Paradigm Shift in Machine Translation: Boosting Translation Performance of Large Language Models, 2023.  URL https://arxiv.org/abs/2309.11674.
  * Yao et al. [2023] Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik Narasimhan.  Tree of Thoughts: Deliberate Problem Solving with Large Language Models, 2023.  URL https://arxiv.org/abs/2305.10601.
  * Zhang et al. [2020] Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q. Weinberger, and Yoav Artzi.  BERTScore: Evaluating Text Generation with BERT.  In _8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020_. OpenReview.net, 2020.  URL https://openreview.net/forum?id=SkeHuCVFDr.
  * Zhang et al. [2023] Yuanhan Zhang, Kaiyang Zhou, and Ziwei Liu.  What Makes Good Examples for Visual In-Context Learning?, 2023.  URL https://arxiv.org/abs/2301.13670.
  * Zhong et al. [2023] Wanjun Zhong, Ruixiang Cui, Yiduo Guo, Yaobo Liang, Shuai Lu, Yanlin Wang, Amin Saied, Weizhu Chen, and Nan Duan.  AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models, 2023.  URL https://arxiv.org/abs/2304.06364.

## Data and Code Availability

Data and code can be found in the following GitHub repository:
https://github.com/mikelixiang88/Automatic_gpt_grader.git

## Acknowledgements

We would like to take this opportunity to thank Dr. Stephan Haas for providing
critical suggestions in designing the research. We would also like to thank
Dr. Daniel Lidar for providing challenging questions and feedback on GPT4’s
performance in the realm of quantum computing. We would like to thank the
users of QuantumGPT for providing feedback and data.

## Author Contributions

X.L. constructed the quantum question bank, developed the evaluation and auto-
grading part of the paper, and proposed the CEFA architecture. H.T. developed
the context summarization methods part and part of the auto-grading module,
S.C. and A.M. implemented the Relevance Evaluation algorithm, A.M. worked on
the optimization algorithm, Z.W. worked on the credibility system and
implemented the sentiment analysis module, M.A. provided supervision, proposed
the experiment measuring impact of the context, and contributed to the
evaluation and the auto-grading part of the paper. All the authors contributed
to writing the original draft of this article.

## Competing Interests

The authors declare no competing interests.

## Supplementary Material

##  Appendix A Sample question

"Question": "At time t = 0 a large ensemble of spin-1/2 particles is prepared,
all of them in the spin-up state (with respect to the z-axis). They are not
subject to any forces or torques. At time t1 > 0 each spin is measured–some
along the z direction and others along the x direction (but we aren’t told the
results). At time t2 > t1 their spin is measured again, this time along the x
direction, and those with spin up (along x) are saved as a subensemble (those
with spin down are discarded). Question: Of those remaining (the subensemble),
what fraction had spin up (along z or x, depending on which was measured) in
the first measurement?"

##  Appendix B Hints example to the sample question

The equations in the hints are fed as plain latex language. Here they are
presented as readable format.

#### Insightful_hint:

If $S_{z}$ was measured at $t_{1}$, everything was spin up; if $S_{x}$ was
measured, half of them were spin down, but the second measurement did nothing,
and these were thrown away.

#### Vague_hint:

Show that the probability of getting spin up (along $b$) in the first
measurement is $P_{+}=\cos^{2}(\theta_{ab}/2)$, and (by extension) the
probability of getting spin up in both measurements is
$P_{++}=\cos^{2}(\theta_{ab}/2)\cos^{2}(\theta_{bc}/2)$. Find the other three
probabilities ($P_{+-}$, $P_{-+}$, and $P_{--}$). Beware: If the outcome of
the first measurement was spin down, the relevant angle is now the supplement
of $\theta_{bc}$.

#### Unrelated_hint:

Example 10.5
Yukawa scattering. The Yukawa potential (which is a crude model for the
binding force in an atomic nucleus) has the form

$V(r)=\beta\frac{e^{-\mu r}}{r}$

where $\beta$ and $\mu$ are constants. The Born approximation gives

$f(\theta)\approx-\frac{2m\beta}{\hbar^{2}\kappa}\int_{0}^{\infty}e^{-\mu
r}\sin(\kappa r)\,dr=-\frac{2m\beta}{\hbar^{2}(\mu^{2}+\kappa^{2})}.$

Example 10.6
Rutherford scattering. If we put in $\beta=q_{1}q_{2}/4\pi\epsilon_{0},\mu=0$,
the Yukawa potential reduces to the Coulomb potential, describing the
electrical interaction of two-point charges. Evidently, the scattering
amplitude is

$f(\theta)\approx-\frac{2mq_{1}q_{2}}{4\pi\epsilon_{0}\hbar^{2}\kappa^{2}}$

or (using Equations 10.89 and 10.51):

$f(\theta)\approx-\frac{q_{1}q_{2}}{16\pi\epsilon_{0}E\sin^{2}(\theta/2)}.$

The differential cross-section is the square of this:

$\frac{d\sigma}{d\Omega}=\left[\frac{q_{1}q_{2}}{16\pi\epsilon_{0}E\sin^{2}(\theta/2)}\right]^{2},$

which is precisely the Rutherford formula (Equation 10.11). It happens that
for the Coulomb potential classical mechanics, the Born approximation, and
quantum field theory all yield the same result. As they say in the computer
business, the Rutherford formula is amazingly "robust."

##  Appendix C Summarization example

#### Summarized_Insightful_hint:

If $S_{z}$ was measured at $t_{1}$, everything was spin up. If $S_{x}$ was
measured, half of them were spin down. But the second measurement did nothing,
and these were thrown away.

#### Summarized_Vague_hint:

Show that the probability of getting spin up (along $b$) in the first
measurement is $P_{+}=\cos^{2}(\theta_{ab}/2)$, and (by extension) the
probabilities of getting spin up in both measurements is
$P_{++}=\cos^{2}(\theta_{ab}/2)\cos^{2}(\theta_{bc}/2)$. Find the other three
probabilities ($P_{+-}$, $P_{-+}$, and $P_{--}$).

#### Summarized_Unrelated_hint:

The Yukawa potential (which is a crude model for the binding force in an
atomic nucleus) has the form. The Rutherford formula is amazingly "robust." It
happens that for the Coulomb potential, classical mechanics, the Born
approximation, and quantum field theory all yield the same result.

##  Appendix D Generated Answer

The equations in the responses are in plain latex language. Here they are
presented in readable format.

#### No_hint:

All the spins are prepared in the up state along the z axis at $t=0$. This
means that initially the state of any spin is $|up\\_z\rangle$. The
measurement at $t_{1}$ doesn’t change the state, it simply gives us
information about the state; however, any given spin is now in a superposition
of spin up and spin down along the x direction. The key result you need here
is that the $|up\\_z\rangle$ state is a superposition of $|up\\_x\rangle$ and
$|down\\_x\rangle$ states. Specifically,

$|up\\_z\rangle=\frac{1}{\sqrt{2}}(|up\\_x\rangle+|down\\_x\rangle)$

Furthermore, the square of the amplitude gives the probability. Thus, prior to
the first measurement, each spin has a $(1/\sqrt{2})^{2}=1/2$ chance of being
measured as $|up\\_x\rangle$, and the same chance of being measured as
$|down\\_x\rangle$. When the second measurement is made at time $t_{2}$, only
the spins that give $|up\\_x\rangle$ are kept. Since the states
$|up\\_x\rangle$ and $|down\\_x\rangle$ were equally likely at $t_{1}$, this
discards half the spins. Thus, the fraction of spins remaining that were spin
up at $t_{1}$ (either along z or x, whichever was measured) is 1/2. However,
keep in mind that the spins measured along the z direction at $t_{1}$ are
still in the state $|up\\_z\rangle$, regardless of the later measurement. So,
for those spins that were measured along the z direction at $t_{1}$, all of
them were spin up. This is independent of the later measurement along the x
axis at $t_{2}$. In summary, the final answer will depend on how many spins
were measured along x or z at $t_{1}$. But for any given spin, the fraction
that were spin up at $t_{1}$ is 1 if measured along z, and 1/2 if measured
along x.

#### Insightful_hint:

Let’s divide the ensemble into two parts based on the measurements at time
$t_{1}$: 1\. Those where the spin was measured along the $z$ direction. 2\.
Those where the spin was measured along the $x$ direction. For particles in
the first category, since all particles are initially in spin up state with
respect to the $z$ axis, every spin measured will turn out to be up.
Therefore, all particles in this category would have been spin up in the first
measurement at $t_{1}$. For the second category, where the spin is measured
along the $x$ axis at $t_{1}$, the outcomes will be in a superposition of spin
up and spin down along the x axis, due to the angle between the $z$ and $x$
direction. When measurement along $x$ is made, it will have equal
probabilities being found in the plus or minus state, due to the orthogonality
of the two axes. Therefore, the fraction of particles that would have been
measured as spin up during the first measurement for the particles remaining
in the subensemble would be: \- 100- 50Without the proportions of the
particles that were measured along $x$ and $z$, it’s not possible to provide a
concrete combined percentage. But for measurements along $z$ it would always
be 100

#### Vague_hint:

The problem involves sequential measurements of spin 1/2 particles, where the
measurements are performed along different axes. Consider the measurement of
these particles along the x-axis at time $t_{1}$ (these constitute one part of
the original ensemble) and the z-axis at $t_{2}$. The basis states with regard
to the z-axis are $|+z\rangle$ and $|-z\rangle$ which, when expressed in the
x-basis, would be represented as follows: 1\. For spins up along the z-axis:
$|+z\rangle=\frac{1}{\sqrt{2}}(|+x\rangle+|-x\rangle)$ 2\. For spins down
along the z-axis: $|-z\rangle=\frac{1}{\sqrt{2}}(|+x\rangle-|-x\rangle)$ If we
only save the post-measurement states which are spin-up along x, then half of
the spins that were measured along x at $t_{1}$ would be thrown away.
Therefore, it is given that states of particles can either be spin-up along z
or spin-up along x (depending on which was measured) in the first measurement.
From here, considering the populations of these particles is consistent with
both measurements, we find that: 1\. For the particles that were measured
along the z-axis at $t_{1}$, the fraction that remains in the saved
subensemble is $P_{+z}=1$. 2\. For the particles that were measured along the
x-axis, half of the spins that were up along x will remain in the saved
subensemble. Thus, the fraction that remains is $P_{+x}=0.5$ It is important
to highlight that these fractions apply under the assumption of ideal
measurements. In practice, the actual probabilities could be influenced by
experimental uncertainties and possible interactions between particles.

Unrelated_hint: We don’t have concrete numbers for the exact measurements, but
let’s denote the fractions in general. Let’s consider the part of the ensemble
on which spin along $z$ was measured first. The spin was initially prepared in
the spin-up state along $z$, so all these particles had spin up along $z$ at
the first measurement. After the second measurement along $x$, about 50% spins
will be found up along $x$ (because spins along $z$ and $x$ are not
correlated), so this sub-ensemble contributes 50% with spin up in the first
measurement. Now consider the part of the ensemble on which spin along $x$ was
measured first. Again, about 50% of the spins will be found up along $x$ in
the first measurement, and the projection postulate tells us that these spins
are all in the spin-up state along $x$ afterwards, assuming ideal
measurements. When measured again along $x$, these spins will all still be
found up. So this second sub-ensemble also contributes 50% with spin up in the
first measurement. In total, considering these two sub-ensembles combined, out
of all particles that are found to have spin up in the second measurement
about 50% (on average) also had spin up in the first measurement (either along
$z$ or along $x$, depending on the measurement). Remember that these arguments
only make sense for large ensembles, as the results for individual particles
are stochastic.

##  Appendix E Model Answer

"Answer": "All of them"

##  Appendix F Autograde Response

"Manual": 6

"Auto-1": 7

"Auto-2": 7