#  Evaluating GPT-3.5 and GPT-4 Models on Brazilian University Admission Exams

Desnes Nunes*1, Ricardo Primi*2, Ramon Pires*3, Roberto Lotufo4,5, and Rodrigo
Nogueira3,4,5
1University of São Paulo (USP), 2University of San Francisco (USF)
3Maritaca AI, 4NeuralMind, 5State University of Campinas (UNICAMP)
Brazil
*Equal contribution

###### Abstract

The present study aims to explore the capabilities of Language Models (LMs) in
tackling high-stakes multiple-choice tests, represented here by the Exame
Nacional do Ensino Médio (ENEM), a multidisciplinary entrance examination
widely adopted by Brazilian universities. This exam poses challenging tasks
for LMs, since its questions may span into multiple fields of knowledge,
requiring understanding of information from diverse domains. For instance, a
question may require comprehension of both statistics and biology to be
solved. This work analyzed responses generated by GPT-3.5 and GPT-4 models for
questions presented in the 2009-2017 exams, as well as for questions of the
2022 exam, which were made public after the training of the models was
completed. Furthermore, different prompt strategies were tested, including the
use of Chain-of-Thought (CoT) prompts to generate explanations for answers. On
the 2022 edition, the best-performing model, GPT-4 with CoT, achieved an
accuracy of 87%, largely surpassing GPT-3.5 by 11 points. The code and data
used on experiments are available at https://github.com/piresramon/gpt-4-enem

##  I Introduction

Over the past decade, advancements in natural language processing (NLP) and
machine learning (ML) have led to the development of increasingly
sophisticated Language Models (LMs). These models have demonstrated remarkable
performance on a wide array of tasks, such as translation, summarization,
question-answering, among many others [1, 2, 3, 4, 5, 6]. As a consequence,
the education sector has begun to explore the potential benefits of these
technologies, with several studies investigating their potential in classrooms
[7] and the concept of “precision education” [8].

However, most existing research focuses on evaluating LMs in the context of
English language tasks, with some notable exceptions [9, 10, 11, 12, 13, 14,
15]. In particular, there is a lack of studies examining performance of LMs on
Portuguese tasks.

In this work, we evaluate state-of-the-art LMs on Exame Nacional do Ensino
Médio (ENEM), a multidisciplinary admission test widely used in Brazilian
universities. The ENEM exam, presented in Brazilian Portuguese, poses a unique
challenge for LMs, as it requires a deep understanding of various fields of
knowledge and the ability to integrate information from diverse domains. We
analyze responses generated by the latest GPT-3.5 and GPT-4 models to
questions of the ENEM exams using different prompt strategies, including zero-
shot and few-shot prompts with Chain-of-Thought (CoT) explanations. Moreover,
to tackle the potential issue of the model memorizing the answers to the
questions during training, an evaluation was performed on the 2022 exam, which
was made public subsequent to the completion of their training.

By assessing the performance of the state-of-the-art GPT models in this non-
English, high-stakes exam context, this study contributes to the growing body
of research exploring the applicability of LMs in educational settings.
Furthermore, the findings of this study have the potential to inform the
development of future LMs that are better equipped to handle diverse,
multidisciplinary tasks in various languages, ultimately promoting a more
inclusive and accessible landscape for AI-driven educational tools.

##  II Related Work

The goal of this research is to promote advances and evaluate the performance
of automatic question resolution of college entrance exams utilizing recent
state-of-the-art LMs. The multiple-choice questions presented in the ENEM
examination pose a complex challenge, requiring advanced NLP techniques to be
solved. Similar to the Scholastic Assessment Test (SAT), ENEM is an exam that
assesses the knowledge of students who intend to enter universities in
Brazil.111The exam is majorly divided into four areas: languages, codes and
their technologies; human sciences and their technologies; natural sciences
and their technologies; mathematics and its technologies. Moreover, the exam
is composed of 180 multiple-choice questions and an essay. A prior study of AI
models applied to ENEM [16] tackled this challenge by employing static Word
Embeddings [17] and WordNet [18], but attained limited results, ranging
between 26-29% on accuracy. This current paper revisits the task, leveraging
the zero-shot and few-shot capabilities of GPT-3.5 and 4 models.

A recent study applied GPT-3.5 to the United States Bar Examination, achieving
a passing performance in two out of seven categories [19]. In a separate
study, it was discovered that GPT-3.5 falls significantly short of human
performance in analytical quantitative reasoning questions on the Certified
Public Accountants (CPA) Examination [20]. Nevertheless, it demonstrates a
comparable level of performance to humans in questions that demand the skills
of remembering, understanding, and applying knowledge.

In the medical domain, researchers finetuned PALM [4], a large LM pretrained
on diverse texts, on a curated set of medical-related question-answering
examples [21]. The resulting model, Med-PALM, was evaluated on questions from
the United States Medical Licensing Examination (USMLE). Their analysis
revealed that the model provided answers in agreement with the scientific
consensus, as determined by clinical experts, for 92.6% of the questions.

More recently, GPT-4 [22] demonstrated performance comparable to humans across
multiple professional and academic benchmarks, such as achieving a score
within the top 10% of participants on a simulated bar exam. It also largely
surpasses Med-PALM on a version of the USMLE benchmark [23]. We are the first
to evaluate this model on a Portuguese benchmark.

##  III Methodology

###  III-A Datasets

We use two evaluation datasets: The ENEM Challenge and ENEM 2022. The ENEM
Challenge dataset222https://www.ime.usp.br/~ddm/project/enem [16] was created
by parsing questions and alternatives from several editions of the annual ENEM
examination. The dataset’s authors have also annotated each question with the
following tags indicating the domain:

  * •

Text Comprehension (TC)

  * •

Encyclopedic Knowledge (EK)

  * •

Image Comprehension (IC)

  * •

Domain Specific Knowledge (DS)

  * •

Mathematical Reasoning (MR)

  * •

Chemical Elements (CE)

This informative knowledge tag labeling is crucial to determine whether a
question contains elements that cannot be treated as text, such as images or
chemical symbols. The complete dataset comprises 1754 ENEM questions spanning
from 2009-2017. Following [16], we eliminated questions requiring image
comprehension (IC), mathematical reasoning (MR), and having chemical elements
(CE), resulting in 916 questions. Even though the evaluated models possess the
means to process equations or chemical symbols, a comprehensive analysis of
these aspects has been limited to the 2022 exam that was parsed and annotated
by us. The questions in the ENEM Challenge dataset used in this study are
composed of three components: a header, which presents the main text; a
statement, which poses the question to the students; and alternatives, which
provide five options for the student to choose the correct answer to the
statement.

In addition, we evaluated the models on ENEM 2022 questions, which is the most
recent edition of the exam. The dataset was structured and annotated similarly
to the ENEM Challenge dataset. Table I presents the statistics of the ENEM
2022 dataset. Using the same filtering criteria applied to the ENEM Challenge
dataset, ENEM 2022 yields 88 questions. By only removing questions that
require image comprehension (IC), we have 118 questions. As for the original
ENEM Challenge dataset, here we also exclude the five questions related to the
foreign language (English or Spanish) chosen at the time of registration.

TABLE I: Number of questions of ENEM 2022 dataset. Area | total |  $\neg$ IC $\land$ $\neg$ MR $\land$ $\neg$ CE |  $\neg$ IC
---|---|---|---
Languages, Codes and their Technologies | 45 | 33 | 33
Human Sciences and their Technologies | 45 | 37 | 37
Natural Sciences and their Technologies | 45 | 18 | 26
Mathematics and its Technologies | 45 | 0 | 22
Total | 180 | 88 | 118

###  III-B The GPT Models

In this study, we use OpenAI’s GPT 3.5 and 4 models, which are available via a
paid API.333https://platform.openai.com The models we evaluated include code-
davinci-002, text-davinci-002, text-davinci-003, gpt-3.5-turbo-0314, and
gpt-4-0314. Except for gpt-4-0314, these models fall under the GPT-3.5 family,
as designated by OpenAI. For all models, we used the following parameters:
temperature of 0, which defaults to greedy decoding, maximum output of 4000
tokens, frequency penalty of 0, and presence penalty of 0.

We do not know the inner workings of the models, save for a blog post
detailing certain distinctions in their training
methodology.444https://platform.openai.com/docs/model-index-for-researchers We
know however that they were exclusively trained on data publicly available
until September 2021.555https://platform.openai.com/docs/models Hence, the
results in the ENEM 2022 test are trustworthy, given that the models could not
have access to the examination questions and answers during their training
phase.666The ENEM 2022 exam was administered on November 13-20, 2022.

###  III-C Prompt Design

We used three types of prompts to analyze multiple-choice questions: zero-
shot, few-shot, and few-shot with CoT. Each prompt is described hereafter.

####  III-C1 Zero-shot

For zero-shot prompts, the question is initially parsed into a pre-defined
prompt format and then inputted into the model, which then generates an
answer. Figure 1 illustrates a zero-shot prompt example (translated into
English).

Question 1:
---
Header: The biogeochemical cycle of carbon comprises several compartments,
including the Earth, atmosphere and oceans, and several processes that allow
the transfer of compounds between these reservoirs. Carbon stocks stored in
the form of non-renewable resources, for example oil, are limited, and it is
of great importance to realize the importance of replacing fossil fuels with
fuels from renewable sources.
Statement: The use of fossil fuels interferes with the carbon cycle, as it
causes
Alternatives:
A. Increase in the percentage of carbon contained in the Earth.
B. reduction in the rate of photosynthesis of higher plants.
C. increased production of plant-based carbohydrates.
D. increase in the amount of carbon present in the atmosphere.
E. Reducing the global amount of carbon stored in the oceans.
Answer: D. increase in the amount of carbon present in the atmosphere.
Figure 1: Example of zero-shot prompt for question 6 from the ENEM 2009 exam.
The text in blue is the output generated by the model.

By not having access to few-shot examples, the model tends to produce
responses with format variations (e.g., “D”, “D. increase in the amount of
carbon present in the atmosphere”, or “Alternative D.”). In preliminary
experiments, we observed that the use of instructions was not effective in
establishing the desired output format. To overcome this challenge, we have
developed a set of regex rules to identify the chosen alternative (A, B, C, D,
or E) and filter out the irrelevant text additions.

####  III-C2 Few-shot

The few-shot prompt is composed of some examples that induce the model to
generate responses in the expected format.

We selected three examples of different knowledge areas from ENEM 2022 as our
few-shot examples. Specifically, we chose one question from “Languages, Codes
and their Technologies", one from “Human Sciences and their Technologies", and
one from “Mathematics and its Technologies”. While the first question was
considered difficult by teachers due to the presence of
distractors777Distractors are components of multiple-choice tests that can
mislead test takers and make the incorrect alternative appears to be correct.,
the other two were graded as moderately difficult. By selecting questions from
different areas and varying levels of difficulty, this research aims to ensure
the generalizability of results to provide a thorough evaluation of this
approach. Figure 2 presents the three few-shot examples (translated into
English) that are used on the prompts in the same order as they appear.
Appendix A-A shows the original questions in Portuguese.

Question 1:
---
Header: Emotional urgency. If everything is for yesterday, if life engages a
first gear and takes off, if there is no more time for strategic stops, we
fatally fall into the addiction of wanting love to be equally resolved in a
split second. We are in a hurry to hear “I love you". We can’t wait for the
rules of coexistence to be established: are we boyfriends, hookups, married,
lovers? Emotional urgency. A trap. We associate several words with LOVE:
passion, romance, sex, adrenaline, palpitation. We forget, however, the word
that makes this feeling possible: “patience". Love without patience does not
avenge. Love cannot be chewed and swallowed with emergency, desperate hunger.
It’s a meal that can last a lifetime. MEDEIROS, M. Available at:
http://porumavidasimples.blogspot.com.br. Accessed on: 20 Aug. 2017 (adapted).
Statement: In this opinion text, the linguistic marks reveal a relaxed
situation with little formality, which is evidenced by the
Alternatives:
A. impersonalization throughout the text, as in: “if there is no more time”.
B. construction of an atmosphere of urgency, in words like: “hurry”.
C. repetition of a certain syntactic structure, as in: “If everything is for
yesterday”.
D. emphasis on the use of hyperbole, as in: “a meal that can last a lifetime”.
E. use of metaphors, as in: “life engages a first gear and takes off”.
Answer: E. use of metaphors, as in: “life engages a first gear and takes off”.
##
Question 2:
Header: Whenever the relevance of discourse comes into play, the issue becomes
political by definition, as it is the discourse that makes man a political
being. And everything men do, know or experience only makes sense to the
extent that it can be discussed. There will perhaps be truths that lie beyond
language and that may be of great relevance to man in the singular, that is,
to man who, whatever he may be, is not a political being. But men in the
plural, that is, the men who live and move and act in this world, can only
experience the meaning of things by being able to speak and be intelligible to
each other and to themselves. ARENDT, H. The human condition. Rio de Janeiro:
University Forensics, 2004.
Statement: In the excerpt, the philosopher Hannah Arendt shows the importance
of language in the process of
Alternatives:
A. understanding of culture.
B. increased creativity.
C. perception of individuality.
D. improvement of technique.
E. construction of sociability.
Answer: E. construction of sociability.
##
Question 3:
Header: A couple plans to build a swimming pool in the shape of a rectangular
parallelepiped with a capacity of 90,000 L of water on their farm. The couple
hired a construction company that presented five projects with different
combinations of the internal dimensions of depth, width and length. The pool
to be built will have the same ceramic coating on its walls and bottom, and
the couple will choose the project that requires the smallest coating area.
The internal dimensions of depth, width and length, respectively, for each of
the projects are: project I: 1.8 m, 2.0 m and 25.0 m; project II: 2.0 m, 5.0 m
and 9.0 m; project III: 1.0 m, 6.0 m and 15.0 m; project IV: 1.5 m, 15.0 m and
4.0 m; project V: 2.5 m, 3.0 m and 12.0 m.
Statement: The project that the couple should choose will be the
Alternatives:
A. I.
B. II.
C. III.
D. IV.
E. V.
Answer: B. II.
Figure 2: Questions from the ENEM 2022 exam used as few-shot examples.

For experiments on the ENEM 2022 dataset, when one of the three selected few-
shot examples is evaluated, we exclude that example from the few-shot context
and only use the remaining two examples as few-shot examples.

####  III-C3 Few-shot with CoT

Withal, this work investigates the enhancement of few-shot prompts with CoT
techniques [24]. These prompts received additional sequences of explanatory
steps, in order to evaluate the hypothesis that the models are able to
decompose complex problems into smaller, more manageable parts; i.e., enabling
the engagement of reasoning prior to compiling final results. This strategy
would allow the model to potentially grasp abstract concepts and manipulate
them, thereby facilitating the resolution of problems that require a deep
understanding of underlying principles.

The CoT design we apply in this study starts with a brief summary of the
statement. Afterward, the explanation points out the correct alternative and
justifies it. This sequence can optionally be preceded by listing all
alternatives that are likely correct, and followed by a sequence that excludes
distractors. This strategy instructs the model to be aware of distractors.
Also, the explanation ends with a justification for the remaining incorrect
alternatives. This format is similar to the original proposal of CoT for
common sense Q&A [24], except for the novel use of a section that intends to
make the model aware of distractors.

To specify requirements and constraints that enable the model to reply with
the expected structure, we have written the instruction at the beginning of
the CoT prompts as follows:

Formulate a chain of explanations that allows you to answer the multiple-
choice question below. Only one alternative is correct. Desired format: point
out the alternatives that make sense, choose the CORRECT alternative and
justify it, and finish justifying why the other alternatives are incorrect.
Finish the explanation with “Answer:” followed by the alternative.

The instruction was translated into English to facilitate the understanding of
the method, but we use it in Portuguese when evaluating the models.

In addition to the instruction, each few-shot example of a prompt with CoT is
modified as follows: after the set of alternatives, we replace “response” with
“explanation” followed by the respective explanation. The explanation ends up
with the alternative predicted as correct.

In our study, we utilize high-school teachers’ discussions888Such as those
from these links: 1, 2, 3, and 4. to formulate explanations for each few-shot
example, as shown in Figure 3. Appendix A-B presents the original explanations
in Portuguese.

Question 1:
---
The text is written in a light, agile language, with little formality. In
addition, it has figures of speech, such as metaphors and hyperboles, which
are not mutually exclusive. In a sequential analysis of the alternatives, it
would be possible to affirm that D) and E) are correct. However, looking in
detail, it is noted that the expression “use of metaphors” proves to be more
appropriate than “emphasis on the use of hyperbole”, since, in order to state
that the use of hyperbole was emphasized, the figure of speech should have
appeared more often. This makes option E) more likely to be CORRECT. In
addition, impersonality should not be pointed out as a mark of low formality.
There is also an atmosphere of urgency, but that is criticized in the text
that highlights the importance of patience and not haste. Finally, the
syntactic structure is not systematically repeated throughout the text.
Answer: E.
Question 2:
Hannah Arendt argues in her work that we are political beings, in the proper
sense of living in a polis, in a collective and social environment. And this
sociability is only possible through discourse, language. Thus, we can
conclude that language is an important tool for building sociability, and
therefore alternative E) is CORRECT. Furthermore, it is not about
understanding the culture, but the social relationship between people of that
culture. Hannah also doesn’t talk about increased creativity, nor does she
talk about technique. Finally, language is used in something more collective
and social, just the opposite of individuality. Answer: E.
Question 3:
We must calculate the area of the four side faces and the area of the lower
base (bottom of the pool) and add these areas to obtain the coating area.
Therefore, calculating the coating area of each project, we have: Project I: A
= 2 x 25 + 2 x 1.8 x (2 + 25) = 147.2; Project II: A = 9 x 5 + 2 x 2 x (9 + 5)
= 101; Project III: A = 15 x 6 + 2 x 1 x (15 + 6) = 132; Project IV: A = 4 x
15 + 2 x 1.5 x (15 + 4) = 117; Project V: A = 3 x 12 + 2 x 2.5 x (3 + 12) =
111. Therefore, the project with the smallest coating area is the project II,
therefore the correct answer is B. Answer: B.
Figure 3: Explanations for the three questions selected from the ENEM 2022
exam as few-shot examples.

##  IV Results

This section presents the results for the latest ENEM exam (2022) and for the
ENEM Challenge dataset (2009-2017). We reinforce that the few-shot experiments
used three examples from ENEM 2022 (see Section III-C2), but when evaluating
one of those, we exclude it from the prompt and use only the remaining two as
few-shot examples.

###  IV-A ENEM 2022

Table II presents the results of the zero-shot, few-shot, and few-shot with
CoT strategies applied on the latest ENEM exam. In zero-shot experiments, the
gpt-4 model achieved an average accuracy of 79.66%. It performed exceedingly
well in human sciences questions (94.59%), whereas its performance was subpar
in mathematics questions (40.91%). By utilizing few-shot prompts, the accuracy
of the model increased by a mere one percentage point. However, we observed a
substantial improvement of over 6 percentage points when using the few-shot
CoT prompt, with mathematical questions showing the highest improvement
(50.00% $\rightarrow$ 72.73%), followed by natural science questions (76.92%
$\rightarrow$ 88.46%). This improvement is exemplified in Figure 4, which
shows questions requiring mathematical reasoning. The model becomes capable of
solving complex problems when it begins to decompose them into smaller parts.

(a) Question 134 of Natural Sciences

(b) Question 141 of Mathematics

Figure 4: Comparison of answers generated by gpt-4 with and without the Chain-
of-Thought prompt. Questions are from ENEM 2022. The texts were translated
into English for better comprehension.

For code-davinci-002 and gpt-3.5-turbo, the CoT prompt resulted in substantial
improvements for mathematical questions. However, this was accompanied by a
marked decline in accuracy in the remaining domains, ultimately leading to a
decrease in overall accuracy when utilizing the CoT prompt.

TABLE II: Results on ENEM 2022. Questions that require image comprehension were removed. Area | code-davinci-002 | gpt-3.5-turbo | gpt-4
---|---|---|---
| zero-shot
---
| three-shot
---
| three-shot
---
with CoT
| zero-shot
---
| three-shot
---
| three-shot
---
with CoT
| zero-shot
---
| three-shot
---
| three-shot
---
with CoT
Languages and Codes | 78.79 | 87.88 | 72.73 | 75.76 | 81.82 | 69.70 | 84.85 | 87.88 | 87.88
Human Sciences | 89.19 | 94.59 | 91.89 | 91.89 | 89.19 | 94.59 | 94.59 | 94.59 | 94.59
Natural Sciences | 69.23 | 61.54 | 53.85 | 73.08 | 84.62 | 65.38 | 84.62 | 76.92 | 88.46
Mathematics | 18.18 | 27.27 | 50.00 | 18.18 | 36.36 | 54.55 | 40.91 | 50.00 | 72.73
Total | 68.64 | 72.88 | 70.34 | 69.49 | 76.27 | 73.73 | 79.66 | 80.51 | 87.29

###  IV-B ENEM Challenge

Table III presents results for more models on the ENEM Challenge dataset999Two
ENEM exams were held in 2016.. Following Silveira et al [16], we removed
questions that require image comprehension, mathematical reasoning and
chemical elements. The average zero-shot accuracy for the GPT-3.5 models
spanned from 77.41% to 82.88%, whereas the gpt-4 model exhibited a pronounced
increase, attaining an accuracy of 94.56%.

TABLE III: Results on ENEM 2009-2017 and 2022 using zero-shot prompts. Questions that require image comprehension, mathematical reasoning, and chemical symbol understanding were removed. Model | 2009 | 2010 | 2011 | 2012 | 2013 | 2014 | 2015 | 2016_1 | 2016_2 | 2017 | avg. | 2022
---|---|---|---|---|---|---|---|---|---|---|---|---
Silveira et. al | 26.96 | 31.37 | 33.85 | 31.34 | 30.33 | 27.01 | 29.96 | 28.42 | 27.95 | 26.68 | 29.39 | -
text-davinci-002 | 74.16 | 77.45 | 80.21 | 78.26 | 82.35 | 82.76 | 80.90 | 77.66 | 74.19 | 79.78 | 78.77 | 84.09
text-davinci-003 | 68.54 | 72.55 | 83.33 | 78.26 | 77.65 | 79.31 | 76.40 | 79.79 | 78.49 | 79.78 | 77.41 | 79.55
code-davinci-002 | 83.15 | 82.35 | 87.50 | 81.52 | 80.00 | 87.36 | 84.27 | 75.53 | 81.72 | 85.39 | 82.88 | 85.23
gpt-3.5-turbo | 78.65 | 77.45 | 85.42 | 84.78 | 74.12 | 82.76 | 84.27 | 85.11 | 76.34 | 84.27 | 81.32 | 86.36
gpt-4 | 95.51 | 92.16 | 97.92 | 98.91 | 91.76 | 96.55 | 96.63 | 92.55 | 89.25 | 94.38 | 94.56 | 90.91

Additionally, we also investigated the zero-shot performance of the ENEM 2022
dataset using the same filtering criteria of the ENEM Challenge. Results are
shown in the last column of Table III. We observe that the GPT-3.5 models
exhibited accuracies comparable to or surpassing the mean of the other ten
examinations, thereby reducing the suspicion of possible data contamination
and validating the results as the new state-of-the-art for the ENEM Challenge
dataset. Conversely, the gpt-4 reached a lower accuracy in the ENEM 2022 than
the other exams, thus raising a concern that the answers to the 2009-2017
exams were, to some extent, memorized by the model during its training.

##  V Conclusion

This study establishes GPT-4 as the new state-of-the-art model for tackling
the ENEM challenge and provides insights into the effectiveness of various
prompting strategies. The CoT prompt yielded significant improvements in terms
of accuracy, while equipping the model with the ability to generate
explanations to answers. This capability has the potential as an educational
tool as it could enhance students’ understanding of complex concepts and
support their learning process by offering more transparent and informative
responses to challenging questions. Additionally, we envision that LMs could
have a substantial impact on the education sector, potentially leading to the
adoption of AI-powered tools for psychometric analysis of exams, including
predicting question difficulty, generating new exam items, and ultimately
supporting the creation of adaptive test-suites.

##  VI Future Work

Building upon the findings of this study, several avenues for future research
can be explored. First, we propose the development and evaluation of
applications that leverage the capabilities of LMs for precision education, as
outlined by [8]. Such applications have the potential to enhance personalized
learning experiences and improve educational outcomes. More specifically, we
suggest further investigation into the generation of questions within specific
knowledge domains, as well as the calculation of difficulty levels of these
questions. This line of research will provide valuable insights into the
ability of LMs to create and adapt to varying degrees of complexity in the
assessment tasks.

Lastly, we recommend the utilization of multimodal models101010The multimodal
feature of GPT-4 was not available to the public at the time of writing this
paper. and the extension of the ENEM evaluation to questions that require
image comprehension. By incorporating these elements, researchers can explore
the full range of the potential of AI models in addressing complex and diverse
educational challenges.

##  VII Acknowledgments

We thank Thales R. Sales Almeida for providing the scrapping code to parse
questions of the ENEM 2022 exam. This research was partially funded by
Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP) (project id
2022/01640-2).

## References

  * [1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [3] Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving.  Scaling language models: Methods, analysis & insights from training gopher, 2022.
  * [4] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  arXiv preprint arXiv:2204.02311, 2022.
  * [5] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.  Training compute-optimal large language models, 2022.
  * [6] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [7] Danijel Kučak, Vedran Juričić, and Goran Đambić.  Machine learning in education-a survey of current research trends.  Annals of DAAAM & Proceedings, 29, 2018.
  * [8] Hui Luan and Chin-Chung Tsai.  A review of using machine learning approaches for precision education.  Educational Technology & Society, 24(1):250–266, 2021.
  * [9] Kabir Ahuja, Rishav Hada, Millicent Ochieng, Prachi Jain, Harshita Diddee, Samuel Maina, Tanuja Ganu, Sameer Segal, Maxamed Axmed, Kalika Bali, and Sunayana Sitaram.  Mega: Multilingual evaluation of generative ai, 2023.
  * [10] Javier De la Rosa and Andres Fernández.  Zero-shot reading comprehension and reasoning for spanish with BERTIN GPT-J-6B.  In Manuel Montes y Gómez, Julio Gonzalo, Francisco Rangel, Marco Casavantes, Miguel Ángel Álvarez Carmona, Gemma Bel-Enguix, Hugo Jair Escalante, Larissa Freitas, Antonio Miranda-Escalada, Francisco Rodríguez-Sánchez, Aiala Rosá, Marco Antonio Sobrevilla-Cabezudo, Mariona Taulé, and Rafael Valencia-García, editors, Proceedings of the Iberian Languages Evaluation Forum (IberLEF 2022). CEUR Workshop Proceedings, 2022.
  * [11] Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, Weng Lam Tam, Zixuan Ma, Yufei Xue, Jidong Zhai, Wenguang Chen, Peng Zhang, Yuxiao Dong, and Jie Tang.  Glm-130b: An open bilingual pre-trained model, 2022.
  * [12] Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel.  mt5: A massively multilingual pre-trained text-to-text transformer.  In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 483–498, 2021.
  * [13] Linting Xue, Aditya Barua, Noah Constant, Rami Al-Rfou, Sharan Narang, Mihir Kale, Adam Roberts, and Colin Raffel.  Byt5: Towards a token-free future with pre-trained byte-to-byte models.  Transactions of the Association for Computational Linguistics, 10:291–306, 2022.
  * [14] Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al.  Bloom: A 176b-parameter open-access multilingual language model.  arXiv preprint arXiv:2211.05100, 2022.
  * [15] Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel.  Crosslingual generalization through multitask finetuning, 2022.
  * [16] I. Cataneo Silveira and D. Deratani Maua.  Advances in automatically solving the enem.  In 2018 7th Brazilian Conference on Intelligent Systems (BRACIS), pages 43–48, Los Alamitos, CA, USA, oct 2018. IEEE Computer Society.
  * [17] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient Estimation of Word Representations in Vector Space.
  * [18] George A. Miller.  Wordnet: A lexical database for english.  Commun. ACM, 38(11):39–41, nov 1995.
  * [19] Michael James Bommarito and Daniel Martin Katz.  Gpt takes the bar exam.  Available at SSRN 4314839, 2022.
  * [20] Jillian Bommarito, Michael James Bommarito, Jessica Katz, and Daniel Martin Katz.  Gpt as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA Capabilities.  SSRN Electronic Journal, 2023.
  * [21] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al.  Large language models encode clinical knowledge.  arXiv preprint arXiv:2212.13138, 2022.
  * [22] OpenAI.  Gpt-4 technical report, 2023.
  * [23] Harsha Nori, Nicholas King, Scott Mayer McKinney, Dean Carignan, and Eric Horvitz.  Capabilities of gpt-4 on medical challenge problems, 2023.
  * [24] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou.  Chain of thought prompting elicits reasoning in large language models.  CoRR, abs/2201.11903, 2022.

##  Appendix A Appendices

###  A-A Few-shot examples

Figure 5 depicts in Portuguese the three few-shot examples that are used to
compose the prompt. We kept the examples in the same order as they appear in
the few-shot context.

Questão 1:
---
Cabeçalho: Urgência emocional. Se tudo é para ontem, se a vida engata uma
primeira e sai em disparada, se não há mais tempo para paradas estratégicas,
caímos fatalmente no vício de querer que os amores sejam igualmente resolvidos
num átimo de segundo. Temos pressa para ouvir “eu te amo”. Não vemos a hora de
que fiquem estabelecidas as regras de convívio: somos namorados, ficantes,
casados, amantes? Urgência emocional. Uma cilada. Associamos diversas palavras
ao AMOR: paixão, romance, sexo, adrenalina, palpitação. Esquecemos, no
entanto, da palavra que viabiliza esse sentimento: “paciência”. Amor sem
paciência não vinga. Amor não pode ser mastigado e engolido com emergência,
com fome desesperada. É uma refeição que pode durar uma vida. MEDEIROS, M.
Disponível em: http://porumavidasimples.blogspot.com.br. Acesso em: 20 ago.
2017 (adaptado).
Enunciado: Nesse texto de opinião, as marcas linguísticas revelam uma situação
distensa e de pouca formalidade, o que se evidencia pelo(a)
Alternativas:
A. impessoalização ao longo do texto, como em: “se não há mais tempo”.
B. construção de uma atmosfera de urgência, em palavras como: “pressa”.
C. repetição de uma determinada estrutura sintática, como em: “Se tudo é para
ontem”.
D. ênfase no emprego da hipérbole, como em: “uma refeição que pode durar uma
vida”.
E. emprego de metáforas, como em: “a vida engata uma primeira e sai em
disparada”.
Resposta: E. emprego de metáforas, como em: “a vida engata uma primeira e sai
em disparada”.
##
Questão 2:
Cabeçalho: Sempre que a relevância do discurso entra em jogo, a questão torna-
se política por definição, pois é o discurso que faz do homem um ser político.
E tudo que os homens fazem, sabem ou experimentam só tem sentido na medida em
que pode ser discutido. Haverá, talvez, verdades que ficam além da linguagem e
que podem ser de grande relevância para o homem no singular, isto é, para o
homem que, seja o que for, não é um ser político. Mas homens no plural, isto
é, os homens que vivem e se movem e agem neste mundo, só podem experimentar o
significado das coisas por poderem falar e ser inteligíveis entre si e consigo
mesmos. ARENDT, H. A condição humana. Rio de Janeiro: Forense Universitária,
2004.
Enunciado: No trecho, a filósofa Hannah Arendt mostra a importância da
linguagem no processo de
Alternativas:
A. entendimento da cultura.
B. aumento da criatividade.
C. percepção da individualidade.
D. melhoria da técnica.
E. construção da sociabilidade.
Resposta E. construção da sociabilidade.
##
Questão 3:
Cabeçalho: Um casal planeja construir em sua chácara uma piscina com o formato
de um paralelepípedo reto retângulo com capacidade para 90 000 L de água. O
casal contratou uma empresa de construções que apresentou cinco projetos com
diferentes combinações nas dimensões internas de profundidade, largura e
comprimento. A piscina a ser construída terá revestimento interno em suas
paredes e fundo com uma mesma cerâmica, e o casal irá escolher o projeto que
exija a menor área de revestimento. As dimensões internas de profundidade,
largura e comprimento, respectivamente, para cada um dos projetos, são:
projeto I: 1,8 m, 2,0 m e 25,0 m; projeto II: 2,0 m, 5,0 m e 9,0 m; projeto
III: 1,0 m, 6,0 m e 15,0 m; projeto IV: 1,5 m, 15,0 m e 4,0 m; projeto V: 2,5
m, 3,0 m e 12,0 m.
Enunciado: O projeto que o casal deverá escolher será o
Alternativas:
A. I.
B. II.
C. III.
D. IV.
E. V.
Resposta: B. II.
Figure 5: Questions from the ENEM 2022 exam used as few-shot examples (in
Portuguese).

###  A-B CoT Explanations

Figure 6 presents in Portuguese the explanations formulated for the three few-
shot examples. The explanations were created from contents of high-school
teachers’ discussions.

Questão 1:
---
O texto é escrito em uma linguagem leve, ágil, e de pouca formalidade. Além
disso, possui figuras de linguagem, como metáforas e hipérboles, que não são
excludentes. Em uma análise sequencial das alternativas, daria para afirmar
que D. e E. estão corretas. Entretanto, observando em detalhes, nota-se que a
expressão "emprego de metáforas" mostra ser mais adequada do que "ênfase no
emprego da hipérbole", visto que, para afirmarmos que o uso de hipérboles foi
enfatizado, a figura de linguagem deveria ter aparecido mais vezes. Isso torna
a alternativa E. mais provável de ser CORRETA. Além disso, impessoalização não
deve ser apontada como marca de pouca formalidade. Existe também uma atmosfera
de urgência, mas que é criticada no texto que destaca a importância da
paciência e não da pressa. Por fim, a estrutura sintática não é repetida
sistematicamente ao longo do texto. Resposta: E.
Questão 2:
Hannah Arendt defende em sua obra que somos seres políticos, no sentido
próprio de vivermos em pólis, em ambiente coletivo e social. E essa
sociabilidade só é possível por meio do discurso, da linguagem. Desse modo,
podemos concluir que a linguagem se apresenta como uma importante ferramenta
para a construção da sociabilidade, e portanto a alternativa E. é a CORRETA.
Além disso, não se trata do entendimento da cultura, mas da relação social
entre as pessoas dessa cultura. Hannah também não fala sobre aumento de
criatividade, tampouco sobre técnica. Por fim, a linguagem é utilizada em algo
mais coletivo e social, justamente o oposto da individualidade. Resposta: E.
Questão 3:
Devemos calcular a área das quatro faces laterais e a área da base inferior
(fundo da piscina) e somar essas áreas para obter a área de revestimento.
Logo, calculando a área de revestimento de cada projeto, temos: Projeto I: A =
2 x 25 + 2 x 1,8 x (2 + 25) = 147,2; Projeto II: A = 9 x 5 + 2 x 2 x (9 + 5) =
101; Projeto III: A = 15 x 6 + 2 x 1 x (15 + 6) = 132; Projeto IV: A = 4 x 15
+ 2 x 1,5 x (15 + 4) = 117; Projeto V: A = 3 x 12 + 2 x 2,5 x (3 + 12) = 111.
Logo, o projeto com menor área de revestimento, é o projeto II, portanto a
resposta corrreta é B. Resposta: B.
Figure 6: Explanations for the three questions selected from the ENEM 2022
exam as few-shot examples (in Portuguese).