11institutetext:  Department of Physics, TU Dortmund University, Dortmund,
NRW, Germany 22institutetext: Centre for Medical Radiation Physics, University
of Wollongong, Wollongong, New South Wales, Australia 33institutetext:
Australian Synchrotron (ANSTO), Clayton, Victoria, Australia 44institutetext:
Illawarra Health and Medical Research Institute, University of
Wollongong,Wollongong, New South Wales, Australia 55institutetext: School of
Computing and Information Technology, University of Wollongong,Wollongong, New
South Wales, Australia

# A step towards treatment planning for microbeam radiation therapy: fast peak
and valley dose predictions with 3D U-Nets

Florian Mentzel 11 Micah Barnes 2233 Kevin Kröninger 11 Michael Lerch 2244
Olaf Nackenhorst 11 Jason Paino 2244 Anatoly Rosenfeld 2244 Ayu Saraswati 55
Ah Chung Tsoi 55 Jens Weingarten 11 Markus Hagenbuchner 55 Susanna Guatelli
2244

###### Abstract

Fast and accurate dose predictions are one of the bottlenecks in treatment
planning for microbeam radiation therapy (MRT). In this paper, we propose a
machine learning (ML) model based on a 3D U-Net. Our approach predicts
separately the large doses of the narrow high intensity synchrotron microbeams
and the lower valley doses between them. For this purpose, a concept of macro
peak doses and macro valley doses is introduced, describing the respective
doses not on a microscopic level but as macroscopic quantities in larger
voxels. The ML model is trained to mimic full Monte Carlo (MC) data. Complex
physical effects such as polarization are therefore automatically taking into
account by the model. The macro dose distribution approach described in this
study allows for superimposing single microbeam predictions to a beam array
field making it an interesting candidate for treatment planning. It is shown
that the proposed approach can overcome a main obstacle with microbeam dose
predictions by predicting a full microbeam irradiation field in less than a
minute while maintaining reasonable accuracy.

###### keywords:

Microbeams, dose prediction, deep learning, U-Net

##  1 Introduction

Microbeam radiation therapy (MRT) is a novel and currently pre-clinical
radiotherapy treatment based on planar arrays of high intensity sub-millimetre
synchrotron gamma rays [1, 2, 3]. The narrow beams cut with a multi-slit
collimator from the broad beam, often only 50$\,\mu$m wide each, result in
high doses only being applied in the volumes passed by the microbeams (peak
dose), while in the space between the microbeams a significantly lower dose is
deposited (valley dose). This was observed to lead to good healthy tissue
sparing and tumour control and is therefore a promising candidate for
treatment of a variety of tumours which have a weak clinical outcome with
currently available treatments [4, 5]. Ongoing research, e.g. conducted at the
Australian Synchrotron (e.g. [6, 7]) and the European Synchrotron Radiation
Facility (ESRF, France, a recent overview can be found in [2]), has provided
important steps for MRT to move towards first clinical trials.
Presently, accurate treatment planning for MRT is only possible using full
Monte Carlo (MC) simulations which take up to 250 h CPUh (hours per CPU used).
A faster algorithm involving only a first-order MC simulation and applying an
additional kernel method to account for the dose depositions from secondary
electrons is available but still requires up to 30 minutes per configuration
[8].
In a recent publication it was shown that a deep learning model adapted from
the so-called 3D U-Net [9] can be used to predict the dose distribution of a
rectangular synchrotron broad beam of 8 mm width and height in a simplified
head phantom fast and accurately [10]. In this study we extend the research to
the prediction of dose distributions resulting from synchrotron microbeams.
For this purpose, we propose an MRT dose prediction approach based on the
superposition of individual predictions of single microbeams to a whole
microbeam array field. This results in the ability to build complex
irradiation fields flexibly from simple building blocks.

(a)

(b)

Figure 1: (a): Dose profile of a single microbeam (red) and a microbeam array
(black). (b): Contribution of a microbeam located at $0\,$mm to the valleys
further away relative to the contribution to its directly neighbouring valley.

Accurately predicting the dose deposited by a microbeam requires high spatial
resolution, resulting in a need for small dose scoring voxels. Irradiation
fields are often significantly larger ($\approx 1-20\,$mm) than one microbeam
is wide. This large aspect ratio leads to a high importance of secondary
electrons even far away from the actual microbeam. Figure (1(a)) highlights
the vast difference in valley doses resulting from one microbeam compared to
the full beam array. At the location of the first valley, directly next to the
central microbeam peak, the valley doses differ by an order of magnitude which
indicates that not only the next neighbouring microbeams can be taken into
account for valley dose calculation. This issue is further investigated in
Figure (1(b)) which shows the relative contribution of a microbeam to valleys
many microbeams away. Given a large field size, a microbeam can for example
contribute more than 10% of the dose it deposits in it’s adjacent valley
($\approx 200\,\mu$m from the beam centre) in the valley which is located 30
microbeams (or 12.5 mm) far away. As a solution to this, we propose a dose
scoring method resulting in the peak and valley doses being treated as
macroscopic quantities assigned to larger voxels of e.g. 0.5 mm or 1 mm edge
length (depending on the treatment requirements) which we will refer to as
macro voxels. We will show how the resulting macro voxel data, i.e. the macro
peak dose and macro valley dose, are suitable for superposition of dose
distributions from single microbeams making them suited as training data for
an ML model for microbeam dose predictions.
The remainder of this paper is structured as follows: Section (2) presents the
MC simulations which were adopted to produce the data shown in this paper.
Subsequently, the novel macro voxel approach is introduced and explained in
detail. Finally, the ML model is presented in Section (2.3) which is trained
to predict the macro peak and valley dose distributions in a simple water
phantom. Section (3) first presents results that support the assumption that
microbeam array irradiations can be modelled as superposition of single
microbeams. Thereafter, we show the results of this proof of concept ML
training on the novel macro voxel data. Results and some limitations are
discussed in Section (4). A summary and an outlook is presented in Section
(5).

##  2 Methods

This Section presents the MC simulations used to generate data samples,
introduces the macro voxel approach to data taking and presents the 3D U-Net
architecture that is trained to predict the microbeam dose distributions
previously simulated with the MC code.

###  2.1 Monte Carlo Simulations

All MC simulations were performed using Geant4 10.6p01 [11, 12]. The physics
list used comprises the option 4 of the electromagnetic physics constructor
which is extended for polarized physics processes using the Livermore
Polarized Physics processes [13].
The microbeam particle source used in the simulations is obtained as phase
space file (PSF) from a previously published and experimentally validated
Geant4 simulation [6] of the full Imaging and Medical Beamline (IMBL) at the
Australian Synchrotron, ANSTO [14]. The microbeams have a nominal width of
$w=40\,\mu$m each and a height of $h=1\,$mm. The individual microbeams are
spaced at a distance of $411\,\mu$m when entering the phantom. The photons
comprising the microbeams have a mean energy of approximately 80 keV, the
energy distribution is shown in Figure (2(a)).

####  2.1.1 Microbeam Array Simulation

The simulation setup for the microbeam array irradiation scenario is shown in
Figure (2(b)). The microbeams are incident on a small 30x30x30 mm3 water
phantom which can be translated to irradiate different sections of the
phantom. This size of phantom is chosen because of the current pre-clinical
focus on small-scale animal phantoms of MRT research. The microbeam array
field obtained from the PSF has an original width of 30 mm and is cropped
using a tungsten mask not visible in the schematic to width $20\,$mm as it is
a more commonly used configuration. To include dose depositions further away
from the beam, the dose is scored in a volume of also 30x30x30 mm3 using
voxels of edge length 5$\,\mu$m leading to a total 6000x6000x6000 voxels per
simulation for a total of 216 billion voxels. An alternative method for dose
scoring is introduced in Section (2.2).

(a)

(b)

Figure 2: Energy distribution of the microbeams (a), adapted from [10], and a
schematic of the microbeam array irradiation simulation (b) showing a 30x30x30
mm3 water phantom in blue and the edges of the microbeam field in dashed
green.

Figure (3) shows the dose profile at the centre of the water phantom resulting
from the simulation. A grey visual guide is included in the figure to allow
for the identification of individual microbeams. The dose deposition at the
centre of a microbeam is referred to as peak dose while the dose deposition in
the middle of two microbeams is referred to as valley dose. Differences in
peak height are not due to low statistics in the simulations run for this
study but originate from the creation of the PSF.

Figure 3: Dose distribution at the centre of the 30x30x30 mm3 water phantom
scored with voxels of 5$\,\mu$m edge length (black). A visual guide is given
in grey to identify the individual microbeams.

####  2.1.2 Single Microbeam Simulation

To simulate a single individual microbeam, the central microbeam is extracted
from the PSF. Figure (4) shows the schematic for this simulation using the
same small-scale water phantom. In addition to the features in Figure (2), an
exemplary translation of the phantom in positive direction is shown. The
scoring area is kept centred around the microbeam containing water and air in
that case.

Figure 4: Single microbeam simulation schematic showing a 3x3x3 mm3 water
phantom in blue (translated in positive direction) and indicates the position
of the microbeam in green.

###  2.2 Macro Quantity Scoring

In the treatment planning for MRT, a highly resolved dose distribution matrix
is often not required. Instead, it is more important to know the distribution
of peak and valley doses throughout the phantom. Especially the valley doses
are of high importance as those determine the impact on healthy tissue. In
contrast to the microscopic dose deposition, those quantities can be computed
in a macroscopic manner in larger voxels. Figure (5) shows a schematic of this
approach to MRT data taking for macro voxels of 0.5 mm edge length. During a
simulation, the whole scoring volume is separated into peak scoring volumes,
valley scoring volumes and inactive volumes which span the whole height of a
voxel but only $10\,\mu$m width for the peak scoring and $100\,\mu$m width for
the valley scoring (due to shallow dose profile). Dose depositions are counted
only towards the peak dose or valley dose for a given voxel, if they occur
within the respective sub-volume.

Figure 5: Calculation of peak and valley doses within larger voxels as
macroscopic quantities. The voxels are indicated as black squares, the
turquoise rectangles indicate that dose depositions (red) which occur within
them are counted as peak dose, the green rectangles indicate counting towards
the valley dose.

This approach simplifies the description of data significantly. Figure (6(a))
shows the same lateral dose profile as Figure (3) but includes the peak and
valley doses computed with the presented approach. A more detailed depiction
of those in Figure (6(b)) shows the lateral peak and valley dose profiles.

(a)

(b)

Figure 6: (a) shows the lateral dose profile from Figure (3) together with the
respective peak (turquoise) and valley (green) doses. (a) allows a closer
inspection of those macroscopic quantities by showing the same values isolated
on individual axis. Visual guides are included in grey.

The peak and valley doses can also be recorded for a single microbeam by
keeping the volumes the same size as in the microbeam array scenario. This is
shown in Figure (7(a)). The peak and valley scoring volumes are shifted
together with the microbeam while the edges of the macro voxels stay the same.
This can be seen in Figure (7(b)) and Figure (7(c)).

(a)

(b)

(c)

Figure 7: (a) shows the peak and valley scoring areas in the case of an array
of microbeams (black) and a single microbeam (red). (b) and (c) show the
translation of a single microbeam and the peak and valley scoring volumes
relative to the macro voxels.

As the dose fall-off is short-ranged compared to the macro voxel size, the
dose distributions depend significantly on the initial position of the
microbeam relative to the scoring voxels. The effect is shown schematically in
Figures (8(a)) and (8(b)) where the peak dose per macro voxel is displayed for
two different microbeam positions.

(a)

(b)

Figure 8: Schematic of the peak dose distribution in dependence of the
microbeam position relative to the macro voxels.

From Figure (7(a)) can also be seen how to superimpose peak and valley doses
from individual single microbeams using the macro voxel technique: two macro
voxel dose distribution can be added together if the distance between the
single microbeams is a multiple of the peak scoring volume distance. In our
experiments, the peak doses are always recorded at every $411\,\mu$m.
Therefore, a microbeam located e.g. at a global horizontal (lateral) position
$y=0\,\mu$m can be added to ones located at $y=\pm 411\,\mu$m or $y=\pm
822\,\mu$m while a microbeam located at $y=200\,\mu$m could only be added to
ones located at e.g. $y=-622\,\mu$m, $y=-211\,\mu$m or $y=+622\,\mu$m. This
assumption is acceptable as in a realistic treatment scenario, the spacing
between applied microbeams will always be the same for a given multi-slit
collimator while the whole array may move relative to the phantom.
Figure (9) shows two exemplary lateral peak and valley dose profiles for a
single microbeam in the water phantom (including phantom shifts) recorded with
macro voxels. In the case of 0 mm beam translation, the centre of the
microbeam lies exactly on the border of two macro voxels, splitting the
contribution. The peak dose in the central valleys in Figure (9(a)) is about
1/3 of the highest peak dose in Figure (9(b)) because in the first case, there
are two potential peak scoring volumes contributing to each macro voxel which
can be seen in Figure (5). This leads to the occurring energy depositions
being divided by more volume. In the second case, the highest voxel contains
only one peak scoring volume resulting in no further division of the deposited
energy.

(a)

(b)

Figure 9: Exemplary lateral peak and valley dose profiles together at the
centre of the scoring area together with the density matrix representing the
phantom translation for a microbeam translation of 0 mm and a phantom
translation of 7 mm (a) and 0.25 mm and -14 mm (b) respectively.

###  2.3 Machine Learning Model

The machine learning model that is trained to predict the macro voxel dose
distributions given a microbeam position and a CT/ density matrix is a 3D
U-Net adapted from [10] and shown schematically in Figure (10). The model is
implemented using the Keras API [15] for Tensorflow 2.2 [16]. 3D U-Nets
comprise a compression stage (left, dark grey), a most compressed so-called
bottleneck stage (centre) and a decompression stage (right, red-white-blue).
Skip connections in each hierarchical level from compression to decompression
allow for feature extraction at different scales and of different complexity.
In our model, each convolutional layer comprises 64 filters with a kernel size
of 5 for the two first and last levels and kernel size 3 for the other levels.
Dropout [17] with a rate of 0.15 is applied after each stage except the output
layer.

Figure 10: Schematic of the used 3D U-Net model. Adapted from [10].

In contrast to the generative adversarial network model discussed in [10], the
3D U-Net in this study is trained as regression using the mean squared error
as loss function. Two separate networks are trained for the prediction of the
peak and the valley macro dose distributions. Both networks are trained with
the Adam optimizer [18] using a learning rate of $1\cdot 10^{-4}$ and a batch
size of 16.

###  2.4 Machine Learning Training Data

Figure (11) shows the split of data points into training, validation and test
samples used for the study. The data points differ in phantom translation
$\Delta_{\mathrm{P}}$ and beam translation $\Delta_{\mathrm{B}}$. Data samples
from phantom translations of
$-11\,\mathrm{mm}<=\Delta_{\mathrm{P}}<=-9\,\mathrm{mm}$,
$-4.5\,\mathrm{mm}<=\Delta_{\mathrm{P}}<=-2.5\,\mathrm{mm}$,
$2.5\,\mathrm{mm}<=\Delta_{\mathrm{P}}<=4.5\,\mathrm{mm}$ and
$9\,\mathrm{mm}<\Delta_{\mathrm{P}}<11\,\mathrm{mm}$ are used as validation
samples meaning that they are not presented to the network for training but
only for performance evaluation during the training to compare different ML
model configurations.
In addition, a test set of data samples is held out during the entire search
for an optimal model which is only used for performance evaluation at the end
of the study to allow for a minimal bias. The test set comprises of several
phantom translations at beam translations of multiples of $411\,\mu$m (modulo
$500\,\mu$m) instead of the regular grid used for training and validation
data. The test data, therefore, resembles the beam predictions needed for
predicting microbeams that can be superimposed to build a microbeam
irradiation array.
For this proof of concept, the simulated data is reduced in size to 30x30x30
voxels of edge length 1 mm instead of 60x60x60 voxels of edge length $0.5\,$mm
for the training of the ML model.

Figure 11: Training (grey), validation (black) and test (red) data points as a
function of the phantom and beam translation.

##  3 Results

###  3.1 Superposition Of Single Microbeams

Figure (12) shows the peak and valley doses for a 20 mm wide microbeam array
field. The red markers indicate the results obtained from the full microbeam
array simulation while the data indicated by the black markers is obtained
from the superposition of multiple translated simulations of the central
microbeam. It can be seen, that the full field simulation shows a lateral
roll-off while the peak doses of the superimposed field are more even.
Together with the peak doses, a black line indicates the effect of the number
of photons that comprise the individual microbeams. Due to beam diversion, the
multi-slit collimator crops more photons from the microbeams further outside
from the field centre leaving them with less photons contributing to the peak
doses. This effect will have to be corrected for when using superimposed
single microbeam dose distributions by scaling the peak doses accordingly. In
Figure (12(b)) it can be seen that the valley doses systematically lie 0-5%
lower for the full simulation over the whole width of the field. Whether the
correction of the peak doses applied to the single beam valley doses would
suffice to correct this difference will be required to be investigated in
future studies.

(a)

(b)

Figure 12: Comparison between peak (a) and valley (b) doses of a microbeam
array from superposition of single microbeams and the full field simulation.
The red line in (a) indicates the effect of the photon number per microbeam.

###  3.2 Peak And Valley Dose Predictions

Each 3D U-Net is trained until the validation loss is not further reduced. The
minimum validation loss for the valley dose prediction is achieved after 1306
epochs, the minimum for the peak dose distribution after 839 epochs. The
corresponding loss curves are shown in Figure (13). The validation loss lies
systematically below the training loss because especially predictions with
large phantom translations with the water-air transition being closer to the
microbeam centre induce large errors. The edge cases are not included in the
validation set, therefore the loss stays lower.

(a)

(b)

Figure 13: Training (grey) and validation (black) loss (mean squared error)
for the valley (a) and peak (b) dose network. The red circle indicates the
epoch with the lowest validation error respectively.

To investigate the generalization of the training, all training data,
validation data and now also the test data is predicted. The prediction time
with the used NVidia GForce 1080Ti averages $t=116\pm 4\,$ms for individual
sequential predictions. In batch mode, predicting multiple configurations
simultaneously, this can be significantly decreased. Using the training batch
size of 16, those samples can be predicted within $t_{\mathrm{batch}}=241.8\pm
0.3\,$ms on average resulting in a prediction time per sample of
$t_{\mathrm{sample}}\approx 15\,$ms. Based on this, a full 20x20 mm2 microbeam
array field consisting of 70 single microbeams, each 20 mm high would take
approximately 40 seconds (20 s for peak and valley dose each) to compute. It
can be expected that a more general model for more complex geometries would
not result in slower prediction times.

(a)

(b)

Figure 14: Ratio of voxels deviating less than 3% from the MC ground truth in
the peak (a) and valley (b) predictions of training, validation and test data.

Figure (14) shows a performance comparison averaged over all beam translations
as a function of phantom translation: the ratio of voxels in which the ML
prediction deviates less than 3% from the MC ground truth. The values are very
low due to the noisy valley data and the rather poor prediction of the peak
dose network which can be inspected with exemplary predictions in Figure (15).
Especially for the peak dose network, but also for the valley dose network,
the performance on the test data set is slightly lower than for the training
and validation data sets, indicating over-training. This could also be due to
more complex data samples being contained in the test data.
For further investigation, exemplary test data samples are predicted and the
depth dose curves and lateral dose profiles are investigated in Figure (15).
All three samples shown are not part of the training, but the sample resulting
in the Figures (15(c), 15(f), 15(i), 15(l)) is of special interest as neither
the microbeam translation ($322\,\mu$m), nor the phantom translation (7 mm)
are included in the training of the network. Nevertheless, the drop in dose in
the lateral profile is correctly predicted at the edge of the phantom. Much of
the scattering has to be attributed to the MC data noise which could not be
reduced within reasonable computing time due to the large distance from the
microbeams. The ML model provides a smoother prediction as it was trained on
many different MC simulations allowing it to learn the average from the data.
This capability, together with the notation of the extensive computing time
requirements for accurate valley dose distributions emphasizes the value of
such an ML model for the valley doses.
The peak doses are predicted with a lower quality compared to the valley
doses. The peak depth dose cuves exhibit deviations of up to 5% and the
lateral profile shows many entries which are clipped to $D=1\cdot 10^{-5}$
which is assigned during the prediction when the network tries to output a
dose value below 0 which is not physically meaningful.

(a)

(b)

(c)

(d)

(e)

(f)

(g)

(h)

(i)

(j)

(k)

(l)

Figure 15: Exemplary normalized predictions of valley (a-f) and peak (g-l)
doses. (a-c) and (g-i) show depth dose curves while (d-f) and (j-l) show
lateral dose profiles at the centre of the phantom respectively. The predicted
configurations are beam translations of 55$\,\mu$m (a,d,g,j), 466$\,\mu$m
(b,e,h,k), 322$\,\mu$m (c,f,i,l) for phantom translations of 0 mm (a,d,g,j),
-14 mm (b,e,h,k), 7 mm (c,f,i,l).

##  4 Discussion

Using the macro voxel approach, microbeam array irradiation fields can be
easily constructed from single microbeams by superposition. For this, the
microbeams are required to be produced in the pre-defined peak-to-peak
distance between two peak scoring volumes. The differences between the
presented dose distributions from a full microbeam field simulation and a
superposition of multiple versions of the central microbeam are to be
investigated in future studies.
The 3D U-Net model was able to predict both the peak and the valley doses.
Especially in the case of the peak doses though, the predictions which are
further away from the central beam are subject to large deviations. As the
peak doses do not influence neighbouring peak doses as much as valley doses
do, as discussed in this study, this is not a major problem for the dose
prediction. Nevertheless, the peak dose network should be investigated more
carefully.
A feature of the ML model worth to note is the fast prediction. Given batch
processing of currently 16 density matrices (in future CT matrices), a full
irradiation field could be predicted in around 40 seconds which is
significantly faster than existing methods that allow accurate dose
predictions [19, 8].
A proper estimate of the performance of the model given more complex training
data is difficult do provide and will need more detailed investigations. So
far, only a very simplified sub-space of possible microbeam applications has
been investigated. The inclusion of more sophisticated models will be an
important next step to showcase the full potential of this approach.

##  5 Conclusion

The presented macro voxel method to score peak and valley doses for microbeam
radiation therapy allows for informed treatment planning without handling
large dose scoring arrays which are both impractical and memory intensive. It
is also possible to integrate the presented method into existing MC
simulations as well.
Two 3D U-Nets are trained to predict the peak and valley dose distribution in
a 30x30x30 voxel matrix where each voxel has an edge length of 1 mm. The
predictions are fast and reasonably accurate but further investigations are
required towards the applicability of the presented approach for more complex
microbeam radiation therapy irradiation scenarios.

## Acknowledgements

The authors gratefully acknowledge the computing time provided on the Linux
HPC cluster at Technical University Dortmund (LiDO3), partially funded in the
course of the Large-Scale Equipment Initiative by the German Research
Foundation (DFG) as project 271512359.

## References

  * [1] D. N. Slatkin, P. Spanne, F. A. Dilmanian, J. O. Gebbers, and J. A. Laissue.  Subacute neuropathological effects of microplanar beams of x-rays from a synchrotron wiggler.  Proceedings of the National Academy of Sciences of the United States of America, 92(19):8783–8787, 1995.
  * [2] S Bartzsch, S. Corde, J. C. Crosbie, L. Day, M. Donzelli, M. Krisch, M. Lerch, P. Pellicioli, L. M. L. Smyth, and M. Tehei.  Technical advances in x-ray microbeam radiation therapy.  Physics in Medicine & Biology, 65(2):02TR01, 2020.
  * [3] H. Fukunaga, K. T. Butterworth, S. J. McMahon, and K. M. Prise.  A Brief Overview of the Preclinical and Clinical Radiobiology of Microbeam Radiotherapy.  Clinical Oncology, 33(11):705–712, 2021.
  * [4] E. Bräuer-Krisch, J. F. Adam, E. Alagoz, S. Bartzsch, J. Crosbie, C. DeWagter, A. Dipuglia, M. Donzelli, S. Doran, Pa. Fournier, J. Kalef-Ezra, A. Kock, M. Lerch, C. McErlean, U. Oelfke, P. Olko, M. Petasecca, M. Povoli, A. Rosenfeld, E. A. Siegbahn, D. Sporea, and B. Stugu.  Medical physics aspects of the synchrotron radiation therapies: Microbeam radiation therapy (MRT) and synchrotron stereotactic radiotherapy (SSRT).  Physica Medica, 31(6):568–583, 2015.
  * [5] M. A. Grotzer, E. Schültke, E. Bräuer-Krisch, and J. A. Laissue.  Microbeam radiation therapy: Clinical perspectives.  Physica Medica, 31(6):564–567, 2015.
  * [6] A. Dipuglia, M. Cameron, J. A. Davis, I. M. Cornelius, A. W. Stevenson, A. B. Rosenfeld, M. Petasecca, S. Corde, S. Guatelli, and M. L. F. Lerch.  Validation of a Monte Carlo simulation for Microbeam Radiation Therapy on the Imaging and Medical Beamline at the Australian Synchrotron.  Scientific Reports, 9(1):1–14, 2019.
  * [7] E. Engels, N. Li, J. Davis, J. Paino, M. Cameron, A. Dipuglia, S. Vogel, M. Valceski, A. Khochaiche, A. O’Keefe, M. Barnes, A. Cullen, A. Stevenson, S. Guatelli, A. Rosenfeld, M. Lerch, S. Corde, and M. Tehei.  Toward personalized synchrotron microbeam radiation therapy.  Scientific Reports, 10(1):1–13, 2020.
  * [8] M. Donzelli, E. Brauer-Krisch, U. Oelfke, J. J. Wilkens, and S. Bartzsch.  Hybrid dose calculation: A dose calculation algorithm for microbeam radiation therapy.  Physics in Medicine and Biology, 63(4):45013, 2018.
  * [9] Ö. Çiçek, A. Abdulkadir, S. S. Lienkamp, T. Brox, and O. Ronneberger.  3D U-net: Learning dense volumetric segmentation from sparse annotation.  In Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), volume 9901 LNCS, pages 424–432, 2016.
  * [10] F. Mentzel, K. Kröninger, M. Lerch, O. Nackenhorst, J. Paino, A. Rosenfeld, A. Saraswati, A. C. Tsoi, J. Weingarten, M. Hagenbuchner, and S. Guatelli.  Fast and accurate dose predictions for novel radiotherapy treatments in heterogeneous phantoms using conditional 3d-unet generative adversarial networks.  Medical Physics, n/a(n/a):1–16, 2022.
  * [11] S. Agostinelli, J. Allison, K. Amako, J. Apostolakis, H. Araujo, P. Arce, M. Asai, D. Axen, S. Banerjee, G. Barrand, F. Behner, L. Bellagamba, J. Boudreau, L. Broglia, A. Brunengo, H. Burkhardt, S. Chauvie, J. Chuma, R. Chytracek, G. Cooperman, G. Cosmo, P. Degtyarenko, A. Dell’Acqua, G. Depaola, D. Dietrich, R. Enami, A. Feliciello, C. Ferguson, H. Fesefeldt, G. Folger, F. Foppiano, A. Forti, S. Garelli, S. Giani, R. Giannitrapani, D. Gibin, J. J. Gomez Cadenas, I. Gonzalez, G. Gracia Abril, G. Greeniaus, W. Greiner, V. Grichine, A. Grossheim, S. Guatelli, P. Gumplinger, R. Hamatsu, K. Hashimoto, H. Hasui, A. Heikkinen, A. Howard, V. Ivanchenko, A. Johnson, F. W. Jones, J. Kallenbach, N. Kanaya, M. Kawabata, Y. Kawabata, M. Kawaguti, S. Kelner, P. Kent, A. Kimura, T. Kodama, R. Kokoulin, M. Kossov, H. Kurashige, E. Lamanna, T. Lampen, V. Lara, V. Lefebure, F. Lei, M. Liendl, W. Lockman, F. Longo, S. Magni, M. Maire, E. Medernach, K. Minamimoto, P. Mora de Freitas, Y. Morita, K. Murakami, M. Nagamatu, R. Nartallo, P. Nieminen, T. Nishimura, K. Ohtsubo, M. Okamura, S. O’Neale, Y. Oohata, K. Paech, J. Perl, A. Pfeiffer, M. G. Pia, F. Ranjard, A. Rybin, S. Sadilov, E. di Salvo, G. Santin, T. Sasaki, N. Savvas, Y. Sawada, S. Scherer, S. Sei, V. Sirotenko, D. Smith, N. Starkov, H. Stoecker, J. Sulkimo, M. Takahata, S. Tanaka, E. Tcherniaev, E. Safai Tehrani, M. Tropeano, P. Truscott, H. Uno, L. Urban, P. Urban, M. Verderi, A. Walkden, W. Wander, H. Weber, J. P. Wellisch, T. Wenaus, D. C. Williams, D. Wright, T. Yamada, H. Yoshida, and D. Zschiesche.  GEANT4 - A simulation toolkit.  Nuclear Instruments and Methods in Physics Research, Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 506(3):250–303, 2003.
  * [12] J. Allison, K. Amako, J. Apostolakis, P. Arce, M. Asai, T. Aso, E. Bagli, A. Bagulya, S. Banerjee, G. Barrand, B. R. Beck, A. G. Bogdanov, D. Brandt, J. M.C. Brown, H. Burkhardt, Ph Canal, D. Cano-Ott, S. Chauvie, K. Cho, G. A.P. Cirrone, G. Cooperman, M. A. Cortés-Giraldo, G. Cosmo, G. Cuttone, G. Depaola, L. Desorgher, X. Dong, A. Dotti, V. D. Elvira, G. Folger, Z. Francis, A. Galoyan, L. Garnier, M. Gayer, K. L. Genser, V. M. Grichine, S. Guatelli, P. Guèye, P. Gumplinger, A. S. Howard, I. Hřivnáčová, S. Hwang, S. Incerti, A. Ivanchenko, V. N. Ivanchenko, F. W. Jones, S. Y. Jun, P. Kaitaniemi, N. Karakatsanis, M. Karamitrosi, M. Kelsey, A. Kimura, T. Koi, H. Kurashige, A. Lechner, S. B. Lee, F. Longo, M. Maire, D. Mancusi, A. Mantero, E. Mendoza, B. Morgan, K. Murakami, T. Nikitina, L. Pandola, P. Paprocki, J. Perl, I. Petrović, M. G. Pia, W. Pokorski, J. M. Quesada, M. Raine, M. A. Reis, A. Ribon, A. Ristić Fira, F. Romano, G. Russo, G. Santin, T. Sasaki, D. Sawkey, J. I. Shin, I. I. Strakovsky, A. Taborda, S. Tanaka, B. Tomé, T. Toshito, H. N. Tran, P. R. Truscott, L. Urban, V. Uzhinsky, J. M. Verbeke, M. Verderi, B. L. Wendt, H. Wenzel, D. H. Wright, D. M. Wright, T. Yamashita, J. Yarba, and H. Yoshida.  Recent developments in Geant4.  Nuclear Instruments and Methods in Physics Research Section A: Accelerators, Spectrometers, Detectors and Associated Equipment, 835:186–225, 2016.
  * [13] Geant4 Collaboration.  Physics Reference Manual Documentation.  https://geant4-userdoc.web.cern.ch/ UsersGuides/ PhysicsReferenceManual/ html/ index.html. Accessed May 18, 2021, 2017.
  * [14] A. W. Stevenson, J. C. Crosbie, C. J. Hall, D. Häusermann, J. Livingstone, and J. E. Lye.  Quantitative characterization of the X-ray beam at the Australian Synchrotron Imaging and Medical Beamline (IMBL).  In Journal of Synchrotron Radiation, volume 24, pages 110–141. International Union of Crystallography, 2017.
  * [15] F. Chollet et al.  Keras.  https://keras.io, 2015.
  * [16] M. Abadi, A. Agarwal, P. Barham, Brevdo E., Chen Z., Citro C., S. G., Davis A., Dean J., Devin M., Ghemawat S., Goodfellow I., Harp A., Irving G., Isard M., Jia Y., Jozefowicz R., Kaiser L., Kudlur M., Levenberg J., Man’e D., Monga R., Moore S., Murray D., Olah C., Schuster M., Shlens J., Steiner B., Sutskever I., Talwar K., Tucker P., Vanhoucke V., Vasudevan V., Vi’egas F., Vinyals O., Warden P., Wattenberg M., Wicke M., Yu Y., and Zheng X.  TensorFlow: Large-scale machine learning on heterogeneous systems, 2015\.  Software available from tensorflow.org.
  * [17] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov.  Dropout: A simple way to prevent neural networks from overfitting.  Journal of Machine Learning Research, 15:1929–1958, 2014.
  * [18] D. P. Kingma and J. L. Ba.  Adam: A method for stochastic optimization.  In 3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings. International Conference on Learning Representations, ICLR, 2015.
  * [19] C. Debus, U. Oelfke, and S. Bartzsch.  A point kernel algorithm for microbeam radiation therapy.  Physics in Medicine and Biology, 62(21):8341–8359, 2017.


# Evaluating GPT-4’s Vision Capabilities on Brazilian University Admission
Exams

Ramon Pires, Thales Sales Almeida, Hugo Abonizio, and Rodrigo Nogueira
Maritaca AI
Brazil

###### Abstract

Recent advancements in language models have showcased human-comparable
performance in academic entrance exams. However, existing studies often
overlook questions that require the integration of visual comprehension, thus
compromising the full spectrum and complexity inherent in real-world
scenarios. To address this gap, we present a comprehensive framework to
evaluate language models on entrance exams, which incorporates both textual
and visual elements. We evaluate the two most recent editions of Exame
Nacional do Ensino Médio (ENEM), the main standardized entrance examination
adopted by Brazilian universities. Our study not only reaffirms the
capabilities of GPT-4 as the state of the art for handling complex
multidisciplinary questions, but also pioneers in offering a realistic
assessment of multimodal language models on Portuguese examinations. One of
the highlights is that text captions transcribing visual content outperform
the direct use of images, suggesting that the vision model has room for
improvement. Yet, despite improvements afforded by images or captions,
mathematical questions remain a challenge for these state-of-the-art models.
The code and data used on experiments are available at
https://github.com/piresramon/gpt-4-enem.

##  I Introduction

Language models (LMs) have demonstrated remarkable performance on a variety of
natural language processing tasks [1, 2, 3, 4, 5, 6]. Recent works have
leveraged the capabilities of state-of-the-art LMs to evaluate automatic
question resolution in college entrance exams [7, 8, 9, 10], shedding light on
prospective advantages offered by LMs and their applicability in classroom
settings [11] and the evolving landscape of “precision education” [12].
Furthermore, these models are increasingly applied across a diverse array of
entrance exams spanning multidisciplinary domains, consistently exhibiting
performance levels achieved by human students [7, 13, 14, 15, 9].

However, many of these studies primarily focus on evaluating text-based
models, often rejecting questions that necessitate image understanding. Such
approaches do not provide a realistic setting, limiting the potential richness
and complexity of questions that demand a combination of both textual and
visual comprehension.

In this work, we address the diverse nature of entrance exams by incorporating
image understanding. We evaluate LMs on Exame Nacional do Ensino Médio (ENEM),
a multidisciplinary entrance exam for admission to high school education
institutes in Brazil. The ENEM exam poses a unique challenge for LMs because
it requires a deep comprehension across diverse fields of knowledge and the
capability to synthesize information from varied domains. We assess the two
latest editions of ENEM: 2022 and 2023, with the latter applied and released
to the public after the completion of the models’ training. Therefore, it is
certain that the model did not encounter this particular exam during its
training phase. For all the experiments we apply few-shot with Chain-of-
Thought (CoT)[16, 17] explanations, and evaluate the multimodal GPT-4 [7] as
well as text-based language models.

This study shows that language models can have impressive performance in
solving multidisciplinary questions using step-by-step reasoning when applied
in realistic settings with all the necessary visual and textual components.
However, the LMs exhibit relatively more difficulty in mathematics compared to
other subjects. Furthermore, our findings suggest that captions prove more
effective than images in current vision language models.

Figure 1: GPT-4V performance on the ENEM 2023 exam.

##  II Related Work

Some recent works have explored the capabilities of state-of-the-art language
models in more challenging and domain-specific tasks [13, 14, 15].

GPT-4 exhibits human-level performance on multiple professional and academic
exams, including multiple-choice and free-response questions, with images
included in the prompt for questions that required them [7]. For all free-
response questions, the images were objectively transcribed to evaluate vision
and no-vision models.

Those benchmarks, however, are all English exams. Parallel to the growing
interest in developing language models specialized in idioms [18], it becomes
urgent to propose benchmarks based on specific language examinations. Zhang et
al. [8], for example, evaluated the GPT-3.5 model on the Chinese college
entrance examination (GAOKAO), excluding questions containing images. Guillen-
Grima et al. [19] assessed the performance of GPT-3.5 and GPT-4 models in the
examination for access to Spanish Medical Residency. The authors evaluated the
2022 exam in the native language and English translations, and had a slightly
enhanced performance with the last one. Similarly, the questions requiring
image evaluations were excluded.

To bridge the gap of scarce benchmarks in Portuguese, Almeida et al. [10]
introduced the BLUEX (Brazilian Leading Universities Entrance eXams), a
multimodal dataset consisting of the entrance exams of the two leading
universities in Brazil: UNICAMP and USP. Despite being a multimodal benchmark,
to date, the authors have solely evaluated text-based LMs on examples that do
not require image comprehension.

Prior studies have evaluated the performance of NLP models on ENEM [20, 9].
Silveira et al [20] employed static Word Embeddings [21] and WordNet [22] to
evaluate the ENEM from 2009 to 2017. The results were restricted, achieving an
accuracy range of 26-29%, even filtering out all the questions that require
image understanding, mathematical reasoning, or having chemical elements, as
well as the questions related to foreign languages.

More recently, Nunes et al. [9] evaluated text-based GPT-3.5 and GPT-4 models
on ENEM 2009-2017 and ENEM 2022. The models showed satisfactory results and
became capable of solving complex problems when decomposing them into smaller
parts with CoT. However, they applied the same filtering criteria of [20],
eliminating nearly half of the questions. We are the first to provide a
realistic evaluation of multimodal language models on Portuguese benchmarks.

##  III Methodology

###  III-A Datasets

We evaluate the LMs using two datasets: ENEM 2022 and ENEM 2023. These
datasets encompass all multiple-choice questions from the last two
editions111Except the annulled questions. Each edition had one question of
Mathematics annulled.. The datasets have been created to allow the evaluation
of both textual-only and textual-visual language models. To evaluate textual-
only models, we incorporated into the datasets the textual descriptions of the
images that appear in the questions’ statements from the orange ENEM exam
booklet222A particular booklet that offers accessibility to people with visual
impairments. The transcription of images, tables, and chemical equations are
read by a reader during the exam..

The datasets were created through a multi-step process. After downloading the
booklets and respective answer sheets, both in PDF format, we use the Adobe
PDFServices API [23] to extract text, tables, and images. The extraction
process is followed by subsequent parsing procedures, and thorough manual
verification and fixings. The last step was taken to remove accessibility
patterns (e.g., replacing open quotation marks, sentence, close quotation
marks with "sentence") and converting mathematical expressions from natural
language to standardized symbolic form (e.g., replacing v is equal to zero and
a is different from zero with v $=$ 0 and a $\neq$ 0).

Different from the well-known datasets of ENEM exams [20, 9], our datasets
address several key differences. First and foremost, our datasets keeps all
the questions, including the ones that necessitate image understanding, table
interpretation, mathematical reasoning, or familiarity with chemical elements.
We also did not exclude the five questions related to the foreign language
(English or Spanish, chosen at the time of registration) and kept the English
ones. This comprehensive approach enables a more rigorous and nuanced
evaluation process of language models across diverse questions.

In addition, a distinctive characteristic of our datasets lies in the
inclusion of textual descriptions for visual elements, such as images and
complex chemical equations. This method enables the evaluation of both
multimodal and textual language models on the entrance exams.

Furthermore, our datasets incorporate table contents in Markdown formatting.
This standardized structure is easily understandable by the language models,
and because of this, we ignore the reader descriptions related to tables.

The complete dataset comprises 358 multiple-choice multidisciplinary ENEM
questions, encompassing all the textual, visual, and description information
necessary to solve the exams. Table I presents the statistics of the ENEM
datasets, including the number of questions that require image understanding
(IU).

TABLE I: Number of questions of ENEM 2022 and 2023 datasets. Area | ENEM 2022 | ENEM 2023
---|---|---
total | IU | total | IU
Languages and Codes | 45 | 9 | 45 | 6
Human Sciences | 45 | 8 | 45 | 6
Natural Sciences | 45 | 15 | 45 | 13
Mathematics | 44 | 12 | 44 | 19
Total | 179 | 44 | 179 | 44

###  III-B The OpenAI Models

In this study, we use the following OpenAI’s models available via API to
paying customers333https://platform.openai.com:

####  III-B1 GPT-3.5 Turbo

We use the most recent gpt-3.5-turbo-1106, whose training data is up to
September 2021 and supports contexts with 16,385 tokens.

####  III-B2 GPT-4

We use the latest version of GPT-4, gpt-4-0613, trained on data up to
September 2021, enabling support for contexts with 8,192 tokens.

####  III-B3 GPT-4 Turbo

For comparative analysis, we employed the most recent gpt-4-1106-preview,
trained on data up to April 2023, that can accommodate contexts with 128,000
tokens. This preview model is not currently recommended for production
traffic.

####  III-B4 GPT-4 Turbo with Vision

We also extended the investigation to multimodal models, evaluating the latest
version of GPT-4 with vision, the gpt-4-vision-preview. The model was trained
on data up to April 2023, with support for contexts of 128,000 tokens. Similar
to the GPT-4 Turbo, this preview model is not intended for production traffic
at present. We will refer to this model as GPT-4V.

For all models, we used the following parameters: temperature of 0, which
defaults to greedy decoding, maximum output of 512 tokens, frequency penalty
of 0, and presence penalty of 0.

Since the last two models were trained with data up to April 2023 and the ENEM
2022 exam was applied on November 12-20 2022, we do not discard the
possibility of data contamination. However, the results on the ENEM 2023 test
are trustworthy, given that the models could not have access to the
examination questions and answers during their training phase.

###  III-C Experiments

We adopted the prompt design that performed best in [9]: 3-shot with Chain-of-
Thought (CoT) [24]. Those prompts include three few-shot examples from
different knowledge areas of ENEM 2022, and include the solutions in
explanatory steps, decomposing complex problems into smaller ones before
compiling the final response. The Appendix A-A presents the instruction, the
few-shot examples and respective CoT explanations applied in the experiments.

We performed three experiments to analyze multiple-choice questions: without
images, with images, and with captions. Each experiment is described
hereafter.

####  III-C1 Without Images

In this experiment, we analyzed multiple-choice questions excluding any visual
elements. This approach involved disregarding all images that could be present
in both the question statement (and even in the answer alternatives). The
primary aim was to lower-bound the language models’ performance in the absence
of visual contextual cues.

####  III-C2 With Images

For this experiment, we aim to evaluate multimodal LMs by embedding the images
directly into the question body. Unlike the previous experiment, the inclusion
of visual elements aimed to test the language models’ ability to comprehend
and process information from both textual and visual modalities.

####  III-C3 With Captions

In this experiment, our focus was on assessing textual LMs by replacing images
with textual descriptions containing all essential information necessary for
understanding the question. The objective was to measure the language models’
proficiency in solving the exam when equipped with all the necessary
information. This approach resembles LENS [25], a modular approach that fed
LMs with image captions generated by vision models. The main difference herein
is that we used human-generated captions.

##  IV Results

This section presents the results for the two latest ENEM exams (2022 e 2023).
All the experiments use 3-shot with CoT, that was the best approach in
previous work [9].

###  IV-A Evaluating Multimodal Model

TABLE II: Results of GPT-4V on ENEM 2022 and ENEM 2023. Area | ENEM 2022 | ENEM 2023
---|---|---
| without images
---
| with images
---
| with captions
---
| without images
---
| with images
---
| with captions
---
Languages and Codes | 73.33 | 82.22 | 84.44 | 84.44 | 86.67 | 91.11
Human Sciences | 88.89 | 95.56 | 95.56 | 95.56 | 100.00 | 100.00
Natural Sciences | 73.33 | 77.78 | 82.22 | 86.67 | 91.11 | 93.33
Mathematics | 54.55 | 61.36 | 61.36 | 54.55 | 65.91 | 75.00
Total | 72.63 | 79.33 | 81.01 | 80.45 | 86.03 | 89.94

Table II shows the results of the OpenAI multimodal model – GPT-4 Turbo with
Vision – over the two exams. We evaluate the model in the three experiments:
without images, with images, and with captions. Both for ENEM 2022 and 2023,
even without images, the GPT-4V model performed well in all the knowledge
areas except in Mathematics (both reached 54.55%). After including the images,
the performance on ENEM 2022 in Mathematics improved to 61.36% and remained
the same when the images were replaced with captions. The ENEM 2023 had a
remarkable improvement in the same area, achieving 65.91% and 75.00% with
images and captions, respectively.

The substantial improvement in Mathematics was expected given the model’s
initial difficulty in the without-images scenario. This enhancement is
particularly noteworthy because Mathematics comprises a higher proportion of
questions demanding image understanding (43%).

Significant improvements are noticeable when incorporating either textual or
visual representations of images, with the difference nearing 10 points,
particularly when utilizing captions. This underscores the crucial role of
having access to the complete statement. On ENEM 2023, noteworthy achievements
are evident in Languages, Natural Sciences, and Human Sciences areas, all
surpassing the 90% accuracy, with special emphasis on the latter achieving a
score of 100%. Figure 1 illustrates the performance of the GPT-4V model on
ENEM 2023.

The benefit of employing captions highlights that there is room for
improvement in the encoding of images by vision models.

###  IV-B Evaluating Textual-Only Models

TABLE III: Results of textual-only LMs on ENEM 2022. Area | GPT-3.5 Turbo | GPT-4 | GPT-4 Turbo
---|---|---|---
without images | with captions | without images | with captions | without images | with captions
Languages and Codes | 68.89 | 77.78 | 75.56 | 88.89 | 73.33 | 86.67
Human Sciences | 84.44 | 88.89 | 88.89 | 95.56 | 86.67 | 95.56
Natural Sciences | 71.11 | 66.67 | 84.44 | 86.67 | 77.78 | 80.00
Mathematics | 45.45 | 40.91 | 63.64 | 68.18 | 50.00 | 56.82
Total | 67.60 | 68.72 | 78.21 | 84.92 | 72.07 | 79.89
TABLE IV: Results of textual-only LMs on ENEM 2023. Area | GPT-3.5 Turbo | GPT-4 | GPT-4 Turbo
---|---|---|---
without images | with captions | without images | with captions | without images | with captions
Languages and Codes | 80.00 | 82.22 | 84.44 | 93.33 | 84.44 | 91.11
Human Sciences | 91.11 | 93.33 | 95.56 | 100.00 | 93.33 | 100.00
Natural Sciences | 75.56 | 73.33 | 88.89 | 95.56 | 86.67 | 86.67
Mathematics | 38.64 | 45.45 | 65.91 | 72.73 | 59.09 | 65.91
Total | 71.51 | 73.74 | 83.80 | 90.50 | 81.01 | 86.03

As presented in the previous section, the use of captions is sufficient so
that the model comprehends and solves the questions. Then, this section aims
to evaluate other state-of-the-art language models that do not have the
capability of understanding images. Tables III and IV show the results of
GPT-3 Turbo, GPT-4, and GPT-4 Turbo when evaluating the multiple-choice
questions of ENEM 2022 and 2023, respectively.

Upon initial analysis of the results, intriguing patterns emerge. Firstly,
even with the inclusion of captions, the GPT-3.5 Turbo had a performance
decline in two areas for ENEM 2022 and one area for ENEM 2023, deviating from
anticipated outcomes. Secondly, across all evaluated models, considerably
superior results were achieved for ENEM 2023 compared to ENEM 2022, which
raises the possibility that the most recent test had a lower level of
difficulty.

Conversely, even though the GPT-4 Turbo was trained with more recent data
(April 2023), it exhibited the smallest discrepancy in accuracy between ENEM
2023 (never seen during training) and ENEM 2022 (potentially seen during
training). This observation challenges the contamination hypothesis, given
that the model’s performance was inferior to that of GPT-4.

Nevertheless, as expected, all models had a substantial improvement after
including captions. The most notable advancement is evident in GPT-4, which
had a 7-point enhancement, reaching 90.50% with ENEM 2023, and surpassing the
multimodal GPT-4V model.

##  V Conclusion

This study extends the applicability of language models (LMs) by incorporating
image understanding to address the diverse challenges posed by entrance exams.
Our evaluation reaffirms the GPT-4 as the state-of-the-art for tackling the
ENEM, reinforcing findings from a previous study. The multimodal GPT-4V LM
exhibits notable performance across multiple experiments, but the superiority
of the use of captions instead of images shows that there is room for
improvement. Through our analyses, we demonstrate the potential of LMs to
contribute significantly to the educational landscape, which positions them as
potential tools for enhancing educational assessments, offering perspectives
on question difficulty prediction, generating new exam items, and ultimately
helping the students in the preparation for admission exams

##  VI Future Work

For future work, we intend to include the essay in the evaluation and ask
experts to correct the generated text using the same ENEM criteria. This would
provide a more complete benchmark.

Additionally, we would like to adopt the same evaluation criteria currently
adopted for the ENEM: the Item Response Theory (IRT). Different from the
Classical Test Theory (CTC), which computes the total sum of correct answers
regardless of successful guesses, the IRT is a mathematical model developed to
make the test correction process fairer and more accurate. This will allow a
fair comparison of performance between LMs and students, checking the position
of the models on the same ruler that measures the student’s level of
knowledge.

##  VII Acknowledgments

This research was partially funded by Fundação de Amparo à Pesquisa do Estado
de São Paulo (FAPESP), project 2022/01640-2.

## References

  * [1] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186, Minneapolis, Minnesota, June 2019. Association for Computational Linguistics.
  * [2] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al.  Language models are few-shot learners.  Advances in neural information processing systems, 33:1877–1901, 2020.
  * [3] Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, Eliza Rutherford, Tom Hennigan, Jacob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Maribeth Rauh, Po-Sen Huang, Amelia Glaese, Johannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Antonia Creswell, Nat McAleese, Amy Wu, Erich Elsen, Siddhant Jayakumar, Elena Buchatskaya, David Budden, Esme Sutherland, Karen Simonyan, Michela Paganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Elena Gribovskaya, Domenic Donato, Angeliki Lazaridou, Arthur Mensch, Jean-Baptiste Lespiau, Maria Tsimpoukelli, Nikolai Grigorev, Doug Fritz, Thibault Sottiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d’Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson, Blake Hechtman, Laura Weidinger, Iason Gabriel, William Isaac, Ed Lockhart, Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Koray Kavukcuoglu, and Geoffrey Irving.  Scaling language models: Methods, analysis & insights from training gopher, 2022.
  * [4] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al.  Palm: Scaling language modeling with pathways.  arXiv preprint arXiv:2204.02311, 2022.
  * [5] Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, and Laurent Sifre.  Training compute-optimal large language models, 2022.
  * [6] Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al.  Llama: Open and efficient foundation language models.  arXiv preprint arXiv:2302.13971, 2023.
  * [7] OpenAI.  Gpt-4 technical report, 2023.
  * [8] Xiaotian Zhang, Chunyang Li, Yi Zong, Zhengyu Ying, Liang He, and Xipeng Qiu.  Evaluating the performance of large language models on gaokao benchmark, 2023.
  * [9] Desnes Nunes, Ricardo Primi, Ramon Pires, Roberto Lotufo, and Rodrigo Nogueira.  Evaluating gpt-3.5 and gpt-4 models on brazilian university admission exams, 2023.
  * [10] Thales Sales Almeida, Thiago Laitz, Giovana K. Bonás, and Rodrigo Nogueira.  Bluex: A benchmark based on brazilian leading universities entrance exams.  In Murilo C. Naldi and Reinaldo A. C. Bianchi, editors, Intelligent Systems, pages 337–347, Cham, 2023. Springer Nature Switzerland.
  * [11] Danijel Kučak, Vedran Juričić, and Goran Đambić.  Machine learning in education-a survey of current research trends.  Annals of DAAAM & Proceedings, 29, 2018.
  * [12] Hui Luan and Chin-Chung Tsai.  A review of using machine learning approaches for precision education.  Educational Technology & Society, 24(1):250–266, 2021.
  * [13] Michael James Bommarito and Daniel Martin Katz.  Gpt takes the bar exam.  Available at SSRN 4314839, 2022.
  * [14] Jillian Bommarito, Michael James Bommarito, Jessica Katz, and Daniel Martin Katz.  Gpt as Knowledge Worker: A Zero-Shot Evaluation of (AI)CPA Capabilities.  SSRN Electronic Journal, 2023.
  * [15] Karan Singhal, Shekoofeh Azizi, Tao Tu, S Sara Mahdavi, Jason Wei, Hyung Won Chung, Nathan Scales, Ajay Tanwani, Heather Cole-Lewis, Stephen Pfohl, et al.  Large language models encode clinical knowledge.  arXiv preprint arXiv:2212.13138, 2022.
  * [16] Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa.  Large language models are zero-shot reasoners.  Advances in neural information processing systems, 35:22199–22213, 2022.
  * [17] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al.  Chain-of-thought prompting elicits reasoning in large language models.  Advances in Neural Information Processing Systems, 35:24824–24837, 2022.
  * [18] Ramon Pires, Hugo Abonizio, Thales Sales Almeida, and Rodrigo Nogueira.  Sabiá: Portuguese large language models.  In Murilo C. Naldi and Reinaldo A. C. Bianchi, editors, Intelligent Systems, pages 226–240, Cham, 2023. Springer Nature Switzerland.
  * [19] Francisco Guillen-Grima, Sara Guillen-Aguinaga, Laura Guillen-Aguinaga, Rosa Alas-Brun, Luc Onambele, Wilfrido Ortega, Rocio Montejo, Enrique Aguinaga-Ontoso, Paul Barach, and Ines Aguinaga-Ontoso.  Evaluating chatgpt efficacy in navigating the spanish medical residency entrance examination (mir): A new horizon for ai in clinical medicine.  Preprints, September 2023.
  * [20] I. Cataneo Silveira and D. Deratani Maua.  Advances in automatically solving the enem.  In 2018 7th Brazilian Conference on Intelligent Systems (BRACIS), pages 43–48, Los Alamitos, CA, USA, oct 2018. IEEE Computer Society.
  * [21] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean.  Efficient Estimation of Word Representations in Vector Space.
  * [22] George A. Miller.  Wordnet: A lexical database for english.  Commun. ACM, 38(11):39–41, nov 1995.
  * [23] Adobe.  Adobe pdf services python sdk.  https://github.com/adobe/pdfservices-python-sdk, 2023.
  * [24] Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou.  Chain of thought prompting elicits reasoning in large language models.  CoRR, abs/2201.11903, 2022.
  * [25] William Berrios, Gautam Mittal, Tristan Thrush, Douwe Kiela, and Amanpreet Singh.  Towards language models that can see: Computer vision through the lens of natural language, 2023.

##  Appendix A Appendices

###  A-A Few-shot examples and CoT Explanations

This work uses the same few-shot examples and respective explanations used in
[9]. The examples, as well as the instruction, are illustrated in Figure 2.
The instruction and few-shot examples were translated into English to
facilitate the understanding of the method, but we use it in Portuguese when
evaluating the models.

Formulate a step-by-step explanation that allows you to answer the multiple-
choice question below. Only one alternative is correct. Desired format: point
out the alternatives that make sense, choose the CORRECT alternative and
justify it, and finish justifying why the other alternatives are incorrect.
Finish the explanation with “Answer:” followed by the alternative.
---
Question 1:
Emotional urgency. If everything is for yesterday, if life engages a first
gear and takes off, if there is no more time for strategic stops, we fatally
fall into the addiction of wanting love to be equally resolved in a split
second. We are in a hurry to hear “I love you". We can’t wait for the rules of
coexistence to be established: are we boyfriends, hookups, married, lovers?
Emotional urgency. A trap. We associate several words with LOVE: passion,
romance, sex, adrenaline, palpitation. We forget, however, the word that makes
this feeling possible: “patience". Love without patience does not avenge. Love
cannot be chewed and swallowed with emergency, desperate hunger. It’s a meal
that can last a lifetime. MEDEIROS, M. Available at:
http://porumavidasimples.blogspot.com.br. Accessed on: 20 Aug. 2017 (adapted).
In this opinion text, the linguistic marks reveal a relaxed situation with
little formality, which is evidenced by the
A. impersonalization throughout the text, as in: “if there is no more time”.
B. construction of an atmosphere of urgency, in words like: “hurry”.
C. repetition of a certain syntactic structure, as in: “If everything is for
yesterday”.
D. emphasis on the use of hyperbole, as in: “a meal that can last a lifetime”.
E. use of metaphors, as in: “life engages a first gear and takes off”.
Explanation: The text is written in a light, agile language, with little
formality. In addition, it has figures of speech, such as metaphors and
hyperboles, which are not mutually exclusive. In a sequential analysis of the
alternatives, it would be possible to affirm that D) and E) are correct.
However, looking in detail, it is noted that the expression “use of metaphors”
proves to be more appropriate than “emphasis on the use of hyperbole”, since,
in order to state that the use of hyperbole was emphasized, the figure of
speech should have appeared more often. This makes option E) more likely to be
CORRECT. In addition, impersonality should not be pointed out as a mark of low
formality. There is also an atmosphere of urgency, but that is criticized in
the text that highlights the importance of patience and not haste. Finally,
the syntactic structure is not systematically repeated throughout the text.
Answer: E.
Question 2:
Whenever the relevance of discourse comes into play, the issue becomes
political by definition, as it is the discourse that makes man a political
being. And everything men do, know or experience only makes sense to the
extent that it can be discussed. There will perhaps be truths that lie beyond
language and that may be of great relevance to man in the singular, that is,
to man who, whatever he may be, is not a political being. But men in the
plural, that is, the men who live and move and act in this world, can only
experience the meaning of things by being able to speak and be intelligible to
each other and to themselves. ARENDT, H. The human condition. Rio de Janeiro:
University Forensics, 2004.
In the excerpt, the philosopher Hannah Arendt shows the importance of language
in the process of
A. understanding of culture.
B. increased creativity.
C. perception of individuality.
D. improvement of technique.
E. construction of sociability.
Explanation: Hannah Arendt argues in her work that we are political beings, in
the proper sense of living in a polis, in a collective and social environment.
And this sociability is only possible through discourse, language. Thus, we
can conclude that language is an important tool for building sociability, and
therefore alternative E) is CORRECT. Furthermore, it is not about
understanding the culture, but the social relationship between people of that
culture. Hannah also doesn’t talk about increased creativity, nor does she
talk about technique. Finally, language is used in something more collective
and social, just the opposite of individuality. Answer: E.
Question 3:
A couple plans to build a swimming pool in the shape of a rectangular
parallelepiped with a capacity of 90,000 L of water on their farm. The couple
hired a construction company that presented five projects with different
combinations of the internal dimensions of depth, width and length. The pool
to be built will have the same ceramic coating on its walls and bottom, and
the couple will choose the project that requires the smallest coating area.
The internal dimensions of depth, width and length, respectively, for each of
the projects are: project I: 1.8 m, 2.0 m and 25.0 m; project II: 2.0 m, 5.0 m
and 9.0 m; project III: 1.0 m, 6.0 m and 15.0 m; project IV: 1.5 m, 15.0 m and
4.0 m; project V: 2.5 m, 3.0 m and 12.0 m.
The project that the couple should choose will be the
A. I.
B. II.
C. III.
D. IV.
E. V.
Explanation: We must calculate the area of the four side faces and the area of
the lower base (bottom of the pool) and add these areas to obtain the coating
area. Therefore, calculating the coating area of each project, we have:
Project I: A = 2 x 25 + 2 x 1.8 x (2 + 25) = 147.2; Project II: A = 9 x 5 + 2
x 2 x (9 + 5) = 101; Project III: A = 15 x 6 + 2 x 1 x (15 + 6) = 132; Project
IV: A = 4 x 15 + 2 x 1.5 x (15 + 4) = 117; Project V: A = 3 x 12 + 2 x 2.5 x
(3 + 12) = 111. Therefore, the project with the smallest coating area is the
project II, therefore the correct answer is B. Answer: B.
Figure 2: Questions from the ENEM 2022 exam used as few-shot examples, and
respective CoT explanations created from contents of high-school teachers’
discussions.