# Diagram of High-Energy Nuclear Collisions111This paper is based on work
performed before February 24th 2022.

Evgeny Andronov  St. Petersburg State University, St. Petersburg, Russia
Magdalena Kuich  University of Warsaw, Warsaw, Poland  Marek Gazdzicki
Geothe-University Frankfurt am Main, Germany  Jan Kochanowski University,
Kielce, Poland

###### Abstract

Many new particles, mostly hadrons, are produced in high-energy collisions
between atomic nuclei. The most popular models describing the hadron-
production process are based on the creation, evolution and decay of
resonances, strings or quark–gluon plasma. The validity of these models is
under vivid discussion, and it seems that a common framework for this
discussion is missing. Here, for the first time, we explicitly introduce the
diagram of high-energy nuclear collisions, where domains of the dominance of
different hadron-production processes in the space of laboratory-controlled
parameters, the collision energy and nuclear-mass number of colliding nuclei
are indicated. We argue that the recent experimental results suggest the
location of boundaries between the domains, allowing for the first time to
sketch an example diagram. Finally, we discuss the immediate implications for
experimental measurements and model development following the proposed sketch
of the diagram.

high-energy collisions, strongly interacting matter, quark-gluon plasma,
strings, resonances

###### pacs:

25.75.q, 25.75.Nq, 24.60.Ky

##  I Introduction

One of the crucial issues of contemporary physics is understanding strong
interactions - the interactions defining properties of atomic nuclei and
collisions between them. Nuclear collisions at high energies lead to the
production of many new particles, predominately strongly interacting hadrons.
With the advent of the quark model of hadrons and the development of the
commonly accepted theory of strong interactions, quantum chromodynamics (QCD)
naturally led to expectations that matter at very high densities may exist in
a state of quasi-free quarks and gluons, the quark–gluon plasma (QGP)
Shuryak:1980tp . There are numerous indications that QGP is created in heavy-
ion collisions at high energies; for review, see Refs. Heinz:2000bk ;
STAR:2005gfr ; Gazdzicki:2010iv ; Gazdzicki:2020jte .

The theoretical description of high-energy nuclear collision is not an easy
task. This may be attributed to the difficulty of obtaining unique and
quantitative predictions from QCD. In particular, even the formation of QGP in
heavy-ion collisions is beyond the predictability of QCD. Consequently, the
bulk properties of high-energy nuclear collisions are described by
phenomenological models. Over time, three classes of them gained in
popularity:

  1. (i)

One postulates that the final hadronic state emerges from the quark–gluon
plasma’s creation, evolution, and hadronisation Florkowski:2010zz . A key
input - the QGP equation of state - can be estimated using lattice-QCD
calculations Karsch:2001cy . This process will be labelled as _QGP_ ;

  2. (ii)

One assumes hadrons originate from the formation, evolution and fragmentation
of strings - the gluon fields between a pair of colour charges forming a
narrow flux tube Werner:1993uh . Strings are typically oriented along the
collision axis, and they have a continuous masses spectrum. Symmetries and
experimental results are used to determine model parameters. This process will
be labelled as _strings_ ;

  3. (iii)

One describes the production of final state hadrons by creation, evolution and
decay of hadronic resonances Bleicher:1999xi  \- excited states of stable
hadrons. Resonances do not have a preferred elongation direction and have a
discrete mass spectrum. Experimental results are used to determine model
parameters. This hadron-production process will be labelled as _resonances_.

Do the processes reflect reality? If yes, what are the domains of their
applicability? Answering these questions is directly related to understanding
intriguing changes of hadron-production properties observed experimentally by
varying collision energy and the mass number of colliding nuclei. This task
goes hand in hand with selecting measurable quantities sensitive to a
transition between the processes. In this paper, we focus the discussion on
the ratio of positively charged kaons and pions measured at mid-rapidity, the
$K^{+}/\pi^{+}$ ratio. This measure can be interpreted as a good approximation
of the strange to non-strange quarks ratio. Due to mass and number differences
between strange and non-strange particles (quarks and gluons or hadrons), the
ratio is expected to be sensitive to the hadron-production process
Rafelski:1982pu ; Gazdzicki:1998vd  \- it is expected to be sensitive to a
changeover between different processes. With the above and the availability of
the rich experimental data, the choice of the $K^{+}/\pi^{+}$ ratio as the
subject of this paper was most suitable.

For a quantitative comparison of the experimental results with model
predictions, we selected PHSD Cassing:2008sv ; Cassing:2009vt  and SMASH
Weil:2016zrk ; Mohs:2019iee  models. This is motivated by their important
features. Both models give predictions in the full range collision energy and
masses of the colliding nuclei covered by the experimental data. The SMASH
model includes resonances and strings, whereas the PHSD model also includes
QGP.

We review the experimental results and suggest the first answers to the
questions asked in Section II. Section III introduces the diagram of high-
energy nuclear collisions, and we summarise our findings in a diagram sketch.
Finally, we discuss the implications following the sketch for experimental
measurements and developing models.

##  II Guiding Ideas and Experimental Results

Heavy-ion collisions.

The richest experimental results on the collision energy dependence of hadron-
production properties concern collisions between two heavy atomic nuclei,
Pb+Pb and Au+Au collisions. Over the last 40 years, they were recorded in the
hunt for QGP and the energy threshold of its creation - the _onset of
deconfinement_. Many fixed-target and collider experiments in the US (Lawrence
Berkeley Laboratory, LBL, and Brookhaven National Laboratory, BNL) and
European (European Organization of Nuclear Research, CERN and Helmholtz Centre
for Heavy Ion Research, GSI) laboratories have been conducting the
measurements. The results are consistent with the onset of deconfinement being
located at ($\sqrt{s_{NN}}\approx 8$ GeV) and the QGP being created at the
early stage of heavy-ion collisions at higher collision energies (for review,
see Refs. Heinz:2000bk ; STAR:2005gfr ; Gazdzicki:2010iv ; Gazdzicki:2020jte
). The most popular plot illustrating this assessment is presented in Figure 1
(left). It shows the collision energy dependence of the $K^{+}/\pi^{+}$ ratio
in central heavy-ion collisions. The ratio shows the so-called _horn_
structure. Following a fast rise, the ratio passes through a maximum in the
CERN SPS energy range, at approximately 8 GeV, then decreases and settles to a
plateau which continues up to the CERN LHC energies. Kaons are the lightest
strange hadrons, and due to approximate isospin symmetry, the $K^{+}$ yield
counts about half of the strange quarks produced in the collisions and
contained in the reaction products Gazdzicki:1998vd . Thus, Figure 1 (left)
demonstrates that the fraction of strangeness carrying particles in the
produced matter passes through a sharp maximum at the SPS energy range in
central heavy-ion collisions; for a detailed explanation; see Ref.
Gazdzicki:2010iv .

The standard modelling of heavy-ion collisions Florkowski:2010zz  includes the
formation of high-density matter (be it QGP or hadronic matter) at the early
stage of a collision, its expansion and the decoupling of hadrons that freely
stream to particle detectors. A statistical description of the early stage
Gazdzicki:1998vd  led to predictions of the collision energy dependence of
bulk hadron production properties. In particular, the horn structure was
predicted as the signal of the onset of deconfinement. In the model, it
reflects the decrease in the ratio of strange to non-strange degrees of
freedom when deconfinement sets in. Experimental data are compared with
calculations of the PHSD model Cassing:2008sv ; Cassing:2009vt  that
incorporates the QGP creation at sufficiently high densities and chiral-
symmetry restoration in the dense hadronic matter. The model catches the basic
properties of the data; see Figure 1 (left). This further supports
interpreting the horn maximum at $\sqrt{s_{NN}}\approx 8$ GeV as the beginning
of the QGP creation. Moreover, the SMASH model Weil:2016zrk ; Mohs:2019iee ,
which does not include the QGP creation qualitatively, fails to reproduce the
results; see Figure 1 (left). One should, however, note that there are
significant uncertainties in modelling both production processes; see below
for an example.

Figure 1: Collision energy dependence of the $K^{+}/\pi^{+}$ multiplicity
ratio at mid-rapidity in central heavy-ion collisions (Pb+Pb NA49:2007stj ;
NA49:2002pzu ; PhysRevC.88.044910 ; ALICE:2019hno  and Au+Au E866:1999ktz ;
E0895:2003oas ; STAR:2017sal ; STAR:2019vcp ; STAR:2008med ; HADES:2018adm ;
HADES:2020ver ) (left) and in inelastic _p+p_ interactions NA61SHINE:2017fne ;
PHENIX:2011rvu ; ALICE:2011gmo ; ALICE:2015ial ; ALICE:2019hno ; ALICE:2020jsh
(right). Open cross points present the PHSD Cassing:2008sv ; Cassing:2009vt
predictions, while open circles - the SMASH Weil:2016zrk ; Mohs:2019iee
predictions. Lines connecting the points are plotted to guide the eye.

Proton–proton interactions.

Measurements of proton–proton interactions started long before the first
experiments studying heavy-ion collisions. The primary goal of the study of
_p+p_ interactions was understanding strong interactions. With increasing
collision energy, more and heavier hadrons have been produced. To understand
the early results, the string model was invented Veneziano:1968yb . The
experimental results also contributed to the formulation of QCD - the nowadays
commonly accepted theory of strong interactions. While there is no first-
principles derivation of strings from QCD, some properties of a string can be
derived from QCD.

Paradoxically, QCD demotivated studies of bulk properties of _p+p_
interactions. This is because of difficulties in obtaining unique and
quantitative predictions from QCD. Consequently, the world data on _p+p_
interactions are not as rich as the corresponding results on heavy-ion
collisions. The compiled data on the $K^{+}/\pi^{+}$ ratio at mid-rapidity in
inelastic _p+p_ interactions is shown in Figure 1 (right). Precise
measurements are available at the CERN SPS and LHC energies. The _p+p_ results
still allow for important observations:

  1. (i)

At the SPS energies, the ratio in _p+p_ interactions is about a factor of two
lower than in heavy-ion collisions;

  2. (ii)

At $\sqrt{s_{NN}}\approx 8$ GeV, a  _break_ in the collision energy dependence
of the ratio is observed in _p+p_ interactions instead of the horn seen in
heavy-ion collisions. For a more detailed analysis of the _p+p_ break, see
Ref. NA61SHINE:2019xkb ;

  3. (iii)

At the LHC energies, the  _p+p_ ratio is about 20% lower than the heavy-ion
one.

The most popular modelling of proton–proton interactions at high energies
includes strings’ formation, evolution, and fragmentation. The widely used
approaches are the Lund Andersson:1983ia , EPOS Werner:1993uh  and Dual Parton
Kaidalov:1982xe ; Capella:1992yb  models. At low collision energies, the
validity of the string approach breaks, and one replaces it with the creation
of resonances and their decay; for a detailed explanation, see Ref.
Mohs:2019iee . These two processes are implemented in the PHSD Cassing:2008sv
; Cassing:2009vt  and SMASH Weil:2016zrk ; Mohs:2019iee  models.

Their predictions for the collision energy dependence of the $K^{+}/\pi^{+}$
ratio in _p+p_ interactions are shown in Figure 1 (right). Significant
differences between them shed light on the uncertainty of the predictions.
Taking into account this uncertainty, one concludes that the models reproduce
the bulk properties of the data.

The effect of the changeover from resonances to strings (_onset of strings_)
was studied in detail within the UrQMD model Bass:1998ca ; Vovchenko:2014vda .
Within SMASH Weil:2016zrk ; Mohs:2019iee , the changeover causes a wiggle in
the collision energy dependence of the $K^{+}/\pi^{+}$ ratio, which can be
seen in Figure 1 (right) by enlarging the plot. In PHSD Cassing:2008sv ;
Cassing:2009vt , a sharp transition is located at $\sqrt{s_{NN}}\approx 2.6$
GeV - close to the threshold for kaon production—and thus its effect on the
ratio is hard to observe.

The open question discussed in Ref. NA61SHINE:2019xkb  is whether the break
(ii) in the collision energy dependence of the experimental ratio at
$\sqrt{s_{NN}}\approx 8$ GeV is due to the onset of strings or is related to
the onset of deconfinement.

One notes the following regarding the similarity of the ratio in _p+p_ and
Pb+Pb collisions at LHC (iii). It was reported that relative strange hadron
yields in _p+p_ interactions at LHC smoothly increase with increasing charged-
particle multiplicity and for high multiplicity interactions are close to
those in Pb+Pb collisions ALICE:2017jyt . Moreover, recent LHC data on the
azimuthal angle distribution of charged particles in high multiplicity _p+p_
interactions CMS:2012qk ; Khachatryan:2016txc ; Nagle:2018nvi  show
anisotropies up to the recently observed only in heavy-ion collisions and
attributed to the hydrodynamical expansion of matter Bozek:2016acp . This
suggests that QGP may also be produced in _p+p_ interactions at the LHC
energies, at least in collisions with sufficiently high hadron multiplicity.

Collisions of intermediate-mass nuclei.

The collision-energy dependence of hadron-production properties in collisions
of intermediate-mass nuclei is the least established one. The only systematic
measurements have been performed at the CERN SPS by NA61/SHINE Abgrall:2014xwa
.

They were motivated by a search for the critical point of strongly interacting
matter and a need to establish the nuclear mass dependence of the horn
structure Gazdzicki:995681 . The data on collisions of intermediate-mass
nuclei are summarised in Figure 2. The results of $K^{+}/\pi^{+}$ ratio for
central Pb+Pb/Au+Au and inelastic _p+p_ are also plotted for comparison in a
light colour. The main observations are:

  1. (i)

The ratio in Be+Be collisions is similar to the one in _p+p_ interactions in
the whole SPS energy range;

  2. (ii)

There is no horn structure in Ar+Sc collisions;

  3. (iii)

The ratio in Ar+Sc collisions at the top SPS energy is similar to the one in
Pb+Pb collisions.

Figure 2: Status of experimental results on energy dependence of the
$K^{+}/\pi^{+}$ ratio at mid-rapidity in high-energy nuclear collisions. In
addition to the previously shown results on central heavy-ion collisions
NA49:2007stj ; NA49:2002pzu ; PhysRevC.88.044910 ; ALICE:2019hno ;
E866:1999ktz ; E0895:2003oas ; STAR:2017sal ; STAR:2019vcp ; STAR:2008med ;
HADES:2018adm ; HADES:2020ver  and inelastic _p+p_ NA61SHINE:2017fne ;
PHENIX:2011rvu ; ALICE:2011gmo ; ALICE:2015ial ; ALICE:2019hno ; ALICE:2020jsh
interactions, results on central Be+Be NA61SHINE:2020czq , C+C NA49:2004jzr ;
NA49:2012rsi , Si+Al E-802:1994hea , Si+Si NA49:2004jzr ; NA49:2012rsi , Ar+Sc
(preliminary) Lewicki:2020mqr ; NA61SHINE:2021nye , Ni+Ni FOPI:1997rgs ;
FOPI:1997tif  and Xe+Xe ALICE:2021lsv  collisions are shown. Points presenting
Pb+Pb/Au+Au and _p+p_ are plotted in pale colours to emphasize intermediate-
mass nuclei results.

Figure 3 shows results on the $K^{+}/\pi^{+}$ ratio measured at
$\sqrt{s_{NN}}\approx 7.7$ GeV (the left plot) and $\sqrt{s_{NN}}\approx 17$
GeV (the right plot) as a function of the mean number of nucleons that
participated in inelastic interactions, the so-called number of wounded
nucleons $\langle W\rangle$. Since,, at high collision energies, the ratio is
weakly dependent on the collision energy, results from central Au+Au
collisions at $\sqrt{s_{NN}}=19.6$ GeV were also included in the plot.

Figure 3:  The $K^{+}/\pi^{+}$ ratio at mid-rapidity measured at
$\sqrt{s_{NN}}\approx 7.7$ GeV (left) and $\sqrt{s_{NN}}\approx 17$ GeV
(right) as a function of a mean number of wounded nucleons, $\langle
W\rangle$, in inelastic p+p interactions NA61SHINE:2017fne  and central Be+Be
NA61SHINE:2020czq , C+C NA49:2004jzr , Si+Si NA49:2004jzr , Ar+Sc
(preliminary) Lewicki:2020mqr ; NA61SHINE:2021nye , Au+Au STAR:2017sal , Pb+Pb
NA49:2007stj ; NA49:2002pzu  collisions. Experimental results were compared
with the PHSD Cassing:2008sv ; Cassing:2009vt  (open crosses) and SMASH
Weil:2016zrk ; Mohs:2019iee  (open circles) predictions. Lines are plotted to
guide the eye.

Let us start a discussion of model predictions concerning the system-size
dependence of the ratio from the string models. For simplicity of the
arguments, we assume that the string formation, evolution and fragmentation is
independent of $\langle W\rangle$. The yields of $K^{+}$ and $\pi^{+}$ mesons
are proportional to the mean number of strings, and consequently, their ratio
is independent of the mean number of strings, and thus it is independent of
$\langle W\rangle$. The predictions of the SMASH model shown in Figure 3
approximately follow this naive expectation. The model qualitatively fails to
reproduce the data.

A different system-size dependence is predicted within statistical models of
nucleus–nucleus collisions. The strangeness conservation imposed on the whole
system leads to a fast increase of the ratio with increasing system size to
its upper limit given by the grand-canonical-ensemble approximation. The
effect is referred to as canonical strangeness suppression and has been
extensively studied since 1980; see, e.g., Refs. Rafelski:1980gk ;
Redlich:1979bf ; Becattini:2003ft . The PHSD model predictions shown in Figure
3 show a gradual ratio increase with $\langle W\rangle$. However, in this
model, the change is likely to be also caused by smoothly increasing
contributions from QGP and chiral symmetry restoration. The PHSD model
describes the main properties of the data significantly better. However, it
fails to reproduce the _jump_ between the results for _p+p_ and Be+Be
collisions and the results for heavier nuclei at $\sqrt{s_{NN}}\approx 17$
GeV; see Figure 3 (right).

With increasing collision energy and nuclear mass number of colliding nuclei,
the number of produced strings and their density is expected to increase. The
idea that, at sufficiently high densities, the strings would be close enough
to interact and change their properties has been developing over the last 40
years. Many approaches have been proposed, in particular, colour ropes
Biro:1984cf , string fusion Braun:1991dg ; Braun:1992ss ; Braun:2014ica ;
Braun:2015eoa ; Ramirez:2020vne , core formation Werner:2007bf , string
melting Lin:2004en  and percolation Digal:2003sg ; Hohne:2005ks . A model that
explicitly involves the rapid string-QGP changeover was proposed recently. It
is a string collapse pictured as the black hole formation using the AdS/CFT
duality Kalaydzhyan:2014zqa ; Kalaydzhyan:2014tfa ; Iatrakis:2015rga . Thus,
it is natural to interpret the jump as due to a rapid changeover from strings
to QGP. This changeover is called the _onset of QGP fireball_.

The gradual increase of the ratio at low collision energies (see Figure 3
(left)) is also not reproduced by the models. This can be due to the

  1. (i)

Approaching equilibrium with increasing system size and evolution time;

  2. (ii)

Weakening of the canonical strangeness suppression with increasing system
size;

  3. (iii)

Increasing role of chiral symmetry restoration in dense hadronic matter.

##  III Diagram of High-Energy Nuclear Collisions

Here, for the first time, we explicitly introduce a concept of the diagram of
high-energy nuclear collisions and then, based on the experimental data and
ideas discussed above, sketch its example version.

The diagram of high-energy nuclear collisions is defined as a plot showing
domains of the dominance of different hadron-production processes in high-
energy nuclear collisions. The domains are indicated in the space of
laboratory-controlled parameters, the collision energy and the nuclear-mass
number of colliding nuclei. For simplicity, we consider only central
nucleus–nucleus collisions - collisions in which a large fraction of nucleons
participated in inelastic interactions ($\langle W\rangle/$A $\approx 1$).

To sketch the example diagram, the hadron-production processes discussed above
are selected:

  1. (i)

Creation, evolution and decay of resonances;

  2. (ii)

Formation, evolution and fragmentation of strings;

  3. (iii)

Creation, evolution and hadronisation of QGP.

In addition, based on the discussion of the experimental results presented in
the previous section, we assume that

  1. (i)

The Pb+Pb horn locates the resonances–QGP changeover at $\sqrt{s_{NN}}\approx
8$ GeV;

  2. (ii)

The _p+p_ break locates the resonances–strings changeover at
$\sqrt{s_{NN}}\approx 8$ GeV;

  3. (iii)

The jump between _p+p_ /Be+Be and Ar+Sc/Pb+Pb plateaus locates the strings–QGP
changeover at $\sqrt{s_{NN}}\approx 17$ GeV;

  4. (iv)

The LHC _p+p_ data imply QGP creation in (high multiplicity) _p+p_
interactions at sufficiently high (order of 1 TeV) energies.

The diagram of high-energy nuclear collisions following these assumptions is
sketched in Figure 4.

Figure 4: Schematic diagram of high-energy nuclear collisions outlined in
colliding nuclei mass number, A, and collision energy, $\sqrt{s_{NN}}$
variables. Domains in which hadron production is dominated by the creation,
evolution and decay of resonances, strings and quark–gluon plasma are
indicated as _resonances_ , _strings_ and _QGP_ , respectively, while thick
coloured lines show the changeover regions between the domains.

Two comments are in order here.

  1. (i)

The changeover resonances–strings and resonances–QGP are located at similar
collision energies ($\approx$8 GeV/c). This suggests that the resonances–QGP
changeover is driven by the resonances–strings one. At high masses of
colliding nuclei, strings produced above at the resonances–strings changeover
would have density exceeding the strings–QGP changeover. Thus the string
domain disappears, and one observes direct resonances–QGP changeover. This
locates the resonances–QGP changeover at the energy of the resonances–strings
one.

  2. (ii)

It is interesting to consider other diagrams of high-energy collisions. Here,
we discuss a simple example of the hadron–resonance gas diagram. Hagedorn’s
early papers postulated that hadrons in high-energy collisions are produced
according to statistical thermodynamics Hagedorn:1965st . Thus, following
Hagedorn’s postulate, the diagram would include only one production process -
the statistical-thermodynamical production, with Hagedorn’s temperature
$T_{H}\approx 150$ MeV. This model is clearly in contradiction with the
experimental results, as it predicts the $K^{+}/\pi^{+}$ ratio to be
independent of energy and nuclear mass number of colliding nuclei. Over the
years, the simple Hagedorn approach evolved into many models that are much
more flexible in fitting the data; for a recent review, see Ref.
Andronic:2017pug . In particular, it has been popular to fit mean hadron
multiplicities, which include multiplicities of kaons and pions, assuming that
a hadron gas in equilibrium is created at high-energy collisions. The
temperature, the baryon chemical potential, and the gas volume are free
parameters of the model and are fitted to the data from each reaction
separately. The model cannot predict the energy and nuclear mass dependence of
hadron production in this formulation. Thus, it is unsuitable for the diagram
construction.

To verify the assumptions and the diagram sketched on Figure 4, further
analysis of the existing data and new experimental measurements as well as the
development of models is needed. Concerning modelling, there is a need for the
development of dynamical models that include all three production processes.
In this paper, these models are represented by PHSD Cassing:2008sv ;
Cassing:2009vt  which reproduces experimental results significantly better
than the SMASH model Weil:2016zrk ; Mohs:2019iee . The latter includes only
two approaches to hadron production, resonances and strings. Still, the PHSD
model misses important features of the experimental data shown in Figures 1
and 3. One must reconsider the nature of the changeover between different
processes to improve predictions.

Concerning the further analysis of the existing data, one should extend the
presented studies to other quantities which characterise hadron production in
high-energy nuclear collisions. In particular, quantities sensitive to the
collective flow of matter, radial and anisotropic should be sensitive to the
production mechanisms discussed. This important study goes beyond the scope of
this introductory paper.

Finally, concerning the new experimental measurements, data on light and
medium mass nuclei collisions are needed—in particular, a precision system-
size dependence to locate the strings–QGP changeover. Such a study was
launched by NA61/SHINE at the CERN SPS, and its continuation is considered in
the following years Fields:2739340 ; Gazdzicki:2810689 . It would be important
to perform the corresponding measurements in the full range of available
energies, from the FAIR SIS-100 through NICA and SPS to CERN LHC energies. In
2024, a beam of oxygen ions is considered at the SPS and LHC in CERN
Brewer:2021kiv ; Gazdzicki:2810689 , making a good start for further study.
Prospects of studies with the intermediate-mass nuclear beams (e.g., Ar+Ar or
Kr+Kr) at LHC energies are also vividly discussed Citron:2018lsq .

###### Acknowledgements.

The authors thank Mark Gorenstein and Edward Shuryak as well as members of the
NA61/SHINE collaboration, in particular, Tomek Matulewicz, Andrzej Rybicki and
Peter Seyboth for discussion and comments. The authors thank Elena
Bratkovskaya, Viktar Kireyeu and Justin Mohs who helped in the calculation of
predictions of the PHSD and SMASH models.

## References

  * (1) E. V. Shuryak, “Quantum Chromodynamics and the Theory of Superdense Matter,” Phys. Rept., vol. 61, pp. 71–158, 1980.
  * (2) U. W. Heinz and M. Jacob, “Evidence for a new state of matter: An Assessment of the results from the CERN lead beam program.” arXiv:nucl-th/0002042, 2000\.
  * (3) J. Adams et al., “Experimental and theoretical challenges in the search for the quark gluon plasma: The STAR Collaboration’s critical assessment of the evidence from RHIC collisions,” Nucl. Phys. A, vol. 757, pp. 102–183, 2005.
  * (4) M. Gazdzicki, M. Gorenstein, and P. Seyboth, “Onset of deconfinement in nucleus-nucleus collisions: Review for pedestrians and experts,” Acta Phys. Polon. B, vol. 42, pp. 307–351, 2011.
  * (5) M. Gazdzicki, M. Gorenstein, and P. Seyboth, “Brief history of the search for critical structures in heavy-ion collisions,” Acta Phys. Polon. B, 4 2020\.
  * (6) W. Florkowski, Phenomenology of Ultra-Relativistic Heavy-Ion Collisions.  World Scientific, 3 2010.
  * (7) F. Karsch, “Lattice QCD at high temperature and density,” Lect. Notes Phys., vol. 583, pp. 209–249, 2002.
  * (8) K. Werner, “Strings, pomerons, and the venus model of hadronic interactions at ultrarelativistic energies,” Phys. Rept., vol. 232, pp. 87–299, 1993\.
  * (9) M. Bleicher et al., “Relativistic hadron hadron collisions in the ultrarelativistic quantum molecular dynamics model,” J. Phys. G, vol. 25, pp. 1859–1896, 1999.
  * (10) J. Rafelski and B. Muller, “Strangeness Production in the Quark - Gluon Plasma,” Phys. Rev. Lett., vol. 48, p. 1066, 1982.  [Erratum: Phys.Rev.Lett. 56, 2334 (1986)].
  * (11) M. Gazdzicki and M. I. Gorenstein, “On the early stage of nucleus-nucleus collisions,” Acta Phys. Polon. B, vol. 30, p. 2705, 1999.
  * (12) W. Cassing and E. L. Bratkovskaya, “Parton transport and hadronization from the dynamical quasiparticle point of view,” Phys. Rev. C, vol. 78, p. 034919, 2008.
  * (13) W. Cassing and E. L. Bratkovskaya, “Parton-Hadron-String Dynamics: an off-shell transport approach for relativistic energies,” Nucl. Phys. A, vol. 831, pp. 215–242, 2009.
  * (14) J. Weil et al., “Particle production and equilibrium properties within a new hadron transport approach for heavy-ion collisions,” Phys. Rev. C, vol. 94, no. 5, p. 054905, 2016.
  * (15) J. Mohs, S. Ryu, and H. Elfner, “Particle Production via Strings and Baryon Stopping within a Hadronic Transport Approach,” J. Phys. G, vol. 47, no. 6, p. 065101, 2020.
  * (16) C. Alt et al., “Pion and kaon production in central Pb + Pb collisions at 20-A and 30-A-GeV: Evidence for the onset of deconfinement,” Phys. Rev. C, vol. 77, p. 024903, 2008.
  * (17) S. V. Afanasiev et al., “Energy dependence of pion and kaon production in central Pb + Pb collisions,” Phys. Rev. C, vol. 66, p. 054902, 2002\.
  * (18) B. Abelev et al., “Centrality dependence of $\pi$, $k$, and $p$ production in pb-pb collisions at $\sqrt{{s}_{NN}}=2.76$ tev,” Phys. Rev. C, vol. 88, p. 044910, Oct 2013.
  * (19) S. Acharya et al., “Production of charged pions, kaons, and (anti-)protons in Pb-Pb and inelastic $pp$ collisions at $\sqrt{s_{NN}}$ = 5.02 TeV,” Phys. Rev. C, vol. 101, no. 4, p. 044907, 2020.
  * (20) L. Ahle et al., “Excitation function of K+ and pi+ production in Au + Au reactions at 2/A-GeV to 10/A-GeV,” Phys. Lett. B, vol. 476, pp. 1–8, 2000.
  * (21) J. L. Klay et al., “Charged pion production in 2 to 8 AGeV central Au+Au collisions,” Phys. Rev. C, vol. 68, p. 054905, 2003.
  * (22) L. Adamczyk et al., “Bulk Properties of the Medium Produced in Relativistic Heavy-Ion Collisions from the Beam Energy Scan Program,” Phys. Rev. C, vol. 96, no. 4, p. 044904, 2017.
  * (23) J. Adam et al., “Bulk properties of the system formed in $Au+Au$ collisions at $\sqrt{s_{\mathrm{NN}}}$ =14.5 GeV at the BNL STAR detector,” Phys. Rev. C, vol. 101, no. 2, p. 024905, 2020.
  * (24) B. I. Abelev et al., “Systematic Measurements of Identified Particle Spectra in $pp,d^{+}$ Au and Au+Au Collisions from STAR,” Phys. Rev. C, vol. 79, p. 034909, 2009.
  * (25) J. Adamczewski-Musch et al., “Deep sub-threshold $\phi$ production in au+au collisions,” Physics Letters B, vol. 778, pp. 403–407, 2018.
  * (26) J. Adamczewski-Musch et al., “Charged-pion production in Au+Au collisions at $\sqrt{s_{NN}}$=2.4 GeV: HADES Collaboration,” Eur. Phys. J. A, vol. 56, no. 10, p. 259, 2020.
  * (27) A. Aduszkiewicz et al., “Measurements of $\pi^{\pm}$ , K± , p and $\overline{p}$ spectra in proton-proton interactions at 20, 31, 40, 80 and 158 GeV/$c$ with the NA61/SHINE spectrometer at the CERN SPS,” Eur. Phys. J. C, vol. 77, no. 10, p. 671, 2017.
  * (28) A. Adare et al., “Identified charged hadron production in $p+p$ collisions at $\sqrt{s}=200$ and 62.4 GeV,” Phys. Rev. C, vol. 83, p. 064903, 2011.
  * (29) K. Aamodt et al., “Production of pions, kaons and protons in $pp$ collisions at $\sqrt{s}=900$ GeV with ALICE at the LHC,” Eur. Phys. J. C, vol. 71, p. 1655, 2011.
  * (30) J. Adam et al., “Measurement of pion, kaon and proton production in proton-proton collisions at $\sqrt{s}=7$ TeV,” Eur. Phys. J. C, vol. 75, no. 5, p. 226, 2015.
  * (31) S. Acharya et al., “Production of light-flavor hadrons in pp collisions at $\sqrt{s}\leavevmode\nobreak\ =\leavevmode\nobreak\ 7\text{ and }\sqrt{s}=13\,\text{ TeV}$,” Eur. Phys. J. C, vol. 81, no. 3, p. 256, 2021.
  * (32) G. Veneziano, “Construction of a crossing - symmetric, Regge behaved amplitude for linearly rising trajectories,” Nuovo Cim. A, vol. 57, pp. 190–197, 1968.
  * (33) A. Aduszkiewicz et al., “Proton-Proton Interactions and Onset of Deconfinement,” Phys. Rev. C, vol. 102, no. 1, p. 011901, 2020.
  * (34) B. Andersson, G. Gustafson, G. Ingelman, and T. Sjostrand, “Parton Fragmentation and String Dynamics,” Phys. Rept., vol. 97, pp. 31–145, 1983.
  * (35) A. B. Kaidalov and K. A. Ter-Martirosian, “Pomeron as Quark-Gluon Strings and Multiple Hadron Production at SPS Collider Energies,” Phys. Lett. B, vol. 117, pp. 247–251, 1982.
  * (36) A. Capella, U. Sukhatme, C.-I. Tan, and J. Tran Thanh Van, “Dual parton model,” Phys. Rept., vol. 236, pp. 225–329, 1994.
  * (37) S. A. Bass et al., “Microscopic models for ultrarelativistic heavy ion collisions,” Prog. Part. Nucl. Phys., vol. 41, pp. 255–369, 1998.
  * (38) V. Y. Vovchenko, D. V. Anchishkin, and M. I. Gorenstein, “Mean transverse mass of hadrons in proton–proton reactions,” Nucl. Phys. A, vol. 936, pp. 1–5, 2015.
  * (39) J. Adam et al., “Enhanced production of multi-strange hadrons in high-multiplicity proton-proton collisions,” Nature Phys., vol. 13, pp. 535–539, 2017.
  * (40) S. Chatrchyan et al., “Observation of long-range near-side angular correlations in proton-lead collisions at the LHC,” Phys.Lett., vol. B718, pp. 795–814, 2013.
  * (41) V. Khachatryan et al., “Evidence for collectivity in pp collisions at the LHC,” Phys. Lett., vol. B765, pp. 193–220, 2017.
  * (42) J. L. Nagle and W. A. Zajc, “Small System Collectivity in Relativistic Hadronic and Nuclear Collisions,” Ann. Rev. Nucl. Part. Sci., vol. 68, pp. 211–235, 2018.
  * (43) P. Bozek and W. Broniowski, “Theory of pp/pA/small systems,” PoS, vol. LHCP2016, p. 116, 2016.
  * (44) N. Abgrall et al., “NA61/SHINE facility at the CERN SPS: beams and detector system,” JINST, vol. 9, p. P06005, 2014.
  * (45) M. Gazdzicki, Z. Fodor, and G. Vesztergombi, “Study of Hadron Production in Hadron-Nucleus and Nucleus-Nucleus Collisions at the CERN SPS,” tech. rep., CERN, Geneva, Nov 2006.  revised version submitted on 2006-11-06 12:38:20.
  * (46) A. Acharya et al., “Measurements of $\pi^{\pm}$, $K^{\pm}$, $p$ and $\overline{p}$ spectra in 7Be+9Be collisions at beam momenta from 19$A$ to 150$A$ GeV/$c$ with the NA61/SHINE spectrometer at the CERN SPS,” Eur. Phys. J. C, vol. 81, no. 1, p. 73, 2021.
  * (47) C. Alt et al., “System-size dependence of strangeness production in nucleus-nucleus collisions at s(NN)**(1/2) = 17.3-GeV,” Phys. Rev. Lett., vol. 94, p. 052301, 2005.
  * (48) T. Anticic et al., “System-size and centrality dependence of charged kaon and pion production in nucleus-nucleus collisions at 40A GeV and158A GeV beam energy,” Phys. Rev. C, vol. 86, p. 054903, 2012.
  * (49) T. Abbott et al., “Charged hadron distributions in central and peripheral Si + A collisions at 14.6-A/GeV/c,” Phys. Rev. C, vol. 50, pp. 1024–1047, 1994.
  * (50) M. P. Lewicki and L. Turko, “NA61/SHINE shining more light on the onset of deconfinement,” 2 2020.
  * (51) A. Acharya et al., “Spectra and mean multiplicities of $\pi^{-}$ in central40Ar+45Sc collisions at 13A, 19A, 30A, 40A, 75A and 150$A$ GeV/$c$ beam momenta measured by the NA61/SHINE spectrometer at the CERN SPS,” Eur. Phys. J. C, vol. 81, no. 5, p. 397, 2021.
  * (52) D. Best et al., “K+ production in the reaction Ni-58 + Ni-58 at incident energies from 1-A/GeV to 2-A/GeV,” Nucl. Phys. A, vol. 625, pp. 307–324, 1997.
  * (53) D. Pelte et al., “Charged pions from Ni on Ni collisions between 1-A/Gev and 2-A/GeV,” Z. Phys. A, vol. 359, pp. 55–65, 1997.
  * (54) S. Acharya et al., “Production of pions, kaons, (anti-)protons and $\phi$ mesons in Xe–Xe collisions at $\sqrt{s_{\mathrm{NN}}}$ = 5.44 TeV,” Eur. Phys. J. C, vol. 81, no. 7, p. 584, 2021.
  * (55) J. Rafelski and M. Danos, “The Importance of the Reaction Volume in Hadronic Collisions,” Phys. Lett. B, vol. 97, pp. 279–282, 1980.
  * (56) K. Redlich and L. Turko, “Phase Transitions in the Statistical Bootstrap Model with an Internal Symmetry,” Z. Phys. C, vol. 5, p. 201, 1980.
  * (57) F. Becattini and L. Ferroni, “Statistical hadronization and hadronic microcanonical ensemble. 1.,” Eur. Phys. J. C, vol. 35, pp. 243–258, 2004\.
  * (58) T. S. Biro, H. B. Nielsen, and J. Knoll, “Color Rope Model for Extreme Relativistic Heavy Ion Collisions,” Nucl. Phys. B, vol. 245, pp. 449–468, 1984.
  * (59) M. Braun and C. Pajares, “A Probabilistic model of interacting strings,” Nucl. Phys. B, vol. 390, pp. 542–558, 1993.
  * (60) M. A. Braun and C. Pajares, “Particle production in nuclear collisions and string interactions,” Phys. Lett. B, vol. 287, pp. 154–158, 1992.
  * (61) M. A. Braun, C. Pajares, and V. V. Vechernin, “Ridge from Strings,” Eur. Phys. J. A, vol. 51, no. 4, p. 44, 2015.
  * (62) M. A. Braun, J. Dias de Deus, A. S. Hirsch, C. Pajares, R. P. Scharenberg, and B. K. Srivastava, “De-Confinement and Clustering of Color Sources in Nuclear Collisions,” Phys. Rept., vol. 599, pp. 1–50, 2015.
  * (63) J. E. Ramírez, B. Díaz, and C. Pajares, “Interacting color strings as the origin of the liquid behavior of the quark-gluon plasma,” Phys. Rev. D, vol. 103, no. 9, p. 094029, 2021.
  * (64) K. Werner, “Core-corona separation in ultra-relativistic heavy ion collisions,” Phys. Rev. Lett., vol. 98, p. 152301, 2007.
  * (65) Z.-W. Lin, C. M. Ko, B.-A. Li, B. Zhang, and S. Pal, “A Multi-phase transport model for relativistic heavy ion collisions,” Phys. Rev. C, vol. 72, p. 064901, 2005.
  * (66) S. Digal, S. Fortunato, and H. Satz, “Predictions for J / psi suppression by parton percolation,” Eur. Phys. J. C, vol. 32, pp. 547–553, 2004.
  * (67) C. Hohne, F. Puhlhofer, and R. Stock, “System-size dependence of strangeness production in high-energy a+a collisions and percolation of strings,” Phys. Lett. B, vol. 640, pp. 96–100, 2006.
  * (68) T. Kalaydzhyan and E. Shuryak, “Collective interaction of QCD strings and early stages of high multiplicity pA collisions,” Phys. Rev. C, vol. 90, no. 1, p. 014901, 2014.
  * (69) T. Kalaydzhyan and E. Shuryak, “Self-interacting QCD strings and string balls,” Phys. Rev. D, vol. 90, no. 2, p. 025031, 2014.
  * (70) I. Iatrakis, A. Ramamurti, and E. Shuryak, “Collective String Interactions in AdS/QCD and High-Multiplicity pA Collisions,” Phys. Rev. D, vol. 92, no. 1, p. 014011, 2015.
  * (71) R. Hagedorn, “Statistical thermodynamics of strong interactions at high-energies,” Nuovo Cim. Suppl., vol. 3, pp. 147–186, 1965.
  * (72) A. Andronic, P. Braun-Munzinger, K. Redlich, and J. Stachel, “Decoding the phase structure of QCD via particle production at high energy,” Nature, vol. 561, no. 7723, pp. 321–330, 2018.
  * (73) L. Fields and P. Podlaski, “Report from the NA61/SHINE experiment at the CERN SPS,” tech. rep., CERN, Geneva, Oct 2020.
  * (74) M. Gazdzicki, “Addendum to the NA61/SHINE Proposal: Request for oxygen beam in Run 3,” tech. rep., CERN, Geneva, May 2022.
  * (75) J. Brewer, A. Mazeliauskas, and W. van der Schee, “Opportunities of OO and $p$O collisions at the LHC,” in Opportunities of OO and pO collisions at the LHC, 3 2021.
  * (76) Z. Citron et al., “Report from Working Group 5: Future physics opportunities for high-density QCD at the LHC with heavy-ion and proton beams,” CERN Yellow Rep. Monogr., vol. 7, pp. 1159–1410, 2019.


Timely feedback is an important part of teaching and learning. Here we describe how a readily available neural network transformer (machine-learning) model (BERT) can be used to give feedback on the structure of the response to an abstracting exercise where students are asked to summarise the contents of a published article after finding it from a publication database. The dataset contained 207 submissions from two consecutive years of the course, summarising a total of 21 different papers from the primary literature. The model was pre-trained using an available dataset (approx. 15,000 samples) and then fine-tuned on 80% of the submitted dataset. This fine tuning was seen to be important. The sentences in the student submissions are characterised into three classes – background, technique and observation – which allows a comparison of how each submission is structured. Comparing the structure of the students’ abstract a large collection of those from the PubMed database shows that students in this exercise concentrate more on the background to the paper and less on the techniques and results than the abstracts to papers themselves. The results allowed feedback for each submitted assignment to be automatically generated.

§ INTRODUCTION

Feedback is recognised as an important part of modern teaching and learning in higher education <cit.>. There is no prescriptive definition of what feedback is most appropriate as it can vary between the type of assignment or assessment involved. The best feedback may often be a balance between corrective (cognitivist) and socio-constructivist (i.e. encouraging the student to learn through self- or peer-group evaluation) and may be written or oral <cit.>. However, whatever the method of feedback chosen for any assignment there is no doubt that any feedback should be timely. This is because the best advice for improving student performance, narrowing the gap between real performance and that expected by the instructor <cit.> will be when the task is still relatively fresh in their mind. In large modules with many students the workload involved with providing timely feedback can be significant and achieving timeliness can become critical when, for example, there is limited time between an assignment and a final examination.

Machine learning (ML) is a computer technology that relies on algorithms built by learning from training data.  In recent years there has been a growing interest in using ML as an aid to assessment in STEM education. Zhai and co-workers <cit.> have recently reviewed approximately 50 papers that used ML as an assessment tool across science, although only one of the papers concerned a specific chemistry content. More recently Yik et al. <cit.> have given a compelling argument that ML approaches are of sufficient accuracy ( 90%) in assigning scores to open-ended (formative) assessments (of how students understand acid-base mechanisms) to make them a valuable tool.  However, there remain some ethical issues with assigning marks using such a system (especially in so-called high stakes assignments where the result can affect the degree classification or progression of a student).

Rather than use ML for automatic scoring per se, the aim of the project reported here was to examine whether ML approaches can be used to provide quick, useful feedback for a second-year chemistry assignment, where the feedback could be used by both the student and the human marker assigning the scores for the work. The chosen assignment is one whose aim is to introduce students to databases and to reading primary literature. For most of the class this is the first time they will have encountered such an assignment. However, this is a skill they will use throughout their degree and, in particular, during research-led teaching and research projects. The assignment consists of two parts: a database search and a paper summary/abstract. In the first part of the assignment the students find a particular paper in a database, locate the journal impact factor, reformat the reference in two different styles and then look up how many citations the paper has received. These types of questions are fairly trivial to give feedback for as they essentially have correct answers (or answers in a narrow band of possibilities). These do not require an ML approach to generate feedback. In the second part the students are required to summarise the content of the paper. Free-form questions, such as the summaries in this assignment, are more appropriate for a machine-learning approach <cit.>. Modern methods to approach this problem use neural architectures to automatically learn which features are important <cit.>. These systems have been improved by the introduction of the attention mechanism <cit.> which allow a model to assign a weight to the importance of each word in the input text, allowing the model to better extract the meaning of the text. However, an extra challenge in the assessment of this assignment is that students are allocated a different paper to summarise – the project has therefore focused on giving feedback on the structure of the responses.
Automatic systems of this kind could be very useful because they significantly lessen the workload on academic staff and increase the efficiency of students’ learning by providing quick feedback on their work. This could allow students to improve their work without direct feedback from a member of staff, increasing students’ independence and the efficiency of their revision. In addition, it can also be used to flag up quickly those students who need urgent intervention from a member of staff early in the process. For some assignments students may be supported by a subsequent small group tutorial where a final mark is given by the tutor based on both their written response to the assignment and their oral contribution during the tutorial itself. Timely automatic feedback will also support this process as it gives the tutor a guide as to which areas would be most fruitful to discuss in the tutorial.

In the following study we have used a two-step approach to providing feedback. As a check to ensure that the machine learning can indeed identify answers of different quality we used an Automated Essay Scoring (AES) system to provide a mark for the assignment, which was then compared to that provided by the human markers. AES systems have been well researched <cit.> and are capable of grading essays almost within the level of disagreement between human markers. However, because of the ethical issues surrounding the implementation of AES we have only used this as a validity check at this time and so

The second step in the process is Automated Feedback Generation (AFG). In contrast to automatic scoring (AES), AFG has not been studied anywhere near so often <cit.> To provide feedback on the abstracts, we propose a method that uses features extracted by a neural model that are used to provide guiding questions to the student to improve their work and to provide topics for discussion in a subsequent class. Since AES and AFG are very different tasks that require different approaches, we discuss each task in turn and finally provide an example of the feedback on the structure of assignment answer that a student receives using this system.

§ THE ASSIGNMENT

The dataset used for these experiments was derived from a chemistry database/abstracting exercise where each tutorial group was given a different paper for which they had to write a summary in abstract form and provide other information. The information the students had to provide is as follows:

* The Impact Factor of the Journal

* The Reference in the Royal Society of Chemistry format

* The Reference in the American Chemical Society format

* The number of times the paper was cited.

Each of these pieces of information are given 0.5 or 1 mark depending on whether they are “partially correct” or “fully correct.” The correct answers for the 4 questions listed above were extracted by selecting the most common answer in the dataset (if this system were deployed, the assessors could choose their own correct answers) as the majority of students get these questions correct.
After finding these 4 pieces of information the students write an abstract/summary for the paper. The prompt they were given is:

Summarise, in abstract form, the content of the paper.

The abstract is awarded 0-6 marks providing a total of 10 marks for the exercise.

The task can then be further broken down such that the AES system first provides a mark and feedback on the accuracy of the 4 pieces of information and then provides a mark and feedback on the abstract.

§ THE DATASET

The dataset contains 207 samples sourced from two consecutive years’ answers to the assignment. The task involved 21 different papers from the primary literature so each tutorial group has a different one. To evaluate the model the dataset was split into two parts, 80% of the data was used to train the neural model and the other 20% was used to evaluate the AES system. Evaluation of the automatic feedback was performed manually as there was no human-given feedback with the samples.

§ THE AUTOMATIC SCORING SYSTEM

In the same way that the exercise contains two distinct types of question, these questions are scored individually by two different systems and then the scores are added to produce the final mark out of 10.

§.§ Scoring Parts 1 to 4

The numerical questions are, in principle, very simple to score. The current impact factor question (part 1) should have a definitively correct answer while the number of times cited (part 4) response is time sensitive: depending on the exact date that the student looked up the information the database may have been updated. One could give either 0 or 1 depending on certain criteria. However, to facilitate the feedback aspects we chose to give three possible marks. The value given by the student is compared to the correct value and if the student’s value is within 10% percentage difference of the correct value the answer is deemed “fully correct” and one mark is given. If the student’s value is between 10% and 25% percentage difference of the correct value then the answer is deemed “partially correct” and 0.5 marks are given. If the student’s value is outside 25% then the answer is deemed “incorrect” and no marks are awarded.

For the referencing formatting questions (parts 2 and 3) a different approach is required. One method to mark such textual questions is to extract the information from the reference e.g. Authors, Year, Journal etc. check that they are correct and in the correct order. However, perhaps a simpler method to implement is to use the cosine similarity, as defined in <cit.>, to compare the two texts. A cosine similarity higher than 0.9 is deemed fully correct and one mark assigned, a cosine similarity between 0.65 and 0.9 is deemed partially correct earning a mark of 0.5 and a cosine similarity less than 0.65 is deemed incorrect and gaining zero marks.

§.§ Scoring the Abstracting Assignment: the Pre-Training Dataset

Because of the small dataset of only  200 samples, we decided to pre-train the model on a much larger dataset. For this the datasets created for the Automated Student Assessment Prize (ASAP) sponsored by the Hewlett Foundation were used. For these competitions two datasets were published, one dataset for Automated Essay Scoring <cit.> and one for Short Answer Scoring (SAS) <cit.>. These datasets contain approximately 15,000 samples each. This allows for the AES model to begin with a much better ‘foundation’ when it begins to score abstracts and means that the model can achieve high performance while not overfitting to the training set. Overfitting is a particular problem as it means that the model cannot generalise from the training set. In effect, overtraining means the system is ‘memorising’ each sample in the training set rather than finding patterns.

The pre-training dataset contain texts from a variety of subjects including Physics, English, Biology and Chemistry among others. This variety makes a combined dataset suitable for training a subject-agnostic AES model that can be fine-tuned on a specific question with a small dataset to achieve higher performance. Each sample was min-max normalised to a range [0,1]; this was done according to the question the sample was answering, effectively converting each mark to a percentage, i.e. a fully correct answer achieves a score of 1, regardless of how many total marks are available for the question.

§.§ The AES Model

The model used for the AES portion of the task uses a Bidirectional Long Short-Term Memory (BiLSTM) <cit.> encoder with an attention mechanism <cit.>. The input sequence is tokenised using the same tokeniser as used in the BERT transformer model <cit.>. This tokeniser is used because of its ability to tokenise to word parts rather than whole words. This means that when technical language is used, as is often the case with texts such as abstracts, no vocabulary needs to be added, this then improves the effectiveness of pre-training such a model.

§.§ Metrics

§.§.§ Training Metric

To train the AES model a dynamic loss function was used that combines the error in the standard deviation of the model’s predictions and the Mean Squared Error (MSE) metric <cit.>. Multiple loss functions can be combined using a weighted sum. This weight, p is defined in Eq.<ref>.

\begin{equation} p(t) = \min\left(a, a \cdot \exp\left(-c \cdot \left(\frac{t}{T} - b \right)\right)\right) \label{eq:min-decay} \end{equation}

where \(a\), \(b\) and \(c\) are constants, \(t\) is the current training step and \(T\) is the total number of training steps. Such that:

\begin{equation} L_T(t) = p(t) \cdot L_1 + (1-p(t))\cdot L_2 \label{eq:combine-loss} \end{equation}

where \(L_T\) is the total loss, and \(L_1\) and \(L_2\) are two loss functions. For this dynamic loss function \(L_1\) is the error in the standard deviation as defined in Eq. <ref> and \(L_2\) is the MSE metric as defined in Eq. <ref>:

\begin{equation} \text{STDE}(\mathbf{x}, \mathbf{y}) = |\sigma(\mathbf{y}) - \sigma(\mathbf{x}) \label{eq:std-error} \end{equation}

where \(mathbf{y}\) are the model's predictions, \(mathbf{x}\) are the correct values and \(\sigma\) is the function calculating the standard deviation.

\begin{equation} \text{MSE}(\mathbf{x}, \mathbf{y}) = \frac{1}{N} \sum^{N}_{i=0}(y_i - x_i)^2 \label{eq:mse} \end{equation}

§.§.§ Evaluation Metrics

To evaluate the performance of the AES model several metrics are used, primarily the coefficient of determination \(r^2\) as defined in Eq. <ref> and also the Mean Absolute Error (MAE), defined in Eq. <ref> and Root Mean Squared Error (RMSE) defined in Eq. <ref>.

\begin{equation} r^2 = 1- \frac{\sum^{N}_{i=0}(y_i - x_i)^2}{\sum^{N}_{i=0}(y_i-\bar{y})^2} \label{eq:r2} \end{equation}

where \(\bar{y}\) is the mean of the predicted values.

\begin{equation} \text{MAE}(\mathbf{x}, \mathbf{y}) = \frac{1}{N} \sum^{N}_{i=0}|y_i-x_i| \label{eq:mae} \end{equation}

\begin{equation} \text{RMSE}(\mathbf{x}, \mathbf{y}) = \sqrt{\frac{1}{N} \sum^{N}_{i=0}(y_i - x_i)^2} \label{eq:rmse} \end{equation}

§.§ Results

After fine-tuning the neural model, the entire AES system was tested on the evaluation split of the dataset. The model was also evaluated after only being pre-trained on the combined ASAP dataset. The model was unable to learn only using the small fine-tuning dataset due to severe overfitting. Table <ref> shows the performance of both models at each stage of training.

Performance of the model after only Pre-Training and after Pre-Training followed by Fine-Tuning. These values are the mean of 5 runs.

Model           $r^2$     MAE      RMSE     Maximum Error
Pre-Trained     0.29      0.12     0.15     0.28

Fine-Tuned      0.51      0.10     0.12     0.23

These results show that fine-tuning a pre-trained model, even though it has been pre-trained on a very similar task, significantly increases the final performance of the model.

Confusion matrix for automatic (machine) scoring system for the abstract section of the assignment compared with the scoring by human markers.

Figure <ref> shows a confusion matrix for this run of the model. It shows the correlation between the results obtained from human marking and the automatic scoring. A perfect correlation between the two would give only scores on the diagonal of the matrix. Here we see that the automatic scoring gives considerably more marks of 4 out of 6 than do the human markers and also more 2 out of 6. The spread of marks for the human marking seems a little greater than for the automatic machine marking. However, given that for ethical reasons we do not recommend using machine-generated marking at the present time, these results were deemed to be promising enough to move forward to see if the desired automatic feedback could be machine generated. It should also be noted that the marking has not been optimised fully for scoring and so the correlation may well be improved by further work. For the overall goal of this project this was not deemed necessary.

§ THE AUTOMATIC FEEDBACK GENERATION SYSTEM

As described above, the assignment is split into two quite different tasks and the AFG system can similarly be split into two components. To provide simple cognitivist feedback on the first four questions is mostly trivial as it is based on whether the answers are right or wrong. The feedback can then be chosen from a pre-prepared list of comments. Comments such as “Make sure your Royal Society of Chemistry reference has exactly the correct format” or “That is the correct Impact Factor, Well done!” can be given simply on the basis of whether the answers given are right or wrong. This form of feedback (rather than simply saying whether the answer is right or wrong) suggests to the student that they should review their work more closely.

For more free form answers to the abstracting/summarising section the feedback provided must be more nuanced. This is particularly true in an assignment where the students are not all summarising the same paper – each group has a different topic. However, the structure of a response is likely to be similar whatever the scientific content of the paper. Therefore, to provide feedback on the structure of the abstracts we decided to classify each sentence in the abstract and provide feedback based on how each sentence was classified. To provide feedback on the abstracts submitted in this assignment we trained a neural model to classify each sentence in an abstract into one of these categories and then used the results as a guide as to the feedback that should be given. To achieve this level of classification a BERT encoder was used with a classification head designed for multilable classification <cit.> as exposed by the simpletransformers python module <cit.>.

§.§ Training

To fine-tune this BERT model to classify abstract sentences a dataset was created based on the PubMed20kRCT (a smaller variant of the PubMed200k-RCT dataset)  containing 20,000 abstracts from manuscripts on the subject of medical science that are in the PubMed database <cit.>. Despite these abstracts being primarily from the medical field the assumption used in deciding to use this training set is that the structure of an abstract of a scientific paper is relatively invariant irrespective of the specific scientific field.

The Pubmed20k-RCT dataset classifies abstract sentences into 5 classes: background, objective, method, result and conclusion. Banerjee et. al <cit.> defined an alternative and slightly simpler approach with only three categories of abstract sentences: background, techniques and observations (results). Table <ref> shows how the classifications used in the Pubmed20k-RCT work map onto those used by Banerjee. Our starting point for feedback is that a ‘good’ abstract should have a structure that includes sentences from all three of the Banerjee classes. An abstract that is missing one class of sentence probably does not reflect the content of the scientific paper in enough detail to make it a useful abstract.

Mapping between PubMed categories (left) and categories used by Banerjee et. al (right)

BACKGROUND     2*BACKGROUND
METHOD         TECHNIQUE
RESULT         2*OBSERVATION

The BERT model was fine-tuned trained on the PubMed dataset for 5 epochs with a batch size of 64. For the purposes of evaluation during training the dataset was split into a training split and an evaluation split, 90% of the data was used for training the model and the other 10% was used to evaluate the model.

§.§ results

On the evaluation split of the PubMed dataset the model achieved an accuracy score of 92.0%, meaning that it classified all but 8% of sentences ‘correctly’. This is very close to the 92.1% accuracy reported by Banjeree et. al on their PubMed20k-non-RCT.

When this model is applied to our dataset the model classifies most sentences as either Background or Observation, only 11.5% of sentences are classified as Technique and 71.5% of the abstracts only contain sentences in 2 of the 3 classes.  Table <ref> shows the distribution of predicted labels throughout the dataset and how students in the assignment strongly emphasise background in their submissions. This is an interesting result in itself in that it shows that when summarising the work the students concentrate on background (and motivation) more than authors do when they write abstracts of their own papers. This is not a great surprise, but of course does reinforce the need for proper setting of expectations for the students before they complete the assignment to make sure they understand what a good structure looks like. It may be perfectly appropriate for the student to write more of a summary rather than an abstract, but the wording of the question, and the instructions/training given to the students needs to be clear as to what the assignment is testing and the types of response expected.

Distribution of predicted labels on student-written abstracts from our data and abstracts from PubMed20k-RCT

Label           Our Data     PubMed20k-RCT
BACKGROUND      49.9         19.8
TECHNIQUE       11.5         33.0
OBSERVATION     38.6         47.3

Below is an example abstract containing at least one sentence from each category with sentences predicted as BACKGROUND being highlighted yellowyellow, sentences predicted as TECHNIQUE being highlighted greengreen and sentences predicted as OBSERVATION being highlighted pinkpink:

yellowNitro-anilides are compounds essential for manufacturing a variety of chemical compounds, including pharmaceuticals, dyes and explosives. yellowTraditional methods use concentrated sulfuric and nitric acids, where reaction conditions are harsh and have low tolerance for other functional groups. greenThey were replaced by using nitrate salt reagents, which can be difficult to prepare and, from the resultant metal oxides, have a low atom economy. yellowMore recent methods use AgNO$_2$ to selectively nitrate arenes, but this method uses high quantities of rare metals, which is unsustainable. pinkThis reaction uses NaNO$_2$ and K$_2$S$_2$O$_8$ in CH$_3$CN, using catalytic AgNO$_2$, to selectively nitrate the ortho positions on a variety of arenes, displaying high regioselectivity and chemoselectivity as well as moderate to high yields with a selection of substitutions on the starting anilide. yellowThe mechanism proceeds through silver chelation and a subsequent radical mechanism, and the silver catalyst is regenerated by the K$_2$S$_2$O$_8$ oxidant.

As seen above, 4 out of the 6 sentences are labelled BACKGROUND with only one sentence each fitting into the other two categories.

The feedback given to this abstract is as follows:

Your discussion of the paper’s background has a good amount of detail.

It might be useful to outline the Techniques the model uses in a bit more detail.

It may be worth making sure that the discussion of the conclusions of the paper are clearer.

§.§ Example 2

In this section we provide an example of the whole exercise and the marks and feedback our system generated for it.

§.§.§ Student Response

Journal "Impact Factor": 6.005

Reference in RSC format:

A. Lator, S. Gaillard, A. Poater and J.-L. Renaud, Organic Letters, 2018, 20, 5985–5990.

Reference in ACS format:

Lator, A.; Gaillard, S.; Poater, A.; Renaud, J.-L. Well-Defined Phosphine-Free Iron-Catalyzed N-Ethylation and N-Methylation of Amines with Ethanol and Methanol. Organic Letters 2018, 20 (19), 5985–5990.

Number of times cited: 10

To combat the issues of toxic chemicals and by-products, noble and precious metal catalysts, and expensive phosphorus ligands, a new method of alkylation of amines was devised. N-ethylation and N-methylation of a broad range of aliphatic and aromatic compounds were demonstrated using a (cyclopentadienone) iron tricarbonyl complex under basic conditions. These compounds were ethylated or methylated using ethanol or methanol. The use of methanol was more energetically demanding due to its higher enthalpy of dehydrogenation. Consequently, a change in hydrogen pressure was required for selective dehydration over dehydrogenation to methylate some compounds. The method shown produced mono- or dialkylated compounds in high yields. DFT calculations revealed potential pathways for the reaction and highlighted the role of hydrogen pressure in driving the equilibrium towards one intermediate and hence the reduction of imines.

§.§.§ Our Evaluation

Impact Factor: 1 mark

Reference in RSC format: 1 mark

Reference in ACS format: 1 mark

Number of times Cited: 0 marks, the correct answer is 42, you gave 10

Abstract: 3 marks

yellowTo combat the issues of toxic chemicals and by-products, noble and precious metal catalysts, and expensive phosphorus ligands, a new method of alkylation of amines was devised. N-ethylation and N-methylation of a broad range of aliphatic and aromatic compounds were demonstrated using a (cyclopentadienone) iron tricarbonyl complex under basic conditions. greenThese compounds were ethylated or methylated using ethanol or methanol. pinkThe use of methanol was more energetically demanding due to its higher enthalpy of dehydrogenation. Consequently, a change in hydrogen pressure was required for selective dehydration over dehydrogenation to methylate some compounds. The method shown produced mono- or di- alkylated compounds in high yields. DFT calculations revealed potential pathways for the reaction and highlighted the role of hydrogen pressure in driving the equilibrium towards one intermediate and hence the reduction of imines.

yellowBACKGROUND greenTECHNIQUE pinkOBSERVATION

A more balanced discussion of the background of the paper, the techniques of the paper and the observations and conclusions the paper made might improve your work.

It might be worth outlining the methods of the paper in greater detail.

The abstract contains discussion of each aspect of the paper in a logical order.

§ DISCUSSION

AES and AFG systems have many advantages: they allow for students to receive immediate results, academic institutions can significantly reduce the “unnecessary” time allocated to students in manually marking their work, governmental organisations responsible for nation-wide exams could cut spending on examiners drastically and reduce the stressful period where students are waiting for results, student’s revision could be streamlined by increasing the number of practice essays they could write, receiving immediate feedback and improving their work before an exam. With all of these advantages it may seem perplexing that these systems have not yet been adopted by major organisations and academic institutions, even when these systems are reaching near human-level performance.

Despite all of these advantages, AES systems in particular do have some disadvantages: most notably their lack of transparency, especially with neural models. This is an issue with almost every neural network, they are effectively a ‘black box.’ It is impossible to tell how the model is coming to its conclusions. It is likely that this lack of transparency is the reason why these systems have not left the research phase yet as this will decrease the confidence that students have in the system’s grades.

These disadvantages are less obvious for AFG systems as there are fewer confidence issues than there are with grades. This means they are likely to be easier to implement. The question is then how can they be used and how should they be used. In the example we have described in this paper we have chosen to concentrate on the structure of the assignment response and so feedback on the content of the response would still have to be completed by a human marker. However, high quality feedback on the structure would still be a very useful starting point for the markers, saving them time in what can be an onerous task if numbers in the class are high. In the first instance, therefore, we would see this feedback going to the human markers first, to be checked and added to before being forwarded to the student. As well as saving the human marker some time, it will also allow confidence in the quality of the feedback to be developed over time. In the long run the goal is to give the students instant, high-quality feedback on their work without the need for intervention by a human marker. This would be particularly advantageous if first drafts of work are passed through the system, which would then allow the students to improve the quality of their assignments before admission. Eventually, it may even be possible to add in automatic marking using an AES system.

§ CONCLUSION

We have shown how available machine learning algorithms can be utilised to give automatic feedback on a chemistry assignment that mixes different types of responses. The most interesting aspect is that there is scope to give instant feedback not only on the parts of the assignment that have objectively correct answers, but that machine learning can be used to give quick feedback on the structure of a response, even if the chemical content of each response differs.

Missing 'biblatex' package
The bibliography requires the 'biblatex' package.

American Educational Research Association

abstractThis article presents a thematic analysis of the research evidence on assessment feedback in higher education (HE) from 2000 to 2012. The focus of the review is on the feedback that students receive within their coursework from multiple sources. The aims of this study are to (a) examine the nature of assessment feedback in HE through the undertaking of a systematic review of the literature, (b) identify and discuss dominant themes and discourses and consider gaps within the research literature, (c) explore the notion of the feedback gap in relation to the conceptual development of the assessment feedback field in HE, and (d) discuss implications for future research and practice. From this comprehensive review of the literature, the concept of the feedback landscape, informed by sociocultural and socio-critical perspectives, is developed and presented as a valuable framework for moving the research agenda into assessment feedback in HE forward.
journaltitleReview of Educational Research
titleMaking Sense of Assessment Feedback in Higher Education

abstractBackground Effective feedback may be defined as feedback in which information about previous performance is used to promote positive and desirable development. This can be challenging as educators must acknowledge the psychosocial needs of the recipient while ensuring that feedback is both honest and accurate. Current feedback models remain reductionist in their approach. They are embedded in the hierarchical, diagnostic endeavours of the health professions. Even when it acknowledges the importance of two-way interactions, feedback often remains an educator-driven, one-way process. Lessons from the Literature An understanding of the various types of feedback and an ability to actively seek an appropriate approach may support feedback effectiveness. Facilitative rather than directive feedback enhances learning for high achievers. High-achieving recipients undertaking complex tasks may benefit from delayed feedback. It is hypothesised that such learners are supported by reducing interruptions during the task. If we accept that medical students and doctors are high achievers, we can draw on some guiding principles from a complex and rarely conclusive literature. Feedback should focus on the task rather than the individual and should be specific. It should be directly linked to personal goals. Self-assessment as a means to identify personal learning requirements has no theoretical basis. Motivated recipients benefit from challenging facilitated feedback from external sources. A New Model To achieve truly effective feedback, the health professions must nurture recipient reflection-in-action. This builds on self-monitoring informed by external feedback. An integrated approach must be developed to support a feedback culture. Early training and experience such as peer feedback may over time support the required cultural change. Opportunities to provide feedback must not be missed, including those to impart potentially powerful feedback from high-stakes assessments. Feedback must be conceptualised as a supported sequential process rather than a series of unrelated events. Only this sustained approach will maximise any effect. Medical Education 2010: 44: 101–108
journaltitleMedical Education
shorttitleState of the Science in Health Professional Education
titleState of the Science in Health Professional Education: Effective Feedback
Archer - 2010 - State of the science in health professional educat.pdf;C
Oscar Morris

abstractThe idea that assessment is intrinsic to effective instruction is traced from early experiments in the individualization of learning through the work of Benjamin Bloom to reviews of the impact of feedback on learners in classrooms. While many of these reviews detailed the adverse impact of assessment on learning, they also indicated that under certain conditions assessment had considerable potential to enhance learning. It is shown that understanding the impact that assessment has on learning requires a broader focus than the feedback intervention itself, particularly the learner's responses to the feedback, and the learning milieu in which the feedback operates. Different definitions of the terms “formative assessment” and “assessment for learning” are discussed, and subsumed within a broad definition that focuses on the extent to which instructional decisions are supported by evidence. The paper concludes by exploring some of the consequences of this definition for classroom practice.
journaltitleStudies in Educational Evaluation
seriesAssessment for Learning
titleWhat Is Assessment for Learning?

Oscar Morris
Assessment for learning,Feedback,Formative assessment

abstractMachine learning (ML) is an emergent computerised technology that relies on algorithms built by `learning' from training data rather than `instruction', which holds great potential to revolutionise science assessment. This study systematically reviewed 49 articles regarding ML-based science assessment through a triangle framework with technical, validity, and pedagogical features on three vertices. We found that a majority of the studies focused on the validity vertex, as compared to the other two vertices. The existing studies primarily involve text recognition, classification, and scoring with an emphasis on constructing scientific explanations, with a vast range of human-machine agreement measures. To achieve the agreement measures, most of the studies employed a cross-validation method, rather than self- or split-validation. ML allows complex assessments to be used by teachers without the burden of human scoring, saving both time and cost. Most studies used supervised ML, which relies on extraction of attributes from student work that was first coded by humans to achieve automaticity, rather than semi- or unsupervised ML. We found that 24 studies were explicitly embedded in science learning activities, such as scientific inquiry and argumentation, to provide feedback or learning guidance. This study identifies existing research gaps and suggests that all three vertices of the ML triangle should be addressed in future assessment studies, with an emphasis on the pedagogy and technology features.
journaltitleStudies in Science Education
shorttitleApplying Machine Learning in Science Assessment
titleApplying Machine Learning in Science Assessment: A Systematic Review

The Royal Society of Chemistry

abstractAcid–base chemistry is a key reaction motif taught in postsecondary organic chemistry courses. More specifically, concepts from the Lewis acid–base model are broadly applicable to understanding mechanistic ideas such as electron density, nucleophilicity, and electrophilicity; thus, the Lewis model is fundamental to explaining an array of reaction mechanisms taught in organic chemistry. Herein, we report the development of a generalized predictive model using machine learning techniques to assess students' written responses for the correct use of the Lewis acid–base model for a variety (N = 26) of open-ended formative assessment items. These items follow a general framework of prompts that ask: why a compound can act as (i) an acid, (ii) a base, or (iii) both an acid and a base (i.e., amphoteric)? Or, what is happening and why for aqueous proton-transfer reactions and reactions that can only be explained using the Lewis model. Our predictive scoring model was constructed from a large collection of responses (N = 8520) using a machine learning technique, i.e., support vector machine, and subsequently evaluated using a variety of validation procedures resulting in overall 84.5–88.9% accuracies. The predictive model underwent further scrutiny with a set of responses (N = 2162) from different prompts not used in model construction along with a new prompt type: non-aqueous proton-transfer reactions. Model validation with these data achieved 92.7% accuracy. Our results suggest that machine learning techniques can be used to construct generalized predictive models for the evaluation of acid–base reaction mechanisms and their properties. Links to open-access files are provided that allow instructors to conduct their own analyses on written, open-ended formative assessment items to evaluate correct Lewis model use.
journaltitleChemistry Education Research and Practice
titleDevelopment of a Machine Learning-Based Tool to Evaluate Correct Lewis Acid–Base Model Use in Written Responses to Open-Ended Formative Assessment Items
omputational Linguistics

abstractTraditional automated essay scoring systems rely on carefully designed features to evaluate and score essays. The performance of such systems is tightly bound to the quality of the underlying features. However, it is laborious to manually design the most informative features for such a system. In this paper, we develop an approach based on recurrent neural networks to learn the relation between an essay and its assigned score, without any feature engineering. We explore several neural network models for the task of automated essay scoring and perform some analysis to get some insights of the models. The results show that our best system, which is based on long short-term memory networks, outperforms a strong baseline by 5.6% in terms of quadratic weighted Kappa, without requiring any feature engineering.
booktitleProceedings of the 2016 Conference on Empirical Methods in Natural Language Processing
titleA Neural Approach to Automated Essay Scoring

abstractNeural machine translation is a recently proposed approach to machine translation. Unlike the traditional statistical machine translation, the neural machine translation aims at building a single neural network that can be jointly tuned to maximize the translation performance. The models proposed recently for neural machine translation often belong to a family of encoder-decoders and consists of an encoder that encodes a source sentence into a fixed-length vector from which a decoder generates a translation. In this paper, we conjecture that the use of a fixed-length vector is a bottleneck in improving the performance of this basic encoder-decoder architecture, and propose to extend this by allowing a model to automatically (soft-)search for parts of a source sentence that are relevant to predicting a target word, without having to form these parts as a hard segment explicitly. With this new approach, we achieve a translation performance comparable to the existing state-of-the-art phrase-based system on the task of English-to-French translation. Furthermore, qualitative analysis reveals that the (soft-)alignments found by the model agree well with our intuition.
eprintclasscs, stat
titleNeural Machine Translation by Jointly Learning to Align and Translate
6 - Neural Machine Translation by Jointly Learning to .pdf;C
Oscar Morris
Computer Science - Computation and Language,Computer Science - Machine Learning,Computer Science - Neural and Evolutionary Computing,Statistics - Machine Learning

abstractAutomated Text Scoring (ATS) provides a cost-effective and consistent alternative to human marking. However, in order to achieve good performance, the predictive features of the system need to be manually engineered by human experts. We introduce a model that forms word representations by learning the extent to which specific words contribute to the text's score. Using Long-Short Term Memory networks to represent the meaning of texts, we demonstrate that a fully automated framework is able to achieve excellent results over similar approaches. In an attempt to make our results more interpretable, and inspired by recent advances in visualizing neural networks, we introduce a novel method for identifying the regions of the text that the model has found more discriminative.
booktitleProceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)
titleAutomatic Text Scoring Using Neural Networks
6 - Automatic Text Scoring Using Neural Networks.pdf
Computer Science - Computation and Language,Computer Science - Machine Learning,Computer Science - Neural and Evolutionary Computing,I.2.6,I.2.7,I.5.1

abstractAutomated essay scoring (AES) is the task of automatically assigning scores to essays as an alternative to grading by humans. Although traditional AES models typically rely on manually designed features, deep neural network (DNN)-based AES models that obviate the need for feature engineering have recently attracted increased attention. Various DNN-AES models with different characteristics have been proposed over the past few years. To our knowledge, however, no study has provided a comprehensive review of DNN-AES models while introducing each model in detail. Therefore, this review presents a comprehensive survey of DNNAES models, describing the main idea and detailed architecture of each model. We classify the AES task into four types and introduce existing DNN-AES models according to this classification.
issn0385-7417, 1349-6964
titleA Review of Deep-Neural Automated Essay Scoring Models
omputer-based evaluation system works only for multiple-choice questions, but there is no proper evaluation system for grading essays and short answers. Many researchers are working on automated essay grading and short answer scoring for the last few decades, but assessing an essay by considering all parameters like the relevance of the content to the prompt, development of ideas, Cohesion, and Coherence is a big challenge till now. Few researchers focused on Content-based evaluation, while many of them addressed style-based assessment. This paper provides a systematic literature review on automated essay scoring systems. We studied the Artificial Intelligence and Machine Learning techniques used to evaluate automatic essay scoring and analyzed the limitations of the current studies and research trends. We observed that the essay evaluation is not done based on the relevance of the content and coherence.
issn0269-2821, 1573-7462
journaltitleArtificial Intelligence Review
shorttitleAn Automated Essay Scoring Systems
titleAn Automated Essay Scoring Systems: A Systematic Literature Review
hina. It would be very useful if students could receive timely and effective feedback about their writing. Automatic essay feedback generation is a challenging task, which requires understanding the relationship between the text features of the essay and feedback. In this study, we first analyzed 1,290 teacher comments on their 327 Englishmajor students and annotated the feedback on seven aspects of writing, including the grammar, spelling, sentence diversity, structure, organization, supporting ideas, coherence, and conclusion, for each paper. Then, an automatic feedback classification experiment was conducted with the machine learning approach. Finally, we investigated the impact of the system generated-indirect corrective feedback (ICF) and human teachers' direct corrective feedback (DCF) in two English writing classes (N = 56 in ICF class; N = 54 in DCF class) at a key Chinese university through a web-based assignment management system. The study results indicated the feasibility of this approach that system generated ICF can be as useful as direct comments made by the teachers in terms of improving the quality of the content regarding to the structure, organization, supporting ideas, coherence, and conclusion, and encouraging students to spend more time on self-correction.
journaltitleIEEE Transactions on Learning Technologies
titleAutomated Essay Feedback Generation and Its Impact on Revision

Oscar Morris
Feature extraction,Grammar,natural language processing,Semantics,text analysis,Text mining,Writing,Writing feedback

Springer International Publishing

abstractWith the development of sophisticated e-learning platforms, educational recommender systems and automatic essay evaluation are becoming an important feature in e-learning systems. Most of the works in educational recommendation techniques are focused on recommending learning materials or learning activities to the learners. In this paper, we proposed and implemented a semantic-based feedback recommendation approach for automatic essay evaluation, which will allow assessors to interact with automatic essay evaluation systems, give feedback on learner's essay solution in the form of textual comments and provide recommendation to other similar essay solution based on the solution which the assessor has given textual feedback. To compute the semantic similarity and to provide feedback recommendation, we used neural word embedding and relaxed word mover's similarity. The proposed approach achieves high-performance accuracy, compared to the state-of-the-art methods, according to our experimental results.
booktitleIntelligent Systems and Applications
seriesAdvances in Intelligent Systems and Computing
titleSemantic-Based Feedback Recommendation for Automatic Essay Evaluation
Educational recommender system,Feedback recommendation,Semantic similarity,Text similarity

abstractDevelop an automated scoring algorithm for student-written essays.
shorttitleThe Hewlett Foundation
titleThe Hewlett Foundation: Automated Essay Scoring

abstractDevelop a scoring algorithm for student-written short-answer responses.
shorttitleThe Hewlett Foundation
titleThe Hewlett Foundation: Short Answer Scoring

abstractLearning to store information over extended time intervals by recurrent backpropagation takes a very long time, mostly because of insufficient, decaying error backflow. We briefly review Hochreiter's (1991) analysis of this problem, then address it by introducing a novel, efficient, gradient based method called long short-term memory (LSTM). Truncating the gradient where this does not do harm, LSTM can learn to bridge minimal time lags in excess of 1000 discrete-time steps by enforcing constant error flow through constant error carousels within special units. Multiplicative gate units learn to open and close access to the constant error flow. LSTM is local in space and time; its computational complexity per time step and weight is O. 1. Our experiments with artificial data involve local, distributed, real-valued, and noisy pattern representations. In comparisons with real-time recurrent learning, back propagation through time, recurrent cascade correlation, Elman nets, and neural sequence chunking, LSTM leads to many more successful runs, and learns much faster. LSTM also solves complex, artificial long-time-lag tasks that have never been solved by previous recurrent network algorithms.
journaltitleNeural Computation
titleLong Short-Term Memory

abstractWe introduce a new language representation model called BERT, which stands for Bidirectional Encoder Representations from Transformers. Unlike recent language representation models (Peters et al., 2018a; Radford et al., 2018), BERT is designed to pretrain deep bidirectional representations from unlabeled text by jointly conditioning on both left and right context in all layers. As a result, the pre-trained BERT model can be finetuned with just one additional output layer to create state-of-the-art models for a wide range of tasks, such as question answering and language inference, without substantial taskspecific architecture modifications.
journaltitlearXiv:1810.04805 [cs]
titleBERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding
Computer Science - Computation and Language,model

abstractNeural networks and in particular the attention mechanism have brought significant advances to the field of Automated Essay Scoring. Many of these systems use a regression-based model which may be prone to underfitting when the model only predicts the mean of the training data. In this paper, we present a dynamic loss function that creates an incentive for the model to predict with the correct distribution, as well as predicting the correct values. Our loss function achieves this goal without sacrificing any performance achieving a Quadratic Weighted Kappa score of 0.752 on the Automated Student Assessment Prize Automated Essay Scoring dataset.
titleThe Effectiveness of a Dynamic Loss Function in Neural Network Based Automated Essay Scoring
023 - The Effectiveness of a Dynamic Loss Function in Ne.pdf;C
Oscar Morris
Computer Science - Artificial Intelligence,Computer Science - Computation and Language

abstractTransformers for Classification, NER, QA, Language Modelling, Language Generation, T5, Multi-Modal, and Conversational AI
titleSimple Transformers

abstractWe present PubMed 200k RCT, a new dataset based on PubMed for sequential sentence classification. The dataset consists of approximately 200,000 abstracts of randomized controlled trials, totaling 2.3 million sentences. Each sentence of each abstract is labeled with their role in the abstract using one of the following classes: background, objective, method, result, or conclusion. The purpose of releasing this dataset is twofold. First, the majority of datasets for sequential short-text classification (i.e., classification of short texts that appear in sequences) are small: we hope that releasing a new large dataset will help develop more accurate algorithms for this task. Second, from an application perspective, researchers need better tools to efficiently skim through the literature. Automatically classifying each sentence in an abstract would help researchers read abstracts more efficiently, especially in fields where abstracts may be long, such as the medical field.
eprintclasscs, stat
shorttitlePubMed 200k RCT
titlePubMed 200k RCT: A Dataset for Sequential Sentence Classification in Medical Abstracts
T a Dataset for Sequential Sentence.pdf;C
Oscar Morris
Computer Science - Artificial Intelligence,Computer Science - Computation and Language,Statistics - Machine Learning

abstractThe abstract of a scientific paper distills the contents of the paper into a short paragraph. In the biomedical literature, it is customary to structure an abstract into discourse categories like BACKGROUND, OBJECTIVE, METHOD, RESULT, and CONCLUSION, but this segmentation is uncommon in other fields like computer science. Explicit categories could be helpful for more granular, that is, discourse-level search and recommendation. The sparsity of labeled data makes it challenging to construct supervised machine learning solutions for automatic discourse-level segmentation of abstracts in non-bio domains. In this paper, we address this problem using transfer learning. In particular, we define three discourse categories BACKGROUND, TECHNIQUE, OBSERVATION-for an abstract because these three categories are the most common. We train a deep neural network on structured abstracts from PubMed, then fine-tune it on a small hand-labeled corpus of computer science papers. We observe an accuracy of 75% on the test corpus. We perform an ablation study to highlight the roles of the different parts of the model. Our method appears to be a promising solution to the automatic segmentation of abstracts, where the labeled data is sparse.
booktitleProceedings of the ACM/IEEE Joint Conference on Digital Libraries in 2020
shorttitleSegmenting Scientific Abstracts into Discourse Categories
titleSegmenting Scientific Abstracts into Discourse Categories: A Deep Learning-Based Approach for Sparse Labeled Data
020 - Segmenting Scientific Abstracts into Discourse Cat.pdf;C
Oscar Morris
Computer Science - Computation and Language,H.3.7,I.5.1