# Non-equilibrium dynamics in a three state opinion formation model with
stochastic extreme switches

Kathakali Biswas  Department of Physics, Victoria Institution (College), 78B
Acharya Prafulla Chandra Road, Kolkata 700009, India.  Department of Physics,
University of Calcutta, 92 Acharya Prafulla Chandra Road, Kolkata 700009,
India.  Parongama Sen  Department of Physics, University of Calcutta, 92
Acharya Prafulla Chandra Road, Kolkata 700009, India.

###### Abstract

We investigate the non-equilibrium dynamics of a three state kinetic exchange
model of opinion formation, where switches between extreme states are
possible, depending on the value of a parameter $q$. The mean field dynamical
equations are derived and analysed for any $q$. The fate of the system under
the evolutionary rules used in BCS  shows that it is dependent on the value of
$q$ and the initial state in general. For $q=1$, which allows the extreme
switches maximally, a quasi-conservation in the dynamics is obtained which
renders it equivalent to the voter model. For general $q$ values, a “frozen”
disordered fixed point is obtained which acts as an attractor for all
initially disordered states. For other initial states, the order parameter
grows with time $t$ as $\exp[\alpha(q)t]$ where $\alpha=\frac{1-q}{3-q}$ for
$q\neq 1$ and follows a power law behaviour for $q=1$. Numerical simulations
using a fully connected agent based model provide additional results like the
system size dependence of the exit probability and consensus times that
further accentuate the different behaviour of the model for $q=1$ and $q\neq
1$. The results are compared with the non-equilibrium phenomena in other well
known dynamical systems.

##  I Introduction

One of the main motivations in studying non-equilibrium phenomena is to check
what kind of steady states can be reached using different initial conditions.
In the well known studies of Ising-Glauber model at zero temperature, on
lattices or networks, several studies have been made to show that the steady
states may not be the equilibrium steady states spirin1 ; spirin2 ; barros ;
svenson ; hagg ; boyer ; castellano ; biswas ; baek ; khaleque ; krap-comp ;
castell2003 ; sood-red2005 ; such-2005-1 . Exit probability, a quantity
related to the type of final state reached from an initially biased state, has
also been studied extensively in recent times in spin and opinion formation
models claudio ; pkm ; sb_ps ; pr_sb_ps ; sb_ps_pr ; prado ; pr_ps15 ; timp ;
pm_ps16 ; pr_ps17 ; pm_ps17 ; sm_sb_ps . In systems with more than two states,
several other interesting features like two stage ordering process has been
noted sm_sb_ps . In addition, how a system evolves to a stable state starting
from an unstable fixed point is also a matter of interest RRPS .

Opinion dynamics models relevant to social phenomena have received extensive
attention recently stauffer ; sen_chak ; soc_rmp ; galam_book . The simplest
models involve binary opinions that cannot capture all the complexities of the
real world. Recently, models with three or more states have been studied,
however, these models generally have some restrictions. In particular, in the
kinetic exchange models with three opinion states quantified by -1,0,1
(assumed to represent e.g., left, central and right ideologies) a transition
from 1 to -1 or vice versa (i.e., an extreme switch) is not allowed to the
best of our knowledge BCS ; vazquez ; gekle ; lima ; migu ; cast ; luca ;
meanfield ; nuno1 ; nuno2 ; sudip . However, human behaviour being complex and
unpredictable such switches cannot be completely ruled out. In fact, there are
real world examples where even political cadres or leaders shift their
allegiance to parties with totally opposite principles tele ; express . The
reasons may be associated with immediate gains and selfish interests, lack of
strong ideological beliefs etc.

In this article, we have therefore considered a kinetic exchange model of
opinion dynamics with three states, with the possibility of switching between
extreme opinions. In the mean field approach, the equations for the time
derivatives are set up for the three population densities of different
opinions and solved numerically. We have introduced a parameter $q$ which
governs the probability with which switches between extreme opinions can occur
and studied its effect on the time evolution. $q$ varies between zero and
unity, the zero case is already considered where no such switch is allowed BCS
. Parallely, numerical simulations have been conducted using a fully connected
agent based model. The model and quantities of interest are discussed in the
next section followed by the results presented in section III and finally in
the concluding section, the results are discussed and compared to existing
results in similar models.

##  II Mean Field Kinetic Exchange Model

We have considered a kinetic exchange model (KEM) for opinion formation which
incorporates three opinion values are quantified by $0,\pm 1$. The possible
correspondence with left, central and right ideologies has already been
mentioned. The three opinion values may even mimic a 2-party voting system,
where the the $\pm 1$ opinions represent support for the two parties while
people with zero opinion (the neutral population) are those who refrain from
voting for either of them. The opinion of an individual is updated by taking
into account her present opinion and an interaction with a randomly chosen
individual in the fully connected model. The opinion of the $i$th individual
is denoted by $o_{i}(t)$. The time evolution of $o_{i}$, after an interaction
with the $k$th individual, chosen randomly, is given by

$o_{i}(t+1)=o_{i}(t)+\mu o_{k}(t),$ (1)

where $\mu$ can be interpreted as an interaction parameter. The opinions are
bounded in the sense $|o_{i}|\leq 1$ at all times and therefore $o_{i}$ is
taken as 1 (-1) if it is more (less) then 1 (-1). There is no self-interaction
so $i\neq k$ in general. This evolutionary rule was introduced in BCS . Here
time is assumed to be discrete but one can easily use a continuous time model
as will be done in this paper.

In several previous works BCS ; sm_sb_ps ; meanfield ; nuno1 ; nuno2 ; sudip ,
$\mu$, the interaction parameter, has been chosen randomly, allowing also
negative values albeit being bounded; $|\mu|\leq 1$. Such a bound allows a
transition between opinion values with a difference of maximum $\pm 1$ only.
In the present work, the interaction parameter $\mu$ is allowed to take two
discrete values. The values are $\mu=1$ and $\mu=2$ which occur with
probabilities $1-q$ and $q$ respectively. Hence, for example, if an agent with
opinion $+1$ interacts with another with opinion $-1$ and $\mu=2$, her opinion
can change to -1, the other extreme value. The possibilities of all the
interactions and resulting opinions are shown in Fig.1 for the extreme values
$q=0$ and $q=1$. Note that in the present work, only positive values of $\mu$
are allowed.

Figure 1: The updated opinions of the $i$th individual following an
interaction with another individual (denoted by $k$) for all possible opinion
values at time $t$ are shown for $q=0$ (left panel), which implies $\mu=1$ and
$q=1$ (right panel) for which $\mu=2$.

The densities of the three populations with opinion $0,\pm 1$ are denoted by
$f_{0},f_{\pm 1}$ with $f_{0}+f_{+1}+f_{-1}=1$. The ensemble averaged order
parameter obtained from the time dependent equations for the densities is
given as $\langle O(t)\rangle=f_{+1}-f_{-1}$ with $-1\leq\langle
O(t)\rangle\leq 1$.

Usually, to study the opinion dynamics models, one starts with a random
disordered configuration such that the average opinion is $~{}0$. Given that
there are three states, one can choose this state with different combinations
of $f_{i}^{\prime}$s, keeping $f_{+1}=f_{-1}$. A conventional choice is
$f_{0}=f_{\pm 1}=1/3$.

One can also study the effect of an initial bias in the distribution of
opinions in the starting configuration of the system. The homogeneous
configuration being one with all the densities equal to 1/3, one can consider
a deviation from this such that the net opinion is nonzero by choosing
$f_{0}=1/3$, $f_{+1}=1/3+\Delta/2$ and $f_{-1}=1/3-\Delta/2$. Here
$-2/3\leq\Delta\leq 2/3$. Apart from this case, one can take other initial
configurations which have a net nonzero opinion. We have discussed such cases
as well to show the initial configuration dependence.

We present in this paper the rate equations derived analytically using mean
field theory for the three densities, and study their behaviour as functions
of time. The fixed point analysis of the equations present some interesting
and non-intuitive results. We also obtain the exit probability. Here the exit
probability $E$ is considered as a function of $\Delta$, i.e., $E(\Delta)$ is
the probability that the final configuration has $f_{+1}=1$ starting from
$f_{+1}=1/3+\Delta/2$ and $f_{-1}=1/3-\Delta/2$. The saturation value of
$\langle O\rangle$ is related to $E(\Delta)$ by

$\langle O\rangle_{sat}=2E(\Delta)-1$ (2)

from which the exit probability can be estimated.

We have also conducted numerical simulations by considering an agent based
model where each agent can interact with any other agent. Here, the order
parameter for a given configuration is defined as $\bar{O}(t)=\frac{|\sum
o_{i}(t)|}{N}$ where $N$ is the system size with $\langle\bar{O}\rangle$
denoting the configuration average. To calculate the exit probability
$E(\Delta)$, we directly estimate the fraction of configurations which reach
the consensus state with all opinions equal to 1.

To solve the coupled differential equations, Euler method has been used and in
the Monte Carlo method, system sizes ranging from 100 to $2^{16}$ have been
simulated with number of configurations ranging between $10^{4}$ to $10^{5}$.

From the simulations, it is also possible to estimate the average consensus
times for different system sizes. All the results are presented in the next
section.

##  III Results

We present in this section the mean field analytical solution in detail and
also the results obtained from numerical simulations.

###  III.1 Mean field rate equations

To set up the rate equations for the $f_{i}$’s, we need to treat the time
variable as continuous. Assume that the opinion changes from $i$ to $j$
($i,j=0,\pm 1$) in time $\Delta t$ with the transition rate given by
$w_{i\rightarrow j}$. Then we have the following set of $w_{ij}^{\prime}s$:

$\displaystyle w_{+1\rightarrow+1}$ $\displaystyle=$ $\displaystyle
f^{2}_{+1}+f_{0}f_{+1}$ $\displaystyle w_{0\rightarrow+1}$ $\displaystyle=$
$\displaystyle f_{0}f_{+1}$ $\displaystyle w_{-1\rightarrow+1}$
$\displaystyle=$ $\displaystyle qf_{-1}f_{+1}$ $\displaystyle w_{+1\rightarrow
0}$ $\displaystyle=$ $\displaystyle(1-q)f_{+1}f_{-1}$ $\displaystyle
w_{0\rightarrow 0}$ $\displaystyle=$ $\displaystyle f^{2}_{0}$ $\displaystyle
w_{-1\rightarrow 0}$ $\displaystyle=$ $\displaystyle(1-q)f_{-1}f_{+1}$
$\displaystyle w_{+1\rightarrow-1}$ $\displaystyle=$ $\displaystyle
qf_{+1}f_{-1}$ $\displaystyle w_{0\rightarrow-1}$ $\displaystyle=$
$\displaystyle f_{0}f_{-1}$ $\displaystyle w_{-1\rightarrow-1}$
$\displaystyle=$ $\displaystyle f^{2}_{-1}+f_{0}f_{-1}$

Hence, in general, we have $f_{i}(t+\Delta t)=f_{i}(t)+\sum_{j}w_{j\to
i}\Delta t-\sum_{j}w_{i\to j}\Delta t$ such that taking $\Delta t\to 0$, we
get

$\frac{df_{+1}}{dt}=f_{0}f_{+1}-(1-q)f_{+1}f_{-1},$ (3)

and

$\frac{df_{-1}}{dt}=f_{0}f_{-1}-(1-q)f_{-1}f_{+1}.$ (4)

The time evolution of the ensemble averaged order parameter $\langle
O(t)\rangle$ satisfies

$\frac{d\langle O(t)\rangle}{dt}=f_{0}\langle O(t)\rangle.$ (5)

Figure 2: The order parameter versus time variation near the frozen fixed
point shows an exponential growth given by $\exp[\alpha(q)t]$ for any $q\neq
1$ as is evident from the (a) analytical as well as (b) simulation results.
Also shown are the best fitted curves of the form given in eq. 8.

###  III.2 Fixed points and steady states

There will be some trivial fixed points corresponding to the initial
conditions that have any of the three densities equal to 1. Here, obviously,
there will be no evolution of the system at all. We consider more general
cases in the following.

Equation 5 shows that a steady state for $\langle O\rangle$ is obtained when
$\langle O\rangle=0$ and/or $f_{0}=0$. Consider first the case when we have a
disordered steady state, i.e., $\langle O(t\to\infty)\rangle=0$. If the
initial state is disordered, eq. 5 indicates that $O$ will remain zero, i.e.,
will not evolve although the individual densities may change in time. We show
in the following that for all values of $q$, there exists a non-trivial
disordered fixed point at which there is no evolution of not only the order
parameter but also of the individual densities. This special fixed point may
be termed the frozen fixed point (FFP) as the system does not undergo any
change at all right from the beginning, although none of the densities have
value unity.

At the FFP, $f_{+1}=f_{-1}=x>0$ is a constant in time. Using this in eq. 3 or
4, one gets

$\dfrac{dx}{dt}=x-(3-q)x^{2}=0.$ (6)

Ignoring the solution $x=0$, we get $x=\dfrac{1}{3-q}$, i.e., the fixed point
is given by

$f_{+1}=f_{-1}=\dfrac{1}{3-q};f_{0}=\dfrac{1-q}{3-q}.$ (7)

The stability of the FFP can be checked by introducing small deviations about
these values. These deviations can be introduced in different ways. We first
consider a deviation such that the initial state has a nonzero order. Taking
$f_{+1}(t)=x^{*}+\delta$ and $f_{-1}(t)=x^{*}-\delta$ where
$x^{*}=\frac{1}{3-q}$ is the FFP value, we get from eq. 3

$\displaystyle\dfrac{d(x^{*}+\delta)}{dt}=(x^{*}+\delta)-(2-q)(x^{*}+\delta)(x^{*}-\delta)-(x^{*}+\delta)^{2}.$

Linearising the above, one finally gets

$\delta(t)=\delta_{0}\exp[\alpha(q)t],$ (8)

where $\delta_{0}$ is the initial value of $\delta$ and

$\alpha(q)=\dfrac{1-q}{3-q}.$ (9)

The order parameter which is equal to $2\delta(t)$ should show the same
behaviour and indeed both the analytical solution and simulations show the
expected initial exponential growth with the value of the exponent very close
to $\alpha$ given by eq. 9 (see Fig. 2). The simulation results have some
finite size effects which is not unexpected (Fig. 2b). The fact that
$\alpha(q)>0$ (for $q\neq 1$) implies the FFP is an unstable one for all
values of $q$ (except $q=1$) when the deviation favours a finite order.

On the other hand, if we start from any disordered state, it can be shown that
the system will flow towards the FFP. Here, with $f_{+}=f_{-1}$ initially,
they will remain the same in time as indicated by the rate equations. Hence
the state can be characterised by $f_{\pm}=x^{*}+\rho$ and
$f_{0}=1-2(x^{*}+\rho)$. In this case, we obtain

$\rho(t)=\rho_{0}\exp[-t],$ (10)

i.e., the state flows to the FFP with a rate independent of $q$. Here
$\rho_{0}$ is the initial value of $\rho$. Hence the FFP acts as an attractor
for all initially disordered state. We have checked that the above form is
indeed obeyed for any value of $q$ (not shown).

###  III.3 Time evolution of the densities and the order parameter

Having obtained the fixed point and the behaviour of the system close to it
for any value of $q$, we proceed to study the time evolution of the relevant
variables in more detail in this subsection. We will first discuss this for
$q=1$, which is obviously a special point in the parameter space. For other
values of $q$ also, we present the results which show consistency with the
theoretical analysis. The data for the time evolution of the three densities
and the order parameters have been obtained by numerically solving the
analytical equations and also using Monte Carlo simulations. Only two of these
four quantities are independent, however, it is more informative to present
the results for all of them.

Figure 3: Results for $q=1$ obtained from the analytical solution using the
initial configurations given in Table 1. The three densities and the ensemble
averaged order parameter are shown as functions of time in (a), (b), (c) and
(d) respectively. The time evolution of sets I and II merge within a few steps
as expected. Figure 4: Results for $q=1$ obtained from Monte Carlo simulation
(of system size $N=2^{10}$ ) using the initial configurations given in Table
1. The three densities and the ensemble averaged order parameter are shown as
functions of time in (a), (b), (c) and (d) respectively.

We have used four different sets of initial conditions stated in Table 1. Of
these, sets I and II are both disordered with set I corresponding to the
homogeneous case. Set III represents an arbitrary initial condition that
favours order. Set IV is also ordered initially and can be regarded as a small
deviation from set II. For all these cases, for $q=1$, $f_{0}$ falls rapidly
within a few steps as shown in Fig. 3a and Fig. 4a obtained using both the
methods. This behaviour of $f_{0}$ may be easily understood from the
transition possibilities, as we note (see Fig. 1) that for opinion zero, there
is no flux to this state from opinions values $\pm 1$ while there is an
outgoing flux when the zero opinion changes to other values. This leads to the
behaviour of $\langle O(t)\rangle$ in eq. 5 as $\frac{d\langle
O\rangle}{dt}\approx 0$, i.e., a quasi-conservative system is obtained. Note
that $f_{0}\to 0$ implies $f_{\pm 1}$ are independent of time for $q=1$, but
not necessarily equal to 1 or 0. Hence, the consensus state (i.e., either
$f_{+1}$ or $f_{-1}$ equal to 1) is not reached in general such that the value
of the ensemble averaged opinion is less than 1. We exclude here the trivial
cases where $f_{+1}=1$ or $f_{-1}=1$ initially. The results using the mean
field equations are shown in Fig. 3 for different initial states given in
Table 1. It is seen that as expected, for sets I and II, which are disordered
initially, the system evolves to the FFP $0,1/2,1/2$. For the other sets we
see that the system reaches a steady state (which is not a consensus state)
within a few steps with the final value of the order parameter close to the
initial one and $f_{0}=0$. For sets III and IV, we have used initial states
with a bias to the +1 opinion and $O(t)$ is therefore positive in all the
cases.

The corresponding simulation results are shown in Fig. 4. Here the consensus
states are reached for all the sets of initial states including the disordered
ones (sets I and II). This is because, in simulations, since we have a finite
system size, a random fluctuation can drive the system to a consensus state in
an individual configuration. Therefore the data which are shown for the
ensemble average of the absolute value of the order parameter shows
$\langle\bar{O}\rangle\to 1$ at large times for all initial states. This is
analogous to the kinetics in Ising Glauber model at zero temperature in one
dimension, where we have a conservation such that the ensemble averaged
magnetisation is zero. In simulations, however, an individual configuration
indeed reaches the all spin up/down state so that the absolute value of
magnetisation reaches unity even after configuration averaging.

Initial configuration | $f_{0}$ | $f_{+1}$ | $f_{-1}$
---|---|---|---
Set-I | 1/3 | 1/3 | 1/3
Set-II | 1/2 | 1/4 | 1/4
Set-III | 5/10 | 3/10 | 2/10
Set-IV | 1/2 | 1/4 + 0.01 | 1/4 - 0.01
Table 1: Fraction of neutral opinion, positive opinion and negative opinion
considered for the initial configuration.  Figure 5: Results for $q=0$
obtained from analytical solution using the initial configurations given in
Table 1. The three densities and the ensemble averaged order parameter are
shown as functions of time in (a), (b), (c) and (d) respectively. The time
evolution of sets I and II merge within a few steps as expected. Figure 6:
Results for $q=0$ obtained from Monte Carlo simulations ( of system size
$N=2^{10}$ ) using the initial configurations given in Table 1. The three
densities and the ensemble averaged order parameter are shown as functions of
time in (a), (b), (c) and (d) respectively. Figure 7:  The order parameter
versus time variations for the initial condition $f_{\pm}=1/3\pm\Delta/2$
shown for different values of $q$ using (a) analytical method and (b)
numerical simulation. The best fitting curves for the growth shows an
exponential form $\exp[\beta(q)t]$.  Figure 8: (a) The values of $\alpha$ and
$\beta$ (suffix A and S denoting analytical and simulation results
respectively) shown against $q$ agree very well with the analytical form given
by eq. 9. (b) The order parameter against time for $q=1$ obtained using the
numerical simulations is shown to follow a power law behaviour with the
exponent $=0.49\pm 0.01$. The initial conditions are the same as in Fig. 2b
and 7b.

Let us next discuss the case for $q=0$, the other extreme limit. This is the
case when extreme switches are not allowed and is identical to the model
considered in BCS  when all interactions are positive and equal to one. In
that case, an ordered state is expected at long times. However, as already
discussed in the last subsection, the $\Delta=0$ point is the FFP here leading
to $\langle O(t)\rangle=0$ for all $t$ when the time evolution is studied
using the analytical equations. The results are presented in Fig. 5. We note
that the set I does not evolve at all and for set II, the densities evolve
before terminating at the FFP, consistent with the analysis presented in the
previous subsection. Initial states with non-zero order show that the system
reaches a consensus state. We also observe that if the initial state is close
to a disordered state (set IV), the system spends a longer time to reach
consensus. Once again, in the numerical simulations
$\langle\bar{O}(t\to\infty)\rangle=1$ for all initial configurations as shown
in Fig. 6. In the completely disordered case, again random fluctuations are
responsible for driving an individual system to a consensus state.

For other values of $q\neq 1$, the qualitative behaviour of the time evolution
is similar to that of $q=0$. One gets consensus states starting from initially
biased states. The disordered states flow to the FFP when the time evolution
is studied by solving the differential equations numerically, as expected.
Consensus is reached in individual configurations starting from any initial
state in the numerical simulations.

We have already discussed the growth of the order parameter for initially
ordered states with small deviations from the FFP taken in a particular
manner. In this section we discussed the time evolution using various other
initial configuration. It is found that any initially ordered state finally
attains consensus and the the growth of the order parameter is found to be
exponential in all cases given by $\exp[\beta(q)t]$. In particular we show in
Fig. 7 the case when the initial condition is $f_{\pm}=1/3\pm\Delta/2$ with
$\Delta=0.002$.

The values of $\alpha(q)$ and $\beta(q)$ obtained from the numerical solution
of the rate equations as well as using Monte Carlo simulations are very close
to each other as shown in Fig. 8. So we conclude that when the system orders
for $q\neq 1$, the initial growth of the order parameter is given by a unique
exponential form, independent of the initial condition, with the exponent
given by eq. 9.

Since the magnitude of the order parameter increases, a steady state must
imply $f_{0}=0$. With $f_{0}=0$, we have from equations 3 and 4 that in the
steady state, either $f_{+1}$ or $f_{-1}$ must be zero (or unity). Hence the
consensus state will be reached for all $q\neq 1$.

For $q=1$, although the analytical results show that the consensus states are
not reached in the thermodynamic limit, for finite systems, we find a unique
behaviour for the growth of the order parameter from the numerical simulation.
Instead of exponential, it displays a slower power law variation with the
exponent very close to 0.5. The data are presented in Fig. 8b.

###  III.4 Consensus times

From the simulations, $\tau$, the average time to reach the consensus state
has been estimated for different system sizes. Once again, we find different
behaviour for $q\neq 1$ and $q=1$. For $q=1$, $\tau\propto N$, while for other
values of $q$, the consensus time $\tau$ depends logarithmically on the system
size, with $\tau$ increasing with $q$. Fig. 9 shows the data.

Figure 9: Variation of average consensus time for different values of $q$. The
main plot shows the $q=1$ results obtained using system sizes up to
$N=2^{14}$. Inset shows results for two values of $q\neq 1$ obtained in system
sizes $\leq 2^{16}$ agents.

###  III.5 Biased initial conditions and exit probability

Figure 10:  (a) $E(\Delta)$ against $\Delta$ obtained in the analytical method
using eq. 2 for different values of $q$ (b) $E(\Delta)$ for $q=1$ using the
numerical simulations for different system sizes shows a system-size
independent linear behaviour. Figure 11: (a) Data collapse of $E(\Delta)$ for
different system sizes is shown against scaled $\Delta$ values for $q=0$.
Inset shows the raw data. (b) The variation of the parameter $\lambda$ against
$q$.

To calculate the exit probability, we consider the biased initial condition
$f_{0}=1/3$ and $f_{\pm 1}=1/3\pm\Delta/2$. As already mentioned, we calculate
the exit probability as a function of $\Delta$.

The exit probability for $q=1$ has a completely different behaviour compared
to other values of $q$. We find that it has linear variation given by
$E(\Delta)=1/2+3\Delta/4$ shown in Fig. 10a. A linear behaviour is expected in
conserved systems but it is intriguing that even here, there is a linear
behaviour although the system is not exactly conserved. In this case, the
simulations also agree as we take into account whether the consensus reached
is for all $+1$ or all $-1$ states. Fig. 10b shows the results are consistent
with a linear variation of $E(\Delta)$ when $q=1$ and shows that it is also
independent of the system size.

Analytical results for the exit probability, for $q\neq 1$, shows a step
function like behaviour; $E(\Delta)=1$ for $\Delta>0$ and equal to zero for
$\Delta<0$. The fixed point analysis also indicates $E(\Delta=0)=1/2$. Hence
any biased state with a majority opinion equal to 1 (-1) will end up with all
opinions equal to 1 (-1).

The simulation results for $E(\Delta)$ for $q\neq 1$ shows strong finite size
dependence. When plotted against $N^{1/\bar{\nu}}\Delta$, the data collapse in
a single curve indicating

$E(\Delta)=g(N^{1/\bar{\nu}}\Delta),$ (11)

where $g$ is a scaling function. This is true for any value of $q\neq 1$ with
a universal value of $\bar{\nu}\approx 2$. Fig. 11a shows the data for $q=0$.

The scaling function $g$ in eq. 11 can be approximated by

$g(y)=[1+\tanh(\lambda y)]/2,$ (12)

as obtained earlier in a few other models pr_sb_ps ; pr_ps17 ; pm_ps17 ;
sm_sb_ps ; RRPS . We find that $\lambda$ decreases as the value of $q$
increases from zero (Fig. 11b.).

In the thermodynamic limit, the exit probability shows the step function
behaviour. However, for finite systems, it is $S$ shaped and from the finite
size analysis we can conclude that the range of $\Delta$ over which it is
neither zero or unity to a large extent, is inversely proportional to $\lambda
N^{1/\bar{\nu}}$.

We end this section commenting that the two different behaviour of the exit
probability shown in Fig. 10a are analogous to the Ising Glauber model in
dimensions greater than unity and voter model (in any dimension) respectively
for $q\neq 1$ and $q=1$.

##  IV Summary and Discussions

In this paper, the evolution of the opinions in a kinetic exchange model has
been studied using both analytical and numerical methods. The three discrete
opinion values used here are quantised by $0,\pm 1$. The mean field
differential equations for the rate of change of the population densities
having the three opinions have been derived and analysed. Here, the parameter
$q$ determines the value of the interaction $\mu$, a random variable, which
can have binary values 1 and 2. When $\mu=2$, which occurs with a probability
$q$, there is a possibility that the opinion value switches from one extreme
value to the other. The $q=0$ case, where $\mu$ can have a single value equal
to unity, has been considered earlier in several studies in different
contexts.

Let us first summarise the main results obtained:
(a) Any initially ordered state will reach a consensus state for $q\neq 1$.
(b) A frozen disordered fixed point exists; all initially disordered states
flow there.
(c) The growth of the order parameter is exponential for $q\neq 1$.
(d) A quasi-conservation exists for $q\neq 1$ leading to different saturation
behaviour of the order parameter and exit probability.

The results are qualitatively different for $q=1$ and $q\neq 1$. The
analytical solution, which is valid in the thermodynamic limit, shows that for
$q=1$ the dynamics are quasi-conservative as the order parameter remains
constant after a very short transient time. This indicates that the system
does not order fully for any initial configuration with initial order
parameter less than 1. The linear behaviour of the exit probability is similar
to what is seen for a conservative dynamics as for example in the Voter model
in all dimensions and the Ising Glauber model in one dimension. This is
actually quite interesting, as the present model does not strictly conserve
the order parameter; the saturation value is not exactly equal to the initial
one. But the linear behaviour of the exit probability can still occur if the
saturation value of the order parameter varies linearly with the initial value
which we have checked to be true here.

The $q=1$ case is in fact very similar to the Voter model; as $f_{0}$ goes to
zero very fast, it effectively renders the system to a binary opinion model
within a short time scale with the transition rates identical to those in the
Voter model bennaim . Like the voter model, here the agent adapts the opinion
of the other agent with whom she interacts irrespective of her own opinion. We
also obtain the result that the average consensus time is proportional to $N$
for $q=1$, a result valid for the mean field voter model.

In the analytical approach, one essentially obtains the ensemble averages in
the thermodynamic limit. Initial configurations with nonzero order will
eventually reach the consensus state for $q\neq 1$. We also find that this
growth behaviour is unique, i.e., does not depend on the initial state but
only on $q$. This is not surprising as it is expected that there will be a
single time scale in the system. Such exponential growths have been recently
observed in the mean field Ising model with finite coordination number also
RRPS .

The analytical approach also leads to the interesting result that initially
disordered state, that can be realised in many ways, will flow towards the so
called frozen fixed point at a rate independent of $q$. In comparison, in
binary models like the Ising model, the disordered state is unique,
characterised by exactly half of the relevant degrees of freedom belonging to
one state. Hence no such flow can be observed there.

In the numerical simulations, one can keep track of the individual
configurations. For all $q$ values we get a consensus state finally for states
starting from partially disordered states. For $q\neq 1$, this is the same
result obtained from analytical treatment. However it is not expected that
consensus will be obtained for $q=1$ for any initial state and for $q\neq 1$,
for initially fully disordered configurations. This contradictory result
obtained in the simulations is argued to be due to finite size effects. In
finite systems, random fluctuations can drive the system to a consensus state
(which implies that the absolute value of the order parameter is unity) even
if the initial configuration is fully disordered. This has been observed in
spin models also, e.g., in the one dimensional Ising Glauber model for which
the ensemble averaged order parameter is conserved but still consensus states
can be reached in numerical simulations starting from disordered states.
Numerical simulations also show that for $q=1$, the growth follows a power law
behaviour, which is much slower than exponential. As a result, the consensus
time is linear in $N$ for $q=1$, compared to the weak logarithmic dependence
on the system size when $q\neq 1$.

The exit probability for $q\neq 1$ indicates a step function behaviour in the
thermodynamic limit. It shows strong finite size effects as indicated from the
numerical simulations. As observed in some other models, a scaling behaviour
is obtained dictated by two parameters $\bar{\nu}$ and $\lambda$. The value of
$\bar{\nu}$ is independent of $q$, a result similar to that in several other
models where also $\bar{\nu}$ does not depend on the model parameter. However,
the value of $\bar{\nu}\approx 2$ is clearly different from the ones found
earlier for Ising-like and other opinion dynamics models pr_sb_ps ; pr_ps17 ;
pm_ps17 ; sm_sb_ps ; RRPS . $\lambda$, on the other hand is dependent on the
parameter $q$, which was also found to be true in the other models. The linear
variation of the exit probability in the $q=1$ case, independent of system
size, also indicates that one will get minority spreading here galam .

In conclusion, the present results indicate that a society attains stability
when people have less influence on others, i.e., $q$ is small, with the
consensus state attained very fast. Essentially, the $q\neq 1$ model is
qualitatively similar to the $q=0$ model, with a $q$ dependent timescale to
reach the consensus state which diverges as $1/\alpha\propto(1-q)^{-1}$. So
the extreme switches cause a delay in reaching the consensus as they increase
in number. $q=1$, which allows the maximum possible switches between extreme
opinion values, essentially leads to a fragmented society. That this does not
happen usually signifies that real systems may be mimicked by a $q\neq 1$
value in this model. It also shows that an initially disordered society will
remain so when one considers the ensemble average, however, individual
configurations do reach consensus. From the perspective of statistical
physics, we have presented a model with a rich behaviour as $q$ is changed; at
$q=1$ a Voter model-like behaviour is seen that changes to a finite
dimensional spin-model like behaviour for any $q<1$. As future studies, it
will be interesting to consider negative interactions between the agents which
will introduce a noise that can drive a order disorder transitions. This will
also make it closer to reality and would open up the possibility to compare
with time dependent real data. Another interesting possibility is to consider
general opinion values instead of $\pm 1,0$ and introduce transition between
any two states and see how it compares with the present case.

Acknowledgement

The authors acknowledge very useful discussions with Soumyajyoti Biswas. PS
acknowledges financial support from SERB (Government of India) through scheme
no MTR/2020/000356.

## References

  * (1) S. Biswas, A. Chaterjee, P. Sen, Disorder induced phase transition in kinetic models of opinion formation, Physica A 391, 3257 (2012).
  * (2) V. Spirin, P. L. Krapivsky, S. Redner, Fate of zero-temperature Ising ferromagnets, Phys. Rev. E 63, 036118 (2001).
  * (3) V. Spirin, P. L. Krapivsky and S. Redner, Freezing in Ising ferromagnets, Phys Rev. E 65, 016119 (2001).
  * (4) K. Barros, P. L. Krapivsky and S. Redner, Freezing into stripe states in two-dimensional ferromagnets and crossing probabilities in critical percolation, Phys. Rev. E. 80, 040101(R) (2009).
  * (5) P. Svenson, Freezing in random graph ferromagnets, Phys. Rev. E 64, 036122 (2001).
  * (6) O. Häggström, Zero-temperature dynamics for the ferromagnetic Ising model on random graphs, Physica A 310, 275 (2002).
  * (7) D. Boyer, O. Miramontes, Interface motion and pinning in small-world networks, Phys. Rev. E 67, 035102(R) (2003).
  * (8) C. Castellano, V. Loreto, A. Barrat, F. Cecconi, D. Parisi, Comparison of voter and Glauber ordering dynamics on networks, Phys. Rev. E 71, 066107 (2005).
  * (9) S. Biswas, P. Sen, Effect of the nature of randomness on quenching dynamics of the Ising model on complex networks, Phys. Rev. E 84, 066107 (2011).
  * (10) Y. Baek, M. Ha, H. Jeong, Absorbing states of zero-temperature Glauber dynamics in random networks, Phys. Rev. E 85, 031123 (2012).
  * (11) A. Khaleque, P. Sen, Frozen states and active-absorbing phase transitions of the Ising model on networks, Journal of Complex Networks 4, 330 (2016).
  * (12) E. Ben-Naim, L. Frachebourg, P. L. Krapivsky, Coarsening and persistence in the voter model, Phys. Rev. E 53, 3078 (1996).
  * (13) C. Castellano, D. Vilone, A. Vespignani, Incomplete ordering of the voter model on small-world networks, EPL 63, 153 (2003).
  * (14) V. Sood, S. Redner, Voter model on heterogeneous graphs, Phys. Rev. Lett. 94, 178701 (2005).
  * (15) K. Suchecki, V. M. Eguíluz, M. San Miguel, Voter model dynamics in complex networks: Role of dimensionality, disorder, and degree distribution, Phys. Rev. E 72, 036132 (2005).
  * (16) C. Castellano, R. Pastor-Satorras, Irrelevance of information outflow in opinion dynamics models, Phys Rev. E 83, 016113 (2011).
  * (17) Przybyla, P., Sznajd-Weron, K. and Tabiszewski, M.: Exit probability in a one-dimensional nonlinear q-voter model, Phys. Rev. E 84, 031117 (2011).
  * (18) S. Biswas, S. Sinha and P. Sen, Opinion dynamics model with weighted influence: Exit probability and dynamics, Phys Rev. E 88, 022152 (2013).
  * (19) P. Roy, S. Biswas and P. Sen, Universal features of exit probability in opinion dynamics models with domain size dependent dynamics, J. Phys. A: Math. Theor. 47 495001 (2014).
  * (20) P. Roy, S. Biswas and P. Sen, Exit probability in inflow dynamics: nonuniversality induced by range, asymmetry and fluctuation, Phys Rev. E 89, 030103(R) (2014).
  * (21) A. M. Timpanaro and C. P. C. Prado, Exit probability of the one-dimensional q-voter model: Analytical results and simulations for large networks, Phys. Rev. E 89, 052808 (2014).
  * (22) P. Roy and P. Sen, Exit Probability in Generalised Kinetic Ising Model’, J Stat Phys 2015 159, 893 (2015).
  * (23) A. M. Timpanaro, and S. Galam, An analytical expression for the exit probability of the q-voter model in one dimension, Phys. Rev. E 92, 012807 (2015).
  * (24) P. Mullick and P. Sen, Minority-spin dynamics in nonhomogeneous Ising model: Diverging timescales and exponents, Phys Rev. E 93, 052113 (2016).
  * (25) P. Roy and P Sen, Interplay of interfacial noise and curvature-driven dynamics in two dimensions, Phys Rev. E 95, 020101(R) (2017).
  * (26) P. Mullick and P. Sen, Zero-temperature coarsening in the Ising model with asymmetric second-neighbor interactions in two dimensions, Phys. Rev. E 95, 052150 (2017).
  * (27) S. Mukherjee, S. Biswas and P. Sen, Long route to consensus: Two stage coarsening in a binary choice voting model, Phys. Rev. E 102, 012316 (2020).
  * (28) R. Roy, P. Sen Non-equilibrium dynamics in Ising like models with biased initial condition Phys. Rev. E 104, 034123 (2021)
  * (29) D. Stauffer, Opinion dynamics and sociophysics, in: R. Meyers (eds) Encyclopedia of Complexity and Systems Science. Springer, New York, NY (2009).
  * (30) P. Sen, B. K. Chakrabarti, Sociophysics: An Introduction, Oxford University Press, Oxford (2014).
  * (31) C. Castellano, S. Fortunato, V. Loreto, Statistical physics of social dynamics, Rev. Mod. Phys. 81, 591 (2009).
  * (32) S. Galam, Sociophysics: A Physicist’s Modeling of Psycho-political Phenomena, Springer, Boston, MA (2012).
  * (33) F Vazquez, P L Krapivsky, S Redner, Constrained opinion dynamics: freezing and slow evolution J. Phys. A: Math. Gen. 36 L61 (2003).
  * (34) S. Gekle, L. Peliti and S. Galam, Opinion dynamics in a three-choice system. Eur. Phys. J. B 45, 569 (2005).
  * (35) S. Galam, The Drastic Outcomes from Voting Alliances in Three-Party Democratic Voting (1990 $\to$ 2013), J. Stat. Phys. 151, 46 (2013).
  * (36) F. W. S. Lima, J. A. Plascak, Kinetic models of discrete opinion dynamics on directed Barabasi-Albert networks, Entropy 2019, 942 (2019).
  * (37) X. Castelló, V. M. Eguíluz and M. San Miguel, Ordering dynamics with two non-excluding options: bilingualism in language competition, New Journal of Physics 8, 308 (2006).
  * (38) X. Castelló, A. Baronchelli, V. Loreto, Consensus and ordering in language dynamics, Eur. Phys. J. B 71, 557 (2009).
  * (39) L. Dall’Asta and T. Galla, Algebraic coarsening in voter models with intermediate states, Journal of Physics A: Mathematical and Theoretical, 41, 435003 (2008).
  * (40) S. Biswas, Mean-field solutions of kinetic-exchange opinion models, Phys. Rev. E 84, 056106 (2011).
  * (41) N. Crokidakis, C. Anteneodo, Role of conviction in nonequilibrium models of opinion formation, Phys. Rev. E 86, 061127 (2012).
  * (42) N. Crokidakis, Phase transition in kinetic exchange opinion models with independence, Phys. Lett. A 378, 1683 (2014).
  * (43) S. Mukherjee, A. Chatterjee, Disorder-induced phase transition in an opinion dynamics model: Results in two and three dimensions, Phys. Rev. E 94, 062317 (2016).
  * (44) https://www.telegraphindia.com/west-bengal/bengal-assembly-elections-2021-480-cpm-cadres-cross-over-to-bjp/cid/1798215
  * (45) https://www.newindianexpress.com/cities/thiruvananthapuram/2018/dec/22/2-bjp-state-leaders-cross-over-to-cpm-1914895.html
  * (46) P. L. Krapivsky, S. Redner, E. Ben-Naim, A Kinetic View of Statistical Physics, Cambridge University Press, Cambridge (2010).
  * (47) S. Galam, Minority opinion spreading in random geometry, Eur. Phys. J. B 25, 403 (2002).


11institutetext: University of Massachusetts Amherst 22institutetext: Eedi
Contact Emails: 22email<EMAIL_ADDRESS>
# Improving the Validity of Automatically Generated Feedback via Reinforcement
Learning

Alexander Scarlatos 11 Digory Smith 22 Simon Woodhead 22 Andrew Lan 11

###### Abstract

Automatically generating feedback via large language models (LLMs) in
intelligent tutoring systems and online learning platforms has the potential
to improve the learning outcomes of many students. However, both feedback
generation and evaluation are challenging: feedback content has to be valid
especially in subjects like math, which requires models to understand the
problem, the solution, and where the student’s error lies. Feedback also has
to be pedagogically valid to reflect effective tutoring strategies, such as
explaining possible misconceptions and encouraging the student, among other
desirable features. In this work, we address both problems of automatically
generating and evaluating feedback while considering both correctness and
alignment. First, we propose a rubric for evaluating math feedback and show
that GPT-4 is able to effectively use it to annotate human-written and LLM-
generated feedback. Second, we propose a framework for feedback generation
that optimizes both correctness and alignment using reinforcement learning
(RL). Specifically, we use GPT-4’s annotations to create preferences over
feedback pairs in an augmented dataset for training via direct preference
optimization (DPO). We show that our methods significantly increase the
correctness and alignment of generated feedback with Llama 2, an open-source
LLM, qualitatively analyze our generation and evaluation systems using case
studies, and outline several areas for future work.111Our code is available at
https://github.com/umass-ml4ed/feedback-gen-dpo.

###### Keywords:

Feedback Generation Human Preference Alignment Math Education Reinforcement
Learning

##  1 Introduction

Providing students with helpful feedback can be critical to their learning,
allowing them to quickly address and learn from mistakes. Prior work has shown
that delivering immediate automated feedback to students in intelligent
tutoring systems and online learning platforms can improve learning outcomes
[12, 23]. However, doing so is challenging since generated feedback should
satisfy a wide variety of requirements: it should convey an understanding of
the question and why the student’s response is incorrect, as well as be
aligned with educational goals and pedagogical theory. For example,
identifying student misconceptions, providing hints, and using encouraging
language to promote a growth mindset [2, 31] can be helpful, but simply giving
away the answer could be detrimental.

Moreover, evaluating generated feedback along these dimensions is also
difficult. Automated evaluations must account for both feedback correctness as
well as their alignment with educational goals, which requires a thorough
understanding of both. Additionally, even when expert-written feedback
examples are given as reference, text similarity-based metrics may be
unreliable since there are many ways to write valid feedback, and text overlap
can emphasize irrelevant features while neglecting more significant ones [1,
19]. While it is common to use human annotators to evaluate feedback, this
approach requires significant effort and expenses. Therefore, the lack of
reliable, automated feedback _evaluation_ methods becomes a bottleneck for
developing feedback _generation_ methods.

In this work, we propose a framework that both generates and evaluates
feedback messages for incorrect student responses to questions, to improve
both their correctness and alignment with educational goals. We ground our
work in math education but note that our framework could potentially be
generalized to other subjects, such as programming or language learning.
First, we propose a rubric for evaluating generated feedback and show that
LLMs, particularly GPT-4, achieve high agreement with humans in their
evaluations.

Second, we use a reinforcement learning (RL)-based approach to generate
feedback messages where the reward given to generated feedback during training
is based on the evaluation rubric. Moreover, to avoid repeatedly using GPT-4
to evaluate feedback during training, we use direct preference optimization
(DPO) [22], an offline RL algorithm, to align the generated feedback with
educational goals. This approach is similar to aligning LLMs with human [36]
or AI [16] preferences. We experiment on a dataset that consists of feedback
messages written by math teachers for each incorrect option in multiple-choice
questions. Our results show that feedback generated using our framework is
significantly more accurate and aligned with educational goals than baselines.
Notably, on alignment metrics, we approach the performance of humans and
GPT-4, estimated to be a 1T parameter model, using the 7B parameter version of
Llama 2.

##  2 Related Work

###  2.1 Feedback Generation

There are many existing approaches for automatic feedback generation. One
common method is to use engineered features to detect errors in student
responses, and then use a rule-based system to provide relevant feedback or
hints [3, 12, 15, 23, 27, 28]. This method is popular since it is
interpretable and reliable but requires significant human effort to adapt to
new question types. A recent and more general approach to feedback generation
is using large language models (LLMs), either through prompting [1, 19, 21,
30, 33] or fine-tuning [10]. However, prompting pre-trained LLMs requires the
existing models to have good behavior and understanding of educational goals,
but fine-tuning can yield poor results without significant amounts of aligned
training data. We address these concerns in our work by fine-tuning on an
augmented dataset annotated with alignment labels.

###  2.2 Feedback Evaluation

Several recent works have used rubrics to evaluate feedback [10, 30, 33], and
works in other domains have found success in using LLMs to evaluate open-ended
text where their judgements correlate with human judgements [6, 13, 20].
However, most prior works on feedback generation tend to rely on human
annotators for reliable evaluation [1, 9, 10, 30, 33]. One recent work [11]
uses GPT-4 to evaluate math feedback with a rubric and finds high agreement
with human annotations. However, they only use GPT-4 to evaluate human-written
feedback, while we evaluate feedback written by both humans and LLMs.
Including this LLM feedback helps us uncover GPT-4’s shortcomings in feedback
evaluation, particularly that it can struggle to identify when feedback
inaccurately addresses student errors or provides invalid suggestions.

##  3 Methodology

We now detail our framework for the two main tasks of feedback generation and
evaluation. Specifically, we first detail our rubric for feedback evaluation
and how we collect annotations with GPT-4, followed by how we construct an
augmented dataset for training, and finally how we use DPO to fine-tune an LLM
for feedback generation. We first define some notations for our tasks. Given a
dataset $\mathcal{D}$ of $N$ math questions, we define the $i$-th question as
$(q^{(i)},c^{(i)},e^{(i)},\\{d^{(i)}_{j},f^{(i)}_{j}|j\in\\{1,\ldots,M\\})$.
Here, $q^{(i)}$ is the question text, $c^{(i)}$ is the correct answer to the
question, $e^{(i)}$ is a textual explanation of the question’s solution,
$d^{(i)}_{j}$ is an incorrect, student-generated answer to the question,
$f^{(i)}_{j}$ is a textual feedback message to give to a student when their
answer is $d^{(i)}_{j}$, and $M$ is the number of different incorrect answers
given for each question. When discussing individual data points, we omit $i$
and $j$ for notation simplicity. We assume that the feedback messages in the
dataset are human-written, and refer to these as the gold, ground-truth
feedback.

###  3.1 Feedback Evaluation

We now detail our rubric for evaluating feedback given to students for their
incorrect answers. In addition to correctness, we aim to evaluate feedback
messages on their alignment with educational goals, including those associated
with a growth mindset [2, 31]. We take inspiration from prior works using
rubrics for feedback evaluation [10, 30] and include aspects to target common
errors that LLMs make when generating feedback. Specifically, our rubric
evaluates feedback on five different aspects, each of them resulting in a
binary-valued label:

  * •

Correct (COR.) The feedback does not make any incorrect statements and is
relevant to the current question and student answer.

  * •

Revealing (REV.) The feedback does not directly reveal the correct answer to
the student.

  * •

Suggestion (SUG.) The feedback provides suggestions to the student that, when
followed, will guide them towards the correct answer.

  * •

Diagnositic (DIA.) The feedback correctly points out the error the student
made or the misconception underlying their answer.

  * •

Positive (POS.) The feedback is positive and has an encouraging tone.

We now define the rubric function, which assigns labels to any feedback, given
a corresponding question and incorrect answer:

$\displaystyle\operatorname{r}(f|q,c,e,d)$
$\displaystyle=(y_{\text{C}},y_{\text{R}},y_{\text{S}},y_{\text{D}},y_{\text{P}})=\mathbf{y}\in\\{0,1\\}^{5}$

We also define a final scalar-valued rubric score for a feedback message by
aggregating the labels in $\mathbf{y}$, indicating the feedback’s overall
quality:

$\displaystyle
s=y_{\text{C}}\cdot\frac{y_{\text{C}}+y_{\text{R}}+y_{\text{S}}+y_{\text{D}}+y_{\text{P}}}{5}\in[0,1]$

where except for correctness, other rubric aspects are equally weighted. The
final rubric score is 0 if the feedback message is incorrect; otherwise, the
score increases by increments of $0.2$ for every rubric aspect the feedback
satisfies.

While the rubric function can be defined by the output of human annotators,
the cost of evaluating feedback using humans is very high, especially when we
require frequent evaluation such as during RL training. To address this issue,
we use GPT-4, known for its ability to generalize to new tasks, to define a
version of the rubric function, $\operatorname{r}_{\text{GPT-4}}$. Using zero-
shot chain-of-thought prompting [14], we ask GPT-4 yes or no questions related
to each of the 5 labels, and use its output to get an estimated label
$\mathbf{y}^{\prime}$ and corresponding score $s^{\prime}$. During prompt
development, we observed that asking GPT-4 questions performed better than
assigning labels based on a formal rubric, that binary labels performed better
than a likert scale, and that asking the negation of the first two questions
and flipping the labels after improved accuracy. We leave further exploration
of the prompt settings, such as the use of in-context examples, for future
work. We provide an example prompt, output, and corresponding labels in Table
1.

Table 1: Example prompt, output, and labels for GPT-4 feedback evaluation. System |  You are a math education expert.
---|---
Prompt |  Your job is to evaluate feedback given to students on math problems.
Here is the question, the correct solution, the incorrect answer the student
gave, and the feedback given to the student:
Question: emma works 36 hours a week. write this as a fraction in its simplest
form. Correct Answer: $\frac{3}{14}$ Solution: There are 24 × 7 = 168 hours in
a week, so the fraction is 36/168. This simplifies to 18/84 = 9/42 = 3/14.
Incorrect Answer: $\frac{18}{84}$
Feedback: It looks like you have simplified, but not fully.
For the following questions, provide a short explanation and then answer with
"Yes" or "No":
1\. Does the feedback make any incorrect statements?
2\. Does the feedback directly reveal the answer to the question?
3\. Does the feedback give suggestions to the student on how to improve the
answer?
4\. Does the feedback correctly point out the misconception underlying the
student’s answer?
5\. Does the feedback have a positive or encouraging tone?
Output |  1\. Explanation: The feedback is correct in stating that the student has simplified, but not fully. The student stopped at $\frac{18}{84}$, which can be further simplified to $\frac{3}{14}$. So, the feedback does not make any incorrect statements. Answer: No. 2\. Explanation: The feedback does not provide the correct answer. It only indicates that the student’s answer is not fully simplified. Answer: No. 3\. Explanation: The feedback does not provide specific suggestions on how to improve the answer. It only states that the student has not fully simplified the fraction. Answer: No. 4\. Explanation: The feedback correctly identifies that the student’s misconception is not fully simplifying the fraction. Answer: Yes. 5\. Explanation: The feedback is neutral in tone. It points out the error but does not use any positive or encouraging language. Answer: No.
Result |  $y^{\prime}_{\text{C}}=1$, $y^{\prime}_{\text{R}}=1$, $y^{\prime}_{\text{S}}=0$, $y^{\prime}_{\text{D}}=1$, $y^{\prime}_{\text{P}}=0$, $s^{\prime}=0.6$

###  3.2 Data Augmentation

We now detail our method for constructing an augmented dataset, which will be
used for RL training as well as calculating agreement between GPT-4 and human
annotations. For both of these tasks, we require both positive examples, i.e.,
feedback messages that score highly on the rubric, and negative examples,
i.e., feedback that score poorly on the rubric. We observe that while most of
the gold feedback messages in $\mathcal{D}$ score highly, feedback generated
by prompting LLMs can be flawed or unaligned with the evaluation metric, thus
scoring poorly. Therefore, we collect 3 LLM-augmented versions of
$\mathcal{D}$, where each feedback $f^{(i)}_{j}$ is replaced with a generated
version: $\mathcal{D}_{R}$, where feedback is generated using few-shot
prompting with random in-context examples, $\mathcal{D}_{S}$, where feedback
is generated using few-shot prompting with the most similar examples, and
$\mathcal{D}_{Z}$, where feedback is generated using zero-shot prompting. We
refer to the union of the original dataset and LLM-augmented datasets
$\mathcal{D}^{\prime}=\bigcup\\{\mathcal{D},\mathcal{D}_{R},\mathcal{D}_{S},\mathcal{D}_{Z}\\}$
as the augmented dataset. We use GPT-4 to annotate the feedback messages in
the augmented dataset, and describe how we use these annotations for RL
training in the next section.

###  3.3 Direct Preference Optimization

In order to generate feedback that scores highly on the rubric, we leverage
direct preference optimization (DPO) [22], an offline RL algorithm, due to its
simplicity and efficiency. We note that online RL algorithms such as PPO could
also apply to our framework, although they would require training a reward
model and introduce additional technical challenges due to training
instability issues; we leave exploration of such algorithms for future work.
At a high level, DPO trains an LLM on pairs of generated outputs given the
same input, where one is preferred over the other. The goal is to use this
preference information to make the LLM generate outputs that more closely
resemble the preferred outputs seen during training. In our context, the
output is the feedback message, $f$, while the input includes the question and
incorrect answer information, $x=(q,c,e,d)$. During training, we minimize the
DPO objective, i.e.,

$\displaystyle\min_{\theta}-\mathbb{E}_{(x,f_{w},f_{l})\sim\mathcal{D}_{\text{DPO}}}\left[\log\sigma\left(\beta\log\frac{\pi_{\theta}(f_{w}|x)}{\pi_{\text{ref}}(f_{w}|x)}-\beta\log\frac{\pi_{\theta}(f_{l}|x)}{\pi_{\text{ref}}(f_{l}|x)}\right)\right],$

where $f_{w}$ is preferred over $f_{l}$ as the feedback for $x$,
$\mathcal{D}_{\text{DPO}}$ is a curated dataset containing these feedback
pairs and preferences, $\pi_{\theta}$ is the trained LLM, i.e., a text
generation “policy”, parameterized by $\theta$, $\pi_{\text{ref}}$ is a frozen
reference LLM, and $\beta$ is a hyperparameter to control how far
$\pi_{\theta}$ can deviate from $\pi_{\text{ref}}$. We now detail how we
construct $\mathcal{D}_{\text{DPO}}$ using both feedback from the augmented
dataset and mismatched feedback from the gold dataset.

We first leverage the augmented dataset to construct feedback preference
pairs. For each unique $x\in\mathcal{D}^{\prime}$, we have 4 feedback
messages, 1 human and 3 LLM-generated, from which we construct ${4\choose
2}=6$ unique pairs. We then use the score $s^{\prime}$ for each feedback to
determine which feedback is preferred, and exclude pairs that have the same
score. For instance, consider a case where some $x$ has possible feedback
messages $f_{1}$, $f_{2}$, $f_{3}$, and $f_{4}$, with scores $1.0$, $0.8$,
$0.4$, and $0.4$, respectively. We then produce the preference pairs
$(f_{1},f_{2})$, $(f_{1},f_{3})$, $(f_{1},f_{4})$, $(f_{2},f_{3})$, and
$(f_{2},f_{4})$, where the first feedback is the preferred one in each pair.

We also use mismatched feedback from the gold dataset to construct additional
preference pairs. We observe that feedback written for different incorrect
answers to the same question will have many semantically similar features and
often the same variables and numbers. However, despite their similarities, the
feedback written for the corresponding incorrect answer is almost always
better suited than feedback written for other incorrect answers. Therefore,
these mismatched feedback are excellent hard negatives since it is hard for
algorithms to distinguish between them and good feedback; finding such hard
negatives has been shown to be the key to contrastive learning [25]. In
addition to using mismatched feedback from the same question, we construct one
more pair using a feedback from a random question in the gold dataset. For
instance, for $x^{(i)}_{1}$ and $M=3$, we construct the preference pairs
$(f^{(i)}_{1},f^{(i)}_{2})$, $(f^{(i)}_{1},f^{(i)}_{3})$, and
$(f^{(i)}_{1},f^{(i^{\prime})}_{j^{\prime}})$, for some random
$i^{\prime}\in[1,N]$ and $j^{\prime}\in[1,M]$, where $f^{(i)}_{1}$ is
preferred in all pairs.

##  4 Experiments

We now detail all experiments we conduct to validate our framework for
feedback generation and evaluation. First, we demonstrate that our methods
improve the correctness and alignment of generated feedback using both
quantitative evaluation from GPT-4 and qualitative case studies. Second, we
demonstrate that GPT-4 has high agreement with human annotations on our
rubric, justifying its use as an evaluator, and further investigate its
shortcomings using case studies.

###  4.1 Dataset

We validate our framework using a dataset of middle school-level math multiple
choice questions from a math learning platform. The questions cover a variety
of number sense concepts including fractions, exponents, rounding, and many
others. All questions and feedback messages are written by real math teachers,
deployed to real students, and are generally high quality. There are a total
of 1,956 questions in the dataset and each question has a total of 3 incorrect
options and a ground truth human-written feedback for each. We remove
questions that require images and ones with processing errors, resulting in
1,418 questions. We divide these into a train/validation/test split of
850/284/284 questions and correspondingly 2,550/852/852 incorrect answers and
corresponding feedback.

###  4.2 Experimental Setting

##### Data Augmentation

We use two LLMs to generate feedback for our augmented dataset: code-
davinci-002 (Codex) [4] for $\mathcal{D}_{R}$ and $\mathcal{D}_{S}$ since it
has strong few-shot prompting ability, and gpt-3.5-turbo for $\mathcal{D}_{Z}$
since its zero-shot ability is much better than code-davinci-002. We use 2 in-
context examples for few-shot prompts, only select examples from the train
set, and use the S-BERT model all-distilroberta-v1 [24] to measure similarity
for $\mathcal{D}_{S}$. We prompt the models with questions, correct answers
and incorrect answers, but not full solutions to make the task harder and
increase the amount of incorrect feedback. To reduce costs, we randomly select
a subset of $\mathcal{D}^{\prime}$ to be annotated by GPT-4. Specifically, we
take 10,000, 1,000 and 1,000 samples from the train, validation and test sets,
respectively, and remove the remaining samples from the augmented dataset.

##### Feedback Generation Models

We primarily use the instruction-tuned Llama-2 7B Chat model [32] from
HuggingFace [34] for feedback generation, loaded with 8-bit quantization [7].
For both supervised fine-tuning (SFT) and DPO, we train LoRA adapters [8] on
all weight matrices, setting $r=32$, $\alpha=16$, and dropout$=0.05$. We train
using the AdamW optimizer with a learning rate of 3e-4 with warmup for 10% of
steps and an effective batch size of 64 using gradient accumulation. We train
for 3 epochs, which we find minimizes the loss on the validation set. For DPO,
we initialize $\theta$ to the SFT weights, use the SFT model as the reference
model, and set $\beta=0.5$. At inference time, we use greedy decoding and set
the maximum new tokens to 200.

##### Metrics

When evaluating feedback, we report the average of each rubric label in
$\mathbf{y}^{\prime}$ and the corresponding scores $s^{\prime}$ assigned by
GPT-4. We note that GPT-4 will very rarely fail to assign labels when feedback
is unrelated to the current question, in which case we automatically assign
label values of 0. We use a temperature of 0 and 300 maximum new tokens for
GPT-4 decoding. We additionally use two popular reference-based metrics where
we use the human-written feedback as a gold reference: ROUGE-L (ROU.) [17]
which is based on textual overlap, and the F1 of the BERTScore (BER.) [35]
using the recommended microsoft/deberta-xlarge-mnli model, which is based on
token-level semantic similarity.

###  4.3 Feedback Generation

We now show that we can improve both the correctness and alignment of
generated feedback using our framework. We primarily focus on using Llama 2
Chat, an open-source LLM with 7B parameters, where we compare several versions
of the model: Zero-Shot, i.e., simply prompting the base LLM, SFT, i.e., fine-
tuning the base LLM on the gold feedback set, DPO (Score), i.e., training the
base LLM with DPO only on the augmented dataset, and DPO (Score + Mismatch),
i.e., training the base LLM with DPO on the augmented dataset and mismatched
feedback. We additionally compare with the gold, human-written feedback in the
dataset, as well as feedback generated by GPT-4. We use the same prompt for
all methods, where we instruct the model to generate short and helpful
feedback and to follow a version of the evaluation rubric.

####  4.3.1 Quantitative Analysis

Table 2: Quantitative results of feedback generation across methods. Our best method outperforms all Llama 2 baselines in both correctness and alignment. | COR. | REV. | SUG. | DIA. | POS. | Score | ROU. | BER.
---|---|---|---|---|---|---|---|---
Human | 0.91 | 0.98 | 0.67 | 0.82 | 0.41 | 0.73 | 1.00 | 1.00
GPT-4 | 0.95 | 0.96 | 0.99 | 0.93 | 1.00 | 0.94 | 0.19 | 0.57
Zero-shot | 0.63 | 0.63 | 0.74 | 0.43 | 1.00 | 0.49 | 0.16 | 0.55
SFT | 0.65 | 0.98 | 0.49 | 0.68 | 0.19 | 0.49 | 0.29 | 0.61
DPO (Score) | 0.70 | 0.93 | 0.95 | 0.82 | 0.66 | 0.65 | 0.22 | 0.57
DPO (Score + Mismatch) | 0.77 | 0.96 | 0.95 | 0.86 | 0.57 | 0.71 | 0.23 | 0.57

Table 2 shows the average rubric labels and scores assigned by GPT-4 on all
feedback in the test set, as well as the ROUGE-L and BERTScore values for
reference. We see that DPO (Score + Mismatch) significantly improves the
feedback scores compared to baselines (a 45% increase compared to Zero-Shot
and SFT), showing that our data augmentation and training setup is highly
effective at improving the quality of feedback generation with Llama 2. We
additionally observe that including the mismatched feedback messages
substantially increases the correctness of generated feedback, confirming
their effectiveness as hard negative examples. Surprisingly, SFT does not
outperform Zero-Shot on score, which shows that the standard fine-tuning setup
is not effective for the specific task of feedback generation. We can also see
that ROUGE-L and BERTScore are unreliable estimates of feedback quality since
they are highest on SFT, primarily because it copies the style of the gold
feedback the closest.

We also see that GPT-4, a much larger model (rumored to have 1T parameters),
performs almost perfectly across all labels; DPO (Score + Mismatch) can only
match its performance on the revealing and suggestion metrics. However, we
note that these results may be inflated, since we also use GPT-4 in evaluation
and it is likely to believe that its own generations conform to the rubric.
Moreover, we observe that it prefers to be conservative and provides less
specific descriptions of student errors, which leads to high scores under our
evaluation metric; see below for a detailed example. Nevertheless, we
emphasize that a smaller, open-source model is easier for deployment and much
cheaper in real-world educational scenarios than a larger, proprietary model.
Additionally, we see that the gold, human-written feedback does not score
perfectly on correctness, and has a relatively low overall score due to the
suggestion and positive metrics; DPO (Score + Mismatch) achieves a similar
overall performance. However, the primary reason for the lower human
performance is that teachers did not have our evaluation rubrics in mind when
they wrote the feedback.

We also performed qualitative studies to compare the outputs of the different
methods and find cases where they succeed or fail; the main findings are:

  * •

Zero-Shot produces feedback with the right style but struggles to follow
instructions, particularly by not identifying the error or revealing the
correct answer, and is prone to hallucinations and numerical errors.

  * •

SFT produces feedback that is generally short and blunt, and usually attempts
to identify the error although is often incorrect.

  * •

DPO (Score) produces feedback that attempts to identify the error, adding
details and questions to provide implicit suggestions and increase positivity.
It also produces incorrect outputs although less so than SFT.

  * •

DPO (Score + Mismatch) is more accurate than DPO (Score) in identifying the
error.

  * •

GPT-4 produces feedback with smooth and coherent language but tends to avoid
mistakes by not clearly pointing out the error. When it does detail the error,
it is often incorrect, although less so than the other methods.

  * •

For all methods, most failures occur because the LLM cannot correctly identify
the error in the incorrect answer. This failure happens more often when
multiple reasoning steps are required to arrive at the student’s answer. This
observation suggests that although state-of-the-art LLMs claim to excel at
mathematical reasoning, they cannot conceive how students may make errors [11,
18, 19]. It may be necessary to prompt the LLM to perform necessary
calculations before writing the feedback [29] and use calculators or code
execution [5] when doing so.

To provide more specific examples of several of our findings, we show a
representative question and incorrect answer along with corresponding feedback
generated by each method in Table 3. In this example, we see that the human-
written feedback directly points out the error in the incorrect answer that
the student only considered the denominators, and also suggests simplifying
the fraction on the right. GPT-4 prefers to be vague to avoid mistakes by
pointing out that the ratio should be simplified and gives a suggestion based
on this, but does not explicitly identify why the student’s answer was wrong.
Zero-Shot does not point out the student’s error and proceeds to answer the
question, even though it has been instructed to not reveal the answer; it then
hallucinates an incorrect and irrelevant multiplication before finishing with
a positive message. SFT makes a very generic guess at what the error is and
does not give a suggestion for improvement. DPO (Score) guesses a more
specific but incorrect error, gives a somewhat misleading suggestion, and uses
questions to encourage the student. Finally, DPO (Score + Mismatch) correctly
identifies the error and gives a helpful suggestion.

Table 3: Feedback generated by all methods for one question and incorrect
answer.

Input
---
Question: Erica and Rebecca share some marbles in the ratio
$\frac{1}{4}:\frac{6}{8}$. Erica has $\square$ marbles than Rebecca. Which of
the following belongs in the gap?
Correct Answer: a third as many
Incorrect Answer: twice as many
Source |  Feedback
Human |  Not quite, the denominator has doubled but the fractions don’t have that relationship. Try simplifying 6/8 and then compare it to 1/4 ✓
GPT-4 |  Good effort, but remember to simplify the ratios first. Erica’s share is not twice as much, but a fraction of Rebecca’s. Try to compare 1/4 to 3/4 again. Keep going! ✓
Zero-shot |  Great effort! You are on the right track. Since the ratio is 1/4: 3/4, Erica has 1/4, which is a third of 3/4. To find the number of marbles Erica has, you can multiply 3/4 by 3, which gives you 9. So, Erica has 9 marbles. Keep practicing, and you’ll get it right next time! ✗
SFT |  I think you have simplified the second fraction incorrectly. ✗
DPO (Score) |  I think you have simplified 6/8 to 1/2. If we simplify 6/8 to 3/4, how many marbles does Erica have? How many does Rebecca have? What is the difference between them? ✗
DPO (Score + Mismatch) |  I think you have found 8/4 = 2. But we need to simplify 6/8 to get the same units as 1/4. Then we can see how many more or less Erica has. ✓

###  4.4 Feedback Evaluation

Table 4: GPT-4’s agreement with human annotations across all rubric labels. Label | Acc. | Prec. | Rec. | F1
---|---|---|---|---
COR. | 0.78 | 0.78 | 0.85 | 0.81
REV. | 0.96 | 1.00 | 0.96 | 0.98
SUG. | 0.75 | 0.67 | 0.79 | 0.72
DIA. | 0.69 | 0.58 | 0.85 | 0.69
POS. | 0.82 | 0.78 | 0.37 | 0.50
Avg. | 0.80 | 0.76 | 0.76 | 0.74

Since we use GPT-4 to quantitatively evaluate feedback, we need to verify that
GPT-4 can indeed label feedback accurately using our rubric. To do so, we
randomly sample 80 feedback messages from the augmented test set and manually
evaluate them on the rubric. To reduce bias, we do not show the human
annotator GPT-4’s annotations or tell them whether a feedback message is
human-written or LLM-generated. We measure the accuracy (Acc.), precision
(Prec.), recall (Rec.), and F1 of GPT-4’s annotations with respect to human
annotations on this set, and report the results in Table 4.

We observe that GPT-4 generally agrees with our annotations, with an average
accuracy of 80% across labels. We also compute the Pearson correlation
coefficient of the final rubric scores between GPT-4 and our annotations,
resulting in 0.58, indicating moderate overall correlation. We note that
ROUGE-L and BERTScore have correlations of 0.49 and 0.50 with our rubric
scores, respectively. Not only are these numbers lower, but we note that they
are biased upward since the human-written feedback messages, which generally
have high rubric scores, automatically get ROUGE-L and BERTScore values of 1.

Table 5: Incorrect GPT-4 rubric annotation examples with explanations.

Label |  Input |  Generated Explanation
---|---|---
COR. (FP) |  Question: a jacket is reduced in a sale by $20\%$. a customer has a loyalty card for a further $10\%$ off the price. what is the total percentage reduction in the price of the jacket?
Correct Answer: $28\%$
Incorrect Answer: $8\%$
Feedback: I think you have added the two discounts together. We need to use multiplication to find the overall discount. |  The feedback does not make any incorrect statements. It correctly identifies the student’s mistake of adding the discounts together instead of multiplying them. ✗
COR. (FN) |  Question: i am thinking of a number. i divide it by 0.5 and my answer is 10. what number am i thinking of?
Correct Answer: 5
Incorrect Answer: 20
Feedback: You multiplied by 0.5 instead of dividing by 0.5. |  The feedback is incorrect because the student actually multiplied by 2, not 0.5. The correct feedback should be “You multiplied by 2 instead of dividing by 0.5.” ✗
DIA. (FP) |  Question: which fraction is equivalent $\frac{3}{7}$
Correct Answer: $\frac{9}{21}$
Incorrect Answer: $\frac{9}{7}$
Feedback: You have multiplied the top and bottom by 3, but you have to multiply by 3 again to get the correct answer |  The feedback correctly identifies that the student has not multiplied the denominator by the same number as the numerator, which is the underlying misconception. However, the suggested correction is incorrect. ✗

However, GPT-4 still struggles in a few key aspects and we provide examples of
erroneous annotations in Table 5. Most importantly, GPT-4 tends to assume that
feedback is correct when it sounds convincing but incorrectly identifies the
student error or provides an invalid suggestion. These issues mostly occur
when calculations are required to verify the feedback. Additionally, GPT-4 can
sometimes confuse the roles of variables in the question, leading it to
believe that a valid feedback is incorrect. GPT-4 also has a high false
positive rate on the diagnostic label due to hallucinating statements that
were not made in the feedback. We note that it may be possible to resolve
these issues using additional prompt engineering or tools such as self-
reflection [26] and code execution to evaluate math expressions [5, 29].
Finally, while the suggestion and positive labels have relatively low
agreement with human annotations, we note that these labels can be very
subjective, and that GPT-4’s judgement on these labels is more reasonable than
these accuracy numbers suggest.

##  5 Conclusions and Future Work

In this work, we proposed a framework for automated feedback generation and
evaluation via LLMs for students’ incorrect answers in math multiple-choice
questions. Our framework accounts for both the mathematical correctness of the
feedback and its alignment with good pedagogical practices. We show that using
a data augmentation and preference optimization approach, we can generate
high-quality feedback using Llama 2 7B, a small and open-source LLM. We also
show that GPT-4 can evaluate feedback rather accurately using a rubric and
that its annotations are helpful for training the feedback generation method.
There are many avenues for future work. First, we can apply our framework to
other RL algorithms such as PPO, or non-RL approaches such as overgenerate-
and-rank. Second, we can evaluate our final feedback generation task via a
large-scale human evaluation or classroom study, which would alleviate
concerns on GPT-4’s annotations being biased. Finally, we can test our
framework’s generalizability by applying it to domains other than math, such
as programming or language learning, or other scenarios such as hint
generation or student-instructor conversations.

## References

  * [1] Al-Hossami, E., Bunescu, R., Teehan, R., Powell, L., Mahajan, K., Dorodchi, M.: Socratic questioning of novice debuggers: A benchmark dataset and preliminary evaluations. In: Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023). pp. 709–726. Association for Computational Linguistics, Toronto, Canada (Jul 2023)
  * [2] Boaler, J.: Ability and mathematics: The mindset revolution that is reshaping education. The Forum 55, 143–152 (2013)
  * [3] Botelho, A., Baral, S., Erickson, J.A., Benachamardi, P., Heffernan, N.T.: Leveraging natural language processing to support automated assessment and feedback for student open responses in mathematics. Journal of Computer Assisted Learning 39(3), 823–840 (2023)
  * [4] Chen, M., Others: Evaluating large language models trained on code (2021)
  * [5] Chen, W., Ma, X., Wang, X., Cohen, W.W.: Program of thoughts prompting: Disentangling computation from reasoning for numerical reasoning tasks. arXiv preprint arXiv:2211.12588 (2022)
  * [6] Chiang, C.H., Lee, H.y.: Can large language models be an alternative to human evaluations? arXiv preprint arXiv:2305.01937 (2023)
  * [7] Dettmers, T., Lewis, M., Belkada, Y., Zettlemoyer, L.: Llm.int8(): 8-bit matrix multiplication for transformers at scale (2022)
  * [8] Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models (2021)
  * [9] Jia, Q., Cui, J., Xiao, Y., Liu, C., Rashid, P., Gehringer, E.F.: All-in-one: Multi-task learning bert models for evaluating peer assessments. arXiv preprint arXiv:2110.03895 (2021)
  * [10] Jia, Q., Young, M., Xiao, Y., Cui, J., Liu, C., Rashid, P., Gehringer, E.: Insta-reviewer: A data-driven approach for generating instant feedback on students’ project reports. International Educational Data Mining Society (2022)
  * [11] Kakarla, S., Thomas, D., Lin, J., Gupta, S., Koedinger, K.R.: Using large language models to assess tutors’ performance in reacting to students making math errors. arXiv preprint arXiv:2401.03238 (2024)
  * [12] Kochmar, E., Vu, D.D., Belfer, R., Gupta, V., Serban, I.V., Pineau, J.: Automated personalized feedback improves learning gains in an intelligent tutoring system. In: Artificial Intelligence in Education: 21st International Conference, AIED 2020, Ifrane, Morocco, July 6–10, 2020, Proceedings, Part II 21. pp. 140–146. Springer (2020)
  * [13] Kocmi, T., Federmann, C.: Large language models are state-of-the-art evaluators of translation quality. arXiv preprint arXiv:2302.14520 (2023)
  * [14] Kojima, T., Gu, S.S., Reid, M., Matsuo, Y., Iwasawa, Y.: Large language models are zero-shot reasoners. Advances in neural information processing systems 35, 22199–22213 (2022)
  * [15] Lan, A.S., Vats, D., Waters, A.E., Baraniuk, R.G.: Mathematical language processing: Automatic grading and feedback for open response mathematical questions. In: Proceedings of the second (2015) ACM conference on learning@ scale. pp. 167–176 (2015)
  * [16] Lee, H., Phatale, S., Mansoor, H., Lu, K., Mesnard, T., Bishop, C., Carbune, V., Rastogi, A.: Rlaif: Scaling reinforcement learning from human feedback with ai feedback. arXiv preprint arXiv:2309.00267 (2023)
  * [17] Lin, C.Y.: ROUGE: A package for automatic evaluation of summaries. In: Text Summarization Branches Out. pp. 74–81. Association for Computational Linguistics, Barcelona, Spain (Jul 2004)
  * [18] Liu, N., Sonkar, S., Wang, Z., Woodhead, S., Baraniuk, R.G.: Novice learner and expert tutor: Evaluating math reasoning abilities of large language models with misconceptions. arXiv preprint arXiv:2310.02439 (2023)
  * [19] McNichols, H., Feng, W., Lee, J., Scarlatos, A., Smith, D., Woodhead, S., Lan, A.: Automated distractor and feedback generation for math multiple-choice questions via in-context learning. NeurIPS’23 Workshop on Generative AI for Education (2023)
  * [20] Naismith, B., Mulcaire, P., Burstein, J.: Automated evaluation of written discourse coherence using GPT-4. In: Proceedings of the 18th Workshop on Innovative Use of NLP for Building Educational Applications (BEA 2023). pp. 394–403. Association for Computational Linguistics, Toronto, Canada (Jul 2023)
  * [21] Nguyen, H.A., Stec, H., Hou, X., Di, S., McLaren, B.M.: Evaluating chatgpt’s decimal skills and feedback generation in a digital learning game. In: Responsive and Sustainable Educational Futures. pp. 278–293. Springer Nature Switzerland, Cham (2023)
  * [22] Rafailov, R., Sharma, A., Mitchell, E., Ermon, S., Manning, C.D., Finn, C.: Direct preference optimization: Your language model is secretly a reward model (2023)
  * [23] Razzaq, R., Ostrow, K.S., Heffernan, N.T.: Effect of immediate feedback on math achievement at the high school level. In: International Conference on Artificial Intelligence in Education. pp. 263–267. Springer (2020)
  * [24] Reimers, N., Gurevych, I.: Sentence-bert: Sentence embeddings using siamese bert-networks. In: Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing. Association for Computational Linguistics (11 2019)
  * [25] Robinson, J.D., Chuang, C.Y., Sra, S., Jegelka, S.: Contrastive learning with hard negative samples. In: International Conference on Learning Representations (2021)
  * [26] Shinn, N., Cassano, F., Labash, B., Gopinath, A., Narasimhan, K., Yao, S.: Reflexion: Language agents with verbal reinforcement learning. arXiv preprint arXiv:2303.11366 14 (2023)
  * [27] Singh, R., Gulwani, S., Solar-Lezama, A.: Automated feedback generation for introductory programming assignments. In: Proceedings of the 34th ACM SIGPLAN conference on Programming language design and implementation. pp. 15–26 (2013)
  * [28] Song, D., Lee, W., Oh, H.: Context-aware and data-driven feedback generation for programming assignments. In: Proceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. pp. 328–340 (2021)
  * [29] Sonkar, S., Le, M., Chen, X., Liu, N., Mallick, D.B., Baraniuk, R.G.: Code soliloquies for accurate calculations in large language models. arXiv preprint arXiv:2309.12161 (2023)
  * [30] Steiss, J., Tate, T., Graham, S., Cruz, J., Hebert, M., Wang, J., Moon, Y., Tseng, W., et al.: Comparing the quality of human and chatgpt feedback on students’ writing (2023)
  * [31] Sun, K.L.: Brief report: The role of mathematics teaching in fostering student growth mindset. Journal for Research in Mathematics Education 49(3), 330–335 (2018)
  * [32] Touvron, H., Others: Llama 2: Open foundation and fine-tuned chat models (2023)
  * [33] Wang, R.E., Zhang, Q., Robinson, C., Loeb, S., Demszky, D.: Step-by-step remediation of students’ mathematical mistakes
  * [34] Wolf, T., Debut, L., Sanh, V., Chaumond, J., Delangue, C., Moi, A., Cistac, P., Rault, T., Louf, R., Funtowicz, M., et al.: Huggingface’s transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771 (2019)
  * [35] Zhang, T., Kishore, V., Wu, F., Weinberger, K.Q., Artzi, Y.: Bertscore: Evaluating text generation with bert. In: International Conference on Learning Representations (2020)
  * [36] Ziegler, D.M., Stiennon, N., Wu, J., Brown, T.B., Radford, A., Amodei, D., Christiano, P., Irving, G.: Fine-tuning language models from human preferences. arXiv preprint arXiv:1909.08593 (2019)