# Species competition: coexistence, exclusion and clustering

Emilio Hernández-García1 Cristóbal López1 Simone Pigolotti2 Ken H. Andersen3
1IFISC (UIB-CSIC), Instituto de Física Interdisciplinar y Sistemas Complejos,
Campus Universitat de les Illes Balears, E-07122 Palma de Mallorca, Spain
2The Niels Bohr Institute, The Niels Bohr International Academy, Blegdamsvej
17 DK-2100, Copenhagen, Denmark
3National Institute of Aquatic Resources, Technical University of Denmark,
Charlottenlund slot, DK-2920 Charlottenlund, Denmark

###### Abstract

Competition, Lotka-Volterra, competitive exclusion, limiting similarity,
pattern formation We present properties of Lotka-Volterra equations describing
ecological competition among a large number of interacting species. First we
extend previous stability conditions to the case of a non-homogeneous niche
space, i.e. that of a carrying capacity depending on the species trait.
Second, we discuss mechanisms leading to species clustering and obtain an
analytical solution for a state with a lumped species distribution for a
specific instance of the system. We also discuss how realistic ecological
interactions may result in different types of competition coefficients.

Philosophical Transactions of the Royal Society A 367, 3183-3195 (2009)
http://dx.doi.org/10.1098/rsta.2009.0086
Published under the Creative Commons Attribution license.

##  1 Lotka-Volterra competition and species distribution

Competitive interactions occur when entities in a system grow by consuming
common finite resources. They are ubiquitous in many fields of science:
examples include biological species competing for food (MacArthur & Levins
1967; Roughgarden 1979; Case 1981), mode competition in nonlinear optical
systems (Benkert & Anderson 1991), or alternative technologies competing for a
market (Pistorius & Utterback 1997). An early, simple, but powerful model for
competitive interactions is the Lotka-Volterra (LV) set of competition
equations (Volterra 1926; Lotka 1932):

$\dot{N_{i}}=r_{i}N_{i}\left(1-\frac{1}{K_{i}}\sum_{j=1}^{m}G_{ij}N_{j}\right),\
\ i=1,...,m.$ (1)

where $m$ is the number of species, $N_{i}$ the population of species $i$,
$r_{i}$ its maximum growth rate, $K_{i}$ its carrying capacity, and $G_{ij}$
is the matrix characterizing the interaction among species $i$ and $j$, more
specifically the decreasing on the growth rate of species $i$ by the presence
of $j$. Competitive interactions are characterized by $G_{ij}\geq 0$, the
situation to be considered here, whereas negative interactions may model
situations of mutualism, predation or symbiosis.

In classical ecological niche theory, species are associated to points in an
abstract niche space. Coordinates in this space represent relevant phenotypic
characteristics, for example size of individuals in a species, or the size of
preferred prey, such that intensity of competition is larger if species are
closer in this space. We assume for simplicity this space to be one-
dimensional (multi-dimensional generalizations are straightforward, as briefly
mentioned later). If niche locations can be considered to be a continuum, we
can write Eq. (1) as:

$\partial_{t}\psi(u,t)=r(u)\psi(u,t)\left[1-\frac{1}{K(u)}\int
G(u,v)\psi(v,t)dv\right],\ \ \ \ \ $ (2)

where now $\psi(u,t)$ is the population density at niche location $u$. The
integral extends over the full niche space, which could be finite or infinity.
For most purposes, Eqs. (1) and (2) can be considered as equivalent, since the
second is obtained from the first in the limit of many close interacting
species, and (1) can be recovered from (2) for a discrete distribution of
species:

$\psi(u)=\sum_{i=1}^{m}N_{i}\delta(u-u_{i}),$ (3)

with $G_{ij}=G(u_{i},u_{j})$, $r_{i}=r(u_{i})$ and $K_{i}=K(u_{i})$.

It is widely believed that (1) or (2) predict a competitive exclusion leading
to a limiting similarity situation (Abrams 1983), in which a pair of species
too close in niche space can not coexist, and one of them would become
extinct. However it is known that the model allows for continuous coexistence
of species in some situations (Roughgarden 1979), and refinements on the
conditions for this coexistence have been developed, with emphasis on the
effect of the shape of the carrying capacity function $K(u)$ (Meszéna _et al._
2006; Szabó & Meszéna 2006). In this context, a particulary surprising result
was the finding by Scheffer & van Nes (2006) of a situation –for uniform
carrying capacity– which was neither of full coexistence nor of full
exclusion, but of clusters or lumps of tightly packed species which did not
exclude each other, but were well separated from other clusters so that there
was a type of limiting similarity leading to a minimum intercluster distance.
Clustering of individuals or entities under competitive interactions of the LV
type had been already observed in other contexts (Fuentes _et al._ 2003;
Hernández-García & López 2004, 2005; Ramos _et al._ 2008), where the mechanism
was the diffusive broadening of an otherwise zero-width species or entity. In
contrast, the lumps in Scheffer & van Nes (2006) appeared even in the absence
of diffusion in niche space, which is the situation also considered here.

The importance of the functional form of the interaction kernel $G_{ij}$ in
(1) or $G(u,v)$ in (2) was stressed by Pigolotti _et al._ (2007) for the case
of uniform carrying capacity and interactions depending only on differences of
niche positions, and found to be relevant in an evolutionary context by Leimar
_et al._ (2008). For that case the positive-definite character of the Fourier
transform of $G(u,v)=G(u-v)$ is a condition implying the absence of limiting
similarity. Species clustering was reported, but for interaction functions
rather different from the Gaussian used in Scheffer & van Nes (2006). In fact,
for the Gaussian interaction case most results are extremely sensitive to
details such as the implementation of the boundary conditions or weak
ecological second order effects (Pigolotti _et al._ 2008). Thus, a
clarification of the mechanisms leading to species clustering in LV models
would be desirable. In addition, the results in Pigolotti _et al._ (2007,
2008) were obtained under the unrealistic assumption of homogeneity in niche
space whereas the inhomogeneities in the carrying capacity are known to play
relevant roles (Szabó & Meszéna 2006). For simplicity we restrict our
description to the standard situation in which competition is stronger among
species closer in niche space. It is worth mentioning the existence of studies
of LV systems where non-local interactions are considered (Doebeli & Dieckmann
2000). That situation can also be described by the general formalism used here
of an integral kernel function, and our general results therefore also apply
to the situation with non-local interactions.

In this Paper we analyse some mathematical properties of the LV model (1) or
(2). In Sect. 2 we show that the positive-definiteness of the kernel $G$
remains a determining condition for stable coexistence even for non constant
$K(u)$. In Sect. 3, we discuss the mechanism producing lumped species
distributions and explicitly give an analytic expression for a particular
interaction kernel. In the Appendix we show that, in contrast with the
earliest characterizations of the interaction kernel $G$ (MacArthur & Levins
1967; Roughgarden 1979), both positive- and non-positive-definite kernels can
arise from more detailed ecological models which consider the dynamics of the
consumed resource. We use periodic boundary conditions in our numerical
simulations. We expect the effects of this simplifying but unrealistic
assumption to be unimportant at least when a non-constant carrying capacity
limits the presence of species to a limited region of niche space.

##  2 The stability of close coexistence

A simplifying assumption for the study of the LV model is that of homogeneity
in niche space. In this case, the carrying capacity and growth rate are
constants, $K_{0}$ and $r_{0}$, and the interaction kernel depends only on
differences of niche positions $G(u,v)=G(|u-v|)$. Niche space could be
infinite, but in the case in which it is finite, homogeneity can only be
achieved under periodic boundary conditions. Under these restrictions it is
easy to see that a steady solution to (2) which is homogeneous and everywhere
non-vanishing always exists: $\psi_{0}=K_{0}/\hat{G}_{0}$, where
$\hat{G}_{0}\equiv\int duG(u)$. This solution represents coexistence of all
possible species without a limit to their similarity. Its stability against
small perturbations can be analysed by linearization of the equation resulting
from substitution of $\psi(u,t)=\psi_{0}+\delta\psi(u,t)$ into (2). The
solution for the Fourier transform of the deviation from the homogeneous
state, $\delta\hat{\psi}_{q}(t)$, is

$\delta\hat{\psi}_{q}(t)=\delta\hat{\psi}_{q}(0)e^{\lambda_{q}t}\ ,{\rm with}\
\ \lambda_{q}=-r_{0}\frac{\hat{G}_{q}}{\hat{G}_{0}}\ .$ (4)

where $\hat{G}_{q}$ is the Fourier transform of $G(u)$. Thus, the homogeneous
solution $\psi_{0}$ is stable if $\hat{G}_{q}$ is positive $\forall q$, while
a instability leading to pattern formation occurs when $\hat{G}_{q}$ may take
negative values (Pigolotti _et al._ 2007; Fuentes _et al._ 2004; Hernández-
García & López 2004; López & Hernández-García 2004). We note that many steady
solutions to Eq. (2) exist besides $\psi_{0}$ (in particular, solutions of the
form (3)). This is so because dynamics preserves $\psi(u)=0$ at all places
where there is no initial population. Notice also that $\psi_{0}$ is the only
strictly positive solution. Among this multiplicity of solutions the ones that
will be more relevant are the ones which are stable under perturbations or
small immigration (Pigolotti _et al._ 2007).

An interesting class of functions to be used as kernels and carrying
capacities is the family $\\{g_{\sigma}^{p}\\}$ given by

$g_{\sigma}^{p}(u)\equiv\exp\left(-|u/\sigma|^{p}\right),$ (5)

which is parameterized by the value of $p$. The widely used Gaussian kernel is
obtained for $p=2$. When $p<2$ the functions are more peaked around $u=0$ (the
case $p=1$ is an exponential) and for $p>2$ they become more box-like
($g_{\sigma}^{\infty}(u)$ is the flat box with value $1$ in the interval
$[-\sigma,\sigma]$ and zero outside). The width of the kernel $\sigma$ gives
the competition range in niche space. We have positivity of the Fourier
transform if $p\leq 2$. This implies that the homogeneous solution is stable
under evolution with uniform $K$ and kernel $G$ of the form (5) if $p\leq 2$.
When $p>2$, the homogeneous solution is unstable and the system approaches
delta comb solutions of the type (3), with a spacing approximately $1.4\sigma$
(Pigolotti _et al._ 2007) which represent limiting similarity situations.

We now generalize the above stability analysis to the more realistic case in
which there is no homogeneity in niche space. First we consider the simpler
case of a symmetric kernel $G(u,v)=G(v,u)$, which in particular includes the
previous case of kernels depending only on species distance:
$G(u,v)=G(|u-v|)$. Note that in this symmetric case one can write Eq. (2) in
potential form:

$\partial_{t}\psi(u,t)=-r(u)\frac{\psi(u,t)}{K(u)}\frac{\delta
V[\psi]}{\delta\psi(u)},$ (6)

with the functional potential given by:

$V[\psi]=-\int K(u)\psi(u,t)du+\frac{1}{2}\int\int G(u,v)\psi(u,t)\psi(v,t)\
du\ dv.$ (7)

Stationary solutions of Eq. (2) are those for which the r.h.s of Eq. (6)
equals $0$. This has many possible solutions. We define the natural stationary
solution, $\psi^{N}(u)$, as the one which is positive and non vanishing for
all $u$, so that

$\left(\frac{\delta V}{\delta\psi}\right)_{\psi^{N}}=0,$ (8)

that is, the one satisfying:

$\int G(u,v)\psi^{N}(v)dv=K(u)\ ,\forall\ u$ (9)

The solution $\psi^{N}(u)$ can be considered the non-homogeneous
generalization of $\psi_{0}$ introduced in the homogeneous case. In the
particular case in which $G(u,v)=G(u-v)$ the natural solution can be
explicitly written in terms of Fourier transforms of the competition kernel
and the carrying capacity, either in an infinite system or in a finite one
with periodic boundary conditions:

$\hat{\psi}^{N}_{q}=\frac{\hat{K}_{q}}{\hat{G}_{q}}.$ (10)

This requires that these Fourier transforms and their inverses exist and lead
to positive populations densities. When this happens, a continuum species
coexistence is obtained, and its existence is generally robust against small
changes in $G$ or $K$. We show later that it is also an attractor of the
dynamics when $\hat{G}_{q}$ satisfy positivity requirements ($p\leq 2$, for
the family in (5), being $p=2$ the marginal case). For a uniform carrying
capacity, the natural solution (9) always exists and is uniform in phenotype
space $\psi^{N}(u)=\psi_{0}$. But the natural solution may lose positivity or
even cease to exist depending on the properties of $G$ and $K$. For example,
when both $G(u)$ and $K(u)$ are of the form (5) with $p=2$, the inverse
Fourier transform of (10) exists when the carrying capacity has a value of
$\sigma$ larger than the kernel $G$, but not in the opposite case.

Figure 1: Long-time solutions of (2) for different kernels and carrying
capacities. Left: $G=g_{\sigma}^{1}$, $K={\rm sech}(u/\sigma)$, with
$\sigma=0.1$. The natural steady solution ($\psi^{N}=a^{-1}{\rm
sech}^{3}(u/\sigma)$), which is positive and non-vanishing everywhere, is
reached at long times. Center: $G=g_{0.1}^{4}$, $K=g_{0.1}^{0.5}$. Under this
non-positive-definite competition kernel, the solution shown is still evolving
and approaches a singular delta comb of the type (3) at long times. Right:
$G=g_{0.1}^{0.5}$, $K=g_{0.1}^{1}$. A positive natural solution does not exist
and the system approaches a single hump solution which vanishes in part of
niche space.

Figure 1 shows stationary solutions attained at long times by the dynamics in
(2) illustrating the situations described above, starting from a smooth non-
vanishing initial condition. In the first case we choose a kernel and carrying
capacity functions ($G(u)=g_{\sigma}^{1}(u)$, $K(u)={\rm sech}(u/\sigma)$)
such that the natural solution exists and is positive everywhere. Thus it is
stable, and it is the steady state attained at long times. In fact it can be
analytically calculated:

$\psi^{N}(u)=a^{-1}{\rm sech}^{3}(u/\sigma)\ .$ (11)

In the second case the non-positiveness of the kernel used (with a carrying
capacity of the type Eq.(5)) breaks down the initial configuration into lumps,
which at long times approach zero-width delta functions with forbidden zones
in between. In the third case, despite $\hat{G}_{q}$ being positive, a
positive natural solution does not exist. Several outcomes are possible but
for the kernel and capacity used, the system approaches a single hump solution
which vanishes in part of the niche space.

More in general, but still in the symmetric $G$ case $G(u,v)=G(v,u)$, writing
the LV model in potential form (Eq. (6)) is of great use since one can show
that, provided $r(u)$ and $K(u)$ are positive, $dV/dt\leq 0$. This implies
that $V$ is a Lyapunov potential and dynamics proceeds towards its absolute
minimum, or if $\psi(u,t=0)=0$ for some $u$, towards the minimum of $V$ under
such constrain. Notice that, since the potential is a quadratic form,
$\psi^{N}$ is a global attractor (starting from non-vanishing initial
conditions) when the competition kernel is a positive definite quadratic form,
which means that $\int\int f(u)G(u,v)f(v)dudv\geq 0$, $\forall f$ (or
$\sum_{ij}x_{i}G_{ij}x_{j}\geq 0$, $\forall\\{x_{i}\\}$ in the discrete case).
This generalizes the previous stability condition on the Fourier transform
$\hat{G}_{q}>0$ to niche inhomogeneous cases, and shows that the stability
result was global indeed. In a multi-dimensional niche space the same analysis
shows that the positive-definiteness of the quadratic form remains the
condition for the global stability of the natural solution. In any case, the
important consequence is that the stability of the natural solution depends
uniquely on the competition kernel and not on the carrying capacity (provided
the relation kernel-capacity is such that the natural solution exists and is
positive). In particular, for competition kernels of the form (5), $\psi^{N}$
is always (if existing and positive) a globally stable solution of the
dynamics for $p\leq 2$, and unstable otherwise.

The crucial difference in the case of a non-symmetric competition kernel is
that there is no obvious Lyapunov potential for the system. This implies that
there are no available global stability results. However, local stability can
be investigated. Let us consider a small perturbation of the positive natural
solution $\psi^{N}(u)+\delta\psi(u,t)$. To linear order, the perturbation
evolves as:

$\frac{d\delta\psi(u,t)}{dt}=-r(u)\frac{\psi^{N}(u)}{K(u)}\int G(u,v)\
\delta\psi(v,t)\,dv.$ (12)

We now consider the functional $H(\delta\psi)\equiv\int\ du\
\left(A(u)K(u)/r(u)\psi^{N}(u)\right)\ (\delta\psi)^{2}$, where $A(u)$ is a
positive function so that $H\geq 0$ and $H(0)=0$. Let us compute its time
derivative:

$\frac{dH}{dt}=-2\int\delta\psi(u)\ A(u)\ G(u,v)\ \delta\psi(v)\,du\,dv.$ (13)

If for some choice of $A(u)$ one has that $A(u)G(u,v)$ is positive definite,
then $dH/dt<0$ and $\delta\psi=0$ will be approached. This shows that
$\psi^{N}$ is linearly stable in such case. We note that the case in which
$G(u,v)$ itself is positive-definite trivially guaranties the positivity of
$A(u)G(u,v)$, with a constant $A$. Thus, even in this more general
nonsymmetric case, it is the character of the interaction kernel $G$, and not
of the carrying capacity (provided it is such that the natural solution exists
and is positive), which determines the stability of the natural solution.

##  3 Lumped species distributions

Scheffer & van Nes (2006) found transient but long-lived solutions of Eq. (1)
consisting of periodically spaced lumps containing many close species. They
used a Gaussian interaction kernel which turned out to introduce an excessive
sensitivity of the results to the numerical implementation of the model and
boundary conditions (Pigolotti _et al._ 2008). They found however similar
solutions as steady configurations when adding an extra predation term acting
effectively only on species with high population. This can be thought as an
extra intraspecific competition since it decreases the growth of species with
many individuals. Exploiting this idea, Pigolotti _et al._ (2007) checked the
effect of using in (2) a kernel of the type (5) but with an enhanced
interaction at $u=0$, i.e. enhanced intraspecific competition. In particular,
they used a constant carrying capacity $K(u)=K_{0}$ and a flat box kernel with
an added delta function at the origin (see Fig. 2),

$G(u)=g_{\sigma}^{\infty}(u)+a\delta(u)\ .$ (14)

Lumped patterns were obtained numerically for $a=1$.

Because the dynamics of (2) usually involves very long transients, it is
interesting to calculate analytically the steady lumped solution in the simple
case of a kernel (14) and uniform carrying capacity $K_{0}$ (in the infinite
line).

Figure 2:  The kernel in Eq. (14) (left), and the analytic steady solution
given by (18) and (21-22) for $a=K_{0}=1$, $\sigma=0.8$, $L=0.3$ and $d=1$
(right). Figure 3:  The solution $\lambda$ (positive branch) of Eq. (20),
giving the inverse width of species lumps. The width is finite for
$d-\sigma<a$, which is favored by larger enhanced intraspecific competition
$a$.

We begin with the steady state condition

$\int G(u,v)\psi(v)dv=K(u)\ ,$ (15)

valid at $u$ such that $\psi(u)\neq 0$, that particularized to (14) and
constant $K$ reads:

$a\psi(u)+\int_{u-\sigma}^{u+\sigma}dv\psi(v)=K_{0}\ .$ (16)

This is transformed into a differential-difference equation after
differentiation with respect to $u$:

$a\psi^{\prime}(u)+\psi(u+\sigma)-\psi(u-\sigma)=0\ ,{\rm where}\ \psi(u)\neq
0\ .$ (17)

This steady equation has many solutions, including the natural one
$\psi_{0}=K_{0}/(a+2\sigma)$ which is non-vanishing everywhere, or delta combs
such as (3). We search for solutions of the type in Fig. 2, i.e. periodic
arrays of lumps, of period $d$, each one having a symmetric hump shape $f(u)$
of width $2L$ (i.e. $f(u)=0$ if $u\notin[-L,L]$):

$\psi(u)=\sum_{n=-\infty}^{\infty}f(x-nd)$ (18)

We are assuming that the lumps do not overlap, so that $d>2L$. We also note
that if $\sigma+2L<d$ there is no interaction between different lumps, so that
for $u\in[-L,L]$ Eq. (17) reduces to $f^{\prime}(u)=0$ and there is no lump
solution. Moreover, analysis is much simplified if each of the lumps interacts
only with its neighbors ($\sigma+2L<2d$). Thus we restrict to
$d<\sigma+2L<2d$, for which (17) with (18) and $u\in(-L,L)$ becomes:

$af^{\prime}(u)+f(u+\sigma-d)-f(u-\sigma+d)=0$ (19)

The general solution of this linear equation is obtained as a sum of
exponentials $\exp(\lambda u)$, with

$a\lambda=\sinh\left(\lambda(d-\sigma)\right)\ .$ (20)

$\lambda=0$ is always a solution, and if $d-\sigma<a$ there are two additional
solutions $\pm\lambda$, plotted in Fig. 3. For $d-\sigma>a$ the only solution
is the constant one, but in the opposite case (the situation favored by
enhanced intraspecific competition $a$) the solution is a linear combination
of three exponentials. Two of the constants of the combination are determined
from $f(L)=f(-L)=0$. The third one, which gives the overall normalization, can
be obtained by returning back to the original equation (16). Finally we get

$\displaystyle f(u)$ $\displaystyle=$ $\displaystyle
A\left(1-\frac{\cosh(\lambda u)}{\cosh(\lambda L)}\right)\ {\rm if}\
u\in[-L,L]$ (21) $\displaystyle=$ $\displaystyle 0\ \qquad{\rm elsewhere}$

with

$A=\frac{K_{0}}{a\left(1-{\rm sech}(\lambda
L)\right)+\frac{2}{\lambda}\left(\lambda L-\tanh(\lambda L)\right)}\ ,$ (22)

and the value of $\lambda$ which is plotted in Fig. 3. Figure 2 shows the
analytic solution (18) with (21)-(22). We have not studied the stability of
this configuration. But the numerical results in Pigolotti _et al._ (2007)
indicate that it is stable for some values of $L$ and $d$.

Figure 4:  The kernel $G=g_{0.2}^{4}+0.8g_{0.02}^{1}$ (left), and the steady
solution obtained numerically from it at long times with constant $K_{0}=1$
(right).

We finally stress that the appearance of the lumped solution is not a
consequence of the singularity of the delta function in the kernel. In fact,
any kernel sufficiently peaked at the origin will favor the coexistence of
close species. If the behavior at larger distances of the kernel makes it not
positive-definite, then full coexistence will be unstable and the natural
solution will split into disjoint lumps. An example of the final steady state
in this situation is shown in Fig. 4, with a kernel
$G=g_{0.2}^{4}+0.8g_{0.02}^{1}$ which has the properties just described and
contains no delta singularity.

###### Acknowledgements.

C.L. and E.H-G. acknowledge support from project FISICOS (FIS2007-60327) of
MEC and FEDER and NEST-Complexity project PATRES (043268). K.H.A was supported
by the Danish Research Council, grant no. 272-07-0485

Models leading to LV competitive interactions

We have seen that the character of the interaction kernel $G$ is of major
importance to determine the qualitative outcome of LV competition. In the
original formulation of the niche model, however, only positive definite
kernels were allowed. The reason is that competition kernels were derived in
terms of utilization functions $u_{i}(x)$, describing how consumer $i$ uses
resource at niche location $x$ (assumed to be continuous) (MacArthur & Levins
1967; Roughgarden 1979):

$G_{ij}=\frac{\int u_{i}(x)u_{j}(x)\,dx}{\int u_{i}^{2}(x)\,dx}.$ (23)

When the resource is directly related to space, (23) can be justified by
considering the probability that consumer $i$ meets consumer $j$ (Roughgarden
1979). It is easy to see that $G_{ij}$ obtained from (23) is positive
definite. We show in the following, however, that relation (23) is by no means
general, and that a greater variety of kernels –positive or non-positive
definite, so that the natural solution representing coexistence can be either
stable or unstable– could be obtained from equations in which resources are
explicitly modelled. Related calculations could be found, for example, in
Schoener (1974).

We consider a set of predators (or consumers), with populations $N_{i}$,
$i=1,2,...m$, competing for different types of prey populations or resources,
$R_{\alpha}$, $\alpha=1,2,...n$, the later growing in a logistic way with
growth rate $\beta_{\alpha}$ and carrying capacity $Q_{\alpha}$ in the absence
of predators. Particular equations modelling this are

$\displaystyle\dot{R}_{\alpha}$ $\displaystyle=$ $\displaystyle-
R_{\alpha}\sum_{i}a_{\alpha
i}N_{i}+\beta_{\alpha}R_{\alpha}\left(1-\frac{R_{\alpha}}{Q_{\alpha}}\right)$
(24) $\displaystyle\dot{N}_{i}$ $\displaystyle=$ $\displaystyle
N_{i}\sum_{\alpha}S_{i\alpha}R_{\alpha}-d_{i}N_{i}$ (25)

$d_{i}$ is the death rate of species $i$. The interaction coefficients are
$a_{\alpha i}$, the depletion rate of resource $\alpha$ produced by species
$i$, and the sensitivity $S_{i\alpha}$, giving the growth rate of $i$ thanks
to resource $\alpha$. Lotka-Volterra type dynamics arises when the time scale
for resource evolution is much faster than that of the consumers (i.e.
$S_{i\alpha}$ and $d_{i}\rightarrow\infty$, but with their ratio finite). In
this case, adiabatic elimination of the resource can be done
($\dot{R}_{\alpha}\approx 0$, so that each prey is at each instant at the
equilibrium determined by their consumers), giving

$R_{\alpha}\approx Q_{\alpha}\left(1-\frac{1}{\beta_{\alpha}}\sum_{i}a_{\alpha
i}N_{i}\right)\ .$ (26)

for the non-vanishing resources. The impact matrix, $D_{\alpha i}$, describing
the depletion of resource $\alpha$ by species $i$ (Meszéna _et al._ 2006), is
$D_{\alpha i}=Q_{\alpha}a_{\alpha i}/\beta_{\alpha}$, which substituted into
the consumers equation leads to:

$\dot{N}_{i}=N_{i}\left(r_{i}-\sum_{j}C_{ij}N_{j}\right)\ ,$ (27)

where $r_{i}=\sum_{\alpha}S_{i\alpha}Q_{\alpha}$ is the maximum growth rate
and $C_{ij}=\sum_{\alpha}S_{i\alpha}D_{\alpha j}$. Thus, the result is an
effective interaction among the predators which is of Lotka-Volterra type. It
is customary to write (27) in terms of the carrying capacity $K_{i}$, defined
as the equilibrium population $N_{i}$ attained in the absence of the other
competitors, i.e. $K_{i}=r_{i}/C_{ii}$. In terms of it, Eq. (27) becomes
identical to (1), with

$G_{ij}=\frac{C_{ij}}{C_{ii}}=\frac{\sum_{\alpha}S_{i\alpha}D_{\alpha
j}}{\sum_{\alpha}S_{i\alpha}D_{\alpha i}}.$ (28)

Having a continuum $R(x)$ of resources instead of a discrete set $R_{\alpha}$
does not introduce important difficulties. Simply one should replace sums by
integrals, replacing the coefficients of Eq. (27) by:

$\displaystyle r_{i}$ $\displaystyle=$ $\displaystyle\int S_{i}(x)Q(x)dx\ ,$
(29) $\displaystyle C_{ij}$ $\displaystyle=$ $\displaystyle\int
S_{i}(x)D_{j}(x)dx\ ,$ (30) $\displaystyle G_{ij}$ $\displaystyle=$
$\displaystyle\frac{\int S_{i}(x)D_{j}(x)dx}{\int S_{i}(x)D_{i}(x)dx}\ ,$ (31)

One can also consider a continuum of species, labelled by their phenotypes
$u$, so that Eq. (1) is replaced by Eq. (2) with $K(u)=r(u)/C(u,u)$,
$G(u,v)=C(u,v)/C(u,u)$, and $r(u)$ and $C(u,v)$ given by obvious
generalizations of (29) and (30).

It is clear that the presence in the kernel $G_{ij}$ of two different
functions (compare with the most restrictive expression (23)) gives enough
freedom to obtain a variety of kernel behaviors under different circumstances.
A particularly popular choice is to assume that impact and sensitivity are
proportional: $S_{i\alpha}=\epsilon D_{\alpha i}$, with a constant efficiency
$\epsilon$. In the continuum resource case the functions can be written in
terms of a single utilization function $u_{i}(x)$ as $D_{i}(x)=u_{i}(x)$ and
$S_{i}(x)=\epsilon u_{i}(x)$, leading to the classical expression (23).
Slightly more general cases arise when the efficiency depends only on the
resource, $\epsilon=\epsilon_{\alpha}$, or on the consumer
$\epsilon=\epsilon_{i}$, or when dependence on the two types of species
factorizes, $\epsilon=v_{i}w_{\alpha}$. In all these cases (if the efficiency
is positive) one is lead to a kernel $G_{ij}$ which is positive definite. In
more general cases, one can have a kernel leading to instability of the
coexistence state.

We conclude with two instances of ecological interactions in (25) which allow
to tune the stability. First, a homogeneous discrete and infinite niche space
in which all resources have the same internal dynamics $Q_{\alpha}=Q$,
$\beta_{\alpha}=\beta$, $\forall\alpha$, as well as the consumers: $d_{i}=d$,
$\forall i$. The interactions are taken to be

$\displaystyle S_{i\alpha}$ $\displaystyle=$ $\displaystyle
g\delta_{i,\alpha}$ (32) $\displaystyle a_{\alpha,j}=\frac{\beta}{Q}D_{\alpha
j}$ $\displaystyle=$ $\displaystyle
a\delta_{\alpha,j}+b(\delta_{\alpha,j-1}+\delta_{\alpha,j+1}).$

This models a situation in which the consumer $k$ grows only by consuming its
optimal resource $R_{k}$, whereas it depletes also the neighboring resources,
$R_{k+1}$ and $R_{k-1}$. We have $r_{i}=Qg$, $K_{i}=\beta/a$,
$C_{ij}=(Qg/\beta)\left(a\delta_{i,j}+b(\delta_{i,j-1}+\delta_{i,j+1})\right)$,
and $G_{ij}=\delta_{ij}+(b/a)\left(\delta_{i,j-1}+\delta_{i,j+1}\right)$ so
that equation (1) is now

$\dot{N}_{i}=QgN_{i}\left[1-\frac{1}{\beta}\left(aN_{i}+b\left(N_{i+1}+N_{i-1}\right)\right)\right].$
(33)

The natural solution, i.e. the one in which all species have positive non zero
population, is $\overline{N}_{i}=\beta/(a+2b)$, $\forall i$. Its linear
stability can be studied by linearization, $N_{l}(t)=\overline{N}_{l}+\delta
N_{l}(t)$ and substitution of the ansatz $\delta N_{l}\approx
e^{\lambda_{q}t}e^{iql}$ (here $i=\sqrt{-1}$). We find
$\lambda_{q}=-(Qg/\beta)\left(a+2b\cos q\right)$, $q\in[-\pi,\pi]$.
$\lambda_{q}$ are the eigenvalues of $-C_{ij}$, and stability of
$\overline{N}_{i}$ requires all these eigenvalues to be negative, i.e.,
$C_{ij}$ to be positive definite. When $a>2b$, then $\lambda_{q}<0$ $\forall
q$, and the natural coexistence solution is globally stable (see results in
Sect. 2). It is unstable otherwise. In this example, there is no well-defined
single utilization function and the positivity properties of the interaction
kernel and thus the stability of the natural solution can be changed by
varying the parameters.

As a second example, with nonconstant carrying capacity, we consider a
continuous distribution of resources and species on the line, and we take

$\begin{array}[]{rclrcl}Q(x)&=&Qg(x),&\beta(x)&=&\beta g(x),\\\
S_{u}(x)&=&s\delta(u-x),&a_{u}(x)&=&f(u-x).\end{array}$

which implies that the consumers of phenotype $u$ grow only from the resource
at location $x=u$, but they deplete a wider range characterized by $f$. This
leads (in the continuous formalism) to $r(u)=sQg(u)$ ,
$C(u,v)=sQf(u-v)/\beta$, $K(u)=\beta g(u)/f(0)$ and $G(u,v)=f(u-v)/f(0)$. In
this example, by choosing the functions $g$ and $f$, we can impose any desired
combinations of carrying capacity and interaction kernel. Gaussianity or
positive definiteness are particular cases, no more natural in this
generalization than alternative choices leading to non-positiveness,
instability, and thus exclusion zones between clumps of species.

## References

  * (1)
  * Abrams (1983) Abrams, P., 1983 The theory of limiting similarity.  _Ann. Rev. Ecol. Syst._ 14, 359–376.  10.1146/annurev.es.14.110183.002043.
  * Benkert & Anderson (1991) Benkert, C. & Anderson, D. Z., 1991 Controlled competitive dynamics in a photorefractive ring oscillator: ‘winner-takes-all’ and the ‘voting-paradox’ dynamics.  _Phys. Rev. A_ 44, 4633–4638.  10.1103/PhysRevA.44.4633.
  * Case (1981) Case, T. J., 1981 Niche packing and coevolution in competition communities.  _Proc. Nat. Acad. Sci. USA_ 78, 5021–5025.
  * Doebeli & Dieckmann (2000) Doebeli, M. & Dieckmann, U., 2000 Evolutionary branching and sympatric speciation caused by different types of ecological interactions.  _Am. Nat._ 156, S77–S101.  10.1086/303417.
  * Fuentes _et al._ (2004) Fuentes, M., Kuperman, M. & Kenkre, V., 2004 Analytical considerations in the study of spatial patterns arising from nonlocal interaction effects.  _J. Phys. Chem. B_ 108, 10505–10508.  10.1021/jp040090k.
  * Fuentes _et al._ (2003) Fuentes, M. A., Kuperman, M. N. & Kenkre, V. M., 2003 Nonlocal interaction effects on pattern formation in population dynamics.  _Phys. Rev. Lett._ 91, 158104.  10.1103/PhysRevLett.91.158104.
  * Hernández-García & López (2004) Hernández-García, E. & López, C., 2004 Clustering, advection, and patterns in a model of population dynamics with neighborhood-dependent rates.  _Phys. Rev. E_ 70, 016216.  10.1103/PhysRevE.70.016216.
  * Hernández-García & López (2005) Hernández-García, E. & López, C., 2005 Birth, death and diffusion of interacting particles.  _J. Phys.: Condens. Matter_ 49, S4263–S4274.  10.1088/0953-8984/17/49/019.
  * Leimar _et al._ (2008) Leimar, O., Doebeli, M. & Dieckmann, U., 2008 Evolution of phenotypic clusters through competition and local adaptation along an environmental gradient.  _Evolution_ 62, 807–822.  10.1111/j.1558-5646.2008.00334.x.
  * López & Hernández-García (2004) López, C. & Hernández-García, E., 2004 Fluctuations impact on a pattern-forming model of population dynamics with non-local interactions.  _Physica D_ 199, 223–234.  10.1016/j.physd.2004.08.016.
  * Lotka (1932) Lotka, A., 1932 The growth of mixed populations: two species competing for a common food supply.  _J. Washington Acad. Sci._ 22, 461–469.
  * MacArthur & Levins (1967) MacArthur, R. & Levins, R., 1967 The limiting similarity, convergence, and divergence of coexisting species.  _Am. Nat._ 101, 377.  10.1086/282505.
  * Meszéna _et al._ (2006) Meszéna, G., Gyllenberg, M., Pásztor, L. & Metz, J., 2006 Competitive exclusion and limiting similarity: A unified theory.  _Theor. Popul. Biol._ 69, 68–87.  10.1016/j.tpb.2005.07.001.
  * Pigolotti _et al._ (2007) Pigolotti, S., López, C. & Hernández-García, E., 2007 Species clustering in competitive Lotka-Volterra models.  _Phys. Rev. Lett._ 98, 258101.  10.1103/PhysRevLett.98.258101.
  * Pigolotti _et al._ (2008) Pigolotti, S., Lopez, C., Hernandez-Garcia, E. & Andersen, K. H., 2008 On the robustness of Gaussian competition in niche models. Available from arXiv.org:0802.3274.
  * Pistorius & Utterback (1997) Pistorius, C. I. & Utterback, J. M., 1997 Multi-mode interaction among technologies.  _Res. Policy_ 26, 67.  10.1016/S0048-7333(96)00916-X.
  * Ramos _et al._ (2008) Ramos, F., López, C., Hernández-García, E. & Muñoz, M. A., 2008 Crystallization and melting of bacteria colonies and brownian bugs.  _Phys. Rev. E_ 77, 021102.  10.1103/PhysRevE.77.021102.
  * Roughgarden (1979) Roughgarden, J., 1979 _Theory of Population Genetics and Evolutionary Ecology: an Introduction_.  Macmillan Publishers.
  * Scheffer & van Nes (2006) Scheffer, M. & van Nes, E. H., 2006 Self-organized similarity, the evolutionary emergence of groups of similar species.  _Proc. Natl. Acad. Sci. USA_ 103, 6230–6235.  10.1073/pnas.0508024103.
  * Schoener (1974) Schoener, T. W., 1974 Some methods for calculating competition coefficients from resource-utilization spectra.  _Am. Nat._ 108, 332.  10.1086/282911.
  * Szabó & Meszéna (2006) Szabó, P. & Meszéna, G., 2006 Limiting similarity revisited.  _Oikos_ 112, 612–619.  10.1111/j.0030-1299.2006.14128.x.
  * Volterra (1926) Volterra, V., 1926 Variazioni e fluttuazioni del numero d’individui in specie animali conciventi (Variations and fluctuations of the number of individuals in animal species living together).  _Memoria della R. Accademia Nazionale dei Lincei, Ser. VI_ 2, 31–113.  Translated in Chapman, R. (1931) Animal Ecology (McGraw Hill, New York), pp. 409–448.


# DREsS: Dataset for Rubric-based Essay Scoring on EFL Writing

Haneul Yoo Jieun Han So-Yeon Ahn Alice Oh
KAIST, South Korea
{haneul.yoo, jieun_han<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Automated essay scoring (AES) is a useful tool in English as a Foreign
Language (EFL) writing education, offering real-time essay scores for students
and instructors. However, previous AES models were trained on essays and
scores irrelevant to the practical scenarios of EFL writing education and
usually provided a single holistic score due to the lack of appropriate
datasets. In this paper, we release DREsS, a large-scale, standard dataset for
rubric-based automated essay scoring. DREsS comprises three sub-datasets:
$\text{DREsS}_{\text{New}}$, $\text{DREsS}_{\text{Std.}}$, and
$\text{DREsS}_{\text{CASE}}$. We collect $\text{DREsS}_{\text{New}}$, a real-
classroom dataset with 1.7K essays authored by EFL undergraduate students and
scored by English education experts. We also standardize existing rubric-based
essay scoring datasets as $\text{DREsS}_{\text{Std.}}$. We suggest CASE, a
corruption-based augmentation strategy for essays, which generates 20K
synthetic samples of $\text{DREsS}_{\text{CASE}}$ and improves the baseline
results by 45.44%. DREsS will enable further research to provide a more
accurate and practical AES system for EFL writing education.

DREsS: Dataset for Rubric-based Essay Scoring on EFL Writing

Haneul Yoo Jieun Han So-Yeon Ahn Alice Oh KAIST, South Korea {haneul.yoo,
jieun_han<EMAIL_ADDRESS><EMAIL_ADDRESS>

##  1 Introduction

In writing education, automated essay scoring (AES) can provide real-time
scores of students’ essays to both students and instructors. For many students
who are hesitant to expose their errors to instructors, the immediate
assessment of their essays with AES can create a supportive environment for
self-improvement in writing skills Sun and Fan (2022). For instructors, AES
models can ease the time-consuming process of evaluation and serve as a means
to validate their assessments, ensuring consistency in their evaluations.

|  | Content | Organization | Language
---|---|---|---|---
$\text{DREsS}_{\text{New}}$ | 1,782 | 1,782 | 1,782
$\text{DREsS}_{\text{Std.}}$ | ASAP P7 | 1,569 | 1,569 | 1,569
ASAP P8 | 723 | 723 | 723
ASAP++ P1 | 1,785 | 1,785 | 1,785
ASAP++ P2 | 1,800 | 1,800 | 1,800
ICNALE EE | 639 | 639 | 639
$\text{DREsS}_{\text{CASE}}$ | 3,924 | 15,696 | 981
Total | 12,222 | 23,994 | 14,845

Table 1: Data statistics

AES systems can provide either a holistic or an analytic view of essays, but
rubric-based, analytical scores are more preferred in the EFL writing
education domain Ghalib and Al-Hattami (2015). However, there is only a
limited amount of rubric-based datasets available for AES, and the rubrics are
not consistent in building generalizable AES systems. Furthermore, AES
datasets must be annotated by writing education experts because the scoring
task requires pedagogical knowledge in English writing. To date, there is a
lack of usable datasets for training rubric-based AES models, as existing AES
datasets provide only overall scores and/or make use of scores annotated by
non-experts.

In this paper, we release DREsS, a large-scale dataset for rubric-based essay
scoring using three key rubrics: content, organization, and language. DREsS
consists of three datasets: 1) $\text{DREsS}_{\text{New}}$ with 1,782 essays
from English as a foreign language (EFL) learners and their scores assessed by
experts, 2) $\text{DREsS}_{\text{Std.}}$ with 6,516 essays and scores from
existing datasets, and 3) $\text{DREsS}_{\text{CASE}}$ with 20,601 synthetic
essay samples. We standardize and rescale existing rubric-based datasets to
align our rubrics. We also suggest CASE, a corruption-based augmentation
strategy for Essays, employing three rubric-specific strategies to augment the
dataset with corruption. $\text{DREsS}_{\text{CASE}}$ improves the baseline
result by 45.44%.

##  2 Related Work

###  2.1 Holistic AES

#### ASAP Prompt 1-6

ASAP dataset 111https://www.kaggle.com/c/asap-aes is widely used in AES tasks,
involving eight different prompts. Six out of eight prompt sets (Prompt 1-6)
have a single overall score. This holistic AES includes 10K essay scoring data
on source-dependent essay (Prompt 3-6) and argumentative essay (Prompt 1-2).
However, these essays are graded by non-expert annotators, though the essays
were written by Grade 7-10 students in the US.

#### TOEFL11

TOEFL11 Blanchard et al. (2013) corpus from ETS introduced 12K TOEFL iBT
essays, which are not publicly accessible now. TOEFL11 only provides a general
score for essays in 3 levels (low/mid/high), which is insufficient for
building a well-performing AES system.

#### Models

The majority of the previous studies used the ASAP dataset for training and
evaluation, aiming to predict the overall score of the essay only Tay et al.
(2018); Cozma et al. (2018); Wang et al. (2018); Yang et al. (2020). Enhanced
AI Scoring Engine (EASE) 222https://github.com/edx/ease is a commonly used,
open-sourced AES system based on feature extraction and statistical methods.
In addition, Taghipour and Ng (2016) and Xie et al. (2022) released models
based on recurrent neural networks and neural pairwise contrastive regression
(NPCR) model, respectively. However, only a limited number publicly opened
their models and code, highlighting the need for additional publicly available
data and further validation of existing models.

###  2.2 Rubric-based AES

#### ASAP Prompt 7-8

ASAP includes only two prompts (Prompt 7-8) that are rubric-based. These two
rubric-based prompts consist of 1,569 and 723 essays for each respective
prompt. The two prompt sets even have distinct rubrics and score ranges, which
poses a challenge in leveraging both datasets for training rubric-based
models. These dataset (Prompt 7-8) is also evaluated by non-expert annotators,
similar to ASAP Prompt 1-6.

#### ASAP++

To overcome the holistic scoring of ASAP Prompt 1-6, Mathias and Bhattacharyya
(2018) manually annotated rubric-based scores on those essays. However, most
samples in ASAP++ were annotated by a single annotator, who is a non-expert,
including non-native speakers of English. Moreover, each prompt set of ASAP++
has different attributes or rubrics to each other, which need to be more
generalizable to fully leverage such dataset for AES model.

#### ICNALE Edited Essays

ICNALE Edited Essays (EE) v3.0 Ishikawa (2018) presents rubric-based essay
evaluation scores and fully edited versions of essays written by EFL learners
from 10 countries in Asia. Even though the essays are written by EFL learners,
the essay is rated and edited only by five native English speakers, non-
experts in the domain of English writing education. In addition, it is not
openly accessible and only consists of 639 samples.

#### Models

The scarcity of publicly available rubric-based AES datasets poses significant
obstacles to the advancement of AES research. There are industry-driven
services such as IntelliMetric® Rudner et al. (2006) and E-rater® Attali and
Burstein (2006); Blanchard et al. (2013), but none of them are accessible to
the public. In order to facilitate AES research in the academic community, it
is crucial to release a publicly available rubric-based AES dataset and
baseline model.

##  3 DREsS Dataset

We construct 1.7K samples of our newly released DREsS dataset (§3.1), 2.9K
standardized samples of existing datasets (§3.2), and 20K synthetic samples
augmented using CASE (§3.3). The detailed number of samples per rubric is
stated in Table 1.

###  3.1 Dataset Collection

Content. Paragraph is well-developed and relevant to the argument, supported
with strong reasons and examples.
---
Organization. The argument is very effectively structured and developed,
making it easy for the reader to follow the ideas and understand how the
writer is building the argument. Paragraphs use coherence devices effectively
while focusing on a single main idea.
Language. The writing displays sophisticated control of a wide range of
vocabulary and collocations. The essay follows grammar and usage rules
throughout the paper. Spelling and punctuation are correct throughout the
paper.

Table 2: Explanation of rubrics

#### Dataset Details

$\text{DREsS}_{\text{New}}$ includes 1,782 argumentative essays on 22 prompts,
having 313.36 words and 21.19 sentences on average. Each sample in DREsS
includes students’ written essay, essay prompt, rubric-based scores, total
score, and a test type (pre-test, post-test). The essays are scored on a range
of 1 to 5, with increments of 0.5, based on the three rubrics: content,
organization, and language. We chose such three conventional rubrics as
standard criteria for scoring EFL essays, according to previous studies from
the language education Cumming (1990); Ozfidan and Mitchell (2022). Detailed
explanations of the rubrics are shown in Table 2. The essays are written by
undergraduate students whose TOEFL writing score spans from 15 to 21 and
enrolled in EFL writing courses at a college in South Korea from 2020 to 2023.
Most students are Korean and their ages span from 18 to 22, with an average of
19.7. During the course, students are asked to write an in-class timed essay
for 40 minutes both at the start (pre-test) and the end of the semester (post-
test) to measure their improvements.

#### Annotator Details

We collect scoring data from 11 instructors, who serve as the teachers of the
students who wrote the essays. All annotators are experts in English education
or Linguistics and are qualified to teach EFL writing courses at a college in
South Korea. To ensure consistent and reliable scoring across all instructors,
they all participate in training sessions with a scoring guide and norming
sessions where they develop a consensus on scores using two sample essays.
Additionally, there was no significant difference among the score distribution
of all instructors tested by one-way ANOVA and Tukey HSD at a p-value of 0.05.

###  3.2 Standardizing the Existing Data

We standardize and unify three existing rubric-based datasets (ASAP Prompt
7-8, ASAP++ Prompt 1-2, and ICNALE EE) to align with the three rubrics in
DREsS: content, organization, and language. We exclude ASAP++ Prompt 3-6,
whose essay type, source-dependent essays, is clearly different from
argumentative essays. ASAP Prompt 7 contains four rubrics – ideas,
organization, style, and convention – while Prompt 8 contains six rubrics –
ideas and content, organization, voice, word choice, sentence fluency, and
convention. Both sets provide scores ranging from 0 to 3. For language, we
first create synthetic labels based on a weighted average. This involves
assigning a weight of 0.66 to the style and 0.33 to the convention in ASAP
Prompt 7, and assigning equal weights to voice, word choice, sentence fluency,
and convention in ASAP Prompt 8. For content and organization, we utilize the
existing data rubric (idea for content, organization as same) in the dataset.
We then rescale the score of all rubrics into a range of 1 to 5. We repeat the
same process with ASAP++ Prompt 1 and 2, which have the same attributes as
ASAP Prompt 8. Similarly, for ICNALE EE dataset, we unify vocabulary, language
use, and mechanics as language rubric with a weight of 0.4, 0.5, and 0.1,
respectively. In the process of consolidating the writing assessment criteria,
we sought professional consultation from EFL education experts and
strategically grouped together those components that evaluate similar aspects.

###  3.3 Synthetic Data Construction

We construct synthetic data for rubric-based AES to overcome the scarcity of
data and provide accurate scores for students and instructors. We introduce a
corruption-based augmentation strategy for essays (CASE), which starts with a
well-written essay and incorporates a certain portion of sentence-level errors
into the synthetic essay. In subsequent experiments, we define well-written
essays as an essay that scored 4.5 or 5.0 out of 5.0 on each criterion.

$\text{n}(S_{c})=\lfloor\text{n}(S_{E})*(5.0-x_{i})/5.0\rceil$ (1)

$\text{n}(S_{c})$ is the number of corrupted sentences in the synthetic essay,
and $\text{n}(S_{E})$ is the number of sentences in the well-written essay,
which serves as the basis for the synthetic essay. $x_{i}$ denotes the score
of the synthetic essay. In this paper, we generate synthetic data with CASE
under ablation study for exploring the optimal number of samples.

#### Content

We substitute randomly-sampled sentences from well-written essays with out-of-
domain sentences from different prompts. This is based on an assumption that
sentences in well-written essays support the given prompt’s content, meaning
that sentences from the essays on different prompts convey different contents.
Therefore, more number of substitutions imply higher levels of corruption in
the content of the essay.

#### Organization

We swap two randomly-sampled sentences in well-written essays and repeat this
process based on the synthetic score, supposing that sentences in well-written
essays are systematically structured in order. The higher number of swaps
implies higher levels of corruption in the organization of the essay.

Model | Fine-tuning Data | Content | Organization | Language | Total
---|---|---|---|---|---
gpt-3.5-turbo | N/A | 0.239 | 0.371 | 0.246 | 0.307
EASE (SVR) | DREsS | - | - | - | 0.360
NPCR Xie et al. (2022) | DREsS | - | - | - | 0.507
BERT Devlin et al. (2019) | $\text{DREsS}_{\text{New}}$ | 0.414 | 0.311 | 0.487 | 0.471
\+ $\text{DREsS}_{\text{Std.}}$ | 0.599 | 0.593 | 0.587 | 0.551
\+ $\text{DREsS}_{\text{CASE}}$ | 0.642 | 0.750 | 0.607 | 0.685
Table 3: Baseline results of rubric-based automated essay scoring on DREsS
(QWK score)

#### Language

We substitute randomly-sampled sentences into ungrammatical sentences and
repeat this process based on the synthetic score. We extract 605 ungrammatical
sentences from BEA-2019 data for the shared task of grammatical error
correction (GEC) Bryant et al. (2019). We define ungrammatical sentences with
the number of edits of the sentence over 10, which is the 98th percentile. The
more substitutions, the more corruption is introduced in the grammar of the
essay. We set such a high threshold for ungrammatical sentences because of the
limitation of the current GEC dataset that inherent noise may be included,
such as erroneous or incomplete correction Rothe et al. (2021).

##  4 Experimental Result

Table 3 shows the baseline results of rubric-based AES on DREsS. Detailed
experimental settings are described in Appendix §A. We provide the baseline
results on DREsS using fine-tuned BERT Devlin et al. (2019), which is the
model that most state-of-the-art AES systems have leveraged. The same
experimental results with different PLMs are provided in Appendix §B, though
Xie et al. (2022) observed no significant improvements among various pre-
trained language models (PLMs) in AES.

Fine-tuned BERT exhibits scalable results with the expansion of training data.
In particular, the model trained with a combination of our approaches
outperforms other baseline models by 45.44%, demonstrating the effectiveness
of data unification and augmentation using CASE.

The results from existing holistic AES models underscore the need to examine
these models using new datasets. EASE and NPCR reported the QWK scores as
0.699 and 0.817 on ASAP, respectively. However, the scores significantly
decrease when these models are trained and evaluated with DREsS dataset. This
result implies that EASE and NPCR are not robust to different datasets.
Additionally, these models, which are trained with ASAP, are not able to
predict general rubric-based scoring.

Figure 1: Ablation experimental results for CASE. $n_{aug}$ is the number of
synthetic data by each class per original data among all classes.

Asking gpt-3.5-turbo to score an essay achieved the worst performances among
all, showing high variances among the essays with the same score. The detailed
results for ChatGPT in different prompt settings are provided in Appendix §C.

We perform an ablation study to find the optimal number of CASE operations per
each rubric. In Figure 1, we investigate how the number of CASE operations
affects the performance over all rubrics for
$n_{aug}=\\{0.125,0.25,0.5,1,2,4,8\\}$. CASE on content, organization, and
language rubrics show their best performances on 0.5, 2, 0.125 of $n_{aug}$,
generating a pair of synthetic essays and corresponding scores in 4.5, 18,
1.125 times, respectively. We suppose that the detailed augmentation
strategies for each rubric and the small size of the original data affect the
optimal number of CASE operations. Organization, where corruption was made
within the essay and irrelevant to the size of the original data, showed the
highest $n_{aug}$. Content, where the corrupted sentences were sampled from
874 well-written essays with 21.2 sentences on average, reported higher
$n_{aug}$ than language, where the corrupted sentences were sampled from 605
ungrammatical sentences.

##  5 Conclusion

We release the DREsS, a large-scale, standard rubric-based essay scoring
dataset with three subsets: $\text{DREsS}_{\text{New}}$,
$\text{DREsS}_{\text{Std.}}$, and $\text{DREsS}_{\text{CASE}}$.
$\text{DREsS}_{\text{New}}$ is the first reliable AES dataset with 1.7K
samples whose essays are authored by EFL undergraduate students and whose
scores are annotated by instructors with expertise. According to previous
studies from language education, we also standardize and unify existing
rubric-based AES datasets as $\text{DREsS}_{\text{Std.}}$. We finally suggest
CASE, corruption-based augmentation strategies for essays, which generates 20K
synthetic samples and improves the baseline result by 45.44%. This work aims
to encourage further AES research and practical application in EFL education.

## Limitations

Our research focuses on learning English as a foreign language because there
already exist datasets, and the current language models perform the best for
English. There are many L2 learners of other languages whose writing classes
can also benefit from AES. Our findings can illuminate the directions of data
collection, annotation, and augmentation for L2 writing education of other
languages as well. We leave that as future work.

Our DREsS dataset is collected through the EFL writing courses from a college
in South Korea, and most of the essays are written by Korean EFL students. EFL
students in different cultural and linguistic backgrounds might exhibit
different essay-writing patterns, which might affect the distribution of
scores and feedback. We suggest a further extension of collecting the DREsS
dataset from diverse countries.

Our augmentation strategy primarily starts from well-written essays and
generates erroneous essays along with corresponding scores; therefore, this
approach faces challenges in synthesizing well-written essays. However, we
believe that well-written essays can be reliably produced by LLMs, which have
demonstrated strong capabilities in generating high-quality English text.

## Ethics Statement

We expect that this paper will make a significant contribution to the
application of NLP for good, particularly in the domain of NLP-driven
assistance in EFL writing education. All studies in this research project were
conducted with the approval of our institutional review board (IRB). To
prevent any potential impact on student scores or grades, we requested
students to share their essays only after the end of the EFL courses. We also
acknowledged and addressed the potential risk associated with releasing a
dataset containing human-written essays, especially considering privacy and
personal information. To mitigate these risks, we plan to 1) employ rule-based
coding and 2) conduct thorough human inspections to filter out all sensitive
information. Additionally, access to our data will be granted only to
researchers or practitioners who submit a consent form, ensuring responsible
and ethical usage.

## References

  * Attali and Burstein (2006) Yigal Attali and Jill Burstein. 2006.  Automated essay scoring with e-rater® v.2.  _The Journal of Technology, Learning and Assessment_ , 4(3).
  * Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020.  Longformer: The long-document transformer.
  * Black et al. (2022) Sidney Black, Stella Biderman, Eric Hallahan, Quentin Anthony, Leo Gao, Laurence Golding, Horace He, Connor Leahy, Kyle McDonell, Jason Phang, Michael Pieler, Usvsn Sai Prashanth, Shivanshu Purohit, Laria Reynolds, Jonathan Tow, Ben Wang, and Samuel Weinbach. 2022.  GPT-NeoX-20B: An open-source autoregressive language model.  In _Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models_, pages 95–136, virtual+Dublin. Association for Computational Linguistics.
  * Blanchard et al. (2013) Daniel Blanchard, Joel Tetreault, Derrick Higgins, Aoife Cahill, and Martin Chodorow. 2013.  Toefl11: A corpus of non-native english.  _ETS Research Report Series_ , 2013(2):i–15.
  * Bryant et al. (2019) Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019.  The BEA-2019 shared task on grammatical error correction.  In _Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications_ , pages 52–75, Florence, Italy. Association for Computational Linguistics.
  * Cozma et al. (2018) Mădălina Cozma, Andrei Butnaru, and Radu Tudor Ionescu. 2018.  Automated essay scoring with string kernels and word embeddings.  In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_ , pages 503–509, Melbourne, Australia. Association for Computational Linguistics.
  * Cumming (1990) Alister Cumming. 1990.  Expertise in evaluating second language compositions.  _Language Testing_ , 7(1):31–51.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Ghalib and Al-Hattami (2015) Thikra K Ghalib and Abdulghani A Al-Hattami. 2015.  Holistic versus analytic evaluation of efl writing: A case study.  _English Language Teaching_ , 8(7):225–236.
  * Ishikawa (2018) Shinichiro Ishikawa. 2018.  The icnale edited essays; a dataset for analysis of l2 english learner essays based on a new integrative viewpoint.  _English Corpus Studies_ , 25:117–130.
  * Mathias and Bhattacharyya (2018) Sandeep Mathias and Pushpak Bhattacharyya. 2018.  ASAP++: Enriching the ASAP automated essay grading dataset with essay attribute scores.  In _Proceedings of the Eleventh International Conference on Language Resources and Evaluation (LREC 2018)_ , Miyazaki, Japan. European Language Resources Association (ELRA).
  * Ozfidan and Mitchell (2022) Burhan Ozfidan and Connie Mitchell. 2022.  Assessment of students’ argumentative writing: A rubric development.  _Journal of Ethnic and Cultural Studies_ , 9(2):pp. 121–133.
  * Rothe et al. (2021) Sascha Rothe, Jonathan Mallinson, Eric Malmi, Sebastian Krause, and Aliaksei Severyn. 2021.  A simple recipe for multilingual grammatical error correction.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 702–707, Online. Association for Computational Linguistics.
  * Rudner et al. (2006) Lawrence M. Rudner, Veronica Garcia, and Catherine Welch. 2006.  An evaluation of intellimetric™ essay scoring system.  _The Journal of Technology, Learning and Assessment_ , 4(4).
  * Sun and Fan (2022) Bo Sun and Tingting Fan. 2022.  The effects of an awe-aided assessment approach on business english writing performance and writing anxiety: A contextual consideration.  _Studies in Educational Evaluation_ , 72:101123.
  * Taghipour and Ng (2016) Kaveh Taghipour and Hwee Tou Ng. 2016.  A neural approach to automated essay scoring.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1882–1891, Austin, Texas. Association for Computational Linguistics.
  * Tay et al. (2018) Yi Tay, Minh Phan, Luu Anh Tuan, and Siu Cheung Hui. 2018.  Skipflow: Incorporating neural coherence features for end-to-end automatic text scoring.  _Proceedings of the AAAI Conference on Artificial Intelligence_ , 32(1).
  * Wang et al. (2018) Yucheng Wang, Zhongyu Wei, Yaqian Zhou, and Xuanjing Huang. 2018.  Automatic essay scoring incorporating rating schema via reinforcement learning.  In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_ , pages 791–797, Brussels, Belgium. Association for Computational Linguistics.
  * Xie et al. (2022) Jiayi Xie, Kaiwei Cai, Li Kong, Junsheng Zhou, and Weiguang Qu. 2022.  Automated essay scoring via pairwise contrastive regression.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 2724–2733, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Yang et al. (2020) Ruosong Yang, Jiannong Cao, Zhiyuan Wen, Youzheng Wu, and Xiaodong He. 2020.  Enhancing automated essay scoring performance via fine-tuning pre-trained language models with combination of regression and ranking.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 1560–1569, Online. Association for Computational Linguistics.
  * Zaheer et al. (2020) Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago Ontanon, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020.  Big bird: Transformers for longer sequences.  In _Advances in Neural Information Processing Systems_ , volume 33, pages 17283–17297. Curran Associates, Inc.

## Appendix

##  Appendix A Experimental Settings

We split our data into train/dev/test with 6:2:2 ratio with a seed of 22. The
AES experiments were conducted under GeForce RTX 2080 Ti (4 GPUs), 128GiB
system memory, and Intel(R) Xeon(R) Silver 4114 CPU @ 2.20GHz (20 CPU cores)
with hyperparameters denoted in Table 4.

Hyperparameter | Value
---|---
Batch Size | 32
Number of epochs | 10
Early Stopping Patience | 5
Learning Rate | 2e-5
Learning Rate Scheduler | Linear
Optimizer | AdamW
Table 4: Model configuration

##  Appendix B Rubric-based AES with Different LMs

Model | Content | Organization | Language | Total
---|---|---|---|---
BERT (2019) | 0.414 | 0.311 | 0.487 | 0.471
Longformer (2020) | 0.409 | 0.312 | 0.475 | 0.463
BigBird (2020) | 0.412 | 0.317 | 0.473 | 0.469
GPT-NeoX (2022) | 0.410 | 0.313 | 0.446 | 0.475

Table 5: Experimental results on rubric-based AES with different fine-tuned
LMs

Experimental results of rubric-based AES with different LMs are provided in
Table 5, showing no significant difference among different LMs. Xie et al.
(2022) also observed that leveraging different LMs has no significant effect
on AES performance, and most state-of-the-art AES methods have leveraged BERT
Devlin et al. (2019).

##  Appendix C Rubric-based AES with ChatGPT

Prompt | Content | Organization | Language | Total
---|---|---|---|---
(A) | 0.320 | 0.248 | 0.359 | 0.336
(B) | 0.330 | 0.328 | 0.306 | 0.346
(C) | 0.357 | 0.278 | 0.342 | 0.364
(D) | 0.336 | 0.361 | 0.272 | 0.385

Table 6: Rubric-based quadratic weighted kappa (QWK) scores of ChatGPT
(gpt-3.5-turbo, temperature: 0) with diverse prompts.
(A): standard zero-shot prompting, (B): 2-shot prompting, (C): zero-shot with
rubric explanation (D): zero-shot with feedback generation.

(A) |  |  Q. Please score the essay with three rubrics: content, organization, and language.
---
### Answer format: {content: score[x], organization: score[y], language:
score[z]}
score = [1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0]
Please answer only in the above dictionary format, without feedback.
### prompt: <essay prompt>
### essay: <student’s essay>
A:
(B) |  |  Q. Please score the essay with three rubrics: content, organization, and language.
---
### Answer format: {content: score[x], organization: score[y], language:
score[z]}
score = [1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0]
Please answer only in the above dictionary format, without feedback.
### 1-shot example:
### 2-shot example:
### prompt: <essay prompt>
### essay: <student’s essay>
A:
(C) |  |  Q. Please score the essay with three rubrics: content, organization, and language.
---
<three rubrics explanation>
### Answer format: {content: score[x], organization: score[y], language:
score[z]}
score = [1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0]
Please answer only in the above dictionary format, without feedback.
### prompt: <essay prompt>
### essay: <student’s essay>
A:
(D) |  |  Q. Please score the essay with three rubrics: content, organization, and language.
---
### Answer format: {content: score[x], organization: score[y], language:
score[z], content_fbk: chatgpt_con_fbk, org_fbk: chatgpt_org_fbk, lang_fbk:
chatgpt_lang_fbk}
score = [1.0, 1.5, 2.0, 2.5, 3.0, 3.5, 4.0, 4.5, 5.0]
Please answer only in the above dictionary format, with feedback.
### prompt: <essay prompt>
### essay: <student’s essay>
A:

Table 7: Different prompts for ChatGPT to get rubric-based scores. Refer to
Table 6 for descriptions of setting (A)–(D).

Table 6 shows AES results of ChatGPT with different prompts described in Table
7. Considering the substantial length of the essay and feedback, we were able
to provide a maximum of 2 shots for the prompt to gpt-3.5-turbo. To examine
2-shot prompting performance, we divided the samples into two distinct groups
and computed the average total score for each group. Subsequently, we randomly
sampled a single essay in each group, ensuring that its total score
corresponded to the calculated mean value.