# Combinatorial exploration of quantum spin liquid candidates in the
herbertsmithite material family

Alex Hallett  Materials Department, University of California, Santa Barbara,
California 93106, USA  Catalina Avarvarei  Materials Department, University of
California, Santa Barbara, California 93106, USA  John W. Harter
<EMAIL_ADDRESS>Materials Department, University of California, Santa Barbara,
California 93106, USA

###### Abstract

Geometric frustration of magnetic ions can lead to a quantum spin liquid
ground state where long range magnetic order is avoided despite strong
exchange interactions. The physical realization of quantum spin liquids
comprises a major unresolved area of contemporary materials science. One
prominent magnetically-frustrated structure is the kagome lattice. The
naturally occurring minerals herbertsmithite [ZnCu3(OH)6Cl2] and Zn-
substituted barlowite [ZnCu3(OH)6BrF] both feature perfect kagome layers of
spin-$1/2$ copper ions and display experimental signatures consistent with a
quantum spin liquid state at low temperatures. To investigate other possible
candidates within this material family, we perform a systematic first-
principles combinatorial exploration of structurally related compounds
[$A$Cu3(OH)${}_{6}B_{2}$ and $A$Cu3(OH)${}_{6}BC$] by substituting non-
magnetic divalent cations ($A$) and halide anions ($B$, $C$). After optimizing
such structures using density functional theory, we compare various structural
and thermodynamic parameters to determine which compounds are most likely to
favor a quantum spin liquid state. Convex hull calculations using binary
compounds are performed to determine feasibility of synthesis. We also
estimate the likelihood of interlayer substitutional disorder and spontaneous
distortions of the kagome layers. After considering all of these factors as a
whole, we select several promising candidate materials that we believe deserve
further attention.

##  I Introduction

In a quantum spin liquid (QSL), frustrated antiferromagnetic exchange
interactions prevent localized spins from ordering at low temperatures,
instead forming a fluid-like phase. The large degeneracy of this state can
give rise to novel phenomena such as fractionalized quasiparticles, emergent
gauge fields, and long-range entanglement [1, 2, 3, 4]. The kagome lattice of
corner-sharing triangles is known to have high geometric frustration and is
capable of hosting such a phase. A leading QSL material candidate possessing
this structure is herbertsmithite [ZnCu3(OH)6Cl2], which contains perfect
kagome layers of spin-$1/2$ copper cations separated by non-magnetic Zn and Cl
ions [5, 6], as shown in Fig. 1(a,c). Indeed, although herbertsmithite has
strong antiferromagnetic exchange interactions, no magnetic phase transition
is observed down to sub-kelvin temperatures [7, 8, 9, 10, 11], and an array of
experimental and theoretical work favors a possible QSL scenario [12, 13, 14,
15, 16, 17, 18, 19, 20, 21, 22, 23, 19, 24].

Figure 1: Crystal structures of herbertsmithite and Zn-barlowite. (a)
Herbertsmithite viewed along the $c$-axis, showing the kagome arrangement of
Cu ions. (b) Zn-barlowite viewed along the $c$-axis. (c) Herbertsmithite
viewed along the [110] direction, showing the shifted stacking arrangement of
the kagome layers. (d) Zn-barlowite viewed along [110], showing the stacking
of the kagome layers and the inequivalence of the Br and F sites.

Despite its many promising features, herbertsmithite is prone to cation
substitutional disorder, where Cu may occupy interlayer sites and Zn may
occupy intralayer kagome sites [7, 14, 25]. The precise amount of this
disorder is debated. Several studies suggest that while there is minimal
substitution of Zn on the kagome layers, the interlayer sites can be occupied
by up to 15% Cu [26, 27, 12, 28], resulting in a decidedly off-stoichiometric
compound. These interlayer “orphan” spin-$1/2$ Cu2+ defects are highly
problematic for the QSL state, causing weak ferromagnetic interactions between
kagome layers and distorting the surrounding matrix of magnetic ions [13]. Zn-
substituted barlowite (Zn-barlowite), a structurally related compound and
another potential QSL candidate [29, 30], is thought to have a much lower
interlayer disorder concentration, largely due to the greater chemical
distinction between the interlayer and intralayer sites, as shown in Fig.
1(b,d) [31, 32]. Experiments indicate that in Zn-barlowite, off-center
interlayer $C_{2v}$ sites can contain up to 5% Cu defects. Like
herbertsmithite, however, Zn-barlowite does not order magnetically, even with
these large concentrations of magnetic defects [33, 34, 35]. While progress on
this class of materials is encouraging, it is nevertheless desirable to
further minimize orphan Cu spins to realize a clean QSL ground state.

Synthesizing compounds structurally similar to herbertsmithite and Zn-
barlowite is a promising route to discover new QSL candidates. For example,
Mg-substituted herbertsmithite, MgxCu4-x(OH)6Cl2 (tondiite), has been
successfully synthesized and shows no magnetic phase transition down to 1.8 K
[36, 37, 38], and a Cd analog [CdCu3(OH)6Cl2] shows no magnetic ordering down
to 2 K, although it exhibits significant distortions of the kagome planes
[39]. Synthesis of the bromide analog of herbertsmithite [ZnCu3(OH)6Br2] was
attempted but unsuccessful [40]. A Zn-barlowite related structure, Zn-
claringbullite [ZnCu3(OH)6ClF], shows no obvious magnetic transition down to 2
K, but a perfectly stoichiometric compound was not achieved [41]. While the Mg
analog of barlowite cannot be synthesized due to the insolubility of MgF2 in
water, the bromide analog was attempted [MgCu3(OH)6Br2], but did not have the
Zn-barlowite structure and ordered antiferromagnetically at 5.4 K [42].

Clearly, more work is needed to search for and identify viable candidates in
this material family. Only a few computational studies exist exploring cation
substitution in barlowite [31, 32], and a complete exploration of the
structural families of herbertsmithite and Zn-barlowite using computational
methods has not been performed. In this paper, we use ab initio calculations
to systematically explore compounds within the herbertsmithite and Zn-
barlowite families. We compare the thermodynamic stability, structural
properties, and tendency towards disorder. After considering all these
criteria together, we select promising QSL candidates that merit further
experimental and theoretical examination.

##  II Computational Procedure

We carry out a systematic exploration of the structural relatives of
herbertsmithite [$A$Cu3(OH)${}_{6}B_{2}$] and Zn-barlowite
[$A$Cu3(OH)${}_{6}BC$] by substituting closed-shell (spinless) $2+$ cations
($A$ = Ba, Be, Ca, Cd, Ge, Hg, Mg, Pb, Sn, Sr, Zn) and halide anions ($B,C$ =
Br, Cl, F, I). We investigate all 44 possible herbertsmithite relatives. While
there are 176 possible Zn-barlowite relatives, we eliminate compounds where
$B=C$ because the herbertsmithite structure always has lower energy in these
cases. We also do not consider compounds in which the less electronegative
anion occupies the $C$ site [the site occupied by F in Fig. 1(b,d)]. All
hydrogen bonds are oriented towards the $C$ site, so the more electronegative
ion will always occupy this position to minimize energy. Thus, a total of 66
relatives in the Zn-barlowite family were selected for consideration.

We perform high-throughput calculations where the structural optimization of
each candidate is followed by a static calculation to extract the ground-state
energy and to compute phonon frequencies at the $\Gamma$ point to confirm
structural stability. In addition to confirming the stability of the relaxed
structures, we perform convex hull calculations to determine if synthesis of
the candidate compounds is thermodynamically feasible. For the most promising
materials, we also calculate defect formation energies and full phonon
dispersions throughout the first Brillouin zone to verify stability at
$k$-points away from the zone center.

All structures were calculated by allowing the lattice parameters, cell
volume, and atomic positions to fully relax using density functional theory
(DFT) as implemented in the Vienna ab initio simulation package (vasp) [43,
44, 45]. We used the supplied projector augmented wave potentials [46] within
the generalized gradient approximation and Perdew-Burke-Ernzerhof scheme [47].
Electronic wave functions were expanded in a plane wave basis set with an
energy cutoff of 800 eV, and reciprocal space was sampled using an $8\times
8\times 8$ $k$-point mesh for herbertsmithite-related structures and an
$8\times 8\times 5$ $k$-point mesh for Zn-barlowite-related structures. A
$\Gamma$-centered mesh is necessary due to the hexagonal symmetry of Zn-
barlowite. The spacing between $k$-points was $\sim$0.15 Å-1 for both
structural families, and this spacing was also used for calculating the
energies of binary compounds used in the convex hull analysis. All structures
were relaxed until forces on the atoms were less than 1 meV/Å. Calculations
were non-spin-polarized. Input files for all calculations can be found in the
Supplemental Material [48].

Figure 2: Structural stability and thermodynamics of candidate compounds. (a)
Lowest optical phonon frequency for herbertsmithite-related candidates. (b)
Lowest optical phonon frequency for Zn-barlowite-related candidates. (c)
Convex hull energies for herbertsmithite-related candidates. (d) Convex hull
energies for Zn-barlowite-related candidates. Structurally unstable compounds
(identified by $f_{0}<0$) are denoted with an ‘X’. Cations are shown on the
vertical axis and anions on the horizontal axis, in order of increasing ionic
radius from bottom to top and left to right, respectively. The reference
compound (either herbertsmithite or Zn-barlowite) is shown in white and marked
with an asterisk. Compounds with parameter values more favorable than the
reference compounds are shown with warm colors, and values less favorable are
shown with cool colors. Figure 3: Structural properties of candidate
compounds. (a) Cu-O-Cu bond angle for herbertsmithite-related candidates. (b)
Cu-O-Cu bond angle for Zn-barlowite-related candidates. (c) Interplane kagome
distance for herbertsmithite-related candidates. (d) Interplane kagome
distance for Zn-barlowite-related candidates. Structurally unstable compounds
are denoted with an ‘X’. Cations are shown on the vertical axis and anions on
the horizontal axis, in order of increasing ionic radius from bottom to top
and left to right, respectively. The reference compound (either
herbertsmithite or Zn-barlowite) is shown in white and marked with an
asterisk. Compounds with parameter values more favorable than the reference
compounds are shown with warm colors, and values less favorable are shown with
cool colors. Figure 4: Dependence of structural properties on ion size. (a)
Cu-O-Cu bond angle versus anion radius. For Zn-barlowite, the radius plotted
is that of the most electronegative anion. Blue (red) traces correspond to Zn-
barlowite (herbertsmithite) relatives. Different cations are plotted as
separate traces where darker (lighter) traces correspond to smaller (larger)
ion sizes. (b) Interplane kagome distance versus cation radius for
herbertsmithite (red) and Zn-barlowite (blue) relatives. Separate traces are
plotted for each anion, where small (large) anions are plotted in dark (light)
shades. (c) Cu-O-Cu bond angle versus the anion $B$ to anion $C$ ratio for
stable compounds. Separate traces are plotted for different cations. (d)
$c$-axis length versus cation size (left, dashed line) and $a$-axis length
versus anion size (right, solid lines). (e) Frequency of the lowest optical
phonon mode versus $c$-axis length for Zn-barlowite (blue) and herbertsmithite
(red) relatives. Stable (unstable) compounds are shown with filled (empty)
markers. The group IV elements (Ge, Sn, Pb) are plotted with darker colors
because they are almost always unstable, regardless of their $c$-axis length.
(f) Cu-O-Cu bond angle versus $a$-axis length for Zn-barlowite (blue) and
herbertsmithite (red) relatives. Stable (unstable) compounds are shown with
filled (empty) markers. Compounds containing group IV cations (shown in darker
colors) tend to be unstable and have much smaller bond angles.

##  III Results and Discussion

###  III.1 Phonon Calculations

Phonon calculations at the $\Gamma$ point for the fully-relaxed structures
were performed in vasp within the finite differences approximation to confirm
structural stability. As expected, many structures have unstable phonon modes.
Fig. 2(a,b) shows the frequency of the lowest energy optical phonon mode,
$f_{0}$, for all compounds. In all subsequent plots, the unstable compounds
(with $f_{0}<0$) are marked with an ‘X’ to distinguish them from structurally
stable and potentially viable candidates. Cations are shown on the vertical
axis and anions on the horizontal axis, in order of increasing ionic radius
from bottom to top and left to right, respectively. The reference compound,
either herbertsmithite or Zn-barlowite, is shown in white and marked with an
asterisk. Compounds with parameter values more favorable than the reference
compound are shown with warm colors, and values less favorable are shown with
cool colors. For example, a higher frequency of the lowest energy optical mode
indicates higher dynamical stability, so higher frequencies are shown with
warm colors. Most compounds containing group IV elements (Ge, Sn, Pb) tend to
be unstable, with the exception of GeCu3(OH)6F2 and PbCu3(OH)6F2. Compounds
containing larger cations are generally unstable, as well as Zn-barlowite
relatives containing Be.

###  III.2 Convex Hull Calculations

The convex hull of a compound is useful for determining if synthesis is
thermodynamically feasible, usually through a comparison of the compound’s
formation energy to the sum of the energies of all other possible combinations
of crystal structures that could be created from the same set of elements in
the same ratios. Due to the prohibitive size of the phase space for our
candidate materials, we perform a simplified procedure. Instead of considering
all possible crystal structures, we consider only simple binary ionic
compounds [e.g. $A$(OH)2, $AB_{2}$], which are most likely to yield the lowest
convex hull energies (see Supplemental Material [48]). Starting structures for
these binary compounds were obtained from the Materials Project [49] and then
re-relaxed with our settings.

Insulators with energies less than $\sim$50 meV above the convex hull tend to
be stable [50]. We therefore use an energy cutoff of 50 meV/atom as our
criteria for thermodynamic stability when identifying candidate materials. The
calculated energy above the hull for each compound is shown in Fig. 2(c,d).
Energies higher than the reference compound are considered unfavorable and are
represented with cool colors, while energies lower than the reference compound
are favorable and represented with warm colors. Again, the reference compounds
are shown in white and marked with an asterisk, and compounds with structural
instabilities (as determined by phonon calculations) are marked with an ‘X’.
There does not appear to be a clear connection between convex hull energy and
structural stability or ion size.

###  III.3 Comparing Structural Parameters

In addition to structural and thermodynamic stability, we use Cu-O-Cu bond
angles and spacings between kagome layers as additional metrics to rank the
candidate compounds. A Cu-O-Cu bond angle approaching 180∘ leads to a large
antiferromagnetic superexchange interaction while minimizing undesirable
Dzyaloshinskii–Moriya interactions. Larger bond angles are therefore highly
desirable. Additionally, a greater separation between the kagome layers
isolates the two-dimensional magnetic subsystems and suppresses unwanted
coupling between planes. In Fig. 3, these two structural properties are
displayed for all candidate compounds. Squares corresponding to specific
compounds are colored and marked according to the same system described for
Fig. 2, where bond angles and interplane distances larger (smaller) than the
reference compounds are favorable (unfavorable) and represented with warm
(cool) colors, and structurally unstable compounds continue to be marked with
an ‘X’. Compounds with larger cation and anion radii generally lead to larger
bond angles and interplane distances, but also tend to be structurally
unstable. Compounds containing group IV elements are unstable and tend to have
smaller bond angles.

In Fig. 4, we investigate the effects of ion size on the physical properties
of the candidate compounds in more detail. In Fig. 4(a), the Cu-O-Cu bond
angle is plotted versus anion radius for the structurally stable materials.
The anion size plotted on the horizontal axis for Zn-barlowite relatives
refers to the $C$-site anion that occupies the same position as F in the
reference compound [ZnCu3(OH)6BrF] because it has the largest influence on
bond angle. For all materials, bond angle increases with increasing anion
size, and for a given anion, the bond angle also increases with increasing
cation size. Figure 4(b) shows the kagome plane spacing versus cation radius
for stable compounds, with separate traces for each anion. As expected, a
larger cation radius leads to greater distance between the kagome layers. For
a given cation, interplane distance also increases with increasing anion size.
In Fig. 4(c), we find that while the $C$-site anion has the greatest effect on
the Cu-O-Cu bond angle, larger bond angles are obtained when the $B$-site
anion is similar in size to the $C$-site anion.

We examine the effect of ion size on the lattice parameters of stable
compounds in Fig. 4(d). The $c$-axis length primarily increases with cation
size while the $a$-axis length primarily increases with anion size, although
anion size has a much weaker affect on the $a$-axis than cation size does on
the $c$-axis. The frequency of the lowest optical phonon mode ($f_{0}$) is
plotted against $c$-axis length in Fig. 4(e) for both stable (filled markers)
and unstable (empty markers) structures. Of all the structural parameters, the
$c$-axis length has the highest correlation with $f_{0}$. For herbertsmithite
relatives, as the $c$-axis increases, $f_{0}$ decreases, meaning compounds
tend to be less dynamically stable. Compounds containing group IV ions (Ge,
Sn, Pb) are plotted in darker shades for both structural families because
nearly all compounds containing these elements are unstable. Of the compounds
not containing group IV ions, $c$-axis lengths that are very small or very
large lead to structural instabilities. Compounds containing cations from
groups IIA and IIB which are close in size to Zn tend to be most stable. Fig.
4(f) shows Cu-O-Cu bond angle versus $a$-axis length. We find that a larger
$a$-axis leads to a larger bond angle, which agrees with the results in Fig.
4(a), where bond angle is positively correlated with anion radius, and Fig.
4(d), which shows the positive correlation between anion size and the length
of the $a$-axis. It should be noted that many unstable compounds containing
group IV elements have much smaller bond angles than most other candidates.

We also explored correlations between Cu-O-Cu bond angle, interplane distance,
and in-plane Cu-Cu bond length. These plots can be found in the Supplemental
Material [48]. The Cu-O-Cu bond angle has a weak positive correlation with
interplane distance. There is also a positive correlation between in-plane Cu-
Cu distance and Cu-O-Cu bond angle, as both are influenced by the length of
the $a$-axis, which increases with increasing anion size. There is no obvious
correlation between the interplane kagome distance and the in-plane Cu-Cu bond
length, as the interplane distance depends mostly on cation size, and in-plane
bond length depends on anion size. Overall, for both structural families,
compounds with cations of intermediate size (Mg, Zn, Cd, and Hg) are most
stable. Compounds containing group IV elements (Ge, Sn, Pb) are mostly
unstable. Larger anions and cations lead to favorable structural properties,
such as larger bond angles and interplane distances, but may also lead to
distortions of the kagome layers or other structural instabilities.

Table 1: Properties of the most promising QSL candidate materials as compared to the reference materials. The references (herbertsmithite and Zn-barlowite) are highlighted in gray, and the final candidates (with no instabilities throughout the Brillouin zone) are marked with asterisks. Compound |  $f_{0}$ (THz) |  $E_{\mathrm{hull}}$ (meV/atom) |  $E_{d}^{f}$ (eV) |  $\theta$ (deg) |  $d_{\mathrm{inter}}$ (Å) |  $d_{\mathrm{in}}$ (Å)
---|---|---|---|---|---|---
BaCu3(OH)6I2 | 0.41 | 42.6 | 2.42 | 128.0 | 6.09 | 3.53
CaCu3(OH)6Br2 | 0.50 | 30.7 | 0.87 | 125.7 | 5.19 | 3.53
CaCu3(OH)6Cl2 | 0.70 | 44.8 | 0.57 | 125.8 | 5.06 | 3.51
MgCu3(OH)6Br2 * | 2.23 | 36.0 | 0.36 | 125.2 | 4.65 | 3.57
ZnCu3(OH)6Cl2 | 2.63 | 41.2 | 0.13 | 125.0 | 4.58 | 3.53
CaCu3(OH)6IBr | 0.77 | 31.6 | 0.74 | 127.7 | 5.20 | 3.58
CaCu3(OH)6ICl * | 0.94 | 19.2 | 0.72 | 125.4 | 5.17 | 3.54
MgCu3(OH)6ClF * | 1.09 | 39.6 | 0.39 | 118.1 | 4.60 | 3.38
MgCu3(OH)6BrCl | 0.35 | 26.9 | 0.30 | 126.1 | 4.61 | 3.56
ZnCu3(OH)6BrF | 1.41 | 38.6 | 0.10 | 118.0 | 4.69 | 3.39
ZnCu3(OH)6ClF | 0.89 | 43.1 | 0.07 | 118.5 | 4.64 | 3.38

###  III.4 Defect Formation Energy

Herbertsmithite and Zn-barlowite are both susceptible to cation disorder. In
herbertsmithite, the Jahn-Teller active $d^{9}$ Cu2+ ion occupies the
tetragonally elongated site in the center of the CuO4Cl2 octahedra. The
$d^{10}$ Zn2+ ions are not Jahn-Teller active, and occupy the higher-symmetry
trigonally compressed octahedral sites between the kagome layers. Due to the
electronic configurations of the ions and distinct coordination environments,
it is not favorable for Zn to occupy the in-plane sites within the kagome
layer. However, herbertsmithite is the $x=1$ end member of the Zn-paratacamite
family [ZnxCu4-x(OH)6Cl2], and there is a preference for some Cu to exist on
the interlayer site instead of full occupation with Zn alone [7]. The
equilibrium occupation of the interlayer site by Cu has been estimated to be
as large as 15% in herbertsmithite [26, 27].

In Zn-barlowite, the interlayer site has a trigonal prismatic geometry, making
it even less favorable for the Jahn-Teller active Cu2+ ion. As a result, the
interlayer Cu occupation is only $\sim$5% in Zn-barlowite [33], confirming
early computational predictions [31, 32]. Site-specific x-ray diffraction
measurements have shown that there are two distinct interlayer sites in Zn-
barlowite: an off-center $C_{2v}$ site and a central $D_{3h}$ site. The
interlayer Cu defects occupy the $C_{2v}$ sites. It should be noted that even
for large concentrations of magnetic impurities on the interlayer site, Zn-
barlowite does not show signs of magnetic ordering, indicating that the
possible QSL phase is somewhat robust against interlayer magnetic impurities
[33].

An ideal QSL candidate will have only non-magnetic ions on the interlayer
sites, and therefore must have a high energy cost for interlayer Cu
substitution. We calculated the formation energy of such defects in a select
number of our most promising candidates (those structurally stable, with
$E_{\mathrm{hull}}<50$ meV/atom, and with bond angles and interplane distances
larger than the reference compounds). Since nearly all experimental and
computational studies indicate that there is negligible substitution of non-
magnetic ions within the kagome layers, we consider only interlayer defects.
The general expression for the formation energy of a charge-neutral
substitutional defect is

${E_{d}^{f}=E[\mathrm{defect}]-E[\mathrm{bulk}]+(\mu_{A}-\mu_{\mathrm{Cu}})=\Delta
E_{s}+\Delta\mu,}$

where $\Delta E_{s}$ is the difference in energy between a structure with a
single defect and the pristine bulk structure and $\Delta\mu$ is the chemical
potential difference of $A$ and Cu. To calculate $E[\mathrm{defect}]$, we
construct defect structures from $2\times 2\times 2$ supercells of
herbertsmithite relatives and $2\times 2\times 1$ supercells of Zn-barlowite
relatives, with a single Cu substitution. A depiction of our defect
configuration can be found in the Supplemental Material [48]. We relax the
atomic positions of the defect structures and subtract the energy of the
original defect-free structure to obtain $\Delta E_{s}$.

Figure 5: Phonon dispersions of final candidates. (a) The phonon dispersion
for MgCu3(OH)6Br2 (blue) overlaid with the reference dispersion for
herbertsmithite (gray). (b) The phonon dispersion for CaCu3(OH)6ICl (blue)
overlaid with the reference dispersion for Zn-barlowite (gray). (c) The phonon
dispersion for MgCu3(OH)6ClF (blue) overlaid with the reference dispersion for
Zn-barlowite (gray). The absence of imaginary phonon frequencies in all three
cases confirms the structural stability of these candidate compounds.

The chemical formulas for the defect-containing and defect-free configurations
are not equivalent, so the chemical potential difference
$\Delta\mu=\mu_{A}-\mu_{\mathrm{Cu}}$ must be considered. Interlayer defects
are primarily created during the initial growth of the material. During
synthesis of $A$Cu3(OH)${}_{6}B_{2}$, the chemical potentials of the
constituent elements must satisfy the inequality

${\mu_{A}+3\mu_{\mathrm{Cu}}+6\mu_{\mathrm{OH}}+2\mu_{B}>E[A\mathrm{Cu}_{3}(\mathrm{OH})_{6}B_{2}].}$

Individual chemical potentials must all be less than zero ($\mu_{A}<0$,
$\mu_{B}<0$, $\mu_{\mathrm{OH}}<0$, and $\mu_{\mathrm{Cu}}<0$). Additionally,
the formation of unwanted side products must be avoided, imposing the
additional inequalities

${\mu_{A}+2\mu_{B}<E[AB_{2}],}$
${\mu_{\mathrm{Cu}}+2\mu_{B}<E[\mathrm{Cu}B_{2}],}$
${\mu_{A}+2\mu_{\mathrm{OH}}<E[A(\mathrm{OH})_{2}].}$

Similar inequality constraints exist for $A$Cu3(OH)${}_{6}BC$. A higher defect
formation energy is preferable to minimize disorder. To maximize $E_{d}^{f}$,
we must maximize the chemical potential difference $\Delta\mu$ subject to the
above inequality constraints. The defect formation energies calculated with
these optimal values of $\Delta\mu$ are given in Table 1. All candidate
compounds investigated had a higher energy cost for interlayer defects than
herbertsmithite and Zn-barlowite except ZnCu3(OH)6ClF (Zn-substituted
claringbullite).

Two previous computational studies investigated doping selectivity in
barlowite [31, 32]. In both cases, the authors investigated the likelihood of
substituting various non-magnetic ions into the interlayer and intralayer
sites of barlowite, in contrast to the present work where we examine the
energy cost of a Cu defect on an interlayer site in fully-substituted
$A$-barlowite ($A$ = Zn, Mg, Ca). Despite differences in the methodology used
to construct defect structures and calculate the chemical potential
differences, our findings are generally consistent with those studies, which
suggested Zn and Mg to be the most favorable ions for synthesizing barlowite-
related compounds. More details on our defect formation energy calculations
can be found in the Supplemental Material [48].

###  III.5 Selecting Promising Candidates

After eliminating all compounds with structural instabilities at the $\Gamma$
point, formation energies greater than 50 meV/atom above the convex hull, and
Cu-O-Cu bond angles smaller than the reference compounds, 9 candidate
materials remained. For these candidates, we calculated the defect formation
energy $E_{d}^{f}$. To determine a final ranking, we used the following
criteria:

  1. 1.

Structural stability ($f_{0}>0$)

  2. 2.

Convex hull energy (E${}_{\mathrm{hull}}<50$ meV/atom)

  3. 3.

Defect energy cost ($E_{d}^{f}[\mathrm{candidate}]>E_{d}^{f}[\mathrm{ref}]$)

  4. 4.

Cu-O-Cu bond angle ($\theta>\theta^{\mathrm{ref}}$)

All compounds satisfying these criteria are listed with their associated
properties in Table 1. Complete data sets for all 44 herbertsmithite relatives
and 66 Zn-barlowite relatives can be found in the Supplemental Material [48].

We also verified structural stability by calculating the full phonon
dispersion throughout the entire Brillouin zone using the finite displacement
method within the phonopy code [51]. Such calculations can identify structural
instabilities associated with an enlargement of the unit cell. Dispersion
curves were calculated for all candidates in Table 1. However, only one
compound in the herbertsmithite family and two compounds in the Zn-barlowite
family were found to be stable throughout the entire Brillouin zone. The
dispersion curves of these compounds are shown in Fig. 5, while dispersions
for all compounds in Table 1 can be found in the Supplemental Material [48].
Surprisingly, while Zn-claringbullite [ZnCu3(OH)6ClF] is known to have perfect
kagome layers at room temperature [41], our ground state dispersion shows
instabilities at the $M$ and $K$ points (see Supplemental Material [48]). The
instabilities we observe in DFT may be avoided by thermal fluctuations at room
temperature, which could explain the discrepancy between our calculations and
the experimental results. Two other Zn-barlowite-related candidate compounds
listed in Table 1, CaCu3(OH)6IBr and MgCu3(OH)6BrF, showed similar
instabilities, and therefore may also be stable at room temperature (see
Supplemental Material [48]).

Our calculations identify MgCu3(OH)6Br2 as a potential candidate within the
herbertsmithite family, as well as CaCu3(OH)6ICl and MgCu3(OH)6ClF in the Zn-
barlowite family. However, some practical considerations related to synthesis
may require further investigation. For instance, the Mg analog of Zn-barlowite
[MgCu3(OH)6BrF] has not been synthesized due to the insolubility of MgF2 in
water. While synthesis of Zn-barlowite using NH4F yields a structurally
equivalent compound, crystals obtained using this method show a similar
magnetic transition to barlowite, suggesting possible differences in defect
structures between the two synthesis methods [52]. The insolubility of MgF2
may therefore present difficulty in synthesizing our candidate MgCu3(OH)6ClF
[41]. Synthesis of MgCu3(OH)6Br2 has been attempted, but the desired product
was a Zn-barlowite analog [42]. The synthesis method, which followed the
typical hydrothermal procedure, resulted in a compound with $P\bar{3}m1$
symmetry, which may mean that the herbertsmithite $R\bar{3}m$ structure is not
favored in this reaction. It is possible that other synthesis methods could
yield different results. To our knowledge, no experimental studies have been
performed on the Ca analog of either herbertsmithite or Zn-barlowite, nor any
related compounds containing I.

##  IV Conclusion

In summary, we performed a systematic combinatorial exploration of
herbertsmithite and Zn-barlowite material relatives and identified those with
properties that may enhance the likelihood of an ideal QSL ground state. We
found several promising candidates—MgCu3(OH)6Br2, CaCu3(OH)6ICl, and
MgCu3(OH)6ClF—that are structurally stable, thermodynamically feasible to
synthesize, have high energy costs for interlayer defects, and whose
structural properties may result in antiferromagnetic superexchange
interactions stronger than herbertsmithite or Zn-barlowite. These compounds,
if they can be synthesized, may prove to be better QSL candidates than their
well-studied counterparts.

## Acknowledgments

We would like to thank Siavash Karbasizadeh for helpful discussions. This work
was supported by the Air Force Office of Scientific Research under AFOSR award
no. FA9550-21-1-0337. C.A. acknowledges support from the UCSB Quantum Foundry
Internship Program, which is funded by the National Science Foundation (NSF)
through Enabling Quantum Leap: Convergent Accelerated Discovery Foundries for
Quantum Materials Science, Engineering, and Information (Q-AMASE-i): Quantum
Foundry at UC Santa Barbara (DMR-1906325). Use was made of computational
facilities purchased with funds from the NSF (CNS-1725797) and administered by
the Center for Scientific Computing (CSC). The CSC is supported by the
California NanoSystems Institute and the Research Science and Engineering
Center (MRSEC; NSF DMR-1720256) at UC Santa Barbara.

## References

  * Balents [2010] L. Balents, Nature 464, 199 (2010).
  * Savary and Balents [2016] L. Savary and L. Balents, Reports on Progress in Physics 80, 016502 (2016).
  * Broholm  _et al._ [2020] C. Broholm, R. J. Cava, S. A. Kivelson, D. G. Nocera, M. R. Norman, and T. Senthil, Science 367, eaay0668 (2020).
  * Semeghini  _et al._ [2021] G. Semeghini, H. Levine, A. Keesling, S. Ebadi, T. T. Wang, D. Bluvstein, R. Verresen, H. Pichler, M. Kalinowski, R. Samajdar, A. Omran, S. Sachdev, A. Vishwanath, M. Greiner, V. Vuletić, and M. D. Lukin, Science 374, 1242 (2021).
  * Mendels and Bert [2010] P. Mendels and F. Bert, Journal of the Physical Society of Japan 79, 011001 (2010).
  * Norman [2016] M. R. Norman, Reviews of Modern Physics 88, 041002 (2016).
  * Shores  _et al._ [2005] M. P. Shores, E. A. Nytko, B. M. Bartlett, and D. G. Nocera, Journal of the American Chemical Society 127, 13462 (2005).
  * de Vries  _et al._ [2009] M. A. de Vries, J. R. Stewart, P. P. Deen, J. O. Piatek, G. J. Nilsen, H. M. Rønnow, and A. Harrison, Physical Review Letters 103, 237201 (2009).
  * Han  _et al._ [2011] T. H. Han, J. S. Helton, S. Chu, A. Prodi, D. K. Singh, C. Mazzoli, P. Müller, D. G. Nocera, and Y. S. Lee, Physical Review B 83, 100402 (2011).
  * Helton  _et al._ [2007] J. S. Helton, K. Matan, M. P. Shores, E. A. Nytko, B. M. Bartlett, Y. Yoshida, Y. Takano, A. Suslov, Y. Qiu, J.-H. Chung, D. G. Nocera, and Y. S. Lee, Physical Review Letters 98, 107204 (2007).
  * Mendels  _et al._ [2007] P. Mendels, F. Bert, M. A. de Vries, A. Olariu, A. Harrison, F. Duc, J. C. Trombe, J. S. Lord, A. Amato, and C. Baines, Physical Review Letters 98, 077204 (2007).
  * Fu  _et al._ [2015] M. Fu, T. Imai, T.-H. Han, and Y. S. Lee, Science 350, 655 (2015).
  * Han  _et al._ [2016] T.-H. Han, M. R. Norman, J.-J. Wen, J. A. Rodriguez-Rivera, J. S. Helton, C. Broholm, and Y. S. Lee, Physical Review B 94, 060409 (2016).
  * Olariu  _et al._ [2008] A. Olariu, P. Mendels, F. Bert, F. Duc, J. C. Trombe, M. A. de Vries, and A. Harrison, Physical Review Letters 100, 087202 (2008).
  * Han  _et al._ [2012] T.-H. Han, J. S. Helton, S. Chu, D. G. Nocera, J. A. Rodriguez-Rivera, C. Broholm, and Y. S. Lee, Nature 492, 406 (2012).
  * Wulferding  _et al._ [2010] D. Wulferding, P. Lemmens, P. Scheib, J. Röder, P. Mendels, S. Chu, T. Han, and Y. S. Lee, Physical Review B 82, 144412 (2010).
  * de Vries  _et al._ [2008] M. A. de Vries, K. V. Kamenev, W. A. Kockelmann, J. Sanchez-Benitez, and A. Harrison, Physical Review Letters 100, 157205 (2008).
  * Khuntia  _et al._ [2020] P. Khuntia, M. Velazquez, Q. Barthélemy, F. Bert, E. Kermarrec, A. Legros, B. Bernu, L. Messio, A. Zorko, and P. Mendels, Nature Physics 16, 469 (2020).
  * Ofer  _et al._ [2011] O. Ofer, A. Keren, J. H. Brewer, T. H. Han, and Y. S. Lee, Journal of Physics: Condensed Matter 23, 164207 (2011).
  * Jeschke  _et al._ [2013] H. O. Jeschke, F. Salvat-Pujol, and R. Valenti, Physical Review B 88, 075106 (2013), arXiv:1303.1310 [cond-mat] .
  * Suttner  _et al._ [2014] R. Suttner, C. Platt, J. Reuther, and R. Thomale, Physical Review B 89, 020408 (2014).
  * [22] O. Janson, , 223.
  * Rigol and Singh [2007] M. Rigol and R. R. P. Singh, Physical Review Letters 98, 207204 (2007).
  * Götze and Richter [2016] O. Götze and J. Richter, EPL (Europhysics Letters) 114, 67004 (2016).
  * Li  _et al._ [2020] Y. Li, A. Pustogow, M. Bories, P. Puphal, C. Krellner, M. Dressel, and R. Valentí, Physical Review B 101, 161115 (2020).
  * Freedman  _et al._ [2010] D. E. Freedman, T. H. Han, A. Prodi, P. Müller, Q.-Z. Huang, Y.-S. Chen, S. M. Webb, Y. S. Lee, T. M. McQueen, and D. G. Nocera, Journal of the American Chemical Society 132, 16185 (2010).
  * Imai  _et al._ [2011] T. Imai, M. Fu, T. H. Han, and Y. S. Lee, Physical Review B 84, 020411 (2011).
  * de Vries  _et al._ [2012] M. A. de Vries, D. Wulferding, P. Lemmens, J. S. Lord, A. Harrison, P. Bonville, F. Bert, and P. Mendels, Physical Review B 85, 014422 (2012).
  * Feng  _et al._ [2017] Z. Feng, Z. Li, X. Meng, W. Yi, Y. Wei, J. Zhang, Y.-C. Wang, W. Jiang, Z. Liu, S. Li, F. Liu, J. Luo, S. Li, G.-q. Zheng, Z. Y. Meng, J.-W. Mei, and Y. Shi, Chinese Physics Letters 34, 077502 (2017).
  * Tustain  _et al._ [2020] K. Tustain, B. Ward-O’Brien, F. Bert, T. Han, H. Luetkens, T. Lancaster, B. M. Huddart, P. J. Baker, and L. Clark, npj Quantum Materials 5, 1 (2020).
  * Liu  _et al._ [2015] Z. Liu, X. Zou, J.-W. Mei, and F. Liu, Physical Review B 92, 220102 (2015).
  * Guterding  _et al._ [2016] D. Guterding, R. Valentí, and H. O. Jeschke, Physical Review B 94, 125136 (2016).
  * Smaha  _et al._ [2020a] R. W. Smaha, I. Boukahil, C. J. Titus, J. M. Jiang, J. P. Sheckelton, W. He, J. Wen, J. Vinson, S. G. Wang, Y.-S. Chen, S. J. Teat, T. P. Devereaux, C. D. Pemmaraju, and Y. S. Lee, Physical review materials 4, 10.1103/physrevmaterials.4.124406 (2020a).
  * Smaha  _et al._ [2020b] R. W. Smaha, W. He, J. M. Jiang, J. Wen, Y.-F. Jiang, J. P. Sheckelton, C. J. Titus, S. G. Wang, Y.-S. Chen, S. J. Teat, A. A. Aczel, Y. Zhao, G. Xu, J. W. Lynn, H.-C. Jiang, and Y. S. Lee, npj Quantum Materials 5, 1 (2020b).
  * Pasco  _et al._ [2018] C. Pasco, I. Heinmaa, R. Stern, C. Broholm, and T. McQueen, 2018, H24.006 (2018).
  * Chu  _et al._ [2010] S. Chu, T. M. McQueen, R. Chisnell, D. E. Freedman, P. Müller, Y. S. Lee, and D. G. Nocera, Journal of the American Chemical Society 132, 5570 (2010).
  * Colman  _et al._ [2010] R. Colman, A. Sinclair, and A. Wills, Chemistry of Materials 22, 5774 (2010).
  * Malcherek  _et al._ [2014] T. Malcherek, L. Bindi, M. Dini, M. R. Ghiara, A. M. Donoso, F. Nestola, M. Rossi, and J. Schlüter, Mineralogical Magazine 78, 583 (2014).
  * McQueen  _et al._ [2011] T. M. McQueen, T. H. Han, D. E. Freedman, P. W. Stephens, Y. S. Lee, and D. G. Nocera, Journal of Solid State Chemistry 184, 3319 (2011).
  * Braithwaite  _et al._ [2004] R. S. W. Braithwaite, K. Mereiter, W. H. Paar, and A. M. Clark, Mineralogical Magazine 68, 527 (2004).
  * Feng  _et al._ [2018] Z. Feng, W. Yi, K. Zhu, Y. Wei, S. Miao, J. Ma, J. Luo, S. Li, Z. Y. Meng, and Y. Shi, Chinese Physics Letters 36, 017502 (2018).
  * Wei  _et al._ [2019] Y. Wei, Z. Feng, C. dela Cruz, W. Yi, Z. Y. Meng, J.-W. Mei, Y. Shi, and S. Li, Physical Review B 100, 155129 (2019).
  * Kresse and Hafner [1993] G. Kresse and J. Hafner, Physical Review B 47, 558 (1993).
  * Kresse and Furthmüller [1996a] G. Kresse and J. Furthmüller, Physical Review B 54, 11169 (1996a).
  * Kresse and Furthmüller [1996b] G. Kresse and J. Furthmüller, Computational Materials Science 6, 15 (1996b).
  * Kresse and Joubert [1999] G. Kresse and D. Joubert, Physical Review B 59, 1758 (1999).
  * Perdew  _et al._ [1996] J. P. Perdew, K. Burke, and M. Ernzerhof, Physical Review Letters 77, 3865 (1996).
  * [48] See {{}}Supplemental Material{}{} at [{{}}URL{}{} Will Be Inserted by Publisher] for complete data set, phonon dispersions, and input files for calculations.
  * Jain  _et al._ [2013] A. Jain, S. P. Ong, G. Hautier, W. Chen, W. D. Richards, S. Dacek, S. Cholia, D. Gunter, D. Skinner, G. Ceder, and K. A. Persson, APL Materials 1, 011002 (2013).
  * Jiang  _et al._ [2022] Y. Jiang, Z. Yu, Y. Wang, T. Lu, S. Meng, K. Jiang, and M. Liu, Chinese Physics Letters 39, 047402 (2022).
  * Togo and Tanaka [2015] A. Togo and I. Tanaka, Scripta Materialia 108, 1 (2015).
  * Smaha  _et al._ [2018] R. W. Smaha, W. He, J. P. Sheckelton, J. Wen, and Y. S. Lee, Journal of Solid State Chemistry 268, 123 (2018).


# Evidence-centered Assessment for Writing with Generative AI

Yixin Cheng
Monash University
<EMAIL_ADDRESS>
Kayley Lyons
University of Melbourne
<EMAIL_ADDRESS>
Guanliang Chen
Monash University
<EMAIL_ADDRESS>
Dragan Gašević
Monash University
<EMAIL_ADDRESS>
Zachari Swiecki
Monash University
<EMAIL_ADDRESS>

###### Abstract

We propose a learning analytics-based methodology for assessing the
collaborative writing of humans and generative artificial intelligence. Framed
by the evidence-centered design, we used elements of knowledge-telling,
knowledge transformation, and cognitive presence to identify assessment
claims; we used data collected from the CoAuthor writing tool as potential
evidence for these claims; and we used epistemic network analysis to make
inferences from the data about the claims. Our findings revealed significant
differences in the writing processes of different groups of CoAuthor users,
suggesting that our method is a plausible approach to assessing human-AI
collaborative writing.

_K_ eywords Generative Artificial Intelligence $\cdot$ Assessment $\cdot$
Evidence-centered Design $\cdot$ Epistemic Network Analysis

##  1 Introduction

Effectively communicating ideas via writing is a critical human skill. Every
day, many of us send text messages, draft emails, and make notes; in many
specialised domains, such as research, writing is a core form of discourse.
The process of writing has, of course, changed over time; writing tools have
transformed from mere storage receptacles to tools that help us craft more
effective writing, such as word processors, spellcheckers, and grammar
checkers [1]. Recently, however, there has been a step-change in tools for
writing. Whereas prior tools helped us to save and process our own writing,
tools-based on _generative artificial intelligence_ (GAI) can now compose
writing for us.

This technological advance has already had far reaching implications for
society. Arguably, these implications have been—and will continue to be—the
most profound for education. Education relies, in part, on assessment. Broadly
speaking, we want students to learn skills that will be valuable to their
well-being and the well-being of society. But to know and communicate whether
learning has occurred, we need to be able to _evidence_ that learning.
Assessments are the structures that make such inferences about learning
possible [2]. A key feature of assessments is that they assign credit to both
pragmatic actions—actions that advance toward a goal state—and epistemic
actions—actions that reduce cognitive effort [3]. However, when students use
tools like ChatGPT [4] to help them write, these actions are mixed between the
human and the tool [5], making the assignment of credit difficult. How, for
example, should a teacher assess writing crafted partly by a human and partly
by AI?

In the last year, educational institutions across the world have scrambled to
put policies in place that seek to address questions like the one above. For
example, the Tertiary Education Quality and Standards Agency (TEQSA),
responsible for regulating and the quality of all providers of higher
education in Australia, recently convened a panel of experts to produce a
whitepaper on “assessment reform in the age of artificial intelligence” that
outlines broad changes institutes of higher education will be expected to make
in response to AI [6]. Rather than cut students off from AI, its authors argue
that assessments should prepare students for a world where collaboration with
AI is commonplace by redesigning assessments to emphasize: (a) appropriate and
authentic engagement with AI; (b) the process of learning; and (c)
opportunities for students to work with each other and AI.

We agree with these recommendations. In turn, we argue that there is a need to
re-develop assessments to account for interactions between humans and AI in
ways that afford reasoned arguments about learning claims from verifiable and
persistent evidence. In this paper, we propose and test such an assessment
informed by (a) the evidence-centered design (ECD) assessment framework (b)
extant theories of the cognitive processes involved in writing (c) recent
interfaces to GAI, and (d) learning analytic process models. Our results
suggest that this method can find expected differences between the processes
people use when writing with GAI, such as differences between genres. This
work is a proof of concept that contributes to a deeper understanding of
human-AI collaboration and suggests a path forward for innovative assessments
in this new age of AI.

##  2 Background and Theory

Educational assessments are a means for determining whether and how students
have learned. While several frameworks for assessment design have been
proposed, the ECD framework has proven to be particularly successful and
adaptable to a variety of domains [2]. ECD defines a Conceptual Assessment
Framework that incorporates three high-level models: a _student model_ , _task
model_ , and _evidence model_. Together, these components suggest a coherent
and logical approach for designing assessments that are aligned with the
constructs they intend to measure.

###  2.1 ECD Student Model

The student model identifies and describes the claims about learning that the
assessment aims to measure. Our study aimed to assess the processes students
enact while writing with GAI as opposed to the quality of the final writing
product. There is a long history of assessing writing ability—as well as
learning more generally—in terms of products like essays, posts, and articles.
For example, automated essay scoring [7] and writing analytics [8] have
developed into active sub-fields of psychometrics and learning analytics,
respectively. Such research traditionally uses natural language processing
(NLP) techniques to assess writing quality and rhetorical structure. More
recently, work has transitioned from machine learning and traditional NLP
models to analyses based on generative language models [9, 10].

While assessing products is undoubtedly important, assessing process has been
recognised as valuable in the learning sciences and learning analytics
communities because it offers a more complete view of student abilities,
facilitates effective feedback, and aids in creating adaptive scaffolds for
personalized needs [11, 12]. We argue that a process-oriented view of
assessment will help to resolve the many of the challenges current educators
face when incorporating or accounting for GAI in their assessment design. Much
of the worry surrounding assessment and GAI is centered on the only evidence
of learning being a product with unknown providence [13]. Altering assessment
to instead focus on writing processes will make it clearer who (the student or
the AI) produced which actions and how these actions might positively or
negatively impact student learning.

While few studies of how people co-write with GAI have been reported to date,
the interfaces of various GAI tools suggest particular kinds of interaction
patterns. ChatGPT [4], for example, includes a text area for prompting the AI
and the ability to copy text, regenerate text, and like or dislike a response.
Similarly, Google’s _Bard_ [14] lets users modify responses within the
interface and export them to emails or word documents directly from the tool.
These affordances suggest that when using GAI writers may engage in varied
processes of prompting, as well as exploring, using, and modifying AI
generated responses.

####  2.1.1 Knowledge Telling and Knowledge Transforming in Writing

Bereiter and Scardamalia’s work on knowledge building offers a lens with which
to view these kinds of interactive writing behaviors [15]. They distinguish
between _knowledge telling_ —a linear process of writing where individuals
focus on transcribing information without deep engagement or critical
restructuring—and knowledge transformation, a more critical engagement that
restructures arguments thoughtfully, synthesizes different viewpoints, and
creates narratives that are both cohesive and compelling. Raković [16]
extended this work to argue that knowledge transformation is a set of actions
whereby writers comprehend, evaluate, and select information sources to
fostering novel connections among previously disconnected fragments of
knowledge to facilitate learning.

In the context of writing with GAI, we operationalize knowledge telling and
knowledge transformation in terms of how learners select, modify, and apply
AI-generated text. More specifically, knowledge telling might appear as
learners accepting suggestions from the AI without alteration, following the
path laid out by the AI at any given moment. Knowledge transformation, on the
other hand, can be seen when users take AI suggestions as a basis for revision
and modify them to fit their needs. Learners may also engage in actions such
as declining initial GAI suggestions and seeking alternatives several times;
or they might deliberate between different sets of suggestions. Such example
defy a simple categorization under knowledge telling or knowledge
transformation. However, we argue below that the cognitive presence framework
can help us to understand these interactions.

####  2.1.2 Cognitive Presence

Garrison situated [17] cognitive presence within the Community of Inquiry
(CoI) framework—a theoretical framework that promotes educational experience
through computer-mediated communication. Cognitive presence refers to the
extent to which learners are able to construct and confirm meaning through
communication in a specific sociocultural context. It is manifested through a
four-phase cycle that includes a _triggering event_ —the initial encounter
with a problem or question that ignites curiosity; _exploration_ —where
learners actively seek information to deepen their understanding to the
question; _integration_ —where gathered information is synthesized to
formulate coherent responses; and finally, _resolution_ —where synthesized
knowledge is used to propose a solution [17]. Prior work has largely focused
on the context of online discussion—for example, automatically identifying
cognitive presence in messages [18] and associating cognitive presence with
particular speech acts [19]. However, cognitive presence has yet to be
investigated in the context of collaborative writing between humans and AI.

In this study, we operationalize cognitive presence in terms of interactions
that take place while co-writing with GAI. Triggering events may occur when
learners approach GAI with an initial question or problem they encounter while
they are writing. Exploration may occur as learners actively engage with the
GAI to brainstorm or search through various suggestions. Integration may occur
as learners synthesize the ideas and suggestions received from the AI, working
towards constructing a cohesive piece of writing. In this way, integration is
similar to knowledge transformation as described above. Finally, resolution
may occur when learners fine-tune their piece with the aid of the AI, seeking
advice on polishing the language and structure to reach a satisfactory final
product.

The combination of knowledge telling, knowledge transformation, and cognitive
presence suggests a set of claims to include in the student model for
assessing the processes involved in co-writing with GAI: using generative AI
responses as is (knowledge telling); modifying generative AI responses to fit
your own goals (knowledge transformation/integration); requesting information
from GAI (triggering); comparing different GAI outputs (exploration); and
arriving at a final product that aligns with the human writer’s intent
(resolution).

###  2.2 ECD Task model

The task model specifies the tasks and task environment students will interact
with to evidence their learning. As mentioned before, several commercial
interfaces to GAI exist. These tools allow individuals to prompt GAI and
integrate the outputs into their own writing. While powerful, they fall short
of requirements of an efficient task model because the data they capture is
not easily accessible for assessment purposes. This lack of the accessibility
leaves educators with few options: either attempt to ban students from using
GAI—which is likely to fail because it is notoriously difficult to invigilate
students’ online activity [20]—or try to document students’ use of GAI—which
is difficult and time-consuming. It is hard to imagine a scenario where a
teacher reviews hours of videos or analyzes numerous screenshots of AI
interactions for each of their students. A more viable solution would be
platforms designed to automatically capture and preserve evidence related to
writing with GAI.

One such platform is CoAuthor [21]—an online tool designed to (a) afford
interactions with GAI and (b) passively capture those interactions for
analysis. CoAuthor was designed as a test-bed for investigating human-AI
collaborative writing. It provides users with writing prompts either for
argumentative writing (e.g. essays) or creative writing (e.g. stories) and
asks users to compose a response. As they write, they can seek suggestions
from GAI that continue their writing. Users are free to explore, accept, or
dismiss suggestions and modify them if accepted. While users are constrained
in their interactions with GAI compared to commercial tools—for example, they
cannot prompt the AI with their own questions—CoAuthor provides a suitable
task environment for assessment purposes because it automatically logs and
makes accessible fine-grained interactions with the system. All keystrokes,
cursor movements, and interactions with GAI are logged in both a human and
machine readable format. Here, we use existing data collected as part of the
CoAuthor project to demonstrate what a valid assessment for writing with GAI
might look like.

###  2.3 ECD Evidence Model

The ECD evidence model describes how students’ responses to the task will be
related to the claims in the student model. Recently, the sub-field of writing
analytics has developed a variety of methods for evidencing claims from
writing data [22]. To understand writing processes, trace data (e.g. keystroke
logging, mouse movement, eye tracking, etc.) [13] are typically analyzed using
process models. For example, recent research [23, 24, 25] has explored
students’ cognitive and metacognitive processes during essay writing via trace
data by applying techniques like semi-Markov processes [26].

To better understand writing processes and process models, researchers often
use process graphs, which show the relationships or connections between
different writing behaviors. For example, Leijten and Waes [27] used the
process graph Inputlog111https://www.inputlog.net to visualize writing
progression over time. They further adapted work into a network visualization
with Pajek222http://mrvar.fdv.uni-lj.si/pajek/ to highlight the key
information sources used during writing. Other researchers have used
representations called revision maps to investigate writing processes. For
example, Southavilay et al. [28] developed a revision map to visually track
changes in paragraphs of a collaborative document over time using color-coded
rectangles to signify the nature and extent of edits. Likewise, Shibani and
colleagues [29] created a revision graph to analyze the editing process in
essay drafts using nodes to represent sentences and edges to depict the
organizational changes across drafts.

Building on revision maps and process graphs, Shibani et al. [30] introduced
CoAuthorViz. This more advanced process graph visualizes co-writing behaviors
with GPT-3 suggestions, drawing upon trace data from _CoAuthor_. Different
from traditional process graphs, it emphasizes three key constructs:
"Suggestion Selection", "Suggestion Modification", "Empty GPT-3 Call", and
"User Text Addition" detailing the sequence of writing actions at the sentence
level. These graphical representations offer insights into sentence-level
differences in compositions and revisions.

The process models and graph approaches described above are commendable for
their focus on the relationships between features that characterise writing;
however, they have important limitations as components of an evidence model.
First, traditional process graphs and revision maps are useful for
understanding individual or aggregated writing processes; however they do a
poor job of effectively comparing processes between individuals or groups.
This is primarily due to their lack of a coordinated semantic space for
network comparisons. Second, while Markov-models, traditional process graphs,
and revision maps excel at showing detailed relationships, they lack
corresponding summary statistics that can be used to quantify the network
interpretation. Such summary statistics are useful because they afford process
comparisons at scale using standard statistical techniques.

The process graph introduced in CoAuthorViz distinguishes itself by analyzing
writing across two groups of users and incorporating summary statistics.
Despite these advancements, the tool has several notable limitations. First,
the constructs included in the model are relatively broad and may therefore
miss the fine-grained behaviors essential for analyzing writing in a nuanced
and theoretically driven way. For example, "Suggestion Selection" consists of
three distinct behaviors—suggestion seeking, exploration, and acquirement, but
these are aggregated together and analyzed as a single construct that is not
explicitly related to theories that describe the cognitive aspects of writing.
Second, their comparison of group of users was based on the final written
product, not writing processes.

To address these limitations, we used epistemic network analysis (ENA) as the
major component of our evidence model. ENA is a widely used learning analytic
technique for modeling actions via network models [31]. ENA coordinates
multiple networks in the same embedding space to facilitate visual and
statistical comparisons of learning processes at scale. While ENA and related
techniques have been used to model metacognitive processes during writing
tasks [32], no work has explored ENA as method for evidencing the cognitive
behaviors inherent to co-writing with GAI.

###  2.4 Research Questions

Our study aimed to demonstrate an assessment method for human-AI collaborative
writing. Framed by ECD, we used elements of knowledge-telling, knowledge
transformation, and cognitive presence to identify claims for our student
model (see hypotheses below); we used data collected from the CoAuthor writing
tool as a proxy for our task model; and we used ENA to evidence claims for the
student model. Ideally, this assessment approach would distinguish high-
quality human-AI collaborative writing processes from low-quality ones.
Unfortunately, the CoAuthor dataset does not include writing quality scores
that could be correlated with writing process measures for this purpose.
However, the dataset does contain a number of conditions that should be
distinguishable with a valid assessment method. In particular, written
products and trace data were divided by how much ownership—i.e., the
proportion of content generated by the human author versus the GAI—the human
had over the final product; whether the author was responding to a creative
writing prompt or an argumentative writing prompt; and whether the GAI had a
high or low temperature setting—that is, whether its outputs had high or low
textual variability. To provide a proof of concept for our assessment method,
we compared these conditions according to the following research questions:

RQ1

Did authors with higher ownership over the final product write differently
than those with lower ownership over the final product?

RQ2

Did authors who responded to creative writing prompts write differently than
those who responded to argumentative prompts?

RQ3

Did authors who use GAI with a higher temperature setting—that is, more
variable output—write differently than those who used a lower temperature
setting?

Regarding RQ1, we hypothesized that: (1) authors who had lower ownership over
the final product would have writing processes more characterised by knowledge
telling and triggering events because they sought and relied more on ready-
made suggestions provided by the GAI; (2) conversely, authors with higher
ownership over the final product would have processes more characterised by
knowledge transformation, exploration, integration, and resolution because
they tended to adapt GAI suggestions to better fit their writing.

To address RQ2, we built upon genre theory [33], which suggests major
differences between fiction (e.g. creative) and non-fiction (e.g.
argumentative) writing. Namely, fiction writing is inherently less bound to a
"reality status" or tight relationship to the real world. Therefore, authors
in this genre are more free to explore the boundaries of imagination and
creativity without the limits of factual accuracy. On the other hand, non-
fiction writing is grounded in reality, requiring a structured approach to
presenting logical arguments based on facts and evidence. Thus, for RQ2 we
hypothesized that: (3) authors responding to creative writing prompts would
have writing processes more characterised by knowledge telling, triggering
events, and exploration as they would likely need to adapt GAI responses less
to maintain a reality status; and (4) authors responding to argumentative
writing prompts would have writing processes more characterised by knowledge
transformation, integration, and resolution as they may have needed to align
GAI outputs (which are known to include falsehoods [34]) with reality.

High temperature settings mean that GAI will tend to produce varied outputs to
the same request, while low temperature settings mean that GAI will tend to
produce repetitive outputs. Regarding RQ3, we hypothesized that: (5) authors
interacting with lower temperature GAI would have processes more characterized
by knowledge transformation, integration, and resolution as they would need to
actively think and rework the available suggestions; (6) while those
interacting with higher temperature GAI would have processes more
characterized by exploration and knowledge telling as they might find it
easier to select suggestions that align closely with their intended message
without significantly transforming the information.

##  3 Method

As shown in Figure 1, our methodological steps included (a) pre-processing the
keystroke-level data from CoAuthor; (b) qualitatively analyzing these data to
derive codes related to knowledge telling, transformation, cognitive presence,
and general writing behaviors for subsequent analysis; (c) the development,
validation, and application of classifiers for automatic assignment of the
codes to the data; (d) applying ENA to the data to create networks that
described the writing processes of authors in different experimental
conditions; and (e) using summary statistics from ENA to statistically compare
the experimental conditions via mixed-effects regression analysis.

Figure 1: Methodology overview (the leftmost image sourced from [21])

###  3.1 Data

We used data collected from the CoAuthor project as a proxy for the task model
of our proposed assessment methodology. These data include digital records of
1,445 writing sessions from 60 authors recruited from Amazon Mechanical Turk.
Authors were asked to respond to one or more prompts. Prompts could be from
the creative writing condition, which were sampled from the r/WritingPrompts
subreddit, or from the argumentative writing condition, which were sampled
from a New York Times prompt library. During the writing sessions, authors
interacted with a user interface that included a standard text editor as well
as a tab that allowed them to request up to five suggestions from GPT-3 for
continuing the currently written text.

Overall, elements of the dataset we used included two distinct types of data:

  * •

Trace Data: Sequential actions and events throughout the writing session.
These included user actions such as text insertion, deletion, and cursor
movement, as well as system interactions like opening, getting, and closing
suggestions. Furthermore, it traced the progression of the written content on
an event-by-event basis.

  * •

Metadata: Details such as the session time, participant ID, session ID, prompt
condition, prompt ID, GPT temperature, as well as metrics that summarise the
interactions between the user and GPT such as ownership—that is the percentage
of the final written text produced by the human author vs. GPT-3.

###  3.2 Pre-processing

The original data included 830 creative writing sessions and 615 argumentative
writing sessions which averaged 1,862 events per session. We removed eight
sessions from the analysis due to participants removing the original prompt
and commencing a different writing piece, and one session due to missing
metadata about identification of the author. In addition, we identified and
manually corrected or excluded events with formatting errors. Our final
dataset consisted of 822 creative writing sessions and 614 argumentative
writing sessions that included more than 2.6M events. For sessions using
argumentative writing prompts, low/high temperature values were 0.2 and 0.9;
for creative writing prompts they were 0.3 and 0.75. Author and GAI ownership
was defined using a median split on the percentage of characters authored by a
human (mdn = 76%).

###  3.3 Qualitative Analysis

ENA requires a coded dataset for analysis—that is, data where the relevant
actions have been labelled. To derive codes, we combined a top-down approach
informed by theory with a bottom-up approach that sought to identify relevant
themes in the data without explicit reference to theoretical frameworks[35].
We qualitatively investigated 40 randomly sampled sessions for code
identification. Two authors discussed and refined these codes to arrive at the
final list below in Table 1.

Table 1: Qualitative codes, definitions, and identifiers.

Code |  Definition |  Identifiers
---|---|---
compose |  Writing new content at the end of the existing text |  event name is "text-insert" in the end of text (space removed where applicable)
relocate |  Rearranging sentences |  The index of same sentence changes in last-current document
reflect |  Revising the content after or near completing a draft |  Revise content after finishing 90% content
seek suggestion |  Obtaining suggestions from GPT |  Event name is ’suggestion-get’
dismiss suggestion |  Dismissing suggestions from GPT |  Event name is ’suggestion-close’ and event source is ’user’
accept suggestion |  Accepting a suggestion from GPT |  Event name is ’suggestion-select’
hover suggestion |  Hovering over the suggestions |  Event name is ’suggestion-hover’
cursor forward |  Moving the position of cursor forward |  Event name is "cursor-forward"
cursor backward |  Moving the position of cursor backward |  Event name is "cursor-backward"
cursor select |  Selecting text |  Event name is "cursor-select"
revise user |  Revising content they wrote |  The inserts or deletes in-text content and ownership of revising sentence is ’user’
revise suggestion |  Revising a GPT suggestion |  The user inserts or deletes in-text content and ownership of revising sentence is ’api’
low modification |  Making minor adjustments to sentences without altering their core meaning |  Sentence semantic similarity > 0.8
high modification |  Making significant changes to sentence meaning |  Sentence semantic similarity < 0.8

In our analysis, codes were related to the student model in terms of the
presence or absence of co-occurrences or _connections_ between them (see 3.5).
We interpreted _knowledge telling_ in terms of connections among seek
suggestion, accept suggestion, revise suggestion, and low modification;
_knowledge transformation_ and _integration_ in terms of connections among
seek suggestion, accept suggestion, revise suggestion, high modification, and
relocate; _triggering events_ in terms of connections to seek suggestion and
accept suggestion; _exploration_ in terms of connections among seek
suggestion, accept suggestion, hover suggestion, and dismiss suggestion; and
_resolution_ in terms of connections among seek suggestion, accept suggestion,
and reflect. In addition to codes that we mapped to knowledge
telling/transformation/cognitive presence, our qualitative analysis suggested
that including codes related to composing, revising one’s own text, and cursor
movements would be useful for more completely understanding writing processes.

###  3.4 Automated Classifier Development

Given the scale of the data, we developed automated classifiers to label it
for our codes. This was simple to implement for the majority of codes as all
that was required was a string match with the corresponding event name in the
data. However, to code for revision behaviors, a more complex algorithm was
required (see below). The details of this algorithm, along with paper-related
data and analysis can be found in our Github
repository333https://github.com/yixin-cheng/coAuthor.

Sentence-level segmentation and analysis allowed us to identify and code the
high and low modifications behaviors. By using NLTK sentence tokenizer and
identifying ending markers, we collected all sentences in their initial form
including their cursor range and ownership (i.e., prompt, user, or api), into
a list. Following this, the list was used to track these sentences under the
following conditions: Sentence Removal, Sentence Merger, and Sentence
Revision. In the case of sentence removal, the corresponding sentence is
marked as ’None’ in our sentence list. For sentence revisions, the sentence
list is updated after every revision. We excluded sentence pairs with a single
word difference identified as a misspelling by the spellchecker Python package
from being counted as a revision. For sentence mergers, we used TF-IDF to
generate word embeddings and calculated the cosine similarity between the
updated sentence and the original sentence. The most similar sentence was
replaced with the merged version, with others marked as ’None’. The final
sentence list’s accuracy was corroborated by its exact match with the final
story or essay versions from the data, validating our proposed algorithm.

Next, we identified the cursor location of revisions between initial and final
sentence lists. The codes revise user and revise suggestion, were identified
based on the combination of sentence ownership and updates to the sentence
list. To code low modification and high modification, we used Sentence-BERT
[36] to compute sentence cosine similarities between initial and final
sentence pairs. We used a threshold value of 0.8 to distinguish high and low
modification [37].

To test the reliability of the high/low modification classifiers and the
validity of the codes, we randomly sampled 80 events from the data. Two human
raters then manually coded these data for the presence/absence of high/low
modifications. Next, for both codes we did pairwise comparisons between each
set of human coding and the automated classifications. The codes were
considered valid and reliable if all pairs of ratings achieved a Cohen’s kappa
greater than 0.65 and Shaffer’s rho less than 0.05. All kappa and rho values
for these two codes exceeded these thresholds. The end result of the automated
coding process was a dataset with binary values that indicated the presence or
absence of a given code for a given event.

###  3.5 Epistemic Network Analysis

To analyze the coded dataset, we used the ENA implementation for R [38]. ENA
creates a separate network representation for each unit of analysis where the
nodes of the networks are codes and edges between nodes indicate the relative
frequency of co-occurrence between those codes in a given unit’s data. To
construct these networks, we used the following ENA specifications:

  * •

Units of analysis: A separate network was created for each combination of
session ID and user ID.

  * •

Codes: All codes listed in Table 1.

  * •

Conversations: The data was grouped by session ID, user ID, and sentence for
co-occurrence accumulation. That is, codes in all events associated with a
given session, user, and sentence could co-occur.

  * •

Window Size: An "infinite" stanza window was used to identify the co-
occurrence between codes within the data for each unit of analysis. This
window begins at the first event in a given conversation and expands to
include all events within that conversation. In this way, all coded events
within the conversation co-occur, but the result is a more fine-grained
accumulation of the co-occurrence structure in the data.

ENA projects the networks into a low dimensional embedding space using
singular value decomposition to produce ENA scores for each network. This
space distinguishes networks in terms of the linear combination of co-
occurrence variables that explain the most variance in the data. To help
interpret the dimensions of this space, ENA co-registers the network graphs in
the embedding space such that the position of the nodes and the connections
they define align with the most influential co-occurrences on that dimension.
Thus, researchers can visually interpret the dimensions according to the
connections at the extremes. For more details, see [39].

###  3.6 Regression Analysis

To directly address our research questions, we conducted a regression analysis
of the ENA results that compared the ownership, genre, and temperature
conditions. Our models regressed the ENA scores for each dimension on
categorical predictors for ownership (user vs. GAI), prompt type
(argumentative vs. creative), and temperature (high vs. low). For each writing
session, these values were derived from the associated metadata in the
CoAuthor dataset.

Authors may have participated in multiple writing sessions. As a result, the
ENA scores were nested within participants. In addition, each participant may
have written multiple responses to prompts of the same kind (e.g., userID
A118B participated in four argumentative writing session and three creative
writing sessions), meaning that participants could also be nested into prompt
kinds. To accommodate this nested structure, we used _mixed-effects_
regression models [40]. To test for the effect of nesting, we calculated the
intraclass correlation coefficient (ICC) for the ENA scores grouped within
participant and prompt using the ICC package for R. Confidence intervals
around the ICC scores suggested that nesting was significant for the
participant variable, but not the prompt variable. In turn, we included the
participant identifier as a random effect (intercept) in all regression
models.

The outputs of interest were the regression coefficients of the ownership
prompt type, and temperature variables, which represent the difference between
the mean outcome scores of the two levels of each variable. We conducted the
regression modelling using the lmer and lmerTest packages for R [41, 42].
lmerTest computes significance tests for the regression coefficients using
Satterthwaite’s method [43]. Confidence intervals around the regression
coefficients were calculated via bootstrapping using the percentile method
[44]. Finally, we calculated the effect size of any significant regression
coefficients of interest using Cohen’s $d$ [45].

###  3.7 Network Comparisons

To interpret the dimensions of the ENA embedding spaces and better visualise
differences between subgroups, we examined the corresponding mean ENA
networks. That is, for given subgroup in the data, we averaged the edge
weights of their associated networks and plotted them in the embedding space.
To compare any two subgroups, we computed their network difference graphs by
subtracting their corresponding edge weights to show which co-occurrences were
more frequent in one group relative to the other.

##  4 Results

###  4.1 ENA Embedding Space

Figure 2: ENA embedding space

The resultant ENA embedding space is shown in Figure 2. The first dimension
primarily distinguishes between authors seeking suggestions from GAI
(seekSugg) on the left and revising their own writing (reviseUser) on the
right. A shift from left to right reflects the degree to which authors tended
to rely on suggestions versus composing and editing their own writing. The
second dimension primarily distinguishes between composing (compose) at the
top and behaviors related to suggestions and modifications (seekSugg, high
modification, and low modification) at the bottom. A higher position along
this dimension indicates a greater emphasis on composition, whereas a lower
position indicates a greater emphasis on modifications and suggestion use. The
dimensions of the space account for 22.4% and 11.3% of the total variation in
the data, respectively.

###  4.2 Regression Results

| X | Y
---|---|---
Intercept | $-0.03$ | $0.05$
| $(0.04)$ | $(0.03)$
prompt types_creative | $0.01$ | $\mathbf{-0.06}^{***}$
| $(0.02)$ | $(0.01)$
ownership_high | $\mathbf{0.09}^{***}$ | $-0.00$
| $(0.02)$ | $(0.02)$
temperature_high | $0.02$ | $-0.02$
| $(0.02)$ | $(0.01)$
AIC | $569.90$ | $33.03$
BIC | $601.52$ | $64.64$
Log Likelihood | $-278.95$ | $-10.51$
Num. obs. | $1435$ | $1435$
Num. groups: author_id | $60$ | $60$
Var: worker_id (Intercept) | $0.07$ | $0.02$
Var: Residual | $0.08$ | $0.05$
${}^{***}p<0.001$; ${}^{**}p<0.01$; ${}^{*}p<0.05$
Table 2: Regression models with ENA scores on either dimension as the
dependent variable. Standard errors in parentheses.

The results of the regression analysis are shown in Table 2. We report two
models that include author_id as a random effect, prompt types, ownership, and
temperature as fixed effects, and ENA scores on either the first or second
dimension of the ENA space as the outcome. Testing for interactions between
the fixed effects did not yield significant results, so we report only the
main effects here.

On the first dimension (X), only the ownership variable was significant.
Authors with more ownership over their final written product (user ownership)
were significantly higher on the dimension than those with less ownership (GAI
ownership): $t=4.35$, $p<0.001$, $d=0.24$, 95% CI $[0.05,0.13]$. Given the
interpretation of the first dimension above, this result suggests that user
ownership authors tended to focus on composing and revising their own writing
significantly more than GAI ownership authors.

On the second dimension (Y), only the prompt condition variable was
significant. When the authors responded to creative writing prompts, they were
significantly lower on the dimension compared to when they responded to
argumentative prompts: $t=-4.71$, $p<0.001$, $d=-0.22$, 95% CI
$[-0.09,-0.04]$. This result suggests that when the authors responded to
creative writing prompts, they tended to focus on editing their writing and
using GAI suggestions significantly more than when they responded to
argumentative writing prompts.

The temperature variable was not statistically significant on either dimension
suggesting that we have insufficient empirical evidence to falsify the null
hypothesis of no difference between the high and low temperature conditions.

###  4.3 User vs. GAI ownership

Figure 3 shows the network comparison for the _ownership_ variable; user
ownership on the left (a) and GAI ownership in the middle (b). Both networks
include a large number of connections to the compose node suggesting that
authors tended to link their composition process to a variety of other
behaviors. For example, both networks show relatively strong connections
between compose and seekSugg, acceptSugg, reviseUser, and cursor movements.
The network differences are shown in plot (c). Here, red edges indicate more
frequent co-occurrences in the GAI ownership group; blue edges indicate more
frequent co-occurrences in the user ownership group. The graph indicates that
the GAI ownership authors made stronger connections between compose and
seekSugg, compose and acceptSugg, and reviseSugg and lowModification. User
ownership authors, on the other hand, made stronger connections between
compose and cursor movements and compose and reviseUser. The centroids of the
networks—that is average ENA scores (blue and red squares)—corroborate the
regression analysis, with the GAI ownership authors being further to the left
on the first dimension than the user ownership authors, on average.

Figure 3: Ownership

###  4.4 Creative vs. Argumentative

Figure 4 shows the network comparisons for the prompt types variable; the mean
network for responses to creative writing prompts is on the left (a), and the
mean network for responses to argumentative writing prompts is in the middle
(b). As before, in both networks, connections to the Compose node feature
prominently. The subtracted network is shown in subplot (c). Here, blue edges
represent connections that occurred more frequently in the creative condition
and red edges represent connections that occurred more frequently in the
argumentative condition. The graph indicates that when responding to creative
writing prompts, authors tended to explore AI generated suggestions more,
having stronger connections among seekSugg, hoverSugg, and compose. When
responding to argumentative prompts, authors tended to focus more on composing
their own writing and revising AI-generated suggestions—as indicated by
stronger connections between compose and cursorSelect, compose and acceptSugg,
and reviseSugg and lowModification. The centroids of the two networks
corroborate the regression analysis, with the argumentative condition being
higher on the second dimension than the creative condition, on average.

Figure 4: Prompt type

###  4.5 High vs. Low Temperature

Figure 5 contrasts for the networks for the high (a) and low (b) temperature
conditions. The network subtraction (c) indicates that the authors in the high
temperature condition made stronger connections between compose and seekSugg,
compose and reviseUser, and compose and cursor movements. The authors in the
high temperature condition made stronger connections between compose and
acceptSugg. The centroids of the two networks overlap on both dimensions,
corroborating the regression results of no significant difference between the
two conditions.

Figure 5: Temperature

##  5 Discussion and Conclusions

In this study, we sought to demonstrate an assessment method for human-AI
collaborative writing. Framed by ECD, we used elements of knowledge-telling,
knowledge transformation, and cognitive presence to identify claims for our
student model; we used data collected from the CoAuthor writing tool as a
proxy for our task model; and we used ENA to evidence claims about the student
model using data from the task model. More specifically, we compared the co-
writing behaviors of users across three conditions: high vs. low ownership;
creative vs. argumentative prompt types; and high vs. low temperature. We
found statistically significant differences between the process of authors in
the high/low ownership conditions and the creative/argumentative conditions.
Specifically, authors with GAI ownership over their final product tended to
rely more on GAI suggestions, while those with user ownership tended to focus
more on composing and revising their own writing. When responding to creative
writing prompts, authors tended to explore GAI suggestions more, while those
responding to argumentative prompts tended focus on composing their own
writing and making small revisions to GAI writing.

In terms of more specific assessment claims, the results support Hypothesis
1—that frequent users of GAI outputs would tend more towards knowledge telling
and triggering events, as evidenced by stronger connections between compose
and seekSugg, as well as compose and acceptSugg. Hypothesis 2—that less
frequent users of GAI would tend toward knowledge transformation, exploration,
integration, and resolution was not supported. Authors with more ownership
over their written product sought and accepted some GAI suggestions but did
not tend to revise them. Instead their revisions tended to be made on their
own text with only slight modifications as evidenced by stronger connections
among compose, reviseUser, and lowModification.

The results were mixed for Hypothesis 3—that authors responding to creative
writing prompts would tend toward knowledge telling, triggering events, and
exploration. On the one hand, they did focus more on triggering events and
exploration, as evidenced by stronger connections between seekSugg and
hoverSugg, as well as compose and hoverSugg. On the other hand, evidence of a
greater focus on knowledge telling is less clear. Authors that responded to
argumentative prompts had stronger connections between acceptSugg and compose.
But they also had stronger connections between reviseSugg and lowModification,
indicating at least some level of transformation that was less prevalent for
those responding to creative writing prompts. The results were similarly mixed
for Hypothesis 4—that authors responding to argumentative writing prompts
would be characterized by knowledge transformation, integration, and
resolution. There is some evidence for this given stronger connections between
reviseSugg and lowModification, but the lack of connections to
highModificaiton and reflect limit this interpretation.

Hypothesis 5 and Hypothesis 6—that authors interacting with lower temperature
GAI would tend more toward knowledge transformation, integration, and
resolution, while authors interacting with higher temperature GAI would tend
more toward knowledge telling and exploration—were not supported. While
authors in the high temperature condition did have stronger connections to
acceptSugg and authors in low temperature condition had stronger connections
to seekSugg, their overall co-occurrence patterns were highly similar.

Our study has several limitations. First, as with any study, our results are
limited by the data at hand. In this case, our data comes from self-selected
participants who engaged with the CoAuthor platform. This potentially
introduces a selection bias, as these individuals represent a subset with
specific characteristics or preferences, which might not be fully
representative of the wider population.

Second, the tasks and environment we used are not necessarily representative
of the broader selection of GAI tools for writing. CoAuthor constrains user
interactions with GAI by only allowing them to seek suggestions that continue
the current text. Widely used tools, however, afford less restricted
interactions where users can phrase questions to the GAI any way they want, as
well as include "steering" instructions that tell the GAI how to respond in
general. Moreover, the available CoAuthor dataset contains interactions with
the now outdated GPT-3. It is possible that interactions with more
contemporary versions of GAI might yield different results.

Third, our coding scheme, and thus our proposed student model, only focused on
a narrow subset of cognitive aspects of writing related to observable
behaviors in the data. Our analysis did not consider other potentially
important features such as the semantic content of the writing and the
metacognitive strategies being used. Future studies will explore these as
possible assessment targets.

Finally, our proposed assessment method is plausible but its inherent
complexity could restrict its scalability and accessibility. We aim to address
these limitations by developing an adaptable and flexible API that includes
customizable parameters, such as event names, to meet diverse task
requirements such as post-event or real-time analysis in human-AI writing
environments. This solution seeks to bridge the gap between the current proof
of concept and its practical, large-scale application.

Despite these limitations, our work provides a proof of concept for the
evidence-centered assessment of writing composed with the help of GAI. Our
methodology posits specific features of human-AI collaborative writing to
target; adopts an existing task model to produce assessment data; and
leverages process models to relate these data to differences between the
writing processes of participants. We hope that this work will continue to
expand such that we have assessments not just for writing, but a variety of
meaningful interactions between learners and AI. Such assessments will be
crucial to preparing learners for an age in which effective human-AI
collaboration is an essential skill.

## Acknowledgments

This research was in part supported by the Australian Research Council
(DP220101209, DP240100069) and Jacobs Foundation (CELLA 2 CERES).

## Citation

Yixin Cheng, Kayley Lyons, Guanliang Chen, Dragan Gašević, and Zachari
Swiecki. 2024. Evidence-centered Assessment for Writing with Generative AI. In
The 14th Learning Analytics and Knowledge Conference (LAK ’24), March 18–22,
2024, Kyoto, Japan. ACM, New York, NY, USA, 16 pages.
https://doi.org/10.1145/3636555.3636866

## References

  * [1] Daniel Naber.  A rule-based style and grammar checker.  01 2003.
  * [2] Robert Mislevy, Russell Almond, and Janice Lukas.  A brief introduction to evidence-centered design.  US Department of Education, 06 2003.
  * [3] Andy Clark and David Chalmers.  The extended mind.  Analysis, 58(1):7–19, 1998.
  * [4] OpenAI.  Gpt-4 technical report, 2023.
  * [5] James V. Wertsch.  Mind As Action.  Oxford University Press, Incorporated, New York, UNITED STATES, 1998.
  * [6] Lodge J. M.and Howard S.and Bearman M.and Dawson P and Associates.  Assessment reform for the age of artificial intelligence.  Tertiary Education Quality and Standards Agency, 2023.
  * [7] Zixuan Ke and Vincent Ng.  Automated essay scoring: A survey of the state of the art.  In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI-19, pages 6300–6308. International Joint Conferences on Artificial Intelligence Organization, 7 2019.
  * [8] Antonette Shibani, Ming Liu, Christian Rapp, and Simon Knight.  Advances in writing analytics: Mapping the state of the field.  In Companion Proceedings of the 9th International Conference on Learning Analytics & Knowledge, LAK ’19. Association for Computing Machinery, 03 2019.
  * [9] Yongjie Wang, Chuang Wang, Ruobing Li, and Hui Lin.  On the use of bert for automated essay scoring: Joint learning of multi-scale essay representation.  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 3416–3425, Seattle, United States, July 2022. Association for Computational Linguistics.
  * [10] Sehrish Iqbal, Mladen Rakovic, Guanliang Chen, Tongguang Li, Rafael Ferreira Mello, Yizhou Fan, Giuseppe Fiorentino, Naif Radi Aljohani, and Dragan Gasevic.  Towards automated analysis of rhetorical categories in students essay writings using bloom’s taxonomy.  In LAK23: 13th International Learning Analytics and Knowledge Conference, LAK2023, page 418–429. Association for Computing Machinery, 2023.
  * [11] Dragan Gasevic, Shane Dawson, and George Siemens.  Let’s not forget: Learning analytics are about learning.  TechTrends, 59, 01 2015.
  * [12] Tongguang Li, Yizhou Fan, Yuanru Tan, Yeyu Wang, Shaveen Singh, Xinyu Li, Mladen Raković, Joep van der Graaf, Lyn Lim, Binrui Yang, Inge Molenaar, Maria Bannert, Johanna Moore, Zachari Swiecki, Yi-Shan Tsai, David Williamson Shaffer, and Dragan Gašević.  Analytics of self-regulated learning scaffolding: effects on learning processes.  Frontiers in Psychology, 14, 2023.
  * [13] Zachari Swiecki, Hassan Khosravi, Guanliang Chen, Roberto Martinez-Maldonado, Jason M. Lodge, Sandra Milligan, Neil Selwyn, and Dragan Gašević.  Assessment in the age of artificial intelligence.  Computers and Education: Artificial Intelligence, 3:100075, 2022.
  * [14] Sunder Pichai.  An important next step on our ai journey, 2023.
  * [15] Carl Bereiter and Marlene Scardamalia.  The psychology of written composition.  Psychology of Education and Instruction Series. Erlbaum, 1987.
  * [16] Mladen Raković.  Automatic Identification of Knowledge Transforming Content in Argument Essays Developed from Multiple Sources.  Phd thesis, Simon Fraser University, British Columbia, CA, sep 2019.
  * [17] D.Randy Garrison, Terry Anderson, and Walter Archer.  Critical inquiry in a text-based environment: Computer conferencing in higher education.  The Internet and Higher Education, 2(2):87–105, 1999.
  * [18] Yuanyuan Hu, Rafael Ferreira Mello, and Dragan Gašević.  Automatic analysis of cognitive presence in online discussions: An approach using deep learning and explainable artificial intelligence.  Computers and Education: Artificial Intelligence, 2:100037, 2021.
  * [19] Sehrish Iqbal, Zachari Swiecki, Srecko Joksimovic, Rafael Ferreira Mello, Naif Aljohani, Saeed Ul Hassan, and Dragan Gasevic.  Uncovering associations between cognitive presence and speech acts: A network-based approach.  In LAK22: 12th International Learning Analytics and Knowledge Conference, LAK22, page 315–325. Association for Computing Machinery, 2022.
  * [20] Joshua Cramp, John F. Medlin, Phoebe Lake, and Colin Sharp.  Lessons learned from implementing remotely invigilated online exams.  Journal of University Teaching and Learning Practice, 16(1):137–155, 2019.
  * [21] Mina Lee, Percy Liang, and Qian Yang.  Coauthor: Designing a human-ai collaborative writing dataset for exploring language model capabilities.  CoRR, abs/2201.06796, 2022.
  * [22] Andrew Gibson and Antonette Shibani.  Natural language processing - writing analytics.  In The Handbook of Learning Analytics, pages 96–104. SoLAR, 2 edition, 2022.
  * [23] H. Guo, M. Zhang, P. Deane, and R. E. Bennett.  Writing process differences in subgroups reflected in keystroke logs.  Journal of Educational and Behavioral Statistics, 44(5):571–596, 2019.
  * [24] Mo Zhang, Hongwen Guo, and Xiang Liu.  Using keystroke analytics to understand cognitive processes during writing.  International Educational Data Mining Society, 2021.
  * [25] Mladen Raković, Sehrish Iqbal, Tongguang Li, Yizhou Fan, Shaveen Singh, Surya Surendrannair, Jonathan Kilgour, Joep van der Graaf, Lyn Lim, Inge Molenaar, Maria Bannert, Johanna Moore, and Dragan Gašević.  Harnessing the potential of trace data and linguistic analysis to predict learner performance in a multi-text writing task.  Journal of Computer Assisted Learning, 39(3):703–718, 2023.
  * [26] Nikolaos Limnios and Ghorghe Oprisan.  Semi-Markov Processes and Reliability.  01 2001.
  * [27] Mariëlle Leijten and Luuk Van Waes.  Keystroke logging in writing research.  Written Communication, 30(3):358–392, 2013.
  * [28] Vilaythong Southavilay, Kalina Yacef, Peter Reimann, and Rafael A. Calvo.  Analysis of collaborative writing processes using revision maps and probabilistic topic models.  In Proceedings of 3rd International Conference on Learning Analytics and Knowledge, LAK ’13, page 38–47. Association for Computing Machinery, 2013.
  * [29] Antonette Shibani, Simon Knight, and Simon Buckingham Shum.  Understanding revisions in student writing through revision graphs.  In Artificial Intelligence in Education, pages 332–336, Cham, 2018. Springer International Publishing.
  * [30] Antonette Shibani, Ratnavel Rajalakshmi, Faerie Mattins, Srivarshan Selvaraj, and Simon Knight.  Visual representation of co-authorship with GPT-3: Studying human-machine interaction for effective writing.  In Proceedings of the 16th International Conference on Educational Data Mining, pages 183–193. International Educational Data Mining Society, July 2023.
  * [31] David Shaffer and Andrew Ruis.  Epistemic Network Analysis: A Worked Example of Theory-Based Learning Analytics, pages 175–187.  01 2017.
  * [32] Yizhou Fan, Mladen Rakovic, Joep van der Graaf, Lyn Lim, Shaveen Singh, Johanna Moore, Inge Molenaar, Maria Bannert, and Dragan Gašević.  Towards a fuller picture: Triangulation and integration of the measurement of self-regulated learning based on trace and think aloud data.  Journal of Computer Assisted Learning, 39(4):1303–1324, 2023.
  * [33] Daniel Chandler.  An introduction to genre theory.  1997\.
  * [34] Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung.  Survey of hallucination in natural language generation.  ACM Computing Surveys, 55(12):1–38, mar 2023.
  * [35] David Williamson Shaffer and A. R. Ruis.  How we code.  In Advances in Quantitative Ethnography, pages 62–77. Springer International Publishing, 2021.
  * [36] Nils Reimers and Iryna Gurevych.  Sentence-bert: Sentence embeddings using siamese bert-networks.  CoRR, abs/1908.10084, 2019.
  * [37] Peiying Zhang, Xingzhe Huang, and Lei Zhang.  Information mining and similarity computation for semi- / un-structured sentences from the social data.  Digital Communications and Networks, 7, 08 2020.
  * [38] Marquart, C. L., Swiecki, Z., Collier, W., Eagan, B., Woodward, R., Shaffer, and D. W.  rENA: Epistemic Network Analysis, 2022.
  * [39] Dale Bowman, Zachari Swiecki, Zhiqiang Cai, Yeyu Wang, Brendan Eagan, Jeff Linderoth, and David Williamson Shaffer.  The mathematical foundations of epistemic network analysis.  In Advances in Quantitative Ethnography, Communications in Computer and Information Science, pages 91–105. Springer, 2021.
  * [40] Alain Zuur, EN Ieno, Neil Walker, Anatoly Saveliev, and GM Smith.  Mixed Effects Models and Extensions in Ecology With R, volume 1-574.  01 2009.
  * [41] Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker.  Fitting linear mixed-effects models using lme4.  Journal of Statistical Software, 67(1):1–48, 2015.
  * [42] Alexandra Kuznetsova, Per B. Brockhoff, and Rune H. B. Christensen.  lmerTest package: Tests in linear mixed effects models.  Journal of Statistical Software, 82(13):1–26, 2017.
  * [43] F. E. Satterthwaite.  An approximate distribution of estimates of variance components.  Biometrics Bulletin, 2(6):110–114, December 1946.
  * [44] Rob Hyndman and Yanan Fan.  Sample quantiles in statistical packages.  The American Statistician, 50:361–365, 11 1996.
  * [45] J. Cohen.  Statistical Power Analysis for the Behavioral Sciences.  Lawrence Erlbaum Associates, 1988.