# Coulombic surface-ion interactions induce non-linear and chemistry-specific
charging kinetics

W.Q. Boon  Institute for Theoretical Physics, Utrecht University,
Princetonplein 5, 3584 CC Utrecht, The Netherlands  M. Dijkstra  Soft
Condensed Matter, Debye Institute for Nanomaterials Science, Utrecht
University, Princetonplein 1, 3584 CC Utrecht, The Netherlands  R. van Roij
Institute for Theoretical Physics, Utrecht University, Princetonplein 5, 3584
CC Utrecht, The Netherlands

###### Abstract

While important for many industrial applications, chemical reactions
responsible for charging of solids in water are often poorly understood. We
theoretically investigate the charging kinetics of solid-liquid interfaces,
and find that the time-dependent equilibration of surface charge contains key
information not only on the reaction mechanism, but also on the valency of the
reacting ions. We construct a non-linear differential equation describing
surface charging by combining chemical Langmuir kinetics and electrostatic
Poisson-Boltzmann theory. Our results reveal a clear distinction between late-
time (near-equilibrium) and short-time (far-from-equilibrium) relaxation
rates, the ratio of which contains information on the charge valency and ad-
or desorption mechanism of the charging process. Similarly, we find that
single-ion reactions can be distinguished from two-ion reactions as the latter
show an inflection point during equilibration. Interestingly, such inflection
points are characteristic of autocatalytic reactions, and we conclude that the
Coulombic ion-surface interaction is an autocatalytic feedback mechanism.

Charged solid-liquid interfaces play a central role in a wide variety of
industries such as food and coating production [1, 2, 3], mining [4, 5, 6],
medicine [7, 8, 9], soil remediation [10, 11, 12] and even carbon capture
[13]. With the advent of nanoscale fluidics one expects that charged surfaces
become ever more important [14, 15]. In water and other polar solvents
chemical reactions are a common mechanism by which surfaces obtain their
charge. For ionic solids the de- or adsorption of a dissolved ionic compound
is often preferred over the sorption of its own counterion [16, 17, 18]; for
covalent solids such as polymers and metal oxides the acidic nature of surface
groups ensures that the surface (de)protonates in polar solvents and hence
becomes charged [19, 20, 21, 17, 18]. However, for many processes of
industrial and environmental importance relatively little is known about the
surface chemistry [19, 17, 18] as the electrolytes in realistic applications
contain a large variety of ions that can all undergo multiple reactions [17,
18, 22]. Due to experimental limitations the majority of studies investigating
surface charging are performed at (quasi)-equilibrium conditions [17, 18],
with the notable exception of pressure-jump experiments [23, 24]. Only
recently, however, it has been shown that the kinetics of chemical surface
reactions can strongly couple to electrokinetic fluid flows, thereby affecting
the physical surface properties on macroscopic scales [25, 26, 27, 28, 29,
18]. Furthermore, with the recent advent of fast and surface specific non-
linear spectroscopy the dynamic measurement of surface charge has become
feasible [30, 31, 32, 33, 34, 29]. In this context it has been explicitly
stated that there is an urgent need for theoretical models to describe such
experiments [35]. Traditionally, sorption kinetics is typically described by
(pseudo)-first-order reactions [12, 36, 37] that exhibit single-exponential
relaxation towards equilibrium; the influence of a time-dependent surface
charge is usually neglected entirely [38, 39, 35]. We are aware of one
theoretical work [40] and associated review [41] that considers a surface
charge that affects the rate constants of ion-association, which, however,
does not consider the (chemistry-specific) non-linear dynamics induced by the
electrostatic feedback as we do here.

In this Letter we present a theory for the charging dynamics of solid
surfaces. We include the Coulombic ion-surface interactions and reveal an
intricate dependence on the reaction mechanism and the valency of the reactive
ions already present in a mean-field description. The Coulomb interactions not
only affect the time constant of the late-time exponential decay of the
surface charge towards equilibrium after an ion concentration (or pH) shock,
but they also induce strongly nonlinear dynamics at early times far from
equilibrium. Combined with the present-day capability to experimentally
measure the time-dependent surface charge density, our theory forms a first
step to unveil the surface chemistry of technologically important but ill-
understood materials [35, 18], such as silica [42, 22] and graphene [43], and
of processes such as the clean-up of radioactive and heavy metals [10, 16, 44,
45].

Surfaces, for instance silica, in water commonly charge either by desorption
of ionic species from neutral surface groups or by adsorption of ionic species
onto neutral surfaces. While the exact charging mechanism of the silica-water
interface is complex, there is support for charging by desorption of protons
at high pH and adsorption of protons at low pH [42, 46, 47, 22],

$\displaystyle\mathrm{SiOH}_{\rm{(s)}}$
$\displaystyle\xrightleftharpoons[k_{\rm{a}}\rho]{k_{\rm{d\
}}}\mathrm{SiO}^{-}_{\rm{(s)}}+\mathrm{H}^{+}_{\rm{(aq)}},$ (1a)
$\displaystyle\mathrm{SiOH}_{\rm{(s)}}+\mathrm{H}^{+}_{\rm{(aq)}}$
$\displaystyle\xrightleftharpoons[k_{\rm{d\
}}]{k_{\rm{a}}\rho}\mathrm{SiOH_{2}}^{+}_{\rm{(s)}},$ (1b)

where $\mathrm{SiOH}_{\rm{(s)}}$ is a neutral silanol group that is covalently
bound to the (solid) glass and where $\mathrm{SiO}^{-}_{\rm{(s)}}$ and
$\mathrm{SiOH_{2}}^{+}_{\rm{(s)}}$ denote a silanol group with a proton
desorbed or adsorbed in Eqs. (1a) and (1b), respectively. Here $\rho$ denotes
the proton density at the solid surface, and the dissociation and association
rate $k_{\rm{d}}$ and $k_{\rm{a}}$ will be discussed below for the charging
kinetics of a single desorptive and a single adsorptive reaction, not only for
monovalent reactive ions as in Eqs. (1a) and (1b) but for general valency $z$.
While adsorption isotherms of real materials can rarely be described by just a
single charging reaction [47, 16], we show in Supplemental Material I (SM I
[48]) that charging by multiple reactions can actually be well-approximated by
the single-reaction kinetics presented in this Letter for a wide range of
experimental conditions.

We consider a macroscopic surface with a density $\Gamma$ of identical surface
groups. A group can only be in either a neutral or a charged state. The
charging is assumed to take place either by desorption (labeled by $-$) of a
cation of charge $ze$, or by adsorption (labeled by $+$) of a cation of charge
$ze$, with $z\geq 0$ and $e$ the proton charge. The surface densities of
charged and neutral groups are denoted by $\sigma_{\pm}>0$ and
$\Gamma-\sigma_{\pm}>0$ respectively, and the surface charge density is given
by $\pm ze\sigma_{\pm}$. Note that the charging dynamics is invariant under
the sign of the reacting ions, and without loss of generality we can restrict
attention to reactive cations of (strictly positive) valency $z$. Assuming the
chargeable surface sites to be independent, we can describe the reaction
kinetics in terms of the time-dependent surface density $\sigma_{\pm}(t)>0$
which satisfies Langmuir kinetics described by [49, 50]

$\partial_{t}\sigma_{-}=k_{\rm{d}}(\Gamma-\sigma_{-})-k_{\rm{a}}\sigma_{-}\rho(\sigma_{-})\\\
$ (1)

for desorptive charging reaction (1a), and

$\partial_{t}\sigma_{+}=k_{\rm{a}}(\Gamma-\sigma_{+})\rho(\sigma_{+})-k_{\rm{d}}\sigma_{+}$
(2)

for adsorptive charging reaction (1b). Here $k_{\mathrm{d}}$ and
$k_{\mathrm{a}}$ are the rate constants of the dissociation and association of
the reactive ion and $\rho(\sigma_{\pm})$ is the volumetric concentration of
reactive ions at the surface, which is defined at the position where the rate-
limiting step for the reaction occurs [51, 50]. We consider this surface to be
impermeable to non-reacting ions and therefore do not account for any Stern
layer other than the charged surface groups [27]. The equilibrium surface
charge follows from $\partial_{t}\sigma_{\pm}=0$ and is given by
$\sigma_{\pm,\rm{eq}}=\Gamma(1+(k_{\rm{a}}\rho_{\rm{eq}}/k_{\rm{d}})^{\mp
1})^{-1}$, which reduces to an explicit “Langmuir isotherm” in the case that
the equilibrium concentration of the reactive ions
$\rho_{\rm{eq}}\equiv\rho(\sigma_{\pm,\rm{eq}})$ is a constant independent of
$\sigma_{\pm,\rm{eq}}$. In general, however, this Langmuir isotherm is a self-
consistency equation for $\sigma_{\pm,\rm{eq}}$ that requires an additional
“closure” relation $\rho(\sigma_{\pm})$ for an explicit equilibrium solution
$\sigma_{\pm,\rm{eq}}$. Without (Coulombic) interactions between surface and
ions, the local concentration $\rho(\sigma_{\pm})$ of reactive species in the
vicinity of the surface would be equal to the bulk concentration
$\rho_{\rm{b}}$ of the reactive ions far from the surface (which is
independent of $\sigma_{\pm}$ and hence also independent from the reaction
mechanism), such that Eqs. (1)-(2) would be linear differential equations
whose solution can be written as
$s_{\pm}(t)=1+(s_{\pm}(0)-1)\exp[-(k_{\rm{d}}+k_{\rm{a}}\rho_{\rm{b}})t]$ with
the dimensionless charge $s_{\pm}=\sigma_{\pm}/\sigma_{\pm,\rm{eq}}$ such that
$s_{\pm,\rm{eq}}=1$; here $s_{\pm}(0)-1$ is the integration constant and
denotes the relative deviation from equilibrium at the initial time $t=0$.
Note that the condition that $0\leq\sigma_{\pm}(t)\leq\Gamma$ implies that
$0\textless s_{\pm}(0)\textless\Gamma/\sigma_{\pm,\rm{eq}}$, where the lower
bound corresponds to an initially neutral surface whereas the upper bound can
be as large as $\mathcal{O}(10-100)$, since typical equilibrium conditions
have a charge occupancy of only a few percent of the total number of
chargeable groups [52]. Thus from measurements of $\sigma_{\pm}(t)$ at various
concentrations of reactive (dissolved) species both $k_{\mathrm{d}}$ and
$k_{\mathrm{a}}$ could in this non-interacting case be determined.

However, as the charged surface attracts or repels reactive ions, Eqs. (1) and
(2) are complicated by a nontrivial relation $\rho(\sigma_{\pm})$, which
causes a charge-dependent decay rate and introduces deviations from purely
single-exponential relaxation of $\sigma_{\pm}(t)$. In fact, an explicit
function $\rho(\sigma_{\pm})$ is needed to investigate and solve the dynamics,
which we will develop here.

Figure 1: Time-dependent relative deviations $s_{\pm}(t)-1$ from the
equilibrium charge density as follows from the kinetic Langmuir-Gouy-Chapman
equations (1)-(3) in (a) and Eqs. (2)-(3) in (b) and (c), for equilibrium zeta
potentials $(k_{\rm{B}}T/e)|\phi_{\rm{eq}}|$ equal to 50 mV (cross) and 100 mV
(circle) for valencies $z=0,1,2,3$ (colors), in (a) $s_{-}(t)$ for desorptive
reactions when $\sigma_{-,\rm{eq}}\ll\Gamma$, in (b) $s_{+}(t)$ for adsorptive
reactions when $\sigma_{+,\rm{eq}}\ll\Gamma$, and in (c) $s_{+}(t)$ for
adsorptive reactions when $\sigma_{+,\rm{eq}}\simeq\Gamma$. Insets denote
semi-logarithmic representations of $|s_{\pm}(t)-1|$. The case
$\sigma_{-,\rm{eq}}\simeq\Gamma$ (not shown) is trivial with single-
exponential decay for all $z$.

We consider the planar and homogeneous chargeable solid surface discussed
above in contact with a bulk solvent with permittivity $\epsilon$ and
temperature $T$ with a three-component $1:1:z$ electrolyte of bulk
concentrations $\rho_{\rm{s}}:(\rho_{\rm{s}}-z\rho_{\rm{b}}):\rho_{\rm{b}}$.
For convenience we assume trace amounts of reactive ions and therefore set
$\rho_{\rm{b}}\ll\rho_{\rm{s}}$, where $\rho_{\rm{s}}$ is the bulk salt
concentration. We also assume the electrolyte volume to be macroscopically
large such that $\rho_{\rm{b}}$ and $\rho_{\rm{s}}$ do not change due to
surface charging. Furthermore, we assume the charging timescale $\tau_{\pm}$,
which remains to be derived, to be the slowest timescale of the system. Given
that the typical timescale for electric double layer (EDL) equilibration is
around $10^{-9}-10^{-6}$ s and that the (geometry and flow dependent)
transport timescale for ions in stirred reactors can be as short as $10^{-4}$
s [53], we find a large window $\tau_{\pm}\gg 10^{-4}$ s for reactions to be
well-described by our (reaction-limited) theory [54]: for example phosphate
desorption shows characteristic reaction timescales of hours [55] and
adsorption of transition metals can occur on millisecond timescales [23, 24].
The slow-reaction assumption allows us to describe the EDL within an
equilibrium theory, for which we take the Gouy-Chapman solution of Poisson-
Boltzmann (PB) theory for simplicity [20, 56]. Although PB theory is based on
a mean-field assumption for a system of point ions, it is known that for all
but the highest salt concentrations this theory is quite accurate for $1:1$
and even $1:2$ aqueous electrolytes [57], and we expect a similar accuracy for
$1:1:z$ electrolytes in the limit $\rho_{\rm{b}}\ll\rho_{\rm{s}}$ of our
interest. Within these assumptions the concentration of reactive ions at the
surface is determined by a Boltzmann distribution
$\rho(\sigma_{\pm})=\rho_{\rm{b}}\exp[-z\phi(\sigma_{\pm})]$, where
$k_{\mathrm{B}}T\phi(\sigma_{\pm})/e$ is the electric potential at the surface
with a surface charge $\pm ze\sigma_{\pm}$, with $k_{\rm{B}}$ the Boltzmann
constant. For desorptive charging the surface and ions have opposite charge
and hence $z\phi(\sigma_{-})<0$, while for adsorptive charging ions and
surface have the same sign yielding $z\phi(\sigma_{+})>0$. With this
observation the Gouy-Chapman solution for a $1:1$ electrolyte, which is
relevant here as $\rho_{\rm{s}}\gg\rho_{\rm{b}}$, gives
$\phi(\sigma_{\pm})=\pm 2\sinh^{-1}(z\sigma_{\pm}/\sigma^{*})$[20, 56], where
$\sigma^{*}=(2\pi\lambda_{\mathrm{B}}\lambda_{\mathrm{D}})^{-1}$ with
$\lambda_{\mathrm{B}}=e^{2}(4\pi\epsilon k_{\mathrm{B}}T)^{-1}$ the Bjerrum
length of the solvent and
$\lambda_{\mathrm{D}}=(8\pi\lambda_{\mathrm{B}}\rho_{\rm{s}})^{-\frac{1}{2}}$
the Debye screening length. Substituting the Gouy-Chapman potential in the
Boltzmann factor yields

$\rho(\sigma_{\pm})=\rho_{\rm{b}}\exp[-z\phi(\sigma_{\pm})]=\rho_{\rm{b}}\bigg{(}\frac{z\sigma_{\pm}}{\sigma^{*}}+\sqrt{1+\big{(}\frac{z\sigma_{\pm}}{\sigma^{*}}\big{)}^{2}}\bigg{)}^{\mp
2z},$ (3)

where the exponent is positive for desorptive charging and negative for
adsorptive charging. Because Eq. (3) is reaction-mechanism dependent, explicit
information on the charging mechanism can be deduced from the reaction
kinetics as described by combining Eq. (3) with Eqs. (1)-(2).

In order to investigate the influence of the Coulombic ion-surface
interactions on the charging dynamics, we numerically solve $\sigma_{-}(t)$
from the kinetic Langmuir-Gouy-Chapman Eqs. (1)-(3). The symbols in Fig.1(a)
present the resulting relative deviations from equilibrium, $s_{-}(t)-1$, for
a desorptive reaction in the experimentally common case of low equilibrium
saturation $\sigma_{-,\rm{eq}}\ll\Gamma$, both for $s_{-}(t=0)=2$ and $0.01$
corresponding to a surface with double the charge compared to equilibrium and
an initially almost uncharged surface, respectively, for equilibrium surface
potentials of 50 mV ($|\phi_{\rm{eq}}|=2$, circles) and 100 mV
($|\phi_{\rm{eq}}|=4$, crosses) and for valencies $z=0,1,2,3$ indicated by the
different colours. Fig.1(a) shows that a desorptive surface that is
overcharged ($s_{-}>1$) decays to equilibrium faster than one that is
undercharged ($s_{-}<1$), the more so for larger valencies $z$. Interestingly,
the sorption of uncharged species ($z=0$, black symbols) reveals perfect
symmetry between the two cases as expected for first order kinetics, which is
also manifest in the semi-logarithmic representation of $|s_{-}(t)-1|$ in the
inset of Fig.1(a) that shows a data collapse and a single-exponential decay
for $z=0$. For $z\geq 1$ the inset reveals a non-exponential time dependence
with an initially slower decay for undercharged surfaces and an initially
faster decay for overcharged surfaces, the difference becoming more pronounced
for higher valencies. Fig. 1(b) shows the deviation $s_{+}(t)-1$ from
equilibrium for numerical solutions of Eqs. (2)-(3) for an adsorptive charging
reaction and the same low equilibrium surface density
$\sigma_{+,\rm{eq}}\ll\Gamma$ and the same surface potentials and valencies as
in (a). Interestingly, for this reaction the relaxation of an initially
undercharged surface to equilibrium is faster, rather than slower as we found
for desorptive undercharged surfaces in (a). Hence the two mechanisms can be
distinguished by inspecting a single time-trace of the surface charge. We do
not plot the dynamics of a desorptive surface that is saturated in equilibrium
$\sigma_{-,\rm{eq}}\simeq\Gamma$ (in which case $k_{\rm{d}}\gg
k_{\rm{a}}\rho_{\rm{eq}}$) as the equilibration (dissociation) rate for such a
surface $\partial_{t}s\simeq-k_{\rm{d}}(s-1)$ is linear and equilibration
occurs through trivial single-exponential decay. The lack of non-linearity for
such a surface stems from the fact that the dissociation process is unaffected
by the electrostatic surface-ion interaction. However, as can be seen in
Fig.1(c) the dynamics of an adsorptively charged surface with a saturated
charge density $\sigma_{+,\rm{eq}}\simeq\Gamma$ is markedly non-linear. As was
the case in Fig.1(b) we see that an undercharged surface equilibrates faster
than a single-exponential. Clearly, these rather distinctive features of the
time-dependent surface charge contain explicit information on not only the
reaction mechanism but also the valency of reacting ions. Interestingly, such
deviations from single-exponential decay have historically been observed in
pressure-jump experiments [23, 24]. In SM II [48] we show that these
experiments are well described by our theory, alleviating the need of
introducing multiple reactions to describe such experiments.

In Figs.1(a)-(b) the dimensionless time on the horizontal axes contains a
factor $(2z+1)$, which as we will show now, is convenient as it leads to a
data collapse in the asymptotic nonlinear-screening regime
$|\phi_{\rm{eq}}|\gg 1$ where $s_{\pm}(t)$ only depends on the valency, the
reaction mechanism, and the initial charge state. To see why the near-
equilibrium decay rate includes a factor $(2z+1)$ in Figs.1(a) and (b) but not
(c) we simplify the Langmuir-Gouy-Chapman Eqs. (1)-(2) and (3) in the
important and common case of large equilibrium surface potentials where
$z\sigma/\sigma^{*}\textgreater 1$, say beyond 50 mV where
$|\phi_{\rm{eq}}|\geq 2$. In this limit Eqs. (1)-(2) can be rewritten as a
single polynomial (Chini [58]) differential equation

$\displaystyle-\partial_{t}s_{\pm}=k_{\rm{a}}\rho_{\rm{eq}}\bigg{(}s_{\pm}^{1\mp
2z}-s_{\pm}^{-z\mp z}\bigg{)}+k_{\rm{d}}\bigg{(}s_{\pm}-s_{\pm}^{-z\mp
z}\bigg{)},$ (4)

for which a closed form solution can be obtained by separation of variables
only for an adsorptively charged surface with $k_{\rm{d}}\gg
k_{\rm{a}}\rho_{\rm{eq}}$ in which case
$s^{2z+1}_{+}(t)=1+(s_{+}^{2z+1}(0)-1)\exp[-(2z+1)k_{\rm{d}}t]$. Near-
equilibrium, $s_{\pm}\simeq 1$, Eq. (4) simplifies to the linear differential
equation $\partial_{t}s_{\pm}\simeq-(s_{\pm}-1)/\tau_{\pm}$ with the near-
equilibrium decay rate for desorptive and adsorptive charging given,
respectively, by

$\displaystyle\tau_{-}^{-1}$
$\displaystyle=(2z+1)k_{\rm{a}}\rho_{\rm{eq}}+k_{\rm{d}},$ (5a)
$\displaystyle\tau_{+}^{-1}$
$\displaystyle=(2z+1)k_{\rm{d}}+k_{\rm{a}}\rho_{\rm{eq}}.$ (5b)

As announced, this timescale $\tau_{\pm}$ shows that electrostatic attraction
can alter the linear, near-equilibrium, decay rate by a factor $(2z+1)$ for
$z\geq 1$ compared to the neutral case ($z=0$) in the experimentally common
regime $(k_{\rm{a}}\rho_{\rm{eq}}/k_{\rm{d}})^{\mp 1}\gg 1$ where
$\sigma_{\pm,\rm{eq}}\ll\Gamma$. As for the majority of surfaces the
equilibrium charge is much lower than saturation, $\sigma_{\rm{eq}}\ll\Gamma$
[52], we expect the correction by a factor $(2z+1)$ to be common and we note
in passing that the only other work focusing on the influence of Coulombic
ion-surface interactions on kinetics [40, 41] does not mention this factor. As
already observed in Fig.1, our simple Eq. (4) shows that far-from-equilibrium
the dynamics becomes non-linear and importantly the $\pm$ sign of the reaction
mechanism breaks the near-equilibrium symmetry of dynamics with regard to a
charge excess or a charge deficit. As can be seen from Eq. (3), desorptively
charged surfaces which are overcharged, $s_{-}>1$, will initially attract an
excess of reactive ions to the surface, thereby having an increased reaction
rate compared to uncharged equilibration. Hence the equilibration for large
overcharging is faster than expected from uncharged Langmuir kinetics.
Conversely, an initially undercharged surface, $s_{-}<1$, will have a shortage
of reactive ions and thus equilibration will be slower. For adsorptively
charged surfaces equilibration will be non-linear regardless of
$\sigma_{\rm{eq}}/\Gamma$ and here undercharging leads to a shortage of
reactive ions compared to equilibrium and hence faster equilibration, as can
be seen in Fig.1(b). The rate changing during equilibration is reminiscent of
autocatalytic reactions where the equilibration rate changes because a
catalyst speeding up the reaction is produced simultaneously with a reaction
product [59, 60, 61], and in SM III [48] we demonstrate the similarity between
Eq. (4) and autocatalytic kinetics. A characteristic feature of such
autocatalytic reactions is an increasing decay rate up to a maximum and a
corresponding inflection point in the time-dependent decay.

Figure 2: Time dependence of the relative deviation from the equilibrium
charge density $s(t)-1$ for the autocatalytic ion displacement reaction Eq.
(7a) and Eq. (7b) for 10 different initial conditions $s(0)$ in the
experimentally common regime $\sigma_{\rm{eq}}\ll\Gamma$. Symbols are
numerical solutions for the full Langmuir-Gouy-Chapman equation (SM IV [48])
for $\phi_{\rm{eq}}=2$ (open symbols) and $\phi_{\rm{eq}}=4$ (crosses), for 5
initial conditions. Gray dashed line denotes the inflection point (top
$s(t)-1\simeq 0.6$, bottom $s(t)-1\simeq-0.25$).

Interestingly, for so-called ion displacement reactions in which ions are
involved in both the forward- and the back-reaction, inflection points are
easily realisable as there are now two ions attracted or repelled from the
charged surface such that the Coulombic feedback is strengthened: inflection
points are hence a smoking gun that multiple ions are involved in a charging
reaction. An example of a two-ion reaction where all reacting ions are
repelled from the charged surface is the calcium charging of silica [42] of
Eq. (6a), while an example of a reaction where all the reacting ions are
attracted to the charged surface is the fluoride charging of the biomineral
carbonato-apatite [62] of Eq. (6b),

$\displaystyle\mathrm{SiOH}_{\rm{(s)}}+\mathrm{Ca}^{2+}_{\rm{(aq)}}$
$\displaystyle\xrightleftharpoons[k_{\rm{b}}\rho_{\rm{H}}]{k_{\rm{f}}\rho_{\rm{Ca}}}\mathrm{SiOCa}^{+}_{\rm{(s)}}+\mathrm{H}^{+}_{\rm{(aq)}},$
(6a)
$\displaystyle\mathrm{X}\mathrm{CO_{3}}_{\rm{(s)}}+\mathrm{F}^{-}_{\rm{(aq)}}$
$\displaystyle\xrightleftharpoons[k_{\rm{b}}\rho_{\rm{CO}}]{k_{\rm{f}}\rho_{\rm{F}}}\mathrm{XF}^{+}_{\rm{(s)}}+\mathrm{CO_{3}}^{2-}_{\rm{(aq)}},$
(6b)

where $\mathrm{X}=\mathrm{Ca_{10}(PO_{4})_{6}}$, $k_{\rm{f}}$ is the forward
(charging) reaction rate, $k_{\rm{b}}$ is the backward (decharging) rate and
$\rho_{i}$ is the concentration of ion species $i$ at the charged surface. In
SM III [48] we derive under the same Gouy-Chapman and large surface potential
conditions of the main text that the charge equilibration for Eq. (6a) and Eq.
(6b) are respectively described by

$\displaystyle-\partial_{t}s$
$\displaystyle=k_{\rm{f}}\rho_{\mathrm{Ca,eq}}(s^{-3}-s^{-4})+k_{\rm{b}}\rho_{\rm{H,eq}}(s^{-1}-s^{-4}),$
(7a) $\displaystyle-\partial_{t}s$
$\displaystyle=k_{\rm{f}}\rho_{\mathrm{F,eq}}(s^{3}-s^{2})\ \ \ \ \ \
+k_{\rm{b}}\rho_{\rm{CO,eq}}(s^{5}-s^{2}),$ (7b)

with the resulting near-equilibrium decay constant
$\tau=k_{\rm{f}}\rho_{i,\rm{eq}}+3k_{\rm{b}}\rho_{j,\rm{eq}}$ for both
reactions. Comparing Eqs. (7a)-(7b) to Eq. (4) we see that now the time
evolution is given by the difference of two polynomials of (non-zero) unequal
degree, ensuring that there is always a maximum in the decay rate and hence an
inflection point. We find that reactions of the form (6b) have two physically
realizable inflection points located at $s=2/3$ (if
$\sigma_{\rm{eq}}\simeq\Gamma$) and $s=(2/5)^{1/3}\simeq 0.75$ (if
$\sigma_{\rm{eq}}\ll\Gamma$): the reaction (6a) has only one accessible
inflection point $s=4^{1/3}\simeq 1.6$ (if $\sigma_{\rm{eq}}\ll\Gamma$), while
its second inflection point $s=4/3$ is inaccessible for a saturated surface
with $\sigma_{\rm{eq}}\simeq\Gamma$.

We plot the dynamics resulting from the Eqs. (7a)-(7b) for a variety of
starting conditions $s(0)$ in Fig.2 in the experimentally common limit
$\sigma_{\rm{eq}}\ll\Gamma$. In Fig.2(a) excellent agreement between the
asymptotic Eq. (7a) and full numerical results can be seen, while in Fig.2(b)
for large undercharging $s(0)\textless 0.2$ a discrepancy between Eq. (7b) and
the full numeric solution is found. However in general Eqs. (7a)-(7b) predict
the location of the inflection point accurately for a range of common surface
potentials $(k_{\rm{B}}T/e)|\phi_{0}|\in[50,100]$ mV. For ion-displacement
reactions of the form (6a) and (6b) involving ions with higher valencies but
monovalent surface charge the inflection point will lie even closer to
equilibrium. Thus surfaces that are initially undercharged by only $\simeq
25\%$ or overcharged by $\simeq 60\%$ will generally equilibrate along
sigmoidal curves, which is a distinguishing feature that cannot be observed
for the single-ion reactions Eqs. (1a)-(1b). Finally we note that ion-
displacement reactions offer a simple explanation for the recently observed
sigmoidal equilibration of the surface charge at an aqueous silica interface
[61] using only a single charging-reaction of the form Eq. (6a) rather than
the proposed autocatalytic dissolution cycle involving multiple steps.

To summarize, in this Letter we present a model for the non-linear dynamics of
reaction-limited surface charging, combining Langmuir dynamics with Poisson-
Boltzmann theory. The model captures how the screened electrostatic surface-
ion interaction affects the reaction rate near and far from equilibrium in
terms of a non-linear differential equation, where the electrostatic
interaction is described by only using the charge valency of the reactive ion.
The Coulombic ion-surface interaction leads to a charge-dependent decay rate,
which can be used to gain information on the valency of reacting ions, initial
charge, and reaction rate. De- and adsorptive reactions can be distinguished
by inspecting whether far-from-equilibrium decay is slower or faster than
near-equilibrium decay, while an inflection point is characteristic for two-
ion reactions. Interestingly, we note that inflection points are a
characteristic feature of autocatalytic reactions and that the electrostatic
ion-surface interaction can be seen as a catalytic feedback loop. Hence
electrostatics offers a straightforward explanation for the recently measured
autocatalytic charging of silica [61].

###### Acknowledgements.

We thank Ben Werkhoven and Cheng Lian for fruitful discussion, and an
anonymous referee for pointing us to the pressure-jump experiments Ref. [23,
24]. This work is part of the D-ITP consortium, a program of the Netherlands
Organisation for Scientific Research (NWO) that is funded by the Dutch
Ministry of Education, Culture and Science (OCW).

## References

  * Nair  _et al._ [2020] M. S. Nair, M. Tomar, S. Punia, W. Kukula-Koch, and M. Kumar, International Journal of Biological Macromolecules (2020).
  * Croll [2002] S. Croll, Progress in Organic Coatings 44, 131 (2002).
  * Karlsson  _et al._ [2018] M. C. Karlsson, Z. Abbas, R. Bordes, Y. Cao, A. Larsson, A. Rolland, P. Taylor, and B.-M. Steenari, Progress in Organic Coatings 125, 279 (2018).
  * [4] C. D. Anderson, P. R. Taylor, and C. G. Anderson, in  _Inter. Miner. Processing Congress Proceedings_ , pp. 1–15.
  * Fuerstenau  _et al._ [1965] M. Fuerstenau, D. Rice, P. Somasundaran, and D. Fuerstenau, Inst. Min. Metall. Trans 74, 381 (1965).
  * Rao [2013] S. R. Rao,  _Surface chemistry of froth flotation: Volume 1: Fundamentals_ (Springer Science & Business Media, 2013).
  * Zheng  _et al._ [2017] W. Zheng, Y. Jia, W. Chen, G. Wang, X. Guo, and X. Jiang, ACS Applied Materials & Interfaces 9, 21181 (2017).
  * Dhall  _et al._ [2021] A. Dhall, S. Islam, M. Park, Y. Zhang, A. Kim, and G. Hwang, ACS Applied Materials & Interfaces 13, 40379 (2021).
  * Leonenko  _et al._ [2006] Z. Leonenko, M. Rodenstein, J. Döhner, L. M. Eng, and M. Amrein, Langmuir 22, 10135 (2006).
  * Brown  _et al._ [1999] J. Brown, G. E., A. L. Foster, and J. D. Ostergren, Proc Natl Acad Sci U S A 96, 3388 (1999).
  * Kleber  _et al._ [2021] M. Kleber, I. C. Bourg, E. K. Coward, C. M. Hansel, S. C. Myneni, and N. Nunan, Nature Reviews Earth & Environment 2, 402 (2021).
  * Ho  _et al._ [2000] Y. S. Ho, J. Ng, and G. McKay, Separation and purification methods 29, 189 (2000).
  * Ringe  _et al._ [2019] S. Ringe, E. L. Clark, J. Resasco, A. Walton, B. Seger, A. T. Bell, and K. Chan, Energy & Environmental Science 12, 3001 (2019).
  * Bocquet [2020] L. Bocquet, Nature Materials 19, 254 (2020).
  * Faucher  _et al._ [2019] S. Faucher, N. Aluru, M. Z. Bazant, D. Blankschtein, A. H. Brozena, J. Cumings, J. Pedro de Souza, M. Elimelech, R. Epsztein, J. T. Fourkas, _et al._ , The Journal of Physical Chemistry C 123, 21309 (2019).
  * Kosmulski [2001a] M. Kosmulski,  _Chemical properties of material surfaces_ , Vol. 102 (CRC press, 2001).
  * Uematsu [2021] Y. Uematsu, Journal of Physics: Condensed Matter 33, 423001 (2021).
  * Borkovec  _et al._ [2001] M. Borkovec, B. Jönsson, and G. J. Koper, in  _Surface and colloid science_ (Springer, 2001) pp. 99–339.
  * Kosmulski [2001b] M. Kosmulski,  _Chemical properties of material surfaces_ , Vol. 102, page 590 (CRC press, 2001) p. 590.
  * Tandon  _et al._ [2008] V. Tandon, S. K. Bhagavatula, W. C. Nelson, and B. J. Kirby, Electrophoresis 29, 1092 (2008).
  * Verwey and Overbeek [1955] E. J. W. Verwey and J. T. G. Overbeek, Journal of Colloid Science 10, 224 (1955).
  * Hiemstra  _et al._ [1989] T. Hiemstra, W. H. Van Riemsdijk, and G. Bolt, Journal of colloid and interface science 133, 91 (1989).
  * Hachiya  _et al._ [1984] K. Hachiya, M. Sasaki, T. Ikeda, N. Mikami, and T. Yasunaga, The Journal of Physical Chemistry 88, 27 (1984).
  * Sparks [2013a] D. L. Sparks,  _Kinetics of soil chemical processes_ (Academic Press, 2013) p. 88.
  * Celebi  _et al._ [2018] A. T. Celebi, M. Barisik, and A. Beskok, Microfluidics and Nanofluidics 22, 7 (2018).
  * Mouterde and Bocquet [2018] T. Mouterde and L. Bocquet, Eur Phys J E Soft Matter 41, 148 (2018).
  * Werkhoven  _et al._ [2018] B. Werkhoven, J. C. Everts, S. Samin, and R. van Roij, Physical Review Letters 120, 264502 (2018).
  * Döpke  _et al._ [2022] M. F. Döpke, F. W. van der Meij, B. Coasne, and R. Hartkamp, Physical Review Letters 128, 056001 (2022).
  * Ober  _et al._ [2021] P. Ober, W. Q. Boon, M. Dijkstra, E. H. Backus, R. van Roij, and M. Bonn, Nature Communications 12, 1 (2021).
  * Gonella  _et al._ [2016] G. Gonella, C. Lütgebaucks, A. G. De Beer, and S. Roke, The Journal of Physical Chemistry C 120, 9165 (2016).
  * Hayashi  _et al._ [2007] M. Hayashi, Y. J. Shiu, K. K. Liang, S. H. Lin, and Y. R. Shen, J Phys Chem A 111, 9062 (2007).
  * Lis  _et al._ [2014] D. Lis, E. H. Backus, J. Hunger, S. H. Parekh, and M. Bonn, Science 344, 1138 (2014).
  * Lützenkirchen  _et al._ [2018] J. Lützenkirchen, T. Scharnweber, T. Ho, A. Striolo, M. Sulpizi, and A. Abdelmonem, Journal of colloid and interface science 529, 294 (2018).
  * Zhang  _et al._ [2011] Z. Zhang, L. Piatkowski, H. J. Bakker, and M. Bonn, Nature Chemistry 3, 888 (2011).
  * Kosmulski [2001c] M. Kosmulski,  _Chemical properties of material surfaces_ , Vol. 102 (CRC press, 2001) p. 531.
  * Rudzinski and Plazinski [2006] W. Rudzinski and W. Plazinski, The Journal of Physical Chemistry B 110, 16514 (2006).
  * Sparks [2013b] D. L. Sparks,  _Kinetics of soil chemical processes_ (Academic Press, 2013) p. 63.
  * Do [1998] D. D. Do,  _Adsorption analysis: equilibria and kinetics_ , Vol. 2 (Imperial college press London, 1998).
  * Sen Gupta and Bhattacharyya [2011] S. Sen Gupta and K. G. Bhattacharyya, Adv Colloid Interface Sci 162, 39 (2011).
  * Koopal and Avena [2001] L. K. Koopal and M. J. Avena, Colloids and Surfaces A: Physicochemical and Engineering Aspects 192, 93 (2001).
  * Koopal  _et al._ [2019] L. Koopal, W. Tan, and M. Avena, Colloids and Surfaces A: Physicochemical and Engineering Aspects 577, 709 (2019).
  * Iler [1955] R. K. Iler,  _The colloid chemistry of silica and silicates_ , Vol. 80 (LWW, 1955) p. 666.
  * Grosjean  _et al._ [2016] B. Grosjean, C. Pean, A. Siria, L. Bocquet, R. Vuilleumier, and M. L. Bocquet, J Phys Chem Lett 7, 4695 (2016).
  * Febrianto  _et al._ [2009] J. Febrianto, A. N. Kosasih, J. Sunarso, Y.-H. Ju, N. Indraswati, and S. Ismadji, Journal of hazardous materials 162, 616 (2009).
  * Venema  _et al._ [1997] P. Venema, T. Hiemstra, and W. H. van Riemsdijk, Journal of Colloid and Interface Science 192, 94 (1997).
  * Westall and Hohl [1980] J. Westall and H. Hohl, Advances in Colloid and Interface Science 12, 265 (1980).
  * Van Riemsdijk  _et al._ [1987] W. Van Riemsdijk, J. De Wit, L. Koopal, and G. Bolt, Journal of colloid and interface science 116, 511 (1987).
  * [48] See Supplemental Material at https://doi.org/10.1103/PhysRevLett.130.058001 for the kinetics resulting from multiple reactions, the description of pressure-jump experiments, the similarity between autocatalytic and charging kinetics and the derivation of Eqs. (7a) and (7b), which includes Refs. [63, 64].
  * Langmuir [1918] I. Langmuir, Journal of the American Chemical society 40, 1361 (1918).
  * Bazant [2013] M. Z. Bazant, Accounts of chemical research 46, 1144 (2013).
  * Grosjean  _et al._ [2019] B. Grosjean, M. L. Bocquet, and R. Vuilleumier, Nat Commun 10, 1656 (2019).
  * Kosmulski [2001d] M. Kosmulski,  _Chemical properties of material surfaces_ , Vol. 102 (CRC press, 2001) p. 582.
  * Avena and Koopal [1999] M. J. Avena and L. K. Koopal, Environmental science & technology 33, 2739 (1999).
  * Rudzinski and Plazinski [2007] W. Rudzinski and W. Plazinski, Applied surface science 253, 5827 (2007).
  * Krumina  _et al._ [2016] L. Krumina, J. P. Kenney, J. S. Loring, and P. Persson, Chemical Geology 427, 54 (2016).
  * Herrero and Joly [2021] C. Herrero and L. Joly, arXiv preprint arXiv:2105.00720 (2021).
  * Ebeling  _et al._ [2011] D. Ebeling, D. van den Ende, and F. Mugele, Nanotechnology 22, 305706 (2011).
  * Cheb-Terrab and Kolokolnikov [2000] E. Cheb-Terrab and T. Kolokolnikov, arXiv preprint math-ph/0007023 (2000).
  * Hanopolskyi  _et al._ [2021] A. I. Hanopolskyi, V. A. Smaliak, A. I. Novichkov, and S. N. Semenov, ChemSystemsChem 3, e2000026 (2021).
  * Schuster [2019] P. Schuster, Monatshefte für Chemie-Chemical Monthly 150, 763 (2019).
  * Schaefer  _et al._ [2018] J. Schaefer, E. H. G. Backus, and M. Bonn, Nat Commun 9, 3316 (2018).
  * Yehia and Ezzat [2009] A. Yehia and K. Ezzat, Adsorption Science & Technology 27, 337 (2009).
  * Bernasconi [1976] C. Bernasconi,  _Relaxation Kinetics_ (Academic Press San Diego, 1976).
  * Teschl [2012] G. Teschl,  _Ordinary differential equations and dynamical systems_ , Vol. 140 (American Mathematical Soc., 2012).


# HAE-RAE Bench: Evaluation of Korean Knowledge in Language Models

Guijin Son1,2∗, Hanwool Lee1∗, Suwan Kim1∗, Huiseo Kim1, Jaecheol Lee1,
Je Won Yeom1, Jihyu Jung1, Jung Woo Kim1, and Songseong Kim1
1EluetherAI, 2OnelineAI

###### Abstract

Large Language Models (LLMs) trained on massive corpora demonstrate impressive
capabilities in a wide range of tasks. While there are ongoing efforts to
adapt these models to languages beyond English, the attention given to their
evaluation methodologies remains limited. Current multilingual benchmarks
often rely on back translations or re-implementations of English tests,
limiting their capacity to capture unique cultural and linguistic nuances. To
bridge this gap for the Korean language, we introduce HAE-RAE Bench, a dataset
curated to challenge models lacking Korean cultural and contextual depth. The
dataset encompasses six downstream tasks across four domains: vocabulary,
history, general knowledge, and reading comprehension. Contrary to traditional
evaluation suites focused on token or sequence classification and specific
mathematical or logical reasoning, HAE-RAE Bench emphasizes a model’s aptitude
for recalling Korean-specific knowledge and cultural contexts. Comparative
analysis with prior Korean benchmarks indicates that the HAE-RAE Bench
presents a greater challenge to non-native models, by disturbing abilities and
knowledge learned from English being transferred.

**footnotetext: These authors contributed equally to this work

##  1 Introduction

Over time, both language models and benchmark datasets have evolved in tandem,
continually becoming more sophisticated and challenging, in recognition of the
reciprocal relationship between them. Despite the pivotal role played by
benchmark datasets in advancing the capabilities of language models, the
evaluation of their multilingual abilities remains largely limited. Existing
evaluation efforts often rely on translated versions of English datasets Shi
et al. (2022) or translation-specific benchmarks such as WMT 21 Akhbardeh et
al. (2021). This approach, while providing some insights into the models’
performance across languages, may not fully capture the intricacies, nuances,
and knowledge specific to each linguistic context.

Some of the existing efforts to evaluate language models in Korean include
Korean-NLI & STS Ham et al. (2020), KLUE Park et al. (2021), and KoBEST Kim et
al. (2022). Korean-NLI & STS is derived from machine and human translations of
English datasets for natural language inference (NLI) and semantic textual
similarity (STS), accordingly, they hardly capture the unique nuances of the
Korean language. KLUE is a Korean version of the GLUE benchmark Wang et al.
(2018), which supports a variety of tasks including NLI, STS, and topic
classification. Unfortunately, its adoption was limited due to its relatively
simple tasks. The latest benchmark, KoBEST, is designed to assess a language
model’s ability to address questions that require advanced reasoning, like
understanding passages of time or causality. However, with the advent of Large
Language Models (LLMs) such as GPT-4 OpenAI (2023) and conversational agents
built upon them, there’s an increasing need to evaluate the cultural knowledge
of language models, to ensure they converse with native speakers without
sounding incoherent. To address this issue, we introduce HAE-RAE Bench***The
link for the dataset will be added after the review due to anonymity issues. a
Korean benchmark dataset originally crafted to capture culture-specific
nuances inherent to the Korean language.

##  2 Related Work

###  2.1 Language Model

Since the introduction of the Transformer architecture Vaswani et al. (2017)
and early derivatives like BERT Devlin et al. (2018) and GPT Radford et al. ,
research in English language models has expanded rapidly. The debut of
InstructGPT Ouyang et al. (2022) and Flan-T5 Chung et al. (2022), with their
instruction-following capabilities, further invigorated this interest. This
has led to the development of various instruction-tuned models, such as
Llama-2-Chat Touvron et al. (2023), Vicuna Vicuna (2023), WizardLM Xu et al.
(2023), and Platypus Lee et al. (2023). While most of these models primarily
focus on English, there are notable exceptions for Chinese, with Qwen QwenLM
(2023), Baichuan Yang et al. (2023), and GLM Zeng et al. (2022). To adapt
these advancements to various languages, several approaches are being
researched. These include:

  1. 1.

Building language-specific models from scratch, such as Polyglot-Ko(Korean) Ko
et al. (2023), Hyperclova(Korean) Kim et al. (2021), Japanese
StableLM(Japanese) StabilityAI (2023), and ruGPT(Russian) ai forever (2023);

  2. 2.

Developing multilingual models like BLOOM Scao et al. (2022), MT5 Xue et al.
(2020), and UMT5 Chung et al. (2023);

  3. 3.

Adapting English models for other languages, as seen with Sabiá Pires et al.
(2023) and Chinese-LLaMA Cui et al. (2023).

Accordingly, essential research questions emerge, including: "When can we
consider a model to have been trained on a sufficient number of language
tokens to produce culturally and grammatically coherent sentences?" This
highlights the importance of benchmarks curated to evaluate the
multilingualism of language models.

###  2.2 Multilingual Evaluation

Along with the English language models, multitask benchmarks like GLUE Wang et
al. (2018) and SuperGLUE Wang et al. (2019) were introduced. Once these were
saturated, they were followed by even bigger benchmarks such as HELM Liang et
al. (2022), MMLU Hendrycks et al. (2020), and Big BENCH Srivastava et al.
(2022). Non-English evaluation research has mirrored this trend, predominantly
through translation or re-implementation of existing English benchmarks.
Examples include JGLUE Kurihara et al. (2022), KLUE Park et al. (2021), and
CMMLU Li et al. (2023), which are Japanese and Korean adaptations of GLUE and
Chinese re-implementation of MMLU, respectively.

Yet, these benchmarks fall short of capturing the native knowledge in language
models. This highlights the need for evaluation suites curated to assess the
cultural context of a model. Recent research in this direction is BHASALeong
et al. (2023), aimed at gauging the cultural depth of language models in
Southeast Asian languages. Nonetheless, limitations are apparent: only 34
questions for Indonesian and 28 for Tamil in the entire dataset specifically
address cultural representation tasks. HAE-RAE Bench advances the field by
introducing an evaluation set of 1.5K questions, curated to assess Korean-
specific knowledge in language models.

###  2.3 Korean Evaluation

Korean language model evaluation is also a field of interest, with resources
emerging after English and Chinese. Examples include Korean-NLI & STS Ham et
al. (2020), KorFin-ASC Son et al. (2023), KLUE Park et al. (2021), and KoBEST
Kim et al. (2022). Korean-NLI & STS are based on translations of English
datasets for natural language inference (NLI) and semantic textual similarity
(STS), potentially missing Korean nuances. KorFin-ASC was originally built
from Korean news however it concentrates on sentiment classification in the
financial domain. KLUE mirrors the GLUE benchmark for Korean, covering tasks
like Topic Classification, Semantic Textual Similarity, Natural Language
Inference, and more. However, these benchmarks, whether translated or task-
oriented, can not fully assess language-specific models. Large English models
can excel in these evaluation suites by leveraging their multilingual
capabilities derived from the scale. Recent research include KoBEST Kim et al.
(2022), which features Korean re-implementations of HellaSwag Zellers et al.
(2019), COPA Gordon et al. (2012), BOOLQ Clark et al. (2019), SentiNeg Savanur
and Sumathi (2023), and WiC Pilehvar and Camacho-Collados (2018). KoBEST
advances the field by introducing reasoning tasks previously overlooked. HAE-
RAE Bench distinguishes itself from above mentioned Korean benchmarks by
evaluating the depth of knowledge encoded in language models instead of their
natural language understanding or reasoning abilities.

##  3 HAE-RAE Bench

|  | Total N. |  | Avg. Words (std) |  | Fertility Rate (std)
---|---|---|---|---|---|---
Category | Type | Question | Unique Morpheme |  | per question | per passage |  | Polyglot-Ko | UMT5 | Llama-2
Loan Words | Q | 169 | 960 |  | 5.1 (0.3) | - |  | 3.9 (0.3) | 4.1 (0.3) | 6.7 (0.6)
Rare Words | Q | 405 | 2721 |  | 13.0 (3.4) | - |  | 3.1 (0.3) | 3.5 (0.3) | 6.1 (0.4)
Standard Nomenclature | Q | 153 | 1018 |  | 8.3 (0.5) | - |  | 3.2 (0.4) | 3.6 (0.4) | 6.4 (0.6)
Reading Comprehension | Q, P | 447 | 5825 |  | 7.1 (1.8) | 69.6 (44.6) |  | 2.5 (0.4) | 2.8 (0.4) | 6.0 (0.5)
General Knowledge | Q, P | 176 | 2099 |  | 7.0 (3.0) | 9.1 (13.6) |  | 3.4 (0.6) | 3.7 (0.6) | 6.4 (0.9)
History | Q | 188 | 1595 |  | 12.8 (3.5) | - |  | 3.3 (0.4) | 3.8 (0.4) | 6.3 (0.6)

Table 1: HAE-RAE Bench Statistics.

The design principle behind HAE-RAE Bench significantly differs from earlier
Korean benchmark suites like KLUE Park et al. (2021) or KoBEST Kim et al.
(2022). While previous benchmarks focused on evaluating natural language
understanding or reasoning abilities, HAE-RAE emphasizes the depth of
knowledge itself. This change is driven by the emergence of LLMs and
conversational agents or search engines built on them. We posit that knowledge
of Korean vocabulary, culture, geography, and history might be as crucial, if
not more so, than traditional NLU tasks such as token or sequence
classification in conversational situations. Accordingly, the resulting
benchmark encompasses six downstream tasks: Loan Words(LW), Standard
Nomenclature(SN), Rare Words(RW), General Knowledge(GK), History(HI) and
Reading Comprehension(RC).

Statistics for the HAE-RAE Bench dataset are provided in Table 1. “Type"
indicates the structure of the question. “Q" denotes that the instance
comprises a question with multiple choices, while “Q, P" indicates the
inclusion of an associated passage. We also present the fertility rate of the
dataset, tokenized using different models: Polyglot-Ko Ko et al. (2023), UMT5
Chung et al. (2023), and Llama-2 Touvron et al. (2023). The fertility rate Ács
(2019) calculates the average number of sub-tokens generated per word. A
fertility rate of 1 implies that the tokenizer’s vocabulary encompasses every
word in the text. A higher fertility rate may suggest potential challenges for
the tokenizer in grasping context. In our observation the fertility rate
increases for models with less emphasis on Korean. To assess the relative
complexity of the vocabularies in the HAE-RAE Bench, we compared its fertility
rate with that of KoBEST, as shown in Table 2. Using the polyglot-ko
tokenizer, the fertility rates for HAE-RAE Bench and KoBEST are 3.0 and 2.7,
respectively. This suggests that the HAE-RAE Bench comprises less common
words. Examples for each subset of the datasets are presented in section 9.3.

Dataset | Polyglot-Ko | UMT5 | Llama-2
---|---|---|---
HAE-RAE Bench | 3.00 (0.38) | 3.56 (0.37) | 6.33 (0.59)
KoBEST | 2.70 (0.34) | 3.39 (0.45) | 6.44 (0.75)

Table 2: Fertility rate (std) of HAE-RAE Bench and KoBEST.

###  3.1 Loan Words

Task Description Loan words refer to vocabularies directly adopted from
foreign languages. In South Korea, the National Institute of Korean Language
(NIKL) ***https://www.korean.go.kr/front/main.do formulates corresponding
Korean terms for such words. In this task, language models are given a foreign
word along with five choices and are tasked to identify the correct Korean
equivalent.
Creation The pairs of foreign words and their Korean equivalents are sourced
from NIKL. Some Korean terms are infrequently used, either because the foreign
word has been entrenched in society for a long time or because it’s a recent
addition and not yet widely recognized. To ensure we focus on reasonably
common terms, we filter the list to only include words present in both “Naver
Knowledge Encyclopedia" ***https://terms.naver.com and “Daum Encyclopedia"
***https://100.daum.net, the two most widely used online encyclopedias in
Korea. From the refined list, we randomly sampled 200 vocabularies. Incorrect
options were selected from the remaining terms based on their Levenshtein
distance Levenshtein et al. (1966) to the correct answer. While Levenshtein
distance may initially seem to prioritize syntax over semantics, it
effectively captures both in the Korean language. “Han" (Chinese logograms)
constitute about 55% of the Korean vocabulary, accordingly words with the same
Korean letter have related meanings. Moreover, the structure of the Korean
language involves compounding, where multiple “roots" (fundamental word units)
merge to form new words. Consequently, words sharing similar meanings often
include the same “root", making the syntactic and semantic distances in Korean
words largely aligned. Finally, we applied a Levenshtein distance threshold of
3, omitting samples with fewer than four incorrect options meeting this
criterion.

###  3.2 Standard Nomenclature

Task Description Standard Nomenclatures, published by NIKL, are unified
terminology for domain-specific words. In this task, language models are
presented with a specialized term along with five options, with the objective
of identifying the official term endorsed by NIKL.

Creation Pairs of domain-specific words and their official terms are collected
from NIKL. We follow the approach used in 3.1 to create questions.

###  3.3 Rare Words

Task Description The Rare Words task aims to probe language models’
understanding of challenging vocabulary. Given a definition and five words,
models are tasked with selecting the word that best suits the provided
definition.

Creation We sourced pairs of definitions and challenging words from past
episodes of the TV program “Woorimal Battle,"
***https://program.kbs.co.kr/1tv/culture/woorimal/pc/index.html known for its
challenging Korean vocabulary quizzes. We follow the approach used in 3.1 to
create questions.

###  3.4 General Knowledge

Task Description General Knowledge evaluates the model’s familiarity with
various aspects of the Korean cultural, using five-option multiple-choice
questions.

Category | Sample N. | Average Length
---|---|---
Tradition | 17 | 35.2
Law | 10 | 32.2
Geography | 49 | 46
Korean Pop | 50 | 42.3
Korean Drama | 50 | 36.7

Table 3: The number of data instances for each category.

Creation We first identify five primary categories for general knowledge: law,
tradition, geography, Korean pop, and Korean drama. We then crowd-sourced
questions to fit these subcategories. Overlapping and factually incorrect
questions were removed, along with those not aligning with the defined
category. Additional investigations were conducted to ensure no superficial
artifacts were inadvertently introduced. Basic statistics for each sub-
category is illustrated in Table 3.

Investigation Following Kaushik and Lipton (2018), we examined the performance
of Polyglot-Ko-12.8B using question-only (Q-only) and context-only (C-only)
settings. Polyglot-Ko-12.8B achieved scores of 25.57% and 23.86% for Q-only
and C-only respectively, while the full setting outperformed both with a score
of 32.95%. Although the Q-only and C-only settings are within 10% accuracy of
the full setting, it’s worth noting that the model’s lower bound is set at
20%. Therefore, we conclude that the dataset were crafted correctly to require
both question and context to answer.

Metric | Full | Q-Only | C-Only |  $\Delta$ (min)
---|---|---|---|---
Acc | 32.95 | 25.57 | 23.86 | -7.38
Macro F1 | 32.01 | 24.35 | 23.64 | -7.56

Table 4: Performance of Polyglot-Ko-12.8B on General Knowledge with truncated
inputs.

###  3.5 History

Task Description The history task assesses the model’s understanding of
historical events. Presented with a question and five options, the model must
identify the correct answer.

Creation We first sourced web pages tagged “Korean history" from Namuwiki,
Korea’s equivalent to Wikipedia, and randomly selected 40 pages. From each
page, authors manually crafted five questions. We refer Malaviya et al. (2022)
and filtered out 12 questions with overlapping tokens between questions and
answers. Moreover, to investigate potential biases introduced while creating
the wrong options, we analyzed two simple linguistic indicators: the
probability of the longest option being correct was 21.53%, and for the
shortest option, it was 17.01%. Through this process, we remove overly
simplistic questions and investigate for potential biases.

###  3.6 Reading Comprehension

Task Description Reading comprehension tasks involve providing paired
questions and passages along with four options. The materials for our Reading
Comprehension (RC) tests were sourced from the Korean Language Ability Test
(KLAT), an exam designed to evaluate proficiency in Korean as a second
language.

Creation The tests were gathered from sample materials publicly released by
the Korea Educational Testing Service (KETS). We omitted questions that
required interpreting images. The sourced KLAT is divided into four
proficiency tiers: three that correspond to the Common European Framework of
Reference (CEFR) levels—A (beginner), B (intermediate), and C (advanced)—plus
an introductory level below A for absolute beginners.

###  3.7 Quality Check

To further filter the collected questions, we reviewed the entire dataset and
conducted factual verification using online resources. In this process, we
manually corrected 23 questions with labeling or crawling errors.

##  4 Evaluation Settings

###  4.1 Language Models

We evaluated 10 models across varying sizes from four model families. From
openly available models we selected (1) Korean-focused models: Polyglot-
ko-1.3B/3.8B/5.8B/12.8B Ko et al. (2023), (2) Multilingual models: UMT5-XL/XXL
Chung et al. (2023), and (3) English-centric models: Llama-2-7B/13B Touvron et
al. (2023). To investigate the influence of the number of pretrained Korean
tokens on model performance we excluded models that do not disclose related
statistics. This leaves out Falcon Penedo et al. (2023) and BLOOM Scao et al.
(2022) from our experiments. Additionally, we included GPT-3.5-Turbo and GPT-4
in our evaluation to gauge HAE-RAE Bench’s efficacy in assessing LLMs.

Polyglot-Ko Ko et al. (2023) is available in four sizes: 1.3B, 3.8B, 5.8B, and
12.8B, all built using the GPT-NeoX codebase. It was pretrained on a Korean-
only corpus, with sizes ranging between 167B to 212B tokens. Despite its
smaller pretraining budget compared to similar-sized English models, Polyglot-
Ko achieved state-of-the-art results on KoBEST, a benchmark comprising five
Korean language understanding and reasoning tasks Kim et al. (2022).

UMT5Chung et al. (2023) was originally trained in five sizes: small (77M),
base (250M), large (800M), xlarge (3B), and xxlarge (13B), closely following
the mT5 architectureXue et al. (2020). However, the large variant wasn’t
released publicly due to pretraining instability. The models are trained on a
corpus of 1T tokens, which includes 14.8 billion Korean tokens. UMT5 surpasses
mT5 in benchmarks such as XNLI Conneau et al. (2018) and TyDi QA Clark et al.
(2020). As the small and base models don’t have counterparts in the Polyglot-
Ko suite, our experiments focus on the xlarge and xxlarge models.

Llama-2 Touvron et al. (2023) is available in three sizes: 7B, 13B, and 70B.
It’s trained on a corpus of 2T tokens, predominantly in English (89.7%), with
Korean comprising a mere 0.06% or about 0.6B tokens. We utilize the version
without fine-tuning. The Llama-2-70B model is excluded from our study due to
the absence of a corresponding Korean model.

###  4.2 Methodology

|  | Loan Words |  | Standard Nomenclature |  | Rare Words
---|---|---|---|---|---|---
Model | Params | n=0 | n=5 | n=10 |  | n=0 | n=5 | n=10 |  | n=0 | n=5 | n=10
Polyglot-Ko | 1.3B | 76.92 | 88.76 | 91.72 |  | 60.13 | 69.93 | 71.24 |  | 47.41 | 61.48 | 61.23
3.8B | 78.70 | 88.76 | 91.72 |  | 63.40 | 79.74 | 77.78 |  | 47.16 | 70.62 | 72.10
5.8B | 82.84 | 93.49 | 94.08 |  | 66.67 | 82.35 | 83.66 |  | 56.79 | 73.09 | 74.57
12.8B | 87.57 | 94.67 | 94.67 |  | 61.44 | 84.97 | 86.93 |  | 53.09 | 75.31 | 76.05
UMT5 | 3B | 58.58 | 61.54 | 59.76 |  | 41.83 | 37.25 | 33.33 |  | 25.68 | 25.43 | 24.44
13B | 58.58 | 59.76 | 60.36 |  | 41.83 | 43.79 | 44.44 |  | 33.09 | 30.37 | 28.64
LLaMA-2 | 7B | 66.86 | 73.96 | 75.15 |  | 39.22 | 49.02 | 50.98 |  | 29.38 | 39.26 | 39.01
13B | 66.86 | 77.51 | 78.11 |  | 49.02 | 57.52 | 64.05 |  | 32.35 | 42.47 | 43.95

Table 5: Evaluation results of the performance on Loan Words, Standard
Nomenclature, and Rare Word tasks.

|  | History |  | General Knowledge |  | Reading Comprehension
---|---|---|---|---|---|---
Model | Params | n=0 | n=5 | n=10 |  | n=0 | n=5 | n=10 |  | n=0 | n=5 | n=10
Polyglot-Ko | 1.3B | 60.11 | 78.19 | 77.13 |  | 26.70 | 30.68 | 28.98 |  | 34.45 | 37.81 | 37.14
3.8B | 69.15 | 86.17 | 85.11 |  | 28.41 | 33.52 | 33.52 |  | 40.49 | 42.06 | 40.04
5.8B | 79.79 | 85.11 | 81.91 |  | 29.55 | 27.84 | 28.41 |  | 40.72 | 42.73 | 41.39
12.8B | 80.32 | 88.30 | 90.43 |  | 32.95 | 33.52 | 34.66 |  | 41.61 | 45.41 | 46.76
UMT5 | 3B | 14.36 | 12.77 | 14.36 |  | 22.73 | 19.32 | 19.32 |  | 25.28 | 24.83 | 25.28
13B | 21.59 | 18.09 | 19.15 |  | 21.81 | 25.00 | 19.32 |  | 29.75 | 25.28 | 27.74
LLaMA-2 | 7B | 28.72 | 35.64 | 35.64 |  | 21.02 | 24.43 | 25.00 |  | 29.98 | 32.89 | 31.32
13B | 35.11 | 38.83 | 40.96 |  | 28.41 | 31.82 | 28.98 |  | 31.99 | 36.47 | 34.00

Table 6: Evaluation results of the performance on History, General Knowledge,
and Reading Comprehension tasks.

To evaluate the models, we employ the “log-likelihood" method implemented via
LM-Eval-Harness Gao et al. (2021) using accuracy as our primary metric. This
involves computing the log-likelihood for each response and selecting the
option with the highest likelihood. All evaluations are implemented using
bfloat16 precision in 0-shot, 5-shot, and 10-shot settings.

The aim of HAE-RAE Bench is to curate a dataset that challenges models lacking
depth in Korean culture and knowledge, thereby guiding researchers in creating
better Korean language models. To compare the ability of this benchmark to
differentiate less native language models against prior benchmarks, we use
KoBEST Kim et al. (2022) as our baseline. We selected KoBEST as it offers a
broad range of language understanding and reasoning tasks. KoBEST comprises
five tasks: BoolQ, COPA, HellaSwag, WiC, and SentiNeg. However, given the
findings of Ko et al. (2023), that both monolingual and multilingual language
models exhibit inconsistent performance on WiC, we opted to omit this task
from our assessment. While there are other available datasets that may be
adopted as baselines, they come with limitations. For instance, Korean-NLI&STS
Ham et al. (2020), being translated from English, is inherently easier for
English models. KLUE Park et al. (2021), despite being handcrafted, primarily
focuses on basic NLU tasks like topic classification and NER. This makes it
incapable of evaluating complex reasoning capabilities. Additionally, its test
set is not publicly available.

##  5 Evaluation Results

Is HAE-RAE bench harder for foreign models? In Tables 5 and 6, we observe that
the performance of LMs scales with model size and the number of exemplars
within the same suite. Yet, despite their extensive training budgets, UMT5 and
Llama-2 consistently fall short of their Polyglot-Ko counterparts.
Furthermore, they rarely surpass the results of Polyglot-Ko-1.3B (0-shot).
These results reaffirm the importance of language-specific corpora in learning
cultural context and knowledge. It also highlights the effectiveness of HAE-
RAE Bench, in assessing the language model’s proficiency in Korean.

| Polyglot-Ko | $\Delta$ |
---|---|---|---
Dataset | Params | Average | UMT5 | Llama-2 |
HAE-RAE Bench | 1.3B | 51.0 | -16.5 | -15.1 |
3.8B | 54.6 | -20.1 | -18.7 |
5.8B | 59.4 | -25.0 | -23.5 |
12.8B | 59.5 | -25.1 | -23.6 |
KoBEST | 1.3B | 56.3 | -6.3 | -6.1 |
3.8B | 55.7 | -5.7 | -5.5 |
5.8B | 56.0 | -6.0 | -5.8 |
12.8B | 65.2 | -15.2 | -15.0 |

Table 7: Average Performance of Polyglot-Ko vs. UMT5-XXL and Llama-2-13B on
HAE-RAE and KoBEST (0-shot).

| Polyglot-Ko | $\Delta$ |
---|---|---|---
Dataset | Params | Average | UMT5 | Llama-2 |
HAE-RAE Bench | 1.3B | 61.1 | -27.4 | -13.7 |
3.8B | 66.8 | -33.1 | -19.4 |
5.8B | 67.4 | -33.7 | -20.0 |
12.8B | 70.4 | -36.6 | -22.9 |
KoBEST | 1.3B | 56.4 | -8.4 | 7.8 |
3.8B | 64.7 | -16.6 | -0.5 |
5.8B | 68.0 | -19.9 | -3.8 |
12.8B | 71.4 | -23.3 | -7.2 |

Table 8: Average Performance of Polyglot-Ko vs. UMT5-XXL and Llama-2-13B on
HAE-RAE and KoBEST (5-shot).

| Polyglot-Ko | $\Delta$ |
---|---|---|---
Dataset | Params | Average | UMT5 | Llama-2 |
HAE-RAE Bench | 1.3B | 61.2 | -28.0 | -12.9 |
3.8B | 66.7 | -33.4 | -18.4 |
5.8B | 67.3 | -34.1 | -19.0 |
12.8B | 71.6 | -38.3 | -23.2 |
KoBEST | 1.3B | 55.0 | -8.7 | 12.1 |
3.8B | 63.3 | -16.9 | 3.9 |
5.8B | 68.0 | -21.6 | -0.8 |
12.8B | 71.8 | -25.4 | -4.6 |

Table 9: Average Performance of Polyglot-Ko vs. UMT5-XXL and Llama-2-13B on
HAE-RAE and KoBEST (10-shot).

Our results illustrated in Tables 7, 8, and 9 suggest that HAE-RAE Bench is
particularly challenging for non-Korean models compared to the KoBEST
benchmark. The performance gap between Polyglot-Ko and its counterparts is
more pronounced on HAE-RAE Bench compared to KoBEST across all exemplar
counts. Notably, for Llama-2-13B, the margin narrows considerably on KoBEST
with an increase in exemplars. This discrepancy highlights that the proposed
benchmark is especially challenging for models not tailored in Korean while
being difficult to mitigate by in-context learning. The entire result for
KoBEST is illustrated in section 9.1.

Does language frequency in the training corpora matter? Despite UMT5 being
trained on a larger volume of Korean tokens, it underperforms Llama-2 on HAE-
RAE Bench. Moreover, the advantage of in-context learning is relatively
minimal for UMT5. Our findings support previous claims that language-specific
reasoning capabilities of language models are not solely tied to the number of
dedicated tokens in the pretraining corpus Shi et al. (2022). These results
indicate that language models under the size of 20B parameters also transfer
their in-context learning abilities to low-resource languages.

How important is the model size for HAE-RAE Bench? In Table 10, we employ
regression and Analysis of Variance(ANOVA) to examine the correlation between
the parameter count of Polyglot-Ko models and their performance. To narrow the
focus solely on the impact of model size, the analysis is limited to the
Polyglot-Ko family, thus setting aside variables like corpus quality or model
architecture. For the KoBEST benchmark, the results demonstrate a marked
relationship between performance and model size, as indicated by the high
$R^{2}$ value of 0.71 and the significant F-statistic and p-value. In
contrast, for HAE-RAE Bench the model size explains only about a quarter of
the performance variability. Additionally, the absence of statistical
significance in both the regression and ANOVA for HAE-RAE Bench implies that
its evaluation is influenced by a broader spectrum of factors, pointing to
challenges beyond just model size.

| Regression | ANOVA
---|---|---
Benchmark | $\beta_{0}$ | $\beta_{1}$ | $R^{2}$ |  $F$-statistic
HAE-RAE Bench | 58.79 | 0.73 | 0.26 | 1.42
KoBEST | 56.49 | 1.17 | 0.71* | 8.23*

Table 10: Results from regression and ANOVA for the HAE-RAE and KoBEST
benchmarks. An asterisk (*) denotes outcomes with a p-value less than 0.01,
indicating statistical significance.

Can GPT-3.5/4 ace HAE-RAE Bench? In Table 11, the performance of GPT-3.5 and
GPT-4 on the HAE-RAE Bench and KoBEST is presented. Unlike openly available
models for which we leveraged a log probability method to gauge accuracy,
these models do not provide log probabilities for individual tokens.
Accordingly, we prompted the models to generate the number of the options they
deemed correct. Direct comparison between these evaluation methods is not
feasible, however, the method used for proprietary models is more challenging
than the log-likelihood method applied to open models. The former entails
generating answers from the entire vocabulary, whereas the latter restricts
choices to five options. Notably, GPT-3.5 and GPT-4 achieved scores of 51.2%
and 67.8% on the HAE-RAE Bench, respectively, indicating potential for further
improvements. Conversely, their performances on KoBEST were 68.0% and 81.1%,
suggesting narrower margins for improvement. In summary, state-of-the-art
language models such as GPT-3.5 and GPT-4 have yet to master either the HAE-
RAE Bench or KoBEST, though more room is left for HAE-RAE Bench. The entire
evaluation results for GPT-3.5 and GPT-4 models is available at section 9.2.

| GPT-3.5-Turbo |  | GPT-4
---|---|---|---
Dataset | Ko | En | $\Delta$ |  | Ko | En | $\Delta$
HAE-RAE Bench | 51.2 | 55.4 | 4.2 |  | 67.8 | 68.2 | 0.4
KoBEST | 68.0 | 79.3 | 11.4 |  | 81.1 | 91.0 | 9.9

Table 11: Evaluation result of GPT-3.5-Turbo and GPT-4 on HAE-RAE Bench and
KoBEST with zero shot setting. We use the snapshot from June 13th 2023 for
both models. Ko and En denote the language of the prompt used.

Can knowledge be transferred from English? Past research indicates that LLMs
can internally transfer knowledge acquired in English to low-resource
languages Huang et al. (2023); Zhou et al. (2023). To investigate whether LLMs
leverage abilities derived from English corpora to solve HAE-RAE Bench, we
employed Cross-lingual thought prompting (XLT) with GPT-3.5 and GPT-4. XLT Shi
et al. (2022) is a technique that aids the transfer of abilities learned in
English to other languages. As illustrated in Table 11, English prompting
enhances the performance of LLMs on both HAE-RAE Bench and KoBEST. However,
the gains for HAE-RAE Bench are modest: 4.2 for GPT-3.5 and 0.4 for GPT-4. In
contrast, the improvements on KoBEST are more substantial, with margins of 9.9
and 11.1, respectively. Given KoBEST’s focus on language understanding and
reasoning, we suspect that such skills are more seamlessly transferable across
languages within models. On the other hand, HAE-RAE Bench probes the nuances
of cultural context and knowledge, aspects that are challenging to learn from
English tokens, thereby undermining the benefits of extensive training across
various languages.

##  6 Error Analysis

Error analysis is important to understand the common errors or likely biases
of language model mistakes and identify areas of future research. Accordingly,
we compare the results of Polyglot-Ko-12.8B (0-shot) and GPT-4 (Korean
Prompting) for possible errors.

Figure 1: Density distribution of answer choices by Polyglot-Ko-12.8B, GPT-4,
and Gold Labels.

We first examine the answer distribution to see if either model has a bias
toward selecting certain numbers. This is shown in Figure 1. We find that both
models are less likely to guess “ 5" compared to other numbers. This pattern
can be traced back to the dataset composition: while most questions offer five
multiple-choice options, the reading comprehension subset provides only four.
Beyond this, neither model displays any notable trends.

|  | GPT-4 |  | Polyglot-Ko-12.8B
---|---|---|---|---
Dataset |  | Correct | Incorrect |  | Correct | Incorrect
Rare Words |  | 1.58 (0.81) | 1.58 (0.80) |  | 1.58 (0.81) | 1.58 (0.80)
Loan Words |  | 1.58 (0.80) | 1.56 (0.80) |  | 1.58 (0.80) | 1.57 (0.80)
Standard Nomenclature |  | 1.58 (0.80) | 1.55 (0.80) |  | 1.57 (0.81) | 1.56 (0.80)

Table 12: Average Levenshtein distance of options for correct and incorrect
questions.

In the Rare Words, Loan Words, and Standard Nomenclature subsets of HAE-RAE
Bench, incorrect options were generated using a sorting method based on
Levenshtein distance. To investigate the impact of Levenshtein distance on
model performance, we compare the average distances for options based on
whether the model answered the question correctly. As shown in Table 12, no
discernible difference in Levenshtein distance is observed for either model
between correct and incorrect answers. We suspect that the set Levenshtein
distance threshold of 3 may not lead to meaningful variations in question
difficulty. To delve deeper, we reviewed all incorrect questions for Polyglot-
Ko-12.8B and GPT-4. However, given the questions’ simple structures, such as
"What is the {official loan word / correct standard nomenclature} for {word}?"
or "Which word is suitable for the definition {def}?", we did not identify any
syntactic characteristics that might explain the incorrect questions.

Figure 2: Accuracy of Polyglot-Ko-12.8B and GPT-4 on sub-categories of General
Knowledge. The striped areas within each bar represent questions that both
models answered correctly. Figure 3: Accuracy of Polyglot-Ko-12.8B and GPT-4
on different levels of Reading Comprehension.

For the General Knowledge subset, we assess the performance of Polyglot-
Ko-12.8B and GPT-4 across the subcategories, as shown in Figure 2. GPT-4
consistently outperforms Polyglot-Ko-12.8B. Polyglot-Ko-12.8B fares better in
law and culture but lags in geography, K-pop, and especially K-drama. The
model’s weaker performance in K-pop and K-drama may stem from its knowledge
cutoff, given the need for up-to-date information in these areas. GPT-4 excels
across all categories, likely benefiting from a diverse training set. Both
models have the lowest scores in the K-drama category, suggesting either model
limitations or ambiguous questions.

The Reading Comprehension subset of HAE-RAE Bench comprises four difficulty
levels: Introductory (for absolute beginners), A (beginner), B (intermediate),
and C (advanced), based on the Common European Framework of Reference (CEFR).
In figure 3 we examine the performance of each model across these levels. Our
findings indicate that GPT-4 consistently outperforms Polyglot-Ko-12.8B across
all difficulty tiers, with the performance gap becoming more pronounced at
higher levels (B and C). The performance of Polyglot-Ko-12.8B peaks at
difficulty level A and declines, suggesting a limitation in handling more
challenging questions.

##  7 License

HAE-RAE Bench is released under a CC BY-NC-ND license. This license prohibits
remixing, redistribution, and commercial use of the dataset. This constraint
is due to the reading comprehension subset, for which the copyright holder of
KLAT has restricted commercial alterations. However, we do not anticipate this
as a significant issue since benchmark datasets are seldom used for commercial
purposes. Researchers can still freely download and evaluate their models
using this dataset.

##  8 Conclusion

In this paper, we introduce HAE-RAE Bench, a dataset curated to evaluate the
cultural knowledge encoded in language models. Unlike previous Korean language
model evaluation suites, HAE-RAE Bench is crafted to present a greater
challenge to non-Korean models, disrupting their ability to guess answers
based on in-context learning or scale-derived multilingualism. Our work is
among the first to propose a non-task-oriented dataset aimed at assessing
whether a language model’s knowledge is adequate for roles like a domestic
conversational agent or search engine. This research suggests a pathway for
advancing non-English NLP, emphasizing the need for language models that are
as proficient in language-specific knowledge as they are in language
understanding and reasoning tasks.

## References

  * ai forever (2023) ai forever. 2023.  rugpt-3.5 13b.
  * Akhbardeh et al. (2021) Farhad Akhbardeh, Arkady Arkhangorodsky, Magdalena Biesialska, Ondřej Bojar, Rajen Chatterjee, Vishrav Chaudhary, Marta R. Costa-jussa, Cristina España-Bonet, Angela Fan, Christian Federmann, Markus Freitag, Yvette Graham, Roman Grundkiewicz, Barry Haddow, Leonie Harter, Kenneth Heafield, Christopher Homan, Matthias Huck, Kwabena Amponsah-Kaakyire, Jungo Kasai, Daniel Khashabi, Kevin Knight, Tom Kocmi, Philipp Koehn, Nicholas Lourie, Christof Monz, Makoto Morishita, Masaaki Nagata, Ajay Nagesh, Toshiaki Nakazawa, Matteo Negri, Santanu Pal, Allahsera Auguste Tapo, Marco Turchi, Valentin Vydrin, and Marcos Zampieri. 2021.  Findings of the 2021 conference on machine translation (WMT21).  In _Proceedings of the Sixth Conference on Machine Translation_ , pages 1–88, Online. Association for Computational Linguistics.
  * Chung et al. (2023) Hyung Won Chung, Noah Constant, Xavier Garcia, Adam Roberts, Yi Tay, Sharan Narang, and Orhan Firat. 2023.  Unimax: Fairer and more effective language sampling for large-scale multilingual pretraining.  _arXiv preprint arXiv:2304.09151_.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.  BoolQ: Exploring the surprising difficulty of natural yes/no questions.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Clark et al. (2020) Jonathan H. Clark, Eunsol Choi, Michael Collins, Dan Garrette, Tom Kwiatkowski, Vitaly Nikolaev, and Jennimaria Palomaki. 2020.  TyDi QA: A benchmark for information-seeking question answering in typologically diverse languages.  _Transactions of the Association for Computational Linguistics_ , 8:454–470.
  * Conneau et al. (2018) Alexis Conneau, Guillaume Lample, Ruty Rinott, Adina Williams, Samuel R Bowman, Holger Schwenk, and Veselin Stoyanov. 2018.  Xnli: Evaluating cross-lingual sentence representations.  _arXiv preprint arXiv:1809.05053_.
  * Cui et al. (2023) Yiming Cui, Ziqing Yang, and Xin Yao. 2023.  Efficient and effective text encoding for chinese llama and alpaca.  _arXiv preprint arXiv:2304.08177_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Gao et al. (2021) Leo Gao, Jonathan Tow, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Kyle McDonell, Niklas Muennighoff, Jason Phang, Laria Reynolds, Eric Tang, Anish Thite, Ben Wang, Kevin Wang, and Andy Zou. 2021.  A framework for few-shot language model evaluation.
  * Gordon et al. (2012) Andrew Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012.  SemEval-2012 task 7: Choice of plausible alternatives: An evaluation of commonsense causal reasoning.  In _*SEM 2012: The First Joint Conference on Lexical and Computational Semantics – Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Evaluation (SemEval 2012)_ , pages 394–398, Montréal, Canada. Association for Computational Linguistics.
  * Ham et al. (2020) Jiyeon Ham, Yo Joong Choe, Kyubyong Park, Ilji Choi, and Hyungjoon Soh. 2020.  KorNLI and KorSTS: New benchmark datasets for Korean natural language understanding.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 422–430, Online. Association for Computational Linguistics.
  * Hendrycks et al. (2020) Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. 2020.  Measuring massive multitask language understanding.  _arXiv preprint arXiv:2009.03300_.
  * Huang et al. (2023) Haoyang Huang, Tianyi Tang, Dongdong Zhang, Wayne Xin Zhao, Ting Song, Yan Xia, and Furu Wei. 2023.  Not all languages are created equal in llms: Improving multilingual capability by cross-lingual-thought prompting.  _arXiv preprint arXiv:2305.07004_.
  * Kaushik and Lipton (2018) Divyansh Kaushik and Zachary C Lipton. 2018.  How much reading does reading comprehension require? a critical investigation of popular benchmarks.  _arXiv preprint arXiv:1808.04926_.
  * Kim et al. (2021) Boseop Kim, HyoungSeok Kim, Sang-Woo Lee, Gichang Lee, Donghyun Kwak, Jeon Dong Hyeon, Sunghyun Park, Sungju Kim, Seonhoon Kim, Dongpil Seo, Heungsub Lee, Minyoung Jeong, Sungjae Lee, Minsub Kim, Suk Hyun Ko, Seokhun Kim, Taeyong Park, Jinuk Kim, Soyoung Kang, Na-Hyeon Ryu, Kang Min Yoo, Minsuk Chang, Soobin Suh, Sookyo In, Jinseong Park, Kyungduk Kim, Hiun Kim, Jisu Jeong, Yong Goo Yeo, Donghoon Ham, Dongju Park, Min Young Lee, Jaewook Kang, Inho Kang, Jung-Woo Ha, Woomyoung Park, and Nako Sung. 2021.  What changes can large-scale language models bring? intensive study on HyperCLOVA: Billions-scale Korean generative pretrained transformers.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3405–3424, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Kim et al. (2022) Dohyeong Kim, Myeongjun Jang, Deuk Sin Kwon, and Eric Davis. 2022.  Kobest: Korean balanced evaluation of significant tasks.  _arXiv preprint arXiv:2204.04541_.
  * Ko et al. (2023) Hyunwoong Ko, Kichang Yang, Minho Ryu, Taekyoon Choi, Seungmu Yang, Sungho Park, et al. 2023.  A technical report for polyglot-ko: Open-source large-scale korean language models.  _arXiv preprint arXiv:2306.02254_.
  * Kurihara et al. (2022) Kentaro Kurihara, Daisuke Kawahara, and Tomohide Shibata. 2022.  JGLUE: Japanese general language understanding evaluation.  In _Proceedings of the Thirteenth Language Resources and Evaluation Conference_ , pages 2957–2966, Marseille, France. European Language Resources Association.
  * Lee et al. (2023) Ariel N Lee, Cole J Hunter, and Nataniel Ruiz. 2023.  Platypus: Quick, cheap, and powerful refinement of llms.  _arXiv preprint arXiv:2308.07317_.
  * Leong et al. (2023) Wei Qi Leong, Jian Gang Ngui, Yosephine Susanto, Hamsawardhini Rengarajan, Kengatharaiyer Sarveswaran, and William Chandra Tjhi. 2023.  Bhasa: A holistic southeast asian linguistic and cultural evaluation suite for large language models.
  * Levenshtein et al. (1966) Vladimir I Levenshtein et al. 1966.  Binary codes capable of correcting deletions, insertions, and reversals.  In _Soviet physics doklady_ , volume 10, pages 707–710. Soviet Union.
  * Li et al. (2023) Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. 2023.  Cmmlu: Measuring massive multitask language understanding in chinese.  _arXiv preprint arXiv:2306.09212_.
  * Liang et al. (2022) Percy Liang, Rishi Bommasani, Tony Lee, Dimitris Tsipras, Dilara Soylu, Michihiro Yasunaga, Yian Zhang, Deepak Narayanan, Yuhuai Wu, Ananya Kumar, et al. 2022.  Holistic evaluation of language models.  _arXiv preprint arXiv:2211.09110_.
  * Malaviya et al. (2022) Chaitanya Malaviya, Sudeep Bhatia, and Mark Yatskar. 2022.  Cascading biases: Investigating the effect of heuristic annotation strategies on data and models.  In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_ , pages 6525–6540, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.  _ArXiv_ , abs/2303.08774.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022\.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Park et al. (2021) Sungjoon Park, Jihyung Moon, Sungdong Kim, Won Ik Cho, Jiyoon Han, Jangwon Park, Chisung Song, Junseong Kim, Yongsook Song, Taehwan Oh, et al. 2021.  Klue: Korean language understanding evaluation.  _arXiv preprint arXiv:2105.09680_.
  * Penedo et al. (2023) Guilherme Penedo, Quentin Malartic, Daniel Hesslow, Ruxandra Cojocaru, Alessandro Cappelli, Hamza Alobeidli, Baptiste Pannier, Ebtesam Almazrouei, and Julien Launay. 2023.  The refinedweb dataset for falcon llm: outperforming curated corpora with web data, and web data only.  _arXiv preprint arXiv:2306.01116_.
  * Pilehvar and Camacho-Collados (2018) Mohammad Taher Pilehvar and Jose Camacho-Collados. 2018.  Wic: the word-in-context dataset for evaluating context-sensitive meaning representations.  _arXiv preprint arXiv:1808.09121_.
  * Pires et al. (2023) Ramon Pires, Hugo Abonizio, Thales Rogério, and Rodrigo Nogueira. 2023.  Sabi$\backslash$’a: Portuguese large language models.  _arXiv preprint arXiv:2304.07880_.
  * QwenLM (2023) QwenLM. 2023.  Introducing qwen-7b: Open foundation and human-aligned models (of the state-of-the-arts).
  * (33) Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al.  Improving language understanding by generative pre-training.
  * Savanur and Sumathi (2023) Sandhya Savanur and R. Sumathi. 2023.  Sentineg: Algorithm to process negations at sentence level in sentiment analysis.  _International Journal of Software Innovation_ , 11:1–27.
  * Scao et al. (2022) Teven Le Scao, Angela Fan, Christopher Akiki, Ellie Pavlick, Suzana Ilić, Daniel Hesslow, Roman Castagné, Alexandra Sasha Luccioni, François Yvon, Matthias Gallé, et al. 2022.  Bloom: A 176b-parameter open-access multilingual language model.  _arXiv preprint arXiv:2211.05100_.
  * Shi et al. (2022) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, et al. 2022.  Language models are multilingual chain-of-thought reasoners.  _arXiv preprint arXiv:2210.03057_.
  * Son et al. (2023) Guijin Son, Hanwool Lee, Nahyeon Kang, and Moonjeong Hahm. 2023.  Removing non-stationary knowledge from pre-trained language models for entity-level sentiment classification in finance.  _arXiv preprint arXiv:2301.03136_.
  * Srivastava et al. (2022) Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao, Abu Awal Md Shoeb, Abubakar Abid, Adam Fisch, Adam R Brown, Adam Santoro, Aditya Gupta, Adrià Garriga-Alonso, et al. 2022.  Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.  _arXiv preprint arXiv:2206.04615_.
  * StabilityAI (2023) StabilityAI. 2023.  Japanese-stablelm-base-alpha-7b.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _arXiv preprint arXiv:2307.09288_.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  _Advances in neural information processing systems_ , 30.
  * Vicuna (2023) Vicuna. 2023.  Vicuna: An open-source chatbot impressing gpt-4 with 90% chatgpt quality.
  * Wang et al. (2019) Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019.  Superglue: A stickier benchmark for general-purpose language understanding systems.  _Advances in neural information processing systems_ , 32.
  * Wang et al. (2018) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018.  Glue: A multi-task benchmark and analysis platform for natural language understanding.  _arXiv preprint arXiv:1804.07461_.
  * Xu et al. (2023) Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. 2023.  Wizardlm: Empowering large language models to follow complex instructions.  _arXiv preprint arXiv:2304.12244_.
  * Xue et al. (2020) Linting Xue, Noah Constant, Adam Roberts, Mihir Kale, Rami Al-Rfou, Aditya Siddhant, Aditya Barua, and Colin Raffel. 2020.  mt5: A massively multilingual pre-trained text-to-text transformer.  _arXiv preprint arXiv:2010.11934_.
  * Yang et al. (2023) Aiyuan Yang, Bin Xiao, Bingning Wang, Borong Zhang, Ce Bian, Chao Yin, Chenxu Lv, Da Pan, Dian Wang, Dong Yan, Fan Yang, Fei Deng, Feng Wang, Feng Liu, Guangwei Ai, Guosheng Dong, Haizhou Zhao, Hang Xu, Haoze Sun, Hongda Zhang, Hui Liu, Jiaming Ji, Jian Xie, JunTao Dai, Kun Fang, Lei Su, Liang Song, Lifeng Liu, Liyun Ru, Luyao Ma, Mang Wang, Mickel Liu, MingAn Lin, Nuolan Nie, Peidong Guo, Ruiyang Sun, Tao Zhang, Tianpeng Li, Tianyu Li, Wei Cheng, Weipeng Chen, Xiangrong Zeng, Xiaochuan Wang, Xiaoxi Chen, Xin Men, Xin Yu, Xuehai Pan, Yanjun Shen, Yiding Wang, Yiyu Li, Youxin Jiang, Yuchen Gao, Yupeng Zhang, Zenan Zhou, and Zhiying Wu. 2023.  Baichuan 2: Open large-scale language models.
  * Zellers et al. (2019) Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019.  Hellaswag: Can a machine really finish your sentence?  _arXiv preprint arXiv:1905.07830_.
  * Zeng et al. (2022) Aohan Zeng, Xiao Liu, Zhengxiao Du, Zihan Wang, Hanyu Lai, Ming Ding, Zhuoyi Yang, Yifan Xu, Wendi Zheng, Xiao Xia, et al. 2022.  Glm-130b: An open bilingual pre-trained model.  _arXiv preprint arXiv:2210.02414_.
  * Zhou et al. (2023) Meng Zhou, Xin Li, Yue Jiang, and Lidong Bing. 2023.  Enhancing cross-lingual prompting with dual prompt augmentation.  In _Findings of the Association for Computational Linguistics: ACL 2023_ , pages 11008–11020.
  * Ács (2019) Judit Ács. 2019.  Exploring bert’s vocabulary.

##  9 Appendix

###  9.1 KoBEST Evaluation Results

Model | Lang | BoolQ | COPA | HellaSwag | SentiNeg
---|---|---|---|---|---
GPT-3.5-Turbo | Ko | 82.34 | 57.88 | 40.00 | 91.69
En | 86.40 | 79.20 | 55.00 | 96.73
GPT-4 | Ko | 96.58 | 52.40 | 76.60 | 98.74
En | 96.65 | 95.70 | 73.00 | 98.49

Table 13: Evaluation results for GPT-3.5 and GPT-4 on KoBEST

Model | Lang |  | Loan
---
Words
| Standard
---
Nomenclature
| Rare
---
Words
History |  | General
---
Knowledge
| Reading
---
Comprehension
GPT-3.5-Turbo | Ko | 62.13 | 55.56 | 63.46 | 30.32 | 35.80 | 60.18
En | 72.19 | 67.32 | 61.73 | 30.85 | 42.61 | 57.72
GPT-4 | Ko | 70.41 | 67.32 | 74.32 | 60.64 | 54.55 | 79.64
En | 66.86 | 79.08 | 73.83 | 54.79 | 55.68 | 79.19

Table 14: Evaluation results for GPT-3.5 and GPT-4 on HAE-RAE Bench

|  | BoolQ |  | COPA |  | HellaSwag |  | SentiNeg
---|---|---|---|---|---|---|---|---
Model | Params | n=0 | n=5 | n=10 |  | n=0 | n=5 | n=10 |  | n=0 | n=5 | n=10 |  | n=0 | n=5 | n=10
Polyglot-Ko | 1.3B | 49.9 | 51.2 | 50.2 |  | 72.1 | 72.0 | 71.8 |  | 41.0 | 40.6 | 42.0 |  | 69.8 | 62.0 | 56.2
3.8B | 50.6 | 53.9 | 52.9 |  | 75.7 | 76.2 | 76.3 |  | 44.6 | 47.8 | 48.0 |  | 58.7 | 81.1 | 76.1
5.8B | 53.7 | 58.6 | 55.7 |  | 77.9 | 76.9 | 77.5 |  | 49.0 | 48.6 | 50.2 |  | 50.4 | 87.9 | 88.4
12.8B | 56.7 | 62.8 | 63.8 |  | 79.6 | 81.0 | 80.2 |  | 49.0 | 51.0 | 49.8 |  | 91.7 | 90.7 | 93.5
UMT5 | 3B | 50.2 | 50.2 | 50.2 |  | 52.4 | 53.4 | 51.1 |  | 32.0 | 31.0 | 28.2 |  | 52.4 | 49.9 | 49.6
13B | 50.2 | 50.3 | 50.3 |  | 57.9 | 58.2 | 57.3 |  | 36.2 | 32.0 | 31.6 |  | 56.9 | 51.9 | 46.4
Llama-2 | 7B | 50.9 | 58.8 | 55.8 |  | 56.1 | 58.1 | 58.2 |  | 41.8 | 43.2 | 43.2 |  | 48.9 | 58.2 | 57.2
13B | 50.6 | 74.2 | 77.1 |  | 59.1 | 64.1 | 63.1 |  | 41.6 | 44.8 | 42.6 |  | 50.6 | 73.8 | 85.9

Table 15: Evaluation results for KoBEST.

The evaluation results for KoBEST are presented in table 15. Polyglot-Ko
achieves the top scores in most settings, with exceptions in the 5-shot and
10-shot configurations for BoolQ.

###  9.2 GPT-3.5/4 Evaluation

In this section, we detail the prompts employed for the XLT evaluation of
GPT-3.5 and GPT-4 models. We also provide their performance metrics for each
downstream task in the HAE-RAE Bench and KoBEST datasets. The specific
instructions used for XLT with both models are presented below. Table 14 and
Table 13 contain the evaluation results for HAE-RAE Bench and KoBEST,
respectively.

> "Read the given question, and choose the most suitable answer. Answer your
> answer with the number of the answer you think to be correct.
> ### question: {question}
> ### options: (1) {option#1} (2) {option#3} (3) {option#3} (4) {option#4} (5)
> {option#5}
> ### answer:

###  9.3 HAE-RAE Bench Examples

Starting from Figure 4, we present examples for each task alongside their
translated versions.

Figure 4: An example from the Loan Words subset. Figure 5: A translated
example from the Loan Words subset. Figure 6: An example from the Rare Words
subset. Figure 7: A translated example from the Rare Words subset. Figure 8:
An example from the Standard Nomenclature subset. Figure 9: A translated
example from the Standard Nomenclature subset. Figure 10: An example from the
Reading Comprehension subset. Figure 11: A translated example from the Reading
Comprehension subset. Figure 12: An example from the General Knowledge subset.
Figure 13: A translated example from the General Knowledge subset. Figure 14:
An example from the History subset. Figure 15: A translated example from the
History subset.