# A remark on $\mathscr{C}^{\infty}$ definable equivalence

Anna Valette  Chair of Optimization and Control
Jagiellonian University
Łojasiewicza 6
30-348 Kraków, Poland<EMAIL_ADDRESS>and  Guillaume Valette
Institute of Mathematics
Jagiellonian University
Łojasiewicza 6
30-348 Kraków, Poland<EMAIL_ADDRESS>
###### Abstract.

We establish that if a submanifold $M$ of $\mathbb{R}^{n}$ is definable in
some o-minimal structure then any definable submanifold
$N\subset\mathbb{R}^{n}$ which is $\mathscr{C}^{\infty}$ diffeomorphic to $M$,
with a diffeomorphism $h:N\to M$ that is sufficiently close to the identity,
must be $\mathscr{C}^{\infty}$ definably diffeomorphic to $M$. The definable
diffeomorphism between $N$ and $M$ is then provided by a tubular neighborhood
of $M$.

###### Key words and phrases:

o-minimal structures, definable diffeomorphism

###### 2020 Mathematics Subject Classification:

Primary 32B20, 58C25; Secondary 03C64

##  1\. Introduction

The framework of o-minimal structures is well adapted to perform analysis as
well as geometry, and definable mappings have many finiteness properties that
are valuable for applications. It is however not always easy to construct
definable diffeomorphisms. For instance, smooth trivializations are often
generated by integration of a vector field [3] and the flow of a definable
vector field may fail to be definable in the same structure. Furthermore, M.
Shiota gave examples of algebraic smooth manifolds that are
$\mathscr{C}^{\infty}$ diffeomorphic but not Nash diffeomorphic (i.e. not
semialgebraically $\mathscr{C}^{\infty}$ equivalent) [5]. This points out that
$\mathscr{C}^{\infty}$ equivalence of definable manifolds does not yield
definable $\mathscr{C}^{\infty}$ equivalence. In this note, we show that such
pathologies cannot arise if the diffeomorphisms considered are sufficiently
close to the identity. Namely, we establish that if a submanifold $M$ of
$\mathbb{R}^{n}$ is definable in some o-minimal structure then any definable
submanifold $N\subset\mathbb{R}^{n}$ which is $\mathscr{C}^{\infty}$
diffeomorphic to $M$, with a diffeomorphism $h:N\to M$ that is sufficiently
close to the identity, must be definably $\mathscr{C}^{\infty}$ diffeomorphic
to $M$ (Theorem 2.1). The definable diffeomorphism between $N$ and $M$ is then
provided by a tubular neighborhood of $M$.

We briefly recall that an o-minimal structure expanding the real field
$(\mathbb{R},+,\cdot)$ is the data for every $n$ of a Boolean algebra
$\mathcal{D}_{n}$ of subsets of $\mathbb{R}^{n}$ containing all the algebraic
subsets of $\mathbb{R}^{n}$ and satisfying the following axioms:

  1. (1)

If $A\in\mathcal{D}_{m}$, $B\in\mathcal{D}_{n}$, then $A\times
B\in\mathcal{D}_{m+n}$;

  2. (2)

If $\pi:\mathbb{R}^{n}\times\mathbb{R}\to\mathbb{R}^{n}$ is the natural
projection and $A\in\mathcal{D}_{n+1}$, then $\pi(A)\in\mathcal{D}_{n}$;

  3. (3)

$\mathcal{D}_{1}$ is nothing else but all the finite unions of points and
intervals.

A set belonging to the structure $\mathcal{D}$ is called a definable set and a
map whose graph is in the structure $\mathcal{D}$ is called a definable map.
Given $B\subset\mathbb{R}^{k}$, we say that $(Z_{t})_{t\in B}$ is a definable
family of subsets of $\mathbb{R}^{n}$ if for each $t\in B$,
$Z_{t}\subset\mathbb{R}^{n}$ and $\bigcup_{t\in B}\\{t\\}\times
Z_{t}\in\mathcal{D}_{k+n}$ (in particular, $Z_{t}$ is definable for all $t$).
A family of mappings $(\varphi_{t})_{t\in B}$ is said to be definable if the
family of the respective graphs of the mappings $\varphi_{t}$ is a definable
family of sets.

Given a definable set $A\subset\mathbb{R}^{n}$, we denote by
$\mathcal{D}^{+}(A)$ the set of positive definable continuous functions on
$A$.

Given a mapping $f:A\to B$, with $A\subset\mathbb{R}^{n}$,
$B\subset\mathbb{R}^{k}$, we denote by $|f|$ the function which assigns to
$x\in A$ the number $|f(x)|$. Here we stress the fact that since $|f|$ is not
a real number but a function, this does not define a norm on the space of
mappings.

If $f$ is a differentiable mapping, we denote by $d_{x}f$ its derivative at
$x$ and we will write $|d_{x}f|$ for the norm of $d_{x}f$ (as a linear
mapping) derived from the euclidean norm $|.|$. We will write $|df|$ for the
function defined by $x\mapsto|d_{x}f|$. We then set

$|f|_{1}:=|f|+|df|.$

We will denote by $B(x,r)$ the open euclidean ball of radius $r$ centered at
$x$, and by $\overline{A}$ the closure of $A$ in the euclidean topology.

##  2\. Definable diffeomorphisms via retractions

Let us recall that given a definable $\mathscr{C}^{p}$ submanifold $M$ of
$\mathbb{R}^{n}$, $p\geq 2$, there is a definable neighborhood $U$ of $M$ in
$\mathbb{R}^{n}$ and a definable retraction $r:U\to M$ such that for all $x\in
U$, $r(x)$ is the point that realizes the distance from $x$ to $M$. The vector
$(x-r(x))$ is then orthogonal to the tangent space to $M$ at $r(x)$ and we say
that $(U,r)$ is a tubular neighborhood of $M$. The mapping $r$ is at least
$\mathscr{C}^{p-1}$ and, if $M$ is $\mathscr{C}^{\infty}$, then so is $r$ [4],
[6, Proposition 2.4.1], see also [2, Theorem 6.11].

######  Theorem 2.1.

Let $M\subset\mathbb{R}^{n}$ be a closed definable $\mathscr{C}^{2}$
submanifold and let $(U,r)$ be a definable tubular neighborhood of $M$. There
exists $\varepsilon\in\mathcal{D}^{+}(\mathbb{R}^{n})$ such that if
$N\subset\mathbb{R}^{n}$ is any definable $\mathscr{C}^{2}$ submanifold for
which there exists a $\mathscr{C}^{1}$ diffeomorphism $h:N\to M$ satisfying
$|h-id_{|_{N}}|_{1}<\varepsilon$ then $N$ is contained in $U$ and the
restriction $r_{|_{N}}:N\to M$ is a $\mathscr{C}^{1}$ diffeomorphism.

###### Proof.

Let $N\subset\mathbb{R}^{n}$ be a definable $\mathscr{C}^{2}$ submanifold and
$h:N\to M$ a diffeomorphism such that $|h(x)-x|_{1}<\varepsilon(x)$ for some
$\varepsilon\in\mathcal{D}^{+}(\mathbb{R}^{n})$. We assume $\varepsilon<1$ and
we will put extra requirements on $\varepsilon$ on the way. For
$\delta\in\mathcal{D}^{+}(M)$ define

(2.1) $U_{\delta}:=\\{x\in U\;:dist(x,M)<\delta(r(x))\\},$

where $dist(x,M)$ is the euclidean distance of $x$ to $M$. If $\delta$ is
sufficiently small we have $\overline{U_{\delta}}\subset U$. Replacing $U$
with $U_{\delta}$ for some small $\delta$, we can assume that $|d_{x}r|$ is
bounded on bounded sets. Moreover, the assumption
$|h-id_{|_{N}}|_{1}<\varepsilon$ entails $dist(x,M)<\varepsilon(x)$ for every
$x$ in $N$ and therefore $N\subset U_{\delta}$ if
$\varepsilon(x)\leq\eta(x):=\inf\\{\frac{\delta(z)}{2},\,|z|<|x|+1\\}$.

Let $x\in N$ and take a unit vector $u\in T_{x}N$. Put $v:=d_{x}h(u)$. By
assumption, we have $|u-v|<\varepsilon(x)$. Observe that

  1. (1)

$|d_{x}r(u)-d_{x}r(v)|\leq|u-v||d_{x}r|$,

  2. (2)

$|d_{h(x)}r(v)-u|=|v-u|$ (as $r$ is the identity on $M$),

  3. (3)

by continuity of $x\mapsto d_{x}r$, there is a definable function $\mu(x)>0$
such that if $|x-h(x)|<\mu(x)$ then $|d_{x}r(v)-d_{h(x)}r(v)|<\frac{1}{8}$.

By the above, if
$\varepsilon(x)<\min\\{\eta(x),\mu(x),\frac{1}{8(|d_{x}r|+1)}\\}$ we have

$\displaystyle|d_{x}r(u)-u|$ $\displaystyle\leq$
$\displaystyle|d_{x}r(u)-d_{x}r(v)|+|d_{h(x)}r(v)-u|+|d_{x}r(v)-d_{h(x)}r(v)|$
$\displaystyle<$
$\displaystyle\varepsilon(x)(|d_{x}r|+1)+\frac{1}{8}<\frac{1}{4},$

which shows that $d_{x}(r_{|N})$ is an isomorphism, implying that $r$ induces
a local diffeomorphism on $N$.

To prove that it is a diffeomorphism on $N$, we start by showing that the
restriction $r_{|_{N}}$ is proper. Observe that $N$ is closed. Indeed, take
$(x_{i})\subset N$ such that $x_{i}\to x\in\overline{N}$. As $h(x_{i})$ is
bounded and $M$ is closed there is a $y\in M$ such that $h(x_{i})\to y$
(extracting some subsequence if necessary), which implies $x=h^{-1}(y)\in N$,
yielding $N$ is closed. Now, notice that if $|x_{i}|\to+\infty$ then, as
$|r(x_{i})-x_{i}|\leq|h(x_{i})-x_{i}|<\varepsilon(x_{i})$, the sequence
$r(x_{i})$ must go to infinity as well. Since $N$ is closed, this shows that
the restriction of $r$ to $N$ is proper.

Hence, by Ehresmann’s theorem $r_{|_{N}}$ is a locally trivial fibration above
every connected component of $M$. We will show that it is indeed one-to-one.
For this purpose, let us fix any $x$ in $M$. There is $q>0$ such that
$V_{x}:=B(x,q)\cap M$ is simply connected and hence $r_{|_{N}}$ is trivial
above $V_{x}$. We first show by way of contradiction that if $\varepsilon$ is
small enough (depending on $x$) then $r^{-1}(x)\cap N$ must be reduced to a
single point. Write $r_{|_{N}}^{-1}(x)=\\{x_{1},\dots,x_{k}\\}$, suppose
$k\geq 2$, and notice that

$r_{|_{N}}^{-1}(V_{x})=V_{1}\cup\dots\cup V_{k}$

is a disjoint union of neighborhoods $V_{i}$ of $x_{i}$ in $N$ respectively.
Take now $y\in B(x,q/8)\cap M$ and set $\varepsilon_{0}:=\sup_{K}\varepsilon$,
where $K$ is some compact neighborhood of $x$ comprising all the sets that we
are considering. Let us observe that

$|h^{-1}(y)-x|\leq|h^{-1}(y)-y|+|y-x|<\varepsilon_{0}+q/8,$

which means that for $\varepsilon_{0}<q/16$, we have $h^{-1}(y)\in B(x,q/4)$,
and consequently

$h^{-1}(B(x,q/8)\cap M)\subset B(x,q/4).$

Moreover, since

$|r(h^{-1}(y))-x|\leq|r(h^{-1}(y))-h^{-1}(y)|+|h^{-1}(y)-y|+|y-x|<q/16+q/16+q/8$

we have that $r(h^{-1}(y))\in B(x,q/4)$, and hence

$h^{-1}(B(x,q/8)\cap M)\subset V_{1}\cup\dots\cup V_{k}.$

Since the last union is disjoint, we can assume that

$h^{-1}(B(x,q/8)\cap M)\subset V_{1}.$

Furthermore, we have

$|h(x_{2})-x|<|h(x_{2})-x_{2}|+|x_{2}-x|<2\varepsilon_{0}<q/8,$

and hence $x_{2}\in h^{-1}(B(x,q/8)\cap M)$, which contradicts the fact that
$V_{1}$ and $V_{2}$ are disjoint, establishing that $r^{-1}(x)\cap N$ must be
reduced to a single point.

Choose now one point in each connected component of $M$, say
$z_{1},\dots,z_{l}$. By the above, for $\varepsilon$ small enough, the set
$r^{-1}(z_{i})\cap N$ is reduced to a single point for all $i\leq l$. Since
$r_{|_{N}}:N\to M$ is a locally trivial fibration, it must be one-to-one.

We turn to show that $r_{|N}$ is onto. As it is a locally trivial fibration on
each connected component of $M$, it suffices to show that $r(N)$ contains at
least one point in every connected component. Take any $x\in M$ and observe
that, for $\varepsilon$ small enough, $h^{-1}(x)\in U$ and $r(h(x))$ is a
point close to $x$, which therefore must belong to the same connected
component of $M$ as $x$, if $\varepsilon$ is sufficiently small. ∎

######  Definition 2.2.

Let $M$ be a $\mathscr{C}^{k}$ submanifold of $\mathbb{R}^{n}$, $k\geq 2$
(possibly infinite). Fix any $\varepsilon\in\mathcal{D}^{+}(M)$ and a positive
integer $p\leq k$. A definable deformation of $M$ is a definable family
$(Z_{t})_{t\in[0,1]}$ of $\mathscr{C}^{p}$ submanifolds
$Z_{t}\subset\mathbb{R}^{n}$ with $M=Z_{0}$. A deformation
$(Z_{t})_{t\in[0,1]}$ is $(\varepsilon,\mathscr{C}^{p})$ trivial if there
exists a family of $\mathscr{C}^{p}$ diffeomorphisms $\varphi_{t}:M\to Z_{t}$,
$t\in[0,1]$, $\mathscr{C}^{p}$ with respect to $t$ and satisfying
$\varphi_{0}(x)=x$ for all $x$ as well as:

(2.2)
$|x-\varphi_{t}(x)|<\varepsilon(x)\quad\mbox{and}\quad|u-d_{x}\varphi_{t}(u)|<\varepsilon(x),$

for every $x\in M=Z_{0}$ and every unit vector $u\in T_{x}M$. When
$(\varphi_{t})_{t\in[0,1]}$ is a definable family of mappings, we say that
$(Z_{t})_{t\in[0,1]}$ is definably $(\varepsilon,\mathscr{C}^{p})$ trivial.

######  Corollary 2.3.

Let $M\subset\mathbb{R}^{n}$ be a closed, definable $\mathscr{C}^{k}$, $k\geq
2$ (possibly infinite), submanifold and let
$\varepsilon\in\mathcal{D}^{+}(M)$. There exists $\delta\in\mathcal{D}^{+}(M)$
such that any $(\delta,\mathscr{C}^{k-1})$ trivial definable deformation of
$M$ is $(\varepsilon,\mathscr{C}^{k-1})$ definably trivial.

###### Proof.

Let $\varepsilon\in\mathcal{D}^{+}(M)$ and take a tubular neighborhood $(U,r)$
of $M$. Take some $\delta$ sufficiently small for $U$ to contain the closure
of $U_{\delta}$ (see (2.1)). The derivative of $r$ is then uniformly
continuous on every bounded subset of $U_{\delta}$. Let $(Z_{t})_{t\in[0,1]}$
be a $(\delta,\mathscr{C}^{k})$ trivial deformation, with corresponding family
of diffeomorphisms $\varphi_{t}:M\to Z_{t}$. If $\delta$ is sufficiently
small, by Theorem 2.1 (applied with $h:=\varphi_{t}^{-1}$ for each $t$), the
restriction $r_{t}$ of $r$ to each $Z_{t}$ induces a definable diffeomorphism.
We are going to verify that for $\delta$ sufficiently small we have for all
$x\in Z_{t}$ and $u\in T_{x}Z_{t}$ unit vector (for any $t$):

(2.3)
$|x-r_{t}(x)|<\varepsilon(r_{t}(x))\quad\mbox{and}\quad|u-d_{x}r_{t}(u)|<\varepsilon(r_{t}(x)).$

Before proving these two inequalities, let us make it clear that this yields
the desired fact. We may assume $\varepsilon<\frac{1}{2}$. Setting
$y=r_{t}(x)\in M$ and $v=\frac{d_{x}r_{t}(u)}{|d_{x}r_{t}(u)|}\in T_{y}M$ (if
$x\in Z_{t}$ and $u\in T_{x}Z_{t}$ is a unit vector), (2.3) immediately
entails $|d_{x}r_{t}(u)|\geq\frac{1}{2}$ so that

$|(r_{t})^{-1}(y)-y|<\varepsilon(y)\quad\mbox{and}\quad|d_{y}(r_{t})^{-1}(v)-v|<2\varepsilon(y),$

showing (2.2) for $r_{t}^{-1}$ (up to the constant $2$).

We can assume that $\delta<1$. Observe that for such $x$

(2.4)
$|r_{t}(x)-x|=dist(x,M)\leq|x-\varphi_{t}^{-1}(x)|\overset{(\ref{eq_istopie_approx})}{\leq}\delta(\varphi_{t}^{-1}(x))$

and hence

(2.5) $|r_{t}(x)-\varphi_{t}^{-1}(x)|\leq 2\delta(\varphi_{t}^{-1}(x))\leq 2.$

For $\delta^{\prime}(y):=\sup\\{3\delta(z)\;:\ z\in M\cap B(y,2)\\},y\in M$,
we obtain from (2.4) and (2.5) that

(2.6) $|r_{t}(x)-x|\leq\delta^{\prime}(r_{t}(x)).$

Now, as $x\mapsto d_{x}r$ is uniformly continuous on bounded sets, if $\delta$
is small enough we have on $U_{\delta}$ for each unit vector $u\in
T_{r_{t}(x)}M$:

(2.7) $|d_{x}r(u)-u|=|d_{x}r(u)-d_{r(x)}r(u)|\leq\varepsilon(r_{t}(x)).$

This is almost the desired estimate (together with (2.6)). The problem is that
we need to have such an estimate for $u\in T_{x}Z_{t}$, $x\in Z_{t}\subset
U_{\delta}$. We thus are going to estimate the distance between $T_{x}Z_{t}$
and $T_{r_{t}(x)}M$ (see (2.8) and (2.9) below).

Denote by $\mathbb{G}^{m}_{n}$, $m=\dim M$, the Grassmannian of
$m$-dimensional linear subspaces of $\mathbb{R}^{n}$, that we endow with the
metric

$\rho(P,Q):=\sup_{a\in P,|a|=1}\inf_{b\in Q}|a-b|.$

Observe that

(2.8)
$\rho(T_{x}Z_{t},T_{\varphi^{-1}_{t}(x)}M)\overset{(\ref{eq_istopie_approx})}{<}\delta(\varphi_{t}^{-1}(x))<\delta^{\prime}(r_{t}(x)),$

by definition of $\delta^{\prime}$ and (2.5). Moreover, since the tangent
bundle of $M$ is at least $\mathscr{C}^{1}$, by (2.5), for $\delta$
sufficiently small, we have

(2.9) $\rho(T_{r_{t}(x)}M,T_{\varphi^{-1}_{t}(x)}M)<\varepsilon(r_{t}(x)).$

By (2.8) and (2.9), we get for $\delta$ small enough

$\rho(T_{x}Z_{t},T_{r_{t}(x)}M)<2\varepsilon(r_{t}(x)),$

which with (2.4) and (2.7) yields (2.3). ∎

### Acknowledgements

Research supported by the National Science Center (Poland) under grant number
2021/43/B/ST1/02359.

## References

  * [1]
  * [2] M. Coste, _An Introduction to o-minimal Geometry_ , Istituti editoriali e poligrafici internazionali, 2000.
  * [3] C. Ehresmann, _Les connexions infinitésimales dans un espace fibré différentiable_ , Colloque de topologie (espaces fibrés), Bruxelles, 1950, Georges Thone, Liège; Masson et Cie., Paris, 29–55.
  * [4] J.-B. Poly & G. Raby, _Fonction distance et singularités_ , Bull. Sci. Math. (2), 108 (1984), 187–195.
  * [5] M. Shiota, _Classification of Nash manifolds_ , Ann. Inst. Fourier, 33 (1983), 209–232.
  * [6] G. Valette, _On subanalytic geometry_ , survey available at http://www2.im.uj.edu.pl/gkw/sub.pdf.


# Influence of Roughening Transition on Magnetic Ordering

Nalina Vadakkayil  Theoretical Sciences Unit and School of Advanced Materials,
Jawaharlal Nehru Centre for Advanced Scientific Research, Jakkur P.O.,
Bangalore 560064, India  Sanat K. Singha  Theoretical Sciences Unit and School
of Advanced Materials, Jawaharlal Nehru Centre for Advanced Scientific
Research, Jakkur P.O., Bangalore 560064, India  Assam Energy Institute, Centre
of Rajiv Gandhi Institute of Petroleum Technology, Sivasagar 785697, India
Subir K. Das<EMAIL_ADDRESS>Theoretical Sciences Unit and School of
Advanced Materials, Jawaharlal Nehru Centre for Advanced Scientific Research,
Jakkur P.O., Bangalore 560064, India

###### Abstract

In the literature of magnetic phase transitions, in addition to a critical
point, existence of another special point has been discussed. This is related
to the broadening of interface between two different ordering phases and is
referred to as the point of roughening transition. While there exists good
understanding on equilibrium properties associated with this transition,
influence of this on nonequilibrium dynamics has not been investigated. In
this paper we present comprehensive results, from Monte Carlo simulations, on
coarsening dynamics in a system, over a wide range of temperature, in space
dimension $d=3$, for which there exists roughening transition at a nonzero
temperature. State-of-the-art analysis of the simulation data, on structure,
growth and aging, shows that the onset of unexpected glass-like slow dynamics
in this system, that has received much attention in recent times, for quenches
to zero temperature, actually occurs at this transition point. This
demonstrates an important structure-dynamics connection in phase-ordering
dynamics. We compare the key results with those from $d=2$, for which there
exists no non-zero roughening transition temperature. Absence of the above
mentioned anomalous features in the latter dimension places our conjecture on
the role of roughening transition on a firmer footing.

##  I Introduction

Over past several decades there has been significant interest spirin1 ;
spirin2 ; oliveira ; kondrat ; arenzon ; olejarz1 ; olejarz2 ; olejarz3 ;
corberi ; blanchard ; blanchard2 ; amar ; shore ; cueille ; das1 ; das2 ; das3
; denholm ; godriche ; mullick ; yu ; cugliandolo ; liu ; ohta ; yeung2  in
the understanding of ordering dynamics following quenches of paramagnetic
configurations to the ferromagnetic region by crossing the critical
temperature mfisher1  $T_{c}$. In recent times glass-like dynamics in a
popular ordering system, despite the absence of in-built frustration,
following quenches to the final temperature $T_{f}=0$, drew attention olejarz1
; olejarz2 ; olejarz3 ; blanchard ; blanchard2 ; shore ; cueille ; das1 ; das2
; das3 . This slow dynamics perhaps is due to non-conventional structure
formation. The observation is striking and it is unknown whether such
unexpected behavior corberi ; das1 ; das2 ; das3  is specific to $T_{f}=0$. It
is possible that the origin is at the roughening transition beijern , that
occurs at a much higher temperature $T_{R}$ $(<T_{c})$, given that below
$T_{R}$ interfaces are sharp. Knowledge of this is crucial not only in the
understanding of this intriguing fact but also in establishing structure-
dynamics coupling in the general context of growth phenomena. Interestingly,
the consequences of the presence or absence of rough interfaces, of importance
in real systems, in such nonequilibrium phenomena has not been investigated,
though the equilibrium aspects of roughening transition is reasonably well
explored.

Some of the key aspects of ordering dynamics bray ; puri ; dfisher ; yeung1 ;
henkel ; lorenz ; liu ; ohta ; das4 ; yeung2 ; huse ; das5 ; allen ; landau
are: i) self-similarity and scaling property of structure, ii) growth of the
latter, and iii) related aging. The structure is typically probed via the two-
point equal time $(t)$ correlation function bray , which, for a spin system,
reads $C(r,t)=\langle S_{i}(t)S_{j}(t)\rangle-\langle S_{i}(t)\rangle\langle
S_{j}(t)\rangle$, $S_{i}$ and $S_{j}$ representing orientations of spins or
atomic magnets at sites $i$ and $j$, located $r$ distance apart. It is also
customary to study the Fourier transform of $C(r,t)$, the structure factor,
$S(k,t)$, $k$ being the wave number bray . The latter has direct experimental
relevance. These quantities obey certain scaling properties when the growth is
self-similar. E.g., in simple situations, when structures are non-fractal,
$C(r,t)$ satisfies bray  $C(r,t)\equiv\tilde{C}(r/\ell)$, $\ell$ being the
average domain size or characteristic length scale of the growing system at
time $t$ and $\tilde{C}(x)$ a time-independent master function. In such a
situation $\ell$ is expected bray  to grow as $\sim t^{\alpha}$. A power-law
behavior is expected for aging phenomena also. In the latter case the
autocorrelation function puri ; dfisher , $C_{\textrm{ag}}(t,t_{w})=\langle
S_{i}(t)S_{i}(t_{w})\rangle-\langle S_{i}(t)\rangle\langle
S_{i}(t_{w})\rangle$, should scale as $\sim$ $(\ell/\ell_{w})^{-\lambda}$, in
the asymptotic limit when $\ell>>\ell_{w}$. Here, $t_{w}$ $(\leq t)$ is the
waiting time or age of the system and $\ell_{w}$ is the value of $\ell$ at
$t=t_{w}$.

For uniaxial ferromagnets one expects bray ; allen  $\alpha=1/2$. The
structure in this case is supposed to be described by the Ohta-Jasnow-Kawasaki
(OJK) function bray ; ohta : $C(r,t)=\frac{2}{\pi}\sin^{-1}[\exp(-r^{2}/Dt)]$,
$D$ being a constant. Note that while the form of $C(r,t)$ and the value of
growth exponent $\alpha$ are independent of space dimension $d$, aging
exponent $\lambda$ changes with it. The values of $\lambda$ for this ordering,
as obtained by Liu and Mazenko (LM) liu , are expected to be $\simeq
1.67=\lambda_{3}^{\rm{LM}}$ in $d=3$, whereas in $d=2$, it is
$\lambda_{2}^{\rm{LM}}\simeq 1.29$. Unless otherwise mentioned, in the rest of
the paper all our discussions are for $d=3$.

While these theoretical predictions were confirmed via the Monte Carlo landau
simulations of the Ising model, for moderately high values of $T_{f}$,
striking deviations were reported for $T_{f}=0$, in $d=3$. In the latter case,
several works concluded that $\alpha=1/3$ or the growth is even slower. Most
recently it was reported that the OJK function does not das3  describe the
pattern at $T_{f}=0$. Furthermore, $\lambda$ was also estimated das3  to be
much weaker than $\lambda_{3}^{\rm{LM}}$. Thorough investigations, we believe,
are necessary, in order to arrive at a complete and correct picture. It needs
to be understood if such anomalies bear any connection with any other special
point. If such a special point turns out to be that of the roughening
transition, important relation concerning structure and dynamics das6  can be
established in the nonequilibrium context. Our study clearly suggests that the
above mentioned anomalous features are not specific to $T_{f}=0$. The onset of
the anomalies occurs at the roughening transition. The key results have been
verified by sophisticated finite-size scaling analysis das4 ; mfisher2 ;
das_pre_fss . We believe that these will inspire novel investigations, thereby
explaining intriguing dynamical phenomena in the nonequilibrium domain.

##  II Model and Methods

We choose $J>0$ in the Ising Hamiltonian landau ; mfisher1
$H=-J\sum_{<ij>}S_{i}S_{j}$, where $S_{i}$ and $S_{j}$ can take values $+1$
and $-1$, corresponding to up and down orientations of the atomic magnets. We
study this model on a simple cubic lattice, having $T_{c}\simeq 4.51J/k_{B}$
landau , where $k_{B}$ is the Boltzmann constant. For the limited set of
results in $d=2$, we considered the square lattice. Note that in this case
$T_{c}\simeq 2.27J/k_{B}$ landau . In $d=3$, the value of $T_{R}$ for this
model is $\simeq 2.57J/k_{B}$ beijern , whereas a non-zero roughening
transition temperature does not exist in $d=2$.

Moves in our MC simulations were tried by randomly choosing a spin and
changing its sign landau ; glauber . These were accepted by following standard
Metropolis criterion landau . Unit of time in our simulations is a MC step
(MCS) that consists of $L^{d}$ trial moves, $L$ being the linear dimension of
a cubic or a square box, in units of the lattice constant.

All our results are presented after averaging over runs with 50 independent
random initial configurations, with $L=512$. Periodic boundary conditions were
applied in all possible directions. Average domain lengths were measured, from
the simulation snapshots, as the first moments das5  of the domain-size
distribution function, in which length of a domain was estimated as the
distance between two successive interfaces along any Cartesian direction.
Results on the structure and growth were obtained after appropriately
eliminating the thermal noise in the snapshots via a majority spin rule das5 .
We repeat, unless otherwise stated, the results are from $d=3$.

##  III Results

In Fig. 1 we show $\ell$ versus $t$ plots for several values of $T_{f}$. For
$T_{f}=0$ few early works were suggestive of olejarz2 ; amar ; shore ; cueille
an exponent $\alpha=1/3$ or even slower growth. Similar quantitative behavior
is seen here as well. However, at very late time a crossover das1 ; das3  of
the exponent to a higher value, viz., $\alpha=1/2$, can be appreciated. The
very early works could not capture this, either due to consideration of small
systems or simulations over short periods, owing, perhaps, to inadequate
computational resources. As can be seen, such a slow looking early growth is
not unique to $T_{f}=0$. The data sets for nonzero $T_{f}$ values also exhibit
similar trend. However, with the increase of $T_{f}$ departure from this slow
behavior occurs earlier, the $1/3$-like regime ceasing to exist for
$T_{f}=T_{R}=2.57$. At this stage, it is worth warning that the early
evolution should not be taken seriously, at the quantitative level. This is
because, during this period satisfaction of the scaling property of the
correlation function is not observed, as demonstrated below.

Figure 1: Average domain lengths, $\ell(t)$, are plotted versus time, on a
log-log scale. Results from different final temperatures are presented. The
solid lines are power-laws. The values of the exponents are mentioned. Figure
2: (a) Two-point equal time correlation functions, $C(r,t)$, are plotted
versus the scaled distance $r/\ell$, for $T_{f}=0.5$. Data from few different
times are shown. (b) Same as (a) but here we have shown $C(r,t)$ from
different final temperatures. In each of the cases we have chosen $t=5000$
that fall in the scaling regimes. In both (a) and (b) the continuous lines
represent the Ohta-Jasnow-Kawasaki (OJK) function. The correlation functions
have been plotted in such a way that there exists good collapse in the early
abscissa range. Figure 3: (a) $C_{\textrm{ag}}(t,t_{w})$, the autocorrelation
functions, are plotted versus $\ell/\ell_{w}$, on a log-log scale. Data from
different $t_{w}$, for $T_{f}=1.5$, are included. The solid line is a power-
law with $\lambda=\lambda_{3}^{\rm{LM}}=1.67$. (b) The instantaneous exponents
$\lambda_{i}$ are shown as a function of $\ell_{w}/\ell$, for the same final
temperature. The arrow-headed line is a linear extrapolation to the
$\ell/\ell_{w}=\infty$ limit, done by excluding the late time finite-size
affected as well as early time domain-magnetization relaxation parts. (c) Here
we have shown $\lambda_{i}$, as a function of $\ell_{w}/\ell$, for a few
different values of $T_{f}$. In each of the cases $t_{w}$ belongs to the
scaling regime. The arrow-headed lines are linear guides to the eyes. (d)
Finite-size scaling plot of $C_{\rm{ag}}(t,t_{w})$ for $T_{f}=0.75$. The solid
line there corresponds to a power-law with the value of the exponent mentioned
near the line.

In Fig. 2(a) we show plots of $C(r,t)$, from different times, by scaling the
distance axis by a characteristic length, extracted by exploiting the
satisfaction of scaling of $C(r,t)$ at small distances, for $T_{f}=0.5$. It
appears that the collapse starts only from $t\gtrsim 1000$, approximately the
time since when departure to $\alpha=1/2$ behavior starts. This general
picture is true for other low temperatures also. In Fig. 2(b) we have shown
$C(r,t)$, again versus $r/\ell$, from the scaling regimes of different $T_{f}$
values. Interestingly, $\tilde{C}(r/\ell)$ at different $T_{f}$ values do not
agree with each other. However, with the increase of $T_{f}$ the agreement
with the OJK function $[\tilde{C}_{\rm{OJK}}(r/\ell)]$ ohta  keeps getting
better. This observation suggests that perhaps there exists a special
temperature $T_{\rm{sp}}$ ($<T_{c}$), beyond which the coarsening dynamics is
more unique than below it. In fact for $T_{f}=T_{R}$ the agreement between
simulation data and the OJK function is quite well. We will return to this
central theme after discussion of the basic results on aging.

Fig. 3(a) shows plots of $C_{\textrm{ag}}(t,t_{w})$, with the variation of
$\ell/\ell_{w}$. The value of $T_{f}$ for this representative case is set at
$1.5$. Data sets from a few different waiting times are shown. Good collapse
of data is visible for the considered values of $t_{w}$. However, there exist
deviations from the master curve, for $\ell/\ell_{w}>>1$. These are related to
finite-size effects das4 . Decay in the finite-size unaffected regime does not
appear consistent with the LM value liu  – see the disagreement with the solid
line. In Fig. 3(b) we show the instantaneous exponent dfisher ; das4
$\lambda_{i}$ $[=-\textrm{d}\ln
C_{\textrm{ag}}(t,t_{w})/\textrm{d}\ln(\ell/\ell_{w})]$ as a function of
$\ell_{w}/\ell$, for multiple choices of $t_{w}$ lying in the scaling regime.
The data sets appear linear in the finite-size unaffected regimes. Note that
at early time relaxation of domain magnetization interferes and should be
discarded from the process of estimation of $\lambda$. A linear extrapolation
to $\ell/\ell_{w}=\infty$ provides $\lambda\simeq 1.5$. Given that the above
quoted number lies between $\lambda_{3}^{\rm{LM}}$ and $\lambda(T_{f}=0)$,
being significantly different from each of these, one gets a strong hint on
the presence of a special point.

In Fig. 3(c) we show plots of $\lambda_{i}$, as a function of $\ell_{w}/\ell$,
for few different values of $T_{f}$. Here we discarded the parts corresponding
to finite-size effects and equilibration of domain magnetization. Furthermore,
in each of the cases the results are from well inside the scaling regimes of
$t_{w}$. The arrow-headed lines are related to the estimations of the values
of $\lambda$, from linear extrapolations to the $\ell_{w}/\ell=\infty$ limit.
Clearly, $\lambda$ depends strongly on $T_{f}$. The accuracy of these
estimates is validated by the independent quantifications of $\lambda$ via a
finite-size scaling method das4 ; das_pre_fss . A representative exercise
related to this is shown in Fig. 3(d), for $T_{f}=0.75$. In this figure $Y$ is
a $t_{w}$-independent scaling function and $y$ is a dimensionless scaling
variable. Note that here we have avoided studying systems of different sizes,
contrary to the standard practice in the literature of such analysis. Instead,
we have obtained collapse of data from different $t_{w}$ values. Note that
when $t_{w}$ is varied a system has different effective sizes to grow further.
Details of the scaling construction is provided below.

Figure 4: (a) Plots are shown by comparing $\Delta\lambda$ and $\Delta I$ with
the variation of $T_{f}$. (b) Plots of $\Delta\lambda$ and $\Delta I$ as a
function of $1/T_{f}$. The locations of $T_{R}$ are marked inside the frames.

The behavior of $\lambda_{i}$ in Fig. 3(b) and Fig. 3(c) suggest
$\lambda_{i}=\lambda-B/x$, with $x=\ell/\ell_{w}$ and $B$ being a constant, in
the finite-size unaffected late time regime. This leads to a form das4 ;
das_pre_fss  $C_{\textrm{ag}}(t,t_{w})=Ae^{-B/x}x^{-\lambda}$. By taking
$y=L/\ell$ as a scaling variable and $y_{w}=L/\ell_{w}$, a finite-size scaling
function can be written as das3
$Y=C_{\textrm{ag}}(t,t_{w})e^{By/y_{w}}y_{w}^{\lambda}$, where $Y$ contains a
factor $y^{\lambda}$. When results from different $t_{w}$ are plotted, for
optimum choices of the unknown parameters, including $\lambda$, there will be
collapse of data sets that will satisfy the expected $y^{\lambda}$ behavior at
large $y$. This is demonstrated in Fig. 3(d) for $T_{f}=0.75$. Here the
collapse is obtained for $\lambda=1.26$, the number being consistent with the
value that was suggested by the exercise in Fig. 3(c). Next we quantify the
special temperature from a more systematic study.

Figure 5: (a) Plots of scaled two-point equal time correlation functions,
$C(r,t)$, from $d=2$, for different $T_{f}$ values. The time has been chosen
from the scaling regime. The continuous line represents the OJK function. (b)
The instantaneous exponents, $\lambda_{i}$, are plotted as a function of
$\ell_{w}/\ell$, for different $T_{f}$ values. The solid line is a common
linear extrapolation of the data sets. The arrow-headed horizontal line points
to the LM value of $\lambda$ in $d=2$.

In Fig. 4(a) we show $\Delta\lambda=\lambda_{3}^{\rm{LM}}-\lambda(T_{f})$, as
a function of $T_{f}$. Given that the general expectation is
$\lambda_{3}^{\rm{LM}}$, it is meaningful to look at the stated difference.
There appears to be a nice convergence of the data set to zero as
$T_{f}\rightarrow T_{\rm{R}}\simeq 2.57$. With respect to the deviation of
$\tilde{C}(r/\ell)$ from the OJK form ohta , that we observed above, there may
also be a similar trend. In this case an appropriate quantity to consider is
$\Delta I=\int dr[\tilde{C}(r/\ell)-\tilde{C}_{\rm{OJK}}(r/\ell)]$. Here note
that there exists an LM form for $C(r,t)$ as well liu . However, this
practically overlaps with the OJK function. It will be interesting to see if
$\Delta I$ approaches zero at the same $T_{f}$ as in the case of
$\Delta\lambda$. Thus, in Fig. 4(a) we have included the $T_{f}$-dependence of
$\Delta I$ as well. The presented data sets are in nice agreement with each
other, over a wide range of temperature, within a factor – see Fig. 4(b) for a
clearer convergence to zero for $T_{f}\rightarrow T_{R}$.

In Fig. 5 we show analogous results from $d=2$. Fig. 5(a) contains results for
the scaled $C(r,t)$ and in Fig. 5(b) we have shown data for $\lambda_{i}$, as
a function of $\ell_{w}/\ell$. For each of the cases results from a wide range
of $T_{f}$ are included. The anomalies present in $d=3$ are clearly absent in
this case. No detectable $T_{f}$-dependence can be observed. The theoretical
expectations are satisfied over the whole range of $T_{f}$. Recall that in
this dimension, a nonzero roughening transition temperature does not exist for
this model.

##  IV Conclusions

From extensive Monte Carlo simulations landau  we have presented results on
nonequilibrium dynamics in the Glauber landau ; glauber  Ising model. This
mimics ordering in uniaxial ferromagnets. Our quantitative analysis of data
from space dimension $d=3$ on structure, growth and aging, over a wide range
of temperature below the critical point, suggests that the low temperature
behavior is anomalous.

We show that that the anomalies are not unique to the case of zero temperature
quench, as was previously thought. Various quantities exhibit zero-
temperature-like trend till a certain nonzero value of $T_{f}$. Above this
temperature, behavior of all the aspects become consistent with various
theoretical expectations bray ; liu ; ohta ; allen . This transition or
special temperature coincides with that of the roughening transition beijern .
Such a conclusion appears more meaningful from the fact that these anomalies
are absent in $d=2$ and for this dimension roughening transition temperature
is zero.

To understand the anomalies, more detailed theoretical investigations by
exploiting the well travelled auxiliary field ansatz should be carried out
yeung . Recently, it was shown mpemba_pccp  that the ordering dynamics in the
Glauber Ising model exhibits Mpemba effect. It needs to be further
investigated in light of the present observation.

SKD acknowledges previous collaborations on similar matter with S.
Chakraborty, S. Majumder and J. Midya. The authors are grateful to SERB, DST,
INDIA for support via Grant No. MTR/2019/001585.

## References

  * (1) V. Spirin, P.L. Krapivsky, and S. Redner, Phys. Rev. E 63, 036118 (2001).
  * (2) V. Spirin, P.L. Krapivsky, and S. Redner, Phys. Rev. E 65, 016119 (2001).
  * (3) P.M.C. de Oliveira, C.M. Newman, V. Sidoravicious, and D.L. Stein, J. Phys. A 39, 6841 (2006).
  * (4) G. Kondrat and K. Sznajd-Weron, Phys. Rev. E 79, 011119 (2009).
  * (5) J.J. Arenzon, A.J. Bray, L.F. Cugliandolo, and A. Sicilia, Phys. Rev. Lett. 98, 145701 (2007).
  * (6) J. Olejarz, P.L. Krapivsky, and S. Redner, Phys. Rev. Lett. 109, 195702 (2012).
  * (7) J. Olejarz, P.L. Krapivsky, and S. Redner, Phys. Rev. E 83, 051104 (2011).
  * (8) J. Olejarz, P.L. Krapivsky, and S. Redner, Phys. Rev. E 83, 030104 (2011).
  * (9) F. Corberi, E. Lippiello, and M. Zannetti, Phys. Rev. E 78, 011109 (2008).
  * (10) T. Blanchard, F. Corberi, L.F. Cugliandolo, and M. Picco, Europhys. Lett. 106, 66001 (2014).
  * (11) T. Blanchard, L.F. Cugliandolo, M. Picco, and A. Tartaglia, J. Stat. Mech. 2017, P113201.
  * (12) J.G. Amar and F. Family, Bull. Am. Phys. Soc. 34, 491 (1989).
  * (13) J.D. Shore, M. Holzer, and J.P. Sethna, Phys. Rev. B 46, 11376 (1992).
  * (14) S. Cueille and C. Sire, J. Phys. A 30, L791 (1997).
  * (15) S.K. Das and S. Chakraborty, Eur. Phys. J. Spec. Top. 226, 765 (2017).
  * (16) S. Chakraborty and S.K. Das, Europhys. Lett. 119, 50005 (2017).
  * (17) N. Vadakkayil, S. Chakraborty and S.K. Das, J. Chem. Phys. 150, 054702 (2019).
  * (18) J. Denholm and B. Hourahina, J. Stat. Mech: Theory and Expt. 2020, 093205 (2020).
  * (19) C. Godriche and M. Pleimling, J. Stat. Mech 2018, 043209 (2018).
  * (20) P. Mullick and P. Sen, Phys. Rev. E 95, 052150 (2017).
  * (21) U. Yu, J. Stat. Mech 2017, 123203 (2017).
  * (22) L.F. Cugliandolo, Comptes Rendus Physique 16, 257 (2015).
  * (23) F. Liu and G.F. Mazenko, Phys. Rev. B 44, 9185 (1991).
  * (24) T. Ohta, D. Jasnow, and K. Kawasaki, Phys. Rev. Lett. 49, 1223 (1982).
  * (25) C. Yeung, Phys. Rev. Lett. 61, 1135 (1988).
  * (26) M.E. Fisher, Rep. Prog. Phys. 30, 615 (1967).
  * (27) H. Van Beijern and I. Nolden, in Structure and Dynamics of Surfaces II: Phenomena, Models and Methods, Topics in Current Physics, vol. 43, ed. W. Schommers and P. von Blanckenhagen (Springer, Berlin, 1987).
  * (28) A.J. Bray, Adv. Phys. 51, 481 (2002).
  * (29) S. Puri and V. Wadhawan (ed.), Kinetics of Phase Transitions (CRC Press, Boca Raton, 2009).
  * (30) D.S. Fisher and D.A. Huse, Phys. Rev. B 38, 373 (1988).
  * (31) C. Yeung, M. Rao, and R.C. Desai, Phys. Rev. E 53, 3073 (1996).
  * (32) M. Henkel, A. Picone, and M. Pleimling, Europhys. Lett. 68, 191 (2004).
  * (33) E. Lorenz and W. Janke, Europhys. Lett. 77, 10003 (2007).
  * (34) J. Midya, S. Majumder, and S.K. Das, J. Phys.: Condens. Matter 26, 452202 (2014).
  * (35) D.A. Huse, Phys. Rev. B 34, 7845 (1986).
  * (36) S. Majumder and S.K. Das, Phys. Rev. E 84, 021110 (2011).
  * (37) S.M. Allen and J.W. Cahn, Acta Metall. 27, 1085 (1979).
  * (38) D.P. Landau and K. Binder, A Guide to Monte Carlo Simulations in Statistical Physics (Cambridge University Press, Cambridge, 2009).
  * (39) J. Midya and S.K. Das, Phys. Rev. Lett. 118, 165701 (2017).
  * (40) M.E. Fisher and M.N. Barber, Phys. Rev. Lett. 28, 1516 (1972).
  * (41) J. Midya, S. Majumder, and S.K. Das, Phys. Rev. E 92, 022124 (2015).
  * (42) R.J. Glauber, J. Math. Phys. 4, 294 (1963).
  * (43) C. Yeung, Phys. Rev. E 97, 062107 (2018).
  * (44) N. Vadakkayil and S.K. Das, Phys. Chem. Chem. Phys 23, 11186 (2021).


# MathViz-E: A Case-study in Domain-Specialized Tool-Using Agents

Arya Bulusu1, Brandon Man2, Ashish Jagmohan1, Aditya Vempaty1, Jennifer Mari-
Wyka1, Deepak Akkil1

1 Emergence AI
2 Massachusetts Institute of Technology

(March 2024)

###### Abstract

There has been significant recent interest in harnessing LLMs to control
software systems through multi-step reasoning, planning and tool-usage. While
some promising results have been obtained, application to specific domains
raises several general issues including the control of specialized domain
tools, the lack of existing datasets for training and evaluation, and the non-
triviality of automated system evaluation and improvement. In this paper, we
present a case-study where we examine these issues in the context of a
specific domain. Specifically, we present an automated math visualizer and
solver system for mathematical pedagogy. The system orchestrates mathematical
solvers and math graphing tools to produce accurate visualizations from simple
natural language commands. We describe the creation of specialized data-sets,
and also develop an auto-evaluator to easily evaluate the outputs of our
system by comparing them to ground-truth expressions. We have open sourced the
data-sets and code for the proposed system.

##  1 Introduction

Large Language Models (LLMs) and Large Multimodal Models (LMMs) have had
extraordinary recent success in tasks involving natural language generation
and code generation. Spurred by this, there has been significant interest in
harnessing LLMs to control software systems and embodied agents through multi-
step reasoning, planning and leveraging tools and APIs Karpas et al. (2022);
Hosseini et al. (2021); Hao et al. (2023); Schick et al. (2023); Patil et al.
(2023). Promising results have been obtained in many tasks, from device and
web-control to game-playing and roboticsWen et al. (2024); Lutz et al. (2024);
Wang et al. (2023a; b); Ahn et al. (2022).

The creation of AI-driven automated systems for specialized domains holds
great economic promise; estimated by a recent study at more than a trillion
dollars Company (2024). While there exist several general multi-agent
frameworks that can be built on, such as Autogen Wu et al. (2023b); Park et
al. (2023), the development of LLM-driven agentic workflows in specialized
domains requires overcoming several additional challenges:

  * •

Firstly, such domains come with specialized tools and problems, different from
the general-purpose problems that past work has often focused on.

  * •

Secondly, there is often a paucity of datasets for training or benchmarking.
For example, common math benchmarks like Cobbe et al. (2021), while important,
are of limited value for settings like math pedagogy.

  * •

Thirdly, automated evaluation of such systems is hard, and human evaluation
does not scale. This makes it hard to create continuous improvement loops,
which are essential for robustness.

In this paper, we investigate the above issues in the context of a specialized
domain; that of math pedagogy. Teachers use a variety of in-classroom
technological tools in day-to-day instruction. The variety and complexity of
operating these tools imposes a cognitive and time-overload, that teachers
would rather spend with students. Generative AI has significant potential to
simplify the tools available in the classroom, allowing teachers to spend more
time interacting with their students instead of their technology (Company,
2023). The combination of specialized tool-use, and the potential benefits of
automation make classroom pedagogy a well-suited use-case for our exploration.

Reflecting the challenges described above, previous LLM-based math research
Mitra et al. (2024); Yu et al. (2023); Liu et al. (2023); Trinh et al. (2024)
has focused on solving math problems and theorem-proving which, while
important, are tangential to in-classroom teaching. Existing math benchmarks
like GSM8K Cobbe et al. (2021) and MATH Hendrycks et al. (2021) are also of
limited value for understanding how LLMs can be applied to the classroom
setting. There are no comprehensive datasets that are purposefully aligned
with educational standards for middle and high school, nor are there datasets
for math visualization pedagogy.

We present an automated math graphing system, that we term MathViz-E, for
mathematical pedagogy. Graphs are an essential tool in the classroom, allowing
students to visualize and interact with mathematical concepts (Donnelly-
Hermosillo et al., 2020). Our automated graphing system takes in voice
utterances, converts them to mathematical expressions, and graphs them with
the Desmos graphing calculator (Desmos Studio, ). This simplifies the process
of creating graphs in the classroom, allowing teachers to more easily
incorporate math visualization techniques into their lessons without
disrupting classroom flow.

Our contributions are:

  * •

We present a voice-driven automated graphing system, combining an LLM with a
mathematical solver and a visual graphing calculator, for the domain of in-
classroom math pedagogy.

  * •

We design new domain-specific datasets for graphing problems, representative
of the Common Core Math standards (CCSSO & NGA-Center, 2010), focused on a
range of learning objectives that teachers currently use visualization tools
to teach.

  * •

Evaluating the accuracy of an automated visual graphing system is non-trivial,
given that the output of the system is a set of math visualizations. We create
an auto-evaluation pipeline to simplify the evaluation of different versions
of our system.

  * •

We present results demonstrating that our proposed system achieves high
accuracy on a wide variety of learning objectives, and show that incorporating
multiple tools significantly out-performs an LLM-only system.

The incorporation of multiple tools, including a solver, provides a foundation
of accuracy, as LLMs alone are incapable of reliably solving several types of
math problems (as we will see in Section 4). This allows the system to produce
accurate graphs even for difficult, multi-step problems requiring complex
reasoning. On the other hand, while mathematical solvers such as Wolfram Alpha
(LLC, 2024) can provide accurate answers for many categories of problems, they
are not capable of understanding all types of natural language. An example of
this is anaphora in multi-turn conversations, where a query refers to objects
already graphed in the calculator (e.g. ”Move the function 5 units
horizontally”). By using an LLM to orchestrate across the solver and the
visual calculator, we create a robust voice- and dialog-based system. Thus the
combination of an LLM with specialized tools produces an automated system with
strong potential for domain use. We have open-sourced our code and datasets at
https://github.com/EmergenceAI/MathViz-E.

##  2 Related Work

Our work in this paper is related to a large body of recent literature in
using LLMs for tool-usage, multi-step reasoning, and plan execution by agents.
There is also related work in the use of LLMs for mathematical reasoning, and
for pedagogy. In this section, we briefly review this literature.

There has been considerable work in the last couple of years on using LLMs to
orchestrate tools and APIs, motivated by the desire to augment language
models’ strong text generation capabilities with other, more specialized
abilities Karpas et al. (2022). Systems like NL2API Hosseini et al. (2021),
ToolkenGPT Hao et al. (2023), Toolformer Schick et al. (2023) and Gorilla
Patil et al. (2023) use language models that learn how to invoke parameterized
APIs. Combining multi-step planning and tool-usage enables the creation of
embodied agents that can plan and act in virtual or real environments.
Examples of this include agents in virtual game environments Baker et al.
(2022); Wang et al. (2023a; b) and real-world robotic agents Ahn et al.
(2022); Bousmalis et al. (2023); Wu et al. (2023a); Bhateja et al. (2023).
Beyond single-agent systems, there has also been much interest in multi-agent
systems and frameworks Wu et al. (2023b); Park et al. (2023). In comparison to
the above, our work focuses on a narrow set of tools, but for domain-specific
capabilities rather than general-purpose usage; we investigate math solvers
and visual calculators specifically for mathematical pedagogy.

There has also been significant work in multi-step reasoning and sequential
decision making. Chain-of-thought (CoT) Wei et al. (2023), and its many
variants Chu et al. (2023) have shown gains on several types of reasoning
tasks. While initially considered an emergent ability in large models like
PaLM Chowdhery et al. (2022), subsequent work has used techniques like
distillation Fu et al. (2023); Li et al. (2023) to specialize smaller models
for specific reasoning tasks. CoT-style step-by-step reasoning can be further
combined with self-critique driven refinement Madaan et al. (2023); Bai et al.
(2022). Another type of approach is to generate code via LLMs, that can be run
via tools like Python interpreters, e.g. Gao et al. (2023). Also related are
approaches like ReAct Yao et al. (2023), Reflexion Shinn et al. (2023) and
many others (e.g. Aksitov et al. (2023)) that employ LLMs for sequential
decision making. In the system we describe in the next section, we use chain-
of-thought with a general instruction-tuned LLM for multiple purposes,
including query reformulation and tool control. While in this paper we’ve used
vanilla CoT and a large model, the use of more specialized CoT variants and
smaller distilled models are intriguing directions for future investigation.

Also related is the literature on training LLMs for mathematical reasoning. A
popular dataset (among many) is the GSM-8K dataset Cobbe et al. (2021) with
grade-school math word problems. Recent work has explored the training of
small, parameter-efficient models, generally through fine-tuning using
augmented data; examples include Mitra et al. (2024); Yu et al. (2023); Liu et
al. (2023). Finally the use of LLMs for various pedagogical tasks includes
work on assessment-generation Wang et al. (2022); Elkins et al. (2023);
Bulathwela et al. (2023), learning-content generation Diwan et al. (2023);
Rodway & Schepman (2023), and the use of commercial models like ChatGPT
through prompt engineering Adeshola & Adepoju (2023); Baidoo-Anu & Ansah
(2023). In contrast to these, we focus on in-classroom math pedagogy, wherein
a teacher controls math tools through voice and language.

##  3 Methodology

###  3.1 Dataset Construction

The Common Core standards (CCSSO & NGA-Center, 2010) are a set of national
educational standards describing what students are expected to know at each
grade level, and they have been widely adopted in the United States. Based on
the math Common Core standards, we identify a set of learning objectives that
teachers use visualization tools to teach in the classroom. We use these
categories as the basis of our dataset, creating approximately 10 questions
per category to evaluate our system on.

The categories included in our datasets and the style of utterances were
refined through teacher feedback, to reflect language that is commonly used by
teachers in math pedagogy (e.g. ”Graph a unit circle” rather than ”Draw a
circle of radius 1”.). This feedback also informed the type of problems we
chose and the learning objectives we targeted in our datasets. From the
identified categories, we create three datasets; the first (referred to as the
utterance-focused dataset) is focused on use cases a teacher might want to
have available in the classroom. The utterances in this dataset are written as
commands a teacher might say, as opposed to written-out problems for a student
to solve. The dataset is mainly comprised of simpler, single-step problems
that a teacher might use to demonstrate intermediate steps in the process of
solving a problem. To ensure robust evaluation, we created variants for
utterances in each category.

Table 1: Example row from utterance-focused dataset

Processed Utterance |  Natural Language Utterance | Graph Input
---|---|---
Reflect y = 5x - 4 across the y-axis |  Reflect y equals five x minus four across the y-axis | $y=-5x-4$

Table 2: Example question from multi-turn dataset

Processed Utterance | Graph Input
---|---
Plot a line that goes through (1, 3) and (4, 8) | $y=\frac{5x}{3}+\frac{4}{3}$
Plot a parallel line through the origin |  $y=\frac{5x}{3}$, $y=\frac{5x}{3}+\frac{4}{3}$

Our second dataset (referred to as the textbook-focused dataset) is focused on
multi-step, complicated problems that require tool use to solve. The main
topics in this dataset are a superset of those in the teacher-focused dataset,
but the problems are geared towards demonstrating the utility of tool use in
LLMs. In contrast to the utterance-focused dataset, we include word problems.
The problems in this dataset are based on representative problems explicitly
written for Common Core standards.

Our third dataset, referred to as the multi-turn dataset, is similar to the
utterance focused dataset but includes multiple turns in each question. This
requires the system to incorporate an understanding of the current calculator
state into its response.

The datasets include a column for the problems and a column for the graph
input associated with the problem. As the system is meant to be used through
natural language commands, we also included a column with the utterance for
the problem (e.g. “Graph $y=5x^{2}+3$” vs. “Graph y equals five x squared plus
three”). This column was automatically generated with GPT-4 (OpenAI, 2023)
based on the original problem column and manually checked over for accuracy.
The utterance-focused dataset contains 70 queries across seven categories, the
textbook-focused dataset contains 147 queries across fourteen categories, and
the multi-turn dataset contains 95 utterances across seven categories.

###  3.2 System

The MathViz-E system consists of four main components: the creation of the
solver query, the generation of a written explanation based on the solver’s
output, the generation of the visual calculator graphing expressions based on
the solver’s output, and the validation and correction of the graphing
expressions based on LLM self-critique. We incorporate multi-turn
functionality into the system by including a calculator state in our prompts
which describes the current expressions graphed on the calculator. As a
result, the system is able to understand queries within the context of the
expressions that have already been graphed. In this paper, we use Desmos as
visual calculator, Wolfram Alpha as the solver and GPT-4 as the LLM, but the
main principles of the system can be broadly applied to other tools and LLMs.
The system is voice-driven; the presented version uses the Web Speech API MDN
Web Docs (2023) for speech recognition, but other ASR pipelines can also be
used.

For a given problem, we create the solver query by prompting the LLM with
instructions and a series of examples demonstrating how to write queries for
certain math problems. These examples were chosen by identifying problems the
LLM consistently misunderstood. The LLM is also provided with the spoken-
utterance version of the problem and the calculator state. The calculator
state contains the equations that have previously been graphed in the graphing
window, and passing this state allows the system to incorporate this
information into its problem-solving process. Below is a truncated version of
the prompt used to create the Wolfram Alpha query:

> Write a Wolfram Alpha query that can be used to solve the problem. The main
> purpose of the task is to find the numerical answer to the problem, not to
> graph the problem. When writing a query for a word problem, only include the
> necessary equation to solve the problem. Ensure that the query is acceptable
> by the Wolfram Alpha engine.
>
> For example, if you are asked:
>
> Graph $y=6x^{2}+4$ and find the local maxima and minima.
>
> Calculator state: []
>
> You generate:
>
> Find the local maxima and minima of $y=6x^{2}+4$

Figure 1: Overview of the MathViz-E automated graphing system

Once the query has been generated, we input it to our solver. Wolfram Alpha
provides a set of pods for each query, with each pod containing a different
category of information related to the query. Wolfram Alpha also provides
step-by-step solutions for some problems. From these results, we extract the
solution (generally the second pod, after the “Input Interpretation” pod) and
the step-by-step solution if it is present.

To generate an explanation of the problem, we prompt the LLM with a zero-shot
instruction. Along with the prompt, we provide the natural language utterance
version of the problem, the calculator state, the numerical solution as given
by the solver, and the solver’s step-by-step solution, if it is present.

In the cases where Wolfram Alpha provides a step-by-step solution, the LLM
only has to expand upon this solution by providing more detail and explaining
the reasoning behind the steps. When there is no step-by-step solution given,
it must write its own explanation from scratch based on the problem and
numerical solution.

In order to generate the Desmos graphing expressions, we prompt the LLM with
instructions and a set of examples. In the prompt, we ask for a chain-of-
thought, which helps to generate more accurate expressions. Chain-of-thought
prompting has been shown to improve the accuracy of LLMs’ reasoning,
especially with regards to math problems.(Wei et al., 2022)(Chu et al., 2023)
As with the explanation prompt, we also provide the natural language utterance
version of the problem, the calculator state, the solver’s numerical solution,
and the solver’s step-by-step solution, if there is one. The large number of
examples in the prompt helps guide the LLM towards producing valid Desmos
expressions. The provided examples were created by identifying common points
of failure, and writing problems that demonstrate how to accurately deal with
these issues.

In the last self-critique step, we ask the LLM to validate and refine the
previously-generated Desmos expressions by checking for common errors. A
majority of these errors arise from Desmos API deviations from standard latex
(e.g. the use of $le$ and $ge$ instead of $leq$ and $geq$ for inequalities, or
using $abs(\cdot)$ instead of $|\cdot|$ for the absolute value function).
These checks also include ensuring that correct graphing variables are used,
operations are formatted correctly, and functions are named properly. This
step helps to eliminate basic errors made by the LLM in the expression-
generating step.

Figure 2: UI of MathViz-E demonstrated through a multi-turn inverse problem

##  4 Experiments

###  4.1 Autoevaluation

Traditional evaluation metrics for text similarity fail for comparing
mathematical statements due to the precise nature of math statements. Consider
the statement 5=2+3. Lexical similarity metrics, such as Jaccard distance,
would consider the statement 5=2+4 more similar than 5=4+1 since the former
statements shares more words in common than the latter. Furthermore, many
existing similarity metrics do not recognize common mathematical symbols as
tokens and thus cannot be converted into a numerical representation.
Similarly, directly examining the visual graph output through a multimodal
approach is unlikely to be precise enough for our purposes. Although LLMs may
be able to evaluate equivalence for simple expressions, their judgement
becomes inconsistent for more complex expressions. As a result, evaluating the
equations output by the automated graphing system at a large scale is
nontrivial.

Due to these limiting factors, we create a new autoevaluation pipeline that
can precisely compare two mathematical statements. We use the computer algebra
system SymPy (Meurer et al., 2017) to evaluate the mathematical equations
output by the LLM in our LLM+Solver system when responding to given questions.
In order to compare two equations, we use SymPy to isolate a variable and
compare the resulting expressions on the other side of the equality.

Although this approach leads to accurate checking of math statements, an issue
is that SymPy cannot parse certain formats of equations, which the LLM in the
LLM+Solver system may produce. To combat this, we use an LLM as a backup in
the autoevaluation process, where if SymPy cannot parse an equation, it will
let the LLM compare the two math statements and output a result.

We construct a set of ground truth evaluations by running all the questions in
our datasets through the LLM-only system, and manually evaluating if the
system’s output matches the correct answer. This manually-benchmarked dataset
allows us to run different versions of the autoevaluator on the dataset and
check how its evaluations compare to our manually-written evaluations.

The simpler version of our autoevaluator only uses an LLM to compare two
equations. Using GPT-4 as the LLM, we compare the results of LLM-only and
LLM+SymPy autoevaluators on the entirety of our utterance-focused and
textbook-focused datasets. An older version of the textbook-focused dataset
was used, with the same categories and question styles as the current version.
In the table below, we display the dataset-wide results as well as the results
for selected categories.

Table 3: Accuracy of LLM-only autoevaluator and LLM+SymPy autoevaluator as compared to manual evaluations |  Utterance-Focused Dataset |  Textbook-Focused Dataset |  Systems of Linear Equations |  Graph Inverse Functions |  Graph Lines
---|---|---|---|---|---
LLM-Only |  77% |  76% |  50% |  40% |  92%
LLM+SymPy |  86% |  88% |  100% |  80% |  85%

The addition of SymPy to the autoevaluation pipeline increases the accuracy of
evaluations significantly on almost all categories, especially in Systems of
Linear Equations. In general, the LLM+SymPy autoevaluator performs better than
the LLM-only autoevaluator on problems that are well-structured but
computationally difficult. These problems are easily interpretable and can be
solved by SymPy, as it can easily handle complex algebraic manipulations. In
contrast, an LLM-only approach would struggle to accurately carry out algebra.

We see a minor drop in performance for a few categories, such as Graph Lines.
This decrease in accuracy is generally due to SymPy and the LLM both
misunderstanding the formatting of the equation. An important point to note is
that the overwhelming majority of incorrectly evaluated answers are the result
of a correct input being marked as incorrect. This occurs because SymPy will
only mark expressions equivalent if they are genuinely equivalent, and GPT-4
rarely marks inequivalent expressions as being equivalent. As a result, when
using the autoevaluation pipeline, we can trust that nearly all the responses
marked as correct are actually correct, and manually check if the responses
marked as incorrect are in fact incorrect. As a result, this greatly reduces
the burden of evaluation when testing the performance of various iterations of
the system.

###  4.2 Results

For the results reported in this section, we comprehensively evaluate the
LLM+Solver system by manually validating all of the outputs generated for the
three datasets (utterance-focused, textbook-focused, and multi-turn). We
compare the performance of the LLM+Solver system to the results of the LLM-
only system. The LLM-only system consists of directly prompting an LLM with
instructions to write Desmos expressions and examples, while also providing
the natural language utterance problem and the calculator state. No solver
solution is provided. Although the framework of the system can be applied to
LLMs and solvers broadly, in this paper we evaluate using GPT-4 as the LLM and
Wolfram Alpha as the solver. Tables 4, 5 and 6 compare the results for the
LLM-only system and the LLM+Solver systems.

Table 4: Accuracy of LLM-only and LLM+Solver models
| LLM-only | LLM+Solver
---|---|---
Utterance-Focused Dataset | 66% | 90%
Textbook-Focused Dataset | 64% | 86%
Multi-turn Dataset | 86% | 91%
Table 5: Accuracy of individual categories in utterance-focused dataset
| LLM-only | LLM+Solver
---|---|---
Graph Circles | 90% | 100%
Transform Shapes | 70% | 70%
Intersections of Lines | 20% | 100%
Transform Functions | 100% | 90%
Graph Lines | 90% | 100%
Local Minima and Maxima | 40% | 90%
X Intercepts, Y Intercepts | 50% | 80%
Table 6: Accuracy of individual categories in textbook-focused and multi-turn
datasets
| LLM-only | LLM+Solver
---|---|---
Proportional Relationships | 100% | 100%
Linear Inequality Systems | 100% | 90%
Graph Inequalities | 90% | 100%
Graph Lines | 100% | 100%
Graph Lines (multi-turn) | 91% | 100%
Graph Polynomials + Identify Zeros | 67% | 100%
Graph Polynomials (multi-turn) | 67% | 100%
Systems of Linear + Quadratic Equations | 15% | 100%
Systems of Lin + Quad Eqns (multi-turn) | 60% | 100%
Graph Circles | 100% | 100%
Graph Circles (multi-turn) | 96% | 83%
Transformations of Functions | 100% | 100%
Transform Functions (multi-turn) | 86% | 90%
Graph Inverse Functions | 70% | 100%
Graph Inverse Functions (multi-turn) | 90% | 100%
Tangents to Parabolas | 0% | 11%
Tangents to Circles | 0% | 75%
Linear and Nonlinear Functions | 60% | 70%
Linear and Nonlinear Functions (multi) | 100% | 70%
Systems of Linear Equations | 40% | 100%
Rigid Transformations + Dilations | 70% | 60%

LLM+Solver vs. LLM-only system Table 4 shows that the addition of the solver
results in a significant overall performance increase across all three
datasets. Tables 5 and 6 further show that the greatest performance increase
occurs in categories such as Local Minima and Maxima, X and Y intercepts,
Intersections of Lines, Systems of Equations, and Tangents to Circles. These
problems require complex calculations which are difficult for GPT-4 to carry
out by itself, meaning that GPT-4 will frequently get them wrong. However,
Wolfram Alpha can solve these problems, given a well-formulated query. As a
result, the LLM+Solver system shows strong improvement over the LLM-only
system for these categories.

Error Modes for LLM+Solver system The proposed system demonstrates excellent
accuracy across a wide variety of problem types, as demonstrated in Table 5
and Table 6. However, there are some notable categories on which it has poor
accuracy. It is instructive to delve deeper into these error modes.

  * •

For the category ”Tangents to Parabolas”, the problem arises from the Wolfram
ALpha solver API, which does not return the correct answer in its Step-by-Step
Solution response. This is a specific instance of a broader problem, where it
is not always clear which of the Wolfram Alpha API response fields should be
input to the LLM for further reasoning; fixing this requires conditioning
response interpretation on the problem type (e.g. ”Tangents to Parabola” and
”Tangents to Circles” produce different types of API responses).

  * •

For ”Rigid Function Transformations”, errors arise from improper handling of
polygon transformations. While the solution is very robust to transformations
of parametric shapes, it struggles with non-parametric shape transformations.
Improving this is an area of future work.

  * •

For other categories, some common (though occasional) error modes include:
incomplete specifications (like ”Draw a circle” without any radius or center
specified) sometimes result in the system producing parametric expressions
without associated sliders; a single query containing a list of sub-queries
(”Plot the x-intercept, the y-intercept, the local minima and maxima, and the
asymptotes”) occasionally result in some sub-queries being missed; and tasks
of sufficient complexity (”Move the circle so it is tangent to a line which
satisfies property x and property y”) may be computed incorrectly. The last
error mode is not generally an issue for the pedagogical levels that the
system is designed for, but could be an issue for higher-ed (compared to
K-12).

Analysis of the LLM-only system The LLM-only system struggles the most in
problems that require complicated reasoning and calculations to solve, such as
”Tangents to Circles”, ”Systems of Linear + Quadratic Equations”, ”Intercepts”
and others. It tends to fail either by solving the problem with an incorrect
method or executing calculations incorrectly. It also sometimes plots the
underlying functions or shapes, but then does not successfully compute points
of interesections, extrema etc.

For many categories, the performance of the LLM-only system was strong to
begin with, such as Circles, Proportional Relationships, and Graph Lines.
These categories contain simple problems with little mathematical calculation,
so GPT-4 is able to succeed at these problems without the help of a solver.
There are some categories for which there are no Wolfram Alpha queries that
can be used to solve the problem, such as Transform Shapes and Transformations
on Functions. These categories do not show much change in the performance of
both systems, as Wolfram Alpha cannot be used to provide answers for these
categories. Incorporation of a more powerful tool, such as Python, could allow
the system to successfully solve these problems.

##  5 Discussion

The results presented in this paper highlight the potential of domain-specific
automation created via LLM orchestration of specialized tools. The presented
case-study highlights some of the main challenges that need to be overcome in
such systems. The paucity of preexisting datasets requires careful creation of
new datasets for benchmarking; there is a need to identify and control
specialized tools; and there is a need for auto-evaluation to validate and
improve system performance.

For the case of mathematical pedagogy, we showed that through the design and
development of an LLM-orchestrated system incorporating multiple specialized
tools, through the creation of new benchmark datasets based on Common Core,
and through the creation of an auto-evaluation pipeline, we created an
effective automated system and the means to easily evaluate future iterations.
The system has strong performance, and for many categories of problems it can
consistently produce accurate outputs.

For the domain under consideration, there are categories of problems for which
the system is not yet fully reliable. To improve system performance in these
categories, there are several directions that we can take in the future. An
especially promising approach is to add retrieval-based techniques to make
adjustments to the system for the individual problem categories. This
individualized approach could improve accuracy for some categories as compared
to our current, one-size-fits-all system. Another key direction is to reduce
latency; we plan to evaluate our system using smaller open-source LLMs,
trained via supervised fine-tuning, instead of a large general model like
GPT-4.

More broadly, for domain-specific agentic solutions, the approach presented in
this paper offers a set of patterns that can be generalized. We expect that
the generalization and application of these patterns to other domains and
problems will continue to remain fertile ground for future research.

#### Acknowledgments

We would like to thank Ravi Kokku, Marc Pickett, Prasenjit Dey, and Paul Haley
for helpful discussions and feedback.

## References

  * Adeshola & Adepoju (2023) Ibrahim Adeshola and Adeola Praise Adepoju.  The opportunities and challenges of chatgpt in education.  _Interactive Learning Environments_ , pp. 1–14, 2023.
  * Ahn et al. (2022) Michael Ahn, Anthony Brohan, Noah Brown, Yevgen Chebotar, Omar Cortes, Byron David, Chelsea Finn, Chuyuan Fu, Keerthana Gopalakrishnan, Karol Hausman, Alex Herzog, Daniel Ho, Jasmine Hsu, Julian Ibarz, Brian Ichter, Alex Irpan, Eric Jang, Rosario Jauregui Ruano, Kyle Jeffrey, Sally Jesmonth, Nikhil J Joshi, Ryan Julian, Dmitry Kalashnikov, Yuheng Kuang, Kuang-Huei Lee, Sergey Levine, Yao Lu, Linda Luu, Carolina Parada, Peter Pastor, Jornell Quiambao, Kanishka Rao, Jarek Rettinghouse, Diego Reyes, Pierre Sermanet, Nicolas Sievers, Clayton Tan, Alexander Toshev, Vincent Vanhoucke, Fei Xia, Ted Xiao, Peng Xu, Sichun Xu, Mengyuan Yan, and Andy Zeng.  Do as i can, not as i say: Grounding language in robotic affordances, 2022.
  * Aksitov et al. (2023) Renat Aksitov, Sobhan Miryoosefi, Zonglin Li, Daliang Li, Sheila Babayan, Kavya Kopparapu, Zachary Fisher, Ruiqi Guo, Sushant Prakash, Pranesh Srinivasan, Manzil Zaheer, Felix Yu, and Sanjiv Kumar.  Rest meets react: Self-improvement for multi-step reasoning llm agent, 2023.
  * Bai et al. (2022) Yuntao Bai, Saurav Kadavath, Sandipan Kundu, Amanda Askell, Jackson Kernion, Andy Jones, Anna Chen, Anna Goldie, Azalia Mirhoseini, Cameron McKinnon, Carol Chen, Catherine Olsson, Christopher Olah, Danny Hernandez, Dawn Drain, Deep Ganguli, Dustin Li, Eli Tran-Johnson, Ethan Perez, Jamie Kerr, Jared Mueller, Jeffrey Ladish, Joshua Landau, Kamal Ndousse, Kamile Lukosuite, Liane Lovitt, Michael Sellitto, Nelson Elhage, Nicholas Schiefer, Noemi Mercado, Nova DasSarma, Robert Lasenby, Robin Larson, Sam Ringer, Scott Johnston, Shauna Kravec, Sheer El Showk, Stanislav Fort, Tamera Lanham, Timothy Telleen-Lawton, Tom Conerly, Tom Henighan, Tristan Hume, Samuel R. Bowman, Zac Hatfield-Dodds, Ben Mann, Dario Amodei, Nicholas Joseph, Sam McCandlish, Tom Brown, and Jared Kaplan.  Constitutional ai: Harmlessness from ai feedback, 2022.
  * Baidoo-Anu & Ansah (2023) David Baidoo-Anu and Leticia Owusu Ansah.  Education in the era of generative artificial intelligence (ai): Understanding the potential benefits of chatgpt in promoting teaching and learning.  _Journal of AI_ , 7(1):52–62, 2023.
  * Baker et al. (2022) Bowen Baker, Ilge Akkaya, Peter Zhokhov, Joost Huizinga, Jie Tang, Adrien Ecoffet, Brandon Houghton, Raul Sampedro, and Jeff Clune.  Video pretraining (vpt): Learning to act by watching unlabeled online videos, 2022.
  * Bhateja et al. (2023) Chethan Bhateja, Derek Guo, Dibya Ghosh, Anikait Singh, Manan Tomar, Quan Vuong, Yevgen Chebotar, Sergey Levine, and Aviral Kumar.  Robotic offline rl from internet videos via value-function pre-training, 2023.
  * Bousmalis et al. (2023) Konstantinos Bousmalis, Giulia Vezzani, Dushyant Rao, Coline Devin, Alex X. Lee, Maria Bauza, Todor Davchev, Yuxiang Zhou, Agrim Gupta, Akhil Raju, Antoine Laurens, Claudio Fantacci, Valentin Dalibard, Martina Zambelli, Murilo Martins, Rugile Pevceviciute, Michiel Blokzijl, Misha Denil, Nathan Batchelor, Thomas Lampe, Emilio Parisotto, Konrad Żołna, Scott Reed, Sergio Gómez Colmenarejo, Jon Scholz, Abbas Abdolmaleki, Oliver Groth, Jean-Baptiste Regli, Oleg Sushkov, Tom Rothörl, José Enrique Chen, Yusuf Aytar, Dave Barker, Joy Ortiz, Martin Riedmiller, Jost Tobias Springenberg, Raia Hadsell, Francesco Nori, and Nicolas Heess.  Robocat: A self-improving foundation agent for robotic manipulation, 2023.
  * Bulathwela et al. (2023) Sahan Bulathwela, Hamze Muse, and Emine Yilmaz.  Scalable educational question generation with pre-trained language models.  In _International Conference on Artificial Intelligence in Education_ , pp. 327–339. Springer, 2023.
  * CCSSO & NGA-Center (2010) CCSSO and NGA-Center.  Common core state standards.  https://www.thecorestandards.org/Math/, 2010.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vinodkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengcheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, Andrew M. Dai, Thanumalayan Sankaranarayana Pillai, Marie Pellat, Aitor Lewkowycz, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel.  Palm: Scaling language modeling with pathways, 2022.
  * Chu et al. (2023) Zheng Chu, Jingchang Chen, Qianglong Chen, Weijiang Yu, Tao He, Haotian Wang, Weihua Peng, Ming Liu, Bing Qin, and Ting Liu.  A survey of chain of thought reasoning: Advances, frontiers and future, 2023.
  * Cobbe et al. (2021) Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, Christopher Hesse, and John Schulman.  Training verifiers to solve math word problems, 2021.
  * Company (2023) McKinsey & Company.  What’s the future of generative AI? an early view in 15 charts.  https://www.mckinsey.com/featured-insights/mckinsey-explainers/whats-the-future-of-generative-ai-an-early-view-in-15-charts, 2023.
  * Company (2024) McKinsey & Company.  The economic potential of generative ai: The next productivity frontier.  https://www.mckinsey.com/capabilities/mckinsey-digital/our-insights/the-economic-potential-of-generative-ai-the-next-productivity-frontier, 2024.
  * (16) PBC Desmos Studio.  Desmos graphing calculator.  https://www.desmos.com/calculator.
  * Diwan et al. (2023) Chaitali Diwan, Srinath Srinivasa, Gandharv Suri, Saksham Agarwal, and Prasad Ram.  Ai-based learning content generation and learning pathway augmentation to increase learner engagement.  _Computers and Education: Artificial Intelligence_ , 4:100110, 2023.
  * Donnelly-Hermosillo et al. (2020) Dermot Francis Donnelly-Hermosillo, Libby F. Gerard, and Marcia C. Linn.  Impact of graph technologies in k-12 science and mathematics education.  _Computers & Education_, 146:103748, 2020.  ISSN 0360-1315.  doi: https://doi.org/10.1016/j.compedu.2019.103748.
  * Elkins et al. (2023) Sabina Elkins, Ekaterina Kochmar, Iulian Serban, and Jackie CK Cheung.  How useful are educational questions generated by large language models?  In _International Conference on Artificial Intelligence in Education_ , pp. 536–542. Springer, 2023.
  * Fu et al. (2023) Yao Fu, Hao Peng, Litu Ou, Ashish Sabharwal, and Tushar Khot.  Specializing smaller language models towards multi-step reasoning, 2023.
  * Gao et al. (2023) Luyu Gao, Aman Madaan, Shuyan Zhou, Uri Alon, Pengfei Liu, Yiming Yang, Jamie Callan, and Graham Neubig.  Pal: Program-aided language models, 2023.
  * Hao et al. (2023) Shibo Hao, Tianyang Liu, Zhen Wang, and Zhiting Hu.  Toolkengpt: Augmenting frozen language models with massive tools via tool embeddings, 2023.
  * Hendrycks et al. (2021) Dan Hendrycks, Collin Burns, Saurav Kadavath, Akul Arora, Steven Basart, Eric Tang, Dawn Song, and Jacob Steinhardt.  Measuring mathematical problem solving with the math dataset, 2021.  URL https://arxiv.org/abs/2103.03874.
  * Hosseini et al. (2021) Saghar Hosseini, Ahmed Hassan Awadallah, and Yu Su.  Compositional generalization for natural language interfaces to web apis, 2021.
  * Karpas et al. (2022) Ehud Karpas, Omri Abend, Yonatan Belinkov, Barak Lenz, Opher Lieber, Nir Ratner, Yoav Shoham, Hofit Bata, Yoav Levine, Kevin Leyton-Brown, Dor Muhlgay, Noam Rozen, Erez Schwartz, Gal Shachaf, Shai Shalev-Shwartz, Amnon Shashua, and Moshe Tenenholtz.  Mrkl systems: A modular, neuro-symbolic architecture that combines large language models, external knowledge sources and discrete reasoning, 2022.
  * Li et al. (2023) Yuanzhi Li, Sébastien Bubeck, Ronen Eldan, Allie Del Giorno, Suriya Gunasekar, and Yin Tat Lee.  Textbooks are all you need ii: phi-1.5 technical report, 2023.
  * Liu et al. (2023) Bingbin Liu, Sebastien Bubeck, Ronen Eldan, Janardhan Kulkarni, Yuanzhi Li, Anh Nguyen, Rachel Ward, and Yi Zhang.  Tinygsm: achieving ¿80
  * LLC (2024) Wolfram Alpha LLC.  Wolfram alpha.  https://www.wolframalpha.com/, 2024.
  * Lutz et al. (2024) Michael Lutz, Arth Bohra, Manvel Saroyan, Artem Harutyunyan, and Giovanni Campagna.  Wilbur: Adaptive in-context learning for robust and accurate web agents.  _arXiv preprint arXiv:2404.05902_ , 2024.
  * Madaan et al. (2023) Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark.  Self-refine: Iterative refinement with self-feedback, 2023.
  * MDN Web Docs (2023) MDN Web Docs.  Mdn web speech api.  https://developer.mozilla.org/en-US/docs/Web/API/Web_Speech_API/Using_the_Web_Speech_API, 2023.
  * Meurer et al. (2017) Aaron Meurer, Christopher P. Smith, Mateusz Paprocki, Ondřej Čertík, Sergey B. Kirpichev, Matthew Rocklin, AMiT Kumar, Sergiu Ivanov, Jason K. Moore, Sartaj Singh, Thilina Rathnayake, Sean Vig, Brian E. Granger, Richard P. Muller, Francesco Bonazzi, Harsh Gupta, Shivam Vats, Fredrik Johansson, Fabian Pedregosa, Matthew J. Curry, Andy R. Terrel, Štěpán Roučka, Ashutosh Saboo, Isuru Fernando, Sumith Kulal, Robert Cimrman, and Anthony Scopatz.  Sympy: symbolic computing in python.  _PeerJ Computer Science_ , 3:e103, January 2017.  ISSN 2376-5992.  doi: 10.7717/peerj-cs.103.
  * Mitra et al. (2024) Arindam Mitra, Hamed Khanpour, Corby Rosset, and Ahmed Awadallah.  Orca-math: Unlocking the potential of slms in grade school math, 2024.
  * OpenAI (2023) OpenAI.  Gpt-4 technical report.  https://arxiv.org/abs/2303.08774, 2023.
  * Park et al. (2023) Joon Sung Park, Joseph C. O’Brien, Carrie J. Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein.  Generative agents: Interactive simulacra of human behavior, 2023.
  * Patil et al. (2023) Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez.  Gorilla: Large language model connected with massive apis, 2023.
  * Rodway & Schepman (2023) Paul Rodway and Astrid Schepman.  The impact of adopting ai educational technologies on projected course satisfaction in university students.  _Computers and Education: Artificial Intelligence_ , 5:100150, 2023.
  * Schick et al. (2023) Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom.  Toolformer: Language models can teach themselves to use tools, 2023.
  * Shinn et al. (2023) Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao.  Reflexion: Language agents with verbal reinforcement learning, 2023.
  * Trinh et al. (2024) T.H. Trinh, Y. Wu, Q.V. Le, H. He, and T. Luong.  Solving olympiad geometry without human demonstrations.  _Nature_ , 36, 2024.
  * Wang et al. (2023a) Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar.  Voyager: An open-ended embodied agent with large language models, 2023a.
  * Wang et al. (2022) Zichao Wang, Jakob Valdez, Debshila Basu Mallick, and Richard G Baraniuk.  Towards human-like educational question generation with large language models.  In _International conference on artificial intelligence in education_ , pp. 153–166. Springer, 2022.
  * Wang et al. (2023b) Zihao Wang, Shaofei Cai, Anji Liu, Yonggang Jin, Jinbing Hou, Bowei Zhang, Haowei Lin, Zhaofeng He, Zilong Zheng, Yaodong Yang, Xiaojian Ma, and Yitao Liang.  Jarvis-1: Open-world multi-task agents with memory-augmented multimodal language models, 2023b.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou.  Chain of thought prompting elicits reasoning in large language models.  _CoRR_ , abs/2201.11903, 2022.  URL https://arxiv.org/abs/2201.11903.
  * Wei et al. (2023) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou.  Chain-of-thought prompting elicits reasoning in large language models, 2023.
  * Wen et al. (2024) Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu.  Autodroid: Llm-powered task automation in android.  In _Proceedings of the 30th Annual International Conference on Mobile Computing and Networking_ , pp. 543–557, 2024.
  * Wu et al. (2023a) Hongtao Wu, Ya Jing, Chilam Cheang, Guangzeng Chen, Jiafeng Xu, Xinghang Li, Minghuan Liu, Hang Li, and Tao Kong.  Unleashing large-scale video generative pre-training for visual robot manipulation, 2023a.
  * Wu et al. (2023b) Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W White, Doug Burger, and Chi Wang.  Autogen: Enabling next-gen llm applications via multi-agent conversation, 2023b.
  * Yao et al. (2023) Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao.  React: Synergizing reasoning and acting in language models, 2023.
  * Yu et al. (2023) Longhui Yu, Weisen Jiang, Han Shi, Jincheng Yu, Zhengying Liu, Yu Zhang, James T. Kwok, Zhenguo Li, Adrian Weller, and Weiyang Liu.  Metamath: Bootstrap your own mathematical questions for large language models, 2023.