is a closed subvariety of $\widetilde{A}\simeq\mathbb{C}^{\dim A}$, which is
thus a Stein space. Since $f:X^{\prime}\to f(X^{\prime})$ is finite, it
follows that $X^{\prime}$ is also Stein. Note that any unramified covering of
a Stein space is Stein. Therefore, $\widetilde{X}$ is a Stein space. ∎

######  Theorem 7.9.

Let $X_{0}$ be a complex projective normal surface and let $p$ be a fixed
prime and $N$ be a fixed positive integer. We assume that $M_{\rm
B}(X_{0},N)_{\mathbb{F}_{p}}$ is large, meaning that for any positive
dimensional closed subvariety $Z$ of $X_{0}$, there exists a linear
representation $\varrho:\pi_{1}(X_{0})\to{\rm GL}_{N}(K)$ with ${\rm
char}\,K=p$ such that $\varrho({\rm Im}[\pi_{1}(Z^{\rm
norm})\to\pi_{1}(X_{0})])$ is infinite. Then the universal covering of $X_{0}$
is Stein.

###### Proof.

Let $\mu:X\to X_{0}$ be a desingularization. It induces a morphism
$\iota:M_{\rm B}(X_{0},N)_{\mathbb{F}_{p}}\hookrightarrow M_{\rm
B}(X,N)_{\mathbb{F}_{p}}$ between affine $\mathbb{F}_{p}$-schemes of finite
type which is a closed immersion. Let $M:=\iota(M_{\rm
B}(X_{0},N)_{\mathbb{F}_{p}})$ which is a Zariski closed subset of $M_{\rm
B}(X,N)_{\mathbb{F}_{p}}$. By Theorem 2.7, the Shafarevich morphism ${\rm
sh}_{M}:X\to{\rm Sh}_{M}(X)$ of $M$ exists.

######  Claim 7.10.

The Shafarevich morphism ${\rm sh}_{M}:X\to{\rm Sh}_{M}(X)$ coincides with
$\mu:X\to X_{0}$.

###### Proof.

Let $Z$ be any closed subvariety of $X$ and let $W:=\mu(Z)$. If ${\rm
sh}_{M}(Z)$ is a point, then $\mu^{*}\sigma({\rm Im}[\pi_{1}(Z^{\rm
norm})\to\pi_{1}(X)])$ is finite for any $[\sigma]\in M_{\rm
B}(X_{0},N)_{\mathbb{F}_{p}}(L)$. Therefore, $\sigma({\rm Im}[\pi_{1}(W^{\rm
norm})\to\pi_{1}(X_{0})])$ is finite. Hence $W$ is a point since $M_{\rm
B}(X_{0},N)_{\mathbb{F}_{p}}$ is assumed to be large.

On the other hand, assume that $\mu(Z)$ is a point. Let
$\sigma:\pi_{1}(X)\to{\rm GL}_{N}(L)$ be any representation where $L$ is a
field of characteristic $p$ such that $[\sigma]\in M(L)$. Then there exists a
representation $\sigma^{\prime}:\pi_{1}(X_{0})\to{\rm GL}_{N}(\bar{L})$ such
that $[\mu^{*}\sigma^{\prime}]=[\sigma]$. By Lemma 2.1, for any subgroup
$\Gamma\subset\pi_{1}(X)$, $\mu^{*}\sigma^{\prime}(\Gamma)$ is finite if and
only if $\sigma(\Gamma)$ is finite. Since $\mu^{*}\sigma^{\prime}({\rm
Im}[\pi_{1}(Z^{\rm norm})\to\pi_{1}(X)])$ is trivial, it follows that
$\sigma({\rm Im}[\pi_{1}(Z^{\rm norm})\to\pi_{1}(X)])$ is finite. By the
properties of ${\rm sh}_{M}$ proven in Theorem 2.7, ${\rm sh}_{M}(Z)$ is a
point. This proves that ${\rm sh}_{M}=\mu$. ∎

Let $H_{0}:=\cap_{\varrho}\ker\varrho$ where $\varrho:\pi_{1}(X_{0})\to{\rm
GL}_{N}(L)$ ranges over all linear representation and $L$ is any field with
${\rm char}\,L=p$. Denote by $H:=\mu^{*}H_{0}$. Let
$\pi_{H}:\widetilde{X}_{H}\to X$ be the Galois covering with the Galois group
$\pi_{1}(X)/H$ and $\pi_{0}:\widetilde{X}_{0}\to X_{0}$ be that with
$\pi_{1}(X_{0})/H_{0}$. Consequently, we have the following commutative
diagram

${\widetilde{X}_{H}}$${X}$${\widetilde{X}_{0}}$${X_{0}}$${{\rm
Sh}_{M}(X)}$$\scriptstyle{\pi_{H}}$$\scriptstyle{f}$$\scriptstyle{\mu}$$\scriptstyle{{\rm
sh}_{M}}$$\scriptstyle{\pi_{0}}$ (7.1)

where $f:\widetilde{X}_{H}\to\widetilde{X}_{0}$ is a holomorphic proper
fibration.

######  Claim 7.11.

Let $\sigma:\pi_{1}(X)\to{\rm GL}_{N}(L)$ be a linear representation such that
$[\sigma]\in M(L)$, where $L$ is a field of characteristic $p$. Then
$H\subset\mathrm{ker}(\sigma^{ss})$.

###### Proof.

There exists a linear representation $\sigma^{\prime}:\pi_{1}(X_{0})\to{\rm
GL}_{N}(\bar{L})$ such that $[\mu^{*}\sigma^{\prime}]=[\sigma]$. Then
$(\mu^{*}\sigma^{\prime})^{ss}$ is conjugate to $\sigma^{ss}$. Hence
$\mathrm{ker}(\mu^{*}\sigma^{\prime})\subset\mathrm{ker}((\mu^{*}\sigma^{\prime})^{ss})=\mathrm{ker}(\sigma^{ss})$.
We have
$H\subset\mu^{*}\mathrm{ker}(\sigma^{\prime})=\mathrm{ker}(\mu^{*}\sigma^{\prime})$.
Thus $H\subset\mathrm{ker}(\sigma^{ss})$. ∎

Consider the affine $\mathbb{F}_{p}$-scheme $R_{\mathbb{F}_{p}}$ and the GIT
quotient $\pi:R_{\mathbb{F}_{p}}\to M_{\rm B}(X,N)_{\mathbb{F}_{p}}$, as
defined in § 2.1. Define $R:=\pi^{-1}(M)$, which is an affine
$\overline{\mathbb{F}_{p}}$-scheme of finite type. According to the structure
of the Shafarevich morphism described in Theorem 2.7 and Definition 2.6, the
Shafarevich morphism ${\rm sh}_{M}:X\to{\rm Sh}_{M}(X)$ of $M$ is obtained
through the simultaneous Stein factorization of the reductions
$\\{s_{\tau}:X\to S_{\tau}\\}_{[\tau]\in M(K)}$. Here $\tau:\pi_{1}(X)\to{\rm
GL}_{N}(K)$ ranges over all reductive representations with $K$ a local field
of characteristic $p$ such that $[\tau]\in M(K)$ and $s_{\tau}:X\to S_{\tau}$
is the reduction map defined in Theorem 1.2.

By Lemma 1.6, there exists a reductive representation
${\bm{\tau}}:\pi_{1}(X)\to{\rm GL}_{N^{\prime}}(K^{\prime})$ with $K^{\prime}$
a local field of characteristic $p$ such that the Katzarkov-Eyssidieux
reduction map $s_{{\bm{\tau}}}:X\to S_{{\bm{\tau}}}$ of ${\bm{\tau}}$
coincides with ${\rm Sh}_{M}:X\to{\rm Sh}_{M}(X)$. Moreover by Lemma 1.6 and
Claim 7.11, we have

$H\subset\mathrm{ker}({\bm{\tau}}).$ (7.2)

Case 1: The spectral 1-forms have rank 2. Assume that the spectral one forms
$\\{\eta_{1},\ldots,\eta_{k}\\}\subset H^{0}(X^{\\!\rm
sp},\pi^{*}\Omega_{X}^{1})$ with respect to ${\bm{\tau}}:\pi_{1}(X)\to{\rm
GL}_{N^{\prime}}(K^{\prime})$ has rank 2.

Let $s_{\bm{\tau}}:X\to S_{\bm{\tau}}$ be the reduction map of ${\bm{\tau}}$,
which coincides with ${\rm sh}_{M}:X\to{\rm Sh}_{M}(X)$ (and thus $\mu:X\to
X_{0}$). Let $T_{\bm{\tau}}$ be the canonical current on $X_{0}=S_{\bm{\tau}}$
defined in Definition 7.2. Then $T_{\bm{\tau}}$ is strictly positive at
general points since the spectral forms associated with ${\bm{\tau}}$ has rank
2. Since $T_{\bm{\tau}}$ has continuous local potentials, it follows that
$T_{\bm{\tau}}$ is a big and nef class.

On the other hand, by Lemma 7.3, we conclude that $T_{\bm{\tau}}$ is strictly
nef. We now apply a theorem by Demailly-Păun [DP04] (or [Lam99]) to conclude
that $\\{T_{\bm{\tau}}\\}$ is a Kähler class.

According to Proposition 7.4 and (7.2), there exists a continuous
plurisubharmonic function $\phi:\widetilde{X}_{0}\to\mathbb{R}_{\geq 0}$ such
that ${\rm d}{\rm d}^{\rm c}\phi\geq\pi_{0}^{*}T_{{\bm{\tau}}}$. We can apply
Proposition 7.5 to conclude that $\widetilde{X}_{0}$ is a Stein space. Note
that any unramified covering of a Stein space is Stein. Therefore, the
universal covering of $X_{0}$ is a Stein space.

Case 2: The spectral 1-forms have rank 1: Assume that the spectral covering
$X^{\\!\rm sp}\to X$ with respect to ${\bm{\tau}}:\pi_{1}(X)\to{\rm
GL}_{N^{\prime}}(K^{\prime})$ has rank 1; i.e., $\eta_{i}\wedge\eta_{j}\equiv
0$ for every $\eta_{i}$ and $\eta_{j}$, where
$\\{\eta_{1},\ldots,\eta_{\ell}\\}\subset H^{0}(X^{\\!\rm
sp},\pi^{*}\Omega_{X}^{1})$ is the spectral forms associated with
${\bm{\tau}}:\pi_{1}(X)\to{\rm GL}_{N^{\prime}}(K^{\prime})$, and
$\pi:X^{\\!\rm sp}\to X$ is the spectral covering associated with
${\bm{\tau}}:\pi_{1}(X)\to{\rm GL}_{N^{\prime}}(K^{\prime})$.

Case 2.1: The dimension of spectral 1-forms is at least two: Suppose ${\rm
dim}_{\mathbb{C}}{\rm Span}\\{\eta_{1},\ldots,\eta_{\ell}\\}\geq 2.$

Without loss of generality, we may assume that $\eta_{1}\wedge\eta_{2}\equiv
0$ and $\eta_{1}\not\in\\{\mathbb{C}\eta_{2}\\}$. According to the
Castelnuovo-De Franchis theorem (cf. [ABC+96, Theorem 2.7]), there exists a
proper fibration $h:X^{\\!\rm sp}\to C$ over a smooth projective curve $C$
such that $\\{\eta_{1},\eta_{2}\\}\subset h^{*}H^{0}(C,\Omega_{C}^{1})$. Since
${\rm sh}_{M}=\mu$ is birational, we can choose a general fiber $F$ of $h$,
which is irreducible and such that ${\rm
sh}_{M}\circ\pi(F)=s_{\bm{\tau}}\circ\pi(F)$ is not a point. There exists some
$i$ such that $\eta_{i}|_{F}\neq 0$ (cf. Lemma 7.1). Given that
$\eta_{1}|_{F}\equiv 0$, this implies that $\eta_{i}\wedge\eta_{1}\neq 0$. It
contradicts with our assumption that the spectral $1$-forms have rank 1.
Therefore, this case cannot occur.

Case 2.2: The dimension of spectral 1-forms is 1: We have ${\rm
dim}_{\mathbb{C}}{\rm Span}\\{\eta_{1},\ldots,\eta_{\ell}\\}=1.$

Let $G$ be the Zariski closure of ${\bm{\tau}}(\pi_{1}(X))$, which is
reductive. Consider the isogeny $g:G\to G/Z\times G/\mathcal{D}G$ where $Z$ is
the central torus of $G$ and $\mathcal{D}G$ is the derived group of $G$. As a
result, $G^{\prime}:=G/Z$ is semisimple and $G^{\prime\prime}:=G/\mathcal{D}G$
is a torus. Let $\tau^{\prime}:\pi_{1}(X)\to
G^{\prime}(\overline{K^{\prime}})$ be the composite of ${\bm{\tau}}$ with the
projection $G\to G^{\prime}$, and $\tau^{\prime\prime}:\pi_{1}(X)\to
G^{\prime\prime}(\overline{K^{\prime}})$ be the composite of ${\bm{\tau}}$
with the projection $G\to G^{\prime\prime}$. Then $\tau^{\prime}$ and
$\tau^{\prime\prime}$ are both Zariski dense representations.

Let $\nu:Y\to X^{\\!\rm sp}$ be a desingularization and denote
$\eta:=\nu^{*}\eta_{1}$. Consider the partial Albanese morphism $a:Y\to A$
induced by $\eta$. Then there exists a one form $\eta^{\prime}\in
H^{0}(A,\Omega_{A}^{1})$ such that $a^{*}\eta^{\prime}=\eta$. If $\dim
a(Y)=1$, then the Stein factorization $h:Y\to C$ of $a$ is a proper
holomorphic fibration over a smooth projective curve $C$ such that
$\eta_{1}\in h^{*}H^{0}(C,\Omega_{C}^{1})$. We are now in a situation akin to
Case 2.1, and we can apply the same arguments to reach a contradiction. Hence
$\dim a(Y)=2$. Let $\pi_{A}:\widetilde{A}\to A$ denote the universal covering
map. We denote by $Y^{\prime}:=Y\times_{\widetilde{A}}A$ a connected component
of the fiber product and let $\pi^{\prime}:Y^{\prime}\to Y$ be the induced
étale cover. It’s worth noting that $\pi^{\prime*}\eta$ is exact.
Consequently, we can define the following holomorphic map:

$\displaystyle h:Y^{\prime}$ $\displaystyle\to\mathbb{C}$ $\displaystyle y$
$\displaystyle\mapsto\int_{y_{0}}^{y}\pi^{\prime*}\eta.$

We then have the following commutative diagram:

${\widetilde{Y}}$${Y^{\prime}}$${Y}$${\widetilde{A}}$${A}$${\mathbb{C}}$$\scriptstyle{p}$$\scriptstyle{\pi_{Y}}$$\scriptstyle{h}$$\scriptstyle{\pi^{\prime}}$$\scriptstyle{a}$$\scriptstyle{\pi_{A}}$

The holomorphic map $\widetilde{A}\to\mathbb{C}$ in the above diagram is
defined by the linear 1-form $\pi_{A}^{*}\eta^{\prime}$ on $\widetilde{A}$. By
Simpson’s Lefschetz theorem [Sim93a], for any $t\in\mathbb{C}$, $h^{-1}(t)$ is
connected and we have the surjectivity
$\pi_{1}(h^{-1}(t))\twoheadrightarrow\pi_{1}(Y^{\prime})$. By the definition
of $h$, $\pi_{Y}^{*}\eta|_{Z}\equiv 0$ where $Z$ is any connected component of
$p^{-1}(h^{-1}(t))$. Here $p:\widetilde{Y}\to Y^{\prime}$ is the natural
covering map.

Consider the Zariski dense representation $\tau^{\prime}:\pi_{1}(X)\to
G^{\prime}\big{(}\overline{K^{\prime}}\big{)}$ as defined previously. Let $L$
be a finite extension of $K^{\prime}$ such that $G^{\prime}$ is defined on $L$
and $\tau^{\prime}:\pi_{1}(X)\to G^{\prime}(L)$. We denote by
$\sigma:\pi_{1}(Y)\to G^{\prime}\big{(}L\big{)}$ the pullback of
$\tau^{\prime}$ via the morphism $Y\to X$. The existence of a
$\sigma$-equivariant harmonic mapping $u:\widetilde{Y}\to\Delta(G^{\prime})$
is guaranteed by [GS92], where $\Delta(G^{\prime})$ is the Bruhat-Tits
building of $G^{\prime}$.

We note that $\pi_{Y}^{*}\eta$ is the (1,0)-part of the complexified
differential of the harmonic mapping $u$ at general points of $\widetilde{Y}$,
with $\pi_{Y}:\widetilde{Y}\to Y$ denoting the universal covering. For any
connected component $Z$ of $p^{-1}(h^{-1}(t))$ for a general $t\in\mathbb{C}$,
since $\pi_{Y}^{*}\eta|_{Z}\equiv 0$, and all the spectral forms are assumed
to be $\mathbb{C}$-linearly equivalent, it follows that $u(Z)$ is constant.
Since $u$ is $\sigma$-equivariant, it follows that $\pi^{\prime*}\sigma({\rm
Im}[\pi_{1}(h^{-1}(t))\to\pi_{1}(Y^{\prime})])$ is contained in the subgroup
of $G^{\prime}(L)$ fixing the point $u(Z)$. Recall that
$\pi_{1}(h^{-1}(t))\to\pi_{1}(Y^{\prime})$ is surjective. Hence
$\pi^{\prime*}\sigma(\pi_{1}(Y^{\prime}))$ is a bounded subgroup of
$G^{\prime}(L)$. Additionally, note that $\mathcal{D}\pi_{1}(Y)\subset{\rm
Im}[\pi_{1}(Y^{\prime})\to\pi_{1}(Y)]$, and it follows that
$\sigma(\mathcal{D}\pi_{1}(Y))$ is bounded. Since $\tau^{\prime}$ is Zariski
dense, and ${\rm Im}[\pi_{1}(Y)\to\pi_{1}(X)]$ is a finite index subgroup of
$\pi_{1}(X)$, according to Lemma 7.6 the Zariski closure of
$\sigma(\pi_{1}(Y))$ contains the identity component of $G^{\prime}$, and it
is also semisimple. We apply Lemma 7.7 to conclude that $\sigma(\pi_{1}(Y))$
is bounded.

Since $\sigma(\pi_{1}(Y))$ is a finite index subgroup of
$\tau^{\prime}(\pi_{1}(X))$, it follows that $\tau^{\prime}(\pi_{1}(X))$ is
also bounded. Then the reduction map $s_{\tau^{\prime}}$ is the constant map.
This implies that the reduction map $s_{\tau}$ is identified with
$s_{\tau^{\prime\prime}}$. Recall that $G^{\prime\prime}$ is a tori. By
[CDY22, Step 6 in Proof of Theorem H], we know that there exists a morphism
$a:X\to A$ with $A$ an abelian variety such that $s_{\tau^{\prime\prime}}$ is
the Stein factorization of $a$.

######  Claim 7.12.

There exists a morphism $b:X_{0}\to A$ such that $b\circ\mu=a$.

###### Proof.

Recall that $[\tau]\in M(K^{\prime})$. Let $F$ be any fiber of the birational
morphism $\mu:X\to X_{0}$. Then $\tau({\rm Im}[\pi_{1}(F)\to\pi_{1}(X)])$ is
finite since ${\rm sh}_{M}:X\to{\rm Sh}_{M}$ coincides with $\mu$. It follows
that $\tau({\rm Im}[\pi_{1}(F)\to\pi_{1}(X)])$ is a finite group. By the
definition of $\tau^{\prime\prime}$, we conclude that
$\tau^{\prime\prime}({\rm Im}[\pi_{1}(F)\to\pi_{1}(X)])$ is also finite, and
is thus bounded. According to the property of the reduction map Theorem 1.2,
$s_{\tau^{\prime\prime}}(F)$ is a point. Hence there exists a morphism
$b:X_{0}\to A$ such that $b\circ\mu=a$. ∎

Consider the map $b:X_{0}\to A$. Note that this map is finite. We apply Lemma
7.8 to conclude that the universal covering of $X_{0}$ is a Stein space. ∎

######  Remark 7.13.

The proof of Theorem 7.9 utilizes techniques similar to those used in the
proof of [DYK23, Theorem C]. In order to extend Theorem 7.9 to any projective
normal variety, we have to establish Simpson’s theory [Sim93b] on _absolutely
constructible subsets_ for character varieties of representations in positive
characteristic. We plan to explore this problem in our future work.

We recall the following definition by Campana [Cam94].

######  Definition 7.14 ($\Gamma$-dimension).

Let $X$ be a projective normal variety. The $\Gamma$-dimension of $X$ is
defined to be $\dim{\rm Sh}(X)$, where ${\rm sh}_{X}:X\dashrightarrow{\rm
Sh}(X)$ is the Shafarevich map constructed by Campana [Cam94] and Kollár
[Kol93] (see Remark 2.11).

######  Theorem 7.15.

Let $X$ be a projective normal variety and let $\varrho:\pi_{1}(X)\to{\rm
GL}_{N}(K)$ be a faithful representation where $K$ is a field of positive
characteristic. If the $\Gamma$-dimension of $X$ is at most two (e.g. when
$\dim X\leq 2$), then the universal covering $\widetilde{X}$ of $X$ is
holomorphically convex.

###### Proof.

By Proposition 4.6, after we replace $X$ by a suitable finite étale cover,
there exists a large representation $\tau:\pi_{1}({\rm
Sh}_{\varrho}({X}))\to{\rm GL}_{N}(K)$ such that $({\rm
sh}_{\varrho})^{*}\tau=\varrho$, where ${\rm sh}_{\varrho}:{X}\to{\rm
Sh}_{\varrho}({X})$ is the Shafarevich morphism of $\varrho$. Since $\varrho$
is faithful, it follows that $\tau$ is also faithful since the homomorphism
$({\rm sh}_{\varrho})_{*}:\pi_{1}(X)\to\pi_{1}({\rm sh}_{\varrho}(X))$ is
surjective. As we assume that the $\Gamma$-dimension of $X$ is at most two,
therefore, $\dim{\rm Sh}_{\varrho}({X})\leq 2$. We apply Theorem 7.9 to
conclude that the universal covering $S$ of $\dim{\rm Sh}_{\varrho}({X})$ is
Stein. Note that $({\rm sh}_{\varrho})_{*}:\pi_{1}(X)\to\pi_{1}({\rm
sh}_{\varrho}(X))$ is an isomorphism. Hence there exists a proper holomorphic
fibration $\widetilde{X}\to S$ between the universal coverings of $X$ and
${\rm Sh}_{\varrho}(X)$ that lifts ${\rm sh}_{\varrho}$. It follows that
$\widetilde{X}$ is holomorphically convex. ∎

## References

  * [ABC+96] J. Amorós, M. Burger, K. Corlette, D. Kotschick & D. Toledo.  _Fundamental groups of compact Kähler manifolds_ , vol. 44 of _Mathematical Surveys and Monographs_.  American Mathematical Society, Providence, RI (1996).  URL http://dx.doi.org/10.1090/surv/044.
  * [BDDM22] D. Brotbek, G. Daskalopoulos, Y. Deng & C. Mese.  “Pluriharmonic maps into buildings and symmetric differentials”.  _arXiv e-prints_ , (2022):arXiv:2206.11835.  2206.11835, URL http://dx.doi.org/10.48550/arXiv.2206.11835.
  * [Cam94] F. Campana.  “Remarks on the universal covering of compact Kähler manifolds”.  _Bull. Soc. Math. Fr._ , 122(1994)(2):255–284.  URL http://dx.doi.org/10.24033/bsmf.2232.
  * [Cam04] ———.  “Orbifolds, special varieties and classification theory.”  _Ann. Inst. Fourier_ , 54(2004)(3):499–630.  URL http://dx.doi.org/10.5802/aif.2027.
  * [Cam11] ———.  “Special orbifolds and birational classification: a survey”.  “Classification of algebraic varieties. Based on the conference on classification of varieties, Schiermonnikoog, Netherlands, May 2009.”, 123–170. Zürich: European Mathematical Society (EMS) (2011):.
  * [CCE15] F. Campana, B. Claudon & P. Eyssidieux.  “Linear representations of Kähler groups: factorizations and linear Shafarevich conjecture”.  _Compos. Math._ , 151(2015)(2):351–376.  URL http://dx.doi.org/10.1112/S0010437X14007751.
  * [CDY22] B. Cadorel, Y. Deng & K. Yamanoi.  “Hyperbolicity and fundamental groups of complex quasi-projective varieties”.  _arXiv e-prints_ , (2022):arXiv:2212.12225.  2212.12225, URL http://dx.doi.org/10.48550/arXiv.2212.12225.
  * [CH13] B. Claudon & A. Höring.  “Compact Kähler manifolds with compactifiable universal cover”.  _Bull. Soc. Math. France_ , 141(2013)(2):355–375.  With an appendix by Frédéric Campana, URL http://dx.doi.org/10.24033/bsmf.2651.
  * [CHK13] B. Claudon, A. Höring & J. Kollár.  “Algebraic varieties with quasi-projective universal cover”.  _J. Reine Angew. Math._ , 679(2013):207–221.  URL http://dx.doi.org/10.1515/crelle.2012.017.
  * [CP19] F. Campana & M. Păun.  “Foliations with positive slopes and birational stability of orbifold cotangent bundles”.  _Publ. Math., Inst. Hautes Étud. Sci._ , 129(2019):1–49.  URL http://dx.doi.org/10.1007/s10240-019-00105-w.
  * [CW16] F. Campana & J. Winkelmann.  “Rational connectedness and order of non-degenerate meromorphic maps from $\mathbb{C}^{n}$”.  _Eur. J. Math._ , 2(2016)(1):87–95.  URL http://dx.doi.org/10.1007/s40879-015-0083-z.
  * [Del10] T. Delzant.  “The Bieri-Neumann-Strebel invariant of fundamental groups of Kähler manifolds”.  _Math. Ann._ , 348(2010)(1):119–125.  URL http://dx.doi.org/10.1007/s00208-009-0468-8.
  * [Den23] Y. Deng.  “Big Picard theorems and algebraic hyperbolicity for varieties admitting a variation of Hodge structures”.  _Épijournal de Géométrie Algébrique_ , Volume 7(2023).  URL http://dx.doi.org/10.46298/epiga.2023.volume7.8393.
  * [DP04] J.-P. Demailly & M. Paun.  “Numerical characterization of the Kähler cone of a compact Kähler manifold”.  _Ann. Math. (2)_ , 159(2004)(3):1247–1274.  URL http://dx.doi.org/10.4007/annals.2004.159.1247.
  * [DYK23] Y. Deng, K. Yamanoi & L. Katzarkov.  “Reductive Shafarevich Conjecture”.  _arXiv e-prints_ , (2023):arXiv:2306.03070.  2306.03070, URL http://dx.doi.org/10.48550/arXiv.2306.03070.
  * [EKPR12] P. Eyssidieux, L. Katzarkov, T. Pantev & M. Ramachandran.  “Linear Shafarevich conjecture”.  _Ann. Math. (2)_ , 176(2012)(3):1545–1581.  URL http://dx.doi.org/10.4007/annals.2012.176.3.4.
  * [Eys04] P. Eyssidieux.  “Sur la convexité holomorphe des revêtements linéaires réductifs d’une variété projective algébrique complexe”.  _Invent. Math._ , 156(2004)(3):503–564.  URL http://dx.doi.org/10.1007/s00222-003-0345-0.
  * [Fuj17] O. Fujino.  “Notes on the weak positivity theorems”.  “Algebraic varieties and automorphism groups. Proceedings of the workshop held at RIMS, Kyoto University, Kyoto, Japan, July 7–11, 2014”, 73–118. Tokyo: Mathematical Society of Japan (MSJ) (2017):.
  * [GS92] M. Gromov & R. Schoen.  “Harmonic maps into singular spaces and $p$-adic superrigidity for lattices in groups of rank one”.  _Publ. Math., Inst. Hautes Étud. Sci._ , 76(1992):165–246.  URL http://dx.doi.org/10.1007/BF02699433.
  * [Hof09] K. R. Hofmann.  “Triangulation of locally semi-algebraic spaces.”  https://deepblue.lib.umich.edu/bitstream/handle/2027.42/63851/krhofman_1.pdf?sequence=1&isAllowed=y (2009).
  * [Kat97] L. Katzarkov.  “On the Shafarevich maps”.  “Algebraic geometry. Proceedings of the Summer Research Institute, Santa Cruz, CA, USA, July 9–29, 1995”, 173–216. Providence, RI: American Mathematical Society (1997):.
  * [Kol93] J. Kollár.  “Shafarevich maps and plurigenera of algebraic varieties”.  _Invent. Math._ , 113(1993)(1):177–215.  URL http://dx.doi.org/10.1007/BF01244307.
  * [Kol95] ———.  _Shafarevich maps and automorphic forms_.  Princeton University Press, Princeton (N.J.) (1995).
  * [KP23] T. Kaletha & G. Prasad.  _Bruhat-Tits theory. A new approach (to appear)_ , vol. 44 of _New Math. Monogr._ Cambridge: Cambridge University Press (2023).
  * [KR98] L. Katzarkov & M. Ramachandran.  “On the universal coverings of algebraic surfaces”.  _Ann. Sci. École Norm. Sup. (4)_ , 31(1998)(4):525–535.  URL http://dx.doi.org/10.1016/S0012-9593(98)80105-5.
  * [Lam99] A. Lamari.  “The Kähler cone of a surface”.  _J. Math. Pures Appl. (9)_ , 78(1999)(3):249–263.  URL http://dx.doi.org/10.1016/S0021-7824(98)00005-1.
  * [LM85] A. Lubotzky & A. R. Magid.  “Varieties of representations of finitely generated groups”.  _Mem. Amer. Math. Soc._ , 58(1985)(336):xi+117.  URL http://dx.doi.org/10.1090/memo/0336.
  * [Mil17] J. S. Milne.  _Algebraic groups. The theory of group schemes of finite type over a field_ , vol. 170 of _Camb. Stud. Adv. Math._ Cambridge: Cambridge University Press (2017).  URL http://dx.doi.org/10.1017/9781316711736.
  * [Nog81] J. Noguchi.  “Lemma on logarithmic derivatives and holomorphic curves in algebraic varieties”.  _Nagoya Math. J._ , 83(1981):213–233.  URL http://projecteuclid.org/euclid.nmj/1118786486.
  * [NWY13] J. Noguchi, J. Winkelmann & K. Yamanoi.  “Degeneracy of holomorphic curves into algebraic varieties. II”.  _Vietnam J. Math._ , 41(2013)(4):519–525.  URL http://dx.doi.org/10.1007/s10013-013-0051-1.
  * [Ses77] C. S. Seshadri.  “Geometric reductivity over arbitrary base”.  _Advances in Math._ , 26(1977)(3):225–274.  URL http://dx.doi.org/10.1016/0001-8708(77)90041-X.
  * [Sim88] C. T. Simpson.  “Constructing variations of Hodge structure using Yang-Mills theory and applications to uniformization”.  _J. Amer. Math. Soc._ , 1(1988)(4):867–918.  URL http://dx.doi.org/10.2307/1990994.
  * [Sim92] ———.  “Higgs bundles and local systems”.  _Inst. Hautes Études Sci. Publ. Math._ , (1992)(75):5–95.  URL http://www.numdam.org/item?id=PMIHES_1992__75__5_0.
  * [Sim93a] ———.  “Lefschetz theorems for the integral leaves of a holomorphic one-form”.  _Compos. Math._ , 87(1993)(1):99–113.
  * [Sim93b] ———.  “Subspaces of moduli spaces of rank one local systems”.  _Ann. Sci. Éc. Norm. Supér. (4)_ , 26(1993)(3):361–401.  URL http://dx.doi.org/10.24033/asens.1675.
  * [ST00] G. Smith & O. Tabachnikova.  _Topics in group theory_.  Springer Undergraduate Mathematics Series. Springer-Verlag London, Ltd., London (2000).  URL http://dx.doi.org/10.1007/978-1-4471-0461-2.
  * [Yam10] K. Yamanoi.  “On fundamental groups of algebraic varieties and value distribution theory”.  _Ann. Inst. Fourier_ , 60(2010)(2):551–563.  URL http://dx.doi.org/10.5802/aif.2532.


# Sequence to General Tree: Knowledge-Guided
Geometry Word Problem Solving

Shih-hung Tsai, Chao-Chun Liang, Hsin-Min Wang, Keh-Yih Su
Institute of Information Science, Academia Sinica, Taiwan
<EMAIL_ADDRESS>

###### Abstract

With the recent advancements in deep learning, neural solvers have gained
promising results in solving math word problems. However, these SOTA solvers
only generate binary expression trees that contain basic arithmetic operators
and do not explicitly use the math formulas. As a result, the expression trees
they produce are lengthy and uninterpretable because they need to use multiple
operators and constants to represent one single formula. In this paper, we
propose sequence-to-general tree (S2G) that learns to generate interpretable
and executable operation trees where the nodes can be formulas with an
arbitrary number of arguments. With nodes now allowed to be formulas, S2G can
learn to incorporate mathematical domain knowledge into problem-solving,
making the results more interpretable. Experiments show that S2G can achieve a
better performance against strong baselines on problems that require domain
knowledge.111Data and code are available at the GitHub repository:
https://github.com/doublebite/Sequence-to-General-tree/

##  1 Introduction

Math word problem (MWP) solving is a special subfield of question answering.
It requires machine solvers to read the problem text, understand it, and then
compose the numbers and operators into a meaningful equation (as shown in
Table 1). This process, even for the simplest problem in elementary school,
demands language understanding and numerical reasoning capabilities, making
this task a long-standing challenge in AI Bobrow (1964); Zhang et al. (2019).

Problem: The outer radius and the inner radius
---
of a circular annulus are 5m and 3m repsectively.
Find the area of this circular annulus.
Equation: $x=5*5*3.14-3*3*3.14$
Answer: 50.24
With formula: $x$ = circle_area(5) - circle_area(3)

Table 1: Example problem that requires geometry knowledge. Figure 1: (a).
binary expression tree and (b). operation tree along with formulas for the
problem in Table 1.

As with any QA task, solving an MWP requires the introduction of external
knowledge or domain knowledge Mishra et al. (2020). However, current state-of-
the-art solvers Xie and Sun (2019); Zhang et al. (2020); Wu et al. (2020) do
not address this issue explicitly. They learn to map the problem text into
binary expression trees regardless of whether it requires any knowledge. This
is counterintuitive for problems that need math concepts or formulas. As
illustrated in Figure 1(a), without explicitly using the corresponding area
formula, the expression tree for the problem is lengthy and uninterpretable.

To address this issue, we propose a sequence-to-general tree (S2G)
architecture where the nodes can be arbitrary math concepts or formulas with
arbitrary number of arguments. In this way, our S2G model can learn to map the
problem text into executable operation trees that contain different formulas
across different domains. For example, S2G can learn to generate tree nodes
that contain the required geometry formula for circles, as shown in Figure
1(b), making the result more intuitive and explainable.

In addition, we propose a knowledge-guided mechanism to guide tree-decoding
using a mathematical knowledge graph (KG). To evaluate our model, we also
construct a middle-sized dataset consisting of 1,398 geometry word problems
which require a diversified set of formulas. Experimental results show that
our S2G model can provide better performance and more interpretable results
against strong baselines on problems that require domain knowledge.

The main contributions of this paper are:

  1. 1.

We propose a seq-to-general tree model that learns to map the problem text
into operation trees where the nodes can be formulas with arbitrary number of
arguments. This helps to incorporate domain knowledge into problem solving and
produce interpretable results.

  2. 2.

We design a knowledge-guided mechanism that guides tree decoding using
mathematical knowledge graphs and GNNs.

  3. 3.

We curate a middle-sized dataset that contains 1,398 geometry word problems.
In addition, we annotate them with detailed formulas that can be readily
converted into operation trees.

##  2 Seq2seq v.s. Seq2tree v.s. Seq2general

Our goal is to design a sequence-to-general tree model that learns to map the
problem text into its corresponding operation tree. Before diving into the
model, we first compare the decoding mechanisms between seq-to-seq, seq-to-
tree and our seq-to-general tree solvers. Figure 2 illustrates the tree
decoding process of these three types of model, respectively.

For seq2seq models, their decoder basically does two things: (1) predicting
the current output and (2) generating the next state. These two steps can be
conditioned on different information including the current state, the current
input, or a context vector calculated using attention. The decoder would
repeat these two steps until it outputs an end token.

Figure 2: Comparison between three types of decoding: (a) seq2seq, (b)
seq2tree, and (c) seq-to-general tree.

For seq2tree models, however, this process is slightly different. The decoder
predicts the current output as in seq2seq, but it will decide whether to
generate the next state based on the current output. If the current output is
a arithmetic operator, the decoder knows it should produce two child states,
and these states are used to expand into its left and right children. If the
current output is a number, then the decoder would end the decoding process,
so the current node becomes a leaf node. As a result, the whole decoding
process resembles generating an expression tree in a top-down manner.

In our work, we generalize the decoding process by making the decoder produce
a variable number of children based on the type of the current output. If the
output is a number or operator, the decoder would produce zero or two child
states as before. If the output is a formula, the decoder will generate the
pre-specified number of child states for this formula.

##  3 Sequence-to-General Tree Model

In this section, we give a detailed description for each part of our S2G
model.

###  3.1 Encoder

The main function of the encoder is to encode the problem text
$P=(x_{1},x_{2},...,x_{n})$ into a sequence of hidden states
$(h_{1},h_{2},...,h_{n})$ and their summary state $h_{encoder}$. The hidden
states $h_{1}$ to $h_{n}$ are expected to contain the information for each
input token $x_{1}$ to $x_{n}$, while the summary state $h_{encoder}$ is
expected to capture the overall information of the problem.

Specifically, we use bidirectional gated recurrent units (GRU) Cho et al.
(2014) as our encoder. Given the current input $x_{t}$, the previous state
$h_{t-1}$, and the next state $h_{t+1}$, the current state
$h_{t}\in(h_{1},h_{2},...,h_{n})$ can be calculated with:

$\displaystyle\overrightarrow{h_{t}}=\mathrm{GRU}(x_{t},\overrightarrow{h_{t-1}}),$
(1)
$\displaystyle\overleftarrow{h_{t}}=\mathrm{GRU}(x_{t},\overleftarrow{h_{t+1}}),$
(2)

where the arrows represent different directions in the bidirectional encoding.
After calculating the hidden state for each input token, we combine the last
state of the forward and backward directions to get the summary state for the
encoder:

$h_{encoder}=\overleftarrow{h_{0}}+\overrightarrow{h_{n}}$ (3)

###  3.2 Geometry Knowledge Graph

To incorporate domain knowledge into problem solving, we propose to utilize
the knowledge from mathematical knowledge graphs. The main idea is that given
a formula predicted as the current node, we could use the physical meaning of
its arguments to help us better predict its children. For example, if the
current node is the formula for rectangle area, then we know its child nodes
should be related to ”length ” and ”width”. We can thus use the node
embeddings of ”length” and ”width” from a geometry KG to provide additional
information for our solver.

We manually collect a geometry knowledge graph which contains the common
geometry shapes (e.g., square, circle) and their geometry quantities (e.g.,
area, length), and we link these nodes to each other if they belong to the
same shape. To embed this KG, we employ a graph convolutional network (GCN)
Kipf and Welling (2017) that transforms the KG into some vector space and
calculates the embedding of each node. Given the feature matrix $X$ and the
adjacency matrix $A$ of the KG, we use a two-layer GCN to encode it as
follows:

$Z=\mathrm{GCN}(X,A),$ (4)

where $Z=(z_{1},...,z_{n})$ are the node embeddings for each node in the
graph. Then, we can use the embedding to represent the physical meaning of a
certain formula argument in the decoding process.

###  3.3 General Tree Decoder

In the decoding stage, the decoder learns to produce the target operation
trees in a recursive manner. It first predicts the current output $y_{t}$ in
order to determine the number of children of the current node. Given the
current decoder state $s_{t}$, the embedding of the last output
$e_{(y_{t-1})}$, and the node embedding $z_{t}$ which represents the physical
meaning in the knowledge graph, the probability of the current output
$P(y_{t})$ is calculated using:

$\displaystyle c_{t}$
$\displaystyle=\mathrm{Attention(e_{(y_{t-1})},h_{1}^{n})}$ (5) $\displaystyle
z_{t}^{\prime}$ $\displaystyle=\mathrm{Attention(z_{t},h_{1}^{n})}$ (6)
$\displaystyle P(y_{t})$
$\displaystyle=\mathrm{Softmax}(W_{y}[s_{t};e_{(y_{t-1})};c_{t};z_{t}^{\prime}]),$
(7)

where $h_{1}^{n}$ is the encoder states $(h_{1},...,h_{n})$, $c_{t}$ is the
context vector of $e_{(y_{t-1})}$ with respect to $h_{1}^{n}$, and
$z_{t}^{\prime}$ is another context vector calculated using the node embedding
$z_{t}$ and $h_{1}^{n}$. Specifically, we use additive attention Bahdanau et
al. (2015) to calculate these context vectors and use $h_{encoder}$ as the
first decoder state $s_{0}$. Given the probability $P(y_{t})$, we can then
determine the output token $\hat{y_{t}}$:

$\hat{y_{t}}=\mathrm{argmax}P(y_{t}).$ (8)

Next, we predict the child states conditioned on the required number of
children for $\hat{y_{t}}$. Unlike previous binary-tree decoders that use two
distinct DNNs to predict the left and right children respectively Xie and Sun
(2019); Zhang et al. (2020); Wu et al. (2020), we employ a GRU to predict a
variable number of children. Given the current state $s_{t}$, its child states
$s_{t_{1}},...,s_{t_{n}}$ are generated in a recurrent manner:

$s_{t_{i}}=\mathrm{Decoder}(s_{t_{i-1}};e_{(y_{t})};c_{t}),$ (9)

where we generate the first child $s_{t_{1}}$ using $s_{t}$, and the following
child state $s_{t_{i}}$ using its previous sibling $s_{t_{i-1}}$until we reach
the required number of children. The decoder is basically a GRU followed by a
linear projection layer and an activation function:

$\displaystyle s_{t_{i}}^{\prime}$
$\displaystyle=\mathrm{GRU}([e_{(y_{t})};c_{t}],s_{t_{i-1}}),$ (10)
$\displaystyle s_{t_{i}}$
$\displaystyle=\mathrm{ReLU}(W_{s}s^{\prime}_{t_{i}}),$ (11)

where the input of GRU is the concatenation of $e_{(y_{t})}$ and $c_{t}$,
$W_{s}$ is the linear projection layer, and ReLU is used as the activation
function. After getting these child states, we push them into a stack and
repeat the steps from Equation (5) to Equation (11) until all the states are
realized into tokens.

###  3.4 Training Objective

For a problem and operation tree pair (P, T), we follow previous seq2tree work
Xie and Sun (2019); Wu et al. (2020) and set our objective to minimize the
negative log likelihood:

$L(T,P)=\sum_{t=1}^{n}-logP(y_{t}|s_{t},P,KG).$ (12)

##  4 Dataset

To evaluate our S2G model on problems that require formulas, we curate a
middle-sized dataset, GeometryQA, that contains 1,398 geometry word problems.
These problems are collected from Math23K Wang et al. (2017) using the
keywords of common geometric objects (e.g., circle, square, etc.) and their
shapes (e.g., rectangular, circular, etc.). Then, we re-annotate each problem
with their associated formulas if the problem belongs to one of the six major
shapes: square, cubic, rectangle, cuboid, triangle and circle. Table 2 shows
the overall statistics of GeometryQA and Table 7 in Appendix B shows the 11
formulas we used to annotate these problems.

Note that not all problems in GeometryQA are annotated with formulas. About
16% of the problems belong to other shapes (e.g., parallelogram, rhombus,
etc.) which currently are not covered in our formula set. About 40% are
problems that contain geometric keywords but do not actually require any
formulas. Table 3 shows such an example. We use these problems to test the
robustness of our model. That is, S2G has to learn to apply the correct
formulas or equations from misleading keywords (as shown in Table3) and has to
learn to generate both binary expression trees and operation trees as a whole.

GeometryQA |
---|---
Number of problems | 1,398
Number of sentences/words | 5.4k / 41.1k
Vocabulary size | 2,872
Annotated with formulas | 604 (43.20%)
Problems of other shapes | 225 (16.09%)
Formulas not required | 569 (40.70%)
Table 2: Dataset statistics of GeometryQA

Problem: The perimeter of a rectangular swim-
---
ming pool is 300 m. If you place a chair every
10 m all the way around its perimeter, how many
chairs do you need?
Equation: $x=300/10$
Answer: 30

Table 3: Example problem that contains misleading keywords (perimeter,
rectangular) but do not require any geometry formulas.

##  5 Experiments

###  5.1 Implementation Details

We implement our S2G model and the GNN module using
Pytorch222https://pytorch.org/ and Pytorch Geometric333https://pytorch-
geometric.readthedocs.io/en/latest/. We set the dimension of word embedding to
128 and the dimension of the hidden state of GRU and GNN to 512. The dropout
rate Srivastava et al. (2014) is set to 0.5 and the batch size is 64. For
optimization, we use ADAM Kingma and Ba (2015) with a learning rate of
$10^{-3}$ and a weight decay of $10^{-5}$. Besides, we use a learning rate
scheduler to reduce the learning rate by half every 20 epochs. During
evaluation, we use beam search Wiseman and Rush (2016) with a beam size of 5.

###  5.2 Experimental Results on GeometryQA

We evaluate our S2G model on GeometryQA to check whether it can learn to
predict the corresponding operation tree for the geometry word problems. Table
4 shows the results of our S2G against other seq2tree SOTA models. S2G is
trained using the re-annotated equations that contain formulas, while the
baselines are trained using the original equations.

First, we find that S2G has about 3.8% performance gain over its baselines
(with p-value $<$ 0.01). We attribute this to the fact that operation trees
are easier to learn and generate since they are less lengthy and complex than
binary expression trees. Hence, there is a better chance for S2G to produce
the correct trees and arrive at the correct answers.

Second, there is a small performance gain by adding Geometry KG. However, the
improvement is not significant (with p-value$\approx$0.8). We guess that is
because the dataset currently has only six geometric objects, which is not
complex enough to show the effectiveness of adding knowledge graphs.

Model | Accuracy(%)
---|---
KA-S2T Wu et al. (2020) | 49.61%
GTS Xie and Sun (2019) | 51.01%
S2G | 54.79%
S2G + Geometry KG | 54.99%
Table 4: Answer accuracy of S2G and other SOTA seq2tree models on GeometryQA
(all evaluated with 5-fold cross validation).

##  6 Conclusion

In this work, we proposed a sequence-to-general tree model (S2G) that aims to
generalize previous seq2tree architectures. Our S2G can learn to generate
executable operation trees where the nodes can be formulas with arbitrary
number of arguments. By explicitly generating formulas as nodes, we make the
predicted results more interpretable. Besides, we also proposed a knowledge-
guided mechanism to guide the tree decoding using KGs and constructed a
dataset in which problems are annotated with associated formulas. Experimental
results showed that our S2G model can achieve better performance against
strong baselines.

## References

  * Bahdanau et al. (2015) Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. 2015.  Neural machine translation by jointly learning to align and translate.  _International Conference on Learning Representations_.
  * Bobrow (1964) Daniel G Bobrow. 1964.  Natural language input for a computer problem solving system.
  * Cho et al. (2014) Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. 2014.  Learning phrase representations using RNN encoder–decoder for statistical machine translation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1724–1734, Doha, Qatar. Association for Computational Linguistics.
  * Dong and Lapata (2016) Li Dong and Mirella Lapata. 2016.  Language to logical form with neural attention.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 33–43, Berlin, Germany. Association for Computational Linguistics.
  * Feigenbaum et al. (1963) Edward A Feigenbaum, Julian Feldman, et al. 1963.  _Computers and thought_.  New York McGraw-Hill.
  * Gu et al. (2016) Jiatao Gu, Zhengdong Lu, Hang Li, and Victor O.K. Li. 2016.  Incorporating copying mechanism in sequence-to-sequence learning.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1631–1640, Berlin, Germany. Association for Computational Linguistics.
  * Hosseini et al. (2014) Mohammad Javad Hosseini, Hannaneh Hajishirzi, Oren Etzioni, and Nate Kushman. 2014\.  Learning to solve arithmetic word problems with verb categorization.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 523–533.
  * Huang et al. (2018) Danqing Huang, Jing Liu, Chin-Yew Lin, and Jian Yin. 2018.  Neural math word problem solver with reinforcement learning.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 213–223.
  * Huang et al. (2017) Danqing Huang, Shuming Shi, Chin-Yew Lin, and Jian Yin. 2017.  Learning fine-grained expressions to solve math word problems.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 805–814.
  * Kingma and Ba (2015) Diederik P Kingma and Jimmy Ba. 2015.  Adam: A method for stochastic optimization.  _International Conference on Learning Representations_.
  * Kipf and Welling (2017) Thomas N Kipf and Max Welling. 2017.  Semi-supervised classification with graph convolutional networks.  _International Conference on Learning Representations_.
  * Koncel-Kedziorski et al. (2015) Rik Koncel-Kedziorski, Hannaneh Hajishirzi, Ashish Sabharwal, Oren Etzioni, and Siena Dumas Ang. 2015.  Parsing algebraic word problems into equations.  _Transactions of the Association for Computational Linguistics_ , 3:585–597.
  * Kushman et al. (2014) Nate Kushman, Yoav Artzi, Luke Zettlemoyer, and Regina Barzilay. 2014.  Learning to automatically solve algebra word problems.  In _Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 271–281.
  * Li et al. (2019) Jierui Li, Lei Wang, Jipeng Zhang, Yan Wang, Bing Tian Dai, and Dongxiang Zhang. 2019.  Modeling intra-relation in math word problems with different functional multi-head attentions.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 6162–6167.
  * Li et al. (2020) Shucheng Li, Lingfei Wu, Shiwei Feng, Fangli Xu, Fengyuan Xu, and Sheng Zhong. 2020\.  Graph-to-tree neural networks for learning structured input-output translation with applications to semantic parsing and math word problem.  In _Findings of the Association for Computational Linguistics: EMNLP 2020_ , pages 2841–2852, Online. Association for Computational Linguistics.
  * Liang et al. (2016) Chao-Chun Liang, Shih-Hong Tsai, Ting-Yun Chang, Yi-Chung Lin, and Keh-Yih Su. 2016\.  A meaning-based English math word problem solver with understanding, reasoning and explanation.  In _Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: System Demonstrations_ , pages 151–155, Osaka, Japan. The COLING 2016 Organizing Committee.
  * Liang et al. (2018) Chao-Chun Liang, Yu-Shiang Wong, Yi-Chung Lin, and Keh-Yih Su. 2018.  A meaning-based statistical English math word problem solver.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 652–662, New Orleans, Louisiana. Association for Computational Linguistics.
  * Liu et al. (2019) Qianying Liu, Wenyv Guan, Sujian Li, and Daisuke Kawahara. 2019.  Tree-structured decoding for solving math word problems.  In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_ , pages 2370–2379.
  * Manning et al. (2014) Christopher D Manning, Mihai Surdeanu, John Bauer, Jenny Rose Finkel, Steven Bethard, and David McClosky. 2014.  The stanford corenlp natural language processing toolkit.  In _Proceedings of 52nd annual meeting of the association for computational linguistics: system demonstrations_ , pages 55–60.
  * Mishra et al. (2020) Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Sachdeva, and Chitta Baral. 2020.  Towards question format independent numerical reasoning: A set of prerequisite tasks.  _arXiv preprint arXiv:2005.08516_.
  * Mitra and Baral (2016) Arindam Mitra and Chitta Baral. 2016.  Learning to use formulas to solve simple arithmetic problems.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 2144–2153.
  * Rabinovich et al. (2017) Maxim Rabinovich, Mitchell Stern, and Dan Klein. 2017.  Abstract syntax networks for code generation and semantic parsing.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1139–1149, Vancouver, Canada. Association for Computational Linguistics.
  * Roy and Roth (2018) Subhro Roy and Dan Roth. 2018.  Mapping to declarative knowledge for word problem solving.  _Transactions of the Association for Computational Linguistics_ , 6:159–172.
  * Shi et al. (2015) Shuming Shi, Yuehui Wang, Chin-Yew Lin, Xiaojiang Liu, and Yong Rui. 2015.  Automatically solving number word problems by semantic parsing and reasoning.  In _Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing_ , pages 1132–1142.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _The journal of machine learning research_ , 15(1):1929–1958.
  * Sutskever et al. (2014) Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014.  Sequence to sequence learning with neural networks.  In _Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2_ , NIPS’14, page 3104–3112, Cambridge, MA, USA. MIT Press.
  * Veličković et al. (2018) Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018.  Graph attention networks.  _International Conference on Learning Representations_.
  * Wang et al. (2017) Yan Wang, Xiaojiang Liu, and Shuming Shi. 2017.  Deep neural solver for math word problems.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 845–854.
  * Wiseman and Rush (2016) Sam Wiseman and Alexander M. Rush. 2016.  Sequence-to-sequence learning as beam-search optimization.  In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_ , pages 1296–1306, Austin, Texas. Association for Computational Linguistics.
  * Wu et al. (2020) Qinzhuo Wu, Qi Zhang, Jinlan Fu, and Xuan-Jing Huang. 2020.  A knowledge-aware sequence-to-tree network for math word problem solving.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7137–7146.
  * Xie and Sun (2019) Zhipeng Xie and Shichao Sun. 2019.  A goal-driven tree-structured neural model for math word problems.  In _IJCAI_ , pages 5299–5305.
  * Yin and Neubig (2017) Pengcheng Yin and Graham Neubig. 2017.  A syntactic neural model for general-purpose code generation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 440–450, Vancouver, Canada. Association for Computational Linguistics.
  * Zhang et al. (2019) Dongxiang Zhang, Lei Wang, Luming Zhang, Bing Tian Dai, and Heng Tao Shen. 2019\.  The gap of semantic parsing: A survey on automatic math word problem solvers.  _IEEE transactions on pattern analysis and machine intelligence_ , 42(9):2287–2305.
  * Zhang et al. (2020) Jipeng Zhang, Lei Wang, Roy Ka-Wei Lee, Yi Bin, Yan Wang, Jie Shao, and Ee-Peng Lim. 2020.  Graph-to-tree learning for solving math word problems.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 3928–3937, Online. Association for Computational Linguistics.

##  Appendix A Data Preprocessing

In this section, we describe the data preprocessing steps required for our S2G
model.

###  A.1 Converting to prefix notation

To perform top-down tree decoding, we follow Xie and Sun (2019) to convert the
equations into their prefix notation, where the operators are placed in front
of their operands, rather than in between. In this way, the order of the
equation tokens would match the order of decoding. In our case, we also need
to consider the formulas used in the equation. For a formula in the form
$"F(arg1,arg2)"$, we turn it into $"[F,arg1,arg2]"$ so that it can fit into
the prefix notation. Table 5 shows an example of this infix-to-prefix
conversion for an equation with formulas.

Problem: The outer radius and inner radius of
---
a circular annulus are 5m and 3m respectively.
Find the area of this circular annulus.
Equation: $x$ = circle_area(5) - circle_area(3)
Prefix form: [ -, circle_area, 5, circle_area, 3]
Table 5: Infix-to-prefix conversion for an equation with formulas.

###  A.2 Vocabulary

We follow the canonical sequence-to-sequence architecture Sutskever et al.
(2014) to prepare for the source vocabulary. For the target vocabulary,
however, we have to take into consideration the way that humans solve MWPs. To
solve a math problem, we use the numbers from the problem text (a dynamic
vocabulary) and the mathematical operators learned before (a static
vocabulary) and try to compose them into an equation. Sometimes, we also need
to use external constant numbers (a static vocabulary) that are not in the
problem text but would appear in the equation (e.g., 1, 2, or 3.14). These
three types of vocabulary make up the vocabulary for the equations in
arithmetic problems (equation 13).

$V_{arith}=V_{number}\cup V_{op}\cup V_{const}$ (13)

We follow Xie and Sun (2019) to use a copy mechanism Gu et al. (2016) to copy
the numbers from the problem text. Hence, we can dynamically get the problem
numbers during decoding. Besides, we extend the original vocabulary by adding
domain-specific formulas into it so that the decoder can generate formulas
during decoding (equation 14). Table 6 shows the overall vocabulary that we
use for our decoder.

$V_{target}=V_{number}\cup V_{op}\cup V_{const}\cup V_{formula}$ (14)

Vocab Type |  Instances
---|---
Operator |  +, -, *, /, ^
Number |  $\langle N0\rangle$, $\langle N1\rangle$, $\langle N2\rangle$, …
Constant |  1, 2, 3.14
*Formula |  circle_area, square_area, rectangle_perimeter, and so on.
Table 6: Types of the vocabulary.

##  Appendix B GeometryQA

Table 7 shows the 11 formulas used for annotation.

Name | Formula | # args
---|---|---
Square |  |
square_area | side * side | 1
square_perimeter | 4 * side | 1
Cubic |  |
cubic_volume | side*side*side | 1
Circle |  |
circle_area |  $\pi$ * radius^2 | 1
circumference_r | 2 * $\pi$ * radius | 1
circumference_d |  $\pi$ * diameter | 1
Triangle |  |
triangle_area | base*height / 2 | 2
Rectangle |  |
rectangle_area | length * width | 2
rectangle_perimeter | 2 * (l+w) | 2
Cuboid |  |
cuboid_volume | l* w* height | 3
cuboid_surface | 2*(l*w+w*h+l*h) | 3

Table 7: Eleven geometry formulas used in annotating GeometryQA.

##  Appendix C Related Work

In this section, we briefly introduce the progress of MWP solvers, and then we
focus on topics that are closer to our work, including seq2tree solvers and
knowledge graphs for problem solving.

###  C.1 Math Word Problem Solving

Ever since 1960s, efforts have been made to build automatic math word problem
solving systems Feigenbaum et al. (1963); Bobrow (1964). Statistical solvers
learn to map problem features into corresponding equation templates or
operations to solve the problem Kushman et al. (2014); Hosseini et al. (2014);
Mitra and Baral (2016); Liang et al. (2016, 2018); Roy and Roth (2018). For
example, Kushman et al. (2014) propose to align MWPs to their templates, while
Hosseini et al. (2014) propose to find the operations by verb categorization.
Semantic parsing approaches, on the other hand, parse the problem into
intermediate representations using semantic parsers Shi et al. (2015); Koncel-
Kedziorski et al. (2015); Huang et al. (2017).

Recently, neural architectures have emerged as a dominant paradigm in math
word problem solving. Wang et al. (2017) first attempt to use a seq2seq solver
that utilize encoder-decoder architectures to encode the problem text and then
decode into equations in a way similar to machine translation. Copy mechanism
Huang et al. (2018) or attention mechanisms Li et al. (2019) are introduced to
improvement the performance of seq2seq models. These seq2seq models, however,
suffer from producing invalid equations, like a binary operator with three
operands, because there is no grammatical constraint on its sequential
decoding. To solve this problem, seq2tree models are proposed to bring into
the grammatical constraints Xie and Sun (2019); Liu et al. (2019). We will
give a more detailed introduction to seq2tree models in Section C.2.

###  C.2 Sequence-to-Tree Models

To convert text into structured representations, several research strands have
utilized sequence-to-tree models. Dong and Lapata (2016) first use seq2tree on
semantic parsing to translate text into structured logical forms. Similar
frameworks are also adopted for code generation Yin and Neubig (2017);
Rabinovich et al. (2017) where they translate code snippets into executable
representations or abstract syntax trees (ASTs).

Inspired by their ideas, MWP solving also adopts seq2tree to map the problem
text into expression trees. This introduces a constraint that the non-leaf
nodes of the tree should be operators and leaf nodes be numbers, and thus the
resulted equations are always guaranteed to be valid. Most seq2tree solvers
choose bidirectional LSTM or GRU as their text encoder and use two separate
models to predict the left and right nodes during decoding respectively Xie
and Sun (2019); Zhang et al. (2020); Wu et al. (2020); Li et al. (2020). Our
model differs from the previous that we use a single RNN-based decoder to
predict a variable number of children nodes during decoding. In addition, our
model can predict formulas as nodes that increase the interpretability of the
model outputs, while previous solvers can only predict basic arithmetic
operators.

###  C.3 Knowledge Graph for Math Word Problem Solving

To incorporate external knowledge into problem solving, some solvers utilize
graph convolutional networks Kipf and Welling (2017) or graph attention
networks Veličković et al. (2018) to encode knowledge graphs (KGs) as an
additional input. Wu et al. (2020) proposed to incorporate commonsense
knowledge from external knowledge bases. They constructed a dynamic KG for
each problem to model the relationship between the entities in the problem.
For example, ”daisy” and ”rose” would be linked to their category ”flower” so
that the solver can use this hyperonymy information when counting the number
of flowers. Zhang et al. (2020) proposed to build graphs that model the
quantity-related information using dependency parsing and POS tagging tools
Manning et al. (2014). Their graphs provide better quantity representations to
the solver. Our model differs from previous models that we aim to incorporate
domain knowledge from mathematical KGs rather than from commonsense knowledge
bases.