$\displaystyle\left(n^{n_{1}}_{1},n_{1}+x_{1},n_{1}+x_{1}-1,\ldots,n_{2}+1,n^{n_{2}}_{2},n_{2}+x_{2},\ldots,n_{3}+1,n^{n_{3}}_{3},n_{3}+x_{3},\ldots,n_{3}+1,n_{4}+x_{4},\ldots\right)$

where $x_{i}\geq n_{i+1}-1$ and $n_{i}\geq 2$. Then, $\Lambda$ is a dominant
Auslander-regular algebra of global dimension three and dominant dimensions of
projective modules are either three or one.

###### Proof.

By direct computation, $\Lambda$ is non-defect invariant reverse of
$\Theta\cong\mathbb{A}_{n_{1}}\oplus\mathbb{A}_{n_{2}}\oplus\cdots\oplus\mathbb{A}_{n_{k}}$.
Hence global dimension of $\Lambda$ is three. Every projective module having
proper injective quotient is of length $n_{i}+x_{i}$. Hence, projective and
co-dominant dimensions of an injective $I$ with $\ell(I)\leq
n_{i}+x_{i}-n_{i+1}$ is one. If $\ell(I)>n_{i}+x_{i}-n_{i+1}$, then
$0<\ell(\Omega^{1}(I))<n_{i+1}-1$. Hence $\ell(\Omega^{2}(I))<n_{i+1}+1$ which
is not projective, and its projective cover is $P$ of length
$n_{i+1}+x_{i+1}$. $\Omega^{3}(I)$ is projective by comparing lengths of $P$
and $\Omega^{2}(I)$. In particular, all projective covers are projective-
injective, hence codominant dimension of $I$ is three. Hence $\Lambda$ is a
dominant Auslander-regular algebra. ∎

######  Proposition 4.5.7.

Let $\Lambda$ be a cyclic Nakayama algebra of global dimension three. If
$\Lambda$ is dominant Auslander-regular algebra satisfying
$\operatorname{p\\!\dim}I=3$ and $\operatorname{p\\!\dim}I^{\prime}=2$, then
it is given by

$\displaystyle((1+g(n_{1})+(g(n_{2})-1)\delta(n_{1},1))^{g(n_{1})},n_{1}+g(n_{2}),n_{1}+g(n_{2})-1,\ldots,1+g(n_{2}),$
$\displaystyle(1+g(n_{2})+(g(n_{3})-1)\delta(n_{2},1))^{g(n_{2})},n_{2}+g(n_{3}),n_{2}+g(n_{3})-1,\ldots,1+g(n_{3})$
$\displaystyle(1+g(n_{3})+(g(n_{4})-1)\delta(n_{3},1))^{g(n_{3})},n_{3}+g(n_{4}),n_{3}+g(n_{4})-1,\ldots,1+g(n_{4}),\ldots$
$\displaystyle\ldots$
$\displaystyle\ldots(1+g(n_{k})+(g(n_{1})-1)\delta(n_{k},1))^{g(n_{k})},n_{k}+g(n_{1}),n_{k}+g(n_{1})-1,\ldots,1+g(n_{k}))$

where $\delta$ is Kronecker-delta function and

$g(n)=\begin{cases}n-1&\text{if}\,\,n\geq 2\\\ 1&\text{if}\,\,n=1\end{cases}$

###  4.6. Higher Auslander Algebras among Nakayama Algebras

######  Proposition 4.6.1.

Let $\Lambda$, $\Theta$ be Nakayama algebras such that
$\bm{\varepsilon}(\Lambda)\cong\Theta$. $\Lambda$ is a higher Auslander
algebra of global dimension three if and only if
$\Theta\cong\mathbb{A}_{n_{1}}\oplus\mathbb{A}_{n_{2}}\oplus\cdots\oplus\mathbb{A}_{n_{k}}$
with $n_{j}\geq 2$. In this case, $\Lambda$ is given by Kupisch series

$\displaystyle\left(n^{n_{1}}_{1},n_{1}+n_{2}-1,n_{1}+n_{2}-2,\ldots,n_{2}+1,n^{n_{2}}_{2},n_{2}+n_{3}-1,\ldots,n_{3}+1,n^{n_{3}}_{3},n_{3}+n_{4}-1,\ldots\right).$

###### Proof.

Global dimension of $\Lambda$ is three, therefore global dimension of
$\bm{\varepsilon}(\Lambda)$ is one. So, connected components of syzygy
filtered algebra are either $\mathbb{A}$-type quivers or simple algebras,
hence
$\Theta\cong\mathbb{A}_{n_{1}}\oplus\mathbb{A}_{n_{2}}\oplus\cdots\oplus\mathbb{A}_{n_{k}}$.
If at least one component is simple, i.e., $n_{j}=1$ for some $j$, then there
is no defect invariant reverse of $\Theta$. By Theorem B 2 we would get
$\operatorname{dom.\\!dim}\Lambda\leq 2$, which contradicts to $\Lambda$ is a
higher Auslander algebra of global dimension three. Therefore, each $n_{j}\geq
2$ . In this case, by proposition 3.1.1, we obtain Kupisch series of
$\Lambda$. ∎

######  Remark 4.6.2.

Let $\Lambda$ be a cyclic Nakayama algebra given by Kupisch series as in
proposition 4.6.1. Then, $\Gamma:=\operatorname{End}_{\Lambda}(Q)^{op}$ is
given by

$\displaystyle\left(n_{1},n_{1},n_{1}-1,\ldots,2,n_{2},n_{2},n_{2}-1,\ldots,2,n_{3},n_{3},n_{3}-1,\ldots,2,\ldots,n_{k},n_{k},n_{k}-1,\ldots,2\right)$

where $Q$ is projective-injective generator of $\Lambda$. $\Gamma$ has
$2$-cluster-tilting object. Moreover, any cyclic Nakayama algebra $\Gamma$
having $2$-cluster-tilting object ${\mathcal{M}}$ such that
${\mathcal{M}}\subseteq\Gamma\oplus D\Gamma\oplus\operatorname{rad}\Gamma$ is
given by 4.7.

####  4.6.1. Global dimension four higher Auslander algebras

We give complete classification of cyclic Nakayama algebras of global
dimension four which are higher Auslander. There are two cases we need to
consider.

######  Remark 4.6.3.

Let $\Theta$ be an Auslander algebra. If $\Theta$ is cyclic, then it is given
by Kupisch series $(2323,\ldots,23)$. If $\Theta$ is linear, then it is given
by Kupisch series $((23)^{k}221)$ where $k\geq 0$.

######  Proposition 4.6.4.

Let $\Theta$ be given by Kupisch series $(23)^{k}$. Then, defect invariant
reverse is given by Kupisch series $(344)^{k}$.

###### Proof.

By proposition 3.1.1, result follows. ∎

######  Proposition 4.6.5.

Let $\mathbb{B}_{k}$ denote the algebra given by Kupisch series
$((23)^{k}221)$. If
$\Theta\cong\mathbb{B}_{n_{1}}\oplus\mathbb{B}_{n_{2}}\oplus\cdots\oplus\mathbb{B}_{n_{k}}$,
then the defect invariant reverse is given by Kupisch series

$\displaystyle\left((344)^{n_{1}-1}333\right)^{\delta(n_{1})}232,2+\delta(n_{2}),\left((344)^{n_{2}-1}333\right)^{\delta(n_{2})}232,2+\delta(n_{3}),\ldots,$
$\displaystyle\ldots,\left((344)^{n_{k}-1}333\right)^{\delta(n_{k})}232,2+\delta(n_{1}))$

where

$\delta(n)=\begin{cases}0&\text{if}\,\,n=0\\\ 1&\text{if}\,\,n\neq
0\end{cases}$

###### Proof.

It follows from proposition 3.1.1. ∎

######  Remark 4.6.6.

Let $\Lambda$ be a cyclic Nakayama algebra given by Kupisch series as in
proposition 4.6.5. Then, $\Gamma:=\operatorname{End}_{\Lambda}(Q)^{op}$ is
given by

(4.7)
$\displaystyle\left(2(33)^{n_{1}},2,2,(33)^{n_{2}},\ldots,2(33)^{n_{k}}2\right)$

where $Q$ is projective-injective generator of $\Lambda$. $\Gamma$ has
$3$-cluster-tilting object. Moreover, any cyclic Nakayama algebra $\Gamma$
having $3$-cluster-tilting object ${\mathcal{M}}$ such that
${\mathcal{M}}\subseteq\Gamma\oplus D\Gamma\oplus\operatorname{rad}\Gamma$ is
given by 4.7.

##  5\. The duality of syzygy filtered algebras

In this section, we reveal a duality between the syzygy filtered algebras
$\bm{\varepsilon}(\Lambda)$ and the cosyzygy filtered algebra $\eta(\Lambda)$
for cyclic Nakayama algebra $\Lambda$. We will show equivalent definitions for
these algebras first, and then establish a connection between them via minimal
projective and minimal injective modules.

###  5.1. Construction

Let $\Lambda$ be a cyclic Nakayama algebra. Denote by
$\Lambda\operatorname{\textendash mod}$ the category of finitely generated
left $\Lambda$ modules, ${\mathcal{P}}$ the subcategory of projective modules,
${\mathcal{I}}$ the subcategory of injective modules and ${\mathcal{Q}}$ the
subcategory of projective-injective modules. Assume $Q_{i}$ is an
indecomposable projective-injective module, denote by $S_{i}$ the (unique)
simple submodule of $Q_{i}$ and $T_{i}$ the unique simple quotient module of
$Q_{i}$. Let ${\mathcal{S}}=\\{S_{i}\\}$ and ${\mathcal{T}}=\\{T_{i}\\}$.

Recall that a module is torsionless, if it is a submodule of a projective
module. A simple module $S\in{\mathcal{S}}$ if and only if $S$ is torsionless.

######  Lemma-Definition 5.1.1.

Let $S$ be a simple submodule of an indecomposable projective-injective module
$Q$. Then there exists a unique largest submodule $\Delta(S)$ of $Q$ such that
$\operatorname{soc}\Delta(S)=S$ and every quotient of $\Delta(S)$ is not
torsionless.

###### Proof.

Assume $k>0$ is the smallest integer such that $\tau^{-(k+1)}S$ is
torsionless. Then $\Delta$ is the unique indecomposable submodule of $Q$ (of
length $k$) with composition factors $S,\tau^{-1}S,\cdots,\tau^{-k}S$. ∎

The syzygy filtration technique was introduced by Sen in [27], [28], where the
set of modules ${\mathcal{B}}(\Lambda)=\\{\Delta(S)|S\in{\mathcal{S}}\\}$
(called _the base-set_) plays an important role in studying projective
resolutions of modules over Nakayama algebras: Let $M$ be an indecomposable
non-projective $\Lambda$ module. If its second syzygy is nontrivial, then
$\Omega^{2}(M)$ has a unique filtration by ${\mathcal{B}}(\Lambda)$.
Therefore, the subcategory $Filt({\mathcal{B}}(\Lambda))$ consisting of
modules with ${\mathcal{B}}(\Lambda)$ filtrations controls second (and higher)
syzygies of modules over Nakayama algebras. To understand the subcategory
$Filt({\mathcal{B}}(\Lambda))$, Sen introduced the following definition of
syzygy filtered algebra:

######  Definition 5.1.2.

[28] For any cyclic Nakayama algebra $\Lambda$, the syzygy filtered algebra
$\bm{\varepsilon}(\Lambda)$ is the endomorphism algebra:

$\displaystyle\bm{\varepsilon}(\Lambda):=\operatorname{End}_{\Lambda}\left(\bigoplus\limits_{S\in{\mathcal{S}}}P(\tau
S)\right).$

The name of syzygy filtered algebra comes from the following result.

######  Theorem 5.1.3.

[28] Denote by $Filt({\mathcal{B}}(\Lambda))$ the full subcategory of
${\mathcal{B}}(\Lambda)$-filtered (right) $\Lambda$-modules and
${\mathcal{P}}=\bigoplus\limits_{S\in{\mathcal{S}}}P(\tau S)$. Let
$\bm{\varepsilon}(\Lambda)\operatorname{\textendash mod}$ be the category of
finitely generated right $\bm{\varepsilon}(\Lambda)$-modules. Then
$\operatorname{Hom}({\mathcal{P}},-):Filt({\mathcal{B}}(\Lambda))\to\bm{\varepsilon}(\Lambda)\operatorname{\textendash
mod}$ is an equivalence.

Throughout this section, $\Lambda$ is a cyclic Nakayama algebra. Denote by
${\mathcal{S}}=\\{\operatorname{soc}Q|Q$ is an indecomposable projective-
injective module$\\}$ and ${\mathcal{T}}=\\{\operatorname{top}Q|Q$ is an
indecomposable projective-injective module$\\}$.

###  5.2. The syzygy filtered algebra

######  Definition 5.2.1.

[28, 25] We call an indecomposable projective $\Lambda$-module $X$ minimal
projective if $\operatorname{rad}X$ is not projective. An indecomposable
injective module $Y$ is minimal injective if $Y/\operatorname{soc}Y$ is not
injective. Denote by ${\mathcal{X}}$ and ${\mathcal{Y}}$ the set of all the
minimal projectives and all the minimal injectives respectively.

######  Proposition 5.2.2.

The set ${\mathcal{X}}=\\{P(\tau^{-}T)|T\in{\mathcal{T}}\\}$ and the set
${\mathcal{Y}}=\\{I(\tau S)|S\in{\mathcal{S}}\\}$.

Realize that the Nakayama functor is a quasi-isomorphism between projectives
and injectives. We have the following equivalent definition of the syzygy
filtered algebras.

######  Corollary 5.2.3.

The syzygy filtered algebra
$\bm{\varepsilon}(\Lambda)\cong\operatorname{End}_{\Lambda}\left(\bigoplus\limits_{S\in{\mathcal{S}}}I(\tau
S)\right)=\operatorname{End}_{\Lambda}\left(\bigoplus\limits_{Y\in{\mathcal{Y}}}Y\right)$

###  5.3. The cosyzygy filtered algebra

Dually, we can consider the cosyzygy filtered algebra.

######  Lemma-Definition 5.3.1.

Let $T$ be the simple top of an indecomposable projective-injective module
$Q$. Then there exists a unique largest quotient module $\nabla(T)$ of $Q$
such that $\operatorname{top}\nabla(T)=T$ and every submodule of $\nabla(T)$
is not a quotient of an injective module.

###### Proof.

Similar to Lemma-Definition 5.1.1. ∎

Let ${\mathcal{B}}^{\circ}(\Lambda)=\\{\nabla(T)|T\in{\mathcal{T}}\\}$ be the
set of $\nabla$-modules, which are the composition factors of second (or
higher) cosyzygy modules. i.e. for an indecomposable non-injective $\Lambda$
module $M$, if its second cosyzygy is nontrivial, then $\Omega^{-2}(M)$ has a
unique filtration by ${\mathcal{B}}^{\circ}(\Lambda)$.

######  Definition 5.3.2.

For any cyclic Nakayama algebra $\Lambda$, the cosyzygy filtered algebra
$\bm{\eta}(\Lambda)$ is the endomorphism algebra:

$\displaystyle\bm{\eta}(\Lambda):=\operatorname{End}_{\Lambda}\left(\bigoplus\limits_{T\in{\mathcal{T}}}I(\tau^{-}T)\right)^{\operatorname{op}}.$

Notice that we have to take the opposite algebra here, because of the
following:

######  Theorem 5.3.3.

Denote by $Filt({\mathcal{B}}^{\circ}(\Lambda))$ the full subcategory of
${\mathcal{B}}^{\circ}(\Lambda)$-filtered (right) $\Lambda$-modules and
$\hat{I}=\bigoplus\limits_{T\in{\mathcal{T}}}I(\tau^{-}T)$. Let
$\eta(\Lambda)\operatorname{\textendash mod}$ be the category of finitely
generated right $\eta(\Lambda)$ modules. Then
$D\operatorname{Hom}(-,\hat{I}):Filt({\mathcal{B}}^{\circ}(\Lambda))\to\eta(\Lambda)\operatorname{\textendash
mod}$ is an equivalence, where $D=\operatorname{Hom}(-,k)$.

###### Proof.

By definitions, the functor $D=\operatorname{Hom}(-,k)$ induces a duality:
$Filt{\mathcal{B}}^{\circ}(\Lambda)\to
Filt{\mathcal{B}}(\Lambda^{\operatorname{op}})$. It follows that $D\hat{I}$ is
a projective generator of $Filt{\mathcal{B}}(\Lambda^{op})$. Hence, by Theorem
5.1.3,
$\operatorname{Hom}_{\Lambda^{\operatorname{op}}}(D\hat{I},-):Filt{\mathcal{B}}(\Lambda^{\operatorname{op}})\to\bm{\varepsilon}(\Lambda^{\operatorname{op}})\operatorname{\textendash
mod}$ is an equivalence. On the other hand,
$\bm{\varepsilon}(\Lambda^{\operatorname{op}})=\operatorname{End}_{\Lambda^{op}}(D\hat{I})\cong\operatorname{End}_{\Lambda}(\hat{I})=\eta(\Lambda)^{\operatorname{op}}$.
In the following diagram:

$\textstyle{Filt{\mathcal{B}}^{\circ}(\Lambda)\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{D}$$\scriptstyle{F}$$\textstyle{Filt{\mathcal{B}}(\Lambda^{\operatorname{op}})\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{\operatorname{Hom}_{\Lambda^{\operatorname{op}}}(D\hat{I},-)}$$\textstyle{\eta(\Lambda)\operatorname{\textendash
mod}\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\scriptstyle{D}$$\textstyle{\bm{\varepsilon}(\Lambda^{\operatorname{op}})\operatorname{\textendash
mod}}$

take
$F=D\circ\operatorname{Hom}_{\Lambda^{\operatorname{op}}}(D\hat{I},-)\circ
D=D\operatorname{Hom}{\Lambda^{\operatorname{op}}}(D\hat{I},D-)\cong
D\operatorname{Hom}_{\Lambda}(-,\hat{I})$, which is the desired equivalence
functor. ∎

Due to Proposition 5.2.2, we have the following equivalent definition of the
cosyzygy filtered algebras.

######  Corollary 5.3.4.

The cosyzygy filtered algebra

$\eta(\Lambda)\cong\operatorname{End}_{\Lambda}\left(\bigoplus\limits_{T\in{\mathcal{T}}}P(\tau^{-}T)\right)^{\operatorname{op}}=\operatorname{End}_{\Lambda}\left(\bigoplus\limits_{X\in{\mathcal{X}}}X\right)^{\operatorname{op}}.$

###  5.4. Connecting Syzygy and Cosyzygy Filtered Algebras

Finally, we are going to show the duality between the syzygy filtered algebra
and the cosyzygy filtered algebra. First, we need to fix a notation for some
irreducible morphisms:

Given an indecomposable module $M$, denote by $i_{M}:\operatorname{rad}M\to M$
the embedding and $p_{M}:M\to M/\operatorname{soc}M$ the quotient map. Both
$i_{M}$ and $p_{M}$ are unique up to isomorphism.

######  Lemma 5.4.1.

Let $f:M\to N$ be homomorphism between indecomposable modules. If $f$ is
neither a monomorphism nor an epimorphism, then up to isomorphism $f$ can be
uniquely decomposed as a composition: $f=i_{N}\circ f^{\prime}\circ p_{M}$.

###### Proof.

Let
$f:M\stackrel{{\scriptstyle\pi}}{{\to}}\operatorname{Im}f\stackrel{{\scriptstyle\rho}}{{\to}}N$
be the canonical epi-mono decomposition. Since $f$ is not an epimorphism,
$\rho$ factors through $\operatorname{rad}N$. i.e.
$\rho=i_{N}\circ\rho^{\prime}$. Since $f$ is not a monomorphism either, $\pi$
factors through $M/\operatorname{soc}M$. i.e. $\pi=\pi^{\prime}\circ p_{M}$.
Hence set $f^{\prime}=\rho^{\prime}\circ\pi^{\prime}$, it follows that
$f=\rho\circ\pi=i_{N}\circ\rho^{\prime}\circ\pi^{\prime}\circ p_{M}=i_{N}\circ
f^{\prime}\circ p_{M}$. The uniqueness of the decomposition follows
immediately from the fact that the monomorphism $i_{N}$ and the epimorphism
$p_{M}$ are unique. ∎

Notice that if $M$ is a minimal injective or $N$ is a minimal projective, any
non-isomorphism $f:M\to N$ is neither a monomorphism nor an epimorphism.

######  Proposition 5.4.2.

For each minimal projective module $X$, there exists a unique minimal
injective module $Y$ such that
$Y/\operatorname{soc}Y\cong\operatorname{rad}X$. Furthermore, such a
correspondence $X\mapsto Y$ is a bijection between ${\mathcal{X}}$ and
${\mathcal{Y}}$.

###### Proof.

See [25] Appendix C. ∎

######  Definition 5.4.3.

Define $\tilde{\tau}:{\mathcal{X}}\to{\mathcal{Y}}$ a bijection as the
following: for any minimal projective X, $\tilde{\tau}X$ is the unique module
in ${\mathcal{Y}}$ such that
$\tilde{\tau}X/\operatorname{soc}\tilde{\tau}X\cong\operatorname{rad}X$.

######  Lemma 5.4.4.

The map $\tilde{\tau}$ induces an equivalence
$F:\operatorname{add}{\mathcal{X}}\to\operatorname{add}{\mathcal{Y}}$.

###### Proof.

Let $F:\operatorname{add}{\mathcal{X}}\to\operatorname{add}{\mathcal{Y}}$ be
an additive functor, which on the objects $F(X)=\tilde{\tau}X$. Suppose there
is a homomorphism $f:X_{1}\to X_{2}$, where $X_{1}$ and $X_{2}$ are minimal
projectives. If $f$ is an isomorphism, then define $F(f):F(X_{1})\to F(X_{2})$
to be an isomorphism. Otherwise, $f$ is neither a monomorphism nor an
epimorphism. According to Lemma 5.4.1, there exists a unique decomposition
$f=i_{X_{2}}\circ f^{\prime}\circ p_{X_{1}}$. Here
$f^{\prime}\in\operatorname{Hom}_{\Lambda}(X_{1}/\operatorname{soc}X_{1},\operatorname{rad}X_{2})$
and both $X_{1}/\operatorname{soc}X_{1}$ and $\operatorname{rad}X_{2}$ are
neither projective nor injective. Hence the Auslander-Reiten translation
$\tau$ induces a bijection:

$\textstyle{\operatorname{Hom}_{\Lambda}(X_{1}/\operatorname{soc}X_{1},\operatorname{rad}X_{2})\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{\operatorname{Hom}_{\Lambda}(\tau(X_{1}/\operatorname{soc}X_{1}),\tau(\operatorname{rad}X_{2}))\ignorespaces\ignorespaces\ignorespaces\ignorespaces}$$\textstyle{\operatorname{Hom}_{\Lambda}(FX_{1}/\operatorname{soc}FX_{1},\operatorname{rad}FX_{2})}$

Therefore, we can define $F(f)=i_{FX_{2}}\circ\tau f^{\prime}\circ
p_{FX_{1}}$.

$FX_{1}$$X_{1}$$FX_{2}$$X_{2}$$\operatorname{Im}F(f)$$\operatorname{Im}f$

Next we want to show that $F$ is fully-faithful.

Let $f:X_{1}\to X_{2}$ be a non-isomorphism, where $X_{1}$ and $X_{2}$ are
minimal projectives. Assume $F(f)=0$. In particular, $f$ is not an
isomorphism. Due to the definition, $F(f)=i_{FX_{2}}\circ\tau f^{\prime}\circ
p_{FX_{1}}$. So $0=\operatorname{Im}F(f)=\operatorname{Im}\tau
f^{\prime}=\tau\operatorname{Im}f^{\prime}=\tau\operatorname{Im}f$, which
implies $f=0$.

Let $g:FX_{1}\to FX_{2}$ be a homomorphism between minimal injectives. If $g$
is an isomorphism, $g=F(f)$ for some isomorphism $f:X_{1}\to X_{2}$.
Otherwise, there is a decomposition $g=i_{FX_{2}}\circ g^{\prime}\circ
p_{FX_{1}}$, where
$g^{\prime}:FX_{1}/\operatorname{soc}FX_{1}\to\operatorname{rad}FX_{2}$ and
both $FX_{1}/\operatorname{soc}FX_{1}$ and $\operatorname{rad}FX_{2}$ are
neither projective nor injective. Hence we can take
$f=i_{X_{2}}\circ\tau^{-}g^{\prime}\circ p_{X_{1}}$ and it is easy to check
that $F(f)=g$. ∎

######  Theorem 5.4.5.

Let $\Lambda$ be a cyclic Nakayama algebra and $\bm{\varepsilon}(\Lambda)$,
$\eta(\Lambda)$ be its syzygy filtered algebra and cosyzygy filtered algebra.
Then we have the following algebra isomorphism:

$\bm{\varepsilon}(\Lambda)\cong\eta(\Lambda)^{\operatorname{op}}.$

###### Proof.

$\bm{\varepsilon}(\Lambda)\stackrel{{\scriptstyle Corollary\ref{eps
inj}}}{{\cong}}\operatorname{End}\left(\bigoplus\limits_{Y\in{\mathcal{Y}}}Y\right)\stackrel{{\scriptstyle
Lemma\ref{conn
equiv}}}{{\cong}}\operatorname{End}\left(\bigoplus\limits_{X\in{\mathcal{X}}}X\right)\stackrel{{\scriptstyle
Corollary\ref{eta pr}}}{{\cong}}\eta(\Lambda)^{\operatorname{op}}.$

∎

###  5.5. A further question

We should point out that the $\bm{\varepsilon}(\Lambda)$-$\eta(\Lambda)$
duality may exist in a more general context. Let $\Lambda$ be a finite
dimensional algebra and $\hat{Q}$ the (basic) additive generator of
projective-injective modules. When $\operatorname{dom.\\!dim}\Lambda\geq 1$,
${\mathcal{T}}=\operatorname{Gen}(\hat{Q})$ is a torsion class. Its
corresponding wide subcategory (or thick subcategory)
${\mathcal{U}}=\\{X\in{\mathcal{T}}:\forall(g:Y\to
X)\in{\mathcal{T}},\,\ker(g)\in{\mathcal{T}}\\}$ is an abelian category and
hence is equivalent to $\eta(\Lambda)\operatorname{\textendash mod}$ for some
finite dimensional algebra $\eta(\Lambda)$. Such algebras have been studied by
Asai [2]. At the same time, ${\mathcal{F}}=\operatorname{Cogen}(\hat{Q})$ is a
torsion-free class. Its corresponding wide subcategory
${\mathcal{V}}=\\{X\in{\mathcal{F}}:\forall(g:X\to
Y)\in{\mathcal{F}},\,\operatorname{coker}(g)\in{\mathcal{F}}\\}$ is an abelian
category and hence is equivalent to
$\epsilon(\Lambda)\operatorname{\textendash mod}$ for some finite dimensional
algebra $\epsilon(\Lambda)$. It is still an open question to us whether
$\bm{\varepsilon}(\Lambda)\cong\eta(\Lambda)^{\operatorname{op}}.$

## References

  * [1] T. Adachi, O. Iyama, and I. Reiten.  $\tau$-tilting theory.  Compos. Math., 150(3):415–452, 2014.
  * [2] S. Asai.  Semibricks.  Int. Math. Res. Not, 07 2018.
  * [3] M. Auslander, M. Platzeck, G. Todorov,  Homological theory of idempotent ideals  Transactions of the American Mathematical Society, Volume 332, Number 2 , August 1992.
  * [4] M.Auslander, I. Reiten, S. Smalo.  Representation theory of Artin algebras.  , Cambridge Studies in Advanced Mathematics, 1995.
  * [5] M. Auslander and S. O. Smalø.  Almost split sequences in subcategories.  J. Algebra, 69:426–454, 1981.
  * [6] Emily Barnard, Andrew Carrol, Shijie Zhu.  Minimal inclusions of torsion classes.  Algebraic Combinatorics, 2(5), 879–901, 2019.
  * [7] Emily Barnard, Gordana Todorov, Shijie Zhu.  Dynamical combinatorics and torsion classes.  Preprint available: arXiv:1911.10712, 2019.
  * [8] A. Chan, O. Iyama, R. Marczinzik  Dominant Auslander-Gorenstein algebras and mixed cluster tilting.  arxiv preprint 2210.06180
  * [9] H. Chen and C. Xi,  Dominant dimensions, derived equivalences and tilting modules,  Israel J. Math., 215 (2016), 349?C395.
  * [10] Laurent Demonet, Osamu Iyama, Nathan Reading, Idun Reiten, and Hugh Thomas.  Lattice theory of torsion classes.  Preprint available: arXiv:1711.01785, 03 2018.
  * [11] W. Gustafson. Global dimension in serial rings. Journal of Algebra 97.1, 14-16 1985
  * [12] K. Igusa, G. Todorov. On the finitistic global dimension conjecture for artin algebras. Representations of algebras and related topics, 201-204. Fields Inst. Commun., 45, Amer. Math. Soc., Province, RI, 2005
  * [13] O. Iyama.  Auslander correspondence.  Advances in Mathematics, 2007.
  * [14] O. Iyama, O. Solberg. ”Auslander-Gorenstein algebras and precluster tilting.” Advances in Mathematics 326 (2018): 200-240.
  * [15] D. Madsen,R. Marczinzik, G. Zaimi.  On the classification of higher Auslander algebras for Nakayama algebras.  Journal of Algebra, 2020.
  * [16] Frederik Marks and Jan Šťovíček.  Torsion classes, wide subcategories and localisations.  Bull. Lond. Math. Soc., 49(3):405-416, 2017.
  * [17] K. Morita,  Duality for modules and its applications in the theory of rings with minimum condition,  Sci. Rep. Tokyo Daigaku A, 6 (1958), 83-142.
  * [18] B.J. Mueller,  The classification of algebras by dominant dimension,  Canadian Journal of Mathematics, 20 (1968), 398-409.
  * [19] Van Nguyen, Idun Reiten, Gordana Todorov, Shijie Zhu,  Dominant dimension and tilting modules,  Mathematische Zeitschrift, 292: 947-973 2018, DOI: 10.1007/s00209-018-2111-4.
  * [20] Matthew Pressland, Julia Sauter,  Special tilting modules for algebras with positive dominant dimension  Preprint available: arxiv:1705.03367v2, 2020.
  * [21] C. Ringel.  Representations of $K$-species and bimodules.  Journal of Algebra, 1976.
  * [22] Claus Michael Ringel.  Reflection functors for hereditary algebras.  J. London Math. Soc. (2), 21(3):465–479, 1980.
  * [23] Claus Michael Ringel.  The Catalan combinatorics of the hereditary Artin algebras.  In Recent developments in representation theory, volume 673 of Contemp. Math., pages 51-177. Amer. Math. Soc., Providence, RI, 2016.
  * [24] H. Tachikawa,  On dominant dimension of QF-3 algebras,  Trans. Amer. Math. Soc., 112 (1964), 249–266.
  * [25] CM. Ringel.  The finitistic dimension of a Nakayama algebra  Journal of Algebra, 2021 .
  * [26] CM. Ringel.  Linear nakayama algebras which are higher auslander algebras.  Communications in Algebra, 50(11):4842–4881, 2022.
  * [27] E. Sen.  The $\varphi$-dimension of cyclic nakayama algebras.  Communications in Algebra 49 (6), 2278-2299, 2021
  * [28] E. Sen.  Syzygy filtrations of cyclic nakayama algebras.  arXiv preprint arXiv:1903.04645, 2019.
  * [29] E. Sen.  Nakayama algebras which are higher auslander algebras.  arXiv preprint arXiv:2009.03383, 2020.
  * [30] E. Sen.  Delooping level of nakayama algebras.  Archiv der Mathematik, 117(2):141–146, 2021.


# Large Language Models Are Cross-Lingual Knowledge-Free Reasoners

Peng Hu$\clubsuit$, Sizhe Liu$\clubsuit$11footnotemark: 1, Changjiang
Gao$\clubsuit$, Xin Huang$\diamond$, Xue Han$\diamond$,
Junlan Feng$\diamond$, Chao Deng$\diamond$, Shujian Huang$\clubsuit$
$\clubsuit$National Key Laboratory for Novel Software Technology, Nanjing
University
$\diamond$China Mobile Research, Beijing, China
{hup, liusz<EMAIL_ADDRESS><EMAIL_ADDRESS>
{huangxinyjy, hanxueai, fengjunlan<EMAIL_ADDRESS>*Equal
contribution

###### Abstract

Large Language Models have demonstrated impressive reasoning capabilities
across multiple languages. However, the relationship between capabilities in
different languages is less explored. In this work, we decompose the process
of reasoning tasks into two separated parts: knowledge retrieval and
knowledge-free reasoning, and analyze the cross-lingual transferability of
them. With adapted and constructed knowledge-free reasoning datasets, we show
that the knowledge-free reasoning capability can be nearly perfectly
transferred across various source-target language directions despite the
secondary impact of resource in some specific target languages, while cross-
lingual knowledge retrieval significantly hinders the transfer. Moreover, by
analyzing the hidden states and feed-forward network neuron activation during
the reasoning tasks, we show that higher similarity of hidden representations
and larger overlap of activated neurons could explain the better cross-lingual
transferability of knowledge-free reasoning than knowledge retrieval. Thus, we
hypothesize that knowledge-free reasoning embeds in some language-shared
mechanism, while knowledge is stored separately in different languages. 111Our
code and data is available at: https://github.com/NJUNLP/Knowledge-Free-
Reasoning.

Large Language Models Are Cross-Lingual Knowledge-Free Reasoners

Peng Hu$\clubsuit$††thanks: *Equal contribution, Sizhe
Liu$\clubsuit$11footnotemark: 1, Changjiang Gao$\clubsuit$, Xin
Huang$\diamond$, Xue Han$\diamond$, Junlan Feng$\diamond$, Chao
Deng$\diamond$, Shujian Huang$\clubsuit$ $\clubsuit$National Key Laboratory
for Novel Software Technology, Nanjing University $\diamond$China Mobile
Research, Beijing, China {hup, liusz<EMAIL_ADDRESS><EMAIL_ADDRESS>{huangxinyjy, hanxueai, fengjunlan,
<EMAIL_ADDRESS>

##  1 Introduction

Large language models (LLMs) today have shown strong multitask and
multilingual performance in various domains Huang and Chang (2022), including
robust reasoning capabilities across multiple languages Ranaldi and Zanzotto
(2023), even for low-resource languages in the training corpus Shi et al.
(2022).

Previous study reveals that these multilingual LLMs possess certain ability of
multilingual transfer Ye et al. (2023), which means the skills or knowledge
learned with one language can be automatically transferred to another language
without extra training. However, the effect of such cross-lingual transfer
varies across tasks. In knowledge-related tasks, current LLMs show
unsatisfactory cross-lingual transfer Gao et al. (2024), while in certain
reasoning tasks, more effective transfer is observed Qi et al. (2023).
Previous study lacks the analysis on the difference between these tasks, and
does not dig further into the specific factors affecting the transfer
effectiveness.

In this study, we divide a reasoning task into two separated components:
knowledge retrieval and knowledge-free reasoning, and investigate the transfer
effectiveness of them in current LLMs. The former means recalling information
encountered during training, while the latter refers to organizing the given
knowledge in the context into a final answer.

Based on such definition, we perform evaluation and interpretability analysis
on LLaMA-2 models Touvron et al. (2023) to compare the cross-lingual
transferability of knowledge retrieval and knowledge-free reasoning. In the
evaluation part, by controlling the requirement of knowledge retrieval in the
StrategyQA (Geva et al., 2021) dataset and constructing a knowledge-free
reasoning dataset, we compare the cross-lingual transfer effectiveness of
knowledge-related and knowledge-free reasoning. In the interpretability
analysis part, we assess the cross-lingual computational similarity of hidden
states and Feed-Forward Network (FFN) neural activation to trace the
computational process of cross-lingual transfer of knowledge-related and
knowledge-free reasoning in LLMs. Our main findings are:

  * •

Knowledge retrieval significantly hinders cross-lingual transfer. The more
knowledge retrieval is required in the task, the lower effectiveness of cross-
lingual transfer is observed.

  * •

The knowledge-free reasoning ability can be effectively transferred to other
languages after finetuning in one, while the model’s ability in these
languages are less important.

  * •

The overall cross-lingual computational similarity of knowledge-free reasoning
is significantly higher than knowledge-related reasoning. Also, the difference
in such similarity level is the most significant in the middle-high layers of
the model. This suggests a language-shared mechanism of knowledge-free
reasoning in multilingual LLMs.

##  2 Methods

The work in this paper consists of two parts: evaluation and interpretability
analysis. For each part, we employ specific datasets and design certain
metrics or measurements.

###  2.1 Datasets

This part introduces the datasets used in this paper for the analysis of the
cross-lingual transfer of reasoning tasks. For datasets available only in
English or part of the tested languages, we translate them into other
languages with Google Translate.

####  2.1.1 Adapted StrategyQA dataset

The StrategyQA dataset is a multi-domain reasoning dataset including common
sense, arithmetic, symbolic, and logical reasoning, where model is given at
least two pieces of evidence from Wikipedia, and required to give "yes" or
"no" answers to the questions. An example can be found in Table 3 in Appendix
B.

Since Wikipedia is a commonly used source in LLM pretraining, the model can
either collect the required information from evidence in the context, or
retrieve it from its internal knowledge strorage. Thus, by masking out some of
the evidence, we can control the need of knowledge retrieval in the reasoning
process, which allows us to analyze the impact of knowledge retrieval on the
cross-lingual transfer of reasoning skills. Namely, we design two kinds of
scenarios in the experiments:

  * •

No Fact (NF): The model is given only the questions and must retrieve the
necessary knowledge (evidence).

  * •

With Fact (WF): The model is provided with the questions and some of the
evidence. To control the degree of knowledge retrieval needed, we further
devide the WF-1, WF-2 and WF-all settings, where one piece, two pieces, and
all pieces of evidence is provided for each question.

####  2.1.2 Knowledge-free reasoning dataset

Although the need for knowledge retrieval can be controlled with the adapted
StrategyQA dataset, the questions themselves may still contain implicit
requirement for knowledge retrieval. To rule out such effect and focus on the
knowledge-free reasoning process, we construct a data that minimizes the need
for knowledge retrieval. Inspired by Wei et al. (2022)’s taxonomy of reasoning
tasks, we developed the Knowledge-Free Reasoning Dataset (KFRD), which
consists of three fundamental reasoning tasks: arithmetic reasoning, symbolic
reasoning, and logic reasoning in the 4-choice format. The dataset is
constructed by filling GPT-4 Achiam et al. (2023) generated entities or simple
English words into a unified template with one input part, one transformation
rule, and one options part. Then, using the translated templates, entities and
words, the dataset is expanded to all the tested languages. The template and
examples can be found in Figure 11 and Table 1 in Appendix A.

##### Arithmetic Reasoning:

transforming one or two input numbers through mathematical operations into one
or two output numbers. The mathematical operations include: basic arithmetic
operations (addition, subtraction, multiplication, division), equality,
geometric progression, arithmetic progression, and sorting.

##### Symbolic Reasoning:

transforming 3-5 words from the respective language input through symbolic
operations to obtain the output. The symbolic operations include: repetition,
addition, deletion, reordering, and their combinations, with up to three
operations applied in combination.

##### Logical Reasoning:

selecting propositions derived from eight given premises using provided
logical rules. The logical rules include: Implication Elimination, Conjunction
Introduction, Conjunction Elimination, Disjunction Introduction, Disjunction
Elimination, and Proof by Contradiction.

####  2.1.3 Knowledge-related reasoning dataset

To further compare the cross-lingual transfer of knowledge retrieval and
knowledge-free reasoning, three knowledge-related reasoning datasets are used
in the interpretability analysis. The three datasets are:

##### MKQA:

The MKQA dataset Longpre et al. (2021) is an open-domain question-answering
evaluation set designed to provide a challenging benchmark for multilingual
question-answering. The dataset consists of 10,000 question-answer pairs and
covers 26 different languages.

##### BoolQ:

BoolQ (Clark et al., 2019) is a question answering dataset for yes/no
questions, comprising 15,942 examples. The questions originate from queries to
the Google search engine with Wikipedia pages as answers.

##### AmbigQA:

AmbigQA (Min et al., 2020) is an open-domain dataset for question answering
that includes 14,042 questions. It addresses the issue of ambiguous answers by
rephrasing the related questions.

Some examples of these datasets are shown in Table 3 in Appendix B.

###  2.2 Evaluation metric

In order to measure the model’s cross-lingual transferability while mitigating
the difficulty difference in of the datasets, we calculate the Cross-lingual
Transfer Ratio (XLTR) following the prior work Gao et al. (2024). The formula
is as follows:

$\text{XLTR}(s,t)=(\frac{|C_{s}\cap C_{t}|}{|C_{s}|}-A_{r})/(1-A_{r})$

where $s$ and $t$ denote the source and target languages in the transfer.
$C_{x}$ represents the set of correct answers in language $x$, and $A_{r}$ is
the accuracy of random choices for the given task. If after training on a task
in one specific language (namely, English), the model shows an XLTR score
close to 100% in other untrained languages, we say it achieves Fully Cross-
Lingual Transfer (FCLT) in this task.

###  2.3 Interpretability measurements

We employ two metrics to evaluate the cross-lingual computational consistency
of the LLM during knowledge-related and knowledge-free reasoning processes.

####  2.3.1 Cosine similarity of hidden states (CS)

We measure the cosine similarity of the hidden representations across multiple
languages during the reasoning process of a same question, in order to observe
how the semantic space in the tested languages approximate each other. The
similarity is calculated by:

$\text{CS}(x)=\frac{\sum_{n=1}^{N}\sum_{\begin{subarray}{c}a,b\in\mathcal{L},a\neq
b\end{subarray}}\frac{\mathbf{h}_{n}^{a}(x)\cdot\mathbf{h}_{n}^{b}(x)}{\left|\mathbf{h}_{n}^{a}(x)\right|\cdot\left|\mathbf{h}_{n}^{b}(x)\right|}}{|\mathcal{L}|(|\mathcal{L}|-1)N}$

where $x$ is a certain question sample, $N$ is the total number of model
layers, $\mathcal{L}$ denotes the set of all tested languages, and
$\mathbf{h}_{n}^{a}(x)$ is the output hidden states of the $n$-th layer for
sample $x$ in language $a$. After that, the cosine similarity of all tested
samples are averaged to report the final score.

####  2.3.2 Neuron Activation Overlap (NAO)

The Neuron Activation Overlap refers to the extent of overlap between
activated neurons across multiple languages for the same corpus. The
calculation steps are as follow:

  1. 1.

Input a question in multiple languages into the model and extract the
activation values of the last token in the question in each language.

  2. 2.

Identify activated neurons. A neuron is considered activated if the absolute
value of its activation exceeds a certain threshold.

  3. 3.

Calculate the overlap of activated neurons in all the tested languages.

For a question sample $x$, the formula is as follows:

$\text{NAO}(x)=\frac{\left|\mathcal{L}\right|\cdot\left|\bigcap_{l\in\mathcal{L}}S^{l}(x)\right|}{\sum_{l\in\mathcal{L}}\left|S^{l}(x)\right|}$

where $\mathcal{L}$ is set of languages, and $S^{l}(x)$ is the set of
activated neurons on sample $x$ in language $l$. After that, the NAO of all
tested samples are averaged to report the final score.

##  3 Experiment Settings

###  3.1 Language and model choice

##### Language choice

To capture a broad spectrum of linguistic diversity, we selected ten languages
based on their extensive use and representation of diverse linguistic
families, following Gao et al. (2024). The languages selected are English
(en), German (de), French (fr), Italian (it), Russian (ru), Polish (pl),
Arabic (ar), Hebrew (he), Chinese (zh), and Japanese (ja). Further details are
provided in Appendix C.

##### Model choice

We selected several LLMs including LLaMA-2-7B-Chat (Touvron et al., 2023),
BLOOMZ-MT-7B (Muennighoff et al., 2023), Mistral-7B-Instruct-v0.1 (Jiang et
al., 2023), and Qwen-1.5-7B-Chat (Bai et al., 2023) to conduct transferability
experiments on existing and synthetic datasets.

For experiments focusing on the impact of the training language (Section
4.3.1) and interpretability (Section 4.4), we used LLaMA-2-7B-Chat as a
representative model for analysis.

For the experiments evaluating impact of the target language proficiency in
specific directions (Section 4.3.2), we selected derived models of LLaMA-2-7B
and Mistral-7B to conduct experiments on Arabic and Hebrew respectively, both
of which are low-resource languages.

###  3.2 Fine-tuning settings

We perform LoRA fine-tuning (Hu et al., 2021) on all model blocks in all
experiments due to the limited computational resources. More details about
fine-tuning can be found in Appendix E.

###  3.3 Decoding settings

In all experiments, we perform constrained decoding to prevent the model from
generating tokens other than the desired choices (Yes/No for StrategyQA,
A/B/C/D for KFRD).

In the interpretability analysis, we use the last token of the question to
collect the hidden states and neural activation values, because the last input
token is used to predict the next token, it gradually incorporates the primary
information of the entire sentence, reflecting the overall thought process for
the entire problem (Meng et al., 2022; Stolfo et al., 2023; Wu et al., 2024).
By focusing on the model’s computational pathway for reasoning rather than
calculating the similarity between multilingual sentences, we can better
understand how the model processes complex queries. Calculating with an output
token, on the other hand, would make it difficult to interpret the reasoning
process. Additionally, token counts differ across languages, complicating
direct comparisons. Therefore, using the last input token helps in
standardizing the analysis across different languages.

To ensure consistency in the final token across different datasets, we added a
language-specific ’?’ to certain datasets. For details on these modifications
and metrics, please refer to Appendix D.

##  4 Results

Figure 1: XLTR of different models on StrategyQA. Solid lines: WF results;
Dashed lines: NF results. The label of training language (en) is capitalized.
Figure 2: XLTR of LLaMA-2-7B-Chat on StrategyQA under different settings. NF
stands for the No Facts setting, while WF-1 and WF-2 indicate that one or two
facts are given for each question, respectively. WF-all signifies that all
facts are provided. Figure 3: XLTR on the different parts of KFRD

Figure 4: Accuracy of LLaMA-2-7B-Chat from different languages on three parts
of KFRD

Figure 5: XLTR of LLaMA-2-7B-Chat from different languages on three parts of
KFRD

###  4.1 Impact of knowledge retrieval on cross-lingual transfer

We analyze the impact of the amount of retrieved knowledge on cross-lingual
transfer in different settings of the StrategyQA dataset. The results for
cross-lingual transfer ratio are presented in Figure 1, while the accuracy
results are detailed in Figure 12 in Appendix G.

##### Knowledge retrieval requirement harms cross-lingual transfer

The experimental results indicate that, for all languages, the cross-lingual
transfer ratio of all models are significantly higher when the necessary
knowledge for reasoning is provided compared to when it is not. This suggests
that the requirement for knowledge retrieval significantly hinders the model’s
cross-lingual transferability when solving reasoning problems.

##### Less knowledge retrieval makes better cross-lingual transfer

Additionally, we conduct a more detailed evaluation using the LLaMA-2-7B-Chat
model to observe the changes in cross-lingual transfer ratio with varying
amounts of knowledge entries. As shown in Figure 2, the experimental results
demonstrate that the transfer rate increases as the number of provided
knowledge entries increases from none to all required knowledge. This further
validates the conclusion that the retrieval of more knowledge significantly
impacts cross-lingual transferability.

###  4.2 The cross-lingual transfer of knowledge-free reasoning

We assess the cross-lingual transferability of the model’s knowledge-free
reasoning capabilities. We evaluate the performance of the KFRD by training in
English and transferring to the other 9 languages. The resulting cross-lingual
transfer ratio are shown in Figure 3, and accuracy results are shown in Figure
13 in Appendix G.

The results demonstrate that the KFRD exhibits extremely high cross-lingual
transfer performance for most language pairs. For 7 out of the 9 languages, it
can be observed that the cross-lingual transfer performance in knowledge-free
reasoning tasks often exceeds 90%, with some instances approaching 100%,
thereby achieving near FCLT.

However, for the two lowest-resource languages Arabic and Hebrew, the cross-
lingual transferability is significantly poorer. We hypothesize that this may
be due to the model’s weaker language proficiency in these two languages,
which adversely affects its transferability. Further analysis on this issue is
provided in the next section.

###  4.3 Impact of language proficiency on cross-lingual transfer

In this section, we explore the impact of language proficiency of the training
and target language with additional experiments.

####  4.3.1 Training language proficiency

To test the impact of training language proficiency, we select German and
Chinese as representatives of high-resource languages, and Arabic and Hebrew
as representatives of low-resource languages as training languages. Then, we
train models on the KFRD in these languages and evaluated their performance
across the 10 languages. The accuracy results are shown in Figure 5, and the
transfer ratio is presented in Figure 5.

The results show that, though training with low-resource languages resulted in
slightly lower accuracy (which is particularly noticeable in the arithmetic
dataset), the models show no significant differences in transfer ratio when
trained with high-resource or low-resource languages, indicating that the
proficiency and resource of the training language has no significant effect on
the cross-lingual transfer of knowledge-free reasoning.

####  4.3.2 Target language proficiency in specific directions

In previous experiments, we observe strong cross-linguistic transfer
performance between most languages. However, the transferability from English
to Arabic and Hebrew was significantly weaker. We hypothesize that this is
related to the model’s weaker language proficiency in these two target
languages.

In this section, we select models from Hugging Face that have undergone
Continuous Pre-Training (CPT), Supervised Fine-Tuning (SFT), and a combination
of both (CPT + SFT) on the LLaMA-2 or Mistral platforms. These adapted models
have better proficiency in the respective languages. The selected models are
listed in Table 2 in Appendix F.

We train a Vanilla model and the above fine-tuned models on the KFRD in
English and test their transferability from English to the respective target
languages. The transfer rate results are shown in Figure 6, and the accuracy
results are provided in Figure 14 in Appendix G.

The experimental results indicate that the Vanilla model exhibits very low
transfer rates for low-resource languages. However, after applying CPT, SFT,
or CPT+SFT, the transfer rates increase significantly. Notably, for Hebrew,
the transfer rate reached over 95%, approaching FCLT. This suggests that the
proficiency in Arabic and Hebrew limits the cross-linguistic transfer of
knowledge-free reasoning abilities from English to these languages. Improving
proficiency in the target language can alleviate this limitation. This
demonstrates that the cross-linguistic transfer of knowledge-free reasoning in
specific directions is influenced by target language proficiency.

###  4.4 Analyzing differences in transfer characteristics

####  4.4.1 Overall computational similarity

In this section, we assess the LLaMA-2-7B-Chat model’s hidden states cosine
similarity and Feed-Forward Network neuron activation overlap on knowledge-
related tasks and knowledge-free reasoning tasks. We also evaluate changes in
these metrics after fine-tuning with LoRA to understand the mechanisms
supporting cross-lingual transfer of knowledge-free reasoning abilities. The
experimental results are shown in Figures 7 and 8.

Figure 6: Averaged XLTR from English to Arabic/Hebrew on KFRD for models in
different stages trained in Arabic/Hebrew Figure 7: CS for different datasets
in the LLaMA-2-7B-Chat model. The black lines on each bar indicate the 99%
confidence intervals estimated with bootstrap sampling (Efron, 1992). Figure
8: NAO for different dataset in the LLaMA-2-7B-Chat at activation thresholds
ranging from 0.1 to 0.9. Shaded areas: 99% confidence intervals estimated with
bootstrap sampling; Solid lines: results of the original model; Dashed lines:
results of the LoRA tuned model. The meanings of the shaded areas and dashed
lines in Figures 9 and 10 are consistent with those described here.

##### Internal representation of knowledge-free reasoning task is better
aligned than retrieval

The results in Figure 7 indicate that the CS of the model on knowledge-free
reasoning tasks is significantly higher than that on knowledge-related
reasoning tasks both before and after training. Additionally, after training
on the knowledge-free reasoning dataset, the model’s CS shows a significant
increase. In contrast, after training on the knowledge-related reasoning
dataset, the increase in CS is not significant and may even decrease. This
suggests that adapting to knowledge-free reasoning tasks results in a more
aligned hidden space processing across different languages, whereas adapting
to knowledge-related reasoning tasks leads to relatively divergent hidden
space processing.

##### Neuron activation pattern of knowledge-free reasoning task is more
similar than retrieval

Neuron analysis further elucidates this phenomenon. The results in Figure 8
show that, across all threshold settings, NAO for knowledge-free reasoning
tasks is significantly higher than for knowledge-related reasoning tasks. This
indicates that the model tends to use similar neurons for processing
knowledge-free reasoning tasks across different languages, resulting in
similar neuron activation patterns and computational processes. Consistent
with the hidden states results, after training on the knowledge-free reasoning
dataset, the model’s NAO shows a significant increase, whereas there is no
significant improvement and even a decline after training on the knowledge-
related reasoning dataset. This suggests that training on knowledge-free
reasoning tasks makes neuron activation characteristics across different
languages more similar, whereas training on knowledge-related reasoning tasks
makes these characteristics relatively divergent.

These results provide a comprehensive analysis of the differing cross-lingual
transfer effectiveness between knowledge-free reasoning and knowledge-related
reasoning from a computational similarity perspective. We hypothesize that
this difference is because the model tends to store knowledge for different
languages in different neurons, while using similar neuron groups for
knowledge-free reasoning functions.

####  4.4.2 Layer-wise computational similarity

To analyze the relationship between hidden states and neuron activation
characteristics with cross-lingual transfer of reasoning abilities more
granularly, we performed a layer-wise analysis of the cross-lingual cosine
similarity of hidden states and cross-lingual neural activation overlap. The
experimental results are shown in Figures 9 and 10.

Figure 9: CS for different layers of the LLaMA-2-7B-Chat. Figure 10: NAO for
different layers of the LLaMA-2-7B-Chat at an activation threshold of 0.4.

It is observed that the phenomena identified in the previous section,
specifically the significantly higher CS and NAO for knowledge-free reasoning
tasks compared to knowledge tasks with more pronounced improvements after
fine-tuning, are most evident in the middle layers (layers 6-25) of the model
and less in the shallow layers (layers 1-6). Previous work (Zhao et al., 2024;
Wendler et al., 2024) suggested that the middle layers of LLMs are primarily
responsible for conceptual reasoning, which is cross-lingual. This hypothesis
aligns with our findings and further supports the view that reasoning
capabilities can transfer across languages.

Additionally, in the upper layers (layers 25-32) of the model, the CS and NAO
for knowledge-free reasoning tasks are similar to those for knowledge tasks
before training, but show significant improvement after training. This
phenomenon is accompanied by a consistent performance improvement across
different languages. The layered functionality theory of LLMs suggests that
the upper layers are mainly responsible for generating the tokens of the
target language responses. Therefore, we infer that task fine-tuning makes the
model’s behavior more consistent during the decoding phase, resulting in
better multilingual task performance.

##  5 Related Work

##### Multilingual reasoning evaluation

Laskar et al. (2023) performed evaluation for multilingual ability of ChatGPT.
Shi et al. (2022) found LLMs can perform reasoning in multiple languages using
CoT, even for those languages with very low resources.

##### Cross-lingual transfer

Gao et al. (2024) evaluated the cross-lingual transferability of models on
multiple reasoning datasets, finding significant variations in transfer
performance across different datasets. Additionally, Zhu et al. (2024)
discovered that training on translated questions can enhance the cross-lingual
transferability of reasoning tasks. Ye et al. (2023) assessed the imbalance of
knowledge across different languages in LLMs, observing weak cross-lingual
transferability of knowledge. Furthermore, Hu et al. (2024) found that
knowledge transferability remains weak across various settings.

##### Analysis of multilingual internal representation

Zhao et al. (2024) analyzed the way LLMs handle multilingualism and suggested
a three-phase working pattern, which includes understanding, task solving and
generation. Wendler et al. (2024) also arrived at a similar conclusion.

##  6 Conclusion

In this study, we analyze the reasons behind the differing cross-lingual
transfer abilities of LLMs on various reasoning tasks. We divide reasoning
tasks into two components: knowledge retrieval and knowledge-free reasoning.
We adapt several existing reasoning datasets and construct a knowledge-free
reasoning dataset for analysis.

Experiments on these datasets demonstrated that the requirement of knowledge
retrieval significantly lowers the cross-lingual transfer performance of
reasoning tasks, while the knowledge-free reasoning ability can nearly fully
transferred between languages.

In addition, we analyze the difference of computational processes between
knowledge-related and knowledge-free reasoning tasks to explain the gap in
cross-lingual transfer. The results show that the overall cross-lingual
computational similarity of knowledge-free reasoning is significantly higher
than that of knowledge-related reasoning, particularly in the middle-high
layers of the model. This suggests that knowledge-free reasoning benefits from
a shared neural mechanism across languages, whereas knowledge storage is more
language-specific.

## Limitations

One key limitation of this paper is the model selection and language coverage.
In our exploration of language proficiency and interpretability experiments,
we primarily rely on the LLaMA-2 model. Additionally, other parts of our
research utilize only a few models, which may oversimplify the descriptions of
model performance and behavior. In terms of language coverage, although we
included ten languages from different language families, this number is still
insufficient compared to the thousands of languages globally. This limitation
is partly due to our computational resource constraints. With adequate
resources, the proposed methods could be extended to other models and
languages to further validate our conclusions.

Another limitation of our study is the depth of the interpretability analysis.
We aim to investigate whether different knowledge-free reasoning tasks utilize
the same neurons and whether knowledge is stored in different neurons for
different languages. However, our support for this hypothesis is primarily
based on macro-level numerical analyses, without precisely identifying
specific reasoning neurons and knowledge neurons. This limitation restricts
our fine-grained understanding of the model’s internal mechanisms. Future
research should conduct more detailed neuron-level analyses to verify these
hypotheses.

## Ethics Statement

The authors declare no competing interests. All datasets used in this study
are sourced from publicly available repositories and do not contain sensitive
information, such as personal data. The data generated by GPT-4 have been
verified to be non-toxic and are used exclusively for research purposes. The
use of LLaMA-2 models, as well as several other large language models,
complies with their respective licenses.

## References

  * Achiam et al. (2023) Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023.  Gpt-4 technical report.  _arXiv preprint arXiv:2303.08774_.
  * Ansel et al. (2024) Jason Ansel, Edward Yang, Horace He, Natalia Gimelshein, Animesh Jain, Michael Voznesensky, Bin Bao, Peter Bell, David Berard, Evgeni Burovski, Geeta Chauhan, Anjali Chourdia, Will Constable, Alban Desmaison, Zachary DeVito, Elias Ellison, Will Feng, Jiong Gong, Michael Gschwind, Brian Hirsh, Sherlock Huang, Kshiteej Kalambarkar, Laurent Kirsch, Michael Lazos, Mario Lezcano, Yanbo Liang, Jason Liang, Yinghai Lu, CK Luk, Bert Maher, Yunjie Pan, Christian Puhrsch, Matthias Reso, Mark Saroufim, Marcos Yukio Siraichi, Helen Suk, Michael Suo, Phil Tillet, Eikan Wang, Xiaodong Wang, William Wen, Shunting Zhang, Xu Zhao, Keren Zhou, Richard Zou, Ajit Mathews, Gregory Chanan, Peng Wu, and Soumith Chintala. 2024.  PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation.  In _29th ACM International Conference on Architectural Support for Programming Languages and Operating Systems, Volume 2 (ASPLOS ’24)_. ACM.
  * Bai et al. (2023) Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, Binyuan Hui, Luo Ji, Mei Li, Junyang Lin, Runji Lin, Dayiheng Liu, Gao Liu, Chengqiang Lu, Keming Lu, Jianxin Ma, Rui Men, Xingzhang Ren, Xuancheng Ren, Chuanqi Tan, Sinan Tan, Jianhong Tu, Peng Wang, Shijie Wang, Wei Wang, Shengguang Wu, Benfeng Xu, Jin Xu, An Yang, Hao Yang, Jian Yang, Shusheng Yang, Yang Yao, Bowen Yu, Hongyi Yuan, Zheng Yuan, Jianwei Zhang, Xingxuan Zhang, Yichang Zhang, Zhenru Zhang, Chang Zhou, Jingren Zhou, Xiaohuan Zhou, and Tianhang Zhu. 2023.  Qwen technical report.  _arXiv preprint arXiv:2309.16609_.
  * Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019.  Boolq: Exploring the surprising difficulty of natural yes/no questions.  _arXiv preprint arXiv:1905.10044_.
  * Csaki et al. (2024) Zoltan Csaki, Bo Li, Jonathan Li, Qiantong Xu, Pian Pawakapan, Leon Zhang, Yun Du, Hengyu Zhao, Changran Hu, and Urmish Thakker. 2024.  Sambalingo: Teaching large language models new languages.  _Preprint_ , arXiv:2404.05829.
  * DICTA (2024) DICTA. 2024.  Dictalm-2.0.  https://huggingface.co/dicta-il/dictalm2.0.  Accessed: 2024-06-15.
  * Efron (1992) Bradley Efron. 1992.  Bootstrap methods: another look at the jackknife.  In _Breakthroughs in statistics: Methodology and distribution_ , pages 569–593. Springer.
  * Gao et al. (2024) Changjiang Gao, Hongda Hu, Peng Hu, Jiajun Chen, Jixing Li, and Shujian Huang. 2024.  Multilingual pretraining and instruction tuning improve cross-lingual knowledge alignment, but only shallowly.  _arXiv preprint arXiv:2404.04659_.
  * Geva et al. (2021) Mor Geva, Yoav Goldberg, and Jonathan Berant. 2021.  Strategyqa: A question answering benchmark for reasoning about strategies.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_.
  * Hu et al. (2021) Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021.  Lora: Low-rank adaptation of large language models.  _Preprint_ , arXiv:2106.09685.
  * Hu et al. (2024) Peng Hu, Changjiang Gao, Ruiqi Gao, Jiajun Chen, and Shujian Huang. 2024.  Limited out-of-context knowledge reasoning in large language models.  _Preprint_ , arXiv:2406.07393.
  * Huang and Chang (2022) Jie Huang and Kevin Chen-Chuan Chang. 2022.  Towards reasoning in large language models: A survey.  _arXiv preprint arXiv:2212.10403_.
  * Icebear-AI (2024) Icebear-AI. 2024.  Llama-2-7b-chat-arabic-lora.  https://huggingface.co/Icebear-AI/Llama-2-7b-chat-arabic-lora.  Accessed: 2024-06-15.
  * Jiang et al. (2023) Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. 2023.  Mistral 7b.  _Preprint_ , arXiv:2310.06825.
  * Laskar et al. (2023) Md Tahmid Rahman Laskar, M Saiful Bari, Mizanur Rahman, Md Amran Hossen Bhuiyan, Shafiq Joty, and Jimmy Xiangji Huang. 2023.  A systematic study and comprehensive evaluation of chatgpt on benchmark datasets.  _Preprint_ , arXiv:2305.18486.
  * Longpre et al. (2021) Shayne Longpre, Yi Lu, and Joachim Daiber. 2021.  Mkqa: A linguistically diverse benchmark for multilingual open domain question answering.  _Transactions of the Association for Computational Linguistics_ , 9:1389–1406.
  * Meng et al. (2022) Kevin Meng, David Bau, Alex Andonian, and Yonatan Belinkov. 2022.  Locating and editing factual associations in gpt.  _Advances in Neural Information Processing Systems_ , 35:17359–17372.
  * Min et al. (2020) Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020.  AmbigQA: Answering ambiguous open-domain questions.  In _EMNLP_.
  * Muennighoff et al. (2023) Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M Saiful Bari, Sheng Shen, Zheng-Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward Raff, and Colin Raffel. 2023.  Crosslingual generalization through multitask finetuning.  _Preprint_ , arXiv:2211.01786.
  * Qi et al. (2023) Jirui Qi, Raquel Fernández, and Arianna Bisazza. 2023.  Cross-lingual consistency of factual knowledge in multilingual language models.  _Preprint_ , arXiv:2310.10378.
  * Ranaldi and Zanzotto (2023) Leonardo Ranaldi and Fabio Massimo Zanzotto. 2023.  Empowering multi-step reasoning across languages via tree-of-thoughts.  _arXiv preprint arXiv:2311.08097_.
  * Rasley et al. (2020) Jeff Rasley, Samyam Rajbhandari, Olatunji Ruwase, and Yuxiong He. 2020.  Deepspeed: System optimizations enable training deep learning models with over 100 billion parameters.  In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_, pages 3505–3506.
  * Saparov et al. (2024) Abulhair Saparov, Richard Yuanzhe Pang, Vishakh Padmakumar, Nitish Joshi, Mehran Kazemi, Najoung Kim, and He He. 2024.  Testing the general deductive reasoning capacity of large language models using ood examples.  _Advances in Neural Information Processing Systems_ , 36.
  * Shi et al. (2022) Freda Shi, Mirac Suzgun, Markus Freitag, Xuezhi Wang, Suraj Srivats, Soroush Vosoughi, Hyung Won Chung, Yi Tay, Sebastian Ruder, Denny Zhou, Dipanjan Das, and Jason Wei. 2022.  Language models are multilingual chain-of-thought reasoners.  _Preprint_ , arXiv:2210.03057.
  * Stolfo et al. (2023) Alessandro Stolfo, Yonatan Belinkov, and Mrinmaya Sachan. 2023.  A mechanistic interpretation of arithmetic reasoning in language models using causal mediation analysis.  In _Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing_ , pages 7035–7052.
  * Touvron et al. (2023) Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Hartshorn, Saghar Hosseini, Rui Hou, Hakan Inan, Marcin Kardas, Viktor Kerkez, Madian Khabsa, Isabel Kloumann, Artem Korenev, Punit Singh Koura, Marie-Anne Lachaux, Thibaut Lavril, Jenya Lee, Diana Liskovich, Yinghai Lu, Yuning Mao, Xavier Martinet, Todor Mihaylov, Pushkar Mishra, Igor Molybog, Yixin Nie, Andrew Poulton, Jeremy Reizenstein, Rashi Rungta, Kalyan Saladi, Alan Schelten, Ruan Silva, Eric Michael Smith, Ranjan Subramanian, Xiaoqing Ellen Tan, Binh Tang, Ross Taylor, Adina Williams, Jian Xiang Kuan, Puxin Xu, Zheng Yan, Iliyan Zarov, Yuchen Zhang, Angela Fan, Melanie Kambadur, Sharan Narang, Aurelien Rodriguez, Robert Stojnic, Sergey Edunov, and Thomas Scialom. 2023.  Llama 2: Open foundation and fine-tuned chat models.  _Preprint_ , arXiv:2307.09288.
  * Wei et al. (2022) Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022.  Chain-of-thought prompting elicits reasoning in large language models.  _Advances in neural information processing systems_ , 35:24824–24837.
  * Wendler et al. (2024) Chris Wendler, Veniamin Veselovsky, Giovanni Monea, and Robert West. 2024.  Do llamas work in english? on the latent language of multilingual transformers.  _arXiv preprint arXiv:2402.10588_.
  * Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. 2020.  Transformers: State-of-the-art natural language processing.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_ , pages 38–45, Online. Association for Computational Linguistics.
  * Wu et al. (2024) Wilson Wu, John X Morris, and Lionel Levine. 2024.  Do language models plan ahead for future tokens?  _arXiv preprint arXiv:2404.00859_.
  * Ye et al. (2023) Jiacheng Ye, Xijia Tao, and Lingpeng Kong. 2023.  Language versatilists vs. specialists: An empirical revisiting on multilingual transfer ability.  _arXiv preprint arXiv:2306.06688_.
  * Zhao et al. (2024) Yiran Zhao, Wenxuan Zhang, Guizhen Chen, Kenji Kawaguchi, and Lidong Bing. 2024.  How do large language models handle multilingualism?  _arXiv preprint arXiv:2402.18815_.
  * Zheng et al. (2024) Yaowei Zheng, Richong Zhang, Junhao Zhang, Yanhan Ye, Zheyan Luo, and Yongqiang Ma. 2024.  Llamafactory: Unified efficient fine-tuning of 100+ language models.  _arXiv preprint arXiv:2403.13372_.
  * Zhu et al. (2024) Wenhao Zhu, Shujian Huang, Fei Yuan, Shuaijie She, Jiajun Chen, and Alexandra Birch. 2024.  Question translation training for better multilingual reasoning.  _arXiv preprint arXiv:2401.07817_.

##  Appendix A Detailed Description of Knowledge-Free Reasoning

The KFRD is generated using a unified template, consisting entirely of
multiple-choice questions with four options. Each question is structured into
three parts: input, output, and transformation rules. Specific examples can be
seen in Table 1, and the templates used for these examples are shown in Figure
11.

Arithmetic Reasoning
---
Input |  11,645 (one or two numbers)
Transformation Rule |  Addition (a mathematical operation)
Output Options |  A) 595 B) 536 C) 771 D) 656 (one number, sometimes the result is two numbers)
Symbolic Reasoning
Input |  education, game, president, night, man (3-5 words in the corresponding language)
Transformation Rule |  Swap the positions of the 5th and 2nd words; Delete the 2nd word (symbolic operations within three layers)
Output Options |  A) education, president, night, game B) education, problem, night, game C) hand, president, night, game D) education, house, night, game
Logical Reasoning
Input |  Alex is Aurora Vale. Everything that is Aurora Vale is Omicron Delta. Stella is not Chronos Wasteland. Max is not Dreamweaver’s Haven. Suppose Sally is Whispering Meadows, then Sally is Chimerical Citadel. Everything that is Ebonwyrm Abyss is Phoenixfire Ridge. (8 propositions)
Transformation Rule |  Implication Elimination (a logical rule)
Output Options |  A) Alex is Seraphim Heights. B) Alex is Tempestwilds. C) Alex is Omicron Delta. D) Polly is Arcadia Reach.
Table 1: Examples of different tasks in the KFRD dataset

###  A.1 Arithmetic Reasoning

This dataset transforms two input numbers through mathematical operations into
one or two output numbers. The mathematical operations include basic
arithmetic operations (addition, subtraction, multiplication, division),
equality, geometric progression, arithmetic progression, and sorting.

  * •

Input: Numbers are randomly generated within the range of 0-999.

  * •

Transformation rules: Each rule generates an equal number of samples.

  * •

Output: Generated through transformation rules, constrained within the range
of 0-999. Other options are randomly generated, ensuring a single correct
answer.

###  A.2 Symbolic Reasoning

This dataset transforms 3-5 input words from the corresponding language
through symbolic operations to generate the output. Symbolic operations
include repetition, addition, deletion, reordering, and their combinations.
Considering that single-step symbolic operations are too simple, we chose up
to three-step symbolic operations.

  * •

Input: Randomly select 3-5 words from a specific language. We chose 100 simple
English words and translated them into other languages using Google Translate.

  * •

Transformation rules: The dataset includes equal amounts of single-step, two-
step, and three-step symbolic operations. For single-step operations, each
rule generates an equal number of samples. For two-step and three-step
operations, rule combinations are randomly selected.

  * •

Output: Generated through transformation rules. Other options are partially
randomly generated and partially based on random replacements from the
original input, ensuring consistent length and a unique correct answer.

###  A.3 Logical Reasoning

This dataset generates output from a subset of eight input propositions using
logical rules. Logical rules include Implication Elimination, Conjunction
Introduction, Conjunction Elimination, Disjunction Introduction, Disjunction
Elimination, and Proof by Contradiction. The Logical rules are referenced from
Saparov et al. (2024).

  * •

Input: Eight propositions are generated using proposition templates and
randomly selected entities, proposition templates referenced from Saparov et
al. (2024) and entities from Saparov et al. (2024) and Gao et al. (2024).
Missing languages were supplemented using Google Translate.

  * •

Transformation rules: Each logical rule generates an equal number of samples.

  * •

Output: Generated through logical rules. Other options are partially based on
entities appearing in the propositions and partially randomly generated,
ensuring a unique correct answer.

Instruction: The output is the result of applying a specific transformation rule to the input. In this question, you will be given an input value and its corresponding transformation rule. Based on this information, determine the correct output from the options provided: A, B, C, or D. Please give the corresponding answer option directly. Transformation Rule: {Transformation Rule} Input: {Input} Based on the above rule and input, choose the correct output from the following options: A. Output: {Output1} B. Output: {Output2} C. Output: {Output3} D. Output: {Output4} Answer: Figure 11: Example prompt template for our KFRD dataset Training | Arabic | Hebrew
---|---|---
Vanilla | LLaMA-2-7B-Chat | Mistral-7B-Instruct-v0.1
SFT | Llama-2-7b-chat-arabic-lora (Icebear-AI, 2024) | -
CPT | SambaLingo-Arabic-Base (Csaki et al., 2024) | DictaLM-2.0 (DICTA, 2024)
CPT+SFT | SambaLingo-Arabic-Chat | DictaLM-2.0-Instruct
Table 2: Training models for Arabic and Hebrew

##  Appendix B Dataset Examples

We provide some examples of adapted datasets used in this paper in Table 3.

StrategyQA
---
Question |  Are more people today related to Genghis Khan than Julius Caesar?
Facts |  1\. Julius Caesar had three children. 2\. Genghis Khan had sixteen children. 3\. Modern geneticists have determined that out of every 200 men today has DNA that can be traced to Genghis Khan.
Answer |  True
MKQA
Query |  Who sings "I Hear You Knocking But You Can’t Come In"?
Answers |  Dave Edmunds
BoolQ
Question |  Do Iran and Afghanistan speak the same language?
Answer |  True
AmbigQA
Question |  How often does spermatogenesis—the production of sperm—occur?
Answer |  74 days
Table 3: Examples of adapted datasets used in this paper

##  Appendix C Language Choice

This section provides an overview of the languages utilized in our research,
highlighting the primary countries where they are spoken and their respective
language families. Refer to Table 4 for detailed information.

ISO | Countries | Language Family
---|---|---
en | US, UK | Germanic
de | Germany, Austria | Germanic
fr | France, Canada | Romance
it | Italy | Romance
pl | Poland | Slavic
ru | Russia, Belarus | Slavic
ar | Egypt, Algeria | Afro-Asiatic
he | Israel | Afro-Asiatic
ja | Japan | Japonic
zh | China (Mainland) | Sino-Tibetan
Table 4: Correspondence between Languages, Countries, and Language Families

##  Appendix D Implementation Details for Interpretability

###  D.1 Dataset adjustments

We made slight modifications of the datasets to ensure that the last character
of different datasets is uniformly a "?" token. Since we are analyzing the
internal representation of the last token, in this way, we can eliminate
interference caused by the inconsistent input token, which may make the
representation unreliable, especially in the bottom layers. Another reason why
we append the token "?" is that it can act as a trigger to let the model start
the process of preparing to answer the question, which is what we are
analyzing.

For knowledge-free reasoning dataset, we added the phrase "Which option should
I choose?" in different languages. For the MKQA and BoolQ datasets, where some
questions did not end with a "?", we added a "?". All other datasets already
ended with a "?".

###  D.2 Calculation method for activation values

We use the output of the gate linear layer in the SwiGLU module of the LLaMA
model, processed through the SiLU function, as the activation values.

Dataset | Samples | Epoch
---|---|---
StrategyQA | 2061 | 4
KFRD Arithmetic | 8000 | 4
KFRD Symbolic | 2000 | 1
KFRD Logical | 4000 | 1
Table 5: Training epoch and number of samples of fine-tuned datasets in the transferability experiments Dataset | Samples
---|---
StrategyQA | 228
KFRD Arithmetic | 800
KFRD Symbolic | 500
KFRD Logical | 500
Table 6: The size of testset used in the transferability experiments

##  Appendix E Experiments Details

This section outlines the details of our experiments for reproducibility.

###  E.1 Infrastructure

We used the following scientific artifacts in our research:

  * •

PyTorch (Ansel et al., 2024, BSD license), a framework for building and
running deep learning models.

  * •

Transformers (Wolf et al., 2020, Apache-2.0 license), a library providing a
user friendly interface for running and fine-tuning pre-trained models.

  * •

DeepSpeed (Rasley et al., 2020, Apache-2.0 license), a library optimizing the
parallel training of the deep learning models.

  * •

LLaMA-Factory (Zheng et al., 2024, Apache-2.0 license), a library that
provides a unifying way to easily fine-tune large language models with
parameter efficient fine-tuning technique like LoRA.

###  E.2 Hyperparameters

In the fine-tuning of all models, we use a learning rate of 2e-4 with a cosine
learning rate scheduler. We clip the gradient norm to 1.0, use a total batch
size of 64, set the rank of LoRA to 128, and alpha to 16. The LoRA adapters
are applied to all the linear layers within Transformer blocks.

The numbers of training epoch and samples used in the transferability
experiments are listed in Table 5. These numbers are tuned to enable
LLaMA-2-7B-Chat to achieve 85% + accuracy on the corresponding tasks. The size
of testsets used in the transferability experiments are shown in Table 6.

In the interpretability experiments, we adjust the number of training epochs
or the size of the syntactic datasets to keep the number of total update steps
on all datasets around 150, which avoids interference of different update
steps on experimental results. We report the average cosine similarity and
neuron activation overlap of 100 samples from each dataset.

###  E.3 Computation resources

All the fine-tuning experiments can be done on 4 NVIDIA Tesla V100 32GB GPUs.
Each fine-tuning can be done in no more than 2 hours.

##  Appendix F Models Used in the Target Language Proficiency Experiment

The continue pre-training or fine-tuning models of LLaMA-2-7B and Mistral-7B
used in the target language proficiency experiment in 4.3.2 are listed in
Table 2.

##  Appendix G Additional Results of Experiment

Here we provide the accuracy of the above experiments in Figure 12, 13 and 14.

Figure 12: Left: Accuracy of different models on StrategyQA. Solid and dashed
line represent the result of With Facts and No Facts setting, respectively.
Middle: Accuracy of different models on StrategyQA before fine-tuning. Right:
Accuracy of LLaMA-2-7B-Chat on StrategyQA under various settings. The
translucent line represents the accuracy before finetuning on the specific
tasks (which are all around 50%). Figure 13: Accuracy of various models on
different parts of KFRD. The translucent line represents the accuracy before
finetuning on the specific tasks. Figure 14: Averaged Accuracy on English and
Arabic/Hebrew KFRD for models in different stages trained in Arabic/Hebrew