cosets the images of $F_{A}^{*}$ can be added to the domain we integrate over
because the kernel function vanishes on those sets. If
$\sigma\in\sigma_{\kappa}^{-1}\Gamma\sigma_{\kappa}$ leaves $\infty$ fixed,
then so does every element in its coset, and the part
$\sigma(\sigma_{\kappa}^{-1}F^{*}_{A})$ is the same as

$\\{z\in
F_{C({\gamma_{\alpha}^{\kappa}})}:\,\sigma^{-1}z\in\sigma_{\kappa}^{-1}\sigma_{\kappa^{\prime}}U_{A},\,\textrm{
for some cusp }\kappa^{\prime}\in\mathcal{S}\\},$

at least if $A$ is big enough. If $\kappa\neq\kappa^{\prime}$, then
$\sigma_{\kappa}^{-1}\sigma_{\kappa^{\prime}}=\left[\begin{array}[]{cc}a&b\\\
c&d\end{array}\right]$ does not fix the point $\infty$ and hence $c\neq 0$.
Since $\sigma^{-1}\infty=\infty$, the values $Y_{0}(\sigma^{-1}z)$ and
$Y_{0}(z)$ are the same. Therefore, if
$\sigma^{-1}z\in\sigma_{\kappa}^{-1}\sigma_{\kappa^{\prime}}U_{A}$, then there
is a $w\in U_{A}$ such that
$\sigma^{-1}z=\sigma_{\kappa}^{-1}\sigma_{\kappa^{\prime}}w$, and hence

$Y_{0}(z)=Y_{0}(\sigma^{-1}z)=Y_{0}(\sigma_{\kappa}^{-1}\sigma_{\kappa^{\prime}}w)\leq\frac{1}{(Nc)^{2}A}.$

The inequality above follows easily from the identity
$\left|c_{k}z+d_{k}\right|^{2}\cdot\textrm{Im}\,(\sigma_{\kappa}^{-1}\sigma_{\kappa^{\prime}}z)_{k}=\textrm{Im}\,z_{k}$
that holds for every $k=1,\dots,n$. The function $\psi$ is compactly
supported, hence for a large enough $A$ the kernel

$k(z,\gamma_{\alpha}^{\kappa}z)=\psi\left(\frac{\left|z_{1}-(z_{1}-\alpha_{1})\right|^{2}}{y_{1}^{2}},\dots,\frac{\left|z_{n}-(z_{n}-\alpha_{n})\right|^{2}}{y_{n}^{2}}\right)=\psi\left(\frac{\alpha_{1}^{2}}{y_{1}^{2}},\dots,\frac{\alpha_{n}^{2}}{y_{n}^{2}}\right)$

vanishes for every $z\in F_{C(\gamma_{\alpha})}$ for which
$\sigma^{-1}z\in\sigma_{\kappa}^{-1}\sigma_{\kappa^{\prime}}U_{A}$ holds for
some $\kappa\neq\kappa^{\prime}$, hence these parts can be simply added to the
domain we integrate over.

Now assume that $\sigma$ does not fix the cusp $\infty$. Then
$\sigma\sigma_{\kappa}^{-1}\sigma_{\kappa^{\prime}}$ cannot fix $\infty$,
because this would be equivalent to

$\kappa=(\sigma_{\kappa}\sigma\sigma_{\kappa}^{-1}\sigma_{\kappa^{\prime}})\infty=(\sigma_{\kappa}\sigma\sigma_{\kappa}^{-1})\kappa^{\prime}.$

But $\sigma_{\kappa}\sigma\sigma_{\kappa}^{-1}\in\Gamma$ and hence
$\kappa\neq\kappa^{\prime}$ cannot hold, because these two cusps are not
equivalent. Also, $\sigma_{\kappa}\sigma\sigma_{\kappa}^{-1}$ does not fix
$\kappa$, so (similarly as above) the parts
$\sigma\sigma_{\kappa}^{-1}\sigma_{\kappa^{\prime}}U_{A}$ can be added.

Hence (54) becomes

$\displaystyle\sum_{\kappa\in\mathcal{S}}\,\sum_{0\neq\alpha\in\mathbf{t}_{\kappa}/\Lambda}\,\,\int\limits_{z\in
F_{C(\gamma_{\alpha}^{\kappa})},Y_{0}(z)\leq
A}\psi\left(\frac{\alpha_{1}^{2}}{y_{1}^{2}},\dots,\frac{\alpha_{n}^{2}}{y_{n}^{2}}\right)u(\sigma_{\kappa}z)\,d\mu(z)=$
$\displaystyle=\sum_{\kappa\in\mathcal{S}}\,\sum_{0\neq\alpha\in\mathbf{t}_{\kappa}/\Lambda}\,\,\int\limits_{0\leq
X_{1}^{\kappa},\dots,X_{n}^{\kappa}<1}\,\,\,\int\limits_{Y_{0}\leq
A}\psi\left(\frac{\alpha_{1}^{2}}{y_{1}^{2}},\dots,\frac{\alpha_{n}^{2}}{y_{n}^{2}}\right)u(\sigma_{\kappa}z)\frac{dy_{1}\
\dots dy_{n}}{y_{1}^{2}\dots y_{n}^{2}}\,dx_{1}\dots dx_{n}.$

Using the Fourier expansion of $u(\sigma_{\kappa}z)$ and that for an
$l\in\mathbf{t}_{\kappa}^{*}$ we have

$\int\limits_{0\leq X_{1}^{\kappa},\dots,X_{n}^{\kappa}<1}e^{2\pi
i<l,x>}\,dx_{1}\dots
dx_{n}=\left\\{\begin{array}[]{ll}\mathrm{vol}(\mathbb{R}^{n}/\mathbf{t}_{\kappa}),&\textrm{if
}l=0,\\\\[2.84526pt] 0&\textrm{otherwise},\end{array}\right.$

the sum above can be written in the following way:

$\sum_{\kappa\in\mathcal{S}}\,\mathrm{vol}(\mathbb{R}^{n}/\mathbf{t}_{\kappa})\sum_{0\neq\alpha\in\mathbf{t}_{\kappa}/\Lambda}\,\,\int\limits_{Y_{0}\leq
A}\psi\left(\frac{\alpha_{1}^{2}}{y_{1}^{2}},\dots,\frac{\alpha_{n}^{2}}{y_{n}^{2}}\right)(\eta_{\kappa}y_{1}^{s_{1}}\dots
y_{n}^{s_{n}}+\phi_{\kappa}y_{1}^{1-s_{1}}\dots
y_{n}^{1-s_{n}})\frac{dy_{1}\dots dy_{n}}{y_{1}^{2}\dots y_{n}^{2}}.$

The substitution $u_{k}=\left|\alpha_{k}\right|/y_{k}$ gives then

$\displaystyle\sum_{\kappa\in\mathcal{S}}\,\mathrm{vol}(\mathbb{R}^{n}/\mathbf{t}_{\kappa})$
$\displaystyle\sum_{0\neq\alpha\in\mathbf{t}_{\kappa}/\Lambda}\,\,\frac{1}{\left|N\alpha\right|}\int\limits_{\begin{array}[]{c}\scriptstyle{0<u_{1},\dots,u_{n}<\infty}\\\
\scriptstyle{\left|N\alpha\right|\leq Au_{1}\dots
u_{n}}\end{array}}\psi\left(u_{1}^{2},\dots,u_{n}^{2}\right)\times$
$\displaystyle\qquad\times\left[\eta_{\kappa}\lambda_{m_{u}}(\left|\alpha\right|)\frac{\left|N\alpha\right|^{s}}{u_{1}^{s_{1}}\dots
u_{n}^{s_{n}}}+\phi_{\kappa}\lambda_{-m_{u}}(\left|\alpha\right|)\frac{\left|N\alpha\right|^{1-s}}{u_{1}^{1-s_{1}}\dots
u_{n}^{1-s_{n}}}\right]\,du_{1}\dots du_{n}$

where $\left|\alpha\right|$ denotes the coordinate-wise absolute value of the
vector $\alpha$. Hence we have to examine two terms:

(58)
$\displaystyle\eta_{\kappa}\mathrm{vol}(\mathbb{R}^{n}/\mathbf{t}_{\kappa})\int\displaylimits_{0<u_{1},\dots,u_{n}<\infty}\psi\left(u_{1}^{2},\dots,u_{n}^{2}\right)u_{1}^{-s_{1}}\dots
u_{n}^{-s_{n}}\\!\\!\\!\sum_{\begin{array}[]{c}\scriptstyle{0\neq\alpha\in\mathbf{t}_{\kappa}/\Lambda}\\\
\scriptstyle{\left|N\alpha\right|\leq Au_{1}\dots
u_{n}}\end{array}}\frac{\lambda_{m_{u}}(\left|\alpha\right|)}{\left|N(\alpha)\right|^{1-s}}\,\,du_{1}\dots
du_{n}$

and

(61)
$\displaystyle\phi_{\kappa}\mathrm{vol}(\mathbb{R}^{n}/\mathbf{t}_{\kappa})\int\displaylimits_{0<u_{1},\dots,u_{n}<\infty}\psi\left(u_{1}^{2},\dots,u_{n}^{2}\right)u_{1}^{s_{1}-1}\dots
u_{n}^{s_{n}-1}\\!\\!\\!\sum_{\begin{array}[]{c}\scriptstyle{0\neq\alpha\in\mathbf{t}_{\kappa}/\Lambda}\\\
\scriptstyle{\left|N\alpha\right|\leq Au_{1}\dots
u_{n}}\end{array}}\frac{\lambda_{-m_{u}}(\left|\alpha\right|)}{\left|N(\alpha)\right|^{s}}\,\,du_{1}\dots
du_{n}.$

We express the sums in (58) and (61) in terms of the zeta functions
$Z_{\kappa}(1-s,-m_{u})$ and $Z_{\kappa}(s,m_{u})$, respectively. By Lemma 2.8
$Z_{\kappa}(s,m_{u})$ can be continued meromorphically to $\mathbb{C}$ with
simple poles at $0$ and $1$ if and only if $m_{u}=0$, and in the latter case
its residue at $1$ is
$\frac{2^{n}|\det\mathcal{E}|}{n\cdot\mathrm{vol}(\mathbb{R}^{n}/\mathbf{t}_{\kappa})}$.
By Proposition 2.7 there is a vector $\nu\in\mathbb{R}^{n}$ with non-zero
coordinates such that the coordinates of $\nu\cdot\mathbf{t}_{\kappa}$ are
conjugate integers. Hence we may write

$\displaystyle Z_{\kappa}(s,m_{u})$
$\displaystyle=\sum_{0\neq\alpha\in\nu\cdot\mathbf{t}_{\kappa}/\Lambda}\frac{\lambda_{-m_{u}}(\left|\alpha\right|/\nu)}{\left|N(\alpha/\nu)\right|^{s}}$
$\displaystyle=|N\nu|^{s}\sum_{0\neq\alpha\in\nu\cdot\mathbf{t}_{\kappa}/\Lambda}\frac{\lambda_{-m_{u}}(\left|\alpha/\nu\right|)}{\left|N(\alpha)\right|^{s}}=|N\nu|^{s}\sum_{k=1}^{\infty}\frac{a_{m_{u}}(k)}{k^{s}},$

where

$a_{m_{u}}(k)=\sum_{0\neq\alpha\in\nu\cdot\mathbf{t}_{\kappa}/\Lambda,\,\left|N\alpha\right|=k}\lambda_{-m_{u}}(\left|\alpha\right|/\nu).$

Since $\Lambda$ is isomorphic to a finite index subgroup of the multiplicative
group of the units in $\mathcal{O}_{K}$, the latter sum can be estimated from
above by the number of integer ideals of norm $k$ in $K$ and hence by
$\tau(k)^{[K:\mathbb{Q}]}\ll_{\delta}k^{\delta}$ for any $\delta>0$, where
$\tau(k)$ is the number of divisors of the rational integer $k$.

Now we can apply Theorem 5.2 and Corollary 5.3 in [9] for the function

$\alpha_{s,m_{u}}(S)=\sum_{k=1}^{\infty}\frac{a_{m_{u}}(k)}{k^{s+S}}=\frac{Z_{\kappa}(s+S,m_{u})}{|N\nu|^{s+S}}.$

If $0<\mathrm{Re}\,s<1$ and $\sigma_{0}>1-\mathrm{Re}\,s$, then

$\displaystyle\sum_{k\leq
A}\mathop{}^{\mkern-5.0mu{}^{\prime}}\frac{a_{m_{u}}(k)}{k^{s}}=\frac{1}{2\pi
i}\int\limits_{\sigma_{0}-iT}^{\sigma_{0}+iT}\alpha_{s,m_{u}}(S)\frac{A^{S}}{S}\,dS+R_{s,m_{u}}$

where $\sum\mathop{}^{\mkern-5.0mu{}^{\prime}}$ indicates that if $A$ is an
integer, then the last term is to be counted with half weight, further

$R_{s,m_{u}}\ll\sum_{A/2<k<2A,\,k\neq
A}\left|a_{m_{u}}(k)\right|k^{-\mathrm{Re}\,s}\min\left(1,\frac{A}{T\left|A-k\right|}\right)+\frac{4^{\sigma_{0}}+A^{\sigma_{0}}}{T}\sum_{k=1}^{\infty}\frac{\left|a_{m_{u}}(k)\right|}{k^{\sigma_{0}+\mathrm{Re}\,s}}.$

Hence for any $\sigma_{0}>\mathrm{Re}\,s$ the integral in (58) can be
rewritten as

$\displaystyle|N\nu|^{1-s}$
$\displaystyle\\!\\!\\!\\!\int\displaylimits_{0<u_{1},\dots,u_{n}<\infty}\psi\left(u_{1}^{2},\dots,u_{n}^{2}\right)u_{1}^{-s_{1}}\dots
u_{n}^{-s_{n}}\times$ $\displaystyle\,\,\qquad\times\left[\frac{1}{2\pi
i}\int\limits_{\sigma_{0}-iT}^{\sigma_{0}+iT}\\!\\!\alpha_{1-s,-m_{u}}(S)\cdot\frac{(|N\nu|Au_{1}\dots
u_{n})^{S}}{S}\,dS+R_{1-s,-m_{u}}\right]\,du_{1}\dots du_{n},$

where with the notation $B=|N\nu|Au_{1}\dots u_{n}$ we have that
$R_{1-s,-m_{u}}$ is bounded by

(62) $\displaystyle\sum_{B/2<k<2B,\,k\neq
B}\left|a_{-m_{u}}(k)\right|k^{\mathrm{Re}\,s-1}\min\left(1,\frac{B}{T\left|B-k\right|}\right)+\frac{4^{\sigma_{0}}+B^{\sigma_{0}}}{T}\sum_{k=1}^{\infty}\frac{\left|a_{-m_{u}}(k)\right|}{k^{\sigma_{0}+1-\mathrm{Re}\,s}}.$

Let us fix an $0<\delta_{0}<1-\mathrm{Re}\,s$ and use the estimate
$a_{-m_{u}}(k)\ll k^{\delta_{0}}$. Also, we set the values
$\sigma_{0}=\mathrm{Re}\,s+\delta_{0}+\frac{1}{\log A}$ and
$T=A^{\mathrm{Re}\,s+\delta_{1}}$ for some
$\delta_{0}<\delta_{1}<1-\mathrm{Re}\,s$. Since $u_{k}$ is bounded from above
for any $1\leq k\leq n$, the second term on the right hand side of (62) is
bounded by $A^{\delta_{0}-\delta_{1}}\log A=o(1)$ (as $A\to\infty$), and the
implied constant depends on $\delta_{0}$.

Turning to the first term we divide the sum in it into three parts. The first
one is where $\left|B-k\right|<BT$. The terms of this part are of the form
$\left|a_{-m_{u}}(k)\right|k^{\mathrm{Re}\,s-1}\ll
k^{\delta_{0}+\mathrm{Re}\,s-1}$ hence they give at most a constant times
$(BT)\cdot B^{\delta_{0}+\mathrm{Re}\,s-1}=(|N\nu|u_{1}\dots
u_{n})^{\delta_{0}+\mathrm{Re}\,s}A^{\delta_{0}-\delta_{1}}$, and since $\psi$
is compactly supported this gives an $o(1)$ term in the last integral above.

The second part is where $BT\leq\left|B-k\right|<BT+1$, i.e. it consists of at
most two terms bounded by a constant times

$k^{\delta_{0}+\mathrm{Re}\,s-1}\frac{B}{T\left|B-k\right|}\ll\frac{B^{\delta_{0}+\mathrm{Re}\,s}}{T^{2}B}\leq
B^{\delta_{0}+\mathrm{Re}\,s-1}=(|N\nu|Au_{1}\dots
u_{n})^{\delta_{0}+\mathrm{Re}\,s-1}.$

Hence we obtain a term in the integral above that can be bounded by

$A^{\delta_{0}+\mathrm{Re}\,s-1}\int\limits_{0<u_{1},\dots,u_{n}<C}(u_{1}\dots
u_{n})^{\delta_{0}-1}\,du_{1}\dots du_{n}$

for some $C>0$, and the latter integral converges at $0$ giving an $o(1)$ term
as $A\to\infty$.

The third part is where $|BT+1|\leq\left|B-k\right|$. Note that in this case
$1\leq\left|B-k\right|\leq B$, hence this error term is bounded by

$\displaystyle
B^{\delta_{0}+\mathrm{Re}\,s-1}\cdot\frac{B}{T}\sum_{1\leq\left|B-k\right|\leq
B}\frac{1}{\left|B-k\right|}$ $\displaystyle\ll(|N\nu|u_{1}\dots
u_{n})^{\delta_{0}+\mathrm{Re}\,s}A^{\delta_{0}-\delta_{1}}\sum_{1\leq k\leq
B}\frac{1}{k}$ $\displaystyle\ll A^{\delta_{0}-\delta_{1}}\max(0,\log B)\ll
A^{\delta_{0}-\delta_{1}}\log A=o(1).$

Finally, to cover also those cases when $k=|N\nu|Au_{1}\dots u_{n}$ is an
integer we may add

$\frac{1}{2}\cdot\frac{a_{-m_{u}}(k)}{k^{1-s}}\ll(|N\nu|Au_{1}\dots
u_{n})^{\delta_{0}+\mathrm{Re}\,s-1}$

to the error term $R_{1-s,-m_{u}}$, which also gives an $o(1)$ term.

It follows that aside from an $o(1)$ term the expression in (58) is

$\displaystyle\eta_{\kappa}\mathrm{vol}(\mathbb{R}^{n}/\mathbf{t}_{\kappa})|N\nu|^{1-s}\\!\\!\\!\\!\\!$
$\displaystyle\int\displaylimits_{0<u_{1},\dots,u_{n}<\infty}\psi\left(u_{1}^{2},\dots,u_{n}^{2}\right)u_{1}^{-s_{1}}\dots
u_{n}^{-s_{n}}\times$ $\displaystyle\,\qquad\times\left(\frac{1}{2\pi
i}\int\limits_{\sigma_{0}-iT}^{\sigma_{0}+iT}\\!\\!\alpha_{1-s,-m_{u}}(S)\cdot\frac{(|N\nu|Au_{1}\dots
u_{n})^{S}}{S}\,dS\right)du_{1}\dots du_{n}$

where $\sigma_{0}=\mathrm{Re}\,s+\delta_{0}+\frac{1}{\log A}$ and
$T=A^{\mathrm{Re}\,s+\delta_{1}}$ for some
$\delta_{0}<\delta_{1}<1-\mathrm{Re}\,s$. Substituting the definition of
$\alpha_{1-s,-m_{u}}(S)$ and interchanging the order of integration this
becomes

$\displaystyle\frac{\eta_{\kappa}\mathrm{vol}(\mathbb{R}^{n}/\mathbf{t}_{\kappa})}{2\pi
i}\int\limits_{\sigma_{0}-iT}^{\sigma_{0}+iT}F(S)Z_{\kappa}(1-s+S,-m_{u})\frac{A^{S}}{S}\,dS,$

where

$\displaystyle
F(S)=\int\limits_{0}^{\infty}\dots\int\limits_{0}^{\infty}\psi\left(u_{1}^{2},\dots,u_{n}^{2}\right)u_{1}^{S-s_{1}}\dots
u_{n}^{S-s_{n}}\,du_{1}\dots\,du_{n}.$

Let us choose a number $\mathrm{Re}\,s-1<\sigma_{1}<0$ and set
$G(S)=F(S)Z_{\kappa}(1-s+S,-m_{u})\frac{A^{S}}{S}$. We shift the line of
integration to the line $\sigma_{1}+it$, then by the residue theorem

$\displaystyle\frac{1}{2\pi
i}\int\limits_{\sigma_{0}-iT}^{\sigma_{0}+iT}G(S)\,dS=$ $\displaystyle
Z_{\kappa}(1-s,-m_{u})F(0)+\delta_{m_{u}}\cdot\frac{2^{n}|\det\mathcal{E}|}{n\cdot\mathrm{vol}(\mathbb{R}^{n}/\mathbf{t}_{\kappa})}\cdot\frac{A^{s}}{s}F\left(s\right)$
(63) $\displaystyle-\frac{1}{2\pi
i}\left(\,\,\int\limits_{\sigma_{0}+iT}^{\sigma_{1}+iT}G(S)\,dS+\int\limits_{\sigma_{1}+iT}^{\sigma_{1}-iT}G(S)\,dS+\int\limits_{\sigma_{1}-iT}^{\sigma_{0}-iT}G(S)\,dS\right).$

We show that the last three integral above is $o(1)$ as $A\to\infty$. Firstly,
repeated integration by parts in $F(S)$ with respect to $u_{1}$ (for example)
gives

$G(S)=Z_{\kappa}(1-s+S,-m_{u})\frac{A^{S}}{S^{N+1}}\int\limits_{0}^{\infty}\dots\int\limits_{0}^{\infty}H_{s_{1}}^{(N)}(u_{1},\dots,u_{n})u_{1}^{S-s_{1}}\dots
u_{n}^{S-s_{n}}\,du_{1}\dots\,du_{n}$

where $N$ is any positive integer and $H_{s_{1}}^{(N)}$ is a compactly
supported smooth function.

To estimate the integrals on the right hand side of (3.5) we apply Lemma 2.8:
if $0\leq\mathrm{Re}\,s$, then we have
$Z_{\kappa}(s,-m_{u})\ll\left|\textrm{Im}\,s\right|^{n/2+\varepsilon}$ for any
$\varepsilon>0$ as $\left|t\right|\to\infty$, hence on the horizontal segments
we have

$\displaystyle Z_{\kappa}(1-s+S,-m_{u})\frac{(u_{1}\dots
u_{n}A)^{S}}{S^{N+1}}$
$\displaystyle\ll\frac{A^{\mathrm{Re}\,S}}{T^{N+1}}\cdot
T^{n/2+\varepsilon}\ll
A^{\mathrm{Re}\,S+(\mathrm{Re}\,s+\delta_{1})(n/2+\varepsilon-N-1)}.$

Here (by the compact support of $H_{s_{1}}^{(N)}$) $u_{1},\dots,u_{n}$ can be
bounded from above by a constant. Choosing an appropriate $N$ it follows that
these integrals give $o(1)$ terms. On the vertical line we have

$\displaystyle Z_{\kappa}(1-s+S,-m_{u})\frac{(u_{1}\dots
u_{n}A)^{S}}{S^{N+1}}$
$\displaystyle\ll\frac{A^{\sigma_{1}}}{(1+\left|\textrm{Im}\,S\right|)^{N+1}}\cdot\left|\textrm{Im}\,S\right|^{n/2+\varepsilon}\ll
A^{\sigma_{1}}\left|\textrm{Im}\,S\right|^{n/2+\varepsilon-N-1}$

if $\left|\textrm{Im}\,S\right|$ is big enough and hence (choosing an
$N>n/2+\varepsilon-1$)

$\displaystyle\int\limits_{\sigma_{1}+iT}^{\sigma_{1}-iT}G(S)\,dS$
$\displaystyle\ll A^{\sigma_{1}}=o(1).$

The expression (58) becomes

$\eta_{\kappa}\mathrm{vol}(\mathbb{R}^{n}/\mathbf{t}_{\kappa})Z_{\kappa}(1-s,-m_{u})F(0)+\delta_{m_{u}}\cdot\frac{\eta_{\kappa}2^{n}|\det\mathcal{E}|}{n}\cdot\frac{A^{s}}{s}F\left(s\right)+o(1)$

One can show similarly that (61) is

$\phi_{\kappa}\mathrm{vol}(\mathbb{R}^{n}/\mathbf{t}_{\kappa})Z_{\kappa}(s,m_{u})\tilde{F}(0)+\delta_{m_{u}}\cdot\frac{\phi_{\kappa}2^{n}|\det\mathcal{E}|}{n}\cdot\frac{A^{1-s}}{1-s}\tilde{F}\left(1-s\right)+o(1)$

where

$\displaystyle\tilde{F}(S)=\int\limits_{0}^{\infty}\dots\int\limits_{0}^{\infty}\psi\left(u_{1}^{2},\dots,u_{n}^{2}\right)u_{1}^{S+s_{1}-1}\dots
u_{n}^{S+s_{n}-1}\,du_{1}\dots\,du_{n}.$

Next we calculate the values $2^{n}F(s)$ and $2^{n}\tilde{F}(s)$ in the case
$m_{u}=0$, when

$\displaystyle 2^{n}F(s)=2^{n}\tilde{F}(1-s)=$ $\displaystyle
2^{n}\int\limits_{0}^{\infty}\dots\int\limits_{0}^{\infty}\psi\left(u_{1}^{2},\dots,u_{n}^{2}\right)\prod\limits_{k=1}^{n}du_{k}$
$\displaystyle=\int\limits_{0}^{\infty}\dots\int\limits_{0}^{\infty}\frac{\psi(t_{1},\dots,t_{n})}{\prod\limits_{k=1}^{n}\sqrt{t_{k}}}\,\prod_{k=1}^{n}\,dt_{k}=g(0,\dots,0).$

Finally, we evaluate $F$ and $\tilde{F}$ at $0$:

$\displaystyle F(0)$
$\displaystyle=\int\limits_{0}^{\infty}\dots\int\limits_{0}^{\infty}\psi(u_{1}^{2},\dots,u_{n}^{2})u_{1}^{-s_{1}}\dots
u_{n}^{-s_{n}}\,du_{1}\dots\,du_{n}$
$\displaystyle=\frac{1}{2^{n}}\int\limits_{0}^{\infty}\dots\int\limits_{0}^{\infty}\frac{\psi(u_{1},\dots,u_{n})}{\sqrt{u_{1}\dots
u_{n}}}u_{1}^{-\frac{s_{1}}{2}}\dots
u_{n}^{-\frac{s_{n}}{2}}\,du_{1}\dots\,du_{n}.$

Using (2.1) we get that this is

$\displaystyle\frac{(-1)^{n}}{(2\pi)^{n}}\int\limits_{0}^{\infty}\dots\int\limits_{0}^{\infty}\left(\,\int\limits_{u_{n}}^{\infty}\dots\int\limits_{u_{1}}^{\infty}\frac{\frac{\partial^{n}Q}{\partial
w_{1}\dots\partial
w_{n}}(w_{1},\dots,w_{n})}{\sqrt{w_{1}-u_{1}}\dots\sqrt{w_{n}-u_{n}}}\,dw_{1}\dots\,dw_{n}\right)\frac{u_{1}^{-\frac{s_{1}}{2}}\dots
u_{n}^{-\frac{s_{n}}{2}}}{\sqrt{u_{1}\dots u_{n}}}\,du_{1}\dots\,du_{n}$
$\displaystyle=\frac{(-1)^{n}}{(2\pi)^{n}}\int\limits_{0}^{\infty}\dots\int\limits_{0}^{\infty}\frac{\partial^{n}Q}{\partial
w_{1}\dots\partial
w_{n}}(w_{1},\dots,w_{n})\left(\prod_{k=1}^{n}\int\limits_{0}^{w_{k}}\frac{u_{k}^{-\frac{s_{k}}{2}}}{\sqrt{w_{k}-u_{k}}\sqrt{u_{k}}}\,du_{k}\right)\,dw_{1}\dots\,dw_{n}.$

We have

$\int\limits_{0}^{w}\frac{u^{-\frac{\alpha}{2}}}{\sqrt{w-u}\sqrt{u}}\,du=w^{-\frac{\alpha}{2}}B\left(\frac{1-\alpha}{2},\frac{1}{2}\right)=w^{-\frac{\alpha}{2}}\frac{\Gamma(\frac{1-\alpha}{2})\Gamma(\frac{1}{2})}{\Gamma(\frac{1-\alpha}{2}+\frac{1}{2})}=w^{-\frac{\alpha}{2}}2^{-\alpha}\frac{\Gamma(\frac{1-\alpha}{2})^{2}}{\Gamma(1-\alpha)}$

for any $w>0$ and $\alpha\in\mathbb{C}$ with $0<\mathrm{Re}\,\alpha<1$, where
$B$ is the beta function and we used the following relations:

$\Gamma\left(1/2\right)=\sqrt{\pi},\qquad\Gamma(z)\Gamma\left(z+\frac{1}{2}\right)=2^{1-2z}\sqrt{\pi}\Gamma(2z).$

Then $F(0)$ is

$\frac{(-1)^{n}}{(2^{1-s}\pi)^{n}}\left(\prod_{k=1}^{n}\frac{\Gamma(\frac{1-s_{k}}{2})^{2}}{\Gamma(1-s_{k})}\right)\int\limits_{0}^{\infty}\dots\int\limits_{0}^{\infty}\frac{\partial^{n}Q}{\partial
w_{1}\dots\partial w_{n}}(w_{1},\dots,w_{n})w_{1}^{-\frac{s_{1}}{2}}\dots
w_{n}^{-\frac{s_{n}}{2}}\,dw_{1}\dots\,dw_{n}.$

We substitute $w_{k}=e^{x_{k}}+e^{-x_{k}}-2$ to express this in terms of the
function $g$, and then the integral above becomes

$\int\limits_{0}^{\infty}\dots\int\limits_{0}^{\infty}\frac{\partial^{n}g}{\partial
x_{1}\dots\partial
x_{n}}(x_{1},\dots,x_{n})\prod_{k=1}^{n}(e^{x_{k}}+e^{-x_{k}}-2)^{-\frac{s_{k}}{2}}\,dx_{k}.$

Now

$\frac{\partial^{n}g}{\partial x_{1}\dots\partial
x_{n}}(x_{1},\dots,x_{n})=\frac{(-i)^{n}}{(2\pi)^{n}}\int\limits_{0}^{\infty}\dots\int\limits_{0}^{\infty}h(r_{1},\dots,r_{n})r_{1}\dots
r_{n}e^{-i(r_{1}x_{1}+\dots+r_{n}x_{n})}\,dr_{1}\dots\,dr_{n},$

and

$\displaystyle\int\limits_{0}^{\infty}(e^{x}+e^{-x}-2)^{-\alpha}e^{-irx}\,dx$
$\displaystyle=\int\limits_{0}^{1}\left(y^{-\frac{1}{2}}-y^{\frac{1}{2}}\right)^{-2\alpha}y^{ir-1}\,dy=\int\limits_{0}^{1}\left(1-y\right)^{-2\alpha}y^{\alpha+ir-1}\,dy$
$\displaystyle=B(\alpha+ir,1-2\alpha)=\frac{\Gamma(\alpha+ir)\Gamma(1-2\alpha)}{\Gamma(1-\alpha+ir)}.$

We conclude that $F(0)$ is

$\left(\frac{i}{2^{2-s}\pi^{2}}\right)^{n}\left(\prod_{k=1}^{n}\Gamma\left(\frac{1-s_{k}}{2}\right)^{2}\right)\int\limits_{0}^{\infty}\dots\int\limits_{0}^{\infty}h(r_{1},\dots,r_{n})\prod_{k=1}^{n}\frac{r_{k}\Gamma(\frac{s_{k}}{2}+ir_{k})}{\Gamma(\frac{2-s_{k}}{2}+ir_{k})}\,dr_{k}.$

Similarly, $\tilde{F}(0)$ is

$\left(\frac{i}{2^{s+1}\pi^{2}}\right)^{n}\left(\prod_{k=1}^{n}\Gamma\left(\frac{s_{k}}{2}\right)^{2}\right)\int\limits_{0}^{\infty}\dots\int\limits_{0}^{\infty}h(r_{1},\dots,r_{n})\prod_{k=1}^{n}\frac{r_{k}\Gamma(\frac{1-s_{k}}{2}+ir_{k})}{\Gamma(\frac{s_{k}+1}{2}+ir_{k})}\,dr_{k},$

and this completes the proof of Theorem 2.9.

## References

  * [1] A. Biró.  On a generalization of the Selberg trace formula.  Acta Arithmetica, 87(4):319–338, 1999.
  * [2] A. Biró.  Cycle integrals of Maass forms of weight 0 and Fourier coefficients of Maass forms of weight 1/2.  Acta Arithmetica, 94(2):103–152, 2000.
  * [3] A. Biró.  A relation between triple products of weight 0 and weight 1/2 cusp forms.  Israel Journal of Mathematics, 182(1):61–101, 2011.
  * [4] A. Biró.  Local average of the hyperbolic circle problem for Fuchsian groups.  Mathematika, 64(1):159–183, 2018.
  * [5] D. Bump.  Automorphic forms and representations.  Number 55 in Cambridge Studies in Advanced Mathematics. Cambridge university press, 1998.
  * [6] I. Y. Efrat.  The Selberg trace formula for $PSL_{2}(R)^{n}$, volume 359 of Mem. Amer. Math. Soc. American Mathematical Soc., 1987.
  * [7] E. Freitag.  Hilbert modular forms.  In Hilbert Modular Forms, pages 5–71. Springer-Verlag, Berlin, 1990\.
  * [8] H. Iwaniec.  Spectral methods of automorphic forms, volume 53 of Graduate Studies in Mathematics.  American Mathematical Soc., 2002.
  * [9] H. L. Montgomery and R. C. Vaughan.  Multiplicative number theory I: Classical theory, volume 97 of Cambridge Studies in Advanced Mathematics.  Cambridge University Press, 2007.
  * [10] A. Selberg.  Harmonic analysis and discontinuous groups in weakly symmetric Riemannian spaces with applications to Dirichlet series.  J. Indian Math. Soc., 20:47–86, 1956.
  * [11] H. Shimizu.  On discontinuous groups operating on the product of the upper half planes.  Annals of Mathematics, 77:33–71, 1963.
  * [12] C. L. Siegel and K. Ramanathan.  Advanced analytic number theory, volume 9 of Tata Institute of Fundamental Research Studies in Mathematics.  Tata Institute of Fundamental Research Bombay, 1980.
  * [13] D. Tóth.  A generalization of the Selberg trace formula.  Doctoral dissertation, Central European University. 2021.  https://www.etd.ceu.edu/2021/toth_david.pdf.
  * [14] J. L. Truelsen.  Quantum unique ergodicity of Eisenstein series on the Hilbert modular group over a totally real field.  Forum Mathematicum, 23:891–931, 2011.


acoustics actuators addictions admsci adolescents aerospace agriculture
agriengineering agronomy ai algorithms allergies analytica animals antibiotics
antibodies antioxidants appliedchem appliedmath applmech applmicrobiol
applnano applsci aquacj architecture arts asc asi atmosphere atoms audiolres
automation axioms bacteria batteries bdcc behavsci beverages biochem
bioengineering biologics biology biomass biomechanics biomed biomedicines
biomedinformatics biomimetics biomolecules biophysica biosensors biotech birds
bloods blsf brainsci breath buildings businesses cancers carbon cardiogenetics
catalysts cells ceramics challenges chemengineering chemistry chemosensors
chemproc children cimb civileng cleantechnol climate clinpract clockssleep cmd
coasts coatings colloids colorant compounds computation computers
condensedmatter conservation constrmater cosmetics covid crops cryptography
crystals ctn curroncol cyber dairy data dentistry dermato dermatopathology
designs diabetology diagnostics digital disabilities diseases diversity dna
drones dynamics earth ebj ecologies econometrics economies education ejihpe
electricity electrochem electronicmat electronics encyclopedia endocrines
energies eng engproc entropy environments environsciproc epidemiologia
epigenomes est fermentation fibers fintech fire fishes fluids foods
forecasting forensicsci forests foundations fractalfract fuels futureinternet
futurepharmacol futurephys futuretransp galaxies games gases gastroent
gastrointestdisord gels genealogy genes geographies geohazards geomatics
geosciences geotechnics geriatrics hazardousmatters healthcare hearts hemato
heritage highthroughput histories horticulturae humanities humans hydrobiology
hydrogen hydrology hygiene idr ijerph ijfs ijgi ijms ijns ijtm ijtpp immuno
informatics information infrastructures inorganics insects instruments
inventions iot j jal jcdd jcm jcp jcs jdb jfb jfmk jimaging jintelligence
jlpea jmmp jmp jmse jne jnt jof joitmc jor journalmedia jox jpm jrfm jsan
jtaer jzbg kidneydial knowledge land languages laws life liquids literature
livers logistics lubricants machines macromol magnetism magnetochemistry make
marinedrugs materials materproc mathematics mca medicina medicines medsci
membranes merits metabolites metals metrology micro microarrays microbiolres
micromachines microorganisms microplastics minerals mining modelling molbank
molecules mps msf mti nanoenergyadv nanomanufacturing nanomaterials ncrna
network neuroglia neurolint neurosci nitrogen notspecified nri nursrep
nutraceuticals nutrients obesities oceans ohbm onco optics oral organics
osteology oxygen parasites parasitologia particles pathogens pathophysiology
pediatrrep pharmaceuticals pharmaceutics pharmacy philosophies photochem
photonics phycology physchem physics physiologia plants plasma pollutants
polymers polysaccharides poultry preprints proceedings processes prosthesis
proteomes psf psych psychiatryint publications quantumrep quaternary qubs
radiation reactions recycling regeneration religions remotesensing reports
reprodmed resources rheumato risks robotics ruminants safety sci scipharm
sensors separations sexes signals sinusitis skins smartcities sna societies
socsci soilsystems solar solids sports standards stats stresses surfaces
surgeries suschem sustainability symmetry systems taxonomy technologies
telecom test textiles thermo tomography tourismhosp toxics toxins
transplantology transportation traumacare traumas tropicalmed universe
urbansci uro vaccines vehicles venereology vetsci vibration viruses vision
water wem wevj wind women world youth zoonoses


Fine-tuning language models on tasks with instructions has demonstrated potential in facilitating zero-shot generalization to unseen tasks.
In this paper, we introduce a straightforward yet effective method for enhancing instruction tuning by employing symbolic tasks.
Compared to crowdsourced human tasks or model-generated tasks, symbolic tasks present a unique advantage as they can be easily generated in vast quantities, theoretically providing an infinite supply of high-quality training instances.
To explore the potential of symbolic tasks, we carry out an extensive case study on the representative symbolic task of SQL execution.
Empirical results on various benchmarks validate that the integration of SQL execution leads to significant improvements in zero-shot scenarios, particularly in table reasoning.
Notably, our $3$B model surpasses both the $175$B GPT-3 and ChatGPT in zero-shot table reasoning across four benchmarks.
Furthermore, experimental results on BBH ($27$ tasks) and MMLU ($57$ tasks) reveal that language models can be enhanced through symbolic tasks without compromising their generality.
We hope that our paper serves as a catalyst, inspiring increased efforts to incorporate symbolic tasks in instruction tuning.

§ INTRODUCTION

In recent years, the development of large language models (LM) has been one of the most significant advances in natural language processing (NLP) <cit.>.
After being trained on large amounts of natural language corpus <cit.> with the language modeling objective, LMs have demonstrated impressive performance on a variety of NLP tasks.
Furthermore, recent progress show that LMs are able to do zero-shot task generalization, which means they can adapt to unseen tasks without any specific fine-tuning on those tasks.
Along this way, one promising direction is the instruction tuning <cit.>.
By fine-tuning LMs to follow instructions on diverse tasks, instruciton tuning enables LMs to perform well on tasks that they have not been explicitly trained on.

Despite the popularity, most existing instruction tuning methods focus on crowdsourced human tasks <cit.> or model-generated tasks <cit.> for instruction tuning, which are either in limited quantity or quality.
As scaling up language models in different dimensions has shown promise in pushing the boundaries of zero-shot performance, the search for high-quality and scalable instruction tuning tasks has become increasingly important.
In this paper, we draw inspiration from the work of TaPEx <cit.> and investigate the use of symbolic tasks as a complementary training resource for instruction tuning.

Language models are fine-tuned with both natural language tasks (e.g., fact verificaiton) and symbolic tasks (e.g., SQL execution), and are evaluated on unseen tasks (e.g., table fact verificaiton).

A symbolic task is a form of problem-solving or computational process that entails the manipulation of symbolic representations or structured data (e.g., sorting a disordered array).
In general, symbolic tasks are characterized by the use of symbols in formal systems, such as logic, mathematics, or programming languages, instead of relying on natural language.
Unlike natural language, symbols in symbolic tasks follow a limited set of grammatical rules, making them easier to synthesize systematically.
Meanwhile, these symbols can quickly expand in scale through composition, and can be executed on symbolic executors to yield the corresponding output.
As a result, generating diverse, complex, and consistent examples for symbolic tasks is relatively easy compared to crowdsourced human tasks or model-generated tasks.
This ease of generation theoretically allows for a large number of high-quality examples to be created for instruction tuning.
Given the advantageous properties of symbolic tasks, we believe symbolic tasks can serve as a strong complement as a source for instruction tuning.
The core insight is that if a model trained with instruction tuning is capable of performing both symbolic tasks and natural language tasks, then the underlying abilities learned from symbolic tasks can be transferred to unseen natural language tasks.

Motivated by the above, this paper presents a comprehensive study on SQL execution as a representative example to examine the power of symbolic tasks in instruction tuning.
SQL execution refers to the process of executing SQL statements to interact with tables.
Our goal is to extend this interaction between SQL and tables to NL-based table reasoning without using any table reasoning example.
Empirical results demonstrate that the incorporation of SQL execution yields substantial improvements across different benchmarks of table reasoning.
To highlight, our $3$B model outperforms both $175$B GPT-3 (code-davinci-002) and ChatGPT (gpt3.5-turbo) in zero-shot table reasoning across four benchmarks.
In addition to being used with training, symbolic tasks can also be leveraged by large language models (e.g., GPT-3) without training, as part of their instructions to boost their performance.
For example, without any realistic example in , SQL execution can boost the GPT-3 performance from $40.4\%$ to $52.9\%$.
Furthermore, we find the task not only enhances performance on table reasoning but also leads to a significant improvement in mathematical reasoning performance as measured by the SVAMP benchmark, increasing the score from $21.4$% to $26.5$%.

More importantly, experimental results on BBH ($27$ tasks) and MMLU ($57$ tasks) show that fine-tuning with symbolic tasks does not hurt the model performance on generic held-out tasks, implying that we can improve LMs through symbolic tasks without compromising their generality.
Based on this observation, we advocate for the research community to expand the scale of symbolic tasks together, pushing the boundaries of zero-shot learning.
As part of our ongoing commitment to collaborative research, we would love to share credits with all contributors who provides any contribution.
For more specific details of our project, please refer to our open-sourced project <https://github.com/sail-sg/symbolic-instruction-tuning>.

§ RELATED WORK

Our work leverages symbolic tasks in instruction tuning, and thus our work is closely related to instruction tuning. Meanwhile, since symbolic tasks are one typical case of synthetic tasks, we also discuss about the relationship between ours and previous work which uses synthetic tasks in training.

Instruction Tuning

Instruction tuning is a popular direction in NLP to enable zero-shot generalization on unseen tasks <cit.>.
The core idea behind instruction tuning is that LMs are fine-tuned to accomplish diverse tasks by following a set of instructions.
Therefore, the task source becomes important in instruction tuning <cit.>.
Previous studies generally collect existing publicly avaiable datasets for the training, including T0 <cit.>, FLAN <cit.> and NaturalInstructions <cit.>, or carefully crowdsource diverse tasks via human annotations, including Super-NaturalInstructions <cit.> and InstructGPT <cit.>.
While these mixtures of tasks are generally of high quality, they often rely on a significant amount of human effort and are usually limited in quantity.
We believe that our proposed symbolic tasks could be a valuable addition to address the quantity limitation in this line.
Recently, several researchers have also explored the usage of model-generated tasks, which invoke a large language model (e.g., GPT-3) to generate a diverse set of instructions, task inputs, and task outputs based on a high-quality seed set.
However, these model-generated datasets not only rely on a powerful LM, but also introduce a lot noise (i.e., task output does not correspond to the task input) in the data.[Some discussion about the quality of GPT-3 generated data in the Alpaca project <cit.> can be found in <https://github.com/gururise/AlpacaDataCleaned>.]
In contrast, in symbolic tasks, we can ensure that the task output corresponds to the task input because the output is automatically obtained from reliable symbolic executors.

Synthetic Training

Synthetic training is a technique that leverages synthetic tasks, whose examples can be synthesized automatically, to improve the model performance on downstream tasks.
Previous research has explored the power of synthetic tasks in several aspects, including attaining natural language pre-training performance using an artificial task <cit.>, the usage of SQL execution in table pre-training <cit.>, the application of synchronous context-free grammars in text-to-SQL parsing <cit.>, the use of synthetic data in numerical reasoning <cit.>, and the use of environment-adaptive synthetic data in language-based environment manipulation tasks <cit.>.
In this line, the most related work to ours is the work of TaPEx <cit.>, which introduces SQL execution in the pre-training phase.
However, TaPEx still required fine-tuning on downstream datasets to work <cit.>, while our work can perform zero-shot generalization on downstream tasks without any fine-tuning on the downstream task.
More recently, <cit.> also made a noteworthy contribution in the field of zero-shot table reasoning.
Nonetheless, our approach differs in its strategy for leveraging symbolic tasks.
For example, their method relies on a trained NL generator to ensure naturalness of the model input, which necessitates parallel program and NL corpus.
In contrast, our approach does not require translating the program in symbolic tasks to NL. Instead, we focus on leveraging instruction tuning to directly project the underlying abilities in symbolic tasks onto unseen NL tasks.

§ EXAMING THE POWER OF SYMBOLIC TASKS

In this section, we first describe how we conduct the data synthesis for the SQL execution corpus, and then we introduce two ways to leverage our proposed symbolic tasks in instruction tuing: multi-task fine-tuning and synthetic demonstrations.
The multi-task fine-tuning method is a training-involved use case of our symbolic tasks, where we need jointly training the LM on the symbolic task and other NL tasks.
The synthetic demonsration method is a training-free use case, where we only need synthesize the symbolic examples and insert them into the current instruction, and then feed them into LMs.
Below we first briefly describe the data synthesis procedure of the SQL execution task, then we describe the two methods in detail.

§.§ Data Synthesis

Since we leverage SQL execution as the only symbolic task, the most important factors for the data synthesis are the table source and the SQL queries.
Regarding the source of the table, we go into more details in the next sections, and first we introduce how the SQL query is synthesized.

In principle, we follow <cit.> to synthesize training instances for SQL execution of symbolic tasks.
Concretely, we obtain them using the SQL templates from the Squall dataset <cit.>.
For instance, a typical SQL template might look like , with  representing a numeric column and  a text column, while  corresponds to a specific cell value related to the  column. To create concrete SQL queries, we randomly choose headers and cell values from a given table to populate the template.
By instantiating SQL templates on the table source, we can synthesize high-quality examples at any scale.

Given an executable SQL query and a table $T$, where $T$ consists of $M$ rows $\{r_i\}_{i=1}^M$ and $N$ headers $\{c_j\}_{j=1}^N$, we first flatten the table into a sequence $T^*$, and then concatenate it with the SQL query and the task instruction to create each training example.
A typical example can be found in fig:mutli_task_fintune, where the task instruction is “” and the model is “”.
The supervision for the model is the execution result of the SQL query “”, which can be obtained by running the SQL query through an off-the-shelf SQL executor, such as MySQL.

It is worthy to note that the task instruction is always “” for all symbolic task examples.
And the table representations shown in fig:model_overview and  fig:mutli_task_fintune are for illustration purpose.
In practise, to obtain the flattend table $T^*$ without losing the table structure information, we include various special tokens to indicate the boundaries inside the table.
Denoting a flattened table as $T^*=\texttt{\small [HEAD]},c_1,{\cdots},c_N,\texttt{\small [ROW]},1,r_1,\texttt{\small [ROW]},2,r_2,{\cdots},r_M$, the special tokens  and  represent the table header and rows respectively. Additionally, the number following  indicates the row index.

§.§ Training-Involved Use Case: Multi-Task Fine-tuning

The illustration of our multi-task fine-tuning procedure, where nearly $200$K examples from the symbolic task are merged with $600$K examples from the FLAN task mixture <cit.>. The language model is initialized from FLAN-T5 <cit.>.

Several works have demonstrated the importance of mult-task fine-tuning in the instructon tuning paradigm <cit.>, especially the diversity of tasks.
Motivated by them, the straightforward method to leverage the symbolic task is to incorporate it as part of the fine-tuned tasks, with diverse examples.
To make the task more diverse, we follow <cit.> to synthesize SQL queries based on a collection of different tables.
Meanwhile, rather than crawling noisy tables from the Internet and subsequently applying heuristic filtering, we choose clean tables directly from available public datasets.
Specifically, we randomly select close to $1000$ tables from the training set  <cit.> to serve as the table source for the SQL execution task.

To verify the proposed approach in multi-task fine-tuning, we follow the popular instruction tuning work FLAN <cit.>, which has demostarted great zero-shot performance on several challeging benchmarks such as MMLU <cit.> and BBH <cit.>.
However, FLAN collected $1836$ NL tasks for instruction tuning, which consumes a lot of time and resources to perform fine-tuning from scratch.
Therefore, inspried by <cit.> which argues that fine-tuned LMs can continually learn new tasks without catastrophic forgetting, we employ the rehearsal strategy, where a small amount (e.g., $1$%) of data in FLAN tasks is replayed during training.
As shown in fig:mutli_task_fintune, the training starts from the weights of , and we leverage $600$K examples from the FLAN task mixture, along with $200$K examples from our synthetic corpus.
Compared to multi-task fine-tuning from scratch, such kind of rehearsal strategy saves a lot of computation and allows us to reuse the well-trained weights.

§.§ Training-Free Use Case: Synthetic Demonstration

The comparison between different instruction strategies on .

Multitasking fine-tuning is applicable to language models in general, but not necessarily to large language models, as the cost of fine-tuning can be high.
To explore whether symbolic tasks work without fine-tuning, we suggest injecting them to LMs by means of synthetic demonstrations, where the symbolic task is employed as part of the instruction to LMs.
fig:demo_methods illustrates different instruction strategies to prompt large LMs, including the zero-shot, synthetic demonstration and realistic demonstration.

Under the zero-shot setting, the instruction to the model only contains the task description and the task input, without any demonstration, while the few-shot setting introduces a few downstream examples as realistic demonstrations for the model to perform in-context learning <cit.>.
Different from both them, we argue that we can leverage the SQL query and its execution result, i.e., the synthetic demonstration, as part of the instruction.
As shown in the middle of fig:demo_methods, the synthetic demonstrations can be flexibly synthesized without accessing realistic examples.
In practice, we have the flexibility to synthesize the corresponding SQL queries and obtain their execution results based on the task related table.

§ EXPERIMENTS

In this section, we perform experiments to answer three research questions (RQ):

* RQ1. Does the method benefit table reasoning without using any realistic example?

* RQ2. Does the method benefit tasks beyond table tasks?

* RQ3. Does the method hurt the model performance on generic tasks?

§.§ Experimental Setup

Evaluation Task We evaluate our approach on $7$ tasks. For RQ1, we perform experiments on typical table reasoning benchmarks, such as weakly-supervised WikiSQL () <cit.>, WikiTableQuestions () <cit.>,  <cit.>, and  <cit.>.
The first three tasks are in the line of table question answering, while the last one focuses on table fact verification.
and are both large crowd-sourced datasets for developing single-turn question answering systems on tables.
Compared to , which necessitates only filtering and optionally aggregating cell values, demands more advanced reasoning capabilities like sorting.
is a conversational benchmark, which requires models to take into account the conversational context.
is a multi-choice task, which is to determine whether a given statement aligns with the facts presented in a specified table.
For RQ2, we conduct experiments on SVAMP <cit.>, a challenging diagnostic dataset for probing general numercial reasoning over text. Note that we do not include GSM8K <cit.> since small models, such as our base model and , cannot obtain a reasonable performance. We leave investigation on more benchmarks in the future.
For RQ3, we follow <cit.> to evaluate models on two benchmark datasets: BBH ($27$ tasks) <cit.> and MMLU ($57$ tasks) <cit.>. These benchmarks consist of a set of multiple-choice questions, sourced from a group of humans across various domains. These benchmarks have been recognized as challenging for language models to provide accurate answers.

The instruction used in zero-shot evaluation of for different models. The instruction for  is borrowed from <cit.>, while the instruction for  is inspried by the Promptify repo.

Model     Prompt with {variable}
Read the table below to answer the following questions.

Question: {question}

The answer is
You are a highly intelligent question answering bot. You take the following Table and Question as input and return the answer from the Table. Retain as much information as needed to answer the question at a later time. Your output format is only “Answer: Extracted Answer” form, no other form. And the output should be number or entities, as short as possible, without any explanation.

/      Here is a table: {table}.

Answer the following question: {question}

Baseline Methods Since we aim to build strong zero-shot models for downstream tasks, we mainly compare the performance of our approach with several zero-shot methods. First, we include the base models and . Second, large language models are known to have good zero-shot performance, and has been demonstrated to have strong abiltieis in reasoning <cit.>, so we also compare our performance with it. Meanwhile, we also report the performance of on these datasets. [The results of GPT-3 and ChatGPT were obtained in March 2023 via the OpenAI API interface.]
It is worthy noting that our claim that Codex and ChatGPT perform zero-shot tasks is based on the assumption that they have not been previously exposed to these datasets.
However, to be honest, we have no way of knowing whether these datasets are already included in their training corpora. Last, we also report the performance of state-of-the-art (SOTA) fine-tuned models on these datasets for reference.
Finally, we reported the performance of TaPEx <cit.> to demonstrate that it does not have strong zero-shot generalization ability.
The full results can be found in Appendix appendix:1.
In the below, we use to refer to models trained based on using our proposed approach.

Evaluation Two evaluation metrics are used in the experiements. The first evaluation metric is denotation accuracy, which assesses whether the predicted answer corresponds to the ground-truth answer based on set-level equivalence. It is used in , and .
The second metric is accuracy, which is widely used in multi-classification problems and checks whether the predicted answer is correct. The accuracy metric is used in , SVAMP, BBH and MMLU.
Unless explicitly stated, the results presented in the following are obtained via zero-shot prompting on the evaluation task.
Considering that zero-shot models are sensitive to instructions, we follow the criterion in  <cit.> to report model performance.
Concretely, the model performance on the test set is reported based on the best validation instruction for the target task.
For and , we employ a rigorous approach by manually crafting $10$ distinct instructions for each task and selecting the best instruction based on validation results, ensuring the reliability of the results.
For GPT-3 and ChatGPT, we iteratively refine the instructions using a limited set of samples until the models are able to generate the anticipated outcomes.
The instructions for different methods can be found in  tab:prompt_for_models.

Implementation Details

We utilize Transformers <cit.> to implement our approach, and leverage DeepSpeed <cit.> to fine-tune models.
The multi-task fine-tuning is conducted using a batch size of $64$ with up to $20,000$ steps, requiring approximately $14$ hours using $8$ A100 GPU cards for .
With respect to the sampling strategy from the FLAN collection <cit.>, the normalized distribution of different mixtures CoT / Muffin / Natural Instructionsv2 / T0-SF is set as $0.1/0.3/0.3/0.3$.
As for the model optimization, we use the AdamW optimizer <cit.> and the peak learning rate is set to $3{\times}{10}^{-5}$.
The optimal checkpoint is selected based on the mixed validation set of and MMLU.
As for and , the most important hyperparameter is the temperature.
We set the hyperparameter temperature as $0$ since we found in practice that $0$ yields the best results. This might explain why the results of in our experiments are much better than those reported by <cit.>, as they used a default temperature of $0.7$.

§.§ Experimental Results

legend style=at=(0.5,1.35),
anchor=north,legend columns=-1,
symbolic x coords=0,1,2,5,10,
xticklabels=$0$ (Zero-shot), $1$, $2$, $5$, $10$,
bar width=.44cm,
enlarge x limits=0.2,
grid style=dashed,
every node near coord/.append style=font=,
nodes near coords,
nodes near coords style=anchor=south,font=,
ytick distance=5,
xlabel=The Number of Demonstrations,
ylabel=Performance (%),
xlabel near ticks
[fill=crimsonred!60!white,draw=crimsonred] coordinates
(0, 40.4)
(1, 51.4)
(2, 52.1)
(5, 51.9)
(10, 52.9)
[fill=royalblue!60!white,draw=royalblue] coordinates
(0, 40.4)
(1, 51.3)
(2, 53.4)
(5, 53.9)
(10, 54.4)
Synthetic Demonstration, Realistic Demonstartion

The performance of different instruction strategies with on . Note that our proposed synthetic demonstration does not leverage any realistic example.

We conduct extensive experiments to answer the following research questions:

RQ1. Does the method benefit table reasoning without using any realistic example? Yes.

tab:zeroshot_results shows the results of different methods on the table reasoning benchmark.
One can observe that GPT-3 and ChatGPT exhibit impressive zero-shot capabilities, such as attaining a denotation accuracy of $42.9$ on the most challenging task .
Similarly, ($780$M) and ($3$B) demonstrate exceptional table reasoning abilities, despite their considerably smaller model scales.
Compared to them, our proposed significantly boosts performance based on the models, surpassing all of baseline methods significantly.
Taking the task as an example, the performance of even exceeds $50$%, which is superior to the fine-tuned performance of TaPaS <cit.>, a well-known table pre-training LM.
However, it is worth noting that the performance of all these zero-shot models still falls short compared to the performance of fine-tuned models.

Our approach is not only effective in the multi-task fine-tuning use case, but also shows promise when applied to large language models.
fig:syn_demo compares the performance of our method with realistic demonstrations when paired with GPT-3.
As depicted in the figure, the most significant performance improvement for both synthetic demonstrations and realistic demonstrations comes from the transition from $0$ to $1$ demonsration.
When using only $1$ demonsration, our method achieves a similar performance as realistic demonstration.
As the number of demonstrations gradually increases, the model performance exhibits an upward trend.
Although our approach falls $1.5$% short of the realistic demonstration when there are $10$ demonstrations, it does not rely on any realistic examples, thereby reducing the need for demonstration tables.
Furthermore, the flexibility of our approach opens the possibility of synthesizing example-specfic demonstrations for different inputs.

RQ2. Does the method benefit tasks beyond table tasks? Yes.

In addition to table reasoning, our method can also be applied to generic tasks.
Inspired by POET <cit.>, we conducted experiments on the SVAMP dataset.
A typical question in the SVAMP dataset is as “Jack had 8 pens and Mary had 5 pens. Jack gave 3 pens to Mary. How many pens does Jack have now?”
It is apparent that the question is unrelated to table reasoning, but necessitates numerical reasoning.
tab:svamp compares the performance of and on the SVAMP dataset, illustrating a notable improvement of our models over .
The experimental results indicate that our approach can benefit a wider array of scenarios, not just table reasoning, thereby expanding its potential applications.

RQ3. Does the method hurt the model performance on generic tasks? No.

Recent studies have shown that models can be made to “specialize” in a particular domain via fine-tuning <cit.>.
However, there seems to be a trade-off between performance in the target domain and in other domains.
For instance, in  <cit.>, models trained on the GSM8K dataset achieved scores close to zero on the general tasks of BBH and MMLU.

To investigate whether the performance degration occurs in our approach, we report the performance of on the MMLU (tab:mmlu_tab) and BBH (tab:bbh_tab) test sets.
Notably, following , we incorporate few-shot demonstrations in the instruction and adopt the standard few-shot templates, as outlined in <cit.>.
As shown in the table, the performance of and on these two datasets is comparable.
For example, can achieve an accuracy of $40.9\%$ on BBH tasks.
Interestingly, our approach demonstrates a greater advantage in specific task categories, such as Humanities in the MMLU dataset and Algorithm tasks in the BBH dataset.
These experimental results demonsrate that our proposed symbolic tasks does not hurt the model performance on generic held-out tasks, implying that we can always improve LMs through symbolic tasks without sacrificing their generality.

§ CONCLUSION

In this paper, we presented a novel approach for instruction tuning that leverages synthetic tasks to improve the performance of language models on unseen tasks. Our proposed approach of utilizing symbolic tasks for instruction tuning has demonstrated its effectiveness in enhancing the performance of language models on unseen tasks, as exemplified by our case study on zero-shot table reasoning. The experimental results on several benchmarks show that our approach outperforms larger language models signifantly and achieves the best zero-shot performance across table reasoning benchmarks.
Meanwhile, the model maintains the original performance on generic datasets, which indicates that we can import symbolic tasks into instruction tuning without sacrificing the generality.
Compared to crowdsourced tasks, symbolic tasks are inherently more controllable, thus offering a number of potential opportunities. In the future, we hope to collaborate with the community to discover more valuable symbolic tasks, ultimately contributing to the development of mainstream instruction fine-tuning models.

§ EXPERIMENTAL RESULTS ON TABULAR REASONING