[a]Marco Garofalo [a]Simone Romiti

# Testing the $\mathrm{SU}(2)$ lattice Hamiltonian built from $S_{3}$
partitionings

Tobias Hartung  Timo Jakobs  Karl Jansen  Johann Ostmeyer  Dominik Rolfes
Carsten Urbach

###### Abstract

We test a possible digitization of SU$(2)$ lattice gauge theories based on
partitionings of the sphere $S_{3}$. In our construction the link operators
are unitary and diagonal, with eigenvalues determined by the vertices of the
partitioning. The canonical momenta are finite difference operators
approximating the Lie derivatives on the manifold. In this formalism we
implement the standard Wilson Hamiltonian. We show results for a 2-site
Schwinger-type model in 1D and a single-plaquette system in 2D. Our
calculations are performed on a classical computer, though in principle they
can be implemented also on a quantum device.

##  1 Introduction

The recent years have seen a revived interest in the Hamiltonian formulation
of lattice gauge theories [1, 2, 3, 4, 5]. This is due to the rapid
advancement of quantum computing resources. As an outstanding technological
achievement, this could open the window for unprecedented efficiency for
solving certain scientific problems. One of them is certainly the quantum
simulation of physical systems, namely the numerical solution of the time-
dependent Schrödinger equation. The goal is indeed to write down a truncated
version of the Hilbert space and of the degrees of freedom of the theory,
which recovers the exact version when the truncation is extrapolated to
infinity.

The Hamiltonian formalism has in general many potential advantages compared to
the classical Monte Carlo techniques based on the action [6]. For instance,
the time evolution of the physical states is performed in real time, hence not
requiring careful analytic continuations. There is no sign problem, one can
simulate at non-zero baryon densities, and study dynamical events such as the
scattering of bound states. It is important to remark, however, that nowadays
the available quantum devices still have a rather limited number of noisy
qubits. Therefore, many calculations are done on classical devices which are
in practice limited to very small and simple systems.

The formulation of Hamiltonian gauge theories on the lattice has a long
history [7], and can be treated in some cases with analytical techniques.
Indeed, the seminal paper by Kogut and Susskind [7] gave the formulation to
infer the properties of an SU$(2)$ lattice gauge theory in the confining
phase, in the approximation of strong coupling. The formalism has been
investigated and developed since then, also with the extension to generic
SU$(N)$ symmetry [8]. Eventually, the main goal for many particle physicists
is SU$(3)$, as it is the gauge group of QCD. The SU$(2)$ gauge theories
represents nevertheless a topic of interest, as they are cheaper to simulate
and show a confining phase similar to QCD [7, 9].

For the above reasons we have developed a digitization of an SU$(2)$ lattice
gauge theory in the Hamiltonian formalism, i.e. an algorithm for truncating
the Hilbert space to a given size and building the matrices that approximate
the degrees of freedom. In the literature there is more than one choice for
the basis of the Hilbert space, such as the electric, magnetic, loop-string-
hadron, etc. Each basis and each prescription is expected to give accurate
results in a given range of the Yang-Mills coupling constant. In this paper we
work in the magnetic basis, with unitary and diagonal links.

The Hilbert space is generated by the tensor product states for each lattice
site and direction. There we define the action of the operators $U$ (the link)
and the $L_{a},R_{a}$ (the canonical momenta) [10, 11, 12]. The space these
operators act on is isomorphic to a set of points on the SU$(2)$ group
manifold, which we chart according to the isomorphism with the sphere $S_{3}$.
We call this set of points a partitioning. The eigenvalues of $U$ are SU$(2)$
color matrices corresponding to the points in the partitioning, while the
canonical momenta are built as finite difference operators that approximate
the Lie derivatives on the continuous manifold. The fermionic fields are the
usual ladder operators in the fermionic part of the Fock space.

##  2 Theoretical background

Here we outline the main features of the Hilbert space and degrees of freedom
for a lattice $\mathrm{SU}(2)$ Hamiltonian. These properties are standard in
the literature. For a review, we defer to Refs. [13, 14, 8, 15, 12].

For a $\mathrm{SU}(2)$ lattice gauge theory the Hilbert space $\mathcal{H}$ is
generated by the states:

$\ket{\mathcal{U}}\otimes\ket{{\chi}}=\bigotimes_{\vec{x}}\left[\bigotimes_{\mu=1}^{d}\ket{\mathcal{U}_{\mu}(\vec{x})}\right]\,\otimes\ket{{\chi}(\vec{x})}\,.$
(1)

$\ket{\mathcal{U}}$ is the magnetic basis for the bosonic excitations, while
$\ket{{\chi}}$ contains information about the fermions. The
$\mathcal{U}_{\mu}(\vec{x})$ are color matrices in the fundamental
representation of $\mathrm{SU}(2)$, generated by the normalized Pauli matrices
$\tau_{a}=\frac{\sigma_{a}}{2}$. The ${\chi}(\vec{x})$ are color spinors with
entries given by the occupation number: either $0$ or $1$. The corresponding
operators $U_{\mu}(\vec{x})$ and $\chi(\vec{x})$ act as:

$\displaystyle{U_{\mu}(\vec{x})^{ab}\ket{\mathcal{U}}\vcentcolon=\mathcal{U}_{\mu}(\vec{x})^{ab}\ket{\mathcal{U}}}\,,$
(2)
$\displaystyle{\chi^{a}}^{\dagger}\ket{\chi^{a}}(\vec{x})=\ket{({\chi^{a}}(\vec{x})+1)\,\mathrm{mod}\,2}\,\forall
a\,.$ (3)

The fermionic operators obey the anti-commutation relations:

$\\{\chi^{a}(\vec{x}),{\chi^{b}}^{\dagger}(\vec{y})\\}=\delta_{ab}\delta_{\vec{x},\vec{y}}\,,\quad\\{\chi^{a}(\vec{x}),{\chi^{b}}(\vec{y})\\}=0\,.$
(4)

The $\mathrm{SU}(2)$ lattice Hamiltonian for a degenerate up-down quarks
doublet coupled to gluons is

$H=H_{\text{el}}+H_{\text{mag}}+H_{F}\,,$ (5)

where:

$\displaystyle H_{\text{el}}$
$\displaystyle=\frac{g^{2}}{2}\sum_{\vec{x}}\sum_{\mu=1}^{d-1}\sum_{a=1}^{3}{(L_{a})}_{\mu}^{2}(\vec{x})\,$
(6) $\displaystyle H_{\text{mag}}$
$\displaystyle=-\frac{1}{4g^{2}}\sum_{\vec{x}}\sum_{\mu=1,\nu<\mu}^{d-1}\,\mathrm{Tr}[{U}_{\mu\nu}(\vec{x})+{U}_{\mu\nu}^{\dagger}(\vec{x})]\,,$
(7) $\displaystyle H_{F}$
$\displaystyle=\mu\sum_{\vec{x}}(-1)^{\vec{x}}{\chi^{c}}^{\dagger}(\vec{x})\chi^{c}(\vec{x})+\sum_{\vec{x}}\sum_{\mu=1}^{d-1}{\chi^{c}}^{\dagger}(\vec{x})U_{\mu}(\vec{x})^{cc^{\prime}}\chi^{c^{\prime}}(\vec{x}+\mu)\,.$
(8)

$\vec{x}$ is a generic site of the lattice and $d$ is the number of
dimensions. $H_{\text{el}}$, $H_{\text{mag}}$, $H_{F}$ are the electric,
magnetic and fermionic part of the Hamiltonian respectively.
${U}_{\mu\nu}(\vec{x})$ is the plaquette operator and the trace is taken in
color space only. The Yang-Mills coupling $g$ is related to the usual coupling
$\beta$ by $\beta=4/g^{2}$.

The system is gauge invariant under gauge transformations generated by the
Left and Right canonical momenta $L_{a}$ and $R_{a}$. In analogy to
electrodynamics, these are often called electric fields (more detail in
Section. 3).

The gauge redundancy of the theory is removed by restricting to the subspace
of $\mathcal{H}$ of physical states, defined by Gauss’ law:

$G_{a}(\vec{x})\ket{\psi}_{\text{phys.}}=\left[\sum_{\mu=1}^{d}{(L_{a})}_{\mu}(\vec{x})+{(R_{a})}_{\mu}(\vec{x}-\mu)-Q_{a}(\vec{x})\right]\ket{\psi}_{\text{phys.}}=0\,\,,$
(9)

where $Q$ is the charge operator:

$Q_{a}=\frac{1}{2}\chi^{\dagger}(\vec{x})\,\tau^{a}\,\chi(\vec{x})\,.$ (10)

We conclude by recalling that despite the simplicity of the fermionic Fock
space, the gauge part has a rich structure already by itself. In fact, a basis
for physical states for the pure gauge theory is isomorphic to the irreps of
the $\mathrm{SU}(2)$ algebra. The latter are labelled by 3 half-integers
$j,m_{L},m_{R}$, with the constraint $|m_{L}|,|m_{R}|=-j,\ldots,j$. $j$ is the
main quantum number, while $m_{L}$ and $m_{R}$ are the left and right magnetic
quantum numbers respectively. When we truncate the Hilbert space, we cannot
preserve the unitarity of the links and the exact canonical commutation
relations simultaneously. In this work we keep the former, sampling the
eigenvalues of $U$ from a discrete set points on $S_{3}$ (see upcoming
sections for more details).

##  3 Construction of the momenta

The main difficulty in our prescription is the construction of the links and
momenta. We do this by using the isomorphism of the $\mathrm{SU}(2)$ group
manifold with the sphere $S_{3}$ in $4$ dimensions. The latter is partitioned
into a finite number of tetrahedra, with $N$ vertices labelled by three angles
$\vec{\alpha}$. The link $U$ is diagonal on the Hilbert space, with
eigenvalues given by $2\times 2$ color matrices $\mathcal{U}$:

$U\ket{\vec{\alpha}}=\mathcal{U(\vec{\alpha})}\ket{\vec{\alpha}}\,.$ (11)

The convention for the chart map $\vec{\alpha}$ defines the form of
$\mathcal{U(\vec{\alpha})}$ in terms of the generators $\tau_{a}$. The
canonical momenta are constructed as the finite difference operators
approximating the Lie derivatives along the directions of the generators:

$\displaystyle{L}_{a}f(U)$
$\displaystyle=-i\frac{\mathrm{d}}{\mathrm{d}\omega}f(e^{-i\omega\tau_{a}}U)|_{\omega=0}\,,$
(12) $\displaystyle{R}_{a}f(U)$
$\displaystyle=-i\frac{\mathrm{d}}{\mathrm{d}\omega}f(Ue^{i\omega\tau_{a}})|_{\omega=0}\,,$
(13)

where $f$ is an arbitrary $L^{2}$-integrable function over the
$\mathrm{SU}(2)$ Haar measure.

In [10, 11] this was done using a finite element method on a Delaunay
triangulation of $S_{3}$, and the derivatives are obtained as average over all
the simplices of the 1st order finite differences. The momenta are ultra-
local, namely the non-vanishing matrix elements are either on the diagonal or
couple nearest neighbors on $S_{3}$. This construction was implemented for
several types of partitionings of $S_{3}$ (linear, Fibonacci, etc.) becoming
dense in the limit of infinite number of points $N$. In that work we showed
numerical evidence for power law convergence in $N$ to the correct Lie algebra
structure and canonical commutation relations, by looking at the action of
these operators on test functions. We also showed the convergence of the
spectrum of the electric part of the Hamiltonian to the continuum manifold
one. For the latter we employ the operator $\sum_{a}L_{a}L_{a}$ found directly
as the approximation of the Laplace-Beltrami operator on $S_{3}$, while the
1st order finite differences $L_{a},R_{a}$ are used for the construction of
the Gauss’ law generators. This requires some care, as the latter contain
spurious effects coming from the discretization of the manifold. In the
following we will refer to this type of construction by the name of the
partitioning of $S_{3}$ employed. For details on the different constructions
see Ref. [11].

In Ref. [12], we have provided a construction that is exact on a subspace of
the Hilbert space, while the remaining subspace can be projected to arbitrary
energies above the cutoff. This property causes a loss of locality of the
matrices in the manifold space, while the operators are still local on the
space lattice. Thus, this construction is expected to work up to some maximum
value for the main quantum number $j$. In the following we will refer to it as
DJT, as the canonical momenta are constructed using what in Ref. [12] is
called Discrete Jacobi Transform.

##  4 Construction of the Hamiltonian

We generate the Hilbert space according to a truncated version of eq. 1.
Hence, it will be a finite dimensional vector space, with the basis given by
the tensor products of the configuration of the degrees of freedom.

At every point, the algebra of the degrees of freedom is the same. Therefore,
it is sufficient to define their action “pointwise”, so that the operators
will simply be tensor products of the identity at all points but for the one
we consider. More explicitly, if we have an operator $\mathcal{O}(x,\mu)$
then:

$\mathcal{O}(x,\mu)=\left[\bigotimes_{i<i_{x,\mu}}\mathds{1}\right]\otimes\mathcal{O}\otimes\left[\bigotimes_{i>i_{x,\mu}}\mathds{1}\right]\,,$
(14)

where $i_{x,\mu}$ is the checkerboard index corresponding to the point of the
space lattice. Furthermore, the fundamental degrees of freedom are
factorizable into the gauge (G) and fermionic (F) part, so that each
$\mathcal{O}$ is either ${\mathcal{O}_{G}\otimes\mathds{1}_{F}}$ or
${\mathds{1}_{G}\otimes\mathcal{O}_{F}}$.

In our case, the gauge operators are the link $U$ and the canonical momenta
${L_{a},R_{a}}$. The fermionic ones are the $\chi$s. In this work we
considered at most one light quark doublet, but the formalism is the identical
for multiple ones. We remark that at finite volume (i.e. finite number of
space lattice points), the fermionic Hilbert space is always finite.
Therefore, the field $\chi$ at every point $\vec{x}$ can be simply implemented
according to eq. 3.

The Hamiltonian is built according to eqs. 5 and 14, and Gauss’ law is
enforced by adding a penalty term:

$H_{\text{pen}}=\kappa\sum_{\vec{x},\mu}\sum_{a}(G_{a})_{\mu}(\vec{x})(G_{a})_{\mu}(\vec{x})\,,$
(15)

where $\kappa>0$. As $\kappa$ increases, so do the energies of the unphysical
states. According to the decoupling theorem, these modes decouple more and
more from the theory. The advantage of this approach is the simplicity of the
implementation, as it doesn’t require a maximal tree construction beforehand.
On the lattice $\kappa$ has to be chosen to be above the cutoff $\sim 1/a$
induced by lattice spacing $a$. In practice, find the optimal value of
$\kappa$ numerically: it has to be large enough such that the results are
independent of a further increase. In the case of the momenta obtained with
the finite element method, some extra care is required. In fact that
prescription possesses intrinsic spurious effects, coming from the
discretization of the manifold. Therefore, $\kappa$ can’t be too large in
order to not induce large residual discretization effects.

##  5 Numerical results

In this section we show our numerical results obtained with the constructions
described in section 4. These are obtained on a classical computer by an exact
diagonalization of the Hamiltonian.

###  5.1 Schwinger model in 1D

We consider the Hamiltonian for a 2-site lattice with open boundary
conditions. Written explicitly, the Hamiltonian reads:

$H=\mu\sum_{k=1}^{N_{\textrm{sites}}}\sum_{c=1}^{2}(-1)^{k}\,\chi^{c\dagger}_{k}\chi^{c}_{k}+\sum_{k=1}^{N_{\text{links}}}({L}^{a}_{k})^{2}+x\sum_{k=1}^{N_{\textrm{links}}}\sum_{c,c^{\prime}=1}^{2}\left(\chi^{c\,\dagger}_{k}{U}_{k}^{cc^{\prime}}\chi^{c^{\prime}}_{k+1}+\textrm{h.c.}\right)\,,$
(16)

where we have just rewritten eq. 5 for this particular system. The fermionic
$\chi$ operators obey the commutation relations of eq. 4.

$H$ describes the dynamics of a Schwinger-type model in one spatial dimension
(1D) with SU$(2)$ gauge symmetry, on a lattice connecting two sites. Since
there is no plaquette term in 1D, $H_{\text{mag}}=0$ and Gauss’ law can be
enforced exactly by integrating out the gauge fields analytically. The
resulting Hamiltonian reads [15]:

$\begin{split}{H}_{\textrm{integrated}}&=x\sum_{k}\left(\chi^{c\,\dagger}_{k}\chi^{c}_{k+1}+\textrm{h.c.}\right)+\mu\sum_{k=1}^{N_{\textrm{sites}}}\sum_{c=1}^{2}(-1)^{k}\,\chi^{c\dagger}_{k}\chi^{c}_{k}+\sum_{a=1}^{3}\sum_{k,k^{\prime}=1}^{N_{\textrm{sites}}}Q_{k}^{a}\,\frac{|k-k^{\prime}|}{2}\,Q_{k^{\prime}}^{a}\,,\end{split}$
(17)

where $Q_{a}$ is the charge operator defined in eq. 10.

We check the convergence of results obtained with Hamiltonian from eq. 16 to
the exact results obtained from ${H}_{\textrm{integrated}}$ numerically. In
fig. 1 we show in the left column the eigenvalue deviations
$(\lambda_{\mathrm{int}}-\lambda_{i})/\lambda_{\mathrm{int}}$ for $i=1$, $i=4$
and $i=5$ as a function of the number of points $N$ in the respective
partitionings for $\mu=1,x=1$ and $\kappa=10$. In the right column the
residual Gauss’ law violations for the respective eigenstates are plotted. We
compare DJT with the linear and the RFCC partitionings.

The eigenvalues $\lambda_{2}$ and $\lambda_{3}$ are omitted, as they
manifestly vanish for any number of points of the partitioning. In fact, the
canonical momenta are finite difference operators with eigenvalue zero on the
$\ket{j=0}$ state. Moreover, the state
$\ket{00}\otimes\ket{j=0}\otimes\ket{00}$ (or the same with
$\ket{00}\to\ket{11}$) is an eigenstate with eigenvalue zero for both
Hamiltonians $H$ and ${H}_{\textrm{integrated}}$. This is a feature of the
model and of the peculiar 2-site lattice configuration.

We observe convergence in $1/N$ for both, linear and RFCC partitionings, see
also the inlets in the upper row. For DJT, however, the eigenvalues agree with
the analytical results up to machine precision and the residual Gauss’ law
violations are also zero up to round off errors.

Figure 1:  Spectrum of the 1D Schwinger-type model with two lattice sites. We
show the (normalized) difference between the eigenvalues of the “standard”
lattice Hamiltonian of eq. 16 and the analytical results obtained from eq. 17
as a function of $1/N$ for $\lambda_{1,4,5}$ in the left column. In the right
column the residual Gauss’ law violations are show, again as a function of
$1/N$ for the respective three eigenstates. We compare Linear and RFCC
partitionings to the DJT construction.

We conclude that in this case the DJT is advantageous when compared to the
Linear and RFCC partitionings. It is likely a feature of this particular
system, as with only two sites Gauss’ law forbids heavier states.

###  5.2 Single plaquette system

Here we present our preliminary results for the single plaquette system. This
is a $4$-sites system in $2$-dimensions, whose Hamiltonian is:

$\hat{H}=\frac{g^{2}}{2}\sum_{c=1}^{3}\sum_{i=0}^{3}(\hat{L}^{c}_{i})^{2}-\frac{2}{g^{2}}\mathrm{Tr}\left[\hat{U}_{0}\hat{U}_{1}\hat{U}_{2}\hat{U}_{3}\right]\,.$
(18)

The latter is eq. 5, without fermions, for this particular setup. Gauss’ law
reads:

$\left(\hat{L}^{c}_{i}+\hat{R}^{c}_{(i+1)\,\,\mathrm{mod}\,\,4}\right)\ket{\psi}_{\text{phys.}}=0\,.$
(19)

In order to ease the numerical simulation, we enforce the spacetime symmetries
analytically, thus reducing the dimension of the relevant Hilbert space. We
write the gauge part of the state as
$\ket{\ell_{0},\ell_{1},\ell_{2},\ell_{3}}$, where $\ell_{i}$ is the
configuration of the $i$-th link. The system is symmetric under spatial
rotations of $\pi/2$ degrees, which form a discrete group $Z_{4}$ with
generator $\hat{r}$ acting as:

$\hat{r}\ket{\ell_{0},\ell_{1},\ell_{2},\ell_{3}}=\ket{\ell_{3},\ell_{0},\ell_{1},\ell_{2}}\,.$
(20)

The eigenstates of $\hat{r}$ are classified by a number $n$ as follows:

$\ket{\phi_{\hat{r}}}_{n}\sim\left(1+\lambda_{\hat{r}}^{n}\hat{r}+(\lambda_{\hat{r}}^{n})^{2}\hat{r}^{2}+(\lambda_{\hat{r}}^{n})^{3}\hat{r}^{3}\,\right)\ket{\ell_{0},\ell_{1},\ell_{2},\ell_{3}}\,,$
(21)

where $\lambda_{n}=e^{in\pi/2}$, $n=0,\ldots,3$. Note that the eigenstates
$\ket{\phi_{\hat{r}}}_{n}$ as defined above vanish for some combinations of
$(\ell_{0},\ell_{1},\ell_{2},\ell_{3})$. In particular states where
$\ell_{0}=\ell_{2}$ and $\ell_{1}=\ell_{3}$ will only give valid eigenstates
for $n=0,2$. For $\ell_{0}=\ell_{1}=\ell_{2}=\ell_{3}$ only $n=0$ gives a non-
vanishing eigenstate. With this, we find numerically that the ground state and
first excited state belong to in the $n=0$ subspace. Thus, it is sufficient to
only diagonalize this smaller block.

We compute the expectation value of the plaquette in the numerically found
vacuum state, together with the mass gap of the theory, for a range of
couplings $\beta$. In fig. 2 we compare our preliminary results to the ones
obtained through a variational ansatz [16]: the solid black lines represent
the variational results, and the points the results from exact diagonalization
of the digitized Hamiltonians. With the RFCC partitioning we observe that with
increasing number of points in the partitioning the results for $\langle
P\rangle$ approach the variational result for $1/\beta\lesssim 0.3$, see the
upper left panel. For the ground state energy this is even true for the full
range of $\beta$-values considered, see the upper right panel. For the gap
$E_{1}-E_{0}$, on the other hand, this approach is less visible, at least for
the number of points in the partitionings investigated here.

For the DJT construction, shown as (blue) crosses and (purple) boxes, we
observe smaller deviations from the variational result than with the
aforementioned RFCC partitionings for $1/\beta\lesssim 0.3$ for all three
quantities. Also $\langle H_{\mathrm{penalty}}\rangle/\kappa$ – a measure for
residual violations of Gauss’ law in the vacuum – is smaller for the DJT
construction compared with the RFCC partitionings. A the same time, the number
of points in the respective partitionings is smaller for DJT: for DJT with
$j\leq 1/2$ $N=9$ and for $j\leq 1$ $N=50$ are required per link, while the
two shown RFCC partitionings do require $N=32$ and $N=90$ points per link,
respectively.

Figure 2:  Expectation value of the plaquette $\langle P\rangle$, of the
vacuum energy $E_{0}$, the gap $E_{1}-E_{0}$, and $\langle
H_{\mathrm{penalty}}\rangle/\kappa$ as a function of $1/\beta=g^{2}/4$. We
compare variational results (solid lines) to results obtained with two
particular RFCC partitionings with $N=32$ and $N=90$ and with two DJT
constructions for $j\leq 1/2$ and $j\leq 1$ with $N=9$ and $N=50$,
respectively.

##  6 Conclusion and outlook

In this proceeding contribution, we have presented preliminary results for
different digitizations of a $\mathrm{SU}(2)$ lattice gauge theory in the
magnetic basis. We compare the constructions presented in Refs. [11, 12],
based on the isomorphism between $\mathrm{SU}(2)$ and the sphere $S_{3}$,
which is sampled with a finite number of points $N$ defining the truncated
basis of the Hilbert space. The gauge link operator is diagonal and unitary,
while the canonical momenta are finite difference matrices, converging to the
continuum manifold Lie derivatives at infinite truncation.

Compared to the construction based on partitionings from Ref. [11], the one
from Ref. [12] based on a discrete Jacobi transform (DJT) has the property
that gauge symmetry is exactly preserved on a subspace of the truncated
Hilbert space for the price of dense matrix representations of the canonical
momenta. Here we have tested and compared the different constructions for two
models, a Schwinger-type model with two sites and SU$(2)$ gauge symmetry, as
well as a SU$(2)$ gauge theory consisting of a single plaquette. For both
models exact results are available to compare with.

With the partitionings from Ref. [11] we observe deviations from the exact
results, which go to zero with increasing number of points $N$ in the
partitionings. With the DJT [12] on the other hand we find exact results up to
machine precision for the 2-site Schwinger-type model as a consequence of the
enhanced gauge symmetry. While not exact anymore, DJT shows also for the
single plaquette gauge theory smaller deviations from the expected result with
smaller number of points in the partitioning, at least for a certain range of
$\beta$-values when compared to the RFCC partitionings.

In the near future, we plan a systematical study of the single plaquette gauge
theory, both in parameter space of the theory as well as the number of points
of the partitioning.

## Acknowledgements

We would like to thank A. Crippa for helpful discussions. This work is
supported by the Deutsche Forschungsgemeinschaft (DFG, German Research
Foundation) and the NSFC through the funds provided to the Sino-German
Collaborative Research Center CRC 110 “Symmetries and the Emergence of
Structure in QCD” (DFG Project-ID 196253076 - TRR 110, NSFC Grant No.
12070131001) as well as by the STFC Consolidated Grant ST/T000988/1. This work
is supported with funds from the Ministry of Science, Research and Culture of
the State of Brandenburg within the Centre for Quantum Technologies and
Applications (CQTA). This work is funded by the European Union’s Horizon
Europe Framework Programme (HORIZON) under the ERA Chair scheme with grant
agreement No. 101087126.

## References

  * [1] A.D. Meglio, K. Jansen, I. Tavernelli, C. Alexandrou, S. Arunachalam, C.W. Bauer et al., _Quantum computing for high-energy physics: State of the art and challenges. summary of the qc4hep working group_ , 2023.  10.48550/arXiv.2307.03236.
  * [2] I. Raychowdhury and J.R. Stryker, _Loop, string, and hadron dynamics in su(2) hamiltonian lattice gauge theories_ , _Phys. Rev. D_ 101 (2020) 114502.
  * [3] Z. Davoudi, A.F. Shaw and J.R. Stryker, _General quantum algorithms for hamiltonian simulation with applications to a non-abelian lattice gauge theory_ , _arXiv preprint arXiv:2212.14030_ (2022) .
  * [4] T.V. Zache, D. González-Cuadra and P. Zoller, _Quantum and Classical Spin-Network Algorithms for q-Deformed Kogut-Susskind Gauge Theories_ , _Phys. Rev. Lett._ 131 (2023) 171902 [2304.02527].
  * [5] C.W. Bauer, I. D’Andrea, M. Freytsis and D.M. Grabowska, _A new basis for Hamiltonian SU(2) simulations_ , 2307.11829.
  * [6] C. Gattringer and C. Lang, _Quantum Chromodynamics on the Lattice - an Introductory Presentation_ , vol. 788, Springer Science & Business Media (2009), 10.1007/978-3-642-01850-3.
  * [7] J. Kogut and L. Susskind, _Hamiltonian formulation of Wilson’s lattice gauge theories_ , _Phys. Rev. D_ 11 (1975) 395.
  * [8] E. Zohar and M. Burrello, _Formulation of lattice gauge theories for quantum simulations_ , _Phys. Rev. D_ 91 (2015) 054506.
  * [9] R. Sommer, _A new way to set the energy scale in lattice gauge theories and its application to the static force and $\alpha_{s}$ in $\mathrm{SU}(2)$ Yang-Mills theory_, _Nucl. Phys. B_ 411 (1994) .
  * [10] M. Garofalo, T. Hartung, K. Jansen, J. Ostmeyer, S. Romiti and C. Urbach, _Defining Canonical Momenta for Discretised SU $(2)$ Gauge Fields_, _PoS_ LATTICE2022 (2023) 040 [2210.15547].
  * [11] T. Jakobs, M. Garofalo, T. Hartung, K. Jansen, J. Ostmeyer, D. Rolfes et al., _Canonical momenta in digitized SU(2) lattice gauge theory: definition and free theory_ , _Eur. Phys. J. C_ 83 (2023) 669 [2304.02322].
  * [12] S. Romiti and C. Urbach, _Digitizing lattice gauge theories in the magnetic basis: reducing the breaking of the fundamental commutation relations_ , 2311.11928.
  * [13] M. Creutz, _Gauge fixing, the transfer matrix, and confinement on a lattice_ , _Phys. Rev. D_ 15 (1977) 1128.
  * [14] T. Banks, S. Raby, L. Susskind, J. Kogut, D.R.T. Jones, P.N. Scharbach et al., _Strong-coupling calculations of the hadron spectrum of quantum chromodynamics_ , _Phys. Rev. D_ 15 (1977) 1111.
  * [15] M.C. Bañuls, K. Cichy, J.I. Cirac, K. Jansen and S. Kühn, _Efficient basis formulation for ( $1+1$)-dimensional su(2) lattice gauge theory: Spectral calculations with matrix product states_, _Phys. Rev. X_ 7 (2017) 041046.
  * [16] M.C. Huang, R.L. Coldwell and M.W. Katoot, _A demonstration of the speed and accuracy of the biased-selection monte carlo methods in hamiltonian su(2) lattice gauge theory_ , _Nuclear Physics B_ 309 (1988) 733.


August 28, 2024

# A New Compound Poisson Process and its fractional Versions

Palaniappan Vellaisamy and Tomoyuki Ichiba  Department of Statistics and
Applied Probability, UC Santa Barbara, Santa Barbara, CA, 93106, USA.
Department of Statistics and Applied Probability, UC Santa Barbara, Santa
Barbara, CA, 93106, USA<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract.

We consider a weighted sum of a series of independent Poisson random variables
and show that it results in a new compound Poisson distribution which includes
the Poisson distribution and Poisson distribution of order $k.$ An explicit
representation for its distribution is obtained in terms of Bell polynomials.
We then extend it to a compound Poisson process and time fractional compound
Poisson process ($TFCPP$). It is shown that the one-dimensional distributions
of the $TFCPP$ exhibit over-dispersion property, are not infinitely divisible
and possess the long-range dependence property. Also, their moments and
factorial moments are derived. Finally, the fractional differential equation
associated with the $TFCPP$ is also obtained.

###### Key words and phrases:

Adomian polynomials, Poisson distribution of order $n$, probabilistic method,
recurrence relations

###### 2010 Mathematics Subject Classification:

Primary: 65L99; Secondary: 93E25, 60E05.

##  1\. Introduction

In recent decades, the classical Poisson process, negative binomial process
and gamma process have been generalized to various forms of their fractional
versions such as fractional Poisson process ($FPP$), fractional negative
binomial processes ($FNBP$) and fractional gamma process ($FGP$); see, for
example, Laskin [Lask03], Begin and Orsingher [BegOrs09], Begin [Beg13],
Mainardi et. al [MGS04], Begin and Macci [BegMac14], Meerchaert et al.
[MNV09], Meerchaert et al. [MNV11], Vellaisamy and Maheswari [PVAM18] and
Kataria and Khandakar [KKMK22]. These time and space fractional versos have
heavy-tailed distributions, non-exponential waiting times and long-range
dependence properties; see Biard and Saussereau [BDSS14], Kataria and
Vellaisamy [KV19] and Maheshwari and Vellaisamy [MV16]. These characteristics
make the these processes more suitable, than the classical Levy processes (see
Applebaum [AB09]) for modeling various phenomena that arise in many
disciplines such as finance, hydrology, atmospheric science, etc. (see Laskin
[Lask09]). The time fractional Poisson process which initially derived from
certain fractional differential equations (see Laskin [Lask03], Begin and
Orshinger [BegOrs09]) can also be viewed as a Poisson process subordinated to
inverse stable subordinator (see Meerchaert et al. [MNV11]). This approach
initiated the study of various subordinated processes leading to time-
fractional and space fractional versions; some references, among others, are
Orshinger and Polito [OrsPoli12], Maheswari and Vellaisamy [MV19], Leonenko
et. al [LMS14] and Begin and Vellaisamy [BegVel18].

In this paper, we look at the sum of a series of independent weighted Poisson
random variables (rvs) which leads to a new compound Poisson distribution
($CPD$). By suitably choosing the associated sequence of parameters, we show
that the $CPD$ includes the Poisson distribution, Poisson distribution of
order $k$ and infinity and so its study leads a unified approach. Replacing
Poisson rvs by Poisson processes, we extend it to a new compound Poisson
process ($CPP$). Later, as a natural extension, we consider its time
fractional version and call it $TFCPP$.

In Section 2, we introduce the notations and required preliminary results and
the $CPD$ is introduced and studied in Section 3. In Section 4, the associated
$CPP$ is studied and its time fractional version $TFCPP$ is investigated in
Section 5. In particular, the one-dimensional distributions of the $TFCPP$
have the over-dispersion property, are not infinitely divisible, and possess
the long-range dependence property. The moments and the factorial moments of
the $TFCPP$ are derived. Finally, its factional differential equation
satisfied by the $TFCPP$ is also derived.

##  2\. Preliminaries

In this section, we introduce the notations and the results that will be used
later. We start with some special functions that will be required later.

###  2.1. Some special functions

######  Definition 2.1.

(i): The one parameter Mittag-Leffler function $M_{\beta}(z)$ is defined as
(see [erde3])

$M_{\alpha}(z)=\sum\limits_{k=0}^{\infty}\frac{z^{k}}{\Gamma(\alpha
k+1)},\,\,\,\alpha,z\in{\mathbb{C}}\text{ and Re}(\alpha)>0.$ (2.1)

(ii): For $z\in\mathbb{C}$, the two parameter Mittag-Leffler function is
defined as

$M_{\alpha,\beta}(z)=\sum_{k=0}^{\infty}\frac{z^{k}}{\Gamma(\alpha
k+\beta)},\leavevmode\nobreak\ \leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \text{$\alpha>0,\beta>0.$}$ (2.2)

When $\beta=1,M_{\alpha,1}$ reduces to the one parameter Mittag-Leffler
function $M_{\alpha}(z).$
(iii): The generalized Mittag-Leffler function (Prabhakar [PB71]) is defined
as

$M_{\alpha,\beta}^{\gamma}(z)=\frac{1}{\Gamma(\gamma)}\sum_{k=0}^{\infty}\frac{\Gamma(\gamma+k)}{k!\Gamma(\alpha
k+\beta)}{z^{k}},\leavevmode\nobreak\ \leavevmode\nobreak\
\leavevmode\nobreak\ \text{$\alpha,\beta,\gamma,z\in\mathbb{C}$}$ (2.3)

where $Re(\alpha),Re(\beta),Re(\gamma)>0$. Note that
$M_{\alpha,\beta}^{\gamma}(0)=1/\Gamma(\beta).$

Let $M_{\alpha,\beta}^{(n)}(x)$ denote the $n$-th derivative of the two
parameter Mittag-Leffler function. Then (see Kilbas et al. [KST06], Eq
(1.9.5))

$M_{\alpha,\beta}^{(n)}(x)=n!M_{\alpha,n\alpha+\beta}^{n+1}(x),\quad n\geq 0.$
(2.4)

It is well known that

$\alpha M_{\alpha}^{(n)}(-\lambda
t^{\alpha})=M_{\alpha,\alpha}^{(n-1)}(-\lambda t^{\alpha}),\quad n\geq 1.$
(2.5)

The Mittag-Leffler $ML(\alpha,\lambda)$ distribution is introduced and studied
by Pillai [Pil90]. Its distribution function is given by

$F(t\mid\alpha,\lambda)=1-M_{\alpha}(-\lambda
t^{\alpha})=\sum_{k=0}^{\infty}(-1)^{k+1}\frac{(\lambda
t^{\alpha})^{k}}{\Gamma(\alpha k+1)},\leavevmode\nobreak\ t>0,$ (2.6)

where $0<\alpha\leq 1$ and $\lambda>0$ are the parameters.

The density of $ML(\alpha,\lambda)$ is given by

$\displaystyle f(t\mid\alpha,\lambda)=$ $\displaystyle\alpha\lambda
t^{\alpha-1}M_{\alpha}^{{}^{\prime}}(-\lambda t^{\alpha})$ $\displaystyle=$
$\displaystyle\lambda t^{\alpha-1}M_{\alpha,\alpha}(-\lambda t^{\alpha}),$
(2.7)

using (2.5).

Let $X\sim ML(\alpha,\lambda)$. The its Laplace transform is

$E(e^{-sX})=\frac{\lambda}{\lambda+s^{\alpha}}.$

If $T=X_{1}+\cdots+X_{n}$, where $X_{i}$’s are IID $ML(\alpha,\lambda)$, then
the density of $T$ (see Kataria and Vellaisamy [KV19a]) is

$\displaystyle f(t\mid\alpha,\lambda)=$
$\displaystyle\frac{\lambda^{n}}{(n-1)!}t^{\alpha
n-1}M_{\alpha,\alpha}^{n-1}(-\lambda t^{\alpha})$ $\displaystyle=$
$\displaystyle\frac{\alpha\lambda^{n}}{(n-1)!}t^{\alpha
n-1}M_{\alpha}^{(n)}(-\lambda t^{\alpha}).$ (2.8)

######  Definition 2.1.

(i): The Wright function $W_{\alpha,\leavevmode\nobreak\ \beta}(z)$ is defined
, for $\alpha>-1$ and $\beta\in\mathbb{C}$, as

$\displaystyle
W_{\alpha,\beta}(z)={\sum_{n=0}^{\infty}\frac{(z)^{n}}{n!\Gamma[\alpha
n+\beta]}}$ (2.9)

which converges in the whole complex plane.

(ii): A particular case of the Wright function, called the $M$-Wright function
$W_{\beta}(z)$ is defined as

$\displaystyle W_{\beta}(z)=$ $\displaystyle
W_{-\beta,1-\beta}(-z)={\sum_{n=0}^{\infty}\frac{(-z)^{n}}{n!\Gamma[-\beta
n+(1-\beta)]}}$ (2.10) $\displaystyle=$
$\displaystyle{\frac{1}{\pi}\sum_{n=1}^{\infty}\frac{(-z)^{n-1}}{(n-1)!}\Gamma(\beta
n)\sin(\pi\beta n)},$

which converges for $z\in\mathbb{C}$ and $0<\beta<1$.

The following results are well known (see Kilbas e͡t. al [KST06], Kataria and
Vellaismy [KV18]).

######  Definition 2.2.

(i): Let $D_{\beta}(t)$ be the $\beta$-stable subordinator. Then the density
of $D_{\beta}(t)$ is

$g_{{}_{\beta}}(x;t)=\beta
tx^{-(\beta+1)}W_{\beta}(tx^{-\beta}),\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ x>0.$ (2.11)

(ii): Let $E_{\beta}(t)$ be the inverse $\beta$-stable subordinator. Then the
density of $E_{\beta}(t)$ is

$h_{\beta}(x;t)=t^{-\beta}W_{\beta}(t^{-\beta}x),\leavevmode\nobreak\
\leavevmode\nobreak\ \leavevmode\nobreak\ x>0.$ (2.12)

###  2.2. Some fractional derivatives

Let $AC[a,b]$ be the space of functions $f$ which are absolutely continuous on
$[a,b]$ and

$AC^{n}[a,b]=\left\\{f:[a,b]\rightarrow\mathbb{R};f^{(n-1)}(t)\in
AC[a,b]\right\\},$

where $AC^{1}[a,b]=AC[a,b]$.

Henceforth, ${\mathbb{Z}}_{+}=\\{0,1,\ldots\\}$ and
${\mathbb{N}}=\\{1,2,\ldots\\}$ the set of nonnegative integers and the set of
positive integers respectively.

######  Definition 2.3.

Let $n\in{\mathbb{N}}$, $\beta\geq 0$ and $f(t)\in AC^{n}[0,T]$. Then
(i): The (left-hand) Riemann-Liouville (R-L) fractional derivative
$\mathbb{D}_{t}^{\beta}f$ of $f$ (see [KST06, Lemma 2.2]) is defined by (with
$\mathbb{D}^{0}f=f$)

$\mathbb{D}_{t}^{\beta}f(t):=\begin{cases}\hfill\dfrac{1}{\Gamma(n-\beta)}\dfrac{d^{n}}{dt^{n}}\displaystyle\int_{0}^{t}\dfrac{f(s)}{(t-s)^{\beta-n+1}}ds,\hfill&\text{if}\leavevmode\nobreak\
\leavevmode\nobreak\ n-1<\beta<n,\\\ &\\\
f^{(n)}(t),\quad\text{if}\leavevmode\nobreak\ \leavevmode\nobreak\
\beta=n.&\end{cases}$ (2.13)

(ii): The (left-hand) Caputo fractional derivative $\partial^{\beta}f$ of $f$
(see Kilbas et. al [KST06, Theorem 2.1]) is defined by (with
$\partial^{0}f=f$)

$\partial_{t}^{\beta}f(t):=\begin{cases}\hfill\dfrac{1}{\Gamma(n-\beta)}\displaystyle\int_{0}^{t}\dfrac{f^{(n)}(s)}{(t-s)^{\beta-n+1}}ds,\hfill&\text{if}\leavevmode\nobreak\
\leavevmode\nobreak\ n-1<\beta<n\\\ &\\\
f^{(n)}(t),\quad\text{if}\leavevmode\nobreak\ \leavevmode\nobreak\
\beta=n.&\end{cases}$ (2.14)

The relation between the R-L fractional derivative and the Caputo fractional
derivative is (see Kilbas et. al [KST06, eq. (2.4.6)])

$\mathbb{D}_{t}^{\beta}f(t)=\partial_{t}^{\beta}f(t)+\displaystyle\sum\limits_{k=0}^{n-1}\frac{t^{k-\beta}}{\Gamma(k-\beta+1)}f^{(k)}(0^{+}),$

where $f^{(k)}(0^{+}):=\lim_{t\rightarrow 0^{+}}\frac{d^{k}}{dt^{k}}f(t)$.

###  2.3. Poisson Distribution of Order $k$

Let $\\{N_{j}\\},j\geq 1,$ where $N_{j}\sim$ $Poi(\lambda_{j})$, be a sequence
of independent Poisson random variables with probability distribution

$\mathbb{P}\\{N_{j}=n\\}=\frac{e^{-\lambda_{j}}\lambda_{j}^{n}}{n!},\leavevmode\nobreak\
n\in\mathbb{Z_{+}},\leavevmode\nobreak\ j\in{\mathbb{N}}.$

It is well known that the Poisson family is stable under convolution, that is,

$S_{k}=\sum_{j=1}^{k}N_{j}\sim\text{Poi}\leavevmode\nobreak\
(\lambda_{1}+\cdots+\lambda_{k}),\quad k\geq 1,$

the Poisson distribution with mean $\lambda_{1}+\cdots+\lambda_{k}$.

For $k\geq 1$, consider now the random variable
$W_{k}=N_{1}+2N_{2}+\cdots+kN_{k}$, a weighted sum of Poisson rvs. Its
distribution is given by, for $n\in\mathbb{Z_{+}}$,

$\displaystyle\mathbb{P}\\{W_{k}=n\\}$
$\displaystyle=\sum_{\left\\{\sum\limits_{j=1}^{k}jx_{j}=n\right\\}}\mathbb{P}\\{N_{1}=x_{1},\cdots,N_{k}=x_{k}\\}$
$\displaystyle=\sum_{\left\\{\sum\limits_{j=1}^{k}jx_{j}=n\right\\}}\prod_{j=1}^{k}\mathbb{P}\\{N_{j}=x_{j}\\}$
$\displaystyle=e^{-(\lambda_{1}+\cdots+\lambda_{k})}\sum_{\left\\{\sum\limits_{j=1}^{k}jx_{j}=n\right\\}}\left(\frac{\lambda_{1}^{x_{1}}\cdots\lambda_{k}^{x_{k}}}{x_{1}!\ldots
x_{k}!}\right),$ (2.15)

where we have used the independence of the $X_{i}$’s in the second line above.
The probability distribution given in (2.15) is called the Poisson
distribution of order $k$ (see Philippou et. al [Philip88]) and is denoted by
Poi ($u_{1},\ldots,u_{k})$. Thus, it follows that

$W_{k}=\sum_{j=1}^{k}jN_{j}\sim\text{Poi}\leavevmode\nobreak\
(u_{1},\ldots,u_{k}),\quad k\geq 1.$

###  2.4. Bell Polynomials

The following definitions and results on Bell polynomials will be required
later.

####  2.4.1. Ordinary Bell polynomials.

Let $c_{j}$’s denote the nonnegative integers. For $1\leq k\leq n$, define

$\displaystyle\Lambda_{k,n}=\Big{\\{}(c_{1},c_{2},\ldots,c_{n}):c_{1}+\cdots+c_{n}=k;\leavevmode\nobreak\
c_{1}+2c_{2}+\cdots+nc_{n}=n\Big{\\}}$ (2.16)

and

$\displaystyle\Lambda_{k,n}^{*}=\Big{\\{}(c_{1},c_{2},\ldots,c_{n-k+1}):c_{1}+\cdots+c_{n-k+1}=k;\leavevmode\nobreak\
c_{1}+2c_{2}+\cdots+(n-k+1)c_{n-k+1}=n\Big{\\}}.$ (2.17)

Also, for $n\geq 1$, let

$\displaystyle\Lambda_{n}=\Big{\\{}(c_{1},c_{2},\ldots,c_{n}):c_{1}+2c_{2}+\cdots+nc_{n}=n\Big{\\}}.$
(2.18)

The ordinary partial Bell polynomials are defined by

$\displaystyle\hat{B}_{n,k}(u_{1},\dots,u_{n-k+1})=\sum_{\Lambda_{k,n}^{*}}{\frac{k!}{c_{1}!\cdots
c_{n-k+1}!}}{{u_{1}}}^{c_{1}}\cdots{{u_{n-k+1}}}^{c_{n-k+1}}$ (2.19)

or equivalently

$\displaystyle\hat{B}_{n,k}(u_{1},\dots,u_{n})=\sum_{\Lambda_{k,n}}{\frac{k!}{c_{1}!\dots
c_{n}!}}{{u_{1}}}^{c_{1}}\dots{u_{n}}^{c_{n}},$ (2.20)

since, for each fixed $k$, there can be no nonzero $c_{j}$’s, for
$j\geq(n-k+1)$ and so $c_{n-k+2}=\cdots=c_{n}=0$.

The ordinary Bell polynomials are defined by

$\displaystyle\hat{B}_{n}(u_{1},\dots,u_{n})=\sum_{k=1}^{n}\hat{B}_{n,k}(u_{1},\dots,u_{n}),\quad
n\geq 1.$ (2.21)

The following results are well known (see Comtet [CT74], pp. 133-137):

$\displaystyle\exp\left(x\sum_{j=1}^{\infty}\frac{u_{j}t^{j}}{j!}\right)=$
$\displaystyle
1+\sum_{n=1}^{\infty}\frac{t^{n}}{n!}\left\\{\sum_{k=1}^{n}\hat{B}_{n,k}\left(u_{1},u_{2},\ldots,u_{n-k+1}\right)x^{k}\right\\}$

and also for the series expansion of the $k$-th power, for $k\geq 1,$

$\left(\sum_{j=1}^{\infty}u_{j}t^{j}\right)^{k}=\sum_{n=k}^{\infty}\hat{B}_{n,k}\left(u_{1},u_{2},\ldots,u_{n-k+1}\right)t^{n}.$
(2.22)

####  2.4.2. Partial exponential Bell polynomials.

The partial or incomplete exponential Bell polynomials are a triangular array
of polynomials given by

$\displaystyle B_{n,k}\left(u_{1},u_{2},\ldots,u_{n-k+1}\right)=$
$\displaystyle\sum_{\Lambda_{k,n}^{*}}\frac{n!}{c_{1}!c_{2}!\cdots
c_{n-k+1}!}\left(\frac{u_{1}}{1!}\right)^{c_{1}}\left(\frac{u_{2}}{2!}\right)^{c_{2}}\cdots\left(\frac{u_{n-k+1}}{(n-k+1)!}\right)^{c_{n-k+1}}$

or compactly

$\displaystyle
B_{n,k}\left(u_{1},u_{2},\ldots,u_{n}\right)=\sum_{\Lambda_{k,n}}\binom{n}{c_{1},c_{2},\ldots,c_{n}}\left(\frac{u_{1}}{1!}\right)^{c_{1}}\left(\frac{u_{2}}{2!}\right)^{c_{2}}\cdots\left(\frac{u_{n}}{n!}\right)^{c_{n}},$

where $B_{0,0}=1$, $B_{0,k}=0\leavevmode\nobreak\ ,k\geq 1$. Here
$\binom{n}{c_{1},c_{2},\ldots,c_{n}}$ denotes the multinomial coefficient.

Let $\Lambda_{n}$ be defined as in (2.18). Then the sum

$\displaystyle B_{n}\left(u_{1},\ldots,u_{n}\right)=$
$\displaystyle\sum_{k=1}^{n}B_{n,k}\left(u_{1},u_{2},\ldots,u_{n}\right)$
$\displaystyle=$
$\displaystyle\sum_{\Lambda_{n}}\binom{n}{c_{1},c_{2},\ldots,c_{n}}\left(\frac{u_{1}}{1!}\right)^{c_{1}}\left(\frac{u_{2}}{2!}\right)^{c_{2}}\cdots\left(\frac{u_{n}}{n!}\right)^{c_{n}},$
(2.23)

with $B_{0}=1$, is called $n$-th complete exponential Bell polynomial.
Henceforth, Bell polynomials always refer to exponential Bell polynomials
unless stated otherwise.

The relation between ordinary Bell polynomials and exponential Bell
polynomials is

$\hat{B}_{n,k}\left(u_{1},u_{2},\ldots,u_{n}\right)=\frac{k!}{n!}B_{n,k}\left(1!u_{1},2!u_{2},\ldots,n!u_{n}\right).$
(2.24)

The following result from Johnson [John02, p. 220] is useful. Let $|A|$ denote
the determinant of the matrix $A.$

######  Lemma 2.1.

If $n\geq 1$, then

$B_{n}\left(u_{1},u_{2},\ldots,u_{n}\right)=\begin{vmatrix}u_{1}&\binom{n-1}{1}u_{2}&\binom{n-1}{2}u_{3}&\cdots&\binom{n-1}{n-2}u_{n-1}&u_{n}\\\
&&&&&\\\
-1&u_{1}&\binom{n-2}{1}u_{2}&\cdots&\binom{n-2}{n-3}u_{n-2}&u_{n-1}\\\
&&&&&\\\ 0&-1&u_{1}&\cdots&\binom{n-3}{n-4}u_{n-3}&u_{n-2}\\\
\vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\\\ 0&0&0&\cdots&u_{1}&u_{2}\\\
0&0&0&\cdots&-1&u_{1}\\\ \end{vmatrix}.$

Note in the above matrix, all the entries on the main subdiagonal are $-1$,
and all entries below it are 0 .

######  Corollary 2.1.

Let $u_{j}=0$ for $j\geq 2$. Then by using Lemma 2.1

$\displaystyle B_{n}(u_{1},0,0,\ldots,0)=$
$\displaystyle\begin{vmatrix}u_{1}&0&0&\cdots&0&0\\\ -1&u_{1}&0&\cdots&0&0\\\
0&-1&u_{1}&\cdots&0&0\\\ \vdots&\vdots&\vdots&\vdots&\vdots&\vdots&\\\
0&0&0&\cdots&u_{1}&0\\\ 0&0&0&\cdots&-1&u_{1}\end{vmatrix}$ $\displaystyle=$
$\displaystyle u_{1}^{n}.$

####  2.4.3.  A Probabilistic Formula.

In this subsection, we present a probabilistic approach to compute the Bell
polynomials. Let $X_{j}\sim Poi(\lambda_{j})$ be a sequence of independent
Poisson rvs and $u_{j}=j!\lambda_{j},j\geq 1$. As before, let
$S_{n}=X_{1}+\cdots+X_{n}$ and $W_{n}=X_{1}+2X_{2}+\cdots+nX_{n}$. Then by
definition

$\displaystyle B_{n,k}(u_{1},\dots,u_{n})$
$\displaystyle=\sum_{\Lambda_{k,n}}\binom{n}{c_{1},\ldots,c_{n}}\left({\frac{u_{1}}{1!}}\right)^{c_{1}}\dots\left({\frac{u_{n}}{n!}}\right)^{c_{n}}$
$\displaystyle=n!\sum_{\Lambda_{k,n}}\dfrac{\lambda_{1}^{c_{1}}}{c_{1}!}\cdots\dfrac{\lambda_{n}^{c_{n}}}{c_{n}!}$
$\displaystyle=n!e^{\lambda_{1}+\cdots+\lambda_{n}}\sum_{\Lambda_{k,n}}\dfrac{e^{-\lambda_{1}}\lambda_{1}^{c_{1}}}{c_{1}!}\cdots\dfrac{e^{-\lambda_{n}}\lambda_{n}^{c_{n}}}{c_{n}!}$
$\displaystyle=\dfrac{n!P\\{S_{n}=k;W_{n}=n\\}}{P\\{S_{n}=0\\}}.$ (2.25)

Using (2.4.2), the complete Bell polynomial is for $n\geq 1$,

$\displaystyle B_{n}(u_{1},\ldots,u_{n})=$
$\displaystyle\sum_{k=1}^{n}B_{n,k}(u_{1},\dots,u_{n})$ $\displaystyle=$
$\displaystyle n!\sum_{k=1}^{n}\dfrac{P\\{S_{n}=k;W_{n}=n\\}}{P\\{S_{n}=0\\}}$
$\displaystyle=$
$\displaystyle\frac{n!}{P\\{S_{n}=0\\}}\sum_{k=0}^{n}P\\{S_{n}=k;W_{n}=n\\}$
$\displaystyle=$ $\displaystyle\frac{n!P\\{W_{n}=n\\}}{P\\{S_{n}=0\\}},$
(2.26)

since $P(S_{n}=0;W_{n}=n)=0$ for $n\geq 1.$

In other words,

$\displaystyle
B_{n}(1!\lambda_{1},2!\lambda_{2},\ldots,n!\lambda_{n})=\frac{n!P\\{W_{n}=n\\}}{P\\{S_{n}=0\\}},$
(2.27)

where $X_{j}\sim Poi(\lambda_{j})$.

For some additional details on the probabilistic connections to the Bell
polynomials, see Kataria and Vellaisamy [KVV22].

##  3\. A Compound Poisson Distribution

Let $\\{N_{j}\\}$ be a sequence of independent Poisson variates, where
$\\{N_{j}\\}\sim Poi(\lambda_{j}),j\geq 1.$ Henceforth, we define $N_{j}\equiv
0\leavevmode\nobreak\ a.s$ if $\lambda_{j}=0,$ that is, a Poisson distribution
with mean zero is defined to be the degenerate distribution at zero. Let
$\boldsymbol{\lambda}=\\{\lambda_{j}\\}_{j\geq 1}$ be the sequence of
associated parameters such that
$\delta=\sum\limits_{j=1}^{\infty}\lambda_{j}<\infty$ and
$\sum\limits_{j=1}^{\infty}t^{j}\lambda_{j}<\infty,$ for $0<|t|<M,$ for some
$M>0.$ We call the distribution of

$G_{P}(\boldsymbol{\lambda})=\sum_{j=1}^{\infty}jN_{j},$ (3.1)

a generalized Poisson distribution ($GPD$). Note that
$G_{P}(\boldsymbol{\lambda})=W_{\infty}.$ Our generalization of the Poisson
distribution is new and different from the ones available in the literature,
see for instance, Consul and Jain [CJ73] where a two parameter generalization
of the Poisson distribution is obtained as a limiting form of the generalized
negative binomial distribution.

######  Remarks 3.1.

(i) Let $\lambda_{j}=0$ for $j\geq 2.$ Then, obviously,
$G_{P}(\boldsymbol{\lambda})\stackrel{{\scriptstyle\mathscr{L}}}{{=}}N_{1}\sim
Poi(\lambda_{1})$, the Poisson distribution with mean$\lambda.$

(ii) Let $\lambda_{j}=0$, for $j\geq k+1.$ Then, clearly,
$G_{P}(\boldsymbol{\lambda})\stackrel{{\scriptstyle\mathscr{L}}}{{=}}N_{1}+2N_{2}+\ldots+kN_{k}=W_{k}\sim
Poi(u_{1},\ldots,u_{k})$, the Poisson distribution of order $k.$

The study of the $GPD$’s provide a unified approach and has not been addressed
in the literature. In view of the form of (3.1), the $GPD$ may be viewed as
the Poisson distribution of order infinity. First we show that the series in
the right-hand side of (3.1) indeed follows a compound Poisson distribution.
One could also apply Kolmogorov’s three series theorem to check the
convergence.

Let now $Y$ be a positive integer-valued random variable with the distribution

$\mathbb{P}\\{Y=j\\}=\frac{\lambda_{j}}{\delta},\leavevmode\nobreak\
j\in{\mathbb{N}}.$ (3.2)

where $\delta=\sum\limits_{j=1}^{\infty}\lambda_{j}.$ Observe that given a
sequence $\boldsymbol{\lambda}$, satisfying the conditions given above, the
distributions of $N\sim Poi(\delta)$ and $Y$ can be determined.

######  Theorem 3.1.

Let $G_{P}(\boldsymbol{\lambda})$ be the $GPD$ defined in (3.1) and
$\\{Y_{j}\\}_{j\geq 1}$ be a sequence of IID positive-integer valued rvs with
distribution defined in (3.2). Also, let $N\sim Poi(\delta)$ be a Poisson rv
with mean $\delta$ and is independent of the sequence $\\{Y_{j}\\}$. Then

$\sum_{j=1}^{\infty}jN_{j}\stackrel{{\scriptstyle\mathscr{L}}}{{=}}\sum_{j=1}^{N}Y_{j}.$
(3.3)

###### Proof.

First note that the $PGF$ of $N$ is

$H_{N}(z)=\mathbb{E}\left(z^{N}\right)=e^{-\delta(1-z)}.$

Consider now the random sum $T_{N}=\sum_{j=1}^{N}Y_{j}.$ Then the $PGF$ of
$T_{N}$ is

$\displaystyle H_{T_{N}}(z)$ $\displaystyle=H_{N}(H_{Y_{1}}(z))$
$\displaystyle=e^{-\delta(1-H_{Y_{1}}(z))}$
$\displaystyle=\exp\left({\delta\left(\sum_{j=1}^{\infty}z^{j}\frac{\lambda_{j}}{\delta}-1\right)}\right)$
$\displaystyle=\exp\left(\sum_{j=1}^{\infty}z^{j}\lambda_{j}-\delta\right)$
(3.4)
$\displaystyle=\exp\left(\sum_{j=1}^{\infty}z^{j}\lambda_{j}-\sum_{n=1}^{\infty}\lambda_{j}\right)$
$\displaystyle=\exp\left(\sum_{j=1}^{\infty}\lambda_{j}(z^{j}-1)\right).$
(3.5)

Also, the $PGF$ of $\sum_{j=1}^{\infty}jN_{j}$ is

$\displaystyle\mathbb{E}\left(z^{N_{1}+2N_{2}+\cdots}\right)=$
$\displaystyle\prod_{j=1}^{\infty}\mathbb{E}\left(z^{jN_{j}}\right)$
$\displaystyle=$
$\displaystyle\prod_{j=1}^{\infty}\mathbb{E}\left((z^{j})^{N_{j}}\right)$
$\displaystyle=$ $\displaystyle\prod_{j=1}^{\infty}e^{-\lambda_{j}(1-z^{j})}$
$\displaystyle=$
$\displaystyle\exp\left({\sum_{j=1}^{\infty}\lambda_{j}(z^{j}-1)}\right).$
(3.6)

which coincides with (3.5). This proves the result. ∎

The above result motivates the following alternative definition.

######  Definition 3.1.

A Compound Poisson Distribution. Let $\\{N_{j}\\}$ be a sequence of Poisson
rvs with parameter $\lambda_{j}$ and
$\delta=\sum\limits_{j=1}^{\infty}\lambda_{j}$. Let $\\{Y_{j}\\}$ be a
sequence of $iid$ rvs with $\mathbb{P}(Y_{1}=j)={\lambda_{j}}/{\delta}$ for
$j\geq 1$. We call the distribution $\sum_{j=1}^{N}Y_{j}=T_{N},$ where $N\sim
Poi(\delta)$ and is independent of the $Y_{j}$, the compound Poisson
distribution ($CPD$) and denote it by $C_{P}(\boldsymbol{\lambda}).$

######  Remarks 3.2.

(i) Let $\lambda_{j}=0$ for $j\geq 2.$ Then, obviously, $\delta=\lambda_{1}$
and $T_{N}\stackrel{{\scriptstyle\mathscr{L}}}{{=}}N_{1}\sim
Poi(\lambda_{1})$, the Poisson distribution.

(ii) Let $\lambda_{j}=0$, for $j\geq k+1.$ Then,
$\delta=\delta_{k}=\sum_{j=1}^{k}\lambda_{j}$ and
$T_{N}\stackrel{{\scriptstyle\mathscr{L}}}{{=}}N_{1}+2N_{2}+\ldots+kN_{k}\sim
Poi(u_{1},\ldots,u_{k})$, the Poisson distribution of order $k$ and has the
$PGF$

$\displaystyle\exp\left({\sum_{j=1}^{k}-\lambda_{j}(1-z^{j})}\right).$ (3.7)

The mean and variance of the $CPD$ follows easily. Since $E(N)=Var(N)$, we
have

$\displaystyle{\mathbb{E}}(T_{N})=E(Y_{1})E(N)=\sum_{j=1}^{\infty}j{\lambda_{j}}$

and similarly

$\displaystyle\operatorname{Var}(T_{N})=E(Y_{1}^{2})E(N)=\sum_{j=1}^{\infty}j^{2}{\lambda_{j}}.$

Next, we obtain the probability distribution of the $CPD$. Define
${T_{0}\equiv 0}\leavevmode\nobreak\ a.s.$ For $1\leq m\leq n$, let

$\displaystyle\Delta_{m,n}=$ $\displaystyle\\{(y_{1},y_{2},\ldots
y_{m}):y_{j}\geq 1;\sum_{j=1}^{m}y_{j}=n\\},$ (3.8)

and $\sum_{y_{j}\in\Delta_{m,n}}\lambda_{y_{1}}\cdots\lambda_{y_{m}}=0$, if
the set $\Delta_{m,n}$ is empty.

######  Theorem 3.2.

(i): The $PMF$ of the $C_{P}(\boldsymbol{\lambda})$ defined in (3.3) is

$\displaystyle\mathbb{P}\left\\{T_{N}=n\right\\}=\begin{cases}e^{\delta},\quad\text{if}\leavevmode\nobreak\
\leavevmode\nobreak\ n=0\\\
e^{-\delta}\sum_{m=1}^{n}\Big{\\{}\sum_{y_{j}\in\Delta_{m,n}}\lambda_{y_{1}}\cdots\lambda_{y_{m}}\Big{\\}}\frac{1}{m!},\quad\text{if}\leavevmode\nobreak\
\leavevmode\nobreak\ n\geq 1.\end{cases}$ (3.9)

(ii): An explicit expression in terms of Bell polynomials is

$\displaystyle P(T_{N}=n)=$
$\displaystyle\frac{e^{-\delta}}{n!}{B}_{n}\left(u_{1},u_{2},\ldots,u_{n}\right),\quad
n\geq 0,$ (3.10)

where $u_{j}=j!\lambda_{j}$, for $1\leq j\leq n.$

###### Proof.

(i): First, clearly,
$\mathbb{P}\left\\{T_{N}=0\right\\}=\mathbb{P}(N=0)=e^{-\delta}$. For $n\geq
1$, we have

$\displaystyle\mathbb{P}\left\\{T_{N}=n\right\\}$
$\displaystyle=\sum_{m=1}^{n}\mathbb{P}\left\\{Y_{1}+Y_{2}+\cdots+Y_{m}=n\mid
N=m\right\\}\mathbb{P}\left\\{N=m\right\\}\quad(\because y_{j}\geq 1)$
$\displaystyle=\sum_{m=1}^{n}\mathbb{P}\left\\{Y_{1}+Y_{2}+\cdots+Y_{m}=n\right\\}\mathbb{P}\left\\{N=m\right\\}$
$\displaystyle=\sum_{m=1}^{n}\Big{\\{}\sum_{y_{j}\in\Delta_{m,n}}\mathbb{P}\left\\{Y_{1}=y_{1},Y_{2}=y_{2},\ldots,Y_{m}=y_{m}\right\\}\Big{\\}}\mathbb{P}\left\\{N=m\right\\}$
$\displaystyle=\sum_{m=1}^{n}\sum_{y_{j}\in\Delta_{m,n}}\prod_{i=1}^{m}\mathbb{P}\left\\{Y_{i}=y_{i}\right\\}\mathbb{P}\left\\{N=m\right\\}$
$\displaystyle=\sum_{m=1}^{n}\Big{(}\sum_{y_{j}\in\Delta_{m,n}}\frac{\lambda_{y_{1}}\cdots\lambda_{y_{m}}}{\delta^{m}}\Big{)}\frac{e^{-\delta}\delta^{m}}{m!}$
$\displaystyle=e^{-\delta}\sum_{m=1}^{n}\Big{\\{}\sum_{y_{j}\in\Delta_{m,n}}\lambda_{y_{1}}\cdots\lambda_{y_{m}}\Big{\\}}\frac{1}{m!},$
(3.11)

where $\delta=\sum_{j=1}^{\infty}\lambda_{j}<\infty.$

(ii): The probability generating function ($PGF$) of the
$C_{P}(\boldsymbol{\lambda})$ (see (3.4) ) is

$\displaystyle H_{T_{N}}(z)=$ $\displaystyle e^{-\delta(1-H_{Y_{1}}(z))}$
$\displaystyle=$ $\displaystyle
e^{-\delta}\sum_{k=0}^{\infty}\dfrac{1}{k!}\left(\sum_{j=1}^{\infty}\lambda_{j}z^{j}\right)^{k}$
$\displaystyle=$ $\displaystyle
e^{-\delta}\sum_{k=0}^{\infty}\dfrac{1}{k!}\Big{\\{}\sum_{n=k}^{\infty}\hat{B}_{n,k}\left(\lambda_{1},\lambda_{2},\ldots,\lambda_{n-k+1}\right)z^{n}\Big{\\}}\quad(\text{using}\leavevmode\nobreak\
\eqref{221})$ $\displaystyle=$ $\displaystyle
e^{-\delta}\sum_{n=0}^{\infty}\Big{\\{}\sum_{k=0}^{n}\dfrac{1}{k!}\hat{B}_{n,k}\left(\lambda_{1},\lambda_{2},\ldots,\lambda_{n-k+1}\right)\Big{\\}}z^{n}$
$\displaystyle=$ $\displaystyle
e^{-\delta}\sum_{n=0}^{\infty}\Big{\\{}\dfrac{1}{n!}\sum_{k=0}^{n}{B}_{n,k}\left(1!\lambda_{1},2!\lambda_{2},\ldots,(n-k+1)!\lambda_{n-k+1}\right)\Big{\\}}z^{n}\leavevmode\nobreak\
\leavevmode\nobreak\ (\text{using \eqref{eqn312}})$ $\displaystyle=$
$\displaystyle
e^{-\delta}\sum_{n=0}^{\infty}\Big{\\{}\dfrac{1}{n!}{B}_{n}\left(1!\lambda_{1},2!\lambda_{2},\ldots,n!\lambda_{n}\right)\Big{\\}}z^{n}$

which shows that, for $n\geq 0,$

$\displaystyle P(T_{N}=n)=$
$\displaystyle\frac{e^{-\delta}}{n!}{B}_{n}\left(1!\lambda_{1},2!\lambda_{2},\ldots,n!\lambda_{n}\right)$
$\displaystyle=$
$\displaystyle\frac{e^{-\delta}}{n!}{B}_{n}\left(u_{1},u_{2},\ldots,u_{n}\right),$

which proves the result.

∎


# Syllable-aware Neural Language Models:
A Failure to Beat Character-aware Ones

Zhenisbek Assylbekov
School of Science and Technology
Nazarbayev University
<EMAIL_ADDRESS>&Rustem Takhanov
School of Science and Technology
Nazarbayev University
<EMAIL_ADDRESS>Bagdat Myrzakhmetov
National Laboratory Astana
Nazarbayev University
<EMAIL_ADDRESS>&Jonathan N. Washington
Linguistics Department
Swarthmore College
jonathan.washington
@swarthmore.edu

###### Abstract

Syllabification does not seem to improve word-level RNN language modeling
quality when compared to character-based segmentation. However, our best
syllable-aware language model, achieving performance comparable to the
competitive character-aware model, has 18%–33% fewer parameters and is trained
1.2–2.2 times faster.

##  1 Introduction

Recent advances in neural language modeling (NLM) are connected with
character-aware models Kim et al. (2016); Ling et al. (2015b); Verwimp et al.
(2017). This is a promising approach, and we propose the following direction
related to it: We would like to make sure that in the pursuit of the most
fine-grained representations one has not missed possible intermediate ways of
segmentation, e.g., by syllables. Syllables, in our opinion, are better
supported as linguistic units of language than single characters. In most
languages, words can be naturally split into syllables:

ES: el par-la-men-to a-po-yó la en-mien-da
RU: par-la-ment pod-der-žal po-prav-ku
(EN: the parliament supported the amendment)

Based on this observation, we attempted to determine whether syllable-aware
NLM has any advantages over character-aware NLM. We experimented with a
variety of models but could not find any evidence to support this hypothesis:
splitting words into syllables does not seem to improve the language modeling
quality when compared to splitting into characters. However, there are some
positive findings: while our best syllable-aware language model achieves
performance comparable to the competitive character-aware model, it has
18%–33% fewer parameters and is 1.2–2.2 times faster to train.

##  2 Related Work

Much research has been done on subword-level and subword-aware111Subword-level
LMs rely on subword-level inputs and make predictions at the level of
subwords; subword-aware LMs also rely on subword-level inputs but make
predictions at the level of words. neural language modeling when subwords are
characters Ling et al. (2015b); Kim et al. (2016); Verwimp et al. (2017) or
morphemes Botha and Blunsom (2014); Qiu et al. (2014); Cotterell and Schütze
(2015). However, not much work has been done on syllable-level or syllable-
aware NLM. Mikolov et al. (2012) show that subword-level language models
outperform character-level ones.222Not to be confused with character-aware
ones, see the previous footnote. They keep the most frequent words untouched
and split all other words into syllable-like units. Our approach differs
mainly in the following aspects: we make predictions at the word level, use a
more linguistically sound syllabification algorithm, and consider a variety of
more advanced neural architectures.

We have recently come across a concurrent paper Vania and Lopez (2017) where
the authors systematically compare different subword units (characters,
character trigrams, BPE Sennrich et al. (2016), morphemes) and different
representation models (CNN, Bi-LSTM, summation) on languages with various
morphological typology. However, they do not consider syllables, and they
experiment with relatively small models on small data sets (0.6M–1.4M tokens).

##  3 Syllable-aware word embeddings

Let $\mathcal{W}$ and $\mathcal{S}$ be finite vocabularies of words and
syllables respectively. We assume that both words and syllables have already
been converted into indices. Let
$\mathbf{E_{\mathcal{S}}}\in\mathbb{R}^{|\mathcal{S}|\times d_{\mathcal{S}}}$
be an embedding matrix for syllables — i.e., it is a matrix in which the $s$th
row (denoted as $\mathbf{s}$) corresponds to an embedding of the syllable
$s\in\mathcal{S}$. Any word $w\in\mathcal{W}$ is a sequence of its syllables
$(s_{1},s_{2},\ldots,s_{n_{w}})$, and hence can be represented as a sequence
of the corresponding syllable vectors:

$[\mathbf{s_{1}},\mathbf{s_{2}},\ldots,\mathbf{s_{n_{w}}}].$ (1)

The question is: How shall we pack the sequence (1) into a single vector
$\mathbf{x}\in\mathbb{R}^{d_{\mathcal{W}}}$ to produce a better embedding of
the word $w$?333The same question applies to any model that segments words
into a sequence of characters or other subword units. In our case “better”
means “better than a character-aware embedding of $w$ via the Char-CNN model
of Kim et al. (2016)”. Below we present several viable approaches.

unconstitutional conditions on stack of two LSTMs word vector Highway layers
(optional) Syllable-aware word embedding model Syllable embeddings un con sti
tu tional imposes unconstitutional conditions

Figure 1: Syllable-aware language model.

###  3.1 Recurrent sequential model (Syl-LSTM)

Since the syllables are coming in a sequence it is natural to try a recurrent
sequential model:

$\mathbf{h_{t}}=f(\mathbf{s_{t}},\mathbf{h_{t-1}}),\qquad\mathbf{h_{0}}=\mathbf{0},$
(2)

which converts the sequence of syllable vectors (1) into a sequence of state
vectors $\mathbf{h_{1:n_{w}}}$. The last state vector $\mathbf{h_{n_{w}}}$ is
assumed to contain the information on the whole sequence (1), and is therefore
used as a word embedding for $w$. There is a big variety of transformations
from which one can choose $f$ in (2); however, a recent thorough evaluation
Jozefowicz et al. (2015) shows that the LSTM Hochreiter and Schmidhuber (1997)
with its forget bias initialized to $1$ outperforms other popular
architectures on almost all tasks, and we decided to use it for our
experiments. We will refer to this model as Syl-LSTM.

###  3.2 Convolutional model (Syl-CNN)

Inspired by recent work on character-aware neural language models Kim et al.
(2016) we decided to try this approach (Char-CNN) on syllables. Our case
differs mainly in the following two aspects:

  1. 1.

The set of syllables $\mathcal{S}$ is usually bigger than the set of
characters $\mathcal{C}$,444In languages with alphabetic writing systems. and
also the dimensionality $d_{\mathcal{S}}$ of syllable vectors is expected to
be greater than the dimensionality $d_{\mathcal{C}}$ of character vectors.
Both of these factors result in allocating more parameters on syllable
embeddings compared to character embeddings.

  2. 2.

On average a word contains fewer syllables than characters, and therefore we
need narrower convolutional filters for syllables. This results in spending
fewer parameters per convolution.

This means that by varying $d_{\mathcal{S}}$ and the maximum width of
convolutional filters $L$ we can still fit the parameter budget of Kim et al.
(2016) to allow fair comparison of the models.

Like in Char-CNN, our syllable-aware model, which is referred to as Syl-
CNN-[L], utilizes max-pooling and highway layers Srivastava et al. (2015) to
model interactions between the syllables. The dimensionality of a highway
layer is denoted by $d_{\text{HW}}$.

###  3.3 Linear combinations

We also considered using linear combinations of syllable-vectors to represent
the word embedding:

$\textstyle\mathbf{x}=\sum_{t=1}^{n_{w}}\alpha_{t}(s_{t})\cdot\mathbf{s_{t}}.$
(3)

The choice for ${\alpha_{t}}$ is motivated mainly by the existing approaches
(discussed below) which proved to be successful for other tasks.

Syl-Sum: Summing up syllable vectors to get a word vector can be obtained by
setting $\alpha_{t}(s_{t})=1$. This approach was used by Botha and Blunsom
(2014) to combine a word and its morpheme embeddings into a single word
vector.

Syl-Avg: A simple average of syllable vectors can be obtained by setting
$\alpha_{t}(s_{t})={1}/{n_{w}}$. This can be also called a “continuous bag of
syllables” in an analogy to a CBOW model Mikolov et al. (2013), where vectors
of neighboring words are averaged to get a word embedding of the current word.

Syl-Avg-A: We let the weights $\alpha_{t}$ in (3) be a function of parameters
$(a_{1},\ldots,a_{n})$ of the model, which are jointly trained together with
other parameters. Here $n=\max_{w}\\{n_{w}\\}$ is a maximum word length in
syllables. In order to have a weighted average in (3) we apply a softmax
normalization:

${\alpha_{t}}=\operatorname{softmax}(\mathbf{a})_{t}=\frac{\exp(a_{t})}{\sum_{\tau=1}^{n}\exp(a_{\tau})}$
(4)

Syl-Avg-B: We can let $\alpha_{t}$ depend on syllables and their positions:

$\alpha_{t}=\alpha_{t}(s_{t})=\operatorname{softmax}(\mathbf{a}_{s_{t}}+\mathbf{b})_{t}$

where $\mathbf{A}\in\mathbb{R}^{d_{\mathcal{S}}\times n}$ (with elements
${a}_{s,t}$) is a set of parameters that determine the importance of each
syllable type in each (relative) position, $\mathbf{b}\in\mathbb{R}^{n}$ is a
bias, which is conditioned only on the relative position. This approach is
motivated by recent work on using an attention mechanism in the CBOW model
Ling et al. (2015a).

We feed the resulting $\mathbf{x}$ from (3) into a stack of highway layers to
allow interactions between the syllables.

###  3.4 Concatenation (Syl-Concat)

In this model we simply concatenate syllable vectors (1) into a single word
vector:

$\textstyle\mathbf{x}=[\mathbf{s_{1}};\mathbf{s_{2}};\ldots;\mathbf{s_{n_{w}}};\underbrace{\mathbf{0};\mathbf{0};\ldots;\mathbf{0}}_{n-n_{w}}]$

We zero-pad $\mathbf{x}$ so that all word vectors have the same length $n\cdot
d_{\mathcal{S}}$ to allow batch processing, and then we feed $\mathbf{x}$ into
a stack of highway layers.

##  4 Word-level language model

Once we have word embeddings $\mathbf{x_{1:k}}$ for a sequence of words
$w_{1:k}$ we can use a word-level RNN language model to produce a sequence of
states $\mathbf{h_{1:k}}$ and then predict the next word according to the
probability distribution

$\Pr(w_{k+1}|w_{1:k})=\operatorname{softmax}(\mathbf{h_{k}}\mathbf{W}+\mathbf{b}),$

where $\mathbf{W}\in\mathbb{R}^{d_{\text{LM}}\times|\mathcal{W}|}$,
$\mathbf{b}\in\mathbb{R}^{|\mathcal{W}|}$, and $d_{\text{LM}}$ is the hidden
layer size of the RNN. Training the model involves minimizing the negative
log-likelihood over the corpus $w_{1:K}$:

$\textstyle-\sum_{k=1}^{K}\log\Pr(w_{k}|w_{1:k-1})\longrightarrow\min$ (5)

As was mentioned in Section 3.1 there is a huge variety of RNN architectures
to choose from. The most advanced recurrent neural architectures, at the time
of this writing, are recurrent highway networks Zilly et al. (2017) and a
novel model which was obtained through a neural architecture search with
reinforcement learning Zoph and Le (2017). These models can be spiced up with
the most recent regularization techniques for RNNs Gal and Ghahramani (2016)
to reach state-of-the-art. However, to make our results directly comparable to
those of Kim et al. (2016) we select a two-layer LSTM and regularize it as in
Zaremba et al. (2014).

##  5 Experimental Setup

We search for the best model in two steps: first, we block the word-level
LSTM’s architecture and pre-select the three best models under a small
parameter budget (5M), and then we tune these three best models’
hyperparameters under a larger budget (20M).

Pre-selection: We fix $d_{\text{LM}}$ (hidden layer size of the word-level
LSTM) at 300 units per layer and run each syllable-aware word embedding method
from Section 3 on the English PTB data set Marcus et al. (1993), keeping the
total parameter budget at 5M. The architectural choices are specified in
Appendix A.

Hyperparameter tuning: The hyperparameters of the three best-performing models
from the pre-selection step are then thoroughly tuned on the same English PTB
data through a random search according to the marginal distributions:

  * •

$d_{\mathcal{S}}\sim U(20,650)$,555$U(a,b)$ stands for a uniform distribution
over $(a,b)$.

  * •

$\log(d_{\text{HW}})\sim U(\log(160),\log(2000))$,

  * •

$\log(d_{\text{LM}})\sim U(\log(300),\log(2000))$,

with the restriction $d_{\mathcal{S}}<d_{\text{LM}}$. The total parameter
budget is kept at 20M to allow for easy comparison to the results of Kim et
al. (2016). Then these three best models (with their hyperparameters tuned on
PTB) are trained and evaluated on small- (DATA-S) and medium-sized (DATA-L)
data sets in six languages.

Optimizaton is performed in almost the same way as in the work of Zaremba et
al. (2014). See Appendix B for details.

Syllabification: The true syllabification of a word requires its grapheme-to-
phoneme conversion and then splitting it into syllables based on some rules.
Since these are not always available for less-resourced languages, we decided
to utilize Liang’s widely-used hyphenation algorithm Liang (1983).

##  6 Results

The results of the pre-selection are reported in Table 1.

Model | PPL | Model | PPL
---|---|---|---
LSTM-Word | 88.0 | Char-CNN | 92.3
Syl-LSTM | 88.7 | Syl-Avg | 88.5
Syl-CNN-2 | 86.6 | Syl-Avg-A | 91.4
Syl-CNN-3 | 84.6 | Syl-Avg-B | 88.5
Syl-CNN-4 | 86.8 | Syl-Concat | 83.7
Syl-Sum | 84.6 |  |
Table 1: Pre-selection results. PPL stands for test set perplexity, all models
have $\approx 5$M parameters.

All syllable-aware models comfortably outperform the Char-CNN when the budget
is limited to 5M parameters. Surprisingly, a pure word-level model,666When
words are directly embedded into $\mathbb{R}^{d_{\mathcal{W}}}$ through an
embedding matrix $\mathbf{E_{\mathcal{W}}}\in\mathbb{R}^{|\mathcal{W}|\times
d_{\mathcal{W}}}$. LSTM-Word, also beats the character-aware one under such
budget. The three best configurations are Syl-Concat, Syl-Sum, and Syl-CNN-3
(hereinafter referred to as Syl-CNN), and tuning their hyperparameters under
20M parameter budget gives the architectures in Table 2.

Model | $d_{\mathcal{S}}$ | $d_{\text{HW}}$ | $d_{\text{LM}}$ | Size | PPL
---|---|---|---|---|---
Syl-CNN | 242 | 1170 | 380 | 15M | 80.5
Syl-Sum | 438 | 1256 | 435 | 18M | 80.3
Syl-Concat | 228 | 781 | 439 | 13M | 79.4
Table 2: Hyperparameters tuning. In Syl-CNN, $d_{\text{HW}}$ is a function of
the primary hyperparameter $c=195$ (see Appendix A).

The results of evaluating these three models on small (1M tokens) and medium-
sized (17M–57M tokens) data sets against Char-CNN for different languages are
provided in Table 3.

Model | EN | FR | ES | DE | CS | RU |
---|---|---|---|---|---|---|---
Char-CNN | 78.9 | 184 | 165 | 239 | 371 | 261 |  DATA-S
Syl-CNN | 80.5 | 191 | 172 | 239 | 374 | 269
Syl-Sum | 80.3 | 193 | 170 | 243 | 389 | 273
Syl-Concat | 79.4 | 188 | 168 | 244 | 383 | 265
Char-CNN | 160 | 124 | 118 | 198 | 392 | 190 |  DATA-L
Syl-CNN777Syl-CNN results on DATA-L are not reported since computational resources were insufficient to run these configurations. | – | – | – | – | – | –
Syl-Sum | 170 | 141 | 129 | 212 | 451 | 233
Syl-Concat | 176 | 139 | 129 | 225 | 449 | 225
Table 3: Evaluation of the syllable-aware models against Char-CNN. In each
case the smallest model, Syl-Concat, has 18%–33% less parameters than Char-CNN
and is trained 1.2–2.2 times faster (Appendix C).

The models demonstrate similar performance on small data, but Char-CNN scales
significantly better on medium-sized data. From the three syllable-aware
models, Syl-Concat looks the most advantageous as it demonstrates stable
results and has the least number of parameters. Therefore in what follows we
will make a more detailed comparison of Syl-Concat with Char-CNN.

Shared errors: It is interesting to see whether Char-CNN and Syl-Concat are
making similar errors. We say that a model gives an error if it assigns a
probability less than $p^{\ast}$ to a correct word from the test set. Figure 2
shows the percentage of errors which are shared by Syl-Concat and Char-CNN
depending on the value of $p^{\ast}$.

Figure 2: Percentage of errors shared by both Syl-Concat and Char-CNN on
DATA-S (left) and DATA-L (right).

We see that the vast majority of errors are shared by both models even when
$p^{\ast}$ is small (0.01).

PPL breakdown by token frequency: To find out how Char-CNN outperforms Syl-
Concat, we partition the test sets on token frequency, as computed on the
training data. We can observe in Figure 3 that, on average, the more frequent
the word is, the bigger the advantage of Char-CNN over Syl-Concat. The more
Char-CNN sees a word in different contexts, the more it can learn about this
word (due to its powerful CNN filters). Syl-Concat, on the other hand, has
limitations – it cannot see below syllables, which prevents it from extracting
the same amount of knowledge about the word.

Figure 3: PPL reduction by token frequency, Char-CNN relative to Syl-Concat on
DATA-L.

PCA of word embeddings: The intrinsic advantage of Char-CNN over Syl-Concat is
also supported by the following experiment: We took word embeddings produced
by both models on the English PTB, and applied PCA to them.888We equalized
highway layer sizes $d_{\text{HW}}$ in both models to have same dimensions for
embeddings. In both cases, word vectors were standardized using the z-score
transformation. Regardless of the threshold percentage of variance to retain,
the embeddings from Char-CNN always have more principal components than the
embeddings from Syl-Concat (see Table 4).

Model | 80% | 90% | 95% | 99%
---|---|---|---|---
Char-CNN | 568 | 762 | 893 | 1038
Syl-Concat | 515 | 729 | 875 | 1035
Table 4: Number of principle components when PCA is applied to word embeddings
produced by each model, depending on % of variance to retain.

This means that Char-CNN embeds words into higher dimensional space than Syl-
Concat, and thus can better distinguish them in different contexts.

LSTM limitations: During the hyperparameters tuning we noticed that increasing
$d_{\mathcal{S}}$, $d_{\text{HW}}$ and $d_{\text{LM}}$ from the optimal values
(in Table 2) did not result in better performance for Syl-Concat. Could it be
due to the limitations of the word-level LSTM (the topmost layer in Fig. 1)?
To find out whether this was the case we replaced the LSTM by a Variational
RHN Zilly et al. (2017), and that resulted in a significant reduction of
perplexities on PTB for both Char-CNN and Syl-Concat (Table 5). Moreover,
increasing $d_{\text{LM}}$ from 439 to 650 did result in better performance
for Syl-Concat. Optimization details are given in Appendix B.

Model | depth | $d_{\text{LM}}$ | Size | PPL
---|---|---|---|---
RHN-Char-CNN | 8 | 650 | 20M | 67.6
RHN-Syl-Concat | 8 | 439 | 13M | 72.0
RHN-Syl-Concat | 8 | 650 | 20M | 69.4
Table 5: Replacing LSTM with Variational RHN.

Comparing syllable and morpheme embeddings: It is interesting to compare
morphemes and syllables. We trained Morfessor 2.0 Creutz and Lagus (2007) in
its default configuration on the PTB training data and used it instead of the
syllabifier in our models. Interestingly, we got $\approx$3K unique morphemes,
whereas the number of unique syllables was $\approx$6K. We then trained all
our models on PTB under 5M parameter budget, keeping the state size of the
word-level LSTM at 300 (as in our pre-selection step for syllable-aware
models). The reduction in number of subword types allowed us to give them
higher dimensionality $d_{\mathcal{M}}=100$ (cf.
$d_{\mathcal{S}}=50$).999$\mathcal{M}$ stands for morphemes.

Convolutional (Morph-CNN-3) and additive (Morph-Sum) models performed better
than others with test set PPLs 83.0 and 83.9 respectively. Due to limited
amount of time, we did not perform a thorough hyperparameter search under 20M
budget. Instead, we ran two configurations for Morph-CNN-3 and two
configurations for Morph-Sum with hyperparameters close to those, which were
optimal for Syl-CNN-3 and Syl-Sum correspondingly. All told, our best
morpheme-aware model is Morph-Sum with $d_{\mathcal{M}}=550$,
$d_{\text{HW}}=1100$, $d_{\text{LM}}=550$, and test set PPL 79.5, which is
practically the same as the result of our best syllable-aware model Syl-Concat
(79.4). This makes Morph-Sum a notable alternative to Char-CNN and Syl-Concat,
and we defer its thorough study to future work.

Source code: The source code for the models discussed in this paper is
available at https://github.com/zh3nis/lstm-syl.

##  7 Conclusion

It seems that syllable-aware language models fail to outperform competitive
character-aware ones. However, usage of syllabification can reduce the total
number of parameters and increase the training speed, albeit at the expense of
language-dependent preprocessing. Morphological segmentation is a noteworthy
alternative to syllabification: a simple morpheme-aware model which sums
morpheme embeddings looks promising, and its study is deferred to future work.

##  Appendix A Pre-selection

In all models with highway layers there are two of them and the non-linear
activation of any highway layer is a ReLU.

LSTM-Word: $d_{\mathcal{W}}=108$, $d_{\text{LM}}=300$.

Syl-LSTM: $d_{\mathcal{S}}=50$, $d_{\text{LM}}=300$.

Syl-CNN-[$L$]: $d_{\mathcal{S}}=50$, convolutional filter widths are
$[1,\ldots,L]$, the corresponding convolutional filter depths are $[c\cdot
l]_{l=1}^{L}$, $d_{\text{HW}}=c\cdot(1+\ldots+L)$. We experimented with
$L=2,3,4$. The corresponding values of $c$ are chosen to be $120,60,35$ to fit
the total parameter budget. CNN activation is $\tanh$.

Linear combinations: We give higher dimensionality to syllable vectors here
(compared to other models) since the resulting word vector will have the same
size as syllable vectors (see (3)). $d_{\mathcal{S}}=175$, $d_{\text{HW}}=175$
in all models except the Syl-Avg-B, where we have $d_{\mathcal{S}}=160$,
$d_{\text{HW}}=160$.

Syl-Concat: $d_{\mathcal{S}}=50$, $d_{\text{HW}}=300$.

##  Appendix B Optimization

LSTM-based models: We perform the training (5) by truncated BPTT Werbos
(1990); Graves (2013). We backpropagate for 70 time steps on DATA-S and for 35
time steps on DATA-L using stochastic gradient descent where the learning rate
is initially set to 1.0 and halved if the perplexity does not decrease on the
validation set after an epoch. We use batch sizes of 20 for DATA-S and 100 for
DATA-L. We train for 50 epochs on DATA-S and for 25 epochs on DATA-L, picking
the best-performing model on the validation set. Parameters of the models are
randomly initialized uniformly in $[-0.05,0.05]$, except the forget bias of
the word-level LSTM, which is initialized to $1$. For regularization we use
dropout Srivastava et al. (2014) with probability 0.5 between word-level LSTM
layers and on the hidden-to-output softmax layer. We clip the norm of the
gradients (normalized by minibatch size) at 5. These choices were guided by
previous work on word-level language modeling with LSTMs Zaremba et al.
(2014).

To speed up training on DATA-L we use a sampled softmax Jean et al. (2015)
with the number of samples equal to 20% of the vocabulary size Chen et al.
(2016). Although Kim et al. (2016) used a hierarchical softmax Morin and
Bengio (2005) for the same purpose, a recent study Grave et al. (2016) shows
that it is outperformed by sampled softmax on the Europarl corpus, from which
DATA-L was derived Botha and Blunsom (2014).

RHN-based models are optimized as in Zilly et al. (2017), except that we
unrolled the networks for 70 time steps in truncated BPTT, and dropout rates
were chosen to be as follows: 0.2 for the embedding layer, 0.7 for the input
to the gates, 0.7 for the hidden units and 0.2 for the output activations.

##  Appendix C Sizes and speeds

On DATA-S, Syl-Concat has 28%–33% fewer parameters than Char-CNN, and on
DATA-L the reduction is 18%–27% (see Fig. 4).

Figure 4: Model sizes on DATA-S (left) and DATA-L, in millions of trainable
variables.

Training speeds are provided in the Table 6. Models were implemented in
TensorFlow, and were run on NVIDIA Titan X (Pascal).

Model | EN | FR | ES | DE | CS | RU |
---|---|---|---|---|---|---|---
Char-CNN | 9 | 8 | 8 | 7 | 6 | 6 |  S
Syl-Concat | 14 | 12 | 12 | 11 | 10 | 9
Char-CNN | 10 | 8 | 7 | 5 | 7 | 4 |  L
Syl-Concat | 22 | 13 | 13 | 6 | 10 | 5
Table 6: Training speeds, in thousands of tokens per second.

## Acknowledgements

We gratefully acknowledge the NVIDIA Corporation for their donation of the
Titan X Pascal GPU used for this research. The work of Bagdat Myrzakhmetov has
been funded by the Committee of Science of the Ministry of Education and
Science of the Republic of Kazakhstan under the targeted program O.0743
(0115PK02473). The authors would like to thank anonymous reviewers and Aibek
Makazhanov for valuable feedback, Makat Tlebaliyev and Dmitriy Polynin for IT
support, and Yoon Kim for providing the preprocessed datasets.

## References

  * Botha and Blunsom (2014) Jan Botha and Phil Blunsom. 2014.  Compositional morphology for word representations and language modelling.  In _Proceedings of ICML_.
  * Chen et al. (2016) Wenlin Chen, David Grangier, and Michael Auli. 2016.  Strategies for training large vocabulary neural language models.  In _Proceedings of ACL_.
  * Cotterell and Schütze (2015) Ryan Cotterell and Hinrich Schütze. 2015.  Morphological word-embeddings.  In _Proceedings of HLT-NAACL_.
  * Creutz and Lagus (2007) Mathias Creutz and Krista Lagus. 2007.  Unsupervised models for morpheme segmentation and morphology learning.  _ACM Transactions on Speech and Language Processing (TSLP)_ , 4(1):3.
  * Gal and Ghahramani (2016) Yarin Gal and Zoubin Ghahramani. 2016.  A theoretically grounded application of dropout in recurrent neural networks.  In _Proceedings of NIPS_.
  * Grave et al. (2016) Edouard Grave, Armand Joulin, Moustapha Cissé, David Grangier, and Hervé Jégou. 2016.  Efficient softmax approximation for gpus.  _arXiv preprint arXiv:1609.04309_.
  * Graves (2013) Alex Graves. 2013.  Generating sequences with recurrent neural networks.  _arXiv preprint arXiv:1308.0850_.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Jean et al. (2015) Sébastien Jean, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio. 2015.  On using very large target vocabulary for neural machine translation.  In _Proceedings of ACL-IJCNLP_.
  * Jozefowicz et al. (2015) Rafal Jozefowicz, Wojciech Zaremba, and Ilya Sutskever. 2015.  An empirical exploration of recurrent network architectures.  In _Proceedings of ICML_.
  * Kim et al. (2016) Yoon Kim, Yacine Jernite, David Sontag, and Alexander M Rush. 2016.  Character-aware neural language models.  In _Proceedings of AAAI_.
  * Liang (1983) Franklin Mark Liang. 1983.  _Word Hy-phen-a-tion by Com-put-er_.  Citeseer.
  * Ling et al. (2015a) Wang Ling, Lin Chu-Cheng, Yulia Tsvetkov, and Silvio Amir. 2015a.  Not all contexts are created equal: Better word representations with variable attention.  In _Proceedings of EMNLP_.
  * Ling et al. (2015b) Wang Ling, Chris Dyer, Alan W Black, Isabel Trancoso, Ramon Fermandez, Silvio Amir, Luis Marujo, and Tiago Luis. 2015b.  Finding function in form: Compositional character models for open vocabulary word representation.  In _Proceedings of EMNLP_.
  * Marcus et al. (1993) Mitchell P Marcus, Mary Ann Marcinkiewicz, and Beatrice Santorini. 1993.  Building a large annotated corpus of english: The penn treebank.  _Computational linguistics_ , 19(2):313–330.
  * Mikolov et al. (2013) Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013.  Efficient estimation of word representations in vector space.  _arXiv preprint arXiv:1301.3781_.
  * Mikolov et al. (2012) Tomáš Mikolov, Ilya Sutskever, Anoop Deoras, Hai-Son Le, Stefan Kombrink, and Jan Cernocky. 2012.  Subword language modeling with neural networks.  _preprint (http://www. fit. vutbr. cz/imikolov/rnnlm/char. pdf)_.
  * Morin and Bengio (2005) Frederic Morin and Yoshua Bengio. 2005.  Hierarchical probabilistic neural network language model.  In _Proceedings of AISTATS_.
  * Qiu et al. (2014) Siyu Qiu, Qing Cui, Jiang Bian, Bin Gao, and Tie-Yan Liu. 2014.  Co-learning of word representations and morpheme representations.  In _Proceedings of COLING_.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of ACL_.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey E Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _Journal of Machine Learning Research_ , 15(1):1929–1958.
  * Srivastava et al. (2015) Rupesh K Srivastava, Klaus Greff, and Jürgen Schmidhuber. 2015.  Training very deep networks.  In _Proceedings of NIPS_.
  * Vania and Lopez (2017) Clara Vania and Adam Lopez. 2017.  From characters to words to in between: Do we capture morphology?  In _Proceedings of ACL_.
  * Verwimp et al. (2017) Lyan Verwimp, Joris Pelemans, Patrick Wambacq, et al. 2017.  Character-word lstm language models.  In _Proceedings of EACL_.
  * Werbos (1990) Paul J Werbos. 1990.  Backpropagation through time: what it does and how to do it.  _Proceedings of the IEEE_ , 78(10).
  * Zaremba et al. (2014) Wojciech Zaremba, Ilya Sutskever, and Oriol Vinyals. 2014.  Recurrent neural network regularization.  _arXiv preprint arXiv:1409.2329_.
  * Zilly et al. (2017) Julian Georg Zilly, Rupesh Kumar Srivastava, Jan Koutník, and Jürgen Schmidhuber. 2017.  Recurrent highway networks.  In _Proceedings of ICML_.
  * Zoph and Le (2017) Barret Zoph and Quoc V Le. 2017.  Neural architecture search with reinforcement learning.  In _Proceedings of ICLR_.