# On momentum operators given by Killing vectors
whose integral curves are geodesics

Thomas Schürmann<EMAIL_ADDRESS>Düsseldorf, Germany

###### Abstract

We consider momentum operators on intrinsically curved manifolds. Given that
the momentum operators are Killing vector fields whose integral curves are
geodesics, it is shown that the corresponding manifold is either flat, or
otherwise of compact type with positive constant sectional curvature and
dimension equal to 1, 3 or 7. Explicit representations of momentum operators
and the associated Casimir element will be discussed for the 3-sphere $S^{3}$.
It will be verified that the structure constants of the underlying Lie algebra
are proportional to $2\hbar/R$, where $R$ is the curvature radius of $S^{3}$.
This results in a countable energy and momentum spectrum of freely moving
particles in $S^{3}$. It is shown that the maximum resolution of the possible
momenta is given by the de-Broglie wave length $\lambda_{R}=\pi R$, which is
identical to the diameter of the manifold. The corresponding canonical
position operators are defined in terms of geodesic normal coordinates and the
associated commutator relations of position and momentum are established.

Every generalization of the ordinary momentum operator in quantum mechanics to
intrinsically curved manifolds strongly depends on the assumptions which are
supposed to be established. Those assumptions are mostly based on the obvious
rules of quantum mechanics in Cartesian coordinates of the flat Euclidean
space.

Let $M$ be an $n$-dimensional smooth Riemannian manifold with metric $g$
(occasionally denoted by $\langle\cdot,\cdot\rangle$). At every point $p\in
M$, smooth manifolds admit a tangent space $T_{p}M$, which is an
$n$-dimensional real vector space. For every smooth function $f$ on $M$,
consider the action of the differential form $df$. Since $df$ is a map on the
tangential space $T_{p}M$ at point $p\in M$, the gradient of $f$ is defined
such that

$\displaystyle df(v)=\langle v,\text{grad}f\rangle$ (1)

for every smooth vector field $v\in T_{p}M$. From this definition one can
obtain the formal expression of the gradient vector field by

$\displaystyle\text{grad}f=(\text{grad}f)^{k}\,\partial_{k}$ (2)

with contravariant components $(\text{grad}f)^{i}=g^{ik}\partial_{k}f$. With
this notation the traditional momentum operator of Euclidean space in
Cartesian coordinates is given by the covariant components
$\hat{p}_{k}f=-i\hbar\,(\text{grad}f)_{k}$, which is commonly written as

$\displaystyle\hat{p}_{k}=-i\hbar\,\partial_{k}$ (3)

in physics textbooks. Following De-Witt dW52 dW57 , for intrinsically curved
manifolds one can obtain more general momentum operators which also satisfy
the canonical commutation relations

$\displaystyle\left[\hat{x}^{j},\hat{p}_{k}\right]$ $\displaystyle=$
$\displaystyle i\hbar\,\delta^{j}_{k}$ (4)
$\displaystyle\left[\,\hat{p}_{j}\,,\hat{p}_{k}\right]$ $\displaystyle=$
$\displaystyle\left[\,\hat{x}_{j}\,,\hat{x}_{k}\right]=0.$ (5)

This yields an enhanced quantization rule

$\displaystyle\hat{p}_{k}=-i\hbar\left(\partial_{k}+\Gamma^{j}_{jk}(x)\right)$
(6)

where the curvature of the manifold is reflected by the contracted Christoffel
symbols $\Gamma^{j}_{jk}(x)$. Compared to the Cartesian quantization rule in
Euclidean space (3), the ordinary partial derivative is replaced by the
definition (6). An advantage of this quantization rule is its applicability to
a wide range of curved spaces and the validity of the canonical commutator
relations. However, in order to set up a quantum Hamiltonian one has to keep
in mind that there is no unique prescription to quantize the classical curved
space Hamilton function. This is because of operator orderings of the kinetic
energy term result in different, inequivalent quantum corrections, such that
the correct Hamiltonian can only be confirmed empirically.

On the other hand, a particular property which is in a sense self-evident for
the Cartesian case is that the partial derivatives $\partial_{k}$ of the
momentum representation in (3) can be understood as orthonormal Killing
vectors whose isometries are ”translations” in Euclidean space. Strictly
speaking, the integral curves of this Killing vectors (called Killing
trajectories) are geodesics, especially for a particle moving in a force-free
surrounding. If this idea is generalized, for instance, to a freely moving
particle on the 3-sphere of radius $R$, then it is already know that the
(geodesic) Killing trajectories are the greater circles. Hence, the structure
coefficients of the underlying Lie algebra are proportional to the fraction
$\hbar/R$ and therefore different from zero and one is compelled to relax the
form of the canonical commutator relations (4) and (5). One of the first
attempts in this direction have been proposed by Segal S60  and were later
developed by Śniatycki S80 , Doebner, Tolar and Nattermann D96 D01 . Without
to go in any detail (a review can be found in A04 ) a generalized momentum
operator is obtained by projection on a given smooth vector field $X$ on $M$
according to

$\displaystyle P_{\\!{}_{X}}=-i\hbar\left(X+\frac{1}{2}\,\text{div}X\right),$
(7)

where $\text{div}X$ is the covariant divergence of the vector field $X$. A
straightforward computation (that is here omitted) yields the general
commutation relation

$\displaystyle\left[P_{\\!{}_{X}},P_{\\!{}_{Y}}\right]=-i\hbar\,P_{\\!{}_{\left[X,Y\right]}}$
(8)

where $\left[X,Y\right]$ denotes the commutator of the two vector fields $X$
and $Y$ in the usual sense of the theory of manifolds S60 . In the case of a
linear manifold this vanishes for two infinitesimal translations, and (8)
specializes to the commutativity of the conventional linear momenta.

Obviously, the operator (6) of De-Witt can be recovered for vector fields $X$
of the special form $X_{k}=\partial_{k}$, with
$\Gamma^{j}_{jk}=\partial_{k}\log\sqrt{g}\equiv\Gamma_{k}$. However, that
means $\text{div}X_{k}=\Gamma_{k}\neq 0$ such that the vector fields $X_{k}$
cannot be Killing vectors if the underlying manifold is intrinsically curved.

Since our focus will be on momentum operators given by Killing vector fields
whose integral curves are geodesics, it seems appropriate to introduce the
following:

Definition 1. (Killing frame) AN68
A Riemannian manifold $M$ is said to have the Killing property if, in some
neighborhood of each point of $M$, there exists an orthonormal frame
$X_{1},...,X_{n}$ such that each $X_{i}$, $i=1,...,n$, is a Killing vector
field (local infinitesimal isometry). Such a frame will be called a Killing
frame.

Since any linear combination, with constant coefficients, of Killing vector
fields is again a Killing vector field, a manifold has the Killing property if
and only if it is always possible to find frames consisting of Killing vector
fields such that $\langle X_{i},X_{j}\rangle=const$, for each choice of $i$
and $j$. The normality condition of the definition implies that the integral
curves of the isometries are geodesics, since a necessary and sufficient
condition for this is that the Killing vector fields have constant length (S54
, p. 349, Y57 , p. 50).

For instance, let $x,y$ and $r,\varphi$ denote Cartesian and polar coordinates
on the Euclidean plane $\mathbb{R}^{2}$ endowed with Euclidean metric. Then
the Killing vector fields corresponding to translations and rotations are
$X_{1}=\partial_{x}$, $X_{2}=\partial_{y}$ and $X_{3}=\partial_{\varphi}$.
Their squared vector norms are $X_{1}^{2}=1,X_{2}^{2}=1$ and
$X_{3}^{2}=r^{2}$. The Killing vector fields $X_{1}$ and $X_{2}$ have a
constant length on the whole plane. Their trajectories are straight lines,
which are geodesics. The Killing trajectories corresponding to rotations
$X_{3}$ are concentric circles around the origin. The length of $X_{3}$ is
constant along the circles, but non-constant on the whole plane. The
corresponding Killing trajectories are circles, which are not geodesics K16 .

In the context given so far we come to the following:

Definition 2. (Momentum operator)
Let $M$ be an $n$-dimensional Riemannian manifold with Killing frame
$X_{1},...,X_{n}$ on $M$. The set of operators defined by

$\displaystyle P_{{}_{X_{k}}}=-i\hbar X_{k}$ (9)

$k=1,...,n$, are called (geodesic) momentum operators in the direction $X_{k}$
on $M$.

This definition is compatible with (7) since every Killing vector field
$X_{i}$ is a priori divergenceless, i.e., $\text{div}X_{i}=0$, $i=1,...,n$.
Moreover, the Lie bracket of two Killing fields is still a Killing field. The
momentum operators (9) thus form a Lie subalgebra of vector fields on $M$. If
$M$ is a complete manifold, this is the Lie algebra of the translation group.
In this case the commutation relations of the Killing vector fields are given
by

$\displaystyle[\,X_{i},X_{j}]=c^{k}_{ij}X_{k}$ (10)

where the structure coefficients $c^{k}_{ij}$ express the multiplication of
pairs of vectors as a linear combination. The corresponding commutator
relations of the momenta are obtained by multiplication with the physical
units $(-i\hbar)^{2}$ on both sides of (10) and subsequently applying the
definition (9), i.e.

$\displaystyle[\,P_{{}_{X_{i}}},P_{{}_{X_{j}}}]=-i\hbar\,c^{k}_{ij}P_{{}_{X_{k}}}$
(11)

which is compatible with the general expression (8). The associated Casimir
element of this Lie algebra is given by AN68 :

Proposition 1. Let $M$ be an $n$-dimensional manifold and $X_{1},...,X_{n}$ be
a Killing frame on $M$. There is a decomposition of the Laplace-Beltrami
operator, such that

$\displaystyle\sum_{j=1}^{n}P^{2}_{\\!{}_{X_{j}}}=-\hbar^{2}\Delta$ (12)

Proof. The vector fields $X_{j}$ can be expressed as a linear combination of
the coordinate vector fields $\partial_{\alpha}=\partial/\partial x^{\alpha}$,
such that

$\displaystyle X_{i}=\xi_{i}^{\alpha}\partial_{\alpha}$ (13)

where each $\xi_{i}^{\alpha}$ is a function. For every smooth $f$ on $M$, we
can write

$\displaystyle\sum_{j}X_{j}^{2}f$ $\displaystyle=$
$\displaystyle\partial_{\alpha}(g^{\alpha\beta}\partial_{\beta}f)-\delta^{ij}(\partial_{\alpha}\xi^{\alpha}_{i})\xi^{\beta}_{j}\partial_{\beta}f$
(14)

which has been obtained by the product rule of differentiation. On the other
hand, the Laplace-Beltrami operator in the natural frame is

$\displaystyle\Delta
f=\partial_{\alpha}(g^{\alpha\beta}\partial_{\beta}f)+\frac{1}{\sqrt{g}}(\sqrt{g}),_{\alpha}g^{\alpha\beta}\partial_{\beta}f$
(15)

Now, it follows that expression (14) is equal to (15) for every $f$, if it can
be shown that

$\displaystyle\frac{1}{\sqrt{g}}(\sqrt{g}),_{\alpha}g^{\alpha\beta}+\delta^{ij}(\partial_{\alpha}\xi^{\alpha}_{i})\,\xi^{\beta}_{j}=0$
(16)

Using the basic property $\nabla g=0$ of the Levi-Civita connection together
with $\delta^{ij}\xi_{i}^{\alpha}\xi_{j}^{\beta}=g^{\alpha\beta}$, we obtain
the following condition

$\displaystyle\delta^{ij}(\nabla_{\\!{}_{X_{i}}}\xi^{\alpha}_{j})=0.$ (17)

This identity can be confirmed as follows:

$\displaystyle 0$ $\displaystyle=$
$\displaystyle\nabla_{\\!{}_{\partial_{l}}}g^{lk}$ (18) $\displaystyle=$
$\displaystyle\delta^{ij}\,\nabla_{\\!{}_{\partial_{l}}}(\xi^{l}_{i}\,\xi^{k}_{j})$
$\displaystyle=$
$\displaystyle\delta^{ij}\Big{(}\text{div}X_{i}\,\,\xi^{k}_{j}\,+\nabla_{\\!{}_{X_{i}}}\xi^{k}_{j}\,\Big{)}$
$\displaystyle=$ $\displaystyle\delta^{ij}\,\nabla_{\\!{}_{X_{i}}}\xi^{k}_{j}$

and thus

$\displaystyle\sum_{j}X_{j}^{2}f=\Delta f$ (19)

With definition (9), in physical units we obtain the statement(12).
$\text{}\hfill\square$

According to this decomposition of the covariant Laplacian there is no
ambiguity concerning operator orderings of the kinetic energy term. It is also
important to know that the decomposition (12) is independent of the particular
choice of the orthonormal basis H15 . Moreover, the commutator of $\Delta$
with the elements $X_{j}$ of the Lie algebra (10) is given by

$\displaystyle[X_{j},\Delta]=0.$ (20)

From the mathematical point of view, the Casimir element has a meaning only
for the theory of representations, but not as an element of the Lie algebra,
since the product in (12) is not defined for the algebra itself. However, from
linear algebra we know that the eigenvectors of a linear operator always form
a basis for the vector space in question. In addition, for any Lie group, one
or more of the generators can be simultaneously diagonalized using similarity
transformations. The set of generators that can be diagonalized simultaneously
are called Cartan generators. Thus, a suggestive and particularly easy basis
for the vector space of each representation is given by the eigenvectors of
the Cartan generators (see below).

The scope of the concept given so far asks for a mathematical classification
of manifolds with Killing property. It has been shown that a Riemannian
manifold having the Killing property must be locally symmetric AN68 . Thus,
each point of a connected Riemannian manifold having the Killing property has
an open neighbourhood which is isometric to an open neighbourhood in a simply
connected Riemannian symmetric space $M$. Then $M$ also has the Killing
property and, moreover, has global Killing frames. In fact, a local Killing
frame exists on $M$ because of the given local isometry, and can be extended
uniquely to give a global Killing frame. The extension of each Killing vector
field to a global Killing vector field is possible since the symmetry implies
completeness. This extension remains orthonormal since the Riemannian
structure on $M$ is subordinate to a real analytic Riemannian structure (cf.
W67  p. 240, H62  p. 187).

A simply connected Riemannian symmetric space is said to be irreducible if it
is not the product of two or more Riemannian symmetric spaces. It can then be
shown that any simply connected Riemannian symmetric space is a Riemannian
product of irreducible ones. Therefore, we furthermore restrict ourselves to
the irreducible, simply connected Riemannian symmetric spaces. Any simply
connected Riemannian symmetric space $M$ is of one of the following three
types: ($i$) Euclidean type: $M$ has vanishing curvature, and is therefore
isometric to a Euclidean space. ($ii$) Compact type: $M$ has nonnegative (but
not identically zero) sectional curvature. ($iii$) Non-compact type: $M$ has
non-positive (but not identically zero) sectional curvature. Actually,
strictly negatively curved manifolds imply that there are no nontrivial (real
valued) orthonormal Killing fields.

Manifolds of constant positive curvature are known AN68  to have the Killing
property only if the dimension of $M$ is equal to 1, 3 or 7. For the spheres
$S^{1}$, $S^{3}$ and $S^{7}$, in fact, there is a global Killing frame. The
construction depends essentially on the existence of a multiplication in
$\mathbb{R}^{2}$ (complex numbers), $\mathbb{R}^{4}$ (quaternions), and
$\mathbb{R}^{8}$ (Cayley numbers).

Circle: For the circle we can take $X_{1}$ to be the unit tangent vector
field, say pointing in the anti-clockwise direction. More precisely consider
the situation of a compact subset $M\subset S^{1}$ embedded in
$\mathbb{R}^{2}$. The general solution of the Killing equation ${\cal
L}_{X_{1}}g=0$ on $S^{1}$, with metric $ds^{2}=\rho^{2}d\varphi^{2}$ is given
by $X_{1}=\xi^{\varphi}\,\partial_{\varphi}$, for
$\xi^{\varphi}\in\mathbb{R}$. Let $\xi^{\varphi}=1/\rho$, where $\rho$ is the
constant (hyper-) radius of the circle, then we have $X_{1}^{2}=1$ and the
Killing trajectory is a geodesic. The associated momentum operator is

$\displaystyle
P_{\varphi}=-i\hbar\,\frac{1}{\rho}\frac{\partial}{\partial\varphi}.$ (21)

This operator is symmetric on any compact set $M\subset S^{1}$, with
$f=\text{const.}$ on the boundary. It has been mentioned in GP04  that in
quantum mechanics on a circle with standard commutation relation for $\varphi$
and $p_{\varphi}$, the uncertainty relation cannot be stronger than
$\sigma_{p}\sigma_{\varphi}\geq 0$, where $\sigma_{\varphi}$ and $\sigma_{p}$
are the standard deviation of position and momentum. Indeed, this inequality
is not informative at all, since a product of two nonnegative values cannot be
negative. Alternatively, we refer to the approach in TS09 , which is not
affected by difficulties arising in defining a proper measure of position
uncertainty on manifolds mentioned in T04 . By applying the substitution
$r=\rho\,\varphi$, which corresponds to the arc-length on $S^{1}$, the
uncertainty principle of TS09  is given by

$\displaystyle\sigma_{p}\Delta r\geq\pi\hbar$ (22)

where $\Delta r$ is the measure (length) of a compact domain on $S^{1}$.

3-sphere The 3-sphere of radius ${R>0}$ can be understood as the three-
dimensional hypersurface in the four-dimensional Euclidean space. This can
naturally be described by standard spherical coordinates of $\mathbb{R}^{4}$,
given by GP09 :

$\displaystyle x^{1}$ $\displaystyle=$ $\displaystyle R\cos\chi$
$\displaystyle x^{2}$ $\displaystyle=$ $\displaystyle R\sin\chi\cos\theta$
$\displaystyle x^{3}$ $\displaystyle=$ $\displaystyle
R\sin\chi\sin\theta\cos\varphi$ $\displaystyle x^{4}$ $\displaystyle=$
$\displaystyle R\sin\chi\sin\theta\sin\varphi.$

In order to cover all points of the 3-sphere with both positive and negative
values of the coordinates $x^{i}$, it is necessary that
$0\leq\chi,\theta\leq\pi$, $0\leq\varphi<2\pi$. In these coordinates the
metric of $S^{3}$ takes the form

$\displaystyle ds^{2}$ $\displaystyle=$ $\displaystyle
R^{2}\big{(}d\chi^{2}+\sin^{2}\\!\chi\,(d\theta^{2}+\sin^{2}\\!\theta\,d\varphi^{2})\big{)}.$
(23)

We solved the corresponding Killing equation for the unit sphere ($R=1$) and
selected the following orthonormal Killing frame:

$\displaystyle X_{1}$ $\displaystyle=$
$\displaystyle\phantom{,}\sin\theta\cos\varphi\,\partial_{\chi}$
$\displaystyle+$
$\displaystyle(\cot\chi\cos\theta\cos\varphi-\sin\varphi)\,\partial_{\theta}$
$\displaystyle-$
$\displaystyle(\cot\chi\csc\theta\sin\varphi+\cot\theta\cos\varphi)\,\partial_{\varphi}$
$\displaystyle X_{2}$ $\displaystyle=$
$\displaystyle\phantom{,}\sin\theta\sin\varphi\,\partial_{\chi}$
$\displaystyle+$
$\displaystyle(\cot\chi\cos\theta\sin\varphi+\cos\varphi)\,\partial_{\theta}$
$\displaystyle+$
$\displaystyle(\cot\chi\csc\theta\cos\varphi-\cot\theta\sin\varphi)\,\partial_{\varphi}$
$\displaystyle X_{3}$ $\displaystyle=$
$\displaystyle\cos\theta\,\partial_{\chi}-\cot\chi\sin\theta\,\partial_{\theta}+\partial_{\varphi}$
(26)

while the case $R\neq 1$ can be obtained by division of the right side by $R$.
The orthonormality relation $g(X_{i},X_{j})=\delta_{ij}$ is easily verified.
We also determined the corresponding representation in Cartesian coordinates
$x^{i}$ of the Euclidean embedding space $\mathbb{R}^{4}$ and obtained the
equivalent representation

$\displaystyle K_{1}$ $\displaystyle=$
$\displaystyle(-x^{4},-x^{3},\phantom{-}x^{2},\phantom{-}x^{1})$ (27)
$\displaystyle K_{2}$ $\displaystyle=$
$\displaystyle(\phantom{-}x^{3},-x^{4},-x^{1},\phantom{-}x^{2})$ (28)
$\displaystyle K_{3}$ $\displaystyle=$
$\displaystyle(-x^{2},\phantom{-}x^{1},-x^{4},\phantom{-}x^{3})$ (29)

which satisfy $K_{i}\cdot K_{j}=\delta_{ij}$ with respect to the Euclidean
scalar product. One can check that the Lie algebra generated by
$\\{X_{1},X_{2},X_{3}\\}$ is given by the commutation relations

$\displaystyle[\,X_{i},X_{j}]=-\frac{2}{R}\,\epsilon_{ijk}X_{k}$ (30)

where $\epsilon_{ijk}$ is the Levi-Civita symbol in three dimensions. In
physical units this can be rewritten as

$\displaystyle[\,P_{\\!{}_{X_{i}}},P_{\\!{}_{X_{j}}}]=\frac{2i\hbar}{R}\,\,\epsilon_{ijk}P_{\\!{}_{X_{k}}}$
(31)

The corresponding Hamilton operator $H$ of a free particle is given by

$\displaystyle
H=\frac{1}{2m}\sum_{i=1}^{3}P_{\\!{}_{X_{i}}}^{2}=-\frac{\hbar^{2}}{2m}\Delta$
(32)

which is equal to the Casimir element of Proposition 1 in three dimensions.
Thus, it follows that

$\displaystyle[\,H,P_{\\!{}_{X_{i}}}]=0$ (33)

for $i=1,2,3$.

An alternative decompositions of of the Laplacian in (32), by using six (non-
orthonormal) Killing vector fields instead of three, has been proposed in
Santander etal. SRC12 . One essential point of the approach in SRC12  is that
the structure coefficients of the associated commutator relations are not
constants, such that the Hamiltonian cannot be considered as a Casimir element
of the operator algebra. This makes the analysis of the eigenvalues and the
corresponding eigenspaces quite complicated. Actually, there seems to be no
reason why we should regard a decomposition of the free Hamiltonian in terms
of additional angular momenta whose integral curves are not geodesics.

The eigenvalues of $H$ can be obtained by the hyperspherical harmonics on
$S^{3}$, which have been discussed as part of investigations of a variety of
gravitational physics problems in spaces with the topology of the 3-sphere L05
L17 . According to L17 , these hyperspherical harmonics on $S^{3}$ are denoted
by $Y^{nlm}$. The integers $n,l$ and $m$ with $n\geq l\geq 0$ and $l\geq
m\geq-l$ indicate the order of the harmonic. These harmonics are
eigenfunctions of the covariant Laplacian according to

$\displaystyle\Delta Y^{nlm}=-\frac{n(n+2)}{R^{2}}\,Y^{nlm}.$ (34)

The corresponding energy eigenvalues $E_{n}$ of $H$ are given by

$\displaystyle E_{n}=\frac{\hbar^{2}}{2m}\frac{n(n+2)}{R^{2}}.$ (35)

for $n=0,1,...$

Now let us consider the corresponding eigenvalue spectrum of the momentum
operators. Although $P_{\\!{}_{X_{i}}}$ and $H$ are commuting Hermitian
operators it is not necessarily given that each eigenbasis of $H$ is also an
eigenbasis of $P_{\\!{}_{X_{i}}}$. Indeed, most of the functions $Y^{nlm}$
given in (34) are not eigenfunctions of the momentum operator
$P_{\\!{}_{X_{3}}}$. A simultaneous eigenbasis can be obtained by applying the
standard textbook formalism, but based on the specific algebra given by (31).
Rather then working with the operators $P_{\\!{}_{X_{1}}}$ and
$P_{\\!{}_{X_{2}}}$, it is convenient to work with the non-Hermitian linear
combinations,

$\displaystyle P_{\\!{}_{\pm}}=P_{\\!{}_{X_{1}}}\\!\\!\pm iP_{\\!{}_{X_{2}}}$
(36)

where by definition $(P_{\\!{}_{-}})^{\dagger}=P_{\\!{}_{+}}$. Using (31) and
(33), it is straight-forward to show that

$\displaystyle\left[P_{\\!{}_{+}},P_{\\!{}_{-}}\right]$ $\displaystyle=$
$\displaystyle 4\,\frac{\hbar}{R}\,P_{\\!{}_{X_{3}}}$ (37)
$\displaystyle\big{[}P_{\\!{}_{X_{3}}},P_{\\!{}_{\pm}}^{k}\big{]}$
$\displaystyle=$
$\displaystyle\pm\,2\,k\,\frac{\hbar}{R}\,P_{\\!{}_{\pm}}^{k}$ (38)

for $k=0,1,2,...$. Obviously, we also have

$\displaystyle\big{[}H,P_{\\!{}_{\pm}}^{k}\big{]}=0.$ (39)

In order to obtain a simultaneous eigenbasis of $H$ and $P_{\\!{}_{X_{3}}}$
for every fixed $n\in\mathbb{N}$, we consider the set of orthogonal states
$\\{\psi^{nk}_{\\!{}_{\pm}}\\}_{k=0}^{n}$ given by applying the ”ladder”
operators $P_{\\!{}_{\pm}}$ according to

$\displaystyle\psi^{nk}_{\\!{}_{\pm}}=P_{\\!{}_{\pm}}^{k}\,Y^{nn(\mp n)}$ (40)

where $k=0,1,...,n$. From this definition it follows that
$P_{\\!{}_{\mp}}\psi^{n0}_{\\!{}_{\pm}}=0$. By applying the general commutator
rule (38), it can be seen that all of these states are eigenstates of the
momentum operator such that

$\displaystyle P_{\\!{}_{X_{3}}}\psi^{nk}_{\\!{}_{\pm}}$ $\displaystyle=$
$\displaystyle\pm\,p_{nk}\,\psi^{nk}_{\\!{}_{\pm}}$ (41) $\displaystyle
p_{nk}$ $\displaystyle=$ $\displaystyle(k-\frac{n}{2})\,\frac{2\hbar}{R}$ (42)

for $k=0,1,...,n$. The physical interpretation becomes clear by recalling that
the diameter of the 3-sphere is, by definition, the maximal possible geodesic
distance ($\pi R$) between two points on $S^{3}$. If we apply the original
definition of Planck’s constant $\hbar=h/2\pi$, the maximal resolution $\Delta
p_{nk}=p_{nk+1}-p_{nk}$ of the possible momenta in (42) is given by

$\displaystyle\Delta p_{nk}=\frac{h}{\pi R}$ (43)

Actually this unit of momentum is corresponding to the de-Broglie wavelength

$\displaystyle\lambda_{R}=\pi R$ (44)

which is identical to the diameter of the manifold. It is hard to think of
higher resolutions than this.

On the other hand, the energy eigenvalues can be verified by applying the
commutator (39), such that we obtain the eigenvalue equations

$\displaystyle H\psi^{nk}_{\\!{}_{\pm}}=E_{n}\psi^{nk}_{\\!{}_{\pm}}$ (45)

where $E_{n}$ is given in (35). For numerical purposes it is helpful to know
the explicit form of the initial functions $\psi^{n0}_{\\!{}_{\pm}}$, which
are given by

$\displaystyle\psi^{n0}_{\\!{}_{\pm}}=C_{n}\,\sin^{n}\\!\\!\chi\,\sin^{n}\\!\theta\,\,\,e^{\mp
in\varphi},$ (46)

and the normalization constant is

$\displaystyle
C_{n}=\frac{\sqrt{2^{2n-1}(n+1)}}{\pi}\,\frac{n!\,(2n-1)!!}{(2n)!}$ (47)

The representation of the operators $P_{\\!{}_{X_{1}}}$ and
$P_{\\!{}_{X_{2}}}$ can be obtained by inverting relation (36) in order to
express them in terms of the ladder operators. This completes the brief
analysis of momentum operators on $S^{3}$.

Finally, let us bring up some remarks concerning the notion of position
operators. In Segal S60  the position operator $Q$ is defined as follows: if
$f$ is a general function on $M$, then $Q_{{\\!}_{f}}$ is defined as the
operation of multiplication by $f$. For real $f,g$, the operators
$Q_{{\\!}_{f}}$ and $Q_{{\\!}_{g}}$ are Hermitian, such that there is no
difficulty in verifying the commutation relations

$\displaystyle\left[\,Q_{{}_{f}},P_{{}_{X}}\,\right]$ $\displaystyle=$
$\displaystyle i\hbar\,Q_{{}_{X\\!f}}$ (48)
$\displaystyle\left[\,Q_{{}_{f}},Q_{{}_{g}}\,\right]$ $\displaystyle=$
$\displaystyle 0$ (49)

Let $q^{k}$, for $k=1,...,n$, be some a priori unknown position type functions
on $M$ and

$\displaystyle Q^{k}=Q_{q^{k}}$ (50)

be the notation of the position operator with respect to the direction
$X_{k}$. From the exterior calculus point of view, the position functions
$q^{k}$ can be considered as differential forms of zero degree. On the other
hand, the dual frame on $M$ is given by an $n$-tuple of 1-forms
$\theta^{1},...,\theta^{n}$, which are linearly independent at each point on
$M$. When applied to the Killing frame $\\{X_{j}\\}$ on $M$ they satisfy the
duality relation

$\displaystyle\theta^{k}(X_{j})=\delta_{j}^{k}$ (51)

for $j,k=1,...,n$. Thus, it is obvious to ask if there exist position
functions $q^{k}$ on $M$ such that the elements of the dual frame forms are
given by $\theta^{k}=dq^{k}$, for $k=1,...,n$. Let
$\gamma_{{\\!}_{X}}:\left[0,\infty\right)\rightarrow M$ be the constant speed
geodesic emanating from $p\in M$ with $\dot{\gamma}_{{\\!}_{X}}\\!(0)=X$ and
$\gamma_{{\\!}_{X}}\\!(0)=p$. According to the Killing property of $M$, at any
point $p$ there is an orthonormal basis $(X_{1},...,X_{n})$ of $T_{p}M$. Let
the position functions $(q^{1},...,q^{n})$ be defined by the corresponding
geodesic normal coordinates around $p$, such that the decomposition of every
$X\in T_{p}M$ is given by $X=q^{i}X_{i}$. Then the dual frame forms
$\theta^{k}$ with respect to the Killing trajectory $\gamma_{{\\!}_{X}}$ are
given by

$\displaystyle\theta^{k}$ $\displaystyle=$ $\displaystyle
g_{\alpha\beta}\,\xi_{k}^{\alpha}\,d\gamma_{{\\!}_{X}}^{\beta}$ (52)
$\displaystyle=$ $\displaystyle
g_{\alpha\beta}\,\xi_{k}^{\alpha}\,\xi_{j}^{\beta}\,dq^{j}$ $\displaystyle=$
$\displaystyle g(X_{k},X_{j})\,dq^{j}$ $\displaystyle=$ $\displaystyle
dq^{k}.$

Together with (48) and (51) this implies the corresponding canonical
commutator relations

$\displaystyle[\,Q^{k},P_{{\\!}_{X_{j}}}\\!]=i\hbar\,\delta^{k}_{j}$ (53)

An uncertainty relation of position and momentum which is compatible with this
approach and was applied to $S^{3}$ can be found in TS18 TS20 .

## References

  * (1) DeWitt B. S., _Point Transformations in Quantum Mechanics,_ Phys. Rev. 85 (1952) 653.
  * (2) DeWitt B. S., _Dynamical Theory in Curved Spaces. I. A Review of the Classical and Quantum Action Principles,_ Rev. Mod. Phys. 29 (1957) 377.
  * (3) Segal I. E., _Quantization of nonlinear systems,_ J. Math. Phys. 1 (1960), 468–488.
  * (4) Śniatycki J., _Geometric quantization and quantum mechanics,_ Springer Publications, New York 1980.
  * (5) Doebner H.-D. and Nattermann P., _Borel quantization: kinematics and dynamics,_ Acta Phys. Polon. B 27 (1996) 2327–2339.
  * (6) Doebner H.-D., S̆t’ovíc̆ek P. and Tolar J., _Quantization of kinematics on configuration manifolds,_ Rev. Math. Phys. 13 (2001) 1–47.
  * (7) Ali S. T and Engliš M., _Quantization Methods: A Guide for Physicists and Analysts,_ Rev. Math. Phys. 17 (2005) 391-490.
  * (8) D’Atri J. E. and Nickerson H. K., _The existence of special orthonormal frames,_ J. Differential Geom. 2 (1968) 393-409.
  * (9) Schouten J. A., _Ricci-calculus,_ 2nd ed., Springer, Berlin, 1954.
  * (10) Yano K., _The theory of Lie derivatives,_ North Holland, Amsterdam, 1957.
  * (11) Katanaev M. O., _Killing vector fields and a homogeneous isotropic universe,_ Phys. Usp. 59 (2016) 689-700.
  * (12) Hall B C., _Lie groups, Lie algebras, and representations,_ Springer Publications, New York 2015.
  * (13) Wolf J. A., _Spaces of constant curvature,_ McGraw-Hill, New York, 1967.
  * (14) Helgason S., _Differential geometry and symmetric spaces,_ Academic Press, New York, 1962.
  * (15) Golovnev A. V. and Prokhorov L. V., _Uncertainty relations in curved spaces,_ J. Phys. A: Math. Gen. 37 (2004) 2765-2775.
  * (16) Schürmann T. and Hoffmann I., _A closer look at the uncertainty relation of position and momentum,_ Found. Phys. 39 (2009) 958.
  * (17) Trifonov D. A., _Position uncertainty measures on the sphere,_ Proceedings of the Fifth International Conference on Geometry, Integrability and Quantization 755 (Softex, Sofia, Bulgaria 2004) 211-224.
  * (18) Griffiths J. B. and Podolský, J., _Exact Space-Times in Einstein’s General Relativity,_ Cambridge University Press, New York 2009.
  * (19) Lachièze-Rey M. and Caillerie S., _Laplacian eigenmodes for spherical spaces,_ Class. Quant. Grav. 22 (2005) 695-708.
  * (20) Lindblom L, Taylor N. W. and Zhang F., _Scalar, vector and tensor harmonics on the three-sphere,_ Gen. Rel. Grav. 49 (2017) 140.
  * (21) Cariñena J. F., Rañada M. F. and Santander M., _The quantum free particle on spherical and hyperbolic spaces: A curvature dependent approach. II,_ J. Math. Phys. 53 (2012) 102109.
  * (22) Edelen D., _Applied exterior calculus,_ Dover Publications, New York 2005.
  * (23) Bishop R. L. and Goldberg S. I. , _Tensor analysis on manifolds,_ Dover Publications, New York 1980.
  * (24) Schürmann T., _Uncertainty principle on 3-dimensional manifolds of constant curvature,_ Found. Phys. 48 (2018) 716.
  * (25) Schürmann T., _On the uncertainty principle in Rindler and Friedmann spacetimes,_ Eur. Phys. J. C80, (2020) 141.


# A Sketch-Based Neural Model for Generating Commit Messages from Diffs

Nicolae-Teodor Pavel  University Politehnica of Bucharest
Romania
<EMAIL_ADDRESS>Traian Rebedea  University Politehnica of
Bucharest
Romania
<EMAIL_ADDRESS>
###### Abstract

Commit messages have an important impact in software development, especially
when working in large teams. Multiple developers who have a different style of
writing may often be involved in the same project. For this reason, it may be
difficult to maintain a strict pattern of writing informative commit messages,
with the most frequent issue being that these messages are not descriptive
enough. In this paper we apply neural machine translation (NMT) techniques to
convert code diffs into commit messages and we present an improved sketch-
based encoder for this task. We split the approach into three parts. Firstly,
we focus on finding a more suitable NMT baseline for this problem. Secondly,
we show that the performance of the NMT models can be improved by training on
examples containing a specific file type. Lastly, we introduce a novel sketch-
based neural model inspired by recent approaches used for code generation and
we show that the sketch-based encoder significantly outperforms existing state
of the art solutions. The results highlight that this improvement is relevant
especially for Java source code files, by examining two different datasets
introduced in recent years for this task.

###### Index Terms:

commit message generation, code summarization, neural machine translation,
sketch-based encoder, deep learning

##  I Introduction

Version control is widely used in software development to maintain the history
of changes for most software projects nowadays. Version control systems have
been adopted by a large spectrum of developers, ranging from industry to
academia. Each change is called a commit and should be described using a
commit message. The purpose of this message is to offer a short description of
the changes. Commit messages are a key factor in software development,
especially when working in teams, because they help different developers to
understand, review and validate the commits. However, every developer has its
own way of writing the commit messages and it is difficult to maintain a
strict pattern for writing informative short texts. Consequently, these
messages are sometimes too short, too cryptic, or may be even left empty by
some developers.

A solution to this problem is to automatically generate messages by examining
the underlying source code diffs from each commit. Most of the systems [1, 2,
3] that were previously developed for automatically generating commit messages
do not use neural networks but rather more conventional ways, including rules
and patterns, to create the messages. These systems proved to have poor
performance and multiple limitations. For example Jiang et al. [2] proposed a
Naive Bayes approach for generating the verb of a commit message. Their
approach was able to achieve only a 43% accuracy.

In this paper, we improve the work of Jiang et al. [4] who proposed the usage
of neural machine translation (NMT) models to ”translate” code diffs to commit
messages. Diffs are files that track the changes between two versions of a
software repository. Besides the actual code changes, diff files contain some
additional context about those changes as well.

In our experiments we used two large datasets containing diffs and commit
messages released in recent year. The former was proposed by Jiang et al. [2]
and the latter was released by Liu et al. [5] which is a cleaned subset of the
first one. Our most important aim is to determine the full potential of the
recent advances in neural models for code generation and machine translation
when applied for commit message generation.

To this extent, our study focuses on answering the following three research
questions.

RQ1: Has NMT achieved its full potential for commit message generation?

We firstly focused on finding a more suitable NMT baseline for generating
commit messages from diff files by extending the work of Jiang et al. [4]. For
this, we explored three verticals that can increase the performance of NMT
models: model depth, embedding dimensionality, and residual connections. In
addition to their work, we also tackle the problem of rare words by using
BPE[6] preprocessing and coping mechanism [7]. By approaching this problem we
aim at making our solution capable of open vocabulary translation from diffs
to messages.

RQ2: Can training different NMT models for specific file types improve
performance?

We discovered a NMT model which is capable of obtaining state of the art
performance. However, this NMT model is still not capable to outperform NNGen,
a solution based on nearest neighbor search for commit messages [5]. To
understand why NMT models are outperformed by NNGen, we conducted a more
thorough analysis of the diff files in the dataset. Thus, we discovered that
the diff files contain a mix of file types extracted from code repositories
and each diff contains only the changes of a single file. We decide to select
the top nine most frequent file types and create a separate dataset for each
file type. The training process on these datasets is similar to training on
the entire dataset. The ensemble of NMT models trained on these specific file
type datasets outperforms NNGen on the initial dataset but not on the cleaned
one. Our experimental results show that models with fewer layers are able to
generate state of the art results.

RQ3: Can a sketch-based neural model outperform NNGen for generating commit
messages for Java files?

During the previous experiments, we conducted a comparison by file type with
NNGen which revealed that it outperforms NMT models by a high margin on
examples containing Java files. Inspired by this finding, we propose a novel
sketch-based neural model which outperforms NNGen for examples containing Java
files. The preprocessing part for the proposed sketch-based approach generates
a simple code sketch by replacing constants, variables, functions, and class
names with placeholders. The training is done similar to the previous NMT
experiments using the code sketch. After training, the placeholders are
replaced with constants, variables, functions, and class names. The ensemble
containing the model trained with the sketch-based approach for Java files and
the previous NMT models trained on each file type is able to outperform NNGen
on both datasets.

In the rest of this paper, we describe the related work, the proposed method
focusing on the sketch-based model, the experimental setup, and a summary of
the results. At the end, we present both a quantitative comparison with the
state of the art, together with a qualitative study using human evaluation and
concluding remarks on using advanced NMT models for commit message generation.

##  II Related work

###  II-A NMT Models for Commit Message Generation

Jiang et al. [4] proposed a neural machine translation (NMT) model to generate
commit messages from diffs, an approach previously used in natural language
translation. They worked with the dataset published by Jiang and McMillan [2]
consisting of diffs and their respective commit messages from several Java
repositories collected from GitHub. The source code diffs are considered as
the source language and the commit messages, filtered by keeping an action
verb and direct object structure, as target language.

Their model consists of two layers, one for encoder and the other for decoder,
both being recurrent neural networks. They employ a standard Bahdanau
attention mechanism [8]. The word embeddings size used by them is 512 and the
hidden layers size is 1024. They train the model using early stopping for
maximum 5000 epochs with a batch size of 80.

They obtain a BLEU-4 [9] score of 31.92. Compared to this approach, we are
proposing an improved neural architecture using a sketch-based encoder, but
also a more complex approach which divides the dataset into smaller datasets
based on the file type found in the diff file and training an ensemble of
models for each file type specific dataset.

###  II-B NNGen - Nearest Neighbour Commit Message Generation

Liu et al. [5] have analyzed in more depth how the neural machine translation
approach proposed by Jiang et al. [4] performs. They have analyzed when and
why a NMT model performs well and proposed a simpler solution inspired by the
nearest neighbour algorithm.

Firstly, they discovered that the approach of Jiang et al. generates very good
messages on diffs that have a high similarity with diffs in the training set.
They also discovered that 16% of the messages were automatically generated or
describe changes that are trivial.

Taking into consideration these findings, Liu et al. created a new dataset by
cleaning the dataset proposed by Jiang et al. [2]. Their cleaning is done by
manually evaluating 200 randomly picked commits from Jiang et al. test set.
They determine that 71 (35.5%) of those messages have good quality. Out of
those 71 messages, all messages except one have high similarity with messages
from the training set.

Lastly, Liu et al. put forward a new method called NNGen based on the nearest
neighbour (NN) algorithm. Their approach outperforms Jiang et al. [4] approach
on both datasets. Their approach obtains 38.55 (vs. 31.92) BLEU-4 score on the
initial dataset and 16.42 (vs. 14.19) BLEU-4 score on the cleaned dataset.

In our work we will first show that Liu et al. compared their results with a
week NMT baseline. We will then describe how to build a NMT approach that has
a similar performance as NNGen and how to subsequently improve it using a
sketch-based NMT model.

###  II-C Neural Machine Translation

Our work is mostly related to approaches in natural language processing that
address the problems using neural machine translation. We took an in-depth
look at articles that explore the current state of the art for neural machine
translation [10, 11]. Currently the most used architecture is the sequence-to-
sequence model [12, 10, 11, 7]. It consists of two recurrent neural networks,
an encoder and a decoder. The encoder and decoder can have one or more layers.
Residual connections are commonly used to improve the performance of models
with more than one layer for decoder or encoder.

The encoder-decoder approach needs to encode a sentence to a fixed size vector
space. This can be a difficult task for sentences that are too long. In order
to address this issue, the model is extended. Attention mechanisms [8] are
added to the model in order to learn to align the input sequence with the
output sequence. Bahdanau[8] attention is one of the most used attention
mechanisms.

NMT models should be able to be robust and to translate words that are out of
the vocabulary. There are two common techniques that address this issue:
copying mechanism [7] and subword units [6]. Copying mechanism is learning to
output out of vocabulary (OOV) words by copying them directly from the input.
The model learns when to generate words from the existing vocabulary and when
to copy words from the input. The subword units approach has the main goal of
dividing rare words into subunits which occur more frequently. By doing so,
the model generates rare words or OOV words from subunits learned during
training.

###  II-D Sketch-based Methods for Code Generation

Sketch-based methods have been recently found to substantially increase the
performance of vanilla NMT models for code generation [13, 14]. Both methods
are using sequence-to-sequence models, also employed in NMT, to generate
sketches for source code [13] and SQL queries [13, 14]. Afterwards a different
model is used in order to generate the final result (code or query). For
example, Hosu et al. [14] use another sequence-to-sequence model with a dual
encoder which receives both the sketch and the question as input in order to
generate the final SQL query. In contrast to their approaches we leverage the
coding style to create a rule-based method that creates the sketches. The
decoding part in our case is performed using a dictionary created during the
sketch generation. Moreover, in our case we have a sketch-based encoder, while
these methods employ a sketch-based decoder for the sequence-to-sequence
model.

##  III Proposed Method

Our proposed method is split in three parts. Firstly, we focus on finding a
more suitable NMT baseline. Secondly, we propose an improved method which
relies on training on smaller datasets which contain the same file type.
Lastly, we introduce a novel sketch-based neural encoder which substantially
improves the performance of standard NMT models when generating commit
messages for Java files. All our code is available on GitHub and can be found
at the following link 111https://github.com/nicolae-teodor-pavel/commit-
message-generation.

###  III-A Baseline NMT Method

In this section we focus at first on NMT architecture exploration for our
task. Afterwards we tackle the problem of rare and out of the vocabulary
words.

##### Architecture exploration

One of our main goals is to find an architecture that better suits this
problem which is very different from the problem of translating from one
language to another. We explore how different variables of the model affect
the overall performance. The problem of translating diffs to language is more
special than (natural) language to (natural) language translation, therefore
the need for advanced models is justified. We explore how the depth of the
model influences the performance by testing with 2, 4 and 8 layers. Diffs are
a combination of comments and code and our intuition says that a bigger
embedding will help the model. In that regard we conduct tests with an
embedding of size 512 and 1024. In recent years, residual connections have
helped deep models to obtain better performances. Although our model has less
layers than Resnet [15] which introduced residual connections, we also explore
if residual the gradient flow helps our models to gain better performances.

##### Rare and OOV words

BPE is a preprocessing method which is making a vocabulary compression based
on the frequency of the n-grams on the sources and targets texts. During our
experiments we are going to test a vocabulary size of 5,000, 10,000 and
32,000. Based on the vocabulary size, the input length is different. We
selected the input sequence length in order to contain 99% of the examples in
the dataset. The output sequence length remains constant at 30. In Table I we
present the three BPE versions used in our experiments.

TABLE I: BPE versions for processing rare and OOV words | Vocabulary | Input sequence | Enc | Dec
---|---|---|---|---
BPE1 | 5000 | 185 | 2 | 2
BPE2 | 1000 | 170 | 4 | 4
BPE3 | 32000 | 160 | 4 | 4

The copying mechanism implies reproducing parts from the input sequence in the
output sequence, as a way to replicate the human way of speaking. After taking
a closer look at the dataset we found that the initial dataset does not
contain any out of the vocabulary word in the training messages. In order to
use the copying mechanism, we reduced the messages and diffs vocabularies
based on the number of occurrences of each word. In other words, we selected
the most frequent words from both vocabularies. We tested two different
configurations:

  1. 1.

The messages vocabulary is reduced to 6,302 which is 38.67% from the initial
messages vocabulary. The diffs vocabulary is not reduced.

  2. 2.

The messages vocabulary is reduced to 1,600 which is 9.81% from the initial
messages vocabulary and the diffs vocabulary is reduced to 5,000 which is 10%
from the initial messages vocabulary.

In the first configuration the messages vocabulary contains all the words that
appeared in messages at least once in the train messages. In the second
configuration the messages vocabulary contains the words that appeared at
least 10 times in the train messages. In the last configuration the diffs
vocabulary contains the words that appeared at least in 10 times in the train
diffs.

###  III-B Improved Ensemble NMT Model

The next step in our work was to take a closer look on the dataset introduced
by Jiang et al. [2]. One of the first observations was that all the diffs in
the dataset contain only a change to a single file. Following this observation
we explored the frequency of each file type. We observed a mix of files in the
dataset. In Table II we present the top 10 most frequent file types in the
original dataset [2] and in the cleaned dataset [5]. 80.49% of the examples
removed from the latter contain changes to Gitrepo files which are usually
very simple to learn and not very relevant for source code commit messages.

TABLE II: Top 10 file types sorted by frequency Dataset | Original [2] | Cleaned [5]
---|---|---
Rank | File type | Frequency | File type | Frequency
1 | Java | 4186 | Java | 4186
2 | Gitrepo | 3297 | Xml | 2097
3 | Xml | 2103 | Gradle | 1817
4 | Gradle | 1945 | Md | 1193
5 | Md | 1619 | Gitignore | 912
6 | Gitignore | 940 | Properties | 869
7 | Properties | 899 | Txt | 820
8 | Txt | 845 | Yml | 806
9 | Yml | 806 | Js | 337
10 | Js | 337 | Groovy | 297

Taking into account these observations, we explored how training a NMT model
on a specific file type influences its performance. In our exploration we
evaluated two scenarios. In the first one we select the top five most frequent
file types and create a dataset for each file type while in the second one we
select the top nine most frequent file types. In both scenarios we group the
rest of the files into a dataset which we will call ”others”.

The Gitrepo, Gradle, Java, Md and Xml represent 50.17% of the examples in the
dataset. The Gitignore, Gitrepo, Gradle, Java, Md, Properties, Txt, Xml and
Yml represent 63.48% of the examples in the dataset. The vocabulary for each
dataset was obtained following these steps:

  1. 1.

the words which do not appear in the initial vocabulary are filtered;

  2. 2.

the words which appear only once (twice for Gitrepo) in the diffs/messages
specific to the file types are filtered;

  3. 3.

the remaining words are selected to be part of the vocabulary.

In order to confirm that our approach is valid, we used kernel density
estimation (KDE) [16] to plot the density of words for the top three most
frequent file types (Gitrepo, Java, Xml). To obtain a relevant comparison we
selected the first 10,000 words from the diffs and messages. Figure 2 and
Figure 2 show that the distribution of the words in diffs and messages is
different for each file type.

Figure 1: Kernel Density Estimation for Gitrepo, Java, Xml diff files; Gitrepo
KDE is BLEU; Java KDE is orange; Xml KDE is green

Figure 2: Kernel Density Estimation for Gitrepo, Java, Xml messages files;
Gitrepo KDE is BLEU; Java KDE is orange; Xml KDE is green

###  III-C Sketch-based Neural Encoder

Inspired by the finding of the poor performances of NMT on examples containing
Java files, we propose our main contribution which is a sketch-based neural
model which achieves a better BLEU-4 score than NNGen. One of the reasons we
focus on examples containing Java files is that this file type is the only one
which contains actual source code. Another reason is that the dataset
introduced by Jiang el al. was created from the most popular Java repositories
on GitHub and any claim of improved performance for this dataset should also
report the score on examples containing Java file types.

The sketch-based solution consists of three parts. First, the dataset is
preprocessed. Constants, variables, functions, and class names are replaced
with placeholders. Then the new dataset with placeholders and smaller
vocabularies is trained. Lastly, the obtained messages are postprocessed and
the placeholders in messages are replaced with constants, variables,
functions, and class names extracted from diffs. An example of a prepossessed
diff can be found in Figure 3.

Figure 3: Example of a preprocessed diff using the encoding sketch

The preprocessing part relies on Java naming conventions in order to
distinguish between constants, variables, functions, and class names. First,
the Java keywords, annotations, imports, numbers, strings and comments from
the diff file are removed. Afterwards the following heuristics are used to
distinguish between constants, variables, functions, and class names:

  1. 1.

constants are written using only upper case letters

  2. 2.

classes start with an uppercase letter and continue with a combination of
upper and lower case letters

  3. 3.

functions start with a lower case and have an open bracket at the end of the
name

  4. 4.

variables start with a lower case but have no open bracket at the end of the
name

In the end, each constant, variable, function, and class name found in the
diff file are replaced with a specific placeholder. If any name found in the
diff file is found in the message, that name is also replaced with same
placeholder as in the diff file. A dictionary is created in order to save the
mapping from placeholders to the actual names.

Our sketch preprocesing part is based on the Java naming convention is
reasonable but risky. A more reliable solution would have been using an AST
tree to find the constants, variables, functions, and class names. Obtaining
the AST from a diff file can be challenging or maybe impossible that is why we
recommend using an AST tree in the future research but in our research we used
our rule-based preprocessing.

The postprocessing part searches for placeholders in the predicted messages
and replaces those placeholders with the mapping found in the dictionary
created during preprocessing. If no mapping is found, a random name is
selected from the names found in that diff. In case no name is found, the
placeholder is removed.

This method has two main advantages. First of all, it reduces the vocabulary
size by replacing constant, variable, function, and class names with
placeholders. Secondly, it enhances the similarity between parts of code which
otherwise would be very different because of the constant, variable, function,
and class names.

##  IV Experimental setup

In this section we briefly present the datasets, training setup, and
evaluation setup used during our experiments.

###  IV-A Datasets

Since the previous work on this topic was done on the dataset introduced by
Jiang et al. [2], we are going to conduct most of our experiments on the same
data. In order to be able to present a broader comparison with the state of
the art we are going to use the cleaned version of Jiang et al. dataset
introduced by Liu et al. [5]. In the next paragraphs we are going to present a
brief description of both datasets.

##### Jiang el al. dataset [2]

The dataset was created by collecting 2 million commits from 1,000 Java
repositories. The full list of projects used in creating the dataset can be
found online [17]. The commits that were rollbacks, merges or have empty
messages or non-English characters were removed. The diff files were obtained
using git diff command, but commits containing diff files that were larger
than 1 MB are filtered. In order to obtain messages with a consistent writing
style, only messages that start with dobj (direct object) dependency were
selected using Stanford CoreNLP[18] framework. From the remaining number of
commits, 26,000 are randomly selected for training, 3,000 are selected for
test, and 3,000 for validation. Figure 4 represents an example of a diff and a
reference message.

Figure 4: Example of a diff and reference message from [5]

##### Liu et al. dataset[5]

is a subset of the original Jiang et al. [2] dataset. In this paper, we refer
to this new dataset as the cleaned dataset as it removes simple examples from
the original set. Liu et al. analyzed the results obtained by Jiang et al.
[4]. They discovered that most of the high quality messages obtained using
Jiang et al. approach have high similarity with messages from the training set
and are not describing the actual changes (e.g. they are ”housekeeping”
messages with trivial changes to non source-code / Java files). After doing an
exhaustive search they identified a couple of trivial message patterns and
removed all messages which were fitting those patterns. The full list of
patterns can be found online [19]. However, as they acknowledged in their
paper, this approach did not clean all the trivial commit messages. This
dataset contains 22,112 examples in the training set, 2,511 examples in the
validation set, and 2,521 examples in testing set.

###  IV-B Training Setup

The training goal of our system is to minimize the categorical cross-entropy.
As training algorithm we use Adam with a learning rate of 0.0001. We used
sequence-to-sequence models in our work with RNN encoder and decoder. The
encoder and the decoder use GRU [20] units. Our models use Bahdanau
attention[8] for the attention mechanism. The size of the hidden units is 1024
and the size of the batches is 32. The models are trained between 5,000 -
30,000 steps depending on the number of layers of each specific model and the
size of the dataset. We use the same input sequence length (100) and output
sequence length (30) as Jiang et al. [2] as 99% of the examples in the
datasets have both the diffs and commit messages length less than these
thresholds.

The training was performed on two types of GPUs: 12GB GeForce GTX TITAN X and
12GB Tesla K40m.

###  IV-C Evaluation Setup

We evaluate the models based on the similarity between the target messages and
the generated messages. In order to evaluate the similarity between two
sentences we compute BLEU scores [9]. BLEU is widely used as a similarity
metric in machine translation [2, 10, 11, 8, 7] and various other language and
code generation tasks [14]. BLEU score is a number between 0 and 100 and is
calculated based on n-gram overlap. We decode our examples using beam search,
with widths of 1, 5, 10, and 10 with length penalty of 1. Beam search is
commonly used to maximize the scoring function for target sequences in neural
decoders using RNNs.

All our experiments were run five times and the average BLEU-4 score over all
runs is presented in the results section.

##  V Experiments and Discussion

###  V-A Baseline NMT Method

The experiments in this section were done on the initial dataset introduced by
Jiang et al. [2].

####  V-A1 Architecture exploration

As we stated before, we explore the following three verticals that may
increase the performance of our baseline: model depth, embedding
dimensionality, and residual connections.

##### Model Depth

We compare the results obtained with NMT models using 2, 4, and 8 layers. Our
results, presented in Table III, show that a larger model is able to encode
more information and to make better predictions. The problem of generating
commit messages out of diffs is difficult enough and the dataset contains
sufficient examples so the larger models do not overfit. All the models obtain
the best results using beam decoding of width 10 and length penalty 1. The
model with 8 layers (4 layers for encoder, 4 for decoder) obtains the best
BLEU-4 score (34.07).

TABLE III: Influence of model depth for NMT models Encoder | Decoder | BLEU-4
---|---|---
1 | 1 | 31.16
2 | 2 | 33.56
4 | 4 | 34.07

##### Embedding Dimensionality

We compare the results obtained with models with embedding size 1024 and
models with embedding size 512. We expected that a larger embedding is able to
capture a better mapping between the words and the vector space. The models
with embedding 512 have better performances than the others when the decoding
is done without beam search. However, when the decoding is done with beam
search the models with embedding 1024 have better performances in most of the
cases. In our opinion, the suspect that the model with 8 layers and 1024
embedding size was not fully trained at 30,000 steps and can achieve a higher
score than the model with 512 embedding. In general, models with 1024
embedding need more training steps than the models with 512 embedding. We also
did not observe overfitting with a larger embedding space. In conclusion, a
1024 embedding size is more suitable for converting diffs into commit messages
as can be seen in Table IV.

TABLE IV: Influence of embedding size for NMT models |  |  | BLEU-4
---|---|---|---
Encoder | Decoder | Embedding size | wo BS | w BS
1 | 1 | 512 | 29.98 | 30.05
1 | 1 | 1024 | 27.11 | 31.16
2 | 2 | 512 | 33.16 | 33.29
2 | 2 | 1024 | 32.77 | 33.56
4 | 4 | 512 | 33.41 | 34.17
4 | 4 | 1024 | 32.58 | 34.07

##### Residual Connections

We compare the results obtained with NMT models with residual connections and
vanilla NMT models. Residual connections help the gradients flow faster
through the model and we expected these models to have better performance. A
better gradient flow thorugh a neural model helps them to converge faster. Our
intuition is confirmed by the results. The models with residual connections
have on average an increase of 3 points in BLEU-4 score compared to the models
without residual connections (see Table V). The biggest impact of residual
connections is seen for the larger NMT models, with 8 layers, which is an
increase of 4.52 in BLEU-4 score.

TABLE V: Influence of residual connections for NMT models Encoder | Decoder | Residual | BLEU-4
---|---|---|---
2 | 2 | No | 33.56
2 | 2 | Yes | 35.35
4 | 4 | No | 34.07
4 | 4 | Yes | 38.45

In the next experiments we are going to use the following three models with an
embedding size of 1024: 1) NMT2 which is a two layer model 2) NMT4 which is a
4 layer model with residual connections, and 3) NMT8 which is a 8 layer model
with residual connections. The results obtained by the three models are
presented in Table VI and the last model can be considered as a strong NMT
baseline for commit message generation.

TABLE VI: Overview of three proposed NMT models Model | Type | BLEU-4
---|---|---
NMT2 | 2 layers, no residual | 31.16
NMT4 | 4 layers, residual | 35.35
NMT8 | 8 layers, residual | 38.45

####  V-A2 Rare and OOV Words

##### Byte Pair Encoding (BPE)

BPE preprocessing has a big impact on language translation. However, for our
problem BPE achieves poor performances and obtains a BLEU-4 score of 32.67
(vs. 38.45). The model trained with the biggest vocabulary size has the best
performances. Our BPE vocabulary contains a lot of words which are rare.
Because of the rarity of the words this preprocessing method has poor
performance on this dataset. BPE shows better performances on a dataset where
rare words are not so frequent. In the Jiang et al [2] dataset, out of 66,295
words in the joint target and source vocabulary, 29,941 appear only once which
is 45% of the total of words in the joint vocabulary. The results are
presented in Table VII.

TABLE VII: Influence of BPE for NMT models | Model | Vocabulary | BLEU-4
---|---|---|---
BPE1 | NMT4 | 5,000 | 26.65
BPE2 | NMT8 | 10,000 | 29.40
BPE3 | NMT8 | 32,000 | 32.67

##### Copying mechanism

As already mentioned, the dataset does not contain any OOV words. However, we
observed that 45% of the words in the vocabulary appear only once. This
encouraged us to reduce the vocabulary size and employ a copying mechanism for
words which became out of vocabulary after vocabulary reduction. We tested our
two configurations of vocabulary reduction on the three proposed models (NMT2,
NMT4, NMT8) and we present the results in Table VIII. Vocabulary reduction and
copying mechanism help models with fewer layers to achieve scores closer to
those obtained with more complex neural architectures. Models trained using
the second configuration, where we reduce both the messages and diffs
vocabularies, have performed better than models trained using the first one.
As we stated in the Proposed method section III, the first configuration
employs a modified messages vocabulary which contains words that appear at
least once and the second configuration employs vocabularies contain words
that appear at least ten times. This happens mainly for two reasons: 1) it
reduces the noise in the input sequence and the embedding layer can capture a
better mapping 2) reduces the number of parameters for the models. The NMT2
and NMT4 models with vocabulary reduction and copying obtain better
performances than the NMT2 and NMT4 models without. However, NMT8 without
vocabulary reduction and copying mechanism obtains a better performance. We
assume that NMT8 is able to extract some information from the noise due to the
fact of having more layers. In most of the cases, beam search with width 10
and penalty 1 with copying mechanism is the decoding algorithm that achieves
the best result. The impact of the copying mechanism is limited but it
increases the BLEU score in average by 0.22 in BLEU-4 score.

TABLE VIII: Copying mechanism and vocabulary reduction for NMT models Model | Configuration | BLEU-4
---|---|---
NMT2 | 1 | 33.01
NMT4 | 1 | 36.25
NMT8 | 1 | 36.87
NMT2 | 2 | 33.58
NMT4 | 2 | 37.36
NMT8 | 2 | 37.39

In this subsection we investigated RQ1 and we discovered a more suitable NMT
architecture which achieves a BLEU-4 score close to the current state of the
art (NMT8, residual, no copy, no BPE).

###  V-B Improved Ensemble NMT Model

In this section we present the results obtained by our improved ensemble
method. Our experimental results are presented in Table IX and were performed
on the original dataset. NMT2 is a two layer RNN architecture trained on the
entire dataset. NMT2-FT, NMT4-FT, NMT8-FT are the 2, 4, and 8 layer RNN
architectures trained on datasets which contain only examples specific to a
specific file type (FT).

We compare the results obtained by training on the nine file type datasets
(Gitignore, Gitrepo, Gradle, Java, Md, Properties, Txt, Xml, Yml) with the
results obtained with the model with two RNN layers (NMT2) on the entire
dataset.

We firstly observe that for Gitrepo files, both the model trained on the
entire dataset and the one trained only on this specific file type obtain
around 99 BLEU-4 score. This means that both approaches were able to solve
almost perfectly examples for Gitrepo files. The very high BLEU-4 score on
this file type is an outlier compared with the scores obtained on the other
file types. This score is artificially increasing the average BLEU-4 score on
the entire dataset and it contains only trivial, ”maintenance” commit
messages.

On the datasets containing Gradle, Java, Md, and Xml file types, the
performance of NMT2-FT is on average 161% higher than the performance of the
baseline NMT2. The increase in performance of Gitignore, Properties, Txt is a
bit lower (117%). This is mostly because of the small number of examples for
these datasets. On the dataset containing Yml examples NMT2 is not able to
learn, but NMT2-FT is able to achieve a BLEU-4 score of 6.42.

We observe that deeper models (NMT4-FT and NMT8-FT) have better performance.
However, the increase of performance is limited compared to the increase of
performance in the case of training on the entire dataset. On average a four
layer model increases the performance with 2.5% and a eight layer model with
6.9%.

In comparison with NNGen, NMT2-FT has similar performances on examples
containing Gitrepo and Gradle files, a slightly better one on Gitignore, Md,
Properties, Xml files and a slightly worse on Yml files. However, NNGen is
still outperforming our ensemble approach on Java and Txt files by a high
margin (13.1 vs. 9.06, 22.57 vs. 8.11).

TABLE IX: Ensemble NMT model results by file type | NNGen[5] | NMT2 | NMT2-FT | NMT4-FT | NMT8-FT
---|---|---|---|---|---
Gitignore | 5.47 | 2.48 | 6.2 | - | -
Gitrepo | 100 | 99.64 | 99.21 | - | -
Gradle | 25.11 | 11.29 | 25.15 | 25.51 | 26.76
Java | 13.1 | 2.04 | 9.06 | 9.38 | 9.70
Md | 18.96 | 11.52 | 21.46 | - | -
Properties | 19.26 | 9.05 | 21.33 | - | -
Txt | 22.57 | 8.11 | 13.69 | - | -
Xml | 22.43 | 7.14 | 26.46 | 27.18 | 28.4
Yml | 8.69 | 0 | 6.85 | - | -

In Table X we present the results obtained with the assembles of NMT2-FT,
NMT4-FT and NMT8-7 in both scenarios in comparison with NNGen, NMT2 and NMT8.
NMT4-FT-S1 and NMT4-FT-S2 are ensembles containing NMT models with four layers
trained on the Gradle, Java, Xml and Others datasets and models with two
layers for the rest of the datasets. Similarly, NMT8-FT-S1 and NMT8-FT-S2
contain eight layers models trained on the Gradle, Java, Xml and Others
datasets and models with two layers for the rest of the datasets. The
description of the scenarios can be found in the following subsection III-C.

The ensemble of NMT2-FT models is slightly outperformed by NNGen on both
scenarios (38.55 vs. 38.33, 38.44). Ensembles of NMT4-FT and NMT8-FT models
were able to outperform NNGen due to the performance increase on Gradle, Java
and Xml datasets. However, on a more thorough analysis NNGen still outperforms
our approach for Java source files.

The performance of the ensemble containing all the NMT2-FT models in the first
scenarios (NMT2-FT-S1) is with 7.17 greater in BLEU-4 score than the
performance of the NMT2 and with 7.28 in the second scenario (NMT2-FT-S2). The
score obtained with NMT8 (8 layer model) is a bit better than the one obtained
with NMT2-FT-S1 (38.44 vs. 38.33) and similar to the one obtained with
NMT2-FT-S2 (38.44). We believe that NMT8 is able to separate the word
distributions by file type due to its larger number of layers.

In both scenarios, the ensemble containing NMT8-FT is achieving a BLEU-4 score
of 39.01. It is fair to say that training Gitignore, Properties, Txt and Yml
separately does not boost the performance. This is mainly due to the reduced
size of those datasets. However, if the dataset size is large enough we would
recommend training examples containing diffs with similar file type
separately.

TABLE X: Ensemble models compared to state of the art Approach | BLEU-4
---|---
NNGen[5] | 38.55
NMT2 | 31.16
NMT8 | 38.44
NMT2-FT-S1 | 38.33
NMT2-FT-S2 | 38.44
NMT4-FT-S1 | 38.65
NMT4-FT-S2 | 38.69
NMT8-FT-S1 | 39.01
NMT8-FT-S2 | 39.01

The experiments in this section revealed that training on a specific file type
improves overall performance (RQ2). Most of the ensembles outperformed the
state of the art on the original dataset.

###  V-C Sketch-based Neural Encoder

We trained our sketch-based method on three different architectures. Table XI
presents the results in BLEU-4 score for NNGen, NMT8-FT, and our sketch-based
method using models with 2 (NMT2-JT), 4 (NMT4-JT) and 8 (NMT8-JT) layers. The
sketch-based method increases the performance of NMT models by 40% for Java
files. It also reduces the complexity of the problem and models with fewer
layers have similar or even better performances than deeper models. It is
interesting that the model with 2 layers can achieve a better BLEU-4 score
than the model with 4 layers. However, in average across all prediction
strategies (without beam, beam 5, beam 10, beam 10 with length penalty 1)
NMT4-JT archives a score of 13.38 and NMT2-JT achieves a score of 13.32. The
score obtained with NMT8-JT is better with 4% than the score in BLEU-4
obtained with NNGen on diffs containing Java files.

TABLE XI: Sketch-based NMT model for Java files Approach | BLEU-4
---|---
NNGen[5] | 13.1
NMT8-FT | 9.7
NMT2-JT | 13.6
NMT4-JT | 13.47
NMT8-JT | 13.64

In this section we showed the positive impact of the sketch-based encoder for
NMT models (RQ3). We improved our score by a high margin and outperformed the
state of the art on Java files. At this moment, NMT sketch-based models
achieve better results both overall, for all file types in the dataset, and
for Java files alone.

##  VI Comparison with state of the art

In this section we compare the results obtained with our eight layer model
trained on the entire dataset (NMT8), our ensemble of models trained on
specific file types (NMT8-FT) and the same ensemble but with Java sketch-based
approach (NMT8-FT-JT) with the current state of the art.

The first, third, forth and fifth row in Table XII show how our improved
approaches are outperforming by a high margin the state of the art in
generating commit messages from diffs using vanilla NMT techniques[4] on both
the initial and the cleaned dataset.

Our file type (NMT8-FT) and Java sketch (NMT8-FT-JT) approaches have better
results that the NNGen approach introduced by Liu el al. [5] and even our
baseline model with 8 layers (NMT8) has similar results with NNGen on the
original dataset. However, on the cleaned one only our Java sketch (NMT8-FT-
JT) is outperforming NNGen.

Although NNGen has good results on these two datasets, we believe the nearest
neighbour approach will lack robustness on larger and more difficult datasets.
We also highlight the fact that the reduced number of examples in our file
type datasets has a negative impact on the performance of our NMT approaches.
Thus, the performance gap between our proposed NMT sketch-based ensemble
approach and NNGen will probably increase on larger and more complex datasets.

Another important aspect when we compare our results with the state of the art
is the run-time performance of commit message generation. In comparison with
Liu et al.[5] which take into account the training time and the prediction
time during during speed evaluation, we compare the average prediction time on
the test dataset because it is the most relevant metric for production run-
time. The results can be found in Table XII. Our approaches are 27-35x slower
than the NNGen [5] but are still fast enough to be used in production. Our
file type (NMT8-FT) approach is faster than our eight layer model trained on
the entire dataset (NMT8) with about 50ms. This speed boost is mainly due to
the vocabulary reduction on each file type. Our Java sketch (NMT8-FT-JT) is
bit slower than our file type (NMT8-FT) but the boost in performance is
significant.

TABLE XII: Comparison with state of the art Dataset | Approach | BLEU-4 | Avg Time
---|---|---|---
Original | NMT[4] | 31.92 | 340ms[5]
| NNGen[5] | 38.55 | 10ms[5]
| Our NMT8 | 38.44 | 323ms
| Our NMT8-FT | 39.01 | 256ms
| Our NMT8-FT-JT | 39.54 | 271ms
Cleaned | NMT[4] | 14.19 | 309ms[5]
| NNGen[5] | 16.42 | 9ms[5]
| Our NMT8 | 16.07 | 314ms
| Our NMT8-FT | 16.07 | 264ms
| Our NMT8-FT-JT | 16.86 | 281ms

##  VII Human evaluation

In this section we present the human evaluation we conducted for assessing the
quality of the generated commit messages. The goals of this section are to
offer a more in-depth evaluation of our results and to confirm that our
solution provides a real improvement.

We evaluate the results obtained by the sketch-based NMT model described in
section V-C and which obtains the best quantitative BLEU-4 scores on both
datasets. Our evaluation is conducted on all 436 test examples which contain
diffs of Java files. We believe and strongly encourage that future evaluations
should be done explicitly on this file type. We use a similar survey design as
Jiang et al. [4] with a score range between 0-7 where 0 means no similarity
and 7 means identical. The commit messages were evaluated by 18 professionals,
all of them have majored in computer science. Each of them evaluated between
36 to 50 messages according to the three examples introduced by Jiang et al.
[4]. An example is presented in Figure 5.

Figure 5: A scoring example introduced by Jiang et al.[4]

We group the mean scores into three groups: 1) low quality (0 - 2.33] 2)
medium quality (2.33 - 4.66] 3) high quality (4.66 - 7]. Unsurprisingly, 55.73
% of the generated commit messages were low quality, 26.16 % medium quality,
and 18.11 % high quality. Cohen’s Kappa coefficient between raters was 0.346
which suggests there was a fair agreement between them. This fair inter-rater
agreement score also shows that assessing the quality of commit messages is
difficult for human professionals as well.

However, the average score obtained by our generated messages is 2.40 on a 0-7
scale which is 1.50 on a 0-4 scale. This average score is slightly higher than
the score obtained by Liu et al. [5] on the cleaned dataset (1.46). Taking
into consideration that Java file type is one of the most complex file types
in the cleaned dataset, we are confident to say, according to our human
evaluation, that our approach outperforms his approach on examples containing
diffs of Java files and on the entire dataset. This analysis should take into
account that the raters are different in the two studies. However, our study
uses a large number of persons (18 professionals, graduates in computer
science) to eliminate potential outlier scores.

In figure 6 we present three examples of generated messages. The first one was
evaluated as a good quality message and the next two were evaluated as low
quality messages. In the first example, the NMT model was able to predict a
message similar with the target message but with a grammatical error. In the
second example, the model was able to partially understand the changes. The
generated message was categorized as low quality because the evaluators did
not have access to the diff files during evaluation. In the last example, we
present a situation where our model failed to generate a meaningful message.
However, it is clear that our sketch method outperforms NNGen on the first two
examples and has a similar performance on the third one.

Example 1:

Reference Message: Fixed problem with getHighestFnScope method
Message Generated by NNGen: Remove fflush from hdr_percentile_print.
Message Generated by NMT8-JT: Fix a bug with getHighestFnScope().

Example 2:

Reference Message: make package protected, so other apps do not link it
Message Generated by NNGen: add function with zero arguments
Message Generated by NMT8-JT: add comments

Example 3:

Reference Message: add not null
Message Generated by NNGen: set visible pack to empty in vcs log ui in
dispose IDEA - 120539
Message Generated by NMT8-JT: JMS now also enlists topic destinations
as managed endpoints in JMX

Figure 6: Examples of generated and the reference messages

##  VIII Conclusions

During our work we were able to show how NMT techniques are able to improve
the generation of comments for code commits and can achieve state of the art
results. Our sketch-based ensemble model is outperforming all existing
solutions on both datasets used to assess the quality of generated commit
messages for this task.

One of our main contributions is the approach of splitting the dataset in
smaller datasets by file type and training them separately in an ensemble of
NMT models. We are confident that an increase of the number of the examples
for each specific file type dataset would have a positive impact in
performance of the overall system.

Inspired by the NMT’s poor performance on Java files, we also propose a
sketch-based method which is able to boost the performance on Java types by a
high margin and outperform NNGen - the top scoring solution for Java files at
this moment.

We believe that the problem of generating messages for code commits is more
challenging and of a somehow different nature than the problem of translating
text from one natural language to another. However, the current datasets for
commit message generation are 100 times smaller than the datasets used in
machine translation (e.g. WMT’14 English-to-French, WMT’14 English-to-German).
A larger dataset is a must in order to ensure progress on this task. We
believe that NMT models did not achieve their full potential on this task and
will perform even better on a larger dataset.

Diff files contain some context of the changes, but that context contains
limited information. A future improvement might likely target adding more
context to training process. One of the challenges of this approach will be
choosing the right size for the context. At the end, we view our work as a
step forward for advancing the state of the art, but current approaches are
far from being able to be used in the industry.

## References

  * [1] R. P. Buse and W. R. Weimer, “Automatically documenting program changes,” in _Proceedings of the IEEE/ACM International Conference on Automated Software Engineering_ , ser. ASE ’10. New York, NY, USA: ACM, 2010, pp. 33–42. [Online]. Available: http://doi.acm.org/10.1145/1858996.1859005
  * [2] S. Jiang and C. McMillan, “Towards automatic generation of short summaries of commits,” _CoRR_ , vol. abs/1703.09603, 2017. [Online]. Available: http://arxiv.org/abs/1703.09603
  * [3] M. Linares-Vásquez, L. F. Cortés-Coy, J. Aponte, and D. Poshyvanyk, “Changescribe: A tool for automatically generating commit messages,” in _2015 IEEE/ACM 37th IEEE International Conference on Software Engineering_ , vol. 2, May 2015, pp. 709–712.
  * [4] S. Jiang, A. Armaly, and C. McMillan, “Automatically generating commit messages from diffs using neural machine translation,” _CoRR_ , vol. abs/1708.09492, 2017. [Online]. Available: http://arxiv.org/abs/1708.09492
  * [5] Z. Liu, X. Xia, A. E. Hassan, D. Lo, Z. Xing, and X. Wang, “Neural-machine-translation-based commit message generation: how far are we?” in _ASE_ , 2018.
  * [6] R. Sennrich, B. Haddow, and A. Birch, “Neural machine translation of rare words with subword units,” _CoRR_ , vol. abs/1508.07909, 2015. [Online]. Available: http://arxiv.org/abs/1508.07909
  * [7] J. Gu, Z. Lu, H. Li, and V. O. K. Li, “Incorporating copying mechanism in sequence-to-sequence learning,” _CoRR_ , vol. abs/1603.06393, 2016. [Online]. Available: http://arxiv.org/abs/1603.06393
  * [8] D. Bahdanau, K. Cho, and Y. Bengio, “Neural machine translation by jointly learning to align and translate,” _CoRR_ , vol. abs/1409.0473, 2014. [Online]. Available: http://arxiv.org/abs/1409.0473
  * [9] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: A method for automatic evaluation of machine translation,” in _Proceedings of the 40th Annual Meeting on Association for Computational Linguistics_ , ser. ACL ’02. Stroudsburg, PA, USA: Association for Computational Linguistics, 2002, pp. 311–318. [Online]. Available: https://doi.org/10.3115/1073083.1073135
  * [10] D. Britz, A. Goldie, M. Luong, and Q. V. Le, “Massive exploration of neural machine translation architectures,” _CoRR_ , vol. abs/1703.03906, 2017. [Online]. Available: http://arxiv.org/abs/1703.03906
  * [11] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, J. Klingner, A. Shah, M. Johnson, X. Liu, L. Kaiser, S. Gouws, Y. Kato, T. Kudo, H. Kazawa, K. Stevens, G. Kurian, N. Patil, W. Wang, C. Young, J. Smith, J. Riesa, A. Rudnick, O. Vinyals, G. Corrado, M. Hughes, and J. Dean, “Google’s neural machine translation system: Bridging the gap between human and machine translation,” _CoRR_ , vol. abs/1609.08144, 2016. [Online]. Available: http://arxiv.org/abs/1609.08144
  * [12] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in _Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2_ , ser. NIPS’14. Cambridge, MA, USA: MIT Press, 2014, pp. 3104–3112. [Online]. Available: http://dl.acm.org/citation.cfm?id=2969033.2969173
  * [13] L. Dong and M. Lapata, “Coarse-to-fine decoding for neural semantic parsing,” _CoRR_ , vol. abs/1805.04793, 2018. [Online]. Available: http://arxiv.org/abs/1805.04793
  * [14] I. A. Hosu, R. C. A. Iacob, F. Brad, S. Ruseti, and T. Rebedea, “Natural language interface for databases using a dual-encoder model,” in _Proceedings of the 27th International Conference on Computational Linguistics_. Santa Fe, New Mexico, USA: Association for Computational Linguistics, Aug. 2018, pp. 514–524. [Online]. Available: https://www.aclweb.org/anthology/C18-1043
  * [15] K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” _CoRR_ , vol. abs/1512.03385, 2015. [Online]. Available: http://arxiv.org/abs/1512.03385
  * [16] D. W. Scott, _Kernel Density Estimation_. American Cancer Society, 2018, pp. 1–7. [Online]. Available: https://onlinelibrary.wiley.com/doi/abs/10.1002/9781118445112.stat07186.pub2
  * [17] S. Jiang, “Commitgen: Generating commit messages using neural machine translation,” 2017. [Online]. Available: https://sjiang1.github.io/commitgen
  * [18] C. D. Manning, M. Surdeanu, J. Bauer, J. Finkel, S. J. Bethard, and D. McClosky, “The Stanford CoreNLP natural language processing toolkit,” in _Association for Computational Linguistics (ACL) System Demonstrations_ , 2014, pp. 55–60. [Online]. Available: http://www.aclweb.org/anthology/P/P14/P14-5010
  * [19] Z. Liu, “Liu et al.’s appendix.” 2018. [Online]. Available: https://goo.gl/63B976
  * [20] K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio, “Learning phrase representations using RNN encoder–decoder for statistical machine translation,” in _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_. Doha, Qatar: Association for Computational Linguistics, Oct. 2014, pp. 1724–1734. [Online]. Available: https://www.aclweb.org/anthology/D14-1179