#  Kink properties in Lorentz-violating scalar field theory

Reza Moazzemi<EMAIL_ADDRESS>Mohammad Mehdi Ettefaghi
<EMAIL_ADDRESS>Amirhosein Mojavezi<EMAIL_ADDRESS>Department of
Physics, University of Qom, Ghadir Blvd., Qom 371614-6611, I.R. Iran

###### Abstract

We consider topological defects for the $\lambda\phi^{4}$ theory in (1+1)
dimensions with a Lorentz-violating background. It has been shown, by M.
Barreto et al. (2006) [34], one cannot have original effects in (the leading
order of) single scalar field model. Here, we introduce a new Lorentz-
violating term, next to leading order which cannot be absorbed by any
redefinition of the scalar field or coordinates. Our term is the lowest order
term which leads to concrete effects on the kink properties. We calculate the
corrections to the kink shape and the corresponding mass. Quantization of the
kink is performed and the revised modes are obtained. We find the bound and
continuum states are affected due to this Lorentz symmetry violation.

###### keywords:

Lorentz symmetry violation - scalar field - kink - solitons

††journal: Nuclear Physics B

##  1 Introduction

Nonlinear field models are very interesting in various areas of physics
including particle physics, condensed matter physics, cosmology and optics [1,
2]. Solitons and solitary waves that have localized energy and propagate
without losing the shape are important solutions of some nonlinear field
equations. We can cite the following examples of solitons; vertices in
superconductors [3] and fluids [4], cosmic strings [5], monopoles [6] and
instantons [7]. There are also many works on supersymetric solitons, see for
example [8, 9, 10, 11]. The kinks which appear in (1+1)-dimensional models
with a potential possessing two or more degenerated minima, are the simplest
examples of solitons [12]. Some realistic models in (3+1) or (2+1) dimensions
can be reduced to the effective (1+1)-dimensional dynamics. In some theories
such as $\lambda\phi^{4}$, the existence of kink in higher dimensions is
banned [13]. Actually, to construct a modified kink in such a case we should
interpolate from field theory in the Minkowski space-time to field theory in
curved space-time background [14].

One of the most important symmetries which should be considered in theories
for high energy physics is the Lorentz symmetry. So far, it has not been
observed any deviation of this symmetry by experimental inspections. However,
Lorentz symmetry may be violated due to the quantum gravity in the Plank scale
[15]. Noncommutative theories also constitute an essential motivation for
study of Lorentz-breaking theories and involve a Lorentz-breaking tensor
$\Theta^{\mu\nu}$ [16, 17]. Therefore, it is interesting to restudy a Lorentz
covariant theory when a small violation in Lorentz symmetry is allowed. The
standard model and QED, for instance, were extended by the LV terms [18]. The
Casimir effect in an LV theory is also considered in some works, see for
example [19, 20, 21].

So far, various issues and problems have been studied within the LV extended
standard model framework (for instance see [22] and references therein). This
investigations have been done with basically a twofold purpose: the
determination of new physical effects induced by the Lorentz-violating
coefficients and the imposition of stringent upper bounds on the LV
coefficients that constrain the magnitude of Lorentz breaking. Our goal in
this paper is to study the former. Such studies have been done in several
papers such as Refs. [23, 24, 25]. In Ref. [23], authors have investigated the
static limit of Lorentz-violating electrodynamics in vacuum and in media and
obtained explicit solutions for some simple cases. More details of this theory
have been investigated within the Refs. [23, 24] and classical solutions are
also properly examined [26, 27].

In addition, LV effects on topological defects are considered e.g. in Refs.
[28, 29, 30, 31] and, in particular, for scalar field theories in Refs. [32,
33, 34]. In Ref. [34], authors considered two models in (1+1) dimensions (with
one and two scalar fields) in which the Lorentz and CPT symmetries are
violated. Actually, in the case of a single scalar field, the LV term, modeled
by $-\frac{1}{2}k^{\mu\nu}\partial_{\mu}\phi\partial_{\nu}\phi$, where
$k^{\mu\nu}$ is LV parameter, can be removed through a linear transformation
of coordinates, such a transformation will generate a Jacobian multiplier in
the new action. In fact, one can imagine $k^{\mu\nu}=cu^{\mu}u^{\nu}$
characterized by the LV vector $u^{\mu}$ which is a spacelike two-vector
denoting the aether [35, 36]. Therefore, in (1+1) dimensions, we can say that
there does not exist any observable effect due to the leading order of this
CPT even Lorentz violating interaction of scalar field to the aether.
Therefore, the nontrivial contributions due to this term appear when we couple
the model with more sophisticated fields. However, the effects of higher order
LV terms have not been yet considered. One may naturally expect that there
must exist some effects due to more complicated LV interacting terms. In this
paper, we investigate the unremovable LV effects that occurred in the next to
leading order. Explicitly, we study the LV interacting scalar field theory
modeled by $-\frac{1}{2}c\phi^{2}(u\cdot\partial\phi)^{2}$ in (1+1)
dimensions. This is the lowest order term which leads to concrete effects on
the kink properties. We discuss two different forms of LV terms $cu^{\mu}$,
time-like and space-like, i.e. Lorentz symmetry breaking in the direction of
time and space, respectively. For the case of the time-like vector, we see
there is no concrete effect. Therefore, we study only the space-like LV
vector.

After a very brief review of the standard kink in Sec. 2, we first, find the
modification to the shape of the kink in subsection 2.1 and then calculate the
mass correction due to this LV term. The stability equation is obtained in
Sec. 3a and based on this new corrected stability equation we find the
modification of bound and continuum states. The last section is devoted to
summarizing the results and conclusions.

##  2 Classical kinks in Lorentz violating theory

Here we first, very briefly, explain the standard kink and derive its mass.
The appropriate Lagrangian for spontaneously broken symmetry in 1+1 dimensions
within the real scalar $\phi^{4}$ theory is:

${\cal
L}(x)=\frac{1}{2}(\partial_{t}\phi)^{2}-\frac{1}{2}(\partial_{x}\phi)^{2}-\frac{\lambda}{4}\left(\phi^{2}-\frac{m^{2}}{\lambda}\right)^{2}.$
(1)

By making the following scaling:

$\phi\rightarrow\frac{\sqrt{m}}{\lambda}\phi,\hskip
28.45274ptx\rightarrow\frac{x}{m},\hskip 28.45274pt{\cal
L}\rightarrow\frac{m^{4}}{\lambda}{\cal L},$ (2)

the dimensionless Lagrangian is achieved as follows:

${\cal
L}(x)=\frac{1}{2}(\partial_{t}\phi)^{2}-\frac{1}{2}(\partial_{x}\phi)^{2}-\frac{1}{4}(\phi^{2}-1)^{2}.$
(3)

Equation of motion for Lagrangian (3) reads as

$\partial^{2}_{t}\phi-\partial^{2}_{x}\phi-\phi=-\phi^{3}.$ (4)

This equation has four static solutions: two trivial non-topological solutions
$\phi_{\rm{tri.}}(x)=\pm 1$, and two topological ones (kink and anti-kink):

$\phi_{\rm{k}}(x)=\pm\tanh\left(\frac{x-x_{0}}{\sqrt{2}}\right),$ (5)

which approach two different values as $x\to\pm\infty$ corresponding to the
two possible vacuum states. (For more details see please [37]).
Putting the solutions (5) in the related Hamiltonian

$H=\int\left[\frac{1}{2}(\partial_{t}\phi)^{2}+\frac{1}{2}(\partial_{x}\phi)^{2}+\frac{1}{4}(\phi^{2}-1)^{2}\right]dx,$
(6)

one can find the familiar classical mass of the kink as

$M_{\rm{cl.}}=\frac{2\sqrt{2}}{3}.$ (7)

###  2.1 $\phi^{4}$ theory with Lorentz symmetry violation

In order to have an LV theory (with preserving the other favorite symmetries
such as parity and $Z_{2}$), one may add the simplest possible term
$-\frac{1}{2}k^{\mu\nu}\partial_{\mu}\phi\partial_{\nu}\phi$ to the standard
Lagrangian (3). However, as we stated in the Introduction, this term gives no
real manifestation of Lorentz violation. Note that, since the existence of the
kink owes the interacting term in the Lagrangian, we expect an LV term in the
interaction part might affect the kink directly. Therefore, we introduce a
model in which a bit more complicated LV term is added to the Lagrangian, as
follows:

${\cal
L}(x)=\frac{1}{2}(\partial_{t}\phi)^{2}-\frac{1}{2}(\partial_{x}\phi)^{2}-\frac{1}{2}c\phi^{2}(u\cdot\partial\phi)^{2}-\frac{1}{4}(\phi^{2}-1)^{2},$
(8)

where the vector $cu^{\mu}$ manifests a preferred direction in the space-time
which is responsible for Lorentz symmetry violation. Here, we consider two
different types of Lorentz symmetry violation; first, a violation occurs in
the direction of time, called time-like (TL), and second in the direction of
space, called space-like (SL).

####  2.1.1 TL vector case

In the case of the TL vector, $u^{\mu}=(1,0)$, the third term in (8) is
$\frac{1}{2}c\phi^{2}(\partial_{t}\phi)^{2}$. Hence, the equation of motion
(4) is modified as:

$\partial^{2}_{t}\phi-\partial^{2}_{x}\phi-c\phi(\partial_{t}\phi)^{2}-c\phi^{2}\partial^{2}_{t}\phi-\phi=-\phi^{3}.$
(9)

It is obvious that the static solutions of this differential equation are not
different with (4). Accordingly, Lorentz violation in the time direction has
no effect on the kink properties.

####  2.1.2 SL vector case

In the SL case, $u^{\mu}=(0,1)$ the LV term in (8) becomes
$-\frac{1}{2}c\phi^{2}(\partial_{x}\phi)^{2}$ and equation (4) changes to

$\partial^{2}_{t}\phi-\partial^{2}_{x}\phi-c\phi(\partial_{x}\phi)^{2}-c\phi^{2}\partial^{2}_{x}\phi-\phi=-\phi^{3}.$
(10)

One can solve this differential equation using the perturbation method, i.e.
substituting
$\phi_{\rm{k}}(x)=\phi^{(0)}_{\rm{k}}(x)+c\,\phi^{(1)}_{\rm{k}}(x)+c^{2}\phi^{(2)}_{\rm{k}}(x)+\dots$
in Eq. (10), the solutions are derived as

$\displaystyle\phi_{\rm{k}}(x)$ $\displaystyle=$
$\displaystyle\phi^{(0)}_{\rm{k}}(x)+c\,\phi^{(1)}_{\rm{k}}(x)$ (11)
$\displaystyle=$
$\displaystyle\pm\bigg{\\{}\tanh\left(\frac{x-x_{0}}{\sqrt{2}}\right)$
$\displaystyle\hskip 28.45274pt-c\
\frac{1}{2}{\rm{sech}}\left(\frac{x-x_{0}}{\sqrt{2}}\right)\left[\frac{x-x_{0}}{\sqrt{2}}-\tanh\left(\frac{x-x_{0}}{\sqrt{2}}\right)\right]\bigg{\\}},$

where we retain terms up to order $c$. In Fig. 1 the correction term due to
the Lorentz violation is plotted. The related Hamiltonian for this case is

$H=\int\left[\frac{1}{2}(\partial_{t}\phi)^{2}+\frac{1}{2}(\partial_{x}\phi)^{2}+\frac{1}{2}c\phi^{2}(\partial_{x}\phi)^{2}+\frac{1}{4}(\phi^{2}-1)^{2}\right]dx.$
(12)

Inserting (11) in this Hamiltonian, we obtain

$M_{\rm{cl.}}=\frac{2\sqrt{2}}{3}+\frac{\sqrt{2}}{15}c+{{\cal{O}}(c^{2})}.$
(13)

This is the classical mass of the kink in the LV theory up to the first order
of Lorentz violation parameter $c$ .

Figure 1:  First order correction to the kink function due to Lorentz
violation

##  3 Quantizing the Kinks

As we know, the trivial vacuum and kink are classical static solutions of a
nonlinear relativistic field equation. In principle, they can be associated
with quantum extended particle states. Therefore, we are now going to study
the relevance of them to the corresponding quantum field theory. To do this,
it may be thought that one needs either the Feynman path-integral or the
canonical quantization methods. However, another simple method which is used
in the literature is an approach through which one expands the energy
corresponding to the static states via a semiclassical method. Then, using the
classical solutions one can obtain a little information about the quantum
behavior of the system. This semiclassical quantization technique has been
firstly introduced for relativistic field theories, using an appropriate
generalization of the WKB approximation in quantum mechanics by Dashen,
Hasslacher and Neveu [38, 39]. For future use, here, we very briefly review
this approach and some of its conclusions.

The energy of static states involves only the potential energy which can be
expanded about a minimum as follows:

$V[\phi]=V[\phi_{0}]+\int
dx\frac{1}{2}\left[\zeta(x)\left(-\nabla^{2}+\left.\frac{d^{2}{\cal
U}}{d\phi^{2}}\right|_{\phi_{0}(x)}\right)\zeta(x)\right]+\dots,$ (14)

where $\zeta(x)=\phi(x)-\phi_{0}(x)$ and $\cal U$ is the interacting potential
density along with the mass term. In fact, this is functional Taylor expansion
of $V[\phi(x)]$ about the static solution $\phi_{0}(x)$. In Eq. (14),
integration by part has been used and dots represent cubic and higher terms
which can be treated by perturbation. This method can be used when the weak
coupling approximation is established. The first term in Eq. (14) is related
to classical contribution in energy. The second term is the leading order
quantum correction which is similar to the harmonic oscillator with well-known
quantum physics. We can find the normal modes by solving the following
stability equation:

$\left(-\nabla^{2}+\left.\frac{d^{2}{\cal
U}}{d\phi^{2}}\right|_{\phi_{0}(x)}\right)\zeta_{n}(x)=\omega^{2}_{n}\zeta_{n}(x).$
(15)

Now, one can construct the energy spectrum of a static state corrected by
quantum fluctuations as follows:

$E_{\\{n_{i}\\}}=V[\phi_{0}]+\sum_{i}\left(n_{i}+\frac{1}{2}\right)\omega_{i},$
(16)

where $n_{i}$ is the excitation number of the $i$th normal mode. Using the
Lagrangian Eq. (3), the Eq. (15) reduces to

$\bigg{[}-\partial^{2}_{x}-1+3\phi_{0}^{2}(x)\bigg{]}\zeta_{n}(x)=\omega_{n}^{2}\zeta_{n}(x).$
(17)

For the trivial vacuum $\phi_{0}=\phi_{\rm{tri.}}=\pm 1$, this equation has
plane wave solutions $e^{\pm ikx}$ on which we can impose the Periodic
Boundary Condition (PBC) in a box with length $L$. Therefore,
$\omega_{n}^{2}=k_{n}^{2}+2$ with $k_{n}=2n\pi/L$, and the energy becomes

$E_{\\{n_{i}\\}}^{\rm{trv.}}=\sum_{i=-\infty}^{+\infty}\left(n_{i}+\frac{1}{2}\right)\sqrt{k_{i}^{2}+2}.$
(18)

For the
$\phi_{0}=\phi_{\rm{k}}=\pm\tanh\left(\frac{x-x_{0}}{\sqrt{2}}\right)$, the
Eq. (17) becomes

$\bigg{[}-\partial^{2}_{x}-1+3\tanh^{2}\left(\frac{x}{\sqrt{2}}\right)\bigg{]}\zeta_{n}(x)=\omega_{n}^{2}\zeta_{n}(x),$
(19)

where we choose $x_{0}=0$. The Eq. (19) has two bound state solutions
(regardless of normalization factors)

$\left\\{\begin{array}[]{cc}\zeta_{0}(x)=\frac{1}{\cosh^{2}(x/\sqrt{2})}&\mbox{with}\quad\omega_{0}=0,\\\
\zeta_{1}(x)=\frac{\sinh(x/\sqrt{2})}{\cosh^{2}(x/\sqrt{2})}&\qquad\mbox{with}\quad\omega_{1}=\sqrt{2/3},\end{array}\right.$
(20)

followed by a continuum

$\begin{array}[]{cc}&\zeta_{q}(x)=e^{iqx/\sqrt{2}}\left[3\tanh^{2}(x/\sqrt{2})-1-q^{2}-3iq\tanh(x/\sqrt{2})\right]\\\
&\quad\mbox{with}\qquad\omega_{q}=\sqrt{2+q^{2}/2}.\end{array}$ (21)

The zero mode is corresponding to the translational invariance of the
potential. The asymptotic behavior of $\zeta_{q}(x)$ at $\pm\infty$ is
$\exp[iqx/\sqrt{2}\pm\delta(q)/2]$ where
$\delta(q)=-2\tan^{-1}\left(\frac{3q}{2-q^{2}}\right)$ is the phase shift of
scattering states. Again, the PBC gives the allowed mumenta

$q_{n}L/\sqrt{2}+\delta(q_{n})=2n\pi,$ (22)

and the energy spectrum is

$E_{\\{n_{1},n_{q_{i}}\\}}=\frac{2\sqrt{2}}{3}+\left(n_{1}+\frac{1}{2}\right)\sqrt{\frac{3}{2}}+\sum_{q_{i}}\left(n_{q_{i}}+\frac{1}{2}\right)\sqrt{2+\frac{q^{2}_{i}}{2}}\
.$ (23)

For more details please see [37], for instance.

###  3.1 Excitations of the trivial vacuum

In this subsection we consider the quantization of the trivial solution in the
LV theory. For this purpose, using Lagrangian (8) we expand the potential
about one of its trivial vacuum. Similar to Eq. (14), the functional expansion
up to the second order of $\zeta$ is written as follows:

$V[\phi]=V[\phi_{\rm{tri.}}]+\int
dx\left[\frac{1}{2}\zeta(x)\left(-(1+c)\partial^{2}_{x}+2\right)\zeta(x)\right]+\dots.$
(24)

Accordingly, the eigenvalues in lowest-order quadratic terms are obtained from
this equation

$\left[-(1+c)\partial^{2}_{x}+2\right]\zeta_{n}(x)=\omega_{n}^{2}\zeta_{n}(x).$
(25)

This equation has plane wave solutions $e^{\pm ikx}$ with
$\omega_{n}^{2}=(1+c)k_{n}^{2}+2$. After imposing the PBC, we can write the
energy as

$E_{\\{n_{i}\\}}=\sum_{i}\left(n_{i}+\frac{1}{2}\right)\omega_{i}=\sum_{i=-\infty}^{+\infty}\left(n_{i}+\frac{1}{2}\right)\sqrt{(1+c)k_{i}^{2}+2},$
(26)

where the classical energy $V[\phi_{\rm{tri.}}]$ is zero. We see from Eq. (26)
(or Eq. (25)) that the Lorentz violation effect can be eliminated by the
redefinition of $k_{n}$ (or spatial coordinate). Therefore, the vacuum
solution does not receive any genuine correction due to the Lorentz violation
in this case.

###  3.2 Excitations of kink

Now, we discuss the quantization of the kink. Similar to previous subsection,
we can expand the potential $V[\phi]$ about the $\phi_{\rm{k}}$. Up to the
second order of $\zeta$, one obtains the following expression for the
potential:

$\begin{array}[]{cc}&V[\phi]=V[\phi_{\rm{\rm{k}}}]+\int
dx\frac{1}{2}\zeta(x)\bigg{[}-(1+c\phi_{k}^{2})\partial^{2}_{x}-3c(\partial_{x}\phi_{k})^{2}-3c(\partial_{x}\phi_{k}^{2})\partial_{x}\\\
&\hskip
85.35826pt-4c\phi_{k}(\partial_{x}^{2}\phi_{k})-1+3\phi_{k}^{2}\bigg{]}\zeta(x).\end{array}$
(27)

Therefore, the stability equation, read from Eq. (27), is

$\bigg{[}-(1+c\phi_{k}^{2})\partial^{2}_{x}-3c(\partial_{x}\phi_{k})^{2}-3c(\partial_{x}\phi_{k}^{2})\partial_{x}-4c\phi_{k}(\partial_{x}^{2}\phi_{k})-1+3\phi_{k}^{2}\bigg{]}\zeta_{n}(x)=\omega_{n}^{2}\zeta_{n}(x)$
(28)

The above equation can be solved via perturbation on LV parameter $c$. Hence,
we expand our states $\zeta_{n}(x)$ and normal modes $\omega_{n}$ as

$\zeta_{n}=\zeta_{n}^{(0)}+c\ {\zeta_{n}^{(1)}}+c^{2}{\zeta_{n}^{(2)}}+\dots,$
(29) $\omega_{n}=\omega_{n}^{(0)}+c\
{\omega^{(1)}}+c^{2}{\omega_{n}^{(2)}}+\dots.$ (30)

In the above equations $\zeta_{n}^{(0)}$ and $\omega_{n}^{(0)}$ are the
leading order terms given by Eqs. (20) and (21). Accordingly, the order $c$ of
Eq. (28) becomes

$\displaystyle\left\\{\partial_{x}^{2}+1+\left(\omega_{n}^{(0)}\right)^{2}-3\left[\phi_{\rm{k}}^{(0)}(x)\right]^{2}\right\\}\zeta_{n}^{(1)}(x)+F_{n}(x)=0,$
(31)

where

$\displaystyle
F_{n}(x)=\bigg{[}\left(\phi_{\rm{k}}^{(0)}\right)^{2}\partial_{x}^{2}+3\partial_{x}\left(\phi_{\rm{k}}^{(0)}\right)^{2}\partial_{x}+2\omega_{n}^{(0)}\omega_{n}^{(1)}$
$\displaystyle\hskip
48.36958pt-6\phi_{\rm{k}}^{(0)}\phi_{\rm{k}}^{(1)}+3\left(\partial_{x}\phi_{\rm{k}}^{(0)}\right)^{2}+4\phi_{\rm{k}}^{(0)}\partial_{x}^{2}\phi_{\rm{k}}^{(0)}\bigg{]}\zeta_{n}^{(0)}(x).$
(32)

For the two bound states, the solutions of this equation are

$\left\\{\begin{array}[]{cc}\zeta^{(1)}_{0}(x)=\frac{1}{8}\text{sech}^{4}\left(\frac{x}{\sqrt{2}}\right)\left(2\sqrt{2}x\sinh\left(\sqrt{2}x\right)-9\cosh\left(\sqrt{2}x\right)+11\right)&\\\
\hskip 179.25244pt\mbox{with}\quad\omega^{(1)}_{0}=0,\\\
\zeta^{(1)}_{1}(x)=\frac{1}{20}\text{sech}\left(\frac{x}{\sqrt{2}}\right)\bigg{\\{}5\sqrt{2}x\left(1-2\text{sech}^{2}\left(x/\sqrt{2}\right)\right)&\\\
\hskip
28.45274pt+\tanh\left(\frac{x}{\sqrt{2}}\right)\left[-2\sqrt{2}x+4\log\left(e^{\sqrt{2}x}+1\right)+50\
\text{sech}^{2}\left(\frac{x}{\sqrt{2}}\right)-91/3\right]\bigg{\\}}&\\\
\hskip 202.01474pt\mbox{with}\quad\omega^{(1)}_{1}=\frac{1}{5\sqrt{6}}.&\\\
\end{array}\right.$ (33)

Figure 2:  The zeroth and first order of two bound states.

Actually, $\omega_{0}$ does not receive any correction. To derive these
solutions and their corresponding modes, we have taken the boundary condition
at infinity into account. For comparison, we have plotted the zeroth and first
order bound states in Fig. 2. For the continuum states, we can obtain
corresponding frequencies by writing Eq. (31) at $x\to\pm\infty$

$\left(2\partial_{x}^{2}+q^{2}\right)\zeta^{(1)}(x)+\left(q^{2}+3iq-2\right)\left(q^{2}-2\sqrt{2}\sqrt{q^{2}+4}\omega_{q}^{(1)}\right)e^{iqx/\sqrt{2}}=0.$
(34)

Therefore, we get

$\displaystyle\zeta^{(1)}_{q}(x)=\zeta^{(0)}_{q}(x)$
$\displaystyle\qquad+Q_{2}^{iq}\left[\tanh\left(x/\sqrt{2}\right)\right]\int_{1}^{\tanh\left(x/\sqrt{2}\right)}{{\cal
F}_{q}(t)}P_{2}^{iq}(t)dt$ $\displaystyle\hskip 56.9055pt-
iP_{2}^{iq}\left[\tanh\left(x/\sqrt{2}\right)\right]\int_{1}^{\tanh\left(x/\sqrt{2}\right)}{{\cal
F}_{q}(t)}Q_{2}^{iq}(t)dt$ (35)
$\displaystyle\quad\mbox{with}\qquad\omega^{(1)}_{q}=\frac{q^{2}}{2\sqrt{2}\sqrt{q^{2}+4}}.$
(36)

where,

$\displaystyle{{\cal
F}_{q}(t)}=\frac{e^{iq\tanh^{-1}t}}{(3-iq)\left[P_{3}^{iq}(t)Q_{2}^{iq}(t)-P_{2}^{iq}(t)Q_{3}^{iq}(t)\right]}$
$\displaystyle\times[q^{4}+\left(-44q^{2}-105iqt+105t^{2}-68\right)t^{2}$
$\displaystyle+6\left(q^{2}+3iqt-3t^{2}+1\right)t\tanh^{-1}t+3i\left(3q^{2}+11\right)qt+4q^{2}+3]$

and $P_{n}^{m}(x)$ and $Q_{n}^{m}(x)$ are the first and second kinds of the
associated Legendre functions, respectively. From Eq. (34) we can see that the
asymptotic form of the $\zeta^{(1)}_{q}(x)$ at infinity is $e^{\pm
iqx/\sqrt{2}}$. Therefore we have

$\zeta_{q}(x)\to
e^{iqx/\sqrt{2}}\left[2-q^{2}-3iq+c\right]=\exp[iqx/\sqrt{2}\pm\delta(q)/2]\quad\mbox{at}\qquad
x\to\pm\infty,$ (38)

where the phase shift is

$\delta(q)=-2\tan^{-1}\left(\frac{3q}{2+c-q^{2}}\right)=-2\tan^{-1}\left(\frac{3q}{2-q^{2}}\right)+\frac{6qc}{q^{4}+5q^{2}+4}+{\cal
O}(c^{2}).$ (39)

Here, the first term of the right hand side is the leading term of the phase
shift (without Lorentz violation) and the second is the correction raised from
Lorentz violation. Now, the correction to the energy spectrum becomes

$E^{(1)}_{\\{n_{1},n_{q_{i}}\\}}=\frac{\sqrt{2}}{15}+\left(n_{1}+\frac{1}{2}\right)\frac{1}{5\sqrt{6}}+\sum_{q_{i}}\left(n_{q_{i}}+\frac{1}{2}\right)\frac{q_{i}^{2}}{2\sqrt{2}\sqrt{q_{i}^{2}+4}}$
(40)

where $q_{i}$ are derived by substituting Eq. (39) in Eq. (22).

##  4 Summary

The Lorentz symmetry may be broken in high energies where the gravity must be
quantized. Therefore, we need to check what causes the Lorentz violation in
low energy theories. In this paper, we have studied the effects of an LV
potential on the energy spectrum predicted for the $\phi^{4}$ scalar field
theory in two space-time dimensions. We have considered an additional
fundamental interaction $\frac{-c}{2}\phi^{2}(u.\partial\phi)^{2}$ that
couples the derivative of a field to a constant vector describing LV. This is
the lowest term which leads to nontrivial effects in the theory. The
modifications of wave functions and the corresponding energy spectrum have
been investigated for both trivial vacuum and kink. The trivial vacuum is not
affected by the LV potential. However, up to the first order of the LV
coefficient $c$, we have found the kink and its excitation modes (all bound
and continuum states) are modified. One may expect such modifications because
kink is a result of potential and this new LV term is also in the same order
in $\phi$’s.

AcknowledgementThe authors would like to thank the Research Deputy of the
University of Qom for financial support.

## References

  * Manton and Sutcliffe [2004] N. Manton and P. Sutcliffe, _Topological solitons_ (Cambridge University Press, 2004).
  * Belova and Kudryavtsev [1997] T. I. Belova and A. E. Kudryavtsev, Physics-Uspekhi 40, 359 (1997).
  * Abrikosov [2004] A. Abrikosov, Reviews of modern physics 76, 975 (2004).
  * Kleckner and Irvine [2013] D. Kleckner and W. T. Irvine, Nature physics 9, 253 (2013).
  * Vilenkin and Shellard [2000] A. Vilenkin and E. P. S. Shellard, _Cosmic strings and other topological defects_ (Cambridge University Press, 2000).
  * t Hooft [1974] G. t’ Hooft, Nucl. Phys. B 79, 276 (1974).
  * Schaefer [2002] T. Schäfer, Physical Review D 65, 094033 (2002).
  * [8] M. Shifman and A. Yung,Reviews of Modern Physics, 79 1139 (2007).
  * [9] A. Opfennann and G. Papadopoulos,Nuclear Physics  B488 177 (1997).
  * [10] J. F. Schonfeld, Nuclear Physics B161 125 (1979).
  * [11] N. Graham and R. L. Jaffe., Nuclear Physics B544 432 (1999).
  * Vachaspati [2006] T. Vachaspati, _Kinks and domain walls: An introduction to classical and quantum solitons_ (Cambridge University Press, 2006).
  * Cuevas-Maraver and Kevrekidis [2019] J. Cuevas-Maraver and P. G. Kevrekidis, in _A Dynamical Perspective on the $\phi^{4}$ Model_ (Springer, 2019), pp. 137–162.
  * Waterhouse [2019] A. Waterhouse, arXiv preprint arXiv:1908.09650 (2019).
  * Kosteleckỳ and Potting [1995] V. A. Kosteleckỳ and R. Potting, Physical Review D 51, 3923 (1995).
  * Carroll et al. [2001] S. M. Carroll, J. A. Harvey, V. A. Kosteleckỳ, C. D. Lane, and T. Okamoto, Physical Review Letters 87, 141601 (2001).
  * Aghababaei and Haghighat [2017] S. Aghababaei and M. Haghighat, Physical Review D 96, 075017 (2017).
  * Colladay and Kosteleckỳ [1998] D. Colladay and V. A. Kosteleckỳ, Physical Review D 58, 116002 (1998).
  * [19] A. F. Santos and F. C. Khanna. Physics Letters B 762 283 (2016).
  * [20] M. B. Cruz, E. R. Bezerra de Mello and A. Yu Petrov. Physical Review D 96 045019 (2017).
  * [21] A. Mojavezi, R. Moazzemi and M. E. Zomorrodian. Nuclear Physics B941 145 (2019).
  * [22] J. D. Tasson, Reports on Progress in Physics 77 062901 (2014).
  * [23] Q. G. Bailey and V. A. Kostelecký, Physical Review D 70 076006 (2004).
  * [24] R. Casana, M. M. Ferreira Jr and C. E. H. Santos, Physical Review D 78 025030 (2008).
  * [25] Casana, Rodolfo, M. M. Ferreira Jr, A. R. Gomes, and P. R. D. Pinheiro. Physical Review D 80 125040 (2009).
  * [26] H. Belich, M. M. Ferreira, and J. A. Helayel-Neto, The European Physical Journal C 38 511 (2005).
  * [27] H. Belich, T. Costa-Soares, M.M. Ferreira Jr., J. A. Helay¨el-Neto, Eur. Phys. J. C 42, 127 (2005).
  * Gopakumar et al. [2000] R. Gopakumar, S. Minwalla, and A. Strominger, Journal of High Energy Physics 2000, 020 (2000).
  * Belich et al. [2005] H. Belich, T. Costa-Soares, M. FerreiraJr, and J. Helayël-Neto, The European Physical Journal C-Particles and Fields 41, 421 (2005).
  * Seifert [2013] M. D. Seifert, in _Proceedings of the Sixth Meeting on CPT and Lorentz Symmetry_ (World Scientific, 2013), pp. 212–215.
  * Lubo [2005] M. Lubo, Physical Review D 71, 047701 (2005).
  * Passos et al. [2018] E. Passos, C. Almeida, F. Brito, R. Menezes, J. Mota-Silva, and J. Santos, Annals of Physics 396, 351 (2018).
  * Bazeia et al. [2010] D. Bazeia, M. Ferreira Jr, A. Gomes, and R. Menezes, Physica D: Nonlinear Phenomena 239, 942 (2010).
  * Barreto et al. [2006] M. N. Barreto, D. Bazeia, and R. Menezes, Physical Review D 73, 065015 (2006).
  * [35] S. M. Carroll and H. Tam, Physical Review D 78 044047 (2008).
  * [36] M. Gomes, J. R. Nascimento, A. Yu Petrov, and A. J. Da Silva, Physical Review D 81 045018 (2010).
  * Rajaraman [1982] R. Rajaraman, _Solitons and instantons_ (North Holland, 1982).
  * Dashen et al. [1974a] R. F. Dashen, B. Hasslacher, and A. Neveu, Physical Review D 10, 4114 (1974a).
  * Dashen et al. [1974b] R. F. Dashen, B. Hasslacher, and A. Neveu, Physical Review D 10, 4130 (1974b).


#  Learning Semantic Textual Similarity via Topic-informed
Discrete Latent Variables

Erxin Yu1,2, Lan Du4, Yuan Jin4, Zhepei Wei1,2, Yi Chang1,2,3
1School of Artificial Intelligence, Jilin University
2Key Laboratory of Symbolic Computation and Knowledge Engineering, Jilin
University
3International Center of Future Science, Jilin University
4Faculty of Information Technology, Monash University, Australia
<EMAIL_ADDRESS><EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
Corresponding Author

###### Abstract

Recently, discrete latent variable models have received a surge of interest in
both Natural Language Processing (NLP) and Computer Vision (CV), attributed to
their comparable performance to the continuous counterparts in representation
learning, while being more interpretable in their predictions. In this paper,
we develop a topic-informed discrete latent variable model for semantic
textual similarity, which learns a shared latent space for sentence-pair
representation via vector quantization. Compared with previous models limited
to local semantic contexts, our model can explore richer semantic information
via topic modeling. We further boost the performance of semantic similarity by
injecting the quantized representation into a transformer-based language model
with a well-designed semantic-driven attention mechanism. We demonstrate,
through extensive experiments across various English language datasets, that
our model is able to surpass several strong neural baselines in semantic
textual similarity tasks.

##  1 Introduction

Semantic Textual Similarity (STS), which concerns the problem of measuring and
scoring the relationships or relevance of pairs of text on real-valued scales,
is a fundamental task in NLP. It has further driven many other important NLP
tasks such as machine translation Zou et al. (2013), text summarization
Mohamed and Oussalah (2019), question answering Bordes et al. (2014), etc.

Recently, deep neural language models have achieved state-of-the-art
performance for STS. The success of these models is attributed to their
adoption of self-supervised learning on text representations, which overcomes
the absence of large labeled STS data in many domains. The text
representations learned by these models, usually consist of continuous latent
variables, and can be fed pairwise into some functions (e.g. a multi-layer
perceptron) to compute their semantic similarity.

Compared to the continuous ones, discrete latent variables have drawn much
less attention in language modeling, despite that natural languages are
discrete in nature, and there is growing evidence, from several NLP tasks Jin
et al. (2020); Bao et al. (2020), that they are equally good, if not more
suitable, as the continuous counterparts. As a widely-used training technique
for learning discrete latent variables, vector-quantized variational
autoencoder (VQ-VAE) Oord et al. (2017) computes the values for the variables
through the nearest neighbor look-up across the quantized vectors from the
shared latent embedding space. Despite its success in speech recognition and
computer vision, VQ-VAE has yet to be investigated in its use and viability in
general NLP tasks. In this paper, we intend to explore the use of VQ-VAE and
its generated discrete latent variables for semantic textual similarity.

Two issues need to be resolved in order for the VQ-VAE to work effectively on
the STS task. First, the codebook embeddings need to be carefully initialized
to prevent the so-called codebook collapse, in which only a few of the
embeddings are selected and learned by the model, causing a reduction in the
representational capacity of the codebook. Past research has indicated that
sufficiently informed and sophisticated manipulation of the embedding
initialization can mitigate this problem. A second issue is that most STS
scenarios are designed to teach a language model to measure the textual
similarity from a local perspective (e.g. over contexts within each sentence).
However, global semantics underlying a broader context has already been shown
to benefit a number of NLP tasks, including language generation Guo et al.
(2020a), textual similarity calculation Peinelt et al. (2020), keyphrase
generation Wang et al. (2019), etc. For STS, it can provide perspectives on
the global correlations and dependencies among sentences, which can be
leveraged to better distinguish the semantic nuances of sentence-level
contexts.

In this paper, we propose to leverage topic modeling to tackle both issues.
The topic information can provide informative guidance on the initialization
and the learning of VQ-VAE’s codebook. Furthermore, since the discovered
topics can capture global semantics (e.g., semantics as distributions over the
vocabulary shared by the whole corpus), they can be used to calibrate the bias
of local semantics on the measure of textual similarity.

Our proposed topic-enhanced VQ-VAE language model features two major
components to enrich the codebook representational capacity with rich
contextual information. One is the topic-aware sentence encoder, which
computes the sentence embedding using its (local) topic distribution and the
corresponding global topic-word distributions. The other is a topic-guided VQ-
VAE codebook, where we align its latent codes/embeddings with the topic
embeddings jointly learned by an NTM throughout the training phase. We further
incorporate the quantized sentence representations learned by our topic-
enhanced VQ-VAE into transformer-based language models to guide the learning
of contextual embeddings. Built on top of the multi-head attention, a
semantics-driven attention mechanism is proposed to compute the attention
scores based on the quantized sentence representations. The proposed attention
mechanism is a flexible plug-and-play module that requires a small number of
extra parameters and a minimum change to the existing multi-head attention
implementation. Our contributions are summarised as:

  * •

To the best of our knowledge, this is the first language model that explores
the use of discrete latent variables learned by VQ-VAE for STS.

  * •

We introduce a topic-enhanced VQ-VAE model that combines the merits of topic
modeling and vector quantization, where the quantization is informed by the
topic information jointly learned by an NTM to enhance its robustness against
the collapsing problem and its capacity of capturing both the global and local
semantics.

  * •

We present a simple yet effective semantics-driven attention mechanism to
inject the quantized representations into transformer models, which can serve
as a plug-and-play module for almost all the existing transformer-based
models.

  * •

Comprehensive experiments on six real-world datasets for semantic textual
matching demonstrate the effectiveness of our model and the significance of
its major components.

##  2 Related Work

###  2.1 Neural Language Models for STS

Neural language models have played a key role in fulfilling the STS task.
Among them, the earlier CNN-based model Hu et al. (2014); Yin et al. (2016)
and the RNN-based model Chen et al. (2017) used to achieve the state-of-the-
art performance before the emergence of the transformer architecture Vaswani
et al. (2017) and the large-scale pre-training of its corresponding models.
These pre-trained models, with their ability to be fine-tuned on various
domains and tasks, have dominated the entire NLP area, significantly improving
the task performance in the specific domains. As the most prominent model,
BERT Devlin et al. (2019), along with its variants including RoBERTa Liu et
al. (2019), BERT-sim Xia et al. (2021), ALBERT Lan et al. (2020) and SemBERT
Zhang et al. (2020), have achieved superior results in the STS task. Despite
the achieved success, current language models are mostly “black-box” models
with low interpretability, which require additional explanatory components for
their predictions in needed fields such as Biomedicine and Finance.

###  2.2 Neural Topic Models

The success of Variational AutoEncoder (VAE) Kingma and Welling (2013) has led
to the development of a series of neural topic models (NTM) Zhao et al. (2017,
2021b), (Refer to Zhao et al. (2021a) for a comprehensive survey on NTMs),
where the posterior distribution of document topics is approximated by a
neural network during the variational inference process, known as the neural
variational inference (NVI) Miao et al. (2016). The NVI also enables NTMs to
be easily combined with various language models for capturing the global
semantics information. These hybrid models are known as the topic-aware
language models. Their combination strategies can be largely characterized by
either concatenation of (document) topic distributions with the local word
embeddings, or (multi-head) attention mechanisms towards the inferred topics.
These models have been shown to achieve higher performance than the plain
language models on various tasks, such as text summarization Wang et al.
(2020), keyphrase generation Wang et al. (2019) and document classification
Zeng et al. (2018).

###  2.3 VQ-VAE

Rather than learning continuous representations, VQ-VAE Oord et al. (2017)
learns quantized representations via vector quantization. It was first applied
to speech representation Chorowski et al. (2019) and video generation Yan et
al. (2021). Recently, it has attracted lots of attention in a variety of NLP
tasks. In machine translation, Kaiser et al. (2018) applied VQ-VAE to decode
the target language sequence from the discrete space, making the model
inference faster and the model performance only slightly compromised. In text
classification, Jin et al. (2020) explored general text representations
induced by VQ-VAE for the classification especially in low-resource domains.
Huang and Ji (2020) designed a semi-supervised VQ-VAE for new event type
induction, while Guo et al. (2020b) equipped the encode-decoder architecture
with the VQ-VAE to automatically select relevant evidence. In this paper, we
move one step further by guiding the vector quantization with topics learned
by NTMs.

##  3 Modelling Framework

Given a pair of sentence $X=\left\\{{\bf\it X}_{i},{\bf\it X}_{j}\right\\}$ as
the input, where each sentence is represented as the bag-of-words (bow)
vectors and contextualized embedding, our modelling framework, as shown in
Figure 1, predicts the semantic similarity $Y$ between ${\bf\it X}_{i}$ and
${\bf\it X}_{j}$.

Figure 1: Our semantic textual matching framework combines the benefit of
NTMs, VQ-VAE and Transformer.

###  3.1 Neural Topic Component

We adopt a VAE-based neural topic model Miao et al. (2017) to learn latent
topics. Different from LDA Blei et al. (2003), NTM parameterises the latent
topics ${\bf\it\theta}$ using a neural network conditioned on a draw from a
Gaussian Softmax Construction. Here, ${\bf\it\theta}\in\mathbb{R}^{K}$
represents the topic proportions of a sentence $X$, where $K$ denotes the
number of topics. Let ${\bf\it\beta}_{k}\in\mathbb{R}^{V}$ be a distribution
over a vocabulary $V$ associated with a topic $k$. Following Wang and YANG
(2020), we use word embeddings ${\bf\it\omega}\in\mathbb{R}^{V\times E}$,
topic embeddings ${\bf\it\phi}\in\mathbb{R}^{K\times E}$ to compute
${\bf\it\beta}_{k}$ as
${\bf\it\beta}_{k}=Softmax({\bf\it\omega}\cdot{\bf\it\phi}_{k}^{T})$.

With NVI, we can use an encoder network to approximate the true posterior.
Specifically, the encoder generates the variational parameters ${\bf\it\mu}$
and ${\bf\it\sigma}^{2}$ through neural networks and the latent variable
${\bf\it\theta}$ is sampled with the Gaussian reparameterization trick. The
decoder reconstructs the BoW representation of a document by maximizing the
log-likelihood of the input. The loss function (i.e., ELBO) contains a
reconstruction error term and a KL divergence term as

$\scriptstyle L_{NTM}=D_{KL}[q({\bf\it\theta}|{\bf\it
x})||p({\bf\it\theta}|{\bf\it x})]-E_{q({\bf\it\theta}|x)}[\sum_{m=1}^{M}\log
p(w_{m}|{\bf\it\theta})],$

where $q({\bf\it\theta}|{\bf\it x})$ denotes the variational posterior
distribution of ${\bf\it\theta}$ given the sentence ${\bf\it
x}=(w_{1},...,w_{m})$, approximating the true posterior $p({\bf\it\theta}|x)$.

###  3.2 Topic-enhanced VQ-VAE

VQ-VAE Oord et al. (2017) takes the contextualized embeddings of words within
a sentence: ${\bf\it X}^{seq}=({\bf\it x}_{1}^{seq},...,{\bf\it x}_{m}^{seq})$
into its encoder to produce the corresponding latent representations ${\bf\it
Z}^{e}$. Then, the quantized representations ${\bf\it Z}^{q}$ are calculated
by the nearest neighbor look-up using the predefined and shared embedding
space ${\bf\it E}$, and further used as an input to the decoder to reconstruct
the original sentence text.

However, since the original embedding space of VQ-VAE is randomly initialized,
these learned embeddings can be arbitrary without clear semantic meaning. To
guide the codebook learning, we incorporate the topic information into VQ-VAE
by designing the following two components:

#### Topic Sensitive Encoder.

We use a standard single-layer Transformer Vaswani et al. (2017) as our
encoder. Specifically, for a sentence consisting of a sequence of $m$ words
$\bm{X}=(w_{1},...,w_{m})$, its corresponding word representations ${\bf\it
X}^{seq}$ are generated by the transformer embedding layer, which concatenates
the word embedding and position embedding. To incorporate the topic
information, we leverage the sentence topic distribution ${\bf\it\theta}$ and
the word topic weight vector ${\bf\it\beta}_{w_{m}}\in\mathbb{R}^{K}$ from the
NTM model to compute the topical embedding of the words ${\bf\it
X}^{t}=\left\\{{\bf\it x}_{1}^{t},...,{\bf\it
x}_{m}^{t}\right\\}\in\mathbb{R}^{m\times K}$, where ${\bf\it
x}_{m}^{t}\in\mathbb{R}^{K}={\bf\it\theta}\otimes{{\bf\it{\beta}}_{w_{m}}}.$
We concatenate the ${\bf\it X}^{seq}$ and ${\bf\it X}^{t}$ and feed them into
the encoder as ${\bf\it Z}^{e}\in\mathbb{R}^{m\times E}=f_{enc}({\bf\it
X}^{seq}\oplus{\bf\it X}^{t}).$

#### Topical Latent Embedding.

We get final word representations ${\bf\it Z}^{e}=\\{{\bf\it
z}_{1}^{e},...,{\bf\it z}_{m}^{e}\\}$ from the topic sensitive encoder. These
representations are mapped onto the nearest element of embedding ${\bf\it E}$.
Different from previous VQ-VAE models that use a randomly initialized
embedding ${\bf\it E}$, we leverage the topic embedding
${\bf\it\phi}\in\mathbb{R}^{K\times E}$ from NTM. The discretization process
is defined as: For each $i\in\\{1,\dots,m\\}$, ${\bf\it
z}_{i}^{q}={\bf\it\phi}_{k}\in\mathbb{R}^{E},\,\mbox{where}\,k=\mathop{\arg\min}_{j\in\left\\{1,\dots,k\right\\}}\left\|{\bf\it
z}_{i}^{e}-{\bf\it\phi}_{j}\right\|_{2}.$ Thus, we have quantized
representations ${\bf\it Z}^{q}=\left\\{{\bf\it z}_{1}^{q},...,{\bf\it
z}_{m}^{q}\right\\}\in\mathbb{R}^{m\times E}$ for a sentence. We then feed
${\bf\it Z}^{q}$ into the decoder which is also a single-layer transformer to
reconstruct the sentence X. The overall training objective is thus defined as

$\displaystyle L_{VQ-VAE}=$ $\displaystyle\sum_{i=1}^{m}(-\log p({\bf\it
x}_{i}|{\bf\it z}_{i}^{q})+\left\|sg[{\bf\it z}_{i}^{e}]-{\bf\it
z}_{i}^{q}\right\|_{2}^{2}$ $\displaystyle+\lambda\left\|{\bf\it
z}_{i}^{e}-sg[{\bf\it z}_{i}^{q}]\right\|_{2}^{2}).$ (1)

The first term above is the reconstruction error of the decoder given ${\bf\it
Z}^{q}$. The last two terms are used to minimize the distance between the
latent embedding ${\bf\it E}$ and the encoder output $Z^{e}$, where
${\bf\it\lambda}$ is a commitment loss and $sg(\cdot)$ means the stop-gradient
operation. It is noteworthy that our topic-enhanced VQ-VAE model jointly
trained with NTM can explicitly learn the topic assignment of each individual
words, which is not feasible for NTM alone.

###  3.3 Semantics-driven Multi-head Attention

The multi-head attention in transformer-based models explores the
relationships among tokens by calculating the token similarities, defined as

$\displaystyle{\bf\it Q}=$ $\displaystyle{\bf\it K}={\bf\it V}={\bf\it
h}^{l-1}$ $\displaystyle\text{MultiHead}({\bf\it Q},{\bf\it K},{\bf\it V})$
$\displaystyle=\text{Concat}({\bf\it head}_{1},\cdots,{\bf\it
head}_{n}){\bf\it W}^{O}$ $\displaystyle{\bf\it head}_{i}=\text{Attention}$
$\displaystyle(\mathbf{Q_{i}}{W}_{i}^{Q},{\bf\it K_{i}}{W}_{i}^{K},{\bf\it
V_{i}}{W}_{i}^{V})$ (2)

where ${\bf\it h}^{l-1}\in\mathbb{R}^{L\times(n\cdot E)}$ is the output of
last layer, $L$ is the length of sentence pair, n is the number of attention
head, $E$ is the hidden size of each attention head. ${\bf\it W}_{i}^{Q}$,
${\bf\it W}_{i}^{K}$, ${\bf\it W}_{i}^{V}$ and ${\bf\it
W}_{i}^{O}\in\mathbb{R}^{E\times E}$ are projection matrices. Scaled Dot-
Product method is adopted to calculate the attention function:

${\text{Attention}({\bf\it Q},{\bf\it K},{\bf\it
V})}=\text{Softmax}(\dfrac{{\bf\it QK}^{T}}{\sqrt{d_{k}}}){\bf\it V}$ (3)

To enable the transformer-based model to learn discrete semantics between two
sentences, we design a semantics-driven attention mechanism. Specifically,
given the quantized representations ${\bf\it Z}_{i}^{q}$, ${\bf\it Z}_{j}^{q}$
for sentence ${\bf\it X}_{i}$ and ${\bf\it X}_{j}$ respectively, we compute
the quantized representation-based query and key matrices as follows:

$\displaystyle{\bf\it Q}_{q}$ $\displaystyle={\bf\it
ZW}_{q}^{Q},~{}~{}~{}{\bf\it K}_{q}={\bf\it ZW}_{q}^{K},~{}~{}~{}{\bf\it
Z}=\text{FFN}[{\bf\it Z}_{i}^{q}\oplus{\bf\it Z}_{j}^{q}],$ (4)

where ${\bf\it Z}$, ${\bf\it Q}_{q}$, ${\bf\it K}_{q}\in\mathbb{R}^{L\times
E}$, ${\bf\it W}_{q}^{Q}$ and ${\bf\it W}_{q}^{K}\in\mathbb{R}^{E\times E}$,
FFN means a fully connected feed-forward network. The semantics-driven
attention can be defined as:

$\text{Attention}({\bf\it Q},{\bf\it K},{\bf\it
V})=\text{Softmax}(\dfrac{{\bf\it QK}^{T}+{\bf\it Q}_{q}{\bf\it
K}_{q}^{T}}{\sqrt{d_{k}}}){\bf\it V}.$ (5)

It is different from the topic select-attention block introduced by Lu et al.
(2021), which is an attention block built separately on top of the multi-head
attention block. In addition to the introduction of the quantized
representations into the self-attention mechanism, we also leverage those
representations to enhance the final output ${\bf\it h}^{l}$ of multi-layer
transformer, i.e., $[{\bf\it h}^{l};{\bf\it Z}_{i}^{q}\oplus{\bf\it
Z}_{j}^{q}]\in\mathbb{R}^{L\times(n+1)E}$, which is further fed into a non-
linear layer to predict label $Y$ for semantic textual matching.

###  3.4 Joint Training

Our proposed framework integrates the three modules in Figure 1, i.e., the
NTM, the topic-enhanced VQ-VAE, and the semantics-attended Transformer
equipped with the semantics-driven attention mechanism. We first pretrain the
NTM to obtain meaningful topic information used to initialize the codebook of
the topic-enhanced VQ-VAE. Then we joint train the NTM and topic-enhanced VQ-
VAE to boost their performance. The joint loss function is defined as:

$\displaystyle L=L_{VQ-VAE}+\gamma L_{NTM},$ (6)

where $\gamma$ is the trade-off parameter controlling the balance between the
topic model and the topic-enhanced VQ-VAE model. Finally, we incorporate the
learned quantized representations from topic-enhanced VQ-VAE into the
transformer block and fine-tune the transformer block on the STS task.

##  4 Experiment and analysis

###  4.1 Datasets

We conducted experiments on the following benchmark datasets used for the STS
task, which include MRPC, Quora, STS-B, and SemEval CQA. The Microsoft
Research Paraphrase dataset (MRPC) Dolan and Brockett (2005) contains pairs of
sentences from news websites with binary labels for paraphrase detection. The
Quora duplicate question dataset contains more than 400k question pairs with
binary labels for predicting if two questions are paraphrases. We use
train/dev/test set partition from Wang et al. (2017). STS-B is a collection of
sentence pairs extracted from news headlines and other sources. It comprises a
selection of the English datasets used in the STS task which were annotated
with a score from 1 to 5 denoting how similar the two sentences are. The
SemEval community question answering has three subtasks: (A) Question–Comment
Similarity, (B) Question–Question Similarity, (C) Question–External Comment
Similarity Nakov et al. (2015, 2016, 2017). Following their settings, we use
the 2016 test set as the development set and the 2017 test set as the test
set. Table 1 summarises their statistics.

Dataset | Train/Dev/Test | Len | Label
---|---|---|---
MRPC | 3,668/408/1,725 | 22 | 0/1
Quora | 384,348/10,000/10,000 | 13 | 0/1
STS-B | 5,749/1,500 /1,379 | 10 | 0~5
SemEval-A | 20,340/3,270/2,930 | 41 | 0/1
SemEval-B | 3,169/700/880 | 46 | 0/1
SemEval-C | 31,690/7,000/8,800 | 39 | 0/1
Table 1: Statistics of Datasets

Models | MRPC | Quora | STS-B | SemEval
---|---|---|---|---
A | B | C
Baselines | $Acc$ | $F1$ | $Acc$ | $F1$ | $PC$ | $SC$ | $Acc$ | $F1$ | $Acc$ | $F1$ | $Acc$ | $F1$
ERNIE | 83.97 | 88.16 | 90.88 | 90.76 | 85.12 | 83.26 | 77.53 | 75.10 | 68.86 | 51.42 | 94.36 | 27.20
tBert | 84.29 | 88.44 | 90.76 | 90.65 | 85.34 | 83.96 | 78.02 | 76.77 | 71.44 | 52.10 | 94.07 | 27.33
SemBert | 84.02 | 88.24 | 90.83 | 90.77 | 87.13 | 85.34 | 78.15 | 76.33 | 71.47 | 52.19 | 94.60 | 27.44
Bert | 84.21 | 88.21 | 90.79 | 90.72 | 86.11 | 84.90 | 77.88 | 75.25 | 66.81 | 51.33 | 94.40 | 27.36
DisBert | 84.70 | 89.06 | 90.94 | 90.81 | 86.67 | 85.64 | 79.15 | 77.57 | 72.38 | 54.44 | 94.96 | 27.61
RoBERTa | 87.01 | 90.61 | 91.03 | 91.45 | 89.04 | 88.38 | 78.53 | 76.56 | 75.09 | 55.75 | 93.52 | 32.46
DisRoBERTa | 88.06 | 91.15 | 91.53 | 91.81 | 89.90 | 89.28 | 78.89 | 77.01 | 77.72 | 57.18 | 95.63 | 33.79

Table 2: Comparisons of performance on six STS datasets. We report average
performance for five different random seeds. The better results of different
transformer-based model are highlighted in bold (according to the pairwise
t-test with 95% confidence).

(a) MRPC

(b) Quora

Figure 2: Ablation study on DisBert and DisRoBERTa model. We reported F1 score
for MRPC and Quora datasets.

###  4.2 Baseline Models

We implemented several strong baselines in the STS task for comparison. To
evaluate the semantics-attended Transformer, we select two widely-used
pretrained language models, Bert Devlin et al. (2019) and RoBERTa Liu et al.
(2019). We used the small version to fine-tune the STS task, i.e., Bert-base
and RoBERTa-base. And we modified these two models by adding the semantics-
driven multi-head attention to the original multi-head attention, which is
denoted as DisBert and DisRoBERTa respectively. We selected tBERT Peinelt et
al. (2020) as an important baseline that also makes use of topic information.
Different from tBert which directly concatenates topic representation and
sentence pair vector, we leverage the topic model to help VQ-VAE capture
global semantics through the topic sensitive encoder and topical latent
embedding. We also select the knowledge-enhanced Bert model ERNIE Zhang et al.
(2019) that incorporates knowledge graph into Bert and achieves improvements
on language understanding tasks. To further verify the performance on the STS
task, we also compared with Semantic-aware Bert (SemBert) Zhang et al. (2020),
which incorporates explicit contextual semantics and outperforms other Bert-
based models on the STS task.

###  4.3 Experimental Settings

For all methods, we performed a greedy search to find their optimal hyper-
parameters using the development set. For NTM, we processed the datasets with
gensim tokenizer 111https://radimrehurek.com/gensim/ utils.html, and a
vocabulary for each dataset was built based on its training set with stop
words and words occurring less than 3 times removed. The BoW input of topic
model ${\bf\it X}^{BoW}$ on each dataset was constructed based on the
corresponding vocabulary. For the VQ-VAE model, we set the dimension of each
code $E$ to 64 and commitment loss $\lambda$ = 0.0001. 222Code is available at
https://github.com/ErxinYu/DisBert.

For joint training, we pretrained the NTM and selected the best model based on
their reconstruction perplexity. Then we jointly trained the selected NTM and
topic-enhanced VQ-VAE with $\gamma$ = 1. We fine-tuned our semantics-driven
language model and the vanilla one with the same parameters for a fair
comparison, following their settings with the publicly available code. We use
Pearson correlation coefficient (PC) and Spearman correlation coefficient (SC)
for STS-B datasets. For other datasets, we report accuracy (ACC) and F1 score.
More detailed parameter settings are described in the appendix.

###  4.4 Performance on Six STS Datasets

Table 2 shows the results drove on the six datasets, for which we have the
following observations.

  * •

Semantics-attended transformer is effective. Our model, DisBert, which
introduces the quantized representations into the transformer, significantly
outperforms Bert, with the improvement ranging from $0.1\%$ to $5.5\%$ on the
six datasets. The results demonstrate the usefulness of our semantics-driven
multi-head attention mechanism. Similarly, RoBERTa equipped with vector
quantized representations learned by our topic-enhanced VQ-VAE also achieves
improved performance. Thus, the results on both Bert and RoBERTa verify that
our semantics-driven attention is an effective plug-and-play module that can
be readily applied to various transformer-based language models.

  * •

DisBert outperforms other bert-based models. Our proposed methods achieve
better performance than tBert does, which directly concatenates the output of
the topic model and Bert. It indicates that injecting topic-enhanced VQ-VAE
into Bert is a more efficient way to provide global semantics. The performance
compared with ERNIE shows that our method is better at capturing sentence
semantics than incorporating knowledge graphs into Bert. Compared with
semBert, which is capable of explicitly absorbing contextual semantics, our
model gains better performance on five out of six STS datasets. Overall, the
performance gain on almost datasets demonstrates the effectiveness of our
proposed framework.

(a) SemEval-A

(b) SemEval-B

(c) SemEval-C

Figure 3: Performance of Bert and DisBert with different amounts of training
data of SemEval datasets.

(a) MPRC

(b) Quora

(c) STS-B

(d) SemEval-A

(e) SemEval-B

(f) SemEval-C

Figure 4: The impact of topic number $K$ on DisBert, where the horizontal axis
shows the number of topics and the vertical axis shows the performance on STS
datasets. Spearman correlation coefficient is reported in STS-B datasets and
F1 scores are reported for other datasets.

###  4.5 Ablation Study

We compared different variants of DisBert and DisRoBERTa in order to study how
topic-informed discrete latent variables affect performance. w/o Topic Model
means using the vanilla VQ-VAE model to get discrete latent variables without
topic information. w/o Topic Sensitive Encoder refers to removing topical
embedding ${\bf\it X^{t}}$ in the encoding phase and w/o Topical Latent
Embedding means randomly initializing embedding instead of the topic embedding
${\bf\it\beta}$. w/o Semantics-driven Attention stands for the variant
excludes the attention scores calculated by discrete variables (i.e., ${\bf\it
Q_{q}K^{t}_{q}}$) in Eq (5). w/o Output-enhanced is the variant does not
concatenate quantized variable ${\bf\it Z^{q}}$ with ${\bf\it h}^{l}$ in the
output layer.

As shown in Figure 2 we have the following observation: (1) Topic information
is informative for the STS task for both DisBert and DisRoBERTa. The
performance of DisBert has dropped 0.96, 0.04, and 0.28 on MRPC datasets
without the topic model, topic sensitive encoder, and topical latent embedding
respectively. (2) The well-designed semantics-driven multi-head attention
benefits the STS task. Without such attention, performance dropped for both
DisBert and DisRoBERTa on both datasets. Meanwhile, when the output is
equipped with quantized representations, performance is significantly
improved, which further verifies that the learned topic-informed discrete
latent variable can capture semantics and improve the STS task.

| DisBert | DisRoBERTa
---|---|---
| Multistage | Joint | Multistage | Joint
MRPC | 88.78 | 89.06 | 91.04 | 91.15
Quora | 90.45 | 90.81 | 90.67 | 91.81
STS-B | 85.32 | 85.64 | 88.89 | 89.28
SemEval-A | 77.49 | 77.57 | 76.89 | 77.01
SemEval-B | 54.21 | 54.44 | 56.95 | 57.18
SemEval-C | 27.50 | 27.61 | 33.57 | 33.79

Table 3: Joint training evaluation on six datasets. Spearman correlation
coefficient is reported in STS-B datasets and F1 scores are reported for other
datasets.

(a) Bert-based

(b) RoBERTa-based

Figure 5: Performance of DisBert and DisRoBERTa with semantics-driven multi-
head attention on different layers. w/o means the origin Bert/RoBERTa without
aforesaid attention and all means such attention is applied to all layers.

Cluster | Words
---|---
7 | dead protesters attack crash weapons kill killed
16 | bus day transport by car bike has what
20 | government vote year elections obama winner
26 | in for you on next out it up before us with

Sentence Pair | Label
---|---
S1: Five ${\textbf{\rm\color[rgb]{1,0,0}{killed}}_{\color[rgb]{1,0,0}{[7]}}}$ ${\textbf{\rm\color[rgb]{0,0,1}{in}}_{\color[rgb]{0,0,1}{[26]}}}$ Belgian ${\textbf{\rm\color[rgb]{0,1,0}{coach}}_{\color[rgb]{0,1,0}{[16]}}}$ ${\textbf{\rm\color[rgb]{1,0,0}{crash}}_{\color[rgb]{1,0,0}{[7]}}}$. | 3.8
S2: Teenagers
${\textbf{\rm\color[rgb]{0,0,1}{among}}_{\color[rgb]{0,0,1}{[26]}}}$ 5
${\textbf{\rm\color[rgb]{1,0,0}{dead}}_{\color[rgb]{1,0,0}{[7]}}}$
${\textbf{\rm\color[rgb]{0,0,1}{in}}_{\color[rgb]{0,0,1}{[26]}}}$ Belgian
${\textbf{\rm\color[rgb]{0,1,0}{bus}}_{\color[rgb]{0,1,0}{[16]}}}$
${\textbf{\rm\color[rgb]{1,0,0}{crash}}_{\color[rgb]{1,0,0}{[7]}}}$ .
S1: Mugabe ${\textbf{\rm\color[rgb]{1,0,0}{declared}}_{\color[rgb]{1,0,0}{[20]}}}$ ${\textbf{\rm\color[rgb]{1,0,0}{winner}}_{\color[rgb]{1,0,0}{[20]}}}$ of disputed ${\textbf{\rm\color[rgb]{1,0,0}{elections}}_{\color[rgb]{1,0,0}{[20]}}}$ | 4.0
S2: Zimbabwe Mugabe
${\textbf{\rm\color[rgb]{1,0,0}{declared}}_{\color[rgb]{1,0,0}{[20]}}}$
${\textbf{\rm\color[rgb]{1,0,0}{winner}}_{\color[rgb]{1,0,0}{[20]}}}$ in
disputed ${\textbf{\rm\color[rgb]{1,0,0}{vote}}_{\color[rgb]{1,0,0}{[20]}}}$

Table 4: Case Study on STS-B datasets.

###  4.6 Joint Training vs Multistage Training

After we pretrained the NTM, there are two ways of training our topic-enhanced
VQ-VAE. One is to utilize the topic embeddings from the pretrained NTM as the
codebook initialization in VQ-VAE and train the latter while holding the
former fixed (denoted by Multistage in Table 3), the other is to joint train
the pretrained NTM and VQ-VAE (denoted by Joint). The results of these two
training methods derived from the six STS datasets are shown in Table 3. We
can see that compared with multistage training, joint training yields better
results on all datasets for both DisBert and DisRoBERTa. The comparison shows
that jointly training the topic embedding matrix ${\bf\it\phi}$ benefits the
STS task.

###  4.7 Results with Varying Hyperparameters.

Impact of Training Data Amounts. In figure 3, we compared the results of Bert
and DisBert with different training data. We randomly selected 20% to 100%
data from the training set as training data. We can observe that DisBert
outperforms Bert consistently across all training data sizes, which suggests
that we can leverage the topic-enhanced discrete variable in all data sizes,
even when the training data is scarce.

Impact of Topic Numbers. Figure 4 shows the performance of DisBert given
varying topic numbers. As we can see, the curves on all datasets are not
monotonic and the best accuracy is achieved with different numbers of topics,
e.g., $k=30$ on the MRPC dataset. We also found that on larger training
datasets DisBert requires larger topic numbers and vice versa, e.g., $k=90$
for Quora and $k=30$ for SemEval-B. It is not unexpected, as larger datasets
can cover more topics. However, how to let the data choose the right number of
topics in NTM is beyond the scope of this paper.

Impact of Bert Layers. Figure 5 shows the effect of semantics-driven multi-
head attention on different layers. We found that the aforesaid attention is
not always helpful for all layers and best performances are achieved when
applying such attention to the last layer of DisBert/DisRoBERTa on MRPC
datasets. Meanwhile, the performance drops when such attention is applied to
all layers.

Dataset | Topic Model | VQ-VAE | Semantics-driven Bert | Total | Bert
---|---|---|---|---|---
MRPC | 247 | 218 | 220 | 685 | 200
Quora | 3,512 | 5,123 | 18,232 | 26,927 | 17,805
STS-B | 197 | 186 | 239 | 622 | 214

Table 5: Limitation—Training time (in seconds) of different models in three
datasets. We trained the topic model for 500 epochs, VQ-VAE for 20 epochs,
semantics-driven Bert, and Bert for 10 epochs. All models are trained in the
same environment with a single GeForce RTX 3090 card.

###  4.8 Case Studies

In Table 4, we conduct a case study on STS-B datasets to show how the model
works with topical information. The left part of the table shows four word
clusters generated by Topic-enhanced VQ-VAE. As described in Section 3.2, the
topic-enhanced VQ-VAE model clusters words in the discrete space given by the
codebook. It can be seen that words in the same cluster form a meaningful
topic: Cluster 7 is about protest, Cluster 6 is about transportation, Cluster
20 is about government election, and Cluster 26 is one of the clusters
capturing all the functional words. The right part of the table shows two
pairs of sentences used in the STS tasks with the assignments of the four
topics to their words. We can observe that sentences, which are predicted to
be similar by our model, tend to have more words assigned to similar topics,
even if those words are morphologically different. For instance, “Killed” and
“dead” are both from Cluster 7, “coach” and “bus” are from Cluster 16, and
“elections” and “vote” are from Cluster 20. The heat map calculated by
semantics-driven multi-head attention is shown on Appendix A.3 for more
discussion.

##  5 Conclusion

In this paper, we developed a topic-enhanced VQ-VAE model to effectively train
discrete latent variables by informing the vector quantization with semantics
(i.e., latent topics) learned from a broader context by a neural topic model.
Then we further designed a semantics-driven multi-head attention mechanism for
enriching the contextual embeddings learned by Transformer with topical
information, which is calculated based on the quantized representations from
topic-enhanced VQ-VAE. Its plug-and-play characteristic allows it to be
readily incorporated into various transformed-based language models. Through
experiments on different scale datasets for the STS task, we proved that
building the semantic information learned by NTM via vector quantization into
the multi-head attention block can improve the STS performance of transformer-
based language models.

##  6 Limitations

The limitations of this work, to the best of our knowledge, can be summarized
into two aspects:

(1) Compared to the origin Bert/RoBERTa, our model DisBert/DisRoBERTa needs a
longer time to train since there are three components in our framework, i.e.,
topic model, VQ-VAE model, and the transformer-based model. From table 5, we
can observe that in a small dataset like MRPC and STS-B, the training time is
almost three times longer. While in the large datasets Quora, our model took
an extra 2.5 hours. Therefore in future work, we need to improve the
efficiency of our model.

(2) The improvement of our method is limited when sentences are long enough or
training datasets are large, e.g., Quora and SemEvalC shown in table 2. Our
model is capable of complete global semantics, hence it works better for small
datasets and short sentences which contain limited semantics.

## Acknowledgments

This work is supported by the National Natural Science Foundation of China
(No.61976102 and No.U19A2065) and the Science and Technology Development
Program of Jilin Province (No.20210508060RQ), and the Fundamental Research
Funds for the Central Universities, JLU.

## References

  * Bao et al. (2020) Siqi Bao, Huang He, Fan Wang, Hua Wu, and Haifeng Wang. 2020.  PLATO: Pre-trained dialogue generation model with discrete latent variable.  In _ACL_ , pages 85–96.
  * Blei et al. (2003) David M Blei, Andrew Y Ng, and Michael I Jordan. 2003.  Latent dirichlet allocation.  _Journal of machine Learning research_.
  * Bordes et al. (2014) Antoine Bordes, Sumit Chopra, and Jason Weston. 2014.  Question answering with subgraph embeddings.  In _EMNLP_ , pages 615–620.
  * Chen et al. (2017) Qian Chen, Xiaodan Zhu, Zhen-Hua Ling, Si Wei, Hui Jiang, and Diana Inkpen. 2017\.  Enhanced LSTM for natural language inference.  In _ACL_ , pages 1657–1668, Vancouver, Canada. Association for Computational Linguistics.
  * Chorowski et al. (2019) Jan Chorowski, Ron J Weiss, Samy Bengio, and Aäron van den Oord. 2019.  Unsupervised speech representation learning using wavenet autoencoders.  _IEEE/ACM transactions on audio, speech, and language processing_ , 27(12):2041–2053.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _NAACL_ , pages 4171–4186.
  * Dolan and Brockett (2005) William B Dolan and Chris Brockett. 2005.  Automatically constructing a corpus of sentential paraphrases.  In _IWP_.
  * Guo et al. (2020a) Dandan Guo, Bo Chen, Ruiying Lu, and Mingyuan Zhou. 2020a.  Recurrent hierarchical topic-guided rnn for language generation.  In _ICML_ , pages 3810–3821.
  * Guo et al. (2020b) Daya Guo, Duyu Tang, Nan Duan, Jian Yin, Daxin Jiang, and Ming Zhou. 2020b.  Evidence-aware inferential text generation with vector quantised variational autoencoder.  In _ACL_ , pages 6118–6129.
  * Hu et al. (2014) Baotian Hu, Zhengdong Lu, Hang Li, and Qingcai Chen. 2014.  Convolutional neural network architectures for matching natural language sentences.  In _NeurIPS_ , pages 2042–2050.
  * Huang and Ji (2020) Lifu Huang and Heng Ji. 2020.  Semi-supervised new event type induction and event detection.  In _EMNLP_ , pages 718–724.
  * Jin et al. (2020) Shuning Jin, Sam Wiseman, Karl Stratos, and Karen Livescu. 2020.  Discrete latent variable representations for low-resource text classification.  In _ACL_ , pages 4831–4842.
  * Kaiser et al. (2018) Lukasz Kaiser, Samy Bengio, Aurko Roy, Ashish Vaswani, Niki Parmar, Jakob Uszkoreit, and Noam Shazeer. 2018.  Fast decoding in sequence models using discrete latent variables.  In _ICML_ , pages 2390–2399.
  * Kingma and Welling (2013) Diederik P Kingma and Max Welling. 2013.  Auto-encoding variational bayes.  _ICLR_.
  * Lan et al. (2020) Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2020.  Albert: A lite bert for self-supervised learning of language representations.  In _ICLR_.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arxiv_.
  * Lu et al. (2021) Ruiying Lu, Bo Chen, Dan dan Guo, Dongsheng Wang, and Mingyuan Zhou. 2021.  Topic-aware contextualized transformers.
  * Miao et al. (2017) Yishu Miao, Edward Grefenstette, and Phil Blunsom. 2017.  Discovering discrete latent topics with neural variational inference.  In _ICML_.
  * Miao et al. (2016) Yishu Miao, Lei Yu, and Phil Blunsom. 2016.  Neural variational inference for text processing.  In _ICML_.
  * Mohamed and Oussalah (2019) Muhidin Mohamed and Mourad Oussalah. 2019.  Srl-esa-textsum: A text summarization approach based on semantic role labeling and explicit semantic analysis.  _Information Processing & Management_, 56(4):1356–1372.
  * Nakov et al. (2017) Preslav Nakov, Doris Hoogeveen, Lluís Màrquez, Alessandro Moschitti, Hamdy Mubarak, Timothy Baldwin, and Karin Verspoor. 2017.  Semeval-2017 task 3: Community question answering.  _SemEval-2017 workshop_.
  * Nakov et al. (2015) Preslav Nakov, Lluís Màrquez, Walid Magdy, Alessandro Moschitti, James Glass, and Bilal Randeree. 2015.  Semeval-2015 task 3: Answer selection in community question answering.  _SemEval-2015 workshop_.
  * Nakov et al. (2016) Preslav Nakov, Lluís Màrquez, Alessandro Moschitti, Walid Magdy, Hamdy Mubarak, Abed Alhakim Freihat, Jim Glass, and Bilal Randeree. 2016.  SemEval-2016 task 3: Community question answering.  In _SemEval-2016 workshop_.
  * Oord et al. (2017) A Oord, Oriol Vinyals, and Koray Kavukcuoglu. 2017.  Neural discrete representation learning.  In _NeurIPS_ , pages 6309–6318.
  * Peinelt et al. (2020) Nicole Peinelt, Dong Nguyen, and Maria Liakata. 2020.  tBERT: Topic models and BERT joining forces for semantic similarity detection.  In _ACL_ , pages 7047–7055.
  * Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. 2017.  Attention is all you need.  In _NeurIPS_.
  * Wang and YANG (2020) Xinyi Wang and YI YANG. 2020.  Neural topic model with attention for supervised learning.  In _AISTATS_ , pages 1147–1156.
  * Wang et al. (2019) Yue Wang, Jing Li, Hou Pong Chan, Irwin King, Michael R Lyu, and Shuming Shi. 2019\.  Topic-aware neural keyphrase generation for social media language.  In _ACL_ , pages 2516–2526.
  * Wang et al. (2020) Zhengjue Wang, Zhibin Duan, Hao Zhang, Chaojie Wang, Long Tian, Bo Chen, and Mingyuan Zhou. 2020.  Friendly topic assistant for transformer based abstractive summarization.  In _EMNLP_ , pages 485–497.
  * Wang et al. (2017) Zhiguo Wang, Wael Hamza, and Radu Florian. 2017.  Bilateral multi-perspective matching for natural language sentences.  In _IJCAI_ , pages 4144–4150.
  * Xia et al. (2021) Tingyu Xia, Yue Wang, Yuan Tian, and Yi Chang. 2021.  Using prior knowledge to guide bert’s attention in semantic textual matching tasks.  _Proceedings of the Web Conference 2021_.
  * Yan et al. (2021) Wilson Yan, Yunzhi Zhang, Pieter Abbeel, and Aravind Srinivas. 2021.  Videogpt: Video generation using vq-vae and transformers.  _arXiv_.
  * Yin et al. (2016) Wenpeng Yin, Hinrich Schütze, Bing Xiang, and Bowen Zhou. 2016.  Abcnn: Attention-based convolutional neural network for modeling sentence pairs.  _TACL_.
  * Zeng et al. (2018) Jichuan Zeng, Jing Li, Yan Song, Cuiyun Gao, Michael R Lyu, and Irwin King. 2018\.  Topic memory networks for short text classification.  In _EMNLP_ , pages 3120–3131.
  * Zhang et al. (2019) Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019.  Ernie: Enhanced language representation with informative entities.  _ACL_ , pages 1441–1451.
  * Zhang et al. (2020) Zhuosheng Zhang, Yuwei Wu, Hai Zhao, Zuchao Li, Shuailiang Zhang, Xi Zhou, and Xiang Zhou. 2020.  Semantics-aware bert for language understanding.  In _AAAI_ , pages 9628–9635.
  * Zhao et al. (2017) He Zhao, Lan Du, and Wray L. Buntine. 2017.  A word embeddings informed focused topic model.  In _ACML_.
  * Zhao et al. (2021a) He Zhao, Dinh Phung, Viet Huynh, Yuan Jin, Lan Du, and Wray Buntine. 2021a.  Topic modelling meets deep neural networks: A survey.  In _IJCAI Survey Track_.
  * Zhao et al. (2021b) He Zhao, Dinh Phung, Viet Huynh, Trung Le, and Wray Buntine. 2021b.  Neural topic model via optimal transport.
  * Zou et al. (2013) Will Y. Zou, Richard Socher, Daniel Cer, and Christopher D. Manning. 2013.  Bilingual word embeddings for phrase-based machine translation.  In _EMNLP_ , pages 1393–1398.

##  Appendix A Example Appendix

###  A.1 Dataset Examples

Table 6 shows examples from different datasets. Labels indicate if the second
sentence is a paraphrase (for paraphrasing tasks) or relevant (for QA tasks).
In STS-B datasets, the label stands for sentence pair similarity score, and
the higher score, the more similar the two sentences are.

Dataset | Sentence Pair | Label
---|---|---
MRPC | S1:The world’s two largest automakers said their U.S. sales declined more than predicted last month | 1
as late summer sales frenzy caused more of an industry backlash than expected.
S2: Domestic sales at both GM and No. 2 Ford Motor Co. declined more than
predicted as a late
summer sales frenzy prompted a larger-than-expected industry backlash.
Quora | S1: Currently , all Supreme Court Justices come from very elite law schools, is it similar for the best | 1
lawyers in private practice .
S2: What ’s your type of jungle -LRB- concrete or nature -RRB- and why ?
STS-B | S1: A man is spreading shreded cheese on a pizza. | 3.8
S2: A man is spreading shredded cheese on an uncooked pizza.
SemEval-A | S1: Massage oil is there any place i can find scented massage oils in qatar? | 0
S2: Whats the name of the shop?
SemEval-B | S1: Music tastes so; what kind of music do you like? | 0
S2: After school program and summer camp Can anyone please tell me if there is
any after school program
and summer camp in Doha? – for a grade two girl (seven years old).
SemEval-C | S1: Best sunglass store in Qatar Can somebody suggest the best store where i can | 1
get the best sunglasses for ladies. Where can i find a store with the best
variety?
S2: Fashion wear - Primark Lingerie - Agent Provocateur Lingerie - Intimo
Lingerie - Peach John

Table 6: Dataset Examples.

###  A.2 Hyper-Parameter settings

Table 7 shows the hyperparameter we chose in our model. All hyper-parameters
were chosen through greedy search based on development set performance.

For topic model, we choose number of topics in (20, 30, 40, 50, 60, 70, 80,
90, 100, 150, 200), batch size in (128, 256, 512), and learning rate in (1e-3,
2e-3, 3e-3).

For the VQ-VAE model, we choose the batch size in (32, 64, 128), and the
learning rate in (2e-5, 3e-5).

For the Bert model, We choose the batch size in (32, 64, 128), and the
learning rate in (2e-5, 3e-5).

Models | MRPC | Quora | STS-B | SemEval |
---|---|---|---|---|---
A | B | C |
Topic Model |  |  |  |  |  |  |
Topic number $K$ | 30 | 90 | 40 | 80 | 30 | 70 |
Vocabulary size $V$ | 5,179 | 37,013 | 4,369 | 12,102 | 3,763 | 13,272 |
Topic Embedding hidden size $E$ | 64 | 64 | 64 | 64 | 64 | 64 |
Batch size | 256 | 512 | 256 | 256 | 256 | 256 |
Epochs | 500 | 150 | 500 | 200 | 500 | 200 |
Learning rate | 1e-3 | 2e-3 | 1e-3 | 1e-3 | 1e-3 | 1e-3 |
VQ-VAE Model |  |  |  |  |  |  |
Max sequence length | 128 | 128 | 128 | 256 | 256 | 256 |
Vocabulary size | 10,734 | 61,040 | 9513 | 31,060 | 8,259 | 31,060 |
Batch size | 32 | 128 | 32 | 64 | 32 | 64 |
Epochs | 10 | 10 | 10 | 10 | 10 | 10 |
Learning rate | 2e-5 | 3e-5 | 2e-5 | 2e-5 | 2e-5 | 2e-5 |
DisBert/DisRoBERTa Model |  |  |  |  |  |  |
Max sequence length | 128 | 128 | 128 | 256 | 256 | 256 |
Batch size | 32 | 128 | 32 | 32 | 32 | 64 |
Epochs | 10 | 5 | 10 | 10 | 10 | 10 |
Learning rate | 2e-5 | 3e-5 | 2e-5 | 3e-5 | 2e-5 | 3e-5 |

Table 7: Parameter settings.

###  A.3 Heat Map

Figure 6 shows the heat map of sentence pair 1 in the case study, which is
calculated by our semantics-driven multi-head attention. We can observe that
words from the same cluster usually pay high attention to each other because
they correspond to the same hidden vector in the codebook. Such attention
could help Bert model focus on the words which may have similar semantics.

Figure 6: Heat Map calculated by semantics-driven multi-head attention.