Also at ]Technische Universität Dresden, Institut für Theoretische Physik,
01069 Dresden, Germany

Also at ]Technische Universität Dresden, Institut für Theoretische Physik,
01069 Dresden, Germany

Also at ]Technische Universität Dresden, Institut für Theoretische Physik,
01069 Dresden, Germany

# Tailoring the escape rate of a Brownian particle by combining a vortex flow
with a magnetic field

I. Abdoli  [  Leibniz-Institut für Polymerforschung Dresden, Institut Theorie
der Polymere, 01069 Dresden, Germany<EMAIL_ADDRESS>H. Löwen  Institut für
Theoretische Physik II: Weiche Materie, Heinrich-Heine-Universität Düsseldorf,
Düsseldorf, 40225, Germany  J.-U. Sommer  [  Leibniz-Institut für
Polymerforschung Dresden, Institut Theorie der Polymere, 01069 Dresden,
Germany  A. Sharma  [  Leibniz-Institut für Polymerforschung Dresden, Institut
Theorie der Polymere, 01069 Dresden, Germany

###### Abstract

The probability per unit time for a thermally activated Brownian particle to
escape over a potential well is in general well-described by Kramers theory.
Kramers showed that the escape time decreases exponentially with increasing
barrier height. The dynamics slow down when the particle is charged and
subjected to a Lorentz force due to an external magnetic field. This is
evident via a rescaling of the diffusion coefficient entering as a prefactor
in the Kramers escape rate without any impact on the barrier-height-dependent
exponent. Here we show that the barrier height can be effectively changed when
the charged particle is subjected to an external vortex flow. While the
external vortex alone does not affect the mean escape time of the particle,
when combined with a magnetic field it effectively pushes the fluctuating
particle either radially outside or inside depending on its sign relative to
that of the magnetic field. In particular, the effective potential over which
the particle escapes can be changed to a flat, a stable, and an unstable
potential by tuning the signs and magnitudes of the external vortex and the
applied magnetic field. Notably, the last case corresponds to enhanced escape
dynamics.

††preprint: AIP/123-QED

##  I Introduction

A Brownian particle undergoes erratic motion as a result of its collisions
with the solvent molecules. If the particle is being initially put at the
bottom of a potential well, the thermal activation of the particle may cause
an escape from the potential well over an energetic barrier. Using the flux-
over-population method Farkas (1927), Kramers first derived the escape rate of
a Brownian particle over an energy barrier moving in a bistable potential,
regardless of what happens after this escape Kramers (1940). He showed that
the probability per unit time for the particle to escape the potential well
exponentially decays with the height of the energy barrier. Kramers derived
limiting expressions for weak friction and strong damping and realized a
global maximum at some intermediate value of the damping, which is known as
Kramers turnover Grabert and Linkwitz (1988); McCann, Dykman, and Golding
(1999); Rondin  _et al._ (2017). The problem has been generalized to include
memory friction Grote and Hynes (1980); Carmeli and Nitzan (1982); Ianconescu
and Pollak (2015) and athermal fluctuations Hänggi, Marchesoni, and Grigolini
(1984); Jung and Hänggi (1988); Sharma, Wittmann, and Brader (2017); Scacchi,
Brader, and Sharma (2019); Caprini  _et al._ (2019) and was extended to
quantum field theory Berera  _et al._ (2019); Darmé, Jaeckel, and Lewicki
(2019).

While Kramers’ framework and its extensions have been thoroughly studied with
the relevant deterministic potential force fields Fleming and Hanggi (1993);
Talkner and Hänggi (1995); Hanggi and Mojtabai (1982); Pollak (1986); Pollak,
Grabert, and Hänggi (1989); Pollak and Ankerhold (2013), much less is known
when the deterministic force is nonconservative, namely when it is not of
potential type Baura, Ray, and Bag (2013); Mondal  _et al._ (2018); Filliger
and Reimann (2007). Recently, by taking into account a nonconservative force
in the form of Lorentz force, we have studied the escape dynamics of a two-
dimensional Brownian system with broken spatial symmetry via two noises with
different strengths Abdoli  _et al._ (2022a). We have shown that while the
escape process becomes anisotropic (i.e. particles tend to escape the
potential well more along the axis with larger noise strength) due to two
different noises, when subjected to an external magnetic field, the spatial
symmetry can be restored Abdoli  _et al._ (2022a). However, to our knowledge,
it is expected that the escape process is reduced (or unaffected in the
direction of the applied magnetic field) by external constant magnetic fields
Filliger and Reimann (2007); Abdoli  _et al._ (2022a) which is evident via a
rescaling of the diffusion coefficient. It has been shown that the combined
influence of a nonconservative force and a magnetic field may cause an
instability in the system Lee and Kwon (2019). Here, taking advantage of such
an instability, we show that the Lorentz force due to a constant magnetic
field can result in enhanced escape dynamics.

In this work, we study the escape dynamics of a Brownian particle from a
harmonic trap which is cut-off at a certain distance in the presence of an
external vortex and the Lorentz force due to an external constant magnetic
field. Taking advantage of the spatial isotropy in the system we derive an
exact expression for the mean first passage time. While the external vortex
alone does not affect the escape dynamics, we observe a nontrivial result when
an external magnetic field is present: the mean first passage time can be
reduced or enhanced. This is attributed to the shape change of the effective
potential well. By tuning the external magnetic field or alternatively the
strength of the external vortex the effective potential can change shape to a
flat, a stable, or an unstable potential. This means that by tuning either
parameters the barrier energy over which the particle may escape can be
effectively altered to a smaller or larger one whose origin can be understood
as follows: the combination of the vortex flow and the magnetic field
effectively pushes the fluctuating particle either radially outside or inside
depending on their signs. In other words, the combination of the two fields,
which individually induces no radial force, gives rise to a radial force.In
what follows, we first introduce the model. Next we calculate the mean first
passage time, which can be written in terms of an effective potential. We then
study the trends of the escape time with respect to the magnetic field
strength and the vortex flow and finally we discuss several experimental
realizations of the set-up considered here.

##  II Model

We consider an overdamped charged Brownian particle with the charge $q$
subjected to an external magnetic field $B$ in the $-\hat{z}$ direction. Since
the Lorentz force due to the field does not affect the motion of the particle
in the $z$ direction, we effectively reduce the system to a two-dimensional
one and study the motion of the particle in the $xy$ plane. The particle is
trapped in an isotropic potential $U(x,y)=k(x^{2}+y^{2})/2$ and undergoes a
vortex flow due to the nonconservative force
$\bm{F}_{nc}=\epsilon(-y,x)^{\top}$. Here $k$ and $\epsilon$ are the stiffness
of the potential and the strength of the nonconservative force, respectively.
A schematic of the system is shown in Fig. 1. It is experimentally and
theoretically known that even statically optically trapped Brownian particles
in the overdamped limit represent nonequilibrium behavior characterized by
Brownian vortices. This is due to the nonconservative forces generated by
optical scattering forces Sun, Grier, and Grosberg (2010); Sun  _et al._
(2009); Roichman  _et al._ (2008); Moyses  _et al._ (2015). Moreover, by
applying a prescribed external vortex flow field such as a rotating bucket to
an underdamped Brownian particle one can induce similar terms to the
nonconservative force, i.e. $-\epsilon y$ and $\epsilon x$ Liebchen and Löwen
(2019).

It has been shown that the overdamped dynamics of the particle derived by
simply setting the inertia term to zero can yield an incorrect description in
the presence of a magnetic field Vuijk, Brader, and Sharma (2019). In this
case, the overdamped Langevin equation describing dynamics of the system can
be derived using the low-mass approach Chun, Durang, and Noh (2018); Abdoli
_et al._ (2022b, a), which can be written as

Figure 1: A single charged particle diffusing in a two-dimensional harmonic
potential $U(x,y)=k(x^{2}+y^{2})/2$, shown by concentric contours, with $k$
being its stiffness. The particle is subjected to an external magnetic field
$B$ in the $-\hat{z}$ direction and a nonconservative force
$\bm{F}_{nc}=\epsilon(-y,x)$ with $\epsilon$ being its strength. The
nonconservative force is shown for $\epsilon>0$. The particle can escape the
trap when reaches the boundary, truncated at $r=a$, shown by dashed circle,
where $r=\sqrt{x^{2}+y^{2}}$ is the distance from the origin.

$\displaystyle\dot{x}$ $\displaystyle=$
$\displaystyle\frac{1}{\gamma(1+\kappa^{2})}\left[-kx-\epsilon y+k\kappa
y-\epsilon\kappa x\right]+\xi_{x}(t),$ (1) $\displaystyle\dot{y}$
$\displaystyle=$
$\displaystyle\frac{1}{\gamma(1+\kappa^{2})}\left[-ky+\epsilon x-k\kappa
x-\epsilon\kappa y\right]+\xi_{y}(t),$ (2)

where $\gamma$ is the friction coefficient and $\kappa=qB/\gamma$ is the
diffusive Hall parameter quantifying the strength of the Lorentz force
relative to the frictional force. We note that $\kappa$ can be positive or
negative depending on the sign of the applied magnetic field. Here
$\bm{\xi}(t)=(\xi_{x},\xi_{y})^{\top}$ is Gaussian nonwhite noise with zero
mean and time correlation
$\langle\bm{\xi}(t)\bm{\xi}^{\top}(t^{\prime})\rangle=T\bm{G}^{-1}\delta_{+}(t-t^{\prime})+T(\bm{G}^{-1})^{\top}\delta_{-}(t-t^{\prime})$
where $T$ is the temperature,
$\bm{G}=\gamma\bigl{(}\begin{smallmatrix}1&\kappa\\\
-\kappa&1\end{smallmatrix}\bigr{)}$, and the notations
$\delta_{\pm}(s=t-t^{\prime})$ are the modified Dirac delta functions which
are zero for $s\neq 0$ while
$\int_{0}^{\infty}\mathop{}\\!\mathrm{d}s\delta_{+}(s)=\int_{-\infty}^{0}\mathop{}\\!\mathrm{d}s\delta_{-}(s)=1$
and
$\int_{0}^{\infty}\mathop{}\\!\mathrm{d}s\delta_{-}(s)=\int_{-\infty}^{0}\mathop{}\\!\mathrm{d}s\delta_{+}(s)=0$.
Throughout this work we set the Boltzmann constant $k_{B}$ to unity. Length
and time are measured in units of $\sqrt{T/k}$ and $\gamma/k$, respectively.

We use Itô calculus to reduce the Langevin equations in Eq. (1) and Eq. (2) to
a one-dimensional problem for the variable $r=\sqrt{x^{2}+y^{2}}$, which is
given as Gardiner (2009)

$\mathop{}\\!\mathrm{d}r=\frac{1}{1+\kappa^{2}}\left[-\frac{k+\epsilon\kappa}{\gamma}r+\frac{D}{r}\right]\mathop{}\\!\mathrm{d}t+\sqrt{\frac{2D}{1+\kappa^{2}}}\eta(t)\mathop{}\\!\mathrm{d}t,$
(3)

where $D=T/\gamma$ is the coefficient of a freely diffusing particle and
$\eta(t)$ is Gaussian white noise with zero mean and the Dirac delta time
correlation $\langle\eta(t)\eta(t^{\prime})\rangle=\delta(t-t^{\prime})$. The
terms in the square brackets on the right hand side of Eq.(3) describe the
force on the the particle due to an effective potential, given as

Figure 2: The effective potential from Eq. (4) for different values of the
stiffness $k_{eff}=k+\epsilon\kappa$. By varying the parameter $\kappa$ (or
$\epsilon$) the effective potential can change shape to a stable one if
$k_{eff}>0$, a flat one if $k_{eff}=0$, or to an unstable one if $k_{eff}<0$.

$U_{eff}(r)=\frac{k_{eff}}{2\gamma(1+\kappa^{2})}r^{2}-\frac{D}{1+\kappa^{2}}\log(r),$
(4)

where $k_{eff}=k+\epsilon\kappa$ is the stiffness of the effective potential.
As it is evident from the effective stiffness, in the absence of the vortex
flow the potential simply gets rescaled by the factor $1/(1+\kappa^{2})$, as
we have shown in the supplemental information of Ref. Abdoli  _et al._
(2022a). Moreover, in the absence of the magnetic field there is no effect of
the vortex flow on the effective potential and Eq. (4) reduces to the well
known results in Ref.Gardiner (2009) for a rotationally symmetric Ornstein-
Uhlenbeck process in two dimensions. The second term on the right and side
comes from the transformation to $r$ and corresponds to an extremely repulsive
potential at the origin due to reduced number of states on the circle of
radius $r$. This term influences the motion of the particle only near the
origin and is negligible for larger distances as compared to the first term.

Figure. 2 represents the scaled effective potential from Eq. (4) for different
values of the parameter $k_{eff}$ without the logarithmic term. By tuning the
diffusive Hall parameter or alternatively the strength of the nonconservative
force, the effective potential changes shape: the potential is stable if
$k_{eff}>0$, flat if $k_{eff}=0$, and unstable if $k_{eff}<0$. It becomes
simple quadratic potential in the absence of $\epsilon$ or/and $\kappa$.

##  III Mean Escape Time

Figure 3: The mean escape time as a function of the diffusive Hall parameter
$\kappa$ from Eq. (5) and Eq. (7) for different values of the scaled barrier
height $\beta\Delta E$ with $\beta=1.0$, $\gamma=1.0$ and $\epsilon=0.2$.
Obviously the mean escape time increases with increasing the barrier height.
It can increase or decrease by tuning the parameter $\kappa$: the presence of
an external vortex field can work together with the applied magnetic field to
effectively push the fluctuating particle either radially outside, if
$\kappa<-k/\epsilon$, or inside, if $\kappa>-k/\epsilon$. The former
corresponds to the case in which the combination helps the particle to escape.
The point $\kappa=0$ corresponds to unaffected escape time by the external
vortex (i.e. $k_{eff}=k$). In the inset, we show the mean escape time which is
scaled by the mean escape time in the absence of the external vortex $\langle
t_{0}\rangle$ where the subscript $0$ indicates zero strength length of the
vortex flow. It implies that the mean escape time can decrease with increasing
$\kappa$ as compared to the mean escape time without the vortex flow.

We consider a particle which is trapped in an isotropic potential $U(x,y)$
which taking advantage of the spatial symmetry whose distance from the origin,
$r=|\bm{r}|$, can be described by Eq. (3). We are interested in the mean time
at which the particle reaches the boundary, truncated at $r=a$, as shown in
Fig. 1. As we show in the Appendix A, the mean escape time can be exactly
calculated from Eq. (3) which reads

$\langle
t\rangle=\frac{\gamma(1+\kappa^{2})}{2k_{eff}}\left[\mathrm{Ei}\left(\beta\Delta
E_{eff}\right)-\log\left(\beta\Delta E_{eff}\right)-\gamma_{EM}\right],$ (5)

if $k_{eff}>0$ corresponding to the effective stable potential and

$\langle
t\rangle=\frac{\gamma(1+\kappa^{2})}{2k_{eff}}\left[-\mathrm{Ei}\left(-\beta|\Delta
E_{eff}|\right)+\log\left(\beta|\Delta E_{eff}|\right)+\gamma_{EM}\right],$
(6)

if $k_{eff}<0$ corresponding to the unstable effective potential where $\beta$
is the inverse of the temperature, $\gamma_{EM}$ is the Euler-Mascheroni
constant, and $\mathrm{Ei}(x)$ is the exponential integral. Here $\Delta
E_{eff}=\Delta E+\epsilon\kappa a^{2}/2$ is the effective barrier energy which
is the real barrier height $\Delta E=ka^{2}/2$ augmented by the coupling
between the magnitude of the applied magnetic field and the strength of the
external vortex. Using the series expansion of the exponential integral at
$k_{eff}=0$ for Eq. (5) and Eq. (6), the mean escape time for the effective
flat potential reads

$\langle t\rangle\sim\frac{(1+\kappa^{2})}{4D}a^{2},$ (7)

which is the mean escape time for a freely diffusing particle scaled by
$1+\kappa^{2}$. In the limit of large barrier heights the exponential integral
in Eq. (5) can be expanded and as a consequence the mean escape time reduces
to $\langle t\rangle\sim\gamma(1+\kappa^{2})\exp(\beta\Delta
E_{eff})/(2k_{eff}\beta\Delta E_{eff})$. In the absence of the external
vortex, which corresponds to $\epsilon=0$, the result reduces to the Kramers
result rescaled by $1+\kappa^{2}$ arising from the trivial rescaling of the
diffusion coefficient. The expression becomes the same as the Kramers one when
the magnetic field is absent $\kappa=0$. This confirms that the external
vortex field alone does not affect the mean escape time. The intuitive reason
for that is that, for $\kappa=0$, the presence of a vortex field only changes
the azimuthal motion but not the radial one which leaves the redial particle
escape unaffected.

Figure 4:  The mean escape time with respect to the strength of the
conservative force $\epsilon$ from Eq. (5) and Eq. (7) for different values of
the scaled barrier heights with $\beta=1.0$ and $\gamma=1.0$. The lines with
circles and squares correspond to the results with $\kappa=2.0$ and
$\kappa=-2.0$, respectively. The mean escape time can increase or decrease
with increasing the strength of the vortex flow, which depends on its sign
relative to that of $\kappa$ and their magnitude compared to the stiffness of
the potential $k$.

Figure 3 shows the mean escape time with respect to the diffusive Hall
parameter $\kappa$. Obviously it takes the particle longer time to escape over
larger barrier heights as is evident in the figure. The magnetic field
together with the vortex flow creates additional fluctuations in radial
direction which can be directed either outwards or inwards depending on its
sign. The former corresponds to the case in which the combination of the
vortex flow and the magnetic field helps the particle to escape. The inset
shows the mean escape time scaled by the mean escape time in the absence of
the external vortex, which is indicated by the subscript $0$. The mean escape
time can decrease with increasing magnetic field as compared to the mean
escape time without the vortex flow and remains almost constant for small
barrier height.

In Fig.4, we show that tuning the strength of the vortex flow is an
alternative way to vary the mean escape time which is evident in Eq.(5) and
Eq.(6) via the production of the two parameters, i.e. $\epsilon\kappa$.
Therefore the similar trends are expected. The figure represents the mean
escape time with respect to the parameter $\epsilon$ for a system with
$\kappa=2.0$, denoted by lines with circles, and a system with $\kappa=-2.0$,
denoted by lines with squares. Our results imply that the mean escape time can
be decreased or increased by tuning the vortex flow strength depending on its
sign relative to that of the magnetic field and their magnitude compared to
the stiffness of the potential $k$.

##  IV Discussion

In this work, we studied the effect of a vortex flow on the escape dynamics of
a Brownian magneto-system made of single charged Brownian particle subjected
to an external magnetic field. We expressed the potential in an effective form
which can change shape to a stable, a flat, or an unstable potential depending
on the stiffness of the effective potential. Taking advantage of the spatial
isotropy in the system we obtained an exact expression for the mean escape
time. In the absence of the external vortex, exerted by the nonconservative
force, the Lorentz force due to the external magnetic field slows down the
dynamics of the system without any qualitative change, which is evident via
the trivial rescaling of the diffusion coefficient. We showed that while the
external vortex alone does not affect the mean escape time, when coupled to
the magnetic field it can enhance or reduce the escape time: this is intuitive
as the magnetic field together with the vortex flow creates additional
fluctuations in radial direction which can be directed either outwards or
inwards depending on its sign. In other words, the combination of the two
fields, which individually induces no radial force, gives rise to a radial
force. We showed that the barrier over which the particle escapes can be
effectively changed to a larger or smaller one depending on the relative signs
of the strength of the vortex flow and the applied magnetic field as well as
their magnitude compared to the stiffness of the potential in which the
particle is trapped. Moreover, the trap can be effectively switched-off by an
appropriate sign and value of the magnetic field.

A possible experimental realisation is to trap the particle using optical
tweezers either in a radio-frequency plasma sheath with a vertical magnetic
field Carstensen  _et al._ (2009); Piel (2017) or in a rotating frame of
reference. By rotating the reference frame a Coriolis force can be induced
which acts the same as the Lorentz force due to an external magnetic field
Kählert  _et al._ (2012); Hartmann  _et al._ (2013, 2019). As it has been
shown that even statically optically trapped Brownian particles undergoe a
nonconservative force induced by optical scattering forces Sun, Grier, and
Grosberg (2010); Sun  _et al._ (2009); Roichman  _et al._ (2008); Moyses  _et
al._ (2015); Mangeat  _et al._ (2019), we expect that the study of the
enhanced escape dynamics does not require an additional external vortex.
Another possibility is to apply a rotating bucket to an underdamped Brownian
particle which induces similar terms to the nonconservative force in the
overdamped limit Liebchen and Löwen (2019).

From a future perspective, it could be interesting to study the escape
dynamics of an opposite charged dimer Shinde  _et al._ (2022). In the limit of
low persistence length, an active chiral particle follows curved trajectories,
similar to the Brownian motion of a charged particle Van Teeffelen and Löwen
(2008); Scholz  _et al._ (2021) under a magnetic field. Therefore, another
study of interest would be the escape dynamics of a chiral active Brownian
particle in the presence of an external vortex. Finally, it could be
interesting to study how an external magnetic field can affect an active
turnover for an active particle in a bistable potential Militaru  _et al._
(2021)–an optimal correlation time where the transition rate is maximized– and
how an external vortex influences new turnovers observed in the presence of a
fluctuating magnetic field Baura, Ray, and Bag (2013); Mondal  _et al._
(2018).

###### Acknowledgements.

A. Sharma and H. Löwen acknowledge the support by the Deutsche
Forschungsgemeinschaft (DFG) within the projects SH 1275/3-1 (A.S.) and LO
418/25-1 (H.L.).

## Data Availability Statement

The data that support the findings of this study are available from the
corresponding author upon reasonable request.

## Author Declarations

The authors have no conflicts to disclose.

##  Appendix A Derivation of the mean escape time

The main purpose of this section is to derive the mean escape time in Eq. (5)
to Eq. (7). We start with the underdamped Langevin equation describing the
dynamics of a charged Brownian particle with mass $m$ and charge $q$ subjected
to a magnetic field $B$ in the $-\hat{z}$ direction. The velocity Langevin
equation for the position $\bm{r}=(x,y)^{\top}$ and the velocity
$\bm{v}=(v_{x},v_{y})^{\top}$ of the particle under the effect of the linear
nonconservative force $\bm{F}_{nc}=\epsilon(-y,x)^{\top}$ and the conservative
force $\bm{F}_{c}=-k(x,y)^{\top}$ due to the isotropic potential
$U(x,y)=k(x^{2}+y^{2})/2$, can be written as

$m\dot{\bm{v}}=-\bm{K}\bm{r}-\bm{G}\bm{v}(t)+\sqrt{2\gamma T}\bm{\eta}(t),$
(8)

where $\bm{\eta}(t)=(\eta_{x}(t),\eta_{y}(t))^{\top}$ is the Gaussian white
noise with zero mean and Dirac delta correlation
$\langle\bm{\eta}(t)\bm{\eta}^{\top}(t^{\prime})\rangle=\delta(t-t^{\prime})$
with $\gamma$ being the friction coefficient and $T$ the temperature. The
matrices $\bm{G}$ and $\bm{K}$ are defined as

$\bm{G}=\gamma\left(\begin{array}[]{cc}1&\kappa\\\ -\kappa&1\\\
\end{array}\right),\,\,\,\,\,\bm{K}=\left(\begin{array}[]{cc}k&\epsilon\\\
-\epsilon&k\\\ \end{array}\right),$ (9)

with $\kappa=qB/\gamma$ being the diffusive Hall parameter which quantifies
the strength of the Lorentz force relative to the frictional force. Using the
low-mass approach, the corresponding overdamped Langevin equation can be
written as Chun, Durang, and Noh (2018); Abdoli  _et al._ (2022b, a)

$\dot{\bm{r}}=\bm{A}\bm{r}+\bm{\xi}(t),$ (10)

where $\bm{A}=\bm{G}^{-1}\bm{K}$ and $\bm{\xi}(t)=(\xi_{x},\xi_{y})^{\top}$ is
Gaussian nonwhite noise with

$\displaystyle\langle\bm{\xi}(t)\rangle$ $\displaystyle=$ $\displaystyle 0,$
(11) $\displaystyle\langle\bm{\xi}(t)\bm{\xi}^{\top}(t^{\prime})\rangle$
$\displaystyle=$ $\displaystyle
T\bm{G}^{-1}\delta_{+}(t-t^{\prime})+T(\bm{G}^{-1})^{\top}\delta_{-}(t-t^{\prime}),\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,\,$
(12)

where $\delta_{\pm}(s=t-t^{\prime})$ are the modified Dirac delta functions
which are zero for $s\neq 0$ while
$\int_{0}^{\infty}\mathop{}\\!\mathrm{d}s\delta_{+}(s)=\int_{-\infty}^{0}\mathop{}\\!\mathrm{d}s\delta_{-}(s)=1$
and
$\int_{0}^{\infty}\mathop{}\\!\mathrm{d}s\delta_{-}(s)=\int_{-\infty}^{0}\mathop{}\\!\mathrm{d}s\delta_{+}(s)=0$.

Equation (10) can be rewritten as Eq. (1) and Eq. (2) in the Cartesian
coordinates and thereafter using Itô calculus can be reduced to a one-
dimensional equation for the variable $r$, which is the distance from the
origin and is given by Eq. (3). The mean time for the particle to escape the
trap, truncated at $r=a$, can be obtained by the following equation

$\langle
t\rangle=\frac{1+\kappa^{2}}{D}\int_{0}^{a}y^{-1}\exp\left[\frac{k_{eff}}{2\gamma
D}y^{2}\right]dy\int_{0}^{y}z\exp\left[-\frac{k_{eff}}{2\gamma
D}z^{2}\right]dz,$ (13)

where $D=T/\gamma$ is the diffusion coefficient for a freely moving particle.
This equation can be exactly solved: using a change of variables the second
integral on the right hand side gives $(\gamma
D/k_{eff})\left[1-\exp\left(-k_{eff}y^{2}/2\gamma D\right)\right]$. By
substitution of this solution into Eq. (13), the resulting integral can be
exactly solved which gives Eq. (5) and Eq. (6).

## References

## References

  * Farkas (1927) L. Farkas, “Keimbildungsgeschwindigkeit in übersättigten Dämpfen,” Zeitschrift für physikalische Chemie 125, 236–242 (1927).
  * Kramers (1940) H. A. Kramers, “Brownian motion in a field of force and the diffusion model of chemical reactions,” Physica 7, 284–304 (1940).
  * Grabert and Linkwitz (1988) H. Grabert and S. Linkwitz, “Effect of time-delayed friction on the escape from a metastable well,” Physical Review A 37, 963 (1988).
  * McCann, Dykman, and Golding (1999) L. I. McCann, M. Dykman, and B. Golding, “Thermally activated transitions in a bistable three-dimensional optical trap,” Nature 402, 785–787 (1999).
  * Rondin  _et al._ (2017) L. Rondin, J. Gieseler, F. Ricci, R. Quidant, C. Dellago, and L. Novotny, “Direct measurement of Kramers turnover with a levitated nanoparticle,” Nature Nanotechnology 12, 1130–1133 (2017).
  * Grote and Hynes (1980) R. F. Grote and J. T. Hynes, “The stable states picture of chemical reactions. ii. rate constants for condensed and gas phase reaction models,” The Journal of Chemical Physics 73, 2715–2732 (1980).
  * Carmeli and Nitzan (1982) B. Carmeli and A. Nitzan, “Non-Markoffian theory of activated rate processes,” Physical Review Letters 49, 423 (1982).
  * Ianconescu and Pollak (2015) R. Ianconescu and E. Pollak, “A study of Kramers’ turnover theory in the presence of exponential memory friction,” The Journal of Chemical Physics 143, 104104 (2015).
  * Hänggi, Marchesoni, and Grigolini (1984) P. Hänggi, F. Marchesoni, and P. Grigolini, “Bistable flow driven by coloured Gaussian noise: a critical study,” Zeitschrift für Physik B Condensed Matter 56, 333–339 (1984).
  * Jung and Hänggi (1988) P. Jung and P. Hänggi, “Bistability and colored noise in nonequilibrium systems: theory versus precise numerics,” Physical Review Letters 61, 11 (1988).
  * Sharma, Wittmann, and Brader (2017) A. Sharma, R. Wittmann, and J. M. Brader, “Escape rate of active particles in the effective equilibrium approach,” Physical Review E 95, 012115 (2017).
  * Scacchi, Brader, and Sharma (2019) A. Scacchi, J. M. Brader, and A. Sharma, “Escape rate of transiently active Brownian particle in one dimension,” Physical Review E 100, 012601 (2019).
  * Caprini  _et al._ (2019) L. Caprini, U. Marini Bettolo Marconi, A. Puglisi, and A. Vulpiani, “Active escape dynamics: The effect of persistence on barrier crossing,” The Journal of Chemical Physics 150, 024902 (2019).
  * Berera  _et al._ (2019) A. Berera, J. Mabillard, B. W. Mintz, and R. O. Ramos, “Formulating the Kramers problem in field theory,” Physical Review D 100, 076005 (2019).
  * Darmé, Jaeckel, and Lewicki (2019) L. Darmé, J. Jaeckel, and M. Lewicki, “Generalized escape paths for dynamical tunneling in qft,” Physical Review D 100, 096012 (2019).
  * Fleming and Hanggi (1993) G. R. Fleming and P. Hänggi,  _Activated Barrier Crossing: applications in physics, chemistry and biology_ , Vol. 4 (World Scientific, 1993).
  * Talkner and Hänggi (1995) P. Talkner and P. Hänggi,  _New trends in Kramers’ reaction rate theory_ , Vol. 11 (Springer Science & Business Media, 1995).
  * Hanggi and Mojtabai (1982) P. Hänggi and F. Mojtabai, “Thermally activated escape rate in presence of long-time memory,” Physical Review A 26, 1168 (1982).
  * Pollak (1986) E. Pollak, “Theory of activated rate processes: A new derivation of Kramers’ expression,” The Journal of Chemical Physics 85, 865–867 (1986).
  * Pollak, Grabert, and Hänggi (1989) E. Pollak, H. Grabert, and P. Hänggi, “Theory of activated rate processes for arbitrary frequency dependent friction: Solution of the turnover problem,” The Journal of Chemical Physics 91, 4073–4087 (1989).
  * Pollak and Ankerhold (2013) E. Pollak and J. Ankerhold, “Improvements to Kramers turnover theory,” The Journal of Chemical Physics 138, 164116 (2013).
  * Baura, Ray, and Bag (2013) A. Baura, S. Ray, and B. C. Bag, “Tuning of barrier crossing time of a particle by time dependent magnetic field,” The Journal of Chemical Physics 138, 244110 (2013).
  * Mondal  _et al._ (2018) S. Mondal, A. Baura, S. Das, and B. C. Bag, “A generic signature of a fluctuating magnetic field: An additional turnover prior to the Kramers’ one,” Physica A: Statistical Mechanics and its Applications 502, 58–76 (2018).
  * Filliger and Reimann (2007) R. Filliger and P. Reimann, “Kramers escape rate for a charged particle in a magnetic field,” EPL (Europhysics Letters) 77, 30008 (2007).
  * Abdoli  _et al._ (2022a) I. Abdoli, J.-U. Sommer, H. Löwen, and A. Sharma, “Escape dynamics in an anisotropically driven Brownian magneto-system,” EPL (Europhysics Letters) 139, 21003 (2022a).
  * Lee and Kwon (2019) S. Lee and C. Kwon, “Nonequilibrium driven by an external torque in the presence of a magnetic field,” Physical Review E 99, 052142 (2019).
  * Sun, Grier, and Grosberg (2010) B. Sun, D. G. Grier, and A. Y. Grosberg, “Minimal model for Brownian vortexes,” Physical Review E 82, 021123 (2010).
  * Sun  _et al._ (2009) B. Sun, J. Lin, E. Darby, A. Y. Grosberg, and D. G. Grier, “Brownian vortexes,” Physical Review E 80, 010401 (2009).
  * Roichman  _et al._ (2008) Y. Roichman, B. Sun, A. Stolarski, and D. G. Grier, “Influence of nonconservative optical forces on the dynamics of optically trapped colloidal spheres: the fountain of probability,” Physical Review Letters 101, 128301 (2008).
  * Moyses  _et al._ (2015) H. W. Moyses, R. O. Bauer, A. Y. Grosberg, and D. G. Grier, “Perturbative theory for Brownian vortexes,” Physical Review E 91, 062144 (2015).
  * Liebchen and Löwen (2019) B. Liebchen and H. Löwen, “Optimal navigation strategies for active particles,” EPL (Europhysics Letters) 127, 34003 (2019).
  * Vuijk, Brader, and Sharma (2019) H. D. Vuijk, J. M. Brader, and A. Sharma, “Anomalous fluxes in overdamped Brownian dynamics with Lorentz force,” Journal of Statistical Mechanics: Theory and Experiment 2019, 063203 (2019).
  * Chun, Durang, and Noh (2018) H.-M. Chun, X. Durang, and J. D. Noh, “Emergence of nonwhite noise in Langevin dynamics with magnetic Lorentz force,” Physical Review E 97, 032117 (2018).
  * Abdoli  _et al._ (2022b) I. Abdoli, R. Wittmann, J. M. Brader, J.-U. Sommer, H. Löwen, and A. Sharma, “Tunable Brownian magneto heat pump,” Scientific Reports 12, 1–10 (2022b).
  * Gardiner (2009) C. Gardiner,  _Stochastic methods_ , Vol. 4 (Springer Berlin, 2009).
  * Mangeat  _et al._ (2019) M. Mangeat, Y. Amarouchene, Y. Louyer, T. Guérin, and D. S. Dean, “Role of nonconservative scattering forces and damping on Brownian particles in optical traps,” Physical Review E 99, 052107 (2019).
  * Carstensen  _et al._ (2009) J. Carstensen, F. Greiner, L.-J. Hou, H. Maurer, and A. Piel, “Effect of neutral gas motion on the rotation of dust clusters in an axial magnetic field,” Physics of Plasmas 16, 013702 (2009).
  * Piel (2017) A. Piel,  _Plasma physics: an introduction to laboratory, space, and fusion plasmas_ (Springer, 2017).
  * Kählert  _et al._ (2012) H. Kählert, J. Carstensen, M. Bonitz, H. Löwen, F. Greiner, and A. Piel, “Magnetizing a complex plasma without a magnetic field,” Physical Review Letters 109, 155003 (2012).
  * Hartmann  _et al._ (2013) P. Hartmann, Z. Donkó, T. Ott, H. Kählert, and M. Bonitz, “Magnetoplasmons in rotating dusty plasmas,” Physical Review Letters 111, 155002 (2013).
  * Hartmann  _et al._ (2019) P. Hartmann, J. C. Reyes, E. G. Kostadinova, L. S. Matthews, T. W. Hyde, R. U. Masheyeva, K. N. Dzhumagulova, T. S. Ramazanov, T. Ott, H. Kählert, _et al._ , “Self-diffusion in two-dimensional quasimagnetized rotating dusty plasmas,” Physical Review E 99, 013203 (2019).
  * Shinde  _et al._ (2022) R. Shinde, J. U. Sommer, H. Löwen, and A. Sharma, “Strongly enhanced dynamics of a charged rouse dimer by an external magnetic field,” PNAS Nexus 1, pgac119 (2022).
  * Van Teeffelen and Löwen (2008) S. Van Teeffelen and H. Löwen, “Dynamics of a Brownian circle swimmer,” Physical Review E 78, 020101 (2008).
  * Scholz  _et al._ (2021) C. Scholz, A. Ldov, T. Pöschel, M. Engel, and H. Löwen, “Surfactants and rotelles in active chiral fluids,” Science Advances 7, eabf8998 (2021).
  * Militaru  _et al._ (2021) A. Militaru, M. Innerbichler, M. Frimmer, F. Tebbenjohanns, L. Novotny, and C. Dellago, “Escape dynamics of active particles in multistable potentials,” Nature Communications 12, 1–6 (2021).


11institutetext:  Ritsumeikan University

# An Investigation on Word Embedding Offset Clustering as Relationship
Classification

Didier Gohourou  Kazuhiro Kuwabara

###### Abstract

Vector representations obtained from word embedding are the source of many
groundbreaking advances in natural language processing. They yield word
representations that are capable of capturing semantics and analogies of words
within a text corpus. This study is an investigation in an attempt to elicit a
vector representation of relationships between pairs of word vectors. We use
six pooling strategies to represent vector relationships. Different types of
clustering models are applied to analyze which one correctly groups
relationship types. Subtraction pooling coupled with a centroid based
clustering mechanism shows better performances in our experimental setup. This
work aims to provide directions for a word embedding based unsupervised method
to identify the nature of a relationship represented by a pair of words.

###### Keywords:

Word Embedding Clustering Relationship Representation

##  1 Introduction

The study of pattern recognition in natural language processing requires a
numerical representation of text. This is achieved by computing a vector
representation for the words that compose text corpora. Words were originally
represented using a one-hot encoding vector representation. The approach had
shortcomings including sparsed and high dimensional vectors that are unable to
capture the contextual meaning of words. Proposed by Bengio et al, word
embedding is a neural language model that addresses the curse of
dimensionality. The proposed model also provides a distributed representation
of words, aligned with linguistic principles such as the context-dependent
nature of meaning. Different implementations have spawned from the proposition
including word2vec [7, 4], GloVe [10], and fastText [1]. Those word
representation models are the backbone of many natural language processing
(NLP) tasks including named entity recognition, sentiment analysis, and
machine translation, which led to state-of-the-art breakthroughs in those
respective NLP fields.

Word embedding demonstrated additional properties, such as the ability to
capture syntactic and semantic regularities in language [8]. Building from
this insight, can we obtain vectors that can capture the type of relationship
between word embedding? Can those relationship representations be effectively
grouped with clustering models? This study is an attempt to answer those
questions. It explores different pooling approaches to obtain a vector that
represents the relationship between word vectors based on their embedding. The
study also uses different clustering models in an attempt to score the ability
of the relationship vectors to be grouped. Answering those problems will point
toward an unsupervised methodology to classify relationships between words.
Our contributions can be emphasized as follows:

  * •

Explore for a word embedding representation of the relationship for a pair of
words.

  * •

Analyze the clustering of those relationship vectors.

  * •

Cue toward an unsupervised classification mechanism of relationships between
words.

The rest of the study unfolds by first discussing related works including the
clustering of word vectors and the elicitation of regularities in word vector
space. Then present the methodologies adopted for word embedding relationship
representation and the descriptions of the families of clustering algorithms
selected. An experiment section follows to detail the data set and specific
cluster algorithms used. A discussion analyzes the results and offers
directions to apply the findings, before concluding the study.

##  2 Related Work

Vector-space word representations learned by continuous space language models
such as word2vec models have demonstrated the ability to capture syntactic and
semantic regularities in language [7]. In their study Mikolov et al. created a
set of analogy questions in the form "a is to b as c is to …". To answer the
analogy question, $y=x_{b}-x_{a}+x_{c}$ is computed. Where $x_{a}$, $x_{b}$,
and $x_{c}$ are the embedding vectors for the words $a$, $b$, and $c$
respectively. $y$ is the continuous space representation of the word expected
as the answer. Because $y$ is not likely to represent an existing word, the
answer is considered to be the word with the embedding that has the greatest
cosine similarity. The results showed better performance than prior
methodologies on the SemEval-2012 Task 2: Measuring Relation Similarity. The
more recent transformer-based language models have been applied to solve
abstract analogy problems [14]. After pointing out the lack of attention to
recognizing analogies in NLP, the study emphasized the strength and limitation
of various models on psychometric analogy data sets from educational problems.
The transformer models used in the study included BERT, GPT-2, RoBERTa, GPT-3.
Their performances were compared against word embedding models, perplexity-
based point-wise mutual information, and random answers. GPT-2 and RoBERTa
were the top performer, while BERT performed worst than word embedding.

The clustering of word vectors has been used to analyze how well an embedding
model’s word vectors can be grouped by topic or other taxonomies. It has been
applied to various domain-specific studies including business [3] and medicine
[13]. Zhang et al. [17] demonstrated that incorporating word embedding with a
kernel-based k-mean clustering method provides superior performances within a
topic extraction pipeline. In their study, word2vec was used to extract
features from bibliometric data set, claiming the advantage to skip human
feature engineering. A specifically designed k-mean clustering model is used
for topic extraction. The clustering methodology is a polynomial kernel
function integrated into a cosine similarity-based k-mean clustering. The
performance of the proposed model was compared to different models used for
topic extraction including a standard k-mean algorithm, principal component
analysis, and a fuzzy c-mean algorithm. Closer to our work, an exploration of
the word-class distribution in word vector spaces [11] investigated how well
distribution models including Centroid-based, Gaussian Model, Gaussian Mixture
Model, k-Nearest Neighbor, Support Vector Machine, and OffSet, can estimate
the likelihood of a word to belong to a class. The study experimented with
pre-trained vectors from Glove, and classes from the WordNet taxonomy.

While analogy tasks try to find the best word to complete a pair to be similar
to another, our problem focuses on probing for a vector representation of
word-pair relationships, so that they are efficiently grouped by clustering
models. This by extension call for investigating the performances of
clustering models for this task.

##  3 Methodology

###  3.1 Relation Vectors

To obtain a relationship vector from vector representations of words, we
experiment with different pooling strategies. Here we define by pooling the
mechanism by which we reduce a set of vectors representing different words,
into a single one. The obtained vector will represent the relationship between
the set of pooled vectors. Our first pooling strategy is to use the
subtraction operator. . This strategy is derived from the linguistic
regularity observation such as $king-man+woman=queen$. We can deduce $king-
man=queen-woman$, where we consider the subtraction, the operator that gives a
representation of the type of relationship between word vectors on both sides
of the equation. Thus if $v_{r}$ is the relation vector representing the
relation between the word vectors $v_{s}$ and $v_{o}$, we have:

$v_{r}=v_{s}-v_{o}$ (1)

The second strategy is to apply the absolute value function to each component
of the vector resulting from the subtraction.

$v_{r}=\langle|v_{o_{1}}-v_{s_{1}}|,|v_{o_{2}}-v_{s_{2}}|,...,|v_{o_{n}}-v_{s_{n}}|\rangle$
(2)

Where $v_{o_{i}}$ and $v_{s_{i}}$ are respectively the $i^{th}$ dimensional
coordinate of $v_{o}$ and $v_{s}$. The third consist of adding the two vectors
involved in the relationship.

$v_{r}=v_{s}+v_{o}$ (3)

The fourth pooling strategy constitutes a vector with the coordinate obtained
by taking the minimum of each dimensional coordinate of the word vectors
involved in the relationship.

$v_{r}=\langle
min(v_{o_{1}},v_{s_{1}}),min(v_{o_{2}},v_{s_{2}}),...,min(v_{o_{n}},v_{s_{n}})\rangle$
(4)

Conversely, the fifth pooling strategy, named max pooling, consists of
creating the relationship vector using the maximum of each dimensional
coordinate.

$v_{r}=\langle
max(v_{o_{1}},v_{s_{1}}),max(v_{o_{2}},v_{s_{2}}),...,max(v_{o_{n}},v_{s_{n}})\rangle$
(5)

The last pooling strategy we use in our exploration is the average pooling.

$v_{r}=\langle(v_{o_{1}}+v_{s_{1}})/2,(v_{o_{2}}+v_{s_{2}})/2,...,(v_{o_{n}}+v_{s_{n}})/2\rangle$
(6)

Table 1 gives the summary of the pooling strategies considered is this study.

Table 1: Pooling mechanisms for relationship vector representation. Name | Formula
---|---
Substraction | $v_{r}=v_{1}-v_{2}$
Substraction absolute value | $|v_{1}-v_{2}|$
Addition | $v_{r}=v_{1}+v_{2}$
Minimum | $v_{r}=\langle min(v_{o_{1}},v_{s_{1}}),min(v_{o_{2}},v_{s_{2}}),...,min(v_{o_{n}},v_{s_{n}})\rangle$
Maximum | $v_{r}=\langle max(v_{o_{1}},v_{s_{1}}),max(v_{o_{2}},v_{s_{2}}),...,max(v_{o_{n}},v_{s_{n}})\rangle$
Mean | $v_{r}=\langle(v_{o_{1}}+v_{s_{1}})/2,(v_{o_{2}}+v_{s_{2}})/2,...,(v_{o_{n}}+v_{s_{n}})/2\rangle$

###  3.2 Clustering

Clustering is an active research field, that consist of grouping similar
objects into sets. It can be achieved with various algorithms that differ in
mechanisms employed to constitute a cluster. Although more than seventy
clustering models that can be classified into nearly twenty categories are
commonly used [16], we experiment with four types of the most widely used
clustering mechanisms, including centroid-based, hierarchical-based,
distribution-based and density-based.

####  3.2.1 Centroid-based clustering

represents their cluster based on a central vector that is usually randomly
selected and then optimized. k-mean [6] and its variants including k-mean++
[2] and k-medians are part of this clustering category.

####  3.2.2 Hierarchical clustering

aims to build a hierarchy of clusters, either bottom-up (agglomerative) or
top-down (divisive). The agglomerative approach starts by considering each
data point as a cluster. Pairs of clusters are then gradually merged, moving
up the hierarchy. Conversely, the divisive approach, start by considering all
data points as one cluster. Clusters are then progressively split moving down
the hierarchy. A measure of dissimilarity is used to determine which clusters
should be merged or split. The dissimilarity is measured with a distance
metric and a linkage criterion.

####  3.2.3 Distribution-based clustering

assumes the data points follow a statistical distribution. While distribution
based clustering models can capture statistical relationships within
attributes of the data point, they work efficiently when the data distribution
is known and the model parameters are set accordingly.

####  3.2.4 Density-based clustering

forms clusters by grouping data points from dense areas into clusters. Such
clustering algorithm allows arbitrary-shaped but they might not be precise
with data sets of varying densities. Furthermore, outlier data points are
considered noise. Common density-based clustering includes Density-Based
Spatial Clustering of Application with Noise (DBSCAN) [5], Ordering points to
identify the clustering structure (OPTICS), and Mean-shift.

##  4 Experiment

This section describes the data set used and selected algorithms used for
experimentation. The experiments are conducted with the clustering model
implementation of the Scikit-learn [9] and Scipy [15] python packages.

We use the adjusted random score to evaluate the clustering accuracy of each
setting.

###  4.1 Datasets

We use GloVe pre-trained word vectors 111
https://nlp.stanford.edu/projects/glove trained on a corpus composed of the
2014’s Wikipedia dump and the 5th edition of the English Gigaword. The
training corpus has six billion tokens, four hundred thousand unique words,
and is lower-cased. Our experiments use the 100 dimensions pre-trained vectors
version.

The word pairs are drawn from the word pairs analogy data set of Mikolov et
al. [8]. It contains 14 categories of word pairs for analogies tasks. Table 2
provides a sample of the data set word pairs for two categories.

Table 2: Sample of the word pairs dataset. Countries’ capital | Currencies
---|---
baghdad - iraq | japan - yen
bangkok - thailand | korea - won
beijing - china | latvia - lats
berlin - germany | lithuania - litas
bern - switzerland | macedonia - denar

For each word pair, their relation vector is obtained following the different
pooling strategies. Figure 1 gives a 2D projection overview of the
relationship vectors for each pooling strategy.

Figure 1: TSNE 2D projections of relation vectors for different pooling
strategy

###  4.2 K-mean

K-mean is not only the most used clustering mechanism but probably the most
well-known clustering algorithm overall, because of its simple approach. The
basic running steps of k-mean are the following:

  1. 1.

Define the number of clusters and randomly initialize their center points.

  2. 2.

Each data point is added to the cluster of its closest center point.

  3. 3.

Center points are recomputed, using the mean of all vectors in a given
cluster.

  4. 4.

The two previous steps are repeated, until the center points respectively
converge.

In addition to being simple, k-mean has a linear run time complexity $O(n)$.
Some drawbacks are the need to specify the number of clusters, and the
randomized initialization can provide different clustering results on
different algorithms run. Table 3 provides the adjusted random scores for
K-mean clustering.

Table 3: Adjusted random scores for K-mean clustering on the pooling strategies. | $X_{subs}$ | $X_{add}$ | $X_{abs}$ | $X_{min}$ | $X_{max}$ | $X_{mean}$
---|---|---|---|---|---|---
kmean | 0.792549 | 0.363478 | 0.296212 | 0.334140 | 0.313051 | 0.363478

###  4.3 Gaussian mixture

The Gaussian Mixture Model is a widely used clustering model of this type. It
assumes that the point in the data set follows a gaussian distribution. The
model can use two parameters such as the mean and the standard deviation to
describe the shape of the clusters. The parameters are found using the
Expectation-Maximization optimization algorithm. The Gaussian Mixture Model
clustering works as follows:

  1. 1.

Randomly initialize the Gaussian distribution parameters for each cluster for
the selected number of clusters.

  2. 2.

Compute the probability of each data point to belong to a cluster, given the
Gaussian distribution.

  3. 3.

Update the set of parameters for the Gaussian distributions, based on the data
point probabilities. The new parameters are computed, maximizing the
probability of data points within the cluster.

  4. 4.

The last two steps are iteratively repeated until the clusters’ centers
respectively converge.

The Gaussian mixture can be seen as a more flexible k-mean. Instead of
assuming clusters with a circular shape such as k-mean, Gaussian mixture can
shape ellipse-like clusters.

Table 4 provides the adjusted random score for the Gaussian mixture model
clustering.

Table 4: The adjusted random score of the Gaussian Mixture Model for different pooling strategies. | $X_{subs}$ | $X_{add}$ | $X_{abs}$ | $X_{min}$ | $X_{max}$ | $X_{mean}$
---|---|---|---|---|---|---
gmm | 0.854000 | 0.293000 | 0.351000 | 0.330000 | 0.228000 | 0.293000

###  4.4 Agglomerative Clustering

The hierarchical agglomerative clustering model can be described as follows:

  1. 1.

Every point in the data set is initially considered a cluster.

  2. 2.

Using a distance metric, clusters are jointly merged by pairs of the closest
to one another.

  3. 3.

The previous step is repeated until a unique cluster is formed. The clustering
structure is then defined by choosing when to stop the clusters combination.

The dissimilarity is measured with a distance metric and a linkage criterion.
We test various configurations of distance metrics as (dis)similarity
measures, and linkage criterion. The distance metrics used include euclidean
distance, cosine similarity, manhattan, l1, and l2. The linkage criterion is
the strategy used to merge clusters at different time steps. We experiment
with the following linkage criterion:

  * •

ward: minimize the variance of the clusters being merged,

  * •

average: merge two clusters with the minimal average distances of each
observation,

  * •

complete/maximum: merges two clusters with the smallest maximum distances
between all observations of the two sets,

  * •

single: merges two clusters with the smallest minimum distances between all
observations of the two sets.

The adjusted random scores of different configurations for the linkage and
distance metric parameters are available in table 5.

Table 5: The adjusted random score for different configurations of distance and linkage parameters of agglomerative clustering, for different pooling strategies | $X_{subs}$ | $X_{add}$ | $X_{abs}$ | $X_{min}$ | $X_{max}$ | $X_{mean}$
---|---|---|---|---|---|---
(ward, euclidean) | 0.682373 | 0.302749 | 0.342485 | 0.317647 | 0.283547 | 0.302749
(single, euclidean) | 0.006886 | 0.007961 | 0.006723 | 0.004333 | 0.006045 | 0.007961
(complete, euclidean) | 0.502632 | 0.296154 | 0.081222 | 0.167146 | 0.299644 | 0.296154
(average, euclidean) | 0.022881 | 0.293066 | 0.016404 | 0.129966 | 0.249915 | 0.293066
(single, cosine) | 0.004966 | 0.009726 | 0.003661 | 0.008308 | 0.008689 | 0.009726
(complete, cosine) | 0.695384 | 0.309572 | 0.420896 | 0.281730 | 0.356601 | 0.309572
(average, cosine) | 0.612819 | 0.292957 | 0.273538 | 0.319111 | 0.448297 | 0.292957
(single, manhattan) | 0.006766 | 0.007777 | 0.007989 | 0.005402 | 0.003136 | 0.007777
(complete, manhattan) | 0.550522 | 0.301899 | 0.030703 | 0.273196 | 0.309888 | 0.301899
(average, manhattan) | 0.021224 | 0.291166 | 0.016537 | 0.132257 | 0.269386 | 0.291166
(single, l1) | 0.006766 | 0.007777 | 0.007989 | 0.005402 | 0.003136 | 0.007777
(complete, l1) | 0.550522 | 0.301899 | 0.030703 | 0.273196 | 0.309888 | 0.301899
(average, l1) | 0.021224 | 0.291166 | 0.016537 | 0.132257 | 0.269386 | 0.291166
(single, l2) | 0.006886 | 0.007961 | 0.006723 | 0.004333 | 0.006045 | 0.007961
(complete, l2) | 0.502632 | 0.296154 | 0.081222 | 0.167146 | 0.299644 | 0.296154
(average, l2) | 0.022881 | 0.293066 | 0.016404 | 0.129966 | 0.249915 | 0.293066

###  4.5 DBSCAN

DBSCAN is the ubiquitous density-based clustering. The mechanism of DBSCAN can
be summarized as follows [12].

  1. 1.

Find points within $\epsilon$ distance of every point and identify the core
points which are points with more than a minimum number of points within
distance $\epsilon$.

  2. 2.

Determine the connected components of core points on the neighbor graph,
excluding all non-core points.

  3. 3.

Assign each non-core point to a nearby cluster within an $\epsilon$ distance
neighbor, consider it a noisy point.

We experimented with different metric types including euclidean, cosine, and
manhattan, as well as different values to consider points as neighbors of a
core point. Table 6 provides the adjusted random score for the different
experimental configurations for DBSCAN.

Table 6: The adjusted random score of DBSCAN for different pooling strategies. | $X_{subs}$ | $X_{add}$ | $X_{abs}$ | $X_{min}$ | $X_{max}$ | $X_{mean}$
---|---|---|---|---|---|---
(euclidan, 050) | 0.028191 | 0.028177 | 0.028177 | 0.028177 | 0.028177 | 0.028177
(cosine, 025) | 0.325427 | 0.217478 | 0.014572 | 0.243042 | 0.266950 | 0.217478
(cosine, 030) | 0.627861 | 0.043688 | 0.000000 | 0.039113 | 0.261000 | 0.043688
(cosine, 050) | 0.512095 | 0.006298 | 0.000000 | 0.003168 | 0.001254 | 0.006298
(manhattan, 050) | 0.028191 | 0.028177 | 0.028177 | 0.028177 | 0.028177 | 0.028177

##  5 Discussion

###  5.1 Results

Our results suggest that the subtraction pooling strategy might be the best
operation for word embedding-based word relationship representation, compared
to the five others, experimented with, in the investigation. This finding
supports the assumption of using the subtraction operator in word vector-based
analogy tasks [8]. Table 7 gives the model configuration with the highest
average score on each pooling strategy for different clustering models.

Table 7: Configurations with the highest score average from the different clustering models experimented with. | $X_{subs}$ | $X_{add}$ | $X_{abs}$ | $X_{min}$ | $X_{max}$ | $X_{mean}$
---|---|---|---|---|---|---
kmean | 0.792549 | 0.363478 | 0.296212 | 0.334140 | 0.313051 | 0.363478
gmm | 0.853806 | 0.293151 | 0.351499 | 0.329660 | 0.228340 | 0.293151
agglomerative: complete, cosine | 0.695384 | 0.309572 | 0.420896 | 0.281730 | 0.356601 | 0.309572
dbscan: cosine, 025 | 0.325427 | 0.217478 | 0.014572 | 0.243042 | 0.266950 | 0.217478

In addition, the k-mean centroid-based clustering algorithm yields the highest
scores for 3 of of 6 pooling strategies. The best score comes from clustering
relation vectors from the subtraction pooling strategy using the Gaussian
Mixture clustering model. Although the Gaussian Mixture Model is a
distribution-based clustering mechanism it is built on top of the centroid-
based k-means. This reinforces the superiority of centroid-based clustering in
our experimental setup, followed by the agglomerative hierarchical-based
clustering configured with complete linkage and cosine similarity.

###  5.2 Application

The ability to group representation of similar relationships between pairs of
words especially named entities. Point toward an unsupervised approach to
categorize relationships among words including named entities. This can in
turn be used as link categorization when building knowledge graphs. Thus
alleviating in some cases the need for manual data labeling for the type of
links between nodes. It can go as far as providing data for graph learning
models such as graph convolutional neural network that aims at including links
type in addition to nodes type in their learning process.

###  5.3 Future Work

This work gives two pointers for further steps. One is extending the
experimentation, with more word relationship representation strategy and
clustering models. Additional relationship representation can include learned
one by using an autoencoder on the pair of word vectors. Another direction is
to derive a formal explanation of the current findings, of this exploratory
study.

##  6 Conclusion

This study explores possibilities for a word embedding based word to word
relationship representation and their clustering ability in regards to
grouping similar relationships. Different relationship representations are
obtained by applying basic operations on the coordinate of pairs of vectors.
The subtraction pooling strategy and centroid-based clustering models tend to
give better results in our exploratory setup. Further work might extend the
exploration or provide a formal explanation of the findings.

## References

  * [1] Bojanowski, P., Grave, E., Joulin, A., Mikolov, T.: Enriching Word Vectors with Subword Information. Transactions of the Association for Computational Linguistics 5, 135–146 (2017). https://doi.org/10.1162/tacl_a_00051https://doi.org/10.1162/tacl_a_00051
  * [2] Bradley, P.S., Mangasarian, O.L., Street, W.N.: Clustering via concave minimization. Advances in Neural Information Processing Systems pp. 368–374 (1997)
  * [3] Gohourou, D., Kurita, D., Kuwabara, K., Huang, H.H.: International business matching using word embedding. In: Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). vol. 10191 LNAI (2017). https://doi.org/10.1007/978-3-319-54472-4_18https://doi.org/10.1007/978-3-319-54472-4_18
  * [4] Le, Q., Mikolov, T.: Distributed representations of sentences and documents. 31st International Conference on Machine Learning, ICML 2014 4, 2931–2939 (2014)
  * [5] Ling, R.F.U.o.C.: On the theory and construction of k-cluster. The Computer Journal. 15(4), 326–332 (1972). https://doi.org/https://doi.org/10.1093/comjnl/15.4.326https://doi.org/https://doi.org/10.1093/comjnl/15.4.326, https://doi.org/10.1093/comjnl/15.4.326
  * [6] Lloyd, S.P.: Least Squares Quantization in PCM. IEEE Transactions on Information Theory 28(2), 129–137 (1982). https://doi.org/10.1109/TIT.1982.1056489https://doi.org/10.1109/TIT.1982.1056489
  * [7] Mikolov, T., Chen, K., Corrado, G., Dean, J.: Efficient estimation of word representations in vector space. In: 1st International Conference on Learning Representations, ICLR 2013 - Workshop Track Proceedings. pp. 1–12 (2013)
  * [8] Mikolov, T., Yih, W.t., Zweig, G.: Linguistic Regularities in Continuous Space Word Representations. In: Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 746—-751. Association for Computational Linguistics, Atlanta, Georgia (2013). https://doi.org/10.3109/10826089109058901https://doi.org/10.3109/10826089109058901, https://aclanthology.org/N13-1090
  * [9] Pedregosa, F., Varoquaux, G., Gramfort, A., Michel, V., Thirion, B., Grisel, O., Blondel, M., Prettenhofer, P., Weiss, R., Dubourg, V., Vanderplas, J., Passos, A., Cournapeau, D., Brucher, M., Perrot, M., Duchesnay, E.: Scikit-learn: Machine learning in Python. Journal of Machine Learning Research 12, 2825–2830 (2011)
  * [10] Pennington, J., Socher, R., Manning, C.: GloVe: Global Vectors for Word Representation. Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543 (2014). https://doi.org/10.1080/02688697.2017.1354122https://doi.org/10.1080/02688697.2017.1354122
  * [11] Sasano, R., Korhonen, A.: Investigating Word-Class Distributions in Word Vector Spaces. In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 3657–3666. Association for Computational Linguistics (2020). https://doi.org/10.18653/v1/2020.acl-main.337https://doi.org/10.18653/v1/2020.acl-main.337
  * [12] Schubert, E., Sander, J., Ester, M., Kriegel, H.P., Xu, X.: DBSCAN revisited, revisited: Why and how you should (still) use DBSCAN. ACM Transactions on Database Systems 42(3) (2017). https://doi.org/10.1145/3068335https://doi.org/10.1145/3068335
  * [13] Suárez-Paniagua, V., Segura-Bedmar, I., Martínez, P.: Word Embedding Clustering for Disease Named Entity Recognition. Proc. of the Fifth BioCreative Challenge Evaluation Workshop pp. 299–304 (2015)
  * [14] Ushio, A., Espinosa-Anke, L., Schockaert, S., Camacho-Collados, J.: BERT is to NLP what AlexNet is to CV: Can pre-trained language models identify analogies? ACL-IJCNLP 2021 - 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, Proceedings of the Conference pp. 3609–3624 (2021). https://doi.org/10.18653/v1/2021.acl-long.280https://doi.org/10.18653/v1/2021.acl-long.280
  * [15] Virtanen, P., Gommers, R., Oliphant, T.E., Haberland, M., Reddy, T., Cournapeau, D., Burovski, E., Peterson, P., Weckesser, W., Bright, J., van der Walt, S.J., Brett, M., Wilson, J., Millman, K.J., Mayorov, N., Nelson, A.R.J., Jones, E., Kern, R., Larson, E., Carey, C.J., Polat, İ., Feng, Y., Moore, E.W., VanderPlas, J., Laxalde, D., Perktold, J., Cimrman, R., Henriksen, I., Quintero, E.A., Harris, C.R., Archibald, A.M., Ribeiro, A.H., Pedregosa, F., van Mulbregt, P., SciPy 1.0 Contributors: SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nature Methods 17, 261–272 (2020). https://doi.org/10.1038/s41592-019-0686-2https://doi.org/10.1038/s41592-019-0686-2
  * [16] Xu, D., Tian, Y.: A Comprehensive Survey of Clustering Algorithms. Annals of Data Science 2(2), 165–193 (2015). https://doi.org/10.1007/s40745-015-0040-1https://doi.org/10.1007/s40745-015-0040-1
  * [17] Zhang, Y., Lu, J., Liu, F., Liu, Q., Porter, A., Chen, H., Zhang, G.: Does deep learning help topic extraction? A kernel k-means clustering method with word embedding. Journal of Informetrics 12(4), 1099–1117 (2018). https://doi.org/10.1016/j.joi.2018.09.004https://doi.org/10.1016/j.joi.2018.09.004, https://doi.org/10.1016/j.joi.2018.09.004