# Electron localization function for non-collinear spins

Jacques K. Desmarais<EMAIL_ADDRESS>Dipartimento di
Chimica, Università di Torino, via Giuria 5, 10125 Torino, Italy  Giovanni
Vignale  Institute for Functional Intelligent Materials, National University
of Singapore, 4 Science Drive 2, Singapore 117544  Kamel Bencheikh  Setif 1
University-Ferhat Abbas. Faculty of Sciences. Department of Physics and
Laboratory of Quantum Physics and Dynamical Systems, Setif, Algeria
Alessandro Erba  Dipartimento di Chimica, Università di Torino, via Giuria 5,
10125 Torino, Italy  Stefano Pittalis<EMAIL_ADDRESS>Istituto
Nanoscienze, Consiglio Nazionale delle Ricerche, Via Campi 213A, I-41125
Modena, Italy

###### Abstract

We extend the electron localization function — a popular descriptor of
molecular bonds and atomic shells — to general non-collinearly magnetized
states, open-shell solutions of the Pauli-Schrödinger equation. The
description is also improved in paradigmatic, collinear states.

###### pacs:

71.15.Mb, 71.15Rf, 31.15.E-

Introduction. An ongoing effort in physics and chemistry is to turn the
concept of bonds among atoms into a quantitative modeling tool to handle
understanding and prediction in most general and novel regimes.Ball (2011)
This quest started before the discovery of quantum mechanics with the Lewis
picture of bonding.Lewis (1916) This was then incarnated in the modern
approach by the valence bond (VB) theory of Heitler and London.Heitler and
London (1927) Which, in turn, inspired the concept of “resonance” by
PaulingPauling (1946) and, eventually, models of condensed matter in terms of
resonant valence bonds by P.W. AndersonAnderson (1987) and valence shell
electron pair repulsion theory.Gillespie and Nyholm (1957) The idea of sharing
electrons is also at the base of the so-called molecular orbital theory, which
introduces single-particle orbitals delocalized over the molecule.

The Kohn-Sham (KS) formulation of density functional theory (DFT) is a
molecular orbital theory and, currently, is also the most applied
computational methodology in chemistry and materials science. Jones (2015) At
the center of the success of KS-DFT is the exchange correlation (xc) energy
functional — a functional of the particle density. A pioneering observation by
Bader revealed that the Laplacian of the particle density handles the
visualization of atomic shells and bonds Gatti et al. (1988); Bader (1994) —
although not always successfully — and kicked off the challenge of describing
bonds within a DFT-like modeling. Two cornerstones of which are the “quantum
theory of atoms in molecules and crystals” and the “electron localization
function”.Gatti (2005) In this letter we shall focus on the latter.Silvi and
Savin (1994)

Following suggestions by Bader, Bader and Stephens (1975); Bader et al. (1988)
Becke and Edgecombe exploited the anti-symmetry of the many-body state to
probe the localization of electrons via the information available at the level
of a single Slater determinant for (globally) collinear states: if an electron
is localized at a given point in space, another electron cannot be brought at
the same position with the same spin. This behavior can be captured via the
same-spin pair densityBecke and Edgecombe (1990) $P_{\sigma\sigma}({\bf
r}_{1},{\bf r}_{2})=N(N-1)\int d3d4...dN~{}|\Psi({\bf r}_{1}\sigma,{\bf
r}_{2}\sigma,3,...,N)|^{2}$ where the many-electron wave function
$\Psi(1,2,3,...,N)$, for simplicity, is taken as an $N$-electron Slater
determinant; $1,2,...,N$ stands for the combined spatial plus spin variables
$({\bf r}_{1},\sigma_{1}),({\bf r}_{2},\sigma_{2}),...,({\bf
r}_{N},\sigma_{N})$. Therefore, the conditional probability
$P_{\sigma\sigma}({\bf r}_{1},{\bf r}_{2})/n_{\sigma}({\bf r}_{1})$ — where
$n_{\sigma}({\bf r}_{1})$ is the spin-electron density — refers to the event
of finding a second electron at ${\bf r}_{2}$ given the first electron at
${\bf r}_{1}$. Restricting to collinear states, setting ${\bf r}_{1}={\bf r}$
and ${\bf r}_{2}={\bf r}_{1}+\mathbf{u}$, and taking the spherical average
around ${\bf r}_{1}$, one then readily finds $P_{\sigma\sigma}({\bf
r},u)/n_{\sigma}({\bf r})=\frac{1}{3}D_{\sigma}({\bf
r})u^{2}+\cdot\cdot\cdot\;,$ where: $D_{\sigma}({\bf
r})=\left[2\tau_{\sigma}({\bf r})-\frac{1}{4}\nabla n_{\sigma}({\bf
r})\cdot\nabla n_{\sigma}({\bf r})/n_{\sigma}({\bf r})\right]$ and
$\tau_{\sigma}({\bf
r})=\frac{1}{2}\Sigma_{i=1,N_{\sigma}}|\nabla\varphi_{i,\sigma}({\bf r})|^{2}$
is the kinetic energy density of the electrons in the $N_{\sigma}$ single-
particle orbitals $\varphi_{i,\sigma}({\bf r})$. Next, Becke and Edgecombe
defined the spin-dependent electron localization function as follows

$\mathrm{ELF_{\sigma}}({\bf r})\equiv\frac{1}{1+\left[D_{\sigma}({\bf
r})/D^{\rm unif}_{\sigma}({\bf r})\right]^{2}}\;$ (1)

where $D^{\rm unif}_{\sigma}({\bf
r})=\frac{3}{5}\left(6\pi^{2}\right)^{2/3}n^{5/3}_{\sigma}({\bf r})$ is
$D_{\sigma}({\bf r})$ evaluated for a homogeneous electron gas with the same
local density of the real system. Eq. (1) is a simple scalar quantity
comprised between 0 (zero localization), 1/2 (electron gas-like localization)
and 1 (full localization). In practice, one usually runs a DFT calculations
and evaluates the ratio $D({\bf r})/D^{\rm unif}({\bf r})$ using the
corresponding KS orbitals.

Since its inception, the ELF has become important beyond the purpose of
rationalizing chemical structures and synthesis thereof: it provided insights
for improving density functional approximations.Perdew et al. (2009); Sun et
al. (2015, 2016); Hodgson et al. (2014); Aschebrock and Kümmel (2019) The ELF
has also been linked to the ability of density functionals to access local
information on the spin-entanglement,Pittalis et al. (2015) to properties of
the electronic stress tensor, Tao et al. (2008) to features of (time-
dependent) density functionals.Burnus et al. (2005); Durrant et al. (2015)

Switching to non-collinear states, the single-particle orbitals must be
upgraded to two-component spinors — none of which may point just “up” or
“down”. The direction of the magnetization (defined more below) may change
from point to point. Locally the magnetization provides us with a meaningful
collective direction of spin, but may not represent the configuration of any
of the underlying single-particle spinors. The restriction used in the
construction of the ELF is eluded, which brings up the question whether it may
be extended to work also for non-collinear states.

Figure 1: Naïve ELF of Eq. (5) for the molecular plane of the Cr3 non-
collinear magnet obtained before (left) and after (right) an SU(2) gauge
transformation (see main text for details). The curious reader can find the
plot of the gauge-invariant ELF in Fig. 2.

Solution. For tackling this problem, let us invoke the expression and the
interpretation of the ELF by Savin and collaboratorsSavin et al. (1992)

$\mathrm{ELF}({\bf r})\equiv\frac{1}{1+\left[D({\bf r})/D^{\rm unif}({\bf
r})\right]^{2}}\;$ (2)

Here, the spin-resolved quantity $D_{\sigma}({\bf r})$ is replaced by the
spin-summed quantity

$D({\bf r})=\tau({\bf r})-\frac{\nabla n({\bf r})\cdot\nabla n({\bf
r})}{8n({\bf r})}\;,$ (3)

which is thus normalized relative to the spin-unpolarized $D^{\rm unif}({\bf
r})=\frac{3}{10}\left(3\pi^{2}\right)^{2/3}n^{5/3}({\bf r})$. It can be
readily shown thatHoffmann-Ostenhof and Hoffmann-Ostenhof (1977) $D({\bf r})$
is a non-negative quantity $D({\bf r})\geq 0$. Thus, $D({\bf r})$ can be
regarded as the excess of the fermionic kinetic energy $\tau({\bf r})$ as
compared to the “bosonic kinetic energy” $\nabla n({\bf r})\cdot\nabla n({\bf
r})/8n({\bf r})$: small values of the electron localization correspond to
large excess fermionic kinetic energy.Savin et al. (1992)

Next, initiating the analysis that will bring us to the extension of the ELF,
let us consider the gauge aspects involved in the electron localization
function. In modern geometric parlance,Aitchison and Hey (1989); Nakahara
(2018) (local) gauge transformations can be understood in terms of the changes
in the local frames used in the representation of the quantum states. This
interpretation makes it apparent that gauge invariance must be a general and
stringent condition for any quantity to hold an intrinsic physical meaning —
i.e., expressing the independence of a physical property from the particular
choices made for its representation (or computation).

As a well-known and enlightening example, consider states carrying a
(paramagnetic) particle current, ${\bf j}({\bf
r})=\sum_{k=1}^{N}\Im\left\\{\varphi^{*}_{k}({\bf
r})\left[\nabla\varphi_{k}({\bf r})\right]\right\\}$, as for instance, a
molecule in an external magnetic field. The necessary modification of the ELF,
can be derived in several ways.Burnus et al. (2005); Räsänen et al. (2008);
James W. Furness and Teale (2016) On parallel developments, the importance of
the dependence of xc functionals on the particle current has been largely
demonstrated Dobson (1993); Becke (1996); Pittalis et al. (2007, 2009);
Räsänen et al. (2009); Oliveira et al. (2010); Bates and Furche (2012); Zhu et
al. (2016); Richter et al. (2023) A unifying approach for adding particle
currents to density functionals and the like is to consider the effect of a
U(1) gauge transformations:Tao and Perdew (2005) ${\varphi}^{\prime}({\bf
r})=\exp\left[\frac{i}{c}\chi(\bf r)\right]\varphi({\bf r})\;$. The particle
density is obviously invariant but the kinetic energy density transforms as
follows:

$\tau_{\rm nc}^{\prime}({\bf r})=\tau_{\rm nc}({\bf r})+\frac{1}{c}{{\bf
j}}({\bf r})\cdot{\bf A}_{\rm G}({\bf r})+\frac{n({\bf r})}{2c^{2}}{\bf
A}_{\rm G}({\bf r})\cdot{\bf A}_{\rm G}({\bf r})\;$ (4)

where ${\bf A}_{\rm G}({\bf r})=c\nabla\chi({\bf r})$ is a pure gauge
(Abelian) vector potential. Therefore, a current-dependent and gauge invariant
ELF is readily recovered by replacing $\tau({\bf
r})\rightarrow\widetilde{\tau}({\bf r})=\tau({\bf r})-{\bf j}({\bf
r})\cdot{\bf j}({\bf r})/2n({\bf r})$ in $D({\bf r})$ in Eq. (3).

Keeping in mind the above observations, let us come back to consider non-
collinear (nc) states, as for instance, the ground state of the planar Cr3
molecular magnet. Ignoring for simplicity the U(1) aspects reviewed just
above, it is tempting to extend the ELF by simply evaluating $\tau({\bf r})$
on the two-component spinors $\Phi({\bf r})$: i.e., $\tau({\bf
r})\rightarrow\tau_{\rm nc}({\bf
r})=\frac{1}{2}\Sigma_{k}\left[\nabla\Phi_{k}({\bf
r})\right]^{\dagger}\cdot\nabla\Phi_{k}({\bf r})$. Performing such a
replacement in $D({\bf r})$, yields $D({\bf r})\rightarrow D_{\rm nc}({\bf
r})\equiv\tau({\bf r})-\frac{1}{8}\nabla n({\bf r})\cdot\nabla n({\bf
r})/n({\bf r})$. Which, in turn, would suggest the following naïve expression

$\text{ELF}_{\text{na\"{i}ve}}({\bf r})=\frac{1}{1+\left[D_{\rm nc}({\bf
r})/D^{\rm unif}({\bf r})\right]^{2}}\;.$ (5)

Recall now that an SU(2) gauge transformation acts on the spinors by locally
rotating them as follows ${\Phi}^{\prime}({\bf
r})=\exp\left[\frac{i}{c}{\lambda}^{a}({\bf r}){\sigma}^{a}\right]{\Phi}({\bf
r})\;,$ according to the spin-vector $\vec{\lambda}({\bf r})$. It is
straightforward to verify that under an SU(2) transformation, $\tau_{\rm
nc}({\bf r})$ transforms as follows:

$\displaystyle\tau^{\prime}_{\rm nc}({\bf r})=\tau_{\rm nc}({\bf
r})+\frac{1}{c}{\bf J}^{a}({\bf r})\cdot{\bf A}_{\rm G}^{a}({\bf
r})+\frac{n({\bf r})}{2c^{2}}{\bf A}_{\rm G}^{a}({\bf r})\cdot{\bf A}_{\rm
G}^{a}({\bf r})\;,$ (6)

where ${\bf J}^{a}({\bf r})=\sum_{k=1}^{N}\Im\left\\{\Phi^{\dagger}_{k}({\bf
r}){\sigma^{a}}\left[\nabla\Phi_{k}({\bf r})\right]\right\\}$ are the
(paramagnetic) spin currents and ${\bf A}_{\rm G}^{a}({\bf
r})=-\frac{ic}{2}{\rm Tr}\left({\sigma^{a}}U_{\rm S}^{\dagger}({\bf
r})\mathbf{\nabla}U_{\rm S}({\bf r})\right)$ are the component of a pure gauge
(non-Abelian) vector potential.not (a)

Let us visualize the effect of an SU(2) gauge transformation in the case of
the aforementioned Cr3 molecule. This is a planar molecule with C3v symmetry.
Consider the transformation where $\frac{1}{c}\lambda^{a}({\bf
r})=r_{a}^{2}\delta_{a,z}$ and, thus, $A^{a}_{\mu}({\bf
r})=c|r_{a}|\delta_{a,z}\delta_{\mu,z}$ applied to the Hartree-Fock spinorial
wavefunction of the system. As a consequence of which, the transformed naïve
ELF is dramatically different from the untransformed image: compare left and
right panels of Figure 1.

But a physically meaningful ELF must not change under gauge transformations.
In fact, a recent work on time reversal symmetric (TRS) states, has shown that
the replacement $\tau_{\rm nc}\to\widetilde{\tau}_{\rm nc}({\bf r})=\tau_{\rm
nc}({\bf r})-{\bf J}^{a}({\bf r})\cdot{\bf J}^{a}({\bf r})/2n({\bf r})$ in
$D_{\rm nc}({\bf r})$, restores invariance under SU(2) transformations, and
simultaneously adds a dependence on paramagnetic spin-currents. Desmarais et
al. (2024) Notice, however, that TRS states have vanishing magnetization. Cr3,
instead, exhibits a non-collinear magnetic structure. Shouldn’t such a
magnetization play a role in deriving a gauge invariant ELF?

This above question urges us to “mind our steps” and, thus, we also must
examine the effect of conjoined U(1)$\times$SU(2) transformations! In detail,
such a transformation acts on a spinor as follows ${\Phi}^{\prime}({\bf
r})=\exp\left[\frac{i}{c}\chi({\bf
r}){\mathrm{I}}+\frac{i}{c}{\lambda^{a}}({\bf
r}){\sigma^{a}}\right]{\Phi}({\bf r})$. As a result,

$\displaystyle\tau_{\rm nc}^{\prime}({\bf r})$ $\displaystyle=\tau_{\rm
nc}({\bf r})+\frac{n({\bf r})}{2c^{2}}\left[{\bf A}_{\rm G}({\bf r})\cdot{\bf
A}_{\rm G}({\bf r})+{\bf A}_{\rm G}^{a}({\bf r})\cdot{\bf A}_{\rm G}^{a}({\bf
r})\right]$ $\displaystyle+\frac{1}{c}{{\bf j}}({\bf r})\cdot{\bf A}_{\rm
G}({\bf r})+\frac{1}{c}{\bf J}^{a}({\bf r})\cdot{\bf A}_{\rm G}^{a}({\bf r})$
$\displaystyle+\frac{1}{c^{2}}{m^{a}}({\bf r}){\bf A}_{\rm G}^{a}({\bf
r})\cdot{\bf A}_{\rm G}({\bf r})\;.$ (7)

Eq. (Electron localization function for non-collinear spins) is lengthy but
worth a careful consideration. Crucially, we notice that Eq. (Electron
localization function for non-collinear spins) does not reduce to the simple
sum of terms due to individual U(1) and SU(2) gauge transformations. In fact,
the expression also includes an additional term coming from the coupling of
the U(1) and SU(2) gauge potentials ${\bf A}_{\rm G}({\bf r})$ and ${\bf
A}_{\rm G}^{a}({\bf r})$ via the spin-magnetization $\vec{m}({\bf r})$.

Direct cancellation of the transformation-induced terms in Eq. (Electron
localization function for non-collinear spins), leads to the gauge invariant
quantity

$\displaystyle\widetilde{\tau}_{\rm nc}({\bf r})$
$\displaystyle=\left(\tau_{\rm nc}({\bf r})-\frac{{\bf j}({\bf r})\cdot{\bf
j}({\bf r})}{2n({\bf r})}+\frac{\nabla{m}^{a}({\bf r})\cdot\nabla{m}^{a}({\bf
r})}{8n({\bf r})}\right)$ $\displaystyle+\left(\frac{{m}^{a}({\bf
r}){\tau}_{\rm nc}^{a}({\bf r})}{n({\bf r})}-\frac{{\bf J}^{a}({\bf
r})\cdot{\bf J}^{a}({\bf r})}{2n({\bf r})}\right)\;,$ (8)

with $\vec{{\tau}}_{\rm nc}({\bf
r})=\frac{1}{2}\sum_{k=1}^{N}\sum_{\mu}\Big{[}\partial_{\mu}\Phi^{\dagger}_{k}({\bf
r})\Big{]}\vec{\sigma}\partial_{\mu}\Phi_{k}({\bf r})$ being the spin-kinetic
energy density. Eq. (Electron localization function for non-collinear spins)
implements the task of enforcing U(1)$\times$SU(2) gauge invariance adding a
minimal set of terms — only using quantities which involve first-order
derivatives. The gauge invariance of Eq. (Electron localization function for
non-collinear spins) can be more easily verified by exploiting its connection
to the curvature of the exchange hole,Pittalis et al. (2017) as shown in
Section 1 of the Supplementary Material.Sup  Furthermore,

$\widetilde{D}_{\rm nc}({\bf r})\equiv\widetilde{\tau}_{\rm nc}({\bf
r})-\frac{1}{8}\frac{\nabla n({\bf r})\cdot\nabla n({\bf r})}{n({\bf r})}$ (9)

is, like $\widetilde{\tau}_{\rm nc}({\bf r})$, gauge invariant and is, like
$D({\bf r})$, a non-negative quantity. Non-negativity is established by
writing $\widetilde{D}_{\rm nc}({\bf r})$ as the trace of a Hermitian positive
semi-definite matrix in a reference frame in which the spin and particle
currents vanish [see Section 2 of the Supplementary Material]. Therefore, an
interpretation emerges naturally: $\widetilde{D}_{\rm nc}({\bf r})$ captures
the U(1)$\times$SU(2) gauge-invariant excess of $\widetilde{\tau}_{\rm
nc}({\bf r})$ as relative to the reference $\frac{1}{8}\frac{\nabla n({\bf
r})\cdot\nabla n({\bf r})}{n({\bf r})}$. Finally, we write the
U(1)$\times$SU(2) gauge-invariant ELF

$\widetilde{\mathrm{ELF}}({\bf r})\equiv\frac{1}{1+\left[\widetilde{D}_{\rm
nc}({\bf r})/D^{\rm unif}({\bf r})\right]^{2}}\;.$ (10)

Notice, Eq. (10) reduces to the SU(2) invariant ELF for TRS states,Desmarais
et al. (2024), it reduces to the U(1) invariant ELF for current-carrying
closed shell states,Burnus et al. (2005); Räsänen et al. (2008); James W.
Furness and Teale (2016) and it reduces to the original ELF for zero current
closed shell states.Becke and Edgecombe (1990); Savin et al. (1992)
Strikingly, Eq. (10) does not reduce to $D({\bf r})$ on globally collinear
spin-polarized states — a surprising fact that we shall carefully consider
more below.

Applications and further analysis. Eq. (10) and, thus, Eq. (9) have been
implemented in a developer’s version of the Crystal23 program.Erba et al.
(2023) Computational details are provided in the Supplementary Material.
Further analyses are carried out below.

Figure 2: U(1)$\times$SU(2) gauge invariant $\widetilde{\mathrm{ELF}}$ [see
Eq. (10)] in the Cr3 plane. Figure 3: (left) U(1)$\times$SU(2) gauge invariant
$\widetilde{\mathrm{ELF}}$ [see Eq. (10)] in the I${}_{2}^{-}$ molecule and
(right) the difference due to SOC.

Let us first return to the example of Cr3. The spin-magnetic moments are
oriented within the plane of the molecule at angles of 120∘ in the triangular
geometry, which breaks mirror symmetry and leads to a chiral electronic state.
Here, the spin-current reaches a magnitude much larger than in any of the
other cases discussed below. We also finds particle currents, which are
however negligible.

In Fig. 2, we provide the plot of the U(1)$\times$SU(2) gauge-invariant
$\widetilde{\mathrm{ELF}}$ for Cr3. The non-collinear spin structure increases
the conditional pair-probability density at a given point in space, leading to
a lower localization in the $\widetilde{\mathrm{ELF}}$, compared to the naïve
(gauge-dependent) ELF of Eq. (5). Differences are largest in the anti-bonding
region, where the naïve ELF significantly overestimates electron localization.
In the central bonding region, the largest contribution to the difference
$\widetilde{\mathrm{ELF}}$-ELF is provided by the $(\nabla m^{a}({\bf
r}))(\nabla m^{a}({\bf r}))/8n$ term. The difference reaches a value as large
as $-0.4$, or $40\%$ of the total.

Let us now consider a globally collinear, yet spin-orbit coupled case: the
I${}_{2}^{-}$ molecule. We provide the corresponding
$\widetilde{\mathrm{ELF}}$ in Figure 3. The plot nicely displays the bonding
valence manifold of both $\sigma$ and $\pi$ character. The effect of spin-
orbit coupling on the $\widetilde{\mathrm{ELF}}$, reported in the right panel
of Figure 3, is to increase electron localization along the molecular axis in
the region of the singly-occupied $\sigma^{\ast}$ anti-bonding orbital (i.e.
both close to the atoms and, mainly, behind them) and decrease localization at
the bond center and in directions orthogonal to the bond, in the region of the
$\pi$ bonding orbitals. The $\widetilde{\mathrm{ELF}}$ thus provides us with a
gauge-invariant description of bonding in spin-orbit-coupled orbital
manifolds.

Next, let us look into the remarkable novelties that emerge even at the level
of simple collinear states. Let us consider the simplest of molecules:
H${}^{+}_{2}$. The (eigen)states of this molecule are spin polarized single-
particle ones. For this one-electron system, the $\mathrm{ELF}({\bf r})=1$
(see top left panel of Figure 4), everywhere. This is fixed by the
$\widetilde{\mathrm{ELF}}$ by accessing the extra information on the spin-
polarization. Figure 4 (top right) reports the plot of
$\widetilde{\mathrm{ELF}}$, which reveals that the electron is shared by the
two nuclei.

In the case of the Li atom (bottom panels of Figure 4), both the ELF and
$\widetilde{\mathrm{ELF}}$ display a shell structure (labels for the closed
1$s$ and open 2$s$ shells are provided to guide the eye). However the ELF
(left bottom of Figure 4), predicts large values of the electron localization
also in regions far away from the atomic center where the electrons is
unlikely to be found. Again, the $\widetilde{\mathrm{ELF}}$ (right bottom of
Figure 4) fixes the issues by accessing extra information on the spin
polarization.not (b)

Figure 4: (left) ELF and (right) $\widetilde{\mathrm{ELF}}$ for (top) the
numerically exact Hartree-Fock wavefunction of the H${}_{2}^{+}$ molecule and
(bottom) the Li atom. The H atoms are centered on each tip of the lemon-shaped
region in the right panel, at a bond distance of 0.74 Å. The plot of the Li
atom is in a 6$\times$6 Å2 square.

Finally, let us consider the H2 molecule (not plotted). As a consequence of
the Pauli principle, two electrons are in the same singlet spin configuration
at each point in space. Moreover ${\vec{m}}({\bf r})=0$, $\vec{\tau}({\bf
r})=0$, and $\vec{{\bf J}}({\bf r})=0$ everywhere. Hence,
$\widetilde{\mathrm{ELF}}({\bf r})\equiv\mathrm{ELF}({\bf r})=1$. A resolution
of the localization of the electron pair in H2, would require to account for
correlations beyond single Slater determinantsKohout et al. (2004, 2005);
Feixas et al. (2010); Pittalis et al. (2018) — a task that is beyond the scope
of a “simple” ELF.

Conclusions. In solving the longstanding issue of extending the electron
localization function (ELF) to general non-collinear states, we have
discovered that a fundamental step had been so far missed. Namely, the key
step is to enforce the conjoint U(1)$\times$SU(2) gauge-invariance. This is
necessary to ensure that the ELF will depend only on intrinsic physical
features rather than on any particular gauge in which the Pauli equation is
solved.

The U(1)$\times$SU(2) gauge-invariance adds to an otherwise gauge dependent
ELF — not only the already known quadratic terms in the particle and spin
currents but also — a quadratic term in the gradient of the spin-magnetization
and a term involving the contraction of the magnetization with the (non-
collinear) spin-kinetic energy-density. These are quantities which, in
general, have sizable magnitude that cannot be ignored. As illustrations, we
have shown the cases of the Cr3 and I${}^{-}_{2}$ molecules. Furthermore, we
have shown that, the U(1)$\times$SU(2) gauge-invariant ELF also improves the
description of the localization of one-electron-like collinearly spin-
polarized states.

In view of the impact of the original ELF on the analysis of bonds and on the
construction of advanced density functional approximations, we believe that
the extended ELF here presented will importantly support the analyses of a
larger class of quantum materials as well as find use in the design of
improved density functional approximations.

###### Acknowledgements.

This research has received funding from the Project CH4.0 under the MUR
program “Dipartimenti di Eccellenza 2023-2027” (CUP: D13C22003520001). GV was
supported by the Ministry of Education, Singapore, under its Research Centre
of Excellence award to the Institute for Functional Intelligent Materials
(I-FIM, project No. EDUNC-33-18-279-V12). We are grateful to Drs. Jefferson
Maul and Chiara Ribaldone for stimulating discussions.

## References

  * Ball (2011) P. Ball, Nature 469, 26– (2011), URL https://doi.org/10.1038/469026a.
  * Lewis (1916) G. N. Lewis, Journal of the American Chemical Society 38, 762 (1916), eprint http://dx.doi.org/10.1021/ja02261a002, URL http://dx.doi.org/10.1021/ja02261a002.
  * Heitler and London (1927) W. Heitler and F. London, Z. Phys. 44, 455 (1927).
  * Pauling (1946) L. Pauling, Oregon State University Libraries Special Collections (1946).
  * Anderson (1987) P. W. Anderson, Science 235, 1196 (1987), URL https://www.science.org/doi/abs/10.1126/science.235.4793.1196.
  * Gillespie and Nyholm (1957) a. R. Gillespie and R. Nyholm, Quarterly Reviews, Chemical Society 11, 339 (1957).
  * Jones (2015) R. O. Jones, Rev. Mod. Phys. 87, 897 (2015), URL https://link.aps.org/doi/10.1103/RevModPhys.87.897.
  * Gatti et al. (1988) C. Gatti, P. J. MacDougall, and R. F. Bader, J. Chem. Phys. 88, 3792 (1988).
  * Bader (1994) R. F. W. Bader, _Atoms in Molecules: A Quantum Theory (International Series of Monographs on Chemistry)_ (Clarendon Press, 1994), 1st ed., ISBN 0198558651\.
  * Gatti (2005) C. Gatti, Z. fur Krist. - Cryst. Mater. 220, 399 (2005).
  * Silvi and Savin (1994) B. Silvi and A. Savin, Nature 371, 683 (1994), URL https://www.nature.com/articles/371683a0.
  * Bader and Stephens (1975) R. F. W. Bader and M. E. Stephens, Journal of the American Chemical Society 97, 7391 (1975), eprint http://dx.doi.org/10.1021/ja00859a001, URL http://dx.doi.org/10.1021/ja00859a001.
  * Bader et al. (1988) R. F. W. Bader, R. J. Gillespie, and P. J. MacDougall, Journal of the American Chemical Society 110, 7329 (1988), eprint http://dx.doi.org/10.1021/ja00230a009, URL http://dx.doi.org/10.1021/ja00230a009.
  * Becke and Edgecombe (1990) A. D. Becke and K. E. Edgecombe, The Journal of Chemical Physics 92, 5397 (1990), eprint http://dx.doi.org/10.1063/1.458517, URL http://dx.doi.org/10.1063/1.458517.
  * Perdew et al. (2009) J. P. Perdew, A. Ruzsinszky, G. I. Csonka, L. A. Constantin, and J. Sun, Phys. Rev. Lett. 103, 026403 (2009), URL https://link.aps.org/doi/10.1103/PhysRevLett.103.026403.
  * Sun et al. (2015) J. Sun, A. Ruzsinszky, and J. P. Perdew, Phys. Rev. Lett. 115, 036402 (2015), URL https://link.aps.org/doi/10.1103/PhysRevLett.115.036402.
  * Sun et al. (2016) J. Sun, R. C. Remsing, Y. Zhang, Z. Sun, A. Ruzsinszky, H. Peng, Z. Yang, A. Paul, U. Waghmare, X. Wu, et al., Nat Chem 8, 831 (2016).
  * Hodgson et al. (2014) M. J. P. Hodgson, J. D. Ramsden, T. R. Durrant, and R. W. Godby, Phys. Rev. B 90, 241107 (2014), URL https://link.aps.org/doi/10.1103/PhysRevB.90.241107.
  * Aschebrock and Kümmel (2019) T. Aschebrock and S. Kümmel, Phys. Rev. Res. 1, 033082 (2019), URL https://link.aps.org/doi/10.1103/PhysRevResearch.1.033082.
  * Pittalis et al. (2015) S. Pittalis, F. Troiani, C. A. Rozzi, and G. Vignale, Phys. Rev. B 91, 075109 (2015), URL https://link.aps.org/doi/10.1103/PhysRevB.91.075109.
  * Tao et al. (2008) J. Tao, G. Vignale, and I. V. Tokatly, Phys. Rev. Lett. 100, 206405 (2008), URL https://link.aps.org/doi/10.1103/PhysRevLett.100.206405.
  * Burnus et al. (2005) T. Burnus, M. A. L. Marques, and E. K. U. Gross, Phys. Rev. A 71, 010501 (2005), URL https://link.aps.org/doi/10.1103/PhysRevA.71.010501.
  * Durrant et al. (2015) T. R. Durrant, M. J. P. Hodgson, J. D. Ramsden, and R. W. Godby, ArXiv e-prints (2015), eprint 1505.07687.
  * Savin et al. (1992) A. Savin, O. Jepsen, J. Flad, O. K. Andersen, H. Preuss, and H. G. von Schnering, Angewandte Chemie International Edition in English 31, 187 (1992), ISSN 1521-3773, URL http://dx.doi.org/10.1002/anie.199201871.
  * Hoffmann-Ostenhof and Hoffmann-Ostenhof (1977) M. Hoffmann-Ostenhof and T. Hoffmann-Ostenhof, Phys. Rev. A 16, 1782 (1977).
  * Aitchison and Hey (1989) I. Aitchison and A. Hey, _Gauge Thetheoriesin Particle Physics, Second Edition_ , Graduate student series in physics (Hilger, 1989), ISBN 9780852743294.
  * Nakahara (2018) M. Nakahara, _Geometry, Topology and Physics_ (CRC Press, 2018), ISBN 9781420056945\.
  * Räsänen et al. (2008) E. Räsänen, A. Castro, and E. K. U. Gross, Phys. Rev. B 77, 115108 (2008), URL https://link.aps.org/doi/10.1103/PhysRevB.77.115108.
  * James W. Furness and Teale (2016) T. H. James W. Furness, Ulf Ekström and A. M. Teale, Molecular Physics 114, 1415 (2016), eprint https://doi.org/10.1080/00268976.2015.1133859, URL https://doi.org/10.1080/00268976.2015.1133859.
  * Dobson (1993) J. F. Dobson, J. Chem. Phys. 98, 8870 (1993).
  * Becke (1996) A. D. Becke, Canadian Journal of Chemistry 74, 995 (1996).
  * Pittalis et al. (2007) S. Pittalis, S. Kurth, S. Sharma, and E. K. U. Gross, The Journal of Chemical Physics 127, 124103 (2007), URL https://doi.org/10.1063/1.2777140.
  * Pittalis et al. (2009) S. Pittalis, E. Räsänen, and E. K. U. Gross, Phys. Rev. A 80, 032515 (2009).
  * Räsänen et al. (2009) E. Räsänen, S. Pittalis, C. R. Proetto, and E. K. U. Gross, Phys. Rev. B 79, 121305 (2009).
  * Oliveira et al. (2010) M. J. T. Oliveira, E. Räsänen, S. Pittalis, and M. A. L. Marques, Journal of Chemical Theory and Computation 6, 3664 (2010).
  * Bates and Furche (2012) J. E. Bates and F. Furche, The Journal of Chemical Physics 137, 164105 (2012), URL https://doi.org/10.1063/1.4759080.
  * Zhu et al. (2016) W. Zhu, L. Zhang, and S. B. Trickey, The Journal of Chemical Physics 145, 224106 (2016), eprint http://aip.scitation.org/doi/pdf/10.1063/1.4971377, URL http://aip.scitation.org/doi/abs/10.1063/1.4971377.
  * Richter et al. (2023) R. Richter, T. Aschebrock, I. Schelter, and S. Kümmel, The Journal of Chemical Physics 159 (2023).
  * Tao and Perdew (2005) J. Tao and J. P. Perdew, Phys. Rev. Lett. 95, 196403 (2005).
  * not (a) Here and elsewhere we use Einstein summation over repeated indices, unless explicitly specified.
  * Desmarais et al. (2024) J. K. Desmarais, J. Maul, B. Civalleri, A. Erba, G. Vignale, and S. Pittalis, _Spin-currents via the gauge-principle for meta-generalized-gradient exchange-correlation functionals_ (2024), eprint 2401.07581.
  * Pittalis et al. (2017) S. Pittalis, G. Vignale, and F. G. Eich, Phys. Rev. B 96, 035141 (2017).
  * (43) See supplementary material at [URL] for a demonstration of the gauge-invariance and positivity of the U(1)xSU(2) gauge-invariant ELF and computational details. See also Refs. Desmarais et al., 2021; Desmarais et al., 2020a, 2019a, b, 2019b; Dunning Jr, 1989; Cao and Dolg, 2011; Stoll et al., 2002; Hurley et al., 1986 included therein.
  * Erba et al. (2023) A. Erba, J. K. Desmarais, S. Casassa, B. Civalleri, L. Doná, I. J. Bush, B. Searle, L. Maschio, L. Edith-Daga, A. Cossard, et al., J. Chem. Theor. Comput. 19, 6891 (2023).
  * not (b) In the many-electron case, the value of the $\widetilde{\mathrm{ELF}}$ at large ${\bf r}$ depends on the asymptotic behaviour of $D^{\mathrm{unif}}$ vs. $\widetilde{D}_{\mathrm{nc}}$.
  * Kohout et al. (2004) M. Kohout, K. Pernal, F. R. Wagner, and Y. Grin, Theoretical Chemistry Accounts 112, 453 (2004), ISSN 1432-2234, URL http://dx.doi.org/10.1007/s00214-004-0615-y.
  * Kohout et al. (2005) M. Kohout, K. Pernal, F. R. Wagner, and Y. Grin, Theoretical Chemistry Accounts 113, 287 (2005), ISSN 1432-2234, URL http://dx.doi.org/10.1007/s00214-005-0671-y.
  * Feixas et al. (2010) F. Feixas, E. Matito, M. Duran, M. Solà, and B. Silvi, Journal of Chemical Theory and Computation 6, 2736 (2010), pMID: 26616075, eprint http://dx.doi.org/10.1021/ct1003548, URL http://dx.doi.org/10.1021/ct1003548.
  * Pittalis et al. (2018) S. Pittalis, D. Varsano, A. Delgado, and C. A. Rozzi, Eur Phys J B 91, 1 (2018).
  * Desmarais et al. (2021) J. K. Desmarais, S. Komorovsky, J.-P. Flament, and A. Erba, J. Chem. Phys. 154, 204110 (2021).
  * Desmarais et al. (2020a) J. K. Desmarais, J.-P. Flament, and A. Erba, Phys. Rev. B 101, 235142 (2020a).
  * Desmarais et al. (2019a) J. K. Desmarais, J.-P. Flament, and A. Erba, J. Phys. Chem. Lett. 10, 3580 (2019a).
  * Desmarais et al. (2020b) J. K. Desmarais, J.-P. Flament, and A. Erba, Phys. Rev. B 102, 235118 (2020b).
  * Desmarais et al. (2019b) J. K. Desmarais, J.-P. Flament, and A. Erba, J. Chem. Phys. 151, 074107 (2019b).
  * Dunning Jr (1989) T. H. Dunning Jr, J. Chem. Phys. 90, 1007 (1989).
  * Cao and Dolg (2011) X. Cao and M. Dolg, Wiley Interdiscip. Rev. Comput. Mol. Sci. 1, 200 (2011).
  * Stoll et al. (2002) H. Stoll, B. Metz, and M. Dolg, J. Comput. Chem. 23, 767 (2002).
  * Hurley et al. (1986) M. Hurley, L. F. Pacios, P. Christiansen, R. Ross, and W. Ermler, J. Chem. Phys. 84, 6840 (1986).


[1]This document is the results of the research project funded by the National
Science Foundation.

[type=author, orcid=https:0000-0003-4414-5706 ]

[type=author, orcid=https:0000-0001-7948-7453]

# Attention-Wrapped Hierarchical BLSTMs for DDI Extraction

Vahab Mostafapour<EMAIL_ADDRESS>Oguz Dikenelli
<EMAIL_ADDRESS>Department of Computer Engineering, Ege University,
Izmir, Turkey

###### Abstract

Drug-Drug Interactions (DDIs) Extraction refers to the efforts to generate
hand-made or automatic tools to extract embedded information from text and
literature in the biomedical domain.

Because of restrictions in hand-made efforts and their lower speed, Machine-
Learning, or Deep-Learning approaches have become more popular for extracting
DDIs. In this study, we propose a novel and generic Deep-Learning model which
wraps Hierarchical Bidirectional LSTMs with two Attention Mechanisms that
outperforms state-of-the-art models for DDIs Extraction, based on the
DDIExtraction-2013 corpora. This model has obtained the macro F1-score of
0.785, and the precision of 0.80.

###### keywords:

Drug-Drug Interaction DDI BLSTM Attention Deep-Learning

##  1 Introduction

Drug-Drug Interactions (DDIs) usually refers to changes in the effects of a
drug which occur as results of the presence of another drug. Although these
changes are some-times useful, the result may be harmful when the interaction
increases the toxicity of a drug [3]. Hazardous effects, along with high costs
for patients and insurance companies, have turned the discovery of drug
interactions into an important, and widespread scientific research area. The
results of these researches are generally reported in medical journals, and
literature [1].

Drug-Drug Interaction (DDI) Extraction problem is described as the application
of Information Extraction (IE) techniques to biomedical literature in order to
discover embedded interactions between drugs.

There are some published databases and resources such as Drugbank [10], and
Stockley’s Drug Interactions [3], have gathered and categorized a number of
Drug-Drug Interactions, and their types. However, there are currently two
major problems with these resources: they are mainly made by health care
professionals, and are usually updated only once every two years. Therefore,
the development of an automatic extraction tool would accelerate and simplify
the browsing of medical publications and literature to extract certain
detected interactions.

SemEval-2013-task-9 challenge [21], and its corpus [7], boosted the
application of Machine Learning (ML) techniques on DDI extraction problem.

Former generations of ML techniques were mainly based on statistical methods
such as Support Vector Machine (SVM) [9, 23, 4, 5], while recent approaches
have moved toward the use of Deep Neural Networks (known as Deep-Learning)
architectures, which have been shown to perform better than statistical models
[19, 13, 29, 28, 11]. These methods have transferred some effective features
such as word features, POS tags, Dependency Graphs, and Parse Trees from
statistical methods, and use them as the input for networks. Additionally, the
inputs or features in deep-learning methods are generally mapped into a low
dimensional vector space, such as a predefined embedding representation
vectors using Mikolov Word2Vec [15], GloVe [18], or BioBERT [12], or into a
randomly generated representation.

In this study, we propose the Attention-Wrapped Hierarchical BLSTMs (AW-
BLSTMs) model for DDIs Extraction. The main objective of AW-BLSTMs is to use
the lowest possible level of external information and without big cleaning and
pruning operations on the input data. This means that we aimed to provide a
model that is robust for real-life scenarios even with uncleaned data and be
easily applicable to different languages even those have not strong syntactic
passers.

The novelty of AW-BLSTMs model is that it utilizes the two levels of attention
mechanisms that wrap two layers of BLSTMs to tackle DDI extraction problem,
and could outperform state-of-the-art DDIs extraction models.

The first level attention mechanism, called entity-level, is applied to assign
a weight to each word based on its contextual relatedness to the target
entities that comes from their embedding space. From this mechanism, the model
can highlight the words that are more related to the target entities. The top
level attention mechanism is applied on the top of BLSTM layers, before the
softmax classifier. This mechanism, which feeds from the output of BLSTM,
gives the model the opportunity to learn the weight for the sentences-wide
features that are related to the annotated relationship.

This model outperforms the classification metrics of CNN and RNN based state-
of-the-art DDIs extraction models (in our knowledge) in both the F1-score
(0.785) and the precision score (0.80) in the macro-average approach.

##  2 Related Works

Deep Neural Networks (DNNs) such as Recurrent Neural Network (RNN) and
Convolutional Neural Network(CNN) models have shown a good performance in the
DDI Extraction task.

One of the successful CNN based models to extract DDIs is presented in [13],
which leverages word and position embedding. They obtained a 0.698 for the
F1-score on the DDI-Extraction-2013 corpora. A multi-channel CNNs model is
presented by [19], which used five different types of word embeddings to get
the best embedding performance possible. Another remarkable CNN model, used to
detect DDIs from biomedical literature, is the Syntax Convolutional Neural
Network (SCNN) which could reach a 0.686 in the F1-score based on the
DDIExtraction-2013 corpora [29]. (Asada et al.) [2] have presented a CNNs
based model for DDI extraction task that is intensified with an Attention
mechanism which could help the model to obtain a 0.6912 F1-score.

Although the CNNs have pointed out a good performance in DDIs Extraction, RNNs
like Long Short-Term Memory (LSTM) networks have managed to outperform them in
this task in recent years [28, 30, 20]. Besides the LSTM networks, applying
Attention Mechanisms to the DNNs is another successful approach to increase
the performance of DDIs or general domain semantic relationship Extractors.
This mechanisms give to their models the opportunity to highlight the effect
of the features with high relatedness to the potential relationships in the
input sentence.

(Sahu et al.) [20] has presented three LSTM based models for the DDI
extraction task that are called B-LSTM, AB-LSTM, and Joint AB-LSTM. These
models have employed LSTMs, and attention mechanisms and achieved a 0.6939
F1-score as their highest value. They are designed to categorize the potential
relationship into five classification types: "Advice", "Effect", "Mechanism",
"Int", "Negative" (instead of Non-type relationship). [28] has introduced a
hierarchical RNNs (combination of LSTM and BLSTM) model to extract DDIs from
entity annotated medical text. In this model, an attention mechanism is
applied to the input layer to take advantage of the closeness of each word to
the target entities. It also shows that the usage SDP can significantly
improve the F1-score that has obtained 0.729. Because the usage of SDP
increases the dependence of the model on language, we do not use it in our
model. [30] has published a RNNs based model for DDIs extraction with focus on
data cleaning operations. The architecture of their model is similar to [28]
model, but a big effort on input data pruning, cleaning and removing redundant
sentences has helped them outperform the other models with F1-score 0.773,
which is the highest value in state-of-are models.

BO-LSTM model [11], has leveraged a domain ontology knowledge to intensify the
model that sound so beneficial and achieved 0.75 percent F1-score that is
higher than most of previous works.

Our model takes advantage of BLSTM networks as its intermediate layers in a
hierarchical form. The first layer is used for capturing the information of
each separated input, and the second layer captures information of whole
inputs together. Some previous works such as [28] and [30] have used an entity
attention mechanism alongside LSTM layers. Some models, such as [20], have
leveraged an attention layer on the top of their LSTM layer. However, this is
the first time that a model that is presented for DDIs extraction, combines
different attention mechanisms (input-attention and output-attention) with
BLSTM layers, so that the attention layers wrap the hierarchical BLSTM layers
and outperformed the state-of-the-art models. This combination helps the model
to highlight the more effective words of the given sentence and selected
features on the extraction performance together.

##  3 Model Architecture and Materials

In this section, we describe the architecture of the AW-BLSTMs model. The main
contribution of this model is to employ two types of attention layers to
surround a hierarchical BLSTMs to tackle the DDI extraction problem as
described above. These attention layers are used to modify the input and
output of BLSTMs layers by different attention mechanisms then outperform the
standard BLSTMs. In the following sections, a detailed description is given.

###  3.1  Input Features and Embeddings

Input Features: Various methods are used to exploit more features and
information from the biomedical text in order to outperform DDIs extractors,
or any biomedical relationship extractor in general. These works vary from
rule-based methods to deep-learning methods [6, 22, 23, 28].

The most recent deep-learning studies on DDIs extraction have boosted their
model by some strong features of the input sentences such as Words, POS tags,
and Positions (related distances) [16, 28, 27]. The words of the sentences
carry essential information about relationships between target entities, but t
the effects words vary due to their distance to each target entity. [27] has
addressed the influence of the embedded information in the positions of the
target entities within the sentence. That is, a network can obtain more
information about target drugs from the words that are closer to them than
those that are farther away which can cause noise in the model. To access this
information, the position feature of words is often used. In addition, it is
argued that the Part Of Speech (POS) tags information fortifies the input
layer and relation extraction model to improve its performance. Moreover, the
influence of these features are also compared in some other DDIs extraction
models such as [28]. The SDP feature is escaped to keep model language
independent and generic enough to be applicable to the other domains and
languages. So in this work Words, POS tags and Position (relative distances)
features are employed as input features. Figure 1 presents an example sentence
with two annotated target entities. Figure 2 shows the Part-of-Speech tags of
the sample sentence 1 that are taken from Stanford-Core-NLP tool [14]. In this
example, the token "interferes" is tagged by VBZ as its POS tag which
indicates that it is a form of a verb.

Figure 1: Sample Sentence  Figure 2: Part-of-Speech tag for the sample
sentence

Embeddings: Words embedding refers to the vector representation of a
vocabulary. It maps each word of the vocabulary to a vector in a real space.
In fact, it is a transformation of vocabulary to a high-dimensional one-hot
vector space (with the dimension of vocabulary size), or to a low dimension
space which can be obtained by randomly initialized vectors or training a
neural network such as Mikolove Word2Vec [15], GloVe [18], BioBERT [12], etc.

Embedding representation has played a successful role in neural networks based
information extraction models in the NPL domain [17, 27, 28]. It enables
models to grabs context and semantic information of words of a document. In
this work, an embedding representation for word and POS tags is trained by
Word2Vec [15] on the PubMed abstracts who includes ‘drug‘ trigger word. Also,
the standard normal distribution is employed to initialize an embedding space
for position feature [25], [28].

Figure 3: Attention-Wrapped Hierarchical BLSTMs (AW-BLSTMs) model for DDIs
Extraction

###  3.2 Model Layers

Figure 3 illustrates the general architecture of our model including its
attentions and BLSTMs layers.

####  3.2.1 Input Layer

Words-Sequence: We partition each sentence into three parts; (i) Before part
(tokens come before the first mentioned entity), (ii) Between part (tokens
between two mentioned entities) and (iii) After part (tokens come after second
mentioned entity). Figure 4 illustrates this partitioning strategy for words
of the example sentence in Figure 1. This strategy helps the network to learn
their effect separately in its lower layers. It also solves the problem of
long sentences that are removed by [30] in their data cleaning steps. Words of
each sentence are indicated by an index to utilize the embedding look-up table
in the embedding layer.

Figure 4: Partitions of a Sentence

POS-Sequence: Part-of-Speech tags that gathered from Stanford-Core-NLP tool
[14]. Parallel to the Words-Sequence, POS tags Sequence is partitioned into
three parts as well.

Position-Sequence: Position sequences or distance sequences are the relative
distance of each word related to position of both target entities that are
presented by $dist_{e_{1}}$, $dist_{e_{2}}$ respectively. For example, in the
given example sentences Figure 1, $dist_{acitretin}(\text{"effect"})=5$,
$dist_{progestin}(\text{"effect"})=-2$. So $dist(\text{"effect"})=[5,-2]$.
Similar to POS tags sequence and word sequence, positions sequence is
partitioned into three parts too.

####  3.2.2 Embedding Layer

Embedding layer encodes the input sequences $S$ of each word $W_{i}$ to a low-
dimensional vector space. Each word $W_{i}$ includes three features Word, POS
tag and position (which is a pairs of related distances to both target
entities). Suppose that $WV_{i}$ is the embedding representation of word
$W_{i}$ with its features. Then each $WV_{i}$ is a concatenation of word-
embedding vector $wv_{i}$ (embedding representation of word $W_{i}$ and
trained on PubMed abstracts), POS-embedding vector $pos_{i}$ (also trained on
PubMed abstract), and position-embedding vector $[dist_{e_{1}},dist_{e_{2}}]$
(mapped each distance value to a 10-dimensional vector initialized by normal
random distribution). Then the input sentence (or a sub-sentence; one part) S
can be displayed as a list of embedding-vectors as follows:

$S=[WV_{0},WV_{1},..,WV_{maxLen}]$ (1)

That is, for each word $W_{i}$ includes its three features, we have the
following concatenated embedding vector:

$WV_{i}=[wv_{i},pos_{i},[dist_{e_{1}},dist_{e_{2}}]]$ (2)

Such that,

$wv_{i}\in R^{WE_{dim}}$ ; $WE_{dim}=$ dimension of word-embedding.

$pos_{i}\in R^{pos_{dim}}$ ; $pos_{dim}=$ dimension of POS-tag-embedding.

$dist_{e_{j}}\in R^{dist_{dim}}$, j = 1, 2; $dist_{e_{j}}$= distance of word
$wv_{i}$ related to $e_{j}$th target entity.

$dist_{dim}=$ dimension of position-embedding or distances embedding;

####  3.2.3 Entity Attention Layer

In long and complicated sentences, and in those having multiple clauses, there
will likely be some words and clauses that could misguide the model to a
different relation label than the desired one. Assigning weight to the words,
is a successfully solution that has been applied to overcome such problems
[25], [28]. This method is called the Attention Mechanism, which allows the
models to learn the weight of each word’s contribution on the target
interaction. The essence of this mechanism is a vector ($\alpha$ vector) with
the size of the sentence length. It is considered to keep a weight for each
word. In our model, this mechanism is applied in two different levels: (i) in
the entity level (Input level attention) and (ii) the top level (feature
level). The first is used to highlight more effective words of the sentence in
terms of two target units. The latter one is applied over the top of the
bidirectional RNNs and gives the capability to the model to precisely learn
the contribution rate of the features. A detailed explanation comes as
follows:

Entity Attention: In this study, entity level attention takes advantage of our
embedding over PubMed training corpus since the silent semantic relationships
between words of the corpora are reflecting in the unsupervised trained
vectors by Mikolove Word2vec [15]. To turn this idea into account, the weight
vector $\alpha$ is computed as follows:

$\alpha_{j}^{i}=\frac{exp(f(e_{i},wv_{j}))}{\sum_{k}{exp(f(e_{i},wv_{k}))}}$
(3)

$f(e_{i},wv_{j})$ is the function that computes semantic relatedness of each
word $wv_{j}$ with target entity $e_{i}$ which in this work it is computed as
the inner product of embedding vectors of $e_{i}$ and $wv_{j}$.

In the next step to benefit from the joint impact of $\alpha{}_{j}^{1}$ and
$\alpha{}_{j}^{2}$ (attention vectors related to target entity $e_{1}$ and
$e_{2}$ respectively) together, a simple average of them is considered as
follows.

$\alpha{}_{j}^{Att}=\frac{\alpha{}_{j}^{1}+\alpha{}_{j}^{2}}{2}$ (4)

Finally by applying attention vector on the embedding vector of each word in
sentence sequence, embedding vector of each word updates due to the following
function:

$WV_{j}^{Att}=dot(\alpha{}_{j}^{Att},WV_{j})$ (5)

After applying thıs attention mechanism layer on the sentences, they feed the
bottom bidirectional LSTM cells.

####  3.2.4 BLSTM Layers

Recurrent Neural Networks (RNNs) have played an important role in Information
Extraction and NLP tasks in recent years. The characteristic that makes RRNs
more appropriate to apply on text mining and NLP problems is their sequential
structure. This structure provides the capability to learn the dependencies
between the inputs by connecting previous information (state) to the present.
The most commonly used, and successful version of RNNs are Long Short-Term
Memories (LSTMs) that are a modified RNNs which introduced by [8] to solve the
long term dependencies and Gradient Vanishing problem. LSTMs units benefits
from structures called cell state and gates. The gates provide the capability
of removing and adding information to the cell state, which then serves as an
information belt over time-steps. Each unit updates in time-step t due to
following operations:

$f_{t}=\sigma(W_{f}.[h_{t-1},x_{t}]+b_{f})$ (6)
$i_{t}=\sigma(W_{i}.[h_{t-1},x_{t}]+b_{i})$ (7)
$g_{t}=tanh(W_{g}.[h_{t-1},x_{t}]+b_{g})$ (8) $C_{t}=(f_{t}\otimes
C_{t-1})\oplus(i_{t}\otimes g_{t})$ (9)

Such that $\sigma$ is the sigmoid activation function and the operations
$\otimes$, $\oplus$ indicate the point-wise tensors multiplication and
summation respectively.

BLSTMs: The information flows of the input sentence are from left to right in
an LSTM network. So, the network is only able to capture the one-side
dependencies and relationships between tokens. To allow the network to keep
the effect of backward relationships between tokens (from the end of the input
sequence toward its begin) another LSTM layer with the reverse direction
(backward-side) is usually applied which is called Bidirectional LSTM or BLSTM
in brief. The output of BLSTMs is the concatenation of these two layers.
Formally, suppose an input sequence $S$ like equation 1: the forward LSTM
takes an element of S in each step from $WV_{0}$ to $WV_{maxLen}$
respectively, and generates its outputs
$H^{\rightarrow}=[h^{\rightarrow}_{1},h^{\rightarrow}_{2},...,h^{\rightarrow}_{n}]$.
In contrast, backward LSTM takes the input $S$ reversely from $WV_{maxLen}$ to
$WV_{0}$ respectively to calculate its outputs
$H^{\leftarrow}=[h^{\leftarrow}_{1},h^{\leftarrow}_{2},...,h^{\leftarrow}_{n}]$
and $n$ is the number of LSTM units and $T$ is the number of time-steps or
sequences length. Thus, the BLSTM turns out the features vector
$H=[H^{\rightarrow},H^{\leftarrow}]$.

In our work, as displayed in Figure 3, BLSTM layer consists of two layers. The
first BLSTMs layer is applied on the entity attention layer. In this layer,
three BLSTMs(with same configurations) are placed on top of each part (Before,
Between, and After parts) of the input sequence to capture information
embedded in each part independently. This lower BLSTMs turn out
$H_{1_{Befor}}$, $H_{1_{Middle}}$.,and $H_{1_{After}}$.The outputs of this
layer, $H_{1}$s, are fed into the higher BLSTM layer. The second BLSTM layer
is applied to the top of the lower BLSTMs. So, the higher BLSTM layer lies on
the whole input sentence and connects separated parts together. that is, it
takes outputs of feature vectors of all three BLSTMs as its inputs then
generates its output vector $H_{2}$,

$H_{2}=[h_{2}^{1},h_{2}^{2},...,h_{2}^{l}]$ (10)

This layer allows our model to learn the features that rely on all parts of
input together.

####  3.2.5 Top Attention Layer

The second attention mechanism is applied to our model to give it the
opportunity to concentrate on the sentence-wide features which are more
effective in the relationship classification phase. This mechanism allows the
model to learn weights for the sentence level syntactic and semantic features
that are reflected in the output of the upper BLSTM layer. That is, these
features convey the effects of all operations from the lowest level operation
that is the embeddings attention, to the highest one, the second bidirectional
LSTMs. The model highlights the vector representation of these features by
assigning weight them. The model learns these weights during its training and
keeps them in a vector called top attention vector or $\beta$. Following
equations show the Mathematical computation steps of this attention vector:

$\beta_{j}=dot(W_{top},h_{2}^{j})+b_{top}$ (11)

s.t. $h_{2}^{j}$ is $j^{th}$ element of $H_{2}$ in the equation 10

$\beta_{j}=tanh(\beta_{j})$ (12)
$\beta_{j}^{Att}=\frac{exp(dot(U_{top},\beta_{j}))}{\sum_{k}{exp(dot(U_{top},\beta_{k}))}}$
(13)

We have employed the simple $dot$ operation to apply the effect of attention
vector $\beta$ on the $H_{2}$ output vector and generate attention affected
output $H_{2}^{Att}$. This vector is ready to feed the softmax classifier
layer.

$H_{2}^{Att}=dot(\beta^{Att},H_{2})$ (14)

####  3.2.6 Classification Layer (Softmax layer)

Attention-based weighted feature vectors, $H_{2}^{Att}$, are final outputs of
the inner layers of our model. These vectors are fed into the softmax
classifier to predict the appropriate relation type. The softmax classifier is
set in a dense or fully connected layer with the number of nodes equal by the
number of class types (5, is this case) as its activation function. This
function computes the conditional probability value for each input sentence
using the $H_{2}^{Att}$ weighted vectors and predicts the highest value as the
class number of the relation type [31, 25]. That is, for DDI type y:

$p(y\mid x)=softmax(W_{dense}.H_{2}^{Att}+b_{dense})$ (15)

The softmax layer returns the class label y as its probability value as the
input sentence x is the highest value. We represent this label $r$ that is
computed as follows:

$r=argmax_{y}p(y\mid x)$ (16)

For the training loss, the negative "log-likelihood" of the correct labels is
leveraged as:

$L=\sum_{x}{log(p_{x})};\hskip 11.38092pt\text{x is the given sentence}$ (17)

##  4 Result and Discussions

###  4.1 Dataset Description

Following previous works and, in order to have a fair comparison, we have
evaluated AW-BLSTMs model using the well-known DDIExtraction-2013 corpus [21].
This corpus is a human annotated corpus, published in XML data format. Its
Data sources are from Drugbank [10] text (792 texts), and Medline abstracts
(233 abstracts). Among these files, there are 624 train files, and 191 test
files, related to drug-drug interactions. This corpus consists of 5,021
sentence-level drug-drug interaction instances which are known as positive
samples. It includes more than 28k negative samples that are tagged as false
or not any desired relationship type, as well. To obtain train and test data,
it is randomly split into about %77 and %23 respectively.

the objective of DDI Extraction problem due to the mentioned corpus is to
classify the extracted relation between two annotated drugs into one of the
"Advice", "Effect", "Mechanism", and "Int" interaction types. We have adapted
this problem to the multi-classification problem. To this end, negative
samples (a pair of drug instances in the sentence which does not express any
type of relationships) are labeled as "other" type interaction. By this
strategy, both detecting and classification sub-tasks of DDI Extraction task
is satisfied.

###  4.2 Evaluation Metrics and Performance comparison

Evaluation Metrics: We evaluate the model by the most commonly used Precision,
Recall, and F1-scores measures. The F1-score is traditionally computed by

$F1-score=2.\frac{Precision.Recall}{Precision+Recall}$ (18)

which is harmonic mean of precision and recall.

|  |  | F1-Score of DDI-Types |  |  | Overall
---|---|---|---|---|---
Model | Advice | Effect | Mechanism | Int | Precision | Recall | F1-Score
B-LSTM [20] | 0.7592 | 0.6515 | 0.7266 | 0.4740 | 0.6907 | 0.6435 | 0.6663
BR-LSTM [26] | 0.7518 | 0.6817 | 0.7911 | 0.4336 | 0.7152 | 0.7079 | 0.7115
BO-LSTM [11] | - | - | - | - | - | - | 0.751
Joint AB-LSTM [20] | 0.8026 | 0.6546 | 0.7226 | 0.4411 | 0.7447 | 0.6496 | 0.6939
HRNN [28] | 0.803 | 0.718 | 0.740 | 0.543 | 0.741 | 0.718 | 0.729
Att-BLSTM [30] | 0.851 | 0.766 | 0.775 | 0.577 | 0.759 | 0.687 | 0.773
RHCNN [24] | 0.8054 | 0.7349 | 0.7825 | 0.589 | 0.773 | 0.7375 | 0.7548
AW-BLSTM | 0.819 | 0.774 | 0.78 | 0.584 | 0.80 | 0.77 | 0.785
(our model) |  |  |  |  |  |  |
Table 1: Performance Comparison of AW-BLSTMs and other state-of-are deep
learning based ddi models

Performance comparison: Comparison of our model with other state-of-the-art
Neural networks-based methods on DDIExtraction-2013 corpus is presented in
this section. These results are reported in macro-averaged evaluation metrics
for DDIs classification. The B-LSTM model presented by [20] has been selected
as the baseline model as contains only simple BLSTM layer the can show the
effect of attention mechanisms. Table 1 presents the comparison of our model
with some other RNN and CNN based state-of-art DDIs extraction models.

In the first part, one can see F1-score of models related to the four drug-
drug interaction types separately. Our model outperforms all in "Effect" type
and shows comparable results in the rest of the types, so that in each type
passes most of the other models except one (variously).

In the second part,right side of the table, it shows the overall (in macro-
averaged) results which our model outperforms them in all Precision, Recall
and F1-score by 0.80, 0.77, 0.785 respectively. The best F1-score of the
state-of-art model is presented in [30] with F1-score 0.773, which is focused
on data cleaning and removing redundant samples from gold standard. Although,
the real world data is notoriously noisy and very often requires data
prepossessing, as the they mentioned the steps are not applicable to all types
data (the don’t apply it on Madeline data) and also is not our purpose because
of losing generality of the model to tackle real-world scenarios. However, the
elimination of redundancy sentences is expected to improve the proposed model.
We notice that the model presented in [20] uses attention mechanism on the top
of its BLSTMs layer but it could obtain at most 0.69 for F10 score and the
model presented in [28] has used entity level attention mechanism which could
reach 0.717 for F1-score without SDP and 0.729 even with using SDP. Indeed
using combined attention mechanisms that could capture information of
relatedness words of with target entities together with information of the
features lied on the sentences can intensify the model to the learn weights
more appropriately.

| DDI-type | Precision | Recall | F1-score
---|---|---|---
Effect | 0.734 | 0.819 | 0.774
Mechanism | 0.818 | 0.745 | 0.78
Advice | 0.772 | 0.873 | 0.819
Int | 0.776 | 0.469 | 0.584
Negative | 0.968 | 0.967 | 0.968
Table 2: Evaluation results of each DDI-types using of AW-BLSTMs model

In Table 2, we show the evaluation results of four drug-drug interaction types
existing in the corpus called "Advice", "Effect", "Mechanism", and "Int". The
maximum Precision is 0.818 "Mechanism" and maximum F1-score value is for
"Advice" with 0.819. Also, we can see that the values of "Negative" is higher
than 0.96.

###  4.3 Implementation and Experimental Setup

Implementation operations took place on the keras library with Tensorflow
Backend. To train the model, code is executed in Jupyter Notebook on Anaconda
environment. Code runs on a PC with core i5 CUP and 12GB RAM. The best result
of F1-score is reached on epoch 101 with batch-size=64 and validation-
split=0.1. In the last layer, classification layer, activation function is
employed.

###  4.4 Conclusion

In this work, we present a novel RNNs and Attention based model for the drug-
drug interactions extraction problem called AW-BLSTMs. In this model, a
hierarchical layer of BLSTMs is surrounded by two types of attention
mechanisms. AW-BLSTMs model uses only Words, Part-of-Speech tags as its input
features and words embedding representation. This lowest level of external
information and syntactic make it easily applicable to different languages and
domains which is planned for the future works. Further, focusing on filtering
and cleaning, limits the model, and its applicability to real-world scenarios.
However, this model has outperformed state-of-the-are DDIs extraction models
with F1-score 0.785 by evaluation on the DDIExtraction-2013 corpus.

For the future works, there are different approaches which we would like to
investigate. With keeping the generality of model one can apply it on other
languages such as the Turkish language. In the other hand, can boost the model
by effective features like shortest-dependency-path (SDP) and also, leveraging
biomedical ontologies to get a better result in DDI extraction problem.

##  5 Cross-references

In electronic publications, articles may be internally hyperlinked. Hyperlinks
are generated from proper cross-references in the article. For example, the
words Fig. 1 will never be more than simple text, whereas the proper cross-
reference `\ref{tiger}` may be turned into a hyperlink to the figure itself:
Fig. 1. In the same way, the words Ref. [1] will fail to turn into a
hyperlink; the proper cross-reference is `\cite{Knuth96}`. Cross-referencing
is possible in LaTeX for sections, subsections, formulae, figures, tables, and
literature references.

## References

  * Aronson [2007] Aronson, J., 2007.  Communicating information about drug interactions.  British journal of clinical pharmacology 63, 637–639.
  * Asada et al. [2017] Asada, M., Miwa, M., Sasaki, Y., 2017.  Extracting drug-drug interactions with attention cnns, in: BioNLP 2017, pp. 9–18.
  * Baxter and Preston [2010] Baxter, K., Preston, C.L., 2010\.  Stockley’s drug interactions. volume 495\.  Pharmaceutical Press London.
  * Bell et al. [2011] Bell, L., Zhang, J., Niu, X., 2011.  A statistical approach for extracting proteinprotein interactions, in: Proceedings of the ACM Conference on Bioinformatics, Computational Biology and Biomedicine (ACM-BCB’11).
  * Bui et al. [2014] Bui, Q.C., Sloot, P.M., Van Mulligen, E.M., Kors, J.A., 2014\.  A novel feature-based approach to extract drug–drug interactions from biomedical text.  Bioinformatics 30, 3365–3371.
  * Hao et al. [2005] Hao, Y., Zhu, X., Huang, M., Li, M., 2005.  Discovering patterns to extract protein–protein interactions from the literature: Part ii.  Bioinformatics 21, 3294–3300.
  * Herrero-Zazo et al. [2013] Herrero-Zazo, M., Segura-Bedmar, I., Martínez, P., Declerck, T., 2013\.  The ddi corpus: An annotated corpus with pharmacological substances and drug–drug interactions.  Journal of biomedical informatics 46, 914–920.
  * Hochreiter and Schmidhuber [1997] Hochreiter, S., Schmidhuber, J., 1997\.  Long short-term memory.  Neural computation 9, 1735–1780.
  * Kim et al. [2015] Kim, S., Liu, H., Yeganova, L., Wilbur, W.J., 2015\.  Extracting drug–drug interactions from literature using a rich feature-based linear kernel approach.  Journal of biomedical informatics 55, 23–30.
  * Knox et al. [2010] Knox, C., Law, V., Jewison, T., Liu, P., Ly, S., Frolkis, A., Pon, A., Banco, K., Mak, C., Neveu, V., et al., 2010.  Drugbank 3.0: a comprehensive resource for ‘omics’ research on drugs.  Nucleic acids research 39, D1035–D1041.
  * Lamurias et al. [2019] Lamurias, A., Sousa, D., Clarke, L.A., Couto, F.M., 2019\.  Bo-lstm: classifying relations via long short-term memory networks along biomedical ontologies.  BMC bioinformatics 20, 10\.
  * Lee et al. [2019] Lee, J., Yoon, W., Kim, S., Kim, D., Kim, S., So, C.H., Kang, J., 2019.  Biobert: pre-trained biomedical language representation model for biomedical text mining.  arXiv preprint arXiv:1901.08746 .
  * Liu et al. [2016] Liu, S., Tang, B., Chen, Q., Wang, X., 2016.  Drug-drug interaction extraction via convolutional neural networks.  Computational and Mathematical Methods in Medicine 2016, 1–8.  URL: https://doi.org/10.1155/2016/6918381, doi:10.1155/2016/6918381.
  * Manning et al. [2014] Manning, C.D., Surdeanu, M., Bauer, J., Finkel, J., Bethard, S.J., McClosky, D., 2014\.  The Stanford CoreNLP natural language processing toolkit, in: Association for Computational Linguistics (ACL) System Demonstrations, pp. 55–60.  URL: http://www.aclweb.org/anthology/P/P14/P14-5010.
  * Mikolov et al. [2013] Mikolov, T., Sutskever, I., Chen, K., Corrado, G.S., Dean, J., 2013.  Distributed representations of words and phrases and their compositionality, in: Advances in neural information processing systems, pp. 3111–3119.
  * Nguyen and Verspoor [2018] Nguyen, D.Q., Verspoor, K., 2018\.  Convolutional neural networks for chemical-disease relation extraction are improved with character-based word embeddings.  arXiv preprint arXiv:1805.10586 .
  * Palangi et al. [2016] Palangi, H., Deng, L., Shen, Y., Gao, J., He, X., Chen, J., Song, X., Ward, R., 2016.  Deep sentence embedding using long short-term memory networks: Analysis and application to information retrieval.  IEEE/ACM Transactions on Audio, Speech and Language Processing (TASLP) 24, 694–707.
  * Pennington et al. [2014] Pennington, J., Socher, R., Manning, C., 2014.  Glove: Global vectors for word representation, in: Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532–1543.
  * Quan et al. [2016] Quan, C., Hua, L., Sun, X., Bai, W., 2016.  Multichannel convolutional neural network for biological relation extraction.  BioMed research international 2016\.
  * Sahu and Anand [2018] Sahu, S.K., Anand, A., 2018\.  Drug-drug interaction extraction from biomedical texts using long short-term memory network.  Journal of biomedical informatics 86, 15–24.
  * Segura Bedmar et al. [2013] Segura Bedmar, I., Martínez, P., Herrero Zazo, M., 2013.  Semeval-2013 task 9: Extraction of drug-drug interactions from biomedical texts (ddiextraction 2013), Association for Computational Linguistics.
  * Segura-Bedmar et al. [2011a] Segura-Bedmar, I., Martínez, P., de Pablo-Sánchez, C., 2011a.  A linguistic rule-based approach to extract drug-drug interactions from pharmacological documents, in: BMC bioinformatics, BioMed Central. p. S1.
  * Segura-Bedmar et al. [2011b] Segura-Bedmar, I., Martinez, P., de Pablo-Sánchez, C., 2011b.  Using a shallow linguistic kernel for drug–drug interaction extraction.  Journal of biomedical informatics 44, 789–804.
  * Sun et al. [2019] Sun, X., Dong, K., Ma, L., Sutcliffe, R., He, F., Chen, S., Feng, J., 2019.  Drug-drug interaction extraction via recurrent hybrid convolutional neural networks with an improved focal loss.  Entropy 21, 37.
  * Wang et al. [2016] Wang, L., Cao, Z., De Melo, G., Liu, Z., 2016\.  Relation classification via multi-level attention cnns .
  * Xu et al. [2018] Xu, B., Shi, X., Zhao, Z., Zheng, W., 2018.  Leveraging biomedical resources in bi-lstm for drug-drug interaction extraction.  IEEE Access 6, 33432–33439.
  * Zeng et al. [2014] Zeng, D., Liu, K., Lai, S., Zhou, G., Zhao, J., et al., 2014\.  Relation classification via convolutional deep neural network .
  * Zhang et al. [2017] Zhang, Y., Zheng, W., Lin, H., Wang, J., Yang, Z., Dumontier, M., 2017.  Drug–drug interaction extraction via hierarchical rnns on sequence and shortest dependency paths.  Bioinformatics 34, 828–835.
  * Zhao et al. [2016] Zhao, Z., Yang, Z., Luo, L., Lin, H., Wang, J., 2016\.  Drug drug interaction extraction from biomedical literature using syntax convolutional neural network.  Bioinformatics 32, 3444–3453.
  * Zheng et al. [2017] Zheng, W., Lin, H., Luo, L., Zhao, Z., Li, Z., Zhang, Y., Yang, Z., Wang, J., 2017.  An attention-based effective neural model for drug-drug interactions extraction.  BMC bioinformatics 18, 445\.
  * Zhou et al. [2016] Zhou, P., Shi, W., Tian, J., Qi, Z., Li, B., Hao, H., Xu, B., 2016.  Attention-based bidirectional long short-term memory networks for relation classification, in: Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 207–212.