# Molecular Interactions Induced by a Static Electric Field in Quantum
Mechanics and Quantum Electrodynamics

Mohammad Reza Karimpour  Department of Physics and Materials Science,
University of Luxembourg, L-1511 Luxembourg City, Luxembourg  Dmitry V.
Fedorov  Department of Physics and Materials Science, University of
Luxembourg, L-1511 Luxembourg City, Luxembourg  Alexandre Tkatchenko
Department of Physics and Materials Science, University of Luxembourg, L-1511
Luxembourg City, Luxembourg<EMAIL_ADDRESS>
###### Abstract

By means of quantum mechanics and quantum electrodynamics applied to coupled
harmonic Drude oscillators, we study the interaction between two neutral atoms
or molecules subject to a uniform static electric field. Our focus is to
understand the interplay between leading contributions to field-induced
electrostatics/polarization and dispersion interactions, as considered within
the employed Drude model for both nonretarded and retarded regimes. For the
first case, we present an exact solution for two coupled oscillators obtained
by diagonalizing the corresponding quantum-mechanical Hamiltonian and
demonstrate that the external field can control the strength of different
intermolecular interactions and relative orientations of the molecules. In the
retarded regime described by quantum electrodynamics, our analysis shows that
field-induced electrostatic and polarization energies remain unchanged (in
isotropic and homogeneous vacuum) compared to the nonretarded case. For
interacting species modeled by quantum Drude oscillators, the developed
framework based on quantum mechanics and quantum electrodynamics yields the
leading contributions to molecular interactions under the combined action of
external and vacuum fields.

###### keywords:

American Chemical Society, LaTeX

IR,NMR,UV

The stable structure and properties of biomolecules, nanostructured materials,
and molecular solids are determined by a delicate balance between different
intermolecular forces 1, 2, 3, 4, 5. In many realistic systems, molecular
interactions are substantially modified by solvents, cell membranes, ionic
channels, and other environments 6, 7, 8, 9. A proper description of such
environments demands robust approaches for modeling both nonretarded and
retarded intermolecular interactions under arbitrary fields. Molecular
interactions in presence of static and dynamic electromagnetic and thermal
fields have been studied using various approaches 10, 11, 12, 13, 14, 15, 16,
17, 18, 19, 20, 21, 22, 23, 9, 24, but a comprehensive understanding is still
missing and some results remain controversial. For example, random and
inhomogeneous fields have been shown to affect the strength and distance
dependence of van der Waals (vdW) interactions or even change their sign 17,
18, 19, 20, 21, 6. However, similar to the discussion on the textbook
dispersion interaction 25, 26, there is still an ongoing debate on the
interpretation of modified vdW interactions as having either an electrostatic
or a quantum-mechanical origin 6, 27, 28, 7. The application of weak static
(in)homogeneous fields in the nonretarded regime 22, 23, 9 yields a visible
modification of molecular interactions in second and third orders of
perturbation theory, however the retarded regime has not been addressed in
these studies. Fiscelli et al. 24 used quantum electrodynamics (QED) to
propose a _dispersion_ energy, scaling as $R^{-4}~{}(R^{-3})$, with respect to
the interatomic distance for retarded (nonretarded) regime, for interacting
two-level hydrogen-like atoms in static electric fields. However, it still
remains unclear 29, 30 whether this term corresponds to dispersion or
electrostatic interactions. In order to resolve existing controversies and
clarify discrepancies in the literature, here we develop a quantum framework
for modeling and understanding intermolecular interactions in electric fields
based on first principles of quantum mechanics and QED.

The advances made in this Letter hinge on the usage of two formalisms that
enable an accurate modeling and conceptual understanding of nonretarded and
retarded interactions for two coupled quantum Drude oscillators (QDO) 31, 32,
33 subject to a static electric field: solving Schrödinger’s equation via
exact diagonalization and using perturbation theory in QED 34, 35, 36, 37, 38,
39, 40. In addition, we employ stochastic electrodynamics (SED) 41, 42, 43,
44, 45, 46, 47, 48, as a semiclassical formalism that transparently connects
molecular interactions to the fields that originate them. The usage of QDOs to
accurately and efficiently model the linear response of valence electrons in
atoms and molecules is a critical aspect because coupled QDOs enable
analytical solutions (with and without electric field) and have been
convincingly demonstrated to provide a reliable quantitative tool to describe
response properties of real atoms and molecules subject to external fields or
confinement 31, 32, 33, 49, 50, 51, 52, 53, 54, 55. QDOs can quantitatively –
within a few percent compared to explicit treatment of electrons – describe
polarization and dispersion interactions 33, 52, 53, capture electron density
redistribution induced by these interactions 56, model intermolecular
interactions in electric fields 6, 9, among many other response phenomena 3.
Our current study benefits from many attractive features of QDOs and
demonstrates their applicability to the retarded regime. By means of the
developed framework, we derive dominant contributions to the interaction
energy of two QDOs in an electric field up to terms $\propto R^{-6}\,(R^{-7})$
for nonretarded (retarded) regime. These contributions, corresponding to a
linear response to the external field, are interpreted as the field-induced
electrostatic and polarization interactions obtained in addition to the
conventional leading-order London/Casimir dispersion interaction found to be
unchanged in the presence of a static electric field within the QDO model. We
note that most previous studies in the QED literature employed the two-level
($s$ and $p$ states) hydrogen-like atom as a model. Unfortunately, there is no
known analytical solution for the case of interacting hydrogen atoms under an
external electric field 24, causing quite some controversy over the
interpretation of field-induced interatomic interactions. 29, 30, 57 Of
course, the coupled QDO model employed here also introduces approximations. In
particular, the Gaussian form of the QDO wavefunction does not capture the
effect of deformation of an electron cloud by a static field, in contrast to a
hydrogen atom for example. In a homogeneous electric field, the ground-state
electron density of a QDO undergoes a rigid displacement and this means that
the $\beta$ and $\gamma$ hyperpolarizabilities vanish 33. Therefore, field-
induced and dispersion-induced changes in the polarizability of interacting
species described by these hyperpolarizabilities 58, 12, 13, 14, 59, 16, 60,
61, 62, 63 are missing within the QDO model 64, 58. Hence, some interactions
corresponding to the coupling of the field-induced and dispersion-induced
changes in polarizabilities will also vanish for coupled QDOs. However, as we
discuss below, such terms are either of higher order ($\beta$-terms) or
smaller magnitude ($\gamma$-terms) in comparison to the dispersion and field-
induced electrostatic energies, for weak static electric fields.

Figure 1: Two interacting quantum Drude oscillators under influence of both,
the fluctuating vacuum electromagnetic field and an applied uniform static
electric field
${\bm{\mathcal{E}}}=(\mathcal{E}_{x},\mathcal{E}_{y},\mathcal{E}_{z})$.

First, we consider the nonretarded case, when the distance $R$ between two
interacting species is small compared to wavelengths $\lambda_{e}$ of their
electronic transitions: $R\ll\lambda_{e}=2\pi c/\omega_{e}\,$, where $c$ is
the speed of light. The system of two interacting QDOs, which represent two
atoms or molecules, separated by a distance $R$ along the $z$ axis is shown in
Fig. 1. In the absence of any field, this system is described by the
Hamiltonian

$\displaystyle\\!\\!\\!H\\!=\\!-\tfrac{\hbar^{2}}{2}\\!\left[\\!\tfrac{\bm{\nabla}_{\bm{r}_{1}}^{2}}{m_{1}}\\!+\\!\tfrac{\bm{\nabla}_{\bm{r}_{2}}^{2}}{m_{2}}\\!\right]\\!+\\!\tfrac{m_{1}\omega_{1}^{2}r_{1}^{2}+m_{2}\omega_{2}^{2}r_{2}^{2}}{2}+\\!V_{\rm
int}(\bm{r}_{1},\bm{r}_{2})\,,\\!\\!\\!\\!$ (1)

where $V_{\rm int}$ is the Coulomb potential approximated here by the dipole-
dipole coupling

$\\!\\!V_{\rm int}(\bm{r}_{1},\bm{r}_{2})\approx V_{\rm int}^{\rm
dd}(\bm{r}_{1},\bm{r}_{2})=\tfrac{q_{1}q_{2}}{4\pi\epsilon_{0}R^{3}}\left(\bm{r}_{1}\cdot\bm{r}_{2}-3z_{1}z_{2}\right).\\!\\!\\!$
(2)

This form of the interaction breaks the initial symmetry between $z$ and $x/y$
parts of the Hamiltonian. However, the symmetry between $x$ and $y$ parts
remains. Thus, one needs to obtain just $H_{x}$ and $H_{z}$ from the
separation $H=H_{x}+H_{y}+H_{z}$. Introducing the normal-mode coordinates

$x_{\pm}=\pm\,\tfrac{\gamma_{x}\sqrt{m_{1}}x_{1}+[(\omega_{2}^{2}-\omega_{1}^{2})/2\pm\sqrt{D}]\sqrt{m_{2}}x_{2}}{\sqrt{[(\omega_{2}^{2}-\omega_{1}^{2})/2\pm\sqrt{D}]^{2}+\gamma_{x}^{2}}}\
,$ (3)

with $\gamma_{x}=\frac{q_{1}q_{2}}{4\pi\epsilon_{0}\sqrt{m_{1}m_{2}}R^{3}}$
and $D=((\omega_{2}^{2}-\omega_{1}^{2})/2)^{2}+\gamma_{x}^{2}\,$, we transform
the $x$-dependent part of Eq. 1 to

$H_{x}=-\tfrac{\hbar^{2}}{2}\left[\tfrac{d^{2}}{dx_{+}^{2}}+\tfrac{d^{2}}{dx_{-}^{2}}\right]+\tfrac{1}{2}(\omega_{+}x_{+}^{2}+\omega_{-}x_{-}^{2})\
,$ (4)

where $\omega_{\pm}^{2}=\frac{(\omega_{2}^{2}+\omega_{1}^{2})}{2}\pm\sqrt{D}$.
The Hamiltonian of Eq. 4 describes two independent QDOs with unit masses and
the frequencies $\omega_{+}$ and $\omega_{-}$ of the two normal modes.

Now we introduce a uniform electric field with the contribution $H_{\rm
f}=-(q_{1}\bm{r}_{1}+q_{2}\bm{r}_{2})\cdot\bm{\mathcal{E}}$ to the
Hamiltonian. With the new coordinates, we obtain

$H^{\prime}_{x}\equiv H_{x}+H_{\rm
f}=H_{x}-\mathcal{E}_{x}(f_{+}x_{+}+f_{-}x_{-})\ ,$ (5)

where $H_{x}$ is given by Eq. 4 and

$f_{\pm}=\tfrac{(q_{1}/\sqrt{m_{1}})\gamma_{x}+(q_{2}/\sqrt{m_{2}})[(\omega^{2}_{2}-\omega^{2}_{1})/2\pm\sqrt{D}]}{\sqrt{\gamma_{x}^{2}+[(\omega^{2}_{2}-\omega^{2}_{1})/2\pm\sqrt{D}]^{2}}}\
.$ (6)

Equation 5 can be rewritten as a quadratic form

$H^{\prime}_{x}=\textstyle\sum\limits_{i=\pm}\left[-\tfrac{\hbar^{2}}{2}\tfrac{d^{2}}{dx_{i}^{2}}+\tfrac{1}{2}\omega_{i}^{2}\left(x_{i}-\tfrac{f_{i}\mathcal{E}_{x}}{\omega_{i}^{2}}\right)^{2}-\tfrac{f_{i}^{2}\mathcal{E}_{x}^{2}}{2\omega_{i}^{2}}\right]\
,$ (7)

which is the Hamiltonian of two one-dimensional (1D) oscillators with the
frequencies $\omega_{+}$ and $\omega_{-}$ and the centers shifted by
$(f_{+}\mathcal{E}_{x}/\omega_{+}^{2})$ and
$(f_{-}\mathcal{E}_{x}/\omega_{-}^{2})$, respectively. The interaction energy
of the two oscillators under an external electric field is given by the
difference between the total energy of the coupled QDOs and the sum of the
total energies of two isolated QDOs in the same field

$\Delta\mathcal{U}_{x}=\textstyle\sum\limits_{i=\pm}\left[\tfrac{\hbar\omega_{i}}{2}-\tfrac{f_{i}^{2}\mathcal{E}_{x}^{2}}{2\omega_{i}^{2}}\right]-\textstyle\sum\limits_{i=1,2}\left[\tfrac{\hbar\omega_{i}}{2}-\tfrac{\alpha_{i}^{2}\mathcal{E}_{x}^{2}}{2}\right]\
,$ (8)

where $\alpha_{i}=q_{i}^{2}/m_{i}\omega_{i}^{2}$ is the static polarizability
of a QDO. Due to the symmetry, $\Delta\mathcal{U}_{y}$ is derived in the same
way as $\Delta\mathcal{U}_{x}$ by replacing subscripts $x$ with $y$. The
$z$-dependent part of the total Hamiltonian, $\Delta\mathcal{U}_{z}$, one can
obtain similarly to $\Delta\mathcal{U}_{x}$, by replacing $\gamma_{x}$ in Eq.
3 with $\gamma_{z}=-2\gamma_{x}$. For this case, we obtain
$z_{\pm}=\left(\gamma_{z}z^{\prime}_{1}+[(a_{2}-a_{1})\pm\sqrt{D_{z}}]z^{\prime}_{2}\right)/\sqrt{\gamma_{z}^{2}+[(a_{2}-a_{1})\pm\sqrt{D_{z}}]^{2}}$
and $\omega_{\pm}=[(a_{1}+a_{2})\mp\sqrt{D_{z}}]^{\nicefrac{{1}}{{2}}}$, where
$D_{z}=(a_{2}-a_{1})^{2}+\gamma_{z}^{2}$.

As shown in Ref. 65, the resulting expressions for the interaction energy can
be expanded as infinite series with respect to small terms proportional to
$\alpha_{1}\alpha_{2}/(4\pi\epsilon_{0})^{2}R^{6}$. Retaining all the leading
terms up to $R^{-6}$ within the QDO model, we obtain

$\displaystyle\Delta\mathcal{U}(R)=-\tfrac{3~{}\alpha_{1}\alpha_{2}\hbar\omega_{1}\omega_{2}}{2(4\pi\epsilon_{0})^{2}(\omega_{1}+\omega_{2})R^{6}}+\tfrac{\alpha_{1}\alpha_{2}(\mathcal{E}_{x}^{2}+\mathcal{E}_{y}^{2}-2\mathcal{E}_{z}^{2})}{4\pi\epsilon_{0}R^{3}}-\tfrac{\alpha_{1}\alpha_{2}(\alpha_{1}+\alpha_{2})(\mathcal{E}_{x}^{2}+\mathcal{E}_{y}^{2}+4\mathcal{E}_{z}^{2})}{2(4\pi\epsilon_{0})^{2}R^{6}}\
.$ (9)

Here, the first contribution is the well-known vdW dispersion energy, stemming
from the difference between $\hbar\omega_{i}/2$ terms in Eq. 8 as well as the
corresponding expressions for $\Delta\mathcal{U}_{y}$ and
$\Delta\mathcal{U}_{z}$ 65. The dispersion energy is not affected by the
static field within the QDO model. The other two terms in Eq. 9 are field-
induced contributions, which originate from the terms in Eq. 8 not containing
$\hbar$. As will be discussed in detail below, these terms correspond to the
field-induced electrostatic and polarization contributions to the interaction
energy, respectively. For real atoms or molecules, Eq. 9 would contain an
additional term $\propto R^{-6}$, which has the approximate form 12, 14, 63
(for an exact analytical expression, see Ref. 16)
$\Delta\mathcal{U}_{\gamma}(R)=-\gamma
C_{6}(4\mathcal{E}_{x}^{2}+4\mathcal{E}_{y}^{2}+7\mathcal{E}_{z}^{2})/18\alpha
R^{6}$, where $\gamma$ is the second hyperpolarizability and $C_{6}$ is the
dipole-dipole vdW dispersion coefficient. This contribution is absent within
the QDO model where $\gamma$ vanishes because of symmetry reasons 58, 64. In
addition, it is worth mentioning that vdW dispersion interaction between two
atoms or molecules also causes dispersion-induced dipoles 25, 66, 67, 68, 15,
69, 70, 71, 72, 73 that vary as $R^{-7}$. With coupling to an applied static
field, it results in an interaction energy which is linear in the field and
scales as $R^{-7}$. As first suggested by Hunt 66, such an interaction energy
corresponds to the hyperpolarization of an atom by the fluctuating field from
the neighboring atom in combination with the applied field, which depends on
the $B$ hyperpolarizability (dipole-dipole-quadrupole hyperpolarizability) and
the static dipole polarizability of the atoms 66. Since a QDO has a
nonvanishing $B$ hyperpolarizability, the dispersion-induced dipole moment can
be captured by taking into account dipole-quadrupole couplings 74 in the
interaction Hamiltonian of Eq. 2.

In order to extend our results to the retarded regime,
$\\!R\\!\gg\\!c/\\!\omega_{e}$, we consider the coupling between two QDOs in
the framework of QED, where the molecular interactions are mediated by the
fluctuating vacuum radiation field. The total system consists of two QDOs, an
external static field, and the vacuum field. For a single QDO coupled to the
static field, the Hamiltonian is given by

$H^{(\mathcal{E})}\equiv
H^{(\mathcal{E})}(\bm{r})=-\tfrac{\hbar^{2}}{2m}\bm{\nabla}_{\bm{r}}^{2}+\tfrac{1}{2}m\omega^{2}\bm{r}^{2}-q\,\bm{r}\cdot\bm{\mathcal{E}}\
.$ (10)

To obtain eigenstates/eigenvalues of $H^{(\mathcal{E})}$, we diagonalize it by
means of the transformation
$\bm{r}=\bm{r}^{\prime}+{q\bm{\mathcal{E}}}/{m\omega^{2}}$ resulting in
$H^{(\mathcal{E})}(\bm{r})=H^{(0)}(\bm{r}^{\prime})-{\alpha\mathcal{E}^{2}}/{2}$.
Thus, the eigenstates and eigenvalues of $H^{(\mathcal{E})}(\bm{r})$ are given
by
$\psi_{\bm{n}}^{{}_{(\mathcal{E})}}(\bm{r})=\phi_{\bm{n}}^{{}_{(0)}}(\bm{r}^{\prime})$
and
$\mathcal{U}_{\bm{n}}^{(\mathcal{E})}=\mathcal{U}_{\bm{n}}^{(0)}-{\alpha\mathcal{E}^{2}}/{2}$,
respectively, where $\phi_{\bm{n}}^{{}_{(0)}}$ and
$\mathcal{U}_{\bm{n}}^{(0)}$ denote the eigenstates and eigenvalues of an
isolated oscillator 75. Similar to Eq. 8, the constant energy shift in
$\mathcal{U}_{\bm{n}}^{(\mathcal{E})}$ arises due to the static dipole induced
by the external electric field. The matrix elements of the dipole moment,
$\bm{\mu}=q\,\bm{r}$, are obtained by

$\langle
l|\mu_{x}|k\rangle=\,q\,\sigma\left[\textstyle{\sqrt{l}}\,\delta_{l,k+1}\\!+\\!\textstyle{\sqrt{l+1}}\,\delta_{l,k-1}\right]\\!+\\!\alpha\,\mathcal{E}_{x}\,\delta_{k,l}\
,$ (11)

with $\sigma=\sqrt{\hbar/2m\omega}$ as the QDO characteristic length and the
bras/kets defined such that $\langle x|i\rangle=\psi_{i}(x)$. In a similar
way, one obtains $\mu_{y}$ and $\mu_{z}$. The first and second terms on the
r.h.s. of Eq. 11 correspond to the fluctuating and field-induced static QDO
dipoles, respectively.

The Hamiltonian of the total non-interacting system is

$H_{0}=H_{\rm rad}+H_{1}^{(\mathcal{E})}+H_{2}^{(\mathcal{E})}\ ,$ (12)

where $H_{\rm rad}$ corresponds to the vacuum radiation field. In the dipole
approximation of the multipolar-coupling formalism, the interaction
Hamiltonian is given by 36

$H_{\rm
int}=-[\bm{\mu}_{1}\cdot\bm{D}_{\perp}(\bm{r}_{1})+\bm{\mu}_{2}\cdot\bm{D}_{\perp}(\bm{r}_{2})]/\epsilon_{0}\
.$ (13)

Here, $\bm{D}_{\perp}$ is the transverse component of the vacuum displacement
field given by

$\\!\\!\\!\bm{D}_{\perp}(\bm{r})=i\textstyle\sum\limits_{\bm{k},\lambda}\sqrt{\tfrac{\hbar
ck\epsilon_{0}}{2V}}\left[\hat{e}_{\bm{k}\lambda}a_{\bm{k}\lambda}\mathrm{e}^{i\bm{k}\cdot\bm{r}}-\bar{\hat{e}}_{\bm{k}\lambda}a_{\bm{k}\lambda}^{\dagger}\mathrm{e}^{-i\bm{k}\cdot\bm{r}}\right]$
(14)

with $V$ as the quantization volume. The annihilation and creation operators
of a mode with the wave vector $\bm{k}$ and polarization
$\hat{e}_{\bm{k}\lambda}$, $a_{\bm{k}\lambda}$ and
$a_{\bm{k}\lambda}^{\dagger}$, obey the bosonic commutation relations 36. The
ground-state of $H_{0}$ is given by
$|0\rangle\\!=\\!|0,0,0\rangle_{1}|0,0,0\rangle_{2}|\\{0\\}\rangle$, where the
QDO ket states are defined such that
$\langle\bm{r}_{i}|n_{x},n_{y},n_{z}\rangle_{i}=\psi_{\\{n_{x},n_{y},n_{z}\\}}(\bm{r}_{i})$,
and $|\\{0\\}\rangle$ is the ground state of the vacuum field. Excited states
of the total unperturbed system can be written similarly. By using these
states, we follow standard perturbation theory to obtain interaction energies
distinguishing between different contributions resulting from coupling of
either fluctuating or field-induced static QDO dipole moments, given by Eq.
11, to the vacuum field.

The 1st- and 3rd-order corrections vanish since in such cases
$a_{\bm{k}\lambda}$ and $a_{\bm{k}\lambda}^{\dagger}$ occur between two
identical states of the vacuum field. The 2nd-order perturbation,

$\mathcal{U}^{(2)}=-\textstyle\sum\limits_{{}_{I\neq 0}}\tfrac{\langle
0|H_{\rm int}|I\rangle\langle I|H_{\rm int}|0\rangle}{E_{I}-E_{0}}\ ,$ (15)

yields non-vanishing interaction terms only when the vacuum field in
$|I\rangle$ is in a single-photon excitation and both QDOs are in their ground
states, namely
$|I\rangle=|0,0,0\rangle_{1}|0,0,0\rangle_{2}|\bm{1}_{k\lambda}\rangle$. For
this case, after removing the self–energies,
$\Delta\mathcal{U}(R)=\mathcal{U}(R)-\mathcal{U}(\infty)$, we obtain an
interaction energy between the two QDOs given by

$\displaystyle\Delta\mathcal{U}^{(2)}(R)=\tfrac{\alpha_{1}\alpha_{2}(\mathcal{E}_{x}^{2}+\mathcal{E}_{y}^{2}-2\mathcal{E}_{z}^{2})}{4\pi\epsilon_{0}R^{3}}\
,$ (16)

which is the same as the second term of Eq. 9. Considering the QDO states in
$|I\rangle$, this interaction energy corresponds to the situation when both
QDOs couple to the vacuum field via their static field-induced dipoles and
exchange one virtual photon, indicating the electrostatic nature of this
interaction term. Taking into account the $R^{-3}$ scaling, this term
corresponds to a field-induced (dipole-dipole) electrostatic interaction.

For the 4th-order correction we have two terms 36

$\displaystyle\mathcal{U}^{(4)}\\!=-\textstyle\sum\limits_{{I,I\\!\\!I,I\\!\\!I\\!\\!I}\neq
0}\tfrac{\langle 0|H_{\rm int}|I\\!\\!I\\!\\!I\rangle\langle
I\\!\\!I\\!\\!I|H_{\rm int}|I\\!\\!I\rangle\langle I\\!\\!I|H_{\rm
int}|I\rangle\langle I|H_{\rm
int}|0\rangle}{(E_{I}-E_{0})(E_{I\\!\\!I}-E_{0})(E_{I\\!\\!I\\!\\!I}-E_{0})}\
\ +\textstyle\sum\limits_{{I,I\\!\\!I}\neq 0}\tfrac{\langle 0|H_{\rm
int}|I\\!\\!I\rangle\langle I\\!\\!I|H_{\rm int}|0\rangle\langle 0|H_{\rm
int}|I\rangle\langle I|H_{\rm
int}|0\rangle}{(E_{I}-E_{0})^{2}(E_{I\\!\\!I}-E_{0})}\ .$ (17)

For non-polar species, only the first term of Eq. 17 is relevant and hence
widely used in the literature. The second term becomes relevant for polar
atoms/molecules with permanent electrostatic moments, as they can couple to
the vacuum field via either fluctuating or static dipoles 36, 39. In the
latter case, the interacting species can emit/absorb a virtual photon without
undergoing a change in their energy eigenstate. In our case, the two QDOs
possess static field-induced dipoles due to the applied electric field and the
second term in Eq. 17 plays an important role. Specifically, when each QDO
couples to the vacuum field via its static field-induced dipole, both 4th-
order terms in Eq. 16 yield non-vanishing contributions $\propto R^{-5}$ of
the same magnitude but opposite sign, cancelling each other 65. When both
atoms couple to the vacuum field via their fluctuating dipoles, the treatment
of Eq. 17 as in Refs. 37, 36 delivers 65 the known London and Casimir-Polder
dispersion energies

$\displaystyle\Delta\mathcal{U}_{1}^{(4)}(R)=\left\\{\begin{matrix}-\frac{3\alpha_{1}\alpha_{2}~{}\omega_{1}\omega_{2}\hbar}{2\,(4\pi\epsilon_{0})^{2}\,(\omega_{1}+\omega_{2})R^{6}}~{}\
,\ \ R\ll c/\omega\ ,\\\ -\frac{23\hbar
c~{}\alpha_{1}\alpha_{2}}{4\pi\,(4\pi\epsilon_{0})^{2}R^{7}}\qquad\ \ \ ,\ \ \
R\gg c/\omega\ ,\end{matrix}\right.$ (18)

respectively. The former is the first term of Eq. 9 and the latter is its
counterpart for the retarded regime, where the frequencies disappear since at
large distances each species effectively “sees” another one as a static
object. Finally, when one of the species couples to the vacuum field via its
static dipole moment and the other one by its fluctuating dipole, the
resulting interaction energy is

$\Delta\mathcal{U}_{2}^{(4)}(R)=-\tfrac{\alpha_{1}\alpha_{2}(\alpha_{1}+\alpha_{2})~{}(\mathcal{E}_{x}^{2}+\mathcal{E}_{y}^{2}+4\mathcal{E}_{z}^{2})}{2[4\pi\epsilon_{0}]^{2}R^{6}}\
,$ (19)

which is the third term of Eq. 9. This interaction stems from the 4th-order
correction and has $R^{-6}$ scaling, similar to the nonretarded dispersion
energy, but in contrast to the latter remains unaffected by retardation. These
features allow us to identify the term of Eq. 19 as the field-induced
polarization energy. Within the employed QED approach, one can also take into
account the influence of a static field on the cloud of virtual photons
surrounding each atom 76, 77, 78, 79, 80. Although the effect of distortions
of such photon clouds on the molecular interactions is usually assumed to be
small, it might become important under external fields, similar to the
aforementioned effects caused by hyperpolarizabilities. Moreover, the impact
of clouds of virtual photons on the molecular interactions can become even
more important for atoms at close separation, i.e. when electron exchange
effects become crucial. Indeed, the distortion of photon clouds by a static
field can change the effective atomic vdW radii, but to investigate such
effects in detail 55 one would need to go beyond the perturbative QED approach
used here.

In addition to the two quantum approaches discussed above, we present a
(semi)classical derivation that allows us to transparently connect different
molecular interactions to the fields originating them. To this end, we employ
a SED approach developed by Boyer 81, 82 based on the theory of classical
electrodynamics with a random zero-point radiation field. Within this picture,
the fluctuating vacuum field induces random polarization of atoms (modeled by
classical oscillators) and couples them to each other through their
electromagnetic fields, as described in classical electrodynamics. For the
retarded regime, $R\gg c/\omega$, only large wavelengths contribute to the
interatomic interaction and the SED equation of motion for a dipole oscillator
reduces to $m\omega^{2}\bm{r}=q\bm{E}(\bm{r},t)$. Solving this equation yields
$\bm{\mu}(\bm{r},t)=\alpha\bm{E}(\bm{r},t)$, with $\bm{E}$ as the total
electric field at point $\bm{r}$ and time $t$. The energy of the electric
dipole moment induced by $\bm{E}$ in the same field is known from
electrodynamics as $\mathcal{U}=-\frac{1}{2}\alpha\langle\bm{E}^{2}\rangle$,
where the bracket indicates time averaging.

Since we apply a static electric field on top of the random radiation field,
the oscillator dipole has two parts, each related to one of these fields.
According to Fig. 1, the first QDO is located at the origin, while we bring
the second QDO to the point $\bm{r}_{2}=(0,0,R)$ from $z=+\infty$. The
difference in the energy of the oscillators for the two configurations,
$\Delta\mathcal{U}(R)=\mathcal{U}(R)-\mathcal{U}(\infty)$, is the interaction
energy. The total electric field at $\bm{r}_{2}$ is given by

$\bm{E}(\bm{r}_{2},t)=\bm{E}_{0}(\bm{r}_{2},t)+\bm{E}_{\mu_{1}}(\bm{r}_{2},t)+\bm{\mathcal{E}}_{\mu_{1}}(\bm{r}_{2})+\bm{\mathcal{E}}\
,$ (20)

with $\bm{E}$ and $\bm{\mathcal{E}}$ as radiation and static fields,
respectively. Here, the random radiation field is defined by 43

$\displaystyle\bm{E}_{0}=\mathrm{Re}\textstyle\sum\limits_{\lambda=1}^{2}\int
d^{3}k\tfrac{\bm{\epsilon}(\bm{k},\lambda)\
\mathfrak{h}(\bm{k},\lambda)}{\sqrt{4\pi\epsilon_{0}}}\
\mathrm{e}^{i[\bm{k}\cdot\bm{r}-kct+\theta(\bm{k},\lambda)]}\ ,$ (21)

where, $\mathfrak{h}^{2}(\bm{k},\lambda)=\hbar kc/2\pi^{2}$ is the energy of
each mode of the random field and the sum runs over the two possible
polarizations. The random phase $\theta$ ranges from $0$ to $2\pi$ and
$\bm{\epsilon}(\bm{k},\lambda)$ are orthogonal unit polarization vectors,
$\bm{\epsilon}(\bm{k},\lambda)\cdot\bm{\epsilon}(\bm{k}^{\prime},\lambda^{\prime})=\delta_{\lambda\lambda^{\prime}}$.
Then, $\bm{E}_{\mu_{1}}(\bm{r}_{2},t)$ is a time-dependent field radiated by
the oscillating dipole of the first species induced by the random radiation
field. Similarly, $\bm{\mathcal{E}}_{\mu_{1}}(\bm{r}_{2})$ is the electric
field of the static dipole of the first oscillator, which is induced by the
uniform external electric field. The electric field of an oscillating electric
dipole is 83

$\displaystyle\begin{array}[]{ll}\\!\\!\bm{E}_{\mu}(\bm{r},t)=\mathrm{Re}\left[\left(\tfrac{k^{2}(\bm{n}\times\bm{\mu})\times\bm{n}}{4\pi\epsilon_{0}r}+(1-ikr)\bm{\mathcal{E}}_{\mu}\right)\mathrm{e}^{ikr}\right],\\!\\!\end{array}$
(23)

where
$\bm{\mathcal{E}}_{\mu}(\bm{r})=\left(3\bm{n}(\bm{\mu}\cdot\bm{n})-\bm{\mu}\right)/4\pi\epsilon_{0}r^{3}$
is the electric field of a static electric dipole. Therefore, the energy of
the second oscillator, with the static polarizability $\alpha_{2}$, under the
total electric field given by Eq. 20 is obtained as
$\mathcal{U}(\bm{r}_{2})=-\frac{1}{2}\alpha_{2}\left\langle\bm{E}^{2}(\bm{r}_{2},t)\right\rangle$.
Then, subtracting energies for $z_{2}=R$ and $z_{2}=+\infty$, yields

$\displaystyle\begin{array}[]{ll}\Delta\mathcal{U}(R)=-\alpha_{2}\left[\big{\langle}\bm{E}_{0}(\bm{R},t)\cdot\bm{E}_{\mu_{1}}(\bm{R},t)\big{\rangle}+\big{\langle}\bm{\mathcal{E}}\cdot\bm{\mathcal{E}}_{\mu_{1}}(\bm{R})\big{\rangle}+\frac{1}{2}\big{\langle}\bm{\mathcal{E}}_{\mu_{1}}(\bm{R})\cdot\bm{\mathcal{E}}_{\mu_{1}}(\bm{R})\big{\rangle}\right]\
,\end{array}$ (25)

where we keep only nonvanishing terms after time and phase averaging 65. The
first term in Eq. 25 gives the interaction energy from the coupling of two
fluctuating dipole moments induced by the random field. This point was
addressed by Boyer 43 who obtained the aforementioned London 84, 85 and
Casimir-Polder 81, 82, 86 results for the nonretarded and retarded regimes,
respectively. Here, we focus on other contributions to the interaction energy.
The second term in Eq. 25 is the coupling of the field-induced dipole of the
second oscillator with the static field of the field-induced dipole of the
first oscillator which gives the electrostatic interaction energy

$\\!\\!\\!\Delta\mathcal{U}_{2}(R)\\!=\\!-\alpha_{2}\bm{\mathcal{E}}\\!\cdot\\!\left[\tfrac{3\hat{z}(\alpha_{1}\bm{\mathcal{E}}\cdot\hat{z})-\alpha_{1}\bm{\mathcal{E}}}{4\pi\epsilon_{0}R^{3}}\right]=\tfrac{\alpha_{1}\alpha_{2}(\mathcal{E}_{x}^{2}+\mathcal{E}_{y}^{2}-2\mathcal{E}_{z}^{2})}{4\pi\epsilon_{0}R^{3}}\
.$ (26)

The third term of Eq. 25 describes interaction of the dipole moment of the
second oscillator induced by the static field of the first oscillator, with
the same field and yields the interaction energy

$\displaystyle\\!\\!\\!\Delta\mathcal{U}_{3}(R)\\!=\\!-\tfrac{\alpha_{2}}{2}\\!\left(\tfrac{3\hat{z}(\alpha_{1}\bm{\mathcal{E}}\cdot\hat{z})-\alpha_{1}\bm{\mathcal{E}}}{4\pi\epsilon_{0}R^{3}}\right)^{2}\\!\\!\\!=\\!-\tfrac{\alpha_{2}\alpha_{1}^{2}(\mathcal{E}_{x}^{2}+\mathcal{E}_{y}^{2}+4\mathcal{E}_{z}^{2})}{2(4\pi\epsilon_{0})^{2}R^{6}}\
.$ (27)

The mechanism responsible for this interaction is similar to the one for
intermolecular polarization (induction) interaction between molecules with
permanent dipole moments, with the difference that here the static dipoles are
induced by the external field and hence the interaction is a field-induced
polarization interaction. Adding the term coming from the interaction of the
dipole moment of the first oscillator with the static field of the field-
induced dipole of the second oscillator, yields the full field-induced
polarization energy given by Eq. 19. The polarization energy of Eq. 27 was
originally derived from a pure classical point of view, by considering the
coupling of a _pair_ of polarizable objects, interacting and therefore
possessing the collisional contribution $\Delta\alpha$ to the _pair
polarizability_ , to an external static field 10, 11, 12, 13, 14, 15, 16.

Figure 2: The dispersion force ($F_{\rm disp}$) as well as the field-induced
electrostatic ($F_{\rm el}$) and polarization ($F_{\rm pol}$) forces for two
interacting benzene molecules separated by the corresponding equilibrium
distance $R_{\rm eq}=4.75$Å and $R_{\rm eq}=3.54$Å within (a) T-Shape and (b)
Sandwich structure, respectively. The results are shown for two different
alignments of the external field and the molecules: the field is either
parallel $(||)$ or perpendicular $(\perp)$ to the line connecting the molecule
centers.

Our results, obtained within the QDO model for the linear-response regime,
show that the field-induced electrostatic and polarization energies are not
influenced by the retardation effects and only the conventional leading-order
London/Casimir dispersion energy changes the distance scaling law going from
the nonretarded to retarded regime. However, without aforementioned
contributions coming from dispersion–induced changes in the polarizability 58,
12, 13, 14, 62, 59, 64, 16, 61, 60, 63 and dipole moments 25, 66, 67, 68, 15,
69, 70, 71, 72, 73, 74 of interacting species as well as field-induced
hyperpolarization effects 87, 57, the dispersion interaction remains unchanged
by the external uniform static field, for both regimes. This fact is in
contradiction to the recent results of Fiscelli _et al._ 24, who obtained a
drastic change in the distance dependence of dispersion interaction energy
between two atoms after applying a static electric field ($R^{-3}$/$R^{-4}$
for the nonretarded/retarded regime). In Ref. 24, the wavefunctions of a two-
level “hydrogen” atom in a static electric field were obtained from
perturbation theory. The interaction energy between two atoms in a field was
calculated from QED perturbation theory using the atomic wavefunctions
obtained in the first step. However, since these functions do not form a
complete set, employing them as a basis set for the second use of the
perturbation theory in Ref. 24 is questionable. A similar conclusion was
recently obtained in Ref. 57 independently of our work 65.

Due to the analytical solutions of the QDOs, the results of Eq. 9 can be
straightforwardly generalized to any number of QDOs, with each of them under
their own static field. This provides an opportunity to effectively model
internal atom-dependent electric fields present in large molecules. Moreover,
the developed framework paves the way for tuning intermolecular interactions
by means of external static electric fields, which is important for many
practical applications including biophysics 88. This is demonstrated in Fig.
2, where the intermolecular forces in a benzene dimer are considered for its
two different equilibrium structures, the so called “T-Shape” and “Sandwich”
89, 90. The interplay between the field-induced forces and the dispersion
force can lead either to cooperation or competition. In the case of the
“T-Shape” structure, the total electrostatic force can overtake the dispersion
force, if a static electric field with strength 2.25 V/Å is applied
perpendicularly to the dimer. For the “Sandwich” configuration, the strength
of the perpendicular compensating field is around 3.22 V/Å. Although for the
considered molecular dimers at the equilibrium distance these fields are
relatively strong, they are still weaker than internal molecular fields
experienced by valence electrons. Moreover, as shown in Ref. 65, the field
required for mutual compensation of the intermolecular forces becomes much
weaker at larger separations. Furthermore, for large molecules or nanoscale
systems with a multiplicity of normal modes contributing to Eq. 8, a
significantly smaller magnitude of the electric field will be required to
overcome the dispersion attraction 91. Altogether, this shows that applied
static electric fields can be used to influence the stability and dynamics of
complex molecular system.

Before summarizing our results, it is important to emphasize some of the
remaining limitations of the employed coupled QDO model, whose resolution
would be needed to form a complete physical picture of intermolecular
interactions under a static electric field. As was exhaustively discussed in
Ref. 33, a QDO does possess multipole hyperpolarizabilities starting from the
dipole-dipole-quadrupole one, but the first ($\beta$) and second ($\gamma$)
hyperpolarizabilities vanish due to the spherical symmetry and Gaussian
wavefunction, respectively. Therefore, a single QDO does not fully capture the
contributions to the intermolecular interaction energy which are related to
hyperpolarization effects of interacting atoms and molecules in static fields.
For a pair of two-level hydrogen atoms it is known 57 that the energy
contribution from the field-induced $\beta$ hyperpolarizabilities of the atoms
scales as $R^{-11}$. Therefore, this contribution can be neglected in
comparison to the dispersion and field-induced electrostatic and polarization
interactions. It has been shown that dispersion interactions modify the
polarizability of an interacting pair where the leading-order correction
scales as $R^{-6}$ and depends on the $\gamma$ hyperpolarizability of atoms or
molecules 58, 12, 13, 14, 62, 59, 64, 16, 61, 60, 63. In a static electric
field, the dispersion-induced polarizability yields an additional interaction
energy (quadratic in the applied field) that scales as $R^{-6}$ being
comparable 14, 63 to the field-induced polarization energy in Eq. 9. To
capture a complete picture of the dipolar interactions in a static field, one
could use more than one QDO to represent atoms or molecules, which brings
anharmonicity to the system and breaks the spherical symmetry of the model
when interacting with external fields. Such a multi-QDO model would exhibit
$\beta$ and $\gamma$ hyperpolarizabilities and thus provide more realistic
response properties in comparison to single QDOs. The dipolar interactions
between two (or many) multi-QDO systems can be expressed as a coupled-QDO
problem, which is still exactly solvable.

In addition, the considered contributions to molecular interactions between
atoms or molecules represented by QDOs were derived assuming large
interspecies distances, effectively neglecting exchange and overlap effects.
Within the QDO model, an inclusion of the exchange interaction is possible
with a further generalization of the existing formalism valid for two
identical QDOs 54, 92 to the case of heteronuclear species. The overlap
effects can be also included following the work of Refs. 59, 93.

In summary, we presented dominant contributions to the interaction energy
between two closed-shell atoms or molecules in a uniform static electric field
for both, short (nonretarded regime) and long (retarded regime) separation
distances. Based on first principles of quantum mechanics and QED, and
employing the QDO model as a reliable tool for modeling atomic/molecular
responses, our framework admits many generalizations. In addition to external
fields, the internal field from atomic charges and dipoles within a molecule
can be treated as well, leading to an efficient many-body model of different
types of intramolecular and intermolecular interactions on equal footing. More
general time-dependent fields can also be included. A particularly interesting
and novel research direction is to develop and apply a QED treatment to many-
body states for a set of many interacting QDOs. This could generalize the
transition from the London to Casimir regime for an assembly of atoms or
molecules beyond the well-known two-body case. All in all, we are confident
that our framework barely scratches the surface of possible developments and
applications in the field of molecular interactions under the combined action
of external and vacuum fields.

The authors acknowledge the financial support from the Luxembourg National
Research Fund through the FNR CORE projects “QUANTION(C16/MS/11360857,
GrNum:11360857)” and “PINTA(C17/MS/11686718)” as well as from the European
Research Council via ERC Consolidator Grant “BeStMo(GA n725291)”.

## References

  * Kaplan 2006 Kaplan, I. G. _Intermolecular interactions: physical picture, computational methods and model potentials_ ; John Wiley & Sons: Sussex, 2006
  * Stone 2013 Stone, A. J. _The theory of intermolecular forces_ ; Oxford University Press: Oxford, 2013
  * Tkatchenko 2015 Tkatchenko, A. _Adv. Func. Mat._ 2015, _25_ , 2054–2061
  * Langbein 1974 Langbein, D. _Theory of van der Waals attraction_ ; Springer tracts in modern physics, Vol. 72, Springer-Verlag: Berlin, 1974
  * Ángyán et al. 2020 Ángyán, J.; Dobson, J.; Jansen, G.; Gould, T. _London dispersion forces in molecules, solids and nano-structures: an introduction to physical models and computational methods_ ; Royal Society of Chemistry: Cambridge, 2020
  * Sadhukhan and Tkatchenko 2017 Sadhukhan, M.; Tkatchenko, A. _Phys. Rev. Lett._ 2017, _118_ , 210402
  * Stöhr et al. 2021 Stöhr, M.; Sadhukhan, M.; Al-Hamdani, Y. S.; Hermann, J.; Tkatchenko, A. _Nat. Commun._ 2021, _12_ , 137
  * Stöhr and Tkatchenko 2019 Stöhr, M.; Tkatchenko, A. _Sci. Adv._ 2019, _5_ , eaax0024
  * Kleshchonok and Tkatchenko 2018 Kleshchonok, A.; Tkatchenko, A. _Nat. Commun._ 2018, _9_ , 3017
  * Silberstein 1917 Silberstein, L. _Phil. Mag._ 1917, _33_ , 92–128
  * Silberstein 1917 Silberstein, L. _Phil. Mag._ 1917, _33_ , 521–533
  * Buckingham 1956 Buckingham, A. D. _Trans. Faraday Soc._ 1956, _52_ , 1035–1041
  * Buckingham et al. 1973 Buckingham, A. D.; Martin, P. H.; Watts, R. S. _Chem. Phys. Lett._ 1973, _21_ , 186–190
  * Buckingham and Clarke 1978 Buckingham, A. D.; Clarke, K. L. _Chem. Phys. Lett._ 1978, _57_ , 321–325
  * Hunt and Bohr 1985 Hunt, K. L. C.; Bohr, J. E. _J. Chem. Phys._ 1985, _83_ , 5198–5202
  * Hunt and Bohr 1986 Hunt, K. L. C.; Bohr, J. E. _J. Chem. Phys._ 1986, _84_ , 6141–6150
  * Thirunamachandran 1980 Thirunamachandran, T. _Mol. Phys._ 1980, _40_ , 393–399
  * Haugland et al. 2021 Haugland, T. S.; Schäfer, C.; Ronca, E.; Rubio, A.; Koch, H. _J. Chem. Phys._ 2021, _154_ , 094113
  * Milonni and Smith 1996 Milonni, P. W.; Smith, A. _Phys. Rev. A_ 1996, _53_ , 3484–3489
  * Sukhov et al. 2013 Sukhov, S.; Douglass, K. M.; Dogariu, A. _Opt. Lett._ 2013, _38_ , 2385–2387
  * Brügger et al. 2015 Brügger, G.; Froufe-Pérez, L. S.; Scheffold, F.; Sáenz, J. J. _Nat. Commun._ 2015, _6_ , 7460
  * Marinescu and You 1998 Marinescu, M.; You, L. _Phys. Rev. Lett._ 1998, _81_ , 4596–4599
  * Muruganathan et al. 2015 Muruganathan, M.; Sun, J.; Imamura, T.; Mizuta, H. _Nano Lett._ 2015, _15_ , 8176–8180
  * Fiscelli et al. 2020 Fiscelli, G.; Rizzuto, L.; Passante, R. _Phys. Rev. Lett._ 2020, _124_ , 013604
  * Feynman 1939 Feynman, R. P. _Phys. Rev._ 1939, _56_ , 340
  * Hunt 1990 Hunt, K. L. C. _The Journal of Chemical Physics_ 1990, _92_ , 1180
  * Podeszwa and Jansen 2018 Podeszwa, R.; Jansen, G. _Phys. Rev. Lett._ 2018, _120_ , 258901
  * Sadhukhan and Tkatchenko 2018 Sadhukhan, M.; Tkatchenko, A. _Phys. Rev. Lett._ 2018, _120_ , 258902
  * Abrantes et al. 2021 Abrantes, P. P.; Pessanha, V.; Farina, C.; de Melo e Souza, R. _Phys. Rev. Lett._ 2021, _126_ , 109301
  * Fiscelli et al. 2021 Fiscelli, G.; Rizzuto, L.; Passante, R. _Phys. Rev. Lett._ 2021, _126_ , 109302
  * Wang and Jordan 2001 Wang, F.; Jordan, K. D. _J. Chem. Phys._ 2001, _114_ , 10717–10724
  * Sommerfeld and Jordan 2005 Sommerfeld, T.; Jordan, K. D. _J. Phys. Chem. A_ 2005, _109_ , 11531–11538
  * Jones et al. 2013 Jones, A. P.; Crain, J.; Sokhan, V. P.; Whitfield, T. W.; Martyna, G. J. _Phys. Rev. B_ 2013, _87_ , 144103
  * Cohen-Tannoudji et al. 1997 Cohen-Tannoudji, C.; Dupont-Roc, J.; Grynberg, G. _Photons and atoms-introduction to quantum electrodynamics_ ; Wiley: New York, 1997
  * Milonni 1994 Milonni, P. W. _The quantum vacuum: an introduction to quantum electrodynamics_ ; Academic press: San Diego, 1994
  * Craig and Thirunamachandran 1994 Craig, D. P.; Thirunamachandran, T. _Molecular quantum electrodynamics: an introduction to radiation-molecule interactions_ ; Academic Press: London, 1994
  * Salam 2009 Salam, A. _Molecular quantum electrodynamics: long-range intermolecular interactions_ ; Wiley: Hoboken, 2009
  * Buhmann 2012 Buhmann, S. Y. _Dispersion forces I: macroscopic quantum electrodynamics and ground-state Casimir, Casimir–Polder and van der Waals forces_ ; Springer: London, 2012
  * Passante 2018 Passante, R. _Symmetry_ 2018, _10_ , 735
  * Schäfer et al. 2018 Schäfer, C.; Ruggenthaler, M.; Rubio, A. _Phys. Rev. A_ 2018, _98_ , 043801
  * Marshall 1963 Marshall, T. W. _Proc. R. Soc. Lond. A_ 1963, _276_ , 475–491
  * Marshall 1965 Marshall, T. W. _Math. Proc. Cambridge Philos. Soc._ 1965, _61_ , 537–546
  * Boyer 1975 Boyer, T. H. _Phys. Rev. D_ 1975, _11_ , 790–808
  * de la Peña and Cetto 1978 de la Peña, L.; Cetto, A. M. _Found. Phys._ 1978, _8_ , 191–210
  * Boyer 1980 Boyer, T. H. _A brief survey of stochastic electrodynamics $\mathrm{in}$ “Foundations of radiation theory and quantum electrodynamics”_; Springer: New York, 1980
  * de la Peña and Cetto 1996 de la Peña, L.; Cetto, A. M. _The quantum dice: an introduction to stochastic electrodynamics_ ; Springer: Dordrecht, 1996
  * Nieuwenhuizen 2019 Nieuwenhuizen, T. M. _Atoms_ 2019, _7_ , 59
  * Boyer 2019 Boyer, T. H. _Atoms_ 2019, _7_ , 29
  * Tkatchenko et al. 2012 Tkatchenko, A.; R. A. DiStasio, J.; Car, R.; Scheffler, M. _Phys. Rev. Lett._ 2012, _108_ , 236402
  * Reilly and Tkatchenko 2015 Reilly, A. M.; Tkatchenko, A. _Chem. Sci._ 2015, _6_ , 3289–3301
  * 51 V. V. Gobre, Efficient modelling of linear electronic polarization in materials using atomic response functions, Ph.D. thesis, Fritz Haber Institute Berlin, 2016.
  * Sadhukhan and Manby 2016 Sadhukhan, M.; Manby, F. R. _Phys. Rev. B_ 2016, _94_ , 115106
  * J. Hermann and Tkatchenko 2017 J. Hermann, J., R. A. DiStasio; Tkatchenko, A. _Chem. Rev._ 2017, _117_ , 4714–4758
  * Fedorov et al. 2018 Fedorov, D. V.; Sadhukhan, M.; Stöhr, M.; Tkatchenko, A. _Phys. Rev. Lett._ 2018, _121_ , 183401
  * Tkatchenko et al. 2021 Tkatchenko, A.; Fedorov, D. V.; Gori, M. _J. Phys. Chem. Lett._ 2021, _12_ , 9488–9492
  * Hermann et al. 2017 Hermann, J.; Alfè, D.; Tkatchenko, A. _Nat. Commun._ 2017, _8_ , 14052
  * Hu et al. 2021 Hu, Y.; Hu, J.; Yu, H. _Phys. Rev. A_ 2021, _103_ , 042803
  * Jansen and Mazur 1955 Jansen, L.; Mazur, P. _Physica_ 1955, _21_ , 193–207
  * Hunt and Buckingham 1980 Hunt, K. L. C.; Buckingham, A. D. _J. Chem. Phys._ 1980, _72_ , 2832–2840
  * Li and Hunt 1994 Li, X.; Hunt, K. L. C. _J. Chem. Phys._ 1994, _100_ , 7875–7889
  * Fowler et al. 1994 Fowler, P. W.; Hunt, K. L. C.; Kelly, H. M.; Sadlej, A. J. _J. Chem. Phys._ 1994, _100_ , 2932–2935
  * Certain and Fortune 1971 Certain, P. R.; Fortune, P. J. _J. Chem. Phys._ 1971, _55_ , 5818–5821
  * Joslin et al. 1996 Joslin, C. G.; Goddard, J. D.; Goldman, S. _Mol. Phys._ 1996, _89_ , 791–808
  * Hunt et al. 1981 Hunt, K. L. C.; Zilles, B. A.; Bohr, J. E. _J. Chem. Phys._ 1981, _75_ , 3079–3086
  * Karimpour et al. 2022 Karimpour, M. R.; Fedorov, D. V.; Tkatchenko, A. _Phys. Rev. Research_ 2022, _4_ , 013011
  * Hunt 1980 Hunt, K. L. C. _Chem. Phys. Lett._ 1980, _70_ , 336–342
  * Galatry and Gharbi 1980 Galatry, L.; Gharbi, T. _Chem. Phys. Lett._ 1980, _75_ , 427–433
  * Galatry and Hardisson 1983 Galatry, L.; Hardisson, A. _The Journal of Chemical Physics_ 1983, _79_ , 1758–1765
  * Craig and Thirunamachandran 1981 Craig, D. P.; Thirunamachandran, T. _Chemical Physics Letters_ 1981, _80_ , 14–17
  * Bohr and Hunt 1987 Bohr, J. E.; Hunt, K. L. C. _J. Chem. Phys._ 1987, _86_ , 5441–5448
  * Bohr and Hunt 1987 Bohr, J. E.; Hunt, K. L. C. _J. Chem. Phys._ 1987, _87_ , 3821–3832
  * Fowler 1990 Fowler, P. W. _Chem. Phys._ 1990, _143_ , 447–457
  * Vigoureux and Galatry 1983 Vigoureux, J.; Galatry, L. _Chem. Phys. Lett._ 1983, _98_ , 324–328
  * Odbadrakh and Jordan 2016 Odbadrakh, T. T.; Jordan, K. D. _J. Chem. Phys._ 2016, _144_ , 034111
  * Atkins and Friedman 2005 Atkins, P.; Friedman, R. _Molecular quantum mechanics_ ; Oxford University Press: Oxford, 2005
  * Passante et al. 1985 Passante, R.; Compagno, G.; Persico, F. 1985, _31_ , 2827–2841
  * Compagno et al. 1985 Compagno, G.; Persico, F.; Passante, R. _Physics Letters_ 1985, _112A_ , 215–219
  * Passante and Power 1987 Passante, R.; Power, E. A. _Phys. Rev. A_ 1987, _35_ , 188–197
  * Compagno and Palma 1988 Compagno, G.; Palma, M. G. _Phys. Rev. A_ 1988, _37_ , 2979–2984
  * Compagno et al. 1992 Compagno, G.; Vivirito, S.; Persico, F. _Phys. Rev. A_ 1992, _46_ , 7303–7306
  * Boyer 1969 Boyer, T. H. _Phys. Rev._ 1969, _180_ , 19–24
  * Boyer 1972 Boyer, T. H. _Phys. Rev. A_ 1972, _5_ , 1799–1802
  * Jackson 1998 Jackson, J. D. _Classical electrodynamics_ ; Wiley: Hoboken, 1998
  * Boyer 1972 Boyer, T. H. _Phys. Rev. A_ 1972, _6_ , 314–319
  * Boyer 2018 Boyer, T. H. _Eur. J. Phys._ 2018, _39_ , 045401
  * Boyer 1973 Boyer, T. H. _Phys. Rev. A_ 1973, _7_ , 1832–1840
  * Salam 1997 Salam, A. _Phys. Rev. A_ 1997, _55_ , 804–807
  * Martin et al. 2016 Martin, J. M.; Li, W.; Delaney, K. T.; Fredrickson, G. H. _J. Chem. Phys._ 2016, _145_ , 154104
  * Dinadayalane and Leszczynski 2009 Dinadayalane, T. C.; Leszczynski, J. _Struct. Chem._ 2009, _20_ , 11–20
  * Archibong and Thakkar 1994 Archibong, E. F.; Thakkar, A. J. _Mol. Phys._ 1994, _81_ , 557–567
  * Ambrosetti et al. 2016 Ambrosetti, A.; Ferri, N.; R. A. DiStasio, J.; Tkatchenko, A. _Science_ 2016, _351_ , 1171–1176
  * Vaccarelli et al. 2021 Vaccarelli, O.; Fedorov, D. V.; Stöhr, M.; Tkatchenko, A. _Phys. Rev. Research_ 2021, _3_ , 033181
  * Hunt 1984 Hunt, K. L. C. _J. Chem. Phys._ 1984, _80_ , 393–407


# Character-Aware Decoder for Translation
into Morphologically Rich Languages

Adithya Renduchintala∗ Pamela Shapiro∗ Kevin Duh  Philipp Koehn
Department of Computer Science
Johns Hopkins University
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Neural machine translation (NMT) systems operate primarily on words (or
subwords), ignoring lower-level patterns of morphology. We present a
_character-aware_ decoder designed to capture such patterns when translating
into morphologically rich languages. We achieve character-awareness by
augmenting both the softmax and embedding layers of an attention-based
encoder-decoder model with convolutional neural networks that operate on the
spelling of a word. To investigate performance on a wide variety of
morphological phenomena, we translate English into $14$ typologically diverse
target languages using the TED multi-target dataset. In this low-resource
setting, the character-aware decoder provides consistent improvements with
BLEU score gains of up to $+3.05$. In addition, we analyze the relationship
between the gains obtained and properties of the target language and find
evidence that our model does indeed exploit morphological patterns. ††∗Equal
Contribution

##  1 Introduction

Traditional attention-based encoder-decoder neural machine translation (NMT)
models learn _word-level_ embeddings, with a continuous representation for
each unique word type [Bahdanau et al., 2015]. However, this results in a long
tail of rare words for which we do not learn good representations. More
recently, it has become standard practice to mitigate the vocabulary size
problem with Byte-Pair Encoding (BPE) [Gage, 1994, Sennrich et al., 2016]. BPE
iteratively merges consecutive characters into larger chunks based on their
frequency, which results in the breaking up of less common words into “subword
units.”

While BPE addresses the vocabulary size problem, the spellings of the subword
units are still ignored. On the other hand, purely _character-level_ NMT
translates one character at a time and can implicitly learn about
morphological patterns within words as well as generalize to unseen
vocabulary. Recently, ?) show that very deep character-level models can
outperform BPE, however, the smallest data size evaluated was 2 million
sentences, so it is unclear if the results hold for low-resource settings and
when translating into a range of different morphologically rich languages.
Furthermore, tuning deep character-level models is expensive, even for low-
resource settings. Deep character-level models are sensitive to the dropout
rate and tuning takes much longer due to longer sequence lengths [Cherry et
al., 2018].

A middle-ground alternative is character-_aware_ word-level modeling. Here,
the NMT system operates over words but uses word embeddings that are sensitive
to spellings and thereby has the ability to learn morphological patterns in
the language. Such character-aware approaches have been applied successfully
in NMT to the _source-side_ word embedding layer [Costa-jussà and Fonollosa,
2016], but surprisingly, similar gains have not been achieved on the target
side [Belinkov et al., 2017].

While source-side character-aware models only need to make the _source
embedding layer_ character-aware, on the target-side we require both the
_target embedding layer_ and the _softmax layer_ to be character-aware, which
presents additional challenges. We find that the trivial application of
methods from ?) to these target-side embeddings results in significant drop in
performance. Instead, we propose mixing compositional and standard word
embeddings via a gating function. While simple, we find it is critical to
successful target-side character awareness.

It is worth noting that unlike some purely character-level methods our aim is
not to generate novel words, though this method can function on top of subword
methods which do so [Shapiro and Duh, 2018]. Rather, the character-aware
representations decrease the sparsity of embeddings for rare words or
subwords, which are a problem in low-resource morphologically rich settings.
We summarize our contribution as follows:

  1. 1.

We propose a method for utilizing character-aware embeddings in an NMT decoder
that can be used over word or subword sequences.

  2. 2.

We explore how our method interacts with BPE over a range of merge operations
(including word-level and purely character-level) and highlight that there is
no “typical BPE” setting for low-resource NMT.

  3. 3.

We evaluate our model on $14$ target languages and observe consistent
improvements over baselines. Furthermore, we analyze to what extent the
success of our method corresponds to improved handling of target language
morphology.

##  2 Related Work

NMT has benefited from character-aware word representations on the source side
[Costa-jussà and Fonollosa, 2016], which follows language modeling work by ?)
and generate source-side input embeddings using a CNN over the character
sequence of each word. Further analysis revealed that hidden states of such
character-aware models have increased knowledge of morphology [Belinkov et
al., 2017]. They additionally try using character-aware representations in the
target side embedding layer, leaving the softmax matrix with standard word
representations, and found no improvements.

Our work is also aligned with the character-aware models proposed in [Kim et
al., 2016], but we additionally employ a gating mechanism between character-
aware representations and standard word representations similar to language
modeling work by [Miyamoto and Cho, 2016]. However, our gating is a learned
type-specific vector rather than a fixed hyperparameter.

There is additionally a line of work on purely character-level NMT, which
generates words one character at a time [Ling et al., 2015, Chung et al.,
2016, Passban et al., 2018]. While initial results here were not strong, ?)
revisit this with deeper architectures and sweeping dropout parameters and
find that they outperform BPE across settings of the merge hyperparameter.
They examine different data sizes and observe improvements in the smaller data
size settings—however, the smallest size is about 2 million sentence pairs. In
contrast, we look at a smaller order of magnitude data size and present an
alternate approach which doesn’t require substantial tuning of parameters
across different languages.

Finally, Byte-Pair Encoding (BPE) [Sennrich et al., 2016] has become a
standard preprocessing step in NMT pipelines and provides an easy way to
generate sequences with a mixture of full words and word fragments. Note that
BPE splits are agnostic to any morphological pattern present in the language,
for example the token politely in our dataset is split into pol+itely, instead
of the linguistically plausible split polite+ly.111We observe this split when
merge parameter was $15$k. Our approach can be applied to word-level sequences
and sequences at any BPE merge hyperparameter greater than $0$. Increasing the
hyperparameter results in more words and longer subwords that can exhibit
morphological patterns. Our goal is to exploit these morphological patterns
and enrich the word (or subword) representations with character-awareness.

##  3 Encoder-Decoder NMT

An attention-based encoder-decoder network [Bahdanau et al., 2015, Luong et
al., 2015] models the probability of a target sentence
${\boldsymbol{\mathbf{y}}}$ of length $J$ given a source sentence
${\boldsymbol{\mathbf{x}}}$ as:

$\displaystyle p({\boldsymbol{\mathbf{y}}}\mid{\boldsymbol{\mathbf{x}}})$
$\displaystyle=\prod_{j=1}^{J}p(y_{j}\mid{\boldsymbol{\mathbf{y}}}_{0:j-1},{\boldsymbol{\mathbf{x}}};{\boldsymbol{\mathbf{\theta}}})$
(1)

where ${\boldsymbol{\mathbf{\theta}}}$ represents all the parameters of the
network. At each time-step the $j^{\prime}$th output token is generated by:

$\displaystyle
p(y_{j}\mid{\boldsymbol{\mathbf{y}}}_{0:j-1},{\boldsymbol{\mathbf{x}}})=\text{softmax}({\boldsymbol{\mathbf{W_{o}}}}{\boldsymbol{\mathbf{s}}}_{j})$
(2)

where ${\boldsymbol{\mathbf{s}}}_{j}\in\mathbb{R}^{D\times 1}$ is the decoder
hidden state at time $j$ and
${\boldsymbol{\mathbf{W_{o}}}}\in\mathbb{R}^{\lvert\mathcal{V}\rvert\times D}$
is the weight matrix of the softmax layer, which provides a continuous
representation for target words. ${\boldsymbol{\mathbf{s}}}_{j}$ is computed
using the following recurrence:

$\displaystyle{\boldsymbol{\mathbf{s}}}_{j}$
$\displaystyle=\text{tanh}({\boldsymbol{\mathbf{W_{c}}}}\;[{\boldsymbol{\mathbf{c}}}_{j};{\boldsymbol{\mathbf{\tilde{s}}}}_{j}])$
(3) $\displaystyle{\boldsymbol{\mathbf{\tilde{s}}}}_{j}$
$\displaystyle=f([{\boldsymbol{\mathbf{s}}}_{j-1};{\boldsymbol{\mathbf{w_{s}}}}^{y_{j-1}};{\boldsymbol{\mathbf{\tilde{s}}}}_{j-1}])$
(4)

where $f$ is an LSTM cell.222Note that our notation diverges from ?) so that
${\boldsymbol{\mathbf{s}}}_{j}$ refers to the state used to make the final
predictions.
${\boldsymbol{\mathbf{W_{s}}}}\in\mathbb{R}^{\lvert\mathcal{V}\rvert\times E}$
is the target-side embedding matrix, which provides continuous representations
for the previous target word when used as input to the RNN. Here,
${\boldsymbol{\mathbf{w_{s}}}}^{y_{j-1}}\in\mathbb{R}^{1\times E}$ is a row
vector from the embedding matrix ${\boldsymbol{\mathbf{W_{s}}}}$ corresponding
to the value of $y_{j-1}$. $\mathcal{V}$ is the target vocabulary set, $D$ is
the is the RNN size and $E$ is embedding size. Often these matrices
${\boldsymbol{\mathbf{W_{o}}}}$ and ${\boldsymbol{\mathbf{W_{s}}}}$ are tied.

The context vector ${\boldsymbol{\mathbf{c}}}_{j}$ is obtained by taking a
weighted average over the concatenation of a bidirectional RNN encoder’s
hidden states.

$\displaystyle{\boldsymbol{\mathbf{c}}}_{j}$
$\displaystyle=\sum_{i=1}^{{I}}\alpha_{i},{\boldsymbol{\mathbf{h}}}_{i}$ (5)
$\displaystyle\alpha_{i}$
$\displaystyle=\frac{\exp{({\boldsymbol{\mathbf{s}}}_{j}^{T}{\boldsymbol{\mathbf{W_{a}}}}{\boldsymbol{\mathbf{h}}}_{i})}}{\sum_{l}\exp{({\boldsymbol{\mathbf{s}}}_{j}^{T}{\boldsymbol{\mathbf{W_{a}}}}{\boldsymbol{\mathbf{h}}}_{l}})}$
(6)

The attention matrix ${\boldsymbol{\mathbf{W_{a}}}}\in\mathbb{R}^{D\times H}$
is learned jointly with the model, multiplying with the previous decoder state
and bidirectional encoder state
${\boldsymbol{\mathbf{h}}}_{i}\in\mathbb{R}^{H\times 1}$, normalized over
encoder hidden states via the softmax operation.

##  4 Character-Aware Extension

In this section we detail the incorporation of character-awareness into the
two decoder embedding matrices ${\boldsymbol{\mathbf{W_{o}}}}$ and
${\boldsymbol{\mathbf{W_{s}}}}$. To begin, we consider an example target side
word (or subword in the case of preprocessing with BPE), cat. In both
${\boldsymbol{\mathbf{W_{o}}}}$ and ${\boldsymbol{\mathbf{W_{s}}}}$, there
exist row vectors, ${\boldsymbol{\mathbf{w_{o}}}}^{\texttt{cat}}$ and
${\boldsymbol{\mathbf{w_{s}}}}^{\texttt{cat}}$ that contain the continuous
vector representation for the word cat. In a traditional NMT system, these
vectors are learned as the entire network tries to maximize the objective in
Equation 1. The objective does not require the vectors
${\boldsymbol{\mathbf{w_{o}}}}^{\texttt{cat}}$ and
${\boldsymbol{\mathbf{w_{s}}}}^{\texttt{cat}}$ to model any aspect of the
spelling of the word. Figure 1a illustrates a simple non-compositional word
embedding.

At a high level, we can view our notion of character-awareness as a
composition function $\text{comp}(.;{\boldsymbol{\mathbf{\omega}}})$,
parameterized by ${\boldsymbol{\mathbf{\omega}}}$, that takes the character
sequence that makes up a word (i.e. its spelling) as input and then produces a
continuous vector representation:

$\displaystyle{\boldsymbol{\mathbf{w}}}_{\text{comp}}^{\texttt{cat}}$
$\displaystyle=\text{comp}(\langle
s\rangle,\texttt{c},\texttt{a},\texttt{t},\langle/s\rangle;{\boldsymbol{\mathbf{\omega}}})$
(7)

${\boldsymbol{\mathbf{\omega}}}$ is learned jointly with the overall
objective. Special characters $\langle s\rangle$ and $\langle/s\rangle$ denote
the beginning and end of sequence respectively.

Figure 1b illustrates our compositional approach to generating embeddings [Kim
et al., 2016]. First, a character-embedding layer converts the spelling of a
word into a sequence of character embeddings. Next, we apply $4$ convolution
operations, with kernel sizes $3,4,5$ and $6$, over the character sequence and
the resulting output matrix is max-pooled. We set the output channel size of
each convolution to $\frac{1}{4}$ of the final desired embedding size. The
max-pooled vector from each convolution is concatenated to create the composed
word representation. Finally, we add highway layers to obtain the final
embeddings.

${\boldsymbol{\mathbf{w}}}^{v}_{\text{std}}$$v=\texttt{cat}$ (a)

ctahighway 1highway 2$\langle/s\rangle$$\langle
s\rangle$${\boldsymbol{\mathbf{w}}}^{v}_{\text{comp}}$$v=\texttt{cat}$ (b)

Figure 1: Different approaches to generating embeddings. (a) standard word
embedding that treats words as a single symbol. (b) CNN-based composition
function. We use multiple CNNs with different kernel sizes over the character
embeddings. The resulting hidden states are combined into a single word
embedding via max pooling. Note that (b) shows only $2$ convolution filters
for clarity, in practice we use 4.

###  4.1 Composed & Standard Gating

The composition is applied to every type in the vocabulary and thus generates
a complete embedding matrix (and softmax matrix). In doing so, we assume that
_every_ word in the vocabulary has a vector representation that can be
composed from its spelling sequence. This is a strong assumption as many
words, in particular high frequency words, are not normally compositional,
e.g. the substring ing in thing is not compositional in the way that it is in
running. Thus, we mix the compositional and standard embedding vectors. We
expect standard embeddings to better represent the meaning of certain words,
such has function words and other high-frequency words. For each word $v$ in
the vocabulary we also learn a gating vector
${\boldsymbol{\mathbf{g}}}^{v}\in[0,1]^{1\times D}$.

$\displaystyle{\boldsymbol{\mathbf{g}}}^{v}$
$\displaystyle=\sigma({\boldsymbol{\mathbf{w}}}_{\text{gate}}^{v})$ (8)

Where, $\sigma$ is a sigmoid operation and type-specific parameters
${\boldsymbol{\mathbf{w}}}_{\text{gate}}^{v}$ are jointly learned along with
all the other parameters of the composition function. These parameters are
regularized to remain close to ${\boldsymbol{\mathbf{0}}}$ using dropout.
333However, in practice we found that this regularization did not affect
performance noticeably in this setting. Our final mixed word representation
for each word $v\in\mathcal{V}$ is given by:

$\displaystyle{\boldsymbol{\mathbf{w}}}^{v}_{\text{mix}}={\boldsymbol{\mathbf{g}}}^{v}\odot{\boldsymbol{\mathbf{w}}}^{v}_{\text{std}}+({\boldsymbol{\mathbf{1.}}}-{\boldsymbol{\mathbf{g}}}^{v})\odot{\boldsymbol{\mathbf{w}}}^{v}_{\text{comp}}$
(9)

Where ${\boldsymbol{\mathbf{w}}}^{v}_{\text{mix}}$ is the final word
embedding, ${\boldsymbol{\mathbf{w}}}^{v}_{\text{std}}$ is the standard word
embedding, ${\boldsymbol{\mathbf{w}}}^{v}_{\text{comp}}$ is the embedding by
the composition function and ${\boldsymbol{\mathbf{g}}}^{v}$ is the type-
specific gating vector for the $v$’th word. The weight matrix is obtained by
stacking the word vectors for each word $v\in\mathcal{V}$. The same
representation is used for the target embedding layer and the softmax layer
i.e. we set
${\boldsymbol{\mathbf{w_{o}}}}^{\texttt{cat}}={\boldsymbol{\mathbf{w_{s}}}}^{\texttt{cat}}={\boldsymbol{\mathbf{w}}}_{\text{mix}}^{\texttt{cat}}$,
when $v=\texttt{cat}$. Thus, tying the composition function parameters for the
softmax weight matrix and the target-side embedding matrix.

Experiments comparing the standard embedding model and the compositional
embedding model with and without gating are summarized in Table 1. Row “C”
shows the performance of naively using the composition function (which works
in the source-side) on the target-side. We observe a catastrophic drop in BLEU
($-14.62$) compared to a standard NMT encoder-decoder. The Character-aware
gated model(CG), however, outperforms the baseline by $0.91$ BLEU points
suggesting that the CNN composition function and standard embeddings work in a
complementary fashion.

Composition Method | BLEU
---|---
Std.- (no composition) | 26.84
CGS (without gating) | 12.22
CG (target embedding only) | 26.61
CG (softmax embedding only) | 27.16
CG (both) | 27.75
Table 1: Experiments to determine the effectiveness of composition based
embeddings and gated embeddings. We used en-de language pair from the TED
multi-target dataset. Std. is our baseline with standard word embeddings,
model C is the composition only model and CG combines the character-aware
(composed) embedding and standard embedding via a gating function.

###  4.2 Large Vocabulary Approximation

In Equation 2 of the general NMT framework, the softmax operation generates a
distribution over the output vocabulary. Our character-aware model requires a
much larger computation graph as we apply convolutions (and highway layers)
over the spellings (character embeddings) of entire target vocabulary, placing
a limitation on the target vocabulary size for our model. Which is problematic
for word-level modeling (without BPE).

To make our character-aware model accommodate large target vocabulary sizes,
we incorporate an approximation mechanism based on [Jean et al., 2015].
Instead of computing the softmax over the entire vocabulary, we uniformly
sample $20$k vocabulary types and the vocabulary types that are present in the
training batch.

During decoding, we compute the forward pass
${\boldsymbol{\mathbf{W_{o}}}}{\boldsymbol{\mathbf{s}}}_{j}$ in Equation 2 in
several splits of the target vocabulary. As no backward pass is required we
clear the memory (i.e. delete the computation graph) after each split is
computed.

##  5 Experiments

Language | BPE Sweep | @ $30$k BPE | @ Word-level
---|---|---|---
|  Std(Best BPE) |  CG(Best BPE) | $\Delta$ | Std | CG | $\Delta$ | Std | CG | $\Delta$
cs | 20.57 (7.5k) | 21.41 (7.5k) | +0.84 | 18.73 | 21.28 | +2.55 | 18.44 | 21.49 | +3.05
uk | 15.79 (7.5k) | 16.60 (30k) | +0.81 | 14.27 | 16.60 | +2.33 | 12.94 | 15.30 | +2.36
pl | 16.76 (15k) | 18.00 (30k) | +1.24 | 15.98 | 18.00 | +2.02 | 15.49 | 17.20 | +1.71
tr | 15.11 (7.5k) | 15.83 (30k) | +0.72 | 13.82 | 15.83 | +2.01 | 12.58 | 14.75 | +2.17
hu | 16.61 (3.2k) | 17.23 (15k) | +0.62 | 15.45 | 17.21 | +1.76 | 14.18 | 16.52 | +2.34
he | 23.36 (3.2k) | 23.86 (30k) | +0.50 | 22.47 | 23.86 | +1.39 | 21.26 | 23.01 | +1.75
pt | 37.85 (15k) | 38.35 (30k) | +0.50 | 37.05 | 38.35 | +1.30 | 37.13 | 38.36 | +1.23
ar | 16.22 (7.5k) | 16.28 (30k) | +0.06 | 15.05 | 16.28 | +1.23 | 14.45 | 16.05 | +1.60
de | 27.37 (7.5k) | 28.12 (30k) | +0.75 | 26.94 | 28.12 | +1.21 | 26.84 | 27.75 | +0.91
ro | 24.02 (3.2k) | 24.20 (15k) | +0.18 | 22.88 | 24.00 | +1.12 | 22.39 | 23.27 | +0.88
bg | 31.63 (7.5k) | 32.20 (15k) | +0.57 | 30.92 | 31.90 | +0.98 | 30.18 | 31.43 | +1.25
fr | 35.97 (1.6k) | 36.17 (7.5k) | +0.20 | 35.31 | 35.92 | +0.61 | 35.28 | 36.01 | +0.73
fa | 12.94 (30k) | 13.52 (30k) | +0.58 | 12.94 | 13.52 | +0.58 | 12.85 | 12.79 | -0.06
ru | 19.28 (30k) | 19.61 (30k) | +0.33 | 19.28 | 19.61 | +0.33 | 17.60 | 19.04 | +1.44
Table 2: Best BLEU scores swept over $6$ different BPE merge setting ($1.6$k,
$3.2$k, $7.5$k, $15$k, $30$k, $60$k), and at a standard setting of $30$k. We
notice a consistent improvement across languages and settings of the merge
operation parameter.

We evaluate our character aware model on $14$ different languages in a low-
resource setting. Additionally, we sweep over several BPE merge hyperparameter
settings from character-level to fully word-level for both our model and the
baseline and find consistent gains in the character-aware model over the
baseline. These gains are stable across all BPE merge hyperparameters all the
way up to word-level where they are the highest.

###  5.1 Datasets

We use a collection of TED talk transcripts [Duh, 2018, Cettolo et al., 2012].
This dataset has languages with a variety of morphological typologies, which
allows us to observe how the success of our character-aware decoder relates to
morphological complexity. We keep the source language fixed as English and
translate into $14$ different languages, since our focus is on the decoder.
The training sets for each vary from 74k sentences pairs for Ukrainian to
around 174k sentences pairs for Russian (provided in Appendix A), but the
validation and test sets are “multi-way parallel”, meaning the English
sentences (the source side in our experiments) are the same across all $14$
languages, and are about $2$k sentences each. We filter out training pairs
where the source sentence was longer that $50$ tokens (before applying BPE).
For word-level results, we used a vocabulary size of $100$k (keeping the most
frequent types) and replaced rare words by an <UNK> token.

###  5.2 NMT Setup

We work with OpenNMT-py [Klein et al., 2017], and modify the target-side
embedding layer and softmax layer to use our proposed character-aware
composition function. A $2$ layer encoder and decoder, with $1000$ recurrent
units were used in all experiments The embeddings sizes were made to match the
RNN recurrent size. We set the character embedding size to $50$ and use four
CNNs with kernel widths $3,4,5$ and $6$. The four CNN outputs are concatenated
into a compositional embeddings and gated with a standard word embedding. The
same composition function (with shared parameters) was used for the target
embedding layer and the softmax layer.

We optimize the NMT objective (Equation 1) using SGD.444SGD outperformed both
Adam and Adadelta. Others have found similar trends, see ?) and ?). An initial
learning rate of 1.0 was used for the first $8$ epochs and then decayed with a
decay rate of $0.5$ until the learning rate reached a minimum threshold of
$0.001$. We use a batch size of 80 for our main experiments. At the end of
each epoch we checkpoint and evaluate our model on a validation datset and
used validation accuracy as our model selection criteria for test time. During
decoding, a beam size of $5$ was chosen for all the experiments.

###  5.3 Results

Lang |  Char- Shallow |  Char- Deep |  CG ($30$k BPE) | $\Delta$
---|---|---|---|---
uk |  14.77 | 13.34 | 16.60 | +3.26
cs | 11.16 | 18.45 | 21.28 | +2.83
de | 23.89 | 25.93 | 28.12 | +2.19
bg | 26.40 | 29.81 | 31.90 | +2.09
tr |  15.29 | 13.94 | 15.83 | +1.89
pl | 10.65 | 16.31 | 18.00 | +1.69
ru | 14.63 | 18.01 | 19.61 | +1.60
ro | 21.58 | 22.45 | 24.00 | +1.55
pt | 35.00 | 37.06 | 38.35 | +1.29
hu |  12.51 | 16.02 | 17.21 | +1.19
fr | 32.71 | 34.76 | 35.92 | +1.16
fa |  17.44 | 12.73 | 13.52 | +0.79
ar |  13.58 | 15.89 | 16.28 | +0.39
he | 22.28 | 23.87 | 23.86 | -0.01
Table 3: BLEU scores (lowercased) comparing character-level models against CG
when used on $30$k BPE sequences. We show that without sweeping BPE, CG
generally outperforms purely character-level methods, even when the purely
character-level networks are deepened as was shown to help in ?).

We provide case insensitive BLEU scores for our main experiments, comparing
our character-aware model (CG) against a baseline model that uses only
standard word (and subword) embeddings. We divide the results of our model’s
performance into three parts:  (i) over a sweep of BPE merge operations,
including a commonly used setting of $30$k merge operations  (ii) with word-
level source and target sequences and finally,  (iii) against a purely
character-level model.

####  5.3.1 BPE Results

Part $1$ of Table 2 compares the best BLEU score obtained by the baseline
model, after performing a BPE sweep from $1.6$k to $60$k, to the best BLEU
obtained by CG after sweeping over the same BPE range. While our study focuses
on the target side, BPE (with the same number of merge operations) was applied
to both source and target for our experiments. We find that after this sweep,
CG outperforms the baseline in all 14 languages. The exhaustive table of
results for these experiments is presented in Appendix A.

#### No Typical BPE Setting

Additionally, we see that the BPE setting that achieves best BLEU in the
baseline model varies considerably from $1.6$k to $30$k depending on the
target language, indicating that _there is no “typical” BPE for low-resource
settings_. In the CG model, however, performance was usually best at $30$k.
Part $2$ of Table 2 compares the baseline and CG at BPE of $30$k where CG
performs optimally.

We find that our CG model consistently outperforms the baseline for almost all
BPE merge hyperparameters across all 14 languages. Figure 2 shows the gains
observed by the CG model as we sweep over BPE merge operations. While the
baseline model does slightly better than CG at small BPE settings for a few
languages (all points below the $0$ value), a majority of the points show
positive gains.

####  5.3.2 Word-Level Results

In Part $3$ of Table 2 we show results with our approximation for word level.
While our best results are generally with BPE, we note that we get the biggest
relative gains using our method at the word level, which we expect is due to
always having the whole word to learn character patterns over. For the CG
model, in $60$k BPE and word-level settings we used the large vocabulary
approximation discussed in Section 4.2.

####  5.3.3 Character-Level Results

Finally, in Table 3, we compare two character-level models against our CG
model at $30$k BPE. The shallow character-level model used $2$ encoder and
decoder layers with $1000$ recurrent units, while the deep model used $6$
encoder and decoder layers with $512$ recurrent units .555Increasing the
recurrent size for deep models resulted in significant drop in BLEU scores. We
set the dropout rate to $0.1$. Furthermore, the improved results from the deep
model were only attainable using the Fairseq toolkit with Noam optimization
and $100$ warmup steps [Gehring et al., 2017]. As Table 3 shows, our CG model
with $30$k BPE compares favorably to even deep character-level models for this
low-resource setting.

1.62k3.2k7.5k15k30k60k∗W∗$0$$1$$2$$3$cscscscscscscsukukukukukukukhuhuhuhuhuhuhuplplplplplplplhehehehehehehetrtrtrtrtrtrtrarararararararptptptptptptptrororororororobgbgbgbgbgbgbgrururururururudededededededefafafafafafafafrfrfrfrfrfrfrBPE
Merge Operations$\Delta$ BLEU(CG - Std.)

Figure 2: Plot of the difference between the BLEU scores from CG model and
baseline model at various BPE settings for each of the $14$ languages (shown
in color, with language identifier). The bold black line shows the average
difference across the languages for each BPE setting.

##  6 Analysis

Features |  | Corpus-
---
dependent
| Corpus-
---
independent
TT | A | H | UT | UTC
Correlation | 0.04 | 0.59 | 0.67 | 0.80 | 0.49

Table 4: The Pearsons correlation between the features and the relative gain
in BLEU obtained by the CG model. See Section 6 for details regarding
features.

We are interested in understanding whether our character-aware model is
exploiting morphological patterns in the target language. We investigate this
by inspecting the relationship between a set of hand-picked features and
improvements obtained by our model over the baseline at word-level inputs.
These features fall into two categories, _corpus-dependent_ and _corpus-
independent_. We following ?), and extract features known to correlate with
human judgments of morphological complexity. The following corpus-dependent
features were used:

  1. (i)

Type-Token Ratio (TT): the ratio of the number of word types to the total
number of word tokens in the target side. We note that a large corpus tends to
have a smaller type-token ratio compared to small corpus.

  2. (ii)

Word-Alignment Score (A): computed as $A=\frac{\mid\text{many-to-
one}\mid-\mid\text{one-to-many}\mid}{\mid\text{all-alignments}\mid}$. One-to-
one, one-to-many and many-to-one alignment types are illustrated in Figure
3.666We use FastAlign [Dyer et al., 2013] for word alignments with the grow-
diag-final-and heuristic from [Och and Ney, 2003] for symmetrization. We
intuit that a morphologically poor source language (like English) paired with
a richer target language should exhibit more many-to-one alignments—a single
word in the target will contain more information (via morphological phenomena)
that can only be translated using multiple words in the source.

  3. (iii)

Word-Level Entropy (H): computed as $H=\sum_{v\in\mathcal{V}}p(v)\log p(v)$
where $v$ is a word type. This metric reflects the average information content
of the words in a corpus. Languages with more dependence on having a large
number of word types rather than word order or phrase structure will score
higher.

$s_{1}$$s_{0}$$s_{2}$$s_{3}$$s_{4}$$t_{1}$$t_{0}$$t_{2}$$t_{3}$ Figure 3:
Example of one-to-many ($s_{0}$ to $t_{0},t_{1}$), one-to-one ($s_{1}$ to
$t_{2}$) and many-to-one ($s_{2},s_{3},s_{4}$ to $t_{3}$) alignments. For this
example $A=(3-2)/6$.

For the corpus-independent features we used a morphological annotation corpus
called UniMorph [Sylak-Glassman et al., 2015]. The UniMorph corpus contains a
large list of inflected words (in several languages) along with the word’s
lemma and a set of morphological tags. For example, the French UniMorph corpus
contains the word marchai (walked), which is associated with its lemma,
marcher and a set of morphological tags $\\{$V,IND,PST,1,SG,PFV$\\}$. There
are $19$ such tags in the French UniMorph corpus. A morphologically richer
language like Hungarian, for example, has $36$ distinct tags. We used the
number of distinct tags (UT) and the number of different tag combinations
(UTC) that appear in the UniMorph corpus for each language. Note that we do
not filter out words (and its associated tags) from the UniMorph corpus that
are absent in our parallel data. This ensures that the UT and UTC features are
completely corpus independent.

The Pearson’s correlation between these hand-picked features and relative gain
observed by our model is shown in Table 4. For this analysis we used the
relative gain obtained from the word-level experiments. Concretely, the
relative gain for Czech was computed as $\frac{21.49-18.44}{18.44}$ We see a
strong correlation between the corpus-independent feature (UT) and our model’s
gain. Alignment score and Word Entropy are also moderately correlated.
Surprisingly, we see no correlation to type-token ratio.

As the correlation analysis only examines the relation between BLEU gains and
an _individual_ feature, we further analyzed how the features _jointly_ relate
to BLEU gains. We fitted a linear regression model, setting the relative gains
as the predicted variable $y$ and the feature values as the input variables
${\boldsymbol{\mathbf{x}}}$, with the goal of studying the linear regression
weights ${\boldsymbol{\mathbf{\phi}}}$.777The input features were min-max
normalized for the regression analysis. We used feature-augmented domain
adaptation where we consider each language as a domain [Daumé III, 2007],
allowing the model to find a set of “general” weights as well language-
specific weights that best fit the data (Equation 11). The general feature
weights can be interpreted as being indicative of the overall trends in the
dataset across all the languages, while the language-specific weights indicate
language deviation from the overall trend.

$\displaystyle{\boldsymbol{\mathbf{\mathcal{L}}}}({\boldsymbol{\mathbf{\phi}}})$
$\displaystyle=\sum_{i\in\mathcal{I}}\mid
y_{i}-\tilde{y}_{i}\mid^{2}-\lambda\mid{\boldsymbol{\mathbf{\phi}}}\mid^{2}$
(10) $\displaystyle\tilde{y}_{i}$
$\displaystyle={\boldsymbol{\mathbf{\phi}}}_{\text{ALL}}^{T}{\boldsymbol{\mathbf{x}}}_{i}+{\boldsymbol{\mathbf{\phi}}}_{i}^{T}{\boldsymbol{\mathbf{x}}}_{i}$
(11)

Where, $y$ is the true relative gain in BLEU, $\tilde{y}$ is the predicted
gain, ${\boldsymbol{\mathbf{x}}}$ is a vector of input feature values,
${\boldsymbol{\mathbf{\phi}}}_{\text{ALL}}$ and
${\boldsymbol{\mathbf{\phi}}}_{i}$ are the general and language-specific
weights, and $i$ indexes into the set of languages in our analysis. We set
$\lambda$ to $0.05$.

Figure 4: Feature weights of the feature-augmented language adapted linear
regression model. The first row represents the “general” set of weights used
for all of the languages. Each row below are the language-adapted weights that
only “fire” for that specific language.

The matrix of learned weights ${\boldsymbol{\mathbf{\phi}}}$ is visualized in
Figure 4. The first row of weights correspond to the “general” weights that
are used for all the languages, followed by language-specific weights sorted
by relative gain.

While the general weights align with the correlation results (Table 4), this
analysis also shows that the UTC weight for Czech and Turkish are much larger
than any of the other languages’ and indeed we can verify that these languages
have $194$ and $300$ different tag combinations while the average tag
combinations is $\approx 110$.

From the corpus-dependent features, word alignment score strongly predicts the
gain in BLEU scores. For Czech, Ukrainian, Turkish, Hungarian, and Polish we
see additional weight placed on this feature. A similar trend can be seen for
the word-entropy feature. While type-token ratio does not exhibit a strong
overall trend, we see that Ukrainian and Farsi are outliers.

Our correlation and regression analysis strongly suggest that CG character-
aware modeling helps the most when the target language has inherent
morphological complexity and that it does indeed have the ability to handle
morphological patterns present in the target languages.

###  6.1 Qualitative Examples

Src | here he is : leonardo da vinci .
---|---
Ref | h*A hw – lywnArdw dA fyn$y .
Std | hnA hw : lywnArdw dA dA .
CG | hnA hw : lywnArdw dA fy+n$y .
Src | i ’m the mexican in the family .
Ref | AnA Almksyky fy AlEA}lp .
Std | AnA mksy+Any fy AlEA}lp .
CG | AnA Almksy+ky fy AlEA}lp .
Src | there was going to be a national referendum .
Ref | wtm AlAEdAd lAHrA’ AstftA’ $Eby .
Std | sykwn hnAk f+tA’ wTny .
CG | sykwn hnAk Ast+f+tA’ wTny .
Src | there are ordinary heroes .
Ref | fhnAk AbTAl TbyEywn .
Std | hnAk ASdqA’ EAdy .
CG | hnAk AbTAl EAdyyn .
Table 5: Examples from En-Ar, transliterated with the Buckwalter schema. We
show the version of our model and the English using ‘+’ to denote where BPE
splits words up, while BPE has not been applied to the target reference.
author=adi,color=green,size=,fancyline,caption=,inline,]room for one or two
more example?

We additionally look at specific examples of where our model is outperforming
the baseline in the case of $30$k BPE in En-Ar. We see a few trends, which we
show examples of in Table 5. The first trend, corresponding to the first
example, is that it gets names better. This might be because Arabic is not
written in the Latin alphabet, and the spelling-aware model may be able to
transliterate better.

Another trend is that CG gets the endings of rare words correct, in particular
when the BPE segmentation is _not_ according to morpheme boundaries. The
second example illustrates this, where the word for “Mexican” appears in the
training data broken up by BPE with various morphological endings, all of
which are spelled beginning with “ky” in the second subword. The morpheme
boundaries here would be “Al+mksyk+y.” Note that CG also gets the definite
article “Al” correct while the baseline does not.

Finally, we see a pattern where our model does better for words which are rare
and appear both with and without the definite article “Al.” Our third example
in Table 5 illustrates this with an infrequent word, the word for
“referendum”, which gets broken up into subwords. In particular, the first
subword sometimes has an “Al” attached in the training data. Our model is able
to translate this subword, while the baseline skips the subword altogether,
outputting two subwords that alone are not a valid word. Again, the word is
not broken up along morpheme boundaries by BPE. Here there would be no way to
break this word up into morphological segments—it consists of non-
concatenative derivational morphology. This occurs again in the fourth example
in the word for “heroes,” where the baseline predicts the word for “friends.”
In this case the word was not split up by BPE, but similarly it is rare but
occurs with the definite article attached in the training data as well.

##  7 Conclusion

We extend character-aware word-level modeling to the decoder for translation
into morphologically rich languages. Our improvements were attained by
augmenting the softmax and the target embedding layers with character-
awareness. We also find it critical to add a gating function to balance
compositional embeddings with standard embeddings. We evaluate our method on a
low-resource dataset translating from English into 14 languages, and on top of
a spectrum of BPE merge operations. Furthermore, for word-level and higher
merge hyperparameter settings, we introduced an approximation to the softmax
layer. We achieve consistent performance gains across languages and subword
granularities, and perform an analysis indicating that the gains for each
language correspond to morphological complexity.

For future work, we would like to explore how our methods might be of use in
higher-resource settings. Furthermore, it would be interesting to see how
these methods might interact with multilingual systems and if they might be
able to improve what information is shared between related languages.
author=pamela,color=cyan,size=,fancyline,caption=,inline,]I added a sentence
to the conclusion to help us hit 8 pages and to end on a bit more of a
positive note (and not the one the sassy reviewer commented on)

## Acknowledgements

This project originated at the Machine Translation Marathon 2018. We thank the
organizers and attendees for their support, feedback and helpful discussions
during the event. This work is supported in part by the Office of the Director
of National Intelligence, IARPA. The views contained herein are those of the
authors and do not necessarily reflect the position of the sponsors.

## References

  * [Bahar et al., 2017]  Bahar, Parnia, Tamer Alkhouli, Jan-Thorsten Peter, Christopher Jan-Steffen Brix, and Hermann Ney.  2017\.  Empirical investigation of optimization algorithms in neural machine translation.  The Prague Bulletin of Mathematical Linguistics, 108(1):13–25.
  * [Bahdanau et al., 2015]  Bahdanau, Dzmitry, Kyunghyun Cho, and Yoshua Bengio.  2015\.  Neural machine translation by jointly learning to align and translate.  International Conference on Learning Representations.
  * [Belinkov et al., 2017]  Belinkov, Yonatan, Nadir Durrani, Fahim Dalvi, Hassan Sajjad, and James Glass.  2017\.  What do neural machine translation models learn about morphology?  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 861–872. Association for Computational Linguistics.
  * [Bentz et al., 2016]  Bentz, Christian, Tatyana Ruzsics, Alexander Koplenig, and Tanja Samardzic.  2016\.  A comparison between morphological complexity measures: typological data vs. language corpora.  In Proceedings of the Workshop on Computational Linguistics for Linguistic Complexity (CL4LC), pages 142–153.
  * [Cettolo et al., 2012]  Cettolo, Mauro, Christian Girardi, and Marcello Federico.  2012\.  Wit3: Web inventory of transcribed and translated talks.  In Conference of European Association for Machine Translation, pages 261–268.
  * [Cherry et al., 2018]  Cherry, Colin, George Foster, Ankur Bapna, Orhan Firat, and Wolfgang Macherey.  2018\.  Revisiting character-based neural machine translation with capacity and compression.  In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4295–4305.
  * [Chung et al., 2016]  Chung, Junyoung, Kyunghyun Cho, and Yoshua Bengio.  2016\.  A character-level decoder without explicit segmentation for neural machine translation.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1693–1703.
  * [Costa-jussà and Fonollosa, 2016]  Costa-jussà, Marta R and José AR Fonollosa.  2016\.  Character-based neural machine translation.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 357–361.
  * [Daumé III, 2007]  Daumé III, Hal.  2007\.  Frustratingly easy domain adaptation.  In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 256–263, June.
  * [Duh, 2018]  Duh, Kevin.  2018\.  The multitarget ted talks task.  http://www.cs.jhu.edu/~kevinduh/a/multitarget-tedtalks/.
  * [Dyer et al., 2013]  Dyer, Chris, Victor Chahuneau, and Noah A Smith.  2013\.  A simple, fast, and effective reparameterization of ibm model 2.  In Proceedings of the 2013 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 644–648.
  * [Gage, 1994]  Gage, Philip.  1994\.  A new algorithm for data compression.  C Users J., 12(2):23–38, February.
  * [Gehring et al., 2017]  Gehring, Jonas, Michael Auli, David Grangier, Denis Yarats, and Yann N Dauphin.  2017\.  Convolutional Sequence to Sequence Learning.  ArXiv e-prints, May.
  * [Jean et al., 2015]  Jean, Sébastien, Kyunghyun Cho, Roland Memisevic, and Yoshua Bengio.  2015\.  On using very large target vocabulary for neural machine translation.  In Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1–10.
  * [Kim et al., 2016]  Kim, Yoon, Yacine Jernite, David Sontag, and Alexander M Rush.  2016\.  Character-aware neural language models.  In 30th AAAI Conference on Artificial Intelligence, AAAI 2016.
  * [Klein et al., 2017]  Klein, Guillaume, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander M. Rush.  2017\.  Opennmt: Open-source toolkit for neural machine translation.  In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics, ACL 2017, Vancouver, Canada, July 30 - August 4, System Demonstrations, pages 67–72.
  * [Ling et al., 2015]  Ling, Wang, Isabel Trancoso, Chris Dyer, and Alan W Black.  2015\.  Character-based neural machine translation.  arXiv preprint arXiv:1511.04586.
  * [Luong et al., 2015]  Luong, Thang, Hieu Pham, and Christopher D. Manning.  2015\.  Effective approaches to attention-based neural machine translation.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 1412–1421.
  * [Maruf and Haffari, 2018]  Maruf, Sameen and Gholamreza Haffari.  2018\.  Document context neural machine translation with memory networks.  In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), volume 1, pages 1275–1284.
  * [Miyamoto and Cho, 2016]  Miyamoto, Yasumasa and Kyunghyun Cho.  2016\.  Gated word-character recurrent language model.  In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1992–1997.
  * [Och and Ney, 2003]  Och, Franz Josef and Hermann Ney.  2003\.  A systematic comparison of various statistical alignment models.  Computational linguistics, 29(1):19–51.
  * [Passban et al., 2018]  Passban, Peyman, Qun Liu, and Andy Way.  2018\.  Improving character-based decoding using target-side morphological information for neural machine translation.  In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), volume 1, pages 58–68.
  * [Sennrich et al., 2016]  Sennrich, Rico, Barry Haddow, and Alexandra Birch.  2016\.  Neural machine translation of rare words with subword units.  In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers.
  * [Shapiro and Duh, 2018]  Shapiro, Pamela and Kevin Duh.  2018\.  Bpe and charcnns for translation of morphology: A cross-lingual comparison and analysis.  arXiv preprint arXiv:1809.01301.
  * [Sylak-Glassman et al., 2015]  Sylak-Glassman, John, Christo Kirov, Matt Post, Roger Que, and David Yarowsky.  2015\.  A universal feature schema for rich morphological annotation and fine-grained cross-lingual part-of-speech tagging.  In International Workshop on Systems and Frameworks for Computational Morphology, pages 72–93. Springer.

##  Appendix A More Detailed Results

In Table 6, we provide the number of training sentences for each language.

In Table 7, we provide the full experiments of our sweep of BPE for both
standard and our CG embeddings. In our baseline, we see a divergence in trends
across languages while sweeping over BPE merge hyperparameters—Czech (cs),
Turkish (tr), and Ukrainian (uk) for example, are highly sensitive to the BPE
merge hyperparameter. On the other hand, for languages like French (fr) and
Farsi (fa), the performance is mostly consistent across different BPE merge
hyperparameters.

author=pamela,color=cyan,size=,fancyline,caption=,inline,]Do we need to submit
this separately?

Language | Number of sentences
---|---
Czech (cs) | 81k
Ukrainian (uk) | 74k
Hungarian (hu) | 108k
Polish (pl) | 149k
Hebrew (he) | 181k
Turkish (tr) | 137k
Arabic (ar) | 168k
Portuguese (pt) | 147k
Romanian (ro) | 155k
Bulgarian (bg) | 159k
Russian (ru) | 174k
German (de) | 146k
Farsi (fa) | 106k
French (fr) | 149k
Table 6: Number of sentences in training data for each language L | M |  Char- Shallow |  Char- Deep | BPE (Subwords) |  Word- Level
---|---|---|---|---|---
1.6k | 3.2k | 7.5k | 15k | 30k | 60k
cs | Std. | 11.16 | 18.45 | 20.28 | 20.51 | 20.57 | 19.60 | 18.73 | 17.60 | 18.44
CG | - | - | 20.71 | 21.04 | 21.41 | 21.14 | 21.28 | 20.97 | 21.49
uk | Std. | 4.77 | - | 13.35 | 15.51 | 15.79 | 15.36 | 14.27 | 12.50 | 12.94
CG | - | - | 13.80 | 16.16 | 15.48 | 16.28 | 16.60 | 15.54 | 15.30
hu | Std. | 2.51 | 16.02 | 15.77 | 16.33 | 15.62 | 16.61 | 15.45 | 14.81 | 14.18
CG | - | - | 16.58 | 16.61 | 16.88 | 17.23 | 17.21 | 17.05 | 16.52
pl | Std. | 10.65 | 16.31 | 16.14 | 16.40 | 16.34 | 16.76 | 15.98 | 15.47 | 15.49
CG | - | - | 16.88 | 17.12 | 16.84 | 17.63 | 18.00 | 17.32 | 17.20
he | Std. | 22.28 | 23.87 | 23.07 | 23.36 | 23.32 | 22.76 | 22.47 | 21.84 | 21.26
CG | - | - | 23.52 | 23.38 | 23.65 | 23.33 | 23.86 | 22.78 | 23.01
tr | Std. | 5.29 | 13.94 | 14.92 | 14.58 | 15.11 | 14.75 | 13.82 | 13.69 | 12.58
CG | - | - | 14.42 | 15.25 | 15.51 | 15.54 | 15.83 | 15.05 | 14.75
ar | Std. | 3.58 | 15.89 | 15.66 | 15.67 | 16.22 | 15.70 | 15.05 | 14.86 | 14.45
CG | - | - | 15.96 | 15.55 | 16.17 | 15.99 | 16.28 | 15.53 | 16.05
pt | Std. | 35.00 | 37.06 | 37.47 | 37.53 | 37.61 | 37.85 | 37.05 | 37.11 | 37.13
CG | - | - | 37.94 | 37.98 | 37.77 | 38.28 | 38.35 | 38.11 | 38.36
ro | Std. | 21.58 | 22.45 | 23.48 | 24.02 | 23.72 | 23.78 | 22.88 | 22.73 | 22.39
CG | - | - | 23.55 | 23.42 | 23.61 | 24.20 | 24.00 | 23.38 | 23.27
bg | Std. | 26.40 | 29.81 | 31.17 | 31.41 | 31.63 | 31.09 | 30.92 | 30.44 | 30.18
CG | - | - | 31.43 | 31.71 | 31.81 | 32.20 | 31.90 | 31.58 | 31.43
ru | Std. | 14.63 | - | 18.17 | 18.71 | 19.05 | 18.80 | 19.28 | 18.28 | 17.60
CG | - | - | 18.68 | 19.26 | 19.40 | 19.30 | 19.61 | 19.23 | 19.04
de | Std. | 23.89 | 25.93 | 26.98 | 27.34 | 27.37 | 27.23 | 26.94 | 27.21 | 26.84
CG | - | - | 26.94 | 27.55 | 27.46 | 27.89 | 28.12 | 27.37 | 27.75
fa | Std. | 7.44 | 12.73 | 12.87 | 12.71 | 12.86 | 12.94 | 12.94 | 13.20 | 12.85
CG | - | - | 12.35 | 12.98 | 13.38 | 13.36 | 13.52 | 13.31 | 12.79
fr | Std. | 32.71 | 34.76 | 35.97 | 35.75 | 35.82 | 35.90 | 35.31 | 35.33 | 35.28
CG | - | - | 35.89 | 35.68 | 36.17 | 36.10 | 35.92 | 36.08 | 36.01
Table 7: BLEU scores (case insensitive) for a standard embedding encoder-
decoder baseline (Std), and character-aware model, composed embedding combined
with standard embedding (CG) for $14$ languages and various BPE merge
hyperparameters. For purely character-level we only train the standard model
as CG would not have a sequence of characters to compose. For BPE of $60$k and
word-level we use the softmax approximation described. We see that CG obtains
the best result in all languages.