# Effects of surface potentials on Goos-Hänchen and Imbert-Fedorov shifts in
Weyl semimetals

Ninad Kailas Dongre  Department of Physics, Indian Institute of Technology
(BHU), Varanasi 221005, India  Department of Physics, The University of Texas
at Dallas, Richardson, TX 75080, USA  Krishanu Roychowdhury  Department of
Physics, Stockholm University, SE-106 91 Stockholm, Sweden

###### Abstract

Weyl semimetals exhibit exotic transport responses, among which, recently
Goos-Hänchen (GH) and Imbert-Fedorov (IF) effects have received a revived
attention, which are, otherwise, well-studied phenomena in optical systems and
certain electronic systems. Besides the usual parametric dependence of the
shifts inherited from the underlying Hamiltonian to describe the Weyl system
and/or that induced by external controls, the IF shift further carries a
topological identity – it depends on the chirality of the Weyl cones. A
realistic system of Weyl semimetal naturally accommodates surface potentials
induced by impurities present on its surface that could pose impediments to
observe clean transport signatures predicted in theoretical models.
Classifying these potentials, we study their effects on GH and IF shifts to
provide useful guidance to future experiments that are tuned to the objective
of characterizing Weyl semimetals and for a possible realisation of novel
devices based on these phenomena. A transfer matrix-based approach is invoked
to study the profile of Weyl wavefunctions across the interface which is
hosting the impurity potentials, revealing that such potentials can lead to
several discerning effects which, in certain cases, extend even to nullifying
the IF shift completely and giving rise to phenomenon like valley inversion.

##  I Introduction

The Goos-Hänchen Goos and H ”a nchen (1947); Artmann (1948); McGuirk and
Carniglia (1977) (GH) and Imbert-Fedorov Fedorov (1955); Schilling (1965);
Imbert (1972) (IF) shifts, first discovered in the context of wave-optics,
have now found application in a variety of systems Wild and Giles (1982);
Bretenaker et al. (1992); Pfleghaar et al. (1993); Emile et al. (1995); Bonnet
et al. (2001); Berman (2002); Li (2003); Fan et al. (2003); Shadrivov et al.
(2003); Felbacq et al. (2003); Yin et al. (2004); Merano et al. (2007);
Beenakker et al. (2009); de Haan et al. (2010); Nieminen et al. (2020) that
include optical waveguides, metamaterials, plasmonics, and quantum systems.
These effects transpire for incident beams of finite width (i.e., a
distributed spectrum of plane waves) whose reflection and refraction do not
quite follow the simple geometric rules of Newtonian optics feature lateral
and angular shifts with respect to the point of incidence instead. For
instance, when a traveling photon beam suffers multiple total internal
reflections inside an optical waveguide, measurable shifts are observed that
may be attributed to a finite penetration of the evanescent beam into the
cladding material at each turn. This is reminiscent of the tunneling of
quantum particles through finite barriers which is accredited to their dual
(wave-particle) nature and has inspired researchers to explore these
scattering effects in quantum systems such as two-dimensional electron gas
(2DEG) nanostructures Chen et al. (2008) (for relativistic corrections, see
Ref. Miller Jr and Ashby, 1972; Fradkin and Kashuba, 1974) and Dirac materials
like graphene Beenakker et al. (2009); Chen et al. (2011); Sharma and Ghosh
(2011); Zhai et al. (2011); Cheng (2012); Wang et al. (2013a); Chen et al.
(2013); Wang et al. (2013b); Grosche et al. (2015); Zeng et al. (2017); Wu et
al. (2018) and transition metal dichalcogenides Das and Pradhan (2021),
enabling new device applications that extend, as well, to terahertz regime Fan
et al. (2016); Zheng et al. (2019); Liu et al. (2020a). The GH shift has been
exploited in a system of 2DEG subject to a tunable electric potential and
hosted between magnetic stripes to construct an efficient spin beam splitter
Chen et al. (2008); Zhang and Chan (2014). While spin manipulation of such
type is highly desirable for spintronics and quantum information applications,
the valley degrees of freedom in electronic systems (such as semiconductors
and semimetals) are also of concurrent and intensive interest Zhai et al.
(2011); Wu et al. (2011), especially for emerging quantum technologies like
valleytronics.

On both fronts, Dirac materials have enticed specific attention over the last
few decades. Graphene (along with other materials of similar band structures),
among them, has emerged as a paradigmatic model wherein a myriad of exotic
electronic phenomena that arise in two dimensions have been proposed and
verified in experiments (see Ref. Peres, 2010 for a review). The GH shift for
the massless electrons in graphene manifests as a pseudospin-dependent
scattering effect that results in a quantized jump in the conductance of
heterojunctions Beenakker et al. (2009). In the same system, a valley-
dependent GH shift is studied by means of tailoring the local strain profile
Wu et al. (2011) exhibiting a close resemblance to the spin manipulation in
Ref. Chen et al., 2008 and also, in part, the aforementioned optical and
electronic analogs.

The IF effect, likewise, has also been explored in graphene systems Kort-Kamp
et al. (2016); Farmani et al. (2017); Xu et al. (2017); Luo et al. (2018); Zhu
(2019), and recently resuscitated in Weyl semimetals Jiang et al. (2015,
2016); Wang and Jian (2017); Hao et al. (2019); Ye et al. (2019);
Chattopadhyay et al. (2019); Liu et al. (2020b). The latter is characterized
with topologically robust nodes (immune to arbitrary perturbations) in the
bulk and Fermi arc surface states Hosur and Qi (2013). The nodes are of
distinct chirality, referred to as Weyl cones (singly degenerate, as opposed
to the doubly degenerate Dirac cones in, e.g., graphene), and responsible for
strange phenomena like chiral anomaly, exclusive in three dimensions. Aside
from the bulk nodes, the surface states are also a distinctive hallmark of
these semimetals which have been experimentally probed and bear important
implications for various transport properties of these systems such as TaAs or
NbAs (see Ref. Hasan et al., 2017 and references therein).

In realistic materials, surfaces of a Weyl semimetal would naturally host
impurities that give rise to different types of surface potentials, akin to a
common scenario in topological insulators Zhang et al. (2012); Roy and Das
(2016); Roy et al. (2016). These potentials could break time-reversal or
inversion (or none), which is anyway required to stabilize a Weyl semimetallic
phase in the bulk. For instance, magnetic impurities can populate the surface
of a magnetic Weyl semimetal such as Co3Sn2S2 Liu et al. (2019) while spin-
orbit type impurities can be found in inversion breaking Weyl semimetals such
as candidates from the TaAs material class, Huang et al. (2015). Apart from
bulk impurities, these potentials are also of great relevance to experiments
dedicated to probing surface transport (such as conductivities) in these
topological materials and also in part attesting the theoretical predictions
based on clean surfaces such as those relying on the bulk-boundary
correspondence.

Among many intriguing consequences, localized bulk impurities in Weyl
semimetals can lead to suppression in the nodal density of states which may or
may not be lifted by impurity-induced resonances Huang et al. (2013). Besides,
these impurities also exhibit prominent features for the surface states such
as giving rise to bound states for certain parameter values in the Hamiltonian
that lead to distinct topological phases He (2018). As the lateral shifts
concerned are exclusively surface phenomena and that such shifts, particularly
for a Weyl semimetal surface, are intricately related to the Fermi arc
structure Chattopadhyay et al. (2019), we naturally ask to what extent are
they influenced by the presence of impurity-induced surface potentials which
are compatible with the symmetry breaking in the bulk of a Weyl system. The
central result of this article engages a thorough analysis of GH and IF shifts
in presence of such imperfections, which have hitherto been studied only for
clean surfaces.

The remainder of the article is structured as follows- In Sec. II, we
introduce the model and review some aspects of the GH and the IF shift for a
clean surface. In Sec. III, we discuss the transfer matrix based approach that
applies to analyze the Weyl wavefunction across an interface hosting impurity-
induced surface potentials. These potentials are classified in Sec. IV where
the shift calculations are revisited for each of the classes. We summarize the
findings in Sec. V and discuss future work.

##  II GH and IF shifts for a clean interface

Figure 1: The schematic of the Weyl semimetal surface hosting surface
potential at $x=0$ with the two lateral shifts indicated. The plane of
incidence is the $x-y$ plane such that in-plane lateral shift (GH shift) is
along the $y$-axis and the out-of-plane shift (IF shift) is along the
$z$-axis.

We start by reviewing the results from a previous work that computed the GH
and the IF shifts for a clean interface Jiang et al. (2015). For calculations,
we consider the surface (or interface) to be the $y-z$ plane at $x=0$ which
breaks the translation symmetry along $x$ but retains along the other two
directions (see Fig. 1), however, the formulation presented in this article,
as well, applies to surfaces of other orientations.

The Hamiltonian of the system is described by

$\displaystyle H=\begin{cases}H_{\rm I}=\chi\hbar v_{F}\leavevmode\nobreak\
k_{i}\sigma_{i}&x\leq 0\\\ H_{\rm II}=\chi\hbar v_{F}\leavevmode\nobreak\
k_{i}\sigma_{i}+V_{0}&x>0,\end{cases}$ (1)

where $i\in\\{x,y,z\\}$ and $\chi$ ($\chi=\pm 1$) denotes the chirality of the
Weyl cones. We refer to the region $x\leq 0$ as region I, and the region $x>0$
as region II as in Fig. 1. These two regions are distinguished by applying a
chemical potential difference $V_{0}$ between them which creates a finite
barrier for the Weyl fermions to scatter off. Further, $v_{F}$ denotes the
Fermi velocity, which is taken the same on both sides, and $\sigma_{i}$ are
the Pauli matrices representing the pseudospin degrees of freedom. Throughout
the calculations that follow, we adopt the units in which $\hbar=v_{F}=1$. A
beam of Weyl fermions, in the form of a Gaussian wavepacket, incident on the
interface at $x=0$ is modelled by

$\displaystyle\Psi^{\rm in}(\mathbf{r})=\int^{\infty}_{-\infty}{\rm
d}k_{y}{\rm d}k_{z}\leavevmode\nobreak\ f(k_{y},k_{z})\psi^{\rm
in}(\mathbf{k},\mathbf{r}),$ (2)

where the Gaussian function $f$ is assumed to be localized around
$(\bar{k}_{y},\bar{k}_{z})$ as

$\displaystyle f(k_{y},k_{z})=\frac{1}{2\pi\sqrt{\Delta_{y}\Delta_{z}}}{\rm
Exp}\bigg{[}{-\frac{(k_{y}-\bar{k}_{y})^{2}}{2\Delta_{y}^{2}}-\frac{(k_{z}-\bar{k}_{z})^{2}}{2\Delta_{z}^{2}}}\bigg{]},$
(3)

$\Delta_{y}$ ($\Delta_{z}$) denoting the width of the wavepacket along $y$
($z$). The spinor part of the incident wavefunction $\psi^{\rm in}$ satisfies
the Schrödinger equation $H_{\rm I}\psi^{\rm in}=E\psi^{\rm in}$, and
including the plane wave phase factor $e^{i{\bf k}\cdot{\bf r}}$,

$\displaystyle\psi^{\rm
in}(\mathbf{k},\mathbf{r})=C_{i}\begin{bmatrix}2\eta\chi e^{-i\theta/2}\\\
\tilde{\eta}e^{i\theta/2}\end{bmatrix}e^{i{\bf k}\cdot{\bf r}},$ (4)

where $k_{x}=\sqrt{E^{2}-k_{y}^{2}-k_{z}^{2}}$, $\tan\theta=k_{y}/k_{x}$,
$\eta=\sqrt{\frac{E-k_{z}}{E+k_{z}}}$,
$\tilde{\eta}=\eta^{2}(1+\chi)+(1-\chi)$, and $C_{i}$ is the normalization
constant. Taking the plane of incidence to be the $x-y$ plane, $\theta$
represents the angle of incidence for the incident beam measured from the
surface normal perpendicular to the $y-z$ plane at the point of incidence (see
Fig. 1).

The reflected wavepacket can be expressed in a similar form as the incident
one. Multiplied by the reflection coefficient $r=|r|e^{i\phi_{r}}$, it is

$\displaystyle\Psi^{\rm re}(\mathbf{r})=\int^{\infty}_{-\infty}{\rm
d}k_{y}{\rm d}k_{z}\leavevmode\nobreak\ f(k_{y},k_{z})\leavevmode\nobreak\
r\psi^{\rm re}(\mathbf{k},\mathbf{r}),$ (5)

where $\phi_{r}$ is the reflection phase and $\psi^{\rm
re}(\mathbf{k},\mathbf{r})$ is obtained from $\psi^{\rm
in}(\mathbf{k},\mathbf{r})$ in Eq. 4 via $k_{x}\rightarrow-
k_{x},\theta\rightarrow\pi-\theta$. For the phenomena concerned, we will be
considering total reflection from the interface in region I with $|r|=1$, and
for this, the mode on the other side (region II) must be evanescent. Note this
happens only for values of the incident angle $\theta$ greater than a critical
value $\theta_{c}$ which, as will be shown later, depends on the ratio of the
barrier height to the incident energy $V_{0}/E$.

Linearizing the phases $\theta$ and $\phi_{r}$ in terms of $\bar{k}_{y}$ and
$\bar{k}_{z}$, the integrals in Eq. 2 and 5 provide the centers of the
incident and the reflected wavepackets respectively from which the (spinor)
component-wise spatial shifts along $y$ and $z$ follow as

$\displaystyle\Delta^{y}_{\pm}$ $\displaystyle=-\frac{\partial}{\partial
k_{y}}\phi_{r}(\bar{k}_{y},\bar{k}_{z})\mp\frac{\partial}{\partial
k_{y}}\theta(\bar{k}_{y},\bar{k}_{z}),$ $\displaystyle\Delta^{z}_{\pm}$
$\displaystyle=-\frac{\partial}{\partial
k_{z}}\phi_{r}(\bar{k}_{y},\bar{k}_{z})\mp\frac{\partial}{\partial
k_{z}}\theta(\bar{k}_{y},\bar{k}_{z}),$ (6)

where, $\pm$ refer to the shifts of the two spinor components. The full
spatial shifts are then given as the weighted average of the individual shifts
for each of the spinor components,

$\displaystyle\Delta^{y(z)}=\frac{4\eta^{2}\Delta^{y(z)}_{+}+\tilde{\eta}^{2}\Delta^{y(z)}_{-}}{4\eta^{2}+\tilde{\eta}^{2}}.$
(7)

As the results do not depend on the shape of the wavepackets, we conveniently
adopt a reference by setting $\bar{k}_{z}=0$ which aligns the GH and the IF
shift along $y$ and $z$-axis respectively and, thus, simplifies Eq. 7 to

$\displaystyle\Delta_{\rm GH}$
$\displaystyle\equiv\Delta^{y}=\frac{4\eta^{2}\Delta^{y}_{+}+\tilde{\eta}^{2}\Delta^{y}_{-}}{4\eta^{2}\chi^{2}+\tilde{\eta}^{2}}=-\frac{\partial\phi_{r}}{\partial
k_{y}},$ $\displaystyle\Delta_{\rm IF}$
$\displaystyle\equiv\Delta^{z}=\frac{4\eta^{2}\Delta^{z}_{+}+\tilde{\eta}^{2}\Delta^{z}_{-}}{4\eta^{2}\chi^{2}+\tilde{\eta}^{2}}=-\frac{\partial\phi_{r}}{\partial
k_{z}}.$ (8)

To quantify the shifts, it then remains to compute the reflection phase
$\phi_{r}$ using the continuity of the wavefunctions at the interface

$\displaystyle\psi^{\rm tr}(\mathbf{k},0)=\psi^{\rm
in}(\mathbf{k},0)+\psi^{\rm re}(\mathbf{k},0)$ (9)

where the transmitted spinor $\psi^{\rm tr}(\mathbf{k},\mathbf{r})$ in region
II has an evanescent form at energy $E$ as

$\displaystyle\psi^{\rm tr}(\mathbf{k},\mathbf{r})=C_{t}\begin{bmatrix}-i\\\
\beta_{\chi}\end{bmatrix}e^{-\kappa
x}e^{i(k_{y}y+k_{z}z)};\leavevmode\nobreak\
\beta_{\chi}=\frac{k_{y}+\kappa}{k_{z}+\chi(E-V_{0})},$ (10)

$C_{t}$ being the normalization constant and
$\kappa=\sqrt{k_{z}^{2}+k_{y}^{2}-(E-V_{0})^{2}}>0$ setting the inverse decay
length. Eq. 9 yields a chirality-dependent reflection phase

$\displaystyle\phi_{r}=-\theta-\frac{\pi}{2}+2\tan^{-1}\Big{[}\frac{\tilde{\eta}\cos\theta}{2\eta\chi\beta_{\chi}-\tilde{\eta}\sin\theta}\Big{]},$
(11)

from which the spatial shifts follow as

$\displaystyle\Delta_{\rm
GH}=\frac{(1+\sin^{2}{\bar{\theta}}-\frac{V_{0}}{E})}{\kappa\sin{\bar{\theta}}\cos{\bar{\theta}}}\leavevmode\nobreak\
\leavevmode\nobreak\ ;\leavevmode\nobreak\ \leavevmode\nobreak\ \Delta_{\rm
IF}=-\frac{\chi}{E\tan{\bar{\theta}}},$ (12)

where $\tan{\bar{\theta}}={\bar{k}}_{y}/\bar{k}_{x}$. The critical angle to
ensure a total reflection from the interface is given by
$\theta_{c}=\sin^{-1}|V_{0}/E-1|$.

Note the GH shift on a clean surface is not a chirality dependent phenomenon
while the IF shift is and can be interpreted as a topological effect Jiang et
al. (2015). This is attributed to the fact that during the reflection, the
Weyl fermions retain their valley characteristics due to momentum conservation
on the surface which is in sharp distinction with the optical analog where the
polarization of the photons does get altered during reflection. We further
note that the IF shift is independent of the ratio of the barrier height to
the incident energy $V_{0}/E$. The GH shift, on the other hand, changes sign
at a given angle of incidence $\theta^{\ast}=\sin^{-1}\sqrt{\sin\theta_{c}}$ –
it is negative for $\theta_{c}<\bar{\theta}<\theta^{\ast}$ while positive for
$\theta^{\ast}<\bar{\theta}$ irrespective of the values of $V_{0}/E$, and
likewise for $\bar{\theta}\rightarrow-\bar{\theta}$.

So far, we have discussed the shifts for a clean interface. In the following
sections, we will demonstrate how they get modified when the same interface
harbors different types of impurity-induced surface potentials that affect the
continuity equation stated above and the resultant reflection phase. In fact,
in the presence of certain types of surface potentials, the shifts feature
distinct asymmetry between the valleys resembling the effects of intervalley
scattering Wang and Jian (2017).

##  III The transfer matrix approach

For Weyl (or Dirac) fermions subject to a delta potential scattering, the
transfer matrix accounts for the rotation between the spinors on the two sides
of the potential Calkin et al. (1987). This approach is adopted extensively in
transport calculations to compute observables like surface conductance,
Aharonov-Bohm oscillations, spin Berry phase in various mesoscopic systems
including topological insulators with impurities Ilan et al. (2012); Xypakis
et al. (2020); Adak et al. (2020).

The spatial profile of the full Hamiltonian in presence of a delta function-
type surface potential $V_{S}(x)={\cal V}\delta(x)$ is given by

$\displaystyle{\cal
H}=\chi(\sigma_{a}k_{a}-i\sigma_{x}\partial_{x})+V_{0}\Theta(x)+{\cal
V}\delta(x),$ (13)

where $a\in\\{y,z\\}$ as $k_{y,z}$ are regarded good quantum numbers for we
have imposed periodic boundary conditions on the interface; ${\cal V}$ denotes
the surface potential [see Fig. 1] which could be a scalar or a spin-valued
operator. The Schrödinger equation ${\cal H}\psi=E\psi$ can be recast as
$\partial_{x}\psi={\cal H}_{0}(x)\psi$ leading to a path-ordered solution of
$\psi(x)$ as $\psi(x_{2})={\cal T}_{x_{2},x_{1}}\psi(x_{1})$ where the
transfer matrix ${\cal T}={\cal P}_{x}{\rm Exp}\big{[}\int_{x_{1}}^{x_{2}}{\rm
d}x\leavevmode\nobreak\ {\cal H}_{0}(x)\big{]}$, ${\cal P}_{x}$ denoting the
path-ordering Adak et al. (2020). For the interface at $x=0$ (see Fig. 1), the
transfer matrix pertinent to the model in Eq. 13 turns out to be

$\displaystyle{\cal T}=\lim_{\epsilon\to 0}{\cal P}_{x}\leavevmode\nobreak\
{\rm Exp}\bigg{[}-i\int_{-\epsilon}^{+\epsilon}{\rm d}x\leavevmode\nobreak\
\sigma_{x}V_{S}(x)\bigg{]}=e^{-i\sigma_{x}{\cal V}}.$ (14)

The continuity equation in Eq. 9 gets modified to

$\displaystyle\psi^{\rm tr}(\mathbf{k},0^{+})={\cal T}[\psi^{\rm
in}(\mathbf{k},0^{-})+\psi^{\rm re}(\mathbf{k},0^{-})],$ (15)

and so does the reflection phase $\phi_{r}$ accordingly. This is how the
spatial shifts are expected to get modified in presence of various surface
potentials.

##  IV Surface potentials

In this section, we will consider a family of surface potentials by expressing
the surface term ${\cal V}$ in Eq. 13 as

$\displaystyle{\cal V}=\lambda+{\bf V}\cdot{\bf\sigma},$ (16)

where $\lambda$ is a real constant, and the components of the vector ${\bf
V}=(V_{x},V_{y},V_{z})$ are, in general, real functions of $k_{y}$ and
$k_{z}$. In particular, we take note of the following cases:

  * •

Scalar potential: $\lambda\neq 0$, ${\bf V}=0$.

  * •

Magnetic potential: Uniform magnetic field specified by $V_{x}=B_{x}$,
$V_{y}=B_{y}$, $V_{z}=B_{z}$ where $B_{x,y,z}$ are constants and $\lambda=0$.

  * •

Spin-orbit potential of Rashba type: $\lambda=V_{x}=0$, but $V_{y}$ and
$V_{z}$ are linear functions of $k_{y}$ and $k_{z}$ as
$V_{y}=-\alpha_{R}k_{z}$ and $V_{z}=\alpha_{R}k_{y}$.

  * •

Spin-orbit potential of Dresselhaus type: $\lambda=V_{x}=0$, but $V_{y}$ and
$V_{z}$ are linear functions of $k_{y}$ and $k_{z}$ as $V_{y}=\alpha_{D}k_{y}$
and $V_{z}=-\alpha_{D}k_{z}$.

In the following subsections, we compute the GH and the IF shift from the
resultant $\phi_{r}$ while addressing the above four cases separately. It
should, however, be noted that combinations of them are also likely to occur
on the surface of a Weyl semimetal. We will illustrate one such combination in
which the surface potential includes both scalar and magnetic impurities.

###  IV.1 Scalar potential

Figure 2: Scalar potential: The valley-dependent GH shift as a function of the
incident angle $\theta$ for $V_{0}/E=1.5$ ($E=1$) for valley A in (a) and
valley B in (b) at different strengths ($\lambda$) of the scalar surface
potential. The critical angle $\theta_{c}=30^{\circ}$ below which no total
reflection takes place. The valley-dependent GH shift as a function of
$V_{0}/E$ for an incident angle $\theta=50^{\circ}$ for valley A in (c) and
valley B in (d). The clean case corresponds to $\lambda=0$ (red). The other
representative values of the scalar potential considered are $\lambda=0.6$
(green), $\lambda=1.9$ (blue), and $\lambda=2.6$ (magenta).

For a scalar surface potential of the form $V_{S}=\lambda\delta(x)$, the
transfer matrix that connects the two spinors across the interface is ${\cal
T}=e^{-i\sigma_{x}\lambda}$ which results in a chirality-dependent reflection
phase

$\displaystyle\phi_{r}$
$\displaystyle=-\theta-\frac{\pi}{2}+2\tan^{-1}\zeta,\leavevmode\nobreak\
\leavevmode\nobreak\ {\rm where}$ $\displaystyle\zeta$
$\displaystyle=\frac{\tilde{\eta}\cos\theta(1-\beta_{\chi}\tan\lambda)}{2\eta\chi(\tan\lambda+\beta_{\chi})-\tilde{\eta}\sin\theta(1-\beta_{\chi}\tan\lambda)}.$
(17)

The GH shift, in presence of a scalar potential, is chirality dependent

$\displaystyle\Delta_{\rm GH}$
$\displaystyle=\frac{\tilde{\kappa}^{2}(1-\tan^{2}\lambda)+2\epsilon\chi\tilde{\kappa}\tan\lambda+\epsilon\sec^{2}\lambda\cos^{2}{\bar{\theta}}}{E\tilde{\kappa}\sin{\bar{\theta}}\cos{\bar{\theta}}(1-\epsilon+2\tilde{\kappa}\chi\tan\lambda+(1+\epsilon)\tan^{2}\lambda)},$
(18)

where $\sin{\theta_{c}}\equiv-\epsilon=V_{0}/E-1$ as defined before and
$\tilde{\kappa}=\sqrt{\sin^{2}{\bar{\theta}}-\sin^{2}\theta_{c}}=\sqrt{\bar{k}_{y}^{2}-(E-V_{0})^{2}}/E=\kappa/E$
for $\bar{k}_{z}=0$.

Figure 3: Magnetic potential: The valley-dependent GH shift as a function of
the incident angle $\theta$ for $V_{0}/E=1.5$ ($E=1$) for valley A in (a) and
valley B in (b) at different strengths of the in-plane magnetic field
$B_{\parallel}$ keeping the orientation $\Phi=\pi/10$ and again for valley A
in (e) and valley B in (f) at different orientations of the in-plane magnetic
field $\Phi$ keeping the strength $B_{\parallel}=1$. The critical angle
$\theta_{c}=30^{\circ}$ as before. The valley-dependent GH shift is plotted as
a function of $V_{0}/E$ with an incident angle $\theta=50^{\circ}$ for valley
A in (c) and valley B in (d) varying $B_{\parallel}$ which shows a plateau-
like behavior as $B_{\parallel}$ is increased. The same varying $\Phi$ is
shown in (g) for valley A and (h) for valley B that features a peak for
certain values of the orientation which are distinct for different valleys.
The clean case corresponds to $B_{\parallel}=0$ (red) in (a)-(d) and the other
parameter values are shown in different colors.

The IF shift, on the other hand, is insensitive to the scalar potential and
retains its form as in Eq. 12 i.e.,

$\displaystyle\Delta_{\rm IF}=-\frac{\chi}{E\tan{\bar{\theta}}},$ (19)

reflecting topological robustness against such type of potential. In the limit
$\lambda\rightarrow 0$, we retrieve the clean results of Eq. 12.

At any finite strength of the scalar potential $\lambda$, the GH shift is
evidently valley-dependent and from the expression of the reflection phase
$\phi_{r}$ in Eq. IV.1, $\lambda=n\pi$ ($n$ integers) has the same effect as
$\lambda=0$ (clean case). Figure 2 (a)-(b) show the functional dependence of
$\Delta_{\rm GH}$ on the incident angle $\theta$ at various values of
$\lambda$. Here, we fix $V_{0}/E=1.5$ which, in turn, gives a critical angle
$\theta_{c}=30^{\circ}$ below which total reflection does not occur. The main
features of Fig. 2 (a)-(b) are highlighted below which are distinct for
different chiralities (also referred to as valleys).

Valley A ($\chi=+1$): For $0\leq\lambda<\lambda_{1}$ where
$\lambda_{1}=\pi/6$, $\Delta_{\rm GH}$ for valley A behaves qualitatively
similar to the clean case, however, the value of $\theta$ at which its changes
the sign, denoted $\theta^{\ast(+)}$, is $\lambda$-dependent. We find,
$\theta^{\ast(+)}=\sin^{-1}\sqrt{\alpha^{2}_{+}+1/4}$, where

$\displaystyle\alpha_{+}=\frac{2\tan\lambda+\sqrt{\left(6\cos(2\lambda)-\frac{1}{2}\cos(4\lambda)+\frac{7}{2}\right)\sec^{4}(\lambda)}}{8-2\sec^{2}\lambda}.$
(20)

In the aforementioned range, $\theta^{\ast(+)}$ gradually moves from
$45^{\circ}$ (at $\lambda=0$) to $90^{\circ}$ (at $\lambda=\lambda_{1}$)
monotonically. The numerator in Eq. 18 for valley A changes sign at
$\bar{\theta}=90^{\circ}$ across $\lambda=\lambda_{1}$, by virtue of which,
for $\lambda_{1}\leq\lambda\leq\lambda_{2}$ where $\lambda_{2}=2\pi/3$,
$\Delta_{\rm GH}$ for valley A remains finite irrespective of the incident
angle $\theta$, i.e., $\theta^{\ast(+)}$ does not exist. The numerator for
valley A changes sign again at $\bar{\theta}=90^{\circ}$ across
$\lambda=\lambda_{2}$. As a result, for $\lambda_{2}<\lambda\leq\pi$,
$\theta^{\ast(+)}$ decreases from $90^{\circ}$ (at $\lambda=\lambda_{2}$) to
$45^{\circ}$ (at $\lambda=\pi$), however not monotonically (it obtains a
minimum somewhere before $\lambda=\pi$).

Valley B ($\chi=-1$): For $0\leq\lambda<\lambda_{3}$ where
$\lambda_{3}=\pi/3$, $\Delta_{\rm GH}$ for valley B behaves qualitatively
similar to the clean case, however, the value of $\theta$ at which its changes
the sign, denoted $\theta^{\ast(-)}$, is $\lambda$-dependent. We find,
$\theta^{\ast(-)}=\sin^{-1}\sqrt{\alpha^{2}_{-}+1/4}$, where

$\displaystyle\alpha_{-}=\frac{-2\tan\lambda+\sqrt{\left(6\cos(2\lambda)-\frac{1}{2}\cos(4\lambda)+\frac{7}{2}\right)\sec^{4}(\lambda)}}{8-2\sec^{2}\lambda}.$
(21)

In the aforementioned range, $\theta^{\ast(-)}$ increase from $45^{\circ}$ (at
$\lambda=0$) to $90^{\circ}$ (at $\lambda=\lambda_{3}$) but nonmonotonically
(it attains a minimum somewhere in-between). The numerator in Eq. 18 for
valley B changes sign at $\bar{\theta}=90^{\circ}$ across
$\lambda=\lambda_{3}$, by virtue of which, for
$\lambda_{3}\leq\lambda\leq\lambda_{4}$ where $\lambda_{4}=5\pi/6$,
$\Delta_{\rm GH}$ for valley B remains finite irrespective of the incident
angle $\theta$, i.e., $\theta^{\ast(-)}$ does not exist, just like valley A.
The numerator for valley B changes sign again at $\bar{\theta}=90^{\circ}$
across $\lambda=\lambda_{4}$. Consequently, for $\lambda_{4}<\lambda\leq\pi$,
$\theta^{\ast(-)}$ decreases monotonically from $90^{\circ}$ (at
$\lambda=\lambda_{4}$) to $45^{\circ}$ (at $\lambda=\pi$).

In summary, a notable aspect, which gains prominence from the above analysis,
is that the GH shift, in presence of scalar surface potential, can behave in
vastly different ways for the two valleys – it can vanish for one of the
valleys for certain values of the strength of the potential while, at the same
time, it remains finite for the other. Note the plots of $\Delta_{\rm GH}$
against $\theta$ for the two valleys overlap at the transition points
$\lambda=\lambda_{1,2,3,4}$. Figure 2 (c)-(d) display the behavior of
$\Delta_{\rm GH}$ against the ratio of the barrier height to the incident
energy $V_{0}/E$ which also feature a strong valley-dependence in presence of
a scalar surface potential, namely, the values of $V_{0}/E$ at which the GH
shift vanishes, depends on the chirality, unlike the clean case. As the IF is
not influenced by the scalar potential, we do not show its parametric
dependence which is already discussed in Ref. Jiang et al., 2015.

###  IV.2 Magnetic potential

Figure 4: Magnetic potential: The valley-dependent IF shift as a function of
the incident angle $\theta$ for $V_{0}/E=1.5$ ($E=1$) for valley A in (a) and
valley B in (b) at different strengths of the in-plane magnetic field
$B_{\parallel}$ keeping the orientation $\Phi=\pi/10$ and again for valley A
in (e) and valley B in (f) at different orientations of the in-plane magnetic
field $\Phi$ keeping the strength $B_{\parallel}=1$. The critical angle
$\theta_{c}=30^{\circ}$ as before. The IF shift is plotted as a function of
$V_{0}/E$ with an incident angle $\theta=50^{\circ}$ for valley A in (c) and
valley B in (d) varying $B_{\parallel}$. A vanishing IF shift at this
incidence is observed in presence of a finite $B_{\parallel}$ which is
specific to only one of the valleys (namely, A). The same varying $\Phi$ is
shown in (g) for valley A and (h) for valley B that displays a peak for
certain values of the orientation which are distinct for different valleys.
The clean case corresponds to $B_{\parallel}=0$ (red) in (a)-(d) and the other
parameter values are shown in different colors.

For a magnetic impurity present all over the interface with field orientation
$\vec{\bf B}=(B_{x},B_{y},B_{z})$, the transfer matrix is

$\displaystyle{\cal T}=e^{-i\sigma_{x}(\vec{\bf
B}\cdot{\vec{\bf\sigma}})}=e^{-iB_{x}}e^{B_{y}\sigma_{z}-B_{z}\sigma_{y}}.$
(22)

The out-of-plane component $B_{x}$ contributes merely as a phase factor and
does not influence the shifts. For the remaining components, it is useful to
introduce the parameters $B_{\parallel}=\sqrt{B_{y}^{2}+B_{z}^{2}}$ and
$\tan\Phi=B_{z}/B_{y}$. In terms of these parameters, the reflection phase is

$\displaystyle\phi_{r}$
$\displaystyle=-\theta-\frac{\pi}{2}+2\tan^{-1}\zeta_{B},\leavevmode\nobreak\
\leavevmode\nobreak\ {\rm where}$ $\displaystyle\zeta_{B}$
$\displaystyle=\frac{\tilde{\eta}\cos{\theta}(b_{B}+\beta_{\chi}a_{B})}{2\eta\chi(\beta_{\chi}+a_{B})-\tilde{\eta}\sin\theta(b_{B}+\beta_{\chi}a_{B})},$
(23)

with the parameters $a_{B}$ and $b_{B}$ defined as

$\displaystyle
a_{B}=\frac{\sin{\Phi}\tanh{B_{\parallel}}}{1+\cos{\Phi}\tanh{B_{\parallel}}}\leavevmode\nobreak\
\leavevmode\nobreak\ ;\leavevmode\nobreak\ \leavevmode\nobreak\
b_{B}=\frac{1-\cos{\Phi}\tanh{B_{\parallel}}}{1+\cos{\Phi}\tanh{B_{\parallel}}}.$
(24)

The corresponding shifts are given by

$\displaystyle\Delta_{\rm
GH}=\frac{1}{E\tilde{\kappa}\cos{\bar{\theta}}}\frac{{\cal N}_{\rm GH}}{{\cal
D}}\leavevmode\nobreak\ \leavevmode\nobreak\ ;\leavevmode\nobreak\
\leavevmode\nobreak\ \Delta_{\rm IF}=\frac{2\cos{\bar{\theta}}}{E}\frac{{\cal
N}_{\rm IF}}{{\cal D}},$ (25)

where

$\displaystyle{\cal N}_{\rm GH}$
$\displaystyle=\tilde{\kappa}^{2}(b_{B}^{2}+1-2a_{B}^{2})+\tilde{\kappa}(1-b_{B}^{2})\sin{\bar{\theta}}$
$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\
+2\epsilon[\chi\tilde{\kappa}a_{B}(1-b_{B})-(a_{B}^{2}-b_{B})\cos^{2}{\bar{\theta}}],$
$\displaystyle{\cal N}_{\rm IF}$
$\displaystyle=a_{B}(1+b_{B})\sin{\bar{\theta}}+a_{B}\tilde{\kappa}(1-b_{B})$
$\displaystyle\leavevmode\nobreak\ \leavevmode\nobreak\
+\chi[(a_{B}^{2}+b_{B})\epsilon+a_{B}^{2}-b_{B}],$ $\displaystyle{\cal D}$
$\displaystyle=[(\sin{\bar{\theta}}+\tilde{\kappa})(1+a_{B}^{2}-2\chi
a_{B}\sin{\bar{\theta}})$
$\displaystyle+(\sin{\bar{\theta}}-\tilde{\kappa})(a_{B}^{2}+b_{B}^{2}-2a_{B}b_{B}\chi\sin{\bar{\theta}})$
$\displaystyle+2\epsilon(a_{B}\chi(1+b_{B})-a_{B}^{2}\sin{\bar{\theta}}-b_{B}\sin{\bar{\theta}})].$
(26)

Eq. 25 matches with the clean results in the limit $B_{\parallel}\rightarrow
0$.

Figure 3 displays the behaviour of the GH shift $\Delta_{\rm GH}$ in presence
of magnetic surface potential at various values of the in-plane strength
$B_{\parallel}$ and the orientation of the magnetic field characterized by the
angle $\Phi$ defined previously. In Fig. 3 (a)-(d), $B_{\parallel}$ is being
varied while keeping the orientation $\Phi$ fixed to $\pi/10$. In Fig. 3
(e)-(h), $B_{\parallel}=1$ while the orientation $\Phi$ is being varied. The
out-of-plane component has no effect on the shifts and taken to be zero. In
Fig. 3 (a)-(b), $\Delta_{\rm GH}$ is plotted against the incident angle
$\theta$ for the two different valleys, given $V_{0}/E=1.5$. Evidently,
$\Delta_{\rm GH}$ is no more an odd function of $\theta$ unlike the clean or
the scalar case. The distinctive behavior of the two valleys are prominent
that can be explained in the same way as the scalar potential. In Fig. 3
(c)-(d), $\Delta_{\rm GH}$ is plotted against $V_{0}/E$ which, for the both
the valleys, features a plateau-like behavior as $B_{\parallel}$ is increased.
When the orientation $\Phi$ is varied, $\Delta_{\rm GH}$ plotted against
$\theta$ as in Fig. 3 (e)-(f) behaves in a qualitatively similar way to Fig. 3
(a)-(b), however, its variation against $V_{0}/E$ is observed to develop
conspicuous peaks for certain orientations whose height can differ in order of
magnitudes between the two valleys as shown in Fig. 3 (g)-(h).

The behavior of the IF shift in presence of the magnetic surface potential is
quite remarkable as shown in Fig. 4. Similar to Fig. 3, the variation with
respect to $B_{\parallel}$ keeping $\Phi$ fixed is displayed in Fig. 4 (a)-(d)
while the opposite is shown in Fig. 4 (e)-(h). Note that at any finite
$B_{\parallel}$, the IF shift is defined only for $|\theta|>\theta_{c}$ where
the total reflection takes place because of the parameter $\tilde{\kappa}$
unlike the clean case. For the clean case, this parameter in absent in the
expression of $\Delta_{\rm IF}$ as in Eq. 12. Besides the difference in
magnitude, $\Delta_{\rm IF}$ as a function of $\theta$ can have the same sign
for the two valleys depending on the values of $B_{\parallel}$ when it is
gradually increased. In Fig. 4 (a)-(b), such a behavior is observed for
$B_{\parallel}=1.5$. Further from Eq. 25, $\Delta_{\rm IF}$ can change sign
for $\chi=+1$ and vanishes at a specific value of $\epsilon$ or equivalently
$V_{0}/E$ that depends on $B_{\parallel}$ and $\Phi$. This is not the case for
the other valley [see Fig. 4 (c)-(d)]. In the clean case, i.e., for
$B_{\parallel}=0$, the factor containing $\epsilon$ drops off which results in
$\Delta_{\rm IF}$ being independent of $V_{0}/E$ with the chirality $\chi$
appearing as a prefactor. Thus, magnetic surface potential can lead to a
vanishing IF shift for a specific valley ($\chi=+1$) at an incident angle
$\theta_{c}<\theta<\pi/2$ while not having any such effect on the other. Upon
varying the orientation $\Phi$ as in Fig. 4 (e)-(f), for both the valleys, the
behavior appears similar to varying $B_{\parallel}$. In addition, when plotted
against $V_{0}/E$, $\Delta_{\rm IF}$ exhibits a peak, similar to $\Delta_{\rm
GH}$, for certain values of $\Phi$ as shown in Fig. 4 (g)-(h) for
$B_{\parallel}=1$.

###  IV.3 Rashba spin-orbit potential

Figure 5: Rashba potential: The valley-dependent GH shift as a function of the
incident angle $\theta$ for $V_{0}/E=1.5$ ($E=1$) for valley A in (a) and
valley B in (b) at different strengths ($\alpha_{R}$) of the Rashba spin-orbit
potential. The same as a function of $V_{0}/E$ for an incident angle
$\theta=50^{\circ}$ for valley A in (c) and valley B in (d). The clean case
corresponds to $\alpha_{R}=0$ (red).

For surface potentials resulting from Rashba type spin-orbit coupling
$\mathcal{V}=\alpha_{R}(k_{y}\sigma_{z}-k_{z}\sigma_{y})$, the transfer matrix
is given as-

$\displaystyle\mathcal{T}$
$\displaystyle=e^{-i\alpha_{R}\sigma_{x}(k_{y}\sigma_{z}-k_{z}\sigma_{y})}=e^{-\alpha_{R}(k_{z}\sigma_{z}+k_{y}\sigma_{y})}$
(27)

and the reflection phase, in this case, $\phi_{r}$ turns out to be

$\displaystyle\phi_{r}$
$\displaystyle=-\theta-\frac{\pi}{2}+2\tan^{-1}\zeta_{R},\leavevmode\nobreak\
\leavevmode\nobreak\ {\rm where}$ $\displaystyle\zeta_{R}$
$\displaystyle=\frac{\tilde{\eta}\cos\theta(1+\beta_{\chi}a_{R})}{2\eta\chi(\beta_{\chi}b_{R}+a_{R})-\tilde{\eta}\sin\theta(1+\beta_{\chi}a_{R})},$
(28)

with the parameters $a_{R}$ and $b_{R}$ defined as

$\displaystyle
a_{R}=\frac{\cos\varphi\tanh{(\alpha_{R}k_{\parallel}})}{1+\sin\varphi\tanh{(\alpha_{R}k_{\parallel}})}\leavevmode\nobreak\
;\leavevmode\nobreak\
b_{R}=\frac{1-\sin\varphi\tanh{(\alpha_{R}k_{\parallel}})}{1+\sin\varphi\tanh{(\alpha_{R}k_{\parallel}})}.$
(29)

Here, $k_{\parallel}=\sqrt{k^{2}_{y}+k^{2}_{z}}$ and
$\tan\varphi={k_{z}}/{k_{y}}$.

Note the expression of $\zeta_{R}$ can be obtained from that of $\zeta_{B}$ by
identifying $B_{y}\leftrightarrow-\alpha_{R}k_{z}$ and
$B_{z}\leftrightarrow\alpha_{R}k_{y}$, or equivalently,
$B_{\parallel}\leftrightarrow\alpha_{R}k_{\parallel}$ and
$\Phi\leftrightarrow\pi/2+\varphi$, which, in turn, identifies
$a_{B}/b_{B}\leftrightarrow a_{R}$ and $b^{-1}_{B}\leftrightarrow b_{R}$. This
readily yields Eq. IV.3 from Eq. IV.2. However, the results for the shifts
that follow are significantly different from the magnetic impurity case since
$a_{R}$ and $b_{R}$ are momentum-dependent parameters unlike $a_{B}$ and
$b_{B}$. This has distinct effects on the two shifts when we adopt a reference
by setting $\bar{k}_{z}=0$.

In detail, in this setting, the Rashba type surface potential appears to have
no effect on the IF shift i.e.,

$\Delta_{\rm IF}=-\frac{\chi}{E\tan\bar{\theta}},$ (30)

same as the clean case. The chirality-dependent GH shift, however, is modified
and given by

$\Delta_{\rm GH}=\frac{1}{\kappa\cos\bar{\theta}}\frac{\mathcal{N}_{\rm
R}}{\mathcal{D}_{\rm R}},$ (31)

where

$\displaystyle\mathcal{N}_{\rm R}$
$\displaystyle=(\tilde{\kappa}^{2}+\epsilon\cos^{2}\bar{\theta}-2\chi\alpha_{R}E\tilde{\kappa}^{2}\cos^{2}\bar{\theta})\operatorname{sech}^{2}(\alpha_{R}E\sin\bar{\theta}),$
$\displaystyle\mathcal{D}_{\rm R}$
$\displaystyle=\sin{\bar{\theta}}(1-\epsilon)[1+\tanh^{2}({\alpha_{R}E\sin\bar{\theta}})]$
$\displaystyle+2\chi\tanh({\alpha_{R}E\sin\bar{\theta}})(\epsilon-\sin^{2}\bar{\theta}).$
(32)

It is straightforward to show that ${\cal N}_{\rm R}$ and ${\cal D}_{\rm R}$
approach their corresponding clean values in the limit $\alpha_{R}\rightarrow
0$.

Figure 6: Dresselhaus potential: The GH shift as a function of the incident
angle $\theta$ for $V_{0}/E=1.5$ ($E=1$) in (a) and the same as a function of
$V_{0}/E$ with $\theta=50^{\circ}$ in (b) for valley A at different strengths
($\alpha_{D}$) of the Dresselhaus spin-orbit potential. The IF shift under the
same setup in (c) and (d). Note here $\Delta_{\rm GH}$ is valley-independent
while $\Delta_{\rm IF}\propto\chi$ like the clean case ($\alpha_{D}=0$).

The influence of Rashba-type surface potential on the shifts is depicted in
Fig. 5. The GH shift behaves qualitatively very similar to the clean case in
the sense that it is odd in $\theta$ and for all values of $\alpha_{R}$ below
$\alpha_{R}\sim 10$ (beyond this, $\Delta_{\rm GH}\sim 10^{-5}$ and gets
further suppressed with increasing $\alpha_{R}$), there exists a
$\theta^{\ast}$ at which the shift vanishes (and across which it changes sign)
as can be seen in Fig. 5 (a)-(b) for both the valleys. The same comparison
holds for the plots of $\Delta_{\rm GH}$ vs. $V_{0}/E$ [Fig. 5 (c)-(d)] except
that at large values of $\alpha_{R}$, the GH shift plateaus over the entire
range of $V_{0}/E$ for both the valleys. Further, as shown above, the IF shift
is not affected, and so, we do not provide any parametric plot of $\Delta_{\rm
IF}$ for this case.

###  IV.4 Dresselhaus spin-orbit potential

For surface potentials that arise from Dresselhaus type spin-orbit coupling
$\mathcal{V}=\alpha_{D}(k_{y}\sigma_{y}-k_{z}\sigma_{z})$, the transfer matrix
is given by

$\displaystyle\mathcal{T}=e^{-i\alpha_{D}\sigma_{x}(k_{y}\sigma_{y}-k_{z}\sigma_{z})}=e^{\alpha_{D}(k_{z}\sigma_{y}+k_{y}\sigma_{z})},$
(33)

which results in a reflection phase

$\displaystyle\phi_{r}$
$\displaystyle=-\theta-\frac{\pi}{2}+2\tan^{-1}\zeta_{D},\leavevmode\nobreak\
\leavevmode\nobreak\ {\rm where}$ $\displaystyle\zeta_{D}$
$\displaystyle=\frac{\tilde{\eta}\cos\theta(b_{D}-\beta_{\chi}a_{D})}{2\eta\chi(\beta_{\chi}-a_{D})-\tilde{\eta}\sin\theta(b_{D}-\beta_{\chi}a_{D})},$
(34)

with the parameters $a_{D}$ and $b_{D}$ defined as

$\displaystyle
a_{D}=\frac{\sin\varphi\tanh(\alpha_{D}k_{\parallel})}{1+\cos\varphi\tanh(\alpha_{D}k_{\parallel})}\leavevmode\nobreak\
;\leavevmode\nobreak\
b_{D}=\frac{1-\cos\varphi\tanh(\alpha_{D}k_{\parallel})}{1+\cos\varphi\tanh(\alpha_{D}k_{\parallel})}.$
(35)

Here a straightforward identification $a_{B}\leftrightarrow-a_{D}$ and
$b_{B}\leftrightarrow b_{D}$ (by virtue of
$B_{\parallel}\leftrightarrow\alpha_{D}k_{\parallel}$ and $\Phi\leftrightarrow
2\pi-\varphi$) would yield Eq. IV.4 from Eq. IV.2.

Following analogous calculations as the Rashba case above, we obtain the
shifts as

$\displaystyle\Delta_{\rm
GH}=\frac{1}{\kappa\cos\bar{\theta}}\frac{\mathcal{N}_{\rm D}^{(\rm
GH)}}{{\cal D}_{\rm D}}\leavevmode\nobreak\ \leavevmode\nobreak\
;\leavevmode\nobreak\ \leavevmode\nobreak\ \Delta_{\rm
IF}=-\frac{\chi}{E\tan\bar{\theta}}\frac{\mathcal{N}_{\rm D}^{(\rm IF)}}{{\cal
D}_{\rm D}},$ (36)

where

$\displaystyle\mathcal{N}_{\rm D}^{(\rm IF)}$
$\displaystyle=(1-\epsilon)[1-\tanh^{2}(\alpha_{D}E\sin\bar{\theta})]\sin\bar{\theta}$
$\displaystyle-2\tilde{\kappa}\tanh(\alpha_{D}E\sin\bar{\theta})-2\tanh^{2}(\alpha_{D}E\sin\bar{\theta})\sin\bar{\theta},$
$\displaystyle\mathcal{N}_{\rm D}^{(\rm GH)}$
$\displaystyle=(\tilde{\kappa}^{2}+\epsilon\cos^{2}\bar{\theta}+2\alpha_{D}\epsilon\tilde{\kappa}E\cos^{2}\bar{\theta})$
$\displaystyle+2\tanh(\alpha_{D}E\sin\bar{\theta})\tilde{\kappa}\sin\bar{\theta}$
$\displaystyle+\tanh^{2}(\alpha_{D}E\sin\bar{\theta})[\tilde{\kappa}^{2}-\epsilon\cos^{2}\bar{\theta}-2\alpha_{D}\epsilon\tilde{\kappa}E\cos^{2}\bar{\theta}],$
(37)

and

$\displaystyle\mathcal{D}_{\rm D}$
$\displaystyle=\sin\bar{\theta}(1-\epsilon)+2\tilde{\kappa}\tanh(\alpha_{D}E\sin\bar{\theta})$
$\displaystyle+\tanh^{2}(\alpha_{D}E\sin\bar{\theta})\sin\bar{\theta}(1+\epsilon).$
(38)

Again, the clean results are readily obtained in the limit
$\alpha_{D}\rightarrow 0$.

The Dresselhaus-type surface potential has quite a few exotic effects on both
the shifts. Firstly, from the above expressions, $\Delta_{\rm GH}$ is
chirality-independent and odd in $\theta$ while $\Delta_{\rm IF}$ carries the
same chirality dependence as the clean case i.e., $\Delta_{\rm
IF}\propto\chi$. This is reflected in Fig. 6. In Fig. 6 (a), $\Delta_{\rm GH}$
is plotted against $\theta$ keeping $V_{0}/E=1.5$. Similar to the other
potentials, a $\theta^{\ast}$ exists at which $\Delta_{\rm GH}$ vanishes in a
given range of $\alpha_{D}$ beyond which $|\Delta_{\rm GH}|$ remains finite.
In Fig. 6 (b), $\Delta_{\rm GH}$ is plotted against $V_{0}/E$ keeping
$\theta=50^{\circ}$. We note while at small values of $\alpha_{D}$, the
behavior of $\Delta_{\rm GH}$ looks qualitatively similar to the clean case,
as $\alpha_{D}$ is gradually increased, the plateauing effect starts
dominating and eventually at very large values of $\alpha_{D}$, $\Delta_{\rm
GH}$ saturates at a value that increases with $\theta$.

For the IF shift, we find that the Dresselhaus type surface potential can
interestingly lead to phenomenon like valley inversion as seen in Fig. 6 (c)
in the following manner. Let us denote the value of $\theta$ at which the IF
shift vanishes by $\theta_{\rm IF}^{\ast}$. For the clean case, $|\theta_{\rm
IF}^{\ast}|=\pi/2$. When $\alpha_{D}$ is increased, there appears another
$\theta^{\ast}_{\rm IF}$ such that $|\theta^{\ast}_{\rm IF}|<\pi/2$ which
gradually approaches $\theta_{c}$ as $\alpha_{D}$ approaches

$\displaystyle\alpha_{D}^{\ast}=\frac{1}{E\sin\theta_{c}}\tanh^{-1}\frac{V_{0}}{V_{0}+2}.$
(39)

If $\alpha_{D}$ is increased further beyond this value, a valley inversion
takes place as $\Delta_{\rm IF}$ changes sign across
$\alpha_{D}=\alpha_{D}^{\ast}$. Similar phenomenon is also observed in the
plot of $\Delta_{\rm IF}$ against $V_{0}/E$ as in Fig. 6 (d) where the valley
inversion manifests as reordering of the curves of $\Delta_{\rm IF}$ against
$V_{0}/E$ at different values of $\theta$ as $\alpha_{D}$ is increased and
consequently, $\Delta_{\rm IF}(\theta)\rightarrow-\Delta_{\rm IF}(\theta)$.

Figure 7: Combination of scalar and Zeeman potential: GH shift as a function
of the incident angle $\theta$ for $V_{0}/E=1.5$ for valley A in (a) and
valley B in (b) at different strengths of the scalar surface potential
($\lambda$) and magnetic potential ($B_{\parallel}$). The same as a function
of $V_{0}/E$ at $\theta=50^{\circ}$ for valley A in (c) and valley B in (d).
The combinations of $\lambda$ and $B_{\parallel}$ are specified in different
colors as $\lambda=1,B_{\parallel}=0$ in red, $\lambda=0,B_{\parallel}=1$ in
green, $\lambda=0.5,B_{\parallel}=0.2$ in blue.

##  V Discussion

In this article, we revisit the phenomena of GH and IF shifts, the lateral
shifts of an incident beam upon total reflection, in a Weyl semimetal system
to discuss the effects of surface potentials on these shifts which could be
probed in real materials. Earlier these shift were studied for a clean Weyl
surface and argued that the IF shift is topological in nature, namely, it is
chirality-dependent and so, can be exploited in experiments to characterize
Weyl systems. The GH shift in Weyl semimetals, on the other hand, does not
have such a feature. However, as we reveal in this article, this is not
entirely true when the concerned surface harbors various kinds of surface
potentials as is the case in real materials exhibiting Weyl nodes. Among the
key observations, presence of any type of surface potential renders the GH
shift strongly chirality-dependent. In fact, strong scalar or magnetic
potential can yield a situation in which the GH shift remains finite
irrespective of the incident angle for the allowed values of the magnitude of
the chemical potential barrier ($V_{0}$) that distinguishes the surface. For
other types of surface potentials such as of the Rashba or Dresselhaus type,
the GH shift shows conspicuous departure from the clean results. When plotted
as a function of the barrier height $V_{0}$, strong surface potentials give
rise to plateauing effect in the GH shift and sometimes, peaks at specific
values of $V_{0}$ for magnetic impurities. These effects can be particularly
useful in probing magnetic Weyl systems that have recently nucleated
experimental activities.

For the IF shift, the scalar surface potentials turn out to have no effect.
However, the other kinds of surface potentials do leave remarkable signatures
compared to the clean case similar to when intervalley scattering takes place.
Most promising are the magnetic and Dresselhaus-type impurities, in presence
of which, the IF shift, which otherwise, is independent of $V_{0}$ and changes
sign between the two valleys for the clean case, develops a strong valley
asymmetry beyond a simple sign inversion and also a parametric dependence on
$V_{0}$ such that it can vanish at certain values of $V_{0}$ depending on the
incident angle $\theta$. Similarly, in distinction to the clean case where the
IF shift vanishes only at $\theta=\pm\pi/2$, the impurities can enforce
intermediate values of $0<|\theta|<\pi/2$ at which the IF shift is nullified
and that too can be valley-dependent. This can potentially mask the IF shift
to be identified as a topological effect in realistic Weyl systems as their
surfaces would typically host various impurities even including a mixed
nature.

To investigate the effects of such mixed impurities, we study the case where
the surface potential includes both the scalar and magnetic contributions as
${\cal V}=\lambda+{\bf B}\cdot\sigma$. The results are summarized in Fig. 7.
While the IF shift (not shown) is dictated by the magnetic contribution, the
GH shift is affected by both. In fact, as revealed by Fig. 7, the plateauing
effect is stronger when both types of surface potentials are present with
moderate strengths, compared to their individual influences at higher
strengths. If the two impurities form distinct domains on the surface, we
expect the resultant shifts to be an weighted average of the individual shifts
as
$\langle\Delta\rangle=w\langle\Delta_{1}\rangle+(1-w)\langle\Delta_{2}\rangle$
where the two kinds of impurities are denoted by 1 and 2 and
$\langle\dots\rangle$ implies ensemble averaging that includes the impurities
with probabilities $w$ and $(1-w)$ respectively. This extends to multiple
types of impurities as well.

In summary, our work extends the phenomena of GH and IF shift in Weyl
semimetals beyond a clean surface and accommodates surface potentials to
unveil novel features of the shifts. Our observations would provide useful
guidance to experiments that are tuned to characterize Weyl systems based on
such phenomena that have already found potential relevance in device
applications engaging other electronic systems. Studying similar effects on
other transport properties of Weyl semimetals will be addressed elsewhere.

###### Acknowledgements.

NKD and KR gratefully acknowledge useful discussion with Qing-Dong Jiang and
Sourin Das and thank them for critically reviewing the manuscript. KR also
thanks the sponsorship, in part, by the Swedish Research Council.

## References

  * Goos and H ”a nchen (1947) F. Goos and H. H ”a nchen, Annals of Physics 436, 333 (1947).
  * Artmann (1948) K. Artmann, Annalen der Physik 437, 87 (1948).
  * McGuirk and Carniglia (1977) M. McGuirk and C. Carniglia, JOSA 67, 103 (1977).
  * Fedorov (1955) F. I. Fedorov, Doklady Akademii Nauk SSSR 105, 465 (1955).
  * Schilling (1965) H. Schilling, Annalen der Physik 471, 122 (1965).
  * Imbert (1972) C. Imbert, Physical Review D 5, 787 (1972).
  * Wild and Giles (1982) W. J. Wild and C. L. Giles, Physical Review A 25, 2099 (1982).
  * Bretenaker et al. (1992) F. Bretenaker, A. Le Floch, and L. Dutriaux, Physical review letters 68, 931 (1992).
  * Pfleghaar et al. (1993) E. Pfleghaar, A. Marseille, and A. Weis, Physical review letters 70, 2281 (1993).
  * Emile et al. (1995) O. Emile, T. Galstyan, A. Le Floch, and F. Bretenaker, Physical review letters 75, 1511 (1995).
  * Bonnet et al. (2001) C. Bonnet, D. Chauvat, O. Emile, F. Bretenaker, A. Le Floch, and L. Dutriaux, Optics letters 26, 666 (2001).
  * Berman (2002) P. Berman, Physical Review E 66, 067603 (2002).
  * Li (2003) C.-F. Li, Physical review letters 91, 133903 (2003).
  * Fan et al. (2003) J. Fan, A. Dogariu, and L. Wang, Optics express 11, 299 (2003).
  * Shadrivov et al. (2003) I. V. Shadrivov, A. A. Zharov, and Y. S. Kivshar, Applied Physics Letters 83, 2713 (2003).
  * Felbacq et al. (2003) D. Felbacq, A. Moreau, and R. Smaâli, Optics letters 28, 1633 (2003).
  * Yin et al. (2004) X. Yin, L. Hesselink, Z. Liu, N. Fang, and X. Zhang, Applied Physics Letters 85, 372 (2004).
  * Merano et al. (2007) M. Merano, A. Aiello, M. Van Exter, E. Eliel, J. Woerdman, et al., Optics express 15, 15928 (2007).
  * Beenakker et al. (2009) C. Beenakker, R. Sepkhanov, A. Akhmerov, and J. Tworzydło, Physical Review Letters 102, 146804 (2009).
  * de Haan et al. (2010) V.-O. de Haan, J. Plomp, T. M. Rekveldt, W. H. Kraan, A. A. van Well, R. M. Dalgliesh, and S. Langridge, Physical review letters 104, 010401 (2010).
  * Nieminen et al. (2020) A. Nieminen, A. Marini, and M. Ornigotti, Journal of Optics 22, 035601 (2020).
  * Chen et al. (2008) X. Chen, C.-F. Li, and Y. Ban, Physical Review B 77, 073307 (2008).
  * Miller Jr and Ashby (1972) S. C. Miller Jr and N. Ashby, Physical Review Letters 29, 740 (1972).
  * Fradkin and Kashuba (1974) D. Fradkin and R. Kashuba, Physical Review D 9, 2775 (1974).
  * Chen et al. (2011) X. Chen, J.-W. Tao, and Y. Ban, The European Physical Journal B 79, 203 (2011).
  * Sharma and Ghosh (2011) M. Sharma and S. Ghosh, Journal of physics: condensed matter 23, 055501 (2011).
  * Zhai et al. (2011) F. Zhai, Y. Ma, and K. Chang, New Journal of Physics 13, 083029 (2011).
  * Cheng (2012) M. Cheng, The European Physical Journal B 85, 1 (2012).
  * Wang et al. (2013a) Y. Wang, Y. Liu, and B. Wang, Physica E: Low-dimensional Systems and Nanostructures 48, 191 (2013a).
  * Chen et al. (2013) X. Chen, X.-J. Lu, Y. Ban, and C.-F. Li, Journal of Optics 15, 033001 (2013).
  * Wang et al. (2013b) Y. Wang, Y. Liu, and B. Wang, Physica E: Low-dimensional Systems and Nanostructures 53, 186 (2013b).
  * Grosche et al. (2015) S. Grosche, M. Ornigotti, and A. Szameit, Optics express 23, 30195 (2015).
  * Zeng et al. (2017) X. Zeng, M. Al-Amri, and M. S. Zubairy, Optics express 25, 23579 (2017).
  * Wu et al. (2018) W. Wu, W. Zhang, S. Chen, X. Ling, W. Shu, H. Luo, S. Wen, and X. Yin, Optics express 26, 23705 (2018).
  * Das and Pradhan (2021) A. Das and M. Pradhan, JOSA B 38, 387 (2021).
  * Fan et al. (2016) Y. Fan, N.-H. Shen, F. Zhang, Z. Wei, H. Li, Q. Zhao, Q. Fu, P. Zhang, T. Koschny, and C. M. Soukoulis, Advanced Optical Materials 4, 1824 (2016).
  * Zheng et al. (2019) Z. Zheng, F. Lu, L. Jiang, X. Jin, X. Dai, and Y. Xiang, Optics Communications 452, 227 (2019).
  * Liu et al. (2020a) J.-Y. Liu, T.-J. Huang, L.-Z. Yin, F.-Y. Han, and P.-K. Liu, IEEE Photonics Journal 12, 1 (2020a).
  * Zhang and Chan (2014) Q. Zhang and K. Chan, Applied Physics Letters 105, 212408 (2014).
  * Wu et al. (2011) Z. Wu, F. Zhai, F. Peeters, H. Xu, and K. Chang, Physical review letters 106, 176802 (2011).
  * Peres (2010) N. Peres, Reviews of modern physics 82, 2673 (2010).
  * Kort-Kamp et al. (2016) W. Kort-Kamp, N. Sinitsyn, and D. Dalvit, Physical Review B 93, 081410 (2016).
  * Farmani et al. (2017) A. Farmani, M. Miri, and M. H. Sheikhi, JOSA B 34, 1097 (2017).
  * Xu et al. (2017) G. Xu, J. Sun, H. Mao, and T. Pan, Physics Letters A 381, 2876 (2017).
  * Luo et al. (2018) C. Luo, Z. Wang, X. Dai, and Y. Xiang, JOSA B 35, 2889 (2018).
  * Zhu (2019) M. Zhu, Journal of Magnetism and Magnetic Materials 487, 165319 (2019).
  * Jiang et al. (2015) Q.-D. Jiang, H. Jiang, H. Liu, Q.-F. Sun, and X.-C. Xie, Physical review letters 115, 156602 (2015).
  * Jiang et al. (2016) Q.-D. Jiang, H. Jiang, H. Liu, Q.-F. Sun, and X. Xie, Physical Review B 93, 195165 (2016).
  * Wang and Jian (2017) L. Wang and S.-K. Jian, Physical Review B 96, 115448 (2017).
  * Hao et al. (2019) Y.-R. Hao, L. Wang, and D.-X. Yao, Physical Review B 99, 165406 (2019).
  * Ye et al. (2019) G. Ye, W. Zhang, W. Wu, S. Chen, W. Shu, H. Luo, and S. Wen, Physical Review A 99, 023807 (2019).
  * Chattopadhyay et al. (2019) U. Chattopadhyay, L.-k. Shi, B. Zhang, J. C. Song, and Y. D. Chong, Physical review letters 122, 066602 (2019).
  * Liu et al. (2020b) S. Liu, Y. Shou, X. Zhou, W. Cheng, and Z. Luo, Optics express 28, 10783 (2020b).
  * Hosur and Qi (2013) P. Hosur and X. Qi, Comptes Rendus Physique 14, 857 (2013).
  * Hasan et al. (2017) M. Z. Hasan, S.-Y. Xu, I. Belopolski, and S.-M. Huang, Annual Review of Condensed Matter Physics 8, 289 (2017).
  * Zhang et al. (2012) F. Zhang, C. L. Kane, and E. J. Mele, Physical Review B 86, 081303 (2012).
  * Roy and Das (2016) S. Roy and S. Das, Physical Review B 93, 085422 (2016).
  * Roy et al. (2016) S. Roy, K. Roychowdhury, and S. Das, New Journal of Physics 18, 073038 (2016).
  * Liu et al. (2019) D. Liu, A. Liang, E. Liu, Q. Xu, Y. Li, C. Chen, D. Pei, W. Shi, S. Mo, P. Dudin, et al., Science 365, 1282 (2019).
  * Huang et al. (2015) S.-M. Huang, S.-Y. Xu, I. Belopolski, C.-C. Lee, G. Chang, B. Wang, N. Alidoust, G. Bian, M. Neupane, A. Bansil, et al., arXiv preprint arXiv:1501.00755 (2015).
  * Huang et al. (2013) Z. Huang, D. P. Arovas, and A. V. Balatsky, New Journal of Physics 15, 123019 (2013).
  * He (2018) C. He, Physics Letters A 382, 440 (2018).
  * Calkin et al. (1987) M. Calkin, D. Kiang, and Y. Nogami, American Journal of Physics 55, 737 (1987).
  * Ilan et al. (2012) R. Ilan, J. Cayssol, J. H. Bardarson, and J. E. Moore, Physical review letters 109, 216602 (2012).
  * Xypakis et al. (2020) E. Xypakis, J.-W. Rhim, J. H. Bardarson, and R. Ilan, Physical Review B 101, 045401 (2020).
  * Adak et al. (2020) V. Adak, K. Roychowdhury, and S. Das, Physical Review B 102, 035423 (2020).


# IITP at WAT 2021: System description for English-Hindi Multimodal
Translation Task

Baban Gain
Indian Institute of Technology Patna
Patna, India
<EMAIL_ADDRESS>
Dibyanayan Bandyopadhyay11footnotemark: 1
Indian Institute of Technology Patna
Patna, India
<EMAIL_ADDRESS>
Asif Ekbal
Indian Institute of Technology Patna
Patna, India
<EMAIL_ADDRESS>
Equal contribution

Neural Machine Translation (NMT) is a predominant machine translation
technology nowadays because of its end-to-end trainable flexibility. However,
NMT still struggles to translate properly in low-resource settings
specifically on distant language pairs. One way to overcome this is to use the
information from other modalities if available. The idea is that despite
differences in languages, both the source and target language speakers see the
same thing and the visual representation of both the source and target is the
same, which can positively assist the system. Multimodal information can help
the NMT system to improve the translation by removing ambiguity on some
phrases or words. We participate in the 8th Workshop on Asian Translation (WAT
- 2021) for English-Hindi multimodal translation task and achieve 42.47 and
37.50 BLEU points for Evaluation and Challenge subset, respectively.

##  1 Introduction

Recent progress in neural machine translation (NMT) focuses on translating a
source language into a particular target language. Various methods have been
proposed for this task and most of them deal with the textual data. There are
certain drawbacks while performing machine translation using only textual
datasets.

Human performs translation which is based upon language grounding: our sense
of meaning emerges from interacting with the world. NMT methods do not have
any mechanism to perform language grounding; thus they are devoid of capturing
the true meaning of sentences or phrases while translating them into the other
languages. For example, it needs to translate the word “cricket”, it can get
confused if it is the game cricket or the insect cricket. But the visual
information can clear the ambiguity. Multi-modal translation aims to alleviate
this issue by training an NMT model on textual data along with associated
images to perform language grounding.
This shared task deals with developing multi-modal NMT models for English-
Hindi translation. The choice of languages depends on the following issues:
i).  Hindi is the most spoken language in India and the fourth most spoken
language in the world with 600 million
speakers111https://www.ethnologue.com/guides/ethnologue200. Despite the huge
amount of speakers, suitable resources in Hindi is limited due to the various
factors. ii) Automatic translation of texts from one language to the another
is a difficult task. Specifically, when one or both of them are resource-poor
and distant from each other.
In Multimodal NMT (MNMT), information from the other modalities like audio,
image, video, etc. are used along with text to generate the translation. In
low-resource languages, this is particularly used to improve the low-quality
translations as even though vocabularies, grammar of two languages are
different but their visual representation is the same. There are several
proposed multi-modal methods for translations that exploit the features of the
associated image for better translation. State-of-the-art methods might
achieve better accuracy than the models we used. Our main motivation for using
simplistic models is to demonstrate a proof-of-concept to be used for multi-
modal translation among the resource-poor language pairs. We achieved good
results on both Challenge and Evaluation set in different evaluation metrics
including BLEU, RIBES, AMFM. In subsequent modifications, we aim to develop
our models incorporating several state-of-the-art features. The following
sections describe our processes in greater details.

Dataset | Type | Sentences | Avg length source | Avg length target
---|---|---|---|---
Pre-training | Parallel | 273,885 | 12.33 | 13.36
Train | parallel + multimodal | 28929 | 4.95 | 5.02
Valid | parallel + multimodal | 998 | 4.93 | 4.99
Evaluation | parallel + multimodal | 1595 | 4.92 | 4.92
Challenge | parallel + multimodal | 1400 | 5.85 | 6.17

Table 1: Descriptions of datasets used for our experiments

##  2 Related Works

There have been many attempts to use information other than the source for
better translation. Uni-modal systems include document-level NMT Wang et al.
(2017), sentence-level NMT with contextual information Gain et al. (2021),
etc. Among multimodal systems, Huang et al. (2016) used an object detection
system and extracted local and global image features. Thereafter, they used
those image features as additional inputs to encoder and decoder. Delbrouck
and Dupont (2017) used attention mechanism on visual inputs for the source
hidden states. Lin et al. (2020) used Dynamic Context-guided Capsule Network
Sabour et al. (2017) (DCCN) for iterative extraction of related visual
features.

Multimodal Machine Translation (MMT) for English-Hindi has not been well
explored yet. Dutta Chowdhury et al. (2018) used synthetic data for training.
Furthermore they used multi-modal, attention-based MMT which incorporate
visual features into different parts of both the encoder and the decoder
Calixto and Liu (2017). Sanayai Meetei et al. (2019) used a Recurrent Neural
Network (RNN) based approach achieving BLEU score of 28.45 on Evaluation set
and 12.58 on Challenge set. Laskar et al. (2020) exploited monolingual data
for better translation. Recent works tried to focus on developing unsupervised
model for multi-modal NMT. Su et al. (2018) demonstrated an unsupervised
method based on the language translation cycle consistency loss conditional on
the image. This is done to learn the bidirectional multi-modal translation
simultaneously. Moreover, Su et al. (2021) showed that jointly learning text-
image interaction instead of modeling them separately using attentional
networks is more useful. This result is in line with several state-of-the-art
visual transformer related models, such as VisualBERT (Li et al., 2019),
UNITER (Chen et al., 2019) etc.

##  3 Methodology

###  3.1 Dataset Description

We use Hindi Visual Genome 1.1 dataset Parida et al. (2019)Nakazawa et al.
(2020)Nakazawa et al. (2021) for our experiments. This dataset consists of
28,929 parallel English-Hindi sentence pairs along with the associated images.
Furthermore, we use HindEnCorp dataset for pre-training containing 273K
English-Hindi sentence pairs without images. Statistics of the datasets are
shown in Table 1.

Multimodal dataset consists of an image along with a description of certain
rectangular portion of the image. We are given the coordinates of the portion.
We aim to translate the description with help of the image. An example of
multimodal dataset is given in Figure 1.

Figure 1: An example of multimodal dataset Figure 2: An example of translation
generated by the system. Here, the target is Ek vyakti railgari mein chad raha
hai (A man climbing into train.) The translation by Google NMT system is Train
mein chadta Aadmi (Man climbs into train); whereas our NMT system translates
it as: Ek aadmi ek train mein chadta hai (A man climbs into a train.)

Team | Evaluation | Challenge
---|---|---
BLEU | RIBES | AMFM | BLEU | RIBES | AMFM
Volta | 44.21 | 0.818689 | 0.835480 | 52.02 | 0.854139 | 0.874220
iitp (Ours) | 42.47 | 0.807123 | 0.819720 | 37.50 | 0.790809 | 0.830230
CNLP-NITS | 40.51 | 0.803208 | 0.820980 | 39.28 | 0.792097 | 0.812360
CNLP-NITS | 39.46 | 0.802055 | 0.823270 | 33.57 | 0.754141 | 0.787320
Organizer | 38.63 | 0.767422 | 0.772870 | 20.34 | 0.644230 | 0.669760

Table 2: Details of obtained results by different submissions

###  3.2 Pre-processing

For text data, we lowercase all the utterances. Then, we jointly learn byte-
pair-encoding Sennrich et al. (2016) combining both source and target with a
vocabulary of 10,000. We treat the images by cropping a specified rectangular
portions. This operation is used to discard the portions that do not
contribute much to the translation performance. After we get those cropped-out
images, we use the pre-trained VGG19-bn Simonyan and Zisserman (2015) to
obtain the image representations. We use OpenNMT-py Klein et al. (2017)
framework to perform this step.

###  3.3 Training

We use OpenNMT-py Klein et al. (2017) for our NMT systems. We use
Bidirectional RNN encoder and doubly attentive RNN decoder Calixto et al.
(2017) for our experiments. We train our system in two ways viz. With pre-
training, and Without pre-training.:

  1. 1.

With pre-training We pre-train one of our models on HindEnCorp dataset. This
step does not use any visual features as the dataset used for pre-training is
devoid of any visual features. After pre-training, we fine-tune the pre-
trained model with VisualGenome dataset containing textual and visual
features.

  2. 2.

Without pre-training We do not pre-train the model. We directly fine-tune the
models on VisualGenome dataset which contains both text and associated image.
Consequently, both textual and visual features are used.

Following step is taken into account while doing inference step:
We take the best hypothesis from both the models and filter out any hypothesis
containing <unk>token. Then, we pick the hypothesis with best log-likelihood
during generation.

###  3.4 Hyper-parameters

We set the word embedding size and size of RNN hidden states to 500. We set
the batch size to 40 and train for a maximum 25 epochs. We restrict maximum
source and target sequence length to 50. We use the Adam optimizer Kingma and
Ba (2017) for optimization with $\beta_{1}=0.9$ and $\beta_{2}=0.999$. During
training, we use 0.3 as dropout rate to avoid over-fitting. During generation
of translation, we use 5 as the beam width.

##  4 Experimental Results

We obtain impressive results on our submissions. There are two sets designed
for evaluating our model, i) Evaluation set, ii) Challenge set. We evaluate
our model on both of these test set and tabulate our results in Table 2. We
use different evaluation metrics (BLEU, RIBES, AMFM) to test our model. The
results shown in the table are sorted according to the obtained BLEU scores.
As it can be seen from Table 2, we obtain $42.47$ BLEU points and achieve
second position in terms of BLEU on Evaluation set on multimodal task. Please
refer to Figure 2 for example of translation by our system. We obtain 37.50
BLEU points on Challenge set. One reason for not so good results on Challenge
set could be:

  * •

The challenge test set was created by searching for (particularly) ambiguous
English words based on the embedding similarity and manually selecting those
where the image helps to resolve the ambiguity. Hence, it is difficult to
translate compared to the Evaluation set, which was randomly selected.

  * •

Difference between utterance length during training and testing, i.e. while
average length of Train, Evaluation and Validation set is 5 but average length
of Challenge set is 6.

##  5 Conclusion

We participate in WAT-2021 Multimodal Translation Task for English to Hindi.
We achieve good results on both the Challenge and Evaluation sets achieving
$42.47$ and $37.50$ BLEU points, respectively. We rank second place on
Evaluation set and third place on Challenge set on WAT-2021 Multimodal
Translation Task for English to Hindi. In future, we would like to extend our
work by training with additional monolingual data and better ways to
incorporate multimodal features.

## References

  * Calixto and Liu (2017) Iacer Calixto and Qun Liu. 2017.  Incorporating global visual features into attention-based neural machine translation. In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 992–1003, Copenhagen, Denmark. Association for Computational Linguistics.
  * Calixto et al. (2017) Iacer Calixto, Qun Liu, and Nick Campbell. 2017.  Doubly-attentive decoder for multi-modal neural machine translation.  In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1913–1924, Vancouver, Canada. Association for Computational Linguistics.
  * Chen et al. (2019) Yen-Chun Chen, Linjie Li, Licheng Yu, Ahmed El Kholy, Faisal Ahmed, Zhe Gan, Yu Cheng, and Jingjing Liu. 2019.  UNITER: learning universal image-text representations.  _CoRR_ , abs/1909.11740.
  * Delbrouck and Dupont (2017) Jean-Benoit Delbrouck and Stéphane Dupont. 2017.  Modulating and attending the source image during encoding improves multimodal translation.
  * Dutta Chowdhury et al. (2018) Koel Dutta Chowdhury, Mohammed Hasanuzzaman, and Qun Liu. 2018.  Multimodal neural machine translation for low-resource language pairs using synthetic data.  In _Proceedings of the Workshop on Deep Learning Approaches for Low-Resource NLP_ , pages 33–42, Melbourne. Association for Computational Linguistics.
  * Gain et al. (2021) Baban Gain, Rejwanul Haque, and Asif Ekbal. 2021.  Not all contexts are important: The impact of effective context in conversational neural machine translation.  In _2021 International Joint Conference on Neural Networks (IJCNN)_.
  * Huang et al. (2016) Po-Yao Huang, Frederick Liu, Sz-Rung Shiang, Jean Oh, and Chris Dyer. 2016.  Attention-based multimodal neural machine translation.  In _Proceedings of the First Conference on Machine Translation: Volume 2, Shared Task Papers_ , pages 639–645, Berlin, Germany. Association for Computational Linguistics.
  * Kingma and Ba (2017) Diederik P. Kingma and Jimmy Ba. 2017.  Adam: A method for stochastic optimization.
  * Klein et al. (2017) Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017\.  OpenNMT: Open-source toolkit for neural machine translation.  In _Proceedings of ACL 2017, System Demonstrations_ , pages 67–72, Vancouver, Canada. Association for Computational Linguistics.
  * Laskar et al. (2020) Sahinur Rahman Laskar, Abdullah Faiz Ur Rahman Khilji, Partha Pakray, and Sivaji Bandyopadhyay. 2020.  Multimodal neural machine translation for English to Hindi.  In _Proceedings of the 7th Workshop on Asian Translation_ , pages 109–113, Suzhou, China. Association for Computational Linguistics.
  * Li et al. (2019) Liunian Harold Li, Mark Yatskar, Da Yin, Cho-Jui Hsieh, and Kai-Wei Chang. 2019\.  Visualbert: A simple and performant baseline for vision and language.  _CoRR_ , abs/1908.03557.
  * Lin et al. (2020) Huan Lin, Fandong Meng, Jinsong Su, Yongjing Yin, Zhengyuan Yang, Yubin Ge, Jie Zhou, and Jiebo Luo. 2020.  Dynamic context-guided capsule network for multimodal machine translation.  _Proceedings of the 28th ACM International Conference on Multimedia_.
  * Nakazawa et al. (2021) Toshiaki Nakazawa, Hideki Nakayama, Chenchen Ding, Raj Dabre, Shohei Higashiyama, Hideya Mino, Isao Goto, Win Pa Pa, Anoop Kunchukuttan, Shantipriya Parida, Ondřej Bojar, Chenhui Chu, Akiko Eriguchi, Kaori Abe, and Sadao Oda, Yusuke Kurohashi. 2021.  Overview of the 8th workshop on Asian translation.  In _Proceedings of the 8th Workshop on Asian Translation_ , Bangkok, Thailand. Association for Computational Linguistics.
  * Nakazawa et al. (2020) Toshiaki Nakazawa, Hideki Nakayama, Chenchen Ding, Raj Dabre, Shohei Higashiyama, Hideya Mino, Isao Goto, Win Pa Pa, Anoop Kunchukuttan, Shantipriya Parida, Ondřej Bojar, and Sadao Kurohashi. 2020.  Overview of the 7th workshop on Asian translation.  In _Proceedings of the 7th Workshop on Asian Translation_ , pages 1–44, Suzhou, China. Association for Computational Linguistics.
  * Parida et al. (2019) Shantipriya Parida, Ondřej Bojar, and Satya Ranjan Dash. 2019.  Hindi Visual Genome: A Dataset for Multimodal English-to-Hindi Machine Translation.  _Computación y Sistemas_ , 23(4):1499–1505.  Presented at CICLing 2019, La Rochelle, France.
  * Sabour et al. (2017) Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. 2017.  Dynamic routing between capsules.
  * Sanayai Meetei et al. (2019) Loitongbam Sanayai Meetei, Thoudam Doren Singh, and Sivaji Bandyopadhyay. 2019.  WAT2019: English-Hindi translation on Hindi visual genome dataset.  In _Proceedings of the 6th Workshop on Asian Translation_ , pages 181–188, Hong Kong, China. Association for Computational Linguistics.
  * Sennrich et al. (2016) Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016.  Neural machine translation of rare words with subword units.  In _Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_ , pages 1715–1725, Berlin, Germany. Association for Computational Linguistics.
  * Simonyan and Zisserman (2015) Karen Simonyan and Andrew Zisserman. 2015.  Very deep convolutional networks for large-scale image recognition.
  * Su et al. (2021) Jinsong Su, Jinchang Chen, Hui Jiang, Chulun Zhou, Huan Lin, Yubin Ge, Qingqiang Wu, and Yongxuan Lai. 2021.  Multi-modal neural machine translation with deep semantic interactions.  _Information Sciences_ , 554:47–60.
  * Su et al. (2018) Yuanhang Su, Kai Fan, Nguyen Bach, C.-C. Jay Kuo, and Fei Huang. 2018.  Unsupervised multi-modal neural machine translation.  _CoRR_ , abs/1811.11365.
  * Wang et al. (2017) Longyue Wang, Zhaopeng Tu, Andy Way, and Qun Liu. 2017.  Exploiting cross-sentence context for neural machine translation.  In _Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing_ , pages 2826–2831, Copenhagen, Denmark. Association for Computational Linguistics.