# Origin of the gap in the surface states of the antiferromagnetic topological
insulator

R.S. Akzyanov  Dukhov Research Institute of Automatics, Moscow, 127055 Russia
Moscow Institute of Physics and Technology, Dolgoprudny, Moscow Region, 141700
Russia  Institute for Theoretical and Applied Electrodynamics, Russian Academy
of Sciences, Moscow, 125412 Russia  A.L. Rakhmanov  Dukhov Research Institute
of Automatics, Moscow, 127055 Russia  Moscow Institute of Physics and
Technology, Dolgoprudny, Moscow Region, 141700 Russia  Institute for
Theoretical and Applied Electrodynamics, Russian Academy of Sciences, Moscow,
125412 Russia

###### Abstract

We study the influence of the antiferromagnetic order on the surface states of
topological insulators. We derive an effective Hamiltonian for these states,
taking into account the space structure of the antiferromagnetic ordering. We
obtain a typical (gapless) Dirac Hamiltonian for the surface states if the
surface of the sample is not perturbed. However, a shift in the chemical
potential of the surface layer opens a gap in the spectrum away from Fermi
energy. Such a gap arises only in systems with a finite antiferromagnetic
order. We observe that the gap is robust against the surface disorder. The
obtained results are consistent with the recent experiments and density
functional theory calculations.

##  I Introduction

Magnetic topological insulators (MTIs) are narrow-gap semiconductors that
exhibit a nontrivial band structure along with magnetic order. A prominent
feature of the topological insulators (TIs) is the presence of the surface
states that are robust against disorder. The exchange interaction in the MTIs
breaks the time-reversal symmetry of the system and can open a band gap in the
spectrum of the surface electron states [1, 2]. This significantly
distinguishes MTIs from non-magnetic TIs and makes it possible to observe the
anomalous quantum Hall effect and chiral Majorana states [3, 4].

The magnetic order in the TIs can be introduced either by doping a non-
magnetic TI with magnetic atoms or by synthesis of the stoichiometric TI with
magnetic ions in its crystal structure. The latter approach looks more
promising since it allows to obtain homogeneous samples. The first synthesized
intrinsic MTI was MnBi2Te4 [5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
18], which is currently being intensively studied. This material has a layered
Van-der-Waals structure. Each seven-atom block or layer of MnBi2Te4 can be
schematically written as Te-Bi-Te-Mn-Te-Bi-Te (see Fig. 1). Magnetic ions of
Mn are ferromagnetically ordered within the layer, and the layers are ordered
antiferromagnetically (AFM). The Neel temperature for MnBi2Te4 is 25 K [5, 8],
which is the largest among existing MTIs.

Figure 1: Schematic structure of the antiferromagnetic topological insulator
MnBi2Te4.

In the case of AFM MTIs, it is an open question whether the spectrum of the
surface states has a gap or not. The ARPES measurements give very different
values of the gap from 0 [12] to 100 meV [19]. Such a scattering of the
results can be attributed to the sample quality. Thus, the effect of the
material parameters and the influence of the defects on the electronic gap in
the MTIs are important problems [20]. To address these issues, the ab initio
calculations of the energy spectrum of MnBi2Te4 [21] were performed as well as
analytical analysis based on the effective Hamiltonian of the system in the
${\mathbf{k\cdot p}}$ approximation [22, 23]. The authors of Ref. 22 obtained
an analytical dependence of the surface energy gap on the bulk properties of
the MTI. They conclude that a possible reason for the scattering of the
experimental data could be attributed to the idea that the intralayer
ferromagnetic order becomes much smaller and more localized in real materials.
The density functional theory (DFT) calculations predict that the charged
impurities at the surface increase the surface band gap in MnBi2Te4 [21, 24].

In this work, we study analytically the surface states of the AFM MTI. We
start with the MTI Hamiltonian, in which we explicitly take into account a
spatial variation of the magnetization transversely across the layers of the
material. Then, we calculate an effective Hamiltonian for the surface states.
The obtained Hamiltonian differs from the Hamiltonian derived in Ref. [23]
since it has an extended basis that includes indexes of the AFM-ordered
layers. The effective Hamiltonian for the surface states has a typical Dirac
form, and the spectrum is gapless. This gapless spectrum is protected by the
extended time-reversal symmetry [23]. However, if we break such symmetry by
the perturbation of the surface the gap arises at the Dirac point. We get that
shift the chemical potential of the surface layer open a gap in the spectrum
at the Dirac point due to the AFM ordering. This gap is robust against
disorder. We discuss the consistency of our results with the experiment and
DFT calculations.

##  II Model

We are interested in the electron spectrum of the AFM MTI of the type MnBi2Te4
near the $\Gamma$-point. Following Ref. [23], we start with a low-energy
Hamiltonian in the form

$H_{0}=-\mu+m\sigma_{z}+v(k_{x}s_{x}+k_{y}s_{y})\sigma_{x}+v_{z}k_{z}s_{z}\sigma_{x},$
(1)

where we neglect the terms of the order of $\mathbf{k}^{2}$ and higher. Here
$k_{i}$ are the components of momentum $\mathbf{k}$, $v$ and $v_{z}$ are the
in-plane and transverse components of the Fermi velocity, respectively. The
Pauli matrices $s_{i}$ act in the spin space $(\uparrow,\downarrow)$ and Pauli
matrices $\sigma_{i}$ act in the space of the low-energy orbitals
$\left(|P1^{+}_{z}\rangle,|P2^{-}_{z}\rangle\right)$, where the superscripts
$\pm$ stand for the parity of corresponding states [23]. In the absence of the
spin-orbit coupling, the states $|P1^{+}_{z}\\!\uparrow\\!(\downarrow)\rangle$
are associated with two Bi orbitals, while
$|P2^{-}_{z}\\!\uparrow\\!(\downarrow)\rangle$ are associated with two Te
ones. Further, we will use more descriptive notations, Bi↑(↓) and Te↑(↓), for
the considered low-lying states. The spectrum of Hamiltonian (1) is
$E_{0}=-\mu\pm\sqrt{m^{2}+k^{2}}$, and the physical meaning of $m$ is a gap in
the spectrum in the bulk. The Hamiltonian (1) is equivalent to the Hamiltonian
for a usual TI [1] up to rotation of the basis.

To describe the magnetic ordering, we should add to $H_{0}$ the corresponding
magnetic terms. To take into account the spacial structure of the AFM ordering
explicitly, one has to consider two seven-atom blocks with Mn atoms having
opposite directions of the magnetic moment, which requires extension of the
Gilbert space of Hamiltonian (1) from 4D to 8D. However, the authors of Refs.
[22, 23] made a projection of the space that neglected this feature. Such an
approach allows them to reduce the space of the Hamiltonian from 8D to 4D.

In order to restore information on the spatial structure of the AFM state, we
introduce an additional Gilbert space $t$ that takes into account pairs of the
magnetic layers with opposite polarization of Mn atoms. First, we transform
the kinetic energy term into the Hamiltonian $H_{0}$. This transformation is
$v_{z}k_{z}s_{z}\sigma_{x}\rightarrow v_{z}k_{z}s_{z}\sigma_{x}t_{x}$, where
$t_{x}$ is the Pauli matrix that acts in the $t$ space, since only the
nearest-neighbour hoping between the Bi and Te orbitals is allowed. In
general, the AFM ordering results, first, in the finite magnetization $M_{z}$
of each seven-atom layer and, second, in a spin imbalance between Bi and Te
orbitals $A_{z}$. The corresponding term in the Hamiltonian reads
$H_{m}=M_{z}s_{z}t_{z}+A_{z}s_{z}\sigma_{z}t_{z}$. Thus, the Hamiltonian of
the AFM MTI in the extended space is

$\displaystyle H=$ $\displaystyle-$
$\displaystyle\mu+m\sigma_{z}+v(k_{x}s_{x}+k_{y}s_{y})\sigma_{x}$ (2)
$\displaystyle+$ $\displaystyle
v_{z}k_{z}s_{z}\sigma_{x}t_{x}+M_{z}s_{z}t_{z}+A_{z}s_{z}\sigma_{z}t_{z}.$

##  III Surface states

We assume that the sample occupies the space $z<0$ and the surface lies in the
$(x,y)$ plane at $z=0$. We replace $k_{z}$ by the operator $-i\partial_{z}$ in
the Hamiltonian ($\hbar=1$) and $k_{x},k_{y}=0$ in the main approximation in
$\mathbf{k}$. As a result, the surface states $\Psi(z)$ obey the equation

$\displaystyle\left[m\sigma_{z}\\!+\\!s_{z}(M_{z}t_{z}\\!+\\!A_{z}\sigma_{z}t_{z}\\!-\\!iv_{z}\sigma_{x}t_{x}\partial_{z})\\!-\\!\mu\right]\Psi=E\Psi,$
(3)

where $E$ is the energy. We choose $E=0$ since we are interested in the states
near the Dirac point. We seek the solution to the problem as an eighth-
component spinor

$\Psi=(\textrm{Bi}_{\uparrow 1},\textrm{Bi}_{\downarrow
1},\textrm{Te}_{\uparrow 1},\textrm{Te}_{\downarrow 1},\textrm{Bi}_{\uparrow
2},\textrm{Bi}_{\downarrow 2},\textrm{Te}_{\uparrow 2},\textrm{Te}_{\downarrow
2}).$ (4)

Each component of the spinor is characterized by three quantum numbers:
orbital index $\sigma=\textrm{Bi,\,Te}$, spin projection
$s=\uparrow,\downarrow$ and magnetic layer number $t=1,2$. In addition, the
normalization condition $\int_{0}^{+\infty}|\Psi(z)|^{2}dz=1$ should be hold.

Now, we need a boundary conditions. A uniform boundary condition $\Psi(0)=0$
for TIs was suggested in Ref. 1. However, in the linear approximation in
$\partial_{z}$ such a problem has only a trivial solution. In Refs. [25, 26]
the authors noted that the Van der Waals system Bi2Se3 is naturally cleaved in
between two five-layer unit cells. This allows to formulate an appropriate
boundary condition. Evidently, a similar situation realizes for MnBi2Te4, but
now we have the seven-layer unit and two AFM-ordered blocks. As a result, in
our matrix notations the boundary conditions can be presented in the form

$(1+\sigma_{x})(1+t_{z})\Psi(0)=0,\quad\Psi(+\infty)=0.$ (5)

Physically, this condition implies that only one magnetic layer and only one
orbital on a proper basis reach the surface. Linear equations (3) along with
the boundary conditions Eq. (5) and normalization conditions form a complete
system of equations that allow us to calculate the spinor $\Psi$ for the
surface states. We solve this linear problem and obtain two linearly
independent solutions that form a 2D space. It is convenient to introduce an
orthonormal basis $(\Psi_{1},\Psi_{2})$ in this space, and any solution $\Psi$
is a linear combination of $\Psi_{1}$ and $\Psi_{2}$. We choose $\Psi_{1}$ and
$\Psi_{2}$ in the form

$\displaystyle\Psi_{1}$ $\displaystyle=$
$\displaystyle\textrm{Bi}^{(0)}_{\uparrow
1}\left(e^{\lambda_{1}z},0,-e^{\lambda_{2}z},0,-i\sqrt{\frac{M_{z}-m-A_{z}-\mu}{M_{z}-m+A_{z}+\mu}}e^{\lambda_{2}z},0,i\sqrt{\frac{M_{z}+m+A_{z}-\mu}{M_{z}+m-A_{z}+\mu}}e^{\lambda_{1}z},0\right),$
(6) $\displaystyle\Psi_{2}$ $\displaystyle=$
$\displaystyle\textrm{Bi}^{(0)}_{\downarrow
1}\left(0,e^{\lambda_{2}z},0,-e^{\lambda_{1}z},0,-i\sqrt{\frac{M_{z}+m-A_{z}+\mu}{M_{z}+m+A_{z}-\mu}}e^{\lambda_{1}z},0,i\sqrt{\frac{M_{z}-m+A_{z}+\mu}{M_{z}-m-A_{z}-\mu}}e^{\lambda_{2}z}\right),$

where

$\displaystyle\textrm{Bi}^{(0)}_{\uparrow 1}\\!$ $\displaystyle=$
$\displaystyle\\!\left[\frac{M_{z}+m}{|\lambda_{1}|(M_{z}\\!+\\!m\\!-\\!A_{z}\\!+\\!\mu)}\\!\\!+\\!\\!\frac{M_{z}-m}{|\lambda_{2}|(M_{z}\\!-\\!m\\!+\\!A_{z}\\!+\\!\mu)}\\!\right]^{-\frac{1}{2}},$
$\displaystyle\textrm{Bi}^{(0)}_{\downarrow 1}$ $\displaystyle=$
$\displaystyle\\!\left[\frac{M_{z}+m}{|\lambda_{1}|(M_{z}\\!+\\!m\\!+\\!A_{z}\\!-\\!\mu)}\\!\\!+\\!\\!\frac{M_{z}-m}{|\lambda_{2}|(M_{z}\\!-\\!m\\!-\\!A_{z}\\!-\\!\mu)}\right]^{-\frac{1}{2}},$
$\displaystyle\lambda_{1,2}$ $\displaystyle=$
$\displaystyle-\frac{1}{v_{z}}\sqrt{(M_{z}\pm m)^{2}-(A_{z}\mp\mu)^{2}}.$ (7)

The surface states exist if Re$\lambda_{i}<0$ for both $i=1,2$. This
restriction imposes conditions on the values of the parameters under which the
surface states can exist.

##  IV Effective Hamiltonian of the surface states

To derive an effective surface Hamiltonian $H_{\textrm{s}}$ we make a
projection of the Hamiltonian (2) on the basis vectors Eqs. (6),
$H_{\textrm{s}}=\langle\Psi_{i}|H|\Psi_{j}\rangle$. After integration over
$z$, and considering $v(k_{x}s_{x}+k_{y}s_{y})\sigma_{x}$ as a perturbation we
obtain a typical Dirac-like Hamiltonian for the surface states

$\displaystyle H_{\textrm{s}}$ $\displaystyle=$
$\displaystyle\tilde{v}(k_{x}\hat{s}_{x}+k_{y}\hat{s}_{y}),$
$\displaystyle\tilde{v}$ $\displaystyle=$ $\displaystyle
v\frac{\textrm{Bi}^{(0)}_{\uparrow 1}\textrm{Bi}^{(0)}_{\downarrow
1}(|\lambda_{1}|+|\lambda_{2}|)}{|\lambda_{1}\lambda_{2}|},$ (8)

where $\hat{s}$ are the Pauli matrices in the space of the vectors $\Psi_{1}$
and $\Psi_{2}$. The Hamiltonian has a linear gapless spectrum
$E=\pm\tilde{v}\sqrt{k_{x}^{2}+k_{y}^{2}}$. If we assume that the bulk gap is
large, $m\gg|M_{z}|,|A_{z}|$, we derive
$\tilde{v}=v(1-\mu^{2}/m^{2})+O(M_{z}^{2})+O(A_{z}^{2})$.

The wave function $\Psi_{1}$ corresponds to the orbitals with the real spin
projection $s_{z}=\uparrow$, while $\Psi_{2}$ corresponds to
$s_{z}=\downarrow$, see Eqs. (6). In addition, in our orthonormalized basis
$\hat{s}_{\alpha}\propto\langle\Psi_{i}|s_{\alpha}|\Psi_{j}\rangle$.
Therefore, we can consider the Pauli matrices in the space of the surface
states $\hat{s}$ as real-spin operators. Note that a similar result for the
surface states was obtained in Ref. 1.

Up to this point, we see that taking into account the spatial AFM ordering
effects only quantitatively the surface states as compared with the case
$M_{z}=A_{z}=0$, which ignores this AFM structure [22, 23]. This result is not
surprising: in both cases, the system has the same symmetries except for the
time-reversal symmetry. However, the AFM TI has emergent time-reversal-like
symmetry in the extended space. Therefore, we need to break symmetry between
layers 1 and 2 with different polarizations of Mn atoms to observe a
qualitatively new result. The simplest way to break this symmetry is to
introduce a difference of $-\mu_{\textrm{s}}$ in the chemical potential
between layers 1 and 2 due to surface doping. For simplicity, we assume that
the chemical potentials in the bulk and in the second seven-atom block are the
same. We introduce the operator of the surface chemical potential

$\hat{\mu}=-\mu_{\textrm{s}}\frac{\hat{1}+t_{z}}{2},$ (9)

where factor $(1+t_{z})/2$ selects layer 1 as the surface termination. On the
basis of the surface states, we have

$\displaystyle\hat{\mu}$ $\displaystyle=$
$\displaystyle-\tilde{\mu}_{\textrm{s}}(1+\delta\hat{s}_{z}),$ (10)
$\displaystyle\tilde{\mu}_{\textrm{s}}$ $\displaystyle=$
$\displaystyle\mu_{\textrm{s}}\frac{\left(\textrm{Bi}_{\uparrow
1}^{(0)2}+\textrm{Bi}_{\downarrow
1}^{(0)2}\right)(|\lambda_{1}|\\!+\\!|\lambda_{2}|)}{2\sqrt{2}|\lambda_{1}\lambda_{2}|},$
$\displaystyle\delta$ $\displaystyle=$
$\displaystyle\frac{\textrm{Bi}_{\uparrow 1}^{(0)2}-\textrm{Bi}_{\downarrow
1}^{(0)2}}{\textrm{Bi}_{\uparrow 1}^{(0)2}+\textrm{Bi}_{\downarrow
1}^{(0)2}}.$

When $m\gg|M_{z}|,|A_{z}|$ we get
$\delta=M_{z}\mu/m^{2}(1+\mu^{2}/2m^{2})+A_{z}/m(1/2+\mu^{2}/m^{2})$. We can
see that the shift of the surface chemical potential brings the term
$\propto\hat{s}_{z}$ that opens a gap in the spectrum of the surface states of
the topological insulator. The effective Hamiltonian (IV) with the surface
perturbation now reads

$H=\tilde{v}(k_{x}\hat{s}_{x}+k_{y}\hat{s}_{y})-{\tilde{\mu}}_{\textrm{s}}(1+\delta\hat{s}_{z}).$
(11)

The spectrum of this Hamiltonian is

$E_{\pm}=-{\tilde{\mu}}_{\textrm{s}}\pm\sqrt{\tilde{v}^{2}k^{2}+{\tilde{\mu}}_{\textrm{s}}^{2}\delta^{2}}.$
(12)

We plot in Fig. 2 (right panel) the parameter $\delta$ that controls the gap
in the surface spectrum as a function of the AFM magnetization $M_{z}$ for
different values of the bulk chemical potential $\mu$ and $A_{z}$. We see that
$\delta\propto M_{z}$, $\mu$ controls the slope of this line, and $A_{z}$
shifts $\delta(M_{z})$ from the origin. The larger $\mu$ and $M_{z}$, the
larger the surface electron gap. The spectrum $E(\mathbf{k})$, Eq. (12), is
shown in Fig. 2 (left panel) for different values of
${\tilde{\mu}}_{\textrm{s}}$. When we ignore the AFM order, $M_{z}=A_{z}=0$,
the gap vanishes since $\delta=0$ in this case.

Figure 2: Left panel: energy spectrum as a function of the momentum $k_{x}$
and $k_{y}=0$. We take $\mu=0.5m$, $M_{z}=0.1m$, $A_{z}=0$. Spectrum have a
gap if $\mu_{s}\neq 0$. Right panel: parameter $\delta$ as a function of the
AFM magnetization $M_{z}$.

##  V Effects of the disorder

An important question is the stability of the surface gap against disorder. We
apply the Hamiltonian given by Eq. (11) to address it. Note that in the case
of a magnetization-induced gap in the surface states of non-magnetic TI, the
strong disorder suppresses the gap [27].

We consider a short-range disorder near the sample surface of the AFM MTI
produced by randomly distributed charged point defects. We denote 2D density
of the point defects as $n$ and the local impurity potential at the position
$\mathbf{r}=\mathbf{R}_{j}$ as $u_{j}$. By analogy with Eq. (10), the operator
of the disorder potential in the basis of the surface states has a form
$\hat{U}=\sum_{j}\hat{U}_{j}$, where $\hat{U}_{j}(\mathbf{r})=(\hat{1}+\delta
t_{z})u_{j}\delta(\mathbf{r}-\mathbf{R}_{j})/2$ and $\delta(\mathbf{r})$ is
the delta function. We assume that the disorder is Gaussian, that is,
$\langle\hat{U}_{j}(\mathbf{r})\rangle=0$ and
$\langle\hat{U}_{i}(\mathbf{r}_{1})\hat{U}_{j}(\mathbf{r}_{2})\rangle=nu_{0}^{2}\delta(\mathbf{r}_{1}-\mathbf{r}_{2})\delta_{ij}$,
where $\langle...\rangle$ means the spatial average, $u_{0}^{2}=\langle
u_{i}^{2}\rangle$, and $\delta_{ij}$ is the Kronecker symbol.

We assume that the disorder is weak, that is,
$j=nu_{0}^{2}/(2\pi\tilde{v}^{2})<1$ and, following a standard procedure,
calculate the self-energy in the Born approximation. As a result, we obtain in
the n-th order a recursive Born series

$\begin{cases}\hat{\Sigma}^{(n+1)}=\sum\limits_{k,i}\langle\hat{U}_{i}G(\hat{\Sigma}^{(n)})\hat{U}_{i}\rangle,\\\
G^{-1}(\hat{\Sigma}^{(n)})=-H-\hat{\Sigma}^{(n)}.\end{cases}$ (13)

If this procedure converges and $\hat{\Sigma}^{(n)}\rightarrow\hat{\Sigma}$
for $n\rightarrow+\infty$, then the sum of the series can be represented as a
self-consistent Born approximation (SCBA) solution:
$\hat{\Sigma}=\sum_{k,i}\langle\hat{U}_{i}G(\hat{\Sigma})\hat{U}_{i}\rangle$.

We obtain that in the AFM MTI, the self-energy of the disorder has a non-
trivial spin structure $\hat{\Sigma}=\Sigma_{0}+\Sigma_{z}s_{z}$. We perform a
transformation $\Sigma_{0}=g_{0}+\delta g_{z},\Sigma_{z}=\delta g_{0}+g_{z}$,
and after integration over momentum and summation over $i$ derive from Eqs.
(13)

$\displaystyle g_{0}^{(n+1)}\\!$ $\displaystyle=$
$\displaystyle\\!\\!j(1-\delta^{2})\frac{{\tilde{\mu}}_{\textrm{s}}-g_{0}^{(n)}}{2}\Xi^{(n)},$
(14) $\displaystyle g_{z}^{(n+1)}\\!$ $\displaystyle=$
$\displaystyle\\!\frac{1}{2}\;j(1-\delta^{2})g_{z}^{(n)}\Xi^{(n)},$
$\displaystyle\Xi^{(n)}=$ $\displaystyle\ln$
$\displaystyle\frac{\tilde{v}^{2}k_{c}^{2}}{(\delta^{2}-1)\left[\left({\tilde{\mu}}_{\textrm{s}}-g_{0}^{(n)}\right)^{2}-g_{z}^{2(n)}\right]},$
(15)

where $k_{c}$ is a cutoff momentum. The obtained result is equivalent to the
self-energy equations for the Dirac Hamiltonian without magnetization [27].
Following Ref. [27], we take $g_{0}^{(0)}=-i0$ and $g_{z}^{(0)}=0$, which
implies that $g_{z}^{(n)}=0$ and $\Sigma_{z}^{(n)}=\delta\Sigma_{0}^{(n)}$. We
plot self-energy components in Fig. 3. We see that the disorder generates a
real part of the self-energy.

The main effect of the disorder is the increase of the surface chemical
potential
${\tilde{\mu}}_{\textrm{s}}\rightarrow{\tilde{\mu}}_{\textrm{s}}-\textrm{Re}\,\Sigma_{0}$,
which gives rise to the increase of the gap
$\delta{\tilde{\mu}}_{\textrm{s}}$.

Figure 3: Upper figure: the value of the renormalized gap
$\delta_{\textrm{eff}}=\delta(\mu_{s}-\textrm{Re}\,\Sigma_{0})/\delta\mu_{s}$
as a function of the disorder for different values of the surface chemical
potential $\mu_{s}$. Lower figure: the self-energy components as a function of
the disorder strength $j$ for $\mu_{s}=0.1vk_{c}$. We take $n=5000$,
$\mu=0.5m$, $M_{z}=0.1m$,$A_{z}=0$ for both figures.

##  VI Discussion

In this work, we investigate the effects of the antiferromagnetic ordering on
the properties of the surface states in the topological insulator. We get that
the change in chemical potential at the surface layer opens a gap in the
spectrum away from Fermi energy due to the finite AFM order. Such a gap is
controlled by the bulk chemical potential and the value of the surface doping.
This gap is robust against disorder. We should note that in a real samples the
effect of the surface defects can be more significant than just density
disorder and can lead to quite complex picture [28].

In the presence of the AFM order, the time-reversal symmetry is broken.
Instead, a combined time-reversal-like symmetry is present that includes
translation between spin-up and spin-down AFM layers. [23]. Such symmetry
protects gapless surface states. Unlike real time-reversal symmetry, this
combined symmetry can be broken by the non-magnetic perturbation. If the
perturbation breaks symmetry between spin-up and spin-down regions, then a gap
in the spectrum of the surface states arises, see Fig. 2.

Recent experiments with MnBi2Te4 reveal that the doping of the first surface
layer by Ge increases the gap in the surface states [29]. Also, DFT
calculations performed in Ref. [24] show that the surface potential opens a
gap in the surface states in the AFM MTI. Our results are consistent with
these data. Surface doping is an effective tool that can allow us to achieve a
large surface gap in the AFM MTIs.

## Acknowledgments

This work is supported by Russian Science Foundation (project № 22-72-10074).

## References

  * Liu  _et al._ [2010] C.-X. Liu, X.-L. Qi, H. Zhang, X. Dai, Z. Fang, and S.-C. Zhang, Model hamiltonian for topological insulators, Phys. Rev. B 82, 045122 (2010).
  * Chang  _et al._ [2023] C.-Z. Chang, C.-X. Liu, and A. H. MacDonald, Colloquium: Quantum anomalous hall effect, Rev. Mod. Phys. 95, 011002 (2023).
  * He [2020] K. He, Mnbi2te4-family intrinsic magnetic topological materials, npj Quantum Materials 5, 90 (2020).
  * Li  _et al._ [2023] S. Li, T. Liu, C. Liu, Y. Wang, H.-Z. Lu, and X. C. Xie, Progress on antiferromagnetic topological insulator MnBi2Te4, National Science Review , nwac296 (2023), https://academic.oup.com/nsr/advance-article-pdf/doi/10.1093/nsr/nwac296/48483824/nwac296.pdf .
  * Otrokov  _et al._ [2019] M. M. Otrokov, I. I. Klimovskikh, H. Bentmann, D. Estyunin, A. Zeugner, Z. S. Aliev, S. Gaß, A. Wolter, A. Koroleva, A. M. Shikin, _et al._ , Prediction and observation of an antiferromagnetic topological insulator, Nature 576, 416 (2019).
  * Zhang  _et al._ [2019a] D. Zhang, M. Shi, T. Zhu, D. Xing, H. Zhang, and J. Wang, Topological axion states in the magnetic insulator mnbi 2 te 4 with the quantized magnetoelectric effect, Physical review letters 122, 206401 (2019a).
  * Gong  _et al._ [2019] Y. Gong, J. Guo, J. Li, K. Zhu, M. Liao, X. Liu, Q. Zhang, L. Gu, L. Tang, X. Feng, _et al._ , Experimental realization of an intrinsic magnetic topological insulator, Chinese Physics Letters 36, 076801 (2019).
  * Lee  _et al._ [2019] S. H. Lee, Y. Zhu, Y. Wang, L. Miao, T. Pillsbury, H. Yi, S. Kempinger, J. Hu, C. A. Heikes, P. Quarterman, W. Ratcliff, J. A. Borchers, H. Zhang, X. Ke, D. Graf, N. Alem, C.-Z. Chang, N. Samarth, and Z. Mao, Spin scattering and noncollinear spin structure-induced intrinsic anomalous hall effect in antiferromagnetic topological insulator $\mathrm{MnB}{\mathrm{i}}_{2}\mathrm{T}{\mathrm{e}}_{4}$, Phys. Rev. Res. 1, 012011 (2019).
  * Aliev  _et al._ [2019] Z. S. Aliev, I. R. Amiraslanov, D. I. Nasonova, A. V. Shevelkov, N. A. Abdullayev, Z. A. Jahangirli, E. N. Orujlu, M. M. Otrokov, N. T. Mamedov, M. B. Babanly, _et al._ , Novel ternary layered manganese bismuth tellurides of the mnte-bi2te3 system: Synthesis and crystal structure, Journal of Alloys and Compounds 789, 443 (2019).
  * Hao  _et al._ [2019] Y.-J. Hao, P. Liu, Y. Feng, X.-M. Ma, E. F. Schwier, M. Arita, S. Kumar, C. Hu, M. Zeng, Y. Wang, _et al._ , Gapless surface dirac cone in antiferromagnetic topological insulator mnbi 2 te 4, Physical Review X 9, 041038 (2019).
  * Chen  _et al._ [2019] Y. Chen, L. Xu, J. Li, Y. Li, H. Wang, C. Zhang, H. Li, Y. Wu, A. Liang, C. Chen, _et al._ , Topological electronic structure and its temperature evolution in antiferromagnetic topological insulator mnbi 2 te 4, Physical Review X 9, 041040 (2019).
  * Swatek  _et al._ [2020] P. Swatek, Y. Wu, L.-L. Wang, K. Lee, B. Schrunk, J. Yan, and A. Kaminski, Gapless dirac surface states in the antiferromagnetic topological insulator mnbi 2 te 4, Physical Review B 101, 161109 (2020).
  * Liang  _et al._ [2022] A. Liang, C. Chen, H. Zheng, W. Xia, K. Huang, L. Wei, H. Yang, Y. Chen, X. Zhang, X. Xu, M. Wang, Y. Guo, L. Yang, Z. Liu, and Y. Chen, Approaching a minimal topological electronic structure in antiferromagnetic topological insulator mnbi2te4 via surface modification, Nano Lett. 22, 4307 (2022).
  * Qiu  _et al._ [2023] J.-X. Qiu, C. Tzschaschel, J. Ahn, A. Gao, H. Li, X.-Y. Zhang, B. Ghosh, C. Hu, Y.-X. Wang, Y.-F. Liu, D. Bérubé, T. Dinh, Z. Gong, S.-W. Lien, S.-C. Ho, B. Singh, K. Watanabe, T. Taniguchi, D. C. Bell, H.-Z. Lu, A. Bansil, H. Lin, T.-R. Chang, B. B. Zhou, Q. Ma, A. Vishwanath, N. Ni, and S.-Y. Xu, Axion optical induction of antiferromagnetic order, Nature Materials 22, 583 (2023).
  * Gao  _et al._ [2023] A. Gao, Y.-F. Liu, J.-X. Qiu, B. Ghosh, T. V. Trevisan, Y. Onishi, C. Hu, T. Qian, H.-J. Tien, S.-W. Chen, M. Huang, D. Bérubé, H. Li, C. Tzschaschel, T. Dinh, Z. Sun, S.-C. Ho, S.-W. Lien, B. Singh, K. Watanabe, T. Taniguchi, D. C. Bell, H. Lin, T.-R. Chang, C. R. Du, A. Bansil, L. Fu, N. Ni, P. P. Orth, Q. Ma, and S.-Y. Xu, Quantum metric nonlinear hall effect in a topological antiferromagnetic heterostructure, Science 381, 181 (2023).
  * Wang  _et al._ [2023] N. Wang, D. Kaplan, Z. Zhang, T. Holder, N. Cao, A. Wang, X. Zhou, F. Zhou, Z. Jiang, C. Zhang, S. Ru, H. Cai, K. Watanabe, T. Taniguchi, B. Yan, and W. Gao, Quantum-metric-induced nonlinear transport in a topological antiferromagnet, Nature 10.1038/s41586-023-06363-3 (2023).
  * Chen  _et al._ [2023] P. Chen, Q. Yao, J. Xu, Q. Sun, A. J. Grutter, P. Quarterman, P. P. Balakrishnan, C. J. Kinane, A. J. Caruana, S. Langridge, A. Li, B. Achinuq, E. Heppell, Y. Ji, S. Liu, B. Cui, J. Liu, P. Huang, Z. Liu, G. Yu, F. Xiu, T. Hesjedal, J. Zou, X. Han, H. Zhang, Y. Yang, and X. Kou, Tailoring the magnetic exchange interaction in mnbi2te4 superlattices via the intercalation of ferromagnetic layers, Nature Electronics 6, 18 (2023).
  * Cao  _et al._ [2023] T. Cao, D.-F. Shao, K. Huang, G. Gurung, and E. Y. Tsymbal, Switchable anomalous hall effects in polar-stacked 2d antiferromagnet mnbi2te4, Nano Lett. 23, 3781 (2023).
  * Zeugner  _et al._ [2019] A. Zeugner, F. Nietschke, A. U. Wolter, S. Gaß, R. C. Vidal, T. R. Peixoto, D. Pohl, C. Damm, A. Lubk, R. Hentrich, _et al._ , Chemical aspects of the candidate antiferromagnetic topological insulator mnbi2te4, Chemistry of Materials 31, 2795 (2019).
  * Garnica  _et al._ [2022] M. Garnica, M. M. Otrokov, P. C. Aguilar, I. I. Klimovskikh, D. Estyunin, Z. S. Aliev, I. R. Amiraslanov, N. A. Abdullayev, V. N. Zverev, M. B. Babanly, _et al._ , Native point defects and their implications for the dirac point gap at mnbi2te4 (0001), npj Quantum Materials 7, 7 (2022).
  * Shikin  _et al._ [2020] A. M. Shikin, D. Estyunin, I. I. Klimovskikh, S. Filnov, E. Schwier, S. Kumar, K. Miyamoto, T. Okuda, A. Kimura, K. Kuroda, _et al._ , Nature of the dirac gap modulation and surface magnetic interaction in axion antiferromagnetic topological insulator mnbi 2 te 4, Scientific Reports 10, 13226 (2020).
  * Sun  _et al._ [2020] H.-P. Sun, C. Wang, S.-B. Zhang, R. Chen, Y. Zhao, C. Liu, Q. Liu, C. Chen, H.-Z. Lu, and X. Xie, Analytical solution for the surface states of the antiferromagnetic topological insulator mnbi 2 te 4, Physical Review B 102, 241406 (2020).
  * Zhang  _et al._ [2019b] D. Zhang, M. Shi, T. Zhu, D. Xing, H. Zhang, and J. Wang, Topological axion states in the magnetic insulator ${\mathrm{mnbi}}_{2}{\mathrm{te}}_{4}$ with the quantized magnetoelectric effect, Phys. Rev. Lett. 122, 206401 (2019b).
  * Shikin  _et al._ [2022] A. Shikin, D. Estyunin, D. Glazkova, S. Fil’nov, and I. Klimovskikh, Electronic and spin structures of intrinsic antiferromagnetic topological insulators of the mnbi2te4 (bi2te3) m family and their magnetic properties (brief review), JETP Letters 115, 213 (2022).
  * Fu and Berg [2010] L. Fu and E. Berg, Odd-parity topological superconductors: Theory and application to ${\mathrm{cu}}_{x}{\mathrm{bi}}_{2}{\mathrm{se}}_{3}$, Phys. Rev. Lett. 105, 097001 (2010).
  * Hsieh and Fu [2012] T. H. Hsieh and L. Fu, Majorana fermions and exotic surface andreev bound states in topological superconductors: Application to ${\mathrm{cu}}_{x}{\mathrm{bi}}_{2}{\mathrm{se}}_{3}$, Phys. Rev. Lett. 108, 107005 (2012).
  * Akzyanov [2023] R. S. Akzyanov, Born approximation study of the strong disorder in magnetized surface states of a topological insulator, Phys. Rev. B 107, 205416 (2023).
  * Tan and Yan [2023] H. Tan and B. Yan, Distinct magnetic gaps between antiferromagnetic and ferromagnetic orders driven by surface defects in the topological magnet ${\mathrm{mnbi}}_{2}{\mathrm{te}}_{4}$, Phys. Rev. Lett. 130, 126702 (2023).
  * Estyunina  _et al._ [2023] T. P. Estyunina, A. M. Shikin, D. A. Estyunin, A. V. Eryzhenkov, I. I. Klimovskikh, K. A. Bokai, V. A. Golyashov, K. A. Kokh, O. E. Tereshchenko, S. Kumar, K. Shimada, and A. V. Tarasov, Evolution of mn1-xgexbi2te4 electronic structure under variation of ge content, Nanomaterials 13, 10.3390/nano13142151 (2023).


# Investigating Cross-Domain Behaviors of BERT in Review Understanding

Albert Lu
University of Notre Dame
Notre Dame, IN 46556, USA
<EMAIL_ADDRESS>
Meng Jiang
University of Notre Dame
Notre Dame, IN 46556, USA
<EMAIL_ADDRESS>

###### Abstract

Review score prediction requires review text understanding, a critical real-
world application of natural language processing. Due to dissimilar text
domains in product reviews, a common practice is fine-tuning BERT models upon
reviews of differing domains. However, there has not yet been an empirical
study of cross-domain behaviors of BERT models in the various tasks of product
review understanding. In this project, we investigate text classification BERT
models fine-tuned on single-domain and multi-domain Amazon review data. In our
findings, though single-domain models achieved marginally improved performance
on their corresponding domain compared to multi-domain models, multi-domain
models outperformed single-domain models when evaluated on multi-domain data,
single-domain data the single-domain model was not fine-tuned on, and on
average when considering all tests. Though slight increases in accuracy can be
achieved through single-domain model fine-tuning, computational resources and
costs can be reduced by utilizing multi-domain models that perform well across
domains.

##  1 Introduction

Figure 1: Seven datasets with varying combinations of domains were made, all
of the same size. BERT models were trained upon all datasets and cross-domain
tested. We investigate behaviors where the fine-tuning and testing domains are
not fully aligned.

Online services such as Amazon and Target have review systems where users can
provide comments and ratings for products, offering valuable information to
both buyers and companies. Analyzing the large volume of review data manually
is impractical, so companies use data science and machine learning to extract
knowledge efficiently AlZu’bi et al. (2019). An important task in review
understanding is review score prediction, akin to sentiment analysis, where
the text of a review is used to determine the corresponding star rating.
Review score prediction aids in distinguishing between the sentiment conveyed
by the star rating and the sentiment expressed in the text, addressing
potential biases and ensuring accurate ratings Ul Haque et al. (2018); Ibrahim
Elmurngi and Gherbi (2018).

Bidirectional encoder representations from transformers (BERT) models, a model
set for natural language understanding, can be fine-tuned to predict star
ratings based on review text Devlin et al. (2019). BERT utilizes a transformer
architecture, processing an entire text sequence at once, capturing contextual
relationships between words. BERT consists of stacked transformer encoder
layers with self-attention mechanisms and is pre-trained on two tasks: masked
language modeling and next sentence prediction. After general pre-training,
BERT is fine-tuned for specific downstream tasks by adding task-specific
layers and adjusting neural network weights based on labeled data Devlin et
al. (2019), where rating score prediction is one of the popular downstream
tasks.

The existence of multiple domains poses a significant challenge for review
score prediction. Platforms like Amazon have diverse product categories,
requiring models to analyze reviews for each domain separately. Though cross-
domain models have been created, it remains unclear whether a single model can
perform optimally across all domains Liu et al. (2021); Al-Moslmi et al.
(2017). To address this, BERT can be fine-tuned using training sets specific
to a domain or a mixture of training sets from various domains Sushil et al.
(2021). Fine-tuning with domain-specific data allows models to focus on
predicting the relevant data they encounter, avoiding the counterproductive
task of predicting irrelevant domains. However, training individual domain-
specific models increases resource use compared to training a single model for
multiple domains, assuming the multi-domain model performs well in different
domains. To understand the trade-offs between a specific versus general
approach for multiple domains, an empirical study analyzing BERT’s cross-
domain behaviors in review understanding is necessary.

We use data from three Amazon review domains to form seven datasets, fine-
tuning from them seven BERT models. Each BERT model was evaluated upon every
dataset in a round-robin format. Figure 1 and the Methods section describe our
study design with details about data processing and model fine-tuning.

Below briefly are our observations. Predictions made before experimentation
included that, though it would be possible to fine-tune BERT models to
specialize on a certain domain, the benefit from single-domain fine-tuning
would be minor. Also predicted was that multi-domain models would achieve a
higher performance when evaluated on single-domain datasets compared to
single-domain models not trained upon that specific domain. Observations lined
up with predictions. Though BERT models were able to achieve domain
specificity, the gains in performance towards single domains were slight.
Moreover, the specificity gained appears to have decreased the single-domain
models’ ability to predict on other domains. Multi-domain models outperformed
single-domain models on all categories besides the specific domain the single-
domain models had been fine-tuned on, as well as achieving a higher average
accuracy across all datasets. In addition, multi-domain models performed
better on the multi-domain datasets the models were trained with compared to
the single-domain models and their corresponding single-domain datasets.
Towards applications in review analysis, review score prediction, and
sentiment analysis, it appears that, though marginal performance improvements
can be achieved through single-domain, specific fine-tuning, computational
resources can be saved through training cross-domain models that achieve a
high level of performance for all domains.

##  2 Method of Our Study

Model | Model was evaluated on test sets in: (RMSE: smaller is better) |
---|---|---
(was trained in) | Auto | Garden | Music | A+G | G+M | A+M | A+G+M | Average
Auto | 0.640 | 0.742 | 0.781 | 0.678 | 0.721 | 0.632 | 0.711 | 0.701
Garden | 0.624 | 0.721 | 0.854 | 0.707 | 0.787 | 0.648 | 0.724 | 0.724
Music | 0.860 | 0.781 | 0.728 | 0.860 | 0.714 | 0.806 | 0.783 | 0.790
A+G | 0.721 | 0.707 | 0.794 | 0.686 | 0.643 | 0.632 | 0.663 | 0.691
G+M | 0.714 | 0.700 | 0.794 | 0.686 | 0.640 | 0.632 | 0.661 | 0.689
A+M | 0.748 | 0.806 | 0.707 | 0.735 | 0.693 | 0.616 | 0.731 | 0.719
A+G+M | 0.678 | 0.714 | 0.714 | 0.663 | 0.641 | 0.624 | 0.660 | 0.670
Average | 0.712 | 0.739 | 0.767 | 0.716 | 0.691 | 0.656 | 0.704 | Best avg: 0.659
Table 1: Results from a comprehensive investigation on cross-domain review score prediction using BERT models. Model | Self | All | Specific | Semi-Specific | General
---|---|---|---|---|---
specificity |  |  | Familiar | Unfamiliar | Familiar | Unfamiliar |
Specific | 0.696 | 0.739 | 0.696 | 0.774 | 0.721 | 0.789 | 0.739
General | 0.649 | 0.698 | 0.721 | 0.754 | 0.667 | NA | 0.697
Table 2: Table of average RMSEs of models towards various groups of datasets.

Three Amazon review datasets with mutually dissimilar domains were chosen to
fine-tune BERT with McAuley et al. (2015); He and McAuley (2016): automobile
part review data (Auto); musical instrument review data (Music); and patio,
lawn, and garden review data (Garden). Datasets consisted of examples with an
average review text length of 582 characters and a one-to-five rating.

Our pipeline process from data sampling to model evaluation is shown in Figure
1. Data for model training was processed through downsampling to reduce domain
and class imbalance. Since the original Auto domain data is significantly
larger than the other datasets, 2500 reviews were sampled from each domain for
model fine-tuning, remedying domain imbalance. To reduce class imbalance, for
each domain, the 2500 training reviews consisted of 200, 250, 300, 600, and
1150 one- to five-star reviews, respectively. 800 reviews were randomly
sampled from each domain for both validation and test splits.

Processing resulted in a final training corpus of 7500 reviews, divided evenly
among the three domains. Four more multi-domain datasets were created through
combining examples from various domains. Three two-domain datasets with 2500
train examples each were formed through taking 1250 examples from each
combination of two single-domain datasets. One three-domain dataset of 2500
train examples was formed from an even mixture of examples from all three
domains.

The pre-trained _BERT-base-cased_ model served as the fine-tuning backbone.
Batches of fine-tuned models configured with varying hyperparameters were
trained for each dataset to discover optimal model performance towards a
specific dataset. Four learning rates (5e-5, 4e-5, 3e-5, and 2e-5) and four
batch sizes (32, 64, 128, and 256) were used along with a 20 epoch training
duration and a learning rate warmup ratio of 0.1. High learning rates and
batch sizes were found to increase model performance, and after selecting the
best model from each batch, each of the seven models was evaluated against the
test split of every dataset.

The root mean squared error (RMSE) metric was used to determine the optimal
model in each dataset batch and evaluate model cross-domain testing. As RMSE
is proportional to the difference between true and predicted values, a smaller
RMSE indicates higher performance.

Models were trained and tested on a GPU server of four NVIDIA GeForce RTX 2080
Ti cards, managed by the Center for Research Computing at the University of
Notre Dame.

##  3 Results and Analysis

###  3.1 Model-level results and analysis

The results of testing each model against each dataset are shown in Table 1.
Every row of the table represents one model being tested on all seven
datasets; every column represents one dataset having all seven models
evaluated upon it. Performance of the models was measured using Root Mean
Squared Error (RMSE). A smaller RMSE value indicates higher model accuracy.
Bold table values represent the highest performance achieved in each column.
The “Average” across each column and each row indicates the mean score for
that column or row. Considering the horizontal row of averages, higher RMSE
values indicate a more difficult dataset. Considering the vertical average
column, lower RMSE values mean a better-performing model. The “Best avg” in
the bottom right corner is found by taking the best (lowest) RMSE for each
test set and calculating the mean of those values. 0.659, the best average, is
quite good considering the rating score ranges from one to five. Within-domain
performance (Auto model with Auto data) is indicated on the table through
underlining. The bottom left cells contain the RMSEs of multi-domain models
tested on single-domain data, while in the top right the RMSEs of single-
domain models on multi-domain data are shown. Red cells mean a higher RMSE
than the underlined baseline value of the column; blue cells indicate the
inverse. Our observations are as follows.

(1) Multi-domain models perform better than single-domain ones averaged across
single- and multi-domain datasets. The three-domain A+M+G model achieved the
lowest RMSE, 0.670, averaged across all seven datasets. This matches intuition
- though single-domain models were expected to and did perform better on
certain specific domains, overall the more cross-domain a model was the lower
the error across all domains. This appears to be due to the multi-domain
models having received exposure to data from varied domains, and so having a
“background” to predict decently well on music, auto, and garden data, whereas
single-domain models, though having high performance on the single domain it
was fine-tuned upon, lack knowledge of domains outside its “scope,” and so
performed poorly on those. It seems that the decreases in performance for
single-domain models on unfamiliar domains outweighed the high accuracy on the
sole familiar domain, causing single-domain models to be inferior to multi-
domain models on average.

(2) Some domains are more difficult for models to predict well upon, perhaps
due to differences in text content and technicality. On average, models when
evaluated on the Music dataset resulted in the worst performance, with a RMSE
of 0.767 averaged across all seven models. Perhaps this was due to the high
musical knowledge of those who purchase musical instruments causing reviews to
contain more difficult vocabulary. One would think automobile and garden
products (on Amazon) would not be often purchased by mechanics or professional
gardeners but by amateurs whose product reviews would not utilize technical
language. Musical instruments, however, are in general more expensive and
niche than a simple spark plug or lawn gnome and so reviews would tend towards
professionals who, instead of merely commenting with “good violin” or “bad
violin” would use terms that may be clear to a human musician but would be
difficult for a BERT model to interpret as positive or negative. The easiest
domain for models seems to be the automobile part data, with an average RMSE
of 0.712. With regards to the Auto data, an outlier presents itself: the
garden model, towards the auto domain, had a higher performance than even the
automobile model itself. Given the uniqueness of this outlier (there is no
other example in the table of a model completely unfamiliar with a domain
outperforming a model trained upon that domain), this unexpected high-
performing result can be treated as a curiosity caused by data variation
between domains, interesting but ultimately unimportant to overall analysis.

(3) Data combining domains allows a wider range of models to predict well upon
it. Looking at the latter four columns of Table 1, it seems that on average
models achieved a lower RMSE when evaluated with multi-domain data (two- and
three-domain data). This should not be taken as a sign that difficult domains
when combined result in an easier dataset. The relative higher performance of
models towards multi-domain data should be taken as more models having
familiarity towards a section of the multi-domain data and thus, overall,
performing better.

(4) Multi-domain models outperformed single-domain models in all multi-domain
data, and can outperform single-domain models in their corresponding single-
domain dataset, perhaps caused by the ability of multi-domain models to adapt
to variations in data. These interesting observations can be found by
considering the blue-and-red shaded top right and bottom left regions of Table
1. First, the top right, single-domain model with multi-domain dataset,
section had only one blue cell - only one model-dataset pair that achieved a
better RMSE than the baseline, underlined value in that column. It seems that,
when considering multi-domain data, no single-domain model can compete with
the multi-domain model. However, through analyzing the bottom left table
quadrant, the inverse turns out to be untrue. Multiple cells are blue;
considering the garden and music datasets, more than half of the multi-domain
model with single-domain dataset tests resulted in a higher performance than
the baseline of the column. It seems that, in the auto column, due to
variations in data domains discussed above, the single-domain auto model was
able to achieve a high enough performance that no multi-domain model was able
to exceed its accuracy. In the garden and music domains, in both cases the
three-domain model outperformed the baseline single-domain model, and in the
garden domain, both of the two-domain models that had garden data as part of
their fine-tuning outperformed the baseline. For the music dataset, the auto
plus music model was the best of its column. This is unintuitive, since one
would expect each single-domain model to have the highest performance towards
its corresponding domain. This may have been caused by subtle effects produced
by the difference in training versus testing data. Undersampled training data
is not representative of a real-world application; however, fine-tuning a
model with class-biased, unprocessed amazon reviews creates a model that only
outputs fives and fours. The single-domain models may have been exposed to
such homogeneous data that the model was overfitted to the processed,
undersampled dataset, and thus could not predict well on the test data. Multi-
domain models may have avoided this issue by heterogeneous, multi-domain data
(still undersampled) allowing the model to be exposed to “variations” in data.
In the training data, the variation is solely a difference in domain; in
evaluation, the testing data both has a difference in domain between examples
and a difference in class bias compared to the training data. Multi-domain
models are more accustomed to variation than single-domain models due to the
nature of multi-domain data; it is not a stretch to deduce that multi-domain
models could better accommodate the differences between training and testing
data.

###  3.2 Specific models vs general models

Table 2 summarizes average RMSEs among specific versus general models for
differing categories of dataset. Specific is defined as single-domain; general
models are the four two- and three-domain models. The “Self” column
corresponds to the underlined values in Table 1 \- the RMSE of each model
tested against its corresponding domain. The “All” column is the average of
the specific or general models’ RMSEs on all datasets. In Table 2, “Specific”,
“Semi-Specific”, and “General” datasets are defined as single-domain, two-
domain, and three-domain datasets respectively. The “Specific” and “Semi-
Specific” columns are divided into two sub-column - “Familiar” and
“Unfamiliar.” Familiar is defined as a model having had any training on any
constituent domain of a dataset - for example, the Auto dataset would be
familiar to both the “Specific” Auto model and the “General” A+G model, but
would be unfamiliar to the M+G model. (The “NA” under “Semi-Specific,
Unfamiliar” is due to all three two-domain datasets being familiar to all two-
and three-domain models, as all pairs of two-domain dataset and two- or three-
domain model share at least one domain.) The two values in the “General”
column are the average of the RMSEs of all three single-domain models tested
on A+M+G and the average of the RMSEs of all four multi-domain models tested
on A+M+G, respectively.

Most of the conclusions found by analyzing the results displayed in Table 1
can also be found in Table 2. Though general models had a lower RMSE than
specific ones across all datasets, single-domain models, on average, still
performed better on their corresponding domain compared to multi-domain models
on both familiar and unfamiliar specifc data; 0.696 compared to 0.721 and
0.754, respectively. However, other interesting conclusions can be found by
further analyzing Table 2.

Domain |  Review Text |  True |  Model |  Predicted
---|---|---|---|---
Music |  I’m another reviewer who has, upon personal inspection, discovered that the rear leg of the stand is too short to provide staedy support for any beloved guitar. I returned it. I’ve First Act brand stands similar in appearance but more intelligently constructed. |  1 |  Music
Auto
A+M |  3
4
2
Garden |  This sprayer is very well built and solid. It is easy to use and with the long hose and 18 inch wand it sprays a very good distance and the spray is of course adjustable to different spray patterns. I am happy with hit. |  5 |  Music
Auto
Garden |  5
5
5
Auto |  Takes the dust off my car without leaving any streaking that some report with other brands. Just don’t press real hard. |  5 |  Garden
Auto
A+G |  3
5
5
Table 3: Examples of reviews from various domains with review text, true
rating, and predicted rating by multiple models, illustrating several
conclusions detailed in the Case Study section.

Despite Table 1 indicating that in some cases multi-domain, general models
could outperform single-domain models, (5) in general multi-domain models
tended to lose accuracy when evaluated on data with a single domain. Compared
to the baseline self-test average RMSE of 0.649, the general models had a
0.072 and 0.105 increase in RMSE when evaluated on familiar and unfamiliar
single-domain data, respectively. A decrease in the performance of multi-
domain models on single-domain data is also apparent when comparing the
“Specific” column with the “Semi-Specific” and “General” columns of Table 2,
specifically the values in the “General” row. The average RMSEs of 0.667 and
0.697 of general models towards two- and three-domain data, respectively, are
superior to both 0.721 and 0.754, the RMSEs of familiar and unfamiliar
specific data, respectively. The loss of performance on unfamiliar data is
expected; however, lower accuracy on familiar single domains is more
interesting. An example of lower accuracy of a multi-domain model on familiar
single-domain data is the A+M model achieving a worse RMSE on the Auto dataset
and the Music dataset compared to the A+M dataset. This lowered performance on
familiar domains in isolation may be caused by multi-domain models, in
general, being unable to fully predict well on single-domain data that may
contain technical vocabulary that, though familiar, has not been completely
fine-tuned to. Single-domain test datasets are also more prone to outliers, at
the data sample size used - for example, a certain subsection of Auto data,
selected for testing, may contain a greater-than-average level of reviews
utilizing technical vocabulary; multi-domain datasets reduce the chances of a
test set with complicated vocabulary the multi-domain model cannot predict
well upon. Regardless of the cause, however, multi-domain models achieved a
relatively lower performance on single-domain data, compared to the self-test
and testing on both two- and three-domain data.

A conclusion made when analyzing Table 1 is further supported by the averages
in Table 2. (6) Specific models, when tested on general data and specific data
not of the model’s domain, exhibited an expected decrease in performance.
Compared to the baseline self-test average of 0.696 among the three single-
domain models, the average RMSEs of the single-domain models tested on
familiar two-domain, unfamiliar two-domain, and three-domain datasets were
0.025, 0.093, and 0.043 higher, respectively. A decrease in performance
towards unfamiliar specific data was also observed, the increase in RMSE being
0.075 there, similar to the 0.093 value mentioned above for the increase in
RMSE with specific models tested on unfamiliar semi-specific data. Note that
the relatively lower performance decreases in familiar two-domain and three-
domain datasets is very probably not caused by Auto or Garden models
predicting better on the Music-domain half of the A+M dataset compared to the
Music data alone. The higher accuracy of the Garden model towards A+G data
versus A+M data is because fifty percent of the former dataset is Garden-
domain, while the Garden domain is not present in the latter dataset. Single-
domain model only perform worse on semi-specific and general datasets because
of the unfamiliar specific data content of the multi-domain datasets.

###  3.3 Case study

Some examples that highlight the conclusions presented above are shown in
Table 3. Echoing points 2, 4, and 7, example 1 shows a review which all models
failed to predict well upon (Music, Auto, and A+M are shown to highlight
various conclusions). This review showcases the varied aspects of Music domain
reviews that may have created difficulty for model prediction - lack of clear
adjectives and a sentiment that requires knowledge of what “guitars", “rear
legs", and “stands" are. The model predictions also support that single-domain
models lose accuracy when tested on a different domain (the Auto model
predicted a 4), and that, in some cases, a two-domain model can outperform a
single-domain one (A+M performance).

Example 2 provides an example of an “easy" review to predict upon. The
presence of “very good" and “happy with hit [sic]" allows all models to
predict well upon the review. It also highlights a point that aids cross-
domain models. As one can see in this example, “simple" adjectives such as
“good" and “well" are present in great amount, a trait common to most reviews.
Models undergoing cross-domain testing can utilize these keywords (Auto and
Music model, which both predicted the true value) to achieve accuracy on
various domains.

Example 3 provides another review that contains domain-specific text with few
“simple" adjectives. The Garden model may have latched onto the “don’t" and
“without", misinterpreting those negative-connotation words as indicative of a
low star rating. Example 3 again shows that single-domain models predict well
on the own dataset, and that multi-domain models with familiarity towards a
certain domain can predict just as well.

#### Summary:

To summarize the conclusions presented through analysis of Table 1 and Table
2:

  * •

On average, multi-domain models perform better than single-domain ones.

  * •

Domains vary in difficulty of prediction.

  * •

Multi-domain data allows for both single- and multi-domain models to predict
well.

  * •

Multi-domain models outperform single-domain models in multi-domain data and
can occasionally outperform single-domain models in the single-domain model’s
corresponding domain.

  * •

Multi-domain models lose accuracy when evaluated upon all single-domain data,
including familiar data.

  * •

Single-domain models lose accuracy when evaluated upon unfamiliar single-
domain data.

  * •

On average, single-domain models achieve higher accuracy towards their domain
compared to multi-domain models.

##  4 Related Work

###  4.1 Domain adaptation and BERT

Work has been done on cross-domain sentiment analysis with BERT Myagmar et al.
(2019). In the real world, the domain-dependent nature of sentiment analysis
often creates issues when the amount of data that can be gathered for a
specific domain is insufficient to properly train a high-performing, single-
domain model. This problem is often solved through domain adaptation, in which
a model is trained upon data from a relevant, secondary domain and deployed
for the primary, data-scarce domain. Existing work has shown that fine-tuned
BERT models outperform state-of-the-art domain adaptation methods Lin et al.
(2020). The multi-domain models in our work show a similar capacity to adapt
to varying domains; however, single-domain models were less capable of
predicting well on unfamiliar domains Wang et al. (2020, 2021).

Another domain-reliant task is automatic speech recognition (ASR), in which it
is common practice to pretrain generic ASR models and then fine-tune the
models to domain-specific data Bell et al. (2020), an approach similar to BERT
pretraining and fine-tuning. Other techniques for ASR domain adaptation
include data augmentation, multi-task learning, teacher-student training,
semi-supervised training, and using synthetic data created by a Text-to-Speech
system Fainberg et al. (2016); Sun et al. (2017); Meng et al. (2019); Synnaeve
et al. (2020); Joshi and Singh (2022). If domain adaptability and the high
performance achieved by single-domain models in our work are required, instead
of multi-domain models the techniques used in ASR could be employed.

###  4.2 Knowledge distillation

Work by Howell et al. on domain-specific knowledge distillation for
conversational commerce is closely related to this work Howell et al. (2022).
Knowledge distillation addresses the issue of large and computationally
expensive models being too costly to be deployed through a "teacher-student"
training approach, in which a trained "teacher" model facilitates the training
of a smaller "student" model Sanh et al. (2019). Howell et al. show knowledge
distillation can both reduce model size and adapt models to a specific domain.
In this work, we show that BERT base models are capable of adapting to a
domain through fine-tuning on domain-specific, instead of knowledge
distillation. However, we also show that, compared to single-domain models,
multi-domain models, though incurring a slight accuracy penalty, still retain
high performance towards a broad range of data. We also propose multi-domain
models as an alternative solution to cost of computation. Multi-domain models
and knowledge distillation could be used in tandem to further reduce
computational resources, using knowledge distillation to train a "student"
model with a broad spectrum of data domains, creating a model that is both
small in size and applicable to multiple domains.

##  5 Conclusion and Future Work

We investigated BERT models fine-tuned on single-domain and multi-domain data
for the task of Amazon review score prediction. Though single-domain models
achieved marginally improved performance on their corresponding domain
compared to multi-domain models, multi-domain models outperformed single-
domain models when evaluated on multi-domain data, single-domain data, and on
average. Though slight increases in accuracy can be achieved through single-
domain model fine-tuning, computational demands and costs can be reduced by
utilizing multi-domain models that perform well across many domains.

###  5.1 Discussion and impact

The primary goal of this work was to determine the tradeoffs of specific
versus general model domain scopes in BERT review score prediction. The
difference between model RMSEs was comparatively low relative to the raw RMSE
values, around 5% for most variations. This may have been caused by the task
of classifying Amazon reviews not requiring much fixation on topic-related
words. Though soap and furniture reviews contain differing nouns, adjectives
for negative and positive soap and furniture reviews would be mostly alike.
The BERT models may have recognized the importance of sentiment-laden
adjectives such as “bad” or “good” that can apply to any topic product when
determining the review’s star rating, placing weight on more general keywords
rather than topic-specific nouns and descriptions. However, review score
prediction still requires a certain level of domain understanding, supported
both by our results and by phrases such as "takes up all my time," which is
negative when referring to furniture setup but positive when applied to books.

The findings in this work support that specific models are still superior to
general models for specific applications. Tasks that come to mind that
specific models would be best suited for include, say, diagnosis of a certain
cancer type through processing a patient’s symptom log or as a search aide for
an online archive containing papers of a narrow breadth. However, these
specific models will necessarily be limited in scope.

Though general models in this predictive task were still capable of achieving
relatively high performance on all domains, they lost accuracy on specific
data, even data with domains that the general model had been exposed to
before. Search engines and chatbots come to mind as tasks that require a
general solution. However, general models often fail to correctly interpret a
specific instruction. For example, a text summarization service may not be
able to interpret a technical text accurately.

A possible solution for the above would be to utilize a general NLP model in
order to determine what domain a certain piece of text falls into – a
relatively simple task that is suited to a broad, general model. The text
could then be sent to a specific model customized to address the particular
domain the general model determined the text fell under. This system, though
resource-intensive, would both allow the general model to perform a task that
does not require the precision of a specific model and prevent the specific
models from encountering any text outside its domain breadth.

###  5.2 Further experimentation

Further experimentation along these lines could include expanding the number
of datasets used in order to minimize the effects outliers or poorly fine-
tuned models have on results. Such experimentation would also be able to
determine how the “interrelatedness” of topics affects BERT and NLP – one
would expect soap and detergent models to be more similar than soap and
furniture models. Single- versus multi-domain testing could also be done upon
tasks requiring NLP models to place weight on topic-specific vocabulary (such
as text summarization or keyword extraction), creating a true “specific”
model, avoiding the problem described in the Discussion section of specific
models fixating on general adjectives. Such investigation would be able to
determine the tradeoffs between general and specific scopes in the broader,
NLP field, and whether the trends detailed in this work regarding review score
prediction with BERT (a narrow subset of NLP) do or do not extend to the rest
of the e-commerce and NLP field.

## Acknowledgement

This work was supported by National Science Foundation (NSF) IIS-2119531,
IIS-2137396, IIS-2142827, IIS-2234058, CCF-1901059, and Office of Naval
Research (ONR) N00014-22-1-2507.

## References

  * Al-Moslmi et al. (2017) Tareq Al-Moslmi, Nazlia Omar, Salwani Abdullah, and Mohammed Albared. 2017.  Approaches to cross-domain sentiment analysis: A systematic literature review.  In _IEEE Access_ , volume 5, pages 16173–16192.
  * AlZu’bi et al. (2019) Shadi AlZu’bi, Abdalraheem Alsmadiv, Sokyna AlQatawneh, Mahmoud Al-Ayyoub, Bilal Hawashin, and Yaser Jararweh. 2019.  A brief analysis of amazon online reviews.  In _2019 Sixth International Conference on Social Networks Analysis, Management and Security (SNAMS)_ , pages 555–560.
  * Bell et al. (2020) Peter Bell, Joachim Fainberg, Klejch Ondrej, Jinyu Li, Steve Renals, and Pawel Swietojanski. 2020.  Adaptation algorithms for neural network-based speech recognition: An overview.  In _IEEE Open Journal of Signal Processing_ , pages 33–66.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  Bert: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of NAACL-HLT_ , pages 4171–4186.
  * Fainberg et al. (2016) Joachim Fainberg, Peter Bell, Mike Lincoln, and Steve Renals. 2016.  Improving children’s speech recognition through out-of-domain data augmentation.  In _INTERSPEECH 2016_ , pages 1598–1602.
  * He and McAuley (2016) Ruining He and Julian McAuley. 2016.  Ups and downs: Modeling the visual evolution of fashion trends with one-class collaborative filtering.  In _proceedings of the 25th international conference on world wide web_ , pages 507–517.
  * Howell et al. (2022) Kristen Howell, Jian Wang, Akshay Hazare, Joseph Bradley, Chris Brew, Xi Chen, Matthew Dunn, Beth Hockey, Andrew Maurer, and Dominic Widdows. 2022.  Domain-specific knowledge distillation yields smaller and better models for conversational commerce.  In _Proceedings of the Fifth Workshop on e-Commerce and NLP (ECNLP 5)_ , pages 151–160.
  * Ibrahim Elmurngi and Gherbi (2018) Elshrif Ibrahim Elmurngi and Abdelouahed Gherbi. 2018.  Unfair reviews detection on amazon reviews using sentiment analysis with supervised learning techniques.  In _Journal of Computer Science_ , pages 714–726.
  * Joshi and Singh (2022) Raviraj Joshi and Anupam Singh. 2022.  A simple baseline for domain adaptation in end to end asr systems using synthetic data.  In _Proceedings of the Fifth Workshop on e-Commerce and NLP (ECNLP 5)_ , pages 217–223.
  * Lin et al. (2020) Chen Lin, Steven Bethard, Dmitriy Dligach, Farig Sadeque, Guergana Savova, and Timothy A Miller. 2020.  Does bert need domain adaptation for clinical negation detection?  In _Journal of the American Medical Informatics Association_ , volume 27, page 584–591.
  * Liu et al. (2021) Jun Liu, Shuang Zheng, Guangxia Xu, and Mingwei Lin. 2021.  Cross-domain sentiment aware word embeddings for review sentiment analysis.  In _International Journal of Machine Learning and Cybernetics_ , volume 12, pages 343–354.
  * McAuley et al. (2015) Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015\.  Image-based recommendations on styles and substitutes.  In _Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval_ , pages 43–52.
  * Meng et al. (2019) Zhong Meng, Jinyu Li, Yashesh Gaur, and Yifan Gong. 2019.  Domain adaptation via teacher-student learning for end-to-end speech recognition.  In _IEEE Automatic Speech Recognition and Understanding Workshop (ASRU)_ , pages 268–275.
  * Myagmar et al. (2019) Batsergelen Myagmar, Jie Li, and Shigetomo Kimura. 2019.  Transferable high-level representations of bert for cross-domain sentiment classification.  In _Proceedings on the International Conference on Artificial Intelligence_ , pages 135–141.
  * Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019.  Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter.  In _NeurIPS EMC 2 Workshop_.
  * Sun et al. (2017) Sining Sun, Binbin Zhang, Lei Xie, and Yanning Zhang. 2017.  An unsupervised deep domain adaptation approach for robust speech recognition.  In _Neurocomputing_ , volume 257, pages 79–87.
  * Sushil et al. (2021) Madhumita Sushil, Simon Suster, and Walter Daelemans. 2021.  Are we there yet? exploring clinical domain knowledge of bert models.  In _Proceedings of the 20th Workshop on Biomedical Language Processing_ , pages 41–53.
  * Synnaeve et al. (2020) Gabriel Synnaeve, Qiantong Xu, Jacob Kahn, Tatiana Likhomanenko, Edouard Grave, Vineel Pratap, Anuroop Sriram, Vitaliy Liptchinsky, and Ronan Collobert. 2020\.  End-to-end asr: from supervised to semi-supervised learning with modern architectures.  In _ICML Self-supervision in Audio and Speech_.
  * Ul Haque et al. (2018) Tanjim Ul Haque, Nudrat Nawal Saber, and Faisal Muhammad Shah. 2018.  Sentiment analysis on large scale amazon product reviews.  In _2018 IEEE international conference on innovative research and development (ICIRD)_ , pages 1–6.
  * Wang et al. (2020) Daheng Wang, Meng Jiang, Munira Syed, Oliver Conway, Vishal Juneja, Sriram Subramanian, and Nitesh V Chawla. 2020.  Calendar graph neural networks for modeling time structures in spatiotemporal user behaviors.  In _Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining_, pages 2581–2589.
  * Wang et al. (2021) Daheng Wang, Zhihan Zhang, Yihong Ma, Tong Zhao, Tianwen Jiang, Nitesh Chawla, and Meng Jiang. 2021.  Modeling co-evolution of attributed and structural information in graph sequence.  _IEEE Transactions on Knowledge and Data Engineering_.