# A Quantum Monte Carlo study of the structural, energetic, and magnetic
properties of two-dimensional (2D) H and T phase VSe2

Daniel Wines<EMAIL_ADDRESS>Materials Science and Engineering
Division, National Institute of Standards and Technology (NIST), Gaithersburg,
MD 20899  Juha Tiihonen  Department of Physics, Nanoscience Center, University
of Jyväskylä, P.O. Box 35, Finland  Kayahan Saritas  Material Science and
Technology Division, Oak Ridge National Laboratory, Oak Ridge, Tennessee 37831
Jaron Krogel  Material Science and Technology Division, Oak Ridge National
Laboratory, Oak Ridge, Tennessee 37831  Can Ataca<EMAIL_ADDRESS>Department
of Physics, University of Maryland Baltimore County, Baltimore MD 21250

###### Abstract

Previous works have controversially claimed near-room temperature
ferromagnetism in two-dimensional (2D) VSe2, with conflicting results
throughout the literature. These discrepancies in magnetic properties between
both phases (T and H phase) of 2D VSe2 are most likely due to the structural
parameters being coupled to the magnetic properties. Specifically, both phases
have a close lattice match and similar total energies, which makes it
difficult to determine which phase is being observed experimentally. In this
study, we used a combination of density functional theory (DFT), highly
accurate diffusion Monte Carlo (DMC) and a surrogate Hessian line-search
optimization technique to resolve the previously reported discrepancy in
structural parameters and relative phase stability. With DMC accuracy, we
determined the freestanding geometry of both phases and constructed a phase
diagram. Our findings demonstrate the successes of the DMC method coupled with
the surrogate Hessian structural optimization technique when applied to a 2D
magnetic system.

††preprint: AIP/123-QED

##  I Introduction

Figure 1: Top and side view of the atomic structure of monolayer VSe2 in the
a) 1T and b) 2H phase.

One of the most promising two-dimensional (2D) magnetic materials that has
been extensively studied experimentally and theoretically is 2D VSe2. Similar
to other 2D transition metal dichalcogenides (such as MoS2) Ataca, Şahin, and
Ciraci (2012), VSe2 exists in two phases, the T (octahedral phase
(1T)-centered honeycombs) phase which is metallic and the H (the trigonal
prismatic phase (2H)-hexagonal honeycombs, see Fig. 1) phase which is
semiconducting. Several experimental and theoretical studies have
controversially claimed near-room temperature ferromagnetism in VSe2, with
conflicting results throughout the literature. Density functional theory (DFT)
along with classical Monte Carlo simulations have been used to obtain an
estimate of the Curie temperature of H-VSe2 (291 K) Fuh  _et al._ (2016), but
the model Ising Hamiltonian used did not take into account the magnetic
anisotropy energies, which are essential for an accurate estimation of the
Curie temperature of a 2D lattice. The Curie temperature of multilayered 2D
H-VSe2 has been experimentally measured to be 425 K, with the ferromagnetism
softening as the thickness of the sample increases Wang  _et al._ (2021).
Additionally, the experimental Curie temperature for monolayer T-VSe2 has
ranged from 300 K to 470 K Bonilla  _et al._ (2018); Yu  _et al._ (2019)
depending on which substrate is used (MoS2, graphite, SiO2-coated silicon).
The experimental magnetization of T-VSe2 has also been met with controversy,
with values of 15 $\mu_{B}$ and 5 $\mu_{B}$ (per formula unit) being reported
from two separate studies Bonilla  _et al._ (2018); Duvjir  _et al._ (2018).
Insight has also been reported with regards to how the ferromagnetism is
enhanced with defects, molecular adsorption and the choice of substrate for
VSe2 Boukhvalov and Politano (2020); Yu  _et al._ (2019); Bonilla  _et al._
(2018). A wide range of values have also been reported for the charge density
wave (CDW) transition temperature for T-VSe2, ranging from 120 K to 350 K
Sahoo  _et al._ (2020); Feng  _et al._ (2018); Chen  _et al._ (2018); Wang
_et al._ (2021); Duvjir  _et al._ (2018).

These discrepancies in the electronic and magnetic properties of either phase
of 2D VSe2 arise from the structural parameters of each phase being coupled
closely to the magnetic and electronic properties and the external factors
(substrates, defects) of the individual samples. One example of this has been
a reported discrepancy on which phase (T or H) is energetically more
favorable. Both the T and H phases have a close lattice match and similar
total energies, which makes it difficult to determine which phase is being
observed experimentally. Recently, it has been reported experimentally that
the T phase is favored for bulk VSe2, but with dimensionality decrease, the H
phase is favored Li  _et al._ (2020); Wang  _et al._ (2021). It has also been
reported that a T-to-H phase transition can be realized by thermal annealing
Li  _et al._ (2020). This same structural phase transition has even been
reported by applying a biaxial strain of $\approx$ 3 $\%$ (from calculated
results) Li  _et al._ (2020); Boukhvalov and Politano (2020); Pushkarev  _et
al._ (2019). Researchers have proposed that this lattice strain can be induced
by the mismatch that occurs from putting 2D VSe2 on a substrate Boukhvalov and
Politano (2020); Pushkarev  _et al._ (2019).

From a computational perspective, results for VSe2 depend heavily on which
methodology is employed. In most cases, DFT with an empirical Hubbard
correction (+$U$) for correlated electrons is used Dudarev  _et al._ (1998).
For example, if the $U$ correction is applied for T and H-VSe2, the T phase is
more energetically favorable, while if no $U$ correction is applied, the H
phase is more favorable Popov  _et al._ (2016). In addition to the
discrepancies in results calculated with DFT+$U$, results between van der
Waals (vdW) corrected functionals and hybrid functionals are also inconclusive
Popov  _et al._ (2016) in terms of predicting the relative phase stability. In
order to alleviate the uncertainty in DFT methods, more sophisticated methods
can be used such as Diffusion Monte Carlo (DMC) Foulkes  _et al._ (2001). DMC
is a correlated, many-body electronic structure method that has demonstrated
success for the electronic and magnetic properties of a variety of bulk and 2D
systems Foyevtsova  _et al._ (2014); Busemeyer  _et al._ (2016); Wines,
Saritas, and Ataca (2020, 2021, 2022); Wines, Choudhary, and Tavazza (2022);
Tiihonen, Kent, and Krogel (2022); Shin  _et al._ (2021); Staros  _et al._
(2021). This method has a weaker dependence on the starting density functional
and $U$ parameter and can successfully achieve results with an accuracy beyond
the DFT+$U$ Foulkes  _et al._ (2001).

Due to the fact that T and H-VSe2 have structural parameters that are coupled
to their electronic and magnetic properties, it makes it difficult to produce
conclusive results that rely solely on DFT or DFT+$U$. For this reason, we
employed our recently developed energy-based surrogate Hessian method for
structural optimization with stochastic electronic structure theories (such as
DMC) Tiihonen, Kent, and Krogel (2022) to obtain the geometry of T and H-VSe2
with DMC accuracy, resulting in high-accuracy bond lengths that resolve
previous functional dependent structural discrepancies. After obtaining an
accurate geometry for both structures, we constructed a phase diagram between
T and H-VSe2 using DMC calculated energies and obtained accurate magnetic
properties of each structure. The accurate estimates for lattice geometry,
relative phase energy and the DMC phase diagram assist in clarifying
previously inconclusive theoretical and experimental results regarding T and H
phase VSe2. For full details of the computational methods used, see the
Supporting Information (SI).

As an initial starting point for our study, we performed benchmarking DFT and
DFT+$U$ calculations using a variety of density functionals (local density
approximation (LDA)Hohenberg and Kohn (1964), Perdew-Burke-Ernzerhof
(PBE)Perdew, Burke, and Ernzerhof (1996), and strongly constrained and
appropriately normed (SCAN)Sun, Ruzsinszky, and Perdew (2015) meta-GGA
functionals, see SI for more details) and the Vienna Ab initio Simulation
Package (VASP) code for monolayer T-VSe2 and H-VSe2. The goal of these
simulations were to assess how sensitive the relative energy between the T and
H phase is with respect to functional and material geometry. Another goal of
these simulations was to benchmark the structural parameters of each material
with respect to several density functionals. It is advantageous to perform
these reference calculations with VASP and PAW pseudopotentials as a precursor
to the more expensive DMC calculations due to the fact that they require a
much smaller cutoff energy and are more cost effective for a large number of
simulations. It is important to note that for all DFT and DMC simulations, we
assumed a ferromagnetic ground state for both T and H-VSe2. Although recent
reports have suggested that T-VSe2 could be experimentally paramagnetic Wang
_et al._ (2021), we infer that this paramagnetism can be induced by magnetic
anisotropy. In addition, the modeling of paramagnetism with computational
methods imposes a great challenge, which is why we focus on the freestanding
ferromagnetic ground states of both phases. A more robust treatment of the
magnetic structure can be explored in future work, but is beyond the scope of
this work which primarily focuses on determining the geometric structure and
phase stability of 2D T and H-VSe2.

Figure 2: Relative (T - H) energy between T and H phase 2D VSe2 as a function
of $U$ parameter for several density functionals and methods of atomic
relaxation: a) fully relaxing the structure, b) fixing the lattice and atomic
positions to the $U$ = 0 eV relaxed geometry of that particular functional and
calculating the static energy, c) fixing the lattice to the $U$ = 0 eV relaxed
geometry of that particular functional and relaxing just the atomic positions.
The dotted line indicates 0 eV.

In Fig. 2 we present a comprehensive look at the difference in total energy
between T-VSe2 and H-VSe2, using several DFT functionals under different
geometric constraints. We performed these calculations for a variety of $U$
values in three different ways: fully relaxing the structure at each value of
$U$ (Fig. 2 a) ), fixing the lattice and atomic positions to the $U$ = 0 eV
relaxed geometry of that particular functional and calculating the static
energy at each value of $U$ (Fig 2 b)), fixing the lattice to the $U$ = 0 eV
relaxed geometry of that particular functional and relaxing just the atomic
positions at each value of $U$ (Fig. 2 c)). The results in Fig. 2 indicate
that there is a significant disagreement between DFT functionals, $U$ value
used, and material geometries, with all three factors playing a significant
role in the energy difference between T and H phase. Specifically, regardless
of relaxation method, all bare (no $U$ correction) SCAN, PBE, and PBEsol
functionals predict H favorable, while bare LDA predicts T favorable. For all
functionals, there is a critical value of $U$ that reverses the relative phase
stability, which is dependent on functional and relaxation method. The SCAN
functional with a $U$ correction predicts T phase favorable, with larger
energy differences. As seen in Fig. 2, the trends in the relative phase
stability between Fig. 2 b) and c) are nearly identical, but significantly
vary from Fig. a). This implies that the density functional is strongly
coupled to material geometry, but the lattice constant change has more of an
effect on phase stability than atomic positions and bond distances. This is
most prevalent for higher $U$ values ($>$ 2 eV), where the relaxed geometry
changes more drastically with $U$. The interrelated nature of the material’s
geometry, density functional, and value of $U$ are reasons to seek out higher
levels of theory beyond DFT/DFT+$U$ such as DMC to accurately determine the
optimal geometry and relative energy between the phases of 2D VSe2.

Figure 3: A summary of the deviation of the geometric properties relative to the DMC calculated geometric properties for a) T-VSe2 and b) H-VSe2 and c) the the deviation of T - H energy relative to the DMC calculated T - H energy for a variety of DFT functionals ($U$ = 2 eV), where the DMC error bar (standard error about the mean) is represented by the red bars. Table 1:  Tabulated results for lattice constant, V-Se distance, and relative energy (T - H) for both T and H phase 2D VSe2 for several computational methods. DMC error bars (standard error about the mean) are included in parenthesis. | T-VSe2 |  | H-VSe2 |  |
---|---|---|---|---|---
Method |  $a$ (Å) |  $d^{\textrm{V}-\textrm{Se}}$ (Å) |  $a$ (Å) |  $d^{\textrm{V}-\textrm{Se}}$ (Å) | E${}^{\textrm{T}-\textrm{H}}$ (eV/f.u.)
PBE | 3.336 | 2.489 | 3.333 | 2.502 | 0.045
PBE+$U$=2 | 3.435 | 2.526 | 3.364 | 2.520 | -0.008
LDA | 3.228 | 2.438 | 3.229 | 2.445 | -0.026
LDA+$U$=2 | 3.277 | 2.455 | 3.266 | 2.464 | 0.045
SCAN | 3.387 | 2.486 | 3.329 | 2.486 | 0.045
SCAN+$U$=2 | 3.462 | 2.524 | 3.353 | 2.502 | -0.202
PBEsol | 3.262 | 2.458 | 3.272 | 2.471 | 0.013
PBEsol+$U$=2 | 3.323 | 2.483 | 3.301 | 2.487 | 0.025
PBE-D2 | 3.323 | 2.484 | 3.318 | 2.496 | 0.010
PBE-D3 | 3.315 | 2.485 | 3.319 | 2.497 | 0.042
SCAN+rvv10 | 3.379 | 2.481 | 3.319 | 2.482 | 0.051
DMC | 3.414(12) | 2.505(7) | 3.335(8) | 2.503(5) | 0.06(2)

The relaxed lattice constants, V-Se distances, and T - H energies from Fig. 2
a) are presented in Table 1 and Fig. 3, along with additional VASP reference
calculations performed with the vdW corrected functionals (PBE-D2 Grimme
(2006), PBE-D3 Grimme, Ehrlich, and Goerigk (2011), SCAN+rvv10 Peng  _et al._
(2016)). The DMC computed parameters are also given for comparison in Table 1
and Fig. 3 (more discussion to follow). We observe a $\approx$ 7 $\%$
variability in lattice constant across the different methods for T-VSe2 and a
$\approx$ 4 $\%$ variability in lattice constant across the different methods
for H-VSe2. Between both phases, we observe a $\approx$ 3 $\%$ variability in
V-Se distance ($d^{\textrm{V}-\textrm{Se}}$). Most strikingly, the energy
difference between the T and H phases (E${}^{\textrm{T}-\textrm{H}}$)
drastically varies depending on the material geometry and computational
methodology, ranging from -0.2 eV/f.u. to 0.06 eV/f.u.. Due to the fact that a
strain-induced phase transition has been reported between T- and H-VSe2 Li
_et al._ (2020); Boukhvalov and Politano (2020); Pushkarev  _et al._ (2019),
we decided to perform additional VASP benchmarking calculations that involved
the application of tensile and compressive strain for each monolayer. We
performed these calculations for PBE, SCAN, and LDA (with $U$ = 0 eV and $U$ =
2 eV), starting from the $U$ = 0 eV geometry for each functional. The
resulting equations of state are depicted in Fig. S3. As seen in the figure,
the equation of state and resulting strain-induced phase transition is
entirely dependent on the functional and $U$ value, with no consistent trend.

The strong sensitivity of each monolayer with respect to geometry and
functional are grounds for using a higher-order method such as DMC to obtain a
statistically accurate estimate of the lattice parameters and relative energy
between phases. Prior to performing the DMC/line-search calculations, we
optimized our nodal surface (orbitals selected for DFT wavefunction
generation). Since DMC has the zero-variance property, it means that as the
trial wave function approaches the exact ground state, the statistical
fluctuations in the energy reduce to zero Foulkes  _et al._ (2001). Although
there have been instances where various sophisticated methods have been used
to optimize the nodal surface Kwon, Ceperley, and Martin (1993, 1998); López
Ríos  _et al._ (2006); Bajdich  _et al._ (2010), we employed the PBE+$U$
approach, where the Hubbard ($U$) value was used as a variational parameter to
optimize the nodal surface using DMC (similar to other successful DMC studies
of magnetic materials Foyevtsova  _et al._ (2014); Ichibha  _et al._ (2021);
Saritas  _et al._ (2019, 2018); Staros  _et al._ (2021); Wines, Saritas, and
Ataca (2022); Wines, Choudhary, and Tavazza (2022)). We performed these
calculations for both T and H-VSe2 (24 atom supercells), where we tuned the
$U$ value from (1 to 4) eV while creating the trial wavefunction and computed
the DMC energy. The results of these calculations are depicted in Fig. S4,
where we observe that $U$ = 2 eV yields the lowest energy for both phases. It
is important to note that for the H phase, the DMC energies for $U$ = 1 and
$U$ = 2 eV are statistically identical. Based on this, we created the trial
wavefunction using PBE+$U$ ($U$ = 2 eV) for all subsequent DMC calculations
within the surrogate Hessian line-search for both phases (all 52 DMC energy
evaluations). Since we obtained an optimal $U$ value of 2 eV for both
materials, we focused our DFT+U benchmarking efforts more on $U$ = 2 eV (Fig.
3, Fig 5, Table 1, Fig. 2, Fig. S3).

Figure 4: (Top) The phase diagram of 2D VSe2 in terms of $a$ and
$d^{\textrm{V}-\textrm{Se}}$. The phase boundary (solid line, black) is
estimated from bicubic fits. To assure quality of the fits, the estimated $\pm
0.01$ eV error contours (dotted line) and the minima from the fits (’x’) and
the line-search (’o’) are all well separated. (Bottom) Slices of the PES at
$d^{\textrm{V}-\textrm{Se}}=2.505$ Å.

Based on the DMC line-search results, we determined accurate bounds on the
lattice parameter ($a$) and off-plane displacement of Se ($z$), within an
error tolerance of $0.018$ Å or lower for both parameters. This translates to
within $\approx 0.5\%$ accuracy in a parameter set of $a$ and
$d^{\textrm{V}-\textrm{Se}}$ with 95% confidence. Convergence (absence of
significant displacements outside of the error tolerance) was achieved after
two parallel line-search iterations for both phases. This convergence is
illustrated in Fig. S5, where the convergence of the parameter offsets of $a$
and $z$ and the convergence of the total energy per f.u. are depicted for both
T and H phase 2D VSe2 for the initial DFT relaxed structure (1) and both
subsequent iterations of DMC (2 - 3). In addition, the final energy of both of
the fitted structures (square points) are given.

The final geometric parameters and relative phase energies determined with DMC
are given in Table 1 and Fig. 3. For T-VSe2, we determined a lattice constant
of 3.414(12) Å and a V-Se distance of 2.505(7) Å . For H-VSe2, we determined a
lattice constant of 3.335(8) Å and a V-Se distance of 2.503(5) Å . The DMC
finite-size extrapolated energy difference (T - H) between the two phases was
determined to be 0.06(2) eV/f.u., indicating that in freestanding form at the
equilibrium geometry, H-VSe2 is favored over T-VSe2. When comparing these DMC
results to the other DFT functionals in Table 1 and Fig. 3, it is clear that
very few DFT functionals can reproduce the DMC results for lattice constant,
V-Se distance and relative energy difference. The SCAN functional comes the
closest to reproducing all three simultaneous DMC values, but still falls
slightly short for the V-Se distances of both phases and the lattice constant
of T-VSe2. The fact that SCAN+U successfully predicts the structural
properties (for H-VSe2) and the fact that SCAN+rvv10 produces an energy
difference closest to the average DMC energy difference for both phases
loosely implies that a simultaneous description of correlated magnetism and
vdW interactions are both needed to correctly represent the physics of VSe2.
Experimental measurements of the lattice constant and V-Se distance of
freestanding monolayer VSe2 are scarce and often times dependent on external
factors such as the substrate (more discussion to follow) and sample
preparation technique Yu  _et al._ (2019); Bonilla  _et al._ (2018); Chen  _et
al._ (2020); Liu  _et al._ (2018). However, Chen et al. Chen  _et al._ (2020)
have recently reported a lattice constant of 3.4 Å for thin films of T-VSe2
and Liu et al. Liu  _et al._ (2018) have recently reported a lattice constant
of 3.3 Å for epitaxially grown monolayer H-VSe2. Both of these measured values
are in excellent agreement with our DMC computed lattice constants.
Additionally, we determined the near-equilibrium PES of both T and H 2D VSe2
with DMC accuracy, which are both depicted in Fig. S6.

The phase diagram presented in Fig. 4 is based on similar fits to data, where
the $z$ displacement has been remapped to $d^{\textrm{V}-\textrm{Se}}$. This
DMC phase diagram can directly be compared to the energy vs. strain DFT
benchmarking calculations in Fig. S3, which emphasizes the need for an
accurate representation of the phase boundary between the two phases. The
freestanding geometries of both T and H lie in the energetic H phase, but a
slice of the phase diagram along $d^{\textrm{V}-\textrm{Se}}=2.505$ Å
indicates that the T phase becomes favorable over H at biaxial strain of
$a\gtrsim 3.5$ Å. This implies that in freestanding form, once T-VSe2 is
positively strained at least $\approx$ 2.5 $\%$, T phase is favored over H.
Alternatively, if freestanding H-VSe2 is positively strained at least
$\approx$ 5 $\%$, T phase is also favored over H This strain can easily be
accomplished by placing monolayer VSe2 on a substrate with significant lattice
mismatch. In fact, this type of mismatch has been reported to alter the
material properties Yu  _et al._ (2019); Bonilla  _et al._ (2018); Karn  _et
al._ (2022); Song  _et al._ (2021), significantly contributing to the
controversies of T and H-VSe2 (for energetic favorability, magnetic
properties). Whether or not the changes in energetic favorability or magnetic
properties with respect to the substrate are due to lattice mismatch or more
complicated interactions between the substrate and the monolayer remains to be
answered and is beyond the scope of this work, which has focused solely on the
freestanding forms of T and H-VSe2. However, such calculations can be employed
for future work using higher order methods such as DMC. The proximity of the
phase boundary between T and H phase (Fig. 4) is emphasized by the small
energy difference between the two phases (0.06(2) eV/f.u., at the equilibrium
geometry) between the two curves. Since this energy difference is so close to
room temperature ($\approx$ 0.024 eV), this implies that a process such as
thermal annealing can easily induce a phase transition. In fact, recently it
was demonstrated that a structural phase transition of multilayer VSe2 from T
to H occurs through annealing at 650 K, along with a metal-insulator
transition Li  _et al._ (2020).

Figure 5: The radially averaged spin density ($\rho_{up}$ \- $\rho_{down}$) as
a function of distance, calculated with DMC and PBE+$U$ ($U$ = 2 eV) of a) V
and b) Se for 2D T-VSe2 and c) V and d) Se for 2D H-VSe2. The inset of a) and
c) depicts the spin isosurface density of T-VSe2 and H-VSe2 respectively,
where the isosurface value was set to 6 x 10-3 e/Å3. The standard error about
the mean for DMC is indicated by error bars in blue.

To gain a deeper understanding of the magnetic properties of 2D T and H-VSe2,
we extracted the spin densities (using a trial wavefunction at $U$ = 2 eV and
24 atom supercell at the final equilibrium geometry predicted by DMC/line-
search). The spin density isosurfaces of each phase ($\rho_{\textrm{up}}$ \-
$\rho_{\textrm{down}}$) are depicted in the insets of Fig. 5 a) and c) for
T-VSe2 and H-VSe2 respectively. For both phases, we observe the V atoms are
highly spin-polarized, while the Se atoms are slightly antiparallel with
respect to the V atoms. For more calculation details regarding spin density,
see SI.

We went on to plot the radial averaged spin densities as a function of
distance, separately for V and Se for T and H-VSe2 (depicted in Fig. 5 a) -
d)). This allows us to view the spatial variations in spin density.
Additionally, we benchmarked these V and Se radially averaged densities with
PBE+$U$ ($U$ = 2 eV) using NC pseudopotentials at the equilibrium geometry
(the calculation required to create the trial WF for the subsequent DMC runs).
As seen in Fig. 5 a) and c), there is a substantial difference in the V spin
density between DMC and PBE+$U$ ($U$ = 2 eV) for both T and H phase. This same
substantial difference between DMC and PBE+$U$ also occurs for the total
charge density. This discrepancy is most prevalent near the radial density
peak (peak of $d$ orbital) and can be attributed to the fact that DFT
functionals (even with the added Hubbard correction) tend to delocalize and
unsuccessfully capture 3$d$ orbitals. This large discrepancy in the spin
densities highlights the need for more accurate, many-body computational
methodologies for correlated materials such as VSe2, where DFT fails. In
contrast, there is closer agreement between the DMC and PBE+$U$ spin densities
for Se in T and H-VSe2 (see Fig. 5 b) and d).

Finally, we estimated the site-averaged atomic magnetic moments per V and Se
for both T and H phase by integrating the DMC and PBE+$U$ spin densities
depicted in Fig. 5. At the DMC level, we estimated a magnetic moment of
1.06(2) $\mu_{\textrm{B}}$ for V and -0.09(2) $\mu_{\textrm{B}}$ for Se in
T-VSe2 and a magnetic moment of 1.02(1) $\mu_{\textrm{B}}$ for V and -0.14(1)
$\mu_{\textrm{B}}$ for Se in H-VSe2. At the PBE+$U$ ($U$ = 2 eV) level, we
estimated a magnetic moment of 1.30 $\mu_{\textrm{B}}$ for V and -0.12
$\mu_{\textrm{B}}$ for Se in T-VSe2 and a magnetic moment of 1.40
$\mu_{\textrm{B}}$ for V and -0.15 $\mu_{\textrm{B}}$ for Se in H-VSe2.
Consistent with the radial spin density results in Fig. 5, we find that the
DMC and PBE+$U$ magnetic moments for Se are in much closer agreement than for
V (for both T and H phase). By analyzing the spin densities and obtaining the
on-site magnetic moments, we obtain a clear picture of how the magnetization
of each ion depends on the computational method used, serving as a benchmark
for the magnetic properties of 2D VSe2.

In this work, we used a combination of DFT, DMC and a recently developed
surrogate Hessian line-search optimization technique to resolve the previously
reported discrepancy in structural parameters and relative phase stability of
monolayer T-VSe2 and H-VSe2. Using these methods, we determined the lattice
constant and V-Se distance (with DMC accuracy) to be 3.414(12) Å and 2.505(7)
Å respectively for T-VSe2 and 3.335(8) Å and 2.503(5) respectively for H-VSe2.
In addition, we find the relative energy between the phases (T - H) to be
0.06(2) eV/f.u. at the DMC level, indicating that in freestanding form, H-VSe2
is more energetically favorable than T-VSe2. We went on to obtain a phase
diagram between T and H phase from the PES and determined that a phase
transition can be induced by strain or mechanisms such as thermal annealing.
Additionally, we benchmarked the magnetic properties such as spin density and
on-site magnetic moment for both phases and find substantial differences
between DMC and DFT. The results of this study demonstrate the successes of
the DMC method coupled with the surrogate Hessian line-search structural
optimization technique when applied to a 2D magnetic system. The estimates for
lattice constant, bond distance, relative phase energy and the extracted
structural-dependent phase diagram assist in clarifying previously
inconclusive theoretical and experimental results regarding T and H phase
VSe2.

##  II Code Availability Statement

Software packages mentioned in the article can be found at
https://github.com/usnistgov/jarvis. Please note that the use of commercial
software (VASP) does not imply recommendation by the National Institute of
Standards and Technology.

##  III Competing interests

The authors declare no competing interests.

##  IV acknowledgments

The authors thank the National Institute of Standards and Technology for
funding, computational, and data-management resources. The authors thank Dr.
Kamal Choudhary and Dr. Francesca Tavazza for fruitful discussions. We
acknowledge grants of computer capacity from the Finnish Grid and Cloud
Infrastructure (persistent identifier urn:nbn:fi:research-infras-2016072533).

## References

## References

  * Ataca, Şahin, and Ciraci (2012) C. Ataca, H. Şahin, and S. Ciraci, “Stable, single-layer MX2 transition-metal oxides and dichalcogenides in a honeycomb-like structure,” The Journal of Physical Chemistry C 116, 8983–8999 (2012).
  * Fuh  _et al._ (2016) H.-R. Fuh, C.-R. Chang, Y.-K. Wang, R. F. L. Evans, R. W. Chantrell, and H.-T. Jeng, “Newtype single-layer magnetic semiconductor in transition-metal dichalcogenides vx2 (x = s, se and te),” Scientific Reports 6, 32625 (2016).
  * Wang  _et al._ (2021) X. Wang, D. Li, Z. Li, C. Wu, C.-M. Che, G. Chen, and X. Cui, “Ferromagnetism in 2d vanadium diselenide,”  _ACS Nano_ , ACS Nano 15, 16236–16241 (2021).
  * Bonilla  _et al._ (2018) M. Bonilla, S. Kolekar, Y. Ma, H. C. Diaz, V. Kalappattil, R. Das, T. Eggers, H. R. Gutierrez, M.-H. Phan, and M. Batzill, “Strong room-temperature ferromagnetism in VSe2 monolayers on van der Waals substrates,” Nature Nanotechnology 13, 289–293 (2018).
  * Yu  _et al._ (2019) W. Yu, J. Li, T. S. Herng, Z. Wang, X. Zhao, X. Chi, W. Fu, I. Abdelwahab, J. Zhou, J. Dan, Z. Chen, Z. Chen, Z. Li, J. Lu, S. J. Pennycook, Y. P. Feng, J. Ding, and K. P. Loh, “Chemically exfoliated vse2 monolayers with room-temperature ferromagnetism,” Advanced Materials 31, 1903779 (2019), https://onlinelibrary.wiley.com/doi/pdf/10.1002/adma.201903779 .
  * Duvjir  _et al._ (2018) G. Duvjir, B. K. Choi, I. Jang, S. Ulstrup, S. Kang, T. Thi Ly, S. Kim, Y. H. Choi, C. Jozwiak, A. Bostwick, E. Rotenberg, J.-G. Park, R. Sankar, K.-S. Kim, J. Kim, and Y. J. Chang, “Emergence of a metal–insulator transition and high-temperature charge-density waves in vse2 at the monolayer limit,”  _Nano Letters_ , Nano Letters 18, 5432–5438 (2018).
  * Boukhvalov and Politano (2020) D. W. Boukhvalov and A. Politano, “Unveiling the origin of room-temperature ferromagnetism in monolayer vse2: the role of extrinsic effects,” Nanoscale 12, 20875–20882 (2020).
  * Sahoo  _et al._ (2020) S. Sahoo, U. Dutta, L. Harnagea, A. K. Sood, and S. Karmakar, “Pressure-induced suppression of charge density wave and emergence of superconductivity in $1t-{\mathrm{vse}}_{2}$,” Phys. Rev. B 101, 014514 (2020).
  * Feng  _et al._ (2018) J. Feng, D. Biswas, A. Rajan, M. D. Watson, F. Mazzola, O. J. Clark, K. Underwood, I. Marković, M. McLaren, A. Hunter, D. M. Burn, L. B. Duffy, S. Barua, G. Balakrishnan, F. Bertran, P. Le Fèvre, T. K. Kim, G. van der Laan, T. Hesjedal, P. Wahl, and P. D. C. King, “Electronic structure and enhanced charge-density wave order of monolayer vse2,” Nano Letters 18, 4493–4499 (2018).
  * Chen  _et al._ (2018) P. Chen, W. W. Pai, Y.-H. Chan, V. Madhavan, M. Y. Chou, S.-K. Mo, A.-V. Fedorov, and T.-C. Chiang, “Unique gap structure and symmetry of the charge density wave in single-layer ${\mathrm{vse}}_{2}$,” Phys. Rev. Lett. 121, 196402 (2018).
  * Li  _et al._ (2020) D. Li, X. Wang, C.-m. Kan, D. He, Z. Li, Q. Hao, H. Zhao, C. Wu, C. Jin, and X. Cui, “Structural phase transition of multilayer vse2,” ACS Applied Materials & Interfaces 12, 25143–25149 (2020).
  * Pushkarev  _et al._ (2019) G. V. Pushkarev, V. G. Mazurenko, V. V. Mazurenko, and D. W. Boukhvalov, “Structural phase transitions in vse2: energetics, electronic structure and magnetism,” Phys. Chem. Chem. Phys. 21, 22647–22653 (2019).
  * Dudarev  _et al._ (1998) S. L. Dudarev, G. A. Botton, S. Y. Savrasov, C. J. Humphreys, and A. P. Sutton, “Electron-energy-loss spectra and the structural stability of nickel oxide: An LSDA+U study,” Phys. Rev. B 57, 1505–1509 (1998).
  * Popov  _et al._ (2016) Z. I. Popov, N. S. Mikhaleva, M. A. Visotin, A. A. Kuzubov, S. Entani, H. Naramoto, S. Sakai, P. B. Sorokin, and P. V. Avramov, “The electronic structure and spin states of 2d graphene/vx2 (x = s, se) heterostructures,” Phys. Chem. Chem. Phys. 18, 33047–33052 (2016).
  * Foulkes  _et al._ (2001) W. M. C. Foulkes, L. Mitas, R. J. Needs, and G. Rajagopal, “Quantum Monte Carlo simulations of solids,” Rev. Mod. Phys. 73, 33–83 (2001).
  * Foyevtsova  _et al._ (2014) K. Foyevtsova, J. T. Krogel, J. Kim, P. R. C. Kent, E. Dagotto, and F. A. Reboredo, “Ab initio Quantum Monte Carlo calculations of spin superexchange in cuprates: The benchmarking case of ${\mathrm{Ca}}_{2}{\mathrm{CuO}}_{3}$,” Phys. Rev. X 4, 031003 (2014).
  * Busemeyer  _et al._ (2016) B. Busemeyer, M. Dagrada, S. Sorella, M. Casula, and L. K. Wagner, “Competing collinear magnetic structures in superconducting FeSe by first-principles quantum Monte Carlo calculations,” Phys. Rev. B 94, 035108 (2016).
  * Wines, Saritas, and Ataca (2020) D. Wines, K. Saritas, and C. Ataca, “A first-principles Quantum Monte Carlo study of two-dimensional (2D) GaSe,” The Journal of Chemical Physics 153, 154704 (2020).
  * Wines, Saritas, and Ataca (2021) D. Wines, K. Saritas, and C. Ataca, “A pathway toward high-throughput quantum Monte Carlo simulations for alloys: A case study of two-dimensional (2d) GaSxSe1-x,” The Journal of Chemical Physics 155, 194112 (2021).
  * Wines, Saritas, and Ataca (2022) D. Wines, K. Saritas, and C. Ataca, “Intrinsic ferromagnetism of two-dimensional (2d) mno2 revisited: A many-body quantum monte carlo and dft+u study,” The Journal of Physical Chemistry C 126, 5813–5821 (2022).
  * Wines, Choudhary, and Tavazza (2022) D. Wines, K. Choudhary, and F. Tavazza, “A systematic dft+u and quantum monte carlo benchmark of magnetic two-dimensional (2d) crx3 (x = i, br, cl, f),”  (2022).
  * Tiihonen, Kent, and Krogel (2022) J. Tiihonen, P. R. C. Kent, and J. T. Krogel, “Surrogate hessian accelerated structural optimization for stochastic electronic structure theories,” The Journal of Chemical Physics 156, 054104 (2022), https://doi.org/10.1063/5.0079046 .
  * Shin  _et al._ (2021) H. Shin, J. T. Krogel, K. Gasperich, P. R. C. Kent, A. Benali, and O. Heinonen, “Optimized structure and electronic band gap of monolayer GeSe from Quantum Monte Carlo methods,” Phys. Rev. Materials 5, 024002 (2021).
  * Staros  _et al._ (2021) D. Staros, G. Hu, J. Tiihonen, R. Nanguneri, J. Krogel, M. C. Bennett, O. Heinonen, P. Ganesh, and B. Rubenstein, “A combined first principles study of the structural, magnetic, and phonon properties of monolayer cri3,”  _The Journal of Chemical Physics_ , The Journal of Chemical Physics 156, 014707 (2021).
  * Hohenberg and Kohn (1964) P. Hohenberg and W. Kohn, “Inhomogeneous electron gas,” Phys. Rev. 136, B864–B871 (1964).
  * Perdew, Burke, and Ernzerhof (1996) J. P. Perdew, K. Burke, and M. Ernzerhof, “Generalized gradient approximation made simple,” Phys. Rev. Lett. 77, 3865–3868 (1996).
  * Sun, Ruzsinszky, and Perdew (2015) J. Sun, A. Ruzsinszky, and J. P. Perdew, “Strongly constrained and appropriately normed semilocal density functional,” Phys. Rev. Lett. 115, 036402 (2015).
  * Grimme (2006) S. Grimme, “Semiempirical GGA-type density functional constructed with a long-range dispersion correction,” Journal of Computational Chemistry 27, 1787–1799 (2006).
  * Grimme, Ehrlich, and Goerigk (2011) S. Grimme, S. Ehrlich, and L. Goerigk, “Effect of the damping function in dispersion corrected density functional theory,” Journal of Computational Chemistry 32, 1456–1465 (2011), https://onlinelibrary.wiley.com/doi/pdf/10.1002/jcc.21759 .
  * Peng  _et al._ (2016) H. Peng, Z.-H. Yang, J. P. Perdew, and J. Sun, “Versatile van der Waals density functional based on a meta-generalized gradient approximation,” Phys. Rev. X 6, 041005 (2016).
  * Kwon, Ceperley, and Martin (1993) Y. Kwon, D. M. Ceperley, and R. M. Martin, “Effects of three-body and backflow correlations in the two-dimensional electron gas,” Phys. Rev. B 48, 12037–12046 (1993).
  * Kwon, Ceperley, and Martin (1998) Y. Kwon, D. M. Ceperley, and R. M. Martin, “Effects of backflow correlation in the three-dimensional electron gas: Quantum monte carlo study,” Phys. Rev. B 58, 6800–6806 (1998).
  * López Ríos  _et al._ (2006) P. López Ríos, A. Ma, N. D. Drummond, M. D. Towler, and R. J. Needs, “Inhomogeneous backflow transformations in quantum monte carlo calculations,” Phys. Rev. E 74, 066701 (2006).
  * Bajdich  _et al._ (2010) M. Bajdich, M. L. Tiago, R. Q. Hood, P. R. C. Kent, and F. A. Reboredo, “Systematic reduction of sign errors in many-body calculations of atoms and molecules,” Phys. Rev. Lett. 104, 193001 (2010).
  * Ichibha  _et al._ (2021) T. Ichibha, A. L. Dzubak, J. T. Krogel, V. R. Cooper, and F. A. Reboredo, “${\mathrm{cri}}_{3}$ revisited with a many-body ab initio theoretical approach,” Phys. Rev. Materials 5, 064006 (2021).
  * Saritas  _et al._ (2019) K. Saritas, J. T. Krogel, S. Okamoto, H. N. Lee, and F. A. Reboredo, “Structural, electronic, and magnetic properties of bulk and epitaxial ${\mathrm{LaCoO}}_{3}$ through Diffusion Monte Carlo,” Phys. Rev. Materials 3, 124414 (2019).
  * Saritas  _et al._ (2018) K. Saritas, J. T. Krogel, P. R. C. Kent, and F. A. Reboredo, “Diffusion monte carlo: A pathway towards an accurate theoretical description of manganese oxides,” Phys. Rev. Materials 2, 085801 (2018).
  * Chen  _et al._ (2020) G. Chen, S. T. Howard, A. B. Maghirang, K. Nguyen Cong, R. A. B. Villaos, L.-Y. Feng, K. Cai, S. C. Ganguli, W. Swiech, E. Morosan, I. I. Oleynik, F.-C. Chuang, H. Lin, and V. Madhavan, “Correlating structural, electronic, and magnetic properties of epitaxial $\mathrm{V}{\mathrm{se}}_{2}$ thin films,” Phys. Rev. B 102, 115149 (2020).
  * Liu  _et al._ (2018) Z.-L. Liu, X. Wu, Y. Shao, J. Qi, Y. Cao, L. Huang, C. Liu, J.-O. Wang, Q. Zheng, Z.-L. Zhu, K. Ibrahim, Y.-L. Wang, and H.-J. Gao, “Epitaxially grown monolayer vse2: an air-stable magnetic two-dimensional material with low work function at edges,” Science Bulletin 63, 419–425 (2018).
  * Karn  _et al._ (2022) A. Karn, Y. H. Chan, U. Chazarin, P. Chen, and W. W. Pai, “Modification of monolayer 1t-vse2 by selective deposition of vanadium and tellurium,” AIP Advances 12, 035240 (2022), https://doi.org/10.1063/6.0001402 .
  * Song  _et al._ (2021) D. Song, Y. Zhou, M. Zhang, X. He, and X. Li, “Structural and transport properties of 1t-vse2 single crystal under high pressures,” Frontiers in Materials 8 (2021), 10.3389/fmats.2021.710849.


# Zero-shot Code-Mixed Offensive Span Identification through Rationale
Extraction

Manikandan Ravikiran$\dagger$ , Bharathi Raja Chakravarthi$\ddagger$
$\dagger$Georgia Institute of Technology, Atlanta, Georgia
$\ddagger$Data Science Institute, National University of Ireland Galway
<EMAIL_ADDRESS><EMAIL_ADDRESS>
Corresponding Author

###### Abstract

This paper investigates the effectiveness of sentence-level transformers for
zero-shot offensive span identification on a code-mixed Tamil dataset. More
specifically, we evaluate rationale extraction methods of Local Interpretable
Model Agnostic Explanations (LIME) Ribeiro et al. (2016a) and Integrated
Gradients (IG) Sundararajan et al. (2017) for adapting transformer based
offensive language classification models for zero-shot offensive span
identification. To this end, we find that LIME and IG show baseline $F_{1}$ of
26.35% and 44.83%, respectively. Besides, we study the effect of data set size
and training process on the overall accuracy of span identification. As a
result, we find both LIME and IG to show significant improvement with Masked
Data Augmentation and Multilabel Training, with $F_{1}$ of 50.23% and 47.38%
respectively. Disclaimer : This paper contains examples that may be considered
profane, vulgar, or offensive. The examples do not represent the views of the
authors or their employers/graduate schools towards any person(s), group(s),
practice(s), or entity/entities. Instead they are used to emphasize only the
linguistic research challenges.

##  1 Introduction

Offensive language classification and offensive span identification from code-
mixed Tamil-English comments portray the same task at different granularities.
In the former case, we classify if the code mixed sentence is offensive or
not, while the latter concentrates on extracting the offensive parts of the
comments. Accordingly, one could do the former using models of the latter and
vice versa. Transformer-based architectures such as BERT Devlin et al. (2019),
RoBERTa Liu et al. (2019) and XLM-RoBERTa Conneau et al. (2020) have achieved
state-of-the-art results on both these tasks Chakravarthi et al. (2021a).
However, often these tasks are treated as independent and model development is
often separated.

This paper studies the rationale extraction methods for inferring offensive
spans from transformer models trained only on comment-level offensive language
classification labels. Such an idea is often vital in the case of code-mixed
Tamil-English comments for which span annotations are often costly to obtain,
but comment-level labels are readily available. Besides, such an approach will
also help in decoding the models’ logic behind the prediction of
offensiveness.

Accordingly, we evaluate and compare two different methods, namely LIME and
IG, for adapting pre-trained transformer models into zero-shot offensive span
labelers. Our experiments show that using LIME with pre-trained transformer
models struggles to infer correct span level annotations in a zero-shot
manner, achieving only 20% $F_{1}$ on offensive span identification for code-
mixed Tamil-English comments. To this end, we find that a combination of
masked data augmentation and multilabel training of sentence transformers
helps to better focus on individual necessary tokens and achieve a strong
baseline on offensive span identification. Besides, IG consistently surpasses
LIME even in cases where there are no data augmentation or multilabel
training. Overall the contributions of this paper are as follows.

  * •

We introduce preliminary experiments on offensive language classification
transformer models for zero-shot offensive span identification from code-mixed
Tamil-English language comments.

  * •

We systematically compare LIME and IG methods for zero-shot offensive span
identification.

  * •

We study the impact of data and training process on offensive span
identification by proposing masked data augmentation and multilabel training.

  * •

We further release our code, models, and data to facilitate further research
in the field111The code and data is made available at
https://github.com/manikandan-ravikiran/zero-shot-offensive-span.

The rest of the paper is organized as follows. In Section 2, we present LIME
and IG methods in brief. Meanwhile section 3 and 4 focus on dataset and
experimental setup. In section 5, we present detailed experiments and conclude
in section 6 with our findings and possible implications on the future work.

##  2 Methods

In this section, we present the two rationale extraction methods LIME and IG
used to turn sentence-level transformer models into zero-shot offensive span
labelers.

###  2.1 Local Interpretable Model Agnostic Explanation (LIME)

LIME Ribeiro et al. (2016b) is a model agnostic interpretability approach that
generates word-level attribution scores using local surrogate models that are
trained on perturbed sentences generated by randomly masking out words in the
input sentence. The LIME model has seen considerable traction in the context
of rationale extraction for text classification, including work by Thorne et
al. (2019), which suggests that LIME outperforms attention-based approaches to
explain NLI models. LIME was also used to probe an LSTM based sentence-pair
classifier Lan and Xu (2018) by removing tokens from the premise and
hypothesis sentences separately. The generated scores are used to perform
binary classification of tokens, with the threshold based on $F_{1}$
performance on the development set. The token-level predictions were evaluated
against human explanations of the entailment relation using the e-SNLI dataset
Camburu et al. (2018).

Meanwhile, for offensive span identification in English Ding and Jurgens
(2021) coupled LIME with RoBERTa trained on an expanded training set to find
expanded training set could help RoBERTa more accurately learn to recognize
toxic span. However, though LIME outperforms other methods, it is
significantly slower than Integrated Gradients methods, presented in the next
section.

Dataset |  | Number of
---
Train Samples
| Number of
---
Test Samples
| Offensive Language
---
Classification
| Offensive Span
---
Identification
Offensive Span Identification Dataset | 4786 | 876 | ✗ | ✓
Mask Augmented Dataset | 109961 | - | ✓ | ✓
Multilabel Dataset | 109961 | - | ✓ | ✓

Table 1: Dataset statistics of various datasets used in this work.

Steps | Example Output
---|---
Offensive Lexicon Creation | [sanghu, thu,thooooo,suthamm,F**k,flop,w*f, p**a,n**y,nakkal]
Data Sourcing |  | Last scene vera level love u
---
[Last, scene, vera, level, love,u]
Mask Generation |  | [0,1, 0, 0, 1,1]
---
[1,0, 1, 1, 1,1]
[0,0, 0, 0, 1,1]
Offensive Word Augmentation |  | [Last, sangh, vera, level, thu, n**y]
---
[flop, scene, thooooo, suthamm, F**k, sanghu]
[Last, scene, p**a, n**y, love, u]
| Position Identification
---
and
Multilabel Creation
| Last sangh vera level thu n**y [1,0,1]
---
flop scene thooooo suthamm F**k sanghu [1,1,1]
Last scene p**a n**y love u [0,1,0]

Table 2: Various steps in Masked Augmented Dataset and Multilabel Dataset
creation with sample outputs.

###  2.2 Integrated Gradients (IG)

Integrated Gradients Sundararajan et al. (2017) focuses on explaining
predictions by integrating the gradient along some trajectory in input space
connecting two points. Integrated gradient and its variants are widely used in
different fields of deep learning including natural language processing Sikdar
et al. (2021).

Specifically, it is an iterative method, which starts with so-called starting
baselines, i.e., a starting point that does not contain any information for
the model prediction. In our case involving textual data, this is the set
exclusively with the start and end tokens. These tokens mark the beginning and
the end of a sentence and do not give any information about whether the
evaluation is offensive or not. Following this, it takes a certain number of
iterations, where the model moves from the starting baseline to the actual
input to the model.

This iterative improvement approach is analogous to the sentence creation
process wherein each step, we create the sentence word by word and calculate
the offensiveness, which in turn gives us the attribution of the input
feature. Across its iterations, whenever IG includes an offensive word, we can
expect offensive classification prediction to swing more towards offensive
class and vice versa. Such behavior will help calculate the attribution of
each word in the identified sentence.

##  3 Datasets

In this section, we present various datasets used in this study. Details on
how they are used across different experiments are presented in Table 3.
Finally, the overall dataset statistics are as shown in Table 1.

###  3.1 Offensive Span Identification Dataset

The Shared task on Offensive Span Identification from Code-Mixed Tamil English
Comments Ravikiran and Annamalai (2021) focuses on the extraction of offensive
spans from Youtube Comments. The dataset contains 4786 and 876 examples across
its train and test set respectively. It consists of annotated offensive spans
indicating character offsets of parts of the comments that were offensive.

###  3.2 Masked Augmented Dataset

The data available from both Ravikiran and Annamalai (2021) is minimal, and
transformer methods are sensitive to dataset size Xu et al. (2021). Thus we
created an additional dataset using Masked Augmentation. Accordingly, the data
is generated by using the following steps.

  * •

Step 1: Offensive Lexicon Creation: First, we create an offensive lexicon from
the train set of offensive span identification datasets. To do this, we do
following

    * –

Extract the phrases corresponding to annotated offensive spans from the
training dataset of Ravikiran and Annamalai (2021).

    * –

Selecting phrases of size less than 20 characters and word tokenizing them to
extract the individual words.

    * –

Manually, post-processing these words to ignore words that are not offensive.
For example, many phrases include conjunctions and pronouns which are not
directly offensive.

Accordingly, an offensive lexicon with 2900 tokens is created.

  * •

Step 2: Data Sourcing: In this step, we select the dataset used for creating
the Masked Augmented dataset. Specifically, we use the 25425 non-offensive
comments from Dravidian Code-Mix dataset Chakravarthi et al. (2021b).

  * •

Step 3: Mask Generation: The mask generation is done as follows

    * –

Each of 25425 non-offensive comments was tokenized to create respective
maskable token list.

    * –

Three random binary masks are generated for each of the tokenized non-
offensive comments. These binary masks have same length as that of its
maskable token list.

  * •

Step 4: Offensive Word Augmentation: Finally, words with a corresponding
binary mask of 1 are replaced with words randomly selected from the offensive
lexicon from step 1. Additionally, the spans corresponding to the words that
were replaced are saved.

Overall, such augmentation resulted in 109961 comments, with 75009 being
offensive comments and 34952 non-offensive comments. Table 2 shows an example
sentence and masked augmented dataset creation process.

Experiment Name | Train dataset | Test dataset
---|---|---
Benchmark | Offensive Span Identification Dataset Ravikiran and Annamalai (2021); Ravikiran et al. (2022)
OS-Baseline | Dravidian CodeMix Chakravarthi et al. (2021b) |  Offensive Span Identification Dataset
OS-Augmentation | Mask Augmented Dataset
OS-Multilabel | Multilabel Dataset

Table 3: Relationship between the datasets and experiments.

Name | Value
---|---
$\gamma$ | 0.1
max seq length | 150
train batch size | 64
eval batch size | 64
warmup ratio | 0.1
learning rate | 3x$10^{-5}$
weight decay | 0.1
initializer | glorot

Table 4: Model Hyperparameters for Training Transformers.

###  3.3 Multilabel Dataset

All the previously mentioned datasets are restricted to classification only
i.e. they contain a binary label indicating if they are offensive or they have
annotated offensive spans. Additionally, these sentences does not explicitly
encode any position information of the offensive words, which is useful for
training. Work of Ke et al. (2021) show encoding relative positional
information based attention directly in each head often improves the overall
result of corresponding down stream task. Similarly Shaw et al. (2018) also
proposed using relative position encoding instead of absolute position
encoding and couple them with key and value projections of Transformers to
improve overall results. As such, in this work, to encode position we create a
multilabel dataset in which the labels indicate the relative position of
offensive words. The multilabel dataset is created as follows.

  * •

Step 1: Dataset Selection: We first select the 109961 comments from the Masked
Augmented Dataset along with their saved spans.

  * •

Step 2: Position Identification and Multilabel Creation: From the identified
spans, we check if the offensive spans are present in (a) start of the comment
(b) end of the comment and (c) middle of the comment. Depending on presence of
offensiveness we create three binary labels. For example in Table 2 for
sentence Last scene p**a n**y love u we can see that the offensive word to be
present in the center of the sentence. Accordingly we give it a label [0,1,0].
Meanwhile for comment Last sangh vera level thu n**y we can see offensive
words to be in center and at the end thus we give label [1,0,1].

##  4 Experimental Setup

In this section, we present our experimental setup in detail. All of our
experiments follow the two steps as explained below.

Transformer Training: We use three different transformer models, namely
Multilingual-BERT, RoBERTa, and XLM-RoBERTa, made available by Hugging Face
Wolf et al. (2019), as our transformer architecture due to their widespread
usage in the context of code-mixed Tamil-English Offensive Language
Identification. In line with the works of Mosbach et al. (2021) all the models
were fine-tuned for 20 epochs, and the best performing checkpoint was
selected. Each transformer model takes 1 hour to train on a Tesla-V100 GPU
with a learning rate of 3x$10^{-5}$. Further, all of our experiments were run
five times with different random seeds and the results so reported are an
average of five runs. The relationship between the datasets used to train the
transformers across various experiments is as shown in Table 3. Meanwhile, the
model hyperparameters are presented in Table 4.

Span Extraction Testing: After training the transformer models for offensive
language identification, we use the test set from the offensive span
identification dataset for testing purposes. For LIME, we use individual
transformer models’ MASK token to mask out individual words and allow LIME to
generate 5000 masked samples per sentence. The resulting explanation weights
are then used as scores for each word, and tokens below the fixed decision
threshold of $\tau=-0.01$ are removed while the spans of the rest of the
comments are used for offensive span identification. Meanwhile, for the IG
model, for each sentence in the test set, we perform 50 iterations to generate
scores for each word and extract the spans in line with LIME.

##  5 Experiments, Results and Analysis

Experiments | Model |  | $F_{1}(\%)$
---
LIME | IG
Benchmark | BENCHMARK 1 | 39.834
BENCHMARK 2 | 37.024
OS-Baseline | BERT | 26.35 | 44.83
RoBERTa | 24.26 | 37.01
XLM-RoBERTa | 22.86 | 43.13
OS-Augmentation | BERT | 24.97 | 44.83
RoBERTa | 26.23 | 44.98
XLM-RoBERTa | 21.93 | 50.23
OS-Multilabel | BERT | 47.137 | 44.83
RoBERTa | 47.38 | 35.83
XLM-RoBERTa | 46.76 | 42.06

Table 5: Consolidated Results on Offensive Span Identification Dataset. All
the values represent character level $F_{1}$ measure.

The consolidated results are presented in Table 5. Each model is trained as an
offensive comment classifier and then evaluated for offensive span
identification. Though we do not explicitly furnish any signals regarding
which words are offensive, we can see an assortment of behaviors across both
the rationale extraction methods when trained differently. For reference
comparison, we also include two benchmark baseline models from Ravikiran et
al. (2022). BENCHMARK 1 is a random baseline model which haphazardly labels
50% of characters in comments to belong to be offensive inline. BENCHMARK 2 is
a lexicon-based system, which first extracted all the offensive words from the
train samples of offensive span identification dataset Ravikiran and Annamalai
(2021). These words were scoured in comments from the test set during
inference, and corresponding spans were noted. We report the character level
$F_{1}$ for extracted spans inline with Ravikiran et al. (2022).

Experiments | Model |  $F_{1}@30$ (%) |  $F_{1}@50$ (%) |  $F_{1}@>50$ (%)
---|---|---|---|---
LIME | IG | LIME | IG | LIME | IG
OS-Baseline | BERT | 47.02 | 45.79 | 32.54 | 50.62 | 23.27 | 42.48
RoBERTa | 37.35 | 36.42 | 32.75 | 42.04 | 20.56 | 34.95
XLM-RoBERTa | 43.05 | 51.7 | 31.54 | 48.69 | 18.63 | 40.49
OS-Augmentation | BERT | 48.45 | 45.79 | 32.62 | 50.62 | 21.29 | 42.48
RoBERTa | 50.21 | 45.86 | 32.71 | 50.71 | 22.8 | 42.66
XLM-RoBERTa | 31.19 | 59.47 | 27.58 | 57.01 | 19.17 | 47.19
OS-Multilabel | BERT | 45.7 | 45.79 | 57.15 | 50.62 | 42.722 | 42.48
RoBERTa | 58.66 | 45.86 | 57.19 | 50.71 | 43 | 42.66
XLM-RoBERTa | 59.84 | 59.47 | 57.62 | 57.01 | 42.95 | 47.19

Table 6: Results across different size of comments.

Category | Comment Type | Examples
---|---|---
Correct Prediction | Comments with less than 30 characters |  |  Dei like poda anaithu 9 p***a
---
Semma mokka and as usual a masala movie
Comments with 30-50 characters |  |  M***u adichutu sagunga da j***i p***********a
---
81k views 89k likes YouTube be like W*F
Comments with greater than 50 characters |  |  Old vijayakanth movie parthathu pola irruku..pidikala….
---
Dei Yappa munjha paarthu Sirichu Sirichu vayiru vazhikuthu
Incorrect Prediction | Comments with less than 30 characters |  |  except last scene its a crap
---
Movie is going to be disaster
Comments with 30-50 characters |  | Kandasamy and Mugamoodi mixed nu nenaikre….
---
Last la psycho ilayaraja nu solitan
Comments with greater than 50 characters |  | All I understood from this video was Vikram likes Dosai..
---
Padam nichiyam oodama poga neriya vaipu iruku poliye !  Oru dislike ah potu
vaipom

Table 7: Example of correct and incorrect predictions. Blue highlight shows
words attributed by LIME. Green highlight shows words attributed by IG. Pink
highlight shows words attributed by both LIME and IG. Yellow highlight shows
parts of comments annotated in ground truth but not identified by both LIME
and IG.

###  5.1 OS-Baseline Experiments

Firstly, both benchmarks exhibit high performance, making the task competitive
for LIME and IG methods. To start with, we analyze the results of OS-Baseline
experiments. From Table 5, we can see that LIME has moderately low performance
compared to IG, which either beats the baseline or produces very close
results. Analogizing the LIME and IG, we can see that IG has an average
difference of 18% compared to LIME. To understand this, we identify various
examples (Table 7) where LIME fails, and IG performs significantly well and
vice versa. Firstly, we can see that LIME explicitly focuses on identifying
overtly offensive words only. Besides, we can also see LIME focuses primarily
on offensive words, while IG accounts for terms such as "Dei", "understood",
"iruku poliye" etc.

Accordingly, to comprehend their performance on offensive comments of
different sizes, we separate results across (a) comments with less than 30
characters ($F_{1}$@30), (b) comments with 30-50 characters ($F_{1}$@50) (c)
comments with more than 50 characters ($F_{1}$@>50). The results so obtained
are as shown in Table 6. Accordingly, we find interesting outcomes. Firstly we
can see that though LIME has lower $F_{1}$ overall, it tends to show
competitive results against IG for comments with less than 30 characters.

With the increase in the comment length, the performance of LIME tends to
lower considerably. We believe such behavior of LIME could be because of two
reasons (a) surrogate models may not be strong enough to distinguish different
classes and (b) dilution of scores due to LIME’s random perturbation
procedure. With random perturbations, the instances generated may be quite
different from training instances drawn from the underlying distribution.
Meanwhile, IG is compatible across all the sizes, and in the case of comments
with less than 30 and 50 characters, we can see IG to show the result as high
as 50%.

###  5.2 OS-Augmentation Experiments

Since transformers are very sensitive to dataset size, we focus on estimating
the impact of dataset size used to train the transformers for offensive
comment classification on the performance of LIME and IG, respectively. To
this end, we used the Mask Augmented dataset to finetune the transformers and
pose the question Does adding data make any difference? The various result so
obtained are as shown in Table 5. Firstly, for LIME, we see no such drastic
difference in $F_{1}$. However, for IG, we can see a significant improvement,
especially for RoBERTa and XLM-RoBERTa models. Specifically, we can see the
XLM-RoBERTa model to reach an accuracy of 50.23% with an average of 12% higher
results compared to benchmark models and 7% compared to OS-Baseline.

Furthermore, analysis of results shows a couple of fascinating characteristics
for XLM-RoBERTa. Firstly, we could see many predictions concentrating on words
part of the long offensive span annotations. We believe this is because of the
ability of the model to learn relations between words in different languages
as part of its pretraining, which is not the case with M-BERT and RoBERTa. To
verify this again, we separate the results across different comment sizes.
From Table 6 we can see that for longer sized comments, the model tends to
outperform M-BERT, RoBERTa when coupled with IG. Meanwhile, LIME has no
changes irrespective of used transformers.

###  5.3 OS-Multilabel Experiments

Finally, we analyze the significance of encoding the position of offensive
words as part of the training process. To this end, we ask Does introducing
position information as part of the training process improve zero-shot
results?. As such, we use the multilabel dataset to finetune the transformers
to obtain results, as shown in Table 5. Firstly, we can see that introducing
multiple labels for training has no impact on the overall results of LG.
However, we can see that LIME demonstrates a significant gain in overall
results. Specifically, with multilabel training, the baseline results improve
by 20% to 47.38%.

Furthermore, we can observe an equivalent trend across the different sizes of
comments as seen in Table 6. In fact, for words of less than 30 and 50
characters, LIME outdoes IG models, which aligns with our hypothesis that the
position is helpful. Overall from all the results from Table 5-6 we can see
XLM-RoBERTa be more suitable for extracting spans, especially with the
addition of more data and position information. Meanwhile, IG is consistent in
producing explanations irrespective of dataset size or training approach.

##  6 Conclusion

This work examines rationale extraction methods for inferring offensive spans
from the transformer model trained for offensive sentence classification.
Experiments revealed that approaches such as LIME do not perform as well when
applied to transformers directly, attributing to potential issues with
surrogate models and perturbation procedures. Meanwhile, we can see IG as the
clear front runner for identifying offensive spans in a zero-shot way. We
think this is due to the inherent nature of the method, where it focuses on
creating the input at the same time learning the reason for offensiveness.

Besides, we also analyzed LIME and IG under large datasets and incorporated
position information in the training process. To this end, we discovered that
only augmenting does not improve the performance of LIME. However, when this
large data is coupled with labels incorporating position information, both LG
and IG improve significantly. Especially LIME prefers this approach with large
improvements on $F_{1}$, despite IG outperforming LIME.

Additionally, we also found XLM-RoBERTa to be a clear winner among the
transformer models owing to its intrinsic learning of relationships which
potentially helps with comments that are longer size. However, many details
were unexplored, including (i) the effect of random perturbations on overall
results (ii) the approach to merge attributions of multilabel predictions,
which we plan to explore in the immediate future.

## Acknowledgements

We thank our anonymous reviewers for their valuable feedback. Any opinions,
findings, and conclusion or recommendations expressed in this material are
those of the authors only and does not reflect the view of their employing
organization or graduate schools. The work is the part of the final project in
CS7643-Deep Learning class at Georgia Tech (Spring 2022). Bharathi Raja
Chakravarthi were supported in part by a research grant from Science
Foundation Ireland (SFI) under Grant Number SFI/12/RC/2289$\\_$P2
(Insight$\\_$2), co-funded by the European Regional Development Fund and Irish
Research Council grant IRCLA/2017/129 (CARDAMOM-Comparative Deep Models of
Language for Minority and Historical Languages).

## References

  * Camburu et al. (2018) Oana-Maria Camburu, Tim Rocktäschel, Thomas Lukasiewicz, and Phil Blunsom. 2018.  e-snli: Natural language inference with natural language explanations.  _CoRR_ , abs/1812.01193.
  * Chakravarthi et al. (2021a) Bharathi Raja Chakravarthi, Ruba Priyadharshini, Navya Jose, Anand Kumar M, Thomas Mandl, Prasanna Kumar Kumaresan, Rahul Ponnusamy, Hariharan R L, John P. McCrae, and Elizabeth Sherly. 2021a.  Findings of the shared task on offensive language identification in Tamil, Malayalam, and Kannada.  In _Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages_ , pages 133–145, Kyiv. Association for Computational Linguistics.
  * Chakravarthi et al. (2021b) Bharathi Raja Chakravarthi, Ruba Priyadharshini, Vigneshwaran Muralidaran, Navya Jose, Shardul Suryawanshi, Elizabeth Sherly, and John P. McCrae. 2021b.  Dravidiancodemix: Sentiment analysis and offensive language identification dataset for dravidian languages in code-mixed text.  _CoRR_ , abs/2106.09460.
  * Conneau et al. (2020) Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzmán, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020.  Unsupervised cross-lingual representation learning at scale.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020_ , pages 8440–8451. Association for Computational Linguistics.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers)_ , pages 4171–4186. Association for Computational Linguistics.
  * Ding and Jurgens (2021) Huiyang Ding and David Jurgens. 2021.  HamiltonDinggg at SemEval-2021 task 5: Investigating toxic span detection using RoBERTa pre-training.  In _Proceedings of the 15th International Workshop on Semantic Evaluation (SemEval-2021)_ , pages 263–269, Online. Association for Computational Linguistics.
  * Ke et al. (2021) Guolin Ke, Di He, and Tie-Yan Liu. 2021.  Rethinking positional encoding in language pre-training.  _ArXiv_ , abs/2006.15595.
  * Lan and Xu (2018) Wuwei Lan and Wei Xu. 2018.  Neural network models for paraphrase identification, semantic textual similarity, natural language inference, and question answering.  In _Proceedings of the 27th International Conference on Computational Linguistics_ , pages 3890–3902, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized BERT pretraining approach.  _CoRR_ , abs/1907.11692.
  * Mosbach et al. (2021) Marius Mosbach, Maksym Andriushchenko, and Dietrich Klakow. 2021.  On the stability of fine-tuning BERT: misconceptions, explanations, and strong baselines.  In _9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021_. OpenReview.net.
  * Ravikiran and Annamalai (2021) Manikandan Ravikiran and Subbiah Annamalai. 2021.  DOSA: Dravidian code-mixed offensive span identification dataset.  In _Proceedings of the First Workshop on Speech and Language Technologies for Dravidian Languages_ , pages 10–17, Kyiv. Association for Computational Linguistics.
  * Ravikiran et al. (2022) Manikandan Ravikiran, Bharathi Raja Chakravarthi, Anand Kumar Madasamy, Sangeetha Sivanesan, Ratnavel Rajalakshmi, Sajeetha Thavareesan, Rahul Ponnusamy, and Shankar Mahadevan. 2022.  Findings of the shared task on Offensive Span Identification in code-mixed Tamil-English comments.  In _Proceedings of the Second Workshop on Speech and Language Technologies for Dravidian Languages_. Association for Computational Linguistics.
  * Ribeiro et al. (2016a) Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016a.  "why should I trust you?": Explaining the predictions of any classifier.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, San Francisco, CA, USA, August 13-17, 2016_ , pages 1135–1144. ACM.
  * Ribeiro et al. (2016b) Marco Túlio Ribeiro, Sameer Singh, and Carlos Guestrin. 2016b.  "why should I trust you?": Explaining the predictions of any classifier.  In _Proceedings of the Demonstrations Session, NAACL HLT 2016, The 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, San Diego California, USA, June 12-17, 2016_ , pages 97–101. The Association for Computational Linguistics.
  * Shaw et al. (2018) Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018.  Self-attention with relative position representations.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_ , pages 464–468, New Orleans, Louisiana. Association for Computational Linguistics.
  * Sikdar et al. (2021) Sandipan Sikdar, Parantapa Bhattacharya, and Kieran Heese. 2021.  Integrated directional gradients: Feature interaction attribution for neural NLP models.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 865–878, Online. Association for Computational Linguistics.
  * Sundararajan et al. (2017) Mukund Sundararajan, Ankur Taly, and Qiqi Yan. 2017.  Axiomatic attribution for deep networks.  In _Proceedings of the 34th International Conference on Machine Learning, ICML 2017, Sydney, NSW, Australia, 6-11 August 2017_ , volume 70 of _Proceedings of Machine Learning Research_ , pages 3319–3328. PMLR.
  * Thorne et al. (2019) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2019\.  Generating token-level explanations for natural language inference.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 963–969, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, and Jamie Brew. 2019.  Huggingface’s transformers: State-of-the-art natural language processing.  _CoRR_ , abs/1910.03771.
  * Xu et al. (2021) Peng Xu, Dhruv Kumar, Wei Yang, Wenjie Zi, Keyi Tang, Chenyang Huang, Jackie Chi Kit Cheung, Simon J.D. Prince, and Yanshuai Cao. 2021.  Optimizing deeper transformers on small datasets.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_ , pages 2089–2102, Online. Association for Computational Linguistics.