# The new heavy fermion compound Ce3Bi4Ni3

D. M. Kirschbaum  Institute of Solid State Physics, Vienna University of
Technology, 1040 Vienna, Austria  X. Yan  Institute of Solid State Physics,
Vienna University of Technology, 1040 Vienna, Austria  M. Waas  Institute of
Solid State Physics, Vienna University of Technology, 1040 Vienna, Austria  R.
Svagera  Institute of Solid State Physics, Vienna University of Technology,
1040 Vienna, Austria  A. Prokofiev  Institute of Solid State Physics, Vienna
University of Technology, 1040 Vienna, Austria  B. Stöger  X-ray Center,
Vienna University of Technology, 1040 Vienna, Austria  P. Rogl  Institute of
Materials Chemistry, University of Vienna, 1090 Vienna, Austria  D.-G. Oprea
Max Planck Institute for Chemical Physics of Solids, 01187 Dresden, Germany
Institute for Theoretical Physics, Goethe University Frankfurt, 60438
Frankfurt a.M., Germany  C. Felser  Max Planck Institute for Chemical Physics
of Solids, 01187 Dresden, Germany  R. Valentí  Institute for Theoretical
Physics, Goethe University Frankfurt, 60438 Frankfurt a.M., Germany  M. G.
Vergniory  Max Planck Institute for Chemical Physics of Solids, 01187 Dresden,
Germany  Donostia International Physics Center, 20018 Donostia–San Sebastian,
Spain  J. Custers  Dept. of Condensed Matter Physics, Charles University, 121
16 Prague, Czech Republic  S. Paschen<EMAIL_ADDRESS>Institute of
Solid State Physics, Vienna University of Technology, 1040 Vienna, Austria  D.
A. Zocco<EMAIL_ADDRESS>Institute of Solid State Physics, Vienna
University of Technology, 1040 Vienna, Austria

###### Abstract

The family of cubic noncentrosymmetric 3-4-3 compounds has become a fertile
ground for the discovery of novel correlated metallic and insulating phases.
Here, we report the synthesis of a new heavy fermion compound, Ce3Bi4Ni3. It
is an isoelectronic analog of the prototypical Kondo insulator Ce3Bi4Pt3 and
of the recently discovered Weyl-Kondo semimetal Ce3Bi4Pd3. In contrast to the
volume-preserving Pt-Pd substitution, structural and chemical analyses reveal
a positive chemical pressure effect in Ce3Bi4Ni3 relative to its heavier
counterparts. Based on the results of electrical resistivity, Hall effect,
magnetic susceptibility, and specific heat measurements, we identify an energy
gap of 65–70 meV, about 8 times larger than that in Ce3Bi4Pt3 and about 45
times larger than that of the Kondo-insulating background hosting the Weyl
nodes in Ce3Bi4Pd3. We show that this gap as well as other physical properties
do not evolve monotonically with increasing atomic number, i.e., in the
sequence Ce3Bi4Ni3-Ce3Bi4Pd3-Ce3Bi4Pt3, but instead with increasing partial
electronic density of states of the $d$ orbitals at the Fermi energy. To
understand under which condition topological states form in these materials is
a topic for future studies.

##  I Introduction

The prospect that strong electron correlations and nontrivial electronic
topology could cooperate in the formation of novel quantum states of matter
has sparked great interest in recent years. Much attention has been given to
heavy fermion materials, with proposals to realize correlated topological
states in Kondo insulators and semimetals [1, 2, 3]. In the family of cubic,
noncentrosymmetric Ce-based “3-4-3” compounds of space group 220, Ce3Bi4Pt3 is
perhaps the best-known case. It was classified early on as a Kondo insulator
(KI) [4, 5, 6], a material where the Kondo interaction between localized
(typically 4$f$) states and conduction electrons (typically from $d$ shells)
promotes the formation of a narrow band gap [7]. More recently it was found
that the isoelectronic and isosize substitution of Pt by Pd consecutively
reduces the gap, ultimately giving rise to a novel Weyl-Kondo semimetal (WKSM)
state in the end compound Ce3Bi4Pd3 [8, 9, 10].

Magnetic field experiments showed that the WKSM (gapless) signatures observed
in Ce3Bi4Pd3 can be readily suppressed at a modest field of 9 T while
maintaining the Kondo interaction essentially intact [11, 12, 13]. The
background Kondo gap, albeit small, remains finite up to a considerably larger
magnetic field, as demonstrated from magnetic torque and susceptibility
experiments [11, 14]. High pressure studies indicate that a larger gap value
similar to that of Ce3Bi4Pt3 is gradually recovered [15]. Yet, a detailed
understanding of how the WKSM state is affected by volume and hybridization
changes produced by chemical substitutions [8, 16] or externally applied
pressure [15, 17, 18] is currently lacking.

Here we extend the previous isoelectronic substitution study (5$d$ to 4$d$)
one step further, to the Ni case (3$d$). We present the synthesis of high-
quality single crystals of Ce3Bi4Ni3, a material that has not been synthesized
in any form before. Structural analysis reveals that the Ni substitution
results in a positive chemical pressure, distinct from the volume-preserving
Pt-Pd substitution, which gives new insight into the pressure effects on
Ce3Bi4Pd3. Electrical resistivity, Hall effect, magnetic susceptibility, and
specific heat measurements suggest that Ce3Bi4Ni3 is a new heavy fermion
compound with a hybridization gap that is large compared to those of the Pd
and Pt compounds. We end by discussing the evolution of the characteristic
energy scales across the Ce3Bi${}_{4}X_{3}$ series ($X$ = Ni, Pd, Pt), and
relate it to the results of electronic band structure calculations.

##  II Single crystal growth and characterization

Materials with the chemical formula $R_{3}Pn_{4}X_{3}$, where $R$ is a rare
earth or actinide element, $Pn$ is a pnictide (As, Sb, Bi), and $X$ is a
transition metal element, have been investigated for decades [19, 20, 21, 22].
Among the Ce compounds, the Kondo insulator Ce3Bi4Pt3 has been known for
decades [4, 5, 6]. Single crystals of the isoelectronic compound Ce3Bi4Pd3
were recently obtained via the Bi-flux technique [8], but the synthesis of the
Ni phase has remained elusive so far. Our initial attempts to prepare
polycrystalline samples confirmed that the Ce3Bi4Ni3 phase indeed forms, and
encouraged us to attempt single crystal growth using various fluxes. A
detailed description of the sample synthesis is presented in Appendix A.

Figure 1: (a) Photo of a single crystal of Ce3Bi4Ni3 with indicated
crystallographic directions obtained from Laue diffraction. Typical sample
length is 1.5 mm. (b) Laue XRD pattern (white spots). The center of the image
corresponds to the [-2, -1, -1] crystallographic direction. (c) Sketch of the
unit cell. The polyhedron emphasizes the close environment of a Ce atom. (d)
Room-temperature powder XRD pattern of Ce3Bi4Ni3 (black dots). Vertical bars
indicate the positions of the expected diffraction peaks. The difference (blue
line) between the measured data and the refinement (red line) shows no signs
of impurity phases. (e) Lattice parameter $a$ across the Ce3Bi4(Ni,Pd,Pt)3
series. Intermediate Pd-Pt values (white diamonds) taken from Ref. [8]. The
grey shaded line is a guide to the eye.

Single crystals of Ce3Bi4Ni3 with faceted surfaces and maximum sizes of 2
$\times$ 0.5 $\times$ 0.5 mm3 were successfully grown using Pb flux [Fig.
1(a)]. Laue X-ray diffraction (XRD) provides clear patterns, confirming the
high quality of the crystals [Fig. 1(b)]. Similarly to the Pt and Pd 3-4-3
compounds, the powder and single-crystal XRD data of Ce3Bi4Ni3 can be refined
with the cubic Y3Sb4Au3 structure, space group 220, $I\overline{4}$3$d$ [23,
19]. The unit cell contains 4 formula units, with 40 atoms forming a
noncentrosymmetric structure. Figure 1(c) displays the crammed unit cell; the
local environment of a Ce atom is comprised of 8 Bi atoms and 4 Ni atoms
(nearest neighbors, see Appendix B, Table B). Rietveld refinements of the
powder XRD data of the most stoichiometric samples [Fig. 1(d)] yield an
average room-temperature lattice parameter $a$ = 9.7715(9) Å (see Appendix C).
The unit cell volume of Ce3Bi4Ni3 is therefore compressed by $\sim$ 8% with
respect to the (essentially isovolume [8]) Pt ($a$ = 10.051 Å) and Pd ($a$ =
10.058 Å) compounds [Fig. 1(e)]. These results were confirmed and studied in
more detail with single crystal XRD experiments performed at 100 K, 200 K, and
300 K. A summary of the structural data obtained from a highly stoichiometric
sample, including interatomic distances, atomic occupancies, and atomic
displacement parameters (ADPs), is presented in Appendix B.

Figure 2: EDX spectrum corresponding to one point measurement of a polished
Ce3Bi4Ni3 crystal (inset, top right). Inset: backscattered SEM images of as-
grown (bottom left) and polished (top right) Ce3Bi4Ni3 samples. The lighter
spots visible in the as-grown crystal correspond to Pb inclusions, and are
absent in the polished sample.

The stoichiometries of the samples were determined via energy dispersive X-ray
spectroscopy (EDX) after each growth campaign. Figure 2 displays a
representative EDX spectrum collected from the polished surface of a high-
quality single crystal. The mean composition Ce3.15(5)Bi3.90(9)Ni2.94(4) was
obtained from spectra taken at 6 different spots of a sample. Scanning
electron microscopy (SEM) images obtained from as-grown single crystals (inset
of Fig. 2, bottom left) show lighter spots, which could originate from
residual flux solidifying after centrifuging. In high-quality single crystals,
these inclusions are only present at the surface of the samples, and can
therefore be removed by etching with hydrogen peroxide (H2O2) and careful
polishing (inset of Fig. 2, top right). Spurious superconductivity can
therefore be prevented or eliminated, as we confirm by magnetic susceptibility
and electrical resistivity measurements. A possible substitution of Bi by Pb
in Ce3Bi4Ni3 has been ruled out by comparing the EDX spectra of a Pb-flux
grown single crystal and that of a Pb-free annealed polycrystalline sample.

Samples of Ce3Bi4Ni3 are sensitive to air and moisture. A distinct degradation
of the powder XRD intensity was observed after a 5 h exposure of the powder to
air. The oxygen contamination can also be detected in the EDX spectrum after a
short exposure time (small peak in the low energy region of Fig. 2).

From a combined analysis of the EDX, powder XRD, and magnetic susceptibility
measurements, we can select with good certainty the most stoichiometric, high-
quality single crystals. A detailed discussion of this analysis is presented
in Appendix C. In the following sections we present the physical properties of
Ce3Bi4Ni3 corresponding to samples with Ni content closest to 3 atoms per mol.

##  III Physical properties

The electrical resistivity $\rho(T)$ of Ce3Bi4Ni3 is displayed in Fig. 3(a).
Similarly to the Pt and Pd 3-4-3 compounds, nonmetallic behavior is observed
across the entire studied temperature range (a detailed comparison is
presented in Section IV). $\rho$ increases by more than two orders of
magnitude from room temperature to 2 K, with a resulting inverse residual
resistivity ratio $iRRR$ = $\rho$(2 K)/$\rho$(300 K) = 440. Above 100 K,
$\rho(T)$ can be well described by a thermally activated scattering process
across a gap. At lower temperatures, the systems enters a regime where the
increase in resistivity with decreasing temperature is less pronounced. First,
a shoulder develops at approximately 50 K, followed by a second, low-
temperature thermally activated regime between 10 K and 30 K. A small energy
gap of $\sim$ 20 K could be extracted from this narrow temperature range,
although one might argue that other effects, such as the presence of in-gap
states, could be responsible for this particular temperature dependence of the
resistivity. Finally, $\rho(T)$ begins to flatten below 10 K, but without
reaching saturation down to 2 K.

Figure 3: (a) Electrical resistivity $\rho$ of Ce3Bi4Ni3 as a function of
temperature $T$ in zero magnetic field. (b) Temperature-dependent Hall
coefficient $R_{\mathrm{H}}$. The shaded grey line is a guide to the eye.
(c),(d) Arrhenius plots of the resistivity (normalized to 300 K) and Hall
coefficient data. The red solid lines are linear fits to the data above 100 K.

Figure 3(b) displays the temperature evolution of the Hall coefficient
$R_{\mathrm{H}}$, obtained from linear fits to the low-field region of
isothermal Hall resistivity curves (linear in $B\leq 9$ T for 10 K $\leq$ $T$
$\leq$ 300 K). $R_{\mathrm{H}}$($T$) reaches a local maximum near 50 K and
tends to saturate at lower temperatures. This peak coincides with the end of
the thermally activated regime. The increase of $R_{\mathrm{H}}(T)$ below 10 K
might be due to the freezing out of thermally activated carriers from shallow
in-gap states, the origin of which needs further investigation.

To determine the size of the gap $\Delta$, the transport data are fitted with
thermally activated behavior, $\rho=\rho_{0}\exp[\Delta/(2k_{\mathrm{B}}T)]$
and $R_{\rm H}=R_{\rm H,0}\exp[\Delta/(2k_{\mathrm{B}}T)]$, as shown in
Arrhenius plots [Fig. 3(c, d)]. $\Delta$ values of 759 K and 807 K are
obtained from the fits of the resistivity and Hall coefficient above 100 K,
respectively. These values are 7–10 times larger than the values reported for
Ce3Bi4Pt3 [5, 8] and at least 45 times larger than the Kondo gap acting on the
background of the WKSM state of Ce3Bi4Pd3 in the absence of applied magnetic
field [8, 11], suggesting the presence of a much larger Kondo energy scale at
play in the Ni compound.

Figure 4: (a) Temperature dependence of the magnetic susceptibility $\chi$ of
Ce3Bi4Ni3, measured with $B$ = 1 T. (b) Inverse magnetic susceptibility
$\chi^{-1}(T)$. The paramagnetic Weiss temperate $\Theta$ and the effective
magnetic moment $\mu_{\mathrm{eff}}$ were determined from a Curie-Weiss fit
for $T>T^{\chi}_{\mathrm{max}}$ (red line). (c) Magnetization $M$ vs $B$ at 2
K (black), 10 K (blue), 100 K (green) and 300 K (red). The curves show up and
down $B$-sweeps, with no signs of magnetic hysteresis detected.

The magnetic susceptibility $\chi(T)$ of Ce3Bi4Ni3 displays a broad maximum at
$T^{\chi}_{\mathrm{max}}$ = 280 K [Fig. 4(a)]. In Kondo insulators, the
occurrence of such maxima has been associated with the onset of Kondo
screening and the opening of a hybridization gap [7]. We estimate a Kondo
temperature using the Kondo lattice expression $T_{\mathrm{K}}\approx
4T^{\chi}_{\mathrm{max}}/(2J+1)=187$ K [24, 5], with a total angular momentum
$J=5/2$ for Ce3+. The upturn in $\chi$ vs $T$ observed at lower temperatures
is generally attributed to small amounts of magnetic impurities or defects
[25] which were not detected in our XRD experiments. Assuming Curie-Weiss
behavior for $T>T^{\chi}_{\mathrm{max}}$, a linear fit to the inverse
susceptibility $\chi^{-1}(T)$ in the limited available temperature range [Fig.
4(b)] yields a large negative Weiss temperature $\Theta\sim-1084$ K —typical
for Kondo insulators— and an effective magnetic moment $\mu_{\mathrm{eff}}\sim
2.37\mu_{\mathrm{B}}$, slightly reduced from the Hund’s rules value of
$2.54\mu_{\mathrm{B}}$ for free Ce3+ ions.

The magnetization curves $M(B)$ are linear in field at 10 K, 100 K, and 300 K
[Fig. 4(c)]. A slight curvature appears only in the 2 K measurement, without
reaching saturation up to 9 T. No indications of magnetic hysteresis were
observed in any of the measurements. Overall, Ce3Bi4Ni3 appears to be well
described as a paramagnetic Kondo insulator, without magnetic ordering down to
2 K.

Figure 5: Specific heat $C$ vs $T$ of Ce3Bi4Ni3 measured between 2 K and 300
K. The red dashed line is a fit with the Debye model plus an additional
Schottky-like electronic contribution derived for Kondo insulators (see main
text). The inset shows the low temperature $C/T$ vs $T^{2}$ and the fit. The
Sommerfeld coefficient ($\gamma$) is then determined by shifting the fitting
line by $\gamma=6.92$ mJ/molCeK2 to match the measured data (blue line).

We can gain further understanding on the nature of the ground state of
Ce3Bi4Ni3 and the strength of the Kondo interaction from the specific heat
$C(T)$ measurements presented in Fig. 5. So far, our efforts to synthesize
La3Bi4Ni3 as a non-$f$-electron reference compound have been unsuccessful. To
isolate the electronic and magnetic (Ce 4$f$ related) contributions to the
specific heat, we therefore model the data as $C(T)=\gamma T+C_{\rm
ph}+C_{\Delta}$. The first term is the linear-in-temperature electronic
(Sommerfeld) contribution, the second term the phonon specific heat described
by the Debye model [26]. $C_{\Delta}$ accounts for an additional Schottky-like
electronic contribution in Kondo insulators, which has previously been
modelled with two sharp peaks in the density of states, located above and
below the energy gap [7]:

$C_{\Delta}\propto\left(\frac{\Delta}{2k_{\mathrm{B}}T}\right)^{2}\frac{(2J+1)\exp\left[\Delta/2k_{\mathrm{B}}T\right]}{\left(2J+1+\exp\left[\Delta/2k_{\mathrm{B}}T\right]\right)^{2}}\;.$
(1)

The contribution from the Sommerfeld term is expected to be very small. In
addition, its extraction is complicated by a low-temperature upturn in $C/T$
(see inset of Fig. 5). We therefore set $\gamma=0$ and fit the data above 20 K
using the full Debye integral. The fit yields a Debye temperature $\Theta_{\rm
D}=(202\pm 3)$ K and a large gap $\Delta=(1693\pm 87)$ K, consistent with the
large value of $T^{\chi}_{\mathrm{max}}$ presented above. We can nevertheless
obtain an estimate of the Sommerfeld coefficient $\gamma$ by matching the fit
to the measured data in a $C/T$ vs $T^{2}$ plot at low temperatures (blue line
in inset of Fig. 5). The required shift along the $C/T$ axis then corresponds
to $\gamma=6.92$ mJ/molCeK2, similarly small to the value determined for
Ce3Bi4Pt3 [5]. Ce3Bi4Ni3 displays a small low-temperature upturn visible in
$C/T$ which could be due to magnetic impurities, as we previously discussed
for the magnetic susceptibility data (Fig. 4). However, the upturn could also
be related to the onset of non-Fermi liquid behavior in the presence of a
nearby quantum critical point. Additional low-temperature investigations are
needed to unveil its true origin.

##  IV Discussion

We can now take a look at Ce3Bi4Ni3 in the context of the Ce3Bi${}_{4}X_{3}$
series ($X$ = Ni, Pd, Pt). The lattice parameters $a$ for the three compounds
are shown in Fig. 1(e). In addition, we include the values corresponding to
the intermediate Pt-Pd substitution Ce3Bi4(Pt1-xPdx)3 from Ref. [8] and one
value corresponding to the intermediate Pd-Ni substitution Ce3Bi4(Pd1-xNix)3
with $x=0.19$. Unlike the Pt-Pd case, which appears to exert very little to
no-effect on the unit cell volume [8], the substitution of Pd with Ni is no
longer isosize. In this context, chemical (and, possibly, external) pressure
becomes now a relevant parameter for tuning the Kondo coupling
$J_{\mathrm{K}}$ across the extended Ce 3-4-3 series.

A recent study on the electronic transport properties of Ce3Bi4Pd3 under
hydrostatic pressure showed that the energy gap extracted from resistivity
increases quadratically with applied pressure, reaching a value of $\sim$ 4.8
meV at 2.3 GPa [15, 27]. Hence, pressure appears to drive Ce3Bi4Pd3 to a more
insulating state. In this work [15], as well as in two recent theoretical
studies [17, 18], it was thus argued that pressure drives Ce3Bi4Pd3 towards
Ce3Bi4Pt3, which, at that point, was the only known isoelectronic compound of
this series. In view of the chemical pressure effect presented in Fig. 1(e),
here we argue that, instead, pressure drives Ce3Bi4Pd3 towards the Ni
compound. We can translate the 8.3% shrinking of the unit cell volume obtained
from Ce3Bi4Pd3 to Ce3Bi4Ni3 to an equivalent externally applied pressure $P$
using the third-order Birch-Murnaghan equation of state [28]

$P(V)=\frac{3B_{0}}{2}\left[\left(\frac{V_{0}}{V}\right)^{\frac{7}{3}}-\left(\frac{V_{0}}{V}\right)^{\frac{5}{3}}\right]\times\left\\{1+\frac{3}{4}(B^{\prime}_{0}-4)\left[\left(\frac{V_{0}}{V}\right)^{\frac{2}{3}}-1\right]\right\\},$
(2)

where $V_{0}$ is the ambient pressure volume, $V$ is the compressed volume,
$B_{0}$ is the bulk modulus, and $B^{\prime}_{0}$ its derivative with respect
to pressure. $B_{0}$ and $B^{\prime}_{0}$ result typically from fits to $V(P)$
data obtained in high-pressure X-ray diffraction experiments. Since such data
are not yet available for Ce3Bi4Pd3, we use the values $B_{0}=59.1$ GPa and
$B^{\prime}_{0}=6.9$ reported for the isostructural (and isosize) compound
Ce3Bi4Pt3 [29], which provide also a good agreement with the pressure-
dependent unit cell parameters calculated via DFT+U [18]. This calculation
yields a pressure of 6.9 GPa equivalent to the compression effect resulting
from the Pd-Ni substitution. Using the quadratic dependence of $\Delta(P)$
from the experiments reported in Ref. [15, 27], a pressure of 9.4 GPa would
then be necessary to reach the energy gap value estimated for Ce3Bi4Ni3 (Fig.
3), close to the value calculated using the Birch-Murnaghan formula.

Figure 6: (a) Electrical resistance $R(T)$ normalized to its room temperature
value $R$(300 K) and (b) magnetic susceptibility $\chi(T)$ of the
Ce3Bi${}_{4}X_{3}$ series, with $X$ = Ni (black dots), Pd (blue line), Pt (red
line), and of a partially Ni-Pd single crystal (19% Ni, white triangles). The
data for Ce3Bi4Pd3 and Ce3Bi4Pt3 were taken from Refs. [8] and [10]. Figure 7:
Comparison of characteristic parameters for the Ce3Bi${}_{4}X_{3}$ series
($X=$ Ni, Pd, Pt). Shaded grey areas are guides to the eyes. (a) $iRRR$ values
(from Fig. 6) and temperature scales $T_{\mathrm{K}}$,
$\Delta/k_{\mathrm{B}}$, and $T^{\chi}_{\mathrm{max}}$. For Ce3Bi4Ni3,
$T^{\chi}_{\mathrm{max}}$ is taken from Fig. 4(a), $T_{\mathrm{K}}$ is
calculated from $T^{\chi}_{\mathrm{max}}$ (see Section III), and
$\Delta/k_{\mathrm{B}}$ is obtained from the fits in Figs. 3(c-d). Values for
the Pd and Pt compounds taken from Ref. [8]. (b) Kondo energy gap
$\Delta/k_{\mathrm{B}}$ as a function of the $d$-electron density of states
($d$ DOS) at the Fermi level (from DFT calculations, see Appendix D).

In Fig. 6 we compare the normalized resistances $R(T)/R(\mathrm{300\,K})$ and
magnetic susceptibilities of the Ni, Pd, and Pt compounds. Perhaps
surprisingly at first sight, the transport and magnetic properties of this
isoelectronic series evolve nonmonotonically with atomic number. The $iRRR$
decreases by more than 2 orders of magnitude from Ce3Bi4Ni3 to Ce3Bi4Pd3, and
recovers the value of the Ni compound when Pd is substituted by Pt. A slight
change in resistivity from the pure Pd compound is already evident when Pd is
substituted by 19% Ni [blue line vs white triangles in Fig. 6(a)]. Similarly,
the characteristic peak in susceptibility at $T^{\chi}_{\mathrm{max}}$
[indicated by arrows in Fig. 6(b)] shifts first from room temperature to
$\sim$ 12 K from the Ni to the Pd compound, and increases back to $\sim$ 75 K
for Ce3Bi4Pt3.

To help visualizing the evolution across the series, we present in Fig. 7(a)
(notice the logarithmic vertical axis) the values of $T^{\chi}_{\mathrm{max}}$
and $iRRR$ extracted from Fig. 6, together with the values
$\Delta/k_{\mathrm{B}}$ and $T_{\mathrm{K}}$ obtained as described in Section
III. As we mentioned earlier, the transport properties of Ce3Bi4Pd3 at low
temperatures and zero magnetic field are largely dominated by the effects of
the gapless WKSM ground state [8, 10, 11]. For this reason, as noted in Ref.
[8], the finite energy gap value of 16.9 K for Ce3Bi4Pd3 obtained via
Arrhenius fits to the $B=0$ resistivity data should be taken with caution, and
therefore it is presented as a shaded/dotted line symbol in Fig. 7.
Nevertheless, the value is in good agreement with magnetic field-derived
energy of $\sim$ 1.5 meV required to close the Kondo gap [11].

Overall, the nonmonotonic change of the characteristic parameters form a “U”
shape centered around the most strongly correlated material in the series
Ce3Bi4Pd3, and with the comparably less correlated, large Kondo-gap compounds
Ce3Bi4Pt3 and Ce3Bi4Ni3 lying at either side. While pressure arguments alone
could suffice, as discussed above, to broadly describe the Ni-Pd side of this
plot, they cannot explain the changes in Kondo coupling observed under the
volume-preserving Pd-Pt substitution, and, therefore, other arguments are
necessary to understand the entire nonmonotonic evolution found across the
series. A recent analysis of electronic band structure calculations suggested
that the radial extension of the $d$ orbitals could play an important role in
the determination of the Kondo coupling in these hybridization-gapped
materials [16]. Our density functional theory (DFT) calculations of the whole
series appears to confirm this hypothesis. The aim of these calculations is to
extract the $d$-electron density of states at the Fermi level, a key factor
determining the Kondo interaction strength. As the Kondo effect itself cannot
be captured by DFT, the $4f$ electron of Ce is treated as core electron (see
Appendix D for more details). Figure 7(b) displays the Kondo scale
$\Delta/k_{\mathrm{B}}$ as a function of the $d$-orbital electron density of
states ($d$ DOS) at the Fermi level for the three Ce3Bi${}_{4}X_{3}$ compounds
(see Appendix D). The Kondo scale increases exponentially with the $d$
DOS($E_{\mathrm{F}}$) (shaded grey line in the semi-log plot), but it does it
nonmonotonically with the principal atomic number $n$, increasing from the Pd
4$d$ to the Pt 5$d$ compounds, and reaching its maximum for the 3$d$
Ce3Bi4Ni3. Hence, we conclude that the nonmonotonic evolution of the physical
properties across the isoelectronic series can be explained by the
relationship between Kondo coupling and $d$-orbital density of states.

##  V Conclusions

In summary, we successfully synthesized single crystals of Ce3Bi4Ni3 using Pb
flux. The high quality of the samples was verified via XRD and EDX
measurements. From electrical resistivity, Hall effect, magnetic
susceptibility, and specific heat experiments, we conclude that Ce3Bi4Ni3 is a
paramagnetic heavy fermion material with a relatively large Kondo gap of 65–70
meV. In the context of the Ce3Bi${}_{4}X_{3}$ series ($X$ = Ni, Pd, Pt) we
find a positive chemical pressure effect in Ce3Bi4Ni3 relative to the heavier
Pd and Pt compounds. Notably, the transport and magnetic properties as well as
the extracted characteristic energy scales evolve nonmonotonically throughout
this isoelectronic series. We attribute this behavior to the nonmonotonic
variation of $d$-orbital DOS at the Fermi level. Obtaining a detailed
understanding of how the variations in orbital hybridization affect the Kondo
coupling across the Ni-Pd-Pt isoelectronic series would have a broader impact
for future experimental and theoretical studies. For example, the giant
topological responses found in various physical properties of Ce3Bi4Pd3 arise
from strongly renormalized Weyl bands located within a Kondo insulating gap
[8, 9, 10], which in turn acts in the background as a shield against the
effects of topologically trivial bands. It would then be interesting to
explore how the WKSM signatures observed in Ce3Bi4Pd3 will evolve as the band
structure is carefully tuned via various Ni-Pd-Pt substitutions, and as a
function of externally applied pressure.

###### Acknowledgements.

The work in Vienna was supported by the Austrian Science Fund (FWF projects I
4047-N27 and I 5868-FOR 5249 - QUAST), the European Microkelvin Platform
(H2020 project 824109), and the European Research Council (ERC Advanced Grant
101055088, CorMeTop). The work in Prague was supported by the Czech Science
Fund (GAČR project No. 19-29614L). M.G.V. acknowledges support to the Spanish
Ministerio de Ciencia e Innovación (grant PID2022-142008NB-I00), partial
support from European Research Council (ERC) grant agreement no. 101020833 and
the European Union NextGenerationEU/PRTR-C17.I1, as well as by the IKUR
Strategy under the collaboration agreement between Ikerbasque Foundation and
DIPC on behalf of the Department of Education of the Basque Government.
D.-G.O., C.F., R.V., and M.G.V. acknowledge funding from the German Research
Foundation (DFG) and the Austrian Science Fund (FOR 5249 - QUAST).

##  Appendix A Materials and Methods

##  Appendix B Structure refinement from powder and single crystal X-ray
diffraction

##  Appendix C Relationship between stoichiometry, structural, and magnetic
susceptibility data

##  Appendix D Density functional theory (DFT) calculations

## References

  * Dzero  _et al._ [2016] M. Dzero, J. Xia, V. Galitski, and P. Coleman, Topological Kondo insulators, Annu. Rev. Condens. Matter Phys. 7, 249 (2016).
  * Li  _et al._ [2020] L. Li, K. Sun, C. Kurdak, and J. W. Allen, Emergent mystery in the Kondo insulator samarium hexaboride, Nat. Rev. Phys. 2, 463 (2020).
  * Paschen and Si [2021] S. Paschen and Q. Si, Quantum phases driven by strong correlations, Nat. Rev. Phys. 3, 9 (2021).
  * Fisk  _et al._ [1988] Z. Fisk, J. D. Thompson, and H. R. Ott, Heavy-electrons: new materials, J. Magn. Magn. Mater. 76-77, 637 (1988).
  * Hundley  _et al._ [1990] M. F. Hundley, P. C. Canfield, J. D. Thompson, Z. Fisk, and J. M. Lawrence, Hybridization gap in Ce3Bi4Pt3, Phys. Rev. B 42, 6842 (1990).
  * Fisk  _et al._ [1992] Z. Fisk, P. C. Canfield, J. D. Thompson, and M. F. Hundley, Ce3Bi4Pt3 and hybridization Gap physics, J. of Alloys and Compounds 181, 369 (1992).
  * Riseborough [2000] P. S. Riseborough, Heavy fermion semiconductors, Adv. Phys. 49, 257 (2000).
  * Dzsaber  _et al._ [2017] S. Dzsaber, L. Prochaska, A. Sidorenko, G. Eguchi, R. Svagera, M. Waas, A. Prokofiev, Q. Si, and S. Paschen, Kondo insulator to semimetal transformation tuned by spin-orbit coupling, Phys. Rev. Lett. 118, 246601 (2017).
  * Lai  _et al._ [2018] H.-H. Lai, S. E. Grefe, S. Paschen, and Q. Si, Weyl-Kondo semimetal in a heavy fermion system, Proc. Natl. Acad. Sci. U.S.A. 115, 93 (2018).
  * Dzsaber  _et al._ [2021] S. Dzsaber, X. Yan, G. Eguchi, A. Prokofiev, T. Shiroka, P. Blaha, O. Rubel, S. E. Grefe, H.-H. Lai, Q. Si, and S. Paschen, Giant spontaneous Hall effect in a nonmagnetic Weyl-Kondo semimetal, Proc. Natl. Acad. Sci. U.S.A. 118, e2013386118 (2021).
  * Dzsaber  _et al._ [2022] S. Dzsaber, D. A. Zocco, A. McCollam, F. Weickert, R. McDonald, M. Taupin, G. Eguchi, X. Yan, A. Prokofiev, L. M. K. Tang, B. Vlaar, L. E. Winter, M. Jaime, Q. Si, and S. Paschen, Control of electronic topology in a strongly correlated electron system, Nat. Commun. 13, 5729 (2022).
  * Grefe  _et al._ [2020] S. E. Grefe, H.-H. Lai, S. Paschen, and Q. Si, Weyl-Kondo semimetal: Towards control of Weyl nodes, JPS Conf. Proc. 30, 011013 (2020).
  * [13] S. E. Grefe, H.-H. Lai, S. Paschen, and Q. Si, Extreme topological tunability of Weyl-Kondo semimetal to Zeeman coupling, arXiv:2012.15841 (2021).
  * Kushwaha  _et al._ [2019] S. K. Kushwaha, M. K. Chan, J. Park, S. M. Thomas, E. D. Bauer, J. D. Thompson, F. Ronning, P. F. S. Rosa, and N. Harrison, Magnetic field-tuned Fermi liquid in a Kondo insulator, Nat. Commun. 10, 5487 (2019).
  * Ajeesh  _et al._ [2022] M. O. Ajeesh, S. M. Thomas, S. K. Kushwaha, E. D. Bauer, F. Ronning, J. D. Thompson, N. Harrison, and P. F. S. Rosa, Ground state of ${\mathrm{Ce}}_{3}{\mathrm{Bi}}_{4}{\mathrm{Pd}}_{3}$ unraveled by hydrostatic pressure, Phys. Rev. B 106, L161105 (2022).
  * Tomczak [2020] J. M. Tomczak, Isoelectronic tuning of heavy fermion systems: Proposal to synthesize ${\mathrm{Ce}}_{3}{\mathrm{Sb}}_{4}{\mathrm{Pd}}_{3}$, Phys. Rev. B 101, 035116 (2020).
  * Xu  _et al._ [2022] C. Xu, C. Cao, and J.-X. Zhu, Pressure-induced concomitant topological and metal-insulator quantum phase transitions in Ce3Pd3Bi4, npj Quantum Mater. 7, 18 (2022).
  * Zhang  _et al._ [2022] S.-K. Zhang, Y.-J. Xu, C.-E. Hu, Y. Cheng, J. Zhu, and G.-F. Ji, Pressure-driven change of ground state of ${\mathrm{Ce}}_{3}{\mathrm{Pd}}_{3}{\mathrm{Bi}}_{4}$: A DFT+DMFT study, Phys. Rev. B 106, 205115 (2022).
  * Dwight [1977] A. E. Dwight, Yttrium–gold–antimony Y3Au3Sb4, Acta Cryst. B 33, 1579 (1977).
  * Dwight [1979] A. E. Dwight, Crystal structure of U3Ni3Sb4 and related compounds, J. Nucl. Mater. 79, 417 (1979).
  * Takabatake  _et al._ [1990] T. Takabatake, S.-i. Miyata, H. Fujii, Y. Aoki, T. Suzuki, T. Fujita, J. Sakurai, and T. Hiraoka, Heavy-Fermion and Semiconducting Properties of the Ternary Uranium Compounds U3T3Sn4 and U3T3Sb4 (T = Ni, Cu, Pd, Pt and Au), J. Phys. Soc. Jpn. 59, 4412 (1990).
  * Wang  _et al._ [1999] M. Wang, R. McDonald, and A. Mar, M3Ni3Sb4 (M = Zr, Hf) and Zr3Pt3Sb4. Ternary Antimonides with the Y3Au3Sb4 Structure, Inorg. Chem. 38, 3435 (1999).
  * Pearson [1972] W. B. Pearson,  _The crystal chemistry and physics of metals and alloys_ , Wiley series on the science and technology of materials (Wiley - Interscience, New York, USA, 1972).
  * Coleman [1983] P. Coleman, $\frac{1}{N}$ expansion for the Kondo lattice, Phys. Rev. B 28, 5255 (1983).
  * Takabatake  _et al._ [1998] T. Takabatake, F. Iga, T. Yoshino, Y. Echizen, K. Katoh, K. Kobayashi, M. Higa, N. Shimizu, Y. Bando, G. Nakamoto, H. Fujii, K. Izawa, T. Suzuki, T. Fujita, M. Sera, M. Hiroi, K. Maezawa, S. Mock, H. v. Löhneysen, A. Brückl, K. Neumaier, and K. Andres, Ce- and Yb-based Kondo semiconductors, J. Magn. Magn. Mater. 177-181, 277 (1998).
  * Kittel [2004] C. Kittel,  _Introduction to Solid State Physics_ , 8th ed. (Wiley, 2004).
  * [27] In Ref. [15], the values of activated energy, which amount to half of the total energy gap $\Delta$ used here, were obtained from Arrhenius fits. For example, the authors reported an activated energy of 28 K at 2.3 GPa, which is equivalent to an energy gap of 56 K $\sim$ 4.8 meV.
  * Birch [1947] F. Birch, Finite elastic strain of cubic crystals, Phys. Rev. 71, 809 (1947).
  * Campbell  _et al._ [2019] D. J. Campbell, Z. E. Brubaker, C. Roncaioli, P. Saraf, Y. Xiao, P. Chow, C. Kenney-Benson, D. Popov, R. J. Zieve, J. R. Jeffries, and J. Paglione, Pressure-driven valence increase and metallization in the Kondo insulator ${\mathrm{Ce}}_{3}{\mathrm{Bi}}_{4}{\mathrm{Pt}}_{3}$, Phys. Rev. B 100, 235133 (2019).


# Pay attention to emoji: Feature Fusion Network with EmoGraph2vec Model for
Sentiment Analysis

Xiaowei Yuan1,2, Jingyuan Hu1, Xiaodan Zhang1(🖂), Honglei Lv1
1Institute of Information Engineering, Chinese Academy of Sciences, Beijing,
100093, China
2School of Cyber Security, University of Chinese Academy of Sciences, Beijing,
100049, China
E-mail: {yuanxiaowei, hujingyuan, zhangxiaodan, lvhonglei<EMAIL_ADDRESS>
###### Abstract

With the explosive growth of social media, opinionated postings with emojis
have increased explosively. Many emojis are used to express emotions,
attitudes, and opinions. Emoji representation learning can be helpful to
improve the performance of emoji-related natural language processing tasks,
especially in text sentiment analysis. However, most studies have only
utilized the fixed descriptions provided by the Unicode Consortium without
consideration of actual usage scenarios. As for the sentiment analysis task,
many researchers ignore the emotional impact of the interaction between text
and emojis. It results that the emotional semantics of emojis cannot be fully
explored. In this work, we propose a method called EmoGraph2vec to learn emoji
representations by constructing a co-occurrence graph network from social data
and enriching the semantic information based on an external knowledge base
EmojiNet to embed emoji nodes. Based on EmoGraph2vec model, we design a novel
neural network to incorporate text and emoji information into sentiment
analysis, which uses a hybrid-attention module combined with TextCNN-based
classifier to improve performance. Experimental results show that the proposed
model can outperform several baselines for sentiment analysis on benchmark
datasets. Additionally, we conduct a series of ablation and comparison
experiments to investigate the effectiveness and interpretability of our
model.

##  I Introduction

With the rapid growth of usage of emoji in all social media posts and
messages, it has become a part of culture in computer-mediated communications
[1]. In fact, the Oxford Dictionary named ‘face with tears of joy’ as the word
of the year in 2015111https://goo.gl/6oRkVg. For most natural language
processing (NLP) tasks, the fundamental step is text representation, by
converting unstructured text into structured information, so that it can be
calculated in many tasks such as text classification and sentiment analysis.
Word embedding is the mainstream method to represent words in finite-
dimensional vector space by word2vec [2] or GloVe [3]. Similarly, emoji
representation learning can be helpful to improve the performance of emoji-
related NLP tasks especially in text sentiment analysis [4]. The ability to
automatically process and interpret text combined with emoji will be vitally
important as the ubiquity of emoji. Representing the emoji meanings like word
embedding models can be used to encode emoji into vectors, which we call emoji
embedding models.

Previous study [1, 5] shows that the usage of emojis has sentiment effects on
plain text, which can even dominate the overall emotional polarity. For
example, the sentiment valence of text “Today is a rainy day.” is originally
neutral. If an emoji ’grin’ (or ’sob’) was added in the end, however, the
sentiment would be totally changed. And we also notice that some emojis are
ambiguous in their meanings depending on different contexts and the co-
occurrence emojis appearing in a sentence have the likeness of their
sentiment, interpretation or intended use. It is of great importance to
understand the emotional semantics of emoji and analyze the overall sentiment
of the sentence. Consider two tweets, 1) The white curtains blew in the wind
like ’ghost’, it’s a little scary ’scream’. 2) I love ’ghost’ movie ’heart-
eyes’ ’heart-eyes’. The intended emotional meaning of ’ghost’, with different
emojis followed, varies from negative to positive. This also reflects that the
emotional information in emojis is not immutable. Therefore, we propose a
model called EmoGraph2vec to obtain better performance on emoji representation
learning. We utilize the co-occurrence of emoji in “big data” as an underlying
feature to construct non-euclidean structured data to learn emoji embeddings
in a graph with a variational graph autoencoder (VGAE) [15]. The semantics of
emojis are provided by a sense inventory called EmojiNet [16] to add node
attributes and edge weights.

With the explosive growth of social media, opinionated postings with emojis
have increased explosively. Social media posts contain a vast amount of
valuable emotional information and have become a hot research target for
sentiment analysis [8]. Text with emojis could resolve ambiguity in written
communication and allow people to vividly convey their feelings in a concise
way. Therefore, fusing text with attached emojis can be more conducive to
sentiment analysis task, which also could be viewed as a bi-modal (text +
emoji) information processing. The most common method to analyze the text
containing emojis is to utilize emojis as an important feature [6, 9].
Nevertheless, they are not fine-grained enough in feature fusion, which means
they failed to explore fully the emotional impact of emojis on the text. In
this paper, we propose a hybrid-attention network to learn the mutual
emotional features between text and emojis. Then we send fused features into a
TextCNN[14]-based classifier to predict the sentiment labels.

In summary, the main contributions of our work are as follows:

  * •

We propose an emoji representation learning method named EmoGraph2vec to learn
emoji embeddings in a graph network. We utilize a large-scale unlabeled corpus
to form an emoji co-occurrence graph network and adopt a VGAE to make use of
network structure and node attribute information to obtain emoji embeddings
based on EmojiNet.

  * •

We propose a novel neural network framework to incorporate text and emoji
information into sentiment analysis, which uses a hybrid-attention network to
fuse the bi-modal features and combines with a TextCNN-based classifier to
predict sentiment labels.

  * •

We confirm the effectiveness of our model for sentiment analysis on several
emoji-rich datasets. We also visualize the Emograph2vec compared with other
method to confirm the interpretability of our emoji representation method.

##  II Relate Work

###  II-A Sentiment Analysis

As a significant branch of natural language processing (NLP), text sentiment
analysis aims to mine and analyze the emotions, opinions, and attitudes from
texts. The rapid development of deep learning has played an important role in
boosting the development of sentiment analysis researches. Socher et al. [13]
applied Recursive Neural Network to text sentiment classification with the
consideration of the syntactic structure information; Santos et al. [11]
proposed Character to Sentence Convolutional Neural Network to analyze
sentiment on the short text.

###  II-B Emoji in Sentiment Analysis

Many studies use emojis as heuristic information in social texts [12, 17, 7],
where emojis serve for unsupervised learning in a large number of unlabeled
data. Emojis can also be regarded as important semantic features about
emotions towards the recipient or subject [18] for sentiment classification.
For instance, Tian et al. [19] used a Bi-directional Gate Recurrent Unit
Attention network, which integrated the emotional polarity of emoji and
embedded it as a feature into the model. But they failed to reflect the
emotional impact of emojis on the text. Lou et al. [9] constructed an emoji-
based bidirectional long short-term memory (Bi-LSTM) model, which combined the
attention mechanism to weigh the contribution of each word on the emotional
polarity based on emoji. But their model only analyzes the microblog data that
contains a single emoji and cannot be generalized to process multiple types of
emojis. Yuan et al. [10] proposed an emoji-based co-attention network to deal
with the text and attached multiple emojis. But all the works only learned the
emoji features through fixed emoji names without consideration of actual usage
scenario.

###  II-C Emoji Representation Learning

Barbieri et al. [20] trained an emoji embedding model on tweets to seek an
understanding of emoji meanings from how emoji are used in a large collection
of tweets. Eisner et al. [21] used a pre-trained word embedding model, and
applied it to emoji names to learn an emoji embedding model which they called
emoji2vec. Wijeratne et al. [4] learned the distributional semantics of the
words in emoji definitions to model the emoji meanings extracted from
EmojiNet. But they failed to explore the emoji meanings in the actual usage
scenarios. Illendula et al. [22] utilized a large Twitter corpus which has
emojis in them and built an emoji co-occurrence network, and trained a network
embedding model to embed emojis into a low dimensional vector space. But they
only utilized the emoji frequency and ignored their textual meanings as node
information which can enhance the emoji semantic features.

##  III Model

For sentiment analysis, our model consists of several major procedures:
feature extraction, feature fusion and sentiment classifier (Fig. 1).

We will illustrate the details in the following section.

Figure 1: The architecture of the whole model

###  III-A Emoji Feature Extraction

Given posts with text and emojis, we first use our emoji embeddings model
EmoGraph2vec to encode emoji into vectors, so as word2vec does. Then we equip
the embeddings with additional information about the emoji position in the
sentence since our model treats emoji as a separate input, which derives from
the original sentence, for the model to make use of the order of the words and
emojis.

####  III-A1 EmoGraph2vec

Based on large-scale Twitter data (IV-A2), we extract the emoji co-occurrences
from it to learn emoji features with the EmojiNet resource. Different from
traditional methods that treat the emoji co-occurrences as sequence data, we
utilize the co-occurrence information from the whole corpus as non-Euclidean
data to construct an undirected network graph.

##### EmojiNet

EmojiNet is a machine readable sense inventory for emojis created by Wijeratne
et al [16]. It consists of 12,904 sense labels over 2,389 emojis, where each
emoji $e$ is represented as a nonuple $(u,n,c,d,K,I,R,H,S)$. The meaning of
each element is shown in the Figure 2.

Figure 2: The architecture of EmoGraph2vec model

We utilize the EmojiNet to transform the Unicode emoji representation into
textual meanings. Specifically, $K$ and $S$ elements of each emoji nonuple are
used to calculate edge weight and the node attribution in the graph network.

####  III-A2 Graph Initialization

Each tweet forms a polygon of $n$ nodes where $n$ represents the number of
unique emojis. Let us denote the graph as $G=(V,E)$, where $v\in V$ denotes
one kind of emoji, and $N=|V|$ equals the number of all emojis. The nodes are
connected when they appear in the same tweet.

The edges of the network are weighted by the co-occurrence frequency and
similarity of two emoji nodes. We extract senses $S$ of each emoji $e_{i}$
from EmojiNet as node text to calculate the TF-IDF (Term Frequency–Inverse
Document Frequency) vectors $v_{i}$. Then, similarity scores between node
pairs $(e_{i},e_{j})$ are computed by cosine similarity. We hypothesize that
the emoji pair that has high co-occurrence frequency may have a strong
emotional connection and the similarity can quantify the emotional semantic
distance. Based on co-occurrence frequency and similarity, the product of
these two indicates edge weight in weighted adjacent matrix $A$, which can
determine the mutual influence of emotional polarity between emoji pairs.

$A[i,j]=cosine(v_{i},v_{j})*frequency(e_{i},e_{j})$ (1)

To enrich the semantic information in the network, we also utilize the $K$
element of emoji $e$ into this graph as node attributes. We embed the keywords
of each emoji by word2vec. The word vectors $k_{i}$ of all words in the emoji
senses are averaged to form the final attribute $x_{e}$.

##### Graph Embedding

To incorporate the network structure and node attribute information, we use an
unsupervised learning method VGAE to learn node embeddings. This model uses a
two-layer graph convolutional network (GCN) encoder and an inner product
decoder. The encoder produces the distribution of vectors, including mean
$\mu$ and variance $\sigma$, from which stochastic latent variable $z_{i}$ is
obtained by sampling.

$\displaystyle q(z_{i}|X,A)=\mathcal{N}(z_{i}|\mu_{i},diag({\sigma}^{2}_{i}))$
(2)

Here, $A$ is a weighted adjacency matrix of $\mathcal{G}$, $X$ represents the
node attribute. And the decoder is given by an inner product between latent
variables:

$\displaystyle
p(A|Z)=\begin{matrix}\prod_{i=1}^{N}\begin{matrix}\prod_{j=1}^{N}p(A_{ij}|z_{i},z_{j})\end{matrix}\end{matrix}$
(3)

where $Z$ is the emoji embedding matrix we needed.

####  III-A3 Positional Encoding

Positional encoding are useful for the emojis because they give the emoji
features a sense of where the emoji would have been. We combine position
encoding with the emoji embeddings to form the input emoji features
$R_{e}=(e_{1}+pe_{1},e_{2}+pe_{2},...,e_{n}+pe_{n})$. The dimension of
positional encoding is the same as the emoji embeddings so as to sum the both.

In our model, we utilize sine and cosine functions for different
positions[23]:

$\displaystyle PE_{(pos,2i)}=sin(\frac{pos}{1000^{2i/d_{model}}})$ (4)
$\displaystyle PE_{(pos,2i+1)}=cos(\frac{pos}{1000^{2i/d_{model}}})$ (5)

Where $pos$ is the absolute position of emoji in original sequence, $i$ is the
dimension. Given the mathematical properties of sinusoidal and cosine
functions, these two functions can also learn to attend by relative positions,
since $PE_{pos+t}$ can be represented as a linear transformation of
$PE_{pos}$.

###  III-B Text Feature Extractor

We obtain the word embeddings by the pre-trained word vectors over the Google
News dataset222https://goo.gl/QaxjVC. Then the plain text can be represented
as $X=[x_{1},x_{2},...,x_{L}]$. Since LSTM can overcome the problem of
gradient vanishing and explosion with the capability to learn long-range
dependencies in sequences, our text feature extractor adopts two stacked Bi-
LSTM layers to learn the text representation bidirectionally. And we
concatenate the hidden vectors from both directions to represent every single
word as the output $h_{l}$ of the layer. The second bi-directional LSTM layer
takes the output of the previous one as its input
$H_{1}=[h_{11},h_{21},..,h_{L1}]$, and computes unit stats of network in the
same pattern before producing the output $H_{2}=[h_{12},h_{22},..,h_{L2}]$ as
$R_{h}$. Through a skip-connection, the outputs of the below three layers (the
embedding layer and the two bi-directional LSTM layers) are concatenated as
$R_{u}=[X,H_{1},H_{2}]$.

###  III-C Feature Fusion

For a post containing text and emoji, not all words contribute equally to
express sentiments, we need to lead the model to attend to sentiment-guided
words. Furthermore, different emojis are also related to different emotional
semantics, which depends on the context. Also text has a strong emotional
impact on the sentiment conveyed by the emoji. To this end, we develop a novel
hybrid-attention module to model the mutual influence between text and emoji
based on self-attention unit and its variant co-attention unit.

Figure 3: self-attention unit

Figure 4: co-attention unit

Self-attention unit is a crucial part in transformer [24] as showed in Figure
4, which consists of self-attention function and a fully connected feed-
forward network with a residual connection and normalization. The input
vectors are used to compute three matrices $Q$, $K$, and $V$ (queries, keys,
and values). And the attention distributions over $V$ are determined by the
dot-product similarity between $Q$ and $K$:

$Attention(Q,K,V)=softmax(\frac{QK^{T}}{\sqrt{d}})V$ (6)

where $d$ is the dimension of of the output feature. The resulting weight-
averaged value vector forms the output of the attention block. The fully
connected feed-forward network consists of two linear transformations with a
ReLU activation function in between.

$FFN(x)=max(0,xW_{1}+b_{1})W_{2}+b_{2}$ (7)

The co-attention unit is a variant from self-attention [25].The only
distinction is that the queries are from emoji (or text) features while keys
and values are from text (or emoji) features. This unit produces an attention-
pooled feature for one modality conditioned on another modality. If $Q$ is
computed by text and $K$ and $V$ computed by emoji, the unit measures text-
guided attention to decide which emoji should weigh more. If $Q$ comes from
emoji and $K$ and $V$ come from text, this unit measures emoji-guided
attention to weigh each word. The whole process we design is to simulate the
mutual influence between text and emoji.

###  III-D Sentiment Classifier

After the hybrid attention module, we obtain the bi-modal feature matrixes
$W_{u}\in\mathbb{R}^{d*L}$, $W_{e}\in\mathbb{R}^{d*N}$ and
$W_{h}\in\mathbb{R}^{d*L}$ ($d$ is the embedding dimension, $L,N$ is the
length of text and emojis) fused features of text and emoji. The three
matrixes are treated as three single-channel input into a TextCNN classifier
to predict the probability distribution of sentiment labels. We use three
sizes $(k_{1},k_{2},k_{3})$ of kernel sets $S_{i}=[s_{i1},s_{i2},…,s_{in}]$,
$s_{i}\in\mathbb{R}^{k_{i}*d}(i=1,2,3)$, that respectively map the input
$W_{u}$, $W_{e}$ and $W_{h}$ to $n$ new feature maps
$U_{i}=S_{i}*W_{i},U_{i}\in\mathbb{R}^{k^{\prime}*1}$. Here, $*$ denotes
convolution. When $i$ takes different values, $W_{i}$ represents $W_{u},W_{e}$
or $W_{h}$. We then apply a max-over-time pooling operation to take the
maximum value for each map, and form the new representation
$V_{u},V_{e},V_{h}\in\mathbb{R}^{n}$. The three outputs are concatenated to be
passed to a fully connected softmax layer whose output is the probability
distribution over sentiment labels.

##  IV Experiment

###  IV-A Dataset

####  IV-A1 Labeled Dataset

The labeled data are collected from multi-source social media platforms and
cover multiple domains to minimize biases. More details about the datasets
(MSD, TD, and ERD) are shown in Appendix A 333In supplementary material. We
use the part of the data that contains the emojis, of which the amount is
limited.

####  IV-A2 Unlabeled Dataset

To train the EmoGraph2vec model, we use a large-scale unlabeled data of Tweets
named EmojifyData444https://www.kaggle.com/rexhaif/emojifydata-en. This
dataset contains 18 million English tweets, all with at least one emoji
included. Based on it, we can learn emoji representations in EmoGraph2vec to
obtain emotional semantic information in their embeddings.

###  IV-B Implementation Details

Our model is trained using the PyTorch library [26] on a cuda GPU. Appendix B
provides a detailed description of our experimental configuration. In VGAE,
the number of units in hidden layer 1 is set to 256, and layer 2 is 300. The
training procedure runs for 50 epochs to learn emoji representations on
EmojifyData. We train our model in the sentiment analysis task with 20 epochs.

###  IV-C Baselines and Performance Comparison

To evaluate the performance of our model, we employ several representative
baseline methods: TextCNN [14], Att-BiLSTM [27], EA-Bi-LSTM [9] and ECN[10].
EA-Bi-LSTM (based on Emoji-Attention and BiLSTM) and ECN (Emoji-based Co-
attention Network) are the latest work that designed for the text containing
emojis. For each method, we use three methods to embed emojis. The line
without symbol uses the EmoGraph2vec method, others use emoji2vec [21] and the
Unicode characters given by the Unicode Consortium. The emoji2vec embeddings
are publicly released555https://github.com/uclmr/emoji2vec.

TABLE I: The accuracy of our model and baseline methods on different dataset

Models | MSD | TD | ERD
---|---|---|---
TextCNN | 0.8329 | 0.7273 | 0.6856
TextCNN† | 0.8258 | 0.7202 | 0.6719
TextCNN* | 0.8524 | 0.7378 | 0.7046
Att-BiLSTM | 0.8217 | 0.7098 | 0.6666
Att-BiLSTM† | 0.8243 | 0.7198 | 0.6700
Att-BLSTM* | 0.8301 | 0.7203 | 0.6792
EA-Bi-LSTM | 0.8607 | 0.7483 | 0.7278
EA-Bi-LSTM† | 0.8527 | 0.7470 | 0.7025
EA-Bi-LSTM* | 0.8301 | 0.7413 | 0.6835
ECN | 0.8672 | 0.7579 | 0.7335
ECN† | 0.8552 | 0.7464 | 0.7256
ECN* | 0.8213 | 0.7295 | 0.6929
Our model | 0.8815 | 0.7703 | 0.7627
Our model† | 0.8567 | 0.7595 | 0.7464
Our model* | 0.8335 | 0.7442 | 0.7401

  * †

use the emoji2vec method to obtain emoji representations

  * *

use the Unicode characters as emoji representations

In Table I, we can see our model outperforms all other baseline methods with
the EmoGraph2vec model. The results prove that our proposed method is more
effective than the traditional deep learning methods (TextCNN, Att-BiLSTM),
which do not pay attention to emojis. Looking more closely, EA-Bi-LSTM and ECN
achieve a certain improvement than the former methods (regardless of
EmoGraph2vec or emoji2vec). It demonstrates the significance of incorporating
emojis and text to analyze sentiment. And it is remarkable to find that our
model obtains higher accuracy than other baselines. What’s more, EmoGraph2vec
can achieve higher accuracy compared with emoji2vec and unicode in the latter
three models that focus on emojis. Especially, when our model is applied on
the ERD (Emotion Recognition Dataset), there is a noticeable drop in the
overall accuracy. Different from sentiment classification, emotion is more
fine-grained and multi-classified, which puts forward higher requirements for
the accuracy of the model; secondly, the amount of training set in the ERD
dataset is also less than the first two sentiment analysis datasets, which may
also have some impact on accuracy.

When using the Unicode characters as emoji representations, one interesting
finding is that the accuracy of our model, EA-Bi-LSTM and ECN have decline
while TextCNN and Att-BiLSTM have increase. A possible explanation for this
might be that the former separate text and emojis as two parts of inputs,
where pre-trained emoji embeddings can work better. When treating the whole
sentence as an integrated input (in TextCNN and Att-BiLSTM), the Unicode
characters can be trained with word vectors together to achieve better
performance.

TABLE II: The accuracy of modified models on different datasets Models | MSD | TD | ERD
---|---|---|---
N-model | 0.7814 | 0.6783 | 0.6709
T-model | 0.8356 | 0.7215 | 0.7148
E-model | 0.8626 | 0.7578 | 0.7484
RA1 | 0.8512 | 0.7448 | 0.7436
RA2 | 0.8157 | 0.7103 | 0.7178
RA3 | 0.8652 | 0.7675 | 0.7568
Our model | 0.8815 | 0.7703 | 0.7627

(a) Emoji2vec

(b) EmoGraph2vec

Figure 5: Comparison of emoji representations between emoji2vec and
emograph2vec. The horizontal and vertical coordinates are emoji names, where
’F’ indicates the facial emoji.

###  IV-D Model Analysis

####  IV-D1 The Power of Emojis

To further explore the influence of emojis in our model, we conduct the
subsequent experiments by removing the inputs of emojis or simplified
architecture of the model to evaluate the effectiveness of emojis. N-model
detaches the emoji inputs. T-model removes the text-guided co-attention module
of emoji representation learning. E-model removes the emoji-guided co-
attention module of text representation learning.

As shown in Table II, we find that the complete model significantly
outperforms the N-model and T-model on all the datasets, both of which only
consider the text features before classification. That demonstrates the plain
text does not contain rich emotional semantic information as emojis do
occasionally in sentiment analysis. The T-model also outperforms the N-model
to a certain degree. This shows emoji-guided text representation learning can
effectively improve the ability of the model to learn the emotional semantic.
It also explains why our emoji-aware method can achieve better accuracy
compared to other baseline methods. The accuracy of the E-model is also higher
than T-model and slightly lower than the complete model. Because E-model
extracts sentiment information from text-guided emoji representation but fails
to capture sentiment patterns of emoji-guided text representation.

####  IV-D2 Effectiveness of Co-Attention

To further explore the effect of the co-attention mechanism in our proposed
method, we compare the complete model with several attention-modified models
as follows: In RA1, we remove the self-attention module and concatenate the
text features of each word $R_{u}$ as text feature matrix $W_{u}$. In RA2, we
replace the text-guided co-attention module with the concatenation of the
emoji vectors $E=[e_{1},e_{2},...,e_{N}]$ as the emoji representation $W_{e}$.
In RA3, we replace the emoji-guided co-attention module with the concatenation
of the text representation $H_{2}=[h_{12},h_{22},..,h_{L2}]$ as the text
representation $W_{h}$.

As shown in Table II, it can be seen that the accuracy of the RA2 has dropped
sharply in most datasets. RA2 directly changes the attention mechanism of the
emoji feature matrix $W_{e}$ to the concatenation of all emoji vectors,
indicating that emojis take most of the weight for the emotional semantic
analysis of the model. That means when the model cannot distinguish which
emoji dominates the text emotion, the accuracy drops significantly. The slight
difference of accuracy between the RA1 and RA3 model reveals that when the
self-attention of the text is removed, the simplified $W_{u}$ matrix will
further affect the representation of $W_{e}$ as the emoji feature, and it
results in the lower accuracy. While the RA3 retains the first two
representations of $W_{u}$ and $W_{e}$ matrices, only the last step of the
text vector representation is replaced, it has the least impact on the model
performance, indicating that the model can still make correct predictions from
the $W_{e}$ matrix with a greater probability.

####  IV-D3 Comparison about Emoji Representation Learning

As shown in the Table I, when using different emoji embedding methods in
sentiment analysis tasks, our model combined with Emograph2vec outperforms
other methods in emoji2vec. To further explore the difference between the two
methods, we perform a hierarchical clustering [28] on
Top64666https://home.unicode.org/emoji/emoji-frequency/ most used emojis and
visualize the clustering results in Fig 5. Each emoji can be meaningfully
represented by a low-dimensional vector in the embedding space where
similarity can be measured. In the report, the Top64 emojis include facial
expressions, gestures, and objects. We suppose that not only facial expression
emojis can carry emotional semantic information but also other categories in
various scenarios. Since we learn emoji features from social network data, we
can learn the sentiment tendency contained in each emoji in a specific scene.
We expect that a well-performed representation can embed emojis with similar
emotional semantics closely in the vector space no matter it is a facial
expression or object.

The color scale of each cell indicates the similarity between the two emojis.
The darker the cell is, the more similar the representations of the two emojis
are. As shown in 5(a), the emoji2vec method directly works on Unicode
descriptions, which reflects on the clustering result. The facial expression
emojis have a high degree of similarity also the similar-shaped (e.g. heart-
shaped) emojis do. And the distance between the groups is large, which
illustrates that the method strictly classifies emojis literally. In the
contrast, we can see in the 5(b), there is no clear boundary to distinguish
each kind of emojis. The bottom right corner contains the darkest cells that
means these emojis have similar emotional semantics. We can see this area
includes expressions as well as gestures and objects (e.g. ”sparkle” and
”Face_joy”).

##  V Conclusions

Emojis have an extremely high occurrence in computer-mediated communications.
We propose a method to learn emoji representations called EmoGraph2vec, which
can also be utilized in other NLP tasks. The feture fusion hybrid-attention
network we designed learns the mutual emotional semantics between text and
emojis on texts of social media.

## Acknowledgment

The research is supported in part by the Strategic Priority Research Program
of the Chinese Academy of Sciences, Grant No. XDC02060400.

## References

  * [1] Hu, Tianran et al., Spice Up Your Chat: The Intentions and Sentiment Effects of Using Emojis, ICWSM, 2017.
  * [2] T. Mikolov, et al.,Distributed Representations of Words and Phrases and their Compositionality, NIPS, 2016.
  * [3] P. Jeffrey, et al.,Glove: Global vectors for word representation, EMNLP, 2014.
  * [4] S. Wijeratne, et al., A Semantics-Based Measure of Emoji Similarity, WI, 2017.
  * [5] H. Cramer, et al., Sender-intended functions of emojis in US messaging, In Proc . MobileHCI, 2016.
  * [6] Zhao J, Dong L, Wu J, Xu K. MoodLens, An emoticon-based sentiment analysis system for Chinese tweets, In Proc. the 18th KDD, Aug. 2012, pp.1528-1531.
  * [7] M. Li, E. Ch’ng, A. Chong, and S. See, Multi-class twitter sentiment classification with emojis, Industrial Management & Data Systems, 2018.
  * [8] E. Cambria, Affective Computing and Sentiment Analysis, Intelligent Systems, 2016.
  * [9] Y. Lou, et al., Emoji-based sentiment analysis using attention networks, In Proc. TALLIP, 2020.
  * [10] Yuan X., Hu J.,et al.Emoji-Based Co-Attention Network for Microblog Sentiment Analysis, Neural Information Processing. ICONIP 2021.
  * [11] C. Santos,et al., Deep Convolutional Neural Networks for Sentiment Analysis of Short Texts, COLING, 2014.
  * [12] D. Davidov, et al., Enhanced sentiment learning using twitter hashtags and smileys, COLING, 2010.
  * [13] R. Socher, et al., Recursive deep models for semantic compositionality over a sentiment treebank, EMNLP pp. 1631-1642, 2013.
  * [14] Y. Kim, Convolutional neural networks for sentence classification, ACL, 2014.
  * [15] T. Kipf, M. Welling, Variational graph auto-encoders. NeurIPS, 2016.
  * [16] S. Wijeratne, et al., EmojiNet: An Open Service and API for Emoji Sense Discovery, ICWSM, 2017.
  * [17] A. Go, R. Bhayani, and L. Huang, Twitter sentiment classification using distant supervision, CS224N Project Report, 2009.
  * [18] J.B. Walther, et al., The impacts of emoticons on message interpretation in computer-mediated communication, SSCR, 2001.
  * [19] Z. Tian, et al., Attention aware bidirectional gated recurrent unit based framework for sentiment analysis, In Proc. KSEM, Springer (2018),pp. 67-78.
  * [20] F. Barbieri, F. Ronzano, and H. Saggion, What does this Emoji Mean? A Vector Space Skip-Gram Model for Twitter Emojis, In LREC, 2016.
  * [21] B. Eisner, et al., emoji2vec: Learning Emoji Representations from their Description, In Proc. The 4th International Workshop on NLP for Social Media, 2016.
  * [22] A. Illendula, et al., Learning Emoji Embeddings using Emoji Co-occurrence Network Graph, ICWSM, 2018.
  * [23] G. Jonas, et al.,Convolutional sequence to sequence learning, ICML, 2017.
  * [24] V. Ashish, et al.,Attention is all you need, In Proc of the 31st NIPS, 2017.
  * [25] L. Jiasen, et al.,ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks, NeurIPS, 2019.
  * [26] A. Paszke, et al., PyTorch: An Imperative Style, High-Performance Deep Learning Library, 2019.
  * [27] P. Zhou, et al., Attention-based bidirectional long short-term memory networks for relation classification, ACL, 2016.
  * [28] Z. Joseph, et al., Fast optimal leaf ordering for hierarchical clustering, ISMB, 2001.