# An Enquiry on similarities between Renormalization Group and Auto-Encoders
using Transfer Learning.

Mohak Shukla<EMAIL_ADDRESS>Ajay D. Thakur<EMAIL_ADDRESS>Department of Physics,
Indian Institute of Technology Patna,
Bihta 801106, India

###### Abstract

Physicists have had a keen interest in the areas of Artificial Intelligence
(AI) and Machine Learning (ML) for some time now, with a special inclination
towards unravelling the mechanism at the core of the process of learning. In
particular, exploring the underlying mathematical structure of a neural net
(NN) is expected to not only help us in understanding the epistemological
meaning of ‘Learning’ but also has the potential to unravel the secrets behind
the workings of the brain. Here, it is worthwhile to establish correspondences
and draw parallels between methods developed in core areas of Physics and the
techniques developed at the forefront of AI and ML. Although recent
explorations indicating a mapping between the Renormalisation Group(RG) and
Deep Learning(DL) have shown valuable insights, we intend to investigate the
relationship between RG and Autoencoders(AE) in particular. We will use
Transfer Learning(TL) to embed the procedure of coarse-graining in a NN and
compare it with the underlying mechanism of encoding-decoding through a series
of tests.

Keywords

Renormalization Group; Deep Learning; Auto-Encoders; Transfer Learning; Ising
model; Unsupervised Learning.

††preprint: APS/123

##  I Introduction

Machine Learning or as some of us would call it, Statistical-Optimization is
not a new field per se. Conceptualization of Artificial Intelligence(AI) (of
which Machine Learning(ML) is a part) began as soon as computers became
capable of executing a complex set of instructions. Hence the words Artificial
Intelligence and Machine Learning were coined in the 1950s and 1960s,
respectively [1, 2].The field, however, remained mostly in the conceptual
realm [3, 4, 5, 6] due to a lack of computational power, which was needed to
scale the algorithms to solve any worthwhile problem. In the last couple of
decades, however, the advancement in silicon fabrication and multi-core
processors has reinvigorated the field, making it possible to crunch large
sets of data on training machine learning models. Today ML is used to solve
various kinds of problem, from object detection to high frequency stock-
trading.

By the end of the 20th century, physicists have found themselves surrounded by
problems that are inconceivably hard to solve. Equations governing millions of
stars in a galaxy are simply not solvable by any known analytical method; the
same goes true for solving the wave-function of exotic particles in a high-
energy physics problem and computational physics appears to be the only way
forward. This is where ML can be of immense assistance [7, 8, 9, 10, 11]. On
the other hand, physicists have also been contributing to ML, drawing
inspiration from the theories and techniques developed for computational
physics, as well as providing insights into the foundation of AI and ML [12,
13, 14, 15]. One such insight manifests as we study deep learning side by side
with the renormalization group (Fig.1).

An artificial neuron was designed to mimic the workings of a biological neuron
using mathematical functions [16]. Rosenblatt used them to make a network that
can learn to act as a binary classifier, called it the Perceptron [17]. After
a while these neural-nets were stacked on top of each other to make a multi-
class discriminator, such NNs were called multilayer perceptron(MLP) or deep
neural network(DNN). DNN are known for their ability to extract
features(relations) from the dataset it is trained on. These features become
more complex as we go deeper inside the layers of a neural net. For example, a
DNN trained on Human faces may recognize the outlines of a face in the first
layer. Going a layer deeper, it might start recognizing facial features like
eyes, nose, lips. A layer deeper, it might start understanding the relative
positioning of eyes/nose/lips in order to recognize and differentiate faces.
This reminds us of the coarse-graining techniques employed in RG [18, 19].
Coarse-graining is essentially summing/integrating over short-range
interactions. Resulting in long-range relations and fewer degrees of freedom.
RG thus gives us a systematic procedure to determine the macroscopic relations
governing the system from microscopic interactions, iteratively. Indicating an
analogy with DL. Some pioneering work has been done in this area [20, 21]
particularly the work on finding an exact mapping between the Variational
Renormalization Group and Deep Learning by Mehta and Schwab [22]. We, however,
see room for further exploration which will refine our understanding of the
subject.

Thus we propose a reductionist approach to investigate the parallels between
RG and DL. For that, we intend to dissect both the processes down to the
individual steps, fine-tuning them to fit a common NN architecture. This will
enable us to study the structures formed in the net as it learns to compress
the Ising lattices and compare it with mapping formed by coarse-graining
through RG. For this, we are going to prepare 3 models of NNs, each having the
same number of layers, neurons, and activation function(s). The difference
lies in their training routine, where one will use Backpropagation to learn
encoding-decoding(i.e. an Autoencoder) while the other has a hard-coded map of
decimation to perform coarse-graining and so on(see Sec.II and table 1). We
will also benchmark their performance on metrics such as information loss,
change in entropy, the flow of inverse temperature($\beta$) and fixed point(s)
which will further help in demonstrating parallels (and differences) between
RG and DL in Sec. III.

Figure 1:  Showing an example of (a) Coarse-graining used in RG for
1-dimensional Ising model using 3-spinblock majority rule. (b) A basic NN
consisting of input, hidden and output layers.

##  II Method of Enquiry

We will be using Python version 3.6 along with libraries such as Keras [23]
which runs over the framework of Tensorflow [24] to build and train NNs. We
utilise NumPy [25] for matrix operation, Scikit-image [26] image processing
and Matplotlib [27] for plotting graphs. The project environment was
configured to use the CUDA [28] framework in order to use Nvidia GPUs for
faster training.

###  II.1 Renormalization through Decimation

Figure 2:  The decimation scheme: (a) The original lattice with alternative
Blue and Red spins. Red spins are then removed. (b) The coarse-grained lattice
with the emergent nearest neighbors, shown as dashed lines in (a)

In a 2-dimensional Ising lattice, each site ‘i’ has a spin denoted by variable
$(\sigma_{i})$, which has values ($+1$) or ($-1$) [29]. The Hamiltonian
‘$H(\sigma)$’ of such system is defined as:

$H(\sigma)=-J\sum_{<i,j>}\sigma_{i}\sigma_{j}$ (1)

For RG, we would be employing coarse-graining through Decimation. As
illustrated in Fig.2, we try to remove every other lattice site by summing
over the effects of decimated Red-spins(R) to form a new lattice with only
Blue-spins(B). The configuration probability in terms of inverse
temperature($\beta$) is:

$P(\sigma)=\frac{e^{-\beta H(\sigma)}}{Z(\beta)}\text{, where partition
function }Z(\beta)=\sum_{\sigma}e^{-\beta H(\sigma)}$ (2)

Expanding the term for a single decimation site (A):

$P(\sigma)=\frac{e^{\beta(B_{1}R_{A}+B_{2}R_{A}+B_{3}R_{A}+B_{4}R_{A})}\times
B}{Z(\beta)}$ (3)

Where $B=exp(\beta\sum_{i,j\neq A}J_{i,j}B_{i}R_{j})$ and the minus sign is
absorbed into ’$J$’. Averaging out the effect of ($R_{A}$) for both (+1 and -1
spin) turns out to be:

$P(\sigma)=B\times\frac{e^{\beta(B_{1}+B_{2}+B_{3}+B_{4})}+e^{-\beta(B_{1}+B_{2}+B_{3}+B_{4})}}{Z(\beta)}$
(4)

Expanding the term averages out the effect of spin at site(A) into pair-wise
and quartet interactions of nearest and next-nearest spins in the new lattice.
Taking approximation with only pair-wise coupling, without changing the
Hamiltonian, while negating the quartet couplings. The new configuration
probability can be approximated as:

$P^{\prime}(\sigma)=\frac{e^{\beta^{\prime}H(\sigma)}}{Z(\beta^{\prime})}\quad\text{,
where {\hbox{\beta' = \frac{3}{8}\ln(\cosh(4\beta))}}}$ (5)

###  II.2 Autoencoder and the common Architecture

Figure 3:  Showing the workings of Autoencoder used to compress the Ising
lattice. This architecture will remain common for the NNs used in all three
models, each consisting one input layer, three hidden layers and one output
layer (further explained in Sec.II.6 and Table 1) .

Autoencoder is a simple NN that learns to reconstruct its input by first
compressing(encoding) it down to a reduced representation and then
decompressing(decoding) it to get something as close to the original input as
possible [30][31]. This encoding is achieved by finding and reinforcing the
hierarchy of relations present in the input dataset until only the ‘relevant’
features remain(Fig. 3). This is done by tuning the weights which connect the
layers to reduce losses through back-propagation [32]. Since the training
dataset need not be labelled, the process is called unsupervised learning(UL).
The number of neurons in the input layer($N_{I}$) will equal to the number of
spins in the square spin-lattice of size ($L\times L$), for $L=32$,
$L^{2}=1024=N_{I}$. In order to remain homologous to the decimation coarse
graining, next two encoding hidden layer will be of size $N_{H1}=512$ and
$N_{H2}=256$ respectively. Similarly, decoding layer $N_{H3}=512$ and finally
the output layers $N_{O}=1024$. _This configuration of layers will remain
common for all three models._

###  II.3 Transfer Learning

Figure 4:  Shallow Networks performing (a)Encoding the $32\times 32$ Ising
lattice down to $32\times 16$ and (b)Decoding the $32\times 16$ Ising lattice
up to $32\times 32$.

Transfer-learning(TL) is a technique which uses learning from one dataset and
applying it to related but different problem after some modifications. It is
essentially a transfer of weights of trained layers from one NN to another,
thus skipping most of training required in the second NN. For example, a NN
trained to recognize cats, has also learned to recognize 4-legged animals in
its hidden layers, this learning can then be transferred with slight
modification to a new NN, which can recognize dogs. In our case, several
shallow-nets with just the input and the output layer will be trained to
perform a single iteration of RG.

In the Fig.4 two such shallow-nets are shown where former is trained to
perform coarse graining (Let us call them CG) and the latter is trained to
perform upscaling (UP). The input as well the output of these shallow-nets are
linearized Ising-lattices. It is important to note that a single iteration of
decimation reduces the number of sites in the lattice by half, and since we
cannot reduce both dimensions of the lattice by the factor of $\sqrt{2}$, the
reconstructed lattices will appear compressed in 1 dimension while the other
remains unchanged. For example in Fig. 4(a), a $32\times 32$ Lattice is
linearized to $1024$ sized array, after one decimation it will be reduced to
an array of size $512$, this can only be reconstructed into a lattice of
$32\times 16$ or $16\times 32$, hence the deformed lattice. No such problem
exists in even number of iterations, where both dimensions of the lattice get
reduced by the factor of 2, see the input lattice of $16\times 16$ in Fig.4(b)
for example. We are going to need four such shallow-nets:

  1. 1.

Coarse-grainer from 32x32 to 32x16 (CG-1024-512)

  2. 2.

Coarse-grainer from 32x32 to 32x16 (CG-512-256)

  3. 3.

Up-scaler from 16x16 to 32x16 (UP-256-512)

  4. 4.

Up-scaler from 32x16 to 32x32 (UP-512-1024)

Figure 5:  Showing the process of inserting shallow-nets into the frame of an
untrained NN through Transfer Learning.

After training, their weights will be transferred to a multilayered NN
(Fig.5). This will enable us to construct a curated DNN, which would be
similar to the Autoencoder in structure but had acquired some (or all) of its
training through transfer-learning. Since the activation-function is linear
and biases are not present, the output(Y) by $n^{th}$ layer is given simply by

$Y_{j}^{(n)}=\sum_{i=1}^{N}Y_{i,j}^{(n-1)}W_{i,j}^{(n-1)}$ (6)

where $W^{(n-1)}$ is the weight matrix connecting $(n-1)^{th}$ layer with
$(n)^{th}$ layer and ‘N’ is the number of neurons/nodes in $(n-1)^{th}$ layer.

###  II.4 Hard-coding the coarse graining map

Figure 6:  Scheme of hard-coding the decimation. Every other site (in Red) is
removed and only Blue sites are carried forward into the next layer.

Techniques used to course-grain the system are relations which map the lattice
from a higher dimension to lower. This map can be translated in into a graph,
which can then act as a NN where weights are hard-coded instead of trained.
This Hard-coded NN will act as a benchmark across which we will compare the
performance of other models. As discussed above the lattice has to be
linearized and decimation has to be mapped on the weight matrices($W_{i,j}$)
keeping Numpy library’s linearizing scheme in mind. Equations 7 and 8 show the
sparse matrices $W_{i,j}$ as function of indices (i,j), following the scheme
which will mimic RG through decimation in a linearized lattice.

$W_{i,j}^{1}\colonequals\begin{cases}1&\text{if
}\lfloor\frac{i}{32}+i\rfloor\text{ is even}\\\ 0&\text{else}\end{cases}$ (7)
$W_{i,j}^{2}\colonequals\begin{cases}1&\text{if
}\lfloor\frac{i}{16}\rfloor\text{ is even}\\\ 0&\text{else}\end{cases}$ (8)

Shallow-nets with hard-coded coarse-graining will be very similar to the
trained shallow-nets mentioned above, but will carry a prefix ‘H’ in their
name, for example: Coarse-grainer from $32\times 32$ to $32\times 16$ will be
called HCG-1024-512.

###  II.5 Preparing the Dataset

Figure 7:  Scheme of classifying the training data into different labelled-
sets.

Data samples are generated from a Monte-Carlo simulation of 2D Ising model for
$\beta=0.4352$, near the critical temperature. 30,000 samples of $32\times 32$
sized Ising model lattice with periodic boundary condition along 2 lattices of
consecutive decimations of each sample, bringing the total to 90,000 samples
(see Fig.2). Additionally 50,000 spin configurations of size $32\times 32$
were generated for various $\beta\in[0.01,0.6)$ to train inverse-temperature
predicting NN and a separate auto-encoder based on Model-1 (explained in
section III.3 and III.4). These sets are then bifurcated into Training and
Testing Datasets respectively. Training datasets are further bifurcated into
Training and Cross-validation datasets. A simple flowchart of dataset
classification is shown in the Fig. 7.

###  II.6 Training

Some common features among all the NN (both Shallow and Deep):

  1. 1.

2-D lattice is flatten to a 1-D vector which is fed to the Input-layer. Output
received from the NNs is also a vector which will be reconstructed into a 2-D
Lattice.

  2. 2.

Activation functions of each layer: Linear (other activations like $‘tanh’$
would have given a better result but we have tried to keep the models as
simple as possible.)

  3. 3.

Layers have no Biases.

  4. 4.

L1 regularization is used to keep the model from over-fitting. L1 penalizes
non-essential weights, thus encouraging their value to remain zero.

Both shallow and deep nets undergo training with:

  1. 1.

Loss Function: Hinge Loss

  2. 2.

Optimizer: Adam (used to overcome the shortcomings of Gradient-descent.)

  3. 3.

4000 epochs

  4. 4.

Batch size: 10

  5. 5.

Validation split: 20% of the Training Dataset

###  II.7 Measuring the Inverse-Temperature($\beta$)

Figure 8:  A 3-layered NN trained to predict inverse temperature $(\beta)$
using supervised learning.

Besides partition function(Z), temperature(T) and entropy(S) are some of the
most important quantities that define an Ising lattice. Since we have been
using inverse-temperature($\beta$) for our calculations we will designing a
NNs to measure the inverse-temperature of the Ising lattice. For this we train
a three-layer-deep NN to perform regression through supervised
learning(Fig.8). The input layer ($Y^{1}$) receives the linearized spin
configuration $(\sigma_{32\times 32})$, hence the size of input layer is 1024
(Eq.9). The hidden layer ($Y^{2}$) consist of 512 neurons, with a ‘tanh’ as
the activation function (Eq.10). Finally the output layer ($Y^{3}$) consists
of a single neuron which is used to predicted the inverse temperature of the
input configuration, activation function used here is rectified linear unit or
ReLU (Eq.11).

$Y_{i}^{(1)}=\text{Linearized }(\sigma_{m,n})$ (9)

$Y_{j}^{(2)}=\tanh\left(\sum_{i=1}^{1024}Y_{i}^{(1)}W_{i,j}^{(1)}+b_{j}^{(1)}\right)$
(10)

$Y_{1}^{(3)}=\max\Bigg{\\{}0,\left(\sum_{i=1}^{128}Y_{i}^{(2)}W_{i,1}^{(2)}+b_{1}^{(2)}\right)\Bigg{\\}}$
(11)

where $W_{i,j}$ is the weight connecting $i^{th}$ node of previous layer with
$j^{th}$ the node of current layer and $b_{i}$ is the bias of $i^{th}$ node of
the respective layer. This network will be trained with mean square err (MSE)
as the loss function.

##  III Results and Discussion

###  III.1 Losses and Learning

Figure 9:  Plot of training losses of Model-1(Blue) and Model-2(Orange) while
training. And the final loss of Model-3(Green). See Fig. A.1 in the appendix
for validation losses. Figure 10:  Reconstruction by various models (a) Input
Ising Lattice $\parallel$ (b) Model-1: Autoencoder $\parallel$ (c) Model-2: RG
Encoder-Auto Decoder $\parallel$ (d) Model-3: RG Encoder-RG Decoder.

After training the models for 4000 epoch, the hinge losses of both Models 1
and 2 are comparable 0.276 and 0.338 respectively, indicating that mapping of
weights produced by RG is a type of encoder NN. The smaller loss of Model-1
with respect to Model-2 indicates that training through classic ML techniques
such as back-propagation can provide much better encoding-decoding than the
one provided by the RG (see Fig. 9).

Model-3 is completely dependent on transfer learning for its training and we
can run this NN across the Ising-dataset to check the loss it accumulates and
the lattice configuration of the predicted output. This means that the
relations learned via RG and Rescaling can act as an Encoder-Decoder.

However, Fig.9 shows a substantial loss by Model-3 (0.6001) when compared to
Model 1 and 2, even though its constituent shallow-nets have gone through the
same number of epochs in training. The final validation losses of Model 1, 2
and 3 were 0.281, 0.341 and 0.601 respectively. This indicates that the RG
approach to learning is not efficient. Fig.10 shows a comparison of Ising
lattice reconstructed by the three models. An increasing loss of information
can be observed as we move from Autoencoder to Full-RG trained NNs.

This is probably due to the inherent difference that lies between these two
approaches.

Conjecture: _The goal of Renormalization group is to find a mathematically
solvable framework for coarse-graining, whereas the job of Autoencoder(and
Deep-Learning is general) is to find a mapping with minimum loss of
information._

Model-3 can also be modified further with hard-coded weights, which carry the
exact-mapping used in real space RG techniques like decimation/majority/naive
coarse-graining.

###  III.2 Weights and Spin-Correlation matrices

Figure 11:  Comparison of weight-matrix ($W_{1}$) $1024\times 512$ connecting
layer ($I$) with ($H_{1}$). (a) Trained on RG $\parallel$ (b) Trained on AE.
As well as weight-matrix ($W_{2}$) $512\times 256$ connecting layer $H_{1}$
with $H_{2}$. (c) Trained on RG $\parallel$ (d) Trained on AE.

Analysis of weights(W) acquired through training on all 3 models shows that ML
and RG follow very different part to optimization. DL is essentially a careful
calibration of weights (and biases) which minimizes the loss function,
therefore it would be fruitful for us to visualize the weights acquired by the
NNs as they complete their training. A plot of values aquired by weights
between various layers of NNs are shown in the Fig.11. As we can see the
layers of Model 2 have a very sparse configuration (Fig.11(a), 11(c)), while
the weights in the layers of Model 1 (the Auto-Encoder) are more homogenous
(Fig.11(b), 11(d)). This shows a stark difference between learning acquired
through RG and Autoencoders. The pattern we observe in RG trained weight
matrices can explained through Eq. 7 and 8, i.e. the trained weights are
trying to mimic the mapping done through hard-coding in Sec. II.4.

Figure 12:  Comparison of spin-correlation matrix ($WW^{T}$): $1024\times
1024(W_{1}W_{1}^{T})$ (a) Trained on RG (b) Trained on AE. $\parallel$
$512\times 512(W_{2}W_{2}^{T})$ (c) Trained on RG (d) Trained on AE.

Another important result for comparison is the spin correlation matrix
$WW^{T}$, which helps us in understanding the relations formed within the NNs
[33]. Although our models were trained differently than the Restricted
Boltzmann Machines(RBMs) used by Iso  _et al._ [33], values in $WW^{T}$ will
carry similar significance. The heat-map of $WW^{T}$ matrix from encoding
layers of model 1 and 2 are shown in (Fig.12). As we can see, large values are
mostly concentrated on the diagonal elements $(\sigma^{\prime}_{i,j})$ where
$i=j$, justifying the self-correlation of the input spins. However, presence
of large valued near-diagonal elements($j=i\pm 32,16$) in model-2 signify that
the layers with RG mapping has more spins-correlation than the layers of
Autoencoders(model-1). Please recall that the lattice($32\times 32$) was
linearized for input, therefore spin$(\sigma_{i,j})$ where $(i,j=0,...,31)$
will be transformed to $\sigma^{L}_{k}$ where $(k=0,...,1023)$. As shown by
Iso  _et al._ [33] this might signify that the AE has learned much more non-
relevant information from the dataset than necessary. This indicates over-
fitting, with relations ‘diffused’ throughout the lattice. Indicating that an
AE with smaller hidden layers may perform more efficiently than the current
one.

NOTE: The figures shown here are zoomed-in versions of the actual matrices,
please see Fig.B.1 for $W_{1}$ and Fig.B.2 for $W_{2}$. For spin correlation
matrices see Fig.B.3, B.4 in the appendix.

###  III.3 Entropy and Compression

Figure 13: Plot comparing Shannon-Entropy of the initial Ising lattice with
output lattices from Autoencoder and RG decimation.

As is the case in statistical mechanics, Entropy and its interpretations hold
a lot of significance information theory(IT). Hence Shannon’s entropy [34] (as
it’s called in IT) signifies the actual amount of information the data
carries. Since coarse-graining done by RG or dimensionality reduction done by
AE are essentially data-compression, calculating and comparing the entropy of
the lattices generated by these models will give us an interesting insight
into their compressing power. All this can be done with the help of well-
established methods in Image-processing. Since we are going to study the
change in entropy over a range of $\beta$, it would be prudent to retrain a NN
of Model-1(AE) configuration on the 50,000 spin configurations generated for
$\beta\in[0.1,0.6)$, lets call it AE-wide-range(AE-WR), see Fig.A.2 for
training losses. Decimation is not dependent on $\beta$, hence require no
changes.

Figure 14:  Comparing the output lattices of: ($1^{st}$ row) Ising lattice
$32\times 32$ at $\beta=0.025,0.25,0.5$ respectively. $\parallel$ ($2^{nd}$
row) Autoencoder $32\times 32$. $\parallel$ ($3^{rd}$ row) RG decimation
$16\times 16$.

In our case, we are interested in calculating the entropy in the neighbourhood
of each spin in the Ising lattice. For this, will we take the help of Scikit-
Image processing library [26]. We have generated a thousand lattices for each
of the 20 values of $\beta$ between 0.025 and 0.5. Each lattice also goes
through the autoencoder and coarse-graining which generates two additional
sets of lattices. Entropy is calculated in $3\times 3$ neighbourhood each spin
of the lattice, which is then averaged to the value of local entropy per spin
of a single lattice. This process is repeated for all 1000 lattices and the
average Entropy is calculated at the particular $\beta$ for all 3 sets.

The plot in Fig.13 indicates that at lower $\beta$ (high temperature, high
entropy), output from AE has much less entropy as compared to the original
lattice indicating the compression done by AE is quite lossy in nature, as it
tends to smoothen out the insignificant degrees of freedom (Fig.14). At higher
$\beta$ (low temperature, low entropy) however, the output entropy from AE
converges with the original lattice while the decimated lattice from RG
retains some of the smaller features, increasing its entropy(relatively). As
we can see from the $2^{nd}$ row of Fig.14 the autoencoder has smoothened the
features of the original lattice. This ‘de-noising’ property of AE is well
known [35]. Decimation on the other hand increases the randomness around the
neighbourhood of the spin, retaining some extra entropy($3^{rd}$ row).

Figure 15:  Plot of the Actual-$\beta$ vs Predicted-$\beta$ produced by NN
trained to measure the inverse temperature of the given Ising lattice. Figure
16:  Plot showing the $\beta$ of lattices produced by AE compared with the
mathematical results from RG decimation.

###  III.4 Inverse Temperature and the RG Flow

The $\beta$-predicting NN described in Sec.II.7 was trained on Ising lattices
for $\beta\in[0.1,0.6)$ until the mean square err(MSE) came down to 0.0035,
then a new set of 10,000 lattices were used to check the test-losses of the NN
(see Fig.15). This NN was then deployed to predict the $\beta$ of Ising
lattices produced by the AE-WR and was compared with the mathematical results
for RG decimation shown in Sec.II.1.

As we have seen in Eq.5, the new $\beta^{\prime}\sim\ln(\cosh(\beta))$ after
every decimation, this causes $\beta$ to diverge from it’s original value,
remaining unchanged only at fixed point(s) like Critical Beta($\beta_{c}$)
where $\beta^{\prime}=\beta$. This divergence/convergence from a fixed point
is called RG flow, which will help us in exploring some more the differences
between AE and RG.

As discussed in the previous section, Autoencoders have a de-noising property
[35], which reduces the randomness in the spin configuration, increasing the
$\beta^{\prime}$ of the system, hence the system with $\beta<\beta_{c}$ flows
toward $\beta_{c}$. At $\beta>\beta_{c}$ however, AE reduces the $\beta$ back
to $\beta_{c}$, again flowing towards the fixed point. This indicates a stark
difference in mechanism of learning done by AE and coarse graining of RG
(Fig.16).

It also worth noting that the Blue-line fitting the Predicted-$\beta$ from AE
intersects the $\beta=\beta^{\prime}$ line (Black) at $\beta=0.45265$ which is
much closer to the actual fixed-point $\beta_{c}=0.4352$ than the
approximation done in Sec.II.1 for RG(Red) at $\beta=0.505$.

###  III.5 Discussions

Our investigation into the parallels between RG and DL has lead us to some
anticipated results and some surprising revelations. Decimating did turn out
to be a viable scheme on which deep-net could be trained to encode Ising
lattices. However, it did not seem to be as efficient as the classic DL
techniques, with a substantial difference between the losses RG accumulated
even when the hardcoded schema was deployed. We can attribute this disparity
to the objectives behind doing RG which is very different from objectives
assigned to AE. We also studied the internal structures formed inside both
techniques and discovered that they were quite different. While RG formed a
sparse pattern, the mapping that emerged from AE was homogenised and diffused
throughout the network. Moreover, the nodes/spins in the AE schema were weakly
correlated with each other as compared to RG, indicating that AE can be
further increased. This means AE is more efficient than the RG as far as the
number of network nodes was concerned.

Since both RG and DL are performing dimensionality reduction, studying the
compression ratio and information loss across the range $\beta$ becomes the
next logical step. As entropy and information are parallel interpretations of
the same quantity, we analysed the entropy of the outputs from RG and AE, then
compared them with the entropy of the input Ising lattice. The
$\beta$-dependence of information loss due to compression revealed a clear
distinction between the two processes. While entropy of the input lattice was
comparable to RG lattice at low $\beta$, it converged to the entropy of the AE
lattice at higher $\beta$.

Finally, we shifted our focus to RG flow and set out to find anything similar
happening in deep learning. For this, we utilized a separate NN trained to
predict the inverse temperature($\beta$) of the input lattice. It was then
employed to detect the change in $\beta$ of the output from the AE and
compared it to the mathematically derived result for the RG. We found that the
outputs from AE indeed show an RG-like flow with $\beta_{c}$ as its fixed
point. Surprisingly, however, the direction of flow was opposite. Where RG
flowed away from the $\beta_{c}$, DL flowed towards it. Essentially changing
$\beta_{c}$ from a repeller to an attractor, which appears contrary to
Wilson’s interpretation of ‘relevant’ couplings/operators in RG [36, 19].
Further exploration in this area may help us in establishing a correspondence
between renormalization, machine learning and epistemic learning. In our view,
the arrival of the DL algorithm at the fixed point should be seen as
‘learning’ about the criticality of the Ising model, while coarse-graining the
system leads to ‘unlearning’ of the information about the lattice. In other
words, with auto-encoding, the unsupervised algorithm will learn about the
criticality of the 2D Ising model, knowing nothing about the mathematics
governing the model beforehand. On the other hand, with decimation, we are
progressively losing knowledge about the criticality and mechanics of the
Ising model. In this sense, RG through the decimation of 2D Ising lattice can
hardly be called ‘learning’. We suspect the same can be said for other
divergent RG flows.

###### Acknowledgements.

We would like to thank Ministry of Education, Govt. of India for funding the
research through Indian Institute of Technology Patna. Figures showing the
architecture of the neural networks were made using: LeNail (2019), NN-SVG:
Publication-Ready Neural Network Architecture Schematics; Journal of Open
Source Software, 4(33), 747, https://doi.org/10.21105/joss.00747

## References

  * McCarthy  _et al._ [2006] J. McCarthy, M. L. Minsky, N. Rochester, and C. E. Shannon, A proposal for the dartmouth summer research project on artificial intelligence, august 31, 1955, AI Magazine 27, 12 (2006).
  * Samuel [1959] A. L. Samuel, Some studies in machine learning using the game of checkers, IBM Journal of Research and Development 3, 210 (1959).
  * Little [1974] W. Little, The existence of persistent states in the brain, Mathematical Biosciences 19, 101 (1974).
  * Hopfield [1982] J. J. Hopfield, Neural networks and physical systems with emergent collective computational abilities, Proceedings of the National Academy of Sciences 79, 2554 (1982).
  * Amit  _et al._ [1985] D. J. Amit, H. Gutfreund, and H. Sompolinsky, Spin-glass models of neural networks, Phys. Rev. A 32, 1007 (1985).
  * Amit [1992] D. J. Amit,  _Modeling brain function: The world of attractor neural networks_ (Cambridge university press, 1992).
  * Teimoorinia [2012] H. Teimoorinia, Spectral classification of galaxies at 0.5 $\leqslant$z$\leqslant$ 1 in the cdfs: The artificial neural network approach, The Astronomical Journal 144, 172 (2012).
  * Decelle  _et al._ [2019] A. Decelle, V. Martin-Mayor, and B. Seoane, Learning a local symmetry with neural networks, Phys. Rev. E 100, 050102 (2019).
  * Umrigar  _et al._ [1988] C. Umrigar, K. Wilson, and J. Wilkins, Optimized trial wave functions for quantum monte-carlo calculations, Phys. Rev. Lett. 60, 1719 (1988).
  * Ch’Ng  _et al._ [2017] K. Ch’Ng, J. Carrasquilla, R. G. Melko, and E. Khatami, Machine learning phases of strongly correlated fermions, Phys. Rev. X 7, 031038 (2017).
  * Torlai and Melko [2016] G. Torlai and R. G. Melko, Learning thermodynamics with boltzmann machines, Phys. Rev. B 94, 10.1103/physrevb.94.165134 (2016).
  * Shah [2019] S. N. Shah, Variational approach to unsupervised learning, Journal of Physics Communications 3, 075006 (2019).
  * Lin  _et al._ [2017] H. W. Lin, M. Tegmark, and D. Rolnick, Why does deep and cheap learning work so well?, Journal of Statistical Physics 168, 1223 (2017).
  * Amin  _et al._ [2018] M. H. Amin, E. Andriyash, J. Rolfe, B. Kulchytskyy, and R. Melko, Quantum boltzmann machine, Phys. Rev. X 8, 021050 (2018).
  * Puškarov and Cubero [2018] T. Puškarov and A. C. Cubero, Machine learning algorithms based on generalized gibbs ensembles, Journal of Statistical Mechanics: Theory and Experiment 2018, 103102 (2018).
  * McCulloch and Pitts [1943] W. S. McCulloch and W. Pitts, A logical calculus of the ideas immanent in nervous activity, The bulletin of Mathematical Biophysics 5, 115 (1943).
  * Rosenblatt [1958] F. Rosenblatt, The perceptron: a probabilistic model for information storage and organization in the brain., Psychological Review 65, 386 (1958).
  * Stueckelberg [1953] A. Stueckelberg, E.C.G.; Petermann, La renormalisation des constants dans la théorie de quanta, Helv. Phys. Acta 26, 499–520 (1953).
  * Wilson [1983] K. G. Wilson, The renormalization group and critical phenomena, Rev. Mod. Phys. 55, 583 (1983).
  * Li and Wang [2018] S.-H. Li and L. Wang, Neural network renormalization group, Phys. Rev. Lett. 121, 260601 (2018).
  * Bény [2013] C. Bény, Deep learning and the renormalization group, arXiv preprint arXiv:1301.3124 (2013).
  * Mehta and Schwab [2014] P. Mehta and D. J. Schwab, An exact mapping between the variational renormalization group and deep learning, arXiv preprint arXiv:1410.3831 (2014).
  * Chollet  _et al._ [2015] F. Chollet _et al._ , Keras, https://keras.io (2015).
  * Abadi  _et al._ [2015] M. Abadi _et al._ , TensorFlow: Large-scale machine learning on heterogeneous systems (2015), software available from tensorflow.org.
  * Harris  _et al._ [2020] C. R. Harris _et al._ , Array programming with NumPy, Nature 585, 357–362 (2020).
  * [26] Img-entropy-scikit.
  * Hunter [2007] J. D. Hunter, Matplotlib: A 2d graphics environment, Computing in Science & Engineering 9, 90 (2007).
  * NVIDIA  _et al._ [2020] NVIDIA, P. Vingelmann, and F. H. Fitzek, Cuda, release: 10.2.89 (2020).
  * Ising [1925] E. Ising, Beitrag zur theorie des ferromagnetismus, Zeitschrift für Physik 31, 253 (1925).
  * Rumelhart  _et al._ [1986] D. E. Rumelhart, G. E. Hinton, and R. J. Williams, Learning internal representations by error propagation, in  _Parallel Distributed Processing: Explorations in the Microstructure of Cognition, Vol. 1: Foundations_ (MIT Press, Cambridge, MA, USA, 1986) p. 318–362.
  * Hinton and Salakhutdinov [2006] G. E. Hinton and R. R. Salakhutdinov, Reducing the dimensionality of data with neural networks, Science 313, 504 (2006).
  * LeCun  _et al._ [1989] Y. LeCun, B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel, Backpropagation applied to handwritten zip code recognition, Neural Computation 1, 541 (1989).
  * Iso  _et al._ [2018] S. Iso, S. Shiba, and S. Yokoo, Scale-invariant feature extraction of neural network and renormalization group flow, Phys. Rev. E 97, 053304 (2018).
  * Shannon [1948] C. E. Shannon, A mathematical theory of communication, The Bell system technical journal 27, 379 (1948).
  * Vincent  _et al._ [2008] P. Vincent, H. Larochelle, Y. Bengio, and P.-A. Manzagol, Extracting and composing robust features with denoising autoencoders, in  _Proceedings of the 25th international conference on Machine learning_ (2008) pp. 1096–1103.
  * Wilson and Fisher [1972] K. G. Wilson and M. E. Fisher, Critical exponents in 3.99 dimensions, Phys. Rev. Lett. 28, 240 (1972).

##  Appendix A Training Plots

Figure A.1:  Plot of validation losses of Model-1(Red) and Model-2(Yellow)
while training. And the final loss of Model-3(Purple). Validation losses after
4000 epochs are 0.281, 0.341 and 0.601 respectively. Corresponding training
losses were 0.275, 0.338 and 0.6 respectively. Figure A.2:  Plot of
training(Blue) and validation(Red) losses of AE-WR while training. Final
validation loss after 4000 epochs was 0.357, training loss was 0.347.

##  Appendix B High Resolution Images

Here are the High-Resolution Images mentioned in the previous sections.

Figure B.1:  Weight-matrix ($W_{1}$) $1025\times 512$ connecting layers $I$,
$H_{1}$. (Left) RG $\parallel$ (Right) AE.

Figure B.2:  Weight-matrix ($W_{2}$) $512\times 256$ connecting layers
$H_{1}$, $H_{2}$. (Left) RG $\parallel$ (Right) AE.

.

Figure B.3:  Spin-correlation matrix $W_{1}W_{1}^{T}$ $(1024\times 1024)$:
(Left) Trained on RG $\parallel$ (Right) Trained on AE.

Figure B.4:  Spin-correlation matrix $W_{2}W_{2}^{T}$ $(512\times 512)$:
(Left) Trained on RG $\parallel$ (Right) Trained on AE.

##  Appendix C Program Skeleton

  1. 1.

Include libraries;

  2. 2.

Initialize hyper-parameters;

…

  3. 3.

Initialize a Multilayer Perceptron;

  4. 4.

Using a tree based model, add layers;

  5. 5.

Compile the Model with optimizer, loss function;

…

  6. 6.

Load the Dataset;

  7. 7.

Parse the data-files into variable tensors;

  8. 8.

Normalise and resize tensors;

  9. 9.

Load weights from external source if using Transfer-learning;

…

  10. 10.

Fit the Model into the training data with ample epochs;

  11. 11.

Test the accuracy of the trained model on the Testing-Dataset;

…

  12. 12.

Plot the graph of Loss Function vs Epochs;

  13. 13.

Plot the actual Output vs predicted Output

  14. 14.

Save the trained Model, weights and biases:

  15. 15.

End;

##  Appendix D Table comparing models

Comparison of the 3 models
---
|  Model-1 Autoencoder |  Model-2 TL Encoder Unsupervised Decoder |  Model-3 TL Encoder TL Decoder
Input-layer: ‘I’ Size: 1024 |  Input of a linearized Ising Lattice of 32x32 |  Input of a linearized Ising Lattice of 32x32 |  Input of a linearized Ising Lattice of 32x32
Hidden-layer: ‘H1’ Size: 512 |  Encode input from ‘I’ down to 512 units. Trainable. |  Encode input from ‘I’ down to 512 units. Not trainable as learning is transferred from HCG-1024-512 |  Encode input from ‘I’ down to 512 units. Not trainable as learning is transferred from CG-1024-512
Hidden-layer: ‘H2’ Size: 256 |  Encode input from ‘E1’ down to 256 units. Trainable. |  Encode input from ‘E1’ down to 256 units. Not trainable as learning is transferred from HCG-512-256 |  Encode input from ‘E1’ down to 256 units. Not trainable as learning is transferred from CG-512-256
Hidden-layer: ‘H3’ Size: 512 |  Decode input from ‘E2’ up scaled to 512 units. Trainable. |  Decode input from ‘E2’ up scaled to 512 units. Trainable. |  Decode input from ‘E2’ up-scaled to 512 units. Not trainable as learning is transferred from UP-256-512
Output-layer: ‘O’ Size: 1024 |  Decode input from ‘D’ up scaled to 1024 units. And is compared with linearized input lattice of 32x32 |  Decode input from ‘D’ up scaled to 1024 units. And is compared with linearized input lattice of 32x32 |  Decode input from ‘D’ up-scaled to 1024 units. Not trainable as learning is transferred from UP-512-1024
Table 1:  A comparison between techniques utilized in training various NNs.


# SensoryT5: Infusing Sensorimotor Norms into T5 for Enhanced Fine-grained
Emotion Classification

###### Abstract

In traditional research approaches, sensory perception and emotion
classification have traditionally been considered separate domains. Yet, the
significant influence of sensory experiences on emotional responses is
undeniable. The natural language processing (NLP) community has often missed
the opportunity to merge sensory knowledge with emotion classification. To
address this gap, we propose SensoryT5, a neuro-cognitive approach that
integrates sensory information into the T5 (Text-to-Text Transfer Transformer)
model, designed specifically for fine-grained emotion classification. This
methodology incorporates sensory cues into the T5’s attention mechanism,
enabling a harmonious balance between contextual understanding and sensory
awareness. The resulting model amplifies the richness of emotional
representations. In rigorous tests across various detailed emotion
classification datasets, SensoryT5 showcases improved performance, surpassing
both the foundational T5 model and current state-of-the-art works. Notably,
SensoryT5’s success signifies a pivotal change in the NLP domain, highlighting
the potential influence of neuro-cognitive data in refining machine learning
models’ emotional sensitivity.

Keywords: emotion classification, sensory information, attention mechanism,
pre-trained language model

SensoryT5: Infusing Sensorimotor Norms into T5 for Enhanced Fine-grained
Emotion Classification

Yuhan Xia1, Qingqing Zhao2, Yunfei Long1, Ge Xu3, Jia Wang4
---
1School of Computer Science and Electronic Engineering, University of Essex
2Institute of Linguistics, Chinese Academy of Social Sciences
3College of Computer and Control Engineering, Minjiang University
4Department of Intelligent Science, Xi’an Jiaotong-Liverpool University
{yx23989<EMAIL_ADDRESS>
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>

Abstract content

## 1\. Introduction

Affective computing stands at the intersection of technology and human
emotions Li et al. (2017), whereby sentiment analysis and emotion recognition
are generally merged to give machines a semblance of human-like emotional
understanding. Specifically, sentiment analysis (SA) seeks to decode the
attitudes and viewpoints of opinion holders using computational methods Lu et
al. (2023), providing a coarse-grained categories of polarities: positive,
negative, or neutral Long et al. (2019b). Driven by recent advancements in
deep learning and bolstered by vast labeled datasets, discriminating
sentiments in standard contexts has become progressively more tractable.
Cutting-edge models, including the likes of BERT Devlin et al. (2018), XLNet
Yang et al. (2019), and the T5 Raffel et al. (2020) series, have consistently
set benchmarks, achieving high accuracies on an array of sentiment
classification tasks.

By contrast, emotion analysis (EA) has received less notable results in recent
years. One of the reasons is that different from SA offering a coarse-grained
outlook, EA paints a detailed picture. That is, EA not only distinguishes
between basic sentiments but also identifies nuanced emotions such as joy,
anger, sadness, surprise, and among others Ekman (1992). Thus, the task of EA
is complicated by the sheer variety of emotional categories. For instance,
distinguishing closely related emotions like "contentment" and "happiness" or
"annoyance" and "anger" requires a discerning approach, especially when the
medium is textual content. Thus, this study introduces a SensoryT5 model,
tailored to infuse sensory data, which is cognitively more related to emotions
and includes linguistically more enriched features, into neural architectures,
to achieve a profound comprehension of emotions.

The relationship between emotion and perception/sensation has been verified
repeatedly in various disciplines. From a neuroscientific perspective, emotion
and sensory information are processed in an overlapping neural region, i.e.,
the amygdala Šimić et al. (2021). Shifting the lens to psychology, emotion and
perception are intertwined Zadra and Clore (2011). For example, the sense of
taste shows an inherent link with reward and aversion mechanisms, such as
sucrose being perceived as sweet and desirable, whereas quinine being
recognized as bitter and repulsive (Yamamoto, 2008). In addition, emotion as a
kind of interoception forms an indispensable part of human sensations, when a
wide definition of sensory perception adopted Connell et al. (2018); Lynott et
al. (2020). In terms of the linguistic conceptualization of emotions, people
more frequently use figurative language instead of literal emotion terms to
convey emotions Fainsilber and Ortony (1987); Lee (2018), and the conceptual
metaphor EMOTION IS PERCEPTION is grounded in abundant language usages to show
that the human senses are fruitful sources for verbalizing emotions (e.g.,
sweet and bitter) Lakoff and Johnson (1980); Kövecses (2019); Müller et al.
(2021).

Given the intertwined relation between emotion and perception/sensation, this
study posits that incorporating sensory information into a computational
framework can capture the nuanced interplay between them, hence offering a
reflection of intricate human affective understanding. Specifically, we utlize
the Lancaster Sensorimotor Norms (Lynott et al., 2020), which include
language-specific lexical properties representing the correlation between
conceptualized lexical meanings and sensory modalities.

Our work boasts three pivotal advancements: (1) We introduce SensoryT5, an
innovative architecture that enhances transformer-based fine-grained emotion
classification models by seamlessly embedding sensory knowledge. Marking one
of the pioneering endeavors, SensoryT5 is adapted at harmonizing both the
nuances of contextual attention and the intricacies of sensory information-
based attention. (2) The SensoryT5 leverages sensorimotor norms within
transformer text classification frameworks, contributing to the ongoing
efforts to incorporate neuro-cognitive data in NLP tasks. Thus, our work not
only demonstrates the practical benefits of this integration in improving
emotion classification tasks, but also encourages continued interdisciplinary
dialogue and research between the domains of language processing and neuro-
cognitive science. (3) Assessments across multiple real-world datasets
pertinent to fine-grained emotion classification affirm that our approach
amplifies the efficacy of pre-existing models considerably, even surpassing
contemporary state-of-the-art methodologies on selected datasets. This
endeavor underscores the value of cognition-anchored data in sculpting
attention models. Our findings illuminate the untapped potential of sensory
information in refining emotion classification, carving fresh prospects for
exploration within the realm of affective computing in NLP.

## 2\. Related work

### 2.1. Emotion analysis

Over recent years, the domain of pre-trained language models (PLMs) and large
language models (LLMs) has witnessed marked advancements. Noteworthy
developments include models like BERT (Devlin et al., 2018), RoBERTa (Liu et
al., 2019), GPT-3 (Brown et al., 2020), T5 (Raffel et al., 2020), PaLM
(Chowdhery et al., 2022), LLaMA (Touvron et al., 2023) and ChatGPT (OpenAI,
2023). These models, through rigorous pre-training on vast text corpora using
self-supervised learning, have the ability to autonomously generate intricate
representations. This capability has significantly advanced the field, setting
new benchmarks in numerous tasks, notably in sentiment analysis (Devlin et
al., 2018; Zhang et al., 2023). An in-depth exploration by Zhang et al. (2023)
elucidated the performances of LLMs in sentiment and emotion analysis tasks.
The study has highlighted that, while LLMs excel over PLMs in few-shot
learning scenarios, PLMs remain superior for more nuanced tasks that demand a
deeper understanding of emotions or structured emotional data. Among the
discussed models, T5 (Raffel et al., 2020) stands out due to its innovative
’text-to-text’ transfer approach, in which every NLP challenge is remodelled
as a text-to-text problem. Consequently, T5 frequently sets the state-of-the-
arts in emotion analysis when utilized as the base model.

However, despite the considerable improvements made with these PLMs/LLMs, some
research gaps remain relatively fulfilled. Present models, although they
possess sophisticated neural architectures capable of discerning patterns from
immense text datasets, often overlook the intricate nature of emotion—a
dynamic interplay of cognitive and physiological responses triggered by
various stimuli Khare et al. (2023). Sensory perceptions, pivotal in shaping
these responses, serve as the bedrock upon which our cognitive processes
evaluate and generate emotions Niedenthal and Wood (2019). Integrating these
models with sensory data can potentially elevate their performance, nudging
them closer to approaching human-like comprehension. This presents a
significant research opportunity: equipping already potent PLMs/LLMs with an
element of sensory perception, an aspect they conventionally lack. With our
proposed SensoryT5 model, our ambition is to fill this gap by synergizing the
strengths of T5 and augmenting it with sensory knowledge, thereby enabling a
deeper and more nuanced understanding of emotions.

### 2.2. Cognition-grounded resources: Sensorimotor norms

In recent years, there is an emergent trend that neuro-cognitive data and
computational approaches are synergized in NLP studies. This interdisciplinary
synergy unlocks new dimensions in understanding language, sentiment, and
emotion, reflecting more accurately the human experience and mental
processing. For instance, Long et al. (2019b) improved the attention model for
sentiment analysis by incorporating a eye-tracking dataset. Chen et al. (2021)
incorporated brain measurement data for modeling word embedding. Wan et al.
(2023) demonstrated the superiority of neural networks for metaphor detection
by leveraging sensorimotor knowledge. These studies collectively underscore a
broader shift in the field towards a more integrated approach to NLP. By
weaving in neuro-cognitive data, researchers are equipping computational
models with a richer and more intricate understanding of human language and
cognition, which are often overlooked by traditional data-driven methods.

Given the intimate connection between emotion and perception as demonstrated
in various studies reviewed in the last section, this study assumes that a
cognitively and linguistically motivated representation of words in text based
on sensorimotor knowledge would improve the performance of computational
models for emotion analysis. That is not only because sensory inputs are
crucial sources of emotions, but also because emotional responses are part of
sensory perceptions for human beings.

Thus, this study utilizes Lynott et al. (2020)’s sensorimotor norms which
encompass metrics of sensorimotor strengths (ranging from 0 to 5) of 39,707
concepts spanning six perceptual domains: touch, hearing, smell, taste,
vision, and interoception, as well as five action effectors: mouth/throat,
hand/arm, foot/leg, head (barring mouth/throat), and torso. To exploit this
wealth of data, SensoryT5 is proposed to construct the sensorimotor vectors
from these norms and to seamlessly embed them into the T5’s decoder mechanism
via an auxiliary attention layer. Positioned after the decoders, this sensory-
centric attention layer is synergized with the decoder’s output, producing an
enriched representation brimming with sensory knowledge for words in text.
Thus, SensoryT5 is adapted at simultaneously discerning contextual cues and
sensory knowledge, allowing for a potent alignment of sensory nuances with
contextual intelligence. This integration augments the model’s efficacy in the
fine-grained emotion classification.

## 3\. Our proposed SensoryT5 model

Figure 1: An overview of SensoryT5. Blue box shows a T5 process of deep
learning, while purple box describing sensory information is quantified and
passed into the T5.

In this section, we elaborate how our SensoryT5 model incorporates the sensory
knowledge into the neural emotion classification framework. Specifically,
sensory knowledge is infused into the T5 using an adapter approach built upon
attention mechanisms. Moreover, the contextual and sensory information
learning branches are amalgamated within a unified loss function to facilitate
joint training. The overarching structure is depicted in Figure 1.

### 3.1. Preliminaries

Despite the relatively large size of the Lancaster Sensorimotor Norms, there
are still many out-of-vocabulary words. Following the method proposed by Li et
al. (2017), we use a word embedding model to regressively predict the sensory
values of unknown words, aiming to obtain sensory values for out-of-vocabulary
words.
Inputs and outputs The objective of emotion analysis is to determine and
categorize opinions for a piece of texts following a defined label schema. Let
$D$ denote a collection of documents for emotion classification. Each document
$d\in D$ is first tokenized into a word sequence with maximum length $n$, then
the word embeddings $w_{i}$ of these sequence are jointly employed to
represent the document $d={w_{1},w_{2},...,w_{i},...,w_{n}}(i\in 1,2,...,n)$.

### 3.2. The core attention mechanism in T5

The word embeddings of these sequence
$d={w_{1},w_{2},...,w_{i},...,w_{n}}(i\in 1,2,...,n)$ first enters the T5.
Each layer of the encoder and decoder has a series of multi-head attention
units. The multi-head attention mechanism for the final decoder layer can be
represented using the following equation:

$\displaystyle V_{d}$ $\displaystyle=\text{MultiHead}(Q_{0},K_{0},V_{0})$ (1)
$\displaystyle=[\text{head}_{1},\text{head}_{2},...,\text{head}_{i}]W_{O}$

Where each head is computed as:

$\displaystyle\text{head}_{i}$
$\displaystyle=\text{Attention}(Q_{0}W_{i}^{Q},K_{0}W_{i}^{K},V_{0}W_{i}^{V})$
(2)
$\displaystyle=\text{softmax}\left(\frac{(Q_{0}W_{i}^{Q})(K_{0}W_{i}^{K})^{T}}{\sqrt{d_{k}}}\right)V_{0}W_{i}^{V}$

$W_{i}^{Q}$, $W_{i}^{K}$, and $W_{i}^{V}$ are weight matrices that are learned
during the training process. They are used to project the input queries ($Q$),
keys ($K$), and values ($V$) to different sub-spaces. $Q_{0}$, $K_{0}$, and
$V_{0}$ are derived from the output of the penultimate decoder layer.
Additionally, following the common practice for text classification with T5,
we employ a zero-padding vector as the sole input for the decoder. The result
$V_{d}$ is the output of the T5 decoder, imbued with context-aware attention.
Both $V_{d}$ and $K_{0}$ will be utilized in section 3.4 for integration with
sensory knowledge.

### 3.3. Sensory information transformation for T5 integration

We project the Lancaster Sensorimotor Norms into a sensory word vector space.
Each word is linked with a six-dimensional vector representing sensory scores
across six perceptual modalities (auditory, gustatory, haptic, interoceptive,
olfactory and visual dimensions). For a word $w$, its sensory vector is
denoted as $s(w)=[s_{1},s_{2},...,s_{6}]$.

To enable effective integration into the T5-large, we use two linear
transformations followed by a ReLU activation function to map the sensory
vectors to the same dimension as the T5-large’s word embeddings. Given a
T5-large model with an embedding dimension of $1024$ , the transformation
process can be formally described as:

$\mathbf{h}_{1}=\text{ReLU}(\mathbf{W}_{1}s(w)+\mathbf{b}_{1})$ (3)

$s^{\prime}(w)=\mathbf{W}_{2}\mathbf{h}_{1}+\mathbf{b}_{2}$ (4)

where $\mathbf{W}_{1}:\mathbf{R}^{6}\rightarrow\mathbf{R}^{128}$ and
$\mathbf{W}_{2}:\mathbf{R}^{128}\rightarrow\mathbf{R}^{1024}$ are two linear
transformation matrices and $\mathbf{b}_{1}$, $\mathbf{b}_{2}$ are the
respective bias terms.The shapes of the two weight matrices $W_{1}$ and
$W_{2}$ are respectively $(6,128)$ and $(128,1024)$. The output $h_{1}$ of the
first linear layer is a vector of shape $(1,128)$, and the output
$s^{\prime}(w)$ of the second linear layer is a vector of shape $(1,1024)$.
After the transformation, the sensory vector $s^{\prime}(w)$ is projected into
the same semantic space as the features generated by T5-large. The output
vector $s^{\prime}(w)$, with $V_{d}$ and $K_{d}$ from the T5, will be applied
in section 3.4 for infusing sensory knowledge into T5.

### 3.4. Sensory attention mechanism in SensoryT5

The sensory vector $s^{\prime}(w)$ generated by the sensory vector
transformation is used as the queries in the attention mechanism of the
sensory adapter, substituting the query vector $Q$ in the T5. The sensory
adapter performs the attention calculation as follows:

$\displaystyle A_{d}$
$\displaystyle=\text{MultiHead}(s^{\prime}(w),K_{0},V_{d})$ (5)
$\displaystyle=[a_{1},a_{2},...,a_{i}]W_{d}$

where each head is computed as:

$\displaystyle a_{i}$
$\displaystyle=\text{Attention}(s^{\prime}(w)W_{i}^{Q},K_{0}W_{i}^{K},V_{d}W_{i}^{V})$
(6)
$\displaystyle=\text{Softmax}\left(\frac{(s^{\prime}(w)W_{i}^{Q})(K_{0}W_{i}^{K})^{T}}{\sqrt{d_{k}}}\right)V_{d}W_{i}^{V}$

Once the output $A_{d}={a_{1},a_{2},...,a_{n}}$ of the sensory adapter is
obtained, we apply dropout and pooling operations to form a final
representation $P_{d}$, which is then used as the input to the classification
layer.

$P_{d}=\text{Dropout}(\text{Pool}(A_{d}))$ (7)

The pooled representation $P_{d}$ is then fed into the classifier of the T5.

$C_{d}=\text{Softmax}(Linear(\text{Dropout}(P_{d})))$ (8)

$C_{d}$ is a probability distribution vector. The class with the highest
probability is selected as the predicted label, denoted as $y$.

The first step of the back-propagation process involves computing the gradient
of the loss function with respect to the parameters of sensory attention
adapter. $\Theta_{\text{A}}$ represents the parameters of the sensory
attention layer, and $A_{d}$ represents the output of the sensory T5. The
computed gradient is used to update the parameters of the attention layer,
enhancing its capacity to integrate sensory information into the T5 model.
This is computed as follows:

$\frac{\partial\mathcal{L}}{\partial\Theta_{\text{A}}}=\frac{\partial\mathcal{L}}{\partial
A_{d}}\cdot\frac{\partial A_{d}}{\partial\Theta_{\text{A}}}$ (9)

After the gradients for the sensory attention mechanism have been computed, we
then compute the gradients for the parameters of the final layer of the T5,
denoted as $\Theta_{\text{E}}$.

$\frac{\partial\mathcal{L}}{\partial\Theta_{\text{E}}}=\frac{\partial\mathcal{L}}{\partial
V_{d}}\cdot\frac{\partial V_{d}}{\partial\Theta_{\text{E}}}$ (10)

Finally, the gradients for the sensory information transformation, denoted as
$\Theta_{\text{S}}$, are computed as follows:

$\frac{\partial\mathcal{L}}{\partial\Theta_{\text{S}}}=\frac{\partial\mathcal{L}}{\partial
s^{\prime}(w)}\cdot\frac{\partial s^{\prime}(w)}{\partial\Theta_{\text{S}}}$
(11)

Here, $\Theta_{\text{S}}$ represents the parameters of the sensory information
transformation component, which includes the weights and biases of the two
linear layers, and $s^{\prime}(w)$ represents the output of this component.
The calculated gradient is used to update the parameters of the sensory
information transformation to improve its ability to capture and model sensory
information. Through these calculations, we are able to update the parameters
of the sensory attention mechanism, the T5, and the sensory information
transformation component.

## 4\. Experimental evaluation

### 4.1. Datasets

We have selected four benchmark datasets of varying sizes to encompass a
variety of classification tasks: Empathetic Dialogues (ED) Rashkin et al.
(2019), GoEmotions (GE) Demszky et al. (2020a), ISEAR Scherer and Wallbott
(1994) and EmoInt Mohammad and Bravo-Marquez (2017). For the GE dataset, we
exclusively utilize samples with a single label and omit those that are
neutral to maintain an equitable comparison with prior studies Suresh and Ong
(2021a); Chen et al. (2023). Table 1 presents a summary of key statistics for
these datasets. Our evaluation utilizes two widely recognized performance
metrics: accuracy and the F1 score, in line with state-of-the-art studies.

Dataset | $N_{\text{train}}$ | $N_{\text{test}}$ | L | C
---|---|---|---|---
ED | 19,533 | 2,547 | 18 | 32
GE | 23,485 | 2,984 | 12 | 27
ISEAR | 4,599 | 1,534 | 22 | 7
EmoInt | 3,612 | 3,141 | 16 | 4
Table 1: Statistics of the four benchmark datasets. In the table,
"$N_{\text{train}}$" and "$N_{\text{test}}$" respectively represent the number
of instances in the training and testing sets. "L" stands for the average text
length within the dataset, and "C" indicates the number of classes/categories.

### 4.2. Sensory knowledge

Figure 2: Histograms showing the distribution six sensory values over words.
X-axis shows the value in an sensory dimension, while y-axis displays the word
density.

Before conducting the emotion analysis experiments, we conducted a preliminary
analysis of our sensory lexicons from the perspective of sensory perception
value distribution. Figure 2 displays histograms of the six sensory measures
across all words within our model. Notably, the distributions for these
measures are quite unbalanced. Gustatory and olfactory measures predominantly
demonstrate a left-skewed distribution, with most values ranging between 0 and
1. This suggests that these two sensory perceptions are less frequently
represented in the textual context. Thus, it might be challenging to represent
gustatory and olfactory perceptions from text.

In contrast, auditory and visual measures show a relatively uniform
distribution. The auditory measure is evenly distributed between 0 and 2.5,
while the visual measure ranges between 2 and 4.5. These distributions
indicate a higher sensitivity of auditory and visual knowledge to textual
information, which suggests that auditory and visual senses may play a
significant role within sensory models.

Lastly, haptic and interoceptive measures exhibit similar trends, declining
from about 2500 to 0 as the values increase from 0 to 5. The decline in the
presence of haptic and interoceptive knowledge across the general textual
context might suggest that they are less informative sensory dimensions in the
majority of cases.

As discussed in section 3.1, the Lancaster Sensorimotor Norms dataset is
subject to size limitations, resulting in a significant number of unknown
words for which corresponding sensory values are unavailable. To address this
challenge, we adopted the method proposed by Li et al. (2017) for predicting
sensory values of unknown words through embedding techniques. In our
experiments, we utilized both the T5 embedding and the GloVe embedding
(Pennington et al., 2014b) for this prediction task.

To assess the accuracy of our predictions, we randomly selected 10% of the
Lancaster Sensorimotor Norms dataset as a validation set and applied the Root
Mean Square Error (RMSE) as the evaluation metric. The experimental results
are presented in Table 2. The results demonstrate that GloVe outperforms T5
Embedding in predicting each sensory dimension. To preserve the original
features of the Lancaster dataset to a minimal extent, we opted for a smaller
version of GloVe with 400,000 data points and 200 dimensions. Following
augmentation, our sensory vocabulary size reached 407,572111We will release
the sensory vector after this paper is accepted.

Sensory Name | T5 Embedding | GloVe
---|---|---
Auditory | 0.949 | 0.803
Gustatory | 0.632 | 0.534
Haptic | 0.893 | 0.698
Interoceptive | 0.831 | 0.662
Olfactory | 0.572 | 0.501
Visual | 0.842 | 0.743
Total | 0.798 | 0.665
Table 2: Comparison of prediction accuracy between T5 Embedding and GloVe
techniques on different sensory dimensions, as measured by RMSE values. Lower
scores indicate higher accuracy in the prediction of sensory values.

For validating our augmentation, we evaluated the coverage rates of sensory
word vectors before and after augmentation across all datasets we employed, as
detailed in Table 3. As evident from the augmentation results, the coverage
range significantly expands in comparison to the original data across all
datasets. This underscores the enhanced impact of integrating sensory
information into the model on the results.

Datasets | Lancaster % | Exten-Lancaster %
---|---|---
ED | 58.23 | 91.78
GE | 46.85 | 83.91
ISEAR | 54.62 | 78.97
EmoInt | 29.65 | 46.21
Table 3:  Word coverage of Lancaster Sensorimotor Norms before and after
expansion using regression prediction.

### 4.3. Experiment settings and Baselines

We compare the proposed SensoryT5 primarily with two group of strong
baselines:

PLMs. We compared against BERT Devlin et al. (2019), RoBERTa Liu et al.
(2019), XLNet Yang et al. (2019) and T5 Raffel et al. (2020). The advent of
PLMs has marked a significant improvement across a multitude of tasks in the
realm of natural language processing, including text classification. This leap
in performance is largely due to the deep and nuanced semantic representations
these models extract from the text, facilitating a more profound understanding
and interpretation of linguistic content.

Label Embedding-aware models. Suresh and Ong (2021a) introduced a concept
called label-aware contrastive loss (LCL). This technique uniquely assigns
varying weights to each negative sample. Importantly, pairs that are more
easily confounded have a higher impact on the objective function, enhancing
outcomes in fine-grained text classification scenarios. Chen et al. (2023)
proposed HypEmo, a framework enhancing fine-grained emotion classification by
utilizing hyperbolic space for label embedding. This model integrates
hyperbolic and Euclidean geometries to discern subtle nuances among labels
effectively.

These two models, LCL and HypEmo, stand as the most potent in the realm of
fine-grained emotion classification, delivering unparalleled results due to
their innovative handling of nuanced label distinctions and hierarchical
intricacies.

Implementation Details.222We will release the open-source code after this
paper is accepted During training, we applied the Adam optimizer in Euclidean
space. We set the learning rate at a consistent $10^{-4}$, maintaining a
balance between rapid adaptation and the stability of learning, reducing the
likelihood of oscillation or divergence.

| Empathetic Dialogue | GoEmotions | ISEAR | EmoInt
---|---|---|---|---
| ACC | F1 | ACC | F1 | ACC | F1 | ACC | F1
*$\text{BERT}_{\text{large}}$ | 0.557 | 0.551 | 0.642 | 0.637 | 0.677 | 0.679 | 0.848 | 0.848
*$\text{RoBERTa}_{\text{large}}$ | 0.596 | 0.590 | 0.652 | 0.644 | 0.723 | 0.720 | 0.865 | 0.865
*$\text{XLNet}_{\text{large}}$ | 0.599 | 0.592 | 0.641 | 0.568 | 0.711 | 0.711 | 0.845 | 0.845
*$\text{T5}_{\text{large}}$ | 0.609 | 0.604 | 0.661 | 0.657 | 0.717 | 0.717 | 0.863 | 0.863
†LCL | 0.601 | 0.591 | 0.655 | 0.648 | 0.724 | 0.724 | 0.866 | 0.866
§HypEmo | 0.596 | 0.610 | 0.654 | 0.663 | 0.707 | 0.712 | 0.846 | 0.846
*SensoryT5 | 0.618 | 0.615 | 0.674 | 0.670 | 0.726 | 0.724 | 0.875 | 0.875
Table 4: Evaluation on fine-grained emotion classification, the result with
the best performance are highlighted in bold. Data marked with †are from
(Suresh and Ong, 2021b), §from (Chen et al., 2023), and *represents our own
results. Note: In §, results from missing datasets (ISEAR and EmoInt) were
supplemented by our experiments.

### 4.4. Baseline comparison

To demonstrate the effectiveness of SensoryT5, we embarked on a comprehensive
set of comparative experiments, analyzing its performance in emotion
classification tasks. The comparison is shown in Table 4. Firstly, we compare
SensoryT5 with PLMs. SensoryT5 registers an impressive enhancement over T5’s
performance, the best of the PLM contenders. For instance, SensoryT5 exhibits
an increase in accuracy by 0.9% for Empathetic Dialogues and 1.3% for
GoEmotions, showcasing its finesse in handling diverse emotional contexts.
This upward trend continues with ISEAR and EmoInt datasets, where SensoryT5
improves by 0.9% and 1.2%, respectively, over T5.

Secondly, we compare with label-aware system. These two models, LCL and
HypEmo, stand as the most potent in the realm of fine-grained emotion
classification. LCL outperforms T5 in the ISEAR and EmoInt datasets, while the
other datasets under the label-aware system category do not compete favorably
with T5. This comparative analysis is critical, considering that LCL utilizes
a synonym substitution technique to effectively double its dataset size. Such
an expansion contributes significantly to its enhanced performance metrics. In
our experiments, we strictly adhered to using original samples without
resorting to any form of data augmentation techniques. Despite this, SensoryT5
surpasses LCL by 0.2% and 0.9% in accuracy on the ISEAR and EmoInt datasets,
respectively. This margin of improvement, although seemingly nominal, is quite
significant in the context of these tasks. It underscores the efficacy of our
proposed method of infusing sensory perceptions into the model.

In summary, compared to previous studies, we have achieved superior results
without the necessity for additional data, marking the current pinnacle in
this field. This accomplishment underscores the effectiveness of SensoryT5.

### 4.5. Ablation studies

Figure 3: Ablation Study Results. Performance of T5 (None), Random SensoryT5
(with sensory values randomly assigned), and SensoryT5 across four datasets,
evaluated using accuracy as the metric. Figure 4: The heat values of the final
sensory layer in SensoryT5 and the encoder layer in T5 for the sentence ’I get
so mad when I see or hear about kids getting bullied…’ sourced from the
Empathetic Dialogues training dataset.

In our efforts to understand the contributions of different components within
the SensoryT5 model, we conducted ablation studies, a critical methodological
step in assessing the impact of our novel sensory integration. These studies
were also carried out on four datasets. The ablation tests were structured
around three primary configurations:

SensoryT5: Our complete model infusing sensory information.

Random SensoryT5: A variant of our model where the sensory values were
substituted with random numbers ranging from 0 to 5, maintaining the same
distribution of sensory scores but eliminating their meaningful association
with the data.

T5 (None): The baseline model without any sensory information, representing
the standard PLM approach in fine-grained emotion classification tasks.

The result is shown in Figure 3. While the SensoryT5 model exhibited the
highest performance in terms of accuracy across all datasets, the Random
SensoryT5 configuration yielded lower results than even the T5. This decrement
in performance was especially pronounced on the more complex datasets,
Empathetic Dialogues and GoEmotions.

The degradation in performance with random sensory values underscores the
importance of meaningful sensory integration. It is not merely the presence of
additional numerical data that enhances the SensoryT5 model’s performance, but
rather the contextually relevant and accurately associated sensory information
that it brings to the emotion classification task.

Furthermore, the fact that the Random SensoryT5 underperformed compared to the
T5 indicates that arbitrarily added sensory information could introduce noise
into the model, disrupting its ability to correctly interpret and classify
emotional content. This revelation is significant, affirming that the
strategic integration of sensory data is crucial, and haphazard integration
could be counterproductive.

In summary, these ablation studies have confirmed the value of our sensory
information layer, as evidenced by the performance drop when this layer is
randomized or removed. This reinforces our assertion that the SensoryT5’s
strength lies in its ability to simulate a more human-like understanding of
textual data, resonating with how humans perceive emotions through a sensory
lens.

### 4.6. Case study

We conducted a focused case study on the SensoryT5 model using a sentence from
the Empathetic Dialogues dataset: "I get so mad when I see or hear about kids
getting bullied…" In Figure 4, attention heatmaps display the model’s focus
during processing. The SensoryT5 heatmap shows the aggregate attention for
each token in the sensory layer, while the T5 section compiles attention
weights across all encoder layers, subsequently averaging them to reveal the
model’s overall focus. The SensoryT5 model exhibited intensified attention on
the emotionally significant phrase "so mad," highlighting its ability to
detect crucial emotional nuances. In contrast, the standard T5’s attention was
more distributed, less focused on the emotional pivot. This micro-level
analysis reveals SensoryT5’s superior capability in recognizing emotional
cues. Such insights substantiate the efficacy of integrating sensory awareness
into language models for improved emotional discernment.

In summary, our extensive evaluations and comparative studies highlight the
superior performance of SensoryT5 over other PLMs based emotion classification
models, including the T5. When benchmarked against the state-of-the-art
methods, SensoryT5 notably surpassed them, establishing a new standard in the
field. Further, our ablation studies convincingly demonstrate that the
effectiveness of SensoryT5 is attributed more to its integration of sensory
perception than to structural enhancements. This assertion is corroborated by
our detailed case studies, which offer a microscopic view into the instances
where SensoryT5’s unique capabilities are distinctly evident. Collectively,
these findings underscore a breakthrough performance of SensoryT5 in the realm
of fine-grained emotion classification. Importantly, it signifies a successful
adaptation within the shift towards incorporating neuro-cognitive data in NLP,
validating the premise that a deeper convergence between sensory data and
language modeling leads to a more profound understanding of emotional nuances.

## 5\. Conclusion

In this paper, we propose the SensoryT5 model designed for the fine-grained
emotion classification. This framework harnesses sensory knowledge, aiming to
boost the prowess of transformers in pinpointing nuanced emotional subtleties.
By integrating sensory knowledge into T5 through attention mechanisms, the
model concurrently evaluates sensory cues alongside contextual hallmarks.
Crucially, SensoryT5 exhibits exceptional adaptability and precision, making
it a formidable tool for tasks in Fine-grained Emotion Classification,
including configurations like 32-class, 27-class, 7-class, and 4-class
delineations. Moreover, SensoryT5 serves as a conduit between sensory
perception and emotional understanding, embodying the recent paradigm shift in
NLP towards a more neuro-cognitive approach. It acknowledges and capitalizes
on the intrinsic relationship between our sensory experiences and our
emotional responses, a connection well-documented in neuro-cognitive science
but often under-explored in computational fields. By interpreting sensory
lexicon through advanced representation learning, SensoryT5 decodes the
implicit emotional undertones conveyed, mirroring the human ability to
associate sensory experiences with specific emotional states. In recognizing
the entwined nature of cognition, sensation and emotive expression, SensoryT5
not only contributes to but also encourages the continuation of
interdisciplinary research efforts. It stands as testament to the potential of
a more nuanced and integrative approach in NLP, where understanding language
transcends the boundaries of words and grammar, delving into the very
experiences and perceptions that shape human emotionality.

## Limitations

In our work, we utilized GloVe and T5 embeddings to predict sensory values for
unknown words using a regression method. This approach learns only from static
values. To derive static T5 embeddings, we passed all tokens sequentially
through the T5 embedding layer, obtaining a static embedding for each token.
This process, however, leads to a limitation: it compromises the original
dynamic context-embedding capabilities of T5. In T5 embeddings, different
embeddings are obtained based on the different contexts. We intended to learn
from these transformer embeddings and then predict. Additionally, when
compared to current state-of-the-art models in emotion classification, such as
the label embedding-aware HypEmo and LCL, SensoryT5 exhibits certain
inadequacies, particularly in terms of interpretability. Both HypEmo and LCL
not only surpass SensoryT5 in explaining their decision-making processes but
also do so with fewer parameters. These models, by leveraging sophisticated
label-aware embedding strategies, provide insights into the nuanced
relationships and hierarchies among labels, something that SensoryT5, with its
reliance on static values, struggles to achieve. This gap highlights a
significant area for improvement in SensoryT5, suggesting the need for an
advanced approach that maintains the richness of context-sensitive embeddings
while enhancing the model’s overall interpretability and efficiency.

## 6\. Bibliographical References

##

  * Andrew and Gao (2007) Galen Andrew and Jianfeng Gao. 2007.  Scalable training of $L_{1}$-regularized log-linear models.  In _Proceedings of the 24th International Conference on Machine Learning_ , pages 33–40.
  * Bengio et al. (1994) Yoshua Bengio, Patrice Simard, and Paolo Frasconi. 1994.  Learning long-term dependencies with gradient descent is difficult.  _IEEE transactions on neural networks_ , 5(2):157–166.
  * Bhalla and Proffitt (1999) Mukul Bhalla and Dennis R Proffitt. 1999.  Visual–motor recalibration in geographical slant perception.  _Journal of experimental psychology: Human perception and performance_ , 25(4):1076.
  * Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020.  Language models are few-shot learners.  _Advances in neural information processing systems_ , 33:1877–1901.
  * Chen et al. (2023) Chih-Yao Chen, Tun-Min Hung, Yi-Li Hsu, and Lun-Wei Ku. 2023.  Label-aware hyperbolic embeddings for fine-grained emotion classification.  _arXiv preprint arXiv:2306.14822_.
  * Chen et al. (2019) I-Hsuan Chen, Qingqing Zhao, Yunfei Long, Qin Lu, and Chu-Ren Huang. 2019.  Mandarin chinese modality exclusivity norms.  _PloS one_ , 14(2):e0211336.
  * Chen et al. (2021) Xin Chen, Zhen Hai, Suge Wang, Deyu Li, Chao Wang, and Huanbo Luan. 2021.  Metaphor identification: A contextual inconsistency based neural sequence labeling approach.  _Neurocomputing_ , 428:268–279.
  * Chowdhery et al. (2022) Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. 2022.  Palm: Scaling language modeling with pathways.  _arXiv preprint arXiv:2204.02311_.
  * Christiano et al. (2017) Paul F Christiano, Jan Leike, Tom Brown, Miljan Martic, Shane Legg, and Dario Amodei. 2017.  Deep reinforcement learning from human preferences.  _Advances in neural information processing systems_ , 30.
  * Chung et al. (2022) Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, et al. 2022.  Scaling instruction-finetuned language models.  _arXiv preprint arXiv:2210.11416_.
  * Clavel and Callejas (2015) Chloe Clavel and Zoraida Callejas. 2015.  Sentiment analysis: from opinion mining to human-agent interaction.  _IEEE Transactions on affective computing_ , 7(1):74–93.
  * Collins et al. (2018) Edward Collins, Nikolai Rozanov, and Bingbing Zhang. 2018.  Evolutionary data measures: Understanding the difficulty of text classification tasks.  _arXiv preprint arXiv:1811.01910_.
  * Connell et al. (2018) Louise Connell, Dermot Lynott, and Briony Banks. 2018.  Interoception: the forgotten modality in perceptual grounding of abstract and concrete concepts.  _Philosophical Transactions of the Royal Society B: Biological Sciences_ , 373(1752):20170143.
  * Demszky et al. (2020a) Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020a.  GoEmotions: A dataset of fine-grained emotions.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 4040–4054, Online. Association for Computational Linguistics.
  * Demszky et al. (2020b) Dorottya Demszky, Dana Movshovitz-Attias, Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. 2020b.  Goemotions: A dataset of fine-grained emotions.  _arXiv preprint arXiv:2005.00547_.
  * Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018.  Bert: Pre-training of deep bidirectional transformers for language understanding.  _arXiv preprint arXiv:1810.04805_.
  * Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019.  BERT: Pre-training of deep bidirectional transformers for language understanding.  In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_ , pages 4171–4186, Minneapolis, Minnesota. Association for Computational Linguistics.
  * Dong et al. (2013) Ruihai Dong, Michael P O’Mahony, Markus Schaal, Kevin McCarthy, and Barry Smyth. 2013.  Sentimental product recommendation.  In _Proceedings of the 7th ACM Conference on Recommender Systems_ , pages 411–414.
  * Ekman (1992) Paul Ekman. 1992.  An argument for basic emotions.  _Cognition & emotion_, 6(3-4):169–200.
  * Ekman et al. (1999) Paul Ekman et al. 1999.  Basic emotions.  _Handbook of cognition and emotion_ , 98(45-60):16.
  * Fainsilber and Ortony (1987) Lynn Fainsilber and Andrew Ortony. 1987.  Metaphorical uses of language in the expression of emotions.  _Metaphor and Symbol_ , 2(4):239–250.
  * Gusfield (1997) Dan Gusfield. 1997.  _Algorithms on Strings, Trees and Sequences_.  Cambridge University Press, Cambridge, UK.
  * Hochreiter and Schmidhuber (1997) Sepp Hochreiter and Jürgen Schmidhuber. 1997.  Long short-term memory.  _Neural computation_ , 9(8):1735–1780.
  * Huang et al. (2020) Minghui Huang, Haoran Xie, Yanghui Rao, Yuwei Liu, Leonard KM Poon, and Fu Lee Wang. 2020.  Lexicon-based sentiment convolutional neural networks for online review analysis.  _IEEE Transactions on Affective Computing_ , 13(3):1337–1348.
  * Irsoy and Cardie (2014) Ozan Irsoy and Claire Cardie. 2014.  Opinion mining with deep recurrent neural networks.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 720–728.
  * Jiang et al. (2022) Xiaotong Jiang, Qingqing Zhao, Yunfei Long, and Zhongqing Wang. 2022.  Chinese synesthesia detection: New dataset and models.  In _Findings of the Association for Computational Linguistics: ACL 2022_ , pages 3877–3887.
  * Juslin and Västfjäll (2008) Patrik N Juslin and Daniel Västfjäll. 2008.  Emotional responses to music: The need to consider underlying mechanisms.  _Behavioral and brain sciences_ , 31(5):559–575.
  * Ke et al. (2019) Pei Ke, Haozhe Ji, Siyang Liu, Xiaoyan Zhu, and Minlie Huang. 2019.  Sentilare: Sentiment-aware language representation learning with linguistic knowledge.  _arXiv preprint arXiv:1911.02493_.
  * Khare et al. (2023) Smith K Khare, Victoria Blanes-Vidal, Esmaeil S Nadimi, and U Rajendra Acharya. 2023.  Emotion recognition and artificial intelligence: A systematic review (2014–2023) and research recommendations.  _Information Fusion_ , page 102019.
  * Kövecses (2019) Zoltán Kövecses. 2019.  Perception and metaphor.  _Perception metaphors_ , 19(327):10–1075.
  * Lakoff and Johnson (1980) George Lakoff and Mark Johnson. 1980.  Metaphors we live by.  _University of Chicago, Chicago, IL_.
  * Lee (2018) Sophia Yat Mei Lee. 2018.  Figurative language in emotion expressions.  In _Chinese Lexical Semantics: 18th Workshop, CLSW 2017, Leshan, China, May 18–20, 2017, Revised Selected Papers 18_ , pages 408–419. Springer.
  * Li et al. (2017) Minglei Li, Qin Lu, Yunfei Long, and Lin Gui. 2017.  Inferring affective meanings of words from word embedding.  _IEEE Transactions on Affective Computing_ , 8(4):443–456.
  * Li et al. (2021) Zhengyan Li, Yicheng Zou, Chong Zhang, Qi Zhang, and Zhongyu Wei. 2021.  Learning implicit sentiment in aspect-based sentiment analysis with supervised contrastive pre-training.  _arXiv preprint arXiv:2111.02194_.
  * Lin et al. (2022) Zijie Lin, Bin Liang, Yunfei Long, Yixue Dang, Min Yang, Min Zhang, and Ruifeng Xu. 2022.  Modeling intra- and inter-modal relations: Hierarchical graph contrastive learning for multimodal sentiment analysis.  In _Proceedings of the 29th International Conference on Computational Linguistics_ , pages 7124–7135, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.  _arXiv preprint arXiv:1907.11692_.
  * Long et al. (2019a) Yunfei Long, Rong Xiang, Qin Lu, Chu-Ren Huang, and Minglei Li. 2019a.  Improving attention model based on cognition grounded data for sentiment analysis.  _IEEE transactions on affective computing_ , 12(4):900–912.
  * Long et al. (2019b) Yunfei Long et al. 2019b.  A study on using personal profiles for a biased reader emotion prediction model.
  * Lu et al. (2023) Qiang Lu, Xia Sun, Yunfei Long, Zhizezhang Gao, Jun Feng, and Tao Sun. 2023.  Sentiment analysis: Comprehensive reviews, recent advances, and open challenges.  _IEEE Transactions on Neural Networks and Learning Systems_.
  * Lynott et al. (2020) Dermot Lynott, Louise Connell, Marc Brysbaert, James Brand, and James Carney. 2020.  The lancaster sensorimotor norms: multidimensional measures of perceptual and action strength for 40,000 english words.  _Behavior Research Methods_ , 52:1271–1291.
  * Maas et al. (2011) Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011.  Learning word vectors for sentiment analysis.  In _Proceedings of the 49th annual meeting of the association for computational linguistics: Human language technologies_ , pages 142–150.
  * Meškelė and Frasincar (2020) Donatas Meškelė and Flavius Frasincar. 2020.  Aldonar: A hybrid solution for sentence-level aspect-based sentiment analysis using a lexicalized domain ontology and a regularized neural attention model.  _Information Processing & Management_, 57(3):102211.
  * Mohammad and Bravo-Marquez (2017) Saif Mohammad and Felipe Bravo-Marquez. 2017.  WASSA-2017 shared task on emotion intensity.  In _Proceedings of the 8th Workshop on Computational Approaches to Subjectivity, Sentiment and Social Media Analysis_ , pages 34–49, Copenhagen, Denmark. Association for Computational Linguistics.
  * Müller et al. (2021) Nadine Müller, Arne Nagels, and Christina Kauschke. 2021.  Metaphorical expressions originating from human senses: Psycholinguistic and affective norms for german metaphors for internal state terms (mist database).  _Behavior Research Methods_ , pages 1–13.
  * Nandwani and Verma (2021) Pansy Nandwani and Rupali Verma. 2021.  A review on sentiment analysis and emotion detection from text.  _Social Network Analysis and Mining_ , 11(1):81.
  * Niedenthal and Wood (2019) Paula M Niedenthal and Adrienne Wood. 2019.  Does emotion influence visual perception? depends on how you look at it.  _Cognition and Emotion_ , 33(1):77–84.
  * OpenAI (2023) OpenAI. 2023.  Gpt-4 technical report.
  * Ouyang et al. (2022) Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022.  Training language models to follow instructions with human feedback.  _Advances in Neural Information Processing Systems_ , 35:27730–27744.
  * Pang et al. (2008) Bo Pang, Lillian Lee, et al. 2008.  Opinion mining and sentiment analysis.  _Foundations and Trends® in information retrieval_ , 2(1–2):1–135.
  * Pennington et al. (2014a) Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014a.  GloVe: Global vectors for word representation.  In _Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 1532–1543, Doha, Qatar. Association for Computational Linguistics.
  * Pennington et al. (2014b) Jeffrey Pennington, Richard Socher, and Christopher D Manning. 2014b.  Glove: Global vectors for word representation.  In _Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP)_ , pages 1532–1543.
  * Picard (2000) Rosalind W Picard. 2000.  _Affective computing_.  MIT press.
  * Proffitt et al. (1995) Dennis R Proffitt, Mukul Bhalla, Rich Gossweiler, and Jonathan Midgett. 1995.  Perceiving geographical slant.  _Psychonomic bulletin & review_, 2:409–428.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _The Journal of Machine Learning Research_ , 21(1):5485–5551.
  * Rashkin et al. (2018) Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2018.  Towards empathetic open-domain conversation models: A new benchmark and dataset.  _arXiv preprint arXiv:1811.00207_.
  * Rashkin et al. (2019) Hannah Rashkin, Eric Michael Smith, Margaret Li, and Y-Lan Boureau. 2019.  Towards empathetic open-domain conversation models: A new benchmark and dataset.  In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_ , pages 5370–5381, Florence, Italy. Association for Computational Linguistics.
  * Riener et al. (2011) Cedar R Riener, Jeanine K Stefanucci, Dennis R Proffitt, and Gerald Clore. 2011.  An effect of mood on the perception of geographical slant.  _Cognition and Emotion_ , 25(1):174–182.
  * Scherer and Wallbott (1994) Klaus R Scherer and Harald G Wallbott. 1994.  Evidence for universality and cultural variation of differential emotion response patterning.  _Journal of personality and social psychology_ , 66(2):310.
  * Šimić et al. (2021) Goran Šimić, Mladenka Tkalčić, Vana Vukić, Damir Mulc, Ena Španić, Marina Šagud, Francisco E Olucha-Bordonau, Mario Vukšić, and Patrick R. Hof. 2021.  Understanding emotions: Origins and roles of the amygdala.  _Biomolecules_ , 11(6):823.
  * Socher et al. (2011) Richard Socher, Jeffrey Pennington, Eric H Huang, Andrew Y Ng, and Christopher D Manning. 2011.  Semi-supervised recursive autoencoders for predicting sentiment distributions.  In _Proceedings of the 2011 conference on empirical methods in natural language processing_ , pages 151–161.
  * Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013.  Recursive deep models for semantic compositionality over a sentiment treebank.  In _Proceedings of the 2013 conference on empirical methods in natural language processing_ , pages 1631–1642.
  * Suresh and Ong (2021a) Varsha Suresh and Desmond Ong. 2021a.  Not all negatives are equal: Label-aware contrastive loss for fine-grained text classification.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 4381–4394, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Suresh and Ong (2021b) Varsha Suresh and Desmond C Ong. 2021b.  Not all negatives are equal: Label-aware contrastive loss for fine-grained text classification.  _arXiv preprint arXiv:2109.05427_.
  * Tang et al. (2015) Duyu Tang, Bing Qin, and Ting Liu. 2015.  Document modeling with gated recurrent neural network for sentiment classification.  In _Proceedings of the 2015 conference on empirical methods in natural language processing_ , pages 1422–1432.
  * Tian et al. (2020) Hao Tian, Can Gao, Xinyan Xiao, Hao Liu, Bolei He, Hua Wu, Haifeng Wang, and Feng Wu. 2020.  Skep: Sentiment knowledge enhanced pre-training for sentiment analysis.  _arXiv preprint arXiv:2005.05635_.
  * Touvron et al. (2023) Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023.  Llama: Open and efficient foundation language models.  _arXiv preprint arXiv:2302.13971_.
  * Wan et al. (2023) Mingyu Wan, Qi Su, Kathleen Ahrens, and Chu-Ren Huang. 2023.  Perceptional and actional enrichment for metaphor detection with sensorimotor norms.  _Natural Language Engineering_ , pages 1–29.
  * Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021.  Finetuned language models are zero-shot learners.  _arXiv preprint arXiv:2109.01652_.
  * Xiang et al. (2021) Rong Xiang, Jing Li, Mingyu Wan, Jinghang Gu, Qin Lu, Wenjie Li, and Chu-Ren Huang. 2021.  Affective awareness in neural sentiment analysis.  _Knowledge-Based Systems_ , 226:107137.
  * Xu et al. (2019) Hu Xu, Bing Liu, Lei Shu, and Philip S Yu. 2019.  Bert post-training for review reading comprehension and aspect-based sentiment analysis.  _arXiv preprint arXiv:1904.02232_.
  * Yamamoto (2008) Takashi Yamamoto. 2008.  Central mechanisms of taste: Cognition, emotion and taste-elicited behaviors.  _Japanese Dental Science Review_ , 44(2):91–99.
  * Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019.  Xlnet: Generalized autoregressive pretraining for language understanding.  _Advances in neural information processing systems_ , 32.
  * Yarkareddy et al. (2022) Sruthi Yarkareddy, T Sasikala, and S Santhanalakshmi. 2022.  Sentiment analysis of amazon fine food reviews.  In _2022 4th International Conference on Smart Systems and Inventive Technology (ICSSIT)_ , pages 1242–1247. IEEE.
  * Yin et al. (2020) Da Yin, Tao Meng, and Kai-Wei Chang. 2020.  Sentibert: A transferable transformer-based architecture for compositional sentiment semantics.  _arXiv preprint arXiv:2005.04114_.
  * Zadra and Clore (2011) Jonathan R Zadra and Gerald L Clore. 2011.  Emotion and perception: The role of affective information.  _Wiley interdisciplinary reviews: cognitive science_ , 2(6):676–685.
  * Zeiler and Fergus (2014) Matthew D Zeiler and Rob Fergus. 2014.  Visualizing and understanding convolutional networks.  In _Computer Vision–ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part I 13_ , pages 818–833. Springer.
  * Zhang et al. (2023) Wenxuan Zhang, Yue Deng, Bing Liu, Sinno Jialin Pan, and Lidong Bing. 2023.  Sentiment analysis in the era of large language models: A reality check.  _arXiv preprint arXiv:2305.15005_.
  * Zhao et al. (2017) Bo Zhao, Jiashi Feng, Xiao Wu, and Shuicheng Yan. 2017.  A survey on deep learning-based fine-grained object classification and semantic segmentation.  _International Journal of Automation and Computing_ , 14(2):119–135.