colorlinks, linkcolor=red!50!black, citecolor=blue!50!black,
urlcolor=blue!80!black

mAP

    Mean Average Precision
HOG

    Histogram of Oriented Gradients
Saghog

     Self-Supervised Autoencoder for Generating HOG features
ViT

    Vision Transformer
CNN

    Convolutional Neural Network
MAE

    Masked Autoencoder
VLAD

    Vector of Locally Aggregated Descriptors
ESVM

    Exemplar SVM
SGR

    Similarity Graph Reranking
WR

    Writer Retrieval
SAM

    Segment Anything Model
11institutetext: Computer Vision Lab, TU Wien
11email: {mpeer, kleber<EMAIL_ADDRESS>
Code: http://github.com/marco-peer/icdar24

# SAGHOG: Self-Supervised Autoencoder for Generating HOG Features for Writer
Retrieval

Marco Peer  0000-0001-6843-0830 Florian Kleber  0000-0001-8351-5066 Robert
Sablatnig  0000-0003-4195-1593

###### Abstract

This paper introduces Saghog, a self-supervised pretraining strategy for
writer retrieval using HOG features of the binarized input image. Our
preprocessing involves the application of the Segment Anything technique to
extract handwriting from various datasets, ending up with about 24k documents,
followed by training a vision transformer on reconstructing masked patches of
the handwriting. Saghog is then finetuned by appending NetRVLAD as an encoding
layer to the pretrained encoder. Evaluation of our approach on three
historical datasets, Historical-WI, HisFrag20, and GRK-Papyri, demonstrates
the effectiveness of Saghog for writer retrieval. Additionally, we provide
ablation studies on our architecture and evaluate un- and supervised
finetuning. Notably, on HisFrag20, Saghog outperforms related work with a mAP
of 57.2 % - a margin of 11.6 % to the current state of the art, showcasing its
robustness on challenging data, and is competitive on even small datasets,
e.g. GRK-Papyri, where we achieve a Top-1 accuracy of 58.0 %.

###### Keywords:

Writer Retrieval Self-Supervised Learning Masked Autoencoder Document
Analysis.

##  1 Introduction

Writer Retrieval (WR) is the task of locating documents written by the same
author within a dataset, achieved by identifying similarities in handwriting
[18]. This task is particularly valuable for historians and paleographers,
allowing them to track individuals or social groups across various historical
periods [9]. Additionally, WR is used for recognizing documents with unknown
authors and uncovering similarities within such documents [6].

WR methods usually consist of multiple steps, such as sampling of handwriting
by applying interest point detection, a neural network for feature extraction,
and feature encoding such as NetVLAD, followed by aggregation of the encoded
features [29]. Although those methods work for large datasets such as
Historical-WI [14], WR still lacks performance for datasets that contain less
handwriting or noise such as degradation, best exemplified by HisFrag20 [32].
While approaches trained on full fragments currently work best [26],
experiments show that those methods do infer features from the background, not
necessarily related on the actual handwriting [30].

With the recent gain of interest in self-supervised algorithms that make use
of large amounts of unlabeled data, this paper introduces Self-Supervised
Autoencoder for Generating HOG features (Saghog), a self-supervised
pretraining strategy based on Histogram of Oriented Gradients (HOG) features.
Saghog is two-staged and relies on the Vision Transformer (ViT) architecture,
as shown in Fig. 1: Firstly, we aim to close that gap of performance for
complex datasets by self-supervised pretraining on reconstructing only HOG
features of the handwriting. Since related work relies on clustering SIFT
descriptors [6, 29], predicting HOG features is an intuitive choice. Secondly,
we add NetRVLAD [29] to the pretrained encoder and finetune the model on the
respective dataset. We apply and evaluate two finetuning strategies:
supervised - using the writer label - or unsupervised finetuning - using the
cluster of the corresponding SIFT descriptor of the input patch as a surrogate
class (Cl-S) [6]. Additionally, as a preprocessing step, we propose to apply
Segment Anything Model (SAM) [20], a foundation model for segmentation, to
extract the handwriting of the data, accelerate the training step, and improve
the generalization ability of Saghog.

Figure 1: Overview of Saghog. The decoder reconstructs the HOG features of the
masked tokens (  ) by only using the non-masked tokens (  ). In the second
stage, the pretrained encoder with NetRVLAD is finetuned, either by training
on writer or pseudo labels.

We evaluate our results on three datasets: Historical-WI [14], HisFrag20 [32]
and GRK-Papyri [25], each corresponding to a different domain. We provide
ablation studies on our preprocessing stage, different target features of
Saghog and hyperparameters of the pretraining stage and the network
architecture. Our experiments show that HOG reconstruction during pretraining
does improve WR when finetuning with NetRVLAD, in particular on HisFrag20,
where we outperform the state of the art by a large margin (11.6 %).
Furthermore, we find that only optimizing NetRVLAD is able to outperform
finetuning the network, showing that Saghog is beneficial for WR. On
Historical-WI, the effect of pretraining is limited, but using Saghog does not
harm the performance and is competitive with other approaches. In the end, we
show that even on small, out-of-domain datasets such as GRK-Papyri, Saghog
shines, in particular in terms of Top-1 accuracy, making ViTs valuable for
further research in the domain of WR.

Concluding, our contributions are summarized to:

  * •

We collect multiple datasets and propose a data curation scheme to enable
pretraining of Saghog.

  * •

Saghog is the first self-supervised masking-based approach for handwriting of
$32\times 32$ patches without the need for any carefully designed data
augmentation pipeline. We investigate two different finetuning schemes -
supervised and unsupervised.

  * •

We thorougly evaluate our approach on Historical-WI [14], HisFrag20 [32] and
GRK-Papyri [25] and outperform state-of-the-art methods on WR as a downstream
task.

Our code is publicly available. The remainder of our paper is structured as
follows. Section 2 gives details on related work in the field of self-
supervised learning and WR. Afterwards, we describe Saghog in Section 3,
including pretraining and finetuning, followed by our evaluation protocol in
Section 4 and our results in Section 5. We summarize our findings in Section
6.

##  2 Related Work

In this section, we briefly discuss related work. We start with the
developments in self-supervised learning with a focus on WR, followed by the
current state-of-the-art WR methods.

#### Self-Supervised Learning

In recent years, self-supervised learning has gained popularity to learn
meaningful representations from unlabeled data. While metric learning
approaches like SimCLR [4] or MoCo [16] involve training two separate networks
to generate similar embeddings for a sample, they are outperformed by self-
distillation methods, e.g. DINO [1], that use a carefully designed
augmentation pipeline to generate different views of an image. In the domain
of WR, we introduced a self-supervised algorithm based on DINO [1] using
morphological operations to generate augmented views of the binarized
Historical-WI dataset [28]. Lastilla et al. [22] apply an online bag-of-words
pretraining to documents of the Vatican Apostolic Library for WR. Both
approaches improve the Mean Average Precision (mAP) on the datasets evaluated,
but they use large crops which slows down training. Additionally, they do not
use any encoding or finetuning at all. However, those augmentation pipelines
suffer when training on small patches only containing a few strokes of
handwriting. Therefore, we focus on another approach of self-supervision
called masked image modeling or Masked Autoencoder (MAE), introduced by He et
al. [15], where a network learns to reconstruct masked parts of an image. An
intuitive adaption of MAE is MaskFeat [36], proposed by Wei et al., where the
target features are handcrafted descriptors rather than pixel values, e.g.
HOG. Saghog also applies this strategy by adapting MaskFeat to the binarized
version of the input image. This is an intuitive choice, since dominating
approaches for WR rely on SIFT descriptors [6, 29] for training, and those
descriptors are formed by aggregating HOG features.

#### Writer Retrieval

WR methods fall into two categories: codebook-based and codebook-free
approaches. Codebooks serve as models to compute handwriting statistics, with
Vector of Locally Aggregated Descriptors (VLAD) standing out as a prominent
method for WR [5, 6, 7, 8]. Handwriting characteristics are extracted either
through handcrafted features [5, 21] or CNNs [6, 7, 8]. For historical
datasets in particular, Christlein et al. [6] demonstrate that training on
$32\times 32$ patches with pseudolabels generated by clustering SIFT
descriptors outperforms supervised methods [35]. Additionally, Exemplar SVM
(ESVM) refines each descriptor’s encoding by training multiple SVMs with only
one positive sample and the remaining ones as negatives. The HisIR19
competition winners [9] and the current state-of-the-art on Historical-WI
dataset utilize handcrafted features (SIFT and pathlet) for retrieval,
encoding them via bagged VLAD (bVLAD) [21]. In our previous work [29] we
employ NetRVLAD, a NetVLAD-based layer with reduced complexity, to train the
neural network in and end-to-end way. Additionally, we introduced Similarity
Graph Reranking (SGR), a reranking strategy that shows superior performance
compared with state of the art. In contrast to training with small patches, on
more complex datasets including fragments, such as HisFrag20 [32], leading
approach train on full images: Ngo et al. [26] propose an attention-based VLAD
called A-VLAD, while our feature mixer [30] uses an codebook-free encoding
stage based on convolutional and fully connected layers. In this work,
finetuning of Saghog is closely related to our training protocol of NetRVLAD
[29], and we show that our approach is outperforming methods on image-level
for datasets such as HisFrag20. Saghog is also based on the ViT architecture
instead of relying on CNNs.

##  3 Methodology

In this section, we cover each part of our proposed approach. We start with
the curation of the dataset for pretraining, followed by describing Saghog. In
the end, NetRVLAD finetuning and feature aggregation are explained.

###  3.1 Data Curation

(a) Handwriting extraction via SAM

(b) Examples of input-output pairs

(c) Patch sampling

Figure 2: Preprocessing scheme. (2(a)) We obtain the handwriting of a page by
processing the segmentation of SAM. (2(b)) Examples of the produced images.
(2(c)) $32\times 32$ patches are extracted by applying SIFT on the binarized
image.

Self-supervised learning relies on a large amount of (unlabelled) data. Hence,
we employ multiple datasets proposed in previous competitions in the domain of
historical document analysis for our pretraining. For the datasets used,
presented in Table 1, we are mainly inspired by the choice of Zenk et al.
[37]. Our preprocessing scheme, also shown in Fig. 2, is two-fold:
_handwriting extraction_ and _patch sampling_.

Table 1: Datasets used for self-supervised pretraining after preprocessing. Dataset | Subset | Images
---|---|---
CLaMM 2016 [11] | Task 1 / Task 2 | 2607
Historical-WI [14] | Train (color) | 1182
CLaMM 2017 [10] | Task 1 / Task 3 | 1572
cBAD 2019 [12] | Validation / Test | 531
HisIR19 [9] | Validation / Test | 1152 / 17641
Total |  | 24685

_Handwriting extraction_ Since datasets such as HisIR19 [9] contain additional
elements like color palettes, book covers, and background noise, we use SAM
[20], a foundation model for segmentation. We use the default pretrained model
and process the data as follows:

  1. 1.

We apply a Canny edge detector on the image.

  2. 2.

Since SAM relies on a grid of points and we only want to obtain the
handwriting part that is usually a significant part of the page, we use a
small $8\times 8$ grid for segmentation.

  3. 3.

To discard wrong results, we filter masks with less than 10 % edge pixels and
a confidence level lower than 0.8.

If the preceding steps generate multiple masks, we merge them, restricting the
maximum number of masks per page to two. Additionally, to address pages with
minimal handwriting or none at all, we discard pages with fewer than 1000 SIFT
keypoints detected. This procedure allows us to take smaller random crops of
the pages to extract keypoints, which significantly accelerates the processing
time during training and removes non-handwriting areas. As depicted in Fig.
2(b), we occasionally oversegment and remove some handwriting. Note that we do
not apply SAM on the trainset of Historical-WI [14] since it contains pure
handwriting. To show the effectiveness of the preprocessing, we annotate
handwriting areas of 200 pages of the HisIR19 [9] test set and evaluate the
bounding boxes. We achieve an<EMAIL_ADDRESS>of 74 %. This is mostly due to SAM
splitting the handwriting areas in multiple zones (see Fig. 2(b)). However,
all of our detected masks are inside the annotated handwriting areas,
indicating that we can successfully segment the handwriting - although we
remove parts of it.

_Patch sampling_ During training, we take a random crop of size 256 and sample
small parts of the handwriting by extracting $32\times 32$ patches located
around SIFT keypoints. The keypoints are detected on the binarized version of
the page via SIFT, and we discard patches with less than 1 % black pixels,
similar to previous work [6, 29]. We investigate the influence of the
binarization in our evaluation in Section 5.

The composition of our final dataset used for pretraining is shown in Table 1.
Although the majority of images are included in the HisIR19 [9] dataset, it
contains various document types such as manuscript books, letters, or
charters.

###  3.2 Self-Supervised Pretraining

Saghog is based on the MAE architecture [15] consisting of an encoder-decoder
structure. We use the vanilla ViT [13] structure with a patch size of four.
Firstly, random tokens are masked out - by default, we mask 75 % of the image.
The encoder projects each unmasked token of the $32\times 32$ input color
image to an embedding of size 512. Those embeddings are then projected by the
decoder to the output feature space: While MAE [15] and MaskFeat [36] use RGB
pixel values or HOG features of the colorized input, we propose to reconstruct
the HOG features of the binarized image due to two reasons. By relying on
binarization, the network automatically learns to focus on handwriting, and on
the other hand, we are able to suppress the influence of the background. HOG
features are calculated by applying a convolutional layer with fixed weights
to obtain the gradients. Afterward, we build a 9-bin histogram on cells of
size $4\times 4$, resulting in a feature vector of dimension 9 for each token.
The final HOG features are $l_{2}$ normalized.

(a)

(b)

(c)

(d)

Figure 3: Examples of reconstructed HOG features. For better visualization, we
stack multiple patches to create images of (128, 512). The images are not seen
during pretraining. (3(a)) Raw input image. (3(b)) Masked input with a ratio
of 0.75. (3(c)) Prediction of the missing HOG features of the binarized image.
(3(d)) Actual HOG features.

We choose, similar to related work [36], a larger encoder consisting of 8
layers (depth) and a smaller decoder (one layer) since this shows improved
performance on the downstream task. We denote the encoder as ViT-4/8.
Qualitative examples of the training process are depicted in Fig. 3. In
general, the network is able to reconstruct the missing tokens (and therefore
the style of handwriting as well as parts of letters), even if only small
parts of the handwriting are visible. We evaluate the choice of target
features as well as the hyperparameters of Saghog in our evaluation.

###  3.3 Finetuning

After pretraining of Saghog, we drop the decoder and only use the encoder for
further finetuning. We use the class token as output of the encoder, a
learnable embedding that describes the global information of the token
sequence. We also studied pooling of all patch embeddings instead of using the
class token, but this did not yield significant differences. Finetuning is
then performed by appending the NetRVLAD layer [30] with 100 cluster centers
and a final $l_{2}$-normalization of the encoded embedding. In our evaluation,
we investigate two finetuning schemes commonly used in the domain of WR:

  1. 1.

_Supervised_ : The target label is the writer of the document, this refers to
the default supervised finetuning strategy.

  2. 2.

_Cl-S_ : The target label is the cluster membership of the clustered SIFT
descriptors extracted at the corresponding keypoints. We follow the protocol
used in [29]: We cluster the trainset by using k-Means with 5000 centers and
only consider patches that pass the ratio test between the distance to the
nearest and the second nearest cluster ($<0.9$). This has shown to perform
better than supervised training for WR [6, 29].

###  3.4 Writer Retrieval

We aggregate all patch embeddings $\boldsymbol{\mathrm{x}}_{i}$ with
$i=0,\dots,n_{p}-1$ of a page using $l_{2}$ normalization followed by sum
pooling
$\boldsymbol{\mathrm{X}}=\sum_{i=0}^{n_{p}-1}\boldsymbol{\mathrm{x}}_{i}$ to
obtain the global page descriptor $\boldsymbol{\mathrm{X}}$. Furthermore, to
reduce visual burstiness [17], we apply power-normalization
$f(x)=\text{sign}(x)|x|^{\alpha}$ with $\alpha=0.4$, followed by
$l_{2}$-normalization. Finally, a dimensionality reduction with whitening via
PCA is performed.

WR is then evaluated by a _leave-one-out strategy_ : Each image of the test
set is once used as a query $q$, and the remaining documents are ranked
according to the cosine similarity of the global page descriptors.

##  4 Evaluation

We provide details on the datasets used for evaluation and our experimental
setup in this section.

###  4.1 Datasets

(a) Historical-WI

(b) HisFrag20

(c) GRK-Papyri

Figure 4: Examples of the datasets used. (4(a)) Binarized example of
Historical-WI testset. (4(b)) HisFrag20: Sample of train- and testset. (4(c))
GRK-Papyri (Dioscorus-5) color and binarized version [8] \- we only use the
binarized set.

We evaluate our approach on three datasets, as shown in Fig. 4, each
representing a different scenario:

#### Historical-WI

Introduced by Fiel et al. [14], the dataset consists of 3600 images and 760
writers, each contributing five pages. We use the binarized version for
evaluation, following related work. The training set includes 1192 pages. We
consider Historical-WI as a dataset of moderate size and since it is binarized
without containing any additional noise, handcrafted features [21] as well as
small networks such as ResNet20 [6, 29] work reasonably well.

#### HisFrag20

The main dataset used in our work is the HisFrag20 [32], introduced via the
_ICFHR 2020 Competition on Image Retrieval for Historical Handwritten
Fragments_. The trainset consists of about 100k fragments; the respective test
set includes 20019 fragments from 1152 writers and 2732 documents. The
generation algorithm for the fragments differs between the train- and test
set, making the dataset challenging, which results in a lack of performance
compared to Historical-WI.

#### GRK-Papyri

Additionally, we provide baseline results for comparison on GRK-Papyri [25], a
small dataset consisting of only 50 documents of 10 different writers. Note
that this dataset represents a heavy domain shift compared to the datasets
used for pretraining - from Latin to ancient Greek, as well as high
degradation, e.g., holes, stains, and low contrast. To compare to related
work, we use the U-Net binarized version provided by Christlein et al. [8].
Even though binarization tackles most of the degradation, binarized images
still suffer from noise and missing strokes.

###  4.2 Experimental Setup

In the following, the setups for both training stages as well as for the
aggregation step are given.

#### Pretraining

We use the vanilla ViT architecture with a patch size of 4 and a depth of 8,
denoted as ViT-4/8. The cell size for calculating HOG features is $4\times 4$,
and the decoder depth is 1, as related work [15, 36] shows negligible
influence of those parameters. During training, we take a random crop of size
$256\times 256$ and apply grayscale and binarization (both with a probability
of $p=0.2$). One batch consists of 64 pages, each contributing 32 patches,
yielding a batch size of 2048. Further hyperparameters are given in Table
2(a).

Table 2: Hyperparameters of our training, if not stated otherwise.

(a) Pretraining Config | ViT-4/8
---|---
Optimizer |  AdamW [24]
Weight decay | 0.05
Learning rate | $8\cdot 10^{-4}$
Scheduler |  Cosine decay [23]
Batch size | 2048
Gradient clipping | 0.02
Warmup epochs | 5
Epochs | 200

(b) Finetuning Config | ViT-4/8 | ResNet
---|---|---
Optimizer |  AdamW [24] |  Adam [19]
Weight decay | 0.01 | 0
Learning rate | $10^{-3}$
Scheduler |  Cosine decay [23]
Batch size | 1024
Gradient clipping | 1.0
Warmup epochs | 5
Epochs | 50

#### Finetuning

We finetune all networks by appending NetRVLAD [29] with 100 cluster centers
to Saghog. The training is done for 50 epochs on triplets with Multi-
Similarity loss [34] and a batch size of 1024 with 16 samples per class. For
data augmentation, we apply erosion and dilation with a randomly sampled
$3\times 3$ kernel and a probability of 0.1. Additionally, token dropout is
applied ($p=0.1$). We use an early stopping of five epochs if the mAP on the
validation set does not increase. All of our networks reach convergence with
these settings (10 % of the writers are used as validation). Further
hyperparameters are given in Table 2(b).

#### Retrieval

If not stated otherwise, we report the mAP on the task of WR for the
respective datasets when Saghog is jointly finetuned with NetRVLAD in an
unsupervised manner (Cl-S). The Top-1 metric indicates if the first document
of the ranked list is a hit. The retrieval is performed on PCA-whitened global
descriptors of dimension 512.

##  5 Results

In this section, we evaluate different parts of our pipeline and study the
gain of performance by (un-)supervised finetuning of Saghog.

###  5.1 Ablation studies

#### Preprocessing

Firstly, in Table 3, we study the influence of preprocessing via SAM. For both
datasets, Saghog performs better when we preprocess data for pretraining with
SAM, in particular on the HisFrag20 dataset, where we report a gain of 4.1 %
mAP. Note that pretraining with no preprocessing includes 27k images instead
of 24k due to the filtering applied after using SAM.

Table 3: SAM preprocessing. | Historical-WI | HisFrag20
---|---|---
From scratch | 71.1 | 47.4
Full image | 72.8 | 50.5
With SAM | 73.3 | 54.6

#### Target features

Next, we evaluate the choice of the target features of the decoder. We train
four networks with different features, as shown in Table 4: _Pixel_ \- which
refers to the default MAE proposed by He et al. [15], and three different
settings which refer to the modality used for calculating HOG features: _RGB_
, _Gray_ , _Binarized_ (Bin.). While for both datasets, the largest gain of
performance is reported for switching from pixel values to HOG features,
Saghog works best when using HOG features of the binarized image as target on
HisFrag20. All of our experiments outperform training from scratch, showing
the effectiveness of pretraining.

Table 4: Decoder target features. Target feature | Historical-WI | HisFrag20
---|---|---
From scratch | 71.1 | 47.4
Pixel (RGB) | 71.9 | 50.6
HOG (RGB) | 73.3 | 53.1
HOG (Gray) | 73.3 | 54.1
HOG (Bin.) | 73.3 | 54.6
Table 5: Ablation studies of Saghog on Historical-WI (HW) and HisFrag20 (HF)
with finetuned NetRVLAD. Default settings are marked in gray.

(a) Masking ratio | mAP
---|---
Ratio | HW | HF
45 | 72.2 | 54.4
60 | 72.1 | 53.6
75 | 73.3 | 54.6
90 | 72.1 | 44.6

(b) Encoder dimension | mAP
---|---
Dim. | HW | HF
128 | 71.3 | 47.6
256 | 72.0 | 51.7
384 | 72.5 | 49.1
512 | 73.3 | 54.6

(c) Binarization algorithm | mAP
---|---
Method | HW | HF
Otsu [27] | 73.0 | 54.1
Sauvola [31] | 73.3 | 54.6
Su [33] | 72.5 | 53.6

We provide results for the critical parts of our pipeline in Table 5 when
finetuning on Historical-WI and HisFrag20: the ratio of the masked patches,
indicating the difficulty of the reconstruction task, the dimension of the
encoder and the binarization algorithm used for patch sampling and calculating
HOG features. We summarize our findings in the following:

Masking ratio Saghog performs best when using a _masking ratio_ of 75 %. This
is higher than values reported in related work ([15]: 60 %, [36]: 40 %). We
argue that this is mostly related to the fact that the background is not
considered for the reconstruction task since we are only focusing on the HOG
features of the handwriting. We observe a drop in performance when setting the
masking ratio too high (90 %).
Embedding dimension The performance increases with higher _encoder
dimensions_. The highest performance is obtained for 512 with a mAP of 73.3 %
and 54.6 %. Note that the encoder dimension increases the network size, we
assume that higher dimensions would further increase the performance, but this
introduces an additional computational burden.
Binarization Finally, the choice of _binarization algorithm_ is the least
influencing factor. Even global binarization algorithms such as Otsu do
perform well and close to our best one, Sauvola. We think this is because the
data used does not contain heavy degradation, decreasing the difficulty of the
task.

###  5.2 Finetuning

In this section, we assess two finetuning strategies: the conventional
supervised approach (Sup.) and the unsupervised method (Cl-S [6]). While
supervised finetuning is the commonplace default in various computer vision
tasks, this shifts for WR, where the benchmark is unsupervised training on
pseudolabels, generated by the clustering of the corresponding SIFT
descriptors. In Table 6, the results for both finetuning strategies are shown.
For a fair comparison, we also train ResNet56 and the ViT architecture ViT-4/8
from scratch for both settings to show the influence of Saghog.

Table 6: Finetuning NetRVLAD with different backbones and target labels: Supervised (_Sup._) and Cl-S [6]. We report mAP in %. | Historical-WI | HisFrag20
---|---|---
| Sup. | Cl-S | Sup. | Cl-S
ResNet56 | 67.2 | 73.3 | 42.5 | 41.2
ViT-4/8 (from scratch) | 60.0 | 71.1 | 43.4 | 47.4
Saghog (ours) |  |  |  |
frozen backbone | 64.7 | 64.0 | 56.3 | 57.2
finetuned | 61.4 | 73.3 | 50.7 | 54.6
$\mathbf{\Delta}$ to best |  -2.5 |  0.0 |  +12.9 |  +9.8

#### Historical-WI

In contrast to [6], we have a surprisingly good baseline when training
ResNet56 with NetRVLAD on writer labels (67.2 % mAP). This setting even
outperforms Saghog with supervised finetuning, indicating that the pretraining
task is contrary to the finetuning - freezing the backbone also improves the
mAP. For unsupervised training, Saghog is on par with ResNet56 and
outperforming training ViT from scratch. We think this is mainly due to the
dataset only containing pure handwriting, where a simple CNN is enough to
achieve good performance. The approach seems to be limited by the finetuning
task (the pseudolabels generated by Cl-S).

#### HisFrag20

However, the application of Saghog on HisFrag20, a more complex and larger
dataset, drastically shifts the results towards Saghog. Firstly, training ViT
on Cl-S, even from scratch, outperforms the current state of the art.
Furthermore, ViTs outperforms the residual baseline in all experiments. Our
best result is achieved by freezing the encoder of Saghog and only training
NetRVLAD, achieving 56.3 % for supervised and 57.2 % for unsupervised
training, which is a margin of 12.9 %/9.8 % to the best setting when training
from scratch. In conclusion, we argue that Saghog works best for large,
complex datasets, where networks trained from scratch fail to perform due to
noisy data - either on image- or label-level - or due to a domain shift
between training and test data.

###  5.3 Comparison to state of the art

We compare the results of Saghog to state of the art. In addition to the
previous evaluation, we extend our analysis to include a comparison of the
GRK-Papyri dataset. All results are shown in Table 7.

Table 7: Comparison to state of the art. ($\dagger$) denotes our own
implementation, subscript i the dimension of the final page descriptor.

(a) Historical-WI Method | mAP | Top-1
---|---|---
Cl-S + ResNet56. \+ NetRVLAD512 [29] | 73.4 | 88.5
Cl-S + ResNet20. \+ mVLAD400 [6] | 73.2 | 87.6
Cl-S + ResNet20. \+ mVLAD32000 [6] | 74.8 | 88.6
Pathlet/SIFT + bVLAD1000 [21] | 77.1 | 90.1
Cl-S + Saghog.. \+ NetRVLAD512 | 73.3 | 88.2
with SGR${}_{k=2}^{\dagger}$ [29] | 80.3 | 91.1

(b) GRK-Papyri Method |  mAP | Top-1
---|---|---
Cl-S + ResNet20 + mVLAD32000 [8] | 42.2 | 52.0
R-SIFT + mVLAD32000 [8] | 42.8 | 48.0
Cl-S + ResNet20 + NetRVLAD${}_{2048}^{\dagger}$ [29] | 38.8 | 56.0
Cl-S + Saghog–. \+ NetRVLAD2048 | 38.2 | 58.0
with SGR${}_{k=3}^{\dagger}$ [29] | 47.6 | 58.0

(c) HisFrag20 | Writer Retrieval | Page Retrieval
---|---|---
Method | mAP | Top-1 | mAP | Top-1
Cl-S + mVLAD3200 \+ ESVM [2] | 33.7 | 68.9 | - | -
FragNet512 [32] | 33.5 | 77.1 | 22.6 | 36.4
ResNet50 + NetVLAD6400 \+ ESVM [3] | 38.1 | 77.9 | - | -
Cl-S + ResNet56 + NetRVLAD${}_{512}^{\dagger}$ [29] | 41.2 | 68.5 | 23.5 | 33.3
Feature Mixer256 [30] | 44.0 | 81.9 | 29.3 | 45.0
A-VLAD500 [26] | 46.6 | 85.2 | - | -
Cl-S + Saghog \+ NetRVLAD512 | 57.2 | 81.7 | 35.2 | 47.6
with SGR${}_{k=2}^{\dagger}$ [29] | 68.7 | 88.7 | 54.5 | 70.6

#### Historical-WI

As discussed in the previous section, self-supervised pretraining has a
limited effect on the Historical-WI dataset - probably due to the dataset
containing only (binarized) handwriting. However, Saghog is on par with all
deep-learning-based approaches (Table 7(a)) and does not harm the performance.
Interestingly, the best approach in terms of mAP (without considering
reranking) is still relying on handcrafted features [21].

#### GRK-Papyri

We also finetune Saghog on GRK-Papyri in an unsupervised manner, a small
dataset of only 50 documents. The results are shown in Table 7(b). The global
page descriptors used are averaged on five different models trained, similar
to the ensemble method used by Christlein et al. [8]. We also choose to
decrease the dimension of Saghog to 128 instead of 512 for a fair comparison.
While NetVLAD is trailing the traditional VLAD on mAP for both networks,
ResNet20 and Saghog, the end-to-end learned method achieves better Top-1
accuracy, with Saghog outperforming related work with 58.0 %.

#### HisFrag20

On the largest and most diverse dataset, HisFrag20, we show that Saghog is
superior to all previous approaches by a large margin on both tasks - WR
(+10.6 %) and page retrieval (+5.9 %). A thorough comparison to related work
is given in Table 7(c). Although the mAP and therefore the ranked list is
better, Saghog has a slightly worse Top-1 accuracy. As we showed in our work
introducing the feature mixer [30], networks trained on full-color images tend
to infer features from the background during training, which is also shown in
our comparison: For example, approaches relying on $32\times 32$ patches such
as [2, 29] suffer from a limited Top-1 accuracy. We think this is due to
approaches on image-level retrieving images from the same page, where the
focus is not on retrieving documents based on the similarity of the
handwriting, but rather on background similarity, which is shared within a
page. This phenomenon is also observed by considering the page retrieval
performance (e.g., [29] vs. [30]). However, we argue that due to Saghog
achieving a superior performance in terms of mAP, we can apply a reranking
scheme such as SGR which is shown to be beneficial in particular for Top-1
accuracy [29]. By using SGR, we are able to boost both metrics by 11.5 % mAP /
7 % Top-1 for WR and 19.3 % mAP / 23 % Top-1 for page retrieval, which
outperforms related work, even on the Top-1 accuracy. We also provide
qualitative results in Fig. 5.

Figure 5: Qualitative results of the retrieval on HisFrag20. Left: Query. We
show the four nearest documents. If the retrieved document is written by the
same author as the query, we highlight the image in green, otherwise red.

##  6 Conclusion

In this paper, we introduce Saghog, the first masking-based self-supervised
approach for the downstream task of WR. Our pretraining includes
reconstructing HOG features of the binarized image, which leads the network to
focus on handwriting and suppress the background. Additionally, we propose a
preprocessing via SAM to make use of previous historical documents for our
pretraining. We evaluate three different datasets and two different finetuning
methods commonly used in the domain of WR. In the following, we summarize our
key results and ideas for future work.

#### Our findings

Firstly, we investigate our preprocessing scheme and show that the
segmentation of handwriting is beneficial for the pretraining of Saghog.
Secondly, Saghog shines on WR for large, complex datasets containing sources
of noise (e.g. HisFrag20 with small, torn fragments). We are able to
outperform related work by a significant margin on this difficult dataset.
Furthermore, we show that for a binarized dataset such as Historical-WI the
effect of pretraining is limited, for which current methods are already
working decent. In the end, we also evaluate on GRK-Papyri, a small dataset,
and show that the application of Saghog improves the Top-1 accuracy upon state
of the art. Saghog also shows the applicability of the ViT architecture for
WR, which heavily depends on CNNs.

#### Future work

We further want to investigate self-supervised learning for. On the one hand,
one could improve the diversity of data for pretraining (we mainly use Latin
script and documents including a limited amount of degradation) to improve
results also on smaller datasets such as GRK-Papyri. On the other hand, tuning
the reconstruction task, e.g., using a U-Net for improved binarization or
integrating style embeddings from specific domains to refine the self-
supervised embeddings, promises to further improve the results.

#### Acknowledgements

We thank Vincent Christlein for providing the binarized images of GRK-Papyri.

## References

  * [1] Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: 2021 IEEE/CVF International Conference on Computer Vision, ICCV 2021, Montreal, QC, Canada, October 10-17, 2021. pp. 9630–9640 (2021)
  * [2] Chammas, M., Makhoul, A., Demerjian, J.: Writer identification for historical handwritten documents using a single feature extraction method. In: 19th IEEE International Conference on Machine Learning and Applications, ICMLA 2020, Miami, FL, USA, December 14-17, 2020. pp. 1–6 (2020)
  * [3] Chammas, M., Makhoul, A., Demerjian, J., Dannaoui, E.: A deep learning based system for writer identification in handwritten arabic historical manuscripts. Multimedia Tools and Applications 81(21), 30769–30784 (2022)
  * [4] Chen, T., Kornblith, S., Norouzi, M., Hinton, G.E.: A simple framework for contrastive learning of visual representations. In: Proceedings of the 37th International Conference on Machine Learning, ICML 2020, 13-18 July 2020, Virtual Event. Proceedings of Machine Learning Research, vol. 119, pp. 1597–1607 (2020)
  * [5] Christlein, V., Bernecker, D., Angelopoulou, E.: Writer identification using VLAD encoded contour-zernike moments. In: 13th International Conference on Document Analysis and Recognition, ICDAR 2015, Nancy, France, August 23-26, 2015\. pp. 906–910 (2015)
  * [6] Christlein, V., Gropp, M., Fiel, S., Maier, A.K.: Unsupervised feature learning for writer identification and writer retrieval. In: 14th IAPR International Conference on Document Analysis and Recognition, ICDAR 2017, Kyoto, Japan, November 9-15, 2017. pp. 991–997 (2017)
  * [7] Christlein, V., Maier, A.K.: Encoding CNN activations for writer recognition. In: 13th IAPR International Workshop on Document Analysis Systems, DAS 2018, Vienna, Austria, April 24-27, 2018. pp. 169–174 (2018)
  * [8] Christlein, V., Marthot-Santaniello, I., Mayr, M., Nicolaou, A., Seuret, M.: Writer retrieval and writer identification in greek papyri. In: Intertwining Graphonomics with Human Movements - 20th International Conference of the International Graphonomics Society, IGS 2021, Las Palmas de Gran Canaria, Spain, June 7-9, 2022, Proceedings. vol. 13424, pp. 76–89 (2022)
  * [9] Christlein, V., Nicolaou, A., Seuret, M., Stutzmann, D., Maier, A.: ICDAR 2019 competition on image retrieval for historical handwritten documents. In: 2019 International Conference on Document Analysis and Recognition, ICDAR 2019, Sydney, Australia, September 20-25, 2019. pp. 1505–1509 (2019)
  * [10] Cloppet, F., Eglin, V., Helias-Baron, M., Kieu, V.C., Vincent, N., Stutzmann, D.: ICDAR2017 competition on the classification of medieval handwritings in latin script. In: 14th IAPR International Conference on Document Analysis and Recognition, ICDAR 2017, Kyoto, Japan, November 9-15, 2017. pp. 1371–1376 (2017)
  * [11] Cloppet, F., Eglin, V., Kieu, V.C., Stutzmann, D., Vincent, N.: ICFHR2016 competition on the classification of medieval handwritings in latin script. In: 15th International Conference on Frontiers in Handwriting Recognition, ICFHR 2016, Shenzhen, China, October 23-26, 2016. pp. 590–595 (2016)
  * [12] Diem, M., Kleber, F., Sablatnig, R., Gatos, B.: cBAD: ICDAR2019 competition on baseline detection. In: 2019 International Conference on Document Analysis and Recognition, ICDAR 2019, Sydney, Australia, September 20-25, 2019. pp. 1494–1498 (2019)
  * [13] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021 (2021)
  * [14] Fiel, S., Kleber, F., Diem, M., Christlein, V., Louloudis, G., Nikos, S., Gatos, B.: ICDAR2017 competition on historical document writer identification (historical-wi). In: 14th IAPR International Conference on Document Analysis and Recognition, ICDAR 2017, Kyoto, Japan, November 9-15, 2017\. pp. 1377–1382 (2017)
  * [15] He, K., Chen, X., Xie, S., Li, Y., Dollár, P., Girshick, R.B.: Masked autoencoders are scalable vision learners. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp. 15979–15988. IEEE (2022)
  * [16] He, K., Fan, H., Wu, Y., Xie, S., Girshick, R.B.: Momentum contrast for unsupervised visual representation learning. In: 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020. pp. 9726–9735. Computer Vision Foundation / IEEE (2020)
  * [17] Jégou, H., Douze, M., Schmid, C.: On the burstiness of visual elements. In: 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA. pp. 1169–1176 (2009)
  * [18] Keglevic, M., Fiel, S., Sablatnig, R.: Learning features for writer retrieval and identification using triplet cnns. In: 16th International Conference on Frontiers in Handwriting Recognition, ICFHR 2018, Niagara Falls, NY, USA, August 5-8, 2018. pp. 211–216 (2018)
  * [19] Kingma, D.P., Ba, J.: Adam: A method for stochastic optimization. In: Bengio, Y., LeCun, Y. (eds.) 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings (2015)
  * [20] Kirillov, A., Mintun, E., Ravi, N., Mao, H., Rolland, C., Gustafson, L., Xiao, T., Whitehead, S., Berg, A.C., Lo, W.Y., Dollar, P., Girshick, R.: Segment anything. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 4015–4026 (2023)
  * [21] Lai, S., Zhu, Y., Jin, L.: Encoding pathlet and SIFT features with bagged VLAD for historical writer identification. IEEE Trans. Inf. Forensics Secur. 15, 3553–3566 (2020)
  * [22] Lastilla, L., Ammirati, S., Firmani, D., Komodakis, N., Merialdo, P., Scardapane, S.: Self-supervised learning for medieval handwriting identification: A case study from the vatican apostolic library. Information Processing & Management 59(3), 102875 (2022)
  * [23] Loshchilov, I., Hutter, F.: SGDR: stochastic gradient descent with warm restarts. In: 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net (2017)
  * [24] Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019 (2019)
  * [25] Mohammed, H.A., Marthot-Santaniello, I., Märgner, V.: Grk-papyri: A dataset of greek handwriting on papyri for the task of writer identification. In: 2019 International Conference on Document Analysis and Recognition, ICDAR 2019, Sydney, Australia, September 20-25, 2019. pp. 726–731 (2019)
  * [26] Ngo, T.T., Nguyen, H.T., Nakagawa, M.: A-VLAD: an end-to-end attention-based neural network for writer identification in historical documents. In: 16th International Conference on Document Analysis and Recognition, ICDAR 2021, Lausanne, Switzerland, September 5-10, 2021, Proceedings, Part II. vol. 12822, pp. 396–409 (2021)
  * [27] Otsu, N.: A threshold selection method from gray-level histograms. IEEE Transactions on Systems, Man, and Cybernetics 9(1), 62–66 (Jan 1979)
  * [28] Peer, M., Kleber, F., Sablatnig, R.: Self-supervised vision transformers with data augmentation strategies using morphological operations for writer retrieval. In: Frontiers in Handwriting Recognition - 18th International Conference, ICFHR 2022, Hyderabad, India, December 4-7, 2022, Proceedings. pp. 122–136 (2022)
  * [29] Peer, M., Kleber, F., Sablatnig, R.: Towards writer retrieval for historical datasets. In: Document Analysis and Recognition - ICDAR 2023 - 17th International Conference, San José, CA, USA, August 21-26, 2023, Proceedings, Part I. pp. 411–427 (2023)
  * [30] Peer, M., Sablatnig, R.: Feature mixing for writer retrieval and identification on papyri fragments. In: Proceedings of the 7th International Workshop on Historical Document Imaging and Processing (2023)
  * [31] Sauvola, J., Pietikäinen, M.: Adaptive document image binarization. Pattern Recognition 33(2), 225–236 (Feb 2000)
  * [32] Seuret, M., Nicolaou, A., Maier, A., Christlein, V., Stutzmann, D.: ICFHR 2020 competition on image retrieval for historical handwritten fragments. In: 17th International Conference on Frontiers in Handwriting Recognition, ICFHR 2020, Dortmund, Germany, September 8-10, 2020. pp. 216–221 (2020)
  * [33] Su, B., Lu, S., Tan, C.L.: Binarization of historical document images using the local maximum and minimum. In: Proceedings of the 9th IAPR International Workshop on Document Analysis Systems (Jun 2010)
  * [34] Wang, X., Han, X., Huang, W., Dong, D., Scott, M.R.: Multi-similarity loss with general pair weighting for deep metric learning. In: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019. pp. 5022–5030 (2019)
  * [35] Wang, Z., Maier, A., Christlein, V.: Towards end-to-end deep learning-based writer identification. In: 50. Jahrestagung der Gesellschaft für Informatik, INFORMATIK 2020 - Back to the Future, Karlsruhe, Germany, 28. September - 2. Oktober 2020. vol. P-307, pp. 1345–1354 (2020)
  * [36] Wei, C., Fan, H., Xie, S., Wu, C., Yuille, A.L., Feichtenhofer, C.: Masked feature prediction for self-supervised visual pre-training. In: IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2022, New Orleans, LA, USA, June 18-24, 2022. pp. 14648–14658 (2022)
  * [37] Zenk, J., Kordon, F., Mayr, M., Seuret, M., Christlein, V.: Investigations on self-supervised learning for script-, font-type, and location classification on historical documents. In: Proceedings of the 7th International Workshop on Historical Document Imaging and Processing. HIP ’23 (Aug 2023)

  *[WR]: Writer Retrieval
  *[Saghog]: Self-Supervised Autoencoder for Generating HOG features
  *[HOG]: Histogram of Oriented Gradients
  *[ViT]: Vision Transformer
  *[SAM]: Segment Anything Model
  *[ViTs]: Vision Transformer
  *[mAP]: Mean Average Precision
  *[MAE]: Masked Autoencoder
  *[VLAD]: Vector of Locally Aggregated Descriptors
  *[ESVM]: Exemplar SVM
  *[SGR]: Similarity Graph Reranking


∎

11institutetext: Binlong Zhang 22institutetext: Northwest University
22email<EMAIL_ADDRESS>33institutetext: *Corresponding Author:
Wei Zhou 44institutetext: Northwest University
44email<EMAIL_ADDRESS>
# Transformer-Encoder-GRU (T-E-GRU) for Chinese Sentiment Analysis on Chinese
Comment Text

Binlong Zhang  Wei Zhou

(Received: date / Accepted: date)

###### Abstract

Chinese sentiment analysis (CSA) has always been one of the challenges in
natural language processing due to its complexity and uncertainty. Transformer
has succeeded in capturing semantic features, but it uses position encoding to
capture sequence features, which has great shortcomings compared with the
recurrent model. In this paper, we propose T-E-GRU for Chinese sentiment
analysis, which combine transformer encoder and GRU. We conducted experiments
on three Chinese comment datasets. In view of the confusion of punctuation
marks in Chinese comment texts, we selectively retain some punctuation marks
with sentence segmentation ability. The experimental results show that T-E-GRU
outperforms classic recurrent model and recurrent model with attention.

###### Keywords:

Chinese reviews Chinese sentiment analysis Recurrent model Transformer-Encoder
T-E-GRU

##  1 Introduction

Sentiment analysis has become one of the most active fields in natural
language processing (NLP) feldman2013techniques ; zhang2018deep . With the
development of sentiment analysis, the research is not only limited to the
computer field, but also expanded to management, social science, etc. Its
potential commercial value has also attracted the attention of all sectors of
society. With the development of GPU, sentiment analysis methods based on
machine learning and deep learning gradually occupy the dominant position and
continue to achieve the state-of-the-art performance.

The earliest CSA is based on sentiment dictionary, which judges the sentiment
tendency of the text through its vocabulary. But none of them have the ability
to capture sequence features. Recurrent neural network (RNN) proposed by
Jeffrey L Elman preRNN  is one of the effective methods to alleviate this
defect, but it was accompanying by vanishing/exploding gradient problem
bengio1994Long-term-dependencies-diffcult . Subsequently, a series of
recurrent models based on RNN were proposed to address these problems, such as
long short-term memory (LSTM) hochreiter1997LSTM , gate recurrent neural
network (GRU) cho2014learningGRU , etc. However, the vanishing/exploding
gradient problem on long sequence is still one of its most important
limitations. To address the vanishing/exploding problems, Bahdanau et al.
bahdanau2014Attention-first  proposed the attention mechanism, which assigns
weights to input sequence to make the model pay more attention to keywords.
After that, the models with attention continues to achieve state-of-the-art
performances. In particular, in 2017, Vaswani et al. vaswani2017Transformer
proposed scaled dot-product attention which has become one of the effective
ways to apply attention mechanism and transformer model. Transformer and its
variants have achieved the state-of-the-art performances on basic problems in
many fields BERT ; vaswani2017Transformer . Transformer-Encoder realizes the
extraction of sequence features through position encoding (PE), which still
has a gap with the natural sequence feature extractor such as RNN.

Although sentiment analysis has made remarkable achievements in recent years,
most of them are based on English. Compared with English, Chinese natural
language processing is more challenging. On the one hand, Chinese has more
complicated vocabulary and semantics. On the other hand, Chinese text
semantics are more dependent on context. Inspired by the powerful global
feature extraction ability of attention & transformer and the powerful
sequence feature extraction ability of recurrent model, we proposed
Transformer-Encoder-GRU (T-E-GRU) which combine the GRU with the structure of
transformer, and use it for sentiment analysis of Chinese reviews. We compare
it with recurrent model, recurrent model with attention, etc. The results show
that our model achieves better results.

The main contributions of this paper are summarized as following:

  1. 1.

We proposed a novel model called T-E-GRU based on GRU and transformer encoder,
which is exactly suitable for CSA.

  2. 2.

Focus on the complexity and confusion of punctuation in Chinese comments, we
have selectively retained some punctuation that has the ability to clauses.

  3. 3.

Extensive experiment demonstrate that T-E-GRU achieves better effect on CSA.

##  2 Related Work

###  2.1 Chinese Text To sequence

In Chinese natural language processing, Chinese word segmentation (CWS) is a
basic and important task, which segments text into independent word units.
Compared with the natural delimiter spaces in English, Chinese has no formal
delimiters and no strict division method, and even sometimes the division
method depends on the context. The main disadvantages of dictionary-based word
segmentation algorithms such as forward maximum matching (FMM), backward
maximum matching (BMM), and bi-direction matching (BiMM) are slow in matching
and cumbersome implementation of supplementing unrecorded words
CWS10yearReview . Based on statistical machine learning algorithms, the main
idea is that the more adjacent characters appear in the context at the same
time, the more likely they are to form a word. The current main models are:
N-gram model (N-gram) N-gram , hidden Markov model (HMM) HMM , condition
random field (CRF) CRF , RNN variant model LSTM-CWS ; Bi-LSTM-CWS . Various
open CWS libraries such as Jieba, FoolNLTK, LTP, etc. have performed well in
CWS. Although some people have raised doubts about the necessity of CWS in
deep learning WS-is-necessary , most of the state-of-the-art models are based
on CWS.

The simplest way to represent words is to use one-hot encoding which brings
curse of dimensionality, semantic gap and poor scalability. Fighting the curse
of dimensionality, Bengio Y et al. proposed a method of using low-dimensional
vector to represent vocabulary proposed-Embedding . Word Embedding is a by-
product of their model. In 2013, Tomas Mikolov proposed word2Vec word2Vec ,
which is a faster and better method for training word embedding model. There
are two algorithms: skip-grams and continuous bag-of-words (CBOW) CBOW , both
of which based on N-gram model.The N-gram model assumes that a word is only
related to N surrounding words. This assumption determines that the
disadvantage is insufficient use of global information. Jeffrey Pennington et
al. proposed global vectors for Word representation (GloVe) which considers
both global and local information glove . However, the word representation
mentioned above is a static model, which cannot cope with the situation where
a word has multiple meanings. In 2018, Peters M E et al. proposed embeddings
from language model (ELMO) ELMO  by using bidirectional LSTM. Since
transformer has been proved to have powerful feature extraction ability in
many researches vaswani2017Transformer , generative pre-training (GPT) was
proposed GPT . It replaces LSTM in ELMO with transformer. Although GPT uses a
unidirectional language model, it achieves the best results in 9 of the 12
tasks in NLP. In 2019, Devlin et al. proposed bidirectional encoder
representations from transformers (BERT) BERT  which replace unidirectional
with bidirectional language models, and also combined with the tricks of CBOW.
BERT, as synthesis of word representation model in recent years, achieved
state-of-the-art performances in multiple NLP basic tasks.

###  2.2 Recurrent models

Recurrent model is one of the effective methods for processing sequence input.
In 1990, neuroscientist Jeffrey L Elman proposed RNN which is based on the
Jordan Network preRNN  back-propagation (BP) BP . However, vanishing/exploding
gradient problem on long sequence comes with RNN bengio1994Long-term-
dependencies-diffcult . In 1997, Hochreiter et al. proposed LSTM, which can
alleviate this problem by introducing delicate gate mechanism
hochreiter1997LSTM . However, LSTM often has overfitting on some tasks due to
the complexity of its structure. Cho et al. simplified the gate of LSTM and
propose GRU cho2014learningGRU . GRU has the same performance as LSTM but with
fewer parameters. RNN, LSTM and GRU have become the most important recurrent
models, but none of them can completely solve the vanishing/exploding gradient
problem. Up to now, methods to improve RNN from different perspectives are
still proposed one after another liu2021selfish .

###  2.3 Attention and Transformer

In 2014, Mnih V et al. applied attention mechanism on RNN to solve image
classification tasks cv_Att . Then, attention was used as a way to solve the
information overload problem in NLP tasks. Bahdanau et al. extended the basic
seq2seq model with attention, and it yields good results on longer sentences
bahdanau2014Attention-first . Then Cheng proposed intra-attention Att-
breakout-Seq2seq , which broke the limitation of attention mechanism used in
seq2seq. Many researches bahdanau2014Attention-first ; Att-breakout-Seq2seq ;
cv_Att ; RecurrentAttNet-AspectSentimentAnalysis  show that attention is
effective in long sequence information loss. Vaswani et al. completely
abandoned RNN and CNN to build transformer which is entirely based on the
fully-connected layer and attention mechanism vaswani2017Transformer . In
2020, Zhou H et al. proposed probsparse self-attention mechanism and informer,
and this model achieved high prediction capacity in the long sequence time-
series forecasting zhou2020informer . Transformer is divided into two parts:
encoding and decoding. Transformer-encoder uses position encoding to make the
model understand sequence features. T-E-GRU replaces the position encoding
with GRU, which has similar performance to LSTM. So T-E-GRU has the advantages
of transformer-encoder and recurrent model, which will be more suitable for
CSA.

###  2.4 Chinese sentiment analysis

Chinese sentiment analysis is very challenging due to the complexity of
Chinese. A plain method is based on the n-gram model and syntax tree, using
some classic machine learning classification models such as naive Bayesian
(NB), support vector machine (SVM), etc. for CWS zou2015sentiment . The
performance of their model were unsatisfactory. Ding et al. ding2008holistic
also proposed a method of matching sentiment words in specific field for CSA.
J Liang et al. liang2015polarity  combined polarity shifting and LSTM for CSA.
Xiao Z xiao2016chinese  proposed bidirectional LSTM with word embedding for
CSA. Their model already has high accuracy in CSA. As the attention mechanism
and transformer were proposed, a series of models were for CSA. Bin et al.
bin2017aspect  proposed multi-attention convolutional neural network (CNN) for
aspect-based sentiment analysis. In 2019, Shi, Xiaoming et al.
shi2019attention  proposed attention-based bidirectional hierarchical LSTM
networks. In 2021, Wang, Shitao et al. wang2021bigru  proposed bigru-multi-
head self-attention network for CSA. In addition, many state-of-the-art models
based on transformer or BERT have been proposed for CSA tang2021chinese ;
zhu2019transformer ; gao2021chinese .

##  3 T-E-GRU

At present, the state-of-the-art models for NLP are mainly based on
transformer or recurrent model. In general, transformer has powerful global
feature extraction capabilities and is very suitable for CSA. Recurrent model
is a natural sequence structure, which is very suitable for capturing sequence
features. In particular, GRU has become one of the best recurrent models due
to its similar performance to LSTM and cheaper calculations.

Inspired by the global feature extraction capabilities of transformer and the
sequence feature extraction capabilities of GRU, we proposed T-E-GRU for CSA.
T-E-GRU combines transformer-encoder and GRU. Compared with recurrent model,
the multi-head self-attention mechanism and residual connection in the
transformer-encoder of T-E-GRU can better cooperate with the loss of long
sequence information. Different from transformer, T-E-GRU uses GRU to extract
sequence features instead of position encoding, which can better deal with the
problem that text sentiment depends on word order. In addition, compared to
recurrent model with attention, T-E-GRU combines the transformer structure
that has achieved good results in NLP.

Figure 1: Transformer-Encoder-GRU

As shown in Figure 1, our model consists of four parts: text to sequence,
transformer-encoder, GRU and classification. Text to sequence converts
original Chinese text into vector sequence. Transformer-encoder reprocesses
the original vector sequence So that each processed vector is determined by
the entire input vector sequence. GRU uses the characteristics of recurrent
model to process sequence features. Due to the reprocessing in the
transformer-encoder, processed vector has global information, which can better
deal with the problem of long sequence information loss, and this problem is
the most difficult one to overcome in recurrent model. Finally, the final
state of the GRU is adopted as the input of the classification to output the
predicted probability.

###  3.1 Text To Sequence

Text to sequence maps the original Chinese text to vector sequence
$X=(x_{1},x_{2},\cdots,x_{n})$, where $x_{i}\in\mathbb{R}^{d_{R}}$. n is the
number of tokens and $d_{R}$ is the dimension of word representation. Text to
sequence mainly consists of three parts: filter, tokenization, representation.
Filter is used to deal with irregular punctuation, characters, etc. in the
original Chinese text. After filtering, the tokens generated after
tokenization will be shorter than before and will be fewer meaningless tokens.
Specifically, the filter selectively removes most of the punctuation that does
not have the ability to clause. Then tokenization convert the filtered text
into a series of tokens. Here, any suitable CWS library can be used, and we
use Jieba to finish CWS. Next, we use vectors to represent these tokens. Word
embedding is one of the important representation methods. Representation takes
these tokens as input and vector sequence $X=(x_{1},x_{2},\cdots,x_{n})$ as
output. After the above processes, text to sequence converts the original
Chinese text into vector sequence $X$.

###  3.2 Transformer-Encoder

Transformer-encoder takes $X$ from text to sequence as input and generates
output $\hat{X}=(\hat{x}_{1},\hat{x}_{2},\cdots,\hat{x}_{n})$. As shown in
Figure 1, transform-encoder is mainly divided into multi-head self-attention
and feedforward network.

Attention mechanism has become one of the most effective methods to capture
global information attentionInMulTask ; attentionInNLP ;
vaswani2017Transformer . Multi-head attention is based on scaled dot-product
attention proposed by Vaswani et al. vaswani2017Transformer . Multi-head
attention is shown in the following equations:

$\begin{split}&MultiHead(Q,K,V)=concat(Att_{1},Att_{2},...,Att_{n})\\\ &where\
\ Att_{i}=softmax(\frac{QK^{T}}{\sqrt{d_{k}}})V\end{split}$ (1)

where $Q,K,V$ represent query, keys and values respectively which they are all
input matrices, $d_{k}$ represents the dimension of the keys, $n$ equals to
the number of heads, and we set $n=2$ in our model. Especially when $Q,K,V$
are the same, it is called self-attention. Here we use the output $X$ from
text to sequence as $Q,K,V$, and then output $X_{Att}$. Then we use residual
connection and feedforward network in the following:

$\begin{split}&X_{Att}=MultiHead(X,X,X)\\\ &X_{residual}=norm(X_{Att}+X)\\\
&\hat{X}=norm(X_{residual}+\text{FFN}(X_{residual}))\end{split}$ (2)

where $norm$ is normalization layer, FFN consists of two linear
transformations with a ReLU:

$\text{FFN}(X_{residual})=Linear(\max(0,Linear(X_{residual})))$ (3)

where the dimension of inner-layer is 2048 and other details of transformer-
encoder are mentioned in vaswani2017Transformer .

Finally, transformer-encoder transfers $X=(x_{1},x_{2},\cdots,x_{n})$ to
$\hat{X}=(\hat{x}_{1},\hat{x}_{2},\cdots,\hat{x}_{n})$, where $\hat{x_{i}}$ is
determined by the entire input $X$.

###  3.3 GRU

Figure 2: GRU

GRU is one of the variants of RNN, which improved from the LSTM. It performs
similarly to LSTM, but is computationally cheaper. As shown in Figure 2. For
any input state $x_{t}$ and the hidden state $h_{t-1}$ generated by the
previous sequence:

$\begin{split}&u_{t}=sigmoid(x_{t}W_{z}+h_{t-1}U_{z})\\\
&r_{t}=sigmoid(x_{t}W_{r}+h_{t-1}U_{r})\\\
&h_{t}=(1-u_{t})h_{t-1}+u_{t}*tanh(x_{t}W_{h}+(h_{t-1}\cdot
r_{t})U_{h})\end{split}$ (4)

where $r_{t}$ is the reset gate, $u_{t}$ is the update gate,$W_{z,r,h}$ and
$U_{z,r,h}$ are the weight matrices that need to be trained.

Here we take $\hat{X}$ from transformer-encoder as input of GRU to generate
the state sequence $(h_{1},h_{2},\cdots,h_{n})$. We use the final state
$h_{n}$ as the output of GRU, and then use the classification to implement
CSA. The classification here is simply composed of a linear layer and
logsoftmax.

##  4 Experimental

In this section, we compare our method with the state-of-the-art frameworks
(Including various recurrent neural networks and recurrent neural networks
with attention) on three datasets for CSA tasks, including DMCS_V2 4.1,
YF_DianPing 4.2 and Online_shopping_10 cats 4.3 datasets. We also conduct
ablation study to research the performances of T-E-GRU.

###  4.1 DMCS_V2

Dataset. Douban movie short comments dataset V2
(DMSC_V2111https://www.kaggle.com/utmhikari/doubanmovieshortcomments) comes
from the Kaggle competition. It collects user reviews, ratings, and likes of
movies on Douban (a platform for Chinese film lovers). This dataset collected
short user reviews of 28 popular movies, with a total of more than 2 million
comments. We label 1, 2-star reviews as “negative reviews” and 4,5-star
reviews as “positive reviews” (3-star reviews are ignored because of its vague
sentimentality). In the experiments we randomly selected 700,000 comments data
including 350,000 positive samples and 350,000 negative samples. We divide
them into three parts, 480,000 for the training set, 120,000 for the
validation set, and 100,000 for the test set.

Data preprocessing. Firstly, we need to deal with the punctuation in the text
data. Since the use of punctuation marks in most short film reviews is not
standardized, and some punctuation has less semantic information, we retain
some punctuation with clause ability, such as comma, full stops, question
marks, semicolons, etc. and deleted others. Our focus is not to discuss CWS
model or the necessity of CWS in deep learning models, so we use open source
library Jieba developed by Baidu engineer Sun Junyi. After word segmentation,
we pad and truncate the tokens for batch processing. In addition, we use pre-
trained word embedding to convert words into vectors, which was trained by Li
S, Zhao Z, Hu R, et al. preTrainEmbedding  by using skip-gram with negative
sample method based on Zhihu Q&A preTrainEmbedding . It contains 260,000 used
high-frequency words, which can cover 93.98% of the words in train set. The
dimension of the word vector is 300. DMCS_V2 is a short movies comments
dataset. After tokenization, 99.8% of samples’ tokens are less than 100. So we
align the input sequence to 100 by padding and truncating it at the front of
it. For word representation, we use 200,000 high-frequency words in the pre-
trained word embedding model, which can cover 93.86% of the word needed in
there.

Train Details. In the experiments of RNN, LSTM and GRU, we use negative log-
likelihood (NLL) loss function, set the batch size and the hidden size to 128
and 256 respectively, adopt Stochastic Gradient Descent (SGD) optimizer with
learning rate of 0.002 for 100 epochs, and decay the learning rate by 0.5
every 50 epochs. We use the same settings for the bi-directional recurrent
model, and the model combined with attention mechanism and recurrent models as
mentioned in before, except that the hidden layer in a single direction is set
to 128, and set the learning rate to 0.01. Finally, we also use the same
setting on our T-E-GRU, and set the dropout of transformer-encoder to 0.3.

Results and Analysis.

Table 1: The performance of various models on the DMCS_V2 Method | Accuracy | F1 | Test Time(ms)
---|---|---|---
RNN RNN-first  | 88.80% | 88.77% | 0.8483
LSTM hochreiter1997LSTM  | 88.41% | 88.28% | 2.0053
GRU cho2014learningGRU  | 88.70% | 88.80% | 1.9884
Bi-RNN | 88.30% | 88.30% | 1.4551
Bi-LSTM | 87.95% | 88.02% | 3.8016
Bi-GRU | 88.00% | 88.80% | 1.4989
RNN-Attention | 88.50% | 88.34% | 1.2503
LSTM-Attention | 89.43% | 89.32% | 2.3592
GRU-Attention | 88.08% | 88.27% | 2.3599
Attention-RNN | 85.77% | 86.09% | 1.2624
Attention-LSTM | 85.67% | 85.55% | 2.4111
Attention-GRU | 84.68% | 84.32% | 2.3835
BiRNN-Attention | 88.64% | 88.50% | 1.8544
BiLSTM-Attention | 88.45% | 88.48% | 4.3638
BiGRU-Attention | 88.18% | 87.60% | 1.9140
T-E-GRU | 90.09% | 90.07% | 2.6987

The performance of the above models on the test set is shown in Table 1. We
use accuracy and F1 as the criteria:

$\begin{split}&Acc=\frac{TP+TN}{TP+TN+FP+FN}\\\ &F1=\frac{2\times TP}{2\times
TP+FP+FN}\end{split}$ (5)

where TP, TN, FP and FN represent true positive, true negative, false
positive, and false negative respectively. Test Time denotes the time required
for trained model to test a comment. As shown in Table 1, the accuracy and F1
of the Bi-directional recurrent model are 0.1%$\sim$1% lower than recurrent
model on DMSC_V2. The best recurrent model is simple RNN, and its accuracy and
F1 have reached 88.8% and 89.0% respectively. At the same time, simple RNN
also has a good performance in time with 0.8483ms. After introducing
attention, the accuracy, F1 of various recurrent model have not improved much,
and it also slow down the testing of the model. In addition, the improvement
is only effective when the attention layer is after the recurrent layer, and
the others have a slight decrease. The best recurrent model with attention is
LSTM-Attention, its accuracy and F1 are 89.43% and 89.32%, which is about 0.5%
improvement compared to the best recurrent model. However, compared with LSTM-
Attention, T-E-GRU has an improvement of about 0.1%. In other word, T-E-GRU is
the best in experimental model, with 90.09% accuracy and 90.07% F1. The
drawback is that T-E—GRU costs 6% more test time than LSTM-Attention.

In general, we can find on DMSC_V2:

  1. 1.

The simple RNN got the best performance in recurrent models.

  2. 2.

LSTM-Attention is the best when recurrent model is combined with attention,but
it also leads to more time cost.

  3. 3.

T-E-GRU achieved the best performance in the models of mentioned above, with
90.09% accuracy and 90.07% F1. And the time required for model testing is very
similar to LSTM-Attention.

###  4.2 YF_DianPing Dataset

Dataset. YF_DianPing DianpingData  is customer reviews of restaurant, which
has more characters and more complicated than DMSC_V2. It contains the user
reviews crawled from a famous Chinese online review webset DianPing.com,
including the 3,605,300 reviews of 510,071 users. After eliminating the NaN
data, we randomly selected 80000 1-points reviews as positive samples, and
80,000 1-points reviews as negative samples. We divide them into two parts,
80% are used as the training set and 20% are used as the validation set. Then
we obtained the test set by taking 30,000 4-points reviews as positive samples
and 30,000 2-points reviews with as negative samples.

Data preprocessing. The processing of punctuation, CWS, the padding/
truncating and pre-trained word embedding model are the same as Section 4.1.
After tokenization, the longest sample has 1386 tokens and 96.2% of the
samples’ tokens are less than 400. So we align the input sequence to 400 by
using the same method as in Section 4.1. For word representation, we use
150,000 high-frequency words, which covering 86.0% of the words in the
training set. Due to the complexity of the words used in the data set, using
of 260,000 high-frequency words only covers 86.5%, so it is more appropriate
to use 150,000 high-frequency.

Train Details. On YF_DianPing Dataset, we did the same experiment and only
modified the dataset. The various hyperparameters, loss functions, optimizers,
etc. of each model are the same as Section 4.1.

Results and Analysis

Table 2: The performance of various models on the YF_DianPing Method | Accuracy | F1 | Test Time(ms)
---|---|---|---
RNN RNN-first  | 87.95% | 88.38% | 3.0484
LSTM hochreiter1997LSTM  | 88.40% | 88.91% | 7.4615
GRU cho2014learningGRU  | 88.83% | 89.44% | 7.5326
Bi-RNN | 89.23% | 89.51% | 5.5511
Bi-LSTM | 87.90% | 87.50% | 14.6120
Bi-GRU | 89.55% | 89.94% | 5.3284
RNN-Attention | 88.92% | 89.46% | 3.6268
LSTM-Attention | 90.45% | 90.81% | 7.8170
GRU-Attention | 88.55% | 89.07% | 7.7112
Attention-RNN | 87.29% | 88.12% | 3.2191
Attention-LSTM | 87.92% | 88.54% | 8.2530
Attention-GRU | 88.06% | 88.64% | 8.1179
BiRNN-Attention | 89.54% | 89.86% | 5.6455
BiLSTM-Attention | 87.70% | 88.20% | 15.5177
BiGRU-Attention | 88.88% | 89.36% | 8.6010
T-E-GRU | 90.76% | 90.98% | 8.0009

The results of various models on the DianPing test set are shown in Table 2.
The test set uses 2 points and 4 points review data, while the training set
uses 1 point and 5 points review data, so there may be more ambiguous texts in
the test set, resulting in poor performances. On this test set, Bi-GRU has the
best performance in recurrent model, with accuracy and F1 reaching 89.55% and
89.94% respectively. Compared with the best-performing model RNN in Douban
data, Bi-GRU has an improvement of about 3% on both metrics. LSTM-Attention
achieves the best performances in attention-based recurrent model with 90.45%
accuracy and 90.81% F1. T-E-GRU is slightly better than LSTM-Attention in both
F1 and accuracy. Although not much improvement, T-E-GRU still achieved the
best performance among all tested models with 90.76% accuracy and 90.98% F1.
Similar to the results on Doublan, T-E-GRU still takes about 2% more test time
than LSTM-Attention on this dataset.

In general, We can find:

  1. 1.

RNN had a good performance on Douban before, in YF_DianPing it did not achieve
good performance due to the longer tokens. The Bi-GRU has achieved the best
performance in recurrent model.

  2. 2.

LSTM-Attention is still the best performance in recurrent model with
attention.

  3. 3.

Although not much improvement, T-E-GRU still achieved the best performance
among all tested models with 90.76% accuracy and 90.98% F1. This also shows
that T-E-GRU can still perform well compared to other models in the face of a
more complex test set than training set. And its test time is close to LSTM-
Attention.

###  4.3 Online_shopping_10 cats Dataset

Dataset. This dataset comes from various Chinese E-commerce platforms, with a
total of more than 60,000 comment data, including 30,000 positive and negative
comments. We ignored their category tags and only used sentimental tags. We
divide them into training set, validation set, and test set according to the
ratio of 6:1:3. This dataset is smaller than the previous two datasets, and we
use it to test the performance of our model on small-scale datasets.

Data preprocessing. Compared with the previous two datasets, it is a small-
scale moderate-length dataset. After tokenization, the longest sample has 1502
tokens and 99.0% of the samples’ tokens are less than 200. So we align the
input sequence to 200. For word representation, we use 200,000 high-frequency
words, which can cover 94.9% of the word needed in the training set. Using all
words in pre-trained word embedding model is only about 0.2% improvement.

Train Details. In the experiments of Online_shopping_10 cats dataset, the
various hyperparameters, loss functions, optimizers, etc. of each model are
the same as Section 4.1. However, since the size of this data set is small,
many models use 100 epochs can not complete the training. To solve this, we
double the initial learning rate and set the training epochs to 300. The
performances of each model are shown in Table 3, where 300 epochs of training
are marked after the model name.

Results and Analysis.

Table 3: The performance of various models on the Online_Shopping Method | Accuracy | F1 | Test Time(ms)
---|---|---|---
RNN | 89.68% | 89.93% | 1.5092
LSTM | 90.21% | 90.06% | 3.7045
GRU | 90.08% | 90.36% | 3.7701
Bi-RNN | 90.10% | 90.07% | 2.6567
Bi-LSTM | 90.10% | 90.10% | 7.1392
Bi-GRU | 90.34% | 90.13% | 2.8293
RNN-Attention | 90.07% | 90.10% | 1.9176
LSTM-Attention | 91.86% | 91.76% | 4.0548
GRU-Attention-300 | 90.93% | 90.96% | 3.9876
Attention-RNN | 90.68% | 90.72% | 1.8351
Attention-LSTM-300 | 90.91% | 90.91% | 4.1661
Attention-GRU-300 | 89.56% | 89.65% | 4.1363
BiRNN-Attention-300 | 92.32% | 92.38% | 3.0296
BiLSTM-Attention-300 | 90.28% | 90.16% | 7.7145
BiGRU-Attention-300 | 91.09% | 91.05% | 3.2287
T-E-GRU | 92.92% | 93.05% | 4.3890

The results of various models on the Online_shopping_10 cats test set are
shown in Table 3. On this test set, various recurrent models have similar
performance. The accuracy and F1 of recurrent models are around 90% and 90%.
BiRNN-Attention has the best performance in recurrent model with attention,
with accuracy and F1 reaching 92.32% and 92.38% respectively. Compared with
the recurrent model, BiRNN-Attention has an improvement of about 2%. In
contrast, LSTM-Attenion, which performed well before, was defeated on this
data set. Compared with BiRNN-Attention, T-E-GRU has about 0.6% and 1%
improvement in accuracy and F1 respectively, and it takes 40% more in test
time.

In general, we can find:

  1. 1.

On this data set, Bi-GRU still achieved the best performance in recurrent
model with 90.34% accuracy and 90.13% F1. But compared to other recurrent
models, the improvement of Bi-GRU is ambiguous.

  2. 2.

BiRNN-Attention has achieved the best results in recurrent model with
attention. Compared with recurrent model, BiRNN-Attention has about 2%
improvement in accuracy and F1.

  3. 3.

T-E-GRU still gains the best accuracy and F1, which are 92.92% and 93.05%
respectively. And it only takes 40% more time than BiGRU-Attention.

###  4.4 Ablation Study

We conducted ablation experiments to better understand the contributions and
the hyperparameter setting of T-E-GRU. We trained the network and tested it on
the DMSC_V2, YF_DianPing and Online_shopping datasets.

####  4.4.1 Replacing GRU with other recurrent layer

Table 4: The performance of replacing GRU with other recurrent layers Dataset | Method | Accuracy | F1 | Test Time(ms)
---|---|---|---|---
DMSC_V2 | T-E-RNN | 89.25% | 89.30% | 1.5390
T-E-LSTM | 89.75% | 89.88% | 2.9321
T-E-BiRNN | 89.26% | 89.30% | 2.1434
| T-E-BiLSTM | 89.81% | 89.74% | 4.7107
| T-E-BiGRU | 89.77% | 89.85% | 2.2185
| T-E-GRU | 90.09% | 90.07% | 2.6987
YF_DianPing | T-E-RNN | 88.91% | 89.42% | 3.4685
T-E-LSTM | 90.25% | 90.70% | 8.0919
T-E-BiRNN | 88.94% | 89.40% | 5.9280
| T-E-BiLSTM | 90.63% | 90.92% | 15.4538
| T-E-BiGRU | 90.73% | 90.89% | 6.3677
| T-E-GRU | 90.76% | 90.98% | 8.0009
Online_shopping | T-E-RNN | 91.16% | 91.01% | 2.1586
T-E-LSTM | 92.43% | 92.59% | 4.4119
T-E-BiRNN | 91.40% | 91.49% | 3.3178
| T-E-BiLSTM | 92.41% | 92.47% | 8.2653
| T-E-BiGRU | 92.38% | 92.60% | 3.3085
| T-E-GRU-300 | 92.92% | 93.05% | 4.3890

In this section, we tried replacing GRU with other recurrent models (RNN,
LSTM). The experimental results are shown in Table 4. On DMSC_V2, T-E-GRU has
achieved the best performance compared to other models, but it has only a
slight improvement in accuracy and F1. On YF_DianPing, T-E-GRU only has less
than 1% improvement in accuracy, F1 than other models. T-E-GRU is still the
best accuracy and F1, which are 90.76% and 90.98% respectively. On
Online_shopping, T-E-GRU still obtain the best accuracy and F1, In general,
GRU is more suitable for combining with transformer-encoder.The multiple of
the time spent is usually positively correlated with the number of tokens.

####  4.4.2 Adjustment of hyperparameters in T-E-GRU

(a) DMSC_V2. (b) YF_DianPing (c) Online_shopping_10 cats Dataset

Figure 3: Hidden size of Transformer-Encoder

Firstly, we tried to adjust the hidden size of the transformer-encoder. The
results on three datasets mentioned in Section 4.1, 4.2, 4.3 are shown in
Figure 3. The accuracy, F1, and Test Time are introduced in Section 4.1. From
the results, we find that the different hidden sizes of transformer-encoder
have little effect on Test Time, mostly less than 0.5(ms). For accuracy and
F1, DMSC_V2 and Online_shopping have the best performance in 2048, while 2048
and 3072 on YF_DianPing have very similar performance.

(a) DMSC_V2. (b) YF_DianPing (c) Online_shopping_10 cats Dataset

Figure 4: Heads of Transformer-Encoder

Then we adjusted the number of heads in transformer-encoder. On the same
datasets, the experiment results are shown in Figure 4. In fact, the number of
heads from 1 to 4 has a less than 0.5(ms) effect on Test Time, and it is very
slight. On DMSC_V2 and Online_shopping, the number of heads 2 can achieve the
best accuracy and F1. On YF_DianPing, the number of heads 1 and 2 are similar
in accuracy and F1.

(a) DMSC_V2. (b) YF_DianPing (c) Online_shopping_10 cats Dataset

Figure 5: Dropout of Transformer-Encoder & GRU

Finally, we adjusted the dropout of transformer-encoder and GRU on the same
datasets. Since dropout does not affect the complexity of the model, we
removed Test Time, and the result is shown in Figure 5. As the result shows,
on DMSC_V2 and YF_DianPing, dropout of 0.4 is the best in accuracy and F1,
while on Online_shopping, 0.3 is the best.

In general, we can find:

  1. 1.

GRU is more suitable for combination with transformer-ecoder than LSTM and
RNN.

  2. 2.

For T-E-GRU, adjusting the hidden size of transformer-encoder, heads of
transformer-encoder and dropout has little effect on Test Time, while the
accuracy and F1 are different depending on the datasets.

##  5 Conclusion

On the three real Chinese review datasets which have not been strictly
cleaned, we have retained punctuation with the ability to clauses as
preprocessing. Then We compared our proposed T-E-GRU with various recurrent
model and recurrent model with attention. On the test set, compared to the
best performance in other models, T-E-GRU has 1% improvement in accuracy and
F1.

## References

  * [1] Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio.  Neural machine translation by jointly learning to align and translate.  arXiv preprint arXiv:1409.0473, 2014.
  * [2] Yoshua Bengio, Réjean Ducharme, Pascal Vincent, and Christian Janvin.  A neural probabilistic language model.  The journal of machine learning research, 3:1137–1155, 2003.
  * [3] Yoshua Bengio, Patrice Simard, and Paolo Frasconi.  Learning long-term dependencies with gradient descent is difficult.  IEEE transactions on neural networks, 5(2):157–166, 1994.
  * [4] Liang Bin, Liu Quan, Xu Jin, Zhou Qian, and Zhang Peng.  Aspect-based sentiment analysis based on multi-attention cnn.  Journal of Computer Research and Development, 54(8):1724, 2017.
  * [5] William B Cavnar, John M Trenkle, et al.  N-gram-based text categorization.  In Proceedings of SDAIR-94, 3rd annual symposium on document analysis and information retrieval, volume 161175. Citeseer, 1994.
  * [6] Peng Chen, Zhongqian Sun, Lidong Bing, and Wei Yang.  Recurrent attention network on memory for aspect sentiment analysis.  In Proceedings of the 2017 conference on empirical methods in natural language processing, pages 452–461, 2017.
  * [7] Xinchi Chen, Xipeng Qiu, Chenxi Zhu, Pengfei Liu, and Xuan-Jing Huang.  Long short-term memory neural networks for chinese word segmentation.  In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 1197–1206, 2015.
  * [8] Jianpeng Cheng, Li Dong, and Mirella Lapata.  Long short-term memory-networks for machine reading.  arXiv preprint arXiv:1601.06733, 2016.
  * [9] Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio.  Learning phrase representations using rnn encoder-decoder for statistical machine translation.  arXiv preprint arXiv:1406.1078, 2014.
  * [10] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  Bert: Pre-training of deep bidirectional transformers for language understanding.  arXiv preprint arXiv:1810.04805, 2018.
  * [11] Xiaowen Ding, Bing Liu, and Philip S Yu.  A holistic lexicon-based approach to opinion mining.  In Proceedings of the 2008 international conference on web search and data mining, pages 231–240, 2008.
  * [12] Jeffrey L Elman.  Finding structure in time.  Cognitive science, 14(2):179–211, 1990.
  * [13] Ronen Feldman.  Techniques and applications for sentiment analysis.  Communications of the ACM, 56(4):82–89, 2013.
  * [14] Jie Gao.  Chinese sentiment classification model based on pre-trained bert.  In 2021 2nd International Conference on Computers, Information Processing and Advanced Education, pages 1296–1300, 2021.
  * [15] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [16] Dichao Hu.  An introductory survey on attention mechanisms in nlp problems.  In Proceedings of SAI Intelligent Systems Conference, pages 432–448. Springer, 2019.
  * [17] Changning Huang and Hai Zhao.  Chinese word segmentation: A decade review.  Journal of Chinese Information Processing, 21(3):8–20, 2007.
  * [18] MI Jordan.  Serial order: a parallel distributed processing approach. technical report, june 1985-march 1986.  Technical report, California Univ., San Diego, La Jolla (USA). Inst. for Cognitive Science, 1986.
  * [19] John Lafferty, Andrew McCallum, and Fernando CN Pereira.  Conditional random fields: Probabilistic models for segmenting and labeling sequence data.  2001\.
  * [20] Shen Li, Zhe Zhao, Renfen Hu, Wensi Li, Tao Liu, and Xiaoyong Du.  Analogical reasoning on chinese morphological and semantic relations.  arXiv preprint arXiv:1805.06504, 2018.
  * [21] Xiaoya Li, Yuxian Meng, Xiaofei Sun, Qinghong Han, Arianna Yuan, and Jiwei Li.  Is word segmentation necessary for deep learning of chinese representations?  arXiv preprint arXiv:1905.05526, 2019.
  * [22] Jun Liang, Yumei Chai, Huibin Yuan, ML Gao, and Hongying Zan.  Polarity shifting and lstm based recursive networks for sentiment analysis.  Journal of Chinese Information Processing, 29(5):152–159, 2015\.
  * [23] Shiwei Liu, Decebal Constantin Mocanu, Yulong Pei, and Mykola Pechenizkiy.  Selfish sparse rnn training.  arXiv preprint arXiv:2101.09048, 2021.
  * [24] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean.  Distributed representations of words and phrases and their compositionality.  arXiv preprint arXiv:1310.4546, 2013.
  * [25] Tomáš Mikolov, Wen-tau Yih, and Geoffrey Zweig.  Linguistic regularities in continuous space word representations.  In Proceedings of the 2013 conference of the north american chapter of the association for computational linguistics: Human language technologies, pages 746–751, 2013.
  * [26] Volodymyr Mnih, Nicolas Heess, Alex Graves, and Koray Kavukcuoglu.  Recurrent models of visual attention.  arXiv preprint arXiv:1406.6247, 2014.
  * [27] Jeffrey Pennington, Richard Socher, and Christopher D Manning.  Glove: Global vectors for word representation.  In Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pages 1532–1543, 2014.
  * [28] Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer.  Deep contextualized word representations.  arXiv preprint arXiv:1802.05365, 2018.
  * [29] Lawrence Rabiner and Biinghwang Juang.  An introduction to hidden markov models.  ieee assp magazine, 3(1):4–16, 1986.
  * [30] Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever.  Improving language understanding by generative pre-training.  2018\.
  * [31] David E Rumelhart, Geoffrey E Hinton, and Ronald J Williams.  Learning internal representations by error propagation.  Technical report, California Univ San Diego La Jolla Inst for Cognitive Science, 1985.
  * [32] Xiaoming Shi and Ran Lu.  Attention-based bidirectional hierarchical lstm networks for text semantic classification.  In 2019 10th International Conference on Information Technology in Medicine and Education (ITME), pages 618–622. IEEE, 2019.
  * [33] Fuhong Tang and Kwankamol Nongpong.  Chinese sentiment analysis based on lightweight character-level bert.  In 2021 13th International Conference on Knowledge and Smart Technology (KST), pages 27–32. IEEE, 2021.
  * [34] Shikhar Vashishth, Shyam Upadhyay, Gaurav Singh Tomar, and Manaal Faruqui.  Attention interpretability across nlp tasks.  arXiv preprint arXiv:1909.11218, 2019.
  * [35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  arXiv preprint arXiv:1706.03762, 2017.
  * [36] Shitao Wang, Jiangfeng Li, and Defeng Hu.  Bigru-multi-head self-attention network for chinese sentiment classification.  In Journal of Physics: Conference Series, volume 1827, page 012169\. IOP Publishing, 2021.
  * [37] Zheng Xiao and PiJun Liang.  Chinese sentiment analysis using bidirectional lstm with word embedding.  In International Conference on Cloud Computing and Security, pages 601–610. Springer, 2016.
  * [38] Yushi Yao and Zheng Huang.  Bi-directional lstm recurrent neural network for chinese word segmentation.  In International Conference on Neural Information Processing, pages 345–353. Springer, 2016.
  * [39] Lei Zhang, Shuai Wang, and Bing Liu.  Deep learning for sentiment analysis: A survey.  Wiley Interdisciplinary Reviews: Data Mining and Knowledge Discovery, 8(4):e1253, 2018.
  * [40] Yongfeng Zhang, Min Zhang, Yiqun Liu, Shaoping Ma, and Shi Feng.  Localized matrix factorization for recommendation based on matrix block diagonal forms.  In Proceedings of the 22nd international conference on World Wide Web, pages 1511–1520, 2013.
  * [41] Haoyi Zhou, Shanghang Zhang, Jieqi Peng, Shuai Zhang, Jianxin Li, Hui Xiong, and Wancai Zhang.  Informer: Beyond efficient transformer for long sequence time-series forecasting.  arXiv preprint arXiv:2012.07436, 2020.
  * [42] Zhengshuai Zhu, Yanquan Zhou, and Shuhao Xu.  Transformer based chinese sentiment classification.  In Proceedings of the 2019 2nd International Conference on Computational Intelligence and Intelligent Systems, pages 51–56, 2019.
  * [43] Huang Zou, Xinhua Tang, Bin Xie, and Bing Liu.  Sentiment classification using machine learning techniques with syntax features.  In 2015 International Conference on Computational Science and Computational Intelligence (CSCI), pages 175–179. IEEE, 2015.