#  ST-KeyS: Self-Supervised Transformer for Keyword Spotting in Historical
Handwritten Documents

Sana Khamekhem Jemni  Sourour Ammar  Mohamed Ali Souibgui  Yousri Kessentini
Abbas Cheddad  Digital Research Center of Sfax, B.P. 275, Sakiet Ezzit, 3021
Sfax, Tunisia  SM@RTS : Laboratory of Signals, systeMs, aRtificial
Intelligence and neTworkS, Sfax, Tunisia  MIR@CL: Multimedia, InfoRmation
systems and Advanced Computing Laboratory  Computer Vision Center, Computer
Science Department, Universitat Autònoma de Barcelona, Spain  Department of
Computer Science, Blekinge Institute of Technology, Karlskrona, Sweden

###### Abstract

Keyword spotting (KWS) in historical documents is an important tool for the
initial exploration of digitized collections. Nowadays, the most efficient KWS
methods are relying on machine learning techniques that require a large amount
of annotated training data. However, in the case of historical manuscripts,
there is a lack of annotated corpus for training. To handle the data scarcity
issue, we investigate the merits of the self-supervised learning to extract
useful representations of the input data without relying on human annotations
and then using these representations in the downstream task. We propose ST-
KeyS, a masked auto-encoder model based on vision transformers where the
pretraining stage is based on the mask-and-predict paradigm, without the need
of labeled data. In the fine-tuning stage, the pre-trained encoder is
integrated into a siamese neural network model that is fine-tuned to improve
feature embedding from the input images. We further improve the image
representation using pyramidal histogram of characters (PHOC) embedding to
create and exploit an intermediate representation of images based on text
attributes. In an exhaustive experimental evaluation on three widely used
benchmark datasets (Botany, Alvermann Konzilsprotokolle and George
Washington), the proposed approach outperforms state-of-the-art methods
trained on the same datasets.

###### keywords:

Keyword spotting , masked autoencoders , self-supervised learning , visual
transformers , siamese neural networks , PHOC embedding.

††journal: Pattern Recognition

##  1 Introduction

Many digitized historical handwritten documents are hard to access due to the
lack of suitable indexing and retrieval tools. One solution is to recognize
the image by transforming it to text with an automatic tool, or manually.
However, fully manual transcription is time-consuming and expensive. Also,
current optical character recognition (OCR) systems are mostly efficient for
modern printed documents [1, 2]. But, these systems suffer difficulties in the
case of a historical manuscript that usually contains many types of
degradation due to bad paper quality, writing style variations, ink flow,
shadows, non-uniform lighting, stains, etc.

Another solution to handle the non-indexed documents is the word-matching
process, which is based on a low-level matching known as $word$ $spotting$
[3]. It is closely associated with content-based image retrieval, since it
searches for a word in a set of non-indexed documents using the query image
content as the single source of information. As a result, the system will
return a ranked list of document word images to the user according to their
similarity to the desired searched word image. Ultimately, word spotting can
be defined as the process of identifying positions on a document image that
have a high potential to correspond to an instance of a query word image,
without recognizing it explicitly.

In the literature, two distinct strategies for word spotting appeared
depending on the search space, which could be either a set of
isolated/segmented word images (segmentation-based approach) or a full
document image (segmentation-free approach). In this work, we assume that
document images have been segmented into isolated word images serving
thereafter for matching the query word image, following previous studies such
as [4, 5, 6, 7] Localizing and retrieving word images has been an active field
of research, with the goal of finding the more effective word representations
that give the most meaningful distances between word images. Early approaches
were using the classic learning-free techniques that rely on expert designed
feature embedding (handcrafted features) [6, 8]. These methods tend to be
immediately applied since they do not rely on a learning stage.

Later, and following the success of other computer vision tasks, machine
learning-based techniques (in particular convolutional neural networks (CNNs))
are now dominating the field of word retrieval [9, 10, 11, 12, 13, 14]. But,
despite the significant improvement in the performance of these models over
classical handcrafted techniques, they have their associated shortcomings.
First, CNNs operate on regular grids and use the same convolutional filter to
extract features from handwritten word images, making this technique sensitive
to rotation. Second, CNNs fail to capture relevant features for long-range
dependencies, as they are more suited to extract low-level spatial information
from images.

With the recent success of transformers in natural language processing (NLP)
[15, 16], their application in computer vision tasks (such as object detection
[17], image recognition [18], question answering [19], image restoration [20],
handwritten text recognition (HTR) [21, 22], named entity recognition [23],
etc.) has also lately been getting more attention. The self-attention
mechanism that has been proposed in [15] allows capturing contextual feature
interactions information. This use of local knowledge combined with the
information of the global long-range spatial arrangement is beneficial for an
efficient keyword spotting (KWS) model.

However, the main issue of transformers is that they are data-hungry.
Nonetheless, huge annotated datasets are hard to obtain and labeling large
amounts of data is work intensive and can be very expensive, while unlabeled
data is much more abundant. Thus, the need to build models that benefit from
the unlabeled data in addition to the annotated data, or even omitting the use
of any annotated data becomes more appropriate in this situation. In this
regard, self-supervised techniques based on ViT (Vision Transformer) models
have recently shown significant results.

Motivated by this success in computer vision, such as image classification and
object detection [24, 25], image retrieval [26] and speech recognition [27]
tasks, we propose in this paper an end-to-end keyword spotting approach in
handwritten documents which is based on a self-supervised technique and makes
use of masked autoencoders with the self-attention mechanism. To the best of
our knowledge, this is the first work that proposes a self supervised learning
paradigm of transformer based model in the context of word spotting in
historical document images. Our framework is built in two stages. The first
stage is the pretraining which is designed for learning useful representations
from the unlabeled data, using a masked encoder-decoder architecture, in a
self-supervised way. The second stage is the fine-tuning which is devoted to
efficiently extracting relevant features from the labeled word image. In the
fine-tuning stage, the pretrained encoder is integrated to a siamese neural
network (SNN) and is fine-tuned using a few labeled data to improve feature
embeddings from the input text images. Then, the resulting model is used as a
core component of our word spotting framework by including contextual
information extracted from the word text. To achieve this, we align the visual
representations extracted by our encoder with the pyramidal histogram of
characters (PHOC) representation. To demonstrate the effectiveness of the
proposed method, we conduct several experiments using various training
conditions and several public image databases.

The overall contributions of this paper can be summarized as follows:

  * 1.

To the best of our knowledge, we present the first self-supervised approach
for the goal of keyword spotting in handwritten text images, composed of
pretraining and fine-tuning stages. The approach is based on vision
transformers in an encoder-decoder fashion, without any dependency on CNNs.

  * 2.

An effective pretext task was learned during pretraining to extract the most
useful representations for keyword spotting without the need of any labeled
data.

  * 3.

Then, a two-stage downstream task based on siamese neural networks and PHOC
attributes is used to further promote the representation and makes it more
powerful in retrieving the best matching images of a given query image.

  * 4.

Extensive comparative experiments are achieved to validate the efficiency of
our proposed method, involving three handwritten word images datasets. We
demonstrate that our proposed method can be generalized across different
databases and languages. We show also the effectiveness of our method to deal
with data variability as well as data scarcity issues.

The rest of this paper is organized as follows. In Section 2 we provide a
review of prior works on keyword spotting for segmented documents. Then we
introduce our proposed model in Section 3. After that, experimental results
and comparisons with existing methods will be described in Section 4. Finally,
in Section 5 we draw the conclusions and we propose open challenges for future
research directions.

##  2 Related work

Keyword spotting in handwritten documents has drawn the interest of the
document analysis research community over the last decades [3] and it has
still been challenging given the complexity of historical documents (diverse
scripts, various writing styles, diverse noise, etc). In the literature, there
are several successful efforts that focus on different aspects of the word
spotting issues like the data modality (printed, handwritten or scene text),
method type (segmentation-based and segmentation free), and the embedding
(representation) type. In the following sub-sections, we categorize the
related methods into two main families depending on the used representations:
learning-free and learning-based techniques.

###  2.1 Learning-free representations

Classical KWS methods were learning-free, different methods were built to find
the best handcrafted matching features, or representations. Earlier works
considered the handwritten word images as a temporal sequence to build a
variable embedding representation. Most of these methods are based on profile
features [28] by computing each word image’s column using diverse statistics
at the pixel level. Within this scope, the Dynamic Time Warping (DTW)
algorithm had been employed to match representations having variable lengths
inspired by its usefulness for sequence matching problems in speech. In [29,
30], the authors combined the DTW with the profile features (vertical profile,
upper and lower word profile, and background to ink transitions) for better
accuracy and faster retrieval. However, these simple structural features were
leading to unsatisfactory accuracy. Thus, statistical-based features,
especially local gradient ones such as SIFT (Scale-invariant feature
transform) [31] and HOG (Histogram of oriented gradients) [32] were employed
for word spotting.

In [33, 34], the authors propose a Bag Of Words (BOW) method which uses the
SIFT descriptors to extract the local features, then projects them to a topic
space that conserves the lexical content of the word images. A similar
approach with an addition of the corner detector features was also introduced
in [35]. Another approach that uses the projections of oriented gradients as
descriptors was proposed in [36]. A sequence of descriptors based on the
combination of a zoning scheme and an appearance descriptor (or, modified
Projections of Oriented Gradients) was also used in [6] to represent the word
images. In [37, 38], document images were represented with a grid of HOG
descriptors, then the document regions that are most similar to a query word
are located in a sliding-window fashion. After that, a second stage of re-
ranking the best retrieved regions using a more discriminative BOW
representation was applied. In [4], a representation to embed the word image
to its corresponding text label in the same space using the developed PHOC was
introduced. Additionally, recently, some graph based approaches have been
introduced. In [39], the text images were represented with a graph structure,
then graph matching was applied for the spotting.

Despite the simplicity of the learning-free approaches and the ability to be
adapted and applied in different domains, their results are still
unsatisfactory. Thus, learning-based approaches have since been in vogue.

###  2.2 Learning-based representations

With the advances in machine learning, especially in the deep learning field,
features are now learned within the model instead of handcrafting them.
Nowadays, machine learning based word spotting models are learning the
features using labeled data. Most of the developed approaches within this
strategy are using CNNs. In [40], a CNN based model is employed to detect
words regions in natural images, then it recognizes these detected words.
Another model called PHOCNet [41] was proposed to embed the image features
extracted by the CNN layers into the PHOC representation. The representations
are calculated by applying a sigmoid on the final fully connected layer. This
latter work was extended in [42, 9] to improve the spotting accuracy. Another
PHOC based model was introduced in [43], where instead of a distance-based
matching of the retrieved words, a probabilistic ranking was used for better
performance. In [5], a model called HWNet was introduced to match image
collections containing handwritten content, where the representations were
learned using a CNN. The HWNet representations were also used in [44, 45] by
embedding them into the word attribute space. This was done by training a
classifier to project both image and textual attributes to a common subspace.
In [11], a triplet loss based CNN approach was proposed to learn the
representations by reducing the distance of the anchor image to a similar
(positive) image while enlarging the distance to a different (negative) word
image.

Other approaches were proposed based on graphs, for instance, [7] learned the
representations by a Graph Neural Network (GNN). A message passing neural
network was employed to capture the graph structure, that is used for the
distance computation.

###  2.3 Self-Supervised Learning in document processing/analysis

Learning-based approaches are effective when a large amount of labeled data is
available. However, with the continual growth of deep learning architectures,
they start overfitting on the usual datasets and requiring more samples, which
is a challenging problem. Thus, a recent development in self-supervised
learning [17, 25, 24], especially with the rise of transformers architectures
[15, 18], is now appearing as a solution. Self-supervised methods aim to
benefit from a huge amount of unlabeled data that can be added to the labeled
datasets for training. Nowadays, several document analysis papers are proposed
following this strategy. These approaches were developed for instance in
document understanding in terms of classification, layout analysis or entity
extraction [46, 47, 48, 49] with different pretraining objectives (text-image
matching, text-image alignment, masked visual-language modeling, document
reconstruction, etc). These models were proposed for optical text images and
using an OCR for the pretraining. Thus, their learned representation can not
be used for fine-tuning our model on handwritten KWS. Other approaches were
developed for handwritten/scene text recognition by learning a representation
that considers the text image as a sequence of characters in a sequence-to-
sequence contrastive learning fashion. In [50], the authors first applied
transformation on each unlabeled word. Then, the feature maps are divided into
different instances, over which the contrastive loss is computed, where from
each image several positive pairs and multiple negative examples are
extracted. However, the segmentation of words was not accurate due to the use
of unlabeled data, which makes the method learns a representation of a
sequence of ”word parts” rather than the actual characters. In [51], a similar
approach was proposed by concatenating different unlabeled words to produce
two views, then their aligned features are used with a contrastive loss to
pull together the positive samples and push apart the negative samples.
However, concatenating unlabeled words can cause the consideration of positive
words as negatives. More recently, a self-supervised approach for HTR using
the masking-recovering strategy with the generative models was proposed [22].
The method applies different degradation on the unlabeled word images
(masking, blurring and background noise) and then learns to reconstruct the
original clean image as a pretraining task. It was shown that this method
overcomes the contrastive learning drawbacks of requiring large batch sizes
and large data points while learning more effective and robust representation,
especially for fine-tuning with fewer data. Another method that combines
contrastive learning based pretraining with the masking based one was also
proposed in [52].

##  3 Proposed method

In this section, we present the proposed framework for word spotting in
handwritten document images. We refer to this framework as ST-KeyS: Self-
supervised Transformer for Keyword Spotting. The idea is to benefit from the
big amount of unlabeled data to learn word image representation in a self-
supervised way. Then further improve and align these representations using the
available labeled data. Fig. 1 illustrates the overall proposed framework,
similar to other recent works in self-supervised representation learning, our
proposed method consists of two phases:

  * 1.

Pretraining phase: a self-supervised pretraining using a masked vision
transformer autoencoder is performed to extract useful representations from
the unlabeled text images.

  * 2.

Fine-tuning phase: a downstream task based on siamese neural networks used to
learn deep representations (step1) of the word image followed by PHOC
embedding enabling the extraction of contextual information from the word
(step 2).

Figure 1: Pipeline of the proposed ST-KeyS framework. It consists of two
stages: a pretraining stage and a fine-tuning stage.

In the following sub-sections, we first describe the used masked autoencoder
architecture for self-supervised word representation learning. Second, we
describe the used model based on the SNN architecture and PHOC representations
for the down-stream task.

###  3.1 Pretraining phase: Learning deep representation using a masked
autoencoder model

The pretraining phase is used to learn a word representation model in an
unsupervised fashion. Inspired by the work [24], we propose to use a masked
autoencoder based on vision transformer. Fig. 2 illustrates the architecture
of the used masked encoder-decoder. Same as the ViT model [18], each input
image $x\in R^{H\times W\times C}$ ($H$, $W$ and $C$ denote respectively the
height, width, and number of channels of $x$) is split into a set of $n$ non
overlapping regular patches of size $p\times p\times C$. Then a fraction of
75% of these patches are randomly masked according to a uniform distribution.
The input image is then represented by only the remaining visible patches at
the input of the encoder.

Figure 2: Proposed masked encoder-decoder model for word representation
learning.

Different from conventional autoencoders, the used masked encoder-decoder
operates in an asymmetric fashion [24] enabling the encoder to operate on only
the partial observed signal (without the masked patches) and the decoder to
rebuild the full image based on the representation given by the encoder and
the masked patches. As shown in Fig. 2, the encoder takes as input the visible
patches with their positional information and maps them into a latent
representation, while the decoder reconstructs the masked pixels based on this
latent representation.

The Encoder. The used encoder architecture is a vision transformer [18] that
operates only on visible patches. First, the visible patches are embedded
through a linear projection operation into a patch embedding set (denoted
$E_{i}$). In order to allow the model to capture the spatial structure of the
2D image, each patch embedding in $E_{i}$ is then associated with its
positional information within the original image. The resulting embedding set
$E^{\prime}_{i}$ is then passed as input to the encoder. After that, a number
of transformer blocks are employed to map this embedding $E^{\prime}_{i}$ into
a latent representation (output patch embedding set referred as
$E^{\prime}_{O}$). These encoder’s blocks have the same structure as in [18].
Each block consists of multi-head self-attention and multi-layer perceptron
(MLP) alternating layers. The MLP network consists of a single layer with a
Gaussian Error Linear Unit (GELU) activation function. Layer normalization
(LayerNorm) is applied before each transformer block allowing to train
efficiently deep encoders [53], and we used residual connections after every
block. An overview of the encoder architecture used in our work is illustrated
in Fig. 3. It should be noted also that our encoder is applied only on visible
set of patches. In our case, it operates on only a fraction of 25 % of the
full set of patches.

Figure 3: Architecture of the Encoder used in our work.

The Decoder. The decoder is used to predict the pixel values for masked
patches and then reconstruct the input image based on the latent
representation provided by the encoder. The decoder takes as input the
representation of the encoded patches $E^{\prime}_{O}$ (visible patches) and
the masked patches with their positional information which provides their
arrangement location inside the image. Similar to the encoder architecture,
the decoder has also series of transformer blocks. The decoder produces a
vector of pixel values to represent each patch. All vectors are then reshaped
to reconstruct the image. Similar to [18], we use the mean squared error (MSE)
between the masked patches from the original and reconstructed images as the
loss function.

We note that the decoder is only used during the pretraining phase in order to
perform the image reconstruction and it will be discarded during the second
phase.

###  3.2 Fine-tunig phase: Proposed down-stream task for word spotting

The goal of our study is to propose a spotting method dealing with the query
by example scenario for KWS that can handle existing challenges in handwritten
documents such as writing style variations. For that, we introduce as the
downstream task a KWS method based on a siamese neural network (SNN)
architecture coupled with PHOC embedding. The SNN is a dedicated network
having two inputs and it is able to learn relevant representations of
handwritten images and therefore it allows distinguishing between similar and
non-similar pairs of images. While the SNN network is used to learn a
descriptor that is representing the image, the PHOC embedding is involved to
provide a contextual representation of the characters embedded inside the
word. Such process shall be a suitable way to benefit from the visual (image)
and language (text) modalities when producing the final representations.

####  3.2.1 Visual image representation

Once the masked autoencoder is pretrained in a self-supervised way on large
unlabeled dataset, the decoder is discarded and only the encoder part is
retained for the downstream task. Therefore, the pretrained encoder is used as
a backbone to build the SNN-transformer architecture as shown in Fig. 4. The
pretrained encoder is considered as a good starting point for the SNN model
because it has learned word image representation from unlabeled data during
the first pretraining phase without any human supervision or annotation. It
should be noted that during this phase, the encoder is applied to uncorrupted
(non-masked) images represented by the full set of their patches. As shown in
Fig. 4, the SNN-transformer architecture comprises two identical encoder
blocks, with shared weights, followed by a dense layer. The dense layer
involves a Linear layer with dropout and sigmoid as activation function.

Figure 4: The proposed SNN-transformer architecture. The H1 and H2 refer to
numerical feature vectors corresponding to Image 1 and Image 2, respectively.

Given two input images, the embedded features are propagated, each through an
encoder block. The SNN-transformer works in parallel on the two different
input vectors to compute comparable output vectors. These output vectors are
compared using a distance function, corresponding to the absolute values of
the descriptors subtraction operation, and then passed to the fully connected
layer with a sigmoid activation function to return a similarity measure
indicating whether or not the two images belong to the same class. The
learning process is designed to adjust the embedding to be representative and
enables thereafter the SNN-transformer to extract relevant features as
effectively as possible.

####  3.2.2 PHOC word embedding

Once the encoder has been fine-tuned firstly in a siamese fashion using a few
labeled data, we use this latter as the main component for the second
adaptation stage. This stage is performed to align the visually extracted
features by the previous stage with the text based features. Particularly, we
make use of the textual representation referred to as PHOC embedding. This
representation combines the histogram of characters in several spatial regions
in a pyramidal manner. Here, each feature indicates the presence or absence of
a specific character in a given spatial region.

Figure 5: The proposed PHOCNet-transformer architecture.

The PHOCNet-transformer’s pipeline is visualized in Fig. 5. It consists of a
single encoder block followed by a final dense layer. This last layer is a
Linear layer with dropout and Sigmoid activation function. Similar to [41], we
represent a word image using PHOC embedding with 2, 3, 4 and 5 levels. For
instance, within the second level, the PHOC captures the presence of a certain
character in the first or the second half of the word. Fig. 6 provides an
exemplary illustration of PHOC extraction from a given text string. This
yields a binary histogram having a size of 504. Additionally, we make use of
the 50 most frequent bigrams (level 2). Thus, by the use of the Latin alphabet
(lower case) with the ten digits, the resulting PHOC has a size of 604 which
is corresponding to the output size of the defined dense layer.

Learning PHOC representations allows the adjustment of the embedding to be
representative and enables the network to perform the spotting accurately.
Therefore, after having trained our encoder using the PHOC representations, we
make use of the layer preceding the final dense layer of this model to extract
feature vectors representing the word images. The extracted feature vectors
are then reduced to 400 dimensions using Principal Component Analysis (PCA),
which is popularly used as a dimensionality reduction technique. To perform
the word spotting task, we use the cosine distance metric which is calculated
from a pair of feature vectors that each corresponds to a word image.

Figure 6: Visualisation of the PHOC embedding extracted from a given text at
level 1, 2 and 3.

##  4 Experiments

In this section, we first describe the datasets used in our experiments. We
then outline the implementation details that are most relevant to our
approach. Lastly, we report our results and provide a comparison with state-
of-the-art methods.

###  4.1 Datasets

Four databases of handwritten documents are used in our experiments: while the
IAM off-line database is used to pretrain the model in a self-supervised way,
three other datasets are used to fine-tune and evaluate the KWS proposed
method. These three datasets are the Botany (BOT) dataset, Alvermann
Konzilsprotokolle dataset (AK), and the George Washington dataset (GW) which
are categorized as historical collections.

IAM dataset. The IAM dataset [54] involves forms of handwritten English text
useful for training and testing handwriting recognition models and writer
identification methods. IAM contains handwriting belonging to 657 different
writers which lead to a large variety of writing styles. This database
consists of 1.539 pages of scanned text from the Lancaster–Oslo/Bergen corpus
[55], leading to 115.320 isolated and labeled word images split into training,
validation and test partitions. Due to its large style variability, we have
used the IAM dataset for the self-supervised pretraining phase. A sample
document image is shown in Fig. 7.

Botany dataset (BOT). This dataset is a part of ICFHR 2016 Handwritten Keyword
Spotting Competition [56]. It consists of more than 100 different botanical
documents made between the year 1800 and 1850 by the then government of
British India. These documents are written in English and show some signs of
deterioration, especially fading as shown in Fig. 7. Variations in writing
style are noticeable, especially in scaling and intra-word variations.

Alvermann Konzilsprotokolle dataset (AK). This dataset is also a part of ICFHR
2016 Handwritten Keyword Spotting Competition [56]. It is a historical
document collection consisting of 18.000 pages of handwritten minutes of
official meetings in the central administration of the University of
Greifswald between the year 1794 and 1797. These documents are written in
German and show minor signs of degradation as shown in Fig. 7. This dataset
presents low variation in the writing style.

George Washington dataset (GW). The GW dataset is well known to the community
of word spotting. It consists of 20 pages of letters written by George
Washington and his secretaries in 1755. It contains 4.860 annotated word
images. Due to the small size of the dataset, there are no available standard
partitions or query selection for GW. Thus, a four-fold cross validation setup
is generally adopted by the majority of learning-based recent works. We have
adopted this setup using the same split as in [4]. For each test split, words
having at least two instances are selected as query images in a leave-one-out
fashion. A sample document image from the GW database is shown in Fig. 7.

(a) IAM dataset

(b) AK dataset

(c) BOT dataset

(d) GW dataset

Figure 7: Sample document images from the used datasets. The IAM database is
used during the self-supervised pretraining phase. The BOT, AK and GW datasets
are used for the second phase and thereby for word spotting evaluation
experiments. We notice that BOT, AK and GW datasets are categorized into
historical collection.

###  4.2 Evaluation protocol

For the word spotting task, the aim is to retrieve all instances of the query
words in a dataset’s partition. Given a query image, the database items are
sorted with respect to their similarity to the introduced query following the
protocol presented in [4]. In our work, we use all the words of the test set
as a query in a leave-one-out style. When performing keyword spotting, the
query image is removed from the dataset, and queries that have no extra
relevant words in the database are discarded. However, the remaining
irrelevant words are maintained in the test set to act as distractors in the
ranking process. This protocol is adopted in all our experiments using the
three evaluation sets, namely, BOT, AK and GW databases. Particularly, for the
GW dataset, we evaluate our proposed method in a four-cross validation fashion
to enable comparison with state-of-the-art systems. For the BOT and AK
datasets, there were three partitions of training sets: small, medium, and
large. Here we use the smallest partition for conducting experiments. Table 1
provides statistics on the databases used in our experiments. To assess the
performance of the performed method, we use the mean Average Precision (mAP)
as the evaluation measure.

Table 1: Statistics derived from the datasets used for evaluation.  Dataset |  Keywords |  Test |  Train
---|---|---|---
Botany (BOT) |  150 |  3380 |  1684
Alvermann Konzilsprotokolle (AK) |  200 |  3737 |  1849
George Washington (GW) |  901 |  1215 |  2447

###  4.3 Implementation details

Since the proposed method is based on the masked encoder-decoder architecture,
we initially performed several preliminary experiments to select the best
model architecture leading to the best results. Moreover, we have picked up
some parameters that have been proven successful in [24]. Subsequently, we
trained the network in an end-to-end fashion. Table. 2 describes the network’s
architecture and the parameter setting.

Table 2: Proposed model’s hyperparameters used in our work.

Parameter |  Value
---|---
Patch size |  $16\times 16$
Image size |  224x224
Train size✞ |  60k
Encoder embedding dimension |  768
Encoder layers number |  12
Encoder head number |  12
Decoder embedding dimension |  384
Decoder layers number |  4
Decoder head number |  6

✞ Training set is augmented using different image transformation operations.

The proposed framework was developed using the Pytorch library designed for
deep learning. To pretrain the self-supervised model, we used a maximum of 500
epochs, with a batch size of 20. During the pretraining stage we used AdamW,
an improved version of the algorithm [57], for parameter optimization. The
learning rate is initially set to $1.5\mathrm{e}{-4}$ and decreased using a
scheduler. For the fine-tuning stage, we employed the SGD (Stochastic gradient
descent) optimization algorithm with a learning schedule. An initial learning
rate of 0.01 was used when fine-tuning the siamese neural network. This rate
is declined every 3 epochs. The experiments were performed using an NVIDIA
TITAN Xp GPU. To improve the generalization power of the proposed ST-KeyS
model, we used different random transformation operations, as data
augmentation techniques, such as erosion, dilation, and skewing.

###  4.4 KeyWord Spotting Results

We present in this section the obtained results. It should be noted that in
all our experiments, we rely on the Cosine similarity distance as a metric
allowing us to measure the similarity between two feature vectors. This metric
computes the angle’s cosine between two vectors and identifies if two vectors
point approximately in the same direction.

####  4.4.1 Evaluation on ICFHR 2016 competition’s datasets

The evaluation process is conducted firstly on the ICFHR 2016 competition’s
datasets, namely, the BOT and the AK. Table 3 shows the obtained results. We
recall that the ST-KeyS model is first pretrained on the IAM database in a
self-supervised way in order to learn deep representation from word images.
Secondly, the pretrained encoder is fine-tuned using a few labeled data from
the target dataset. Since the IAM, BOT, AK and GW databases are belonging to
different datasets distributions, we are intended to deal with challenges
related to domain variations and writers styles shifts. That’s why we extend
the pretraining stage with some epochs on a subset from the target dataset
(BOT, AK and GW) in a self-supervised way (without annotation/labels). Such a
process allows to reduce the gap between domains originating from the
calligraphic style fluctuations of the IAM and the used evaluation datasets.

From Table 3, we notice that the proposed ST-KeyS model yields good results on
the BOT dataset reaching a performance of 59.10% which outperforms TPP-
PHOCNet(CPS), PHOCNet(CPS) and Triplet CNN architectures [9]. However, we
obtained moderate results on the AK dataset comparing to TPP-PHOCNet(CPS) and
PHOCNet(CPS) architectures [9]. Moreover, we observe that the ST-KeyS leads to
better results than learning-free based methods [36, 39, 6] on both the BOT
and AK datasets although they represent different calligraphic styles compared
to the IAM database utilized in the first pretraining stage and also despite
the difference in the written language script. Additionally, our ST-KeyS model
gives better results compared to the Graph Edit Distance method proposed in
[7] thanks to the transformer’s capacity to focus on meaningful information
included in the word image. This capacity is owed to the attention mechanism
allowing to extract discriminant features from word images and which are used
later in the matching process.

In summary, we note that the ST-KeyS semi-supervised model outperforms the
fully supervised PHOC-based methods [9] that were conducted in the literature
on the BOT dataset and yields moderate results with these methods on the AK
dataset. Generally, these methods rely on synthetically generated data to
learn the PHOC contextual representations. Thus, such models are able to treat
large vocabulary allowing to alleviate to the out of vocabulary words issue
[58]. In the remainder of our work, we limit the PHOC learning stage to the
data provided in the competition image databases. In other words, we are based
on a closed vocabulary since our goal is to build a robust model allowing to
retrieve handwritten words independently of the out of vocabulary word
problem. Therefore, it is unfair to compare our results to PHOC-based methods
that were conducted in the literature. For that reason, we restrict the
comparison to methods using closed vocabulary.

Table 3: Evaluation results (mAP %) of the word spotting proposed method
performed on the BOT and the AK datasets.

Method |  BOT |  AK
---|---|---
Learning-free methods
Projections of Oriented Gradients [36] |  46.70 |  56.50
Graph matching [39] |  49.57 |  77.24
PSeq-mPOG+MISM [6] |  58.30 |  76.20
Learning-based methods
Graph Edit Distance [7] |  52.83 |  79.55
TPP-PHOCNet(CPS) [9]✝✧ |  51.25 |  90.97
PHOCNet(CPS) [9]✝✧ |  45.82 |  88.31
Triplet-CNN [9]✝✧ |  54.95 |  82.15
Ours (ST-KeyS )✝ |  59.10 |  85.16

✝ Methods based on PHOC representation.

✧ Methods based on generated synthetic data (large vocabulary).

####  4.4.2 Evaluation on the GW dataset

Further experiments are conducted on the GW dataset to assess the performance
of our proposed ST-KeyS method. The obtained results on GW dataset are
illustrated in Table 4. ST-KeyS achieved a mAP of 95.70% on the GW database.
Table 4 shows that our proposed method leads to interesting results that are
exceeding all considered learning-free methods [37, 4, 36, 6]. This result
proves the efficiency of the ST-KeyS model to extract deep meaningful features
from word images enabling an accurate word spotting task compared to the use
of handcrafted ones (HOG, mPOG, etc.). In addition, the ST-KeyS method gives
better results than the current state-of-the-art methods that are based on
supervised approaches [11, 41, 59] and using a closed vocabulary. Moreover,
ST-KeyS outperforms the graph-based approach [7] by a large margin.

It is noteworthy that our method focus on designing a robust model allowing
retrieval of word images in handwritten documents when only few labelled data
are available. Despite this, we can see from Table 4 that ST-KeyS achieved
competitive results compared to PHOC based methods trained on large generated
synthetic data. ST-KeyS outperforms the methods presented in [4, 5] and gets a
slightly lower performance compared the methods [60, 42] without the need for
any external data.

Table 4: Evaluation results (mAP %) of the word spotting proposed method
performed on the GW dataset.

Method | mAP %
---|---
Learning-free methods
HOG + SVM [37] | 49.40
DTW [4] | 60.63
FV [4] | 62.72
Projections of Oriented Gradients [36] | 37.00
PSeq-mPOG+MISM [6] | 77.10
Learning-based methods
Zoning Aggregated features (CNN) [59] | 58.30
Softmax CNN [41] | 78.24
Siamese Triplet CNN [11] | 91.63
Graph Edit Distance [7] | 78.48
Att. + Platts (PHOC) [4]✝✧ | 93.04
PHOCNet [41] ✝✧ | 96.71
TPP-PHOCNet [42]✝✧ | 97.78
HWNet [5]✝✧ | 94.84
HWNet v2 (ROI) [60]✝✧ | 96.01
HWNet v2 (TPP) [60]✝✧ | 98.24
Ours (ST-KeyS)✝ | 95.70

✝ Methods based on PHOC representations.

✧ Methods based on generated synthetic data (large vocabulary).

Fig.8 shows qualitative evaluation of our proposed ST-KeyS method on the GW
database where the top five retrieved images for each query word image are
given. This figure highlights the robustness of the features extracted using
the proposed method. Such features make the method invariant across different
word capitalization forms and some degradation characterizing historical
documents. Some failure cases are presented at the end of the figure where our
model fails to retrieve the correct matching word image. This can occur as a
result of the stroke corruption of some words like the word ’Waggons’ at the
fifth row where the last letter ’s’ is not clearly noticeable. Hence, the
model faces an ambiguity in deciding whether this letter is present at the end
of the word. In other cases, this can be attributed to the lexical complexity
of the word image. For instance, the word ”Sergeant” at the sixth row is well
retrieved at the top 1 and 2 instances, but it was then conflicted with the
word ”Regiment” which makes the model not able to decide about some regions of
the word image. In other cases, the model is missing the appropriate instances
because of the scarcity of this image’s representation in the used dataset as
is the case of the seventh row of Fig. 8. However, our model was able to
retrieve the unique right instance in the second rank for the example shown in
the eighth row. To summarize, we conclude that the ambiguity and the
complexity of the script writing and the close similarity among different
words are the principal causes of the failures.

Query Top 5 retrieved images
---
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
|  |  |  |  |
Figure 8: Qualitative results of the proposed ST-KeyS method on sample
examples from the GW evaluation dataset. Green boxes correspond to correctly
retrieved words and red boxes correspond to incorrectly retrieved words.

###  4.5 Ablation study

In preliminary research, we have conducted different experiments to first
select the best architecture and the optimal parameters leading to the most
efficient ST-KeyS model, then to compare the proposed method to different
self-supervised/supervised methods. These investigations are detailed in this
section.

\- Investigating the design choice: In here, we discuss the self-supervised
pretraining and the different downstream task options. We performed several
experiments to prove the efficiency of our ST-KeyS model that consists of two
phases: a self-supervised pretraining phase followed by a downstream task
(supervised fine-tuning phase). The fine tuning stage is further composed of
siamese network architecture followed by PHOC embedding (SNN-PHOC). Thus, we
study the effectiveness of the pretraining, as well as the utility of the
fine-tuning in the chosen order. The obtained results of this investigation on
the three datasets are presented in Table 5, where we studied different
scenarios. Particularly, we compare the efficiency of the proposed model while
using different downstream tasks such as 1) a siamese architecture followed by
a PHOC alignement (SNN-PHOC), 2) PHOC followed by a siamese architecture
(PHOC-SNN), 3) the use of PHOC representations exclusively and 4) the use of a
siamese network uniquely (SNN). Also, we check the utility of the pretraining
by comparing it with a supervised-only approach.

As it can be seen from the table, the best performance by our model is with
the setting SNN-PHOC. We obtain using the GW and AK databases 95.70% and
85.16%, respectively. For the BOT dataset, the performance is beyond 59%.
Hence, we can conclude that the use of a SNN and PHOC downstream task at the
fine-tuning stage is the better option, especially in the order SNN-PHOC
(first train the SNN and then train with PHOC). The main justification behind
this lies in the fact that the siamese network has the capacity to extract
relevant deep features leading to a relevant representation of the word image
and that is subsequently processed by the PHOC-based encoder allowing the
contextual content of the word image to be promoted.

Moreover, we conducted an evaluation of the self-supervised approaches
compared to the supervised ones. As it can be depicted from Table 5, the use
of the self-supervised approach followed by a downstream task (SNN) performs
better than using a supervised approach based on a siamese architecture (row 5
vs. row 4). A performance of 92.48% is reached using a pretraining stage while
it was at 69.98% when performing supervised approach on the GW database.
Therefore, we can affirm the importance of the pretraining stage in a self-
supervised fashion to improve the word spotting results.

Finally, we explored the relevance of the pretraining stage in a self-
supervised way without any labeled data. We observe (from the last row in
Table 5) that we obtained interesting results from only the image
representations. For instance, we achieved a performance of 61.85% using the
GW dataset which is good and even better than some learning-free word spotting
methods.

Table 5: Evaluation of different downstream tasks for word spotting performed
on the GW, BOT and AK datasets. (results are presented in term of mAP% metric)

Approach |  Downstream Task |  GW |  BOT |  AK
---|---|---|---|---
Self-Supervised |  SNN-PHOC |  95.70 |  59.10 |  85.16
Self-Supervised |  PHOC-SNN |  92.62 |  59.30 |  82.80
Self-Supervised |  PHOC |  92.50 |  59.17 |  82.53
Self-Supervised |  SNN |  92.48 |  58.06 |  81.51
Supervised (SNN) |  – |  69.98 |  $NP$ |  $NP$
Self-Supervised (encoder) |  – |  61.85 |  43.79 |  58.63

NP : Not Performed

\- Evaluation of the model utility in the low resource scenario: This
evaluation is based on siamese downstream task using different annotated data
amounts. To further prove the importance of the self-supervised approach
followed by a downstream task according to the amount of annotated data
utilized in the fine-tuning phase, we have conducted a study while varying the
amount of labeled data to simulate a low resource scenario. From Table 6, we
remark that we have achieved a good result using merely 20% of the labeled
data which is relatively comparable to the use of 60% of annotated data and
even to the utilization of the entire annotated data provided in the GW
database. This confirms the utility of our approach, especially for the low
resource datasets, when the size of the labeled data is limited.

Table 6: Evaluation results (mAP %) of the word spotting task performed on the GW dataset for the Self-supervised approach with a siamese downstream task using different data amounts. Annotated Fraction% |  mAP%
---|---
20 |  80.98
40 |  85.06
60 |  86.88
100 |  92.48

\- Model’s parameters optimisation: To perform the proposed word spotting
method, we have conducted extensive preliminary experiments to set up the
suitable parameters for the model configuration that leads to the best
performance. Therefore, we start our experiments by selecting the optimum
patch size and model architecture.

In our experiments, we define three types of model’s variants which are ST-
KeyS-Base-16, ST-KeyS-Base-32 and ST-KeyS-Small-16, as enlisted in Table 7.
Obviously, implementing a larger model requires more computational memory and
learning time as the number of model parameters increases. Thus, a trade-off
between the size of the model and its performance must be considered.

Table 7: Details of our model variants (EL: Encoder Layers, EAH: Encoder Attention Heads, DL: Decoder Layers, DAH: Decoder Attention Heads). Model |  EL |  EAH |  DL |  DAH |  Patch |  Params
---|---|---|---|---|---|---
ST-KeyS-Base-16 |  12 |  6 |  4 |  6 |  $16\times 16$ |  9.3M
ST-KeyS-Base-32 |  12 |  6 |  4 |  6 |  $32\times 32$ |  9.5M
ST-KeyS-Small-16 |  6 |  3 |  4 |  3 |  $16\times 16$ |  1.2M

In Table 8, we present the results of the Self-supervised approach using only
the pretraining representations (without downstream task) from the three
defined model variants. We note that the performance of the word spotting
method is improved when using a patch size of $16\times 16$ instead of using a
patch size of $32\times 32$ for the same encoder-decoder architecture (row 1
vs. row 2 in Table 8). The explanation behind this behavior is that, by
utilizing a smaller patch size, we make each patch of the image have more
local information during self-attention. Therefore, the model is able to use
more and finer information during the word spotting process with a patch size
of $16\times 16$. However, in the case of using a smaller patch size such as
$8\times 8$, it will lead to the augmentation of the model parameters which
results in the requirement of more computation resources. For that reason, we
have discarded this setting. Also, we remark that the use of a model having a
large number of layers and heads leads to a significant improvement in the
word spotting results (row 1 vs. row 3 in Table 8). Since we attempt to
achieve a compromise between these hyper-parameters and resource consumption
while maintaining good performance, we selected the ST-KeyS-Base-16 model’s
setting as the base model architecture in all our performed experiments.

Table 8: Evaluation of the Self-supervised approach (without a downstream task) for word spotting task in terms of mAP% on the GW database using different model variants. Model |  mAP %
---|---
ST-KeyS-Base-16 |  61.85
ST-KeyS-Base-32 |  40.29
ST-KeyS-Small-16 |  37.53

##  5 Conclusion and Future Work

In this paper, we proposed a novel self-supervised approach for keyword
spotting called ST-KeyS. Our method is based on a pure vision transformer
architecture without any CNN layer, and it is composed of a pretraining phase
and a fine-tuning phase. In the pretraining stage, a masking-recovering
autoencoder is used to learn useful representations from the unlabeled word
images. In the fine-tuning stage, a two steps strategy is employed to extract
further promoted representations for the spotting task. To produce more robust
and meaningful features, a siamese approach is first utilized to embed the
images visually, followed by an alignment approach to the PHOC attributes
produced from the text.

The underlined architecture deals with the data scarcity issue since we have
restricted our study to the available data provided in the evaluation datasets
without using synthetic data or a more extensive dictionary. Extensive
experiments have been made to validate the design choice of our proposal and
evaluate it according to the state-of-the-art. The experiment results
demonstrated the efficiency of self-supervised learning for word spotting by
breaking with the need to have large annotated datasets. We have shown that
our model allows achieving close, and sometimes better, performance to fully
supervised approaches based on large synthetic datasets despite our
restriction to limited data amounts and closed vocabulary.

In the future, we will enhance the performance of our proposed word spotting
method by adding an n-gram language model to re-rank the retrieved word image
list. We would also like to apply the self-supervised approach in a
segmentation-free fashion and for other connected fields, such as the
handwriting recognition area.

## Acknowledgement

This work has been supported by the DocPRESERV (Preserving & Processing
Historical Document Images with Artificial Intelligence) project, funded by
the Swedish Foundation for International Cooperation in Research and Higher
Education (STINT), grant AF2020-8892.

## References

  * [1] S. Fernández, A. Graves, J. Schmidhuber, An application of recurrent neural networks to discriminative keyword spotting, in: J. M. de Sá, L. A. Alexandre, W. Duch, D. Mandic (Eds.), Artificial Neural Networks – ICANN 2007, 2007, pp. 220–229.  doi:doi.org/10.1007/978-3-540-74695-9_23.
  * [2] V. Frinken, A. Fischer, R. Manmatha, H. Bunke, A novel word spotting method based on recurrent neural networks, IEEE Transactions on Pattern Analysis and Machine Intelligence 34 (2) (2012) 211–224.  doi:10.1109/TPAMI.2011.113.
  * [3] A. P. Giotis, G. Sfikas, B. Gatos, C. Nikou, A survey of document image word spotting techniques, Pattern recognition 68 (2017) 310–332.  doi:doi.org/10.1016/j.patcog.2017.02.023.
  * [4] J. Almazán, A. Gordo, A. Fornés, E. Valveny, Word spotting and recognition with embedded attributes, IEEE Transactions on Pattern Analysis and Machine Intelligence 36 (12) (2014) 2552–2566.  doi:10.1109/TPAMI.2014.2339814.
  * [5] P. Krishnan, C. Jawahar, Matching handwritten document images, in: Computer Vision – ECCV 2016, 2016.  doi:doi.org/10.1007/978-3-319-46448-0_46.
  * [6] G. Retsinas, G. Louloudis, N. Stamatopoulos, B. Gatos, Efficient learning-free keyword spotting, IEEE Transactions on Pattern Analysis and Machine Intelligence 41 (7) (2019) 1587–1600.  doi:10.1109/TPAMI.2018.2845880.
  * [7] P. Riba, A. Fischer, J. Lladós, A. Fornés, Learning graph edit distance by graph neural networks, Pattern Recognition 120 (2021) 108132.  doi:doi.org/10.1016/j.patcog.2021.108132.
  * [8] E. Vats, A. Hast, A. Fornés, Training-free and segmentation-free word spotting using feature matching and query expansion, in: 2019 International Conference on Document Analysis and Recognition (ICDAR), 2019, pp. 1294–1299.  doi:10.1109/ICDAR.2019.00209.
  * [9] S. Sudholt, G. A. Fink, Attribute cnns for word spotting in handwritten documents, International Journal on Document Analysis and Recognition (IJDAR) 21 (2018) 199–218.  doi:doi.org/10.1007/s10032-018-0295-0.
  * [10] T. Wilkinson, J. Lindström, A. Brun, Neural ctrl-f: Segmentation-free query-by-string word spotting in handwritten manuscript collections, in: 2017 IEEE International Conference on Computer Vision (ICCV), 2017, pp. 4443–4452.  doi:10.1109/ICCV.2017.475.
  * [11] Y. Serdouk, V. Eglin, S. Bres, M. Pardoen, Keyword spotting using siamese triplet deep neural networks, in: 2019 International Conference on Document Analysis and Recognition (ICDAR), 2019, pp. 1157–1162.  doi:10.1109/ICDAR.2019.00187.
  * [12] A. H. Toselli, V. Romero, J. A. Sánchez, E. Vidal, Making two vast historical manuscript collections searchable and extracting meaningful textual features through large-scale probabilistic indexing, in: 2019 International Conference on Document Analysis and Recognition (ICDAR), 2019, pp. 108–113.  doi:10.1109/ICDAR.2019.00026.
  * [13] F. Daraee, S. Mozaffari, S. M. Razavi, Handwritten keyword spotting using deep neural networks and certainty prediction, Computers & Electrical Engineering 92 (2021) 107111.  doi:doi.org/10.1016/j.compeleceng.2021.107111.
  * [14] G. De Gregorio, S. Biswas, M. A. Souibgui, A. Bensalah, J. Lladós, A. Fornés, A. Marcelli, A few shot multi-representation approach for n-gram spotting in historical manuscripts, in: Frontiers in Handwriting Recognition: 18th International Conference, ICFHR 2022, Springer, 2022, pp. 3–17.  doi:doi.org/10.1007/978-3-031-21648-0_1.
  * [15] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, I. Polosukhin, Attention is all you need, in: Proceedings of the 31st International Conference on Neural Information Processing Systems, 2017, pp. 6000–6010.
  * [16] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, Bert: Pre-training of deep bidirectional transformers for language understanding, arXiv preprint arXiv:1810.04805.
  * [17] N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, S. Zagoruyko, End-to-end object detection with transformers, in: European Conference on Computer Vision, Springer, 2020, pp. 213–229.
  * [18] A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, N. Houlsby, An image is worth 16x16 words: Transformers for image recognition at scale, in: International Conference on Learning Representations, 2021.
  * [19] A. F. Biten, R. Litman, Y. Xie, S. Appalaraju, R. Manmatha, Latr: Layout-aware transformer for scene-text vqa (2022) 16527–16537doi:10.1109/CVPR52688.2022.01605.
  * [20] M. A. Souibgui, S. Biswas, S. Khamekhem Jemni, Y. Kessentini, A. Fornés, J. Lladós, U. Pal, DocEnTr: An end-to-end document image enhancement transformer, in: ICPR, 2022, pp. 1699–1705.  doi:10.1109/ICPR56361.2022.9956101.
  * [21] M. Li, T. Lv, L. Cui, Y. Lu, D. A. F. Florêncio, C. Zhang, Z. Li, F. Wei, Trocr: Transformer-based optical character recognition with pre-trained models, ArXiv abs/2109.10282.
  * [22] M. A. Souibgui, S. Biswas, A. Mafla, A. F. Biten, A. Fornés, Y. Kessentini, J. Lladós, L. Gomez, D. Karatzas, Text-DIAE: A self-supervised degradation invariant autoencoders for text recognition and document enhancement, arXiv preprint arXiv:2203.04814.
  * [23] A. Cheikh Rouhou, M. Dhiaf, Y. Kessentini, S. Ben Salem, Transformer-based approach for joint handwriting and named entity recognition in historical document, Pattern Recognition Letters 155 (2022) 128–134.  doi:doi.org/10.1016/j.patrec.2021.11.010.
  * [24] K. He, X. Chen, S. Xie, Y. Li, P. Dollar, R. Girshick, Masked autoencoders are scalable vision learners, in: 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 15979–15988.  doi:10.1109/CVPR52688.2022.01553.
  * [25] H. Bao, L. Dong, S. Piao, F. Wei, BEit: BERT pre-training of image transformers, in: International Conference on Learning Representations, 2022.
  * [26] Y. K. Jang, N. I. Cho, Self-supervised product quantization for deep unsupervised image retrieval, in: Proceedings of the International Conference on Computer Vision (ICCV), 2021.
  * [27] S. Sadhu, D. He, C.-W. Huang, S. H. Mallidi, M. Wu, A. Rastrow, A. Stolcke, J. Droppo, R. Maas, Wav2vec-c: A self-supervised model for speech representation learning, in: Proc. Interspeech, 2021, pp. 711–715.  doi:10.21437/Interspeech.2021-717.
  * [28] R. Manmatha, C. Han, E. M. Riseman, Word spotting: A new approach to indexing handwriting, in: Proceedings CVPR IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE, 1996, pp. 631–637.
  * [29] T. M. Rath, R. Manmatha, Word image matching using dynamic time warping, in: 2003 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2003. Proceedings., Vol. 2, IEEE, 2003.
  * [30] B. Zhang, S. N. Srihari, C. Huang, Word image retrieval using binary features, in: Document Recognition and Retrieval XI, Vol. 5296, SPIE, 2003, pp. 45–53.
  * [31] D. G. Lowe, Distinctive image features from scale-invariant keypoints, International journal of computer vision 60 (2) (2004) 91–110.
  * [32] N. Dalal, B. Triggs, Histograms of oriented gradients for human detection, in: 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR’05), Vol. 1, IEEE, 2005, pp. 886–893.
  * [33] M. Rusinol, D. Aldavert, R. Toledo, J. Lladós, Browsing heterogeneous document collections by a segmentation-free word spotting method, in: 2011 International Conference on Document Analysis and Recognition, IEEE, 2011, pp. 63–67.
  * [34] M. Rusiñol, D. Aldavert, R. Toledo, J. Lladós, Efficient segmentation-free keyword spotting in historical document collections, Pattern recognition 48 (2) (2015) 545–555.
  * [35] R. Shekhar, C. Jawahar, Word image retrieval using bag of visual words, in: 2012 10th IAPR International Workshop on Document Analysis Systems, IEEE, 2012, pp. 297–301.
  * [36] G. Retsinas, G. Louloudis, N. Stamatopoulos, B. Gatos, Keyword spotting in handwritten documents using projections of oriented gradients, in: 2016 12th IAPR Workshop on Document Analysis Systems (DAS), 2016, pp. 411–416.
  * [37] J. Almazán, A. Gordo, A. Fornés, E. Valveny, Efficient exemplar word spotting, in: British Machine Vision Conf. Surrey, UK, 2012.
  * [38] J. Almazán, A. Gordo, A. Fornés, E. Valveny, Segmentation-free word spotting with exemplar svms, Pattern recognition 47 (12) (2014) 3967–3978.  doi:doi.org/10.1016/j.patcog.2014.06.005.
  * [39] M. Stauffer, A. Fischer, K. Riesen, Keyword spotting in historical handwritten documents based on graph matching, Pattern Recognition 81 (2018) 240–253.  doi:doi.org/10.1016/j.patcog.2018.04.001.
  * [40] M. Jaderberg, A. Vedaldi, A. Zisserman, Deep features for text spotting, in: European conference on computer vision, Springer, 2014, pp. 512–528.
  * [41] S. Sudholt, G. Fink, PHOCNet: A deep convolutional neural network for word spotting in handwritten documents, in: 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR), 2016, pp. 277–282.  doi:10.1109/ICFHR.2016.0060.
  * [42] S. Sudholt, G. Fink, Evaluating word string embeddings and loss functions for cnn-based word spotting, in: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), Vol. 01, 2017, pp. 493–498.  doi:10.1109/ICDAR.2017.87.
  * [43] E. Rusakov, L. Rothacker, H. Mo, G. A. Fink, A probabilistic retrieval model for word spotting based on direct attribute prediction, in: 2018 16th International Conference on Frontiers in Handwriting Recognition (ICFHR), 2018, pp. 38–43.  doi:10.1109/ICFHR-2018.2018.00016.
  * [44] P. Krishnan, K. Dutta, C. Jawahar, Deep feature embedding for accurate recognition and retrieval of handwritten text, in: 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR), IEEE, 2016, pp. 289–294.
  * [45] P. Krishnan, K. Dutta, C. Jawahar, Word spotting and recognition using deep embedding, in: 2018 13th IAPR International Workshop on Document Analysis Systems (DAS), 2018, pp. 1–6.  doi:10.1109/DAS.2018.70.
  * [46] Y. Xu, Y. Xu, T. Lv, L. Cui, F. Wei, G. Wang, Y. Lu, D. Florencio, C. Zhang, W. Che, et al., LayoutLMv2: Multi-modal pre-training for visually-rich document understanding, arXiv preprint arXiv:2012.14740.
  * [47] P. Li, J. Gu, J. Kuen, V. I. Morariu, H. Zhao, R. Jain, V. Manjunatha, H. Liu, Selfdoc: Self-supervised document representation learning, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 5652–5660.  doi:10.1109/CVPR46437.2021.00560.
  * [48] J. Li, Y. Xu, T. Lv, L. Cui, C. Zhang, F. Wei, Dit: Self-supervised pre-training for document image transformer, arXiv preprint arXiv:2203.02378.
  * [49] S. Appalaraju, B. Jasani, B. U. Kota, Y. Xie, R. Manmatha, Docformer: End-to-end transformer for document understanding, in: Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 993–1003.
  * [50] A. Aberdam, R. Litman, S. Tsiper, O. Anschel, R. Slossberg, S. Mazor, R. Manmatha, P. Perona, Sequence-to-sequence contrastive learning for text recognition, in: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 15302–15312.
  * [51] X. Zhang, B. Zhu, X. Yao, Q. Sun, R. Li, B. Yu, Context-based contrastive learning for scene text recognition, in: AAAI Conference on Artificial Intelligence, Vol. 36, 2022, pp. 3353–3361.  doi:10.1609/aaai.v36i3.20245.
  * [52] M. Yang, M. Liao, P. Lu, J. Wang, S. Zhu, H. Luo, Q. Tian, X. Bai, Reading and writing: Discriminative and generative modeling for self-supervised text recognition, in: Proceedings of the 30th ACM International Conference on Multimedia, 2022, pp. 4214–4223.
  * [53] Q. Wang, B. Li, T. Xiao, J. Zhu, C. Li, D. F. Wong, L. S. Chao, Learning deep transformer models for machine translation, in: Annual Meeting of the Association for Computational Linguistics, Association for Computational Linguistics, 2019, pp. 1810–1822.  doi:10.18653/V1/P19-1176.
  * [54] U.-V. Marti, H. Bunke, A full english sentence database for off-line handwriting recognition, in: Proceedings of the Fifth International Conference on Document Analysis and Recognition. ICDAR ’99 (Cat. No.PR00318), 1999, pp. 705–708.  doi:10.1109/ICDAR.1999.791885.
  * [55] A. Fischer, A. Keller, V. Frinken, H. Bunke, Lexicon-free handwritten word spotting using character hmms, Pattern Recognition Letters 33 (7) (2012) 934–942, special Issue on Awards from ICPR 2010.  doi:doi.org/10.1016/j.patrec.2011.09.009.
  * [56] I. Pratikakis, K. Zagoris, B. Gatos, J. Puigcerver, A. H. Toselli, E. Vidal, Icfhr2016 handwritten keyword spotting competition (h-kws 2016), in: roc. Int. Conf. Frontiers Handwriting Recognit., 2016, pp. 613–618.  doi:10.1109/ICFHR.2016.0117.
  * [57] D. P. Kingma, J. B. Adam, A method for stochastic optimization, in: CoRR, abs/1412.6980, 2014.
  * [58] S. Khamekhem Jemni, Y. Kessentini, S. Kanoun, Out of vocabulary word detection and recovery in arabic handwritten text recognition, Pattern Recognition 93 (2019) 507–520.  doi:doi.org/10.1016/j.patcog.2019.05.003.
  * [59] G. Sfikas, G. Retsinas, B. Gatos, Zoning aggregated hypercolumns for keyword spotting, in: 2016 15th International Conference on Frontiers in Handwriting Recognition (ICFHR), 2016, pp. 283–288.  doi:10.1109/ICFHR.2016.0061.
  * [60] P. Krishnan, C. Jawahar, HWnet v2: an efficient word image representation for handwritten documents, Int. Journal on Document Analysis and Recognition 22 (4) (2019) 387–405.  doi:doi.org/10.1007/s10032-019-00336-x.


11institutetext: Institut für Theoretische Physik, Technische Universität
Wien,
Wiedner Hauptstr. 8-10, A-1040 Vienna, Austria

# Colliding poles with colliding nuclei

Alexander Soloviev 11<EMAIL_ADDRESS>
###### Abstract

In these proceedings, I will discuss collisions of poles in the complex plane
as a signature of phase transitions for theories relevant to the quark gluon
plasma. I will begin with an illustrative example, namely the chiral phase
transition, which can be characterized by colliding poles as a function of
temperature. Then, recognizing the interplay between weak and strong coupling
sectors in a typical collision, I will introduce a hybrid model with a weakly
broken symmetry, which has a rich quasi-hydrodynamic phenomenological
description where hydrodynamic and non-hydrodynamic poles are unified by a
common dispersion relation. I will show that energy is transferred initially
from the soft to the hard sector before irreversibly transferring back to the
soft sector at late times, and that the model reproduces many features common
to dissipative systems with a weakly broken symmetry including the k-gap.

##  1 Introduction

In the wake of heavy ion collisions, performed in experiments in the LHC at
CERN and RHIC at BNL, a state of matter of liberated quarks and gluons, known
as the quark-gluon plasma (QGP), is formed STAR:2005gfr . There are three
common effective descriptions used to study the QGP, which have different
regions of applicability within the evolution of a typical heavy ion
collision. At early times, kinetic theory captures the weakly coupled dynamics
Arnold:2002zm . At later times, as the system undergoes hydrodynamization,
strongly coupled holography Heller:2016gbp  and relativistic hydrodynamics
Teaney:2009qa  become increasingly more relevant.

Following the discussion in Kurkela:2019kip , we can examine the analytic
structure of the mentioned theories. Each theory mentioned has in common a
hydrodynamic pole, but other non-analytic structures are radically different.
For instance, kinetic theory in the relaxation time approximation typically
has a branch cut between $-k$ and $k$ at $-i/\tau$ where $\tau$ is the
relaxation time Kurkela:2017xis . Holographic theories exhibit a so-called
"Christmas tree" structure of an infinite number of poles with increasing real
and imaginary values in the lower half plane Grozdanov:2018gfx . Hydrodynamic
theories such as Müller-Israel-Stewart (MIS) Muller:1967zza ; Israel:1979wp
have a non-propagating dissipative pole. One is left naturally to ask the
question: what is the analytic structure and, ultimately, the microscopic
structure of the QGP?

Here, we will aim to partly answer the above question by first underlining in
Sec. 2 that the chiral phase transition in the $O(4)$ model Rajagopal:1992qz ;
Son:1999pa ; Son:2002ci , a characteristic feature present in a typical heavy
ion collision, can be understood via a collision of poles in the complex
plane, as the system moves from diffusive to propagating degrees of freedom.
Then in Sec. 3, we will demonstrate a similar collision of poles in a theory
with weak/strong coupling dynamics via a quasinormal mode analysis of a
semiholographic model with a boundary scalar field.

##  2 Colliding poles near the O(4) critical point

Here, we will discuss reinterpret the chiral phase transition, explored via
the $O(4)$ model in Grossi:2020ezz ; Grossi:2021gqi ; Florio:2021jlx ;
Soloviev:2021syx , as a collision of poles in the complex plane. We describe
the $O(4)$ physics via the effective Hamiltonian

$\displaystyle\mathcal{H}=\int_{x}$ $\displaystyle
p(T)+\frac{\chi_{0}}{4}\mu_{ab}^{2}-\frac{1}{2}\Delta^{\mu\nu}D_{\mu}\phi_{a}D_{\nu}\phi_{a}+\frac{1}{2}m^{2}_{0}(T-T_{c})\phi_{a}\phi_{a}+\frac{\lambda}{4}(\phi_{a}\phi_{a})^{2}-H_{a}\phi_{a}$
(1)

where we denote the $O(4)$ vector via $\phi_{a}=(\sigma,\varphi_{i})$,
$\Delta^{\mu\nu}=u^{\mu}u^{\nu}+g^{\mu\nu},$ $\mu_{ab}$ is the $O(4)$ chemical
potential, $\chi_{0}$ is the static susceptibility and $H_{a}$ is the explicit
symmetry breaking term. Note that the mass term, $m^{2}=m_{0}^{2}(T-T_{c})$,
changes sign in the vicinity of the critical temperature. Choosing
$H_{a}=(H,0,0,0),$ we will work in the mean field limit, which is found by
minimizing the potential in (1) to find

$\displaystyle m^{2}_{0}(T-T_{c})\sigma+\lambda\sigma^{3}-H=0,$ (2)

which determines the mean field value of the condensate as a function of
temperature, $\sigma(T)$, with its behavior shown in Fig. 1.

Figure 1: Comparison of mean field order parameter to the lattice in
Engels:2009tv ; Engels:2011km .

Computing the ideal equations of motion from (1), including dissipation and
linearizing around mean field leads to the following coupled equations for the
pions and chemical potential:

$\displaystyle{\partial_{t}\varphi}$
$\displaystyle=-{{\mu}_{A}}+{\Gamma(\nabla^{2}-m^{2})\varphi}$ (3)
$\displaystyle{\partial_{t}\mu_{A}}$
$\displaystyle={v^{2}(-\nabla^{2}+m^{2})\varphi}+{D_{0}\nabla^{2}\mu_{A}}$ (4)

We do not consider the dynamics of the condensate, which decouple from the
pions and the chemical potential in the linearized limit. From the above, we
can determine the symmetrized propagators $G_{\rm
sym}=T(G_{R}-G_{A})/(i\omega)$. Here, we will focus on the the spectral
density for the axial charge density-density correlator, which is related to
the $\varphi\varphi$ symmetrized propagator via

$\displaystyle\rho_{AA}=\frac{\omega}{T(\chi_{0}\omega_{k})^{2}}G^{\varphi\varphi}_{\rm
sym},$ (5)

where $\omega_{k}^{2}=v^{2}(k^{2}+m^{2}).$ The behavior of this propagator is
best understood by considering Fig. 2. In the left panel, reproduced from
Grossi:2021gqi , the spectral function is plotted as a function of frequency
for various $z\sim(T-T_{c})/T_{c}$. At high temperatures, the system is in the
unbroken phase, characterized by a diffusive peak that we can interpret as the
diffusion of quarks in the QGP. As the temperature decreases, the system
eventually goes through the phase transition, with the spectral function
exhibiting two well-separated peaks that we can interpret as pions.

The right panel presents another way to view the same information in the left
panel, but from the perspective of the pole structure of the spectral
function. Starting from the green diffusive poles, representing the unbroken
phase with $T\gg T_{c}$, the direction of decreasing temperature is indicated
by the black arrows. Eventually the diffusive from the two poles collide,
indicating that the system is undergoing a phase transition. From the
collision point, further decreasing the temperature leads to two well
separated poles, namely the pion modes.

In this way, the collision of poles as a function of temperature in the
complex plane indicates a phase transition, a feature that should be studied
in understanding the analytic structure of the QGP.

Figure 2: Two different ways to see the $O(4)$ phase transition via the
spectral density. Left: The spectral density over frequency for different
$z\sim T-T_{c}$. For the choice of parameters used in this plot, see the
discussion in Grossi:2021gqi . The diffusive peak in the unbroken phase $z\gg
0$ melts to two peaks in the broken phase, $z\ll 0$, representing the
propagating pions. Right: the pole structure of the spectral density as a
function of temperature. The green points represent $z=16$, i.e. the high
temperature, unbroken phase, while the arrows indicate the decrease in
temperature until the red point $z=-16.$ Like in the left plot, the diffusive
poles on the imaginary axis melt down to two propagating poles.

##  3 Colliding poles in holography

We now turn our attention to understanding the analytic structure of the QGP
from another perspective. Although strong and weak coupling dynamics are often
studied separately, a complete description of the QGP would entail
understanding the interplay between the two sectors. One approach in this line
of reasoning is known as semiholography Faulkner:2010tq ; Iancu:2014ava ;
Mukhopadhyay:2015smb ; Banerjee:2017ozx ; Kurkela:2018dku ; Ecker:2018ucc ;
Mitra:2020mei ; Mitra:2022uhv , a framework which self-consistently mixes
holography in the infrared with perturbative degrees of freedom in the
ultraviolet via dynamical boundary fields.

As a quick recap we first discuss a simple example of quasinormal modes (QNMs)
found in holography, before moving onto the example of semiholography. QNMs
describe dissipation of linearized perturbations around equilibrium solutions,
i.e. the ring down of modes. As an example, consider the equation of motion
for a bulk massless scalar field:

$\displaystyle\nabla_{M}\nabla^{M}\Phi=0,$ (6)

living in the Eddington-Finkelstein Schwarzschild-AdS4 background:

$\displaystyle
ds^{2}=-\frac{L^{2}}{r^{2}}(1-Mr^{3})dt^{2}-2\frac{L^{2}}{r^{2}}dtdr+\frac{L^{2}}{r^{2}}(dx^{2}+dy^{2}).$
(7)

We can decompose the scalar field into Fourier modes
$\Phi(r,x^{\mu})\rightarrow e^{-ix\cdot k}f(r,k_{\mu})$, which leads to the
linear equation

$\displaystyle 0$
$\displaystyle=(Mr^{3}-1)f^{\prime\prime}+\frac{Mr^{3}+2-2ir\omega}{r}f^{\prime}+\frac{k^{2}r+2i\omega}{r}f$
(8)

Computing the poles gives rise to the typical "Christmas tree" structure found
in holography Horowitz:1999jd ; Kovtun:2005ev ; Berti:2009kk , see the red
points of Fig. 3.

We now add an additional dynamic scalar field at the boundary of the
holographic theory described above and follow the discussion in
Mondkar:2021qsf . The semiholographic action reads

$\displaystyle S={W_{\rm CFT}}[{h(x)}=-\beta{\chi}]-\frac{1}{2}\int
d^{3}x\;{\partial_{\mu}\chi\partial^{\mu}\chi},$ (9)

where $W_{\rm CFT}$ is the generating functional of the conformal field theory
(dual to the gravitational theory), sourced by the boundary scalar, $\chi$,
with $\beta$ governing the coupling between the two sectors. The equation of
motion for the scalar field is

$\displaystyle{\eta^{\mu\nu}\partial_{\mu}\partial_{\nu}\chi}=\beta{\frac{\delta
W_{\rm CFT}}{\delta h}}=\beta{\mathcal{H}},$ (10)

where $\mathcal{H}$ is the vacuum expectation value of the bulk scalar field,
$\Phi.$ The equations of motion of the dual gravity theory to the CFT are

$\displaystyle{R_{MN}-\frac{1}{2}RG_{MN}-3G_{MN}}$
$\displaystyle={\kappa(\nabla_{M}\Phi\nabla_{N}\Phi-\frac{1}{2}G_{MN}(\nabla_{P}\Phi)^{2})},$
(11) $\displaystyle{\nabla_{M}\nabla^{M}\Phi}$ $\displaystyle=0.$ (12)

The bulk scalar field, ${\Phi}$, has the near boundary expansion:

$\displaystyle{\Phi}={-\beta{\chi}}{+\ldots+\frac{3}{\kappa}\mathcal{H}r^{3}+\ldots}.$
(13)

Note that $\Phi$ is sourced by the dynamical boundary field, $\chi.$

The individual subsector Ward identities are

$\displaystyle\partial_{\mu}{t^{\mu\nu}_{\chi}}=-{\mathcal{H}\partial^{\nu}h},\quad\partial_{\mu}{\mathcal{T}^{\mu\nu}}={\mathcal{H}\partial^{\nu}h},$

where $t^{\mu\nu}_{\chi}$ is the boundary scalar stress tensor and
$\mathcal{T}^{\mu\nu}$ is the holographic stress tensor The energy momentum
tensor of the full system is conserved,
$\partial_{\mu}T^{\mu\nu}=\partial_{\mu}({t^{\mu\nu}_{\chi}}+{\mathcal{T}^{\mu\nu}})=0$.

We now study hybrid fluctuations of bulk dilaton ${\Phi}$ and the boundary
scalar field ${\chi}$. In the case of homogeneous QNM, we see in Fig. 3 that
varying the coupling $\beta\sqrt{T}\rightarrow\infty$ moves the "Christmas
tree" structure down into the complex plane and in the limit
$\beta\sqrt{T}\rightarrow\infty,$ the system exhibits emergent conformality.
Moreover, as we vary the coupling, there is a similar collision of poles on
the imaginary axis as in the previous section. In this case, the diffusion-
like poles become aligned on the "Christmas tree" structure for large
coupling.

Figure 3: Homogeneous quasinormal modes with arrows indicating the direction
of increasing coupling, $\beta$. Red points denote the $\beta=0$ case, which
is the typical "Christmas tree" pattern of the QNMs. A collision of poles
indicates the breaking of the softly broken global symmetry.

Figure 4: Evolution of the black hole energy and the kinetic energy in the
left and right panel, respectively, for a variety of couplings.

In Fig. 3, there is also a conspicious unstable mode in the upper half plane.
However, the full nonlinear simulations show that the apparently unstable mode
indicates that energy is driven from the holographic sector to the boundary
scalar at early times, as can be seen in Fig. 4. Eventually the irreversible
transfer of energy back to the holographic sector at later times is driven by
the quasi-hydro mode. As such, the total system has no instability!

We now turn our attention to the inhomogeneous case. In particular, we examine
the motion of the lowest lying poles for a demonstrative value of coupling, as
can be seen in Fig. 5. The first and last plot represent small and large $k$
behavior. The figures in between indicate the two collisions of poles that
occur for intermediate momenta. All collisions occur on the imaginary axis,
with the poles acquiring a real part after the collision. The first collision
occurs between the diffusion pole and the quasi-hydro pole for $k\sim
0.554\pi$, followed by a collision of the transient unstable mode (discussed
above) and the semiholographic pole for $k\sim 2.433\pi$. The two typical
holographic poles (and the rest of the "Christmas tree" poles deeper in the
complex plane) do not appreciably change as a function of momentum.

Finally, we also point out that in our model, we observe the formation of a
$k$-gap as seen in Fig. 6, which is characteristic of systems with a diffusive
to propagating mode crossover. This has been observed in quasi-hydrodynamic
frameworks Grozdanov:2018fic , where the $k-$gap can be related to a softly
broken global symmetry. Here, global shift symmetry of the theory is

$\displaystyle{\chi}\rightarrow{\chi}+{\chi_{0}},\quad{\Phi}\rightarrow{\Phi}-\beta{\chi_{0}}$
(14)

The k-gap has been observed experimentally in the dispersion of transverse
sound-like excitations of gallium Khusnutdinoff_2020 .

Figure 5: The first 6 poles in the complex plane for
$k=\\{0,0.554\pi,0.6\pi,2.4\pi,2.4334\pi,2.54\pi\\}$ (from top left to bottom
right), for $\beta\sqrt{T}=0.35$. We distinguish the different poles via the
following colors diffusion pole, quasi-hydro pole, transient unstable mode,
semiholographic pole and holographic QNM.

Figure 6: The formation of the k-gap, shown for the particular case
$\beta\sqrt{T}=0.15$.

##  4 Outlook

The analytic structure of the QGP is rich. Due to the presence of phase
transitions, such as the chiral phase transition and the one due to
strong/weak coupling dynamics, we should expect to see colliding poles in the
complex plane of theories relevant to the QGP. In the hybrid model outlined
Sec. 3, an immediate follow-up includes understanding the interaction of the
quasinormal mode spectrum of a holographic theory with Israel-Stewart
hydrodynamics mondkar:tbp . Another important direction is to see the
interplay of QNM poles with branch cuts typical to kinetic theory, especially
in the vicinity of a phase transition.

## Acknowledgments

The author is supported by the Austrian Science Fund (FWF), project no. J4406.

## References

  * (1) J. Adams et al. (STAR), Nucl. Phys. A 757, 102 (2005), nucl-ex/0501009
  * (2) P.B. Arnold, G.D. Moore, L.G. Yaffe, JHEP 01, 030 (2003), hep-ph/0209353
  * (3) M.P. Heller, Acta Phys. Polon. B 47, 2581 (2016), 1610.02023
  * (4) D.A. Teaney, _Viscous Hydrodynamics and the Quark Gluon Plasma_ (2010), pp. 207–266, 0905.2433
  * (5) A. Kurkela, U.A. Wiedemann, B. Wu, Eur. Phys. J. C 79, 965 (2019), 1905.05139
  * (6) A. Kurkela, U.A. Wiedemann, Eur. Phys. J. C 79, 776 (2019), 1712.04376
  * (7) S. Grozdanov, A.O. Starinets, JHEP 04, 080 (2019), 1812.09288
  * (8) I. Müller, Z. Phys. 198, 329 (1967)
  * (9) W. Israel, J. Stewart, Annals Phys. 118, 341 (1979)
  * (10) K. Rajagopal, F. Wilczek, Nucl. Phys. B 399, 395 (1993), hep-ph/9210253
  * (11) D.T. Son, Phys. Rev. Lett. 84, 3771 (2000), hep-ph/9912267
  * (12) D.T. Son, M.A. Stephanov, Phys. Rev. D 66, 076011 (2002), hep-ph/0204226
  * (13) E. Grossi, A. Soloviev, D. Teaney, F. Yan, Phys. Rev. D 102, 014042 (2020), 2005.02885
  * (14) E. Grossi, A. Soloviev, D. Teaney, F. Yan, Phys. Rev. D 104, 034025 (2021), 2101.10847
  * (15) A. Florio, E. Grossi, A. Soloviev, D. Teaney, Phys. Rev. D 105, 054512 (2022), 2111.03640
  * (16) A. Soloviev, EPJ Web Conf. 258, 05008 (2022), 2111.11375
  * (17) J. Engels, O. Vogt, Nucl. Phys. B 832, 538 (2010), 0911.1939
  * (18) J. Engels, F. Karsch, Phys. Rev. D 85, 094506 (2012), 1105.0584
  * (19) T. Faulkner, J. Polchinski, JHEP 06, 012 (2011), 1001.5049
  * (20) E. Iancu, A. Mukhopadhyay, JHEP 06, 003 (2015), 1410.6448
  * (21) A. Mukhopadhyay, F. Preis, A. Rebhan, S.A. Stricker, JHEP 05, 141 (2016), 1512.06445
  * (22) S. Banerjee, N. Gaddam, A. Mukhopadhyay, Phys. Rev. D 95, 066017 (2017), 1701.01229
  * (23) A. Kurkela, A. Mukhopadhyay, F. Preis, A. Rebhan, A. Soloviev, JHEP 08, 054 (2018), 1805.05213
  * (24) C. Ecker, A. Mukhopadhyay, F. Preis, A. Rebhan, A. Soloviev, JHEP 08, 074 (2018), 1806.01850
  * (25) T. Mitra, S. Mondkar, A. Mukhopadhyay, A. Rebhan, A. Soloviev, Phys. Rev. Res. 2, 043320 (2020), 2006.09383
  * (26) T. Mitra, S. Mondkar, A. Mukhopadhyay, A. Rebhan, A. Soloviev (2022), 2211.05480
  * (27) G.T. Horowitz, V.E. Hubeny, Phys. Rev. D 62, 024027 (2000), hep-th/9909056
  * (28) P.K. Kovtun, A.O. Starinets, Phys. Rev. D 72, 086009 (2005), hep-th/0506184
  * (29) E. Berti, V. Cardoso, A.O. Starinets, Class. Quant. Grav. 26, 163001 (2009), 0905.2975
  * (30) S. Mondkar, A. Mukhopadhyay, A. Rebhan, A. Soloviev, JHEP 11, 080 (2021), 2108.02788
  * (31) S. Grozdanov, A. Lucas, N. Poovuttikul, Phys. Rev. D 99, 086012 (2019), 1810.10016
  * (32) R.M. Khusnutdinoff, C. Cockrell, O.A. Dicks, A.C.S. Jensen, M.D. Le, L. Wang, M.T. Dove, A.V. Mokshin, V.V. Brazhkin, K. Trachenko, Physical Review B 101 (2020)
  * (33) S. Mondkar, A. Mukhopadhyay, A. Rebhan, A. Soloviev, Manuscript in preparation (2022), 2212.XXXXX