2021

[1]Ravi Bhatt These authors contributed equally to this work.

These authors contributed equally to this work.

[1]Department of Computer Science & Engineering, Indian Institute of
Technology Ropar, Rupnagar, 140001, Punjab, India

2]Department of Computer Science and Communication, Østfold University
College, Halden, 1757, Norway

# Pho(SC)-CTC - A Hybrid Approach Towards Zero-shot Word Image Recognition

<EMAIL_ADDRESS>Anuj Rai<EMAIL_ADDRESS>Sukalpa Chanda
<EMAIL_ADDRESS>Narayanan C. Krishnan<EMAIL_ADDRESS>*  [

###### Abstract

Annotating words in a historical document image archive for word image
recognition purpose demands time and skilled human resource (like historians,
paleographers). In a real-life scenario, obtaining sample images for all
possible words is also not feasible. However, Zero-shot learning methods could
aptly be used to recognize unseen/out-of-lexicon words in such historical
document images. Based on previous state-of-the-art method for zero-shot word
recognition “Pho(SC)Net”, we propose a hybrid model based on the CTC framework
(Pho(SC)-CTC) that takes advantage of the rich features learned by Pho(SC)Net
followed by a “connectionist temporal classification” (CTC) framework to
perform the final classification. Encouraging results were obtained on two
publicly available historical document datasets and one synthetic handwritten
dataset, which justifies the efficacy of Pho(SC)-CTC and Pho(SC)Net.

###### keywords:

PHo(SC)Net , CTC , Zero-shot word recognition , Historical Documents , Zero-
shot learning , Word recognition.

##  1 Introduction

Historical documents narrates condition about human societies in the past.
Easy availability and usability of image acquisition devices have led to the
digitization and archival of such historical documents. Searching for
important and relevant information from the large pool of images in those
digital archives is a challenging task. Earlier, end-to-end transcription of
the text using OCR was a popular way to achieve this goal. However, the
performance of OCR often depends on character-segmentation accuracy, which is
error prone, specially in the context of cursive handwritten text. Moreover,
end-users of such digital archives (historians, paleographers etc.) are often
not interested in an end-to-end transcription of the text, rather they are
interested in specific document pages where a query incident, place name,
person name has been mentioned. To cater to this requirement, word-spotting
and recognition techniques play an important role as they help directly in
document indexing and retrieval.

Deep learning models have been quite successful in many document analysis
problems. Thus, it is a natural fit for word recognition in historical
documents as well. However, training deep networks for this problem is a
challenging task due to many reasons. The lack of a large corpus of word
images, partly due to the changes in the appearance of characters and
spellings of words over centuries, makes it difficult to train a deep model.
The complexity is further compounded by the requirement of learning large
number of word labels, using only a small set of samples. In addition, the
historical documents exhibit many undesirable characteristics such as torn and
blurred segments, unwanted noise and faded regions, handwritten annotations by
historians and artefacts; all of them contributing to the difficulty of the
task.

Figure 1: Difference between word spotting and recognition: Given a word image
as input, word recognition is finding the word in the lexicon that is the most
likely to be the word present in the image. Word spotting involves finding
image segments similar to the query image. Document image belongs to the GW
dataset.

Word spotting and recognition sound similar to each other but are two
different tasks. This is illustrated in Figure 1. Given an image containing a
word as input, a word recognition system identifies the word from the lexicon
that is present in the image. On the other hand, word spotting refers to
detecting other image segments in the document exhibiting patterns similar to
the query image. This work is focused on the former problem and extends it to
the zero-shot learning (ZSL) setting. Classical word recognition involves
training a machine learning model to recognise the words given the images
containing them. It is assumed that the test query images also contain only
the words that were presented during the training. However, in the ZSL
setting, the test query images can contain words that the model did not see
during training. This is a more challenging task requiring a visual
representation (akin to the semantic embedding in ZSL literature) that can
bridge the set of seen and unseen words.

In this work, we propose a machine learning model based on CTC framework that
takes advantage of the rich features learned by Pho(SC)Net such that it can be
used to recognise out-of-lexicon words. We call this model Pho(SC)-CTC.
Overall we make the following contributions

  * •

We proposed a CTC-based framework for zero-shot word image recognition.

  * •

We investigated the efficacy of Pho(SC)Net features while fusing them in a CTC
framework for zero-shot word recognition.

  * •

The proposed hybrid model Pho(SC)-CTC_P significantly outperforms the state-
of-the-art method “Pho(SC)Net ” for zero-shot word recognition accuracy on a
real-world dataset and a synthetic dataset.

##  2 Related Work

Word spotting and recognition have been well explored over the last 25 years,
with a spurt in deep learning based solutions in the recent past sudholtF16 ,
himanis2017 ,tomas2017 , kang2018gcpr ,chanda2018 , krishnandas2018 ,dutta2018
,alex2009 ,krishnan2019hwnet ,prl_alicia_2020 ,wolf20 . The seminal work
sudholtF16  on word spotting involved training of a CNN for predicting the
PHOC representation almazanPami2014 . This work considered both contemporary
as well as historical document images in their experiments. The proposed
system can be used in both “Query By Example”(QBE) and “Query By String”(QBS)
settings. In krishnandas2018 , the authors proposed an End2End embedding
scheme to learn a common representation for word images and its labels,
whereas in dutta2018 , a CNN-RNN hybrid model was proposed for handwritten
word recognition. The method proposed in himanis2017  uses a deep recurrent
neural network (RNN) and a statistical character language model to attain high
accuracy for word spotting and indexing. A recent work on robust learning of
embeddings presents a generic deep CNN learning framework that includes pre-
training with a large synthetic corpus and augmentation methods to generate
real document images, achieving state-of-the-art performance for word spotting
on handwritten and historical document images krishnan2019hwnet . In
KRISHNAN2020 , the authors introduce a novel semantic representation for word
images that uses linguistic and contextual information of words to perform
semantic word spotting on Latin and Indic scripts. From the brief discussion
it is evident that though deep learning based methods have been used in the
recent past for word spotting and recognition tasks, zero-shot word
recognition - recognizing a word image without having seen examples of the
word during training; has not been studied. Only chanda2018  explore Latin
script word recognition problem in the ZSL framework; however the number of
test classes were limited and no publicly available datasets were used in
their experiments.

ZSL techniques have demonstrated impressive performances in the field of
object recognition/detection Li_2018_CVPR , Zhang_2018_CVPR ,
Annadani_2018_CVPR ,Niu_2018_CVPR , Li2019 ,Xie_2019_CVPR . Li et.al.
Li_2018_CVPR  propose an end-to-end model capable of learning latent
discriminative features jointly in visual and semantic space. They note that
user-defined signature attributes loose its discriminativeness in
classification because they are not exhaustive even though they are
semantically descriptive. Zhang et. al. Zhang_2018_CVPR , use a Graph
Convolutional Network (GCN) along with semantic embeddings and the categorical
relationships to train the classifiers. This approach takes as input semantic
embeddings for each node (representing the visual characteristic). It predicts
the visual classifier for each category after undergoing a series of graph
convolutions. During training, the visual classifiers for a few categories are
used for learning the GCN parameters. During the test phase, these filters are
used to predict the visual classifiers of unseen categories Zhang_2018_CVPR .
In Annadani_2018_CVPR , the objective functions were customized to preserve
the relations between the labels in the embedding space. In akatapami2016 ,
attribute label embedding methods for zero-shot and few-shot learning systems
were investigated. Later, a benchmark and systematical evaluation of zero-shot
learning w.r.t. three aspects, i.e. methods, datasets and evaluation protocol
was done in akatacvpr2017 . In paulcvpr19 , the authors propose a conditional
generative model to learn latent representations of unseen classes using the
generator trained for the seen classes. The synthetically generated features
are used to train a classifier for predicting the labels of images from the
unseen object category.

Figure 2: 11 primary shape attributes: ascender, descender, left small semi-
circle, right small semi-circle, left large semi-circle, right large semi-
circle, circle, vertical line, diagonal line, diagonal line at a slope of 135
degrees, and horizontal line

In summary, the current ZSL methods are mainly focused towards object
detection. One must note that semantic attribute space in ZSL-based object
detection is rich as attributes like colour and texture pattern play a crucial
role. But in case of ZSL-based word recognition the semantic attribute space
is rather constrained due to the absence of such rich visual features.
Further, the major bottleneck for ZSL based word recognition is the absence of
the semantic embedding or attribute signature that establishes the
relationship between the various word labels. This is further challenged by
large number of word classes with relatively few examples.
Graves et.al. ctc2006  introduces connectionist temporal classification, which
is a framework that can be used to train sequential models and decode their
output. The paper shows that CTC outperformed the baseline HMM model on speech
recognition task. This motivates us to use CTC on handwritten word recognition
as recognising the character sequence in the goal for both the approaches. The
ZSL techniques discussed depend on the semantic embedding, CTC on the other
hand is free from these dependencies. In this work, we propose a novel CTC-
based method which takes into account a rich feature representation method
based for zero-shot word recognition. We hypothesize that the CTC model will
be able to recognize the unseen words without the need for the visual
description of the characters in the word. Further prior knowledge about the
visual description can be successfully transferred to the CTC framework for
improving its performance.

Figure 3: Architecture of the multi-task Pho(SC)Net

##  3 Methodology

We begin by defining our problem of interest. Let
$\mathcal{S}=\\{(x_{i},y_{i},c(y_{i}))\\}_{i=1}^{N}$, where
$x_{i}\in\mathcal{X},y_{i}\in\mathcal{Y}^{s},c(y_{i})\in\mathcal{C}$,
$\mathcal{S}$ stands for the training examples of seen word labels, $x_{i}$ is
the image of the word, $y_{i}$ is the corresponding word label in
$\mathcal{Y}^{s}=\\{s_{1},s_{2},\ldots,s_{K}\\}$ consisting of $K$ discrete
seen word labels, and $c(y_{i})\in\mathcal{R}^{Q}$ is the unique word label
embedding or attribute signature that models the visual relationship between
the word labels. In addition, we have a disjoint word label set
$\mathcal{Y}^{U}=\\{u_{1},\ldots,u_{L}\\}$ of unseen labels, whose attribute
signature set $U=\\{u_{l},c(u_{l})\\}_{l=1}^{L},c(u_{l})\in\mathcal{C}$ is
available, but the corresponding images are missing. Given $\mathcal{S}$ and
$\mathcal{U}$, the task of zero-shot word recognition is to learn a classifier
$f_{zsl}:\mathcal{X}\rightarrow\mathcal{Y}^{u}$ and in the generalized zero-
shot word recognition, the objective is to learn the classifier
$f_{gzsl}:\mathcal{X}\rightarrow\mathcal{Y}^{u}\cup\mathcal{Y}^{s}$. In the
absence of training images from the unseen word labels, it is difficult to
directly learn $f_{zsl}$ and $f_{gzsl}$. Instead, we learn a mapping ($\phi$)
from the input image space $\mathcal{X}$ to the attribute signature space
$\mathcal{C}$ that is shared between $\mathcal{Y}^{s}$ and $\mathcal{Y}^{u}$.
The word label for the test image $x$ is obtained by performing a nearest
neighbor search in the attribute signature space using $\phi(x)$. Thus, the
critical features of zero-shot word recognition are the attribute signature
space $\mathcal{C}$ that acts as a bridge between the seen and unseen word
labels and the mapping $\phi$. In this work, we propose a novel attribute
signature representation that can effectively model the visual similarity
between seen and unseen word labels. The mapping $\phi$ is modeled as a deep
neural network. A very popular word label representation that can serve as the
attribute signature for our problem is the pyramidal histogram of characters
(PHOC). A PHOC is a pyramidal binary vector that contains information about
the occurrence of characters in a segment of the word. It encodes the presence
of a character in a certain split of the string representation of the word.
The splits of different lengths result in the pyramidal representation. The
PHOC allows to transfer information about the attributes of words present in
the training set to the test set as long as all attributes in the test set are
also present in the training set. However, this constraint may be violated in
the context of zero-shot word recognition. Further, the PHOC also misses the
visual shape features of the characters as they appear in a word image. We
also observe these limitations of PHOC from our experiments on unseen word
recognition. Literature also suggests parity between various representations
that only encode the occurrence and position of characters within a
wordembedeval17 . Thus we are motivated to present a novel attribute signature
representation that complements the existing word label characterizations.

###  3.1 The Pyramidal Histogram of Shapes

We propose the pyramidal histogram of shapes (PHOS) as a robust bridge between
seen and unseen words. Central to the PHOS representation is the assumption
that every character can be described in terms of a set of primitive shape
attributes chanda2018 . We consider the following set of primitive shapes :-
ascender, descender, left small semi-circle, right small semi-circle, left
large semi-circle, right large semi-circle, circle, vertical line, diagonal
line, diagonal line at a slope of 135 degrees, and horizontal line. These
shapes are illustrated in Fig. 2. Only the counts of these shapes is
insufficient to adequately characterize each word uniquely. Inspired by the
pyramidal capture of occurrence and position of characters in a word, we
propose the pyramidal histogram of shapes that helps in characterizing each
word uniquely.

The process of capturing the PHOS representation for a word is illustrated in
Fig.5. At the highest level of the pyramid, there exists only a single
segment, which is the entire word. At every level $h$ of the pyramid, we
divide the word into $h$ equal segments. Further, at every level $h$, we count
the occurrence of the 11 primary shapes in every $h$ segments of the word. The
concatenation of the count vectors for every segment in a level and across all
the levels of the pyramid results in the PHOS representation of the word. In
this work, we have used levels 1 through 5, resulting in a PHOS vector of
length (1+2+3+4+5)*11 = 165. Thus the PHOS vector encodes the occurrence and
relative position of the shapes in the word string.

Figure 4: Pyramidal structure of PHOS representation

For example, let us consider a pair of anagrams “listen” and “silent” for 3
levels of segmentation. The segments at three levels for “listen” are:
$L_{1}=\\{listen\\},L_{2}=\\{lis,ten\\},L_{3}=\\{li,st,en\\}$. Similarly for
“silent”: $L_{1}=\\{silent\\}$, $L_{2}=\\{sil,ent\\}$, $L_{3}=\\{si,le,nt\\}.$
The corresponding shape counts and their PHOS vector at each level for both
words has been illustrated in Fig.5.

For the zero-shot word recognition problem, it is important to encode the
occurrence and relative position of characters within a word, as well as that
of visual shapes. Therefore, we propose to use the concatenated PHOC and PHOS
vector of a word as its attribute signature representation $\mathcal{C}$.
Thus, the attribute signature representation for the word label $y_{i}$ is
$[c_{c}(y_{i}),c_{s}(y_{i})]$, where $c_{c}(y_{i})$ and $c_{s}(y_{i})$ are the
PHOC and PHOS representations respectively.

###  3.2 Pho(SC)Net Architecture

Having defined the augmented attribute signature space $\mathcal{C}$, our next
objective is to learn the mapping $\phi$ to transform an input word image into
its corresponding attribute signature representation - PHOC+PHOS vector. We
use the architecture of SPP-PhocNetsudholtF16  as the backbone for the
Pho(SC)Net ($\phi$) that is used to predict the combined representation. The
Pho(SC)Net is a multi-task network with shared feature extraction layers
between the two tasks (PHOC and PHOS). The shared feature extraction network
is a series of convolution layers, followed by a spatial pyramid pooling (SPP)
layer. The SPP layer facilitates the extraction of features across multiple
image resolutions. The Pho(SC)Net separates out into two branches after the
SPP layer to output the two representations. The two branches contain two
independent fully connected layers. As the PHOC representation is a binary
vector, the PHOC branch ends with a sigmoid activation layer. On the other
hand the PHOS representation being a non-negative vector, the PHOS branch ends
with a ReLU activation layer. The multi-task Pho(SC)Net architecture is
illustrated in Fig.3.

The output of the Pho(SC)Net for an input word image is the vector
$\phi(x)=[\phi_{C}(x),\phi_{S}(x)]$, where $\phi_{C}(x)$ and $\phi_{S}(x)$ are
the predicted PHOC and PHOS representations respectively. Given a mini batch
of $B$ instances from the training set consisting of seen word images and
their labels, we minimize the following loss function during training.

$L=\sum_{i=1}^{B}\lambda_{c}L_{c}(\phi_{c}(x_{i}),c_{c}(y_{i}))+\lambda_{s}L_{s}(\phi_{s}(x_{i}),c_{s}(y_{i}))$
(1)

where $L_{c}(\phi_{c}(x_{i}),c_{c}(y_{i}))$ is the cross entropy loss between
the predicted and actual PHOC representations,
$L_{s}(\phi_{s}(x_{i}),c_{s}(y_{i}))$ is the squared loss between the
predicted and actual PHOS representations, and $\lambda_{c},\lambda_{s}$ are
hyper-parameters used to balance the contribution of the two loss functions.

Given a test image $x$, the Pho(SC)Net is used to predict the PHOC and PHOS
representations to obtain the predicted attribute signature representation
$[\phi_{c}(x),\phi_{s}(x)]$. The word whose attribute signature representation
has the highest similarity (measured as cosine similarity) with
$[\phi_{c}(x),\phi_{s}(x)]$ is the predicted word label for the test image, in
the conventional ZSL setting, as defined below

$\hat{y}=\operatorname*{argmax}_{k\in\mathcal{Y}^{U}}\cos([\phi_{c}(x),\phi_{s}(x)]^{T}[c_{c}(k),c_{s}(k)])$
(2)

Figure 5: Overview of CTC based neural network for handwritten word
recognition

###  3.3 Connectionist temporal classification (CTC)

Given a dataset of images containing words and the word label, it is difficult
and time consuming to annotate each image. Fortunately, the Connectionist
temporal classification framework, can train neural networks with sequential
input and decode the output. In the case of handwritten word recognition, each
character in the word does not need to be aligned to its exact location in the
image.
For a given input image $X$ and label $Y$, there can be various alignments
possible between the input and the output.CTC sums up the probabilities of all
such alignments to calculate the likelihood $p(Y\mid X)$. Suppose we use the
trivial approach of collapsing consecutive similar characters to one character
to obtain the label from the decoded output. In that case, it becomes
impossible to predict words that have consecutive duplicate characters. To
solve this problem, CTC introduces a unique token called the blank token. If
the output has two consecutive identical characters, the decoded output must
have a blank character between the two characters. For example, consider the
following outputs : 1) $AAAB=AB$ and 2) $AA-AB=AAB$
In the first example multiple instance of a character are considered to be a
single character by the collapsing function, while in second example as two
A’s are separated by a ’-’(special character in this case) they are considered
to be two different characters.
The easiest way to decode the RNN output is to take the most probable
character at each time step, the best path decoding algorithm. There are two
more algorithms to decode the RNN - output beam search and word beam search.
In beam search decoding, text candidates (beams) are created and scored
iteratively. An empty beam and a corresponding score are added to a list of
beams at the start. The algorithm then iterates through the output matrix’s
all-time steps. Only the highest-scoring $k$ beams from the previous time-step
are retained at each time-step. Here $k$ is the beamwidth. Word beam search
makes use of a language model; therefore, using it for zero-shot word
recognition will not be useful.
The decoded output is further passed into a collapsing function as a single
character could be encoded by duplicate characters and the special character.
The decoding function removes duplicate characters and the special character
and returns the predicted label.

###  3.4 Pho(SC)-CTC Architecture

Our CTC model has two primary components, first is a CNN, followed by a RNN (2
Bi-directional Stacked LSTM cells). In this hybrid model “Phos(SC)-CTC”, we
considered the CNN part of the Phos(SC)Net, i.e. we feed the RNN part of the
CTC model with the output of the last Convolution layer (which is the input to
the first dense layer in the PhoscNet architecture) (Fig 3). Max pooling
layers are added such that the output of the CNN can be passed to the RNN. The
Pho(SC)-CTC architecture was trained using the following two different
approaches.

  * •

Pho(SC)-CTC_Pre-Trained-Weights: The weights of the convolutional layers of
Pho(SC)-CTC are initialized with the trained weights of Pho(SC)Net.
Henceforth, we will refer to this approach as Pho(SC)-CTC_P

  * •

Pho(SC)-CTC: All weights in the entire network are obtained after training the
whole end-to-end network from scratch.

Given a mini-batch of B instances from the training set consisting of seen
word images and their labels, we minimize the following loss function during
training.

$L=-\sum_{i=1}^{B}\ln(p(y_{i}\mid x_{i})$ (3)

Where $p(y_{i}\mid x_{i})$ is the probability of getting $y_{i}$ as output
given $x_{i}$ as input.

Split | Train Set | Validation Set | Test (Seen Classes) | Test (Unseen Classes)
---|---|---|---|---
MFU Dataset
MFU 2000 | 36000(2000) | 3600 | 4000 | 8000(1000)
George Washington (GW Dataset)
Split 1 | 1585(374) | 662(147) | 628(155) | 121(110)
Split 2 | 1657(442) | 637(165) | 699(155) | 114(100)
Split 3 | 1634(453) | 709(164) | 667(148) | 105(89)
Split 4 | 1562(396) | 668(149) | 697(163) | 188(152)
IAM Handwriting Dataset
ZSL Split | 30414(7898) | 2500(1326) | 1108(748) | 538(509)
Standard Split | 34549(5073) | 9066(1499) | 8318(1355) | 1341(1071)

Table 1: Details of dataset used for experiments Numbers inside the parentheses represent the number of word classes in the set. Split | PHOC | PHOS | Pho(SC)
---|---|---|---
MFU Dataset
MFU 2000 | .94 | .96 | .98
GW Dataset
GW Split 1 | .46 | .61 | .68
GW Split 2 | .64 | .72 | .79
GW Split 3 | .65 | .71 | .80
GW Split 4 | .35 | .62 | .60
IAM Handwriting Dataset
ZSL Split | .78 | .79 | .86
Standard Split | .89 | .88 | .93
Table 2: ZSL Accuracy on all the splits

##  4 Experiments

###  4.1 Datasets

We validate the effectiveness of the Pho(SC) representation for the zero-shot
word recognition problem on the following three datasets.

####  4.1.1 Most Frequently Used Words (MFU) Dataset

A synthetic dataset was created from the most frequently used English words
list. $\mathcal{Y}^{s}$ was chosen to be the first 2000 words and the
subsequent 1000 words were made part of $\mathcal{Y}^{u}$. Eight handwriting
fonts were used to generate a total of 16000-word images (split into 12000 for
training and 4000 for testing) for $\mathcal{Y}^{s}$ and 8000 for
$\mathcal{Y}^{u}$. We ensure that the word labels of the 4000 test images of
$\mathcal{Y}^{s}$ are present in the corresponding training set. These details
are summarized in Table 1.

####  4.1.2 George Washington (GW) Dataset

The George Washington datasetgwdataset  exhibits homogeneous writing style and
contains 4894 images of 1471 word labels. We used the lower-case word images
from the standard four-fold cross-validation splits accompanying the dataset
to evaluate the Pho(SC)Net. We modified the validation and test sets to suit
the zero-shot word recognition problem. Specifically, the test sets in each
split was further divided into two parts: seen and unseen word label images,
and the validation set contained only seen word label images. The details for
each set across all the splits are presented in Table 1.

####  4.1.3 IAM Handwriting (IAM) Dataset

IAM handwriting datasetiam_dataset_marti2002  is a multi-writer dataset that
consists of 115320 word-images, from 657 different writers. We used the
lowercase word-class images from this dataset to create train, validation, and
test sets. Specifically we created two different splits. In the first split
(overlapping writers ZSL split), we ensured that the writers in the test set
are also part of the training set. Further, the test set contained both unseen
and seen word labels, while the train and validation sets contained only seen
word images. In the second split (standard ZSL split) derived from the
standard split accompanying the dataset we removed the unseen word images from
the validation set, and divided the test split further into seen word and
unseen word images. The standard ZSL split has an additional challenge as the
writers in the training, validation, and test sets are non-overlapping. The
number of images and the (seen and unseen) word labels for each split is
presented in Table 1.

Figure 6: Examples of word images and their labels from the three datasets

The training set of all the three datasets were further augmented through
shearing, and addition of Gaussian noise. The size of a word image (in the
training, validation, and test sets) depends on the length of the word and the
handwriting style (font), but we needed images of uniform size for training.
Hence, the binarized images were resized to the best fitting sizes (without
changing the aspect ratio) and then padded with white pixels to get images of
size 250*50. Fig 6 presents a few examples of word images from the three
datasets.

Split | PHOC | PHOS | Pho(SC)
---|---|---|---
|  $A_{u}$ |  $A_{s}$ |  $h$ |  $A_{u}$ |  $A_{s}$ |  $h$ |  $A_{u}$ |  $A_{s}$ |  $h$
MFU Dataset
MFU 2000 | .74 | .99 | .85 | .92 | .93 | .92 | .92 | .99 | .96
GW Dataset
GW Split 1 | .01 | .96 | .03 | .24 | .95 | .39 | .15 | .97 | .27
GW Split 2 | .10 | .98 | .19 | .30 | .98 | .46 | .30 | .98 | .46
GW Split 3 | .09 | .97 | .17 | .40 | .94 | .56 | .35 | .96 | .51
GW Split 4 | .04 | .94 | .08 | .31 | .92 | .47 | .25 | .95 | .39
IAM Dataset
ZSL Split | .58 | .88 | .70 | .71 | .82 | .76 | .77 | .93 | .84
Standard Split | .46 | .87 | .61 | .64 | .82 | .72 | .70 | .90 | .79

Table 3: Generalized ZSL accuracy on various splits. $A_{u}$ = Accuracy with
unseen word classes, $A_{s}$ = Accuracy with seen word classes, Generalized
ZSL accuracy, $h$ = Harmonic mean of $A_{u}$ and $A_{s}$.

Split | PHO(SC)-CTC | PHO(SC)-CTC_P | Pho(SC)NET
---|---|---|---
|  $A_{u}$ |  $A_{s}$ |  $h$ |  $A_{u}$ |  $A_{s}$ |  $h$ |  $A_{u}$ |  $A_{s}$ |  $h$
MFU Dataset
MFU 2000 | .99 | 1.0 | .99 | 1.0 | 1.0 | 1.0 | .92 | .99 | .96
GW Dataset
GW Split 1 | .50 | .95 | .65 | .50 | .91 | .64 | .15 | .97 | .27
GW Split 2 | .63 | .95 | .76 | .75 | .96 | .84 | .30 | .98 | .46
GW Split 3 | .62 | .90 | .73 | .70 | .88 | .78 | .35 | .96 | .51
GW Split 4 | .44 | .91 | .59 | .60 | .89 | .72 | .25 | .95 | .39
IAM Dataset
ZSL Split | .65 | .82 | .73 | .65 | .81 | .76 | .77 | .93 | .84
Standard Split | .54 | .82 | .65 | .56 | .83 | .72 | .70 | .90 | .79

Table 4: Generalized ZSL accuracy on various splits. $A_{u}$ = Accuracy with
unseen word classes, $A_{s}$ = Accuracy with seen word classes, $h$ = Harmonic
mean of $A_{u}$ and $A_{s}$.

###  4.2 Training and Baselines

The Pho(SC)Net was trained using an Adam optimizer with learning rate 1e-4,
weight decay set as 5e-5, momentum at 0.9. The batch size is kept as 16. The
hyper-parameters $\lambda_{c}$ and $\lambda_{s}$ were fine tuned using the
validation set. The final values chosen for these parameters are 1 and 4.5
respectively. We also conducted ablation studies with the individual PHOCNet
(SPP-PHOCNet) and the PHOSNet counter part. These two networks were also
trained in a similar fashion for all the three datasets. This helps to
investigate the effect of adding visual shape representations to the default
attribute signature vector (PHOC vector). Early stopping was applied to the
training process using reducedLR on plateau on the validation set. As this is
the first work to perform zero-shot word recognition on publicly available
benchmark datasets, we compare Pho(SC)Net with the classical SPP-PHOCNet model
(trained using the settings insudholtF16 ).

Figure 7: Examples of correct and incorrect predictions in Generalized ZSL
setting from the standard IAM Split Figure 8: Generalized ZSL accuracy and
Character Error Rate (CER) of Pho(SC)-CTC on various datasets.

The Pho(SC)-CTC and Pho(SC)-CTC_P models were trained using an Adam optimizer
with a learning rate of 0.001. Early stopping was applied on the validation
set to avoid over-fitting. We compare Pho(SC)-CTC with Pho(SC)-CTC_P and
Pho(SC)net.
There is no defined search space for a CTC framework as the network’s output
is a sequence of characters. So we cannot compare it with the conventional
zeroshot learning results of Pho(SC)net. Thus we compare both the models in
the generalized setting. 111https://github.com/raviRB/Pho-SC--CTC.

###  4.3 Performance Metrics

The top-1 accuracy of the model’s prediction is used as the performance metric
for all the experiments. The top-1 accuracy measures the proportion of test
instances whose predicted attribute signature vector is closest to the true
attribute signature vector. At test time, in the ZSL setting, the aim is to
assign an unseen class label, i.e. $\mathcal{Y}^{u}$ to the test word image
and in the generalized ZSL setting (GZSL), the search space includes both seen
and unseen word labels i.e. $\mathcal{Y}^{u}\cup\mathcal{Y}^{s}$. Therefore,
in the ZSL setting, we estimate the top-1 accuracy over $\mathcal{Y}^{u}$. In
the GZSL setting, we determine the top-1 accuracy for both $\mathcal{Y}^{u}$
and $\mathcal{Y}^{s}$ independently and then compute their harmonic mean.
The output of CTC is a sequence of characters, thus we can calculate the
character error rate (CER) for a set of testing images T, using the following
formula.

$CER=\sum_{i=1}^{T}\frac{ED(y_{ip},y_{i})}{\lvert y_{i}\rvert}$ (4)

here $y_{ip}$ is the predicted label for the input image $x_{i}$ and $y_{i}$
is the true label, ED(a,b) is the minimum number of insertion,deletion and
substitutions required to change a to b, i.e the edit distance between a and
b.

Figure 9: Generalized ZSL accuracy and Character Error Rate (CER) of
Pho(SC)-CTC_P on various datasets.

##  5 Results and Discussion

The accuracies for the unseen word labels under the conventional ZSL setting
is presented in Table 2. Overall, it is observed that the PHOC representation
is not well-suited for predicting unseen word labels. However, the PHOS
representation is more accurate in predicting the unseen word labels. Further,
the combination of both the vectors (Pho(SC)) results in a significant
improvement (on an average $>5\%$) in the unseen word prediction accuracy. The
MFU dataset, on the account of being synthetically generated and noise-free,
has the highest unseen word recognition accuracy. We obtained the least
accuracy on the GW dataset split 4. We attribute this low accuracy to a rather
large number of unseen word classes in the test set for this split.

It is also observed that the accuracy of the model on MFU and IAM datasets are
significantly higher than that of any of the GW splits. This is explained by
observing the number of seen classes the model is presented during training.
Both MFU and IAM datasets have more than 2000 seen word labels, allowing the
model to learn the rich relationships between the word labels as encoded by
the attribute signatures. Learning this relationship is essential for the
model to perform well on unseen word images.

We also observe that using only PHOS as the attribute signature results in
better performance than using PHOC on datasets that have homogeneous writing
style. This is inferred by noticing the significant increase in the unseen
word accuracy of over 14% on the GW dataset splits by PHOS over PHOC.

Table 3 presents the test seen and unseen word accuracies, along with the
harmonic mean for the GZSL setting. The high accuracies on the seen word
labels and low accuracies on unseen word labels for the model using only PHOC
seems to suggest that the PHOC representation is more suitable for scenarios
where the word labels across the train and test are similar. In contrast, the
PHOS model has marginally lower accuracies on the test seen word labels (in
comparison to PHOC), but significantly higher accuracies on the unseen word
labels (again in comparison to PHOC). This indicates that looking at visual
shapes is more reliable when the train and test sets contain different word
labels. Further, we also observe that the combined model yields higher
performance for both MFU and IAM datasets that have a large number of seen
classes. However, on the GW dataset, that has a significantly smaller number
of seen word classes, the PHOS model is able to even outperform the combined
model. The poor performance of the PHOC model that is pulling down the
performance of the combined model on the unseen classes for the GW dataset can
also be attributed to the small number of seen word classes that the model is
exposed to during training. However, note that the PHOC model achieves
significantly higher accuracies on the test images of the seen word classes,
indicating that the model has not overfit, but is biased towards these
classes. Table 4 presents the results for the various CTC models considered in
this work.
Pho(SC)-CTC performs better than Pho(SC)net on the GW dataset and MFU dataset,
but for the IAM dataset Pho(SC)Net outperforms Pho(SC)-CTC by a considerable
margin. Pho(SC)-CTC_P performs better than Pho(SC)-CTC on GW Split 2, GW Split
3, GW Split 4, and IAM dataset in standard setting. In other datasets, both
the models show similar performance. We observe that Pho(SC)-CTC_P performs
better when there is not much heterogeneity in the dataset, furthermore
Pho(SC)Net is able to capture some rich features from the images, which are
not being extracted by Pho(SC)CTC.

####  5.0.1 Error Analysis

Fig. 7 illustrates a few predictions by the Pho(SC) model on the IAM dataset
under the GZSL setting. It can observed that even when the model incorrectly
predicts the word, there is a good overlap between the characters of the
predicted and true word label.
Fig. Figures 9 and 8 shows the CER for all the datasets with the word
accuracies. It can be observed that the CER is quite less even when the word
accuracies are high, signifying that there is a good overlap between the
characters of the predicted and true word label.

Fig. 10 presents the confusion matrix for predictions on the IAM dataset
standard split in the GZSL setting. The confusion matrix has been computed
between words of different lengths to uncover any biases of the model (if
any). It is difficult to visualize the class specific confusion matrix as
there are over 1000 word labels with very few (often only 1) images per word
label. The length of the predicted word labels is mostly within a range of the
length of the true word label. In general, the model is not biased towards
words of any specific length. The high values along the diagonal indicates
that the model is often predicting the word of the correct length (except when
the word length is 15, for which there are only 2 samples and 2 classes).

##  6 Conclusion

We propose a hybrid model that combines novel attribute signature
representation (PHOSC) (that characterizes the occurrence and position of
elementary visual shapes and characters in a word) along with LSTM in a CTC
framework. Our experiments demonstrate the effectiveness of PHOSC-CTC for
predicting unseen word labels in the generalized ZSL setting for two datasets
with more homogeneous handwriting and lesser number of training classes, while
the classical Pho(SC)Net representation is more suitable for recognition of
unseen word classes with more heterogeneous handwriting and lesser number of
training classes. Future directions to this work include extending the PHOS
representations to include other non-Latin scripts like Arabic, Chinese, and
Indic scripts.

Figure 10: Lengthwise confusion matrix(normalised) for predicted word
length(left axis) and true word length(bottom) for Standard IAM Split. Labels
on top(red) indicate the number of word classes in lexicon of that length
while on right(green) represent the number of samples in test set for the
corresponding word length.

## References

  *   * (1) S. Sudholt, G.A. Fink, in _15th International Conference on Frontiers in Handwriting Recognition, 2016_ (2016), pp. 277–282
  * (2) T. Bluche, S. Hamel, C. Kermorvant, J. Puigcerver, D. Stutzmann, A.H. Toselli, E. Vidal, in _International Conference on Document Analysis and Recognition_ (2017), pp. 311–316
  * (3) T. Wilkinson, J. Lindström, A. Brun, in _International Conference on Computer Vision_ (2017), pp. 4443–4452
  * (4) L. Kang, J.I. Toledo, P. Riba, M. Villegas, A. Fornés, M. Rusiñol, in _German Conference on Pattern Recognition_ (2018), pp. 459–472
  * (5) S. Chanda, J. Baas, D. Haitink, S. Hamel, D. Stutzmann, L. Schomaker, in _International Conference on Frontiers of Handwriting Recognition_ (2018), pp. 345–350
  * (6) P. Krishnan, K. Dutta, C.V. Jawahar, in _Document Analysis Systems_ (2018), pp. 1–6
  * (7) K. Dutta, P. Krishnan, M. Mathew, C.V. Jawahar, in _International Conference on Frontiers of Handwriting Recognition_ (2018), pp. 80–85
  * (8) A. Graves, J. Schmidhuber, in _Advances in Neural Information Processing Systems_ (2009), pp. 545–552
  * (9) P. Krishnan, C. Jawahar, Hwnet v2: An efficient word image representation for handwritten documents.  International Journal on Document Analysis and Recognition 22(4), 387–405 (2019)
  * (10) M. Carbonell, A. Fornés, M. Villegas, J. Lladós, A neural model for text localization, transcription and named entity recognition in full pages.  Pattern Recognition Letters 136, 219–227 (2020)
  * (11) F. Wolf, G.A. Fink, in _Document Analysis Systems_ , _Lecture Notes in Computer Science_ , vol. 12116 (Springer, 2020), pp. 293–308
  * (12) J. Almazán, A. Gordo, A. Fornés, E. Valveny, Word spotting and recognition with embedded attributes.  IEEE Transactions on Pattern Analysis and Machine Intelligence 36(12), 2552–2566 (2014)
  * (13) P. Krishnan, C. Jawahar, Bringing semantics into word image representation.  Pattern Recognition 108 (2020)
  * (14) Y. Li, J. Zhang, J. Zhang, K. Huang, in _The IEEE Conference on Computer Vision and Pattern Recognition_ (2018), pp. 7463–7471
  * (15) H. Zhang, P. Koniusz, in _IEEE Conference on Computer Vision and Pattern Recognition_ (2018), pp. 7670–7679
  * (16) Y. Annadani, S. Biswas, in _The IEEE Conference on Computer Vision and Pattern Recognition_ (2018)
  * (17) L. Niu, A. Veeraraghavan, A. Sabharwal, in _IEEE Conference on Computer Vision and Pattern Recognition_ (2018)
  * (18) K. Li, M.R. Min, Y. Fu, in _IEEE International Conference on Computer Vision_ (2019), pp. 3583–3592
  * (19) G.S. Xie, L. Liu, X. Jin, F. Zhu, Z. Zhang, J. Qin, Y. Yao, L. Shao, in _IEEE Conference on Computer Vision and Pattern Recognition_ (2019)
  * (20) Z. Akata, F. Perronnin, Z. Harchaoui, C. Schmid, Label-embedding for image classification.  IEEE Transactions on Pattern Analysis and Machine Intelligence 38(7), 1425–1438 (2016)
  * (21) Y. Xian, B. Schiele, Z. Akata, in _2017 IEEE Conference on Computer Vision and Pattern Recognition_ (2017), pp. 3077–3086
  * (22) A. Paul, N.C. Krishnan, P. Munjal, in _IEEE/CVF Conference on Computer Vision and Pattern Recognition_ (2019), pp. 7056–7065
  * (23) A. Graves, S. Fernández, F. Gomez, J. Schmidhuber, (2006), pp. 369–376.  10.1145/1143844.1143891
  * (24) S. Sudholt, G.A. Fink, in _International Conference on Document Analysis and Recognition_ (2017), pp. 493–498
  * (25) A. Fischer, A. Keller, V. Frinken, H. Bunke, Lexicon-free handwritten word spotting using character hmms.  Pattern recognition letters 33(7), 934–942 (2012)
  * (26) U.V. Marti, H. Bunke, The iam-database: an english sentence database for offline handwriting recognition.  International Journal on Document Analysis and Recognition 5(1), 39–46 (2002)