# CALText: Contextual Attention Localization for Offline Handwritten Text

Tayaba Anjum
Department of Computer Science
University of the Punjab
Lahore, Pakistan
<EMAIL_ADDRESS>
& Nazar Khan
Department of Computer Science
University of the Punjab
Lahore, Pakistan
<EMAIL_ADDRESS>
http://faculty.pucit.edu.pk/nazarkhan/

###### Abstract

Recognition of Arabic-like scripts such as Persian and Urdu is more
challenging than Latin-based scripts. This is due to the presence of a two-
dimensional structure, context-dependent character shapes, spaces and
overlaps, and placement of diacritics. Not much research exists for offline
handwritten Urdu script which is the 10th most spoken language in the world.
We present an attention based encoder-decoder model that learns to read Urdu
in context. A novel localization penalty is introduced to encourage the model
to attend only one location at a time when recognizing the next character. In
addition, we comprehensively refine the only complete and publicly available
handwritten Urdu dataset in terms of ground-truth annotations. We evaluate the
model on both Urdu and Arabic datasets and show that contextual attention
localization outperforms both simple attention and multi-directional LSTM
models.

_Keywords_ Handwriting $\cdot$ Urdu $\cdot$ Offline $\cdot$ Recognition
$\cdot$ Text $\cdot$ Attention $\cdot$ Localization $\cdot$ Encoder $\cdot$
CNN $\cdot$ DenseNet $\cdot$ Decoder $\cdot$ Recurrent $\cdot$ GRU $\cdot$
LSTM $\cdot$ OCR

##  1 Introduction

Despite being the 10th most spoken language in the world [1], there has been
very little focus on the offline recognition of handwritten Urdu text. As a
spoken language, Urdu is similar to Hindi but in script form, it is closer to
Arabic and Persian. Offline Handwritten Text Recognition (OHTR) refers to the
problem of recognizing handwritten text from an image. OHTR provides the
interface between humans and machines by automatic digitization of handwritten
documents such as letters, lecture notes, bank checks, medical records, and
library books. The OHTR problem varies from Optical Character Recognition
(OCR) because of the wide variety of writing styles, sizes, sentence lengths,
pen types, page backgrounds, and rule violations.

For Latin-based handwritten text, good quality datasets are readily available
[27, 22, 41, 28]. Similar datasets exist for handwritten Arabic [2, 7, 8, 42,
26, 4] and Persian [38] as well. Recognition of Arabic-like scripts (Arabic,
Persian and Urdu) presents interesting challenges in addition to those for
Latin-based scripts [33]. For example, the presence of two-dimensional
structure, character and ligature overlap, ligature deformation, context-
sensitive shapes of ligatures, placement and quantities of diacritical marks,
stretching, and spacing. Figure 1 depicts handwritten Urdu text in the first
row, the corresponding typed text in the second row and the individual
characters in the third row. It can be seen that within cursively written Urdu
words, the individual characters appear with widely varying ligatures. While
typed Urdu conforms to some rules, handwriting can be restricted to very few
rules.

Figure 1: Top: Handwritten Urdu text. Middle: Corresponding typed version.
Bottom: Isolated typed characters appear significantly different from their
corresponding cursive ligatures. Urdu is neither written nor typed in the form
of isolated characters.

Previous attempts at OHTR for Arabic-like scripts have either used raw pixels
or features extracted from a Convolutional Neural Network (CNN) that are then
decoded using a recurrent model such as a bidirectional Long Short-Term Memory
(BLSTM) network [4, 19] that aims to capture the horizontal nature of
_deskewed_ text. Multidimensional LSTM (MDLSTM) has also been employed to
capture the horizontal as well as vertical nature of Arabic-like scripts [18,
3]. These approaches use the complete image for recognition of a character at
a certain time step. However, a single character can be recognized from a
specific location of an image at a certain time step using attention.
Attention mechanism helps to recognize a character from an image by focusing
on a specific region at a certain time step.

In previous efforts for Arabic-like scripts, error reporting has been
restricted to the level of characters. World level accuracy is not reported.
This, we speculate, is due to the character level accuracy not being good
enough for recognition of words since a word is recognized correctly if and
only if each and every character is recognized correctly.

Traditional variants of LSTM (1 direction), BLSTM (2 directions) and MDLSTM (4
directions) all limit the number of directions that a decoder can proceed in.
In contrast, an attention-based decoder is direction-less since it can focus
on any relevant part of the image at any given time. However, characters in a
sequence can’t appear at arbitrary locations. For text, _attention itself
needs context_. That is, where to focus next depends on where the model has
already focused previously. One such decoder was introduced in [9]. In solving
problems related to recognition of a line of text, each character appears
within a small contiguous region instead of being scattered all over the
image. Therefore, in addition to the context of attention, text requires
localization of attention as well. We show in this work how localization can
be introduced into a contextual attention-based decoder. This results in our
proposed _contextual attention localization_ (CAL) method for handwritten
text.

In this paper we present a thorough explanation of the encoder/decoder
architecture from [9], extend it using attention localization and introduce a
technique for compact spatiotemporal attention visualization. We use a
DenseNet encoder to extract high level features from the input image. The
features are then decoded into a contextual sequence of output characters
using localized attention. Using localized attention in context encourages our
model to attend only to relevant image regions while decoding an output
sequence. The encoder and decoder are trained jointly so that the decoder can
guide the encoder towards generation of richer, more useful features. The
paper contributes as follows:

  1. 1.

We present a detailed contextual attention-based, encoder-decoder model for
the recognition of offline handwritten text. In particular, we describe the
model introduced in [9] in much greater detail and extend it.

  2. 2.

We show that, armed with attention, direction-less decoders out-perform uni-
directional, bidirectional and multi-directional decoders.

  3. 3.

We show that for text recognition, encouraging the model to localize its
attention improves recognition accuracy.

  4. 4.

We comprehensively re-annotate the PUCIT-OHUL dataset [9] of offline
handwritten Urdu text using 7 annotators.

     * •

Missing ground-truth lines are added.

     * •

We synchronize ground-truth text with the text on the images.

     * •

We record instances of text that is crossed-out, overwritten, or covered with
correction fluid.

     * •

We annotate incorrectly spelled words as they are. We do not correct spelling
mistakes in the written text.

  5. 5.

We additionally report word level accuracy instead of only character level
accuracy.

  6. 6.

To show the robustness of the proposed model, we use Urdu PUCIT-OHUL and
Arabic KHATT dataset for experiments.

  7. 7.

We introduce a method for compact visualization of spatiotemporal attention.

  8. 8.

Compared to previous approaches, we report $2\times$ accuracy improvement at
both character level and word level.

For reproducible research, code for this work is publicly available at
https://github.com/nazar-khan/CALText.

The rest of the paper is structured as follows. Section 2 reviews the state-
of-the-art related to offline handwritten text recognition. Challenges
specific to Urdu and other Arabic-like scripts are discussed in Section 3.
This is followed by detailed explanations of our encoding and decoding
frameworks in Section 4. Section 5 describes the datasets used for training
and evaluation. Quantitative as well as qualitative experimental results are
presented in Section 6 and the paper is concluded in Section 7.

##  2 Literature Review

Deep convolutional networks with some form of recurrence, attention or self-
attention represent state-of-the-art for the OHTR problem for multiple scripts
such as Latin-based [16, 25], Arabic [3, 19], CJK111Chinese Japanese Korean
[48, 24, 34] and even mathematical expressions [23] and music notes [12].
Historically, OHTR has evolved from convolutional [22] to recurrent [17] to
attention-based [18] models. The use of attention remains a central theme in
other problems involving text, such as scene text detection [47, 46].

Early efforts at Urdu recognition were restricted to typed documents with
fixed font styles and sizes [21, 6]. Uni-, bi- and multi-directional LSTMs
[43, 44, 30, 31, 32] were also applied for typed Urdu recognition.

Handwritten Urdu text recognition is more difficult compared to printed text
as writing style, pen type, page background, and rule violations heavily
impact the recognition process. Early attempts were naturally restricted to
recognition of isolated characters. Isolated Urdu numerals were recognized in
[13] using a Support Vector Machine (SVM) classifier with Quad-Tree based
Longest-Run (QTLR) features. The CENPARMI-U dataset introduced in [39] focused
on isolated characters and contained 44 handwritten Urdu characters and
numerals, 5 special characters, and only 57 Urdu words. It also contained Urdu
dates and numeral strings. Isolated characters were recognized in [35] using
an SVM classifier of moment invariant features. For the recognition of
isolated words in online Urdu handwritten text, a Hidden Markov Model (HMM)
with fuzzy logic was used in [37] to achieve $74.1\%$ accuracy on the IFN/ENIT
dataset for Urdu Naskh style and $87.6\%$ accuracy for Urdu Nastaleeq style.
In [29], isolated words were recognized using gradient and structural features
with SVM classifier. They created a dataset of 100 words written by 2 writers.
Each word was written 8 times by 1 writer to obtain a total of 1600 images.

Deep learning based solutions for offline handwritten Urdu text include [5]
and [19] which both employ bi-directional LSTM models. The only publicly
available dataset is UCOM/UNHD [5] and that too is only partially available.

##  3 Challenges of OHTR

Recognition of Arabic-like scripts such as Urdu is difficult due to Sayre’s
paradox [40] that a cursively written word cannot be recognized without being
segmented and cannot be segmented without being recognized. In addition,
handwritten text can employ a wide variety of pen types, writing styles, sizes
and page backgrounds. But most importantly, in handwritten Urdu and text in
general, almost every rule can be violated until the text becomes illegible.
Some of the main challenges in offline handwritten Urdu text recognition as
explained in Figure 2 are:

  1. 1.

One challenge is that cursive Urdu can have extreme overlap among characters.
Sometimes, parts of a letter can overlap multiple other letters or even other
words. In other cases, letters can lie exactly on top of other letters.

  2. 2.

Another challenge is context-dependent shape and location of letters. The
placement of characters is not just in right-to-left order. Urdu script has a
2-dimensional structure as well.

  3. 3.

A third challenge is that Urdu is not entirely cursive. It is a mix of cursive
and non-cursive. As a result, sometimes the space within characters of a word
can be larger than the space between words. This makes tokenization of a
sentence into words difficult.

  4. 4.

Within cursive words, characters to be recognized often appear with widely
varying ligatures. While typed Urdu conforms to some rules, handwriting can be
restricted to very few rules.

Therefore, handwritten Urdu and Arabic recognition is not a trivial problem.

Figure 2: Challenges in offline handwritten Arabic-like scripts. This Urdu
sentence contains i) extreme overlap between ligatures due to the
2-dimensional structure of Urdu, ii) context dependent ligature shapes as well
as locations, and iii) intra-word spaces occasionally larger than inter-word
spaces.

##  4 Methodology

For handwritten text, even if the input images are restricted to have a fixed
size, the characters may constitute a variable-length sequence. The
encoder/decoder model with a recurrent decoder can be used for the recognition
of such variable text sequences [14]. The encoder transforms an input image
into an intermediate representation. The decoder then transforms the
intermediate representation into a sequence of output characters. While
decoding, an attention mechanism is used to focus on a specific part of the
image to produce different characters. An overview of our approach is shown in
Figure 3. Given an input image, the model produces a variable-length sequence
of 1-hot vectors

$Y=\\{\mathbf{y}_{1},\mathbf{y}_{2},\dots,\mathbf{y}_{T}\\}$ (1)

where $\mathbf{y}_{t}\in\mathbb{R}^{k}$ is the $1$-of-$k$ encoding of the
$t$-th character in the image text containing $T$ characters including any
blank spaces and punctuation marks. Size of the vocabulary is denoted by $k$.

Figure 3: Proposed encoder/decoder framework. An image containing handwritten
text is input to a DenseNet encoder. Extracted features are passed through a
decoder that attends to specific image regions in context so as to produce
probabilities of output characters at every time step. The probabilities are
finally decoded into output characters using beam search.

###  4.1 Encoder: DenseNet with Bottlenecks

For extraction of high-level salient features, we use the DenseNet [20]
architecture which is a variation of CNNs with dense blocks. In a dense block,
convolution is applied to all previous layers within the block instead of just
the previous layer. The use of dense blocks provides a more diverse set of
features by minimizing information loss during forward and backward
propagation. Since connecting all previous layers becomes costly, bottleneck
layers ($1\times 1$ convolutions) are inserted to compress the volume of
previous layers. The convolutional output volume of layer $i$ is calculated as

${\mathbf{F}_{i}}={\mathcal{C}_{i}}(\mathcal{B}_{i}([\mathbf{F}_{1};\mathbf{F}_{2};\dots;\mathbf{F}_{i-1}]))$
(2)

where $\mathcal{C}_{i}(\cdot)$ represents regular convolution layers,
$\mathcal{B}_{i}(\cdot)$ represents $1\times 1$ convolution of a bottleneck
layer and $[\mathbf{F}_{1};\mathbf{F}_{2};\dots;\mathbf{F}_{i-1}]$ represents
the depth-wise concatenation of the outputs of all previous layers. Pooling
layers between dense blocks are used for extraction of features at multiple
scales. The detailed architecture of our DenseNet encoder is shown in Figure
4. There are $3$ dense blocks. Each block consists of $32$ sub-layers
alternating between $1\times 1$ convolution bottleneck layers and $3\times 3$
convolution layers. Each dense block is preceded by a $2\times 2$ pooling
layer with stride $2$.

Figure 4: Architecture of the DenseNet encoder that transforms an input image
into a 3-dimensional feature volume. BRD = Batchnorm + ReLU + Dropout.

The output of the dense encoder, for a given input image, is a volume of
dimensions $h\times w\times d$. It can be interpreted as $d$-dimensional
_annotation vectors_ of $h\times w$ overlapping blocks of the input image.
This interpretation is illustrated in Figure 5. The correspondence between the
annotation vectors and image regions is because convolution represents moving,
overlapping, and localized dot-products and pooling results in effectively
larger receptive fields. Therefore each depth vector in the output volume
corresponds to a $d$-dimensional encoding of a localized image region. The set
of all annotation vectors produced by the DenseNet is represented by

$A=\\{\mathbf{a}_{1},\mathbf{a}_{2},\dots,\mathbf{a}_{L}\\}$ (3)

where $\mathbf{a}_{l}\in\mathbb{R}^{d}$ and number of regions is given by
$L=hw$.

Figure 5: DenseNet output volume of size $h\times w\times d$ represents
$d$-dimensional _annotation vectors_ of $h\times w$ overlapping blocks of the
input image.

###  4.2 Decoder: Gated Recurrent Units with Contextual Attention Localization

When an encoder extracts high-level visual features from input images, a Gated
Recurrent Unit (GRU) [15] can be used to generate a variable-length text
sequence character by character, conditioned on the previous output character
$\mathbf{y}_{t-1}$, the current hidden state $\mathbf{h}_{t}$ and a context
vector $\mathbf{c}_{t}$. Our decoder consists of a Contextual Attention
Localization (CAL) unit sandwiched between two GRUs as shown in Figure 6.

Figure 6: The decoder contains a contextual attention localization (CAL) unit
between two gated recurrent units (GRUs).

####  4.2.1 First GRU

The first GRU is used to compute a predicted hidden state
$\mathbf{\hat{h}}_{t}$ via the following recurrent equations

$\displaystyle\mathbf{z}_{1,t}$
$\displaystyle=\sigma(\mathbf{W}_{yz}\mathbf{E}\mathbf{y}_{t-1}+\mathbf{U}_{hz}\mathbf{h}_{t-1})$
(4) $\displaystyle\mathbf{r}_{1,t}$
$\displaystyle=\sigma(\mathbf{W}_{yr}\mathbf{E}\mathbf{y}_{t-1}+\mathbf{U}_{hr}\mathbf{h}_{t-1})$
(5) $\displaystyle\mathbf{\widetilde{h}}_{t}$
$\displaystyle=\tanh(\mathbf{W}_{yh}\mathbf{E}\mathbf{y}_{t-1}+\mathbf{r}_{1,t}\otimes(\mathbf{U}_{rh}\mathbf{h}_{t-1}))$
(6) $\displaystyle\mathbf{\hat{h}}_{t}$
$\displaystyle=\mathbf{z}_{1,t}\otimes\mathbf{h}_{t-1}+(1-\mathbf{z}_{1,t})\otimes\mathbf{\widetilde{h}}_{t}$
(7)

where $\sigma$ is the logistic sigmoid function, $\otimes$ is a point wise
multiplication, $\mathbf{E}\in\mathbb{R}^{m\times k}$ is a learnable lower-
dimensional projection matrix for getting rid of the wasteful $1$-of-$k$
coding of character vector $\mathbf{y}_{t-1}$. All other matrices within each
equation represent learnable transformations into a constant dimensional
space. They can be divided into character embedding transformations
$\\{\mathbf{W}_{yz},\mathbf{W}_{yr},\mathbf{W}_{yh}\\}\in\mathbb{R}^{h\times
m}$ and recurrent transformations
$\\{\mathbf{U}_{hz},\mathbf{U}_{hr},\mathbf{U}_{rh}\\}\in\mathbb{R}^{h\times
h}$. The recurrent hidden state $\mathbf{h}_{t-1}$ from the previous time step
is computed using Equation (16). The predicted hidden state
$\mathbf{\hat{h}}_{t}$ is then used in the dynamic computation of the
attention weights as described next.

####  4.2.2 Contextual Attention

Attention has been successfully used in computer vision [11] and also in
handwriting recognition [50, 49]. The benefit of using attention is that it
adds context to the decoder. For example, an image region should be classified
as a dog by looking at the dog and not by looking at its owner. While a dog
can appear anywhere in an image, characters in a text sequence cannot appear
at arbitrary locations. Text has a strong structure and therefore _attention
itself needs context_. In simpler words, where to focus next depends on where
the model has focused previously.

Ideally, we would like our text recognizer to _read_ like we do. That is, it
should focus on the relevant image region when recognizing each character or
word. However, for images containing text, multiple instances of a character
or word can appear at multiple locations. Nothing stops an attention model
from re-attending a previously attended location or from giving the right
answer by attending the wrong location. Therefore, _for text_ , the decision
for attention needs to depend on the history of attention. In our model,
attention at time $t$ is made to depend on previous values of attention. This
is achieved through a coverage vector, that represents a history of attention
already given to each location.

Let attention weight $\alpha_{tl}$ denote the importance of image region $l$
at time $t$. By aggregating the attentions through time, we can compute the
$h\times w$ aggregated attention array $S^{\alpha}$ at time $t$ via

$\displaystyle S^{\alpha}_{tl}$ $\displaystyle=\sum_{\tau=1}^{t-1}\alpha_{\tau
l}$ (8)

The 2-dimensional array $S^{\alpha}$ can be convolved with $q$ filters of size
$f\times f$ arranged as a volume $Q$ to obtain an $h\times w\times q$ volume
$F_{t}$ as

$\displaystyle F_{t}$ $\displaystyle=S^{\alpha}\ast Q$ (9)

Features in $F_{t}$ can be interpreted as the _history of attention_ until
time $t$ and $Q$ is a learnable layer of convolution filters. Similar to how
annotation vector $\mathbf{a}_{l}$ is formed, the coverage vector
$\mathbf{f}_{l}$ is the $q$-dimensional vector at location $l$ of volume
$F_{t}$ as shown in Figure 7. Each vector $\mathbf{f}_{l}$ can be interpreted
as the _coverage_ given to location $l$ until time $t$. Therefore,
$\mathbf{f}_{l}$ is called a _coverage vector_. The coverage vectors describe
the attention given _so far_ to different regions in a _dynamic_ fashion. In
contrast, the annotation vectors $\mathbf{a}_{1},\dots,\mathbf{a}_{L}$
describe input image regions in a _static_ fashion.

Figure 7: Computation of coverage vector.

The dynamic attention values $\alpha_{tl}$ for the current time step $t$ can
then be computed via

$\displaystyle e_{tl}$
$\displaystyle=\mathbf{v}_{a}^{T}\tanh(\mathbf{W}_{a}\mathbf{\hat{h}}_{t-1}+\mathbf{U}_{a}\mathbf{a}_{l}+\mathbf{U}_{f}\mathbf{f}_{l})$
(10) $\displaystyle\alpha_{tl}$
$\displaystyle=\frac{\exp(e_{tl})}{\sum_{u=1}^{L}\exp(e_{tu})}$ (11)

where $\mathbf{v}_{a}\in\mathbb{R}^{n}$, $\mathbf{W}_{a}\in\mathbb{R}^{n\times
h}$, $\mathbf{U}_{a}\in\mathbb{R}^{n\times d}$, and
$\mathbf{U}_{f}\in\mathbb{R}^{n\times q}$ are all learnable projection
parameters. Notice that the softmax operation in Equation (11) ensures that
attention weights $\alpha_{tl}$ at time $t$ represent probabilities.

Figure 8: Attention weight $\alpha_{tl}$ determines importance of location $l$
in determining context vector $\mathbf{c}_{t}$ at time $t$.

A _dynamic_ representation of the relevant part of the input image at time $t$
can now be defined. As illustrated in Figure 8, it is computed as the
expectation of the annotation vectors

$\mathbf{c}_{t}=\sum_{l=1}^{L}\alpha_{tl}\mathbf{a}_{l}$ (12)

where attention weights $\alpha_{tl}$ determine the importance of each image
region at time $t$. They determine the image _context_ in which the decision
for output character at time $t$ will be made. Accordingly, the expectation of
the annotation vectors can be termed as the _context vector_ $\mathbf{c}_{t}$.

####  4.2.3 Second GRU

The second GRU computes the final hidden state $\mathbf{h}_{t}$ using
predicted hidden state $\mathbf{\hat{h}}_{t}$ and the context vector
$\mathbf{c}_{t}$ via the following recurrent equations

$\displaystyle\mathbf{z}_{2,t}$
$\displaystyle=\sigma(\mathbf{U}^{\prime}_{hz}\mathbf{\hat{h}}_{t}+\mathbf{C}_{cz}\mathbf{c}_{t})$
(13) $\displaystyle\mathbf{r}_{2,t}$
$\displaystyle=\sigma(\mathbf{U}^{\prime}_{hr}\mathbf{\hat{h}}_{t}+\mathbf{C}_{cr}\mathbf{c}_{t})$
(14) $\displaystyle\mathbf{\widetilde{h}}_{t}$
$\displaystyle=\tanh(\mathbf{r}_{2,t}\otimes(\mathbf{U}^{\prime}_{rh}\mathbf{\hat{h}}_{t})+\mathbf{C}_{ch}\mathbf{c}_{t})$
(15) $\displaystyle\mathbf{h}_{t}$
$\displaystyle=\mathbf{z}_{2,t}\otimes\mathbf{\hat{h}}_{t}+(1-\mathbf{z}_{2,t})\otimes\mathbf{\widetilde{h}}_{t}$
(16)

where learnable matrices can be divided into recurrent transformations
$\mathbf{U}^{\prime}_{hz},\mathbf{U}^{\prime}_{hr},\mathbf{U}^{\prime}_{rh}\\}\in\mathbb{R}^{h\times
h}$, and context transformations
$\\{\mathbf{C}_{cz},\mathbf{C}_{cr},\mathbf{C}_{ch}\\}\in\mathbb{R}^{h\times
d}$.

The conditional probabilities of the next character are computed as

$p(\mathbf{y}_{t}|\mathbf{y}_{t-1};\mathbf{X})=\mathcal{S}(\mathbf{W}_{o}(\mathbf{W}_{y}\mathbf{E}\mathbf{y}_{t-1}+\mathbf{W}_{h}\mathbf{h}_{t}+\mathbf{W}_{c}\mathbf{c}_{t}))$
(17)

where $\mathcal{S}(\cdot)$ is the softmax activation function and
$\mathbf{W}_{o}\in\mathbb{R}^{k\times m}$,
$\mathbf{W}_{h}\in\mathbb{R}^{m\times h}$ and
$\mathbf{W}_{c}\in\mathbb{R}^{m\times d}$ are all learnable projection
parameters.

The sequence of probability vectors is converted into an optimal sequence of
output characters $\mathbf{y}_{1},\mathbf{y}_{2},\dots,\mathbf{y}_{T}$ via
beam search [45] whereby only the $b$ most-probable branches are pursued at
each time step.

###  4.3 Attention Localization

Text is made up of very localized image regions so that a single character
appears in a small contiguous area. It cannot appear scattered all over the
image. Therefore, in addition to context of attention, text requires
_localization of attention_ as well.

This behaviour can be induced by encouraging the attention weights
$\alpha_{tl}$ to be sparse. Accordingly, we define the attention localization
penalty as

$\mathcal{L}(\theta)=\sum_{t=1}^{T}\sum_{l=1}^{L}|\alpha_{tl}|$ (18)

which is the $\ell_{1}$-norm of the vector of attention values. Minimizing
$\ell_{1}$-norm encourages only few $\alpha_{tl}$ to be large and all the rest
to be close to zero. This discourages the model from attending to multiple
locations at the same time.

###  4.4 Error Function

The proposed model is trained jointly for recognition of offline handwritten
text. In the model, the next character is predicted depending on the previous
character and input image as explained in Equation (17). The objective of the
training is to maximize the probability of the predicted character. In order
to train the model to classify using contextual attention, regularized cross-
entropy is used as the error function

$\mathcal{C}_{n}(\theta)=-\sum_{t=1}^{T_{n}}\ln
p(\mathbf{y}_{n}^{(t)}|\mathbf{d}_{n}^{(t)},\mathbf{x}_{n})+\lambda r(\theta)$
(19)

where $\mathbf{y}_{n}^{(t)}$ is the predicted character at timestamp $t$ of
the ${n}^{th}$ training image, $r(\theta)$ is the $\ell_{2}$-norm of all
weights except those from the convolution layers and $\lambda>0$ is the
regularization hyperparameter.

In order to train the model to classify using _localized contextual attention_
, cross-entropy can be augmented with our attention localization penalty (18)
to yield our proposed error function

$\mathcal{E}_{n}(\theta)=\mathcal{C}_{n}(\theta)+\gamma\mathcal{L}_{n}(\theta)$
(20)

where hyperparameter $\gamma>0$ controls the tradeoff between classification
and localization.

##  5 Datasets

We evaluate our proposed method on handwritten Urdu as well as Arabic.

###  5.1 Urdu Dataset

The PUCIT-OHUL dataset introduced in [9] is a multi-writing style dataset with
different pen types, ink types, text sizes, and background types and colours.
It was collected using $100$ students between $20$ and $24$ years of age. A
total of $479$ pages of text were collected. Each page was scanned at $200$
DPI and text lines were manually segmented. The line images were not deskewed.
As shown in Figure 9, the dataset contains different types of page
backgrounds, page colors, ink types and pen types. An overview of the subject
categories covered in the dataset is shown in Table 1. The writers were not
given any instructions regarding what subject to choose and were completely
free in their choices. The detailed vocabulary of unique characters and
symbols extracted from the dataset is given in Table 2.

Blank paper |  |  |
---|---|---|---
Ruled paper |  |  |
Background colour |  |  |
Ink colour |  |  |
Pen type |  |  |

Figure 9: Sample page images from the PUCIT-OHUL dataset demonstrating
challenges related to different paper types, backgrounds, colours, ink types
and pen types.

Unsurprisingly, labeling errors in datasets adversely impact training as well
as error rates [10]. Accordingly, we have re-annotated the complete PUCIT-OHUL
dataset. The key outcomes of our re-annotation process are as follows.

  1. 1.

We have added $92$ new lines to the original test set.

  2. 2.

In an attempt to reduce annotator bias, $7$ annotators were used.

  3. 3.

Vocabulary size was increased from $98$ in the original ground-truth to $130$
in our updated version. We annotated $31$ additional characters from the
English alphabet and $1$ from Urdu that had been used by the original writers
but the original ground-truth did not include them. In the updated
annotations, we label each and every character.

  4. 4.

For a few cases, there was a mismatch between images and annotations. Such
mismatches have been fixed.

  5. 5.

In the original ground-truth, spelling mistakes in the handwritten text were
sometimes _corrected_ by the annotators. In our re-annotations, such mistakes
have been recorded as they appear with no attempt to correct them.

  6. 6.

Original ground-truth had a non-trivial amount of errors in the annotation of
diacritics. Since diacritics are usually small and the mind can _auto-correct_
mistakes in diacritics, such errors are easy to overlook. A lot of attention
was focused on correct re-annotation of diacritics.

  7. 7.

We categorize crossed-out text as either readable or non-readable. Such
information can be used to _not penalize_ a recognizer that correctly
recognizes readable crossed-out text.

Table 1: Categorization of the content in the PUCIT-OHUL dataset in terms of subjects, references, pages, and lines. | Subject | References | Pages | Lines
---|---|---|---|---
1 | Art | 1 | 5 | 72
2 | World | 11 | 52 | 805
3 | Economy | 3 | 15 | 277
4 | Social | 19 | 84 | 1291
5 | Literature | 17 | 83 | 1293
6 | Wildlife | 1 | 5 | 106
7 | Sports | 4 | 21 | 317
8 | Health | 1 | 5 | 77
9 | Religion | 8 | 40 | 662
10 | Nature | 2 | 9 | 129
11 | Science | 3 | 14 | 203
12 | Politics | 14 | 67 | 1005
13 | History | 10 | 52 | 683
14 | Education | 6 | 27 | 479
| Total | 100 | 479 | 7,399
Table 2: Vocabulary extracted from the PUCIT-OHUL dataset. Table 3: Comparison with existing offline handwritten Urdu text datasets Dataset |  Total Lines |  Total Words |  Total characters |  Total Writers |  Vocabulary Size
---|---|---|---|---|---
UCOM/UNHD [5] | 10,000 | 312,000 | 1,872,000 | 500 | 59
CENIP-UCCP [36] | 2,051 | 23,833 | - | 200 | -
Custom dataset [19] | 6,000 | 86,400 | 432,000 | 600 | -
PUCIT-OHUL | 7,401 |  80,059 |  367,925 | 100 | 130

Table 3 compares the PUCIT-OHUL dataset with existing datasets. Existing
datasets are problematic for two reasons. Firstly, sometimes data statistics
are not reported correctly. For instance, UCOM/UNHD [5] claims to have
$10,000$ text lines, but only $700$ of those lines are unique in terms of
semantic content. Secondly, all $3$ datasets in [5, 36, 19] are either
publicly unavailable or only partially available. In contrast, the PUCIT-OHUL
dataset with our updated annotations is publicly
available222http://faculty.pucit.edu.pk/nazarkhan/work/urdu_ohtr/pucit_ohul_dataset.html
in its entirety.

The dataset is divided randomly into training, validation, and testing sets
with respect to writers. Out of the $100$ writers, we randomly selected $75$
writers for the training set, $13$ for the validation set and the remaining
$12$ for the testing set. In terms of lines, $5580~{}(75.42\%)$ of the lines
were included in the training set, $907~{}(12.33\%)$ in the validation set,
and $912~{}(12.25\%)$ in the testing set.

###  5.2 Arabic Dataset

The KFUPM Handwritten Arabic TexT (KHATT) [26] dataset is an unconstrained
offline handwritten Arabic dataset. We use the default split of the unique
text portion of the dataset which consists of $4,825$ training, $937$
validation, and $966$ testing line images. The dataset contains $63$ unique
symbols.

##  6 Experiments and Results

###  6.1 Experimental Setup

The proposed model is composed of a DenseNet encoder and a contextual
attention mechanism sandwiched between two GRU decoders. Dimensions of all
learnable matrices and vectors in our decoding framework are given in Table 4.

Table 4: Dimensions of learnable matrices and vectors in the proposed decoding framework. Equation | Dimensions
---|---
(4), (5), (6) | $k=130,m=256,h=256$
(9) | $f=11,q=512$
(10) | $n=256,d=684$

To reduce overfitting, $4\%$ salt and pepper noise is added to training images
with probability of salt being $20\%$ and probability of pepper being $80\%$.
During training, we use batch normalization and dropout. Dropout is only used
at convolution layers with $20\%$ drop ratio. We also apply regularization
with $\lambda=1e^{-4}$ in the cross-entropy loss (19) and $\gamma=1$ in the
overall error function (20). We use an Adadelta optimizer with gradient
clipping for better optimization. All training and testing images are resized
to $100\times 800$ pixels. Before resizing, if the width is less than $300$,
we first double the width of the image by extra white padding and then resize
the result to $100\times 800$. This way we avoid excessive stretching in
images with small width.

The proposed model is trained and tested on Urdu PUCIT-OHUL [9] as well as
Arabic KHATT [26] datasets.

###  6.2 Quantitative Results

We use character and word error rates to quantitatively compare the proposed
method with other approaches. The character error rate (CER) between target
and output text is calculated using the Levenshtein edit distance formula

$\text{CER}=\frac{\text{I}+\text{S}+\text{D}}{\text{N}}\times 100$ (21)

where I, S and D represent the minimum numbers of insertions, substitutions
and deletions required to convert the target text string into the output text
string and N represents the total number of characters in the target
text333Theoretically, CER can be greater than $100$ but for any reasonably
performing recognizer, it is typically less than $100$.. To calculate word
error rate (WER) Equation (21) can be used after replacing characters with
words. To calculate accuracies, both error rates are subtracted from $100$.

Table 5: Comparison of different models for offline handwritten Urdu text recognition on PUCIT-OHUL dataset. Models |  Character Level Accuracy |  Word Level Accuracy
---|---|---
CNN+GRU | $33.00$ | $22.44$
CNN+LSTM | $32.04$ | $21.42$
CNN+BGRU | $32.76$ | $23.16$
CNN+BLSTM [19] | $32.02$ | $22.23$
Contextual Attention [9] | $77.74$ | $45.55$
Contextual Attention Localization | $\mathbf{79.76}$ | $\mathbf{47.93}$

####  6.2.1 PUCIT-OHUL dataset

Table 5 shows the comparison of previous models with our proposed model. We
compare our model with CNN encoder based uni-directional as well as
bidirectional decoders that do not use attention. The CNN architecture that we
use has previously been used for handwritten Urdu, Arabic, English and French.
The CNN model used in the first $4$ models at Table 5 is based on the
architecture from [19] and each method was trained on our dataset. In our
experiments, bidirectional decoders were able to improve character level
accuracies by less than $2\%$. Since character level accuracies were so low,
word level accuracies are understandably poor. This is why previous work on
handwritten Arabic-like scripts does not report word level accuracies. In
contrast, our DenseNet encoder jointly trained with an attention based decoder
almost doubles the character level accuracy and word level accuracy. We
believe the reasons for such drastic improvement in results are that: The
DenseNet encoder learns more diverse features compared to the CNN model of
previous approaches. In addition, attention reduces the need for bidirectional
decoding. Finally, using the concept of _coverage of attention_ captures the
right-to-left nature of Urdu.

Results show that incorporation of the DenseNet encoder with attention based
decoding almost doubles the character level accuracy and word level accuracy.
Furthermore, it can be observed that by using localized attention accuracy
furthers increases by a value of around $2\%$.

The proposed recognizer based LSTM along with DenseNet significantly improves
the character and word level accuracy as compared to already existing
techniques in literature for Urdu text recognition. The use of attention helps
the model in looking at the appropriate location, and attention localization
further helps to localize it. This greatly benefits character/word level
accuracy.

####  6.2.2 KHATT dataset

We also compare proposed models with the MDLSTM model used for recognition of
Arabic offline handwritten KHATT dataset in Table 6. We use KHATT unique text
lines dataset. We don’t use any skew and slant correction. Instead of a
multidimensional decoder, an attention based uni-dimensional encoder without
any specific pre-processing steps gives almost equal results as achieved by
[3]. Adding localization further increases the accuracy by almost $2\%$.

Table 6: Comparison of proposed models with models proposed for recognition of offline handwritten Arabic KHATT dataset.  Models | Pre-processing |  Character Level Accuracy |  Word Level Accuracy
---|---|---|---
MDLSTM [3] |  Skew, slant correction, pruning, and normalization | $75.80$ | $-$
Contextual Attention [9] | Normalization | $75.16$ | $33.88$
Contextual Attention Localization | Normalization | $\mathbf{77.15}$ | $\mathbf{35.83}$

###  6.3 Qualitative

In order to explain the results of the model, we can visualize the attention
weights $\alpha_{tl}$ for region $l$ at time $t$. Let $\alpha_{t}$ denote the
attention map array for all locations at time $t$. This attention map can be
superimposed on the original image. Figure 10 demonstrates the regions of
attention for the proposed CALText model when decoding a sequence of output
characters. It can be seen that the model implicitly segments and recognizes
each character by focusing only on localized, relevant areas while considering
the context of previously attended locations. The purple region indicates the
region of attention. It can be observed that the model has learned to attend
from right-to-left which is the natural order of Urdu script. The most
probable character at each location inferred after beam-search is shown on the
right.

(a) CER=$0\%$ Figure 10: Attention visualization. Our model implicitly
segments and recognizes every character by focusing only on localized,
relevant areas. It also learns to focus in context.

A more compact visualization can be achieved by exploiting color to represent
time. Let the optimal output after beam-search consist of $T$ steps. By
dividing time by $T$ we can normalize it so that $0\leq t\leq 1$. The
normalized value of $t$ can be used to compute the color vector
$\mathbf{c}_{t}$ at time $t$ by linearly interpolating between the color
vector $\mathbf{c}_{0}$ at time $0$ and $\mathbf{c}_{1}$ at time $1$ as

$\displaystyle\mathbf{c}_{t}$
$\displaystyle=(1-t)\mathbf{c}_{0}+t\mathbf{c}_{1}$ (22)

The final colored attention through time can be visualized as

$\displaystyle\tilde{\alpha}$
$\displaystyle=\sum_{t\in\\{0,\dots,1\\}}\mathbf{c}_{t}\alpha_{t}$ (23)

The colored spatiotemporal attention image $\tilde{\alpha}$ visualizes
attention both in terms of spatial intensity and temporal sequence. We set
$\mathbf{c}_{0}$ equal to yellow and $\mathbf{c}_{1}$ equal to green.
Therefore, in our visualizations, temporal sequence of attended locations
moves from yellow in the beginning to green at the end. All subsequent
visualizations (Figures 11 – 15) use this mechanism.

On handwritten Urdu text, Figures 11 and 12 demonstrate three properties of
the proposed model.

  1. 1.

Encouraging the decoder to localize its attention reduces the character error
rate.

  2. 2.

Temporally, learned attention moves from right-to-left which is the natural
order of Urdu script.

  3. 3.

For skewed lines, attention follows the text.

Figure 13 demonstrates the same properties when our model is evaluated for
handwritten Arabic text.

Contextual Attention | Contextual Attention Localization
---|---
|
|
|
|
Figure 11: Urdu recognition results. In each panel, top row is the input image, second row visualizes spatiotemporal attention, third row superimposes attention over the input image and fourth row is the recognized text string followed by the CER in brackets. The proposed CALText model learns to attend to localized regions of the image in context. The transition from yellow to green indicates a temporal sequence of spatial attention. The model learns to attend to Urdu text in a right-to-left direction. The model also _follows_ the text when the handwritten text is not exactly horizontal. Contextual Attention | Contextual Attention Localization
---|---
|
|
|
|
|
Figure 12: Some more results on Urdu recognition. Contextual Attention | Contextual Attention Localization
---|---
|
|
|
|
Figure 13: The proposed CALText model learns to attend to localized regions of
the image in context. The transition from yellow to green indicates a temporal
sequence of attention. The model learns to attend to Arabic text in a right-
to-left direction.

Figure 14 demonstrates the regions attended by our model when observing an
empty white image containing no text. The model trained for right-to-left
ground-truth annotations correctly outputs an empty string as the recognition
result at the first time step $t=1$ by focusing on the left-end of the image.
In contrast, a model trained on left-to-right ground-truth annotations outputs
an empty string after attending the right end of the image.

Figure 15 demonstrates that the proposed model has learned to attend directly
to text. Instead of blindly scanning the image from left to right, it starts
attending the beginning of the sentence text and proceeds until the end of the
sentence text, irrespective of the spatial locations of the beginning and end.

(a)

(b)

Figure 14: Behavior of the proposed CALText model on an empty white image.
Attention visualization for a model trained on (a) right-to-left text, and (b)
left-to-right text. Both models output an empty string. As indicated by the
yellow color, the outputs were obtained at the very first time step $t=1$. It
can be noted that before outputting an empty string, the models focus more on
the relevant end of the image. However, they do focus, albeit less, on the
whole horizontal extant of the image as well. This is intuitively satisfying
since the whole image _should_ be scanned before outputting an empty string.

Figure 15: In order to produce its sequence of output characters, the proposed
CALText model learns to attend to text regions exclusively.

##  7 Conclusion

We have proposed an encoder-decoder recognition of offline, handwritten text.
The model has been evaluated for Urdu and Arabic which share numerous common
features such as right-to-left ordering, similar characters and two-
dimensional writing structure with inconsistent overlaps and spacing.
Multiscale features are extracted through a DenseNet encoder. To incorporate
the role of context in the process of _reading_ and to attend to specific
characters, we have introduced a contextual attention localizer between two
gated recurrent units. Results on test images show that our proposed CALText
model learns to read and recognize text in a way that is similar to human
reading by focusing only on relevant and localized image regions. The model
determines relevancy of a region through contextual attention.

We have comprehensively re-annotated all the ground truth text lines of the
PUCIT-OHUL dataset. As a result of our re-annotation, the count of unique
characters increased from $98$ to $130$. Numerous mistakes and omissions in
the original annotations have been corrected. For Urdu text, the use of
contextual attention improves upon the state-of-the-art by a factor of
$2\times$ in terms of both character and word level accuracy. Localization has
been shown to further improve accuracies by around $2\%$. Code is made
publicly available at https://github.com/nazar-khan/CALText.

## Acknowledgements

This research was supported by HEC-NRPU Grant 8029.

## References

  * [1] What are the top 200 most spoken languages?  https://www.ethnologue.com/guides/ethnologue200, 2021.  accessed: 18.09.2021.
  * [2] Haikal El Abed and Volker Margner.  The IFN/ENIT-database - a tool to develop Arabic handwriting recognition systems.  Int’l Journal on Document Analysis and Recognition, 5(10):39 – 46, 2002.
  * [3] Riaz Ahmad, Saeeda Naz, M. Zeshan Afzal, S. Faisal Rashid, Marcus Liwicki, and Andreas Dengel.  KHATT: A deep learning benchmark on Arabic script.  In 14th IAPR International Conference on Document Analysis and Recognition (ICDAR). IEEE, 2017.
  * [4] Saad Bin Ahmed, Saeeda Naz, Salahuddin Swati, and Muhammad Imran Razzak.  Handwritten Urdu character recognition using one-dimensional BLSTM classifier.  Neural Computing and Applications, 31(4):1143–1151, 2019.
  * [5] Saad Bin Ahmed, Saeeda Naz, Salahuddin Swati, and Muhammad Imran Razzak.  Handwritten Urdu character recognition using one-dimensional BLSTM classifier.  Neural Computing and Applications, 31(4):1143–1151, April 2019\.
  * [6] Q. A. Akram, S. Hussain, A. Niazi, U. Anjum, and F. Irfan.  Adapting Tesseract for complex scripts: An example for Urdu Nastalique.  In Proceedings of 11th IAPR Workshop on Document Analysis Systems (DAS 14), 2014.
  * [7] Huda Alamri, Javad Sadri, Ching Y. Suen, and Nicola Nobile.  A novel comprehensive database for Arabic offline handwriting Recognition .  Journal Pattern Recognition, 60:378–393, 2016.
  * [8] Najoua Ben Amara, Omar Mazhoud, Noura Bouzrara, and Noureddine Ellouze.  ARABASE: A relational database for Arabic OCR systems.  International Arab Journal of Information Technology, 2(4):259–266, 2005.
  * [9] T. Anjum and N. Khan.  An attention based method for offline handwritten Urdu text recognition.  In 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 169–174, 2020.
  * [10] J. Aradillas, J. Murillo-Fuentes, and P. M. Olmos.  Improving offline HTR in small datasets by purging unreliable labels.  In 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 25–30, Los Alamitos, CA, USA, sep 2020\. IEEE Computer Society.
  * [11] D. Bahdanau, J. Chorowski, D. Serdyuk, P. Brakel, and Y. Bengio.  End-to-end attention-based large vocabulary speech recognition.  Acoustics, Speech and Signal Processing (ICASSP), pages 4945–4949, 2016.
  * [12] A. Baro, C. Badal, and A. Fornes.  Handwritten historical music recognition by sequence-to-sequence with attention mechanism.  In 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 205–210, Los Alamitos, CA, USA, sep 2020\. IEEE Computer Society.
  * [13] S. Basu, N. Das, R. Sarkar, M. Kundu, M. Nasipuri, and D. K. Basu.  A novel framework for automatic sorting of postal documents with multi-script address blocks.  Pattern Recognition, 43(10):3507–3521, 2010.
  * [14] K. Cho, B. van Merrienboer, C. Gulcehre, D. Bahdanau, F. Bougares, H. Schwenk, and Y. Bengio.  Learning phrase representations using RNN encoder-decoder for statistical machine translation.  ArXiv e-prints, 2014.
  * [15] J. Chung, C. Gulcehre, K. Cho, and Y. Bengio.  Empirical evaluation of gated recurrent neural networks on sequence modeling.  arXiv preprint, 2014.
  * [16] D. Coquenet, C. Chatelain, and T. Paquet.  Recurrence-free unconstrained handwritten text recognition using gated fully convolutional network.  In 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 19–24, Los Alamitos, CA, USA, sep 2020\. IEEE Computer Society.
  * [17] A. Graves and J. Schmidhuber.  Offline handwriting recognition with multidimensional recurrent neural networks.  In Advances in Neural Information Processing Systems (NIPS), 21:5454–552, 2009.
  * [18] Gui, Liangke, Xiaodan Liang, Xiaojun Chang, and Alexander G Hauptmann.  Adaptive context-aware reinforced agent for handwritten text recognition.  In Proceedings of the British Machine Vision Conference (BMVC), 2018.
  * [19] S. Hassan, A. Irfan, A. Mirza, and I. Siddiqi.  Cursive handwritten text recognition using bi-directional LSTMs: A case study on Urdu handwriting.  In 2019 International Conference on Deep Learning and Machine Learning in Emerging Applications (Deep-ML), pages 67–72, 2019.
  * [20] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
  * [21] S. T. Javed and S. Hussain.  Improving Nastalique specific pre recognition process for Urdu OCR.  In Proceedings of 13th IEEE International Multitopic Conference (INMIC), 2009.
  * [22] Yann LeCun, Corinna Cortes, and Christopher J.C. Burges.  The MNIST database of handwritten digits.  http://yann.lecun.com/exdb/mnist/, 1998.  accessed: 19.07.2019.
  * [23] Z. Li, L. Jin, S. Lai, and Y. Zhu.  Improving attention-based handwritten mathematical expression recognition with scale augmentation and drop attention.  In 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 175–180, Los Alamitos, CA, USA, sep 2020\. IEEE Computer Society.
  * [24] N. Ly, C. Nguyen, and M. Nakagawa.  Attention augmented convolutional recurrent network for handwritten japanese text recognition.  In 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 163–168, Los Alamitos, CA, USA, sep 2020\. IEEE Computer Society.
  * [25] N.T. Ly, H.T. Nguyen, and M. Nakagawa.  2D self-attention convolutional recurrent network for offline handwritten text recognition.  In 2021 International Conference on Document Analysis and Recognition (ICDAR), 2021.
  * [26] Sabri A. Mahmoud, Irfan Ahmad, Wasfi G. Al-Khatib, Mohammad Alshayeb, Mohammad Tanvir Parvez, Volker Margner, and Gernot A. Fink.  KHATT: An open Arabic offline handwritten text database.  Pattern Recognition, 47(3):1096–1112, 2014.
  * [27] U. Marti and H. Bunke.  The IAM-database: An English sentence database for off-line handwriting recognition .  Int’l Journal on Document Analysis and Recognition, 5(10):39 – 46, 2002.
  * [28] H. Mouchere, C. Viard-Gaudin, R. Zanibi, D. H. Kim, J. H. Kim, and U. Garain.  ICDAR 2013 CROHME: Third international competition on recognition of online handwritten Mathematical expressions.  In Proc. ICDAR, 2013.
  * [29] O. Mukhtar, S. Setlur, and V. Govindaraju.  Experiments on Urdu text recognition Guide to OCR for Indic Scripts.  Advances in Pattern Recognition, pages 163–171, 2010.
  * [30] S. Naz, S.B. Ahmed, R. Ahmad, and M.I. Razzak .  Zoning features and 2D LSTM for Urdu text-line recognition.  In Procedia Comput Sci, pages 16–22, 2016.
  * [31] S. Naz, A.I. Umar, R. Ahmad, S.B. Ahmed, S.H. Shirazi, and M.I. Razzak.  Urdu Nastaliq text recognition system based on multi-dimensional recurrent neural network and statistical features.  Neural Computing and Applications, pages 219–231, 2017.
  * [32] S. Naz, A.I. Umar, R. Ahmad, I. Siddiqi, S.B. Ahmed, M.I. Razzak, and F. Shafait.  Urdu Nastaliq recognition using convolutional recursive deep learning.  Neurocomputing, pages 80–27, 2017.
  * [33] Saeeda Naz, Arif I. Umar, Syed H. Shirazi, Saad B. Ahmed, Muhammad I. Razzak, and Imran Siddiqi.  Segmentation techniques for recognition of Arabic-like scripts: A comprehensive survey.  Educ Inf Technol, 20(1), 2015.
  * [34] K. Nguyen, C. Nguyen, and M. Nakagawa.  A semantic segmentation-based method for handwritten Japanese text recognition.  In 2020 17th International Conference on Frontiers in Handwriting Recognition (ICFHR), pages 127–132, Los Alamitos, CA, USA, sep 2020\. IEEE Computer Society.
  * [35] I.K. Pathan and A. Ali.  Recognition of offline handwritten isolated Urdu character.  Advances in Computational Research, 4(1):117–121, 2012.
  * [36] A. Raza, I. Siddiqi, A. Abidi, and F. Arif.  An unconstrained benchmark Urdu handwritten sentence database with automatic line segmentation.  In 2012 International Conference on Frontiers in Handwriting Recognition, pages 491–496, 2012.
  * [37] M. I. Razzak, F. Anwar, S. A. Husain, A. Belaïd, and M. Sher.  HMM and fuzzy logic: a hybrid approach for online Urdu script-based languages’ character recognition.  Knowledge Based Systems, 23(8):914–923, 2010.
  * [38] Javad Sadri, Mohammad Reza Yeganehzad, and Javad Saghi.  A novel comprehensive database for offline Persian handwriting recognition.  Pattern Recognition, 60:378–393, 2016.
  * [39] Malik Waqas Sagheer, Chun Lei He, Nicola Nobile, and Ching Y Suen.  A new large Urdu database for off-line handwriting recognition.  In International Conference on Image Analysis and Processing, pages 538–546. Springer, 2009.
  * [40] Kenneth M Sayre.  Machine recognition of handwritten words: A project report.  Pattern recognition, 5(3):213–228, 1973.
  * [41] Kazem Taghva, Thomas A. Nartker, Julie Borsack, and Allen Condit.  UNLV-ISRI document collection for research in OCR and information retrieval.  In Proc. SPIE 3967, Document Recognition and Retrieval VII, 1999\.
  * [42] A. Tong, M. Przybocki, V. Margner, and H. El Abed.  NIST 2013 open handwriting recognition and translation evaluation.  In Proc. NIST, 2013.
  * [43] A. Ul-Hasan, S.B. Ahmed, F. Rashid, F. Shafait, and T.M. Breuel.  Offline printed Urdu Nastaleeq script recognition with bidirectional LSTM networks.  In Proceedings of the 12th international conference on document analysis and recognition (ICDAR), pages 1061–1065. IEEE, 2013.
  * [44] A. Ul-Hasan, F. Shafait, and M. Liwicki.  Curriculum learning for printed text line recognition of ligature-based scripts.  In Proceedings of the 13th international conference on document analysis and recognition (ICDAR), pages 1001–1005. IEEE, 2015.
  * [45] Sam Wiseman and Alexander M. Rush.  Sequence-to-sequence learning as beam-search optimization.  CoRR, abs/1606.02960, 2016.
  * [46] Yan Wu, Jiaxin Fan, Renshuai Tao, Jiakai Wang, Haotong Qin, Aishan Liu, and Xianglong Liu.  Sequential alignment attention model for scene text recognition.  Journal of Visual Communication and Image Representation, 80:103289, 2021.
  * [47] Yirui Wu, Wenxiang Liu, and Shaohua Wan.  Multiple attention encoded cascade R-CNN for scene text detection.  Journal of Visual Communication and Image Representation, 80:103261, 2021.
  * [48] Yao Xiao, Dan Meng, Cewu Lu, and Chi-Keung Tang.  Template-instance loss for offline handwritten chinese character recognition.  In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 315–322, 2019.
  * [49] Jianshu Zhang, Jun Du, and Lirong Dai.  Multi-scale attention with dense encoder for handwritten mathematical expression recognition.  In 2018 24th international conference on pattern recognition (ICPR), pages 2245–2250. IEEE, 2018.
  * [50] Jianshu Zhang, Jun Du, Shiliang Zhang, Dan Liu, Yulong Hu, Jinshui Hu, Si Wei, and Lirong Dai.  Watch, attend and parse: An end-to-end neural network based approach to handwritten mathematical expression recognition.  Pattern Recognition, 71:196–206, 2017.


# One-Electron Quantum Cyclotron as a Milli-eV Dark-Photon Detector

Xing Fan<EMAIL_ADDRESS>Department of Physics, Harvard University,
Cambridge, Massachusetts 02138, USA  Center for Fundamental Physics,
Department of Physics and Astronomy, Northwestern University, Evanston,
Illinois 60208, USA  Gerald Gabrielse<EMAIL_ADDRESS>Center for Fundamental Physics, Department of Physics and Astronomy,
Northwestern University, Evanston, Illinois 60208, USA  Peter W. Graham
<EMAIL_ADDRESS>Stanford Institute for Theoretical Physics, Department
of Physics, Stanford University, Stanford, California 94305, USA  Kavli
Institute for Particle Astrophysics & Cosmology, Department of Physics,
Stanford University, Stanford, California 94305, USA  Roni Harnik
Superconducting Quantum Materials and Systems Center (SQMS), Fermilab,
Batavia, Illinois 60510, USA  Theoretical Physics Division, Fermi National
Accelerator Laboratory, Batavia, Illinois 60510, USA  Thomas G. Myers  Center
for Fundamental Physics, Department of Physics and Astronomy, Northwestern
University, Evanston, Illinois 60208, USA  Harikrishnan Ramani
<EMAIL_ADDRESS>Stanford Institute for Theoretical Physics, Department of
Physics, Stanford University, Stanford, California 94305, USA  Benedict A. D.
Sukra  Center for Fundamental Physics, Department of Physics and Astronomy,
Northwestern University, Evanston, Illinois 60208, USA  Samuel S. Y. Wong
Stanford Institute for Theoretical Physics, Department of Physics, Stanford
University, Stanford, California 94305, USA  Yawen Xiao  Stanford Institute
for Theoretical Physics, Department of Physics, Stanford University, Stanford,
California 94305, USA

###### Abstract

We propose using trapped electrons as high-$Q$ resonators for detecting meV
dark photon dark matter. When the rest energy of the dark photon matches the
energy splitting of the two lowest cyclotron levels, the first excited state
of the electron cyclotron will be resonantly excited. A proof-of-principle
measurement, carried out with one electron, demonstrates that the method is
background free over a 7.4 day search. It sets a limit on dark photon dark
matter at 148 GHz (0.6 meV) that is around 75 times better than previous
constraints. Dark photon dark matter in the 0.1–1 meV mass range (20–200 GHz)
could likely be detected at a similar sensitivity in an apparatus designed for
dark photon detection.

††preprint: FERMILAB-PUB-22-482-SQMS-T

The particle nature of dark matter (DM) and its interactions with the standard
model (SM) of particle physics remains a mystery, despite decades of
experimental scrutiny DMDiscovery1933 ; DMRotationCurve1980 ;
DMGravitationalLensing2006 ; DMBulletCollision2000 ; DM_WMAPGalaxyCenter2007 ;
Planck2018-1 . The mass of the DM is unknown and the possibility that it is
made of ultralight bosons and can be described as a classical wave has
received significant inquiry in recent years FuzzyCDM2000 ;
DarkPhotonMisalignmentMechanism2011 ; ATheoryOfDarkMatter2009 ;
WISPyDarkMatter2012 ; ReviewScalarFieldDarkMatter ;
UltraLightScalarCosmologicalDarkMatter . One such ultralight dark matter
candidate is the dark photon (DP), a hypothetical spin-1 particle DPTheory1986
; Okun:1982xi  that is theoretically well motivated and possesses cosmological
production mechanisms that can produce the observed DM abundance
Graham:2015rva ; Dror:2018pdh ; Agrawal:2018vin ; LowEnergyFrontierReview2010
; Ahmed:2020fhc ; GravitationalProductionOfDP2021 ; Co_2019 ; Co_2021 . Such a
DP will generically have a kinetic mixing with the SM photon because this term
is allowed by the symmetries of the theory (so long as the DP does not have a
non-Abelian gauge symmetry). This kinetic mixing allows dark photon dark
matter (DPDM) to be looked for in existing DPLimitsReview2021 ;
HuntForDarkPhoton2020  and forthcoming experiments Antypas:2022asj .

In this Letter, we propose a promising new direct detection technique using
one-quantum transitions of one or more trapped electrons that are initially
cooled to their cyclotron ground state. We demonstrate the viability of this
technique with a proof-of-principle measurement that sets a limit 75 times
better than previous constraints. This new limit is only for a narrow mass
range because of limitations of an apparatus designed for making the most
accurate measurements of the electron and positron magnetic moments
atomsNewMeasurement2019 —to test the standard model’s most precise predictions
HarvardMagneticMoment2008 ; HarvardMagneticMoment2011 ; Nio2018TenthOrder ;
CsAlphaScience2018 ; RbAlpha2020Nature ; Alpha2006fixed ;
atomsTheoryReview2019 ; DehmeltMagneticMoment ; EfficientPositronAccumulation
. With an apparatus designed for DPDM detection, including efficient scanning
of the resonant frequency, the mass range could be greatly extended.

The relevant properties of the DP are captured by the Lagrangian (in natural
units) DPTheory1986

$\displaystyle\mathcal{L}\supset-\frac{1}{4}F^{\prime}_{\mu\nu}F^{\prime\mu\nu}+\frac{\epsilon}{2}F^{\mu\nu}F^{\prime}_{\mu\nu}+\frac{1}{2}m_{A^{\prime}}^{2}A^{\prime}_{\mu}A^{\prime\mu}.$
(1)

Here $A^{\prime}_{\mu}$ is the DP vector, $F^{\prime}$ and $F$ are the DP and
SM photon field strengths respectively, $\epsilon$ is the kinetic mixing
parameter, and $m_{A^{\prime}}$ is the mass of the DP. The DPDM manifests as
dark electric and magnetic fields oscillating at a frequency set by the DP
mass $\omega_{A^{\prime}}=m_{A^{\prime}}c^{2}/\hbar$, where $c$ is the speed
of light and $\hbar$ is the reduced Planck constant. In the presence of a
kinetic mixing with the SM photon, these dark fields cause effective
($\epsilon$-suppressed) SM electromagnetic fields. These can be detected by
devices sensitive to tiny electric or magnetic fields at the frequency
$\omega_{A^{\prime}}$.

A plethora of complementary experiments have been designed with sensitivities
to different DM masses. The frequency range we focus on, 20 to 200 GHz (i.e.,
0.1 to 1 meV) is particularly challenging experimentally, yet well-motivated
theoretically by the minimal DPDM model with purely gravitational production
Graham:2015rva . This range is too high for extremely high-$Q$ resonators
(e.g., as used by ADMX ADMXSideCar2018 ; ADMXTechnicalDesignReview2021 , CAPP
CAPP2020_7ueV ; CAPP2020_13ueV ; CAPP2021_10ueV , and HAYSTAC
HAYSTAC2018_24ueV ; HAYSTAC2021_17ueV ). At the same time, the corresponding
photons are below the energy threshold for existing single photon detection
experiments such as those in Refs. Chiles:2021gxk ; Hochberg:2021yud ;
SinglePhotonReview2011 . Alternate experiments involving dish antennae or
metal plates have been proposed or are underway around our frequency range
DOSUE2022_100ueV ; DarkPhoton_Tomita2020_115ueV ;
DarkPhoton_Knirck_2018_0.8meV ; BREAD:2021tpx . The use of trapped ion
crystals was proposed for the MHz frequency range gilmore2021quantum .

The new DPDM detector proposed and demonstrated here is one (or more) electron
in a Penning trap [Fig. 1(a)]—a “one-electron quantum cyclotron”
QuantumCyclotron . The trapped electron is a high-$Q$ resonator with a 20–200
GHz resonant frequency determined by the applied magnetic field of the trap.
The DPDM wave would drive the electron to jump from the cyclotron ground state
to the first excited state [Fig. 1(b)] if the cyclotron level spacing
corresponds to the DP’s frequency. The cyclotron quantum state is monitored in
real time [Fig. 1(c)] to search for excitations. To determine the cyclotron
excitation rate, we compute the SM electric field induced by the DPDM inside
the microwave cavity formed by the electrodes of the Penning trap. Similar
techniques were employed in the context of ion traps to search for millicharge
dark matter in Ref. Budker:2021quh .

Two motions of the trapped electron (mass $m_{e}$ and charge $-e$) are key.
The quantized cyclotron oscillation (in a plane perpendicular to a strong
magnetic field $B_{0}\hat{z}$) is potentially excited by the DPDM-generated
photon field in the $xy$ plane. For $B_{0}=5.3$ T, a photon resonant at the
cyclotron frequency, $\omega_{c}/(2\pi)=eB_{0}/(2\pi m_{e})=148$ GHz could
increase the cyclotron energy by one quantum, $\hbar\omega_{c}$ [Fig. 1(b)].
The frequency of the electron’s classical axial oscillation, along the
magnetic field, is used to detect cyclotron excitations DehmeltMagneticBottle
. The axial oscillation frequency is set at $\omega_{z}/(2\pi)=114$ MHz for
$n_{c}=0$ by the static potentials applied to the trap electrodes.

Figure 1: (a) Single isolated electron in a cylindrical Penning trap. (b)
Quantum cyclotron energy levels of the trapped electron. (c) Resolution of
cyclotron states $n_{c}$ by measuring axial frequency shift. A resonant
external cyclotron excitation drive is applied to cause stimulated transitions
of $n_{c}$.

A quantum nondemolition coupling of the two motions makes it possible to
detect one-quantum cyclotron excitations without causing a change to the
cyclotron quantum number FanBackActionPRL2021 . The monitored axial frequency
shifts in proportion to the cyclotron quantum number $n_{c}$ by
$\Delta\omega_{z}=n_{c}\delta$ [Fig. 1(c)], due to a magnetic bottle gradient
that adds $B_{2}z^{2}\hat{z}$ to the magnetic field. The potential along $z$
now includes a quadratic magnetic potential
$V(z)={m_{e}\omega_{z}^{2}z^{2}}/{2}+\hbar eB_{2}n_{c}z^{2}/m_{e}$
DehmeltMagneticBottle  given by the product of the cyclotron orbital magnetic
moment and $B_{2}z^{2}$. A nickel ring encircling the trap generates
$B_{2}=300$ T/m2, making $\delta/(2\pi)\equiv\hbar eB_{2}/(2\pi
m_{e}^{2}\omega_{z})=1.3$ Hz. The axial frequency is read out by keeping the
axial oscillation amplitude at large $z_{\text{max}}=60~{}\mu\mathrm{m}$ by
feeding back an electrical signal induced by the oscillation itself
SelfExcitedOscillator . The axial shift $\delta/(2\pi)$ from a one-quantum
cyclotron excitation is 8 times larger than the
$\sigma/(2\pi)=0.16~{}\mathrm{Hz}$ standard deviation for fluctuations,
predominantly from the axial amplitude fluctuation coupled to the trap
potential’s anharmonicitySelfExcitedOscillator . The axial detection is
continuously on, but the coupling is too weak to cause the quantum Zeno effect
and suppress the cyclotron transitionsQuantumZenoEffectWineland ; ThesisPeil .
The distribution of measured axial frequencies for 2 s averaging time is
displayed in Fig. 2.

Because of the $B_{2}$ coupling, the cyclotron resonance frequency is also
broadened to $Q=\omega_{c}/\Delta\omega_{c}=B/(B_{2}z_{\text{max}}^{2})\approx
10^{7}$ from the unbroadened $Q=10^{9}$ HarvardMagneticMoment2008 . This does
not worsen the smallest $\epsilon$ we can prove, and slightly broadens the
DPDM sensitivity bandwidth beyond the intrinsic $Q=10^{6}$ bandwidth of the
dark matter Bovy_2012_DMVelocity ; Schonrich_DMVelocity ; Golubov_DMVelocity .

Figure 2: Distribution of measured axial frequency fluctuations
$\Delta\omega_{z}/\delta$ showing the standard deviation $\sigma$ and the
chosen threshold at $5\sigma$.

The electron is suspended at the center of a trap [Fig. 1(a)] that a dilution
refrigerator keeps at a temperature of $T=50$ mK. The electron cyclotron
motion cools via synchrotron radiation and is not excited by blackbody
photons. In thermal equilibrium, the average quantum number is
QuantumCyclotron

$\bar{n}_{c}=\left[\exp\left(\frac{\hbar\omega_{c}}{k_{\mathrm{B}}T}\right)-1\right]^{-1}=1.9\times
10^{-62}\approx 0,$ (2)

where $k_{B}$ is the Boltzmann constant. The electron is thus essentially
always in its quantum cyclotron ground state $n_{c}=0$, with no background
excitations from blackbody photons estimated to take place for many years
QuantumCyclotron .

Photons dynamically induced by the DPDM field could produce cyclotron
transitions from $n_{c}=0\rightarrow n_{c}=1$. For radiation broader than the
linewidth, the transition rate is loudon2000quantum ; NoiseDrivenTransition

$\Gamma=\int\frac{\pi
e^{2}}{2m_{e}\hbar\omega_{c}}S_{E}\left(\omega\right)\chi(\omega,\omega_{c})d\omega,$
(3)

where
$\chi(\omega,\omega_{c})=\tfrac{1}{\sqrt{2\pi}\Delta\omega_{c}}\exp\left[-\tfrac{1}{2}\left(\tfrac{\omega-\omega_{c}}{\Delta\omega_{c}}\right)^{2}\right]$
is the normalized cyclotron line shape with linewidth $\Delta\omega_{c}$, and
$S_{E}(\omega)\,d\omega$ is the power in the interval
$\\{\omega,\omega+d\omega\\}$ for the component of the DM-induced electric
field in the $xy$ plane. For DPDM with spread
$\Delta\omega_{A^{\prime}}\approx 10^{-6}\omega_{A^{\prime}}$ DMDispersion2000
, and for a cylindrical cavity, $S_{E}(\omega)$ can be approximated as a
boxcar window function with value

$\displaystyle S_{E}(\omega)=\kappa^{2}\times\epsilon^{2}\frac{\rho_{\rm
DM}c^{2}}{\varepsilon_{0}\Delta\omega_{A^{\prime}}}\langle\sin^{2}\theta\rangle$
(4)

in the interval
$\\{\omega_{A^{\prime}},\omega_{A^{\prime}}+\Delta\omega_{A^{\prime}}\\}$ and
zero outside, where $\rho_{\mathrm{DM}}c^{2}=0.3$ GeV/cm3 is the local DM
density Planck2018-1  and $\varepsilon_{0}$ is the vacuum permittivity. We
assume that the angle between the DPDM electric field and the $z$ axis,
$\theta$, changes randomly and is adequately sampled in observation times
$T_{\rm obs}\gg{1}/{\Delta\omega_{A^{\prime}}}\approx
10^{-6}~{}\mathrm{s}\times\left(\frac{2\pi\times
148~{}\textrm{GHz}}{\omega_{A^{\prime}}}\right)$. Thus the angular average
that captures the component along the $xy$ plane evaluates to
$\langle\sin^{2}\theta\rangle=2/3$. A fixed DPDM polarization
DPLimitsReview2021  would essentially not change the result given that our
apparatus, off the Earth’s rotation axis, changes orientation as the Earth
rotates during the $T_{\rm obs}\gg$ 1 day observation time for this experiment
forthcoming . We calculate the effect of fixed DPDM polarization on our
projected future sensitivity in Ref. forthcoming .

Finally, $\kappa$ is the enhancement of the DPDM-induced electric field at the
position of the electron by the trap’s microwave structure DMRadioPRD_2015 :

$\kappa=\left|\sum_{n}\frac{\omega^{2}}{\omega^{2}-\omega_{n}^{2}(1-\tfrac{2i}{Q_{n}})}\frac{\int
dV\vec{E}_{n}^{*}(\bf{r})\cdot\hat{\mathbf{x}}}{\int
dV|\vec{E}_{n}(\bf{r})|^{2}}\vec{E}_{n}(\bm{0})\cdot\hat{\mathbf{x}}\right|.$
(5)

Without loss of generality, $\mathbf{x}$ is taken to be the DPDM polarization
direction; $n$ runs over all resonant modes; and $\omega_{n}$, $Q_{n}$, and
$\vec{E}_{n}(\bf{r})$ are the resonant frequency, quality factor, and electric
field of the mode at position $\bf{r}$ respectively. The last term
$\vec{E}_{n}(\bm{0})\cdot\hat{\mathbf{x}}$ captures the transverse electric
field at the center that drives electron cyclotron transition.

Figure 3 shows the calculated frequency spectrum for $\kappa^{2}$ using
measured resonant frequencies and $Q$ factors. The sharp peaks are from cavity
modes that couple strongly to the cyclotron motion of an electron suspended at
the cavity center. The microwave cavity resonances below 170 GHz for the
cylindrical Penning trap CylindricalPenningTrap ;
CylindricalPenningTrapDemonstrated  (with radius $\rho_{0}=4.527$ mm and
height $2z_{0}=7.790$ mm) have all been carefully mapped using parametrically
pumped electrons SynchronizedElectronsPRL ; SynchronizedElectronsPRA . The
measured frequencies agree with an ideal cylindrical model to within a few
percentages.

Figure 3: Calculated $\kappa^{2}$ for the measured microwave resonant
frequencies and $Q$ factors. For this demonstration search (at 148.04786 GHz),
$\kappa^{2}=2.37$ (dashed line).

The high $\kappa$ values at cavity mode resonances are unfortunately not
compatible with the existing $2\,\text{s}$ averaging time needed to resolve
the one-quantum cyclotron transitions in our apparatus. For this averaging
time, the magnetic field must be chosen to keep the electron cyclotron
frequency far from resonance with all cavity modes that couple to the centered
electron. This inhibits the spontaneous emission of synchrotron radiation to
lengthen the lifetime of the excited cyclotron state $\tau_{c}$
InhibitionLetter . For this demonstration, $\tau_{c}$ was set at the longest
value, $\tau_{c}=7.2$ s, a factor of 80 longer than its free space value
InhibitionLetter ; HarvardMagneticMoment2011 , to avoid missing a signal. The
photons induced by the DPDM, being away from resonance, thus cannot build up
in a cavity radiation mode. Fortunately, our calculation shows that
$\kappa^{2}$ remains usefully large, with $\kappa^{2}=2.37$ pertaining to our
demonstration at 148 GHz. The cylindrical symmetry of the conducting cavity
boundary causes an enhancement in the DM-induced electric field at the trap
center (akin to the “focusing” effect found in the dish antenna proposals
DishAntennaProposal-1 ; DishAntennaProposal-2 ). A future optimization by
carefully choosing $\tau_{c}$ and by using a larger spherical trap cavity can
result in a 25-fold increase in $\kappa$.

The new search for 148 GHz DPDM [Fig. 4(a)] is for $n_{c}=0$ to $n_{c}\geq 1$
cyclotron excitations over $T_{\mathrm{obs}}=7.4$ days (Table 1). Shifts of
the electron’s axial frequency are averaged over $t_{\mathrm{ave}}=2$ second
intervals and recorded as a function of time. The trapping potential is slowly
adjusted to eliminate slow drifts of the axial frequency. Figure 4(b) shows
$\Delta\omega_{z}/\delta$ for 24 hours of the 7.4 day search. A cyclotron
excitation to the first excited state would produce
$\Delta\omega_{z}/\delta=1$. Any $\Delta\omega_{z}$ larger than a $5\sigma$
threshold (i.e. $\Delta\omega_{z}/\delta=5\sigma/\delta\geq 0.65$) would be
interpreted as being potentially caused by DPDM. No such excitation is
detected during the 7.4 days.

The search was suspended for 25 min of calibration every 6 hours, indicated by
the breaks in Fig. 4(b). The one-quantum response is confirmed using microwave
photons sent into the cavityHarvardMagneticMoment2011 . The detector bandwidth
of 33 kHz is also deduced by measuring the shift $\Delta\omega_{z}/\delta$ as
a microwave drive is swept through resonance with the 148 GHz cyclotron
frequency [Fig. 4(c)]. The width is broadened by the large self-excited axial
oscillation in the magnetic gradient described above. Cyclotron frequency
shifts are negligible given the extremely low magnetic field drift rate of
$\Delta B/B=10^{-10}$ per hour that is realized using a carefully shimmed
self-shielding solenoid SelfShieldingSolenoid ; Helium3NMR2019 ;
atomsNewMeasurement2019 . This 33 kHz detector bandwidth slightly broadens the
sensitivity bandwidth beyond the ${\sim}100\,\text{kHz}$ bandwidth expected of
the dark matter.

Figure 4: (a) The DPDM measurement cycle. (b) Monitored axial frequency shift shows no DPDM-induced cyclotron excitations during 24 hours of the 7.4 day search. (c) Measured cyclotron line shape $\chi(\omega,\omega_{c})$ with the self-excitation on. Run No. | Time (date.hour:minute) | Observation length (s)
---|---|---
1 | 11.12:46 – 13.13:15 | 148058
2 | 14.18:26 – 15.11:33 | 58162
3 | 15.11:50 – 17.17:22 | 179698
4 | 17.18:38 – 18.18:40 | 80640
5 | 19.12:15 – 21.15:43 | 172312
Total | $\cdots$ | 638870
Table 1: Datasets for DPDM search in March 2022. Each run consists of the
repeated measurement cycle in Fig. 4.

The lowest cyclotron excited state decays to the ground state by the
spontaneous emission of synchrotron radiation photons. The decay time for each
excitation is a random selection from an exponential distribution with an
average lifetime of $\tau_{c}=7.2$ s. The choice of a detection threshold at
$5\sigma=0.65\delta$ means that an excitation that decays in less than
$0.65\times t_{\mathrm{ave}}=1.3$ s will be missed, giving a detection
efficiency of

$\zeta=\int_{1.3\mathrm{s}}^{\infty}\frac{1}{\tau_{c}}\exp\left(-\frac{t}{\tau_{c}}\right)dt=83\%.$
(6)

Correcting for the observed fluctuation spectrum in Fig. 2 affects the result
by only 1%.

Figure 5: (a) The new experimental limit on DPDM with 90 % confidence level.
In a narrow frequency band, the exclusion limit from this work is about 75
times better than the existing constraints from XENON1T (hatched)
DMDetectorAsDPHelioscope ; Xenon1TResultDarkPhoton ; Xenon1TPRL2019  and DM
cosmology (dashed) WISPyDarkMatter2012 . (b) Possible sensitivity with a
dedicated apparatus (blue) is compared to the limits from DOSUE
DOSUE2022_100ueV  and Tokyo DarkPhoton_Knirck_2018_0.8meV .

The conversion to $\Gamma$ is now straightforward. Using the standard estimate
of the upper limit of the null measurement LeoParticlePhysicsTextBook , the
upper limit on the DPDM excitation rate with $CL=90$% confidence level is

$\Gamma<-\frac{1}{\zeta T_{\mathrm{obs}}}\log\left(1-CL\right)=4.33\times
10^{-6}~{}\mathrm{s}^{-1},$ (7)

which is also the upper limit of the one-electron detector’s dark count rate.
Using measured values and Eqs. (3) and (4), our limit on the kinetic mixing
parameter is

$\epsilon<3.2\times 10^{-11}$ (8)

at $m_{A^{\prime}}=0.612\,276$ meV, which improves on previous limits by a
factor of 75. The corresponding microwave electric field detected, specified
by $\sqrt{2\pi S_{E}(\omega)}$ is given by
$2.5~{}\mathrm{pV/(cm}\sqrt{\text{Hz}})$, with the measurement bandwidth, 0.45
nV/cm. The new $\epsilon$ limit is shown in Fig. 5(a), with the limit from the
XENON1T (black hatched) DMDetectorAsDPHelioscope ; Xenon1TResultDarkPhoton ;
Xenon1TPRL2019  and the limit from DM cosmology (dashed) WISPyDarkMatter2012 .

Only a narrow DPDM mass range is accessed in this initial demonstration due to
limitations of the apparatus, which was designed to make the magnetic field
exceptionally stable rather than readily swept. Searching 20–200 GHz
($\sim$0.1 meV to 1 meV) seems feasible in an apparatus that is designed for
dark matter searches. Affordably sweeping the magnetic field over such a broad
range requires cooling with a refrigerator rather than cryogenic liquids. For
DM with $Q=10^{6}$, making $t_{m}=15$ s measurements spaced by $10^{-6}$
relative frequency steps would cover the mentioned range in about a year. The
DPDM sensitivity established above is approximately

$\displaystyle\epsilon\approx 8\times
10^{-11}\frac{\omega_{A^{\prime}}}{2\pi\times
150~{}\textrm{GHz}}\frac{40}{\kappa}\left(\frac{10}{n_{\rm
e}}\right)^{\frac{1}{2}}\left(\frac{15~{}\textrm{s}}{t_{\rm
m}}\right)^{\frac{1}{2}},$ (9)

where $n_{e}$ is the number of electrons used to sense DM and $t_{m}$ is the
measurement time. A shorter measurement time (15 s rather than 7.4 days) would
decrease the sensitivity $\epsilon$ by a factor of 200. It seems feasible to
largely recapture this factor by using $n_{e}=10$ electrons, increasing
$\kappa$ to ${\sim}40$ by using a spherical geometry to get more focusing, and
increasing the radius of the sphere to $r=25~{}\textrm{mm}$ forthcoming .
Resulting reductions in the induced axial oscillation signal needed to observe
one-quantum cyclotron jumps would be compensated for by greatly increasing the
size of the magnetic bottle gradient that couples the cyclotron and axial
motion. The blue dashed line in Fig. 5(b) is an estimate of what may be
possible, assuming that the trap cavity can be tuned during the sweep to avoid
cavity mode resonances. For a spherical cavity,
$\kappa\propto\frac{\omega_{A^{\prime}}r}{c}$ which cancels the
$\omega_{A^{\prime}}$ in Eq. (9), leading to a flat sensitivity curve. A more
detailed optimization is clearly warranted forthcoming .

In conclusion, we have proposed and demonstrated the possibility of using one-
electron quantum cyclotrons within a microwave trap cavity to search for DPDM.
A big advantage is that detection is essentially free of background
excitations, making the detection sensitivity of the transition rate scale
with observation time as $T_{\mathrm{obs}}^{-1}$ rather than
$T_{\mathrm{obs}}^{-\frac{1}{2}}$. The obtained limit is the most sensitive
ever obtained in the challenging meV range and all required parameters for the
DPDM search are measured in situ. The narrow frequency range realized in this
first demonstration could be greatly extended in an apparatus designed and
optimized for dark matter detection. Furthermore, the trapped electron is
naturally compatible with a large magnetic field, and application to an axion
search is promising with a dedicated antenna forthcoming . This proposal and
demonstration thus open a new direction for DPDM searches.

###### Acknowledgements.

This work was supported by the U.S. DOE, Office of Science, National QIS
Research Centers, Superconducting Quantum Materials and Systems Center (SQMS)
under Contract No. DE-AC02-07CH11359. Additional support was provided by NSF
Grants No. PHY-1903756, No. PHY-2110565, and No. PHY-2014215; by the John
Templeton Foundation Grants No. 61906 and No. 61039; by the Simons
Investigator Award No. 824870; by the DOE HEP QuantISED Award No. 100495; by
the Gordon and Betty Moore Foundation Grant No. GBMF7946; and by the Masason
Foundation. S.W. was supported in part by the Clark Fellowship. Y.X. was
supported in part by the Vincent and Lily Woo Fellowship.

## References

  * (1) F. Zwicky, Helv. Phys. Acta 6, 110 (1933).
  * (2) V. C. Rubin, W. K. Ford Jr., and N. Thonnard, Astrophys. J. 238, 471 (1980).
  * (3) D. Clowe, M. Bradač, A. H. Gonzalez, M. Markevitch, S. W. Randall, C. Jones, and D. Zaritsky, Astrophys. J. 648, L109 (2006).
  * (4) D. N. Spergel and P. J. Steinhardt, Phys. Rev. Lett. 84, 3760 (2000).
  * (5) D. Hooper, D. P. Finkbeiner, and G. Dobler, Phys. Rev. D 76, 083012 (2007).
  * (6) Planck Collaboration, Astron. Astrophys. 641, A1 (2020).
  * (7) W. Hu, R. Barkana, and A. Gruzinov, Phys. Rev. Lett. 85, 1158 (2000).
  * (8) A. E. Nelson and J. Scholtz, Phys. Rev. D 84, 103501 (2011).
  * (9) N. Arkani-Hamed, D. P. Finkbeiner, T. R. Slatyer, and N. Weiner, Phys. Rev. D 79, 015014 (2009).
  * (10) P. Arias, D. Cadamuro, M. Goodsell, J. Jaeckel, J. Redondo, and A. Ringwald, J. Cosmol. Astropart. Phys. 013 (2012).
  * (11) J. Magaña and T. Matos, J. Phys. Conf. Ser. 378, 012012 (2012).
  * (12) L. Hui, J. P. Ostriker, S. Tremaine, and E. Witten, Phys. Rev. D 95, 043541 (2017).
  * (13) B. Holdom, Phys. Lett. 166B, 196 (1986).
  * (14) L. B. Okun, Sov. Phys. JETP 56, 502 (1982).
  * (15) P. W. Graham, J. Mardon, and S. Rajendran, Phys. Rev. D 93, 103520 (2016).
  * (16) J. A. Dror, K. Harigaya, and V. Narayan, Phys. Rev. D 99, 035036 (2019).
  * (17) P. Agrawal, N. Kitajima, M. Reece, T. Sekiguchi, and F. Takahashi, Phys. Lett. B 801, 135136 (2020).
  * (18) J. Jaeckel and A. Ringwald, Annu. Rev. Nucl. Part. Sci. 60, 405 (2010).
  * (19) A. Ahmed, B. Grzadkowski, and A. Socha, J. High Energy Phys. 08, 059 (2020).
  * (20) E. W. Kolb and A. J. Long, J. High Energy Phys. 03, 283 (2021).
  * (21) R. T. Co, A. Pierce, Z. Zhang, and Y. Zhao, Phys. Rev. D 99, 075002 (2019).
  * (22) R. T. Co, K. Harigaya, and A. Pierce, J. High Energy Phys. 12, 099 (2021).
  * (23) A. Caputo, A. J. Millar, C. A. J. O’Hare, and E. Vitagliano, Phys. Rev. D 104, 095029 (2021).
  * (24) A. Filippi and M. De Napoli, Rev. Phys. 5, 100042 (2020).
  * (25) D. Antypas et al., arXiv:2203.14915 .
  * (26) G. Gabrielse, S. E. Fayer, T. G. Myers, and X. Fan, Atoms 7, 45 (2019).
  * (27) D. Hanneke, S. Fogwell, and G. Gabrielse, Phys. Rev. Lett. 100, 120801 (2008).
  * (28) D. Hanneke, S. Fogwell Hoogerheide, and G. Gabrielse, Phys. Rev. A 83, 052122 (2011).
  * (29) T. Aoyama, T. Kinoshita, and M. Nio, Phys. Rev. D 97, 036001 (2018).
  * (30) R. H. Parker, C. Yu, W. Zhong, B. Estey, and H. Müller, Science 360, 191 (2018).
  * (31) L. Morel, Z. Yao, P. Cladé, and S. Guellati-Khélifa, Nature (London) 588, 61 (2020).
  * (32) G. Gabrielse, D. Hanneke, T. Kinoshita, M. Nio, and B. Odom, Phys. Rev. Lett. 97, 030802 (2006).
  * (33) T. Aoyama, T. Kinoshita, and M. Nio, Atoms 7, 28 (2019).
  * (34) R. S. Van Dyck, Jr., P. B. Schwinberg, and H. G. Dehmelt, Phys. Rev. Lett. 59, 26 (1987).
  * (35) S. Fogwell Hoogerheide, J. C. Dorr, E. Novitski, and G. Gabrielse, Rev. Sci. Instrum. 86, 053301 (2015).
  * (36) C. Boutan et al., Phys. Rev. Lett. 121, 261302 (2018).
  * (37) R. Khatiwada et al., Rev. Sci. Instrum. 92, 124502 (2021).
  * (38) S. Lee, S. Ahn, J. Choi, B. R. Ko, and Y. K. Semertzidis, Phys. Rev. Lett. 124, 101802 (2020).
  * (39) J. Jeong, S. W. Youn, S. Bae, J. Kim, T. Seong, J. E. Kim, and Y. K. Semertzidis, Phys. Rev. Lett. 125, 221302 (2020).
  * (40) O. Kwon et al., Phys. Rev. Lett. 126, 191802 (2021).
  * (41) L. Zhong et al., Phys. Rev. D 97, 092001 (2018).
  * (42) K. M. Backes et al., Nature (London) 590, 238 (2021).
  * (43) J. Chiles et al., Phys. Rev. Lett. 128, 231802 (2022).
  * (44) Y. Hochberg, B. V. Lehmann, I. Charaev, J. Chiles, M. Colangelo, S. W. Nam, and K. K. Berggren, arXiv:2110.01586 .
  * (45) M. D. Eisaman, J. Fan, A. Migdall, and S. V. Polyakov, Rev. Sci. Instrum. 82, 071101 (2011).
  * (46) S. Kotaka et al., arXiv:2205.03679 .
  * (47) N. Tomita, S. Oguri, Y. Inoue, M. Minowa, T. Nagasaki, J. Suzuki, and O. Tajima, J. Cosmol. Astropart. Phys. 09 (2020) 012 .
  * (48) S. Knirck, T. Yamazaki, Y. Okesaku, S. Asai, T. Idehara, and T. Inada, J. Cosmol. Astropart. Phys. 11 (2018) 031 .
  * (49) J. Liu et al., Phys. Rev. Lett. 128, 131801 (2022).
  * (50) K. A. Gilmore, M. Affolter, R. J. Lewis-Swan, D. Barberena, E. Jordan, A. M. Rey, and J. J. Bollinger, Science 373, 673 (2021).
  * (51) S. Peil and G. Gabrielse, Phys. Rev. Lett. 83, 1287 (1999).
  * (52) D. Budker, P. W. Graham, H. Ramani, F. Schmidt-Kaler, C. Smorra, and S. Ulmer, PRX Quantum 3, 010330 (2022).
  * (53) R. Van Dyck, Jr., P. Ekstrom, and H. Dehmelt, Nature (London) 262, 776 (1976).
  * (54) X. Fan and G. Gabrielse, Phys. Rev. Lett. 126, 070402 (2021).
  * (55) B. D’Urso, R. Van Handel, B. Odom, D. Hanneke, and G. Gabrielse, Phys. Rev. Lett. 94, 113002 (2005).
  * (56) W. M. Itano, D. J. Heinzen, J. J. Bollinger, and D. J. Wineland, Phys. Rev. A 41, 2295 (1990).
  * (57) S. Peil, Ph.D. thesis, Harvard University, 1999.
  * (58) J. Bovy et al., Astrophys. J. 759, 131 (2012).
  * (59) R. Schönrich, Mon. Not. R. Astro. Soc. 427, 274 (2012).
  * (60) O. Golubov et al., Astron. Astrophys. 557, A92 (2013).
  * (61) R. Loudon, The Quantum Theory of Light, 3rd ed. (Oxford University Press, New York, 2000).
  * (62) T. A. Savard, K. M. O’Hara, and J. E. Thomas, Phys. Rev. A 56, R1095 (1997).
  * (63) K. Gebhardt et al., Astrophys. J. 539, L13 (2000).
  * (64) X. Fan, G. Gabrielse, P. W. Graham, R. Harnik, T. G. Myers, H. Ramani, B. A. D. Sukra, S. S. Y. Wong, and Y. Xiao, (to be published).
  * (65) S. Chaudhuri, P. W. Graham, K. Irwin, J. Mardon, S. Rajendran, and Y. Zhao, Phys. Rev. D 92, 075012 (2015).
  * (66) G. Gabrielse and F. C. MacKintosh, Int. J. Mass Spec. Ion Proc. 57, 1 (1984).
  * (67) J. Tan and G. Gabrielse, Appl. Phys. Lett. 55, 2144 (1989).
  * (68) J. Tan and G. Gabrielse, Phys. Rev. Lett. 67, 3090 (1991).
  * (69) J. Tan and G. Gabrielse, Phys. Rev. A 48, 3105 (1993).
  * (70) G. Gabrielse and H. Dehmelt, Phys. Rev. Lett. 55, 67 (1985).
  * (71) J. Jaeckel and J. Redondo, J. Cosmol. Astropart. Phys. 11 (2013) 016 .
  * (72) D. Horns, J. Jaeckel, A. Lindner, A. Lobanov, J. Redondo, and A. Ringwald, J. Cosmol. Astropart. Phys. 04 (2013) 016 .
  * (73) G. Gabrielse and J. Tan, J. Appl. Phys. 63, 5143 (1988).
  * (74) X. Fan, S. E. Fayer, and G. Gabrielse, Rev. Sci. Instrum. 90, 083107 (2019).
  * (75) H. An, M. Pospelov, and J. Pradler, Phys. Rev. Lett. 111, 041302 (2013).
  * (76) H. An, M. Pospelov, J. Pradler, and A. Ritz, Phys. Rev. D 102, 115022 (2020).
  * (77) E. Aprile et al., Phys. Rev. Lett. 123, 251801 (2019).
  * (78) W. R. Leo, Techniques for Nuclear and Particle Physics E12 1xperiments (Springer-Verlag, Berlin, 1994).