heightadjust=all, floatrowsep=columnsep

# Explore Faster Localization Learning For Scene Text Detection

Yuzhong Zhao1 Yuanqiang Cai2 Weijia Wu3 Weiqiang Wang1†

###### Abstract

Generally pre-training and long-time training computation are necessary for
obtaining a good-performance text detector based on deep networks. In this
paper, we present a new scene text detection network (called FANet) with a
Fast convergence speed and Accurate text localization. The proposed FANet is
an end-to-end text detector based on transformer feature learning and
normalized Fourier descriptor modeling, where the Fourier Descriptor Proposal
Network and Iterative Text Decoding Network are designed to efficiently and
accurately identify text proposals. Additionally, a Dense Matching Strategy
and a well-designed loss function are also proposed for optimizing the network
performance. Extensive experiments are carried out to demonstrate that the
proposed FANet can achieve the SOTA performance with fewer training epochs and
no pre-training. When we introduce additional data for pre-training, the
proposed FANet can achieve SOTA performance on MSRA-TD500, CTW1500 and
TotalText. The ablation experiments also verify the effectiveness of our
contributions.

††1 University of Chinese Academy of Sciences, China.††2 Beijing University of
Posts and Telecommunications, China.††3 Zhejiang Unversity, China.†††
Corresponding author<EMAIL_ADDRESS>
##  1 Introduction

Scene text detection is an important task of computer vision, and a basis of
various text-related applications, so many researchers are concerned about the
issue [34, 39, 14, 30, 28]. The rise and wide application of deep learning
make great progress in scene text detection, so the best performance on
benchmark datasets is refreshed constantly. However, most of the state-of-the-
art methods [26, 18, 38, 8, 31, 29] rely on long-time training to achieve a
good performance. Generally researchers use related large datasets for long-
time pre-training, and then finetune the network on the target dataset, or
directly carry out long-time training on the target dataset. These approaches
are not suitable for scenarios that require to rapidly generate models or no
large dataset for pre-training.

Figure 1: The comparison of convergence performance between the proposed FANet
and other state-of-the-art (SOTA) methods. After the training of 100 epochs,
the FANet outperforms current best SOTA method by $6.5\%$ $(83.3\%vs.76.8\%)$
and $16.5\%$ $(84.8\%vs.68.3\%)$ based on F-measure on datasets CTW1500 and
MSRA-TD500 respectively. Figure 2: Overview of the proposed FANet.
${\bm{c}}^{d}$ denotes the Fourier proposal predicted by $d$-th transformer
decoder layer, ${\bm{q}}^{d-1}$ denote the input query for $d$-th transformer
decoder layer, $d=1,\cdots,D$, where $D$ is the number of decoder layers (6 in
our experiments). As some special cases, ${\bm{c}}^{0}$ denotes the Fourier
proposal predicted by Fourier Descriptor Proposal Network (FDPN),
${\bm{q}}^{D}$ is the encoded memory of the last transformer decoder layer. We
only describe the calculation process of a single proposal ${\bm{c}}^{D}$ in
the network for simplicity. In the Iterative Text Decoding Network (ITDN), the
blue top-down arrow indicates that the Fourier descriptor is used as the
reference location for the Multi-Scale Deformable Attention module, the orange
bottom-up arrow indicates that the offset predicted by the transformer decoder
layer is used to refine the Fourier proposals.

Inspired by the feature learning ability of the transformer [22] and
deformable DETR[37] and the representation ability of Fourier descriptor for
arbitrary contours [38], we present a scene text detection network (called
FANet) with a Fast convergence speed and Accurate text localization, where the
transformer is combined with Fourier descriptor to localize arbitrary-shaped
text regions. The DETR based metheds [1] can scale-invariantly localize the
rectangular target regions. The Fourier descriptor proposed by [38] can model
target regions of arbitrary contours, but it cannot be embedded into the DETR
detection framework. Thus, we propose a normalization method for Fourier
descriptor to enable the DETR based methods to predict the normalized text
regions of arbitrary shape. However, we find that the established framework
based on the proposed normalization method has slow convergence and low
accuracy due to two aspects of reasons, i.e., (1) the change of regression
target makes the deformable DETR component become sub-optimal and (2) the
Hungarian Matching Strategy hinders the rapid convergence of the network.
Correspondingly, we first make some effective changes to the structure of
deformable DETR. Concretely, we propose a Fourier Descriptor Proposal Network
(FDPN) to get better candidates for the text decoder. Then, we build an
Iterative Text Decoding Network (ITDN) to iteratively refine Fourier
proposals. Finally, we propose a well-designed loss function to optimize the
descriptor representations and calculate the matching cost. Additionally, we
propose a Dense Matching Strategy (DMS) to greatly speed up the convergence
and improve the detection accuracy within fewer training epochs. As shown in
Figure 1, the proposed FANet can obtain an F-measure of $83.3\%$ and $84.8\%$
respectively, after training only 100 epochs on datasets CTW1500 [16] and
MSRA-TD500 [33] without pre-training, which outperforms current best SOTA
methods [35, 26] by $6.5\%$ $(83.3\%vs.76.8\%)$ and $16.5\%$
$(84.8\%vs.68.3\%)$ respectively. The main contributions of this paper are
summarized as follows.

  * •

We present a scene text detection network FANet with fast convergence speed
and accurate localization, which uses the transformer to learn text features,
and the normalized Fourier descriptor to represent text regions. The proposed
FANet has achieved SOTA performances on multiple public benchmarks,$e.g.$,
MSRA-TD500,CTW1500 and TotalText.

  * •

We make many changes to the original deformable DETR, including Fourier
Descriptor Proposal Network (FDPN), Iterative Text Decoding Network (ITDN) and
well-designed loss function. These components can effectvely improve the
convergence speed and accuracy of FANet.

  * •

We propose a Dense Matching Strategy (DMS), which significantly improves the
convergence speed and accuracy of FANet within fewer training epochs.

##  2 Related Works

###  2.1 Transformer modeling

The transformer [22] with both self-attention and cross-attention mechanism
has achieved great success in both machine translation [6] and visual
recognition [7]. For example, DETR [1] first adopts the transformer
architecture for the object detection task. deformable DETR [37] extends DETR
with a deformable attention module that reduces the training time
significantly. Some previous works also try to explore the potential of
transformer on text spotting task. Wu _et al._ [27, 32] proposed to track and
spot text in video with transformer sequence modeling. [18] adopts the
transformer architecture in multi-orientation text detection for the first
time, but it still suffers from the problems of requiring massive data for
pre-training, slow convergence, poor performance and inadequate representation
ability. By using better contour representation, feature extraction and
network optimization methods, we make the proposed FANet based on transformer
surpasses the SOTA text detection algorithm based on CNNs [38, 4, 26].

###  2.2 Text Region Representation

Text regions can be modeled via per-pixel masks [25, 26], or modeled by
parameters in specified representation spaces. For example, TextRay [24]
represents the text contours in the polar system. ABCNet [15] introduced
Bezier curves to parameterize curved texts. FCENet [38] represents the text
instances in the Fourier domain, which allows to represent any closed
continuous contour in robust and simple manners. In this paper, we further
present a new normalized Fourier descriptor to represent the normalized text
regions of arbitrary shapes, which makes it possible to embed the text
representation based on Fourier descriptor into the detection framework based
on transformers.

##  3 Method Description

###  3.1 Overview

As shown in Figure 2, the proposed FANet mainly consists of three parts:
Feature Extraction Network (FEN), Fourier Descriptor Proposal Network (FDPN)
and Iterative Text Decoding Network (ITDN). For a given image, it is first
encoded as features by the FEN, which consists of a backbone and a transformer
encoder. The encoded features are then fed into the FDPN to obtain a set of
arbitrary-shaped text contours represented as normalized Fourier descriptor
and the object queries transformed from them. Further, the encoded features,
the object queries and the $N_{q}$ Fourier proposals with the highest scores
are jointly fed into the ITDN to obtain the refined Fourier proposals, where
$N_{q}$ is the number of queries of the transformer decoder (300 in our
experiments). Finally, we get the detection results after applying Inverse
Fourier transform (IFT), Inverse Normalization (IN) and Non-Maximum
Suppression (NMS) to the refined Fourier proposals.

###  3.2 Fourier Descriptor Normalization

Figure 3: Description of the Fourier descriptor normalization.

Target generation. FCENet uses a complex-value function $z(t)=x(t)+iy(t)$,
$t\in[0,1]$ to represent a text contour, where $(x(t),y(t))$ denotes the
spatial coordinate of the text contour. To make the contour representation
scale-invariant, we present a normalized form
$\mathcal{Z}(t)=\frac{x(t)}{W}+i\frac{y(t)}{H}=\mathcal{X}(t)+i\mathcal{Y}(t)$,
where $H,W$ are the height and width of a given image. Since we cannot obtain
the analytical form of text contour function $\mathcal{Z}$ in real scenario,
we can discretize the continual function $\mathcal{Z}$ into $N$ points, i.e.,
$\\{\mathcal{Z}(\frac{n}{N})\\}$, $n\in\\{0,...,N-1\\}$. Correspondingly, we
can calculate the Fourier descriptor of a normalized text contour by Discrete
Fourier Transform (DFT), $i.e.$,

$\displaystyle\hat{c}_{k}$
$\displaystyle=\frac{1}{N}\sum_{n=0}^{N-1}\mathcal{Z}(\frac{n}{N})e^{-2\pi
ik\frac{n}{N}},k\in\Psi,$ (1)

where $\hat{c}_{k}=\hat{u}_{k}+i\hat{v}_{k}\in{\mathbb{C}}$, $\hat{u}_{k}$ and
$\hat{v}_{k}$ are real part and imaginary part of $\hat{c}_{k}$,
$\Psi\triangleq\\{-K,\cdots,0,\cdots,K\\}$, $K$ is the highest frequency index
reserved for the Fourier descriptor of the text contour ($K$=5 in our
experiments). Thus, each ground-truth text contour can be represented as a
normalized Fourier descriptor,
$\hat{{\bm{c}}}=[\hat{u}_{-K},\hat{v}_{-K},\cdots,\hat{u}_{0},\hat{v}_{0},\cdots,\hat{u}_{K},\hat{v}_{K}]\in
R^{4K+2}$. We take normalized Fourier descriptor as regression targets. As
shown in Figure 3, the values of original Fourier descriptor are generally
large and unbounded, while the values of normalized Fourier descriptor all
fall into a small region with fixed boundaries, $i.e.$,

$\displaystyle(\hat{u}_{0},\hat{v}_{0})$ $\displaystyle\in[0,1]^{2},$ (2)
$\displaystyle(\hat{u}_{k},\hat{v}_{k})$
$\displaystyle\in[-\frac{2}{\pi},\frac{2}{\pi}]^{2},k\in\Psi,k\neq 0.$ (3)

New activation function. Considering the above properties of normalized
Fourier descriptor, the commonly used activation functions cannot meet our
requirements. For example, the identity function is an unbounded function that
does not fit the bounded regression target well. The sigmoid function $\sigma$
cannot output negative values. The range of $\tanh$ is much larger than our
regression target, which results in too many invalid predictions. So we define
a new activation function $f(\widetilde{{\bm{c}}})$ suitable for the
regression of proposed normalized Fourier descriptor as

$\displaystyle{\bm{c}}_{i}=f(\widetilde{{\bm{c}}}_{i})=\left\\{\begin{array}[]{cc}\sigma(\widetilde{{\bm{c}}}_{i})&i=2K,2K+1,\\\
\frac{\tanh(\widetilde{{\bm{c}}}_{i})}{\Delta}&i\in\Omega/\\{2K,2K+1\\},\\\
\end{array}\right.$ (6)

where $\widetilde{{\bm{c}}}\in{\mathbb{R}}^{4K+2}$ is the stimulus, and
${\bm{c}}\in{\mathbb{R}}^{4K+2}$ is the prediction of regression branches,
where $\Omega=\\{0,\cdots,4K+1\\}$ is the index set aligned with $\Psi$,
$\Delta$ is a hyperparameter that controls the output range of function
$f(\widetilde{{\bm{c}}})$ and we set it to $\frac{\pi}{2}$ to best match the
range of our regression target. As shown in Figure 3, we use the proposed
activation function for the unbounded network logits of the regression
branches where we need to predict the normalized Fourier descriptor by the
networks, such as FDPN and ITDN.

###  3.3 Fourier Descriptor Proposal Network

We build a lightweight Fourier Descriptor Proposal Network (FDPN) following
the Feature Extraction Network. As shown in Figure 2, through a single
convolution module, we predict two feature maps, $i.e.$,
$F_{score}\in{\mathbb{R}}^{B\times 2\times h_{f}\times
w_{f}},F_{reg}\in{\mathbb{R}}^{B\times(4K+2)\times h_{f}\times w_{f}}$, where
$B$ is the batch size, $h_{f}$ and $w_{f}$ are the height and width of the
feature map. Each pixel on the feature map is assigned as an object query,
which directly predicts a normalized Fourier descriptor and a corresponding
score, i.e., a Fourier proposal. The top $N_{q}$ Fourier proposals with the
highest scores are selected, and we convert them into the initial queries of
the ITDN through a transformation layer, which consists of a linear layer and
a position encoding layer as described in deformable DETR [37]. Finally, the
proposals and the queries are sent to the ITDN, and $N_{q}$ is also the number
of queries of the transformer decoder.

###  3.4 Iterative Text Decoding Network

As shown in Figure 2, Iterative Text Decoding Network (ITDN) is composed of
$D$ duplicate modules, each module is consist of a deformable transformer
decoder layer and a refinement layer. We let each module refine the Fourier
descriptor based on the prediction from the previous module and we select the
output of the last module as the final predictions. The refinement target of
FANet is the normalized Fourier descriptor while the refinement target of
deformable DETR [37] is normalized bounding boxes. To adapt to this change of
regression target, we make some effective changes for the transformer decoder
layer as follows:
(1) Take the Fourier descriptor as the reference location for the Multi-Scale
Deformable Attention Module. As shown in Figure 2, in the calculation process
of $d$-th transformer decoder layer, we first calculate the bounding box
${\bm{b}}^{d-1}=(x,y,w,h)\in[0,1]^{4}$ of the Fourier proposal
${\bm{c}}^{d-1}$ through a function $b$. Then we use $((p_{x}+x)w,(p_{y}+y)h)$
instead of $(p_{x},p_{y})$ as the sampling location for the Multi-Scale
Deformable Attention (MSDeformAttn) module [37], where $(p_{x},p_{y})$ are the
sampling offset predicted by the MSDeformAttn module. Such modifications make
the sampled locations of the MSDeformAttn module related to the center and
size of previously predicted text contour.
(2) Use multiplication based refinement instead of the original addition based
refinement for the nondc component of Fourier descriptor. As shown in Figure
2, the refinement function maps a proposal
${\bm{c}}^{d-1}\in{\mathbb{R}}^{4K+2}$ and an offset prediction
${\bm{o}}^{d}\in{\mathbb{R}}^{4K+2}$ to a new proposal
${\bm{c}}^{d}\in{\mathbb{R}}^{4K+2}$, which can be formulated as
${\bm{c}}^{d}_{i}=g({\bm{c}}^{d-1}_{i},{\bm{o}}^{d}_{i}),i\in\Omega$, where
${\bm{c}}^{d-1}$ and ${\bm{c}}^{d}$ denote the prediction of the regression
branch of $(d-1)$-th and $d$-th transformer decoder layer (${\bm{c}}_{0}$
denotes the proposal output by FDPN), ${\bm{o}}^{d}$ is the offset prediction
of the $d$-th transformer decoder layer. refinement function used by
deformable DETR is as follows:

$\displaystyle{\bm{c}}^{d}_{i}$
$\displaystyle=g_{1}({\bm{c}}^{d-1}_{i},{\bm{o}}^{d}_{i})=f(f^{-1}({\bm{c}}^{d-1}_{i})+{\bm{o}}^{d}_{i}),i\in\Omega,$
(7)

where $f$ and $f^{-1}$ are the activation function and its reverse function.
We propose a new multiplication based refinement function for the nondc
component of the normalized Fourier descriptor as

$\displaystyle{\bm{c}}^{d}_{i}$
$\displaystyle=g_{2}({\bm{c}}^{d-1}_{i},{\bm{o}}^{d}_{i})=f(f^{-1}({\bm{c}}^{d-1}_{i})e^{{\bm{o}}^{d}_{i}}),$
$\displaystyle i\in\Omega/\\{2K,2K+1\\}.$ (8)

The reason why we use such refinement function are summarized. If we assume
that ${\bm{o}}^{d}$ close to $\vec{\textbf{0}}$, which is reasonable because
${\bm{o}}^{d}$ is the residual term of the prediction. Based on this
assumption, we can prove that for the refinement function based on addition,
we have

$\displaystyle\lim_{{\bm{o}}^{d}_{i}\rightarrow 0}\left|\frac{\partial
g_{1}}{\partial{\bm{o}}^{d}_{i}}\right|$ $\displaystyle=\left|\frac{\partial
f(z)}{\partial z}|_{z=f^{-1}({\bm{c}}^{d-1}_{i})}\right|,i\in\Omega,$ (9)

and for the refinement function based on multiplication, we have

$\displaystyle\lim_{{\bm{o}}^{d}_{i}\rightarrow 0}\left|\frac{\partial
g_{2}}{\partial{\bm{o}}^{d}_{i}}\right|$ $\displaystyle=\left|\frac{\partial
f(z)}{\partial
z}|_{z=f^{-1}({\bm{c}}^{d-1}_{i})}\right|\left|f^{-1}({\bm{c}}^{d-1}_{i})\right|,$
$\displaystyle i\in\Omega/\\{2K,2K+1\\}.$ (10)

As we can see, when calculating the gradient of ${\bm{o}}^{d}_{i}$, our
multiplication based refinement function has an extra
$\left|f^{-1}({\bm{c}}^{d-1}_{i})\right|$ term compared with the original
definition. It is easy to prove that the function $\left|f^{-1}(x)\right|$ is
a monotonically increasing function of $|x|$ when $f$ is $\sigma$ or $\tanh$.
We can deduce that $\left|f^{-1}({\bm{c}}^{d-1}_{i})\right|$ term can
adaptively increase the gradient of the Fourier descriptor if
$|{\bm{c}}^{d-1}_{i}|$ becomes larger. This weighting scheme is intuitive and
effective, because the numerically larger Fourier descriptor generally plays a
leading role in the construction of the target contour, which means they
should occupy a larger gradient in back propagation than others. As a special
case, we still use the refinement function based on addition for the dc
component $(u_{0},v_{0})$ of Fourier descriptor. $(u_{0},v_{0})$ is the center
coordinate of the text contour and the optimization of the center coordinate
of different targets should not be weighted by the absolute value of their
position. In the ITDN, the refinement function is defined as

$\displaystyle
g({\bm{c}}^{d}_{i})=\left\\{\begin{array}[]{cc}g_{1}({\bm{c}}^{d-1}_{i},{\bm{o}}^{d}_{i})&i=2K,2K+1,\\\
g_{2}({\bm{c}}^{d-1}_{i},{\bm{o}}^{d}_{i})&i\in\Omega/\\{2K,2K+1\\}.\\\
\end{array}\right.$ (13)

###  3.5 Optimization strategies

Dense Matching Strategy. The traditional Hungarian Matching Strategy (HMS)
used in [21, 1, 37] only matches one query for each ground-truth text
instance, which we find is one of the causes of slow convergence of the
network. The number of text instances in an image is usually limited,and we
denote it as $N_{g}$. $N_{g}$ is usually much less than the number of queries
of the network $N_{q}$ (300 in our experiments), i.e., $N_{g}\ll N_{q}$. Using
HMS means that only $N_{g}$ queries are used as positive samples for the
training of regression branch in each iteration, which makes the network easy
to overfit and damages the performance of the network due to lack of positive
samples. We propose to match multiple queries for each ground-truth text
instance. Dense Matching Strategy (DMS) can effectively alleviate the problem
of slow convergence, especially in the early stage of training. Since the DMS
makes the FANet output overlapping detection results, we have to use the NMS
as post-processing. Due to the small number of queries of DETR based methods
and small number of $N_{m}$, the number of predictions is generally small, so
the NMS only brings a low cost. The pseudo code of Dense Matching Strategy is
summarized:

Algorithm 1 Dense Matching Strategy

Input:
$\hat{\mathcal{A}}$ : a set of ground-truth text contours in a given image.
$\mathcal{A}$ : a set of predict text contours in the image.
Parameter:
Cost : the cost function of the proposed FANet.
$N_{m}$ : the number of predicted text contour matched for each ground truth
text contour.
Output:
$\mathcal{P}$ : a set of positive samples.
$\mathcal{N}$ : a set of negtive samples.

1: $\mathcal{P}\leftarrow\varnothing$

2: $\hat{a}=|\hat{\mathcal{A}}|$

3: $a=|\mathcal{A}|$

4: $i=1$

5:
$cost=\text{Cost}(\mathcal{A},\hat{\mathcal{A}})\in{\mathbb{R}}^{a\times\hat{a}}$

6: while $i\leq N_{m}$ do

7: $k,l=\text{Hungarian\\_matcher}(cost)$

8: $\mathcal{P}\leftarrow\mathcal{A}[k]$

9: $cost[k,:]=+\infty$

10: $i=i+1$

11: end while

12: $\mathcal{N}=\mathcal{A}-\mathcal{P}$

13: return $\mathcal{P},\mathcal{N}$

To use the DMS algorithm, we need to define a Cost function that takes the set
of predicted text contours $\mathcal{A}$ and the set of ground-truth text
contours $\hat{\mathcal{A}}$ as input and outputs the cost between any two
elements in the two sets. We use $\mathcal{C}_{cls}+\lambda\mathcal{C}_{reg}$
as the matching cost for any $({\bm{c}},\hat{{\bm{c}}})$ pair in the two sets,
where $\mathcal{C}_{cls}=-log(s)$, where $s$ is the predicted text confidence
score corresponding to proposal ${\bm{c}}$, $\lambda$ is the hyperparameter
that balance the costs. We define $\mathcal{C}_{reg}$ as follows:

$\displaystyle\mathcal{C}_{reg}({\bm{c}},\hat{{\bm{c}}})$
$\displaystyle=\mathcal{L}_{SD}({\bm{c}},\hat{{\bm{c}}})+\alpha_{1}\mathcal{L}_{FD}({\bm{c}},\hat{{\bm{c}}})+\alpha_{2}\mathcal{L}_{bbox}({\bm{c}},\hat{{\bm{c}}}),$
(14) $\displaystyle\mathcal{L}_{SD}({\bm{c}},\hat{{\bm{c}}})$
$\displaystyle=\text{L1}\big{[}\mathcal{F}^{-1}({\bm{c}}),\mathcal{F}^{-1}(\hat{{\bm{c}}})\big{]},$
(15) $\displaystyle\mathcal{L}_{FD}({\bm{c}},\hat{{\bm{c}}})$
$\displaystyle=\text{L1}({\bm{c}},\hat{{\bm{c}}}),$ (16)
$\displaystyle\mathcal{L}_{bbox}({\bm{c}},\hat{{\bm{c}}})$
$\displaystyle=\text{GIOU}\big{[}b({\bm{c}}),b(\hat{{\bm{c}}})\big{]}.$ (17)

Where $\alpha_{1},\alpha_{2}$ are the hyperparameters that balance the three
matching costs. L1 is the L1 loss function, $b$ is a function that convert
Fourier proposal to bounding box, GIOU is the GIOU loss in [19],
$\mathcal{F}^{-1}$ is the Inverse Discrete Fourier Transform. For a given
image, we first calculate the $cost$ between $\mathcal{A}$ and
$\hat{\mathcal{A}}$, then we iteratively perform Hungarian Matching (HM) for
$N_{m}$ times. Each time we first add the proposals that match any ground-
truth text contour to the positive sample set $\mathcal{P}$ and then remove
these proposals from the matching queue by setting the matched rows of $cost$
to $+\infty$ to avoid repeated matching. Finally, we collect all the matched
proposals as positive samples and the unmatched proposals as negative samples.

Loss function. The loss function of the proposed network is given by

$\displaystyle\mathcal{L}$
$\displaystyle=\mathcal{L}_{cls,0}+\lambda\mathcal{L}_{reg,0}+\sum_{i=1}^{D}w_{i}(\mathcal{L}_{cls,i}+\lambda\mathcal{L}_{reg,i}),$
(18)

where $\mathcal{L}_{cls,0}$ and $\mathcal{L}_{reg,0}$ denote the
classification and regression loss of the FDPN respectively,
$\mathcal{L}_{cls,i}$ and $\mathcal{L}_{reg,i}$ denote the classification and
regression loss of the $i$-th decoder layer, $w_{i}$ is the hyperparameter
that balance the losses of different transformer decoder layers, $D$ is the
number of decoder layers. We use focal loss [13] as our default classification
loss for $\mathcal{L}_{cls,i},i=0,1,\cdots,D$. We define $\mathcal{L}_{reg,i}$
as follows:

$\displaystyle\mathcal{L}_{reg,i}$
$\displaystyle=\frac{1}{|\mathcal{M}_{i}|}\sum_{({\bm{c}},\hat{{\bm{c}}})\in\mathcal{M}_{i}}(\mathcal{L}_{SD}+\alpha_{1}\mathcal{L}_{FD}+\alpha_{2}\mathcal{L}_{bbox}),$
(19)

where $\mathcal{M}_{i},i=0,1,\cdots,d$ are the set of matched Fourier
descriptor pairs of $i$-th decoder layer, $i.e.$,
$\forall({\bm{c}},\hat{{\bm{c}}})\in\mathcal{M}_{i},{\bm{c}}\in\mathcal{P}_{i}$,
where $\mathcal{P}_{i}$ is the positive sample set obtained by DMS at the
$i$-th decoding layer.

Table 1: Comparison with recent state-of-the-art methods on ICDAR 2015, MSRA-TD500, CTW1500 and TotalText under the protocol of<EMAIL_ADDRESS>where ’Ext.’ denotes extra training data. We use ResNet50 as the default backbone for our proposed FANet and all the compared algorithms. Methods |  Venue |  Backbone |  Ext. |  ICDAR2015 |  MSRA-TD500 |  CTW1500 |  TotalText
---|---|---|---|---|---|---|---
R(%) | P(%) | F(%) | R(%) | P(%) | F(%) | R(%) | P(%) | F(%) | R(%) | P(%) | F(%)
TextSnake [17] | ECCV’18 |  VGG16 [20] | ✓ | 80.4 | 84.9 | 82.6 | 73.9 | 83.2 | 78.3 | 85.3 | 67.9 | 75.6 | 74.5 | 82.7 | 78.4
PAN [26] | ICCV’19 |  | ✓ | 81.9 | 84.0 | 82.9 | 83.8 | 84.4 | 84.1 | 81.2 | 86.4 | 83.7 | 81.0 | 89.3 | 85.0
DB [12] | AAAI’20 |  Res50-DCN [3] | ✓ | 83.2 | 91.8 | 87.3 | 79.2 | 91.5 | 84.9 | 80.2 | 86.9 | 83.4 | 82.5 | 87.1 | 84.7
CRNet [36] | MM’20 |  | ✓ | 84.5 | 88.3 | 86.4 | 82.0 | 86.0 | 84.0 | 80.9 | 87.0 | 83.8 | 82.5 | 85.8 | 84.1
[18] | CVPRW’21 |  | ✓ | 78.3 | 89.8 | 83.7 | 83.8 | 90.9 | 87.2 | - | - | - | - | - | -
DRRG [35] | CVPR’20 |  | ✓ | 84.7 | 88.5 | 86.6 | 82.3 | 88.1 | 85.1 | 83.0 | 85.9 | 84.5 | 84.9 | 86.5 | 85.7
PCR [4] | CVPR’21 |  DLA34 [4] | ✓ | - | - | - | 83.5 | 90.8 | 87.0 | 82.3 | 87.2 | 84.7 | 82.0 | 88.5 | 85.2
PSENet [25] | CVPR’19 |  |  | 79.7 | 81.5 | 80.6 | - | - | - | 75.6 | 80.6 | 78.0 | 75.1 | 81.8 | 78.3
PAN [26] | ICCV’19 |  |  | 77.8 | 82.9 | 80.3 | 77.3 | 80.7 | 78.9 | 77.7 | 84.6 | 81.0 | 79.4 | 88.0 | 83.5
TextRay [24] | MM’20 |  |  | - | - | - | - | - | - | 80.4 | 82.8 | 81.6 | 77.9 | 83.5 | 80.6
PCR [4] | CVPR’21 |  |  | - | - | - | 77.8 | 87.6 | 82.4 | 79.8 | 85.3 | 82.4 | 80.2 | 86.1 | 83.1
FCENet [38] | CVPR’21 |  |  | 84.2 | 85.1 | 84.6 | - | - | - | 80.7 | 85.7 | 83.1 | 79.8 | 87.4 | 83.4
FANet |  |  |  | 83.8 | 85.6 | 84.7 | 83.3 | 91.7 | 87.3 | 84.3 | 85.6 | 84.9 | 83.3 | 86.2 | 84.8
FANet |  |  | ✓ | 87.6 | 85.0 | 86.3 | 84.2 | 92.1 | 88.0 | 84.0 | 87.8 | 85.9 | 84.7 | 87.1 | 85.9
Table 2: Convergence performance comparison with recent state-of-the-art methods. We use F-measure as the evaluation protocol, and $\\{30,100,500\\}$e means training $\\{30,100,500\\}$ epochs on the dataset. The results of other algorithms are reproduced in open-mmocr [11]. $\dagger$ denote using original Hungarian Matching Strategy (HMS). Methods |  ICDAR2015 |  MSRA-TD500 |  CTW1500 |  TotalText
---|---|---|---|---
30e | 100e | 500e | 30e | 100e | 500e | 30e | 100e | 500e | 30e | 100e | 500e
PAN [26] | 62.3 | 73.8 | 74.9 | 61.4 | 65.1 | 70.0 | 72.0 | 73.4 | 74.6 | 66.9 | 73.7 | 76.0
FCENet [38] | 65.6 | 78.5 | 83.5 | 47.2 | 62.2 | 74.2 | 65.5 | 75.6 | 81.0 | 71.3 | 80.2 | 83.0
FANet$\dagger$ | 73.9 | 82.6 | 83.5 | 62.1 | 70.6 | 83.5 | 75.6 | 81.2 | 84.6 | 79.0 | 81.2 | 84.6
FANet | 77.2 | 84.1 | 84.7 | 70.7 | 84.8 | 87.3 | 79.1 | 83.3 | 84.9 | 81.9 | 84.1 | 84.8

##  4 Experiment

###  4.1 Implementation details

The backbone of FANet is ResNet-50 [9] which is pre-trained on ImageNet [5].
Following FCENet [38], during the target generation stage, we sample
equidistantly a fixed number $N$ ($N$ = 400 in our experiments) points on the
text contour, obtaining the resampled point sequence, i.e.,
$\\{\mathcal{Z}(\frac{n}{N})\\}$, $n\in\\{0,...,N-1\\}$. Then, we transform
the resampled point sequence into its corresponding normalized Fourier
descriptor with Equation 1. We set the highest level reserved for the Fourier
descriptor $K$ to 5. The number of queries of the transformer decoder $N_{q}$
to 300. The number of transformer layers $D$ to 6. For the hyperparameters
that balance the losses, we set $[\lambda,\alpha_{1},\alpha_{2}]$ equals to
$[0.25,5,0.4]$ and set $w_{d}$ to $1$, $d=1,2,\cdots,D$.

Optimization Setting. The experiments are conducted on the workstation with 8
Tesla V100 GPU. The data augmentation contains color jitter, random rotation,
random horizontal flip, random scale, random crop, and random resize.
Following the common practices, we ignore the blurred text regions labeled as
“DO NOT CARE” during training. The network is trained using an Adam optimizer
with the weight decay ratio equals to 0.0001 and we fix the batch size to 16.
We use poly as our default learning rate policy, and we set its power as 0.9.
For the experiments without pre-training, we set the match number of the DMS
$N_{m}$ to 3 by default and MSRA-TD500 to 10 and train the proposed FANet for
500 epochs separately on the dataset we report the results. For the experiment
with pre-training, we first pre-trained our model for 25 epochs on COCOTextv2
[23], and then finetune our model for 250 epochs on the benchmark datasets
respectively. we use HMS as default and DMS $N_{m}$ to 5 for MSRA-TD500 by
default when FANet is pretrained use COCOTextv2. For the convergence
experiments in Table 2, as the learning rate attenuation strategy will damage
the performance of the algorithm under the restriction of very little
iterations, we keep the initial learning rate unchanged for the experiments of
training 30 epochs. For the experiments of training more than 30 epochs, we
keep the optimization settings of the re-implemented algorithms consistent
with the original paper.

Inference setting. In the inference stage, we resize the long dimension of
test images to 1080, 1280, 2200 and 1312 for MSRA-TD500, CTW1500, ICDAR2015
and TotalText respectively.

###  4.2 Comparison with the state-of-the-art methods

We evaluate the proposed FANet on four public benchmark datasets, $i.e.$,
ICDAR2015 [10], MSRA-TD500 [33], CTW1500 [16] and TotalText [2], and the
experiments results are summarized in Table 1.

ICDAR2015. The proposed FANet achieves comparable performance with SOTA
methods on ICDAR2015. It’s worth noting that the proposed FANet outperforms
the previous DETR based text detection algorithm [18] by 2.6% (86.3% $vs.$
83.7%) based on F-measure with pre-training.

MSRA-TD500. The proposed FANet achieves the SOTA performance on MSRA-TD500.
Specifically, if compared with the methods without pre-training, the proposed
FANet outperforms the previous SOTA method PCR by 4.9% (87.3% $vs.$ 82.4%) in
F-measure. If compared with the algorithm with pre-training, the proposed
FANet can still outperforms the previous SOTA method [18] by 0.8% (88.0% $vs.$
87.2%).

CTW1500. The proposed FANet achieves the SOTA performance on CTW1500. if
compared with the methods without pre-training, the proposed FANet outperforms
the previous SOTA method FCENet by 1.8% (84.9% $vs.$ 83.1%) in F-measure. If
compared with the algorithm with pre-training, the proposed FANet can still
outperforms the previous SOTA method [4] by 1.2% (85.9% $vs.$ 84.7%).

TotalText. The proposed FANet achieves the SOTA performance on TotalText. If
compared with the methods without pre-training, the proposed FANet outperforms
the previous SOTA method PAN by 1.3% (84.8% $vs.$ 83.5%) in F-measure. If
compared with the algorithm with pre-training, the proposed FANet can still
outperforms the previous SOTA method [35] by 0.2% (85.9% $vs.$ 85.7%).

Convergence Performance. Under the constraint of limited epochs, the proposed
FANet can still achieve good results. In particular, with only 30 epochs of
training, FANet surpass FCENet by 11.6%, 23.5%, 13.6%, 10.6%, surpass PAN by
14.9%, 9.3%, 7.1%, 15.0% based on F-measure on ICDAR2015, MSRA-TD500, CTW1500
and TotalText respectively, which shows that the proposed FANet can achieve
much better performance than the previous SOTA methods with fewer epochs. With
only 100 epochs of training, FANet can achieve the detection performance of
84.8% on MSRA-TD500, 83.3% on CTW1500 and 84.1% on TotalText, which already
have achieved the SOTA performance if only compared with the methods without
pre-training. In other words, we only need to train FANet on a server with 8
Tesla V100 for about 20 minutes, and we can get a SOTA text detector that
achieve 84.8% F-measure on MSRA-TD500.

Qualitative Results. Some detection results of the proposed FANet are shown in
Figure 4. For the datasets whose text instances are all rectangular boxes, we
take the minimum enclosing rectangle for the text contour. As we can see,
FANet is capable of detecting text in a variety of scenarios, including
extremely long text in MSRA-TD500, scene text in ICDAR2015, curved text in
CTW1500 and TotalText. Qualitative and quantitative results show that the
proposed FANet has a wide range of application scenarios.

Figure 4: Some detection results of our proposed method on MSRA-TD500,
ICDAR2015, CTW1500 and TotalText.

###  4.3 Ablation study

We conduct ablation studies on CTW1500 and MSRA-TD500 under the protocol of
<EMAIL_ADDRESS>We fix the number of training epochs for all experiments to 500
epochs.

Table 3: Ablation experiments on our proposed core components. ITDN |  FDPN |  DMS |  CTW1500 |  MSRA-TD500
---|---|---|---|---
R(%) | P(%) | F(%) | R(%) | P(%) | F(%)
|  |  | 78.9 | 82.0 | 80.4 | 19.1 | 9.2 | 12.4
✓ |  |  | 79.2 | 87.4 | 83.1 | 58.8 | 67.7 | 62.9
✓ | ✓ |  | 82.5 | 86.9 | 84.6 | 79.2 | 88.3 | 83.5
✓ | ✓ | ✓ | 84.3 | 85.6 | 84.9 | 83.3 | 91.7 | 87.3

Baseline and core components. Based on deformable DETR, we directly use
normalized Fourier descriptor to replace bounding box as the regression target
to obtain our baseline algorithm. As shown in Table 3, the baseline of the
proposed method reaches 80.4% on CTW1500 and 12.4% on MSRA-TD500. Compared
with our baseline, ITDN can bring relative improvements of 2.7% (83.1% $vs.$
80.4%) and 50.5% (62.9% $vs.$ 12.4%) on CTW1500 and MSRA-TD500 respectively.
Then, the addition of FDPN can bring relative improvements of 1.5% (84.6%
$vs.$ 83.1%) and 20.6% (83.5% $vs.$ 62.9%) on CTW1500 and MSRA-TD500
respectively. Finally, the DMS can bring relative improvements of 0.3% (84.9%
$vs.$ 84.6%) and 3.8% (87.3% $vs.$ 83.5%) on CTW1500 and MSRA-TD500
respectively. The above modules and strategies can significantly improve the
performance of the network, especially for MSRA-TD500 which only has small
number of training data.

Figure 5: Comparison between normalized Fourier Descriptor regression target and original Fourier Descriptor regression target when DETR like detection architecture is used. (a) denotes using normalized Fourier Descriptor as regression target and our activation function, (b) denotes using original Fourier Descriptor as regression target and identity activation function like FCENet. Table 4: Comparison between our activation function and other commonly used activation functions. AF denotes Activation Function and NT denotes whether to use the Normalized Fourier Descriptor as the regression target. ”—” denotes that the network becomes untrainable and no result can be obtained. AF | NT |  CTW1500 |  MSRA-TD500
---|---|---|---
R(%) | P(%) | F(%) | R(%) | P(%) | F(%)
identity |  | 0.0 | 0.0 | 0.0 | 0.0 | 0.0 | 0.0
identity | ✓ | — | — | — | — | — | —
$\sigma$ | ✓ | 0.0 | 0.0 | 0.0 | 1.6 | 1.9 | 1.7
$\tanh$ | ✓ | 82.9 | 85.7 | 84.3 | 75.8 | 86.6 | 80.8
ours | ✓ | 82.5 | 86.9 | 84.6 | 79.2 | 88.3 | 83.5

Activation function. As shown in Figure 5, we first compare the performance
between the normalized Fourier descriptor and the original Fourier descriptor
when DETR like architecture is used. The network does not locate text well
when the original Fourier descriptor is used which means that the original
Fourier descriptor do not fit with DETR like detection architecture. As shown
in line 2 of Table 4, the network will not be trainable if the activation
function in the proposed FANet is replaced by identity function. Then we
replace the activation function in the proposed FANet with $\sigma$ in line 3
of Table 4, which makes the detection performance decrease to a very low
level, since the range of $\sigma$ can not completely cover the normalized
Fourier descriptor to be predicted. Finally, we test the performance when
replacing the activation function by $\tanh$ in line 4 of Table 4, which makes
the corresponding F-measure of the network decrease by 0.3% (84.6% $vs.$
84.3%) and 2.7% (83.5% $vs.$ 80.8%) on CTW1500 and MSRA-TD500 respectively.
The experimental results show that it is more appropriate to use the
activation function which can well match the range of regression target.

Table 5: The impact of the match number of the Dense matching Strategy for the performance of the network under the restriction of training 500 epochs.  Match number |  MSRA-TD500 |  ICDAR2015 |  CTW1500 |  TotalText
---|---|---|---|---
R(%) | P(%) | F(%) | R(%) | P(%) | F(%) | R(%) | P(%) | F(%) | R(%) | P(%) | F(%)
1 | 79.2 | 88.3 | 83.5 | 79.9 | 87.4 | 83.5 | 82.5 | 86.9 | 84.6 | 82.4 | 87.0 | 84.6
3 | - | - | - | 83.8 | 85.6 | 84.7 | 84.3 | 85.6 | 84.9 | 83.3 | 86.2 | 84.8
5 | 83.2 | 88.8 | 85.9 | - | - | - | - | - | - | 81.5 | 87.2 | 84.3
10 | 83.3 | 91.7 | 87.3 | 78.1 | 85.6 | 81.7 | 80.3 | 88.7 | 84.3 | 73.1 | 84.9 | 78.6
15 | 82.3 | 90.2 | 86.1 | - | - | - | - | - | - | - | - | -
Table 6: Ablation of our proposed Iterative Text Decoding Network. ”add” and ”mul” indicate the cases where our refinement function is replaced by add and multiplication based refinement function respectively. ”w/o reference” indicates that Fourier descriptor is not used as the reference location for the Multi-Scale Deformable Attention Module. ”w/o refinement” indicates the case where the refinement module is removed.  Refinement module |  CTW1500 |  MSRA-TD500
---|---|---
R(%) | P(%) | F(%) | R(%) | P(%) | F(%)
w/o refinement | 78.9 | 82.0 | 80.4 | 19.1 | 9.2 | 12.4
w/o reference | 76.4 | 79.9 | 78.1 | 40.2 | 43.3 | 41.7
add | 77.7 | 85.0 | 81.2 | 36.1 | 47.3 | 40.9
mul | 78.8 | 85.4 | 82.0 | 50.3 | 65.7 | 57.0
ours | 79.2 | 87.4 | 83.1 | 58.8 | 67.7 | 62.9
Table 7: Ablation of the loss functions.  $\mathcal{L}_{SD}$ |  $\mathcal{L}_{FD}$ |  $\mathcal{L}_{bbox}$ |  CTW1500 |  MSRA-TD500
---|---|---|---|---
R(%) | P(%) | F(%) | R(%) | P(%) | F(%)
| ✓ | ✓ | 75.3 | 84.4 | 79.6 | 56.7 | 59.9 | 58.3
✓ | ✓ |  | 82.6 | 86.5 | 84.5 | 82.0 | 83.5 | 82.7
✓ |  | ✓ | 81.8 | 86.2 | 84.0 | 81.6 | 84.2 | 82.9
✓ | ✓ | ✓ | 82.5 | 86.9 | 84.6 | 79.2 | 88.3 | 83.5

Iterative Text Decoding Network. As shown in Table 6, replacing the refinement
function by the refinement function based on addition or multiplication can
result in the decrease of F-measure for the proposed FANet by 1.9% and 1.1% on
CTW1500, and by 22.0% and 5.9% on MSRA-TD500 respectively, this shows that the
proposed refinement module is more suitable for the normalized Fourier
descriptor. Not taking the Fourier descriptor as the reference location for
the Multi-Scale Deformable Attention module reduce the F-measure performance
of the proposed FANet by 5.0% and 21.2% on CTW1500 and MSRA-TD500
respectively, which shows that aggregating features near the text region is
conducive to the accurate regression of the network to the text instances.

Loss function. As shown in Table 7, removing $\mathcal{L}_{SD}$,
$\mathcal{L}_{FD}$ or $\mathcal{L}_{bbox}$ in the matching cost calculation
process and loss calculation process reduce the F-measure performance of the
proposed FANet by 5.0%, 0.6% and 0.1% on CTW1500 respectively, and by 25.2%,
0.6% and 0.8% on MSRA-TD500 respectively. From this result, we can know that
L1 loss for the resampled points of the text contour is more important than
other two losses. Nevertheless, mixing the three losses submit the best
performance, which indicates that the gains brought by the three losses are
not completely overlapping. In order to achieve the best performance, we take
the mixed loss of the three losses as the loss function of the proposed FANet.

Dense matching strategy. As shown in Table 2, with only 30 epochs of training,
the Dense Matching Strategy (DMS) can bring improvements of 3.3%, 8.6%, 3.5%
and 2.9% based on F-measure on ICDAR2015, MSRA-TD500, CTW1500 and TotalText
respectively. We attribute this to the fact that DMS can alleviate the
overfitting problem caused by lack of positive samples. As the number of
epochs increases, the gain brought by DMS becomes smaller, especially on
CTW1500 and TotalText. We attribute this to the fact that as the number of
training epochs increases, data enhancement can alleviate the performance
degradation caused by overfitting and narrows the performance gap between HMS
and DMS. So when we are faced with scenarios that enough training epochs are
provided such as pre-training, using the original HMS or use DMS with small
$H_{m}$ instead is a better choice. Match number $N_{m}$ in DMS is an
important hyperparameter, compared with the case when $N_{m}=1$, setting
$N_{m}$ to 10 will bring relative gain of $3.8\%$ $(87.3\%vs.83.5\%)$ based on
F-measure on MSRA-TD500, which shows the effectiveness of the DMS. Compared
with the case when $N_{m}=10$, setting $N_{m}$ to 5 or 15 will reduce the
performance of our network by $1.4\%$ $(87.3\%vs.85.9\%)$ and $1.2\%$
$(87.3\%vs.86.1\%)$ in F-measure respectively, which indicates that too large
or too small $N_{m}$ will damage the performance of the network. Compared with
the case when $N_{m}=1$, setting $N_{m}$ to 3 will bring relative gain of
$1.2\%$ $(84.7\%vs.83.5\%)$, $0.3\%$ $(84.9\%vs.84.6\%)$ and $0.2\%$
$(84.8\%vs.84.6\%)$ in F-measure on ICDAR2015, CTW1500 and TotalText
respectively. Same as MSRA-TD500, too large or too small $N_{m}$ will damage
the performance of the network. As we can see in Table 5, the optimal choice
of match number is 10 on MSRA-TD500 and 3 on ICDAR2015, CTW1500 and TotalText.
We can deduce that for a dataset with only a small number of text instances in
each image like MSRA-TD500, the problem of overfitting is exacerbated by the
lack of positive samples, so a larger $N_{m}$ can bring the network
considerable performance gain, but for datasets with dense text instances in
each image like ICDAR2015, CTW1500 and TotalText, a smaller $N_{m}$ is more
appropriate.

##  5 Conclusion

This paper focus on the fast localization learning and accurate detection for
scene text detection. We propose FANet, a Fast convergence and Accurate scene
text detection Network, which can achieve the SOTA performance on ICDAR2015,
MSRA-TD500, CTW1500 and TotalText. More importantly, FANet can accurately
detect scene text of arbitrary shapes with fewer training epochs.

##  Appendix A Fourier Descriptor Normalization

######  Theorem 1.

If $\mathcal{Z}(t)=\mathcal{X}(t)+i\mathcal{Y}(t)$ is a univariate continuous
periodic function about independent variable $t$, satisfying $t\in[0,1]$, the
period is 1, $(\mathcal{X}(t),\mathcal{Y}(t))\in[0,1]^{2}$. Then we have the
following conclusions:

$\displaystyle c_{k}$ $\displaystyle=\int_{0}^{1}\mathcal{Z}(t)e^{-2\pi
ikt}dt,k\in{\mathbb{Z}}$ (20) $\displaystyle c_{0}$
$\displaystyle=u_{0}+iv_{0},u_{0},v_{0}\in[0,1]$ (21) $\displaystyle c_{l}$
$\displaystyle=u_{l}+iv_{l},u_{l},v_{l}\in[-\frac{2}{\pi},\frac{2}{\pi}],l\in{\mathbb{Z}},l\neq
0$ (22)

###### Proof.

$\displaystyle c_{k}$ $\displaystyle=\int_{0}^{1}\mathcal{Z}(t)e^{-2\pi
ikt}dt$ $\displaystyle=\int_{0}^{1}[\mathcal{X}(t)+i\mathcal{Y}(t)][cos(-2\pi
kt)+isin(-2\pi kt)]dt$ $\displaystyle=\int_{0}^{1}[\mathcal{X}(t)cos(2\pi
kt)+\mathcal{Y}(t)sin(2\pi kt)]dt$
$\displaystyle+i\int_{0}^{1}[-\mathcal{X}(t)sin(2\pi
kt)+\mathcal{Y}(t)cos(2\pi kt)]dt$ $\displaystyle=u_{k}+iv_{k}$ (23)

When $k=0$, there are the following conclusions:

$\displaystyle u_{0}$ $\displaystyle=\int_{0}^{1}\mathcal{X}(t)dt\in[0,1]$
(24) $\displaystyle v_{0}$
$\displaystyle=\int_{0}^{1}\mathcal{Y}(t)dt\in[0,1]$ (25)

$\text{MSDeformAttn}({\bm{z}}_{q},\hat{{\bm{p}}}_{q},\\{{\bm{x}}^{l}\\}_{l=1}^{L})=\sum_{m=1}^{M}{\bm{W}}_{m}\big{[}\sum_{l=1}^{L}\sum_{s=1}^{S}{A}_{qmls}\cdot{\bm{W}}^{\prime}_{m}{\bm{x}}^{l}(\phi_{l}(\hat{{\bm{p}}}_{q})+\Delta{\bm{p}}_{qmls})\big{]}$
(26)

When $k\neq 0$, there are the following conclusions:

$\displaystyle|u_{k}|$ $\displaystyle=|\int_{0}^{1}[\mathcal{X}(t)cos(2\pi
kt)+\mathcal{Y}(t)sin(2\pi kt)]dt|$
$\displaystyle\leq|\int_{0}^{1}\mathcal{X}(t)cos(2\pi
kt)dt|+|\int_{0}^{1}\mathcal{Y}(t)sin(2\pi kt)dt|$ (27)

For term $\int_{0}^{1}\mathcal{X}(t)cos(2\pi kt)dt$, there are the following
inequalities:

$\displaystyle-\frac{1}{\pi}$ $\displaystyle=\int_{0}^{1}cos(2\pi
kt)I[cos(2\pi kt)<0]dt$ $\displaystyle\leq\int_{0}^{1}\mathcal{X}(t)cos(2\pi
kt)I[cos(2\pi kt)<0]dt$ $\displaystyle\leq\int_{0}^{1}\mathcal{X}(t)cos(2\pi
kt)dt$ $\displaystyle\leq\int_{0}^{1}\mathcal{X}(t)cos(2\pi kt)I[cos(2\pi
kt)>0]dt$ $\displaystyle\leq\int_{0}^{1}cos(2\pi kt)I[cos(2\pi kt)>0]dt$
$\displaystyle=\frac{4k}{2}\big{|}\int_{0}^{\frac{1}{4k}}cos(2\pi
kt)dt\big{|}$ $\displaystyle=\frac{4k}{2}\big{|}\frac{1}{2\pi k}sin(2\pi
kt)|_{t=\frac{1}{4k}}\big{|}=\frac{1}{\pi}$ (28)

We can get $|\int_{0}^{1}\mathcal{X}(t)cos(2\pi kt)dt|\leq\frac{1}{\pi}$, and
we can easily prove it in the same way that
$|\int_{0}^{1}\mathcal{Y}(t)sin(2\pi kt)dt|\leq\frac{1}{\pi}$. We substitute
it into Eq 27, then we can get:

$\displaystyle|u_{k}|$
$\displaystyle\leq\frac{1}{\pi}+\frac{1}{\pi}=\frac{2}{\pi},k\in{\mathbb{Z}},k\neq
0$ (29)

In the same way, we can easily prove that:

$\displaystyle|v_{k}|$ $\displaystyle\leq\frac{2}{\pi},k\in{\mathbb{Z}},k\neq
0$ (30)

∎

##  Appendix B Fourier descriptor as the reference location

Let $\\{{\bm{x}}^{l}\\}_{l=1}^{L}$ be the input multi-scale feature maps,
where ${\bm{x}}^{l}\in{\mathbb{R}}^{C\times H_{l}\times W_{l}}$. Let
$\hat{{\bm{p}}}_{q}\in[0,1]^{2}$ be the normalized coordinates of the
reference point for each query element $q$, then the Multi-Scale Deformable
Attention module [37] is applied as Equation 26,

where $m$ indexes the attention head, $l$ indexes the input feature level, and
$k$ indexes the sampling point. $\Delta{\bm{p}}_{qmls}$ and ${A}_{qmls}$
denote the sampling offset and attention weight of the $s$-th sampling point
in the $l$-th feature level and the $m$-th attention head, respectively. The
scalar attention weight ${A}_{qmls}$ is normalized by
$\sum_{l=1}^{L}\sum_{s=1}^{S}{A}_{qmls}=1$. Here, we use normalized
coordinates $\hat{{\bm{p}}}_{q}\in[0,1]^{2}$ for the clarity of scale
formulation, in which the normalized coordinates $(0,0)$ and $(1,1)$ indicate
the top-left and the bottom-right image corners, respectively. Function
$\phi_{l}(\hat{{\bm{p}}}_{q})$ in Equation 26 rescales the normalized
coordinates $\hat{{\bm{p}}}_{q}$ to the input feature map of the $l$-th level.
The multi-scale deformable attention samples $LS$ points from multi-scale
feature maps. In the proposed FANet, we consider the calculation process of
for query $q$ in $d$-th transformer decoder layer. We first obtain the
normalized Fourier descriptor predicted by the $(d-1)$-th decoder layer of
query $q$, which we denote as
${\bm{c}}=[u_{-K},v_{-K},\cdots,u_{0},v_{0},\cdots,u_{K},v_{K}]\in{\mathbb{R}}^{2(2K+1)}$,
We then calculate the normalized coordinate points on the predict text contour
in spatial domain by Inverse Discrete Fourier Transform (IDFT):

$\displaystyle{\bm{p}}_{qn}=\mathcal{F}^{-1}(\frac{n}{{N}},{\bm{c}}),n=1,\cdots,{N}$
(31)

where ${\bm{p}}_{qn}=(x_{qn},y_{qn})$ is the $n$-th point on the text contour
predicted by query $q$, $\mathcal{F}^{-1}$ is the Inverse Fourier Transform
(IFT) function, ${N}$ is the sampling number on the text contour. We then
calculate the bounding box of the text contour:

$\displaystyle x_{q}$ $\displaystyle=E_{i}(x_{qn})$ (32) $\displaystyle y_{q}$
$\displaystyle=E_{i}(y_{qn})$ (33) $\displaystyle w_{q}$
$\displaystyle=max_{i}(x_{qn})-min_{i}(x_{qn})$ (34) $\displaystyle h_{q}$
$\displaystyle=max_{i}(y_{qn})-min_{i}(y_{qn})$ (35)

Where $\\{x_{q},y_{q},w_{q},h_{q}\\}$ is the bounding box of the text contour.
We use $\phi_{l}((x_{q},y_{q}))$ instead of $\phi_{l}(\hat{{\bm{p}}}_{q})$ as
the new reference point for query $q$. The sampling offset
$\Delta{{\bm{p}}_{mlqh}}$ is also modulated by the box size, as
$(\Delta{{\bm{p}}_{qmlsx}}w_{q},\Delta{{\bm{p}}_{qmlsy}}h_{q})$.

##  Appendix C Refinement function

######  Theorem 2.

If $y=f[x+f^{-1}(x_{0})]$ is a continuous function about $x$, where $f$ is a
differentiable function and $x_{0}\in{\mathbb{R}}$ is a constant value, then:

$\displaystyle\lim_{x\rightarrow 0}\frac{\partial y}{\partial
x}=\frac{\partial f(z)}{\partial z}|_{z=f^{-1}(x_{0})}$ (36)

###### Proof.

$\displaystyle\lim_{x\rightarrow 0}\frac{\partial y}{\partial x}$
$\displaystyle=\lim_{x\rightarrow 0}\frac{\partial f(z)}{\partial
z}|_{z=x+f^{-1}(x_{0})}$ $\displaystyle=\frac{\partial f(z)}{\partial
z}|_{z=f^{-1}(x_{0})}$ (37)

∎

######  Theorem 3.

If $y=f[e^{x}f^{-1}(x_{0})]$ is a continuous function about $x$, where $f$ is
a differentiable function and $x_{0}\in{\mathbb{R}}$ is a constant value,
then:

$\displaystyle\lim_{x\rightarrow 0}\frac{\partial y}{\partial
x}=\frac{\partial f(z)}{\partial z}|_{z=f^{-1}(x_{0})}f^{-1}(x_{0})$ (38)

###### Proof.

$\displaystyle\lim_{x\rightarrow 0}\frac{\partial y}{\partial x}$
$\displaystyle=\lim_{x\rightarrow 0}\frac{\partial f(z)}{\partial
z}|_{z=e^{x}f^{-1}(x_{0})}f^{-1}(x_{0})e^{x}$ $\displaystyle=\frac{\partial
f(z)}{\partial z}|_{z=f^{-1}(x_{0})}f^{-1}(x_{0})$ (39)

∎

## References

  * [1] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In Andrea Vedaldi, Horst Bischof, Thomas Brox, and Jan-Michael Frahm, editors, Computer Vision - ECCV 2020 - 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part I, volume 12346 of Lecture Notes in Computer Science, pages 213–229. Springer, 2020\.
  * [2] Chee Kheng Chng and Chee Seng Chan.  Total-text: A comprehensive dataset for scene text detection and recognition.  In 14th IAPR International Conference on Document Analysis and Recognition, ICDAR 2017, Kyoto, Japan, November 9-15, 2017, pages 935–942. IEEE, 2017.
  * [3] Jifeng Dai, Haozhi Qi, Yuwen Xiong, Yi Li, Guodong Zhang, Han Hu, and Yichen Wei.  Deformable convolutional networks.  In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 764–773. IEEE Computer Society, 2017.
  * [4] Pengwen Dai, Sanyi Zhang, Hua Zhang, and Xiaochun Cao.  Progressive contour regression for arbitrary-shape scene text detection.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 7393–7402. Computer Vision Foundation / IEEE, 2021.
  * [5] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, pages 248–255. IEEE Computer Society, 2009.
  * [6] Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova.  BERT: pre-training of deep bidirectional transformers for language understanding.  In Jill Burstein, Christy Doran, and Thamar Solorio, editors, Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171–4186. Association for Computational Linguistics, 2019\.
  * [7] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  * [8] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross B. Girshick.  Mask R-CNN.  In ICCV, pages 2980–2988, 2017.
  * [9] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770–778. IEEE Computer Society, 2016.
  * [10] Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman K. Ghosh, Andrew D. Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, Faisal Shafait, Seiichi Uchida, and Ernest Valveny.  ICDAR 2015 competition on robust reading.  In 13th International Conference on Document Analysis and Recognition, ICDAR 2015, Nancy, France, August 23-26, 2015, pages 1156–1160. IEEE Computer Society, 2015.
  * [11] Zhanghui Kuang, Hongbin Sun, Zhizhong Li, Xiaoyu Yue, Tsui Hin Lin, Jianyong Chen, Huaqiang Wei, Yiqin Zhu, Tong Gao, Wenwei Zhang, Kai Chen, Wayne Zhang, and Dahua Lin.  MMOCR: A comprehensive toolbox for text detection, recognition and understanding.  In Heng Tao Shen, Yueting Zhuang, John R. Smith, Yang Yang, Pablo Cesar, Florian Metze, and Balakrishnan Prabhakaran, editors, MM ’21: ACM Multimedia Conference, Virtual Event, China, October 20 - 24, 2021, pages 3791–3794. ACM, 2021.
  * [12] Minghui Liao, Zhaoyi Wan, Cong Yao, Kai Chen, and Xiang Bai.  Real-time scene text detection with differentiable binarization.  In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 11474–11481. AAAI Press, 2020.
  * [13] Tsung-Yi Lin, Priya Goyal, Ross B. Girshick, Kaiming He, and Piotr Dollár.  Focal loss for dense object detection.  In IEEE International Conference on Computer Vision, ICCV 2017, Venice, Italy, October 22-29, 2017, pages 2999–3007. IEEE Computer Society, 2017.
  * [14] Xiyan Liu, Gaofeng Meng, and Chunhong Pan.  Scene text detection and recognition with advances in deep learning: a survey.  Int. J. Document Anal. Recognit., 22(2):143–162, 2019.
  * [15] Yuliang Liu, Hao Chen, Chunhua Shen, Tong He, Lianwen Jin, and Liangwei Wang.  Abcnet: Real-time scene text spotting with adaptive bezier-curve network.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 9806–9815. Computer Vision Foundation / IEEE, 2020.
  * [16] Yuliang Liu, Lianwen Jin, Shuaitao Zhang, Canjie Luo, and Sheng Zhang.  Curved scene text detection via transverse and longitudinal sequence connection.  Pattern Recognit., 90:337–345, 2019.
  * [17] Shangbang Long, Jiaqiang Ruan, Wenjie Zhang, Xin He, Wenhao Wu, and Cong Yao.  Textsnake: A flexible representation for detecting text of arbitrary shapes.  In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Computer Vision - ECCV 2018 - 15th European Conference, Munich, Germany, September 8-14, 2018, Proceedings, Part II, volume 11206 of Lecture Notes in Computer Science, pages 19–35. Springer, 2018.
  * [18] Zobeir Raisi, Mohamed A. Naiel, Georges Younes, Steven Wardell, and John S. Zelek.  Transformer-based text detection in the wild.  In IEEE Conference on Computer Vision and Pattern Recognition Workshops, CVPR Workshops 2021, virtual, June 19-25, 2021, pages 3162–3171. Computer Vision Foundation / IEEE, 2021.
  * [19] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian D. Reid, and Silvio Savarese.  Generalized intersection over union: A metric and a loss for bounding box regression.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 658–666. Computer Vision Foundation / IEEE, 2019.
  * [20] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  In Yoshua Bengio and Yann LeCun, editors, 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
  * [21] Russell Stewart, Mykhaylo Andriluka, and Andrew Y. Ng.  End-to-end people detection in crowded scenes.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 2325–2333. IEEE Computer Society, 2016.
  * [22] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998–6008, 2017.
  * [23] Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie.  Coco-text: Dataset and benchmark for text detection and recognition in natural images.  In arXiv preprint arXiv:1601.07140, 2016.
  * [24] Fangfang Wang, Yifeng Chen, Fei Wu, and Xi Li.  Textray: Contour-based geometric modeling for arbitrary-shaped scene text detection.  In Chang Wen Chen, Rita Cucchiara, Xian-Sheng Hua, Guo-Jun Qi, Elisa Ricci, Zhengyou Zhang, and Roger Zimmermann, editors, MM ’20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020, pages 111–119. ACM, 2020.
  * [25] Wenhai Wang, Enze Xie, Xiang Li, Wenbo Hou, Tong Lu, Gang Yu, and Shuai Shao.  Shape robust text detection with progressive scale expansion network.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, Long Beach, CA, USA, June 16-20, 2019, pages 9336–9345. Computer Vision Foundation / IEEE, 2019.
  * [26] Wenhai Wang, Enze Xie, Xiaoge Song, Yuhang Zang, Wenjia Wang, Tong Lu, Gang Yu, and Chunhua Shen.  Efficient and accurate arbitrary-shaped text detection with pixel aggregation network.  In 2019 IEEE/CVF International Conference on Computer Vision, ICCV 2019, Seoul, Korea (South), October 27 - November 2, 2019, pages 8439–8448. IEEE, 2019.
  * [27] Weijia Wu, Yuanqiang Cai, Debing Zhang, Sibo Wang, Zhuang Li, Jiahong Li, Yejun Tang, and Hong Zhou.  A bilingual, openworld video text dataset and end-to-end video text spotter with transformer.  arXiv preprint arXiv:2112.04888, 2021.
  * [28] Weijia Wu, Ning Lu, Enze Xie, Yuxing Wang, Wenwen Yu, Cheng Yang, and Hong Zhou.  Synthetic-to-real unsupervised domain adaptation for scene text detection in the wild.  In Proceedings of the Asian Conference on Computer Vision, 2020\.
  * [29] Weijia Wu, Enze Xie, Ruimao Zhang, Wenhai Wang, Guan Pang, Zhen Li, Hong Zhou, and Ping Luo.  Selftext beyond polygon: Unconstrained text detection with box supervision and dynamic self-training.  arXiv preprint arXiv:2011.13307, 2020.
  * [30] Weijia Wu, Jici Xing, Cheng Yang, Yuxing Wang, and Hong Zhou.  Texts as lines: text detection with weak supervision.  Mathematical Problems in Engineering, 2020, 2020.
  * [31] Weijia Wu, Jici Xing, and Hong Zhou.  Textcohesion: Detecting text for arbitrary shapes.  arXiv preprint arXiv:1904.12640, 2019.
  * [32] Weijia Wu, Debing Zhang, Ying Fu, Chunhua Shen, Hong Zhou, Yuanqiang Cai, and Ping Luo.  End-to-end video text spotting with transformer.  arXiv preprint arXiv:2203.10539, 2022.
  * [33] Cong Yao, Xiang Bai, Wenyu Liu, Yi Ma, and Zhuowen Tu.  Detecting texts of arbitrary orientations in natural images.  In 2012 IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, June 16-21, 2012, pages 1083–1090. IEEE Computer Society, 2012.
  * [34] Qixiang Ye and David S. Doermann.  Text detection and recognition in imagery: A survey.  IEEE Trans. Pattern Anal. Mach. Intell., 37(7):1480–1500, 2015\.
  * [35] Shi-Xue Zhang, Xiaobin Zhu, Jie-Bo Hou, Chang Liu, Chun Yang, Hongfa Wang, and Xu-Cheng Yin.  Deep relational reasoning graph network for arbitrary shape text detection.  In CVPR, CVPR 2020, Seattle, WA, USA, June 13-19, 2020, pages 9696–9705, 2020.
  * [36] Yu Zhou, Hongtao Xie, Shancheng Fang, Yan Li, and Yongdong Zhang.  Crnet: A center-aware representation for detecting text of arbitrary shapes.  In MM ’20: The 28th ACM International Conference on Multimedia, Virtual Event / Seattle, WA, USA, October 12-16, 2020, pages 2571–2580. ACM, 2020.
  * [37] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable DETR: deformable transformers for end-to-end object detection.  In 9th International Conference on Learning Representations, ICLR 2021, Virtual Event, Austria, May 3-7, 2021. OpenReview.net, 2021.
  * [38] Yiqin Zhu, Jianyong Chen, Lingyu Liang, Zhanghui Kuang, Lianwen Jin, and Wayne Zhang.  Fourier contour embedding for arbitrary-shaped text detection.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2021, virtual, June 19-25, 2021, pages 3123–3131. Computer Vision Foundation / IEEE, 2021.
  * [39] Yingying Zhu, Cong Yao, and Xiang Bai.  Scene text detection and recognition: recent advances and future trends.  Frontiers Comput. Sci., 10(1):19–36, 2016.