# Towards Robust Real-Time Scene Text Detection: From Semantic to Instance
Representation Learning

Xugong Qin∗ School of Cyber Science and Engineering, Nanjing University of
Science and Technology<EMAIL_ADDRESS>,  Pengyuan Lyu∗ Department of
Computer Vision Technology (VIS), Baidu Inc<EMAIL_ADDRESS>,  Chengquan
Zhang  Department of Computer Vision Technology (VIS), Baidu Inc.
<EMAIL_ADDRESS>,  Yu Zhou† Institute of Information Engineering,
Chinese Academy of SciencesSchool of Cyber Security, University of Chinese
Academy of Sciences<EMAIL_ADDRESS>,  Kun Yao  Department of Computer Vision
Technology (VIS), Baidu Inc<EMAIL_ADDRESS>,  Peng Zhang  School of Cyber
Science and Engineering, Nanjing University of Science and Technology
<EMAIL_ADDRESS>,  Hailun Lin  Institute of Information Engineering,
Chinese Academy of Sciences<EMAIL_ADDRESS>and  Weiping Wang  Institute
of Information Engineering, Chinese Academy of Sciences<EMAIL_ADDRESS>
(2023)

###### Abstract.

Due to the flexible representation of arbitrary-shaped scene text and simple
pipeline, bottom-up segmentation-based methods begin to be mainstream in real-
time scene text detection. Despite great progress, these methods show
deficiencies in robustness and still suffer from false positives and instance
adhesion. Different from existing methods which integrate multiple-granularity
features or multiple outputs, we resort to the perspective of representation
learning in which auxiliary tasks are utilized to enable the encoder to
jointly learn robust features with the main task of per-pixel classification
during optimization. For semantic representation learning, we propose global-
dense semantic contrast (GDSC), in which a vector is extracted for global
semantic representation, then used to perform element-wise contrast with the
dense grid features. To learn instance-aware representation, we propose to
combine top-down modeling (TDM) with the bottom-up framework to provide
implicit instance-level clues for the encoder. With the proposed GDSC and TDM,
the encoder network learns stronger representation without introducing any
parameters and computations during inference. Equipped with a very light
decoder, the detector can achieve more robust real-time scene text detection.
Experimental results on four public datasets show that the proposed method can
outperform or be comparable to the state-of-the-art on both accuracy and
speed. Specifically, the proposed method achieves 87.2% F-measure with 48.2
FPS on Total-Text and 89.6% F-measure with 36.9 FPS on MSRA-TD500 on a single
GeForce RTX 2080 Ti GPU.

Scene Text Detection, Real-Time, Representation Learning

††journalyear: 2023††copyright: rightsretained††conference: Proceedings of the
31st ACM International Conference on Multimedia; October 29-November 3, 2023;
Ottawa, ON, Canada††booktitle: Proceedings of the 31st ACM International
Conference on Multimedia (MM ’23), October 29-November 3, 2023, Ottawa, ON,
Canada††doi: 10.1145/3581783.3611801††isbn:
979-8-4007-0108-5/23/10††submissionid: 465††ccs: Applied computing Optical
character recognition

ACM Reference Format:

Xugong Qin, Pengyuan Lyu, Chengquan Zhang, Yu Zhou, Kun Yao, Peng Zhang,
Hailun Lin, and Weiping Wang. 2023. Towards Robust Real-Time Scene Text
Detection: From Semantic to Instance Representation Learning. In Proceedings
of the 31st ACM International Conference on Multimedia (MM ’23), October
29–November 3, 2023, Ottawa, ON, Canada. ACM, New York, NY, USA, 10 pages.
https://doi.org/10.1145/3581783.3611801

††*Equal contribution. The work was mainly done at IIE, CAS when Xugong Qin
was an intern at Baidu Inc.. $\dagger$Yu Zhou is the corresponding author.
Figure 1. The comparison of recent SOTA methods on the MSRA-TD500 dataset. SIR
achieves the best accuracy/speed trade-off. All results listed here are
reproduced from the corresponding official implementation on the same device.
“R18” and “R50” denote ResNet-18 and ResNet-50.

##  1\. Introduction

Optical character recognition has attracted much attention due to its
practical applications, e.g., scene understanding (Wang et al., 2022a; Zeng et
al., 2021, 2023), blind navigation, and document analysis (Long et al., 2022;
Shen et al., 2023). As an essential part of the text reading system (Wang et
al., 2022c; Wei et al., 2022), scene text detection (STD) is widely studied.
Significant progress has been witnessed in the past decades (Ye and Doermann,
2015; Zhu et al., 2016; Long et al., 2021; Yao et al., 2014b; Huang et al.,
2014; Tian et al., 2016; Gupta et al., 2016; Yao et al., 2016; Shi et al.,
2017a; Lyu et al., 2018; Xing et al., 2019; Liao et al., 2020; Wang et al.,
2022b; Wan et al., 2021; Song et al., 2022; Xue et al., 2022; Yu et al., 2023;
Guo et al., 2021, 2022). However, STD remains challenging due to significant
variations in scale, orientation, and aspect ratio as well as curved shape.

Due to the flexible representation of arbitrary-shaped scene text,
segmentation-based methods have become mainstream and popular in academic and
industrial communities. Top-down methods (He et al., 2017a; Wang et al.,
2020c) perform foreground/background segmentation based on the “locate-then-
seg” pipeline. To make it simple and efficient, the whole image can be parsed
as text/non-text region (Long et al., 2015; Yao et al., 2016; Zhang et al.,
2016) from the bottom-up view. However, distinguishing different instances
becomes an essential problem since scene text could be born crowded. Among
methods settling this, shrinking text regions (Zhou et al., 2017; Wu and
Natarajan, 2017; Wang et al., 2019b, c; Liao et al., 2020) become the most
prevalent way. Nevertheless, these methods are with different implementations,
which makes it hard to compare them in one framework. Moreover, the
performance is generally inferior to top-down methods, which are not robust to
instance adhesion and false positives issues.

Figure 2. Different learning paradigms for robust scene text detection within
an encoder-decoder framework. (a): Fusing multi-granularity features (Ye et
al., 2020) for rich representation. (b): Fusing multiple outputs from
different modeling perspectives (Xie et al., 2019; Liu et al., 2019c). (c):
Representation learning with auxiliary tasks (ours). “Enc”, “Feat”, “Dec”, and
“Out” denote encoder, feature, decoder and output respectively.

The learning paradigms of existing robust STD methods can be roughly
summarized into two categories as shown in Fig. 2 (a) and (b). As shown in
Fig. 2 (a), one kind of method is fusing multi-granularity features to obtain
rich representation. TextfuseNet (Ye et al., 2020) is a typical work in which
features from semantic segmentation, word level, and character level are
extracted and fused. The other category method follows the pipeline listed in
Fig. 2 (b), which fuses multiple outputs from different modeling perspectives
to re-correct the confidence of predictions (Xie et al., 2019; Liu et al.,
2019c) to suppress the false positives. The above methods all focus on top-
down methods, and enjoy improved performance but are slowed by the burdened
pipelines. To achieve robust real-time scene text detection, we choose popular
bottom-up segmentation which is efficient and accurate yet still not fully
exploited. To avoid bringing additional computation and parameters in testing,
we resort to the perspective of representation learning and develop a
different learning paradigm in which auxiliary tasks are paralleled with the
decoder in training to help the encoder learn robust representation.

However, two issues matter in robust representation learning. One is the
insufficiency of semantic-level modeling. Specifically, long-range relation
modeling is only achieved by a stack of locally connected convolution layers,
which makes it inefficient to relate distant pixels and maintain semantic
consistency across the whole image. The other lies in the absence of instance-
level modeling. Since scene text detection is essentially an instance-level
task. Though the shrinking text region is widely adopted to distinguish
different text instances, it works in a bottom-up way. We argue that the lack
of global instance-level concepts which naturally exist in top-down methods
makes it one of the key reasons for the inferior performance.

To address the issues, we first compose a unified baseline model which unifies
existing methods into a weighted three-channel segmentation problem. For
semantic representation learning, we propose global-dense semantic contrast
(GDSC) as an auxiliary task in which a global semantic vector is extracted and
contrasted to the dense feature vectors, which not only helps to maintain
semantic consistency but also is efficient in computation. Different from SAE
(Tian et al., 2019) and PAN (Wang et al., 2019c) focus on pulling features
from the same instances, and pushing away features from different instances,
GDSC operates on the semantic level which keeps the intra-class (text)
distance small and inter-class distances (text/non-text) large. For instance-
level representation learning, we propose to integrate top-down modeling (TDM)
with bottom-up segmentation into one framework, inspired by GTC (Hu et al.,
2020) which uses a top-down attention mechanism (Shi et al., 2019) to
supervise bottom-up CTC (Shi et al., 2017b) methods in text recognition (Shi
et al., 2017b, 2019; Du et al., 2022; Qiao et al., 2020a, b, 2021).
Specifically, the widely adapted two-stage detection framework Faster R-CNN is
utilized to perform regional recognition tasks, which provide instance-level
information during the encoder learning.

In this work, we propose a robust real-time scene text detection framework,
namely SIR, which explores from Semantic to Instance Representation learning
within a bottom-up segmentation framework. The main contributions of the paper
are as follows:

  * •

We propose a new framework for robust real-time scene text detection which
utilizes auxiliary tasks to help the encoder learn robust representation
without introducing any computations or parameters during inference.

  * •

We revisit existing works, unify existing segmentation-based methods into a
simple weighted three-channel cross-entropy loss and propose a strong baseline
model.

  * •

Global-dense semantic contrast is proposed for effective semantic
representation learning. We exploit top-down modeling in a segmentation-based
framework and find it significantly boosts the performance of the bottom-up
segmentation pipeline.

  * •

Experiments on four public datasets including horizontal, multi-oriented, and
curved text demonstrate the effectiveness and robustness of the proposed
method on real-time scene text detection.

##  2\. Related Work

According to different modeling perspectives, existing scene text detection
methods can be divided into bottom-up (BU) and top-down (TD) methods.
Considering different modeling granularity, methods can be divided into pixel-
based and region-based/contour-based methods, which correspond to
segmentation-based (SEG) and regression-based (REG) methods in previous work.
To better summarize existing works, we provide a detailed categorization of
existing STD methods: TD-REG, TD-SEG, BU-REG, and BU-SEG.

TD-REG methods (Zhou et al., 2017; Liu and Jin, 2017; He et al., 2017d, b;
Liao et al., 2018a; Wang et al., 2018; Liao et al., 2018b; Xue et al., 2018;
Wang et al., 2019d; Liu et al., 2020a; Qin et al., 2021a; Tang et al., 2022;
Wang et al., 2022c) regress the geometry of text bounding boxes, e.g., the
horizontal rectangles (Ren et al., 2015), oriented rectangles (Ma et al.,
2018; Zhou et al., 2017), quadrilaterals (Zhou et al., 2017; Liu and Jin,
2017; Liao et al., 2018a; Wang et al., 2018; Qin et al., 2021a), polygons (Liu
et al., 2019a) or parametric curves (Liu et al., 2020a; Wang et al., 2020a;
Zhu et al., 2021; Wang et al., 2022c), and contours (Dai et al., 2021; Zhang
et al., 2021). These methods are modified from the modern general detectors
(Liu et al., 2016; Ren et al., 2015) which perform relative position
regression based on anchor boxes or anchor points with one-stage prediction or
multi-stage refinement.

TD-SEG methods (He et al., 2017a; Yang et al., 2018; Liao et al., 2021; Liu et
al., 2019c; Qin et al., 2019; Ye et al., 2020; Wang et al., 2020b; Liu et al.,
2020b; Qin et al., 2021b; He et al., 2021) perform instance segmentation based
on region features or grid features. Mask R-CNN (He et al., 2017a) build a
mask head upon Faster R-CNN (Ren et al., 2015), which performs per-pixel
classification in a given region of interest (RoI) whether it belongs to the
instance corresponding to the RoI. To alleviate errors accumulated from
multiple stages, one-stage instance segmentation (Wang et al., 2020c) is
proposed to directly produce an instance-level mask. However, TD methods
require multiple stages or deep stack convolution due to the limitation of the
receptive field. On the contrary, BU methods alleviate the demand by
predicting local units, which gives a more flexible representation.

BU-REG methods (Tian et al., 2016; Shi et al., 2017a; Long et al., 2018; Tang
et al., 2019; Xue et al., 2019; Zhang et al., 2020b; Zhou et al., 2020; Ma et
al., 2021) learns to regress local geometry which is also based on predefined
anchors. Local bounding boxes (Tian et al., 2016; Shi et al., 2017a; Long et
al., 2018; Tang et al., 2019; Zhang et al., 2020b; Ma et al., 2021) or
boundary points (Xue et al., 2019; Zhou et al., 2020; Sheng et al., 2021) are
predicted in BU-SEG methods, which can handle long text well. However, these
kinds of methods usually require complex post-processing and tend to suffer
from false positives.

BU-SEG methods (Zhang et al., 2016; He et al., 2017c; Wu and Natarajan, 2017;
Deng et al., 2018; Xu et al., 2019; Tian et al., 2019; Wang et al., 2019b, c;
Liao et al., 2020; Sheng et al., 2021; Liao et al., 2023) regard text
detection as semantic segmentation problem and perform per-pixel
classification on the input image. However, different from the general
semantic segmentation tasks, different text instances must be separable on the
output semantic maps to obtain the bounding box of each text instance. Thus
shrinking text regions is widely adopted in BU-SEG methods. To reconstruct the
text regions from the text kernel, dilating text contour (Wu and Natarajan,
2017; Liao et al., 2020), progressive scale expansion (Wang et al., 2019b),
and embedding clustering (Wang et al., 2019c; Tian et al., 2019) are proposed.
Compared with TD methods, BU methods need to link the local unit predictions
to generate instance-level results. Due to the lack of instance-level
supervision, BU-SEG methods not only struggled to deal with false positives
but also suffered from instance adhesion problems.

Generally speaking, methods can be divided into multiple categories, e.g.,
Mask R-CNN (He et al., 2017a) involves TD-Reg and TD-SEG stages, and Corner
(Lyu et al., 2018) includes BU-REG corner localization and BU-SEG position-
aware segmentation. The proposed method is also inspired by the complementary
of different modeling perspectives.

Real-time scene text detection (Zhou et al., 2017; Wang et al., 2019c; Liao et
al., 2020, 2023) not only cares about the performance but also the inference
speed. Existing work mainly focuses on light structures with elaborate
designs. Efficient feature fusion is widely studied in EAST (Zhou et al.,
2017), PAN (Wang et al., 2019c), DB (Liao et al., 2020), DB++ (Liao et al.,
2023) as well as compact model design and efficient learning mechanisms. Due
to shallow network design and simple supervision, existing real-time methods
suffer from insufficient representation and inaccurate detection results. In
this work, different from these methods, we aim to improve the representation
learned by the encoder without introducing parameters and computation during
inference, which enables robust real-time scene text detection.

##  3\. Methodology

The proposed method consists of three parts: a BU-SEG baseline, GDSC, and TDM.
We first briefly describe the overall pipeline of the proposed method and then
detail these parts. The label generation and optimization objectives are
presented at last.

The architecture of SIR is illustrated in Fig. 3, which consists of the main
and auxiliary tasks. The main task is a BU-SEG task which is a typical
encoder-decoder-based framework. The auxiliary tasks include GDSC and TDM,
which are in parallel with the decoder to promote the encoder in training and
removed during inference. Given an input image, the feature
$F_{enc}\in{\mathbb{R}}^{H\times W\times C}$ is obtained through the encoder
network and then decoded to produce the probability map via a lightweight
decoder. In training, besides the main segmentation task, GDSC and TDM perform
as the auxiliary tasks which take the encoded feature $F_{enc}$ as the input
and propagate the gradients back to the encoder. The GDSC and TDM promote
representation learning from the semantic and instance levels, providing
strong complements to the man task without bringing burdens during inference.

Figure 3. The overall pipeline of SIR. It mainly contains one BU-SEG main
network and two auxiliary tasks including GDSC and TDM. The auxiliary tasks
only work in training.

###  3.1. A Unified Bottom-Up Segmentation Baseline

In this subsection, we explore building a unified segmentation baseline for
different implementations of shrinking text regions and fairly compare
different variants (Yao et al., 2016; Xue et al., 2018; Liao et al., 2020)
within the same framework. As shown in Fig. 3 (a), the baseline model follows
an encoder-decoder segmentation framework.

Encoder. The encoder consists of the backbone and the neck. ResNet (He et al.,
2016) with deformable convolution (Zhu et al., 2019) is used as the backbone,
followed by an FPN (Lin et al., 2017; Liao et al., 2020, 2023) and the
adaptive scale fusion module (Liao et al., 2023). The structure of the encoder
is the same as that in (Liao et al., 2023). The output dimension of the
encoder is 256 and the spatial dimension is $\frac{1}{4}$ of the input image.

Decoder. The decoder part includes a 3$\times$3 convolution with 64 channels,
followed by a BN and a ReLU, and a convolution layer for prediction. To be
compatible with the border class (Wu and Natarajan, 2017), we set the channel
of prediction convolution to 3, which represents non-text, text kernel, and
text border. The output is up-sampled to the same scale as the input image and
a “softmax” function is used to normalize the probabilities of different
classes.

Loss Function. Due to a serious imbalance between the foreground and
background classes, class-aware balancing is widely used. We use a simple
class-aware weighting strategy that is inversely proportional to the numbers
of the corresponding classes. The segmentation loss can be defined as a
weighted cross-entropy loss:

(1) $\mathcal{L}_{seg}=\frac{1}{HW}\sum_{i=1}^{H}\sum_{j=1}^{W}w_{ij}{\rm
CE}(p_{ij},y_{ij}),$

where $p$, $y$, and $w$ denote the prediction, the ground truth, and the
corresponding weight. And the weight $w$ is defined as follows:

(2) $w_{ij}={\mathbb{W}}^{\top}y_{ij},$

in which per pixel is weighted based on a class-aware weight $\mathbb{W}$ and
can be defined as follows if taking border class into account:

(3) ${\mathbb{W}}=[1,\frac{N_{nt}}{N_{tk}},\frac{N_{nt}}{N_{tb}}],$

or

(4) ${\mathbb{W}}=[1,\frac{N_{nt}+N_{tb}}{N_{tk}}],$

if consider the border class as background, where $N_{nt}$, $N_{tk}$, and
$N_{tb}$ mean the number of pixels which belong to the non-text, text kernel,
and text border regions.

Supervision. Different supervision within one model is achieved as follows.
(1): Kernel/ non-kernel segmentation (Wang et al., 2019c; Liao et al., 2020;
Sheng et al., 2021; Liao et al., 2023). The probability of non-kernel is
obtained by the sum of probabilities of the border and non-text. (2): (1) +
class-aware weights (consider kernel/non-kernel only, denoted as W_kn). (3):
(1) + class-aware weights (kernel/border/non-text, W_kbn). (4): (3) + explicit
border and non-text supervision (Wu and Natarajan, 2017). (5): Text region
segmentation (Yao et al., 2016; Ch’ng and Chan, 2017) with W_kbn. The
probability of text region is given out by summing the prediction from the
kernel and border channels.

Post-process. During inference, the probability map is first obtained by
selecting the kernel class prediction from the output of the decoder and then
binarized with a given threshold. Then connected components labeling (CCL) is
performed on the binarized map. Finally, we obtain predicted polygons by
dilating the contour of connected components with the Vatti clipping algorithm
(Vatti, 1992) following (Liao et al., 2020, 2023).

Experiments with different supervision are performed on the ICDAR2015 (IC15)
and Total-Text (TT) datasets for comparison. As shown in Table 1, with W_kn,
kernel segmentation can be promoted from 74.1% to 80.2% F-measure on TT.
However, we find a drop of 2.6% on ICDAR2015 in which incident text occupies.
Moreover, we observed, the kernel/non-kernel boundary is hard to decide with
W_kn. We find this phenomenon originates from two aspects. One is the
imbalanced weight. The border region is taken as non-kernel which has a much
smaller weight compared to the kernel region. When the border regions are
explicitly weighted, the results are significantly improved (52.9%-¿80.3% on
IC15, 80.2%-¿82.4% on TT). The other is the fussy semantic labels in which the
border and non-text classes are coupled with W_kn. The boundary pixels are
hard to be correctly discriminated due to the fussy semantics of the hand-
craft boundary class. When the border class is independently weighed and
decoupled from the non-kernel class, the network is given clear semantic
boundary information, leading to better results (80.3%-¿81.2% on IC15,
82.4%-¿83.7% on TT). The last row in Table 1 shows that the weighted three-
channel classification is also compatible with the text region segmentation
settings (Yao et al., 2016; Ch’ng and Chan, 2017).

Table 1. Detection results with different settings of the BU-SEG baseline on ICDAR2015 and Total-Text. “P”, “R”, and “F” denote precision, recall, and F-measure respectively. # | Method | ICDAR2015 | Total-Text
---|---|---|---
P |  R |  F |  P |  R |  F
1 | kernel |  66.3 |  47.7 |  55.5 |  85.9 |  65.2 |  74.1
2 | kernel + W_kn |  53.0 |  52.7 |  52.9 |  83.8 |  76.9 |  80.2
3 | kernel + W_kbn |  82.3 |  78.5 |  80.3 |  82.7 |  82.2 |  82.4
4 | kernel + W_kbn + border |  85.0 |  77.8 |  81.2 |  88.8 |  79.2 |  83.7
5 | text + W_kbn |  42.8 |  29.2 |  34.7 |  70.2 |  66.3 |  68.2

###  3.2. Global-Dense Semantic Contrast

Global-dense semantic contrast is proposed to enhance the semantic
representation in a contrastive way. As shown in Fig. 3 (b), given encoded
feature $F_{enc}\in{\mathbb{R}}^{H\times W\times C}$, a shared project layer
is first used to change the channel dimension to $C^{{}^{\prime}}$, then two
individual project heads are further utilized to embed the feature to obtain
$F_{q}\in{\mathbb{R}}^{H\times W\times C^{{}^{\prime}}}$ and
$F_{k}\in{\mathbb{R}}^{H\times W\times C^{{}^{\prime}}}$. The vector $f_{q}$
is extracted by feature sampling on $F_{k}$ according to the proposal mask
$M\in{\mathbb{R}}^{H\times W}$, which aggregates semantic-rich features and
averages to obtain the global semantic representation:

(5)
$f_{q}=\frac{\sum_{x,y}F_{k}(x,y)\mathbb{I}[M(x,y)=1]}{\sum_{x,y}\mathbb{I}[M(x,y)=1]},$

in which $M$ is obtained by interpolation of the text region mask $T$ to the
shape of $H\times W$. We use a $3\times 3$ convolution followed by a BN and a
ReLU and $1\times 1$ convolutions to implement the share project layer and the
separated project heads. $C^{{}^{\prime}}$ is set to 128 in implementation.

After that, we measure the similarity between $f_{q}$ and each element
$F(x,y)$ of $F_{k}$ by computing the inner product, which can be efficiently
implemented with matrix multiplication. Then a normalized similarity score $S$
is further obtained by scaling the similarity to (0, 1) with the “sigmoid”
function, which avoids introducing the hyper-parameter margin (Tian et al.,
2019; Wang et al., 2019c) in metric learning:

(6) $S(x,y)={\rm sig}(f_{q}^{\top}F_{k}(x,y)).$

$S$ is up-sampled to the same scale as the input image. We use the text region
binary map $T$ to supervise the learning process with the dice loss (Milletari
et al., 2016), the loss can be defined as follows:

(7)
$\mathcal{L}_{gdsc}=1-\frac{2\sum{T(x,y)S(x,y)}}{\sum{T(x,y)}+\sum{S(x,y)}}.$

Since $T$ is a binary map, in which 1 represents text (+), 0 represents non-
text (-). The above equation can be written as:

(8) $\mathcal{L}_{gdsc}=1-\frac{2\sum_{k_{+}}{\rm
sig}(f_{q}^{\top}k_{+})}{\sum{T}+\sum_{k_{+}}{\rm
sig}(f_{q}^{\top}k_{+})+\sum_{k_{-}}{\rm sig}(f_{q}^{\top}k_{-})},$

where $\\{k_{+}\\}$ and $\\{k_{+}\\}$ satisfy:

(9) $\\{k_{+}\\}=\\{F_{k}(x,y)|T(x,y)=1\\},$ (10)
$\\{k_{-}\\}=\\{F_{k}(x,y)|T(x,y)=0\\},$

which is a similar form to the typical InfoNCE (Oord et al., 2018), in which
contrastive learning is performed between the global semantic representation
and features within the same image.

GDSC performs dense contrast between a global semantic representation and the
dense grid features, which can be viewed as a dynamic convolution (Jia et al.,
2016) in which the convolution parameters are conditioned on a global
representation, enabling better semantic representation learning in an elegant
way.

###  3.3. Combining Top-Down Modeling

In the BU-SEG framework, text instances are obtained from CCL on the
probability map during inference, which lacks the guidance of instance-level
concepts in representation learning and is sensitive to false positives and
text adhesion. To settle this issue, we introduce TDM through which the
instance concept is naturally given out to the BU-SEG framework.

Without loss of generality, we choose the classical region-based TD modeling
method Faster R-CNN (Ren et al., 2015). As shown in Fig. 3 (c), given the
encoded feature $F_{enc}$, TDM performs anchor-based dense supervision and
proposal-based spare supervision at pixel-level and region-level respectively,
enabling the encoder to learn instance-aware representation.

Anchor-Based Dense Supervision. From the view of BU-SEG, the encoded $F_{enc}$
is used to decode to produce a probability of each class. Considering from the
TD perspective, each vector from the spatial position of
$F_{enc}\in{\mathbb{R}}^{H\times W\times C}$ encoded the information of one
area on the original input image. Associated with one predefined anchor per
location as that in RPN (Ren et al., 2015) and SSD (Liu et al., 2016),
$F_{enc}$ can be utilized to produce the prediction of text instances by
text/non-text classification and relative geometry regression, which provide
instance-level information in a pixel-wise manner.

Standard RPN (Ren et al., 2015) is used to provide anchor-based dense
supervision. To process scene text of different scales, we perform consecutive
max-pooling on $F_{enc}$ and construct a feature pyramid with scales of
$\\{\frac{1}{4},\frac{1}{8},\frac{1}{16},\frac{1}{32},\frac{1}{64}\\}$ of the
input image.

As stated in ATSS (Zhang et al., 2020a), despite the way of encoding relative
position information, the result of using the anchor box and anchor point are
very close. If we don’t take the interpolation into consideration, the
supervision of the predicted probability map (Liao et al., 2020; Wang et al.,
2019c) equals the supervision of the anchor point in EAST (Zhou et al., 2017),
from which we can infer that anchor-based dense supervision has been performed
in the BU-SEG framework on a certain degree. When anchor-based dense
supervision is applied, the instance concept is further strengthened.

Proposal-Based Sparse Supervision. Given the encoded feature $F_{enc}$ and a
set of proposals, we first extract regional features
$F_{r}\in{\mathbb{R}}^{N\times 7\times 7\times C}$ with RoI-Align (He et al.,
2017a), in which $N$ denote the numbers of proposals. Then regional
recognition tasks including region-wise classification and regression are
performed. Specifically, regional classification of text/non-text complements
pixel-level classification with high-level clues. Besides classifying pixels
into one semantic category, regional classification learns to distinguish
whether the region contains a complete instance that is visually and
semantically self-consistent. Regional regression learns to predict region-
wise relative position, enabling the encoder to learn to identify the boundary
of text instances, which not only helps to promote the accuracy of semantic
segmentation but also benefits obtaining complete instance contour by CCL.

We use the fast R-CNN branch (Ren et al., 2015) to implement the proposal-
based sparse supervision. The proposals are obtained by sorting and performing
NMS on the output of RPN, which proves to work quite well in practice.

Figure 4. Qualitative detection results of SIR on the Total-Text, CTW1500,
ICDAR2015, and MSRA-TD500 datasets.

###  3.4. Label Generation and Optimization

Label generation. Given an input training sample, we generate labels for the
above-mentioned tasks with the raw annotations.

For the main semantic segmentation task, we assign a category label for each
pixel to generate the ground truth mask. The categories include text kernel,
text border (Wu and Natarajan, 2017), and non-text region. Text kernel is
obtained by shrinking the text region in an instance-wise way following (Wang
et al., 2019b, c; Liao et al., 2020). The region in the boundary between a
text instance and the corresponding kernel is regarded as the text border. In
default, the shrink ratio is set to 0.6 empirically.

For GDSC, the ground truth $T$ is generated in the same way as the semantic
segmentation task. The only difference is $T$ is a binary mask that only
considers text and non-text.

For TDM, the ground truth of the bounding box is obtained by finding the
minimum and maximum values among the vertex coordinates of the polygon
annotation. The instance label is set to the ”text” category.

Optimization. The network is optimized with multiple supervision and the
overall loss $\mathcal{L}$ can be formulated as below:

(11)
$\mathcal{L}=\mathcal{L}_{seg}+\lambda_{1}\mathcal{L}_{gdsc}+\lambda_{2}\mathcal{L}_{tdm},$

where $\mathcal{L}_{seg}$, $\mathcal{L}_{gdsc}$, and $\mathcal{L}_{tdm}$
denote the loss functions of the semantic segmentation, GDSC, and TDM.
$\lambda_{1}$ and $\lambda_{2}$ are set to 3.0 and 1.0 respectively.

The loss of TDM contains multiple parts:

(12)
$\mathcal{L}_{tdm}=\lambda_{3}\mathcal{L}_{rpn}+\lambda_{4}\mathcal{L}_{cls}+\mathcal{L}_{reg},$

where $\mathcal{L}_{rpn}$, $\mathcal{L}_{cls}$, and $\mathcal{L}_{reg}$ refer
to the losses of anchor-based dense supervision and proposal-based sparse
supervision, which are the same as that defined in Faster R-CNN (Ren et al.,
2015). Both $\lambda_{3}$ and $\lambda_{4}$ are set to 1.0 empirically.

##  4\. Experiments

###  4.1. Datasets

SynthText (Gupta et al., 2016) is a synthetic dataset which consists of 800k
images. These images are synthesized from 8k background images. This dataset
is only used for pre-training when compared with the SOTA methods.

ICDAR2015 (Karatzas et al., 2015) is a multi-oriented text detection dataset
only for English words, which includes 1000 training images and 500 testing
images. The text regions are annotated with quadrilaterals.

MSRA-TD500 (Yao et al., 2012) focus on the Chinese and English incidental text
lines, which are always presented in a long and multi-oriented pattern. Large
variations of text scale and orientation are presented in this dataset. It
consists of 300 training images and 200 testing images. Since the training
images are too few, we follow the common practice of previous works (Zhou et
al., 2017; Long et al., 2018; Wang et al., 2019c) to add 400 images from HUST-
TR400 (Yao et al., 2014a) to the training data.

Total-Text (Ch’ng and Chan, 2017) has 1255 training images and 300 testing
images, which contain horizontal, multi-oriented, and curve texts. Each text
is labeled as a polygon at the word level.

CTW1500 (Liu et al., 2019a) contains 1000 training images and 500 testing
images. There are 10751 text instances in total, where 3530 are curved texts.
In this dataset, text instances are annotated with 14-point polygons at the
text line level.

###  4.2. Implementation Details

Training. Pre-trained weights on ImageNet are used to initialize the backbone.
The models are trained for 1200 epochs on the corresponding datasets with an
SGD optimizer. The initial learning rate is set to 0.005 with a batch size of
16, following a “poly” learning rate policy as (Chen et al., 2017). The warmup
strategy is used in the first 1000 iterations for stable optimization.
Additionally, when comparing with the SOTA methods, we first pre-train our
models on the SynthText dataset for 2 epochs and then fine-tune the models on
the corresponding datasets.

Several data augmentation technologies are used in training: (1) Random
flipping; (2) Random scaling of the image with a scale factor sampling from
$[0.5,3.0]$; (3) Random rotation with an angle range of
$[-10^{\circ},10^{\circ}]$; (4) Random cropping and padding to the size of
$640\times 640$.

Inference. During inference, the input image is resized along the short side
with the respect ratio unchanged. The models are evaluated under the
environment of a single GeForce RTX 2080 Ti GPU and an Intel Xeon Silver 4214R
CPU with a batch size of 1.

###  4.3. Comparisons with State-of-the-Art Methods

We compare the proposed method on four pubic datasets of different types
including Total-Text, CTW1500, ICDAR2015, and MSRA-TD500 to demonstrate the
effectiveness of SIR on real-time arbitrary-shaped scene text detection. As
shown in Fig. 4, SIR is able to detect four types of scene text, which
demonstrate robustness against large variations in aspect ratio and complex
shape. Specially, we reproduce several real-time scene text detection methods
including PAN111https://github.com/whai362/pan pp.pytorch.git (Wang et al.,
2019c) and DB/ DB++222https://github.com/MhLiao/DB.git (Liao et al., 2020,
2023) from the corresponding official implementation for a fair comparison.

Table 2. Detection results on the Total-Text, CTW1500, ICDAR2015, and MSRA-TD500 datasets. * denotes reproduced results by us. $\dagger$ denotes the use of character-level supervision. Method |  Ext | Total-Text | CTW1500 | ICDAR2015 | MSRA-TD500
---|---|---|---|---|---
|  P |  R |  F |  FPS |  P |  R |  F |  FPS |  P |  R |  F |  FPS |  P |  R |  F |  FPS
TextField (Xu et al., 2019) |  Synth |  81.2 |  79.9 |  80.6 |  - |  83.0 |  79.8 |  81.4 |  - |  84.3 |  83.9 |  84.1 |  - |  87.4 |  75.9 |  81.3 |  -
LOMO (Zhang et al., 2019) |  Synth |  88.6 |  76.7 |  81.6 |  4.4 |  89.2 |  69.6 |  78.4 |  4.4 |  91.3 |  83.5 |  87.2 |  3.4 |  - |  - |  - |  -
CRAFT (Baek et al., 2019)$\dagger$ |  MLT17 |  87.6 |  79.9 |  83.6 |  - |  86.0 |  81.1 |  83.5 |  - |  89.8 |  84.3 |  86.9 |  - |  88.2 |  78.2 |  82.9 |  -
PSENet-1s (Wang et al., 2019b) |  MLT17 |  84.0 |  78.0 |  80.9 |  3.9 |  84.8 |  79.7 |  82.2 |  3.9 |  86.9 |  84.5 |  85.7 |  1.6 |  - |  - |  - |  -
ATRR (Wang et al., 2019a) |  - |  80.9 |  76.2 |  78.5 |  - |  80.1 |  80.2 |  80.1 |  - |  89.2 |  86.0 |  87.6 |  - |  85.2 |  82.1 |  83.6 |  -
SAE (Tian et al., 2019) |  Synth |  - |  - |  - |  - |  82.7 |  77.8 |  80.1 |  - |  88.3 |  85.0 |  86.6 |  - |  84.2 |  81.7 |  82.9 |  -
CSE (Liu et al., 2019b) |  MLT17 |  81.4 |  79.1 |  80.2 |  2.4 |  81.1 |  76.0 |  78.4 |  2.6 |  - |  - |  - |  - |  - |  - |  - |  -
MSR (Xue et al., 2019) |  Synth |  83.8 |  74.8 |  79.0 |  - |  85.0 |  78.3 |  81.5 |  4.3 |  86.6 |  78.4 |  82.3 |  4.3 |  87.4 |  76.7 |  81.7 |  -
DRRG (Zhang et al., 2020b) |  MLT17 |  86.5 |  84.9 |  85.7 |  - |  85.9 |  83.0 |  84.5 |  - |  88.5 |  84.7 |  86.6 |  - |  88.1 |  82.3 |  85.1 |  -
ContourNet (Wang et al., 2020b) |  - |  86.9 |  83.9 |  85.4 |  3.8 |  83.7 |  84.1 |  83.9 |  4.5 |  87.6 |  86.1 |  86.9 |  3.5 |  - |  - |  - |  -
TextFuseNet (Ye et al., 2020)$\dagger$ |  Synth |  87.5 |  83.2 |  85.3 |  7.1 |  85.8 |  85.0 |  85.4 |  7.3 |  91.3 |  88.9 |  90.1 |  8.3 |  - |  - |  - |  -
SD (Xiao et al., 2020)$\dagger$ |  MLT17 |  89.2 |  84.7 |  86.9 |  - |  85.8 |  82.3 |  84.0 |  - |  88.7 |  88.4 |  88.6 |  |  - |  - |  - |  -
TextRay (Wang et al., 2020a) |  ArT |  83.5 |  77.9 |  80.6 |  - |  82.8 |  80.4 |  81.6 |  - |  - |  - |  - |  - |  - |  - |  - |  -
FCENet (Zhu et al., 2021) |  - |  89.3 |  82.5 |  85.8 |  - |  87.6 |  83.4 |  85.5 |  - |  90.1 |  82.6 |  86.2 |  - |  - |  - |  - |  -
PCR (Dai et al., 2021) |  MLT17 |  88.5 |  82.0 |  85.2 |  - |  87.2 |  82.3 |  84.7 |  - |  - |  - |  - |  - |  90.8 |  83.5 |  87.0 |  -
BPN (Zhang et al., 2021) |  MLT17 |  90.7 |  85.2 |  87.9 |  10.7 |  86.5 |  83.6 |  85.0 |  12.2 |  - |  - |  - |  - |  86.6 |  84.5 |  85.6 |  12.3
FSG (Tang et al., 2022) |  Synth |  90.7 |  85.7 |  88.1 |  - |  88.1 |  82.4 |  85.2 |  - |  91.1 |  86.7 |  88.8 |  - |  91.4 |  84.7 |  87.9 |  -
PAN (Wang et al., 2019c) |  Synth |  89.3 |  81.0 |  85.0 |  39.6 |  86.4 |  81.2 |  83.7 |  39.8 |  84.0 |  81.9 |  82.9 |  26.1 |  84.4 |  83.8 |  84.1 |  30.2
DB-R18 (Liao et al., 2020) |  Synth |  88.3 |  77.9 |  82.8 |  50.0 |  84.8 |  77.5 |  81.0 |  55.0 |  86.8 |  78.4 |  82.3 |  48.0 |  90.4 |  76.3 |  82.8 |  62.0
DB-R50 (Liao et al., 2020) |  Synth |  87.1 |  82.5 |  84.7 |  32.0 |  86.9 |  80.2 |  83.4 |  22.0 |  91.8 |  83.2 |  87.3 |  12.0 |  91.5 |  79.2 |  84.9 |  32.0
DB++-R18 (Liao et al., 2023) |  Synth |  87.4 |  79.6 |  83.3 |  48.0 |  84.3 |  81.0 |  82.6 |  49.0 |  90.1 |  77.2 |  83.1 |  44,0 |  87.9 |  82.5 |  85.1 |  55.0
DB++-R50 (Liao et al., 2023) |  Synth |  88.9 |  83.2 |  86.0 |  28.0 |  87.9 |  82.8 |  85.3 |  26.0 |  90.9 |  83.9 |  87.3 |  10.0 |  91.5 |  83.3 |  87.2 |  29.0
PAN (Wang et al., 2019c)* |  Synth |  88.5 |  81.7 |  85.0 |  41.3 |  86.0 |  81.0 |  83.4 |  40.4 |  86.6 |  79.7 |  83.0 |  31.5 |  85.7 |  83.4 |  84.5 |  43.3
DB-R18 (Liao et al., 2020)* |  Synth |  89.7 |  78.2 |  83.5 |  49.3 |  - |  - |  - |  - |  87.6 |  77.5 |  82.2 |  44.1 |  86.8 |  79.9 |  83.2 |  55.7
DB-R50 (Liao et al., 2020)* |  Synth |  87.6 |  81.2 |  84.3 |  26.4 |  - |  - |  - |  - |  90.9 |  84.2 |  87.4 |  10.2 |  91.7 |  81.1 |  86.1 |  29.4
DB++-R18 (Liao et al., 2023)* |  Synth |  89.0 |  78.0 |  83.1 |  44.1 |  - |  - |  - |  - |  90.0 |  77.2 |  83.1 |  40.1 |  89.6 |  81.3 |  85.2 |  48.8
DB++-R50 (Liao et al., 2023)* |  Synth |  89.2 |  82.8 |  85.9 |  24.8 |  - |  - |  - |  - |  90.7 |  83.8 |  87.1 |  9.7 |  92.4 |  83.2 |  87.5 |  27.7
SIR-R18 (Ours) |  Synth |  91.8 |  83.1 |  87.2 |  48.2 |  84.7 |  82.9 |  83.8 |  58.8 |  89.2 |  80.7 |  84.7 |  38.8 |  89.4 |  83.6 |  86.4 |  58.0
SIR-R50 (Ours) |  Synth |  90.9 |  85.6 |  88.2 |  30.9 |  87.4 |  83.7 |  85.5 |  38.7 |  90.4 |  85.4 |  87.8 |  10.1 |  93.6 |  86.0 |  89.6 |  36.9

Detecting curved words. We verify the ability of our method to detect curved
words on Total-Text. The test scale is set to 640. As listed in Table 2, on
Total-Text, SIR-R50 achieves the SOTA performance of 88.2% F-measure with 30.9
FPS which outperforms the recent SOTA methods BPN (Zhang et al., 2021) and FSG
(Tang et al., 2022) and runs much faster. Compared with BPN (Zhang et al.,
2021) and FSG (Tang et al., 2022) which involves several refinement stages,
SIR directly learns robust representation with auxiliary tasks without
burdening the inference speed. SIR could outperform DB++ (Liao et al., 2023)
on both accuracy and speed. With ResNet-18, SIR achieves 87.2% F-measure with
48.2 FPS, which is only a little slower than DB (Liao et al., 2020) with 49.3
FPS, but outperforms DB by a large margin of 3.7% in terms of F-measure,
demonstrating the superiority of SIR.

Detecting curved lines. We verify the ability of the proposed method to detect
curved lines on CTW1500. The short side is set to 512. As listed in Table 2,
on CTW1500, a competitive result of 85.5% with 38.7 FPS is achieved by
SIR-R50, which share the same performance with the SOTA methods FCENet (Zhu et
al., 2021) and outperform recently proposed FSG (Tang et al., 2022) and BPN
(Zhang et al., 2021). With ResNet-18, SIR archives 83.8% F-measure with 58.8
FPS, which is faster and better in terms of FPS and F-measure than PAN (Wang
et al., 2019c), DB (Liao et al., 2020), and DB++ (Liao et al., 2023), which
illustrate the ability on detecting long curve text.

Detecting oriented words. The test scale on ICDAR2015 is set to 736 for
SIR-R18 and 1152 for SIR-R50 following (Liao et al., 2023). As shown in Table
2, SIR achieves a competitive performance of $87.8\%$ F-measure, which is
comparable to FSG (Tang et al., 2022) and SD (Xiao et al., 2020) but inferior
to TextFuseNet (Ye et al., 2020). However, TextFuseNet uses character-level
supervision, which is not suitable for direct comparison. SIR also outperforms
other real-time works, e.g., PAN (Wang et al., 2019c), DB (Liao et al., 2020),
and DB++ (Liao et al., 2023). With ResNet-18, SIR gets 84.7% F-measure with
38.8 FPS, which is faster and better than PAN with 83.0 F-measure and 31.5
FPS. SIR runs as fast as DB++ but outperforms it by 1.6%, which illustrates
the ability of SIR in challenging blur text situations.

Detecting oriented lines. The long text is the common case in the natural
scene. We test the robustness of our models of detection of long text lines on
MSRA-TD500. The test scale is 640. As shown in Table 2, our proposed method
shows powerful ability in long text detection. SIR-R50 achieves 89.6%
F-measure with 36.9 FPS, outperforms all the previous methods, and boosts the
best performance on this dataset by $1.7\%$ in terms of F-measure, which
demonstrates that SIR can deal with large variations in scales and rotation
well. Note that even with a light backbone ResNet-18, SIR achieves 86.4
F-measure and the fastest speed of 58.0 FPS, already outperforming most
existing methods, which illustrates the ability of SIR on learning
discriminative features.

###  4.4. Ablation Study

The ablation study is conducted on an oriented text dataset ICDAR2015 and a
curved text dataset Total-Text to demonstrate the effectiveness of GDSC and
TDM. ResNet-18 is used as the backbone. The test scale is also set to 736 and
640 on the two datasets.

The effectiveness of GDSC/TDM. As shown in Table 3, we can infer that both
GDSC and TDM improve the performance of the baseline model on the two
datasets, even though the BU-SEG baseline is quite strong. GDSC achieves 1.6%
and 1.3% performance gain in terms of F-measure on the ICDAR2015 dataset and
the Total-Text dataset, which illustrate the effectiveness of GDSC. TDM brings
1.8% (on ICDAR2015) and 2.1% (on Total-Text) improvements, demonstrating the
superiority of TDM.

The complementarity between GDSC and TDM. As shown in Table 3, due to the
modeling from different perspectives, GDSC and TDM act in a complementary
manner. Specifically, when combining TDM, the model with GDSC is further
improved by 1.2% (on the ICDAR2015 dataset) and 1.5% (on the Total-Text
dataset); With GDSC, the baseline with TDM is increased by 1.0% and 0.7% on
the ICDAR2015 dataset and the Total-Text dataset respectively.

Table 3. Detection results with different settings of GDSC and TDM. “Seg” denotes the segmentation baseline.  Seg |  GDSC |  TDM | ICDAR2015 | Total-Text
---|---|---|---|---
|  |  |  P |  R |  F |  P |  R |  F
✓ |  $\times$ |  $\times$ |  85.0 |  77.8 |  81.2 |  88.8 |  79.2 |  83.7
✓ |  ✓ |  $\times$ |  87.9 |  78.2 |  82.8 |  87.0 |  83.2 |  85.0
✓ |  $\times$ |  ✓ |  84.6 |  81.5 |  83.0 |  89.4 |  82.5 |  85.8
✓ |  ✓ |  ✓ |  88.7 |  79.7 |  84.0 |  88.2 |  84.8 |  86.5
Figure 5. Visualization of the probability maps generated by the baseline
segmentation model (b) and SIR (c). (a) denotes the ground truth.

Comparison between the baseline and SIR. In Table 3, with GDSC and TDM, SIR
outperforms the baseline model by 2.8% on both the ICDAR2015 dataset and the
Total-Text dataset quantitatively in total. We further provide qualitative
results for better illustration. As shown in Fig. 5, the baseline model tends
to suffer from false positives and text adhesion issues, which alleviate the
robustness of the model. Due to auxiliary tasks of GDSC and TDM, which promote
the representation learning on semantic and instance level, SIR learns robust
representation and produce high-quality probability maps as well as accurate
detection.

The function of modules in TDM. We perform a detailed study on the modules of
TDM. The result is listed in Table 4. Several conclusions can be drawn: (1)
Effectiveness of dense anchor-based supervision. RPN brings 0.5% and 1.2%
improvements in terms of F-measure on the ICDAR2015 dataset and the Total-Text
dataset. (2) Both regional classification and regression help. When further
combining “Cls”, the model achieves 0.6% and 0.7% performance gain on the
ICDAR2015 dataset and the Total-Text dataset; “Reg” brings 0.8% and 0.7%
improvements on the ICDAR2015 dataset and the Total-Text dataset. (3) Regional
recognition tasks can be performed even with anchors. As shown in Table 4,
when RPN is removed in which the anchors are directly taken as proposals,
regional recognition tasks can bring 0.5% and 1.2% improvements on the
ICDAR2015 dataset and the Total-Text dataset.

Table 4. Detection results with different settings of TDM. “Cls” and “Reg” refer to the classification and regression tasks in regional recognition. Seg |  RPN |  Cls |  Reg | ICDAR2015 | Total-Text
---|---|---|---|---|---
|  |  |  |  P |  R |  F |  P |  R |  F
✓ |  $\times$ |  $\times$ |  $\times$ |  85.0 |  77.8 |  81.2 |  88.8 |  79.2 |  83.7
✓ |  ✓ |  $\times$ |  $\times$ |  85.2 |  78.5 |  81.7 |  89.7 |  80.6 |  84.9
✓ |  ✓ |  ✓ |  $\times$ |  86.9 |  78.2 |  82.3 |  89.9 |  81.8 |  85.6
✓ |  ✓ |  $\times$ |  ✓ |  85.9 |  79.3 |  82.5 |  86.9 |  84.3 |  85.6
✓ |  ✓ |  ✓ |  ✓ |  84.6 |  81.5 |  83.0 |  89.4 |  82.5 |  85.8
✓ |  $\times$ |  ✓ |  ✓ |  84.4 |  79.1 |  81.7 |  89.7 |  80.5 |  84.9

Evaluation of inference time and training time. As shown in Table 5, compared
to the baseline model, SIR not only brings no extra burden but also runs a
little faster than the baseline model during inference. We find it comes from
the reduction of the number of predicted boxes in the post-process, which also
demonstrates the improvement in the quality of the probability map through
robust representation learning. During training, SIR acts like a two-stage
detector, compared to the one-stage baseline model, which can be seen as
Faster R-CNN compared to RPN during training. SIR costs 0.72 seconds per
iteration against 0.47 seconds per iteration for the baseline model, which
brings about 50% extra training time and is acceptable in real applications
due to the efficiency of the baseline.

Table 5. Comparison of the number of processed boxes and the running time between the baseline and SIR. Method | Box number (per image) | Inference speed (FPS)
---|---|---
ICDAR2015 |  Total-Text |  ICDAR2015 |  Total-Text
Baseline |  12.4 |  10.8 |  38.1 |  47.4
SIR (Ours) |  10.8 |  9.8 |  38.8 |  48.2

###  4.5. Limitation

Our method shares the same limitation with existing BU-SEG methods (Liao et
al., 2023, 2020; Wang et al., 2019c, b). Text instances with a large overlap
in the text kernel will lead to a single connected component, which can not be
well detected by the proposed method.

##  5\. Conclusion

In this paper, we propose a robust real-time scene text detection method (SIR)
to improve the robustness of the existing encoder-decoder segmentation-based
framework. SIR originates from the perspective of representation learning and
consists of a unified BU-SEG baseline, GDSC, and TDM in which GDSC and TDM act
as auxiliary tasks that only work in training and are removed during
inference. GDSC learns the semantic representation by contrasting a global
semantic representation to a dense representation. TDM performs regional
recognition tasks for instance-level representation learning that bring strong
instance clues to the encoder. As a result, robust representations are learned
by the encoder, which effectively mitigates the text/non-text pixel
misclassification and false text instance discrimination issues in the
existing encoder-decoder framework. Experimental results on four public
datasets demonstrate the efficiency and robustness of the proposed method.

###### Acknowledgements.

Supported by the Key Research Program of Frontier Sciences, CAS, Grant NO
ZDBS-LY-7024.

## References

  * (1)
  * Baek et al. (2019) Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee. 2019.  Character region awareness for text detection. In _CVPR_. 9365–9374.
  * Chen et al. (2017) Liang-Chieh Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. 2017\.  Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected crfs.  _IEEE TPAMI_ 40, 4 (2017), 834–848.
  * Ch’ng and Chan (2017) Chee Kheng Ch’ng and Chee Seng Chan. 2017.  Total-text: A comprehensive dataset for scene text detection and recognition. In _ICDAR_. 935–942.
  * Dai et al. (2021) Pengwen Dai, Sanyi Zhang, Hua Zhang, and Xiaochun Cao. 2021\.  Progressive Contour Regression for Arbitrary-Shape Scene Text Detection. In _CVPR_. 7393–7402.
  * Deng et al. (2018) Dan Deng, Haifeng Liu, Xuelong Li, and Deng Cai. 2018\.  Pixellink: Detecting scene text via instance segmentation. In _AAAI_. 6773–6780.
  * Du et al. (2022) Yongkun Du, Zhineng Chen, Caiyan Jia, Xiaoting Yin, Tianlun Zheng, Chenxia Li, Yuning Du, and Yu-Gang Jiang. 2022\.  SVTR: Scene Text Recognition with a Single Visual Model. In _IJCAI_. 884–890.
  * Guo et al. (2021) Youhui Guo, Yu Zhou, Xugong Qin, and Weiping Wang. 2021\.  Which and Where to Focus: A Simple yet Accurate Framework for Arbitrary-Shaped Nearby Text Detection in Scene Images. In _ICANN_. 271–283.
  * Guo et al. (2022) Youhui Guo, Yu Zhou, Xugong Qin, Enze Xie, and Weiping Wang. 2022.  UNITS: Unsupervised Intermediate Training Stage for Scene Text Detection. In _ICME_. 1–6.
  * Gupta et al. (2016) Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman. 2016\.  Synthetic data for text localisation in natural images. In _CVPR_. 2315–2324.
  * He et al. (2017c) Dafang He, Xiao Yang, Chen Liang, Zihan Zhou, Alexander G Ororbi, Daniel Kifer, and C Lee Giles. 2017c.  Multi-scale FCN with cascaded instance aware segmentation for arbitrary oriented word spotting in the wild. In _CVPR_. 3519–3528.
  * He et al. (2017a) Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. 2017a.  Mask R-CNN. In _ICCV_. 2980–2988.
  * He et al. (2016) Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016\.  Deep residual learning for image recognition. In _CVPR_. 770–778.
  * He et al. (2021) Minghang He, Minghui Liao, Zhibo Yang, Humen Zhong, Jun Tang, Wenqing Cheng, Cong Yao, Yongpan Wang, and Xiang Bai. 2021.  MOST: A Multi-Oriented Scene Text Detector with Localization Refinement. In _CVPR_. 8813–8822.
  * He et al. (2017b) Pan He, Weilin Huang, Tong He, Qile Zhu, Yu Qiao, and Xiaolin Li. 2017b.  Single shot text detector with regional attention. In _ICCV_. 3047–3055.
  * He et al. (2017d) Wenhao He, Xu-Yao Zhang, Fei Yin, and Cheng-Lin Liu. 2017d.  Deep direct regression for multi-oriented scene text detection. In _ICCV_. 745–753.
  * Hu et al. (2020) Wenyang Hu, Xiaocong Cai, Jun Hou, Shuai Yi, and Zhiping Lin. 2020.  Gtc: Guided training of ctc towards efficient and accurate scene text recognition. In _AAAI_. 11005–11012.
  * Huang et al. (2014) Weilin Huang, Yu Qiao, and Xiaoou Tang. 2014.  Robust scene text detection with convolution neural network induced mser trees. In _ECCV_. 497–511.
  * Jia et al. (2016) Xu Jia, Bert De Brabandere, Tinne Tuytelaars, and Luc V Gool. 2016.  Dynamic filter networks. In _NeurIPS_. 667–675.
  * Karatzas et al. (2015) Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al. 2015\.  ICDAR 2015 competition on robust reading. In _ICDAR_. 1156–1160.
  * Liao et al. (2021) Minghui Liao, Pengyuan Lyu, Minghang He, Cong Yao, Wenhao Wu, and Xiang Bai. 2021\.  Mask textSpotter: An end-to-end trainable neural network for spotting text with arbitrary shapes.  _IEEE TPAMI_ (2021), 532–548.
  * Liao et al. (2018a) Minghui Liao, Baoguang Shi, and Xiang Bai. 2018a.  Textboxes++: A single-shot oriented scene text detector.  _IEEE TIP_ (2018), 3676–3690.
  * Liao et al. (2020) Minghui Liao, Zhaoyi Wan, Cong Yao, Kai Chen, and Xiang Bai. 2020.  Real-time scene text detection with differentiable binarization. In _AAAI_. 11474–11481.
  * Liao et al. (2018b) Minghui Liao, Zhen Zhu, Baoguang Shi, Gui-song Xia, and Xiang Bai. 2018b.  In _CVPR_. 5909–5918.
  * Liao et al. (2023) Minghui Liao, Zhisheng Zou, Zhaoyi Wan, Cong Yao, and Xiang Bai. 2023.  Real-time scene text detection with differentiable binarization and adaptive scale fusion.  _IEEE TPAMI_ 45, 1 (2023), 919–931.
  * Lin et al. (2017) Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. 2017\.  Feature pyramid networks for object detection. In _CVPR_. 2117–2125.
  * Liu et al. (2016) Wei Liu, Dragomir Anguelov, Dumitru Erhan, Christian Szegedy, Scott Reed, Cheng-Yang Fu, and Alexander C Berg. 2016.  SSD: Single shot multibox detector. In _ECCV_. 21–37.
  * Liu et al. (2020a) Yuliang Liu, Hao Chen, Chunhua Shen, Tong He, Lianwen Jin, and Liangwei Wang. 2020a.  ABCNet: Real-time scene text spotting with adaptive bezier-curve network. In _CVPR_. 9806–9815.
  * Liu and Jin (2017) Yuliang Liu and Lianwen Jin. 2017.  Deep matching prior network: Toward tighter multi-oriented text detection. In _CVPR_. 3454–3461.
  * Liu et al. (2020b) Yuliang Liu, Lianwen Jin, and Chuanming Fang. 2020b.  Arbitrarily shaped scene text detection with a mask tightness text detector.  _IEEE TIP_ (2020), 2918–2930.
  * Liu et al. (2019a) Yuliang Liu, Lianwen Jin, Shuaitao Zhang, Canjie Luo, and Sheng Zhang. 2019a.  Curved scene text detection via transverse and longitudinal sequence connection.  _PR_ (2019), 337–345.
  * Liu et al. (2019c) Yuliang Liu, Sheng Zhang, Lianwen Jin, Lele Xie, Yaqiang Wu, and Zhepeng Wang. 2019c.  Omnidirectional scene text detection with sequential-free box discretization. In _IJCAI_. 3052–3058.
  * Liu et al. (2019b) Zichuan Liu, Guosheng Lin, Sheng Yang, Fayao Liu, Weisi Lin, and Wang Ling Goh. 2019b.  Towards robust curve text detection with conditional spatial expansion. In _CVPR_. 7269–7278.
  * Long et al. (2015) Jonathan Long, Evan Shelhamer, and Trevor Darrell. 2015\.  Fully convolutional networks for semantic segmentation. In _CVPR_. 3431–3440.
  * Long et al. (2021) Shangbang Long, Xin He, and Cong Yao. 2021.  Scene Text Detection and Recognition: The Deep Learning Era.  _IJCV_ 129, 1 (2021), 161–184.
  * Long et al. (2022) Shangbang Long, Siyang Qin, Dmitry Panteleev, Alessandro Bissacco, Yasuhisa Fujii, and Michalis Raptis. 2022.  Towards end-to-end unified scene text detection and layout analysis. In _CVPR_. 1049–1059.
  * Long et al. (2018) Shangbang Long, Jiaqiang Ruan, Wenjie Zhang, Xin He, Wenhao Wu, and Cong Yao. 2018\.  Textsnake: A flexible representation for detecting text of arbitrary shapes. In _ECCV_. 20–36.
  * Lyu et al. (2018) Pengyuan Lyu, Cong Yao, Wenhao Wu, Shuicheng Yan, and Xiang Bai. 2018.  Multi-oriented scene text detection via corner localization and region segmentation. In _CVPR_. 7553–7563.
  * Ma et al. (2021) Chixiang Ma, Lei Sun, Zhuoyao Zhong, and Qiang Huo. 2021\.  ReLaText: Exploiting visual relationships for arbitrary-shaped scene text detection with graph convolutional networks.  _PR_ (2021), 337–345.
  * Ma et al. (2018) Jianqi Ma, Weiyuan Shao, Hao Ye, Li Wang, Hong Wang, Yingbin Zheng, and Xiangyang Xue. 2018.  Arbitrary-oriented scene text detection via rotation proposals.  _IEEE TMM_ (2018), 3111–3122.
  * Milletari et al. (2016) Fausto Milletari, Nassir Navab, and Seyed-Ahmad Ahmadi. 2016\.  V-net: Fully convolutional neural networks for volumetric medical image segmentation. In _3DV_. 565–571.
  * Oord et al. (2018) Aaron van den Oord, Yazhe Li, and Oriol Vinyals. 2018.  Representation learning with contrastive predictive coding.  _arXiv preprint arXiv:1807.03748_ (2018).
  * Qiao et al. (2020a) Zhi Qiao, Xugong Qin, Yu Zhou, Fei Yang, and Weiping Wang. 2020a.  Gaussian Constrained Attention Network for Scene Text Recognition. In _ICPR_. 3328–3335.
  * Qiao et al. (2021) Zhi Qiao, Yu Zhou, Jin Wei, Wei Wang, Yuan Zhang, Ning Jiang, Hongbin Wang, and Weiping Wang. 2021.  PIMNet: A Parallel, Iterative and Mimicking Network for Scene Text Recognition. In _ACM MM_. 2046–2055.
  * Qiao et al. (2020b) Zhi Qiao, Yu Zhou, Dongbao Yang, Yucan Zhou, and Weiping Wang. 2020b.  SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition. In _CVPR_. 13525–13534.
  * Qin et al. (2021b) Xugong Qin, Yu Zhou, Youhui Guo, Dayan Wu, Zhihong Tian, Ning Jiang, Hongbin Wang, and Weiping Wang. 2021b.  Mask is all you need: Rethinking mask R-CNN for dense and arbitrary-shaped scene text detection. In _ACM MM_. 414–423.
  * Qin et al. (2021a) Xugong Qin, Yu Zhou, Youhui Guo, Dayan Wu, and Weiping Wang. 2021a.  FC${}^{\mbox{2}}$RN: A Fully Convolutional Corner Refinement Network for Accurate Multi-Oriented Scene Text Detection. In _ICASSP_. 4350–4354.
  * Qin et al. (2019) Xugong Qin, Yu Zhou, Dongbao Yang, and Weiping Wang. 2019\.  Curved Text Detection in Natural Scene Images with Semi- and Weakly-Supervised Learning. In _ICDAR_. 559–564.
  * Ren et al. (2015) Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2015\.  Faster R-CNN: Towards real-time object detection with region proposal networks. In _NeurIPS_. 91–99.
  * Shen et al. (2023) Huawen Shen, Xiang Gao, Jin Wei, Liang Qiao, Yu Zhou, Qiang Li, and Zhanzhan Cheng. 2023\.  Divide Rows and Conquer Cells: Towards Structure Recognition for Large Tables. In _IJCAI_.
  * Sheng et al. (2021) Tao Sheng, Jie Chen, and Zhouhui Lian. 2021.  Centripetaltext: An efficient text instance representation for scene text detection. In _NeurIPS_. 335–346.
  * Shi et al. (2017a) Baoguang Shi, Xiang Bai, and Serge Belongie. 2017a.  Detecting oriented text in natural images by linking segments. In _CVPR_. 2550–2558.
  * Shi et al. (2017b) Baoguang Shi, Xiang Bai, and Cong Yao. 2017b.  An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition.  _IEEE TPAMI_ 39, 11 (2017), 2298–2304.
  * Shi et al. (2019) Baoguang Shi, Mingkun Yang, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai. 2019\.  Aster: An attentional scene text recognizer with flexible rectification.  _IEEE TPAMI_ 41, 9 (2019), 2035–2048.
  * Song et al. (2022) Sibo Song, Jianqiang Wan, Zhibo Yang, Jun Tang, Wenqing Cheng, Xiang Bai, and Cong Yao. 2022.  Vision-language pre-training for boosting scene text detectors. In _CVPR_. 15681–15691.
  * Tang et al. (2019) Jun Tang, Zhibo Yang, Yongpan Wang, Qi Zheng, Yongchao Xu, and Xiang Bai. 2019\.  SegLink++: Detecting dense and arbitrary-shaped scene text by instance-aware component grouping.  _PR_ (2019), 106954\.
  * Tang et al. (2022) Jingqun Tang, Wenqing Zhang, Hongye Liu, Mingkun Yang, Bo Jiang, Guanglong Hu, and Xiang Bai. 2022.  Few Could Be Better Than All: Feature Sampling and Grouping for Scene Text Detection. In _CVPR_. IEEE.
  * Tian et al. (2016) Zhi Tian, Weilin Huang, Tong He, Pan He, and Yu Qiao. 2016.  Detecting text in natural image with connectionist text proposal network. In _ECCV_. 56–72.
  * Tian et al. (2019) Zhuotao Tian, Michelle Shu, Pengyuan Lyu, Ruiyu Li, Chao Zhou, Xiaoyong Shen, and Jiaya Jia. 2019.  Learning shape-aware embedding for scene text detection. In _CVPR_. 4234–4243.
  * Vatti (1992) Bala R Vatti. 1992\.  A generic solution to polygon clipping.  _Commun. ACM_ 35, 7 (1992), 56–63.
  * Wan et al. (2021) Qi Wan, Haoqin Ji, and Linlin Shen. 2021.  Self-attention based text knowledge mining for text detection. In _CVPR_. 5983–5992.
  * Wang et al. (2020a) Fangfang Wang, Yifeng Chen, Fei Wu, and Xi Li. 2020a.  TextRay: Contour-based geometric modeling for arbitrary-shaped scene text detection. In _ACM MM_. 111–119.
  * Wang et al. (2018) Fangfang Wang, Liming Zhao, Xi Li, Xinchao Wang, and Dacheng Tao. 2018.  Geometry-aware scene text detection with instance transformation network. In _CVPR_. 1381–1389.
  * Wang et al. (2022a) Hao Wang, Junchao Liao, Tianheng Cheng, Zewen Gao, Hao Liu, Bo Ren, Xiang Bai, and Wenyu Liu. 2022a.  Knowledge mining with scene text for fine-grained recognition. In _CVPR_. 4624–4633.
  * Wang et al. (2019d) Pengfei Wang, Chengquan Zhang, Fei Qi, Zuming Huang, Mengyi En, Junyu Han, Jingtuo Liu, Errui Ding, and Guangming Shi. 2019d.  A single-shot arbitrarily-shaped text detector based on context attended multi-task learning. In _ACM MM_. 1277–1285.
  * Wang et al. (2019b) Wenhai Wang, Enze Xie, Xiang Li, Wenbo Hou, Tong Lu, Gang Yu, and Shuai Shao. 2019b.  Shape robust text detection with progressive scale expansion network. In _CVPR_. 9336–9345.
  * Wang et al. (2019c) Wenhai Wang, Enze Xie, Xiaoge Song, Yuhang Zang, Wenjia Wang, Tong Lu, Gang Yu, and Chunhua Shen. 2019c.  Efficient and accurate arbitrary-shaped text detection with pixel aggregation network. In _ICCV_. 8440–8449.
  * Wang et al. (2022c) Wei Wang, Yu Zhou, Jiahao Lv, Dayan Wu, Guoqing Zhao, Ning Jiang, and Weipinng Wang. 2022c.  Tpsnet: Reverse thinking of thin plate splines for arbitrary shape scene text representation. In _ACM MM_. 5014–5025.
  * Wang et al. (2019a) Xiaobing Wang, Yingying Jiang, Zhenbo Luo, Cheng-Lin Liu, Hyunsoo Choi, and Sungjin Kim. 2019a.  Arbitrary shape scene text detection with adaptive text region representation. In _CVPR_. 6449–6458.
  * Wang et al. (2020c) Xinlong Wang, Rufeng Zhang, Tao Kong, Lei Li, and Chunhua Shen. 2020c.  Solov2: Dynamic and fast instance segmentation.  _NeurIPS_ (2020), 17721–17732.
  * Wang et al. (2022b) Yuxin Wang, Hongtao Xie, Mengting Xing, Jing Wang, Shenggao Zhu, and Yongdong Zhang. 2022b.  Detecting Tampered Scene Text in the Wild. In _ECCV_. 215–232.
  * Wang et al. (2020b) Yuxin Wang, Hongtao Xie, Zheng-Jun Zha, Mengting Xing, Zilong Fu, and Yongdong Zhang. 2020b.  ContourNet: Taking a further step toward accurate arbitrary-shaped scene text detection. In _CVPR_. 11750–11759.
  * Wei et al. (2022) Jin Wei, Yuan Zhang, Yu Zhou, Gangyan Zeng, Zhi Qiao, Youhui Guo, Haiying Wu, Hongbin Wang, and Weiping Wang. 2022.  TextBlock: Towards Scene Text Spotting without Fine-Grained Detection. In _ACM MM_. 5892–5902.
  * Wu and Natarajan (2017) Yue Wu and Prem Natarajan. 2017.  Self-organized text detection with minimal post-processing via border learning. In _ICCV_. 5000–5009.
  * Xiao et al. (2020) Shanyu Xiao, Liangrui Peng, Yan Ruijie, An Keyu, Yao Gang, and Min Jaesik. 2020\.  Sequential deformation for accurate scene text detection. In _ECCV_. 108–124.
  * Xie et al. (2019) Enze Xie, Yuhang Zang, Shuai Shao, Gang Yu, Cong Yao, and Guangyao Li. 2019\.  Scene text detection with supervised pyramid context network. In _AAAI_. 9038–9045.
  * Xing et al. (2019) Linjie Xing, Zhi Tian, Weilin Huang, and Matthew R Scott. 2019\.  Convolutional character networks. In _ICCV_. 9126–9136.
  * Xu et al. (2019) Yongchao Xu, Yukang Wang, Wei Zhou, Yongpan Wang, Zhibo Yang, and Xiang Bai. 2019\.  TextField: Learning a deep direction field for irregular scene text detection.  _IEEE TIP_ (2019), 5566–5579.
  * Xue et al. (2018) Chuhui Xue, Shijian Lu, and Fangneng Zhan. 2018.  Accurate scene text detection through border semantics awareness and bootstrapping. In _ECCV_. 355–372.
  * Xue et al. (2019) Chuhui Xue, Shijian Lu, and Wei Zhang. 2019.  MSR: multi-scale shape regression for scene text detection. In _IJCAI_. 989–995.
  * Xue et al. (2022) Chuhui Xue, Wenqing Zhang, Yu Hao, Shijian Lu, Philip HS Torr, and Song Bai. 2022\.  Language matters: A weakly supervised vision-language pre-training approach for scene text detection and spotting. In _ECCV_. 284–302.
  * Yang et al. (2018) Qiangpeng Yang, Mengli Cheng, Wenmeng Zhou, Yan Chen, Minghui Qiu, and Wei Lin. 2018\.  Inceptext: A new inception-text module with deformable PSROI pooling for multi-oriented scene text detection. In _IJCAI_. 1071–1077.
  * Yao et al. (2014a) Cong Yao, Xiang Bai, and Wenyu Liu. 2014a.  A unified framework for multioriented text detection and recognition.  _IEEE TIP_ (2014), 4737–4749.
  * Yao et al. (2012) Cong Yao, Xiang Bai, Wenyu Liu, Yi Ma, and Zhuowen Tu. 2012.  Detecting texts of arbitrary orientations in natural images. In _CVPR_. 1083–1090.
  * Yao et al. (2016) Cong Yao, Xiang Bai, Nong Sang, Xinyu Zhou, Shuchang Zhou, and Zhimin Cao. 2016\.  Scene text detection via holistic, multi-channel prediction.  _arXiv preprint arXiv:1606.09002_ (2016).
  * Yao et al. (2014b) Cong Yao, Xiang Bai, Baoguang Shi, and Wenyu Liu. 2014b.  Strokelets: A learned multi-scale representation for scene text recognition. In _CVPR_. 4042–4049.
  * Ye et al. (2020) Jian Ye, Zhe Chen, Juhua Liu, and Bo Du. 2020\.  TextFuseNet: Scene text detection with richer fused features. In _IJCAI_. 516–522.
  * Ye and Doermann (2015) Qixiang Ye and David S. Doermann. 2015.  Text Detection and Recognition in Imagery: A Survey.  _IEEE TPAMI_ 37, 7 (2015), 1480–1500.
  * Yu et al. (2023) Wenwen Yu, Yuliang Liu, Wei Hua, Deqiang Jiang, Bo Ren, and Xiang Bai. 2023\.  Turning a CLIP Model into a Scene Text Detector.  _CVPR_ , 6978–6988.
  * Zeng et al. (2021) Gangyan Zeng, Yuan Zhang, Yu Zhou, and Xiaomeng Yang. 2021\.  Beyond OCR + VQA: Involving OCR into the Flow for Robust and Accurate TextVQA. In _ACM MM_. 376–385.
  * Zeng et al. (2023) Gangyan Zeng, Yuan Zhang, Yu Zhou, Xiaomeng Yang, Ning Jiang, Guoqing Zhao, Weiping Wang, and Xu-Cheng Yin. 2023\.  Beyond OCR+ VQA: Towards end-to-end reading and reasoning for robust and accurate textvqa.  _PR_ 138 (2023), 109337.
  * Zhang et al. (2019) Chengquan Zhang, Borong Liang, Zuming Huang, Mengyi En, Junyu Han, Errui Ding, and Xinghao Ding. 2019.  Look more than once: An accurate detector for text of arbitrary shapes. In _CVPR_. 10552–10561.
  * Zhang et al. (2020a) Shifeng Zhang, Cheng Chi, Yongqiang Yao, Zhen Lei, and Stan Z. Li. 2020a.  Bridging the gap between anchor-based and anchor-free detection via adaptive training sample selection. In _CVPR_. 9756–9765.
  * Zhang et al. (2020b) Shi-Xue Zhang, Xiaobin Zhu, Jie-Bo Hou, Chang Liu, Chun Yang, Hongfa Wang, and Xu-Cheng Yin. 2020b.  Deep relational reasoning graph network for arbitrary shape text detection. In _CVPR_. 9696–9705.
  * Zhang et al. (2021) Shi-Xue Zhang, Xiaobin Zhu, Chun Yang, Hongfa Wang, and Xu-Cheng Yin. 2021.  Adaptive Boundary Proposal Network for Arbitrary Shape Text Detection. In _ICCV_. 1305–1314.
  * Zhang et al. (2016) Zheng Zhang, Chengquan Zhang, Wei Shen, Cong Yao, Wenyu Liu, and Xiang Bai. 2016\.  Multi-oriented text detection with fully convolutional networks. In _CVPR_. 4159–4167.
  * Zhou et al. (2017) Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang. 2017.  EAST: An efficient and accurate scene text detector. In _CVPR_. 2642–2651.
  * Zhou et al. (2020) Yu Zhou, Hongtao Xie, Shancheng Fang, Yan Li, and Yongdong Zhang. 2020.  CRNet: A center-aware representation for detecting text of arbitrary shapes. In _ACM MM_. 2571–2580.
  * Zhu et al. (2019) Xizhou Zhu, Han Hu, Stephen Lin, and Jifeng Dai. 2019\.  Deformable convnets v2: More deformable, better results. In _CVPR_. 9308–9316.
  * Zhu et al. (2021) Yiqin Zhu, Jianyong Chen, Lingyu Liang, Zhanghui Kuang, Lianwen Jin, and Wayne Zhang. 2021\.  Fourier Contour Embedding for Arbitrary-Shaped Text Detection. In _CVPR_. 3123–3131.
  * Zhu et al. (2016) Yingying Zhu, Cong Yao, and Xiang Bai. 2016.  Scene text detection and recognition: Recent advances and future trends.  _Frontiers of Computer Science_ 10 (2016), 19–36.

##  Appendix A More Details in Implementation

###  A.1. The Architecture of Encoder

The architecture of the encoder of SIR is exactly the same as that in (Liao et
al., 2023), which consists of the backbone part and the neck part as shown in
Fig. 6.

Figure 6. The architecture of the encoder. “s” and “p” denote stride and
padding of the convolution.

Backbone. Following (Liao et al., 2023) and (Zhu et al., 2019), modulated
deformable convolutions are applied in all the $3\times 3$ convolutional
layers in stages conv3, conv4, and conv5 in the ResNet backbone (He et al.,
2016).

Neck. The neck includes a “concat” FPN (Lin et al., 2017; Liao et al., 2020)
and the adaptive scale fusion (ASF) (Liao et al., 2023). Firstly, the input
image is fed into an FPN to obtain features. Secondly, the pyramid features
are up-sampled to the same scale and cascaded. At last, the cascaded features
are fed into the ASF module (Liao et al., 2023) to produce the encoded feature
$F_{enc}$.

###  A.2. Anchor Setting in TDM

To handle the scale and aspect ratio variation in TDM, we construct a feature
pyramid from $F_{enc}$ with four max-pooling operations, resulting in a
feature pyramid of {$F_{enc}\in{\mathbb{R}}^{H\times W\times C}$,
$F_{enc}^{1}\in{\mathbb{R}}^{\frac{H}{2}\times\frac{W}{2}\times C}$,
$F_{enc}^{2}\in{\mathbb{R}}^{\frac{H}{4}\times\frac{W}{4}\times C}$,
$F_{enc}^{3}\in{\mathbb{R}}^{\frac{H}{8}\times\frac{W}{8}\times C}$,
$F_{enc}^{4}\in{\mathbb{R}}^{\frac{H}{16}\times\frac{W}{16}\times C}$}.
Vectors from different pyramids naturally encode different amounts of
information on the input image, which can be associated with anchors with
different scales. The aspect ratios of anchors per location are set to
$\\{\frac{1}{5},\frac{1}{2},1,2,5\\}$ (Xie et al., 2019; Liu et al., 2019c)
since text instances are usually long.

###  A.3. Label Generation

The label generation for the probability map follows PSENet (Wang et al.,
2019b) and DB (Liao et al., 2020). Given a text image, each polygon of text
instance is described by a set of segments:

(13) $G=\\{S_{k}\\}_{k=1}^{n},$

$n$ is the number of vertexes, which may vary according to different
instances. Then the kernel region is generated by shrinking the polygon $G$ to
$G_{s}$ using the Vatti clipping algorithm (Vatti, 1992). The offset $D$ of
shrinking is computed from the perimeter $L$ and area $A$ of the original
polygon:

(14) $D=\frac{A(1-r^{2})}{L},$

where $r$ is the shrink ratio and is set to 0.6 empirically. The border region
is obtained by $G-G_{s}$, which is the area between the kernel region and the
non-text region.

###  A.4. Inference

In the inference period, the probability map is used to generate text-bounding
boxes. The post-processing consists of three steps: (1) the probability map is
firstly binarized with a constant threshold (0.65 by default) to get the
binary map; (2) the connected regions (shrunk text regions) are obtained from
the binary map by CCL; (3) the shrunk regions are dilated with an offset
$D^{{}^{\prime}}$ in the Vatti clipping algorithm. $D^{{}^{\prime}}$ is
calculated as:

(15) $D^{{}^{\prime}}=\frac{A^{{}^{\prime}}\times
r^{{}^{\prime}}}{L^{{}^{\prime}}},$

where $A^{{}^{\prime}}$ is the area of the shrunk polygon; $L^{{}^{\prime}}$
is the perimeter of the shrunk polygon; $r^{{}^{\prime}}$ is set to 1.3 for
oriented datasets and 1.5 for curved datasets empirically. Especially, the
unclip ratio is set to 0 when performing text region segmentation as the
setting # 5 in Table 1. When evaluating the speed, we follow the way of most
existing methods, in which the inference time is averaged on the whole
dataset.

##  Appendix B More Experimental Results

We provide additional ablation and generalization experiments to further
demonstrate the superiority of the proposed method.

###  B.1. Ablation Experiments

The comparison of PAN (Wang et al., 2019c) loss and the proposed GDSC. The PAN
loss (Wang et al., 2019c) focuses on instance-level embedding learning which
consists of pulling features from the same instance and pushing features from
different instances. It is quite different from GDSC which aggregates features
within the same semantic category (text) and pushes text features away from
non-text that is consistent with the target of probability map learning. We
supplement the result with the PAN loss on the ICDAR2015 dataset based on
Table 3, leading to Table 6. As shown in the table, when the PAN_loss
(instance embedding loss) is equipped with the segmentation baseline, a direct
decrease of 3.1% F-measure is obtained. On the contrary, GDSC brings a 1.6%
performance gain in terms of F-measure, which illustrate the superiority of
the proposed GDSC. Compared with GDSC, PAN loss may fall behind for two
reasons: 1) absence of consideration on text/non-text contrast; 2) the
discrimination of different text instances is contradictory to text/non-text
discrimination.

Table 6. Detection results on ICDAR2015 with different model settings. “P”, “R”, and “F” denote precision, recall, and F-measure respectively. Method |  P |  R |  F
---|---|---|---
Seg |  85.0 |  77.8 |  81.2
Seg + GDSC |  87.9 |  78.2 |  82.8
Seg + PAN_loss |  85.8 |  71.6 |  78.1

Does the mask branch (He et al., 2017a) work in TDM? We tried the mask branch
but found it does not work well. Here we provide the results on ICDAR2015, in
which the settings maintain the same as those in Table 7 except for model
settings. As shown in Table 7, we can infer several points. Comparing the
first two rows, we find that using the mask branch leads to a decrease of 3.5%
F-measure. Compared to the last two rows, adding the mask branch to TDM also
brings no performance gains. We think the reason lies in two points: 1) the
use of the mask branch is to provide the pixel-level representation of
arbitrary-shape objects. However, via the main semantic segmentation task, the
network has encoded this type of information; 2) the mask branch
differentiates pixels as the foreground/background in a given bounding box.
When multiple text instances lie in one box, only one instance is taken as the
positive, which is opposite to the goal of semantic segmentation.

Table 7. Detection results on ICDAR2015 with different settings of TDM. “Cls” and “Reg” refer to the classification and regression tasks in regional recognition. “Mask” refers to the instance segmentation task as that in Mask R-CNN (He et al., 2017a). Seg |  RPN |  Cls |  Reg |  Mask |  P |  R |  F
---|---|---|---|---|---|---|---
✓ |  $\times$ |  $\times$ |  $\times$ |  $\times$ |  85.0 |  77.8 |  81.2
✓ |  $\times$ |  $\times$ |  $\times$ |  ✓ |  84.8 |  71.7 |  77.7
✓ |  ✓ |  ✓ |  ✓ |  $\times$ |  84.6 |  81.5 |  83.0
✓ |  ✓ |  ✓ |  ✓ |  ✓ |  83.8 |  82.1 |  83.0

###  B.2. Generalization Experiments

Generalization experiments of GDSC and TDM on other BU-SEG STD methods. We
have supplemented further experimental results with three SOTA methods upon
the corresponding official implementation including
DB++/DB333https://github.com/MhLiao/DB and
PSENet444https://github.com/whai362/pan_pp.pytorch.git, which demonstrate good
generalization ability of the proposed method in this subsection.

Experiments on DB++ (Liao et al., 2023)/DB (Liao et al., 2020) . We follow the
ablation studies in DB++/DB experimenting on the MSRA-TD500 dataset and
pretraining the network with SynthText. The test scale is set to 736. As shown
in Table 8 and Table 9, the proposed GSDC and TDM consistently improve the
DB++/DB baseline. Specifically, GDSC, TDM, and SIR (GDSC + TDM) respectively
bring 0.6%, 0.5%, and 1.1% upon DB++ and 0.6%, 0.7%, and 1.1% upon DB in terms
of F-measure.

Table 8. Detection results on MSRA-TD500 with different model settings upon DB++. Method |  P |  R |  F
---|---|---|---
DB++ (Paper) |  87.9 |  82.5 |  85.1
DB++ |  89.6 |  81.3 |  85.2
DB++ + GDSC |  88.8 |  83.0 |  85.8
DB++ + TDM |  88.2 |  83.3 |  85.7
DB++ + GDSC + TDM |  88.4 |  84.4 |  86.3
Table 9. Detection results on MSRA-TD500 with different model settings upon DB. Method |  P |  R |  F
---|---|---|---
DB (Paper) |  90.4 |  76.3 |  82.8
DB |  86.8 |  79.9 |  83.2
DB + GDSC |  88.5 |  79.6 |  83.8
DB + TDM |  88.7 |  79.6 |  83.9
DB + GDSC + TDM |  86.7 |  82.0 |  84.3
Table 10. Detection results on ICDAR2015 with different model settings upon PSENet. Method |  P |  R |  F
---|---|---|---
PSENet (Official reproduction) |  83.6 |  74.0 |  78.5
PSENet |  83.2 |  74.3 |  78.5
PSENe + GDSC |  82.8 |  74.0 |  78.2
PSENe + TDM |  86.4 |  79.5 |  82.8
PSENe + GDSC + TDM |  88.9 |  79.1 |  83.7

Experiments on PSENet (Wang et al., 2019b). The experiments are performed on
the ICDAR2015 dataset without pretraining. The short side is set to 736 pixels
during evaluation. As shown in Table 10, several interesting points can be
concluded: (1) Only using GDSC doesn’t help PSENet; (2) TDM brings a
significant increment of 4.3% F-measure upon the baseline; (3) Adding GDSC and
TDM together brings additional 0.9% than TDM only. We find the phenomenons
originate from the specific designs of PSENet which learn muti-scale text
kernels and use progressive scale expansion for post-processing. The reason
for point (1) is that learning multi-scale kernels may introduce fussy
semantics of foreground/background for GDSC. For point (2), besides the errors
from learned multi-scale kernels, the progressive scale expansion further
amplifies the effect due to the accumulated errors, making PSENet benefit
greatly from the top-down instance-level information brought by TDM. Point (3)
demonstrates the complementarity of GDSC and TDM; GDSC performs as a global
semantic regularizer for TDM.

##  Appendix C More Visualization Results

In this section, for better illustration, we provide more visualization
results including the BU-SEG baseline (Fig. 7 and 8), and comparison of the
baseline and SIR (Fig. 9), and more detection results (Fig. 10).

###  C.1. The BU-SEG Baseline

As shown in Fig. 7 and 8, among settings from #1 to #5 in Table 1, #4 which
performs a three-category classification weighted by class-aware coefficients
performs best. We can find that the kernel boundary is not entirely clear in
#2 due to an imbalance of samples around the kernel boundary. When comparing
#3 to #2, the weights of samples in the border region are raised to the same
level as the kernel region, which leads to a more balanced learning process
and better kernel boundary. When comparing #4 to #3, the border and non-text
classes are decoupled to be supervised independently, which introduces
explicit semantics and makes the kernel boundary more smooth. Looking at the
results of #5, which performs text segmentation, we find the learned
probability maps are also reasonable, demonstrating the generalization of the
proposed BU-SEG segmentation baseline.

Figure 7. Heatmaps generated with the probability maps and the original images
on the ICDAR2015 dataset. #1-#5 rows are the corresponding results produced by
the settings #1-#5 in Table 1 of the main body. #1: “kernel”; #2:
“kernel+W_kn”; #3: “kernel+W_kbn”; #4: “kernel+W_kbn+border”; #5:
“text+W_kbn”. Figure 8. Heatmaps generated with the probability maps and the
original images on the Total-Text dataset. #1-#5 rows are the corresponding
results produced by the settings #1-#5 in Table 1 of the main body. #1:
“kernel”; #2: “kernel+W_kn”; #3: “kernel+W_kbn”; #4: “kernel+W_kbn+border”;
#5: “text+W_kbn”.

###  C.2. Comparison of the Baseline and SIR

In this section, we provide more samples of the probability maps produced by
the baseline model and SIR on the ICDAR2015 and Total-Text datasets. As shown
in Fig. 9, with the proposed auxiliary modules and tasks, the prediction
results are more robust against false positives and text adhesion. The network
learns more discriminative features on text/non-text classification and nearby
text discrimination. SIR improves the quality of the probability maps, which
can produce more robust detection results.

Figure 9. Visualization results of the probability maps on ICDAR2015 and
Total-Text. (b) and (c) denote the probability maps generated by the baseline
segmentation model and SIR respectively; (a) denotes the ground truth. SIR
improves the quality of the probability map, which can produce more robust
detection results.

###  C.3. Detection Results

As shown in Fig. 10, we provide more qualitative results of SIR in different
cases on the Total-Text, CTW1500, ICDAR2015, and MSRA-TD500 datasets,
including multi-oriented text, long text, multi-lingual text, low-resolution
text, curved text, and dense text, which illustrate the robustness of SIR on
detecting various kinds of scene text.

Figure 10. Qualitative results of SIR in different cases on the Total-Text,
CTW1500, ICDAR2015, and MSRA-TD500 datasets, including multi-oriented text,
long text, multi-lingual text, low-resolution text, curved text, and dense
text. The green and yellow boxes denote the detection results and the ground
truth respectively. The cyan color represent “do-not-care” region.