# A Bilingual, Open World Video Text Dataset and End-to-end Video Text Spotter
with Transformer

Weijia Wu
Zhejiang University
${{\tt<EMAIL_ADDRESS>
&Yuanqiang Cai∗
Beijing University of Posts and Telecommunications
${{\tt<EMAIL_ADDRESS>
Debing Zhang
Kuaishou Technology
&Sibo Wang
Kuaishou Technology
&Zhuang Li
Kuaishou Technology
&Jiahong Li
Kuaishou Technology
&Yejun Tang
Kuaishou Technology
&Hong Zhou
Zhejiang University
Equal contribution.This work was done when Weijia Wu were interns in MMU,
Kuaishou Technology, Beijing, China.Corresponding author.

###### Abstract

Most existing video text spotting benchmarks focus on evaluating a single
language and scenario with limited data. In this work, we introduce a large-
scale, Bilingual, Open World Video text benchmark dataset(BOVText). There are
four features for BOVText. Firstly, we provide 2,000+ videos with more than
1,750,000+ frames, 25 times larger than the existing largest dataset with
incidental text in videos. Secondly, our dataset covers 30+ open categories
with a wide selection of various scenarios, _e.g.,_ Life Vlog, Driving, Movie,
etc. Thirdly, abundant text types annotation (_i.e.,_ title, caption or scene
text) are provided for the different representational meanings in video.
Fourthly, the BOVText provides bilingual text annotation to promote multiple
cultures’ live and communication.

Besides, we propose an end-to-end video text spotting framework with
Transformer, termed TransVTSpotter, which solves the multi-orient text
spotting in video with a simple, but efficient attention-based query-key
mechanism. It applies object features from the previous frame as a tracking
query for the current frame and introduces a rotation angle prediction to fit
the multi-orient text instance. On ICDAR2015(video), TransVTSpotter achieves
the state-of-the-art performance with 44.1% MOTA, 9 fps. The dataset and code
of TransVTSpotter can be found at $\tt github.com/weijiawu/BOVText$ and $\tt
github.com/weijiawu/TransVTSpotter$, respectively.

##  1 Introduction

Text spotting [31, 19] has received increasing attention due to its numerous
applications in computer vision, e.g., document analysis, image-based
translation, image retrieval [46, 38], etc. With the advent of deep learning
and abundance in digital data, reading text from static images has made
extraordinary progress in recent years with a lot of great public datasets
[14, 20, 8] and algorithms [60, 76, 34, 30]. By contrast, video text spotting
almost remains at a standstill for the lack of large-scale multidimensional
practical datasets, which limited numerous applications of video text, _e.g.,_
video understanding [51], video retrieval [10], video text translation, and
license plate recognition [1], etc.

Video text spotting(VTS) is the task that requires simultaneously classifying,
detecting, tracking and recognizing text instances in a video sequence. There
have been a few previous works [67, 64] and datasets [40, 21] in the community
for attempting to develop video text spotting. ICDAR2015 (Text in Videos) [20]
was introduced during the ICDAR Robust Reading Competition in 2015 and mainly
includes a training set of 25 videos (13k frames) and a test set of 24 videos
(14k frames). The videos were categorized into seven scenarios: walking
outdoors, searching for a shop in a shopping street, etc. YouTube Video Text
(YVT) [40] dataset harvested from YouTube, contains 30 videos with 13k frames,
15k for training, and 15k for testing. The text content in the dataset mainly
includes overlay text and scene text (_e.g.,_ street signs, business signs,
words on shirt). RoadText-1K [41] are sampled from BDD100K [70], includes 700
videos (210k frames) for training and 300 videos for testing. The texts are
all obtained from driving videos and match for driver assistance and self-
driving systems. LSVTD [6] includes 100 text videos, 13 indoor (_e.g.,_
bookstore, shopping mall) and 9 outdoor (_e.g.,_ highway, city road)
scenarios. However, as shown in Figure. 1 (a), most existing video text
datasets are limited by the amount of training data (less than 300k frames),
single video scenarios, and a single language. There are only a few outdoor
scene text videos with 13k frames in ICDAR2015 (video). Similar situation for
YVT, RoadText-1k, and LSVTD, the training set is limited and the dataset
scenarios are single. This makes it difficult to evaluate the effectiveness of
more advanced deep learning models for more open scenarios, such as game,
sport and news report. Besides, most existing video text datasets are proposed
before 2019 years, and some of them are no longer being maintained without an
open-source evaluation script. The download links of YVT even have become
invalid, which is not conducive to the development of video text spotting.

Figure 1: Comparison of dataset distribution and pipeline. (a) Data
distribution. BOVText provides various open world scenarios with unique NBA,
Game, etc. (b) Pipelines: (I) Multi-stage pipeline in [65], [16], [68] etc;
(II) Multi-orient video text spotting pipeline with Faster R-CNN [42] proposed
by Wang et al. [61]; (III) Online text tracking pipeline in Yu et al. [71];
(IV) Fast video text spotting pipeline in Cheng et al. [6]; (V) An end-to-end
pipeline with transformer in this work.

In this work, we contribute a large-scale, bilingual open-world benchmark
dataset (BOVText) to the community for developing and testing video text
spotting that can fare in a realistic setting. Our dataset has several
advantages. Firstly, the large training set (_i.e.,_ 2,000+video and
1,750,000+ video frames) from KuaiShou and YouTube enables the development of
deep design specific for video text spotting. Secondly, unlike the existing
datasets, BOVText support 30+ open scenarios, including many new scenarios
such as Sportscast(NBA, FIFA World Cup…), Life Vlog, Game, etc, as shown in
Figure. 1 (a). These data is collected from the worldwide user of
YouTube111https://www.youtube.com/ and KuaiShou222https://www.kuaishou.com/en,
cover various daily scenarios without region limitation and virtual scenes.
But the previous video text datasets usually are collected toward a special
city or language from the hand-held camcorder. Thirdly, BOVText is the first
benchmark for supporting abundant text types annotation. Caption, title, and
scene text are separately tagged for the different representational meanings
in the video. This made our BOVText has the potential to promote other video-
and-language tasks, such as video understanding. Fourthly, bilingual text
annotation(_i.e.,_ Chinese, English) is provided in BOVText to promote
multiple cultures’ live and communication.

Except for the promising benchmark, we also proposed a simple, but effective
video text spotter with transformer (TransVTSpotter). As shown in Figure. 1
(b), unlike previous methods that involve multiple steps, such as proposal
generation, text aggregation, and post-processing(NMS), TransVTSpotter only
requires two steps. 1) Text tracking: for each consecutive frame image, we
obtain the multi-orient boxes tracking trajectories of text by boxes IoU
matching between the predicted detection boxes [5] and the predicted tracking
boxes [52], where the detection boxes are obtained by taking an object query
as input, just like DETR [5]. And features from previously detected objects to
form another “track query” to discover associated objects (_i.e.,_ the
predicted tracking boxes) on the current frames. Besides, an additional angle
loss of multi-orient box and Hungarian angle cost are design to obtain the
angle of multi-orient. 2) Text recognition: recognizing the tracked texts with
attention-based text recognizer [33]. Without bells and whistles,
TransVTSpotter achieves state-of-the-art performance on ICDAR2015 with 44.1%
MOTA, 9 fps. The main contributions of this work are three folds:

(1) We propose a large-scale, bilingual and open world video text spotting
benchmark named BOVText. The proposed dataset provides 2,000+ videos,
1,750,000 frames, open videos scenarios (_e.g.,_ Indoor, Outdoor, Game,
Sport), abundant text types (_i.e.,_ title, caption or scene text), multi-
stage tasks and is 25 times the existing largest dataset with incidental text.

(2) Caption, title, scene text, and other overlap texts are firstly separately
tagged for the different representational meanings in the video. Based on the
previous works [46, 24], this favors other tasks theoretically, such as video
understanding, video retrieval, and video text translation.

(3) We first propose a new video text spotting framework with Transformer,
termed TransVTSpotter, which solves the video multi-orient text spotting with
a simple, but effective pipeline based on the tracking query-key mechanism and
rotated boxes angle prediction.

(4) We evaluate and compare TransVTSpotter and other techniques for scene text
detection, recognition, text tracking, and end-to-end video text spotting on
BOVText and other existing datasets. Besides, a thorough analysis of
performance on the proposed dataset is provided.

##  2 Related Work

###  2.1 End-to-End Text Spotting

For image-level text spotting, various methods [25, 15, 34] based on deep
learning have been proposed and have improved the performance considerably. Li
et al. [25] proposed the first end-to-end trainable scene text spotting
method. The method successfully uses a RoI Pooling [42] to joint detection and
recognition features. Liao et al. [34] propose a Mask TextSpotter which subtly
refines Mask R-CNN and uses character-level supervision to detect and
recognize characters simultaneously.

However, these methods based on the static image can not obtain temporal
information in the video, which is essential for some downstream tasks such as
video understanding. Compared to text spotting in a static image, video text
spotting methods are rare. Yin et al. [69] provides a detailed survey,
summarizes text detection, tracking and recognition methods in video. Wang et
al. [61] introduced an end-to-end video text recognition method through
associations of texts in the current frame and several previous frames to
obtain final results. Cheng et al. [6] propose a video text spotting framework
by only recognizing the localized text one-time. Nguyen et al. [40] improves
detection and recognition performance by temporal redundancy and linearly
interpolate to recover missing detection results. Rong et al. [45] tracked
video text using tracking-by-detection. An MSER detector was used to locate
scene text character, which was used as a constraint to optimize the
trajectory search. To promote video text spotting, we attempt to establish a
standardized benchmark (BOVText), covering various open scenarios and
bilingual text annotation.

###  2.2 Text Spotting Datasets for Images and Videos

The various and practical benchmark datasets [20, 54, 21, 8, 21] contribute to
the huge success of scene text detection and recognition at the image level.
ICDAR2015 [20] was provided from the ICDAR2015 Robust Reading Competition.
Google glasses capture these images without taking care of position, so text
in the scene can be in arbitrary orientations. ICDAR2017MLT [39] is a large-
scale bilingual text dataset, which is composed of complete scene images which
come from 9 languages. The COCO-Text dataset [54] is currently the largest
dataset for scene text detection and recognition. It contains 50,000+ images
for training and testing.

The development of video text spotting is limited in recent years due to the
lack of efficient data sets. ICDAR 2015 Video [21] consists of 28 videos
lasting from 10 seconds to 1 minute in indoors or outdoors scenarios. Limited
videos (_i.e.,_ 13 videos) used for training and 15 for testing. Minetto
Dataset [37] consists of 5 videos in outdoor scenes. The frame size is 640 x
480 and all videos are used for testing. YVT [40] contains 30 videos, 15 for
training and 15 for testing. Different from the above two datasets, it
contains web videos except for scene videos. USTB-VidTEXT [67] with only five
videos mostly contain born-digital text sourced from Youtube. RoadText-1K [41]
provides a driving videos dataset with 1000 videos. The 10-second long video
clips in the dataset are sampled from BDD100K [70]. As shown in Table. 1, the
existing datasets contain a limited training set and single video scenarios.
To promote the development of video text spotting, we create a large-scale,
bilingual open-world benchmark dataset.

###  2.3 Transformers in Vision

Transformer is first proposed in [53] as a new paradigm for machine
translation. But recently, there is a popularity of using transformer
architecture in vision tasks, such as detection [5, 78], segmentation [75], 3D
data processing [74], video object tracking [58] and even backbone
construction [11]. Lately, some works [63, 73, 58] show using a transformer in
processing sequential visual data also make remarkable shots. MOTR [73]
introduce the concept of track query and the contiguous query passing
mechanism for multiple-object Tracking. VisTR [63] solves instance
segmentation by learning the pixel-level similarity and instance tracking is
to learn the similarity between instances. But for video text spotting or
multi-orient text tracking, to the best of our knowledge, there are still no
transformer-based solutions while it is intuitive for its good capacity in
temporal processing. Here, we propose the TransVTSpotter method and provide an
affirmative answer to that, which shows convincingly high performance on the
popular benchmark.

##  3 BOVText Benchmark

###  3.1 Data Collection and Annotation

Data Collection. To obtain abundant videos with various text types, we first
start by acquiring a large list of different scenarios with text (_e.g.,_
_game scenario_ , _travel scenario_) using YouTube333https://www.youtube.com/
and KuaiShou444https://www.kuaishou.com/en \- an online resource that contains
billions of videos with various scene text from cartoon movies to human
relation. Then, we choose 31 open-domain categories with 1 unknown category,
_i.e.,_ , _Game_ , _Home_ , _Fashion_ , and _Technology_ , as shown in Figure.
2 (a). With each raw video category, we first choose the video clips with
text, then make two rounds of manual screening to remove the ordinary videos
without scene text and caption text. As a result, we obtain $2,021$ videos
with $1,620,305$ video frames, as shown in Table 1. Finally, to fair
evaluation, we divide the dataset into two parts: the training set with
$1,328,575$ frames from $1,541$ videos, and the testing set with $429,023$
frames from $480$ videos. As shown in Figure 2, different from the existing
data sets, our dataset not only cares about scene text spotting in the real
world, but also focuses on caption texts in the video. For the most part,
caption text represents more global information than scene text, which is
quite favorable for some downstream tasks, _e.g.,_ video understanding, video
caption translation, etc.

Figure 2: Distributions of BOVText. (a) The benchmark dataset covers a wide
and open range of life scenes (32 open-domain categories). (b) Caption text
(blue box) and scene text (red box) are distinguished in BOVText, which is
favorable for downstream tasks.

Data Annotation. We invite a professional annotation team to label each video
text with four kinds of description information: the rotated bounding box
describing the location information, judging the tracking identification(ID)
of the same text, identifying the content of the text information, and
distinguishing the category of the caption, title or scene text. To save the
annotation cost, we first sample the videos, annotate each sampled video
frame, and then transform the annotation information from the sampled video
frame to the unlabeled video frame by interpolation. Finally, we invite an
audit team to carry out another round of annotation checks, and re-label part
video frames with unqualified annotation. _For video sampling_ , we use
uniform sampling with a sampling frequency of $3$ to sample all the videos in
the dataset, and obtain the sampled video frame set. _For sampling video frame
annotation_ , each text instance is labeled in the same quadrilateral way as
in the ICDAR2015 [77]. In addition, the text instance also will be marked with
two description information: the category of the caption, title or scene, the
recognition content, and the tracking ID. _For interpolation on unlabeled
video frames_ , each text instance is marked with tracking ID and recognition
content, so we can judge whether the texts in adjacent sampling frames are the
same text with the same ID. For the same text instance, we first determine
whether the text annotation of the sampled video frame is the starting and end
frame of the text instance. If not, we look forward and backward for the
starting and end position of the text instance and label it. Then we use the
linear interpolation way to calculate the position of the text object in the
middle of the unmarked video frame, and give tracking ID, recognition content,
and category. _For check and re-label bad cases,_ the linear interpolation
shows a dissatisfied performance in some cases, _e.g., the new text appears on
starting frame, text suddenly disappears on ending frame,_ which are difficult
to capture. Therefore, we invite an audit team to carry out another round of
annotation checks. Around $150,000$ video frames with unqualified annotation
from $1,670,305$ video frames are selected to refine, taking 20 men in three
weeks. As a labor-intensive job, the whole labeling process takes 30 men in
three months, _i.e.,_ 21,600 man-hours, to complete about 600,000 sampled
video frame annotations.

###  3.2 Dataset Analysis

The statistic comparison between BOVText and other datasets are visualized in
Figure 1 (a), $\tt YouTube$, and summarized in Table 1. Besides, we provide
more detailed information, such as ‘data distribution for 32 open scenarios’,
‘text language and category distribution’.

To provide the community with unified text-level quantitative descriptions, we
will compare our dataset with the previous datasets from four aspects, _i.e.,_
text description, video scene, dataset size, and supported tasks. _For text
description attribute_ (_i.e.,__Category, MLingual_), our BOVText supports
four types of text annotations(caption, title, scene, and other text) of video
text with multi-language, which obviously has more extensive description
ability than the existing dataset.

Table 1: Statistical Comparison. ‘D’,‘R’, ‘T’, ‘S’ and ‘BI’ denotes the Detection, Recognition, Tracking, Spotting and bi-lingual text, respectively. ‘Incidental’ denotes indoor and outdoor scenarios in daily life (_e.g., walking outdoors, driving)_. ‘Open’ refers to any scenarios, __e.g.,_ Game, Sport(NBA)._ In green refers to these scenarios only supported by BOVText. Dataset |  Category | BI | Task | Videos | Frames | Texts |  Supported Scenario
---|---|---|---|---|---|---|---
AcTiV-D[72] |  Caption | - | D | 8 | 1,843 | 5,133 |  News video
UCAS-STLData[4] |  Caption | - | D | 3 | 57,070 | 41,195 |  Teleplay
USTB-VidTEXT[67] |  Caption | - | D&S | 5 | 27,670 | 41,932 |  Web video
YVT[40] |  Scene, Caption | - | D&R&T&S | 30 | 13,500 | 16,620 |  Incidental: Cartoon, Outdoor(supermarket, shopping street, driving…)
ICDAR2015 VT[77] |  Scene | - | D&R&T&S | 51 | 27,824 | 143,588 |  Incidental: Outdoor(walking, driving, supermarket, shopping street…)
LSVTD[6] |  Scene | ✓ | D&R&T&S | 100 | 66,700 | 569,300 |  Incidental: Indoor(shopping mall, supermarket, hotel…), Outdoor(driving…)
RoadText-1K[41] |  Scene | - | D&R&T&S | 1,000 | 300,000 | 1,280,613 |  Driving
BOVText(ours) |  Scene, Caption, Title, Others | ✓ | D&R&T&S | 2,021 | 1,757,598 | 7,292,261 |  Open: Cartoon, Vlog(supermarket, shopping street, driving), Travel (indoor and outdoor), Game(PUBG mobile…), Sport(NBA, world cup…), News , TV program, Education(campus, classroom, book…),Technology(introductory video, scientific propaganda…)…

_For video scene attribute_ (_i.e.,__Scenario_), we present the 31 open
scenarios and an "Unknown" scenarios distribution on BOVText in three levels ,
_i.e.,_ video, video frames, and text instances , as shown in Table 2. Unlike
the existing datasets, BOVText spans various video domains with these
scenarios in the existing datasets (_e.g.,_ Driving for RoadText-1k [41],
Vlog(supermarket, shopping street, indoor), Travel(hotel, railway station) for
LSVTD [6]) and more open domains that are not yet supported (_e.g.,_ Game(PUBG
mobile, Honor of Kings…), Sport(NBA, world cup…), News). _For the size of the
dataset_(_i.e.,__Videos, Frames, Texts_), BOVText is 25 times larger than the
existing largest dataset (_i.e.,_ LSVTD [6]) with various
scenarios($1,757,598$ v.s $66,700$ video frames). RoadText1k [41] contains
300k videos frames, but the supported scenario is too single for only supports
driving video scenarios. _For the supported tasks_ , the proposed BOVText
supports all video text tasks: detection, recognition, video text tracking,
and end to end video text spotting. For comprehensive research, we not only
focus the scale, location, recognition content, and tracking ID, but also
additionally collect and annotate the category of caption, title, scene or
other texts for each text instance. As shown in Figure. 2 (b), in a video,
different types of text instances may exist simultaneously, and they are
helpful to understand videos synergistically. Caption text can directly show
the dialogue between people in video scenes and represent the time or topic of
the video scenes, scene text can unambiguously define the object and can
identify important localization and road paths in video scenes. Therefore, the
text category annotation is favoring downstream tasks (_e.g.,_ video text
translation, video understanding, and video retrieval), more details in the
appendix. In conclusion, the high efficiency of BOVText for evaluating
advanced deep learning methods is very favorable for promoting various text
spotting applications in real life. Statistic Comparison. As shown in Table.
1, _Category_ denotes the category of the text type in the corresponding
dataset. _MLingual_ denotes whether the dataset contains multiple language
texts. _Scenario_ denotes the scene range of the video. _Videos, Frames,
Texts_ represents the number of videos, video frames, video texts in the
dataset, respectively. _Task_ denotes which tasks the dataset supports.

Table 2: The Data Distribution for 32 Open Scenarios. In green refers to these scenarios only supported by BOVText. "%" denotes the percentage of each scenario data for whole data. Scenarios | Video | Video Frames | Text Instances | Scenarios | Video | Video Frames | Text Instances
---|---|---|---|---|---|---|---
Cartoon | 67(3.2%) | 64,359(3.6%) | 123,191(2.1%) | Sport | 71(4.8%) | 54,643(3.1%) | 266,996(4.6%)
Vlog | 90(4.2%) | 83,891(4.7%) | 214,910(3.7%) | News Report | 100(4.1%) | 66,207(3.7%) | 178,000(3.1%)
Driving | 71(3.8%) | 61,626(3.5%) | 151,994(2.6%) | Celebrity | 50(2.4%) | 39,958(2.3%) | 121,235(2.1%)
Advertising | 32(1.7%) | 28,645(1.6%) | 91,090(1.0%) | Technology | 68(3.1%) | 53,305(3.1%) | 140,172(2.4%)
Activity | 35(1.6%) | 26,837(1.4%) | 67,879(1.2%) | Program | 42(2.3%) | 38,108(2.2%) | 214,561(3.7%)
Comedy | 88(4.5%) | 79,206(4.5%) | 317,865(5.5%) | Game | 21(1.0%) | 33,925(1.9%) | 84,106(1.5%)
Interview | 37(1.3%) | 31,440(1.8%) | 63,616(1.1%) | Livestreaming | 64(3.1%) | 62,130(3.6%) | 211,569(3.6%)
Government | 66(2.5%) | 45,457(2.6%) | 93,874(1.6%) | Speech | 69(3.2%) | 56,646(3.1%) | 175,119(3.0%)
Travel | 74(4.3%) | 71,291(4.1%) | 280,446(4.8%) | Movie | 106(5.6%) | 108,110(6.3%) | 299,760(5.2%)
Campus | 52(2.3%) | 43,469(2.5%) | 139,760(2.4%) | Photograph | 70(2.8%) | 64,025(3.6%) | 173,832(3.0%)
International | 55(3.1%) | 52,774(3.6%) | 132,117(2.3%) | Education | 74(3.4%) | 59,824(3.6%) | 360,774(6.2%)
Short Video | 70(4.4%) | 59,756(3.4%) | 326,930(5.6%) | Dance | 43(1.9%) | 27,941(1.6%) | 71,740(1.2%)
Makeup | 63(3.1%) | 54,643(3.1%) | 111,814(1.9%) | Fishery | 81(4.5%) | 75,018(4.3%) | 230,085(4.0%)
Talent | 86(4.1%) | 71,024(4.1%) | 339,382(5.9%) | Fashion | 63(3.0%) | 46,337(2.6%) | 98,942(1.7%)
Beauty Industry | 40(1.9%) | 41,350(2.4%) | 132,025(2.3%) | Introduction | 64(3.8%) | 59,048(3.4%) | 236,721(4.1%)
Eating | 56(2.7%) | 62,893(3.6%) | 191,035(3.3%) | Unknown | 53(2.5%) | 33,712(1.9%) | 150,721(2.6%)

###  3.3 Supported Tasks and Metrics

The proposed BOVText supports four tasks: (1) Video Text Detection. (2) Video
Text Recognition. (3) Video Text Tracking. (4) End to End Text Spotting in
Videos.

Following ICDAR2015 [77] 555https://rrc.cvc.uab.es/?ch=4&com=tasks, the
evaluation protocols [57] are used for text detection and recognition task.
For video text tracking and spotting task, the existing video text datasets
such as ICDAR2015 (video) [21] 666https://rrc.cvc.uab.es/?ch=3&com=tasks and
RoadText-1k [41] all adopted the MOT metrics (_i.e.,_ Multiple Object Tracking
Accuracy ($MOTA$) and Multiple Object Tracking Precision ($MOTP$)). However,
there are two sets of measures for Multiple Object Tracking: the MOT metrics
($MOTA$,$MOTP$) [2] and ID metrics ($ID_{F1}$) [26, 44]. The CVPR19
MOTChallenge evaluation framework [9] presents that different measures serve
different purposes. $Event\mbox{-}based$ measures like MOT help pinpoint the
source of some errors and are thereby informative for the designer of certain
system components. $Identity\mbox{-}based$ measure($ID_{F1}$) is more
favorable for evaluating how well computed identities conform to true
identities. Except for using MOTA, MOTP, $Identity\mbox{-}based$
measures($ID_{F1}$), as a new metric is adopted firstly for video text
spotting task. More detailed information for metric can be obtained in the
appendix.

###  3.4 Our Method: TransVTSpotter

Two ingredients are essential for direct text spotting for TransVTSpotter: (1)
A set prediction loss that forces unique matching between predicted and ground
truth multi-orient boxes. (2) An architecture that predicts a set of objects
and associates the same objects during different frames.

Multi-orient Box Matching. Compare to DETR [5], the difference is that we
propose an angle prediction and corresponding loss while only horizontal boxes
prediction for DETR. Let us denote the ground truth set of objects by $y$, and
$\hat{y}=\\{\hat{y}_{i}\\}_{i=1}^{N}$ the set of $N$ predictions. $y$ is as a
set of size $N$ padded with $\varnothing$ (no object). To find a bipartite
matching between these two sets we search for a permutation of $N$ elements
$\sigma\in\mathfrak{S}_{N}$ with the lowest cost:

$\hat{\sigma}=\operatorname*{arg\,min}_{\sigma\in\mathfrak{S}_{N}}\sum_{i}^{N}{\cal
L}_{\rm match}(y_{i},\hat{y}_{\sigma(i)}),$ (1)

where ${\cal L}_{\rm match}(y_{i},\hat{y}_{\sigma(i)})$ is a pair-wise
_matching cost_ between ground truth $y_{i}$ and a prediction with index
$\sigma(i)$. The matching cost takes into account the class prediction, boxes
prediction and the boxes rotated angle prediction. Each element $i$ of the
ground truth set can be seen as a $y_{i}=(c_{i},b_{i},a_{i})$ where $c_{i}$ is
the target class label, $b_{i}\in[0,1]^{4}$ is a vector that defines ground
truth box center coordinates and its height and width relative to the image
size, and $a_{i}$ is rotation angle between the longest edge of ground truth
multi-orient box and horizontal line (x-axis). For the prediction with index
$\sigma(i)$ we define probability of class $c_{i}$ as
$\hat{p}_{\sigma(i)}(c_{i})$, the predicted box as $\hat{b}_{\sigma(i)}$, and
the predicted angle as $\hat{a}_{\sigma(i)}$. Thus we define ${\cal L}_{\rm
match}(y_{i},\hat{y}_{\sigma(i)})$ as
$-\mathds{1}_{\\{c_{i}\neq\varnothing\\}}\hat{p}_{\sigma(i)}(c_{i})+\mathds{1}_{\\{c_{i}\neq\varnothing\\}}{\cal
L}_{\rm
box}(b_{i},\hat{b}_{\sigma(i)})+\mathds{1}_{\\{c_{i}\neq\varnothing\\}}{\cal
L}_{\rm angle}(a_{i},\hat{a}_{\sigma(i)})$. Finally, we could compute the loss
function with all pairs matched:

${\cal L}_{\rm
Hungarian}(y,\hat{y})=\sum_{i=1}^{N}\left[-\log\hat{p}_{\hat{\sigma}(i)}(c_{i})+\mathds{1}_{\\{c_{i}\neq\varnothing\\}}{\cal
L}_{\rm
box}(b_{i},\hat{b}_{\hat{\sigma}}(i)+\mathds{1}_{\\{c_{i}\neq\varnothing\\}}{\cal
L}_{\rm angle}(a_{i},\hat{a}_{\sigma(i)}))\right]\,,$ (2)

where ${\cal L}_{\rm box}(\cdot)$ a linear combination of the $\ell_{1}$ loss
and the generalized IoU loss [43, 5]. And ${\cal L}_{\rm angle}(\cdot)$ refers
to a cosine embedding loss with $1-cos(\hat{a}_{\sigma(i)}-a_{i})$.

Figure 3:  Pipeline of TransVTSpotter. It contains four main components: 1) A
transformer backbone(PVT [59]) extracts feature representation of multiple
images; 2) A transformer encoder models the relations of pixel-level features;
3) Two transformer decoders (shared weight) decode the instance-level
features; 4) Attention-based recognizer [33] recognizes each text instance.

TransVTSpotter architecture. The overall TransVTSpotter architecture is
surprisingly simple and depicted in Figure. 3. A transformer-based backbone
[59] is used to extract feature representation, transformer-based encoder-
decoder framework learned current object query and previous frame tracking
query as input and predicts detection boxes and tracking boxes [52]. With the
detection boxes and tracking boxes, box IoU matching is used to obtain the
final tracking result. Finally, attention-based recognizer [33] is utilized to
obtain the final recognition results. Text Tracking with Transformer. Text
Tracking with Transformer includes three components: backbone, transformer
encoder and transformer decoder, as shown in Figure. 3. Backbone. Starting
from the two consecutive frames $x_{\rm t}\in\mathbb{R}^{3\times H_{0}\times
W_{0}}$ and $x_{\rm t\mbox{-}1}\in\mathbb{R}^{3\times H_{0}\times W_{0}}$, a
transformer backbone [59] generates a lower-resolution activation map for the
two frames($f_{\rm t}\in\mathbb{R}^{C\times H\times W}$ and $f_{\rm
t\mbox{-}1}\in\mathbb{R}^{C\times H\times W}$), then a new feature
sequence$f_{\rm t}^{*}$ can be obtained by simple concatenating $f_{\rm t}$
and $f_{\rm t\mbox{-}1}$. The extracted features $f_{\rm t}$ of the current
frame are temporarily saved and then re-used for the next frame. Transformer
Encoder. We adopted deformable transformer encoder [78] to model the
similarities among all the pixel level features for the extracted features
$f_{\rm t}$. Transformer Decoder. Two parallel decoders [52] are employed. The
object decoder takes learned object query [5] as input and predicts detection
rotated boxes. The tracking decoder takes the object feature from previous
frames as input and predicts the corresponding tracking rotated boxes.
Finally, with detection rotated boxes and tracking rotated boxes, TransTT
obtains the final tracking result by box IoU matching and the Kuhn-Munkres(KM)
algorithm [23]. Recognizer. Following MASTER [33] is utilized to predict
output sequence with 2D-attention.

##  4 Experimental

In this section, we mainly conduct experiments on our BOVText. More
experiments, such as the performance of TransVTSpotter in other datasets,
would be provided in the appendix.

###  4.1 Implementation Details

Except for the TransVTSpotter, we also adopt CRNN [47], RARE [48] as the
recognition baseline and PSENet [60], EAST [76], DB [28] as the detection
baseline to evaluate our BOVText. Detection: we train detectors with pre-
trained model on COCOText [54]. Recognition: the network is pre-trained on the
chinese ocr777https://github.com/YCG09/chinese_ocr and MJSynth [17], then
fine-tuned on our BOVText. All of our experiments are conducted on 8 V100
GPUs. AdamW [32] as the optimizer and the batch size is set to be 16. The
initial learning rate is 2e-4 for the transformer and 2e-5 for the backbone.
The weight decay is 1e-4 All transformer weights are initialized with Xavier-
init [13]. The data augmentation includes random horizontal, scale
augmentation, resizing the input images whose shorter side is by 480-800
pixels while the longer side is by at most 1333 pixels. The model is first
pre-trained on COCOText [54] and then fine-tuned on other video text training
sets. For each iteration, two adjacent frames are randomly selected from one
video from training set to train our model.

###  4.2 Attribute Experiments Analysis for BOVText

New Scenarios, New Challenge for Video Text Tasks. Figure. 5 and Table. 3
gives the tracking performance $ID_{F1}$ of TransVTSpotter in different
scenarios of BOVText. Two new insights can be present from the figure and
table: 1) The existing benchmark datasets cannot effectively test the
effectiveness of advancing algorithm on some novel scenarios (_e.g.,_
NewsReport, Cartoon) for first proposed in BOVText. LSVTD [6] and RoadText-1k
[41], as the two largest data sets on the existing video text datasets are
used to compare with our BOVText. TransVTSpotter achieves a tracking
performance ${\rm ID_{F1}}$ of $70.4\%$ on $NewsReport$ scenario with BOVText
training set, around $20$ percent point improvement than training with LSVTD
[6] and RoadText-1k [41]. We argue that the main cause is that there existing
a mass of caption texts in $NewsReport$ scenario, but LSVTD and RoadText
almost no such dense text scenario, which is a new challenge for algorithms.
Besides, training with only RoadText-1k obtains a low performance no matter
which scenarios. There are two main causes for this. Firstly, the location
annotation of RoadText-1k is an upright bounding box(two points), but the
counterpart of BOVText is multi-orient boxes(four points). Secondly, the data
domain is entirely different for the two datasets. Compare with various
scenarios (_e.g.,_ Game, Sports) and text types (_e.g.,_ long caption text,
big text), the scenario of RoadText-1k only contains small and low-resolution
road signs, plate number on driving scenarios. 2) Huge performance gap
existing during different scenarios. As shown in Figure. 5, the model achieves
the best performance with a $ID_{F1}$ of $88.4\%$ in $Fishery$ videos, since
the conspicuous text instances, simple foreground (_i.e.,_ caption texts) and
background are designed in cartoon videos. By comparison, several scene
categories obtain extremely dissatisfied performance due to complex
background, various text appearance, and unsteady camera movements, such as
$Sports$ of $46.7\%$ and $Travel$ of $55.8\%$.

Figure 4: Attribute Experiments of BOVText. (a) Recognition accuracy of different models in different languages. (b) Detection of different model in caption or scene text. ‘CH’, ‘AN’, ‘EN’ and ‘ALL’ refer to ‘Chinese’, ‘Alphanumeric’, ‘English’ and ‘All Characters’, respectively. Figure 5: Tracking performance (_i.e.,_ $ID_{F1}$) with TransVTSpotter in different scenarios. ‘Training on LSVTD’ and ‘Training on BOVText’ denotes training on LSVTD train set and BOVText train set, respectively. Huge performance gap existing during different scenarios. Table 3: Attribute Experiments for Scenarios. Random scenarios are selected to present here. $\rm Method$ |  Training Set | Detection (F-score/%) on BOVText |  Tracking (${\rm ID_{F1}}$/%) on BOVText
---|---|---|---
Cartoon | NewsReport | Driving | Program | Avg. | Cartoon | NewsReport | Driving | Program | Avg.
TransVTSpotter | LSVTD | 48.2 | 68.2 | 60.3 | 74.2 | 63.8 | 33.7 | 42.1 | 37.9 | 56.5 | 38.1
RoadText | 5.6 | 4.3 | 3.0 | 9.6 | 6.7 | 0.7 | 3.2 | 2.5 | 4.2 | 3.5
LSVTD&RoadText | 45.9 | 70.4 | 62.3 | 75.3 | 67.6 | 35.7 | 50.5 | 40.9 | 54.5 | 41.3
BOVText | 80.7 | 82.3 | 78.1 | 91.3 | 81.7 | 53.3 | 70.4 | 61.0 | 83.1 | 64.7

Bilingual Recognition, New Challenge. As shown in Figure. 4 (a), the text
recognition results for different languages are provided. In summary, the
alphanumeric recognition result (about $47\%$) is better than the Chinese
recognition result (about $35\%$), regardless of the models. The final results
(about $40\%$) for all characters are satisfactory, can not meet the
requirement of the application. Unlike English, Chinese contains thousands of
characters($3,856$ Chinese characters v.s. $26$ English characters on
BOVText), which are difficult to recognize.

Long Caption Text, New Challenge. As shown in Figure. 4 (b), for DB [28],
PSENet [60] and our TransVTSpotter, the performance of caption text is better
than the counterpart of scene text (around $80\%$ vs. $60\%$) due to the more
clear and bigger caption text. But for EAST [76], long caption text show a low
performance with $36\%$ F-score. The prime reason is that caption texts are
all long text ( average width-height ratio: $6.8$ for caption $v.s.$ $2.3$ for
scene text on BOVText), a different case for EAST [76], as shown in $\tt
YouTubeDemo$. However, the existing video text datasets hardly contain long
caption texts, our BOVText can fill out the gap for a more comprehensive
evaluation of text types.

Table 4: Detection and Recognition Performance on BOVText. Detection Performance/% | Recognition Performance/%
---|---
Method | Precision | Recall | F-score | Method | Pretrained | Fine tuned
Chinese | Alphanumeric | English | All | Chinese | Alphanumeric | English | All
EAST [76] | 55.4 | 40.8 | 47.0 |  CRNN [47] | 26.0 | 32.1 | 36.1 | 23.2 | 33.2 | 47.1 | 49.5 | 38.6
PSENet [60] | 78.3 | 75.7 | 77.0 |  RARE [48] | 25.2 | 34.2 | 37.4 | 23.5 | 35.6 | 45.7 | 50.4 | 40.2
DB [28] | 84.3 | 77.6 | 80.8 |  GRCNN [56] | 23.1 | 39.8 | 40.4 | 26.7 | 35.6 | 49.2 | 52.4 | 40.3
TransVTSpotter | 86.2 | 77.4 | 81.7 | - | 26.2 | 40.3 | 42.1 | 29.1 | 36.2 | 48.9 | 52.1 | 40.4
Table 5: Text Tracking and End to End Video Text Spotting Performance on BOVText. Text tracking trajectory id generation use a method proposed in [61]. $\rm Method$ | Text Tracking on BOVText | End to End Text Spotting on BOVText
---|---|---
Detection | Recognition |  $\rm ID_{P}$/% |  $\rm ID_{R}$/% |  ${\rm ID_{F1}}$/% | $\rm MOTA$ | $\rm MOTP$ |  $\rm ID_{P}$/% |  $\rm ID_{R}$/% |  ${\rm ID_{F1}}$/% | $\rm MOTA$ | $\rm MOTP$
EAST [76] |  CRNN [47] | 29.9 | 26.5 | 28.1 | -0.216 | 0.758 | 6.8 | 6.9 | 6.8 | -0.793 | 0.763
RARE [48] | 4.2 | 5.3 | 4.7 | -1.05 | 0.772
PSENet [60] |  CRNN [47] | 52.4 | 40.9 | 45.9 | 0.521 | 0.775 | 31.3 | 26.7 | 28.8 | -0.170 | 0.792
RARE [48] | 35.6 | 28.8 | 31.7 | -0.162 | 0.803
DB [28] |  CRNN [47] | 55.2 | 42.9 | 48.3 | 0.532 | 0.783 | 38.8 | 30.1 | 33.7 | -0.132 | 0.813
RARE [48] | 41.1 | 29.3 | 34.2 | -0.126 | 0.811
TransVTSpotter(ours) | 71.0 | 59.7 | 64.7 | 0.682 | 0.821 | 43.6 | 38.4 | 40.8 | -0.014 | 0.820

###  4.3 Text Detection, Recognition, Tracking and Spotting on BOVText

Video Text Detection and Recognition. As shown in Table. 4, image-based text
detection on BOVText is not unsatisfactory, with lower results than these
methods report on existing text datasets. For example, EAST obtains an f-score
of $47.0\%$ compared to the F-score of $80.7\%$ on icdar2015 [77], but our
TransVTSpotter obtain an f-score of $81.7\%$ on BOVText, at least $1\%$
improvement compare to the image-based detectors (_i.e.,_ DB, PSENet and
EAST). For text recognition, CRNN [47] based on CTC loss, RARE [48] with
attention mechanism and GRCNN [56] as the base text recognizers to test our
BOVText. The text annotation in our BOVText covers two languages (_i.e.,_
English and Chinese), thus we conduct several experiments for each language.
The recognition model only yields about $40\%$ accuracy on our dataset, but
the same model reports at least $90\%$ on most benchmark datasets [21] for
text recognition. The main reasons have two points: (1) The proposed BOVText
is bilingual, and the category number of Chinese characters in real-world is
much larger than those of Latin languages. (2) The video texts are quite
blurred, out-of-focus, and the distribution of characters is relatively
smaller than the static image counterparts, which presents more challenges.

Video Text Tracking. As shown in Table. 5, $\rm ID_{F1}$ ($64.7\%$) of our
TransVTSpotter achieves the best performance, at least $10\%+$ improvement
than other methods. Besides, without NMS and other post-processing,
TransVTSpotter presents 9 fps no matter which dataset. More details and
analysis concerning TransVTSpotter can be obtained in the appendix. And EAST
shows the worst performance with a $\rm ID_{F1}$ of $28.1\%$. The $\rm
IDF_{1}$ of EAST [76] is lower with $17.8\%$ gap than that of PSENet [60]. The
main reason is that BOVText contains a mass of long text instances, but
regression-based EAST can not deal with the long text cases well. The
performance of DB is similar to that of PSENet for both all are the
segmentation-based methods.

End to End Text Spotting in Video. Detection and text tracking tasks are
paving the way for the recognition task. Table. 5 shows the performance of
text spotting on BOVText. Similar to the text tracking performance, our
TransVTSpotter achieves the state-of-the-art performance with at least $6\%$
$\rm ID_{F1}$ improvement compared to the other methods. Besides, the MOTP of
TransVTSpotter achieves $82\%$, $1\%$ percent points improvement than the
counterpart of using DB and RARE. The great performance for $40.8\%$ $\rm
ID_{F1}$ and $82\%$ MOTP present satisfactory tracking and recognition
trajectory and detection results, respectively. The corresponding performance
using EAST [76] as the detector in video text spotting is still not satisfied
with around $5\%$ $\rm ID_{F1}$ and $-0.8$ MOTA. Without TransVTSpotter, the
combination of DB [28] and RARE [48] achieves the best performance with a
$34.2\%$ $\rm ID_{F1}$, but there is at least $6\%$ gap compare to our method.

###  4.4 Potential Negative Societal Impacts and Solution

We argue that there mainly exits slight potential negative societal impacts
for personal privacy. Although much personal information, _e.g.,__names,
identifying information, human faces_ , have been blurred to protect privacy,
there still might exist a little risk.

How to blur human faces. We blur the human faces in BOVText with four steps.
Firstly, human faces in each frame would be detected by face
recognition888https://github.com/ageitgey/face_recognition \- a powerful,
simple, and easy-to-use face recognition open source project with complete
development documents and application cases. Secondly, with the location box
from the previous step, we extract face ROI from the original image. Thirdly,
we blur the face ROI with Gaussian Blur operation in
OpenCV999https://www.tutorialspoint.com/opencv/opencv_gaussian_blur.htm.
Finally, we store the blurred face in the original image and recover to video.

###  4.5 Link to Other Video-and-Language Applications

In this section, we show that the practicability of the proposed BOVText, not
a toy benchmark, which can promote other video-and-text application research.
Text spotting in static images has numerous application scenarios: (1)
Automatic data entry. SF-Express 101010https://www.sf-express.com/cn/sc/
utilizes OCR techniques to accelerate the data entry process. NEBO
111111https://www.myscript.com/nebo/ performs instant transcription as the
user writes down notes. (2) Autonomous vehicle [36, 35]. Text-embedded panels
carry important information, _e.g.,_ geo-location, current traffic condition,
navigation, and etc. (3) Text-based reading comprehension. TextCaps [49] and
text-based VQA [50, 3] show the new vision-and-language tasks, which need to
recognize text, relate it to its visual context, semantic, and visual
reasoning between multiple text tokens and visual entities, such as objects.
Similarly, there are many application demands for video text understanding
across various industries and in our daily lives. We list the most outstanding
ones that significantly impact, improving our productivity and life quality.
Firstly, automatically describing video with natural language [66, 62] can
bridge video and language. Secondly, video text automatic translation
121212https://translate.google.com/intl/en/about/ can be extremely helpful as
people travel, and help video-sharing websites 131313https://www.youtube.com/
to cut down language barriers. Finally, text-based video retrieval [24, 29] is
a irreplaceable business for many companies, such as Google and YouTube. More
details and analyses for application scenarios concerning BOVText in the
appendix.

##  5 Conclusion and Future Work

In this paper, we establish a large-scale, bilingual open-world benchmark
dataset for video text tracking and spotting, termed BOVText, with four
description information, _i.e.,_ , bounding box, tracking ID, recognition
content, and text category label. Compare with the existing benchmarks, the
proposed BOVText mainly contains four advantages: large-scale training set
(_i.e.,_ 2,021+video), $32$ open real scenarios (Sportscast, Life Vlog, Game),
bilingual annotation, and abundant text types annotation(Caption, title, and
scene text). Besides, we first propose an end-to-end video text spotting
framework with Transformer, termed TransVTSpotter, which presents a simple,
but efficient attention-based query-key pipeline. On ICDAR2015(video),
TransVTSpotter achieves the state-of-the-art performance. In general, we hope
the proposed BOVText and TransVTSpotter would provide a standard benchmark to
facilitate the advance of video-and-text research.

##  6 Appendix

###  6.1 BOVText Metrics

The proposed BOVText includes four tasks: (1) Video Text Detection; (2) Video
Text Recognition; (3) Video Text Tracking. (4) End to End Text Spotting in
Videos. $MOTP$ (Multiple Object Tracking Precision) [2], $MOTA$ (Multiple
Object Tracking Accuracy) and $IDF_{1}$ [9, 44] as the three important metrics
are used to evaluate task3 (text tracking) and task4 (text spotting) for
BOVText. Following the previous works [21, 41], BOVText evaluates text
tracking methods in video and compare their performance with the MOTA and
MOTP, which are given by:

$MOTP=\frac{{\textstyle\sum_{i,t}}(1-d^{i}_{t})}{\textstyle\sum_{t}c_{t}}\,,$
(3)

where $c_{t}$ denotes the number of matches found for time $t$. For each of
these matches, calculate the iou $d^{i}_{t}$ between the object $o^{i}$ and
its corresponding hypothesis. It shows the ability of the tracker to estimate
precise object positions. MOTA is calculated as follows:

$MOTA=1-\frac{{\textstyle\sum_{t}}(m_{t}+fp_{t}+mme_{t})}{\textstyle\sum_{t}g_{t}}\,,$
(4)

where $m_{t}$, $fp_{t}$ and $mme_{t}$ are the number of misses, false
positives, and mismatches, respectively. $g_{t}$ is the number of objects
present at time $t$. It shows the tracker’s performance at detecting objects
and keeping their trajectories, independent of the precision of the location.
$ID_{F1}$ is the ratio of correctly identified detections over the average
number of ground-truth and computed detections. And the metric is more
reasonable to evaluate ID switches in some cases. We evaluate the metrics in
BOVText by:

$ID_{tp}=\sum_{h}\sum_{t}m(h,o,\bigtriangleup_{t},\bigtriangleup_{s})\,,$ (5)

$ID_{F1}=\frac{2ID_{tp}}{2ID_{tp}+ID_{fp}+ID_{fn}}\,,$ (6)

where $ID_{tp}$, $ID_{fp}$ and $ID_{fn}$ refer to true positive, false
positive and false negative of matching ID. Besides, the ID metric [9] also
includes $MT$ (Mostly Tracked) Number of objects tracked for at least 80
percent of lifespan, $ML$ (Mostly Lost) Number of objects tracked less than 20
percent of lifespan. $\bigtriangleup_{t}$ and $\bigtriangleup_{s}$ refer to
time matching and space location matching, respectively.

For Task4 (End to End Text Spotting in Videos), the objective of this task is
to recognize words in the video as well as localize them in terms of time and
space. We use $ID_{F1}$ to evaluate our BOVText, which focuses on text
instance ID tracking and recognition results that be required by many
downstream tasks. More specifically,

$ID_{tp}=\sum_{h}\sum_{t}m(h,o,\bigtriangleup_{t},\bigtriangleup_{s},\bigtriangleup_{r})\,,$
(7)

$ID_{F1}=\frac{2TID_{tp}}{2TID_{tp}+TID_{fp}+TID_{fn}}\,,$ (8)

where $\bigtriangleup_{t}$, $\bigtriangleup_{s}$ and $\bigtriangleup_{r}$
refer to ID matching, space location matching and recognition result matching.
$h$ and $o$ denote hypothesis set (_e.g.,_ predicted ID $I_{p}$ , box
locations $L_{p}$, recognition results $R_{p}$) and ground truth set with (ID
$I_{g}$, box locations $L_{g}$, recognition ground true $R_{g}$). And the
three matching can be obatined by:

$\displaystyle\bigtriangleup_{t}:I_{p}=I_{g},$
$\displaystyle\bigtriangleup_{s}:IoU(L_{p},L_{g})>0,$
$\displaystyle\bigtriangleup_{r}:R_{p}=R_{g}.$ (9)

The match of $h$ and $o$ is a true positives of text ID (_i.e.,_ $ID_{tp}$)
when these conditions (_i.e.,_ $\bigtriangleup_{t}$, $\bigtriangleup_{s}$ and
$\bigtriangleup_{r}$ are met. Similarly, false positive (_i.e.,_ $ID_{fp}$)
and false negative (_i.e.,_ $ID_{fn}$) of text ID can be obtained for
$ID_{F1}$ calculation.

###  6.2 More details for experiments

TransVTSpotter. All experiments are conducted on Tesla V100 GPU. We train the
model for 150 epochs and the learning rate drops by a factor of 10 at the
100th epoch.

Baselines concerning the existing methods. Video-based text spotting methods
are rare and lack open-source algorithms. Therefore, we adopt various mature
image-based techniques to compare and evaluate the efficiency of BOVText.
Detection. EAST [76] as one of the popular regression-based methods is used to
test our BOVText. The method adopts FCNs to predict shrinkable text score
maps, rotation angles. For segmentation based methods, we adopt PSENet [60]
and DB [28] to evaluate our BOVText. PSENet [60] generates various scales of
shrinked text segmentation maps, then gradually expands kernels to generate
the final text instance. Recognition. Recent methods mainly include two
techniques, Connectionist Temporal Classification (CTC) and attention
mechanism. In CTC-based methods, CRNN [47] as the representation, which
introduced CTC decoder into scene text recognition with BiLSTM to model the
feature sequence. In Attention-based methods, RARE [48] normalizes the input
text image using the Spatial Transformer Network (STN [18]). Then, it
estimates the output character sequence from the identified features with the
attention module. Text Tracking Trajectory Generation. With text detection and
recognition in a static image, we only obtain text localization and
recognition information without temporal information, which are insufficient
for video spotting evaluation (_e.g.,_ $ID_{F1}$,$MOTA$ and $MOTP$). Following
the work [61], we link and match text objects in the current frame and several
frames by IOU and edit the distance of text. All of the experiments use the
same strategy: (1) Training detector and recognizer with BOVText. (2) Matching
text objects with corresponding text tracking trajectory id.

###  6.3 Ablation study for TransVTSpotter

Table 6: Ablation study for angle prediction. The gaps of at least (+7.1%) improvement after using angle prediction are shown in green. $\rm Method$ | Text Detection on ICDAR2015/% | Text Tracking on ICDAR2015(video)/%
---|---|---
Precision | Recall | F-score |  ${\rm ID_{F1}}$/% | $\rm MOTA$ | $\rm MOTP$
Ours,w/o angle prediction | 73.5 | 65.1 | 68.5 | 36.1 | 16.4 | 68.7
Ours,w/ angle prediction(L1 loss) | 84.2 | 82.3 | 83.2 | 54.7 | 43.2 | 74.5
Ours,w/ angle prediction(cosine loss) | 86.8 | 81.7 |  84.2(+15.7) |  57.3(+21.2) |  44.1(+27.7) |  75.8(+7.1)
Table 7: Ablation study for input query and input image size. Tracking query bring huge improvement with $3.1\%$ ${\rm ID_{F1}}$. $\rm Query$ | Text Tracking on ICDAR2015(video)/% | $\rm ShortSide$ | Text Tracking on ICDAR2015(video)/%
---|---|---|---
${\rm ID_{F1}}$/% | $\rm MOTA$ | $\rm MOTP$ |  ${\rm ID_{F1}}$/% | $\rm MOTA$ | $\rm MOTP$ | $\rm FPS$
Obejct query | 54.2 | 40.5 | 76.6 | 480 | 52.6 | 41.5 | 74.5 | 16
Obejct + tracking query | 57.3 | 44.1 | 75.8 | 640 | 56.7 | 43.8 | 74.5 | 13
- | - | - | - | 800 | 57.3 | 44.1 | 75.8 | 9

Angle Prediction. The angle prediction and corresponding loss are the main
contributions for TransVTSpotter. As shown in Table 6, we conduct three
experiments to test the effectiveness of the angle prediction and
corresponding cosine loss. Without using angle prediction, model with upright-
bounding box(two points) results show a dissatisfactory performance(_i.e.,_
$68.5\%$ f-score for detection and $36.1\%$ ${\rm ID_{F1}}$ for tracking).
Compared with using angle prediction, there is around $20\%$ performance gap.
Besides, using cosine loss present a better performance than the counterpart
of L1 loss($84.2\%$ v.s $83.2\%$ for text detection F-score)

Tracking Query Tracking query is important for our framework, which uses the
knowledge of previously detected objects to obtain a set of tracking boxes. As
shown in Table 7, for only object query, with only learned object query is
input as decoder query, and associating the generated detection to track. This
solution achieves a not bad performance by $54.2$ ${\rm ID_{F1}}$. But, with
tracking query, the performance present further improvement($3.1\%$ ${\rm
ID_{F1}}$)

Accuracy vs. Speed We analyze the inference speed of TransVTSpotter. The time
cost is measured using a single Tesla V100 GPU. Table 7 shows the effect of
input image size. With input image size increasing, the model present a better
${\rm ID_{F1}}$ performance from $52.6\%$ to $57.3\%$. When the short-side of
the input image is by 800 pixels, the speech and performance all present
relatively satisfactory results, so we set it as the default setting in the
experiment.

Table 8: Experiments for TransVTSpotter on ICDAR2015(video), Minetto and YVT. ‘D’,‘R’, ‘T’ and ‘S’ denotes the Detection, Recognition, Tracking, Spotting, respectively. ‘video’ or ‘image’ denote the image or video level dataset. $\rm Method$ | ICDAR2015(video,T)/% | ICDAR2015(video,S)/% |  Minetto [37](video,T)/% |  YVT [40](video,T)/%
---|---|---|---|---
$\rm MOTA$ | $\rm MOTP$ | ${\rm ID_{F1}}$ | $\rm MOTA$ | $\rm MOTP$ | ${\rm ID_{F1}}$ | $\rm MOTA$ | $\rm MOTP$ | ${\rm ID_{F1}}$ | $\rm MOTA$ | $\rm MOTP$ | ${\rm ID_{F1}}$
USTB_TexVideo [20] | 7.4 | 70.7 | 25.9 | 15.6 | 68.5 | 28.2 | - | - | - | - | - | -
StradVision-1 [20] | 7.9 | 70.2 | 25.8 | 8.9 | 70.2 | 31.9 | - | - | - | - | - | -
USTB_TexVideo II-2 [20] | 12.3 | 71.8 | 21.9 | 13.2 | 66.6 | 21.3 | - | - | - | - | - | -
AJOU [20, 22] | 16.4 | 72.7 | 36.1 | - | - | - | - | - | - | - | - | -
AGD with AGD [71] | - | - | - | - | - | - | 75.6 | 74.7 | - | - | - | -
AGD with EAGD [71] | - | - | - | - | - | - | 81.3 | 75.7 | - | - | - | -
Wei et al. [12] | - | - | - | - | - | - | 83.5 | 76.8 | - | - | - | -
Free [7] | 43.2 | 76.8 | 57.9 | 52.9 | 74.8 | 61.8 | - | - | - | 54.0 | 78.0 | -
TransVTSpotter(ours) | 44.1 | 75.8 | 57.3 | 53.2 | 74.9 | 61.5 | 84.1 | 77.6 | 74.7 | 53.9 | 75.9 | 64.5

###  6.4 Comparison with State-of-The-Arts for TransVTSpotter

As shown in Table 8, we evaluate TransSpotter on ICDAR2015(video) [77],
Minetto(video) [37] and YVT(video) [40] for tracking and video text spotting
task. Minetto consists of 5 videos in outdoor scenes. The frame size is 640 x
480 and all videos are used for test when the model training on
ICDAR2015(video). Our TransVTSpotter obtains $44.1\%$ and $84.1\%$ for
tracking task(MOTA) on ICDAR2015(video) and Minetto, at least $0.6\%$
improvement than the previous models.

###  6.5 Link to Other Video-and-Language Applications

In this section, we show that the practicability of the proposed BOVText, not
a toy benchmark, which can promote other video-and-text application research.

Text spotting in static images has numerous application scenarios: (1)
Automatic data entry. SF-Express 141414https://www.sf-express.com/cn/sc/
utilizes OCR techniques to accelerate the data entry process. NEBO
151515https://www.myscript.com/nebo/ performs instant transcription as the
user writes down notes. (2) Autonomous vehicle [36, 35]. Text-embedded panels
carry important information, _e.g.,_ geo-location, current traffic condition,
navigation, and etc. (3) Text-based reading comprehension. TextCaps [49] and
text-based VQA [50, 3] show the new vision-and-language tasks, which need to
recognize text, relate it to its visual context, semantic, and visual
reasoning between multiple text tokens and visual entities, such as objects.
Similarly, there are many application demands for video text understanding
across various industries and in our daily lives. We list the most outstanding
ones that significantly impact, improving our productivity and life quality.
Firstly, automatically describing video with natural language [66, 62] can
bridge video and language. Secondly, video text automatic translation
161616https://translate.google.com/intl/en/about/ can be extremely helpful as
people travel, and help video-sharing websites 171717https://www.youtube.com/
to cut down language barriers. Finally, text-based video retrieval [24, 29] is
an irreplaceable business for many companies, such as Google and YouTube. More
details and analyses for application scenarios concerning BOVText in the
supplementary material.

Video Understanding. As shown in Figure. 6 (a), the example concerning the
task of describing video with natural language is from MSR-VTT [66], and there
has been increasing interest in video understanding [62, 27]. However, video
description with only visual information is difficult and limited, even for a
human. For the annotation of the sample video, _i.e.,_ "A man in a blue suit
and purple tie discusses millennial investing fear", we can not learn the
information of "millennial investing fear" from the visual information in the
video. By comparison, caption and scene texts in the video contain accurate
information of "millennial investing fear", which can help the model to
describe the video better. We argue that the same as general human
understanding, videos without captions and audio, is difficult to be properly
understood by the model. We hope the release of BOVText can promote efficient
video text reading, further enhancing automatic video description.

Video Text Automatic Translation. Another practical application is video text
automatic translation, as shown in Figure. 6 (b). The application may be
unnecessary for several professional teams or classic movies due to the
professional translator or huge cost investment. But for international video-
sharing websites 181818https://www.youtube.com/
191919https://www.kuaishou.com/en with millions of users, it isn’t easy to
apply multilingual caption and scene text in billions of videos. Therefore,
efficient translation concerning caption text (e.g., overlap, song title,
logos) and scene text (e.g., street signs, business signs, words on shirt)
still need further exploration and research. The large-scale and multilingual
BOVText contributes various real scenarios for the development of video text
automatic translation.

Figure 6: The Real Application Tasks Link to BOVText. (a) Video Understanding,
automatically describing visual content with natural language. (b) Video
Caption Translation, extremely helpful for people who travel abroad and video-
sharing websites such as YouTube. (c) Video Retrieval, accurate semantic
information for text in videos can promote video retrieval.

Video Retrieval. Video retrieval with textual cues [38, 55] is also a very
important application direction for video-and-text research, as shown in
Figure. 6 (c). To the best of my knowledge, video retrieval with text
information in the video is still almost a blank field of study and immature
application in the industry. The most existing video retrieval methods are
stiff combinations of text detection and recognition, invalid for the example
with a sentence query. Besides, similar to video understanding, for the query
of the sample video, _i.e.,_ "The lakers play host to Golden State", we can
not obtain the correct related video without scene text or caption
information. The missing information needs to recover by understanding the
video with key video text information such as "GOLDEN STATE WARRIORS, LOS
ANGELES LAKERS". The proposed BOVText with various text types (_e.g.,_
caption, song title, logos, street signs, business signs) and annotation can
promote the research concerning efficient video retrieval.

####  6.5.1 Limitations

Although the proposed BOVText supports all video text spotting tasks, _i.e.,_
text detection, recognition, tracking end to end video text spotting, the
potential contributions for other tasks still need mining. For example, as
shown in Figure. 6 (c), we do not provide the corresponding annotation
(_i.e.,_ the query for each video) and metrics concerning video retrieval, but
the annotation and metric are easy to obtain due to text spotting annotation
already existed. Therefore, there are still many potential contributions for
other tasks on BOVText, we want to take these as the future research
directions and provide a complete solution method.

###  6.6 License and Copyright

The released video dataset includes two parts: $1,494$ videos from KuaiShou
202020https://www.kuaishou.com/en and $356$ videos from YouTube
212121https://www.youtube.com/. For those videos from KuaiShou, we mask the
private information such as the human face, which has passed the examination
of the legal department and copyright department of KuaiShou corporation.
Thus, we own the copyright for these videos. For those videos from YouTube, to
the best of our knowledge at the time of download, we have exercised caution
to download only those videos that were available on YouTube with a Creative
Commmons CC-BY (v3.0) License. We don’t own the copyright of those videos and
provide them for non-commercial research purposes only. All data in our
project is open source under CC-by 4.0 license and only be used for research
purposes.

## References

  * [1] Christos-Nikolaos E Anagnostopoulos, Ioannis E Anagnostopoulos, Ioannis D Psoroulas, Vassili Loumos, and Eleftherios Kayafas.  License plate recognition from still images and video sequences: A survey.  IEEE Transactions on intelligent transportation systems, 9(3):377–391, 2008.
  * [2] Keni Bernardin and Rainer Stiefelhagen.  Evaluating multiple object tracking performance: the clear mot metrics.  EURASIP Journal on Image and Video Processing, 2008:1–10, 2008\.
  * [3] Ali Furkan Biten, Ruben Tito, Andres Mafla, Lluis Gomez, Marçal Rusinol, Ernest Valveny, CV Jawahar, and Dimosthenis Karatzas.  Scene text visual question answering.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 4291–4301, 2019.
  * [4] Yuanqiang Cai, Weiqiang Wang, Shao Huang, Jin Ma, and Ke Lu.  Spatiotemporal text localization for videos.  Multimedia Tools and Applications, 77(22):29323–29345, 2018.
  * [5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European Conference on Computer Vision, pages 213–229. Springer, 2020.
  * [6] Zhanzhan Cheng, Jing Lu, Yi Niu, Shiliang Pu, Fei Wu, and Shuigeng Zhou.  You only recognize once: Towards fast video text spotting.  In ACM International Conference on Multimedia, pages 855–863, 2019.
  * [7] Zhanzhan Cheng, Jing Lu, Baorui Zou, Liang Qiao, Yunlu Xu, Shiliang Pu, Yi Niu, Fei Wu, and Shuigeng Zhou.  Free: A fast and robust end-to-end video text spotter.  IEEE Transactions on Image Processing, 30:822–837, 2020.
  * [8] Chee Kheng Ch’ng and Chee Seng Chan.  Total-text: A comprehensive dataset for scene text detection and recognition.  In IEEE International Conference on Document Analysis and Recognition, pages 935–942, 2017.
  * [9] Patrick Dendorfer, Hamid Rezatofighi, Anton Milan, Javen Shi, Daniel Cremers, Ian Reid, Stefan Roth, Konrad Schindler, and Laura Leal-Taixe.  Cvpr19 tracking and detection challenge: How crowded can it get?  arXiv preprint arXiv:1906.04567, 2019.
  * [10] Jianfeng Dong, Xirong Li, Chaoxi Xu, Xun Yang, Gang Yang, Xun Wang, and Meng Wang.  Dual encoding for video retrieval by text.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [11] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [12] Wei Feng, Fei Yin, Xu-Yao Zhang, and Cheng-Lin Liu.  Semantic-aware video text detection.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1695–1705, 2021.
  * [13] Xavier Glorot and Yoshua Bengio.  Understanding the difficulty of training deep feedforward neural networks.  In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249–256, 2010.
  * [14] Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman.  Synthetic data for text localisation in natural images.  In IEEE Conference on Computer Vision and Pattern Recognition, pages 2315–2324, 2016.
  * [15] Tong He, Zhi Tian, Weilin Huang, Chunhua Shen, Yu Qiao, and Changming Sun.  An end-to-end textspotter with explicit alignment and attention.  In IEEE conference on computer vision and pattern recognition, pages 5020–5029, 2018.
  * [16] Xian-Sheng Hua, Pei Yin, and Hong-Jiang Zhang.  Efficient video text recognition using multiple frame integration.  In Proceedings. International Conference on Image Processing, volume 2, pages II–II. IEEE, 2002.
  * [17] Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Synthetic data and artificial neural networks for natural scene text recognition.  arXiv preprint arXiv:1406.2227, 2014.
  * [18] Max Jaderberg, Karen Simonyan, Andrew Zisserman, and Koray Kavukcuoglu.  Spatial transformer networks.  In Neural Information Processing Systems, pages 2017–2025, 2015\.
  * [19] Keechul Jung, Kwang In Kim, and Anil K Jain.  Text information extraction in images and video: a survey.  Pattern recognition, 37(5):977–997, 2004.
  * [20] Dimosthenis Karatzas, Lluis Gomez-Bigorda, Anguelos Nicolaou, Suman Ghosh, Andrew Bagdanov, Masakazu Iwamura, Jiri Matas, Lukas Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, et al.  Icdar 2015 competition on robust reading.  In IEEE International Conference on Document Analysis and Recognition, pages 1156–1160, 2015.
  * [21] Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, Masakazu Iwamura, Lluis Gomez i Bigorda, Sergi Robles Mestre, Joan Mas, David Fernandez Mota, Jon Almazan Almazan, and Lluis Pere De Las Heras.  Icdar 2013 robust reading competition.  In IEEE International Conference on Document Analysis and Recognition, pages 1484–1493, 2013.
  * [22] Hyung Il Koo and Duck Hoon Kim.  Scene text detection via connected component clustering and nontext filtering.  IEEE transactions on image processing, 22(6):2296–2305, 2013.
  * [23] Harold W Kuhn.  The hungarian method for the assignment problem.  Naval research logistics quarterly, 2(1-2):83–97, 1955.
  * [24] Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal.  Tvr: A large-scale dataset for video-subtitle moment retrieval.  In Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part XXI 16, pages 447–463. Springer, 2020.
  * [25] Hui Li, Peng Wang, and Chunhua Shen.  Towards end-to-end text spotting with convolutional recurrent neural networks.  In IEEE International Conference on Computer Vision, pages 5238–5246, 2017.
  * [26] Yuan Li, Chang Huang, and Ram Nevatia.  Learning to associate: Hybridboosted multi-target tracker for crowded scene.  In 2009 IEEE conference on computer vision and pattern recognition, pages 2953–2960. IEEE, 2009.
  * [27] Yuncheng Li, Yale Song, Liangliang Cao, Joel Tetreault, Larry Goldberg, Alejandro Jaimes, and Jiebo Luo.  Tgif: A new dataset and benchmark on animated gif description.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4641–4650, 2016.
  * [28] Minghui Liao, Zhaoyi Wan, Cong Yao, Kai Chen, and Xiang Bai.  Real-time scene text detection with differentiable binarization.  In AAAI Conference on Artificial Intelligence, pages 11474–11481, 2020.
  * [29] Jingzhou Liu, Wenhu Chen, Yu Cheng, Zhe Gan, Licheng Yu, Yiming Yang, and Jingjing Liu.  Violin: A large-scale dataset for video-and-language inference.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10900–10910, 2020.
  * [30] Xuebo Liu, Ding Liang, Shi Yan, Dagui Chen, Yu Qiao, and Junjie Yan.  Fots: Fast oriented text spotting with a unified network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5676–5685, 2018.
  * [31] Shangbang Long, Xin He, and Cong Yao.  Scene text detection and recognition: The deep learning era.  International Journal of Computer Vision, 129(1):161–184, 2021\.
  * [32] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [33] Ning Lu, Wenwen Yu, Xianbiao Qi, Yihao Chen, Ping Gong, Rong Xiao, and Xiang Bai.  Master: Multi-aspect non-local network for scene text recognition.  Pattern Recognition, 117:107980, 2021.
  * [34] Pengyuan Lyu, Minghui Liao, Cong Yao, Wenhao Wu, and Xiang Bai.  Mask textspotter: An end-to-end trainable neural network for spotting text with arbitrary shapes.  In European Conference on Computer Vision, pages 67–83, 2018.
  * [35] Abdelhamid Mammeri, Azzedine Boukerche, et al.  Mser-based text detection and communication algorithm for autonomous vehicles.  In 2016 IEEE symposium on computers and communication (ISCC), pages 1218–1223. IEEE, 2016.
  * [36] Abdelhamid Mammeri, El-Hebri Khiari, and Azzedine Boukerche.  Road-sign text recognition architecture for intelligent transportation systems.  In 2014 IEEE 80th Vehicular Technology Conference (VTC2014-Fall), pages 1–5. IEEE, 2014.
  * [37] Rodrigo Minetto, Nicolas Thome, Matthieu Cord, Neucimar J Leite, and Jorge Stolfi.  Snoopertrack: Text detection and tracking for outdoor videos.  In IEEE International Conference on Image Processing, pages 505–508, 2011.
  * [38] Anand Mishra, Karteek Alahari, and CV Jawahar.  Image retrieval using textual cues.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3040–3047, 2013.
  * [39] Nibal Nayef, Fei Yin, Imen Bizid, Hyunsoo Choi, Yuan Feng, Dimosthenis Karatzas, Zhenbo Luo, Umapada Pal, Christophe Rigaud, Joseph Chazalon, et al.  Icdar2017 robust reading challenge on multi-lingual scene text detection and script identification-rrc-mlt.  In IEEE International Conference on Document Analysis and Recognition, volume 1, pages 1454–1459, 2017.
  * [40] Phuc Xuan Nguyen, Kai Wang, and Serge Belongie.  Video text detection and recognition: Dataset and benchmark.  In IEEE winter conference on applications of computer vision, pages 776–783, 2014.
  * [41] Sangeeth Reddy, Minesh Mathew, Lluis Gomez, Marçal Rusinol, Dimosthenis Karatzas, and CV Jawahar.  Roadtext-1k: Text detection & recognition dataset for driving videos.  In IEEE International Conference on Robotics and Automation, pages 11074–11080, 2020.
  * [42] Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun.  Faster r-cnn: Towards real-time object detection with region proposal networks.  Advances in neural information processing systems, 28:91–99, 2015\.
  * [43] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese.  Generalized intersection over union: A metric and a loss for bounding box regression.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 658–666, 2019.
  * [44] Ergys Ristani, Francesco Solera, Roger Zou, Rita Cucchiara, and Carlo Tomasi.  Performance measures and a data set for multi-target, multi-camera tracking.  In Workshops of European conference on computer vision, pages 17–35, 2016.
  * [45] Xuejian Rong, Chucai Yi, Xiaodong Yang, and Yingli Tian.  Scene text recognition in multiple frames based on text tracking.  In 2014 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2014.
  * [46] Georg Schroth, Sebastian Hilsenbeck, Robert Huitl, Florian Schweiger, and Eckehard Steinbach.  Exploiting text-related features for content-based image retrieval.  In 2011 IEEE international symposium on multimedia, pages 77–84. IEEE, 2011.
  * [47] Baoguang Shi, Xiang Bai, and Cong Yao.  An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition.  IEEE transactions on pattern analysis and machine intelligence, 39(11):2298–2304, 2016.
  * [48] Baoguang Shi, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai.  Robust scene text recognition with automatic rectification.  In IEEE conference on computer vision and pattern recognition, pages 4168–4176, 2016.
  * [49] Oleksii Sidorov, Ronghang Hu, Marcus Rohrbach, and Amanpreet Singh.  Textcaps: a dataset for image captioning with reading comprehension.  In European Conference on Computer Vision, pages 742–758. Springer, 2020.
  * [50] Amanpreet Singh, Vivek Natarajan, Meet Shah, Yu Jiang, Xinlei Chen, Dhruv Batra, Devi Parikh, and Marcus Rohrbach.  Towards vqa models that can read.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8317–8326, 2019.
  * [51] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov.  Unsupervised learning of video representations using lstms.  In International Conference on Machine Learning, pages 843–852, 2015.
  * [52] Peize Sun, Yi Jiang, Rufeng Zhang, Enze Xie, Jinkun Cao, Xinting Hu, Tao Kong, Zehuan Yuan, Changhu Wang, and Ping Luo.  Transtrack: Multiple-object tracking with transformer.  arXiv preprint arXiv:2012.15460, 2020.
  * [53] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Advances in neural information processing systems, pages 5998–6008, 2017.
  * [54] Andreas Veit, Tomas Matera, Lukas Neumann, Jiri Matas, and Serge Belongie.  Coco-text: Dataset and benchmark for text detection and recognition in natural images.  arXiv preprint arXiv:1601.07140, 2016.
  * [55] Hao Wang, Xiang Bai, Mingkun Yang, Shenggao Zhu, Jing Wang, and Wenyu Liu.  Scene text retrieval via joint text detection and similarity learning.  arXiv preprint arXiv:2104.01552, 2021.
  * [56] Jianfeng Wang and Xiaolin Hu.  Gated recurrent convolution neural network for ocr.  In Neural Information Processing Systems, pages 334–343, 2017.
  * [57] Kai Wang, Boris Babenko, and Serge Belongie.  End-to-end scene text recognition.  In 2011 International Conference on Computer Vision, pages 1457–1464. IEEE, 2011.
  * [58] Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li.  Transformer meets tracker: Exploiting temporal context for robust visual tracking.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 1571–1580, 2021.
  * [59] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  arXiv preprint arXiv:2102.12122, 2021.
  * [60] Wenhai Wang, Enze Xie, Xiang Li, Wenbo Hou, Tong Lu, Gang Yu, and Shuai Shao.  Shape robust text detection with progressive scale expansion network.  In IEEE conference on computer vision and pattern recognition, pages 9336–9345, 2019.
  * [61] Xiaobing Wang, Yingying Jiang, Shuli Yang, Xiangyu Zhu, Wei Li, Pei Fu, Hua Wang, and Zhenbo Luo.  End-to-end scene text recognition in videos based on multi frame tracking.  In IEEE International Conference on Document Analysis and Recognition, pages 1255–1260, 2017.
  * [62] Xin Wang, Jiawei Wu, Junkun Chen, Lei Li, Yuan-Fang Wang, and William Yang Wang.  Vatex: A large-scale, high-quality multilingual dataset for video-and-language research.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4581–4591, 2019.
  * [63] Yuqing Wang, Zhaoliang Xu, Xinlong Wang, Chunhua Shen, Baoshan Cheng, Hao Shen, and Huaxia Xia.  End-to-end video instance segmentation with transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8741–8750, 2021.
  * [64] Liang Wu, Palaiahnakote Shivakumara, Tong Lu, and Chew Lim Tan.  A new technique for multi-oriented scene text line detection and tracking in video.  IEEE Transactions on multimedia, 17(8):1137–1152, 2015.
  * [65] Jie Xi, Xian-Sheng Hua, Xiang-Rong Chen, Liu Wenyin, and Hong-Jiang Zhang.  A video text detection and recognition system.  In IEEE International Conference on Multimedia and Expo, 2001. ICME 2001., pages 222–222. IEEE Computer Society, 2001.
  * [66] Jun Xu, Tao Mei, Ting Yao, and Yong Rui.  Msr-vtt: A large video description dataset for bridging video and language.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5288–5296, 2016.
  * [67] Chun Yang, Xu-Cheng Yin, Wei-Yi Pei, Shu Tian, Ze-Yu Zuo, Chao Zhu, and Junchi Yan.  Tracking based multi-orientation scene text detection: A unified framework with dynamic programming.  IEEE Transactions on Image Processing, 26(7):3235–3248, 2017\.
  * [68] Jian Yi, Yuxin Peng, and Jianguo Xiao.  Using multiple frame integration for the text recognition of video.  In 2009 10th International Conference on Document Analysis and Recognition, pages 71–75. IEEE, 2009.
  * [69] Xu-Cheng Yin, Ze-Yu Zuo, Shu Tian, and Cheng-Lin Liu.  Text detection, tracking and recognition in video: a comprehensive survey.  IEEE Transactions on Image Processing, 25(6):2752–2773, 2016\.
  * [70] Fisher Yu, Wenqi Xian, Yingying Chen, Fangchen Liu, Mike Liao, Vashisht Madhavan, and Trevor Darrell.  Bdd100k: A diverse driving video database with scalable annotation tooling.  arXiv preprint arXiv:1805.04687, 2018.
  * [71] Hongyuan Yu, Yan Huang, Lihong Pi, Chengquan Zhang, Xuan Li, and Liang Wang.  End-to-end video text detection with online tracking.  Pattern Recognition, 113:107791, 2021.
  * [72] Oussama Zayene, Mathias Seuret, Sameh Masmoudi Touj, Jean Hennebert, Rolf Ingold, and Najoua Essoukri Ben Amara.  Text detection in arabic news video based on SWT operator and convolutional auto-encoders.  In Workshop on Document Analysis Systems, pages 13–18, 2016.
  * [73] Fangao Zeng, Bin Dong, Tiancai Wang, Cheng Chen, Xiangyu Zhang, and Yichen Wei.  Motr: End-to-end multiple-object tracking with transformer.  arXiv preprint arXiv:2105.03247, 2021.
  * [74] Hengshuang Zhao, Li Jiang, Jiaya Jia, Philip Torr, and Vladlen Koltun.  Point transformer.  arXiv preprint arXiv:2012.09164, 2020.
  * [75] Sixiao Zheng, Jiachen Lu, Hengshuang Zhao, Xiatian Zhu, Zekun Luo, Yabiao Wang, Yanwei Fu, Jianfeng Feng, Tao Xiang, Philip HS Torr, et al.  Rethinking semantic segmentation from a sequence-to-sequence perspective with transformers.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6881–6890, 2021.
  * [76] Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang.  East: an efficient and accurate scene text detector.  In IEEE conference on computer vision and pattern recognition, pages 5551–5560, 2017.
  * [77] Xinyu Zhou, Shuchang Zhou, Cong Yao, Zhimin Cao, and Qi Yin.  Icdar 2015 text reading in the wild competition.  arXiv preprint arXiv:1506.03184, 2015.
  * [78] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  arXiv preprint arXiv:2010.04159, 2020.