# DSText V2: A Comprehensive Video Text Spotting Dataset for Dense and Small
Text

Weijia Wua, Yiming Zhanga, Yefei Hea, Luoming Zhanga, Zhenyu Loua, Hong
Zhoua(✉), and Xiang Baib

###### Abstract

Recently, video text detection, tracking, and recognition in natural scenes
are becoming very popular in the computer vision community. However, most
existing algorithms and benchmarks focus on common text cases (_e.g.,_ normal
size, density) and single scenario, while ignoring extreme video text
challenges, _i.e.,_ dense and small text in various scenarios. In this paper,
we establish a video text reading benchmark, named DSText V2, which focuses on
Dense and Small text reading challenges in the video with various scenarios.
Compared with the previous datasets, the proposed dataset mainly include three
new challenges: 1) Dense video texts, a new challenge for video text spotters
to track and read. 2) High-proportioned small texts, coupled with the
blurriness and distortion in the video, will bring further challenges. 3)
Various new scenarios, _e.g.,_ ‘Game’, ‘Sports’, etc. The proposed DSText V2
includes 140 video clips from 7 open scenarios, supporting three tasks,
_i.e.,_ video text detection (Task 1), video text tracking (Task 2), and end-
to-end video text spotting (Task 3).

In this article, we describe detailed statistical information of the dataset,
tasks, evaluation protocols, and the results summaries. Most importantly, a
thorough investigation and analysis targeting three unique challenges derived
from our dataset are provided, aiming to provide new insights. Moreover, we
hope the benchmark will promise video text research in the community. DSText
v2 is built upon DSText v1, which was previously introduced to organize the
ICDAR 2023 competition for dense and small video text. The dataset website can
be found at $\tt RRC$ or $\tt Zenodo$.

###### keywords:

Video Text Spotting; Small Text; Text Tracking; Dense Text

††journal: Pattern Recognition

organization=Zhejiang University,city=HangZhou, country=China
organization=Huazhong University of Science and Technology,city=WuHan,
country=China

##  1 Introduction

The field of reading text from static images has witnessed remarkable progress
in recent years, thanks to advancements in deep learning and the availability
of extensive public datasets such as MJSynth Jaderberg et al. (2014),
SynthText Gupta et al. (2016), ICDAR2015 Karatzas et al. (2015), FlowText Zhao
et al. (2023), and Total-Text Ch’ng and Chan (2017), as well as various
algorithms including PSENet Wang et al. (2019), EAST Zhou et al. (2017),
MaskTextSpotter Lyu et al. (2018), , Polygon-Free Wu et al. (2022) , DText Cai
et al. (2022), TextMountain Zhu and Du (2021), TextCohesion Wu et al. (2019),
and FOTS Liu et al. (2018). In contrast, the progress in video-level text
spotting Yin et al. (2016) has been notably slow, which limited numerous
applications of video text, _e.g.,_ video understanding Srivastava et al.
(2015), video retrieval Dong et al. (2021); Wu et al. (2023), video text
translation, and license plate recognition Anagnostopoulos et al. (2008), etc.
There have been a few previous video text spotting benchmarks attempting to
develop video text spotting, which focuses on easy cases, _e.g.,_ normal text
size, and density in a single scenario. ICDAR2015 (Text in Videos) Karatzas et
al. (2015), as the most popular benchmark, was introduced during the ICDAR
Robust Reading Competition in 2015 focus on wild scenarios: walking outdoors,
searching for a shop in a shopping street, etc. YouTube Video Text (YVT)
Nguyen et al. (2014) contains 30 videos from YouTube. The text category mainly
includes overlay text (caption) and scene text (_e.g.,_ driving signs,
business signs). RoadText-1K Reddy et al. (2020) provides 1,000 driving
videos, which promote driver assistance and self-driving systems. LSVTD Cheng
et al. (2019) proposes 100 text videos, 13 indoor (_e.g.,_ bookstore, shopping
mall) and 9 outdoor (_e.g.,_ highway, city road) scenarios, and support two
languages, _i.e.,_ English and Chinese. BOVText Wu et al. (2021) establishes a
large-scale, bilingual video text benchmark, including abundant text types,
_i.e.,_ title, caption, or scene text.

Figure 1: Visualization of DSText. Different from previous benchmarks, DSText
focuses on dense and small text challenges. Each frame of the video contains
dense small texts, coupled with camera movements, posing unique challenges for
detection, tracking, and recognition models.

However, as shown in Figure 1, the above benchmarks still suffer from some
limitations: 1) Most text instances present standard text size without
challenge, _e.g.,_ ICDAR2015(video) YVT, BOVText. 2) Sparse text density in a
single scenario, _e.g.,_ RoadText-1k, and YVT, which can not evaluate the
small and dense text robustness of the algorithm effectively. 3) Most
benchmarks present unsatisfactory maintenance. YVT, RoadText-1k, and BOVText
all do not launch a corresponding competition and release open-source
evaluation scripts. Besides, the download links of YVT even have become
invalid. The poor maintenance plan is not helpful to the development of video
text tasks in the community. To break these limitations, in this work, we
establish one new benchmark, which focuses on dense and small texts in various
scenarios, as shown in Figure 1. Our dataset has several advantages and unique
challenges. Firstly, the high-quality, high-resolution, and various videos
with dense and small texts (_i.e.,_ 140 videos, 62.1k video frames, and 2.2m
text instances) are collected from YouTube enabling the development of deep
design specific for video text spotting. Secondly, Each frame of the video
contains a significant amount of small text, which presents significant
challenges for detection, tracking, and recognition models when encountering
distortions and motion blur. Thirdly, The average text density per frame
reaches a high value of $24$, significantly surpassing the maximum text
density of previous datasets ($5.55$ from ICDAR 2015). This poses significant
challenges for tracking models, as the high text density can lead to ID
switches in tracking, resulting in a decrease in tracking accuracy. The
benchmark mainly supports three tasks, _i.e.,_ video text detection, video
text tracking, and end-to-end video text spotting tasks, including 140 videos
with 62.1k frames.

To further advance the field, we also organize the ICDAR 2023 Video Text
Reading competition for dense and small text, which includes two competition
tracks: video text tracking, and spotting tasks. This competition can serve as
a standard benchmark for assessing the robustness of algorithms that are
designed for video text spotting in complex natural scenes, which is more
challenging. In this paper, we have expanded the dataset with an additional 40
videos, resulting in a total of 140 videos. And we also provide comprehensive
data analysis, experimental analysis, and additional insights into the unique
challenges posed by our dataset. In the Section 3 of this paper, we provide a
detailed overview of how the V2 version of the dataset was constructed,
annotated, and present comprehensive information on the data distribution and
statistical analysis. In the Section 4, we list all tasks and metrics, along
with a brief analysis. Section 5 includes various experimental comparisons and
introduces new insights.

The main contributions of this work are three folds:

  * 1.

We collect and annotate a high-quality, high-resolution video text benchmark
with various video domains, which includes $140$ videos, $62.1k$ video frames,
and $2.2m$ text instances.

  * 2.

Compared to the current existing video text reading datasets, the proposed
DSText V2 provides some unique features and challenges, including 1) Abundant
video scenarios, high-quality videos, 2) Massive and high-proportion of small
text, and 3) dense text distribution per frame.

  * 3.

We provide comprehensive data analysis, experimental analysis, and additional
insights into the unique challenges of our dataset, enabling future
researchers to better understand and leverage its potential.

##  2 Related Work

In this section, we provide a concise overview of the relevant literature and
benchmarks related to our work, specifically focusing on end-to-end text-
spotting methods and corresponding benchmarks.

###  2.1 Image Text Spotting

Numerous image-level deep learning-based approaches Li et al. (2017); He et
al. (2018); Lyu et al. (2018); Liu et al. (2020, 2023); Lu et al. (2021);
Zheng et al. (2020); Wu et al. (2020a, b, 2023) have been introduced to tackle
the task of image text spots, resulting in significant performance
improvements. Li et al. (2017) pioneered the development of an end-to-end
trainable scene text spotting method. Their approach effectively integrates
detection and recognition features using RoI Pooling Ren et al. (2015),
achieving notable success. Lyu et al. (2018) introduced a novel approach
called Mask TextSpotter, which extends the capabilities of Mask R-CNN by
incorporating character-level supervision for simultaneous character detection
and recognition. In addition, to mature algorithms, there are also numerous
excellent datasets Zhou et al. (2015); Veit et al. (2016); Ch’ng and Chan
(2017); Karatzas et al. (2013) available that contribute significantly to the
advancement of research in this field. These datasets provide a diverse range
of annotated images, offering valuable resources for training and evaluating
text detection and recognition models. Some notable datasets such as ICDAR
2015 Zhou et al. (2015) benchmarks and COCOText Veit et al. (2016), which
offer a wide variety of text instances in different contexts. This data is
collected using Google Glasses, which captures a wide range of diverse scenes,
including but not limited to street views, indoor environments, and shopping
malls. Furthermore, there are emerging datasets like Total-Text Ch’ng and Chan
(2017) that focus on curved and irregular text, pushing the boundaries of
existing algorithms and fostering innovation in the field of text analysis.

###  2.2 Video Text Spotting

Unlike image text detection, the development of video-level tracking in the
context of text spotting has been relatively slow. In recent years, there have
been several efforts Yu et al. (2021); Tu et al. (2018); Yin et al. (2016);
Rong et al. (2014); Wu et al. (2021); Wang et al. (2017) to address the
challenges of video text tracking and spotting. Researchers have explored
various approaches to overcome the complexities associated with tracking text
in videos, including visual feature association Yu et al. (2021), learnable
query embedding Wu et al. (2021), detection bounding box association Rong et
al. (2014). Nguyen et al. (2014) performs character detection and recognition
via scanning-window templates trained with mixture models. Yu et al. (2021)
leverage the advantages of contrastive learning to track text across
consecutive frames. They employ a contrastive loss to minimize the distance
between the embeddings of the same text instance across frames while
maximizing the distance between embeddings of different text instances. Wang
et al. (2017) propose a method to link text across adjacent frames using a
combination of detection box IoU and transcription-based edit distance
matching. Cheng et al. (2019) and Free Cheng et al. (2020) propose a video
text spotting framework that performs text recognition only once for the
localized text. This approach reduces the computational costs and improves
efficiency. Wu et al. (2021) propose a video text spotting approach that
utilizes object features from the previous frame as a tracking query for the
current frame. They employ IoU matching to obtain accurate text tracking and
train a separate recognition model to obtain the final recognition results.
TransDETR Wu et al. (2022) was the first to propose an end-to-end framework
that addresses the tasks of detection, tracking, and recognition in a unified
framework. This method employs a query embedding to represent a text instance
across multiple frames, effectively modeling the long sequence relationship.
Overall, end-to-end video text spotting remains scarce and there is still
significant room for improvement in terms of speed and accuracy.

Table 1: Statistical Comparison. ‘Box’, and ‘Text Area’ denote the box annotation type and average area (# pixels) of text while the shorter side of the image is 720 pixels. ‘Text Density’ refers to the average text number per frame.  Dataset | Video | Frame | Text | Box |  Text Area |  Text Density |  Supported Scenario (Domain)
---|---|---|---|---|---|---|---
AcTiV-D Zayene et al. (2016) | 8 | 1,843 | 5,133 | Upright |  - |  - |  News video
YVT Nguyen et al. (2014) | 30 | 13k | 16k | Upright |  8,664 |  1.15 |  Cartoon, Outdoor(supermarket, shopping street, driving…)
ICD15 VTZhou et al. (2015) | 51 | 27k | 144k | Oriented |  5,013 |  5.33 |  Driving, Supermarket, Shopping street…
RoadTextReddy et al. (2020) | 1k | 300k | 1.2m | Upright |  2,141 |  0.75 |  Driving
LSVTDCheng et al. (2019) | 100 | 66k | 569k | Oriented |  2,254 |  5.52 |  Shopping mall, Supermarket, Hotel…
BOVText Wu et al. (2021) | 2k | 1.7m | 8.8m | Oriented |  10,309 |  5.12 |  Cartoon, Vlog, Travel, Game, Sport, News …
DSText V1 Wu et al. (2023) | 100 | 56.0k | 671k | Oriented |  1,984 |  23.5 |  Driving, Activity, Street View (indoor), Street View (outdoor), Sports, News, Movie, Cooking
DSText V2 | 140 | 62.1k | 2.2m | Oriented |  1,758 |  42.4 |  Driving, Activity, Street View (indoor), Street View (outdoor), Game, Sports, Unknown

###  2.3 Video Text Dataset

The progress in video text spotting has been limited in recent years due to
the scarcity of efficient datasets. The ICDAR 2015 Video dataset Zhou et al.
(2015) comprises only 25 training videos and 25 test videos. The videos are
categorized into a few specific scenarios, such as walking outdoors or
searching for a shop in a street, where the majority of the text instances are
considered relatively easy cases, as they do not pose significant challenges
in terms of text size, density, or other characteristics. The Minetto Dataset
Minetto et al. (2011) is a relatively small dataset that includes only 5
videos captured in outdoor scenes. The frame size of the videos is 640 x 480
pixels. YVT Nguyen et al. (2014), consists of a total of 30 videos. Out of
these, 15 videos are used for training, and the remaining 15 videos are
designated for testing. RoadText-1K Reddy et al. (2020) offers a collection of
driving videos consisting of 1000 videos, which are sampled from the BDD100K
Yu et al. (2018). The dataset primarily focuses on the driving scenario,
making it specific to road scenes. BOVText Wu et al. (2021), is a bilingual
dataset collected from YouTube and Kuaishou. It comprises over 2,000 videos. A
majority of these videos in this dataset may not exhibit significant
challenges in terms of text spotting, as shown in Figure 1. Existing datasets
primarily focus on normal text and lack challenging examples of dense and
small text. Besides, many datasets suffer from poor maintenance.

Figure 2: The Data Distribution for $7$ Open Scenarios. (a) Video number
distribution (b). The number distribution of video frames. (c) Text instance
distribution. (d) Average text area (# pixels) distribution while the shorter
side of the image is $720$. (e) The distribution of average text number per
frame (density).

##  3 Dataset

###  3.1 Dataset and Annotations

####  3.1.1 Dataset Source.

We collect high-quality videos with dense and small text from three data
sources:

  * 1.

1) $30$ videos sampled from the large-scale video text dataset BOVText Wu et
al. (2021). BOVText as the largest video text dataset, comprising over $2,000$
videos, is a valuable data source. It covers a wide range of scenarios and
includes a substantial number of videos with small and dense text. We employ a
selection process to identify the top $30$ videos with small and dense texts
based on criteria that the average text area within the video and the average
number of text instances per frame. This ensures that we focus on videos that
exhibit a high concentration of small and dense text.

  * 2.

2) We collect $10$ videos for driving scenario from RoadText-1k Reddy et al.
(2020). As shown in Figure 1, the RoadText-1k dataset exhibits a significant
presence of small texts, which aligns with the challenge of our dataset. Thus
we also randomly select $10$ videos to enrich the driving scenario.

  * 3.

3) $100$ videos for other scenarios, such as street view scenes and
supermarkets, are collected from YouTube. Except for BOVText and RoadText-1k,
we also need more high-quality videos with dense and small texts for other
scenarios, such as games, street view scenes, and supermarkets. Therefore, we
also collect $100$ videos with dense and small texts from YouTube.

Therefore, we obtain $140$ videos with $62.1$k video frames, as shown in Table
1. Then the dataset is randomly divided into two parts: the training set with
$34,907$ frames from $90$ videos, and the testing set with $27,234$ frames
from $50$ videos. In this paper, we have added an additional $40$ videos to
the training set in DSText expanding the size and diversity of the dataset.
Furthermore, the original $12$ video scenarios have been reorganized into $7$
more representative scenarios. This categorization involved grouping certain
categories with fewer data samples (_i.e.,_ movie, news) into the ”unknown”
category.

Figure 3: The distribution of different text size range on different datasets
”%” denotes the percentage of text size region over the whole data. Text area
(# pixels) is calculated while the shorter side of the image is 720 pixels.
Figure 4: Comparison for frame number of different text numbers. ”%” denotes
the percentage of the corresponding frame over the whole data. The majority of
video frames from the existing datasets only contain 0 to 5 text instances. In
contrast, our dataset includes a significant number of text-dense scenes, with
$58\%$ of video frames containing 15 or more text instances. Figure 5:
Distribution of frame for different text numbers on DSText V2. Frames with
more than 10 texts account for around $70\%$ of the dataset. Figure 6: The
Data Distribution for the newly added 40 videos on DSText V2. Figure 7:
Wordcloud visualizations for DSText V1.

####  3.1.2 Annotation

We adopt two main annotation approaches based on the different sources of
data: 1) For the $30$ videos from BOVText, we just adopt the original
annotation, which includes four kinds of description information: the rotated
bounding box of detection, the tracking identification(ID) of the same text,
the content of the text for recognition, the category of text, _i.e.,_
caption, title, scene text, or others. 2) As for the $110$ videos from
RoadText-1k and YouTube, we hire a professional annotation team to label each
text for each frame. The annotation format is the same as BOVText. For each
video, we extract frames at a rate of 15 frames per second (FPS) and require
each annotator to annotate frame by frame. Then, we invite an audit team with
around 5 persons to carry out another round of annotation checks, and re-label
part video frames with unqualified annotation. We require a bounding box and
text transcription accuracy of over $95\%$ for acceptance. This means that all
the correctly annotated boxes should cover the entire text region, and any
missed or incorrect annotations are considered annotation errors. As for text
transcription, every letter should be accurately transcribed. Similar to the
ICDAR2015 video dataset Zhou et al. (2015), for blurry or non-English texts,
we require the annotators to only annotate the bounding box and tracking ID
while setting the text transcription as ‘ignored’. These texts will not be
considered when calculating tracking or spotting metrics. In other words, the
detection of these texts will not receive any reward in terms of metrics, and
there will be no penalty for missing them. One mentionable point is that the
videos from RoadText-1k only provide the upright bounding box (two points),
thus we abandon the original annotation and annotate these videos with the
oriented bounding box. As a labor-intensive job, the whole labeling process
took 30 men in one a and a half months, _i.e.,_ around 7,200 man-hours, to
complete the 110 video frame annotations. In our dataset, due to the high
density of text in each frame, the average annotation cost per frame is
significantly higher compared to other datasets, requiring approximately three
to four times more time and effort. As shown in Figure 10, it is quite time-
consuming and expensive to annotate a mass of text instances at each frame.

###  3.2 Dataset Comparison and Analysis

In this section, we present detailed statistical data and comparative
analysis. Table 2 presents an overall comparison for the basic information,
_e.g.,_ the number of videos, frame, text, average text area, text density,
and supported scenarios. In comparison with previous works, the proposed
DSText V2 shows the denser text instances density per frame (_i.e.,_ average
$42.4$ texts per frame) and smaller text size (_i.e.,_ average $1,758$ pixels
area of texts). Figure 3, Figure 2, and Figure 4 present detailed data
distributions and comparisons with the existing datasets for DSText v2. In
addition, we have provided comprehensive data analytics for the newly added 40
videos in DSText V2, as illustrated in Figure 6. This includes statistics on
the distribution of video quantities per video scenario, the average text area
distribution per video scenario, and the average text count (per frame)
distribution per video scenario category. Due to the substantial presence of
Street (indoor) and Street (outdoor) scenes in the V1 version, in the V2
version, we primarily increased the number of videos in other scene
categories, such as Activity and Sports scenes. It is noteworthy that all
videos of Unknown video scenario are newly added; this scenario category was
not present in the V1 version.

####  3.2.1 Video Scenario Attribute

As shown in Figure 2, we present the detailed distribution of video, frame,
and frame of $7$ open scenarios and an ”Unknown” scenario on DSText V2.
‘Street View (Outdoor)’ and ‘Sport’ scenarios present most video and text
numbers, respectively. And the frame number of each scenario is almost the
same. It is worth mentioning that the text in the Street (Indoor) scenario
exhibits extremely high text density, with an average of around $140$ texts
per frame, as shown in Figure 2 (e). This poses a significant challenge,
particularly for transformer-based architectures with a limited capacity of
handling 100 queries, such as TransDETR Wu et al. (2022) (which can output
only 100 bounding boxes). Additionally, the text in this scenario is also
extremely small, with an average pixel area size of less than 1000, which is
much smaller than existing datasets. We also present more visualizations for
‘Game’, ‘Driving’, ‘Sports’, and ‘Street View’ in Figure 9.

####  3.2.2 Higher Proportion of Small Text

Figure 3 presents the proportion of different text areas for different
datasets. The proportion of big text (more than $1,000$ pixel area) on our
DSText V2 is less than that of BOVText and ICDAR2015(video) with at least
$20\%$. Moreover, it is also about $5\%$ smaller than DSText V1, which is due
to the inclusion of smaller-sized text in our V2 version. DSText V2 also
presents a higher proportion for small texts (less 400 pixels) with up to
$50\%$. This type of small text poses significant challenges, as demonstrated
in the supermarket (street indoor) scene shown in Figure 9. For text detectors
and recognizers, such text is prone to false negatives, false positives, or
recognition errors. RoadText-1K Reddy et al. (2020) and LSVTD Cheng et al.
(2019) also exhibit relatively small average text areas in Table 1. However,
their text density is quite sparse, with only $0.75$ texts and $5.12$ texts
per frame, which is significantly lower than our text density of $42.4$.
Besides, RoadText-1k only focuses on the driving domain, which limits the
evaluation of other scenarios.

Figure 8: Wordcloud visualizations for DSText V2.

:

Figure 9: More Qualitative Video Text Visualization of DSText V2. DSText V2
covers small and dense texts in various scenarios, which is more challenging.

####  3.2.3 Dense Text Distribution

Figure 4 presents the comparison of text density at each frame. The frames
with more than $15$ text instances occupies $58\%$ in our dataset, at least
$30\%$ improvement than the previous work ($7\%$ and $8\%$ for BOVText and
ICDAR2015 video), which presents more dense text scenarios. Besides, the
proportion of frames with fewer than $5$ text instances is significantly lower
in our dataset, accounting for only $22\%$. In contrast, ICDAR2015 video had
$70\%$ and BOVText had $66\%$ of frames with fewer than $5$ text instances.
Therefore, the proposed DSText V2 shows the challenge of dense text tracking
and recognition. Figure 5 shows the detailed data distribution for our DSText
V2. Frames containing 5-40 text instances comprise the majority, while there
are also over 4,000 frames with densely packed scenes containing more than 150
text instances. More visualization can be found in Figure 9 (Visualization for
various scenarios) and Figure 10 (Representative case with around 200 texts
per frame).

####  3.2.4 WordCloud

We also visualize the word cloud for text content in Figure 8 and Figure 7.
All words from annotation must contain at least $3$ characters, we consider
the words less than four characters usually insignificant, _e.g.,_ ‘is’.
Comparing Figure 7 and Figure 8, we can observe significant changes in the
word cloud distribution due to the addition of the extra 40 videos. Some new
words have emerged, such as ”variety” and ”PACK.”

:

Figure 10: One Case for around 200 Texts per Frames. DSText V2 includes huge
amounts of small and dense text scenarios.

##  4 Tasks and Metric

The proposed dataset supports three tasks: 1) video text detection, where the
objective is to obtain a rough estimation of the text areas in the image, in
terms of bounding boxes that correspond to parts of text. 2) video text
tracking, where the objective is to localize and track all words in the video
sequences. and 3) the end-to-end video text spotting: where the objective is
to localize, track and recognize all words in the video sequence. In this
paper, we will conduct a comprehensive evaluation of these three tasks.

###  4.1 Task 1: Video Text Detection

Video text detection is similar to image-level detection, the method requires
detecting all the text in the video frames and returning the corresponding
detection boxes. And we will directly use the image-level evaluation metrics
from ICDAR 2015 Karatzas et al. (2015) to evaluate.

###  4.2 Task 2: Video Text Tracking

The task requires one network to detect and track text over the video sequence
simultaneously. Given an input video, the network should produce two results:
a rotated detection box, and tracking ids of the same text. For simplicity, we
adopt the evaluation method from the ICDAR 2015 Video Robust Reading
competition Zhou et al. (2015) for the task. The evaluation was revised in
2020, utilizing MOTChallenge Dendorfer et al. (2019) for multiple object
tracking For each method, MOTChallenge provides three different metrics: the
Multiple Object Tracking Precision (MOTP), the Multiple Object Tracking
Accuracy (MOTA), and the IDF1. See the 2013 competition report Karatzas et al.
(2013) and MOTChallenge Dendorfer et al. (2019) for details about these
metrics.

Table 2: The Related Metric for Video Text Spotting Task. These metrics have been adopted by ICDAR2015(V), ICDAR2013(V), RoadText-1k, BOVText, DSText V1, and DSText V2. Metric | Better |  Description
---|---|---
${\rm ID_{F1}}$ | higher |  ${\rm ID_{F1}}$ Score Ristani et al. (2016). The ratio of correctly identified detections&recognitions over the average number of ground-truth and computed detections&recognitions. Formula: $\frac{{\rm ID_{Recall}}\times{\rm ID_{Precision}}\times 2}{{\rm ID_{Recall}}+{\rm ID_{Precision}}}$
${\rm ID_{Recall}}$ | higher |  Ratio of correct detections&recognitions to total number of GTs. Formula: $\frac{{\rm TP}}{{\rm TP}+{\rm FN}}$
${\rm ID_{Precision}}$ | higher |  Ratio of correct detections&recognitions to total number of predicted detections&recognitions. Formula: $\frac{{\rm TP}}{{\rm TP}+{\rm FP}}$
$\rm MOTA$ | higher |  Multi-Object Tracking Accuracy Bernardin and Stiefelhagen (2008). This measure combines three error sources: false positives, missed targets and identity switches. Formula: $1-\frac{{\rm FN}+{\rm FP}+{\rm ID\,\,Sw.}}{{\rm TP}+{\rm FN}}$
$\rm MOTP$ | higher |  Multiple Object Tracking Precision Bernardin and Stiefelhagen (2008) measures the average precision of target position predictions in multiple object tracking tasks.
${\rm Mostly\,\,Tracked}$ | higher |  Mostly tracked targets. The ratio of ground-truth trajectories that are covered by a track hypothesis for at least 80% of their respective life span.
${\rm Partially\,\,Matched}$ | lower |  The percentage of ground-truth trajectories covered by a track hypothesis, within the range of 20% to 80% of their respective lifespan.
${\rm Mostly\,\,Lost}$ | lower |  Mostly lost targets. The ratio of ground-truth trajectories that are covered by a track hypothesis for at most 20% of their respective life span.
${\rm False\,\,Postive(FP)}$ | lower |  The total number of false positives.
${\rm False\,\,Negative(FN)}$ | lower |  The total number of false negative.
${\rm True\,\,Postive(TP)}$ | higher |  The total number of true positives.
${\rm True\,\,Negative(TN)}$ | higher |  The total number of true negatives.
${\rm ID\,\,Sw.}$ | lower |  Number of Identity Switches Li et al. (2009). The ID Switch metric measures the frequency of target ID changes in text tracking, representing the occurrences of target identity transitions within a sequence.

###  4.3 Task 3: End-to-End Video Text Spotting

Video Text Spotting (VTS) task that requires simultaneously detecting,
tracking, and recognizing text in the video. The word recognition performance
is evaluated by simply whether a word recognition result is completely
correct. And the word recognition evaluation is case-insensitive and accent-
insensitive. All non-alphanumeric characters are not taken into account,
including decimal points, such as ‘1.9’ will be transferred to ‘19‘ in our GT.
Similarly, the evaluation method (_i.e.,_ ${\rm ID_{F1}}$, ${\rm MOTA}$ and
${\rm MOTP}$) from the ICDAR 2015 Robust Reading competition is also adopted
for the task. Table 2 shows all metrics and their corresponding explanations.
The main issue with MOTA is its primary focus on the number of incorrect
decisions made by the tracker, such as ID switches (IDSW). In certain
scenarios, there is a greater concern for the duration of tracking for a
specific ID. For instance, when a trajectory is consistently tracked for the
majority of its lifespan, even if it experiences multiple ID switches in the
early or late stages, we consider the trajectory to be a reasonable
prediction. Therefore, ${\rm ID_{F1}}$ is introduced to address this
phenomenon. ${\rm ID_{F1}}$ is particularly concerned with the length of time
a tracker correctly matches the same ID. In cases where a trajectory maintains
correct ID matches for the majority of its tracking duration, occasional
acceptable ID switches may occur. Conversely, even if there is only one ID
switch, but the majority of ID matches are incorrect, this is deemed
unacceptable in practical applications.

##  5 Experimental

###  5.1 Implementation Details for Baseline

Figure 11: Tracking performance (_i.e.,_ ${\rm ID_{F1}}$) with TransDETR Wu et
al. (2022) in different scenarios of DSText V2. The ”Street Indoor” scenario
presents smaller text areas and denser text data, which introduces significant
challenges for existing algorithms and datasets.

TransDETR Wu et al. (2022), as the official baseline
111https://rrc.cvc.uab.es/?ch=22&com=downloads of DSText V1, is used to
further evaluate and validate our dataset. TransDETR is a novel, simple, and
end-to-end video text DEtection, Tracking, and Recognition framework
(TransDETR), which views the video text spotting task as a direct long-
sequence temporal modeling problem. We followed the official open-source code
222https://github.com/weijiawu/TransDETR for various implementation details,
including learning rate, data augmentation, and optimizer. All speed and
performance are tested with a batch size of 1 on a V100 GPU and a 2.20GHz CPU
in a single thread.

Other baselines. In addition to the TransDETR, we also included several other
baselines. As shown in Table 5, image-level detectors, including EAST Zhou et
al. (2017), and PSENet Wang et al. (2019), were used to detect text objects.
We strictly followed the settings and configurations provided in the official
open-source code for all aspects of object detection. Wang et al. (2017) was
utilized to link and match text objects across frames using IOU and edit
distance. Finally, CRNN Shi et al. (2016) was employed as the recognition
model to identify each text trajectory in the video. We used the re-
implementation code of DTRB 333https://github.com/clovaai/deep-text-
recognition-benchmark for our experiments.

Figure 12: Tracking performance (_i.e.,_ ${\rm ID_{F1}}$) of videos with
different average text area. TransDETR is the baseline, and is trained on the
training set of DSText V2.

###  5.2 Attribute Experiments Analysis for DSText V2

Small Text, New Challenge for Video Text. A key contribution of this work is
the distribution of a high proportion of small text. To validate the technical
challenges posed by small text, we analyze the problem from two perspectives.
Firstly, we compare the performance of videos with different text densities.
As shown in Figure 12, we compute the average text area and corresponding
${\rm ID_{F1}}$ metric for 50 test videos. It can be observed that within a
certain range, larger text areas result in better tracking performance.
However, when the average text area is less than 500 pixels, it becomes
challenging to achieve an ${\rm ID_{F1}}$ score above $50\%$. Secondly, we
evaluate the tracking performance based on video scenarios. As shown in Figure
11, we categorize the evaluation based on video scenarios. It is evident that
scenarios with smaller average text areas exhibit lower tracking performance.
For example, the ”Street Indoor” scenario presents the lowest performance,
with an average text area of only 900 pixels. Additionally, we explore the
transferability of models trained on other existing video text benchmarks and
tested on our dataset. Figure 11 presents that the challenge of small text
persists, even when models are trained on other benchmarks.

Figure 13: Tracking performance (_i.e.,_ ${\rm ID_{F1}}$) of videos with
different text density. TransDETR is the baseline, and is trained on the
training set of DSText V2. Figure 14: Ablation Study for the Number of Query.
When the number of queries is less than 200, increasing the query quantity
results in significant performance gains.

Dense Texts, New Challenge for Video Text. Similar to analyzing the challenge
posed by small text, we also examined the distributional changes between text
density and ${\rm ID_{F1}}$ metric. As shown in Figure 13, it is evident that
as the text density increases, the ${\rm ID_{F1}}$ performance gradually
decreases. Interestingly, when the average text density is less than 100,
almost no video performance is below $30\%$. However, among videos with an
average text density greater than 150, three of them exhibit performance below
$30\%$.

Effect of Different Scenarios Figure 11 compares the performance on the test
set across different scenarios. The “Sport” scenario demonstrates the best
performance, while the “Street Indoor” scenario performs the worst. This is
reasonable since the “Street Indoor” scenario contains a large amount of dense
small text, with an average text area of only $900$ pixels. However, the
average text per frame is as high as $140$. It is important to note that the
“Unknown” scenario does not exist in the test set. The “Unknown” scenario is
introduced as a new scenario in the training set of the V2 version, and we did
not expand it to the test set.

Effect of the Number of Query. Table 14 presents the ablation study for the
number of query. When the number of queries is less than 200, increasing the
query quantity results in significant performance gains. However, when the
number of queries exceeds 200, further increases seem to yield diminishing
returns, indicating a saturation in performance.

###  5.3 Video Text Detection

As shown in Table 3, we provide six baselines for evaluation. EAST, PSENet,
and DB are image-level detectors, while TransDETR is an end-to-end video text
spotter. For TransDETR, we directly use the tracking detection boxes as
predictions to calculate the metrics. With 200 queries, TransDETR achieves the
highest performance with an F-measure of $62.9\%$. In comparison to existing
benchmarks, the detection performance is noticeably lower. For example, the
highest performance on ICDAR15 is close to $90\%$, while the current highest
performance on the video version of ICDAR15 is $75\%$. There still seems to be
significant room for improvement. Furthermore, it is worth noting that dense
text poses a significant challenge not only in terms of detection accuracy but
also in terms of detection speed. For algorithms that cannot process text
instances in parallel, such as DB and PSENet, the inference speed is greatly
restricted due to the need to process over 100 text instances in each frame.
As a result, the inference speed is only around 5 frames per second (fps),
which is unacceptable for video tasks.

Table 3: Video Text Detection Performance on DSText V2. ‘Size’ and ‘Q’ means the shorter side of the input and query, respectively. Method | Backbone | Pretrain | Video Text Detection/%
---|---|---|---
Precision | Recall | F-measure | FPS
EAST (size: 800) Wu et al. (2021) | VGG16 | COCO-Text | 72.1 | 37.8 | 49.6 | 8.7
PSENet (size: 800) Wang et al. (2019) | ResNet18 | COCO-Text | 76.1 | 38.5 | 51.1 | 3.6
DB (size: 800) Liao et al. (2020) | ResNet18 | COCO-Text | 76.5 | 40.3 | 52.8 | 6.3
DB++ (size: 800) Liao et al. (2022) | ResNet18 | COCO-Text | 77.8 | 41.6 | 54.2 | 5.5
TransVTSpotter (Q:200) Wu et al. (2021) | ResNet50 | COCO-Text | 77.2 | 45.3 | 57.1 | 8.9
TransDETR (Q:100) Wu et al. (2022) | ResNet50 | - | 74.9 | 42.0 | 53.8 | 11.9
TransDETR (Q:100) Wu et al. (2022) | ResNet50 | COCO-Text | 75.0 | 52.3 | 61.6 | 11.9
TransDETR (Q:200) Wu et al. (2022) | ResNet50 | COCO-Text | 78.5 | 52.5 | 62.9 | 11.4
Table 4: Video Text Tracking Performance on DSText V2. ‘Size’ and ‘Q’ means the shorter side of the input and learnable query embedding, respectively. Mostly Tracked (${\rm M\mbox{-}T}$) denotes the number of tracked objects at least 80% of lifespan, Mostly Lost (${\rm M\mbox{-}L}$) denotes the number of objects tracked less than 20% of lifespan. EAST Zhou et al. (2017), VMFT Wang et al. (2017), TransDETR Wu et al. (2022), and PSENet Wang et al. (2019) are used as the baselines. Method | $\rm Backbone$ | Pre. | Video Text Tracking/% | FPS
---|---|---|---|---
Detection | Tracking | ${\rm ID_{F1}}$ | $\rm MOTA$ | $\rm MOTP$ |  ${\rm M\mbox{-}T}$$\uparrow$ |  ${\rm P\mbox{-}T}$$\uparrow$ |  ${\rm M\mbox{-}L}$$\downarrow$
EAST | VMFT | VGG16 | COCO-Text | 32.1 | 24.8 | 76.1 | 2890 | 2670 | 7017 | 7.1
PSENet | VMFT | ResNet18 | COCO-Text | 33.8 | 24.5 | 77.3 | 3025 | 2509 | 7043 | 2.3
YORO | ResNet50 | COCO-Text | 33.2 | 23.4 | 73.5 | 985 | 1762 | 9893 | 5.4
TransVTSpotter (Q:200) | ResNet50 | COCO-Text | 50.1 | 31.0 | 77.2 | 2378 | 1892 | 8307 | 8.9
TransDETR (Q:100) | ResNet50 | - | 26.8 | 35.2 | 76.7 | 3191 | 2347 | 7039 | 11.9
TransDETR (Q:200) | ResNet50 | - | 47.1 | 30.4 | 77.1 | 2878 | 2091 | 7608 | 11.4
TransDETR (Q:100) | ResNet50 | COCO-Text | 52.7 | 36.0 | 77.8 | 3796 | 2484 | 6297 | 11.9
TransDETR (Q:200) | ResNet50 | COCO-Text | 55.2 | 39.3 | 77.4 | 4368 | 2505 | 5704 | 11.4
Table 5: End-to-End Video Text Spotting Performance on DSText V2. ‘Q’ means the number of query. EAST Zhou et al. (2017), VMFT Wang et al. (2017), TransDETR Wu et al. (2022), CRNN Shi et al. (2016), and PSENet Wang et al. (2019) are used as the baselines. Method | $\rm Backbone$ | Pre. | Video Text Spotting/% | FPS
---|---|---|---|---
Detection | Tracking | Recognition | ${\rm ID_{F1}}$ | $\rm MOTA$ | $\rm MOTP$ |  ${\rm M\mbox{-}T}$$\uparrow$ |  ${\rm P\mbox{-}T}$$\uparrow$ |  ${\rm M\mbox{-}L}$$\downarrow$
EAST | VMFT | CRNN | VGG16 | COCOText | 9.3 | -31.2 | 77.3 | 553 | 527 | 11497 | 6.7
PSENet | VMFT | CRNN | ResNet18 | COCOText | 10.2 | -31.0 | 77.3 | 536 | 587 | 11454 | 2.0
TransDETR (Q:100) | ResNet50 | - | 9.8 | -16.4 | 77.9 | 407 | 404 | 11766 | 11.9
TransDETR (Q:200) | ResNet50 | - | 12.0 | -11.9 | 78.8 | 446 | 313 | 11818 | 11.4
TransDETR (Q:100) | ResNet50 | COCOText | 27.0 | -9.7 | 80.4 | 1458 | 780 | 10339 | 11.9
TransDETR (Q:200) | ResNet50 | COCOText | 25.5 | -1.3 | 81.3 | 1110 | 725 | 10742 | 11.4
Table 6: Cross-dataset Evaluations on DSText V2. TransDETR with 200 queries is used as the baseline. Pre. | Train Set | Test Set | Video Text Tracking/%
---|---|---|---
${\rm ID_{F1}}$ | $\rm MOTA$ | $\rm MOTP$ |  ${\rm M\mbox{-}T}$$\uparrow$ |  ${\rm P\mbox{-}T}$$\uparrow$ |  ${\rm M\mbox{-}L}$$\downarrow$
COCO-Text | BOVText | DSText V2 | 29.6 | 10.9 | 74.5 | 925 | 1801 | 9851
- | COCO-Text | DSText V2 | 43.4 | 27.9 | 79.0 | 1533 | 1210 | 9834
- | ICDAR2015 video | DSText V2 | 44.8 | 29.1 | 73.1 | 1388 | 1329 | 9860
COCO-Text | ICDAR2015 video | DSText V2 | 49.9 | 32.3 | 72.2 | 2406 | 1994 | 8177
COCO-Text | DSText V2 | DSText V2 | 55.2 | 39.3 | 77.4 | 4368 | 2505 | 5704
- | DSText V2 | ICDAR 2015 video | 54.2 | 35.3 | 76.7 | 760 | 219 | 937
COCO-Text | DSText V2 | ICDAR 2015 video | 63.4 | 44.0 | 73.5 | 947 | 390 | 579
Table 7:  Detailed information on the size, density, and tracking performance of text in each video. Dense and small text scenarios are typically more challenging. Video Name | 209 | 184 | 208 | 202 | 210 | 122 | 228 | 199 | 214 | 156
---|---|---|---|---|---|---|---|---|---|---
Text Area (# pixel) | 418.8 | 436.1 | 465.3 | 466.3 | 469.5 | 506.9 | 507.7 | 518.0 | 580.2 | 600.6
Text Number per Frame | 254.1 | 28.5 | 391.1 | 69.6 | 178.1 | 19.6 | 22.4 | 65.2 | 207.1 | 213.2
${\rm ID_{F1}}/\%$ | 21.8 | 42.4 | 31.9 | 40.9 | 47.1 | 58.4 | 43.7 | 58.2 | 37.0 | 70.0
MOTA/% | 21.4 | 24.6 | 20.8 | 23.6 | 33.3 | 48.6 | 22.3 | 21.1 | 29.1 | 51.5
Video Name | 128 | 239 | 162 | 121 | 135 | 175 | 192 | 125 | 143 | 216
Text Area (# pixel) | 644.9 | 666.4 | 686.7 | 688.4 | 730.5 | 735.2 | 808.4 | 812.7 | 847.1 | 872.7
Text Number per Frame | 24.4 | 5.1 | 75.7 | 14.2 | 26.6 | 24.1 | 195.5 | 29.7 | 17.7 | 24.9
${\rm ID_{F1}}/\%$ | 67.7 | 58.6 | 63.6 | 55.0 | 50.0 | 57.1 | 24.0 | 66.7 | 51.7 | 54.8
MOTA/% | 54.0 | 38.9 | 48.3 | 32.4 | 39.4 | 36.2 | 27.1 | 40.7 | 24.3 | -9.6
Video Name | 145 | 149 | 126 | 174 | 112 | 232 | 222 | 211 | 133 | 197
Text Area (# pixel) | 900.9 | 902.5 | 987.9 | 992.0 | 998.8 | 1017.5 | 1018.7 | 1033.4 | 1039.5 | 1068.3
Text Number per Frame | 30.2 | 23.2 | 70.5 | 19.5 | 30.8 | 20.0 | 138.1 | 56.2 | 42.7 | 84.7
${\rm ID_{F1}}/\%$ | 68.9 | 51.7 | 54.5 | 65.3 | 80.6 | 50.5 | 45.0 | 51.5 | 47.6 | 62.8
MOTA/% | 48.6 | 46.8 | 32.4 | 46.9 | 68.4 | 29.8 | 41.5 | 39.4 | 31.1 | 45.2
Video Name | 196 | 171 | 167 | 223 | 237 | 172 | 132 | 231 | 220 | 218
Text Area (# pixel) | 1128.8 | 1195.4 | 1196.8 | 1259.0 | 1312.1 | 1379.9 | 1385.7 | 1424.5 | 1511.8 | 1615.7
Text Number per Frame | 168.3 | 158.1 | 17.8 | 8.7 | 10.9 | 90.4 | 16.8 | 32.2 | 60.4 | 251.8
${\rm ID_{F1}}/\%$ | 32.5 | 27.7 | 62.2 | 66.9 | 66.3 | 49.2 | 59.7 | 37.6 | 59.4 | 42.0
MOTA/% | 42.7 | 27.9 | 36.8 | 42.0 | 51.6 | 33.3 | 29.0 | 17.2 | 46.2 | 50.2
Video Name | 230 | 242 | 136 | 105 | 108 | 144 | 178 | 134 | 114 | 123
Text Area (# pixel) | 1769.3 | 1776.1 | 1783.3 | 2079.6 | 2366.3 | 2435.2 | 2959.7 | 3092.6 | 3310.9 | 6919.1
Text Number per Frame | 6.3 | 11.8 | 24.6 | 25.8 | 24.1 | 14.5 | 79.8 | 12.5 | 18.3 | 22.3
${\rm ID_{F1}}/\%$ | 62.4 | 63.4 | 58.6 | 63.9 | 68.8 | 64.6 | 51.9 | 68.3 | 84.9 | 86.0
MOTA/% | 34.9 | 42.8 | 38.5 | 53.8 | 56.9 | 38.3 | 35.5 | 51.1 | 90.9 | 78.4

###  5.4 Video Text Tracking

Table 4 presents the corresponding tracking performance for DSText V2. With
200 queries, TransDETR achieves the best performance with an IDF1 of $55.2\%$
and a MOTA of $39.3\%$, showing a significant improvement compared to a simple
combination of image text detectors and tracking link algorithms. It’s worth
noting that compared to YORO and TransVTSpotter, our method has a simpler and
more efficient pipeline, eliminating the need for multiple models to assist
each other. Additionally, pretraining on COCOText Veit et al. (2016) leads to
a significant improvement, with at least a $10\%$ increase in ${\rm ID_{F1}}$.
Furthermore, increasing the number of queries results in further improvement.
This is reasonable since, as shown in Figure 5, there are approximately $4000$
frames with over $150$ texts. In such scenarios, having more queries is
necessary to capture all the texts accurately. We also provide more detailed
information for each video Figure 7.

###  5.5 Video Text Spotting

We also provide the performance of end-to-end video text spotting in Table 5.
Compared to the text tracking task, the performance of the end-to-end task
appears to be significantly unsatisfactory. One main reason is that, unlike
tracking metrics, the end-to-end task requires both tracking and recognizing
the text with complete accuracy. The predicted result is considered correct
only when every letter in each word is correctly recognized. However, this is
a challenging task, especially in video scenarios where there are camera
movements, distortions, and occlusions. Overall, with 200 queries and
pretraining on COCOText Veit et al. (2016), the model achieves the best
performance with an ${\rm ID_{F1}}$ of $25.5\%$ and a MOTA of $-1.3\%$. This
indicates that there is still significant room for improvement, such as using
more powerful recognition models, among other approaches. Additionally,
further iterations are needed to enhance the inference speed in dense scenes.
The current inference speed is insufficient for practical applications,
especially for video streaming applications.

###  5.6 Cross-datasets Evaluations

Table 6 presents the related cross-datasets evaluations for studying the
generalizability and transferability between different domains. The ICDAR2015
video dataset, being the most popular dataset currently, is primarily used as
a benchmark for comparison. It is observed that a model trained on the
ICDAR2015 video dataset fails to achieve satisfactory performance on DSText
V2, yielding only a 49.6 ${\rm ID_{F1}}$. However, a model trained on DSText
V2 achieves a notable 63.4 ${\rm ID_{F1}}$ score on the ICDAR2015 video
dataset. This suggests that the model trained on DSText V2 possesses stronger
generalization capabilities, performing well on other domain data. The
dissatisfactory performance of the model trained on ICDAR2015 video in DSText
V2 indicates the presence of unique challenges in the video scenarios of
DSText V2 that are absent in the ICDAR2015 video dataset. This is reasonable,
as DSText V2 presents numerous challenges related to small and densely packed
text. In contrast, the majority of text in the ICDAR 2015 video dataset
corresponds to typical scenes, as indicated by the data comparison in Table 1.
Therefore, a model trained on ICDAR 2015 video may struggle to handle very
small and dense text scenarios.

##  6 Potential Limitations

Although our dataset addresses important gaps and challenges, there are still
some potential limitations. Here, we briefly enumerate and discuss several
limitations. Firstly, the dataset mainly focuses on multi-oriented English
text and does not support curved text annotation. While multi-oriented English
text is the most common occurrence in real-life scenarios, the inability to
support curved text is a limitation. This is particularly relevant in
scenarios where curved text, such as text on billboards, cannot be adequately
fit by multi-oriented bounding boxes. Furthermore, the scale of this dataset
is still not very large. Exploring ways to further expand the dataset size at
a lower cost remains one of the directions worth considering.

##  7 Conclusion and Future Work

Here, we present a new video text reading benchmark, which focuses on dense
and small video text. We provide a well-maintained project page
444https://rrc.cvc.uab.es/?ch=22&com=introduction with corresponding links for
dataset download. Compare with the previous datasets, the proposed dataset
mainly includes two new challenges for dense and small video text spotting.
High-proportioned small texts are a new challenge for the existing video text
methods. Furthermore, we provide a comprehensive data analysis that includes
the number of videos in different scenarios, the average text area, and the
average text density. Additionally, we provide more experimental analysis
corresponding to the two unique challenges, namely small text and dense text,
further validating their impact. Overall, we believe and hope the benchmark,
as one standard benchmark, develops and improve the video text tasks in the
community.

##  8 Acknowledgements

This work is supported by the National Key Research and Development Program of
China (# 2022YFC3602601), the Key Research and Development Program of Zhejiang
Province of China (# 2021C02037), and the National Natural Science Foundation
(NSFC#62225603) Besides, we extend our appreciation to OpenAI for the valuable
contribution of ChatGPT, which played a crucial role in refining the grammar
and enhancing the writing of this paper.

## References

  * Jaderberg et al. (2014) M. Jaderberg, K. Simonyan, A. Vedaldi, A. Zisserman,  Synthetic data and artificial neural networks for natural scene text recognition,  in: Workshop on Deep Learning, NIPS, 2014.
  * Gupta et al. (2016) A. Gupta, A. Vedaldi, A. Zisserman,  Synthetic data for text localisation in natural images,  in: IEEE Conference on Computer Vision and Pattern Recognition, 2016, pp. 2315–2324.
  * Karatzas et al. (2015) D. Karatzas, L. Gomez-Bigorda, A. Nicolaou, S. Ghosh, A. Bagdanov, M. Iwamura, J. Matas, L. Neumann, V. R. Chandrasekhar, S. Lu, et al.,  Icdar 2015 competition on robust reading,  in: IEEE International Conference on Document Analysis and Recognition, 2015, pp. 1156–1160.
  * Zhao et al. (2023) Y. Zhao, W. Wu, Z. Li, J. Li, W. Wang,  Flowtext: Synthesizing realistic scene text video with optical flow estimation,  arXiv preprint arXiv:2305.03327 (2023).
  * Ch’ng and Chan (2017) C. K. Ch’ng, C. S. Chan,  Total-text: A comprehensive dataset for scene text detection and recognition,  in: IEEE International Conference on Document Analysis and Recognition, 2017, pp. 935–942.
  * Wang et al. (2019) W. Wang, E. Xie, X. Li, W. Hou, T. Lu, G. Yu, S. Shao,  Shape robust text detection with progressive scale expansion network,  in: IEEE conference on computer vision and pattern recognition, 2019, pp. 9336–9345.
  * Zhou et al. (2017) X. Zhou, C. Yao, H. Wen, Y. Wang, S. Zhou, W. He, J. Liang,  East: an efficient and accurate scene text detector,  in: IEEE conference on computer vision and pattern recognition, 2017, pp. 5551–5560.
  * Lyu et al. (2018) P. Lyu, M. Liao, C. Yao, W. Wu, X. Bai,  Mask textspotter: An end-to-end trainable neural network for spotting text with arbitrary shapes,  in: European Conference on Computer Vision, 2018, pp. 67–83.
  * Wu et al. (2022) W. Wu, E. Xie, R. Zhang, W. Wang, P. Luo, H. Zhou,  Polygon-free: Unconstrained scene text detection with box annotations,  in: 2022 IEEE International Conference on Image Processing (ICIP), IEEE, 2022, pp. 1226–1230.
  * Cai et al. (2022) Y. Cai, Y. Liu, C. Shen, L. Jin, Y. Li, D. Ergu,  Arbitrarily shaped scene text detection with dynamic convolution,  Pattern Recognition 127 (2022) 108608.
  * Zhu and Du (2021) Y. Zhu, J. Du,  Textmountain: Accurate scene text detection via instance segmentation,  Pattern Recognition 110 (2021) 107336.
  * Wu et al. (2019) W. Wu, J. Xing, H. Zhou,  Textcohesion: Detecting text for arbitrary shapes,  arXiv preprint arXiv:1904.12640 (2019).
  * Liu et al. (2018) X. Liu, D. Liang, S. Yan, D. Chen, Y. Qiao, J. Yan,  Fots: Fast oriented text spotting with a unified network,  in: Proceedings of the IEEE conference on computer vision and pattern recognition, 2018, pp. 5676–5685.
  * Yin et al. (2016) X.-C. Yin, Z.-Y. Zuo, S. Tian, C.-L. Liu,  Text detection, tracking and recognition in video: a comprehensive survey,  IEEE Transactions on Image Processing 25 (2016) 2752–2773.
  * Srivastava et al. (2015) N. Srivastava, E. Mansimov, R. Salakhudinov,  Unsupervised learning of video representations using lstms,  in: International Conference on Machine Learning, 2015, pp. 843–852.
  * Dong et al. (2021) J. Dong, X. Li, C. Xu, X. Yang, G. Yang, X. Wang, M. Wang,  Dual encoding for video retrieval by text,  IEEE Transactions on Pattern Analysis and Machine Intelligence (2021).
  * Wu et al. (2023) W. Wu, Y. Zhao, Z. Li, J. Li, H. Zhou, M. Z. Shou, X. Bai,  A large cross-modal video retrieval dataset with reading comprehension,  arXiv preprint arXiv:2305.03347 (2023).
  * Anagnostopoulos et al. (2008) C.-N. E. Anagnostopoulos, I. E. Anagnostopoulos, I. D. Psoroulas, V. Loumos, E. Kayafas,  License plate recognition from still images and video sequences: A survey,  IEEE Transactions on intelligent transportation systems 9 (2008) 377–391.
  * Nguyen et al. (2014) P. X. Nguyen, K. Wang, S. Belongie,  Video text detection and recognition: Dataset and benchmark,  in: IEEE winter conference on applications of computer vision, 2014, pp. 776–783.
  * Reddy et al. (2020) S. Reddy, M. Mathew, L. Gomez, M. Rusinol, D. Karatzas, C. Jawahar,  Roadtext-1k: Text detection & recognition dataset for driving videos,  in: IEEE International Conference on Robotics and Automation, 2020, pp. 11074–11080.
  * Cheng et al. (2019) Z. Cheng, J. Lu, Y. Niu, S. Pu, F. Wu, S. Zhou,  You only recognize once: Towards fast video text spotting,  in: ACM International Conference on Multimedia, 2019, pp. 855–863.
  * Wu et al. (2021) W. Wu, D. Zhang, Y. Cai, S. Wang, J. Li, Z. Li, Y. Tang, H. Zhou,  A bilingual, openworld video text dataset and end-to-end video text spotter with transformer,  in: Thirty-fifth Conference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2), 2021.
  * Li et al. (2017) H. Li, P. Wang, C. Shen,  Towards end-to-end text spotting with convolutional recurrent neural networks,  in: IEEE International Conference on Computer Vision, 2017, pp. 5238–5246.
  * He et al. (2018) T. He, Z. Tian, W. Huang, C. Shen, Y. Qiao, C. Sun,  An end-to-end textspotter with explicit alignment and attention,  in: IEEE conference on computer vision and pattern recognition, 2018, pp. 5020–5029.
  * Liu et al. (2020) Y. Liu, H. Chen, C. Shen, T. He, L. Jin, L. Wang,  Abcnet: Real-time scene text spotting with adaptive bezier-curve network,  in: CVPR, 2020, pp. 9809–9818.
  * Liu et al. (2023) C. Liu, C. Yang, H.-B. Qin, X. Zhu, C.-L. Liu, X.-C. Yin,  Towards open-set text recognition via label-to-prototype learning,  Pattern Recognition 134 (2023) 109109.
  * Lu et al. (2021) N. Lu, W. Yu, X. Qi, Y. Chen, P. Gong, R. Xiao, X. Bai,  Master: Multi-aspect non-local network for scene text recognition,  Pattern Recognition 117 (2021) 107980.
  * Zheng et al. (2020) Y. Zheng, Y. Xie, Y. Qu, X. Yang, C. Li, Y. Zhang,  Scale robust deep oriented-text detection network,  Pattern Recognition 102 (2020) 107180.
  * Wu et al. (2020a) W. Wu, N. Lu, E. Xie, Y. Wang, W. Yu, C. Yang, H. Zhou,  Synthetic-to-real unsupervised domain adaptation for scene text detection in the wild,  in: Proceedings of the Asian Conference on Computer Vision, 2020a.
  * Wu et al. (2020b) W. Wu, J. Xing, C. Yang, Y. Wang, H. Zhou,  Texts as lines: text detection with weak supervision,  Mathematical Problems in Engineering 2020 (2020b) 1–12.
  * Wu et al. (2023) W. Wu, Y. Zhao, M. Z. Shou, H. Zhou, C. Shen,  Diffumask: Synthesizing images with pixel-level annotations for semantic segmentation using diffusion models,  arXiv preprint arXiv:2303.11681 (2023).
  * Ren et al. (2015) S. Ren, K. He, R. Girshick, J. Sun,  Faster r-cnn: Towards real-time object detection with region proposal networks,  Advances in neural information processing systems 28 (2015) 91–99.
  * Zhou et al. (2015) X. Zhou, S. Zhou, C. Yao, Z. Cao, Q. Yin,  Icdar 2015 text reading in the wild competition,  arXiv preprint arXiv:1506.03184 (2015).
  * Veit et al. (2016) A. Veit, T. Matera, L. Neumann, J. Matas, S. Belongie,  Coco-text: Dataset and benchmark for text detection and recognition in natural images,  arXiv preprint arXiv:1601.07140 (2016).
  * Karatzas et al. (2013) D. Karatzas, F. Shafait, S. Uchida, M. Iwamura, L. G. i Bigorda, S. R. Mestre, J. Mas, D. F. Mota, J. A. Almazan, L. P. De Las Heras,  Icdar 2013 robust reading competition,  in: 2013 12th international conference on document analysis and recognition, IEEE, 2013, pp. 1484–1493.
  * Yu et al. (2021) H. Yu, Y. Huang, L. Pi, C. Zhang, X. Li, L. Wang,  End-to-end video text detection with online tracking,  Pattern Recognition 113 (2021) 107791.
  * Tu et al. (2018) D. Tu, L. Chen, M. Lv, H. Shi, G. Chen,  Hierarchical online nmf for detecting and tracking topic hierarchies in a text stream,  Pattern Recognition 76 (2018) 203–214.
  * Rong et al. (2014) X. Rong, C. Yi, X. Yang, Y. Tian,  Scene text recognition in multiple frames based on text tracking,  in: 2014 IEEE International Conference on Multimedia and Expo (ICME), IEEE, 2014, pp. 1–6.
  * Wang et al. (2017) X. Wang, Y. Jiang, S. Yang, X. Zhu, W. Li, P. Fu, H. Wang, Z. Luo,  End-to-end scene text recognition in videos based on multi frame tracking,  in: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 1, IEEE, 2017, pp. 1255–1260.
  * Cheng et al. (2020) Z. Cheng, J. Lu, B. Zou, L. Qiao, Y. Xu, S. Pu, Y. Niu, F. Wu, S. Zhou,  Free: A fast and robust end-to-end video text spotter,  IEEE Transactions on Image Processing 30 (2020) 822–837.
  * Wu et al. (2022) W. Wu, Y. Cai, C. Shen, D. Zhang, Y. Fu, H. Zhou, P. Luo,  End-to-end video text spotting with transformer,  arXiv preprint arXiv:2203.10539 (2022).
  * Zayene et al. (2016) O. Zayene, M. Seuret, S. M. Touj, J. Hennebert, R. Ingold, N. E. B. Amara,  Text detection in arabic news video based on SWT operator and convolutional auto-encoders,  in: Workshop on Document Analysis Systems, 2016, pp. 13–18.
  * Wu et al. (2023) W. Wu, Y. Zhao, Z. Li, J. Li, M. Z. Shou, U. Pal, D. Karatzas, X. Bai,  Icdar 2023 video text reading competition for dense and small text,  arXiv preprint arXiv:2304.04376 (2023).
  * Minetto et al. (2011) R. Minetto, N. Thome, M. Cord, N. J. Leite, J. Stolfi,  Snoopertrack: Text detection and tracking for outdoor videos,  in: IEEE International Conference on Image Processing, 2011, pp. 505–508.
  * Yu et al. (2018) F. Yu, W. Xian, Y. Chen, F. Liu, M. Liao, V. Madhavan, T. Darrell,  Bdd100k: A diverse driving video database with scalable annotation tooling,  arXiv preprint arXiv:1805.04687 (2018).
  * Dendorfer et al. (2019) P. Dendorfer, H. Rezatofighi, A. Milan, J. Shi, D. Cremers, I. Reid, S. Roth, K. Schindler, L. Leal-Taixe,  Cvpr19 tracking and detection challenge: How crowded can it get?,  arXiv preprint arXiv:1906.04567 (2019).
  * Ristani et al. (2016) E. Ristani, F. Solera, R. Zou, R. Cucchiara, C. Tomasi,  Performance measures and a data set for multi-target, multi-camera tracking,  in: European conference on computer vision, Springer, 2016, pp. 17–35.
  * Bernardin and Stiefelhagen (2008) K. Bernardin, R. Stiefelhagen,  Evaluating multiple object tracking performance: the clear mot metrics,  EURASIP Journal on Image and Video Processing 2008 (2008) 1–10.
  * Li et al. (2009) Y. Li, C. Huang, R. Nevatia,  Learning to associate: Hybridboosted multi-target tracker for crowded scene,  in: 2009 IEEE conference on computer vision and pattern recognition, IEEE, 2009, pp. 2953–2960.
  * Shi et al. (2016) B. Shi, X. Bai, C. Yao,  An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition,  IEEE transactions on pattern analysis and machine intelligence 39 (2016) 2298–2304.
  * Liao et al. (2020) M. Liao, Z. Wan, C. Yao, K. Chen, X. Bai,  Real-time scene text detection with differentiable binarization,  in: Proceedings of the AAAI conference on artificial intelligence, volume 34, 2020, pp. 11474–11481.
  * Liao et al. (2022) M. Liao, Z. Zou, Z. Wan, C. Yao, X. Bai,  Real-time scene text detection with differentiable binarization and adaptive scale fusion,  IEEE Transactions on Pattern Analysis and Machine Intelligence 45 (2022) 919–931.


# Formant Estimation and Tracking using Probabilistic Heat-Maps

###### Abstract

Formants are the spectral maxima that result from acoustic resonances of the
human vocal tract, and their accurate estimation is among the most fundamental
speech processing problems. Recent work has been shown that those frequencies
can accurately be estimated using deep learning techniques. However, when
presented with a speech from a different domain than that in which they have
been trained on, these methods exhibit a decline in performance, limiting
their usage as generic tools.

The contribution of this paper is to propose a new network architecture that
performs well on a variety of different speaker and speech domains. Our
proposed model is composed of a shared encoder that gets as input a
spectrogram and outputs a domain-invariant representation. Then, multiple
decoders further process this representation, each responsible for predicting
a different formant while considering the lower formant predictions. An
advantage of our model is that it is based on heatmaps that generate a
probability distribution over formant predictions. Results suggest that our
proposed model better represents the signal over various domains and leads to
better formant frequency tracking and estimation.

Index Terms: formant estimation, formant tracking, heat maps, deep neural
networks.

##  1 Introduction

Formants are resonances of the vocal tract. Typically there are 3 to 5 formant
frequencies, roughly corresponding to each band of 1 kHz. As well as being
essential for the perception of speech, they play an important role in speech
coding, synthesis, and enhancement. A reliable estimation of these frequencies
is essential in many phonological experiments in the fields of phonology,
sociolinguistics, and bilingualism [1, 2, 3, 4, 5].

Recently, it was proposed to use deep learning methods for the tasks of
formant estimation and tracking [6, 7, 8]. The network was trained and tested
on the Vocal Tract Resonance (VTR) database [9] and achieved state-of-the-art
results. However, as with traditional systems, experienced a dropoff in
performance when the network was presented at test time with other recorded
speech databases [10, 3] due to the phenomenon of over-fitting to the speaker
and speech domains represented in that database.

Inspired by the tremendous advancement in object detection and pose estimation
in the vision domain, we propose a similar yet modified modeling for the
formant estimation based on probabilistic heatmaps. In the work of [11, 12,
13] heatmaps and probabilistic approaches were presented for estimating body
parts and detecting objects rather than a direct coordinates prediction. We
borrow ideas from those works to increase generalization in the task of
formants estimation.

Our goal is to train a single network that performs well on a variety of
different speaker and speech domains. A straightforward approach is to use
combined corpora consisting of multiple datasets instead of the VTR alone at
the training phase. However, when incorporating these databases into a single
corpus, the results degraded for all domains of speech, including the VTR
dataset. These datasets were collected for different research purposes, and
there is significant variability among the datasets due to gender, age,
dialects, recording conditions, etc. A visualization of this phenomenon for
the first two formants ($F_{1}$ and $F_{2}$) is depicted in the couple left
sub-figures in Fig. 2. As one can see, there is a noticeable shift in the
formants values between the gender and age groups. There is no direct
transformation for the same group across datasets; the formants values among
men recordings in VTR differ from those in Hillenbrand for the same vowel.
Hence, direct optimization in the form of a regression task may not replicate
the same performance if tested on samples from another dataset.

This paper suggests a new modeling technique for formant tracking
incorporating both the heatmap and the probabilistic approach. First, we use a
convolutional neural network (CNN) model to generate a latent informative and
explainable input representation without direct supervision. Then, multiple
decoders are to predict formants as in a classification setup rather than
traditional regression, which also improves generalization to new domains.
Finally, an empirical evaluation of the proposed approach is presented on both
in-domain examples as well as new datasets, and the generalization capability
is discussed.

Figure 1: Our model. The spectrogram is encoded into a domain-invariant
representation and decoded into probabilistic heatmaps while preserving the
original input dimensions. A heatmap aggregation is shown on the left, where
the max value at each frame is taken as a prediction.

##  2 Problem settings

The input to our model is a spectrogram mathematically represented as a
sequence of $D$-dimensional real values,
$\bar{{\mathbf{x}}}=({\mathbf{x}}_{1},\ldots,{\mathbf{x}}_{T})$, where
${\mathbf{x}}_{t}\in\mathcal{X}\subseteq\mathbb{R}^{D}$ for $t\in[1,T]$, where
$T$ is the number of frames in the input, and $D$ is the number of bins in the
spectrogram. Associated with the spectrogram is an annotated sequence of $K$
formants $\bar{\bm{F}}=(\bm{F}^{1},\ldots,\bm{F}^{T})$, where
$\bm{F}^{t}=(F^{t}_{1},\ldots,F^{t}_{K})\in\mathbb{R}^{K}$ is a vector of $K$
annotated formants of the $t$-th frame (we used $K=3$). Our goal is to predict
the sequence $\bar{\bm{F}}$.

The proposed model is built from two components: an _encoder_ and multiple
_decoders_. The encoder generates a latent representation of the same size as
the input spectrogram, namely $D\times T$. The encoder is denoted as a
function $g:\mathcal{X}^{T}\to\mathcal{Z}^{T}$ from the domain of speech
spectrogram to the domain of latent representations,
$\mathcal{Z}\subseteq\mathbb{R}^{D}$.

The latent representation serves as input to a set of $K$ decoders that work
jointly, where each decoder is responsible for estimating the conditional
probability $\Pr(F_{k}|F_{k-1},\bm{z})$ of the $k$-th formant given the
prediction of the lower formant and the latent frame representation $\bm{z}$.
At inference, the probability $\Pr(F_{1}|\bm{z})$ is estimated first, then the
second formant probability is estimated given the first formant,
$\Pr(F_{2}|F_{1},\bm{z})$, and last, the third formant probability is
estimated given the first two, $\Pr(F_{3}|F_{2},F_{1},\bm{z})$. Overall the
formants at a given frame are predicted as the value
$\bm{F}^{*}=(F^{*}_{1},F^{*}_{2},F^{*}_{3})$ that maximizes the conditional
probability:

$\Pr(F_{1},F_{2},F_{3}|\bm{z})\\\
=\Pr(F_{1}|\bm{z})\Pr(F_{2}|F_{1},\bm{z})\Pr(F_{3}|F_{2},F_{1},\bm{z})\vspace{-5pt}$
(1)

In our setting, the formant values are real numbers but quantized to $D$ bins.
Then, the softmax function estimates the conditional probability of the
quantized formant at each frame, and the median frequency at the associated
bin is taken is predicted. The decoders are a set of functions $h_{k}$, $1\leq
k\leq K$, where the decoder for $F_{i}$ receives as input the latent
representation $\bm{z}$, where the formants $F_{j}$ for $j<i$ have been
masked. We can consider the probability distribution over the formant values
as _heatmaps_. The heatmap is a graphical representation of the conditional
probability space, showing the magnitude of a phenomenon based on the color of
every area. A spectrogram is a heatmap visualization where the frequency is
being squashed into bins, and the value of each bin indicates the strength of
the signal in a specific frequency range.

Figure 2: The average values for $F_{1}$ and $F_{2}$ for each vowel.
Hillenbrand annotations (left), VTR annotations(middle), VTR predictions by
our model (right). Groups are Men (m), Women (w) and Children (c).

##  3 Proposed model

In this section, we go over the model architecture and the optimization
process in detail. At first, we describe the suggested model, components, and
motivation.

###  3.1 Background

A fundamental property of “working in the wild” is the ability to leverage
information from a given data to new and unseen recordings. When splitting a
given set of examples into train and test i.i.d, we rely on the assumption
that both sets belong to an unknown yet particular distribution.
Unfortunately, this assumption does not always hold while relating future
samples as a new test set, i.e., new recordings may come from various acoustic
environments or different speakers. Needless to say, such distribution shifts
are critical and may frequently lead to severe performance degradation. In our
case, the main issue limiting current methods is the fixation of a trained
model to the frequency range where each formant was mainly seen during
training. For example, as depicted in the middle subfigure in Fig. 2, the
frequency of $F_{1}$ in a particular dataset speaker ranges between $350$ to
$750$Hz; thus, a new sample with $F_{1}$ at $1,000$Hz can be presumed as out-
of-distribution. Such phenomena could be due to the speaker’s age or gender.
In that case, the model behavior is undefined. In this paper, we address this
issue and provide a robust formant tracking system consisting of two
components - the first is invariant to the frequency range while the latter
is.

Tracking formants can be broken down into two steps: (i) creating a shared
representation, where information regarding all trackable formants is
extracted using the encoder block; and (ii) predicting a probabilistic heatmap
per formant ($F_{1},F_{2},$ and $F_{3}$) using the set of decoders
$\\{h_{k}\\}_{k=1}^{K}$. By maintaining the spectrogram dimensions, we have a
one-to-one mapping between a formant activation in the output heatmap and the
corresponding frequency bin in the spectrogram. We begin by describing the
shared representation encoder and then describe the set of decoders.

###  3.2 Shared encoder

As distribution shifts in unseen data are often presented as shifts of the
spectra (e.g., an unseen speaker with a higher than usual pitch), our goal is
to learn a representation that is robust to such changes. One way of
incorporating such inductive bias into our encoder is to use 2D-convolutional
layers on the spectrogram, which are translation-invariant to changes in the
spectra. In other words, a formant should be detected even when it manifests
in an unusual frequency band.

As in the encoder-decoder framework, supervision for the encoder output is not
required. By being invariant to shifts in the spectra and receiving
supervision in the form of gradients from the individual decoders, the encoder
learns the general structure of formants in the spectrogram space, regardless
of their prototypical frequency band. Upon closer inspection of the encoder
output in Fig. 1, we can see that it successfully captures additional formants
in the given spectrogram (beyond $F_{1},F_{2},F_{3}$), even for untracked
formants originating in higher frequency bands.

The encoder consists of $3\times 3$ kernels with residual connections and
layered as follows: $1-16-32-64-128-128-64-32-1$, where the input channel
dimension is one and the output as well. The height and width preserve the
original dimension. Batch normalization and dropout layers are combined
between the layers.

###  3.3 Multiple decoders

Given the heatmap produced by the shared encoder, the task of the multiple
decoders is to estimate the formants frequencies accurately by generating
three formant-specific probabilistic heatmaps. At this stage, each decoder has
to capture only the relevant information related to the associated formant
while ignoring the rest of the intermediate heatmap. A masking function is
required to prevent multiple heads from tracking the same formant. Thus, the
decoding works sequentially by setting the input for each head as the shared
representation where the output of previous heads is masked. As opposed to the
encoder, the decoding phase consists of 1D-convolution. The convolved filter
spans over the entire frequency axis, allowing it to detect in various
optional bands. The output of each head is a probabilistic heatmap targeting a
specific formant.

The decoder consists of three identical heads; each has a three-layer
bottleneck with residual connection starting with $257-64-257$ where each
kernel has a window of 3 frames, and the output shape of $257$ reconstructs
the original spectrogram dimensions. Interestingly, in preliminary
experiments, we observed that the bias term in these convolutional layers
tends to converge to the prototypical location of the learned formant, leaving
the kernel weights to be near zero (i.e., ignoring the input). To prevent this
phenomenon, we remove the bias term in these layers. Batch normalization and
dropout function are combined between the layers.

##  4 Evaluation

Table 1: Formant tracking difference reported in mean absolute error on broad
phone classes in VTR.

| Inter-labler | Praat | WaveSurfer | MSR | DeepFormant | Ours
---|---|---|---|---|---|---
| $F_{1}$ | $F_{2}$ | $F_{3}$ | $F_{1}$ | $F_{2}$ | $F_{3}$ | $F_{1}$ | $F_{2}$ | $F_{3}$ | $F_{1}$ | $F_{2}$ | $F_{3}$ | $F_{1}$ | $F_{2}$ | $F_{3}$ | $F_{1}$ | $F_{2}$ | $F_{3}$
vowels | $55$ | $69$ | $84$ | $130$ | $230$ | $267$ | $70$ | $94$ | $154$ | $64$ | $105$ | $125$ | $\bf{54}$ | $81$ | $112$ | $57$ | $\bf{75}$ | $\bf{95}$
semivowels | $68$ | $80$ | $103$ | $136$ | $295$ | $334$ | $89$ | $126$ | $222$ | $83$ | $122$ | $154$ | $\bf{67}$ | $114$ | $168$ | $\bf{67}$ | $\bf{90}$ | $\bf{130}$
nasal | $75$ | $112$ | $106$ | $219$ | $409$ | $381$ | $96$ | $229$ | $239$ | $67$ | $\bf{120}$ | $\bf{112}$ | $\bf{66}$ | $175$ | $151$ | $74$ | $130$ | $125$
fricatives | $91$ | $113$ | $125$ | $564$ | $593$ | $700$ | $209$ | $263$ | $439$ | $129$ | $\bf{108}$ | $\bf{131}$ | $131$ | $135$ | $159$ | $\bf{112}$ | $120$ | $140$
affricatives | $89$ | $118$ | $135$ | $730$ | $515$ | $583$ | $292$ | $407$ | $390$ | $141$ | $\bf{129}$ | $149$ | $164$ | $162$ | $189$ | $\bf{140}$ | $138$ | $\bf{145}$
stops | $91$ | $110$ | $116$ | $258$ | $270$ | $351$ | $168$ | $210$ | $286$ | $130$ | $\bf{113}$ | $\bf{119}$ | $131$ | $135$ | $168$ | $\bf{117}$ | $124$ | $130$

###  4.1 Datasets

In order to estimate a model’s cross-datasets generalization performance, we
used three corpora of manually annotated formants, each of which consists of
different speakers and a recording setup. The first dataset is the Vocal Tract
Resonance (VTR) [9], a subset of the TIMIT dataset that has been split into
train and test set containing 346 and 192 utterances, respectively. There are
173 speakers in the train set and 24 in the test set, presenting a diverse
selection of speakers, dialects, gender, and phoneme. The first three
formants($F_{1}$-$F_{3}$) were initially estimated using the MSR algorithm
[14] and then were corrected manually. Worth mentioning that the fourth
formant is also annotated; however, it has not been manually corrected and is
based only on an automatic extraction. Hence we do not evaluate it in this
paper. The second dataset collected by Clopper and Tamati [10] consists 777
utterances produced by 20 female speakers in the age of 18-22. Each utterance
contains a single word with a single vowel, and the $F_{1}$ and $F_{2}$ were
annotated at the midpoint of the vowel. The third dataset collected by
Hillenbrand et al. [3] during a laboratory study consists of 1668 recordings
produced by 45 men, 48 women, and 46 children (27 boys and 19 girls). The
participants read a list of 12 single vowel words, and each recording was
later split into 12 short utterances, one for each word. The first three
formants ($F_{1}$-$F_{3}$) were annotated at 4 points across the vowel- at
20%,50%, 80% of the vowel duration and in additional time where the annotators
have called a _steady-point_. For pre-processing we apply a pre-emphasis
filter, $H(z)=1-0.97z^{-1}$ , and the number of points for the FFT is $512$.
Spectrograms were created with Torchaudio [15]. The utterances sampling rate
is $16$ kHz; thus, each cell in both the spectrogram and heatmaps spans over
approximately $31$ Hz.

Table 2: Formant estimation error in Hz for multiple datasets and training
techniques.

Dataset | Method | $F_{1}$ | $F_{2}$ | $F_{3}$
---|---|---|---|---
VTR | WaveSurfer | $70$ | $96$ | $154$
DeepFormants | $50$ | $86$ | $104$
Ours | $\bf{39}$ | $\bf{30}$ | $\bf{47}$
Hillenbrand | WaveSurfer | $68$ | $190$ | $182$
DeepFormants (Train VTR) | $71$ | $160$ | $131$
DeepFormants (Train All) | $36$ | $100$ | $116$
Ours (Train VTR) | $74$ | $150$ | $125$
Ours (Train All) | $\bf{26}$ | $\bf{78}$ | $\bf{82}$
Clopper | WaveSurfer | $128$ | $181$ | $-$
DeepFormants (Train VTR) | $228$ | $168$ | $-$
DeepFormants (Train All) | $103$ | $157$ | $-$
Ours (Train VTR) | $99$ | $147$ | $-$
Ours (Train All) | $\bf{49}$ | $\bf{64}$ | $-$

###  4.2 Augmentation & hyperparameters

Unfortunately, having numerous low-volume formants annotated datasets is a key
challenge for training a deep learning model; hence, an artificial enlargement
is required. Data augmentations and manipulation are often used for increasing
and enriching the training set. While traditional audio-related augmentation
like input noising, random-cropping, and spectrogram reversal may boost a
model performance, the modified instances are yet considered as in-
distribution. We applied an additional augmentation to simulate out-of-
distribution samples synthetically, allowing the model to predict formants in
a never seen range. Furthermore, we used a speed-up augmentation by a factor
of two, directly counteracting the issue of a model focusing on popular
frequency bands in the dataset. As a result, we maintained the ability to
yield reliable annotations - the values were merely doubled, and the multi-
head decoder was prevented from fixing the formants’ values. Setting a non-
integer speed-up factor resulted in inaccurate new labels: the originally
annotated frame could have been undefined in the new representation, i.e.,
between adjacent time frames. Hence the augmentation must be tractable. In the
following experiments, we optimized parameters using Adam [16] with an initial
learning rate of $1e-4$, annealed by a factor of 10 at epochs 300 and 600. In
addition, we used label smoothing [17] and randomly speeded up 20% of the
samples every epoch during training.

###  4.3 Formant estimation

We begin by presenting our results for the estimation task and proceed with
the tracking. The former demonstrates the generalization capabilities to a
different domain, i.e., out-of-distribution. We compare the suggested method
with both analytically and machine-learning-based methods. Namely, (i) Praat
[18] and (ii) WaveSurfer [19], widely used tools for phonetic research and
speech analysis based on Linear Predictive Coding (LPC). We also report the
results for the (iii) MSR algorithm [14] and DeepFormants [7, 6]. We followed
the baseline setups as described in [7].

For the estimation task, which applies to vowels only, we averaged the
frequencies for the entire vowel segment and reported the difference from the
annotated test sample in Table 2. Samples in the Clopper dataset are annotated
only for $F_{1}$ and $F_{2}$; hence we leave blanks for the estimation
difference in $F_{3}$. The results present two training methods: (i) training
with VTR alone and estimating formants for Hillenbrand and Clopper, (ii)
training with all three datasets and predicting for set-aside samples from
each. Since there is no predefined test set for Clopper and Hillenbrand, a
third from each speakers group was chosen at random for the test set, leaving
the rest for training.

As seen in the results, replicating the performance obtained from the trained
samples to a new domain is a critical obstacle for machine-learning-based
methods [20, 21]. There is a performance gap for DeepFormants and our model
when we combine samples from the train set of Clopper and Hillenbrand during
the optimization compared to train only with the VTR. The difference in
$F_{1}$ estimation dropped by around 50% for each model when we used the
entire corpora during training. Our heatmap approach presented better
performance than Wavesurfer and DeepFormants in both cases. Moreover, where
only the VTR was used for training, the suggested method outperformed the
previous system even where the latter was trained with samples from those
datasets.

To demonstrate the learned underlying structure, we computed the average
annotated and predicted vowel frequencies in the VTR for each speakers group
(men, women, children) and plotted them with the surrounding polygon in Fig.
2.

###  4.4 Formant tracking

This section addresses the VTR alone, being the only annotated frame-wised
corpus. We consider tracking formants in VTR as an in-distribution task since
utterances from the train and the test set share the same recording setup.

We compared our predictions with the annotations for each spoken frame and
reported the mean absolute error in Hz for each phonetic class in Table 1. The
authors of the VTR dataset [9] reported the variation among the annotators for
each formant and phonetic class as an anchor for evaluation. When tested on
the vowel and semivowel segments, the probabilistic approach presented here
outperformed the previous approaches with a negligible imprecision in $F_{1}$
among vowels. Specifically, the difference for the most indicative formant,
i.e., $F_{1}$, is on par with the expected variation across human annotators
or the most accurate among previous methods, i.e., 57 and 67 Hz difference in
$F_{1}$ in vowels and semivowels compared to 55 and 68 Hz with human
annotator, respectively. Results suggest a minor difference in $F_{2}$ vs.
$F_{3}$ in the vowels phone class. However, it shows the most accurate
measurements in comparison to previous automated systems. The higher precision
in the consonant class presented by MSR is highly likely due to the VTR
annotation process; the initial formants were extracted using the MSR
algorithm and manually corrected afterward. In general, the higher the
formant, the more difficult it is to track it, possibly due to its broad
frequency bandwidth movement. An interesting phenomenon in our measurements is
that the difference in $F_{2}$ is higher than $F_{3}$ among nasals - possibly
a result of inaccurate annotating since most models reproduce the same
behavior as the inter-labeler variance suggests.

A second assessment of the tracking performance involves analyzing the
response time for a model to track a formant when switching from vowel to
consonant (VC) and vice versa. At these transition points, the formant
frequency shifts rapidly and has to be tracked again at the appropriate
bandwidth. In line with previous study [9, 7, 6], we consider three frames
before and after the transition as a six-frames window and calculate the mean
absolute difference as reported in Table 3.

Table 3: Formant tracking error in Hz for a six framed window with a transition from consonant to wel and vice versa. | MSR | DeepFormant | Ours
---|---|---|---
| $F_{1}$ | $F_{2}$ | $F_{3}$ | $F_{1}$ | $F_{2}$ | $F_{3}$ | $F_{1}$ | $F_{2}$ | $F_{3}$
CV transitions | $106$ | $101$ | $119$ | $110$ | $142$ | $166$ | $\bf{91}$ | $\bf{85}$ | $\bf{97}$
VC transitions | $\bf{48}$ | $92$ | $120$ | $53$ | $80$ | $111$ | $59$ | $\bf{62}$ | $\bf{77}$

##  5 Conclusions

In this paper, we presented a probabilistic approach for the task of format
tracking and estimation based on heatmaps classification rather than
regression as in previous models. Focusing on generalization, we introduced
novel modeling that can generate predictions with superior accuracy for both
in-domain and out-of-distribution samples. In future work, we would like to
further explore the generalization effect by combining additional augmentation
techniques to simulate out-of-distribution samples. In addition, we believe
the generated shared heatmap representation captures the acoustic structure of
the speaker and can be used in a variety of speech-related tasks. The code is
publicly available at https://github.com/MLSpeech/FormantsTracker.

## References

  * [1] H. K. Kathania, S. R. Kadiri, P. Alku, and M. Kurimo, “Study of formant modification for children asr,” in _ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_. IEEE, 2020, pp. 7429–7433.
  * [2] R. D. Kent and H. K. Vorperian, “Static measurements of vowel formant frequencies and bandwidths: A review,” _Journal of communication disorders_ , vol. 74, pp. 74–97, 2018.
  * [3] J. Hillenbrand, L. A. Getty, M. J. Clark, and K. Wheeler, “Acoustic characteristics of american english vowels,” _The Journal of the Acoustical society of America_ , vol. 97, no. 5, pp. 3099–3111, 1995.
  * [4] S. Lee, A. Potamianos, and S. Narayanan, “Acoustics of children’s speech: Developmental changes of temporal and spectral parameters,” _The Journal of the Acoustical Society of America_ , vol. 105, no. 3, pp. 1455–1468, 1999.
  * [5] H. K. Vorperian and R. D. Kent, “Vowel acoustic space development in children: a synthesis of acoustic and anatomic data.” _Journal of speech, language, and hearing research : JSLHR_ , vol. 50 6, pp. 1510–45, 2007.
  * [6] Y. Dissen and J. Keshet, “Formant estimation and tracking using deep learning.” in _Interspeech_ , 2016, pp. 958–962.
  * [7] Y. Dissen, J. Goldberger, and J. Keshet, “Formant estimation and tracking: A deep learning approach,” _The Journal of the Acoustical Society of America_ , vol. 145, no. 2, pp. 642–653, 2019.
  * [8] W. Dai, J. Zhang, Y. Gao, W. Wei, D. Ke, B. Lin, and Y. Xie, “Formant tracking using dilated convolutional networks through dense connection with gating mechanism,” _arXiv preprint arXiv:2005.10803_ , 2020.
  * [9] L. Deng, X. Cui, R. Pruvenok, J. Huang, S. Momen, Y. Chen, and A. Alwan, “A database of vocal tract resonance trajectories for research in speech processing,” in _2006 IEEE International Conference on Acoustics Speech and Signal Processing Proceedings_ , vol. 1. IEEE, 2006, pp. I–I.
  * [10] C. G. Clopper and T. N. Tamati, “Effects of local lexical competition and regional dialect on vowel production,” _The Journal of the Acoustical Society of America_ , vol. 136, no. 1, pp. 1–4, 2014.
  * [11] J. Tompson, R. Goroshin, A. Jain, Y. LeCun, and C. Bregler, “Efficient object localization using convolutional networks,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 648–656.
  * [12] S.-E. Wei, V. Ramakrishna, T. Kanade, and Y. Sheikh, “Convolutional pose machines,” in _Proceedings of the IEEE conference on Computer Vision and Pattern Recognition_ , 2016, pp. 4724–4732.
  * [13] J. Redmon, S. Divvala, R. Girshick, and A. Farhadi, “You only look once: Unified, real-time object detection,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2016, pp. 779–788.
  * [14] L. Deng, L. J. Lee, H. Attias, and A. Acero, “A structured speech model with continuous hidden dynamics and prediction-residual training for tracking vocal tract resonances,” in _2004 IEEE International Conference on Acoustics, Speech, and Signal Processing_ , vol. 1. IEEE, 2004, pp. I–557.
  * [15] Y.-Y. Yang, M. Hira, Z. Ni, A. Chourdia, A. Astafurov, C. Chen, C.-F. Yeh, C. Puhrsch, D. Pollack, D. Genzel, D. Greenberg, E. Z. Yang, J. Lian, J. Mahadeokar, J. Hwang, J. Chen, P. Goldsborough, P. Roy, S. Narenthiran, S. Watanabe, S. Chintala, V. Quenneville-Bélair, and Y. Shi, “Torchaudio: Building blocks for audio and speech processing,” _arXiv preprint arXiv:2110.15018_ , 2021.
  * [16] D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” _arXiv preprint arXiv:1412.6980_ , 2014.
  * [17] R. Müller, S. Kornblith, and G. E. Hinton, “When does label smoothing help?” _Advances in neural information processing systems_ , vol. 32, 2019\.
  * [18] P. Boersma, “Praat: doing phonetics by computer [computer program],” _http://www. praat. org/_ , 2011.
  * [19] K. Sjölander and J. Beskow, “Wavesurfer-an open source speech tool,” in _Sixth International Conference on Spoken Language Processing_. Citeseer, 2000.
  * [20] Y. Shrem, M. Goldrick, and J. Keshet, “Dr. vot: Measuring positive and negative voice onset time in the wild,” _arXiv preprint arXiv:1910.13255_ , 2019.
  * [21] M. Goldrick, Y. Shrem, O. Kilbourn-Ceron, C. Baus, and J. Keshet, “Using automated acoustic analysis to explore the link between planning and articulation in second language speech production,” _Language, cognition and neuroscience_ , vol. 36, no. 7, pp. 824–839, 2021.