# Turning a CLIP Model into a Scene Text Detector

Wenwen Yu ,1, Yuliang Liu∗,1, Wei Hua1, Deqiang Jiang2, Bo Ren2, Xiang Bai†,1
1Huazhong University of Science and Technology 2Tencent YouTu Lab
<EMAIL_ADDRESS><EMAIL_ADDRESS>Equal contribution. †Corresponding author.

###### Abstract

The recent large-scale Contrastive Language-Image Pretraining (CLIP) model has
shown great potential in various downstream tasks via leveraging the
pretrained vision and language knowledge. Scene text, which contains rich
textual and visual information, has an inherent connection with a model like
CLIP. Recently, pretraining approaches based on vision language models have
made effective progresses in the field of text detection. In contrast to these
works, this paper proposes a new method, termed TCM, focusing on Turning the
CLIP Model directly for text detection without pretraining process. We
demonstrate the advantages of the proposed TCM as follows: (1) The underlying
principle of our framework can be applied to improve existing scene text
detector. (2) It facilitates the few-shot training capability of existing
methods, _e.g._ , by using 10% of labeled data, we significantly improve the
performance of the baseline method with an average of 22% in terms of the
F-measure on 4 benchmarks. (3) By turning the CLIP model into existing scene
text detection methods, we further achieve promising domain adaptation
ability. The code will be publicly released at
https://github.com/wenwenyu/TCM.

##  1 Introduction

Scene text detection is a long-standing research topic aiming to localize the
bounding box or polygon of each text instance from natural images, as it has
wide practical applications scenarios, such as office automation, instant
translation, automatic driving, and online education. With the rapid
development of fully-supervised deep learning technologies, scene text
detection has achieved remarkable progresses. Although supervised approaches
have made remarkable progress in the field of text detection, they require
extensive and elaborate annotations, _e.g._ , character-level, word-level, and
text-line level bounding boxes, especially polygonal boxes for arbitrarily-
shaped scene text. Therefore, it is very important to investigate text
detection methods under small amount of labeled data, _i.e._ , few-shot
training.

Figure 1: Comparisons of different paradigms of using text knowledge for scene
text detection.

Recently, through leveraging the pretrained vision and language knowledge, the
large-scale Contrastive Language-Image Pretraining (CLIP) model [27] has
demonstrated its significance in various downstream tasks. _e.g._ , image
classification [56], object detection [5], and semantic segmentation [28, 45,
13].

Compared to general object detection, scene text in natural images usually
presents with both visual and rich character information, which has a natural
connection with the CLIP model. Therefore, how to make full use of cross-modal
information from visual, semantic, and text knowledge to improve the
performance of the text detection models receives increasing attentions in
recent studies. For examples, Song et al. [31], inspired by CLIP, adopts fine-
grained cross-modality interaction to align unimodal embeddings for learning
better representations of backbone via carefully designed pretraining tasks.
Xue et al. [48] presents a weakly supervised pretraining method to jointly
learn and align visual and partial textual information for learning effective
visual text representations for scene text detection. Wan et al. [37] proposes
self-attention based text knowledge mining to enhance backbone via an image-
level text recognition pretraining tasks.

Different from these works, as shown in Figure 1, this paper focuses on
turning the CLIP model for text detection without pretraining process.
However, it is not trivial to incorporate the CLIP model into a scene text
detector. The key is seeking a proper method to exploit the visual and
semantic prior information conditioned on each image. In this paper, we
develop a new method for scene text detection, termed as TCM, short for
Turning a CLIP Model into a scene text detector, which can be easily plugged
to improve the scene text detection frameworks. We design a cross-modal
interaction mechanism through visual prompt learning, which is implemented by
cross-attention to recover the locality feature from the image encoder of CLIP
to capture fine-grained information to respond to the coarse text region for
the subsequent matching between text instance and language. Besides, to steer
the pretrained knowledge from the text encoder conditioned independently on
different input images, we employ the predefined language prompt, learnable
prompt, and a language prompt generator using simple linear layer to get
global image information. In addition, we design an instance-language matching
method to align the image embedding and text embedding, which encourages the
image encoder to explicitly refine text regions from cross-modal visual-
language priors. Compared to previous pretraining approaches, our method can
be directly finetuned for the text detection task without pretraining process,
as elaborated in Fig. 1. In this way, the text detector can absorb the rich
visual or semantic information of text from CLIP. We summarize the advantages
of our method as follows:

  * •

We construct a new text detection framework, termed as TCM, which can be
easily plugged to enhance the existing detectors.

  * •

Our framework can enable effective few-shot training capability. Such
advantage is more obvious when using less training samples compared to the
baseline detectors. Specifically, by using 10% of labeled data, we improve the
performance of the baseline detector by an average of 22% in terms of the
F-measure on 4 benchmarks.

  * •

TCM introduces promising domain adaptation ability, _i.e._ , when using
training data that is out-of-distribution of the testing data, the performance
can be significantly improved. Such phenomenon is further demonstrated by a
NightTime-ArT text dataset111NightTime-ArT Download Link, which we collected
from the ArT dataset.

  * •

Without pretraining process using specific pretext tasks, TCM can still
leverage the prior knowledge from the CLIP model, outperforming previous scene
text pretraining methods [37, 31, 48].

##  2 Related works

#### Unimodal Scene Text Detection.

Unimodal scene text detection represents the method directly adopts the
bounding boxes annotation only [21]. It can be roughly divided into two
categories: Segmentation-based methods and regression-based methods. The
segmentation-based methods usually conduct pixel-level [18, 35, 47, 14, 39,
43, 17], segment-level [29, 23, 54, 1, 46, 34, 32, 51], or contour-level [38,
41] segmentation, then grouping segments into text instances via post-
processing. The regression-based methods [60, 55, 9, 10, 15, 58, 8, 53, 40]
regards text as a whole object and regress the bounding boxes of the text
instances directly.

#### Cross-modal Assisted Scene Text Detection.

Unlike unimodal based scene text detection, cross-modal assisted scene text
detection aims to make full use of cross-modal information including visual,
semantic, and text knowledge to boost the performance. Wan et al. [37]
utilized an image-level text recognition pretraining tasks to enhance backbone
via the proposed self-attention based text knowledge mining mechanism. Song et
al. [31], inspired by CLIP, designed three pretraining fine-grained cross-
modality interaction tasks to align unimodal embeddings for learning better
representations of backbone. Xue et al. [48] jointly learned and aligned
visual and partial text instances information for learning effective visual
text representations via the proposed weakly supervised pretraining method.
Long et al. [22] proposed an end-to-end model to perform unified scene text
detection and visual layout analysis simultaneously. The above methods
explicitly leverage text or visual information to assist text detection.
Instead, our method focuses on improving the performance results by turning a
CLIP model into a scene text detector via leveraging pretrained text
knowledge.

##  3 Methodology

We begin by illustrating the CLIP model which we used for fetching the prior
knowledge. Next, we introduce the technical details of TCM as well as the
rationale behind it. An overview of our approach is shown in Fig. 2.

###  3.1 Contrastive Language-Image Pretraining

CLIP [27], which collects 400 million image-text pairs without human
annotation for model pretraining, has well demonstrated the potential of
learning transferable knowledge and open-set visual concepts. Previous study
[4] shows that different neurons in CLIP model can capture the corresponding
concept literally, symbolically, and conceptually, As shown in Fig. 4, the
CLIP model is an inborn text-friendly model which can effectively abstract the
mapping space between image and text [26]. During training, CLIP learns a
joint embedding space for the two modalities via a contrastive loss. Given a
batch of image-text pairs, for each image, CLIP maximizes the cosine
similarity with the matched text while minimizing that with all other
unmatched text. For each text, the loss is computed similarly as each image.
In this way, CLIP can be used for zero-shot image recognition [56]. However,
to exploit the relevant information from such a model, there are two
prerequisites: 1) A proper method to effectively request the prior knowledge
from the CLIP. 2) The original model can only measure the similarity between
an integrated image and a single word or sentence. For scene text detection,
there are usually many text instances per image, which are all required to be
recalled equivalently.

###  3.2 Turning a CLIP into a Text Detector

To turn the CLIP model into the scene text detector, we propose TCM, as shown
in Fig. 2 and Fig.3. TCM is a pluggable module that can be directly applied to
enhance the existing scene text detectors. It extracts the image and text
embeddings from the image encoder and text encoder of CLIP model,
respectively. We then design a cross-modal interaction mechanism through
visual prompt learning to recover the locality feature from the image encoder
of CLIP, which can capture fine-grained information to respond to the coarse
text region for the subsequent matching between text instance and language.
For better steering the pretrained knowledge, we introduce a language prompt
generator to generate conditional cue for each image and design a visual
prompt generator that learns image prompts for adapting the frozen clip text
encoder for the text detection task. The TCM can be directly applicable to
broader text detection methods only with some minor modifications. In
addition, we design an instance-language matching method to align the image
embedding and text embedding, which encourages the image encoder to explicitly
refine text regions from cross-modal visual-language priors.

Figure 2: The overall framework of our approach. Figure 3: The details of the
TCM. The image encoder and text encoder are directly from the CLIP model. Det.
Head short for detection head.

#### Image Encoder.

We use the pretrained ResNet50 [7] of CLIP as the image encoder, which
produces an embedding vector for every input pixel. Given the input image
$\bm{I}^{\prime}\in\mathbb{R}^{H\times W\times 3}$, image encoder outputs
image embedding $\bm{I}\in\mathbb{R}^{\tilde{H}\times\tilde{W}\times C}$,
where $\tilde{H}=\frac{H}{s}$, $\tilde{W}=\frac{W}{s}$, and $C$ is the image
embedding dimension ($C$ is set to 1024) and $s$ is the downsampling ratio (s
is empirically set to 32), which can be expressed as:

$\bm{I}=\operatorname{ImageEncoder}(\bm{I}^{\prime})\,.$ (1)

#### Text Encoder.

The text encoder takes input a number of of $K$ classes prompt and embeds it
into a continuous vector space $\mathbb{R}^{C}$, producing text embeddings
$\bm{T}=\\{\bm{t}_{1},\ldots,\bm{t}_{K}\\}\in\mathbb{R}^{K\times C}$ as
outputs of the text encoder, where $\bm{t}_{i}\in\mathbb{R}^{C}$.
Specifically, we leverage the frozen pretrained text encoder of CLIP
throughout as the text encoder can provide language knowledge prior for text
detection. $K$ is set to 1 because there is only one text class in text
detection task. Different from the original model that uses templates like “a
photo of a [CLS].”, we predefine discrete language prompt as “ _Text_ ”. Then,
a part of the text encoder input $\bm{t}_{in}^{\prime}$ is defined as follows:

$\bm{t}_{in}^{\prime}=\operatorname{WordEmbedding}(\rm
Text)\in\mathbb{R}^{D}\,,$ (2)

where $\operatorname{WordEmbedding}(\cdot)$ denotes word embedding for
predefined prompt “Text” class. $D$ is the word embedding dimension and set to
512.

Inspired by CoOp [57, 56], we also add learnable prompt
$\\{\bm{c}_{1},\ldots,\bm{c}_{n}\\}$ to learn robust transferability of text
embedding for facilitating zero-shot transfer of CLIP model, where $n$ is the
number of learnable prompt, which is set to 4 by default, and
$\bm{c}_{i}\in\mathcal{R}^{D}$. Thus, the input $\bm{t}_{in}$ of the text
encoder is as follows:

$\bm{t}_{in}=[\bm{c}_{1},\ldots,\bm{c}_{n},\bm{t}_{in}^{\prime}]\in\mathbb{R}^{(n+1)\times
D}\,.$ (3)

The text encoder takes $\bm{t}_{in}$ as input and generates text embedding
$\bm{T}=\\{\bm{t}_{1}\\}\in\mathbb{R}^{C}$, and $\bm{T}$ is donated by
$\bm{t}_{out}\in\mathcal{R}^{C}$ for simplification:

$\bm{t}_{out}=\operatorname{TextEncoder}(\bm{t}_{in})\in\mathbb{R}^{C}\,.$ (4)

Figure 4: The neurons in the clip model can directly respond to the text. The
source images are from [4].

#### Language Prompt Generator.

Although the predefined prompt and learnable prompt are effective for steering
the CLIP model, it may suffer from limited few-shot or generalization ability
to open-ended scenarios where the testing text instance is out-of-distribution
from the training images. To this end, we present a language prompt generator
to generate a feature vector, termed as conditional cue ($\bm{cc}$). For each
image, the $\bm{cc}$ is then combined with the input of the text encoder
$\bm{t}_{in}$, formulated as follows:

$\hat{\bm{t}}_{in}=\bm{cc}+\bm{t}_{in}\in\mathbb{R}^{(n+1)\times D}\,,$ (5)

where $\hat{\bm{t}}_{in}$ is the new prompt input of the text encoder
conditioned on the input image, and we replace $\bm{t}_{in}$ with
$\hat{\bm{t}}_{in}$ in Eq. 4.

In practice, the language prompt generator is built with a two-layer feed-
forward network, which is applied to generate conditional cue ($\bm{cc}$) from
the globality image embedding $\bm{I}$. This consists of two layer
normalization followed by linear transformations, with a ReLU activation in
between, which is formulated as follows:

$\bm{cc}=\operatorname{LN}(\sigma(\operatorname{LN}(\bm{\bar{I}})\bm{W}_{1}+\bm{b}_{1}))\bm{W}_{2}+\bm{b}_{2}\in\mathbb{R}^{D}\,,$
(6)

where $\bm{\bar{I}}\in\mathbb{R^{C}}$ is the global image-level feature
generated from image embedding $\bm{I}$ by the same global attention pooling
layer as in CLIP. $\bm{W}_{1}\in\mathbb{R}^{C\times C}$,
$\bm{W}_{2}\in\mathbb{R}^{C\times D}$, $\bm{b}_{1}\in\mathbb{R}^{C}$,
$\bm{b}_{2}\in\mathbb{R}^{D}$, and we broadcast $\bm{cc}$ with $\bm{t}_{in}$
to get $\hat{\bm{t}}_{in}$ in Eq. 5.

#### Visual Prompt Generator.

We design a visual prompt generator to adaptively propagate fine-grained
semantic information from textual features to visual features. Formally, we
use the cross-attention mechanism in Transformer [36] to model the
interactions between image embedding ($\bm{Q}$) and text embedding ($\bm{K}$,
$\bm{V}$). The visual prompt $\tilde{\bm{I}}$ is then learned for transferring
the information prior from image-level to text instance-level, which is
defined as:

$\tilde{\bm{I}}=\operatorname{TDec}(Q=\bm{I},K=\bm{t}_{out},V=\bm{t}_{out})\in\mathbb{R}^{\tilde{H}\times\tilde{W}\times
C}\,,$ (7)

where TDec denotes the Transformer Decoder.

Based on the conditional visual prompt, the original image embedding $\bm{I}$
is equipped with $\tilde{\bm{I}}$ to produce the prompted text-aware locality
embeddings $\hat{\bm{I}}$ used for instance-language matching (Eq. 9) and
downstream detection head:

$\hat{\bm{I}}=\bm{I}+\tilde{\bm{I}}.$ (8)

#### Instance-language Matching.

Given the output of the text encoder and image encoder, we perform text
instance-language matching alignment on text-aware locality image embedding
$\hat{\bm{I}}$ and text embedding $\bm{t}_{out}$ by the dot product followed
by sigmoid activation to get binary score map. The mixture of the generated
conditional fine-grained embedding $\tilde{\bm{I}}$ and visual embedding
$\bm{I}$ can allow text instance existing in visual features to be better
matched with pretrained language knowledge in collaboration. The matching
mechanism is formulated as follows:

$\bm{P}=\operatorname{sigmoid}(\hat{\bm{I}}\bm{t}_{out}^{T}/\tau)\in\mathbb{R}^{\tilde{H}\times\tilde{W}\times
1},$ (9)

where $\bm{t}_{out}$ is text embedding because of only one text class in text
detection scenarios, and $\bm{P}$ is the binary text segmentation map. The
segmentation maps are supervised using the ground-truths as an auxiliary loss
and concatenated by the prompted embedding $\hat{\bm{I}}$ for downstream text
detection head to explicitly incorporate language priors for detection. During
training, we minimize a binary cross-entropy loss between the segmentation map
$\bm{P}$ and ground-truth, which is defined as follows:

$\mathcal{L}_{aux}={\sum_{i}^{\tilde{H}}}\sum_{j}^{\tilde{W}}y_{ij}\log(P_{ij})+(1-y_{ij})\log(1-P_{ij})\,,$
(10)

where $y_{ij}$ and $P_{ij}$ are the label and predicted probability of pixel
$(i,j)$ belonging to the text instances, respectively.

#### Optimization.

The loss function $\mathcal{L}_{total}$ is the sum of detection loss
$\mathcal{L}_{det}$ and auxiliary loss $\mathcal{L}_{aux}$, formulated as
follows:

$\mathcal{L}_{total}=\mathcal{L}_{det}+\lambda\mathcal{L}_{aux}\,,$ (11)

where $\lambda$ is a trade-off hyper-parameters and set to 1 in this paper.
$\mathcal{L}_{det}$ depends on downstream text detection method including
segmentation and regression categories. In the inference period, we use the
output of the detection head as the final result.

##  4 Experiments

We conduct four sets of experiments to validate TCM. Our first set of
experiment examines how TCM can be incorporated into existing text detectors
to achieve consistent performance improvements. Next, we demonstrate the few-
shot training capability and generalization ability by incorporating the TCM
method. In the third set of experiments, we compare our method with previous
pretraining methods. Finally, we provide thorough experiments to evaluate the
sensitivity w.r.t. the proposed designs.

#### Datasets.

Our experiments are conducted on a number of commonly known scene text
detection benchmarks including ICDAR2013 (IC13) [12], ICDAR2015 (IC15) [11],
MSRA-TD500 (TD) [50], CTW1500 (CTW) [20], Total-Text (TT) [3], ArT [2], MLT17
[25], and MLT19 [24]. More details of the datasets refer to appendix.

#### Evaluation Metric.

We use intersection over union (IoU) to determine whether the model correctly
detects the region of text, and we calculate precision (P), recall (R), and
F-measure (F) for comparison following common practice [12]. For fair
comparisons, text regions labeled with either “do not care” or “###” will be
ignored in all datasets during training and testing.

#### Implementation Details.

For text detection tasks, we experiment with the popular text detection
methods including DBNet (DB) [18]222https://github.com/MhLiao/DB, PAN
[39]333https://github.com/whai362/pan_pp.pytorch, and FCENet (FCE)
[60]444https://github.com/open-mmlab/mmocr/tree/main/configs/textdet/fcenet to
evaluate TCM. For consistent settings with these methods, we train the
detector using both SynthText and the real datasets. Specifically, the
backbone is instantiated with the pretrained image encoder ResNet50 [7] of the
CLIP unless specified. The visual prompt generator has 3 transformer decoder
layers with 4 heads; transformer width is 256; and the feed-forward hidden
dimension is set to 1024. We use the corresponding detection head of the
DBNet, PAN, and FCENet to predict the final results. For testing few-shot
learning of model, we directly train on the benchmark with different
proportions of training data without pretraining and test it on the
corresponding test data. For testing the generalization ability, we use the
model trained on the corresponding source datasets and evaluating it on the
target dataset that has dissimilar distribution. We consider two kinds of
adaptation including synthtext-to-real and real-to-real, to validate the
domain adaptation of the TCM. The ablation studies are conducted w.r.t. the
predefined prompt, the learnable prompt, the language prompt generator, the
visual prompt generator, and the different settings. The DBNet is used as
baseline for TCM.

###  4.1 Cooperation with Existing Methods

We report the text detection results of our TCM combined with three text
detection methods on IC15, TD, and CTW in Table 1. Our method is +0.9%, +1.7%,
and +1.9% higher than the original FCENet, PAN, and DBNet, respectively, in
terms of F-measure on IC15. TD and CTW also have similar consistent
improvement. Note that the inference speed of our method is 18, 8.4, and 10
FPS evaluated on IC15, TD, and CTW datasets, respectively, with PAN, FCENet,
and DBNet, remaining the high efficiency of the detector.

| Method | IC15 | TD | CTW | FPS
---|---|---|---|---|---
| F | $\Delta$ | F | $\Delta$ | F | $\Delta$
Reg. | FCENet [60] | 86.2 | - | 85.4† | - | 85.5 | - | 11.5
TCM-FCENet | 87.1 | +0.9 | 86.9 | +1.5 | 85.9 | +0.4 | 8.4
Seg. | PAN [39] | 82.9 | - | 84.1 | - | 83.7 | - | 36
TCM-PAN | 84.6 | +1.7 | 85.3 | +1.2 | 84.3 | +0.6 | 18
DBNet [18] | 87.3 | - | 84.9 | - | 83.4 | - | 14.5
TCM-DBNet | 89.2 | +1.9 | 88.8 | +3.9 | 84.9 | +1.5 | 10
Table 1:  Text detection results of cooperating with existing methods on IC15,
TD, and CTW. † indicates the results from [52]. Reg. and Seg. short for
regression and segmentation methods, respectively. FPS are reported with
ResNet50 backbone on a single V100.

We visualize our method in Fig. 7. It shows that the fine-grained features
$\tilde{\bm{I}}$ containing text information is recovered from the global
image embedding $\bm{I}$, demonstrating that TCM can identify text regions and
provide this prior cues for downstream text detection.

###  4.2 Few-shot Training Ability

To further verify the few-show training ability of our method, we directly
train our model on real datasets using various training data ratio without
pretraining, and evaluate it on the corresponding 4 benchmarks. As shown in
Fig. 5, our method shows robust on limited data and outperforms the three
baseline methods including DB, PAN and EAST [58]. The results show that the
TCM can capture the inherent characteristic of text via leveraging the
pretrained vision and language knowledge of the zero-shot trained CLIP model.

(a) TD500 (b) ICDAR15

(c) TotalText (d) CTW1500

Figure 5: Few-shot training ability with varying training data ratio. “F”
represents F-measure.

###  4.3 Generalization Ability

We conduct two types of experiments including synthtext-to-real adaptation and
real-to-real adaptation, as shown in Table 2 and Table 3, respectively. From
the tables, we can see that by plugging the TCM to DBNet, we significantly
improve the performance by an average of 8.2% in terms of F-measure for four
different settings including synthtext-to-real and real-to-real, which further
demonstrates the effectiveness of our method for domain adaptation.

Method | ST $\rightarrow$ IC13 | ST $\rightarrow$ IC15
---|---|---
EAST† [58] | 67.1 | 60.5
PAN [39] | - | 54.8
CCN [44] | - | 65.1
ST3D [16] | 73.8 | 67.6
DBNet [18] | 71.7 | 64.0
TCM-DBNet | 79.6 | 76.7
Table 2: Synthtext-to-real adaptation. † indicates the results from [42]. ST
indicates SynthText. F-measure (%) is reported.

Method | IC13 $\rightarrow$ IC15 | IC13 $\rightarrow$ TD
---|---|---
EAST† [58] | 53.3 | 46.8
GD(AD) [52] | 64.4 | 58.5
GD(10-AD)[52] | 69.4 | 62.1
CycleGAN [59] | 57.2 | -
ST-GAN [19] | 57.6 | -
CycleGAN+ST-GAN | 60.8 | -
TST [42] | 52.4 | -
DBNet [18] | 63.9 | 53.8
TCM-DBNet | 71.9 | 65.1

Table 3: Real-to-real adaptation. † indicates that the results are from [52].
Note that the proposed method outperforms other methods. F-measure (%) is
reported.

###  4.4 Comparison with Pretraining Methods

The pretraining methods based on specifically designed pretext tasks has made
effective progress in the field of text detection. In contrast to these
efforts, TCM can turn the CLIP model directly into a scene text detector
without pretraining process. The comparison results are shown in Table 4, from
which we can see that without pretext tasks for pretraining, DB+TCM
consistently outperforms previous methods including DB+STKM [37], DB+VLPT
[31], and DB+oCLIP [48]. Especially on IC15, our method outperforms previous
state-of-the-art pretraining method by a large margin, with 89.4% versus 86.5%
in terms of the F-measure.

| Methods | Pretext task | IC15 | TT | TD | CTW
---|---|---|---|---|---|---
Convention | SegLink [29] | $\times$ | - | - | 77.0 | -
PSENet-1s [14] | $\times$ | 85.7 | 80.9 | - | 82.2
LOMO [53] | $\times$ | 87.2 | 81.6 | - | 78.4
MOST [8] | $\times$ | 88.2 | - | 86.4 | -
Tang et al.[33] | $\times$ | 89.1 | - | 88.1 | -
VLP | DB+ST† | $\times$ | 85.4 | 84.7 | 84.9 | -
DB+STKM† [37] | ✓ | 86.1 | 85.5 | 85.9 | -
DB+VLPT† [31] | ✓ | 86.5 | 86.3 | 88.5 | -
DB+oCLIP* [48] | ✓ | - | - | - | 84.4
| DB+TCM(Ours) | $\times$ | 89.4 | 85.9 | 88.8 | 85.1
Table 4: Comparison with existing scene text pretraining techniques on DBNet
(DB). † indicates the results from [31]. ST and VLP denote SynthText
pretraining and visual-language pretraining methods, respectively. * stand for
our reimplementation results. F-measure (%) is reported.

###  4.5 Ablation Studies

Pretrained CLIP Backbone. First, we conduct experiments that we only replace
the original backbone of the DBNet with the pretrained image encoder ResNet50
of the CLIP to quantify the performance variance of the backbones. As shown in
Table 5, the original pretrained model of CLIP is insufficient for leveraging
the visual-language knowledge of the CLIP. Therefore, it is necessary to use a
proper method to excavate the knowledge of the CLIP model.

Method |  BB | IC15 | TD | TT | CTW
---|---|---|---|---|---
DBNet | R50 | 87.3 | 84.9 | 84.7 | 83.4
DBNet | CR50 | 87.7 (+0.4) | 86.8 (+1.9) | 84.7 | 83.4

Table 5: Ablation study of the ResNet50 backbone on IC15, TD, TT, and CTW. BB indicates Backbone. R50 and CR50 represent the ResNet50 backbones of the DBNet and the CLIP, respectively. F-measure (%) is reported.  Method | PP | LP |  LG |  VG | IC15 | TD | TT | CTW
---|---|---|---|---|---|---|---|---
F | F | F | F
BSL | $\times$ | $\times$ | $\times$ | $\times$ | 87.7 | 86.8 | 84.7 | 83.4
BSL+ | ✓ | $\times$ | $\times$ | $\times$ | 87.75 | 87.0 | 84.74 | 83.5
BSL+ | ✓ | 4 | $\times$ | $\times$ | 88.0 | 87.1 | 84.8 | 83.6
BSL+ | $\times$ | 4 | $\times$ | $\times$ | 87.8 | 87.7 | 85.1 | 83.9
BSL+ | $\times$ | 18 | $\times$ | $\times$ | 88.1 | 87.8 | 85.3 | 83.9
BSL+ | $\times$ | 32 | $\times$ | $\times$ | 88.4 | 88.2 | 85.4 | 84.5
BSL+ | ✓ | 4 | ✓ | $\times$ | 88.6 | 88.4 | 85.5 | 84.6
TCM | ✓ | 4 | ✓ | ✓ | 89.2 | 88.8 | 85.6 | 84.9
TCM | ✓ | 32 | ✓ | ✓ | 89.4 | 88.8 | 85.9 | 85.1
$\Delta$ |  |  |  |  | +1.7 | +2.0 | +1.2 | +1.7
Table 6: Ablation study of our proposed components on IC15, TD, TT and CTW. “BSL”, “PP”, “LP”, “LG”, and “VG” represent the baseline method DBNet, the predefined prompt, the learnable prompt, the language prompt generator, and the visual prompt generator, respectively. F (%) represents F-measure. $\Delta$ represents the variance.  Method | IC13 $\rightarrow$ IC15 | IC13 $\rightarrow$ TD | IC15 $\rightarrow$ MLT17(en) | TT $\rightarrow$ ArT(-) | ST $\rightarrow$ IC13 | ST $\rightarrow$ IC15
---|---|---|---|---|---|---
TCM | 71.9 | 65.1 | 85.1 | 68.9 | 79.5 | 76.7
w/o VG | 68.4 (-3.5) | 59.4 (-5.7) | 81.8 (-3.3) | 59.1 (-9.8) | 76.3 (-3.2) | 72.6 (-4.1)
w/o LG | 66.1 (-5.8) | 56.8 (-8.3) | 79.7 (-5.4) | 57.8 (-11.1) | 74.5 (-5.0) | 68.2 (-8.5)
w/o VG & LG | 64.8 (-7.1) | 55.7 (-9.4) | 78.4 (-6.7) | 54.2 (-14.7) | 71.7 (-7.8) | 63.9 (-12.8)
Table 7:  Ablation study of the effect of LG and VG on generalization
performance. F-measure (%) is reported.

Ablation Study for the Predefined Prompt. When using the predefined prompt, as
illustrated in the second row of Table 6, the performances are slightly
improved on all four datasets (IC15, TD, TT, and CTW), with 0.05%, 0.2%,
0.04%, and 0.1% higher than the baseline method, respectively.

Ablation Study for the Learnable Prompt. Besides, results combing the
learnable prompt with the predefined prompt on four datasets are provided in
the third row of Table 6. We notice that a consistent improvement can be
achieved by adding the learnable prompt. We also show the influence of using
different numbers of the learnable prompt in row 4 to row 6 of Table 6. We
observe that as the value of the number of the learnable prompt increases, the
performance increases gradually on all datasets. Compared to the value 4, the
value 32 obtains obvious improvements on CTW, TD, and TT. We conjecture that
this is because the larger number of the learnable prompt can better steer the
pretrained text encoder knowledge which is useful for text detection. In the
following experiments, the default number of the learnable prompt is set to 4
for simplicity.

Ablation Study for the Language Prompt Generator. Furthermore, we evaluate the
performance of the proposed language prompt generator shown in 7th row of
Table 6. With the help of the language prompt generator, we find that TCM
achieves further improvements on all four datasets, especially on ICDAR2015,
indicating that the conditional cue generated by the language prompt generator
for each image can ensure better generalization over different types of
datasets.

Ablation Study for the Visual Prompt Generator. Finally, combining the
proposed visual prompt generator with the above other components, the
improvement of F-measure is better than the baseline on all four datasets,
with larger margins of 1.7% and 2.0% on IC15 and TD, respectively. The reason
for this obvious complementary phenomenon is that the visual prompt generator
can propagate fine-grained visual semantic information from textual features
to visual features. Besides, the prompted locality image embedding generated
by the visual prompt generator can guide the model to obtain more accurate
text instance-level visual representations, which boosts the ability of
instance-language matching and generates a precise segmentation score map that
is useful for downstream detection head.

Ablation Study for the VG and LG on Generalization Performance. As described
in Table 7, removing the VG and LG elements from TCM dramatically deteriorates
the generalization performance, which further indicates the effectiveness of
the VG and LG.

Ablation Study for Image Encoder and Text Encoder. We have investigated how
the quality of the frozen text encoder and image encoder affects the
performance via adjusting the corresponding learning rate (LR) factor. The
experimental results of TCM-DBNet on the TD500 dataset are shown in Table 8.
The results show that using a lower learning rate for both encoders and fixing
the text encoder is the optimal setting for training the whole model. Note
that we observe performance degradation when directly using $1.0\times$
learning rate for both encoders, which suggests the frozen text encoder can
stabilize the training process. The cores of the architecture, including the
language prompt generator and visual prompt generator, are designed to better
steer knowledge of the pretrained CLIP. Appropriate design of the network
architecture and the use of the pretrained CLIP are complementary.

| Image encoder | Text encoder | F (%)
---|---|---|---
LR Factor | 0.1 | 0.0 | 88.7
0.1 | 0.1 | 87.8
0.1 | 1.0 | 87.1
1.0 | 1.0 | 86.3
Table 8: Ablation study of exploration on image encoder and text encoder. “LR”
represents the learning rate.

Ablation Study for Different Amount of Data. To further explore whether the
TCM can learn the additional knowledge which is hard to be obtained from
increasing data, we have trained the model on a large-scale public joint data
including IC13, IC15, TD, CTW, TT, and MLT17, with total 13,784 image, and
testing it on a NightTime-ArT data (326 images) carefully collected from ArT.
The nighttime examples of ArT are shown in Fig. 6. Results are shown in Table
9. The results show that even with the addition of large amounts of training
data, existing methods still show limitation to the nighttime data that is
obviously out-of-distribution from the training set. However, TCM can still
perform robust in such case, indicating its irreplaceable potential
generalization ability.

Figure 6: The examples of our constructed NightTime-ArT. Method | Training Data | Testing Data | F (%)
---|---|---|---
FCENet | Joint data | NightTime-ArT | 55.2
DBNet | Joint data | NightTime-ArT | 52.8
TCM-DBNet | Joint data | NightTime-ArT | 70.2
Table 9: Ablation study of exploration on large amounts of training data.

Ablation Study for the Parameters Comparison. For a fair comparison, we have
increased the parameters of DBNet by replacing the backbone with a larger
ResNet and then conduct experiments on TD500 dataset. Trainable parameters and
FLOPs are calculated with an input size 1280$\times$800\. Results are shown in
Table 10. The results show that TCM-DBNet has better performance than DBNet
with less model size and computation overhead, demonstrating its effectiveness
for scene text detection.

Method | Backbone | Params | FLOPs | F (%)
---|---|---|---|---
DBNet | R50 | 26 (M) | 98 (G) | 84.9
DBNet | R101 | 46 (M) | 139 (G) | 85.9
DBNet | R152 | 62 (M) | 180 (G) | 87.3
TCM-DBNet | R50 | 50 (M) | 156 (G) | 88.7
Table 10: Ablation study of the parameters comparison with DBNet.

Ablation Study for the Auxiliary Loss. We further compare the results of with
and without auxiliary loss on TD500 dataset, as shown in Table 11. We see that
using auxiliary loss achieves higher performance. The results indicate
auxiliary loss is beneficial to train the model via imposing constraints on
instance-language matching score map. In addition, the improvement of the
performance suggests that it might help the image encoder of pretrained CLIP
to perceive locality text region effectively.

Model | F (%)
---|---
TCM-DBNet with auxiliary loss | 88.7
TCM-DBNet w/o auxiliary loss | 85.1
Table 11: Ablation study of the auxiliary Loss.

(a) CTW1500 (b) MSRA-TD500

Figure 7: Visualization results of our method. For each pair, the left is the
image embedding $\bm{I}$, and the right is the generated visual prompt
$\tilde{\bm{I}}$. Best view in screen. More results can be found in appendix.
Figure 8: Failure cases. Red circle means false positive region.

##  5 Discussion of Failure Cases

There are some insightful failed cases as shown in Figure 8. The instance-
language matching score map generates false positive region that is very
similar to the characteristics of text, as shown in the region of the red
circle in Fig. 8, which will be considered as noise. Therefore, it is
necessary that the downstream text detection head can further refine this
initial score map instead of directly using the score map of instance-language
matching as the final results. We leave this problem as future work to
alleviate the false positive score map of instance-language matching.

##  6 Conclusion

This paper proposes the TCM, which can directly excavate the prior knowledge
from the CLIP model into a scene text detector without pretraining process.
Such a new text detection paradigm reveals the importance of using visual-
language prior for seeking information from the zero-shot off-the-rack model,
and thus guiding the text detector adapting to small-scale data, divergent
data distribution, and complicated scenes, without relying on carefully-
designed pretraining tasks. Experiments comprehensively demonstrate the
effectiveness of our method. It is worth mentioning that we also construct a
NightTime-ArT dataset to further demonstrate that the TCM can steer useful
prior knowledge from the CLIP model. As the CLIP model is an inborn-friendly
framework for text, extension of TCM to scene text spotting is also a
promising direction for future work.

Acknowledgements This work was supported by the National Natural Science
Foundation of China (No.62225603, No.6220073278, No.62206103), and the
National Key Research and Development Program (No.2022YFC3301703,
No.2022YFC2305102).

## References

  * [1] Youngmin Baek, Bado Lee, Dongyoon Han, Sangdoo Yun, and Hwalsuk Lee.  Character region awareness for text detection.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9357–9366, 2019.
  * [2] Chee-Kheng Chng, Yuliang Liu, Yipeng Sun, Chun Chet Ng, Canjie Luo, Zihan Ni, ChuanMing Fang, Shuaitao Zhang, Junyu Han, Errui Ding, et al.  ICDAR2019 Robust Reading Challenge on Arbitrary-Shaped Text (RRC-ArT).  In ICDAR, pages 1571–1576, 2019.
  * [3] Chee-Kheng Ch’ng, Chee Seng Chan, and Cheng-Lin Liu.  Total-text: toward orientation robustness in scene text detection.  IJDAR, pages 1–22, 2019.
  * [4] Gabriel Goh, Nick Cammarata, Chelsea Voss, Shan Carter, Michael Petrov, Ludwig Schubert, Alec Radford, and Chris Olah.  Multimodal neurons in artificial neural networks.  Distill, 6(3):e30, 2021.
  * [5] Xiuye Gu, Tsung-Yi Lin, Weicheng Kuo, and Yin Cui.  Open-vocabulary object detection via vision and language knowledge distillation.  In ICLR, pages 1–20, 2022.
  * [6] Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman.  Synthetic data for text localisation in natural images.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2315–2324, 2016.
  * [7] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [8] Minghang He, Minghui Liao, Zhibo Yang, Humen Zhong, Jun Tang, Wenqing Cheng, Cong Yao, Yongpan Wang, and Xiang Bai.  Most: A multi-oriented scene text detector with localization refinement.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8809–8818, 2021.
  * [9] Pan He, Weilin Huang, Tong He, Qile Zhu, Yu Qiao, and Xiaolin Li.  Single shot text detector with regional attention.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 3066–3074, 2017.
  * [10] Wenhao He, Xu-Yao Zhang, Fei Yin, and Cheng-Lin Liu.  Deep direct regression for multi-oriented scene text detection.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 745–753, 2017.
  * [11] D. Karatzas, L. Gomez-Bigorda, et al.  ICDAR 2015 competition on robust reading.  In ICDAR, pages 1156–1160, 2015.
  * [12] Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, M. Iwamura, Lluís Gómez i Bigorda, Sergi Robles Mestre, Joan Mas Romeu, David Fernández Mota, Jon Almazán, and Lluís-Pere de las Heras.  Icdar 2013 robust reading competition.  In 2013 12th International Conference on Document Analysis and Recognition, pages 1484–1493, 2013.
  * [13] Boyi Li, Kilian Q. Weinberger, Serge J. Belongie, Vladlen Koltun, and René Ranftl.  Language-driven semantic segmentation.  In ICLR, 2022.
  * [14] Xiang Li, Wenhai Wang, Wenbo Hou, Ruo-Ze Liu, Tong Lu, and Jian Yang.  Shape robust text detection with progressive scale expansion network.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9328–9337, 2019.
  * [15] Minghui Liao, Baoguang Shi, Xiang Bai, Xinggang Wang, and Wenyu Liu.  Textboxes: A fast text detector with a single deep neural network.  In AAAI, pages 4161–4167, 2017.
  * [16] Minghui Liao, Boyu Song, Minghang He, Shangbang Long, Cong Yao, and Xiang Bai.  Synthtext3d: synthesizing scene text images from 3d virtual worlds.  Science China Information Sciences, 63:1–14, 2020.
  * [17] Minghui Liao, Zhaoyi Wan, Cong Yao, Kai Chen, and Xiang Bai.  Real-time scene text detection with differentiable binarization and adaptive scale fusion.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 45:919–931, 2019.
  * [18] Minghui Liao, Zhaoyi Wan, Cong Yao, Kai Chen, and Xiang Bai.  Real-time scene text detection with differentiable binarization.  In AAAI, pages 11474–11481, 2020.
  * [19] Chen-Hsuan Lin, Ersin Yumer, Oliver Wang, Eli Shechtman, and Simon Lucey.  St-gan: Spatial transformer generative adversarial networks for image compositing.  In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9455–9464, 2018.
  * [20] Yuliang Liu, Lianwen Jin, Shuaitao Zhang, Canjie Luo, and Sheng Zhang.  Curved scene text detection via transverse and longitudinal sequence connection.  Pattern Recognition, 90:337–345, 2019.
  * [21] Shangbang Long, Xin He, and Cong Yao.  Scene text detection and recognition: The deep learning era.  International Journal of Computer Vision, 129:161–184, 2020.
  * [22] Shangbang Long, Siyang Qin, Dmitry Panteleev, A. Bissacco, Yasuhisa Fujii, and Michalis Raptis.  Towards end-to-end unified scene text detection and layout analysis.  2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 1039–1049, 2022.
  * [23] Shangbang Long, Jiaqiang Ruan, W. Zhang, Xin He, Wenhao Wu, and Cong Yao.  Textsnake: A flexible representation for detecting text of arbitrary shapes.  In ECCV, pages 1–17, 2018.
  * [24] Nibal Nayef, Yash Patel, Michal Busta, Pinaki Nath Chowdhury, Dimosthenis Karatzas, Wafa Khlif, Jiri Matas, Umapada Pal, Jean-Christophe Burie, Cheng-lin Liu, et al.  ICDAR2019 Robust Reading Challenge on Multi-lingual Scene Text Detection and Recognition–RRC-MLT-2019.  In ICDAR, pages 1454–1459, 2019.
  * [25] Nibal Nayef, Fei Yin, Imen Bizid, Hyunsoo Choi, Yuan Feng, Dimosthenis Karatzas, Zhenbo Luo, Umapada Pal, Christophe Rigaud, Joseph Chazalon, Wafa Khlif, Muhammad Muzzamil Luqman, Jean-Christophe Burie, Cheng-Lin Liu, and Jean-Marc Ogier.  Icdar2017 robust reading challenge on multi-lingual scene text detection and script identification - rrc-mlt.  In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 01, pages 1454–1459, 2017.
  * [26] Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander H. Miller, and Sebastian Riedel.  Language models as knowledge bases?  In EMNLP, page 1772–1791, 2019.
  * [27] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever.  Learning transferable visual models from natural language supervision.  In ICML, pages 1–16, 2021.
  * [28] Yongming Rao, Wenliang Zhao, Guangyi Chen, Yansong Tang, Zheng Zhu, Guan Huang, Jie Zhou, and Jiwen Lu.  Denseclip: Language-guided dense prediction with context-aware prompting.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 18061–18070, 2022.
  * [29] Baoguang Shi, Xiang Bai, and Serge J. Belongie.  Detecting oriented text in natural images by linking segments.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3482–3490, 2017.
  * [30] Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojciech Galuba, and Tal Hassner.  Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text.  2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8798–8808, 2021.
  * [31] Sibo Song, Jianqiang Wan, Zhibo Yang, Jun Tang, Wenqing Cheng, Xiang Bai, and Cong Yao.  Vision-language pre-training for boosting scene text detectors.  In CVPR, pages 15681–15691, 2022.
  * [32] Jun Tang, Zhibo Yang, Yongpan Wang, Qi Zheng, Yongchao Xu, and Xiang Bai.  Seglink++: Detecting dense and arbitrary-shaped scene text by instance-aware component grouping.  Pattern Recognit., 96:106954, 2019.
  * [33] Jingqun Tang, Wenqing Zhang, Hong yi Liu, Mingkun Yang, Bo Jiang, Guan-Nan Hu, and Xiang Bai.  Few could be better than all: Feature sampling and grouping for scene text detection.  In CVPR, pages 4563–4572, 2022.
  * [34] Zhi Tian, Weilin Huang, Tong He, Pan He, and Yu Qiao.  Detecting text in natural image with connectionist text proposal network.  In ECCV, pages 1–16, 2016.
  * [35] Zhuotao Tian, Michelle Shu, Pengyuan Lyu, Ruiyu Li, Chao Zhou, Xiaoyong Shen, and Jiaya Jia.  Learning shape-aware embedding for scene text detection.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 4229–4238, 2019.
  * [36] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, pages 1–11, 2017.
  * [37] Qi Wan, Haoqin Ji, and Linlin Shen.  Self-attention based text knowledge mining for text detection.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5979–5988, 2021.
  * [38] Fangfang Wang, Yifeng Chen, Fei Wu, and Xi Li.  Textray: Contour-based geometric modeling for arbitrary-shaped scene text detection.  In Proceedings of the 28th ACM International Conference on Multimedia, page 111–119, 2020.
  * [39] Wenhai Wang, Enze Xie, Xiaoge Song, Yuhang Zang, Wenjia Wang, Tong Lu, Gang Yu, and Chunhua Shen.  Efficient and accurate arbitrary-shaped text detection with pixel aggregation network.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 8439–8448, 2019.
  * [40] Xiaobing Wang, Yingying Jiang, Zhenbo Luo, Cheng-Lin Liu, Hyunsoo Choi, and Sungjin Kim.  Arbitrary shape scene text detection with adaptive text region representation.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 6442–6451, 2019.
  * [41] Yuxin Wang, Hongtao Xie, Zhengjun Zha, Mengting Xing, Zilong Fu, and Yongdong Zhang.  Contournet: Taking a further step toward accurate arbitrary-shaped scene text detection.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 11750–11759, 2020.
  * [42] Weijia Wu, Ning Lu, Enze Xie, Yuxiang Wang, Wenwen Yu, Cheng Yang, and Hong Zhou.  Synthetic-to-real unsupervised domain adaptation for scene text detection in the wild.  In ACCV, pages 1–14, 2020.
  * [43] Enze Xie, Yuhang Zang, Shuai Shao, Gang Yu, Cong Yao, and Guangyao Li.  Scene text detection with supervised pyramid context network.  In AAAI, pages 9038–9045, 2019.
  * [44] Linjie Xing, Zhi Tian, Weilin Huang, and Matthew R. Scott.  Convolutional character networks.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9125–9135, 2019.
  * [45] Mengde Xu, Zheng Zhang, Fangyun Wei, Yutong Lin, Yue Cao, Han Hu, and Xiang Bai.  A simple baseline for zero-shot semantic segmentation with pre-trained vision-language model.  In ECCV, 2021.
  * [46] Yongchao Xu, Yukang Wang, Wei Zhou, Yongpan Wang, Zhibo Yang, and Xiang Bai.  Textfield: Learning a deep direction field for irregular scene text detection.  IEEE Transactions on Image Processing, 28:5566–5579, 2019.
  * [47] Chuhui Xue, Shijian Lu, and Wei Zhang.  Msr: Multi-scale shape regression for scene text detection.  In IJCAI, pages 989–995, 2019.
  * [48] Chuhui Xue, Wenqing Zhang, Yu Hao, Shijian Lu, Philip H. S. Torr, and Song Bai.  Language matters: A weakly supervised vision-language pre-training approach for scene text detection and spotting.  In ECCV, pages 1–19, 2022.
  * [49] Cong Yao, Xiang Bai, and Wenyu Liu.  A unified framework for multi-oriented text detection and recognition.  IEEE Transactions on Image Processing, 23:4737–4749, 2014.
  * [50] Cong Yao, Xiang Bai, Wenyu Liu, Yi Ma, and Zhuowen Tu.  Detecting texts of arbitrary orientations in natural images.  In 2012 IEEE conference on computer vision and pattern recognition, pages 1083–1090. IEEE, 2012.
  * [51] Jian Ye, Zhe Chen, Juhua Liu, and Bo Du.  Textfusenet: Scene text detection with richer fused features.  In International Joint Conference on Artificial Intelligence, 2020\.
  * [52] Fangneng Zhan, Chuhui Xue, and Shijian Lu.  Ga-dan: Geometry-aware domain adaptation network for scene text detection and recognition.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 9104–9114, 2019.
  * [53] Chengquan Zhang, Borong Liang, Zuming Huang, Mengyi En, Junyu Han, Errui Ding, and Xinghao Ding.  Look more than once: An accurate detector for text of arbitrary shapes.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 10544–10553, 2019.
  * [54] Shi-Xue Zhang, Xiaobin Zhu, Jie-Bo Hou, Chang Liu, Chun Yang, Hongfa Wang, and Xu-Cheng Yin.  Deep relational reasoning graph network for arbitrary shape text detection.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 9696–9705, 2020.
  * [55] Zheng Zhang, Chengquan Zhang, Wei Shen, Cong Yao, Wenyu Liu, and Xiang Bai.  Multi-oriented text detection with fully convolutional networks.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 4159–4167, 2016.
  * [56] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Conditional prompt learning for vision-language models.  In CVPR, pages 16816–16825, 2022.
  * [57] Kaiyang Zhou, Jingkang Yang, Chen Change Loy, and Ziwei Liu.  Learning to prompt for vision-language models.  International Journal of Computer Vision, page 2337–2348, 2022\.
  * [58] Xinyu Zhou, Cong Yao, He Wen, Yuzhi Wang, Shuchang Zhou, Weiran He, and Jiajun Liang.  East: An efficient and accurate scene text detector.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 2642–2651, 2017.
  * [59] Jun-Yan Zhu, Taesung Park, Phillip Isola, and Alexei A. Efros.  Unpaired image-to-image translation using cycle-consistent adversarial networks.  In 2017 IEEE International Conference on Computer Vision (ICCV), pages 2242–2251, 2017.
  * [60] Yiqin Zhu, Jianyong Chen, Lingyu Liang, Zhuanghui Kuang, Lianwen Jin, and Wayne Zhang.  Fourier contour embedding for arbitrary-shaped text detection.  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3122–3130, 2021.

##  Appendix A Appendix

###  A.1 Datasets

ICDAR2013 [12] is high-resolution English dataset for focused scene text
detection, including 229 images for training and 233 images for testing.

ICDAR2015 [11] is a multi-oriented text detection dataset for English text
that includes 1,000 training images and 500 testing images. Scene text images
in this dataset were taken by Google Glasses without taking care of
positioning, image quality, and viewpoint.

MSRA-TD500 [50] is a multi-language dataset that includes English and Chinese,
including 300 training images and 200 testing images. We also include extra
400 training images from HUST-TR400 [49] following the previous methods [18,
58].

CTW1500 [20] consists of 1,000 training images and 500 testing images which
focuses on the curved text. The text instances are annotated in the text-line
level by polygons with 14 vertices.

Total-Text [3] contains 1,255 training images and 300 testing images. The text
instances are labeled at the word level. It includes horizontal, multi-
oriented, and curved text shapes.

ArT [2] includes 5,603 training images and 4,563 testing images. It is a
large-scale multi-lingual arbitrary-shape scene text detection dataset. The
text regions are annotated by the polygons with an adaptive number of key
points. Note that it contains Total-Text and CTW1500.

MLT17 [25] includes 9 languages text representing 6 different scripts
annotated by quadrangle. It has 7,200 training images, 1,800 validation
images, and 9,000 testing images. We use both the training set and the
validation set in the finetune period.

MLT19 [24] is a large-scale multi-lingual scene text detection datasets. It
contains 10,000 training images and 10,000 testing images, and labeled at word
level.

SynthText [6] It contains 800k synthetic images generated by blending natural
images with artificial text, which are all word-level annotated.

TextOCR [30] is a large-scale high quality scene text datasets collected from
Open Images555https://storage.googleapis.com/openimages/web/index.html. It
contains 30 words on average per image. It has 24,902 training images and
3,232 testing images, and is annotated with polygons.

###  A.2 More Quantitative Results

Multi-lingual Real-to-real Adaptation. We conducted multi-lingual
generalization ability experiments as shown in Table 12. The results show that
the pluggable TCM can also benefit to multi-lingual scenarios text detection
via leveraging the pretrained knowledge of CLIP, which demonstrates the
effectiveness of our method for domain adaptation.

Method | MLT17 $\rightarrow$ MLT19
---|---
DBNet [18] | 47.4
TCM-DBNet | 67.5
Table 12: Real-to-real adaptation. F-measure (%) is reported.

Ablation Study for the Different Predefined Language Prompt. We conducted
ablation study on the predefined language prompt with different string using
TCM-DBNet in Table 13. Results show that without predefined language prompt
the performance is harmed. In addition, it can be seen that there is little
performance variation with different predefined language prompt.

Predefined language prompt | IC15
---|---
“Text” | 89.2
“A set of arbitrary-shape text instances” | 89.0
“The pixels of many arbitrary-shape text instances” | 88.9
without predefined language prompt | 87.7
Table 13: Ablation study of the different predefined language prompt.

Ablation Study for Training with Large-scale Dataset. We conducted ablation
study of training TCM-DBNet on IC15 with extra TextOCR [30] data. As shown in
Table 14, when using additional large-scale TextOCR as training data, our
model can achieve further improvement, suggesting the compatibility of our
method with large-scale datasets.

Model | Training data | F (%)
---|---|---
TCM-DBNet | IC15 | 89.2
TCM-DBNet | IC15+TextOCR | 90.4
Table 14: Ablation study of training TCM-DBNet on IC15 with extra TextOCR
data.

Ablation study for CLIP Backbone Generalization. We conducted ablation study
to investigate the generalization performance of DBNet by directly replacing
the backbone of DBNet with CLIP backbone, as shown in Table 15. It shows that
the CLIP-R50 can indeed bring benefit for generalization. However, integrating
with TCM, the performance can be significantly improved. It suggests that
directly using the pre-trained CLIP-R50 is not strong enough to improve the
generalization performance of the existing text detector, which further
indicates that synergistic interaction between the detector and the CLIP is
important.

Model | Backbone | ST $\rightarrow$ IC13 | ST $\rightarrow$ IC13
---|---|---|---
DBNet | R50 | 71.7 | 64.0
DBNet | CLIP-R50 | 73.1 | 67.4
TCM-DBNet | CLIP-R50 | 79.6 | 76.7
Table 15: Ablation study on CLIP backbone. R50 means ResNet50.

###  A.3 More Visualization Results

Conditional Cue. We visualize the t-SNE of the generated conditional cue
($\mathbf{cc}$) on six datasets, as illustrated in Fig. 9. The structured
distribution indicates our model has learned the distribution of every domain
dataset in high-dimensional feature space, which is useful for improving the
generalization ability.

Figure 9: t-SNE of conditional cue ($\mathbf{cc}$). MLT short for MLT17.

Visual Prompt. Fig. 10 \- Fig. 13 are more qualitative results of the image
embedding $\bm{I}$ and the generated visual prompt $\tilde{\bm{I}}$ on
CTW1500, Total-Text, MSRA-TD500, and ICDAR2015, respectively. The visual
prompt $\tilde{\bm{I}}$ has contains fine-grained information of text regions.

Figure 10: Visualization results of our method on CTW1500. For each pair, the
left is the image embedding $\bm{I}$, and the right is the generated visual
prompt $\tilde{\bm{I}}$. Best view in screen. Figure 11: Visualization results
of our method on Total-Text. For each pair, the left is the image embedding
$\bm{I}$, and the right is the generated visual prompt $\tilde{\bm{I}}$. Best
view in screen. Figure 12: Visualization results of our method on MSAR-TD500.
For each pair, the left is the image embedding $\bm{I}$, and the right is the
generated visual prompt $\tilde{\bm{I}}$. Best view in screen. Figure 13:
Visualization results of our method on ICDAR2015. For each pair, the left is
the image embedding $\bm{I}$, and the right is the generated visual prompt
$\tilde{\bm{I}}$. Best view in screen.


$E_{1}=(\psi_{1}<B)\lor((\alpha<\alpha_{R})\land(B<\psi_{1}<A))$ is true, we
can use the first or third implication to conclude that $\frac{{\rm d}O}{{\rm
d}x}\big{\rvert}_{x=x_{R}}<0$. Also, if
$E_{2}=(\psi_{1}>A)\lor((\alpha>\alpha_{R})\land(B<\psi_{1}<A))$ is true, we
can use the second or third implication to conclude that $\frac{{\rm d}O}{{\rm
d}x}\big{\rvert}_{x=x_{R}}>0$.

A direct calculation shows that $A\geq B$ and that $B<\psi_{2}$ if and only if
$1/\rho>\gamma\triangleq\min\\{1,\max\\{0,2\psi_{2}-1\\}\\}$. A direct
calculation also shows that $\alpha_{R}\in[0,1]$ if and only if the conditions
in (167) and (168) hold.

Condition (166) can be obtained through following the observations. First,
notice that both (160) and (163) are linear decreasing functions of $\alpha$.
Second, when $\alpha=1$ we always have $O(x_{L})-O(x_{R})<0$. Therefore, there
exists $\alpha_{C}\leq 1$ such that $(\alpha<\alpha_{C}\Rightarrow
O(x_{L})-O(x_{R})>0)$ and $(\alpha>\alpha_{C}\Rightarrow
O(x_{L})-O(x_{R})<0))$. The expression for $\alpha_{C}$ is given by (41). From
this expression, a direct calculation shows that $\alpha_{C}\geq 0$ if and
only if $\psi_{1}\leq\psi_{2}/(1-\rho|1-\psi_{2}|)$. ∎

To finish the proof of Theorem 9 for $\psi_{1}>\psi_{2}$ we consider the
different scenarios in Table 1. These are studied via cases that are exactly
the same to the Case 1.1 to Case 1.14 for $\psi_{1}<\psi_{2}$ but with
$\alpha<\alpha_{R}$ replaced by $E_{1}$ and $\alpha>\alpha_{R}$ replaced by
$E_{2}$ because when $\psi_{1}>\psi_{2}$ it is $E_{1}$ and $E_{2}$ that
determine the sign of $\frac{{\rm d}O}{{\rm d}x}\big{\rvert}_{x=x_{R}}$. For
example, for the top left-most cell in Table 1, both when $\psi_{1}<\psi_{2}$
or $\psi_{1}>\psi_{2},$ we have that $O$ is convex and its decreasing at
$x_{L}$ and $x_{R}$, so its minimum is at $x=x_{R}$. We omit repeating the
arguments for Case 1.2 to Case 1.14. ∎

###  D.6 Proof of Theorem 10

###### Proof of Theorem 10.

The proof amounts to a long calculus exercise, which we shorten by some
careful observations.

We first notice that $\mathcal{E}^{\infty}_{R_{2}}$ and
$\mathcal{S}^{\infty}_{R_{2}}$ can both be written as function of
${\omega_{2}}$, and we can use this to reduce the optimization problem (29) to
an optimization problem over just one variable.

The variable ${\omega_{2}}$ is a function of $\mu_{0},\mu_{1},\mu_{2}\geq 0$,
and has range $[-\infty,0]$, the value $-\infty$ being achieved when
$\mu^{2}_{\star}=\mu_{2}-\mu^{2}_{1}-\mu^{2}_{2}=0$.

To avoid having to deal with infinities, we make use of the Mobius
transformation $x=\frac{1+{\omega_{2}}}{{\omega_{2}}-1}$, and instead work
with $x$.

If we substitute $x=\frac{1+{\omega_{2}}}{-1+{\omega_{2}}}$ into the left hand
side (45), and substitute (16) in the resulting expression we confirm that
(45) is satisfied.

Furthermore, if we use $x=\frac{1+{\omega_{2}}}{-1+{\omega_{2}}}$ and (16) to
write $x$ as a function of $\mu_{0},\mu_{1},\mu_{2}$, we can use the fact that
$\mu_{0},\mu_{1},\mu_{2}\geq 0$, to conclude that
$x\in[-1,\min\\{1,-1+2\psi_{2}\\}]$.

Our problem is thus equivalent to solving
$\min_{x}(1-\alpha)\mathcal{E}^{\infty}_{R_{2}}+\alpha\mathcal{S}^{\infty}_{R_{2}}$
subject to $x\in[-1,\min\\{1,-1+2\psi_{2}\\}]$. Once we know $x$, any
$\mu_{0},\mu_{1},\mu_{2}$ that satisfies (45) will be a minimizer.

At this point we compute $\frac{{\rm d}}{{\rm
d}x}[(1-\alpha)\mathcal{E}^{\infty}_{R_{2}}+\alpha\mathcal{S}^{\infty}_{R_{2}}]$
and observe that this is a rational function of $x$. The numerator is, apart
from a multiplying constant, $p(x)$, and the denominator is zero if and only
if $x=-1-2\sqrt{\psi_{2}}$ or $x=-1+2\sqrt{\psi_{2}}$. Both are outside of the
range of $x$ unless $\psi_{2}=x=1$. We treat this case separately soon. Since
the zeros of the numerator and of the denominator never cancel out, all of the
critical points are given by $p(x)=0$.

Now we compute the value of the derivative at the extremes of the range of
$x$, namely, $x=-1,1$, or $-1+2\psi_{2}$. The value of the derivative at
$x=-1$ is $F1^{2}(-1+\alpha)<0$, which implies that $x=-1$ is not a minimizer.
The value of the derivative at $x=1$ is $\alpha
F_{1}^{2}+\frac{\psi_{2}\left(F_{\star}^{2}+\tau^{2}\right)}{(\psi_{2}-1)^{2}}>0$,
possibly $+\infty$, which implies that $x=1$ is not a minimizer. The value of
the derivative at $x=-1+2\psi_{2}$ is $\alpha
F_{1}^{2}+\frac{F_{\star}^{2}+\tau^{2}}{(\psi_{2}-1)^{2}}>0$, possibly
$+\infty$, which implies that $x=-1+2\psi_{2}$ is not a minimizer. This
calculation implies that we can assume that
$x\in(-1,\min\\{1,-1+2\psi_{2}\\})$, which we will assume from now on.

Finally, we show that the objective is convex in the domain of $x$, which
implies that there is only one critical point – that is $p(x)=0$ has only one
solution in the domain $(-1,\min\\{1,-1+2\psi_{2}\\})$ – and that this
critical point is a global minimum. To show that the objective is convex, we
compute its second derivative, which is

$\displaystyle\frac{8\psi_{2}\left(F_{1}^{2}\left(4\psi_{2}^{2}+\psi_{2}(3(x-2)x-5)-x^{3}+3x+2\right)+\left(F_{\star}^{2}+\tau^{2}\right)\left(4\psi_{2}+3(x+1)^{2}\right)\right)}{\left(4\psi_{2}-(x+1)^{2}\right)^{3}}.$
(169)

The minimum of denominator for $x\in[-1,\min\\{1,-1+2\psi_{2}\\}]$ is
$\begin{cases}4(\psi_{2}-1)&,\psi_{2}>1\\\
-4(\psi_{2}-1)\psi_{2}&,0\leq\psi_{2}\leq 1\end{cases}$ , which is always non-
negative, and is zero only if $x=1$, or $x=-1+2\psi_{2}$, which have already
been excluded because they are not minimizers. Hence,for
$x\in(-1,\min\\{1,-1+2\psi_{2}\\})$, the denominator is strictly positive.

To show that the numerator is non-negative, we only need to show that
$4\psi_{2}^{2}+\psi_{2}(3(x-2)x-5)-x^{3}+3x+2\geq 0$ in the range of $x$. The
minimum of $4\psi_{2}^{2}+\psi_{2}(3(x-2)x-5)-x^{3}+3x+2$ for
$x\in[-1,\min\\{1,-1+2\psi_{2}\\}]$ is
$\begin{cases}4(\psi_{2}-1)^{2}&,\psi_{2}>1\\\
4(\psi_{2}-1)^{2}\psi_{2}&,0\leq\psi_{2}\leq 1,\end{cases}$ which is always
non-negative. ∎

###  D.7 Proof of Theorem 11

###### Proof of Theorem 11.

Similar to proof of Theorem 10, the majority of the proof is a long calculus
exercise.

We first notice that $\mathcal{E}^{\infty}_{R_{3}}$ and
$\mathcal{S}^{\infty}_{R_{3}}$ can both be written as function of
${\omega_{1}}$ and $\zeta^{2}$.

The variable ${\omega_{1}}$ is a function of $\mu_{0},\mu_{1},\mu_{2}\geq 0$,
and has range $[-\infty,0]$, the value $-\infty$ being achieved when
$\mu^{2}_{\star}=\mu_{2}-\mu^{2}_{1}-\mu^{2}_{2}=0$.

To avoid having to deal with infinities, we make use of the Mobius
transformation $x=\frac{1+{\omega_{1}}}{{\omega_{1}}-1}$, and instead work
with $x$. If we use $x=\frac{1+{\omega_{1}}}{{\omega_{1}}-1}$ and (18) to
write $x$ as a function of $\mu_{0},\mu_{1},\mu_{2}$, we can use the fact that
$\mu_{0},\mu_{1},\mu_{2}\geq 0$, to conclude that
$x\in[-1,\min\\{1,-1+2\psi_{1}\\}]$.

Our problem is thus equivalent to solving
$\min_{x}(1-\alpha)\mathcal{E}^{\infty}_{R_{3}}+\alpha\mathcal{S}^{\infty}_{R_{3}}$
subject to $x\in[-1,\min\\{1,-1+2\psi_{1}\\}]$ and $\zeta^{2}\geq 0$.

Let
$O=(1-\alpha)\mathcal{E}^{\infty}_{R_{3}}+\alpha\mathcal{S}^{\infty}_{R_{3}}$.
With a direct calculation we can check the following. The derivative
$\frac{{\rm d}O}{{\rm d}x}|_{x=-1}$ is always negative which implies that
there is no local minimum at $x=-1$. The derivatives $\frac{{\rm d}O}{{\rm
d}x}|_{x=1}$ and $\frac{{\rm d}O}{{\rm d}x}|_{x=-1+2\psi_{1}}$ are always
positive, which implies that there is no local minimum at either $x=1$ or
$x=-1+2\psi_{1}$. Therefore, we know that the minimizer has
$x\in(-1,\min\\{1,-1+2\psi_{1}\\})$. For $x\in(-1,\min\\{1,-1+2\psi_{1}\\})$,
the derivative $\frac{{\rm d}O}{{\rm d}(\zeta^{2})}|_{\zeta^{2}=0}$ is always
positive, which implies that there is no local minimum at $\zeta^{2}=0$. We
thus know that the minimizer of $O$ is in the interior of the domain and hence
it can be found via $\nabla O=0$.

The remainder of the proof considers a few different cases depending on the
value of $\alpha$ and $\psi_{1}$.

Case when $\alpha=0$:

In this case $O=\mathcal{E}^{\infty}_{R_{3}}$ and $\frac{{\rm d}O}{{\rm
d}(\zeta^{2})}=\frac{{F_{1}}^{2}(x+1)^{2}}{\zeta^{4}\left((x+1)^{2}-4{\psi_{1}}\right)}$.
The only way that $\frac{{\rm d}O}{{\rm d}(\zeta^{2})}=0$ is if
$\zeta^{2}=\infty$, which corresponds to $\mu_{\star}=0$.

As we noted in the beginning, $O$ is a function of ${\omega_{1}}$ and
$\zeta^{2}$, and since ${\omega_{1}}$ is a function $\zeta^{2}$ and
$\mu^{2}_{1}$ and $\zeta^{2}$ is a function of $\mu^{2}_{1}$ and
$\mu^{2}_{\star}$, we know that $O$ is a function of $\mu^{2}_{1}$ and
$\mu^{2}_{\star}$. We express $O$ in these variables and compute $\frac{{\rm
d}O}{{\rm d}(\mu_{1}^{2})}$ when $\mu_{\star}\to 0$. We get that

$\frac{{\rm d}O}{{\rm
d}(\mu_{1}^{2})}=-\frac{2F_{1}^{2}\lambda^{2}\psi_{1}^{3}}{\left(2\mu_{1}^{2}\psi_{1}\left(\lambda-\mu_{1}^{2}\right)+\psi_{1}^{2}\left(\lambda+\mu_{1}^{2}\right){}^{2}+\mu_{1}^{4}\right){}^{3/2}}.$
(170)

By minimizing the denominator with respect to $\mu_{1}\geq 0$, we conclude
that its minimum is $\lambda^{2}\psi^{2}_{1}>0$, which implies that
$\frac{{\rm d}O}{{\rm d}(\mu_{1}^{2})}<0$, which implies that
$\mu_{1}\to\infty$.

In this case, if we express $O$ as a function of $\mu_{1}$ and $\mu_{\star}$
and take $\mu_{1}\to\infty$ and $\mu_{\star}\to 0$, we get $O\to F^{2}_{s}$.

Case when $\psi_{1}=1\land 0<\alpha\leq\frac{1}{4}$:

In this case $\frac{{\rm d}O}{{\rm
d}(\zeta^{2})}=\frac{(\alpha-1){F_{1}}^{2}(x+1)^{2}}{(x-1)(x+3){\zeta^{4}}}$.
The only way that $\frac{{\rm d}O}{{\rm d}(\zeta^{2})}=0$ is if
$\zeta^{2}=\infty$, which corresponds to $\mu_{\star}=0$.

As we noted in the beginning, $O$ is a function of ${\omega_{1}}$ and
$\zeta^{2}$, and since ${\omega_{1}}$ is a function $\zeta^{2}$ and
$\mu^{2}_{1}$ and $\zeta^{2}$ is a function of $\mu^{2}_{1}$ and
$\mu^{2}_{\star}$, we know that $O$ is a function of $\mu^{2}_{1}$ and
$\mu^{2}_{\star}$. We express $O$ in these variables and compute $\frac{{\rm
d}O}{{\rm d}(\mu_{1}^{2})}$ when $\mu_{\star}\to 0$. We get that

$\frac{{\rm d}O}{{\rm
d}(\mu_{1}^{2})}=\frac{2F_{1}^{2}\lambda^{2}\left(\frac{4\alpha\lambda\left(\lambda+4\mu_{1}^{2}\right)}{\left(\sqrt{\lambda\left(\lambda+4\mu_{1}^{2}\right)}+\lambda\right){}^{2}}-1\right)}{\left(\lambda\left(\lambda+4\mu_{1}^{2}\right)\right){}^{3/2}}.$
(171)

By maximizing the numerator with respect to $\mu^{2}_{1}\geq 0$, we conclude
that its value is always strictly smaller than
$2F^{2}_{1}\lambda^{2}(-1+4\alpha)\leq 0$. Hence, $\frac{{\rm d}O}{{\rm
d}(\mu_{1}^{2})}<0$, which implies that the optimum is achieved for
$\mu_{1}\to\infty$.

In this case, if we express $O$ as a function of $\mu_{1}$ and $\mu_{\star}$
and take $\mu_{1}\to\infty$ and $\mu_{\star}\to 0$, we get $O\to\alpha
F^{2}_{1}+(1-\alpha)F^{2}_{s}$.

Case when $\psi=1\land\frac{1}{4}<\alpha<1$:

This case is very similar to the previous case. The only difference being
that, because $\frac{1}{4}<\alpha<1$, when we solve

$\frac{{\rm d}O}{{\rm
d}(\mu_{1}^{2})}=\frac{2F_{1}^{2}\lambda^{2}\left(\frac{4\alpha\lambda\left(\lambda+4\mu_{1}^{2}\right)}{\left(\sqrt{\lambda\left(\lambda+4\mu_{1}^{2}\right)}+\lambda\right){}^{2}}-1\right)}{\left(\lambda\left(\lambda+4\mu_{1}^{2}\right)\right){}^{3/2}}=0,$
(172)

we now get two possible solutions, namely,

$\displaystyle\mu^{2}_{1}\in\left\\{\frac{-4\alpha^{2}\lambda+3\alpha\lambda+\left(-\sqrt{\alpha}\right)\lambda}{16\alpha^{2}-8\alpha+1},\frac{-4\alpha^{2}\lambda+3\alpha\lambda+\sqrt{\alpha}\lambda}{16\alpha^{2}-8\alpha+1}\right\\}.$
(173)

First notice that if $\lambda=0$, both expressions give $\mu_{1}=0$. Let us
assume now that $\lambda>0$. If we maximize the first expression with respect
to $\frac{1}{4}<\alpha<1$, we conclude that its value is always smaller than
$-((3\lambda)/16)<0$, which implies that it is not a valid solution in the
range $\mu^{2}_{1}\geq 0$. If we minimize the second expression with respect
to $\frac{1}{4}<\alpha<1$, we conclude that its value is always non-negative,
which implies it is a valid solution in the range $\mu^{2}_{1}\geq 0$. Given
that this is the only stationary point of $O$ and given that $O$ is bounded
from below it must be a global minimum.

If we substitute the optimal values for $\mu^{2}_{1}$ and $\mu^{2}_{\star}=0$
in $O$, we get
$O=\left(4\sqrt{\alpha}-1-3\alpha\right){F_{1}}^{2}+(1-\alpha){F_{s}}^{2}$.

Case when $\psi_{1}\neq 1$:

In this case $\frac{{\rm d}O}{{\rm
d}(\zeta^{2})}=\frac{(\alpha-1){F_{1}}^{2}(x+1)^{2}}{{\zeta}^{4}\left(4{\psi_{1}}-(x+1)^{2}\right)}$.
The only way that $\frac{{\rm d}O}{{\rm d}(\zeta^{2})}=0$ is if
$\zeta^{2}=\infty$, which corresponds to $\mu_{\star}=0$.

If we substitute $x=\frac{1+{\omega_{1}}}{{\omega_{1}}-1}$ into the left hand
side of the last equality in (54), namely,
$\mu_{1}^{2}(-1+2\psi_{1}-x)(-1+x)+2\lambda\psi_{1}(1+x)$, and let
$\mu_{\star}\to 0$, we get $0$, which confirms the condition on $x$ when
$\psi_{1}\neq 1$.

We take $\zeta\to\infty$ in $O$ to obtain,

$\displaystyle
O={F_{1}}^{2}\left(\frac{{\psi_{1}}(x-1)^{2}}{4{\psi_{1}}-(x+1)^{2}}+\alpha
x\right)-(\alpha-1){F_{\star}}^{2},$ (174) $\displaystyle\frac{{\rm d}O}{{\rm
d}x}={F_{1}}^{2}\left(\alpha+\frac{4{\psi_{1}}(x-1)(2{\psi_{1}}-x-1)}{\left((x+1)^{2}-4{\psi_{1}}\right)^{2}}\right),$
(175) $\displaystyle\frac{{\rm d}^{2}O}{{\rm
d}x^{2}}=\frac{8{F_{1}}^{2}{\psi_{1}}\left(4{\psi_{1}}^{2}+{\psi_{1}}(3(x-2)x-5)-x^{3}+3x+2\right)}{\left(4{\psi_{1}}-(x+1)^{2}\right)^{3}}.$
(176)

If we minimize $\frac{{\rm d}^{2}O}{{\rm d}x^{2}}$ over
$x\in[-1,\min(1,-1+2\psi_{1})]$ and $\psi_{1}\geq 0$, we obtain
$F^{2}_{1}/8>0$. This shows that our objective $O$ is strictly convex in the
range $x\in[-1,\min(1,-1+2\psi_{1})]$ and hence there is only one solution to
$\frac{{\rm d}O}{{\rm d}x}=0$.

The rational function $\frac{{\rm d}O}{{\rm d}x}$ has a denominator which is
zero only if $x=-1-2\sqrt{\psi_{1}}$ or $x=-1+2\sqrt{\psi_{1}}$, both of which
are outside the valid range for $x$. Hence the unique solution to $\frac{{\rm
d}O}{{\rm d}x}=0$ comes from the zeros of the numerator of $\frac{{\rm
d}O}{{\rm d}x}$. This numerator is (a constant times) a polynomial in $x$
whose coefficients are described in Theorem 11.

∎


# Audio Spoofing Verification using Deep Convolutional Neural Networks by
Transfer Learning

###### Abstract

Automatic Speaker Verification systems are gaining popularity these days;
spoofing attacks are of prime concern as they make these systems vulnerable.
Some spoofing attacks like Replay attacks are easier to implement but are very
hard to detect thus creating the need for suitable countermeasures. In this
paper, we propose a speech classifier based on deep-convolutional neural
network to detect spoofing attacks. Our proposed methodology uses acoustic
time-frequency representation of power spectral densities on Mel frequency
scale (Mel-spectrogram), via deep residual learning (an adaptation of
ResNet-34 architecture). Using a single model system, we have achieved an
equal error rate (EER) of 0.9056% on the development and 5.32% on the
evaluation dataset of logical access scenario and an equal error rate (EER) of
5.87% on the development and 5.74% on the evaluation dataset of physical
access scenario of ASVspoof 2019.

Index Terms: Spoofing detection, Mel-Spectrogram, Deep-Convolutional Neural
Network

##  1 Introduction

Biometrics technologies play a critical role in regulating access to
informational resources in today's world. One of the reliable approaches for
attaining a suitable secure system is speaker verification.

Automatic speaker verification (ASV) has undergone rapid improvements in the
recent decade but continues to show high vulnerability towards different
spoofing attacks. Spoofing methods are categorized as speech synthesis (SS),
voice conversion (VC), impersonation and replay attacks [1]. Among these,
replay attacks are arguably the most simple yet highly indistinguishable ASV
spoofing technique as they do not require the attackers to have any
specialized knowledge, and can be mounted with relative ease using consumer
devices.

For the first automatic speaker verification spoofing and counter-measures
challenge (ASVspoof 2015 [2]) even though the best results showed an overall
average detection EER of less than 1.5%, the EER of unknown attacks is five
times higher than that of known attacks. In addition, while some attacks were
easily and consistently detected, others provoked extremely high error rates
nearing 50%.

Systems submitted under ASVspoof 2017 [3] challenge inspected distinct front-
end features and the usage of various classifiers to detect replay attack
under conditions. Among them, the best-performing system [4], had an equal
error rate (EER) of 6.73% and used a light convolutional neural network (LCNN)
to extract high-level features from the log power spectrum, together with a
Gaussian Mixture Model (GMM) as classifier. Variable length Teager energy
operator-energy separation algorithm-instantaneous frequency cosine
coefficients (VESA-IFCCs) [5] were proposed as a single system with the aim of
capturing the spectral changes due to the transmission and channel
characteristic of replay devices. A single frequency filtering feature with a
high spectro-temporal resolution was proposed [6] to capture the channel
information embedded in the low signal to noise ratio region.

ASVspoof 2019 [7] Challenge was organized to emphasize the development of
reliable countermeasures that could efficiently segregate bona fide and
spoofed speech. The initiative aims specifically to encourage the design of
generalized countermeasures, i.e. countermeasures that would perform well when
encountered with spoofing attacks which are unpredictable in nature. The data
used for ASVspoof 2015 [2] contained spoofed speech samples generated using
text-to-speech (TTS) and voice conversion (VC) systems at that time. The
ASVspoof 2017 [3] challenge emphasized on the design of countermeasures
strived at detecting replay spoofing attacks that could be implemented easily
by anyone using conventional devices. The ASVspoof 2019 [7] edition was the
first to focus on countermeasures for all three major spoofing attack types,
namely those originating from TTS, VC and replay attacks. ASVspoof 2019 [7]
focuses to develop next-generation countermeasures for the automatic detection
of spoofed or fake audio. The challenge encloses two separate sub-challenges
in logical and physical access control and provides a common database of the
most advanced spoofing attacks to date. The aim was to study the extremities
of spoofing countermeasures with respect to automatic speaker verification and
spoofed audio detection.

In this paper, we propose the use of Mel-spectrograms [8] which is obtained
from the audio frames as a time-frequency representation of power spectral
density, for the training of deep-convolutional neural networks for audio
spoofing attack detection. The use of Mel-spectrograms provides a time-
frequency representation with sufficient spectral resolution, which is
required for robust replay attack detection. Our framework is based on
adapting the ResNet-34 architecture [9].

##  2 Methodology

Our proposed framework (Figure 1) employs transfer learning [10] of a pre-
trained convolutional neural network (CNN) for fast adaptation to the Mel-
spectrograms extracted from speech inputs. Transfer learning is an analysis
technique in machine learning that used to solve a problem by reusing
knowledge gained from solving another but similar problem. Extracted
spectrograms are fed into the deep convolutional neural network in which the
speech signals are classified as bona fide or spoofed. In the following
subsections, we describe the three major components of our system: Transfer
Learning [10], Mel-spectrograms [8] and ResNet [9], followed by a functional
overview of our proposed framework.

###  2.1 Transfer Learning

Transfer Learning is a machine learning technique that is used to solve a
problem based on knowledge gained from solving another but related problem. In
this approach, the features gained from training a base neural network is
reused or adjusted to a target neural network for its training. Transfer
learning is usually expressed through the use of pre-trained models. A pre-
trained model is trained using a large benchmark dataset to solve a similar
problem. The computational cost of training such models is high, So it is
common practice to reuse models from published literature (e.g. VGG,
Inception, MobileNet, ResNet). Pre-trained models used in transfer learning
are usually based on large convolutional neural networks (CNN). High
performance and the easiness in training are two of the main factors that
makes CNN popular for such applications.

Figure 1: Overview of the proposed framework for speech spoofing detection.

###  2.2 Mel-spectrograms

Figure 2: Mel-spectrogram of an audio.

A Mel-spectrogram represents an acoustic time-frequency representation of a
sound, i.e. the power spectral density $P(f,t)$. It is sampled into a number
of points at equal intervals of time $t\textsubscript{i}$ and frequencies
$f\textsubscript{j}$ (on a Mel frequency scale).

The Mel-frequency scale is defined as:

$Mel=2595log\textsubscript{10}(1+\frac{Hertz}{700})$ (1)

And its inverse is:

$Hertz=700(10\textsuperscript{$\frac{Mel}{2595}$}-1)$ (2)

STFT (short-time Fourier transform) and Mel-spectrogram have been the most
popular input representations for audio classifications [8]. Mel-spectrograms
provide an efficient and perceptually relevant representation compared to STFT
[11] and have been shown to perform well in various tasks [12]-[16]. However,
an STFT is closer to the original signal and neural networks may be able to
learn a representation that is more optimal to the given task than Mel-
spectrograms. The latter requires large amount of training data; however as
reported in [17] where using Mel-spectrograms outperformed STFT with a smaller
dataset.

Mel-spectrogram transforms the input raw sequence to a 2-D feature map where
one of the dimensions represents time and the other one represents frequency
and the values represent amplitude. The above Figure 2 shows the Mel-
spectrogram of an audio signal. 2-D Mel Model betters the 1-D raw wave model
but the average of the two outperforms each individual model significantly
[18]. The Mel-spectrogram is selected since it is psychologically relevant and
computationally efficient. It produces a Mel-scaled frequency representation
which is a more appropriate conjecture of human auditory understanding [11]
and typically involves compressing the frequency axis of short-time Fourier
transform representation. And the magnitude of Mel-spectrogram is mapped to
the decibel scale.

###  2.3 Deep Residual Net (ResNet)

The deeper the Neural Network the more difficult it is to train. It is well
known that the depth of a network is a determining factor of the network
performance. Deeper neural networks are used in the field of computer vision.
However, it is not easy to train a deeper network due to the notorious
gradient vanishing problem. This was one of the drawbacks of VGG net [19] as
they started losing generalization capability after some depth. To address
this problem, an effective method called residual neural network(ResNet)[9] is
introduced. ResNet provides a training framework to ease the training of
networks that are substantially deeper than those used previously. It was
motivated by the counter intuitive experimental findings that adding more
layers lead to higher training error. Theoretically, as the number of layers
increased, the modeling capabilities of the Neural Networks should be better
and therefore, the deeper networks should produce no higher training error.
Experiments in [9] showed that ResNet greatly improves training efficiency
since the gradients can propagate many layers through the shortcut connection.
In addition, ResNet allows deeper networks to be trained, resulting in models
that usually perform better. One of the problems ResNets solves is known as
the vanishing gradient. This is because when the network is too deep, the
gradients from where the loss function is calculated easily shrink to zero
after several applications of the chain rule. This results in the weights
never getting updated and therefore, no learning process is involved. With
ResNets, the gradients flow directly through the identity function.

Deep residual learning [9] enables the training of CNNs that have
substantially deeper architectures. It introduces skip connections that enable
gradient flow across a large number of layers thus relieves the problem of
vanishing gradients in deep CNNs. These skip connections enables the outputs
to learn a residual mapping. The residual block forms the basic building block
of a ResNet (Figure 3). If the desired mapping to be learned is
$\mathcal{H}(x)$, the stacked residual layers learn the residual mapping,
$\mathcal{F}(x)=\mathcal{H}(x)-x$. Thus, the original mapping to be learned
becomes $\mathcal{F}(x)+x$. ResNet uses the Rectified linear unit (ReLU)
activation function.

Figure 3: Residual learning: a building block [9].

##  3 Experiments

###  3.1 Dataset

Table 1: Number of non-overlapping target speakers and number of utterances in
training and development sets of the ASVspoof 2019 database [20].

| Speakers | Utterances
---|---|---
Subset | Male | Female | Logical Access | Physical Access
Bona fide | Spoofed | Bona fide | Spoofed
Training | 8 | 12 | 2,580 | 22,800 | 5,400 | 48,600
Development | 8 | 12 | 2,548 | 22,296 | 5,400 | 24,300

####  3.1.1 Data Conditions: Logical Access

Logical access includes spoofing attacks generated with text-to-speech (TTS)
and voice conversion (VC) attacks. Genuine speech is collected from speakers
with no significant channel or background noise effects. Spoofed speech is
generated from genuine data using a number of possible spoofing algorithms.
The full dataset is split into three subsets, the first one is for training,
the second is for development and the third is for evaluation. The number of
speakers in the former two subsets is illustrated in Table 1. There is no
speaker overlap in these three subsets regarding target speakers used in voice
conversion or TTS technique. The voice conversion systems are based on (1)
neural-network-based and (2) transfer-function-based methods. The speech
synthesis systems were done with (1) waveform concatenation, (2) neural-
network-based parametric speech synthesis using source-filter vocoders and (3)
neural-network-based parametric speech synthesis using Wavenet.

####  3.1.2 Data Conditions: Physical Access

The spoofing attacks that at the sensor level are considered as physical
access scenario. Both bona fide and spoofed signals propagate through a
physical space prior to the acquisition. Spoofing attacks in this scenario are
therefore referred to as replay attacks, whereby are cording of a bona fide
access attempt is first captured, presumably surreptitiously, before being
replayed to the ASV microphone.

Table 1 shows the number of speakers and the number of bona fide and replay
spoofing access attempts (utterances) in and development set. Both bona fide
and replay spoofing access attempts in both training and development data
partitions were generated with respect to the same set of randomly selected
acoustic and replay configurations.

In a similar fashion to the logical access scenario, the evaluation set is
disjoint in terms of speakers. Evaluation data is generated in the same manner
as training and development data, albeit with different, randomly acoustic and
replay configurations. Specifically, the set of room sizes, levels of
reverberation, speaker-to-ASV microphone distances, attacker-to-speaker
recording distances, and loudspeaker qualities, while drawn from the same
categories, will be different. Full details of the dataset and performance
measures of ASVspoof 2019 are presented in evaluation plan [20].

###  3.2 Performance measures

Improvements in Countermeasure (CM) technology need not necessarily imply an
improved complete system that involves co-operation of CM and ASV. Thus, the
main focus is on assessment of tandem systems where the CM act as a ‘mediator’
to determine whether a given speech input originates from a bona fide
(genuine) user, before passing it as the main biometric verifier (the ASV
system). The cost function used in this scenario is a recently-proposed tandem
detection cost function (t-DCF) [21] as its primary performance metric. Other
than t-DCF, the evaluation of the equal error rate (EER) of each submitted
countermeasure is used as a secondary performance metric [20].

####  3.2.1 Tandem-Detection Cost Function (t-DCF)

The t-DCF [21] is mainly consisted of statistical detection theory and
involves detailed specification of a specified application. A key feature of
t-DCF involves the assessment of a tandem system while on the other hand,
keeping the two subsystems (CM and ASV) isolated from each other, i.e. they
can be developed independently of each other.

The t-DCF metric has 6 parameters: (i) false alarm and miss costs for both
systems, and (ii) prior probabilities of target and spoof trials (with an
implied third, non-target prior).

$t-DCF(s)=C_{1}P_{miss}^{cm}(s)+C_{2}P_{fa}^{cm}(s)$ (3)

where $P_{miss}^{cm}(s)$ and $P_{fa}^{cm}(s)$ are, respectively, the miss rate
and the false alarm rate of the CM system at threshold s. The constants C2 and
C1 are dictated by the t-DCF costs, priors, and the ASV system detection
errors.

####  3.2.2 Equal Error Rate (EER)

In scoring-based spoofing detection tasks, EER is used as a secondary metric
to analyze the performance of different counter measures methods. Let
FRR($\theta$) and FAR($\theta$) denote the false rejection rate and false
acceptance rate at threshold $\theta$.

$FAR(\theta)=\frac{count(spoof\ trials\ with\ score)>\theta}{total\ spoof\
trials}$ (4) $FRR(\theta)=\frac{count(human\ trials\ with\
score)<\theta}{total\ human\ trials}$ (5)

$FRR(\theta)$ and $FAR(\theta)$ are monotonically decreasing and increasing
functions of $\theta$. The EER corresponds to the threshold $\theta$EER at
which two detection error rates are equal.

###  3.3 Model Setup

Our model was trained only on the training set and validated on the
development set of the ASVspoof 2019 dataset. The model was trained in Google
Colab with the granted GPU (Tesla K80). Audio preprocessing was performed via
the Google Cloud Platform. Python’s fastai library was used to train CNN which
provides support for vision, text, tabular, and collaborative filtering
models. The fastai library is a high-level library build on PyTorch which
enables easy prototyping and gives you access to a lot of state-of-the-art
methods such as marketing application (horizontal application) and health care
application (vertical application). Mel-spectrograms were extracted from the
audio files using FFT window of size 2048 and the number of samples between
successive frames was taken as 512. The Mel-spectrograms were further resized
to 224x224 before feeding into the CNN. ResNet-34 training ran on Google Colab
with ADAM optimizer and a learning rate in range of 0.000001, over 8 epochs
and a batch size of 64.

##  4 Results and Discussion

Table 2: The experiment result obtained from ASVspoof 2019.

Dataset | EER | t-DCF
---|---|---
Development | Evaluation | Development | Evaluation
| Logical Access
---
0.9056 | 5.32 | - | 0.1514
| Physical Access
---
5.87 | 5.74 | - | 0.1351

Table 3: The experiment result of baseline in ASVspoof 2019. Baseline Implementation | Logical Access | Physical Access
---|---|---
EER | t-DCF | EER | t-DCF
CQCC - GMM | 9.57 | 0.2366 | 11.04 | 0.2454
LFCC - GMM | 8.09 | 0.2116 | 13.54 | 0.3017

Table 2 shows the result of our model in ASVspoof 2019 challenge based on the
two performance matrices (t-DCF and EER) on evaluation and development
dataset. Comparing the result of ASVspoof 2019 challenge, our single model
outperforms the baseline models provided by ASVspoof 2019 organizers given in
Table 3. But when compared to other teams, the result that we obtained in the
testing phase was low. We had investigated the reason behind this problem and
found that in the test dataset, spoofed data are generated according to
diverse unseen spoofing algorithms. However, they are variations of the
spoofing algorithms used to generate the development dataset.

Among the systems submitted under ASVspoof 2019 challenge, the best-performing
system under logical access scenario, had a min-tDCF of 0.2093 and an equal
error rate (EER) of 11.40% (while evaluating a single system). By comparing
this with our single system, we had obtained a better result in both the
performance metrics. Along with that, while analyzing the result and spoofing
attack algorithms provided by ASVspoof 2019 committee, our model performed
well in detecting spoofing attacks which uses voice conversion techniques by
waveform filtering (A17). In both logical and physical access scenario, we are
one among the top three teams while considering both primary and single system
as same. We had proposed the same methodology, rather than going for different
methodologies to counter both logical and physical access scenarios. Also,
t-DCF and EER showed correlation of 0.99686 and 0.96886 in physical and
logical access scenarios respectively.

##  5 Conclusion

In this paper, we explored the applicability of the transfer learning approach
for the solution to the problem of spoofing detection. Our first finding is
that, instead of feeding raw audio directly to the model, we use an advanced
approach to audio classification using Mel-spectrogram. Mel-spectrogram
transforms the input raw sequence to a 2-D feature map in which one dimension
represents time and the other represents frequency and the values represent
amplitude. A second finding is that residual neural networks can help us build
deeper models that outperform conventional deep neural networks.

Finally, we demonstrated that even with a single model, better system
performance can be obtained. By using different features and models system
performance can be significantly improved. In future, we would like to explore
the effectiveness of using more advanced generative adversarial networks (GAN)
for the spoofing detection task. Denoising methods like Adaptive filtering and
Autoencoders could also be implemented.

##  6 Acknowledgements

The authors would like to thank ASVspoof 2019 organizers for providing the
dataset and detailed analysis of our system. We also acknowledge the technical
support from Sachin Dev S, working at Tricodia Inc.

## References

  * [1] Z. Wu, N. Evans, T. Kinnunen, J. Yamagishi, F. Alegre, and H. Li, ”Spoofing and countermeasures for speaker verification: A survey,” in speech communication, 66, 2015, pp. 130–153.
  * [2] Z. Wu, T. Kinnunen, N. Evans, J. Yamagishi, C. Hanilçi, M. Sahidullah, and A. Sizov, ”ASVspoof 2015: the first automatic speaker verification spoofing and countermeasures challenge,” in Sixteenth Annual Conference of the International Speech Communication Association, 2015.
  * [3] T. Kinnunen, M. Sahidullah, H. Delgado, M. Todisco, N. Evans, J. Yamagishi, and K.A. Lee, ”The ASVspoof 2017 challenge: Assessing the limits of replay spoofing attack detection,” 2017.
  * [4] G. Lavrentyeva, S. Novoselov, E. Malykh, A. Kozlov, O. Kudashev, and V. Shchemelinin, ”Audio Replay Attack Detection with Deep Learning Frameworks,” in Interspeech, August, 2017, pp 82–86.
  * [5] H.A. Patil, H.A. Kamble, T.B. Patel, and M.H. Soni, ”Novel Variable Length Teager Energy Separation Based Instantaneous Frequency Features for Replay Detection,” in INTERSPEECH, August, 2017, pp 12–16.
  * [6] K.R. Alluri, S. Achanta, S.R. Kadiri, S.V. Gangashetty, and A.K. Vuppala, ”SFF Anti-Spoofer: IIIT-H Submission for Automatic Speaker Verification Spoofing and Countermeasures Challenge 2017,” in Interspeech, August, 2017, pp. 107–111.
  * [7] ASVspoof2019:the automatic speaker verification spoofing and countermeasure.[Online]. www.asvspoof.org
  * [8] K. Choi, G. Fazekas, K. Cho, and M. Sandler, ”A tutorial on deep learning for music information retrieval,” arXiv preprint arXiv:1709.04396, 2017.
  * [9] K. He, X. Zhang, S. Ren, and J. Sun, ”Deep residual learning for image recognition,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770–778.
  * [10] S.J. Pan, and Q. Yang, 2010. ”A survey on transfer learning,” in IEEE Transactions on knowledge and data engineering, 22(10), 2010, pp. 1345–1359.
  * [11] B.C. Moore, ”An introduction to the psychology of hearing,” Brill, 2012.
  * [12] A. Van den Oord, S. Dieleman, and B. Schrauwen, “Deep content-based music recommendation,” in Advances in Neural Information Processing Systems, 2013, pp. 2643-–2651.
  * [13] S. Dieleman, and B. Schrauwen, “End-to-end learning for music audio,” in Acoustics, Speech and Signal Processing (ICASSP), 2014 IEEE International Conference on. IEEE, 2014, pp. 6964–-6968.
  * [14] J. Schlüter, and S. Bock, “Improved musical onset detection with convolutional neural networks,” in Acoustics,Speech and Signal Processing, IEEE International Conference on. IEEE, 2014.
  * [15] K. Ullrich, J. Schlüter, and T. Grill, “Boundary detection in music structure analysis using convolutional neural networks,” in Proceedings of the 15th International Society for Music Information Retrieval Conference (ISMIR 2014), Taipei, Taiwan, 2014.
  * [16] K. Choi, G. Fazekas, and M. Sandler, “Automatic tagging using deep convolutional neural networks,” in The 17th International Society of Music Information Retrieval Conference, New York, USA. ISMIR, 2016.
  * [17] K. Choi, G. Fazekas, and M. Sandler, ”Automatic tagging using deep convolutional neural networks,” arXiv preprint arXiv:1606.00298, 2016.
  * [18] https://github.com/CVxTz/audio_classification
  * [19] K. Simonyan and A. Zisserman, ”Very deep convolutional networks for large-scale image recognition,” arXiv preprint arXiv:1409.1556, 2014.
  * [20] ASVspoof2019:the automatic speaker verification spoofing and countermeasure challenge evaluation plan.[Online]. Available: http://www.asvspoof.org/asvspoof2019/asvspoof2019evaluation-
plan.pdf

  * [21] T. Kinnunen, K. Lee, H. Delgado, N. Evans, M. Todisco, M. Sahidullah, J. Yamagishi, and D.A. Reynolds, ”t-DCF: a detection cost function for the tandem assessment of spoofing countermeasures and automatic speaker verification,” in Proc. Odyssey, Les Sables d’Olonne,France, June 2018.