# Generalized Face Anti-Spoofing via Multi-Task Learning
and One-Side Meta Triplet Loss

Chu-Chun Chuang1, Chien-Yi Wang2,and Shang-Hong Lai1,2
1 National Tsing Hua University, Taiwan 2 Microsoft AI R&D Center, Taiwan

###### Abstract

With the increasing variations of face presentation attacks, model
generalization becomes an essential challenge for a practical face anti-
spoofing system. This paper presents a generalized face anti-spoofing
framework that consists of three tasks: depth estimation, face parsing, and
live/spoof classification. With the pixel-wise supervision from the face
parsing and depth estimation tasks, the regularized features can better
distinguish spoof faces. While simulating domain shift with meta-learning
techniques, the proposed one-side triplet loss can further improve the
generalization capability by a large margin. Extensive experiments on four
public datasets demonstrate that the proposed framework and training
strategies are more effective than previous works for model generalization to
unseen domains.

††publicationid: pubid: 979-8-3503-4544-5/23/$31.00 © 2023 IEEE

##  I INTRODUCTION

Owing to the progressive development of face recognition techniques, the
information securities are integral to recognition systems. Thanks to face
anti-spoofing, the probability of accessing the systems with presentation
attacks have strongly reduced.

Over the past few years, face anti-spoofing methods can be roughly divided
into appearance-based and temporal-based methods. However, the performance of
these works significantly degrades while the distributions between testing
data and training data exist significant discrepancies (sensors, environments,
and spoofing mediums). Therefore, domain generalization becomes significant
while dealing with the face anti-spoofing task. Some of the previous works [9,
19, 20, 3, 13, 25] have employed domain generalization techniques to face
anti-spoofing. Adversarial training and meta-learning frameworks were used to
find generalized features among different domains, and the performance showed
verified improvements. The way that meta-learning achieves generalization is
keeping simulating domain shift scenarios during the training process. The
model learns the ability of transfer between domains. In this paper, we
conduct meta-learning process to improve generalization and strengthen the
optimization of inter-domain feature distributions.

Figure 1: Illustrated idea of the proposed face anti-spoofing system.

To find generalized features, employing multiple aspects of face information
may help us achieve this goal. For example, the model would understand the
face composition and depth conditions better with face parsing and depth
information. Based on this idea, multi-task framework may be helpful to
address this thought. In this way, the features our model learned would
include various facial information. With understanding the key information of
faces, we could find more generalized facial features. Moreover, spoof regions
are not always the whole faces. There are many attack types of local regions
such as eyes. Understanding face parsing would also help in this situation.
Thus, here we employ face parsing to strengthen the face understanding
capability and enrich there presentations that we learned. In the past, [10,
11, 15] also employ face parsing to improve the understanding of faces. For
the depth, it is one of the important facial information as well. Pixel-wise
binary masks are employed in many works as supervision for training. [5]
employs binary masks that determines the real region as pixel-wise supervision
for partial attacks. Here we employ binary depth masks as pixel-wise
supervision. Moreover, the depth maps of attacks like print and replay are
flat. Thus, we encourage our model to understand faces much better and to
regularize learning directions by leveraging depth facial features in this
paper.

RFMetaFAS [20] conducts meta-learning process to improve generalized ability
with domain shift simulations. It also employs depth information as the prior
knowledge to regularize the learning direction. Inspired by RFMetaFAS [20] and
our thoughts mentioned above, we propose a multi-task meta-learning framework
that includes depth estimation, face parsing, and spoof classification tasks.
For the face parsing task, we employ an U-net based face parsing module into
the framework, which can serve as pixel-wise supervision and strengthen the
understanding of different facial parts. However, meta-learning focuses on
domain shift simulations which optimizes the inter-domain distributions.
Hence, the distributions of intra-domain may not well constrained. At the same
time, triplet loss can enforce similar data of different label to separate,
and this is crucial due to the distribution of intra-domain is usually close.
Moreover, as the spoof images are more dispersed due to the large variation of
attack mediums and collection environments, the one-side triplet loss only
samples triplet anchors from the features of live images. Hence, we propose to
apply the one-side triplet loss in the meta-optimization procedure in the meta
learner module. The one-side triplet loss encourages the aggregation of live
features and is proven to be very effective for domain generalization in face
anti-spoofing.

The main contributions of our work contain three aspects:

1\. We introduce a multi-task meta-learning framework for learning more
generalized features for face anti-spoofing. The framework and training
techniques demonstrate superior performance over previous approaches in
various domain generalization protocols of face anti-spoofing.

2\. A U-net based face parsing module and the attention-based skip connection
are utilized for encoding facial priors into the network and further promoting
the feature generalization capability.

3\. During the meta-optimization procedure, we propose to use the one-side
triplet loss, which encourages the aggregation of live features and
discriminates the diverse spoof features better in unseen domains.

##  II Related Works

###  II-A Face Anti-spoofing

Due to the different aspects of exploiting face features, existing methods
dealing with face anti-spoofing can be roughly divided into two groups:
temporal-based methods and appearance-based methods.

####  II-A1 Temporal-based Methods

Temporal-based methods aim to exploit the temporal information from multiple
frames of face videos to distinguish live and spoof faces. Recently, several
temporally-based methods have been proposed. CNN-LSTM [27] proposed to extract
temporal features from multiple face frames with a CNN-LSTM architecture for
anti-spoofing. Furthermore, [14] exploited rPPG signals as useful information
in face videos to distinguish spoof attacks from live faces.

####  II-A2 Appearance-based Methods

Comparatively, appearance-based methods aim to differentiate live and spoof
faces from spatial information in images due to the different textures and
features between live and spoof images. In recent years, CNN based methods
[28] have been employed for face anti-spoofing. Two-stream CNN-based method
[1] combines patch-based CNN and depth-based CNN models for face anti-
spoofing.

However, [1] learns the fused face representations via finding patch and depth
features separately, whereas our work directly leverages face parsing and
depth tasks to regularize the learning direction. The face representations
extracted in our work contains both face parsing and depth information with
one common feature extractor, and thus to obtain more generalized features.

Regardless of the tempotal-based or appearance-based methods mentioned above,
the prior works are poor to generalize to unseen domains since data
distributions are different from training data. Thus, several works including
this paper exploit the concepts of domain generalization for face anti-
spoofing. The next subsection we would introduce these works.

###  II-B Domain Generalization for Face Anti-Spoofing

Domain generalization(DG) aims to learn from several source domains and then
test on the unseen target domain. Recently, domain generalization is an
important challenge for face anti-spoofing as we have mentioned in
introduction. Several learning methods such as adversarial learning and meta-
learning are adopted to improve generalization. MADDG [19] uses a multi-
adversarial discriminative deep domain generalization framework to learn
generalized features from multiple domains. SSDG [9] proposes single-side
domain generalization for face anti-spoofing with single-side adversarial
learning and an asymmetric triplet loss. RFMetaFAS [20] learns generalized
features via meta-learning and employs depth information as prior knowledge.
We deal with domain generalization of face anti-spoofing with also a meta-
learning-based method. Different from others, our work employs multi-task
framework to learn more complete and generalized facial features, and
optimizes distributions with meta-learning and one-side triplet loss for
inter-domain and intra-domain, separately.

##  III Proposed Method

Our work aims to learn more robust features and improve the generalization
capability of the face anti-spoofing model. Meta-learning is also effective
for domain generalization due to domain shifting simulations. Here we propose
a multi-task framework with one-side triplet loss during meta-optimization.

In the following section, we first introduce the proposed multi-task
framework, followed by the components of the proposed framework: multi-task
meta-learning, U-net based face parsing module, one-side triplet loss, and
detailed objective functions.

###  III-A Overview

Fig. 2 illustrates the overall framework. The proposed method is a multi-task
meta-learning based approach, which contains a feature extractor, a meta
learner for spoof classification, a depth estimator branch for depth
estimation, and a U-net based face parsing module for semantic segmentation on
faces.

Figure 2: Our framework contains depth estimation, face parsing, and meta-
learned spoof classification branches. Here we denote the feature extractor as
$F$, U-net based face parsing module as $S$, depth estimator as $D$, and meta
learner as $M$. Source images are divided into meta-train set and meta-test
set in each iteration. The depth estimator $D$ is to predict depth maps, which
serves as prior knowledge to regularize the model. U-net based face parsing
module $S$ aims to improve the generalization by encoding the semantic facial
priors with the attention-based skip-connection. In the end, features are fed
into the meta learner $M$ to conduct meta-optimization, which involves the
one-side triplet and classification losses.

We assume each training data contains a face image _x_ associated with the
corresponding label _y_ , face depth image _d_ , and face parsing image _s_.
For the network, we denote the feature extractor as $F$, U-net based face
parsing module as $S$, depth estimator as $D$, and meta learner as $M$. In the
network, the face image _x_ is fed into feature extractor $F$ to extract a
feature vector _f_. After the feature extraction, a depth estimator $D$
estimates the corresponding depth map _d_ from the feature vector _f_.
Simultaneously, the feature _f_ is fed into a U-net based face parsing module
$S$, which helps the model learn better representations to obtain and
strengthen the information of face parsing. Lastly, the aggregated feature
vector, which is concatenated from feature _f_ and face parsing feature, is
fed into the meta learner $M$ for live/spoof classification. Besides, we apply
one-side triplet loss in meta learner to improve the data distribution in
feature space. We explain the details in the coming sections.

###  III-B Multi-Task Meta Learning

We adopt the fine-grained learning strategy in RFMetaFAS [20] that we randomly
choose one of the source domain as meta-test domain and the others are meta-
train domains during each iteration. Here we assume N source domains in total,
including N-1 meta-train domains and a meta-test domain. We assume $D_{mtrn}$
and $D_{mtst}$ as meta-train domains and meta-test domain, respectively. The
meta-learning process contains meta-train, meta-test, and meta-optimization
stages. The meta learner conducts meta-learning by exploiting a variety of
domain shift scenarios in each iteration. Fig. 3 illustrates the detailed
gradient flows of our meta-learning process.

Figure 3: Illustration of the optimization process in meta learner. Given
$D_{1}$, $D_{2}$ as meta-train domains, and $D_{3}$ as meta-test domain.
$L_{mtrn}$, $L_{mtst}$, $L_{Cls}$ and $L_{Trip}$ denotes loss of meta-train,
meta-test, spoof classification, and one-side triplet loss.
$\theta_{1}^{\prime}$ and $\theta_{2}^{\prime}$ are found by $D_{1}$ and
$D_{2}$, and $D_{3}$ is applied for finding learning directions by both
$\theta_{1}^{\prime}$ and $\theta_{2}^{\prime}$. Every domain division
contains a gradient and meta-gradient from meta-train and meta-test domains.

####  III-B1 Meta-train

For meta-train stage, data of each meta-train domain is sampled for batches.
Here we exploit classification loss and one-side triplet loss to find learning
directions. During the meta-train stage, the parameters of meta learner are
first found by
$\theta_{M_{i}^{\prime}}=\theta_{M}-\alpha((\nabla_{\theta_{M}}L_{Cls(D_{mtrn}(i))}(\theta_{F},\theta_{M}))+(\nabla_{\theta_{M}}L_{Trip(D_{mtrn}(i))}(\theta_{F},\theta_{M})))$
for each meta-train domain, where $\theta_{F}$ and $\theta_{M}$ denote the
parameters of feature extractor and meta learner, $i\in{D_{mtrn}}$, $L_{Cls}$
and $L_{Trip}$ are the classification loss and one-side triplet loss,
respectively. Besides, $L_{Dep}$ is adopted to regularize the feature
extractor as well. For live faces, depth image is supposed to be a face-shape
like depth maps, while depth images of spoof faces are all set to zero.
Moreover, $L_{Seg}$ is adopted to learn face parsing information and find more
generalized representations.

####  III-B2 Meta-test

The generalization capability could be further enhanced with meta-learning by
domain shifting simulations, and here meta-test domain acts as a role of the
unseen domain. Thus, our model should perform well on the meta-test domain
based on learning directions found by the meta-train domain. Parameters of the
meta learner here are denoted as $\theta_{M_{i}^{\prime}}$.

####  III-B3 Meta-Optimization

In meta-optimization, we optimize our model, which includes $\theta_{F}$,
$\theta_{M}$, $\theta_{S}$, and $\theta_{D}$, with the sum of meta-train and
meta-test losses.

###  III-C U-net Based Face Parsing Module

U-net based face parsing module aims to learn semantic facial priors and
discriminate the spoof images with the attention to different facial parts.
Fig. 4 shows the architecture of U-net face parsing module. Our face parsing
module contains two parts: a face parsing U-net and an attention-based skip
connection.

####  III-C1 Face Parsing U-net

U-net is a commonly used architecture for dealing with semantic segmentation
tasks. We employ a U-net based architecture that includes an encoder-decoder
structure. Both encoder and decoder contain three convolutional blocks, and
skip connections are added to aggregate segmented information between them.
The face parsing U-net aims to predict face parsing images _s_. The input of
the face parsing module is the face image _x_ , which is the same as the
feature extractor. Specifically, since we want face parsing information to
regulate the feature extractor, we set the encoder and feature extractor with
shared weights. After we obtain the feature _f_ , the decoder outputs the
13-dimension face parsing binary maps.

####  III-C2 Attention-Based Skip Connection

We apply the attention-based skip connection to encode semantic facial priors
into the main branch of the network. Here we take the last stage of the face
parsing decoder as the input for the attention-based module. Due to the reason
that different channels of the face parsing feature may exist specific
connections between each other, here we apply a channel attention module ECA-
Net [24] to learn inter-channel relations of the feature in the skip
connection branch. Finally, the output feature is concatenated with feature
_f_ for feeding into the meta learner.

Figure 4: The architecture of U-net based face parsing module. Our face
parsing module contains two parts: face parsing U-net and attention-based skip
connection. The encoder of the U-net includes three convolutional blocks,
which share weights with the feature extractor. Features after the encoder are
fed into the decoder and upsampled three times, and there are skip connections
to the corresponding convolutional block of the encoder. Apart from the output
of 13 dimensions segmentation maps, there is another path called attention-
based skip-connection, which contains a convolutional block and an ECA module
to transfer important face parsing information to the main network.

###  III-D One-Side triplet loss

Here we propose one-side triplet loss to combine the inter-domain advantage of
meta learning and intra-domain one of triplet loss. First, triplet loss is
suitable for face anti-spoofing due to aggregations of the same class and
separations of different classes. In the meantime, meta learning aims to
simulate domain shift and thus improves the generalization capability in
unseen domains. However, meta learning mainly focuses on inter-domain
distribution. Hence, we propose one-side triplet loss for both meta-train and
meta-test domains. Besides, since meta learning emphasizes domain shifting
directions for unseen domains, inter-domain triplet loss is redundant. Second,
different from the normal triplet loss, one-side triplet only aggregates live
domain data due to the reason below. Since there are countless kinds of
attacks due to variations such as cameras, illuminations, etc., we believe
that spoof data are more widely distributed than live ones. Moreover, since
spoof domains could be kept dividing into smaller spoof subdomains, we believe
distributions of different spoof datasets should be partially overlapped
instead of completely independent to others. Thus, it is not necessary to
separate different spoof domains. Separating live and spoof domains is a
critical task. Therefore, different from SSDG [9], we propose one-side triplet
loss, which only applies live domain for anchor and applies both live and
spoof domains for positive and negative samples.

Due to the spoof domains are widely distributed, it is prone to pick extreme
data while processing the one-side triplet loss. To avoid interference, the
triplet loss mining strategy becomes pretty crucial. Hence, we apply two-stage
mining triplet loss. A smaller margin is chosen at first and applies batch all
mining strategy, optimizing all of the valid triplets. Then we increase the
margin with the batch hard mining strategy, which optimizes only the hardest
triplet. By doing this, we could get more stable results. Equation (III-E2)
and (III-E2) show one-side triplet loss $L_{triplet}$ in meta-train and meta-
test.

###  III-E Objective Function

Our network is trained with four different objective functions, i.e., the
classification loss, one-side triplet loss, segmentation loss, and depth loss.
The four objective functions will be described in detail in this subsection.
Here we assume $J$ denotes the domains belonging to $D_{mtrn}$,
$J_{i}(i=1,2,...,N-1)$ and ${K}$ denotes the domain belonging to $D_{mtst}$.

####  III-E1 Classification Loss

We adopt binary cross-entropy classification loss to both meta-train and meta-
test domains for classification. Equation (III-E1) and (III-E1) give the
classification losses during each meta-train and meta-test domain,
respectively.

$\displaystyle L_{Cls(J_{i})}(\theta_{F},\theta_{M})=$
$\displaystyle\sum_{(x,y)\sim{J_{i}}}(y\log{M(F(x))}$ $\displaystyle+$
$\displaystyle(1-y)\log{(1-M(F(x)))})$ (1)
$\displaystyle\sum\limits_{i=1}^{N-1}L_{Cls({K})}(\theta_{F},\theta_{M_{i}^{\prime}})=$
$\displaystyle\sum\limits_{i=1}^{N-1}\sum\limits_{(x,y)\sim{K}}(y\log{M_{i}^{\prime}(F(x))}$
$\displaystyle+$ $\displaystyle(1-y)\log{(1-M_{i}^{\prime}(F(x)))})$ (2)

where $\theta_{F}$, $\theta_{M}$, and $\theta_{M_{i}^{\prime}}$ denote the
parameters of feature extractor, meta learner, and meta learner after update
from $i$-th domain in meta-train. $x$ and $y$ are the input and the
corresponding label.

####  III-E2 One-Side Triplet Loss

In order to combine with meta learning, we adopt one-side triplet loss into
our method. The detailed equations of one-side triplet loss for each meta-
train and meta-test domain are given in (III-E2) and (III-E2), respectively.

$\displaystyle L_{Trip(J_{i})}(\theta_{F},\theta_{M})=$
$\displaystyle\sum_{(x_{i}^{a},x_{i}^{p},x_{i}^{n})}(\|{M(F(x_{i}^{a}))}-{M(F(x_{i}^{p}))}\|_{2}^{2}$
$\displaystyle-$
$\displaystyle\|M(F(x_{i}^{a}))-{M(F(x_{i}^{n}))}\|_{2}^{2}+\alpha)$ (3)
$\displaystyle\sum$
${}_{i=1}^{N-1}L_{Trip({K})}(\theta_{F},\theta_{M_{i}^{\prime}})=\sum\limits_{i=1}^{N-1}\sum\limits_{(x_{t}^{a},x_{t}^{p},x_{t}^{n})}(\|{M_{i}^{\prime}(F(x_{t}^{a}))}$
$\displaystyle-$
$\displaystyle{M_{i}^{\prime}(F(x_{t}^{p}))}\|_{2}^{2}-\|M_{i}^{\prime}(F(x_{t}^{a}))-{M_{i}^{\prime}(F(x_{t}^{n}))}\|_{2}^{2}+\alpha)$
(4)

where $x_{i}\sim{J}_{i}$ and $x_{t}\sim{K}$; $x^{a}$, $x^{p}$, $x^{n}$ denote
anchor, positive, negative sample, respectively, and $\alpha$ is a margin.

####  III-E3 Segmentation Loss

Since we expect the U-net based face parsing module to predict the face
parsing masks corresponding to the source face image, here we adopt the multi-
class cross-entropy loss. The segmentation masks are 13-dimensional binary
images, whose pseudo ground truth is pre-computed by [29]. The segmentation
loss is given by (III-E3) as follows:

$\displaystyle L_{Seg{J_{i}}}(\theta_{F},\theta_{S})=$ $\displaystyle CE(x,Y)$
$\displaystyle=$
$\displaystyle-\sum_{(x,Y)\sim{J_{i}}}\sum_{C}y\log{S(F(x_{j,k}))}$ (5)

where $\theta_{S}$ denotes the parameters in the face parsing module, $Y$
denotes the ground truth of face parsing masks corresponding to the input, $C$
is the channel of face parsing masks, and $x_{j,k}$ denotes a pixel in our
predicted output. $b$ is either a meta-train or meta-test domain depends on
the meta-train or meta-test stage.

####  III-E4 Depth Loss

Similar to RFMetaFAS [20], in order to exploit depth image as domain knowledge
to regularize the feature extractor, we apply the L2 loss between the
prediction and the pseudo depth ground truth obtained from [7]. It is given by
(6).

$\displaystyle
L_{Dep_{b}}(\theta_{F},\theta_{D})=\sum_{(x,I)\sim{b}}\|{D(F(x))-I}\|^{2}$ (6)

where $\theta_{D}$ consists of the parameters in the depth estimator, and $I$
denotes the pre-computed depth image.

####  III-E5 Overall Loss

Finally, the overall objective function is given as follows:

$\displaystyle L$ ${}_{all}=\lambda_{mtrn}L_{J}+\lambda_{mtst}L_{{K}}$
$\displaystyle=$
$\displaystyle\lambda_{mtrn}(\lambda_{Cls}L_{Cls(J)}+\lambda_{Dep}L_{Dep(J)}+\lambda_{Seg}L_{Seg(J)}+\lambda_{Trip}L_{Trip(J)})$
$\displaystyle+$
$\displaystyle\lambda_{mtst}(\lambda_{Cls}L_{Cls({K})}+\lambda_{Dep}L_{Dep({K})}+\lambda_{Seg}L_{Seg({K})}+\lambda_{Trip}L_{Trip({K})})$
(7)

where
$\lambda_{mtrn},\lambda_{mtst},\lambda_{Cls},\lambda_{Dep},\lambda_{Seg},\lambda_{Trip}$
are pre-defined hyperparameters. Here we set $\lambda_{mtrn}=1$,
$\lambda_{mtst}=1$, $\lambda_{Cls}=1$, $\lambda_{Dep}=10$, $\lambda_{Seg}=1$,
$\lambda_{Trip}=0.5$ in every experiment. To balance the contributions of
meta-train and meta-test set, we set $\lambda_{mtrn}=\lambda_{mtst}$.
Moreover, the values of $\lambda_{Cls}$, $\lambda_{Dep}$, $\lambda_{Seg}$, and
$\lambda_{Trip}$ are modified to balance $L_{Cls}$, $L_{Dep}$, $L_{Seg}$, and
$L_{Trip}$ because we find that the results are more stable while they are
balanced as we test.

TABLE I: Experimental Comparisons of different face anti-spoofing methods on the four domain generalization experiments. Method | O&C&I to M | O&M&I to C | O&C&M to I | I&C&M to O
---|---|---|---|---
| HTER(%) | AUC(%) | HTER(%) | AUC(%) | HTER(%) | AUC(%) | HTER(%) | AUC(%)
MADDG[19] | 17.69 | 88.06 | 24.50 | 84.51 | 22.19 | 84.99 | 27.98 | 80.02
RFMetaFAS[20] | 13.89 | 93.98 | 20.27 | 88.16 | 17.30 | 90.48 | 16.45 | 91.16
CCDD[17] | 15.42 | 91.13 | 17.41 | 90.12 | 15.87 | 91.47 | 14.72 | 93.08
PAD-GAN[23] | 17.02 | 90.10 | 19.68 | 87.43 | 20.87 | 86.72 | 25.02 | 81.47
NAS-FAS[30] | 16.85 | 90.42 | 15.21 | 92.64 | 11.63 | 96.98 | 13.16 | 94.18
SSDG-M[9] | 16.67 | 90.47 | 23.11 | 85.45 | 18.21 | 94.61 | 25.17 | 81.83
MT-FAS[16] | 11.67 | 93.09 | 18.44 | 89.67 | 11.93 | 94.95 | 16.23 | 91.18
D2AM[3] | 12.70 | 95.66 | 20.98 | 85.58 | 15.43 | 91.22 | 15.27 | 90.87
ANRL[13] | 10.83 | 96.75 | 17.85 | 89.26 | 16.03 | 91.04 | 15.76 | 91.90
FGHV[12] | 9.17 | 96.92 | 12.47 | 93.47 | 16.29 | 90.11 | 13.58 | 93.55
SSAN-M[25] | 10.42 | 94.76 | 16.47 | 90.81 | 14.00 | 94.58 | 19.51 | 88.17
Ours | 7.38 | 96.66 | 13.2 | 94.27 | 8.07 | 96.85 | 8.75 | 95.95

##  IV Experimental Results

###  IV-A Experimental Setting

####  IV-A1 Datasets

Here we conduct domain generalization experiments with four public datasets:
Oulu-NPU [2](denoted as O), Idiap Replay-Attack [4](denoted as I), CASIA-FASD
[31](denoted as C), and MSU-MFSD [26](denoted as M). These four datasets
contain large domain shifts which include variations in illumination,
background, imaging devices, and attack types.

The experiment setting follows the work in [20]. We take one of the four
source datasets as the unseen testing domain and the other three datasets as
training domains in each experiment. Therefore, we conduct four experiments:
O&C&I to M, O&M&I to C, O&C&M to I, I&C&M to O.

####  IV-A2 Implementation Details

Our network is implemented with PyTorch. Adam is adopted as the optimizer with
momentum of 0.9, weight decay is set to 5e-5, learning rate is 1e-3, and batch
size is set to 20 for each domain. For the hyperparameters in the objective
functions, we set $\lambda_{1}=1$, $\lambda_{2}=1$, $\lambda_{3}=1$,
$\lambda_{4}=10$, $\lambda_{5}=1$, $\lambda_{6}=0.5$. We concatenate RGB and
HSV images as input to our network, which is resized to $256\times 256\times
6$. The detailed network structure is given in the supplemental material.

####  IV-A3 Evaluation Metrics

Here we adopt Half Total Error Rate (HTER) and Area Under Curve (AUC) as
evaluation metrics. For visualization, we exploit t-SNE [21] and Grad-CAM [18]
to show the effect of our approach.

###  IV-B Experimental Comparisons

In the following section, we compare performance of our approach with several
state-of-the-art face anti-spoofing methods, including MADDG [19], and
RFMetaFAS [20], CCDD [17], PAD-GAN [23], NAS-FAS [30], SSDG-M [9], MT-FAS
[16], D2AM [3], ANRL [13], FGHV [12], and SSAN-M [25]. Table I demonstrates
significant improvements by using the proposed method compared with the
baseline RFMetaFAS [20] and other state-of-the-art methods.

Note that we use the architecture of MADDG [19] as our backbone, which is the
same as SSDG-M [9]. However, SSDG-R [9] and PatchNet [22] adopts ResNet-18 [8]
as the backbone, and LMFD-PAD [6] adopts ResNet-50 as backbone. Thus, they are
not included in the experimental comparison because its model size is much
larger than MADDG.

TABLE II: Ablation study of using different components in our method. The upper part is about using face parsing information, and the lower part is about one-side triplet loss. For face parsing information, we compare the results of our method with and without using the attention-based skip connection (ASC) and U-net based face parsing module (parsing). For one-side triplet loss, we compare the results of our method with and without one-side triplet loss, without meta learning, and with normal triplet loss. Method | O&C&I to M | O&M&I to C | O&C&M to I | I&C&M to O
---|---|---|---|---
| HTER(%) | AUC(%) | HTER(%) | AUC(%) | HTER(%) | AUC(%) | HTER(%) | AUC(%)
Ours w/o parsing | 10.17 | 94.05 | 20.43 | 88.64 | 13.12 | 94.49 | 17.26 | 90.95
Ours w/o ASC | 9.31 | 94.30 | 26.68 | 83.72 | 9.13 | 96.46 | 15.01 | 93.06
Ours w/o one-side trip. | 9.25 | 95.72 | 25.03 | 83.97 | 12.30 | 95.08 | 13.79 | 93.51
Ours w/o meta | 10.94 | 95.30 | 31.05 | 75.78 | 27.79 | 79.26 | 29.24 | 77.70
Ours w/ normal trip. | 9.94 | 95.43 | 22.54 | 87.17 | 10.68 | 95.59 | 14.42 | 92.78
Ours | 7.38 | 96.66 | 13.20 | 94.27 | 8.07 | 96.85 | 8.75 | 95.95

###  IV-C Face Parsing Results

We depict visualization of the output of U-net based face parsing module in
Fig. 5. 13 labels are chosen for the face parsing: skin, left/right brow,
left/right eye, eyeglasses, left/right ear, nose, mouth, upper/lower lip, and
background. The pixel-wise output of semantic facial priors is well performed
both for real and spoof images.

###  IV-D Ablation Study

####  IV-D1 U-net Based Face Parsing Module

To understand the effect of U-net based face module, here we compare our
approach with two different experiment settings: our approach without the
U-net face parsing module and the attention-based skip connection. For the
frontal one, we want to realize the effect of face parsing for the face anti-
spoofing task. For the latter one, we want to discuss whether the face parsing
information should be encoded and added back to the main network by the
attention-based connection. Table II shows the experimental results of the
three settings. It shows that the U-net based face parsing module is effective
for regularizing the feature, and the attention-based skip connection(ASC)
indeed helps to generalize better in the unseen domains for face anti-
spoofing.

Figure 5: Segmentation output from the face parsing module. From top to
bottom: input image, parsing output, and ground truth. The left three columns
are live samples, and the right three columns are spoof ones.

####  IV-D2 One-Side Triplet Loss with Meta-learning

In the following section, we perform an experiment to show the effect of the
combination of one-side triplet loss and meta-learning. Table II shows results
of our approach with and without one-side triplet loss, without meta-learning,
and with normal triplet loss instead of one-side triplet loss. It is worth
noting that HTER and AUC are improved by over 10% in O&M&I to C experiment.
Without one-side triplet loss or meta-learning, the AUCs of O&M&I to C degrade
dramatically. In our opinion, O&M&I to C is the harder experiment because
there is no cut attack in source domains, but target domain exists cut attack.
Thus, the performance improves with one-side triplet loss or meta-learning
means that out model performs much generalized. For O&C&I to M, the
performance does not decline obviously due to the data of M is comparatively
easy. In summary, removing meta-learning from our method and changing one-side
triplet loss into ordinary triplet loss cause large degradation of the
generalization performance in most protocols.

Figure 6: Visualization of Grad-CAM. From top to bottom: input image, our
approach, and ours without U-net based face parsing module. The left four
columns are live images, and the remaining columns are spoof ones. Figure 7:
Visualization results of t-SNE based on our approach. From left to right: our
approach without triplet loss, ours with normal triplet loss, and our
approach. Src means the source domains, and tgt means the target domain.

###  IV-E Visualization

####  IV-E1 Grad-CAM Visualization

In addition to the above experimental results, we depict visualization of some
examples with Grad-CAM[18], which can localize the important regions that the
network concentrated on according to a specific class.

Fig. 6 shows the results of the two settings: our approach with and without
U-net based face parsing module. The three rows are input images, ours, and
ours without U-net based face parsing module. The first four columns are live
samples, and the others are spoofed ones. We encourage our model to learn
discriminative cues from the facial region instead of background because the
background contain large variations for different datasets. As Fig. 6 shows,
our network tends to focus on facial parts for both live and spoof images.

####  IV-E2 t-SNE Visualization

t-SNE is adopted for visualizing the effect of one-side triplet loss. Fig. 7
shows three settings from left to right: our approach without triplet loss,
ours with normal triplet loss, and ours with one-side triplet loss. Here we
apply the margin of normal triplet loss as 0.1, which is the same as our
beginning setting. As shown in Fig. 7, target domain samples are well
discriminated against those with one-side triplet loss. To achieve
generalization, we do not separate different domains and thus reduce the
domain gaps.

Figure 8: Segmentation output from the face parsing module. Four rows are our
approach without attention-based skip connection, our approach, ground truth,
and the input image.

####  IV-E3 Effect of Attention-Based Skip Connection for Face Parsing

We visualize the segmentation output from the face parsing module. Here we
mainly focus on comparing our approach to ours without attention-based skip
connection. As Fig. 8 shown, adding attention-based skip connection to our
approach makes the face parsing results more stable and refined.

##  V Conclusion

In this paper, we propose a multi-task framework based on meta learning to
improve the model generalization capability for face anti-spoofing. For the
face parsing task, a U-net based face parsing module is proposed to learn
important face parsing information. For the spoof classification task, a one-
side triplet loss is employed to combine the advantages of meta learning and
triplet loss. Our experiments on cross-domain generalization for face anti-
spoofing demonstrate that our method provides superior performance compared to
the state-of-the-art methods on public datasets.

## References

  * [1] Y. Atoum, Y. Liu, A. Jourabloo, and X. Liu.  Face anti-spoofing using patch and depth-based cnns.  In In Proceeding of International Joint Conference on Biometrics, 2017.
  * [2] Z. Boulkenafet, J. Komulainen, L. Li, X. Feng, and A. Hadid.  Oulu-npu: A mobile face presentation attack database with real-world variations.  In 12th IEEE International Conference on Automatic Face Gesture Recognition (FG), 2017.
  * [3] Z. Chen, T. Yao, K. Sheng, S. Ding, Y. Tai, J. Li, F. Huang, and X. Jin.  Generalizable representation learning for mixture domain face anti-spoofing.  In AAAI, 2021.
  * [4] I. Chingovska, A. Anjos, and S. Marcel.  On the effectiveness of local binary patterns in face anti-spoofing.  In 2012 BIOSIG - Proceedings of the International Conference of Biometrics Special Interest Group (BIOSIG), 2012.
  * [5] M. Fang, F. Boutros, A. Kuijper, and N. Damer.  Partial attack supervision and regional weighted inference for masked face presentation attack detection.  2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021), pages 1–8, 2021.
  * [6] M. Fang, N. Damer, F. Kirchbuchner, and A. Kuijper.  Learnable multi-level frequency decomposition and hierarchical attention mechanism for generalized face presentation attack detection.  In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), pages 3722–3731, January 2022.
  * [7] Y. Feng, F. Wu, X. Shao, Y. Wang, and X. Zhou.  Joint 3d face reconstruction and dense alignment with position map regression network.  In Proceedings of the European Conference on Computer Vision (ECCV), 2018.
  * [8] K. He, X. Zhang, S. Ren, and J. Sun.  Deep residual learning for image recognition.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016.
  * [9] Y. Jia, J. Zhang, S. Shan, and X. Chen.  Single-side domain generalization for face anti-spoofing.  In Proc. IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [10] M. M. Kalayeh, B. Gong, and M. Shah.  Improving facial attribute prediction using semantic segmentation, 2017\.
  * [11] K. Khan, M. Attique, R. Khan, I. Syed, and T.-S. Chung.  A multi-task framework for facial attributes classification through end-to-end face parsing and deep convolutional neural networks.  Sensors, 20, 01 2020.
  * [12] S. Liu, S. Lu, H. Xu, J. Yang, S. Ding, and L. Ma.  Feature generation and hypothesis verification for reliable face anti-spoofing.  In AAAI, 2022.
  * [13] S. Liu, K.-Y. Zhang, T. Yao, M. Bi, S. Ding, J. Li, F. Huang, and L. Ma.  Adaptive normalized representation learning for generalizable face anti-spoofing.  In Proceedings of International Conference on Multimedia. Association for Computing Machinery, 2021.
  * [14] Y. Liu, A. Jourabloo, and X. Liu.  Learning deep models for face anti-spoofing: Binary or auxiliary supervision.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
  * [15] Z. Lu, T. Hu, L. Song, Z. Zhang, and R. He.  Conditional expression synthesis with face parsing transformation.  In Proceedings of the 26th ACM International Conference on Multimedia, page 1083–1091. Association for Computing Machinery, 2018.
  * [16] Y. Qin, Z. Yu, L. Yan, Z.Wang, C. Zhao, and Z. Lei.  Meta-teacher for face anti-spoofing.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021\.
  * [17] S. Saha, W. Xu, M. Kanakis, S. Georgoulis, Y. Chen, D. P. Paudel, and L. Van Gool.  Domain agnostic feature learning for image and video based face anti-spoofing.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2020.
  * [18] R. R. Selvaraju, M. Cogswell, A. Das, R. Vedantam, D. Parikh, and D. Batra.  Grad-cam: Visual explanations from deep networks via gradient-based localization.  In 2017 IEEE International Conference on Computer Vision (ICCV), 2017.
  * [19] R. Shao, X. Lan, J. Li, and P. C. Yuen.  Multi-adversarial discriminative deep domain generalization for face presentation attack detection.  In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2019.
  * [20] R. Shao, X. Lan, and P. C. Yuen.  Regularized fine-grained meta face anti-spoofing.  In Thirty-Fourth AAAI Conference on Artificial Intelligence (AAAI), 2020.
  * [21] L. van der Maaten and G. Hinton.  Visualizing data using t-sne.  Journal of Machine Learning Research, 2008.
  * [22] C.-Y. Wang, Y.-D. Lu, S.-T. Yang, and S.-H. Lai.  Patchnet: A simple face anti-spoofing framework via fine-grained patch recognition.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 20281–20290, June 2022.
  * [23] G. Wang, H. Han, S. Shan, and X. Chen.  Cross-domain face presentation attack detection via multi-domain disentangled representation learning.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [24] Q. Wang, B. Wu, P. Zhu, P. Li, W. Zuo, and Q. Hu.  Eca-net: Efficient channel attention for deep convolutional neural networks.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [25] Z. Wang, Z. Wang, Z. Yu, W. Deng, J. Li, and S. Li.  Domain generalization via shuffled style assembly for face antispoofing.  In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2022.
  * [26] D. Wen, H. Han, and A. K. Jain.  Face spoof detection with image distortion analysis.  IEEE Transactions on Information Forensics and Security, 2015.
  * [27] Z. Xu, S. Li, and W. Deng.  Learning temporal features using lstm-cnn architecture for face anti-spoofing.  2015 3rd IAPR Asian Conference on Pattern Recognition (ACPR), 2015\.
  * [28] J. Yang, Z. Lei, and S. Z. Li.  Learn convolutional neural network for face anti-spoofing, 2014.
  * [29] C. Yu, J. Wang, C. Peng, C. Gao, G. Yu, and N. Sang.  Bisenet: Bilateral segmentation network for real-time semantic segmentation.  In Proceedings of the European conference on computer vision (ECCV), 2018.
  * [30] Z. Yu, J. Wan, Y. Qin, X. Li, S. Z. Li, and G. Zhao.  Nas-fas: Static-dynamic central difference network search for face anti-spoofing.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020\.
  * [31] Z. Zhang, J. Yan, S. Liu, Z. Lei, D. Yi, and S. Z. Li.  A face antispoofing database with diverse attacks.  In 2012 5th IAPR International Conference on Biometrics (ICB), 2012\.


# Punctuation Restoration for Singaporean Spoken Languages: English, Malay,
and Mandarin

Abhinav Rao
BITS Pilani, Hyderabad Campus
<EMAIL_ADDRESS>Ho Thi-Nga
Nanyang Technological University
<EMAIL_ADDRESS>Chng Eng-Siong
Nanyang Technological University
<EMAIL_ADDRESS>
###### Abstract

This paper presents the work of restoring punctuation for ASR transcripts
generated by multilingual ASR systems. The focus languages are English,
Mandarin, and Malay which are three of the most popular languages in
Singapore. To the best of our knowledge, this is the first system that can
tackle punctuation restoration for these three languages simultaneously.
Traditional approaches usually treat the task as a sequential labeling task,
however, this work adopts a slot-filling approach that predicts the presence
and type of punctuation marks at each word boundary. The approach is similar
to the Masked-Language Model approach employed during the pre-training stages
of BERT, but instead of predicting the masked word, our model predicts masked
punctuation. Additionally, we find that using
Jieba111https://github.com/fxsjy/jieba instead of only using the built-in
SentencePiece tokenizer of XLM-R can significantly improve the performance of
punctuating Mandarin transcripts.

Experimental results on English and Mandarin IWSLT2022 datasets and Malay News
show that the proposed approach achieved state-of-the-art results for Mandarin
with 73.8% F1-score while maintaining a reasonable F1-score for English and
Malay, i.e. 74.7% and 78% respectively. Our source code that allows
reproducing the results and building a simple web-based application for
demonstration purposes is available on
Github222https://github.com/AetherPrior/Multilingual-Sentence-Boundary-
detection.

##  I Introduction

Automatic punctuation restoration is a problem of restoring punctuation
symbols on unpunctuated text. One of its common inputs is automatic
transcripts generated by Automatic Speech Recognition (ASR) systems. This is
because most current ASR systems still yield transcripts with no punctuation,
which poses a challenge for human reading [1] and also downstream text
processing applications. Performance of text processing applications such as
machine translation [2], and sentence dependency parsing [3], has been shown
to decrease when processed on unpunctuated text. Thus, restoring punctuation
is crucial for not only improving the human readability of ASR-generated
transcripts but also opening up their usability in many text processing
applications.

There are two broad groups of input features that are used in the research on
the task of punctuation restoration for ASR transcripts. One is prosody-based
features that are extracted from speech output. For example, [4] uses pause
duration and pitch with a Multi-Layer Perceptron (MLP) to predict punctuation
marks after each word, while [5] supplements prosodic features with POS-tags
for every word in the input sequence. [6] supplement their texts with pause
durations, after aligning their audio data with text, and use an LSTM to
punctuate texts. [7] use an acoustic feature extractor, supplemented by BERT
for lexical feature extraction. However, there are disadvantages of using
prosody-based features in practical use, owing to the variations in tone per
person and the non-availability of speech audio for different transcripts. The
other is lexical features which are extracted purely from text sources. For
example, [8] uses a Hidden-Markov-Model approach, [9] uses the superior
Conditional Random Field for punctuation restoration, and [10] uses word-
vectors for embeddings. Punctuation restorers using only lexical features face
a limitation of not utilizing audio information but are generally more
straightforward and simple to implement.

Over the past few years, there has been increasing research in multilingual
language models (MLM), with the advent of state-of-the-art models that can
produce multilingual embeddings and can be fine-tuned for downstream tasks.
Multilingual-BERT, a variation of BERT [11] developed by Google, and XLM-
RoBERTa [12] developed by Facebook are two such models that capture the
vocabularies of over a hundred languages. These large transformer models are
shipped with pre-trained weights and only require fine-tuning for other tasks
such as Question-Answering, Named-Entity Recognition, and Natural Language
Inference (NLI) with minimal modifications to the architecture.

In this work, we treat the punctuation prediction task as a slot-filling task,
where, similar to the MLM objective, we consider the masking of punctuation
marks as a fine-tuning step to predict punctuation. To be capable of
performing the task in a multilingual setting, we use an XLM-RoBERTa (XLM-R)
base model as the backbone pre-trained layer, and a classifier that outputs
the punctuation marks to be filled in each masked position. XLM-R has shown to
be performed well on Cross-Lingual tasks such as Cross-Lingual Natural
Language Inference [13], Question-Answering [14], and Information Retrieval
[15]. Additionally, we modify the tokenization process and improve the model
performance for the Mandarin test set. We evaluate this architecture for three
languages, i.e. English, Mandarin, and Malay, that are popular in Singapore.

The rest of this paper is presented as follows. Section II discusses the
related work. Our proposed approach is described in Section III with details
about our model architecture and our analysis of the selected datasets.
Experiments and results are presented in Section V. We interpret our findings
in Section VI. Section VII concludes our work.

##  II Background

Punctuation restoration using Deep Learning has had its fair share of
development over the past few years. Tilk et al. [6], [16] used an LSTM-based
approach for restoring punctuation using sequence tagging, with [17] using a
similar architecture for applying punctuation restoration to medical reports.
Upon the advent of transformers [18], Makhija et al. [19] use a BERT [11]
model as an embedding generator and an LSTM+CRF for punctuation restoration.
Yi et al. [20] supplement the punctuation prediction task with POS tagging for
adversarial training and [21] employ a teacher-student model for punctuation
prediction, similar to knowledge distillation. Zhang et al. [22] focus on
Mandarin, capturing Mandarin word-boundary information using a CNN, while
labelling the punctuation marks using BERT-Chinese combined with a sequence
model. While all of the above models treat punctuation restoration as a
sequence modeling task, SAPR [23] treats the problem as a machine translation
task, and uses a transformer-based model for generating embeddings for the
model.

Work has also been developed towards multilingual approaches: Alam et al. [24]
train multiple models for English and a lower-resource Indian language,
Bangla, and A. Nagy. et al. [25] train for English and Hungarian, using
English-BERT and Hungarian-BERT respectively. Chordia [26] separately trains
and compares multilingual models for high resource languages such as English,
Hindi, and low resource Indian languages. Guerreiro et al. [27] train and
compare a multilingual and a monolingual punctuator using BERT models for
European languages. Li et al. [28] use a single LSTM model for forty-three
different languages, with a modification to the tokenizer and vocabulary by
using a byte-pair encoding (BPE) scheme to capture shared instances of
vocabulary.

##  III Proposed Model

In this work, we propose an architecture similar to that of the MLM pre-
training objective of BERT. We use a pre-trained XLM-R model, producing
multilingual embeddings, followed by a classifier with two linear layers that
output the punctuation for each masked token. The proposed architecture is
depicted in Figure 1 and more details of the components are described in
Section III-A and Section III-B.

Figure 1: Schematic diagram of our proposed model architecture

###  III-A XLM-R embedding generator

The XLM-R model is a variation of the XLM model [29], which itself enabled
better multilingual training from BERT. XLM-R works in a self-supervised
fashion and is trained using the Masked-Language Model objective, similar to
RoBERTa [30]. XLM-R is trained on 2.5 Terabytes of common-crawled text, while
focusing on low-resource languages. Similar to BERT, it contains multiple
self-attention transformer-encoder layers stacked on one on top of the other.
However, unlike BERT, XLM-R is not trained with the Next Sentence Prediction
(NSP) task. Each of these encoder layers consists of transformer multi-head
self-attention and feed-forward networks. The self-attention layer, also
occasionally referred to as Query-Key-Value attention, maps queries and keys
to an output using a scaled dot-product.

###  III-B Classifier

The classifier consists of two Fully-connected (FC) layers. The layers are
shared between every mask token for each input sample. The outputs of the
classifier of only mask-tokens are considered for back-propagation while
others’ are discarded, since we only predict punctuation, and the masked
tokens represent punctuation marks between two words. An example of the input
and output is described below:

Input string:
hello <mask> this <mask> is <mask> josh <mask> here <mask> to <mask> help
<mask> you <mask>

Output string:
hello P this O is O josh C here O to O help O you P

The input data has no punctuation and a <mask> token is appended after every
word. The output contains $[C,P,Q]$ tokens representing Comma, Period (full-
stop), Question for the three punctuation marks that we are predicting and $O$
represents empty space or no-punctuation.

###  III-C Tokenization

In order to feed the text into the model, we used the tokenizer called
‘SentencePiece’ provided together with the XLM-R framework to tokenize and
encode it into XLM-R input format. The SentencePiece tokenizer is capable of
tokenizing multiple languages at once, including English, Malay, and Mandarin.
Punctuation marks that do not conform to the four classes are either modified
to the most suitable class or discarded. The English and Malay text are
entirely lower-cased to mimic ASR transcripts which usually come without any
punctuation or capitalization.

The entire tokenization and masking process for input data is demonstrated in
Figure 2. Specifically, word boundaries are firstly determined, spaces are
added between determined words, and subsequently, space or punctuation is
replaced with a <mask> token. Word boundaries are easily determined by the
space in the case of English and Malay data. For Mandarin, since there is no
space between characters or words, we employed $Jieba$[31] as an additional
external tokenizer to determine boundaries between each “word”. A space token
$O$ or a punctuation token $[C,Q,P]$ is then added between the determined
“words” for the input sequences. During training, a <mask> token replaces each
space and punctuation token to produce the input sequence $X$, while the
accompanying slot-filling label sequence $Y$ is represented by a sequence of
$O,C,Q,P$ tokens. The entire processed training data, which consists of
${X,Y}$ pairs in English, Malay, and Mandarin, is then shuffled and merged,
and subsequently used for model training. During validation and evaluation,
only the $X$ sequences are provided to the model, and there is no shuffling
and merging after masking.

Figure 2: Tokenization and masking process for input data to the model
training and evaluation. Note that $[O,C,P]$ are labels for “Space”, “Comma”,
and “Period”

##  IV Data and Data Pre-procesisng

###  IV-A Datasets

We train and evaluate our model on monolingual datasets from English,
Mandarin, and Malay. For English and Mandarin, we use the IWSLT2012 [32]
datasets. They consist of transcribed TED talks for Automatic Speech
Recognition (ASR), Speech-Language Translation (SLT), and Machine Translation
(MT). We consider the MT track for our use case, choosing the EN-FR parallel
tracks for English, and the ZH-EN parallel tracks for Mandarin. We use the
same train, validation, and test sets from the IWSLT 2012 dataset, similar to
previous work on punctuation restoration such as [19], [16], [23] for English
and [22] for Mandarin.

Since there is no IWSLT track for Malay, or any publicly available well-formed
Malay-transcript dataset, we crawled news articles that were made public by
Zolkepli [33], similar to the work on Bangla punctuation by [24]. The dataset
is well-formed in sentences and punctuation, and consists of 30,000 news
articles crawled on Google News using keyword-matching to find news articles
written in Malay, for instance, “saham malaysia”, “ekonomi malaysia”,
“hiburan”. We then normalized the crawled Malay text, the details are
presented in Section IV-B1.

###  IV-B Data Preprocessing

####  IV-B1 Text Normalization

The selected English and Mandarin datasets have primarily well-formed texts
that require little to no normalization. Similar to [25], we converted or
removed punctuation marks that do not belong to the four classes. For
instance, we replaced ‘?!’ with ‘?’, and ‘!’ with ‘.’ to mark an end of a
sentence.

The Malay dataset however required additional text normalization to mimic the
form of ASR output. Specifically, we have performed a number-to-word
conversion, URL address normalization, currency normalization, and phone
number conversion. This normalization has been done using the tool developed
by Zolkepli [33].

####  IV-B2 Data Sampling for Malay

The original Malay dataset comes with a huge amount of text with a total
30,000 articles. We have chosen not to use the entire dataset but only retain
about 11,000 articles that have more than two sentences in their content. We
also noticed that the ratio of question marks is significantly lesser than the
other punctuation marks, thus, we have performed oversampling on the number of
question marks using random sampling. Specifically, we selected question
sentences from the dataset and randomly insert them into parts of the selected
articles. Notice that the oversampling has only been performed on the train
set, and not on the validation and test sets. After pre-processing, each
dataset then has been split into train, validation, and test sets. For English
and Mandarin, we adopted the same train, validation, and test sets released by
IWSLT. For Malay, we split the dataset with a ratio of 80/10/10 for train,
validation, and test sets respectively.

####  IV-B3 Final Datasets

Table I shows the number of words, commas, question marks, and periods
presented in each of the selected datasets after pre-processing.

TABLE I: Number of words, commas, question marks, and periods presented in
train, valid, and test sets

| Train | Valid | Test
---|---|---|---
Word | Period | Question | Comma | Word | Period | Question | Comma | Word | Period | Question | Comma
English | 2,374,992 | 190,510 | 10,332 | 141,628 | 15141 | 909 | 71 | 1274 | 16,208 | 1100 | 46 | 1185
Mandarin | 2,321,146 | 156,373 | 10,957 | 85,881 | 15099 | 446 | 89 | 1227 | 15,099 | 446 | 89 | 1227
Malay | 3,135,762 | 266,108 | 9,198 | 252,521 | 428,959 | 24,462 | 837 | 26,023 | 449,694 | 26,440 | 979 | 27,045

##  V Experiments

###  V-A Experimental Setup

We set up our experiment with the following settings:

  * •

The classifier comprises two linear layers with dimensions of 1568 and 4,
respectively.

  * •

Learning rates were set at 1e-4 for the classification layers and 3e-5 for the
XLM-R model which is a similar setting to work presented in [25].

  * •

We use the RAdam [34] optimizer, with a warm-up of 300.

  * •

The training data contains English, Mandarin, and Malay datasets and were
shuffled before training.

The final model was obtained after approximately 9000 iterations, with a batch
size of 4 and a sequence length of 256 on a cluster with an Nvidia RTX-3070
and a Tesla-V100 GPUs.

###  V-B Experiment Results

We report the performance of our model with Precision, Recall, and F1-score
for the Period, Comma, and Question mark classes. We compared our model
performance against baselines that are three recent works on the task, i.e.
SAPR[23], BertPunct[19], and BERT-CNN-RNN[22]. The models are briefly
described below:

  * •

SAPR: This architecture by Wang et al. [23] treats the problem as a machine
translation task, using a transformer decoder as the base architecture. The
work only focuses on English.

  * •

BertPunct: This work by Makhija et al. [19] treats the problem as a sequence
labeling task, using a BERT layer as an embedding and a BiLSTM+CRF layer for
classification. This work also focuses only on English. We use the
BertPunctBASE model for comparison, since we use a similar base model in our
work.

  * •

BERT-CNN-RNN: This model proposed by Zhang et al. [22] focuses on Mandarin.
The architecture involves the use of two layers, a word-CNN layer to capture
word boundary information, and a layer compromising of BERT-Chinese and an RNN
layer for classification of the input into the punctuation class.

  * •

MLM-Punct: Our multilingual model. The model was obtained by performing
multilingual training with datasets being shuffled for all languages.

####  V-B1 Performance on English test set

TABLE II: Performance on English IWSLT2012 Test Set

Model | Period | Comma | Question | Overall
---|---|---|---|---
PR/RC/F1 | PR/RC/F1 | PR/RC/F1 | PR/RC/F1
SAPR | 96.7/97.3/96.8 | 57.2/50.8/ 55.9 | 70.6/69.2/70.3 | 78.2/74.4/77.4
BertPunct | 82.6/83.5/83.1 |  72.1/72.4/72.3 |  77.4/89.1/82.8 |  77.4/81.7/79.4
MLM-Punct |  76.8/86.7/81.5 |  60.4/78.5/68.2 |  78.4/81.7/80.0 | 68.5/82.4/74.7

Table II shows performance of SAPR[23], BertPunct[19] and MLM-Punct on
IWSLT2012 English test set. Note that SAPR and BertPunct are monolingual
models while MLM-Punct is multilingual. MLM-Punct’s performance is close to
the BertPunct model on all three classes with the highest Recall score for
Comma and Precision score for the Question mark, while the BertPunct performs
better on the rest. SAPR performs well on predicting period which is
significantly higher on Precision, Recall, and F1-score for this class,
however, it shows poorer performance on Comma and Question classes. The
difference in performance is possibly due to the English-only BERT encoder
providing better word embeddings for English tokens, as opposed to the
multilingual XLM-R encoder [35]. English is also a very high-resource language
compared to other languages, and the much larger amounts of English data that
go into pre-training English BERT could be another factor.

####  V-B2 Performance on Mandarin test set

Table III shows the performance of our model, MLM-Punct, and the baseline
model, BERT-CNN-RNN, on the IWSLT2012 Mandarin test set. Compared to BERT-CNN-
RNN, MLM-Punct performs better overall, i.e. by 4.7% in F1-score. It
specifically outperforms on Comma and Period classes with an approximate 17%
and 5% improvement in F1-score respectively.

TABLE III: Performance on Mandarin IWSLT2012 Test Set

Model | Period | Comma | Question | Overall
---|---|---|---|---
PR/RC/F1 | PR/RC/F1 | PR/RC/F1 | PR/RC/F1
Zhang et al. |  68.4/83.5/75.2 | 52.0/60.1/55.8 |  78.8/73.9/76.3 | 66.4/72.5/69.1
MLM-Punct |  73.8/80.3/76.9 | 66.3/80.0/72.5 |  75.0/88.7/81.3 | 68.6/80.5/73.8

####  V-B3 Performance on Malay test set

To the best of our knowledge, there is no publicly available punctuation
restoration model for Malay language. Our model performs well for $Period$ and
$Question$ classes, and slightly lower F1-score for $Comma$ class with
F1-scores at 86.1%, 80.0% and 70.8% for each class respectively.

TABLE IV: Performance on Malay News Test Set

Model | Period | Comma | Question | Overall
---|---|---|---|---
PR/RC/F1 | PR/RC/F1 | PR/RC/F1 | PR/RC/F1
MLM-Punct | 81.0/91.9/86.1 | 61.7/83.0/70.8 | 73.6/87.6/80.0 | 70.8/87.2/78.0

##  VI Discussion

In addition to building a multilingual model that can restore punctuation for
the three languages at once, we have experimented with building monolingual
models for each language. We present a comparison between these models in
Section VI-A. Subsequently, Section VI-B discusses the effectiveness of using
the Jieba tokenizer on Mandarin text.

###  VI-A A Comparison between Monolingual and Multilingual training

Table V presents the experimental results on monolingual and multilingual
training settings. Monolingual models, i.e. Mono-English, Mono-Mandarin, Mono-
Malay were trained on the same network architecture as MLM-Punct with pre-
trained multilingual XLM-RoBERTa. However, they were fine-tuned by using only
the monolingual dataset of the respective language i.e. English IWSLT2012,
Mandarin IWSLT2012, and Malay News. The experimental results show that MLM-
Punct outperforms monolingual training across three test sets on overall
Precision, Recall, and F1 scores. Specifically, MLM-Punct gains 1.6%, 1.6% and
3.6% of F1 scores on English, Mandarin, and Malay test sets respectively.

The gains in the overall performance of the multilingual model are possibly
attributed to XLM-R’s cross-lingual pre-training that brings out linguistic
similarities across languages when the model is fine-tuned on a multilingual
dataset. Another possibility is the cross-lingual nature of XLM-R shares the
latent properties relating to punctuation and phrasal information across
languages and thus benefit each other. However, there are some dips in some
specific classes. For example, The Precision score is dipped for $Period$
class on Mandarin and Malay test sets. Another dip is from the $Question$
class in Mandarin, which shows an F1-score dip of 1.6% as compared to the
Mono-Mandarin model.

TABLE V: Results on monolingual and multilingual training. MLM-Punct
represents multilingual training. Mono-X represents model fine-tuned on
Language ‘X’

Models | English IWSLT Test set | Mandarin IWSLT Test set | Malay news Test set
---|---|---|---
Period | Comma | Question | Overall | Period | Comma | Question | Overall | Period | Comma | Question | Overall
PR/RC/F1 | PR/RC/F1 | PR/RC/F1 | PR/RC/F1 | PR/RC/F1 | PR/RC/F1 | PR/RC/F1 | PR/RC/F1 | PR/RC/F1 | PR/RC/F1 | PR/RC/F1 | PR/RC/F1
MLM-Punct | 76.8/86.7/81.5 |  60.4/78.5/68.2 | 78.4/81.7/80.0 | 68.5/82.4/74.7 | 73.8/80.3/76.9 |  66.3/80.0/72.5 | 75.0/88.7/76.3 | 68.6/80.5/73.8 | 81.0/91.9/86.1 |  61.7/83.0/70.8 | 73.6/87.6/80.0 | 70.8/87.2/78.0
Mono-English | 73.6/86.1/79.3 | 58.7/78.7/67.3 | 74.6/74.6/74.6 | 66.0/82.1/73.1 | - | - | - | - | - | - | - | -
Mono-Mandarin | - | - | - | - |  75.1/77.6/76.3 | 60.3/82.9/69.8 |  81.9/86.5/84.2 | 65.1/81.7/72.2 | - | - | - | -
Mono-Malay | - | - | - | - | - | - | - | - |  83.5/81.5/82.5 | 55.8/85.0/67.4 | 69.2/58.3/63.2 | 68.9/83.1/74.4

###  VI-B Using Jieba tokenization for Mandarin

Mandarin text usually comes with no space between characters. Thus, we first
have to split the text into “word” tokens and then insert [SPACE] tokens after
every token. The [SPACE] tokens and the punctuation marks are masked before
feeding the input to the model. To split the text, we have experimented with
using only the XLM-R built-in SentencePiece tokenizer and by using $Jieba$
tokenizer [31]. Our experimental results show that using $Jieba$ helps to
improve the model performance significantly.

The experiment results are presented in Table VI. We named the model using the
SentencePiece tokenizer as $MLM-Punct_{s}$, while our $MLM-Punct$ model
represents the model using $Jieba$ tokenizer. $MLM-Punct$ outperforms $MLM-
Punct_{s}$ on the Mandarin test set with 17.2% improvement on F1-score. This
is probably because $Jieba$ tokenizes Mandarin text into “words”, while
SentencePiece tends to tokenize into sub-words. Predicting punctuation marks
between words is more syntactically and semantically accurate than predicting
punctuation marks between sub-words. In [36], the authors reported a similar
issue in dealing with Mandarin text in using WordPiece tokenizer for BERT.
Specifically, it tends to perform poorly for languages with non-trivial word
boundaries on the tasks that involve word-level information such as POS
tagging and NER.

TABLE VI: Comparison between using SentencePiece tokenizer and $Jieba$
tokenizer for Mandarin data. $MLM-Punct_{s}$ represents the model using
SentencePiece tokenizer and MLM-Punct represents the model using $Jieba$
tokenizer

Model | Period | Comma | Question | Overall
---|---|---|---|---
PR/RC/F1 | PR/RC/F1 | PR/RC/F1 | PR/RC/F1
$MLM-Punct_{s}$ | 34.2/84.4/48.7 | 47.1/75.1/57.9 | 70.3/87.4/77.9 | 45.0/78.1/56.6
$MLM-Punct$ | 73.8/80.3/76.9 | 66.3/80.0/72.5 | 75.0/88.7/76.3 | 68.6/80.5/73.8

##  VII Conclusions

In this work, our proposed model has the ability to punctuate multilingual
text in English, Mandarin, and Malay, treating the problem as a slot-filling
task using the XLM-RoBERTa language representation model. We show that
multilingual training can perform better than monolingual training and achieve
state-of-the-art results for Mandarin punctuation.

For future work, we will consider acquiring more standard evaluation sets for
English and Mandarin, considering that the IWSLT track mainly involves
describing TED Talks. Malay does not have a proper speech transcription
dataset, and requires additional dataset procurement. We will also train and
evaluate on a larger set of languages. Another direction would be to
incorporate prosody features into the architecture to predict uncommon
punctuation. A further contribution following this could be on-the-fly
punctuation in the automatic speech recognition pipeline for the multilingual
space. Previous work exists for the English language [37], [38], [39], but
there has been currently been no work towards the multilingual space. Finally,
we could explore the task of punctuating code-switching text, which is a
challenge due to no defined grammatical rules surrounding them, thus being a
purely machine-learning-oriented task.

##  VIII Acknowledgments

This research is supported by ST Engineering Mission Software & Services Pte.
Ltd under a collaboration programme (Research Collaboration No: REQ0149132).
The computational work for this paper was partially performed on resources of
the National Supercomputing Center, Singapore (https://www.nscc.sg), and on
Sharanga, BITS Pilani, Hyderabad Campus.

## References

  * [1] D. Jones, F. Wolf, E. Gibson, E. Williams, E. Fedorenko, D. Reynolds, and M. Zissman, “Measuring the readability of automatic speech-to-text transcripts,” 01 2003.
  * [2] S. Peitz, M. Freitag, A. Mauser, and H. Ney, “Modeling punctuation prediction as machine translation,” 12 2011, pp. 238–245.
  * [3] V. I. Spitkovsky, H. Alshawi, and D. Jurafsky, “Punctuation: Making a point in unsupervised dependency parsing,” in _Proceedings of the Fifteenth Conference on Computational Natural Language Learning_. Portland, Oregon, USA: Association for Computational Linguistics, Jun. 2011, pp. 19–28. [Online]. Available: https://aclanthology.org/W11-0303
  * [4] H. Christensen, Y. Gotoh, and S. Renals, “Punctuation annotation using statistical prosody models,” _Prosody and Speech Recognition_ , 10 2001.
  * [5] F. Batista, D. Caseiro, N. J. Mamede, and I. Trancoso, “Recovering punctuation marks for automatic speech recognition,” in _INTERSPEECH_ , 2007.
  * [6] O. Tilk and T. Alumäe, “Lstm for punctuation restoration in speech transcripts,” in _Sixteenth annual conference of the international speech communication association_ , 2015.
  * [7] Y. Zhu, L. Wu, S. Cheng, and M. Wang, “Unified multimodal punctuation restoration framework for mixed-modality corpus,” in _ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)_ , 2022, pp. 7272–7276.
  * [8] Y. Liu, E. Shriberg, A. Stolcke, D. Hillard, M. Ostendorf, and M. Harper, “Enriching speech recognition with automatic detection of sentence boundaries and disfluencies,” _Audio, Speech, and Language Processing, IEEE Transactions on_ , vol. 14, pp. 1526 – 1540, 10 2006.
  * [9] W. Lu and H. Ng, “Better punctuation prediction with dynamic conditional random fields.” _EMNLP 2010 - Conference on Empirical Methods in Natural Language Processing, Proceedings of the Conference_ , pp. 177–186, 01 2010\.
  * [10] X. Che, C. Wang, H. Yang, and C. Meinel, “Punctuation prediction for unsegmented transcript based on word vector,” in _Proceedings of the Tenth International Conference on Language Resources and Evaluation LREC 2016, Portorož, Slovenia, May 23-28, 2016_ , N. Calzolari, K. Choukri, T. Declerck, S. Goggi, M. Grobelnik, B. Maegaard, J. Mariani, H. Mazo, A. Moreno, J. Odijk, and S. Piperidis, Eds. European Language Resources Association (ELRA), 2016. [Online]. Available: http://www.lrec-conf.org/proceedings/lrec2016/summaries/103.html
  * [11] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” 2019.
  * [12] A. Conneau, K. Khandelwal, N. Goyal, V. Chaudhary, G. Wenzek, F. Guzmán, E. Grave, M. Ott, L. Zettlemoyer, and V. Stoyanov, “Unsupervised cross-lingual representation learning at scale,” 2020.
  * [13] A. Conneau, G. Lample, R. Rinott, A. Williams, S. R. Bowman, H. Schwenk, and V. Stoyanov, “XNLI: evaluating cross-lingual sentence representations,” _CoRR_ , vol. abs/1809.05053, 2018. [Online]. Available: http://arxiv.org/abs/1809.05053
  * [14] M. Artetxe, S. Ruder, and D. Yogatama, “On the cross-lingual transferability of monolingual representations,” _CoRR_ , vol. abs/1910.11856, 2019. [Online]. Available: http://arxiv.org/abs/1910.11856
  * [15] J. Tiedemann, “The tatoeba translation challenge–realistic data sets for low resource and multilingual mt,” _arXiv preprint arXiv:2010.06354_ , 2020.
  * [16] O. Tilk and T. Alumäe, “Bidirectional recurrent neural network with attention mechanism for punctuation restoration.” in _Interspeech_ , 2016, pp. 3047–3051.
  * [17] W. Salloum, G. Finley, E. Edwards, M. Miller, and D. Suendermann-Oeft, “Deep learning for punctuation restoration in medical reports,” in _BioNLP 2017_. Vancouver, Canada,: Association for Computational Linguistics, Aug. 2017, pp. 159–164. [Online]. Available: https://aclanthology.org/W17-2319
  * [18] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” _CoRR_ , vol. abs/1706.03762, 2017. [Online]. Available: http://arxiv.org/abs/1706.03762
  * [19] K. Makhija, T.-N. Ho, and E.-S. Chng, “Transfer learning for punctuation prediction,” in _2019 Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC)_ , 2019, pp. 268–273.
  * [20] J. Yi, J. Tao, Y. Bai, Z. Tian, and C. Fan, “Adversarial transfer learning for punctuation restoration,” 2020.
  * [21] Q. Chen, W. Wang, M. Chen, and Q. Zhang, “Discriminative self-training for punctuation prediction,” _Interspeech 2021_ , Aug 2021. [Online]. Available: http://dx.doi.org/10.21437/Interspeech.2021-246
  * [22] Z. Zhang, J. Liu, L. Chi, and X. Chen, “Word-level bert-cnn-rnn model for chinese punctuation restoration,” in _2020 IEEE 6th International Conference on Computer and Communications (ICCC)_ , 2020, pp. 1629–1633.
  * [23] F. Wang, W. Chen, Z. Yang, and B. Xu, “Self-attention based network for punctuation restoration,” in _2018 24th International Conference on Pattern Recognition (ICPR)_ , 2018, pp. 2803–2808.
  * [24] T. Alam, A. Khan, and F. Alam, “Punctuation restoration using transformer models for high-and low-resource languages,” in _Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020)_ , 01 2020, pp. 132–142.
  * [25] A. Nagy, B. Bial, and J. Ács, “Automatic punctuation restoration with BERT models,” _CoRR_ , vol. abs/2101.07343, 2021. [Online]. Available: https://arxiv.org/abs/2101.07343
  * [26] V. Chordia, “PunKtuator: A multilingual punctuation restoration system for spoken and written text,” in _Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations_. Online: Association for Computational Linguistics, Apr. 2021, pp. 312–320. [Online]. Available: https://aclanthology.org/2021.eacl-demos.37
  * [27] N. M. Guerreiro, R. Rei, and F. Batista, “Towards better subtitles: A multilingual approach for punctuation restoration of speech transcripts,” _Expert Systems with Applications_ , vol. 186, p. 115740, 2021. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0957417421011180
  * [28] X. Li and E. Lin, “A 43 Language Multilingual Punctuation Prediction Neural Network Model,” in _Proc. Interspeech 2020_ , 2020, pp. 1067–1071.
  * [29] G. Lample and A. Conneau, “Cross-lingual language model pretraining,” _CoRR_ , vol. abs/1901.07291, 2019. [Online]. Available: http://arxiv.org/abs/1901.07291
  * [30] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov, “Roberta: A robustly optimized BERT pretraining approach,” _CoRR_ , vol. abs/1907.11692, 2019. [Online]. Available: http://arxiv.org/abs/1907.11692
  * [31] S. Junyi, “Jieba, word tokenizer for chinese,” https://github.com/fxsjy/jieba, 2013.
  * [32] _2012 International Workshop on Spoken Language Translation, IWSLT 2012, Hong Kong, December 6-7, 2012_. ISCA, 2012. [Online]. Available: http://www.isca-speech.org/archive/iwslt_12/
  * [33] Z. Husein, “Malay-dataset,” https://github.com/huseinzol05/malay-dataset, 2018.
  * [34] L. Liu, H. Jiang, P. He, W. Chen, X. Liu, J. Gao, and J. Han, “On the variance of the adaptive learning rate and beyond,” in _International Conference on Learning Representations_ , 2020. [Online]. Available: https://openreview.net/forum?id=rkgz2aEKDr
  * [35] T. Pires, E. Schlinger, and D. Garrette, “How multilingual is multilingual bert?” 2019.
  * [36] P. Rust, J. Pfeiffer, I. Vulić, S. Ruder, and I. Gurevych, “How good is your tokenizer? on the monolingual performance of multilingual language models,” 2021\.
  * [37] C. J. Chen, “Speech recognition with automatic punctuation,” in _Sixth European Conference on Speech Communication and Technology_ , 1999.
  * [38] Y. Guan, “End to end asr system with automatic punctuation insertion,” 2020. [Online]. Available: https://arxiv.org/abs/2012.02012
  * [39] P. Hlubík, M. Španěl, M. Boháč, and L. Weingartová, “Inserting punctuation to asr output in a real-time production environment,” in _International Conference on Text, Speech, and Dialogue_. Springer, 2020, pp. 418–425.


# A note on the extended Bruinier-Kohnen conjecture

Mohammed Amin Amri  M’hammed Ziane

###### Abstract

Let $f$ be a cusp form of half-integral weight $k+1/2$, whose Fourier
coefficients $a(n)$ not necessarily real. We prove an extension of the
Bruinier-Kohnen conjecture on the equidistribution of the signs of $a(n)$ for
the families $\\{a(tp^{2\nu})\\}_{p,\text{prime}}$, where $\nu$ and $t$ be
fixed odd positive integer and square-free integer respectively.

Keywords: Sign changes; Fourier coefficients of cusp forms; Sato-Tate
Equidistribution.
Mathematics subject classification(2010): 11F03; 11F30; 11F37.

##  1 Introduction and statement of result

Over the recent years, the problem of sign changes of Fourier coefficients of
various kind of automorphic forms has been studied by several authors [14, 16,
7, 13, 3]. The case of modular forms of half-integral weight was first
considered by Bruinier and Kohnen in [6], there, they proved sign change
results for some subfamilies of Fourier coefficients of half-integral weight
modular forms which are accessible via the Shimura-lift. Moreover, they
conjectured that the signs of the sequence of Fourier coefficients of a cusp
form which lies in the Kohnen’s plus space are equidistributed. Through this
problem in [8, 9, 4] Arias, Inam and Wiese proved the Bruinier-Kohnen
conjecture in the special case when the Fourier coefficients of a Hecke
eigenforms of half-integral weight are indexed by $tn^{2}$ with $t$ a fixed
square-free number and $n\in\mathbb{N}$. In [1], the results of [8, 4] were
generalized to Hecke eigenforms with not necessarily real Fourier
coefficients, based on this and empirical evidence the first author
generalized the Bruinier-Kohnen conjecture to cusp forms in Kohnen’s plus
space with not necessarily real Fourier coefficients $a(n)$. More precisely,
he conjectured that

$\lim_{x\to\infty}\dfrac{\\#\\{n\leq x\;:\;{\Re
e\,}\\{a(n)e^{-i\phi}\\}\gtrless 0\\}}{\\#\\{n\leq x\;:\;{\Re
e\,}\\{a(n)e^{-i\phi}\\}\neq 0\\}}=\frac{1}{2},$

for each $\phi$ belonging to $[0,\pi)$.

In the present note we address the above conjecture. Indeed, we shall prove
that it holds for the families $\\{a(tp^{2\nu})\\}_{p\in\mathbb{P}}$ where
$\nu$ and $t$ be a fixed odd positive integer and a square-free integer. In
order to state our result we need to introduce some notations. Let
$\mathbb{P}$ be the set of all prime numbers, and let $k,N$ be natural
numbers, assume from now on that $k\geq 2$ and $N$ be an odd and square-free
integer. Let $\chi$ be a Dirichlet character modulo $4N$, write
$S_{k+1/2}(4N,\chi)$ for the space of cusp forms of weight $k+1/2$ for the
congruence subgroup $\Gamma_{0}(4N)$ with character $\chi$. In this set-up we
have.

######  Theorem.

thm:1 Let ${\mathfrak{f}}\in S_{k+1/2}(4N,\chi)$ be a cuspidal Hecke
eigenform, and let the Fourier expansion of ${\mathfrak{f}}$ at $\infty$ be

${\mathfrak{f}}(z)=\sum_{n\geq 1}a(n)q^{n}\quad q:=e^{2\pi iz}.$

Let $\nu$ be a positive odd integer, and $t$ be a square-free integer such
that $a(t)\neq 0$. Let $\phi\in[0,\pi)$, define the set of primes

$P_{>0}(\phi,\nu):=\\{p\in\mathbb{P}\;:\;{\Re
e\,}(a(tp^{2\nu})e^{-i\phi})>0\\},$

and similarly $P_{<0}(\phi,\nu),P_{\neq 0}(\phi,\nu)$. Then the sequence
$\\{a(tp^{2\nu})\\}_{p\in\mathbb{P}}$ is oscillatory (in the sense of [10]),
moreover the sets $P_{<0}(\phi,\nu)$ and $P_{>0}(\phi,\nu)$ have equal
positive natural density, that is, both are precisely half of the natural
density of the set $P_{\neq 0}(\phi,\nu)$.

The above theorem improves the result of the first author in [1, Theorem 5,
non-CM-case]. The proof which will be given in the next section follows
broadly the same lines of the proof of [1, Theorem 5, non-CM-case] and [2,
Theorem 1.4]. The essential ingredients are the Shimura-lift [17], and the
Sato-Tate conjecture [5, Theorem B].

##  2 Proof of the result

We start by recalling some basic properties of the Shimura-lift and introduce
some notations. The Shimura correspondence [17, 15] maps ${\mathfrak{f}}$ to a
Hecke eigenform $f_{t}$ of weight $2k$ for the group $\Gamma_{0}(2N)$ with
character $\chi^{2}$. Let the Fourier expansion of $f_{t}$ be given by

$f_{t}(z)=\sum_{n\geq 1}A_{t}(n)q^{n}.$

According to [17], the $n$-th Fourier coefficient of $f_{t}$ is given by

$A_{t}(n)=\sum_{d|n}\chi_{t,N}(d)d^{k-1}a\left(\frac{n^{2}}{d^{2}}t\right),$
(2.1)

where $\chi_{t,N}$ denotes the character
$\chi_{t,N}(d):=\chi(d)\left(\frac{(-1)^{k}N^{2}t}{d}\right)$, we let
$\chi_{0}(d):=\left(\frac{(-1)^{k}N^{2}t}{d}\right)$. Since ${\mathfrak{f}}$
is a Hecke eigenform, then so is the Shimura lift. Indeed, we have
$f_{t}=a(t)f$ where $f$ is a normalized Hecke eigenform, write

$f(z)=\sum_{n\geq 1}\lambda(n)n^{k-1/2}q^{n},$

for its Fourier expansion at $\infty$. We shall assume that $a(t)=1$, in the
general case we may apply the proof to $\frac{{\mathfrak{f}}}{a(t)}$. Since
$2N$ is square-free, it follows that $f$ is a Hecke eigenform without complex
multiplication.

Let $\zeta$ be a root of unity belonging to $\mathrm{Im}(\chi)$, and let $p$
be a prime number such that $\chi(p)=\zeta$, it is clear that
$\frac{\lambda(p)}{\zeta}$ is real. By Deligne’s bound we have
$\left|\frac{\lambda(p)}{\zeta}\right|\leq 2$, thus we may write

$\lambda(p)=2\zeta\cos(\theta_{p}),$ (2.2)

for a uniquely defined angel $\theta_{p}\in[0,\pi]$. At this point we state
the following theorem which will be crucial for our purpose.

######  Theorem 1.

(Barnet-Lamb, Geraghty, Harris, Taylor) Assume the set-up above. The sequence
$\\{\theta_{p}\\}_{p}$ is equidistributed in $[0,\pi]$ as $p$ varies over
primes satisfying $\chi(p)=\zeta,$ with respect to the Sato-Tate measure
$\mu_{\text{ST}}:=\frac{2}{\pi}\sin^{2}\theta d\theta$. In particular, for any
sub-interval $I\subset[0,\pi]$ we have

$\lim_{x\to\infty}\dfrac{\\#\\{p\leq x\;:\;\chi(p)=\zeta,\;\theta_{p}\in
I\\}}{\\#\\{p\leq x\;:\;\chi(p)=\zeta\\}}=\frac{2}{\pi}\int_{I}\sin^{2}\theta
d\theta.$

We shall need the following technical lemmas.

######  Lemma 1.

lem1 Let $\zeta$ be a root of unity belonging to $\mathrm{Im}(\chi)$, then the
set

$\\{p\in\mathbb{P}:\chi(p)=\zeta\\},$

have natural density equal to $\frac{1}{r_{\chi}}$, where $r_{\chi}$ denotes
the order of the character $\chi$.

###### Proof.

To establish the above lemma, it suffices to see

$\\{p\in\mathbb{P}:\chi(p)=\zeta\\}=\coprod_{\genfrac{}{}{0.0pt}{}{a\in(\mathbb{Z}/4N\mathbb{Z})^{*}}{\chi(a)=\zeta}}\\{p\in\mathbb{P}\;:\;p\equiv
a\pmod{4N}\\}.$

Note that the number of the sets in the union is
$\\#\mathrm{ker}(\chi)=\frac{\phi(4N)}{r_{\chi}},$ hence by Dirichlet’s
theorem on arithmetic progressions we have

$\lim_{x\to\infty}\dfrac{\\#\\{p\leq
x:\chi(p)=\zeta\\}}{\pi(x)}=\frac{\\#\mathrm{ker}(\chi)}{\phi(4N)}=\frac{1}{r_{\chi}},$

where here and subsequently $\pi(x)$ denotes the prime counting function. ∎

######  Lemma 2.

lem:1 Assume the set-up above. Let $\nu$ be an odd positive integer. Define
the set of primes

$\mathbb{P}_{>0}(\zeta,\nu):=\left\\{p\in\mathbb{P}\;:\;\chi(p)=\zeta,\;\frac{a(tp^{2\nu})}{\zeta^{\nu}}>0\right\\},$

and similarly $\mathbb{P}_{\geq 0}(\zeta,\nu)$, $\mathbb{P}_{<0}(\zeta,\nu)$
and $\mathbb{P}_{\leq 0}(\zeta,\nu)$. Then the sets

$\mathbb{P}_{>0}(\zeta,\nu),\mathbb{P}_{\geq
0}(\zeta,\nu),\mathbb{P}_{<0}(\zeta,\nu),\mathbb{P}_{\leq 0}(\zeta,\nu)$

have a natural density equal to $\frac{1}{2r_{\chi}}$, where $r_{\chi}$
denotes the order of the character $\chi$.

###### Proof.

Denote by $\pi_{>0}(x,\zeta):=\\#\\{p\leq x:p\in\mathbb{P}_{>0}(\zeta,\nu)\\}$
and similarly $\pi_{<0}(x,\zeta)$, $\pi_{\leq 0}(x,\zeta)$,
$\pi_{>0}(x,\zeta)$, $\pi_{\geq 0}(x,\zeta)$. Applying the Möbius inversion
formula to (2.1), we derive that

$a(tn^{2})=\sum_{d|n}\mu(d)\chi_{t,N}(d)d^{k-1}A_{t}\left(\frac{n}{d}\right).$

Set $n=p^{\nu}$ with $\chi(p)=\zeta$ and normalizing by
$p^{\nu(k-1/2)}\zeta^{\nu}$, the above equality specialises to

$\dfrac{a(tp^{2\nu})}{p^{\nu(k-1/2)}\zeta^{\nu}}=\dfrac{\lambda(p^{\nu})}{\zeta^{\nu}}-\frac{\chi_{0}(p)}{\sqrt{p}}\dfrac{\lambda(p^{\nu-1})}{\zeta^{\nu-1}}.$
(2.3)

Since $f$ is a Hecke eigenform its $p^{\nu}$-th Fourier coefficient with
$\chi(p)=\zeta$ is expressible (see [1, Lemma 1]) by the following
trigonometric identity

$\lambda(p^{\nu})=\dfrac{\sin((\nu+1)\theta_{p})}{\sin\theta_{p}}\zeta^{\nu},$
(2.4)

for $\theta_{p}\in(0,\pi)$, and in the limiting cases when $\theta_{p}=0$ and
$\theta_{p}=\pi$ respectively we have $\lambda(p^{\nu})=(\nu+1)\zeta^{\nu}$
and $\lambda(p^{\nu})=(-1)^{\nu}(\nu+1)\zeta^{\nu}$, which can happen for at
most finitely many primes $p$ only (see [11, Remark 2] ). Thus we may assume
and do that $\theta_{p}\in(0,\pi)$. Altogether from (2.3) and (2.4) we have

$\dfrac{a(tp^{2\nu})}{\zeta^{\nu}}>0\Longleftrightarrow\sin((\nu+1)\theta_{p})>\frac{\chi_{0}(p)}{\sqrt{p}}\sin(\nu\theta_{p}).$
(2.5)

Let $\epsilon>0$ (small enough). Since for all $p>\frac{1}{\epsilon^{2}}$ one
has $\left|\frac{\chi_{0}(p)}{\sqrt{p}}\sin(\nu\theta_{p})\right|<\epsilon$,
then from (2.5) we have the inclusion of sets

$\left\\{p\leq
x:\chi(p)=\zeta,\sin((\nu+1)\theta_{p})>\epsilon\right\\}\subset\left\\{p\leq\frac{1}{\epsilon^{2}}:\chi(p)=\zeta\right\\}\cup\\{p\leq
x:p\in\mathbb{P}_{>0}(\zeta,\nu)\\}.$ (2.6)

On the other hand we have

$\sin((\nu+1)\theta_{p})>\epsilon\Longleftrightarrow\theta_{p}\in
I_{\epsilon}:=\bigcup_{j=1}^{\frac{\nu+1}{2}}\left(\frac{(2j-2)\pi+\arcsin(\epsilon)}{\nu+1},\frac{(2j-1)\pi-\arcsin(\epsilon)}{\nu+1}\right),$

Thus (2.6) is equivalent to

$\left\\{p\leq x:\chi(p)=\zeta,\theta_{p}\in
I_{\epsilon}\right\\}\subset\left\\{p\leq\frac{1}{\epsilon^{2}}:\chi(p)=\zeta\right\\}\cup\\{p\leq
x:p\in\mathbb{P}_{>0}(\zeta,\nu)\\}.$

Consequently we have

$\pi_{>0}(x,\zeta)+\pi_{\zeta}\left(\frac{1}{\epsilon^{2}}\right)\geq\\#\left\\{p\leq
x:\chi(p)=\zeta,\theta_{p}\in I_{\epsilon}\right\\},$

where, $\pi_{\zeta}(x):=\\#\\{p\leq x:\chi(p)=\zeta\\}$. Now divide the above
inequality by $\pi_{\zeta}(x)$, we obtain

$\dfrac{\pi_{>0}(x,\zeta)}{\pi_{\zeta}(x)}+\dfrac{\pi_{\zeta}\left(\frac{1}{\epsilon^{2}}\right)}{\pi_{\zeta}(x)}\geq\dfrac{\\#\left\\{p\leq
x:\chi(p)=\zeta,\theta_{p}\in I_{\epsilon}\right\\}}{\pi_{\zeta}(x)}.$ (2.7)

Since $\pi_{\zeta}\left(\frac{1}{\epsilon^{2}}\right)$ is finite the term
$\frac{\pi_{\zeta}\left(\frac{1}{\epsilon^{2}}\right)}{\pi_{\zeta}(x)}$ tends
to zero as $x\to\infty$. By Theorem 1 we have

$\lim\limits_{x\to\infty}\frac{\\#\left\\{p\leq x:\chi(p)=\zeta,\theta_{p}\in
I_{\epsilon}\right\\}}{\pi_{\zeta}(x)}=\mu_{ST}(I_{\epsilon}).$

Hence a passage to the limit in (2.7) yields

$\liminf_{x\to\infty}\dfrac{\pi_{>0}(\zeta,x)}{\pi_{\zeta}(x)}\geq\mu_{ST}(I_{\epsilon}).$

Letting $\epsilon$ goes to zero in the above inequality we find
$\liminf\limits_{x\to\infty}\dfrac{\pi_{>0}(\zeta,x)}{\pi_{\zeta}(x)}\geq\mu_{ST}(I)$,
where
$I:=\bigcup_{j=1}^{\frac{\nu+1}{2}}\left(\frac{(2j-2)\pi}{\nu+1},\frac{(2j-1)\pi}{\nu+1}\right)$.
Now from [12] we have $\mu_{ST}(I)=\frac{1}{2}$, therefore

$\liminf_{x\to\infty}\dfrac{\pi_{>0}(\zeta,x)}{\pi_{\zeta}(x)}\geq\frac{1}{2}.$

A similar reasoning yields $\liminf\limits_{x\to\infty}\frac{\pi_{\leq
0}(\zeta,x)}{\pi_{\zeta}(x)}\geq\frac{1}{2}$, in combination with $\pi_{\leq
0}(x,\zeta)=\pi_{\zeta}(x)-\pi_{>0}(x,\zeta)$, one sees
$\limsup\limits_{x\to\infty}\frac{\pi_{>0}(\zeta,x)}{\pi_{\zeta}(x)}\leq\frac{1}{2}$,
whence $\lim\limits_{x\to\infty}\frac{\pi_{>0}(\zeta,x)}{\pi_{\zeta}(x)}$
exists and equal to $\frac{1}{2}$. Now from Lemma LABEL:lem1, we conclude

$\lim_{x\to\infty}\frac{\pi_{>0}(\zeta,x)}{\pi(x)}=\frac{1}{2r_{\chi}}.$

Similar arguments apply to the other cases. ∎

Now we proceed to prove our main theorem. Fix $\phi\in[0,\pi)$. Let $\zeta$ be
a root of unity belonging to $\mathrm{Im}(\chi)$. If $p$ a prime number
satisfying $\chi(p)=\zeta$, from Lemma LABEL:lem:1 we know that
$\frac{a(tp^{2\nu})}{\zeta^{\nu}}$ is real, thus we my write

${\Re e\,}(a(tp^{2\nu})e^{-i\phi})=\dfrac{a(tp^{2\nu})}{\zeta^{\nu}}{\Re
e\,}(\zeta^{\nu}e^{-i\phi})\quad\text{with}\quad\chi(p)=\zeta.$ (2.8)

If $\mathrm{arg}(\zeta^{\nu})\equiv\phi\pm\frac{\pi}{2}\pmod{2\pi}$, the
sequence $\\{{\Re e\,}(a(tp^{2\nu})e^{-i\phi})\\}_{p,\chi(p)=\zeta}$ is
trivial. Assume
$\mathrm{arg}(\zeta^{\nu})\not\equiv\phi\pm\frac{\pi}{2}\pmod{2\pi}$, in this
case the sequence $\\{{\Re e\,}(a(tp^{2\nu})e^{-i\phi})\\}_{p,\chi(p)=\zeta}$
is not trivial. Without restriction of generality we can assume ${\Re
e\,}(\zeta^{\nu}e^{-i\phi})>0$. Thus for any fixed $\epsilon>0$ (but small)
from (2.5) we have

$\left\\{p\in\mathbb{P}:p>\frac{1}{\epsilon^{2}},\;\chi(p)=\zeta,\;\theta_{p}\in
I_{\epsilon}\right\\}\subset\\{p\in\mathbb{P}:\chi(p)=\zeta,\;{\Re
e\,}(a(tp^{2\nu})e^{-i\phi})>0\\},$ (2.9)

and

$\left\\{p\in\mathbb{P}:p>\frac{1}{\epsilon^{2}},\;\chi(p)=\zeta,\;\theta_{p}\in
I^{\prime}_{\epsilon}\right\\}\subset\\{p\in\mathbb{P}:\chi(p)=\zeta,\;{\Re
e\,}(a(tp^{2\nu})e^{-i\phi})<0\\},$ (2.10)

where

$I_{\epsilon}:=\bigcup_{j=1}^{\frac{\nu+1}{2}}\left(\frac{(2j-2)\pi+\arcsin(\epsilon)}{\nu+1},\frac{(2j-1)\pi-\arcsin(\epsilon)}{\nu+1}\right)\subset[0,\pi],$

and

$I^{\prime}_{\epsilon}:=\bigcup_{j=1}^{\frac{\nu+1}{2}}\left(\frac{(2j-1)\pi+\arcsin(\epsilon)}{\nu+1},\frac{2j\pi-\arcsin(\epsilon)}{\nu+1}\right)\subset[0,\pi].$

From Theorem 1 we know that the sequence $\\{\theta_{p}\\}_{p,\chi(p)=\zeta}$
is equidistributed in $[0,\pi]$ with respect to the Sato-Tate measure
$\mu_{ST}$, it follows that there are infinitely many primes $p$ satisfying
$\chi(p)=\zeta$ such that $\theta_{p}\in I^{\prime}_{\epsilon}$ and infinitely
many primes $p$ satisfying $\chi(p)=\zeta$ such that $\theta_{p}\in
I_{\epsilon}$, thus the sets in (2.9) and (2.10) are infinite. Hence for all
root of unity $\zeta$ belonging to $\mathrm{Im}(\chi)$ the sequence
$\\{a(tp^{2\nu})\\}_{p,\chi(p)=\zeta}$ is oscillatory, then the sequence
$\\{a(tp^{2\nu})\\}_{p\in\mathbb{P}}$ so is.

What is left is to calculate the natural densities $\delta(P_{>0}(\phi,\nu))$
and $\delta(P_{<0}(\phi,\nu))$ of the sets $P_{>0}(\phi,\nu)$ and
$P_{<0}(\phi,\nu)$. From (2.8) one observes that

$P_{>0}(\phi,\nu)=\coprod_{\genfrac{}{}{0.0pt}{}{\zeta,\text{root of
unity}}{\zeta\in\text{Im}(\chi),{\Re
e\,}(\zeta^{\nu}e^{-i\phi})>0}}\mathbb{P}_{>0}(\zeta,\nu)\bigsqcup\coprod_{\genfrac{}{}{0.0pt}{}{\zeta,\text{root
of unity}}{\zeta\in\text{Im}(\chi),{\Re
e\,}(\zeta^{\nu}e^{-i\phi})<0}}\mathbb{P}_{<0}(\zeta,\nu),$

and

$P_{<0}(\phi,\nu)=\coprod_{\genfrac{}{}{0.0pt}{}{\zeta,\text{root of
unity}}{\zeta\in\text{Im}(\chi),{\Re
e\,}(\zeta^{\nu}e^{-i\phi})<0}}\mathbb{P}_{>0}(\zeta,\nu)\bigsqcup\coprod_{\genfrac{}{}{0.0pt}{}{\zeta,\text{root
of unity}}{\zeta\in\text{Im}(\chi),{\Re
e\,}(\zeta^{\nu}e^{-i\phi})>0}}\mathbb{P}_{<0}(\zeta,\nu),$

up to finitely many primes. Then the above formula combined with Lemma
LABEL:lem:1, gives

$\displaystyle\delta\left(P_{>0}(\phi,\nu)\right)$ $\displaystyle=$
$\displaystyle\lim_{x\to\infty}\sum_{\genfrac{}{}{0.0pt}{}{\zeta,\text{root of
unity}}{\zeta\in\text{Im}(\chi),{\Re
e\,}(\zeta^{\nu}e^{-i\phi})>0}}\dfrac{\pi_{>0}(x,\zeta)}{\pi(x)}+\lim_{x\to\infty}\sum_{\genfrac{}{}{0.0pt}{}{\zeta,\text{root
of unity}}{\zeta\in\text{Im}(\chi),{\Re
e\,}(\zeta^{\nu}e^{-i\phi})<0}}\dfrac{\pi_{<0}(x,\zeta)}{\pi(x)}$
$\displaystyle=$
$\displaystyle\frac{1}{2}\sum_{\genfrac{}{}{0.0pt}{}{\zeta,\text{root of
unity}}{\zeta\in\text{Im}(\chi),{\Re e\,}(\zeta^{\nu}e^{-i\phi})\neq
0}}\frac{1}{r_{\chi}},$ $\displaystyle=$ $\displaystyle\frac{\delta(P_{\neq
0}(\phi,\nu))}{2}.$

Likewise we get $\delta\left(P_{<0}(\phi,\nu)\right)=\frac{\delta(P_{\neq
0}(\phi,\nu))}{2}$, which concludes the proof.

## References

  * [1] Amri M. A.: Oscillatory behavior and equidistribution of signs of Fourier coefficients of cusp forms. to appear in Ramanujan J, (2018). https ://doi.org/10.1007/s11139-018-0078-8
  * [2] Amri M. A.: Simultaneous sign change and equidistribution of signs of Fourier coefficients of two cusp forms. Arch. Math. 111(2), (2018), 257–266.
  * [3] Amri, M. A .: Angular changes of complex Fourier coefficients of cusp forms. https://arxiv.org/abs/1704.00982
  * [4] Arias-de-Reyna, S., Ilker I., and Gabor W, .: On conjectures of Sato-Tate and Bruinier-Kohnen.  Ramanujan J, 36(3), (2015): 455–481.
  * [5] Barnet-Lamb, Geraghty, D., Harris, M., Taylor, R.: A Family of Calabi-Yau Varities and Potential Automorphy II, Pub. Res. Inst. Math. Sci., 47, (2011), 29-98,
  * [6] Bruinier, J.-H., Kohnen, W.: Sign changes of coefficients of half integral weight modular forms.In: Edixhoven, B. (ed.) Modular forms on Schiermonnikoong, (2008), pp. 57–66. Cambridge University Press, Cambridge.
  * [7] Das, S., Kohnen, W.: On sign changes of eigenvalues of Siegel cusp forms of genus 2 in prime powers. Acta Arith. 183(2), (2018), 167–172.
  * [8] Inam, I., Wiese, G.: Equidistribution of Signs for Modular Eigenforms of Half Integral Weight. Arch. Math, 101, (2013), 331-339,
  * [9] Inam, I. and Wiese, G .: A short note on the Bruiner–Kohnen sign equidistribution conjecture and Halàsz’ theorem. Int. Journal of Number Theory. 12 (02), (2016), 357-360.
  * [10] Knopp, M., Kohnen, W., Pribitkin, W.: On the signs of Fourier coefficients of cusp forms, Ramanujan J., 7(1), (2003),269-277.
  * [11] Kohnen, W., Lau, Y-K. and Wu, J.: Fourier coefficients of cusp forms of half-integral weight J. Math. Z, 273,(2013), 29-41.
  * [12] Meher, J., Shankhadhar, K. D., Viswanadham, G. K.: On the coefficients of symmetric power $L$-functions. Int. Journal of number theory. 14(3), (2018), 813–824.
  * [13] Meher, J., and Naomi Tanabe. Sign changes of Fourier coefficients of Hilbert modular forms. J. of Number Theory. 145, (2014), 230-244.
  * [14] Meher, J., and M. Murty. Oscillations of coefficients of Dirichlet series attached to automorphic forms. Proc. AMS, 145(2), (2017), 563-575.
  * [15] Niwa, S.: Modular forms of half integral weight and the integral of certain theta-functions. Nagoya Math J,56, (1975), 147-161.
  * [16] Pribitkin, W.: On the oscillatory behavior of certain arithmetic functions associated with automorphic forms. J. of Number Theory, 131(11), (2011), 2047-2060.
  * [17] Shimura, G.: On Modular Forms of Half-Integral Weight, Ann. of Math., 97, (1973), 440-481.

Mohammed Amin Amri, ACSA Laboratory, Department of Mathematics,Faculty of
Sciences, Mohammed First University,Oujda, Morocco.

E-mail address, Mohammed Amin Amri<EMAIL_ADDRESS>
M’hammed Ziane, ACSA Laboratory, Department of Mathematics,Faculty of
Sciences, Mohammed First University,Oujda, Morocco.

E-mail address, M’hammed Ziane<EMAIL_ADDRESS>