# Looking and Listening: Audio Guided Text Recognition

Wenwen Yu1, Mingyu Liu1, Biao Yang1, Enming Zhang1, Deqiang Jiang2,
Xing Sun2, Yuliang Liu1, Xiang Bai1
1Huazhong University of Science and Technology 2Tencent YouTu Lab
<EMAIL_ADDRESS>
###### Abstract

Text recognition in the wild is a long-standing problem in computer vision.
Driven by end-to-end deep learning, recent studies suggest vision and language
processing are effective for scene text recognition. Yet, solving edit errors
such as add, delete, or replace is still the main challenge for existing
approaches. In fact, the content of the text and its audio are naturally
corresponding to each other, i.e., a single character error may result in a
clear different pronunciation. In this paper, we propose the AudioOCR, a
simple yet effective probabilistic audio decoder for mel spectrogram sequence
prediction to guide the scene text recognition, which only participates in the
training phase and brings no extra cost during the inference stage. The
underlying principle of AudioOCR can be easily applied to the existing
approaches. Experiments using 7 previous scene text recognition methods on 12
existing regular, irregular, and occluded benchmarks demonstrate our proposed
method can bring consistent improvement. More importantly, through our
experimentation, we show that AudioOCR possesses a generalizability that
extends to more challenging scenarios, including recognizing non-English text,
out-of-vocabulary words, and text with various accents. Code will be available
at https://github.com/wenwenyu/AudioOCR.

##  1 Introduction

Reading text from a natural image is a long-standing and profound problem in
computer vision. It facilitates various applications such as document
scanning, automatic assisting, and archiving data entry. Driven by the success
of the deep neural network, text recognition has achieved remarkable progress
in the past decades.

Scene text recognition methods usually adopt visual processing and language
modeling pipelines for parsing text content. The former usually adopts deep
feature extraction for sequential prediction, while the latter is then used to
rectify the recognition results both implicitly and explicitly. Although such
a pipeline greatly advances the development of the field of scene text
recognition, the edit errors such as add, delete, or replace are still one of
the main challenges.

Figure 1: Audio information can provide the salient information for guiding
text recognition.

To address this issue, we attempt to use audio information to guide the scene
text recognition. In fact, as shown in Figure 1, a single character such as
“h” or “n” may not have a significant difference simply based on the visual
appearance, as scene text could be presented with various fonts, vague,
perspective distortion, _etc_ ; however, such error may result in completely
different of the pronunciation of the entire word. Recently, using audio
information to assist vision tasks have been proven effective [57, 6, 7, 65],
_e.g._ , Tian et al.  [56]. show that using audio-visual integration can
strengthen the perception ability and thus improve the robustness of audio-
visual models. Tang et al.  [55] uses an Automatic Speech Recognition (ASR)
model [4, 1] to convert annotations from audio modal to text modal for text
spotting tasks, and accelerate the annotation procedure. To the best of our
knowledge, few methods consider using audio information for guiding scene text
recognition.

In this paper, we propose a novel module, termed AudioOCR, using a
transformer-based probabilistic audio decoder for spectrogram sequence
prediction. It can be easily plugged into the existing approaches only during
the training phase, and thus it brings no extra cost to the baseline model yet
introduces obvious improvement. To enable the training, as shown in Figure 4,
we adopt a PaddleSpeech [69]111https://github.com/PaddlePaddle/PaddleSpeech
engine to convert the text label to the target of mel spectrogram recognition.
The text labels are directly from the existing synthesized datasets like MJ
[19] and ST [14], which are commonly served as standard training data for
scene text recognition. The underlying principle of our method is that the
text always has the correct pronunciation regardless of its shape, size,
clarity, _etc_. We summarize our method as follows:

  * •

We discovered a new approach, termed AudioOCR, whereby speech information can
supplement text recognition, paving the way for further advances in the field.

  * •

The simplicity of our AudioOCR allows it to be applied to the existing
approaches, _i.e._ , it can apparently improve the performance of the
recognizers, incurring acceptable computational overhead during the training
phase and none whatsoever during the testing phase.

  * •

We evaluate our method by integrating the AudioOCR into 7 previous scene text
recognition methods on 12 benchmarks including regular, irregular, and
occluded datasets, which demonstrates our method can improve the performance
of each recognition method by an average of 1.75% in terms of the recognition
accuracy.

##  2 Related Works

Visual-audio modeling. In the visual-audio domain, there are several
advancements that happened in visual-audio encoders in the past. Qu et al.
[40] proposed an encoder-decoder architecture LipSound2 which directly
predicts mel-scale spectrogram from raw face sequences. Chung et al. [10]
proposed a ‘Watch, Listen, Attend and Spell’ (WLAS) network that learns to
transcribe videos of mouth motion to characters for lip reading. Arandjelovic
et al. [2] designed new network architectures that can be trained for cross-
modal retrieval and localizing the sound source in an image through audio-
visual correspondence tasks. Oh et al. [36] proposed Speech2Face model for
reconstructing a facial image of a person from a short audio recording of that
person speaking.

Text-audio modeling. In the text-audio domain, it’s a rapidly evolving field
with ongoing research in many different areas, such as Automatic Speech
Recognition (ASR) [15, 9, 4, 27] and Text-to-Speech (TTS) Synthesis [63, 53,
43, 41].

Compared to the visual-audio and text-audio domain, our method is different in
several aspects: 1) The audio-decoder is a Transformer-based model that is
parallelly trained via self-attention mechanism; 2) our method takes the input
of visual modality and outputs the audio modality information; 3) our method
develops a simple yet effective plug-and-play pipeline specifically designed
for OCR, which does not introduce computational overheads in inference stage.

Scene text recognition. Scene text recognition has been an active and
challenging research topic in the field of computer vision. The popular
methods have relied on encoder-decoder architectures [13, 52, 33, 3, 62, 5,
61, 66, 5], and recent approaches have utilized convolutional neural networks
(CNNs) as encoders, followed by recurrent neural networks (RNNs) or
connectionist temporal classification (CTC) for decoding [17, 45, 52]. Then
attention mechanisms [24] and spatial transformer networks [46, 47] based
methods have been proposed to handle irregular text with arbitrary shapes and
rotations. Apart from this, some methods [28, 60] attempt to recognize text
via character segmentation.

Recently, some methods tend to leverage auxiliary information to assist the
recognition process [67, 64, 11, 39, 12, 54, 34], such as the linguistic
knowledge. SRN [67] introduces a global semantic reasoning module to refine
vision model predictions. SEED [39] designs a semantic module, supervised by
embedding outputs of a pretrained language model. Fang et al. [12] propose a
method that is based entirely on CNN for both vision and language models.
ABINet [11] utilizes a bidirectional network as the language model and an
explicit language model by blocking gradient flow. Besides, Song et al. [50]
mask random characters in the visual model and restore them by language model
to learn linguistic knowledge in a union.

As shown above, previous works mainly leverage linguistic knowledge to assist
recognition task; however, audio, another meaningful information is less
studied in the literature. Thus, we explicitly utilize audio information
through automatic generation without extra annotation costs.

##  3 Methodology

In this section, we first introduce the overall architecture of the proposed
audio guided text recognition framework. Then, we elaborate on the proposed
plug-and-play audio decoder, termed AudioOCR, and further analyze the
advantages of audio in feature modeling. Finally, we describe the details of
the generation of the mel spectrogram of the audio decoder as well as the
training process of cooperating with existing recognition methods.

Figure 2: The pipeline of audio guided text recognition method. AudioOCR
consists of the image encoder, the recognition decoder, and the pluggable
audio decoder for guiding text recognition. Dashed arrows are the training-
only operators. Solid arrows indicate forward pass in both training and
inference.

###  3.1 Audio Guided Text Recognition

Our main objective is to precisely recognize each character in images with the
help of not only looking but also listening. To this end, we jointly train a
simple audio decoder with existing methods to predict the spectrogram of the
audio of the corresponding transcription. As shown in Figure 2, our proposed
audio-guided text recognition architecture is composed of an image encoder, a
recognition decoder, and an audio decoder.

Given an input image $\mathbf{X}$, the image encoder generates visual image
embeddings $\mathbf{\hat{X}}$, which is formulated as follows:

$\mathbf{\hat{X}}=\mathtt{ImageEncoder}\left(\mathbf{X}\leavevmode\nobreak\
;\Theta_{\text{enc}}\right)\,,$ (1)

where $\mathbf{X}\in\mathbb{R}^{H^{\prime}\times W^{\prime}\times 3}$ denotes
the vector of input image, $H^{\prime}$ and $W^{\prime}$ represent the height
and width of image, respectively. $\mathbf{\hat{X}}\in\mathbb{R}^{H\times
W\times d_{model}}$ represents the output of the image encoder, and
$d_{model}$ indicates the dimension of model. $\mathbf{\hat{X}}$ will be used
as input for the recognition decoder and audio decoder. $\Theta_{\text{enc}}$
represents the parameters of the image encoder. In practice, image encoder,
depending on different recognizers, can be instantiated into a wide variety of
model structures, which usually includes the CNN based _e.g._ ResNet [16] and
transformer [58] based image encoder along with the extra module _e.g._ ,
flexible rectification [47].

The recognition decoder receives the visual image embedding $\mathbf{\hat{X}}$
to produce a sequence of predicted characters
$\hat{Y}=\\{y_{1},y_{2},\ldots,y_{n}\\}$, in which $n$ is the length of
predicted sequence and $y_{i}$ is the $i$-th predicted characters, which can
be expressed as:

$\hat{Y}=\mathtt{RecoDecoder}\left(\mathbf{\hat{X}}\leavevmode\nobreak\
;\Theta_{\text{rec}}\right)\,,$ (2)

where RecoDecoder means the recognition decoder module, and
$\Theta_{\text{rec}}$ represents the parameters of the recognition decoder.

In practice, there are three types of recognition decoders commonly used in
real scenarios, including (1) Connectionist Temporal Classification (CTC) [13,
45]; (2) attention-based sequence prediction (Attn) [47, 26]; and (3)
transformer decoder based sequence prediction (Trans) [25, 31, 37]. CTC allows
for the prediction of a non-fixed number of a sequence even though a fixed
number of the features are given. Attn-based decoder automatically captures
the information flow within the input sequence to predict the output sequence.
It enables a text recognition model to learn a character-level language model
representing output class dependencies. Trans-based decoder leverages self-
attention to model long-range dependency between features and predicts the
characters of the sequence via an auto-regressive manner.

###  3.2 Audio Decoder

The audio decoder, as shown in Figure 3, is proposed to guide the recognizers
to capture audio modal information. It consists of a Prenet, a visual-audio
decoder, and a Mel Linear.

Prenet is composed of two fully connected layers (each has 256 hidden units)
with ReLU activation. Prenet is responsible for projecting mel spectrograms
into the phoneme subspace of audio, and thus the similarity of a phoneme and
mel frame pair can be measured to force the following visual-audio decoder to
easily capture visual and audio interaction, which is defined as:

$\mathbf{\hat{Z}}=\mathtt{Prenet}\left(\mathbf{\dot{Z}}\leavevmode\nobreak\
;\Theta_{\text{pre}}\right)\,,$ (3)

where $\mathbf{\dot{Z}}$ is the shifted-right format [58] of the mel
spectrogram $\mathbf{Z}\in\mathbb{R}^{T\times 80}$ (Sec. 3.3 describes the
generation way), where $T$ is the time length of audio.
$\mathbf{\hat{Z}}\in\mathbb{R}^{T\times 256}$ denotes the transformed vector
of mel spectrogram containing compact and low dimensional subspace features of
audio. $\Theta_{\text{pre}}$ represents the parameters of the Prenet.

The visual-audio decoder is implemented by the transformer decoder. It
utilizes the mel spectrogram features $\mathbf{\hat{Z}}$ as query and image
embeddings $\mathbf{\hat{X}}$ as key and value to model cross-modal
interaction between visual and audio features for learning effective scene
text visual representations via cross-attention. The visual-audio decoder
hence learns the relationships between visual and audio features. To this end,
the visual features $\mathbf{\hat{X}}$ implicitly contain the audio
information which is helpful for text recognition of complex scenarios. We add
a set of learned positional encoding [58] to mel spectrogram
$\mathbf{\hat{Z}}$ for capturing the sequential information, then passing it
to visual-audio decoder to refine the mel spectrogram conditioned on visual
features:

$\mathbf{\tilde{Z}}=\mathtt{VADecoder}\left(\mathbf{\mathtt{Q}=\hat{Z},\mathtt{K}=\hat{X},\mathtt{V}=\hat{X}}\leavevmode\nobreak\
;\Theta_{\text{va}}\right)\,,$ (4)

where $\mathbf{\tilde{Z}}\in\mathbb{R}^{T\times 256}$ denotes the refined mel
spectrogram. VADecoder means the visual-audio decoder. $\Theta_{\text{va}}$
represents the parameters of the visual-audio decoder. In practice, the
default settings of the visual-audio decoder consist of $N=3$ stacked decoder
layers. Each decoder layer contains a masked multi-head attention, a multi-
head attention layer, and an MLP network.

The Mel Linear implemented by a linear projection and the $\mathbf{\tilde{Z}}$
is then used to predict the mel spectrogram, which can be formulated by:

$\mathbf{\bar{Z}}=\mathtt{MelLinear}\left(\mathbf{\tilde{Z}}\leavevmode\nobreak\
;\Theta_{\text{mel}}\right)\,,$ (5)

where $\mathbf{\bar{Z}}\in\mathbb{R}^{T\times 80}$ denotes the predicted mel
spectrogram. MelLinear means the Mel Linear. $\Theta_{\text{mel}}$ represents
the parameters of the Mel Linear. Then $\mathbf{\bar{Z}}$ and $\mathbf{Z}$
will calculate the L1 loss $L_{mel}$:

$L_{mel}=\mathtt{L1}\left(\mathbf{\check{Z}}\leavevmode\nobreak\
,\mathbf{\bar{Z}}\right)\,,$ (6)

where the mel spectrogram label $\mathbf{\check{Z}}$ and the input of Prenet
$\mathbf{\dot{Z}}$ are basically the same (mel spectrogram $\mathbf{Z}$). The
only difference is the input $\mathbf{\dot{Z}}$ is the shifted-right format
[58] of the $\mathbf{Z}$.

Figure 3: The architecture of the audio decoder. It includes a Prenet, a
visual-audio decoder, and a Mel Linear. LayerNorm and Dropout layers are
omitted due to space constraints. Figure 4: The pipeline of converting the
text label to the mel spectrogram used in the audio decoder.

###  3.3 Mel Spectrogram $\mathbf{Z}$ Generation

Since there is no publicly available dataset that includes text-audio pairs,
we use the off-the-shelf text-to-speech tool PaddleSpeech to generate the
audio of the transcription of every image and store the audio file with the
suffix $.wav$. We then convert the audio data to the mel spectrogram used for
the training of the audio decoder.

Specifically, we first get the waveform $\bm{x}$ of the audio from the $.wav$
file, with the shape of $[C,T]$ where $C$ is the number of channels ($C=2$ for
stereo audio), and $T$ the number of points used to represent the audio, as
shown in Figure 4. Then Short-Time Fourier Transform (STFT) is employed over
the waveform $\bm{x}$ to represent audio into a complex-valued tensor of shape
$[C,F,L]$ where $F$ is the number of frequencies and $L$ is the length. The
STFT is computed on overlapping windowed segments of the waveform $\bm{x}$.
The STFT at a point ($f$, $l$) is defined as:

$\mathtt{STFT}\\{\bm{x}\\}(f,l)=\sum_{t=0}^{T}\bm{x}_{t}\cdot\bm{w}_{t-l}\cdot\mathtt{exp}\left(-i\frac{2\pi
t}{T}f\right)\,,$ (7)

where $\bm{w}$ is a slide window function of length $W$ that is used to
isolate the chunk of interest from the waveform signal. $f$, $l$, and $i$ are
frequency, time step, and imaginary part, respectively.

Instead of real and imaginary parts, often the STFT is converted into
magnitude and phase, a representation that can be inverted back to real and
imaginary parts, and hence back to the waveform. Magnitude, called linear
sepctrogram, is defined as $|\mathtt{STFT}\\{\bm{x}\\}(f,l)|$.

Finally, we perform a mathematical operation on the frequencies of the linear
spectrogram to convert the linear scale to the mel scale [51], called mel
spectrogram. The reason is that humans do not perceive frequencies on a linear
scale [51], and equal distances in mel scale sound equally distant to the
listener. The mathematical operation is formulated as:

$m=2595*\log_{10}\left(1+\frac{f}{700}\right)\,,$ (8)

where $f$ denotes the linear scale frequency, and $m$ denotes the mel scale
frequency. The configuration of mel spectrogram generation refers to appendix.

###  3.4 Training strategy

Our model is designed to jointly learn with recognition loss and audio loss.
The loss function $L$ can be expressed as a weighted sum of the recognition
losses and audio losses:

$L=L_{rec}+\alpha\times L_{mel},$ (9)

where $L_{rec}$ is the loss for the recognition decoder, and $L_{mel}$ (Eq. 6)
is the L1 loss for the audio decoder. According to the numeric values of the
losses, $\alpha$ is set to $1.0$ by default.

##  4 Experiment

We first incorporate the AudioOCR into the existing recognizers to validate
its effectiveness. Next, we conduct ablation studies to measure the
sensitivity of the designs and generalization ability for AudioOCR.

###  4.1 Datasets

We use three types of datasets including synthetic datasets, real scene text
recognition benchmarks, and our SynthAudio data. Synthetic datasets including
MJSynth (MJ) [19], SynthText (ST) [14], and SynAdd (SA) [26]. The details
descriptions of the synthetic datasets refer to appendix.

Scene text recognition benchmarks including (1) IIIT5k-Words (IIIT) [32]; (2)
Street View Text (SVT) [48] ; (3) ICDAR 2013 (IC13) [21]; (4) ICDAR 2015
Incidental Text (IC15) [20]; (5) SVTP-Perspective (SVTP) [38]; (6) CUTE80
(CT80) [44]; (7) COCOText-Validation (COCO) [59]; (8) CTW dataset [30]; (9)
Total-Text dataset (TT) [8]; (10) Occlusion Scene Text (OST) [64], including
weakly occluded scene text (WOST) and heavily occluded scene text (HOST) are
used to evaluate our recognition models; and (11) WordArt dataset [66]. The
detailed descriptions of the benchmarks refer to appendix.

SynthAudio data is generated using the open-source text-to-speech toolkit
PaddleSpeech [69]. Our corpus is directly from the label of public datasets,
including synthetic datasets ST, MJ, real datasets COCO-Text, IIIT5K, SVT,
_etc_. The detailed information of SynthAudio refers to appendix.

###  4.2 Implementation Details

Our model is implemented based on MMOCR [23]222https://github.com/open-
mmlab/mmocr. The model is trained using 4 NVIDIA GTX 3090 GPUs. We choose to
perform experiments on the existing reproduced models on MMOCR, including
MASTER [31], CRNN [45], SAR [26], SATRN [25], Robust_Scanner (Robust_S) [68],
ABINet [11], and ASTER [11], which contains CTC, Attention and Transformer
based recognizers.

For the training dataset, we keep the same configuration as the original
paper. For different models, we have different data configurations, as shown
in Table 1. Unless specified, we set the number of audio decoder layers as 3.
We report the word accuracy metric (%) in all experiments.

###  4.3 Cooperation with Existing Recognizers

As illustrated in Table 1, we validate the benefits of the pluggable AudioOCR
for existing recognizers. Specifically, we evaluate AudioOCR by combining it
with 7 scene text recognition methods on 12 benchmarks including regular,
irregular, and occluded datasets. Equipping with audio decoder methods
achieves consistent improvement compared to baseline methods in average
recognition accuracy. It is worth mentioning that such improvement is free in
the inference stage without introducing any extra costs, while for the
training stage, it only brings an additional 4.6 ms/image on average for
computation overhead as shown in Table 9.

| Method | Img. data | Audio data | Regular | Irregular | Occluded | WordArt | Avg.
---|---|---|---|---|---|---|---|---
| IIIT5K | SVT | IC13 | IC15 | SVTP | CT80 | CTW | TT | COCO | HOST | WOST
CTC | CRNN-paper | MJ | - | 78.2 | 80.8 | 86.7 | - | - | - | - | - | - | - | - | - | -
CRNN* | MJ | - | 80.5 | 81.0 | 86.5 | 54.1 | 59.1 | 55.6 | 54.3 | 52.2 | 37.1 | 30.1 | 45.7 | 33.1 | 55.8
CRNN | MJ | 600k | 82.0 | 81.0 | 88.3 | 57.5 | 65.4 | 60.4 | 55.4 | 53.5 | 37.7 | 36.9 | 53.0 | 38.4 | 59.1
Attention | ASTER-paper | ST+MJ | - | 91.9 | 88.8 | 89.8 | - | 74.1 | 73.3 | - | - | - | - | - | - | -
ASTER* | ST+MJ | - | 92.5 | 89.0 | 91.2 | 73.3 | 80.2 | 82.3 | 69.0 | 69.6 | 52.3 | 40.4 | 61.4 | 58.2 | 71.6
ASTER | ST+MJ | 4m | 93.7 | 89.5 | 93.2 | 74.7 | 81.1 | 86.1 | 69.8 | 71.6 | 53.1 | 47.8 | 65.5 | 60.6 | 73.9
SAR-paper | S+R | - | 95.0 | 91.2 | 94.0 | 78.8 | 86.4 | 89.6 | - | - | - | - | - | - | -
SAR* | S+R | - | 95.0 | 89.6 | 93.7 | 79 | 82.2 | 88.9 | 75.6 | 75.8 | 63.2 | 41.4 | 65.3 | 63.3 | 76.1
SAR | S+R | 4m | 95.5 | 90.9 | 93.5 | 79.3 | 82.6 | 89.9 | 76.0 | 78.9 | 64.3 | 41.2 | 65.4 | 63.4 | 76.7
Robust_S-paper | S+R | - | 95.4 | 89.3 | 94.1 | 79.2 | 82.9 | 89.6 | - | - | - | - | - | - | -
Robust_S* | S+R | - | 95.1 | 89.2 | 93.1 | 77.8 | 80.3 | 90.3 | 75.4 | 75.7 | 62.0 | 38.6 | 65.0 | 61.0 | 75.3
Robust_S | S+R | 2m | 95.1 | 90.6 | 94.1 | 79.6 | 81.4 | 91 | 75.5 | 77.9 | 64.6 | 42.4 | 66.1 | 62.4 | 76.7
Transformer | SATRN-paper | ST+MJ | - | 92.8 | 91.3 | 94.1 | 79 | 86.5 | 87.8 | - | - | - | - | - | - | -
SATRN* | ST+MJ | - | 94.7 | 90.7 | 95.4 | 81.9 | 85.7 | 86.5 | 75.6 | 78.4 | 59.8 | 56.7 | 72.0 | 63.7 | 78.4
SATRN | ST+MJ | 4m | 95.4 | 92.9 | 96.6 | 84.2 | 88.8 | 89.9 | 76.2 | 79.1 | 59.9 | 64.7 | 76.1 | 67.5 | 81.0
MASTER-paper | ST+MJ+SA | - | 95 | 90.6 | 95.3 | 79.4 | 84.5 | 87.5 | - | - | - | - | - | - | -
MASTER* | ST+MJ+SA | - | 95.3 | 89.9 | 95.2 | 77.0 | 83.0 | 89.9 | 74.2 | 77.4 | 54.0 | 52.2 | 72.7 | 65.3 | 77.2
MASTER | ST+MJ+SA | 4m | 95.7 | 91.2 | 95.9 | 79.7 | 84.7 | 89.9 | 75.7 | 80.2 | 55.5 | 52.8 | 74.2 | 66.3 | 78.5
ABINet-LV-paper | ST+MJ | - | 96.2 | 93.5 | 97.4 | 86 | 89.3 | 89.2 | - | - | - | - | - | - | -
ABINet-LV* | ST+MJ | - | 95.7 | 94.6 | 95.7 | 85.1 | 89.3 | 90.3 | 76.7 | 81.2 | 63.7 | 62.8 | 76.5 | 66.7 | 81.5
ABINet-LV | ST+MJ | 4m | 95.9 | 94.9 | 97.8 | 86.1 | 89.9 | 91.4 | 76.8 | 82.8 | 64.0 | 63.5 | 76.8 | 66.7 | 82.3
Table 1: The results of cooperating with existing recognizers. * represents our reproduced results. S+R means ST+MJ+SA+real data. k: one thousand, m: one million. For ABINet, we use the same version of IC13 and IC15 as its paper. In each column, the best performance result of every method is shown in bold font, and the global best result is shown with an underline. Method | Layers | Img. data | Audio data | Regular | Irregular | Occluded | WordArt | Avg.
---|---|---|---|---|---|---|---|---
IIIT5K | SVT | IC13 | IC15 | SVTP | CT80 | CTW | TT | COCO | HOST | WOST
CRNN | 1 | MJ | 600k | 81.3 | 79.3 | 86.8 | 57.6 | 62.9 | 60.4 | 55.5 | 51.0 | 37.3 | 33.7 | 48.1 | 37.0 | 57.6
CRNN | 2 | MJ | 600k | 81.0 | 80.2 | 87.8 | 56.7 | 62.5 | 60.1 | 54.7 | 52.1 | 38.0 | 33.3 | 48.6 | 36.4 | 57.6
CRNN | 3 | MJ | 600k | 82.0 | 81.0 | 88.3 | 57.5 | 65.4 | 60.4 | 55.4 | 53.5 | 37.7 | 36.9 | 53.0 | 38.4 | 59.1
CRNN | 4 | MJ | 600k | 81.8 | 80.5 | 86.7 | 57.3 | 62.6 | 60.4 | 53.8 | 52.1 | 38.2 | 34.5 | 47.9 | 37.3 | 57.8
ASTER | 1 | ST+MJ | 600k | 93.3 | 88.7 | 93.0 | 73.9 | 80.0 | 83.7 | 69.1 | 70.3 | 52.7 | 46.34 | 65.0 | 59.6 | 73.0
ASTER | 2 | ST+MJ | 600k | 93.4 | 87.3 | 92.0 | 73.5 | 79.1 | 84.7 | 68.8 | 70.0 | 52.3 | 47.0 | 65.2 | 60.2 | 72.8
ASTER | 3 | ST+MJ | 600k | 93.6 | 89.0 | 93.0 | 74.2 | 80.0 | 85.4 | 69.7 | 71.0 | 53.1 | 47.1 | 65.4 | 60.2 | 73.5
ASTER | 4 | ST+MJ | 600k | 93.5 | 88.6 | 93.0 | 73.4 | 79.8 | 84.0 | 69.7 | 70.7 | 52.2 | 47.0 | 65.3 | 60.1 | 73.1
ABINet | 1 | ST+MJ | 4m | 95.8 | 94.7 | 97.3 | 85.8 | 89.6 | 90.3 | 76.9 | 81.3 | 63 | 62.3 | 76.3 | 66.7 | 81.6
ABINet | 2 | ST+MJ | 4m | 95.9 | 94.6 | 97.8 | 85.9 | 89.5 | 90.3 | 76.7 | 81.2 | 63.7 | 62.5 | 76.4 | 65.9 | 81.7
ABINet | 3 | ST+MJ | 4m | 95.9 | 94.9 | 97.8 | 86.1 | 89.9 | 91.4 | 76.8 | 82.8 | 64 | 63.5 | 76.8 | 66.7 | 82.2
ABINet | 4 | ST+MJ | 4m | 95.4 | 94.6 | 97.1 | 85.7 | 89.6 | 91 | 76.3 | 81.1 | 64.7 | 63.4 | 76.4 | 65.7 | 81.75
Table 2: Ablation study of the depth of the audio decoder layer. Img. and Avg. are short for image and average, respectively. In each column, the best performance result of every method is shown in bold font. Method | Speaker | Img. data | Audio data | Regular | Irregular | Occluded | WordArt | Avg.
---|---|---|---|---|---|---|---|---
IIIT5K | SVT | IC13 | IC15 | SVTP | CT80 | CTW | TT | COCO | HOST | WOST
CRNN | Female, British | MJ | 600k | 82.0 | 81.0 | 88.3 | 57.5 | 65.4 | 60.4 | 55.4 | 53.5 | 37.7 | 36.9 | 53.0 | 38.4 | 59.1
CRNN | Male, British | MJ | 600k | 81.8 | 79.9 | 88.4 | 57.0 | 62.6 | 63.1 | 54.9 | 52.3 | 37.3 | 33.9 | 47.6 | 36.7 | 58.0
CRNN | Male, American | MJ | 600k | 81.4 | 79.3 | 87.0 | 55.6 | 64.0 | 58.6 | 55.3 | 52.4 | 37.5 | 33.2 | 48.8 | 36.6 | 57.5
CRNN | Female, American | MJ | 600k | 82.2 | 80.4 | 88.7 | 57.1 | 65.5 | 62.5 | 54.7 | 53.6 | 37.8 | 36.4 | 52.6 | 38.1 | 59.1
ASTER | Female, British | ST+MJ | 600k | 93.6 | 89.0 | 93.0 | 74.2 | 80.0 | 85.4 | 69.7 | 71.0 | 53.1 | 47.1 | 65.4 | 60.2 | 73.5
ASTER | Male, British | ST+MJ | 600k | 93.5 | 87.1 | 92.3 | 73.9 | 77.5 | 85.3 | 69.2 | 70.3 | 52.8 | 47.1 | 64.6 | 60.1 | 72.8
ASTER | Male, American | ST+MJ | 600k | 93.7 | 87.3 | 93.0 | 73.1 | 79.2 | 85.0 | 69.2 | 69.6 | 52.3 | 46.4 | 64.6 | 60.1 | 72.8
ASTER | Female, American | ST+MJ | 600k | 93.7 | 88.1 | 92.7 | 73.2 | 79.3 | 84.0 | 69.3 | 70.6 | 52.3 | 47.0 | 65.4 | 60.2 | 73.0
ABINet | Female, British | ST+MJ | 4m | 95.9 | 94.9 | 97.8 | 86.1 | 89.9 | 91.4 | 76.8 | 82.8 | 64.0 | 63.5 | 76.8 | 66.7 | 82.3
ABINet | Male, British | ST+MJ | 4m | 95.7 | 94.7 | 95.6 | 84.4 | 89.6 | 90.0 | 77.9 | 84.2 | 65.2 | 61.6 | 76.2 | 66.4 | 81.8
ABINet | Male, American | ST+MJ | 4m | 95.7 | 94.6 | 95.9 | 84.8 | 89.5 | 90.6 | 77.8 | 84.3 | 64.9 | 61.6 | 76.4 | 66.8 | 82.0
ABINet | Female, American | ST+MJ | 4m | 95.6 | 94.2 | 95.7 | 84.9 | 89.9 | 90.9 | 78.1 | 84.5 | 64.8 | 61.5 | 76 | 65.5 | 81.8
Table 3: Ablation study of using 4 different accents for the audio information. Img. and Avg. are short for image and average, respectively. In each column, the best performance result of every method is shown in bold font. Method | Img. data | Audio data | Regular | Irregular | Occluded | WordArt | Avg.
---|---|---|---|---|---|---|---
IIIT5K | SVT | IC13 | IC15 | SVTP | CT80 | CTW | TT | COCO | HOST | WOST
CRNN | MJ | 100k | 81.0 | 79.8 | 87.4 | 55.3 | 62.1 | 57.6 | 54.6 | 52.1 | 37.2 | 31.1 | 46.8 | 35.2 | 56.7
CRNN | MJ | 600k | 82.0 | 81.0 | 88.3 | 57.5 | 65.4 | 60.4 | 55.4 | 53.5 | 37.7 | 36.9 | 53.0 | 38.4 | 59.1
CRNN | MJ | 4m | 81.9 | 81.2 | 88.2 | 57.4 | 64.5 | 62.2 | 55.3 | 53.4 | 37.8 | 36.7 | 52.2 | 39.5 | 59.2
ASTER | ST+MJ | 100k | 93.2 | 88.2 | 93.1 | 73.9 | 79.3 | 82.9 | 69.2 | 70.8 | 53.0 | 44.8 | 64.9 | 59.4 | 72.7
ASTER | ST+MJ | 600k | 93.6 | 89.0 | 93.0 | 74.2 | 80.0 | 85.4 | 69.7 | 71.0 | 53.1 | 47.1 | 65.4 | 60.2 | 73.5
ASTER | ST+MJ | 4m | 93.7 | 89.5 | 93.2 | 74.7 | 81.1 | 86.1 | 69.8 | 71.6 | 53.1 | 47.8 | 65.5 | 60.6 | 73.9
ABINet | ST+MJ | 100k | 95.7 | 94.5 | 97.2 | 85.7 | 88.4 | 90.3 | 76.4 | 81.1 | 63.1 | 62.2 | 76.1 | 66.1 | 81.4
ABINet | ST+MJ | 600k | 96.1 | 94.5 | 97.8 | 86.0 | 88.9 | 90.3 | 76.4 | 81.1 | 63.7 | 61.8 | 76.9 | 67.6 | 81.8
ABINet | ST+MJ | 4m | 95.9 | 94.9 | 97.8 | 86.1 | 89.9 | 91.4 | 76.8 | 82.8 | 64.0 | 63.5 | 76.8 | 66.7 | 82.3
Table 4: Ablation study of the number of the audio data. Img. and Avg. are short for image and average, respectively. In each column, the best performance result of every method is shown in bold font. Method | Label type | Img. data | Audio data | Regular | Irregular | Occluded | WordArt | Avg.
---|---|---|---|---|---|---|---|---
IIIT5K | SVT | IC13 | IC15 | SVTP | CT80 | CTW | TT | COCO | HOST | WOST
CRNN | Mel | MJ | 600k | 82.0 | 81.0 | 88.3 | 57.5 | 65.4 | 60.4 | 55.4 | 53.5 | 37.7 | 36.9 | 53.0 | 38.4 | 59.1
CRNN | Linear | MJ | 600k | 80.7 | 80.1 | 87.0 | 56.3 | 62.2 | 56.6 | 54.2 | 51.8 | 37.6 | 32.6 | 48.6 | 36.1 | 57.0
ASTER | Mel | ST+MJ | 600k | 93.6 | 89.0 | 93.0 | 74.2 | 80.0 | 85.4 | 69.7 | 71.0 | 53.1 | 47.1 | 65.4 | 60.2 | 73.5
ASTER | Linear | ST+MJ | 600k | 93.6 | 88.0 | 92.7 | 73.4 | 78.5 | 83.7 | 68.8 | 70.5 | 52.4 | 46.2 | 65.3 | 59.6 | 72.7
ABINet | Mel | ST+MJ | 600k | 95.9 | 94.9 | 97.8 | 86.1 | 89.9 | 91.4 | 76.8 | 82.8 | 64 | 63.5 | 76.8 | 66.7 | 82.2
ABINet | Linear | ST+MJ | 600k | 95.7 | 94.8 | 96.9 | 85.8 | 89.3 | 90.5 | 76.2 | 81.2 | 63.8 | 63.1 | 75.6 | 66.7 | 81.6
Table 5: Ablation study of the spectrogram format. Img. and Avg. are short for
image and average, respectively. In each column, the best performance result
of every method is shown in bold font.

###  4.4 Ablation Studies

The number of the audio decoder layers. To study the relationship between the
performance and the ability of speech information extraction, we compare the
result of models which are implemented with a different number of audio
decoder layers $N$. As shown in Table 2, we can see that $N=3$ gets the best
performance, and the performance of $N=4$ decreases a lot compared to $N=3$.

The influence of the accents. To examine the influence of the accents, we
compare the performance of four different settings: American female accent,
American male accent, British female accent, and British male accent. As shown
in Table 3, the guidance by the American female and British female audio
achieves the best for CRNN, with 3.33% higher than the baseline method in
terms of the average accuracy. For ASTER, the best setting is using the
British female audio, which is 1.9% higher than the baseline ASTER. Note that
for both methods and genders, the American and British accents bring similar
gains for the baseline method, while the female accent is clearly more
beneficial to the recognizers than the male accent. By analyzing the sound, we
find that the female voice is clearer while the male voice is deeper.
Reflected in the waveforms, as shown in Figure 5, the female waveform is
smoother, clearly identifiable, and may easier to learn. In the following
experiments, we adopt the British female audio.

The influence of audio data size. To further demonstrate the effectiveness of
audio information, we explore the influence of the audio data volume on text
recognition. As shown in Table 4, the increasing amount of data may gradually
increase the recognition performance.

Figure 5: The waveforms of different speakers. The female waveform is smoother
and has a higher frequency than the male waveform.

The influence of spectrogram format. We perform a group of experiments on the
spectrogram format including linear and mel scale to investigate how it
affects the model performance. We report the results in Table 5, which shows
that our method achieves the best performance under the mel spectrogram
setting compared with linear spectrogram settings on all benchmarks. This is
because the mel spectrogram has better discriminant validity for different
audios than the linear spectrogram, and thus the audio decoder can easily
capture the discrepancy of different text conditioned on the visual features.
To this end, the audio decoder can facilitate better learning of
discriminatory content by the image encoder.

Figure 6: Failure cases of our methods. GT and Pred. represent the ground-
truth and the predicted text, respectively.

Input Images | Audio-guided MASTER | MASTER | GT
---|---|---|---
|  SIMENS |  SHEIVENS | SIMENS
|  Welcome |  Welcerne | Welcome
|  equip |  coulf | equip
|  VISA |  VINA | VISA

Figure 7: Recognition result samples. Green and red are correct and wrong
predictions, respectively. More results are in appendix.

How useful on large-scale real data. As shown in Table 6, based on the large-
scale real data TextOCR [49], using 690k audio data of TextOCR can improve
CRNN and ABINet from 56.0% to 59.5% and 78.2% to 79.3%, respectively,
demonstrating the effectiveness when training on real data.

Usefulness on sentence-level data vs. language modeling methods. Despite the
advantages of language modeling, our method provides further enhancement. We
examined this using the line-level SROIE [18] dataset and ABINet that
possesses language modeling capabilities. Incorporating audio into ABINet
increased performance by 5.4% over the original method (92.3% vs. 86.9%),
demonstrating its effectiveness for text recognition when handling more
sentences for actually text recognition.

Usefulness vs. multiple decoders methods. We assessed the performance of audio
decoder against the use of ensemble decoders (CTC+Attention+Transformer)
during the training stage. For a fair comparison, the parameters of these
additional decoders were matched with those of the audio decoder.
Subsequently, inference was carried out using one decoder with MASTER as the
baseline. The average accuracy are 78.5% and 77.7%, respectively. This
demonstrates the superiority of the proposed audio decoder over using multiple
decoders.

|  |  | Regular | Irregular | Occluded |  |  |
---|---|---|---|---|---|---|---|---
Method | Img. ddata | Audio data | IIIT5K | SVT | IC13 | IC15 | SVTP | CT80 | CTW | TT | COCO | HOST | WOST | WordArt | TextOCR | Avg.
CRNN | TextOCR | - | 77.4 | 79.3 | 88.1 | 58.1 | 62.2 | 61.8 | 54.9 | 52.6 | 41.7 | 23 | 41.3 | 36.3 | 51.5 | 56.0
CRNN | TextOCR | 690k | 78.9 | 81.9 | 90.4 | 62.1 | 66.4 | 66.7 | 56.8 | 54.6 | 43.4 | 29.3 | 48.8 | 41.2 | 53.2 | 59.5
ABINet | TextOCR | - | 85.4 | 94 | 93.6 | 80.3 | 90.1 | 90.6 | 71.3 | 79.7 | 63.7 | 60 | 72.6 | 66 | 70 | 78.2
ABINet | TextOCR | 690k | 86.03 | 94.1 | 95.4 | 81.1 | 89.8 | 92.1 | 72.1 | 80.5 | 65.2 | 61.26 | 75.04 | 68.2 | 70.5 | 79.3
Table 6: Ablation study of training on real data. Img. and Avg. are short for
image and average, respectively. In each column, the best performance result
of every method is shown in bold font. Figure 8: Heat map of image features.
Best view in screen.

Ablation study on non-English data. Our audio-guided text recognition method,
primarily designed and evaluated for English scenarios, has been tested for
universality with non-English data as well. Specifically, we selected datasets
of images containing scene text in Latin German from MLT19 [35], and Chinese
from ReCTS [29]. These German and Chinese datasets comprise diverse
pronunciations. We trained CRNN and ABINet on the German and Chinese datasets
using the same procedure as for the English dataset. Table 7 shows that our
audio-guided method can improve the performance of existing methods by
leveraging audio features for multilingual text recognition.

Method | Audio data | MLT19-German | ReCTS-Chinese
---|---|---|---
CRNN | - | 70.6 | 63.1
CRNN | 600k | 71.7 | 64.8
ABINet-LV | - | 74.8 | 81.5
ABINet-LV | 60k | 75.6 | 82.6
Table 7: Ablation study on non-English data.

Ablation study on out-of-vocabulary words. Table 8 presents our investigation
into the generalization capacity of the audio-guided method for out-of-
vocabulary (OOV) data. We designed our OOV experimental setup to ensure that
the words in the test set do not exist in the training data, thus facilitating
an exploration of the method’s generalization potential. Specifically, we
selected French and German words from MLT19, composed solely of alphabetic
characters, as our test set to guarantee they do not overlap with the English
words in the training set. We utilized existing models trained on English
datasets to directly test this set. The results demonstrated consistent
improvements offered by AudioOCR. The conducted experiments provide empirical
evidence supporting our method’s capacity to generalize to OOV data. It’s
suggested that the model might learn pronunciation rules of vowels and
consonants from the training data, enabling it to infer unseen words.
Moreover, these findings suggest the feasibility of employing minimal audio
data for real-world application training, thereby eliminating the need to
cover all English words.

Method | French (3,883 imgs) | German (6,916 imgs)
---|---|---
CRNN | 70.9 | 76.1
CRNN+Audio | 72.2 | 76.5
ASTER | 77.8 | 79.9
ASTER+Audio | 78.3 | 82.2
ABINet-LV | 85.4 | 82.2
ABINet-LV+Audio | 86.1 | 82.5
Table 8: Ablation study on out-of-vocabulary words.

Training cost of the proposed auxiliary task. For the training stage of
baseline models, as shown in Table 9, using our method introduces an
additional 19.4M, 0.44G, and 4.6 ms/image on average, respectively. It is
valuable for real applications, since this auxiliary loss is only added during
training and the training cost is acceptable, and no additional cost is
incurred during the inference phase.

Model | Params(M) | FlOPs(G) | ms/image | Acc. (%)
---|---|---|---|---
MASTER | 58.98 | 15.04 | 122.3 | 77.2
MASTER+Audio | 78.42 | 15.47 | 125.8 | 78.5
ABINet-LV | 36.74 | 5.94 | 21.8 | 81.5
ABINet-LV+Audio | 56.17 | 6.39 | 27.5 | 82.3
Table 9: The training cost of the proposed auxiliary task.

Analyze the impact of common syllables. We conducted additional experiments
with a dataset containing 45 common English prefixes and suffixes used in
scene text collected from TT and CTW. We compared the performance of the
ABINet method with and without audio. The results showed a 1.5% performance
improvement with the addition of audio. This indicates that the audio
information, particularly the presence of commonly used syllables, contributes
to the enhancement of recognition performance.

Qualitative Analysis. As shown in Figure 7, compared to the baseline methods,
audio-guided methods show more robust performance for complicated cases such
as blurry or flipping images, distorted, occluded, curved, or art font text.
We deduce this may be contributed by the spectrogram of corresponding text
features in the frequency domain, which somewhat makes the visual
representations more robust. We further visualize failure cases in Figure 6,
from which we find the audio-guided recognizers tend to predict phonetically
legal words even if the images missing partly characters.

How audio helps improve accuracy? The use of audio information during training
enhances the focus of the image encoder on the text areas, thereby boosting
its ability to capture subtle visual signals. This is illustrated in Figure 8,
where the heat map of the image features highlights larger text areas in both
English and Chinese scenarios, even if Chinese has predominantly monosyllabic
characters. This audio-visual fusion fortifies the discriminative capability
of the image encoder, thereby improving text recognition accuracy. While not
involved in inference, the encoder still uses training-derived information to
extract relevant features. Hence, the audio decoder aids the recognition
process by enriching the encoder with implicit text information.

##  5 Conclusion

In this work, we propose a simple yet effective plug-and-play module AudioOCR.
To exploit the audio knowledge, we design a visual-audio decoder implemented
by a transformer to excavate cross-modal information between visual and audio
features for learning effective scene text visual representations. It can be
easily connected to existing methods during the training phase meanwhile
freely improve the performance of recognizers without extra cost in the
inference stage. We evaluate the AudioOCR by combining it with 7 scene text
recognition methods on 12 benchmarks including regular, irregular, and
occluded datasets. The results demonstrate the consistent improvements
contributed by the proposed AudioOCR. Besides, it can also deal with more
challenging problems including non-English text, out-of-vocabulary words, and
text with various accents.

Since the text-to-speech tool, PaddleSpeech, can only handle letters and does
not support special symbols for now. The SynthAudio dataset only contains non-
special symbolic audio. The improvement of AudioOCR may be further improved if
such special symbols can also be guided by the audio, which leaves for future
exploration.

As the audio information is naturally connected to the scene text recognition
information, the successful attempt of using the AudioOCR further sheds a
broad space of cross-modal interaction for scene text recognition.

## References

  * [1] Dario Amodei, Sundaram Ananthanarayanan, Rishita Anubhai, Jin Bai, Eric Battenberg, Carl Case, Jared Casper, Bryan Catanzaro, Jingdong Chen, Mike Chrzanowski, Adam Coates, Gregory Frederick Diamos, Erich Elsen, Jesse Engel, Linxi (Jim) Fan, Christopher Fougner, Awni Y. Hannun, Billy Jun, Tony Xiao Han, Patrick LeGresley, Xiangang Li, Libby Lin, Sharan Narang, A. Ng, Sherjil Ozair, Ryan J. Prenger, Sheng Qian, Jonathan Raiman, Sanjeev Satheesh, David Seetapun, Shubho Sengupta, Anuroop Sriram, Chong-Jun Wang, Yi Wang, Zhiqian Wang, Bo Xiao, Yan Xie, Dani Yogatama, Junni Zhan, and Zhenyao Zhu.  Deep speech 2 : End-to-end speech recognition in english and mandarin.  In ICML, pages 173–182, 2016.
  * [2] Relja Arandjelović and Andrew Zisserman.  Objects that sound.  In ECCV, 2018.
  * [3] Jeonghun Baek, Geewook Kim, Junyeop Lee, Sungrae Park, Dongyoon Han, Sangdoo Yun, Seong Joon Oh, and Hwalsuk Lee.  What is wrong with scene text recognition model comparisons? dataset and model analysis.  In 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 4714–4722, 2019.
  * [4] Alexei Baevski, Henry Zhou, Abdel rahman Mohamed, and Michael Auli.  wav2vec 2.0: A framework for self-supervised learning of speech representations.  In NeurIPS, 2020.
  * [5] Darwin Bautista and Rowel Atienza.  Scene text recognition with permuted autoregressive sequence models.  In ECCV, 2022.
  * [6] Otavio Braga and Olivier Siohan.  Best of both worlds: Multi-task audio-visual automatic speech recognition and active speaker detection.  ICASSP 2022 - 2022 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6047–6051, 2022.
  * [7] Ying Cheng, Ruize Wang, Zhihao Pan, Rui Feng, and Yuejie Zhang.  Look, listen, and attend: Co-attention network for self-supervised audio-visual representation learning.  Proceedings of the 28th ACM International Conference on Multimedia, 2020.
  * [8] Chee-Kheng Chng and Chee Seng Chan.  Total-text: A comprehensive dataset for scene text detection and recognition.  In 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR), volume 01, pages 935–942, 2017.
  * [9] Jan Chorowski, Dzmitry Bahdanau, Dmitriy Serdyuk, Kyunghyun Cho, and Yoshua Bengio.  Attention-based models for speech recognition.  In NeurIPS, 2015.
  * [10] Joon Son Chung, Andrew W. Senior, Oriol Vinyals, and Andrew Zisserman.  Lip reading sentences in the wild.  In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 3444–3453, 2016.
  * [11] Shancheng Fang, Hongtao Xie, Yuxin Wang, Zhendong Mao, and Yongdong Zhang.  Read like humans: Autonomous, bidirectional and iterative language modeling for scene text recognition.  In CVPR, pages 7098–7107, 2021.
  * [12] Shancheng Fang, Hongtao Xie, Zheng-Jun Zha, Nannan Sun, Jianlong Tan, and Yongdong Zhang.  Attention and language ensemble for scene text recognition with convolutional sequence modeling.  In Proceedings of the 26th ACM International Conference on Multimedia, MM ’18, page 248–256, New York, NY, USA, 2018. Association for Computing Machinery.
  * [13] Alex Graves, Santiago Fernández, Faustino Gomez, and Jürgen Schmidhuber.  Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks.  In International Conference on Machine Learning, pages 369–376, 2006.
  * [14] Ankush Gupta, Andrea Vedaldi, and Andrew Zisserman.  Synthetic data for text localisation in natural images.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2315–2324, 2016.
  * [15] Awni Y. Hannun, Carl Case, Jared Casper, Bryan Catanzaro, Gregory Frederick Diamos, Erich Elsen, Ryan J. Prenger, Sanjeev Satheesh, Shubho Sengupta, Adam Coates, and A. Ng.  Deep speech: Scaling up end-to-end speech recognition.  ArXiv, abs/1412.5567, 2014.
  * [16] Kaiming He, X. Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pages 770–778, 2016.
  * [17] Pan He, Weilin Huang, Yu Qiao, Chen Change Loy, and Xiaoou Tang.  Reading scene text in deep convolutional sequences.  In AAAI, pages 3501–3508, 2016.
  * [18] Zheng Huang, Kai Chen, Jianhua He, Xiang Bai, Dimosthenis Karatzas, Shijian Lu, and CV Jawahar.  Icdar2019 competition on scanned receipt ocr and information extraction.  In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1516–1520. IEEE, 2019.
  * [19] Max Jaderberg, Karen Simonyan, Andrea Vedaldi, and Andrew Zisserman.  Synthetic data and artificial neural networks for natural scene text recognition.  In NeurIPS Workshop, 2014.
  * [20] Dimosthenis Karatzas, Lluís Gómez i Bigorda, Anguelos Nicolaou, Suman K. Ghosh, Andrew D. Bagdanov, M. Iwamura, Jiri Matas, Lukás Neumann, Vijay Ramaseshan Chandrasekhar, Shijian Lu, Faisal Shafait, Seiichi Uchida, and Ernest Valveny.  Icdar 2015 competition on robust reading.  In 2015 13th International Conference on Document Analysis and Recognition (ICDAR), pages 1156–1160, 2015.
  * [21] Dimosthenis Karatzas, Faisal Shafait, Seiichi Uchida, M. Iwamura, Lluís Gómez i Bigorda, Sergi Robles Mestre, Joan Mas Romeu, David Fernández Mota, Jon Almazán, and Lluís-Pere de las Heras.  Icdar 2013 robust reading competition.  In 2013 12th International Conference on Document Analysis and Recognition, pages 1484–1493, 2013.
  * [22] Jungil Kong, Jaehyeon Kim, and Jaekyoung Bae.  Hifi-gan: Generative adversarial networks for efficient and high fidelity speech synthesis.  In NeurIPS, page 17022–17033, 2020.
  * [23] Zhanghui Kuang, Hongbin Sun, Zhizhong Li, Xiaoyu Yue, Tsui Hin Lin, Jianyong Chen, Huaqiang Wei, Yiqin Zhu, Tong Gao, Wenwei Zhang, Kai Chen, Wayne Zhang, and Dahua Lin.  Mmocr: A comprehensive toolbox for text detection, recognition and understanding.  In Proceedings of the 29th ACM International Conference on Multimedia, 2021.
  * [24] Chen-Yu Lee and Simon Osindero.  Recursive recurrent nets with attention modeling for ocr in the wild.  In Computer Vision and Pattern Recognition, pages 2231–2239, 2016\.
  * [25] Junyeop Lee, Sungrae Park, Jeonghun Baek, Seong Joon Oh, Seonghyeon Kim, and Hwalsuk Lee.  On recognizing texts of arbitrary shapes with 2d self-attention.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 2326–2335, 2020.
  * [26] Hui Li, Peng Wang, Chunhua Shen, and Guyu Zhang.  Show, attend and read: A simple and strong baseline for irregular text recognition.  In AAAI, page 8610–8617, 2019.
  * [27] Jinyu Li.  Recent advances in end-to-end automatic speech recognition.  APSIPA Transactions on Signal and Information Processing, 2022.
  * [28] Minghui Liao, Jian Zhang, Zhaoyi Wan, Fengming Xie, Jiajun Liang, Pengyuan Lyu, Cong Yao, and Xiang Bai.  Scene text recognition from two-dimensional perspective.  In AAAI, page 8714–8721, 2019.
  * [29] Xi Liu, Rui Zhang, Yongsheng Zhou, Qianyi Jiang, Qi Song, Nan Li, Kai Zhou, Lei Wang, Dong Wang, Minghui Liao, Mingkun Yang, Xiang Bai, Baoguang Shi, Dimosthenis Karatzas, Shijian Lu, and C. V. Jawahar.  Icdar 2019 robust reading challenge on reading chinese text on signboard.  In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1577–1581, 2019.
  * [30] Yuliang Liu, Lianwen Jin, Shuaitao Zhang, Canjie Luo, and Sheng Zhang.  Curved scene text detection via transverse and longitudinal sequence connection.  Pattern Recognit., 90:337–345, 2019.
  * [31] Ning Lu, Wenwen Yu, Xianbiao Qi, Yihao Chen, Ping Gong, and Rong Xiao.  Master: Multi-aspect non-local network for scene text recognition.  Pattern Recognit., 117:107980, 2021.
  * [32] Anand Mishra, Karteek Alahari, and CV Jawahar.  Top-down and bottom-up cues for scene text recognition.  In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pages 2687–2694. IEEE, 2012.
  * [33] Anand Mishra, Karteek Alahari, and C. V. Jawahar.  Enhancing energy minimization framework for scene text recognition with top-down cues.  Computer Vision and Image Understanding, pages 30–42, 2016.
  * [34] Byeonghu Na, Yoonsik Kim, and Sungrae Park.  Multi-modal text recognition networks: Interactive enhancements between visual and semantic features.  In European Conference on Computer Vision, 2022.
  * [35] Nibal Nayef, Yash Patel, Michal Busta, Pinaki Nath Chowdhury, Dimosthenis Karatzas, Wafa Khlif, Jiri Matas, Umapada Pal, Jean-Christophe Burie, Cheng-Lin Liu, and Jean-Marc Ogier.  Icdar2019 robust reading challenge on multi-lingual scene text detection and recognition — rrc-mlt-2019.  In 2019 International Conference on Document Analysis and Recognition (ICDAR), pages 1582–1587, 2019.
  * [36] Tae-Hyun Oh, Tali Dekel, Changil Kim, Inbar Mosseri, William T. Freeman, Michael Rubinstein, and Wojciech Matusik.  Speech2face: Learning the face behind a voice.  In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 7531–7540, 2019.
  * [37] Dezhi Peng, Xinyu Wang, Yuliang Liu, Jiaxin Zhang, Mingxin Huang, Songxuan Lai, Jing Li, Shenggao Zhu, Dahua Lin, Chunhua Shen, et al.  Spts: single-point text spotting.  In Proceedings of the 30th ACM International Conference on Multimedia, pages 4272–4281, 2022.
  * [38] Trung Quy Phan, Palaiahnakote Shivakumara, Shangxuan Tian, and Chew Lim Tan.  Recognizing text with perspective distortion in natural scenes.  In 2013 IEEE International Conference on Computer Vision, pages 569–576, 2013.
  * [39] Zhi Qiao, Y. Zhou, Dongbao Yang, Yucan Zhou, and Weiping Wang.  Seed: Semantics enhanced encoder-decoder framework for scene text recognition.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13525–13534, 2020.
  * [40] Leyuan Qu, Cornelius Weber, and Stefan Wermter.  Lipsound2: Self-supervised pre-training for lip-to-speech reconstruction and lip reading.  IEEE transactions on neural networks and learning systems, PP, 2021\.
  * [41] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu.  Fastspeech 2: Fast and high-quality end-to-end text to speech.  In ICLR, 2021.
  * [42] Yi Ren, Chenxu Hu, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu.  Fastspeech 2: Fast and high-quality end-to-end text to speech.  In ICLR, 2021.
  * [43] Yi Ren, Yangjun Ruan, Xu Tan, Tao Qin, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu.  Fastspeech: Fast, robust and controllable text to speech.  In NeurIPS, 2019.
  * [44] Anhar Risnumawan, Palaiahnakote Shivakumara, Chee Seng Chan, and Chew Lim Tan.  A robust arbitrary text detection system for natural scene images.  Expert Syst. Appl., 41:8027–8048, 2014.
  * [45] Baoguang Shi, Xiang Bai, and Cong Yao.  An end-to-end trainable neural network for image-based sequence recognition and its application to scene text recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 2298–2304, 2015.
  * [46] Baoguang Shi, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai.  Robust scene text recognition with automatic rectification.  In CVPR, pages 4168–4176, 2016.
  * [47] Baoguang Shi, Mingkun Yang, Xinggang Wang, Pengyuan Lyu, Cong Yao, and Xiang Bai.  Aster: An attentional scene text recognizer with flexible rectification.  IEEE Transactions on Pattern Analysis and Machine Intelligence, pages 2035–2048, 2019.
  * [48] Cunzhao Shi, Chunheng Wang, Baihua Xiao, Song Gao, and Jinlong Hu.  End-to-end scene text recognition using tree-structured models.  Pattern Recognit., 47:2853–2866, 2014.
  * [49] Amanpreet Singh, Guan Pang, Mandy Toh, Jing Huang, Wojciech Galuba, and Tal Hassner.  Textocr: Towards large-scale end-to-end reasoning for arbitrary-shaped scene text.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8802–8812, 2021.
  * [50] Sibo Song, Jianqiang Wan, Zhibo Yang, Jun Tang, Wenqing Cheng, Xiang Bai, and Cong Yao.  Vision-language pre-training for boosting scene text detectors.  In CVPR, 2022.
  * [51] S. S. Stevens, John E. Volkmann, and Edwin B. Newman.  A scale for the measurement of the psychological magnitude pitch.  Journal of the Acoustical Society of America, 8:185–190, 1937.
  * [52] Bolan Su and Shijian Lu.  Accurate recognition of words in scenes without character segmentation using recurrent neural network.  Pattern Recognition, pages 397–405, 2017.
  * [53] Hideyuki Tachibana, Katsuya Uenoyama, and Shunsuke Aihara.  Efficiently trainable text-to-speech system based on deep convolutional networks with guided attention.  In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 4784–4788, 2017.
  * [54] Yew Lee Tan, Adams Wai-Kin Kong, and Jung-Jae Kim.  Pure transformer with integrated experts for scene text recognition.  In ECCV, 2022.
  * [55] Jing Rui Tang, Su Qiao, Benlei Cui, Yuhang Ma, Sheng Zhang, and D. Kanoulas.  You can even annotate text with voice: Transcription-only-supervised text spotting.  In Proceedings of the 30th ACM International Conference on Multimedia, pages 4154–4163, 2022.
  * [56] Yapeng Tian and Chenliang Xu.  Can audio-visual integration strengthen robustness under multimodal attacks?  In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5597–5607, 2021.
  * [57] Arun Balajee Vasudevan, Dengxin Dai, and Luc Van Gool.  Sound and visual representation learning with multiple pretraining tasks.  In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 14596–14606, 2022.
  * [58] Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, volume 30. Curran Associates, Inc., 2017.
  * [59] Andreas Veit, Tomas Matera, Lukás Neumann, Jiri Matas, and Serge J. Belongie.  Coco-text: Dataset and benchmark for text detection and recognition in natural images.  ArXiv, abs/1601.07140, 2016.
  * [60] Zhaoyi Wan, Minghang He, Haoran Chen, Xiang Bai, and Cong Yao.  Textscanner: Reading characters in order for robust scene text recognition.  In AAAI, pages 12120–12127, 2020.
  * [61] P. Wang, Cheng Da, and Cong Yao.  Multi-granularity prediction for scene text recognition.  In European Conference on Computer Vision, 2022.
  * [62] Tianwei Wang, Yuanzhi Zhu, Lianwen Jin, Canjie Luo, Xiaoxue Chen, Y. Wu, Qianying Wang, and Mingxiang Cai.  Decoupled attention network for text recognition.  In AAAI, pages 12216–12224, 2020.
  * [63] Yuxuan Wang, R. J. Skerry-Ryan, Daisy Stanton, Yonghui Wu, Ron J. Weiss, Navdeep Jaitly, Zongheng Yang, Ying Xiao, Z. Chen, Samy Bengio, Quoc V. Le, Yannis Agiomyrgiannakis, Robert A. J. Clark, and Rif A. Saurous.  Tacotron: Towards end-to-end speech synthesis.  In Interspeech, 2017.
  * [64] Yuxin Wang, Hongtao Xie, Shancheng Fang, Jing Wang, Shenggao Zhu, and Yongdong Zhang.  From two to one: A new scene text recognizer with visual language modeling network.  In 2021 IEEE/CVF International Conference on Computer Vision (ICCV), pages 14174–14183, 2021.
  * [65] Peng Wu, Jing Liu, Yujiao Shi, Yujia Sun, Fang Shao, Zhaoyang Wu, and Zhiwei Yang.  Not only look, but also listen: Learning multimodal violence detection under weak supervision.  In ECCV, 2020.
  * [66] Xudong Xie, Ling Fu, Zhifei Zhang, Zhaowen Wang, and Xiang Bai.  Toward understanding wordart: Corner-guided transformer for scene text recognition.  In ECCV, pages 303–323, 2022.
  * [67] Deli Yu, Xuan Li, Chengquan Zhang, Junyu Han, Jingtuo Liu, and Errui Ding.  Towards accurate scene text recognition with semantic reasoning networks.  In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12110–12119, 2020.
  * [68] Xiaoyu Yue, Zhanghui Kuang, Chenhao Lin, Hongbin Sun, and Wayne Zhang.  Robustscanner: Dynamically enhancing positional clues for robust text recognition.  In European Conference on Computer Vision, pages 135–151, 2020\.
  * [69] Hui Zhang, Tian Yuan, Junkun Chen, and et al.  Paddlespeech: An easy-to-use all-in-one speech toolkit.  In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies: Demonstrations, page 114–123. Association for Computational Linguistics, 2022.

##  Appendix A Appendix

###  A.1 Datasets

MJSynth (MJ) is the synthetic text dataset proposed in [19] a.k.a. Syn90k. The
dataset has 9 million images generated from a set of 90k common English words.
Every image is annotated with word-level ground-truth. We randomly generate
partial corresponding audio according to the text of the images.

SynthText (ST) [14] is a synthetic text dataset originally introduced for text
detection. The generating procedure is similar to [19], and the words are
rendered onto a full image with a large resolution. 800 thousand full images
are used as background images, and usually, each rendered image contains
around 10 text lines. It is also widely used for scene text recognition via
cropped images. We randomly generated partial corresponding audio according to
the text of the images.

SynAdd (SA) is the synthetic text dataset proposed in [26]. The dataset
contains 1.6 million word images using the synthetic engine proposed by [19]
to compensate for the lack of special characters like punctuations.

The Details of Benchmarks. The detailed descriptions of the benchmarks we used
are briefly shown in Table 10.

ID | Benchmarks | Test images | Description
---|---|---|---
#1 | IIIT5K | 3,000 | regular scene text
#2 | SVT | 647 | regular scene text
#3 | IC13 | 1,015 | regular scene text
#4 | IC15 | 1,811 | incidental scene text
#5 | SVTP | 645 | perspective scene text
#6 | CT80 | 288 | irregular scene text
#7 | CTW | 1,572 | irregular scene text
#8 | TT | 2,201 | irregular scene text
#9 | COCO | 9,896 | incidental scene text
#10 | WOST | 2,416 | weakly occluded scene text
#11 | HOST | 2,416 | heavily occluded scene text
#12 | WordArt | 1,511 | wordart scene text
Table 10: Details of the scene text recognition benchmarks.

The Details of SynthAudio. The detailed information of the SynthAudio can be
found in Table 11.

ID | Text Source | Audio Num | Audio Type
---|---|---|---
#1 | MJ | 600,000 | Female, British
#2 | MJ | 600,000 | Female, American
#3 | MJ | 600,000 | Male, British
#4 | MJ | 600,000 | Male, American
#5 | MJ | 4,000,000 | Female, British
#6 | ST | 4,000,000 | Female, British
#7 | COCO | 42,142 | Female, British
#8 | IC11 | 3,567 | Female, British
#9 | IC13 | 848 | Female, British
#10 | IC15 | 4,468 | Female, British
#11 | IIIT5K | 2,000 | Female, British
#12 | TextOCR | 690,000 | Female, British
#13 | MLT19 | 600,000 | Female, German
#14 | ReCTS | 600,000 | Female, Chinese
Table 11: Details of the generated SynthAudio data. British and American
represent two different accents.

###  A.2 Implementation Details

The Details of English and Chinese Audio Generation. We use PaddleSpeech
[69]333https://github.com/PaddlePaddle/PaddleSpeech engine to convert the
English and Chinese text labels to the audio file via neural network based
text to speech (TTS) models. The procedure of TTS models first generates mel
spectrograms from text sequences using an acoustic model, and then synthesizes
speech from the generated mel spectrograms using a separately trained vocoder.
The configurations for audio generation are shown in Table 12. Specifically,
we chose Fastspeech2 [42] as our acoustic model, which can remarkably promote
the speed of generating audio data. The acoustic model can represent the
relationship between an audio signal and the phonemes or other linguistic
units that make up speech. The vocoder is used to convert mel spectrograms to
an audio waveform. We chose HiFiGAN [22] to be our vocoder since it is
commonly used in academic and industrial scenarios. The text corpus is in
English and directly from existing synthesized datasets like MJ [19], ST [14],
and real data including COCO-Text, IC11, IC13, IC15, IIIT5K, TextOCR, and
ReCTS-Chinese [29], which are commonly served as training data for scene text
recognition. The generated audio will be used to generate the mel spectrogram
$\mathbf{Z}$.

The Details of German Audio Generation. Because PaddleSpeech does not support
German audio generation, we use Google Text-to-Speech API engine to convert
the German text label to the audio file including MLT19-German [35]. The
configuration of the Google Text-to-Speech API is depicted in Table 13.

The Details of Mel Spectrogram Generation. The generation configurations of
the mel spectrogram $\mathbf{Z}$ are shown in Table 14. The sample ratio
reflects the frequency we sample from the original audio signal, a higher
sample ratio tends to deliver a better-quality audio reproduction. Here we set
it to 22050, which means we sample 22050 times in one second. For the window
function in Eq. 10, we set the window stride to 275.625 (ms) and the window
length $W$ to 1102.5 (ms), which decides how many times we compute the Fast
Fourier Transform (FFT) on the origin audio signal, and the dimension of FFT
is set to 2048. The dimension of the mel spectrogram $F$ is set to 80. The
higher the dimension of the mel spectrogram is, the more information can be
learned from the audio data, but it also brings extra computation costs.

$\mathtt{STFT}\\{\bm{x}\\}(f,l)=\sum_{t=0}^{T}\bm{x}_{t}\cdot\bm{w}_{t-l}\cdot\mathtt{exp}\left(-i\frac{2\pi
t}{T}f\right)\,.$ (10)

Parameters | Configuration
---|---
Acoustic Model | Fastspeech2
Vocoder | HiFiGAN
Text Corpus | MJ+ST+Real
Language | English/Chinese
Table 12: The configurations of PaddleSpeech for English/Chinese audio generation. Real consists of COCO, IC11, IC13, IC15, IIIT5K, TextOCR, and ReCTS-Chinese [29]. Parameters | Configuration
---|---
Voice Type | Neural2
Language Code | de-DE
Voice Name | de-DE-Neural2-F
SSML Gender | FEMALE
Text Corpus | MLT19-German
Language | German
Table 13: The configurations of Google Text-to-Speech API for German audio generation. Parameters | Configuration
---|---
Sample Ratio (SR) | 22050
Window Stride | 275.625 (ms)
Window Length $W$ | 1102.5 (ms)
Dimension of FFT | 2048
Dimension of Mel Spectrogram $F$ | 80
Table 14: The generation configurations of the mel spectrogram $\mathbf{Z}$.

###  A.3 More Results.

Ablation study for the weight of losses. As shown in Table 15, we vary the
trade-off audio loss weight $\alpha$ ranging in [0.5, 1, 2], observing that
$\alpha=1$ can get the best performance. We guess that $\alpha=2$ may break
the balance between visual information and audio information, and $\alpha=0.5$
results in insufficient training using the audio feature.

Qualitative Analysis. Fig. 9 are more qualitative samples of recognition
results of audio-guided and original methods.

Method | Loss weight | Img. data | Audio data | Regular | Irregular | Occluded | WordArt | Avg.
---|---|---|---|---|---|---|---|---
IIIT5K | SVT | IC13 | IC15 | SVTP | CT80 | CTW | TT | COCO | HOST | WOST
CRNN | 1 | MJ | 600k | 82.0 | 81.0 | 88.3 | 57.5 | 65.4 | 60.4 | 55.4 | 53.5 | 37.7 | 36.9 | 53.0 | 38.4 | 59.1
CRNN | 2 | MJ | 600k | 81.2 | 80.2 | 87.2 | 56.7 | 62.2 | 58.7 | 54.8 | 52.1 | 37.5 | 33.6 | 48.5 | 35.4 | 57.3
CRNN | 0.5 | MJ | 600k | 81.3 | 80.6 | 87.0 | 57.3 | 62.3 | 60.1 | 55.6 | 52.5 | 37.3 | 34.3 | 49.2 | 37.7 | 57.9
ABINet | 1 | ST+MJ | 4m | 95.9 | 94.9 | 97.8 | 86.1 | 89.9 | 91.4 | 76.8 | 82.8 | 64 | 63.5 | 76.8 | 66.7 | 82.2
ABINet | 2 | ST+MJ | 4m | 95.8 | 94.9 | 96 | 85.8 | 89.6 | 90.3 | 76.9 | 81.6 | 63.4 | 63.1 | 76.8 | 66.7 | 81.7
ABINet | 0.5 | ST+MJ | 4m | 95.8 | 94.7 | 96.9 | 85.3 | 89.8 | 90.2 | 76.7 | 81.4 | 63.9 | 62.9 | 76.5 | 66.5 | 81.7
Table 15: Ablation study of loss weight of audio decoder. Img. and Avg. are
short for image and average, respectively. In each column, the best
performance result of every method is shown in bold font.

Input Images | Audio-guided MASTER | MASTER | GT
---|---|---|---
|  DEPARTMENT |  SARTMENT | DEPARTMENT
|  EXTRA |  EXTRN | EXTRA
|  everyone |  eucyone | everyone
|  own |  owy | own
|  island |  isenndd | island
|  alexandra |  alexandia | alexandra
|  WORLD |  140.1221.21 | WORLD

Figure 9: Samples of recognition results of audio-guided and original methods.
Green and red characters represent correct and wrong predictions,
respectively.


[table]capposition=top

# A Construction of Type-II ZCCS of Arbitrary Sequence Length with Low PMEPR

Rajen Kumar, Prashant Kumar Srivastava and Sudhan Majhi

###### Abstract

In this letter, we propose a novel construction of type-II $Z$-complementary
code set (ZCCS) having arbitrary sequence length. The proposed construction
features type-II $(K,K,NP-P+1,NP)$-ZCCS using Kronecker product between
$(K,K,N)$-complete complementary code (CCC) set and unimodular sequence. In
this construction, for the first time, Barker sequences are used to reduce row
sequence peak-to-mean envelope power ratio (PMEPR) for some specific lengths
sequence and column sequence PMEPR for some specific size of codes. The column
sequence PMEPR of the proposed type-II ZCCS is upper bounded by a number
smaller than $2$, which is lower than the bounds reported in existing
literature for such codes. The proposed construction also contributes new
lengths of type-II $Z$-complementary pair (ZCP) and type-II $Z$-complementary
set (ZCS), which are also not reported before in literature. Furthermore, the
PMEPR of the obtained type-II ZCP is also lower than some of the existing
work. The PMEPR of type-II ZCS can also be ensured to be low.

###### Index Terms:

Type-II ZCP, Type-II ZCS, Type-II ZCCS, CCC, PMEPR.

##  I Introduction

Golay, in his pioneering work in $1961$, proposed a pair of sequences,
popularly known as a Golay complementary pair (GCP) [1]. Aperiodic auto-
correlation sum (AACS) of GCPs is zero except at the zero-shift position.
Tseng and Liu [2] extended the idea of GCPs to a complementary set (CS) that
includes two or more constituent sequences with the similar AACS property.
Paterson [3] proposed a construction of CS as a generalization of the
construction by Davis and Jedwab in [4]. Rathinakumar and Chaturvedi [5]
further extended the idea of Paterson [3] to obtain the mutually orthogonal
Golay complementary set (MOGCS).

The sequence construction plays a major role to reduce row sequence peak-to-
mean envelope power ratio (PMEPR) and column sequence PMEPR. In $2020$, a new
class of CS was introduced by Sarkar et al. [6] for multi-carrier code-
division multiple-access (MC-CDMA) where column sequence PMEPR is bounded by
$2$. Recently in $2021$, Kumar et al. [7] proposed a novel construction for
$2^{h}$-ary complete complementary code (CCC) of length in the form of
$10$$\cdot$$2^{m}$ and $26$$\cdot$$2^{m}$, where $h$ and $m$ are positive
integers, where column sequence PMEPR upper bound by $2$.

In $2007$, Fan et al. introduced the concept of zero correlation zone (ZCZ)
width, in their construction on Z-complementary pair (ZCP) where AACSs is zero
within this ZCZ width [8]. Due to the flexibility in lengths, ZCPs have been
widely used over GCPs. ZCPs are known as type-I ZCP, when ZCZ width exists
around zero shift and type-II ZCP, when ZCZ width exists towards end shift
[9]. The construction for type-I ZCPs of even length has been proposed in [10,
11, 12, 13, 14, 15, 16], but optimal ZCZ width could not be achieved.
Similarly, construction for type-II ZCPs has been proposed in [9, 16, 17, 18,
19] where optimal ZCZ width has been achieved for some even lengths. Type-I
and type-II ZCP for some odd length with optimal ZCZ width has been proposed
in [9, 16, 17]. Instead of pair, if a set of sequences follow the property of
zero AACS within ZCZ width, it is known as $Z$-complementary set (ZCS).

A collection of ZCS is called $Z$-complementary code set (ZCCS), if aperiodic
cross-correlation sum (ACCS) of two different ZCS is zero in the ZCZ region
including the zero time shift. The ZCCS is advantageous in the sense that it
includes the CCC as a special case with a larger number of sequence lengths
and set size. Type-I ZCCS refers to a set of codes where the ZCZ exist near
zero time shift, whereas for type-II ZCCS, the ZCZ width exists near end time
shift. When the delay spread in wideband communication exceeds the symbol
time, the reflected components of a signal arrive with a temporal delay to the
receiver. In such scenarios, type-II ZCCS may be more effective than type-I
ZCCS in suppressing inter-symbolic interference (ISI) in wavelet-based MC CDMA
[20]. To the best of the authors’ knowledge, there are no constructions on
type-II ZCS and ZCCS have been reported in the literature.

In this letter, we use the Kronecker product between the CCC and the
unimodular sequence to produce a type-II ZCCS of any arbitrary length. For a
given $(K,K,N)$-CCC, the proposed construction provides a new type-II
$(K,K,NP-P+1,NP)$-ZCCS, for $P>1$. Furthermore, for the fixed value of
constituent sequence, the proposed type-II ZCCS provides larger set size
compared to that of type-I ZCCS. Also, for the first time, we introduce an
idea of the Barker sequences to reduce the PMEPR of a code. The column
sequence PMEPR of type-II ZCCS is upper bounded by a number smaller than $2$,
which is lower than the bounds reported in existing literature for a code. The
proposed construction also deduces type-II ZCPs with more lengths compared to
[18] and [19], and provides lower PMEPR for type-II ZCP compared to [19]. The
proposed construction also contributes type-II ZCS which is also not been
provided and reported in the existing literature.

The rest of the letter is stated as follows. Section II contains the
introduction and notations. Section III describes the proposed construction,
proofs, and example. In section IV, a special case of our construction has
been compared with some of the existing work. Finally, section V includes the
final observations.

##  II Preliminaries

This section introduces the fundamental notations, definitions, and lemmas
utilised in this letter.

###  II-A Basic Notations

######  Definition 1 ([21])

Let $\mathbf{a}=(a_{0},a_{1},\ldots,a_{N-1})$ and
$\mathbf{a^{\prime}}=(a^{\prime}_{0},a^{\prime}_{1},$$\ldots,$$a^{\prime}_{N-1})$
be two complex-valued sequences of length $N$. The ACCS between $\mathbf{a}$
and $\mathbf{a^{\prime}}$ is defined as

$\Lambda(\mathbf{a},\mathbf{a^{\prime}})(\tau)=\left\\{\begin{array}[]{ll}\sum_{i=0}^{N-1-\tau}a_{i+\tau}a_{i}^{{}^{\prime}*},&0\leq\tau<N,\\\
\sum_{i=0}^{N+\tau-1}a_{i}a_{i-\tau}^{{}^{\prime}*},&-N<\tau<0,\\\ 0,&\text{
otherwise, }\end{array}\right.$

where $a_{i}^{{}^{\prime}*}$ is complex conjugate of $a^{\prime}_{i}$. When
$\mathbf{a}=\mathbf{a^{\prime}}$,
$\Lambda(\mathbf{a},\mathbf{a^{\prime}})(\tau)$ is called AACS of $\mathbf{a}$
and is denoted as $\Lambda(\mathbf{a})(\tau)$.

###### Lemma 1

Let $\mathbf{a}$ and $\mathbf{a^{\prime\prime}}$ be two complex-valued
sequences of length $M$, where
$\mathbf{a^{\prime\prime}}=e^{\sqrt{-1}\theta}\mathbf{a}$ and
$\theta\in[0,2\pi)$. Then
$\Lambda(\mathbf{a^{\prime\prime}})(\tau)=\Lambda(\mathbf{a})(\tau).$

###### Definition 2

Let $\mathbf{C}=\left\\{{C}_{1},{C}_{2},\ldots,{C}_{K}\right\\}$ be a set of
$K$ matrices (codes), each having order $M\times N$. Here $C_{k}$ is defined
as

$C_{k}=\left[\begin{array}[]{c}\mathbf{c}_{1}^{k}~{}\mathbf{c}_{2}^{k}~{}\ldots~{}\mathbf{c}_{M}^{k}\end{array}\right]^{T}_{M\times
N},$

where $\mathbf{c}_{\nu}^{k}(1\leq\nu\leq M,1\leq k\leq K)$ is the $\nu$-th row
sequence of $C_{k}$ and $\left[\cdot\right]^{T}$ represents transpose of
matrix $\left[\cdot\right]$.
Let $C_{k_{1}}$ and $C_{k_{2}}$ be any two matrices in $\mathbf{C}$, then ACCS
of $C_{k_{1}}$ and $C_{k_{2}}$ is defined by

$\Lambda\left({C}_{k_{1}},{C}_{k_{2}}\right)(\tau)=\sum_{\nu=0}^{M-1}\Lambda\left(\mathbf{c}_{\nu}^{k_{1}},\mathbf{c}_{\nu}^{k_{2}}\right)(\tau).$
(1)

When ${C}_{k_{1}}={C}_{k_{2}}={C}_{k}$, $\Lambda(C_{k_{1}},C_{k_{2}})(\tau)$
is called AACS of ${C}_{k}$ and is denoted as $\Lambda(C_{k})(\tau)$.

###### Definition 3

Let $\mathbf{Z}=\left\\{{Z}_{1},{Z}_{2},\ldots,{Z}_{K}\right\\}$ be a set of
$K$ matrices (codes), each having order $M\times N$. $\mathbf{Z}$ is called a
type-II $(K,M,Z,N)$-ZCCS if it satisfy following properties

$\Lambda\left(Z_{k_{1}},Z_{k_{2}}\right)(\tau)=\left\\{\begin{array}[]{ll}NM,&\tau=0,k_{1}=k_{2},\\\
0,&\tau=0,k_{1}\neq k_{2},\\\ 0,&N-Z<|\tau|<N.\end{array}\right.$

Each code $Z_{k}\in\mathbf{Z}$ is called a type-II ZCS. It is known as type-II
ZCP, when number of rows in a type-II ZCS is $2$. When $Z=N$, a type-II ZCCS
is called MOGCS and when $K=M$, a MOGCS is called $(K,K,N)$-CCC. If ACCS
values of two codes are zero for all time shifts these are called uncorrelated
codes.

######  Lemma 2 ([22])

Let $\mathbf{a}=(a_{0},a_{1},\ldots,a_{N-1})$ and
$\mathbf{a^{\prime}}=(a^{\prime}_{0},a^{\prime}_{1},\ldots,a^{\prime}_{N-1})$
of length $N$ and $\mathbf{b}=(b_{0},b_{1},\ldots,b_{P-1})$ and
$\mathbf{b^{\prime}}=(b^{\prime}_{0},b^{\prime}_{1},\ldots,b^{\prime}_{P-1})$
be unimodular sequences (each element of the sequence is of modulus $1$) of
length $P$. Let $\otimes$ represents Kronecker product. Then
$\mathbf{c}=\mathbf{a}\otimes\mathbf{b}$ and
$\mathbf{c^{\prime}}=\mathbf{a^{\prime}}\otimes\mathbf{b^{\prime}}$ are
sequences of length $NP$ and ACCS of $\mathbf{c}$ and $\mathbf{c}^{\prime}$
depends on ACCS of $(\mathbf{a},\mathbf{a^{\prime}})$ and
$(\mathbf{b},\mathbf{b^{\prime}})$, as

$\displaystyle\Lambda(\mathbf{c,c^{\prime}})(Pj+k)=$
$\displaystyle\Lambda(\mathbf{a,a^{\prime}})(j)\Lambda(\mathbf{b},\mathbf{b^{\prime}})(k)$
(2)
$\displaystyle+\Lambda(\mathbf{a,a^{\prime}})(j+1)\Lambda(\mathbf{b},\mathbf{b^{\prime}})(-P+k),$
$\displaystyle\text{for }0\leq k<P\text{ and }-N<j<N.$

###### Lemma 3

Let $\mathbf{a}=(a_{0},a_{1},\ldots,a_{N-1})$ and
$\mathbf{b}=(b_{0},b_{1},\ldots,b_{P-1})$ be unimodular sequences of length
$N$ and $P$ respectively. Then $\mathbf{c}=\mathbf{a}\otimes\mathbf{b}$ is a
sequence of length $NP$ and AACS of $\mathbf{c}$ depends on AACS of
$\mathbf{a}$ and $\mathbf{b}$.

###### Proof:

In Lemma 2, if we consider $\mathbf{a}=\mathbf{a^{\prime}}$ then
$\mathbf{c}=\mathbf{c^{\prime}}$ and ACCS of $\mathbf{a},\mathbf{a^{\prime}}$
and $\mathbf{c},\mathbf{c^{\prime}}$ become AACS of $\mathbf{a}$ and
$\mathbf{c}$. From Lemma 2, we have

$\displaystyle\Lambda(\mathbf{c})(Pj+k)=$
$\displaystyle\Lambda(\mathbf{a})(j)\Lambda(\mathbf{b})(k)+\Lambda(\mathbf{a})(j+1)\Lambda(\mathbf{b})(-P+k),$
(3) $\displaystyle\text{for }0\leq k<P\text{ and }-N<j<N.$

∎

###  II-B Barker sequence [23]

###### Definition 4

A binary sequence
$\boldsymbol{\gamma}=(\gamma_{0},\gamma_{1},\ldots,\gamma_{N-1})$ of length
$N$ is said to be a Barker sequence if the AACS of the sequence is either $\pm
1$ or $0$, i.e., $|\Lambda(\boldsymbol{\gamma})(\tau)|$ is either $1$ or $0$
for $\tau\neq 0$.

The only lengths $N>1$, for which a binary Barker sequence,
$\boldsymbol{\gamma}$ is known to exist, are $2,3,4,5,7,11$ and $13$. Under
one or more of the following transformations, these Barker sequences are
unique for their length

$\gamma_{i}\mapsto\\{-\gamma_{i},\gamma_{N-1-i},(-1)^{i}\gamma_{i}\\}$ (4)

###  II-C Peak-to-mean envelope power ratio (PMEPR)

######  Lemma 4 ([4])

Let $\mathbf{a}$ be a sequence of length $N$. Then PMEPR of $\mathbf{a}$ is
bounded by

$\text{PMEPR}(\mathbf{a})\leq\frac{1}{N}\sum_{\tau=-(N-1)}^{N-1}|\Lambda(\mathbf{a})(\tau)|.$
(5)

PMEPR of orthogonal frequency-division multiplexing (OFDM) system is
calculated by considering the row of a CS or ZCS, and PMEPR of MC-CDMA system
is calculated by the column of a CS or ZCS. The maximum over the obtained
PMEPR of row or column sequence is known as row or column sequence PMEPR upper
bound of a code, respectively.

##  III Proposed Construction of Type-II ZCCS

In this section, we present our proposed type-II ZCCS construction utilising
Kronecker products. We also explore methods to reduce a code’s PMEPR.

###### Theorem 1

Let $\mathbf{C}=\\{C_{1},C_{2},\ldots,C_{K}\\}$ be a $(K,K,N)$-CCC, where
$C_{i}$ is a code of order $K\times N$ and $\mathbf{b}$ is a unimodular
sequence of length $P$. Then $\mathbf{Z}=\\{Z_{0},Z_{1},\ldots,Z_{K-1}\\}$ is
a type-II $(K,K,NP-P+1,NP)$-ZCCS, where $Z_{i}=C_{i}\otimes\mathbf{b}$.

###### Proof:

Each code of $\mathbf{C}$ follows the property that $\Lambda(C_{i})(\tau)=0$
for non-zero value of $\tau$ and $\Lambda(C_{i},C_{j})(\tau^{\prime})=0$ for
any value of $\tau^{\prime}$ for $i,j\in\\{0,1,\ldots,K-1\\}$. Let

$C_{i}=[\mathbf{c}_{0}^{i}~{}\mathbf{c}_{1}^{i}~{}\ldots~{}\mathbf{c}_{K-1}^{i}]^{T},$

then we define

$Z_{i}=C_{i}\otimes\mathbf{b}=[\mathbf{c}_{1}^{i}\otimes\mathbf{b}~{}\mathbf{c}_{2}^{i}\otimes\mathbf{b}~{}\ldots~{}\mathbf{c}_{K-1}^{i}\otimes\mathbf{b}]^{T}.$

Since $\sum_{k=1}^{K}\Lambda(\mathbf{c}_{k}^{i})(\tau)=0$ for any non-zero
value of $\tau$, for $u\geq P-1$, from Lemma 3, it is straight forward
$\sum_{k=1}^{K}\Lambda(\mathbf{c}_{k}^{i}\otimes\mathbf{b})(u)=0$, i.e.,
$\Lambda(Z_{i})(u)=0$ for $\lvert u\rvert\geq P-1$.
Since
$\sum_{k=1}^{K}\Lambda(\mathbf{c}_{k}^{i}\mathbf{c}_{k}^{j})(\tau^{\prime})=0$
for any integer $\tau^{\prime}$, from Lemma 2, it is straight forward that
$\sum_{k=1}^{K}\Lambda(\mathbf{c}_{k}^{i}\otimes\mathbf{b},\mathbf{c}_{k}^{j}\otimes\mathbf{b})(u)=0$,
i.e., $\Lambda(Z_{i},Z_{j})(u)=0$ for any integral value of $u^{\prime}$.
Therefore,

$\Lambda(Z_{i},Z_{j})(u)=\left\\{\begin{array}[]{ll}0,&P-1<|u|<NP,i=j,\\\
0,&|u|<NP,i\neq j.\end{array}\right.$

∎

###### Remark 1

Let $C$ be a CS of size $M\times N$ and $\mathbf{b}$ be a sequence of length
$P$. From Theorem 1, $Z=C\otimes\mathbf{b}$ be a type-II ZCS of size $M\times
NP$ with ZCZ width $NP-P+1$.

###### Remark 2

Let ($\mathbf{s},\mathbf{t}$) be a GCP of length $N$ and $\mathbf{b}$ be a
sequence of length $P$. From Theorem 1,
($\mathbf{s}\otimes\mathbf{b},\mathbf{t}\otimes\mathbf{b}$) be a type-II ZCP
of length $NP$ with ZCZ width $NP-P+1$.

###### Corollary 1

Suppose ${Z_{\mu}}$ is a code from type-II $(K,K,NP-P+1,NP)$-ZCCS obtained by
Theorem 1, then we can say that $|\Lambda(Z_{\mu})(\tau)|$ is a multiple of
$KN$, where $\mu\in\\{0,1,\ldots,K-1\\}$. When $\tau=0$, we get
$|\Lambda({Z_{\mu}})(\tau)|=KNP$. For $\tau>P-1$, we get
$|\Lambda({Z_{\mu}})(\tau)|=0$, but for $0<\tau\leq P-1$, we get

$\displaystyle\Lambda({Z_{\mu}})(\tau)$
$\displaystyle=\Lambda(\mathbf{b})(\tau)KN,$
$\displaystyle\text{i.e.,~{}}|\Lambda({Z_{\mu}})(\tau)|$
$\displaystyle\leq(P-\tau)KN.$

From Corollary 1 and Lemma 3, the AACS value of the proposed code outside the
ZCZ width is directly proportional to the AACS of the unimodular sequences
$\mathbf{b}$.

From Corollary 1, we can also conclude the following for AACS values out of
ZCZ width :

  * •

$|\Lambda(Z_{\mu})(P-1)|=KN$.

  * •

For $0<\tau<P-1$, $0\leq|\Lambda(Z_{\mu})(\tau)|\leq\tau KN$. For example, let
$\mathbf{b}=(e^{\iota\pi\theta_{0}},e^{\iota\pi\theta_{1}},\ldots,e^{\iota\pi\theta_{P-1}})$,
where $\iota=\sqrt{-1}$, such that
$\theta_{P-2}-\theta_{1}=\theta_{P-1}-\theta_{2}\pm 1$, then
$\Lambda(Z_{\mu})(P-2)=0$. This case is possible if the phase of $\mathbf{b}$
is even. In the case of binary sequence
$\mathbf{b}=(b_{0},b_{1},\ldots,b_{P-1})$, it must follows

$b_{0}+b_{1}+b_{P-1}+b_{P-2}=\pm 2.$ (6)

Since we can make $|\Lambda({Z_{\mu}})(\tau)|=0$ for $\tau=P-2$, so the
obvious question is, can we get $\lvert\Lambda({Z_{\mu}})(\tau)\rvert=0$ for
all $0<\tau<P-2$? The answer is no. To verify the answer, let us investigate
it for $\tau=P-3$. Let us also consider $\Lambda({Z_{\mu}})(P-3)=0$ and
$\Lambda({Z_{\mu}})(P-2)=0$, from Lemma 3, this gives sequence $\mathbf{b}$
needs to be satisfied $\Lambda(\mathbf{b})(P-3)=0$ and
$\Lambda(\mathbf{b})(P-2)=0$. Suppose, the sequence $\mathbf{b}$ follows (6)
and implies $\Lambda(\mathbf{b})(P-2)=0$ then $\Lambda(\mathbf{b})(P-3)$ can
be written as
$e^{\iota\pi(\theta_{P-3}-\theta_{1})}+e^{\iota\pi(\theta_{P-2}-\theta_{2})}+e^{\iota\pi(\theta_{P-1}-\theta_{3})}$,
where $e^{\iota\pi\theta_{i}}$ is of the even phase, value of
$e^{\iota\pi(\theta_{P-3}-\theta_{1})}+e^{\iota\pi(\theta_{P-2}-\theta_{2})}+e^{\iota\pi(\theta_{P-1}-\theta_{3})}$
only be $\pm 1$, for some choice of $\mathbf{b}$. For an example, let a binary
sequence $\mathbf{b}=(b_{0},b_{1},\ldots,b_{P-1})$, be such that

$\displaystyle b_{0}+b_{1}+b_{P-2}+b_{P-1}$ $\displaystyle=\pm 2,$
$\displaystyle b_{0}+b_{1}+b_{P-3}+b_{P-2}$ $\displaystyle=\pm 2,$
$\displaystyle b_{1}+b_{2}+b_{P-2}+b_{P-1}$ $\displaystyle=\pm 2,$

then

$\Lambda(\mathbf{b})(P-2)=0,~{}\Lambda(\mathbf{b})(P-3)=\pm 1.$

Therefore, we can have $|\Lambda({Z_{\mu}})(P-3)|=KN$. For the odd phase, we
cannot make $\Lambda({Z_{\mu}})(P-2)=0$ but if the phase of $\mathbf{b}$ is
multiple of $3$, we can find a sequence $\mathbf{b}$, such that
$\Lambda(\mathbf{b})(P-3)=0$.
From Lemma 4, PMEPR of OFDM system by using Barker sequences of lengths
$3,5,7,11$ and $13$ has upper bound $5/3,9/5,13/7,21/11$ and $25/13$
respectively. PMEPR of a Golay sequence has upper bound $2$.

###### Remark 3

If $\mathbf{b}$ is a Barker sequence then AACS for any code generated by
Theorem 1, is either $\pm KN$ or $0$, outside the ZCZ width.

Now, if we replace $\mathbf{b}$ by Barker sequence in Theorem 1, then the AACS
of $Z_{i}$ can be reduced outside of the ZCZ width. However, as the Barker
sequences are limited in length, the Kronecker product of Barker sequences is
being used. According to Lemma 3, the Kronecker product of Barker sequences
likewise has a low AACS value. With the help of the Kronecker product of
Barker sequences, sequences with low ACCS of lengths in the form of
$2^{n_{1}}3^{n_{2}}5^{n_{3}}7^{n_{4}}11^{n_{5}}13^{n_{6}}$, where
$n_{1},n_{2},\ldots,n_{6}\geq 0$. A type-II $(K,K,NP-P+1,NP)$-ZCCS that are
being constructed, row PMEPR can be decreased if
$P=2^{n_{1}}3^{n_{2}}5^{n_{3}}7^{n_{4}}11^{n_{5}}13^{n_{6}}$, and column
sequence PMEPR can be reduced if a code has
$2^{n_{1}}3^{n_{2}}5^{n_{3}}7^{n_{4}}11^{n_{5}}13^{n_{6}}$ constituent
sequences.

###### Corollary 2

Consider $\mathbf{C}$ be a CCC of set size $3,5,7,11$ or $13$ constructed by
the method in [24]. Using Theorem 1, we have
$\mathbf{Z}=\\{Z_{0},Z_{1},\ldots,Z_{K-1}\\}$ a type-II ZCCS of set size
$3,5,7,11$ or $13$, where
$Z_{i}=[\mathbf{z}_{0}^{i}~{}\mathbf{z}_{1}^{i}~{}\ldots~{}\mathbf{z}_{K-1}^{i}]^{T}$
, for $i\in\\{0,1,\ldots,K-1\\}$. Let
$\boldsymbol{\gamma}=\\{\gamma_{0},\gamma_{1},\ldots,\gamma_{K-1}\\}$ be a
Barker sequence of the same length as number of rows in $\mathbf{Z}_{i}$. We
generate new code set as
$\mathbf{Z^{\prime}}=\\{Z^{\prime}_{0},Z^{\prime}_{1},\ldots,Z^{\prime}_{K-1}\\}$,
where
$Z^{\prime}_{i}=[\gamma_{0}\mathbf{z}_{0}^{i}~{}\gamma_{1}\mathbf{z}_{1}^{i}~{}\ldots~{}\gamma_{K-1}\mathbf{z}_{K-1}^{i}]^{T}$.
Then the column sequence PMEPR of $\mathbf{Z}^{\prime}$ is the same as PMEPR
of $\boldsymbol{\gamma}$.

###### Proof:

Let $\mathbf{C}$ be a CCC constructed using [24], where each column sequence
of $\mathbf{C}$ can be represented by some linear function. Let us construct
$\mathbf{Z^{\prime}}=\\{Z^{\prime}_{0},Z^{\prime}_{1},\ldots,Z^{\prime}_{K-1}\\}$
by using Theorem 1 and this $\mathbf{C}$, where
$Z^{\prime}_{i}=[\gamma_{0}\mathbf{z}_{0}^{i}~{}\gamma_{1}\mathbf{z}_{1}^{i}~{}\ldots~{}\gamma_{K-1}\mathbf{z}_{K-1}^{i}]^{T}$.
Note that, each column of each code in $\mathbf{Z}^{\prime}$ is either
multiplication between $\boldsymbol{\gamma}$ and unimodular constant or
multiplication between transformed $\boldsymbol{\gamma}$ and unimodular
constant, using the transformation (4). Now AACS value of each column sequence
is same as the AACS value of $\boldsymbol{\gamma}$. Therefore, column sequence
PMEPR is the same as PMEPR of $\boldsymbol{\gamma}$. This complete the proof
of Corollary 2. ∎

An example of proposed type-II ZCCS is presented below.

###### Example 1

Let $\mathbf{C}=\\{C_{1},C_{2},C_{3},C_{4}\\}$ is a binary $(4,4,8)$-CCC given
in Table I, where $+$ and $-$ represent $1$ and $-1$, respectively.

$\begin{smallmatrix}-++++++-\\\ \--+-+-++\\\ -+--++-+\\\ +++--+++\end{smallmatrix}$ | $\begin{smallmatrix}--+-+-++\\\ -++++++-\\\ \---++---\\\ +-++--+-\end{smallmatrix}$ | $\begin{smallmatrix}-+--++-+\\\ +++--+++\\\ -++++++-\\\ \--+-+-++\end{smallmatrix}$ | $\begin{smallmatrix}+++--+++\\\ -+--++-+\\\ ++-+-+--\\\ +------+\end{smallmatrix}$
---|---|---|---
TABLE I: Binary CCC

Let $\mathbf{b}=(1,1,-1)$ be a Barker sequence of length $3$. Using Theorem 1,
binary type-II $(4,4,24)$-ZCCS can be obtained as given in the Table II. AACS
of each code from $\mathbf{Z}$ is $(0_{21},-32,0,96,0,-32,0_{21})$ and ACCS of
any two code from $\mathbf{Z}$ is zero for all time shifts. AACS value of each
column sequence is either $(1,0,-1,0,4,0,-1,0,1)$ or $(-1,0,1,0,4,0,1,0,-1)$.
Therefore, column sequence PMEPR is upper bounded by $2$ for this given
example.

$\begin{smallmatrix}--+++-++-++-++-++-++---+\\\ \--+--+++---+++---+++-++-\\\
\--+++---+--+++-++---+++-\\\ ++-++-++---+--+++-++-++-\end{smallmatrix}$
---
$\begin{smallmatrix}--+--+++---+++---+++-++-\\\ \--+++-++-++-++-++-++---+\\\
\--+--+--+++-++---+--+--+\\\ ++---+++-++---+--+++---+\end{smallmatrix}$
$\begin{smallmatrix}--+++---+--+++-++---+++-\\\ ++-++-++---+--+++-++-++-\\\
\--+++-++-++-++-++-++---+\\\ \--+--+++---+++---+++-++-\end{smallmatrix}$
$\begin{smallmatrix}++-++-++---+--+++-++-++-\\\ \--+++---+--+++-++---+++-\\\
++-++---+++---+++---+--+\\\ ++---+--+--+--+--+--+++-\end{smallmatrix}$
TABLE II: Binary type-II ZCCS.

Let $\mathcal{B}=\\{\mathbf{b_{1}},\mathbf{b_{2}},\ldots,\mathbf{b_{r}}\\}$ be
a set of sequences such that $\Lambda(\mathbf{b_{i}},\mathbf{b_{j}})(0)=0$ for
$i\neq j$, i.e., $\mathbf{b_{i}}\cdot\mathbf{b_{j}}^{*}=0$, we call such
sequence mutually orthogonal to conjugate sequences.

###### Corollary 3

Let $\mathbf{C}=\\{C_{1},C_{2},\ldots,C_{K}\\}$ be a $(K,K,N)$-CCC and
$\mathcal{B}=\\{\mathbf{b_{1}},\mathbf{b_{2}},\ldots,\mathbf{b_{r}}\\}$ be a
set of mutually orthogonal to conjugate sequences of length $P$, i.e.,
$\mathbf{b_{p}}\cdot\mathbf{b_{p^{\prime}}^{*}}=0$, for $p\neq p^{\prime}$.
Then
$\mathbf{Z}=\\{C_{1}\otimes\mathbf{b_{1}},C_{2}\otimes\mathbf{b_{1}},\ldots,C_{K}\otimes\mathbf{b_{1}},C_{1}\otimes\mathbf{b_{2}},C_{2}\otimes\mathbf{b_{2}},\ldots,C_{K}\otimes\mathbf{b_{2}},\ldots,C_{1}\otimes\mathbf{b_{r}},C_{2}\otimes\mathbf{b_{r}},\ldots,C_{K}\otimes\mathbf{b_{r}}\\}$
is a type-II $(rK,K,NP-P+1,NP)$-ZCCS.

###### Proof:

From Theorem 1, $\Lambda(Z_{i})(\tau)=0$, for $\lvert\tau\rvert\geq P$. Now,
we partition the set
$\mathbf{Z}=\\{\mathbf{Z^{1}},\mathbf{Z^{2}},\ldots,\mathbf{Z^{r}}\\}$, where
$\mathbf{Z}^{p}=\\{C_{1}\otimes\mathbf{b_{p}},C_{2}\otimes\mathbf{b_{p}},\ldots,C_{K}\otimes\mathbf{b_{p}}\\}$,
for $p=1,2,\ldots,r$. From Theorem 1, ACCS value between any two codes from
$\mathbf{Z^{p}}$ is zero for all time shifts. Let
$Z_{\alpha}=C_{k_{1}}\otimes\mathbf{b_{p}}$ and
$Z_{\beta}=C_{k_{2}}\otimes\mathbf{b_{p^{\prime}}}$ be any two codes from
$\mathbf{Z}$. From Lemma 2,

$\displaystyle\Lambda(Z_{\alpha},Z_{\beta})(Pj+k)$
$\displaystyle=\Lambda({C_{k_{1}},C_{k_{2}}})(j)\Lambda(\mathbf{b_{p}},\mathbf{b_{p}^{\prime}})(k)$
(7)
$\displaystyle+\Lambda(C_{k_{1}},C_{k_{2}})(j+1)\Lambda(\mathbf{b_{p}},\mathbf{b_{p}^{\prime}})(-P+k),$

$\text{for }0\leq k<P\text{ and }-N<j<N$.

For any $k_{1},k_{2}$ and $u\neq 0$, $\Lambda({C_{k_{1}},C_{k_{2}}})(u)=0$
which implies that $\Lambda(Z_{\alpha},Z_{\beta})(\tau)=0$, for
$\lvert\tau\rvert\geq P$. For $\tau=0$, $k=j=0$ and we also have
$\Lambda(\mathbf{b_{p}},\mathbf{b_{p}^{\prime}})(0)=0$. Therefore, from (7),
$\Lambda(\mathbf{b_{p}},\mathbf{b_{p}^{\prime}})(0)=0$, implying,
$\Lambda(Z_{\alpha},Z_{\beta})(0)=0$. This complete the proof. ∎

For example, consider $\mathbf{C}$ from Example 1, and consider $\mathcal{B}$
as a collection of rows of a Hadamard matrix of size $8$. Using Corollary 3,
we have binary type-II $(32,4,57,64)$-ZCCS.

The number of codes for a type-I $(K,M,Z,N)$-ZCCS is bounded by $K\leq
M\left\lfloor\frac{N}{Z}\right\rfloor$ [25]. The number of codes for the
proposed type-II $(K,M,Z,N)$-ZCCS is bounded by $K\leq M(N-Z+1)$, which is
larger than that of type-I ZCCS.

##  IV Comparison With Existing Type-II ZCP

The proposed construction also features type-II ZCP of length in the form of
$NP$, where $P$ is any natural number and $N$ is the length of available GCPs,
thus type-II ZCP can be constructed of lengths in the form of $N,2N,3N,$ and
so on. However, [18] constructed quadriphase type-II ZCP with the length in
the forms of $3N$, $7N$, $9N$, $14N$ and $15N$ only.

In the proposed construction, if we consider GCPs of length in the form of
power-of-two, then every length of type-II ZCP mentioned in [19] can obtain
with the same ZCZ width. As we are using GCPs of every possible length, our
proposed construction provides not only more lengths but also large ZCZ width
with low AACS value out side the ZCZ width. Therefore, PMEPR of our ZCPs are
lower than the PMEPR of ZCPs constructed in [19].

##  V Conclusion

In this paper, we have provided a construction for type-II ZCCS. For a given
$(K,K,N)$-CCC, type-II $(K,K,NP-P+1,NP)$-ZCCS can be generated by the proposed
construction. The upper bound column sequence PMEPR of proposed type-II ZCCS
with set sizes $3,5,7,11,$ and $13$ can be reduced upto $5/3,9/5,13/7,21/11$
and $25/13$, respectively. The set size can be offered as a product of $r$
(number of mutually orthogonal to conjugate sequences) and the number of
constituent sequences in a code. The proposed construction also generalizes
some of the existing type-II ZCP and produces type-II ZCS for the first time
in the literature. An open problem could be to construct a set of mutually
orthogonal to conjugate sequence of arbitrary length with lower AACS values
similar to Barker sequences.

## References

  * [1] M. Golay, “Complementary series,” _IRE Trans. Inf. Theory_ , vol. 7, no. 2, pp. 82–87, Apr. 1961.
  * [2] C.-C. Tseng and C. Liu, “Complementary sets of sequences,” _IEEE Trans. Inf. Theory_ , vol. 18, no. 5, pp. 644–652, 1972.
  * [3] K. Paterson, “Generalized Reed-Muller codes and power control in OFDM modulation,” _IEEE Trans. Inf. Theory_ , vol. 46, no. 1, pp. 104–120, 2000\.
  * [4] J. A. Davis and J. Jedwab, “Peak-to-mean power control in OFDM, Golay complementary sequences and Reed-Muller codes,” _IEEE Trans. Inf. Theory_ , vol. 45, no. 7, pp. 2397–2417, Nov. 1999.
  * [5] A. Rathinakumar and A. K. Chaturvedi, “Complete mutually orthogonal Golay complementary sets from Reed–Muller codes,” _IEEE Trans. Inf. Theory_ , vol. 54, no. 3, pp. 1339–1346, 2008.
  * [6] P. Sarkar, S. Majhi, and Z. Liu, “A direct and generalized construction of polyphase complementary sets with low PMEPR and high code-rate for OFDM system,” _IEEE Trans. Commun._ , vol. 68, no. 10, pp. 6245–6262, 2020.
  * [7] P. Kumar, S. Majhi, and S. Paul, “A direct construction of GCP and binary CCC of length non power of two,” 2021. [Online]. Available: https://arxiv.org/abs/2109.08567
  * [8] P. Fan, W. Yuan, and Y. Tu, “Z-complementary binary sequences,” _IEEE Signal Process. Lett._ , vol. 14, no. 8, pp. 509–512, Aug. 2007.
  * [9] Z. Liu, U. Parampalli, and Y. L. Guan, “Optimal odd-length binary Z-complementary pairs,” _IEEE Trans. Inf. Theory_ , vol. 60, no. 9, pp. 5768–5781, 2014.
  * [10] A. R. Adhikary, P. Sarkar, and S. Majhi, “A direct construction of $q$-ary even length Z-complementary pairs using generalized Boolean functions,” _IEEE Signal Process. Lett._ , vol. 27, pp. 146–150, 2020.
  * [11] A. R. Adhikary, S. Majhi, Z. Liu, and Y. L. Guan, “New sets of even-length binary Z-complementary pairs with asymptotic ZCZ ratio of $3/4$,” _IEEE Signal Process. Lett._ , vol. 25, no. 7, pp. 970–973, July, 2018.
  * [12] Z. Liu, U. Parampalli, and Y. L. Guan, “On even-period binary Z-complementary pairs with large ZCZs,” _IEEE Signal Process. Lett._ , vol. 21, no. 3, pp. 284–287, Mar. 2014.
  * [13] C.-Y. Chen and C.-Y. Pai, “Binary Z-complementary pairs with bounded peak-to-mean envelope power ratios,” _IEEE Commun. Lett._ , vol. 23, no. 11, pp. 1899–1903, 2019.
  * [14] C.-Y. Pai, S.-W. Wu, and C.-Y. Chen, “Z-complementary pairs with flexible lengths from generalized Boolean functions,” _IEEE Commun. Lett._ , vol. 24, no. 6, pp. 1183–1187, 2020.
  * [15] C.-Y. Chen, “A novel construction of Z-complementary pairs based on generalized Boolean functions,” _IEEE Signal Process. Lett._ , vol. 24, no. 7, pp. 987–990, 2017.
  * [16] A. R. Adhikary, S. Majhi, Z. Liu, and Y. L. Guan, “New sets of optimal odd-length binary Z-complementary pairs,” _IEEE Trans. Inf. Theory_ , vol. 66, no. 1, pp. 669–678, 2020.
  * [17] Z. Gu, Z. Zhou, Q. Wang, and P. Fan, “New construction of optimal type-II binary Z-complementary pairs,” _IEEE Trans. Inf. Theory_ , vol. 67, no. 6, pp. 3497–3508, 2021.
  * [18] F. Zeng, X. He, Z. Zhang, and L. Yan, “Constructions of Z-optimal type-II quadriphase Z-complementary pairs,” _IEEE Wirel. Commun. Lett._ , vol. 11, no. 3, pp. 568–572, 2022.
  * [19] R. Kumar, P. Sarkar, P. K. Srivastava, and S. Majhi, “A direct construction of asymptotically optimal type-II ZCP for every possible even length,” _IEEE Signal Process. Lett._ , vol. 28, pp. 1799–1802, 2021.
  * [20] W. C. Y. Lee, _Mobile communications Design Fundamentals_. Wiley Series in Telecommunication and Signal Processing, 1993.
  * [21] P. Sarkar, S. Majhi, and Z. Liu, “Optimal Z-complementary code set from generalized Reed-Muller codes,” _IEEE Trans. Commun._ , vol. 67, no. 3, pp. 1783–1796, Mar. 2019.
  * [22] Y. Jin and H. Koga, “Basic properties of the complete complementary codes using the DFT matrices and the Kronecker products,” in _2008 International Symposium on Information Theory and Its Applications_ , 2008, pp. 1–6.
  * [23] J. Jedwab and M. G. Parker, “A construction of binary Golay sequence pairs from odd-length Barker sequences,” _J. Comb. Des._ , vol. 17, no. 6, pp. 478–491, 2009.
  * [24] P. Sarkar, Z. Liu, and S. Majhi, “Multivariable function for new complete complementary codes with arbitrary lengths,” 2021. [Online]. Available: https://arxiv.org/abs/2102.10517
  * [25] L. Feng, P. Fan, and X. Zhou, “Lower bounds on correlation of Z-complementary code sets,” _Wirel. Pers. Commun._ , vol. 72, pp. 1475–1488, 2013.