11institutetext: On-Device AI, Samsung R&D Institute, Bangalore, India
11email:  {r.moharir, arun.prabhu, msukumar, gopi.ramena,
<EMAIL_ADDRESS>
# On-Device Spatial Attention based Sequence Learning Approach for Scene Text
Script Identification

Rutika Moharir 11 Arun D Prabhu 11 Sukumar Moharana 11 Gopi Ramena 11 Rachit S
Munjal 11

###### Abstract

Automatic identification of script is an essential component of a multilingual
OCR engine. In this paper, we present an efficient, lightweight, real-time and
on-device spatial attention based CNN-LSTM network for scene text script
identification, feasible for deployment on resource constrained mobile
devices. Our network consists of a CNN, equipped with a spatial attention
module which helps reduce the spatial distortions present in natural images.
This allows the feature extractor to generate rich image representations while
ignoring the deformities and thereby, enhancing the performance of this fine
grained classification task. The network also employs residue convolutional
blocks to build a deep network to focus on the discriminative features of a
script. The CNN learns the text feature representation by identifying each
character as belonging to a particular script and the long term spatial
dependencies within the text are captured using the sequence learning
capabilities of the LSTM layers. Combining the spatial attention mechanism
with the residue convolutional blocks, we are able to enhance the performance
of the baseline CNN to build an end-to-end trainable network for script
identification. The experimental results on several standard benchmarks
demonstrate the effectiveness of our method. The network achieves competitive
accuracy with state-of-the-art methods and is superior in terms of network
size, with a total of just 1.1 million parameters and inference time of 2.7
milliseconds.

##  1 Introduction

Most multilingual OCR systems require knowledge of the underlying script for
each text instance present in any natural image. Hence, script identification
acts as an essential step in increasing the overall efficiency of any text
recognition pipeline. Many languages share the same script family, which makes
script identification a fine-grained classification task.

Earlier works in script identification have been proposed for document [16],
[2], handwritten text [11], [13] and video-overlaid text [27], [6] where the
texts are fairly simple with non-complex backgrounds and have been able to
achieve great performance. However, scene text images pose additional
challenges such as varied appearances and distortions, various text styles and
degradation like background noise and light conditions, etc. Our work on scene
text script identification focuses on two challenges – first, handling spatial
distortions in scene text and exploiting sequential dependencies within the
text which can help focus on discriminative features for efficient
classification of scripts with relatively subtle differences, for example
Chinese and Japanese which share a common subset of characters. Second, be
suitable, have low latency and memory footprint, for on-device deployment.

Figure 1: The architecture of the proposed script identification model.

While recent works have been trying to handle script identification in complex
backgrounds [1], [3], [26], [5], [4], they have their own challenges. [4]
tries to enhance discriminative features using intermediate supervision on
patch-level predictions. While, [1] proposed an attention-based Convolutional-
LSTM network, analyzing features globally and locally. [3] uses an end-to-end
trainable single fully convolutional network (FCN) while [7] adopted a patch-
based method containing an ensemble of identical nets to learn discriminative
strokepart representations. In [26] they used BLCT to enhance the
discriminative features and imposed inverse document frequency (idf) to get
codewords occurrences. [5] use Encoder and Summarizer to get local features
and fuse them to a single summary by attention mechanism to reflect the
importance of different patches. However, few works have been reported to
address an important factor that often leads to script identification failures
– the degradation of scene text such as blur, tilt, low contrast to the
background, etc. which are not rare in natural images. Instead of using
computationally expensive attention-based patch weights [1] or optimizing
patch level predictions [4], we tackle this problem using a spatial attention
mechanism that is capable of directly learning the region of interest for each
character in the text for a superior foreground-background separation. Figure
3 shows the effect of using our spatial attention mechanism. The distorted
text images are transformed and made suitable for script identification.

As for the problem of failing to capture discriminative features, [21]
presented a method based on hand-crafted features, a LBP variant, and a deep
multi-Layer perceptron. Pre-defined image classification algorithms, such as
[17] who proposed a combination of CNN and LSTM model, the robustly tested CNN
[24] or the Single-Layer Networks (SLN) [23] normally consider holistic
representation of images. Hence they perform poorly in distinguishing
similarly structured script categories as they fail to take into account the
discriminative features of the scripts. For example, a few languages belonging
to the Chinese and Japanese script share a common subset of characters. A word
is more likely to be Japanese if both Chinese and Japanese characters occur in
an image. But when no Japanese character occurs, the word has to be Chinese.
We propose a sequence to sequence mapping of individual characters’ shape to
their script, from which we can know what scripts the character of a given
image could be. After that, the majority of the predicted probability
distributions is calculated,and the dominant script is determined.

In this paper, we propose a novel scene text script identification method
which uses spatial attention based sequence learning approach for script
identification in scene text images. We use spatial attention blocks which
extract rich text features while ignoring the disruptions present in the scene
text. Inspired by the successful application of residual connections in image
classification tasks [12], [25], we employ hierarchical residual connections
to build a deeper network that helps in learning representative image
features. In training phase, instead of learning a single representation of
the text and failing to capture the discriminating chars of each script, a
segmentation free, sequence to sequence mapping is done using Connectionist
Temporal Classification (CTC) to supervise the predictions. The key
contributions of our scene text script identification method are:

1\. We propose a novel end-to-end trainable scene text script identification
network which integrates spatial attention mechanism and residue convolutional
blocks with the sequence learning capabilities of the LSTM to generate
sequence to sequence mapping for discriminative learning of the script. The
module has low latency and low memory footprint and thus is feasible for
deployment in on-device text recognition systems.

2\. A deep feature extractor with residue convolutional blocks generates a
rich image representation for the text.

3\. The proposed sequence learning based script identification method does
supervision using Connectionist Temporal Classification (CTC). To the extent
of our knowledge, this is the very first time that CTC demonstrates strong
capabilities for scene text script identification.

Figure 2: Illustration of the Spatial Attention Block (left) and the Residue
Convolutional Block (right) which are the components of the feature extractor.

##  2 Methodology

In the following sections, we describe in detail the three key components of
our network, namely, the spatial attention based feature extractor (2.1),
sequence modelling (2.2) and prediction (2.3). Figure 1 illustrates the
overall architecture of our network.

###  2.1 Spatial Attention based Feature Extractor

The feature extractor consists of a combination of two modules – Spatial
Attention Block and the Residue Convolutional Block (see Figure 2).

Spatial Attention Block Spatial attention mechanism in a convolutional neural
network (CNN) ignores the information provided by the channels and treats all
the channels in a feature map equally. In the CNN, the low level feature maps
having fewer channels, are responsible for extracting the spatial
characteristics such as edges and contours. Hence in our network, the spatial
attention block is introduced for the low level feature maps to learn the
interaction of spatial points and focus on the key informative areas while
ignoring the irrelevant distortions.

As can be seen from Figure 2, to compute the spatial attention, firstly, we
pass the feature map $F\in R^{H\times W\times C}$ to the channel aggregation
operation which generates a global distribution of spatial features, the
spatial descriptor $S\in R^{H\times W}$. Global average pooling $G_{avg}$ in
the channel dimension (C) is used as the aggregation function.

$s_{hw}\ =\ G_{avg}(f_{hw})\ =\ \frac{1}{C}\sum_{i=l}^{C}f_{hw}(i)$ (1)

where $f_{hw}\in R^{C}$ refers to the local feature at spatial position (h,
w). On the generated spatial descriptor $S$, weight learning is implemented
using a convolutional layer to generate the spatial attention map $T\in
R^{H\times W}$ which is computed as:

$T\ =\ \sigma(\delta(f_{1}p))$ (2)

where p is spatial position, $f_{1}$ represents a convolution operation with
filter size $3\times 3$, $\delta$ refers to activation function ReLU [18] and
$\sigma$ is a sigmoid activation function used to generate spatial weight
$t_{h,w}\in(0,1)$ at position (h, w). T is finally applied to the feature map
F. The feature values at different positions in F are multiplied by different
weights in T to generate the output O of the spatial attention block as:

$O_{hw}\ =\ f_{hw}\cdot t_{hw}$ (3)

Residue Convolutional Block The residue block (see Figure 2 (b)) consists of
two convolutional layers, each followed by ReLU activation and a skip-
connection between the input of the first and the output of the second
convolutional layer. If the input x is mapped to the true distribution H(x),
then the difference can be stated as:

$R(x)\ =\ H(x)-x$ (4)

The layers in a traditional network learn the true output H(X) while the
residue convolutional block layers learn the residue R(x), since they have an
identity connection reference due to x. Other than the fact that it is easier
to optimise the residual mapping than the original unreferenced mapping, an
added advantage is that it can learn the identity function by simply setting
the residual as zero. This enables to address the problem of vanishing
gradients, giving us the ability to build deep feature extractor consisting of
7 convolutional layers.

Figure 3: Illustration of the effect of the Spatial Attention Block during
feature extraction. The 2nd column represents feature maps generated in the
absence of the attention blocks while the 3rd column shows enhanced
foreground-background separation with the removal of distortions on the
application of the attention blocks.

###  2.2 Sequence Modelling

Previous works fail to take the spatial dependencies within text word into
consideration, which may be discriminative for script identification as
mentioned in section 1.

Long Short-Term Memory (LSTM) [14] solves this problem by introducing three
gates: input gate, forget gate, output gate. They are integrated together to
capture spatial dependencies for a long time by keeping the gradient in the
same order of magnitude during back-propagation. The hidden state $h_{t}$
depends not only on the current input $x_{t}$, but also previous time step
hidden state $h_{t-1}$.

In this work, we use one Bidirectional-LSTM (Bi-LSTM) layer with 256 hidden
states which calculates $h_{t}$ using the memory block from both forward and
backward directions. For faster inference, we use LSTM with recurrent
projection layer [22] to decrease the computational complexity. The projection
layer projects the hidden state to a lower dimension of 96 at every time step.

###  2.3 Prediction

We concatenate the features generated by the Bi-LSTM layer and the per-frame
predictions are fed into a fully connected layer, followed by a softmax layer
which outputs the normalized probabilities for each script class.

We use Connectionist Temporal Classification (CTC) [9] to help the model
associate the shape of the character to a script class.

The formulation of the conditional probability is described as follows. Let
$L$ represent the set of all script classes in our task and
$L^{{}^{\prime}}=L\cup\\{blank\\}$. The output from the sequential modelling
block is transformed into per-frame probability distribution
$y=(y_{1},y_{2}..y_{T})$ where $T$ is the sequence length. Each $y_{t}$ is the
probability distribution over the set $L^{\prime}$. A sequence-to-sequence
mapping function $\beta$ is defined on the sequence $\pi\in L^{\prime T}$
which deletes repeated characters and the blank symbol. The conditional
probability is calculated by summing the probabilities of all paths mapped
onto the label sequence $l$ by $\beta$ :

$p(l|y)\ =\ \sum_{\beta(\pi)=l}p(\pi|y)$ (5)

As directly computing the above equation is computationally expensive,
forward–backward algorithm [9] is used to compute it efficiently. While
predicting the sequence of labels, we pick the label with the highest
probability at each time step and then apply $\beta$ to the entire output
path.

During inference, we calculate the number of times each script is predicted
for the given word crop. The script having the highest count is considered as
the script for the word crop.

Method | MLT17 | CVSI 15 | MLe2e | Parameters | Time
---|---|---|---|---|---
Bhunia[1] | 90.23 | 97.75 | 96.70 | 12 | 85
ECN [8] | 86.46 | 97.20 | 94.40 | 24 | 13
Cheng[4] | 89.42 | 98.60 | - | 26.7 | 2.5
Mei[17] | - | 94.20 | - | 5.2 | 92
Zdenek[26] | - | 97.11 | - | - | 60
Ours | 89.59 | 97.44 | 97.33 | 1.1 | 2.7
Table 1: Script Identification accuracy on benchmark datasets. The best
performing result for each column is set in bold font. The number of
parameters are in millions (M) and time is denoted in milliseconds per image
(ms).

##  3 Experiments

###  3.1 Dataset

We evaluate our scene text script identification methods on following
benchmark datasets:

ICDAR 2019 [19] (IC19) consists of 8 script classes - Latin, Korean, Chinese,
Japanese, Arabic, Hindi, Bangla and Symbols. The test set consists of
approximately 102462 cropped word images and is highly biased towards Latin.

RRC-MLT 2017 [20] (MLT17) consists of 97,619 test cropped images. This dataset
holds 7 scripts: Arabic, Latin, Chinese, Korean, Japanese, Bangla, Symbols.
There exist some multioriented, curved and perspectively distorted texts which
make it challenging.

CVSI 2015 [23] (CVSI 15) dataset contains scene text images of ten different
scripts: Latin, Hindi, Bengali, Oriya, Gujrati, Punjabi, Kannada, Tamil,
Telegu, and Arabic. Each script has at least 1,000 text images collected from
different sources (i.e. news, sports etc.)

MLe2e [7] is a dataset for the evaluation of scene text end-to-end reading
systems. The dataset contains 643 word images for testing covering four
different scripts: Latin, Chinese, Kannada and Hangul many of which are
severely corrupted by noise and blur and may have very low resolutions.

###  3.2 Implementation Details

For feature extraction, the input text crops are scaled to fixed height of 24
while maintaining the aspect ratio. The residue convolutional blocks are
stacked with the spatial attention block being introduced after the 1st and
2nd block as shown in Figure 1. The filter size, stride and padding size of
all the convolutional layers is 3, 1 and 1 respectively and each convolutional
layers is followed by a max-pooling layer. The output filter size for the
convolutional layers are 32, 64, 96, 128, 164, 196, 256 respectively and each
layer is followed by the ReLU activation function. Batch normalization [15] is
adopted after the 3rd and the 5th convolutional layer, speeding up the
training significantly. The input layer is followed by a 3 × 3-stride max-pool
layer. This is followed by three residual blocks, with each unit consisting of
two 3×3 convolution operations. The first residual block further downsample
the feature maps with a 2 × 2-stride max-pool layer, while the last two
residual blocks each employ a 2 × 1-stride max-pool layer to reduce the sizes
of feature maps only in the height dimension. The two LSTMs of a BiLSTM layer
in the encoder each have 256 hidden units for capturing dependencies in each
direction of the feature sequence.

We initialize the network by first training on a synthetic dataset generated
using the method followed by [10]. The training set consists of 1.5 million
word crops with equal distribution for all the scripts, which are randomly
sampled to form minibatches of size 96. Further, the network is fine-tuned on
the benchmark datasets and evaluated. We adopt some data augmentation
techniques such as varying contrast, adding noise and perspective
transformations to make the dataset robust. The Adam optimizer is used with
the initial learning rate being set to 0.001 and is automatically adjusted by
the optimizer during training. The training of the network on Nvidia Geforce
GTX 1080ti GPU, 16GB RAM takes about 36 hours for 10 epochs. The network has
approximately 1.1 million parameters and takes 2.7 ms per image for
identification.

Module | Accuracy | Params
---|---|---
CNN+LSTM | 84.79 | 0.65
ResCNN+LSTM | 84.66 | 1.0
Atten+CNN+LSTM | 86.75 | 1.03
Atten+ResCNN+LSTM | 88.60 | 1.1
Atten+CNN+LSTM+CTC | 87.85 | 1.03
Atten+ResCNN+LSTM+CTC | 90.17 | 1.1

Position of SA blocks | Accuracy
---|---
After conv and 1st RB | 83.81
After conv and 2nd RB | 84.59
After 1st and 2nd RB | 85.63
After 2nd and 3rd RB | 84.81

Table 2: (a) Comparison with different variations of the proposed network on
MLT17 validation set. Parameters are mentioned in millions. (b) Script
Identification accuracy on MLT17 with respect to the position of spatial
attention blocks, where RB stands for Residual Block and SA stands for Spatial
Attention Block

###  3.3 Effectiveness of Spatial Attention and Residue Convolutional Blocks
for Script Identification

To validate the effectiveness of the proposed spatial attention based residue
convolutional network for scene text script identification, in Table 2 (a), we
compare the performance with that of a baseline CNN network without the
application of the spatial attention mechanism or residue convolutional blocks
but with the same sequence modelling backbone. Specifically, in the ’CNN+LSTM’
model we skip the spatial attention block and residual connections, feeding
the input text image into the ’CNN+LSTM’ backbone and using cross entropy
classification as the loss function. The model ‘ResCNN+LSTM’ omits the spatial
attention block and makes use of the residue convolutional blocks, enhancing
the quality of image feature representation. On the other hand, the model
’Atten+CNN+LSTM’ skips using the residual connections and employs the spatial
attention block, using cross entropy as the optimization function while
’Atten+CNN+LSTM+CTC’ makes use of the CTC loss function.The model
’Atten+ResCNN+LSTM’ combines the spatial attention blocks with the residue
convolutional blocks, and the model ’Atten+Res+CNN+LSTM+CTC’ instead uses CTC
loss function to optimise the prediction.

Contribution of spatial attention blocks and residue convolutional blocks
Compared to the baseline model, the introduction of the proposed spatial
attention module and the residue convolutional module effectively increases
the accuracy of the script identification model. Combining both modules helps
further enhance the performance on all the datasets. ’Atten+ResCNN+LSTM’
attains more accurate residual enhancement representations as compared to
’Atten+CNN+LSTM’, thus capturing low level features, which help improve the
identification accuracy. The overall enhanced identification accuracy of the
proposed spatial attention based sequence learning approach for script
identification relative to the baseline, demonstrates the effectiveness of
spatial attention for scene text script identification.

Arrangement of spatial attention blocks To ensure low latency and
computational complexity we limit to using two spatial attention blocks. Given
an input image, the two attention blocks compute attention, focusing on
exploiting image features while ignoring distortions. Considering this, the
two blocks can be placed at either the low level or at the higher level in the
network. In this experiment we compare four different ways of arranging the
two spatial attention blocks: After conv and the 1st residual block, after
conv and 2nd residual block, after the 1st and 2nd residual block and finally,
after the 2nd and 3rd residual block. Since the spatial attention works
locally, the order may affect the overall performance. Table 2 (b) summarises
the experimental results on different combinations. From the results we can
find that spatial attention blocks after the 1st residual block and the 2nd
residual block is slightly better than the other arrangements.

Effect of CTC loss To verify the superiority of the proposed sequence to
sequence mapping method for script identification, which performs per
character script classification, we replace the proposed CTC module in our
script identification model ’Atten+ Res+ CNN+ LSTM+ CTC’, with the Cross
entropy classification technique, the model ’Atten+ Res+ CNN+ LSTM’, which
generates a single representation for the input image and the spatial
attention based residual network is trained. The experimental results show
that CTC module yields 1.57% higher script identification accuracy on the
benchmark dataset compared to the cross entropy module whose output features
may be ignoring the discriminative features between similarly structured
scripts.

Method | Accuracy (%)
---|---
Tencent - DPPR team | 94.03
SOT: CNN based classifier | 91.66
GSPA_HUST | 91.02
SCUT-DLVC-Lab | 90.97
TH-ML | 88.85
Multiscale_HUST | 88.64
Conv_Attention | 88.41
ELE-MLT based method | 82.86
Ours | 89.82

Script | Accuracy
---|---
Latin | 98.30
Korean | 94.28
Chinese | 90.28
Japanese | 84.15
Arabic | 92.61
Hindi | 86.28
Bangla | 87.28
Symbols | 85.44
Average | 89.82

Table 3: (a) Comparison of our results on MLT 19 with other methods. (b)
Script wise results of our method on MLT19 dataset.

###  3.4 Comparison with State-of-the-Art Script Identification Methods

Our method achieves the highest script identification accuracy on one of the
three benchmark datasets and a highly competitive accuracy with the last two
datasets. Note that our network is significantly lighter and faster as
compared to other state-of-the-art methods which qualifies it for real-time
on-device pre-ocr applications.

For the scene text images in Mle2e, a great improvement has been made. Since
in this dataset, many of the text regions are severely distorted and few of
them have very low resolutions, the performance of our network shows the
ability and the need for a spatial attention mechanism in handling disruptions
occurring in natural images. As can been seen from the Table 1, our method
also performs well on the MLT17 dataset.Table 4 compares our method with
others on CVSI 2015 dataset. The result demonstrates that our method performs
well on both document text images and complex background images.

Table 3 mentions script wise performance accuracy of our method ICDAR19 RRC
(IC19). Table 3 also compares the performance of our method with other models
described in [19]. The baseline methods have based their methods on famous
deep nets for text recognition such as ResNet, VGG16, Seq2Seq with CTC etc.
and use the recognised characters to identify the scripts while adopting
improvements such as multiscale techniques, voting strategy for combining
results from multiple nets, training statistics of the scripts etc. Even
though the proposed network is slightly less accurate the benchmarks, it is
superior in terms of latency and model size and thus, in contrast, is feasible
for on-device deployment.

Given the relatively standard feature extraction backbone employed in our
method, the results of the experiment demonstrates the significant effect of
the proposed spatial attention mechanism coupled with the sequence learning
capabilities of LSTM on improving the accuracy of the script identification
model. Particularly, compared to the existing script identification methods,
Section 1, which either use complex self-attention mechanisms or heavy feature
extractors, our method with a simple convolution based attention block and a
small CNN backbone achieves overall enhanced accuracy, with the additional
benefit of being computationally lighter and faster.

Script | C-DAC | HUST | CVC-2 | CIJK | Cheng | Ours
---|---|---|---|---|---|---
Eng | 68.33 | 93.55 | 88.86 | 65.69 | 94.20 | 98.54
Hin | 71.47 | 96.31 | 96.01 | 61.66 | 96.50 | 97.24
Ben | 91.61 | 95.81 | 92.58 | 68.71 | 95.60 | 97.74
Ori | 88.04 | 98.47 | 98.16 | 79.14 | 98.30 | 98.16
Guj | 88.99 | 97.55 | 98.17 | 73.39 | 98.70 | 96.94
Pun | 90.51 | 97.15 | 96.52 | 92.09 | 99.10 | 97.47
Kan | 68.47 | 92.68 | 97.13 | 71.66 | 98.60 | 97.45
Tam | 91.90 | 97.82 | 99.69 | 82.55 | 99.20 | 96.88
Tel | 91.33 | 97.83 | 93.80 | 57.89 | 97.70 | 96.60
Ara | 97.69 | 100.00 | 99.67 | 89.44 | 99.60 | 97.36
Average | 84.66 | 96.69 | 96.00 | 74.06 | 97.75 | 97.44
Table 4: Script wise performance comparison with various methods on CVSI 2015
dataset.

##  4 Conclusion

In this paper, we present a novel lightweight real-time on-device spatial
attention based sequence learning approach for scene text script
identification. The spatial attention mechanism helps removing spatial
disruptions in natural images for improving script identification accuracy in
complex scenarios. Our network employs residue convolutional blocks to
generate rich feature representations of the input image thus capturing
discriminative features of a script. Sequence learning using the Bidirectional
LSTM layer, optimised using the Connectionist Temporal Classification enables
the model to associate the characters shape to its script, thereby enhancing
script identification accuracy. While achieving competitive accuracy with
state-of-the-art methods, our network is superior, as it achieves 4 times
reduction in model size and has low latency, making it suitable for on-device
inference.

## References

  * [1] Bhunia, A.K., Konwer, A., Bhunia, A.K., Bhowmick, A., Roy, P.P., Pal, U.: Script identification in natural scene image and video frames using an attention based convolutional-lstm network. Pattern Recognition (2019)
  * [2] Busch, A., Boles, W.W., Sridharan, S.: Texture for script identification. IEEE Transactions on Pattern Analysis and Machine Intelligence (2005)
  * [3] Bušta, M., Patel, Y., Matas, J.: E2e-mlt-an unconstrained end-to-end method for multi-language scene text. In: Asian Conference on Computer Vision. Springer (2018)
  * [4] Cheng, C., Huang, Q., Bai, X., Feng, B., Liu, W.: Patch aggregator for scene text script identification. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). IEEE (2019)
  * [5] Fujii, Y., Driesen, K., Baccash, J., Hurst, A., Popat, A.C.: Sequence-to-label script identification for multilingual ocr. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR). vol. 1. IEEE (2017)
  * [6] Gllavata, J., Freisleben, B.: Script recognition in images with complex backgrounds. In: Proceedings of the Fifth IEEE International Symposium on Signal Processing and Information Technology, 2005. pp. 589–594. IEEE (2005)
  * [7] Gomez, L., Karatzas, D.: A fine-grained approach to scene text script identification. In: 2016 12th IAPR Workshop on Document Analysis Systems (DAS). IEEE (2016)
  * [8] Gomez, L., Nicolaou, A., Karatzas, D.: Boosting patch-based scene text script identification with ensembles of conjoined networks. arXiv preprint arXiv:1602.07480 (2016)
  * [9] Graves, A., Fernández, S., Gomez, F., Schmidhuber, J.: Connectionist temporal classification: labelling unsegmented sequence data with recurrent neural networks. In: Proceedings of the 23rd international conference on Machine learning (2006)
  * [10] Gupta, A., Vedaldi, A., Zisserman, A.: Synthetic data for text localisation in natural images. In: Proceedings of the IEEE conference on computer vision and pattern recognition (2016)
  * [11] Hangarge, M., Dhandra, B.: Offline handwritten script identification in document images. International Journal of Computer Applications (2010)
  * [12] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (2016)
  * [13] Hochberg, J., Bowers, K., Cannon, M., Kelly, P.: Script and language identification for handwritten document images. International Journal on Document Analysis and Recognition (1999)
  * [14] Hochreiter, S., Schmidhuber, J.: Long short-term memory. Neural computation (1997)
  * [15] Ioffe, S., Szegedy, C.: Batch normalization: Accelerating deep network training by reducing internal covariate shift. In: International conference on machine learning. PMLR (2015)
  * [16] Joshi, G.D., Garg, S., Sivaswamy, J.: A generalised framework for script identification. International Journal of Document Analysis and Recognition (IJDAR) (2007)
  * [17] Mei, J., Dai, L., Shi, B., Bai, X.: Scene text script identification with convolutional recurrent neural networks. In: 2016 23rd International Conference on Pattern Recognition (ICPR). IEEE (2016)
  * [18] Nair, V., Hinton, G.E.: Rectified linear units improve restricted boltzmann machines. In: Icml (2010)
  * [19] Nayef, N., Patel, Y., Busta, M., Chowdhury, P.N., Karatzas, D., Khlif, W., Matas, J., Pal, U., Burie, J.C., Liu, C.l., et al.: Icdar2019 robust reading challenge on multi-lingual scene text detection and recognition—rrc-mlt-2019. In: 2019 International Conference on Document Analysis and Recognition (ICDAR). IEEE (2019)
  * [20] Nayef, N., Yin, F., Bizid, I., Choi, H., Feng, Y., Karatzas, D., Luo, Z., Pal, U., Rigaud, C., Chazalon, J., et al.: Icdar2017 robust reading challenge on multi-lingual scene text detection and script identification-rrc-mlt. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR). IEEE (2017)
  * [21] Nicolaou, A., Bagdanov, A.D., Gomez, L., Karatzas, D.: Visual script and language identification. In: 2016 12th IAPR Workshop on Document Analysis Systems (DAS). IEEE (2016)
  * [22] Sak, H., Senior, A.W., Beaufays, F.: Long short-term memory recurrent neural network architectures for large scale acoustic modeling (2014)
  * [23] Sharma, N., Mandal, R., Sharma, R., Pal, U., Blumenstein, M.: Icdar2015 competition on video script identification (cvsi 2015). In: 2015 13th International Conference on Document Analysis and Recognition (ICDAR). IEEE (2015)
  * [24] Shi, B., Bai, X., Yao, C.: Script identification in the wild via discriminative convolutional neural network. Pattern Recognition 52 (2016)
  * [25] Szegedy, C., Ioffe, S., Vanhoucke, V., Alemi, A.: Inception-v4, inception-resnet and the impact of residual connections on learning. In: Proceedings of the AAAI Conference on Artificial Intelligence (2017)
  * [26] Zdenek, J., Nakayama, H.: Bag of local convolutional triplets for script identification in scene text. In: 2017 14th IAPR International Conference on Document Analysis and Recognition (ICDAR). vol. 1. IEEE (2017)
  * [27] Zhao, D., Shivakumara, P., Lu, S., Tan, C.L.: New spatial-gradient-features for video script identification. In: 2012 10th IAPR International Workshop on Document Analysis Systems. IEEE (2012)


††thanks: Email<EMAIL_ADDRESS>Email:
<EMAIL_ADDRESS>Email<EMAIL_ADDRESS>
# Investigating nonflow contribution subtraction in d-Au collisions with AMPT
model

Zuman Zhang  School of Physics and Mechanical Electrical & Engineering, Hubei
University of Education, Wuhan 430205, China  Institute of Theoretical
Physics, Hubei University of Education, Wuhan 430205, China  Key Laboratory of
Quark and Lepton Physics (MOE), Central China Normal University, Wuhan 430079,
China  Sha Li  School of Physics and Mechanical Electrical & Engineering,
Hubei University of Education, Wuhan 430205, China  Ning Yu  School of Physics
and Mechanical Electrical & Engineering, Hubei University of Education, Wuhan
430205, China  Institute of Theoretical Physics, Hubei University of
Education, Wuhan 430205, China  Key Laboratory of Quark and Lepton Physics
(MOE), Central China Normal University, Wuhan 430079, China  Qiao Wu  School
of Physics and Mechanical Electrical & Engineering, Hubei University of
Education, Wuhan 430205, China

###### Abstract

This paper presents research that focuses on nonflow contribution subtraction
in heavy-ion collisions, using a multiphase transport model (AMPT).
Specifically, the study aims to investigate the behavior of charged particle
elliptic flow ($v_{\rm 2}$) in d-Au collisions at a collision energy of
$\sqrt{s_{\rm NN}}=200$ GeV and to determine the impact of nonflow sources,
such as jet correlations and resonance decays, in small collision systems. To
reduce nonflow effects, the per-trigger yield distribution in peripheral d-Au
collisions or pp collisions with the same collision energy is subtracted. Our
results show that the nonflow effects in central and mid-central collisions
are not strongly dependent on subtracting the per-trigger yield distribution
in peripheral d-Au collisions or pp collisions. Furthermore, the elliptic flow
of charged particles, after removing nonflow effects through two subtracting
methods from this work, exhibits consistency in various collision centrality
classes. We also discuss comparisons with measurements from d-Au collisions at
$\sqrt{s_{\rm NN}}=200$ GeV. Overall, this work provides valuable insights and
serves as a reference for researchers studying nonflow contribution
subtraction in experiments with small collision systems.

PACS numbers

`25.75.Ld Collective flow, 24.10.Lx Monte Carlo` simulations (including hadron
and parton cascades and string breaking models)

##  I Introduction

In high-energy heavy-ion collision research, the goal is to understand the
properties of the quark-gluon plasma (QGP), a state of matter characterized by
high energy density and temperature Muller:2006ee ; Bazavov:2011nk . One of
the key observables in this field is the azimuthal anisotropy of final state
particles, which provides information about the transport features of the QGP
Ollitrault:1992bk . The elliptic flow ($v_{\rm 2}$) is a crucial observable in
studying the collective motion of the QGP and is obtained through the Fourier
expansion of the azimuthal distribution of emitted particles in transverse
momentum space. The second-order coefficient in this expansion provides
valuable insights into the behavior of the QGP and its properties. The study
of elliptic flow is an important aspect of high-energy heavy-ion collision
research as it helps to deepen our understanding of the quark-gluon plasma and
its characteristics.

In order to better understand the effects of cold nuclear matter on the
interpretation of measurements in heavy-ion collisions, we conducted a study
of small collision systems such as p(d)+A collisions. Our research focused on
exploring key aspects of cold nuclear matter effects, such as the modification
of parton distribution functions Eskola:2009uj , the broadening of $k_{\rm T}$
Kopeliovich:2002yh , and energy loss in cold nuclear matter Kang:2014hha . To
our surprise, in small collision systems, using the two-particle azimuthal
correlation method, high-multiplicity p+A collisions at $\sqrt{s_{\rm
NN}}=5.02$ TeV in the midrapidity region by the ALICE, ATLAS, and CMS
collaborations CMS:2012qk ; Abelev:2012ola ; Aad:2012gla ; ABELEV:2013wsa ;
Aaboud:2016yar  and at forward rapidity by the LHCb collaboration Aaij:2015qcq
, we discovered long-range structures in two-particle correlations that were
associated with a positive $v_{\rm 2}$ for hadrons. In addition, at lower beam
energies, long-range correlations were also observed in d-Au Adare:2013piz ;
Adare:2014keg ; Adamczyk:2015xjc  and 3He-Au collisions Adare:2015ctn  by the
PHENIX and STAR collaborations at RHIC.

To address nonflow effects, various strategies have been developed to
eliminate correlations that are not associated with collectivity and arise
from sources such as jet interactions, resonance decays, and so on. Typically,
in small collision system experiments these nonflow contributions were
suppressed by requiring a separation in pseudorapidity between paired
particles or by subtracting correlations measured in low-multiplicity
Abelev:2012ola ; Aad:2014lta  or pp collisions Aad:2019lta . The long-range
correlations were then isolated using a standard template fit procedure
Aad:2019ajj . In the experimental analysis, the contribution of nonflow has
not been calculated by both methods for same collision system. So, we analyze
the $p_{\rm T}$ distribution of anisotropic flow ($v_{\rm 2}$) in d-Au
collisions at $\sqrt{s_{\rm NN}}=200$ GeV, using the multiphase transport
(AMPT) model Lin:2004en  to investigate the nonflow contribution subtraction
from both peripheral and pp collisions. The paper starts with a brief
introduction to the AMPT model. Then, the method of nonflow contribution
subtraction is detailed. Finally, the results are presented, along with a
discussion and conclusion.

##  II Event Generation and Definition of anisotropic flow

###  II.1 A Multi-Phase Transport (AMPT) model

The AMPT model Lin:2004en  is a hybrid transport model used to study
collective behavior in heavy ion collisions. It consists of four components:
initial conditions, partonic interactions, conversion from partonic to
hadronic matter, and hadronic interactions.

The Lund string fragmentation function is determined by the parameters $a$ and
$b$ in HIJING Wang:1991hta  as $f(z)\propto
z^{-1}(1-z)^{a}\exp{(-bm^{2}_{\perp}/z)}$, where $z$ is the light-cone
momentum fraction of the produced hadron of transverse mass $m_{\perp}$ with
respect to the fragmenting string. The Zhang’s Parton Cascade (ZPC) model
Zhang:1997ej , which calculates parton-parton scattering using cross sections
$\sigma\approx\frac{9\pi\alpha_{s}^{2}}{2\mu^{2}}$ based on a Debye screening
mass $\mu$, is used to simulate the evolution of the partonic phase. After the
partons stop interacting in ZPC, the quarks are subjected to a hadronization
process based on the quark coalescence model, which combines the nearest
quarks in coordinate space into hadrons. The hadrons formed during quark
coalescence are then subjected to hadronic stage evolution, which is handled
by a relativistic transport (ART) model Li:1995pra  with the input cross
section for various hadron-hadron scattering channels.

In this work we use the program version with string melting AMPT model. We
have conducted approximately 10 million AMPT events for d-Au collisions at
$\sqrt{s_{\rm NN}}=200$ GeV and pp collisions at $\sqrt{s}=200$ GeV. The event
centrality in $d$$+$Au is determined by impact parameter $b$ from AMPT events.
We use central, mid-central and peripheral event samples comprising the top
5%, 10–20%, 20–30%, 30–40%, 40–50% and 50–88% collision centrality intervals
of the total charge particle elliptic flow ($v_{\rm 2}$) distributions,
respectively.

###  II.2 Definition of anisotropic flow

Typically, the magnitude of azimuthal anisotropies is quantified using a
Fourier decomposition of the particle azimuthal distribution given by

${{\rm d}^{2}N\over{{\rm d}p_{\rm T}{\rm d}\varphi}}={1\over 2\pi}{{\rm
d}N\over{\rm d}p_{\rm T}}\bigg{(}1+2\sum_{{n}=1}^{\infty}v_{n}(p_{\rm T}){\rm
cos}[{n}(\varphi-\Psi_{n})]\bigg{)},$ (1)

where $\varphi$ and $p_{\rm T}$ are the particle azimuthal angle and
transverse momentum, respectively. The anisotropy of produced particles is
defined by the Fourier coefficients $v_{n}$ Voloshin:1994mz  and the azimuthal
angle of the symmetry plane for the $n^{\rm th}$ harmonic is denoted by
$\Psi_{n}$. The largest contribution to the asymmetry of collisions is
provided by the second Fourier coefficient $v_{2}$ referred to as elliptic
flow Ollitrault:1992bk ; Voloshin:1994mz .

##  III Nonflow contribution subtraction method

The method using two-particle correlations to extract the azimuthal anisotropy
is extensively discussed in Refs. CMS:2012qk ; Abelev:2012ola ; Aad:2012gla ;
ABELEV:2013wsa ; Aaboud:2016yar ; Aaij:2015qcq ; Adam:2015bka ;
Acharya:2017tfn ; Acharya:2018dxy . The correlation between two particles
(denoted trigger and associated particle) is measured as a function of the
azimuthal angle difference $\Delta\phi$ (defined within $-\pi/2$ and $3\pi/2$)
and pseudorapidity difference $\Delta\eta$. While the trigger particles are
charged particles, the analysis is done for charged associated particles
(denoted $h-h$). In this work, we follow the analysis in experiments at the
RHIC energy Adare:2013piz . In AMPT events, charged hadrons with
$0.5<p_{T}<3.5$ GeV/$c$ are used, each pair includes at least one particle at
low $p_{T}$ ($0.5<p_{T}<0.75$ GeV/$c$). To minimize the contribution from
small-angle correlations pairs are restricted to pseudorapidity separations of
$0.48<|\Delta\eta|<0.7$. The correlation is expressed in terms of $Y$, the
associated yield per trigger particle defined as:

$Y=\frac{1}{N_{\rm trig}}\frac{{\rm d}N_{\rm assoc}}{{\rm d}\Delta\phi},$ (2)

where $N_{\rm trig}$ is the total number of trigger particles in the event
class and $p_{\rm T}$ interval, $N_{\rm assoc}$ is the total number of
associatied particles in the event class and $p_{\rm T}$ interval.

We use the zero-yield-at-minimum (ZYAM) method Ajitanand:2005jj , where one
assumes that the number of correlated pairs is zero at the correlation
function minimum. This background contribution is obtained for the central,
mid-central, peripheral and pp collisions samples by performing fits to the
conditional yields using a functional form composed of a constant pedestal and
two Gaussian peaks, centered at $\Delta\phi=0$ and $\pi$. The minimum of this
function, $b_{\rm ZYAM}$, is subtracted from the conditional yields, and the
result is: $Y(\Delta\phi)=\frac{1}{N_{\rm trig}}\frac{dN_{\rm
assoc}}{d\Delta\phi}-b_{\rm ZYAM}$. The conditional yields $Y_{\rm
c}(\Delta\phi)$, $Y_{\rm mc}(\Delta\phi)$, $Y_{\rm p}(\Delta\phi)$ and $Y_{\rm
pp}(\Delta\phi)$ are related to central, mid-central, peripheral and pp
collisions events, respectively. Their difference are $\Delta Y_{\rm
cp}(\Delta\phi)=Y_{\rm c}(\Delta\phi)-Y_{\rm p}(\Delta\phi)$, $\Delta Y_{\rm
mcp}(\Delta\phi)=Y_{\rm mc}(\Delta\phi)-Y_{\rm p}(\Delta\phi)$, $\Delta Y_{\rm
cpp}(\Delta\phi)=Y_{\rm c}(\Delta\phi)-Y_{\rm pp}(\Delta\phi)$ and $\Delta
Y_{\rm mcpp}(\Delta\phi)=Y_{\rm mc}(\Delta\phi)-Y_{\rm pp}(\Delta\phi)$ are
associated with subtracting the per-trigger yield distribution in peripheral
d-Au collisions or pp collisions, respectively. These subtraction removes any
centrality independent correlations, such as effects from jet correlations,
resonance decays, and so on.

Fourier coefficients can be extracted from the $\Delta\phi$ projection of the
per-trigger yield by a fit with:

$\frac{1}{N_{\rm trig}}\frac{{\rm d}N_{\rm assoc}}{{\rm
d}\Delta\phi}=a_{0}+\sum_{{\rm n}=1}^{3}2a_{\rm n}{\rm cos}({\rm
n}\Delta\phi)$ (3)

To quantify the relative amplitude of the azimuthal modulation, we define

$c_{n}=a_{n}/\left(b^{c}_{\rm ZYAM}+a_{0}\right),$ (4)

where $b^{c}_{\rm ZYAM}$ is $b_{\rm ZYAM}$ in central events Ajitanand:2005jj
.

The method using two-particle correlations to the $v_{n}^{h}\\{{\rm 2PC}\\}$
coefficient of order $n$ for a particle $h$ is defined as ABELEV:2013wsa :

$v_{n}^{h}\\{{\rm 2PC}\\}=\sqrt{c_{n}^{h-h}}$ (5)

##  IV Results and Discussions

Figure 1: Charged particles azimuthal conditional yields
$Y\left(\Delta\phi\right)$ for 0%–5% most central ($Y_{\rm c}(\Delta\phi)$,
orange circles) and 50%–85% peripheral ($Y_{\rm p}(\Delta\phi)$, blue squares)
collisions with $0.48<|\Delta\eta|<0.7$. Difference
$\Delta$$Y\left(\Delta\phi\right)$ between most central and peripheral d-Au
collisions ($\Delta Y_{\rm cp}(\Delta\phi)=Y_{\rm c}(\Delta\phi)-Y_{\rm
p}(\Delta\phi)$, black triangle-ups), which is fit to
$a_{0}+2a_{2}\cos(2\Delta\phi)$(red curve), where $a_{0}$ and $a_{2}$ are
computed from the AMPT events.  Figure 2: Charged particles azimuthal
conditional yields $Y\left(\Delta\phi\right)$ for 0%–5% most central ($Y_{\rm
c}(\Delta\phi)$, orange circles) and pp ($Y_{\rm pp}(\Delta\phi)$, blue
squares) collisions with $0.48<|\Delta\eta|<0.7$. Difference
$\Delta$$Y\left(\Delta\phi\right)$ between most central d-Au collisions and pp
collisions ($\Delta Y_{\rm cpp}(\Delta\phi)=Y_{\rm c}(\Delta\phi)-Y_{\rm
pp}(\Delta\phi)$, black triangle-ups), which is fit to
$a_{0}+2a_{2}\cos(2\Delta\phi)$(red curve), where $a_{0}$ and $a_{2}$ are
computed from the AMPT events.

From Eq. (2), the charged particles conditional yields $Y_{\rm
c}(\Delta\phi)$, $Y_{\rm p}(\Delta\phi)$ and $Y_{\rm pp}(\Delta\phi)$(0%–5%
most central, peripheral and pp collisions events, respectively) are shown in
Fig. 1 and Fig. 2 , along with their difference $\Delta Y_{\rm
cp}(\Delta\phi)=Y_{\rm c}(\Delta\phi)-Y_{\rm p}(\Delta\phi)$ and $\Delta
Y_{\rm cpp}(\Delta\phi)=Y_{\rm c}(\Delta\phi)-Y_{\rm pp}(\Delta\phi)$ are
expressed as subtracting the per-trigger yield distribution in peripheral d-Au
collisions at $\sqrt{s_{\rm NN}}=200$ GeV or pp collisions at $\sqrt{s}=200$
GeV, respectively. It is worth noting that any signal in the peripheral events
and pp events is subtracted from the signal in the central events. For
$\Delta\phi$ near 0 and $\pi$, $Y_{c}(\Delta\phi)$ is significantly larger
than $Y_{\rm p}(\Delta\phi)$ and $Y_{\rm pp}(\Delta\phi)$.

We discover that the distinction with 0–5% most central, 50–85% peripheral
collisions and pp collisions are well described by the Eq. (3) as demonstrated
in Fig. 1 and Fig. 2. The charged particles coefficients $a_{n}$ are computed
from the $\Delta Y(\Delta\phi)$ distributions as: $a_{n}=\langle\Delta
Y(\Delta\phi)\cos(n\Delta\phi)\rangle$. The bracket $\langle...\rangle$
denotes an average over particles in the event sample with $\Delta\phi$.

Figure 3: With $0.5<p^{t}_{T}<0.75$ GeV/$c$, $0.48<|\Delta\eta|<0.7$, the
charged particles $2$th-order pair anisotropy $c_{2}$ of the 0–5% most central
collision excess as a function of associated particle $p_{\rm T}$. The blue
solid line is for $c_{2}$ of the 0–5% most central d-Au collisions with
nonflow effects. The nonflow effects for 0–5% most central d-Au collisions
reduced by subtracting the per-trigger yield distribution in peripheral d-Au
collisions (green dotted line) or pp collisions (red dashed line). The $c_{2}$
of the central d-Au collision in the data is also shown (black circles) from
Ref. Adare:2013piz .

From Eq. (4),charged particles $c_{2}$ is shown as a function of associated
$p_{\rm T}$ in Fig. 3 for central (0–5%) d-Au collisions. With the nonflow
effect, the blue solid line is higher than red dashed line or green dotted
line (without the nonflow effect). The nonflow effects for 0–5% most central
d-Au collisions reduced by subtracting the per-trigger yield distribution in
peripheral d-Au collisions (green dotted line) or pp collisions (red dashed
line) are very similar. The charged particles $c_{2}$ from AMPT events and
data are consistent from 0.5 to 2.0 GeV/$c$, then $c_{2}$ from AMPT events is
higher than from data.

Figure 4: In upper panel, the charged particles elliptic flow $v_{2}$ of the
0–5% most central d-Au collision excess as a function of associated particle
$p_{\rm T}$. The blue solid line is for $v_{2}$ of the 0–5% most central d-Au
collisions with nonflow effects. The nonflow effects for 0–5% most central
d-Au collisions reduced by subtracting the per-trigger yield distribution in
peripheral d-Au collisions (green dotted line) or pp collisions (red dashed
line). The $v_{2}$ of the central d-Au collision in the data is also shown
(black circles) from Ref. Adare:2013piz , which obtained under the assumption
of factorization: $c_{2}\left(p^{t}_{\rm T},p^{a}_{\rm
T}\right)=v_{2}\left(p^{t}_{\rm T}\right)v_{2}\left(p^{a}_{\rm T}\right)$. In
bottom panel, the ratio between charged particles $v_{2}$ of most central
collisions reduced by subtracting the per-trigger yield distribution in
peripheral d-Au collisions and pp collisions is shown by cyan shortdash-dotted
line, the ratio between $v_{2}$ of most central d-Au collisions reduced by
subtracting the per-trigger yield distribution in peripheral d-Au collisions
and without nonflow subtracting is shown by pink longdash-dotted line.  Figure
5: The charged particles elliptic flow $v_{2}$ of the 10–20%, 20–30%, 30–40%
and 40–50% collision excess as a function of associated particle $p_{\rm T}$.
The blue solid line is for charged particles $v_{2}$ of the different
centrality collisions with nonflow effects. The nonflow effects reduced by
subtracting the per-trigger yield distribution in peripheral d-Au collisions
(green dotted line) or pp collisions (red dashed line).

From Eq. (5),charged particles $v_{2}$ is shown as a function of associated
$p_{\rm T}$ in Fig. 4 for central (0–5%) d-Au collisions. We can observe the
charged particles $v_{2}$ value increase with $p_{\rm T}$ distribution, this
is because at higher $p_{\rm T}$, the particles are more affected by
reflecting the initial spatial anisotropy of the collision geometry. With the
nonflow effect, the blue solid line is higher than red dashed line or green
dotted line (without the nonflow effect). The nonflow effects for 0–5% most
central d-Au collisions reduced by subtracting the per-trigger yield
distribution in peripheral d-Au collisions (green dotted line) or pp
collisions (red dashed line) are very similar. This could indicate that the
nonflow effects (such as jet correlations and resonance decays) in the 0-5%
most central d-Au collisions are not strongly dependent on the subtracting the
per-trigger yield distribution in peripheral d-Au collisions or pp collision
system. The charged particles $v_{2}$ from AMPT events are in agreement with
data from 0.5 to 1.2 GeV/$c$, then $v_{2}$ from AMPT events is lower than data
from 1.2 to 2.5 GeV/$c$.

In bottom panel of Fig. 4 , we can observe the ratio between charged particles
$v_{2}$ of most central d-Au collisions reduced by subtracting the per-trigger
yield distribution in peripheral d-Au collisions and pp collisions is shown by
cyan shortdash-dotted line, the maximum deviation is less than 2%. The ratio
between $v_{2}$ of most central d-Au collisions reduced by subtracting the
per-trigger yield distribution in peripheral d-Au collisions and without
nonflow subtracting is shown by pink longdash-dotted line, the ratio value at
higher $p_{\rm T}$ is larger than at lower $p_{\rm T}$ and the maximum
deviation is less than 6%. It is worth noting that the nonflow contribution
can vary with $p_{\rm T}$ due to different physical mechanisms that contribute
to particle production in different $p_{\rm T}$ ranges. For example, at low
$p_{\rm T}$, the nonflow contribution may be dominated by resonance decays,
while at high $p_{\rm T}$, the nonflow contribution may be dominated by jet-
like correlations. Therefore, studying the nonflow contribution ratio as a
function of $p_{\rm T}$ can provide important insights into the underlying
physics of particle production in heavy-ion collisions.

The charged particles $v_{2}$ is shown as a function of associated $p_{\rm T}$
in Fig. 5 at 10–20%, 20–30%, 30–40% and 40–50% d-Au collision centrality
intervals. We can find the nonflow effects in the mid-central d-Au collisions
are not strongly dependent on the subtracting the per-trigger yield
distribution in peripheral d-Au collisions or pp collision system. On other
hand, we observe the nonflow effects at different collision centrality
intervals reduced by subtracting the per-trigger yield distribution in
peripheral d-Au collisions (green dotted line) or pp collisions (red dashed
line) are very similar.

##  V Summary

In this work, a multiphase transport model(AMPT) has been used to
comprehensively study the behavior of elliptic flow($v_{\rm 2}$) for d-Au
collisions at $\sqrt{s_{\rm NN}}=200$ GeV. Two-particle angular correlations
of charged particles have been calculated in d-Au collisions at
$\sqrt{s_{\mathrm{NN}}}=200$ GeV and expressed as associated yields per
trigger particle. The Fourier coefficient $v_{2}$ was extracted from these
correlations and studied as a function of $p_{\rm T}$. The nonflow effects for
central/mid-central d-Au collisions at $\sqrt{s_{\rm NN}}=200$ GeV reduced by
subtracting the per-trigger yield distribution in peripheral d-Au collisions
at $\sqrt{s_{\rm NN}}=200$ GeV or pp collisions at $\sqrt{s}=200$ GeV. Both
techniques give compatible results. Discussions about comparisons with
measurements from d-Au collisions at $\sqrt{s_{\rm NN}}=200$ GeV are included.
We can find the nonflow effects (such as jet correlations and resonance
decays) in the central and mid-central d-Au collisions are not strongly
dependent on the subtracting the per-trigger yield distribution in peripheral
d-Au collisions or pp collision system. By subtracting the per-trigger yield
distribution in peripheral d-Au collisions and without nonflow subtraction,
the ratio of $v_{2}$ for most central d-Au collisions shows a larger value at
higher $p_{\rm T}$ than at lower $p_{\rm T}$. The maximum deviation is less
than 6%. By analyzing the nonflow contribution ratio as a function of $p_{\rm
T}$, we can gain valuable insights into the underlying physics of particle
production in heavy-ion collisions. At low $p_{\rm T}$, resonance decays may
dominate the nonflow contribution, whereas at high $p_{\rm T}$, jet-like
correlations may be the main contributor. On the other hand, we have observed
nonflow effects in different collision centrality intervals. These effects
have been reduced by subtracting the per-trigger yield distribution in
peripheral d-Au collisions or pp collisions, which are remarkably similar.
Therefore, our work provides a reference for researchers looking into nonflow
contribution subtraction in small collision system experiments.

The authors appreciate the referee for his/her careful reading of the paper
and valuable comments. This work is supported in part by the Natural Science
Foundation of Henan Province (No.212300410386), Key Research Projects of Henan
Higher Education Institutions (No.20A140024), and NSFC Key Grant 12061141008,
the Scientific Research Foundation of Hubei University of Education for Talent
Introduction (No. ESRC20220028 and No. ESRC20230002).

## References

  * (1) B. Muller and J. L. Nagle, “Results from the relativistic heavy ion collider”, Ann. Rev. Nucl. Part. Sci. 56 (2006) 93–135.
  * (2) A. Bazavov et al., “The chiral and deconfinement aspects of the QCD transition”, Phys. Rev. D85 (2012) 054503.
  * (3) J.-Y. Ollitrault, “Anisotropy as a signature of transverse collective flow”, Phys. Rev. D 46 (1992) 229–245.
  * (4) K. J. Eskola, H. Paukkunen, and C. A. Salgado, “EPS09: A New Generation of NLO and LO Nuclear Parton Distribution Functions”, JHEP 04 (2009) 065.
  * (5) B. Z. Kopeliovich, J. Nemchik, A. Schafer, and A. V. Tarasov, “Cronin effect in hadron production off nuclei”, Phys. Rev. Lett. 88 (2002) 232303.
  * (6) Z.-B. Kang, I. Vitev, E. Wang, H. Xing, and C. Zhang, “Multiple scattering effects on heavy meson production in p+A collisions at backward rapidity”, Phys. Lett. B 740 (2015) 23–29.
  * (7) CMS Collaboration, S. Chatrchyan et al., “Observation of Long-Range Near-Side Angular Correlations in Proton-Lead Collisions at the LHC”, Phys. Lett. B 718 (2013) 795–814.
  * (8) ALICE Collaboration, B. Abelev et al., “Long-range angular correlations on the near and away side in pPb collisions at $\sqrt{s_{\rm NN}}=5.02$ TeV”, Phys. Lett. B 719 (2013) 29–41.
  * (9) ATLAS Collaboration, G. Aad et al., “Observation of Associated Near-Side and Away-Side Long-Range Correlations in $\sqrt{s_{\rm NN}}$ = 5.02 TeV Proton-Lead Collisions with the ATLAS Detector”, Phys. Rev. Lett. 110 no. 18, (2013) 182302.
  * (10) ALICE Collaboration, B. B. Abelev et al., “Long-range angular correlations of $\rm\pi$, K and p in p–Pb collisions at $\sqrt{s_{\rm NN}}$ = 5.02 TeV”, Phys. Lett. B 726 (2013) 164–177.
  * (11) ATLAS Collaboration, M. Aaboud et al., “Measurements of long-range azimuthal anisotropies and associated Fourier coefficients for pp collisions at $\sqrt{s}=5.02$ and $13$ TeV and p+Pb collisions at $\sqrt{s_{\mathrm{NN}}}=5.02$ TeV with the ATLAS detector”, Phys. Rev. C 96 no. 2, (2017) 024908.
  * (12) LHCb Collaboration, R. Aaij et al., “Measurements of long-range near-side angular correlations in $\sqrt{s_{\rm NN}}$ = 5 TeV proton-lead collisions in the forward region”, Phys. Lett. B 762 (2016) 473–483.
  * (13) PHENIX Collaboration, A. Adare et al., “Quadrupole Anisotropy in Dihadron Azimuthal Correlations in Central $d$$+$Au Collisions at $\sqrt{s_{{}_{\rm NN}}}$ = 200 GeV”, Phys. Rev. Lett. 111 no. 21, (2013) 212301.
  * (14) PHENIX Collaboration, A. Adare et al., “Measurement of long-range angular correlation and quadrupole anisotropy of pions and (anti)protons in central $d$$+$Au collisions at $\sqrt{s_{{}_{NN}}}$ = 200 GeV”, Phys. Rev. Lett. 114 no. 19, (2015) 192301.
  * (15) STAR Collaboration, L. Adamczyk et al., “Long-range pseudorapidity dihadron correlations in $d$+Au collisions at $\sqrt{s_{\rm NN}}=200$ GeV”, Phys. Lett. B 747 (2015) 265–271.
  * (16) PHENIX Collaboration, A. Adare et al., “Measurements of elliptic and triangular flow in high-multiplicity 3He$+$Au collisions at $\sqrt{s_{{}_{\rm NN}}}=200$ GeV”, Phys. Rev. Lett. 115 no. 14, (2015) 142301.
  * (17) ATLAS Collaboration, G. Aad et al., “Measurement of long-range pseudorapidity correlations and azimuthal harmonics in $\sqrt{s_{NN}}=5.02$ TeV proton-lead collisions with the ATLAS detector”, Phys. Rev. C 90 no. 4, (2014) 044906.
  * (18) Vojtěch Pacík (for the ALICE Collaboration), “Elliptic flow of identified hadrons in small collisional systems measured with ALICE”, Nucl.Phys. A 982 (2019) 451-454.
  * (19) ATLAS Collaboration, G. Aad et al., “Transverse momentum and process dependent azimuthal anisotropies in $\sqrt{s_{\mathrm{NN}}}=8.16$ TeV p+Pb collisions with the ATLAS detector”, Eur. Phys. J. C 80 no. 1, (2020) 73.
  * (20) Z.-W. Lin, C. M. Ko, B.-A. Li, B. Zhang, and S. Pal, “A Multi-phase transport model for relativistic heavy ion collisions”, Phys. Rev. C 72 (2005) 064901.
  * (21) X.N. Wang, M. Gyulassy, “hijing: A Monte Carlo model for multiple jet production in pp, pA, and AA collisions”, Phys. Rev. D 44, (1991) 3501.
  * (22) B. Zhang, “ZPC 1.0.1: A Parton cascade for ultrarelativistic heavy ion collisions”, Comput. Phys. Commun. 109 (1998) 193–206.
  * (23) B.-A. Li and C. M. Ko, “Formation of superdense hadronic matter in high-energy heavy ion collisions”, Phys. Rev. C 52 (1995) 2037–2063.
  * (24) S. Voloshin and Y. Zhang, “Flow study in relativistic nuclear collisions by Fourier expansion of Azimuthal particle distributions”, Z. Phys. C 70 (1996) 665–672.
  * (25) ALICE Collaboration, J. Adam et al., “Forward-central two-particle correlations in p–Pb collisions at $\sqrt{s_{\rm NN}}$ = 5.02 TeV”, Phys. Lett. B 753 (2016) 126–139.
  * (26) ALICE Collaboration, S. Acharya et al., “Search for collectivity with azimuthal J/$\psi$-hadron correlations in high multiplicity p–Pb collisions at $\sqrt{s_{\rm NN}}$ = 5.02 and 8.16 TeV”, Phys. Lett. B 780 (2018) 7–20.
  * (27) ALICE Collaboration, S. Acharya et al., “Azimuthal Anisotropy of Heavy-Flavor Decay Electrons in $p$-Pb Collisions at $\sqrt{s_{\rm NN}}$ = 5.02 TeV”, Phys. Rev. Lett. 122 no. 7, (2019) 072301.
  * (28) Ajitanand, N. N. and Alexander, J. M. and Chung, P. and Holzmann, W. G. and Issah, M. and Lacey, Roy A. and Shevel, A. and Taranenko, A. and Danielewicz, P., “Decomposition of harmonic and jet contributions to particle-pair correlations at ultrarelativistic energies”, Phys. Rev. C 72, (2005) 011902.
  * (29) M. Luzum, “Collective flow and long-range correlations in relativistic heavy ion collisions”, Phys. Lett. B 696, (2011) 499.


# Comprehensive Video Understanding: Video summarization with content-based
video recommender design

Yudong Jiang1, Kaixu Cui1, Bo Peng2333Bo Peng is an intern in Xinhua Zhiyun
Technology Co., Ltd. when doing this work., Changliang Xu1
1Xinhua Zhiyun Technology Co., Ltd.
2Dept. of Statistics, Columbia University in the City of New York
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

Video summarization aims to extract keyframes/shots from a long video.
Previous methods mainly take diversity and representativeness of generated
summaries as prior knowledge in algorithm design. In this paper, we formulate
video summarization as a content-based recommender problem, which should
distill the most useful content from a long video for users who suffer from
information overload. A scalable deep neural network is proposed on predicting
if one video segment is a useful segment for users by explicitly modelling
both segment and video. Moreover, we accomplish scene and action recognition
in untrimmed videos in order to find more correlations among different aspects
of video understanding tasks. Also, our paper will discuss the effect of audio
and visual features in summarization task. We also extend our work by data
augmentation and multi-task learning for preventing the model from early-stage
overfitting. The final results of our model win the first place in ICCV 2019
CoView Workshop Challenge Track.

##  1 Introduction

Figure 1: The whole picture of our video summarization system. Feature
extractor can be a model trained on different video understanding tasks.
Summarization system use viewers’ feedback and features from different source
to generate the video summary. Users’ feedback is treated as supervised
signals but not a necessary condition in the prediction process.

As information overload becomes a more and more serious topic in modern
society, many efficient tools have been designed to overcome “information
anxiety”. Videos, the fastest-growing information carrier, will account for
more than 80% of all Internet traffic by 2020 [3]. Video summarization aims to
address this problem by extracting keyframes/shots from a long video, which
contains the most useful content for people. It serves as a vital way of
comprehensively understanding video data in research area while saving
viewers’ time on information acquisition. This technology also gains the
attention of both the academic world and the industry. Adobe has already
presented video summarization feature in their products for video editing.
Some cloud computing companies, like Microsoft Azure, AliCloud, provide this
function as an online service.

One of the main challenges in video summarization is subjectivity, different
people may have different selections on key shots for a same video. We
investigate two top referenced dataset SumMe [6] and TVSum [15], which both
provide manually curated consistency benchmark, with F1 score of 0.31 for
SumMe, F1 score of 0.36 for TVSum. Based on the assumptions above, we don’t
take diversity or representativeness and other subjective attributes into
consideration, instead, we focuse on solely the key shots and the full video.
The mission of our algorithm is to find the most attractive video segments for
a group of annotators/users when they watch a long video.

Content-based recommender modelling is one of the most important technology in
recommendation system, especially for alleviating cold start problem, which
recommend items with similar content to what users like. It is usually
formalized as a similarity learning problem [13]. A content-based video
recommender deep model learns compact representation of videos and constructs
a bridge between users’ feedback and video semantics information. We develop
and deploy the HighlightNet to learn annotators’ preferences, the model is
supervised by the segment’s importance feedbacks. It can easily combine
different inputs, such as raw features, high-level features, audio features
and vision feature together under this framework. The holistic picture of our
work is shown in figure 1.

Currently, as video summarization problem appeals to a lot of researchers,
many state-of-the-art approaches have been presented in solving this problem
[3, 24, 25, 26]. Generally, they treat video summarization as sequence-to-
sequence learning problem. Since RNN and its variants LSTM, GRU are very
efficient in modelling long time dependencies under encoder-decoder
architectures, many works in machine translation, image/video captioning,
reading comprehension adopt these technologies. In this paper, GRU modelling
is adopted in order to consider both the isolated segment, and its roles in
the whole video.

Video summarization, as one of the comprehensive video understanding tasks, is
extremely difficult and requires a large amount of data in deep learning
architecture. However, the collection of such summarization labels is time-
consuming and labor-intensive, resulting in an insufficient dataset. Since
supervised learning can easily overfit on small data, many methods have been
explored on alleviating this issue. Many state-of-the-art videos summarization
work address this problem by unsupervised learning, semi-supervised learning,
or multi-task learning [25, 26, 28]. We explore the self-supervised learning
on video sequence modelling and joint training with segments importance score
prediction. Another way to confront overfitting is to shuffle information
flows which potentially improves the model’s generalization ability. The
contribution of our work:

[1] We unify various inputs on different semantic levels to one framework by
formatting video summarisation into a recommender problem.

[2] We develop an algorithm that models independent segments and segment
sequence(whole video).

[3] We extend the summarisation framework with self-supervised learning and
data augmentation to deal with lack of labelled data.

##  2 Related Work

Video classification, as a fundamental task for video understanding, has been
studied seriously. Many high-quality datasets [1, 10, 11, 16]are published
which drives the research to a higher level. Supervised learning brings the
performance of algorithms nearly to human performance on large-scale video
classification tasks [4, 7, 9, 18, 19, 20, 21].

A significant number of deep learning based frameworks have been explored
recently in solving video summarization [3, 22, 24, 26, 28]. K. Zhang et al.
creatively applied LSTM in supervised video sequence labelling to model video
temporal information with good performance [24]. Jiri Fajtl et.al introduced
self-attention instead of RNN to enhance computation efficiency [3]. K. Zhou
et al. showed that fully unsupervised learning can outperform many supervised
methods by considering diversity and representativeness in reinforcement
learning-based framework [28]. Y. Zhang et al. introduced adversarial loss to
video summarization which learns a dilated temporal relational generator and a
discriminator with three-player loss [26].

Researchers believe learning good visual representation can help deep neural
network improve both fitting and generalization ability, especially in
classification task [5, 12, 14, 17]. Some video summarization work adopts
unsupervised learning as an auxiliary task to help to improve the supervised
learning system’s performance. K. Zhang et al. used “retrospective encoder”
that embeds the predicted summary and original video into the same abstract
semantic space with closer distance. The semi-supervised settings help
increase F-score on TVSum dataset from 63.9% to 65.2% [25]. K. Zhou et al. on
the other hand, extending their reinforcement learning based framework to
semi-supervised style elevated performance on both SumMe and TVSum comparing
to purely supervised or unsupervised methods [28].

For the subjectivity of video summarization task, Kanehira et al. provided a
solution by building a summary that depends on the particular aspect of a
video the viewer focuses on [8]. Joonseok Lee et.al from Google published a
content-only video recommendation system which we regarded as a work close to
video summarization [13].

##  3 Approach

In this section, we detail our method on CoView 2019 comprehensive video
understanding challenge track. First, we simply introduce our work on action
and scene recognition in untrimmed video. Second, we formalize the
summarization problem and present our solution based on deep neural network.
Third, we talk about some important technology on how to prevent the DNN from
early-stage overfitting.

###  3.1 Action and Scene Recognition in Untrimmed Video

We first adopt I3D with non-local blocks video classification [2, 21] in this
subtask. A ResNet101 backbone is used for this framework. We tried two ways of
training:

1\. Action and scene recognition task sharing the same backbone with two
SoftMax loss branches for each classification tasks.

2\. We get one model each for action and scene recognition.

Since the training data just comes from only 1000 videos, the scene data for
training is too small and lacks variation for recognition. There are many
public image scene classification datasets such as Place360 [27] and SUN [23]
for image scene recognition. We assume the video scene classification task can
be trained by image classification without much information loss. We can
easily make full use of public datasets to potentially improve models’
robustness. We use average pooling operation on image classification
predictions for final video level results. To capture the advantages of both
video-based and image-based models, we ensemble the output scores of two
models together with logistic regression.

###  3.2 Video Summarization Framework

We take the video summarization task as a content-based recommender problem.
Let

$V_{m}=\left[s_{1},s_{2},s_{3},...,s_{T_{N}}\right]^{T}\in{\mathbb{R}}^{T_{N}\times
sd},$

be the feature matrix for a video m with $T_{N}$ segments in $sd$ dimension.

Figure 2: The Video Summarization Network structure. It contains three
subnetworks including: SegNet, VideoNet, and HighlightNet.

For each segment we have a mean importance score $a_{i}$ as users’ feedback,
can be written by

$A_{m}=\left[a_{1},a_{2},a_{3},...,a_{T_{N}}\right]^{T}\in{\mathbb{R}}^{T_{N}\times
1}.$

Our goal is to select segments from the video by Top-k highest importance
prediction score as summarization set $\left\\{S\right\\}_{k}$. So the problem
can be defined as finding a ranking function $f\left(s\right)$ that can
predict the segment importance score in video segments sequence

$\hat{A_{m}}=f\left(V_{m}\right).$

A loss function can be defined by mean-square error(MSE),

$L({A}_{m}\quad,\hat{A_{m}})=\frac{1}{H\times
T_{N}}\sum_{m=1,i=1}^{H,T_{N}}{\left(\left(A_{m}^{i}-f(V_{m}^{i})\right)^{2}\right)},$

where H is the number of videos in training set, our algorithm is to find
optimal function $f$ that minimizes the overall loss:

$\hat{f}=\underset{R(f)}{\quad argmin}\quad L(f),$

in the prediction model space $R$.

Since $s_{i}$ lacks information from the whole video sequence as the input of
$f$, we set ${V_{m}}^{\prime}$ defined by

${V_{m}}^{\prime}=V_{m}\oplus V_{s},$

$\oplus$ is the operator that concatenate each row of $V_{m}$ to $V_{s}$,
$V_{s}$ is a $T_{N}\times vd$ matrix that represent $vd$ dimension whole video
feature to each segment, which can also be taken as a learnable function of
$V_{s}=\phi(V_{m})$. In this work, we set the same value for each row of
$V_{s}$.

We can also get sequence descriptors for one segment, like image frame, audio
frame, a segment is defined by

$S_{p}=\left[f_{1},f_{2},f_{3},\ldots,f_{T_{G}}\right]^{T}\in{\mathbb{R}}^{T_{G}\times
fd}.$

Another learnable function $V_{r}=\theta(S_{p})$ is used to map frames
sequence features from $T_{G}\times fd$ to $1\times sfd$ space. We combine
image based feature $V_{r}$ and segment based feature $V{w}$ by learnable
function $\varphi$ to final segment-level feature

$V_{m}=\varphi(V{r},V{w}).$

We can minimize $f$, $\phi$, $\theta$, $\varphi$ in one framework by
minimizing

$\begin{split}L(f)&=L(f({V_{m}}^{\prime}))\\\ &=L(f(V_{m}\oplus V_{s}))\\\
&=L(f(V_{m}\oplus\phi(V_{m})))\\\
&=L(f(\varphi(\theta(S_{p}),V_{w})\oplus\phi(\varphi(\theta(S_{p}),V_{w})))).\end{split}$

We design a video summarization network which consists of three subnetworks,
SegNet for function $\theta$ and $\varphi$, VideoNet for $\phi$, and
HighlightNet for $f$ as shown in figure 2.

As we utilize ImageNet features for each sampled video frame, SegNet combines
all frame features’ sequence to a frame-based vector. SegNet also fuses frame-
based feature and segment-level feature together. The SegNet is designed to
process either 2D or 3D frame sequence. For 2D features, we adopt temporal-
convolution and pooling to squeeze temporal dimension. And for 3D features, we
use three 3D convolution blocks. The 2D convolution blocks adopt “bottleneck”
structure for spatial fully connected layers(figure 3).

Figure 3: The SegNet structure details. The network is designed to process
either 2D or 3D frame sequence. Only one output(output1 or output2) is taken
as the segment embedding.

For each long video, the segments’ feature sequence generated by SegNet are
taken as inputs to VideoNet for video-level modelling. VideoNet use Bi-GRU for
sequence encoding. The final GRU unit hidden layer, which is a fixed-length
context vector, is taken as the output of VideoNet representing the video-
level feature.

Each segment-level feature concatenates with the video-level feature of the
same long video as the final segment representation. The representation
contains not only the independent video segment information, but also the
whole video sequence information, which is passed to highlight subnetwork to
predict the segment importance score. Figure 4 shows the structure of
highlight subnetwork.

Figure 4: The HighlightNet structure details. The network is designed to
predict the segment importance score by fusion features.

###  3.3 Multi-task Learning and Data Augmentation for Video Summarization
Network

We consider an auxiliary self-supervised task to better modelling video
sequence. Some segments are selected from one video by the fixed proportion
before inputting to the network. We shuffle the selected segments and train
the network to distinguish the odd-position segments as shown in figure 5.
This operation assumes that a good video sequence encoder has the ability to
model the right segments order. We control the difficulty of the task by only
indicate the odd-position segments but not sort the shuffled sequence to right
order. Parameters $\alpha$ and $\beta$ are used for adjusting the shuffled
segments‘ proportion and the weight of self-supervised task in multi-task
learning. The final loss can be defined by:

$L(f)=L(f({V_{m}}^{\prime})+\beta*\delta(shuffle(\alpha,V_{m}))).$

The advantages of applying multi-task learning are:

1\. Learning several tasks simultaneously can suppress early-stage
overfitting, by sharing the same representations.

2\. The auxiliary task is helpful in providing more useful information.

3\. Our method implicitly performs data augmentation since we shuffle the
input video sequence, which may improve the robustness of the algorithm.

4\. This method utilize unlabelled data for video sequence modelling.

Figure 5: Self-taught learning for odd-position labelling. The input segment
sequence are shuffled randomly by $\alpha$ ratio. The video encoder learn to
find the odd-position segments in the sequence.

A data augmentation method is proposed which only choose a portion of
information from each segment when modelling the whole video each time. After
modelling all portion of segments, we average the embeddings from different
portion to one vector(figure 6).

Figure 6: Data augmentation method. VideoNet only models a portion of each
segment to video embedding. Multiple embeddings are combined by average
pooling.

##  4 Experiments

In this section, we provide our comparison experiment on CoView 2019 dataset.
The dataset consists of 1200 videos for training and 300 for testing. Videos
are sampled from Youtube-8M, Dense-Captioning, and Video summarization
dataset. Every video from the dataset is segmented into a set of 5-second-long
segments and asked 20 users to annotate importance score and 99 action / 79
scene labels. The average of importance scores and the most voted action/scene
labels are provided as ground-truth in segment-level. Before any experiments,
we randomly split videos into training(1000) and validation (200) set based on
original videos instead of five-seconds videos.

###  4.1 Scene and Action Classification in Untrimmed Video

For scene/action classification task, top-5 accuracy on validation results
show in table 1. a. joint training I3D+non-local attention with two SoftMax
branches(JT). b. independent I3D+non-local model(IT). c. independent I3D+non-
local model with external action classification data(AED). d. ResNet-50 with
external scene image classification data(SED).

Method | Scene Top-5 | Action Top-5
---|---|---
a (JT) | 82.76 | 81.72
b (IT) | 86.92 | 87.77
c (AED) | - | 84.86
d (SED) | 82.06 | -
Table 1: Top-5 accuracy of different models on validation.

We train I3D+non-local with the pre-trained model on Kinetics, and the
ResNet-50 pre-trained on ImageNet. The results indicate: 1. Although
scene/action classification task may have inner connection, both joint
training and ensemble failed to improve the results. 2. Pure 2D convolution-
based network method is less accurate than 3D convolution-based network on
both scene and action recognition task. 3. External data didn’t help improve
classification accuracy on validation set, but we still use it for potential
model generalization enhancement.

###  4.2 Video Summarization

We investigate our summarization network on different parameter scales, visual
and visual-audio feature inputs, single-task and multi-task learning, with and
without data augmentation.

Evaluation Protocol. The video summarization evaluation metric computes the
sum of importance scores of selected segments comparing with ground truth
summary segments. The importance score metric is defined as

$\frac{1}{N}\sum_{k=1}^{N}{\frac{\sum_{i=1}^{{N}_{s}}{ImportanceScore(k,\quad
Sub_{i})}}{\sum_{i=1}^{{N}_{s}}{ImportanceScore(k,\quad GT_{i})}}},$

where N is the number of test videos, $N_{s}$ is the number of summarized
segments. $ImportanceScore(k,GT_{i})$ is the importance score from the i-th
segment of the ground truth summary of the k-th video, and
$ImportanceScore(k,Sub_{i})$ is the importance score of the i-th segment of
the submitted summary for the k-th video. Importance Score is shared in both
ground truth and submitted summary. $N_{s}$ is set to 6 for all videos, the
top $N_{s}$importance segments are ground truth summary. We randomly choose
$N_{s}$ segments for 10 times as the baseline of summary score, the mean value
is 74.92%.

Embedding Dimension. We explore different embedding dimension for the network
by regularizing the same length of segNet and videoNet outputs. For each
segment, we sample 16 frames for image-based extractor and 32 frames for
video-based extractor. We use GoogleNet trained on ImageNet to extract image-
based feature and I3D+non-local blocks trained on Kinetics to extract video-
based feature. Table 2 shows our results for different feature length from 512
to 64. The 256-feature length is slightly better.

Feature Length | SummaryScore
---|---
Dim = 512 | 81.47
Dim = 256 | 81.81
Dim = 128 | 81.49
Dim = 64 | 81.19
Table 2: Summary score on different embedding dimension.

Semantic Features Combination. Image-based feature is extracted by image
classifier. We select three types of image classifier: 1. ResNet-50 trained on
CoView2019 scene classification task(R50_s). 2. ResNet-152 trained on
ImageNet(R152_i). 3. GoogleNet trained on ImageNet(G_i). Video-based feature
is extracted by video classifier. We choose two video classifier: 1. I3D+non-
local model trained on Kinetics(INK). 2. I3D+non-local model trained on
CoView2019 action recognition task(INA). Table 3 shows different visual
feature combination on feature length 256. Our model didn’t benefit a lot from
action and scene recognition task.

Feature Type | SummaryScore
---|---
G_i+INK | 81.81
G_i+INA | 81.06
R50_s+INA | 80.83
R152_i+INA | 80.99
R50_s+INK | 81.45
R152_i+INK | 81.45
Table 3: Summary score with different visual semantic feature combination.

Multi-Task Learning. Since the self-taught learning increases the learning
difficulty, we adopt 3-layers bi-GRU on video net and feature length equals to
512 with R50_s+INK features. Dropout and bottleneck structure are removed from
segNet. The self-taught learning is firstly trained for two days using videos
on CoView2019 with $\alpha=15\%$ shuffle ratio. We use the trained model, with
96.64% accuracy on self-taught task and 78.6% recall on odd order segments, as
pre-trained model in multi-task learning setting. In joint training, we set
$\alpha=0.02,\beta=1$, and $\alpha=0$ when evaluate on validation. Table 4
shows the comparison results, we evaluate three models: a. the baseline
supervised learning without pre-trained model(sup_no_prt). b. supervised
learning with self-taught pre-trained model(sup_with_prt). c. multi-task
learning with self-taught pre-trained model(multi_with_prt). The results shows
that multi-task setting is better than the baseline model with small margin.

Method | SummaryScore
---|---
a. (sup_no_prt) | 80.69
b. (sup_with_prt) | 80.98
c. (multi_with_prt) | 81.64
Table 4: Summary score with Multi-Task learning.

Data augmentation. The basic network setting is the same as R50_s+INK in
”Semantic Features Combination” experiment, we get 0.29% improvment from
81.45% to 81.74%.

Audio feature. We extract audio features MFCC, Chroma-gram for each segment by
using python audio processing package LibROSA. We concatenate the audio
features to 5450-dimension segment-level features before inputing to SegNet.
The audio fusion model builds bases on Multi-Task setting in ”Multi-Task
Learning” experiment. The result decrease from 81.64% to 80.42%, that may due
to increased model complexity caused by larger feature dimension without
important information.

Optical flow. Optical flow as a low-level feature describing video motion, is
shown as complementary information to RGB in action recognition task. Optical
flow feature sequence are obtained by using Gunnar Farneback’s algorithm to
extract 16 frames for each segment. We modified SegNet in Multi-Task setting
to 4 layers 3D convolution for processing optical flow features on temporal
and spectral. Same as audio feature, this modification decrease summary score
from 81.64% to 80.78%.

Model ensemble. We ensemble five models: base 256 feature length
model(81.81%), audio feature model(80.42%), optical flow model(80.78%), multi-
task model(81.64%), data augmentation model(81.74%) by linear regression. Then
we do model selection according to linear regression weights which direct us
choosing base 256-length model and data augmentation model ensemble
result(81.86%) on validation as the final submission.

##  5 Conclusion and Future Work

In this paper, we propose a scalable deep neural network for video
summarization in content-based recommender formulation, which predicts the
segments importance score by considering both segment and video level
information. Our work shows that data augmentation and Multi-Task learning is
helpful in solving the limitation of dataset problem. To better understand the
video content, we also perform action and scene recognition in untrimmed video
with state-of-the-art video classification algorithm. We experiment with
combinations of high-level visual semantic features, audio features and
optical flow, and concluded that visual semantic features play the most
important role in this summarization task.

There are some areas we can further explore to improve comprehensive video
understanding in the future. First of all, we didn’t benefit from action and
scene connection in both recognition and summarization task, which leaves us
room for better utilize this prior knowledge. Second, as we re-formalize video
summarization in recommender framework, some state-of-the-art recommendation
technologies can be introduced such as Collaborative Filtering, Factorization
Machine, Wide & Deep Learning and so on. Last but not the least, although
CoView2019, to our knowledge, provides the largest public video summarization
dataset, it is too small in the scale of deep learning. We need more, like
users’ action when browsing video websites, video language description, and
large-scale dataset to accomplish such complex task.

## References

  * [1] Sami Abu-El-Haija, Nisarg Kothari, Joonseok Lee, Paul Natsev, George Toderici, Balakrishnan Varadarajan, and Sudheendra Vijayanarasimhan.  Youtube-8m: A large-scale video classification benchmark.  arXiv preprint arXiv:1609.08675, 2016.
  * [2] Joao Carreira and Andrew Zisserman.  Quo vadis, action recognition? a new model and the kinetics dataset.  In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 6299–6308, 2017.
  * [3] Jiri Fajtl, Hajar Sadeghi Sokeh, Vasileios Argyriou, Dorothy Monekosso, and Paolo Remagnino.  Summarizing videos with attention.  In ACCV Workshops, 2018.
  * [4] Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He.  Slowfast networks for video recognition.  arXiv preprint arXiv:1812.03982, 2018.
  * [5] Basura Fernando, Hakan Bilen, Efstratios Gavves, and Stephen Gould.  Self-supervised video representation learning with odd-one-out networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 3636–3645, 2017.
  * [6] Michael Gygli, Helmut Grabner, Hayko Riemenschneider, and Luc Van Gool.  Creating summaries from user videos.  In European conference on computer vision, pages 505–520. Springer, 2014.
  * [7] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [8] Atsushi Kanehira, Luc Van Gool, Yoshitaka Ushiku, and Tatsuya Harada.  aware video summarization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7435–7444, 2018.
  * [9] Andrej Karpathy, George Toderici, Sanketh Shetty, Thomas Leung, Rahul Sukthankar, and Li Fei-Fei.  Large-scale video classification with convolutional neural networks.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 1725–1732, 2014.
  * [10] Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijayanarasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al.  The kinetics human action video dataset.  arXiv preprint arXiv:1705.06950, 2017.
  * [11] Hildegard Kuehne, Hueihan Jhuang, Estíbaliz Garrote, Tomaso Poggio, and Thomas Serre.  Hmdb: a large video database for human motion recognition.  In 2011 International Conference on Computer Vision, pages 2556–2563. IEEE, 2011.
  * [12] Hsin-Ying Lee, Jia-Bin Huang, Maneesh Singh, and Ming-Hsuan Yang.  Unsupervised representation learning by sorting sequences.  In Proceedings of the IEEE International Conference on Computer Vision, pages 667–676, 2017.
  * [13] Joonseok Lee and Sami Abu-El-Haija.  Large-scale content-only video recommendation.  In Proceedings of the IEEE International Conference on Computer Vision, pages 987–995, 2017.
  * [14] Ishan Misra, C Lawrence Zitnick, and Martial Hebert.  Shuffle and learn: unsupervised learning using temporal order verification.  In European Conference on Computer Vision, pages 527–544. Springer, 2016.
  * [15] Yale Song, Jordi Vallmitjana, Amanda Stent, and Alejandro Jaimes.  Tvsum: Summarizing web videos using titles.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 5179–5187, 2015.
  * [16] Khurram Soomro, Amir Roshan Zamir, and Mubarak Shah.  Ucf101: A dataset of 101 human actions classes from videos in the wild.  arXiv preprint arXiv:1212.0402, 2012.
  * [17] Nitish Srivastava, Elman Mansimov, and Ruslan Salakhudinov.  Unsupervised learning of video representations using lstms.  In International conference on machine learning, pages 843–852, 2015.
  * [18] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1–9, 2015.
  * [19] Du Tran, Heng Wang, Lorenzo Torresani, Jamie Ray, Yann LeCun, and Manohar Paluri.  A closer look at spatiotemporal convolutions for action recognition.  In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition, pages 6450–6459, 2018.
  * [20] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal segment networks: Towards good practices for deep action recognition.  In European conference on computer vision, pages 20–36. Springer, 2016.
  * [21] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7794–7803, 2018.
  * [22] Huawei Wei, Bingbing Ni, Yichao Yan, Huanyu Yu, Xiaokang Yang, and Chen Yao.  Video summarization via semantic attended networks.  In Thirty-Second AAAI Conference on Artificial Intelligence, 2018\.
  * [23] Jianxiong Xiao, James Hays, Krista A Ehinger, Aude Oliva, and Antonio Torralba.  Sun database: Large-scale scene recognition from abbey to zoo.  In 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, pages 3485–3492. IEEE, 2010.
  * [24] Ke Zhang, Wei-Lun Chao, Fei Sha, and Kristen Grauman.  Video summarization with long short-term memory.  In European conference on computer vision, pages 766–782. Springer, 2016.
  * [25] Ke Zhang, Kristen Grauman, and Fei Sha.  Retrospective encoders for video summarization.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 383–399, 2018.
  * [26] Yujia Zhang, Michael Kampffmeyer, Xiaoguang Zhao, and Min Tan.  Dtr-gan: Dilated temporal relational adversarial network for video summarization.  In Proceedings of the ACM Turing Celebration Conference-China, page 89. ACM, 2019.
  * [27] Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba.  Places: A 10 million image database for scene recognition.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2017\.
  * [28] Kaiyang Zhou, Yu Qiao, and Tao Xiang.  Deep reinforcement learning for unsupervised video summarization with diversity-representativeness reward.  In Thirty-Second AAAI Conference on Artificial Intelligence, 2018\.