#  Video Quality Assessment Based on Swin TransformerV2 and Coarse to Fine
Strategy

Zihao Yu∗ Fengbin Guan∗ Yiting Lu∗ Xin Li∗ and Zhibo Chen∗
∗University of Science and Technology of China Hefei, Anhui, China
<EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

The objective of non-reference video quality assessment is to evaluate the
quality of distorted video without access to reference high-definition
references. In this study, we introduce an enhanced spatial perception module,
pre-trained on multiple image quality assessment datasets, and a lightweight
temporal fusion module to address the no-reference visual quality assessment
(NR-VQA) task. This model implements Swin Transformer V2 as a local-level
spatial feature extractor and fuses these multi-stage representations through
a series of transformer layers. Furthermore, a temporal transformer is
utilized for spatiotemporal feature fusion across the video. To accommodate
compressed videos of varying bitrates, we incorporate a coarse-to-fine
contrastive strategy to enrich the model’s capability to discriminate features
from videos of different bitrates. This is an expanded version of the one-page
abstract.

##  1 Introduction

With the increase in video resolution, the necessity to process high-
resolution video has become paramount. To address this issue, we employ Swin
TransformerV2[1] for spatial perception. The advantage of Swin TransformerV2
lies in its ability to efficiently process large-scale data while maintaining
computational efficiency. However, videos with different compression bitrates
may pose challenges to the model’s distinguishable capabilities.
For compression distortion, we propose a coarse-to-fine contrastive strategy
to equip the model with the ability to handle compressed videos of varying
bitrates. From the global perspective, we introduce a group contrastive
loss[2]. This loss function empowers the model to coarsely discern differences
in videos when processing different bitrates. However, group contrast loss may
cause the model to lack sufficient discriminative ability when processing
videos with the same bitrate. To mitigate this issue, from the local
perspective, we introduce rank loss. By employing rank loss, we can ensure
that the model retains good discriminative ability when processing videos
within the same bitrate.

##  2 Methodology

In this section, we delineate our proposed model as illustrated in Fig. 1, a
Non-reference Video Quality Assessment (NR-VQA) method comprising an enhanced
spatial perception module and a lightweight temporal fusion module, equipped
with coarse-to-fine contrastive strategy.

Figure 1: An overview of our proposed method

Feature Extraction In the spatial feature extraction part, we feed the sampled
frames into SwinTransformerV2 [1]. SwinTransformerV2 is capable of obtaining
local features of video frames at different levels. After normalizing and
rescaling the local features through a feature pooling operation, we derive
the global frame-level features by fusing the local representations across a
series of transformer layers. Finally, we concatenate the local features and
global features along the channel dimension and feed them into a fully
connected layer to obtain the quality-aware features of the frames. This
method fully utilizes global and local information, which helps to improve the
perceptual ability of video quality assessment.
Coarse to Fine Contrastive Strategy In our model, we incorporate Group
Contrastive loss as part of the loss function. We minimize the Group
Contrastive loss that is defined as follows:

$\mathcal{L}_{i,j}^{gc}=-\log\frac{\exp\left(\operatorname{sim}\left(\boldsymbol{z}_{(i)},\boldsymbol{z}_{(j)}\right)/\tau\right)}{\sum_{k>(1-p)N}^{N}\exp\left(\operatorname{sim}\left(\boldsymbol{z}_{(i)},\boldsymbol{z}_{(k)}\right)/\tau\right)}$
(1) $\mathcal{L}^{gc}=\sum_{i=1}^{pN}\sum_{\begin{subarray}{c}j=1\\\ j\neq
i\end{subarray}}^{pN}\mathcal{L}_{i,j}^{gc}+\sum_{i=(1-p)N+1}^{N}\sum_{\begin{subarray}{c}j=(1-p)N+1\\\
j\neq i\end{subarray}}^{N}\mathcal{L}_{i,j}^{gc}$ (2)

Where $\tau$ is the temperature parameter, $p$ is a proportion parameter of
$0.5$, $N$ is the size of the input vector , and $\text{sim}(z_{i},z_{j})$ is
the similarity between vectors $z_{i}$ and $z_{j}$. The Group Contrastive loss
minimizes the feature similarity within the same bitrate group and maximizes
the feature similarity of the different bitrate groups. Through this approach,
we can make the features of videos with the same bitrate closer, and the
features of videos with different bitrates farther apart. This design enables
our model to effectively handle multi-bitrate compressed video quality
assessment.
We incorporate rank loss as part of the loss function to ensure that the model
retains the discriminative ability when processing videos with the same bit
rate. We minimize the rank loss that is defined as follows:

$\mathcal{L}^{Rank}=\sum_{i=1}^{n}\max(0,\text{margin}-\
(\text{pred}_{1}-\text{pred}_{2})_{i}(\text{label}_{1}-\text{label}_{2})_{i})$
(3)

Where $margin$ is a preset interval value, $pred$ is the predicted value, and
label is the label value. The total loss for our model is defined as:

$\mathcal{L}=\mathcal{L}^{MSE}+\mathcal{L}^{L1}+\lambda_{1}\
\mathcal{L}^{gc}+\lambda_{2}\ \mathcal{L}^{Rank}$ (4)

##  3 Experiment

Implementation details We pre-trained the spatial module on datasets such as
CLIVE[3], LIVE[4], KonIQ-10k[5], and KADID-10K[6]. The diversity and
extensiveness of these datasets provide our model with a wealth of training
samples for enhancing image-level perception.
For performance evaluation, we employ two commonly used criteria, namely
Spearman’s rank-order correlation coefficient (SROCC) and Pearson’s linear
correlation coefficient (PLCC). Both SROCC and PLCC range from 0 to 1, and a
higher value indicates a better performance.
During the pre-training phase, our spatial module was pre-trained using the
loss function and training scheme of TReS[7]. TReS enhances the feature
extraction capability of the model by using Relative Ranking and Self-
Consistency. We trained on an NVIDIA 1080Ti GPU, with a batch size of 8, using
an exponential decay scheme to adjust the learning rate, with a decay rate of
0.95.
Results As shown in Table 1, our spatial model achieved excellent results in
the BIQA task. In our study, due to the lack of annotation for compressed
video quality assessment, we decided to train and test on pseudo labels
created using VMAF. This approach allows us to explore the performance of
Video Quality Assessment (VQA) in an opinion-free manner.
As shown in Table 2, our model achieved better performance on the pseudo-label
dataset when using group contrastive loss and rank Loss. This indicates that
the Coarse to Fine Contrastive Strategy is helpful for VQA tasks on compressed
videos at different bit rates. This coarse-to-fine contrastive strategy
facilitates perception for multi-bitrate video quality at both global and
local level, empowering the model to gain an enhanced understanding and
representation of changes in multiple distortion levels. This subsequently
improves the predictive capabilities of the model.

##  4 Conclusion

In this work, we propose an enhanced spatial perception module and lightweight
temporal module implementing a coarse-to-fine contrastive strategy for
compressed video quality assessment. For spatial feature extraction, we pre-
train the spatial module on multiple blind image quality assessment (BIQA)
datasets, achieving performance comparable to UNIQUE [8] and TRES [7]. This
demonstrates the excellence of Swin Transformer V2 for image feature
extraction. Concurrently, we hold high expectations for the coarse-to-fine
contrastive strategy in the compressed video quality assessment across
multiple bitrates. We believe this can further enrich the performance for
compressed video quality assessment.

Table 1: Performance comparison measured by SRCC and PLCC, where bold entries indicate the top one results. | LIVE | KADID | LIVEC | KonIQ
---|---|---|---|---
Method | PLCC | SRCC | PLCC | SRCC | PLCC | SRCC | PLCC | SRCC
TReS[7] | 0.968 | 0.969 | 0.858 | 0.859 | 0.877 | 0.846 | 0.928 | 0.915
UNIQUE[8] | 0.968 | 0.969 | 0.876 | 0.878 | 0.890 | 0.854 | 0.901 | 0.896
MUSIQ[9] | 0.911 | 0.940 | 0.872 | 0.875 | 0.746 | 0.702 | 0.928 | 0.916
DEIQT[10] | 0.982 | 0.980 | 0.887 | 0.889 | 0.894 | 0.875 | 0.934 | 0.921
ours | 0.965 | 0.966 | 0.931 | 0.931 | 0.883 | 0.867 | 0.935 | 0.924
Table 2: Test Results of Coarse to Fine Contrastive Strategy Method | PLCC | SRCC
---|---|---
GC and Rank | 0.924 | 0.903
no GC and Rank | 0.919 | 0.900

##  5 References

## References

  * [1] Ze Liu, Han Hu, Yutong Lin, Zhuliang Yao, Zhenda Xie, Yixuan Wei, Jia Ning, Yue Cao, Zheng Zhang, Li Dong, et al.,  “Swin transformer v2: Scaling up capacity and resolution,”  in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 12009–12019.
  * [2] Subhadeep Roy, Shankhanil Mitra, Soma Biswas, and Rajiv Soundararajan,  “Test time adaptation for blind image quality assessment,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2023, pp. 16742–16751.
  * [3] Deepti Ghadiyaram and Alan C Bovik,  “Massive online crowdsourced study of subjective and objective picture quality,”  IEEE Transactions on Image Processing, vol. 25, no. 1, pp. 372–387, 2015.
  * [4] Hamid R Sheikh, Muhammad F Sabir, and Alan C Bovik,  “A statistical evaluation of recent full reference image quality assessment algorithms,”  IEEE Transactions on image processing, vol. 15, no. 11, pp. 3440–3451, 2006.
  * [5] Vlad Hosu, Hanhe Lin, Tamas Sziranyi, and Dietmar Saupe,  “Koniq-10k: An ecologically valid database for deep learning of blind image quality assessment,”  IEEE Transactions on Image Processing, vol. 29, pp. 4041–4056, 2020.
  * [6] Hanhe Lin, Vlad Hosu, and Dietmar Saupe,  “Kadid-10k: A large-scale artificially distorted iqa database,”  in 2019 Eleventh International Conference on Quality of Multimedia Experience (QoMEX). IEEE, 2019, pp. 1–3.
  * [7] S Alireza Golestaneh, Saba Dadsetan, and Kris M Kitani,  “No-reference image quality assessment via transformers, relative ranking, and self-consistency,”  in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2022, pp. 1220–1230.
  * [8] Weixia Zhang, Kede Ma, Guangtao Zhai, and Xiaokang Yang,  “Uncertainty-aware blind image quality assessment in the laboratory and wild,”  IEEE Transactions on Image Processing, vol. 30, pp. 3474–3486, 2021.
  * [9] Junjie Ke, Qifei Wang, Yilin Wang, Peyman Milanfar, and Feng Yang,  “Musiq: Multi-scale image quality transformer,”  in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 5148–5157.
  * [10] Guanyi Qin, Runze Hu, Yutao Liu, Xiawu Zheng, Haotian Liu, Xiu Li, and Yan Zhang,  “Data-efficient image quality assessment with attention-panel decoder,”  arXiv preprint arXiv:2304.04952, 2023.


# Quadruplet Loss For Improving the Robustness to Face Morphing Attacks

Iurii Medvedev
1Institute of Systems
and Robotics,
University of Coimbra,
Coimbra, Portugal
<EMAIL_ADDRESS>Nuno Gonçalves 1,2
2Portuguese Mint and Official
Printing Office (INCM),
Lisbon, Portugal
<EMAIL_ADDRESS>
###### Abstract

Recent advancements in deep learning have revolutionized technology and
security measures, necessitating robust identification methods. Biometric
approaches, leveraging personalized characteristics, offer a promising
solution. However, Face Recognition Systems are vulnerable to sophisticated
attacks, notably face morphing techniques, enabling the creation of fraudulent
documents. In this study, we introduce a novel quadruplet loss function for
increasing the robustness of face recognition systems against morphing
attacks. Our approach involves specific sampling of face image quadruplets,
combined with face morphs, for network training. Experimental results
demonstrate the efficiency of our strategy in improving the robustness of face
recognition networks against morphing attacks.

###### Index Terms:

quadruplet loss, face morphing, face recognition, computer vision, deep
learning

##  I Introduction

In recent years, the evolution of deep learning technologies has profoundly
influenced the technology and modern security measures. Security concerns
continue to expand prompting the need for more robust and reliable
identification methods. To address these challenges, biometric approaches have
emerged as a promising solution, personalized characteristics for the
recognition process.

Among the various modalities of biometric identification, face recognition has
garnered significant attention and adoption due to its simplicity of
acquisition and recent advancements in computer vision techniques. The
widespread use of Face Recognition Systems (FRSs) underscores the importance
of facial traits in modern biometric applications, facilitating identification
and verification processes across various domains. However, FRSs remain
vulnerable to attacks, particularly from sophisticated image manipulation
techniques that aim to deceive the system.

One such technique is face morphing, which involves merging or blending two or
more digital face images to create a synthetic image that shares biometric
properties with the originals, potentially matching different individuals.
Face morphing can facilitate acceptance of manipulated face images for the
creation of fraudulent documents, which can then be utilized by unauthorized
individuals engaging in fraudulent activities. While such fraudulent documents
are occasionally detected during border control procedures, the true extent of
the existence of fraudulent documents remains unknown [1]. The emergence of
deep learning techniques has significantly advanced the field of face
recognition, yet face morphing poses a persistent security risk, challenging
traditional human or computer-based recognition methods. Ensuring the
robustness of face recognition systems against face morphing attacks is
crucial for maintaining the integrity of biometric identification.

In this work, we follow and extend contrastive methods in face recognition and
introduce a novel quadruplet loss function aimed at enhancing the robustness
of face recognition systems to morphing attacks. Our approach involves the
specific sampling of quadruplets of face images, combined with face morphs, to
train the network. Results of our experiments demonstrate that our strategy
effectively increases the robustness of face recognition networks against
morphing attacks, thereby contributing to the advancement of biometric
security measures.

##  II Related Work

In this section, we will review recent advancements in face morphing and face
recognition that are relevant to our research objectives.

###  II-A Face Morphing

Generation of a face morph from original images usually includes several
stages: extracting facial features $\rightarrow$ averaging features
$\rightarrow$ generating a morphed image from features $\rightarrow$ image
refining/retouching. Early methods [2, 3] relied on facial landmark features
for aligning original images and generating face morphs through image
blending. However, many of these techniques are prone to producing images with
blending artifacts.

Recent advancements in generative deep learning methods have led to the
development of face morphing methods that utilize the deep latent feature
domain. These approaches may employ a range of deep learning tools, such as
variational autoencoders (VAE) [4] or generative adversarial networks (GANs)
[5, 6] or diffusion autoencoders [7].

Significant number of recent research studies has been dedicated to the
detection of face morphs within biometric systems. [8, 9, 10, 11].

At the same time the issue of the robustness and stability of facial template
(representation) is also very important [12], and even famous private
benchmarks begin to support performance estimation for this [13].

For instance Marriott et al.[14] tested several facial recognition algorithms
to the robustness against face-morphing attacks and demonstrated the potential
threat of face morphing attacks.

###  II-B Face Recognition

Deep learning tools, known for their effectiveness in pattern recognition
tasks, have found widespread application in various biometric fields,
including face recognition [15]. Among these tools, convolutional neural
networks (CNNs) have gained prominence for their capacity to extract
discriminative features from unconstrained images, making them particularly
efficient in this context.

Recent approaches in face recognition primarily focused on extracting a
compact facial biometric template from deep features generated by a backbone
network. The main objective is to enhance the discriminative capability of
this template under specific conditions. The strategies for training deep
networks in face recognition can be categorized into classification and
contrastive approaches.

Many recent methods or training face recognition networks approach it via a
multi-class closed-set classification problem using existing face image
datasets. The discriminative information learned by the network is embedded in
its hidden feature layer, which can be utilized for open-set identity
discrimination tasks. These methods typically employ softmax loss and its
variations for classification [16]. Softmax classification-based methods were
broadly modified by techniques to enhance intra-class compactness and inter-
class separation. [17, 18, 19, 20, 21, 22] and with sample-specific strategies
for additional control of the feature domain with samples hardness,
classification uncertainty or quality [23, 24, 25, 26, 27].

Contrastive methods (or metric learning methods) utilize the target similarity
metric (for instance euclidean or cosine distance) to straightforwardly
optimize the distance between deep features by matching face image pairs
during the learning process [28]. These methods are usually characterized by
the high demands of dataset diversity and sophisticated sampling strategies
for reliable convergence of the training process [29]. They are shown to be
effective on large datasets and useful in the tasks of transfer learning and
fine-tuning.

Initial works straightforwardly optimised pairwise distance contrast utilising
small neural networks for feature extraction [30, 31].

The major improvement of contrastive methods in face recognition was
introduced by the FaceNet approach [32]. FaceNet proposed the use of triplet
loss, which directly optimizes the embedding of face images in a high-
dimensional space by the distances within triplets of samples. Triplets are
composed of an anchor image, a positive image of the same identity as the
anchor, and a negative image of a different identity. This approach required
sophisticated sampling of those triplets to learns discriminative feature
representation that ensures close proximity between embeddings of images of a
same individual while maximizing the separation between embeddings of images
of different individuals. Many of the following works on contrastive learning
methods in face recognition were inspired by the triplet loss strategy.

For instance DocFaceID [33] introduces the MPS loss function for fine-tuning a
pair of sibling networks for the ID document photo matching problem. MPS loss
aims to enhance representation learning by maximizing the margin between match
pair similarities and non-match pair similarities. This approach simulates a
scenario where ID photos serve as templates, while selfies from different
subjects act as probes seeking verification, or vice versa.

CoReFace [34] revisited the contrastive learning approach for face recognition
by incorporating an adaptive margin for better training regularisation.
Additionally, it proposed a new pair-coupling protocol to address the
similarity issue stemming from pair symmetry, thereby enhancing the
effectiveness of the approach.

###  II-C Quadruplet loss

As an evolution of traditional triplet loss, the Quadruplet loss has emerged
as a compelling method in the realm of deep metric learning for face
recognition. By leveraging four samples instead of three, the Quadruplet loss
can offer enhanced discriminative power and robustness. However such approach
can increase the complexity of efficient sampling strategy.

The Quadruplet loss found its application in the tasks of Re-Identification.
For instance, it was applied to the Re-Identification of baggage [35] or
persons [36, 37].

The Quadruplet loss was also used in applications for Heterogeneous Face
Recognition where it addresses challenges posed by cross-domain discrepancies
and limited training data [38]. It integrates domain-level and class-level
alignment within a unified network to learn domain-invariant discriminative
features (DIDF). Domain-level alignment reduces distribution discrepancy,
while a specialized quadruplet loss further diminishes intra-class variations
and enhances inter-class separability.

In applications of visual tracking the Quadruplet loss was used to develop a
discriminative model to one-shot learning (recognize objects of the same class
by a single exemplar given) and helped to improve tracking accuracy and real-
time speed [39].

##  III Methodology

In our work we intend to leverage the potential of quadruplet loss by
attaching additional branch which inputs the face morph sample to the common
triplet.

###  III-A Quadruplet Loss With Face Morphing

In order to define the Quadruplet loss in our work we revising the formulation
of the triplet loss in the FaceNet approach [32]. The triplet loss is computed
for three face images (anchor $x_{i}^{a}$, positive $x_{i}^{p}$ and negative
$x_{i}^{n}$) where anchor and a positive belong to the same identity and
negative belong to another identity. Next the distance between anchor and a
positive is minimised and the distance between anchor and a negative is
maximised:

$\left\|x_{i}^{a}-x_{i}^{p}\right\|_{2}^{2}+\alpha<\left\|x_{i}^{a}-x_{i}^{n}\right\|_{2}^{2},\forall(x_{i}^{a},x_{i}^{p},x_{i}^{n})\in\tau$
(1)

where $\alpha$ is a margin that is enforced between positive and negative
pairs. $\tau$ is the set of all possible triplets in the training set.

The embedding is represented by $f(x)\in\mathbb{R}^{d}$ and the loss that is
being minimized is then:

$L=\sum(\left\|f(x_{i}^{a})-f(x_{i}^{p})\right\|_{2}^{2}-\left\|f(x_{i}^{a})-f(x_{i}^{n})\right\|_{2}^{2}+\alpha)$
(2)

We modify this formulation by inserting a morph sample into the consideration.
This face morph sample may be generated from two face images, which belong to
the same identities of positive samples. The source images for generating may
be indeed different from other components of the initial triplet. With this
modification the cross-sample distance condition have the following
formulation:

$\displaystyle\left\|x_{i}^{a}-x_{i}^{p}\right\|_{2}^{2}+\alpha<\omega_{a/n}\left\|x_{i}^{a}-x_{i}^{n}\right\|_{2}^{2}+$
(3)
$\displaystyle+\omega_{a/m}\left\|x_{i}^{a}-x_{i}^{m}\right\|_{2}^{2}+\omega_{n/m}\left\|x_{i}^{n}-x_{i}^{m}\right\|_{2}^{2}+$
$\displaystyle+\omega_{p/m}\left\|x_{i}^{p}-x_{i}^{m}\right\|_{2}^{2},\forall(x_{i}^{a},x_{i}^{p},x_{i}^{n},x_{i}^{m})\in\mathbb{Q}$

where $x_{i}^{a},x_{i}^{p},x_{i}^{n},x_{i}^{m}$ correspond to anchor,
positive, negative, and morphed (for positive/negative pairs) in the
quadruplet $\mathbb{Q}$ and $\omega_{i}$ signify the weight for each part of
the sum, which are selected to balance identity matched and non-matched
distances ($\omega_{i}=0.25$) on Fig. 1.

Such methods in fact is directed onto achieving the maximum discriminative
power of the learned features, while also enlarging variances between original
images and their morphed combinations. The approximate visualisation of
desired feature distribution (in case of 2D feature layer dimensionality) is
depicted on the Fig. 2.

Figure 1: The proposed Quadruplet Loss minimizes the distance between an
anchor and a positive, both of which have the same identity, and maximizes the
distance between the anchor and a negative, anchor and morphed, negative and
morphed where anchor and negative belongs to different identities and morphed
is taken between the positive and negative.

Figure 2: The distribution of features $x$ (with two dimensional feature
layer) with common face recognition approaches (left) and desired feature
distribution in proposed approach (right). Each point on 2D surface
corresponds to a single image features (data is not based on real
experiments).

###  III-B Data Harvesting

Our training data is based on the collection, which is developped and used in
[10]. It is build on the filtered version of VGGFace2 dataset[40], where
quality-based filtering is used to select a subset of images suitable for face
morphing (frontal image with acceptable quality). Morphs are generated from
those filtered originals with a customized landmark-based morphing approach is
utilized with a blending coefficient set to 0.5. Additionally, GAN-based
morphs are generated using the StyleGAN method. These morphs are synthesized
by projecting original images onto a latent domain, followed by interpolation
of their deep representations to produce the resulting morph. We also employ
”selfmorphs” generated by applying face morphing to images of the same
identity. These selfmorphs serve as bona fide samples, allowing to focus on
the behavior of deep face features while mitigating the influence of
perceptual artifacts. In our work we assume that deep discriminative face
features remain unaffected by selfmorphing.

###  III-C Data Sampling

Training data generator combines the batches of quadruplets with the above
data. In our sampling strategy we go thought the list of available face morphs
and for each one we select anchor, positive and negative from the source morph
sample identities to combine the quadruplet (see Fig. 3). The resulting
quadruplets are combined into batches and fed to the Siamese network input.

Figure 3: Quadruplet samples

a b

c d

Figure 4: MinMax-MMPMR and ProdAvg-MMPMR for different models and protocols as
a dependency from FNMR. a,b - LDM benchmark; c,d - STG benchmark.

##  IV Experiments

We perform several experiment with our methodology and selected data. In all
our experiments we train the Resnet-50 architecture with image size $300\times
300$ pixels by 10 epochs with Adam optimizer. However first we need to discuss
the metrics and benchmarks that will be used for performance estimation.

###  IV-A Benchmarking

There are several metrics for estimating the robustness of face recognition
system to face morphing. Following the [12] we compute the MinMax-MMPMR (Eq.
4) and ProdAvg-MMPMR (Eq. 5), which are motivated by the international
standard ISO/IEC 30107-3 [41].

$\displaystyle MinMax-
MMPMR(\tau)={\color[rgb]{1,1,1}\cdot\cdot\cdot\cdot\cdot\cdot\cdot\cdot}$ (4)
$\displaystyle\frac{1}{M}\cdot\sum_{m=1}^{M}\left\\{\left(\begin{matrix}min\\\
i=1,...,N_{m}\end{matrix}\left[\begin{matrix}max\\\
i=1,...,I_{m}^{n}\end{matrix}S_{m}^{n,i}\right]\right)>\tau\right\\}$

where $I_{m}^{n}$ is the number of samples of subject $n$ within morph $m$.

$\displaystyle ProdAvg-MMPMR(\tau)={\color[rgb]{1,1,1}\cdot\cdot\cdot}$ (5)
$\displaystyle\frac{1}{M}\cdot\sum_{m=1}^{M}\left\\{\prod_{n=1}^{N_{m}}\frac{1}{I_{m}^{n}}\cdot\sum_{i=1}^{I_{m}^{n}}\left\\{S_{m}^{n,i}>\tau\right\\}\right\\}$

To estimate those metrics we build a custom benchmark basing on the dataset
from [42]. This dataset is collected fur the studies of ICAO-requirements[43]
conformity and thus contain both ICAO-compliant and non-compliant images. One
of the benefits of the data is that it contain images collected with a
professional digital camera and a smartphone giving these two modalities of
the data. We use those different modalities to simulate the differences of
enrollment and life capture (reference) in our benchmark utilities. Namely we
select enrollment images from the ones that are collected with professional
digital camera and the reference from the smartphone images. This also imply
strict ICAO-compliance for the enrollment images and relaxed conditions for
the reference ones. In total we have 1567 and 8760 images correspondingly.

In order to normalize the result metric rates they are usually computed at the
same level of FNMR in some standard face recognition 1-1 performance value. We
define such standard 1-1 verification protocol by pairing enrollment and
reference images. This protocols consist of $\sim$71k of pairs ($\sim$2k match
pairs and $\sim$69k non-match pairs)

We define a standard face pairing protocol for morphing, which standardize the
images selection for performing face morphing. This protocol consist of 2142
pairs, thus giving the same amount of face morphs in our benchmarks. face
morphs are generated only from the enrollment images. Following this protocols
we generate face morphs with a landmark based approach (LDM) and a StyleGAN
(STG)[44] approach to have in total two different benchmarks for performing
comparisons in our work.

###  IV-B Results

In our experiments we compare the results of with the Triplet loss baseline
(trained on the same date with the same conditions) and also investigating the
effect of self-morph presence. We present our results as the MMPMR dependency
from the FNMR values (see Fig. 4) and values of MMPMR at the FNMR=0.01 (see
Table I).

TABLE I: MinMax-MMPMR and ProdAvg-MMPMR for different models and protocols at FNMR=0.01. Model & Protocol | Rate @ FNMR=0.01
---|---
MinMax-MMPMR | ProdAvg-MMPMR
Triplet LDM | 0.99953 | 0.97890
Triplet+Self LDM | 1.0 | 0.97849
Triplet STD | 0.99906 | 0.96828
Triplet+Self STG | 1.0 | 0.97456
Quadruplet LDM | 0.99766 | 0.93530
Quadruplet+Self LDM | 1.0 | 0.978870
Quadruplet STG | 0.91876 | 0.642203
Quadruplet+Self STG | 0.999533 | 0.9666627

Withing the same setup Quadruplet Loss with our sampling strategy allowed to
achieve better face morphing robustness then the Triplet Loss baseline. At the
same time self morphs didn’t demonstrate significant impact to the
performance.

##  V Conclusion

In this work we revisit the contrastive methods in face recognition and
introduce a novel quadruplet loss function, which is designed to improve the
robustness of face recognition systems against morphing attacks. Through the
specific sampling of quadruplets of face images, coupled with face morphs, our
approach demonstrates promising results in enhancing the resilience of face
recognition networks against morphing attacks. By introducing the formulation
of the quadruplet loss and conducting extensive experiments across various
data settings, we prove the effectiveness of our strategy in resisting face
morphing attacks. In our future work we plan to extend this study by enlarging
our benchmark utilities with other types of morph samples and also print/scan
morphs and perform corresponding experiments. We also plan to perform a
extensive study of our benchmark utilities with various face recognition
utilities and networks.

## References

  * [1] M. Torkar, “Morphing cases in slovenia,” NIST IFPS, 2022, ministry of the Interior Police, Slovenia.
  * [2] M. Ferrara, A. Franco, and D. Maltoni, “The magic passport,” _IJCB 2014 - 2014 IEEE/IAPR_ , 12 2014.
  * [3] Biometric System Laboratory, “UBO-Morpher,” 2018, http://biolab.csr.unibo.it/Research.asp. (accessed: September 1, 2022).
  * [4] N. Damer, A. M. Saladié, A. Braun, and A. Kuijper, “MorGAN: Recognition Vulnerability and Attack Detectability of Face Morphing Attacks Created by Generative Adversarial Network,” in _2018 IEEE 9th International Conference on BTAS_ , 2018, pp. 1–10.
  * [5] S. K. Venkatesh, H. Zhang, R. Ramachandra, K. B. Raja, N. Damer, and C. Busch, “Can GAN Generated Morphs Threaten Face Recognition Systems Equally as Landmark Based Morphs? - Vulnerability and Detection,” _2020 8th IWBF_ , pp. 1–6, 2020.
  * [6] H. Zhang, S. Venkatesh, R. Ramachandra, K. Raja, N. Damer, and C. Busch, “MIPGAN - Generating Robust and High Quality Morph Attacks Using Identity Prior Driven GAN,” _ArXiv_ , vol. abs/2009.01729, 2020.
  * [7] N. Damer, M. Fang, P. Siebke, J. N. Kolf, M. Huber, and F. Boutros, “Mordiff: Recognition vulnerability and attack detectability of face morphing attacks created by diffusion autoencoders,” 2023. [Online]. Available: https://arxiv.org/abs/2302.01843
  * [8] M. Ferrara, A. Franco, and D. Maltoni, “Face morphing detection in the presence of printing/scanning and heterogeneous image sources,” _ArXiv_ , vol. abs/1901.08811, 2019.
  * [9] S. Lorenz, U. Scherhag, C. Rathgeb, and C. Busch., “Morphing attack detection: A fusion approach,” in _IEEE Fusion_ , 2021.
  * [10] I. Medvedev, F. Shadmand, and N. Gonçalves, “Mordeephy: Face morphing detection via fused classification,” in _Proceedings of ICPRAM_. SciTePress, 2023, pp. 193–204.
  * [11] P. C. Neto, T. Gonçalves, M. Huber, N. Damer, A. F. Sequeira, and J. S. Cardoso, “Orthomad: Morphing attack detection through orthogonal identity disentanglement,” in _2022 International Conference of the Biometrics Special Interest Group (BIOSIG)_ , 2022, pp. 1–5.
  * [12] U. Scherhag, A. Nautsch, C. Rathgeb, M. Gomez-Barrero, R. Veldhuis, L. Spreeuwers, M. Schils, D. Maltoni, P. Grother, S. Marcel, R. Breithaupt, R. Ramachandra, and C. Busch, “Biometric systems under morphing attacks: Assessment of morphing techniques and vulnerability reporting,” in _2017 International Conference of the Biometrics Special Interest Group, BIOSIG 2017_. Germany: Gesellschaft für Informatik, 9 2017.
  * [13] NIST, “NIST FRVT MORPH,” 2022, https://pages.nist.gov/frvt/html/frvt_morph.html.
  * [14] R. T. Marriott, S. Romdhani, S. Gentric, and L. Chen, “Robustness of facial recognition to gan-based face-morphing attacks,” _ArXiv_ , vol. abs/2012.10548, 2020. [Online]. Available: https://api.semanticscholar.org/CorpusID:229339565
  * [15] O. Russakovsky, J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, M. B., A. C. Berg, and L. Fei-Fei, “ImageNet Large Scale Visual Recognition Challenge,” _IJCV_ , vol. 115, no. 3, pp. 211–252, 2015.
  * [16] Y. Sun, X. Wang, and X. Tang, “Deeply learned face representations are sparse, selective, and robust,” _2015 IEEE Conference on CVPR_ , pp. 2892–2900, 2015.
  * [17] Y. Wen, K. Zhang, Z. Li, and Y. Qiao, “A Discriminative Feature Learning Approach for Deep Face Recognition,” in _Computer Vision – ECCV 2016_ , 2016, pp. 499–515.
  * [18] W. Liu, Y. Wen, Z. Yu, M. Li, B. Raj, and L. Song, “SphereFace: Deep Hypersphere Embedding for Face Recognition,” in _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , 2017, pp. 6738–6746.
  * [19] H. Wang, Y. Wang, Z. Zhou, X. Ji, D. Gong, J. Zhou, Z. Li, and W. Liu, “CosFace: Large Margin Cosine Loss for Deep Face Recognition,” in _2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2018, pp. 5265–5274.
  * [20] J. Deng, J. Guo, N. Xue, and S. Zafeiriou, “ArcFace: Additive Angular Margin Loss for Deep Face Recognition,” in _2019 Conference on CVPR_ , 2019, pp. 4685–4694.
  * [21] F. Wang, J. Cheng, W. Liu, and H. Liu, “Additive Margin Softmax for Face Verification,” _IEEE Signal Processing Letters_ , vol. 25, no. 7, pp. 926–930, 2018.
  * [22] J. Sun, W. Yang, J. Xue, and Q. Liao, “An Equalized Margin Loss for Face Recognition,” _IEEE Transactions on Multimedia_ , pp. 1–1, 2020.
  * [23] D. Zeng, H. Shi, H. Du, J. Wang, Z. Lei, and T. Mei, “NPCFace: A Negative-Positive Cooperation Supervision for Training Large-scale Face Recognition,” _CoRR_ , vol. abs/2007.10172, 2020. [Online]. Available: https://arxiv.org/abs/2007.10172
  * [24] Y. Shi and A. Jain, “Probabilistic Face Embeddings,” in _2019 IEEE/CVF International Conference on Computer Vision (ICCV)_ , 2019, pp. 6901–6910.
  * [25] Q. Meng, S. Zhao, Z. Huang, and F. Zhou, “Magface: A universal representation for face recognition and quality assessment,” in _2021 IEEE/CVF Conference on CVPR_ , 2021, pp. 14 220–14 229.
  * [26] J. Tremoço, I. Medvedev, and N. Gonçalves, “QualFace: Adapting Deep Learning Face Recognition for ID and Travel Documents with Quality Assessment,” in _2021 International Conference of the BIOSIG_ , 2021, pp. 1–6.
  * [27] I. Medvedev, J. Tremoço, B. Mano, L. E. Santo, and N. Gonçalves, “Towards understanding the character of quality sampling in deep learning face recognition,” _IET Biometrics_ , vol. 11, no. 5, pp. 498–511, 2022. [Online]. Available: https://ietresearch.onlinelibrary.wiley.com/doi/abs/10.1049/bme2.12095
  * [28] S. Chopra, R. Hadsell, and Y. LeCun, “Learning a similarity metric discriminatively, with application to face verification,” _2005 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR’05)_ , vol. 1, pp. 539–546 vol. 1, 2005.
  * [29] A. Mishchuk, D. Mishkin, F. Radenović, and J. Matas, “Working hard to know your neighbor’s margins: local descriptor learning loss,” in _Proceedings of the 31st International Conference on Neural Information Processing Systems_ , ser. NIPS’17. Red Hook, NY, USA: Curran Associates Inc., 2017, p. 4829–4840.
  * [30] H. V. Nguyen and L. Bai, “Cosine similarity metric learning for face verification,” in _Asian Conference on Computer Vision_ , 2010. [Online]. Available: https://api.semanticscholar.org/CorpusID:18453384
  * [31] J. Hu, J. Lu, and Y.-P. Tan, “Discriminative deep metric learning for face verification in the wild,” _2014 IEEE Conference on Computer Vision and Pattern Recognition_ , pp. 1875–1882, 2014. [Online]. Available: https://api.semanticscholar.org/CorpusID:3217198
  * [32] F. Schroff, D. Kalenichenko, and J. Philbin, “FaceNet: A unified embedding for face recognition and clustering,” in _2015 IEEE Conference CVPR_ , 2015, pp. 815–823.
  * [33] Y. Shi and A. K. Jain, “DocFace: Matching ID Document Photos to Selfies*,” in _2018 IEEE 9th International Conference on BTAS_ , 2018, pp. 1–8.
  * [34] Y. Song and F. Wang, “Coreface: Sample-guided contrastive regularization for deep face recognition,” 2023.
  * [35] H. Yang, X. Chu, L. Zhang, Y. Sun, D. Li, and S. J. Maybank, “Quadnet: Quadruplet loss for multi-view learning in baggage re-identification,” _Pattern Recognition_ , vol. 126, p. 108546, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S0031320322000279
  * [36] H. Proença, E. Yaghoubi, and P. Alirezazadeh, “A quadruplet loss for enforcing semantically coherent embeddings in multi-output classification problems,” _IEEE Transactions on Information Forensics and Security_ , vol. 16, pp. 800–811, 2021.
  * [37] W. Chen, X. Chen, J. Zhang, and K. Huang, “Beyond triplet loss: A deep quadruplet network for person re-identification,” _2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_ , pp. 1320–1329, 2017. [Online]. Available: https://api.semanticscholar.org/CorpusID:14795862
  * [38] S. Yang, K. Fu, X. Yang, Y. Lin, J. Zhang, and C. Peng, “Learning domain-invariant discriminative features for heterogeneous face recognition,” _IEEE Access_ , vol. 8, pp. 209 790–209 801, 2020.
  * [39] X. Dong, J. Shen, D. Wu, K. Guo, X. Jin, and F. M. Porikli, “Quadruplet network with one-shot learning for fast visual object tracking,” _IEEE Transactions on Image Processing_ , vol. 28, pp. 3516–3527, 2017. [Online]. Available: https://api.semanticscholar.org/CorpusID:3972862
  * [40] Q. Cao, L. Shen, W. Xie, O. M. Parkhi, and A. Zisserman, “Vggface2: A dataset for recognising faces across pose and age,” in _International Conference on FG_ , 2018.
  * [41] International Organization for Standardization, “ISO/IEC 30107–3:2017. Information Technology—Biometric Presentation Attack Detection — Part 3: Testing and Reporting,” ISO/IEC JTC 1/SC 37 Biometrics, p. 33, 09 2017.
  * [42] C. Guerra, J. Marcos, and N. Gonçalves, “Automatic validation of ICAO compliance regarding head coverings: An inclusive approach concerning religious circumstances,” in _International Conference of the Biometrics Special Interest Group, BIOSIG 2023, Darmstadt, Germany, September 20-22, 2023_ , N. Damer, M. Gomez-Barrero, K. B. Raja, C. Rathgeb, A. F. Sequeira, M. Todisco, and A. Uhl, Eds. IEEE, 2023, pp. 1–4. [Online]. Available: https://doi.org/10.1109/BIOSIG58226.2023.10345995
  * [43] E. V. C. L. Borges, I. L. P. Andrezza, J. R. T. Marques, R. A. T. Mota, and J. J. B. Primo, “Analysis of the Eyes on Face Images for Compliance with ISO/ICAO Requirements,” in _2016 29th SIBGRAPI Conference on Graphics, Patterns and Images (SIBGRAPI)_ , 2016, pp. 173–179.
  * [44] T. Karras, S. Laine, and T. Aila, “A Style-Based Generator Architecture for Generative Adversarial Networks,” in _2019 IEEE/CVF Conference on CVPR_ , 2019, pp. 4396–4405.


# Attention-based CNN-LSTM and XGBoost hybrid model for stock prediction

Zhuangwei Shi, Yang Hu, Guangliang Mo, Jian Wu This work was supported by the
Orient Securities Co. Ltd, and the Entropy-controlled Research Project of
Shenzhen Wukong Investment Management Co. Ltd. Correspondence author:
Zhuangwei Shi (email:zwshi@mail.nankai.edu.cn)Zhuangwei Shi and Yang Hu are
with the College of Artificial Intelligence, Nankai University, Tianjin,
China, and the Shenzhen Research Institute, Nankai University, Shenzhen,
Guangdong, China.Guangliang Mo is with the Orient Securities Co. Ltd,
Shanghai, China.Jian Wu is with the Research Center of Pearl River-West River
Economic Belt (PWEB) Development and Finance, and the Institute of Regional
Development, Guangxi Academy of Social Science, Nanning, Guangxi, China.

###### Abstract

Stock market plays an important role in the economic development. Due to the
complex volatility of the stock market, the research and prediction on the
change of the stock price, can avoid the risk for the investors. The
traditional time series model ARIMA can not describe the nonlinearity, and can
not achieve satisfactory results in the stock prediction. As neural networks
are with strong nonlinear generalization ability, this paper proposes an
attention-based CNN-LSTM and XGBoost hybrid model to predict the stock price.
The model constructed in this paper integrates the time series model, the
Convolutional Neural Networks with Attention mechanism, the Long Short-Term
Memory network, and XGBoost regressor in a non-linear relationship, and
improves the prediction accuracy. The model can fully mine the historical
information of the stock market in multiple periods. The stock data is first
preprocessed through ARIMA. Then, the deep learning architecture formed in
pretraining-finetuning framework is adopted. The pre-training model is the
Attention-based CNN-LSTM model based on sequence-to-sequence framework. The
model first uses convolution to extract the deep features of the original
stock data, and then uses the Long Short-Term Memory networks to mine the
long-term time series features. Finally, the XGBoost model is adopted for
fine-tuning. The results show that the hybrid model is more effective and the
prediction accuracy is relatively high, which can help investors or
institutions to make decisions and achieve the purpose of expanding return and
avoiding risk. Source code is available at
https://github.com/zshicode/Attention-CLX-stock-prediction.

###### Index Terms:

Attention mechanism, Convolutional Neural Networks, Long Short-Term Memory,
XGBoost, stock prediction

##  I Introduction

Stock market plays an important role in the economic development. Due to the
high return characteristics of stocks, the stock market has attracted more and
more attention from institutions and investors. However, due to the complex
volatility of the stock market, sometimes it will bring huge loss to
institutions or investors. Considering the risk of the stock market, the
research and prediction on the change of the stock price can avoid the risk
for the investors.

The traditional time series model ARIMA can not describe the nonlinear time
series, and needs to satisfy many preconditions before modeling, and can not
achieve remarkable results in the stock forecasting. In recent years, with the
rapid development of artificial intelligence theory and technology, more and
more researchers apply artificial intelligence method to the financial market.
On the other hand, the sequence modeling problem, focusing on natural language
sequences, protein sequences, stock price sequences, and so on, is important
in the field of artificial intelligence research [12, 8]. The most
representative artificial intelligence method is neural networks, which are
with strong nonlinear generalization ability.

Recurrent Neural Network (RNN) was adopted for analyzing sequential data via
neural network architecture, and Long Short-Term Memory (LSTM) model is the
most commonly used RNN. LSTM introduced gate mechanism in RNN, which can be
seen as simulation for human memory, that human can remember useful
information and forget useless information [6]. Attention Mechanism [15, 7]
can be seen as simulation for human attention, that human can pay attention to
useful information and ignore useless information. Attention-based
Convolutional Neural Networks (ACNN) are widely used for sequence modeling [4,
10]. Combining Attention-based Convolutional Neural Networks and Long Short-
Term Memory, is a self-attention based sequence-to-sequence (seq2seq) [14]
model to encode and decode sequential data. This model can solve long-term
dependency problem in LSTM, hence, it can better model long sequences. LSTM
can capture particular long-distance correspondence that fits the sturcture of
LSTM itself, while ACNN can capture both local and global correspondence.
Therefore, this architecture is more flexible and robust.

Transformer [15] is the most successful sequential learning self-attention
based model. Experiments on natural language processing demonstrates that
Transformer can better model long sequences. Bidirectional Encoder
Representation Transformer (BERT) with pretraining [2] can perform better than
the basic Transformer. Pretraining is a method to significantly improve the
performance of Transformer (BERT).

This paper proposes a hybrid deep learning model to predict the stock price.
Different from the traditional hybrid prediction model, the proposed model
integrates the time series model ARIMA and the neural networks in a non-linear
relationship, which combines the advantages of the two vanilla models, and
improves the prediction accuracy. The stock data is first preprocessed through
ARIMA. The stock sequence is put into neural networks (NN) or XGBoost after
preprocessing via ARIMA(p=2,q=0,d=1). Then, the deep learning architecture
formed in pretraining-finetuning framework [2, 5] is adopted. The pre-training
model is the Attention-based CNN-LSTM model based on sequence-to-sequence
framework, where the Attention-based CNN is encoder, and the Bidirectional
LSTM is decoder. The model first uses convolution to extract the deep features
of the original stock data, and then uses the Long Short-Term Memory networks
to mine the long-term time series features. Finally, the XGBoost model is
adopted for fine-tuning, which can fully mine the information of the stock
market in multiple periods. Our proposed Attention-based CNN-LSTM and XGBoost
hybrid model is so called AttCLX.

The results show that the model is more effective and the prediction accuracy
is relatively high, which can help investors or institutions to make decisions
and achieve the purpose of expanding returns and avoiding risks. The source
code of this paper is available at https://github.com/zshicode/Attention-CLX-
stock-prediction. We conduct empirical study on the stock price of Back of
China (601988.SH) in Chinese stock market. The data is downloaded from
Tushare(www.tushare.pro). The stock price data on Tushare is with public
availability.

##  II Materials and Methods

###  II-A ARIMA

Classical stock prediction methods are based on ARMA (Auto Regressive Moving
Average) model and ARIMA (Auto Regressive Integrated Moving Average) model. An
ARMA(p,q) model

$s_{t}=a_{0}+\sum_{i=1}^{p}a_{i}s_{t-i}+w_{t}+\sum_{i=1}^{q}b_{i}w_{t-i},$ (1)

where a,b are parameters, w is noise. ARMA model can be used when sequence
$s_{1:N}$ is stationary, which means

$\mathbb{E}[s_{t}]=\mathrm{Constant}.$ (2)
$\mathrm{Cov}(s_{t},s_{t-k})=\mathrm{Constant}.$ (3)

where $t=1,2,...,N$ and $k=1,2,...,t$.

When sequence is non-stationary, ARIMA(p,q,d) adopts $d$-order difference to
the sequence. In stock prediction, the first-order difference
$x_{1:N}=\dot{s}_{1:N}$ (i.e. $x_{k}=s_{k}-s_{k-1}$) is usually considered as
stationary sequence.

We conduct empirical study on the stock price of Back of China (601988.SH) in
Chinese stock market. The data is downloaded from Tushare(www.tushare.pro).
The stock price data on Tushare is with public availability. The data is
selected from the data from January 1, 2007 to March 31, 2022, the data in one
day denotes a point of the sequence.

ADF test is adopted for testing the stationary condition of time series.
Adopting ADF test for the original sequence and first-order difference
sequence. The results are in Table I and II. When p-value is more than 0.562
or Critical Value (1%) is more than -3.44, the sequence is non-stationary. The
ADF test shows that the original sequence is non-stationary and first-order
difference sequence is stationary. The first-order difference sequence and
second-order difference sequence are shown on Fig.1 and 2.

TABLE I: ADF test for original sequence Metric | Value
---|---
Test Statistic Value | -2.35539
p-value | 0.154726
Lags Used | 16
Number of Observations Used | 3484
Critical Value(1%) | -3.43223
Critical Value(5%) | -2.86237
Critical Value(10%) | -2.56721
TABLE II: ADF test for first-order diff sequence Metric | Value
---|---
Test Statistic Value | -14.7498
p-value | 2.49565e-27
Lags Used | 15
Number of Observations Used | 3484
Critical Value(1%) | -3.43223
Critical Value(5%) | -2.86237
Critical Value(10%) | -2.56721
Figure 1: The first-order difference. Figure 2: The second-order difference.

###  II-B Deep learning on sequential data

In basic feed-forward neural network (FFNN), output of current moment $o_{t}$
is only determined by input of current moment $i_{t}$, which suppress the
ability of FFNN to model time-series data. In recurrent neural network (RNN),
a delay is used to save the latent state of latest moment $h_{t-1}$, then,
latent state of current moment $h_{t}$ is determined by both $h_{t-1}$ and
$i_{t}$. See Fig. 3. [3] suggested that RNN may vanish the gradient as error
propagates through time dimension, which leads to long-term dependency
problem. Human can selectively remember information. Through gated activation
function, LSTM (long short-term memory) model can selectively remember updated
information and forget accumulated information.

Figure 3: RNN unit.

Sequence-to-sequence (seq2seq) [14] model adopted autoencoder (i.e. encoder-
decoder architecture) for analyzing sequential data. Sequence-to-sequence
model (seq2seq) [14] is constructed through an encoder-decoder architecture,
which enhances the ability of LSTM to learn hidden information through data
with noise. In seq2seq, the encoder is an LSTM that encodes the inputs into
the context (commonly the hidden state at last $h_{N}$), then decode the
context in the decoder. In the decoder, output at previous moment is input at
next moment.

To optimize the decoded sequence, beam search [16] was adopted in seq2seq
model. Both beam search and Viterbi algorithm in hidden Markov model (HMM) are
based on dynamic programming. Solving optimal estimation of current state
according to observation and previous state, is called decoding or inference
in HMM. Solving $p(x_{k}|y_{1:k})$ and $p(x_{k}|y_{1:N})$ respectively, is
equivalent to forward-backward algorithm in HMM. The optimal bidirectional
estimation can be obtained through the distribution of $x_{k}$. That is the
probabilistic perspective that the bidirectional LSTM which combines forward
and backward, is proposed [6].

###  II-C Attention mechanism

Human usually pays attention to salient information. Attention mechanism is a
technique in deep learning based on human cognitive system. For input
$X=(x_{1},x_{2},...,x_{N})$, give query vector $q$, depict the index of
selected information by attention $z=1,2,...,N$, then the distribution of
attention

$\alpha_{i}=p(z=i|X,\mathbf{q})=\frac{\exp\left(s\left(\mathbf{x}_{i},\mathbf{q}\right)\right)}{\sum_{j=1}^{N}\exp\left(s\left(\mathbf{x}_{j},\mathbf{q}\right)\right)}.$
(4)

i.e.

$\alpha_{i}=\mathrm{softmax}\left(s\left(\mathbf{x}_{i},\mathbf{q}\right)\right)$
(5)

Here

$s\left(\mathbf{x}_{i},\mathbf{q}\right)=\frac{\mathbf{x}_{i}^{\mathrm{T}}\mathbf{q}}{\sqrt{d}}$
(6)

is attention score through scaled dot product, $d$ is the dimension of input.
Suppose the input key-value pairs
$(K,V)=\left[\left(\mathbf{k}_{1},\mathbf{v}_{1}\right),\cdots,\left(\mathbf{k}_{N},\mathbf{v}_{N}\right)\right]$,
for given $q$, attention function

$\mathrm{att}((K,V),\mathbf{q})=\sum_{i=1}^{N}\alpha_{i}\mathbf{v}_{i}=\sum_{i=1}^{N}\frac{\exp\left(s\left(\mathbf{k}_{i},\mathbf{q}\right)\right)}{\sum_{j}\exp\left(s\left(\mathbf{k}_{j},\mathbf{q}\right)\right)}\mathbf{v}_{i}.$
(7)

Multi-head mechanism is usually adopted through multi-query
$Q=\left[\mathbf{q}_{1},\cdots,\mathbf{q}_{M}\right]$ for attention function
computation.

$\mathrm{att}((K,V),Q)=\left(\mathrm{att}\left((K,V),\mathbf{q}_{1}\right)||\cdots||\mathrm{att}\left((K,V),\mathbf{q}_{M}\right)\right).$
(8)

Here, $||$ denotes Concatenate operation. This is so called multi-head
attention (MHA).

Attention mechanism can be adopted to generate data-driven different weights.
Here, $Q,K,V$ are all obtained through linear transform of $X$, and
$W_{Q},W_{K},W_{V}$ can be adjusted dynamicly.

$Q=W_{Q}X,K=W_{K}X,V=W_{V}X.$ (9)

This is so called self-attention. Similarly, output

$\mathbf{h}_{i}=\operatorname{att}\left((K,V),\mathbf{q}_{i}\right).$ (10)

Hence

$h_{i}=\sum_{j=1}^{N}\alpha_{ij}\mathbf{v}_{j}=\sum_{j=1}^{N}\operatorname{softmax}\left(s\left(\mathbf{k}_{j},\mathbf{q}_{i}\right)\right)\mathbf{v}_{j}$
(11)

Adopting scaled dot product score, the output

$H=V\operatorname{softmax}\left(\frac{K^{\mathrm{T}}Q}{\sqrt{d}}\right).$ (12)

###  II-D Method

####  II-D1 Preprocessing

This paper proposes a hybrid deep learning model to predict the stock price.
Different from the traditional hybrid prediction model, the proposed model
integrates the time series model ARIMA and the neural networks in a non-linear
relationship, which combines the advantages of the two vanilla models, and
improves the prediction accuracy.

The stock data is first preprocessed through ARIMA. The stock sequence is put
into neural networks (NN) or XGBoost after preprocessing via
ARIMA(p=2,q=0,d=1). Putting original stock market data into ARIMA, can output
a new series that depict state more effectively.

The ADF results in Table I and II shows that the original sequence is non-
stationary and first-order difference sequence is stationary. After
determining d=1, we need to determine AR(p) and MA(q) in ARIMA. We use the
Autocorrelation Figure (ACF) and Partial Autocorrelation Figure (PACF). The
ACF and PACF of the original sequence and first-order difference sequence are
shown on Fig. 4 and 5. Fig. 4 shows that PACF truncates when order=2, meaning
that we should adopt AR(2). The ACF is with long tail for any order, meaning
that we should adopt MA(0). Hence p=2, q=0.

Figure 4: The ACF and PACF of the original sequence. Figure 5: The ACF and
PACF of the first-order difference.

####  II-D2 Pretraining

Then, the deep learning architecture formed in pretraining-finetuning
framework is adopted. The pre-training model is the Attention-based CNN-LSTM
model based on sequence-to-sequence framework, where the Attention-based CNN
is encoder, and the Bidirectional LSTM is decoder. The model first uses
convolution to extract the deep features of the original stock data, and then
uses the Long Short-Term Memory networks to mine the long-term time series
features. Finally, the XGBoost model is adopted for fine-tuning, which can
fully mine the information of the stock market in multiple periods.

Seq2seq suppress the effect of noise through encoder-decoder architecture.
Based on deep learning, hidden information of state is depicted more
effectively, while the model would not satisfy the assumptions of linear
property of stock price. LSTM receives the context from Attention-based CNN
(ACNN)encoder.

The ACNN encoder block consists of self-attention layer and CNN. $Q,K,V$ are
computed through Eq. (9) after self-attention layer, and $H$ is computed
through Eq. (12). This is the input of LSTM decoder block. Encoder-decoder
layers depicts relationship between current sequence and previous sequence,
and relationship between current sequence and embedding. Encoder is still with
multi-head mechanism. When the $k$-th embedding is being decoded, only
$k-1$-th and previous decoding can be seen. This multi-head mechanism is
masked multi-head attention.

Attention-based CNN (ACNN) can capture both global and local dependency that
LSTM may not [4], which enhance the robustness. In our proposed encoder-
decoder framework, we can adopt a ACNN-LSTM structure. Attention is usually
before memory in human cognitive system. The reason why ACNN can capture long-
term dependency, is that it integrates multi-head self attention and
convolution. Combining LSTM and ACNN can enhance both structural advantages
and ability for time-series modeling. Integrating multi-head attention and
multi-scale convolutional kernel, ACNN enocder can capture saliency that LSTM
may not, while LSTM can better depict time-series property.

####  II-D3 Fine-tuning

After decoding, the output is obtained through a XGBoost regressor for precise
extraction of features and fine-tuning. Our proposed Attention-based CNN-LSTM
and XGBoost hybrid model is so called AttCLX, which is shown on Fig. 6.

Figure 6: Attention-based CNN-LSTM and XGBoost hybrid model (AttCLX).

As the fine-tuning model, XGBoost [1] is with strong expansion and
flexibility. It integrates multiple tree models to build a stronger learner
model. Based on the pre-training, we propose fine-tuning based on XGBoost, and
establish a regression prediction model for stock data. XGBoost fine-tuning
model also achieves better predictive ability and generalization ability.

##  III Experiments

###  III-A Modification of model

After preprocessing by ARIMA, the input of the neural networks, is a two-
dimensional matrix of data at intervals of a period of time, with a size of
TimeWindow$\times$Features. In Empirical studies on stock prediction, features
include the basic stock market data (opening price, closing price, highest
price, lowest price, trading volume, trading amount). The ARIMA-processing
sequence along with the residual sequence are also concatenated as features.

We adopted look back trick for time-series forecasting, and the look back
number is 20, i.e. $o_{t}$ can be obtained through $o_{t-1},\cdots,o_{t-5}$.
This means that the TimeWindow width is 20. The layer number of LSTM is 5, and
the size is 64. The epoch number is 50. Model is trained by introducing
dropout [13], and the dropout rate is 0.3. The head number is 4.

The experiments are on an NVIDIA GTX2070 GPU with 8GB memory. The model is
trained through Adam optimizer [9], and learning rate is 0.01.

The data used in this article comes from the open and free public dataset in
Tushare (https://www.tushare.pro/) for the research of stock market in China,
which has the characteristics of rich data, simple use, and convenient
implementation. It is very convenient to obtain the basic market data of
stocks by calling the API.

The implementation details of this paper can refer to source code of this
paper at https://github.com/zshicode/Attention-CLX-stock-prediction. We
conduct empirical study on the stock price of Back of China (601988.SH) in
Chinese stock market. The data is downloaded from Tushare(www.tushare.pro).
The stock price data on Tushare is with public availability. The data is
selected from the data from January 1, 2007 to March 31, 2022, the data in one
day denotes a point of the sequence. The train set and test set was divided on
June 22, 2021, as shown on Fig. 7. This means that there are 3500 training
samples and 180 testing samples. The batch size is 32.

Figure 7: The train set and test set.

###  III-B Prediction performance

The stock price prediction result of ARIMA model is shown on Fig. 8. The
resudial and residual density plot are shown on Fig. 9.

Figure 8: The ARIMA for stock price prediction. Figure 9: The resudial and
residual density plot.

The stock price prediction result of ARIMA+XGBoost model is shown on Fig. 10.
The resudial and residual density plot are shown on Fig. 11.

Figure 10: The ARIMA+XGBoost for stock price prediction. Figure 11: The
ARIMA+XGBoost for resudial prediction.

The loss curves of original sequence and residual sequence of ARIMA+SingleLSTM
are shown on Fig. 12 and Fig. 13.

Figure 12: The loss curves of original sequence of ARIMA+SingleLSTM. Figure
13: The loss curves of residual sequence of ARIMA+SingleLSTM.

The stock price prediction result of ARIMA+SingleLSTM model and ARIMA+BiLSTM
model are shown on Fig. 14 and Fig. 15.

Figure 14: The stock price prediction result of ARIMA+SingleLSTM. Figure 15:
The stock price prediction result of ARIMA+BiLSTM.

The loss curve of our proposed model is shown on Fig. 16.

Figure 16: The loss curve of our proposed model.

The stock price prediction result of our proposed model is shown on Fig. 17.

Figure 17: The stock price prediction result of our proposed model.

###  III-C Compared with other methods

The evaluation metrics are mean absolute error (MAE), root of mean square
error (RMSE), mean absolute percentage error (MAPE) and $R^{2}$.

$MAE=\frac{1}{n}\sum\limits_{t=1}^{n}{\left|{{{\hat{X}}_{t}}-{X_{t}}}\right|}$
(13)
$RMSE=\sqrt{\frac{1}{n}\sum\limits_{t=1}^{n}{{{({{\hat{X}}_{t}}-{X_{t}})}^{2}}}}$
(14)
$MAPE=\frac{1}{n}\sum\limits_{t=1}^{n}{\left|{\frac{{{{\hat{X}}_{t}}-{X_{t}}}}{{{X_{t}}}}}\right|}$
(15)
$R^{2}=\frac{\sum\limits_{t=1}^{n}\|\hat{X}_{t}-\bar{X}_{t}\|^{2}}{\sum\limits_{t=1}^{n}\|X_{t}-\bar{X}_{t}\|^{2}}$
(16)

Here $\bar{X}_{t}$ denotes the mean value of $X_{t}$. Lower error and higher
$R^{2}$ denote better performance.

First, we conduct comparison with different pre-training and fine-tuning
models. Table III demonstrates that our proposed model outperforms other
baselines.

TABLE III: Comparison with different pre-training and fine-tuning models Pre-training | Fine-tuning | MSE | RMSE | MAE | R2
---|---|---|---|---|---
None | None | 0.00057 | 0.02734 | 0.02368 | 0.74402
None | XGBoost | 0.00031 | 0.01755 | 0.01223 | 0.82405
SL-LSTM | SL-LSTM | 0.00045 | 0.02282 | 0.01960 | 0.79434
ML-LSTM | ML-LSTM | 0.00031 | 0.01720 | 0.01265 | 0.82351
BiLSTM | BiLSTM | 0.00027 | 0.01652 | 0.01201 | 0.84210
BiLSTM | XGBoost | 0.00024 | 0.01605 | 0.01187 | 0.86301
CNN-BiLSTM | XGBoost | 0.00022 | 0.01529 | 0.01145 | 0.87720
ACNN-BiLSTM | XGBoost | 0.00020 | 0.01424 | 0.01126 | 0.88342

Then, we conduct comparison with current methods. The compared methods
include:

  * •

ARIMA

  * •

ARIMA-NN [17] improved time-series forecasting using a hybrid ARIMA and neural
network model.

  * •

LSTM-KF, Transformer-KF, TL-KF [11] proposed Kalman Filter, along with LSTM
and Transformer for stock prediction.

Table IV demonstrate that the proposed neural networks outperforms current
methods.

TABLE IV: Results on different methods Model | MAE | RMSE | MAPE | R2
---|---|---|---|---
ARIMA | 0.00057 | 0.02734 | 0.02368 | 0.74402
ARIMA-NN | 0.00052 | 0.02608 | 0.02350 | 0.75037
LSTM-KF | 0.00047 | 0.02381 | 0.02192 | 0.76245
Transformer-KF | 0.00037 | 0.01924 | 0.01525 | 0.80230
TL-KF | 0.00033 | 0.01656 | 0.01372 | 0.81923
AttCLX | 0.00020 | 0.01424 | 0.01126 | 0.88342

##  IV Conclusions

Stock market is of great importance in the financial and economic development.
Due to the complex volatility of the stock market, the prediction on the
tendency of the stock price, can secure the return for the investors. The
traditional time series model ARIMA can not describe the nonlinearity in the
stock prediction. As neural networks are with strong nonlinear modeling
ability, this paper proposes an attention-based CNN-LSTM and XGBoost hybrid
model to predict the stock price. The model in this paper integrates the ARIMA
model, the Convolutional Neural Networks with Attention mechanism, the Long
Short-Term Memory network, and XGBoost regressor in a non-linear relationship,
and improves the prediction accuracy. The model can capture the information of
the stock market in multiple periods. The stock data is first preprocessed
through ARIMA. Then, the deep learning architecture formed in pretraining-
finetuning framework is adopted. The pre-training model is the Attention-based
CNN-LSTM model based on sequence-to-sequence framework. The model first uses
attention-based multi-scale convolution to extract the deep features of the
original stock data, and then uses the Long Short-Term Memory networks to mine
the time series features. Finally, the XGBoost model is adopted for fine-
tuning. The results show that the hybrid model is more effective, which can
help investors or institutions to achieve the purpose of expanding return and
avoiding risk.

## References

  * [1] T. Chen and C. Guestrin, “Xgboost: A scalable tree boosting system,” in _KDD_ , 2016.
  * [2] J. Devlin, M.-W. Chang, K. Lee, and K. Toutanova, “Bert: Pre-training of deep bidirectional transformers for language understanding,” _CoRR_ , 2018.
  * [3] A. Graves, “Long short-term memory,” _Neural Computation_ , vol. 9, no. 8, pp. 1735–1780, 1997.
  * [4] C. Jin, J. Gao, Z. Shi, and H. Zhang, “Attcry: Attention-based neural network model for protein crystallization prediction,” _Neurocomputing_ , vol. 463, pp. 265–274, 2021.
  * [5] C. Jin, Z. Shi, C. Kang, K. Lin, and H. Zhang, “Tlcrys: Transfer learning based method for protein crystallization prediction,” _International Journal of Molecular Sciences_ , vol. 23, p. 972, 1 2022.
  * [6] C. Jin, Z. Shi, W. Li, and Y. Guo, “Bidirectional lstm-crf attention-based model for chinese word segmentation,” _arXiv preprint arXiv:2105.09681_ , 2021.
  * [7] C. Jin, Z. Shi, K. Lin, and H. Zhang, “Predicting mirna-disease association based on neural inductive matrix completion with graph autoencoders and self-attention mechanism,” _Biomolecules_ , vol. 12, no. 1, p. 64, 2022.
  * [8] C. Jin, Z. Shi, H. Zhang, and Y. Yin, “Predicting lncrna–protein interactions based on graph autoencoders and collaborative training,” in _IEEE International Conference on Bioinformatics and Biomedicine (BIBM), Houston, USA, 9-12 December_ , 2021.
  * [9] D. P. Kingma and J. Ba, “Adam: a method for stochastic optimization,” in _3rd International Conference for Learning Representations(ICLR)_ , 2015.
  * [10] K. Lin, X. Quan, C. Jin, Z. Shi, and J. Yang, “An interpretable double-scale attention model for enzyme protein class prediction based on transformer encoders and multi-scale convolutions,” _Frontiers in Genetics_ , vol. 13, p. 885627, 2022.
  * [11] Z. Shi, “Incorporating transformer and lstm to kalman filter with em algorithm for state estimation,” _arXiv preprint arXiv:2105.00250_ , 2021.
  * [12] Z. Shi, H. Zhang, C. Jin, X. Quan, and Y. Yin, “A representation learning model based on variational inference and graph autoencoder for predicting lncrna-disease associations,” _BMC Bioinformatics_ , vol. 22, p. 136, 2021\.
  * [13] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov, “Dropout: a simple way to prevent neural networks from overfitting,” _Journal of Machine Learning Research_ , vol. 15, no. 1, pp. 1929–1958, 2014\.
  * [14] I. Sutskever, O. Vinyals, and Q. V. Le, “Sequence to sequence learning with neural networks,” in _NeurIPS_ , 2014.
  * [15] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all you need,” in _NeurIPS_ , 2017.
  * [16] S. Wiseman and A. M. Rush, “Sequence-to-sequence learning as beam-search optimization,” in _EMNLP_ , 2016.
  * [17] G. Zhang, “Time series forecasting using a hybrid arima and neural network model,” _Neurocomputing_ , vol. 50, pp. 159–175, 2003.