# A New Perspective for Shuttlecock Hitting Event Detection

Yu-Hsi Chen Institute of Communications Engineering, National Tsing Hua
University, Taiwan<EMAIL_ADDRESS>
###### Abstract

This article introduces a novel approach to shuttlecock hitting event
detection. Instead of depending on generic methods, we capture the hitting
action of players by reasoning over a sequence of images. To learn the
features of hitting events in a video clip, we specifically utilize a deep
learning model known as SwingNet. This model is designed to capture the
relevant characteristics and patterns associated with the act of hitting in
badminton. By training SwingNet on the provided video clips, we aim to enable
the model to accurately recognize and identify the instances of hitting events
based on their distinctive features. Furthermore, we apply the specific video
processing technique to extract the prior features from the video, which
significantly reduces the learning difficulty for the model. The proposed
method not only provides an intuitive and user-friendly approach but also
presents a fresh perspective on the task of detecting badminton hitting
events. The source code will be available at https://github.com/TW-
yuhsi/A-New-Perspective-for-Shuttlecock-Hitting-Event-Detection.

##  1 Introduction

In this task, we need to develop computer vision technology capable of
automatically extracting comprehensive technical data from the broadcast video
of badminton matches. This data includes crucial information about each shot,
such as the precise timing, spatial positioning, player postures, and skill
levels exhibited as the shuttlecock is struck during the match. It encompasses
various elements, such as the shot’s timing, ball’s location, player
responsible for the hit, swing posture, standing positions of both players,
type of ball used, and the eventual winner. These extensive datasets serve as
valuable resources for analyzing the technical and strategic aspects of
badminton matches. The scoring system is based on individual rallies, where
each rally is evaluated separately. Participants can earn a maximum of $1$
point per rally based on the accuracy of the rally’s shot count and the
precision of the predicted attributes. The final rankings are determined by
the total scores, with the highest score securing the top position and the
lowest score occupying the last spot.

##  2 Related Work

For this badminton competitions, it is common practice to use CoachAI Hsu et
al. (2019), which is an AI-based coaching assistant system aimed at providing
professional guidance and personalized advice in the field of sports. The
system utilizes advanced machine learning and deep learning techniques to
extract crucial information from athletes’ data and videos, enabling
comprehensive analysis and evaluation. Next, we will introduce the deep
learning models and techniques utilized in CoachAI.

###  2.1 TrackNetV2

TrackNetV2 Sun et al. (2020) is an advanced computer vision model designed
specifically for object tracking tasks. Building upon its predecessor,
TrackNetV1, this updated version offers enhanced performance and improved
accuracy in tracking objects across video sequences. It employs deep learning
techniques, particularly convolutional neural networks (CNNs), to extract
relevant features from consecutive frames of a video. These features are then
used to predict the location and trajectory of a specified object in
subsequent frames. By learning the spatiotemporal patterns and motion
characteristics of the tracked object, TrackNetV2 can accurately follow its
movements over time.

After obtaining the predicted badminton trajectories from TrackNetV2, further
processing steps are performed, including removing extreme offset points,
curve fitting, and interpolation. Once the data is prepared accordingly,
roughly rally segmentation and event detection are applied to identify
specific events within the processed videos.

###  2.2 Court Detection

Court detection in Coach AI plays a crucial role in enhancing the accuracy and
effectiveness of the system, enabling it to provide valuable insights and
actionable information based on the specific dynamics and requirements of the
sports court.

###  2.3 MoveNet

MoveNet is a deep learning model designed for human pose estimation and
tracking. It is specifically developed to accurately detect and track human
body movements in real-time from video or camera input. This will be further
used to analyze the BallType played by the players.

###  2.4 OpenPose

OpenPose is a computer vision library and framework for real-time multi-person
keypoint detection and pose estimation. It enables the accurate estimation of
human body poses, including the positions of body joints such as the head,
shoulders, elbows, wrists, hips, knees, and ankles. Using OpenPose nodes in
this competition allows for more precise identification of the location of the
hitter and the defender.

##  3 Methodology

The methods we employ include video processing, SwingNet, ViT, YOLOv5, and
TrackNetV2 deep learning models.

###  3.1 Video Processing

Video processing is a critical step to improve the score in the entire task.
In this process, we first calculate the optical flow images using the method
of Optical Flow Calculation embedded in Reynolds Transport Theorem111A Prior
Feature Enhanced Layer for Small Objects Detection and Tracking Bruhn et al.
(2005); Harouna and Mémin (2017) to capture the prior features of the video,
then, we remove the background information to achieve a mechanism similar to
attention as shown in Figure 1. And we will use the term ”optical flow video
without background” to specifically describe this type of video. Subsequently,
we input these processed video into SwingNet for shuttlecock hitting event
detection.

Figure 1: Comparison between the original and optical frames.

###  3.2 SwingNet

SwingNet McNally et al. (2019) plays a crucial role in the field of computer
vision and sports analytics, specifically in the domain of golf. Its
application enables more efficient and detailed analysis of golf swings,
contributing to the overall development and understanding of the sport. In
here, we utilize the SwingNet, a deep learning model combining with the
MobileNetV2 and bidirectional LSTM structure, for shuttlecock hitting event
detection. That is, utilizing SwingNet enables us to extract the ShotSeq and
HitFrame features for the desired csv file. Figure 2 demonstrates the inferred
event probabilities for a video clip.

Figure 2: SwingNet performs event probability inference.

###  3.3 Vision Transformer

ViT, short for Vision Transformer Dosovitskiy et al. (2020), is a deep
learning model that applies the transformer architecture to computer vision
tasks. In contrast to the CNNs’ approach, ViT adopts a distinct strategy by
leveraging the transformer architecture, initially designed for tasks in
natural language processing. Moreover, the strength of ViT lies in its
attention mechanism, as depicted in Figure 3, accompanied by the corresponding
attention map illustrated in Figure 4. Here, we utilize ViT-B/16 to extract
the information including Hitter, RoundHead, Backhand, BallHeight, BallType,
and Winner from the videos.

Figure 3: Attention map for BallType classification task. Figure 4:
Visualization of attention maps for each layer.

###  3.4 YOLOv5

The YOLO-series detector is a renowned family of object detection models.
Specifically, YOLOv5 Zhu et al. (2021) is a popular and highly efficient
object detection model, which bilds upon the success of its predecessors,
YOLOv1, YOLOv2, and YOLOv3, and introduces several improvements to achieve
better performance in terms of accuracy and speed. We utilize YOLOv5m to
extract the information for LandingX, LandingY, HitterLocationX,
HitterLocationY, DefenderLocationX, and DefenderLocationY.

To begin with, we opted for YOLOv5m as our chosen detection model. The
rationale behind not selecting newer models like YOLOv7 is based on my
experience. While YOLOv7 may achieve higher scores on benchmark datasets, it
lacks the desired level of robustness. Additionally, once we have obtained the
detection results for both players and the ball, we integrate them with the
Hitter prediction generated by ViT. We designate the output generated by ViT
as the hitter, while the remaining player is assigned the role of the
defender. As the landing result of the ball is the desired information, we
utilize the y-value of the bottom-right corner of hitter’s bounding box to
represent the ball’s landing y-value. The landing coordinate of the ball is
indicated by the black cross symbol depicted in Figure 5. Next, we use the
vertices of the detection boxes closest to the ball as the positions of the
Hitter and the Defender for the AICUP competition, as illustrated by the light
blue and orange rectangles in Figure 5.

Figure 5: Detection result from YOLOv5m.

###  3.5 TrackNetV2

The intention behind incorporating TrackNetV2 in this context is to compensate
for the shortcomings of YOLO in detecting badminton. Nevertheless, there may
still arise scenarios where both YOLOv5m and TrackNetV2 fail to detect the
shuttlecock simultaneously. In that case, I will assign the coordinates of
LandingX and LandingY as $(0,0)$.

###  3.6 YOLOv8-pose

YOLOv8 Jocher et al. (2023) is the latest version of YOLO by Ultralytics.
Being at the forefront of advanced technology, YOLOv8 represents a SOTA model
that builds upon the achievements of its predecessors. It incorporates
innovative features and enhancements to deliver superior performance,
versatility, and efficiency. Furthermore, YOLOv8 provides comprehensive
support for a wide array of vision AI tasks, encompassing detection,
segmentation, pose estimation, tracking, and classification. In the CodaLab
competition, we adopt a different approach for determining the positions of
the players. Instead of relying on the vertices of the detection boxes, we
utilize the pose estimated by YOLOv8x-pose-p6 model to acquire the players’
foot positions with enhanced accuracy. The image with the updated method is
depicted in Figure 6.

Figure 6: Detection results from YOLOv5m and YOLOv8-pose.

##  4 Results and Discussion

This section encompasses the Evaluation Criteria, Experimental Results, and
Discussion.

###  4.1 Evaluation Criteria

The scoring system is based on each individual video clip, with a maximum
score of $1$ point per clip. The first step is to validate the number of shots
within the rally video, with each row in the corresponding csv file
representing one shot. If the predicted number of shots differs from the
ground truth, the question will receive a score of $0$ points. However, if the
number of shots is accurately predicted, a score of $0.1$ points will be
awarded, and the evaluation will proceed to the next stage, which involves
comparing the contents of the columns. In this stage, the maximum score for
correctly matching column content is $0.9$ points.

When the number of shots is accurate for the test video, each shot is
evaluated individually, and its score is determined based on the cumulative
score of its columns. The average score of all the columns is then used as the
shot’s content score. The evaluation process begins by validating the HitFrame
column. If the prediction error exceeds $2$ frames compared to the true value,
the shot is awarded $0$ points in terms of content and the scoring proceeds to
the next shot. However, if the prediction error is within or equal to $2$
frames, a score of $0.1$ points is given, and the remaining columns within the
same shot are compared.

  * •

Hitter: $0.1$ points if correct, $0$ points otherwise.

  * •

BallHeight: $0.1$ points if correct, $0$ points otherwise.

  * •

Landing: If the prediction error is not greater than $6$ pixels in the
Euclidean distance, it is considered correct and is awarded $0.1$ points;
otherwise, $0$ points.

  * •

HitterLocation: If the prediction error is not greater than $10$ pixels in the
Euclidean distance, it is considered correct with $0.05$ points, otherwise $0$
points.

  * •

DefenderLocation: If the prediction error is not greater than $10$ pixels in
the Euclidean distance, it is considered correct with $0.05$ points, otherwise
$0$ points.

  * •

Backhand: $0.05$ points if correct, $0$ points otherwise.

  * •

RoundHead: $0.05$ points if correct, $0$ points otherwise.

  * •

BallType: $0.2$ points if correct, $0$ points otherwise.

  * •

Winner: $0.1$ points if correct, $0$ points otherwise. (Note: If it is not the
last shot but is filled in, it will be considered incorrect.)

Assuming that there are R rally videos in the data set, and the $i$-th video
has $S_{i}$ shots, the scoring formula is given below:

$\frac{1}{R}\sum\limits_{i=1}^{R}1_{S_{i}=S_{i}^{pred}}(0.1+ASS_{i})$ (1)

in which $ASS_{i}$ (the Average Shot Score of content of the $i$-th rally
video) is given by:

$ASS_{i}=\frac{1}{S_{i}}\sum\limits_{j=1}^{S_{i}}1_{|HitFrame_{j}-HitFrame_{j}^{pred}\leq
2}SS_{j}$ (2)

with $SS_{j}$ (the $j$-th Shot Score) given by:

$\displaystyle SS_{j}$ $\displaystyle=0.1+0.1\times
1_{Hitter_{j}=Hitter_{j}^{pred}}$ (3) $\displaystyle+0.1\times
1_{BallHeight_{j}=BallHeight_{j}^{pred}}$ $\displaystyle+0.1\times
1_{||Landing_{j}-Landing_{j}^{pred}<6}$ $\displaystyle+0.05\times
1_{||HitterLocation_{j}-HitterLocation_{j}^{pred}<10}$
$\displaystyle+0.05\times
1_{||DefenderLocation_{j}-DefenderLocation_{j}^{pred}<10}$
$\displaystyle+0.05\times 1_{Backhand_{j}=Backhand_{j}^{pred}}$
$\displaystyle+0.05\times 1_{RoundHead_{j}=RoundHead_{j}^{pred}}$
$\displaystyle+0.2\times 1_{BallType_{j}=BallType_{j}^{pred}}$
$\displaystyle+0.1\times 1_{Winner_{j}=Winner_{j}^{pred}}$

###  4.2 Experimental Results

To begin with, we will discussing the hyperparameters and performance of
SwingNet, as this aspect determines the upper limit of the overall score. This
is due to the fact that any errors in the detection of ShotSeq and HitFrame
would render subsequent feature accuracy irrelevant, as they would not
contribute to the score. Subsequently, we will delve into the hyperparameters
and performance analysis of the ViT-B/16 and YOLOv5m models. Consequently, the
contribution of each model to the score will be illustrated at the conclusion
of this section. Note that since the official pretrained TrackNetV2 and
YOLOv8x-pose-p6 models are employed directly for inference, no further details
will be provided here.

####  4.2.1 Hyperparameters and Performance of SwingNet

To enhance the performance of SwingNet in capturing key frames more
effectively, we conducted numerous experiments focused on optimizing the
hyperparameters. During the training phase, various parameters could be fine-
tuned, including image size, sequence length, event ratio, the number of
frozen layers, batch size, and iteration number. Additionally, during the
inference phase, we can adjust parameters like image size, sequence length,
probabilities for the quantiles, and the filter threshold for successive
predicted events to further refine the results. After lots of trials, we have
determined the hyperparameters that lead to improved predictions for the
model. These optimized hyperparameters are summarized in Table 1.

Hyperparameters | Train | Inference
---|---|---
image size | 180$\times$180 | 180$\times$180
sequence length | 64 | 64
number of frozen layers | 10 | 10
batch size | 8 | 8
event ratio | 0.0305 | –
iteration number | 10000 | –
probabilities for the quantiles | – | 0.8
filter threshold | – | 3
Table 1: Optimized hyperparameters for SwingNet.

Once the hyperparameters of SwingNet have been determined, it becomes
essential to examine how various video representations influence the detection
results. We utilized three distinct types of input videos, including original
video, optical flow video (Opt video), and optical flow video without
background (Opt video w/o BG). Based on the information presented in Table 2,
it can be observed that utilizing the optical flow video without background
yields the highest score compared to other types of videos.

Video type | Iter. $\backslash$ Loss | Score
---|---|---
Original video | 2000 $\backslash$ 0.5821 | 0.0234
4000 $\backslash$ 0.5074 | 0.0275
6000 $\backslash$ 0.4622 | 0.0184
8000 $\backslash$ 0.4269 | 0.0276
10000 $\backslash$ 0.3979 | 0.0239
Opt video | 2000 $\backslash$ 0.3799 | 0.0229
4000 $\backslash$ 0.3325 | 0.0325
6000 $\backslash$ 0.3037 | 0.0333
8000 $\backslash$ 0.2839 | 0.0249
10000 $\backslash$ 0.2664 | 0.0334
Opt video w/o BG | 2000 $\backslash$ 0.3808 | 0.0274
4000 $\backslash$ 0.3294 | 0.0331
6000 $\backslash$ 0.2982 | 0.0349
8000 $\backslash$ 0.2760 | 0.0225
10000 $\backslash$ 0.2582 | 0.0263
Table 2: Performance of SwingNet on different types of videos.

Next, we made some architectural adjustments in an attempt to achieve higher
scores. These adjustments included MobileNetV3 + bidirectional LSTM and
MobileNetV2 + TCN. Before conducting the experiments, it was anticipated that
both of these architectures would outperform the original one, as MobileNetV3
Howard et al. (2019) and Temporal Convolution Network (TCN) Lea et al. (2017)
have been proven to be effective in other literature. However, compare to
Table 2, it can be clearly seen that the performance of the original
architecture still remained the best. The performance of tuned architectures
are shown in Table 3.

Architecture | Iter. $\backslash$ Loss | Score
---|---|---
MobileNetV3 | 2000 $\backslash$ 0.4364 | 0.0136
4000 $\backslash$ 0.3893 | 0.0293
+ | 6000 $\backslash$ 0.3628 | 0.0162
bidirectional LSTM | 8000 $\backslash$ 0.3437 | 0.0253
10000 $\backslash$ 0.3288 | 0.0115
MobileNetV2 | 2000 $\backslash$ 0.5417 | 0.0252
4000 $\backslash$ 0.5065 | 0.0313
+ | 6000 $\backslash$ 0.4816 | 0.0289
TCN | 8000 $\backslash$ 0.4613 | 0.0207
10000 $\backslash$ 0.4440 | 0.0234
Table 3: Performance of tuned architectures.

Therefore, in this study, we primarily used optical flow without background as
the main type of video for detection, and performed 5-fold inference. All the
scores for the 5-fold inference are presented in Table 4.

Iter.$\backslash$fold | 1 | 2 | 3 | 4 | 5
---|---|---|---|---|---
1000 | 0.0259 | 0.0262 | 0.0168 | 0.027 | 0.0263
2000 | 0.0241 | 0.0261 | 0.0354 | 0.034 | 0.0339
3000 | 0.0212 | 0.0352 | 0.037 | 0.0318 | 0.0426
4000 | 0.0275 | 0.0266 | 0.0389 | 0.0372 | 0.0399
5000 | 0.0288 | 0.027 | 0.0289 | 0.0366 | 0.0351
6000 | 0.0312 | 0.0167 | 0.0362 | 0.0269 | 0.0283
7000 | 0.0233 | 0.0372 | 0.0144 | 0.0326 | 0.0319
8000 | 0.0285 | 0.0347 | 0.0375 | 0.0396 | 0.0213
9000 | 0.0236 | 0.0237 | 0.0387 | 0.0279 | 0.0226
10000 | 0.0324 | 0.0182 | 0.0348 | 0.0262 | 0.0251
Table 4: Scores obtained from 5-fold inference.

After multiple trials, the highest score obtained in the first two columns, as
shown in Table 4, is $0.0426$. In addition, we attempted to ensemble the
detection results of SwingNet, but it did not effectively improve the score.
Furthermore, it is notable that the maximum score for the first two columns
can reach $0.2$ according to the evaluation criteria. This observation further
emphasizes that SwingNet’s predictions for the first two columns still have
great potential for improvement.

Upon completing the first two columns, we will exhibit the hyperparameters
employed for classification and detection tasks using ViT-B/16 and YOLOv5m,
respectively.

####  4.2.2 Hyperparameters of ViT-B/16 and YOLOv5m

Based on numerous experiments conducted, we have determined the optimal
hyperparameters that improve the performance of ViT-B/16 and YOLOv5m, as
presented in Table 5. To enhance the inference performance of the ViT-B/16 and
YOLOv5m models, the training strategy involves maximizing the image size. The
main reason behind this setting is that larger image sizes have a tendency to
uncover finer details, thereby significantly enhancing the value of both
training and inference processes. In addition, I divided the training data
into 5-fold for the classification tasks.

Hyperparameters | ViT-B/16 | YOLOv5m
---|---|---
image size | 480 | 2880
optimizer | SGD | SGD
learning rate | 3E-2 | 1E-2
loss function | CE | BCE
batch size | 4 | 1
iteration number | 10000 | –
epochs | – | 100
Table 5: Hyperparameters of ViT-B/16 and YOLOv5m.

Once the hyperparameters of ViT-B/16 and YOLOv5m have been determined, it is
important to delve into the stage of model inference. After completing the
intact training phase, I will have a total of $30$ ViT-B/16 models for these
classification tasks, specifically $5$ models for each. In the inference
process, I have employed ensemble techniques, namely vote ensemble and mean
ensemble. These approaches involve merging the predictions of multiple models
by either averaging the probabilities or selecting the choice that receives
the most votes, which aims to enhance the overall performance of the
prediction. Table 6 illustrates the comparison of scores achieved with and
without the application of these ensemble techniques, using the previous
predictions made by SwingNet as a baseline with a score of $0.0426$.

Features$\backslash$Ensemble | fold1 | Vote | Mean
---|---|---|---
Hitter | +0.0068 | +0.0068 | +0.0068
RoundHead | +0.0033 | +0.0033 | +0.0033
Backhand | +0.0031 | +0.0031 | +0.0031
BallHeight | +0.0066 | +0.0066 | +0.0067
BallType | +0.0089 | +0.0091 | +0.0089
Winner | +0.0005 | +0.0005 | +0.0006
Table 6: Ablation study of ensemble techniques.

It can be observed that there is only a slight difference between using and
not using ensemble techniques. Furthermore, the variations among different
ensemble techniques are also small from Table 6. The main factor contributing
to this observation is the notable disparity in probabilities predicted by
ViT-B/16. As illustrated in Figure 3, the model assigns a remarkably high
probability of $0.9745$ to the second category, while the probability of the
ball belonging to the next highest category only $0.0255$, the disparity
between these probabilities is substantial. Consequently, the utilization of
ensemble techniques does not significantly impact the classification results,
as the dominant probability already indicates a clear classification
preference. Therefore, we can train only one ViT-B/16 model for each feature
during classification tasks, rather than five models. This approach greatly
reduces the time and resources required for training and inference.

Now, let’s analyze the individual contributions of different models to their
respective features. Firstly, we will use SwingNet to obtain the predicted
answers for ShotSeq and HitFrame. Subsequently, we utilize ViT-B/16 for
predicting the features related to the classification task. Furthermore, we
employ the YOLOv5m model with TrackNetV2 to detect the position of the players
and the trajectory of the badminton. Moreover, we utilized the YOLOv8x-pose-p6
model to estimate the poses of players, resulting in a more accurate detection
of their positions as shown in Figure 6. These steps allow us to generate the
complete submission file. The corresponding contribution table can be found in
Table 7.

Features | Models | Score
---|---|---
ShotSeq | SwingNet | 0.0426
&HitFrame
Hitter | ViT | 0.0494 (+0.0068)
RoundHead | ViT | 0.0527 (+0.0033)
Backhand | ViT | 0.0558 (+0.0031)
BallHeight | ViT | 0.0625 (+0.0067)
LandingX | YOLOv5m | 0.0625 (+0.0000)
&LandingY | +TrackNetV2
HitterLocationX |  YOLOv5m +YOLOv8x -pose-p6 | 0.0630 (+0.0005)
&HitterLocationY
&DefenderLocationY
&DefenderLocationY
BallType | ViT | 0.0721 (+0.0091)
Winner | ViT | 0.0727 (+0.0006)
Table 7: The distinct contributions of various models to the features.

###  4.3 Discussion

In this section, we will primarily address four key points of discussion. The
first one is the extensive GPU memory resources required when using the method
proposed in this article for shuttlecock hitting event detection with
SwingNet. The main reason is that the model reads in $64$ frames of video at
once for evaluation. For example, if we input $64$ full-color images with
resolution of $720p$, the model needs to accommodate a size of $64\times
24\times 1280\times 720/8/1000\approx 169$GB. Due to the limited $6$GB memory
of my GPU, the maximum image resolution we can use for training and testing is
only $180\times 180$. I believe that if there is sufficient GPU memory
available, utilizing the method proposed in this article and increasing the
image size could be a promising approach. Compared to analyzing the trajectory
of the shuttlecock, this method is more intuitive and can overcome situations
where the shuttlecock is obstructed by the player.

The second point is that, according to the rules of badminton, we can infer
that the information in the Hitter column will alternate between $A$ and $B$
$(A\rightarrow B\rightarrow A\rightarrow B\cdots)$. This implies that there
will never be more than two consecutive instances of either $A$ or $B$.
Consequently, by predicting the first hitter, we can determine the order of
each hitter in a video. It is crucial to emphasize that when employing the
alternative showup method, the accuracy of the predicted first hitter becomes
paramount. Any inaccuracies in this prediction may result in a complete
reversal of the order for every hitter in the entire video.

The third point to consider is that by exclusively focusing on the hitter
during the classification process, it is possible to enhance the performance
of the classification problem. That is, instead of utilizing the entire image
for classification as depicted in Figure 3, one can solely utilize the area
occupied by the hitter. Implementing this method has the potential to decrease
the defender’s propensity to confuse hitter’s information, such as RoundHead,
Backhand, and BallType.

Lastly, it is worth noting that in this competition, several features were not
utilized while training, such as LandingX, LandingY, HitterLocationX,
HitterLocationY, DefenderLocationX, DefenderLocationY. Without fully utilizing
the available information, I believe there will be a certain degree of
misalignment. Henceforth, we will analyze the unused information with the aim
of leveraging it to aid in both the training and inference stages.

##  5 Summary

This article proposes a new perspective for the shuttlecock hitting event
detection task, which involves using a deep learning model to learn the
meaning conveyed by the image sequence, and thus capturing the desired
information from the video. Furthermore, the video processing method presented
in this article can effectively reduce the difficulty of model learning,
resulting in better performance in keyframe detection task. In the future, we
will continue to explore adjustments to the model architecture in order to
construct a more precise method for obtaining keyframes. This will be combined
with suitable image processing techniques to further enhance the overall
framework.

## Acknowledgments

I am delighted to have the opportunity to integrate the models used in this
competition and engage in deeper discussions about their structures and
mechanisms. I am also grateful to my family and friends for their unwavering
support and encouragement throughout this period, which enabled me to fully
immerse myself in the contests and generate innovative ideas. I truly
appreciate it.

## References

  * Bruhn et al. [2005] Andrés Bruhn, Joachim Weickert, and Christoph Schnörr.  Lucas/kanade meets horn/schunck: Combining local and global optic flow methods.  International journal of computer vision, 61:211–231, 2005.
  * Dosovitskiy et al. [2020] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * Harouna and Mémin [2017] S Kadri Harouna and Etienne Mémin.  Stochastic representation of the reynolds transport theorem: revisiting large-scale modeling.  Computers & Fluids, 156:456–469, 2017.
  * Howard et al. [2019] Andrew Howard, Mark Sandler, Grace Chu, Liang-Chieh Chen, Bo Chen, Mingxing Tan, Weijun Wang, Yukun Zhu, Ruoming Pang, Vijay Vasudevan, et al.  Searching for mobilenetv3.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 1314–1324, 2019.
  * Hsu et al. [2019] Tzu-Han Hsu, Ching-Hsuan Chen, Nyan Ping Jut, Tsi-Ui Ik, Wen-Chih Peng, Yu-Shuen Wang, Yu-Chee Tseng, Jiun-Long Huang, Yu-Tai Ching, Chih-Chuan Wang, et al.  Coachai: A project for microscopic badminton match data collection and tactical analysis.  In 2019 20th Asia-Pacific Network Operations and Management Symposium (APNOMS), pages 1–4. IEEE, 2019.
  * Jocher et al. [2023] Glenn Jocher, Ayush Chaurasia, and Jing Qiu.  YOLO by Ultralytics, January 2023.
  * Lea et al. [2017] Colin Lea, Michael D Flynn, Rene Vidal, Austin Reiter, and Gregory D Hager.  Temporal convolutional networks for action segmentation and detection.  In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 156–165, 2017.
  * McNally et al. [2019] William McNally, Kanav Vats, Tyler Pinto, Chris Dulhanty, John McPhee, and Alexander Wong.  Golfdb: A video database for golf swing sequencing.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  * Sun et al. [2020] Nien-En Sun, Yu-Ching Lin, Shao-Ping Chuang, Tzu-Han Hsu, Dung-Ru Yu, Ho-Yi Chung, and Tsì-Uí İk.  Tracknetv2: Efficient shuttlecock tracking network.  In 2020 International Conference on Pervasive Artificial Intelligence (ICPAI), pages 86–91. IEEE, 2020.
  * Zhu et al. [2021] Xingkui Zhu, Shuchang Lyu, Xu Wang, and Qi Zhao.  Tph-yolov5: Improved yolov5 based on transformer prediction head for object detection on drone-captured scenarios.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 2778–2788, 2021.


# Performance Analysis of a Foreground Segmentation Neural Network Model

Joel Tomás Morais1, António José Borba Ramires Fernandes1, André Leite
Ferreira2 and Bruno Faria2
1Minho University
2Bosch Car Multimedia Portugal S.A.
<EMAIL_ADDRESS>arf@di.uminho.pt,{Andre.Ferreira2,
<EMAIL_ADDRESS>

###### Abstract

In recent years the interest in segmentation has been growing, being used in a
wide range of applications such as fraud detection, anomaly detection in
public health and intrusion detection. We present an ablation study of
FgSegNet_v2, analysing its three stages: (i) Encoder, (ii) Feature Pooling
Module and (iii) Decoder. The result of this study is a proposal of a
variation of the aforementioned method that surpasses state of the art
results. Three datasets are used for testing: CDNet2014, SBI2015 and
CityScapes. In CDNet2014 we got an overall improvement compared to the state
of the art, mainly in the LowFrameRate subset. The presented approach is
promising as it produces comparable results with the state of the art (SBI2015
and Cityscapes datasets) in very different conditions, such as different
lighting conditions

##  1 INTRODUCTION

Over the years object detection has seen different application domains with
the aim of detecting an object type and location in a specific context of an
image. Nevertheless, for some applications detecting an object location and
type is not enough. For these cases, the labelling of each pixel according to
its surroundings in an image presents itself as an alternative approach. This
task is known as segmentation [Hafiz and Bhat, 2020] and finds applications in
video monitoring, intelligent transportation, sports video analysis,
industrial vision, amongst many other fields [Setitra and Larabi, 2014]. Some
of the techniques used in traditional segmentation include Thresholding,
K-means clustering, Histogram-based image segmentation and Edge detection.
Throughout the last years, modern segmentation techniques have been powered by
deep learning technology.

Segmentation methods isolate the objects of interest of a scene by segmenting
it into background and foreground. Throughout the aforementioned applications,
segmentation is usually applied to localizing the objects of interest in a
video using a fixed camera, presenting robust results.

Nevertheless, when the camera is not fixed or the background is changing,
detecting objects of interest using background subtraction could be more
demanding, generating diverse false positives. Moreover, it demands a high
computational cost when applied to videos [Liu et al., 2020]. Another issue
with the method consists in developing a self-adaptive background environment,
accurately describing the background information. This can be challenging
since the background could be changing a lot, e.g., in lighting and blurriness
[Minaee et al., 2020].

After exploring numerous state of the art methods in this field, FgSegNet_v2
was chosen as a suitable candidate to explore since it outperforms every state
of the art method in the [Wang, 2014] challenge. Through the analysis of its
components, and exploring variations for each, we have achieved a more robust
method that can cope with both fixed and moving cameras amongst datasets with
different scenarios.

This work is organized as follows: section 2 describes the current state of
the art in instance segmentation; section 3 presents FgSegNet_v2 architecture
and our proposed variations; in section 4 we introduce the used metrics and
datasets, followed by the experiments made in the FgSegNet_v2, followed by our
results showing that the current implementation achieves state of the art
performance. A conclusion and some avenues for future work are presented in
section 5.

##  2 RELATED WORK

Background subtraction has been studied in the Statistics field since the 19th
century [Chandola et al., 2009]. Nowadays, this subject is being strongly
developed in the Computer Science field, from using arbitrary to automated
techniques [Lindgreen and Lindgreen, 2004], in Machine Learning, Statistics,
Data Mining and Information Theory.

Some of the earlier background subtraction techniques are non-recursive, such
as _Frame differencing_ , _Median Filter_ and _Linear predictive filter_ , in
which a sliding window approach is used for the background estimation [ching
S. Cheung and Kamath, 2004], maintaining a buffer with past video frames and
estimating a background model based on the statistical properties of these
frames, resulting in a high consumption of memory [Parks and Fels, 2008]. As
for the recursive techniques, _Approximated Median Filter_ , _Kalman filter_
and _Gaussian related_ do not keep a buffer for background subtraction,
updating a single background based on the input frame [ching S. Cheung and
Kamath, 2004]. Due to the recursive usage, by maintaining a single background
model which is being updated with each new video frame, less memory is going
to be used when compared to the non-recursive methods [Parks and Fels, 2008].

Regarding segmentation, and using DL, the top state of the art techniques
include Cascade CNN [Wang et al., 2017b], _FgSegNet_v2_ [Lim and Keles, 2019]
and _BSPVGAN_ [Zheng et al., 2020]. These methods have achieved the top three
highest scores in the [Wang, 2014] challenge.

_Cascade CNN_ [Wang et al., 2017b] is a semi-automatic method for segmenting
foreground moving objects, it is an end-to-end model based on Convolutional
Neural networks (CNN) with a cascaded architecture. This approach starts by
manually selecting a sub-set of frames containing objects properly delimited,
which are then used to train a model. The model embodies three networks,
differentiating instances, estimating masks and categorizing objects. These
networks form a cascaded structure, allowing them to share their convolutional
features while also being able to generalize to cascades that have more
stages, maintaining the prediction phase extremely fast.

In _FgSegNet_v2_ [Lim and Keles, 2019], the aim is to segment moving objects
from the background in a robust way under various challenging scenarios. It
can extract multi-scale features within images, resulting in a sturdy feature
pooling against camera motion, alleviating the need of multi-scale inputs to
the network. A modified VGG 16 is used as an encoder for the network,
obtaining higher-resolution feature maps, which will be used as input for the
Feature Pooling Module (FPM) and consequently as input for the decoder,
working with two Global Average Pooling (GAP) modules.

_BSPVGAN_ [Zheng et al., 2020] starts by using the median filtering algorithm
in order to extract the background. Then, in order to classify the pixels into
foreground and background, a background subtraction model is built by using
Bayesian GANs. Last, parallel vision [Wang et al., 2017a] theory is used in
order to improve the background subtraction results in complex scenes. Even
though the overall results in this method don’t outperform the FgSegNet_v2, it
shows some improvements regarding lighting changes, outperforming other
methods in Specificity and False Negative changes.

##  3 EXPLORING VARIATIONS FOR FGSEGNET_V2

In this section, an ablation study of the FgSegNet_v2 components is presented,
in order to potentially improve the previous method by reworking individually
these components, obtaining a more robust method. The section is divided in
different modules, starting from the details of the architecture, along with
the description of the elements in it, i.e, Encoder, Feature Pooling module
and Decoder, ending with the training details.

Following a segmentation strategy, the proposed work intends to segregate the
background pixels from the foreground ones. In order to achieve this, a VGG-16
architecture is used as an encoder, in which the corresponding output is used
as input to a FPM ( Feature Pooling Module ) and consequently to a decoder.
The overall architecture can be seen in Figure 1.

Figure 1: Proposed method architecture.

###  3.1 Encoder

A modified version of VGG-16 [Simonyan and Zisserman, 2015] is used as the
encoder, both in FgSegNet_v2 and the proposed method, in which the dense
layers and the last convolutional block were removed, leaving the current
adaptation with 4 blocks. The first 3 blocks contain 2 convolutional layers,
each block followed by a MaxPooling layer. The last block holds 3
convolutional layers, each layer followed by Dropout [Cicuttin et al., 2016].
The architecture is depicted in Figure 2.

After comparing this configuration with other models, i.e., Inception v_3,
Xception and Inception ResNet v_2, we concluded that VGG-16 has given the best
results so far, as seen in Section 4.4.3 . This allows the extraction of the
input image features with high resolution, using these features as input to
the FPM.

Figure 2: VGG-16 architecture.

###  3.2 Feature Pooling Module

The Feature Pooling Module receives the features from the encoder in order to
correlate them at different scales, making it easier for the decoder to
produce the corresponding mask.

Lim [Lim and Keles, 2019] proposed a Feature Pooling Module, in which the
features from a 3 X 3 convolutional layer are concatenated with the Encoder
features, resulting as input to the following convolutional layer with
dilation rate of 4. The features from this layer are then concatenated with
the Encoder features, hence used as input to a convolutional layer with
dilation rate of 8. The process repeats itself to a convolutional layer with
dilation rate of 8. The output of every layer is concatenated with a pooling
layer, resulting in a 5 X 64 multi-features layers. These will finally be
passed through InstanceNormalization and SpatialDropout.

After numerous experiments and tests, an improvement to this configuration was
found. A convolutional layer with dilation rate of 2 is introduced, followed
by the removal of the pooling layer. The output of the last layer, i.e.,
convolutional with dilation rate of 16, proceeds to BatchNormalization [Ioffe
and Szegedy, 2015] (providing better results than InstanceNormalization) and
SpatialDropout [Cicuttin et al., 2016] (Figure 3).

Due to the removal of the multi-features layers, SpatialDropout is used
instead of Dropout. The former promoted independence between the resultant
feature maps by dropping them in their entirety, correlating the adjacent
pixels.

Figure 3: (1) Original Feature Pooling Module architecture and (2) Modified
Feature Pooling Module architecture.

##  4 EXPERIMENTS AND RESULTS

When training is concluded, a test set containing the raw images and their
corresponding masks is loaded, along with the trained model.

For each loaded image, a set of metrics are applied in order to evaluate the
performance of the model. To complement these metrics, some visual results are
also exploited. A probability mask is produced by applying a specific
threshold in order to remove the low probability scores. These probabilities
are translated in a Jet heatmap, i.e., when the color is red there is a high
probability of containing an object and when the color is purple there is a
low probability, which will be blend with the corresponding raw image, making
it easier to analyse the results.

###  4.1 Metrics

In order to fully understand the viability of the model, three different
metrics are used throughout the evaluation. The first one is the _AUC_ ( Area
Under Curve ) [Flach et al., 2011], being equal to the probability that the
classifier will rank a randomly chosen positive example higher than a randomly
chosen negative example. This metrics allows the perception of the model
detecting an object correctly, i.e., if the AUC is equal to 100% then the
model detected every object in it. Nevertheless, this metric only focus on
classification successes, failing to consider classification errors and not
providing information regarding the segmentation. This metric plots two
parameters: _TPR_ (true positive rate) and _FPR_ (false positive rate), seen
in Equations 1 and 2.

$TPR=\frac{TP}{TP+FN}$

(1)

$FRP=\frac{FP}{FP+TN}$

(2)

The _Accuracy_ and the _F-Measure_ could be potential candidates, since they
are highly used and usually produce reliable results, however, since the
dataset is highly imbalanced when regarding the foreground and the background
pixels, these two metrics are not an option, since they are highly sensitive
to imbalanced data, therefore _MCC_ (Mathews Correlation Coefficient ) is used
as an alternative [Chicco and Jurman, 2020], see Equation 3. This metric
ranges from $[-1,+1]$ where $+1$ represents a perfect segmentation according
to the ground truth, and $-1$ the opposite.

$MCC=\frac{(TP\times TN)-(FP\times FN)}{\sqrt{(TP+FP)(TP+FN)(TN+FN)(TN+FP)}}$

(3)

To complement the previous metric, _mIoU_ ( mean intersection over union ) is
also used. IoU is the area of overlap between the predicted segmentation and
the ground truth divided by the area of union between the predicted
segmentation and the ground truth, see Equation 4. When applied to a binary
classification, a mean is computed by taking the IoU of each class and
averaging them. This metric ranges from $[0,1]$ where $1$ represents a perfect
overlap with the ground truth and $0$ no overlap at all.

$IoU=\frac{\text{Area of Overlap}}{\text{Area of Union}}$

(4)

Metrics such as _Recall_ , _Specificity_ , _PWC ( Percentage of wrong
classifications )_ , _Precision_ and _F-Measure_ will also be used in some
cases, in order to allow a comparison with other state of the art methods.

###  4.2 Datasets

Three different datasets are used to evaluate our proposal. The first one is
_CityScapes_ , which pretends to capture images regarding outdoor street
scenes [Ramos et al.,]. Every image was acquired with a camera in a moving
vehicle across different months, this way different scenarios were covered,
such as seasons and different cities. It has $20000$ coarse pixel-level
annotations, where $5000$ of them were manually selected for dense pixel-level
annotations. This way, there will be high diversity in foreground objects,
background and scene layout. This dataset contains an additional challenge due
to the moving camera.

The second is _CDNet2014_ dataset [Wang et al., 2014], it consists of 31
camera-captured videos with a fixed camera, containing eleven categories:
Baseline, Dynamic Background, Camera Jitter, Intermittent Object Motion,
Shadows, Thermal, Challenging Weather, Low Frame-Rate, Night, PTZ and Air
Turbulence. The spatial resolution from the captured images ranges from
320x240 to 720x576. This dataset covers various challenging anomaly
detections, both indoor and outdoor.

The last dataset is SBI2015 ( Scene Background Initialization )
[Christodoulidis and Anthimopoulos, 2015]. It contains frames taken with a
fixed camera from 14 video sequences with its corresponding ground-truths. It
was assembled with the objective of evaluating and comparing the results of
background initialization algorithms.

###  4.3 Training details

The proposed method receives a raw image and its corresponding mask, i.e.,
ground-truth, as input, both $512\times 512$. A pre-trained VGG-16 on ImageNet
is used. The first three layer blocks of the VGG-16 are frozen in order to
apply transfer learning to our model.
Binary cross entropy loss is used as the loss function, assigning more weight
to the foreground in order to address the imbalancement in the data, i.e.,
number of foreground pixels much higher than the background ones, and also
taking in consideration when there are not any foreground pixels. The
optimizer used is Adam [Kingma and Ba, 2015], as it provides better results
amongst other tested optimizers, i.e., RMSProp, Gradient Descent, AdaGrad
[Duchi et al., 2012] and AdaDelta [Zeiler, 2012], and the batch size is 4.
A maximum number of 80 epochs is defined, stopping the training if the
validation loss does not improve after 10 epochs. The learning rate starts at
1e-4, and is reduced after 5 epochs by a factor of 0.1 if the learning
stagnates.

###  4.4 Experiments

Using [Lim and Keles, 2019] as a starting point, some ablation tests were made
in CDNet2014 dataset using 25 frames for training, in order to understand the
importance of the different components in the architecture and what could be
improved. The metrics used throughout these evaluations are presented in 4.1.

Note that results presented in the following subsections have as only purpose
the comparison of the proposed changes. Hence, these do not represent an
improvement over the original FgSegNet_v2. Only when combined do these changes
provide an increase in performance.

####  4.4.1 Feature Pooling Module

The goal is to break down the Feature Pooling Module, understanding its
components and evaluate if improvements could be done.

The first stage consisted in changing the layers with different dilation
rates, these dilated convolutions are used in order to systematically
aggregate multi-scale contextual information without losing resolution [Yu and
Koltun, 2016]. The tests are (1) Removing the convolutional layer with
dilation rate equal to $16$, (2) Add a convolutional layer with dilation rate
equal to $2$ and (3) Add a convolutional layer with dilation rate equal to $2$
and remove a convolutional layer with dilation rate equal to $16$. As seen in
Table 1, when removing the layer with dilation rate of $16$, information is
going to get lost, while when adding a layer with dilation rate equal to $2$,
better results are being obtained. Since adding a layer with dilation rate
equal to $2$ (2) improves the overall results, it will remain for the duration
of the project.

Table 1: Changing layers with different dilation rates in the Feature Pooling
Module.

| Baseline | LowFrameRate | BadWeather | CameraJitter
---|---|---|---|---
| F-M. |  MCC |  mIoU |  F-M. |  MCC |  mIoU |  F-M. |  MCC |  mIoU |  F-M. |  MCC |  mIoU
(1) | 0.9491 |  0.9313 |  0.9317 |  0.6250 |  0.6870 |  0.6930 |  0.8916 |  0.8817 |  0.8927 |  0.9030 |  0.8912 |  0.9000
(2) | 0.9731 |  0.9697 |  0.9721 |  0.7102 |  0.7054 |  0.7098 |  0.9172 |  0.9102 |  0.9148 |  0.9218 |  0.9199 |  0.9205
(3) | 0.9528 |  0.9487 |  0.9510 |  0.6970 |  0.6890 |  0.6900 |  0.9034 |  0.8987 |  0.9019 |  0.9190 |  0.9076 |  0.9119

The second stage is to change some of the concatenations between the layers.
These tests consist in (1) Remove output from layer with dilation rate of 2 to
the final concatenations, (2) Only concatenate layer with dilation rate=16 to
the pooling and layer with r=1, (3) Delete every connection from pooling
layer, (4) Only keep the final output of layer with dilation rate of 16 as
input to the SpatialDropout and (5) Delete every concatenation.

As seen in Table 2, when comparing the concatenations, (4) produces the best
results. Hence, the other concatenation tests will be discarded.

Table 2: Changing concatenations between layers in the Feature Pooling Module.

| Baseline | LowFrameRate | BadWeather | CameraJitter
---|---|---|---|---
| F-M. |  MCC |  mIoU |  F-M. |  MCC |  mIoU |  F-M. |  MCC |  mIoU |  F-M. |  MCC |  mIoU
(1) | 0.9281 |  0.9198 |  0.9260 |  0.6623 |  0.6596 |  0.6602 |  0.8916 |  0.8897 |  0.8904 |  0.9082 |  0.9047 |  0.9058
(2) | 0.9528 |  0.9502 |  0.9516 |  0.6898 |  0.6885 |  0.6890 |  0.8994 |  0.8978 |  0.8983 |  0.9036 |  0.9011 |  0.9023
(3) | 0.9804 |  0.9789 |  0.9799 |  0.7328 |  0.7311 |  0.7320 |  0.9207 |  0.9200 |  0.9202 |  0.9386 |  0.9368 |  0.9374
(4) | 0.9824 | 0.9817 | 0.9820 | 0.7517 | 0.7502 | 0.7513 | 0.9305 | 0.9296 | 0.9300 | 0.9492 | 0.9486 |  0.9490
(5) | 0.9528 | 0.9518 | 0.9520 | 0.6917 | 0.6909 | 0.6912 | 0.8923 | 0.8917 | 0.8920 | 0.9029 | 0.9010 |  0.9019

####  4.4.2 Decoder

The decoder in [Lim and Keles, 2019] uses three Convolutional layers 3X3,
followed by InstanceNormalization, and multiplies the output of the first two
layers with the output of the Encoder and GlobalAveragePooling (GAP).

The first stage consists in analysing the importance of the GAP, maintaining
the output of the Encoder or removing it ( according to its corresponding GAP
). The tests are (1) Remove the first GAP and its corresponding Encoder’s
output, (2) Remove the second GAP and its corresponding Encoder’s output, (3)
Remove both GAP and the Encoder’s output, (4) Remove first GAP but keep the
corresponding Encoder’s output, (5) Remove second GAP but keep the
corresponding Encoder’s output and (6) Remove both GAP but keep both Encoder’s
output.

As seen in Table LABEL:tab:GAP1, (4) and (6) produce the worst results,
decreasing the AUC, MCC and mIoU. As for the other configurations, they
produce an overall increase in every metric. When removing both GAP modules
and the Encoder’s outputs, option (3), the MCC and mIoU record the highest
values, therefore the other configurations will be discarded.

Table 3: Changes in the Decoder, by changing the configuration of the GAP. | Baseline | LowFrameRate
---|---|---
|  F-M. |  MCC |  F-M. |  MCC
(1) | 0.9527 | 0.9502 | 0.8010 | 0.7927
(2) | 0.9683 | 0.9650 | 0.8438 | 0.8412
(3) | 0.9856 | 0.9851 | 0.9680 | 0.9655
(4) | 0.8716 | 0.8678 | 0.7617 | 0.7600
(5) | 0.9628 | 0.9611 | 0.8017 | 0.8000
(6) | 0.8847 | 0.8826 | 0.7729 | 0.7711

After analysing the GAP, the relevance of the multiplications between the
output of the first two layers in the Decoder and the output of the Encoder
after applying the GAP must be considered. Therefore, additional tests were
performed: (1) Remove first multiplication, (2) Remove second multiplication
and (3) Remove both multiplications.

As seen in Table LABEL:tab:GAP2 the results decrease when compared to (3) from
the previous stage ( Table LABEL:tab:GAP1 ), hence these configurations will
be discarded.

Table 4: Changes in the Decoder, by changing the multiplications between the GAP and the dense layers. | Baseline | LowFrameRate
---|---|---
|  F-M. |  MCC |  F-M. |  MCC
(1) | 0.8926 | 0.8911 | 0.8618 | 0.8601
(2) | 0.9327 | 0.9316 | 0.9137 | 0.9122
(3) | 0.9126 | 0.9111 | 0.9017 | 0.8985

####  4.4.3 Encoder

Keeping the previous configurations of the FPM and the Decoder, three tests
were made in the Encoder, changing the VGG-16 backbone to (1) Inception v3
[Szegedy et al., 2016], (2) Xception [Chollet, 2017] and (3) Inception ResNet
v2 [Szegedy et al., 2017].

When comparing these three models with VGG-16, the latter produces better
results in every metric, e.g., in the F-Measure and MCC as seen in Table
LABEL:tab:encoder, while also keeping the number of parameters much lower.
Therefore no changes were made when compared to the FgSegNet_v2.

Table 5: Different Encoders. | Baseline | LowFrameRate
---|---|---
|  F-M. |  MCC |  F-M. |  MCC
(1) | 0.8816 | 0.8798 | 0.8126 | 0.8109
(2) | 0.9014 | 0.9002 | 0.8428 | 0.8406
(3) | 0.9218 | 0.9202 | 0.8828 | 0.8811

###  4.5 Final results and Comparison

With the previous configurations established, some results applied to the full
datasets are compared with the state of the art.

Regarding the CDNet2014 dataset, the proposed method outperforms the top state
of the art technique in this dataset, i.e., _FgSegNet_v2_ , when using 200
frames for training, improving by a long margin the LowFrameRate class, going
from 0.8897 to 0.9939 in the F-Measure, more details in Table 6. Some visual
results can also be seen in Figure 4, presenting results close to the ground
truth, even when dealing with LowFrameRate images.

Table 6: Results comparison between (1) Our proposed method and (2) FgSegnet v2 using 200 frames for training across the 11 categories. |  | FPR | FNR | Recall | Precision | PWC | F-Measure
---|---|---|---|---|---|---|---
Baseline | (1) | 6e-4 | 2e-3 |  0.9979 | 0.9974 |  0.0129 | 0.9975
(2) | 4e-5 | 3.8e-3 | 0.9962 | 0.9985 | 0.0117 | 0.9974
Low Fr. | (1) | 4.8e-3 | 4.1e-3 | 0.9956 | 0.9910 | 0.0569 | 0.9939
(2) | 8e-5 | 9.5e-2 | 0.9044 | 0.8782 | 0.0299 | 0.8897
Night V. | (1) | 7.4e-4 | 1.5e-2 | 0.9848 | 0.9785 | 0.1245 | 0.9816
(2) | 2.2e-4 | 3.6e-2 | 0.9637 | 0.9861 | 0.0802 | 0.9747
PTZ | (1) | 6e-4 | 2.3e-2 | 0.9888 | 0.9902 | 0.0471 | 0.9922
(2) | 4e-5 | 2.1e-2 | 0.9785 | 0.9834 | 0.0128 | 0.9809
Turbulence | (1) | 3e-4 | 2.5e-2 | 0.9861 | 0.9826 | 0.0417 | 0.9878
(2) | 1e-4 | 2.2e-2 | 0.9779 | 0.9747 | 0.0232 | 0.9762
Bad Wea. | (1) | 1.3e-4 | 1e-2 | 0.9913 | 0.9914 | 0.0379 | 0.9881
(2) | 9e-5 | 2.1e-2 | 0.9785 | 0.9911 | 0.0295 | 0.9848
Dyn.Back. | (1) | 3e-5 | 7.4e-3 | 0.9958 | 0.9959 | 0.0067 | 0.9960
(2) | 2e-5 | 7.5e-3 | 0.9925 | 0.9840 | 0.0054 | 0.9881
Cam.Jitter | (1) | 1.6e-4 | 2.5e-3 | 0.9974 | 0.9940 | 0.0275 | 0.9957
(2) | 1.2e-4 | 9.3e-3 | 0.9907 | 0.9965 | 0.0438 | 0.9936
Int.Obj.Mot. | (1) | 2.3e-4 | 7.4e-3 | 0.9925 | 0.9972 | 0.0672 | 0.9948
(2) | 1.5e-4 | 1e-2 | 0.9896 | 0.9976 | 0.0707 | 0.9935
Shadow | (1) | 4e-4 | 1.6e-2 | 0.9909 | 0.9942 | 0.0542 | 0.9938
(2) | 1e-4 | 5.6e-3 | 0.9944 | 0.9974 | 0.0290 | 0.9959
Thermal | (1) | 2e-4 | 2.7e-3 | 0.9972 | 0.9954 | 0.0302 | 0.9963
(2) | 2.4e-4 | 8.9e-3 | 0.9911 | 0.9947 | 0.0575 | 0.9929
Figure 4: Results comparison in the LowFrameRate class. The rows represent raw
images, ground-truth, FgSegNet_v2 output masks, difference between FgSegNet_v2
and ground truth, our proposed method output masks and difference between our
proposed method and ground truth, respectively.

In the SBI2015 dataset, the overall F-Measure decreased from 0.9853 to 0.9447
when compared with the _FgSegNet_v2_ , but increased from 0.8932 to 0.9447
when compared with the CascadeCNN, [Wang et al., 2017b]. Nevertheless, the
results still confirm a good overall evaluation on this dataset, compensating
the higher results assembled in the CDNet2014 dataset. More details can be
seen in Table LABEL:tab:sbi.

Table 7: Results on the SBI dataset on 13 categories.

| AUC | F-M. |  MCC
---|---|---|---
Board | 99.84 |  0.9734 |  0.9724
Candela |  99.92 | 0.9640 |  0.9631
CAVIAR1 | 97.25 | 0.9475 |  0.9466
CAVIAR2 | 97.51 | 0.9011 |  0.9001
Cavignal | 99.94 | 0.9881 |  0.9872
Foliage | 99.10 | 0.9124 |  0.9115
HallAndMonitor | 98.49 | 0.9169 |  0.9160
Highway1 | 99.35 | 0.9593 |  0.9583
Highway2 | 99.56 | 0.9528 |  0.9518
Humanbody2 | 99.82 | 0.9579 |  0.9580
IBMtest2 | 99.42 | 0.9521 |  0.9512
PeopleAndFoliage | 99.77 | 0.9570 |  0.9560
Snellen | 98.84 | 0.8977 |  0.8967

Last, some preliminary tests ( without direct comparison to other datasets )
were made in the CityScapes dataset in order to evaluate the behaviour of the
proposed method in a dataset with complex background changes, since
FgSegNet_v2 was not tested in such dataset. A test was made in three different
classes (Road, Citizens and Traffic Signs). As seen in Table
LABEL:tab:cityscapes and in Figure 5, the proposed method is able to detect
almost every object, confirmed by AUC metric.

Table 8: Results in the CityScapes dataset in the class Road, Citizens and Traffic Signs. | AUC | MCC | mIoU
---|---|---|---
Road | 99.61 | 0.9555 | 0.9564
Citizens | 99.31 | 0.7552 | 0.8019
Traffic Signs | 97.72 | 0.6618 | 0.7425
Figure 5: Results comparison in the CityScapes dataset. The columns represent
raw images, our proposed method output mask and ground truth, respectively.
The first row corresponds to Citizens class, second row to Road class and last
two to Traffic Signs class.

##  5 CONCLUSION

An improved FgSegNet_v2 is proposed in the presented paper. By changing the
Feature Pooling Module, i.e., deleting the pooling layer and only maintaining
the output from the layer with dilation rate of 16, and the Decoder, i.e.,
removing the GAP modules, a more simplified and efficient approach is made,
preserving the low number of needed training images feature while improving
the overall results. It outperforms every state of the art method in the
_ChangeDetection2014_ challenge, in particular in the _LowFrameRate_ images,
showing a very significant improvement and also maintaining great results in
the SBI and CityScapes datasets, resulting in a more generalized method than
the others since no experiments have been shown when using these datasets
simultaneously. As future work, we are going to focus on the _CityScapes_
dataset, maintaining or improving the good results in other datasets.

## ACKNOWLEDGEMENTS

This work is supported by: European Structural and Investment Funds in the
FEDER component, through the Operational Competitiveness and
Internationalization Programme (COMPETE 2020) [Project nº 039334; Funding
Reference: POCI-01-0247-FEDER-039334].

This work has been supported by national funds through FCT – Fundação para a
Ciência e Tecnologia within the Project Scope: UID/CEC/00319/2019.

## REFERENCES

  * Chandola et al., 2009 Chandola, V., BANERJEE, A., and KUMAR, V. (2009).  Survey of Anomaly Detection.  ACM Computing Survey (CSUR), 41(3):1–72.
  * Chicco and Jurman, 2020 Chicco, D. and Jurman, G. (2020).  The advantages of the Matthews correlation coefficient (MCC) over F1 score and accuracy in binary classification evaluation.  BMC Genomics, 21(1):1–13.
  * ching S. Cheung and Kamath, 2004 ching S. Cheung, S. and Kamath, C. (2004).  Robust techniques for background subtraction in urban traffic video.  In Panchanathan, S. and Vasudev, B., editors, Visual Communications and Image Processing 2004, volume 5308, pages 881 – 892. International Society for Optics and Photonics, SPIE.
  * Chollet, 2017 Chollet, F. (2017).  Xception: Deep learning with depthwise separable convolutions.  Proceedings - 30th IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2017, 2017-January:1800–1807.
  * Christodoulidis and Anthimopoulos, 2015 Christodoulidis, S. and Anthimopoulos, M. (2015).  Food Recognition for Dietary Assessment Using Deep Convolutional Neural Networks Stergios.  New Trends in Image Analysis and Processing – ICIAP 2015 Workshops, 9281(June):458–465.
  * Cicuttin et al., 2016 Cicuttin, A., Crespo, M. L., Mannatunga, K. S., Garcia, V. V., Baldazzi, G., Rignanese, L. P., Ahangarianabhari, M., Bertuccio, G., Fabiani, S., Rachevski, A., Rashevskaya, I., Vacchi, A., Zampa, G., Zampa, N., Bellutti, P., Picciotto, A., Piemonte, C., and Zorzi, N. (2016).  A programmable System-on-Chip based digital pulse processing for high resolution X-ray spectroscopy.  2016 International Conference on Advances in Electrical, Electronic and Systems Engineering, ICAEES 2016, 15:520–525.
  * Duchi et al., 2012 Duchi, J. C., Bartlett, P. L., and Wainwright, M. J. (2012).  Randomized smoothing for (parallel) stochastic optimization.  Proceedings of the IEEE Conference on Decision and Control, 12:5442–5444.
  * Flach et al., 2011 Flach, P., Hernández-Orallo, J., and Ferri, C. (2011).  A coherent interpretation of AUC as a measure of aggregated classification performance.  Proceedings of the 28th International Conference on Machine Learning, ICML 2011, (January 2014):657–664.
  * Hafiz and Bhat, 2020 Hafiz, A. M. and Bhat, G. M. (2020).  A survey on instance segmentation: state of the art.  International Journal of Multimedia Information Retrieval, 9(3):171–189.
  * Ioffe and Szegedy, 2015 Ioffe, S. and Szegedy, C. (2015).  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  32nd International Conference on Machine Learning, ICML 2015, 1:448–456.
  * Kingma and Ba, 2015 Kingma, D. P. and Ba, J. L. (2015).  Adam: A method for stochastic optimization.  3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, pages 1–15.
  * Lim and Keles, 2019 Lim, L. A. and Keles, H. Y. (2019).  Learning multi-scale features for foreground segmentation.  Pattern Analysis and Applications, (0123456789).
  * Lindgreen and Lindgreen, 2004 Lindgreen, A. and Lindgreen, A. (2004).  Corruption and unethical behavior: report on a set of Danish guidelines.  Journal of Business Ethics, 51(1):31–39.
  * Liu et al., 2020 Liu, Y., Shen, C., Yu, C., and Wang, J. (2020).  Efficient Semantic Video Segmentation with Per-frame Inference.  pages 1–17.
  * Minaee et al., 2020 Minaee, S., Boykov, Y., Porikli, F., Plaza, A., Kehtarnavaz, N., and Terzopoulos, D. (2020).  Image Segmentation Using Deep Learning: A Survey.  pages 1–23.
  * Parks and Fels, 2008 Parks, D. and Fels, S. (2008).  Evaluation of background subtraction algorithms with post-processing.  pages 192–199.
  * Ramos et al.,  Ramos, S., Rehfeld, T., Enzweiler, M., Benenson, R., and Roth, S.  The Cityscapes Dataset for Semantic Urban Scene Understanding.
  * Setitra and Larabi, 2014 Setitra, I. and Larabi, S. (2014).  Background subtraction algorithms with post-processing: A review.  Proceedings - International Conference on Pattern Recognition, pages 2436–2441.
  * Simonyan and Zisserman, 2015 Simonyan, K. and Zisserman, A. (2015).  Very deep convolutional networks for large-scale image recognition.  3rd International Conference on Learning Representations, ICLR 2015 - Conference Track Proceedings, pages 1–14.
  * Szegedy et al., 2017 Szegedy, C., Ioffe, S., Vanhoucke, V., and Alemi, A. A. (2017).  Inception-v4, inception-ResNet and the impact of residual connections on learning.  31st AAAI Conference on Artificial Intelligence, AAAI 2017, pages 4278–4284.
  * Szegedy et al., 2016 Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., and Wojna, Z. (2016).  Rethinking the Inception Architecture for Computer Vision.  Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2016-December:2818–2826.
  * Wang et al., 2017a Wang, K., Gou, C., Zheng, N., Rehg, J. M., and Wang, F. Y. (2017a).  Parallel vision for perception and understanding of complex scenes: methods, framework, and perspectives.  Artificial Intelligence Review, 48(3):299–329.
  * Wang, 2014 Wang, Y. (2014).  Change detection 2014 challenge.
  * Wang et al., 2014 Wang, Y., Jodoin, P. M., Porikli, F., Konrad, J., Benezeth, Y., and Ishwar, P. (2014).  CDnet 2014: An expanded change detection benchmark dataset.  IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops, pages 393–400.
  * Wang et al., 2017b Wang, Y., Luo, Z., and Jodoin, P. M. (2017b).  Interactive deep learning method for segmenting moving objects.  Pattern Recognition Letters, 96:66–75.
  * Yu and Koltun, 2016 Yu, F. and Koltun, V. (2016).  Multi-scale context aggregation by dilated convolutions.  4th International Conference on Learning Representations, ICLR 2016 - Conference Track Proceedings.
  * Zeiler, 2012 Zeiler, M. D. (2012).  ADADELTA: An Adaptive Learning Rate Method.
  * Zheng et al., 2020 Zheng, W., Wang, K., and Wang, F. Y. (2020).  A novel background subtraction algorithm based on parallel vision and Bayesian GANs.  Neurocomputing, 394:178–200.