# Study and Survey on Gesture Recognition Systems

Kshitij Deshpande  Information Technology
Pune Institute of Computer Technology
Pune, India
<EMAIL_ADDRESS>Varad Mashalkar  Information Technology
Pune Institute of Computer Technology
Pune, India
<EMAIL_ADDRESS>Kaustubh Mhaisekar  Information Technology
Pune Institute of Computer Technology
Pune, India
<EMAIL_ADDRESS>Amaan Naikwadi  Information Technology
Pune Institute of Computer Technology
Pune, India
<EMAIL_ADDRESS>Dr. Archana Ghotkar  Information Technology
Pune Institute of Computer Technology
Pune, India
<EMAIL_ADDRESS>
###### Abstract

In recent years, there has been a considerable amount of research in the
Gesture Recognition domain, mainly owing to the technological advancements in
Computer Vision. Various new applications have been conceptualised and
developed in this field. This paper discusses the implementation of gesture
recognition systems in multiple sectors such as gaming, healthcare, home
appliances, industrial robots, and virtual reality. Different methodologies
for capturing gestures are compared and contrasted throughout this survey.
Various data sources and data acquisition techniques have been discussed. The
role of gestures in sign language has been studied and existing approaches
have been reviewed. Common challenges faced while building gesture recognition
systems have also been explored.

###### Index Terms:

gestures, gesture recognition, computer vision, sign language

##  I Introduction

Gestures are nonverbal movements and signals made by the body, such as hand
movements, facial expressions, and body posture. They are an essential part of
human communication and can convey a wide range of meanings and emotions
without the use of words. Gestures can be used in HCI applications as they can
provide an intuitive way for users to interact with technology. This can
reduce the learning curve for new users and make the interaction more
accessible to a broader range of users. Gestures can also be a more efficient
way of completing actions and interacting with the system. They can be an
effective way to provide an accessible interaction for users with physical
disabilities or impairments. For example, using gestures for navigation can be
easier for users with limited mobility than using a mouse or keyboard.
Gestures facilitate communication with people who do not share a common
language. Also for people who are disabled and cannot communicate verbally,
sign language is used.
Sign language is a visual language that makes use of a combination of hand
gestures, facial expressions, and body movements to communicate with others
who are deaf or hard of hearing. Hand gestures, also known as signs, are the
most important part of sign language. Each sign corresponds to a word or
concept, and they are combined to form sentences and express complex ideas.
For example, the sign for “allergy” in the Indian Sign Language (ISL) involves
using one hand to scratch the other to convey the word “allergy”. Body posture
and movement are also important in sign language, for example, to convey the
sign “hospital” in Indian Sign Language one needs to turn sideways and draw a
“+” symbol on the side of their shoulder. Facial expressions are also used in
sign language, as they can convey emotional nuance and tone.
In this study, we describe the architecture and operation of a general gesture
recognition system. We will look at the different methodologies that can be
used for detecting and interpreting gestures by a system as well as various
applications of gesture detection. A comparative analysis of various tools and
techniques for recognising gestures, along with the comparison of various data
sources for the same is also conducted. We look at various benchmark datasets
in this domain and review multiple technologies and algorithms that have been
implemented in gesture recognition systems, along with the challenges faced
while doing so. We understand need of using gestures for communicating using
sign language and its applications as well.

##  II Gesture Recognition Systems and Datasets

Gesture recognition systems are an increasingly important area of research
that enables natural and intuitive interaction between humans and machines.
These systems rely on high-quality datasets to train and improve their
accuracy, making dataset creation and curation a crucial aspect of gesture
recognition research. In this review, we will explore some of the most notable
gesture recognition systems and datasets in the field.

###  II-A Algorithms

Eng-Jon, et al [1] propose to model the spatio-temporal signatures using
Sequential Interval Patterns (SIP), which is based on the Greek Sign Language.
A novel multi-class classifier is then used to organise these SIP into a
hierarchical

TABLE I: Comparison of Existing Sign Language Datasets Sr no. |  Name |  Country |  Classes |  Subjects |  Samples |  Language Level
---|---|---|---|---|---|---
1 |  SIGNUM |  Germany |  450 |  25 |  33210 |  Sentence
2 |  GSL 20 |  Greek |  20 |  6 |  840 |  Word
3 |  Boston ASL LVD |  USA |  3300+ |  6 |  9800 |  Word
4 |  PSL Kinect 30 |  Poland |  30 |  1 |  300 |  Word
5 |  LSA64 |  Argentina |  64 |  10 |  3200 |  Word
6 |  MSR Gesture 3D |  USA |  12 |  10 |  336 |  Word
7 |  DEVISIGN-L |  China |  2000 |  8 |  24000 |  Word
8 |  LSFB-CONT |  Belgium |  6883 |  100 |  85000+ |  Word / Sentence
9 |  LSFB-ISOL |  Belgium |  400 |  85 |  50000+ |  Word
10 |  WLASL |  EEUU |  2000 |  119 |  21083 |  Word

form which makes it possible to take advantage of any sub-sequence sharing
that may occur between various SIPs of various classes. Dilsizian, Mark, et
al. [2] put forward a new framework based on the American Sign Language that
offers a new tracking technique that is less reliant on laboratory settings
and can handle variations in background and skin regions (like the face,
forearms, or other hands) , identifies 3D hand configurations that are
significant for ASL linguistically and includes statistical data that reflects
linguistic restrictions on sign creation. Oszust, M., and M. Wysocki. [3]
compares the strategy using models of subunits to a complete word model
approach using the closest neighbour classifier which uses dynamic time
warping (DTW) based on the Polish Sign Language. Kinect’s skeletal pictures
were used for one set of features and simplified descriptions of the hands
retrieved as skin-colored regions were employed for the other. The second
strategy proved to be more effective.

Masood, Sarfaraz, et al. [4] implement a very straightforward approach to
Argentinian sign language recognition in video sequences, which consists of
temporal and spatial features. RNNs have been used for the former while CNNs
have been used for the latter. An accuracy of 95.217 was achieved. Li, Dongxu,
et al. [5] suggests a strategy that involves exploiting cross-domain knowledge
in subtitled sign news videos to enhance sign language recognition models
based on word-level American Sign Language. A domain-invariant descriptor is
learned for each isolated sign, class centroids are stored in prototypical
memory, and isolated signs and news signs are aligned. This improves
performance and reduces the need for labelling by enabling the classifier to
concentrate on learning class-specific features and localise sign words from
sentences automatically.

A general probabilistic model for categorising signs is presented by
Ronchetti, Franco, et al. [6] and includes sub-classifiers based on numerous
variables, including location, movement, and handshape in the Argentinian Sign
Language. In all categorization steps, the model employs a bag-of-words
strategy. The suggested model had a 97 percent accuracy rate when tested on a
dataset of Argentine Sign Language with 64 classes of signs and 3200 samples,
providing some evidence that recognition without the sequential ordering of
frames to limit the number of hypotheses is possible.

###  II-B Related Work

Bokstaller, Jonas, and Costanza Maria Improta. [7] present a basic fundamental
approach towards dynamic sign language recognition by simply training a
vanilla Conv3DRNN network on dynamic signs. It consists of a LSTM layer that
is fed by a 3D-CNN, performing with a best accuracy of around 77 percent.
Chen, Yutong, et al. [8] demonstrate an approach, in which two different
streams are used to model RGB videos and keypoint sequences. These are then
used together for sign language recognition by applying techniques like
bidirectional lateral connection which does an element wise addition between
the two feature maps. A sign pyramid network (SPN) is defined that involves a
top-down pathway and a lateral connection to generate frame level
probabilites. Hu, Hezhen, et al. [9] modelled an algorithm called SignBERT
which is a self-supervised pretrainable sign language recognition system with
model-aware hand prior incorporated. In the first self supervised pre training
step it masks and reconstructs the hand, and captures the temporal context of
the sign. In the second step the model is fine tuned to the task of sign
language recognition by replacing the decoder from the intial model by a
prediction head. Boh aˇcek, Maty aˇs, and Marek Hr uz. [10] presented a paper
that shows a new approach for word-level sign language recognition based on a
sign pose-based transformer model. The proposed method takes sign language
videos as input and extracts pose information to represent the sign language
signs. The extracted pose information is then fed into a transformer model
that learns to recognize the corresponding sign language word.

Zuo, Ronglai, Fangyun Wei, and Brian Mak. [11] demonstrated an approach in
which a new method for sign language recognition that utilizes natural
language processing (NLP) techniques is proposed. The proposed system takes
advantage of the fact that sign languages often have a natural language
equivalent and leverages this to improve accuracy in recognizing signs. The
system uses a pre-trained NLP model to generate possible natural language
phrases for a given sign, and then compares these phrases to a database of
known sign language phrases to determine the correct sign.

Lai, Kenneth, and Svetlana N. Yanushkevich [12] propose a method that uses the
depth and skeleton information of the hand to detect and classify hand
gestures. The method consists of two main stages: a pre-processing stage and a
gesture recognition stage. In the pre-processing stage, the depth and skeleton
data of the hand are extracted using a Kinect sensor. Then, the data is
normalized and segmented to extract individual gestures. In the gesture
recognition stage, the normalized gesture data is fed into a CNN to extract
features, followed by an RNN to capture the temporal dynamics of the gesture.
The output of the RNN is then fed into a softmax classifier for gesture
recognition. Alam, Mohammad Mahmudul, Mohammad Tariqul Islam, and SM Mahbubur
Rahman. [13] also demostrate a unified learning approach for simultaneous
recognition of egocentric hand gestures and detection of fingertips in first-
person videos. The proposed method utilizes a single deep learning network
that takes raw video frames as input and produces hand gesture labels and
fingertip locations as outputs. The model is trained on a large-scale
egocentric hand gesture dataset and achieves state-of-the-art performance in
both hand gesture recognition and fingertip detection tasks.

Chen, Yuxiao, et al. [14] propose a method for recognizing hand gestures using
dynamic graphs constructed with spatial-temporal attention. The proposed
method creates a dynamic graph from extracted features where nodes represent
the features and the edges represent the relation between them. A spatial-
temporal attention mechanism is employed to assign weights to the nodes and
edges.

Chen, Huizhou, et al. [15] demonstrate a new method for recognizing hand
gestures using a combination of 3D CNNs and multi-scale attention mechanisms.
The proposed network takes in multimodal data from depth and color cameras,
and learns to extract features at different spatial and temporal scales. The
network performs better overall when the multi-scale attention mechanism is
employed to selectively emphasise key features at various scales.

##  III Features used in gesture recognition

###  III-A Types of gestures

####  III-A1 Static gestures

Static gestures involve holding the hand or body in a specific pose or
configuration without any movement. Some common features used in static
gesture recognition include:

TABLE II: Techniques for feature extraction for static signs Feature |  Technique |  Tool
---|---|---
Hand shape |  Shape analysis |  OpenCV, MATLAB
Palm orientation |  Normalized depth map |  OpenCV, MATLAB
Hand position |  Convex hull and centroid detection |  OpenCV, MATLAB
Distance |  Euclidean distance between hand and other objects |  OpenCV, MATLAB
Texture |  Local binary patterns (LBP) |  OpenCV, MATLAB
Color |  Color histogram of selected regions |  OpenCV, MATLAB
Saliency |  Saliency map analysis |  OpenCV, MATLAB

####  III-A2 Dynamic gestures

Dynamic gestures are gestures that involve movement of the hand or body. Some
of the features used in dynamic gesture recognition are:

TABLE III: Techniques for feature extraction for dynamic signs Feature |  Technique |  Tool
---|---|---
Trajectory |  Motion tracking |  OpenCV, MATLAB
Velocity |  Finite difference method |  OpenCV, MATLAB
Acceleration |  Finite difference method |  OpenCV, MATLAB
Jerk |  Finite difference method |  OpenCV, MATLAB
Curvature |  Curvature estimation |  OpenCV, MATLAB
Hand shape |  Shape analysis |  OpenCV, MATLAB
Joint angles |  Kinematic modeling |  OpenCV, MATLAB
Frequency |  Fourier transform |  OpenCV, MATLAB

###  III-B Method of recognition

There are primarily 2 ways of recognising gestures, Vision based and Sensor
based. Vision-based gesture recognition systems rely on image and video
processing techniques. Sensor-based gesture recognition systems typically use
a combination of hardware sensors and signal processing algorithms to capture
and analyze gesture data.

TABLE IV: Techniques for feature extraction in vision-based recognition Feature |  Technique |  Tool
---|---|---
Hand and body pose |  Pose estimation |  OpenCV, TensorFlow
Hand and body movement |  Motion tracking |  OpenCV, MATLAB
Skin color and texture |  Color histogram, LBP |  OpenCV, MATLAB
Edge detection |  Canny edge detection |  OpenCV, MATLAB
Optical flow |  Lucas-Kanade method |  OpenCV, MATLAB
Depth information |  Structured light, Time-of-flight camera |  OpenCV, MATLAB
Hand landmarks |  Landmark detection |  OpenCV, TensorFlow
TABLE V: Techniques for feature extraction for sensor-based recognition Feature |  Technique |  Tool
---|---|---
Inertial measurement units |  Kalman filtering, quaternion |  MATLAB, Python
Acceleration |  Signal processing |  MATLAB, Python
Gyroscopic rotation |  Signal processing |  MATLAB, Python
Magnetic field |  Signal processing |  MATLAB, Python
Flex sensors |  Resistance measurement |  Arduino, Raspberry Pi
Pressure sensors |  Resistance measurement |  Arduino, Raspberry Pi
Infrared sensors |  Reflection measurement |  Arduino, Raspberry Pi
Ultrasonic sensors |  Time-of-flight measurement |  Arduino, Raspberry Pi

In summary, the features used in gesture recognition depend on the type of
gesture and the method of recognition. Static gestures involve hand shape,
orientation, and palm orientation, while dynamic gestures involve trajectory,
velocity, and acceleration. Vision-based recognition involves shape analysis,
motion analysis, and texture analysis, while sensor-based recognition involves
IMUs, flex sensors, and pressure sensors

##  IV Methodologies for capturing gestures

There are multiple methodologies identified for capturing gestures in a
computational environment. These methodologies can utilize various available
tools and provide a set of features as required. The selection of the required
methodology can be based on the provided features. Table VI of this paper
elaborates the features and challenges associated with these methodologies
along with their applications in various sectors

TABLE VI: Comparison of Data Collection Methods Source of Data Collection |  Challenges |  Tools |  Features |  Applications
---|---|---|---|---
Camera-based systems |  \- Limited field of view |  \- Webcams, Kinect |  \- High accuracy |  \- Human-computer interaction
|  \- Poor lighting conditions |  \- Leap Motion |  \- Can track multiple joints |  \- Sign language recognition
|  \- Occlusions |  \- Intel RealSense |  \- Can capture RGB and depth information |  \- Gaming
|  \- Privacy concerns |  \- OpenPose |  \- Can recognize body pose and movement |  \- Sports training
Wearable devices |  \- Limited battery life |  \- Smartwatches, armbands |  \- Continuous data collection |  \- Fitness tracking
|  \- Comfort and fit |  \- Smart gloves |  \- Can capture hand gestures |  \- Industrial control
|  \- Limited sensor range |  \- Myo armband |  \- Can recognize muscle activation |  \- Virtual reality
|  \- Complex data processing |  \- Nymi wristband |  \- Can authenticate user based on heart rate |  \- Healthcare
Inertial measurement units |  \- Calibration errors |  \- Accelerometers, gyroscopes |  \- Low power consumption |  \- Robotics
|  \- Drift and noise |  \- Magnetometers |  \- Small form factor |  \- Motion analysis
|  \- Limited accuracy |  \- IMU sensor fusion |  \- Can provide 3D orientation |  \- Gait analysis
|  \- Sensor placement |  |  |
Depth sensors |  \- Limited range |  \- Kinect, Intel RealSense |  \- Can capture 3D information |  \- Gaming
|  \- Poor lighting conditions |  \- Structure Sensor |  \- Can attach to mobile devices |  \- Robotics
|  \- Occlusions |  \- Zed Camera |  \- Can capture RGB and depth information |  \- Augmented reality
|  \- Complexity and cost |  \- Orbbec Astra |  \- Can track multiple users |  \- Human-computer interaction
Electromyography |  \- Limited sensor placement |  \- Surface EMG sensors |  \- Can capture muscle activation |  \- Prosthetics
|  \- Signal interference |  \- Intramuscular EMG sensors |  \- High accuracy |  \- Rehabilitation
|  \- Limited range |  \- Wireless EMG sensors |  \- Continuous data collection |  \- Sports training
|  \- Complex data processing |  |  |

##  V Gesture based applications

###  V-A Gaming

Gesture based interfaces provide the users with a better interaction than
tradition controls. Gestures can be used to control actions, directions and
make choices. However, the gesture recognition needs to be fast and accurate
to provide an absolute experience.

Vision based methods can identify gestures to reproduce the same in a gaming
environment. Nintendo Wii, Sony Playstation Eye, Microsoft Kinect and Leap
Motion are some popular gesture control devices used in the gaming industry.

###  V-B Healthcare

FAce MOUSe was introduced by Nishikawa et al. [16], which uses the facial
expressions of surgeons to control the position of laparoscope, thus
eliminating the use of direct contact devices. Wachs et al. [17] invented
Gestix, a real-time, vision-based hand gesture interpretation system that
employs the user’s gestures to navigate and manipulate pictures in an
electronic medical record (EMR) database. Through captured video and
navigation, gestures are converted to instructions based on their temporal
trajectories.

###  V-C Home Appliances

Users can lock or open doors, change camera angles, arm or disarm security
systems, and receive alerts when someone is detected all with the use of
simple hand gestures. With gesture recognition, users can control their TVs by
using simple hand gestures, such as swiping, pointing, and waving.

###  V-D Industrial Robots

Gesture recognition technology is particularly useful in environments where
workers need to keep their hands free, such as in cleanrooms, hazardous areas,
or when wearing protective gear.

ABB’s YuMi Robot is a collaborative robot that makes use of gesture
recognition technology to improve its usability and functionality in
industrial manufacturing environments. The robot’s gesture-based programming
allows operators to teach it new tasks simply by guiding its arms and hands
through the required movements.

###  V-E Virtual Reality

The hand tracking feature of the Quest Oculus VR headset enables you to use
your hands to navigate through various menu options. The motion data is used
as input to decide what action to take.

UltraLeap motion is integrated with virtual and augmented reality to let users
pick up objects without the use of a gadget. The Leap Motion Controller is a
compact USB device that lets users enter gesture commands into an application
by using infrared LEDs to track hand and finger movements.

##  VI Role of Gestures in Sign Language

Gestures play a crucial role in sign language as they are the basic units of
communication in this visual language. Signers use gestures to create signs,
which are combinations of movements, shapes, and locations that have specific
meanings. For example, the sign for ”happy” may involve a smiling facial
expression, an upward movement of the hands, and a specific handshape.

Gestures are also important in sign language because they can convey
grammatical information. Gestures can also be used to show tense, aspect, and
other aspects of grammar.

Gestures allow signers to convey complex information and grammatical nuances,
and they form the foundation of this unique visual language.

##  VII Common challenges faced

  * •

Variability in human gestures: Variability in gestures due to various
factors(culture, gender, age, lighting, ambiguity etc) makes it difficult to
design a universal gesture recognition system that can accurately interpret
different types of gestures.

  * •

Complex hand movements: Hand movements are complex and involve the use of
multiple joints and muscles and hence requires a sophisticated algorithm that
can analyze the subtle changes in hand position and movement for
interpretation.

  * •

Limited training data: Training a gesture recognition system requires a large
amount of data. However, there is a limited amount of training data available,
which can make it difficult to create accurate models.

  * •

Real-time processing: Real-time processing is necessary for many applications
of gesture recognition, such as gaming and virtual reality. However, real-time
processing requires high-speed computing and processing power, which can be a
challenge for some devices.

##  VIII Conclusion

In conclusion, this survey paper has explored the concepts of gestures and
gesture recognition systems, their applications and methodologies. It compares
and contrasts various tools and techniques for perceiving gestures. It also
analyses various data sources and corresponding data acquisition techniques.
We have seen that gestures are a fundamental means of communication and can
convey a wide range of meanings and emotions. Gesture recognition systems
enable machines to interpret and respond to human gestures, which has numerous
applications in various fields, including healthcare, gaming, and robotics.

In particular, gesture recognition systems have proven to be invaluable in
sign language recognition and translation, facilitating communication between
hearing and deaf communities. By recognizing and interpreting sign language
gestures, these systems enable deaf individuals to communicate more
effectively and bridge the communication gap between deaf and hearing
communities.

Overall, this survey paper provides a comprehensive overview of the state-of-
the-art in gesture recognition systems and highlights the current challenges
and future directions in this field.

## References

  * [1] Eng-Jon, et al. ”Sign spotting using hierarchical sequential patterns with temporal intervals.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2014.
  * [2] Dilsizian, Mark, et al. ”A New Framework for Sign Language Recognition based on 3D Handshape Identification and Linguistic Modeling.” LREC. 2014.
  * [3] Oszust, M., and M. Wysocki. ”Some approaches to recognition of sign language dynamic expressions with kinect.” Human-Computer Systems Interaction: Backgrounds and Applications 3 (2014): 75-86.
  * [4] Masood, Sarfaraz, et al. ”Real-time sign language gesture (word) recognition from video sequences using CNN and RNN.” Intelligent Engineering Informatics: Proceedings of the 6th International Conference on FICTA. Springer Singapore, 2018.
  * [5] Li, Dongxu, et al. ”Transferring cross-domain knowledge for video sign language recognition.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2020.
  * [6] Ronchetti, Franco, et al. ”Sign languague recognition without frame-sequencing constraints: A proof of concept on the argentinian sign language.” Advances in Artificial Intelligence-IBERAMIA 2016: 15th Ibero-American Conference on AI, San José, Costa Rica, November 23-25, 2016, Proceedings 15. Springer International Publishing, 2016.
  * [7] Bokstaller, Jonas, and Costanza Maria Improta. ”Dynamic Gesture Recognition.” arXiv preprint arXiv:2109.09396 (2021).
  * [8] Chen, Yutong, et al. ”Two-Stream Network for Sign Language Recognition and Translation.” arXiv preprint arXiv:2211.01367 (2022).
  * [9] Hu, Hezhen, et al. ”Signbert: pre-training of hand-model-aware representation for sign language recognition.” Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.
  * [10] Boháček, Matyáš, and Marek Hrúz. ”Sign pose-based transformer for word-level sign language recognition.” Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision. 2022.
  * [11] Zuo, Ronglai, Fangyun Wei, and Brian Mak. ”Natural Language-Assisted Sign Language Recognition.” arXiv preprint arXiv:2303.12080 (2023).
  * [12] Lai, Kenneth, and Svetlana N. Yanushkevich. ”CNN+ RNN depth and skeleton based dynamic hand gesture recognition.” 2018 24th international conference on pattern recognition (ICPR). IEEE, 2018.
  * [13] Alam, Mohammad Mahmudul, Mohammad Tariqul Islam, and SM Mahbubur Rahman. ”Unified learning approach for egocentric hand gesture recognition and fingertip detection.” Pattern Recognition 121 (2022): 108200.
  * [14] Chen, Yuxiao, et al. ”Construct dynamic graphs for hand gesture recognition via spatial-temporal attention.” arXiv preprint arXiv:1907.08871 (2019).
  * [15] Chen, Huizhou, et al. ”Multi-scale attention 3D convolutional network for multimodal gesture recognition.” Sensors 22.6 (2022): 2405.
  * [16] Atsushi Nishikawa, Toshinori Hosoi, Kengo Koara, Daiji Negoro, Ayae Hikita, Shuichi Asano, Haruhiko Kakutani, Fumio Miyazaki, Mitsugu Sekimoto, Masayoshi Yasui, Yasuhiro Miyake, Shuji Takiguchi, and Morito Monden : ”FAce MOUSe: A Novel Human–Machine Interface for Controlling the Position of a Laparoscope”, _IEEE TRANSACTIONS ON ROBOTICS AND AUTOMATION_ VOL. 19, NO. 5, OCTOBER 2003
  * [17] JUAN P. WACHS, PHD, HELMAN I. STERN, PHD, YAEL EDAN, PHD, MICHAEL GILLAM, MD, JON HANDLER, MD, CRAIG FEIED, MD, PHD, MARK SMITH, MD : ”A Gesture-based Tool for Sterile Browsing of Radiology Images”, _Journal of the American Medical Informatics Association_ Volume 15 Number 3 May / June 2008


11institutetext: INT UMR7289, Aix Marseille Univ, CNRS; 27 Bd Moulin, 13005
Marseille, France
%First␣names␣are␣abbreviated␣in␣the␣running␣head.%If␣there␣are␣more␣than␣two␣authors,␣’et␣al.’␣is␣used.https://laurentperrinet.github.io/publication/perrinet-23-icann/
11email<EMAIL_ADDRESS>
#  Accurate Detection of Spiking Motifs by Learning Heterogeneous Delays of a
Spiking Neural Network ††thanks: Supported by A*MIDEX grant AMX-21-RID-025
“Polychronies”.

Laurent U Perrinet  0000-0002-9536-010X

###### Abstract

Recently, interest has grown in exploring the hypothesis that neural activity
conveys information through precise spiking motifs. To investigate this
phenomenon, various algorithms have been proposed to detect such motifs in
Single Unit Activity (SUA) recorded from populations of neurons. In this
study, we present a novel detection model based on the inversion of a
generative model of raster plot synthesis. Using this generative model, we
derive an optimal detection procedure that takes the form of logistic
regression combined with temporal convolution. A key advantage of this model
is its differentiability, which allows us to formulate a supervised learning
approach using a gradient descent on the binary cross-entropy loss. To assess
the model’s ability to detect spiking motifs in synthetic data, we first
perform numerical evaluations. This analysis highlights the advantages of
using spiking motifs over traditional firing rate based population codes. We
then successfully demonstrate that our learning method can recover
synthetically generated spiking motifs, indicating its potential for further
applications. In the future, we aim to extend this method to real
neurobiological data, where the ground truth is unknown, to explore and detect
spiking motifs in a more natural and biologically relevant context.

###### Keywords:

Neurobiology spike trains population coding spiking motifs heterogeneous
delays pattern detection.

##  1 Introduction

###  1.1 The age of large-scale neurobiological event-based data

Over the past decade, remarkable technological progress across multiple
disciplines has expanded the potential for experimental neuroscience research.
These cutting-edge methods, such as in vivo two-photon imaging, large
population recording arrays, optogenetic circuit control tools, transgenic
manipulations, and large volume circuit reconstructions, allow researchers to
explore neural networks’ function, structure, and dynamics with unparalleled
precision.

The complexity revealed by these advanced technologies underscores the
significance of neurobiological knowledge in bridging the gap between abstract
brain function principles and their biological implementation in neural
circuits. Consequently, there is a growing need to scale up analysis methods
to handle the vast amounts of data generated by these powerful techniques. By
meeting this demand, researchers can gain deeper insights into brain function,
further our understanding of neural circuits, and make groundbreaking
discoveries in neuroscience.

One approach aimed at addressing this challenge is the Rastermap algorithm
[24]. This algorithm rearranges neurons in the raster map based on the
similarity of their activity and utilizes a deconvolution strategy with a
linear model. However, it’s worth noting that the Rastermap algorithm’s
primary testing has been on calcium imaging data, which may introduce some
imprecision in the timing of spiking activity observed in Single Unit Activity
(SUA) recordings. Another significant contribution is from the work of
Williams et al. [32]. They propose a point process model that overcomes
limitations present in existing models, such as the need for discretized spike
times or lack of uncertainty estimates for model predictions and estimated
parameters. By incorporating learnable time-warping parameters to model
sequences of varying durations, the model effectively captures experimentally
observed patterns in neural circuits.

###  1.2 Decoding neural activity using spike distances

Neuroscience research heavily relies on defining appropriate metrics to
compute the distance between spike trains, and one well-known measure for this
purpose is the Victor-Purpura distance [30]. This metric effectively addresses
inconsistencies observed with firing rate-based estimation of spike trains.
Another study refines the Victor-Purpura distance by introducing a time
constant as a parameter, allowing for interpolation between a coincidence
detector and a rate difference counter [27]. Additionally, researchers have
extended these distance measures to non-Euclidean metrics and morphological
manipulations, enabling the computation of spike train dissimilarity.

Regarding spike timings, various methods have been developed to estimate the
latency of neural responses. Bayesian binning [19] is one such method. Unitary
event analysis, based on a statistical model of chance detection, has been
widely used to detect significant synchronous patterns above chance in neuron
pair recordings [11]. Recent extensions of these methods, such as the 3D-SPADE
approach [29], enable the identification of reoccurring patterns in parallel
spike train data and assess their statistical significance. Incorporating
possible temporal dithering in spike timings has been shown to improve
performance, particularly in the presence of patterns with varying durations,
such as surrogates used to evaluate precisely timed higher-order spike
correlations.

However, some of these methods may suffer from computational complexity,
block-based implementations, and narrow specialization for specific tasks. To
address these challenges, novel methods like SpikeShip [28] are being
developed. The complexity and diversity of these spike train distance and
timing comparison methods demonstrate the growing interest in integrating such
measures to understand the neural code. A critical step in testing their
potential usefulness is scaling these methods to handle larger amounts of
data, enabling broader applications and deeper insights into neural activity
patterns and their significance.

###  1.3 A novel hypothesis: spiking motifs

Figure 1:  Core Mechanism of Spiking Motif Detection: In this illustrative
example, we consider a scenario involving three presynaptic neurons denoted as
$a_{1}$, $a_{2}$, and $a_{3}$, which are fully connected to two postsynaptic
neurons $b_{1}$ and $b_{2}$. The synaptic delays for the connections to
$b_{1}$ are $1$, $5$, and $9~{}$\mathrm{ms}$$, while for $b_{2}$ they are $8$,
$5$, and $1~{}$\mathrm{ms}$$, respectively. In the middle panel, when the
three presynaptic neurons emit synchronous pulses, the postsynaptic potentials
generated in $b_{1}$ and $b_{2}$ reach them asynchronously due to the
heterogeneous delays. Consequently, the postsynaptic potentials may not be
sufficient to reach the membrane threshold (dashed line) in either of the
postsynaptic neurons, and no output spike is generated. In the right panel,
the pulses emitted by the presynaptic neurons are arranged in such a way that,
taking into account the delays, they reach the postsynaptic neuron $b_{1}$ at
the same time (at $t=10~{}$\mathrm{ms}$$ in this example). As a result, the
postsynaptic potentials $V_{t}$ evoked by the three presynaptic neurons sum
up, causing the voltage threshold to be crossed. This leads to the emission of
an output spike, signaling the detection of a spiking motif in the presynaptic
population (highlighted in red color). This core mechanism illustrates how the
interplay between heterogeneous delays in the network allows for precise spike
timing, enabling the detection of spiking motifs in neural populations.

In recent studies, the importance of spike timing has been emphasized,
especially in the barn owl auditory system, where precise spike timing in
response to the sound of a mouse allows the brain to determine the prey’s
position [7]. This discovery aligns with a growing body of literature
suggesting that the brain’s dynamics often exhibit stereotyped sequences known
as _spiking motifs_ [9]. The concept of spiking motifs is a generalization of
the patterns observed in the polychronization model developed by Izhikevich
[16]. This theoretical model comprises a random recurrent network of spiking
neurons with biologically realistic synaptic delays and evolving weights
governed by Spike-Time Dependent Plasticity (STDP) learning rule.

The interplay between the synaptic delays and STDP leads to the spontaneous
organization of neurons into groups called "polychronous groups." Despite
neurons in one of these groups firing at different times, the heterogeneous
delays enable their spikes to converge synchronously on the postsynaptic
neuron. This convergence results in the summation of excitatory postsynaptic
potentials, leading to the firing of the postsynaptic neuron (see Figure 1).
The polychronization model allows spiking neurons to self-organize into groups
and generate reproducible time-locked spiking motifs. The STDP rule increases
synaptic weights selectively for neurons involved in these polychronous
groups, thereby consolidating the formation of such groups.

While the polychronization model provides valuable insights into understanding
spiking neural networks and their potential role in generating spatio-temporal
spiking motifs, it has a limitation. The model’s heterogeneous delays are
fixed and cannot evolve over time, which may limit its applicability in
certain scenarios. However, the underlying mechanism offers valuable
implications for studying neural activity motifs and their significance in the
brain. To effectively detect spiking motifs, we propose a novel metric
inspired by this model.

###  1.4 The Heterogeneous Delays Spiking Neural Network (HD-SNN)

In this work, we propose to accurately detect spatio-temporal spiking motifs
using a feed-forward, single layer heterogeneous delays spiking neural network
(HD-SNN). The paper is organized as follows. We develop a theoretically
defined HD-SNN for which we can attune both the weights and delays. We first
detail the methodology by defining the basic mechanism of spiking neurons that
utilize heterogeneous delays. This will allow us to formalize the spiking
neuron used to learn the model’s parameters in a supervised manner and test
its effectiveness. In the results section, we will first evaluate the
efficiency of the learning scheme. We will also study the robustness of the
spiking motif detection mechanism and in particular its resilience to changing
the dimensions of the presynaptic or postsynaptic populations, or the depth in
the number of different possible delays. Then, we will explore how the spiking
motifs may be learned using supervised learning, and evaluate how the
efficiency of the algorithm may depend on the parameters of the HD-SNN
architecture. This will allow us to show how such a model can provide an
efficient solution which may in the future be applied to neurobiological data.
Finally, we will conclude by highlighting the main contributions of this
paper, while defining some limitations which will open perspectives for future
detection methods.

##  2 Methods

Let us formally define the HD-SNN model. First, we will define raster plots
similar to those obtained from Single Unit Activity (SUA) recordings using an
event-based and then binarized setting. We will then derive a generative model
for raster plots using a HD-SNN, and derive a model for efficient detection of
event-based motifs using a similar HD-SNN with “inverted” delays.

###  2.1 Raster plots: from event-based to binarized

In neurobiological recordings, any generic raster plot consists of a stream of
_spikes_. This can be formalized as a list of neural addresses and timestamps
tuples $\epsilon=\\{(a_{r},t_{r})\\}_{r\in[1,N_{ev}]}$ where
$N_{ev}\in\mathbb{N}$ is the total number of events in the data stream and the
rank $r$ is the index of each event in the list of events. Each event has a
time of occurrence $t_{r}$ (these are typically ordered) and an associated
address $a_{r}$ in the space $\mathcal{A}$ of the neural population. In a
neurobiological recording like that of SUAs, this can be the identified set of
neurons.

Events are generated by neurons which are defined on the one hand by the
equations governing the evolution of its membrane potential dynamics on their
soma and on the other hand by the integration of the synaptic potential
propagating on their dendritic tree. A classical characterization consists in
detailing the synaptic weights of each synaptic contact, the so-called weight
matrix. As we saw above, neurons can receive inputs from multiple presynaptic
neurons with heterogeneous delays. These delays represent the time it takes
for a presynaptic spike to reach the soma of the postsynaptic neuron. In such
neurons, input presynaptic spikes $\epsilon$ will be multiplexed in time by
the dendrites defined by this synaptic set (see Figure 1).

Let’s formalize such a layer of spiking neurons in the HD-SNN model. Each
postsynaptic neuron $b\in\mathcal{B}$ connects to presynaptic neurons from a
set of addresses in $\mathcal{A}$. In biology, a single cortical neuron has
generally several thousands of synapses. Each may be defined by its synaptic
weight and also its delay. Note that two neurons may contact with multiple
synapses, and thus different delays. Scanning all neurons $b$, we thus define
the set of $N_{s}\in\mathbb{N}$ synapses as
$\mathcal{S}=\\{(a_{s},b_{s},w_{s},\delta_{s})\\}_{s\in[1,N_{s}]}$, where each
synapse is associated to a presynaptic address $a_{s}$, a postsynaptic address
$b_{s}$, a weight $w_{s}$, and a delay $\delta_{s}$.

This defines the full connectivity of the HD-SNN model. The receptive field of
a postsynaptic neuron refers to the set of synapses that connect to it.
Similarly, the emitting field of a presynaptic neuron refers to the set of
synapses it connects to. These fields determine the synaptic inputs and
outputs of individual neurons. More formally, the receptive field of a
postsynaptic neuron is defined
$\mathcal{S}^{b}=\\{(a_{s},b_{s},w_{s},\delta_{s})\|b_{s}=b\\}_{s\in[1,N_{s}]}$,
and the emitting field of a presynaptic neuron as
$\mathcal{S}_{a}=\\{(a_{s},b_{s},w_{s},\delta_{s})\|a_{s}=a\\}_{s\in[1,N_{s}]}$.
Following this definition, an event stream which evokes neurons in the
presynaptic address space is multiplexed by the synapses into a new event
stream which is defined by the union of the sets generated by each emitting
field from the presynaptic space:
$\cup_{r\in[1,N_{ev}]}\\{(b_{s},w_{s},t_{r}+\delta_{s})\\}_{s\in\mathcal{S}_{a_{r}}}$.
In biology, this new stream of events is naturally ordered in time as events
reach the soma of post-synaptic neurons. Synchronous activation of
postsynaptic neurons, where multiple spikes converge on the soma
simultaneously, will increase the firing probability of those neurons.

From the perspective of simulating such event-based computations on standard
CPU- or GPU-based computers, it is useful to transform this event-based
representation into a dense representation. Indeed, we may transform any
event-based input as the boolean matrix $A\in\\{0,1\\}^{N\times T}$, where $N$
is the number of presynaptic neurons in $\mathcal{A}$ and $T$ is the number of
time bins (see Figure 2a). In this simplified model, we will consider that
heterogeneous delays are integers limited in range between $0$ and $D$ (that
is, $\forall{s\in[1,N_{s}]}$, $0\leq\delta_{s}<D$) such that the synaptic set
can be represented by the dense matrix $K^{b}\in\mathbb{R}^{N\times D}$ giving
for each neuron $b$ the weights as a function of presynaptic address and delay
(see Figure 2b). It is equal to zero except on synapses:
$\forall{s\in\mathcal{S}^{b}},K^{b}(a_{s},\delta_{s})=w_{s}$. Equivalently,
one may define for each presynaptic neuron $a$ the emitting kernel as the
transpose kernel $K^{T}_{a}\in\mathbb{R}^{M\times D}$, where $M$ is the number
of postsynaptic neurons, whose values are zero except on synapses:
$\forall{s\in\mathcal{S}_{a}},K^{T}_{a}(b_{s},\delta_{s})=w_{s}$.

Figure 2: From generating raster plots to inferring spiking motifs. (a) As an
illustration for the generative model, we draw a multiunit raster plot
synthesized from $4$ different spiking motifs and for $10$ presynaptic
neurons. (b) We show these motifs, each identified at the top by a different
color. The evidence of activation (red) or deactivation (blue) is assigned to
each presynaptic neuron and $31$ different possible delays. (c) The activation
in time of the different motifs (denoted by stars) is drawn at random and then
used to generate a raster plot on the multi-unit address space (see panel a).
By inverting this model, an inference model can be defined for their efficient
detection, outputting an evidence value (continuous line) from which the
identity and timing of SMs can be inferred (vertical bars). (d) The original
raster plot can be annotated with each identified spiking motif (as
represented by the respective color assigned to SMs).

###  2.2 A generative model for raster plots

As described in Figure 1, a spiking motif can be detected using a properly
tuned HD-SNN that maximizes spike synchronization at the postsynaptic
terminal. Taking the argument the other way around, one may form a generative
model for realistic raster plots in which spikes in the presynaptic address
space are generated as the conjunction of spiking motifs defined in the
postsynaptic space, knowing that both populations are connected by a set of
weights and delays whose structure is stable relatively to the coding
timescale. When connection weights are strong and sparsely distributed, this
firing will robustly cause a specific temporal motif. Overall, these examples
show that raster plots may be considered as a mixture of the effects of
different elementary causes, and that each event triggers a specific spatio-
temporal spiking motif.

Formally, the activation of spiking motifs can occur independently and at
random times. The activity is represented as a boolean matrix
$B\in\\{0,1\\}^{M\times T}$, where $M$ is the number of different spiking
motifs (see Figure 2c). Each entry $B(b,t)$ indicates whether a particular
motif $b$ is activated at time $t$. The firing of a neuron $a$ at time $t$ is
considered a Bernoulli trial with a bias parameter $p(a,t)\in[0,1]$. This bias
is conditioned by the presence of spiking motifs on postsynaptic neurons with
corresponding delays. Assuming that this bias is conditioned by the presence
of spiking motifs on _all_ efferent postsynaptic neurons with the
corresponding delays, it can be shown that the logit (inverse of the sigmoid)
of this probability bias can be written as the sum of the logit of each of
these factors, whose values we will define as the corresponding weights in the
kernel. We can thus write the probability bias $p(a,t)$ as the accumulated
evidence given these factors as

$p(a,t)=\sigma\big{(}K_{\mathcal{A}}(a)+\sum_{b\in\mathcal{S}_{a},0\leq\delta\leq
D}B(b,t+\delta)\cdot K_{a}(b,\delta)\big{)}$

where $\sigma$ is the sigmoid function. We will further assume that kernel’s
weights are balanced (their mean is zero) and that $K_{\mathcal{A}}$ is a bias
such that $\forall a,t$, $\sigma(K_{\mathcal{A}}(a))$ is the average
background firing rate.

Finally, we obtain the raster plot $A\in\\{0,1\\}^{N\times T}$ by drawing
spikes using independent Bernoulli trials based on the computed probability
biases $A\sim\mathcal{B}(p)$. Note that, depending on the definition of
kernels, the generative model can model a discretized Poisson process,
generate rhythmic activity or more generally propagating waves. This
formulation thus defines a simple generative model for raster plots as a
combination of independent spiking motifs. This generative model can be easily
extented to include a refractory period in order to ensure that there is a
minimum time gap between successive action potentials, preventing them from
overlapping. This temporal separation allows for discrete and well-defined
neural signals, enabling accurate information processing and mitigating signal
interference. The refractory period contributes to energy efficiency in neural
systems and plays a crucial role in temporal coding by creating distinct time
windows between successive spikes.

###  2.3 Detecting spiking motifs

Assuming the spiking motifs (as defined by the kernel $K$) are known, the
generative model allows to determine an inference model for detecting sources
$\hat{B}$ when observing a raster plot $A$. Indeed, by using this forward
model, it is possible to estimate the likelihood $p(b,t)$ for the presence of
a spiking motif of address $b$ and at time $t$ by using the transpose
convolution operator. This consists in using the emitting field
$\mathcal{S}_{a}$ of presynaptic neurons in place of the receptive field
$\mathcal{S}^{b}$ of postsynaptic neurons. It thus comes that when observing
$A$, then one may infer the logit of the probability as the sum of evidences:

$p(b,t)=\sigma\big{(}K_{\mathcal{B}}(b)+\sum_{a\in\mathcal{S}^{b},0\leq\delta\leq
D}A(a,t-\delta)\cdot K^{b}(a,\delta)\big{)}$

This also takes the form of a temporal convolution. This assumption holds as
long as the kernels are uncorrelated, a condition which is met here
numerically by choosing a relatively sparse set of synapses (approximately
$1\%$ of active synapses). Finally, we compute $\hat{B}$ by selecting the most
likely items, allowing to identify the spiking motifs in the input raster plot
(see Figure 2d).

One may naturally extend this algorithm when the spiking motifs (that is, the
weights) are not known, but that we know the timing and identity of the
spiking motifs. Indeed, the equation above is differentiable. Indeed, the
activation function of our spiking neural is a sigmoid function implementing a
form of Multinomial Logistic Regression (MLR) [10]. The underlying metric is
the binary cross-entropy, as used in the logistic regression model. In
particular, if we consider kernels with similar decreasing exponential time
profile, one can prove that this detection model is similar to the method of
Berens et al. [2]. In our specific case, the difference is that the regression
is performed in both dendritic and delay space by extending the summation
using a temporal convolution operator.

##  3 Results

To quantify the efficiency of this operation, we generated raster plots
parameterized by $N=128$ presynaptic inputs and $M=144$ synthetic spiking
motifs as random independent kernels and with $D=31$ possible delays. We drew
random independent instances of $B$ with a length of $T=1000$ time steps and
an average of $1.0$ spikes per neuron. This allowed us to generate a large
number of synthetic raster plots, which we use to infer $\hat{B}$. We compute
accuracy as the rate of true positive detections (both for inferring the
address and its exact timing) and observe on average $\approx 98.8\%$ correct
detections.

We extended this result by showing how accuracy evolves as a function of the
number of simultaneous spiking motifs, holding the frequency of occurrence
constant. We show in Fig. 3 (left) that the accuracy of finding the right
spiking motif is still above $80\%$ accuracy with more than $1364$ overlapping
spiking motifs. This observation illustrates quantitatively the capacity of
the HD-SNN in representing a high number of simultaneous motifs. Furthermore,
we show in Fig. 3 (middle) that (with $M=144$ spiking motifs fixed) the
accuracy increases significantly with increasing temporal depth $D$ of the
spiking motif kernel, quantitatively demonstrating the computational advantage
of using heterogeneous delays. These results were obtained under the
assumption that we know the spiking motifs through $K$. However, this is
generally not the case, for example, when considering the raster plot of
biological neurons.

Finally, we evaluated the performance of the supervised learning scheme in
inferring the connection kernel when the address and timing of spiking motifs
are known. The kernel was initialized with random independent values, and we
used stochastic gradient descent with a learning rate of $110-4$ over $1104$
trials (i.e., over rasters as defined above with $T=1000$ and $N=128$).
Qualitatively, the convergence was monotonous, and the correct values of the
$M=144$ spiking motifs were quickly recovered. Quantitatively, the correlation
between the true and learned kernel weights showed that all kernels were
correctly recovered (see Figure 3, right). Performing inference with the
learned weights was as efficient as with the true kernels, and showed no
significant difference (not shown).

Figure 3: Detecting spiking motifs using spiking neurons with heterogeneous
delays. Accuracy of detection for the classical correlation (red) and the HD-
SNN method (blue) as a function of (Left) the number $M$ of kernels, (Middle)
the number of presynaptic neurons, (Right) Correlation matrix of true vs
learned kernels.

##  4 Discussion

###  4.1 Synthesis and Main Contributions

In this paper, we present a novel Heterogeneous Delays Spiking Neural Network
(HD-SNN) model designed for the detection of spiking motifs in synthetic
neurobiologically-inspired raster plots.

Our contributions encompass several innovations. Firstly, we formulate the HD-
SNN model from first principles, optimizing the detection of event-based
spatiotemporal motifs. Unlike previous models like the tempotron, which are
evaluated on simplified problems, our model is rigorously tested on realistic
synthetic data. The results demonstrate that, assuming that the spiking motifs
are known, our model accurately detects the identity and timing of spiking
motifs, even when multiple motifs are superimposed. Additionally, we show that
our method outperforms correlation-based heuristics, such as those used in
previous works like [6, 33], in terms of efficiency. Secondly, compared to
other event-based methods, like HOTS [18], our model’s weights are
interpretable. These weights are directly related to the logit, which is the
inverse sigmoid of the probability of detecting each spatiotemporal spiking
motif. Finally, a crucial novelty lies in the simultaneous learning of weights
and delays in our model. In contrast, models like the polychronization model
[16] only learn weights and delays are frozen. These contributions highlight
the significance and effectiveness of our HD-SNN model for detecting spiking
motifs, offering insights into the neural mechanisms involved in pattern
recognition and information processing.

###  4.2 Main limits

The model comes with certain limitations. First, the entire framework is based
on discrete time binning, which is incompatible with the continuous nature of
biological time. While this choice facilitated efficient implementation on
conventional hardware such as GPUs, it can be extended to a purely event-based
SNN framework [8]. By analytically incorporating a precision term in the
temporal value of the input spikes, a purely event-based scheme can be
achieved, promising speedups and computational energy gains.

Second, the current model is purely feed-forward, i.e. the spikes generated by
postsynaptic neurons are based solely on information from their classical
receptive fields. However, neural systems often involve lateral interactions
between neurons in the same layer and feedback connections, which can be
crucial for computational principles and modulation of neural information.
While our theoretical model can incorporate these recurrent connections by
inserting new spikes into the list of spikes reaching presynaptic addresses,
it requires proper tuning to avoid perturbations of the homeostatic state. For
the implementation of predictive or anticipatory processes, recurrent activity
would be essential, especially when dealing with multiple different delays
that require temporal alignment. Such recurrent activity has previously been
modelled to explain phenomena such as the flash-lag illusion. Implementing
this using generalised coordinate and delay operators would allow predictive
mechanisms to be incorporated into our proposed HD-SNN model, providing an
elegant solution to this problem.

Addressing these limitations and exploring the extension of the HD-SNN model
to event-based schemes and recurrent connections would enrich its potential
applications and pave the way for a better understanding of neural information
processing in complex systems.

###  4.3 Perspectives

The coding results were obtained under the assumption that we know the spiking
motifs by way of $K$, or using supervised learning by knowing the identity and
timing of spiking motifs. However, this is generally not the case, e.g. when
observing the neurobiological raster plot of a population of neurons. One
perspective would be to extend the model to a fully self-supervised learning
paradigm, i.e. without any labeled data [1]. This type of learning is thought
to be prevalent in the central nervous system and, assuming the signal is
sparse [23], one could extend these Hebbian sparse learning schemes to spikes
[25, 22].

We expect that this would be particularly adapted for exploring
neurobiological data [21]. Indeed, there is a large literature showing that
brain dynamics often organize into stereotyped sequences such as synfire
chains [15], packets [20], or hippocampal sequences [31] (for a review, see
[9]). These motifs are stereotyped and robust, as they can be activated in the
same motif from day to day [13]. In contrast to conventional methods used to
process neurobiological data, such an event-based model would be able to
answer key questions regarding the representation of information in
neurobiological data.

## References

  * [1] Barlow, H.: Unsupervised Learning. Neural Computation 1(3), 295–311 (Sep 1989).
  * [2] Berens, P., Ecker, A.S., Cotton, R.J., Ma, W.J., Bethge, M., Tolias, A.S.: A Fast and Simple Population Code for Orientation in Primate V1. Journal of Neuroscience 32(31), 10618–10626 (2012).
  * [3] Boutin, V., Franciosini, A., Chavane, F.Y., Ruffier, F., Perrinet, L.U.: Sparse Deep Predictive Coding captures contour integration capabilities of the early visual system. PLoS Computational Biology (May 2020).
  * [4] Boutin, V., Franciosini, A., Ruffier, F., Perrinet, L.U.: Effect of top-down connections in Hierarchical Sparse Coding. Neural Computation 32(11), 2279–2309 (Feb 2020).
  * [5] Chavane, F., Perrinet, L.U., Rankin, J.: Revisiting horizontal connectivity rules in V1: from like-to-like towards like-to-all. Brain Structure and Function (Feb 2022).
  * [6] Ghosh, R., Gupta, A., Silva, A.N., Soares, A., Thakor, N.V.: Spatiotemporal filtering for event-based action recognition (2019).
  * [7] Goodman, D.F.M., Brette, R.: Spike-timing-based computation in sound localization. PLoS Comput Biol 6(11) (Nov 2010).
  * [8] Grimaldi, A., Boutin, V., Ieng, S.H., Benosman, R., Perrinet, L.U.: A robust event-driven approach to always-on object recognition. Neural Networks (2023).
  * [9] Grimaldi, A., Gruel, A., Besnainou, C., Jérémie, J.N., Martinet, J., Perrinet, L.U.: Precise Spiking Motifs in Neurobiological and Neuromorphic Data. Brain Sciences 13(1), 68 (Jan 2023).
  * [10] Grimaldi, A., Perrinet, L.U.: Learning heterogeneous delays in a layer of spiking neurons for fast motion detection. Biological Cybernetics (Jun 2023).
  * [11] Grün, S., Diesmann, M., Aertsen, A.: Unitary Events in Multiple Single-Neuron Spiking Activity: II. Nonstationary Data. Neural Computation 14(1), 81–119 (Jan 2002).
  * [12] Gütig, R., Sompolinsky, H.: The tempotron: a neuron that learns spike timing–based decisions. Nature Neuroscience 9(3), 420–428 (2006).
  * [13] Haimerl, C., Angulo-Garcia, D., Villette, V., Reichinnek, S., Torcini, A., Cossart, R., Malvache, A.: Internal representation of hippocampal neuronal population spans a time-distance continuum. Proceedings of the National Academy of Sciences 116(15), 7477–7482 (Apr 2019).
  * [14] Hogendoorn, H., Burkitt, A.N.: Predictive Coding with Neural Transmission Delays: A Real-Time Temporal Alignment Hypothesis. eneuro 6(2), ENEURO.0412–18.2019 (Mar 2019).
  * [15] Ikegaya, Y., Aaron, G., Cossart, R., Aronov, D., Lampl, I., Ferster, D., Yuste, R.: Synfire Chains and Cortical Songs: Temporal Modules of Cortical Activity. Science 304(5670), 559–564 (Apr 2004).
  * [16] Izhikevich, E.M.: Polychronization: Computation with Spikes. Neural Computation 18(2), 245–282 (Feb 2006).
  * [17] Khoei, M.A., Masson, G.S., Perrinet, L.U.: The Flash-Lag Effect as a Motion-Based Predictive Shift. PLOS Computational Biology 13(1), e1005068 (Jan 2017).
  * [18] Lagorce, X., Orchard, G., Galluppi, F., Shi, B.E., Benosman, R.B.: HOTS: A Hierarchy of Event-Based Time-Surfaces for Pattern Recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence 39(7), 1346–1359 (2017).
  * [19] Levakova, M., Tamborrino, M., Ditlevsen, S., Lansky, P.: A review of the methods for neuronal response latency estimation. Biosystems 136, 23–34 (Oct 2015).
  * [20] Luczak, A., Barthó, P., Marguet, S.L., Buzsáki, G., Harris, K.D.: Sequential structure of neocortical spontaneous activity in vivo. Proceedings of the National Academy of Sciences 104(1), 347–352 (Jan 2007).
  * [21] Mackevicius, E.L., Bahle, A.H., Williams, A.H., Gu, S., Denisenko, N.I., Goldman, M.S., Fee, M.S.: Unsupervised discovery of temporal sequences in high-dimensional datasets, with applications to neuroscience. eLife 8, e38471 (Feb 2019).
  * [22] Masquelier, T., Guyonneau, R., Thorpe, S.J.: Competitive STDP-Based Spike Pattern Learning. Neural Computation 21(5), 1259–1276 (May 2009).
  * [23] Olshausen, B.A., Field, D.J.: Emergence of simple-cell receptive field properties by learning a sparse code for natural images. Nature 381(6583), 607–609 (1996).
  * [24] Pachitariu, M., Stringer, C., Harris, K.D.: Robustness of Spike Deconvolution for Neuronal Calcium Imaging. The Journal of Neuroscience 38(37), 7976–7985 (2018).
  * [25] Perrinet, L.: Emergence of filters from natural scenes in a sparse spike coding scheme. Neurocomputing 58-60(C), 821–826 (2004).
  * [26] Perrinet, L.U., Adams, R.A., Friston, K.J.: Active inference, eye movements and oculomotor delays. Biological Cybernetics 108(6), 777–801 (Dec 2014).
  * [27] van Rossum, M.: A novel spike distance. Neural Comp 13(4), 751–763 (2001).
  * [28] Sotomayor-Gómez, B., Battaglia, F.P., Vinck, M.: SpikeShip: A method for fast, unsupervised discovery of high-dimensional neural spiking patterns. bioRxiv : the preprint server for biology pp. 2020–06 (2021).
  * [29] Stella, A., Quaglio, P., Torre, E., Grün, S.: 3d-SPADE: Significance evaluation of spatio-temporal patterns of various temporal extents. Biosystems 185, 104022 (Nov 2019).
  * [30] Victor, J.D., Purpura, K.P.: Nature and precision of temporal coding in visual cortex: a metric-space analysis. Journal of Neurophysiology 76(2), 1310–1326 (1996).
  * [31] Villette, V., Malvache, A., Tressard, T., Dupuy, N., Cossart, R.: Internally Recurring Hippocampal Sequences as a Population Template of Spatiotemporal Information. Neuron 88(2), 357–366 (Oct 2015).
  * [32] Williams, A.H., Degleris, A., Wang, Y., Linderman, S.W.: Point process models for sequence detection in high-dimensional neural spike trains. Tech. Rep. 2010.04875, arXiv (Oct 2020).
  * [33] Yu, C., Gu, Z., Li, D., Wang, G., Wang, A., Li, E.: STSC-SNN: Spatio-Temporal Synaptic Connection with Temporal Convolution and Attention for Spiking Neural Networks (Oct 2022), arXiv:2210.05241


11institutetext:  MARS Motor AI Recognition Solution, Bangkok, Thailand
11email<EMAIL_ADDRESS>
22institutetext: OZT Robotics, Bangkok, Thailand
22email<EMAIL_ADDRESS>
# MARS: Mask Attention Refinement with Sequential Quadtree Nodes for Car
Damage Instance Segmentation

Teerapong Panboonyuen Corresponding author:
<EMAIL_ADDRESS>Naphat Nithisopa 11 Panin Pienroj 22
Laphonchai Jirachuphun 22 Chaiwasut Watthanasirikrit 11 Naruepon Pornwiriyakul
11

###### Abstract

Evaluating car damages from misfortune is critical to the car insurance
industry. However, the accuracy is still insufficient for real-world
applications since the deep learning network is not designed for car damage
images as inputs, and its segmented masks are still very coarse. This paper
presents MARS (Mask Attention Refinement with Sequential quadtree nodes) for
car damage instance segmentation. Our MARS represents self-attention
mechanisms to draw global dependencies between the sequential quadtree nodes
layer and quadtree transformer to recalibrate channel weights and predict
highly accurate instance masks. Our extensive experiments demonstrate that
MARS outperforms state-of-the-art (SOTA) instance segmentation methods on
three popular benchmarks such as Mask R-CNN [9], PointRend [14], and Mask
Transfiner [13], by a large margin of +1.3 maskAP-based R50-FPN backbone and
+2.3 maskAP-based R101-FPN backbone on Thai car-damage dataset. Our demos are
available at https://www.marssolution.io.

###### Keywords:

Quadtree Transformer Mask Transfiner PointRend Instance Segmentation.

##  1 Introduction

Evaluation of car damages in Thailand (see Figure 1) from an accident is a
critical aspect of the car insurance business. When a car is involved in an
accident, it can result in significant damage to the vehicle, ranging from
minor dents and scratches to more severe structural damage. Evaluating the
extent of the damage is essential for the insurance company to determine the
cost of repairs or replacement. The evaluation process is typically carried
out by a trained professional, such as a claims adjuster, who assesses the
damage and determines the cost of repairs or replacement. This process helps
the insurance company determine the payout amount to the policyholder. The
accuracy of the damage assessment is crucial in ensuring a fair payout to the
policyholder and avoiding fraudulent claims. The timely and accurate
evaluation of car damages is critical to the car insurance business’s smooth
operation and helps to maintain the trust of policyholders in the insurance
industry [12, 27, 23, 11, 17].

Figure 1: Instance segmentation with MARS. Our module can be easily
incorporated into existing semantic and instance segmentation systems. This
paper demonstrates that using MARS instead of the default Sequential Quadtree
Nodes in Mask Transfiner [13] leads to much more detailed results.

Recent instance segmentation methods [26, 22, 3, 16, 2, 9, 5, 15, 25, 4] have
been based on object detection pipelines, where objects [14, 13] are first
localized with bounding boxes and then refined into a segmentation. However,
these methods may suffer from false-positive detections and poorly localized
bounding boxes that do not cover the entire object. They also do not consider
the entire image but independent proposals. In contrast, this paper proposes a
new framework, MARS, that models instance masks and considers the image as a
whole when making predictions, producing more accurate segmentations. Figure 2
clearly demonstrates that there is still a considerable disparity between the
performance of the latest state-of-the-art methods in bounding box and
segmentation, particularly for the most recent query-based approaches. This
suggests that while detection capability has progressed significantly, the
quality of masks has not advanced at the same rate.

Figure 2: The performance gap between object detection and segmentation for
instance segmentation models on Thai car-damage set using R50-FPN as backbone.
Detailed comparisons are in Table 2.

Our proposed method is inspired by [13], decomposes and represents the image
regions as a quadtree. The main concept of this research is to overcome false-
positive detections and imperfectly localized bounding boxes that do not cover
the entire object, which is a limitation of detection-based methodsdf. The
proposed method takes into account the entire image while making predictions
and aims to address occlusions between different objects. The segmentation
maps produced by the proposed method, as shown in Figure 4, do not require
post-processing. We start our end-to-end trained network with a semantic
segmentation module, which generates a variable number of instances for each
input image.

The main contributions of this work are two-fold:

  * •

We introduce MARS, a new framework for modeling instance masks that enables
our method to generate more precise segmentations. MARS is designed to be
straightforward and efficient.

  * •

We propose the use of self-attention with sequential quadtree nodes in our
framework. We modify the relative positional encoding to encode information
about spatial distances and relationships between sequential quadtree and
quadtree transformer module, which helps capture critical local dependencies
and correlations.

Our method, MARS, was extensively evaluated on the Thai car-damage corpus
benchmark using both quantitative and qualitative analysis. Our results
demonstrate that MARS is highly effective in producing accurate segmentations
of Thai car-damage imagery, surpassing state-of-the-art methods in high APs
family thresholds on the dataset. In addition, our proposed method improves
the semantic segmentation task while being trained for the instance
segmentation task. Qualitatively, MARS generates sharp object boundaries as
shown in Figure 3 and Figure 5. Despite the standard intersection-over-union
based metrics being biased towards object-interior pixels and being
insensitive to boundary improvements, we observed quantitative improvements
with MARS. Our method achieved a significant improvement over the strong Mask
R-CNN model. Qualitatively, MARS efficiently computes sharp boundaries between
objects, as illustrated in Figure 3.

Figure 3: Instance Segmentation on Thai car-damage validation set by a) Mask
R-CNN [9], b) PointRend [14], c) Mask Transfiner [13], d) MARS (Ours) using
R50-FPN as backbone, where MARS produces significantly more detailed results
at high-frequency image regions by replacing Mask R-CNN’s default mask head
(Zoom in for better view).

##  2 Related Works

Several efforts have been made in the literature to automatically analyze car
damage. The field of instance segmentation, which is a complex and challenging
area in machine vision research, has gained significant attention. Its
objective is to predict the object class label and the pixel-specific object
instance mask, and it locates different types of object instances in various
images. A number of studies such as [20, 10, 7, 6] have been carried out in
this area. This study employs instance segmentation to identify and segment
areas of automobile damage in traffic accidents.

Instance Segmentation. Faster R-CNN [8] is extended by Mask R-CNN [9] with the
addition of a branch for predicting an object mask in parallel with the
existing component for bounding box recognition. Mask R-CNN is simple to train
and adds only a small overhead to Faster R-CNN.

PointRend (Point-based Rendering) [14] offers a module that performs point-
based segmentation predictions at adaptively selected locations based on an
iterative subdivision algorithm. It can be applied flexibly to both instance
and semantic segmentation tasks by building on top of existing state-of-the-
art models. PointRend is not restricted to instance segmentation and can be
expanded to other pixel-level recognition tasks.

Mask Transfiner [13] identifies and decomposes image regions to construct a
hierarchical quadtree. They use a multi-scale deep feature pyramid, take
sequential input, and execute powerful local and non-local reasoning through
the multi-head attention layers. It receives as input sequence the sparsely
detected feature points in incoherent image regions across the RoI feature
pyramid levels and generates the corresponding segmentation labels. First, it
detects and decomposes image regions to build a hierarchical quadtree. Then,
all points on the quadtree are transformed into a query sequence for the
transformer to forecast the final labels. Unlike previous segmentation methods
using convolutions constrained by uniform image grids, it produces high-
quality masks with low computation and memory costs.

Car Damage Task. The task of analyzing car damage has been approached by
various researchers. Zhang et al. [26] propose an improved Mask R-CNN with
ResNet and FPN for feature extraction. Parhizkar et al. [18] introduce a CNN-
based approach for accurate vehicle detection and damage localization on the
exterior surface of the car. Pasupa et al. [19] develop an automatic car part
identification system based on Mask R-CNN and GCNet, for both object detection
and semantic segmentation tasks in average weather conditions. Finally,
Amirfakhrian et al. [1] present a particle swarm optimization (PSO) algorithm
to identify the damaged parts of cars involved in accidents in Iran.

Although previous works have employed mask instance segmentation, there are
limitations in its accuracy as some areas with damage may not be visible or
segmented correctly. Unlike previous works that use Mask R-CNN for feature
representation, our approach uses it to generate masks for instance
segmentation. Additionally, our approach involves learning Mask R-CNN specific
to each image instead of using global prototypes shared across the car damage
dataset.

##  3 Proposed Method

We next describe Mask Attention Refinement with Sequential Quadtree Nodes
(MARS) for instance segmentation. MARS has three components and illustrated in
Figure 4: (i) node encoder, (ii) sequence encoder and (iii) pixel decoder like
Mask Transfiner [13]. The node encoder first enriches the feature embedding
for each incoherent point. The sequence encoder then takes these encoded
feature vectors across multiple quadtree levels as input queries. Finally, the
pixel decoder predicts their corresponding mask labels. Also, we employ a
multi-scale deep feature pyramid. The object detection head then predicts
bounding boxes as instance proposals. This component also generates a coarse
initial mask prediction at low resolution. Given this input data, our aim is
to predict highly accurate instance segmentation masks.

Figure 4: The framework of MARS. Our end-to-end trained network consists of
semantic and instance segmentation modules.

Our MARS operates in the detected incoherent regions. Since it operates on
feature points on the constructed quadtree, and not in a uniform grid, we
design a transformer architecture which jointly processes all incoherent nodes
in all levels of the quadtree. Finally, we present the training strategy of
MARS along with the implementation details.

MARS Architecture. Figure 4 shows the overall architecture of our MARS. Based
on the Mask Transfiner [13], instance segmentation is tackled in a multi-level
and coarse-to-fine manner. Instead of using single-level FPN feature for each
object, MARS takes as input sequence the sparsely detected feature points in
incoherent image regions across the RoI feature pyramid levels, and outputs
the corresponding segmentation labels. We interpret the projections of
queries, keys, and values in self-attention module as multiple 1×1
convolutions, followed by the computation of attention weights and aggregation
of the values before take into sequential quadtree nodes. Then, the sequence
transformer encoder of Transfiner jointly processes the en- coded nodes from
all levels in the quadtree. The transformer thus performs both global spatial
and inter-scale reasoning. Each sequence encoder layer has a standard
transformer structure, formed by a multi-head self-attention module and a
fully connected feed forward network (FFN). To equip the incoherent points
sequence with adequate positive and negative references, we also use all
feature points from the coarsest FPN level with small size 7x7. Different from
the standard transformer decoder [4] with deep attention layers, the pixel
decoder in MARS is a small two- layer MLP, which decodes the output query for
each node in the tree, in order to predict the final mask labels.

Training and inference. Based on the constructed quadtree, we develop flexible
and adaptive training and inference schemes for MARS, where all detected
incoherent nodes across quadtree levels are formed into a sequence for
parallel prediction. During inference, to obtain final object masks, MARS
follows the quadtree propagation scheme after obtaining the refined labels for
incoherent nodes. During training, the whole Mask Trans- finer framework can
be trained in an end-to-end manner. We employ a multi-task loss

${\mathcal{L}}=\lambda_{1}{\mathcal{L}}_{Detect}+\lambda_{2}\mathcal{L}_{Coarse}+\lambda_{3}\mathcal{L}_{Refine}+\lambda_{4}\mathcal{L}_{Inc}$
(1)

Here, $\mathcal{L}_{Refine}$ denotes the refinement with L1 loss between the
predicted labels for incoherent nodes and their ground- truth labels. A Binary
Cross Entropy loss $\mathcal{L}_{Inc}$ is for detecting incoherent regions.
The detection loss $\mathcal{L}_{Detect}$ includes the localization and
classification losses from the base detector, e.g. Faster R-CNN or DETR
detector. Finally, $\mathcal{L}_{Coarse}$ represents the loss for the initial
coarse segmentation prediction used by $\lambda_{\\{1,2,3,4\\}}$ are
hyperparameter weights ${\\{0.75,0.75,0.8,0.5\\}}$.

##  4 Experiments

###  4.1 Experimental Setup

This paper uses a dataset of Thai car-damage, which is divided into four
categories: Cracked Paint, Dent, Loose, and Scrape. The number of instances
for each category is summarized in Table 1. To improve the models’ performance
and increase data diversity, the training and validation data are subjected to
data augmentation. The augmented dataset is split into training (60%),
validation (20%), and testing (20%) sets to evaluate the models’ performance
on unseen data and real-world scenarios. The data augmentation process
addresses several challenges in the dataset, including variations in
backgrounds, unique car damage, unclear or angled body damage, and flipped
images. Additionally, cross-class bias was observed between the Scrape, Loose,
Dent, and Cracked categories. The experiment’s results provide valuable
insights into instance segmentation models’ performance on the dataset.

Table 1: Object detection performance comparison Category | Instances
---|---
Cracked Paint | 273,121
Dent | 332,342
Loose | 114,345
Scrape | 434,237

###  4.2 Comparison with SOTA

This work compares the performance of the proposed method, MARS, with state-
of-the-art instance segmentation methods such as Mask R-CNN, PointRend, and
Mask Transfiner on the Thai car-damage dataset. The comparison is based on
several metrics, including AP, AP50, AP75, APs, APm, API, and FPS, and the
results are presented in Table 2.

The study shows that MARS consistently outperforms Mask R-CNN and PointRend by
4.5 AP and 2.3 AP, respectively, using R50-FPN, and surpasses Mask Transfiner
by 2.4 AP using R101-FPN. The superior performance of MARS is not limited to
overall AP, but extends to various AP metrics such as APs, APm, APl, AP50, and
AP75. For example, MARS exhibits a significant improvement in APs, with a 4.8
AP gain over Mask R-CNN using R50-FPN and a 5.1 AP gain using R101-FPN.
Furthermore, MARS outperforms other methods in APm and APl scores, indicating
its robustness across different object scales. The higher AP50 and AP75 scores
also demonstrate MARS’s improved localization and segmentation accuracy
compared to other methods.

In addition to its superior performance, MARS is computationally efficient. It
almost achieves the highest FPS on the Thai car-damage dataset, possibly
because the model was fine-tuned or optimized for this specific dataset. This
fine-tuning process could involve adjusting the model’s hyperparameters,
architecture, or training strategies to improve its efficiency and performance
on the Thai car-damage dataset. Task-specific optimizations, such as focusing
on learning the dataset’s most relevant features, may lead to a more compact
model that requires fewer computations during inference, resulting in faster
performance.

Table 2: Object detection performance comparison Method | Backbone | AP | AP50 | AP75 | APs | APm | APl | FPS
---|---|---|---|---|---|---|---|---
Mask R-CNN [9] | R50-FPN | 31.7 | 50.1 | 34.7 | 11.9 | 29.9 | 41.3 | 8.4
PointRend [14] | R50-FPN | 33.9 | 51.7 | 36.4 | 12.3 | 31.0 | 42.2 | 4.6
Mask Transfiner [13] | R50-FPN | 34.9 | 52.4 | 37.1 | 13.8 | 32.5 | 45.0 | 6.7
MARS (Ours) | R50-FPN | 36.2 | 53.0 | 38.9 | 15.8 | 34.6 | 47.3 | 6.8
Mask R-CNN [9] | R101-FPN | 32.4 | 51.5 | 35.1 | 17.6 | 33.6 | 42.0 | 8.1
PointRend [14] |  | 34.5 | 52.8 | 37.0 | 19.6 | 35.6 | 43.7 | 5.5
Mask Transfiner [13] | R101-FPN | 35.1 | 54.9 | 37.7 | 20.9 | 37.5 | 44.4 | 7.1
MARS (Ours) | R101-FPN | 37.5 | 55.7 | 41.2 | 22.7 | 38.7 | 45.1 | 7.2

Figure 5 shows qualitative comparisons on Thai car-damage dataset, where our
MARS produces masks with substantially higher precision and quality than
previous methods [9, 14, 13], especially for the hard regions, such as the
small rear mirrors and high-heeled shoes. Refer to supplementary file for more
visual comparisons.

Figure 5: Example result pairs from SOTA with its standard mask head (columns
3 and 4) vs. with MARS (right image). Note how MARS predicts masks with
substantially finer detail around object boundaries.

###  4.3 Implementation details

We utilized advanced hardware and software technologies, including the NVIDIA
T4 GPU with 16 GB memory, Intel® Xeon® Scalable (Cascade Lake) 4 vCPUs, 16 GiB
RAM, PyTorch (1.13.1), Detectron2 [24], Label Studio [21] and CUDA (11.7.0),
to perform the training and testing tasks for our project. The combination of
high-performance hardware and optimized software stack resulted in efficient
computations and superior-quality outcomes.

##  5 Conclusions

Our new instance segmentation method, MARS (Mask Attention Refinement with
Sequential Quadtree Nodes), is introduced in this work. MARS first detects and
decomposes image regions to create a hierarchical quadtree. Then, all points
on the quadtree are transformed into a self-attention layer before being
passed through our MARS model to predict car-damage classes. Unlike previous
segmentation methods that rely on convolutions with uniform image grids, MARS
produces high-quality masks with low computation and memory cost. We
demonstrate the effectiveness of MARS on query-based segmentation frameworks
and show that it outperforms existing methods on the Thai car-damage dataset.
We anticipate that the proposed MARS framework can serve as a solid baseline
for car-damage instance segmentation tasks.

## Acknowledgement

This work was partially supported by Thaivivat Insurance PCL by providing
financial support and expertise in car insurance. We would like to thank
Natthakan Phromchino, Darakorn Tisilanon, and Chollathip Thiangdee in MARS
(Motor AI Recognition Solution) for annotating the data. Teerapong Panboonyuen
and his colleague received no financial support for the research, authorship,
and/or publication of this article.

## References

  * [1] Amirfakhrian, M., Parhizkar, M.: Integration of image segmentation and fuzzy theory to improve the accuracy of damage detection areas in traffic accidents. Journal of big data 8(1), 1–17 (2021)
  * [2] Arnab, A., Torr, P.H.: Pixelwise instance segmentation with a dynamically instantiated network. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 441–450 (2017)
  * [3] Bolya, D., Zhou, C., Xiao, F., Lee, Y.J.: Yolact: Real-time instance segmentation. In: Proceedings of the IEEE/CVF international conference on computer vision. pp. 9157–9166 (2019)
  * [4] Chen, H., Sun, K., Tian, Z., Shen, C., Huang, Y., Yan, Y.: Blendmask: Top-down meets bottom-up for instance segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 8573–8581 (2020)
  * [5] Chen, K., Pang, J., Wang, J., Xiong, Y., Li, X., Sun, S., Feng, W., Liu, Z., Shi, J., Ouyang, W., et al.: Hybrid task cascade for instance segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 4974–4983 (2019)
  * [6] Chen, L.C., Hermans, A., Papandreou, G., Schroff, F., Wang, P., Adam, H.: Masklab: Instance segmentation by refining object detection with semantic and direction features. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 4013–4022 (2018)
  * [7] Deng, D., Liu, H., Li, X., Cai, D.: Pixellink: Detecting scene text via instance segmentation. In: Proceedings of the AAAI conference on artificial intelligence. vol. 32 (2018)
  * [8] Girshick, R.: Fast r-cnn. In: Proceedings of the IEEE international conference on computer vision. pp. 1440–1448 (2015)
  * [9] He, K., Gkioxari, G., Dollár, P., Girshick, R.: Mask r-cnn. In: Proceedings of the IEEE international conference on computer vision. pp. 2961–2969 (2017)
  * [10] Iglovikov, V., Seferbekov, S., Buslaev, A., Shvets, A.: Ternausnetv2: Fully convolutional network for instance segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition workshops. pp. 233–237 (2018)
  * [11] Isaksson-Hellman, I., Lindman, M.: Real-world performance of city safety based on swedish insurance data. In: 24th international technical conference on the enhanced safety of vehicles (ESV). vol. 8, pp. 15–0121 (2015)
  * [12] Jõeveer, K., Kepp, K.: What drives drivers? switching, learning, and the impact of claims in car insurance. Journal of Behavioral and Experimental Economics 103, 101993 (2023)
  * [13] Ke, L., Danelljan, M., Li, X., Tai, Y.W., Tang, C.K., Yu, F.: Mask transfiner for high-quality instance segmentation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 4412–4421 (2022)
  * [14] Kirillov, A., Wu, Y., He, K., Girshick, R.: Pointrend: Image segmentation as rendering. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 9799–9808 (2020)
  * [15] Lee, Y., Park, J.: Centermask: Real-time anchor-free instance segmentation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 13906–13915 (2020)
  * [16] Liu, S., Qi, L., Qin, H., Shi, J., Jia, J.: Path aggregation network for instance segmentation. In: Proceedings of the IEEE conference on computer vision and pattern recognition. pp. 8759–8768 (2018)
  * [17] Macedo, A.M., Cardoso, C.V., Neto, J.S.M., et al.: Car insurance fraud: the role of vehicle repair workshops. International Journal of Law, Crime and Justice 65, 100456 (2021)
  * [18] Parhizkar, M., Amirfakhrian, M.: Car detection and damage segmentation in the real scene using a deep learning approach. International Journal of Intelligent Robotics and Applications 6(2), 231–245 (2022)
  * [19] Pasupa, K., Kittiworapanya, P., Hongngern, N., Woraratpanya, K.: Evaluation of deep learning algorithms for semantic segmentation of car parts. Complex & Intelligent Systems 8(5), 3613–3625 (2022)
  * [20] Tian, Z., Shen, C., Chen, H.: Conditional convolutions for instance segmentation. In: Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part I 16. pp. 282–298. Springer (2020)
  * [21] Tkachenko, M., Malyuk, M., Holmanyuk, A., Liubimov, N.: Label Studio: Data labeling software (2020-2022), https://github.com/heartexlabs/label-studio, open source software available from https://github.com/heartexlabs/label-studio
  * [22] Wang, X., Zhang, R., Kong, T., Li, L., Shen, C.: Solov2: Dynamic and fast instance segmentation. Advances in Neural information processing systems 33, 17721–17732 (2020)
  * [23] Weisburd, S.: Identifying moral hazard in car insurance contracts. Review of Economics and Statistics 97(2), 301–313 (2015)
  * [24] Wu, Y., Kirillov, A., Massa, F., Lo, W.Y., Girshick, R.: Detectron2. https://github.com/facebookresearch/detectron2 (2019)
  * [25] Xie, E., Sun, P., Song, X., Wang, W., Liu, X., Liang, D., Shen, C., Luo, P.: Polarmask: Single shot instance segmentation with polar representation. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 12193–12202 (2020)
  * [26] Zhang, Q., Chang, X., Bian, S.B.: Vehicle-damage-detection segmentation algorithm based on improved mask rcnn. IEEE Access 8, 6997–7004 (2020)
  * [27] Zhang, W., Cheng, Y., Guo, X., Guo, Q., Wang, J., Wang, Q., Jiang, C., Wang, M., Xu, F., Chu, W.: Automatic car damage assessment system: Reading and understanding videos as professional insurance inspectors. In: Proceedings of the AAAI Conference on Artificial Intelligence. vol. 34, pp. 13646–13647 (2020)