# TraClets: Harnessing the power of computer vision for trajectory
classification

Ioannis Kontopoulos, Antonios Makris, Konstantinos Tserpes
Department of Informatics and Telematics
Harokopio University
Athens, Greece
{kontopoulos, amakris<EMAIL_ADDRESS>
Vania Bogorny
Programa de Pós-Graduação em Ciência da Computação
Universidade Federal de Santa Catarina (UFSC)
Florianópolis, Brazil
<EMAIL_ADDRESS>

###### Abstract

Due to the advent of new mobile devices and tracking sensors in recent years,
huge amounts of data are being produced every day. Therefore, novel
methodologies need to emerge that dive through this vast sea of information
and generate insights and meaningful information. To this end, researchers
have developed several trajectory classification algorithms over the years
that are able to annotate tracking data. Similarly, in this research, a novel
methodology is presented that exploits image representations of trajectories,
called TraClets, in order to classify trajectories in an intuitive humans way,
through computer vision techniques. Several real-world datasets are used to
evaluate the proposed approach and compare its classification performance to
other state-of-the-art trajectory classification algorithms. Experimental
results demonstrate that TraClets achieves a classification performance that
is comparable to, or in most cases, better than the state-of-the-art, acting
as a universal, high-accuracy approach for trajectory classification.

_K_ eywords trajectory classification $\cdot$ deep learning $\cdot$
convolutional neural networks $\cdot$ computer vision

##  1 Introduction

The sudden increase in mobile devices and tracking sensors has led to the
creation of an abundance of data. These kind of data have shifted both
researchers’ and industries’ attention alike towards algorithms and
methodologies able to extract knowledge from a large pool of spatio-temporal
data (e.g. trajectory classification). Trajectory classification is an
important mining task as it can uncover information regarding potential animal
migration patterns [1, 2], the type and strength level of hurricanes [3] and
illegal vessel activities at sea [4], thus improving overall surveillance and
safety in various domains.

Several approaches for trajectory classification [5, 4] try to exploit global
features such as average speed, acceleration or the standard deviation of
them. Other approaches try to apply trajectory partitioning [3] or find
relevant sub-trajectories [6] in an attempt to identify more discriminative
features. The context of the analysis is typically the physical world and the
geography. Position, speed and acceleration are the basic features in a
possibly multi-dimensional space. However, experts rely heavily on the
visualization of trajectories to manually classify trajectories that are of
some significance. This provides an intuition to move the analysis in a
different domain, leveraging computer vision approaches on classification. In
computer vision, the most commonly used techniques include Convolutional
Neural Networks (CNNs) [7, 8, 9]. Each layer of a CNN identifies a different
feature of the image, including but not limited to shape and color. As an
attempt to increase the performance of CNNs, deep learning emerged [8, 9],
increasing the complexity of the networks. One of the most common goals of
such networks is to classify a set of images to a predefined set of labels of
interest.

Our proposed methodology builds upon previous works [10, 11] that represent
trajectories of moving objects as images and accurately classify them, by
incorporating the notion of acceleration to the images. Furthermore, in this
work we compare our proposed methodology to four state-of-the-art trajectory
classifiers over three well-known datasets. The novelty of our approach lies
in the use of a computer vision methodology for the classification of moving
objects’ trajectories. Specifically, the proposed approach leverages an image
classification approach by visually representing trajectories as images,
called TraClets. The use of TraClets for the problem at hand and its main
contributions are:

  * •

patterns formed by the trajectories of moving objects tend to be visually
distinct [10]. This visual distinctiveness leads to an increased trajectory
classification performance;

  * •

image classification works for trajectory classification even when data are
not transmitted at fixed intervals (e.g. hourly). This is in contrast to time-
series methodologies [12] that are inherently unsuitable for such tasks and
require data points at fixed time points;

  * •

most trajectory classification approaches found in the literature [13] require
a pre-processing step that involves the understanding and analysis of data and
the selection of features suitable only for the moving objects’ trajectories
to be classified. This means that features selected for a certain trajectory
(e.g. cars) cannot be applied to other patterns as well (e.g. vessels) [14]. A
computer vision approach for trajectory classification skips entirely the
aforementioned pre-processing step; the same technique for classifying an
image (e.g. CNNs) can be applied for the classification of other trajectories
since they are transformed into images. Therefore, a computer vision approach
for trajectory classification yields a promising universal approach for the
classification of moving objects’ trajectories;

  * •

the partitioning of trajectories often constitutes a first step when dealing
with trajectory classification [15, 3]. Such approaches often require input
parameters which are hard to determine and that eventually have a significant
impact on the partitioning results. As our method entirely skips this step, as
stated above, we have eliminated the need of arbitrary or empirical user-
defined parameters, making our approach scalable and robust.

The rest of the paper is organized as follows. Section 2 serves as a
literature review in the field of trajectory classification. Section 3
describes in detail the TraClets and the way they are used in the
classification process while Section 4 evaluates the proposed approach in
terms of classification performance and compares the TraClets with other
techniques. Finally, Section 5 summarizes the merits of our work and
highlights some perspectives that require further attention in the future.

##  2 Related Work

Over the years, several techniques for trajectory classification have been
proposed, such as transportation mode prediction of mobility objects, animal
specie identification, hurricane level determination, etc. Trajectory
classification methods extract different features from the spatiotemporal
properties of trajectories, such as speed, acceleration and direction change,
to use as input for trajectory classifiers. As referred in [13], the
difference between the classification techniques is focused on the type of
trajectory features extracted for creating the classification model. Effective
trajectory classification requires generating a set of features that
discriminate the class.

One of the first trajectory classification methods was TRACLASS [16]. The
framework generated a hierarchy of features by combining two types of
clustering: i) region-based which discovers regions of trajectories that
belong on one class and ii) trajectory-based which discovers sub-trajectories
that present common moving patterns of each class. It is based on dividing the
space in grids, and by reducing the grid sizes until the trajectories inside
the grid belong to the same class. If the trajectories inside a grid cell are
from the same class then it is selected as a feature, otherwise its size is
reduced until the lowest possible size, given by a threshold. If the lowest
threshold size is reached and the trajectories inside the grid are not from
the same class, then the trajectories are split by direction change, and these
subtrajectories are clustered using a density-based clustering algorithm. The
grid cells and subtrajectory clusters are used as the features to feed the
classification model, by identifying whether the trajectory is part of that
features or not.

The works of Bolbol [17], Soleymani [18], Dabiri [19], and [20] extract local
features from trajectories, i.e., attributes from trajectory parts. More
specifically, Bolbol [17] segments the trajectories in a pre-defined number of
subtrajectories and uses a sliding window to cover a certain number of
subtrajectories. The proposed framework is based on Support Vector Machines
(SVMs) classification and extracts features as the average acceleration and
average speed of the trajectory. Soleymani [18] computes and analyzes features
in both spatial and temporal domains. The method segments the trajectories by
using two types of grids. The first grid splits the trajectories based on
their spatial location, and the technique extracts the time duration of the
subtrajectories inside each grid cell. The second grid divides the trajectory
by using a time window, and the technique calculates features as the standard
deviation of speed and maximum turning angle from each subtrajectory inside a
grid cell. Dabiri [19] presented a Convolutional Neural Network architecture
for trajectory mode classification. Four features from sequential trajectory
points (speed, acceleration, direction change and stop rate) are calculated.
Then, trajectories are represented by a vector of four dimensions, one for
each feature. This vector is fed into the CNN to estimate the transportation
mode. Vicenzi in [20] extracts local features from each trajectory point and
selects the most frequent ones to use as input to a classifier. However, the
process of finding the most discriminant dimension must be done manually by
the user.

The works of Zheng [21], Sharma [22] and Junior [23] extract global features,
i.e., features from the entire trajectory. Zheng [21] propose a supervised
learning approach for transportation mode classification. For each trajectory
are extracted features such as length, maximum speed, acceleration, average
expectation and variance of speed, heading change rate, stop rate and velocity
change rate. Subsequently, these features are used for training a Decision
Tree-based model in order to perform predictions. Junior [23] presented an
active learning approach called ANALYTiC, which enabled semantic annotation on
the learning set. The proposed approach extracts the maximum, minimum and
average speed, direction change and traveled distance for training an Active
Learning model. Sharma [22] uses the Nearest Neighbour Trajectory
Classification (NNTC), where a trajectory is assigned to the same class as its
neighbour, i.e., the closest trajectory. For data enrichment, an initial pre-
processing is performed which adds additional features such as the time of the
next position in the sequence, the time interval and the distance in space to
the next position, the speed, direction, acceleration and direction change.

A segmentation and feature extraction method for trajectories which considers
local and global features was presented in [24]. The method calculates
features every two consecutive trajectory points and then the trajectories are
represented as sequences of each feature. Local features were extracted from
sub-trajectories with the same characteristics (e.g. same speed, same
acceleration, etc.) and global features were statistics of the entire
trajectory as the minimum speed, maximum speed, average speed, minimum
acceleration, etc. Xiao [25] and Etemad [26] extract several statistics from
trajectory points, as the percentiles, interquatile range, skewness,
coefficient of variation and kurtosis from the speed, acceleration, and
heading change. Patel in [27] extended the work of Lee [16] by considering the
temporal dimension in the grid cells, where the local features were the grids,
and the global features were, for example, the total duration and the traveled
distance of a trajectory. A trajectory classification method called Movelets,
which is based on shapelet analysis is presented in [6]. The Movelets
technique extracts relevant sub-trajectories called movelets, in order to
generate local features of each trajectory and compares the distance of each
sub-trajectory to all trajectories in the dataset. Movelets is able to
consider all trajectory dimensions as space, time and semantic information. An
extension of this method is proposed in [28] to choose the best dimension
combination, but as movelets, it still has a heavy preprocessing step that
needs to explore all sub-trajectory sizes and dimension combinations to
achieve a high accuracy, while the approach proposed in this paper skips this
expensive step.

In addition, neural networks have been widely used for trajectory
classification. Jiang et al. [29] employed Recurrent Neural Networks (RNNs)
for point-based trajectory classification, by utilizing embedding of GPS data
to map the original low-dimensional and heterogeneous feature space into
distributed vector representations. The feature space was enriched with
segment-based information, and maxout activations were employed by RNNs for an
increased performance. Zhang et al. [30] employed a deep multi-scale learning
model to model grid data under different space and time granularities, thus
capturing the impact of space and time on the classification results. An
attention dense module was designed by combining an attention mechanism, which
was able to select major features, and the DenseNet model, which was able to
enhance the propagation of local and spatial features throughout the network.
Furthermore, Jiang et al. [31] employed RNNs and specifically a novel
partition-wise Gated Recurrent Unit (pGRU) architecture for point-based
trajectory classification on detecting fishing activities. The proposed method
maps low-dimensional features into another space with the use of a partition-
wise activation function applied before the linear transformation and then
receives different parameters for distinct partitions, to model them jointly
in a nonlinear hierarchical deep structure. Thus, the proposed method is less
sensitive to the quality of partitions and achieves better predictive results.
In [32], Petry proposed to use embeddings over the trajectory dimensions of
each point, and for the spatial dimension the geohash is used in the
embedding. Finally, Makris et al. [33] proposed a semi-supervised
convolutional auto-encoder (CAE) model for vessel trajectory classification.
While classic auto-encoders are often limited to learning low-level structures
of an image such as lines and edges, convolutional auto-encoders can cope with
multidimensional images. The main advantage of CAEs over traditional
supervised CNNs is their ability to achieve high classification performance in
many applications.

##  3 Methodology

This section describes the creation of TraClets from the trajectories and the
way these TraClets are employed in the classification process via a
Convolutional Neural Network.

###  3.1 TraClets

In this section we present TraClets, the image representation of trajectories.
These representations are indicative of the mobility patterns of the moving
objects. TraClets need to efficiently visualize and capture three key features
that characterize the trajectory patterns of moving objects: i) the shape of
the trajectory which indicates the way the object moves in space taking into
account the heading, ii) the speed, and iii) the acceleration that indicates
how fast the object moves in space.

####  3.1.1 Trajectory shape

Trajectories of the similar moving objects (e.g. cars) tend to form similar
patterns in terms of the way the object moves. However, the distance each
moving object travels through space is different (e.g. a car in a highway
travels greater distances compared to a car in an urban area). Therefore, the
bounding box or the surveillance area in which the object moves needs to be
normalized. To this end, the total distance of both the $x$ and the $y$ axis
in which the object moves must be defined first. For this reason, both the
total horizontal distance (Eq. 1) and the total vertical distance the object
has travelled are calculated (Eq. 2) based on the minimum and maximum
longitudes and latitudes respectively. The total horizontal distance is
defined as:

$d_{x}=longitude_{max}-longitude_{min}$ (1)

and the total vertical distance the object has travelled is defined as:

$d_{y}=latitude_{max}-latitude_{min}$ (2)

Then, the distance that each position $m$ has travelled from the minimum
longitude and latitude can be calculated from the equations 3 and 4
respectively as follows:

$d(m_{x})=longitude_{m}-longitude_{min}$ (3)

and

$d(m_{y})=latitude_{m}-latitude_{min}$ (4)

From equations 1,3 and 2,4, the percentage of the total distance each position
$m$ has travelled so far can be calculated from the minimum coordinate in both
$x$ and $y$ axes:

$normalized(m_{x})=d(m_{x})\div d_{x}$ (5)

and

$normalized(m_{y})=d(m_{y})\div d_{y}$ (6)

Given a predefined image size of $N\times N$, the exact position of $m$ inside
an image can be calculated as follows:

$pixel_{x}=normalized(m_{x})\times N$ (7)

and

$pixel_{y}=normalized(m_{y})\times N$ (8)

Therefore, each position is placed inside a normalized bounding box or a
surveillance space of size $N\times N$ that is essentially an image
representation. Figure 1 demonstrates an example of the surveillance space
normalization for $N=10$. Each green circle corresponds to a position. The
total horizontal distance is $\mathit{d_{x}=28.75-2.875=25.875}$ while the
total vertical distance is $\mathit{d_{y}=92.5-9.25=83.25}$. Based on equation
3 and 4, $d(m_{x})$ of the upper right position
($longitude=23.0,latitude=9.25$) is equal to $20.125$ and $(m_{y})$ is equal
to $0$. According to equations 5 and 6, $normalized(m_{x})$ is equal to $0.7$
and $normalized(m_{y})$ is equal to $0$. Multiplying each normalized value by
$N=10$ results in $pixel_{x}=7$ and $pixel_{y}=0$. In order to fit boundary
trajectory positions into the normalized surveillance space, pixel values
smaller than $N$ are transformed into 1 and pixel values larger than $N$ are
transformed into $N$. Therefore, $pixel_{y}$ is transformed into $1$ and the
final pixel position inside the $10\times 10$ image is $x=7$ and $y=1$ as
shown in Figure 1.

(a) Positions of the trajectory.

(b) Positions placed in a $10\times 10$ raster.

Figure 1: An example of space normalization.

####  3.1.2 Trajectory speed

To capture the changes in speed, previous research conducted on the
classification of vessel trajectories [10, 11] is taken into consideration, in
which the maximum speed $max(speed)$ of the vessels was segmented to $11$
equally sized increments with each speed increment corresponding to a
different RGB color value in the final image. Let $S=[0,max(speed)]$ be the
possible speed values at which the moving objects move. A speed increment
$incr(speed)$ is defined as:

$incr(speed)=max(speed)\div 11$ (9)

resulting in an arithmetic sequence such that:

$\alpha_{i}=\alpha_{1}+(i-1)\times incr(speed)$ (10)

where $\alpha_{i}$ is the $i^{th}$ term of the sequence, $\alpha_{1}$ is the
initial term of the sequence (in our case $0~{}m/s$) and $incr(speed)$ is the
common difference of successive members. Each pixel $p(pixel_{x},pixel_{y})$
that corresponds to a position $m$ is colored with a different color value
that corresponds to the $i^{th}$ term of the sequence depending on the speed
the position has at the moment. Moreover, pixels that do not contain any
positions are colored white.

####  3.1.3 Trajectory acceleration

To represent the acceleration of the trajectory, a straight line between each
temporally consecutive $pixel(pixel_{x},pixel_{y})$ or position $m$ is drawn
initially using the Bresenham’s line algorithm [34], in order to make the
pattern created by each moving object more distinctive. The Bresenham’s line
algorithm is a line-drawing algorithm that generates points that form a close
approximation to a straight line between two points of an $N$-dimensional
raster. Then, similar to Section 3.1.2, the acceleration is segmented to $11$
equally sized increments and each line is colored based on the absolute value
of the acceleration and the $i^{th}$ term of the acceleration sequence that
the acceleration value falls into. The acceleration of the line is defined as
the acceleration between the starting and ending position $m$ of the
respective line. The final result is an image, called TraClet, that depicts
the movement, the speed and the acceleration of the trajectory. An example of
a TraClet is illustrated in Figure 2.

Figure 2: An example of a TraClet.

###  3.2 Deep learning for trajectory classification

This section describes the deep learning approach employed for mobility data
classification from trajectory images, based on fine-tuning and Convolutional
Neural Networks.

CNNs are able to learn deep features of trajectories such as shape and color
(e.g., speed, acceleration) automatically, thus eliminating hand-created
feature extraction. Furthermore, they are able to capture more local spatial
correlations, as neurons in the CNN receive signals from other neurons in the
local area in the preceding layer [35]. In addition, the weight-sharing
characteristic in the connection of adjacent layers can significantly reduce
the number of variables. The main disadvantage of image classification through
deep learning is that these methods require a large amount of data in order to
perform accurate feature extraction and classification. In order to overcome
this limitation, transfer learning was adopted. Transfer learning is a common
and effective method, which aims at training a network with fewer samples, as
the knowledge extracted by a pre-trained model is then reused and applied to
the given task of interest. The intuition behind transfer learning is that
generic features learned on a general large dataset, can be shared among
seemingly disparate datasets. The learned features can be used to solve a
different, but related task [36].

Fine-tuning was employed as the standard method for transfer learning. The
weights were pre-trained on the ImageNet [37] dataset for the examined CNN,
namely VGG16. Figure 3 illustrates the fine-tuning process on the VGG16
network. The VGG16 model consists of $13$ convolutional and $3$ fully
connected ($FC$) layers. The final set of layers which contain the $FC$ layers
along with the activation function is called the “head”. The network was
instantiated with weights pre-trained on ImageNet, as shown at the top of the
figure. Afterwards, the $FC$ layers were truncated, and the final $pooling$
layer was treated as a feature extractor. The body of the network, i.e., the
weights of the convolution layers of the pre-trained network, were freezed
such that only the $FC$ head layer was trained. This was because the
convolution layers had already learned discriminative filters and captured
universal features like curves and edges; thus, these weights had to remain
intact. Finally, the truncated layers were replaced by a new $FC$ head layer,
which was randomly initialized and placed on top of the original architecture
(bottom of the figure). In other words, the $FC$ head layer was randomly
initialized from scratch and focused on learning dataset-specific features.

Figure 3: Fine-tuning on the VGG16 network architecture.

Trajectory data were transformed into images and labeled based on concrete
mobility patterns’ annotation into several classes based on the dataset.
Animals dataset for example, contains three classes which represent three
animal species, namely elk, mule deer and cattle. These images, i.e. TraClets,
were fed as input into the deep learning model for training. Input images were
scaled to the fixed size of $224\times 224$ pixels. Training was conducted for
$100$ epochs with a learning rate of $1e-3$ and a batch size of $16$. The
output of the convolution layers was activated by the non-linear activation
function called the Rectified Linear Unit (ReLU), due to its reduced
likelihood of vanishing gradients and its efficient computation. After each
convolution layer, the pooling layer was introduced to carry out downsampling
operations, which reduced the in-plane dimensionality of the feature maps.
Downsample operations after convolutional layers introduce a translation
invariance to small shifts and distortions and decrease the number of
subsequent learnable parameters. Average pooling [38] was employed as the
pooling strategy and the number of learnable parameters was reduced,
preventing overfitting. The output feature maps of the final layer were
flattened and connected to the fully connected layers, in which every input
was connected to every output by a learnable weight. The output layer
generated a probability distribution over the classification labels, by
resorting the softmax function. As we are facing multi-class classification,
the “categorical_crossentropy” is utilized as the loss function which compares
the distribution of the predictions with the true distribution.

Adam optimization method [39] was used to find the minimum of the objective
(error) function making use of its gradient. Furthermore, a dropout layer [40]
of $0.2$ was applied, which means that 20% of neurons were randomly set to
zero during each training epoch, thus avoiding overfitting on the training
dataset. Dropout is one of the most popular regularization techniques, which
forces the weights in the network to receive only small values, making the
distribution of weight values more regular. As a result, this technique is
able to reduce overfitting on small training examples [41]. In addition, in
order to prevent model overfitting, data augmentation was performed during
training, leveraging several multi-processing techniques. Specifically, the
transformations employed included random rotation of the images (the maximum
rotation angle was 30 degrees), horizontal flips, shearing, zooming, cropping,
and small random noise perturbations. Data augmentation improved the
generalization and enhanced the learning capability of the model. Table 1
summarizes the parameters employed by the model.

Parameter | Value
---|---
Epochs | $100$
Learning rate | $1e-3$
Batch size | $16$
Activation function | $ReLU$
Pooling | $Average2D$
Optimizer | $Adam$
Output activation function | $Softmax$
Loss function | $categorical\\_crossentropy$
Dropout probability | $0.2$
Table 1: Configuration of the CNN architecture

We have implemented an open-source deep learning toolkit called
d-Look111https://github.com/AntonisMakris/d-LOOK under the MIT Licence, which
provides an automated way to execute various supervised deep learning
Keras222https://keras.io/ models. In this research work, we have utilized only
one model provided by d-Look, VGG16, as it is proven to obtain the best
classification accuracy results in comparison with other models such as
Inception, DenseNet etc. [11, 42].

##  4 Experimental Evaluation

###  4.1 Dataset Description

In general, trajectories can be classified into four major categories
regarding mobility: people, vehicles, animals and natural phenomena [43]. The
datasets employed for the evaluation of the different trajectory
classification techniques, were selected according to some criteria:

  1. 1.

the data are categorized in classes, so they can be used to evaluate
classification problems.

  2. 2.

they contain trajectories of different objects (people, vehicles, hurricanes,
and animals).

  3. 3.

they present variations in their sampling rates, ranging from highly sampled
to very low sampling rate.

Many of these datasets have already being used in [6] and [44] for comparing
several classification methods. A detailed description of the trajectory
datasets is presented below.

####  4.1.1 Microsoft GeoLife dataset

This GPS trajectory dataset [45, 21, 46], was collected in (Microsoft Research
Asia) Geolife project by 182 users in a period of over three years (from April
2007 to August 2012). This dataset recorded a broad range of users’ outdoor
movements, including not only life routines like go home and go to work but
also some entertainments and sports activities, such as shopping, sightseeing,
dining, hiking, and cycling. Furthermore, it contains several transportation
modes as cars, bike, bus, and foot. As this dataset has a high sampling rate,
some issues such as outliers and duplicated records needed to be fixed before
applying data mining techniques. The pre-processing phase contains the
following steps: a) removing duplicated records, b) splitting the trajectories
where two consecutive points had more than 300 seconds of difference between
them, as it represents a large gap in such dense dataset, c) removing
trajectories with less than 100 points, as it represents a very small portion
of time due to the density, d) excluding the transportation modes with too
less trajectory examples, which are the classes of airplane, boat, running and
motorcycle e) removing trajectories with unreal average velocity given the
transportation mode, as for instance, trajectories labeled with walk with
average velocity of more than 10m/s, and f) from the remaining trajectories, a
proportion of 20% of the trajectories of each transportation mode was
selected, as some techniques were unable to perform in a reasonable amount
time.

####  4.1.2 Hurricane Dataset

This dataset333https://www.nhc.noaa.gov/data/ is provided by the National
Hurricane Service (NHS) and is related to Atlantic Hurricanes Database
collected between 1950 and 2008. It contains sparse points with hurricane
trajectories, where the hurricanes are classified in strength level. It has a
low sampling rate and the class is the degree of the hurricane.

####  4.1.3 Animals Dataset

This dataset444https://www.fs.fed.us/pnw/starkey/index.shtml contains
information of three animal species from the Starkey project. It contains
sparse points of major habitat variables derived for radiotelemetry studies of
elk, mule deer and cattle at the Starkey Experimental Forest and Range, 28
miles southwest of La Grande, Oregon. It has a low sampling rate and the class
is the type of animal.

Table 2 provides information about the datasets employed for evaluating the
various trajectory classification methods.

Dataset | Trajectories | Points | Sampling rate | Type
---|---|---|---|---
GeoLife | 1,763 | 953,966 | high | dense
Hurricanes | 1003 | 26,783 | low | sparse
Animals | 253 | 287,136 | low | sparse
Table 2: Datasets Overview

###  4.2 Trajectory Classification Techniques

Trajectory classification techniques aim to identify the classes of the
trajectories by extracting features that are capable of distinguishing the
classes. In this research work, the proposed deep learning method is compared
against four more methods. These methods, which are used for extracting
trajectory features, were selected because of their best classification
accuracy. Table 3 summarizes the classification methods, the trajectory
dimensions they support and the features they extract from trajectories that
are used as input for the classification algorithms.

Technique | Trajetory Dimensions | Features Extracted
---|---|---
Dodge [24], Zheng [21] and Xiao [25] | Space and time | Numerical features, e.g. speed, acceleration, deviation rate, etc.
Movelets [6] | Space and Time | Movelets extracted from space dimension + global features as indicated in [6]
TraClets | Space and Time | Image representations extracted from raw trajectories
Table 3: Summary of the classification techniques, with the trajectory
dimensions they use, and the features they extract from trajectories.

The techniques of Dodge [24], Zheng [21] and Xiao [25] extract the numerical
features from trajectories, e.g. the average speed, the maximum acceleration,
etc., which are formulas that need the spatio-temporal dimensions from
trajectories. The Movelets [6] extract the movelets, which are the sub-
trajectories that better discriminate the classes. Although the Movelets can
deal with any trajectory dimension, Ferrero in [6] indicated that the best
dimension is the space, with the aggregation of some numerical features
extracted from the whole trajectories. In TraClets, image representations are
extracted from raw trajectory data.

###  4.3 Experimental Results

After extracting the features, the Random Forest (RF) is trained and used to
compare Dodge, Zheng, Xiao and Movelets, as it is commonly used by
classification techniques in the literature. The works of [6], [5] and [10]
have compared several classifiers including SVM (Support Vector Machine), MLP
(Multi Layer Perceptron) and Random Forest, and they have shown a similar
accuracy. We have selected RF because it performs faster than SVM and MLP.
Random Forest was constructed with $100$ decision trees with the default
structure. The threshold values used by Dodge, Zheng and Xiao are those
reported in the original paper. To extract the local features of [24] the
sinuosity threshold is necessary to identify similar subtrajectories, and this
threshold is given by the mean between the minimum and maximum sinuosity
value. To extract the Zheng features that require thresholds (e.g. the heading
change rate) we used the method suggested in [21] for finding the thresholds,
which consists on evaluating each feature individually for finding the best
set of thresholds for each dataset. In the Movelets technique the Euclidean
distance measure is used. The datasets were split in a hold-out manner, in a
proportion of $70\%$ for training and $30\%$ for testing.

In order to evaluate the classification performance, the Accuracy (ACC) metric
was adopted. $Accuracy$ indicates how well a classification algorithm can
discriminate the classes of the trajectories in the test set. As shown in
Equation (11), $ACC$ can be defined as the proportion of the predicted correct
labels (true positives/negatives) to the total number of labels ($N$).

$Accuracy(ACC)=\frac{|Correctly\ Labeled\ Examples|}{N}$ (11)

The classification performance of VGG16 was evaluated on the effectiveness of
trajectory pattern recognition. The datasets were split in the same hold-out
manner as previous ($70\%$ \- $30\%$) and reported the macro-average results.
The Keras package and a TensorFlow backend were employed along with the Python
programming language for training the deep learning models. Keras is a simple-
to-use neural network library built on top of Theano or TensorFlow.

Dataset | [24] | [21] | [25] | [6] | TraClets
---|---|---|---|---|---
GeoLife | $81.01\%$ | $82.33\%$ | $83.77\%$ | $83.98\%$ | $\textbf{86.5}\%$
Hurricanes | $60.52\%$ | $58.22\%$ | $60.19\%$ | $\textbf{61.84}\%$ | $59.2\%$
Animals | $82.05\%$ | $87.17\%$ | $85.89\%$ | $91.02\%$ | $\textbf{92.15}\%$
Table 4: Summary of the classification results in terms of accuracy.

Table 4 illustrates the classification accuracy of all classifiers over all
datasets. It is shown that TraClets achieved the best accuracy in two out of
three datasets with an accuracy of $86.5\%$ in the GeoLife dataset and
$92.15\%$ in the Animals dataset while Movelets are the next best classifier
succeeding TraClets with an accuracy of $83.98\%$ and $91.02\%$ respectively.
On the other hand, Movelets achieved the best accuracy in the Hurricanes
dataset ($61.84\%$). Furthermore, it is worth noting that the Hurricanes
dataset challenged every classifier yielding the lowest classification
accuracies.

(a) GeoLife dataset.

(b) Hurricanes dataset.

(c) Animals dataset.

Figure 4: Accuracy and Loss during training.

To further visualize the TraClets’ performance we illustrated the loss and the
accuracy during the training process in Figure 4. TraClets demonstrates a
smooth training process for all datasets, during which the loss gradually
decreases and the accuracy increases. Moreover, it can be observed that the
accuracy of both training and validation do not deviate much from one another,
a phenomenon that can also be observed for the training and validation loss,
indicating that the deep learning model does not overfit. Finally, in Figure
4(b) that demonstrates the training process in the hurricanes dataset, it can
be seen that the accuracy does not increase much and the loss stops decreasing
after epoch 20, indicating that the deep learning model cannot distinguish
between the classes beyond that point.

The fact that the model cannot distinguish between the classes of hurricanes
is also apparent in Table 5 that illustrates the precision, recall and
f1-score of our approach in all datasets. It can be observed that the scores
achieved in the hurricanes dataset are two low (F1-score of $0.26$). On the
other hand, the classes of the rest of the datasets can be visually distinct
from each other and for that reason the proposed approach achieved an F1-score
of $0.83$ and $0.88$ in the GeoLife and the Animals dataset respectively. It
is worth noting that the proposed methodology for trajectory classification
achieved a state-of-the-art classification performance with no configurations,
i.e., no manual adjustment of the speed and acceleration increments was
performed as mentioned in Sections 3.1.2 and 3.1.3 respectively. Therefore, we
can safely assume that TraClets can act as a universal trajectory
classification technique that can work well in a wide range of datasets
requiring minimal or no configurations in most cases to achieve a high-
accuracy classification performance.

Dataset | Precision | Recall | F1-score
---|---|---|---
GeoLife | $0.84$ | $0.83$ | $0.83$
Hurricanes | $0.26$ | $0.28$ | $0.26$
Animals | $0.94$ | $0.86$ | $0.88$
Table 5: Summary of the classification results of TraClets.

##  5 Conslusion

This work presented a novel and high-accuracy trajectory classification
approach in an attempt to provide an efficient and alternative way to treat
the problem of trajectory classification. The proposed methodology employs a
state-of-the-art deep learning algorithm and creates a universal approach for
the classification of trajectories. This universal approach can be achieved
through image representations of trajectories, called TraClets, that amplify
the distinct visual difference that most of the moving objects’ trajectories,
if not all, have in three distinct domains.

To demonstrate the applicability of our work, three datasets were used to
evaluate the performance of the proposed approach. As deep learning approaches
require a large amount of data in order to perform an accurate classification,
transfer learning was also employed. Experimental evaluation demonstrated that
TraClets achieve a state-of-the-art classification accuracy that is comparable
to or better than other state-of-the-art trajectory classification algorithms
in the literature.

As a future work, we plan on extending and evaluating our approach on more
domains such as airplanes that may require to take into account the height
instead of only longitude and latitude.

## Acknowledgments

This work was supported by the MASTER and SmartShip Projects through the
European Union’s Horizon 2020 research and innovation program under Marie-
Sklodowska Curie Grant Agreement Nos. 777695 and 823916, respectively. The
work reflects only the authors’ view, and the EU Agency is not responsible for
any use that may be made of the information it contains. The works was also
supported by the Brazilian agencies CAPES PRINT, CNPQ and FAPESC.

## References

  * [1] M. L. Damiani, H. Issa, G. Fotino, M. Heurich, F. Cagnacci, Introducing ’presence’ and ’stationarity index’ to study partial migration patterns: an application of a spatio-temporal clustering technique, Int. J. Geogr. Inf. Sci. 30 (5) (2016) 907–928.  doi:10.1080/13658816.2015.1070267.
URL https://doi.org/10.1080/13658816.2015.1070267

  * [2] J. De Groeve, N. Van de Weghe, N. Ranc, T. Neutens, L. Ometto, O. Rota-Stabelli, F. Cagnacci, Extracting spatio-temporal patterns in animal trajectories: an ecological application of sequence analysis methods, Methods in Ecology and Evolution 7 (3) (2016) 369–379.  arXiv:https://besjournals.onlinelibrary.wiley.com/doi/pdf/10.1111/2041-210X.12453, doi:https://doi.org/10.1111/2041-210X.12453.
URL
https://besjournals.onlinelibrary.wiley.com/doi/abs/10.1111/2041-210X.12453

  * [3] J. Lee, J. Han, X. Li, H. Gonzalez, _TraClass_ : trajectory classification using hierarchical region-based and trajectory-based clustering, Proc. VLDB Endow. 1 (1) (2008) 1081–1094.  doi:10.14778/1453856.1453972.
URL http://www.vldb.org/pvldb/vol1/1453972.pdf

  * [4] I. Kontopoulos, K. Chatzikokolakis, K. Tserpes, D. Zissis, Classification of vessel activity in streaming data, in: J. Gascon-Samson, K. Zhang, K. Daudjee, B. Kemme (Eds.), DEBS ’20: The 14th ACM International Conference on Distributed and Event-based Systems, Montreal, Quebec, Canada, July 13-17, 2020, ACM, 2020, pp. 153–164.  doi:10.1145/3401025.3401763.
URL https://doi.org/10.1145/3401025.3401763

  * [5] C. L. da Silva, L. M. Petry, V. Bogorny, A survey and comparison of trajectory classification methods, in: 8th Brazilian Conference on Intelligent Systems, BRACIS 2019, Salvador, Brazil, October 15-18, 2019, IEEE, 2019, pp. 788–793.  doi:10.1109/BRACIS.2019.00141.
URL https://doi.org/10.1109/BRACIS.2019.00141

  * [6] C. A. Ferrero, L. O. Alvares, W. Zalewski, V. Bogorny, Movelets: Exploring relevant subtrajectories for robust trajectory classification (2018) 9–13.
  * [7] W. Rawat, Z. Wang, Deep convolutional neural networks for image classification: A comprehensive review, Neural Comput. 29 (9) (2017) 2352–2449.  doi:10.1162/neco\\_a\\_00990.
URL https://doi.org/10.1162/neco_a_00990

  * [8] S. Zhong, Y. Liu, Y. Liu, Bilinear deep learning for image classification, in: K. S. Candan, S. Panchanathan, B. Prabhakaran, H. Sundaram, W. Feng, N. Sebe (Eds.), Proceedings of the 19th International Conference on Multimedia 2011, Scottsdale, AZ, USA, November 28 - December 1, 2011, ACM, 2011, pp. 343–352.  doi:10.1145/2072298.2072344.
URL https://doi.org/10.1145/2072298.2072344

  * [9] J. Wu, Y. Yu, C. Huang, K. Yu, Deep multiple instance learning for image classification and auto-annotation, in: IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2015, Boston, MA, USA, June 7-12, 2015, IEEE Computer Society, 2015, pp. 3460–3469.  doi:10.1109/CVPR.2015.7298968.
URL https://doi.org/10.1109/CVPR.2015.7298968

  * [10] I. Kontopoulos, A. Makris, D. Zissis, K. Tserpes, A computer vision approach for trajectory classification, in: 2021 22nd IEEE International Conference on Mobile Data Management (MDM), IEEE, 2021, pp. 163–168.
  * [11] I. Kontopoulos, A. Makris, K. Tserpes, A deep learning streaming methodology for trajectory classification, ISPRS Int. J. Geo Inf. 10 (4) (2021) 250.  doi:10.3390/ijgi10040250.
URL https://doi.org/10.3390/ijgi10040250

  * [12] H. I. Fawaz, G. Forestier, J. Weber, L. Idoumghar, P. Muller, Deep learning for time series classification: a review, Data Min. Knowl. Discov. 33 (4) (2019) 917–963.  doi:10.1007/s10618-019-00619-1.
URL https://doi.org/10.1007/s10618-019-00619-1

  * [13] C. L. da Silva, L. M. Petry, V. Bogorny, A survey and comparison of trajectory classification methods, in: 2019 8th Brazilian Conference on Intelligent Systems (BRACIS), IEEE, 2019, pp. 788–793.
  * [14] E. Souza, K. Boerder, B. Worm, Improving fishing pattern detection from satellite ais using data mining and machine learning, PLOS ONE 11 (2016) e0158248.  doi:10.1371/journal.pone.0158248.
  * [15] J. Lee, J. Han, K. Whang, Trajectory clustering: a partition-and-group framework, in: C. Y. Chan, B. C. Ooi, A. Zhou (Eds.), Proceedings of the ACM SIGMOD International Conference on Management of Data, Beijing, China, June 12-14, 2007, ACM, 2007, pp. 593–604.  doi:10.1145/1247480.1247546.
URL https://doi.org/10.1145/1247480.1247546

  * [16] J.-G. Lee, J. Han, X. Li, H. Gonzalez, Traclass: Trajectory classification using hierarchical region-based and trajectory-based clustering, Proceedings of The Vldb Endowment - PVLDB 1 (08 2008).  doi:10.14778/1453856.1453972.
  * [17] A. Bolbol, T. Cheng, I. Tsapakis, J. Haworth, Inferring hybrid transportation modes from sparse gps data using a moving window svm classification, Computers, Environment and Urban Systems 36 (6) (2012) 526–537.
  * [18] A. Soleymani, J. Cachat, K. Robinson, S. Dodge, A. Kalueff, R. Weibel, Integrating cross-scale analysis in the spatial and temporal domains for classification of behavioral movement, Journal of Spatial Information Science 2014 (8) (2014) 1–25.
  * [19] S. Dabiri, K. Heaslip, Inferring transportation modes from gps trajectories using a convolutional neural network, Transportation research part C: emerging technologies 86 (2018) 360–371.
  * [20] F. Vicenzi, L. M. Petry, C. L. da Silva, L. O. Alvares, V. Bogorny, Exploring frequency-based approaches for efficient trajectory classification, in: C. Hung, T. Cerný, D. Shin, A. Bechini (Eds.), SAC ’20: The 35th ACM/SIGAPP Symposium on Applied Computing, online event, [Brno, Czech Republic], March 30 - April 3, 2020, ACM, 2020, pp. 624–631.  doi:10.1145/3341105.3374045.
URL https://doi.org/10.1145/3341105.3374045

  * [21] Y. Zheng, Q. Li, Y. Chen, X. Xie, W.-Y. Ma, Understanding mobility based on gps data, in: Proceedings of the 10th international conference on Ubiquitous computing, ACM, 2008, pp. 312–321.
  * [22] L. K. Sharma, O. P. Vyas, S. Schieder, A. K. Akasapu, Nearest neighbour classification for trajectory data, in: International Conference on Advances in Information and Communication Technologies, Springer, 2010, pp. 180–185.
  * [23] A. S. Júnior, C. Renso, S. Matwin, Analytic: An active learning system for trajectory classification, IEEE computer graphics and applications 37 (5) (2017) 28–39.
  * [24] S. Dodge, R. Weibel, E. Forootan, Revealing the physics of movement: Comparing the similarity of movement characteristics of different types of moving objects, Computers, Environment and Urban Systems 33 (6) (2009) 419–434.
  * [25] Z. Xiao, Y. Wang, K. Fu, F. Wu, Identifying different transportation modes from trajectory data using tree-based ensemble classifiers, ISPRS International Journal of Geo-Information 6 (2) (2017) 57.
  * [26] M. Etemad, A. Soares Júnior, S. Matwin, Predicting transportation modes of gps trajectories using feature engineering and noise removal, in: Advances in Artificial Intelligence: 31st Canadian Conference on Artificial Intelligence, Canadian AI 2018, Toronto, ON, Canada, May 8–11, 2018, Proceedings 31, Springer, 2018, pp. 259–264.
  * [27] D. Patel, C. Sheng, W. Hsu, M. L. Lee, Incorporating duration information for trajectory classification, in: Data Engineering (ICDE), 2012 IEEE 28th International Conference on, IEEE, 2012, pp. 1132–1143.
  * [28] C. A. Ferrero, L. M. Petry, L. O. Alvares, C. L. da Silva, W. Zalewski, V. Bogorny, Mastermovelets: discovering heterogeneous movelets for multiple aspect trajectory classification, Data Mining and Knowledge Discovery 34 (3) (2020) 652–680.
  * [29] X. Jiang, E. N. de Souza, A. Pesaranghader, B. Hu, D. L. Silver, S. Matwin, Trajectorynet: an embedded GPS trajectory representation for point-based classification using recurrent neural networks, in: M. Mindel, K. A. Lyons, J. Wigglesworth (Eds.), Proceedings of the 27th Annual International Conference on Computer Science and Software Engineering, CASCON 2017, Markham, Ontario, Canada, November 6-8, 2017, IBM / ACM, 2017, pp. 192–200.
URL http://dl.acm.org/citation.cfm?id=3172817

  * [30] R. Zhang, P. Xie, C. Wang, G. Liu, S. Wan, Classifying transportation mode and speed from trajectory data via deep multi-scale learning, Comput. Networks 162 (2019).  doi:10.1016/j.comnet.2019.106861.
URL https://doi.org/10.1016/j.comnet.2019.106861

  * [31] X. Jiang, X. Liu, E. N. de Souza, B. Hu, D. L. Silver, S. Matwin, Improving point-based AIS trajectory classification with partition-wise gated recurrent units, in: 2017 International Joint Conference on Neural Networks, IJCNN 2017, Anchorage, AK, USA, May 14-19, 2017, IEEE, 2017, pp. 4044–4051.  doi:10.1109/IJCNN.2017.7966366.
URL https://doi.org/10.1109/IJCNN.2017.7966366

  * [32] L. M. Petry, C. L. da Silva, A. Esuli, C. Renso, V. Bogorny, MARC: a robust method for multiple-aspect trajectory classification via space, time, and semantic embeddings, Int. J. Geogr. Inf. Sci. 34 (7) (2020) 1428–1450.  doi:10.1080/13658816.2019.1707835.
URL https://doi.org/10.1080/13658816.2019.1707835

  * [33] A. Makris, I. Kontopoulos, E. Psomakelis, K. Tserpes, Semi-supervised trajectory classification using convolutional auto-encoders, in: Proceedings of the 1st ACM SIGSPATIAL International Workshop on Animal Movement Ecology and Human Mobility, 2021, pp. 27–32.
  * [34] F. L. Gaol, Bresenham algorithm: Implementation and analysis in raster shape, J. Comput. 8 (1) (2013) 69–78.  doi:10.4304/jcp.8.1.69-78.
URL https://doi.org/10.4304/jcp.8.1.69-78

  * [35] R. Chen, M. Chen, W. Li, J. Wang, X. Yao, Mobility modes awareness from trajectories based on clustering and a convolutional neural network, ISPRS Int. J. Geo Inf. 8 (5) (2019) 208.  doi:10.3390/ijgi8050208.
URL https://doi.org/10.3390/ijgi8050208

  * [36] S. J. Pan, Q. Yang, A survey on transfer learning, IEEE Transactions on knowledge and data engineering 22 (10) (2009) 1345–1359.
  * [37] J. Deng, W. Dong, R. Socher, L. Li, K. Li, F. Li, Imagenet: A large-scale hierarchical image database, in: 2009 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR 2009), 20-25 June 2009, Miami, Florida, USA, IEEE Computer Society, 2009, pp. 248–255.  doi:10.1109/CVPR.2009.5206848.
URL https://doi.org/10.1109/CVPR.2009.5206848

  * [38] M. Lin, Q. Chen, S. Yan, Network in network, arXiv preprint arXiv:1312.4400 (2013).
  * [39] D. P. Kingma, J. Ba, Adam: A method for stochastic optimization, in: Y. Bengio, Y. LeCun (Eds.), 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings, 2015.
URL http://arxiv.org/abs/1412.6980

  * [40] G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Improving neural networks by preventing co-adaptation of feature detectors, CoRR abs/1207.0580 (2012).  arXiv:1207.0580.
URL http://arxiv.org/abs/1207.0580

  * [41] D. M. Hawkins, The problem of overfitting, Journal of chemical information and computer sciences 44 (1) (2004) 1–12.
  * [42] A. Makris, I. Kontopoulos, K. Tserpes, Covid-19 detection from chest x-ray images using deep learning and convolutional neural networks, in: 11th Hellenic Conference on Artificial Intelligence, 2020, pp. 60–66.
  * [43] P. Sun, S. Xia, G. Yuan, D. Li, An overview of moving object trajectory compression algorithms, Mathematical Problems in Engineering 2016 (2016).
  * [44] A. Makris, C. L. da Silva, V. Bogorny, L. O. Alvares, J. A. Macedo, K. Tserpes, Evaluating the effect of compressing algorithms for trajectory similarity and classification problems, GeoInformatica (2021) 1–33.
  * [45] Y. Zheng, X. Xie, W.-Y. Ma, et al., Geolife: A collaborative social networking service among user, location and trajectory., IEEE Data Eng. Bull. 33 (2) (2010) 32–39.
  * [46] Y. Zheng, L. Zhang, X. Xie, W.-Y. Ma, Mining interesting locations and travel sequences from gps trajectories, in: Proceedings of the 18th international conference on World wide web, ACM, 2009, pp. 791–800.