# Multi-modal Visual Tracking: Review and Experimental Comparison

Pengyu Zhang, Dong Wang, Huchuan Lu

###### Abstract

Visual object tracking, as a fundamental task in computer vision, has drawn
much attention in recent years. To extend trackers to a wider range of
applications, researchers have introduced information from multiple modalities
to handle specific scenes, which is a promising research prospect with
emerging methods and benchmarks. To provide a thorough review of multi-modal
tracking, we summarize the multi-modal tracking algorithms, especially
visible-depth (RGB-D) tracking and visible-thermal (RGB-T) tracking in a
unified taxonomy from different aspects. Second, we provide a detailed
description of the related benchmarks and challenges. Furthermore, we conduct
extensive experiments to analyze the effectiveness of trackers on five
datasets: PTB, VOT19-RGBD, GTOT, RGBT234, and VOT19-RGBT. Finally, we discuss
various future directions from different perspectives, including model design
and dataset construction for further research.

###### keywords:

Visual tracking, Object tracking, Multi-modal fusion, RGB-T tracking, RGB-D
tracking

††journal: Pattern Recognition

##  1 Introduction

Visual object tracking is a fundamental task in computer vision, which is
widely applied in many areas, such as smart surveillance, autonomous driving,
and human-computer interaction. Traditional tracking methods are mainly based
on visible (RGB) images captured by a monocular camera. When the target
suffers long-term occlusion or is in low-illumination scenes, the RGB tracker
can hardly work well and may cause tracking failure. With the easy-access
binocular camera, tracking with multi-modal information (e.g. visible-depth,
visible-thermal, visible-radar and visible-laser) is a prospective research
direction that has become popular in recent years. Many datasets and
challenges have been presented [1, 2, 3, 4, 5, 6]. Motivated by these
developments, trackers with multi-modal cues have been proposed with the
potential accuracy and robustness against extreme tracking scenarios [7, 8, 9,
10, 11].

With the emergence of multi-modal trackers, a comprehensive and in-depth
survey has not been conducted. To this end, we revisit existing methods from a
unified view and evaluate them on popular datasets. The contributions of this
work can be summarized as follows.

  * 1.

Substantial review of multi-modal tracking methods from various aspects in a
unified view. We exploit the similarity of RGB-D and RGB-T tracking and
classify them in a unified framework. We category existing 56 multi-modal
tracking methods based on auxiliary modality, tracking framework, and related
datasets with corresponding metrics. Taxonomy with detailed analysis can cover
the main knowledge in this field, and provide an in-depth introduction to
multi-modal tracking models.

  * 2.

A comprehensive and fair evaluation of popular trackers on several datasets.
We collect 29 methods consisting of 14 RGB-D and 15 RGB-T trackers, and
evaluate them on 5 datasets in accuracy and speed for various applications. We
further analyze the advantages and drawbacks of different frameworks in
qualitative and quantitative experiments.

  * 3.

A prospective discussion for multi-modal tracking. We present the potential
direction of multi-modal tracking in model design and dataset construction,
which can provide prospective guidance to researchers.

The rest of the paper is organized as follows. In section 2, we introduce
existing related basic concepts and previous related surveys. Section 3
provides a taxonomical review of multi-modal tracking. We represent the
introduction of existing datasets, challenges, and corresponding evaluation
metrics described in section 4. In section 5, we report the experimental
results on several datasets and different challenges. Finally, we discuss the
future direction of multi-modal tracking in section 6. All the collected
materials and analysis will be released at https://github.com/zhang-
pengyu/Multimodal_tracking_survey.

##  2 Background

###  2.1 Visual Object Tracking

Visual object tracking aims to estimate the coordinates and scales of a
specific target throughout the given video. In general, tracking methods can
be divided into two types according to used information: (1) single-modal
tracking and (2) multi-modal tracking. Single-modal tracking locates the
target captured by a single sensor, such as laser, visible and infrared
cameras, to name a few. In previous years, tracking with RGB image, being
computationally efficient, easily accessible and high-quality, became
increasingly popular. Numerous methods have been proposed to improve tracking
accuracy and speed.

In RGB tracking, several frameworks including Kalman filter (KF) [12, 13],
particle filter (PF) [14, 15], sparse learning (SL) [16, 17], correlation
filter (CF) [18, 19], and CNN [20, 21] have been involved to improve the
tracking accuracy and speed. In 2010, Bolme _et al._ [18] proposed a CF-based
method called MOSSE, which achieves high-speed tracking with reasonable
performance. Thereafter, many researchers have aimed to develop the CF
framework to achieve state-of-the-art performance. Li _et al._ [19] achieve
scale estimation and multiple feature integration on the CF framework. Martin
_et al._ [22] eliminate the boundary effect by adding a spatial regularization
to the learned filter at the cost of speed decrease. Galoogahi _et al._ [23]
provide another efficient solution to solve the boundary effect, thereby
maintaining a real-time speed. Another popular framework is Siamese-based
network, which is first introduced by Bertinetto _et al._ [20]. Then, deeper
and wider networks are utilized to improve target representation. Zhang _et
al._ [21] find that the padding operation in the deeper network induces a
position bias, suppressing the capability of powerful network. They address
the position bias problem, and improve the tracking performance significantly.
Some methods perform better scale estimation by predicting segmentation masks
rather than bounding boxes [24, 25]. Above all, many efforts have been
conducted in this field. However, target appearance, as the main cue from
visible images, is not reliable for tracking when target suffers extreme
scenarios including low illumination, out-of-view and heavy occlusion. To this
end, more complementary cues are added to handle these challenges. A visible
camera is assisted by other sensors, such as laser [26], depth [7], thermal
[10], radar [27], and audio [28], to satisfy different requirements.

Since 2005, series of methods have been proposed using various multi-modal
information. Song _et al._ [26] conduct multiple object tracking by using
visible and laser data. Kim _et al._ [27] exploit the traditional Kalman
filter method for multiple object tracking with radar and visible images.
Megherbi _et al._ [28] propose a tracking method by combining vision and audio
information using belief theory. In particular, tracking with RGB-D and RGB-T
data has been the focus of attention using a portable and affordable binocular
camera. Thermal data can provide a powerful supplement to RGB images in some
challenging scenes, including night, fog, and rainy. Besides, a depth map can
provide an additional constrain to avoid tracking failure caused by heavy
occlusion and model drift. Lan _et al._ [29] apply the sparse learning method
to RGB-T tracking, thereby removing the cross-modality discrepancy. Li _et
al._ [11] extend an RGB tracker to the RGB-T domain, which achieves promising
results. Zhang _et al._ [10] jointly model motion and appearance information
to achieve accurate and robust tracking. Kart _et al._ [7] introduce an
effective constraint using a depth map to guide model learning. Liu _et al._
[30] transform the target position to 3D coordinate using RGB and depth
images, and then perform tracking using the mean shift method.

Table 1: Summary of existing surveys in related fields. Index | Year | Reference | Area |  Description | Publication
---|---|---|---|---|---
1 | 2010 | [31] | Multi-modal fusion |  This paper provides a overview on how to fuse multimodal data. | MS
2 | 2016 | [32] | multimodal object tracking |  This paper provides a general review of both single-modal and multi-modal tracking methods. | AIR
3 | 2016 | [33] | RGB-D dataset |  This paper collects popular RGB-D datasets for different applications and provides an anaysis of the popularity and difficulty. | MTA
4 | 2017 | [34] | RGB-D multiple human tracking |  This paper surveys the existing multiple human tracking methods with RGB-D data into two aspects. | IET CV
5 | 2019 | [35] | Multimodal machine learning |  A general survey covers how to represent, translate and fuse multimodal data according to various tasks. | TPAMI
6 | 2019 | [36] | RGB-T Image Fusion |  This paper gives a detailed survey on existing methods and applications for RGB-T image fusion. | IF
7 | 2020 | [37] | RGB-T object tracking |  A survey of the existing RGB-T tracking methods. | IF

###  2.2 Previous Surveys and Reviews

As shown in Table 1, existing surveys are introduced, which are related to
multi-modal processing, such as, image fusion, object tracking, and multi-
modal machine learning. Some of them focus on specific multi-modal information
or single tasks. Cai _et al._ [33] collect the datasets captured by RGB-D
sensors, which are used in many different applications, such as object
recognition, scene classification, hand gesture recognition, 3D-simultaneous
localization and mapping, and pose estimation. Camplani _et al._[34] focus on
multiple human tracking with RGB-D data and conduct an in-depth review of
different aspects. A comprehensive and detailed survey by Ma _et al._ [36] is
presented to summarize the methods regarding RGB-T image fusion. Recently, a
survey on RGB-T object tracking [37] is presented, which analyzes various
RGB-T trackers and conducts quantitative analysis on several datasets.

Other surveys aim to give a general introduction on how to utilize and
represent multi-modal information among a series of tasks. Atrey _et al._ [31]
present a brief introduction on multi-modal fusion methods and analyze
different fusion types in 2010. Walia _et al._ [32] introduce a general survey
on tracking with multiple modality data in 2016. Baltrusaitis _et al._ [35]
provide a detailed review of the machine learning method using multi-modal
information.

Various differences and developments are observed among the most related works
[32, 37]. First, we aim to conduct a general survey on how to utilize multi-
modal information, especially RGB-D and RGB-T tracking, on visual object
tracking in a unified view. Furthermore, different from [32], we pay much
attention to the recent deep-learning-based methods, which have not been
proposed in 2016. Finally, compared with the literature [37] that only focuses
on RGB-T tracking, our study provides a more substantial and comprehensive
survey in a large scope, including RGB-D and RGB-T tracking.

##  3 Multi-modal Visual Tracking

This section provides an overview of multi-modal tracking from three aspects:
(1) auxiliary modality purpose: how to utilize the information of auxiliary
modality to improve tracking performance; (2) tracking framework: the types of
framework that trackers belong to. Note that, in this study, we mainly focus
on visible-thermal (RGB-T), visible-depth (RGB-D) tracking, and we consider
visible modality as the main modality and other sources (i.e. thermal and
depth) as auxiliary modalities. The taxonomic structure is shown in Figure 1.

Figure 1: Structure of three classification methods and algorithms in each
category.

Multimodal Tracking

Auxiliary Modality PurposeTracking FrameworkDatasetFeature LearningPre-
processing [38, 39, 30, 40, 41, 7]Post-
processingGenerativeDiscriminativePublic DatasetsChallenges[5, 6] EF [42, 43,
44, 45, 46, 47, 48, 49, 50, 51, 52, 53]
---
[54, 55, 29, 56, 57, 58, 59, 2, 60, 3, 61, 62]
[63, 64, 1, 65, 66, 11, 67, 68, 69, 70, 71, 72]
LF[73, 74, 75, 4, 76, 77, 8, 78, 10]SL [51, 57, 29, 56, 58, 60, 61, 63, 64, 1,
2]MS[39, 30, 77]Others[53, 73] CF [42, 43, 46, 7, 47, 48, 49, 50]
---
[4, 76, 78, 79, 55, 68, 65, 9, 80, 10]
DL[40, 66, 11, 67, 69, 70, 71, 72, 81, 62]PF[38, 45, 51, 52, 54, 8,
64]Others[82, 74, 75, 41, 59, 44, 83]OR[43, 82, 44, 46, 47, 48, 30, 51, 74,
79, 76, 41, 9, 80, 10]SE [43, 46, 49, 79, 80]RGB-D[45, 84, 85, 86, 87,
4]RGB-T[88, 89, 3, 1, 2]

###  3.1 Auxiliary Modality Purpose

We first discuss the auxiliary modality purpose in multi-modal tracking. There
are three main categories: (a) feature learning, where the feature
representations of auxiliary modality image are extracted to help locate the
target; (b) pre-processing, where the information from auxiliary modality is
used before the target modeling; and (c) post-processing, where the
information from auxiliary modality aims to improve the model or refine the
bounding box.

####  3.1.1 Feature Learning

Methods based on feature learning extract information from auxiliary modality
through various feature methods, and then adopt modality fusion to combine the
data from different sources. Feature learning is an explicit way to utilize
multi-modal information, and most of corresponding methods consider the image
of auxiliary modality as an extra channel of the model. According to different
fusion methods, as shown in Figure 2, it can be further categorized as methods
based on early fusion (EF) and late fusion (LF) [31, 90]. EF-based methods
combine multi-modal information in the feature level using concatenation and
summation approaches; while LF-based methods model each modality individually
and obtain the final result by considering both decisions of modalities.

Figure 2: Workflows of early fusion (EF) and late fusion (LF). EF-based
methods conduct feature fusion and model them jointly; while LF-based methods
aim to model each modality individually and then combine their decisions.

Early Fusion (EF)

. In EF-based methods, the features extracted from both modalities are first
aggregated as a larger feature vector and then sent to the model to locate the
target. The workflow of EF-based trackers is shown in the left part of Figure
2. For most of the trackers, EF is the primary choice in the multi-modal
tracking task, while visible and auxiliary modalities are treated alike with
the same feature extraction methods. Camplani _et al._ [43] utilize HOG
feature for both visible and depth maps. Kart _et al._ [47] extract multiple
features to build a robust tracker for RGB-D tracking. Similar methods exist
in [44, 48, 49, 42, 54, 56, 58, 2, 60, 3]. However, auxiliary modality often
indicates different information against the visible map. For example, thermal
and depth images contain temperature and depth data, respectively. The
aforementioned trackers apply feature fusion, ignoring the modality
discrepancy, which decreases the tracking accuracy and causes the tracker to
drift easily. To this end, some trackers differentiate the heterogeneous
modalities by applying different feature methods. In [45], the gradient
feature is extracted in a depth map, while the average color feature is used
to represent the target in the visible modality. Meshgi _et al._ [52] use the
raw depth information and many feature methods (HOG, LBP, and LoG) for RGB
images. In [29, 57, 64], the HOG and intensity features are used for visible
and thermal modalities, respectively. Due to the increasing cost involved in
feature concatenation and the misalignment of multi-modal data, some methods
tune the feature representation after feature extraction by the pruning [67]
or re-weighting operation [50, 72], which can compress the feature space and
exploit the cross-modal correlation. In DAFNet [67], a feature pruning module
is proposed to eliminate noisy and redundant information. Liu _et al._ [50]
introduce a spatial weight to highlight the foreground area. Zhu et al. [72]
exploit modality importance using the proposed multi-modal aggregation
network.

Late fusion (LF)

. LF-based methods process both modalities simultaneously and the independent
models for each modality are built to make decisions. Then, the decisions are
combined by using weighted summation [78, 74, 4, 76], calculating joint
distribution function [73, 8, 77], and conducting multi-step localization
[75]. Conaire _et al._ [73] assume the independence between multi-modal data,
and then obtain the result by multiplying the target’s likelihoods in both
modalities. A similar method is adopted in literature [77]. Xiao _et al._ [4]
fuse two single-modal trackers via an adaptive weight map. In MCBT [75], data
from multiple sources are used stepwise to locate the target. A rough target
position is first estimated by optical flow in the visible domain, and the
final result is determined by part-based matching method with RGB-D data.

####  3.1.2 Pre-Processing

Due to the available depth map, the second purpose of auxiliary modality is to
transform the target into 3D space before target modeling via RGB-D data.
Instead of tracking in the image plane, these types of methods model the
target in the world coordinate, and 3D trackers are designed [38, 39, 7, 30,
40, 41]. Liu _et al._ [30] extend the classical mean shift tracker to 3D
extension. In OTR [7], the dynamic spatial constraint generated by the 3D
target model enhances the discrimination of DCF trackers in dealing with out-
of-view rotation and heavy occlusion. Although a significant performance is
achieved, the computation cost of 3D reconstruction cannot be neglected.
Furthermore, the performance is highly subject to the quality of depth data
and the accessibility of mapping functions between the 2D and 3D spaces.

####  3.1.3 Post-processing

Compared with the RGB image that brings more detailed content, the depth image
highlights the contour of objects, which can segment the target among
surroundings via depth variance. Inspired by the nature of depth map, many
RGB-D trackers utilize the depth information to determine whether the
occlusion occurs and estimate the target scale [43, 46, 49, 79].

Occlusion Reasoning (OR)

. Occlusion is a traditional challenge in the tracking task because the
dramatic appearance variation leads the model drifting. Depth cue is a
powerful feature to detect target occlusion; thus, the tracker can apply a
global search strategy or model updating mechanism to avoid learning from the
occluded target. In [43], occlusion is detected when the depth variance is
large. Then, tracker enlarges the search region to detect the re-appeared
target. Ding _et al._ [44] propose an occlusion recovery method, where a depth
histogram is recorded to examine whether the occlusion occurs. If the
occlusion is detected, the tracker locates the occluder and searches the
candidate around. In [10], Zhang _et al._ propose a tracker switcher to detect
occlusion based on the template matching method and tracking reliability. The
tracker can dynamically select which information is used for tracking between
appearance and motion cues, thereby improving the robustness of the tracker
significantly.

Scale Estimation (SE)

. SE is an important module in tracking task, which can obtain a tight
bounding box and avoid drift. CF-based trackers estimate the target scale by
sampling the search region in multiple resolutions [91], learning a filter for
scale estimation [92], which cannot effectively adapt to the target’s scale
change [49]. Both thermal and depth maps provide clear contour information and
a coarse pixel-wise target segmentation map. With such information, the target
shape can be effectively estimated. In [46], the number of scales is
adaptively changed to fit the scale variation. SEOH [49] uses space
continuity-of-depth information to achieve accurate scale estimation with
minor time cost. The pixels belonging to the target are clustered by the
K-means method in the depth map, and the sizes of the target and search
regions are determined by clustering result.

###  3.2 Tracking Framework

In this section, multi-modal trackers are categorized based on the methods
used in target modeling, including generative and discriminative. The
generative framework focuses on directly modeling the representation of the
target. During tracking, the target is captured by matching the data
distribution in the incoming frame. However, generative methods only learn the
representations for the foreground information while ignoring the influence of
surroundings, suffering from background cluttering or distractions [93]. In
comparison, the discriminative models construct an effective classifier to
distinguish the object against the surroundings. The tracker outputs the
confidence score of sampled candidates and chooses the best matching patch as
the target. Various patch sample manners are exploited, e.g. sliding window
[50], particle filter [38, 45], and Gaussian sampling [11]. Furthermore, a
crucial task is utilizing powerful features to represent the target. Thanks to
the emerging convolution networks, more trackers have been built via efficient
CNNs. We will introduce the various frameworks in the following paragraphs.

####  3.2.1 Generative Methods

Figure 3: Framework of OAPF [52]. The particle filter method is applied, with
occlusion handling, in which the occlusion model is constructed against the
template model. When the target is occluded, the occlusion model is used to
predict the position without the updating template model.

Sparse Learning (SL)

. SL has been popular in many tasks including image recognition [94] and
classification [95], object tracking [96], and others. In SL-based RGB-T
trackers, the tracking task can be formulated as a minimization problem for
the reconstruction error with the learned sparse dictionary [57, 29, 56, 58,
60, 63, 64, 1]. Lan _et al._ [29] propose a unified learning paradigm to learn
the target representation, modality-wise reliability and classifier,
collaboratively. Similar methods are also applied in the RGB-D tracking task.
Ma _et al._ [51] construct an augmented dictionary consisting of target and
occlusion templates, which achieves accurate tracking even in heavy occlusion.
SL-based trackers achieve promising results at the expense of computation
cost. These trackers cannot meet the requirements of real-time tracking.

Mean Shift (MS)

. MS-based methods maximize the similarity between the histograms of
candidates and the target template, and conduct fast local search using the
mean shift technique. These methods usually assume that the object overlaps
itself in consecutive frames [77]. In [39, 30], the authors extend the 2D MS
method to 3D with RGB-D data. Conaire _et al._ [77] propose an MS tracker
using spatiogram instead of histogram. Compared with discriminative methods,
MS-based trackers directly regress the offset of the target, which omits dense
sampling. These methods with lightweight features can achieve real-time
performance, whereas the performance advantage is not obvious.

Other Frameworks

. Other generative methods have been applied to tracking tasks. Coraire _et
al._ [73] model the tracked object via Gaussian distribution and select the
best-matched patch via similarity measure. Chen _et al._ [53] model the
statistics of each individual modality and the relationship between RGB and
thermal data using the expectation maximization algorithm. These methods can
model individual or complementary modalities, thereby achieving a flexible
framework for different scenes.

####  3.2.2 Discriminative Methods

Figure 4: Workflow of the JMMAC [10]. The CF-based tracker is used to model
appearance cue, while both camera and target motion are considered, thereby
achieving substantial performance.

Particle Filter (PF)

. The PF framework is a Bayesian sequential importance sampling technique
[97]. It consists of two steps, i.e., prediction and updating. In the
prediction step, given the state observations ${\bf z}_{1:t}=\\{{\bf
z}_{1},{\bf z}_{2},...,{\bf z}_{t}\\}$ during the previous $t$ frames, the
posterior distribution of the state $\mathbf{x}_{t}$ is predicted using
Bayesian rule as follows:

$p\left(\mathbf{x}_{t}\mid\mathbf{z}_{1:t}\right)=\frac{p\left(\mathbf{z}_{t}\mid\mathbf{x}_{t}\right)p\left(\mathbf{x}_{t}\mid\mathbf{z}_{1:t-1}\right)}{p\left(\mathbf{z}_{t}\mid\mathbf{z}_{1:t-1}\right)},$
(1)

where $p\left(\mathbf{x}_{t}\mid\mathbf{z}_{1:t-1}\right)$ is estimated by a
set of $N$ particles. Each particle has a weight, $w_{t}^{i}$. In the updating
process, $w_{t}^{i}$ is updated as

$w_{t}^{i}\propto p\left({\bf z}_{t}\mid{\bf x}_{t}={\bf x}_{t}^{i}\right).$
(2)

In the PF framework, the restrictions of linearity and Gaussianity imposed by
Kalman filter are relaxed, thereby leading accurate and robust tracking [8].
Several works improve the PF method for multi-modal tracking task. Bibi _et
al._ [38] formulate the PF framework in 3D, which considers both
representation and motion models and propose a particle pruning method to
boost the tracking speed. Meshgi _et al._ [52] consider occlusion in
approximation step to improve PF in occlusion handling. Liu _et al._ [64]
propose a new likelihood function for PF to determine the goodness of
particles, thereby promoting the performance.

Figure 5: Framework of MANet [11]. Generic adapter (GA) is used to extract
common information of RGB-T images. Modality adapter (MA) aims to exploit the
different properties of heterogeneous modalities. Finally, instance adapter
(IA) models the appearance properties and temporal variations of a certain
object.

Correlation Filter (CF)

. CF-based tracker learns the discriminative template denoted as CF to
represent the target. Then, the online learned filter is used to detect the
object in the next frame. As the circular convolution can be accelerated in
Fourier domain, these trackers can maintain approving accuracy with high
speed. In recent years, many CF-based variants are proposed, such as adding
spatial regularization [98], introducing temporal constraint [99], and
equipping discriminative features [100], to increase the tracking performance.
Due to the advantage of CF-based trackers, many researchers aim to build
multi-modal trackers with the CF framework. Zhai  _et al._ [65] introduce low-
rank constraint to learn the filters of both modalities collaboratively,
thereby exploiting the relationship between RGB and thermal data. Hannuna  _et
al._ [46] effectively handle the scale change with the guidance of the depth
map. Kart  _et al._ propose a long-term RGB-D tracker [7], which is designed
based on CSRDCF [101] and applies online 3D target reconstruction to
facilitate learning robust filters. The spatial constraint is learned from the
3D model of the target. When the target is occluded, view-specific DCFs are
used to robustly localize the target. Camplani  _et al._ [43] improve the CF
method in scale estimation and occlusion handling, while maintaining a real-
time speed.

Deep Leraning (DL)

. Due to the discriminative ability in feature representation, CNN is widely
used in the tracking task. Various networks provide a powerful alternative to
the traditional hand-crafted feature, which is the simplest way to utilize
CNN. Liu _et al._ [50] extract the deep features from VGGNet [102] and hand-
crafted features to learn a robust representation. Li _et al._[68] concatenate
deep features from visible and thermal images, and then adaptively fuse them
using the proposed FusionNet to achieve robust feature representation.
Furthermore, some methods aim to learn an end-to-end network for multi-modal
tracking. In [11, 67, 69], a similar framework borrowed from MDNet [103] is
applied for tracking with different structures to fuse the cross-modal data.
These trackers achieve obvious performance promotion while the speed is poor.
Zhang _et al._ [71] propose an end-to-end RGB-T tracking framework with real-
time speed and balanced accuracy. They apply ResNet [104] as the feature
extractor and fuse RGB and thermal information in the feature level, which are
used for target localization and box estimation.

Other Frameworks

. Some methods use an explicit template matching method to localize the
object. These methods find the best-matched candidate with the target captured
in frames through a pre-defined matching function [75, 41]. Ding _et al._ [44]
learn a Bayesian classifier and consider the candidate with maximal score as
the target location, which can reduce the model drift. In [83], a structured
SVM [105] is learned by maximizing a classification score, which can prevent
the labeling ambiguity in the training process.

##  4 Datasets

With the emergence of multi-modal tracking methods, several datasets and
challenges for RGB-D and RGB-T tracking are released. We summarize the
available datasets in Table 2.

Table 2: Summary of multi-modal tracking datasets. | Name | Seq. Num. | Total Frames | Min. Frame | Max. Frame | Attr. | Resolution | Metrics | Year
---|---|---|---|---|---|---|---|---|---
RGB-D |  PTB [86] | 100 | 21.5K | 0.04K | 0.90K | 11 |  640 $\times$ 480 | CPE, SR | 2013
STC [4] | 36 | 18.4K | 0.13K | 0.72K | 10 |  640 $\times$ 480 | SR, Acc., Fail. | 2018
CTDB [87] | 80 | 101.9K | 0.4K | 2.5K | 13 |  640 $\times$ 360 | F-score, Pr, Re | 2019
RGB-T |  OTCBVS [88] | 6 | 7.2K | 0.6K | 2.3K | – |  320 $\times$ 240 | – | 2007
LITIV [89] | 9 | 6.3K | 0.3K | 1.2K | – |  320 $\times$ 240 | – | 2012
GTOT [1] | 50 | 7.8K | 0.04K | 0.37K | 7 |  384 $\times$ 288 | SR, PR | 2016
RGBT210 [2] | 210 | 104.7K | 0.04K | 4.1K | 12 |  630 $\times$ 460 | SR, PR | 2017
RGBT234 [3] | 234 | 116.6K | 0.04K | 8.1K | 12 |  630 $\times$ 460 | SR, PR, EAO | 2019
VOT-RGBT [5, 6] | 60 | 20.0K | 0.04K | 1.3K | 5 |  630 $\times$ 460 | EAO | 2019

###  4.1 Public dataset

####  4.1.1 RGB-D dataset

In 2012, a small-scale dataset called BoBoT-D [45] is constructed, consisting
of five RGB-D video sequences captured by Kinect V1 sensor. Both overlap and
hit rate are used for evaluation, which indicate the mean overlap between
result and ground truth and percentage of frame where the overlap is larger
than 0.33. Song _et al._ [86] propose the well-known Princeton tracking
benchmark (PTB) of 100 high-diversity RGB-D videos, five of which are used for
validation and others without available ground truth are used for testing. The
PTB dataset contains 11 annotations, which are separated by 5 categories
including target type, target size, movement, occlusion, and motion type. Two
metrics are conducted to evaluate the tracking performance: center position
error (CPE) and success rate (SR). CPE measures the Euclidean distance between
centers of result and ground truth and SR is the average intersection over
union (IoU) during all frames, which is defined as

$SR=\frac{1}{N}\sum_{i=1}^{N}u_{i}\ \ \ \ \ \
u_{i}=\left\\{\begin{matrix}1&IoU(bb_{i},gt_{i})>t_{sr}\\\
0&otherwise\end{matrix}\right.,$ (3)

where the $IoU(\cdot,\cdot)$ denotes the IoU between the bounding box $bb_{i}$
and ground truth $gt_{i}$ in the $i$-th frame. If the IoU is larger than the
threshold $t_{sr}$, we consider the target to be successfully tracked. The
final rank of the tracker is determined by the Avg. Rank, which is defined as
the average ranking of SR in each attribute. The STC dataset [4] consists of
36 RGB-D sequences and covers some extreme tracking circumstances, such as
outdoor and night scenes. This dataset is captured by still and moving ASUS
Xtion RGB-D cameras to evaluate the tracking performance under conditions of
arbitrary camera motion. A total of 10 attributes are labeled to thoroughly
analyze the dataset bias. _The detailed introduction of each attributes are
shown in the supplementary file._

Figure 6: Examples in RGB-D tracking datasets (PTB and STC datasets).

The trackers are measured by using both SR and VOT protocols. The VOT protocol
evaluates the tracking performance in terms of two aspects: accuracy and
failure. Accuracy (Acc.) considers the IoU between the ground truth and
bounding box, and failure (Fail.) measures the times when the overlap is zero
and the tracker is set to re-initialize using the ground truth and continues
to track. CTDB [87] is the latest RGB-D tracking dataset, which contains 80
short-term and long-term videos. The target is out-of-view and occluded
frequently, which needs the tracker to handle both tracking and re-detection
cases. The metrics are Precision (Pr.), Recall (Re.) and the overall F-score
[106]. The precision and recall are defined as follows,

$Pr=\frac{\sum_{i=1}^{N}u_{i}}{\sum_{i=1}^{N}s_{i}}\ \ \
u_{t}=\left\\{\begin{matrix}1&bb_{i}\ \ exists\\\
0&otherwise\end{matrix}\right.,$ (4)

$Re=\frac{\sum_{i=1}^{N}u_{i}}{\sum_{i=1}^{N}g_{i}}\ \ \
g_{t}=\left\\{\begin{matrix}1&gt_{i}\ \ exists\\\
0&otherwise\end{matrix}\right.,$ (5)

where $u_{i}$ is defined in Eq. 3. The F-score combines both precision and
recall through

$F-score=\frac{2Pr\times Re}{Pr+Re}.$ (6)

####  4.1.2 RGB-T Dataset

In previous years, two RGB-T people detection datasets are used for tracking.
The OTCBVS dataset [88] has six grayscale-thermal video clips captured from
two outdoor scenes. The LITIV dataset [89] contains nine sequences,
considering the illumination influence and being captured indoors. These
datasets with limited sequences and low diversity have been depreciated. In
2016, Li _et al._ construct the GTOT dataset for RGB-T tracking, which
consists of 50 grayscale-thermal sequences under different scenarios and
conditions. A new attribute for RGB-T tracking is labeled as thermal crossover
(TC), which indicates that the target has similar temperature with the
background. Inspired by [107, 108], GTOT adopts success rate (SR) and
precision rate (PR) for evaluation. PR denotes the percentage of frames whose
CPE is smaller than a threshold $t_{pr}$, which is set to 5 in GTOT to
evaluate small targets. Li _et al._ [2] propose a large-scale RGB-T tracking
dataset, namely RGBT210, which contains 210 videos and 104.7k image pairs.
This dataset also extends the number of attributes to 12. _The detailed
description of attributes can be found in supplementary file._ The metric is
the same as GTOT, except $t_{pr}$ is set to 20 normally. In 2019, the
researchers enlarge the RGBT210 dataset and propose RGBT234 [3], which
provides individual ground truth for each modality. Furthermore, besides SR
and PR, expected average overlap (EAO) is used for evaluation, combining the
accuracy and failures in a principled manner.

Figure 7: Examples and corresponding attributes in GTOT and RGBT234 tracking
datasets.

###  4.2 Challenges for Multi-modal Tracking

Since 2019, both RGB-D and RGB-T challenges have been held by VOT Committee
[6, 5]. For RGB-D challenge, trackers are evaluated on the CDTB dataset [87]
with the same evaluation metrics. All the sequences are annotated on the basis
of 5 attributes, namely, occlusion, dynamics change, motion change, size
change, and camera motion. RGB-T challenge constructs the dataset as a subset
of RGBT234 with slight change in ground truth, which consists of 60 RGB-T
public videos and 60 sequestered videos. Compared with RGBT234, VOT-RGBT
utilizes different evaluation metrics, i.e., EAO, to measure trackers. In
VOT2019-RGBT, trackers need to be re-initialized, when tracking failure is
detected (the overlap between bounding box and ground truth is zero). Besides,
VOT2020-RGBT introduces a new anchor mechanism to avoid a causal correlation
between the first reset and the later ones [5] instead of the re-
initialization mechanism.

##  5 Experiments

In this section, we conduct analysis on both public datasets and challenges
from the overall comparison, attribute-based comparison, and speed. For fair
comparison on speed, we refer to the device used (CPU or GPU), platform used
(M: Matlab, MCN: Matconvnet, P: Python, and PT: PyTorch), and setting
(detailed information on CPU and GPU). _The available codes and detailed
description of trackers are collected and listed in the supplementary files._

###  5.1 Experimental Comparison on RGB-D Datasets

Table 3: Experimental results on the PTB dataset. Numbers in parentheses indicate their ranks. The top three results are in red, blue, and green fonts. Algorithm |  Avg. Rank | Target type | Target size | Movement | Occlusion | Motion type
---|---|---|---|---|---|---
Human | Animal | Rigid | Large | Small | Slow | Fast | Yes | No | Passive | Active
OTR [7] | 2.91 | 77.3(3) | 68.3(6) | 81.3(3) | 76.5(4) | 77.3(1) | 81.2(2) | 75.3(1) | 71.3(2) | 84.7(6) | 85.1(1) | 73.9(3)
WCO [50] | 3.91 | 78.0(2) | 67.0(7) | 80.0(4) | 76.0(5) | 75.0(2) | 78.0(7) | 73.0(5) | 66.0(6) | 86.0(2) | 85.0(2) | 82.0(1)
TACF [48] | 4.64 | 76.9(5) | 64.7(9) | 79.5(5) | 77.2(3) | 74.0(4) | 78.5(5) | 74.1(3) | 68.3(5) | 85.1(3) | 83.6(4) | 72.3(5)
CA3DMS [30] | 6 | 66.3(12) | 74.3(2) | 82.0(1) | 73.0(10) | 74.2(3) | 79.6(3) | 71.4(8) | 63.2(13) | 88.1(1) | 82.8(5) | 70.3(8)
CSR-rgbd [9] | 6.36 | 76.6(6) | 65.2(8) | 75.9(9) | 75.4(6) | 73.0(6) | 79.6(3) | 71.8(6) | 70.1(3) | 79.4(10) | 79.1(7) | 72.1(6)
3DT [38] | 6.55 | 81.4(1) | 64.2(11) | 73.3(11) | 79.9(2) | 71.2(9) | 75.1(12) | 74.9(2) | 72.5(1) | 78.3(11) | 79.0(8) | 73.5(4)
DLST [42] | 7 | 77.0(4) | 69.0(5) | 73.0(12) | 80.0(1) | 70.0(12) | 73.0(13) | 74.0(4) | 66.0(6) | 85.0(5) | 72.0(13) | 75.0(2)
OAPF [52] | 7.27 | 64.2(13) | 84.8(1) | 77.2(6) | 72.7(11) | 73.4(5) | 85.1(1) | 68.4(12) | 64.4(11) | 85.1(3) | 77.7(10) | 71.4(7)
CCF [80] | 7.55 | 69.7(10) | 64.5(10) | 81.4(2) | 73.1(9) | 72.9(7) | 78.4(6) | 70.8(9) | 65.2(8) | 83.7(7) | 84.4(3) | 68.7(12)
OTOD [40] | 8.91 | 72.0(8) | 71.0(3) | 73.0(12) | 74.0(7) | 71.0(10) | 76.0(9) | 70.0(11) | 65.0(9) | 82.0(8) | 77.0(12) | 70.0(9)
DMKCF [47] | 9 | 76.0(7) | 58.0(13) | 76.7(7) | 72.4(12) | 72.8(8) | 75.2(11) | 71.6(7) | 69.1(4) | 77.5(13) | 82.5(6) | 68.9(11)
DSKCF [46] | 9.09 | 70.9(9) | 70.8(4) | 73.6(10) | 73.9(8) | 70.3(11) | 76.2(8) | 70.1(10) | 64.9(10) | 81.4(9) | 77.4(11) | 69.8(10)
DSOH [43] | 11.45 | 67.0(11) | 61.2(12) | 76.4(8) | 68.8(13) | 69.7(13) | 75.4(10) | 66.9(13) | 63.3(12) | 77.6(12) | 78.8(9) | 65.7(13)
DOHR [44] | 14 | 45.0(14) | 49.0(14) | 42.0(14) | 48.0(14) | 42.0(14) | 50.0(14) | 43.0(14) | 38.0(14) | 54.0(14) | 54.0(14) | 41.0(14)

Overall Comparison

. PTB provides a website111 http://tracking.cs.princeton.edu/ for
comprehensive evaluating RGB and RGB-D methods in an online manner. We collect
the results of 14 RGB-D trackers on the website and sort them based on rank.
The results are shown in Table 3. We list the Avg. Rank, SR and corresponding
rank of each attribute. The Avg. Rank is calculated by averaging the rankings
of all attributes. According to Table 3, OTR achieves the best performance
among all the competitors, which is based on the CF framework without deep
features. Reason for the promising result is that the 3D construction provides
a useful constraint for filter learning. The same conclusion can be obtained
by CA3DMS and 3DT, which construct a 3D model to locate the target via mean-
shift and sparse learning methods. These trackers with traditional features
are competitive to the deep trackers. DL-based trackers (WCO, TACF, and CSR-
RGBD) achieve substantial performance, which indicates the discrimination of
deep features. CF-based trackers achieve various results and are the most
widely-applied framework. Trackers based on original CF methods (DMKCF, DSKCF
and DSOH) perform significantly worse than those developed on improved CF
(OTR, WCO and TACF). OTOD based on point cloud does not exploit the
effectiveness of CNN and obtains the 10th rank on the PTB dataset.

Attribute-based Comparison

. PTB provided 11 attributes from five aspects for comparison. CF-based
trackers, including OTR, WCO, TACF, CSR-RGBD, and CCF, are not well-performed
on tracking animals. As animals move fast and irregularly, these trackers with
online learning are easy to drift. While the target is in small size, CF can
provide precise tracking results. The occlusion handling mechanism contributes
greatly to videos with target occlusion. The 3D mean shift method shows
obvious advantage in tracking targets with rigid shape and no occlusion. OAPF
obtains an above-average performance on tracking small objects, thereby
indicating the effectiveness of the scale estimation strategy.

Speed Analysis

. The speed report of RGB-D trackers are listed in Table 4. Most of the
trackers cannot meet the requirements of real-time tracking. Trackers based on
the improved CF framework (OTR [7], DMKCF [47], CCF [80], WCO [50], and TACF
[48]), are subject to their speed. Only two real-time trackers (DSKCF [46] and
DSOH [43]) are based on the original CF architecture.

Table 4: The speed analysis of RGB-D trackers. Trackers | Speed | Device | Platform | Setting
---|---|---|---|---
OTR [7] | 2.0 | CPU | Matlab |<EMAIL_ADDRESS>
WCO [50] | 9.5 | GPU | M & MCN |<EMAIL_ADDRESS>GTX TITAN
TACF [48] | 13.1 | GPU | M & MCN |<EMAIL_ADDRESS>
CA3DMS [30] | 63 | CPU | C++ |<EMAIL_ADDRESS>
DLST [42] | 4.8 | CPU | M | I5-<EMAIL_ADDRESS>
OAPF [52] | 0.9 | CPU | M | –
CCF [80] | 6.3 | CPU | M |<EMAIL_ADDRESS>
DMKCF [47] | 8.3 | CPU | M |<EMAIL_ADDRESS>
DSKCF [46] | 40 | CPU | M & C++ |<EMAIL_ADDRESS>
DSOH [43] | 40 | CPU | M |<EMAIL_ADDRESS>

###  5.2 Experimental Comparison of RGB-T Datasets

We select 14 trackers as our baseline to perform overall comparison of the
GTOT and RGBT234 datasets. As only part of the trackers (JMMAC, MANet, mfDiMP)
release their codes, We run these trackers on these two datasets and record
the performance of other trackers in their original papers. The overall
results are shown in Table 5.

Table 5: Experimental results on the GTOT and RGBT234 datasets. Tracker | GTOT | RGBT234 | Speed | Device | Platform | Setting
---|---|---|---|---|---|---
SR | PR | SR | PR
CMPP [81] | 73.8 | 92.6 | 57.5 | 82.3 | 1.3 | GPU | PyTorch | RTX 2080Ti
JMMAC [10] | 73.2 | 90.1 | 57.3 | 79.0 | 4.0 | GPU | MatConvNet | RTX 2080Ti
CAT [121] | 71.7 | 88.9 | 56.1 | 80.4 | 20.0 | GPU | PyTorch | GTX 2080Ti
MaCNet [62] | 71.4 | 88.0 | 55.4 | 79.0 | 0.8 | GPU | PyTorch | GTX 1080Ti
TODA [69] | 67.7 | 84.3 | 54.5 | 78.7 | 0.3 | GPU | PyTorch | GTX1080Ti
DAFNet [66] | 71.2 | 89.1 | 54.4 | 79.6 | 23.0 | GPU | PyTorch | RTX 2080Ti
MANet [11] | 72.4 | 89.4 | 53.9 | 77.7 | 1.1 | GPU | PyTorch | GTX1080Ti
DAPNet [67] | 70.7 | 88.2 | 53.7 | 76.6 | – | GPU | PyTorch | GTX1080Ti
FANet [72] | 69.8 | 88.5 | 53.2 | 76.4 | 1.3 | GPU | PyTorch | GTX1080Ti
CMR [59] | 64.3 | 82.7 | 48.6 | 71.1 | 8.0 | CPU | C++ | –
SGT [2] | 62.8 | 85.1 | 47.2 | 72.0 | 5.0 | CPU | C++ | –
mfDiMP [71] | 49.0 | 59.4 | 42.8 | 64.6 | 18.6 | GPU | PyTorch | RTX 2080Ti
CSR [1] | – | – | 32.8 | 46.3 | 1.6 | CPU | Matlab & C++ | –
L1-PF [54] | 42.7 | 55.1 | 28.7 | 43.1 | – | – | – | –
JSR [64] | 43.0 | 55.3 | 23.4 | 34.3 | 0.8 | CPU | Matlab | –

Overall Comparison

. All the high-performance trackers are equipped with the learned deep
features and most of them are based on MDNet variants (CMPP, MaCNet, TODA,
DAFNet, MANet, DAPNet, and FANet), which achieve satisfactory results. The CF-
based tracker (JMMAC) obtains the second rank on GTOT and RGBT234 datasets,
which combines appearance and motion cues. Compared with CF trackers, MDNet-
based trackers can provide precise target position with higher PR, but are
inferior to CF framework in scale estimation, reflecting on SR. Trackers with
sparse learning technique (CSR, SGT) are better than L1-PF based on the PF
method. Although mfDiMP utilizes the state-of-the-art backbone, the
performance is not positive. The main reason may be that mfDiMP utilizes
different training data, which are generated by image translation methods
[120] and brings a gap between existing real RGB-T data.

Attribute-based Comparison

. We conduct attribute-based comparison on RGBT234, as shown in Figure 8.
Improved MDNet-based trackers achieve satisfactory performance on low-
resolution, deformation, background clutter, fast motion, and thermal
crossover. Since modeling both camera motion and target motion, JMMAC has
strength in camera movement and partial occlusion, but fails easily in fast-
moving targets. This condition may result from CF-based trackers having a
fixed search region. When the target moves outside the region, the target
cannot be detected, thereby causing tracking failure. CMPP, which exploits the
inter-modal and cross-modal correlation, have great promotion on low
illumination, low resolution, and thermal crossover. Targets in these
attributes have unavailable modality and CMPP can eliminate the gap between
heterogeneous modalities. _The detailed figure on attribute-based comparison
can be found in supplementary file._

Speed Analysis

. For tracking speed, we list the platform and setting for fair comparison in
Table 5. DAFNet based on a real-time MDNet variant achieves fast tracking with
23.0 fps. Although mfDiMP is equipped with ResNet-101, it achieves the second
fastest tracker because most parts of the network are trained offline without
tuning online. Other trackers are constrained by their low speed, which cannot
be utilized in some real-time applications.

Figure 8: Attribute-based Comparison on RGBT234.

###  5.3 Challenge Results on VOT2019-RGBD

We list the challenge results in Table 6. Both original RGB trackers without
utilizing depth information and RGB-D trackers are merged for evaluation.
Trackers who obtain top three ranks on F-score, precision, and Recall, are
designed with the same component and framework. Unlike the PTB dataset, DL-
based methods have potential performance on VOT-RGBD19, which results from
these trackers utilizing large-scale visual datasets for offline training and
equipping deeper networks. For instance, the original RGB tracker with DL
framework also achieves excellent performance. Furthermore, occlusion handling
is another necessary component of the high-performance tracker because
VOT2019-RGBD focuses on long-term tracking with target frequent reappearance
and out-of-view and most of the trackers are equipped with a redetection
mechanism. The CF framework (FuCoLoT, OTR, CSR-RGBD, and ECO) does not perform
well, which may stem from the online updating using occlusion patches that
degrade the discriminability of the model.

Table 6: Challenge results on VOT2019-RGBD dataset. Tracker | Rank | Modality | Type | F-score | Precison | Recall
---|---|---|---|---|---|---
SiamDW-D [21] | 1 | RGB-D | OR, DL | 0.681 | 0.677 | 0.685
ATCAIS [6] | 2 | RGB-D | OR, DL | 0.676 | 0.643 | 0.712
LTDSE-D [6] | 3 | RGB-D | OR, DL | 0.658 | 0.674 | 0.643
SiamM-Ds [24] | 4 | RGB-D | SE, DL | 0.455 | 0.463 | 0.447
MDNet [103] | 5 | RGB | DL | 0.455 | 0.463 | 0.447
MBMD [6] | 6 | RGB | OR,DL | 0.441 | 0.454 | 0.429
FuCoLoT [122] | 7 | RGB | OR, CF | 0.391 | 0.459 | 0.340
OTR [7] | 8 | RGB-D | CF, EF, Pre | 0.336 | 0.364 | 0.312
SiamFC [20] | 9 | RGB | DL | 0.333 | 0.356 | 0.312
CSR-rgbd [9] | 10 | RGB-D | CF, EF, OR | 0.332 | 0.375 | 0.397
ECO [100] | 11 | RGB | CF | 0.329 | 0.317 | 0.342
CA3DMS [30] | 12 | RGB | MS, OR, Pre | 0.271 | 0.284 | 0.259

###  5.4 Challenge Results on VOT2019-RGBT

For VOT2019-RGBT dataset shown in Table 7, JMMAC with exploiting both
appearance and motion cues shows high accuracy and robust performance and
obtains the highest EAO in a large margin. Early fusion is the primary manner
of RGB-T fusion, while the late fusion method (JMMAC) has great potential in
improving tracking accuracy and robustness, which has not been fully utilized.
All top six trackers are equipped with CNN as feature extractor, thereby
indicating the powerful ability of CNN. SiamDW using a Siamese network is a
general method that performs well in both RGB-D and RGB-T tasks. ATOM variants
(mfDiMP and MPAT) are used to handle RGB-T tracking.

Table 7: Challenge results on the VOT2019-RGBT dataset. Tracker | Rank | Modality | Type | EAO | Accuracy | Robustness
---|---|---|---|---|---|---
JMMAC [10] | 1 | RGBT | CF, LF | 0.4826 | 0.6649 | 0.8211
SiamDW-T [21] | 2 | RGBT | DL, EF | 0.3925 | 0.6158 | 0.7839
mfDiMP [71] | 3 | RGBT | DL, EF | 0.3879 | 0.6019 | 0.8036
FSRPN [6] | 4 | RGBT | DL, EF | 0.3553 | 0.6362 | 0.7069
MANet [11] | 5 | RGBT | DL, EF | 0.3463 | 0.5823 | 0.7010
MPAT [6] | 6 | RGB | DL | 0.3180 | 0.5723 | 0.7242
CISRDCF [6] | 7 | RGBT | CF, EF | 0.2923 | 0.5215 | 0.6904
GESBTT [6] | 8 | RGBT | Lucas–Kanade | 0.2896 | 0.6163 | 0.6350

##  6 Further Prospects

###  6.1 Model Design

Multi-modal fusion

. Compared with tracking with unimodality data, multi-modal tracking can
easily exploit a powerful data fusion mechanism. Existing methods mainly focus
on feature fusion, whereas the effectiveness of other fusion types has not
been exploited. Compared with early fusion, late fusion eliminates the bias
that heterogeneous features may be learned from different modalities.
Furthermore, another advantage of late fusion is that we can utilize various
methods to model each modality independently. The hybrid fusion method
combining the early and late fusion strategies has been used in image
segmentation [123] and sports video analysis [124], which is also a better
choice for multi-modal tracking.

Specific Network for Auxiliary Modality

. As the gap of different modalities exists and the semantic information is
also heterogeneous, traditional methods use different features to extract more
useful data [57, 64, 45]. Although sufficient works on network structures for
visible image analysis have been conducted, the specific architecture for
depth and thermal maps has not been deeply explored. Thus, DL-based methods
[11, 66, 67, 71] trade the data in auxiliary modality as an additional
dimension of the RGB image with the same network architecture, e.g., VGGNet
and ResNet, and extract the feature in the same level (layer). A crucial task
is to design a network for processing multi-modal data. Since 2017, AutoML
method, especially neural architecture search (NAS), has been popular which
design the architecture automatically and obtain highly competitive results in
many areas, such as image classification [125], and recognition [126].
However, researchers pay less attention to NAS method for multi-modal
tracking, which is a good direction to explore.

Multi-modal Tracking with Real-time Speed

. The additional modality multiplies the computation, which causes difficulty
for the existing tracking frameworks to achieve the requirements of real-time
performance. A speed-up mechanism needs to be designed, such as feature
selection [67], knowledge distillation technology, and others. Furthermore,
Huang _et al._ [127] propose a trade-off method, where the agent decides which
layer is more suitable for accurate localization, thereby providing 100 times
speed boost.

###  6.2 Dataset Construction

Large-scale Dataset for Training

. With the emergence of deep neural networks, more powerful methods are
equipped with CNN to achieve accurate and robust performance. However, the
existing datasets focus on testing with no training subset. For instance, most
DL-based trackers use the GTOT dataset as the training set when testing
RGBT234, which has a small amount of data with limited scenes. The
effectiveness of DL-based methods has not been fully exploited. Zhang _et al._
[71] generate synthetic thermal data from the numerous existing visible
datasets by using the image translation method [120]. However, this data
augmentation does not bring significant performance improvement. Above all,
constructing a large-scale dataset for training is the main direction for
multi-modal tracking.

Figure 9: Unregistration examples in RGBT234 dataset. We show the ground truth
of visible modality in both images. The coarse bounding box degrades the
discriminability of the model.

Modality Registration

. As multi-modal data is captured by different sensors and the binocular
camera has a parallax error that cannot be ignored when the target is small
and has low resolution, registering the data in spacial and temporal aspects
is essential. As shown in Figure 9, the target is out of the box and the model
is degraded by learning meaningless background information. In the VOT-RGBT
challenge, the dataset ensures the precise annotation in infrared modality and
the misalignment of the RGB image is required to be handled by the tracker. We
state that the image pre-registration process is necessary during dataset
construction by cropping the shared visual field and applying image
registration method.

Metrics for Robustness Evaluation

. In some extreme scenes and weather conditions, such as rainy, low
illumination and hot sunny days, visible or thermal sensors cannot provide
meaningful data. The depth camera cannot obtain precise distance estimation
when the object is far from the sensor. Therefore, a robust tracker needs to
avoid tracking failure when any of the modality data is unavailable during a
certain period. To handle this case, both complementary and discriminative
features have to be applied in localization. However, none of the datasets
measures the tracking robustness with missing data. Thus, a new evaluation
metric for tracking robustness needs to be considered.

##  7 Conclusion

In this study, we provide an in-depth review of multi-modal tracking. First,
we conclude multi-modal trackers in a unified framework, and analyze them from
different perspectives, including auxiliary modality purpose and tracking
framework. Then, we present a detailed introduction on the datasets for multi-
modal tracking and corresponding metrics. Furthermore, a comprehensive
comparison of five popular datasets is conducted and the effectiveness of
trackers belonging to various types are analyzed in the views of overall
performance, attribute-based performance, and speed. Finally, as an emerging
field, several possible directions are identified to facilitate the
improvement of multi-modal tracking. The comparison results and analysis will
be available at https://github.com/zhang-pengyu/Multimodal_tracking_survey.

## References

  * [1] C. Li, H. Cheng, S. Hu, X. Liu, J. Tang, L. Lin, Learning collaborative sparse representation for grayscale-thermal tracking, IEEE Transactions on Image Processing 25 (12) (2016) 5743–5756.
  * [2] C. Li, N. Zhao, Y. Lu, C. Zhu, J. Tang, Weighted sparse representation regularized graph learning for RGB-T object tracking, in: ACM International Conference on Multimedia, 2017.
  * [3] C. Li, X. Liang, Y. Lu, N. Zhao, J. Tang, RGB-T object tracking: Benchmark and baseline, Pattern Recognition 96 (12) (2019) 106977.
  * [4] J. Xiao, R. Stolkin, Y. Gao, A. Leonardis, Robust fusion of color and depth data for RGB-D target tracking using adaptive range-invariant depth models and spatio-temporal consistency constraints, IEEE Transactions on Cybernetics 48 (8) (2018) 2485–2499.
  * [5] M. Kristan, J. Matas, A. Leonardis, M. Felsberg, et al., The eighth visual object tracking VOT2020 challenge results, in: European Conference on Computer Vision Workshop, 2020.
  * [6] M. Kristan, J. Matas, A. Leonardis, M. Felsberg, et al., The seventh visual object tracking VOT2019 challenge results, in: IEEE International Conference on Computer Vision Workshop, 2019.
  * [7] U. Kart, A. Lukezic, M. Kristan, J.-K. Kamarainen, J. Matas, Object tracking by reconstruction with view-specific discriminative correlation filters, in: IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  * [8] N. Cvejic, S. G. Nikolov, H. D. Knowles, A. Loza, A. Achim, A. Achim, C. N. Canagarajah, The effect of pixel-level fusion on object tracking in multi-sensor surveillance video, in: IEEE Conference on Computer Vision and Pattern Recognition, 2007.
  * [9] U. Kart, J.-K. Kamarainen, J. Matas, How to make an RGBD tracker?, in: European Conference on Computer Vision Workshop, 2018.
  * [10] P. Zhang, J. Zhao, D. Wang, H. Lu, X. Yang, Jointly modeling motion and appearance cues for robust rgb-t tracking, CoRR abs/2007.02041.
  * [11] C. Li, A. Lu, A. Zheng, Z. Tu, J. Tang, Multi-adapter RGBT tracking, in: IEEE International Conference on Computer Vision Workshop, 2019.
  * [12] S.-K. Weng, C.-M. Kuo, S.-K. Tu, Video object tracking using adaptive kalman filter, Journal of Visual Communication and Image Representation 17 (6) (2006) 1190–1208.
  * [13] G. Y. Kulikov, M. V. Kulikova, The accurate continuous-discrete extended kalman filter for radar tracking, IEEE Transactions on Signal Processing 64 (4) (2016) 948–958.
  * [14] C. Yang, R. Duraiswami, L. Davis, Fast multiple object tracking via a hierarchical particle filter, in: IEEE International Conference on Computer Vision, 2005.
  * [15] K. Okuma, A. Taleghani, N. de Freitas, J. J. Little, D. G. Lowe, A boosted particle filter: Multitarget detection and tracking, in: European Conference on Computer Vision, 2004.
  * [16] T. Zhang, B. Ghanem, S. Liu, N. Ahuja, Low-rank sparse learning for robust visual tracking, in: European Conference on Computer Vision, 2012.
  * [17] T. Zhang, B. Ghanem, S. Liu, N. Ahuja, Robust visual tracking via multi-task sparse learning, in: IEEE Conference on Computer Vision and Pattern Recognition, 2012.
  * [18] D. S. Bolme, J. R. Beveridge, B. A. Draper, Y. M. Lui, Visual object tracking using adaptive correlation filters, in: IEEE Conference on Computer Vision and Pattern Recognition, 2010.
  * [19] Y. Li, J. Zhu, A scale adaptive kernel correlation filter tracker with feature integration, in: European Conference on Computer Vision Workshop, 2014.
  * [20] L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, P. H. S. Torr, Fully-convolutional siamese networks for object tracking, in: European Conference on Computer Vision Workshop, 2016.
  * [21] Z. Zhang, H. Peng, Deeper and wider siamese networks for real-time visual tracking, in: IEEE Conference on Computer Vision and Pattern Recognition, 2019\.
  * [22] M. Danelljan, G. Hager, F. S. Khan, M. Felsberg, Learning spatially regularized correlation filters for visual tracking, in: IEEE International Conference on Computer Vision, 2015.
  * [23] H. K. Galoogahi, A. Fagg, S. Lucey, Learning background-aware correlation filters for visual tracking, in: IEEE International Conference on Computer Vision, 2017.
  * [24] Q. Wang, L. Zhang, L. Bertinetto, W. Hu, P. H. Torr, Fast online object tracking and segmentation: A unifying approach, in: IEEE Conference on Computer Vision and Pattern Recognition, 2019.
  * [25] A. Lukezic, J. Matas, M. Kristan, D3S – a discriminative single shot segmentation tracker, in: IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  * [26] X. Song, H. Zhao, J. Cui, X. Shao, R. Shibasaki, H. Zha, An online system for multiple interacting targets tracking: Fusion of laser and vision, tracking and learning, ACM Transactions on Intelligent Systems and Technology 4 (1) (2013) 1–21.
  * [27] D. Y. Kim, M. Jeon, Data fusion of radar and image measurements for multi-object tracking via kalman filtering, Information Fusion 278 (10) (2014) 641–652.
  * [28] N. Megherbi, S. Ambellouis, O. Colot, F. Cabestaing, Joint audio-video people tracking using belief theory, in: IEEE Conference on Advanced Video and Signal based Surveillance, 2005.
  * [29] X. Lan, M. Ye, S. Zhang, P. C. Yuen, Robust collaborative discriminative learning for RGB-infrared tracking, in: AAAI Conference on Artificial Intelligence, 2018.
  * [30] Y. Liu, X.-Y. Jing, J. Nie, H. Gao, J. Liu, Context-aware three-dimensional mean-shift with occlusion handling for robust object tracking in RGB-D videos, IEEE Transactions on Multimedia 21 (3) (2019) 664–677.
  * [31] P. K. Atrey, M. A. Hossain, A. E. Saddik, M. S. Kankanhalli, Multimodal fusion for multimedia analysis: a survey, Multimedia Systems 16 (2010) 345–379.
  * [32] G. S. Walia, R. Kapoor, Recent advances on multicue object tracking: A survey, Artificial Intelligence Review 46 (1) (2016) 1–39.
  * [33] Z. Cai, J. Han, L. Liu, L. Shao, Rgb-d datasets using microsoft kinect or similar sensors: A survey, Multimedia Tools and Applications 76 (2016) 4313–4355.
  * [34] M. Camplani, A. Paiement, M. Mirmehdi, D. Damen, S. Hannuna, T. Burghardt, L. Tao, Multiple human tracking in rgb-depth data: A survey, IET Computer Vision 11 (4) (2016) 265–285.
  * [35] T. Baltrusaitis, C. Ahuja, L.-P. Morency, Multimodal machine learning:a survey and taxonomy, IEEE Transactions on Pattern Analysis and Machine Intelligence 41 (2) (2019) 423–443.
  * [36] J. Ma, Y. Ma, C. Li, Infrared and visible image fusion methods and applications: A survey, Information Fusion 45 (2019) 153–178.
  * [37] X. Zhang, P. Ye, H. Leung, K. Gong, GangXiao, Object fusion tracking based on visible and infrared images: A comprehensive review, Information Fusion 63 (2020) 166–187.
  * [38] A. Bibi, T. Zhang, B. Ghanem, 3D part-based sparse tracker with automatic synchronization and registration, in: IEEE Conference on Computer Vision and Pattern Recognition, 2016.
  * [39] A. Gutev, C. J. Debono, Exploiting depth information to increase object tracking robustness, in: International Conference on Smart Technologies, 2019\.
  * [40] Y. Xie, Y. Lu, S. Gu, RGB-D object tracking with occlusion detection, in: International Conference on Computational Intelligence and Security, 2019.
  * [41] B. Zhong, Y. Shen, Y. Chen, W. Xie, Z. Cui, H. Zhang, D. Chen, T. Wang, X. Liu, S. Peng, J. Gou, J. Du, J. Wang, W. Zheng, Online learning 3D context for robust visual tracking, Neurocomputing 151 (2015) 710–718.
  * [42] N. An, X.-G. Zhao, Z.-G. Hou, Online RGB-D tracking via detection-learning-segmentation, in: International Conference on Pattern Recognition, 2016.
  * [43] M. Camplani, S. Hannuna, M. Mirmehdi, D. Damen, A. Paiement, L. Tao, T. Burghardt, Real-time RGB-D tracking with depth scaling kernelised correlation filters and occlusion handling, in: British Machine Vision Conference, 2015.
  * [44] P. Ding, Y. Song, Robust object tracking using color and depth images with a depth based occlusion handling and recovery, in: International Conference on Fuzzy Systems and Knowledge Discovery, 2015.
  * [45] G. M. Garcia, D. A. Klein, J. Stuckler, Adaptive multi-cue 3D tracking of arbitrary objects, in: Joint DAGM and OAGM Symposium, 2012.
  * [46] S. Hannuna, M. Camplani, J. Hall, M. Mirmehdi, D. Damen, T. Burghardt, A. Paiement, L. Tao, DS-KCF: a real-time tracker for RGB-D data, Journal of Real-Time Image Processing 16 (2019) 1439–1458.
  * [47] U. Kart, J.-K. Kamarainen, J. Matas, L. Fan, F. Cricri, Depth masked discriminative correlation filter, in: International Conference on Pattern Recognition, 2018.
  * [48] Y. Kuai, G. Wen, D. Li, J. Xiao, Target-aware correlation filter tracking in RGBD videos, Sensors 19 (20) (2019) 9522–9531.
  * [49] J. Leng, Y. Liu, Real-time RGB-D visual tracking with scale estimation and occlusion handling, Access 6 (2018) 24256–24263.
  * [50] W. Liu, X. Tang, C. Zhao, Robust RGBD tracking via weighted convolution operators, Sensors 20 (8) (2020) 4496–4503.
  * [51] Z. Ma, Z. Xiang, Robust object tracking with RGBD-based sparse learning, Frontiers of Information Technology and Electronic Engineering 18 (7) (2017) 989–1001.
  * [52] K. Meshgi, S. ichi Maeda, S. Oba, H. Skibbe, Y. zhe Li, S. Ishii, An occlusion-aware particle filter tracker to handle complex and persistent occlusions, Computer Vision and Image Understanding 150 (2016) 81–94.
  * [53] S. Chen, W. Zhu, H. Leung, Thermo-visual video fusion using probabilistic graphical model for human tracking, in: International Symposium on Circuits and Systems, 2008.
  * [54] Y. Wu, E. Blasch, G. Chen, L. Bai, H. Ling, Multiple source data fusion via sparse representation for robust visual tracking, in: International Conference on Information Fusion, 2011.
  * [55] Y. Wang, C. Li, J. Tang, D. Sun, Learning collaborative sparse correlation filter for real-time multispectral object tracking, in: International Conference on Brain Inspired Cognitive Systems, 2018.
  * [56] X. Lan, M. Ye, R. Shao, B. Zhong, Online non-negative multi-modality feature template learning for RGB-assisted infrared tracking, Access 7 (2019) 67761–67771.
  * [57] X. Lan, M. Ye, S. Zhang, H. Zhou, P. C. Yuen, Modality-correlation-aware sparse representation for RGB-infrared object tracking, Pattern Recognition Letters.
  * [58] X. Lan, M. Ye, R. Shao, B. Zhong, P. C. Yuen, H. Zhou, Learning modality-consistency feature templates: A robust RGB-Infrared tracking system, IEEE Transactions on Industrial Electronics 66 (12) 9887–9897.
  * [59] C. Li, C. Zhu, Y. Huang, J. Tang, L. Wang, Cross-modal ranking with soft consistency and noisy labels for robust RGB-T tracking, in: European Conference on Computer Vision, 2018.
  * [60] C. Li, S. Hu, S. Gao, , J. Tang, Real-time grayscale-thermal tracking via laplacian sparse representation, 2016.
  * [61] C. Li, C. Zhu, J. Zhang, B. Luo, X. Wu, J. Tang, Learning local-global multi-graph descriptors for RGB-T object tracking, IEEE Transactions on Circuits and Systems for Video Technology 29 (10) (2018) 2913–2926.
  * [62] H. Zhang, L. Zhang, L. Zhuo, J. Zhang, Object tracking in RGB-T videos using modal-aware attention network and competitive learning, Sensors 20 (2).
  * [63] C. Li, X. Sun, X. Wang, L. Zhang, J. Tang, Grayscale-thermal object tracking via multitask laplacian sparse representation 47 (4) 673–681.
  * [64] H. Liu, F. Sun, Fusion tracking in color and infrared images using joint sparse representation, Information Sciences 55 (3) (2012) 590–599.
  * [65] S. Zhai, P. Shao, X. Liang, X. Wang, Fast RGB-T tracking via cross-modal correlation filters, Neurocomputing 334 172–181.
  * [66] Y. Gao, C. Li, Y. Zhu, J. Tang, T. He, F. Wang, Deep adaptive fusion network for high performance RGBT tracking, in: IEEE International Conference on Computer Vision Workshop, 2019.
  * [67] Y. Zhu, C. Li, B. Luo, J. Tang, X. Wang, Dense feature aggregation and pruning for RGBT tracking, in: ACM International Conference on Multimedia, 2019.
  * [68] C. Li, X. Wu, N. Zhao, X. Cao, J. Tang, Fusing two-stream convolutional neural networks for RGB-T object tracking, Neurocomputing 281 (2018) 78–85.
  * [69] R. Yang, Y. Zhu, X. Wang, C. Li, J. Tang, Learning target-oriented dual attention for robust RGB-T tracking, in: IEEE International Conference on Image Processing, 2019.
  * [70] X. Zhang, X. Zhang, X. Du, X. Zhou, J. Yin, Learning multi-domain convolutional network for RGB-T visual tracking, 2018.
  * [71] L. Zhang, M. Danelljan, A. Gonzalez-Garcia1, J. van de Weijer, F. S. Khan, Multi-modal fusion for end-to-end RGB-T tracking, in: IEEE International Conference on Computer Vision Workshop, 2019.
  * [72] Y. Zhu, C. Li, Y. Lu, L. Lin, B. Luo, J. Tang, FANet: Quality-aware feature aggregation network for RGB-T tracking, CoRR abs/1811.09855.
  * [73] C. O. Conaire, N. E. O’Connor, E. Cooke, A. F. Smeaton, Comparison of fusion methods for thermo-visual surveillance tracking, in: International Conference on Information Fusion, 2006.
  * [74] H. Shi, C. Gao, N. Sang, Using consistency of depth gradient to improve visual tracking in RGB-D sequences, in: Chinese Automation Congress, 2015.
  * [75] Q. Wang, J. Fang, Y. Yuan, Multi-cue based tracking, Neurocomputing 131 (2014) 227–236.
  * [76] H. Zhang, M. Cai, J. Li, A real-time RGB-D tracker based on KCF, in: Chinese Control And Decision Conference, 2018.
  * [77] C. O. Conaire, N. E. O’Connor, A. F. Smeaton, Thermo-visual feature fusion for object tracking using multiple spatiogram trackers, Machine Vision and Applications 19 (5) (2008) 483–494.
  * [78] C. Luo, B. Sun, K. Yang, T. Lu, W.-C. Yeh, Thermal infrared and visible sequences fusion tracking based on a hybrid tracking framework with adaptive weighting scheme, Infrared Physics and Technology 99 265–276.
  * [79] Y. Zhai, P. Song, Z. Mou, X. Chen, X. Liu, Occlusion-aware correlation particle filter target tracking based on rgbd data, Access 6 (2018) 50752–50764.
  * [80] G. Li, L. Huang, P. Zhang, Q. Li, Y. Huo, Depth information aided constrained correlation filter for visual tracking, in: International Conference on Geo-Spatial Knowledge and Intelligence, 2019.
  * [81] C. Wang, C. Xu, Z. Cui, L. Zhou, T. Zhang, X. Zhang, J. Yang, Cross-modal pattern-propagation for RGB-T tracking, in: IEEE Conference on Computer Vision and Pattern Recognition, 2020.
  * [82] Y. Chen, Y. Shen, X. Liu, B. Zhong, 3d object tracking via image sets and depth-based occlusion detection, Signal Processing 112 (2015) 146–153.
  * [83] C. Li, C. Zhu, S. Zheng, B. Luo, J. Tang, Two-stage modality-graphs regularized manifold ranking for RGB-T tracking, Signal Processing: Image Communication 68 207–217.
  * [84] A. Geiger, P. Lenz, C. Stiller, R. Urtasun, Vision meets robotics: The KITTI dataset, International Journal of Robotics Research 32 (11) (2013) 1231–1237.
  * [85] J. Liu, Y. Liu, Y. Cui, Y. Chen, Real-time human detection and tracking in complex environments using single RGBD camera, in: IEEE International Conference on Image Processing, 2013.
  * [86] S. Song, J. Xiao, Tracking revisited using RGBD camera: Unified benchmark and baselines, in: IEEE International Conference on Computer Vision, 2013.
  * [87] A. Lukezic, U. Kart, J. Kapyla, A. Durmush, J.-K. Kamarainen, J. Matas, M. Kristan, Cdtb: A color and depth visual object tracking dataset and benchmark, in: IEEE International Conference on Computer Vision, 2019.
  * [88] J. W.Davis, V. Sharma, Background-subtraction using contour-based fusion of thermal and visible imagery, Computer Vision and Image Understanding 106 (2-3) 162–182.
  * [89] A. Torabi, G. Massé, G.-A. Bilodeau, An iterative integrated framework for thermal-visible image registration, sensor fusion, and people tracking for video surveillance applications, Computer Vision and Image Understanding 116 (2) 210–221.
  * [90] D. Ramachandram, G. W. Taylor, Deep multimodal learning: A survey on recent advances and trends 34 (6) (2017) 96–108.
  * [91] Y. Li, J. Zhu, A scale adaptive kernel correlation filter tracker with feature integration, in: European Conference on Computer Vision, 2014.
  * [92] M. Danelljan, G. Hager, F. S. Khan, M. Felsberg, Discriminative scale space tracking, IEEE Transactions on Pattern Analysis and Machine Intelligence 39 (8) (2017) 1561–1575.
  * [93] X. Li, W. Hu, C. Shen, Z. Zhang, A. Dick, A survey of appearance models in visual object tracking, ACM Transactions on Intelligent Systems and Technology 58 (4).
  * [94] S. Shojaeilangari, W.-Y. Yau, K. Nandakumar, J. Li, E. K. Teoh, Robust representation and recognition of facial emotions using extreme sparse learning, IEEE Transactions on Image Processing 24 (7).
  * [95] M. Yang, L. Zhang, X. Feng, D. Zhang, Sparse representation based fisher discrimination dictionary learning for image classification, International Journal of Computer Vision 109.
  * [96] Y. Xie, W. Zhang, C. Li, S. Lin, Y. Qu, Y. Zhang, Discriminative object tracking via sparse representation and online dictionary learning, IEEE Transactions on Cybernetics 44 (4).
  * [97] M. Isard, A. Blake, Condensation—conditional density propagation for visual tracking, International Journal of Computer Vision 29.
  * [98] M. Danelljan, G. Häger, F. S. Khan, M. Felsberg, Learning spatially regularized correlation filters for visual tracking, in: IEEE International Conference on Computer Vision, 2015.
  * [99] F. Li, C. Tian, W. Zuo, L. Zhang, M.-H. Yang, Learning spatial-temporal regularized correlation filters for visual tracking, in: IEEE Conference on Computer Vision and Pattern Recognition, 2018.
  * [100] M. Danelljan, G. Bhat, F. S. Khan, M. Felsberg, ECO: Efficient convolution operators for tracking, in: IEEE Conference on Computer Vision and Pattern Recognition, 2017.
  * [101] A. Lukežič, T. Vojíř, L. Čehovin Zajc, J. Matas, M. Kristan, Discriminative correlation filter tracker with channel and spatial reliability, International Journal of Computer Vision 126.
  * [102] K. Simonyan, A. Zisserman, Very deep convolutional networks for large-scale image recognition (2015).
  * [103] H. Nam, B. Han, Learning multi-domain convolutional neural networks for visual tracking, in: IEEE Conference on Computer Vision and Pattern Recognition, 2016\.
  * [104] K. He, X. Zhang, S. Ren, J. Sun, Deep residual learning for image recognition, in: IEEE Conference on Computer Vision and Pattern Recognition, 2016.
  * [105] S. Hare, A. Saffari, P. H. S. Torr, Struck:structured output tracking with kernels, in: IEEE International Conference on Computer Vision, 2011.
  * [106] A. Lukežič, L. Čehovin Zajc, T. Vojíř, J. Matas, M. Kristan, Now you see me: evaluating performance in long-term visual tracking, CoRR abs/1804.07056.
  * [107] Y. Wu, J. Lim, M.-H. Yang, Online object tracking: A benchmark, in: IEEE Conference on Computer Vision and Pattern Recognition, 2013.
  * [108] Y. Wu, J. Lim, M.-H. Yang, Object tracking benchmark, IEEE Transactions on Pattern Analysis and Machine Intelligence 37 (9).
  * [109] D. A. Klein, A. B. Cremers, Boosting scalable gradient features for adaptive real-time tracking, in: International Conference on Robotics and Automation, 2011\.
  * [110] T. Brox, A. Bruhn, N. Papenberg, J. Weickert, High accuracy optical flow estimation based on a theory for warping, in: European Conference on Computer Vision, 2004.
  * [111] F. M. Porikli, Integral histogram: a fast way to extract histograms in cartesian spaces, in: IEEE Conference on Computer Vision and Pattern Recognition, 2005.
  * [112] N. Dalal, B. Triggs, Histograms of oriented gradients for human detection, in: IEEE Conference on Computer Vision and Pattern Recognition, 2005.
  * [113] P. Viola, M. Jones, Rapid object detection using a boosted cascade of simple features, in: IEEE Conference on Computer Vision and Pattern Recognition, 2001\.
  * [114] B. K.P.Horn, B. G.Schunck, Determining optical flow, in: Artificial Intelligence, 1981.
  * [115] H. Bay, A. Ess, T. Tuytelaars, L. V. Gool, Speeded-up robust features, Computer Vision and Image Understanding 110 346–359.
  * [116] J. van de Weijer, C. Schmid, C. Schmid, D. Larlus, Learning color names for real-world applications, IEEE Transactions on Image Processing 18 (7) 1512–1523.
  * [117] A. Krizhevsky, I. Sutskever, G. E. Hinton, Imagenet classification with deep convolutional neural networks, in: Advances in Neural Information Processing Systems, 2012.
  * [118] S. Birchfield, S. Rangarajan, Spatiograms versus histograms for region-based tracking, in: IEEE Conference on Computer Vision and Pattern Recognition, 2005\.
  * [119] J. Canny, A computational approach to edge detection, IEEE Transactions on Pattern Analysis and Machine Intelligence 8 (6) 679–698.
  * [120] L. Zhang, A. Gonzalez-Garcia, J. van de Weijer, M. Danelljan, F. S. Khan, Synthetic data generation for end-to-end thermal infrared tracking, IEEE Transactions on Image Processing 28 (4) (2018) 1837–1850.
  * [121] C. Li, L. Liu, A. Lu, Q. Ji, J. Tang, Challenge-aware rgbt tracking, in: European Conference on Computer Vision, 2020.
  * [122] A. Lukezic, L. C. Zajc, T. Vojir, J. Matas, M. Kristan, FuCoLoT – a fully-correlational long-term tracker, in: Asian Conference on Computer Vision, 2018.
  * [123] A. Bendjebbour, ves Delignon, L. Fouque, V. Samson, W. Pieczynski, Multisensor image segmentation usingdempster–shafer fusion in markov fields context, IEEE Transactions on Geoscience and Remote Sensing 39 (8).
  * [124] H. Xu, T.-S. Chua, Fusion of av features and external information sources for event detection in team sports video, ACM Transactions on Multimedia Computing Communications and Applications 2 (1).
  * [125] H. Liu, K. Simonyan, Y. Yang, DARTS: Differentiable architecture search, 2019\.
  * [126] B. Zoph, V. Vasudevan, J. Shlens, Q. V. Le, Learning transferable architectures for scalable image recognition, in: IEEE Conference on Computer Vision and Pattern Recognition, 2018.
  * [127] C. Huang, S. Lucey, D. Ramanan, Learning policies for adaptive tracking with deep feature cascades, in: IEEE International Conference on Computer Vision, 2017\.