# Visual Object Tracking on Multi-modal RGB-D Videos: A Review

Xue-Feng Zhu, Tianyang Xu, Xiao-Jun Wu X.-F Zhu and X.-J Wu are with the
School of Artificial Intelligence and Computer Science, Jiangnan University,
Wuxi, P.R. China. (e-mail<EMAIL_ADDRESS>wu_xiaojun@jiangnan.edu.cn)T. Xu is with the School of Artificial Intelligence
and Computer Science, Jiangnan University, Wuxi, P.R. China and the Centre for
Vision, Speech and Signal Processing, University of Surrey, Guildford, GU2
7XH, UK. (e-mail<EMAIL_ADDRESS>
###### Abstract

The development of visual object tracking has continued for decades. Recent
years, as the wide accessibility of the low-cost RGBD sensors, the task of
visual object tracking on RGB-D videos has drawn much attention. Compared to
conventional RGB-only tracking, the RGB-D videos can provide more information
that facilitates objecting tracking in some complicated scenarios. The goal of
this review is to summarize the relative knowledge of the research filed of
RGB-D tracking. To be specific, we will generalize the related RGB-D tracking
benchmarking datasets as well as the corresponding performance measurements.
Besides, the existing RGB-D tracking methods are summarized in the paper.
Moreover, we discuss the possible future direction in the field of RGB-D
tracking.

###### Index Terms:

Visual objecting tracking, RGB-D tracking, multi-modal tracking

##  I Introduction

Visual object tracking, as a fundamental research topic in computer vision and
pattern recognition, aims to automatically predict the state of an object of
interest in each frame of a video. It plays a significant role in various
practical applications, including automatic driving, human-computer
interaction, intelligent surveillance, and so on [1, 2, 3]. Over the past
decades, visual object tracking has achieved significant progress [4, 5, 6,
7]. Conventional visual object tracking mainly focus on RGB videos captured by
a monocular camera. Despite the fact that considerable development has been
achieved in the research of visual object tracking, due to some challenging
factors in RGB videos such as background clutter, illumination variation,
deformation, occlusion, etc, it is still a challenging task to implement
robust tracking.

Trying to alleviate the limitation of RGB only devices, various reliable and
low-priced RGB-D sensors have been widely accessible recent years. An RGB-D
image is comprised of a three-channel RGB image and a single-channel depth
map. On account of the supplementary spatial information provided by an
accurate depth map, which can offset the defect of insufficient information of
RGB image in some complicated scenes, RGB-D data has been widely utilized for
multifarious computer vision applications [8, 9, 10, 11, 12]. In the visual
object tracking community, the progress of RGB-D tracking research has been
promoted significantly since the emergence of several RGB-D tracking
benchmarking datasets [13, 14, 15, 16]. Up to now, there are four RGB-D
tracking benchmarking datasets in total including PTB [13], STC [14], CDTB
[15] and DepthTrack [16]. The videos of these datasets are captured by
different kinds of cameras including ToF-RGB pair of cameras, stereo-camera
sensor and Kinect sensor. On these datasets, various measurements to evaluate
the performance of RGB-D trackers are also proposed.

In general, the aim of RGB-D tracking is to track an object of interest with a
given initial state in multi-modal RGB-D videos. Compared to the RGB-only
tracking, an additional depth map can provide more 3D spatial perception,
which can be adopted to avoid or address tracking failures caused by model
drifts or severe occlusions [17, 18]. Considering this advantage that the
depth map can benefit to address target occlusion and re-detect the target, as
well as the excellent performance of current RGB-only tracker, the existing
RGB-D trackers are generally based on high-performance short-term RGB
trackers, using the depth information additionally for occlusion estimation
[19]. Besides the challenge of occlusion, the problem of target disappearance
and reappearance should also be more manageable in RGB-D tracking datasets.
Accordingly, existing RGB-D tracking methods are endowed with a long-term
tracking mechanism with a re-detection module to address the problem of
tracking failures [15, 20, 21, 22]. Exploiting the depth channel with the help
of a long-term tracking mechanism, RGB-D trackers have been shown to report a
great performance improvement [20, 21].

To summarise, the key contributions of this work include the following:

  * •

We provide the details of the existing RGB-D tracking datasets and analyse the
characteristic of each dataset. As well, the corresponding performance
measurements are presented and analysed.

  * •

We summarize the existing RGB-D tracking algorithms and give the corresponding
analysis.

  * •

We present the potential development trend of RGB-D tracking. We hope that it
could provide some inspiration for relevant researchers.

The rest of this paper is organised as follows. In Section II, we discuss the
existing datasets and corresponding performance measurement. In this part, we
also present some comparison of the results of the existing RGB-D tracking. In
Section III, we review the existing RGB-D tracking methods. In Section IV, we
discuss the potential development trend of RGB-D tracking. Conclusions are
offered in Section V.

##  II RGBD tracking Dataset and Performance Measurements

In the recent years, the rapid development of sensor equipment and the
emergence of various RGB-D benchmarking datasets have significantly promoted
the progress of the RGB-D tracking research. In this section, we mainly
introduce the existing RGB-D datasets and corresponding performance
measurements. According to different imaging principles, there are various
kinds of RGB-D sensors, such as ToF-RGB pair of cameras, stereo-camera sensor
and Kinect sensor and so on. All these camera can capture a three-channel RGB
image and a single-channel depth map simultaneously. Four RGB-D tracking
datasets, PTB [13], STC [14], CDTB [15] and DepthTrack [16] have been
proposed. The sizes of these four datasets are provided in Table I.

TABLE I: The details of existing RGB-D datasets.

Dataset name | Video number | Total frames | ST/ LT
---|---|---|---
PTB | 100 | 21,542 | ST
STC | 36 | 9,009 | ST
CDTB | 80 | 100,045 | LT
DepthTrack | 200 | 294,600 | LT

Thanks to the emergence of the seminal publicly available RGB-D tracking
dataset Princeton Tracking Benchmark (PTB) [13], the research on RGB-D
tracking has drawn widespread attention. PTB contains 100 challenging videos
for RGB-D tracking evaluation. Among them, 95 videos only give the groundtruth
bounding boxes of the initial frame, and the remaining 5 videos provide all
the groundtruth labels for the parameters tuning. According to the target
category, target size, movement, occlusion and motion type, these sequences
are annotated with 11 attributes, including human, animal, rigid, large,
small, slow, fast, occlusion, no occlusion, passive and active. For the
performance evaluation on PTB, Successful Rate (SR) is proposed. The SR is the
percentage of frames where the overlap ratio between the predicted target
bounding box and the ground truth annotation exceeds the threshold of 0.5. For
evaluation, users can test their trackers on the 95 videos and submit the raw
results to the online evaluation page to get the overall performance. For more
details of the PTB dataset and the corresponding evaluation metric, the
readers can refer to [13].

STC [14] is an RGB-D tracking dataset comprising 36 video sequences with 10
per-frame annotated attributes, including illumination variation (IV), depth
variation (DV), scale variation (SV), colour distribution variation (CDV),
depth distribution variation (DDV), surrounding depth clutter (SDC),
surrounding colour clutter (SCC), background-colour camouflages (BCC),
background shape camouflages (BSC) and partial occlusion (PO). To evaluate the
performance of trackers, the Area Under Curve (AUC) as a measurement is used.
The AUC is defined as the area under the curve of the success plot. The
success plot shows the percentage of the instances where the bounding box
overlap ratio exceeds a given threshold. The reference [14] provides more
detailed information about the STC dataset and the corresponding evaluation
metrics.

CDTB is the existing largest testing dataset for RGB-D tracking, comprising 80
video sequences captured in a long-term tracking scenario. All these sequences
are annotated per-frame with thirteen attributes, including fast motion,
target size change, aspect ratio change, similar targets, target out-of-view,
full occlusion, partial occlusion, out-of-plane rotation, similar objects,
deformable target, reflective target, depth change and dark scene. Considering
that during a long term tracking, the target may disappear and reappear
several times, the ability to localise the target as well as to predict the
target absence, and re-detect the missed target, is very important. In this
case, as to tracking performance evaluation on long-term scenarios, the high-
profile VOT long-term tracking challenges [23, 20, 21, 22] have proposed long-
term tracking evaluation protocol measure the tracking performance on the CDTB
dataset. The tracking Precision and Recall are applied as the performance
measures. Specifically, the Precision is defined as the average ratio of
overlap of the predicted and ground truth targets on the frames where the
target is detected. The Recall represents the average ratio of overlap of the
predicted target bounding box and the ground truth annotation, measured on the
frames where the target is visible. The primary performance measurement on the
CDTB dataset is F-score obtained by calculating the tracking F-measurement
that combines tracking Precision and Recall. More specific information about
the CDTB dataset and the performance measurements is provided in [15].

DepthTrack [16] is the existing largest RGB-D tracking dataset, and is also
the first dataset for offline training of RGB-D trackers. It contains 200
RGB-D video sequences captured by using the Intel Realsense 415 sensor, in
which 150 videos can be adopted for the training offline of end-to-end network
based RGB-D tracker. The remaining 50 videos are used for the tracking
performance evaluation. For detailed scene-type level analysis of tracking
results, all videos are annotated 15 per-frame attributes, in which 13
attributes are the same as the attributes of CDTB. Two additional attributes,
background clutter and camera motion, are introduced in DepthTrack. Similar to
CDTB, DepthTrack is a long-term RGB-D tracking dataset, therefore, the
performance metrics of the DepthTrack are the same as the metrics of CDTB.
More details about the DepthTrack can be available in [16].

TABLE II: The SR scores of 13 trackers on the PTB dataset.

Method | Human | Animal | Rigid | Large | Small | Slow | Fast | Occ. | No-Occ. | Passive | Active
---|---|---|---|---|---|---|---|---|---|---|---
DAL [24] | 0.78 | 0.86 | 0.81 | 0.76 | 0.84 | 0.83 | 0.80 | 0.72 | 0.93 | 0.78 | 0.82
OTR [25] | 0.77 | 0.68 | 0.81 | 0.76 | 0.77 | 0.81 | 0.75 | 0.71 | 0.84 | 0.85 | 0.74
ca3dms+toh [26] | 0.66 | 0.74 | 0.82 | 0.73 | 0.74 | 0.80 | 0.71 | 0.63 | 0.88 | 0.83 | 0.70
CSR-rgbd++ [19] | 0.77 | 0.65 | 0.76 | 0.75 | 0.80 | 0.72 | 0.72 | 0.70 | 0.79 | 0.79 | 0.72
3DT [27] | 0.81 | 0.64 | 0.73 | 0.80 | 0.71 | 0.75 | 0.75 | 0.73 | 0.78 | 0.79 | 0.73
PT [13] | 0.74 | 0.63 | 0.78 | 0.78 | 0.70 | 0.76 | 0.72 | 0.72 | 0.75 | 0.82 | 0.70
OAPF [18] | 0.64 | 0.85 | 0.77 | 0.73 | 0.73 | 0.85 | 0.68 | 0.64 | 0.85 | 0.78 | 0.71
DLST [28] | 0.77 | 0.69 | 0.73 | 0.80 | 0.70 | 0.73 | 0.74 | 0.66 | 0.85 | 0.72 | 0.75
DM-DCF [29] | 0.76 | 0.58 | 0.77 | 0.72 | 0.73 | 0.75 | 0.72 | 0.69 | 0.78 | 0.82 | 0.69
DS-KCF-Shape [30] | 0.71 | 0.71 | 0.74 | 0.74 | 0.70 | 0.76 | 0.70 | 0.65 | 0.81 | 0.77 | 0.70
DS-KCF [31] | 0.67 | 0.61 | 0.76 | 0.69 | 0.70 | 0.75 | 0.67 | 0.63 | 0.78 | 0.79 | 0.66
STC [14] | 0.65 | 0.67 | 0.74 | 0.68 | 0.69 | 0.72 | 0.68 | 0.61 | 0.80 | 0.78 | 0.66
hiob_lc2 [32] | 0.53 | 0.72 | 0.78 | 0.61 | 0.70 | 0.72 | 0.64 | 0.53 | 0.85 | 0.77 | 0.62

TABLE III: The area under the curve (AUC) scores of 9 RGB-D trackers on the
STC dataset.

Method | IV | DV | SV | CDV | DDV | SDC | SCC | BCC | BSC | PO | ALL
---|---|---|---|---|---|---|---|---|---|---|---
DAL [24] | 0.64 | 0.51 | 0.63 | 0.50 | 0.60 | 0.62 | 0.64 | 0.63 | 0.57 | 0.58 | 0.58
OTR [25] | 0.39 | 0.48 | 0.31 | 0.19 | 0.45 | 0.44 | 0.46 | 0.42 | 0.42 | 0.50 | 0.49
ca3dms+toh [26] | 0.25 | 0.39 | 0.29 | 0.17 | 0.33 | 0.41 | 0.48 | 0.35 | 0.39 | 0.44 | 0.43
CSR-rgbd++ [19] | 0.35 | 0.43 | 0.30 | 0.14 | 0.39 | 0.40 | 0.43 | 0.38 | 0.40 | 0.46 | 0.45
OAPF [18] | 0.15 | 0.21 | 0.15 | 0.15 | 0.18 | 0.24 | 0.29 | 0.18 | 0.23 | 0.28 | 0.26
DS-KCF-Shape [30] | 0.29 | 0.38 | 0.21 | 0.04 | 0.25 | 0.38 | 0.47 | 0.27 | 0.31 | 0.37 | 0.39
DS-KCF [31] | 0.26 | 0.34 | 0.16 | 0.07 | 0.20 | 0.38 | 0.39 | 0.23 | 0.25 | 0.29 | 0.34
STC [14] | 0.28 | 0.36 | 0.24 | 0.24 | 0.36 | 0.38 | 0.45 | 0.32 | 0.34 | 0.37 | 0.40
PT [13] | 0.20 | 0.32 | 0.13 | 0.02 | 0.17 | 0.32 | 0.39 | 0.27 | 0.27 | 0.30 | 0.35

TABLE IV: Evaluation of 10 trackers on the CDTB dataset in terms of Precision,
Recall and F-score.

Method |  STARK_RGBD [22] |  TALGD [22] |  DRefine [22] |  ATCAIS [21] |  DDiMP [21] |  CLGS_D [21] |  SiamDW_D [21] |  LTDSEd [21] |  OTR [25] |  CSRDCF-D [19]
---|---|---|---|---|---|---|---|---|---|---
Precision | 0.742 | 0.728 | 0.707 | 0.709 | 0.703 | 0.725 | 0.677 | 0.674 | 0.364 | 0.375
Recall | 0.769 | 0.717 | 0.715 | 0.696 | 0.689 | 0.664 | 0.685 | 0.643 | 0.312 | 0.297
F-score | 0.756 | 0.722 | 0.711 | 0.702 | 0.696 | 0.693 | 0.681 | 0.658 | 0.336 | 0.340

TABLE V: The tracking results of 12 RGB-D trackers on the DepthTrack dataset.

| Sequence-based | Frame-based
---|---|---
Method | Pr | Re | F-score | Pr | Re | F-score
DeT [16] | 0.560 | 0.506 | 0.532 | 0.523 | 0.479 | 0.500
DDiMP [21] | 0.503 | 0.469 | 0.485 | 0.490 | 0.458 | 0.474
ATCAIS [21] | 0.500 | 0.455 | 0.476 | 0.496 | 0.470 | 0.483
CLGS_D [21] | 0.584 | 0.369 | 0.453 | 0.534 | 0.376 | 0.441
SiamDW_D [20] | 0.429 | 0.436 | 0.432 | 0.420 | 0.429 | 0.424
LTDSEd [21] | 0.430 | 0.382 | 0.405 | 0.425 | 0.383 | 0.403
SiamM_Ds [21] | 0.463 | 0.264 | 0.336 | 0.415 | 0.245 | 0.308
Siam_LTD [21] | 0.418 | 0.342 | 0.376 | 0.410 | 0.306 | 0.350
DAL [24] | 0.512 | 0.369 | 0.429 | 0.496 | 0.385 | 0.433
CA3DMS [26] | 0.218 | 0.228 | 0.223 | 0.211 | 0.221 | 0.216
DS-KCF-Shape [30] | 0.070 | 0.071 | 0.071 | 0.070 | 0.072 | 0.071
DS-KCF [31] | 0.075 | 0.077 | 0.076 | 0.074 | 0.076 | 0.075

##  III RGBD Tracking Methods

Since the the advent of the seminal publicly available RGB-D tracking dataset
PTB, the research on RGB-D tracking has received widespread attention. In
terms of the remarkable performance of existing RGB-only tracking framework,
most of the RGB-D tracking algorithms in the literature are based on the
existing RGB-only tracker, with different solutions for utilizing the
supplementary depth channel information effectively to improve performance.

Firstly, the Discrimative Correlation Filter (DCF) based trackers have shown
promising tracking performance on RGB videos. Therefore, there are a number of
DCF-based RGB-D trackers. For instance, Camplani et al. present a real-time
RGB-D tracker based on the Kernelized Correlation Filters (KCF) [33], fusing
colour and depth cues as features, with the depth information used to manage
occlusions and scale changes [31]. Similarly, exploiting the depth cue to
handle occlusions, scale variation, and shape change, a real-time RGB-D
tracker DS-KCF, built upon the fast KCF tracker, is proposed [30]. Awwad et
al. propose a local depth pattern feature for RGB-D tracking based on the
colour-only Struck tracker [34]. To convert an arbitrary short-term RGB
tracker into an RGB-D tracker, Kart et al. propose a general framework [19]
and adopt a novel depth segmentation-based occlusion detection module into the
discriminative correlation filter framework [29]. More recently, a long-term
RGB-D tracker OTR based on a DCF tracker is proposed to model appearance
changes via 3D target reconstruction [25]. Based on ECO tracker [35], Jiang et
al. propose hierarchical multi-modal fusion to fuse the features from RGB
image and depth map [36].

Secondly, there are also some RGB-D trackers based on other RGB-only trackers,
such as conventional mean-shift trackers and current end-to-end offline
trained RGB trackers. Not limiting the use of the depth information only for
occlusion reasoning, Xiao et al. propose an adaptive range-invariant target
depth model to exploit depth in tracking [14]. Considering that 3-D point
clouds can be generated from RGB-D data, a 3-D extension of the classical
mean-shift tracker is proposed to deal with occlusions more effectively [26].
Recently, as a result of the incorporation of a long-term RGB-D tracking
benchmark with a clearly defined performance evaluation methodology in the VOT
toolkit [15], more and more advanced algorithms based on end-to-end offline
trained RGB trackers are proposed for the VOT-RGBD challenge, focusing on
different ways of exploring depth information and designing effective long-
term tracking mechanisms [20, 21, 22].

Thirdly, Yan et al. proposes a end-to-end offline trained RGB-D tracker DeT
[16] based on the framework of the RGB-only trackers ATOM [37] and DiMP [38],
using a additional feature fusion module. As far as we know, it is the first
offline trained RGB-D tracking method. Considering the fact that the offline
trained algorithms require the support of large-scale data, they generate a
considerable number of RGB-D videos by conducting monocular depth prediction
on existing RGB tracking training data. With a small amount of real RGB-D
videos of the training set of DepthTrack and a large number of generated RGB-D
videos, the offline trained RGB-D tracker DeT achieves remarkable performance
on the testing set of DepthTrack.

For the articles and codes of the above methods, readers can refer to the
relevant references. Here we also collect some experimental results of
existing RGB-D methods for the convenience of readers, as shown in Table II,
III, IV, V.

##  IV Future Directions

Although the RGB-D tracking has achieved significant progress recent yeas,
there is still a huge space for improvement. Most of the existing RGB-D
trackers fail to exploit all the visual information conveyed by the two
modalities efficiently, only using the depth channel for occlusion reasoning
and target re-detection. For comptuter vison task, robustness and
expressiveness of features are critical to methods performance [39, 40, 41,
42, 43]. For RGB-D data, the modal asymmetry of RGB and depth is obvious,
therefore how to extract robust features from RGB and depth modalities is
extremely important to improve the performance of tracking. Additionally, the
fusion of the features of two modality is also vital for multi-modal tracking,
due to the fact that information fusion is beneficial to research of multi-
modal computer vision tasks [44, 45, 46, 47, 48]. Besides, in view of the
scarcity of RGB-D tracking videos for offline training of an RGB-D tracking
network, self-supervised learning method can be considered. Moreover, most
existing RGB-D tracking algorithms are learned online rather than trained
offline with sufficient RGB-D data. Considering the outstanding performance of
current RGB-only trackers offline trained using large-scale RGB videos,
existing RGB-D tracking data needs to be augmented. Futhermore, the RGB-D
image can be converted to 3D point cloud data, so that some processing methods
for 3D point cloud data can be applied to RGB-D tracking. With the increase of
RGB-D annotation data and the proposal of various algorithms in the future,
the RGB-D tracking filed will witness a great development.

##  V Conclusion

In this paper, we present a brief review of multi-modal RGB-D tracking.
Firstly, We provide the details of the existing RGB-D tracking datasets and
analyse the characteristic of each dataset. As well, the corresponding
performance measurements are presented and analysed. Then, we summarize the
existing RGB-D tracking algorithms and collect the results of the existing
RGB-D trackers. Finally, we present the potential directions of RGB-D
tracking. We hope this brief review can give valuable insight into the
research of RGB-D tracking and encourage new research.

## References

  * [1] X. Xue, Y. Li, X. Yin, C. Shang, T. Peng, and Q. Shen, “Semantic-aware real-time correlation tracking framework for uav videos,” _IEEE Transactions on Cybernetics_ , pp. 01–12, 2020.
  * [2] E. Griffiths, A. Kalyanaraman, J. Ranjan, and K. Whitehouse, “An empirical design space analysis of doorway tracking systems for real-world environments,” _ACM Transactions on Sensor Networks (TOSN)_ , vol. 13, no. 4, pp. 1–34, 2017.
  * [3] A. W. Smeulders, D. M. Chu, R. Cucchiara, S. Calderara, A. Dehghan, and M. Shah, “Visual tracking: An experimental survey,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 36, no. 7, pp. 1442–1468, 2013.
  * [4] X.-F. Zhu, X.-J. Wu, T. Xu, Z.-H. Feng, and J. Kittler, “Complementary discriminative correlation filters based on collaborative representation for visual object tracking,” _IEEE Transactions on Circuits and Systems for Video Technology_ , vol. 31, no. 2, pp. 557–568, 2020.
  * [5] X.-F. Zhu, X.-J. Wu, T. Xu, Z. Feng, and J. Kittler, “Robust visual object tracking via adaptive attribute-aware discriminative correlation filters,” _IEEE Transactions on Multimedia_ , 2021.
  * [6] Xu, Tianyang and Feng, Zhen-Hua and Wu, Xiao-Jun and Kittler, Josef, “Joint group feature selection and discriminative filter learning for robust visual object tracking,” in _Proceedings of the IEEE International Conference on Computer Vision_ , 2019, pp. 7950–7960.
  * [7] T. Xu, Z. Feng, X.-J. Wu, and J. Kittler, “Adaptive channel selection for robust visual object tracking with discriminative correlation filters,” _International Journal of Computer Vision_ , vol. 129, no. 5, pp. 1359–1375, 2021.
  * [8] A. Wang, J. Cai, J. Lu, and T.-J. Cham, “Modality and component aware feature fusion for rgb-d scene classification,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 5995–6004.
  * [9] J.-F. Hu, W.-S. Zheng, J. Lai, and J. Zhang, “Jointly learning heterogeneous features for rgb-d activity recognition,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2015, pp. 5344–5352.
  * [10] D. Lin, R. Zhang, Y. Ji, P. Li, and H. Huang, “Scn: switchable context network for semantic segmentation of rgb-d images,” _IEEE transactions on cybernetics_ , vol. 50, no. 3, pp. 1120–1131, 2018.
  * [11] K. Fu, D.-P. Fan, G.-P. Ji, Q. Zhao, J. Shen, and C. Zhu, “Siamese network for rgb-d salient object detection and beyond,” _IEEE Transactions on Pattern Analysis and Machine Intelligence_ , pp. 01–18, 2021.
  * [12] C. Li, R. Cong, S. Kwong, J. Hou, H. Fu, G. Zhu, D. Zhang, and Q. Huang, “Asif-net: Attention steered interweave fusion network for rgb-d salient object detection,” _IEEE transactions on cybernetics_ , vol. 51, no. 1, pp. 88–100, 2020.
  * [13] S. Song and J. Xiao, “Tracking revisited using rgbd camera: Unified benchmark and baselines,” in _Proceedings of the IEEE international conference on computer vision_ , 2013, pp. 233–240.
  * [14] J. Xiao, R. Stolkin, Y. Gao, and A. Leonardis, “Robust fusion of color and depth data for rgb-d target tracking using adaptive range-invariant depth models and spatio-temporal consistency constraints,” _IEEE transactions on cybernetics_ , vol. 48, no. 8, pp. 2485–2499, 2017.
  * [15] A. Lukezic, U. Kart, J. Kapyla, A. Durmush, J.-K. Kamarainen, J. Matas, and M. Kristan, “Cdtb: A color and depth visual object tracking dataset and benchmark,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 10 013–10 022.
  * [16] S. Yan, J. Yang, J. Käpylä, F. Zheng, A. Leonardis, and J.-K. Kämäräinen, “Depthtrack: Unveiling the power of rgbd tracking,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 10 725–10 733.
  * [17] T. Bagautdinov, F. Fleuret, and P. Fua, “Probability occupancy maps for occluded depth images,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2015, pp. 2829–2837.
  * [18] K. Meshgi, S.-i. Maeda, S. Oba, H. Skibbe, Y.-z. Li, and S. Ishii, “An occlusion-aware particle filter tracker to handle complex and persistent occlusions,” _Computer Vision and Image Understanding_ , vol. 150, pp. 81–94, 2016.
  * [19] U. Kart, J.-K. Kamarainen, and J. Matas, “How to make an rgbd tracker?” in _Proceedings of the European Conference on Computer Vision (ECCV) Workshops_ , 2018, pp. 01–15.
  * [20] M. Kristan, J. Matas, A. Leonardis, M. Felsberg, R. Pflugfelder, J.-K. Kamarainen, L. Cehovin Zajc, O. Drbohlav, A. Lukezic, A. Berg _et al._ , “The seventh visual object tracking vot2019 challenge results,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops_ , 2019, pp. 01–36.
  * [21] M. Kristan, A. Leonardis, J. Matas, M. Felsberg, R. Pflugfelder, J.-K. Kämäräinen, M. Danelljan, L. Č. Zajc, A. Lukežič, O. Drbohlav _et al._ , “The eighth visual object tracking vot2020 challenge results,” in _European Conference on Computer Vision_. Springer, 2020, pp. 547–601.
  * [22] M. Kristan, J. Matas, A. Leonardis, M. Felsberg, R. Pflugfelder, J.-K. Kämäräinen, H. J. Chang, M. Danelljan, L. Cehovin, A. Lukežič _et al._ , “The ninth visual object tracking vot2021 challenge results,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2021, pp. 2711–2738.
  * [23] M. Kristan, A. Leonardis, J. Matas, M. Felsberg, R. Pflugfelder, L. ˇCehovin Zajc, T. Vojir, G. Bhat, A. Lukezic, A. Eldesokey _et al._ , “The sixth visual object tracking vot2018 challenge results,” in _Proceedings of the European Conference on Computer Vision Workshops_. Springer, 2018, pp. 01–52.
  * [24] Y. Qian, S. Yan, A. Lukežič, M. Kristan, J.-K. Kämäräinen, and J. Matas, “Dal: A deep depth-aware long-term tracker,” in _2020 25th International Conference on Pattern Recognition (ICPR)_. IEEE, 2021, pp. 7825–7832.
  * [25] U. Kart, A. Lukezic, M. Kristan, J.-K. Kamarainen, and J. Matas, “Object tracking by reconstruction with view-specific discriminative correlation filters,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 1339–1348.
  * [26] Y. Liu, X.-Y. Jing, J. Nie, H. Gao, J. Liu, and G.-P. Jiang, “Context-aware three-dimensional mean-shift with occlusion handling for robust object tracking in rgb-d videos,” _IEEE Transactions on Multimedia_ , vol. 21, no. 3, pp. 664–677, 2018.
  * [27] A. Bibi, T. Zhang, and B. Ghanem, “3d part-based sparse tracker with automatic synchronization and registration,” in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition_ , 2016, pp. 1439–1448.
  * [28] N. An, X.-G. Zhao, and Z.-G. Hou, “Online rgb-d tracking via detection-learning-segmentation,” in _2016 23rd International Conference on Pattern Recognition (ICPR)_. IEEE, 2016, pp. 1231–1236.
  * [29] U. Kart, J.-K. Kämäräinen, J. Matas, L. Fan, and F. Cricri, “Depth masked discriminative correlation filter,” in _2018 24th International Conference on Pattern Recognition (ICPR)_. IEEE, 2018, pp. 2112–2117.
  * [30] S. Hannuna, M. Camplani, J. Hall, M. Mirmehdi, D. Damen, T. Burghardt, A. Paiement, and L. Tao, “Ds-kcf: a real-time tracker for rgb-d data,” _Journal of Real-Time Image Processing_ , vol. 16, no. 5, pp. 1439–1458, 2019\.
  * [31] M. Camplani, S. L. Hannuna, M. Mirmehdi, D. Damen, A. Paiement, L. Tao, and T. Burghardt, “Real-time rgb-d tracking with depth scaling kernelised correlation filters and occlusion handling.” in _BMVC_ , vol. 3, 2015, pp. 01–12.
  * [32] P. Springstübe, S. Heinrich, and S. Wermter, “Continuous convolutional object tracking.” in _ESANN_ , vol. 4, 2018, p. 5.
  * [33] J. F. Henriques, R. Caseiro, P. Martins, and J. Batista, “High-speed tracking with kernelized correlation filters,” _IEEE transactions on pattern analysis and machine intelligence_ , vol. 37, no. 3, pp. 583–596, 2014.
  * [34] S. Awwad, F. Hussein, and M. Piccardi, “Local depth patterns for tracking in depth videos,” in _Proceedings of the 23rd ACM international conference on Multimedia_ , 2015, pp. 1115–1118.
  * [35] M. Danelljan, G. Bhat, F. Shahbaz Khan, and M. Felsberg, “Eco: Efficient convolution operators for tracking,” in _Proceedings of the IEEE conference on computer vision and pattern recognition_ , 2017, pp. 6638–6646.
  * [36] M.-x. Jiang, C. Deng, J.-s. Shan, Y.-y. Wang, Y.-j. Jia, and X. Sun, “Hierarchical multi-modal fusion fcn with attention model for rgb-d tracking,” _Information Fusion_ , vol. 50, pp. 1–8, 2019.
  * [37] M. Danelljan, G. Bhat, F. S. Khan, and M. Felsberg, “Atom: Accurate tracking by overlap maximization,” in _Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition_ , 2019, pp. 4660–4669.
  * [38] G. Bhat, M. Danelljan, L. V. Gool, and R. Timofte, “Learning discriminative model prediction for tracking,” in _Proceedings of the IEEE/CVF International Conference on Computer Vision_ , 2019, pp. 6182–6191.
  * [39] M. Wang, S.-T. Wang, and X.-J. Wu, “Initial results on fuzzy morphological associative memories,” _Acta Electronica Sinica_ , vol. 31, no. 5, pp. 690–693, 2003.
  * [40] J. Sun, W. Fang, X. Wu, and W. Xu, “Quantum-behaved particle swarm optimization: principles and applications,” 2011.
  * [41] S.-G. Chen and X.-J. Wu, “A new fuzzy twin support vector machine for pattern classification,” _International Journal of Machine Learning and Cybernetics_ , vol. 9, no. 9, pp. 1553–1564, 2018.
  * [42] Y.-J. Zheng, J.-Y. Yang, J. Yang, X.-J. Wu, and Z. Jin, “Nearest neighbour line nonparametric discriminant analysis for feature extraction,” _Electronics Letters_ , vol. 42, no. 12, pp. 679–680, 2006.
  * [43] C. Li, W. Yuan, A. Bovik, and X. Wu, “No-reference blur index using blur comparisons,” _Electronics letters_ , vol. 47, no. 17, pp. 962–963, 2011\.
  * [44] X. Luo, Z. Zhang, B. Zhang, and X.-J. Wu, “Image fusion with contextual statistical similarity and nonsubsampled shearlet transform,” _IEEE Sensors Journal_ , vol. 17, no. 6, pp. 1760–1771, 2017.
  * [45] X. Luo, Z. Zhang, and X. Wu, “A novel algorithm of remote sensing image fusion based on shift-invariant shearlet transform and regional selection,” _AEU-International Journal of Electronics and Communications_ , vol. 70, no. 2, pp. 186–197, 2016.
  * [46] H. Li and X.-J. Wu, “Multi-focus image fusion using dictionary learning and low-rank representation,” in _International Conference on Image and Graphics_. Springer, Cham, 2017, pp. 675–686.
  * [47] J. Sun, C. Li, X.-J. Wu, V. Palade, and W. Fang, “An effective method of weld defect detection and classification based on machine vision,” _IEEE Transactions on Industrial Informatics_ , vol. 15, no. 12, pp. 6322–6333, 2019\.
  * [48] H. Li, X.-J. Wu, and T. Durrani, “Nestfuse: An infrared and visible image fusion architecture based on nest connection and spatial/channel attention models,” _IEEE Transactions on Instrumentation and Measurement_ , vol. 69, no. 12, pp. 9645–9656, 2020.