# TRAT: Tracking by Attention Using Spatio-Temporal Features

Hasan Saribas1, Hakan Cevikalp2, Okan Köpüklü3, Bedirhan Uzun2
1 Eskisehir Technical University 2 Eskisehir Osmangazi University 3 Technical
University of Munich

###### Abstract

Robust object tracking requires knowledge of tracked objects’ appearance,
motion and their evolution over time. Although motion provides distinctive and
complementary information especially for fast moving objects, most of the
recent tracking architectures primarily focus on the objects’ appearance
information. In this paper, we propose a two-stream deep neural network
tracker that uses both spatial and temporal features. Our architecture is
developed over ATOM tracker and contains two backbones: (i) 2D-CNN network to
capture appearance features and (ii) 3D-CNN network to capture motion
features. The features returned by the two networks are then fused with
attention based Feature Aggregation Module (FAM). Since the whole architecture
is unified, it can be trained end-to-end. The experimental results show that
the proposed tracker TRAT (TRacking by ATtention) achieves state-of-the-art
performance on most of the benchmarks and it significantly outperforms the
baseline ATOM tracker. The source code and models are available at
https://github.com/Hasan4825/TRAT.

##  1 Introduction

The visual object tracking is an important computer vision task, and the goal
is to track a target object in the subsequent frames of a video where the
target object is identified in the first frame. It is widely used in many
domains including surveillance, video and activity analysis, and robotics.
However, it is a challenging task since there are numerous factors such as
significant deformation and appearance variations of the target object,
illumination changes, background clutter, occlusion, etc. that can make the
tracking of the target extremely difficult. The main difficulty arises from
the fact that the tracker must learn an appearance model of the target object
at the initial frame just by using bounding box information. Then, it must
have the adaptability to generalize to all variations of the target object
appearances in the subsequent frames.

Figure 1: Visual comparison of the TRAT with state-of-the-art trackers. ATOM
[11] and DiMP50 [3] trackers are based on only the appearance information
coming from 2D-CNN features. Consequently, they fail to track fast moving
objects. Our approach TRAT employs 3D-CNN and 2D-CNN backbones to
simultaneously extract appearance and motion information and provide accurate
bounding box predictions for slow and fast moving objects.

Object tracking requires the incorporation of spatial and temporal features
captured within video frames. The spatial part carries information about the
scenes and object appearances in individual video frames, whereas the temporal
part contains the motion information of the tracked objects across the video
frames. So far, the majority of the trackers that employ deep neural network
architectures only used spatial features coming from video frames for
tracking. On the other hand, spatio-temporal features have been widely used
for both activity recognition and detection in video frames [51, 65, 48, 35,
31]. However, capturing spatio-temporal features is not very straightforward
in tracking settings as in activity recognition since we do not have access to
all video frames during online tracking. Instead, we have only access to the
previous frames of the current frame, and many tracking videos do not include
any motion for long time periods. As a result, special attention must be given
to combine spatial and temporal information during online tracking. In fact,
temporal information is mostly useful when the object appearance changes
significantly between consecutive frames because of object speed, shape
deformation, illumination variations, and out-of-plane rotations. Motion
information also helps when there are similar objects in the vicinity of the
tracked object. Fig. 1 illustrates several examples where the motion
information is critical.

In this paper we propose a novel two-stream deep neural network tracker that
uses both spatial and temporal features. We use an attention based Feature
Aggregation Module (FAM) to fuse the information coming from two-channels
effectively. Proposed FAM makes use of local channel correlations for
efficiency and a more discriminative pooling strategy to improve the tracking
performance.

###  1.1 Related Work

The most recent state-of-the-art tracking methods can be roughly divided into
three categories: CNN (Convolutional Neural Network) based trackers,
correlation filter based trackers and Siamese network based trackers. CNN
based trackers [47, 44, 59, 41, 58, 61, 27] usually use shallow neural
networks since it has been shown in many studies that the last layers of large
CNNs are more effective in capturing semantics. Small networks are also
necessary for speed issues since updating parameters of large networks is slow
and it cannot be accomplished in real-time for online visual tracking. There
are also methods using larger networks, but they usually update either the
last classifier stage or a few last layers of the network during online
learning due to the computational efficiency. One of the earliest deep neural
network learning tracker [61] used a stacked de-noising auto-encoder to learn
generic image features from a large dataset as auxiliary data and then
transferred the learned features to the online tracking task. Hong et al. [27]
used CNNs and SVM classifier to learn target-specific saliency maps. Li et al.
[41] introduced an online learning method based on a CNN using multiple image
cues. Wang et al. [59] pre-trained a large CNN offline and transferred the
learned features to the online tracking as in [61]. Nam and Han [47]
introduced the Multi-Domain Network (MDNet) for object tracking. Beside some
online training tricks such as long-term and short-term update strategies, the
main novelty of the paper was to show how to transfer rich and effective
features for tracking. A fast version of MDNet tracker, called Real-Time
MDNet, [32] used Fast R-CNN [20] approach to accelerate the slow feature
extraction stage of the MDNet method. Cevikalp et al. [7] proposed a deep
neural network tracker using ranking loss which enforces the network to return
better bounding boxes framing the target object. For the same purpose, both
[11, 3] used novel deep neural tracking architectures that utilize IoU-Net
[30] whose goal is to estimate and increase the Intersection over Union (IoU)
overlap between the target an estimated bounding box to improve the accuracy.

Correlation filter (CF) based trackers learn a correlation filter to localize
the target object in consecutive frames by solving a ridge regression problem
efficiently in the Fourier frequency domain. The learned filter is applied to
a region of interest in the next frame, and the maximum correlation filter
response determines the object location in the new frame. Then, the filter is
updated by using this new object location. CF based trackers are extremely
fast compared to the deep neural network based trackers owing to the fact that
the problem is solved efficiently in the frequency domain. Bolme et al. [5]
introduced a very fast CF based method using the minimum output of squared
error (MOSSE) for visual tracking. Kernelized correlation filters using
circulant matrices and multi-channel features have been proposed in [26].
Danelljan et al. [14] introduced a formulation using continuous convolutional
operators, which paved the way for efficiently integrating multi-resolution
deep feature maps into the convolution filter based trackers. In [12], this
method has been improved by the introduction of factorized convolutional
operators. Initially, CF trackers used gray levels or hand-crafted features
such as histogram of oriented gradients (HOGs) [26], color names [15] or color
histograms [1]. On the other hand, CF trackers proposed in [44, 13] utilized
CNN features extracted from pre-trained CNNs. The most recent studies
introduced methods to learn both deep CNN features and correlation filters
simultaneously [55, 62, 22]. The best performing tracker [22] of the VOT2017
[36] challenge uses this methodology. ROI Pooled Correlation Filter (RPCF)
method [52] used a region of interest (ROI) based pooling operation in the
correlation filters algorithm. Wang et al. [60] integrated forward tracking
and backward verification steps, which are based on forward-backward
trajectory analysis, into the unsupervised training phase. Xu et al. [67]
proposed a feature compression in both spatial and channel dimensions by using
group channel feature selection method to learn correlation filters. [10] used
adaptive spatially regularized correlation filters model to estimate the
object location and scale. Saribas et al. [50] and Huang et al. [29] achieved
high accuracy and speed on tracking of objects in aerial videos by using
correlation filters.

The trackers that are based on Siamese networks on the other hand are built
based on distance (or similarity) metric learning for visual object tracking
[53, 2, 56, 23]. Initially, these methods [53, 2] are trained on larger
datasets such as ILSVRC15 [49] and the learned metric (i.e., matching
function) is simply evaluated online during tracking, and these methods did
not have the ability to update the previously learned distance function during
tracking. Later, Guo et al. [23] and Wang et al. [56] proposed methods that
update the learned distance function online during tracking. One of the main
challenges in Siamese-based trackers is how to adapt aspect ratio changes. To
address this problem, Siamese Region Proposal Network (SiamRPN) [40] used
region proposal networks. SiamRPN++ [39] proposed a layer-wise feature
aggregation structure to improve performance and reduce the model size. To
solve the data imbalance problem of RPN, Fan and Ling [17] have used hard
negative sampling process and cascaded RPN instead of one-stage RPN. Gradient-
Guided network (GradNet) method [42] used a novel gradient-guided network to
avoid overfitting and increase adaptation ability. SiamMask [64] utilized a
multi task learning approach to solve both object tracking and object
segmentation simultaneously. Wang et al. [57] used a two-stage series-parallel
matching for tracking where the coarse matching (CM) and fine matching (FM)
stages enhance the robustness performance and discrimination power of Siamese
networks. Choi et al. [9] presented a novel meta-learner network by adding
target aware feature space to improve tracking accuracy.

Trackers Using Spatio-Temporal Features: Regarding trackers using both spatial
and temporal features, [21] showed that a correlation filter tracker using
deep motion (optical flow) features combined with deep RGB features
outperforms the same tracker using appearance (deep RGB features) information
alone. Teng et al. [54] proposed a deep neural network that includes temporal
and spatial networks, where the temporal network collects key historical
temporal samples by solving a sparse optimization problem. The output of the
temporal network is fed to the spatial network that further refines the
localization of tracked target. A Graph Convolutional Tracking (GCT) method
built based on Siamese framework, which uses spatio-temporal features, has
been given in [18]. Zhang et al. [69] learned a spatio-temporal context model
between the tracked target and its local surrounding background and used it
for robust tracking of the targets. The method which is the most related to
our proposed method is the FlowTrack introduced in [71]. FlowTrack also uses
both spatial and temporal information for better tracking of the object.
However, there are significant differences between the FlowTrack and our
tracker TRAT. FlowTrack is a CF based tracker by making use of several 2D-CNNs
to extract RGB and optical flow modality features, and an attention module to
fuse them. In contrast, we use a 3D-CNN backbone to extract temporal features
in addition to the 2D-CNN backbone which extract spatial features. The feature
maps coming from these two backbones are fused with an effective and
discriminative attention based feature aggregation module using local channel
correlations. The reason to include 3D-CNN backbone in our tracker is that 3D
convolutional kernels inherently captures pixel movements (i.e., motion
information) on consecutive frames. Although 3D-CNNs are considered
computationally heavy, they are much more efficient compared to optical flow
calculation. To the best of our knowledge, TRAT is the first architecture,
which employs 3D-CNNs to capture motion information for the task of object
tracking.

Figure 2: The architecture of the proposed two-stream tracker. It uses both
spatial and temporal features through 2D-CNN and 3D-CNNs. The feature maps of
the two-networks are fused by using a feature aggregation module shown with
dashed line rectangle.

##  2 Method

We propose a two-stream deep neural network tracker for visual object
tracking, which is illustrated in Fig. 2. Our tracker TRAT is developed over
ATOM tracker architecture in a way that it uses a two-stream network including
both 3D-CNN and 2D-CNN architectures rather than using 2D-CNN alone. In
addition, we propose an efficient feature aggregation module (FAM) to
aggregate the feature maps of 3D-CNN and 2D-CNN backbones to track the target
object more successfully. Therefore, the proposed tracker mainly includes 3
important components as seen in the figure: 3D-CNN, 2D-CNN, and feature
aggregation module (FAM). In the following, we will first briefly describe
ATOM (Accurate Tracking by Overlap Maximization) tracker and explain each
proposed components in details. Next, we give the implementation details at
the end of this section.

###  2.1 ATOM

ATOM tracker includes two components designed exclusively for target
estimation and classification. Target estimation module is learned offline and
it predicts the Intersection over Union (IoU) ratio between the target and
estimated bounding box. Target classification module uses a discriminative
correlation filter (DCF) to return an initial bounding box candidate which
will be used in target estimation module. Then, 10 object bounding box
proposals are generated by adding uniform random noise to this initial
position. The target estimation module estimates the IoU ratios of these boxes
and the final prediction is obtained by taking the mean of the 3 bounding
boxes with the highest IoU ratios.

Target classification module of the ATOM consists of a 2-layer fully
convolutional neural network defined as,

$f(\mathbf{x};\mathbf{w})=\Phi_{2}(\mathbf{w}_{2}*\Phi_{1}(\mathbf{w}_{1}*\mathbf{x})),$
(1)

where $\mathbf{x}$ is the backbone feature map,
$\mathbf{w}=\left\\{\mathbf{w}_{1},\>\mathbf{w}_{2}\right\\}$ are the
parameters of the network, $\Phi_{1},\Phi_{2}$ are activation functions. The
module uses the following discriminative correlation filter objective function
given as,

$L(\mathbf{w})=\sum_{j=1}^{m}\gamma_{j}\left\|f(\mathbf{x}_{j};\mathbf{w})-\mathbf{y}_{j})\right\|^{2}+\sum_{k}\lambda_{k}\left\|\mathbf{w}_{k}\right\|^{2}.$
(2)

Here, $\mathbf{y}_{j}$ represents the classification confidences for feature
maps $\mathbf{x}_{j}$. The impact of training samples is adjusted by using the
weight $\gamma_{j}$, and $\lambda_{k}$ is the weight for regularization.

Instead of using only spatial features $\mathbf{x}$ coming from 2D-CNN as in
ATOM, TRAT makes use of spatio-temporal features extracted by 3D-CNN and
2D-CNN, which are fused by attention based FAM.

###  2.2 3D-CNN

Tracking methods using spatial (appearance) information alone struggle in
situations where the object appearance changes significantly between
consecutive frames because of shape deformation, out-of-plane rotations,
illumination variations, and background distractors with similar appearances
to the tracked object. In such cases, motion information provides rich
complementary information and it may help to disambiguate the target object
from the background. Most studies used optical-flow features for motion
information. However, in addition to the computational complexity spent for
extracting optical-flow features, many tracking videos lack motion for long
time periods. As a result, optical-flow features will be completely zero for
many video sequences which limits its use for tracking. On the other hand,
3D-CNNs can capture motion information by applying convolution operation in
both space and time dimensions. Moreover,3D-CNNs can also provide information
when the object is stationary. Therefore, 3D-CNN backbone is selected to be
used as motion capturing unit in TRAT architecture.

In our 3D-CNN, we use 3D-ResNet architecture. During tracking, we give a
sequence of cropped ROIs (Region of Interests) to the network as input. The
ROIs are cropped from a sequence of consecutive frames in time order and the
size of ROIs is determined at the beginning by using the given target ground-
truth location in the first frame. Given a cropped ROIs with size $h\times w$,
the size of the 3D-CNN input is $\left(c\times f\times h\times w\right)$,
where $c=3$, $f$ is the number of consecutive frames, which is set to 4 in our
experiments. After convolution operations, 3D-CNN outputs a feature map
$\mathcal{X}_{3D}\in{\rm I\\!R}^{c^{{}^{\prime}}\times f^{{}^{\prime}}\times
h^{{}^{\prime}}\times w^{{}^{\prime}}}$, where $c^{{}^{\prime}}$ is the number
of output channels, $f^{{}^{\prime}}=1$, $h^{{}^{\prime}}=\frac{h}{d}$, and
$w^{{}^{\prime}}=\frac{w}{d}$. $d$ refers to the applied downsampling rate.
The depth dimension of the resulting feature map is reduced to 1 such that
output volume is squeezed to $\left(c^{{}^{\prime}}\times
h^{{}^{\prime}}\times w^{{}^{\prime}}\right)$ in order to match the output
feature map of 2D-CNN.

###  2.3 2D-CNN

To exploit spatial (appearance) information for tracking, we use a 2D-CNN,
which employs ResNet [25] architecture. In contrast to 3D-CNN, 2D-CNN takes a
single ROI image cropped from the most recent frame, and its size is also
$h\times w$. The 2D-CNN outputs a feature map $\mathcal{X}_{2D}\in{\rm
I\\!R}^{c^{{}^{\prime\prime}}\times h^{{}^{\prime}}\times w^{{}^{\prime}}}$,
where $c^{{}^{\prime\prime}}$ is the number of output channels, and
$h^{{}^{\prime}}=\frac{h}{d}$, $w^{{}^{\prime}}=\frac{w}{d}$ as in 3D-CNN
output. Since, $h^{{}^{\prime}}$ and $w^{{}^{\prime}}$ are common for both
3D-CNN and 2D-CNNs, the feature maps of these two networks can be concatenated
directly. The resulting stacked features include both appearance and motion
information and they are passed to the subsequent FAM as input.

Figure 3: The architecture of the proposed feature aggregation module.

###  2.4 Feature Aggregation Module (FAM)

The fusion of feature maps coming from 2D-CNN and 3D-CNN backbones should be
handled properly in order to achieve superior performance compared to single
backbones alone. To this end, we use an effective channel fusion and attention
mechanism to aggregate the feature maps. The proposed module is a variant of
the Efficient Channel Attention (ECA) module [63] which can be simply
implemented via 1D convolution. As opposed to other channel attention modules
that use correlations among all channels, ECA captures only local cross-
channel interactions by considering local neighborhoods of each channel. This
makes the method both efficient and effective since ECA can be implemented by
fast 1D convolutions of size $k$, where kernel size $k$ represents the
coverage of local cross-channel interactions.

However, ECA loses discriminative details since it employs global average
pooling (GAP) at the beginning, which represents an entire channel with the
average sum of the values in the channel. Still, not all values in a feature
channel contribute equally to the localization of the tracked object and some
features are more discriminative than the others. Therefore, we replace the
GAP in ECA module with a more discriminative pooling strategy that uses
weighted sum of the values in each channel, which is called as adaptive
weighted pooling (AWP) in [6, 19]. AWP learns to give higher weights to more
discriminative features during pooling and this largely increases the
detection performance as reported in [19]. This is somewhat similar to spatial
attention since more discriminative features are more heavily weighted during
summation.

The proposed attention based feature aggregation module is given in Fig. 3. We
use the same architectures for both 3D-CNN and 2D-CNN, therefore the outputs
of these networks have same channel size
$(c^{{}^{\prime}}=c^{{}^{\prime\prime}})$. As a result,
$\mathcal{X}_{2D}\in{\rm I\\!R}^{c^{{}^{\prime}}\times h^{{}^{\prime}}\times
w^{{}^{\prime}}}$ and $\mathcal{X}_{3D}\in{\rm I\\!R}^{c^{{}^{\prime}}\times
h^{{}^{\prime}}\times w^{{}^{\prime}}}$. To fuse feature maps, we utilized two
fusion methods where the first one simply concatenates feature channels
whereas the second one adds the feature channels as described below.

Concatenation fusion. As given in Fig. 3, our concatenation fusion strategy
contains concatenation operation and convolution operation with $1\times 1$
kernel size. The concatenation operation stacks two feature maps along
channels to produce
$\mathcal{X}_{Cat}=f^{cat}(\mathcal{X}_{2D},\mathcal{X}_{3D})\in{\rm
I\\!R}^{2c^{{}^{\prime}}\times h^{{}^{\prime}}\times w^{{}^{\prime}}}$. Then,
we feed the stacked feature maps into 2D convolutional layer to return to
original size by reducing the channel size by half $\mathcal{X}_{S}\in{\rm
I\\!R}^{c^{{}^{\prime}}\times h^{{}^{\prime}}\times w^{{}^{\prime}}}$.

Sum fusion. The sum operation simply computes the sum of the two feature maps
at the same spatial locations to produce
$\mathcal{X}_{S}=f^{sum}(\mathcal{X}_{2D},\mathcal{X}_{3D})\in{\rm
I\\!R}^{c^{{}^{\prime}}\times h^{{}^{\prime}}\times w^{{}^{\prime}}}$.

After fusion operation, we employ adaptive weighted pooling (AWP) to downscale
feature map, $\mathcal{X}_{S}$. To this end, we first feed $\mathcal{X}_{S}$
into convolution layer and apply amplified sigmoid function to the convolved
feature map for computing $W(\mathcal{X}_{S})$. In order to obtain new
weighted feature map $\tilde{\mathcal{X}}_{S}\in{\rm
I\\!R}^{c^{{}^{\prime}}\times h^{{}^{\prime}}\times w^{{}^{\prime}}}$, we use
the following formulation,

$\tilde{\mathcal{X}}_{S}=W(\mathcal{X}_{S})\odot\mathcal{X}_{S},$ (3)

where $\odot$ represents Hadamard product (i.e., element-wise product). This
operation acts like a spatial attention mechanism since the more
discriminative locations related to the tracked object are weighted more
heavily compared to other regions in the background. This largely improves the
tracking performance as demonstrated in the experiments. Then, we applied a
global average pooling on $\tilde{\mathcal{X}}_{S}$ to obtain discriminative
pooling weights $\mathcal{X}_{AWP}=g(\tilde{\mathcal{X}}_{S})\in{\rm
I\\!R}^{c^{{}^{\prime}}\times 1\times 1}$ by using,

$g(\tilde{\mathcal{X}}_{S})=\frac{1}{h\times
w}\sum_{i=1}^{h}\sum_{j=1}^{w}\tilde{\mathcal{X}}_{S}(i,j).$ (4)

Finally, we apply 1D convolution and sigmoid operations to $\mathcal{X}_{AWP}$
to get our channel attention map $\mathcal{W}_{C}\in{\rm
I\\!R}^{c^{{}^{\prime}}\times 1\times 1}$,

$W_{C}=\sigma(Conv1D_{k}(\mathcal{X}_{AWP})),$ (5)

where $k=5$ is the kernel size of 1D convolution and $\sigma(.)$ is the
Sigmoid function. At the end, the fused feature map $\mathcal{X}_{S}$ is
multiplied with the channel attention map $\mathcal{W}_{C}$ to produce the
aggregated feature map $\mathcal{X}_{FAM}\in{\rm I\\!R}^{c^{{}^{\prime}}\times
h^{{}^{\prime}}\times w^{{}^{\prime}}}$,

$\mathcal{X}_{FAM}=\mathcal{X}_{S}\otimes\mathcal{W}_{C}.$ (6)

###  2.5 Implementation Details

Offline training. For offline training, we use the training splits of the
GOT-10k [28], LaSOT [16] and TrackingNet [46] datasets. We used ResNet-18 and
ResNet-50 backbones for both 2D-CNN and 3D-CNN streams. We pretrained our
3D-ResNet architecture on Kinetics dataset [24] using 4 consecutive frames to
learn motion information of the moving targets. For 2D-ResNet, we initialize
network weights with model pre-trained on ImageNet. Image pixel values are
normalized to lie in the range [0,1]. During offline training, we freeze all
the weights of the 3D-CNN backbone so that it does not lose its ability to
generate temporal information due to the training video frames where no motion
occurs. Also we freeze all weights of the 2D-CNN backbone as in [11].

In a batch, for both train and test branches, we use 64 images and 64 4-frame
clips as input to 2D-CNN and 3D-CNN backbones, respectively. Our sampling
strategy can be defined as follows: we first determine the key frame to
extract spatial features using 2D-CNN, and then we create a corresponding
video clip with 4 consecutive frames in time order, with the key frame being
the last frame. Therefore, when the frame number of key frame is smaller than
4, we apply same padding to the beginning of the clip in time dimension. We
also apply data augmentations including, color jittering and image flipping.
In order to ensure pixel-wise correspondence, the same data augmentations are
applied to the key frame and clip. We trained FAM and IoU predictor modules
for 50 epochs using Adam [34] optimizer with initial learning rates of
$5\times 10^{-4}$ and $10^{-3}$, respectively, and using a factor of 0.2 decay
at every 15 epochs. The mean-squared error is used as loss function.

Online tracking. We employ the online tracking strategy of the ATOM tracker.
During online tracking, our two-stream network is initialized with offline
pre-trained model weights and all weights of our network except for the target
classification module are frozen. Spatial resolution of $288\times 288$ is
used for input images. While the target estimation module receives layer-2 and
layer-3 features of both 2D-ResNet and 3D-ResNet as input, the target
classification module receives only layer-3 features of both 2D-ResNet and
3D-ResNet as input. Downsampling rates $d$ of the backbones are 8 and 16 for
layer-2 and layer-3, respectively. The target classification module learns the
weights of 2-layer fully convolutional network
$\mathbf{w}=\left\\{\mathbf{w}_{1},\>\mathbf{w}_{2}\right\\}$ using the
$L^{2}(.)$ loss function as given in Eq. (2). While the first layer applies a
$1\times 1$ convolutional layer to reduce the channel size of layer-3 features
to 64, the second layer of the classification module applies a convolutional
layer with $4\times 4$ kernel size. For the fast minimization of the objective
function given in Eq. (2), we use the Gauss-Newton approach proposed in ATOM
during online learning. In the initial frame, while the classification module
learns both $\mathbf{w}_{1}$ and $\mathbf{w}_{2}$, the network updates just
the weights of $\mathbf{w}_{2}$ at every 10th frame.

We utilize data augmentation strategies such as rotation, blur, dropout,
varying degrees of translation, image flipping, and color jittering to
construct an initial training set including 30 samples using given the first
frame. During online tracking, the new frame is determined as the key frame in
time order. We use 4 frames to create a video clip with the key frame being
the last frame as in offline training. Also, we copy the initial frame to
complete the video clip when the key frame number is less than 4.

##  3 Experiments

The proposed tracker is implemented by using PyTorch. On Tesla V100 GPU, TRAT
runs at 21 fps and 28 fps using backbones of ResNet-18 and ResNet-50,
respectively. We evaluate TRAT on eight different benchmark datasets,
including the OTB-100 [66], UAV123 [45], NfS [33], VOT2018 [37], VOT2019 [38],
TrackingNet [46], LaSOT [16] and GOT-10k [28].

On the OTB-100 [66], UAV123 [45] and NfS [33] datasets, we report the results
in one-pass evaluation (OPE) protocol with both precision and success plots,
by using OTB evaluation toolkit. Tracking algorithms are ranked based on the
area-under-curve (AUC) scores obtained from the precision and success plots.
On the VOT2018 [37] and VOT2019 [38] datasets, we use VOT challenge protocol
which uses reset-based methodology in which a tracker is re-initialized
whenever the tracking fails. The accuracy is measured in terms of expected
average overlap (EAO), which quantitatively reflects both bounding box overlap
ratio (accuracy-A) and re-initialization times (robustness-R). We report the
AUC scores as well as precision (Pre) and normalized precision (Prenorm)
scores on the TrackingNet [46] and LaSOT [16] datasets. Finally, to evaluate
results on the GOT-10k [28] dataset, we calculate average overlap (AO) and
success rates (SR) at overlap thresholds set to 0.5 and 0.75.

To ensure the reliability of the results, we run the proposed tracker 5 times
on the OTB-100, UAV123, NfS, LaSOT, 3 times on the GOT-10k, 1 time on the
TrackingNet, 15 times on both VOT2018 and VOT2019 datasets then we report the
averages of these results.

Table 1: Ablation study on the OTB-100, NfS and UAV123 datasets. (The red
fonts indicate the best result.)

|  |  |  |  |  |  |  | OTB-100 | NfS | UAV123
---|---|---|---|---|---|---|---|---|---|---
Res-18 | Res-50 | 2D-CNN | 3D-CNN | Sum F. | Cat F. | Attn | AWP | AUC | Pre | AUC | Pre | AUC | Pre
✓ |  | ✓ |  |  |  |  |  | 66.3 | 87.3 | 58.5 | 69.5 | 64.2 | 84.3
| ✓ | ✓ |  |  |  |  |  | 67.2 | 87.4 | 59.6 | 70.2 | 64.9 | 84.9
✓ |  |  | ✓ |  |  |  |  | 60.9 | 79.8 | 55.3 | 65.2 | 57.3 | 78.2
| ✓ |  | ✓ |  |  |  |  | 61.5 | 81.1 | 56.4 | 65.8 | 58.9 | 79.8
| ✓ | ✓ | ✓ | ✓ |  |  |  | 67.9 | 87.5 | 61.3 | 72.4 | 64.3 | 84.1
| ✓ | ✓ | ✓ |  | ✓ |  |  | 67.8 | 87.4 | 61.5 | 72.7 | 65.2 | 85.2
| ✓ | ✓ | ✓ |  | ✓ | ✓ |  | 68.4 | 88.3 | 62.6 | 73.1 | 65.8 | 86.2
| ✓ | ✓ | ✓ |  | ✓ | ✓ | ✓ | 68.6 | 88.7 | 62.8 | 73.6 | 66.2 | 86.8

###  3.1 Ablation Studies

We conduct an ablation study on the OTB-100 [66], NfS [33], and UAV123 [45]
datasets, to show the impact of individual components of the proposed tracker
TRAT. Table 1 shows the results of the used components on these datasets.

Impact of Backbone. We take ATOM tracker as baseline to analyze the impact of
using ResNet-18 and ResNet-50 as its backbone. As expected, usage of the
ResNet-50 backbone improves AUC scores by about 0.7-1% and precision scores by
about 0.1-0.6%.

We also explore the applicability of 3D-CNNs for object tracking problem.
Therefore, we experiment using only 3D-CNN backbone. Although the architecture
with 3D-CNN performs inferior compared to the architecture with 2D-CNN
backbone, it still outperforms some of the recent trackers such as ECO, MDNet
and UPDT on AUC and precision metrics for NfS dataset, where the motion of the
tracked objects is significant. However, we will see in the next ablation
study that 3D-CNN backbone produces complementary features to 2D-CNN features.
As expected, deeper 3D-CNN backbone again achieves better performance.

Impact of Fusion Strategies. We conduct experiments for both sum fusion and
concatenation fusion in order to explore the impact of feature aggregation. As
given in Table 1, fusion of 2D-CNN and 3D-CNN features improves both AUC and
precision scores on the all datasets. Concatenation fusion strategy has more
improvements than sum fusion strategy on the NfS and UAV123 datasets. On the
OTB-100 dataset, both fusion approaches obtain similar results. Therefore, we
have used concatenation fusion for the rest of the ablation studies.

Impact of Chanel Attention. After aggregating the feature maps using
concatenation fusion, we have applied the channel attention module to weight
the importance of aggregated channels. For this analysis, we have used global
average pooling instead of AWP. Channel attention module improves AUC scores
by about 0.6-1.0% and precision scores by about 0.4-1.0%.

Impact of AWP. Finally, we have investigated the impact of the AWP component.
It improves AUC scores by about 0.2-0.4% and precision scores by around
0.4-0.6%. To compare with other state-of-the-art trackers, we utilize this
final tracker, which is called TRAT.

###  3.2 State-of-the-art Comparison

Object Tracking Benchmark (OTB-100). The OTB-100 [66] is a well-known single
object tracking dataset. This dataset includes 100 fully annotated video
sequences, with 11 various challenging factors. We report results in Table 2
using the OPE protocol. UPDT [4] and SiamRPN++ [39] trackers achieve the best
accuracy and precision scores, respectively. Our proposed method obtains
competitive results on this dataset.

NfS. We evaluate our approach on need for speed dataset [33] (30 FPS version),
which is the first high frame rate dataset recorded at real-world scenes. It
includes 100 fully annotated videos (380K frames) with fast moving target
objects. The AUC and precision scores of state-of-the-art trackers are shown
in Table 2. TRAT achieves a success score of 62.8% which is higher than the
previous best result. In terms of precision, TRAT obtains the best results
after DiMP50 [3] tracker. Moreover, TRAT outperforms our baseline tracker,
ATOM [11], with relative gains of 7.4% and 5.9% in terms of AUC and precision
scores, respectively. These results demonstrate the importance of using 3D-CNN
in videos involving a fast-moving target object.

Figure 4: Precision and success plots of the TRAT and state-of-the-art
trackers on the NfS using one pass evaluation (OPE) protocol.

UAV123. The UAV123 [45] includes 123 videos captured from low-altitude
unmanned aerial vehicles. We report the success and precision scores in Table
2. Our tracker achieves an AUC score of 66.2% and precision score of 86.2%,
which are higher than the previous best performing trackers DiMP50 [3] with
65.3% and 86.0% in terms of AUC and precision scores, respectively. TRAT
outperforms ATOM [11] considerably by AUC score of 63.1% $\rightarrow$ 66.2%
and precision score of 84.1% $\rightarrow$ 86.2%.

Table 2: State-of-the-art comparison on the OTB-100, NfS and UAV123 datasets.
(The red and blue fonts respectively indicate the best and the second best
results.)

| OTB-100 | NfS | UAV123
---|---|---|---
| AUC | Precision | AUC | Precision | AUC | Precision
UPDT [4] | 70.7 | - | 54.1 | - | 55.0 | -
RankingT [7] | 70.1 | 91.1 | - | - | 53.5 | 74.6
DaSiam-RPN [70] | 65.8 | 88.0 | 60.6 | 75.2 | 56.9 | 78.1
MDNet [47] | 67.8 | 90.9 | 41.5 | 49.2 | 56.9 | 78.1
ECO [12] | 69.1 | 91.0 | 46.6 | - | 52.5 | 74.1
SiamRPN++ [39] | 69.6 | 91.4 | - | - | 61.3 | 80.7
SiamBAN [8] | 69.6 | 91.0 | 59.4 | - | 63.1 | 83.3
DiMP50 [3] | 68.4 | 89.4 | 62.0 | 73.7 | 65.3 | 86.0
ATOM [11] | 66.3 | 87.4 | 58.5 | 69.5 | 63.1 | 84.3
TRAT | 68.6 | 88.7 | 62.8 | 73.6 | 66.2 | 86.8

TrackingNet. We evaluate our tracker on the large-scale TrackingNet [46]
benchmark using success, precision and normalized precision metrics. Table 3
shows the results of state-of-the-art trackers on the TrackingNet test split
which contains 511 videos. TRAT ranks second for all metrics and achieves
competitive results compared to the best performing trackers, SiamRPN++ [39]
and SiamAttn [68].

Table 3: State-of-the-art comparison on the TrackingNet dataset. (The red and
blue fonts respectively indicate the best and the second best results.)

| UPDT [4] | DaSiam-RPN [70] | D3S [43] | SiamAttn [68] | SiamRPN++ [39] | DiMP50 [3] | ATOM [11] | TRAT
---|---|---|---|---|---|---|---|---
Pre (%) | 55.7 | 59.1 | 66.4 | - | 69.4 | 68.7 | 64.8 | 68.9
Prenorm (%) | 70.2 | 73.3 | 76.8 | 81.7 | 80.0 | 80.1 | 77.1 | 80.3
Success (%) | 61.1 | 63.8 | 72.8 | 75.2 | 73.3 | 74.0 | 70.3 | 74.2

LaSOT. Large scale object tracking [16] dataset consist of 1400 sequences with
3.52 million frames, which have high-quality dense annotations. We evaluate
our tracker on test split which consist of 280 videos. TRAT achieves the
second best result on success metric and competitive results for normalized
precision metric. More importantly, TRAT significantly outperforms ATOM by
7.0% and 5.1% on normalized precision and success metrics, respectively.

Figure 5: Normalized precision and success plots of the TRAT and state-of-the-
art trackers on the LaSOT. Table 4: State-of-the-art comparison on the LaSOT
dataset. (The red and blue fonts respectively indicate the best and the second
best results.)

| DaSiam-RPN [70] | SiamBAN [8] | SiamAttn [68] | SiamRPN++ [39] | DiMP50 [3] | ATOM [11] | TRAT
---|---|---|---|---|---|---|---
Prenorm (%) | 49.6 | 59.8 | 64.8 | 56.9 | 64.8 | 57.6 | 64.6
Success (%) | 41.5 | 51.4 | 56.0 | 49.6 | 56.8 | 51.5 | 56.6

GOT-10k. The GOT-10k dataset [28] is another large-scale object tracking
dataset which contains 10K video sequences for the training set and 180 videos
with an average length of 127 frames for the test set. We evaluate TRAT on the
GOT-10k dataset using the GOT-10k evaluation toolkit. Table 5 shows the state-
of-the-art comparison on the GOT-10k test split. We report the results in
terms of success rates (SR) and average overlap (AO). In terms of SR at
overlap thresholds set to 0.5 and 0.75, TRAT obtains the best and second-best
results of 72.0% and 46.7% respectively. Again TRAT has a comparable AO result
to DiMP50 [3]. Compared with the baseline tracker ATOM [11], TRAT has
significant improvements of 8.6% on SR0.50, 6.5% on SR0.75, and 5.2% on AO.

Table 5: State-of-the-art comparison on the GOT-10k dataset. (The red and blue
fonts respectively indicate the best and the second best results.)

| MDNet [47] | CCOT [14] | ECO [12] | SiamMask [64] | D3S [43] | DiMP50 [3] | ATOM [11] | TRAT
---|---|---|---|---|---|---|---|---
SR0.50 (%) | 30.3 | 32.8 | 30.8 | 58.7 | 67.6 | 71.7 | 63.4 | 72.0
SR0.75 (%) | 9.9 | 10.7 | 11.1 | 36.6 | 46.2 | 49.2 | 40.2 | 46.7
AO (%) | 29.9 | 32.5 | 31.6 | 51.4 | 59.7 | 61.1 | 55.6 | 60.8

VOT2018-VOT2019. We evaluate trackers on the short-term challenges of the
VOT2018 [37] and VOT 2019 [38] datasets which consist of 60 fully annotated
video sequences by rotated bounding boxes. All frames are also annotated for
different challenging factors including motion change, size change, occlusion,
illumination change, camera motion, and unassigned which did not correspond to
the other attributes. We run our trackers 15 times to extract results by
provided VOT evaluation toolkit. Table 6 and Table 7 show the state-of-the-art
comparison on the VOT2018 and VOT2019 datasets, respectively. We report all
results using a reset-based methodology which contains accuracy (A),
robustness (R) and expected average overlap (EAO) metrics for both datasets.
Our proposed method TRAT outperforms the state-of-the-art trackers with 0.456
EAO, 0.605 accuracy, and 0.148 robustness scores on the VOT2018 dataset.
Especially in terms of EAO, TRAT achieves large gains of 5.5%, compared to our
baseline method ATOM [11].

Table 6: State-of-the-art comparison on the VOT2018 dataset.

| RankingT [7] | SiamBAN [8] | DaSiam-RPN [4] | SiamRPN++ [39] | DiMP50 [3] | ATOM [11] | TRAT
---|---|---|---|---|---|---|---
AUC | 0.554 | 0.597 | 0.503 | 0.604 | 0.597 | 0.590 | 0.605
Robustness | 0.213 | 0.178 | 0.159 | 0.234 | 0.153 | 0.204 | 0.148
EAO | 0.335 | 0.452 | 0.389 | 0.414 | 0.440 | 0.401 | 0.456

On the VOT2019 dataset, TRAT achieves the best accuracy score of 0.606.
Moreover, in terms of robustness and EAO, it obtains the second-best scores of
0.302 and 0.381, respectively.

Table 7: State-of-the-art comparison on the VOT2019 dataset.

| RankingT [7] | SiamBAN [8] | SiamMask [64] | SiamRPN++ [39] | DiMP50 [3] | ATOM [11] | TRAT
---|---|---|---|---|---|---|---
AUC | 0.528 | 0.602 | 0.594 | 0.599 | 0.594 | 0.603 | 0.606
Robustness | 0.360 | 0.327 | 0.461 | 0.482 | 0.278 | 0.411 | 0.302
EAO | 0.270 | 0.396 | 0.287 | 0.285 | 0.379 | 0.292 | 0.381

##  4 Conclusion

This paper introduces a two-stream deep neural network tracker using 2D-CNN
and 3D-CNN backbones. The proposed tracker is built based on ATOM tracker
architecture, and the 2D-CNN stream is used to capture appearance information
whereas the 3D-CNN is used for capturing motion cues. The feature maps coming
from these two streams are fused by using an effective channel fusion and
attention mechanism that considers only local correlations between feature
channels. This significantly decreases the computational complexity, and
allows the network to learn the feature aggregation parameters during online
tracking efficiently. The proposed tracker, TRAT, is evaluated on most of the
tracking datasets and results show that it achieves the state-of-the-art
accuracies on most of the tested datasets. Moreover, it significantly
outperforms the baseline tracker ATOM that uses only appearance information
alone. This indicates the importance of motion information in tracking
applications.

## References

  * [1] Luca Bertinetto, Jack Valmadre, Stuart Golodetz, Ondrej Miksik, and Philip HS Torr.  Staple: Complementary learners for real-time tracking.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1401–1409, 2016.
  * [2] Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Torr.  Fully-convolutional siamese networks for object tracking.  In European conference on computer vision, pages 850–865. Springer, 2016.
  * [3] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte.  Learning discriminative model prediction for tracking.  arXiv preprint arXiv:1904.07220, 2019.
  * [4] Goutam Bhat, Joakim Johnander, Martin Danelljan, Fahad Shahbaz Khan, and Michael Felsberg.  Unveiling the power of deep tracking.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 483–498, 2018.
  * [5] David S Bolme, J Ross Beveridge, Bruce A Draper, and Yui Man Lui.  Visual object tracking using adaptive correlation filters.  In 2010 IEEE computer society conference on computer vision and pattern recognition, pages 2544–2550. IEEE, 2010.
  * [6] Jiale Cao, Hisham Cholakkal, Rao Muhammad Anwer, Fahad Shahbaz Khan, Yanwei Pang, and Ling Shao.  D2det: Towards high quality object detection and instance segmentation.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11485–11494, 2020.
  * [7] Hakan Cevikalp, Hasan Saribas, Burak Benligiray, and Sinem Kahvecioglu.  Visual object tracking by using ranking loss.  In The IEEE International Conference on Computer Vision (ICCV) Workshops, Oct 2019.
  * [8] Zedu Chen, Bineng Zhong, Guorong Li, Shengping Zhang, and Rongrong Ji.  Siamese box adaptive network for visual tracking.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6668–6677, 2020.
  * [9] Janghoon Choi, Junseok Kwon, and Kyoung Mu Lee.  Deep meta learning for real-time target-aware visual tracking.  In Proceedings of the IEEE International Conference on Computer Vision, pages 911–920, 2019.
  * [10] Kenan Dai, Dong Wang, Huchuan Lu, Chong Sun, and Jianhua Li.  Visual tracking via adaptive spatially-regularized correlation filters.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4670–4679, 2019.
  * [11] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg.  Atom: Accurate tracking by overlap maximization.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4660–4669, 2019.
  * [12] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg.  Eco: Efficient convolution operators for tracking.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6638–6646, 2017.
  * [13] Martin Danelljan, Gustav Hager, Fahad Shahbaz Khan, and Michael Felsberg.  Convolutional features for correlation filter based visual tracking.  In Proceedings of the IEEE international conference on computer vision workshops, pages 58–66, 2015.
  * [14] Martin Danelljan, Andreas Robinson, Fahad Shahbaz Khan, and Michael Felsberg.  Beyond correlation filters: Learning continuous convolution operators for visual tracking.  In European conference on computer vision, pages 472–488. Springer, 2016.
  * [15] Martin Danelljan, Fahad Shahbaz Khan, Michael Felsberg, and Joost Van de Weijer.  Adaptive color attributes for real-time visual tracking.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1090–1097, 2014.
  * [16] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling.  Lasot: A high-quality benchmark for large-scale single object tracking.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5374–5383, 2019.
  * [17] Heng Fan and Haibin Ling.  Siamese cascaded region proposal networks for real-time visual tracking.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 7952–7961, 2019.
  * [18] Junyu Gao, Tianzhu Zhang, and Changsheng Xu.  Graph convolutional tracking.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4649–4659, 2019.
  * [19] Ziteng Gao, Limin Wang, and Gangshan Wu.  Lip: Local importance-based pooling.  In Proceedings of the IEEE International Conference on Computer Vision, pages 3355–3364, 2019.
  * [20] Ross Girshick.  Fast r-cnn.  In Proceedings of the IEEE international conference on computer vision, pages 1440–1448, 2015.
  * [21] Susanna Gladh, Martin Danelljan, Fahad Shahbaz Khan, and Michael Felsberg.  Deep motion features for visual tracking.  In 2016 23rd international conference on pattern recognition (ICPR), pages 1243–1248. IEEE, 2016.
  * [22] Erhan Gundogdu and A Aydın Alatan.  Good features to correlate for visual tracking.  IEEE Transactions on Image Processing, 27:2526–2540, 2018.
  * [23] Qing Guo, Wei Feng, Ce Zhou, Rui Huang, Liang Wan, and Song Wang.  Learning dynamic siamese network for visual object tracking.  In Proceedings of the IEEE international conference on computer vision, pages 1763–1771, 2017.
  * [24] Kensho Hara, Hirokatsu Kataoka, and Yutaka Satoh.  Can spatiotemporal 3d cnns retrace the history of 2d cnns and imagenet?  In CVPR, 2018.
  * [25] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [26] João F Henriques, Rui Caseiro, Pedro Martins, and Jorge Batista.  High-speed tracking with kernelized correlation filters.  IEEE Transactions on PAMI, 37:583–596, 2015.
  * [27] Seunghoon Hong, Tackgeun You, Suha Kwak, and Bohyung Han.  Online tracking by learning discriminative saliency map with convolutional neural network.  In International conference on machine learning, pages 597–606, 2015.
  * [28] Lianghua Huang, Xin Zhao, and Kaiqi Huang.  Got-10k: A large high-diversity benchmark for generic object tracking in the wild.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019\.
  * [29] Ziyuan Huang, Changhong Fu, Yiming Li, Fuling Lin, and Peng Lu.  Learning aberrance repressed correlation filters for real-time uav tracking.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2891–2900, 2019.
  * [30] B. Jiang, R. Luo, J. Mao, T. Xiao, and Y. Jiang.  Acquisition of localization confidence for accurate object detection.  In ECCV, 2018.
  * [31] Boyuan Jiang, MengMeng Wang, Weihao Gan, Wei Wu, and Junjie Yan.  Stm: Spatiotemporal and motion encoding for action recognition.  In Proceedings of the IEEE International Conference on Computer Vision, pages 2000–2009, 2019.
  * [32] Ilchae Jung, Jeany Son, Mooyeol Baek, and Bohyung Han.  Real-time mdnet.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 83–98, 2018.
  * [33] Hamed Kiani Galoogahi, Ashton Fagg, Chen Huang, Deva Ramanan, and Simon Lucey.  Need for speed: A benchmark for higher frame rate object tracking.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1125–1134, 2017.
  * [34] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [35] Okan Köpüklü, Xiangyu Wei, and Gerhard Rigoll.  You only watch once: A unified cnn architecture for real-time spatiotemporal action localization.  arXiv preprint arXiv:1911.06644, 2019.
  * [36] Matej Kristan, Ales Leonardis, Jiri Matas, Michael Felsberg, Roman Pflugfelder, Luka Cehovin Zajc, Tomas Vojir, Gustav Hager, Alan Lukezic, Abdelrahman Eldesokey, et al.  The visual object tracking vot2017 challenge results.  In Proceedings of the IEEE international conference on computer vision workshops, pages 1949–1972, 2017.
  * [37] Matej Kristan, Ales Leonardis, Jiri Matas, Michael Felsberg, Roman Pflugfelder, Luka ˇCehovin Zajc, Tomas Vojir, Goutam Bhat, Alan Lukezic, Abdelrahman Eldesokey, et al.  The sixth visual object tracking vot2018 challenge results.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 0–0, 2018.
  * [38] Matej Kristan, Jiri Matas, Ales Leonardis, Michael Felsberg, Roman Pflugfelder, Joni-Kristian Kamarainen, Luka Cehovin Zajc, Ondrej Drbohlav, Alan Lukezic, Amanda Berg, et al.  The seventh visual object tracking vot2019 challenge results.  In Proceedings of the IEEE International Conference on Computer Vision Workshops, pages 0–0, 2019.
  * [39] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan.  Siamrpn++: Evolution of siamese visual tracking with very deep networks.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 4282–4291, 2019.
  * [40] Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu.  High performance visual tracking with siamese region proposal network.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 8971–8980, 2018.
  * [41] Hanxi Li, Yi Li, and Fatih Porikli.  Deeptrack: Learning discriminative feature representations online for robust visual tracking.  IEEE Transactions on Image Processing, 25(4):1834–1848, 2015.
  * [42] Peixia Li, Boyu Chen, Wanli Ouyang, Dong Wang, Xiaoyun Yang, and Huchuan Lu.  Gradnet: Gradient-guided network for visual object tracking.  In Proceedings of the IEEE International Conference on Computer Vision, pages 6162–6171, 2019.
  * [43] Alan Lukezic, Jiri Matas, and Matej Kristan.  D3s-a discriminative single shot segmentation tracker.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7133–7142, 2020.
  * [44] Chao Ma, Jia-Bin Huang, Xiaokang Yang, and Ming-Hsuan Yang.  Hierarchical convolutional features for visual tracking.  In Proceedings of the IEEE international conference on computer vision, pages 3074–3082, 2015.
  * [45] Matthias Mueller, Neil Smith, and Bernard Ghanem.  A benchmark and simulator for uav tracking.  In European conference on computer vision, pages 445–461. Springer, 2016.
  * [46] Matthias Muller, Adel Bibi, Silvio Giancola, Salman Alsubaihi, and Bernard Ghanem.  Trackingnet: A large-scale dataset and benchmark for object tracking in the wild.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 300–317, 2018.
  * [47] Hyeonseob Nam and Bohyung Han.  Learning multi-domain convolutional neural networks for visual tracking.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4293–4302, 2016.
  * [48] Sanath Narayan, Hisham Cholakkal, Fahad Shahbaz Khan, and Ling Shao.  3c-net: Category count and center loss for weakly-supervised action localization.  In Proceedings of the IEEE International Conference on Computer Vision, pages 8679–8687, 2019.
  * [49] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, Alexander C. Berg, and Li Fei-Fei.  ImageNet Large Scale Visual Recognition Challenge.  International Journal of Computer Vision (IJCV), 115(3):211–252, 2015.
  * [50] Hasan Saribas, Bedirhan Uzun, Burak Benligiray, Onur Eker, and Hakan Cevikalp.  A hybrid method for tracking of objects by uavs.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 0–0, 2019.
  * [51] Karen Simonyan and Andrew Zisserman.  Two-stream convolutional networks for action recognition in videos.  In Advances in neural information processing systems, pages 568–576, 2014.
  * [52] Yuxuan Sun, Chong Sun, Dong Wang, You He, and Huchuan Lu.  Roi pooled correlation filters for visual tracking.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5783–5791, 2019.
  * [53] Ran Tao, Efstratios Gavves, and Arnold WM Smeulders.  Siamese instance search for tracking.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1420–1429, 2016.
  * [54] Zhu Teng, Junliang Xing, Qiang Wang, Congyan Lang, Songhe Feng, and Yi Jin.  Robust object tracking based on temporal and spatial deep networks.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1144–1153, 2017.
  * [55] Jack Valmadre, Luca Bertinetto, Joao Henriques, Andrea Vedaldi, and Philip HS Torr.  End-to-end representation learning for correlation filter based tracking.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2805–2813, 2017.
  * [56] Bing Wang, Li Wang, Bing Shuai, Zhen Zuo, Ting Liu, Kap Luk Chan, and Gang Wang.  Joint learning of convolutional neural networks and temporally constrained metrics for tracklet association.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops, pages 1–8, 2016.
  * [57] Guangting Wang, Chong Luo, Zhiwei Xiong, and Wenjun Zeng.  Spm-tracker: Series-parallel matching for real-time visual object tracking.  arXiv preprint arXiv:1904.04452, 2019.
  * [58] Lijun Wang, Wanli Ouyang, Xiaogang Wang, and Huchuan Lu.  Visual tracking with fully convolutional networks.  In Proceedings of the IEEE international conference on computer vision, pages 3119–3127, 2015.
  * [59] Naiyan Wang, Siyi Li, Abhinav Gupta, and Dit-Yan Yeung.  Transferring rich feature hierarchies for robust visual tracking.  arXiv preprint arXiv:1501.04587, 2015.
  * [60] Ning Wang, Yibing Song, Chao Ma, Wengang Zhou, Wei Liu, and Houqiang Li.  Unsupervised deep tracking.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1308–1317, 2019.
  * [61] Naiyan Wang and Dit-Yan Yeung.  Learning a deep compact image representation for visual tracking.  In Advances in neural information processing systems, pages 809–817, 2013.
  * [62] Qiang Wang, Jin Gao, Junliang Xing, Mengdan Zhang, and Weiming Hu.  Dcfnet: Discriminant correlation filters network for visual tracking.  arXiv preprint arXiv:1704.04057, 2017.
  * [63] Qilong Wang, Banggu Wu, Pengfei Zhu, Peihua Li, Wangmeng Zuo, and Qinghua Hu.  Eca-net: Efficient channel attention for deep convolutional neural networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11534–11542, 2020.
  * [64] Qiang Wang, Li Zhang, Luca Bertinetto, Weiming Hu, and Philip HS Torr.  Fast online object tracking and segmentation: A unifying approach.  In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 1328–1338, 2019.
  * [65] Philippe Weinzaepfel, Zaid Harchaoui, and Cordelia Schmid.  Learning to track for spatio-temporal action localization.  In Proceedings of the IEEE international conference on computer vision, pages 3164–3172, 2015.
  * [66] Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang.  Object tracking benchmark.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 37(9):1834–1848, 2015.
  * [67] Tianyang Xu, Zhen-Hua Feng, Xiao-Jun Wu, and Josef Kittler.  Joint group feature selection and discriminative filter learning for robust visual object tracking.  In Proceedings of the IEEE International Conference on Computer Vision, pages 7950–7960, 2019.
  * [68] Yuechen Yu, Yilei Xiong, Weilin Huang, and Matthew R Scott.  Deformable siamese attention networks for visual object tracking.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6728–6737, 2020.
  * [69] Kaihua Zhang, Lei Zhang, Qingshan Liu, David Zhang, and Ming-Hsuan Yang.  Fast visual tracking via dense spatio-temporal context learning.  In European conference on computer vision, pages 127–141. Springer, 2014.
  * [70] Zheng Zhu, Qiang Wang, Bo Li, Wei Wu, Junjie Yan, and Weiming Hu.  Distractor-aware siamese networks for visual object tracking.  In Proceedings of the European Conference on Computer Vision (ECCV), pages 101–117, 2018.
  * [71] Zheng Zhu, Wei Wu, Wei Zou, and Junjie Yan.  End-to-end flow correlation tracking with spatial-temporal attention.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 548–557, 2018.