# Transformers in Single Object Tracking: An Experimental Survey

Janani Thangavel111These authors contributed equally to this work.
<EMAIL_ADDRESS>Thanikasalam Kokul222These authors contributed
equally to this work<EMAIL_ADDRESS>Amirthalingam Ramanan
<EMAIL_ADDRESS>Subha Fernando<EMAIL_ADDRESS>
###### Abstract

Single-object tracking is a well-known and challenging research topic in
computer vision. Over the last two decades, numerous researchers have proposed
various algorithms to solve this problem and achieved promising results.
Recently, Transformer-based tracking approaches have ushered in a new era in
single-object tracking by introducing new perspectives and achieving superior
tracking robustness. In this paper, we conduct an in-depth literature analysis
of Transformer tracking approaches by categorizing them into CNN-Transformer
based trackers, Two-stream Two-stage fully-Transformer based trackers, and
One-stream One-stage fully-Transformer based trackers. In addition, we conduct
experimental evaluations to assess their tracking robustness and computational
efficiency using publicly available benchmark datasets. Furthermore, we
measure their performances on different tracking scenarios to identify their
strengths and weaknesses in particular situations. Our survey provides
insights into the underlying principles of Transformer tracking approaches,
the challenges they encounter, and the future directions they may take.

###### keywords:

Deep Learning , Tracking Review , Transformer Tracking , Vision Transformer ,
Visual Object Tracking

[1]organization=Department of Computational Mathematics, University of
Moratuwa, country=Sri Lanka [2]organization=Department of Interdisciplinary
Studies, University of Jaffna, country=Sri Lanka [3]organization=Department of
Computer Science, University of Jaffna, country=Sri Lanka

##  1 Introduction

Visual object tracking (VOT) algorithms are intended to estimate the state
(i.e., spatial location and size) of an object in a given video sequence.
Given the initial state of the target in the first frame of a video sequence,
these algorithms tracks the target’s states in the remaining frames. VOT can
be classified as single and multi-object tracking [1, 2] approaches based on
the number of targets to be tracked. In single-object tracking, a single
instance of an object class is tracked, while in multi-object tracking,
multiple instances of an object class are tracked.

Single object tracking algorithms became popular and gained interest in recent
years because of its wide range of applications in computer vision, including
video surveillance [3], augmented reality [4], automated driving [5], mobile
robotics [6], traffic monitoring [7], sports video analysis [8], scene
understanding [9], and human computer interaction [10]. Single object VOT
approaches captures the target’s appearance features in the first frame of a
video sequence and then use it to locate the target in the remaining frames.
Although many appearance-based approaches have been proposed in VOT, it is
still far from reaching the tracking robustness of humans with real-time speed
due to many challenges such as appearance and pose variations, occlusions,
motion blurring, background clutter, similar object distractors, and
deformation.

Figure 1: Performance comparison of state-of-the-art CNN-based, CNN-
Transformer based, and fully-Transformer based trackers on LaSOT benchmark
dataset. For some trackers, their reported tracking speed may be different
than the above graph since this comparison study is conducted on a NVIDIA
Quadro P4000 GPU with a 64GB of RAM.

Single object tracking algorithms can be classified and analysed in several
ways. Based on the features used in a tracking model, VOT approaches can be
categorized as hand-crafted and deep feature-based trackers. Hand-crafted
feature-based tracking approaches [11, 12, 13, 14, 15] extract the features
from images according to a certain manually predefined algorithm based on
expert knowledge. On the other hand, deep feature-based trackers [16, 17, 18,
19] capture the semantic cues from raw images by using the Convolutional
Neural Networks (CNNs) [20]. Because of the hierarchical learning capability,
deep features-based appearance trackers significantly outperform hand-crafted
feature-based trackers [21]. Based on how trackers differentiate the target
object from its surroundings, they can be discriminative and generative
trackers. Discriminative trackers [22, 23] treats the VOT as a binary
classification task and separate the target object from the background. On the
other hand, most of the generative trackers [24, 25] treat VOT as a similarity
matching problem by searching for the best candidate that closely matches the
reference template in each frame. In recent trackers, a two branch CNN
architecture, known as Siamese network [26], is utilized in similarity
matching tracking. Over the past few years, a large number of Siamese-based
trackers [27, 28, 29, 30, 31, 32, 33, 34] have been proposed in VOT since they
showed excellent performance with high computational efficiency. However,
discriminative capability of Siamese-based trackers are considerably poor
[35], since they only focus on learning a visual representation of the target
object to match the similarity while ignoring the background and target
specific information and hence showed low performance in occlusion and
deformation scenes.

Transformer [36] was introduced in the field of Natural Language Processing
(NLP) to capture the long range dependencies between input and output
sequences for machine translation tasks. Encoder and decoder stacks are the
two primary components of a Transformer architecture and they are used to
learn contextual information of the inputs and to generate the output
sequences, respectively. Based on great success of the Transformers in NLP
applications, researchers modified the Transformer architecture [37] to solve
the computer vision tasks. In the last three years, several Transformer
architectures are proposed and showed state-of-the-art performances in various
vision tasks such as object detection[38], image classification[37, 39],
semantic segmentation [40], and point cloud learning[41]. In addition, a few
review studies [42, 43, 44] have also been conducted to analyze the
performance of Transformers in vision tasks and to compare their performance
with state-of-the-art CNN architectures.

Although Transformers and CNNs are two types of artificial neural network
models, they differ from each other in several ways, including their
architecture, design principles, and performances. The CNNs and Transformers
consume images differently, with CNNs consuming images as arrays of pixel
values and Transformers consuming images as sequences of patches,
respectively. Based on the studies [42], Transformers are good at capturing
global information of an image than the state-of-the-art CNN models.
Furthermore, Transformers are better equipped to capture long-range
dependencies in an image without sacrificing computational efficiency. On the
other hand, increasing the size of convolutional kernels in CNNs can hinder
their learning capability. Researchers [37] also found that Transformers are
difficult to optimize and hence needs more training data than the CNN models.
Additionally, Transformers have a larger number of parameters, which can lead
to overfitting if not enough training data is available. However, once
trained, Transformers can produce testing outputs with fewer computational
resources than the corresponding CNN models, as they can process the data in a
parallel manner. In terms of transfer learning, Transformers shown promising
results and has been gaining popularity in recent years while CNNs are
generally better suited for small and medium-sized datasets. Overall,
Transformer models are replacing the CNNs in computer vision tasks because of
their attention mechanism and global feature capturing capabilities. Although
accuracy of Transformer is modest in small and medium-sized datasets, they are
expected to replace CNNs in all tasks in the coming years.

For the last three years, due to the great success of Transformers in other
computer vision tasks, a set of Transformer based VOT approaches [45, 46, 47,
48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66,
67, 68, 69] have been proposed and showed excellent performance on publicly
available single object tracking benchmark datasets. They have been proposed
as Siamese-based template matching trackers or discriminative approaches and
showed outstanding performances than the CNN-based trackers. In some early
approaches [45, 46, 47, 48, 49, 50], a hybrid-type tracking model is proposed
by combining the CNN and Transformer architectures together. As shown in Fig.
1, these hybrid-type trackers, called as CNN-Transformer based trackers,
showed better tracking robustness while maintaining considerable tracking
speed than the CNN-based trackers since they combined the attention mechanism
of Transformers with the hierarchical learning capability of CNNs. From last
year, researchers have initiated a new era in VOT by proposing a set of
trackers that rely solely on Transformer architecture. These trackers are
referred to as fully-Transformer based trackers, and they have demonstrated
exceptional tracking robustness compared to CNN-based and CNN-Transformer
based trackers because they successfully handle the information flow between
target template and search region features. Overall, the introduction of
Transformers in VOT opens up several new possibilities and perspectives.
Therefore, there is a need to analyze the performance of Transformers in
single object tracking to identify future research directions.

VOT research has been conducted for more than two decades and several
approaches have been proposed. Over these years, several review studies [35,
70, 21, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81] have been conducted to
analyze and compare the performance of VOT approaches. Most of the early
review studies [70, 72, 74, 77, 78, 79] were conducted to evaluate the
performance of hand-crafted feature-based trackers, while recent studies [35,
21, 71, 73, 75, 76, 81] focus solely on deep feature-based trackers.
Furthermore, while most studies [35, 70, 72, 73, 74, 75, 76, 79] only
summarize the literature or compare the reported results of tracking
approaches, a few studies [21, 77, 80, 71] perform an analysis of tracker
performance through experiments conducted on the same computing platform. To
the best of our knowledge, no survey studies have been conducted that include
Transformer-based trackers.

In recent years, several challenging benchmark datasets [82, 83, 84, 85] have
been introduced with various challenging attributes and performance measures.
To identify the future direction of VOT, it is necessary to experimentally
evaluate and compare all recently proposed trackers on the same computing
platform in order to avoid bias. Additionally, even though the reported
results of Transformer-based trackers are better than those of CNN-based
trackers, there is a need to evaluate their performance through detailed
experiments in order to identify research gaps and propose future tracking
models. With the aim of achieving these objectives, we have conducted an
experimental survey on recently proposed single-object trackers. This study
specifically focuses on trackers that utilize the Transformer in their
tracking pipeline. We have considered all the Transformer trackers that were
published in indexed journals and prestigious conferences.

This study makes the following notable contributions:

  1. 1.

We have conducted a comprehensive review of the literature on Transformer-
based tracking approaches. Non-Transformer based trackers were not included in
our literature review as they have already been covered by previous studies.

  2. 2.

We have experimentally evaluated and then compared the tracking robustness of
Transformers in VOT on five challenging benchmark datasets by categorizing and
analyzing the state-of-the-art trackers as fully-Transformer based trackers,
CNN-Transformer based trackers, and CNN-based trackers.

  3. 3.

The computational efficiencies of the state-of-the-art trackers were evaluated
on a common computing platform using the source code and models provided by
the authors for fair comparison.

  4. 4.

We have conducted the tracking attribute-wise evaluation on three benchmark
datasets to identify the most challenging scenarios for recent trackers.

  5. 5.

Based on the experimental findings, we provide the suggestions for future
directions of Transformer based single object tracking.

The rest of the manuscript is structured as follows: The previously conducted
review studies are discussed and the rationale behind this survey study is
established in Section 2. Section 3 provides an in-depth overview of the
Transformer architecture. The detailed literature review of Transformer-based
tracking approaches is presented in Section 4. The experimental results of
tracking robustness and computational efficiencies are presented in Section 5.
The summary of the findings and the proposed future directions for
Transformer-based tracking are discussed in Section 6. Finally, the paper is
concluded in Section 7.

##  2 Related Work

In this section, we justify the need for this study, despite the several
experimental surveys and review studies conducted in the past two decades for
single object tracking. We have summarized the previous studies in Table 1,
categorizing them by their review type and feature type.

Table 1: Summary of the review and experimental survey studies in single object tracking. Study | Year | Review Type | Feature Type
---|---|---|---
Literature | Reported result | Experimental | Attribute | Efficiency | Hand | Deep
Survey | based Analysis | Survey | Analysis | Analysis | crafted
Yilmaz _et al._ [72] | 2006 | ✓ | - | - | - | - | ✓ | -
Cannons _et al._ [79] | 2008 | ✓ | - | - | - | - | ✓ | -
Li _et al._ [70] | 2013 | ✓ | - | - | ✓ | - | ✓ | -
Deori _et al._ [74] | 2014 | ✓ | - | - | - | - | ✓ | -
Wu _et al._ [77] | 2015 | ✓ | - | ✓ | ✓ | ✓ | ✓ | -
Pflugfelder _et al._ [73] | 2017 | ✓ | ✓ | - | - | - | - | ✓
Li _et al._ [71] | 2018 | ✓ | - | ✓ | ✓ | ✓ | - | ✓
Soleimanitaleb _et al._ [75] | 2019 | ✓ | - | - | - | - | ✓ | ✓
Ondrašovič _et al._ [35] | 2021 | ✓ | ✓ | - | - | - | - | ✓
Marvasti-Zadeh _et al._ [21] | 2021 | ✓ | - | ✓ | ✓ | ✓ | - | ✓
Zhang _et al._ [76] | 2021 | ✓ | ✓ | - | - | - | ✓ | ✓
Chen _et al._ [80] | 2022 | ✓ | - | ✓ | ✓ | ✓ | ✓ | ✓
Javed _et al._ [81] | 2022 | ✓ | ✓ | - | - | - | - | ✓

During the period from 2006 to 2014, several review studies were conducted on
hand-crafted feature-based VOT approaches. However, since no benchmark
datasets were developed during that period, the evaluation of trackers was
performed on various video sequences using different performance metrics.
Therefore, comparing the performance of trackers is a challenging task, and as
a result, the review studies focused only on conducting literature surveys.
Yilmaz _et al._ [72] conducted a literature survey on object tracking by
classifying the trackers into different categories based on the target
representation, feature type, model learning algorithm, and searching
technique. Similar to that work, Cannons _et al._ [79] developed a detailed
technical report on visual tracking and conducted the literature survey by
classifying the trackers based on their target representation as point,
contour, and region trackers. Li _et al._ [70] performed a study on VOT by
providing a detailed literature survey. Also, they have compared a few
trackers based on the qualitative results of some tracking attributes such as
occlusion, illumination, and deformation. Deori _et al._ [74] conducted a
literature survey by categorising the trackers based on their searching
algorithms. Since deep learning-based trackers outperform hand-crafted
trackers by a large margin, these survey studies are no longer useful in the
present era.

The first experimental survey on single object tracking was conducted by Wu
_et al._ [77] based on the evaluation of thirty-one appearance based trackers.
They have developed a benchmark dataset, known as Object Tracking Benchmark
(OTB), and then evaluated the tracking performances and efficiencies of the
trackers based on their overall accuracies and tracking speed on a same
computing platform. In addition, trackers are evaluated and compared in eleven
challenging tracking attributes (scenarios). Although this study only
considers the hand-crafted feature based trackers, it is still followed by
many researchers due to the important of the OTB dataset.

In the last ten years, deep learning-based single object trackers have shown
excellent tracking performance compared to hand-crafted feature-based
trackers. Currently, only a few studies have been conducted to review the
literature on deep trackers. Soleimanitaleb _et al._ [75] analyzed the
literature of a few number of hand-crafted and deep trackers. Pflugfelder _et
al._ [73] only considered the Siamese based deep trackers and compared their
performances based on their reported results. Similar to their work,
Ondrašovič _et al._ [35] considered a detailed literature survey on Siamese
based deep trackers and then compared their tracking performances based on
their reported results on the well-known OTB100 [77], VOT2015 [86], VOT2016
[87], VOT2017 [88], VOT2018 [89] and GOT-10k [83] benchmark datasets.
Recently, Zhang _et al._ [76] conducted a survey by classifying the recent
trackers as deep trackers and Discriminative Correlation Filter (DCF) based
trackers. Also, based on the reported results of corresponding trackers, they
have compared the tracking performance and efficiency. Another similar survey
study is conducted by Javed _et al._ [81] by categorizing the tracking
approaches as Siamese based trackers and DCF trackers. These review studies
have analyzed the literature of the trackers by classifying them into
different categories and provided future directions for VOT. However, since
they did not conduct experimental evaluations, their reported results and
comparisons are not considered acceptable. Moreover, these survey studies did
not consider the efficiency comparison of trackers, even though it is an
important aspect in evaluation as the tracking speed is directly related to
the complexity of the tracker. Although some survey studies compared tracking
speeds based on reported results, this comparison is not considered acceptable
because the speed of a tracker is mainly dependent on the efficiency of the
GPU on which it was tested.

The first experimental survey on deep-learning-based trackers is conducted by
Li _et al._ [71] by evaluating the performance and efficiency of twenty-two
trackers on OTB100 and VOT2015 benchmarks. Also, they have compared the
individual performance of trackers based on eleven tracking attributes and
then identified the future directions. Recently, Marvasti-Zadeh _et al._ [21]
conducted a comprehensive and experimental survey on deep learning-based VOTs
based on the network architecture, tracking methods, long-term tracking,
aerial-view tracking, and online tracking ability of the trackers. In
addition, they have compared tracking benchmarks based on their challenging
attributes and conducted the tracking speed comparison of trackers. Similar to
their work, Chen _et al._ [80] conducted an experimental survey on deep and
hand-crafted trackers on recently developed benchmarks: OTB, VOT, LaSOT [82],
GOT-10k, and TrackingNet [84]. Although these experimental evaluation studies
reviewed the literature and compared the tracking performances of recently
proposed deep trackers in several aspects, they did not give much attention to
the comparison based on tracking efficiency.

In the last three years, similar to other computer vision tasks, language
models have played a crucial role in single object tracking. Transformer-based
language models have opened up new possibilities and have shown excellent
tracking performances and efficiencies compared to CNN-based deep trackers.
Since previous review studies have not included Transformer-based trackers, as
they were recently proposed, there is a need to categorize and review their
model architecture and to identify their strengths and weaknesses for the
future directions of single object tracking. To achieve this objective, we
conducted a detailed literature review on Transformer-based single object
trackers and conducted an experimental survey, which clearly showed that they
significantly outperformed CNN-based trackers by a large margin. In addition,
unlike many other review studies, we have also experimentally compared the
efficiency of recent trackers based on their tracking speed, number of
floating-point operations (FLOPs), and the number of parameters in the
tracking model. Furthermore, we have discussed the future directions of
Transformer-based tracking by identifying research gaps and providing
suggestions.

##  3 Transformer

In this section, we have covered the Transformer[36] and Vision Transformer
(ViT) [37] architectures, as they form the foundation for Transformer-based
single object trackers.

Transformer [36] was initially introduced in machine translation tasks and
then based on its great success and efficiency it was used in other NLP tasks
such as document summarisation and generation. The Transformer architecture is
based on an attention mechanism that enables the model to weigh the importance
of different parts of the input sequence during processing, leading to
improved information flow and the capturing of long-range dependencies. As
shown in Fig. 2, the Transformer architecture consists of encoder and decoder
components. The encoder component is made up of $N$ number of identical
encoder layers by stacked them on top of each other and the decoder component
is also made up of $N$ number of identical decoder layers, which are also
stacked on each other. Encoder and decoder components of a Transformer
architecture are illustrated in the left and right side of Fig. 2,
respectively. In the Transformer architecture, all encoder layers have two
sub-layers: a self-attention layer, and a fully connected feed-forward layer.
In addition to those two sub-layers, all decoder layers have an encoder-
decoder attention layer in the middle.

Figure 2: Architecture of the Transformer [36] model used in machine
translation task.

Transformer [36] receive the input as a vector sequence and use a positional
embedding algorithm to add information about the position of each token within
that sequence to its representation. After the embedding, the input data is
fed to the self-attention sub-layer of the encoder, as it helps to capture the
contextual relationships. On the other side, in the decoder layer, an encoder-
decoder attention sub-layer is used to concentrate on relevant parts of the
input data. The self-attention mechanism of the Transformer is described in
detail in Sections 3.1 and 3.2.

After the self-attention, a fully-connected feed forward layer is used to
learn the complex representation of attention features. It has a simple
architecture with two linear transformations and a non-linear activation in
between them. This layer can be described as two convolutions with the kernel
size of 1. In both encoder and decoder layers, residual connection is included
and then it is followed by a normalization layer. Residual connections are
used to preserve the cues from the original input data and to enable the model
to learn more accurate representations of the input data.

After the decoder layer stacks, a liner layer is used to produce the output
vectors. Finally, a Softmax layer is used to produce the probabilities of the
outputs.

###  3.1 Self-Attention

Transformer architectures are designed based on an attention concept known as
self-attention. It used to process input sequences in parallel, rather than in
a sequential manner like RNNs or LSTMs. In the Transformer architecture, self-
attention mechanism captures contextual relationships between input tokens by
calculating the attention weights between each element in the input sequence
and all other elements in the same sequence. Transformer used “Query”, “Key”,
and “Value” abstractions to calculate the attention of an input sequence.

In the first step of the self-attention computation, three distinct vectors: a
query vector $\mathbf{q}$, a key vector $\mathbf{k}$, and a value vector
$\mathbf{v}$, are created by multiplying an input vector ($\mathbf{x}$) with
three corresponding matrices: $W_{Q}$, $W_{K}$, and $W_{V}$, that were trained
in the training phase. Similarly, all input vectors are packed together to
form an input matrix $X$ and then the corresponding “Query”, “Key”, and
“Value” matrices: $Q$, $K$, and $V$ are generated, respectively.

As the second step of self-attention, a score matrix $S$ is computed for the
input $X$ by taking the dot product of $Q$ and $K$ as follows:

$S=Q\cdot K^{T},$ (1)

The score matrix $S$ provides how much attention should be given to other
parts of the input sequence based on the input vector at a particular
position. In the next step, score matrix $S$ is normalized to obtain more
stable gradients as follows:

$S_{n}=S/\sqrt{d_{\mathbf{k}}},$ (2)

where $S_{n}$ is the normalized score matrix, and $d_{\mathbf{k}}$ is the
dimension of the key vector. Then the SoftMax function is applied to the
normalized score matrix to convert the scores into probabilities ($P$) as
denoted in the following equation:

$P=\mathrm{SoftMax}(S_{n}),$ (3)

In the final step of self-attention, the obtained probabilities ($P$) are
multiplied with the value matrix $V$ to find the self-attention output values
($Z$) as denoted in the following equation:

$Z=P\cdot V,$ (4)

In summary, the entire self-attention mechanism can be unified into a single
equation as follows:

$\mathrm{Attention}(Q,K,V)=\mathrm{SoftMax}\left(\frac{Q\cdot
K^{T}}{\sqrt{d_{\mathbf{k}}}}\right)\cdot V$ (5)

The encoder-decoder attention sub-layer in the decoder layer is similar to the
self-attention sub-layer of the encoder except the key matrix $K$, and the
value matrix $V$ are obtained from the encoder block, and the query matrix $Q$
is obtained from its previous layer.

###  3.2 Multi-Head Self-Attention

The self-attention mechanism is not much capable of focusing on a specific
location of input without affecting the attention on other equally vital
locations at the same time. Therefore, the performance of self-attention
mechanism is boosted by using multiple heads and this technique is known as
multi-head self-attention.

Multiple sets of weight matrices ($W_{Q}$, $W_{K}$, and $W_{V}$) are used in
multi-head self-attention mechanism. They are randomly initialized and trained
separately since they are used to project the same input data into a different
subspace. Then, the attention function is computed concurrently on each of
these projected versions of queries, keys, and values to produce the
corresponding output values. At the final stage of multi-head self-attention,
outputs of all attention heads are concatenated and then multiplied with
another trainable weight matrix $W_{O}$ to obtain the multi-head self-
attention ($M_{\mathrm{atten}}$) as follows:

$M_{\mathrm{atten}}=\mathrm{Concat}(\mathrm{head}_{1},...,\mathrm{head}_{n})\cdot
W_{O}$ (6)

where $\mathrm{head}_{i}$ is the output of the attention head $i$. Multi-head
self-attention layer is used to concentrate on different positions of input
and to represent the input into different subspaces.

###  3.3 Vision Transformer (ViT)

Figure 3: Architecture of the Vision Transformer (ViT) [37], proposed for
image recognition task.

Based on the success of Transformers in NLP tasks, several researchers have
attempted to apply them in computer vision tasks and have proposed various
architectures. Among these models Vision Transformer (ViT) [37] is more
efficient than others with a simple architecture as demonstrated in Fig. 3.

In the initial step of ViT, an input image $I\in\mathbb{R}^{H\times W\times
C}$ is split as equal size patches with the size of $P\times P$. Here, $H$,
$W$, and $C$ represent the height, width, and number of channels of the input
image, respectively. Then the patches are flattened to form a 2D sequence with
the size of $(N\times(P^{2}\cdot C))$, where $N$ is the number of extracted
patches from an input image. In the next stage of ViT architecture, patches
are embedded with position and class information. Then the embedded patches
are fed to a set of encoder layers in a sequence manner. The output of the
encoder layers is then obtained and fed to a Multi-Layer Perceptron (MLP)
network to produce the class-specific scores. The encoder layers of ViT are
much similar to the Transformer [36] architecture except GELU [90] non-linear
function is used instead of ReLU function.

ViT, when trained on large datasets, has demonstrated superior results
compared to state-of-the-art CNN models. Also, authors of ViT tested it in
small and medium-sized datasets with fine-tuning and showed modest results.
After the success of ViT in image recognition, several vision Transformer
models have been proposed and it has been used in other computer vision tasks.
To reduce the computational complexity of ViT, Swin Transformer [91] performs
self-attention locally within non-overlapping windows that partition an image
and introduces a shifting window partitioning mechanism for cross-window
connections. Unlike fixed-size tokens in ViT, Swin Transformer constructs a
hierarchical representation by starting from small-sized patches and then
gradually merge the neighboring patches in deeper Transformer layers for
multi-scale prediction to overcome the scaling problem. Since pure Transformer
models are poor to capture the local information, CVT [92] incorporates two
convolution-based operations into the ViT architecture, namely convolutional
token embedding, and convolutional projection. The predefined positional
embedding scheme of the ViT is replaced by a conditional position embedding in
the CPVT [93] Transformer architecture. The TNT [94] Transformer further
subdivides a $16\times 16$ image patch into $4\times 4$ sub-patches using a
Transformer-in-Transformer framework. Inner Transformer blocks and outer
Transformer blocks are used in TNT to capture the interaction between sub-
patches and the relationship between patches, respectively. Since ViT is less
efficient at encoding finer-level features, VOLO [95] introduces light-weight
attention mechanism called Outlooker to encode the token representations with
finer-level information efficiently. Overall, ViT has ushered in a new era in
computer vision tasks.

##  4 Transformer in Single Object Tracking

Based on the model architecture, feature extraction, and feature integration
techniques, recent deep trackers can be classified as three categories: CNN-
based trackers [29, 96, 97, 98, 31, 99, 32, 100, 101, 34, 33, 102], CNN-
Transformer based trackers [45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56]
and fully-Transformer based trackers [57, 58, 59, 60, 61, 62, 63]. CNN-based
trackers rely solely on a CNN architecture for feature extraction and target
detection, while CNN-Transformer based trackers and fully-Transformer based
trackers partially and fully rely on a Transformer architecture, respectively.
The literature on CNN-Transformer based trackers and fully-Transformer based
trackers is reviewed in this section.

Table 2: Summary of CNN-Transformer based and fully-Transformer based trackers.  Tracker |  Year |  Type |  Backbone |  Template Update |  Training Dataset |  Training Scheme
---|---|---|---|---|---|---
HiFT [50] |  2021 |  CNN-Transformer |  AlexNet |  N |  COCO, ImageNet VID, GOT-10k, Youtube-BB |  cross-entropy loss, binary cross-entropy loss, IoU loss, SGD
STARK [47] |  2021 |  CNN-Transformer |  ResNet-101 |  Y |  COCO, LaSOT, GOT-10k, TrackingNet |  binary cross-entropy loss, L1 loss, GIoU loss, AdamW
TransT [46] |  2021 |  CNN-Transformer |  ResNet-50 |  N |  COCO, LaSOT, GOT-10k, TrackingNet |  binary cross-entropy loss, L1 loss, GIoU loss, AdamW
DTT [49] |  2021 |  CNN-Transformer |  ResNet-50 |  Y |  COCO, LAaSOT, GOT-10k, TrackingNet |  cross-entropy loss, IoU loss
TrDiMP [45] |  2021 |  CNN-Transformer |  ResNet-50 |  Y |  COCO, LaSOT, GOT-10k, TrackingNet |  L2 norm loss, AdamW
TrTr [48] |  2021 |  CNN-Transformer |  ResNet-50 |  N |  Youtube-BB, ImageNet VID, GOT-10k, LaSOT, COCO |  focal loss, L1 loss ,AdamW
DualTFR [57] |  2021 |  fully-Transformer |  Custom |  N |  ImageNet-1K, COCO, LaSOT, GOT-10k, TrackingNet |  cross-entropy loss, GIoU, L1 loss, AdamW
SwinTrack [58] |  2021 |  fully-Transformer |  Swin Transformer |  N |  COCO, LaSOT, GOT-10k, TrackingNet |  varifocal loss, GIoU loss, AdamW
ProContEXT [63] |  2022 |  fully-Transformer |  MAE-ViT |  Y |  COCO, LaSOT, GOT-10k, TrackingNet |  focal loss, IoU loss, L1 Loss, AdamW
OSTrack [62] |  2022 |  fully-Transformer |  MAE-ViT |  N |  COCO, LaSOT, GOT-10k, TrackingNet |  L1 loss, GIoU loss, Gaussian weighted focal loss, AdamW
MixFormer [60] |  2022 |  fully-Transformer |  CVT |  Y |  COCO, LaSOT, GOT-10k, TrackingNet |  L1 loss, GIoU loss, cross-entropy, AdamW
SimTrack [61] |  2022 |  fully-Transformer |  ViT |  N |  COCO, LaSOT, GOT-10k, TrackingNet |  L1 loss, GIoU loss, AdamW
SparseTT [59] |  2022 |  fully-Transformer |  Swin Transformer |  N |  COCO, LaSOT, GOT-10k, TrackingNet, ILSVRC VID, ILSVRC DET |  focal loss, IoU loss, AdamW
ToMP [51] |  2022 |  CNN-Transformer |  ResNet-101 |  Y |  COCO, LaSOT, GOT-10k, TrackingNet |  GIoU loss, AdamW
CSWinTT [55] |  2022 |  CNN-Transformer |  ResNet-50 |  Y |  LaSOT, GOT-10k, TrackingNet |  L1 loss, GIoU loss, cross-entropy, AdamW
AiATrack [56] |  2022 |  CNN-Transformer |  ResNet-50 |  Y |  COCO, LaSOT, GOT-10k, TrackingNet |  L1 loss, GIoU loss, AdamW
CTT [54] |  2022 |  CNN-Transformer |  ResNet-50 |  N |  COCO, LaSOT, GOT-10k, TrackingNet |  cross-entropy loss, GIoU loss, L1 loss,AdamW
UTT [53] |  2022 |  CNN-Transformer |  ResNet |  N |  COCO, LaSOT, GOT-10k, TrackingNet |  GIoU loss, L1 loss, AdamW
SiamTPN [52] |  2022 |  CNN-Transformer |  ShuffleNetV2 |  N |  COCO, LaSOT, GOT-10k, TrackingNet |  cross-entropy loss, GIoU loss, L1 loss, AdamW
BANDT [64] |  2023 |  CNN-Transformer |  ResNet-50 |  Y |  COCO, LaSOT, GOT-10k, TrackingNet |  cross-entropy loss, GIoU loss, Focal loss, smooth L1 loss, AdamW
GRM [67] |  2023 |  fully-Transformer |  MAE-ViT |  N |  COCO, LaSOT, GOT-10k, TrackingNet |  GIoU loss, Focal loss, L1 loss, AdamW
DropTrack [65] |  2023 |  fully-Transformer |  DropMAE-ViT |  N |  COCO, LaSOT, GOT-10k, TrackingNet |  L1 loss, GIoU loss, Gaussian weighted focal loss, AdamW
MATTrack [103] |  2023 |  fully-Transformer |  MAT-ViT |  N |  COCO, LaSOT, GOT-10k, TrackingNet, VID |  L1 loss, GIoU loss, AdamW
VideoTrack [66] |  2023 |  fully-Transformer |  MAE-ViT |  Y |  COCO, LaSOT, GOT-10k, TrackingNet |  -
SeqTrack [69] |  2023 |  fully-Transformer |  MAE-ViT |  Y |  COCO, LaSOT, GOT-10k, TrackingNet |  cross-entropy loss AdamW
ARTrack [68] |  2023 |  fully-Transformer |  ViT |  N |  LaSOT, GOT-10k, TrackingNet |  cross-entropy loss, SIoU loss, AdamW

Generally, Transformer architectures require a massive number of training
samples [37] to train their models. Since the target is given in the first
frame of a tracking sequence, obtaining a large number of samples is not
possible in VOT and hence all of the fully-Transformer based and CNN-
Transformer based trackers use a pre-trained network and considered it as the
backbone model. In addition, some of these trackers update the target template
to capture the temporal cues during tracking, while others do not. Moreover,
they trained their models on various benchmark datasets such as COCO [104],
LaSOT, GOT-10k, TrackingNet, and Youtube-BB [105]. We have summarized these
details in Table 2 for all Transformer-based and CNN-Transformer based
approaches, providing information on their backbone network, template update
details, training dataset, and training scheme details.

Figure 4: General architectures of CNN-Transformer, One-stream One-stage
fully-Transformer, and Two-stream Two-stage fully-Transformer based trackers.
Although ARTrack [68] and SeqTrack [69] belong to the One-stream One-stage
category, they are capable of predicting the target without the need for a
prediction head.

We have conducted a literature review of CNN-Transformer based and fully-
Transformer based trackers, focusing on their model architecture. As shown in
Fig. 4, CNN-Transformer based trackers used a CNN backbone for feature
extraction and then used a Transformer architecture for feature integration.
Fully-Transformer based trackers can be classified further as ”Two-stream Two-
stage” trackers, and ”One-stream One-stage” trackers. In Two-stream Two-stage
trackers, two identical pipeline of network branches (two-stream) are used to
extract the features of the target image and search image. Also, in this
category of trackers, feature extraction and feature fusion of target template
and search region are done in two distinguishable stages (two-stage). On the
other side, in One-stream One-stage trackers, a single pipeline of networks
are used and the feature extraction and feature fusion are done together
through a single stage. Fig. 5 shows the classification of CNN-Transformer
based, and fully-Transformer based trackers.

ptTransformers in VOT CNN-Transformer based Trackers TrDiMP [45], DTT [49],
HiFT [50], TransT [46], STARK [47], ToMP [51], CSWinTT [55], AiATrack [56],
SiamTPN [52], UTT [53], TrTr [48], CTT [54], BANDT [64] Fully-Transformer
based Trackers Two-stream Two-stage Trackers DualTFR [57], SwinTrack [58],
SparseTT [59] One-stream One-stage Trackers MixFormer [60], SimTrack [61],
OSTrack [62], ProContEXT [63], GRM [67], DropTrack [65], MATTrack [103],
VideoTrack [66], SeqTrack [69], ARTrack [68]

Figure 5: Classification of CNN-Transformer and fully-Transformer trackers
based on their model architecture and tracking pipeline. Multiple pipeline of
networks are used in ‘Two-stream Two-stage’ trackers while single pipeline of
networks are used in ’One-stream’ trackers. Also, feature extraction and
feature fusion are done through two distinguishable stages in ‘Two-stage’
trackers while they are done together in ‘One-Stage’ trackers.

###  4.1 CNN-Transformer based Trackers

Most of the recent CNN-based trackers [29, 96, 97, 98, 31, 99, 32, 100, 101,
34, 33, 102] followed a Siamese network architecture by using two identical
pipelines of CNNs. In these trackers, features of the target template and
search region are extracted by using two identical CNN branches. Then the
target localization is done by finding the similarity of target’s features in
search region’s features using a correlation function. Although the
correlation operation is simple and fast for feature similarity matching
process, it is not good enough to capture the non-linear interaction
(occlusion, deformation, and rotation) between the target template and search
region and hence the tracker’s performance is limited. To successfully address
this issue, researchers have started incorporating the Transformer for the
feature fusion process, resulting in trackers known as CNN-Transformer based
trackers.

Similar to most of the CNN-based trackers, CNN-Transformer based trackers also
use two Siamese-like identical pipeline of networks. In the beginning of these
pipelines, target template’s and search region’s features are extracted using
a CNN backbone. Then the extracted deep features are flattened into vectors
and thereafter fed to a Transformer to capture the similarity features of the
target in the search region.

Figure 6: Architecture of the first CNN-Transformer based tracker[45] with
TrSiam and TrDiMP branches. Target template set’s features and search region
features are fed to the Transformer encoder and decoder to locate the target.

The first CNN-Transformer based tracker was proposed by Wang _et al._ [45] by
introducing a Transformer into both generative and discriminative tracking
paradigm. In their Siamese-like tracking architecture, a set of template
patches and search region are fed to a CNN backbone to extract the deep
features. Then, as shown in Fig. 6, extracted template features are fed to the
Transformer’s encoder to capture the high-quality target’s features using the
attention mechanism. Similarly, search region features are fed to the
Transformer decoder to produce the decoded features by aggregating the
informative target cues from previous frames with the search region features.
In the TrSiam pipeline, similar to the SiamFC[24] tracker, the target feature
is cropped from the encoded feature, and then cross-correlated with the
decoded features to locate the target location. In the TrDiMP pipeline, end-
to-end Discriminative Correlation Filters(DCF) [97] are applied on the encoded
features to generate a target model, which is then used to convolve with
decoded features to locate the target in the search region. Since a collection
of target template features is fed to the Transformer of this tracker, it is
able to locate the target even with severe appearance changes.

Similar to the TrDiMP [45] tracker, Yu _et al._ [49] introduced the encoder-
decoder Transformer architecture [36] in VOT and their tracker is referred to
as DTT. They have also used a Siamese-like tracking pipeline and extracted the
deep features using a backbone CNN architecture. In their tracking model,
target templates are fed with background scenes and then the Transformer
architecture captures the most discriminative cues of the target. Since their
approach involves conducting the tracking without the need to train a separate
discriminative model, it is simple and has demonstrated high tracking speed in
benchmark datasets.

Recently, Yang _et al._ [64] found that the TrDiMP [45] tracker is
computationally expensive since it computes the attention in all possible
spatial locations of the target template feature, and hence the model weights
are increased with respect to the pixel counts of the feature maps. To address
this issue, they proposed a border-aware tracking framework with deformable
Transformers, referred to as the BANDT tracker. Instead of computing attention
among all possible locations of the target template features, the BANDT
tracker only calculates attention in a few specific locations around the
target boundary. Based on the inspiration from the Deformable DETR Transformer
[106], the BANDT tracker extracts a small set of key sampling points around
the border of the target object in the template image. Since the BANDT tracker
enhances the border features of the target object, it shows better performance
than the TrDiMP tracker with fewer computational costs.

Figure 7: Architecture of the TransT tracker [46]. It has a CNN backbone
network, a Transformer network, and a prediction network for deep feature
extraction, feature fusion, and target localization, respectively.

Another Siamese-like architecture is proposed in TransT [46] tracker. As
demonstrated in Fig. 7, TransT tracker has three modules: a CNN backbone
network, a Transformer based feature fusion network, and a prediction network.
Similar to other CNN-Transformer based trackers, target template’s and search
region’s features are extracted using the ResNet50 [107] model. Then these
features are reshaped using a $1\times 1$ convolutional layer and fed to the
feature fusion network. The Transformer-based feature fusion network has a set
of layers, and each layer has an ego-context augment module (ECA) and a cross-
feature augment (CFA) module to enhance the self-attention and cross-
attention, respectively. Finally, fused features are fed to a prediction head,
and it locates the target and find the coordinates using a simple
classification and regression branches, respectively. TransT tracker showed
excellent performance than the CNN-based trackers by utilizing the Transformer
for feature fusion instead of the correlation matching of the previous
approaches.

Several CNN-Transformer trackers are proposed by training the Transformer to
capture the relationship between the target template and search region
features. The TrTr tracker [48] trained a Transformer to capture the global
information of the target template and then used that cues to find the
accurate correlation between target and search region. Another similar
approach is proposed by Xiuhua _et al._ [108] with a template update
mechanism. Recently, Zhong _et al._ [54] proposed a tracker referred to as
CTT, and improved the feature fusion between target and search region features
by including a correlation module into the Transformer architecture. CTT
tracker avoids the background interference in long-term tracking by using
their correlation based Transformer architecture.

Figure 8: Architecture of the STARK tracker [47]. It captures the spatial and
temporal information of the target using the Transformer.

In contrast to the previously mentioned CNN-Transformer based trackers, Yan
_et al._ [47] proposed a new Transformer architecture for VOT based on the
DETR [38] object detection Transformer. Their tracker is referred to as STARK
and they have trained the Transformer to capture the spatial and temporal cues
of the target object. ResNet [107] is used to extract the deep features of the
initial target template, dynamic target template, and search regions. Then, as
shown in Fig. 8, these features are flattened, concatenated, and then fed to a
Transformer with encoder and decoder architectures. The STARK tracker’s
Transformer consists of a set of encoder layers, with each layer containing a
multi-head self-attention module and a feed-forward network module. The
encoder captures the feature dependencies between every element in the
tracking sequence, and reinforces the original features with global contextual
information. This enables the model to learn discriminative features for
object localization. The decoder of the tracker learns a query embedding to
predict the spatial positions of the target object by using the Transformer-
based detection approach of the DETR. To determine the target’s bounding box
in the current frame, STARK introduced a corner-based prediction head.
Additionally, a score head is learned to control the updates of the dynamic
template images. Since the architecture of the STARK is simple and capable of
capturing the spatio-temporal information of the target, it has demonstrated
better tracking robustness and good tracking speed compared to other CNN-based
trackers.

Similar to the STARK tracker, Mayer _et al._ [51] proposed a tracker called
ToMP, utilizing the DETR object detection Transformer. In this tracker, the
target template and search region features from the CNN backbone are
concatenated and jointly processed by the Transformer encoder and decoder.
Additionally, similar to the DiMP [97] tracker, the ToMP tracker incorporates
a target model for localizing the target. However, unlike DiMP, the weights of
the target model are obtained using the Transformer architecture instead of a
discriminative correlation filter. According to the reported experimental
results, the ToMP tracker demonstrates superior tracking performance and speed
compared to the STARK tracker.

Figure 9: Architecture of the HiFT tracker[50]. It extracted the multi-level
convolutional features of the target template and search region and then fed
the high-resolution features to encoder and low-resolution features to
decoder.

Most of the CNN-Transformer based trackers [47, 51, 46, 49, 45] utilized the
deep features from the last convolutional layer of a backbone CNN. Different
from these trackers, as illustrated in Fig. 9, Cho _et al._ [50] utilized the
multi-level convolutional features of the target template and search region
from the last three layers (3 to 5) of the AlexNet [20] model. Their tracker
is referred to as HiFT and it was originally proposed for aerial tracking. In
this tracker, high resolution cross-correlated features are fed to the
Transformer’s encoder to capture the spatial cues of the target with different
scales. Also, low resolution cross-correlated features of the last
convolutinal layers of the target template and search region are fed to the
Transformer’s decoder to capture the semantic features. Finally, a prediction
head is applied on the Transformer features to locate the target state. HiFT
tracker showed better performance than the CNN-based trackers because of its
strong discriminative capability.

Similar to HiFT tracker, Xing _et al._ [52] proposed an aerial tracking
approach, referred to as SiamTPN, by utilizing the multi-level convolutional
features of a backbone CNN. However, their Transformer architecture is
different than the HiFT tracker since they used a Transformer Pyramid Network
(TPN) to fuse the multi-level features of the target and search region. In
addition, instead of the multi-head self attention layer, a pooling attention
(PA) layer is used in the Transformer architecture of this tracker to reduce
the computational complexity and memory load. Based on the reported results,
SiamTPN tracker is able to track a target in real-time speed on a CPU.

Figure 10: Architecture of the CSWinTT tracker [55]. Instead of pixel-level
attention, this tracker considered the window-level attention by fed the
flattened window feature pairs to the Transformer.

All of the early CNN-Transformer based approaches [47, 51, 46, 49, 45, 50, 52]
extracted the deep features of the target template and search region using a
CNN backbone and then flattened them to fed a Transformer network. Although
the Transformer captures the pixel-wise attention from the flattened features,
the valuable object-level cues from the relative pixel positions are lost.
Recently, Song _et al._ [55] noticed this issue and proposed a multi-scale
cyclic shifting window Transformer tracker (CSWinTT) to tackle this issue.
Similar to other CNN-Transformer based trackers, CSWinTT also extracted the
deep features of the target template and search region using the ResNet-50.
Thereafter, as shown in Fig. 10, these features are split as small windows and
then each windows are flattened and then fed to the Transformer to capture the
window-wise attention. In addition, a multi-head multi-scale attention module
is used in the Transformer of the CSWinTT to find the relevance between target
template and search region windows at a particular scale. Moreover, a cyclic
shifting technique is proposed in the window-wise attention module of the
CSWinTT to fuse the outputs of each attention heads with different scales.
Although the cyclic shifting and window-wise attention are computationally
expensive, CSWinTT approach showed excellent tracking performance than the
other CNN-Transformer based trackers because it considered the window-level
attention instead of the pixel-level attention.

The first CNN-Transformer based unified tracker is proposed by Ma _et al._
[53] and it is referred to as UTT. This unified tracker provided a single
model architecture for single object tracking and multi object tracking to
track the single and multiple targets together. In this tracker, deep features
of the target template and search region, obtained from a backbone CNN, are
fed to a Transformer to locate a target. In this UTT approach, target-specific
features are extracted by using a decoder component of the Transformer.
Similarly, a proposal decoder component is used to extract the search region
features with respect to the target. Finally, these two features are fed to a
Transformer to predict the coordinates of the target in the search region.
Experimental results showed that UTT approach successfully handles the single
and multi object tracking through a single tracking architecture.

Figure 11: Architecture of the AiATrack approach [56]. An attention-in-
attention (AiA) module is used in this tracker to enrich the cross-correlation
attention by searching for consensus among all correlation vectors and hence
ignores the noisy correlations.

In most of the CNN-Transformer based trackers, CNN features of the target
template and search regions are enriched using the self-attention mechanism.
Then, the relationship between them is computed using the cross-attention
modules. Based on the correlation between each query-key pair, the highest
correlated pair is selected and then used to identify the target in the search
region. Gao _et al._ [56] found that if a key has high correlation with a
query and the neighbors of that key has less correlation with that particular
query then that correlation should be a noise. Based on this finding, as
illustrated in Fig. 11, they have proposed an attention-in-attention (AiA)
module by including an inner attention module in Transformer architecture and
their tracker is referred to as AiATrack. The proposed inner attention module
of the AiATrack approach improves the attention by searching for consensus
among all correlation vectors and ignores the noisy correlations. In addition,
AiATrack approach introduces a learnable target-background embedding to
differentiate the target from the background while maintaining the contextual
information. Because of the enhanced attention mechanism, AiATrack approach
successfully avoid the distractor objects and hence improves the tracking
performance.

In summary, CNN-Transformer based trackers extracted the deep features of the
target template and search region using a CNN backbone such as ResNet or
AlexNet. Then the relationship between these features is computed using the
attention mechanism of a Transformer. Finally, a prediction head is employed
to locate the target using the features generated by the Transformer. CNN-
Transformer based trackers are successfully outperforms the CNN-based trackers
since they have used a learnable Transformer instead of the linear cross-
correlation operation. Although, a few early trackers borrowed the Transformer
architectures from the object detection task and used them without any
modification, recent approaches identified the Transformer based tracking
issues and then modified their architectures accordingly. In overall, CNN-
Transformer based trackers showed excellent tracking performance than the CNN-
based trackers.

###  4.2 Fully-Transformer Based Trackers

Although CNN-Transformer based trackers utilize the attention mechanism of
Transformers for the feature integration of the target template and search
region, they still rely on convolutional features as they use a backbone CNN
for feature extraction. Since CNNs capture features through local
convolutional kernels, CNN-Transformer based trackers struggle to capture
global feature representations.

Due to the success of the fully-Transformer architectures in other computer
vision tasks, researchers start using them in single object tracking. Based on
the tracking network formulation, we classified the fully-Transformer trackers
as Two-stream Two-stage trackers and One-stream One-stage trackers and
reviewed their literature in the following subsections.

####  4.2.1 Two-stream Two-stage Trackers

Two-stream Two-stage trackers consist of two identical and individual
Transformer-based tracking pipelines to extract the features of the target
template and search region. Another Transformer network is then employed to
find the relationship between these features. Finally, a prediction head is
utilized to locate the target by using the attended features.

Figure 12: Architecture of the DualTFR tracker [57]. It has a set of local
attention blocks and a global attention block to capture the attention in a
small sized window and whole image, respectively. At the end of the network, a
cross-attention block is used to fuse the features of both branches.

The first fully-Transformer based Two-stream Two-stage tracker is proposed by
Xie _et al._ [57], and it is referred to as DualTFR. In this tracker, template
and search region images are split as tokens and then fed to the corresponding
feature extraction branches. DualTFR has a set of local attention blocks (LAB)
in feature extraction branches to capture the attention in a small size
window. Then the extracted features are fed to a global attention block (GAB)
to capture the long range dependencies. Finally, as demonstrated in Fig. 12,
output features of both branches are fed to a cross-attention block to compute
the attention between target template and search region. Since the LAB
computes attention within a small window of tokens on the high-resolution
feature maps and the GAB computes the attention among all tokens of the same
picture on the low-resolution feature maps, DualTFR tracker successfully
achieved the high accuracy while maintaining above real-time speed.

Lin _et al._ [58] also used a two-branch fully-Transformer architecture for
VOT and it is referred to as SwinTrack. They have utilized the Swin
Transformer [91] for feature extraction since it showed better performance
with fewer computational cost in object detection. In Swin Transformer, images
and features are partitioned into non-overlapping windows and then the self-
attention is computed within the window. Also, windows are shifted in the next
layers to maintain the connectivity. After the feature extraction using the
Swin Transformer, features of the template and search region are concatenated
as shown in Fig. 13. Then the concatenated features and their corresponding
positional embedding are fed to the Transformer encoder to enrich them using
the self-attention mechanism. Then a Transformer decoder block is used to find
the relationship between the template and search region features using the
cross-attention mechanism. Finally, the target is located by feeding the
features to a prediction head, which has a binary classifier and a bounding
box regression module. Based on the tracking accuracy, SwinTrack outperformed
CNN-Transformer based trackers, and CNN-based trackers in several benchmark
datasets.

Figure 13: Architecture of the SwinTrack tracker [58]. It utilized the Swin
Transformer [91] for feature extraction. Then the features are concatenated
with their positional encoding and then to fed to the encoder and decoder to
enhance them using self-attention and cross-attention, respectively.

While the self-attention mechanism of the Transformer captures long-range
dependencies, it may not adequately capture target-specific cues, making it
susceptible to distraction by background clutter. Fu _et al._ [59] noticed
this issue and then proposed a tracker, referred to as SparseTT, by using a
sparse attention module to provide the target-specific attention in the search
region features. In this tracker, the Swin Transformer is used to extract the
target template and search region features and then fed them to a sparse
Transformer. Encoder blocks of the sparse Transformer are similar to the other
Transformer architectures. Sparse multi-head self-attention modules are used
in the decoder blocks to focus the foreground regions. Finally, enhanced
features are fed to a double-head prediction network to detect the target with
bounding box coordinates.

In overall, Two-stream Two-stage fully-Transformer trackers used a backbone
Transformer model to extract the features of the target template and search
region. Another Transformer is used to fuse and enhance the extracted futures.
Finally, a small prediction network is used to locate the target. All of the
Two-stream Two-stage trackers have a simple and neat tracking architecture and
showed better performance than the CNN-based trackers and CNN-Transformer
based trackers since they fully utilized the attention mechanism of the
Transformer for tracking a target.

####  4.2.2 One-stream One-stage Trackers

Figure 14: Architecture of the MixFormer Tracker [60]. It has a set of Mixed
Attention Modules (MAM) to simultaneously extract and integrate the features
of the target template and search region and to extract the target-specific
discriminative cues.

One-stream One-stage Trackers have a single pipeline of fully-Transformer
based network architecture. Also, feature extraction and feature fusion
process are done in a single stage in these approaches instead of two stages
in the previously mentioned tracking methods.

Recently, Cui _et al._ [60] found that combining the feature extraction and
feature fusion processes is important for object tracking, as it enables
extraction of more target-specific cues in the search region and improves
correlation. Based on this fact, they proposed a fully-Transformer based One-
stream One-stage tracker referred to as MixFormer. As illustrated in Fig. 14,
a set of Mixed Attention Modules (MAM) are used in the MixFormer tracker to
simultaneously extract and integrate the features of the target template and
search region. MixFormer Tracker consumes multiple target templates and search
regions are the inputs and locate the target using a simple fully-
convolutional based prediction head network. Also, MixFormer utilized the pre-
trained CVT Transformer [92] to design the MAM modules since CVT is excellent
to capture the local and global feature dependencies in an image. Instead of
the self-attention mechanism of the CVT, MAM employs a dual attention
mechanism on target template and search region tokens to capture the target-
specific and search-specific cues, respectively. In addition, an asymmetric
mixed attention technique is used in the MAM modules to reduce the
computational cost by eliminating the unnecessary cross-attention between the
tokens of the target and search region. Based on the reported results,
MixFormer showed excellent tracking performance in benchmark datasets.
However, MixFormer showed poor tracking speed since the MAM modules are
computationally inefficient.

Figure 15: Architecture of the SimTrack Approach [61]. In this tracker, a
target template, a search region, and an exact target region from the template
are split as tokens and then fed to the ViT [37] backbone to locate the
target.

Another One-stream One-stage tracker is proposed by Chen _et al._ [61] and it
is referred to as SimTrack. In this tracker, as shown in Fig. 15, the pre-
trained ViT [37] model is utilized as the backbone Transformer to combine
feature extraction and fusion. In SimTrack approach, target template and
search region are split as a set of tokens, concatenated, and then fed to the
backbone Transformer with their positional embedding. Since the target
template tokens contain some background regions due to the splitting process,
SimTrack followed a foveal windowing technique to accurately capture the
target-specific cues. In the foveal windowing technique, a smaller region of
the template image is cropped with the target in middle and then serialized
into image tokens. In addition to the tokens of target template and search
region, foveal sequence also fed to the Transformer to capture more target-
specific features.

OSTrack [62] is another One-stream One-stage approach that combines the
feature learning and feature fusion processes using the ViT backbone, as
demonstrated in Fig. 16. OSTrack utilizes a self-supervised learning-based
Masked Auto Encoder (MAE) [109] pre-trained model to initialize the ViT
backbone. The authors of OSTrack have found that some of the tokens from the
search image contain background information, and hence including these tokens
in the tracking process is unnecessary. Based on this fact, OSTrack includes
an early candidate elimination module in some of the encoder layers to remove
tokens containing background information. Due to the candidate elimination
module, the tracking speed and accuracy of OSTrack are increased. Since
OSTrack efficiently uses the information flow between the features of the
target template and search region, target-specific discriminative cues are
extracted, and unnecessary background features are eliminated. Therefore, it
has shown excellent tracking performance with high tracking speed in several
benchmark datasets.

Figure 16: Architecture of the OSTrack Approach [62]. It used an early
elimination module in encoder layers to identify and remove the features of
the background tokens from the search image and hence the performance of the
tracker is increased.

Due to the great success of OSTrack in the tracking community, several recent
follow-up approaches [63, 66, 67, 65] have been proposed, utilizing self-
supervised learning based MAE pre-trained model to initialize the backbone
network. Lan _et al._ [63] upgraded the performance of OSTrack by using a
candidate token elimination module and including a set of dynamic templates to
adopt the temporal features of the target. Their tracker, referred to as
ProContEXT, is shown in Fig. 17. The ProContEXT tracker focuses on capturing
the spatial and temporal cues of the target templates by utilizing a context-
aware self-attention module. In this tracker, static target templates, dynamic
target templates with spatial and temporal cues, and the search region are
split and then fed to the attention module. Additionally, the candidate token
elimination module of ProContEXT performs better than OSTrack as it
incorporates temporal cues in the removal of background tokens. Based on the
reported results, the ProContEXT approach outperformed OSTrack and
demonstrated state-of-the-art performance in tracking benchmarks.

Figure 17: Architecture of the ProContEXT Tracker [63]. Spatial and temporal
cues are captured using a context-aware self-attention module by feeding a set
of static and dynamic target templates. Also, an upgraded candidate token
elimination module is used in this tracker.

The success of the One-stream One-Stage trackers lies in the information flow
between the features of target template patches and search region patches.
Although the attention operation is used to identify target-specific features
in search regions, it is not necessary to calculate the attention between the
template patches and all of the search patches since patches containing
background and distractor features can degrade the efficiency and accuracy of
the tracker. Although the OSTrack [62] and ProContEXT [63] trackers
incorporate modules to eliminate background patches in some encoder layers,
there is a need for a more robust and generalized mechanism. Recently, Gao _et
al._ [67] have proposed a Generalized Relation Modeling (GRM) mechanism to
address this limitation in One-stream One-stage trackers. In the first stage
of GRM tracker, a simple MLP network is utilized to determine the
probabilities of target template features in search region patches.
Subsequently, the search region patches are divided into two groups, with only
the high probability search patches allowed to interact with the template
patches in the encoder layers. This adaptive selection of search region
patches enhances the flexibility of relation modeling and improves
discrimination capabilities. The experimental results demonstrate that the GRM
tracker outperforms OSTrack with an approximately 2% higher success rate and
improved tracking speed.

The state-of-the-art One-stream One-stage trackers [62, 61, 60] do not fully
utilize the temporal features, making them less robust for appearance changes.
Recently, Xie _et al._ [66] identified this limitation and proposed an end-to-
end Transformer tracker called VideoTrack, which aims to exploit the temporal
context. VideoTrack captures the spatio-temporal features of the tracking
sequence by feeding patches from a set of adjacent frames to the Transformer.
The VideoTrack approach encodes inter-frame and intra-frame patches using
distinct encoding schemes and then feeds them to a Transformer encoder to
capture spatio-temporal features. Due to the spatio-temporal feature learning,
VideoTrack outperforms OSTrack[62] and MixFormer [60] approaches with a
considerable margin.

Figure 18: Architecture of the DropMAE [65] masked auto encoder. It is used to
capture the spatial cues within an individual image and to capture the
correlated spatial cues between target and search images. The OSTrack [62]
approach showed better performance while fine-tuning the DropMAE as the
backbone compared to the MAE [109] backbone.

Most single-object tracking approaches traditionally rely on pre-trained
models initialized through supervised learning. However, recent fully
Transformer-based trackers, such as OSTrack [62], ProContEXT [63], GRM [67],
and VideoTrack [66], have demonstrated that initializing the backbone with a
self-supervised learning based Masked Autoencoder (MAE) [109] pre-trained
model can achieve higher tracking accuracy compared to models based on
supervised learning. This improvement can be attributed to the MAE’s ability
to capture fine-grained local structures within an image, which are essential
for accurate target localization. Recently, Wu _et al._ [65] discovered that
the MAE architecture exhibits a lack of robustness when applied to feature
matching tasks between two images. As a result, a modified variant of the MAE
architecture named DropMAE was introduced, as depicted in Figure 18, which is
specifically designed to improve feature matching for tracking and
segmentation tasks. DropMAE captures spatial cues within individual images and
also captures the correlated spatial cues between two frames by randomly
masking the input frames and processing them through the encoder-decoder
architecture. Additionally, DropMAE follows an attention dropout mechanism
that restricts the interaction between tokens within the same frame. This
restriction leads to increased interaction between the tokens of search and
target image pairs, resulting in more reliable capture of temporal cues. The
experimental results demonstrated that the OSTrack [62] approach achieved
superior performance when utilizing DropMAE to initialize the backbone
compared to initializing with MAE backbone.

Closely related to DropMAE, another masked encoder-based pre-trained model has
been specifically designed and trained for the tracking task, referred to as
MAT [103]. Similar to DropMAE, MAT randomly masks patches of template and
search image pairs, which are then jointly processed by the encoder to capture
their visual representations. After encoding, in contrast to the single
decoder of other models [109, 65], MAT employs two identical decoders to
separately reconstruct the search image and the target region in the search
image. The encoded features of the target template are used to reconstruct the
target region in the search image that enables the capturing of tracking-
specific representations in the search image. Finally, a simple and
lightweight tracker (MATTrack) is designed to evaluate the representation of
the MAT masked encoder.

All present CNN-based and fully-Transformer based approaches treat object
tracking as a template matching problem between the target image and the
search region. Although some approaches follow a template update mechanism,
the temporal dependencies between the frames are neglected or not fully
investigated. Recently, Wei _et al._ [68] proposed a tracker called ARTrack,
which treats object tracking as a coordinate sequence interpretation problem.
Taking inspiration from language modeling, ARTrack interprets target
trajectories by learning from previous states in a sequence of frames. The
tracking pipeline of ARTrack differs from other One-stream One-stage
Transformer trackers as it employs an encoder-decoder architecture and lacks a
prediction head. By treating tracking as a coordinate sequence interpretation
task and adopting a general encoder-decoder architecture, ARTrack simplifies
the tracking pipeline and achieves superior performance compared to existing
trackers that rely on prediction heads.

Figure 19: Architecture of the SeqTeack [69] approach. It treats the tracking
as a sequence learning problem instead of template matching. Without a
prediction head, it is able to locate a target using an end-to-end Transformer
architecture.

Closely related to the ARTrack[68], the recently proposed SeqTrack [69]
tracker also considers tracking as a sequence learning problem. This tracker
utilizes an encoder-decoder Transformer architecture to extract visual
features and autoregressively generate the bounding box coordinates of the
target, respectively. Different from ARTrack, in SeqTrack, the predicted
bounding box values rely solely on the target stage in the previous frame,
achieved through the use of an attention masking technique in the decoder.
Experimental results shows that SeqTrack better performance than the template
matching based trackers.

In summary, One-stream One-stage trackers combined the feature extraction and
feature fusion process using a fully-Transformer architecture. In these
trackers, target template and search region images are split as tokens and
concatenated with their positional embedding, and then fed to a Transformer.
Since these trackers extract features with a single Transformer network, the
features of the template and search regions are efficiently integrated,
leading to more discriminative features being identified and unnecessary
features being eliminated. Based on these facts, fully-Transformer based One-
stream One-stage trackers have shown outstanding performance compared to other
types of trackers on all benchmark datasets.

##  5 Experimental Analysis

In the last two decades, numerous approaches have been proposed for single
object tracking. Since their performance is evaluated on various benchmark
datasets using different evaluation metrics, it is crucial to experimentally
evaluate these approaches to identify future directions, particularly after
the introduction of Transformers in VOT. This study focuses on evaluating the
tracking robustness and computational efficiency performances of CNN-
Transformer based trackers and fully-Transformer based trackers. Additionally,
we have included 12 recently proposed CNN-based trackers in this experimental
evaluation study. Five benchmark datasets are utilized, and their details and
evaluation metrics are described in Section 5.1. The results of tracking
robustness and efficiency are presented in Section 5.2 and Section 5.3,
respectively.

###  5.1 Benchmark Datasets and Evaluation Metrics

Several benchmark datasets are constructed and publicly available for VOT.
Each dataset is different from the others based on the target object class,
number of sequences, annotation method, length of the tracking sequence,
attributes, and complexity. Additionally, these datasets follow various
performance metrics to measure the trackers’ performance. To analyze the
performance of recent trackers, we have selected five benchmark datasets:
OTB100, UAV123, LaSOT, TrackingNet, and GOT-10k. The details of these datasets
are summarized in Table 3.

Table 3: Details of the datasets used in experimental analysis study. The abbreviations are denoted as SV for scale variation, ARC for aspect ratio change, FM for fast motion, LR for low resolution, OV for out-of-view, IV for illumination variation, CM for camera motion, MB for motion blur, BC for background clutter, SOB for Similar Object, DEF for deformation, ROT for rotation, IPR for in-plane rotation, OPR for out-of-plane rotation, OCC for occlusion, FOC for full occlusion, POC for partial occlusion, and VC for viewpoint change. Dataset | Year | No of | No of | Average | Absent |  Attributes |  Evaluation
---|---|---|---|---|---|---|---
|  | Sequences | Classes | Frames | Label |  |  Metrics
OTB100 [77] | 2015 | 100 | 16 | 598 | No |  SV, FM, LR, OV, IV, MB, |  Precision, Success
|  |  |  |  |  |  BC, DEF, IPR, OPR, OCC |
UAV123 [85] | 2016 | 123 | 9 | 915 | No |  SV, FM, LR, OV, IV, CM, |  Precision, Success
|  |  |  |  |  |  BC, SOB, OCC, POC, FOC |
TrackingNet [84] | 2018 | 30643 | 27 | 471 | No |  SV, ARC, FM, LR, OV, IV, |  Precision, Success,
|  |  |  |  |  |  CM, MB, BC, SOB, DEF, IPR, |  Normalized Precision
|  |  |  |  |  |  OPR, POC, FOC |
LaSOT [82] | 2019 | 1400 | 70 | 2506 | Yes |  SV, ARC, FM, LR, OV, IV, |  Precision, Success,
|  |  |  |  |  |  CM, MB, BC, DEF, ROT, FOC, |  Normalized Precision
|  |  |  |  |  |  POC, VC |
GOT-10k [83] | 2019 | 10000 | 563 | 149 | Yes |  SV, ARC, FM, LR, IV, OCC |  Average Overlap,
|  |  |  |  |  |  |  Success Rate

OTB was the first widely used benchmark dataset in VOT. It gained widespread
acceptance among researchers due to its simplicity and better evaluation
scheme. It is initially proposed with 51 short tracking video sequences and
they are referred to as OTB2013 [110]. Few years later, another 49 sequences
are included into the benchmark and then the entire sequences are referred to
as OTB100 [77]. We have selected OTB100 to evaluate the performance of the
trackers with various tracking attributes (scenarios). Similar to other
researchers, precision and success plots are used to measure the performance
of a tracker in OTB100. Precision in OTB100 is computed by calculating the
distance between the midpoints of the tracked target bounding box and the
corresponding ground truth bounding box. The precision plot is used to show
the proportion of frames in which the tracking locations are within a
threshold distance. In this study, threshold of 20 pixel distance is used to
rank the trackers based on their precision scores. Success of a tracker is
measured in OTB100 by measuring the average overlap scores and it is computed
as follows:

$\mathrm{Overlap\\_Score}=\frac{\mid B_{t}\cap B_{g}\mid}{\mid B_{t}\cup
B_{g}\mid},$ (7)

where $B_{t}$ and $B_{g}$ are the bounding boxes of tracked target and ground-
truth, respectively. Also, $\cap$ and $\cup$ are denoting intersection and
union operations, respectively. The area-under-curve (AUC) scores are used to
rank the trackers in success plots, representing the average number of
successful frames when the overlapping scores change from 0 to 1.

UAV123 [85] dataset contains a total of 123 video sequences from an aerial
viewpoint. All the video sequences of UAV123 are captured by using unmanned
aerial vehicles (UAV). Compare to OTB100 and other benchmark datasets, target
object is small in UAV123 and the tracking sequences have several distractor
objects and long occlusions. Since both the OTB100 and UAV123 datasets have a
small number of sequences with fewer frames, they are not sufficient to train
deep learning-based tracking architectures. We have used the UAV123 dataset to
evaluate the aerial tracking performances of trackers and to measure their
occlusion handling capability. UAV123 dataset follows the same evaluation
scheme of OTB100 to compare the performance of trackers.

TrackingNet [84] is the first large scale benchmark dataset and contains more
than 30K video sequences. Tracking video sequences of TrackingNet are sampled
from real-world YouTube videos and 30,132 and 511 numbers of sequences are
allocated for training and testing, respectively. The TrackingNet dataset
contains a rich distribution of target object classes, with almost 14 million
bounding boxes. The average length of tracking sequences is less than 500
frames, and hence the TrackingNet dataset is not suitable to evaluate a
tracker’s long-term tracking capability. However, we have used the TrackingNet
dataset to evaluate the performance of trackers in real-world scenarios across
a wide range of classes. Similar to the OTB100, TrackingNet dataset also used
the precision and success plots to evaluate the performance of trackers. In
addition to these metrics, normalized precision is used since the precision is
depending on the sizes of the images and bounding boxes. The normalized
precision ($P_{n}$) metric is computed as:

$P_{n}=\lVert W(B_{t}-B_{g})\rVert_{2}$ (8)

where $W$ denotes the size of the bounding box.

Recently, the large-scale single object tracking (LaSOT) benchmark [82] has
been developed, and it is one of the largest tracking datasets to date. On
average, more than 2500 frames are in a tracking sequence in LaSOT, and the
total duration of video sequences are more than 32 hours. In addition, target
objects are annotated with detail information such as absent label, in which
out-of-view and fully occluded target are labeled. Furthermore, the LaSOT
dataset maintained the class balance in video sequences by containing equal
number of video sequences for each target class. Similar to the TrackingNet
dataset, LaSOT dataset used the precision, success, and normalized precision
metrics to measure the performance of the trackers. In this experimental
analysis study, LaSOT benchmark is mainly used to evaluate the long-term
tracking performances of the trackers.

GOT-10k [83] is another large scale tracking benchmark and it has 10,000 video
sequences which were captured from real-world scenarios. Compare to other
benchmark datasets, GOT-10k has a huge number of target object classes and
almost 1.5 million bounding boxes are used in annotation. Furthermore, the
annotations include motion and absent labels, allowing for the efficient
evaluation of trackers that can handle motion and occlusions. The average
overlap (AO) and success rate are used to evaluate the trackers in GOT-10k and
they are calculated by measuring the average overlaps between $B_{t}$ and
$B_{g}$, and the percentage of correctly tracked frames, respectively. We have
evaluated the trackers on GOT-10k to measure the one-shot tracking
performances of the trackers since the training object classes are not
overlapped with testing object classes.

###  5.2 Tracking Performance Analysis

We have presented the qualitative results of the trackers in this section. To
conduct an unbiased evaluation and compare their tracking performances, we
reproduced the success and precision scores of the trackers and their
attribute-wise tracking results using their source codes. Additionally, we
considered the reported results of a few recent trackers since their source
codes are unavailable.

Table 4: Tracking robustness comparison of the recently proposed state-of-the-art approaches on OTB100, UAV123, TrackingNet, LaSOT, and GOT-10k benchmarks. The abbreviations are denoted as $\mathrm{SR}$: Success Rate, $\mathrm{P}$:Precision, $\mathrm{P_{n}}$: Normalized Precision, $\mathrm{AO}$: Average Overlap, $\mathrm{SR_{0.5}}$: Success Rate at $0.5$ threshold, and $\mathrm{SR_{0.75}}$: Success Rate at $0.75$ threshold. All the results are reported in percentage. The evaluation considered the top-performing models of each tracker. The red, blue, and green colors are used to mark the top three results, respectively. The symbols †and ‡are denoting One-stream One-stage trackers, and Two-stream Two-stage trackers, respectively. The $\ast$ symbol is used to indicate tracking models that are only trained on the training set of GOT-10k.  | Trackers | OTB100 | UAV123 | LaSOT | TrackingNet | GOT-10k $\ast$
---|---|---|---|---|---|---
| SR | P | SR | P | SR | $\mathrm{P_{n}}$ | P | SR | $\mathrm{P_{n}}$ | P | AO | $\mathrm{SR_{0.5}}$ | $\mathrm{SR_{0.75}}$
Fully-Transformer based Trackers |  SeqTrack [69]† | 68.3 | 89.1 | 68.5 | 89.1 | 72.5 | 81.5 | 79.2 | 85.5 | 89.8 | 85.8 | 74.8 | 81.9 | 72.2
DropTrack [65]† | 69.5 | 91.0 | 70.9 | 92.4 | 71.5 | 81.5 | 77.9 | 84.1 | 88.9 | - | 75.9 | 86.8 | 72.0
GRM [67]† | 68.9 | 90.0 | 69.0 | 89.8 | 69.9 | 79.3 | 75.8 | 84.0 | 88.7 | 83.3 | 73.4 | 82.9 | 70.4
VideoTrack [66]† | - | - | 69.7 | 89.9 | 70.2 | - | 76.4 | 83.8 | 88.7 | 83.1 | 72.9 | 81.9 | 69.8
MATTrack [103]† | - | - | 68.0 | - | 67.8 | 77.3 | - | 81.9 | 86.8 | - | 67.7 | 78.4 | -
ProContEXT [63] † | 66.9 | 91.1 | 69.2 | 90.0 | 71.4 | 81.2 | 77.9 | 84.6 | 89.2 | 83.8 | 74.6 | 84.7 | 72.9
OSTrack [62]† | 68.1 | 88.7 | 70.7 | 92.3 | 71.1 | 81.1 | 77.6 | 83.9 | 88.5 | 83.2 | 73.7 | 83.2 | 70.8
MixFormer [60]† | 70.4 | 92.2 | 69.5 | 90.9 | 70.0 | 79.9 | 76.3 | 83.9 | 88.9 | 83.1 | 70.7 | 80.0 | 67.8
SimTrack-B/16 [61]† | 66.1 | 85.7 | 69.8 | 89.6 | 69.3 | 78.5 | 74.0 | 82.3 | 86.5 | 80.2 | 68.6 | 78.9 | 62.4
SparseTT [59]‡ | 69.8 | 90.5 | 68.8 | 88.0 | 66.0 | 74.8 | 70.1 | 81.7 | 86.6 | 79.5 | 69.3 | 79.1 | 63.8
SwinTrack[58]‡ | 69.1 | 90.2 | 69.8 | 89.1 | 70.2 | 78.4 | 75.3 | 84.0 | 88.2 | 83.2 | 69.4 | 78.0 | 64.3
|  DualTFR [57]‡ | - | - | 68.2 | - | 63.5 | 72.0 | 66.5 | 80.1 | 84.9 | - | - | - | -
CNN-Transformer based Trackers |  BANDT [64] | 70.6 | 91.1 | 69.6 | 90.0 | 64.4 | 73.0 | 67.0 | 78.5 | 82.7 | 74.5 | 64.5 | 73.8 | 54.2
AiATrack [56] | 69.6 | 91.7 | 69.3 | 90.7 | 69.0 | 79.4 | 73.8 | 82.7 | 87.8 | 80.4 | 69.6 | 80.0 | 63.2
ToMP [51] | 70.0 | 90.6 | 65.9 | 85.2 | 68.5 | 79.2 | 73.5 | 81.5 | 86.4 | 78.9 | - | - | -
CSWinTT [55] | 67.1 | 87.2 | 70.5 | 90.3 | 66.2 | 75.2 | 70.9 | 81.9 | 86.7 | 79.5 | 69.4 | 78.9 | 65.4
SiamTPN [52] | 70.2 | 90.2 | 63.6 | 82.3 | 58.1 | 68.3 | 57.8 | 70.8 | 77.1 | 65.1 | 57.6 | 68.6 | 44.1
UTT [53] | - | - | - | - | 64.6 | - | 67.2 | 79.7 | - | 77.0 | - | - | -
CTT [54] | - | - | - | - | 65.7 | 75.0 | 69.8 | 81.4 | 86.4 | - | - | - | -
DTT [49] | - | - | - | - | 60.1 | - | - | 79.6 | 85.0 | 78.9 | 68.9 | 79.8 | 62.2
TrTr [48] | 71.2 | 93.1 | 63.3 | 83.9 | 52.7 | 61.9 | 54.4 | 70.7 | 79.5 | 67.4 | - | - | -
STARK [47] | 68.0 | 88.4 | 68.5 | 89.5 | 67.1 | 76.9 | 72.2 | 82.0 | 86.9 | 79.1 | 68.8 | 78.1 | 64.1
TransT [46] | 69.5 | 89.9 | 68.1 | 87.6 | 64.9 | 73.8 | 69.0 | 81.4 | 86.7 | 80.3 | 67.1 | 76.8 | 60.9
TrDiMP [45] | 70.8 | 92.5 | 67.0 | 87.6 | 63.9 | 73.0 | 66.2 | 78.4 | 83.3 | 73.1 | 67.1 | 77.7 | 58.3
HiFT [50] | 61.4 | 81.4 | 58.9 | 78.7 | 45.1 | 52.7 | 42.1 | 66.7 | 73.8 | 60.9 | - | - | -
CNN-based Trackers |  KeepTrack [102] | 70.9 | 92.2 | 69.1 | 90.9 | 67.3 | 77.4 | 70.4 | 78.1 | 83.5 | 73.8 | 68.3 | 79.3 | 61.0
SiamGAT [33] | 71.0 | 91.6 | 64.6 | 84.3 | 53.9 | 63.3 | 53.0 | 75.3 | 80.7 | 69.3 | 62.7 | 74.3 | 48.8
SiamRN [34] | 70.1 | 93.1 | 64.3 | 86.1 | 52.7 | 63.5 | 53.1 | - | - | - | - | - | -
SiamR-CNN [32] | 70.1 | 89.1 | 64.9 | 83.4 | 64.8 | 72.2 | 68.4 | 81.2 | 85.4 | 80.0 | 64.9 | 72.8 | 59.7
PrDiMP-50 [101] | 69.7 | 89.7 | 66.9 | 87.8 | 59.9 | 69.0 | 60.8 | 75.8 | 81.6 | 70.4 | 63.4 | 73.8 | 54.3
Ocean [100] | 68.4 | 92.0 | 62.1 | 82.3 | 56.0 | 65.1 | 56.6 | 69.2 | 79.4 | 68.7 | 61.1 | 72.1 | 47.3
SiamFC++ [99] | 68.3 | 89.6 | 62.3 | 81 | 54.3 | 62.3 | 54.7 | 75.4 | 80.0 | 70.5 | 59.5 | 69.5 | 47.3
SiamBAN [31] | 69.6 | 91.0 | 63.1 | 83.3 | 51.4 | 59.8 | 52.1 | - | - | - | - | - | -
SiamAttn [98] | 71.2 | 92.6 | 65.0 | 84.5 | 56.0 | 64.8 | - | 75.2 | 81.7 | - | - | - | -
DiMP-50 [97] | 68.0 | 88.8 | 64.8 | 85.8 | 56.8 | 64.8 | 56.4 | 74.0 | 80.1 | 68.7 | 61.1 | 71.7 | 49.2
SiamRPN++ [29] | 69.6 | 91.5 | 64.2 | 84.0 | 49.6 | 56.9 | 49.1 | 73.3 | 80.0 | 69.4 | - | - | -
SiamDW [96] | 67.0 | 89.2 | 53.6 | 77.6 | 38.5 | 48.0 | 38.9 | 61.1 | 71.3 | 56.3 | 42.9 | 48.3 | 14.7

We have selected 37 single-object trackers for this experimental evaluation
study. They were published in the last four years in reputed conferences and
indexed journals. Although we have included the ARTrack[68] in the literature
review, it is not included in the experimental evaluation because the raw
results or tracking model of this tracker are still not publicly available. In
this evaluation, these selected trackers are categorized as CNN-based
trackers, CNN-Transformer based trackers, and fully-Transformer based trackers
and then their category-wise performances are discussed. The overall
performances of these trackers are summarized in the Table 4 and their
attribute-wise results are detailed in the Table 5. Tracking performances of
these approaches for each benchmark datasets are discussed in the following
subsections.

Table 5: Tracking attribute-based comparison on OTB100, UAV123, and LaSOT datasets. In each tracking attribute, percentage of success rate scores (SR) of best three trackers and their types are listed. The symbols †, ‡, and $\Diamond$ are denoting One-stream One-stage trackers, Two-stream Two-stage trackers, and CNN-Transformer based trackers, respectively.  Tracking Attribute | OTB100 | UAV123 | LaSOT
---|---|---|---
SR | Tracker | SR | Tracker | SR | Tracker
ARC | - | - | 70.6 |  CSWinTT [55] $\Diamond$ | 70.9 |  SeqTrack [69] †
- | - | 70.2 |  SimTrack-B/16 [61] † | 70.1 |  DropTrack [65]†
- | - | 70.1 |  OSTrack [62] † | 69.8 |  ProContEXT [63]†
BC | 71.5 |  TrTr [48]$\Diamond$ | 57.6 |  OSTrack [62]† | 65.9 |  SeqTrack [69]†
70.4 |  SiamAttn [98] | 56.1 |  DropTrack [65]† | 64.4 |  ProContEXT [63]†
70.3 |  TrDiMP [45] $\Diamond$ | 55.4 |  AiATrack [56] $\Diamond$ | 63.7 |  DropTrack [65]†
CM | - | - | 72.7 |  CSWinTT [55] $\Diamond$ | 75.8 |  ProContEXT [63]†
- | - | 72.4 |  MixFormer [60] † | 74.6 |  SeqTrack [69]†
- | - | 72.4 |  DropTrack [65]† | 74.0 |  AiATrack [56] $\Diamond$
DEF | 69.1 |  SiamGAT [33] | - | - | 74.9 |  SeqTrack [69]†
69.0 |  TrTr [48] $\Diamond$ | - | - | 72.9 |  ProContEXT [63] †
68.6 |  SiamTPN [52] $\Diamond$ | - | - | 72.3 |  GRM [67] †
FM | 72.9 |  SparseTT [59] ‡ | 68.3 |  AiATrack [56] $\Diamond$ | 60.1 |  DropTrack [65]†
72.6 |  SwinTrack [58] ‡ | 67.9 |  CSWinTT [55] $\Diamond$ | 59.1 |  OSTrack [62]†
72.5 |  MixFormer [60] † | 67.6 |  DropTrack [65]† | 58.6 |  ProContEXT [63]†
FOC | - | - | 56.5 |  SimTrack-B/16 [61]† | 64.5 |  DropTrack [65] †
- | - | 56.1 |  OSTrack [62]† | 63.1 |  ProContEXT [63] †
- | - | 54.4 |  DropTrack [65] † | 63.0 |  SeqTrack [69]†
IV | 74.2 |  TrTr [48] $\Diamond$ | 67.0 |  AiATrack [56] $\Diamond$ | 71.7 |  ProContEXT [63] †
73.8 |  SiamAttn [98] | 65.9 |  CSWinTT [55] $\Diamond$ | 71.4 |  OSTrack [62]†
72.8 |  KeepTrack [102] | 65.5 |  OSTrack [62] † | 71.3 |  DropTrack [65]†
IPR | 72.1 |  TrDiMP [45]$\Diamond$ | - | - | - | -
72.0 |  SiamAttn [98] | - | - | - | -
71.7 |  SiamBAN [31] | - | - | - | -
LR | 74.5 |  SiamAttn [98] | 58.6 |  DropTrack [65] † | 65.6 |  DropTrack [65] †
74.2 |  SparseTT [59] ‡ | 58.4 |  SwinTrack [58] ‡ | 65.2 |  SeqTrack [69]†
73.3 |  SiamGAT [33] | 57.4 |  OSTrack [62]† | 65.0 |  OSTrack [62] †
MB | 75.5 |  MixFormer [60]† | - | - | 71.0 |  ProContEXT [63] †
75.1 |  SparseTT [59] ‡ | - | - | 70.5 |  SeqTrack [69]†
74.5 |  AiATrack [56] $\Diamond$ | - | - | 69.7 |  DropTrack [65] †
OCC/POC | 68.5 |  TrTr [48] $\Diamond$ | 66.0 |  OSTrack [62]† | 69.7 |  SeqTrack [69]†
68.3 |  SiamAttn [98] | 65.8 |  CSWinTT [55] $\Diamond$ | 69.5 |  DropTrack [65] †
68.2 |  KeepTrack [102] | 65.8 |  DropTrack [65] † | 68.9 |  ProContEXT [63] †
OPR | 70.9 |  SiamAttn [98] | - | - | - | -
70.7 |  SiamGAT [33] | - | - | - | -
70.4 |  TrTr [48] $\Diamond$ | - | - | - | -
OV | 70.3 |  MixFormer [60]† | 68.7 |  ProContEXT [63] † | 67.4 |  DropTrack [65] †
69.8 |  ProContEXT [63] † | 68.5 |  DropTrack [65] † | 67.2 |  SeqTrack [69]†
69.6 |  SeqTrack [69]† | 68.0 |  SimTrack-B/16 [61] † | 66.4 |  OSTrack [62]†
ROT | - | - | - | - | 72.4 |  SeqTrack [69] †
- | - | - | - | 71.1 |  ProContEXT [63] †
- | - | - | - | 70.8 |  DropTrack [65] †
SV | 73.4 |  ProContEXT [63]† | 70.0 |  DropTrack [65] † | 72.4 |  SeqTrack [69]†
72.8 |  KeepTrack [102] | 69.7 |  OSTrack [62]† | 71.3 |  DropTrack [65]†
72.8 |  TransT [46] $\Diamond$ | 69.4 |  CSWinTT [55] $\Diamond$ | 71.2 |  ProContEXT [63] †
SOB | - | - | 69.7 |  SimTrack-B/16 [61] † | - | -
- | - | 69.1 |  OSTrack [62] † | - | -
- | - | 68.6 |  CSWinTT [55] $\Diamond$ | - | -
VC | - | - | 74.0 |  SimTrack-B/16 [61] † | 74.9 |  SeqTrack [69] †
- | - | 73.9 |  DropTrack [65] † | 73.4 |  ProContEXT [63] †
- | - | 73.4 |  OSTrack [62] † | 72.6 |  DropTrack [65] †

####  5.2.1 Analysis on OTB100 Dataset

We used the official toolkit of OTB100 to evaluate the tracking performance of
the trackers. Success plots were ranked based on the area-under-curve (AUC)
scores, while the precision (P) scores at a threshold of 20 pixels were used
to evaluate the trackers. In addition to the overall evaluation, the
performances of the trackers were measured based on eleven tracking attributes
using the toolkit and their success plots, as shown in Fig. 20.

Based on the overall success rates and precision scores in Table 4, both CNN-
Transformer based trackers and CNN-based trackers have achieved overall high
performance. TrTr [48] tracker showed excellent performance on OTB100 in terms
of accuracy and precision by replacing the cross-correlation mechanism of
Siamese tracking with a Transformer. Similarly, TrDiMP [45] showed competitive
success and precision scores by exploiting temporal cues with a CNN-
Transformer based architecture. On the other side, CNN-based Siamese trackers:
SiamAttn [98] and SiamRN [34] showed high success and precision scores,
respectively.

Compared to other benchmark datasets, the tracking performances of fully-
Transformer based trackers are slightly lower than the other two types on
OTB100. CNN-Transformer based trackers and CNN-based trackers showed better
performances in OTB100 by capturing and matching convolutional features that
represent local-regional cues. Since most of the OTB videos have fewer frames,
the appearance of the target remains the same in many sequences. Therefore,
CNN-based feature extraction and matching have shown excellent tracking
results. On the other hand, the performances of fully-Transformer based
approaches mainly rely on their attention mechanism and global feature
capturing capabilities, and their performances are slightly limited on OTB100
since most tracking sequences have low-resolution videos with fewer frames.

The attribute-based evaluation on OTB100 showed that the dataset is no longer
challenging for recent trackers, based on the results in Table 5 and the plots
in Fig. 20. The fully-Transformer based trackers: MixFormer [60], SparseTT
[59], and ProContEXT [63] showed better performance than the CNN-based
trackers in many challenging attributes. In particular, almost all of the
fully Transformer-based trackers successfully handled the fast motion (FM) and
out-of-view (OV) scenarios and outperformed the CNN-based trackers by a large
margin, due to their long-range feature capturing capability. On the other
side, fully-Transformer trackers showed poor performances in background
clutters (BC), and deformation (DEF) frames because of their poor
discriminative capabilities in short-range videos. In overall, CNN-Transformer
based tracker TrTr [48] successfully handles all tracking challenges in OTB100
by combining the CNN features with a Transformer architecture.

Figure 20: The success plots of trackers for 11 challenging attributes on
OTB100.

####  5.2.2 Analysis on UAV123 Dataset

Figure 21: The success plots of the trackers for 12 challenging attributes on
UAV123 dataset.

Tracking a target in the UAV123 dataset is more difficult than in other
benchmark datasets, since the target objects are relatively small in aerial
tracking sequences. Therefore, the trackers can capture limited visual cues
and are unable to rely on powerful appearance models. In addition, tracking a
target in UAV123 is more challenging since the target object and camera
frequently change position and orientation. We have used the official toolkit
of UAV123 to measure the precision and success scores of the trackers and to
conduct the tracking attribute-wise evaluation.

The One-stream One-stage trackers showed outstanding performances in UAV123
based on the overall precision and success rate scores due to their combined
feature learning and future fusion process. Especially, the DropTrack [65]
tracker showed excellent performance in UAV123 with a success rate score of
70.9% and a precision score of 92.4% due to its exceptional ability to capture
correlated spatial cues. The OSTrack [62] and MixFormer [60] approaches are
also showed good results in UAV123 dataset. Among the CNN-Transformer based
trackers, CSWinTT [55] and AiATrack [56] approaches showed better success and
precision scores, respectively. CNN-based tracker: KeepTrack [102] obtained
third highest precision score because of its distractor object handling
capability without relying heavily on an appearance model.

Fully-Transformer based and CNN-Transformer based trackers showed excellent
performance compared to the CNN-based trackers in every attribute on the
UAV123 benchmark, as shown in Fig. 21 and Table 5. The OSTrack [62] and
DropTrack [65] trackers successfully handle the background clutter (BC),
partial occlusion (POC), out-of-view (OV), and scale variation (SV) scenarios
because of their strong discriminative capability and the background feature
eliminating technique. On the other hand, CNN-Transformer based trackers:
CSWinTT [55] and AiATrack [56] significantly outperforms fully-Transformer
based trackers in fast motion (FM), illumination variation (IV), and aspect
ratio change (ARC) scenarios.

Based on the experimental analysis, full occlusion (FOC), background clutter
(BC), and low resolution (LR) are the most challenging attributes in UAV123
dataset since all the trackers are struggling to capture the strong appearance
cues in these scenarios in aerial tracking videos. To summarize, the
evaluation of state-of-the-art trackers on the UAV123 dataset showed that
their performance was only average. This finding highlights the need for
further research and innovation to improve the accuracy and reliability of
aerial tracking systems.

####  5.2.3 Analysis on LaSOT Dataset

Figure 22: The success plots comparison of trackers on LaSOT for 14
challenging attributes.

Experimental analysis on LaSOT dataset is important to identify the future
direction of VOT since it has long-term tracking sequences with several
challenging scenarios. We have utilized the official toolkit of the LaSOT
dataset to measure the tracking performance of the approaches.

As summarized in the overall performance results in Table 4, One-stream One-
stage tracker SeqTrack [69] showed outstanding performance in LaSOT in terms
of success rate, normalized precision, and precision scores since it trained
the Transformer by treating tracking as a sequence learning problem rather
than template matching. The DropTrack [65] approach also demonstrated
excellent performance in LaSOT due to the correlated spatial feature learning
capability of its backbone network. In addition to these trackers, the
ProContEXT [63] and OSTrack [62] approaches showed good performances since
they efficiently used the information flow between the target template and
search region and hence eliminated the background features in feature
matching. Overall, all One-stream One-stage fully-Transformer based trackers
significantly outperformed other tracker types with a large margin in the
LaSOT dataset, as the target-aware Transformer features are better suited for
long-term tracking than CNN-based local feature extraction.

Based on the attribute-wise success plots in Fig. 22, and Table 5, SeqTrack
[69] and DropTrack [65] trackers showed best performance in most of the
tracking scenarios in the LaSOT benchmark. In particular, the SeqTrack tracker
outperforms the competitors with a large scale in background clutter (BC),
deformation (DFE), scale variation (SV), and view point change (VC) scenes
because of its sequence learning mechanism. In addition to these trackers, the
ProContEXT [63] tracker showed better performance in camera motion (CM),
illumination variation (IV), and motion blur (MB) scenarios because of its
strong discriminative capability and background feature removal mechanism.

Overall, One-stream One-stage fully-Transformer trackers showed excellent
performance, while CNN-Transformer based trackers showed considerable
performance in the LaSOT benchmark. On the other hand, the overall tracking
and attribute-wise performances of CNN-based approaches are very limited in
the LaSOT dataset, as they fail to include temporal cues and extract target-
specific features in search regions. Based on the attribute-wise success
rates, fast motion and full occlusion are the most challenging attributes for
state-of-the-art trackers.

####  5.2.4 Analysis on TrackingNet Dataset

TrackingNet dataset has more than 30k training and 511 testing video sequences
with 14 million and 225k annotations, respectively. Since the TrackingNet
dataset has the videos with a large diversity of resolutions, target object
classes, and frame rates, evaluating the tracking performance on this dataset
is important for many real-world applications. Similar to LaSOT dataset, we
ranked the trackers in TrackingNet dataset using success rate, precision
score, and normalized precision. Since the test set annotations are not
publicly available, we are unable to conduct the attribute-wise comparison in
this dataset.

Similar to the LaSOT benchmark, the SeqTrack [69] tracker showed superior
performance in TrackingNet dataset with 85.5% of success rate, 89.8% of
normalized precision, and 85.8% of precision scores. Also, other One-stream
One-stage based fully-Transformer trackers: ProContEXT [63] and DropTrack [65]
approaches also showed competitive tracking performances. Moreover, the GRM
[67] and MixFormer [60] achieved the third rank in terms of precision and
normalized precision, respectively.

Experimental analysis on TrackingNet dataset showed that fully-Transformer
based trackers significantly outperformed other two categories by a large
margin. Although CNN-Transformer based trackers: AiATrack [56] and ToMP [51]
obtained considerable competitive performances, tracking accuracy of other
CNN-Transformer based trackers are lower. On the other hand, similar to the
LaSOT dataset, all CNN-based trackers are struggling to show good performance
in TrackingNet dataset. The experimental results clearly show that fully-
Transformer based feature extraction and fusion successfully handle real-world
tracking scenarios, such as various resolutions and frame rates, better than
CNN-based feature extraction and target localization.

####  5.2.5 Analysis on GOT-10k dataset

Evaluating performance on the GOT-10k dataset is essential for VOT approaches,
as it has a large diversity of target object classes. This allows for
measuring the one-shot tracking performance of the trackers, i.e., the
performance when the target object class is not included in the training.
Furthermore, the evaluation results of this dataset are not biased towards
familiar objects, as the tracking models are only trained on the training set
of GOT-10k. We strictly adhere to the evaluation protocols to measure the
generalization capability of trackers. Average Overlap and success rate at the
threshold of 0.5 ($\mathrm{SR_{0.5}}$) and 0.75 ($\mathrm{SR_{0.75}}$) are
used to rank the trackers in GOT-10k dataset.

Based on the obtained results in Table 4, the DropTrack [65] tracker achieved
the highest average overlap score of 75.9% and $\mathrm{SR_{0.5}}$ (success
rate at 0.5 threshold) of 86.6%. The ProContEXT [63] and SeqTrack [69]
trackers also demonstrated competitive performances. Overall, the experimental
results showed that all One-stream One-stage fully-Transformer based trackers
possess a good generalization capability, as their attention mechanism perform
effectively across a diverse range of target object classes. On the other
hand, the tracking performances of CNN-based trackers are lower in the GOT-10k
dataset, as CNNs are not as effective in handling unfamiliar objects.

###  5.3 Tracking Efficiency Analysis

Analyzing tracking efficiency is as important as analyzing tracking
performance, as it is crucial for many practical situations and real-world
applications. In this efficiency analysis comparison, we have included all the
recent trackers except a few approaches since their source codes and tracking
models are not publicly available. To conduct the unbiased comparison,
efficiency results of each tracker is obtained by executing their source codes
on a computer with a NVIDIA Quadro P4000 GPU and a 64GB of RAM. We did not
change the parameters of the tracking models of these approaches and obtained
the efficiency results by evaluating the trackers on the LaSOT benchmark
dataset.

Table 6: Tracking efficiency comparison of the recently proposed state-of-the-art trackers on LaSOT dataset. The search region size, number of parameters (in million), tracking speed (in frame per second), number of floating point operations (FLOPs) of each tracker are listed. The efficiency experiments are conducted on a NVIDIA Quadro P4000 GPU with a 64GB of RAM. The symbols †and ‡are denoting One-stream One-stage trackers, and Two-stream Two-stage trackers, respectively. | Trackers | Tracking | No. of | Search | FLOPs (G) | Success
---|---|---|---|---|---|---
| Speed (FPS) | Parameters (M) | Reign Size |  | Rate
Fully-Transformer based Trackers |  SeqTrackL-384 [69]† | $5.81$ | $308.98$ |  384 $\times$ 384 | $535.85$ | $72.5$
DropTrack [65]† | $20.64$ | $92.83$ |  384 $\times$ 384 | $48.32$ | $71.5$
ProContEXT [63] † | $10.44$ | $92.83$ |  384 $\times$ 384 | $85.27$ | $71.4$
OSTrack-384 [62]† | $20.64$ | $92.83$ |  384 $\times$ 384 | $48.32$ | $71.1$
SwinTrack-B-384 [58]‡ | $11.92$ | $90.96$ |  384 $\times$ 384 | $61.85$ | $70.2$
MixFormer-L [60]† | $8.02$ | $195.40$ |  320 $\times$ 320 | $113.02$ | $70.0$
GRM-B [67]† | $36.02$ | $99.83$ |  256 $\times$ 256 | $30.90$ | $69.9$
SimTrack-B/16 [61]† | $36.34$ | $88.64$ |  224 $\times$ 224 | $22.26$ | $69.3$
SparseTT [59]‡ | $30.50$ | $46.33$ |  289 $\times$ 289 | $9.21$ | $66.0$
CNN-Transformer based Trackers |  AiATrack [56] | $31.22$ | $17.95$ |  320 $\times$ 320 | $9.45$ | $69.0$
ToMP-101 [51] | $13.16$ | $66.86$ |  288 $\times$ 288 | $22.12$ | $68.5$
STARK-ST101 [47] | $17.90$ | $47.17$ |  320 $\times$ 320 | $18.48$ | $67.1$
CSWinTT [55] | $8.76$ | $28.23$ |  384 $\times$ 384 | $16.45$ | $66.2$
TransT [46] | $21.15$ | $23.02$ |  256 $\times$ 256 | $16.71$ | $64.9$
TrDiMP [45] | $15.75$ | $43.75$ |  352 $\times$ 352 | $16.59$ | $63.9$
SiamTPN(Shuffle) [52] | $31.86$ | $6.17$ |  256 $\times$ 256 | $2.09$ | $58.1$
TrTr-online [48] | $16.28$ | $11.93$ |  255 $\times$ 255 | $21.23$ | $52.7$
HiFT [50] | $37.06$ | $11.07$ |  287 $\times$ 287 | $6.47$ | $45.1$
CNN-based Trackers |  KeepTrack [102] | $8.11$ | $43.10$ |  352 $\times$ 352 | $28.65$ | $67.3$
PrDiMP-50 [101] | $18.05$ | $43.10$ |  288 $\times$ 288 | $15.51$ | $59.9$
DiMP-50 [97] | $30.67$ | $43.10$ |  288 $\times$ 288 | $10.35$ | $56.8$
Ocean [100] | $18.60$ | $37.16$ |  255 $\times$ 255 | $20.26$ | $56.0$
SiamAttn [98] | $15.72$ | $140.24$ |  255 $\times$ 255 | $-$ | $56.0$
SiamFC++ (GoogLeNet) [99] | $45.27$ | $13.89$ |  303 $\times$ 303 | $-$ | $54.3$
SiamGAT [33] | $41.99$ | $14.23$ |  287 $\times$ 287 | $17.28$ | $53.9$
SiamRN [34] | $6.51$ | $56.56$ |  255 $\times$ 255 | $116.87$ | $52.7$
SiamBAN [31] | $23.71$ | $53.93$ |  255 $\times$ 255 | $48.84$ | $51.4$
SiamRPN++ [29] | $5.17$ | $53.95$ |  255 $\times$ 255 | $48.92$ | $49.6$
SiamDW (CIRResNet22-FC) [96] | $52.58$ | $2.46$ |  255 $\times$ 255 | $12.90$ | $38.5$

We have evaluated the efficiency of the trackers in terms of their tracking
speed, number of parameters, and the number of floating point operations
(FLOPs) in their tracking model. Tracking speed is important for many real-
world applications and hence we have considered it as an important metric in
efficiency comparison. It is computed by calculating the average number of
frames processed by an approach in a second. Since the reported tracking
speeds of the approaches depend on the hardware and implementation platforms,
we calculated the tracking speed of the trackers on the same hardware platform
using the PyTorch deep learning framework.

The number of parameters is another efficiency metric in deep learning-based
tracking approaches, as models with fewer parameters are more hardware-
efficient and can work on small devices such as mobiles and tablets. Number of
parameters of a tracking approach are the total number of arguments passed to
the optimizer and in most situations it does not rely on the input size of the
tracking approach. We have used the PyTorch default function to measure the
total number of parameters of a tracking model. We have considered the number
of floating point operations (FLOPs) as the third metric to measure the
efficiency of a tracking model. Except for fully-Transformer based trackers,
the number of FLOPs of an approach depends on the tracking model and
corresponding search image size since the target template features are
computed only in the first frame of a tracking sequence. In fully-Transformer
based trackers, features of the target template are computed in each and every
frame, and hence the size of the template influences the FLOPs. Although the
number of FLOPs of some trackers is very high, they are still able to track a
target at high speed since their models are highly parallel, and nowadays GPUs
can handle them successfully. However, these approaches are not suitable for
some applications that only run on CPUs and mobile devices. The overall
efficiency results of the trackers, along with their corresponding search
image size and success scores on the LaSOT benchmark, are reported in Table 6.

Based on the obtained efficiency results in Table 6, CNN-based trackers showed
better performances than the other two types of trackers. In particular,
SiamDW [96] tracker achieved top efficiency results with 52.58 FPS of tracking
speed and 2.46 million of parameters by using the light-weighted cropping-
inside residual units based CNN backbone. The SiamGAT [33] tracker achieved a
tracking speed of 41.99 FPS. Its tracking model has 14.23 million parameters
and 14.23 gigaFLOPs for a search image size of $287\times 287$. Although most
of the recent CNN-based trackers are computationally efficient, SiamRPN++ [29]
tracker achieved poor efficiency results with 5.17 FPS of tracking speed since
it used a deeper CNN backbone architecture for feature extraction. The SiamRN
[34] approach also showed the second-lowest efficiency results among the CNN-
based trackers, with an average tracking speed of 6.51 FPS and 116.87
gigaFLOPs, due to its computationally expensive relation detector module.

Overall, most of the CNN-Transformer based trackers successfully balance
tracking robustness and computational efficiency. In particular, the number of
FLOPs of the CNN-Transformer based trackers is considerably lower than the
other two categories because they successfully capture robust cues from CNN-
based features, even with a lightweight backbone network. Particularly,
AiATrack [56] tracker achieved 31.22 FPS of the average tracking speed and
17.95 million parameters while maintaining 69% of success score on the LaSOT
dataset. Although the AiATrack approach searches the target in a large search
region, it has only 9.45 gigaFLOPs because its model update mechanism uses a
feature-reusing technique to avoid additional computational costs. Among the
CNN-Transformer based trackers, HiFT [50] approach achieved top efficiency
results with 37.06 FPS of the tracking speed and 11.07 million of parameters
while showing a considerable tracking accuracy. Since the HiFT tracker
utilized the lightweight AlexNet [20] as the backbone feature extraction
network, it achieved top efficiency results with fewer FLOPs. Based on its
average tracking speed, the CSWinTT [55] tracker showed poor results with 8.76
FPS because the cyclic shifting attention mechanism of this approach is
computationally expensive.

Although most of the fully Transformer-based trackers are computationally
inefficient, GRM-B [67], SimTrack-B/16 [61], and SparseTT [59] trackers
considerably balance the tracking robustness and efficiency. Since the GRM
tracker searches for the target in a narrower region compared to other fully-
Transformer trackers and avoids unnecessary attention computation by
eliminating background patches, it successfully balances accuracy and
efficiency. The reason behind the better efficiency of the SimTrack approach
is its simple and computationally efficient ViT-B/16 backbone Transformer
architecture. Similarly, the SparseTT [59] tracking model has a computational
load of 9.21 gigaFLOPs since it utilizes the smallest version of the Swin
Transformer [91] as the backbone network. Also, the SparseTT tracker removed
the non-similarity image pairs between target template and search image in the
searching process and hence improved the tracking efficiency.

Among the fully-Transformer based approaches, SeqTrack [69] achieved lower
efficiency results with 5.81 FPS of tracking speed, 308.98 million parameters,
and 535.85 gigaFLOPs. Although SeqTrack has shown superior accuracy in several
benchmarks, it is computationally expensive since it relies on the large-scale
ViT-L model. Additionally, unlike other trackers, SeqTrack uses a large size
($384\times 384$) of the target template and search region, leading to an
increase in the number of floating-point operations. The MixFormer-L [60]
tracker also achieved low efficiency results with a tracking speed of 8.02
FPS, 195.40 million parameters, and 113.02 gigaFLOPs. The tracking model of
the MixFormer-L approach was created from the wider model of CVT Transformer
[92] (CVT-W24), and hence it achieved poor results in terms of efficiency.
Although the ProContEXT [63] tracker is a follow-up work of OSTrack [62]
tracking approach, it can able to track a target at the half speed of the
OSTrack since its context-aware self-attention module is computationally
expensive and hence the number of FLOPs are doubled.

In summary, CNN-based trackers achieve better results in terms of tracking
speed, number of parameters, and number of floating-point operations. However,
their tracking robustness is poorer than the other two categories. On the
other hand, although fully-Transformer based trackers show excellent tracking
robustness, their efficiency is lower than the other two types. Number of
floating point operations of the fully-Transformer based approaches are
generally higher than the CNN-Transformer based and CNN-based trackers because
of their attention mechanism. CNN-Transformer based trackers successfully
balance tracking robustness and efficiency by combining CNN-based feature
extraction and Transformer-based feature fusion.

##  6 Discussions

In this section, we discuss the summary of the findings based on the
literature review and experimental survey, as well as the future directions of
Transformer-based single object tracking.

###  6.1 Summary of Findings

This survey study focused on analyzing the literature and performances of a
subset of visual object trackers that use Transformers in their tracking
pipeline. We covered different types of Transformer-based trackers and
analyzed their individual performances based on how they addressed the
tracking challenges. In addition, we compared the performances of Transformer
trackers with state-of-the-art CNN-based trackers to demonstrate how they have
surpassed them by a significant margin within a short period of time.

Before the introduction of the Transformer in object tracking, CNN-based
trackers dominated the tracking world. Specifically, Siamese-based approaches
achieved a considerable balance between tracking robustness and efficiency in
benchmark datasets by treating object tracking as a template matching problem.
However, since CNN-based Siamese tracking approaches primarily rely on the
correlation operation, which is a local linear matching process, their
performance is limited in challenging tracking scenarios. Additionally, based
on the results of our experimental analysis on the large-scale LaSOT
benchmark, the long-term tracking capability of CNN-based trackers is very
limited. This limitation arises from their disregard for temporal cues in the
template matching process. Furthermore, attribute-wise experimental results
showed that CNN-based trackers still struggle to track a target in fully
occluded, rotated, viewpoint-changed, and scale-varied scenarios due to their
poor target discriminative and feature matching capabilities. While all CNN-
based trackers have demonstrated poor tracking robustness in recently
developed challenging datasets, they still exhibit excellent performance in
the OTB100 dataset. This notable distinction arises from the fact that the
targets in the OTB100 dataset do not undergo substantial appearance
alterations, unlike those in the other challenging datasets.

The Transformer was initially introduced to the single-object tracking
community as a module with a CNN backbone, and these approaches are referred
to as CNN-Transformer based trackers in this study. Researchers replaced the
correlation operation of Siamese tracking approaches with a Transformer
architecture in CNN-Transformer based trackers, utilizing attention mechanism.
Based on the results of this study, CNN-Transformer based trackers
successfully balanced tracking robustness and efficiency in benchmark
datasets. Although their tracking robustness is good in short-term and aerial
tracking sequences, their long-term tracking capabilities are considerably
lower than fully-Transformer based approaches. Compared to CNN-based trackers,
CNN-Transformer based approaches successfully utilize the backbone CNN
architecture for feature extraction, and they show excellent performances even
with smaller pre-trained CNN models. However, our attribute-wise experimental
analysis reveals that CNN-Transformer based trackers exhibit limited
performance in tracking scenarios involving full occlusion, low resolution,
out-of-view targets, and illumination variation since their inability to
capture the temporal cues and target-specific cues using CNN features.

Recent VOT approaches fully rely on Transformer architectures, leveraging
their global feature learning capabilities in object tracking. Based on their
model architecture, we classify the literature of fully-Transformer approaches
into two categories: Two-stream Two-stage trackers and One-stream One-stage
trackers. Two-stream Two-stage trackers perform feature extraction and fusion
in two distinguishable stages, utilizing two identical Transformer network
branches and another Transformer network, respectively. On the other hand,
One-stream One-stage trackers use a single pipeline of a Transformer network.
Based on our experiential evaluation on challenging benchmark datasets, fully-
Transformer based trackers significantly outperform other approaches by a wide
margin while maintaining moderate efficiency scores.

Our experimental comparison study clearly shows that One-stream One-stage
fully-Transformer trackers significantly outperform other types of trackers
and are expected to dominate the single object tracking community for the next
couple of years. Since the feature extraction and fusion are conducted
together by a single Transformer network architecture in these trackers, the
information flow between the features of target template and search region
patches is efficiently utilized. As a result, target-specific discriminative
features are enhanced, while unnecessary background and distractor features
are eliminated. Therefore, the One-stream One-stage trackers successfully
handle long-term tracking scenarios, even when the target undergoes severe
appearance changes and full occlusion.

Based on our experimental study, we have found that most One-stream One-stage
fully-Transformer trackers exhibit poor efficiency compared to CNN-based
trackers in terms of FLOPs, tracking speed, and the number of parameters. The
primary reason for this is that ViT based Transformers employ a self-attention
mechanism that calculates attention between all patches, resulting in a
quadratic computational complexity in the input sequence. Additionally,
certain trackers such as SeqTrack [69], which utilize encoders and decoders
for self and cross-attention, exhibit very poor computational efficiency. In
addition to that reason, most Transformer trackers search for the target in a
larger search region compared to CNN-based trackers, which results in slower
tracking speeds.

The experimental results of this study clearly indicate that the performance
of recent trackers on the OTB100 dataset has reached saturation, as evident
from the overall performances and attribute-wise results. As a result, some
recent trackers have already observed this fact and started avoiding the
evaluation of their performances on OTB100. We have also observed that while
the tracking accuracy of some trackers is excellent on certain challenging
benchmark datasets, their performances are only average on the GOT-10k dataset
since these trackers tend to be biased towards familiar target objects. These
findings justify the need for a massive tracking dataset comprising long
tracking sequences and a large number of non-overlapping training and testing
target object classes, which will truly evaluate the performance of future
trackers.

The increasing number of Transformer-based approaches proposed this year,
along with their exceptional performances, clearly indicates that Transformers
have replaced CNNs in single object tracking. These trackers are expected to
continue impacting the tracking community in the coming years, as they have
introduced novel possibilities and perspectives, such as treating object
tracking as a sequence learning problem rather than the template matching. The
influence of Transformer-based language models on the field has further
inspired researchers to propose similar architectures for object tracking.
Moreover, recent advancements in self-supervised learning-based mask
autoencoders have motivated the development of more robust trackers that
effectively address data redundancy issues and capture more generalized
representations. Consequently, it is evident that Transformer-based trackers
are positioned to dominate the field of single object tracking in the future.

###  6.2 Future Directions of Transformer Tracking

Our experimental survey clearly shows that One-stream One-stage fully-
Transformer based trackers significantly outperform other Transformer-based
trackers by a large margin. Despite their impressive performance on
challenging benchmark datasets, there are still several issues that need to be
addressed and require further attention in future work. Taking this into
account, we provide some recommendations for future directions by exclusively
considering the One-stream One-stage fully-Transformer based trackers.

Designing a new spatio-temporal Transformer architecture for object tracking
to capture the spatial and temporal cues together: All of the current
Transformer-based trackers utilize the Vision Transformer (ViT) [37] or a
variant of ViT as their backbone and fine-tune it for tracking. These
Transformer backbone networks were originally designed for image recognition
or object classification tasks, primarily focusing on capturing spatial
relationships within individual images. However, they lack the inherent
capability to capture temporal cues between a sequence of continuous reference
frames, which is crucial for object tracking. This limitation stems from the
fundamental difference between object classification and tracking tasks. To
overcome this limitation, there is a need to design a novel spatio-temporal
Transformer architecture specifically tailored for tracking tasks, which
effectively captures spatial and temporal cues together. These Transformers
can be capture spatial cues by computing attention between image patches
within a frame and temporal cues by computing attention between patches in
adjacent frames. Similar to the recently proposed spatio-temporal Transformers
in other tasks [111, 112], a novel spatio-temporal Transformer could be
employed to capture spatial-temporal cues from the continuous reference frame
sequence, leading to the attainment of more robust results.

Improving computational efficiency through utilization of lightweight
Transformer architectures, quantization techniques, and feature reusing
techniques: Although fully-Transformer based trackers have shown outstanding
tracking robustness, their computational efficiencies are considerably poor
and hence not suitable for many real-world applications. Recently, several
lightweight Transformers have been proposed [113, 114, 115], demonstrating
excellent efficiency scores while maintaining accuracy in various computer
vision tasks. By utilizing these lightweight Transformers in VOT,
computational costs could be improved. In addition, similar to the work of
[116], utilizing proper quantization techniques could reduce the computational
complexity of Transformers without compromising performance. Moreover, the
current Transformer trackers process the template frame patches in each and
every frame using an encoder, resulting in increased computational costs. To
address this issue, an efficient feature reusing mechanism could be employed
to reduce the model’s computational complexity.

Utilizing self-supervised learning based masked autoencoder pre-trained models
to enhance tracking performance: Recent Transformer trackers [62, 63, 67, 69]
have enhanced their tracking accuracy by utilizing self-supervised learning
based masked autoencoder pre-trained models [109] to initialize the tracker
encoder. These pre-trained models helped the tracker extract more
discriminative features by focusing on relevant features and discarding
irrelevant or noisy information. In addition, the trackers that utilized these
pre-trained models showed better performance for unseen target objects due to
the generalization capability of the masked autoencoders. Due to these facts,
self-supervised learning-based masked autoencoder pre-trained models could be
further investigated as a potential future direction for object tracking.
Additionally, since current masked autoencoders are primarily designed to
capture spatial cues within an image for image recognition tasks, there is a
need to develop a self-supervised pre-trained model for object tracking that
can effectively capture both spatial and temporal cues. However, it is
important to note that developing these powerful pre-trained mask autoencoder
models requires significant computational resources and can be expensive for
real-time tracking tasks. Therefore, it is crucial to adopt an efficient fine-
tuning mechanism, similar to LoRA [117], to reduce the computational
complexity without compromising tracking performance.

Enriching the accuracy of fully-Transformer trackers to track a small target
object with less appearance cues: The experimental results on the UAV123
dataset show that fully-Transformer based approaches struggle to track small
target objects with limited appearance cues, as other types of trackers
outperform them in attribute-wise comparisons. In particular, their success
scores are poor when tracking small target objects in full occlusion,
background clutter, and low resolution scenarios. Since the patch-level
attention mechanism of the Transformer fails to capture the correct appearance
cues of a small target object, fully-Transformer based trackers exhibit