# MixFormer: End-to-End Tracking with
Iterative Mixed Attention

Yutao Cui, Cheng Jiang, Gangshan Wu, and Limin Wang Yutao Cui, Cheng Jiang,
Gangshan Wu and Limin Wang are with the State Key Laboratory for Novel
Software Technology, Nanjing University, Nanjing, China, 210023
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>lmwang@nju.edu.cn).

###### Abstract

Visual object tracking often employs a multi-stage pipeline of feature
extraction, target information integration, and bounding box estimation. To
simplify this pipeline and unify the process of feature extraction and target
information integration, in this paper, we present a compact tracking
framework, termed as MixFormer, built upon transformers. Our core design is to
utilize the flexibility of attention operations, and propose a Mixed Attention
Module (MAM) for simultaneous feature extraction and target information
integration. This synchronous modeling scheme allows to extract target-
specific discriminative features and perform extensive communication between
target and search area. Based on MAM, we build our MixFormer trackers simply
by stacking multiple MAMs and placing a localization head on top.
Specifically, we instantiate two types of MixFormer trackers, a hierarchical
tracker MixCvT, and a non-hierarchical tracker MixViT. For these two trackers,
we investigate a series of pre-training methods and uncover the different
behaviors between supervised pre-training and self-supervised pre-training in
our MixFormer trackers. We also extend the masked pre-training to our
MixFormer trackers and design the competitive TrackMAE pre-training technique.
Finally, to handle multiple target templates during online tracking, we devise
an asymmetric attention scheme in MAM to reduce computational cost, and
propose an effective score prediction module to select high-quality templates.
Our MixFormer trackers set a new state-of-the-art performance on seven
tracking benchmarks, including LaSOT, TrackingNet, VOT2020, GOT-10k, OTB100
and UAV123. In particular, our MixViT-L achieves AUC score of 73.3% on LaSOT,
86.1% on TrackingNet, EAO of 0.584 on VOT2020, and AO of 75.7% on GOT-10k.
Code and trained models are publicly available at https://github.com/MCG-
NJU/MixFormer.

###### Index Terms:

Visual tracking, transformer, mixed attention, compact tracking framework,
vision transformer, convolutional vision transformer.

##  1 Introduction

Visual object tracking [6, 59, 51, 24, 34, 6, 2, 61] has been a long-standing
problem in computer vision area for decades, which aims to estimate the state
of an arbitrary target in video sequences given its initial state. It has been
deployed in various applications such as human computer interaction [48] and
visual surveillance [78]. However, so far how to design a simple yet effective
end-to-end tracker is still challenging in the real-world scenarios. The main
challenges are from aspects of scale variations, object deformations,
occlusion, and confusion from similar objects.

Current prevailing trackers [3, 40, 13, 88, 51, 18, 4, 15, 11, 70, 80]
typically employ a multi-stage pipeline as shown in Fig. 1. It contains
several components to accomplish the tracking task: (1) a backbone to extract
generic features of tracking target and search area, (2) an integration module
to allow information communication between tracking target and search area for
subsequent target-aware localization, (3) task-specific heads to precisely
localize the target and estimate its bounding box. Integration module is the
key of tracking algorithms as it is responsible for incorporating the target
information to bridge the steps of generic feature extraction and target-aware
localization. Traditional integration methods include correlation-based
operations (e.g. SiamFC [3], SiamRPN [40], CRPN [24], SiamFC++ [79], SiamBAN
[13], OCEAN [88]) and online learning algorithms (e.g., DCF [51], KCF [32],
CSR-DCF [52], ATOM [18], DiMP [4], FCOT [15]). Recently, thanks to its global
and dynamic modeling capacity, Transformers [66] are introduced to perform
attention based integration and yields good tracking performance (e.g., TransT
[11], TMT [70], STMTrack [25], TREG [14], STARK [80], DTT [83]). However,
these transformer based trackers still depend on the CNN for generic feature
extraction, and simply apply attention operations for target information
integration in the latter high-level and abstract representation space. We
analyze that these CNN representations are limited as they are typically pre-
trained for generic object recognition and might neglect finer structure
information for tracking. In addition, these CNN representations employ local
convolutional kernels and are ineffective in global modeling. Therefore, the
decoupled CNN representation and integration modules still fail to fully
unleashing power of attention operations (Transformers) in the whole tracking
pipeline.

Figure 1: Comparison of tracking pipeline. (a) The dominant tracking framework
contains three components: a convolutional or transformer backbone, a
carefully-designed integration module, and task-specific heads. (b) Our
MixFormer is more compact and only composed of two components: a target-search
mixed attention based backbone and a simple localization head.

To overcome the above issues, we present a new perspective on tracker design
by unifying visual feature extraction and target information integration
within a single module. This unified processing paradigm shares several key
advantages. First, it enables the feature extraction to be aware of the
corresponding tracking target and thus capture more target-specific features.
Second, it also allows the target information to be more extensively
integrated into search area, and thereby better to capture their correlation.
In addition, as shown in Fig. 1, this will result in a more compact and
efficient tracking pipeline only with a single backbone and a tracking head,
without an explicit integration module.

To this end, in this paper, we present the _MixFormer_ , a neat tracking
framework designed for unifying the feature extraction and target integration
with a transformer-based architecture. Attention module is a very flexible
architectural building block with dynamic and global modeling capacity, which
makes few assumption about the data structure and could be applied for general
relation modeling. Our core idea is to utilize this flexibility of attention
operation, and devise the mixed attention module (MAM) that performs both
feature extraction and information interaction of target template and search
area at the same time. Specifically, in our MAM, we propose a hybrid
interaction scheme with both self-attention and cross-attention operations on
the tokens from target template and search area. The self-attention is
responsible to extract their own features of target template or search area,
while the cross-attention allows for the communications between them to mix
their information. To further reduce the computational cost of MAM and allow
for multiple templates to handle object deformation, we further present a
customized asymmetric attention scheme by pruning the unnecessary target-to-
search area cross-attention. This enables our MixFormer trackers to be easily
adapted for multiple target template inputs. In practice, we propose a score
based target template update mechanism, to select reliable and high-quality
online templates.

We instantiate two types of MixFormer trackers, a hierarchical tracker with
progressive downsampling and depth-wise convolutional projections (termed as
MixCvT), and a non-hierarchical tracker built upon the plain backbone of ViT
(termed as MixViT). For the former one, motivated by the success of hybrid
convolutional and transformer backbones [73, 72, 44, 85, 49], we want to
introduce the inductive bias in the tracker design by using a progressive
downsampling architecture and introducing locality modeling in the Wide Mixed
Attention Module (W-MAM). Specifically, we use the CvT [73] backbone to build
our hierarchical MixFormer tracker MixCvT by stacking the layers of
Convolutional Patch Embedding and W-MAM and finally placing a simple
localization head on top. Due to the progressive downsampling and locality
modeling in MixCvT, it exhibits desirable properties (i.e., shift, scale and
distortion invariance) of CNNs [73] and achieves promising performance.
However, these adaptations yield a hierarchical architecture with a tremendous
amount of tokens in the first two stages and relatively complex computational
steps.

In the other instantiation, we explore a more general,  _simpler_ and  _non-
hierarchical_ backbone of Vision Transformer (ViT) [21] for designing
MixFormer tracker. Motivated by its design simplicity and promising results of
ViT in image and video recognition, we aim to build a plain MixFormer tracker
MixViT without inductive bias for high modeling flexibility and tracking
efficiency. We construct the MixViT tracker with a simple patchification
scheme and the Slimming Mixed Attention Module (S-MAM) by removing the
convolutional projection in Wide MAM. Specifically, we first slice the image
into non-overlapping patches and project them into a token sequence with patch
embeddings and position embeddings. Then we stack multiple S-MAMs on the
fixed-length token sequence and finally place a pyramidal localization head.
For these two types of MixFormer trackers, we design customized localization
heads for accurate bounding box estimation. Especially, we design a pyramidal
corner head for non-hierarchical MixViT, which can supplement some multi-scale
information.

Due to the small size of tracking datasets and the overall complexity of
tracking task, pre-training is an necessary step for training effective
transformer-based trackers. The existing successful trackers [3, 40, 13, 88,
14, 12, 80] often uses the ImageNet pre-trained models to initialize the
weights of their feature extraction module. Although we mix the tokens from
the target template and search area in our MAM, we could directly apply the
ImageNet pre-trained CvT or ViT models as the initialization of the
corresponding trackers, thanks to the flexibility of attention operations. In
practice, for MixCvT, we empirically study the supervised pre-training on
ImageNet-1k and ImageNet-22k, demonstrating that pre-trained model with more
semantic supervision also contributes to a better tracking performance. For
MixViT, due to its less inductive bias and higher flexibility, the ViT unlocks
the capability of dropping image patch tokens and unleash the strong power of
masked autoencoder (MAE) [30] as a scalable visual learner. We extensively
explore the supervised and self-supervised pre-training strategies for our
MixViT, and also try to implement a new pre-training strategy TrackMAE solely
on the tracking datasets without ImageNet. Our TrackMAE achieves very
competitive performance to the large-scale ImageNet pre-training. We perform
extensive experiments on several benchmarks and demonstrate that a series of
instantiated MixFormer trackers (including MixCvT, MixCvT-L, MixViT and
MixViT-L) largely surpass the state-of-the-art trackers. Especially, with
plain ViT pre-trained by MAE, our MixViT achieves the AUC of 69.6% on LaSOT
and AO of 72.5% on GOT-10k with a real-time running speed of 75 FPS. The main
contributions are summarized as follows:

  * •

We propose a compact end-to-end tracking framework, termed as _MixFormer_ ,
based on Mixed Attention Modules (MAM). MAM allows for extracting target-
specific features and extensive communication between target and search
simultaneously.

  * •

We instantiate two types of MixFormer trackers, a hierarchical tracker with
progressive downsampling and depth-wise convolutional projections (coined as
MixCvT), and a non-hierarchical tracker built upon plain backbone of ViT
(coined as MixViT). Especially, we design a pyramidal corner head for plain
MixViT, which can supplement the multi-scale information for accurate target
localization.

  * •

We empirically study a series of pre-training strategies for our MixFormer
framework and uncover the important ingredients of developing an efficient and
effective tracker. In particular, we come up with simple self-supervised pre-
training strategy TrackMAE, which achieves competitive performance to the
large-scale ImageNet pre-training.

  * •

For online template update, we devise a customized asymmetric attention in MAM
for high efficiency, and propose an effective score prediction module to
select high-quality templates, leading to an efficient and effective online
transformer-based tracker.

  * •

The proposed MixCvT and MixViT significantly outperform the state-of-the-art
trackers on seven challenging benchmarks, including VOT2020 [36], LaSOT [23],
TrackingNet [58], GOT-10k [33], VOT2022, UAV123 [57] and OTB100 [74].

A preliminary version of this work [16] was published on the conference of
IEEE CVPR 2022 as an oral presentation. In this new version, we make the
following extensions: i) we instantiate two types of MixFormer framework, the
original MixCvT based on Wide Mixed Attention Module and the new MixViT based
on Slimming Mixed Attention Module (S-MAM). MixViT generally enjoys better
tracking performance and higher running speed. ii) For the non-hierarchical
tracker MixViT, we design a customized pyramidal corner head, supplementing
some multi-scale information for bounding box localization. iii) We further
study a series of pre-training methods for MixFormer framework and uncover the
key ingredient of training a more effective tracker. In addition, we proposed
a new pre-training strategy of TrackMAE solely on the tracking dataset and
demonstrate its competitive performance to the large-scale ImageNet pre-
training. iv) We provide more experimental analysis on the design of MixFormer
trackers and the pre-training methods. Our MixViT-L obtains consistent
performance improvement over the original MixCvT, and sets a new state-of-the-
art performance on most of the tracking datasets. We also report the
performance of MixViT on the latest VOT 2022 challenge and it (MixFormerL)
obtains the best performance of 0.602 EAO score on the VOT-STb2022 public
challenge.

##  2 Related Work

###  2.1 Tracking Paradigm

Current prevailing tracking methods often employ a three-stage architecture,
containing (i) a backbone to extract generic features, (ii) an integration
module to fuse the information of target and search region, (iii)
classification and localization heads to produce the target states. Generally,
most modern trackers [39, 18, 4, 12, 67] used ResNets as the backbones. For
the most important integration module, researchers have explored various types
of methods. First, Siamese-based trackers [3, 40, 29, 91, 79, 71] attract a
lot of attention due to its simplicity and efficiency. These methods combined
a correlation operation with the Siamese network, modeling the appearance
similarity and correlation between the target and search. SiamFC [3] employed
a Siamese network to measure the similarity between the template and the
search area with a high tracking speed. SiamRPN++ [39] improved cross
correlation to depth-wise cross correlation, which can increase both the
performance and efficiency. For more precise regression, CGACD [22] and Alpha-
Refine [81] proposed pixel-wise cross correlation. PG-Net [45] then replaced
the cross-correlation with pixel-to-global matching correlation, which
combined the advantages of the both two and achieved surprising performance.

Second, in contrast to Siamese trackers, another family of trackers [17, 32,
34, 41, 51, 18, 4, 15] learned an online target-dependent discriminative model
for object tracking. [7, 32, 51] employed online correlation filter to
distinguish targets from background. However, these approaches rely on
complicated online learning procedures that cannot be easily deployed in an
end-to-end learning architecture. To gain benefits from end-to-end training,
CFNet [65] integrated the correlation filter into deep network. DiMP [4]
further introduced a target model predictor to online optimizing the target
model instructed by the discriminative loss, which achieved leading
performance in various benchmarks at that time.

Finally, some recent trackers [12, 80, 14, 25, 70] introduced a transformer-
based integration module to capture the global dependencies between target and
search area. CGCAD [22] and SiamAttn [84] introduced a correlation-guided
attention and self-attention to perform discriminative tracking. TransT [12]
developed a ego-context augment module based on self-attention and a cross-
feature augment module based on cross-attention, which established
associations between distant features and thus achieving excellent
performance. STARK [80] implemented end-to-end tracking with the designed
transformer encoder and decoder. However, these transformer trackers still
follow the traditional paradigm of _Backbone-Integration-Head_. TREG [14]
proposed a target-aware transformer for regression branch and can generate
accurate prediction in VOT2021 [38] challenge. Inspired by TREG, we formulate
mixed attention mechanism by using both self attention and cross attention. In
this way, our MixFormer unifies the two processes of feature extraction and
information integration with an iterative MAM based backbone, leading to a
more compact, neat and effective end-to-end tracker. After our CVPR 2022
conference version [16], there are some co-occurent works in ECCV 2022 such as
OSTrack [82] and SimTrack [9], which also use Transformer as backbone to
perform both feature extraction and information fusion. Our work starts
earlier and obtains better performance than them due to our customized design
of localization head, well-explored pre-trained strategies and backbone, and
effective online templates selection method. Compared to OSTrack, our
MixFormer framework works without any post-processing such as hanning window
penalty, thus yielding a more simple tracker.

###  2.2 Transformers in Computer Vision

The Vision Transformer (ViT) [21] presented a pure transformer architecture
for image modeling, with a patch embedding to convert the 2D image into a
token sequence and stacked layers of multi-head self-attention (MHSA) [66] on
the visual tokens. ViT has obtained an impressive performance on image
classification and also shown the great potential on vision tasks such as
object detection [43]. Some works [49, 85, 44, 72] introduced the customized
designs into ViT to capture local context in images. For example, PVT [72]
incorporated a multi-stage design (without convolutions) for Transformer
similar to multi-scales in CNNs. Swin Transformer [49] proposed a hierarchical
architecture with the designed shifted window scheme. CvT [73] combined CNNs
and Transformers to model both local and global dependencies for image
classification in an efficient way. Our MixFormer aims to extend vision
transformer to the object tracking area. Inspired by CvT [73] and ViT [21], we
instantiate two types of MixFormer, including the _hierarchical_ MixCvT and
the _non-hierarchical_ MixViT. MixCvT aims to utilize the inductive bias of
the multi-stage design and the adoption of depth-wise convolutional
projections to relieve the data requirement. MixViT try to enjoy the design
simplicity and high efficiency of ViT backbone, and present a faster tracker.
But due to its plain structure without inductive bias, MiXViT often requires
to careful initialization, such as supervised pre-training or MAE pre-
training.

Compared to ViT [21] and CvT [73], there are some fundamental differences. (i)
The proposed MAM performs dual attentions for both feature extraction and
information integration, while ViT and CvT uses self attention to solely
extract features. (ii) The tasks are different, and the corresponding input
and the head are different. We use multiple templates together with the search
region as input and employ a corner-based or query-based localization head for
bounding box generation, while ViT and CvT are originally designed for image
classification. (iii) We further introduce an asymmetric mixed attention and a
score prediction module for the specific task of online tracking.

###  2.3 Pre-training of Trackers

Pre-training has been well recognized as an effective initialization method to
improve the training stability and boost the performance of downstream tasks.
In the past few years, the most common way [8, 69, 39, 18] is to pre-train the
backbone of trackers with the image classification models from ImageNet [20],
containing more than one million labeled images. Recently, self-supervised
learning [31, 10, 30, 63] has attracted broad attention due to its superiority
to traditional supervised learning on multiple tasks. BEiT [1] first
introduced masked image modeling (MIM) via a two-stage training process, i.e.
tokenizer training and token-based masked image modeling. MAE [30] proposed an
end-to-end masked autoencoder to encode representation for unmasked patches
and predicted the masked patches. Concurrently, SimMIM [77] presented a
similar masked image modeling architecture.

In tracking area, most of the prevailing trackers pre-train the backbone on
ImageNet-1k [20]. OSTrack [82] first employed MAE pre-training to its ViT-
based tracker. In this work, we first demonstrate that the MixCvT pre-trained
on a larger dataset ImageNet-22k significantly outperforms that on
ImageNet-1k. Then we experiment on MixViT and uncover the differences between
the MAE pre-training and supervised pre-training with detailed ablations on
depth of ViT backbone. To further verify the effectiveness of MAE pre-training
on tracking, we propose a new masked pre-training technique (TrackMAE) solely
on the tracking datasets instead of the large-scale ImageNet, and demonstrate
its excellent performance on MixViT pre-training.

##  3 Method

In this section, we present our end-to-end tracking framework, termed as
MixFormer, based on iterative mixed attention modules (MAM), and the overall
pipeline is depicted in Fig. 1 (b). We instantiate two types of MixFormers,
including the hierarchical MixCvT built upon W-MAM and the non-hierarchical
MixViT built upon S-MAM. Specifically, MixCvT aims to utilize the visual
inductive bias with progressive downsampling to embrace the success of hybrid
structure of CNN and transform. MixViT focuses on designing a simpler and more
efficient tracker with less inductive bias, but requires careful pre-training
to achieve superior performance. In Section 3.1, we introduce our proposed MAM
to unify the process of feature extraction and target information
incorporation. This simultaneous processing scheme will enable our feature
extraction to be more specific to the corresponding tracking target. In
addition, it also allows the target information integration to be performed
more extensively and thus to better capture the correlation between target and
search area. Then, we describe MixCvT and MixViT in Section 3.2 and Section
3.3 respectively. Furthermore, we introduce the details of pre-training
methods in Section 3.4. Finally, in Section 3.5, we describe the training and
inference of MixCvT and MixViT, and the confidence score based target template
update mechanism to handle object deformation in tracking procedure.

###  3.1 Mixed Attention Module (MAM)

Figure 2: (a) Mixed Attention Module (MAM) is a flexible attention operation
that unifies the process of feature extraction and information integration for
target template and search area. This mixed attention has dual attention
operations where self-attention is performed to extract features from itself
while cross-attention is conducted to communicate between target and search.
This MAM could be easily implemented with a concatenated token sequence. To
further improve efficiency, we propose an asymmetric MAM by pruning the
target-to-search cross attention (denoted by dashed lines) (b) We instantiate
two types of MAMs, including Wide MAM for MixCvT and Slimming MAM for MixViT.
The difference only lies in the share-weights projections.

Mixed attention module (MAM) is the core design to pursue a compact end-to-end
visual object tracker. The input to our MAM is the target template and search
area. MAM aims to simultaneously extract their own visual features and fuse
the interaction information between them. In contrast to the original Multi
Head Attention [66], MAM performs dual mixed attention operations on two
separate tokens sequences of target template and search area. It carries out
self-attention on tokens from each sequence themselves to capture the target
or search area information. Meanwhile, it conducts cross-attention between
tokens from two sequences to allow information communication between target
template and search area. As shown in Fig. 2, this mixed attention mechanism
could be implemented efficiently via a concatenated token sequence. In this
work, we instantiate two types of MAM, including Wide MAM for MixCvT and
Slimming MAM for MixViT. The former one introduces the desirable properties of
CNN (i.e. shift, scale, and distortion invariance) to transformer architecture
via the depth-wise convolutional projections. This enables the tracker to
realize promising performance when the scale of training data is limited.
While the latter one pursue a simple and universal transformer operation. The
details of W-MAM and S-MAM are described as follows.

Wide Mixed Attention Module (W-MAM). Formally, given a concatenated tokens of
multiple targets and search, we first split it into two parts and reshape them
to 2D feature maps. In order to achieve additional modeling of local spatial
context, a separable depth-wise convolutional projection layer is performed on
each feature map (i.e., _query_ , _key_ and _value_). It also provides
efficiency benefits by allowing the down-sampling in _key_ and _value_
matrices. Then each feature map of target and search is flattened and
processed by a linear projection to produce queries, keys and values of the
attention operation. We use $q_{t}$, $k_{t}$ and $v_{t}$ to represent target,
$q_{s}$, $k_{s}$ and $v_{s}$ to represent search region. The mixed attention
is defined as:

$\displaystyle k_{m}={\rm Concat}(k_{t},k_{s}),\ \ \ v_{m}={\rm
Concat}(v_{t},v_{s}),$ (1) $\displaystyle{\rm Attention_{t}}={\rm
Softmax}(\frac{q_{t}k_{m}^{T}}{\sqrt{d}})v_{m},$ $\displaystyle{\rm
Attention}_{s}={\rm Softmax}(\frac{q_{s}k_{m}^{T}}{\sqrt{d}})v_{m},$

where $d$ represents the dimension of the key, ${\rm Attention}_{t}$ and ${\rm
Attention}_{s}$ are the attention maps of the target and search respectively.
It contains both self attention and cross attention which unifies the feature
extraction and information integration. Then the targets token and search
token are concatenated and processed by a linear projection. Finally, the
concatenated token sequence is processed by a Layer Normalization and a MLP
function as in Fig. 2.

Slimming Mixed Attention Module (S-MAM). While the W-MAM produces a strong
visual representation through introducing translation-equivariant priors to
vanilla transformer (i.e., depth-wise convolutional projection), it also
possesses the a few issues. Primarily, it leads to a lower tracking speed,
since the reshaping operation and depth-wise convolution operations for each
attention element (i.e., query, key and value). In addition, it lacks the
flexibility to adapt to the recent ViT development, such as the simple masked
pre-training [30]. To address these issues, we further present the Slimming
Mixed Attention Module (S-MAM) by removing the depth-wise convolutional
projection in W-MAM. The detailed structure of S-MAM is depicted in Fig. 2.

First, we split the input token sequence into multiple targets token
sequences, including the static one and online ones if we use multiple online
templates, and a search token sequence. Then the split features of targets and
search are processed by a layer normalization and simple linear projections.
Next, the mixed attention operation is performed on the generated queries,
keys and values. The remaining operations remain the same as W-MAM.

Figure 3: MixFormer Tracking Pipeline. Based on the above MAM, our MixFormer
presents a compact end-to-end framework for tracking with unified process of
feature extraction and target information integration. Specifically, we
instantiate two types MixFormer trackers. The first one is a hierarchical
tracker with progressive downsampling based on the CvT backbone (termed as
MixCvT). The second one is a non-hierarchical tracker with plain backbone of
ViT (termed as MixViT). For these two types of trackers, we design two
customized localization heads for bounding box estimation.

Asymmetric Mixed Attention Scheme. Intuitively, the cross attention from the
targets query to search area is not so important and might bring negative
influence due to potential distractors. To reduce computational cost of MAM
and thereby allowing for efficiently using multiple templates to deal with
object deformation, we further present a customized asymmetric mixed attention
scheme by pruning the unnecessary target-to-search area cross-attention. This
asymmetric mixed attention is defined as follows:

$\displaystyle{\rm Attention_{t}}={\rm
Softmax}(\frac{q_{t}k_{t}^{T}}{\sqrt{d}})v_{t},$ (2) $\displaystyle{\rm
Attention}_{s}={\rm Softmax}(\frac{q_{s}k_{m}^{T}}{\sqrt{d}})v_{m}.$

In this manner, the template tokens in each MAM could remain unchanged during
tracking process and so it needs to be processed only once.

Discussions. To better expound the insight of the mixed attention, we make a
comparison with the attention mechanism used by other transformer trackers.
Different with our mixed attention, TransT [11] uses ego-context augment and
cross-feature augment modules to perform self attention and cross attention
progressively _in two steps_. Compared to the transformer encoder of STARK
[80], our MAM shares a similar attention mechanism but with three notable
differences. First, in W-MAM, we incorporate the spatial structure information
with a depth-wise convolution while they use positional encoding. More
importantly, our MAM is built as a multi-stage backbone for both joint feature
extraction and information integration, while they depend on a separate CNN
backbone for feature extraction and only focus on information integration in
another single stage. Finally, we also propose a different asymmetric MAM to
further improve the tracking efficiency without much accuracy drop.

###  3.2 Hierarchical MixFormer: MixCvT

Overall Architecture. Based on the W-MAM blocks, we build the hierarchical
MixFormer tracker of MixCvT. The main idea of MixCvT is to progressively
extract coupled features for target template and search area, and deeply
perform the information integration between them. Basically, it comprises two
components: a backbone composed of iterative W-MAMs, and a simple localization
head to produce the target bounding box. Compared with other prevailing
trackers by decoupling the steps of feature extraction and information
integration, it leads to a more compact and neat tracking pipeline only with a
single backbone and tracking head, without an explicit integration module or
any post-processing. The overall architecture is depicted in Fig. 3.

W-MAM Based Backbone. Our goal is to couple both the generic feature
extraction and target information integration within a unified transformer-
based architecture. The W-MAM based backbone employs a progressive multi-stage
architecture design. Each stage is composed of $N$ MAM layers operating on the
same-scaled feature maps with the identical channel number. All stages share
the similar architecture, which consists of an overlapped patch embedding
layer and $N_{i}$ wide mixed attention modules.

Specifically, given $T$ templates (i.e., the first template and $T-1$ online
templates) with the size of $T\times H_{t}\times{W_{t}}\times 3$ and a search
region (a cropped region according to the previous target states) with the
size of $H_{s}\times{W_{s}}\times 3$, we first map them into overlapped patch
embeddings using a _convolutional Token Embedding_ layer with stride $4$ and
kernel size $7$. The convolutional token embedding layer is introduced in each
stage to grow the channel resolution while reducing the spatial resolution.
Then we flatten the patch embeddings and concatenate them, yielding a fused
token sequence with the size of
$(T\times\frac{H_{t}}{4}\times\frac{W_{t}}{4}+\frac{H_{s}}{4}\times\frac{W_{s}}{4})\times
C$, where $C$ equals to 64 or 192, $H_{t}$ and $W_{t}$ is 128, $H_{s}$ and
$W_{s}$ is 320 in this work. After that, the concatenated tokens pass through
$N_{i}$ target-search MAM to perform both feature extraction and target
information incorporation. Finally, we obtain the token sequence of size
$(T\times\frac{H_{t}}{16}\times\frac{W_{t}}{16}+\frac{H_{s}}{16}\times\frac{W_{s}}{16})\times
6C$. More details about the MAM backbones could be found in the Section 4.1
and Table I. Before passed to the prediction head, the search tokens are split
and reshaped to the size of $\frac{H_{s}}{16}\times\frac{W_{s}}{16}\times 6C$.
Particularly, we do not apply the multi-scale feature aggregation strategy,
commonly used in other trackers (e.g., SiamRPN++ [39]).

Corner Based Localization Head. Inspired by the corner detection head in STARK
[80], we employ a fully-convolutional corner based localization head to
directly estimate the bounding box of tracked object, solely with several $\rm
Conv$-$\rm BN$-$\rm ReLU$ layers for the top-left and the bottom-right corners
prediction respectively. Finally, we can obtain the bounding box by computing
the expectation over corner probability distribution [42]. The difference with
STARK lies in that ours is a fully convolutional head while STARK highly
relies on both encoder and the decoder with more complicated design.

Query Based Localization Head. Inspired by DETR [8], we propose to employ a
simple query based localization head. This sparse localization head can verify
the generalization ability of our MAM backbone and yield a pure transformer-
based tracking framework. Specifically, we add an extra learnable _regression
token_ to the sequence of the final stage and use this token as an anchor to
aggregate information from entire target and search area. Finally, a FFN of
three fully connected layers is employed to directly regress the bounding box
coordinates. This framework does not use any post-processing technique either.

Figure 4: Different Heads employed in MixCvT and MixViT. Two types of head are
developed to validate the generalization of MixFormer. We also explore
building a pyramidal corner head for the non-hierarchical MixViT to supplement
the multi-scale information.

###  3.3 Plain MixFormer: MixViT

Overall Architecture. To remove the designs of the hierarchical structure and
the convolutional projections, we build a simple and non-hierarchical tracker,
termed as MixViT. The architecture of MixViT is depicted in Fig. 3 (b). MixViT
is composed of a patch embedding layer, several Slimming Mixed Attention
Module (S-MAM) on the token sequence and a new pyramidal corner head on top.
In this section, we describe the Slimming Mixed Attention Module, S-MAM based
backbone, different positional embeddings for MixFormer-ViT and the proposed
pyramidal corner head.

S-MAM Based Backbone. Compared to MixCvT tracker, the following adaptions are
made in MixViT tracker. First, the backbone of MixViT is non-hierarchical and
one-stage, composed of a single non-overlapped patch embedding layer and
multiple slimming mixed attention modules. This avoids a tremendous number of
tokens in the high-resolution stages as in MixCvT, thus guaranteeing a high
tracking efficiency. Second, we explore different types of positional
embeddings for targets and search tokens to explicitly encode the absolute
position information. Third, since MixViT backbone has the same trainable-
weights with vanilla Vision Transformer [21], it can be compatible with the
recent ViT development, such as MAE pretraining,.

First, given $T$ templates with the size of $T\times H_{t}\times W_{t}\times
3$ and the search region with the size of $H_{s}\times W_{s}\times 3$, we map
them into non-overlapped patch embeddings using a convolutional token
embedding layer with kernel size of 16 and stride of 16. Then we add two
different positional embeddings to search region and templates patch
embeddings respectively. Three different types of positional embeddings are
experimented as in Section 3.3. Next, we concatenate the templates (with size
of $T\times\frac{H_{t}}{16}\times\frac{W_{t}}{16}\times C$) and search region
tokens (with size of $\frac{H_{s}}{16}\times\frac{W_{s}}{16}\times C$) and
input it to multiple Slimming Mixed Attention Modules. Finally, the split
search region token is prepared to input the localization head.

Positional Embedding. We investigate three types of positional embeddings in
this work to give a thorough inspection to the targets and search region
position modeling. For all of the three types, we use two positional
embeddings of different length for targets and search region. The differences
between the three types of positional embeddings lie in the initialization
methods and whether to be learnable. For the first type, we adopt 2D bilinear
interpolation to interpolate the pretrained positional embeddings from MAE-
pretrained ViT models. And the embeddings are set to be frozen during training
process. For the second one, the same initialization method is employed as the
first type. However, the embeddings are set to be learnable so as to realize
dynamic adjustment. For the last one, we use frozen sin-cos positional
embeddings as adopted in vanilla ViT [21].

Pyramidal Corner Head. Since MixViT adopts a non-hierarchical backbone and may
lacks the multi-scale information, we build a pyramidal corner heads from only
the last feature map of MixViT backbone. The structure of pyramidal head is
depcited in Fig. 4 (c). Specifically, we first build the feature pyramid via
several convoluational layers and interpolation layers, thereby producing
three different-resolution feature maps. Then, the multi-scale feature maps
are fused by convolutional layers to get robust representations. We find that,
compared with plain corner head employed in MixCvT, the pyramidal one can
obtain better performance as shown in experiments.

###  3.4 Pre-training of MixFormer

In this work, we find that the pre-training of backbone plays an important
role in building an effective MixFormer tracker. Thanks to the flexibility of
attention operation, the pre-trained CvT or ViT models on ImageNet for image
classification could be directly adapted to initialize the weights of backbone
of our MixFormer trackers. Specifically, we investigate different supervised
pre-trained models for MixCvT, and both supervised and self-supervised pre-
trained models for MixViT. Inspired by the excellent performance of masked
pre-training [30], we further design a masked auto-encoder on the commonly
used tracking datasets (i.e., LaSOT [23], TrackingNet [58], GOT-10k [33] and
COCO [47]), and apply it as the pretraining weights for MixViT.

Supervised Pre-training on ImageNet. For MixCvT, the model weights of CvT
trained on ImageNet-1k or ImageNet-22k are used as the initialization of
MixCvT backbone. Pre-training on ImageNet-22k provides stronger visual
representations than ImageNet-1k. For MixViT, we carry out the supervised pre-
training via the recent DeiT3 [64] model, which successfully trains vanilla
ViT models with high performance on the ImageNet-1k and ImageNet-21k. It is
noteworthy that we add LayerScale in transformer blocks of MixViT when
pretraining it with DeiT3 to keep consistency.

Self-supervised Pre-training on ImageNet. Inspired by the great success of
masked autoencoder (MAE) pre-training [30] and its application in object
detection [43], we also investigate the MAE pre-training in our MixViT. To
better investigate the difference in the learned representations from
supervised and self-supervised pre-training, we perform detailed studies on
the effect of pre-training by varying the ViT depth. We observe that the
different behaviors of supervised pre-training and MAE pre-training, where
supervised pre-trained models tend to learn high-level semantic
representations in the latter stages, while the MAE pre-trained models seem to
focus on learning the low-level signal structure progressively along all
layers. Therefore, reducing several layers in the supervised pre-trained model
leads to very small performance drop as the high-level semantic features in
the latter stages contribute little to object tracking. In addition, we
further increase the depth of the MAE pre-trained ViT by borrowing transformer
blocks from the MAE decoder, and obtain the further performance improvement.

To combine the advantages of masked pre-training and visual inductive prior of
progressive modeling, we further explore the ConvMAE [26] pretraining in our
MixViT tracker. The architecture of ConvMAE pre-trained model is similar to
the plain ViT, except that it replaces the original one-layer strided patch
embedding with a lightweight CNN for progressive patch embedding. So, we can
easily adapt the ConvMAE pre-trained model to our MixViT by simply changing
the patch embedding layer to the corresponding CNN. The modified MixViT with
ConvMAE pre-training consists of a three-stage patch embedding CNN, the plain
ViT encoder of S-MAM, and a pyramidal corner head. The patch embedding CNN
uses simple convolution blocks to separately transform the search and
templates to token embeddings. The details on ViT backbone design is similar
to the original MAE pretraining as shown in Table II.

Simple MAE Pre-training on Tracking Datasets. Inspired by the excellent
performance MAE pre-training [30] and its data-efficiency [63] on the training
samples, we are wondering whether we are able to pre-train our MixVit with MAE
solely on the common tracking datasets without using the large-scale ImageNet
dataset. First, we design a simple MAE baseline by directly pre-training a
masked auto-encoder on the tracking datasets and use this pre-trained MAE to
initialize our MixViT. In details, we first train a masked auto-encoder on
tracking datasets (including LaSOT, TrackingNet, GOT-10k and coco) and then
fine-tune MixViT on the same tracking datasets. The main procedure of training
masked auto-encoder keeps consistent with MAE [30] excepting for the datasets
and data augmentations. Surprisingly, we observe that MixViT with self-
supervised pretraining solely on the tracking datasets outperforms that with
training from scratch by a large margin.

Figure 5: Framework of TrackMAE. We perform random masking on the search image
with a mask ratio of 75% while keeping the target template unmasked. The
decoder reconstructs the invisible patches in the search image.

TrackMAE Pre-training on Tracking Datasets. To better adapt the MAE pre-
training to our MixViT tracker, we present a new self-supervised pre-training
method for visual object tracking, termed as TrackMAE, as shown in Fig. 5.
TrackMAE inherits the simple pipeline of random token dropping and
reconstructing the masked ones. However, to make it more suitable for the
tracking framework and narrow the gap between tracker pre-training and fine-
tuning, the search images are masked with a mask ratio of 75% while the
templates are fully visible and are placed for information integration. The
decoder reconstructs the invisible parts of search images. Especially, we only
use the tracking datasets for TrackMAE pre-training.

###  3.5 Training and Inference

Training. The training process of our MixFormer generally follows the standard
training recipe of current trackers [80, 12]. We first pre-train our MixCvT
and MixViT backbones with CvT model [73] and encoders of MAE [30]
respectively, and then fine-tune the whole tracking framework on the tracking
datasets. For MixCvT training, a combination of ${L_{1}}$ loss and GIoU loss
[60] is employed as follows:

$L_{loc}=\lambda_{L1}L_{1}(B_{i},\hat{B_{i}})+\lambda_{giou}L_{giou}(B_{i},\hat{B_{i}}),$
(3)

where $\lambda_{L1}=5$ and $\lambda_{giou}=2$ are the weights of the two
losses, $B_{i}$ is the ground-truth bounding box and $\hat{B_{i}}$ is the
predicted bounding box of the targets. For MixViT training, we replace the
GIoU loss with the CIoU loss [89].

Figure 6: Structure of the Score Prediction Module (SPM).

Template Online Update. Online templates play an important role in capturing
temporal information and dealing with object deformation and appearance
variations. However, poor-quality online templates may lead to tracking
performance degradation. As a consequence, we introduce a Score Prediction
Module (SPM), described in Fig. 6, to select reliable online templates
determined by the predicted confidence score. The SPM is composed of two
attention blocks and a three-layer perceptron. First, a learnable _score
token_ serves as a query to attend the search ROI tokens. It enables the score
token to encode the mined target information. Next, the score token attends to
all positions of the initial target token to implicitly compare the mined
target with the first target. Finally, the score is produced by the MLP layer
and a sigmoid activation. The online template is treated as negative when its
predicted score is below than 0.5. For the SPM training, it is performed after
the backbone training and we use a standard cross-entropy loss:

$\begin{split}L_{score}=y_{i}{\rm log}(p_{i})+(1-y_{i}){\rm
log}(1-p_{i}),\end{split}$ (4)

where $y_{i}$ is the ground-truth label and $p_{i}$ is the predicted
confidence score.

Discussions on SPM. We compare our SPM with the score prediction branch in
STARK [80]. In contrast to STARK, i) we explicitly input the tracked object
RoI feature (cropped based on MixFormer’s tracking results) of the current
frame to the SPM, while STARK implicitly extract the target information
through an encoder-decoder structure; ii) Unlike STARK, there is no decoder in
MixFormer backbone, so that we introduce an extra score token and two
attention modules. iii) Our SPM is composed of two progressive attention
blocks to take comparison between tracked object and the target, and a MLP
head to produce the online template confidence score. In contrast, STARK only
employs a three-layer perceptron head to output the template confidence score.

Inference. During inference, multiple templates, including one static template
and $N$ dynamic online templates, together with the cropped search region are
fed into MixFormer trackers to produce the target bounding box and the
confidence score. We update the online templates only when the update interval
is reached and select the sample with the highest confidence score.
Especially, the predicted bounding box is served as the final output without
any post-processing such as cosine window or bounding box smoothing.

TABLE I: W-MAM based backbone architectures for MixCvT and MixCvT-L. The input
is a tuple of templates with shape of $128\times 128\times 3$ and search
region with shape of $320\times 320\times 3$. $S$ and $T$ represent for the
search region and template. $H_{i}$ and $D_{i}$ is the head number and
embedding feature dimension in the $i$-th stage. $R_{i}$ is the feature
dimension expansion ratio in the MLP layer.

| Output Size | Layer Name | MixCvT | MixCvT-L
---|---|---|---|---
Stage1 | $\begin{array}[]{c}S:80\times 80,\\\ T:32\times 32\end{array}$ | Conv. Embed. |  $7\times 7$, $64$, stride $4$ |  $7\times 7$, $192$, stride $4$
$\begin{array}[]{c}S:80\times 80,\\\ T:32\times 32\end{array}$ |  | W-MAM
---
$\left[\begin{array}[]{c}H_{1}=1\\\ D_{1}=64\\\ R_{1}=4\end{array}\right]\times 1$ | $\begin{bmatrix}\begin{array}[]{c}H_{1}=3\\\ D_{1}=192\\\ R_{1}=4\end{array}\end{bmatrix}\times 2$
Stage2 | $\begin{array}[]{c}S:40\times 40,\\\ T:16\times 16\end{array}$ | Conv. Embed. |  $3\times 3$, $192$, stride $2$ |  $3\times 3$, $768$, stride $2$
$\begin{array}[]{c}S:40\times 40,\\\ T:16\times 16\end{array}$ |  | W-MAM
---
$\begin{bmatrix}\begin{array}[]{c}H_{2}=3\\\ D_{2}=192\\\ R_{2}=4\end{array}\end{bmatrix}\times 4$ | $\begin{bmatrix}\begin{array}[]{c}H_{2}=12,\\\ D_{2}=768,\\\ R_{2}=4\end{array}\end{bmatrix}\times 2$
Stage3 | $\begin{array}[]{c}S:20\times 20,\\\ T:8\times 8\end{array}$ | Conv. Embed. |  $3\times 3$, $384$, stride $2$ |  $3\times 3$, $1024$, stride $2$
$\begin{array}[]{c}S:20\times 20,\\\ T:8\times 8\end{array}$ |  | W-MAM
---
$\begin{bmatrix}\begin{array}[]{c}H_{3}=6\\\ D_{3}=384\\\ R_{3}=4\end{array}\end{bmatrix}\times 16$ | $\begin{bmatrix}\begin{array}[]{c}H_{3}=16\\\ D_{3}=1024\\\ R_{3}=4\end{array}\end{bmatrix}\times 12$
MACs |  $35.61$ M |  $183.89$ M
FLOPs |  $23.04$ G |  $127.81$ G
Speed (RTX 8000) |  $31$ FPS |  $27$ FPS

TABLE II: S-MAM based backbone architectures for MixViT and MixViT-L. $S$ and
$T$ represent for the search region and the template respectively. ‘$*$’
denotes for the hidden dim in the third stage of MixViT with ConvMAE pre-
training.

Model | Layers | Hidden | Input Size | Params | Speed
---|---|---|---|---|---
Dims | ($S$ / $T$) | (RTX 8000)
MixViT | 12 | 768 | ($288\times 288$) / ($128\times 128$) | 97M | 75 FPS
MixViT-L | 24 | 1024 | ($384\times 384$) / ($192\times 192$) | 318M | 13 FPS
MixViT (ConvMAE) | 15 | 768∗ | ($288\times 288$) / ($128\times 128$) | 96 M | 45 FPS
MixViT-L (ConvMAE) | 24 | 1024∗ | ($384\times 384$) / ($192\times 192$) | 286 M | 10 FPS

##  4 Experiments

###  4.1 Implementation Details

Our trackers are implemented using Python 3.6 and PyTorch 1.7.1. The MixFormer
training is conducted on 8 Tesla V100 GPUs. Especially, MixFormer is a neat
tracker without some post-processing strategies. The inference of MixCvT and
MixViT are performed on a Quadro RTX-8000 GPU.

Architectures. As shown in Table I, we instantiate two types of MixCvT
trackers, denoted by MixCvT and MixCvT-L, with different parameters and FLOPs
by varying the number of W-MAM blocks and the hidden feature dimension in each
stage. The backbones of MixCvT and MixCvT-L are initialized with the CvT-21
and CvT24-W [73] (first 16 layers are employed) pre-trained on ImageNet [20]
respectively. In addition, as shown in Table II, we instantiate two models of
MixViT, i.e., MixViT and MixViT-L, by varying the number of S-MAM blocks and
the hidden feature dimension. The backbone weights of MixViT and MixViT-L are
initialized with corresponding MAE encoders. Besides, we try ConvMAE pre-
training that has comparable parameters and Flops to the plain ViT for the
initialization of MixViT.

Training. The training set includes TrackingNet [58], LaSOT [23], GOT-10k [33]
and COCO [47] training dataset, which is the same as DiMP [4] and STARK [80].
While for GOT-10k test, we re-train our trackers by only using the GOT-10k
train split following its standard protocol. The whole training process of
MixFormer consists of two stages, which contains the first 500 epochs for the
tuning of backbones and heads, and extra 40 epochs for score prediction head
tuning. For MixCvT training, we use ADAM [35] with weight decay of $1\times
10^{-4}$. The learning rate is initialized as $1\times 10^{-4}$ and decreased
to $1\times 10^{-5}$ at the epoch of 400. The sizes of search images and
templates are $320\times 320$ pixels and $128\times 128$ pixels respectively.
For data augmentations, we use the horizontal flip and brightness jittering.
For MixViT training, we use ADAMW with weight decay of $1\times 10^{-4}$ and
learning rate of weight decay of $4\times 10^{-4}$ and decreased to $4\times
10^{-5}$ at the epoch of 400. In MixViT setting, the sizes of search images
and templates are $288\times 288$ pixels and $128\times 128$ pixels
respectively. In the MixViT-L setting, to improve the representation power of
trackers, the sizes of search images and templates are set to $384\times 384$
pixels and $192\times 192$ pixels, respectively.

Inference. We use the first template and multiple online templates together
with the current search region as input of MixFormer. The dynamic templates
are updated when the update interval of 200 is reached by default (the update
interval varies slightly with different test datasets). The templates with the
highest predicted score in the interval is selected to replace the previous
one.

###  4.2 Exploration Studies

To verify the effectiveness and give a thorough analysis on our proposed
MixFormer trackers (i.e. MixViT-L, MixViT, MixCvT-L and MixCvT) and pre-
training methods, we perform detailed ablation studies on the large-scale
LaSOT dataset. AUC is employed as the evaluation metric.

####  4.2.1 Study on Mixed Attention Module

TABLE III: Analysis of the MAM based framework. ‘-’ denotes the component is not used. SAM represents for self attention module, CAM for cross attention module and W-MAM for the proposed mixed attention module. The numbers in brackets represent the number of the blocks. Performance is evaluated on LaSOT. # | Backbone | Integration | Head | Params. | FLOPs | AUC
---|---|---|---|---|---|---
1 | SAM(21) | CAM(1) | Corner | 37.35M | 20.69G | 59.8
2 | SAM(21) | CAM(3) | Corner | 40.92M | 22.20G | 60.5
3 | SAM(21) | ECA+CFA(4) | Corner | 49.75M | 27.81G | 66.9
4 | SAM(20)+W-MAM(1) | - | Corner | 35.57M | 19.97G | 65.8
5 | SAM(15)+W-MAM(6) | - | Corner | 35.67M | 20.02G | 66.2
6 | SAM(10)+W-MAM(11) | - | Corner | 35.77M | 20.07G | 67.4
7 | SAM(5)+W-MAM(16) | - | Corner | 35.87M | 20.12G | 68.1
8 | W-MAM(21) | - | Corner | 35.97M | 20.85G | 68.4
9 | W-MAM(21) | - | Query | 31.46M | 19.13G | 66.0

Simultaneous process vs. Separate process. As the core part of our MixFormer
is to unify the procedure of feature extraction and target information
integration in a single module, we compare it to the separate processing
architecture (e.g. TransT [11]). The comparison results are shown in Table III
#1, #2, #3 and #8. Experiments of #1 and #2 are end-to-end trackers comprising
a self-attention based backbone, ${n}$ cross attention modules to perform
information integration and a corner head. #3 is the tracker with CvT as
backbone and TransT’s ECA+CFA(4) as interaction. Experiment of #8 is our
MixCvT without multiple online templates and asymmetric mechanism, denoted by
MixCvT-Base. MixCvT-Base largely increases the model of #1 (using one CAM) and
#2 (using three CAMs) by 8.6% and 7.9% with smaller parameters and FLOPs. This
demonstrates the effectiveness of unified feature extraction and information
integration, as both of them would benefit each other.

TABLE IV: Ablation for asymmetric mixed attention mechanism. All models are deployed without online templates update mechanism and pretrained on ImageNet-22k. Performance is evaluated on LaSOT. Tracker | Asymmetric. | FPS (RTX 8000) | AUC
---|---|---|---
MixCvT | - | 25 | 68.4
MixCvT | ✓ | 31 | 68.1

Study on stages of MAM. To further verify the effectiveness of the MAMs, we
conduct experiments as in Table III #4, #5, #6, #7 and #8, to investigate the
performance of different numbers of W-MAM in our MixCvT. We compare our W-MAM
with the self-attention operations (SAM) with out cross-branch information
communication. We find that more W-MAMs contribute to higher AUC score. It
indicates that extensive target-aware feature extraction and hierarchical
information integration play a critical role to construct an effective
tracker, which is realized by the iterative MAM. Especially, when the number
of W-MAM reaches 16, the performance reaches 68.1, which is comparable to the
MixFormer-Base containing 21 W-MAMs.

Study on asymmetric MAM. The asymmetric MAM is used to reduce computational
cost and allows for usage of multiple templates during online tracking. As
shown in Table IV, the asymmetric MixCvT-Base tracker increases the running
speed of 24% while achieving a comparable performance, which demonstrates
asymmetric MAM is important for building an efficient tracker.

####  4.2.2 Study on MixCvT & MixViT Architectures

To compare the hierarchical tracker MixCvT and non-hierarchical tracker MixViT
fairly, we perform ablation studies without any pre-training (i.e., training
from scratch) and with the same localization head. We aim to explore whether
the MixCvT backbone exhibits superiority of the progressive downsampling and
locality modeling over MixViT backbone and we gradually evolve MixViT into
MixCvT as in Table V. For fair comparison, all models in Table V are trained
from scratch, employ plain corner head, and are deployed without online
scheme. Notably, the Batch Normalization in original MixCvT are substituted to
Layer Normalization, otherwise the performance is extremely poor. The MixViT
model as in the first line of Table V obtains AUC of 56.2 without multi-stage
strategies and depth-wise convolutional projections. When extending it to a
multi-stage one with three convolutional patch embeddings as MixCvT, the AUC
score increases by 2.6%. Further, when adding depth-wise convolutional
projections, the AUC score slightly increases by 0.2%. We can conclude that,
MixCvT obtains superior performance to MixViT under the setting of training
from scratch only on the tracking datasets, and the multi-stage strategy plays
a more important role than depth-wise convolutional projections. It implies
that incorporating visual inductive bias into backbone design has the benefits
to relieve the risk of over-fitting and improve the generalization ability
when without extra pre-training techniques.

TABLE V: Study on MixCvT and MixViT backbone architectures, including the analysis of multi-stage downsampling scheme and depth-wise convolutional projections. For fair comparison, all models are trained from scratch, employ plain corner head and are deployed without online scheme. Tracker | Backbone | Multi-stage | DW-Conv Proj. | AUC
---|---|---|---|---
MixViT | S-MAM(12) | - | - | 56.2
- | S-MAM(12) | ✓ | - | 58.8
MixCvT | W-MAM(12) | ✓ | ✓ | 59.0
TABLE VI: Study on different heads and different positional embeddings employed in MixViT. All models are pretrained with MAE encoders and are deployed without online scheme. Tracker | Head | Positional Emb. | AUC
---|---|---|---
MixViT | Plain Corner Head | V1 | 68.4
MixViT | Plain Corner Head | V2 | 68.4
MixViT | Plain Corner Head | V3 | 68.5
MixViT | Plain Corner Head | V3 | 68.5
MixViT | Pyramidal Corner Head | V3 | 69.0

####  4.2.3 Study on Heads and Positional Embeddings

MixCvT’s localization heads. To verify the generalization ability of our MAM
backbone, we evaluate the MixCvT-Base with two types of localization head as
described in Section 3.2 (fully convolutional head vs. query based head). The
results are shown as in Table III, where #8 and #9 are for the corner head and
the query-base head respectively. MixCvT-Base with the fully convolutional
corner head outperforms that of the query-based head. In particular, MixCvT-
Base with corner head surpass the state-of-the-art TransT trackers even
without any post-processing and online templates. Besides, MixCvT-Base with
the query head, a pure transformer-based tracking framework, obtains a much
better performance than the query head of STARK-ST (66.0 vs 63.7). These
results demonstrate the generalization ability of our MAM backbone to
different localization heads.

MixViT’s localization heads. To verify the effectiveness of our proposed
pyramidal corner head on the MixViT tracker, we evaluate the performance of
MixViT with plain corner head and pyramidal corner head as shown in Table VI.
All trackers in Table VI are pre-trained with MAE encoders and deployed
without the online template updating. When substituting plain corner head to
pyramid corner head, the AUC score increases by 0.5. It indicates the
effectiveness of the proposed pyramidal corner head, which can supplement the
multi-scale information to the non-hierarchical MixViT backbone for better
object localization.

MixViT’s positional embeddings. We evaluate the three types of positional
embeddings of MixViT as referred in Section 3.3. The results are shown in
Table VI. All trackers are pre-trained with MAE encoders and are deployed
without online scheme. We can see that the MixViT models with the three
different positional embeddings achieve comparable performance, which prove
that it’s no matter what initialization methods of positional embeddings are
emplyed and whether to set it to be learnable.

TABLE VII: Study on pretraining methods. ‘Tracking Data’ denotes for using the whole tracking datasets including GOT-10k, LaSOT, TrackingNet and COCO. ‘Scratch’ means the model is trained from scratch. ‘MAE’ with ‘Tracking data.’ pre-training represents that we pretrain the MixViT backbone with MAE performed on tracking datasets including LaSOT, TrackingNet, GOT-10k and coco. All models employ plain corner head, asymmetric mechanism and are deployed without online scheme. Trackers | Pre-training Methods | Pre-training Data | AUC
---|---|---|---
MixCvT | From Scratch | - | 59.0
MixCvT | Supervised (CvT) | ImageNet-1k | 67.2
MixCvT | Supervised (CvT) | ImageNet-22k | 68.1
MixViT | From Scratch | - | 56.2
MixViT | MAE | Tracking Data. | 65.1
MixViT | TrackMAE | Tracking Data. | 66.0
MixViT | Supervised (DeiT3) | ImageNet-1k | 66.5
MixViT | Supervised (DeiT3) | ImageNet-21k | 67.7
MixViT | MAE Fine-tune | ImageNet-1k | 68.0
MixViT | MAE Pre-train | ImageNet-1k | 68.5
Figure 7: The performance of AUC on LaSOT when we use MAE pretraining and
DeiT3 pretraining on MixViT backbones with different number of layers. All
models use plain corner head and are deployed without online scheme.

####  4.2.4 Study on Pre-training Methods

Study on different pre-training methods. Pre-training of MAM based backbone
plays an important role in MixFormer tracking. We investigate some different
pre-training methods of MixCvT and MixViT as shown in Table VII. Both MixCvT
and MixViT with supervised pre-training or self-supervised pre-training
outperform training from scratch by more than 9%, which indicates the
importance of pre-training for MixFormer trackers. MixCvT and MixViT with
ImageNet-21k pre-training obtain higher AUC than that with ImageNet-1k pre-
training, which demonstrates the importance of semantic features trained with
more supervision. MixViT with self-supervised pre-training of MAE outperforms
that with the supervised pre-training of DeiT3. The MixViT with tracking-data
only MAE (trained with the tracking datasets instead of ImageNet) pre-training
acquire comparable performance with that of supervised pre-training and
largely surpass that trained from scratch. Meanwhile, our proposed TrackMAE
surpasses the tracking-data only MAE by 0.9. This demonstrates that it is
meaningful to design a customized self-supervised pre-training pipeline for
tracking framework, so as to narrow the gap between pre-training and fine-
tuning.

TABLE VIII: Study on online templates update mechanism and score prediction module. All models are pretrained on ImageNet-22k. Performance is evaluated on LaSOT. Tracker | Online | Score of SPM | AUC
---|---|---|---
MixCvT | - | - | 68.1
MixCvT | ✓ | - | 66.6
MixCvT | ✓ | ✓ | 69.2
TABLE IX: State-of-the-art comparison on VOT2020 [36]. The best two results
are shown in red and blue fonts. Our trackers use Alpha-Refine [81] to predict
masks. MixFormer-1k is pretrained with ImageNet-1k.

| SiamMask | D3S | SuperDiMP | AlphaRef | OceanPlus | RPT | DualTFR | STARK | ToMP | MixCvT-1k | MixCvT-22k | MixCvT-L | MixViT | MixViT-L | MixViT | MixViT-L
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---
|  [71] |  [50] |  [4] |  [81] |  [86] |  [54] |  [76] |  [80] |  [55] |  |  |  |  |  | (ConvMAE) | (ConvMAE)
EAO | 0.321 | 0.439 | 0.305 | 0.482 | 0.491 | 0.530 | 0.528 | 0.505 | 0.497 | 0.527 | 0.535 | 0.555 | 0.518 | 0.584 | 0.528 | 0.567
Accuracy | 0.624 | 0.699 | 0.492 | 0.754 | 0.685 | 0.700 | 0.755 | 0.759 | 0.750 | 0.746 | 0.761 | 0.762 | 0.728 | 0.755 | 0.734 | 0.747
Robustness | 0.648 | 0.769 | 0.745 | 0.777 | 0.842 | 0.869 | 0.836 | 0.817 | 0.798 | 0.833 | 0.854 | 0.855 | 0.841 | 0.890 | 0.857 | 0.870

Figure 8: Visualization results of different MixCvT’s attention weights on
car-2 of LaSOT. S-to-t is search-to-template cross attention, S-to-OT is
search-to-online-template cross attention, S-to-S is self attention of search
region and OT-to-T is online-template-to-template cross attention. S$i$-B$j$
represents for Stage-$i$ and Block-$j$ of MixFormer. Best viewed with zooming
in. Figure 9: Visualization results of different MixViT’s attention weights on
car-2 of LaSOT. S-to-t is search-to-template cross attention, S-to-OT is
search-to-online-template cross attention, S-to-S is self attention of search
region and OT-to-T is online-template-to-template cross attention. S$i$-B$j$
represents for Stage-$i$ and Block-$j$ of MixFormer. Best viewed with zooming
in.

Study on different number of encoder and decoder layers. To take a closer look
at the difference between Deit3 pre-training and MAE pre-training, we
investigate MixViT with different number of transformer layers using the two
pre-training methods as shown in Fig. 7. “+D$N$” in X-axis of Fig. 7
represents for adding extra $N$ MAE decoders (with mixed attention). For the
supervised pre-training of DeiT3, MixViT of 8 layers reaches comparable
performance with that of 12 layers. It indicates the supervised pre-training
might learn high-level semantic features that have less relation with the
object tracking. However, for the self-supervised pre-training of MAE
encoders, we see a different curve and the MixViT of 12 layers increases that
of 8 layers by 2.6%. Furthermore, we add extra MAE decoders and see a small
performance improvement. We suppose that the last few layers provide very
little help, since the representations are gradually adjusted to adapt to the
pixel-level reconstruction task. Finally, we surprisingly find that MixViT of
4 layers with DeiT3 pre-training still obtains a high AUC of 63.4% with an
extremely high running speed of 300 FPS.

####  4.2.5 Study on online scheme and SPM

We evaluate the effectiveness of the proposed Score Prediction Module and the
online scheme. As demonstrated in Table VIII, MixCvT with online templates,
sampled by a fixed update interval, performs worse than that with only the
first template, and the online MixCvT with our score prediction module
achieves the best AUC score. Specifically, the online MixCvT with our SPM
improves the original MixCvT by 0.9% of AUC on LaSOT. It suggests that
selecting reliable templates with our score prediction module is of vital
importance, since low-quality online samples can bring noises to the
templates.

###  4.3 Visualization Results

To explore how the mixed attention works in our MixFormer trackers, we
visualize the attention maps of MixCvT in Fig. 8 and the attention maps of
MixViT in Fig. 9. From the four types of attention maps, we derive that: (i)
distractors in background get suppressed layer by layer, demonstrating the
effectiveness of mixed attention on enhancing the tracker’s discriminative
ability; (ii) online templates are supplements to the static template, which
may be more adaptive to appearance variation and help to discriminate the
target from the background; (iii) the foreground of multiple templates can be
augmented by mutual cross attention (from the fourth line of ‘OT-to-T Cross’
visualization), (iv) a certain position tends to interact with the surrounding
local patch (from the ‘S-to-S Self.’ visualization). (v) compared with the
visualization results of MixCvT, MixViT yields a cleaner attention maps,
especially in the last blocks. We think this is possibly due to the different
pre-training methods.

TABLE X: State-of-the-art comparison on TrackingNet [58], LaSOT [23], GOT-10k
[33] and UAV123 [57]. The best two results are shown in red and blue fonts.
The underline results of GOT-10k are not considered in the comparison, since
the models are trained with datasets other than GOT-10k. MixCvT-1k is the
model pre-trained with ImageNet-1k. Other MixCvT models are pre-trained on
ImageNet-22k and all MixViT models are pre-trained with corresponding MAE-
pretraining encoders. ‘*’ denotes for trackers trained only with GOT-10k train
split.

Method | LaSOT | TrackingNet | GOT-10k | UAV123
---|---|---|---|---
AUC(%) |  $P_{Norm}$(%) | P(%) | AUC(%) | $P_{Norm}(\%)$ | P(%) | AO(%) |  $SR_{0.5}$(%) |  $SR_{0.75}$(%) | AUC(%) | P(%)
MixViT-L (ConvMAE) | 73.3 | 82.8 | 80.3 | 86.1 | 90.3 | 86.0 | 75.4 | 84.0 | 75.4 | 70.0 | 92.5
MixViT (ConvMAE) | 70.4 | 80.4 | 76.7 | 84.5 | 89.1 | 83.7 | 74.3 | 84.1 | 73.0 | 69.0 | 91.2
MixViT-L | 72.4 | 82.2 | 80.1 | 85.4 | 90.2 | 85.7 | 78.0 | 87.3 | 78.7 | 68.7 | 90.8
MixViT | 69.6 | 79.9 | 75.9 | 83.5 | 88.3 | 82.0 | 72.7 | 82.3 | 70.8 | 68.1 | 90.3
MixViT-L* | - | - | - | - | - | - | 75.7 | 85.3 | 75.1 | - | -
MixViT* | - | - | - | - | - | - | 72.5 | 82.4 | 69.9 | - | -
MixCvT-L | 70.1 | 79.9 | 76.3 | 83.9 | 88.9 | 83.1 | 75.6 | 85.7 | 72.8 | 69.5 | 91.0
MixCvT | 69.2 | 78.7 | 74.7 | 83.1 | 88.1 | 81.6 | 72.6 | 82.2 | 68.8 | 70.4 | 91.8
MixCvT-1k | 67.9 | 77.3 | 73.9 | 82.6 | 87.7 | 81.2 | 73.2 | 83.2 | 70.2 | 68.7 | 89.5
MixCvT* | - | - | - | - | - | - | 70.7 | 80.0 | 67.8 | - | -
MixCvT-1k* | - | - | - | - | - | - | 71.2 | 79.9 | 65.8 | - | -
SimTrack-L [9] | 70.5 | 79.7 | - | 83.4 | 87.4 | - | 69.8 | 78.8 | 66.0 | 71.2 | 91.6
SimTrack-B [9] | 69.3 | 78.5 | - | 82.3 | 86.5 | - | 68.6 | 78.9 | 62.4 | 69.8 | 89.6
OSTrack-384 [82] | 71.1 | 81.1 | 77.6 | 83.9 | 88.5 | 83.2 | 73.7 | 83.2 | 70.8 | 70.7 | -
OSTrack-256 [82] | 69.1 | 78.7 | 75.2 | 83.1 | 87.8 | 82.0 | 71.0 | 80.4 | 68.2 | 68.3 | -
AiATrack [27] | 69.0 | 79.4 | 73.8 | 82.7 | 87.8 | 80.4 | 69.6 | 80.0 | 63.2 | 70.6 | -
SwinTrack-B-384 [46] | 71.3 | - | 76.5 | 84.0 | - | 82.8 | 72.4 | 80.5 | 67.8 | - | -
ToMP-101 [55] | 68.5 | 79.2 | 73.5 | 81.5 | 86.4 | 78.9 | - | - | - | 66.9 | -
CSWinTT [62] | 66.2 | 75.2 | 70.9 | 81.9 | 86.7 | 79.5 | 69.4 | 78.9 | 65.4 | 70.5 | 90.3
SBT-large [75] | 66.7 | - | 71.1 | - | - | - | 70.4 | 80.8 | 64.7 | - | -
UTT [53] | 64.6 | - | 67.2 | 79.7 | - | 77.0 | 67.2 | 76.3 | 60.5 | - | -
STARK [80] | 67.1 | 77.0 | - | 82.0 | 86.9 | - | 68.8 | 78.1 | 64.1 | - | -
KeepTrack [56] | 67.1 | 77.2 | 70.2 | - | - | - | - | - | - | 69.7 | -
DTT [83] | 60.1 | - | - | 79.6 | 85.0 | 78.9 | 63.4 | 74.9 | 51.4 | - | -
SAOT [90] | 61.6 | 70.8 | - | - | - | - | 64.0 | 75.9 | - | - | -
AutoMatch [87] | 58.2 | - | 59.9 | 76.0 | - | 72.6 | 65.2 | 76.6 | 54.3 | - | -
TREG [14] | 64.0 | 74.1 | - | 78.5 | 83.8 | 75.0 | 66.8 | 77.8 | 57.2 | 66.9 | 88.4
DualTFR [76] | 63.5 | 72.0 | 66.5 | 80.1 | 84.9 | - | - | - | - | 68.2 | -
TransT [12] | 64.9 | 73.8 | 69.0 | 81.4 | 86.7 | 80.3 | 67.1 | 76.8 | 60.9 | 69.1 | -
TrDiMP [70] | 63.9 | - | 61.4 | 78.4 | 83.3 | 73.1 | 67.1 | 77.7 | 58.3 | 67.5 | -
STMTracker [25] | 60.6 | 69.3 | 63.3 | 80.3 | 85.1 | 76.7 | 64.2 | 73.7 | 57.5 | 64.7 | -
SiamR-CNN [67] | 64.8 | 72.2 | - | 81.2 | 85.4 | 80.0 | 64.9 | 72.8 | 59.7 | 64.9 | 83.4
PrDiMP [19] | 59.8 | 68.8 | 60.8 | 75.8 | 81.6 | 70.4 | 63.4 | 73.8 | 54.3 | 68.0 | -
OCEAN [88] | 56.0 | 65.1 | 56.6 | - | - | - | 61.1 | 72.1 | 47.3 | - | -
KYS [5] | 55.4 | 63.3 | - | 74.0 | 80.0 | 68.8 | 63.6 | 75.1 | 51.5 | - | -
FCOT [15] | 57.2 | 67.8 | - | 75.4 | 82.9 | 72.6 | 63.4 | 76.6 | 52.1 | 65.6 | 87.3
SiamAttn [84] | 56.0 | 64.8 | - | 75.2 | 81.7 | - | - | - | - | 65.0 | 84.5
SiamGAT [28] | 53.9 | 63.3 | 53.0 | - | - | - | 62.7 | 74.3 | 48.8 | 64.6 | 84.3
CGACD [22] | 51.8 | 62.6 | - | 71.1 | 80.0 | 69.3 | - | - | - | 63.3 | 83.3
SiamBAN [13] | 53.1 | - | 54.1 | 71.6 | 79.4 | 68.5 | - | - | - | 64.4 | 84.6
SiamCAR [29] | 50.7 | 60.0 | 51.0 | - | - | - | 56.9 | 67.0 | 41.5 | 61.4 | 76.0
SiamFC++ [79] | 54.4 | 62.3 | 54.7 | 75.4 | 80.0 | 70.5 | 59.5 | 69.5 | 47.9 | - | -
MAML [68] | 52.3 | - | - | 75.7 | 82.2 | 72.5 | - | - | - | - | -
D3S [50] | - | - | - | 72.8 | 76.8 | 66.4 | 59.7 | 67.6 | 46.2 | - | -
DiMP [4] | 56.9 | 65.0 | 56.7 | 74.0 | 80.1 | 68.7 | 61.1 | 71.7 | 49.2 | 65.4 | -
ATOM [18] | 51.5 | 57.6 | 50.5 | 70.3 | 77.1 | 64.8 | 55.6 | 63.4 | 40.2 | 64.3 | -
SiamRPN++ [39] | 49.6 | 56.9 | 49.1 | 73.3 | 80.0 | 69.4 | 51.7 | 61.6 | 32.5 | 61.0 | 80.3
SiamMask [71] | - | - | - | 72.5 | 77.8 | 66.4 | 51.4 | 36.6 | 58.7 | - | -
ECO [17] | 32.4 | 33.8 | 30.1 | 55.4 | 61.8 | 49.2 | 31.6 | 30.9 | 11.1 | 53.2 | -
MDNet [59] | 39.7 | 46.0 | 37.3 | 60.6 | 70.5 | 56.5 | 29.9 | 30.3 | 9.9 | 52.8 | -
SiamFC [3] | 33.6 | 42.0 | 33.9 | 57.1 | 66.3 | 53.3 | 34.8 | 35.3 | 9.8 | 48.5 | 69.3

###  4.4 Comparison with the state-of-the-art trackers

We verify the performance of our proposed MixCvT-1k, MixCvT-22k, MixCvT-L,
MixViT and MixViT-L on six benchmarks, including VOT2020 [36], LaSOT [23],
TrackingNet [58], GOT10k [33], UAV123 [57] and OTB100 [74].

Figure 10: Comparisons on OTB100 [74] with different challenging aspects: fast
motion, background clutter, motion blur, deformation, illumination variation,
in-plane rotation, low resolution, occlusion, out-of-plane rotation, out of
view and scale variation. Figure 11: State-of-the-art comparison on the LaSOT
dataset. Best viewed with zooming in. Figure 12: State-of-the-art comparison
on the OTB100 dataset. Best viewed with zooming in.

VOT2020. VOT2020 [36] consists of 60 videos with several challenges including
fast motion, occlusion, etc. Our trackers are tested on the dataset in
comparison with state-of-the-art trackers. As shown in Table IX, MixViT-L
achieves the top-ranked performance on EAO criteria of 0.584, which
outperforms the transformer tracker STARK with a large margin of 7.9% of EAO.
MixCvT-22k and MixViT-L outperform other trackers including RPT (VOT2020
short-term challenge winner). Besides, MixViT-L with ConvMAE pre-training
obtains lower EAO than MixViT-L. We analyze that this is mainly because
MixViT-L with ConvMAE pre-training removes the last three layers of original
ConvMAE which may lack some high-level representation and thus yielding a
negative impact on Robustness.

LaSOT. LaSOT [23] has 280 videos in its test set. We evaluate our MixFormer on
the test set to validate its long-term capability. The Table X shows that our
MixFormer models surpass all other trackers with a large margin. Specifically,
MixViT-L pretrained with ConvMAE achieves the top-ranked performance on AUC of
73.3%, surpassing ToMP [55] by 4.8% and SimTrack-L [9] (based on ViT-Large) by
2.8%. To give a further analysis, we provide Success plot and Precision plot
for LaSOT in Fig. 11. It proves that improvement is due to both higher
accuracy and robustness.

TrackingNet. TrackingNet [58] provides over 30K videos with more than 14
million dense bounding box annotations. The videos are sampled from YouTube,
covering target categories and scenes in real life. We validate our MixFormer
models on its test set. From Table X, we find that our MixCvT-L and MixViT-L
pretrained with ConvMAE set a new state-of-the-art performance on the large
scale benchmark, with AUC of 83.9% and 86.3% respectively. The results
indicate that the proposed framework is effective for tracking.

GOT10k. GOT10k [33] is a large-scale dataset with over 10000 video segments
and has 180 segments for the test set. Apart from generic classes of moving
objects and motion patterns, the object classes in the train and test set are
zero-overlapped. As shown in Table X, our MixViT-L obtains state-of-the-art
performance on the test split, outperforming SBT-large by 5.3%. MixViT-L
pretrained with ConvMAE-L obtains lower AO than MixViT-L pretrained with MAE-L
due to the reason similar with that of VOT2020.

UAV123. UAV123 [57] is a large dataset containing 123 Sequences with average
sequence length of 915 frames, which is captured from low-altitude UAVs. Table
X shows our results on UAV123 dataset. MixViT-L and MixCvT achieve competitive
performance with the ToMP-101 [55] and CSWinTT [62].

TABLE XI: The Top eight trackers for VOT-STb2022 public challenge [37]. Expected average overlap (EAO), accuracy and robustness are shown.  Tracker | EAO | Accuracy | Robustness
---|---|---|---
MixFormerL | 0.602 | 0.831 | 0.859
DAMT | 0.602 | 0.776 | 0.887
OSTrackSTB | 0.591 | 0.790 | 0.869
APMT_MR | 0.591 | 0.787 | 0.877
MixFormer | 0.587 | 0.797 | 0.874
APMT_RT | 0.581 | 0.787 | 0.877
ADOTstb | 0.569 | 0.775 | 0.862
SRATransT | 0.560 | 0.764 | 0.864
TABLE XII: The Top eight trackers for VOT-RGBD2022 challenge [37]. Expected average overlap (EAO), accuracy and robustness are shown.  Tracker | EAO | Accuracy | Robustness
---|---|---|---
MixForRGBD | 0.779 | 0.816 | 0.946
SAMF | 0.762 | 0.807 | 0.936
OSTrack | 0.729 | 0.808 | 0.894
ProMix | 0.722 | 0.798 | 0.900
SBT_RGBD | 0.708 | 0.809 | 0.864
DMTracker | 0.658 | 0.758 | 0.851
DeT_DiMP50_Max | 0.657 | 0.760 | 0.845
SPT | 0.651 | 0.798 | 0.851

OTB-100. OTB100 [74] is a commonly used benchmark, which evaluates performance
on Precision and AUC scores. Figure. 12 presents results of our trackers on
both the two metrics. MixCvT-L reaches competitive performance w.r.t. state-
of-the-art trackers, surpassing the transformer tracker TransT by 1.3% on AUC
score. MixViT with MAE pre-training obains the highest AUC score of 71.6% and
Precision score of 94.4%. In Figure. 10, we compare MixFormer models with
prevailing trackers on some different challenging aspects. The results prove
that a series of MixCvT and MixViT models consistently outperform other
trackers such as TransT [12], SiamBAN [13] and PrDiMP [19] on most of the
challenging aspects. Especially, MixViT significantly improves TransT [12] by
8.7% and 3.6% on the aspects of background clutter and motion blur
respectively, which indicates the strong discriminative ability of the
proposed mixed attention.

TABLE XIII: The Top six trackers for VOT-D2022 challenge [37]. Expected average overlap (EAO), accuracy and robustness are shown.  Tracker | EAO | Accuracy | Robustness
---|---|---|---
MixFormerD | 0.600 | 0.758 | 0.806
RSDiMP | 0.573 | 0.734 | 0.759
OSTrack_D | 0.568 | 0.735 | 0.774
DOT | 0.469 | 0.672 | 0.673
SBT_Depth | 0.462 | 0.756 | 0.571
UpDoT | 0.439 | 0.652 | 0.627

VOT2022 Challenges Results. We also test the performance of our MixFormer
trackers in the VOT2022 challenge. As shown in Table XI, ours MixFormerL
(based on the MixViT-L tracker) ranks 1/41 on the VOT2022-STb public
challenge. Besides, we notice that the VOT2022-RGBD and VOT2022-D winners of
MixForRGBD and MixForD, implemented by Lai Simiao, are both constructed upon
our MixFormer. This demonstrates the effectiveness and generalization of the
proposed MixFormer framework.

##  5 Conclusions

We have presented MixFormer, an compact end-to-end tracking framework with
iterative mixed attentions, aiming to unify the feature extraction and target
integration. Mixed attention module performs both feature extraction and
mutual interaction for target template and search area. We instanitate two
types of MixFormer trackers: a hierarchical tracker of MixCvT and a non-
hierarchical tracker of MixViT. We have performed extensive study on the
design and pre-training techniques of MixFormer trackers. We also extend the
MAE pre-training to the tracking framework and propose the TrackMAE to achieve
the competitive performance to the large-scale ImageNet pre-training.
Extensive evaluations on six common tracking benchmarks demonstrates that our
MixFormer tracker obtains a notable improvement over other prevailing
trackers. In the future, we consider extending MixFormer to multiple object
tracking or long-term object tracking.

## Acknowledgments

This work is supported by National Natural Science Foundation of China
(No.62076119, No.61921006), Program for Innovative Talents and Entrepreneur in
Jiangsu Province, and Collaborative Innovation Center of Novel Software
Technology and Industrialization.

## References

  * [1] Hangbo Bao, Li Dong, and Furu Wei.  Beit: Bert pre-training of image transformers.  International Conference on Learning Representations, ICLR, 2022\.
  * [2] Luca Bertinetto, Jack Valmadre, Stuart Golodetz, Ondrej Miksik, and Philip H. S. Torr.  Staple: Complementary learners for real-time tracking.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2016.
  * [3] Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Torr.  Fully-convolutional siamese networks for object tracking.  In Proceedings of the European Conference on Computer Vision, ECCV Workshops, 2016.
  * [4] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte.  Learning discriminative model prediction for tracking.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, pages 6182–6191, 2019.
  * [5] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte.  Know your surroundings: Exploiting scene information for object tracking.  In Proceedings of the European Conference on Computer Vision, ECCV, 2020.
  * [6] Goutam Bhat, Joakim Johnander, Martin Danelljan, Fahad Shahbaz Khan, and Michael Felsberg.  Unveiling the power of deep tracking.  In Vittorio Ferrari, Martial Hebert, Cristian Sminchisescu, and Yair Weiss, editors, Proceedings of the European Conference on Computer Vision, ECCV, 2018.
  * [7] David S Bolme, J Ross Beveridge, Bruce A Draper, and Yui Man Lui.  Visual object tracking using adaptive correlation filters.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2010.
  * [8] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In Proceedings of the European Conference on Computer Vision, ECCV, 2020.
  * [9] Boyu Chen, Peixia Li, Lei Bai, Lei Qiao, Qiuhong Shen, Bo Li, Weihao Gan, Wei Wu, and Wanli Ouyang.  Backbone is all your need: A simplified architecture for visual object tracking.  In Proceedings of the European Conference on Computer Vision, ECCV, 2022.
  * [10] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton.  A simple framework for contrastive learning of visual representations.  In International conference on machine learning. PMLR, 2020.
  * [11] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu.  Transformer tracking.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [12] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu.  Transformer tracking.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2021.
  * [13] Zedu Chen, Bineng Zhong, Guorong Li, Shengping Zhang, and Rongrong Ji.  Siamese box adaptive network for visual tracking.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2020.
  * [14] Yutao Cui, Cheng Jiang, Limin Wang, and Gangshan Wu.  Target transformed regression for accurate tracking.  CoRR, 2021.
  * [15] Yutao Cui, Cheng Jiang, Limin Wang, and Gangshan Wu.  Fully convolutional online tracking.  Computer Vision and Image Understanding, 224:103547, 2022.
  * [16] Yutao Cui, Cheng Jiang, Limin Wang, and Gangshan Wu.  Mixformer: End-to-end tracking with iterative mixed attention.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2022.
  * [17] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg.  ECO: efficient convolution operators for tracking.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2017.
  * [18] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg.  ATOM: accurate tracking by overlap maximization.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2019.
  * [19] Martin Danelljan, Luc Van Gool, and Radu Timofte.  Probabilistic regression for visual tracking.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2020.
  * [20] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2009.
  * [21] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby.  An image is worth 16x16 words: Transformers for image recognition at scale.  In International Conference on Learning Representations, ICLR, 2021.
  * [22] Fei Du, Peng Liu, Wei Zhao, and Xianglong Tang.  Correlation-guided attention for corner detection based visual tracking.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2020.
  * [23] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling.  Lasot: A high-quality benchmark for large-scale single object tracking.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2019.
  * [24] Heng Fan and Haibin Ling.  Siamese cascaded region proposal networks for real-time visual tracking.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2019.
  * [25] Zhihong Fu, Qingjie Liu, Zehua Fu, and Yunhong Wang.  Stmtrack: Template-free visual tracking with space-time memory networks.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2021.
  * [26] Peng Gao, Teli Ma, Hongsheng Li, Jifeng Dai, and Yu Qiao.  Mcmae: Masked convolution meets masked au- toencoders.  Neural Information Processing Systems, NIPS, 2022.
  * [27] Shenyuan Gao, Chunluan Zhou, Chao Ma, Xinggang Wang, and Junsong Yuan.  Aiatrack: Attention in attention for transformer visual tracking.  In Proceedings of the European Conference on Computer Vision, ECCV, 2022.
  * [28] Dongyan Guo, Yanyan Shao, Ying Cui, Zhenhua Wang, Liyan Zhang, and Chunhua Shen.  Graph attention tracking.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2021.
  * [29] Dongyan Guo, Jun Wang, Ying Cui, Zhenhua Wang, and Shengyong Chen.  Siamcar: Siamese fully convolutional classification and regression for visual tracking.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2020.
  * [30] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2022.
  * [31] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick.  Momentum contrast for unsupervised visual representation learning.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2020.
  * [32] João F. Henriques, Rui Caseiro, Pedro Martins, and Jorge Batista.  High-speed tracking with kernelized correlation filters.  IEEE Trans. Pattern Anal. Mach. Intell., 37(3):583–596, 2015\.
  * [33] Lianghua Huang, Xin Zhao, and Kaiqi Huang.  Got-10k: A large high-diversity benchmark for generic object tracking in the wild.  IEEE Trans. Pattern Anal. Mach. Intell., 43(5):1562–1577, 2021\.
  * [34] Hamed Kiani Galoogahi, Ashton Fagg, and Simon Lucey.  Learning background-aware correlation filters for visual tracking.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, 2017.
  * [35] Diederik P. Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In Yoshua Bengio and Yann LeCun, editors, International Conference on Learning Representations, ICLR, 2015.
  * [36] Matej Kristan, Ales Leonardis, and et. al.  The eighth visual object tracking VOT2020 challenge results.  In Adrien Bartoli and Andrea Fusiello, editors, Proceedings of the European Conference on Computer Vision, ECCV Workshops, 2020.
  * [37] Matej Kristan, Ales Leonardis, Jiri Matas, Michael Felsberg, Roman Pflugfelder, Joni-Kristian Kamarainen, Hyung Jin Chang, Martin Danelljan, Luka Čehovin Zajc, Alan Lukežič, Ondrej Drbohlav, Johanna Bjorklund, Yushan Zhang, Zhongqun Zhang, Song Yan, Wenyan Yang, Dingding Cai, Christoph Mayer, and Gustavo Fernandez.  The tenth visual object tracking vot2022 challenge results.  In Proceedings of the European Conference on Computer Vision, ECCV Workshops, 2022.
  * [38] Matej Kristan, Jiří Matas, Aleš Leonardis, Michael Felsberg, Roman Pflugfelder, Joni-Kristian Kämäräinen, Hyung Jin Chang, Martin Danelljan, Luka Cehovin, Alan Lukežič, Ondrej Drbohlav, Jani Käpylä, Gustav Häger, Song Yan, Jinyu Yang, Zhongqun Zhang, and Gustavo Fernández.  The ninth visual object tracking vot2021 challenge results.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV Workshops, 2021.
  * [39] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan.  Siamrpn++: Evolution of siamese visual tracking with very deep networks.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2019.
  * [40] Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu.  High performance visual tracking with siamese region proposal network.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2018.
  * [41] Feng Li, Cheng Tian, Wangmeng Zuo, Lei Zhang, and Ming-Hsuan Yang.  Learning spatial-temporal regularized correlation filters for visual tracking.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2018.
  * [42] Xiang Li, Wenhai Wang, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang.  Generalized focal loss V2: learning reliable localization quality estimation for dense object detection.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2021.
  * [43] Yanghao Li, Hanzi Mao, Ross B. Girshick, and Kaiming He.  Exploring plain vision transformer backbones for object detection.  In Shai Avidan, Gabriel J. Brostow, Moustapha Cissé, Giovanni Maria Farinella, and Tal Hassner, editors, ECCV, pages 280–296, 2022.
  * [44] Yawei Li, Kai Zhang, Jiezhang Cao, Radu Timofte, and Luc Van Gool.  Localvit: Bringing locality to vision transformers.  CoRR, abs/2104.05707, 2021.
  * [45] Bingyan Liao, Chenye Wang, Yayun Wang, Yaonong Wang, and Jun Yin.  Pg-net: Pixel to global matching network for visual tracking.  In Proceedings of the European Conference on Computer Vision, ECCV, 2020.
  * [46] Liting Lin, Heng Fan, Yong Xu, and Haibin Ling.  Swintrack: A simple and strong baseline for transformer tracking.  Neural Information Processing Systems, NIPS, 2022.
  * [47] Tsung-Yi Lin, Michael Maire, Serge J. Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick.  Microsoft COCO: common objects in context.  In Proceedings of the European Conference on Computer Vision, ECCV, 2014.
  * [48] Liwei Liu, Junliang Xing, Haizhou Ai, and Xiang Ruan.  Hand posture recognition using finger geometric feature.  In ICPR, 2012.
  * [49] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, 2021.
  * [50] Alan Lukezic, Jiri Matas, and Matej Kristan.  D3S - A discriminative single shot segmentation tracker.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2020.
  * [51] Alan Lukezic, Tomas Vojir, Luka Cehovin Zajc, Jiri Matas, and Matej Kristan.  Discriminative correlation filter with channel and spatial reliability.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2017.
  * [52] Alan Lukezic, Tomas Vojir, Luka Cehovin Zajc, Jiri Matas, and Matej Kristan.  Discriminative correlation filter with channel and spatial reliability.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2017.
  * [53] Fan Ma, Mike Zheng Shou, Linchao Zhu, Haoqi Fan, Yilei Xu, Yi Yang, and Zhicheng Yan.  Unified transformer tracker for object tracking.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2022.
  * [54] Ziang Ma, Linyuan Wang, Haitao Zhang, Wei Lu, and Jun Yin.  RPT: learning point set representation for siamese visual tracking.  In Adrien Bartoli and Andrea Fusiello, editors, Computer Vision \- ECCV 2020 Workshops, Lecture Notes in Computer Science, 2020.
  * [55] Christoph Mayer, Martin Danelljan, Goutam Bhat, Matthieu Paul, Danda Pani Paudel, Fisher Yu, and Luc Van Gool.  Transforming model prediction for tracking.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2022.
  * [56] Christoph Mayer, Martin Danelljan, Danda Pani Paudel, and Luc Van Gool.  Learning target candidate association to keep track of what not to track.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, 2021.
  * [57] Matthias Mueller, Neil Smith, and Bernard Ghanem.  A benchmark and simulator for UAV tracking.  In Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling, editors, Proceedings of the European Conference on Computer Vision, ECCV, 2016\.
  * [58] Matthias Müller, Adel Bibi, Silvio Giancola, Salman Al-Subaihi, and Bernard Ghanem.  Trackingnet: A large-scale dataset and benchmark for object tracking in the wild.  In Proceedings of the European Conference on Computer Vision, ECCV, 2018.
  * [59] Hyeonseob Nam and Bohyung Han.  Learning multi-domain convolutional neural networks for visual tracking.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2016.
  * [60] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian D. Reid, and Silvio Savarese.  Generalized intersection over union: A metric and a loss for bounding box regression.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2019.
  * [61] Yibing Song, Chao Ma, Xiaohe Wu, Lijun Gong, Linchao Bao, Wangmeng Zuo, Chunhua Shen, Rynson W. H. Lau, and Ming-Hsuan Yang.  VITAL: visual tracking via adversarial learning.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2018.
  * [62] Zikai Song, Junqing Yu, Yi-Ping Phoebe Chen, and Wei Yang.  Transformer tracking with cyclic shifting window attention.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2022.
  * [63] Zhan Tong, Yibing Song, Jue Wang, and Limin Wang.  Videomae: Masked autoencoders are data-efficient learners for self-supervised video pre-training.  Neural Information Processing Systems, NIPS, 2022.
  * [64] Hugo Touvron, Matthieu Cord, and Hervé Jégou.  Deit iii: Revenge of the vit.  Proceedings of the European Conference on Computer Vision, ECCV, 2022.
  * [65] Jack Valmadre, Luca Bertinetto, Joao Henriques, Andrea Vedaldi, and Philip HS Torr.  End-to-end representation learning for correlation filter based tracking.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2017.
  * [66] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In Neural Information Processing Systems, NIPS, 2017.
  * [67] Paul Voigtlaender, Jonathon Luiten, Philip H.S. Torr, and Bastian Leibe.  Siam r-cnn: Visual tracking by re-detection.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2020.
  * [68] Guangting Wang, Chong Luo, Xiaoyan Sun, Zhiwei Xiong, and Wenjun Zeng.  Tracking by instance detection: A meta-learning approach.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2020.
  * [69] Limin Wang, Yuanjun Xiong, Zhe Wang, Yu Qiao, Dahua Lin, Xiaoou Tang, and Luc Van Gool.  Temporal segment networks: Towards good practices for deep action recognition.  In Proceedings of the European Conference on Computer Vision, ECCV, 2016.
  * [70] Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li.  Transformer meets tracker: Exploiting temporal context for robust visual tracking.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
  * [71] Qiang Wang, Li Zhang, Luca Bertinetto, Weiming Hu, and Philip H. S. Torr.  Fast online object tracking and segmentation: A unifying approach.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2019.
  * [72] Wenhai Wang, Enze Xie, Xiang Li, Deng-Ping Fan, Kaitao Song, Ding Liang, Tong Lu, Ping Luo, and Ling Shao.  Pyramid vision transformer: A versatile backbone for dense prediction without convolutions.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, 2021.
  * [73] Haiping Wu, Bin Xiao, Noel Codella, Mengchen Liu, Xiyang Dai, Lu Yuan, and Lei Zhang.  Cvt: Introducing convolutions to vision transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, pages 22–31, 2021.
  * [74] Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang.  Object tracking benchmark.  IEEE Trans. Pattern Anal. Mach. Intell., 37(9):1834–1848, 2015\.
  * [75] Fei Xie, Chunyu Wang, Guangting Wang, Yue Cao, Wankou Yang, and Wenjun Zeng.  Correlation-aware deep tracking.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2022.
  * [76] Fei Xie, Chunyu Wang, Guangting Wang, Wankou Yang, and Wenjun Zeng.  Learning tracking representations via dual-branch fully transformer networks.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV Workshops, 2021.
  * [77] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu.  Simmim: A simple framework for masked image modeling.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2022.
  * [78] Junliang Xing, Haizhou Ai, and Shihong Lao.  Multiple human tracking based on multi-view upper-body detection and discriminative learning.  In ICPR, 2010.
  * [79] Yinda Xu, Zeyu Wang, Zuoxin Li, Ye Yuan, and Gang Yu.  Siamfc++: Towards robust and accurate visual tracking with target estimation guidelines.  In Proceedings of the AAAI Conference on Artificial Intelligence, AAAI, 2020.
  * [80] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu.  Learning spatio-temporal transformer for visual tracking.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, 2021.
  * [81] Bin Yan, Xinyu Zhang, Dong Wang, Huchuan Lu, and Xiaoyun Yang.  Alpha-refine: Boosting tracking performance by precise bounding box estimation.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2021.
  * [82] Botao Ye, Hong Chang, Bingpeng Ma, and Shiguang Shan.  Joint feature learning and relation modeling for tracking: A one-stream framework.  Proceedings of the European Conference on Computer Vision, ECCV, 2022.
  * [83] Bin Yu, Ming Tang, Linyu Zheng, Guibo Zhu, Jinqiao Wang, Hao Feng, Xuetao Feng, and Hanqing Lu.  High-performance discriminative tracking with transformers.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, 2021.
  * [84] Yuechen Yu, Yilei Xiong, Weilin Huang, and Matthew R. Scott.  Deformable siamese attention networks for visual object tracking.  In IEEE Conference on Computer Vision and Pattern Recognition, CVPR, 2020.
  * [85] Li Yuan, Yunpeng Chen, Tao Wang, Weihao Yu, Yujun Shi, Zi-Hang Jiang, Francis EH Tay, Jiashi Feng, and Shuicheng Yan.  Tokens-to-token vit: Training vision transformers from scratch on imagenet.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, 2021.
  * [86] Zhipeng Zhang, Yufan Liu, Bing Li, Weiming Hu, and Houwen Peng.  Toward accurate pixelwise object tracking via attention retrieval.  IEEE Transactions on Image Processing, 30:8553–8566, 2021.
  * [87] Zhipeng Zhang, Yihao Liu, Xiao Wang, Bing Li, and Weiming Hu.  Learn to match: Automatic matching network design for visual tracking.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, 2021.
  * [88] Zhipeng Zhang, Houwen Peng, Jianlong Fu, Bing Li, and Weiming Hu.  Ocean: Object-aware anchor-free tracking.  In Proceedings of the European Conference on Computer Vision, ECCV, 2020.
  * [89] Zhaohui Zheng, Ping Wang, Wei Liu, Jinze Li, Rongguang Ye, and Dongwei Ren.  Distance-iou loss: Faster and better learning for bounding box regression.  In Proceedings of the AAAI Conference on Artificial Intelligence, AAAI, 2020.
  * [90] Zikun Zhou, Wenjie Pei, Xin Li, Hongpeng Wang, Feng Zheng, and Zhenyu He.  Saliency-associated object tracking.  In Proceedings of the IEEE/CVF International Conference on Computer Vision, ICCV, 2021.
  * [91] Zheng Zhu, Qiang Wang, Bo Li, Wei Wu, Junjie Yan, and Weiming Hu.  Distractor-aware siamese networks for visual object tracking.  In Proceedings of the European Conference on Computer Vision, ECCV, 2018.