# Efficient Training for Visual Tracking with Deformable Transformer

Qingmao Wei, Bi Zeng, Guotian Zeng
Guangdong University of Technology
<EMAIL_ADDRESS>
###### Abstract

Recent Transformer-based visual tracking models have showcased superior
performance. Nevertheless, prior works have been resource-intensive, requiring
prolonged GPU training hours and incurring high GFLOPs during inference due to
inefficient training methods and convolution-based target heads. This
intensive resource use renders them unsuitable for real-world applications. In
this paper, we present DETRack, a streamlined end-to-end visual object
tracking framework. Our framework utilizes an efficient encoder-decoder
structure where the deformable transformer decoder acting as a target head,
achieves higher sparsity than traditional convolution heads, resulting in
decreased GFLOPs. For training, we introduce a novel one-to-many label
assignment and an auxiliary denoising technique, significantly accelerating
model’s convergence. Comprehensive experiments affirm the effectiveness and
efficiency of our proposed method. For instance, DETRack achieves 72.9% AO on
challenging GOT-10k benchmarks using only 20% of the training epochs required
by the baseline, and runs with lower GFLOPs than all the transformer-based
trackers.

##  1 Introduction

Figure 1: Comparison of our DETRack with other trackers on GOT-10k benchmark
in terms of trained epochs following the official one-shot protocol [18]. The
bubble size represents the relative GFLOPs. Our DETRack sets up a best trade-
off among accuracy, trained epochs and running GFLOPs.

Visual object tracking remains a critical challenge in computer vision,
finding applications in diverse areas from surveillance to robotics and
autonomous driving. The recent adoption of the Transformer [35] within visual
trackers [8, 37, 43, 9, 34, 6, 14] has introduced both innovation and
complexity to this domain. Although deep learning advancements have elevated
performance, training state-of-the-art (SOTA) trackers leveraging Transformers
remains demanding, both in terms of time and computational resources. The
significant GPU hours necessary for training a competitive tracker pose
challenges, particularly for researchers with limited computational resources.
Furthermore, the substantial parameters and GFLOPs associated with a SOTA
tracker hinder their applicability in downstream tasks.

The present mainstream approaches for visual object tracking typically follow
three primary stages: (i) deep neural network based feature extraction from
the search and template images (ii) an integration module using either
convolution or attention mechanisms for feature matching/fusion, and (iii)a
head for bounding-box localization through customized heads for corner, center
or scale estimation, and target classification. In some cases, the first two
stages can be combined using a unified architecture, _e.g_. Transformer
Encoder and thus enjoying the powerful mask-image-modeling pretraining [16, 7,
39]. For accelerating running speed, sparsity is bringed into this unified
process by some works. Specifically, some image features, also called as
tokens in the Transformer, can be dropped if they are considered irrelevant to
the target to be tracked. The sparicified process reduces about 66% tokens
compared to the original one [43] before feeding the tokens into the head.
Those dropped tokens reduce about 20% to 30% GFLOPs in the encoder. However,
the prevailing head designs in the last stage are usually convolutional based,
which require a fully 2D feature map as input. Before feeding the sparicified
tokens into the convolution head, those dropped ones must be padded (usually
by zeros). In the end, convolutional computaions are performed redundantly
over padded dropped tokens. On the other hand, the learning objective for
classification in the prevailing head designs, such as center and corner
require the model predicting a unique and sharp class map, which is difficult
to optimize.

Figure 2: (a) Trackers with convolutional head have to padded the dropped
features; (b) Our DETRack with the decoder maintains the full sparsity without
redundant computation on the padded features.

To tackle the redundant cumputation problem, we introduce the transformer
decoder as the target head and thus make our model a encoder-decoder
framework, dubbed as DETRack. With capability of handling seqeunce of
features, the decoder only deal with the reserved tokens after sparcified
process by the encoder, avoiding redundant computation on the dropped tokens.
It enables our model keeping fully sparcified as shown in Fig. 2. Insead of
applying one-to-one Hungarian matching during training like the other encoder-
decoder designs in object detection [5, 23], we found that a loose one-to-many
label assignment significantly accelerating the training convergence. We allow
all the feature pixels within the GT bounding box to predict a positive
classification score, which is expected to reflect the localization quality,
_e.g_., IoU between the predicted bounding box and the GT. For localization,
we pick up the multiple predicted bounding box with the highest classification
scores, thus introducing more supervision signal in the training. Further
more, we design a novel denoising branch as a auxiliary training strategy. The
noised GT bounding boxes as region proposals enrich the diversity of training
samples, which further acccelerate the training speed.

Our extensive experiments validate the effectiveness and efficiency of our
DETRack. Specifically, compared to the most training-efficient transformer-
based tracker OSTrack, our method further reduce the training epochs to 20%,
while achieving a higher performance with lower GFLOPs as shown in Fig. 1 and
Tab. 1. Our main contributions are summarized as follows:

  * •

We propose a DETR-like encoder-decoder framework for visual object tracking
without convolution head, thus maintain the computation efficiency for a
sparsified backbone.

  * •

We design a novel one-to-many label assignment during training, which
significantly accelerate the trainging convergence.

  * •

To avoid low quality prediction and further acccelerate training convergence,
we introduce a denoising training strategy which bring in rich supervision
signal.

##  2 Related Work

###  2.1 Visual Tracking Paradims

Over the past few years, Siamese trackers [1, 22, 10] have gained much
popularity. Typically, they adopt a two-stream pipeline to separately extract
the features of the template and search region. Cross-relations between the
two streams are modeled by additional correlation modules. To exploit the
power of highly discriminative features, most Siamese trackers [47, 20, 3, 48]
use a pre-trained deep neural networks as the backbone, _e.g_. ResNet-50 [17],
leave them frozen in training, thus they only need very few epochs to train
the tracker. Recently, the Transformer [36] architecture has achieved
promising results in visual tracking and has become the de-facto choice for
many high-performance trackers [37, 8, 41, 29, 14, 40, 25].

To further enhance the feature interaction, several attempts [44, 38, 15] have
investigated cross-relation modeling inside the backbone. Recently, another
thread of progress [41, 25] concatenates the template and search tokens to
conduct cross-relation modeling and self-relation modeling jointly. Inspired
by these explorations, more recent trackers [9, 43, 6] adopt a one-stream
pipeline to jointly extract the features and model the relations of both the
template and search region by the self-attention mechanism. Based on this
pipeline, they can utilize advanced pretrained models, _e.g_. MAE [16],
instead of randomly initialized correlation modules for cross-relation
modeling, thereby yielding a remarkable performance gain.

###  2.2 DEtection with TRansformer (DETR)

Carion et al. [5] proposed a Transformer-based end-to-end object detector
named DETR (DEtection TRansformer) without using hand-designed components like
anchor design and NMS. Many follow-up papers have attempted to address the
slow training convergence issue of DETR introduced by decoder cross-attention.
Deformable-DETR [49] proposed a deformable attention module to focus on
important regions from multiple feature levels. DN-DETR [23] and DINO [45]
develop a denoising training strategy to helps the model to avoid duplicate
outputs of the same target. Borrowing inspiration from DETR, STARK [41] casts
target tracking as a bounding box prediction problem and solve it with an
encoder-decoder transformer, in which the encoder models the global
spatiotemporal feature dependencies between targets and search regions.
However, it still adopt a convolutional head after the transformer for the
final prediction. We also adopt the encoder-decoder framework for our tracker
inspired by the DETR-like models, but totally get rid off the dense operation
like convolution. Besides, we apply a novel label assignment and denoising
training strategy, significantly reduced the training GPU hours.

##  3 Methods

This section presents the DETRack method in detail. First, we briefly overview
the model architecture of our DETRack framework, including the encoder and the
decoder. Then, we introduce the training strategy, including label assignment
and denoising training.

###  3.1 Preliminary

DETR [5] is an end-to-end Transformer-based framework for object detection. In
DETR, each query in the transformer is expected to associated with one object
in the image. As studied in some variants of DETR [27, 45], it becomes clear
that queries in DETR are formed by two parts: a positional part and a content
part, which are referred to as positional queries and content queries in this
paper. DAB-DETR [27] explicitly formulates each positional query in DETR as a
4D anchor box $(x,y,w,h)$, where $x$ and $y$ are the center coordinates of the
box and $w$ and $h$ correspond to its width and height. Such an explicit
anchor box formulation makes it easy to dynamically refine anchor boxes layer
by layer in the decoder.

DN-DETR [23] and DINO [45] introduces a denoising (DN) training method to
accelerate the training convergence of DETR-like models. It shows that the
slow convergence problem in DETR is caused by the instability of bipartite
matching. To mitigate this problem, DN-DETR proposes to additionally feed
noised ground-truth (GT) labels and boxes into the Transformer decoder and
train the model to reconstruct the ground-truth ones.

###  3.2 Model Architecture

Figure 3:  (a) Illustration of the architecture of our proposed DETRack. The
denoising part in the decoder is only added during training. (b) The label
assignment for the denoising training. The most center feature within the GT
bounding box is picked as the positive object query in one denoising group,
while the corner ones are selected as negative queries.

We use an encoder-decoder structure for learning and inference. Such a network
architecture is widely used in modern visual recognition, especially DETR-like
models [23, 49, 23]. Unlike DETR [5] usually using a convolutional backbone
for feature extraction in object detection, we simply use a single transformer
encoder as backbone.

Encoder. The encoder is identical to the one used in Vision Transformer (ViT).
We start by dividing a given image pair, comprising a template and a search
region, into smaller image patches. Specifically, we denote the template image
patch as $z\in\mathbb{R}^{3\times H_{z}\times W_{z}}$ and the search region
patch as $x\in\mathbb{R}^{3\times H_{x}\times W_{x}}$. These patches are then
transformed into tokens, $\bm{H}_{z}\in\mathbb{R}^{D}$ and
$\bm{H}_{x}\in\mathbb{R}^{D}$, using a linear projection layer. Then we add
template and search tokens with positional embeddings, concatenate as
$[\bm{H}_{z};\bm{H}_{x}]$ and feed them into a plain ViT encoder to encode
visual features. For computation efficiency, some tokens can be dropped if
they are considered irrelevant to the target in the encoder. Here we simply
adopt candidate elimination from OSTrack [43] for the sparcified process.

Query Selection. Tokens from the encoder directly serve as object queries in
the decoder. If there is no spacified process for dropping some tokens in the
encoder, it will bring unacceptable computational and memory cost for the
self-attention modules in the decoder. To avoid this problem, we only select
$K$ (we set $K=64$ in our implemention) tokens for the decoder no matter how
many tokens are reserved after the encoder. Specically, each tokens output by
the encoder directly predicts a bounding box and a foreground score by a
linear layer. The tokens with top-K score will be selected as content queires,
and the bouding box of them are picked as region proposals.

Decoder. There are mainly self attention and cross attention in each layer of
the decoder. The query elements for both types of attention modules are of
object queries. The self attention is a standard multi-head self attention,
where object queries interact with each other. In the cross attention modules,
object queries interact with the features from the encoder, where the key
elements are of the output tokens from the encoder. The output object quiries
by the decoder will be fed in to a MLP for the bounding box prediction and a
linear layer for the foreground score prediction. Unlike the encoder, we adopt
the deformable attention [49] in the decoder, which will be detailed describe
in Sec. 3.3.

###  3.3 Deformable Transformer Decoder

Figure 4:  Illustration of one layer in the Deformable Transformer Decoder. In
each layer, the query tokens elementwise add the position embeddings generated
by the box proposals to form the object queries for the attention module. In
the end of the layer, the predicted offset plus the box proposal as refined
boxes are fed into the next layer.

As shown in Fig. 4, we formulate each object query in the deformable
transformer decoder as two part: position part and content part in the
decoder. The postion part are initialized from the region proposals (see Fig.
3) by $\sin$-$\cos$ embdeding in the fisrt layer. For content part, we
initialze it directly with the tokens from the encoder. We simply element-wise
add the two parts for the object queires. After a standard multi-head
attention, the core module in the decoder is deformable attention, in which
the queires only attend to a small set of key sampling points around a
reference point, regardless of the spatial size of the feature maps.

Deformable Attention. Given an input feature map
${\bm{x}}\in{\mathbb{R}}^{C\times H\times W}$, let $q$ index a query element
with content feature ${\bm{z}}_{q}$ and a 4-d reference box $(x,y,w,h)$,
denoted as ${\bm{p}}_{q}$. The deformable attention feature is calculated by

$\displaystyle\text{DeformAttn}({\bm{z}}_{q},{\bm{p}}_{q},{\bm{x}})=$ (1)
$\displaystyle\sum_{m=1}^{M}{\bm{W}}_{m}\big{[}\sum_{k=1}^{K}{A}_{mqk}\cdot{\bm{W}}^{\prime}_{m}{\bm{x}}({\bm{p}}_{q}+\Delta{\bm{p}}_{mqk})\big{]},$

where $m$ indexes the attention head, $k$ indexes the sampled keys, and $K$ is
the total sampled key number ($K\ll HW$). $\Delta{\bm{p}}_{mqk}$ and
${A}_{mqk}$ denote the sampling offset and attention weight of the
$k^{\text{th}}$ sampling point in the $m^{\text{th}}$ attention head,
respectively. The scalar attention weight ${A}_{mqk}$ lies in the range
$[0,1]$, normalized by $\sum_{k=1}^{K}{A}_{mqk}=1$.
$\Delta{\bm{p}}_{mqk}\in{\mathbb{R}}^{4}$ are of 4-d real numbers with
unconstrained range. As ${\bm{p}}_{q}+\Delta{\bm{p}}_{mqk}$ is fractional,
bilinear interpolation is applied in computing
${\bm{x}}({\bm{p}}_{q}+\Delta{\bm{p}}_{mqk})$. Both $\Delta{\bm{p}}_{mqk}$ and
${A}_{mqk}$ are obtained via linear projection over the query feature
${\bm{z}}_{q}$.

After a layer of decoder, the object queires are fed into a MLP to predict
another 4-D offset $(\Delta x,\Delta y,\Delta w,\Delta h)$, as the adjustment
to the target bounding box from the last layer’s output. As shown in Fig. 4,
the refined box output by the current layer serves as the 4-d reference box
for the next layer of decoder.

###  3.4 Lable Assignment

In object detection, DETR fundamentally addresses the issue of encouraging
redundant predictions during training and the need for additional modules to
eliminate such duplications during inference. DETR’s one-to-one Hungarian
matching promotes independent predictions during training. However, in single-
object tracking, a given input image contains at most one target to be
tracked. Thus, it’s sufficient to merely select the prediction with the
highest confidence from the final output.

Certain studies in object detection have indicated that encouraging repetitive
predictions can significantly accelerate the convergence of training. We treat
all tokens within the Ground Truth (GT) bounding box as potential positive
samples. However, simply assigning a hard label to the positive samples makes
it difficult to select a high-quality prediction. To make the positive samples
different, we introduce the localization quality (i.e., IoU score) into the
classification score inpired by QFL [24], where its supervision softens the
standard one-hot category label and leads to a possible float target
$y\in[0,1]$. Specifically, $y=0$ denotes the negative samples with $0$ quality
score, and $0<y\leq 1$ stands for the positive samples with target IoU score
$y$. Therefore, the loss function for the classification is:

$\mathcal{L}_{\operatorname{cls}}(\sigma)=-|y-\sigma|^{\beta}((1-y)\log(1-\sigma)+y\log(\sigma)),$
(2)

where that $\sigma=y$ is the global minimum solution, standing for a accurate
quality estimation. The parameter $\beta$ controls the down-weighting rate
$|y-\sigma|^{\beta}$ smoothly. In experiments, we set $\beta=2$ following
[24].

For the box regression, to prevent supervision of low-quality tokens (e.g.,
tokens at the very edges or corners within the GT box), we filter out and
retain only the top-k tokens based on predicted classification scores. We then
compute the localization loss exclusively for the boxes output by these
selected tokens. We combine the $l_{1}$ loss and the generalized IoU loss [33]
as the training objective for the localization. The loss function can be
formulated as:

$\mathcal{L}_{\operatorname{loc}}(b)=\lambda_{\operatorname{G}}\mathcal{L}_{\operatorname{GIoU}}(b_{i},\hat{b}_{i})+\lambda_{l_{1}}\mathcal{L}_{l_{1}}(b_{i},\hat{b}_{i}),$
(3)

where $b_{i}$ represents the groundtruth, and $\hat{b}_{i}$ represents the
predicted box. In experiments, we set the weights
$\lambda_{\operatorname{G}}=2$ and $\lambda_{l_{1}}=5$.

###  3.5 DeNoising Training

For each search image, we generate extra queires after the query selection by
adding the noise to the GT. The noised queires enrich the supervision signal
for the decoder during the training process. In implementation, the noised
queires are also formulated as two part: position (box) and content (class).

Box Denoising. We consider adding noise to boxes in two ways as in [23]:
center shifting and box scaling. We define $\lambda_{1}$ and $\lambda_{2}$ as
the noise scale of these 2 noises. 1) center shifting: we add a random noise
$(\Delta x,\Delta y)$, to the box center and make sure that $|\Delta
x|<\frac{\lambda_{1}w}{2}$, $|\Delta y|<\frac{\lambda_{1}h}{2}$ , where
$\lambda_{1}\in(0,1)$ so that the center of the noised box will still lie
inside the original bounding box. 2) box scaling: we set a hyper-parameter
$\lambda_{2}\in(0,1)$. The width and height of the box are randomly sampled
respectively. The noise scale for negative queires are set larger than that
for the positive.

For class denoising, the token positioned at the very center of the GT
bounding box is chosen as the content part of the positive query, as
illustrated in Fig. 3(b). To sidestep low-quality predictions, we select the
corner tokens for the negative queries. We adopt a soft label assignment for
alignment with the regular part in Tab. 6. Specifically, the labels for
positive queries are determined by the IoU between the predicted refined
bounding box and the GT bounding box. In contrast, labels for negative queries
are set to zero.

Attention Mask. It is proved to compromise the performance instead of
improving it, without an attention mask preventing the GT information leaking
from the denoising quires to un-denoising part [23]. We devide the denoising
queires as multiple groups, in which contains a positive and a negative query.
Queries in different groups and different parts cannot interact with each
other. The devision is performed by adding a mask to the self-attention module
in the decoder as in [23].

###  3.6 Training Objective

The training objective for both denoising branch and regular branch are
similar. Specifically, we apply QFL in both regular branch and denoising
branch for classification. For localization in denoising branch, we compute
loss only on the positive queries. As in DETR [5], we add auxiliary losses
after each decoder layer and query selection module. Considering the loss of
denoising branch after each decoder layer, our final loss can be written as
generalized format:

$\mathcal{L}=\sum_{l=0}^{L_{\operatorname{dec}}}[\lambda_{\operatorname{cls}}(\mathcal{L}_{\operatorname{cls}}+\mathcal{L}_{\operatorname{cls}}^{\operatorname{DN}})+\lambda_{\operatorname{loc}}(\mathcal{L}_{\operatorname{loc}}+\mathcal{L}_{\operatorname{cls}}^{\operatorname{DN}})],$
(4)

where $l=0$ denotes for the output before the query selection and
$l=L_{\operatorname{dec}}$ represents for the final output of the decoder. We
simply set the parameter $\lambda_{\operatorname{cls}}$ and
$\lambda_{\operatorname{loc}}$ to 1 following [45].

##  4 Experiments

Tracker | Source | GOT-10k* | #Epochs∗∗ | TrackingNet | LaSOT | #Epochs∗∗ | GFLOPs
---|---|---|---|---|---|---|---
AO | SR0.5 | SR0.75 |  AUC | P${}_{\text{norm}}$ | P | AUC | P${}_{\text{norm}}$ |  P
DETRack | Ours | 72.9 | 82.1 | 69.9 |  20 |  83.2 | 88.3 | 83.1 | 69.0 | 78.9 |  75.1 |  60 | 15.6
OSTrack [43] | ECCV’22 | 71.0 | 80.4 | 68.2 |  100 |  83.1 | 87.8 | 82.0 | 69.1 | 78.7 |  75.2 |  300 | 21.5
OSTrack† [43] | ECCV’22 | 71.5 | 80.7 | 68.9 |  100 |  83.0 | 87.7 | 82.0 | 68.5 | 78.1 |  74.9 |  300 | 16.4
AiATrack [14] | ECCV’22 | 69.6 | 80.0 | 63.2 |  100 |  82.7 | 87.8 | 80.4 | 69.0 | 79.4 |  73.8 |  300 | 18.5
SimTrack [6] | ECCV’22 | 68.6 | 78.9 | 62.4 |  500 |  82.3 | 86.5 | - | 69.3 | 78.5 |  74.0 |  500 | 20.1
Unicorn [40] | ECCV’22 | - | - | - |  - |  83.0 | 86.4 | 82.2 | 68.5 | 76.6 |  74.1 |  - | 33.5
MixFormer [9] | CVPR’22 | 70.7 | 80.0 | 67.8 |  180 |  83.1 | 88.1 | 81.6 | 69.2 | 78.7 |  74.7 |  550 | 23.0
ToMP [29] | CVPR’22 | - | - | - |  - |  81.2 | 86.2 | 78.6 | 67.6 | 78.0 |  72.2 |  200 | -
CSWinTT [34] | CVPR’22 | 69.4 | 78.9 | 65.4 |  600 |  81.9 | 86.7 | 79.5 | 66.2 | 75.2 |  70.9 |  500 | 19.3
STARK [41] | ICCV’21 | 68.0 | 77.7 | 62.3 |  500 |  81.3 | 86.1 | 78.1 | 66.4 | 76.3 |  71.2 |  500 | 18.5
KeepTrack [30] | ICCV’21 | - | - | - |  - |  - | - | - | 67.1 | 77.2 |  70.2 |  - | -
AutoMatch [46] | ICCV’21 | 65.2 | 76.6 | 54.3 |  - |  76.0 | - | 72.6 | 58.3 | - |  59.9 |  - | -
TransT [8] | CVPR’21 | 67.1 | 76.8 | 60.9 |  - |  81.4 | 86.7 | 80.3 | 64.9 | 73.8 |  69.0 |  - | -
Alpha-Refine [42] | CVPR’21 | - | - | - |  - |  80.5 | 85.6 | 78.3 | 65.3 | 73.2 |  68.0 |  - | -
TMT [37] | CVPR’21 | 67.1 | 77.7 | 58.3 |  - |  78.4 | 83.3 | 73.1 | 63.9 | - |  61.4 |  - | -
Ocean [48] | ECCV’20 | 61.1 | 72.1 | 47.3 |  50 |  - | - | - | 56.0 | 65.1 |  56.6 |  50 | -
DiMP [4] | ICCV’19 | 61.1 | 71.7 | 49.2 |  - |  74.0 | 80.1 | 68.7 | 56.9 | 65.0 |  56.7 |  - | -
ATOM [10] | CVPR’19 | - | - | - |  - |  70.3 | 77.1 | 64.8 | 51.5 | 57.6 |  50.5 |  - | -
SiamRPN++ [21] | CVPR’19 | 51.7 | 61.6 | 32.5 |  - |  73.3 | 80.0 | 69.4 | 49.6 | 56.9 |  49.1 |  - | -
|  |  |  |  |  |  |  |  |  |  |  |  |

Table 1: State-of-the-art comparison on GOT-10k, TrackingNet and LaSOT. The
best three results are shown in red ,blue and green fonts, respectively. We
use * to denote that the results on GOT-10k are obtained following the
official one-shot protocol. ** denotes the we calculate the effective number
of epochs for $6\times 10^{4}$ image-pairs sampled per epoch if the training
detail is provided. † indicates the tracker we replace the backbone with the
same CAE [7] pretrained ViT-Base model as our DETRack.

Tracker |  SiamRPN++[21] |  ATOM [10] |  DiMP [4] |  TransT [8] |  TMT [37] |  STARK [41] |  ToMP [29] |  OSTrack [43] | DETRack
---|---|---|---|---|---|---|---|---|---
NFS [19] | 50.2 | 59.0 | 62.0 | 65.7 | 66.5 | 65.2 | 66.9 | 64.7 | 65.3
UAV123 [31] | 61.3 | 65.0 | 65.4 | 69.1 | 67.5 | 69.1 | 69.0 | 68.3 | 68.7
#Epochs | - | - | - | - | - | 500 | 200 | 300 | 60
|  |  |  |  |  |  |  |  |

Table 2: Comparison with the state-of-the-art trackers on NFS and UAV123 in
terms of AUC score.

Sparsity | Head | Params(M) | FLOPs(G) | AO(%)
---|---|---|---|---
- | conv | 70.9 | 22.3 | 71.5
- | decoder |  68.9 ($\downarrow$ 2.0) |  21.4 ($\downarrow$ 0.9) | 72.9
CE | conv | 70.9 | 16.4 | 71.7
CE | decoder |  68.9 ($\downarrow$ 2.0) |  15.6 ($\downarrow$ 0.8) | 72.9
|  |  |  |

Table 3:  Comparison of params, FLOPs, and performance(Average Overlap) on
GOT-10k. The sparsified method CE indicates the Candidate Elimination [43].

###  4.1 Implemention Details

Our trackers are implemented using Python 3.8 and PyTorch 1.12. The models are
trained on 2 RTX 3090 GPUs. We test the model on a single RTX 2080Ti GPU.

Model. We adopt the vanilla ViT-Base [12] model and initialze it with CAE [7]
pre-trained weights on ImageNet [11] as the encoder of our DETRack. We leave
the weight of the decoder randomly initialized. The sizes of the template and
search images are 128$\times$128 pixels and 256$\times$256 pixels,
respectively. All the input images are split into 16$\times$16 patches. The
hidden dimension in the encoder and the decoder are 768 and 256. The dimension
of tokens output by the encoder are transformed by a linear layer. The bouding
box and its offset in each decoder layer is obtain by three-layer perceptrons,
which shares the parameter. The classification score is predicted by a simple
linear layer.

Training. For the GOT-10k [18] benchmark, we only use the training split of
GOT-10k following the one-shot protocols and train the model for 20 epochs.
For the other benchmarks, the training splits of GOT-10k, COCO [26], LaSOT
[13] and TrackingNet [32] are used for training in 60 epochs. For video
datasets, we sample the image pair from a random video sequence. For the image
dataset COCO, we randomly select an image and apply data augmentations to
generate an image pair. Common data augmentations such as scaling,
translation, and jittering are applied on the image pair. The search region
and the template are obtained by expanding the target box by a factor of 4 and
2, respectively. The optimizer is the AdamW optimizer [28], with the weight
decay of 1e-4. The initial learning rate of the encoder and the decoder are
4e-5 and 4e-4, respectively. We reduce the learning rate to 10% in the last
20% epochs. Each GPU holds 64 image-pairs, resulting a batch size of 128 in
total. Training takes $<$2 hours for GOT-10k and $<$6 hours for the other
benchmarks on two RTX 3090 GPUs. Note that the training time of the previous
most efficient transformer-based tracker, OSTrack, is about 8 hours, which is
4$\times$ that of our method.

Testing. During testing, we adopt Hanning window penalty to utilize positional
prior like scale change and motion smoothness in tracking, following the
common practice [2, 25, 43]. The output scores of each object queires by the
decoder are reflected back on the 2D map and simply element-wise multiplied by
the Hanning window with the same size, and we choose the box with the highest
multiplied score as the target box.

###  4.2 Comparison with State-of-the-art Trackers

We compare our DETRack with state-of-the-art(SOTA) trackers on 3 different
large-scale benchmarks and 2 small benchmarks, including GOT-10k, TrackingNet,
LaSOT, UAV123 and NFS. For fair comparison, the number of training epochs in
Tab. 1 and Tab. 2 of different trackers are counted with the same settings as
in [43], _e.g_., $6\times 10^{4}$ image pairs sampled in each epoch.

GOT-10k. GOT-10k [18] is a large-scale dataset containing more than 10000
video segments of real-world moving objects. The object classes between train
and test sets are zero-overlapped. We strictly follow the one-shot protocol to
only train our model on the GOT-10k training split and evaluate the results
through the evaluation server. As presetented in Tab. 1, DETRack improves all
matrics by a large margin, _e.g_., 1.9% in Average Overlap (AO) compared with
OSTrack which indicates the capability in accurate discrimination and
localization of unseen objects. Notice that our tracker is trained with only
20 epochs, which is 20% of the most training-efficient transformer-based
traker OSTrack.

TrackingNet. TrackingNet [32] is a large-scale short-term tracking benchmark
that provides more than 30000 video sequences with over 14 million boxes. The
test split of TrackingNet contains 511 sequences without publicly available
ground truth and covers diverse target classes andscenes. We submit the
tracking results to the official evaluation server and make comparisons with
previous SOTA trackers in Tab. 1. The results show that our DETRack achieves
83.2% in success score(AUC) and 83.1% in precision score, overtaking all
previously published trackers with the same backbone. It is notable that
training our DETRack is very easy: only 60 trained epochs, which is only 10%
to 20% of the previous SOTA trackers.

LaSOT. LaSOT [13] is a densely annotated large-scale dataset that contains 280
long-term video sequences for public evaluation. We evaluate our DETRack on
the test set to compare with previous SOTA trackers. From Tab. 1, we find that
our method acheives a comparable results, surpassing the OSTrack with the same
backbone in all three matrics. Specifically, DETRack achieves 69.0% AUC score
with merely 60 training epochs, which comsumes quite a few GPU hours compared
with the other SOTA methods, demonstrateing the efficiency of our approach.

NFS and UAV123. NFS [19] comprises 100 video sequences featuring fast-moving
objects, and is often used to test the robustness of tracking algorithms.
UAV123 [31], with its 123 video sequences captured from a low-altitude
unmanned aerial vehicle, poses challenges for long-term tracking due to its
average sequence length of 915 frames. As reported in Tab. 2, our method
consistently outperforms the baseline on both datasets. Moreover, our approach
requires significantly fewer training epochs, leading to reduced GPU training
time.

Params, GFLOPs and Speed. We provide the GFLOPs with state-of-the-art trackers
in Tab. 1. For more details about params, GFLOPs and speed, we compare our
DETRack with the baseline OSTrack [43] in Tab. 3 on a Nvidia RTX 2080Ti GPU.
We re-implemented the OSTrack by replacing the MAE [16] backbone with CAE [7]
for fair comparison. No matter with or without saprisified technique like
candidate elimination (CE) [43], our DETRack has a lower number of parameters
and FLOPs than the tracker using the same backbone but with a convolutional
head. Still, the implemention detials in engineering make our tracker run a
bit slower.

##  5 Ablation Study and Analysis

We analyze the main properties of the DETRack framework. For the following
experimental studies, we follow GOT-10k test protocol unless otherwise noted.

###  5.1 Analysis on the Number of Decoder Layers.

| AO(%) | GFLOPs
---|---|---
Ltest Ltrain | 2 | 3 | 4 |
4 | - | - | 72.5 | 15.9
3 | - | 72.9 | 72.3 | 15.6
2 | 71.4 | 72.1 | 71.7 | 15.3
1 | 71.2 | 71.4 | 71.1 | 15.1
|  |  |  |

Table 4: Results of the ablation study on the number of decoder layers. Ltrain
represents the number of decoder layers under training. Ltest means the number
of first n layers are used during the inference. The results of AO(%) means
the accuracy obtained by only using the first n layers of the decoder. The
experiment are reported on the GOT-10k test split following the one-shot
protocol with 20 epochs.

We investigate the influence of varying numbers of decoder layers. As shown in
Tab. 4, the decoder with 3 layers strikes the best trade-off between the
efficiency and accuracy. Decreasing the number of decoder layers hurts the
performance significantly, _e.g_., a two-layer decoder decrease the accuracy
of 1.5 AO% to a three-layer decoder. Adding layers in the decoder, _e.g_., a
four-layer decoder does not brings gains on performance but introducing
redundant cumputation in terms of GFLOPs.

Benefits from applying auxiliary loss on each layer of decoder and sharing the
prediction head among these layers, we can design a flexible configuration for
activating different decoder layers between training and testing. For
instance, we can obtain the intermediate prediction in the early layer of the
decoder as the final prediction, leaving the fianl layers un-performed. For
instance, only using first-two layers of a three-layer also achieves a
acceptable performance as 72.1 AO% as shown in Tab. 4. Furthermore, we can
only use the first layer in testing no matter how many layers we set for
training. The performance of all one-layer decoder configurations shown in
Tab. 4 surpass the convolutional baseline OSTrack(shown in the Tab. 1). The
results demostrate that our tracker can adapt flexibly between efficiency and
accuracy, even if the training is done.

###  5.2 Denoising Strategy

# | Positive | Negative | GOT-10k
---|---|---|---
AO | SR0.5 | SR0.75
baseline | - | - | 71.2 | 80.1 | 68.3
\footnotesize1⃝ | embedding | embedding | 71.3 | 80.3 | 68.2
\footnotesize2⃝ | center | corner | 72.9 | 82.1 | 69.9
\footnotesize3⃝ | center | outside | 72.4 | 81.9 | 69.9
|  |  |  |  |

Table 5: Comparison of different label assignment for denoising part in the
training. Baseline indicates the tracker trained without any auxiliary
denoising technique. The column Positve and Negative represents the source
feature for the positive and negative queries in the denoising part when
applying denoising training.

We also investigate the strategy for denoising training. The origional
denoising design for object detection (OD) adopts a embdeding layer to encode
the positive and negative labels [23]. The encoded classification label is
static to the input image because the classes are usually fixed in OD.
However, in single object tracking (SOT), the class of the target is generic
and dynamic. As shown in Tab. 5, the fixed embedding for classification label
does not help improving the performance too much over the baseline which is
trained without denoising, _e.g_., 71.4% AO vs. 71.2% AO in \footnotesize1⃝
and baseline. The slight increase of performance is come from the localization
part, i.e. , the bounding box denoising. As shown in \footnotesize2⃝ of Tab.
5, our proposed denoising strategy which the center feature is picked up as
the positive query and the corner ones are selected as negative, improving the
accuracy to 72.9% AO, leading the baseline by 1.7 point. The results also
demostrate that dynamic selected features as denoising quiries are more
suitable for SOT, compared with fixed and static label embeddings. Compared
with \footnotesize3⃝, which randomly pick the feature pixels outside the GT
bounding boxas as negative quiries, the corner features acting as hard
negative samples closer to the GT boxes are more helpful to improve the
performance, _e.g_., 72.9% vs. 72.4% AO.

###  5.3 Label Assignment

Method | GOT-10k | LaSOT
---|---|---
AO${}_{\text{20}}$ | AO${}_{\text{50}}$ | AO${}_{\text{100}}$ | AUC${}_{\text{60}}$ | AUC${}_{\text{100}}$ | AUC${}_{\text{300}}$
\footnotesize1⃝center | 55.8 | 65.4 | 71.2 | 43.0 | 52.3 | 68.8
\footnotesize2⃝Hungarian | 51.0 | 59.1 | 65.2 | 35.6 | 47.9 | 66.4
\footnotesize3⃝hard label | 65.7 | 65.4 | 66.3 | 62.1 | 63.4 | 62.9
\footnotesize4⃝loc. quality | 72.9 | 72.9 | 72.4 | 69.0 | 68.6 | 68.9
|  |  |  |  |  |

Table 6: Comparison with different label assignment (for un-denoising part).
AO${}_{\text{k}}$ and AUC${}_{\text{k}}$ denotes for the performance obtained
by training k epochs for the benchmark.

Incorporating the decoder lets us set the training objective for
classification, akin to convolution-based heads, such as centerness. From Tab.
6, it’s clear that one-to-one assignments, represented by centerness
\footnotesize1⃝ and bipartite matching \footnotesize2⃝, require a large number
of training epochs to be effective, _e.g_., 55.8% AO at 20 epoch to 71.2% AO
at 100 epoch in \footnotesize1⃝ on GOT-10k. In contrast, one-to-many
assignments (\footnotesize3⃝ and \footnotesize4⃝) notably decrease the epochs
required to reach peak performance. Using \footnotesize3⃝ to allocate each
positive sample with a rigid label results in sub-optimal performance.
However, emphasizing localization quality, as represented by IoU, considerably
enhances both convergence rate and performance. For instance, performance
obtained in the early stage like 72.9% AO at 20 epoch on GOT-10k and 69.0% AUC
at 60 epoch on LaSOT are higher than that with more epochs.

###  5.4 Convergence

Figure 5:  The label assignment for the denoising training. The most center
feature within the GT bounding box is picked as the positive object query in
one denoising group, while the corner ones are selected as negative queries.

We provide the training IoU vs. epoch curve in the Fig. 5. All results are
reported on 2 RTX 3090 GPUs with ViT-B intialized from CAE [7]. As we can see,
our proposed DETRack achieve the similar training IoU in 20 epochs with the
OSTrack in 100 epochs. The combined results from Tab. 6 and Fig. 5 also
demostrate a interesting fact: the more accurate localization ability in the
template-search image pair during training does not mean a higher overall
tracking performance over a video sequence in testing.

##  6 Conclusion

In this research, we unveiled DETRack, an innovative encoder-decoder framework
that leverages the deformable transformer decoder to supersede the
conventional convolutional head, paving the way for a more pronounced sparsity
and consequent reduction in GFLOPs. Through our novel implementation of the
one-to-many label assignment and unique denoising technique during training,
DETRack improves tracking accuracy with a significantly reduced training
epochs. The marked reductions in GPU hours for traiing are especially
beneficial for researchers with limited computational resources, potentially
democratizing access to high-quality visual object tracking.

Limitation. Although our work achieves comparable accuracy on per-frame
classification and localization with little GPU resource consuming, the
overall performance on some long-term seqeunce tracking benchmark is limited.
Besides, even if our tracker runs with lower FLOPs and params than the
convolution-based ones with the same or weaker performance, the actual running
speed of ours, _e.g_., FPS, is a little lower than the latters due to the
engineering implementation.

## References

  * [1] Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Torr.  Fully-convolutional siamese networks for object tracking.  In European conference on computer vision, pages 850–865. Springer, 2016.
  * [2] Luca Bertinetto, Jack Valmadre, Joao F Henriques, Andrea Vedaldi, and Philip HS Torr.  Fully-convolutional siamese networks for object tracking.  In Proc. of ECCV, 2016.
  * [3] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte.  Learning discriminative model prediction for tracking.  In Proceedings of the IEEE/CVF international conference on computer vision, pages 6182–6191, 2019.
  * [4] Goutam Bhat, Martin Danelljan, Luc Van Gool, and Radu Timofte.  Learning discriminative model prediction for tracking.  In Proc. of ICCV, 2019.
  * [5] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In European conference on computer vision, pages 213–229. Springer, 2020.
  * [6] Boyu Chen, Peixia Li, Lei Bai, Lei Qiao, Qiuhong Shen, Bo Li, Weihao Gan, Wei Wu, and Wanli Ouyang.  Backbone is all your need: A simplified architecture for visual object tracking.  arXiv preprint arXiv:2203.05328, 2022.
  * [7] Xiaokang Chen, Mingyu Ding, Xiaodi Wang, Ying Xin, Shentong Mo, Yunhao Wang, Shumin Han, Ping Luo, Gang Zeng, and Jingdong Wang.  Context autoencoder for self-supervised representation learning.  arXiv preprint arXiv:2202.03026, 2022.
  * [8] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu.  Transformer tracking.  In Proc. of CVPR, 2021.
  * [9] Yutao Cui, Cheng Jiang, Limin Wang, and Gangshan Wu.  Mixformer: End-to-end tracking with iterative mixed attention.  In Proc. of CVPR, 2022.
  * [10] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg.  Atom: Accurate tracking by overlap maximization.  In Proc. of CVPR, 2019.
  * [11] J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei.  Imagenet: A large-scale hierarchical image database, 2009.
  * [12] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al.  An image is worth 16x16 words: Transformers for image recognition at scale.  arXiv preprint arXiv:2010.11929, 2020.
  * [13] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling.  Lasot: A high-quality benchmark for large-scale single object tracking.  In Proc. of CVPR, 2019.
  * [14] Shenyuan Gao, Chunluan Zhou, Chao Ma, Xinggang Wang, and Junsong Yuan.  Aiatrack: Attention in attention for transformer visual tracking.  In Proc. of ECCV, 2022.
  * [15] Mingzhe Guo, Zhipeng Zhang, Heng Fan, Liping Jing, Yilin Lyu, Bing Li, and Weiming Hu.  Learning target-aware representation for visual tracking via informative interactions.  arXiv preprint arXiv:2201.02526, 2022.
  * [16] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick.  Masked autoencoders are scalable vision learners.  In Proc. of CVPR, 2022.
  * [17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [18] Lianghua Huang, Xin Zhao, and Kaiqi Huang.  Got-10k: A large high-diversity benchmark for generic object tracking in the wild.  IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019.
  * [19] Hamed Kiani Galoogahi, Ashton Fagg, Chen Huang, Deva Ramanan, and Simon Lucey.  Need for speed: A benchmark for higher frame rate object tracking.  In Proceedings of the IEEE International Conference on Computer Vision, pages 1125–1134, 2017.
  * [20] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan.  Siamrpn++: Evolution of siamese visual tracking with very deep networks.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4282–4291, 2019.
  * [21] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan.  Siamrpn++: Evolution of siamese visual tracking with very deep networks.  In Proc. of CVPR, 2019.
  * [22] Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu.  High performance visual tracking with siamese region proposal network.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 8971–8980, 2018.
  * [23] Feng Li, Hao Zhang, Shilong Liu, Jian Guo, Lionel M Ni, and Lei Zhang.  Dn-detr: Accelerate detr training by introducing query denoising.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13619–13627, 2022.
  * [24] Xiang Li, Wenhai Wang, Lijun Wu, Shuo Chen, Xiaolin Hu, Jun Li, Jinhui Tang, and Jian Yang.  Generalized focal loss: Learning qualified and distributed bounding boxes for dense object detection.  Advances in Neural Information Processing Systems, 33:21002–21012, 2020.
  * [25] Liting Lin, Heng Fan, Yong Xu, and Haibin Ling.  Swintrack: A simple and strong baseline for transformer tracking.  arXiv preprint arXiv:2112.00995, 2021.
  * [26] Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick.  Microsoft coco: Common objects in context.  In Proc. of ECCV, 2014.
  * [27] Shilong Liu, Feng Li, Hao Zhang, Xiao Yang, Xianbiao Qi, Hang Su, Jun Zhu, and Lei Zhang.  Dab-detr: Dynamic anchor boxes are better queries for detr.  arXiv preprint arXiv:2201.12329, 2022.
  * [28] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [29] Christoph Mayer, Martin Danelljan, Goutam Bhat, Matthieu Paul, Danda Pani Paudel, Fisher Yu, and Luc Van Gool.  Transforming model prediction for tracking.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8731–8740, 2022.
  * [30] Christoph Mayer, Martin Danelljan, Danda Pani Paudel, and Luc Van Gool.  Learning target candidate association to keep track of what not to track.  In Proc. of ICCV, 2021.
  * [31] Matthias Mueller, Neil Smith, and Bernard Ghanem.  A benchmark and simulator for uav tracking.  In European conference on computer vision, pages 445–461. Springer, 2016.
  * [32] Matthias Muller, Adel Bibi, Silvio Giancola, Salman Alsubaihi, and Bernard Ghanem.  Trackingnet: A large-scale dataset and benchmark for object tracking in the wild.  In Proc. of ECCV, 2018.
  * [33] Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese.  Generalized intersection over union: A metric and a loss for bounding box regression.  In Proc. of CVPR, 2019.
  * [34] Zikai Song, Junqing Yu, Yi-Ping Phoebe Chen, and Wei Yang.  Transformer tracking with cyclic shifting window attention.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8791–8800, 2022.
  * [35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Proc. of NeurIPS, 2017.
  * [36] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  Advances in neural information processing systems, 30, 2017.
  * [37] Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li.  Transformer meets tracker: Exploiting temporal context for robust visual tracking.  In CVPR, pages 1571–1580, 2021.
  * [38] Fei Xie, Chunyu Wang, Guangting Wang, Yue Cao, Wankou Yang, and Wenjun Zeng.  Correlation-aware deep tracking.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 8751–8760, 2022.
  * [39] Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu.  Simmim: A simple framework for masked image modeling.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9653–9663, 2022.
  * [40] Bin Yan, Yi Jiang, Peize Sun, Dong Wang, Zehuan Yuan, Ping Luo, and Huchuan Lu.  Towards grand unification of object tracking.  In ECCV, pages 733–751, 2022.
  * [41] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu.  Learning spatio-temporal transformer for visual tracking.  In Proc. of ICCV, 2021.
  * [42] Bin Yan, Xinyu Zhang, Dong Wang, Huchuan Lu, and Xiaoyun Yang.  Alpha-refine: Boosting tracking performance by precise bounding box estimation.  In CVPR, pages 5289–5298, 2021.
  * [43] Botao Ye, Hong Chang, Bingpeng Ma, Shiguang Shan, and Xilin Chen.  Joint feature learning and relation modeling for tracking: A one-stream framework.  In Proc. of ECCV, 2022.
  * [44] Yuechen Yu, Yilei Xiong, Weilin Huang, and Matthew R Scott.  Deformable siamese attention networks for visual object tracking.  In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6728–6737, 2020.
  * [45] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M Ni, and Heung-Yeung Shum.  Dino: Detr with improved denoising anchor boxes for end-to-end object detection.  arXiv preprint arXiv:2203.03605, 2022.
  * [46] Zhipeng Zhang, Yihao Liu, Xiao Wang, Bing Li, and Weiming Hu.  Learn to match: Automatic matching network design for visual tracking.  In Proc. of ICCV, 2021.
  * [47] Zhipeng Zhang and Houwen Peng.  Deeper and wider siamese networks for real-time visual tracking.  In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 4591–4600, 2019.
  * [48] Zhipeng Zhang, Houwen Peng, Jianlong Fu, Bing Li, and Weiming Hu.  Ocean: Object-aware anchor-free tracking.  In Proc. of ECCV, 2020.
  * [49] Xizhou Zhu, Weijie Su, Lewei Lu, Bin Li, Xiaogang Wang, and Jifeng Dai.  Deformable detr: Deformable transformers for end-to-end object detection.  arXiv preprint arXiv:2010.04159, 2020.


# BEVERS: A General, Simple, and Performant Framework
for Automatic Fact Verification

Mitchell DeHaven
Information Sciences Institute
University of Southern California
<EMAIL_ADDRESS>
Stephen Scott
University of Nebraska-Lincoln
<EMAIL_ADDRESS>

###### Abstract

Automatic fact verification has become an increasingly popular topic in recent
years and among datasets the Fact Extraction and VERification (FEVER) dataset
is one of the most popular. In this work we present BEVERS, a tuned baseline
system for the FEVER dataset. Our pipeline uses standard approaches for
document retrieval, sentence selection, and final claim classification,
however, we spend considerable effort ensuring optimal performance for each
component. The results are that BEVERS achieves the highest FEVER score and
label accuracy among all systems, published or unpublished. We also apply this
pipeline to another fact verification dataset, Scifact, and achieve the
highest label accuracy among all systems on that dataset as well. We also make
our full code available111https://github.com/mitchelldehaven/bevers.

##  1 Introduction

The danger of misinformation online has gained significant attention in recent
years. This has been reignited by the recent COVID-19 pandemic, where social
media sites and other entities were tasked with identifying misleading content
or false content to warn users. Being able to develop systems to automate or
build tools to improve this process could reduce the need for human annotators
to mark content as being misleading or false.

The Fact Extraction and VERification (FEVER) dataset Thorne et al. (2018) is
one the largest and most popular datasets aimed at automated fact
verification. The FEVER dataset is comprised of 185,445 claims and uses a 2017
dump of Wikipedia as the corpus to verify the claims, which results in a
corpus size of over 5,000,000 articles. For each claim, the task is to find
the relevant Wikipedia page(s), the relevant sentence(s) within the page(s),
and finally given the relevant sentences and claim determine if the claim is
supported, refuted, or there is not enough information. As such, a fairly
standard pipeline of a document retrieval system, a sentence selection system,
and a final claim classification system is used by most of the systems for the
task. The primary metric for the dataset is FEVER score. The FEVER score
requires both that the predicted label is correct as well as at least one
piece of correct evidence being retrieved as predicted evidence.

Much of the recent work has examined parts of the pipeline and made novel
improvements over baseline approaches. For our system, rather than making
novel improvements against the baseline pipeline, we instead tune each of
these components to ensure maximum performance. In fact, our pipeline is quite
similar to one of the first FEVER systems to utilize Transformer models
Soleimani et al. (2020). We call our system Baseline fact Extraction and
VERification System (BEVERS). Despite its relative simplicity, our system
attains state of the art (SOTA) performance on the FEVER blind test set. When
applying our baseline pipeline to another popular fact verification dataset,
Scifact Wadden et al. (2020), our system achieves the highest label F1 score
on that dataset as well.

##  2 Related Work and Methods

###  2.1 Document Retrieval

The initial baseline for FEVER Thorne et al. (2018) utilized a standard TF-IDF
document retrieval model. Hanselowski et al. (2018) improved on this by using
named entity recognition (NER) to extract query terms from the claim text and
query those terms against WikiMedia’s
API222https://www.mediawiki.org/wiki/API:Main_page, which has become widely
used among other systems. Recently systems such as those from Stammbach (2021)
and Jiang et al. (2021) have used a combination of traditional IR approaches
with Hanselowski et al.’s (2018) NER approach. We follow a similar setup,
however, we replace the approach of Hanselowski et al.’s (2018) use of
WikiMedia’s API. We similarly extract named entities to form query terms,
however, we run those against a fuzzy string search system using the titles of
the documents. For our TF-IDF, we build separate representations for documents
and titles. This is for two reasons. First, it allows us to separately
optimize the parameters for titles and documents. Second, it forces the
retrieval system to give titles more attention as it is forced to retrieve
half of all documents based on the title alone. We give an ablation over these
design decisions in Appendix B.

###  2.2 Sentence Selection

After retrieving documents, the next step is to score evidence and form a
ranking for the predicted evidence of the claim. The simplest approach to
doing this is referred to as “point-wise” ranking, in which each sentence is
scored individually against the claim. This is the approach utilized by most
systems. Soleimani et al. (2020) looked at improving on this utilizing a
pairwise approach to ranking. Stammbach (2021) found that utilizing document-
wide context via sparse attention Transformers improves on point-wise
approaches. Our system utilizes a simple point-wise approach to sentence
selection to form the predicted evidence. We look at two cases, treating the
task as both a binary classification task and a ternary classification task.
In the binary case, the label set is simply RELEVANT and IRRELEVANT with the
softmax score of RELEVANT being used for ranking. In the ternary case, we use
REFUTES, NOT ENOUGH INFO, and SUPPORTS as the labels and use $1-$ NOT ENOUGH
INFO softmax score for ranking. We randomly sample sentences from the document
retrieved from our document retrieval approach for negative samples. In the
binary case, these random negative samples are assigned to the IRRELEVANT
label class and all true evidence is assigned to RELEVANT. In the ternary
case, the negative samples are assigned to NOT ENOUGH INFO, and the true
evidence is assigned to its respective labels, REFUTES and SUPPORTS.

In addition, we utilize a process we call evidence-based re-retrieval. The
FEVER dataset includes hyperlink information for each sentence in the dataset.
This process takes the initial set of predicted evidence for a claim and
extracts additional documents based on hyperlinks found in the initial
sentences retrieved. Sentences from these additional documents are scored and
combined with the initial sentences to form a final top 5 predicted evidence.
This process is very similar to Stammbach’s (2021) “multi-hop retrieval”, with
slight differences in how sentences are discounted when combining the two sets
of sentences. Stammbach sets evidence from re-retrieved documents just above a
predefined threshold for selection to prevent re-retrieved evidence from
pushing evidence from the initial retrieval outside of the top 5. We similarly
found that simply combining both sets together actually hurts recall, because
evidence from re-retrieval sometimes pushes out relevant evidence from the
initial retrieval. In our approach, we scale the sentence selection scores of
the re-retrieved sentences by the score of the original sentence that was
responsible for its retrieval. Thus, if evidence $s_{j}$ was retrieved due to
a hyperlink in $s_{i}$ the final retrieval score is $score(s_{i})\times
score(s_{j})$. Scaling this way reduces re-retrieved evidence pushing evidence
from the initial retrieval from the top-5 selection. It also allows re-
retrieved evidence scores to be proportional to the score of the initial
evidence responsible for its retrieval.

###  2.3 Claim Classification

The claim classification portion has recently seen the most diversity in
approaches to the task. The initial Transformer approach of Soleimani et al.
(2020) formed predictions for each claim and evidence pair, using a simple set
of rules to aggregate labels across the different pieces of evidence.
Subsequently, several works examined the use of graph neural networks as the
claim classification model Liu et al. (2020); Zhong et al. (2020), showing
improvements over simply using Transformers due to their ability to aggregate
information over different pieces of evidence. More recently, increasing the
size of the Transformer models and concatenating all evidence sentences
together have shown further improvements, with Jiang et al. (2021) using T5
Raffel et al. (2020) and Stammbach (2021) using DeBERTa V2 XL MNLI He et al.
(2021). Finally, the previous SOTA among public systems, ProofVER Krishna et
al. (2022), utilizes natural language proofs generated via seq2seq models for
interpretable inference steps.

For our approach, we look at prediction over singleton, concatenated, and a
mixed case. We predict a top-5 evidence set for each claim for training using
our document selection and sentence selection. In the singleton case, we
generate a prediction for each piece of evidence using as input the
$\langle$claim, evidence$\rangle$ pairs. In the concatenated case, we
concatenate all the evidence together and form the input based on
$\langle$claim, evidence1, evidence${}_{2},\ldots\rangle$. For the mixed
approach, we mix the singleton approach and concatenated approach together.
For the singleton and mixed approach, we have multiple predictions for each
claim. To aggregate these into a single score, we use the softmax scores for
each prediction with the retrieval scores and train a gradient boosting
classifier Friedman (2001) on these inputs to produce a single prediction. In
the singleton case, the input is a $5\times 4$ matrix (5 pieces of evidence, 3
softmax scores and a retrieval score). In the mixed case, the input is a
$6\times 4$ matrix (includes the additional concatenated input softmax scores
and the retrieval score, computed from the average retrieval scores of the 5
pieces of evidence). The singleton and concatenated approaches have been used
previously Soleimani et al. (2020); Jiang et al. (2021), while we are not
aware of any works that look at simply mixing these approaches together.

##  3 Experimental Setup

Hyperparameter | Values
---|---
Force Lowercase | True, False
Force ASCII | True, False
Norm | L2, None
Sublinear TF | True, False
Max Ngram | 1, 2
Table 1: The hyperparameter search for our TF-IDF system. Hyperparameter | Values
---|---
Label Set | binary, ternary
Negative Samples | 5, 10, 20, 40
Learning Rate | 1e-5, 6e-6, 3e-6
Label Smoothing | 0.0, 0.1, 0.2
Table 2: The hyperparameter search our sentence selection model.

What we believe to be the source of improvements for our system is
hyperparameter tuning for each component. We identify hyperparameters and
potential values and run a grid search to find the optimal configurations for
each component. In this section, we will go over each of the grid searches
providing additional details on the exact setup.

For our TF-IDF system, we utilize SciKit Learn’s Pedregosa et al. (2011) TF-
IDF representation. In Table 1 we list the hyperparameters and their candidate
values used in the grid search. We use recall @ 5 on the development set for
finding the best configuration. The fuzzy string search is implemented using
Sqlite’s spellfix1 virtual table333https://www.sqlite.org/spellfix1.html. We
set a simple edit distance threshold for retrieving additional documents.

Our sentence selection hyperparameter tuning is split into two sections.
First, we optimize the number of negative samples selected as well as binary
vs ternary classes for ranking. Since the FEVER dataset does not provide
evidence for NOT ENOUGH INFO claims, negative samples must be used to generate
training examples for these. Using the best selection from the initial setup,
we tune the learning rate and label smoothing. The candidate values for the
tuning can be found in Table 2. Given the imbalance in the training set and
the balanced nature of the dev and test set, we oversample the minority
classes so that label distribution in the training set matches that of the dev
and test sets. We use the dev set for determining optimal hyperparameter
values. RoBERTa Large Liu et al. (2019) is used as the initial model for fine-
tuning.

The claim classification tuning setup is quite similar to sentence selection.
We initially tune the learning rate and label smoothing using the same
candidate values for the concatenated case. Instead of tuning the model types
of singleton, concatenated, and mixed, we simply use the best hyperparameter
configuration and train a model for each of these to draw final comparisons.
Again, given the imbalance in classes in the train set, we use class weighting
to compensate for this imbalance. For fine-tuning we use RoBERTa Large MNLI
and DeBERTa V2 XL MNLI.

Hyperparameter | Values
---|---
Learning Rate | 0.1, 0.3
Estimators | 20, 40, 60, 80, 100
Max Depth | 2, 4, 6, 8
Table 3: The hyperparameter search our gradient boosting model. System | Test LA | Test FEVER
---|---|---
Soleimani et al. (2020) | 71.86% | 69.66%
KGAT Liu et al. (2020) | 74.07% | 70.38%
LisT5 Jiang et al. (2021) | 79.35% | 75.87%
Stammbach (2021) | 79.20% | 76.80%
ProoFVer Krishna et al. (2022) | 79.47 % | 76.82%
Ours (RoBERTa Large MNLI) singleton | 78.01% | 76.09%
Ours (RoBERTa Large MNLI) concatenated | 79.14% | 76.69%
Ours (RoBERTa Large MNLI) mixed | 79.39% | 76.89%
Ours (DeBERTa V2 XL MNLI) mixed |  80.24% |  77.70%
Table 4: Full system comparison for label accuracy (LA) and FEVER score on the
blind test set.

Finally, for the singleton and mixed approaches, we use XGBoost Chen and
Guestrin (2016) for training a classifier to aggregate the predictions into a
single prediction. Similarly, we define a hyperparameter grid to find the
optimal values. Since the previous steps were all trained on the train set and
thus the softmax scores and retrieval scores will be overly optimistic on the
training set, we instead train the XGBoost classifier on the dev set. We use
4-fold cross-validation to find the optimal configuration.

##  4 Results

System | Dev Recall @ 5
---|---
Hanselowski et al. (2018) | 87.10%
Liu et al. (2020) | 94.37%
Soleimani et al. (2020) | 88.38%
Jiang et al. (2021) | 90.54%
Stammbach (2021) | 93.62%
Ours | 92.03%
\+ re-retrieval |  94.41%
Table 5: The results of several sentence selection systems in terms of recall
@ 5 on the dev set.

For sentence selection, the primary metric used is recall @ 5. This is due to
the fact that when computing FEVER score, the scoring metric will only
consider up to 5 pieces of predicted evidence. In Table 5 we compare our
sentence selection system against several other top systems on the dev set. As
can be seen, our sentence selection system outperforms all previous systems in
terms of recall @ 5 on the dev set. This is despite using a substantially
smaller model relative to Jiang et al.’s (2021) T5 approach as well as only
using pointwise scoring for sentence selection as opposed to Stammbach’s
(2021) full document context approach. We separate our results from using
initial retrieval and including evidence-based re-retrieval, which shows a
very large improvement in recall by doing re-retrieval, consistent with
Stammbach’s (2021) findings.

For claim classification results, we present the entire end-to-end results for
our system in Table 4. The simple approach of mixing the singleton and
concatenate approaches gives a small improvement, although is not a
substantial source of improvement. Despite the singleton approach being
incapable of modeling claims that require multi-hop evidence, it still
performs well. Despite using a relatively smaller model of 300 million
parameters when compared to 3 billion with T5 and 900 million with DeBERTa V2
XL MNLI, our RoBERT Large MNLI system achieves the highest FEVER score among
all published systems. When we utilize DeBERTa V2 XL MNLI using our mixed
approach, we achieve the highest label accuracy and FEVER score amongst all
systems, published or unpublished, on the blind test set.

##  5 Beyond FEVER: Scifact

System | SS + L | Abstract LO
---|---|---
Pradeep et al. (2021) | 58.8 | 64.9
Zhang et al. (2021) | 63.1 | 68.1
Wadden et al. (2022) | 67.2 | 72.5
Ours | 58.1 | 73.2
Table 6: System comparison for SS + L F1 score and Abstract LO F1 score on
SciFact blind test set.

To test this pipeline for automatic fact verification beyond the FEVER
dataset, we also apply these methods to the SciFact dataset Wadden et al.
(2020). SciFact is very similar in structure to the FEVER dataset, however,
the corpus is composed of scientific articles. A source of difficulty is that
claims are often phrased in lay terms, which can be a stark difference in form
from how topics are presented in scientific articles. The overall size of the
dataset is quite a bit smaller as well, containing only 1,409 claims and 5,183
article abstracts, which serve as the corpus. Despite this, we keep our
pipeline nearly identical to FEVER, excluding only the fuzzy string search
component. We follow the approach of Wadden et al. (2022) for improving the
initial models for finetuning given the low resource nature of the dataset.

We show the results of our pipeline in Table 6 compared to the current SOTA
Wadden et al. (2022) and other top systems. The metrics reported are sentence
selection + label (SS + L) and abstract label only (Abstract LO). These
metrics roughly correspond to FEVER Score and label accuracy for FEVER. As can
be seen in the SS + L metric, the simplicity of our document retrieval system
appears to hold the overall system back. Our system only uses TF-IDF whereas
the three others add neural re-rankers on top of their retrieval. Despite
this, on the Abstract LO metric our system achieves the highest F1 score on
the blind test set, outperforming the SOTA on this metric.

##  6 Conclusion

We presented BEVERS, a strong baseline approach for the FEVER and SciFact
datasets. Despite being similar to previous works in structure Soleimani et
al. (2020) and utilizing little in terms of novel improvements, our system was
able to achieve SOTA performance on FEVER and the highest label accuracy on
SciFact. We primarily attribute these improvements to diligent hyperparameter
tuning and error analysis. While several previous works have shown novel
contributions to portions of the pipeline can yield improvements, in this work
we show a well-tuned baseline is very strong.

##  7 Limitations

As shown with SciFact, this pipeline struggles in situations where there is a
mismatch in how claims are phrased and how evidence is phrased in the corpus.
Since our retrieval method is term-based, synonymous terms are often missed,
and thus in such systems utilizing neural retrieval methods will offer better
performance. In addition, this work does not thoroughly examine which design
decisions or approaches led to the improvements seen in this pipeline. We note
that evidence-based re-retrieval does give substantial improvements, yet even
without re-retrieval, our sentence selection outperforms most previous systems
by a substantial margin, so it is not the sole source of improvement.

## References

  * Chen and Guestrin (2016) Tianqi Chen and Carlos Guestrin. 2016.  XGBoost: A scalable tree boosting system.  In _Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ , KDD ’16, pages 785–794, New York, NY, USA. ACM.
  * Friedman (2001) Jerome H. Friedman. 2001.  Greedy function approximation: A gradient boosting machine. _The Annals of Statistics_ , 29(5):1189 – 1232.
  * Hanselowski et al. (2018) Andreas Hanselowski, Hao Zhang, Zile Li, Daniil Sorokin, Benjamin Schiller, Claudia Schulz, and Iryna Gurevych. 2018.  UKP-athene: Multi-sentence textual entailment for claim verification.  In _Proceedings of the First Workshop on Fact Extraction and VERification (FEVER)_ , pages 103–108, Brussels, Belgium. Association for Computational Linguistics.
  * He et al. (2021) Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021.  Deberta: Decoding-enhanced bert with disentangled attention.  In _International Conference on Learning Representations_.
  * Jiang et al. (2021) Kelvin Jiang, Ronak Pradeep, and Jimmy Lin. 2021.  Exploring listwise evidence reasoning with t5 for fact verification.  In _Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers)_ , pages 402–410, Online. Association for Computational Linguistics.
  * Krishna et al. (2022) Amrith Krishna, Sebastian Riedel, and Andreas Vlachos. 2022.  ProoFVer: Natural logic theorem proving for fact verification.  _Transactions of the Association for Computational Linguistics_ , 10:1013–1030.
  * Liu et al. (2019) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019.  Roberta: A robustly optimized bert pretraining approach.
  * Liu et al. (2020) Zhenghao Liu, Chenyan Xiong, Maosong Sun, and Zhiyuan Liu. 2020.  Fine-grained fact verification with kernel graph attention network.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 7342–7351, Online. Association for Computational Linguistics.
  * Pedregosa et al. (2011) F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay. 2011.  Scikit-learn: Machine learning in Python.  _Journal of Machine Learning Research_ , 12:2825–2830.
  * Pradeep et al. (2021) Ronak Pradeep, Xueguang Ma, Rodrigo Nogueira, and Jimmy Lin. 2021.  Scientific claim verification with VerT5erini.  In _Proceedings of the 12th International Workshop on Health Text Mining and Information Analysis_ , pages 94–103, online. Association for Computational Linguistics.
  * Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020.  Exploring the limits of transfer learning with a unified text-to-text transformer.  _J. Mach. Learn. Res._ , 21(1).
  * Soleimani et al. (2020) Amir Soleimani, Christof Monz, and Marcel Worring. 2020.  Bert for evidence retrieval and claim verification.  In _Advances in Information Retrieval: 42nd European Conference on IR Research, ECIR 2020, Lisbon, Portugal, April 14–17, 2020, Proceedings, Part II_ , page 359–366, Berlin, Heidelberg. Springer-Verlag.
  * Stammbach (2021) Dominik Stammbach. 2021.  Evidence selection as a token-level prediction task.  In _Proceedings of the Fourth Workshop on Fact Extraction and VERification (FEVER)_ , pages 14–20, Dominican Republic. Association for Computational Linguistics.
  * Thorne et al. (2018) James Thorne, Andreas Vlachos, Christos Christodoulopoulos, and Arpit Mittal. 2018\.  FEVER: a large-scale dataset for fact extraction and VERification.  In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_ , pages 809–819, New Orleans, Louisiana. Association for Computational Linguistics.
  * Wadden et al. (2020) David Wadden, Shanchuan Lin, Kyle Lo, Lucy Lu Wang, Madeleine van Zuylen, Arman Cohan, and Hannaneh Hajishirzi. 2020.  Fact or fiction: Verifying scientific claims.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7534–7550, Online. Association for Computational Linguistics.
  * Wadden et al. (2022) David Wadden, Kyle Lo, Lucy Wang, Arman Cohan, Iz Beltagy, and Hannaneh Hajishirzi. 2022.  MultiVerS: Improving scientific claim verification with weak supervision and full-document context.  In _Findings of the Association for Computational Linguistics: NAACL 2022_ , pages 61–76, Seattle, United States. Association for Computational Linguistics.
  * Ye et al. (2020) Deming Ye, Yankai Lin, Jiaju Du, Zhenghao Liu, Peng Li, Maosong Sun, and Zhiyuan Liu. 2020.  Coreferential Reasoning Learning for Language Representation.  In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_ , pages 7170–7186, Online. Association for Computational Linguistics.
  * Zhang et al. (2021) Zhiwei Zhang, Jiyi Li, Fumiyo Fukumoto, and Yanming Ye. 2021.  Abstract, rationale, stance: A joint model for scientific claim verification.  In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_ , pages 3580–3586, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
  * Zhong et al. (2020) Wanjun Zhong, Jingjing Xu, Duyu Tang, Zenan Xu, Nan Duan, Ming Zhou, Jiahai Wang, and Jian Yin. 2020.  Reasoning over semantic-level graph for fact checking.  In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_ , pages 6170–6180, Online. Association for Computational Linguistics.

##  Appendix A Optimal Hyperparameter Settings

In Table 7 we show the optimal hyperparameter settings for the various TF-IDF
configurations. To minimize space, we use "Cat" to refer to the concatenated
TF-IDF setup. In Table 8 and Table 9 we show the optimal hyperparameter values
for sentence selection and claim classification models. Finally, in Table 10
we include the optimal hyperparameter values for the XGBoost classifier.

Hyperparameter | Cat |  Title, Document
---|---|---
Force Lowercase | True | True, False
Force ASCII | True | True, True
Norm | None | L2, None
Sublinear TF | True | True, True
Max Ngram | 2 | 2, 2
Table 7: Optimal hyperparameters for the concatenated and separated TF-IDF configurations. Hyperparameter | Optimal Value
---|---
Label Set | Ternary
Negative Samples | 10
Learning Rate | 3e-6
Label Smoothing | 0.0
Table 8: Optimal hyperparameters for sentence selection model. Hyperparameter | Optimal Value
---|---
Learning Rate | 3e-6
Label Smoothing | 0.2
Table 9: Optimal hyperparameters for claim classification model. Hyperparameter | Optimal Value
---|---
Max Depth | 2
Number of Estimators | 60
Learning Rate | 0.3
Table 10: Optimal hyperparameters for XGBoost aggregation classifier.

##  Appendix B Ablation Studies

In Table 11 we show the impacts of various design choices for document
retrieval and their impacts on sentence selection. We use our best sentence
selection model for ranking the sentences retrieved by the document retrieval
approaches. Previous works use OFEVER from the original paper as a metric for
comparing document retrieval methods, however, OFEVER does not account for
different approaches retrieving different numbers of documents given that is
an oracle approach. Thus, we find measuring the sentence selection in this way
gives a better representation of improvements.

Retrieval Approach | Dev Recall $@$ 5
---|---
TF-IDF (concatenated) | 84.49 %
\+ fuzzy string search | 91.35 %
\+ document re-retrieval | 93.58 %
TF-IDF (separated) | 87.09 %
\+ fuzzy string search | 92.03 %
\+ document re-retrieval |  94.41%
Table 11: Dev set recall $@$ 5 using various document retrieval approaches. Author (Model) | Test LA | Test FEVER
---|---|---
KGAT (RoBERTa Large) Liu et al. (2020) | 74.07 % | 70.38 %
KGAT (CorefRoBERTa) Ye et al. (2020) | 75.96 % | 72.30 %
Ours (RoBERTa Large) | 76.60 % | 73.21 %
Ours (RoBERTa Large MNLI) | 77.95 % | 74.08 %
Table 12: Comparison between KGAT’s claim classification and ours. We use
KGAT’s released outputs for evidence retrieval, so differences in performance
are not attributable to improvements in our system’s retrieval approach.

In Table 12 we compare our claim classification setup with KGAT’s. Rather than
utilizing our document retrieval and sentence selection, we use KGAT’s
sentence selection outputs which they make publicly available. This allows for
a more direct comparison since we are using the same evidence for forming
predictions. The only changes we make: re-score the top 5 evidence from KGAT’s
sentence selection using our own best sentence selection model and re-train
the gradient boosting classifier. Despite using the same evidence as KGAT, our
claim classification still outperforms using either RoBERTa Large or RoBERTa
Large MNLI. So while some of the improvement in our system is attributable to
improvements in document retrieval and sentence selection our approach to
claim classification still outperforms previous systems when using the same
retrieval outputs.