Alan<EMAIL_ADDRESS>Žiga<EMAIL_ADDRESS>Jiří<EMAIL_ADDRESS>Matej<EMAIL_ADDRESS>Faculty
of Computer and Information Science
University of Ljubljana
Ljubljana, Slovenia Center for Machine Perception,
Czech Technical University in Prague,
Prague, Czech Republic Trans2k

# Trans2k: Unlocking the Power of Deep Models for Transparent Object Tracking

###### Abstract

Visual object tracking has focused predominantly on opaque objects, while
transparent object tracking received very little attention. Motivated by the
uniqueness of transparent objects in that their appearance is directly
affected by the background, the first dedicated evaluation dataset has emerged
recently. We contribute to this effort by proposing the first transparent
object tracking training dataset Trans2k that consists of over 2k sequences
with 104,343 images overall, annotated by bounding boxes and segmentation
masks. Noting that transparent objects can be realistically rendered by modern
renderers, we quantify domain-specific attributes and render the dataset
containing visual attributes and tracking situations not covered in the
existing object training datasets. We observe a consistent performance boost
(up to 16%) across a diverse set of modern tracking architectures when trained
using Trans2k, and show insights not previously possible due to the lack of
appropriate training sets. The dataset and the rendering engine will be
publicly released to unlock the power of modern learning-based trackers and
foster new designs in transparent object tracking.

††∗ The authors contributed equally.

##  1 Introduction

Visual object tracking is a fundamental computer vision problem that emerges
in a broad range of downstream applications such as human-computer
interaction, surveillance, autonomous robotics and video editing, to name a
few. The substantial advances observed in the last decade have been primarily
driven by emergence of challenging evaluation datasets [Wu et al.(2015)Wu,
Lim, and Yang, Kristan et al.(2016)Kristan, Matas, Leonardis, Vojir,
Pflugfelder, Fernandez, Nebehay, Porikli, and Cehovin, Huang et
al.(2019)Huang, Zhao, and Huang, Fan et al.(2019)Fan, Lin, Yang, Chu, Deng,
Yu, Bai, Xu, Liao, and Ling] and diverse training sets [Muller et
al.(2018)Muller, Bibi, Giancola, Alsubaihi, and Ghanem, Russakovsky et
al.(2015)Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla,
Bernstein, et al., Huang et al.(2019)Huang, Zhao, and Huang] that enabled end-
to-end learning of modern deep tracking architectures. While most benchmarks
addressed opaque objects, very little attention has been dedicated to tracking
of transparent objects. These are unique in that they are often reflective and
their appearance is affected by the background texture, thus reducing the
reliability of the deep features trained for opaque objects.

Figure 1:  Trackers trained on opaque object training sets fail due to
specifics of transparent object appearance dynamics (upper row). After
training with the proposed Trans2k, their performance remarkably improves
(bottom row).

Recently, the TOTB benchmark [Fan et al.(2021)Fan, Miththanthaya, Harshit,
Rajan, Liu, Zou, Lin, and Ling] was proposed to facilitate research in
transparent object tracking. The benchmark results show that classical
trackers underperform on transparent objects and that, contrary to opaque
object tracking and to many other vision problems, shallow backbones
outperform the deep ones. However, it is crucial to note that the results were
obtained without re-training the state-of-the-art trackers on representative
training sets, which opens the question whether these observations are not
just a consequence of the domain shift rather than an inherent property of
shallow and deep modern learning-based tracking architectures. There is thus a
pressing need for a high quality transparent object training video dataset to
answer this question and to potentially unlock the power of deep learning
trackers, as well as to facilitate in-depth analysis and foster further
research.

Construction of the training dataset presents many challenges. First, the
training set should be large, diverse, and focus on visual attributes and
challenging situations specific for transparent objects, which are not already
covered in the opaque tracking datasets. Second, the targets should be
accurately annotated. Presented with these challenges, various sequence
selection and annotation protocols have emerged [Huang et al.(2019)Huang,
Zhao, and Huang, Kristan et al.(2013)Kristan, Pflugfelder, Leonardis, Matas,
Porikli, Čehovin, Nebehay, Fernandez, and Vojir, Fan et al.(2021)Fan,
Miththanthaya, Harshit, Rajan, Liu, Zou, Lin, and Ling, Kristan et
al.(2014)Kristan, Pflugfelder, Leonardis, Matas, Čehovin, Nebehay, Vojir, and
et al. Fernandez]. In related fields like 6DoF estimation [Hodan et
al.(2020a)Hodan, Barath, and Matas, Hodan et al.(2018)Hodan, Michel,
Brachmann, Kehl, GlentBuch, Kraft, Drost, Vidal, Ihrke, Zabulis, Sahin,
Manhardt, Tombari, Kim, Matas, and Rother] and scene parsing [Du et
al.(2018)Du, Liu, Basevi, Leonardis, Freeman, Tenenbaum, and Wu, Zhang et
al.(2019)Zhang, Chen, Huang, Lin, and Zhang, Qi et al.(2019)Qi, Wang, Qin, and
Li], image rendering has been applied to avoid the aforementioned issues.
However, the realism of rendered general objects remains limited, reducing the
training potential. We note that transparent objects are unique in that,
contrary to their opaque counterparts, non-textured transparent materials may
be faithfully rendered by modern renderers [Denninger et al.(2020)Denninger,
Sundermeyer, Winkelbauer, Olefir, Hodan, Zidan, Elbadrawy, Knauer, Katam, and
Lodhi]. Thus highly realistic sequences with precisely specified visual
attributes and pixel-level ground truth free of subjective annotation bias can
be generated.

We propose the first transparent object tracking training dataset Trans2k. To
maximise its utility, a protocol is designed that identifies challenging
visual attributes and tracking situations not covered in existing datasets.
The identified attribute ranges are then used in rendering over 2k sequences
with 104,343 images overall.

A set of trackers representing the major modern deep learning approaches is
evaluated on [Fan et al.(2021)Fan, Miththanthaya, Harshit, Rajan, Liu, Zou,
Lin, and Ling]. We report a consistent performance boost (up to $16\%$) across
all architectures when trained with Trans2k. Contrary to [Fan et al.(2021)Fan,
Miththanthaya, Harshit, Rajan, Liu, Zou, Lin, and Ling], we show that deep
backbones outperform shallow ones on transparent object tracking, which is
consistent with observations in opaque tracking. We see transformers as the
most promising approach and identify the visual attributes that future
architectural designs should address to make significant progress in
performance.

In summary, our contributions are: (i) Trans2k, the first training dataset for
transparent object tracking that unlocks the power of deep trainable trackers
and allows training bounding box or segmentation trackers, (ii) a
complementary analysis on [Fan et al.(2021)Fan, Miththanthaya, Harshit, Rajan,
Liu, Zou, Lin, and Ling] with new findings indicating future research
directions. The dataset and the sequence generation engine will be made
publicly available. The paper reports two surprising observations: first, that
transparent object tracking results are comparable to opaque object tracking
for state-of-the-art trackers trained with Trans2k, and second, that training
with Trans2k leads to substantial performance boost on transparent objects at
minimal reduction on opaque objects. The dataset, rendering engine and
instructions how to use it are available here:
https://github.com/trojerz/Trans2k

##  2 Related Work

Object tracking. Deep trackers excel across various benchmarks [Matej Kristan,
et al.(2020), Kristan et al.(2021)Kristan, Matas, Leonardis, Felsberg,
Pflugfelder, Kämäräinen, Chang, Danelljan, Cehovin, Lukežič, Drbohlav, Käpylä,
Häger, Yan, Yang, Zhang, and Fernández, Huang et al.(2019)Huang, Zhao, and
Huang, Fan et al.(2019)Fan, Lin, Yang, Chu, Deng, Yu, Bai, Xu, Liao, and Ling,
Wu et al.(2015)Wu, Lim, and Yang, Fan et al.(2021)Fan, Miththanthaya, Harshit,
Rajan, Liu, Zou, Lin, and Ling] compared to their hand-crafted counterparts.
Initially, pre-trained general backbones were used for feature extraction,
primarily by the discriminative correlation filter (DCF) trackers [Danelljan
et al.(2016)Danelljan, Robinson, Khan, and Felsberg, Danelljan et
al.(2017)Danelljan, Bhat, Shahbaz Khan, and Felsberg, Bhat et al.(2018)Bhat,
Johnander, Danelljan, Shahbaz Khan, and Felsberg, Danelljan et
al.(2015)Danelljan, Häger, Khan, and Felsberg, Liu(2021)], which learned a
discriminative localization models online during tracking. Later, backbone
end-to-end training techniques that maximize DCF localization were proposed
[Valmadre et al.(2017)Valmadre, Bertinetto, Henriques, Vedaldi, and Torr].
Most recently, the DCF optimization has been introduced as part of the deep
network. Milestone representatives were proposed in [Danelljan et
al.(2019a)Danelljan, Bhat, Khan, and Felsberg, Danelljan et
al.(2019b)Danelljan, Bhat, Van Gool, and Timofte], which also proposed a post-
processing network for bounding box refinement that accounted for target
aspect changes. In parallel, siamese trackers have been explored and grown
into a major tracker design branch. The seminal work [Bertinetto et
al.(2016)Bertinetto, Valmadre, Henriques, Vedaldi, and Torr] trained AlexNet-
based network [Krizhevsky et al.(2012)Krizhevsky, Sutskever, and Hinton] such
that localization accuracy is maximized simply by correlation between a
template and search region in feature space. These trackers afford fast
processing since no training is required during tracking. Siamese trackers
were extended by anchor-based region proposal networks [Li et al.(2018)Li,
Yan, Wu, Zhu, and Hu, Li et al.(2019)Li, Wu, Wang, Zhang, Xing, and Yan] and
recently an anchor-free extension has been proposed [Chen et al.(2020)Chen,
Zhong, Li, Zhang, and Ji] with improved localization performance. Drawing on
advances in object detection [Carion et al.(2020)Carion, Massa, Synnaeve,
Usunier, Kirillov, and Zagoruyko], transformer-based trackers have recently
emerged [Yan et al.(2021)Yan, Peng, Fu, Wang, and Lu, Chen et al.(2021)Chen,
Yan, Zhu, Wang, Yang, and Lu, Wang et al.(2021)Wang, Zhou, Wang, and Li].
These are the current state-of-the-art, and computationally efficient with
remarkable real-time performance [Kristan et al.(2021)Kristan, Matas,
Leonardis, Felsberg, Pflugfelder, Kämäräinen, Chang, Danelljan, Cehovin,
Lukežič, Drbohlav, Käpylä, Häger, Yan, Yang, Zhang, and Fernández].

Benchmarks. The developments in visual object tracking have been facilitated
by introduction of benchmarks. The first widely-used benchmark [Wu et
al.(2015)Wu, Lim, and Yang, Wu et al.(2013)Wu, Lim, and Yang] proposed a
dataset and evaluation protocol that allowed standardised comparsion of
trackers. Later, the VOT initiative explored dataset construction as well as
performance evaluation protocols for efficient in-depth analysis [Kristan et
al.(2016)Kristan, Matas, Leonardis, Vojir, Pflugfelder, Fernandez, Nebehay,
Porikli, and Cehovin, Kristan et al.(2013)Kristan, Pflugfelder, Leonardis,
Matas, Porikli, Čehovin, Nebehay, Fernandez, and Vojir, Kristan et
al.(2014)Kristan, Pflugfelder, Leonardis, Matas, Čehovin, Nebehay, Vojir, and
et al. Fernandez]. Further improvements were made in the subsequent yearly
challenges, e.g., [Matej Kristan, et al.(2020), Kristan et al.(2021)Kristan,
Matas, Leonardis, Felsberg, Pflugfelder, Kämäräinen, Chang, Danelljan,
Cehovin, Lukežič, Drbohlav, Käpylä, Häger, Yan, Yang, Zhang, and Fernández].
With advent of deep learning, tracking training sets have emerged. [Muller et
al.(2018)Muller, Bibi, Giancola, Alsubaihi, and Ghanem] constructed a huge
training set from public video repository and applied a semi-automatic
annotation. Recently, [Huang et al.(2019)Huang, Zhao, and Huang] presented ten
thousand annotated video sequences, divided into a large training and a
smaller evaluation set. Concurrently, a long-term tracking benchmark [Fan et
al.(2019)Fan, Lin, Yang, Chu, Deng, Yu, Bai, Xu, Liao, and Ling] with fifteen
pre-defined categories, containing training and test set was proposed. All
these benchmarks focus on opaque objects, while recently as transparent object
tracking evaluation dataset [Fan et al.(2021)Fan, Miththanthaya, Harshit,
Rajan, Liu, Zou, Lin, and Ling] has been proposed. However, training datasets
for transparent object tracking have not been proposed.

Use of synthesis. Rendering has been previously considered in computer vision
to avoid costly manual dataset acquisition. In [Krahenbuhl(2018), Richter et
al.(2017)Richter, Hayder, and Koltun], synthetic data was generated by a video
game engine, which provided an unlimited amount of annotated training data for
various computer vision tasks. A rendered dataset of urban scenes, Synthia
[Ros et al.(2016)Ros, Sellart, Materzynska, Vazquez, and Lopez], was shown to
substantially improve the trained deep models for semantic segmentation. A
similar dataset [Wrenninge and Unger(2018)] was proposed for training and
evaluation of scene parsing networks. A fine-grained vegetation and terrain
dataset [Metzger et al.(2021)Metzger, Mortimer, and Wuensche] was recently
proposed for training drivable surfaces and natural obstacles detection
networks in outdoor scenes. [Saleh et al.(2018)Saleh, Aliakbarian, Salzmann,
Petersson, and Alvarez] showed that foreground and background should be
treated differently when training segmentation on synthetic images. The
benefits of using mixed real and synthetic 6DoF training data has been
recently shown in [Hodan et al.(2020b)Hodan, Barath, and Matas]. The major
6DoF object detection challenge [Hodaň et al.(2020)Hodaň, Sundermeyer, Drost,
Labbé, Brachmann, Michel, Rother, and Matas] thus provides a combination of
real and synthetic images for training as well as evaluation. Synthesis has
been used in the UAV123 tracking benchmark [Mueller et al.(2016)Mueller,
Smith, and Ghanem] in which eight of the sequences are rendered by a game
engine. A rendering approach was used in [Cehovin Zajc et al.(2017)Cehovin
Zajc, Lukezic, Leonardis, and Kristan] to parameterize camera motion for fine-
grained tracker performance analysis. However, using synthetic data for
training in visual tracking remains unexplored.

Transparent objects. Highlighting the difference from opaque counterparts,
transparent objects have been explored in computer vision in various tasks.
Recognition of transparent objects was studied in [Fritz et al.(2009)Fritz,
Bradski, Karayev, Darrell, and Black, Maeno et al.(2013)Maeno, Nagahara,
Shimada, and Taniguchi], while 3D shape estimation and reconstruction of
transparent objects on RGB-D images was proposed in [Klank et al.(2011)Klank,
Carton, and Beetz, Sajjan et al.(2020)Sajjan, Moore, Pan, Nagaraja, Lee, Zeng,
and Song]. Segmentation of transparent objects has been studied in [Xu et
al.(2015)Xu, Nagahara, Shimada, and Taniguchi, Kalra et al.(2020)Kalra,
Taamazyan, Rao, Venkataraman, Raskar, and Kadambi], while a benchmark was
proposed in [Xie et al.(2020)Xie, Wang, Wang, Ding, Shen, and Luo]. All these
works consider single-image tasks and little attention has been dedicated to
videos. In fact, a transparent object tracking benchmark [Fan et al.(2021)Fan,
Miththanthaya, Harshit, Rajan, Liu, Zou, Lin, and Ling] has been proposed only
recently and reported a performance gap between transparent and opaque object
tracking. However, due to the lack of a dedicated training dataset, the gap
source remains unclear.

##  3 Trans2k dataset

Transparent objects, which are often reflective and glass-like, can be
rendered with a high level of realism by the modern photo-realistic rendering
engines [Denninger et al.(2020)Denninger, Sundermeyer, Winkelbauer, Olefir,
Hodan, Zidan, Elbadrawy, Knauer, Katam, and Lodhi]. In our approach, we first
identify and parameterize the sequence attributes specific to transparent
objects (Section 3.1). A BlenderProc-based sequence generator is implemented
that enables parameterized sequence rendering. Attribute levels useful for
learning are identified empirically and the final training dataset is
generated (Section 3.2).

###  3.1 Parametrization of sequence attributes

The dataset should reflect the diversity of visual attributes typical for
transparent object tracking scenes for efficient learning. After carefully
examining various videos of transparent and opaque objects, the following
attributes were identified (Figure 2).

Scene background. Since background affects the transparent object appearance,
a high background diversity is required in training. We ensure this by
randomly sampling videos from GoT10k [Huang et al.(2019)Huang, Zhao, and
Huang] training set and use them as backgrounds over which the transparent
object is rendered.

Object types. 3D models of 25 object types from open source online
repositories are selected with several instances of the same type. The set was
chosen such to cover a range of nontrivial as well as smooth shapes, with some
objects rendered with empty and some with full volume. This amounts to 148
object instances.

Target motion. To increase the object-background appearance diversity, the
objects are moving in the videos. The motion trajectory is generated by a
cubic Hermite spline spanned by four uniformly sampled points. The motion
dynamics is not critical in training, since deep models are typically trained
on pairs of image patches cropped at target position. Thus a constant velocity
is applied.

Distractors. In realistic environments, the target may be surrounded by other
visually similar transparent objects (e.g., glasses on a table), which act as
distractors. We thus render an additional transparent object following the
target object. The distractor object is from a different type to keep the
appearance-based localization learning task feasible.

Transparency. The level of transparency crucially affects the target
appearance. We thus identify four levels ranging from clearly visible to
nearly invisible.

Motion blur. Fast motions, depending on the aperture speed, result in various
levels of blurring. We identify four levels of blur intensity, ranging from no
blurring to extreme blurriness.

Partial occlusion. Objects are commonly occluded by other objects in practical
situations (e.g., handling of the target). We thus simulate partial occlusions
by rendering coloured stripe pattern moving across the video frame. The stripe
width is fixed, while the occlusion intensity is simulated by the number of
stripes (0, 7, 11, 20) per image, i.e., from zero to severe occlusion.

Rotation. To present realistic object appearance change, the object rotates in
3D in addition to position change. The rotation dynamics is specified by the
angular velocity along each axis, which is kept constant throughout the
sequence. We identify four rotation speed levels, (0, 1.3, 5.4, 10.6) degrees
per frame, thus ranging from no rotation to fast rotation.

###  3.2 Attribute selection and dataset generation

To maximize the dataset application utility, the sequences should be
complementary to existing datasets from tracking perspective and should focus
on attributes that the learning-based trackers cannot already learn from
opaque object tracking training sets. An empirical study was designed to
determine which intensity levels of the attributes (i) transparency, (ii)
partial occlusion, (iii) rotation and (iv) motion blur should be considered in
the final dataset. The intensity levels are visualized in Figure 2.

Figure 2: Trans2k attribute levels for "Transparency", "Motion blur", "Partial
occlusion", "Distractor" (binary), "Target motion" (four control points) and
"Rotation".

Seven state-of-the-art deep learning trackers pretrained on opaque object
tracking datasets (see Section 4 for details) that cover the major current
trends in tracking were selected. The difficulty level of individual attribute
is quantified as the overall performance of these trackers on test sequences
rendered with that attribute. Specifically, for each attribute level, five
sequences were sampled from Got10K [Huang et al.(2019)Huang, Zhao, and Huang]
training set and used as backgrounds in the rendered sequences. The same
background sequences with same object, traveling over the same trajectory are
used with all attributes to ensure consistent evaluation. This resulted in 80
test sequences (4 attributes $\times$ 4 levels $\times$ 5 variations).

The results are shown in Figure 3. We observe that most of the attribute
levels result in performance reduction and are thus kept as relevant in our
final dataset, except from two at which the trackers score quite high. The
lowest transparency level and zero rotation appear to be well addressed by the
opaque object training sets, thus we decide not to include them in our dataset
for better use of its capacity. The following parameters are thus applied when
rendering Trans2k. The GoT10k training set sequences are sampled at random and
at most once. All object types are sampled with equal probability. The
transparency levels (excluding the lowest level) are sampled with equal
probability. Blur presence in a sequence is sampled with 0.15 probability,
with blur levels sampled uniformly. Occlusion presence is sampled with 0.2
probability, while occlusion levels are sampled uniformly. Rotation level is
uniformly sampled. The resulting training dataset Trans2k thus contains 2,039
challenging sequences and 104,343 frames in total.

Since the sequences are rendered, the ground truth can be exactly computed. We
provide the ground truth in two standard forms, the widely accepted target
enclosing axis-aligned bounding-box and the segmentation mask to cater to the
emerging segmentation trackers [Matej Kristan, et al.(2020)]. The ground
truths for distractors are generated as well. Trans2k is thus the first
dataset with per-frame distractor annotation to facilitate development of
future learning-based methods that could exploit this.

Figure 3: Average IoU of trackers reflect the difficulty level of individual
attribute intensity. The blue shaded columns show performance of trackers pre-
trained on opaque datasets, while the gray shaded column shows performance
after training with Trans2k.

##  4 Experiments

###  4.1 Selected trackers and training setup

We selected state-of-the-art learning-based trackers that cover the major
trends in modern architecture designs for validating Trans2k: (i) two siamese
trackers SiamRPN++ [Li et al.(2019)Li, Wu, Wang, Zhang, Xing, and Yan] and
SiamBAN [Chen et al.(2020)Chen, Zhong, Li, Zhang, and Ji], (ii) two deep
correlation filter trackers ATOM [Danelljan et al.(2019a)Danelljan, Bhat,
Khan, and Felsberg] and DiMP [Danelljan et al.(2019b)Danelljan, Bhat, Van
Gool, and Timofte], (iii) the recent state-of-the-art transparent object
tracker TransATOM [Fan et al.(2021)Fan, Miththanthaya, Harshit, Rajan, Liu,
Zou, Lin, and Ling], and (iv) a transfomer-based tracker STARK [Yan et
al.(2021)Yan, Peng, Fu, Wang, and Lu]. These trackers localize the target by a
bounding box. To account for the recent trend in localization by per-pixel
segmentation [Matej Kristan, et al.(2020)], we include (v) the recent state-
of-the-art segmentation-based tracker D3S [Lukežič et al.(2020)Lukežič, Matas,
and Kristan].

During training, the trackers were initialized by the pre-trained weights
provided by their authors, while all the training details were the same as in
the original implementations. The trackers were trained for 50 epochs with
10000 training samples per epoch. Since Trans2k was designed as a
complementary dataset covering situations not present in existing datasets,
the training considers samples from Trans2k as well as opaque object
sequences. In particular, we merged the opaque training datasets GOT10k [Huang
et al.(2019)Huang, Zhao, and Huang], LaSoT [Fan et al.(2019)Fan, Lin, Yang,
Chu, Deng, Yu, Bai, Xu, Liao, and Ling] and TrackingNet [Muller et
al.(2018)Muller, Bibi, Giancola, Alsubaihi, and Ghanem] into a single dataset,
abbreviated as opaque object training dataset (OTD). A training batch is then
constructed by sampling from Trans2k and OTD with 5:3 ratio.

###  4.2 Validation of Trans2k

We first validated the contribution of Trans2k by measuring performance of
trackers on the recent transparent object tracking benchmark TOTB [Fan et
al.(2021)Fan, Miththanthaya, Harshit, Rajan, Liu, Zou, Lin, and Ling].
Following the regime described in Section 4.1 the selection of seven state-of-
the-art trackers was trained using Trans2k. Their performance was then
compared to their original performance, i.e., when trained only with opaque
object tracking sequences. Thus any change in performance is contributed only
by the training dataset. The trackers were evaluated by the standard one-pass
evaluation protocol (OPE) that quantifies the performance by AUC and center
error measures on success and precision plots. For more information on the
protocol, please refer to [Wu et al.(2015)Wu, Lim, and Yang, Fan et
al.(2021)Fan, Miththanthaya, Harshit, Rajan, Liu, Zou, Lin, and Ling].

Figure 4: Trackers evaluated on TOTB dataset shown in precision and success
plots. Trackers trained with Trans2k are denoted by a star (*). The right
graph shows absolute improvements in tracking performance measured by the AUC
measure after training with the proposed Trans2k.

The results are shown in Figure 4. The performance of all trackers
substantially improved when trained using Trans2k. The performance gains are
at a level usually expected for a clear methodological improvement. Recently,
TransATOM [Fan et al.(2021)Fan, Miththanthaya, Harshit, Rajan, Liu, Zou, Lin,
and Ling], a transparent object tracking extension of ATOM [Danelljan et
al.(2019a)Danelljan, Bhat, Khan, and Felsberg], was proposed, which
outperformed ATOM by 2.1%. Without any methodological modification and only
training with Trans2k, ATOM outperforms this extension by 1.7%. Nevertheless,
TransATOM gains 3.3% when trained with Trans2k. The largest performance boost
is achieved by DiMP, which improves by over 16% and scores as the second-best
among all the tested trackers. Consistent with the observation on opaque
object tracking benchmarks, the transformer-based tracker STARK achieves the
best performance. Note that even without training with Trans2k, STARK
surpasses all trackers, but when trained with Trans2k, an additional healthy
2.5% performance boost is observed. Since Trans2k provides segmentation ground
truths in addition to bounding boxes, it boosts the segmentation-based tracker
D3S [Lukežič et al.(2020)Lukežič, Matas, and Kristan] as well. The version
trained with Trans2k gains a remarkable 6% in performance.

###  4.3 Re-evaluating the significance of backbone depth

The recent benchmark [Fan et al.(2021)Fan, Miththanthaya, Harshit, Rajan, Liu,
Zou, Lin, and Ling] reported a remarkable case that, specific to transparent
object tracking, shallow backbones outperform deep ones, which conflicts
common observations in opaque object tracking. Since [Fan et al.(2021)Fan,
Miththanthaya, Harshit, Rajan, Liu, Zou, Lin, and Ling] could only analyze the
performance with using opaque tracking training sets, we re-evaluate this
claim but in the context of using a transparent object tracking training set.
We select three deep discriminative correlation filters DiMP [Danelljan et
al.(2019b)Danelljan, Bhat, Van Gool, and Timofte], ATOM [Danelljan et
al.(2019a)Danelljan, Bhat, Khan, and Felsberg] and TransATOM [Fan et
al.(2021)Fan, Miththanthaya, Harshit, Rajan, Liu, Zou, Lin, and Ling] and
study their performance with a shallow (ResNet18) and a deep (ResNet50)
backbone when trained with and without transparent objects.

Results in Table 1 reveal that ATOM and TransATOM with shallow backbones
indeed outperform their deep backbone counterparts. In contrast, DiMP with
deep backbone substantially outperforms the shallow backbone counterpart. This
apparent discrepancy comes from the different designs of ATOM and DiMP. While
ATOM uses a pre-trained backbone and allows training of only post-processing
steps, DiMP trains the backbone as well. The ATOM’s apparent preference of
shallow backbones comes from the fact that shallow backbones generalize better
to transparent objects when trained only with opaque training examples. To
verify this, we replace ATOM’s and TransATOM’s backbone by those trained by
DiMP. Both trackers improve their performance with deep backbones trained on
transparent objects compared to shallow ones. Interestingly, their deep
backbone variants reach performance near DiMP’s and their performance
difference becomes negligible – thus properly trained vanilla ATOM should be
preferred to its more complex extension TransATOM. The experiments thus
reveal, that deep backbones in fact lead to substantial improvements over
shallow counterparts, if trained on the transparent object dataset.

Table 1: Tracking performance (AUC) of three trackers using different backbones. Opaque indicates training with only OTD, \+ Trans2k to using the transparent dataset as well. Pretrained ResNet18 and ResNet50 backbones are denoted by R18 and R50, respectively, while their versions trained by DiMP are denoted by D18 and D50. | DiMP | ATOM | TransATOM
---|---|---|---
| D18 | D50 | R18 | R50 | D18 | D50 | R18 | R50 | D18 | D50
Opaque | 0.552 | 0.600 | 0.618 | 0.608 | 0.551 | 0.588 | 0.631 | 0.608 | 0.582 | 0.603
\+ Trans2k | 0.613 | 0.699 | 0.642 | 0.648 | 0.629 | 0.695 | 0.664 | 0.664 | 0.647 | 0.697

###  4.4 How does Trans2k affect opaque object tracking?

To quantify how much the trackers trained with Trans2k lose in generalization
to opaque objects, we evaluate the trackers on the GOT10k [Huang et
al.(2019)Huang, Zhao, and Huang] validation dataset. Table 2 shows results for
trackers trained only with OTD and with added Trans2k (as described in Section
4.1). The tracking performance on opaque objects slightly drops, but still
remains high. This result suggests that, while substantial boosts are observed
in transparent object tracking (Figure 4) with the use of Trans2k, the
generalization to opaque objects is not lost.

Table 2: Tracking performance (AUC) on the opaque tracking dataset GoT-10k val. Opaque – training with only OTD, \+ Trans2k – using the transparent dataset as well.  | STARK | DiMP | SiamBAN | D3S | TransATOM | SiamRPN | ATOM
---|---|---|---|---|---|---|---
Opaque | 0.777 | 0.706 | 0.679 | 0.676 | 0.662 | 0.656 | 0.650
\+ Trans2k | 0.752 | 0.696 | 0.676 | 0.663 | 0.650 | 0.656 | 0.650

###  4.5 The role of using opaque objects in training

To further study the impact of the training set content from perspective of
the presence of opaque and transparent objects, the training sets were varied.
We selected two well-known state-of-the-art trackers that performed well in
our previous experiments, yet could be trained sufficiently fast. The deep
discriminative correlation filter DiMP [Danelljan et al.(2019b)Danelljan,
Bhat, Van Gool, and Timofte] and the siamese tracker SiamBAN [Chen et
al.(2020)Chen, Zhong, Li, Zhang, and Ji] were selected. The original versions
trained by the authors were evaluated on TOTB [Fan et al.(2021)Fan,
Miththanthaya, Harshit, Rajan, Liu, Zou, Lin, and Ling] along with the
versions re-trained using the following variations of the training set: (i)
only Trans2k without OTD, (ii) only OTD, (iii) Trans2k+OTD. In experiments
(i), (ii) and (iii) the tracker networks are initialized by their pre-trained
models provided by the authors. Thus an additonal experiment (iv) is performed
where the trackers were trained from scratch using the dataset from (iii). The
trained trackers were evaluated on TOTB [Fan et al.(2021)Fan, Miththanthaya,
Harshit, Rajan, Liu, Zou, Lin, and Ling].

Results in Table 4 show that using only Trans2k reduces the tracking
performance compared to training on opaque objects training datasets. A closer
look revealed that the trackers trained only with Trans2k tend to focus on
transparent objects in general rather than localizing the target, which was
reflected in tracker often jumping to nearby transparent objects when multiple
such objects were close to the target. Training with OTD when initialized with
original tracker parameters does not bring improvements in general (DiMP
performance drops slightly, while that of SiamBAN increases a bit). However,
when using transparent as well as opaque objects in the training set, the
performance improves substantially. When training from scratch, the
performance of both trackers drops compared to the version initialized with
pre-trained networks. This suggests pre-training is beneficial for both
trackers, but particularly for SiamBAN as it requires learning more parameters
than DiMP.

Table 3:  Comparison of different training setups for SiamBAN and DiMP. Performance of pre-trained trackers is indicated by orig., while scr. indicates training from scratch.  | orig. | Trans2k | OTD | OTD+Trans2k | scr.
---|---|---|---|---|---
DiMP | 0.600 | 0.554 | 0.584 | 0.699 | 0.667
SiamBAN | 0.656 | 0.650 | 0.658 | 0.680 | 0.649

##  5 Conclusion

The first transparent object tracking training dataset Trans2k is proposed.
The fact that transparent objects can be sufficiently realistically rendered
by modern renderers is exploited. Using a specialized protocol, we identified
visual attributes not covered well in existing datasets and rendered a dataset
with over 2k training sequences containing transparent objects.

Trans2k was validated on the recent transparent object tracking benchmark TOTB
[Wu et al.(2015)Wu, Lim, and Yang]. Training with Trans2k improves performance
at levels usually observed in fundamental methodological advancements in
tracking algorithms. This behavior is observed over a wide range of tracking
methodologies. Analysis shows that significant performance gains in
transparent object tracking come at a minor performance loss in opaque object
tracking, which indicates to excellent generalization of modern trackers. In
contrast to the findings in [Fan et al.(2021)Fan, Miththanthaya, Harshit,
Rajan, Liu, Zou, Lin, and Ling], experiments show that trackers benefit from
training deeper backbones on transparent objects. Additional experiments
showed the benefits of using transparent as well as opaque objects in the
training dataset. Overall, the best performance was observed with
transformers.

While the field of transparent object tracking has recently obtained an
excellent test set [Fan et al.(2021)Fan, Miththanthaya, Harshit, Rajan, Liu,
Zou, Lin, and Ling], the main ingredient crucial for advancements, i.e., a
curated training set was missing. Trans2k fills this void and will enable
future development of new learnable modules specifically addressing the
challenges in transparent object tracking, thus fully unlocking the power of
modern deep learning trackers on this scientifically interesting domain. Our
sequence generator engine will be released along with Trans2k. We envision
that the engine will allow inovative learning modes in which the sequences
with specific challenges can be generated on demand to specialize the trackers
to niche tasks or to improve their overall performance. In addition, the
rendering engine could be used to generate training data for 6-DoF video pose
estimation, thus benefiting research beyond 2D transparent object tracking.

Acknowledgements: This work was supported by Slovenian research agency program
P2-0214 and projects Z2-4459, J2-2506 and J2-9433. J. Matas was supported by
the Czech Science Foundation grant GA18-05360S.

## References

  * [Bertinetto et al.(2016)Bertinetto, Valmadre, Henriques, Vedaldi, and Torr] Luca Bertinetto, Jack Valmadre, João F Henriques, Andrea Vedaldi, and Philip H S Torr.  Fully-convolutional siamese networks for object tracking.  In _European Conference on Computer Vision Workshops_ , pages 850–865, 2016.
  * [Bhat et al.(2018)Bhat, Johnander, Danelljan, Shahbaz Khan, and Felsberg] Goutam Bhat, Joakim Johnander, Martin Danelljan, Fahad Shahbaz Khan, and Michael Felsberg.  Unveiling the power of deep tracking.  In _Proc. European Conf. Computer Vision_ , pages 493–509, 2018.
  * [Carion et al.(2020)Carion, Massa, Synnaeve, Usunier, Kirillov, and Zagoruyko] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko.  End-to-end object detection with transformers.  In _Proc. European Conf. Computer Vision_ , pages 213–229, 2020.
  * [Cehovin Zajc et al.(2017)Cehovin Zajc, Lukezic, Leonardis, and Kristan] Luka Cehovin Zajc, Alan Lukezic, Ales Leonardis, and Matej Kristan.  Beyond standard benchmarks: Parameterizing performance evaluation in visual object tracking.  In _Int. Conf. Computer Vision_ , 2017.
  * [Chen et al.(2021)Chen, Yan, Zhu, Wang, Yang, and Lu] Xin Chen, Bin Yan, Jiawen Zhu, Dong Wang, Xiaoyun Yang, and Huchuan Lu.  Transformer tracking.  In _Comp. Vis. Patt. Recognition_ , pages 8126–8135, 2021.
  * [Chen et al.(2020)Chen, Zhong, Li, Zhang, and Ji] Zedu Chen, Bineng Zhong, Guorong Li, Shengping Zhang, and Rongrong Ji.  Siamese box adaptive network for visual tracking.  In _Comp. Vis. Patt. Recognition_ , pages 6668–6677, 2020.
  * [Danelljan et al.(2015)Danelljan, Häger, Khan, and Felsberg] Martin Danelljan, Gustav Häger, Fahad Shahbaz Khan, and Michael Felsberg.  Convolutional features for correlation filter based visual tracking.  In _2015 IEEE International Conference on Computer Vision Workshop (ICCVW)_ , pages 621–629, 2015.
  * [Danelljan et al.(2016)Danelljan, Robinson, Khan, and Felsberg] Martin Danelljan, Andreas Robinson, Fahad Shahbaz Khan, and Michael Felsberg.  Beyond correlation filters: learning continuous convolution operators for visual tracking.  In _Proc. European Conf. Computer Vision_ , pages 472–488, 2016.
  * [Danelljan et al.(2017)Danelljan, Bhat, Shahbaz Khan, and Felsberg] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg.  Eco: Efficient convolution operators for tracking.  In _Comp. Vis. Patt. Recognition_ , pages 6638–6646, 2017.
  * [Danelljan et al.(2019a)Danelljan, Bhat, Khan, and Felsberg] Martin Danelljan, Goutam Bhat, Fahad Shahbaz Khan, and Michael Felsberg.  ATOM: Accurate tracking by overlap maximization.  In _Comp. Vis. Patt. Recognition_ , 2019a.
  * [Danelljan et al.(2019b)Danelljan, Bhat, Van Gool, and Timofte] Martin Danelljan, Goutam Bhat, Luc Van Gool, and Radu Timofte.  Learning discriminative model prediction for tracking.  In _Int. Conf. Computer Vision_ , pages 6181–6190, 2019b.
  * [Denninger et al.(2020)Denninger, Sundermeyer, Winkelbauer, Olefir, Hodan, Zidan, Elbadrawy, Knauer, Katam, and Lodhi] Maximilian Denninger, Martin Sundermeyer, Dominik Winkelbauer, Dmitry Olefir, Tomas Hodan, Youssef Zidan, Mohamad Elbadrawy, Markus Knauer, Harinandan Katam, and Ahsan Lodhi.  Blenderproc: Reducing the reality gap with photorealistic rendering.  In _International Conference on Robotics: Sciene and Systems, RSS 2020_ , 2020.
  * [Du et al.(2018)Du, Liu, Basevi, Leonardis, Freeman, Tenenbaum, and Wu] Yilun Du, Zhijian Liu, Hector Basevi, Ales Leonardis, Bill Freeman, Josh Tenenbaum, and Jiajun Wu.  Learning to exploit stability for 3d scene parsing.  In _Advances in Neural Information Processing Systems_ , volume 31, 2018.
  * [Fan et al.(2019)Fan, Lin, Yang, Chu, Deng, Yu, Bai, Xu, Liao, and Ling] Heng Fan, Liting Lin, Fan Yang, Peng Chu, Ge Deng, Sijia Yu, Hexin Bai, Yong Xu, Chunyuan Liao, and Haibin Ling.  Lasot: A high-quality benchmark for large-scale single object tracking.  In _Comp. Vis. Patt. Recognition_ , June 2019.
  * [Fan et al.(2021)Fan, Miththanthaya, Harshit, Rajan, Liu, Zou, Lin, and Ling] Heng Fan, Halady Akhilesha Miththanthaya, Harshit, Siranjiv Ramana Rajan, Xiaoqiong Liu, Zhilin Zou, Yuewei Lin, and Haibin Ling.  Transparent object tracking benchmark.  In _Int. Conf. Computer Vision_ , pages 10734–10743, 2021.
  * [Fritz et al.(2009)Fritz, Bradski, Karayev, Darrell, and Black] Mario Fritz, Gary Bradski, Sergey Karayev, Trevor Darrell, and Michael Black.  An additive latent feature model for transparent object recognition.  _Advances in Neural Information Processing Systems_ , 22, 2009.
  * [Hodan et al.(2018)Hodan, Michel, Brachmann, Kehl, GlentBuch, Kraft, Drost, Vidal, Ihrke, Zabulis, Sahin, Manhardt, Tombari, Kim, Matas, and Rother] Tomas Hodan, Frank Michel, Eric Brachmann, Wadim Kehl, Anders GlentBuch, Dirk Kraft, Bertram Drost, Joel Vidal, Stephan Ihrke, Xenophon Zabulis, Caner Sahin, Fabian Manhardt, Federico Tombari, Tae-Kyun Kim, Jiri Matas, and Carsten Rother.  Bop: Benchmark for 6d object pose estimation.  In _Proc. European Conf. Computer Vision_ , September 2018.
  * [Hodan et al.(2020a)Hodan, Barath, and Matas] Tomas Hodan, Daniel Barath, and Jiri Matas.  Epos: Estimating 6d pose of objects with symmetries.  In _Comp. Vis. Patt. Recognition_ , 2020a.
  * [Hodan et al.(2020b)Hodan, Barath, and Matas] Tomas Hodan, Daniel Barath, and Jiri Matas.  Epos: Estimating 6d pose of objects with symmetries.  In _Comp. Vis. Patt. Recognition_ , pages 11703–11712, 2020b.
  * [Hodaň et al.(2020)Hodaň, Sundermeyer, Drost, Labbé, Brachmann, Michel, Rother, and Matas] Tomáš Hodaň, Martin Sundermeyer, Bertram Drost, Yann Labbé, Eric Brachmann, Frank Michel, Carsten Rother, and Jiří Matas.  Bop challenge 2020 on 6d object localization.  In _Proc. European Conf. Computer Vision_ , pages 577–594, 2020.
  * [Huang et al.(2019)Huang, Zhao, and Huang] Lianghua Huang, Xin Zhao, and Kaiqi Huang.  GOT-10k: A large high-diversity benchmark for generic object tracking in the wild.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 2019.
  * [Kalra et al.(2020)Kalra, Taamazyan, Rao, Venkataraman, Raskar, and Kadambi] Agastya Kalra, Vage Taamazyan, Supreeth Krishna Rao, Kartik Venkataraman, Ramesh Raskar, and Achuta Kadambi.  Deep polarization cues for transparent object segmentation.  In _Comp. Vis. Patt. Recognition_ , pages 8602–8611, 2020.
  * [Klank et al.(2011)Klank, Carton, and Beetz] Ulrich Klank, Daniel Carton, and Michael Beetz.  Transparent object detection and reconstruction on a mobile platform.  In _Int. Conf. Robotics and Automation_ , pages 5971–5978. IEEE, 2011\.
  * [Krahenbuhl(2018)] Philipp Krahenbuhl.  Free supervision from video games.  In _Comp. Vis. Patt. Recognition_ , pages 2955–2964, 2018.
  * [Kristan et al.(2013)Kristan, Pflugfelder, Leonardis, Matas, Porikli, Čehovin, Nebehay, Fernandez, and Vojir] M. Kristan, R. Pflugfelder, A. Leonardis, J. Matas, F. Porikli, L. Čehovin, G. Nebehay, G. Fernandez, and T. et al. Vojir.  The visual object tracking vot2013 challenge results.  In _Vis. Obj. Track. Challenge VOT2013, In conjunction with ICCV2013_ , pages 98–111, 2013.
  * [Kristan et al.(2016)Kristan, Matas, Leonardis, Vojir, Pflugfelder, Fernandez, Nebehay, Porikli, and Cehovin] M. Kristan, J. Matas, A. Leonardis, T. Vojir, R. Pflugfelder, G. Fernandez, G. Nebehay, F. Porikli, and L. Cehovin.  A novel performance evaluation methodology for single-target trackers.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 2016.
  * [Kristan et al.(2014)Kristan, Pflugfelder, Leonardis, Matas, Čehovin, Nebehay, Vojir, and et al. Fernandez] Matej Kristan, Roman Pflugfelder, Aleš Leonardis, Jiri Matas, Luka Čehovin, Georg Nebehay, Tomas Vojir, and Gustavo et al. Fernandez.  The visual object tracking vot2014 challenge results.  In _Proc. European Conf. Computer Vision_ , pages 191–217, 2014.
  * [Kristan et al.(2021)Kristan, Matas, Leonardis, Felsberg, Pflugfelder, Kämäräinen, Chang, Danelljan, Cehovin, Lukežič, Drbohlav, Käpylä, Häger, Yan, Yang, Zhang, and Fernández] Matej Kristan, Jiří Matas, Aleš Leonardis, Michael Felsberg, Roman Pflugfelder, Joni-Kristian Kämäräinen, Hyung Jin Chang, Martin Danelljan, Luka Cehovin, Alan Lukežič, Ondrej Drbohlav, Jani Käpylä, Gustav Häger, Song Yan, Jinyu Yang, Zhongqun Zhang, and Gustavo Fernández.  The ninth visual object tracking vot2021 challenge results.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops_ , pages 2711–2738, 2021.
  * [Krizhevsky et al.(2012)Krizhevsky, Sutskever, and Hinton] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton.  Imagenet classification with deep convolutional neural networks.  In _Advances in Neural Information Processing Systems_ , volume 25, 2012.
  * [Li et al.(2018)Li, Yan, Wu, Zhu, and Hu] Bo Li, Junjie Yan, Wei Wu, Zheng Zhu, and Xiaolin Hu.  High performance visual tracking with siamese region proposal network.  In _Comp. Vis. Patt. Recognition_ , 2018.
  * [Li et al.(2019)Li, Wu, Wang, Zhang, Xing, and Yan] Bo Li, Wei Wu, Qiang Wang, Fangyi Zhang, Junliang Xing, and Junjie Yan.  SiamRPN++: Evolution of siamese visual tracking with very deep networks.  In _Comp. Vis. Patt. Recognition_ , 2019.
  * [Liu(2021)] X. Liu.  Deep correlation filters for robust visual tracking.  In _2021 IEEE International Conference on Multimedia and Expo (ICME)_ , pages 1–6, 2021.
  * [Lukežič et al.(2020)Lukežič, Matas, and Kristan] Alan Lukežič, Jiří Matas, and Matej Kristan.  D3S – a discriminative single shot segmentation tracker.  In _Comp. Vis. Patt. Recognition_ , pages 7133–7142, 2020.
  * [Maeno et al.(2013)Maeno, Nagahara, Shimada, and Taniguchi] Kazuki Maeno, Hajime Nagahara, Atsushi Shimada, and Rin-ichiro Taniguchi.  Light field distortion feature for transparent object recognition.  In _Comp. Vis. Patt. Recognition_ , pages 2786–2793, 2013.
  * [Matej Kristan, et al.(2020)] Matej Kristan, et al.  The eighth visual object tracking VOT2020 challenge results.  In _European Conference on Computer Vision Workshops_ , 2020.
  * [Metzger et al.(2021)Metzger, Mortimer, and Wuensche] Kai A. Metzger, Peter Mortimer, and Hans-Joachim Wuensche.  A fine-grained dataset and its efficient semantic segmentation for unstructured driving scenarios.  In _Proc. Int. Conf. Pattern Recognition_ , pages 7892–7899, 2021\.
  * [Mueller et al.(2016)Mueller, Smith, and Ghanem] Matthias Mueller, Neil Smith, and Bernard Ghanem.  A benchmark and simulator for uav tracking.  In _Proc. European Conf. Computer Vision_ , pages 445–461, 2016.
  * [Muller et al.(2018)Muller, Bibi, Giancola, Alsubaihi, and Ghanem] Matthias Muller, Adel Bibi, Silvio Giancola, Salman Alsubaihi, and Bernard Ghanem.  TrackingNet: A large-scale dataset and benchmark for object tracking in the wild.  In _Proc. European Conf. Computer Vision_ , September 2018.
  * [Qi et al.(2019)Qi, Wang, Qin, and Li] Mengshi Qi, Yunhong Wang, Jie Qin, and Annan Li.  Ke-gan: Knowledge embedded generative adversarial networks for semi-supervised scene parsing.  In _Comp. Vis. Patt. Recognition_ , 2019.
  * [Richter et al.(2017)Richter, Hayder, and Koltun] Stephan R Richter, Zeeshan Hayder, and Vladlen Koltun.  Playing for benchmarks.  In _Int. Conf. Computer Vision_ , pages 2213–2222, 2017.
  * [Ros et al.(2016)Ros, Sellart, Materzynska, Vazquez, and Lopez] German Ros, Laura Sellart, Joanna Materzynska, David Vazquez, and Antonio M Lopez.  The synthia dataset: A large collection of synthetic images for semantic segmentation of urban scenes.  In _Comp. Vis. Patt. Recognition_ , pages 3234–3243, 2016.
  * [Russakovsky et al.(2015)Russakovsky, Deng, Su, Krause, Satheesh, Ma, Huang, Karpathy, Khosla, Bernstein, et al.] Olga Russakovsky, Jia Deng, Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma, Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al.  Imagenet large scale visual recognition challenge.  _Int. J. Comput. Vision_ , 115(3):211–252, 2015\.
  * [Sajjan et al.(2020)Sajjan, Moore, Pan, Nagaraja, Lee, Zeng, and Song] Shreeyak Sajjan, Matthew Moore, Mike Pan, Ganesh Nagaraja, Johnny Lee, Andy Zeng, and Shuran Song.  Clear grasp: 3d shape estimation of transparent objects for manipulation.  In _Int. Conf. Robotics and Automation_ , pages 3634–3642. IEEE, 2020\.
  * [Saleh et al.(2018)Saleh, Aliakbarian, Salzmann, Petersson, and Alvarez] Fatemeh Sadat Saleh, Mohammad Sadegh Aliakbarian, Mathieu Salzmann, Lars Petersson, and Jose M Alvarez.  Effective use of synthetic data for urban scene semantic segmentation.  In _Proc. European Conf. Computer Vision_ , pages 84–100, 2018.
  * [Valmadre et al.(2017)Valmadre, Bertinetto, Henriques, Vedaldi, and Torr] Jack Valmadre, Luca Bertinetto, Joao Henriques, Andrea Vedaldi, and Philip H. S. Torr.  End-to-end representation learning for correlation filter based tracking.  In _Comp. Vis. Patt. Recognition_ , pages 2805–2813, 2017.
  * [Wang et al.(2021)Wang, Zhou, Wang, and Li] Ning Wang, Wengang Zhou, Jie Wang, and Houqiang Li.  Transformer meets tracker: Exploiting temporal context for robust visual tracking.  In _Comp. Vis. Patt. Recognition_ , pages 1571–1580, 2021.
  * [Wrenninge and Unger(2018)] Magnus Wrenninge and Jonas Unger.  Synscapes: A photorealistic synthetic dataset for street scene parsing.  _arXiv preprint arXiv:1810.08705_ , 2018.
  * [Wu et al.(2015)Wu, Lim, and Yang] Y. Wu, J. Lim, and M. H. Yang.  Object tracking benchmark.  _IEEE Trans. Pattern Anal. Mach. Intell._ , 37(9):1834–1848, 2015.
  * [Wu et al.(2013)Wu, Lim, and Yang] Yi Wu, Jongwoo Lim, and Ming-Hsuan Yang.  Online object tracking: A benchmark.  In _Comp. Vis. Patt. Recognition_ , pages 2411– 2418, 2013.
  * [Xie et al.(2020)Xie, Wang, Wang, Ding, Shen, and Luo] Enze Xie, Wenjia Wang, Wenhai Wang, Mingyu Ding, Chunhua Shen, and Ping Luo.  Segmenting transparent objects in the wild.  In _Proc. European Conf. Computer Vision_ , pages 696–711. Springer, 2020.
  * [Xu et al.(2015)Xu, Nagahara, Shimada, and Taniguchi] Yichao Xu, Hajime Nagahara, Atsushi Shimada, and Rin-ichiro Taniguchi.  Transcut: Transparent object segmentation from a light-field image.  In _Int. Conf. Computer Vision_ , pages 3442–3450, 2015.
  * [Yan et al.(2021)Yan, Peng, Fu, Wang, and Lu] Bin Yan, Houwen Peng, Jianlong Fu, Dong Wang, and Huchuan Lu.  Learning spatio-temporal transformer for visual tracking.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)_ , pages 10448–10457, October 2021.
  * [Zhang et al.(2019)Zhang, Chen, Huang, Lin, and Zhang] Junyi Zhang, Ziliang Chen, Junying Huang, Liang Lin, and Dongyu Zhang.  Few-shot structured domain adaptation for virtual-to-real scene parsing.  In _Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops_ , 2019.

##  Appendix A Per-attribute analysis

Figure 5 shows per-attribute tracking performance on TOTB dataset [Fan et
al.(2021)Fan, Miththanthaya, Harshit, Rajan, Liu, Zou, Lin, and Ling] with and
without training on Trans2k. Most of the trackers improve at all attributes
when trained with Trans2k, which leads to a consistent average tracker per-
attribute boost. Interestingly, after training with Trans2k, the per-attribute
difference between trackers decreases, thus reducing the influence of the
methodological differences. The per-attribute tracker ranks change, but STARK
[Yan et al.(2021)Yan, Peng, Fu, Wang, and Lu] remains the top tracker in all
attributes. The most difficult attributes remain full occlusion and out-of-
view, which is expected since the short-term trackers are not specifically
designed for target re-detection, which is a long-term property. In addition,
the fast motion and motion blur appear to be the next most difficult
attributes. These are the attributes future research should address in
transparent object tracking. The least challenging attributes are illumination
variation and deformation, which are apparently well addressed by the powerful
deep backbone features. Partial occlusion also seems to be addressed well by
the existing trackers.

Figure 5: Per-attribute performance on TOTB [Fan et al.(2021)Fan,
Miththanthaya, Harshit, Rajan, Liu, Zou, Lin, and Ling]. Blue ribbons indicate
trackers trained with opaque training set, while gray indicate performance
after training with Trans2k.

##  Appendix B The role of training set size

To study the required number of transparent object training sequences in the
training regime, we re-trained DiMP [Danelljan et al.(2019b)Danelljan, Bhat,
Van Gool, and Timofte] and SiamBAN [Chen et al.(2020)Chen, Zhong, Li, Zhang,
and Ji] with varying the training dataset size. In particular, the number of
transparent object sequences in the training set (originally 2,039 sequences)
was reduced to 100, 500 and 1000.

The results in Table 4 show that using only 100 sequences already
significantly improves the results for DiMP, while SiamBAN requires at least
500 sequences to achieve a noticeable improvement over its pre-trained version
on opaque datasets. We speculate that the reason is that SiamBAN relies more
on the dataset diversity since it needs to learn features capable of object
localization only by template correlation. On the other hand, the DiMP
backbone only needs to learn features that sufficiently well reflect the
transparent object presence, while the discriminative template is learned
online. However, DiMP further improves with the additional sequences in the
full Trans2k dataset.

Table 4:  Comparison of different sizes of the training dataset Trans2k for SiamBAN and DiMP. The numbers 100, 500, 1000 and 2039 denote number of sequences containing transparent objects, which were used for training. Performance of trackers pre-trained on opaque objects only is indicated by orig. | orig. | 100 | 500 | 1000 | 2039
---|---|---|---|---|---
DiMP | 0.600 | 0.649 | 0.679 | 0.687 | 0.699
SiamBAN | 0.656 | 0.655 | 0.667 | 0.670 | 0.680

##  Appendix C Qualitative examples

In this section we present some qualitative examples of the proposed dataset.
Figure 6 shows objects which was used for rendering the Trans2k. We defined 25
object types and each object type has several instances, which results in
total 148 object instances.

Figure 7 shows three sequences from the proposed Trans2k dataset. Both types
of ground-truth annotations are visualized: bounding boxes as well as
segmentation masks.  Note that while transparent objects flying over a scene
are seldom observed in real life, they convey sufficient level of realism
considering the object and immediate surrounding required to train deep
models.

In Figure 8 we present qualitative comparison of five original trackers and
their versions, retrained using Trans2k. Trackers trained on Trans2k track the
target much more accurately and reliably. Since no methodological changes were
made to the original trackers, these results clearly demonstrate the
effectiveness of the proposed training dataset.

Figure 6: A diverse set of object instances used in Trans2k sequences. Figure
7: Frames from three sequences of the proposed Trans2k dataset. Objects are
labeled by bounding boxes and segmentation masks (cropped in square). Figure
8: Qualitative examples on TOTB dataset. Trackers trained only with opaque
object tracking sequences are shown in blue, trackers trained with Trans2k are
shown in red and the ground is shown in green.


# Determination of the Gluonic D-term and Mechanical Radii of Proton from
Experimental data

Wei Kou<EMAIL_ADDRESS>Institute of Modern Physics, Chinese Academy of
Sciences, Lanzhou 730000, China  University of Chinese Academy of Sciences,
Beijing 100049, China  Rong Wang<EMAIL_ADDRESS>Institute of Modern
Physics, Chinese Academy of Sciences, Lanzhou 730000, China  University of
Chinese Academy of Sciences, Beijing 100049, China  Xurong Chen
<EMAIL_ADDRESS>Institute of Modern Physics, Chinese Academy of Sciences,
Lanzhou 730000, China  University of Chinese Academy of Sciences, Beijing
100049, China  Guangdong Provincial Key Laboratory of Nuclear Science,
Institute of Quantum Matter, South China Normal University, Guangzhou 510006,
China

###### Abstract

We apply a “color” tripole ansatz for describing the $D$-term of the proton.
By fitting the experimental data of the vector meson J/$\psi$ and $\phi$
photoproductions near the thresholds, we firstly obtained the gluonic $D$-term
of the proton. $D_{g}(0)$ is estimated to be $-2.16\pm 0.42$ for J/$\psi$ and
$-1.31\pm 0.48$ for $\phi$, and the mechanical root mean square radius of
proton is estimated to be $0.61\pm 0.29$ fm for $\phi$ and $0.42\pm 0.11$ fm
for J/$\psi$.

##  I Introduction

Recently in the study of hadron physics, the mechanical properties and the
quantum chromodynamics (QCD) Gross and Wilczek (1973a, b, 1974) structure of
the proton are among the hot topics under discussions. The proton is composed
of quarks and gluons according to the quark model and the modern QCD theory.
Although one can break the atomic nucleus to get the nucleons inside, we
cannot observe the individual quarks and gluons by breaking up the nucleons.
This is because of the color confinement of QCD theory. Nevertheless the
proton structure can be revealed in the asymptotic region under high energy
transfer. There are some theoretical ways to envisage the internal structure
of the proton. The QCD factorization theory indicates that the scattering
amplitudes can be expressed in terms of the generalized parton distribution
(GPD) Ji (1997a); Müller  _et al._ (1994); Radyushkin (1996) in the
high-$Q^{2}$ region. Although less rigorous, there are some phenomenal models
which also successfully explain parts of physical processes.

The basic mechanical properties of proton can be deduced from the
gravitational form factors (GFFs) of Energy-Momentum-Tensor (EMT) Ji (1997b);
Pagels (1966); Belitsky and Radyushkin (2005). Matrix elements of EMT contain
basic mechanical properties such as mass and angular momentum. Over the past
two decades people have gradually understood the origins of the proton mass
and spin within the framework of quantum field theory and from the prior
experimental studies. But much less is known about the $D$-term of the proton,
which is one of the GFFs that reflects the shear force and the pressure inside
the proton Polyakov (2003); Polyakov and Schweitzer (2018). The old fashion
models on the $D$-term were performed in the bag model Ji  _et al._ (1997) and
the chiral quark soliton model ($\chi$QSM) Petrov  _et al._ (1998). In 2018 V.
Burkert, et. al. Burkert  _et al._ (2018) adopted GPD measurement from deeply
virtual Compton scattering (DVCS) Ji (1997b, a) to firstly give a direct
mapping of the pressure distribution inside proton with the extracted $D$-term
$D_{q}(0)$ of quark. In the following year, P. Shanahan and W. Detmold gave
the gluonic and total $D$-term by Lattice QCD (LQCD) calculation Shanahan and
Detmold (2019a, b). Hatta et. al. have analyzed the $D$-term by studying the
photoproduction process of heavy quarkomium from the holographic QCD with
Operator Product Expansion theory Hatta and Yang (2018); Boussarie and Hatta
(2020); Hatta and Strikman (2021). H. Dutrieux et. al. apply the artificial
neural network technique recently to analyze the DVCS process in extracting
the $D$-term of the proton Dutrieux  _et al._ (2021).

On experimental side, the extraction of the gluonic $D$-term is rarely
mentioned in the literatures. The exclusive leptoproduction of vector mesons
$ep\rightarrow e^{\prime}\gamma^{*}p\rightarrow e^{\prime}p^{\prime}V$ can
broadly unveil the details of the structure of proton. The proton has only the
up and down valence components, and the light mesons can be used to detect
only these partons in proton. But in the Regge limit, the heavy quarkonium
like J/$\psi\ \mathrm{and}\ \Upsilon$ can be used to probe the gluon part of
the proton. It is similar to the method of extracting the gravitational radius
of the proton from the GFFs of EMT Kharzeev (2021); Wang  _et al._ (2021). In
this work, we apply the vector meson $\phi,\ \mathrm{J}/\psi$ near-threshold
photoproduction data Mibe  _et al._ (2005); Ali  _et al._ (2019) to extract
the gluons’ contribution of $D$-term $D_{g}(0)$. Then we present the internal
pressure shear force distributions of the proton with the obtained $D$-term.

The organization of the paper is as follows. The EMT and $D$-term of the
proton are briefly introduced in Sec. II. The result on the parameterized
$D$-term extracted from vector meson photoproduction is present in Sec. III.
The pressure and shear forces distribution inside the proton is shown in Sec.
IV. We also get the $D$-term radius defined by Zahed’s work Mamo and Zahed
(2021) and mechanical radius defined by Polyakov and Schweitzer (2018) in Sec.
V. A summary is given in Sec. VI.

##  II Energy-momentum tensor and $\bf D$-term

The QCD EMT is the conserved current associated with the space-time
translational symmetry based on Noether’s theorem. The total tensor satisfies
the conservation law:

$\displaystyle\partial^{\mu}\hat{T}_{\mu\nu}=0,$ (1)
$\displaystyle\hat{T}_{\mu\nu}=\sum_{q}\hat{T}_{\mu\nu}^{q}+\hat{T}_{\mu\nu}^{g}.$

The covariant normalization condition is set as $\left\langle p^{\prime}\mid
p\right\rangle=2p^{0}(2\pi)^{3}\delta^{(3)}\left(\bm{p}^{\prime}-\bm{p}\right)$,
where the state $|\bm{p}\rangle$ is one particle state. The form factors of
EMT depend the kinematic variables
$P=\frac{1}{2}\left(p^{\prime}+p\right),\Delta=p^{\prime}-p,t=\Delta^{2}$. As
the proton is the focus of this paper, the matrix element of the QCD EMT of
the Spin-$\frac{1}{2}$ hadron is defined as Polyakov and Schweitzer (2018),

$\displaystyle\left\langle
p^{\prime},s^{\prime}\left|\hat{T}_{\mu\nu}^{a}(x)\right|p,s\right\rangle=\bar{u}^{\prime}\left[A^{a}(t)\frac{\gamma_{\\{\mu}P_{\nu\\}}}{2}\right.$
(2)
$\displaystyle\left.+B^{a}(t)\frac{iP_{\\{\mu}\sigma_{\nu\\}\rho}\Delta^{\rho}}{4m}\right.\left.+D^{a}(t)\frac{\Delta_{\mu}\Delta_{\nu}-g_{\mu\nu}\Delta^{2}}{4m}\right.$
$\displaystyle+m\bar{c}^{a}(t)g_{\mu\nu}\bigg{]}ue^{i\left(p^{\prime}-p\right)x},$

where the normalization of the spinor is $\bar{u}(p,s)u(p,s)=2m$, with the
notation $a_{\\{\mu}b_{\nu\\}}=a_{\mu}b_{\nu}+a_{\nu}b_{\mu}$. The individual
quark and gluon form factors $A^{a}(t),\ B^{a}(t),\ D^{a}(t)$ and
$\bar{c}^{a}(t)$ depend on the renormalization scale Polyakov and Schweitzer
(2018). From the lattice QCD calculation Alexandrou  _et al._ (2017), the
$B^{a}(t)$ is suggested to be zero for the gluon.

The other GFFs besides the $D$-term form factor contain the information of the
mass and the spin distributions of the proton. In this paper we mainly focus
on the $D$-term form factors which can describe the coupling between the
vector meson and the proton. The GFFs related to the vector production is
discussed in Ref. Hatta and Strikman (2021). The tripole parametrization
ansatz for the $D$-term is suggested by the perturbative counting rule at
large $t$ Tanaka (2018); Hatta and Strikman (2021),

$D_{q,g}(t)=\frac{D_{q,g}(0)}{\left(1-t/m_{D}^{2}\right)^{3}}.$ (3)

The tripole form contributes more to the scattering amplitude in the small $t$
region.

##  III Extraction of $\bf{D_{g}(0)}$

Usually both GFF $A(t)$ and GFF $D(t)$ couple to the vector meson
photoproduction. $D$-term dominates at $t$ approaching zero and it has the
steeper $t$-dependence. To extract the $D$-term from the vector meson near-
threshold photoproduction data, we assume that Hatta and Yang (2018),

$\frac{d\sigma}{dt}\sim
D^{2}(t)=\frac{D_{g}^{2}(0)}{\left(1-t/m_{D}^{2}\right)^{6}}.$ (4)

Since the form factors are evaluated at a large scale $\mu^{2}=Q^{2}$, in an
approximation we can use the following asymptotic results,

$\displaystyle A_{q}(0)$ $\displaystyle\approx\frac{n_{f}}{4C_{F}+n_{f}},\
\quad A_{g}(0)$ $\displaystyle\approx\frac{4C_{F}}{4C_{F}+n_{f}},\ \quad
D_{q}(0)$ $\displaystyle\approx\frac{n_{f}}{4C_{F}}D_{g}(0),$ (5)

where $C_{F}=\frac{N_{c}^{2}-1}{2N_{c}}$ is the Casimir operator with
$N_{c}=3$ and $n_{f}=3$ represents the number of light flavors in the proton.
The aim of the analysis is to extract $D(0)$, therefore we use the third row
in Eq. (5), which relates the quark $D$-term and the gluon $D$-term with
$D_{q}(0)=\frac{9}{16}D_{g}(0)$. In analyzing the experimental data of the
photoproductions of $\phi$ and J/$\psi$, we assume that only the gluon
component of the proton are involved in the color interaction near the
threshold. This is reasonable given that s and c quarks in the nucleon are
generated by the gluon splitting $g\to s\bar{s}/c\bar{c}$ Frankfurt and
Strikman (2002). The gluon $D(0)$ is directly regarded as the total one, as
$D_{g}(0)=D(0)$, because of the weak gravitational field approximation
Kharzeev (2021). For $\phi$ and J/$\psi$ meson productions, Eq. (4) is used to
fit the experimental data Mibe  _et al._ (2005); Ali  _et al._ (2019). The
cutoff parameter $m_{D}$ is also a free parameter to be determined by the
fits.

Figure 1: (Color online) (a) The differential cross section of $\phi$
photoproduction near threshold at $E_{\gamma}=$ 1.62 GeV by LEPS Collaboration
Mibe  _et al._ (2005). The red curve is our fit with the tripole form, with
$m_{D}^{2}=1.25\pm 0.60$ GeV and $D_{g}(0)=-1.31\pm 0.48$. (b) The result by
Ali  _et al._ (2019) which have the similar marks by (a).

The near-threshold photoproduction data of $\phi$ and J/$\psi$ vector mesons
from the LEPS Mibe  _et al._ (2005) and GlueX Ali  _et al._ (2019)
Collaborations with the fits of the tripole form $D$-term are shown in FIG. 1
and FIG. 1, respectively. Note that only the gluon contribution is considered.
The obtained values of the $D$-term are shown in FIG. 2, compared with the
result from LQCD Shanahan and Detmold (2019a). The extracted parameters and
the qualities of the fits are summarized in Table 1. The results of $\Upsilon$
will be verified with the future high-precision experiments.

Figure 2: (Color online) The gluon contribution to the $D$-term inside proton
extracted from the experimental data (black dots) and the LQCD calculation
Shanahan and Detmold (2019a) (blue circle). The renormalization scale is
indicated with the horizontal axis, which is defined as the the binding energy
of the vector meson. The scale of LQCD calculation corresponds the
$\overline{\mathrm{MS}}$ scheme at a scale of $\mu=2$ GeV. The magenta doted
line marks the binding energy of $\Upsilon$ meson.

##  IV Pressure and shear force distribution inside the proton

In the previous section, the values of $D_{g}(0)$ are given by analyzing the
experimental data. Generally, $D(t)$ contains the information of the internal
pressure and shear force distributions of the proton. Using the tripole
parameterization for $D_{g}(t)$ and $-t=K^{2}$, we get,

$D_{g}(K)=\frac{D_{g}(0)}{\left(1+\frac{K^{2}}{m_{D}^{2}}\right)^{3}}.$ (6)

The Fourier transform of Eq. (6) gives the $D$-term in the three-dimensional
coordinate space ($E=m_{N}$), as,

$\displaystyle\tilde{D}_{g}(r)=D_{g}(0)\int\frac{d^{3}K}{2E(2\pi)^{3}}\frac{e^{-iK\cdot
r}}{\left(1+\frac{K^{2}}{m_{D}^{2}}\right)^{3}}=\frac{D_{g}(0)}{64\pi
m_{N}}\left(\frac{1}{m_{D}}+r\right)m_{D}^{4}\exp(-m_{D}r).$ (7)

The pressure distribution $p(r)$ and shear force $s(r)$ inside the proton then
can be computed with $\tilde{D}_{g}(r)$ Polyakov and Schweitzer (2018):

$\displaystyle p(r)$
$\displaystyle=\frac{1}{3}\frac{1}{r^{2}}\frac{d}{dr}r^{2}\frac{d}{dr}\tilde{D}_{g}(r)=\frac{D_{g}(0)}{192\pi
m_{N}}(m_{D}r-3)m_{D}^{5}\exp(-m_{D}r),$ (8) $\displaystyle s(r)$
$\displaystyle=-\frac{1}{2}\frac{d}{dr}\frac{1}{r}\frac{d}{dr}\tilde{D}_{g}(r)=-\frac{D_{g}(0)}{128\pi
m_{N}}m_{D}^{5}r\exp(-m_{D}r).$

According to Eq. (7–8), we calculate the pressure and shear force
distributions inside the proton with the associated uncertainties. The
uncertainties come from the uncertainties of the parameters $D_{g}(0)$ and
$m_{D}$. Using Eq. (8) the pressure and shear force distributions inside the
proton are obtained and displayed in FIG. 3 to FIG. 4.

Figure 3: (Color online) (a) The radial pressure distribution inside the
proton based on the $D$-term extracted from the $\phi$ photoproduction.
$r^{2}p(r)$ is the pressure distribution from the interactions of the gluons
and $r$ is the radial distance to the center of the proton. The solid curve
shows the center value of the pressure and band represents the statistical
uncertainty. (b) The radial pressure distribution inside the proton based on
the $D$-term extracted from the J/$\psi$ photoproduction. The description as
same as (a).

Figure 4: (Color online) (a) The shear force distribution inside the proton
based on the $D$-term extracted from the $\phi$ photoproduction. $r^{2}s(r)$
is the pressure distribution from the interactions of the gluons and $r$ is
the radial distance to the center of the proton. The solid curve shows the
center value of the pressure and band represents the statistical uncertainty.
(b) The shear force distribution inside the proton based on the $D$-term
extracted from the J/$\psi$ photoproduction. The description as same as (a).

As shown in this analysis, the $D_{g}(t)$ extracted from the exclusive $\phi$
and J/$\psi$ production channels are different, which results in the different
pressure and shear force distributions as well. According to our uncertainty
analysis, the uncertainty of the pressure and shear force distributions
largely depend on the error of parameter $m_{D}$ from the fit. Constraining
better the slope parameter $m_{D}$ would better to give the precise pressure
and shear force distributions inside the proton.

##  V Root Mean square radii of proton

According to the definition of root mean square radius Polyakov and Schweitzer
(2018), we can get the corresponding radius of $D$-term. To compare our
results, we now introduce the mechanical force there. In previous section, we
use the experimental data to have the fitting parameters and the pressure and
shear force distributions $p(r)$ and $s(r)$. According the argument from
Perevalova  _et al._ (2016), authors thought that for the mechanical stability
of the system the corresponding force must be directed outwards. Therefore the
local criterion for the mechanical stability can be formulated as the
inequality,

$\frac{2}{3}s(r)+p(r)>0.$ (9)

Of course, our results satisfy the inequality. We also use the parameters from
fitting data of J/$\psi$ to give the figure as FIG. 5.

Figure 5: (Color online) The positive combination satisfy the relation from
Eq. (9) using experimental data by J/$\psi$ production.

The positive combination $\frac{2}{3}s(r)+p(r)$ is described as the meaning of
the normal force distribution in the proton system. The mechanical radius for
proton is introduced as Polyakov and Schweitzer (2018)

$\left\langle r^{2}\right\rangle_{\mathrm{mec}}=\frac{\int
d^{3}rr^{2}\left[\frac{2}{3}s(r)+p(r)\right]}{\int
d^{3}r\left[\frac{2}{3}s(r)+p(r)\right]}=\frac{6D_{g}(0)}{\int_{-\infty}^{0}dtD_{g}(t)}.$
(10)

For the tripole ansatz like Eq. (3) the resulting mechanical mean square
radius is $\langle r^{2}\rangle_{\mathrm{mech}}=12/m_{D}^{2}$. To compare with
the $D$-term radius, we use the classical radius definition like previous
works Adamuscin  _et al._ (2012); Song (2018); Kumano  _et al._ (2019); Wang
_et al._ (2021)

$\left\langle
r^{2}\right\rangle_{\mathrm{D}}=\frac{6}{D_{g}(0)}\frac{dD_{g}(t)}{dt}\Bigg{|}_{t=0}=\frac{18}{m_{D}^{2}}.$
(11)

Table 1: The least-square fit results to the experimental data Mibe  _et al._ (2005); Ali  _et al._ (2019) using Eq. (4) and the obtained parameters, mechanical radii and $D$-term radii for the tripole parametrization. The last line come from Burkert  _et al._ (2021) and we calculate the radii corresponding the DVCS data.  Data | $D_{g}(0)$ |  $m_{D}^{2}$ (GeV2) |  $\sqrt{\langle r^{2}\rangle_{\mathrm{mec}}}$ (fm) |  $\sqrt{\langle r^{2}\rangle_{\mathrm{D}}}$ (fm)
---|---|---|---|---
$\phi$ | $-1.31\pm 0.48$ | $1.25\pm 0.60$ | $0.61\pm 0.29$ | $0.75\pm 0.36$
J/$\psi$ | $-2.16\pm 0.42$ | $2.70\pm 0.73$ | $0.42\pm 0.11$ | $0.51\pm 0.14$
DVCS Burkert  _et al._ (2021) | $-1.47\pm 0.06$ | $1.02\pm 0.13$ | $0.68\pm 0.09$ | $0.83\pm 0.10$

Table 1 concludes the different radii of proton using Eq. (10) and (11). It
can be clearly seen from these results that the mechanical radii are obviously
smaller than the $D$-term radii. Both sets of experimental data illustrate
this conclusion.

##  VI Discussions and Summary

One important outcome of the QCD factorization theorem is that for the process
dominated by the two-gluon exchange, the $t$-dependence should reach a
universal limit which is independent of the flavor of the quark constituents
of the meson Brodsky  _et al._ (1994); Frankfurt and Strikman (2002), at large
$Q^{2}$ and fixed $x$. The mechanism for such universality is the transverse
squeezing of the meson wave function. From LQCD calculation Shanahan and
Detmold (2019a, b), the authors concluded that the gluons play an important
role in the internal dynamics of the proton, which is quite distinct from that
of quarks. Hence in this limit the $t$-dependence of the amplitude is given
solely by the gluons’ contribution. Therefore it is a reasonable assumption to
extract the gluonic GFFs from the differential cross section data of the
vector meson photoproductions in the region close to the threshold energy.

In this work, the gluons’ contribution to the proton $D$-term and the pressure
distribution inside the proton are determined from the vector meson
photoproduction experiments near the threshold. The vector meson mass
dependence of the gluonic $D$-term is shown in FIG. 2. Our speculation for the
dependence is that the gluonic $D$-term is scale-dependent and different meson
probes correspond to different energy scales. We note that a recent work
presents a determination of the shear force distribution in the proton using a
DVCS experiment which measures the GPDs of the proton Burkert  _et al._
(2021). Our work sketches an alternative method beside the DVCS experiment in
studying the $D$-term, pressure and shear forces inside the proton. More
precise data at JLab and future EIC in US Accardi  _et al._ (2016); Abdul
Khalek  _et al._ (2021) and China Chen (2018); Chen  _et al._ (2020); Anderle
_et al._ (2021) are of importance for probing the $D$-term form factor and
enhancing our understanding of the QCD structure of the proton.

###### Acknowledgements.

We thank Prof. Fan WANG for the fruitful discussions and the suggestions. This
work is supported by the Strategic Priority Research Program of Chinese
Academy of Sciences under the Grant NO. XDB34030301.

## References

  * Gross and Wilczek (1973a) D. J. Gross and F. Wilczek, Phys. Rev. Lett. 30, 1343 (1973a).
  * Gross and Wilczek (1973b) D. J. Gross and F. Wilczek, Phys. Rev. D8, 3633 (1973b).
  * Gross and Wilczek (1974) D. J. Gross and F. Wilczek, Phys. Rev. D9, 980 (1974).
  * Ji (1997a) X.-D. Ji, Phys. Rev. Lett. 78, 610 (1997a), arXiv:hep-ph/9603249 [hep-ph] .
  * Müller  _et al._ (1994) D. Müller, D. Robaschik, B. Geyer, F. M. Dittes, and J. Hořejši, Fortsch. Phys. 42, 101 (1994), arXiv:hep-ph/9812448 [hep-ph] .
  * Radyushkin (1996) A. V. Radyushkin, Phys. Lett. B380, 417 (1996), arXiv:hep-ph/9604317 [hep-ph] .
  * Ji (1997b) X.-D. Ji, Phys. Rev. D55, 7114 (1997b), arXiv:hep-ph/9609381 [hep-ph] .
  * Pagels (1966) H. Pagels, Phys. Rev. 144, 1250 (1966).
  * Belitsky and Radyushkin (2005) A. V. Belitsky and A. V. Radyushkin, Phys. Rept. 418, 1 (2005), arXiv:hep-ph/0504030 [hep-ph] .
  * Polyakov (2003) M. V. Polyakov, Phys. Lett. B 555, 57 (2003), arXiv:hep-ph/0210165 .
  * Polyakov and Schweitzer (2018) M. V. Polyakov and P. Schweitzer, Int. J. Mod. Phys. A33, 1830025 (2018), arXiv:1805.06596 [hep-ph] .
  * Ji  _et al._ (1997) X.-D. Ji, W. Melnitchouk, and X. Song, Phys. Rev. D 56, 5511 (1997), arXiv:hep-ph/9702379 .
  * Petrov  _et al._ (1998) V. Yu. Petrov, P. V. Pobylitsa, M. V. Polyakov, I. Bornig, K. Goeke, and C. Weiss, Phys. Rev. D57, 4325 (1998), arXiv:hep-ph/9710270 [hep-ph] .
  * Burkert  _et al._ (2018) V. D. Burkert, L. Elouadrhiri, and F. X. Girod, Nature 557, 396 (2018).
  * Shanahan and Detmold (2019a) P. E. Shanahan and W. Detmold, Phys. Rev. D99, 014511 (2019a), arXiv:1810.04626 [hep-lat] .
  * Shanahan and Detmold (2019b) P. E. Shanahan and W. Detmold, Phys. Rev. Lett. 122, 072003 (2019b), arXiv:1810.07589 [nucl-th] .
  * Hatta and Yang (2018) Y. Hatta and D.-L. Yang, Phys. Rev. D98, 074003 (2018), arXiv:1808.02163 [hep-ph] .
  * Boussarie and Hatta (2020) R. Boussarie and Y. Hatta, Phys. Rev. D101, 114004 (2020), arXiv:2004.12715 [hep-ph] .
  * Hatta and Strikman (2021) Y. Hatta and M. Strikman, (2021), arXiv:2102.12631 [hep-ph] .
  * Dutrieux  _et al._ (2021) H. Dutrieux, C. Lorcé, H. Moutarde, P. Sznajder, A. Trawiński, and J. Wagner, (2021), arXiv:2101.03855 [hep-ph] .
  * Kharzeev (2021) D. E. Kharzeev, (2021), arXiv:2102.00110 [hep-ph] .
  * Wang  _et al._ (2021) R. Wang, W. Kou, Y.-P. Xie, and X. Chen, (2021), arXiv:2102.01610 [hep-ph] .
  * Mibe  _et al._ (2005) T. Mibe _et al._ (LEPS), Phys. Rev. Lett. 95, 182001 (2005), arXiv:nucl-ex/0506015 .
  * Ali  _et al._ (2019) A. Ali _et al._ (GlueX), Phys. Rev. Lett. 123, 072001 (2019), arXiv:1905.10811 [nucl-ex] .
  * Mamo and Zahed (2021) K. A. Mamo and I. Zahed, (2021), arXiv:2103.03186 [hep-ph] .
  * Alexandrou  _et al._ (2017) C. Alexandrou, M. Constantinou, K. Hadjiyiannakou, K. Jansen, C. Kallidonis, G. Koutsou, A. Vaquero Avilés-Casco, and C. Wiese, Phys. Rev. Lett. 119, 142002 (2017), arXiv:1706.02973 [hep-lat] .
  * Tanaka (2018) K. Tanaka, Phys. Rev. D98, 034009 (2018), arXiv:1806.10591 [hep-ph] .
  * Frankfurt and Strikman (2002) L. Frankfurt and M. Strikman, Phys. Rev. D 66, 031502 (2002), arXiv:hep-ph/0205223 .
  * Perevalova  _et al._ (2016) I. A. Perevalova, M. V. Polyakov, and P. Schweitzer, Phys. Rev. D 94, 054024 (2016), arXiv:1607.07008 [hep-ph] .
  * Adamuscin  _et al._ (2012) C. Adamuscin, S. Dubnicka, and A. Z. Dubnickova,  _From quarks and gluons to hadrons and nuclei. Proceedings, International Workshop on Nuclear Physics, 33rd course, Erice, Italy, September 16-24, 2011_ , Prog. Part. Nucl. Phys. 67, 479 (2012).
  * Song (2018) Q.-T. Song,  _Proceedings, 26th International Workshop on Deep Inelastic Scattering and Related Subjects (DIS 2018): Port Island, Kobe, Japan, April 16-20, 2018_ , PoS DIS2018, 142 (2018), arXiv:1808.03422 [hep-ph] .
  * Kumano  _et al._ (2019) S. Kumano, Q.-T. Song, and O. V. Teryaev,  _Proceedings, 23rd International Symposium on Spin Physics (SPIN2018): Ferrara, Italy, September 10-14, 2018_ , PoS SPIN2018, 074 (2019), arXiv:1902.04333 [hep-ph] .
  * Burkert  _et al._ (2021) V. D. Burkert, L. Elouadrhiri, and F. X. Girod, (2021), arXiv:2104.02031 [nucl-ex] .
  * Brodsky  _et al._ (1994) S. J. Brodsky, L. Frankfurt, J. F. Gunion, A. H. Mueller, and M. Strikman, Phys. Rev. D50, 3134 (1994), arXiv:hep-ph/9402283 [hep-ph] .
  * Accardi  _et al._ (2016) A. Accardi _et al._ , Eur. Phys. J. A 52, 268 (2016), arXiv:1212.1701 [nucl-ex] .
  * Abdul Khalek  _et al._ (2021) R. Abdul Khalek _et al._ , (2021), arXiv:2103.05419 [physics.ins-det] .
  * Chen (2018) X. Chen, PoS DIS2018, 170 (2018), arXiv:1809.00448 [nucl-ex] .
  * Chen  _et al._ (2020) X. Chen, F.-K. Guo, C. D. Roberts, and R. Wang, Few Body Syst. 61, 43 (2020), arXiv:2008.00102 [hep-ph] .
  * Anderle  _et al._ (2021) D. P. Anderle _et al._ , (2021), arXiv:2102.09222 [nucl-ex] .