11institutetext: Department of Electrical and Computer Engineering, Duke
University, NC, USA
corresponding author 11email<EMAIL_ADDRESS>Department of Radiology, Duke University, NC, USA 33institutetext: Department
of Computer Science, Duke University, NC, USA 44institutetext: Department of
Biostatistics & Bioinformatics, Duke University, NC, USA 44email:
<EMAIL_ADDRESS>
# The Intrinsic Manifolds of Radiological Images and their Role in Deep
Learning

Nicholas Konz*(✉) 11 0000-0003-0230-1598 Hanxue Gu 11 0000-0003-2622-753X
Haoyu Dong 22 0000-0002-5132-0341 Maciej A. Mazurowski(✉) 11223344
0000-0003-4202-8602

###### Abstract

The manifold hypothesis is a core mechanism behind the success of deep
learning, so understanding the intrinsic manifold structure of image data is
central to studying how neural networks learn from the data. Intrinsic dataset
manifolds and their relationship to learning difficulty have recently begun to
be studied for the common domain of natural images, but little such research
has been attempted for radiological images. We address this here. First, we
compare the intrinsic manifold dimensionality of radiological and natural
images. We also investigate the relationship between intrinsic dimensionality
and generalization ability over a wide range of datasets. Our analysis shows
that natural image datasets generally have a higher number of intrinsic
dimensions than radiological images. However, the relationship between
generalization ability and intrinsic dimensionality is much stronger for
medical images, which could be explained as radiological images having
intrinsic features that are more difficult to learn. These results give a more
principled underpinning for the intuition that radiological images can be more
challenging to apply deep learning to than natural image datasets common to
machine learning research. We believe rather than directly applying models
developed for natural images to the radiological imaging domain, more care
should be taken to developing architectures and algorithms that are more
tailored to the specific characteristics of this domain. The research shown in
our paper, demonstrating these characteristics and the differences from
natural images, is an important first step in this direction.

###### Keywords:

Radiology Generalization Dimension Manifold

##  1 Introduction

Although using deep learning-based methods to solve medical imaging tasks has
become common practice, there lacks a strong theoretical understanding and
analysis of the effectiveness of such methods. This could be a potential
problem for future algorithm development, as most successful methods for
medical images are adapted from techniques solving tasks using natural image
datasets [6]. Due to the apparent differences in relevant semantics between
natural and medical domains [20], it is not clear what design choices are
necessary when adapting these networks to medical images. This difference in
domain is especially true when considering radiological images. Our goal is to
provide a better, quantified footing for developing such radiology-specialized
methods, by (1) analyzing the underlying structure of common radiological
image datasets and determining how it relates to learning dynamics and
generalization ability and (2) comparing these characteristics to common
natural image datasets.

Figure 1: Sample images from each dataset studied. From the left: CheXpert
[15], OAI [23], MURA [25], DBC [26], BraTS 2018 [19], Prostate-MRI [29] and
RSNA-IH-CT [10] (see Sec. 3).

The Manifold Hypothesis [9, 30, 5] states that high-dimensional data, such as
images, can be well described by a much smaller number of features/degrees of
freedom than the number of pixels in an image; this number is the intrinsic
dimension (ID) of the dataset. This is central to deep computer vision because
these abstract visual features can be learned from data, allowing inference in
a tractable, lower-dimensional space. It is therefore important to study the
relationship of the ID of datasets with the learning process of deep models.
This was recently explored for standard natural image datasets [24], but a
similarly comprehensive study has yet to be conducted for radiological image
datasets, which is important because of the apparent differences between these
two domains.

####  1.0.1 Contributions.

Our contributions are summarized as follows.

  1. 1.

We investigate the intrinsic manifold structure of common radiology datasets
(Fig. 1), and find that their IDs are indeed much lower than the number of
pixels, and also generally lower than for natural image datasets (Fig. 2).

  2. 2.

We also find that classification is generally harder with radiology datasets
than natural images for moderate-to-low training set sizes.

  3. 3.

We show that classification performance is negatively linear to dataset ID
within both data domains, invariant to training set size. However, the
absolute value of the slope of this relationship is much higher for
radiological data than for natural image data.

  4. 4.

We test these linearity findings on a wide range of common classification
models, and find that performance for radiological images is almost
independent of the choice of model, relying instead on the ID of the dataset.

Our results show that while the ID of a dataset affects the difficulty of
learning from this dataset, what also matters is the complexity of the
intrinsic features themselves, which we find to be indicated by the sharpness
of the relationship between generalization ability and ID, that is more severe
for radiological images. These findings give experimental evidence for the
differences in intrinsic dataset structure and learning difficulty between the
two domains, which we believe is the first step towards a more principled
foundation for developing deep methods specially designed for radiology.

Figure 2: Intrinsic dimension of radiological (blue) and natural image
(orange) datasets, the latter from [24]. Figure recommended to be viewed in
color.

####  1.0.2 Related Work.

Intrinsic dimension (ID) estimation methods ([3]), have only recently been
applied to datasets used in modern computer vision, beginning with [24], which
explored the ID of common natural image datasets and how it relates to
learning ability and generalization. There have been a few studies of the ID
of medical datasets, e.g., [7], but these are targeted at an individual
modality or dataset. The most common ID estimator obtains a maximum-likelihood
(MLE) solution for the ID by modeling the dataset as being sampled from a
locally uniform Poisson process on the intrinsic data manifold [17]. Other
estimators exist ([11, 8]), but these are unreliable estimators for images
[24], so we use the MLE estimator in this work. Note that we do not estimate
dataset ID with some learned latent space dimension found by training an
autoencoder-type model or similar; relatedly, we also do not study the ID of
the learned feature structure, or parameters of a trained feature extraction
model, e.g., [1, 4]. In contrast, we study the intrinsic, model-independent
structure of the dataset itself.

##  2 Methods: Intrinsic Dimension Estimation

Consider some dataset $\mathcal{D}\subset\mathbb{R}^{d}$ of $N$ images, where
$d$ is the extrinsic dimension, i.e., the number of pixels in an image. The
Manifold Hypothesis [9] assumes that the datapoints $x\in\mathcal{D}$ lie near
some low-dimensional manifold $\mathcal{M}\subseteq\mathbb{R}^{d}$ that can be
described by an intrinsic dimension $\mathrm{dim}(\mathcal{M})=m\ll d$.

In order to obtain an estimate of $m$, [17] models the data as being sampled
from a Poisson process within some $m$-dimensional sphere at each datapoint
$x$; the density of points is assumed to be approximately constant within the
radius of the sphere. Rather than specifying the radius as a hyperparameter,
the authors set this radius to be the distance of the $k^{th}$ nearest
neighboring datapoint to $x$ (instead specifying $k$). By maximizing the
likelihood of the data given the parameters of the Poisson model, we then
obtain an ID for $x$ of

$\hat{m}_{k}(x)=\left[\frac{1}{k-1}\sum_{j=1}^{k-1}\log\frac{T_{k}(x)}{T_{j}(x)}\right]^{-1},$
(1)

where $T_{j}(x)$ is the $\ell_{2}$ distance from $x$ to its $j^{th}$ nearest
neighbor. The authors of [18] then showed that MLE can be used again to obtain
an estimate for the global dataset ID, as

$\hat{m}_{k}=\left[\frac{1}{N}\sum_{i=1}^{n}\hat{m}_{k}\left(x_{i}\right)^{-1}\right]^{-1}=\left[\frac{1}{N(k-1)}\sum_{i=1}^{N}\sum_{j=1}^{k-1}\log\frac{T_{k}\left(x_{i}\right)}{T_{j}\left(x_{i}\right)}\right]^{-1},$
(2)

which is the estimator that we use for this work. Note here that $k$ is a
hyperparameter of the estimator; just as in [24] we set $k=20$, a moderate
value as recommended by [17].

##  3 Datasets and Tasks

In this work, we use the common task of binary classification to analyze the
effect of dataset intrinsic dimension on the learning of radiological images.
We chose radiology datasets that are varied in modality and well-
representative of the domain, while being large enough for a broad study of
training set sizes for at least one realistic classification task. We explore
using alternate tasks for the same datasets in Section 4.2.2.

The datasets are as follows. (1) CheXpert [15], where we detect pleural
effusion in chest X-ray images. Next is (2) the Knee Osteoarthritis Initiative
(OAI) [23], where we select the OAI-released screening packages 0.C.2 and
0.E.1 containing knee X-ray images. Following [2, 31], we build a binary
osteoarthritis (OA) detection dataset by combining Kellgren-Lawrence (KL)
scores of $\\{0,1\\}$ as OA-negative and combining scores of $\\{2,3,4\\}$ as
positive. (3) is MURA [25], where we detect abnormalities in musculoskeletal
X-ray images. Next, (4) is the Duke Breast Cancer MRI (DBC) dataset [26],
where we detect cancer in fat-saturated breast MRI volume slices; we take
slices containing a tumor bounding box to be positive, and all other slices at
least five slices away from the positives to be negative. We follow this same
slice-labeling procedure for dataset (5), BraTS 2018 [19] where we detect
gliomas in T2 FLAIR brain MRI slices. Dataset (6) is Prostate-MRI-Biopsy [29],
where we take slices from the middle 50% of each MRI volume, and label each
slice according to the volume’s assigned prostate cancer risk score; scores of
$\\{0,1\\}$ are negative, and scores of $\geq 2$, which correlates with risk
of cancer [22], are positive. Our final dataset (7) is the RSNA 2019
Intracranial Hemorrhage Brain CT Challenge (RSNA-IH-CT) [10], where we detect
any type of hemorrhage in Brain CT scans. Sample images are shown in Fig. 1.

##  4 Experiments and Results

###  4.1 The Intrinsic Dimension of Radiology Datasets

We first estimate the intrinsic dimension (ID) of the considered radiology
datasets; the results are shown in Figure 2, alongside the natural image
dataset results of [24]. For each dataset, we estimate the MLE ID (Equation
(2)) on a sample of $7500$ images such that there is an exact $50/50$ split of
negative and positive cases in the sample, to minimize estimator bias
(although we found that when using fewer images, the estimates were little
affected). Like natural images, we found the ID of radiological images to be
many times smaller than the extrinsic dimension; however, radiological image
datasets tend to have lower ID than natural images datasets. Intuitively, we
found that modifying the dataset extrinsic dimension (resizing the images) had
little effect on the ID.

###  4.2 Generalization Ability, Learning Difficulty and Intrinsic Dimension

We now wish to determine what role the intrinsic dimension (ID) of a radiology
dataset has in the degree of difficulty for a deep model to learn from it, and
how this compares to the natural image domain. As in [24], we use the test set
accuracy obtained when the model has maximally fit to the training set as a
proxy for the generalization ability (GA) of the model on the dataset. We
train and test with each dataset separately on it’s respective binary
classification task (Sec. 3), for a range of models and training set sizes
$N_{\text{train}}$, from $500$ to $2000$. We train for 100 epochs with the
same hyperparameters for all experiments; further details are provided in the
supplementary materials. For each experiment with a studied dataset, we sample
$2750$ images from the given dataset such that there is an exact $50/50$ split
of negative and positive images. From this, we sample $750$ test images, and
from the remaining $2000$ images, we sample some $N_{\text{train}}$ training
examples.

Beginning with $N_{\text{train}}=2000$ on ResNet-18, we plot the GA with
respect to the dataset ID in Fig. 3, alongside the corresponding results for
natural image datasets where binary classification was explored from [24].
Intriguingly, even across the range of tasks and datasets, we see that the
relationship of GA with ID is approximately linear within each domain. Indeed,
when fitting a simple ordinary least squares linear model

$\text{GA}=a_{\text{GA},\text{ID}}\times\text{ID}+b$ (3)

to each of the two domains, we obtain a Pearson linear correlation coefficient
of $R^{2}=0.824$ and slope of $a_{\text{GA},\text{ID}}=-0.019$ for
radiological images, and $R^{2}=0.981$ with $a_{\text{GA},\text{ID}}=-0.0075$
for natural images.

Figure 3: Linearity of model generalization ability with respect to dataset
intrinsic dimension, for radiological (blue) and natural (orange) image
datasets, for $N_{\text{train}}=2000$ on ResNet-18. Figure recommended to be
viewed in color.

When repeating the same experiments over the aforementioned range of
$N_{\text{train}}$ and ResNet models, we find that averaged over
$N_{\text{train}}$, $R^{2}=0.76\pm 0.05$, $a_{\text{GA},\text{ID}}=-0.0199\pm
0.0009$ and $R^{2}=0.91\pm 0.12$, $a_{\text{GA},\text{ID}}=-0.0077\pm 0.0004$
($\pm$ standard deviation) for the radiological and natural domains,
respectively. These low deviations imply that even across a range of training
sizes, both domains have a high, and mostly constant, correlation between GA
and ID. Similarly, both domains have approximately constant slopes of GA vs.
ID with respect to $N_{\text{train}}$, but between domains, the slopes differ
noticeably.

On the left of Fig. 4, we show how GA varies with respect to
$N_{\text{train}}$ for datasets of both domains. We see that datasets with
higher ID pose a more difficult classification task within both domains, i.e.,
more training samples are required to achieve some test accuracy. However,
between these two domains, radiological images are generally more difficult to
generalize to than natural images, for these moderate-to-low training set
sizes (that are typical for radiology tasks [28]). For example, OAI
($\text{ID}=13$), MURA ($\text{ID}=14$) and CheXpert ($\text{ID}=20$), all
prove to be more difficult than ImageNet, even though it has more than double
the number of intrinsic dimensions ($43$).

This implies that the intrinsic dataset features described by these dimensions
(and the correlations between them) can vary in learning difficulty between
domains, indicated by the aforementioned sharper slope
$a_{\text{GA},\text{ID}}$. Our results show that radiological images generally
have more difficult intrinsic features to learn than natural images, even if
the number of these intrinsic feature dimensions is higher for the latter.

Figure 4: Left: model generalization ability (GA) vs. training set size
$N_{\text{train}}$ for various radiological (solid line) and natural (dashed
line) image datasets, using ResNet-18. Right: linearity of GA with respect to
$\log N_{\text{train}}$ and dataset intrisic dimension. Accompanies Fig. 3;
recommended to be viewed in color.

We can also explore the dependence of generalization ability (GA) on both
training set size $N_{\text{train}}$ and ID, shown on the right of Fig. 4.
[21] found that learning requires a training sample size that grows
exponentially with the data manifold’s ID; this implies that GA should scale
with $\log N_{\text{train}}$. Indeed we see that GA is approximately linear
with respect to ID and $\log N_{\text{train}}$: for each domain of
radiological and natural, we find multiple linear correlation coefficients
$R^{2}$ of $0.766$ and $0.902$ between these variables, respectively. Given
some new radiology dataset, we could potentially use this to estimate the
minimum $N_{\text{train}}$ needed to obtain a desired GA/test accuracy for the
dataset, which would save overall training time. However, extensive
experiments would need to be conducted before this could be widely applicable,
due to confounding factors such as if the chosen model has a high enough
capacity to fully fit to the training set; as such, we leave such an
investigation for future work.

####  4.2.1 Dependence on Model Choice.

As mentioned, we repeated the same experiments with a number of additional
models for the radiology domain, to see if these linear relationships change
with different model choices. In addition to ResNet-18, we tested on ResNet-34
and -50 ([12]), VGG-13, -16 and -19 ([27]), Squeezenet 1.1 ([14]), and
DenseNet-121 and -169 ([13]). Averaged over $N_{\text{train}}$ and all models,
we obtained $R^{2}=0.699\pm 0.080$ and $a_{\text{GA},\text{ID}}=-0.019\pm
0.001$ (individual model results are provided in the supplementary materials).
By the low deviations of both $R^{2}$ and the actual regressed slope
$a_{\text{GA},\text{ID}}$ of GA vs. ID, we see that the same linear
relationship between GA and ID also exists for these models. We therefore
infer that this relationship between classification GA and ID is largely
independent of model size/choice, assuming that the model has a high enough
capacity to fully fit to the training set.

####  4.2.2 Dependence on Task Choice.

Logically, there should be other factors affecting a model’s GA beyond the
dataset’s ID; e.g., for some fixed dataset, harder tasks should be more
difficult to generalize to. This section aims to determine how changing the
chosen tasks for each dataset affects the preceding results. We will consider
realistic binary classification tasks that have enough examples to follow the
dataset generation procedure of Sec. 4.2. The datasets that support this are
CheXpert—detect edema instead of pleural effusion, RSNA-IH-CT—detect
subarachnoid hemorrhage, rather than any hemorrhage, and Prostate-MRI—detect
severe cancer ($\text{score}>2$), rather than any cancer. For robustness we
will experiment on all three aforementioned ResNet models.

From switching all three datasets to their modified tasks, the linear fit
parameters (averaged over $N_{\text{train}}$) changed as $R^{2}=0.76\pm
0.05\Rightarrow 0.78\pm 0.15$ and $a_{\text{GA},\text{ID}}=-0.0199\pm
0.0009\Rightarrow-0.012\pm 0.002$ for ResNet-18; $R^{2}=0.77\pm
0.07\Rightarrow 0.76\pm 0.21$ and $a_{\text{GA},\text{ID}}=-0.019\pm
0.001\Rightarrow-0.011\pm 0.003$ for ResNet-34; and $R^{2}=0.78\pm
0.07\Rightarrow 0.82\pm 0.15$ and $a_{\text{GA},\text{ID}}=-0.021\pm
0.001\Rightarrow-0.013\pm 0.004$ for ResNet-50. We therefore conclude that the
choice of task has some, but a small effect on the significance of the linear
relationship of GA with ID $(R^{2})$, but can affect the parameters (slope
$a_{\text{GA},\text{ID}}$) of the relationship. Certainly, the choice of task
represents a very large space of possibilities, so we leave more comprehensive
investigations for future work.

##  5 Conclusion

Our results provide empirical evidence for the practical differences between
the two domains of radiological and natural images, in both the intrinsic
structure of datasets, and the difficulty of learning from them. We found that
radiological images generally have a lower intrinsic dimension (ID) than
natural images (Fig. 2), but at the same time, they are generally harder to
learn from (Fig. 4, left). This indicates that the intrinsic features of
radiological datasets are more complex than those in natural image data.
Therefore, assumptions about natural images and models designed for natural
images should not necessarily be extended to radiological datasets without
consideration for these differences. Further study of the differences between
these two domains and the conceptual reasons for why they arise could lead to
helpful guidelines for deep learning with radiology. We believe that the
results in this work are an important step in this direction, and they lay the
foundation for further research on this topic.

## References

  * [1] Alessio Ansuini, Alessandro Laio, Jakob H Macke, and Davide Zoccolan.  Intrinsic dimension of data representations in deep neural networks.  Advances in Neural Information Processing Systems, 32, 2019.
  * [2] Joseph Antony, Kevin McGuinness, Kieran Moran, and Noel E O’Connor.  Automatic detection of knee joints and quantification of knee osteoarthritis severity using convolutional neural networks, 2017.
  * [3] Jonathan Bac, Evgeny M Mirkes, Alexander N Gorban, Ivan Tyukin, and Andrei Zinovyev.  Scikit-dimension: a python package for intrinsic dimension estimation.  Entropy, 23(10):1368, 2021.
  * [4] Tolga Birdal, Aaron Lou, Leonidas J Guibas, and Umut Simsekli.  Intrinsic dimension, persistent homology and generalization in neural networks.  Advances in Neural Information Processing Systems, 34, 2021.
  * [5] Matthew Brand.  Charting a manifold.  Advances in neural information processing systems, 15, 2002.
  * [6] Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter.  A downsampled variant of imagenet as an alternative to the cifar datasets.  arXiv preprint arXiv:1707.08819, 2017.
  * [7] Dietmar Cordes and Rajesh R Nandy.  Estimation of the intrinsic dimensionality of fmri data.  Neuroimage, 29(1):145–154, 2006.
  * [8] Elena Facco, Maria d’Errico, Alex Rodriguez, and Alessandro Laio.  Estimating the intrinsic dimension of datasets by a minimal neighborhood information.  Scientific reports, 7(1):1–8, 2017.
  * [9] Charles Fefferman, Sanjoy Mitter, and Hariharan Narayanan.  Testing the manifold hypothesis.  Journal of the American Mathematical Society, 29(4):983–1049, 2016\.
  * [10] Adam E Flanders, Luciano M Prevedello, George Shih, Safwan S Halabi, Jayashree Kalpathy-Cramer, Robyn Ball, John T Mongan, Anouk Stein, Felipe C Kitamura, Matthew P Lungren, et al.  Construction of a machine learning dataset through collaboration: the rsna 2019 brain ct hemorrhage challenge.  Radiology: Artificial Intelligence, 2(3):e190211, 2020.
  * [11] Marina Gomtsyan, Nikita Mokrov, Maxim Panov, and Yury Yanovich.  Geometry-aware maximum likelihood estimation of intrinsic dimension.  In Asian Conference on Machine Learning, pages 1126–1141. PMLR, 2019.
  * [12] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770–778, 2016.
  * [13] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger.  Densely connected convolutional networks.  In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4700–4708, 2017.
  * [14] Forrest N Iandola, Song Han, Matthew W Moskewicz, Khalid Ashraf, William J Dally, and Kurt Keutzer.  Squeezenet: Alexnet-level accuracy with 50x fewer parameters and< 0.5 mb model size.  arXiv preprint arXiv:1602.07360, 2016.
  * [15] Jeremy Irvin, Pranav Rajpurkar, Michael Ko, Yifan Yu, Silviana Ciurea-Ilcus, Chris Chute, Henrik Marklund, Behzad Haghgoo, Robyn Ball, Katie Shpanskaya, et al.  Chexpert: A large chest radiograph dataset with uncertainty labels and expert comparison.  In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 590–597, 2019.
  * [16] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [17] Elizaveta Levina and Peter Bickel.  Maximum likelihood estimation of intrinsic dimension.  Advances in neural information processing systems, 17, 2004.
  * [18] David JC MacKay and Zoubin Ghahramani.  Comments on’maximum likelihood estimation of intrinsic dimension’by e. levina and p. bickel (2004).  The Inference Group Website, Cavendish Laboratory, Cambridge University, 2005.
  * [19] Bjoern H Menze, Andras Jakab, Stefan Bauer, Jayashree Kalpathy-Cramer, Keyvan Farahani, Justin Kirby, Yuliya Burren, Nicole Porz, Johannes Slotboom, Roland Wiest, et al.  The multimodal brain tumor image segmentation benchmark (brats).  IEEE transactions on medical imaging, 34(10):1993–2024, 2014.
  * [20] Lia Morra, Luca Piano, Fabrizio Lamberti, and Tatiana Tommasi.  Bridging the gap between natural and medical images through deep colorization.  In 2020 25th International Conference on Pattern Recognition (ICPR), pages 835–842. IEEE, 2021.
  * [21] Hariharan Narayanan and Sanjoy Mitter.  Sample complexity of testing the manifold hypothesis.  Advances in neural information processing systems, 23, 2010.
  * [22] S. Natarajan, A. Priester, D. Margolis, J. Huang, and L. Marks.  Prostate mri and ultrasound with pathology and coordinates of tracked biopsy (prostate-mri-us-biopsy) [data set].  DOI: 10.7937/TCIA.2020.A61IOC1A, 2020.  Accessed: 2022-02-21.
  * [23] M. Nevitt, D. Felson, and Gayle Lester.  The osteoarthritis initiative.  Protocol for the cohort study, 1, 2006.
  * [24] Phillip Pope, Chen Zhu, Ahmed Abdelkader, Micah Goldblum, and Tom Goldstein.  The intrinsic dimension of images and its impact on learning.  arXiv preprint arXiv:2104.08894, 2021.
  * [25] Pranav Rajpurkar, Jeremy Irvin, Aarti Bagul, Daisy Ding, Tony Duan, Hershel Mehta, Brandon Yang, Kaylie Zhu, Dillon Laird, Robyn L Ball, et al.  Mura: Large dataset for abnormality detection in musculoskeletal radiographs.  arXiv preprint arXiv:1712.06957, 2017.
  * [26] Ashirbani Saha, Michael R Harowicz, Lars J Grimm, Connie E Kim, Sujata V Ghate, Ruth Walsh, and Maciej A Mazurowski.  A machine learning approach to radiogenomics of breast cancer: a study of 922 subjects and 529 dce-mri features.  British journal of cancer, 119(4):508–516, 2018.
  * [27] Karen Simonyan and Andrew Zisserman.  Very deep convolutional networks for large-scale image recognition.  arXiv preprint arXiv:1409.1556, 2014.
  * [28] Shelly Soffer, Avi Ben-Cohen, Orit Shimon, Michal Marianne Amitai, Hayit Greenspan, and Eyal Klang.  Convolutional neural networks for radiologic images: a radiologist’s guide.  Radiology, 290(3):590–606, 2019.
  * [29] Geoffrey A Sonn, Shyam Natarajan, Daniel JA Margolis, Malu MacAiran, Patricia Lieu, Jiaoti Huang, Frederick J Dorey, and Leonard S Marks.  Targeted biopsy in the detection of prostate cancer using an office based magnetic resonance ultrasound fusion device.  The Journal of urology, 189(1):86–92, 2013.
  * [30] Joshua B Tenenbaum, Vin de Silva, and John C Langford.  A global geometric framework for nonlinear dimensionality reduction.  science, 290(5500):2319–2323, 2000.
  * [31] Aleksei Tiulpin, Jérôme Thevenot, Esa Rahtu, Petri Lehenkari, and Simo Saarakkala.  Automatic Knee Osteoarthritis Diagnosis from Plain Radiographs: A Deep Learning-Based Approach.  Scientific Reports, 8(1):1727, 2018.

## Supplementary Materials for “The Intrinsic Manifolds of Radiological Images
and their Role in Deep Learning”

##  Appendix 0.A Experimental Settings (for reproducibility)

Table 1: Training/Inference Parameters and Settings Name | Value
---|---
Loss Function | Binary Cross-Entropy
Optimizer | Adam [16]
Learning rate | 0.001
Weight decay | 0.0001
Image normalization | $[0,255]$
Image resolution | $224\times 224$
Training image augmentations | None
GPUs |  $4\times$ NVIDIA GeForce RTX 3070 (8 GB)
Experiment Software | PyTorch 1.8.1, Numpy 1.22.1, CUDA 11.4
Visualization Software | MatPlotLib 3.4.2
Typical model training time (on $N_{\text{train}}=2000$) | 10-30 min.
Table 2: Model-Specific Details and Results (averaged over $N_{\text{train}}$ with std. dev.) Model | Training batch size |  $R^{2}$ (GA vs. ID) | slope $a_{\text{GA},\text{ID}}$ (GA vs. ID)
---|---|---|---
ResNet-18 [12] | 200 | $0.756\pm 0.052$ | $-0.0199\pm 0.0009$
ResNet-34 [12] | 128 | $0.772\pm 0.071$ | $-0.0193\pm 0.0012$
ResNet-50 [12] | 64 | $0.781\pm 0.066$ | $-0.0207\pm 0.0010$
VGG-13 [27] | 32 | $0.646\pm 0.048$ | $-0.0194\pm 0.0009$
VGG-16 [27] | 32 | $0.623\pm 0.066$ | $-0.0184\pm 0.0008$
VGG-19 [27] | 32 | $0.597\pm 0.100$ | $-0.0168\pm 0.0031$
Squeezenet 1.1 [14] | 32 | $0.580\pm 0.073$ | $-0.0173\pm 0.0011$
DenseNet-121 [13] | 32 | $0.770\pm 0.073$ | $-0.0190\pm 0.0009$
DenseNet-169 [13] | 32 | $0.765\pm 0.061$ | $-0.0189\pm 0.0008$


# PTT: Point-Track-Transformer Module for 3D Single Object Tracking in Point
Clouds

Jiayao Shan†, Sifan Zhou†, Zheng Fang*, Yubo Cui †Authors with equal
contribution.The authors are with the Faculty of Robot Science and
Engineering, Northeastern University, Shenyang, China; Corresponding author:
Zheng Fang, e-mail<EMAIL_ADDRESS>
###### Abstract

3D single object tracking is a key issue for robotics. In this paper, we
propose a transformer module called Point-Track-Transformer (PTT) for point
cloud-based 3D single object tracking. PTT module contains three blocks for
feature embedding, position encoding, and self-attention feature computation.
Feature embedding aims to place features closer in the embedding space if they
have similar semantic information. Position encoding is used to encode
coordinates of point clouds into high dimension distinguishable features.
Self-attention generates refined attention features by computing attention
weights. Besides, we embed the PTT module into the open-source state-of-the-
art method P2B to construct PTT-Net. Experiments on the KITTI dataset reveal
that our PTT-Net surpasses the state-of-the-art by a noticeable margin
($\sim$10%). Additionally, PTT-Net could achieve real-time performance
($\sim$40FPS) on NVIDIA 1080Ti GPU. Our code is open-sourced for the robotics
community at https://github.com/shanjiayao/PTT.

##  I INTRODUCTION

3D single object tracking (SOT) has a wide range of applications in robotics
and autonomous driving [1, 2]. However, most existing 3D SOT methods are
equipped with RGB-D cameras [3, 4], which inherit the characteristics of 2D
images and depend heavily on RGB-D information, thus trackers may fail in
visually degraded or illumination changing environments.

In addition to RGB-D sensors, 3D LIDAR sensors are also widely used in object
tracking tasks [5, 6] because they are less sensitive to illumination changes
and could directly capture geometric information more accurately. However,
using only point clouds for 3D SOT has its challenges. First, point cloud is
sparse and disordered [7], which requires the network to be permutation-
invariant. Second, 3D object tracking needs to estimate higher space dimension
(e.g. $x,y,z,w,h,l,ry$) than 2D visual tracking, which takes more
computational complexity. Third, compared with rigid object tracking (e.g.
car), it’s more challenging to track non-rigid objects (e.g. pedestrian) since
it is too hard to extract stable features.

Figure 1: Exemplified illustration to show how PTT module works. Compared with
the existing 3D single object tracking method, our PTT module works after
calculating the similarity features, and weighs the features based on their
importance to improve tracking performance.

Currently, most point cloud based 3D SOT methods follow 2D visual object
tracking (VOT) approaches (e.g. Siamese networks [8, 9]) to track 3D target.
The Siamese network formulates VOT task as learning a similarity function
between the template branch and the search branch. SC3D [10] is the pioneer
point cloud based 3D Siamese Tracker based on shape completion network.
However, it could not be trained end-to-end and run in real-time. Besides, Qi
et al. [11] also proposed a point-to-box (P2B) network to estimate target
bounding box from the raw point cloud. However, this approach does not cope
well with sparse scenarios. Recently, Fang et al. [12] jointed Siamese network
and point cloud based Region Proposal Network (RPN) [13] to tackle 3D SOT
task. Nonetheless, the performance of their method is unsatisfying. It is
worth noting that points located in different geometric positions often have
different importance in representing targets. However, these aforementioned
methods do not weigh point cloud features based on this characteristic.
Besides, the point cloud features extracted from the template and the search
area contain less potential object information due to the sparsity of point
clouds. Therefore, how to pay attention to the important clues is the key to
improving the performance of the 3D object tracker.

Recently, transformer has revolutionized natural language processing and image
analysis [14, 15, 16]. Self-attention operator, which is the core of
transformer networks, is intrinsically a set operator: positional information
is provided as attributes of elements that are processed as a set [14, 17].
Therefore, transformer is suitable for point cloud due to its positional
attributes.

In this paper, we explore the application of the transformer network for 3D
SOT task, and propose a transformer module named PTT (Point-Track-
Transformer). To pay more attention to the important feature of the object, we
use the transformer’s powerful self-attention and position encoding mechanism
to weigh the point cloud features. To evaluate the effect of our PTT module,
we embed our PTT module into the open-source state-of-the-art method P2B [11]
to build a new network called PTT-Net. Finally, the experimental results of
our PTT-Net on KITTI [18] dataset demonstrate the superiority of our method
($\sim$10%’s improvement on both Success and Precision). Besides, PTT-Net
could run at 40FPS.

Overall, there are three main contributions as follows:

$\bullet$ PTT module: a Point-Track-Transformer (PTT) module for 3D single
object tracking using only point clouds, weighing point cloud features to
focus on deeper-level object clues during tracking.

$\bullet$ PTT-Net: a 3D single object tracking network named PTT-Net embedded
with PTT modules which can be end-to-end trained. To the best of our
knowledge, this is the first work to apply transformer to 3D object tracking
task based on point cloud.

$\bullet$ Open-source: Experiments on KITTI tracking dataset [18] show our
method outperforms the state-of-the-art methods with remarkable margins.
Besides, we open source our method to the research community.

##  II RELATED WORK

This section will briefly discuss the related work in 3D single object
tracking, transformer and self-attention mechanism.

###  II-A 3D SOT Using Point Cloud

Giancola et al. [10] proposed the first pioneer point cloud based 3D single
object tracker which utilized the Kalman Filter to generate massive target
proposals. They introduced the shape completion module to enrich the feature
learning on points. However, their method has a poor generalization ability
and could not run in real-time. Zarzar et al. [19] leveraged 2D Siamese
network which converted raw point clouds into Bird-Eye-View (BEV)
representation to generate 3D proposals. This method may lose fine-grain
geometry details which are important for tracking tiny objects. Cui et al.
[20] also adopted a 3D Siamese tracker only using point cloud. However, they
could not estimate the orientation and size information of the target. Fang et
al. [12] jointed 3D Siamese network and 3D RPN network to track targets, but
their performance is limited by the one-stage RPN network. Besides, Zou et al.
[21] integrated 2D image and 3D point cloud information for 3D SOT. However,
this method relies more on 2D trackers and the performance is not satisfactory
when using previous results to initialize tracker. Qi et al. [11] proposed P2B
which used deep hough voting to obtain the potential centers (votes) and
estimated target center based on those votes. However, it ignores the fact
that points in different positions have different contributions to tracking.
Furthermore, its random sampling mechanism loses the location distribution
information of the raw point cloud. Based on these shortcomings, we propose a
PTT module to weigh different point features and use farthest point sampling
instead of random sampling to obtain more raw point cloud information.

###  II-B Transformer and Self-attention

Recently, there have been many wonderful works based on Transformer [14, 15].
Hu et al. [15] and Ramachandran et al. [16] applied scalar dot product self-
attention in local pixel neighbors. Zhao et al. [17] utilized vector self-
attention operations to image tasks.

Inspired by these works, Zhao et al. [22] used a Point Transformer layer by
applying vector self-attention operations, which had a great performance
improvement in point cloud classification and segmentation tasks. Nico et al.
[23] proposed SortNet as a part of Point Transformer, and achieved competitive
performance on point cloud classification and partial segmentation tasks.
Meanwhile, Guo et al. [24] also introduced Point Cloud Transformer (PCT),
which performed well on shape classification, part segmentation, and normal
estimation tasks. Obviously, transformer has unique advantages for point cloud
feature learning. We hence aim to extend the transformer paradigm to our 3D
SOT task with more attention to features.

Figure 2: PTT module architecture. It consists of three blocks: feature
embedding, position encoding, and self-attention. The whole inputs are the
coordinates and their corresponding features. Feature embedding module maps
input features into embedding space. In position encoding module, the
k-nearest neighbor algorithm is used to obtain local position information,
then the encoded position features will be learned by a Multi Layer Perceptron
(MLP). The self-attention module learns refined attention features for input
features based on local context. The output features of PTT module are the sum
of input and residual features.

##  III METHODOLOGY

Here, we focus on the application of transformer network in 3D SOT. Given an
input of M points with XYZ coordinates, a backbone network modified on the
basis of PointNet++ [25] is used to extract the point cloud and learn deep
features. It outputs a subset of the input containing N interest points
(seeds) ${S=\left\\{s_{i}\right\\}^{N}_{i=1}}$. ${s_{i}=(c_{i},f_{i})}$ is
composed by a vector ${c_{i}}$ of 3D coordinate and a D-dimensional descriptor
${f_{i}}$ of the local object geometry. Our goal of using transformer is to
perform an attention weighting operation on the feature space of $f_{i}$, and
output refined features ${f_{i}}^{*}$ with the same dimension.

###  III-A Transformer

The architecture of transformer can be divided into three parts: input feature
embedding, position encoding, and self-attention. Self-attention is the core
module, which mainly focuses on the differences of input features and
generates refined attention features based on global or local context. Given
the input feature ${G=\left\\{g_{i}\right\\}^{N}_{i=1}}$ after feature
embedding, the general formula of self-attention is:

$\displaystyle Q,K,V=\alpha(G),\beta(G),\gamma(G)$ (1)
$\displaystyle\mathcal{A}=\rho(\sigma(Q,K)+P)\odot(V)$

where $\alpha$, $\beta$ and $\gamma$ are point-wise feature transformations
(e.g. linear layers or MLPs). $Q$, $K$, and $V$ are the $query$, $key$ and
$value$ matrices, respectively. $\sigma$ is the relation function between $Q$
and $K$. $P$ is the position encoding feature. $\rho$ is a normalization
function (e.g. Softmax). $\odot$ means Hadamard product. $\mathcal{A}$ is the
attention feature produced by self-attention layer.

###  III-B PTT Module

We modify the transformer module proposed in Point Transformer [22] to weigh
the point cloud features. In [22], the point transformer layer is proposed to
process the raw point cloud for classification and segmentation tasks. Here,
we set an importance-based transformer module to focus on the differences
among the input point cloud features for 3D SOT task.

Therefore, we explore how to embed transformer modules into 3D SOT task and
propose our Point-Track-Transformer (PTT) module. PTT module processes
features by utilizing shape and geometry information. Given a point set
${S=\left\\{s_{i}\right\\}^{N}_{i=1}}$, $s_{i}=(c_{i},f_{i})$,
$c_{i}\in\mathbb{R}^{3}$ and $f_{i}\in\mathbb{R}^{D}$. ${c_{i}}$ and ${f_{i}}$
represent 3D coordinates and descriptor of point $s_{i}$.

Feature embedding module maps input features into embedding space
$\mathbb{R}^{M}$: $f_{i}\rightarrow g_{i}$, $g_{i}\in\mathbb{R}^{M}$. Position
encoding module extracts higher-level M-dimensional features $p_{i}$ from
input coordinates $c_{i}$: $c_{i}\rightarrow p_{i}$,
$p_{i}\in\mathbb{R}^{K\times M}$. Finally, the self-attention module
calculates attention weights and attention features
$f_{i}^{*}$,$f_{i}^{*}\in\mathbb{R}^{D}$ by taking embedding features and
position features as inputs. To avoid the vanishing gradient problem in
training stage, we also adopt the residual architecture in [22], and take the
sum of the attention features and input features as output features.

####  III-B1 Feature Embedding

The original feature embedding module in Natural Language Processing (NLP) is
to map each word in the input sequence to a high-dimensional vector. In this
work, we use the linear layer to complete the feature embedding operation, and
map the input point cloud feature dimension from $D$ To $M$:
$\mathbb{R}^{D}\rightarrow\mathbb{R}^{M}$, which can place the feature closer
in the embedding space if the semantics are more similar and make the network
have a stronger fitting ability.

####  III-B2 Position Encoding

Position encoding module plays a crucial role in transformer, which allows
operators to adapt to the local structure of the input data [14]. And 3D point
coordinates themselves are the natural input for position encoding. Therefore,
we utilize the coordinates directly as the input of the position encoding
module. Besides, we use the relative coordinates to make the network better
capture the spatial correlation between points and local geometric shape
information. Since the feature $f_{i}$ is extracted by [25] which can provide
the local context information, we obtain the position encoding features
${P=\left\\{p_{i}\right\\}^{N}_{i=1}}$ with function $\eta$. For input point
set $S$ including N points, the position encoding feature for each point is:

${p_{i}}=\eta(c_{i}-c_{j})$ (2)

where $c_{i}$ is the coordinate of the i-th point in $S$. $c_{j}$ is the j-th
coordinate in local neighborhood region of $c_{i}$ by using K-nearest-neighbor
(KNN). And $\eta$ is an MLP with two linear layers and one ReLU layer.

Figure 3: The pipeline of PTT-Net. In order to verify the effect of our PTT
module, we embedded two PTT modules into seeds voting stage and proposals
generation stage of [11]. The first transformer block takes the augmented
feature as input, and outputs the refine features for voting. The second
transformer block refines the cluster features to help PTT-Net obtain better
proposals.

####  III-B3 Self-Attention

As Fig. 2 shows, self-attention module computes three vectors for each point:
$Q$, $K$, $V$ through $\alpha$, $\beta$, $\gamma$, where $\alpha$, $\beta$,
$\gamma$ are all linear layers. It is worth noting that K and V are aggregated
from the features of the $k$ neighborhood points, which aim to encode more
local context information. Here, $Q\in\mathbb{R}^{M}$,
$K\in\mathbb{R}^{K\times M}$, and $V\in\mathbb{R}^{K\times M}$.

The relation function $\sigma$ can be classified into two types: scalar [14]
and vector [17]. And it has been proved in [22] that vector attention is more
suitable for point cloud than scalar attention since it supports adaptive
modulation of individual feature channels, not just whole feature vectors.
Thus we set $\sigma(Q,K)=Q-K$ to obtain point-wise attention weights. And an
MLP layer $\gamma$ is used to introduce additional trainable transformations
and match the output dimension. Then, we add the position encoding features
$P$ to both the attention vector $\sigma$ and the transformed features $K$.
Finally, the residual features recorded as $\mathcal{A}$ are defined as the
weighted sum of the attention weights with all $V$ vectors. The formula is as
follows:

$\mathcal{A}=\rho\big{(}\gamma(Q-{K}+{P})\big{)}\odot\big{(}{V}+{P}\big{)}$
(3)

where $\rho$ is a normalization function (Softmax) and $\gamma$ is a non-
linear mapping function (MLP) that includes two linear layers and one ReLU
layer. $\mathcal{A}$ is attention features.

###  III-C PTT-Net

The ability of the transformer to learn self-attention weights inspires us to
try it on 3D SOT task. We formulate the problem of focusing on the differences
in features as self-attention weighting. In order to verify the effect of our
method, we embed our PTT module into P2B [11]. More specifically, the PTT
modules are inserted in seeds voting stage and proposals generation stage of
P2B.

In seeds voting stage, P2B generates votes based on the augmented features,
which are from backbone in Fig. 3. We notice that [11] ignores the differences
among different point cloud features in the search area, and gives no
preference to the points in different locations when generating votes.
However, it is important to focus on the points which contain more geometric
information and suppress the background noise. Therefore, we apply PTT module
to weigh the augmented features and obtain the weighted features focused on
foreground points (in Fig. 4(a)(b)).

In proposals generation stage, P2B generates proposals based on local context
features. However, their method ignores the global semantic features of
targets, so that they could not distinguish similar objects (e.g. two
pedestrians, in Fig. 4(c)(d)). Therefore, we use the PTT module to further
weigh the target-wise context features obtained by the aggregation network in
P2B for tracking deeper-level target clues.

As shown in Fig. 3, we embed our PTT module in the P2B [11] to build PTT-Net.
We add PTT module to the seeds voting and proposal generation stages, and
weigh the augmented features and cluster features respectively. Experiments
show that our PTT-Net outperforms [11] with remarkable margins.

####  III-C1 Loss Function

The PTT module is trained with the other subnetworks in [11]. So we follow
[11] to design our loss function. The overall loss consists of two parts as
follows:

$\displaystyle{L}_{all}={L}_{cv}+\lambda_{1}{L}_{cb}+\lambda_{2}{L}_{rv}+\lambda_{3}{L}_{rb}$
(4)

where $\lambda_{1}$, $\lambda_{2}$, $\lambda_{3}$ represent the weighting
coefficient of each loss. Classification loss includes voting classification
loss ${L}_{cv}$ and proposal box classification loss ${L}_{cb}$. The
regression loss includes the voting loss ${L}_{rv}$ and the proposal box
regression loss ${L}_{rb}$.

##  IV EXPERIMENTS

We used KITTI tracking dataset [18] as the benchmark, and set up more detailed
experiments to show the superiority of our PTT-Net by comparing the different
performance of the dominant methods [10, 11, 12, 21] on rigid (Car,Van) and
non-rigid (Pedestrian,Cyclist) objects. The experiments show that PTT-Net
outperforms the previous SOTA method with remarkable margins at 40fps.

###  IV-A Experimental protocols

####  IV-A1 Dataset

We used the training set of KITTI which includes more than 20,000 manually
labeled 3D objects using Velodyne HDL-64E 3D lidar (10HZ). For the splits of
dataset, we follow [10, 11, 12, 21], which divide 20 sequences into three
parts 00-16, 17-18, 19-20, corresponding to training set, validation set, and
test set respectively.

####  IV-A2 Evaluation Metric

Following previous work [10, 11, 12, 21], we report Success and Precision
metrics defined by One Pass Evaluation (OPE) [26], which represent overlap and
error Area Under the Curve (AUC) respectively.

####  IV-A3 Implementation Details

We use the farthest point sampling (FPS) instead of random sampling (RS) in
origin P2B [11]. In training stage, we use the Adam optimizer and set the
initial learning rate to 0.001 and decrease by 5 times after 12 epochs. The
batch size is 48 and training epoch is 60. Besides, we extend the offset from
(x, y, $\theta$) to (x, y, z, $\theta$) when generating more template samples
during data augmentation in [11]. In testing stage, we also add $Z$ axis
offset to generate predicted box. Other parameters are consistent with
settings of [11].

###  IV-B Quantitative Experiments

TABLE I: Performance comparison on the KITTI dataset for the car category. R
and L denote RGB ans LiDAR respectively.

Module | Modality | 3D Success | 3D Precision | FPS
---|---|---|---|---
AVOD-Tracking[27] | R+L | 63.1 | 69.7 | -
F-Siamese[21] | R+L | 37.1 | 50.6 | -
SC3D[10] | L | 41.3 | 57.9 | 1.8
P2B[11] | L | 56.2 | 72.8 | 45.5
3D-SiamRPN[12] | L | 58.22 | 76.2 | 20.8
PTT-Net(Ours) | L | 67.81 | 81.8 | 40.0

1 2Red and blue mean the performance score is ranked first and second
respectively.

TABLE II: Extensive comparisons with different categories. “Ped" denotes “Pedestrian".  | Category | Car | Ped | Van | Cyclist | Mean
---|---|---|---|---|---|---
| Frame Number
---
6424 | 6088 | 1248 | 308 | 14068
Success | SC3D[10] | 41.3 | 18.2 | 40.4 | 41.5 | 31.2
P2B[11] | 56.2 | 28.7 | 40.8 | 32.1 | 42.4
FSiamese[21] | 37.1 | 16.2 | - | 47.0 | -
3DSiamRPN[12] | 58.2 | 35.2 | 45.6 | 36.1 | 46.6
PTT-Net(Ours) | 67.8 | 44.9 | 43.6 | 37.2 | 55.1
Precision | SC3D[10] | 57.9 | 37.8 | 47.0 | 70.4 | 48.5
P2B[11] | 72.8 | 49.6 | 48.4 | 44.7 | 60.0
FSiamese[21] | 50.6 | 32.2 | - | 77.2 | -
3DSiamRPN[12] | 76.2 | 56.2 | 52.8 | 49.0 | 64.9
PTT-Net(Ours) | 81.8 | 72.0 | 52.5 | 47.3 | 74.2

To better evaluate our method, we designed two quantitative experiments on
KITTI dataset. In the first experiment, we quantitatively evaluated our method
for 3D car tracking. In the second experiment, we further compared PTT-Net
with the previous methods on Pedestrian, Van, and Cyclist.

####  IV-B1 Comparisons on car category

We compared the performance of our PTT-Net with the existing methods on the
KITTI dataset and reported results for 3D car tracking in Tab. I. In order to
fit the requirement of real scenarios, we generate the search area centered on
the previous result. The results show our PTT-Net has achieved SOTA
performance in all evaluation metrics. Compared with the baseline algorithm
P2B [11], our performance has been greatly improved by $\sim$11%.
Additionally, compared with [27] and [21] which both use RGB+LIDAR fusion
information, the Success/Precision results of PTT-Net outperform them
4.7%/12.1% and 30.7%/31.2% respectively.

####  IV-B2 Comparisons on other categories

We also compared with the dominant methods on Pedestrian, Van, and Cyclist
(Tab. II). The average performance of PTT-Net outperforms P2B [11] $\sim$13%.
It is worth noting that the Success/Precision results of PTT-Net show an
improvement (9.7%/15.8%) on non-rigid object (Pedestrian) tracking. This also
proves that our PTT module can help the network understand and learn the
important characteristics of the target better.

###  IV-C Ablation Study

TABLE III: Different ways for template generation. "GT" denotes "ground truth". "First & Previous" denotes "The first ground truth and Previous result".  | Method |  The First GT |  Previous Result |  First & Previous |  All Previous
---|---|---|---|---|---
Success | SC3D[10] | 31.6 | 25.7 | 34.9 | 41.3
P2B[11] | 46.7 | 53.1 | 56.2 | 51.4
3DSiamRPN[12] | 57.2 | - | 58.2 | -
PTT-Net(Ours) | 62.9 | 64.9 | 67.8 | 59.8
Precision | SC3D[10] | 44.4 | 35.1 | 49.8 | 57.9
P2B[11] | 59.7 | 68.9 | 72.8 | 66.8
3DSiamRPN[12] | 75.0 | - | 76.2 | -
PTT-Net(Ours) | 76.5 | 77.5 | 81.8 | 74.5
TABLE IV: Different embedded positions of PTT module. Ablation | 3D Success | 3D Precision
---|---|---
baseline[11] | 56.2 | 72.8
Only PTT in Vote | 62.1 | 76.9
Only PTT in Prop | 65.7 | 78.9
PTT in all(PTT-Net) | 67.8 | 81.8
Figure 4: Visualization of classification(a-b) and tracking(c-d) results with
or without PTT module. The point will be paid more attention if it has a
higher score. Compared (a) with (b), PTT module pays more attention to the
foreground points. Compared (c) with (d), PTT module could still track targets
robustly in crowded scenes (with multiple pedestrians). Figure 5: Exemplified
illustration to show the attention scores of rigid cases(a-b) and non-rigid
cases(c-d). And we also divide the easy and hard cases according to the number
of foreground points. It is obvious that PTT module guides tracker in focusing
on foreground points even if they are few or extremely similar to others.

####  IV-C1 Template Area Generation

We explored four different settings of template point cloud generation,
including the first ground truth, previous result, the fusion of the first
ground truth and previous result, and all previous results. We reported
results in Tab. III. Obviously, our method achieves SOTA performances in all
settings.

####  IV-C2 Embedding location of PTT module

To verify our design in Sec. III-C of positions where PTT modules are
embedded, we tried different schemes (Tab. IV). The results show that
embedding PTT module in both two stages of [11] can obtain the best
improvement. Besides, as shown in Fig. 4, compared with (a) and (b), PTT-Net
has better point cloud classification results which focus on foreground
points. Comparing (c) with (d), PTT-Net could still track target pedestrian
robustly when more proposal centers are generated from another pedestrian.
This result effectively shows that transformer can learn more target-wise
information.

Figure 6: The number of points on the first frame’s car and 3D Success. There
are 120 points in this scatter figure corresponding to 120 tracklets in
testing sequences (19-20). Orange points indicate tracking off course or even
failed. Figure 7: Advantageous cases of PTT-Net (ours) compared with SC3D and
P2B. In sparse point cloud scenes, our method can track the target well. Both
P2B and SC3D failed to track. Figure 8: Failure cases. no points in the
initial search area

###  IV-D Qualitative Experiment

Attention can be understood as the place where the network focuses on. It is
obvious that PTT module guides tracker in focusing on foreground points even
if they are few.

####  IV-D1 Advantageous cases

We first exemplified the attention score in the voting stage in PTT-Net in
Fig. 5. To further demonstrate the performance of our method, we selected four
scenarios according to the type and difficulty of the tracking target. It is
obvious that PTT module guides tracker in focusing on foreground points even
if they are few by comparing (a) and (b). Besides, (c) and (d) also show good
performance in non-rigid object (pedestrian) tracking, even if the target and
background are extremely similar (d), PTT module can distinguish them.

We then visualized our advantageous cases over P2B and SC3D in Fig. 7. We can
observe that both SC3D and P2B tracked eventually failed in the sparse
scenarios (less than 50 points), but our PTT-Net tracks the target tightly.

####  IV-D2 Failure cases

To show the performance of our method in more detail, we exemplified the
impact of varying density of points on the first frame’s car to PTT-Net in
Fig. 6. The orange points indicate failure cases that the number of points
from the initial frame are mostly less than 20. And even if four tracklets are
initialized with 0 points. In these cases, our PTT-Net could not learn
effective object characteristics, so it fails to track. As shown in Fig. 8,
our PTT-Net could not learn effective object characteristics since no points
in the initial search area.

###  IV-E Timing Breakdown

We calculated the average running time of all test frames in the Car category
to evaluate running speed. PTT-Net achieved 40 FPS on a single NVIDIA 1080Ti
GPU, including 8.3 ms for preparing point cloud, 16.2 ms for model forward
propagation, and 0.5 ms for post-processing. The running time of SC3D [10],
P2B [11] and 3D-SiamRPN [12] on the same platform are 1.8FPS, 45.5FPS and
20.8FPS, respectively.

##  V Conclusions

In this work, we explored the application of transformer network in 3D SOT
task and proposed PTT module. The PTT module aims at weighing point cloud
features to focus on the important features of objects. We also embedded the
PTT modules into the open-source state-of-the-art method [11] and construct a
novel 3D SOT tracker named PTT-Net. Experiments show that PTT-Net outperforms
previous state-of-the-art methods with remarkable margins. We hope that our
work will inspire further investigation of the application of transformers to
3D object tracking.

## ACKNOWLEDGMENT

This work was supported by National Natural Science Foundation of China
(62073066, U20A20197), Science and Technology on Near-Surface Detection
Laboratory (6142414200208), the Fundamental Research Funds for the Central
Universities (N182608003), Major Special Science and Technology Project of
Liaoning Province (No.2019JH1/10100026), and Aeronautical Science Foundation
of China (No. 201941050001).

## References

  * [1] U.Kart, A.Lukezic, M.Kristan, J.K.Kamarainen, and J.Matas, “Object tracking by reconstruction with view-specific discriminative correlation filters,” 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1339–1348, 2019.
  * [2] E. Machida, M. Cao, T. Murao, and H. Hashimoto, “Human motion tracking of mobile robot with kinect 3d sensor,” 2012 Proceedings of SICE Annual Conference (SICE), pp. 2207–2211, 2012.
  * [3] Y. Liu, X.-Y. Jing, J. Nie, H. Gao, J. Liu, and G.-P. Jiang, “Context-aware three-dimensional mean-shift with occlusion handling for robust object tracking in rgb-d videos,” IEEE Transactions on Multimedia, pp. 664–677, 2018.
  * [4] U.Kart, J.K.Kamarainen, and J.Matas, “How to make an rgbd tracker?,” Proc. European Conference on Computer Vision (ECCV), pp. 148–161, 01 2018.
  * [5] W. Luo, B. Yang, and R. Urtasun, “Fast and furious: Real time end-to-end 3d detection, tracking and motion forecasting with a single convolutional net,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3569–3577, 2018.
  * [6] M. Simon, K. Amende, A. Kraus, J. Honer, T. Sämann, H. Kaulbersch, S. Milz, and H. Groß, “Complexer-yolo: Real-time 3d object detection and tracking on semantic point clouds,” 2019 IEEE Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pp. 1190–1199, 2019.
  * [7] R. Q. Charles, S. Hao, K. Mo, and L. J. Guibas, “Pointnet: Deep learning on point sets for 3d classification and segmentation,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 652–660, 2017.
  * [8] L. Bertinetto, J. Valmadre, J. F. Henriques, A. Vedaldi, and P. H. S. Torr, “Fully-convolutional siamese networks for object tracking,” in European Conference on Computer Vision (ECCV), pp. 850–865, 2016.
  * [9] B. Li, J. Yan, W. Wu, Z. Zhu, and X. Hu, “High performance visual tracking with siamese region proposal network,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8971–8980, 2018.
  * [10] S. Giancola, J. Zarzar, and B. Ghanem, “Leveraging shape completion for 3d siamese tracking,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1359–1368, 2019.
  * [11] H. Qi, C. Feng, Z. Cao, F. Zhao, and Y. Xiao, “P2b: Point-to-box network for 3d object tracking in point clouds,” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 6328–6337, 2020.
  * [12] Z. Fang, S. Zhou, Y. Cui, and S. Scherer, “3d-siamrpn: An end-to-end learning method for real-time 3d single object tracking using raw point cloud,” IEEE Sensors Journal, vol. 21, no. 4, pp. 4995–5011, 2021.
  * [13] S. Shi, X. Wang, and H. Li, “Pointrcnn: 3d object proposal generation and detection from point cloud,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770–779, 2019.
  * [14] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin, “Attention is all you need,” in Advances in Neural Information Processing Systems, 2017.
  * [15] H. Hu, Z. Zhang, Z. Xie, and S. Lin, “Local relation networks for image recognition,” 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3463–3472, 2019.
  * [16] P. Ramachandran, I. Bello, N. Parmar, A. Levskaya, A. Vaswani, and J. Shlens, “Stand-alone self-attention in vision models,” in Advances in Neural Information Processing Systems, 2019.
  * [17] H. Zhao, J. Jia, and V. Koltun, “Exploring self-attention for image recognition,” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 10073–10082, 2020.
  * [18] A. Geiger, “Are we ready for autonomous driving? the kitti vision benchmark suite,” in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 3354–3361, 2012.
  * [19] J. Zarzar, S. Giancola, and B. Ghanem, “Efficient tracking proposals using 2d-3d siamese networks on lidar,” arXiv preprint arXiv:1903.10168, 2019\.
  * [20] Y. Cui, Z. Fang, and S. Zhou, “Point Siamese Network for Person Tracking Using 3D Point Clouds,” Sensors, vol. 20, p. 143, dec 2019.
  * [21] H. Zou, J. Cui, X. Kong, C. Zhang, Y. Liu, F. Wen, and W. Li, “F-siamese tracker: A frustum-based double siamese network for 3d single object tracking,” 2020 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 8133–8139, 2020.
  * [22] H. Zhao, L. Jiang, J. Jia, P. H. S. Torr, and V. Koltun, “Point transformer,” arXiv preprint arXiv:2012.09164, 2020.
  * [23] V. B. Nico Engel and K. Dietmayer, “Point transformer,” arXiv preprint arXiv:2011.00931, 2020.
  * [24] M. H. Guo, J. X. Cai, Z. N. Liu, T. J. Mu, R. R. Martin, and S. M. Hu, “Pct: Point cloud transformer,” Computational Visual Media, vol. 7, no. 2, pp. 187–199, 2021.
  * [25] C. R. Qi, L. Yi, H. Su, and L. J. Guibas, “Pointnet++: Deep hierarchical feature learning on point sets in a metric space,” in Advances in neural information processing systems, pp. 5099–5108, 2017.
  * [26] M. Kristian, J. Matas, A. Leonardis, and F. Porikli, “A novel performance evaluation methodology for single-target trackers,” IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), pp. 2137–2155, 2016.
  * [27] J. Ku, M. Mozifian, J. Lee, A. Harakeh, and S. Waslander, “Joint 3d proposal generation and object detection from view aggregation,” in IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 1–8, 2018\.