figure table

# Self-Supervised Learning for Interventional Image Analytics: Towards Robust
Device Trackers

Saahil Islam  Pattern Recognition Lab, Friedrich-Alexander-Universität
Erlangen-Nürnberg, 91058 Erlangen, Germany  Digital Technology and Innovation,
Siemens Healthineers, Erlangen, Germany  Venkatesh N. Murthy  Digital
Technology and Innovation, Siemens Healthineers, Princeton, NJ, USA  Dominik
Neumann  Digital Technology and Innovation, Siemens Healthineers, Erlangen,
Germany  Badhan Kumar Das  Pattern Recognition Lab, Friedrich-Alexander-
Universität Erlangen-Nürnberg, 91058 Erlangen, Germany  Digital Technology and
Innovation, Siemens Healthineers, Erlangen, Germany  Puneet Sharma  Digital
Technology and Innovation, Siemens Healthineers, Princeton, NJ, USA  Andreas
Maier  Pattern Recognition Lab, Friedrich-Alexander-Universität Erlangen-
Nürnberg, 91058 Erlangen, Germany  Dorin Comaniciu  Digital Technology and
Innovation, Siemens Healthineers, Princeton, NJ, USA  Florin C. Ghesu  Digital
Technology and Innovation, Siemens Healthineers, Erlangen, Germany

###### Abstract

Purpose: An accurate detection and tracking of devices such as guiding
catheters in live X-ray image acquisitions is an essential prerequisite for
endovascular cardiac interventions. This information is leveraged for
procedural guidance, e.g., directing stent placements. To ensure procedural
safety and efficacy, there is a need for high robustness / no failures during
tracking. To achieve that, one needs to efficiently tackle challenges, such
as: device obscuration by contrast agent or other external devices or wires,
changes in field-of-view or acquisition angle, as well as the continuous
movement due to cardiac and respiratory motion.
Approach: To overcome the aforementioned challenges, we propose a novel
approach to learn spatio-temporal features from a very large data cohort of
over 16 million interventional X-ray frames using self-supervision for image
sequence data. Our approach is based on a masked image modeling technique that
leverages frame interpolation based reconstruction to learn fine inter-frame
temporal correspondences. The features encoded in the resulting model are
fine-tuned downstream in a light-weight model.
Results: Our approach achieves state-of-the-art performance and in particular
robustness compared to ultra optimized reference solutions (that use multi-
stage feature fusion, multi-task and flow regularization). The experiments
show that our method achieves 66.31% reduction in maximum tracking error
against reference solutions (23.20% when flow regularization is used);
achieving a success score of 97.95% at a 3$\times$ faster inference speed of
42 frames-per-second (on GPU). In addition, we achieve a 20% reduction in the
standard deviation of errors, which indicates much more stable tracking
performance.
Conclusions: The proposed data-driven approach achieves superior performance
particularly in robustness and speed compared to the frequently used multi-
modular approaches for device tracking. The results encourage the use of our
approach in various other tasks within interventional image analytics that
require effective understanding of spatio-temporal semantics.

###### keywords:

self-supervised learning, device tracking, interventional imaging

*Saahil Islam<EMAIL_ADDRESS>111Address all correspondence to Saahil Islam<EMAIL_ADDRESS>
##  1 Introduction

The tracking of interventional devices is an important prerequisite for
interventional specialists during catheterized cardiac interventions such as
percutaneous coronary interventions (PCI), cardiac electrophysiology (EP), or
transarterial chemoembolization (TACE) [1, 2, 3].

Tracking the tip of the catheter as a visual guidance facilitates navigation
to the desired anatomy. Furthermore, the tip of the catheter serves as an
anchor point separating the catheter from the vessel structures. The anchor
point can provide a basis for mapping angiography (high-dose X-ray with
injected contrast agent) to fluoroscopy (low-dose X-ray), thereby reducing
usage of contrast for visualizing vessels [1, 4]. In order to co-register
intravascular ultrasonography (IVUS) with angiography and perform a complete
examination of the vessel, lumen, and wall structure, catheter tip tracking
also offers a significant cue [5, 6, 7].

Figure 1: Tracking error ($\downarrow$) versus average speed ($\uparrow$) for
catheter tip tracking in coronary X-ray sequences acquired during procedures
such as invasive coronary angiography (ICA) or percutaneous coronary
intervention (PCI): (a) showing average tracking error; and (b) showing
maximum tracking error. Note that the average tracking error has $2$ different
scales indicated with a horizontal break-point for better visualization.
Runtime is measured on a Tesla V100 GPU.

However, tracking the tip of the catheter in X-ray images can be challenging
in presence of various occlusions due to contrast agent and other devices.
This is in addition to the cardiac and breathing motion of the patient.
Recently, self-supervised learning methods have been developed with the aim to
learn general features from unlabeled data to boost the performance in various
natural sequence imaging tasks. Most self-supervised pretraining methods learn
such features by identifying and removing inherent redundancies from sequence
image data. VideoMAE [8] conducts temporal downsampling on the pixel level
followed by symmetrical masking over all the sampled frames with a high
masking ratio of $90\%$. This deliberate design choice prevents the network
from learning fine inter-frame correspondences. SiamMAE [9] improves upon this
baseline by using highly asymmetric masking. However, the proposed asymmetric
masking requires feeding in the first frame entirely with $0\%$ masking which
increases the compute complexity quadratically and prevents the network from
learning spatio-temporal features over a longer period of time.

The space-time semantics in interventional cardiac image sequences differ from
natural videos in terms of both redundancies and motion. For example,
visibility may largely vary based on X-ray dosage along with varying motion
based on the acquisition frame-rate, patient’s breathing and cardiac motion.
In angiography sequences, vessels have high structural similarity with devices
such as catheters and guidewires and can gradually appear or disappear over
time.

Figure 2: Overview of key differences between our approach and previous
approaches for device tracking.

To address these challenges, in this work we bring the following contributions
in terms of both self-supervised pretraining and the downstream device
tracking:

  1. 1.

We pretrain a spatio-temporal encoder on a large database of interventional
cardiac X-ray sequences from over 20,000 patients (over 16,000,000 frames) for
robust device tracking.

  2. 2.

We propose a novel frame interpolation masked auto-encoder (FIMAE) to learn
generalized spatio-temporal features from this dataset. The pretrained spatio-
temporal features play an essential role in feature extraction and feature
matching for tracking. Our pretrained features efficiently capture the
underlying temporal motion needed for tracking, which is typically
accomplished through highly optimized supplementary modules in other device
tracking models [10, 11].

  3. 3.

To the best of our knowledge, this is the first approach which leverages
spatio-temporal pretrained features to replace a commonly used Siamese-like
architecture for single object tracking.

  4. 4.

A lightweight Vision Transformer (ViT) [12] based model is designed that
leverages the learned features to replace a traditional two-stage tracking
encoder for feature extraction and feature fusion into one spatio-temporal
encoder for highly accurate and robust real-time device tracking with an
inference speed of 42 fps on a single Tesla V100 GPU (refer to Figs. 1 and 2).

  5. 5.

We conduct comprehensive numerical experiments and demonstrate that our method
outperforms other state-of-the-art tracking methods in robustness, accuracy
and speed.

  6. 6.

We conduct a comprehensive analysis of our model’s robustness in handling long
temporal sequences and demonstrate its ability to maintain consistent
performance across diverse scenarios, including angiography, fluoroscopy, and
sequences featuring additional obstructions caused by other devices.

##  2 Related Work

#### Self-Supervised Learning

methods have been used in a variety of contexts to learn features from
unlabeled data that boost the performance in downstream tasks, such as using
pretext tasks [13, 14, 15] and contrastive learning [16, 17, 18, 19, 20, 21].
In the space of sequential image data processing (e.g., video), temporal
information has been leveraged in various ways [22, 23, 24, 25, 26, 27, 28].
However, self-supervised methods based on masked image modeling (MIM) have
shown significant promise recently, where the input is masked to a high
percentage and fed through an encoder-decoder network to predict the missing
information [29, 30, 31, 32]. Some methods use symmetrical masking on
temporally downsampled video frames to reduce space-time redundancies over a
long time period [33] [8]. In contrast, others [9] use asymmetrical masking to
learn inter-frame correspondence between frame pairs. However, we propose a
method for both reducing space-time redundancies over a long time period along
with learning fine inter-frame correspondence.

#### Siamese Natural Image Tracking

strategies leverage a Siamese architecture for matching between search and
target templates, where the extracted spatial search and template features are
matched via feature fusion or a similar matching module [34, 35, 36, 37, 38,
39, 40]. With the rise of transformers, Siamese trackers have been extended to
incorporate transformer-based models like Stark [41], Mixformer [42], among
other methods cited in [43, 44, 45].

#### Historical-Trajectory-based Natural Image Tracking

approaches leverage prompt-based methods to integrate relevant information. In
particular, the temporal information is passed into the network as prompts to
incorporate the historical trajectory information. ARTrack [46] employs a
decoder that receives these encodings as well as coordinates of the searched
object from previous frames as spatio-temporal prompts for a trajectory
proposal. Another approach, SwinTrack [47], uses a multi-head cross-attention
decoder that leverages both the encoder output and a motion token, which
represents the past object trajectory given previous bounding box predictions.

#### Device tracking in X-Ray:

Specifically for the tracking of devices in X-Ray images, multiple approaches
have been proposed, including multiple Siamese-based architectures similar to
those in natural image object tracking [48, 34]. Other methods such as Cycle
Ynet [10] employ a semi-supervised approach to address the lack of annotated
frames in the medical domain, or leverage deep learning-based Bayesian
filtering for catheter tip tracking [1]. One of the most recent approaches,
ConTrack [11], uses a Siamese architecture and a transformer-based feature
fusion model. To further refine the tracking, it incorporates a RAFT [49]
model applied to catheter body masks for estimating optical flow.

##  3 Methods

We propose a novel Frame Interpolation Masked Autoencoder (FIMAE) approach to
train a transformer model to extract spatio-temporal features based on a large
internal dataset $\mathcal{D}_{u}$. The model is designed specifically to
learn inter-frame correspondences over a large number of frames. The
pretrained encoder is then used as backbone for the downstream tracking task
using supervised learning on a dataset $\mathcal{D}_{l}$ (with expert
annotations). The pretraining method and the tracking pipeline are explained
in the following subsections.

###  3.1 Self-supervised Model Training

#### Learning space-time embeddings

Given the unlabeled dataset $\mathcal{D}_{u}$, $n$ frames are sampled from an
arbitrary sequence $S_{k}\in\mathcal{D}_{u}$, $\forall k>0$, where
$S_{k,n}=[I_{1},I_{2},...,I_{n}]$. All image frames are randomly cropped to a
size of $(h,w)=384\times 384$ pixels on a sequence level (i.e., same crop
applied to each image). Each input of size $(h,w)$ is spatially encoded into
$n\times\frac{h}{16}\times\frac{w}{16}$ tokens of dimension $D_{m}$ with no
temporal downsampling.

#### Masking strategy based on frame interpolation

In order to learn features that capture fine spatial information and fine
temporal correspondences between frames, we propose a novel masking strategy
based on frame interpolation, that overcomes the limitation of the symmetrical
tube masking proposed by VideoMAE [8]. Recall that the VideoMAE approach is
limited in capturing fine inter-frame correspondences. Traditionally, in the
domain of natural imaging, the frame interpolation task [50, 51] is defined as
the as a sum of forward warping and backward warping of any two neighboring
frames (indexed by $t>0$):

Figure 3: Overview of our framework. First, the encoder is trained to learn
spatio-temporal features from a large unlabeled dataset of angiography and
fluorscopy using Frame Interpolation Masked Autoencoder (FIMAE) (left). Then,
the weights are transfered into ViT encoder for feature extraction and feature
matching for tracking the catheter tip (right).

$I_{t+1}=\tau_{\theta_{1}}(I_{t})+\tau_{\theta_{2}}(I_{t+2})\enspace,$ (1)

where $\tau_{\theta_{1}}$ denotes the forward warping operator and
$\tau_{\theta_{2}}$ denotes the backward warping operator (parametrized by
$\theta_{1},\theta_{2}$). However, the change of appearance in coronary vessel
structures in presence of contrast can be much more complex than natural
images. Hence, a linear operation of forward and backward warping can limit
the potential of the network. In our case, we reformulate this to a learning
problem, seeking to optimize the parameters $\theta$ of a deep neural network
to learn a combined warping operation $F$ as:

$I_{t+1}=F_{\theta}(I_{t},I_{t+2})\enspace.$ (2)

In our approach, we use tube masking for every alternate frame with a ratio of
$75\%$ and combine it with frame masking. However, with such high tube masking
ratio, further masking an entire intermediate frame for frame interpolation
can make the task extremely challenging. In addition, masking an entire frame
may also lead the network to never attend to certain patch positions during
training. Hence, we mask the intermediate frame randomly to a high ratio of
$98\%$, instead. See Fig. 4 for a schematic visualization.

Figure 4: Schematic visualization of tube-frame masking.

Let $p_{t}\in\Omega_{tube}$ be the token indices of the tube masked tokens for
frame $t$, where $\Omega_{tube}$ denotes the set of all tube masked token
indices. Similarly, $q_{t}\in\Omega_{frame}$ refers to the frame masked token
indices for frame $t$ in all randomly frame masked token indices.
Mathematically, if $\rho$ is the probability for masking, $\Omega_{tube}\sim
Bernoulli(\rho_{tube})$ where different time $t$ shares the same value. On the
other hand, $\Omega_{frame}\sim Bernoulli(\rho_{frame})$, and is drawn
uniquely for each frame at $t$. Let $p^{\prime}_{t}\in\Omega^{\prime}_{tube}$
and $q^{\prime}_{t}\in\Omega^{\prime}_{frame}$ be the sets of remaining
visible token indices. Combining tube and frame masking strategies, we obtain
the following reconstruction objective for any $3$ given frames:

$I_{t},I_{t+1},I_{t+2}=F_{\theta}(I_{t}(p^{\prime}_{t}),I_{t+1}(q^{\prime}_{t+1}),I_{t+2}(p^{\prime}_{t+2}))$
(3)

where $0<t<n-1$ denotes the index of an arbitrary frame from the sampled
sequence, $I_{t}(p^{\prime}_{t})$ denotes the visible patches of frame $I_{t}$
with tube/frame masking applied. The 3-frame objective shown in Eq. 3 can be
generalized to all $n$ frames.

#### Encoder-Decoder Training

The unmasked patches are passed through a ViT encoder which adopts a joint
space-time attention. That is, each token for frame $t$, is projected and
flattened into $D_{m}$-dimensional vector query, key and value embedding:
$(q_{t},k_{t},v_{t})$. The joint space-time attention is based on the
concatenated vectors:

$\operatorname{Attention}(\mathrm{Q},\mathrm{K},\mathrm{V})=\operatorname{softmax}\left(\frac{QK^{T}}{\sqrt{d}}\right)V,$
(4)

where the variables $(Q,K,V)$ are defined as
$\mathrm{Q}=\operatorname{Concat}(q_{1},q_{2},\ldots,q_{n}),\\\
\mathrm{K}=\operatorname{Concat}(k_{1},k_{2},\ldots,k_{n}),\mathrm{V}=\operatorname{Concat}(v_{1},v_{2},\ldots,v_{n})$,
for $n$ sampled consecutive frames. The encoded visible patches are then
concatenated with learnable masked tokens. A lightweight transformer decoder
attends to the encoded patches and the masked token to reconstruct the
initially masked patches. The decoder incorporates additional positional
encoding to ensure the correct positions of the masked and unmasked patches as
per the original frames.

#### Pretraining Loss Function

We use a weighted mean squared error (MSE) loss,
$\mathcal{L}=\mathcal{L}_{\text{tube}}+\gamma\mathcal{L}_{\text{frame}}$
between the masked tokens and the reconstructed ones in the pixel space based
on the masking strategy, where $\gamma$ is the weighting factor:

$\mathcal{L}_{\text{tube}}=\frac{1}{|\Omega_{tube}|}\sum_{t={2\eta+1}}^{n}\sum_{p_{t}\in\Omega_{tube}}\|I_{t}(p_{t})-\hat{I}_{t}(p_{t})\|^{2},$
(5)

$\mathcal{L}_{\text{frame}}=\frac{1}{|\Omega_{frame}|}\sum_{t={2\eta+2}}^{n}\sum_{q_{t}\in\Omega_{frame}}\|I_{t}(q_{t})-\hat{I}_{t}(q_{t})\|^{2},$
(6)

where $I$ is the input image, $\hat{I}$ is the reconstructed image, and
$0\leq\eta\leq(n-2)/2$. We use a weighted loss for reconstruction to
compensate for the imbalance between low masked frames (less reconstruction
tokens) and highly masked frames (more reconstruction tokens). The variable
$\gamma$ is defined as the ratio of number of $\Omega_{tube}$ tokens and the
number of $\Omega_{frame}$ tokens.

###  3.2 Downstream Application: Device Tracking

Particularly for tracking the tip of the catheter, our goal is to track its
location, $\hat{y}_{t}=(u_{t},v_{t})$ at any time $t,t>0$ given a sequence of
X-ray images $\\{I_{t}\\}_{t=1}^{n}$ with a known initial location of the
catheter tip $y_{1}=(u_{1},v_{1})$ on the labeled dataset $\mathcal{D}_{l}$.
We consider the sequences $S_{k}\in\mathcal{D}_{l}$, $\forall k>0$ to have
only few annotated labels,
$S_{k,n}=[(I_{1},y_{1}),(I_{2}),...,(I_{7},y_{7}),(I_{8}),...]$. To identify
the location of the tip of the catheter at current search frame, existing
approaches build a correlation with a template frame. The template frame is
usually a small crop around the catheter tip location from a previously
predicted frame. Similar to ConTrack, during training we use three template
frames that are cropped from the first annotated frame and the previous two
annotated frames, respectively. We use the current frame for template frames
if no previously annotated frames are available. During inference, the initial
location of the catheter tip serves as the first template crop and is kept
intact. The rest two template frames are updated dynamically based on the
model’s predictions.

#### Feature transfer

The spatio-temporal transformer backbone inputs three template frames and a
search frame as four distinct frames. We interpolate the positional encoding
from the pretraining frame positions appropriately to ensure that the network
distinguishes each template and search frame as distinct frames. In
particular, each template frame and the search frame correspond to the
positions of center crops of individual frames in the pretraining setup.
Therefore, the encoder inputs
$\operatorname{Concat}(te_{1},te_{2},te_{3},se)$, where $te_{1,2,3}$ and $se$
are template patches and search patches respectively. Given that transformers
are isotropic models, we obtain an encoded feature set,
$f_{c}=\operatorname{Concat}(f_{te_{1}},f_{te_{2}},f_{te_{3}},f_{se})$. The
spatio-temporal transformer backbone is trained to extract fine inter-frame
correspondences. Hence, this results in a joint feature extraction and feature
matching between the template frames and the search frame. The overview of the
proposed model is depicted in Fig. 3.

#### Multi-task Transformer Decoder

We use a lightweight Transformer decoder similar to the original Transformer
model [52]. First, all the features $f_{c}$ are projected to a lower dimension
$d_{m}$. The decoder uses two learnable query tokens $(h_{d},m_{d})$, each for
a heatmap head and a mask head. Then, each layer first computes attention on
the query tokens as per Eq. 4. It is followed by cross-attention with encoded
features $f_{c}$, where key and value embeddings are computed by projecting
the features $f_{c}$ to dimension $d_{m}$. The resulting query tokens are then
correlated with the search features, unflattened and passed through a CNN
head:

$P_{h}=\operatorname{Conv_{h}}(\operatorname{Unflatten}(\operatorname{corr}(f_{se},h_{d}))),$
(7)

$P_{m}=\operatorname{Conv_{m}}(\operatorname{Unflatten}(\operatorname{corr}(f_{se},m_{d}))).$
(8)

The final tip coordinates are obtained by $\hat{y}=\operatorname{max}(P_{h})$,
where $P_{h}$ and $P_{m}$ refer to predicted heatmap of the catheter tip and
predicted mask of the catheter respectively. We compute soft dice loss
$\mathbf{L}_{\text{dice }}=\mathbf{L}_{h}+\lambda\mathbf{L}_{m}$, for both
heatmap and mask predictions, given by:

$\mathbf{L}_{h}=\frac{2*\sum G_{h}*P_{h}}{\sum G_{h}^{2}+\sum
P_{h}^{2}+\epsilon},$ (9)

$\mathbf{L}_{m}=\begin{cases}\frac{2*\sum G_{m}*P_{m}}{\sum G_{m}^{2}+\sum
P_{m}^{2}+\epsilon},&\text{if }G_{m}\text{ exists}\\\
0&\text{otherwise}\end{cases},$ (10)

where $G$ represents ground truth labels, and $\lambda$ is the weight for
weighting mask loss.

##  4 Experiments and Results

###  4.1 Dataset

An unlabeled internal dataset $\mathcal{D}_{u}$ of coronary X-ray sequences is
utilized to pretrain our model. $\mathcal{D}_{u}$ consists of 241,362
sequences collected from 21,589 patients, comprising 16,342,992 frames in
total. It contains both fluoroscopy (”Fluoro”) and angiography (”Angio”)
sequences. We randomly sample 10 frames at a time, with varying temporal gaps
between them, ranging from 1 to 4 frames. We repeat the last frame in
sequences where the number of frames is less than 10. The model is then
pretrained for 200 epochs with a learning rate of $1\mathrm{e}{-4}$.

Figure 5: Distribution of the datasets based on the Field of View (Positioner
Primary angle and Positioner Secondary angle): The left plot denotes the
unlabled dataset ($\mathcal{D}_{u}$) and the right plot denotes the catheter
tip dataset ($\mathcal{D}_{l}$).

For the downstream tracking task, we use dataset $\mathcal{D}_{l}$. Note that
$\mathcal{D}_{l}\cap\mathcal{D}_{u}=\emptyset$. The distribution of field of
view for both $\mathcal{D}_{u}$ and $\mathcal{D}_{l}$ is depicted in Fig. 5
and is estimated based on the Positioner angles. The Positioner Primary Angle
is defined in the transaxial plane at the imaging device’s isocenter with zero
degrees in the direction perpendicular to the patient’s chest and +90 degrees
at the Patient left hand side and -90 at the Patient right hand side. The
Positioner Secondary Angle is defined in the Sagittal Plane at the imaging
device’s isocenter with zero degrees in the direction perpendicular to the
patient’s chest. Fig. 5 shows that the distribution of the sequences in both
the datasets are concentrated around similar Positioner angles. Other
attributes from both the datasets $\mathcal{D}_{l}$ and $\mathcal{D}_{u}$ are
depicted in Table 1.

Table 1: Dataset Statistics (Range and median) for Unlabeled Dataset ($\mathcal{D}_{u}$) and Catheter Tip Dataset ($\mathcal{D}_{l}$) Attributes | Unlabeled Dataset ($\mathcal{D}_{u}$) | Catheter Tip Dataset ($\mathcal{D}_{l}$)
---|---|---
| Range | median | Range | median
FPS | 1 to 30 | 15 | 1 to 30 | 15
#Frames | 1 to 552 | 83 | 4 to 920 | 77
Resolution (mm/pixel) | 0.129 to 0.616 | 0.279 | 0.108 to 0.368 | 0.279
Peak Kilo Volt | 45.16 to 125.0 | 87.1 | 61.0 to 125.0 | 86.3
Tube Current (mA) | 1.0 to 928.0 | 757.0 | 7.0 to 904.0 | 740.0
Exposure Time (msec) | 3 to 20235 | 522 | 5 to 14160 | 503
Table 2: SNR of different categories in catheter tip dataset ($\mathcal{D}_{l}$) Fluoro | Angio | Devices
---|---|---
24.72 dB | 21.38 dB | 23.64 dB

The annotations on the frames in $\mathcal{D}_{l}$ represent the coordinates
of the tip of the catheter, which are converted to Gaussian heatmaps with
standard deviation of $\approx$ 5mm. Mask annotations of the catheter body are
also available for a subset of these annotated frames. In an average, the
catheter body takes up 0.009% of the total area of a frame. The training and
validation set consists of 2,314 sequences totaling 198,993 frames, out of
which 44,957 have annotations. In this set, 2,098 sequences are Angio and only
216 sequences are Fluoro. The test set consists of 219 sequences, where all
17,988 frames are annotated. For evaluation, we split the test set into three
categories: 94 Fluoro sequences (8,494 frames, 82 patients), 101 Angio
sequences (6,904 frames, 81 patients), and 24 Devices sequences (2,593 frames,
10 patients) [11]. The latter category, ”Devices”, covers all sequences where
sternal wires are present, which cause occlusion, thus further increasing the
difficulty of catheter tip tracking. Examples of these cases are illustrated
in Fig. 6. The SNR of the image intensity at the catheter tip with respect to
the background is shown in Table 2, further quantifying the challenge of
tracking. The SNR was calculated based on the following formula:

$SNR=20\log_{10}\frac{P_{w}}{\sigma_{f}}$ (11)

where $P_{w}$ is the mean intensity in the window of $6\times 6$ ($\approx$
2mm $\times$ 2mm) and $\sigma_{f}$ denotes the standard deviation of the
intensity of the background in the window of $30\times 30$ ($\approx$ 10mm
$\times$ 10mm) with the catheter tip as the centre of both windows.

Figure 6: Visualization of tip of the catheter in fluoroscopy, angiography and
cases with other devices.

We follow the same image pre-processing pipeline as ConTrack, i.e., we
resample and pad to size of 512 × 512 with 0.308 mm isotropic pixel spacing.
We use 160$\times$160 crops for search image, and 64$\times$64 crops for
template images. We train our model for 100 epochs, with a learning rate of
$2\mathrm{e}{-4}$ using AdamW optimizer and Cosine Annealing scheduler with
warm restarts.

###  4.2 Performance Evaluation

Table 3: Comparison study of sequence-level tracking errors (mean euclidean distance) and runtime for different methods for catheter tip tracking in coronary X-ray sequences. Best numbers are marked in bold black and the second best is marked in blue. We also show the performance of different versions of ConTrack. ConTrack-base refers to its base version which has no additional modules, ConTrack-mtmt refers to multi-task and multi-template version and ConTrack-optim is its final optimal version which has all modules including flow refinement.  Models | Median | Mean | Std | 95 Percentile | Max | Speed
---|---|---|---|---|---|---
| (mm) $\downarrow$ | (mm) $\downarrow$ | (mm) $\downarrow$ | (mm) $\downarrow$ | (mm) $\downarrow$ | (fps) $\uparrow$
SiameseRPN [34] | 7.13 | 9.01 | 6.81 | 22.37 | 46.23 | 18
STARK [41] | 2.65 | 4.14 | 4.93 | 9.24 | 31.34 | 22
MixFormer [42] | 2.68 | 5.15 | 7.1 | 19.20 | 49.29 | 20
Cycle Ynet [10] | 1.96 | 2.68 | 2.4 | 6.75 | 21.04 | 109
ConTrack-base [11] | 1.13 | 2.17 | 3.75 | 6.34 | 31.35 | 21
ConTrack-mtmt [11] | 1.12 | 1.97 | 3.61 | 5.53 | 30.37 | 19
ConTrack-optim [11] | 1.08 | 1.63 | 1.7 | 5.18 | 13.32 | 12
Ours | 1.02 | 1.44 | 1.35 | 3.52 | 10.23 | 42

We evaluate our work against state-of-the-art methods, explore the impact of
the proposed pretraining strategy, and investigate whether complex additional
tracking refinement modules are necessary. All the evaluations are performed
based on expert annotations.

#### Benchmarking against State-of-the-Art

We report the performance of our model against the state of the art device
tracking models in Table 3. Here, we evaluate the euclidean distance error in
mm between the prediction and the ground truth annotations. Overall, our
method demonstrates the best performance on the test dataset, excelling in
both precision and robustness. Our approach significantly reduces the overall
maximum error, e.g., by 66.31% against the comparable version of ConTrack
(ConTrack-mtmt) and by 23.20% against ConTrack-optim, a highly optimized
solution leveraging multi-stage feature fusion, multi-task learning and flow
regularization. In comparison to previous state-of-the-art approaches, our
approach results in fewer failures, as depicted by the error distribution in
Fig. 7. Atleast 95% of the all test cases has an error below the average
diameter of the vessels ($\approx$ 4mm). Notably, our approach stands out from
other tracking models by eliminating the need for a two-stage process
involving the extraction of spatial features and subsequent matching using
feature fusion. Instead, our spatio-temporal encoder jointly performs both.

Figure 7: Percentile plot of Cycle YNet, ConTrack and ours (a) for all test
cases and (b) zoomed in for percentiles from 90th to 100th. 95th percentile of
our method’s performance is lesser than the average diameter of the vessels
($\approx$ 4mm).

Other approaches often require two or more forward passes for the two-stage
processing to incorporate varying template-search size, which increases
computational complexity. This is further amplified by the inclusion of
additional modules, such as multi-task decoders and the flow-refinement
network in ConTrack-optim [11]. In contrast, our model accomplishes the task
with a single forward pass for both the multiple templates and the search
frame. The only additional modules in our model are the two CNN heads for
multi-task decoding. This design choice enables us to achieve a significantly
higher real-time inference speed of 42 fps on a single Tesla V100 GPU without
compromising on accuracy, as shown in Fig. 1. Despite Cycle Ynet [10] also
relying on multiple forward passes for feature extraction, its simplicity and
computationally friendly CNN architecture allows it to reach higher speed,
albeit at the expense of accuracy and robustness.

Table 4: Study of effect of pretraining startegies on the performance of the catheter tip tracking. Pretraining is performed either on our internal dataset (denoted as $\mathcal{D}_{u}$) or on natural images (in case of the first approach). Pretraining | Median | Mean | Std | Max
---|---|---|---|---
strategy | (mm) | (mm) | (mm) | (mm)
VideoMAE-Kinetics | 1.93 | 3.67 | 4.95 | 36.99
VideoMAE ($\mathcal{D}_{u}$) | 1.48 | 2.75 | 4.64 | 53.26
SiamMAE ($\mathcal{D}_{u}$) | 1.54 | 2.79 | 3.44 | 23.76
Ours: FIMAE ($\mathcal{D}_{u}$) | 1.02 | 1.44 | 1.35 | 10.23

#### Impact of Pretraining

Next, we focused on the impact of pretraining by comparing tracking
performance utilizing our proposed pretraining strategy (FIMAE) against
current prevalent pretraining methods for sequential image processing, see
Table 4. The findings indicate that pretraining on domain-specific data, as
opposed to natural images (VideoMAE-Kinetics), offers significant advantages.
However, even when including the models trained on $D_{u}$ (VideoMAE and
SiamMAE) into the comparison, our model surpasses all by more than 30% across
all reported metrics. VideoMAE lacks fine temporal correspondence between
frames, leading to non-efficient feature matching between template and search
frames. While SiamMAE has the ability to learn inter-frame correspondence, it
relies on only two frames at a time, which is insufficient to fully capture
the underlying motion. Qualitative results are shown in Fig. 8, based on a
challenging angiography sequence with contrast-based device obstruction and
other visible sternal wires. The figure shows how our model is able to handle
this challenging case by not losing track of the tip of the catheter where the
other models fail to differentiate the catheter from the sternal wires.

Figure 8: Qualitative Results. Comparison of different methods on a
challenging sequence of angiography, where tracking receives obstruction from
vessels and sternal wires (other devices). Note that the images have been
cropped around the region of interest for better visualization. The mean error
depicted in the figure is the average error computed over the entire sequence.

#### Performance without Complexity

The strength of our approach comes from the pretrained spatio-temporal
features that facilitate effective feature matching between the template
frames and the search frame. Another key advantage is its prior understanding
of the inherent cardiac/respiratory motion. This knowledge significantly
reduces or even eliminates the impact of additional modules such as flow
refinement. Our approach thereby achieves high robustness in tracking, with
minimal variations across different additional modules such as multi-task. To
illustrate this, Fig. 9(a) highlights the relative stability of the maximum
error across different versions of our model compared to the high volatility
observed in ConTrack under different module configurations. In addition,
ConTrack reaches its best performance only when utilizing all modules, in
particular including flow-refinement, which in turn leads to increased
inference time. Contrary to ConTrack, adding the flow refinement module to our
model even reduced its performance marginally in terms of accuracy ($1.54$ mm)
and robustness (max error of $11.38$ mm). We postulate that this is
attributable to the fact that while flow refinement can indeed learn intricate
temporal correspondences between the previous and current frames, it can also
propagate noise originating from inaccurately predicted catheter masks.

Figure 9: Comparison of robustness between our method and different versions
of Contrack, via (a) maximum distance error ($\downarrow$), (b) sequence-level
TSUC ($\uparrow$) and (c) frame-level TSUC ($\uparrow$).

To further assess the robustness of the tracking systems, we introduce
Tracking Success Score (TSUC), which draws parallels with most tracking
benchmarks prevalent in single object tracking in the natural image domain
[53]. TSUC is computed as the ratio of number of instances (frame or
sequence), in which the distance error falls below a specific threshold, to
the total number of instances. To establish a relevant threshold, we set it at
twice the average vessel diameter in our test dataset ($\approx 8mm$). Fig.
9(b) and Fig. 9(c) summarize the results for sequence-level and frame-level
TSUC respectively. Our approach consistently achieves an impressive 99.08%
sequence-level TSUC across all additional modules, with only a small drop to
98.61% in the multi-task configuration. At the frame level, our optimal
version (multi-task multi-template) yields a TSUC of 97.95%, compared to
93.53% for ConTrack under the same configuration. ConTrack achieves its best
frame-level TSUC of 95.44% using the flow-refinement variant.

Figure 10:  Robustness with respect to the sequence length: mean TSUC for all
sequences greater than the frame count (nframes). Note the dataset consists of
only 4 sequences with frame count greater than 210.

The robustness of a method is also influenced by its ability to effectively
handle long sequences, as the accuracy of current frame predictions is
dependent on previous frame predictions, resulting in gradual accumulation of
errors over time. We examine the mean TSUC for sequences exceeding a certain
frame count (nframes) in Fig. 10. The plot shows that our method consistently
demonstrates stable TSUC values across various sequence lengths, indicating
its robust performance. Conversely, different versions of the ConTrack exhibit
a gradual decline in mean TSUC as the frame count threshold increases,
suggesting a reduced reliability in predicting outcomes over extended
sequences.

#### Performance breakdown for different cases

We further conduct detailed comparison with the best-performing state-of-the-
art method, ConTrack, for the different image categories defined earlier, see
Fig. 11. We further compare our model’s performance with ConTrack for the
challenging cases, i.e., angiography and devices, via percentile plots in Fig.
12. In the cases of angiography, our method shows 15% improved accuracy and
45% reduction in maximum error. Similarly for the devices (occlusion)
category, where we achieve 43% better accuracy and 60% reduction in maximum
error (Fig. 11 and Fig. 12). Our model’s performance on Angio and Devices
cases is compared qualitatively with ConTrack in Fig. 13. The example cases in
the figure shows the effectiveness of our approach in the presence of complex
occlusions from the vessels and sternal wires. ConTrack achieves a better
performance than our method in Fluoro cases with a slightly better median and
lesser maximum error. However, for Fluoro, ConTrack achieves a TSUC of 99.01%
(inaccurate in 1 sequence) compared to our model’s TSUC of 97.69% (inaccurate
in 3 sequences). The inaccuracy of our model is seen in sequences where the
visibility of the catheter is faint due to low-dose X-rays. We hypothesise
that this is due to Transformer’s architecture that uses $16\times 16$ non-
overlapping patches making it less effective towards faint visibility in low-
dose X-rays compared to CNNs in ConTrack, which uses overlapping $3\times 3$
windows.

Figure 11: Breakdown of different cases in a violin plot for comparison of our
method with ConTrack. Figure 12: Percentile plots of different versions of
ConTrack and Ours for (a) Angio Cases and (b) Device cases. Figure 13:
Visualization of predictions of ConTrack and our model in two Angio sequences
(top two) and an extra device case (bottom). Note that the frames are sampled
randomly from the sequence for visualization.

###  4.3 Ablations

The following ablation studies investigate the impact of three key components
on overall tracking performance.

#### Positional Encoding

As reported in Table 5, the positional encoding strategy has notable impact on
downstream task performance.

Table 5: Effect of different positional encoding incorporated in the downstream task. Positional Encoding | Median | Mean | Std | Max
---|---|---|---|---
Naive | 1.47 | 2.51 | 3.43 | 36.24
Learnable | 1.37 | 1.86 | 1.54 | 11.22
Frame-aware (Ours) | 1.02 | 1.44 | 1.35 | 10.23

The naive positional encoding simply applies $1D$ sine-cosine positional
encoding over all the patches, and hence loses the temporal information about
the patches, resulting in unsatisfactory results. If learnable positional
encoding is used, the temporal positions are still needed to be learnt leading
to sub-optimal performance. Interpolating from the central patch positions of
the pretrained frames (frame-aware positional encoding) gives the best
results.

#### Masking Ratio

We further compare the performance of different intermediate frame masking
ratios in Table 6.

Table 6: Tracking performance with FIMAE trained with different intermediate frame masking ratios. i.e. masking ratio of $\Omega_{frame}$. Frame Masking ratio | Median | Mean | Std | Max
---|---|---|---|---
95% | 1.09 | 1.47 | 1.24 | 10.34
98% | 1.02 | 1.44 | 1.35 | 10.23
100% | 1.08 | 1.78 | 2.09 | 15.12

Best results are obtained with an intermediate frame masking ratio of $98\%$.
While results with $95\%$ are largely equivalent, there is a notable reduction
in performance when the entire frame is masked, which may be due to the lack
of patches and its relative positions information during pretraining.

#### Effect of initialization

Recall that the first template crop during both training and inference was
obtained from the initial catheter tip location and is not updated. We explore
its impact in Table 7. To assess its importance, we conduct two experiments.
First, we dynamically update the initial template frame during inference, like
the rest. Second, we introduce random noise (2 to 16 pixels) to the initial
tip location instead of updating the template. Our findings highlight the
crucial role of initialization in tracking. Updating the initial template
frame worsens performance due to greater accumulated prediction errors over
time compared to the original setup. Additionally, even small noise levels of
2 pixels can noticeably affect performance, increasing the maximum error by 5
pixels.

Table 7: Significance of initialization in catheter tip tracking: How the performance is affected if first template frame is updated or some noise is introduced to the initial tip coordinates. Upate First template | init noise ($\mp$ px) | Median | Mean | Std | Max
---|---|---|---|---|---
✓ | 0 | 1.17 | 1.90 | 2.51 | 24.55
✗ | 16 | 1.53 | 2.44 | 3.18 | 25.42
✗ | 8 | 1.45 | 1.94 | 2.25 | 26.45
✗ | 4 | 1.13 | 1.69 | 2.07 | 20.72
✗ | 2 | 1.05 | 1.55 | 1.60 | 15.36
✗ | 0 | 1.02 | 1.44 | 1.35 | 10.23

#### Modality Bias

Table 8: Performance variation across modalities based on modality-specific training. Trained on | Fluoro | Angio | Devices
---|---|---|---
| mean | median | max | mean | median | max | mean | median | max
Fluoro data | 1.44 | 0.84 | 10.54 | 4.15 | 2.36 | 22.96 | 6.58 | 4.62 | 19.47
Angio data | 1.41 | 0.75 | 11.42 | 1.49 | 1.14 | 5.56 | 2.80 | 0.99 | 22.55
All data | 1.24 | 0.75 | 10.23 | 1.61 | 1.38 | 7.33 | 1.54 | 0.98 | 4.27

The distribution between Angio and Fluoro varies to some degree in terms of
dosage and presence of contrasted vessel structures. We remind the reader that
in our training dataset the distribution of Angio:Fluoro sequences was
2098:216 of the total of 2314 sequences. Our objective in this study is to
develop a model that exhibits strong performance across both modalities. We
present the results of training on individual modalities compared to training
on combined data in Table 8. Our findings indicate that training solely on one
modality results in suboptimal performance on the other modality. Notably,
while training on Angio data yields an improvement in Angio performance,
training exclusively on Fluoro data fails to enhance performance in Fluoro. We
hypothesize that a possible reason for this effect is the imbalance of
2098:216 (Angio to Fluoro sequences), with the following effects:

  1. 1.

2098 Angio sequences is a large enough dataset to ensure good Angio
performance when training on this data alone;

  2. 2.

216 Fluoro sequences is too little to power the training of a large
transformer model, leading to inferior results when training/testing on Fluoro
only;

  3. 3.

transitioning from Angio to using all data for training has a negative effect
on the Angio test performance - we hypothesize that adding the few Fluoro
sequences to training increases the complexity of the training problem, as the
distribution of Angio training cases is enhanced with the distribution of
Fluoro cases, based on only 216 examples; and

  4. 4.

transitioning from Fluoro to using all data for training has a positive effect
on the Fluoro test performance - we hypothesize this is because the 216 Fluoro
sequences are complement with many more non-contrasted frames from all Angio
sequences to substantially increase the dataset, and thereby improve
performance.

Furthermore, the challenges posed by device obstruction exhibit nuanced
differences between fluoro and angio, contributing to reduced performance when
the model is trained on a single modality.

##  5 Conclusion

In this study, we present Frame Interpolation Masked Autoencoder (FIMAE), a
Masked Imaging Modeling (MIM) approach, which is introduced for the purpose of
acquiring generalized features from a large unlabeled dataset containing more
than 16 million interventional X-ray frames, with the objective of device
tracking. FIMAE overcomes the limitation of tube masking as proposed in
VideoMAE, and applies frame interpolation-based masking for capturing fine
inter-frame correspondences. The acquired features are subsequently applied to
the task of device tracking within fluoroscopy and angiography image
sequences. Our pre-trained FIMAE encoder surpasses all prevalent MIM-based
pretraining methods for sequential imaging processing.

The spatio-temporal features acquired during the pretraining phase
significantly influence the extraction and matching of features for the
purpose of device tracking. We demonstrate that an efficient spatio-temporal
encoder can replace the frequently utilized Siamese-like architecture,
yielding a computationally lightweight model that maintains a high degree of
precision and robustness in the tracking task. By adopting our methodology, we
achieve a noteworthy 23.2% reduction in maximum tracking error, even without
the incorporation of supplementary modules such as flow refinement, when
compared to the state-of-the-art multi-modular optimized approach. This
performance enhancement is accompanied by a frame-level TSUC score of 97.95%
at $3\times$ faster inference speed than the state-of-the-art method. The
results also show that our approach achieves superior tracking performance,
particularly in the challenging cases where occlusions and distractors are
present.

#### Limitations and Future Work

Our investigation is primarily centered on leveraging pre-trained features for
the tracking of devices within X-ray sequences. Consequently, we contend that
the pre-trained model can be further extended to other tasks within
interventional image analytics, such as stenosis detection, guidewire
localization, and vessel segmentation. Furthermore, the absence of annotated
frames within our sequential imaging dataset imposes a constraint on the
utilization of historical trajectory information, a commonly exploited
approach in recent single object tracking methodologies in the natural imaging
domain. Thus, a more comprehensive investigation is needed to effectively make
use of this information in our specific context.

### Disclaimer

The concepts and information presented in this paper are based on research
results that are not commercially available.

### Disclosures

There are no conflicts of interest.

### Code, Data, and Materials Availability

Based on the data usage agreements, the data cannot be shared with the
community. More information about the code can be shared upon request.

## References

  * [1] H. Ma, I. Smal, J. Daemen, et al., “Dynamic coronary roadmapping via catheter tip tracking in x-ray fluoroscopy with deep learning based bayesian filtering,” Medical image analysis 61, 101634 (2020).
  * [2] K. E. Odening, A.-M. Gomez, D. Dobrev, et al., “Esc working group on cardiac cellular electrophysiology position paper: relevance, opportunities, and limitations of experimental models for cardiac electrophysiology research,” EP Europace 23(11), 1795–1814 (2021).
  * [3] A. Facciorusso, R. Licinio, N. Muscatiello, et al., “Transarterial chemoembolization: evidences from the literature and applications in hepatocellular carcinoma patients,” World journal of hepatology 7(16), 2009 (2015).
  * [4] K. Piayda, L. Kleinebrecht, S. Afzal, et al., “Dynamic coronary roadmapping during percutaneous coronary intervention: a feasibility study,” European journal of medical research 23, 1–7 (2018).
  * [5] P. Wang, T. Chen, O. Ecabert, et al., “Image-based device tracking for the co-registration of angiography and intravascular ultrasound images,” in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2011: 14th International Conference, Toronto, Canada, September 18-22, 2011, Proceedings, Part I 14, 161–168, Springer (2011).
  * [6] T. Araki, N. Ikeda, N. Dey, et al., “A comparative approach of four different image registration techniques for quantitative assessment of coronary artery calcium lesions using intravascular ultrasound,” Computer methods and programs in biomedicine 118(2), 158–172 (2015).
  * [7] P. Wang, O. Ecabert, T. Chen, et al., “Image-based co-registration of angiography and intravascular ultrasound images,” IEEE transactions on medical imaging 32(12), 2238–2249 (2013).
  * [8] Z. Tong, Y. Song, J. Wang, et al., “Videomae: masked autoencoders are data-efficient learners for self-supervised video pre-training,” Advances in neural information processing systems 35, 10078–10093 (2022).
  * [9] A. Gupta, J. Wu, J. Deng, et al., “Siamese masked autoencoders,” arXiv preprint arXiv:2305.14344 (2023).
  * [10] J. Lin, Y. Zhang, A.-a. Amadou, et al., “Cycle ynet: semi-supervised tracking of 3d anatomical landmarks,” in Machine Learning in Medical Imaging: 11th International Workshop, MLMI 2020, Held in Conjunction with MICCAI 2020, Lima, Peru, October 4, 2020, Proceedings 11, 593–602, Springer (2020).
  * [11] M. Demoustier, Y. Zhang, V. N. Murthy, et al., “Contrack: contextual transformer for device tracking in x-ray,” arXiv preprint arXiv:2307.07541 (2023).
  * [12] A. Dosovitskiy, L. Beyer, A. Kolesnikov, et al., “An image is worth 16x16 words: transformers for image recognition at scale,” arXiv preprint arXiv:2010.11929 (2020).
  * [13] C. Doersch, A. Gupta, and A. A. Efros, “Unsupervised visual representation learning by context prediction,” in Proceedings of the IEEE international conference on computer vision, 1422–1430 (2015).
  * [14] D. Pathak, R. Girshick, P. Dollár, et al., “Learning features by watching objects move,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2701–2710 (2017).
  * [15] S. Gidaris, P. Singh, and N. Komodakis, “Unsupervised representation learning by predicting image rotations,” arXiv preprint arXiv:1803.07728 (2018).
  * [16] Z. Wu, Y. Xiong, S. X. Yu, et al., “Unsupervised feature learning via non-parametric instance discrimination,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 3733–3742 (2018).
  * [17] K. He, H. Fan, Y. Wu, et al., “Momentum contrast for unsupervised visual representation learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 9729–9738 (2020).
  * [18] M. Caron, H. Touvron, I. Misra, et al., “Emerging properties in self-supervised vision transformers. 2021 ieee,” in CVF International Conference on Computer Vision (ICCV), 3 (2021).
  * [19] T. Chen, S. Kornblith, M. Norouzi, et al., “A simple framework for contrastive learning of visual representations,” in International conference on machine learning, 1597–1607, PMLR (2020).
  * [20] J.-B. Grill, F. Strub, F. Altché, et al., “Bootstrap your own latent-a new approach to self-supervised learning,” Advances in neural information processing systems 33, 21271–21284 (2020).
  * [21] X. Chen and K. He, “Exploring simple siamese representation learning,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 15750–15758 (2021).
  * [22] P. Sermanet, C. Lynch, Y. Chebotar, et al., “Time-contrastive networks: Self-supervised learning from video,” in 2018 IEEE international conference on robotics and automation (ICRA), 1134–1141, IEEE (2018).
  * [23] C. Sun, F. Baradel, K. Murphy, et al., “Learning video representations using contrastive bidirectional transformer,” arXiv preprint arXiv:1906.05743 (2019).
  * [24] T. Han, W. Xie, and A. Zisserman, “Video representation learning by dense predictive coding,” in Proceedings of the IEEE/CVF International Conference on Computer Vision Workshops, 0–0 (2019).
  * [25] C. Feichtenhofer, H. Fan, B. Xiong, et al., “A large-scale study on unsupervised spatiotemporal representation learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 3299–3309 (2021).
  * [26] A. Recasens, P. Luc, J.-B. Alayrac, et al., “Broaden your views for self-supervised video learning,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 1255–1265 (2021).
  * [27] R. Qian, T. Meng, B. Gong, et al., “Spatiotemporal contrastive video representation learning,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 6964–6974 (2021).
  * [28] N. Park, W. Kim, B. Heo, et al., “What do self-supervised vision transformers learn?,” arXiv preprint arXiv:2305.00729 (2023).
  * [29] J. Devlin, M.-W. Chang, K. Lee, et al., “Bert: Pre-training of deep bidirectional transformers for language understanding,” arXiv preprint arXiv:1810.04805 (2018).
  * [30] K. He, X. Chen, S. Xie, et al., “Masked autoencoders are scalable vision learners,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 16000–16009 (2022).
  * [31] H. Bao, L. Dong, S. Piao, et al., “Beit: Bert pre-training of image transformers,” arXiv preprint arXiv:2106.08254 (2021).
  * [32] Z. Xie, Z. Zhang, Y. Cao, et al., “Simmim: A simple framework for masked image modeling,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9653–9663 (2022).
  * [33] C. Feichtenhofer, Y. Li, K. He, et al., “Masked autoencoders as spatiotemporal learners,” Advances in neural information processing systems 35, 35946–35958 (2022).
  * [34] B. Li, J. Yan, W. Wu, et al., “High performance visual tracking with siamese region proposal network,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 8971–8980 (2018).
  * [35] B. Li, W. Wu, Q. Wang, et al., “Evolution of siamese visual tracking with very deep networks,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 (2019).
  * [36] H. Fan and H. Ling, “Siamese cascaded region proposal networks for real-time visual tracking,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 7952–7961 (2019).
  * [37] Z. Zhu, Q. Wang, B. Li, et al., “Distractor-aware siamese networks for visual object tracking,” in Proceedings of the European conference on computer vision (ECCV), 101–117 (2018).
  * [38] H. Fan and H. Ling, “Cract: Cascaded regression-align-classification for robust tracking,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), 7013–7020, IEEE (2021).
  * [39] Y. Yu, Y. Xiong, W. Huang, et al., “Deformable siamese attention networks for visual object tracking,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 6728–6737 (2020).
  * [40] Z. Zhang, Y. Liu, X. Wang, et al., “Learn to match: automatic matching network design for visual tracking,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, 13339–13348 (2021).
  * [41] B. Yan, H. Peng, J. Fu, et al., “Learning spatio-temporal transformer for visual tracking,” in Proceedings of the IEEE/CVF international conference on computer vision, 10448–10457 (2021).
  * [42] Y. Cui, C. Jiang, L. Wang, et al., “Mixformer: end-to-end tracking with iterative mixed attention,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 13608–13618 (2022).
  * [43] J. Kugarajeevan, T. Kokul, A. Ramanan, et al., “Transformers in single object tracking: an experimental survey,” IEEE Access (2023).
  * [44] N. Wang, W. Zhou, J. Wang, et al., “Transformer meets tracker: Exploiting temporal context for robust visual tracking,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 1571–1580 (2021).
  * [45] X. Chen, B. Yan, J. Zhu, et al., “Transformer tracking,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 8126–8135 (2021).
  * [46] X. Wei, Y. Bai, Y. Zheng, et al., “Autoregressive visual tracking,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 9697–9706 (2023).
  * [47] L. Lin, H. Fan, Z. Zhang, et al., “Swintrack: a simple and strong baseline for transformer tracking,” Advances in Neural Information Processing Systems 35, 16743–16754 (2022).
  * [48] J. Bromley, I. Guyon, Y. LeCun, et al., “Signature verification using a” siamese” time delay neural network,” Advances in neural information processing systems 6 (1993).
  * [49] Z. Teed and J. Deng, “Raft: Recurrent all-pairs field transforms for optical flow,” in Computer Vision–ECCV 2020: 16th European Conference, Glasgow, UK, August 23–28, 2020, Proceedings, Part II 16, 402–419, Springer (2020).
  * [50] H. Jiang, D. Sun, V. Jampani, et al., “Super slomo: High quality estimation of multiple intermediate frames for video interpolation,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), (2018).
  * [51] S. Niklaus, L. Mai, and F. Liu, “Video frame interpolation via adaptive separable convolution,” in Proceedings of the IEEE International Conference on Computer Vision (ICCV), (2017).
  * [52] A. Vaswani, N. Shazeer, N. Parmar, et al., “Attention is all you need,” Advances in neural information processing systems 30 (2017).
  * [53] H. Fan, L. Lin, F. Yang, et al., “Lasot: A high-quality benchmark for large-scale single object tracking,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 5374–5383 (2019).

##  Appendix A Pretraining details

The detailed architecture illustration and the implementation details of the
pretraining are illustrated in Table 9 and Table 10 respectively. We use a
10-frame vanilla ViT-Base as our encoder architecture which incorporates joint
space-time attention on visible patches. The decoder is of lower dimension and
lower depth than encoder which incorporates similar joint space-time attention
on all patches. The decoder is only responsible for reconstruction and is
discarded for downstream tasks.

Stage | Vision Transformer (Base) | Output Size
---|---|---
data | temporal gaps = [1,2,3,4] | 1×10×384×384
Patch Embed | 1×16×16, 768 | 768×10×576
| stride 1×16×16 |
mask |  $\rho$ = Tube 75% + frame 98% | 768×10×[576×(1-$\rho$)]
encoder | [MHA(768), MLP(3072)] x 12 | 768×10×[576×(1-$\rho$)]
projector | MLP(384) & |
| concat learnable tokens | 768×10×576
decoder | [MHA(384), MLP(1536)] x 4 | 384×10×576
projector | MLP(256) | 256×10x576
reshape | from 256 to 1×1×16×16 | 1×10×384×384
Table 9: Architecture details of FIMAE. We use a 10-frame vanilla ViT-Base as our architecture. ”MHA” here denotes the joint space-time self-attention. The output sizes are denoted by C×T×S for channel, temporal and spatial sizes. Config | Name/Params
---|---
Optimizer | AdamW
Base learning rate | 1.5e-4
weight decay | 1e-4
optimizer momentum |  $\beta_{1}$, $\beta_{2}$ = 0.9, 0.95
batch size | 8
learning rate schedule | cosine decay
warmup epochs | 15
augmentation | MultiScaleCrop
Table 10: Pretraining Setting

##  Appendix B Downstream Model Details

The architectural detail of the downstream tracking model is depicted in Table
11. The encoder is same as the pretraining encoder, whereas the decoder is a
lightweight Transformer decoder, followed by two CNN heads that outputs the
catheter tip heatmap and catheter body mask respectively. The implementation
details are further explained in Table 12.

Stage | ViT-Base + Multi-task Decoder | Output Size
---|---|---
data | 3 Templates + 1 Search | 1×3×64×64, 1×1×160×160
Patch Embed | 1×16×16, 768 | 768×148
| stride 1×16×16 |
| & concatenate |
encoder | [MHA(768), MLP(3072)] x 12 | 768×148
projector | MLP(256) | 256×148
decoder | Query = (256, 2) & | 256×148, 256×2
| [CA(256), MLP(1024)] x 6 |
cross-correlate | Unconcatenate and matmul | 256×2×10×10
mask head | (Upconv, Conv, Batchnorm, GeLU) x 4 | 4×160×160
mask project | Linear(4,1) | 1×160×160
heatmap head | (Upconv, Conv, Batchnorm, GeLU) x 4 | 4×160×160
heatmap project | Linear(4,1) | 1×160×160
Table 11: Architecture details of downstream tracking model. ”CA” refers to Cross-Attention. Config | Name/Params
---|---
Optimizer | AdamW
Base learning rate | 6e-4
weight decay | 1e-4
optimizer momentum |  $\beta_{1}$, $\beta_{2}$ = 0.9, 0.95
batch size | 42
learning rate schedule | cosine decay
warmup epochs | 10
augmentation | Horizontal Flip
| Vertical Flip
| Random Rotation (-10∘, 10∘)
Table 12: Finetuning Setting

Saahil Islam is a second-year PhD student at Friedrich Alexander University in
Erlangen, Germany. Having obtained an MSc from the same institution with a
specialization in computer vision and segmentation for glacier calving front
detection from remote sensing images, Saahil’s current research focuses on
medical imaging within the realm of image-guided therapy, conducted in
collaboration with Siemens Healthineers. Specifically, Saahil is dedicated to
leveraging artificial intelligence to enhance real-time systems in image-
guided therapy, aiming to contribute to advancements in this critical field.

Venkatesh N. Murthy a seasoned computer scientist, brings over a decade of
experience in Computer Vision and Machine Learning. Having earned his Ph.D.
from UMass Amherst, Venkatesh has garnered acclaim through numerous
publications in prestigious conferences and journals, amassing over 700
citations and securing multiple patents. Currently, as a Staff Research
Scientist at Siemens Healthineers in Princeton, NJ, Venkatesh focuses he
focuses on advancing object classification, detection, and tracking
technologies, driving innovation in healthcare technology.

Biographies and photographs of the other authors are not available.

###### List of Figures

  1. 1 Tracking error ($\downarrow$) versus average speed ($\uparrow$): (a) showing average tracking error; and (b) showing maximum tracking error.
  2. 2 Overview of key differences between our approach and previous approaches for device tracking.
  3. 3 Overview of our framework: Pretraining using Frame Interpolation Masked Autoencoder (FIMAE) (left) and downstream tracking model (right).
  4. 4 Schematic visualization of tube-frame masking.
  5. 5 Distribution of the datasets based on the Field of View
  6. 6 Visualization of tip of the catheter in fluoroscopy, angiography and cases with other devices.
  7. 7 Percentile plot of Cycle YNet, ConTrack and ours (a) for all test cases and (b) zoomed in for percentiles from 90th to 100th.
  8. 8 Qualitative Results. Comparison of different methods on a challenging sequence of angiography, where tracking receives obstruction from vessels and sternal wires (other devices)
  9. 9 Comparison of robustness between our method and different versions of Contrack, via (a) maximum distance error ($\downarrow$), (b) sequence-level TSUC ($\uparrow$) and (c) frame-level TSUC ($\uparrow$).
  10. 10 Robustness with respect to the sequence length.
  11. 11 Breakdown of different cases in a violin plot for comparison of our method with ConTrack.
  12. 12 Percentile plots of different versions of ConTrack and Ours for (a) Angio Cases and (b) Device cases.
  13. 13 Visualization of predictions of ConTrack and our model in two Angio sequences (top two) and an extra device case (bottom).

###### List of Tables

  1. 1 Dataset Statistics (Range and median) for Unlabeled Dataset ($\mathcal{D}_{u}$) and Catheter Tip Dataset ($\mathcal{D}_{l}$)
  2. 2 SNR of different categories in catheter tip dataset ($\mathcal{D}_{l}$)
  3. 3 Comparison study of sequence-level tracking errors and runtime for different methods.
  4. 4 Study of effect of pretraining startegies on the performance of the catheter tip tracking.
  5. 5 Effect of different positional encoding incorporated in the downstream task.
  6. 6 Tracking performance with FIMAE trained with different intermediate frame masking ratios.
  7. 7 Significance of initialization in catheter tip tracking.
  8. 8 Performance variation across modalities based on modality-specific training.
  9. 9 Architecture details of FIMAE. We use a 10-frame vanilla ViT-Base as our architecture. ”MHA” here denotes the joint space-time self-attention. The output sizes are denoted by C×T×S for channel, temporal and spatial sizes.
  10. 10 Pretraining Setting
  11. 11 Architecture details of downstream tracking model. ”CA” refers to Cross-Attention.
  12. 12 Finetuning Setting