# Towards Two-view 6D Object Pose Estimation: A Comparative Study on Fusion
Strategy

Jun Wu1, Lilu Liu1, Yue Wang1 and Rong Xiong1 1Jun Wu, Lilu Liu, Yue Wang, and
Rong Xiong are with the State Key Laboratory of Industrial Control Technology
and Institue of Cyber-Systems and Control, Zhejiang University, Zhejiang,
China.Corresponding author<EMAIL_ADDRESS>Co-corresponding author,
<EMAIL_ADDRESS>
###### Abstract

Current RGB-based 6D object pose estimation methods have achieved noticeable
performance on datasets and real world applications. However, predicting 6D
pose from single 2D image features is susceptible to disturbance from changing
of environment and textureless or resemblant object surfaces. Hence, RGB-based
methods generally achieve less competitive results than RGBD-based methods,
which deploy both image features and 3D structure features. To narrow down
this performance gap, this paper proposes a framework for 6D object pose
estimation that learns implicit 3D information from 2 RGB images. Combining
the learned 3D information and 2D image features, we establish more stable
correspondence between the scene and the object models. To seek for the
methods best utilizing 3D information from RGB inputs, we conduct an
investigation on three different approaches, including Early- Fusion, Mid-
Fusion, and Late-Fusion. We ascertain the Mid- Fusion approach is the best
approach to restore the most precise 3D keypoints useful for object pose
estimation. The experiments show that our method outperforms state-of-the-art
RGB-based methods, and achieves comparable results with RGBD-based methods.

##  I Introduction

6D object pose estimation aspires to estimate the rotation and translation of
interested objects with regard to certain canonical coordinates. Accurate
object pose estimation is the key to many real-world applications, such as
robotic manipulation, augmented reality, and human-robot interactions. This is
a challenging problem due to the variety of objects appearance, occlusions
between objects, and clutter in the scene.

Based on the sensors they adopt, current model-based object pose estimation
methods can be roughly categorized into two classes: RGB-based methods
[1][2][3] and RGBD-based methods [4][5][6]. Though the scale is a known
quantity in model-based object pose estimation problem, previous researches
have shown that the performance of RGB-based methods are generally less
competitive than the RGBD-based methods[7], mostly due to the lacking of 3D
structure information. Since an image is the projection of the object under
certain lighting condition and observation angle, predicting object pose with
only RGB inputs could be affected by low resolutions, ill observation pose,
changing of environment, and textureless or resemblant surface, thus involves
more ambiguities. On the other hand, RGBD-based methods usually integrate 3D
structures with 2D image features to estimate object poses[8][9]. The factors
impacting one kind of input modality could barely impact the other, thus this
integration of two modalities of input features creates more robust prediction
results.

Figure 1: Illustration of our ideas. We show the general pipelines of RGB-
based, RGBD-based methods and our methods. RGB-based methods take RGB image as
input, establish correspondence between image features and object models, then
solve for the pose. While RGBD-based methods deploy image features and point
cloud features to build the correspondence, which enhances the robustness of
estimation. Our motivation is to narrow down the performance gap between RGB-
based methods and RGBD-based methods, by proposing a framework combining image
features from 2 input images to learn the implicit 3D information.

Therefore, a reasonable approach to enhance the pose estimation accuracy of
RGB-based methods would be learning 3D geometric information from the RGB
inputs. Though 2D images lack awareness of 3D geometric information in nature,
two or more frames of such images combined together implicitly bring out the
depth information, as has been verified in many stereo or multi-view stereo
tasks [10][11][12]. Thus, we opt to learn the implicit 3D information from two
or more RGB images to enhance the accuracy in estimating object poses.
Luckily, making two or more observations of the scene with a certain relative
transformation is commonly accessible for robots, rendering the applicability
of this pipeline.

In this paper, we investigate 3 different approaches to learn the 3D
information from 2 input images, including fusing images to restore depth map
(Early-Fusion), fusing 2D keypoints predicted from each image to restore 3D
keypoints (Late-Fusion), and fusing 2D feature maps to restore characterized
3D space points (Mid-Fusion). We analyze all approaches on real word datasets
and discover that the Mid-Fusion approach is able to restore the most precise
3D information useful for object pose estimation task. And by comparing to
other state-of-the-art methods, we prove that combining 2D image features with
learned 3D information effectively enhance the pose estimation accuracy.

To this end, the major contributions of this paper are as follows:

  * •

To narrow down the performance gap between RGB-based methods and RGBD-based
methods, we propose a framework for 6D object pose estimation, which learns
the implicit 3D information from two RGB images to solve the pose.

  * •

To seek for the method best utilizing 3D information useful for object pose
estimation task, we conduct an investigation on 3 different approaches,
including Early-Fusion, Mid-Fusion, and Late-Fusion, and show that the Mid-
Fusion approach yields more precise 3D information.

  * •

We compare our method to state-of-the-art methods on benchmarks. Our results
show that integrating 3D information with 2D features effectively enhance the
object pose estimation accuracy.

##  II Related Works

RGB-based Object Pose Estimation. Many object pose estimation methods take RGB
images as input, and tackle this problem by learning the correspondence
between 2D image features and 3D CAD models [2][13][3][14][15]. [16] and [17]
employs CNN to predict the 3D bounding box corners of the object in the image,
and associate the corners with those in 3D CAD models to solve the pose. Since
the corners are artificial, the estimation results are not satisfactory. To
use more reliable correspondence, PVNet [18] selects keypoints from the
object’s model, and train a CNN to predict the vertex from every pixel to
those keypoints then vote with confidence. Besides keypoints, HybridPose [19]
employs edge vectors and symmetry correspondence to enrich the feature space
for better estimation. Beyond establishing correspondence between 2D features
and 3D models, some methods seek to directly learn the rationale between 2D
features and 6D poses with neural networks[1][20]. Furthermore, [21] and [22]
explicitly learn the dense 2D-3D correspondence between images and canonical
models, then regress poses by a Patch-PnP network.

RGBD-based Object Pose Estimation. To further improve the estimation accuracy,
another pipeline, RGBD-based methods additionally deploy depth information
[4][5][23][24]. Early research [25] [26] compose contour vectors from RGB
image and surface normal vectors from depth image, and estimate the pose by
template matching. PVN3D [9] uses a neural network to separately extract image
and point features, then encode them pointwisely to vote for 3D keypoints, and
solve the pose with 3D-3D correspondence. REDE [6] and L6DNet [27] both encode
multimodal features, and apply a 3D registration algorithm to solve the pose.
Because of the extra depth information, RGBD-based methods generally achieve
better results than RGB-based methods.

Multi-view Stereo Object Pose Estimation. Recently, some methods deploy
multiframe observations to further refine pose estimation results in a
optimization framework [28][29][30][31][7]. They formulate a global
optimization function to refine the object poses predicted from each
observation by minimizing the reprojection loss or average distance loss. This
backend framework is generally adaptive to work with other frontend pose
estimation methods, including our proposed one. [32] is the method most
similar to ours, which proposes a pose estimation framework for transparent
objects. They fuse the stereo RGB inputs to restore dense depth map, assigning
depth value to the predicted 2D keypoints, then solve the 3D-3D registration
problem. Our proposed Early-Fusion approach shares the same idea with them,
and will be discussed later in the experiments.

Multi-view Stereo Scene Reconstruction. Restoring scene geometry from multiple
overlapping images is a problem widely studied. Early Methods decouples the
complex multi-view problem into relatively small problems with only one
reference and a few source images at a time [33][34]. Based on plane-sweep
stereo, many recent learning-based methods[35][36][37] build cost volumes with
warped 2D image features from multiple observations, regularize them with 3D
CNNs and regress the depth. Cas-MVSNet[38] proposes a cascade cost volume
mechanism based on feature pyramids and estimates the depth in a coarse-to-
fine manner to reduce the computation resources. We follow them to build our
Early-Fusion pipeline to restore the depth and evaluate its capability in
restoring useful 3D keypoints for pose estimation.

Figure 2: Methods Overview. We show the overall pipeline of our proposed
methods. Taking the input reference image $I^{r}$ and query image $I^{q}$, we
extract 2D image feature $F^{r}$ and $F^{q}$ with a shared multi-scale feature
extractor network. Then, 3 different approaches are developed to predict 3D
keypoints $P_{k}^{r}$, including Early-Fusion, Mid-Fusion, and Late-Fusion.
The Early-Fusion approach fuses the two input images to predict depth map
$D^{r}$ via a DepthNet, thus the predicted 2D keypoints $x_{k}^{r}$ are
projected to $P_{k}^{r}$. The Late-Fusion approach fuses $x_{k}^{r}$ and
$x_{k}^{q}$ by directly triangulating them to get $P_{k}^{r}$. The Mid-Fusion
approach fuse the two feature maps to restore a characterized 3D geometry
volume $V_{r}$, then predicts $P_{k}^{r}$ with 3D CNNs. Notice that for a fair
comparison, the 2D keypoints $x_{k}^{r}$ are predicted by an additional head
of the shared feature extractor. Last, $P_{k}^{r}$ are used to solve the pose
in a closed form with model keypoints $P_{k}^{m}$.

##  III Method

In this section, we introduce our overall pipeline to combine two images for
object pose estimation, as illustrated in Fig. 2. Taking the input reference
image $I^{r}$ and query image $I^{q}$, we extract 2D image feature $F^{r}$ and
$F^{q}$ with a shared feature extractor network. Then, 3 different approaches
are developed to predict 3D keypoints $P_{k}^{r}$, including Early-Fusion,
Mid-Fusion, and Late-Fusion. The Early-Fusion approach fuse the two input
images to obtain depth map $D^{r}$, thus the predicted 2D keypoints
$x_{k}^{r}$ are projected to $P_{k}^{r}$. The Mid-Fusion approach fuse the two
feature maps to restore a characterized 3D geometry volume $V^{r}$, then
predicts $P_{k}^{r}$ with 3D CNNs. And the Late-Fusion approach fuses
$x_{k}^{r}$ and $x_{k}^{q}$ directly to get $P_{k}^{r}$. Notice that for a
fair comparison, the 2D keypoints $x_{k}^{r}$ are predicted by an additional
head of the shared feature extractor. Last, $P_{k}^{r}$ are used to solve the
pose in a closed form with model keypoints $P_{k}^{m}$.

###  III-A 2D Feature Extraction

For a fair comparison, we adopt a shared feature extraction network for all
our approaches. Since feature extraction is not the focus of this paper, we
follow [18] to build a multi-scale convolutional neural network, to extract
features at multiple resolutions, and predicts 2D keypoints
$\\{x_{k}^{r},x_{k}^{q}\\}$ for Early-Fusion pipeline and Late-Fusion
pipeline.

###  III-B 3D Keypoints Prediction

With the goal of learning useful 3D information from two input RGB images, we
explore 3 different approaches to predict the 3D keypoints, including Early-
Fusion, Mid-Fusion, and Late-Fusion. Here we present how we develop these
approaches separately.

Early-Fusion Approach: The most intuitive way to learn 3D information from
input images is to directly recover depth from them. This approach takes
advantage of the abundant off-the-shelf methods in the multi-view stereo depth
estimation field, thus is rather facile to implement. Without loss of
generality, we follow [38] to build a cascade of 3D geometry volumes from
warped 2D image features, regularize them with independent 3D CNN blocks and
regress the depth map. Then, for a fair comparison with the other two
approaches, we predict 2D keypoints from the input reference image following
[18]. The extracted image features are fed into a vector prediction head to
predict the vectors from each pixel to its predicted keypoints pixels, then a
RANSAC voting mechanism is applied to decide the mean and covariance of each
keypoint. Last, the 3D keypoints are obtained by assigning depth values to the
predicted 2D keypoints.

Late-Fusion Approach: Another nature approach to learn 3D information is to
fuse the predicted 2D keypoints from both inputs into 3D keypoints. Likewise,
we follow [18] to predict 2D keypoint for equitable contrast. Then, we
triangulate these 2D keypoints to 3D with their relative camera poses by
OpenCV library algorithm.

Mid-Fusion Approach: Despite the two straightforward approaches, we take a
deeper look at the essence of keypoint prediction process. Predicting
keypoints is to seek the location where the features best match the target
keypoint previously chosen in the object models. Hence, the distinctiveness of
the features is significant. Considering this, the Early-Fusion and the Late-
Fusion approaches both rely solely on the 2D image features to predict, and
the 3D information is only used to project the 2D keypoints to 3D. Therefore,
we consider this Mid-Fusion approach to combine 2D image features and 3D
information together to predict the keypoints.

As shown in Fig. 2, we build a 3D geometry volume from the two image feature
maps, regularize it with 3D CNNs, then reduce the divergence between the
feature field and the local keypoint heatmaps to predict 3D keypoints.

To build the 3D geometry volume, we divide the interested 3D space into
regular 3D grids of size $(H_{v},W_{v},D_{v})$, and the size of each grid is
$(h_{v},w_{v},d_{v})$. The axes of the grid coordinates are centralized in an
initial guess and parallel to the reference camera coordinates. We create a
many-to-one projection from 3D grid space to the image space by known camera
intrinsic parameters. The grid space is also projected to the query image
space by the relative camera pose between the two views. Then we assign the
features extracted before of the 2D point to its related 3D point. Points
projected outside the image are assigned to initial feature values. At each 3D
point, we simply concatenate the two feature vectors from two inputs to keep
as much information as possible.

Then, to predict 3D keypoints, we regularize this 3D volume with 3D CNNs to
obtain a field representing the distribution of target keypoint locations.
This distribution is learned by reducing the divergence between the 3D feature
field and the local target keypoint heatmaps. The local keypoint heatmaps are
defined as

$Q_{i}^{k}(p|P_{ki}^{m},\theta)\sim\mathcal{N}(\theta\cdot
P_{ki}^{m},\sigma_{i})$ (1)

where $\\{P_{ki}^{m}\\}_{i=1}^{N}$ refers to the model keypoints, $\theta$ is
the target pose, and $\sigma_{i}$ is the hyperparameters. This distribution
represents the local heatmap we expect to be highlighted as target keypoints
locations. And the feature field is defined as

$Q_{i}^{v}(p|V(\cdot))=\sum_{j\in\Omega}w_{j}V([p_{j}])$ (2)

where $\\{w_{j}\\}_{j\in\Omega}$ are the trilinear interpolation coefficients,
$[p_{j}]$ are the 8 neighbour grids of the conditioned position $p$, and
$V(\cdot)$ is the value operation in the feature field. This distribution
describes how likely a position in the field is to be the keypoints. To
minimal the divergence of these two distributions, we adopt a Kullback-Leibler
divergence loss

$Loss_{KL}=D_{KL}(Q^{v}\|Q^{k})=-\sum_{i=1}^{N}Q_{i}^{v}log(\frac{Q_{i}^{k}}{Q_{i}^{v}})$
(3)

Last, we predict the 3D keypoint by calculating the expectation of the
distribution in the 3D volume. Besides, we employ a coarse-to-fine strategy to
further enhance the prediction accuracy.

###  III-C Soft RANSAC Solver

Given the predicted 3D keypoints $\\{\hat{P}_{ki}\\}_{i=1}^{N}$ and the model
keypoints $\\{P_{ki}^{m}\\}_{i=1}^{N}$, we then solve the object pose by
minimizing the distance between predicted points and model points. The
optimization problem is

$\hat{\theta}=\arg\min_{\theta}\|\theta\cdot P_{ki}^{m}-\hat{P}_{ki}\|_{2}$
(4)

where $\theta$ represents the 6D object pose.

Though this optimization problem can be solved by SVD in closed form, it is
easy to be disturbed by outliers. Therefore, we propose a soft RANSAC solver
to softly count the inliers to solve a more robust pose. Taking the predicted
keypoints $\\{\hat{P}_{ki}\\}_{i=1}^{N}$, we calculate all possible poses with
every 3 points by SVD, which brings us a set of pose hypothesis
${\\{\theta_{k}\\}_{k=1}^{{C_{N}^{3}}}}$. For each hypothesis, we evaluate the
distance for every predicted and model keypoints under the hypothesis pose

$d_{k,i}=\|\theta_{k}\cdot P_{ki}^{m}-\hat{P}_{ki}\|_{2}$ (5)

Then we sum up the soft inlier numbers for each hypothesis to evaluate the
pose, and softly aggregate all the hypothesis into a final pose with a
regularized scores

$\hat{\theta}=\sum_{k}^{K}\theta_{k}\cdot\sum_{i}^{N}sigmoid(\gamma_{1}(-d_{k,i}+\gamma_{2}))$
(6)

This solver is shared among all the 3 proposed approaches.

For the differentiability in the entire process in Mid-Fusion approach, the
pose error could be end-to-end back-propagated to train the 3D networks. The
loss to evaluate the predicted pose is

$Loss_{pose}=\|\hat{t}-t\|_{2}+\alpha\|\hat{R}R^{T}-I\|_{F}$ (7)

In total, the Mid-Fusion approach is trained with joint losses

$Loss=\sum_{j}\beta_{1}Loss_{pose_{j}}+\beta_{2}Loss_{kpt_{j}}+\beta_{3}Loss_{KL_{j}}$
(8)

where $j=\\{0,1\\}$ refers to the coarse and fine levels, and
$\\{\beta_{1},\beta{2},\beta{3}\\}$ are hyperparameters.

Figure 3: Example of query images under different pairing gaps. The reference
image and query image are paired up by a fixed gap, in which $gap=1$ means
pairing two images with a gap of one frame, and so on. Generally larger gaps
lead to farther views.

##  IV Experiments

In this section, we evaluate our method by experiments to answer two
questions: (1) Which of the proposed fusing approaches can recover the most
precise and useful 3D information from 2 RGB images for object pose
estimation? (2) Combined with 2D image features and 3D information, can our
method enhance the pose estimation accuracy? To answer (1), we assess and
compare the precision of the predicted 3D keypoints of the proposed
approaches. To answer (2), we compare the estimation results between our
method and other state-of-the-art methods on public benchmark LineMOD
dataset[26] and Occlusion LineMOD dataset[39].

TABLE I: evaluation on 3d keypoints prediction. ($3D\mbox{-}Keypoint\mbox{-}Distance/m$) | Early-Fusion | Mid-Fusion | Late-Fusion | REDE[6]
---|---|---|---|---
$gap=1$ | 0.044 | 0.024 | 0.079 |
$gap=5$ | 0.031 | 0.009 | 0.014 |
$gap=10$ | 0.031 | 0.007 | 0.011 |
mean | 0.035 | 0.013 | 0.035 | 0.024

(a) ape

(b) can

(c) cat

(d) glue

(e) eggbox

(f) iron

Figure 4: 3D Keypoint Recall ($<0.03m$) with respect to relative camera
distances. Each point represents a ratio of the numbers of keypoints with an
error less than $0.03m$ to the number of all keypoints under certain relative
camera distances. The relative camera distances are computed as the distances
between two focal points between the two cameras. The Mid-Fusion approach
achieves better results in most of the camera distances, especially in small
parallax.

###  IV-A Dataset

LineMOD dataset[26] is a widely used benchmark for object 6D pose estimation
tasks. It contains 13 objects with varieties of textures and structures. For a
fair comparison, we follow [18] [6] to split it into train, valid and test
data. For each object, about 180 images are for training and more than 1000
images are for testing. Also, we follow [18] to further create 10000 images by
”Cut and Paste” strategy to train all the objects.

Occlusion LineMOD[39] dataset shares the same objects with LineMOD, but
additionally annotates a subset of images with serious occlusion and clutters.
It contains 8 different objects, and 1214 images. All images are used for
testing with models trained on LineMOD dataset.

###  IV-B Metrics

We follow convention to use ADD [25] and ADD-S [1] as evaluation metrics.
Given an object model of $M$ points, ADD metric evaluates the average distance
between the model points transformed with predicted and ground truth pose
respectively

$ADD=\frac{1}{M}\sum_{i=1}^{M}\|\theta p_{i}-\hat{\theta}p_{i}\|_{2}$ (9)

While ADD-S metric calculates the average distance between the closest points,
which is used for evaluating symmetric object

$ADD\mbox{-}S=\frac{1}{M}\sum_{i=1}^{M}\min_{j\in M}\|\theta
p_{j}-\hat{\theta}p_{i}\|_{2}$ (10)

An estimation is regarded successful if the ADD(-S) is less than $10\%$ of the
object’s diameter.

Besides, to evaluate the quality of restored 3D information, we apply 3D
keypoint Euclidean distance between predicted 3D keypoints and ground truth 3D
keypoints.

Figure 5: Visualization of keypoint prediction results from different fusion
approach. From left to right: input image pairs with ground truth 3D keypoints
(green), predicted keypoints from Early-Fusion (yellow) and the predicted
depth map, predicted keypoints from Late-Fusion (blue), predicted keypoints
from Mid-Fusion (red) and four slices of the feature volume, and all the
ground truth and predicted 3D keypoints in two different views with respective
colors.  TABLE II: Performance comparison on LineMOD (ADD(-S)$<0.1d$).

| RGB | RGBD
---|---|---
| Pix2Pose | PVNet | DPOD | CDPN | HybridPose | SO-Pose | GDR-Net | Ours | DenseFusion | REDE
| [2] | [18] | [3] | [40] | [19] | [22] | [21] |  | [4] | [6]
ape | 58.1 | 43.6 | 53.3 | 64.4 | 63.1 |  |  | 94.9 | 92.3 | 96.3
benchvise | 91.0 | 99.9 | 95.3 | 97.8 | 99.9 |  |  | 99.9 | 93.2 | 98.9
cam | 60.9 | 86.9 | 90.4 | 91.7 | 90.4 |  |  | 93.7 | 94.4 | 99.8
can | 84.4 | 95.5 | 94.1 | 95.9 | 98.5 |  |  | 99.0 | 93.1 | 99.4
cat | 64.0 | 79.3 | 60.4 | 83.8 | 89.4 |  |  | 98.3 | 96.5 | 99.2
driller | 76.3 | 96.4 | 97.7 | 96.2 | 98.5 |  |  | 99.1 | 87.0 | 99.3
duck | 43.8 | 52.6 | 66.0 | 66.8 | 65.0 |  |  | 96.0 | 92.3 | 96.2
eggbox* | 96.8 | 99.1 | 99.7 | 99.7 | 100.0 |  |  | 99.3 | 99.8 | 100.0
glue* | 79.4 | 95.7 | 93.8 | 99.6 | 98.8 |  |  | 99.0 | 100.0 | 100.0
holepuncher | 74.8 | 81.9 | 65.8 | 85.8 | 89.7 |  |  | 94.5 | 86.9 | 99.2
iron | 83.4 | 98.9 | 99.8 | 97.9 | 100.0 |  |  | 98.8 | 97.0 | 99.9
lamp | 82.0 | 99.3 | 88.1 | 97.9 | 99.5 |  |  | 98.4 | 95.3 | 99.5
phone | 45.0 | 92.4 | 74.2 | 90.1 | 94.9 |  |  | 96.5 | 92.8 | 98.9
mean | 72.4 | 86.3 | 83.0 | 89.9 | 91.3 | 96.0 | 94.1 | 97.5 | 94.3 | 99.0

  * •

*denotes symmetric objects.

###  IV-C Implementation Details

In implementation, we follow [18] to select 9 keypoints for every object, and
perform the same data augmentation. Every two adjacent frames are paired up
for training, and every two frames with a certain gap are paired up for
testing. The relative camera poses are obtained from the dataset.

In the Early-Fusion approach, we finetune the pretrained models provided by
[38] on our datasets with an initial learning rate of $0.0001$ and a stepwise
decay strategy. The depth range is set according to the depth range of the
dataset, and the depth resolution is set to $3.5mm$.

In the Mid-Fusion approach, the keypoint prediction network contains three 3D
convolutional layers all with one 3D BatchNorm layer and 1 ReLU layer
respectively, 1 output 3D convolutional layer, and finally 1 LogSoftmax layer.
The 3D geometric volume is set in range
$[-0.3,0.3]\times[-0.3,0.3]\times[-0.3,0.3]$(meters) with grid size of $0.01m$
in the coarse level. While the range of the fine volume is dependent on the
diameter of each object and its grid size is set to $0.005m$.

We run all our experiments on a machine equipped with an Intel(R) Xeon(R)
Silver 4216 CPU at 2.10GHz, and an NVIDIA GeForce RTX 3090 GPU.

###  IV-D Evaluation on 3D Keypoint Precision

To validate the ability of our method to restore 3D information from the input
RGB images, we train all the networks in the 3 proposed approaches on LineMOD
dataset, then analyze and compare the quality of the predicted 3D object
keypoints in terms of average 3D Keypoint Distance metric. During testing, we
adopt a fixed gapping strategy to pair up images as inputs for our proposed
method. Since the dataset is collected with handheld cameras, the relative
camera poses between each frame are diverse even with fixed gaps. But
generally a larger gap still leads to two farther views, as the example in
Fig. 3 shows. Thus, to analyze the effect of pairing gaps, the evaluation is
conducted under different gaps ($\\{1,5,10\\}$), in which $gap=1$ means
pairing two frames with a gap of one frame, and so on.

As can be seen in Table. I, with the increase of gap frames, the 3D keypoints
prediction precision of the proposed methods are all enhanced. Among all the 3
approaches, the Mid-Fusion approach achieves the best prediction results, with
an average keypoint error of $0.007m$ in $gap=10$. The Late-Fusion approach is
the second-best in $gap=5$ and $gap=10$ occasions. While the Early-Fusion
approach obtains the worst results in all pairing settings. We also compare
the keypoints predicted by our method and those predicted by a SOTA RGBD-based
method [6]. Table. I shows that, compared to [6] ($0.024m$), the Early-Fusion
approach achieves slightly worse results, and the Late-Fusion approach gets
better results with $gap=5$ ($0.014m$) and $gap=10$ ($0.011m$). What’s more,
the Mid-Fusion approach achieves the same results even under $gap=1$, and
better results with larger gaps.

To further verify the performance of the 3 proposed approaches under different
parallax, we draw an accuracy ratio curve under different relative camera
distances in 4 different objects (ape, can, cat, and glue), as shown in Fig.
4. Each point in the figure represents a ratio of the numbers of keypoints
with an error less than $0.03m$ to the number of all keypoints under certain
relative camera distances. The relative camera distances are computed as the
distances between two focal points between the two cameras. The Mid-Fusion
approach achieves the best results in most of the camera distances, especially
in small ones. It exhibits the robustness and superiority of the Mid-Fusion
approach in different situations.

Fig. 5 shows a visualized result comparison among the 3 proposed approaches.
The 3D predicted keypoints are projected to the reference image by camera
parameters, as well as shown in 2 different views in 3D space. Also, the depth
map predicted by the Early-Fusion approach and the feature slices learned by
the Mid-Fusion approach are exhibited. As can be seen, though some predictions
show good performance when projected to 2D image plane, their errors are more
clear when observed in the 3D space, such as the top keypoint predicted by the
Early-Fusion approach.

To this end, we argue that among all the 3 proposed methods, the Mid-Fusion
approach achieves the best prediction results under different situations,
which shows its ability to restore the most useful 3D information for object
pose estimation from two input RGB images. Therefore, we adopt the Mid-Fusion
approach as the keypoint predictor to conduct experiments compared to other
state-of-the-art methods.

TABLE III: Performance comparison on Occlusion LineMOD (ADD(-S)$<0.1d$).

| RGB | RGBD
---|---|---
| PoseCNN | PVNet | DPOD | HybridPose | SO-Pose | GDR-Net | Ours | PVN3D | REDE | FFB6D
| [1] | [18] | [3] | [19] | [22] | [21] |  | [9] | [6] | [24]
ape | 9.6 | 15.8 |  | 20.9 | 48.4 | 46.8 | 49.2 | 33.9 | 55.9 | 47.2
can | 45.2 | 63.3 |  | 75.3 | 85.8 | 90.8 | 89.3 | 88.6 | 87.9 | 85.2
cat | 0.93 | 16.7 |  | 24.9 | 32.7 | 40.5 | 37.7 | 39.1 | 37.5 | 45.7
driller | 41.4 | 65.7 |  | 70.2 | 77.4 | 82.6 | 90.1 | 78.4 | 75.3 | 81.4
duck | 19.6 | 25.2 |  | 27.9 | 48.9 | 46.9 | 57.8 | 41.9 | 48.5 | 53.9
eggbox* | 22.0 | 50.2 |  | 52.4 | 52.4 | 54.2 | 57.3 | 80.9 | 73.3 | 70.2
glue* | 38.5 | 49.6 |  | 53.8 | 78.3 | 75.8 | 71.6 | 68.1 | 74.8 | 60.1
holepuncher | 22.1 | 39.7 |  | 54.2 | 75.3 | 60.1 | 80.0 | 74.7 | 61.9 | 85.9
mean | 24.9 | 40.8 | 47.3 | 47.5 | 62.3 | 62.2 | 66.6 | 63.2 | 64.2 | 66.2

  * •

*denotes symmetric objects.

###  IV-E Evaluation on Object Pose Estimation

To validate the enhancement of our method after combining 2D image features
and learned 3D information, we evaluate the performance in LineMOD dataset and
Occlusion LineMOD dataset and compete with state-of-the-art RGB-based methods
and RGBD-based methods. In this section, all our methods are evaluated with
$gap=10$ pairing strategy.

Table. II displays the performance on LineMOD dataset. Best ADD(-S) recalls
for each object are in bold, excluding RGBD-based methods. Our method excels
all other RGB-based methods with a mean ADD recall of $97.5$, and most of the
methods in 9 object classes. Also, our method performs slightly better than
the RGBD-based method [4] in 10 object classes and in the mean recall metric.
Both indicate the advantage of combining 2D image features and 3D information
together for pose estimation. Nonetheless, though predicts more accurate 3D
keypoints than [6], our method performs worse than [6] in pose estimation. We
consider the reason lies in the pose solver they adopt. In [6], after
predicting sparse 3D keypoints, they deploy a mechanism called minimal solver
bank to use dense real scene points recovered from the depth sensors to softly
aggregate all the possible poses calculated from 3D registration, which is
indeed an exhaustive RANSAC algorithm. The solver is robust enough to remedy
the uncertainty brought by less accurate keypoints. Yet out Mid-Fusion
approach is unable to deploy such an algorithm due to the lack of dense 3D
points, while the Early-Fusion approach restores dense points with certain
errors which will bring more disturbance to the solver.

Pose estimation under serious occlusion situations is a difficult task,
especially for RGB-based methods. Table. III shows the performance on
Occlusion LineMOD dataset. Best ADD(-S) recalls for each object is in bold,
excluding RGBD-based methods. In this dataset with more occlusion and clutter,
the mean ADD(-S) recall of our method beats all other RGB-based and RGBD-based
methods. If excluding RGBD-based, our method beats RGB-based methods in 5
object classes. Note that we also beat [6] in this dataset. Under the heavily
occluded situation, our method not only take advantage of both 2D image
features and learned 3D information, but is also benefited from extra
observation views. Hence, we have more opportunities to acquire useful
information for keypoint prediction and pose estimation.

Some visualization results are shown in Fig. 6. We project the object CAD
model transformed by the estimated pose and draw the points on the reference
view. All the images are cropped for better visualization. Compared with RGB-
based method PVNet[18], our method achieves more accurate results in
textureless surfaces, bizarre viewing angle, and some normal situations. Also,
our method beats both [18] and RGBD-based method[6] in occluded occasions.

Figure 6: Visualization Results Samples. Some visualization results on LindMOD
and Occlusion LineMOD dataset. The pose results are shown by projecting the
model points via the predicted pose from each method. We also show the contour
of the model projection mask. All images are cropped for better visualization.
The cases with few dislocated projected points represent a predicted pose with
large errors. The last row shows our result with Mid-Fusion approach. Compared
with RGB-based method PVNet[18], our method achieves more accurate results in
textureless surfaces (e), bizarre viewing angle (c), and some normal
situations (a)(b)(d). Also, our methods beats [18] and RGBD-based method[6] in
occluded occasions (f)(g).

##  V CONCLUSIONS

In this paper, we propose a framework to learn implicit 3D information from 2
input RGB images for 6D object pose estimation. In this framework, we conduct
an investigation on 3 different 3D information learning approaches. Based on
the experiments, we reveal a Mid-Fusion mechanism that fuses 2D image features
geometrically to 3D space produces the most precise keypoints. We further show
that with this Mid-Fusion mechanism, our framework is capable of enhancing the
pose estimation performance of RGB-based methods, and achieving comparable
results with RGBD-based methods. However, our performance in less occluded
environments are still inferior to RGBD-based methods, mostly because of the
lack of dense reliable scene points. Thus, we plan to tackle the challenge by
employing two or more views of RGBD inputs to solve the pose in the future.

## References

  * [1] Y. Xiang, T. Schmidt, V. Narayanan, and D. Fox, “Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes,” arXiv preprint arXiv:1711.00199, 2017.
  * [2] K. Park, T. Patten, and M. Vincze, “Pix2pose: Pixel-wise coordinate regression of objects for 6d pose estimation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7668–7677, 2019.
  * [3] S. Zakharov, I. Shugurov, and S. Ilic, “Dpod: 6d pose object detector and refiner,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1941–1950, 2019.
  * [4] C. Wang, D. Xu, Y. Zhu, R. Martín-Martín, C. Lu, L. Fei-Fei, and S. Savarese, “Densefusion: 6d object pose estimation by iterative dense fusion,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3343–3352, 2019.
  * [5] K. Wada, E. Sucar, S. James, D. Lenton, and A. J. Davison, “Morefusion: Multi-object reasoning for 6d pose estimation from volumetric fusion,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14540–14549, 2020.
  * [6] W. Hua, Z. Zhou, J. Wu, H. Huang, Y. Wang, and R. Xiong, “Rede: End-to-end object 6d pose robust estimation using differentiable outliers elimination,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 2886–2893, 2021\.
  * [7] I. Shugurov, S. Zakharov, and S. Ilic, “Dpodv2: Dense correspondence-based 6 dof pose estimation,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
  * [8] Y. Shi, J. Huang, X. Xu, Y. Zhang, and K. Xu, “Stablepose: Learning 6d object poses from geometrically stable patches,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15222–15231, 2021.
  * [9] Y. He, W. Sun, H. Huang, J. Liu, H. Fan, and J. Sun, “Pvn3d: A deep point-wise 3d keypoints voting network for 6dof pose estimation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11632–11641, 2020.
  * [10] J.-R. Chang and Y.-S. Chen, “Pyramid stereo matching network,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5410–5418, 2018.
  * [11] P. Li, X. Chen, and S. Shen, “Stereo r-cnn based 3d object detection for autonomous driving,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7644–7652, 2019.
  * [12] Y. Yao, Z. Luo, S. Li, T. Fang, and L. Quan, “Mvsnet: Depth inference for unstructured multi-view stereo,” in Proceedings of the European Conference on Computer Vision (ECCV), pp. 767–783, 2018.
  * [13] M. Rad and V. Lepetit, “Bb8: A scalable, accurate, robust to partial occlusion method for predicting the 3d poses of challenging objects without using depth,” in Proceedings of the IEEE International Conference on Computer Vision, pp. 3828–3836, 2017.
  * [14] T. Hodan, D. Barath, and J. Matas, “Epos: Estimating 6d pose of objects with symmetries,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11703–11712, 2020.
  * [15] F. Manhardt, D. M. Arroyo, C. Rupprecht, B. Busam, T. Birdal, N. Navab, and F. Tombari, “Explaining the ambiguity of object detection and 6d pose from visual data,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 6841–6850, 2019.
  * [16] B. Tekin, S. N. Sinha, and P. Fua, “Real-time seamless single shot 6d object pose prediction,” in Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 292–301, 2018.
  * [17] Q. Luo, H. Ma, L. Tang, Y. Wang, and R. Xiong, “3d-ssd: Learning hierarchical features from rgb-d images for amodal 3d object detection,” Neurocomputing, vol. 378, pp. 364–374, 2020.
  * [18] S. Peng, Y. Liu, Q. Huang, X. Zhou, and H. Bao, “Pvnet: Pixel-wise voting network for 6dof pose estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4561–4570, 2019.
  * [19] C. Song, J. Song, and Q. Huang, “Hybridpose: 6d object pose estimation under hybrid representations,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 431–440, 2020.
  * [20] Y. Hu, P. Fua, W. Wang, and M. Salzmann, “Single-stage 6d object pose estimation,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2930–2939, 2020.
  * [21] G. Wang, F. Manhardt, F. Tombari, and X. Ji, “Gdr-net: Geometry-guided direct regression network for monocular 6d object pose estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 16611–16621, 2021.
  * [22] Y. Di, F. Manhardt, G. Wang, X. Ji, N. Navab, and F. Tombari, “So-pose: Exploiting self-occlusion for direct 6d pose estimation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 12396–12405, 2021.
  * [23] L. Saadi, B. Besbes, S. Kramm, and A. Bensrhair, “Optimizing rgb-d fusion for accurate 6dof pose estimation,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 2413–2420, 2021.
  * [24] Y. He, H. Huang, H. Fan, Q. Chen, and J. Sun, “Ffb6d: A full flow bidirectional fusion network for 6d pose estimation,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 3003–3013, 2021.
  * [25] S. Hinterstoisser, V. Lepetit, S. Ilic, S. Holzer, G. Bradski, K. Konolige, and N. Navab, “Model based training, detection and pose estimation of texture-less 3d objects in heavily cluttered scenes,” in Asian conference on computer vision, pp. 548–562, Springer, 2012.
  * [26] S. Hinterstoisser, S. Holzer, C. Cagniart, S. Ilic, K. Konolige, N. Navab, and V. Lepetit, “Multimodal templates for real-time detection of texture-less objects in heavily cluttered scenes,” in 2011 international conference on computer vision, pp. 858–865, IEEE, 2011.
  * [27] M. Gonzalez, A. Kacete, A. Murienne, and E. Marchand, “L6dnet: Light 6 dof network for robust and precise object pose estimation with small datasets,” IEEE Robotics and Automation Letters, vol. 6, no. 2, pp. 2914–2921, 2021\.
  * [28] A. Collet and S. S. Srinivasa, “Efficient multi-view object recognition and full pose estimation,” in 2010 IEEE International Conference on Robotics and Automation, pp. 2050–2055, IEEE, 2010.
  * [29] A. Collet, M. Martinez, and S. S. Srinivasa, “The moped framework: Object recognition and pose estimation for manipulation,” The international journal of robotics research, vol. 30, no. 10, pp. 1284–1306, 2011.
  * [30] Y. Labbé, J. Carpentier, M. Aubry, and J. Sivic, “Cosypose: Consistent multi-view multi-object 6d pose estimation,” in European Conference on Computer Vision, pp. 574–591, Springer, 2020.
  * [31] J. Fu, Q. Huang, K. Doherty, Y. Wang, and J. J. Leonard, “A multi-hypothesis approach to pose ambiguity in object-based slam,” in 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 7639–7646, IEEE, 2021.
  * [32] X. Liu, R. Jonschkowski, A. Angelova, and K. Konolige, “Keypose: Multi-view 3d labeling and keypoint estimation for transparent objects,” in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11602–11610, 2020.
  * [33] N. D. F. Campbell, G. Vogiatzis, C. Hernández, and R. Cipolla, “Using multiple hypotheses to improve depth-maps for multi-view stereo,” in European Conference on Computer Vision, 2008.
  * [34] Engin, Tola, Christoph, Strecha, Pascal, and Fua, “Efficient large-scale multi-view stereo for ultra high-resolution image sets,” Machine Vision and Applications, vol. 23, no. 5, pp. 903–920, 2011.
  * [35] Y. Yao, Z. Luo, S. Li, T. Shen, T. Fang, and L. Quan, “Recurrent mvsnet for high-resolution multi-view stereo depth inference,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 5525–5534, 2019.
  * [36] K. Luo, T. Guan, L. Ju, H. Huang, and Y. Luo, “P-mvsnet: Learning patch-wise matching confidence aggregation for multi-view stereo,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10452–10461, 2019.
  * [37] J. Yang, W. Mao, J. M. Alvarez, and M. Liu, “Cost volume pyramid based depth inference for multi-view stereo,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 4877–4886, 2020.
  * [38] X. Gu, Z. Fan, S. Zhu, Z. Dai, and P. Tan, “Cascade cost volume for high-resolution multi-view stereo and stereo matching,” in 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
  * [39] E. Brachmann, A. Krull, F. Michel, S. Gumhold, J. Shotton, and C. Rother, “Learning 6d object pose estimation using 3d object coordinates,” in European conference on computer vision, pp. 536–551, Springer, 2014.
  * [40] Z. Li, G. Wang, and X. Ji, “Cdpn: Coordinates-based disentangled pose network for real-time rgb-based 6-dof object pose estimation,” in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 7678–7687, 2019\.


11institutetext: Norwegian University of Science and Technology, Norway
22institutetext: Clinic of Cardiology, St. Olavs Hospital, Norway
33institutetext: SINTEF Digital, Norway
33email<EMAIL_ADDRESS>
# EchoTracker: Advancing Myocardial Point Tracking in Echocardiography

Md Abulkalam Azad 11 0009-0006-7177-4961 Artem Chernyshov 11 John Nyberg 11
Ingrid Tveten 1133 Lasse Lovstakken 11 Håvard Dalen 1122 Bjørnar Grenne 1122
Andreas Østvik 1133 0000-0003-3895-2683

###### Abstract

Tissue tracking in echocardiography is challenging due to the complex cardiac
motion and the inherent nature of ultrasound acquisitions. Although optical
flow methods are considered state-of-the-art (SOTA), they struggle with long-
range tracking, noise occlusions, and drift throughout the cardiac cycle.
Recently, novel learning-based point tracking techniques have been introduced
to tackle some of these issues. In this paper, we build upon these techniques
and introduce EchoTracker, a two-fold coarse-to-fine model that facilitates
the tracking of queried points on a tissue surface across ultrasound image
sequences. The architecture contains a preliminary coarse initialization of
the trajectories, followed by reinforcement iterations based on fine-grained
appearance changes. It is efficient, light, and can run on mid-range GPUs.
Experiments demonstrate that the model outperforms SOTA methods, with an
average position accuracy of 67% and a median trajectory error of 2.86 pixels.
Furthermore, we show a relative improvement of 25% when using our model to
calculate the global longitudinal strain (GLS) in a clinical test-retest
dataset compared to other methods. This implies that learning-based point
tracking can potentially improve performance and yield a higher diagnostic and
prognostic value for clinical measurements than current techniques. Our source
code is available at: https://github.com/riponazad/echotracker/.

###### Keywords:

Deep learning Point-tracking Motion estimation Ultrasound imaging Strain
measurements.

##  1 Introduction

Myocardial imaging in echocardiography uses ultrasound (US) image analysis to
assess and quantify the morphology and function of the cardiac muscle. These
methods can be used to uncover reduced pumping efficiency, detect muscle
irregularities, and diagnose various heart conditions, facilitating early
identification of cardiac dysfunction. Myocardial strain, a measure of
deformation, has shown high sensitivity with superior diagnostic and
prognostic value compared to common anatomical measurements, such as ejection
fraction [6]. Motion estimation is vital for precise strain, but is hampered
by variability in image acquisition, measurements, and inherent limitations of
US. Currently, motion estimation and strain imaging rely on speckle tracking
using block- and feature-matching approaches [18]. Recent advances in
learning-based techniques, such as optical flow-based architectures like
FlowNet 2.0 [9] and PWC-Net [17], have inspired researchers to use and adapt
them for US imaging and enhanced strain calculations [5, 11, 12]. However,
optical flow estimates dense displacement fields between consecutive frames
without considering long-range temporal context. This limitation makes
tracking susceptible to noise, out-of-plane motion, and decorrelation of
speckle patterns. Therefore, it hinders optimal tracking across multiple
frames and causes drift within the cardiac cycle.

In this study, we propose EchoTracker, a novel method for tissue tracking in
echocardiography. It is designed based on the latest point tracking approaches
for general applications [2, 3, 7, 10, 20]. To the best of our knowledge, this
is the first work that uses such an approach in the field of medical imaging.
It addresses the limitations of dense optical flow by leveraging the temporal
context of longer sequences. Herein, we build upon this and design our model
architecture tailored for US data, enabling efficient learning with enhanced
performance and a lightweight structure. We assess the tracking performance of
our model compared to related approaches. Finally, we utilize our model for
GLS calculations and compare the clinical performance with other SOTA
solutions.

##  2 Tracking Any Point (TAP)

Tracking of arbitrary points in video sequences is a new research area in deep
learning, evolving to mitigate the limitations of optical flow-based tracking.
It also possesses the capability to tackle deformations when queried points
are selected on the surface of non-rigid objects. Doersch et al. were the
first to formalize the TAP problem, provided a benchmark comprising real and
synthetic data, and proposed a simple baseline, TAP-Net, for evaluation [2].
The TAP algorithm takes a video and query points from any frame $t$ as input
and predicts locations $(x_{t},y_{t})$ and binary occlusions $(o_{t})$ as
output for each point in all other frames. Doersch et al. also mentioned that
the output location $(x_{t},y_{t})$ is meaningless when occluded $(o_{t}=1)$.
However, this poses a contradiction to our problem, as we still require the
location to compute GLS even if the point is out of the plane. Our
modifications to the formal definition of TAP align with Persistent
Independent Particles (PIPs) [7]. It aimed to track particles as pixels across
long-range video sequences, inspired by the classic Particle Video approach
[15].

However, PIPs has notable drawbacks, as it operates on long videos using
temporal sliding, which can lead to drift when points are occluded for more
than a single window. Expanding the temporal window makes the model slow and
unsuitable for parallel computation. To this end, Doersch et al. later
introduced TAPIR [3], leveraging both TAP-Net and PIPs. Also in parallel,
Zheng et al. proposed PIPs++ by modifying PIPs with expanded temporal
receptive fields and a multi-template update mechanism [20].

Although these approaches differ, a common factor is that they all track
points independently, not sharing information between trajectories. This
limitation may hinder the tracking of deformable objects like myocardial
tissue and lead to drift for points outside the US probe view. This issue has
been addressed by CoTracker [10] and OmniMotion [19]. CoTracker iteratively
refines trajectories using a transformer architecture in a sliding window
manner after a naive initialization. Consequently, it shares the same
disadvantage as PIPs for long-range tracking and also exponentially increases
the time complexity in case of longer temporal windows. On the other hand,
OmniMotion provides a test-time optimization approach based on the canonical
3D volume of the input video.

##  3 Methods

The overall goal of our learning-based model is to track tissue points through
the cardiac cycle while dealing with complex motion, deformation, and noise.
As depicted in Fig. 1, the input comprises a sequence of US images
$U=\\{u_{s}\in\bbbr^{H\times W}\\}$ for $s=0,1,...,S$, and a set of query
points on the first frame $p_{0}=\\{(x^{n}_{0},y^{n}_{0})\\}$ for
$n=0,1,...,N$. Here, $H$ is the height of the image, $W$ is the width, while
$x$ and $y$ are the horizontal and vertical pixel locations of a given point.
The proposed solution outputs the locations of the queried points in all other
frames, $P=\\{p_{s}\in(x_{s}^{n},y_{s}^{n})\\}$. Hence, the problem can be
summarized as EchoTracker$(u_{s},p_{0})=p_{s}\in(x_{s}^{n},y_{s}^{n})$.

Figure 1: An illustration of tracking queried points (highlighted in red) from
the first frame throughout one heart cycle.

###  3.1 EchoTracker

Our proposed model, named EchoTracker, includes two stages as shown in Fig. 2,
initialization and iterative reinforcement. The approach follows a two-fold
coarse-to-fine strategy inspired by TAPIR [3]. In the initial stage,
trajectories are initialized based on the coarse resolution of the feature
maps using a coarse network. Subsequently, in the second stage, the
trajectories are iteratively refined using fine-grained feature maps by a fine
network, thus constituting a two-fold coarse-to-fine approach. This technique
not only speeds up computation but also prevents the loss of important
information due to downsampling. Although the networks in both stages estimate
trajectories independently, they exploit point locations from the first frame
to maintain spatial correlation and estimate coherent trajectories.
Additionally, frame flow (i.e. $u_{s}-u_{s-1}$), representing the difference
between consecutive frames, is naively passed to the model to make it aware of
global appearance changes. The model can run on US sequences of any length and
with any number of query points, depending on available memory.

Figure 2: EchoTracker is a two-fold coarse-to-fine model. Initially, it
estimates coarse trajectories (yellow points) based on the cost volume for the
given query points (red). It then imposes iterative reinforcement to obtain
the fine trajectories (green points).

####  3.1.1 Initialization.

The input contains $S$ ultrasound images and $N$ number of query points, as
highlighted by red dots in Fig. 2. We utilize a pruned 2D residual
convolutional network (basic encoder) [8] to generate coarse feature maps
$F_{s}\in\bbbr^{d\times\frac{H}{k}\times\frac{W}{k}}$ for each frame with
$k=8$ and $d=64$. The pruning is motivated by a reduction in computational
costs and the limited variability of data representation (e.g. grayscale,
cyclic, velocity) present in echocardiography. Given a query point in the
first frame $p_{0}^{n}$, we extract a feature vector
$f_{p_{0}^{n}}=\texttt{sample}(F_{0},p_{0}^{n})$, which captures the
appearance of the point, through a bilinear sampling of the feature maps
$F_{0}$ at that specific point location. Following that, we compute the cost
volume $C_{s}^{n}=f_{p_{0}^{n}}\cdot\texttt{pyramid}(F_{s})$ by correlating
$f_{p_{0}^{n}}$ with features across the entire video, using multi-scale
feature pyramids with $L=4$ levels and kernel size $r=3$. Finally, the cost
volume is fed to a coarse 1D ResNet, similar to PIPs++ [20], which convolves
across time to estimate the trajectory $P_{s}^{n}$ for $s=0,1,...,S$. This is
highlighted in yellow in Fig. 2. We choose a 1D ConvNet here rather than a 2D
to prioritize temporal information, assuming that bilinear sampling already
perceives the required spatial information present in the US images.

####  3.1.2 Iterative Reinforcement.

Taking inspiration from PIPs [7] and more recent point tracking methods, we
also refine the initial coarse trajectories through an iterative reinforcement
process illustrated in Fig. 2. We hypothesize that our strong initialization
can substantially decrease the number of iterations necessary to converge to a
refined trajectory, and thus keep it at $I=4$ for both training and
evaluation. After initialization, we use the same basic encoder with a reduced
downsampling factor $k=2$ to produce fine feature maps
$F_{s}\in\bbbr^{d\times\frac{H}{2}\times\frac{W}{2}}$. Subsequently, unlike
the initialization, we compute the cost volume
$C_{s}^{n}=f_{p_{s}^{n}}\cdot\texttt{multicrop-pyramid}(F_{s})$ for the query
point by correlating the feature vector $f_{p_{s}^{n}}$ from the current frame
$s$ with multi-scale crops of pyramid features on the fine feature maps
surrounding the point location. Inspired by the feature extraction within a
fixed temporal span of the current frame [1] and multi-template update [20],
we track changes in the point appearance by acquiring additional cost volumes
(i.e., $C_{s-2}^{n},C_{s-4}^{n}$) at a fixed temporal span from the current
frame and always for the first $C_{0}^{n}$. Afterwards, we concatenate these
cost volumes and pass them through a linear layer to obtain the score maps for
the current frame. Similarly, we obtain score maps for all frames and
concatenate them before passing to the next layer to generate updates $\Delta
p_{s}^{n}$ for the trajectory. This layer contains a fine 1D ResNet, which is
a deeper and more weighted version of the coarse ResNet utilized in the
initialization. Finally, updates are applied to the points to obtain the
refined trajectory $p_{s,i}^{n}=p_{s,i-1}^{n}+\Delta p_{s,i-1}^{n}$, and the
iterative reinforcement ensures the most fine-grained estimated trajectory. We
supervise the model by considering all iterations in an end-to-end manner
using the same loss function as in [20].

##  4 Experiments and Results

###  4.1 Datasets

Learning-based methods are usually trained using synthetic data and tested on
real data annotated by humans [5]. Unfortunately, the availability of high-
quality synthetic point tracking data for echocardiography that can adequately
replicate real-world data remains scarce. In this work, we chose to rely on
human-annotated data, similar to real-world data annotations in TAP benchmarks
[2]. Our trajectories are generated in a semi-supervised fashion using a
traditional tracking algorithm and undergo tuning for optimal tracking by
clinical experts. In addition, experts perform quality assurance by rejecting
points that do not follow the tissue properly, and we discard those from
training and evaluation. We retrieve four splits, as illustrated in Table 1,
focusing on tracking the left ventricle myocardium from three acoustic views,
namely the apical four-chamber, two-chamber and long-axis. All data were
collected using GE Vivid E95 scanners, and received approval for research use
from the regional ethics committee. DS-A is a test-retest dataset, meaning
that the same patient is being scanned twice in immediate succession by two
different operators. Therefore, we expect the patient to be in the same
physical condition for both exams, serving as a reference for the
reproducibility of the method. This dataset is independent and used
exclusively for testing. On the other hand, DS-B, DS-C, and DS-D are subsets
of the same dataset but analyzed by three different observers through random
selection. We train the model using these datasets and evaluate its
performance on the DS-A dataset.

Table 1: Ultrasound point tracking datasets and selected characteristics. The number of points and frames, as well as the height and width of the images, are given as average with range (min, max) in parenthesis. Dataset | Patients | Videos | Points | Frames | Height | Width
---|---|---|---|---|---|---
DS-A | 40 | 210 | 73 (57-91) | 93 (57-150) | 593 (583-640) | 571 (496-650)
DS-B | 913 | 2731 | 81 (49-115) | 85 (44-151) | 592 (583-640) | 630 (453-844)
DS-C | 615 | 1837 | 80 (49-111) | 84 (44-150) | 592 (583-640) | 631 (496-778)
DS-D | 643 | 1922 | 66 (45-111) | 84 (46-151) | 592 (421-640) | 634 (392-856)

###  4.2 Implementation details

Similar to Zheng et al. [20], we use a learning rate of $5\cdot 10^{-4}$, a
one-cycle scheduler [16], and the AdamW optimizer. Images are resized to
$256\times 256$ for both training and evaluation by default to limit the GPU
memory footprint. Initially, we train our model on DS-B for 22 epochs using a
sequence length of $S=36$. We fine-tune the model on DS-C for 50 epochs with
$S=68$, and then on DS-D for a single epoch with $S=68$. Throughout the
training process, we consistently use all available points with a batch size
of 1. The training time on a single GPU, completing 50 epochs on the DS-B
dataset typically takes over one week. We implemented our framework in PyTorch
and used an NVIDIA GeForce RTX 3090 (24GB) GPU for both training and
evaluation of models.

###  4.3 Evaluation

####  4.3.1 Evaluation Metrics.

Following SOTA point tracking literature, we report average position accuracy
($<\delta_{avg}^{x}$) as proposed in TAP-Vid [2] for the technical
performance. Position accuracy ($<\delta^{x}$), measures the proportion (%) of
all the query points that fall within the threshold distance of pixels (e.g.,
$x=1,2$) from their ground truth across the entire video and
$<\delta_{avg}^{x}$ averages over five thresholds: 1, 2, 4, 8, and 16 pixels.
Given the sensitivity of tracking in US, we also report $<\delta^{x}$ for
individual threshold and median trajectory error (MTE) [20], which calculates
the distance in pixels between the estimated and ground truth trajectories. To
show the efficiency of our model, we also include the average inference time
(AIT) per video measured in seconds. Furthermore, to evaluate our model in a
clinical setting, we calculate the peak GLS, which is defined as the relative
change of the longitudinal ventricular length from end-diastole to minimum
ventricular length.

####  4.3.2 Technical Performance.

For reference, we conduct an evaluation of current SOTA models directly on our
DS-A dataset without fine-tuning. These results are summarized in Table 2
which shows that PIPs++ and CoTracker stand out. Thus, we use these two models
as baselines for our subsequent experiments.

Table 2: Performance comparison of the state-of-the-art methods on the DS-A
dataset without fine-tuning.

Method | $<\delta^{1}\uparrow$ | $<\delta^{2}\uparrow$ | $<\delta^{4}\uparrow$ | $<\delta^{8}\uparrow$ | $<\delta^{16}\uparrow$ | $<\delta^{x}_{avg}\uparrow$ | MTE$\downarrow$
---|---|---|---|---|---|---|---
PIPs | 5 | 12 | 28 | 54 | 81 | 36 | 10.22
TAP-Net | 5 | 17 | 41 | 70 | 89 | 44 | 6.50
TAPIR | 7 | 21 | 47 | 78 | 95 | 50 | 5.63
PIPs++ | 8 | 22 | 47 | 78 | 96 | 50 | 5.31
CoTracker | 13 | 27 | 52 | 79 | 95 | 53 | 5.28

$\delta$: Position accuracy (%), MTE: Median trajectory error (pixel)

The performance of EchoTracker compared with PIPs++ [20] and CoTracker [10]
fine-tuned with the same training datasets is displayed in Table 3. CoTracker
shows a slight decline in performance after fine-tuning, likely attributed to
suboptimal implementation details or data processing, necessitating in-depth
investigation in future studies. Our model demonstrates superior performance
across all metrics, surpassing other methods by a significant margin.
Moreover, it shows faster inference times compared to these alternatives.

Table 3: Technical performance of EchoTracker on the DS-A test-retest dataset
compared to state-of-the-art methods.

Method | $<\delta^{1}\uparrow$ | $<\delta^{2}\uparrow$ | $<\delta^{4}\uparrow$ | $<\delta^{8}\uparrow$ | $<\delta^{16}\uparrow$ | $<\delta^{x}_{avg}\uparrow$ | MTE$\downarrow$ | AIT$\downarrow$
---|---|---|---|---|---|---|---|---
PIPs++ | 15 | 36 | 70 | 94 | 100 | 63 | 3.28 | 0.42
CoTracker | 8 | 22 | 47 | 78 | 96 | 50 | 5.45 | 1.67
EchoTracker (ours) | 19 | 43 | 76 | 96 | 100 | 67 | 2.86 | 0.24

$\delta$: Position accuracy (%), MTE: Median trajectory error (pixel), AIT:
Average inference time (s)

To investigate if our two-stage architecture can reduce the number of
iterations in the reinforcement stage, we solely train the initialization part
on a limited part of the DS-B dataset. This initialization part yields
$<\delta^{x}_{avg}=48\%$, surpassing the performance of baseline PIPs and TAP-
Net. Thus, it can be expected that the iterative reinforcement stage would
require less effort to refine and smooth the initial trajectory. Our empirical
investigations found that models trained on short sequences struggle when
tracking longer videos. Therefore, a straightforward improvement was to train
on longer sequences of one heart cycle. Furthermore, we experiment with
replacing the fine ResNet with a transformer, following the approach in
CoTracker [10]. Surprisingly, this modification led to a drop in performance.
The reason may be attributed to the fact that transformers typically require
pretraining on large-scale datasets to outperform ConvNets [4]. Exploring this
aspect further could be an intriguing avenue for future research as we witness
CoTracker outperform all the other methods without fine-tuning.

####  4.3.3 Clinical Performance.

In Table 4, we present the GLS results compared to the reference measurements
and in a test-retest situation. We also list results from solutions developed
by others, albeit calculated on their private datasets. Our method, as well as
fine-tuned PIPs++, performs favourably compared to other published work.
However, the methods are tested on different private datasets, so a direct
comparison was not possible.

Table 4: Clinical results for GLS calculations compared to reference
measurements and in a test-retest scenario.

Method | Reference |  | Test-retest
---|---|---|---
$\mu$ | $\sigma\downarrow$ | $MAD\downarrow$ |  | $\mu$ | $\sigma\downarrow$ | $MAD\downarrow$
c-PWC-Net-60A [5] | 1.85 | 2.73 | N/A |  |  |  |
us2ai [11] | 0.68 | 2.52 | 2.0 |  |  |  |
EchoPWCNet [13, 14] | -1.4 | 1.9 | 1.8 |  | 0.0 | 1.9 | 1.6
PIPs++ | -1.21 | 1.95 | 1.76 |  | 0.11 | 1.62 | 1.28
CoTracker | -0.82 | 2.40 | 1.98 |  | -0.11 | 2.47 | 1.96
EchoTracker (ours) | -0.13 | 1.78 | 1.36 |  | -0.13 | 1.55 | 1.21

$\mu$: Bias (%), $\sigma$: Standard deviation (%), $MAD$: Mean absolute
deviation (%)

##  5 Conclusion

We have introduced modern general-purpose point tracking approaches in
echocardiography. Through a comprehensive analysis of several SOTA methods, we
design a two-fold coarse-to-fine architecture and propose a novel model for
tracking myocardial tissue. Our assessment demonstrates that this new approach
not only outperforms SOTA solutions but also enhances the measurements of GLS
in a test-retest scenario. Thus, learning-based point tracking holds the
potential to elevate both the diagnostic and prognostic utility of myocardial
function measurements, marking a notable step forward in the field of
echocardiography.

## References

  * [1] Azad, M.A., Mohammed, A., Waszak, M., Elvesæter, B., Ludvigsen, M.: Multi-label video classification for underwater ship inspection. In: OCEANS 2023 - Limerick. pp. 1–10 (2023)
  * [2] Doersch, C., Gupta, A., Markeeva, L., Recasens, A., Smaira, L., Aytar, Y., Carreira, J., Zisserman, A., Yang, Y.: Tap-vid: A benchmark for tracking any point in a video. Advances in Neural Information Processing Systems (NeurIPS) 35, 13610–13626 (2022)
  * [3] Doersch, C., Yang, Y., Vecerik, M., Gokay, D., Gupta, A., Aytar, Y., Carreira, J., Zisserman, A.: Tapir: Tracking any point with per-frame initialization and temporal refinement. ICCV (2023)
  * [4] Dosovitskiy, A., Beyer, L., Kolesnikov, A., Weissenborn, D., Zhai, X., Unterthiner, T., Dehghani, M., Minderer, M., Heigold, G., Gelly, S., Uszkoreit, J., Houlsby, N.: An image is worth 16x16 words: Transformers for image recognition at scale. In: International Conference on Learning Representations (ICLR) (2021)
  * [5] Evain, E., Sun, Y., Faraz, K., Garcia, D., Saloux, E., Gerber, B.L., De Craene, M., Bernard, O.: Motion estimation by deep learning in 2d echocardiography: synthetic dataset and validation. IEEE transactions on medical imaging 41(8), 1911–1924 (2022)
  * [6] Farsalinos, K.E., Daraban, A.M., Ünlü, S., Thomas, J.D., Badano, L.P., Voigt, J.U.: Head-to-head comparison of global longitudinal strain measurements among nine different vendors: the eacvi/ase inter-vendor comparison study. Journal of the American Society of Echocardiography 28(10), 1171–1181 (2015)
  * [7] Harley, A.W., Fang, Z., Fragkiadaki, K.: Particle video revisited: Tracking through occlusions using point trajectories. In: European Conference on Computer Vision (ECCV). pp. 59–75. Springer (2022)
  * [8] He, K., Zhang, X., Ren, S., Sun, J.: Deep residual learning for image recognition. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). pp. 770–778 (2016)
  * [9] Ilg, E., Mayer, N., Saikia, T., Keuper, M., Dosovitskiy, A., Brox, T.: Flownet 2.0: Evolution of optical flow estimation with deep networks. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). pp. 2462–2470 (2017)
  * [10] Karaev, N., Rocco, I., Graham, B., Neverova, N., Vedaldi, A., Rupprecht, C.: Cotracker: It is better to track together. arXiv:2307.07635 (2023)
  * [11] Myhre, P.L., Hung, C.L., Frost, M.J., Jiang, Z., Ouwerkerk, W., Teramoto, K., Svedlund, S., Saraste, A., Hage, C., Tan, R.S., et al.: External validation of a deep learning algorithm for automated echocardiographic strain measurements. European Heart Journal-Digital Health 5(1), 60–68 (2024)
  * [12] Østvik, A., Salte, I.M., Smistad, E., Nguyen, T.M., Melichova, D., Brunvand, H., Haugaa, K., Edvardsen, T., Grenne, B., Lovstakken, L.: Myocardial function imaging in echocardiography using deep learning. ieee transactions on medical imaging 40(5), 1340–1351 (2021)
  * [13] Salte, I.M., Østvik, A., Olaisen, S.H., Karlsen, S., Dahlslett, T., Smistad, E., Eriksen-Volnes, T.K., Brunvand, H., Haugaa, K.H., Edvardsen, T., et al.: Deep learning for improved precision and reproducibility of left ventricular strain in echocardiography: A test-retest study. Journal of the American Society of Echocardiography (2023)
  * [14] Salte, I.M., Østvik, A., Smistad, E., Melichova, D., Nguyen, T.M., Karlsen, S., Brunvand, H., Haugaa, K.H., Edvardsen, T., Lovstakken, L., Grenne, B.: Artificial intelligence for automatic measurement of left ventricular strain in echocardiography. JACC: Cardiovascular Imaging 14(10), 1918–1928 (2021). https://doi.org/10.1016/j.jcmg.2021.04.018, https://www.jacc.org/doi/abs/10.1016/j.jcmg.2021.04.018
  * [15] Sand, P., Teller, S.: Particle video: Long-range motion estimation using point trajectories. International journal of computer vision 80, 72–91 (2008)
  * [16] Smith, L.N., Topin, N.: Super-convergence: Very fast training of neural networks using large learning rates. In: Artificial intelligence and machine learning for multi-domain operations applications. vol. 11006, pp. 369–386. SPIE (2019)
  * [17] Sun, D., Yang, X., Liu, M.Y., Kautz, J.: Pwc-net: Cnns for optical flow using pyramid, warping, and cost volume. In: Proceedings of the IEEE conference on computer vision and pattern recognition (CVPR). pp. 8934–8943 (2018)
  * [18] Voigt, J.U., Pedrizzetti, G., Lysyansky, P., Marwick, T.H., Houle, H., Baumann, R., Pedri, S., Ito, Y., Abe, Y., Metz, S., et al.: Definitions for a common standard for 2d speckle tracking echocardiography: consensus document of the eacvi/ase/industry task force to standardize deformation imaging. European Heart Journal-Cardiovascular Imaging 16(1), 1–11 (2015)
  * [19] Wang, Q., Chang, Y.Y., Cai, R., Li, Z., Hariharan, B., Holynski, A., Snavely, N.: Tracking everything everywhere all at once. In: International Conference on Computer Vision (ICCV) (2023)
  * [20] Zheng, Y., Harley, A.W., Shen, B., Wetzstein, G., Guibas, L.J.: Pointodyssey: A large-scale synthetic dataset for long-term point tracking. In: Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV). pp. 19855–19865 (2023)