# Multi-Head Deep Metric Learning Using Global and Local Representations

Mohammad K. Ebrahimpour, Gang Qian, and Allison Beach
ObjectVideo Labs, Inc., 8281 Greensboro Dr., Tysons, VA 22102
<EMAIL_ADDRESS>{gqian<EMAIL_ADDRESS>
###### Abstract

Deep Metric Learning (DML) models often require strong local and global
representations, however, effective integration of local and global features
in DML model training is a challenge. DML models are often trained with
specific loss functions, including pairwise-based and proxy-based losses. The
pairwise-based loss functions leverage rich semantic relations among data
points, however, they often suffer from slow convergence during DML model
training. On the other hand, the proxy-based loss functions often lead to
significant speedups in convergence during training, while the rich relations
among data points are often not fully explored by the proxy-based losses. In
this paper, we propose a novel DML approach to address these challenges. The
proposed DML approach makes use of a hybrid loss by integrating the pairwise-
based and the proxy-based loss functions to leverage rich data-to-data
relations as well as fast convergence. Furthermore, the proposed DML approach
utilizes both global and local features to obtain rich representations in DML
model training. Finally, we also use the second-order attention for feature
enhancement to improve accurate and efficient retrieval. In our experiments,
we extensively evaluated the proposed DML approach on four public benchmarks,
and the experimental results demonstrate that the proposed method achieved
state-of-the-art performance on all benchmarks.

R@1 |
---|---
| Epochs
Figure 1: Accuracy in Recall@1 versus epochs on the Cars-196 [29] dataset.
Note that all methods were trained on a single Quadro p5000 GPU with a batch
size of 100. Our method achieves the highest accuracy while converging at the
same order as the proxy-based baselines in terms of the number of epochs.

##  1 Introduction

Learning semantically meaningful representations has been a vital step in
numerous computer vision applications such as representation learning [60,
68], content-based visual retrieval [26, 25, 36, 53], person or vehicle re-
identification [69, 56, 34], and face verification [30, 18]. Deep
Convolutional Neural Networks (CNNs) have proven repeatedly their
effectiveness in the large spectrum of applications [11, 10, 14, 58, 6, 13,
12] including Deep Metric Learning (DML). The neural networks in DML are
trained to map the data to a lower-dimensional embedding space in which
similar data (data in the same class) are pulled together and the dissimilar
data (data in different classes) are pushed away [53, 61]. For such an
embedding space, rich representations and special loss functions are
inevitable.

High image retrieval often requires global and local representations [7]. The
global features [47, 1], or “global descriptors” compactly summarize the
contents of an image. Often global descriptors are taken from the deepest
layers in CNNs; therefore, they only involve the most abstract information,
and the vital identifiers such as geometry and spatial information are lost.
On the other hand, local features [4, 33], involve information about the
geometry and spatial information of the input image. Generally speaking,
global features lead to better recall, while local features are essential in
better precision [7]. A typical retrieval system setup takes advantage of both
global and local features in its final embeddings to obtain the best of both
worlds.

Recently, self-attention or Second-Order Attention (SOA) in feature space has
received a significant attraction [55, 59, 38, 54, 65]. The SOA can be
considered as a spatial enhancement technique that reflects the correlation
among spatial locations and enhances the highly correlated parts of the
feature map. Although recent deep-learning-based global descriptors provide
effective ways to aggregate features into a compact global vector, they have
not explored the correlations of low-level and high-level features within
feature maps simultaneously.

Figure 2: Our proposed deep metric learning architecture with local and global
features. Our model jointly extracts deep local and global features. Both of
these features will be further enhanced spatially by an SOA mechanism.

The other vital factor in DML is the loss function. The loss functions are
essential to provide a powerful supervisory signal based on the problem
objectives [25, 61]. The loss functions in the DML problems are classified
into pairwise-based [61, 48, 57, 50, 41], and proxy-based [36, 25, 53] models.
The pairwise-based losses are built based on comparing the pairwise distances
between data in the batch. While the pairwise-based losses provide a strong
supervisory signal for training the model by considering _data-to-data_
relations [50, 61], they suffer from sample mining and slow convergence [25].

The proxy-based losses address the above issues by introducing a limited
number of proxies [36, 25, 53]. A proxy is a representative of a subset of
training data (for instance, a proxy per class) and learned along with network
parameters. Since the number of proxies is substantially smaller than the
data-points, the proxy-based models benefit from faster convergence rates than
the pairwise-based losses. Note the proxy-based models are associated with
_data-to-proxy_ relations and they miss the rich supervisory information of
data-to-data relations.

In this paper, we propose a multi-head network that benefits from the fast
convergence of the proxy-based loss functions and rich data-to-data relation
of the pairwise-based models. Although we are using a hybrid of both proxy-
based and pairwise-based loss functions in our multi-head network, our
approach does not introduce any hyper-parameter tuning for tuple sampling. Our
framework also involves an SOA mechanism to exploit the correlation between
features at different spatial locations to further enhance the deep local and
global features. Also, we combine both global and local descriptors to produce
the final descriptor that holds the content information as well as geometry
and spatial information to efficiently select the most similar images. With
the above advantages, our proposed method achieves state-of-the-art
performance in terms of Recall@1 and quickly converges as exhibited in Figure
1.

The contribution of this paper unfolds as follows: (a) We propose a multi-head
network that takes advantage of both pairwise-based and proxy-based methods;
it leverages rich data-to-data relations and enables fast and reliable
convergence. (b) We explore the SOA for further enhancement of both local and
global features based on higher-order information. (c) We demonstrate the
impact of using local and global descriptors, proxy-based and pairwise-based,
SOA, and the embedding dimensions via a thorough ablation study on their
effects. (d) An embedding neural network trained with our approach achieves
state-of-the-art performance on four publicly available benchmarks for metric
learning [29, 63, 42, 31].

---
Figure 3: Our proposed multi-head metric learning architecture with joint
local and global features. The local and global features are jointly extracted
from the backbone and they are sent to the SOA block (the blue block) for
further enhancement. Then, we apply pooling layers on top of the refined and
re-weighted features. The final embedding involves the concatenation of local
and global representations that are used for retrieval to efficiently select
the most similar images based on both local and global identifiers
simultaneously. Finally, a hybrid loss involving a proxy-based and a pairwise
based is applied to the final embedding.

##  2 Related Work

In this section, we categorize the DML approaches based on their use of
descriptors and loss functions into two broad categories, then we review
relevant papers in each category.

###  2.1 Loss Functions

Loss functions in DML can be divided into two groups, pairwise-based and
proxy-based.

Pairwise-based Losses. Contrastive loss [8, 19] and Triplet loss [48, 57] are
influential examples of loss functions for pairwise-based DML. Contrastive
loss takes a pair of embedding vectors as input and aims to push them apart if
they are of different classes or pull them together if they are of the same
class. Triplet loss considers a data point as an anchor. Each anchor is
associated with a positive (an embedding with an identical class label to the
anchor) and a negative data point (an embedding with different class labels)
and involves the distance of the anchor-positive pair to be smaller than that
of the anchor-negative pair in the embedding space.

One potential issue with pairwise-based models is that a large number of
tuples have a limited contribution to the learning algorithm and sometimes
even diminish the quality of the learned embedding space [64]. To address this
issue, most pairwise-based losses [50, 43, 62] employ hard sample mining
techniques [64, 20]. However, these techniques involve tuning hyper-parameters
and consequently increases the risk of over-fitting. Pairwise-based losses are
rich in data-to-data relations. However, the number of tuples increases
polynomially with regard to the number of training data, resulting in
prohibitive complexity and significantly slow convergence [25].

Proxy-based Losses. Proxy-based metric learning endeavors to address the
complexity and slow convergence issue of the pairwise-based losses. The proxy-
based methods require a small set of proxies to capture the global structure
of an embedding space and assign each data point to relevant proxies instead
of the other data points during training. Since the number of proxies is
significantly smaller than the training data, the training complexity reduces
substantially. For instance, Proxy-NCA [36] loss assigns a single proxy to
each class and associates data points to each proxy and encourages positive
pairs to be close together and negative pairs to be far apart. Proxy-NCA++
[53] is an extension of the Proxy-NCA and aims to enhance the limitations of
the Proxy-NCA in terms of temperature factor and pooling layer.

SoftTriple loss [46], inspired by the Proxy-NCA yet assigns multiple proxies
to each class instead of one to improve the likelihood of capturing the intra-
class variance. Proxy-Anchor loss [25] assigns a proxy to each class and
treats each proxy as an anchor and assigns positive and negative pairs to each
anchor. Although introducing proxies in proxy-based losses significantly
improves the convergence in model training, it has an inherent limitation of
data-to-proxy relation instead of data-to-data relation that results in
limited supervisory information. Our multi-head network overcomes this
limitation by proposing a hybrid approach involves in both pairwise-based and
proxy-based methods to benefit data-to-data relations as well as high
convergence rates.

###  2.2 Descriptors

DML algorithms can be divided into three groups based on their use of
descriptors: local descriptors, global descriptors, and joint local and global
descriptors.

Local Descriptors. Hand-engineered features such as SIFT [33] and SURF [4]
have been widely used and adopted for retrieval systems especially before the
deep learning era. The key advantage of local features over global ones for
image retrieval is their capacity to perform spatial matching, often by
utilizing RANSAC [15]. Due to the efficiency of local features, recently,
several deep learning-based local features have been proposed [66, 39, 35,
44].

Global Descriptors. Global descriptors are often involved the most abstract
information about the input, leading to high-performance image retrieval.
Before the deep learning era, most global descriptors were obtained using the
combination of local descriptors [23, 24]. However, recently most high-
performing global features are obtained based on CNNs [3, 2, 17].

Joint Local and Global Descriptors. Global descriptors are essential for high
recalls yet local descriptors are necessary for better precision; therefore,
researchers develop hybrid methods to take advantage of both descriptors. For
instance, Taira  _et al_. [52] used NetVLAD [2] to extract global features for
candidate pose retrieval, followed by dense local feature matching using
feature maps from the same network for indoor localization. Simeoni  _et al_.
’s DSM [49] detected key points in activation maps from global feature models.
Activation channels are interpreted as visual words, to propose
correspondences between a pair of images. Cao  _et al_. [7] extracted global
and local features from the same network. They utilize the global descriptors
to retrieve the most similar images and then re-rank the retrieved images by
local descriptors to increase the precision.

##  3 Proposed Algorithm

Our proposed algorithm involves two essential components: Refined deep local
and global representations along with a multi-head loss function that enables
the data-to-data relation as well as fast convergence. Our model design is
illustrated in Figure 2.

###  3.1 Deep Global and Local Representations

We propose to leverage hierarchical representations from a CNN to represent
different types of descriptors. While deep layers are associated with the most
abstract representations and representing higher-level features, the
intermediate layers are more informative in terms of local representations and
lower-level features.

Given an image, from the backbone we obtain two feature maps:
$f_{l}\in\mathbb{R}^{H_{l}\times W_{l}\times C_{l}}$ and
$f_{g}\in\mathbb{R}^{H_{g}\times W_{g}\times C_{g}}$ , representing local
($l$) and global ($g$) feature maps where $H,W,C$ indicate the height, width,
and number of channels respectively. For off-the-shelf convolutional networks,
$H_{g}\leq H_{l}$, $W_{g}\leq W_{l}$, and $C_{g}\geq C_{l}$; indicating that
deeper layers have larger number of channels and spatially smaller feature
maps.

###  3.2 Second-Order Attention (SOA)

Let $(i_{I},j_{I})$ in the input image ($I$) correspond to location $(i,j)$ in
feature map $f$. To incorporate higher-order spatial information into the
feature map, we adopt the second-order attention block [55, 38]. A
computational flow of the SOA concept is shown in Figure 3. For each feature
map, we produce two projections of feature map $f$ named _query_ $q$, and
_key_ $k$, each obtained through $1\times 1$ 2d-convolutions with possible
reduction of number of channels ($d$). Then, by flattening both tensors, we
obtain both $q$ and $k$ in $\mathbb{R}^{d\times hw}$. The second-order
attention map $A$ is computed as follows:

$a=\text{softmax}(\zeta q^{T}k),$ (1)

where $\zeta$ is a scaling factor and $a\in\mathbb{R}^{hw\times hw}$,
indicating the correlation of each $f_{i,j}$ to the whole map $f$. A third
projection of $f$ and _value_ $v$ is then obtained by $1\times 1$
2d-convolution, and after flattening, results in $\mathbb{R}^{hw\times d}$
shape. Then, the second-order attention map $f^{soa}$ is obtained from linear
combination of the first-order features $f$ and the second-order attention
map:

$f^{soa}=f+\phi(a\times v),$ (2)

where $\phi$ is yet another $1\times 1$ convolution to manage the effect of
the obtained attention map. Thus, a new feature $f^{soa}_{i,j}$ in the second-
order map $f^{soa}\in\mathbb{R}^{h\times w\times d}$, is a function of
features from all locations in $f$:

$f^{soa}_{i,j}=h(a_{ij}\odot f),$ (3)

where $h$ denotes the combination of all convolutional operations within the
non-local block.

###  3.3 Pooling

To aggregate deep activations in both global and local features, we adopt the
combination of Global Max Pooling (GMP) and Global Average Pooling (GAP) as
follows:

$\begin{split}f=\frac{1}{W\times H}\sum_{i\in W,j\in
H}f^{soa}+\text{max}_{(i\in W,j\in H)}f^{soa}\end{split}$ (4)

After the aggregation, we whiten the aggregated representation for both
refined local and global representations; we integrate this into our model
with two separated fully-connected layers. The fully connected layer
associated with enhanced local representations
$F_{l}\in\mathbb{R}^{C_{f^{soa}_{l}}\times\frac{D}{2}}$ , with learned bias
$b_{f^{soa}_{l}}\in\mathbb{R}^{C_{f^{soa}_{l}}}$, which $C_{f^{soa}_{l}}$
indicates the number of channels in the $f^{soa}_{l}$ and $\frac{D}{2}$ is the
dimension of the local embedding space. Similarly, we have a fully connected
layer associated with enhanced global representations
$F_{g}\in\mathbb{R}^{C_{f^{soa}_{g}}\times\frac{D}{2}}$ , with learned bias
$b_{f^{soa}_{g}}\in\mathbb{R}^{C_{f^{soa}_{g}}}$, which $C_{f^{soa}_{g}}$
indicates the number of channels in the $f^{soa}_{g}$ and $\frac{D}{2}$ is the
dimension of the global embedding space.

After computing $F_{l}\in\mathbb{R}^{D/2}$ and $F_{g}\in\mathbb{R}^{D/2}$, the
final embedding $F\in\mathbb{R}^{D}$ computes by concatenation of the $F_{l}$
and $F_{g}$.

Table 1: Recall@K ($\%$) on the Cars-196 [29] and CUB-200-2011 [63] datasets. Superscripts indicate embedding sizes. Backbone networks of the models are denoted by abbreviations: G–GoogleNet [51], BN–Inception with batch normalization [22], R50–ResNet50 [21]. For each group of methods, the best performance is bolded and the second best is underlined. Algorithms | BackBone | Cars-196 | CUB-200-2011
---|---|---|---
|  | R@1 | R@2 | R@4 | R@8 | R@1 | R@2 | R@4 | R@8
Clustering64 [40] | BN | 58.1 | 70.6 | 80.3 | 87.8 | 48.2 | 61.4 | 71.8 | 81.9
Proxy-NCA${}^{64}~{}\cite[cite]{[\@@bibref{}{proxy-NCA}{}{}]}$ | BN | 73.2 | 82.4 | 86.4 | 87.8 | 49.2 | 61.9 | 67.9 | 72.4
Smart Mining${}^{64}~{}\cite[cite]{[\@@bibref{}{smart-mining}{}{}]}$ | G | 64.7 | 76.2 | 84.2 | 90.2 | 49.8 | 62.3 | 74.1 | 83.3
MS64 [61] | BN | 78.6 | 86.6 | 91.8 | 95.4 | 60.1 | 71.9 | 81.2 | 88.5
Proxy-Anchor64 [25] | R50 | 78.8 | 87.0 | 92.2 | 95.5 | 61.7 | 73.0 | 81.8 | 88.8
lightgrayOurs64 | R50 | 81.1 | 88.1 | 92.3 | 95.3 | 63.1 | 74.6 | 83.2 | 89.4
Margin128 [64] | R50 | 79.6 | 86.5 | 91.9 | 95.1 | 63.6 | 74.4 | 83.1 | 90.0
lightgray Ours128 | R50 | 84.9 | 90.6 | 94.0 | 96.6 | 66.6 | 77.1 | 85.2 | 91.3
HDC384 [67] | G | 73.7 | 83.2 | 89.5 | 93.8 | 53.6 | 65.7 | 77.0 | 85.6
A-BIER${}^{512}~{}\cite[cite]{[\@@bibref{}{a-bier}{}{}]}$ | G | 82.0 | 89.0 | 93.2 | 96.1 | 57.5 | 68.7 | 78.3 | 86.2
ABE512 [27] | G | 85.2 | 90.5 | 94.0 | 96.1 | 60.6 | 71.5 | 79.8 | 87.4
HTL512 | BN | 81.4 | 88.0 | 92.7 | 95.7 | 57.1 | 68.8 | 78.7 | 86.5
RLL-H512 [62] | BN | 74.0 | 83.6 | 90.1 | 94.1 | 57.4 | 69.7 | 79.2 | 86.9
MS512 [61] | R50 | 84.1 | 90.4 | 94.0 | 96.5 | 65.7 | 77.0 | 86.3 | 91.2
SoftTriple512 [46] | BN | 84.5 | 90.7 | 94.5 | 96.9 | 65.4 | 76.4 | 84.5 | 90.4
Proxy-Anchor512 [25] | BN | 86.1 | 91.7 | 95.0 | 97.3 | 68.4 | 79.2 | 86.8 | 91.6
ProxyNCA++${}^{512}~{}\cite[cite]{[\@@bibref{}{proxy-nca++}{}{}]}$ | R50 | 86.5 | 92.5 | 95.7 | 97.7 | 69.0 | 79.8 | 87.3 | 92.7
Proxy-Anchor512 [25] | R50 | 87.7 | 92.9 | 95.8 | 97.9 | 69.7 | 80.0 | 87.0 | 92.4
lightgray Ours512 | R50 | 90.1 | 94.2 | 96.4 | 98.1 | 70.6 | 80.9 | 88.0 | 92.3

###  3.4 A Hybrid Loss Function

Our loss is designed to overcome the limitation of both proxy-based and
pairwise-based models by introducing a hybrid loss involving the proxy-anchor
[25] loss from the proxy-based category and the MS [61] loss from the
pairwise-based class.

Proxy-based Loss. Proxy-anchor loss [25] assigns a proxy to each class. Proxy-
anchor approach considers each proxy as an anchor and associate it with entire
data in a batch to find positive and negative samples. The proxy-anchor loss
defined as follows:

$\begin{split}\ell_{p}(X)=\frac{1}{|P^{+}|}\sum_{p\in
P^{+}}\text{log}(1+\sum_{x\in X_{p}^{+}}\text{exp}(-\alpha(s(x,p)-\delta)))\\\
+\frac{1}{|P|}\sum_{p\in P}\text{log}(1+\sum_{x\in
X_{p}^{-}}\text{exp}(\alpha(s(x,p)+\delta))),\end{split}$ (5)

where $\delta>0$ is a margin, $\alpha>0$ is a scaling factor, $P$ is the set
of all proxies, $s(.,.)$ measures the similarity among its arguments ,and
$P^{+}$ indicates the set of positive proxies of data in the batch. Also, for
each proxy $p$, a batch of embedding vectors $X$ is divided into the set of
positive $X^{+}_{p}$ and negative $X^{-}_{p}=X-X^{+}_{p}$ embedding vectors.

By utilizing the proxy-anchor loss we incorporate data-to-proxy relations as
well as fast convergence. For incorporating the data-to-data relation, we
integrate the MS loss.

Pairwise-based Loss. We employ the MS loss [61] as a pairwise-based loss since
it considers the self, negative, and positive similarities. Self-similarity
ensures that the instances belonging to a positive class remains closer to the
anchor than the instances associated with negative classes. The positive
similarity exclusively deals with positive pairs. $\sigma$ represents the
similarity margin that controls the closeness of positive pairs by heavily
penalizing those pairs whose cosine similarities are less or equal to
$\sigma$. The negative similarity ensures that negative samples have
similarity with the anchor as low as possible. The MS loss function is
formulated as follows:

$\begin{split}\ell_{m}(X)=\frac{1}{m}\sum_{i=1}^{m}(\frac{1}{\gamma}\text{log}(1+\sum_{k\in
P_{i}}\text{exp}(-\gamma(S_{i,k}-\sigma)))\\\
+\frac{1}{\beta}\text{log}(1+\sum_{k\in
N_{i}}\text{exp}(\beta(S_{i,k}+\sigma))),\end{split}$ (6)

where $\gamma$, $\beta$, and $\sigma$ are hyper-parameters. $m$ is the number
of samples, $P_{i}$,$N_{i}$ represents positive and negative samples, and
$S_{i,j}$ denotes the pairwise similarity between $x_{i}$ and $x_{j}$.

Our Objective Function. Our hybrid objective function is a combination of
proxy-anchor and MS losses balanced by normalization factor $\lambda$:

$\mathcal{L}=\ell_{m}+\lambda\ell_{p}$ (7)

##  4 Experimental Results

Table 2: Recall@K ($\%$) on the SOP. Superscripts indicates embedding sizes. For each group of methods, the best performance is bolded and the second best is underlined. Recall@K | 1 | 10 | 100 | 1000
---|---|---|---|---
Clustering64 [40] | 67.0 | 83.7 | 93.2 | -
Proxy-NCA64 [36] | 73.7 | - | - | -
MS64 | 74.1 | 87.8 | 94.7 | 98.2
SoftTriple64 [46] | 76.3 | 89.1 | 95.3 | -
Proxy-Anchor64 [25] | 76.5 | 89.0 | 95.1 | 98.2
lightgray Ours64 | 77.3 | 89.5 | 95.4 | 98.3
Margin128 [64] | 72.7 | 86.2 | 93.8 | 98.0
lightgray Ours128 | 79.1 | 90.6 | 95.8 | 98.5
HDC384 [67] | 69.5 | 84.4 | 92.8 | 97.7
A-BIER512 [45] | 74.2 | 86.9 | 94.0 | 97.8
ABE512 [27] | 76.3 | 88.4 | 94.8 | 98.2
HTL512 [16] | 74.8 | 88.3 | 94.8 | 98.4
RLL-H512 [62] | 76.1 | 89.1 | 95.4 | -
MS512 [61] | 78.2 | 90.5 | 96.0 | 98.7
SoftTriple512 [46] | 78.3 | 90.3 | 95.9 | -
Proxy-Anchor512 [25] | 79.1 | 90.8 | 96.2 | 98.7
Proxy-NCA++512 [53] | 80.7 | 92.0 | 96.7 | 98.9
lightgray Ours512 | 81.7 | 92.0 | 96.6 | 98.8
Table 3: Recall@K ($\%$) on the In-Shop. Superscripts indicates embedding sizes. For each group of methods, the best performance is bolded and the second best is underlined. Recall@K | 1 | 10 | 20 | 40
---|---|---|---|---
HDC384 [67] | 62.1 | 84.9 | 89.0 | 92.3
HTL128 [16] | 80.9 | 94.3 | 95.8 | 97.4
MS128 [61] | 88.0 | 97.2 | 98.1 | 98.7
Proxy-Anchor128 [25] | 90.8 | 97.9 | 98.5 | 99.0
lightgray Ours128 | 90.9 | 97.9 | 98.6 | 98.9
FashionNet4096 [31] | 53.0 | 73.0 | 76.0 | 79.0
A-BIER512 [45] | 83.1 | 95.1 | 96.9 | 97.8
ABE512 [27] | 87.3 | 96.7 | 97.9 | 98.5
MS512 [61] | 89.7 | 97.9 | 98.5 | 99.1
Proxy-Anchor512 [25] | 91.5 | 98.1 | 98.8 | 99.1
ProxyNCA++512 [53] | 90.4 | 98.1 | 98.8 | 99.2
lightgray Ours512 | 93.1 | 98.3 | 99.0 | 99.2

In this section, our method is compared with current state-of-the-art methods
on four public benchmark datasets employed for deep metric learning [29, 63,
42, 31]. We also perform a thorough investigation of local and global
features, the SOA, the MS loss and proxy-anchor loss, and embedding
dimensionality to study their effects on the proposed method.

###  4.1 Dataset

We evaluated our model on the CUB-200-2011 [63], Cars-196 [29], Stanford
Online Product (SOP) [42] and In-Shop Clothes Retrieval (In-Shop) [31]
datasets. For CUB-200-2011, we set aside 5,864 images of its first 100 classes
as a training set and 5,924 images of the other classes as a test set. For
Cars-196, 8,054 images of its first 98 classes are set aside as a training set
and 8,131 images of the other classes are used as a test set. For SOP, we
follow the standard dataset split in [43, 25, 53] using 59,551 images of
11,318 classes as a training set and the remaining 60,502 images as a test
set. Also for In-Shop dataset, we follow the setting in [25] to use 25,882
images of the first 3,997 classes as a training set and 28,760 images of the
remaining classes for test set; the test set is further partitioned into a
query and gallery sets with 14,218 images of 3,985 classes and 12,612 images
of 3,985 classes, respectively. For all datasets, we set aside $20\%$ of the
training set as a validation set for hyper-parameter tuning [37].

###  4.2 Implementation Setup

Backbone: For a fair comparison [37] to recent works, the Resnet50 [21] pre-
trained on ImageNet classification [9] is adopted as our backbone network.

Global and Local Features: For all experiments on all datasets, we obtained
the global features from resnet50-conv5-x$\in\mathbb{R}^{7\times 7\times
2084}$ layer and local features are extracted from
resnet50-conv4-x$\in\mathbb{R}^{14\times 14\times 1024}$ .

Training: In all of the experiments, AdamW algorithm [32] has been adopted and
used as our optimizer. AdamW has the same update step as Adam [28] with
separate decays of weights. In all experiments, our model is trained only for
20 epochs and the initial learning rate is fixed to $10^{-4}$. Note that the
learning rate for proxies is set to $10^{-2}$ for faster convergence.

Proxy Setting: We assign a single proxy to each class as suggested in Proxy-
NCA [36] and Proxy-Anchor [25]. The proxies are initialized with a normal
distribution.

Input Setting: To have a fair comparison with state-of-the-art methods, the
input is re-scaled to $256\times 256$ and then center-cropped to $224\times
224$. We used random cropping and horizontal flipping during training as the
data augmentation strategy, as suggested in [25, 36]. During the test, the
images are only center-cropped. The default size of cropped images is fixed to
$224\times 224$ [37].

Hyperparameter Setting: $\zeta$ in Eq. 1 is set to $1$. $\alpha$ and $\delta$
in Eq. 5 is set to $32$ and $10^{-1}$ respectively. $\gamma$, $\beta$, and
$\sigma$ in Eq. 6 are set to $2,50,1$. Finally, $\lambda$ in Eq. 7 is set to
$3\times 10^{-2}$, for all experiments.

###  4.3 Comparison to Other Methods

We demonstrate the strength of our proposed method quantitatively by
evaluating its image retrieval performance on four public benchmark datasets.
For a fair comparison to the previous arts [37], the accuracy of our model is
computed in three different settings: we used 64, 128, and 512 embedding
dimension on Cars-196, CUB-200-2011, and SOP datasets with the default image
size $224\times 224$. On the In-Shop dataset the performance is only measured
with embedding dimensions of 128 and 512 with the default image size
$224\times 224$. Results on the Cars-196 and CUB-200-2011 datasets are
exhibited in Table 1. According to Table 1, our model outperforms all the
previous state-of-the-art methods including the proxy-based [25, 36, 53],
pairwise-based [62, 61, 46] and ensemble methods [27] in all three settings
often with a large margin on top 1 recall. In particular, on the challenging
Cars-196 dataset, our method improves the previous best score by a large
margin, 2.3$\%$, 5.3$\%$, and 2.4$\%$ in Recall$@1$ in embedding size of 64,
128, and 512 respectively. As reported in Table 2, our model also achieves
state-of-the-art performance on the SOP dataset. It outperforms previous
methods in all cases except for Recall@10 and Recall@100 in 64-dimensional
embedding, but even in these cases, it achieves the second best. Finally, on
the In-Shop dataset, it obtains the best scores in all two settings as shown
in Table 3. On the In-Shop dataset, our model outperforms the state-of-the-art
by a large margin of 2.7$\%$ in Recall@1. In our experimental results, we
noted that our model outperforms numerous state-of-the-art methods even with
low-dimensional embedding vectors while they have higher embedding dimensions.
This observation suggests that our model is capable of learning a more compact
and effective embedding space. Last but not least, our hybrid method converges
in the same order as the proxy-based method as results are summarized in
Figure 1.

---
Figure 4: Qualitative results on the Cars-196 (a) CUB-200-201l (b), SOP (c), and In-Shop (d). For each query image (leftmost), top 4 retrievals are exhibited. The results with red boundaries are false cases but they are substantially similar to the query images in terms of appearance. (rows 2,3,6, and 7). (a) | (b)
---|---
| Cars-196 | CUB-200-2011 | Cars-196 | CUB-200-2011
R@1 |  |  |  |
| Epochs | Epochs | Epochs | Epochs
Figure 5: (a): The impact of local and global descriptors in terms of Recall@1
performance on both Cars-196 and CUB-200-2011 datasets. The blue and green
colors illustrate the performance of the global and local descriptors, while
the red indicates the performance of the combined global and local
descriptors. The (a) part of this figure demonstrates the combination of local
and global features are vital in our design. (b): The impact of the SOA
component on the mentioned datasets. The blue color exhibits the model without
second-order attention and red depicts the model with second-order attention.

Cars196 | CUB-200-2011
---|---
|

Figure 6: Qualitative examples of SOA maps on the Cars-196 and CUB-200-2011
datasets. (a) corresponds to Cars196 dataset and (b) corresponds to
CUB-200-2011 dataset. Each row depicts the source image and four corresponding
SOA maps obtained for specific spatial locations (marked by stars).

###  4.4 Qualitative Results

To further exhibiting the visual performance of our method, we illustrate the
qualitative retrieval results of our model on four datasets in Figure 4. Note
that these datasets are challenging especially due to their large intra-class
variations. For instance, the CUB200-2011 has a variety of poses and
background clutter, the Cars-196 has various colors and shapes, and SOP and
In-Shop datasets have challenging view-points of objects that make the
retrieval tasks even harder. In contrast to all of these challenges, our
proposed method performs robust retrieval.

###  4.5 Ablation Study

| Cars-196 | CUB-200-2011
---|---|---
R@1 |  |
| Epochs | Epochs
Figure 7: The impact of the MS and Proxy-Anchor loss on Cars-196 and CUB-200-2011 datasets. The red color indicates our hybrid loss while the green and blue represent the Proxy-Anchor and MS losses, respectively. The Figure demonstrates the combination of two losses is crucial in our proposed method. R@1 |
---|---
| Dimensions
Figure 8: Accuracy in terms of Recall@1 versus embedding dimension on both
Cars-196 and CUB-200-2011 datasets.

Local and Global Features. To investigate the impact of the local and global
features on the performance of our proposed method, we examined the Recall@1
while training our model with only local and global descriptors separately on
the Cars-196 and CUB-200-2011 datasets. The result of the analysis is
summarized in Figure 5 (a). This Figure illustrates the local descriptors on
both Cars-196 and CUB-200-2011 datasets are getting slightly better
performance (2$\%$) than the global descriptors but neither of these
descriptors individually achieved the performance as the combination of these
descriptors obtained on both datasets based on Recall@1 performance (see Table
1).

Second-order Attention. One of the vital components of our proposed method is
SOA. We employ this attention mechanism to further enhance deep features based
on their correlation. To evaluate the impact of the SOA, we trained our model
with and without having the SOA, and the results are summarized in Figure 5
(b). According to this figure, the performance of the model significantly
improves in terms of Recall@1 when the model is trained with SOA especially on
critical datasets like CUB-200-2011. This observation confirms our assumption
on the essence of higher-order attention for further enhancements of the
representations. We visualize the effects of second-order attention in Figure
9. This figure exhibits the attention map of samples from Cars-196 and
CUB-200-2011 datasets. For each image, four parts have been selected with
different stars and different colors. The attention map associated with each
star has a border with identical colors. For locations in the background,
interestingly, the attention from that feature is distributed within the main
object in the image. It confirms the second-order attention has learned to
focus on the main object in the image to accurately retrieve the most similar
images to the query. On the other hand, when the star is located within the
main object, the attention is on highly distinctive regions.

A Single Head Attention. Our proposed method requires two attention heads for
local and global descriptors. We are interested in probing a test case with
having only a single attentional head for both local and global descriptors.
After extracting local descriptor $f_{l}\in\mathbb{R}^{14\times 14\times
1024}$ and global descriptor $f_{g}\in\mathbb{R}^{7\times 7\times 2048}$ from
backbone, we use a focus layer [5] to reshape the local descriptor
$f_{l}^{new}\in\mathbb{R}^{7\times 7\times 4096}$ to spatially match the
global descriptor. Then, we concatenate the global and local descriptors
resulting in a combined feature map $f\in\mathbb{R}^{\in 7\times 7\times
6144}$. Then, we apply the SOA for further enhancement of this feature map. We
evaluated this approach on Cars-196 and CUB-200-2011 datasets. The single head
attention obtained 86.6$\%$ and 66.6$\%$ in terms of Recall@1 on Cars-196 and
CUB-200-2011, respectively. Note the obtained results are slightly degraded
from the multi-head approach 90.1$\%$ on Cars-196 and 70.6$\%$ on CUB-200-2011
(see Table 1), that suggests the multi-head SOA is necessary.

Multisimilarity and Proxy-anchor Loss. The other crucial component of our
architecture is the combination of pairwise-based and proxy-based loss
functions. To study the impact of each loss function on our proposed method,
we trained our model separately with MS loss and proxy anchor loss and we
evaluated the performance based on Recall@1 on Cars-196 and CUB-200-2011
datasets. The results is exhibited in Figure 7. This Figure demonstrates that
the combination of these two losses are crucial in our design since none of
them individually achieves our performance (see Table 1).

Embedding Dimension. The dimension of embedding vectors is a vital factor that
controls the trade-off between speed and accuracy in image retrieval systems.
We thus investigate the effect of embedding dimensions on the retrieval
accuracy in our method. We evaluated our model with embedding dimensions
varying from 64 to 2048 following the experiment in [61, 25]. The result of
the analysis is illustrated in Figure 8, in which the retrieval performance of
our model is reported on both Cars-196 and CUB-200-2011 dataset. The
performance of our loss is fairly stable when the dimension is equal to or
larger than 128. The performance of our model on the Cars-196 dataset improves
until reaching 1024 dimensional embedding and after that slightly degrades. On
the other hand, the performance consistently increases with the embedding
dimension on the CUB-200-2011 dataset showing that more information on that
dataset helps the retrieval performance.

##  5 Conclusion

We have proposed a novel metric learning algorithm that takes the best of both
proxy-based and pairwise-based losses. Also, it leverages enhanced local and
global descriptors to improve the recall and precision simultaneously. Our
method benefits from having a reach data-to-data relation as well as fast and
reliable convergence. We extensively evaluated our model on 4 public
benchmarks and our model has achieved state-of-the-art performance on all
datasets in terms of Recall@1 accuracy. Also, our model converged quickly
without any careful data sampling technique.

## Acknowledgements

This research work was sponsored by the Office of Naval Research (ONR) via
contract N00014-17-C-2007. Disclaimer: The views and conclusions contained
herein are those of the authors and should not be interpreted as necessarily
representing the official positions, policies or endorsements, either
expressed or implied, of ONR or the U.S. Government.

## References

  * [1] Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic.  Netvlad: Cnn architecture for weakly supervised place recognition.  In CVPR, 2016.
  * [2] Relja Arandjelovic, Petr Gronat, Akihiko Torii, Tomas Pajdla, and Josef Sivic.  Netvlad: Cnn architecture for weakly supervised place recognition.  In CVPR, 2016.
  * [3] Artem Babenko, Anton Slesarev, Alexandr Chigorin, and Victor Lempitsky.  Neural codes for image retrieval.  In ECCV, 2014.
  * [4] Herbert Bay, Andreas Ess, Tinne Tuytelaars, and Luc Van Gool.  Speeded-up robust features (surf).  CVIU, 2008.
  * [5] Alexey Bochkovskiy, Chien-Yao Wang, and Hong-Yuan Mark Liao.  Yolov4: Optimal speed and accuracy of object detection.  arXiv preprint arXiv:2004.10934, 2020.
  * [6] Necati Cihan Camgoz, Oscar Koller, Simon Hadfield, and Richard Bowden.  Sign language transformers: Joint end-to-end sign language recognition and translation.  In CVPR, 2020.
  * [7] Bingyi Cao, Andre Araujo, and Jack Sim.  Unifying deep local and global features for image search.  In ECCV, 2020.
  * [8] Sumit Chopra, Raia Hadsell, and Yann LeCun.  Learning a similarity metric discriminatively, with application to face verification.  In CVPR, 2005.
  * [9] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei.  Imagenet: A large-scale hierarchical image database.  In CVPR, 2009.
  * [10] Mohammad Ebrahimpour, Timothy Shea, Andreea Danielescu, David Noelle, and Chris Kello.  End-to-end auditory object recognition via inception nucleus.  In ICASSP, 2020.
  * [11] Mohammad K Ebrahimpour, J Ben Falandays, Samuel Spevack, and David C Noelle.  Do humans look where deep convolutional neural networks “attend”?  In ISVC, 2019.
  * [12] Mohammad K Ebrahimpour, J Ben Falandays, Samuel Spevack, Ming-Hsuan Yang, and David C Noelle.  Ww-nets: Dual neural networks for object detection.  In IJCNN, 2020.
  * [13] Mohammad K Ebrahimpour, Jiayun Li, Yen-Yun Yu, Jackson Reesee, Azadeh Moghtaderi, Ming-Hsuan Yang, and David C Noelle.  Ventral-dorsal neural networks: object detection via selective attention.  In WACV, 2019.
  * [14] Mohammad K Ebrahimpour and David C Noelle.  Fast object localization via sensitivity analysis.  In ISVC, 2019.
  * [15] Martin A Fischler and Robert C Bolles.  Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography.  Communications of the ACM, 1981.
  * [16] Weifeng Ge.  Deep metric learning with hierarchical triplet loss.  In ECCV, 2018.
  * [17] Albert Gordo, Jon Almazan, Jerome Revaud, and Diane Larlus.  End-to-end learning of deep visual representations for image retrieval.  IJCV, 2017.
  * [18] Jianzhu Guo, Xiangyu Zhu, Chenxu Zhao, Dong Cao, Zhen Lei, and Stan Z Li.  Learning meta face recognition in unseen domains.  In CVPR, 2020.
  * [19] Raia Hadsell, Sumit Chopra, and Yann LeCun.  Dimensionality reduction by learning an invariant mapping.  In CVPR, 2006.
  * [20] Ben Harwood, Vijay Kumar BG, Gustavo Carneiro, Ian Reid, and Tom Drummond.  Smart mining for deep metric learning.  In ICCV, 2017.
  * [21] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [22] Sergey Ioffe and Christian Szegedy.  Batch normalization: Accelerating deep network training by reducing internal covariate shift.  arXiv preprint arXiv:1502.03167, 2015.
  * [23] Hervé Jégou, Matthijs Douze, Cordelia Schmid, and Patrick Pérez.  Aggregating local descriptors into a compact image representation.  In CVPR, 2010.
  * [24] Herve Jegou, Florent Perronnin, Matthijs Douze, Jorge Sánchez, Patrick Perez, and Cordelia Schmid.  Aggregating local image descriptors into compact codes.  PAMI, 2011.
  * [25] Sungyeon Kim, Dongwon Kim, Minsu Cho, and Suha Kwak.  Proxy anchor loss for deep metric learning.  In CVPR, 2020.
  * [26] Sungyeon Kim, Minkyo Seo, Ivan Laptev, Minsu Cho, and Suha Kwak.  Deep metric learning beyond binary supervision.  In CVPR, 2019.
  * [27] Wonsik Kim, Bhavya Goyal, Kunal Chawla, Jungmin Lee, and Keunjoo Kwon.  Attention-based ensemble for deep metric learning.  In ECCV, 2018.
  * [28] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  arXiv preprint arXiv:1412.6980, 2014.
  * [29] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei.  3d object representations for fine-grained categorization.  In CVPRW, 2013.
  * [30] Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song.  Sphereface: Deep hypersphere embedding for face recognition.  In CVPR, 2017.
  * [31] Ziwei Liu, Ping Luo, Shi Qiu, Xiaogang Wang, and Xiaoou Tang.  Deepfashion: Powering robust clothes recognition and retrieval with rich annotations.  In CVPR, 2016.
  * [32] Ilya Loshchilov and Frank Hutter.  Decoupled weight decay regularization.  arXiv preprint arXiv:1711.05101, 2017.
  * [33] David G Lowe.  Distinctive image features from scale-invariant keypoints.  IJCV, 2004.
  * [34] Dechao Meng, Liang Li, Xuejing Liu, Yadong Li, Shijie Yang, Zheng-Jun Zha, Xingyu Gao, Shuhui Wang, and Qingming Huang.  Parsing-based view-aware embedding network for vehicle re-identification.  In CVPR, 2020.
  * [35] Anastasiia Mishchuk, Dmytro Mishkin, Filip Radenovic, and Jiri Matas.  Working hard to know your neighbor’s margins: Local descriptor learning loss.  In NIPS, 2017.
  * [36] Yair Movshovitz-Attias, Alexander Toshev, Thomas K Leung, Sergey Ioffe, and Saurabh Singh.  No fuss distance metric learning using proxies.  In ICCV, 2017.
  * [37] Kevin Musgrave, Serge Belongie, and Ser-Nam Lim.  A metric learning reality check.  arXiv preprint arXiv:2003.08505, 2020.
  * [38] Tony Ng, Vassileios Balntas, Yurun Tian, and Krystian Mikolajczyk.  Solar: Second-order loss and attention for image retrieval.  arXiv preprint arXiv:2001.08972, 2020.
  * [39] Hyeonwoo Noh, Andre Araujo, Jack Sim, Tobias Weyand, and Bohyung Han.  Large-scale image retrieval with attentive deep local features.  In ICCV, 2017.
  * [40] Hyun Oh Song, Stefanie Jegelka, Vivek Rathod, and Kevin Murphy.  Deep metric learning via facility location.  In CVPR, 2017.
  * [41] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese.  Deep metric learning via lifted structured feature embedding.  In CVPR, 2016.
  * [42] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese.  Deep metric learning via lifted structured feature embedding.  In CVPR, 2016.
  * [43] Hyun Oh Song, Yu Xiang, Stefanie Jegelka, and Silvio Savarese.  Deep metric learning via lifted structured feature embedding.  In CVPR, 2016.
  * [44] Yuki Ono, Eduard Trulls, Pascal Fua, and Kwang Moo Yi.  Lf-net: learning local features from images.  In NIPS, 2018.
  * [45] Michael Opitz, Georg Waltner, Horst Possegger, and Horst Bischof.  Deep metric learning with bier: Boosting independent embeddings robustly.  PAMI, 2018.
  * [46] Qi Qian, Lei Shang, Baigui Sun, Juhua Hu, Hao Li, and Rong Jin.  Softtriple loss: Deep metric learning without triplet sampling.  In ICCV, 2019.
  * [47] Filip Radenović, Giorgos Tolias, and Ondřej Chum.  Fine-tuning cnn image retrieval with no human annotation.  PAMI, 2018.
  * [48] Florian Schroff, Dmitry Kalenichenko, and James Philbin.  Facenet: A unified embedding for face recognition and clustering.  In CVPR, 2015.
  * [49] Oriane Siméoni, Yannis Avrithis, and Ondrej Chum.  Local features and visual words emerge in activations.  In CVPR, 2019.
  * [50] Kihyuk Sohn.  Improved deep metric learning with multi-class n-pair loss objective.  In NeurIPS, 2016.
  * [51] Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich.  Going deeper with convolutions.  In CVPR, 2015.
  * [52] Hajime Taira, Masatoshi Okutomi, Torsten Sattler, Mircea Cimpoi, Marc Pollefeys, Josef Sivic, Tomas Pajdla, and Akihiko Torii.  Inloc: Indoor visual localization with dense matching and view synthesis.  In CVPR, 2018.
  * [53] Eu Wern Teh, Terrance DeVries, and Graham W Taylor.  Proxynca++: Revisiting and revitalizing proxy neighborhood component analysis.  In ECCV, 2020.
  * [54] Yurun Tian, Xin Yu, Bin Fan, Fuchao Wu, Huub Heijnen, and Vassileios Balntas.  Sosnet: Second order similarity regularization for local descriptor learning.  In CVPR, 2019.
  * [55] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [56] Dongkai Wang and Shiliang Zhang.  Unsupervised person re-identification via multi-label classification.  In CVPR, 2020.
  * [57] Jiang Wang, Yang Song, Thomas Leung, Chuck Rosenberg, Jingbin Wang, James Philbin, Bo Chen, and Ying Wu.  Learning fine-grained image similarity with deep ranking.  In CVPR, 2014.
  * [58] Li Wang, Dong Li, Yousong Zhu, Lu Tian, and Yi Shan.  Dual super-resolution learning for semantic segmentation.  In CVPR, 2020.
  * [59] Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He.  Non-local neural networks.  In CVPR, 2018.
  * [60] Xiaolong Wang and Abhinav Gupta.  Unsupervised learning of visual representations using videos.  In ICCV, 2015.
  * [61] Xun Wang, Xintong Han, Weilin Huang, Dengke Dong, and Matthew R Scott.  Multi-similarity loss with general pair weighting for deep metric learning.  In CVPR, 2019.
  * [62] Xinshao Wang, Yang Hua, Elyor Kodirov, Guosheng Hu, Romain Garnier, and Neil M Robertson.  Ranked list loss for deep metric learning.  In CVPR, 2019.
  * [63] P Welinder, S Branson, T Mita, C Wah, F Schroff, S Belongie, P Perona, and Caltech-UCSD Birds.  200\.  California Institute of Technology, 2010.
  * [64] Chao-Yuan Wu, R Manmatha, Alexander J Smola, and Philipp Krahenbuhl.  Sampling matters in deep embedding learning.  In ICCV, 2017.
  * [65] Bryan Ning Xia, Yuan Gong, Yizhe Zhang, and Christian Poellabauer.  Second-order non-local attention networks for person re-identification.  In CVPR, 2019.
  * [66] Kwang Moo Yi, Eduard Trulls, Vincent Lepetit, and Pascal Fua.  Lift: Learned invariant feature transform.  In ECCV, 2016.
  * [67] Yuhui Yuan, Kuiyuan Yang, and Chao Zhang.  Hard-aware deeply cascaded embedding.  In ICCV, 2017.
  * [68] Sergey Zagoruyko and Nikos Komodakis.  Learning to compare image patches via convolutional neural networks.  In CVPR, 2015.
  * [69] Zhizheng Zhang, Cuiling Lan, Wenjun Zeng, Xin Jin, and Zhibo Chen.  Relation-aware global attention for person re-identification.  In CVPR, 2020.

##  6 Additional Experimental Results

This supplementary material exhibits additional experimental results excluded
from the main paper due to space limitation. Section 6.1 analyzes the impact
of the batch size in our framework. Section 6.2 visualizes attention maps
generated from Cars-196 [29] and CUB-200-2011 [63]. Section 6.3 provides
additional qualitative results including, image retrieval results of our model
compared with MS [61] and Proxy-Anchor [25] losses, and t-SNE visualization of
the learned embedding space on four benchmark datasets [29, 63, 42, 31].

###  6.1 Impact of Batch size

To investigate the impact of the batch size on the performance of our proposed
method, we examined the Recall@1 by training our model on both Cars-196 and
CUB-200-2011 datasets by varying batch sizes from 30 to 120. The result of the
analysis is summarized in Table 4 where it reveals our model is robust in
terms of batch sizes, especially for batch sizes larger or equal to 60.

###  6.2 Second-Order Attention

We visualize the effects of second-order attention in Figure 9. This Figure
exhibits the attention map of samples from Cars-196 and CUB-200-2011 datasets.
For each image, four parts have been selected with different stars and
different colors. The attention map associated with each star has a border
with identical colors. For locations in the background (row 2, col 2 or row 4,
col 3), interestingly, the attention from that feature is distributed within
the main object in the image. It confirms the second-order attention has
learned to focus on the main object in the image to accurately retrieve the
most similar images to the query. On the other hand, when the star is located
within the main object, the attention is on highly distinctive regions(row 1,
col 3 or row 2, col 4).

Table 4: Recall@1 ($\%$) on the Cars-196 and CUB-200-2011 datasets with varying batch sizes from 30 to 120. | Recall@1
---|---
Batch Size | Cars-196 | CUB-200-2011
30 | 90.0 | 68.5
60 | 91.1 | 70.6
90 | 91.7 | 70.7
120 | 91.2 | 70.8

###  6.3 Additional Qualitative Results

More qualitative examples for image retrieval on Cars-196 and CUB-200-2011
datasets are exhibited in Figure 10 and Figure 11, respectively. The results
of our model are compared with the MS loss –from the pairwise-based class– and
Proxy-Anchor –from the proxy-based category– using the same backbone network.
The overall results demonstrate that our model learned better embedding space
than our baselines. In the examples in the 2nd, 3rd, 5th and 6th rows of
Figure 10, the MS loss and proxy anchor loss picked wrongful images while our
hybrid loss retrieved the most similar cars to the query. Also, in Figure 11
MS and Proxy-Anchor loss missed several images, but our approach was able to
retrieve the most similar images to the query. Even in Figure 11, row 1 that
our model wrongfully retrieved the first image, but as one can see, the image
is significantly similar to the query image in terms of appearance.

Based on the qualitative results given in Figures 10 and 11, one can see that
our model produces accurate results. Also, the example in the first row of
Figure 11 shows successful retrievals despite different view-point and color
changes. Figures 12, 13 , 14 and 15 exhibit t-SNE visualizations of the
embedding spaces learned by our hybrid loss on the test sets of the four
benchmark datasets. The results demonstrate that all data points in the
embedding space have relevant nearest neighbors, which suggests that our model
learns a semantic similarity that can be generalized even in the test set.

---
Figure 9: Qualitative examples of SOA maps on the Cars-196 and CUB-200-2011
datasets. Each row depicts the source image and four corresponding SOA maps
obtained for specific spatial locations (marked by stars).
---
Figure 10: Qualitative results on the Cars-196 dataset generated from our
hybrid loss, MS loss from the pairwise-based category, and Proxy-Anchor (PA)
loss from proxy-based losses. For each query image (far most image), we
retrieved the top 4 similar images.
---
Figure 11: Qualitative results on the CUB-200-2011 dataset generated from our
hybrid loss, MS loss from the pairwise-based category, and Proxy-Anchor (PA)
loss from proxy-based losses. For each query image (far most image), we
retrieved the top 4 similar images.
---
Figure 12: t-SNE visualization of our embedding space learned on the test set
of Cars-196 dataset in a grid.
---
Figure 13: t-SNE visualization of our embedding space learned on the test set
of CUB-200-2011 dataset in a grid.
---
Figure 14: t-SNE visualization of our embedding space learned on the test set
of SOP dataset in a grid.
---
Figure 15: t-SNE visualization of our embedding space learned on the gallery
split of Inshop dataset in a grid.


[1]Ido Zuckerman

1]Faculty of Data and Decision Sciences, Technion – Israel Institute of
Technology, Haifa, 3200003, Israel

2]Department of Surgery, University of Wisconsin-Madison School of Medicine
and Public Health, 600 Highland Ave, Madison, 53792, WI, USA

3]Clinical Simulation Program, University of Wisconsin Hospitals and Clinics,
600 Highland Ave, Madison, 53792, WI, USA

# Depth Over RGB: Automatic Evaluation of Open Surgery Skills Using Depth
Camera

<EMAIL_ADDRESS>Nicole Werner  Jonathan Kouchly  Emma Huston
Shannon DiMarco  Paul DiMusto  Shlomi Laufer  [  [  [

###### Abstract

Purpose: In this paper, we present a novel approach to the automatic
evaluation of open surgery skills using depth cameras. This work is intended
to show that depth cameras achieve similar results to RGB cameras, which is
the common method in the automatic evaluation of open surgery skills.
Moreover, depth cameras offer advantages such as robustness to lighting
variations, camera positioning, simplified data compression, and enhanced
privacy, making them a promising alternative to RGB cameras.

Methods: Experts and novice surgeons completed two simulators of open
suturing. We focused on hand and tool detection, and action segmentation in
suturing procedures. YOLOv8 was used for tool detection in RGB and depth
videos. Furthermore, UVAST and MSTCN++ were used for action segmentation. Our
study includes the collection and annotation of a dataset recorded with Azure
Kinect.

Results: We demonstrated that using depth cameras in object detection and
action segmentation achieves comparable results to RGB cameras. Furthermore,
we analyzed 3D hand path length, revealing significant differences between
experts and novice surgeons, emphasizing the potential of depth cameras in
capturing surgical skills. We also investigated the influence of camera angles
on measurement accuracy, highlighting the advantages of 3D cameras in
providing a more accurate representation of hand movements.

Conclusion: Our research contributes to advancing the field of surgical skill
assessment by leveraging depth cameras for more reliable and privacy
evaluations. The findings suggest that depth cameras can be valuable in
assessing surgical skills and provide a foundation for future research in this
area.

###### keywords:

Surgical training, Computer vision, Depth Camera, Object detection, Open
surgery

##  1 Introduction

The complexity and high-stakes nature of open surgery necessitates the
development of reliable and robust systems for evaluating surgical skills [1].
The evaluation of surgical skills has been an active area of research, with
methodologies ranging from subjective assessments by expert surgeons to
objective metrics using sensors and data analytics.

Studies have shown the capability of motion sensors to distinguish between
expert and novice surgeons. For instance, novices tend to move their hands
with less efficiency, resulting in longer path lengths [2]. Additionally, they
exhibit slower movements [3] and employ a more expansive working volume [4].
Unfortunately, hand sensors come with drawbacks such as high costs and
discomfort. Furthermore, their integration into the operating room environment
poses significant challenges.

The combination of RGB cameras and computer vision provides a new approach for
assessing surgical skills. Goldbraikh et al. [5] utilized a standard webcam in
combination with object detection to track hand movements, showing significant
differences between students and experts. This technique paves the way for the
creation of simple and accessible training systems, providing learners with
the opportunity to practice independently and receive objective feedback.
However, motion captured by an RGB camera is the 2D in-plane of the camera
rather than the actual 3D distance. The measurements can vary significantly if
the camera’s angle in relationship to the suture area changes. This study aims
to investigate the potential of depth cameras to address this limitation,
proposing a method that not only resolves this issue but also preserves the
simplicity and accessibility of the training systems.

The use of RGB cameras is not limited to object detection and motion analysis,
In recent years deep learning techniques have been used for general tasks such
as tool detection in laparoscopic surgeries [6] or surgical gesture
recognition [7]. Additionally, other studies have harnessed computer vision to
formulate task-specific performance metrics [8, 9]. Therefore, RGB cameras may
have a broad impact on the quality, efficiency, and safety of surgical
procedures.

Nevertheless, using RGB cameras, especially in a clinical scenario, poses
several challenges. First, privacy concerns emerge due to factors like
capturing facial details and text. Second, lighting in the operating room is
very challenging [10], as there is a wide variation in the amount of light in
different areas [11, 12]. Depth cameras have been suggested as an alternative
to RGB cameras to overcome these issues [13, 14]. They require no contact with
the operating environment while still being capable of accurately tracking
real-life hand motion data. They may be used to measure pose estimation and
gait analysis [15] as well as patient activity recognition [16].

This study introduces an approach that employs depth cameras to automatically
evaluate open surgery skills, specifically focusing on hand and tool detection
and action segmentation in suturing procedures. We show that depth cameras can
achieve comparable results to RGB cameras in a more robust way and provide an
alternative approach for assessing surgical skills. The paper’s main
contributions are : (1) Demonstrating that depth cameras are as effective as
RGB cameras for object detection and action segmentation. (2) Analysing how
the angle between the camera and suture area can affect the accuracy of their
results, thus demonstrating the advantage of depth data. (3) Introducing a
novel metric that relies solely on depth cameras.

##  2 Methods

###  2.1 The Dataset

The study included 28 participants: 22 first-year surgical residents (8 male
and 14 female) and 6 attending surgeons (3 male and 3 female) at a Midwestern
academic hospital. The residents participated in this study as part of an
annual surgical intern simulation series in which all first-year surgical
residents complete a selection of basic surgical skills. Each participant was
informed of the research prior to the session, and their decision to
participate had no influence on the simulation series. One week before the
simulation series, each intern received a video showing a faculty member
accurately performing each skill. There was no limit on the amount of video
views. During the simulation series, each intern is paired with a faculty
member in a room within the hospital simulation center. The intern is then
given standardized written instructions with scoring metrics and asked to
complete each skill using a simulator. After task completion, the faculty
member provides feedback to the resident.

The participants were engaged in conducting various surgical tasks utilizing
two simulators: a “Suture pad” and a “Fascia Closure”. The execution of these
tasks was documented through an Azure Kinect, which features a 4K RGB camera,
a Depth Camera, and an IR Camera.

The first simulator, the “Suture Pad” simulator 2, was made of silicone. It
was constructed to resemble human tissue and allows trainees to practice basic
suturing techniques, such as creating knots and closing incisions. This
simulator is similar to the simulator presented in [17]. In this study,
participants executed four tasks using this simulator: simple suture,
horizontal mattress suture, vertical mattress suture, and running suture. The
goal was to train and assess medical professionals in the technique of
suturing wounds. The initial task averaged 54 seconds, the second task 84
seconds, the third 81 seconds, and the final task approximately 206 seconds.

Figure 1: Suture Pad

Figure 2: Fascia Closure

The second simulator, the “Fascia Closure” simulator 2, simulated the process
of suturing and closing the connective tissue layer called fascia during
surgical procedures. This simulator design is taken from Mayo Clinic’s
Surgical Olympics where it has been used since 2006 [18]. It provides a
simulation of the fascia layer, enabling trainees to practice the skills
required for successful closure. This simulator focuses on running suture in a
distinct type of tissue. This simulator took an average of 379 seconds.

We focused on several computer vision tasks: object detection of the hands and
tools as well as temporal segmentation of the surgical gestures. For object
detection, about 900 frames from simulator 1 and an additional 900 frames from
simulator 2 were annotated. These frames were drawn from 14 participants, who
were chosen randomly, at a rate of one frame every 5 seconds. The number of
participants was limited to 14 due to constraints on labeling resources. In
each frame, all present tools, hands, and the simulator itself were marked
with bounding boxes. The tools identified include Needle Driver, Tissue
Forceps, Dressing Forceps, and Scissors. Additionally, the entire video set
was annotated for temporal segmentation. The catalog of gestures consists of
“Holding needle with a tool”, “Needle passing”, “Pull the suture”,
“Instrumental tie”, “Lay the knot”, “Cut the suture”, “No Gesture”, and “Hand
tie”.

We converted the depth matrix from each frame of the depth camera videos into
grayscale video to better visualize and analyze the spatial information. In
this format, objects nearest to the camera are represented in white, while
those at a greater distance appear black, thereby simplifying the
representation of depth information. The annotations applied to the RGB videos
were also used for these depth videos.

###  2.2 Hardware and Software

We conducted all our experiments, including training, testing, and
evaluations, using a hardware setup consisting of two NVIDIA RTX A6000 GPUs
and a single Intel Core i9-10940X CPU equipped with 28 logical cores. For
running these experiments, we employed the PyTorch library, and for experiment
tracking, we utilized WANDB [19].

###  2.3 Object Detection

For the purpose of object detection, encompassing tools, hands, and the
simulator, the YOLOv8 architecture [20] was employed. This architecture was
trained using the Ultralytics framework. Notably, four distinct models were
trained for each simulator scenario: one for RGB data encompassing all tools,
another for depth data encompassing all tools, a third for RGB data focusing
solely on hands, and a fourth for depth data dedicated to hands. In the hand-
focused models, we had two classes: “Right Hand” and “Left Hand”, While, for
the models aimed at detecting all tools, we used: “Right Hand”, “Left Hand”,
“Needle Driver”, “Tissue Forceps”, “Dressing Forceps”, “Scissors”, and
“Simulator”.

During the training process, several data augmentation techniques were applied
to enhance the model’s robustness, including rotations, and brightness
adjustments.

Modifications were made to the prediction head of the model to accommodate the
aforementioned classes. The evaluation of the model’s performance was carried
out using the mean average precision (mAP) based on intersection over union
(IoU). Finally, the trained model was applied to extract per-frame bounding
boxes, employing a confidence threshold ranging from 0.5 to 0.95, with
increments of 0.05. Every epoch took an estimated one minute to complete, and
the model was trained for a total of 300 epochs. The memory footprint was
about 19GB.

###  2.4 Action Segmentation

In our action segmentation experiments, we employed two architectures: UVAST
[21] and MSTCN++ [22]. These architectures were selected because they
complement each other effectively. MS-TCN++ is a lighter and less complex
model, suitable for online, real-time inference. Conversely, UVAST, being a
more feature-rich and complex model, offers greater accuracy but requires
longer inference times. Both models were implemented using their original
frameworks as described in the cited papers. Both architectures leveraged RGB
and optical flow features, extracted using the I3D model [23] trained on the
Kinetics 400 dataset [24]. Specifically for depth videos, we initially
converted them to greyscale and then adapted them to RGB format by
triplicating each frame for compatibility with the I3D model. In terms of
training, MSTCN++ was trained over 100 epochs, with each epoch averaging
around 5 seconds, while UVAST was trained 600 epochs, each averaging about one
minute. The memory footprint was about 5GB when extracting features and 3GB
for model training and prediction.

The UVAST architecture incorporated the Viterbi algorithm [25] during the
inference stage. For experiments on simulator 2, due to limited resources and
the longer video durations, we limited the hypothesis space to 10,000 during
inference.

In the Suture Pad simulator, the first three tasks of the suturing are closely
related as they all present the execution of what we term a “stationary knot”
— a simple suture, a horizontal mattress suture, and a vertical mattress
suture. Consequently, the model was trained on a unified dataset that included
these tasks, with each task separated into a distinct video. These videos were
then divided into four separate train-test splits, ensuring that all videos
from a single participant fell within the same split. The results for this
model will be labeled as “Simple Suture”. For the fourth task, in the Suture
Pad simulator, the results will be categorized under “Running Suture”, while
results from the Fascia Closure simulator will be designated as “Fascia”.

We used distinct action segmentation labels for each simulator. For the suture
pad simulator, labels included G0 for holding the needle with a tool, G1 for
needle passing, G2 for pulling the suture, G3 for instrumental tie, G4 for
laying the knot, G5 for cutting the suture, and G6 for no action. The fascia
closure simulator employed similar labels, with the addition of G7 for hand
tie.

As previously established in the literature [21, 22], three evaluation metrics
were employed. Frame-wise accuracy, segmental edit distance, and F1@k for
$k\in\\{10,25,50\\}$. Frame-wise Accuracy assesses the ratio of correctly
classified frames to total frames. Segmental Edit Distance, adapted from the
Levenshtein distance, compares activity segments and is normalized by the
greater length between ground truth and prediction. F1@k calculates the
Intersection over Union (IoU) for each segment, categorizing them as true or
false positives based on a threshold $k$.

###  2.5 3D Hand Path Length

According to [26], the 3D hand path length metric is used to evaluate surgical
skills by measuring the efficiency of a surgeon’s movements. Shorter, more
direct paths typically indicate higher skill and experience because they
reflect a surgeon’s ability to perform movements more efficiently and
precisely, making this metric a valuable tool for assessing and improving
surgical proficiency.

To quantify the path length traversed by the hands in a three-dimensional
space, we employed the object detection algorithm 2.3 to identify the hands.
Subsequently, we extracted the coordinates of the bounding box’s center.
Utilizing the depth camera provided by the Azure Kinect, we transformed the
depth information into a point cloud using the Open3D (O3D) library [27]. The
coordinates of the bounding box were then used to extract the [x,y,z]
coordinates from the point cloud, representing each hand’s spatial location.
By aggregating these spatial coordinates across frames, we calculated the
total path length using Euclidean distance metrics. For the statistical
analysis, we adopted the Wilcoxon rank-sum test to compare the total path
lengths between the two groups (experts and residents). The significance level
was set at $p<0.05$.

In our previous work [9], we explored temporal data obtained through action
segmentation tools and examined its correlation with skill. In the current
work, we extend our investigation into spatial data. Specifically, we
introduce a novel metric to quantify the average distance a surgeon’s hands
move during each unique gesture. This approach will facilitate the provision
of more focused practice recommendations, honing in on gestures that require
further refinement.

###  2.6 2D Different Angles

In order to investigate the influence of RGB camera angles on measurement
accuracy, our approach involved analyzing the movement of hands from different
angles, emphasizing how each angle uniquely captures aspects of the movement
in 3D space. This approach underscores our aim to demonstrate the superiority
of depth cameras, which provide 3D imagery, over RGB cameras that offer only
2D perspectives. To accomplish this, we determined the geometric center of
each hand for every frame. We then computed the [x, y, z] coordinates
representing the hand’s spatial position within the simulator’s point cloud, a
methodology previously established in 2.5. These 3D coordinates were then
projected onto three orthogonal 2D planes: XY, YZ, and XZ. This projection
onto 2D planes serves to mimic the limited perspective of RGB cameras. By
comparing these projections, we aim to highlight the constraints of 2D imaging
in capturing the full complexity of hand movements in 3D space.

Subsequent to the projection, we quantified the distances covered by the hand
within these 2D planes as if they were captured by a 2D camera. This
comparison is critical for demonstrating that depth cameras, with their 3D
imaging capabilities, provide a more comprehensive and accurate representation
of hand movements in 3D space than 2D RGB cameras.

##  3 Results

###  3.1 Object Detection

This section presents the results of the YOLOv8 algorithm applied to object
detection. Tables 1 and 2 provide a detailed overview of the algorithm’s
performance, specifically in terms of Average Precision (AP) for each class.
The evaluation was conducted on two distinct models: one trained on RGB video
data and the other on depth video data. These models were rigorously tested on
a separate test set comprising 313 frames for the first simulator and 354
frames for the second from different participants, ensuring that the model’s
performance was evaluated on previously unseen data.

In the first simulator, the suture pad simulator, for the model trained on RGB
video data using all the tools and the hands, we obtained $mAP_{50-95}(RGB)$
of 0.890. Similarly, for the same model trained on depth video data, the
corresponding $mAP_{50-95}(Depth)$ was found to be 0.888. For the models
trained only on the hands, we obtained $mAP_{50-95}(RGB)$ of 0.976 and
$mAP_{50-95}(Depth)$ of 0.963. These results highlight the consistency in the
models’ performance despite being trained on distinct data types, emphasizing
the similar value that the depth camera gives us.

The second simulator, the fascia closure simulator, the model trained on RGB
video data, exhibited performance with a $mAP_{50-95}(RGB)$ of 0.830. The
model trained on depth video data for all the tools and the hands achieved a
$mAP_{50-95}(Depth)$ of 0.801. For the models trained only on the hands, we
obtained $mAP_{50-95}(RGB)$ of 0.945 and $mAP_{50-95}(Depth)$ of 0.966.

| Suture Pad Simulator |  | Fascia Closure Simulator
---|---|---|---
Class | Occurrence | $AP_{50-95}^{RGB}$ | $AP_{50-95}^{Depth}$ |  | Occurrence | $AP_{50-95}^{RGB}$ | $AP_{50-95}^{Depth}$
Left Hand | 316 | 0.967 | 0.964 |  | 352 | 0.935 | 0.937
Right Hand | 306 | 0.953 | 0.942 |  | 332 | 0.944 | 0.973
Needle Driver | 295 | 0.931 | 0.922 |  | 313 | 0.915 | 0.882
Tissue Forceps | 299 | 0.648 | 0.634 |  | 246 | 0.349 | 0.290
Dressing Forceps | 273 | 0.792 | 0.819 |  | 297 | 0.646 | 0.506
Scissors | 298 | 0.932 | 0.927 |  | 287 | 0.816 | 0.814
Simulator | 309 | 0.999 | 0.999 |  | 353 | 0.989 | 0.981
Average | - | 0.890 | 0.888 |  | - | 0.830 | 0.801

Table 1: Suture pad and fascia closure simulators - All Tools and Hands

| Suture Pad Simulator |  | Fascia Closure Simulator
---|---|---|---
Class | Occurrence | $AP_{50-95}^{RGB}$ | $AP_{50-95}^{Depth}$ |  | Occurrence | $AP_{50-95}^{RGB}$ | $AP_{50-95}^{Depth}$
Left Hand | 316 | 0.980 | 0.965 |  | 352 | 0.955 | 0.964
Right Hand | 306 | 0.972 | 0.961 |  | 332 | 0.933 | 0.968
Average | - | 0.976 | 0.963 |  | - | 0.945 | 0.966

Table 2: Suture pad and fascia closure simulators - Only Hands

###  3.2 Action Segmentation

In the case of the Suture Pad simulator, models trained using depth features
outperformed others across all evaluation measures, with the sole exception
being UVAST’s marginally higher edit score in the simple suture task. Using
depth features, UVAST attained an accuracy of $78.22\%$ for the Simple Suture
and $70.97\%$ for the Running Suture. At the same time, MS-TCN++ achieved
accuracies of $76.75\%$ and $66.98\%$ for the same tasks, outperforming their
respective RGB-based versions.

In the case of the Fascia Closure simulator, models trained using RGB showcase
higher results across evaluation metrics, MS-TCN++ achieving an accuracy of
$75.24\%$, and UVAST achieving an accuracy of $71.69\%$. Nonetheless, as
indicated in Table 3, these results remain comparable to those achieved using
depth features.

|  | Simple Suture |  | Running Suture |  | Fascia
---|---|---|---|---|---|---
|  | F1@{10, 25, 50} | Edit | Acc |  | F1@{10, 25, 50} | Edit | Acc |  | F1@{10, 25, 50} | Edit | Acc
RGB | MS-TCN++ | 80.43 77.38 65.29 | 77.78 | 72.40 |  | 66.74 61.89 44.96 | 62.56 | 62.09 |  | 77.53 74.68 62.66 | 72.21 | 75.24
UVAST + Viterbi | 83.84 81.39 67.63 | 81.57 | 74.92 |  | 69.94 65.65 52.04 | 70.26 | 64.73 |  | 73.66 70.41 59.33 | 69.47 | 71.69
Depth | MS-TCN++ | 82.63 80.40 69.56 | 79.13 | 76.75 |  | 71.96 68.02 52.33 | 67.83 | 66.98 |  | 70.22 66.71 53.33 | 63.36 | 67.20
UVAST + Viterbi | 84.19 82.55 71.47 | 79.74 | 78.22 |  | 75.21 72.26 56.69 | 71.80 | 70.97 |  | 66.20 63.67 51.58 | 62.90 | 65.24

Table 3: Suture pad and fascia closure simulators - action segmentation
results

###  3.3 3D Hand Path Length

As we expected, and shown for the 2D case in [5], the box plots in Fig. 3
reveal a consistent pattern across all tasks. Experts consistently navigated a
shorter hand path compared to residents. In the suture pad simulator, for Task
1 - sub-figure 3(a), the p-value of the Wilcoxon rank-sum test was 0.003, for
Task 2 - sub-figure 3(b) it was 0.038, for Task 3 - sub-figure 3(c) the
p-value was 0.021, for Task 4 - sub-figure 3(d) the p-value was 0.021, and for
the fascia closure simulator two - sub-figure 3(e) the p-value was 0.038. All
the p-values are $p<0.05$.

These results indicate that the differences in hand path length between
experts and residents are statistically significant, similar to [28],
underscoring the value of expertise in surgical efficiency that can be
captured using a depth camera. Additionally, it’s noteworthy that despite the
small sample size in our datasets, we were able to achieve statistically
significant p-values. This fact further reinforces the validity of our
results, highlighting the robustness of our findings even with limited data.

(a) Simulator 1 - Task 1

(b) Simulator 1 - Task 2

(c) Simulator 1 - Task 3

(d) Simulator 1 - Task 4

(e) Simulator 2

Figure 3: Task-Based Box-plot of Hand Path Length for Residents and Experts

Fig. 4 serves as an initial exploration based on data collected from the
Simple Suture simulator and gives us a more nuanced look at the hands’ path
length, showing which gestures require the most movement. Our results show a
statistically significant difference in the distance passed when passing the
needle ($p=0.001$), tying a knot ($p<0.001$), laying a knot ($p=0.008$), and
holding the needle ($p=0.021$). No significant difference was found when
pulling the suture ($p=0.707$), cutting the suture ($p=0.056$), and for the
distance moved when no action is performed ($p=0.283$). This offers an initial
validation for our proposed gesture distance metric.

Figure 4: Gesture distance for experts and residents in the Simple Suture task

###  3.4 2D Different Angles

Our data in Table 4 quantitatively confirmed these visual distortions,
revealing a correlation between camera angle and measurement error. We found
that certain angles disproportionately amplified or minimized specific types
of movement, thereby providing an inaccurate representation of the true hand
path. It becomes clear from these results that, because the hand’s path
distance in RGB video is always calculated along a 2D plane, results are
inherently subject to variance due to camera angles. This issue can lead to a
loss of up to a third of the actual, real-world data, as demonstrated in the
XY case under Table 4. This presents an inherent problem when using 2D cameras
for applications that demand high accuracy and reliability.

In contrast, 3D cameras offer a solution to this issue by capturing the real-
world position of the hands in a three-dimensional space, thereby eliminating
the distortions introduced by varying camera angles. This allows for a more
authentic and nuanced understanding of hand movements, as it captures the
complete spatial relationships between different points in the hand’s path.

|  | XYZ |  | XY |  | YZ |  | XZ
---|---|---|---|---|---|---|---|---
|  | Experts | Residents |  | Experts | Residents |  | Experts | Residents |  | Experts | Residents
Suture Pad | Task 1 | 18.52 $\pm$ 3.51 | 29.95 $\pm$ 7.59 |  | 11.00 $\pm$ 1.65 | 17.77 $\pm$ 5.28 |  | 16.06 $\pm$ 3.20 | 26.30 $\pm$ 6.88 |  | 17.43 $\pm$ 3.18 | 28.11 $\pm$ 7.16
Task 2 | 32.97 $\pm$ 6.76 | 45.94 $\pm$ 11.86 |  | 19.36 $\pm$ 4.07 | 27.69 $\pm$ 9.18 |  | 28.77 $\pm$ 6.03 | 40.17 $\pm$ 10.71 |  | 31.05 $\pm$ 6.41 | 43.00 $\pm$ 11.04
Task 3 | 33.39 $\pm$ 3.88 | 45.70 $\pm$ 11.98 |  | 19.38 $\pm$ 2.43 | 27.41 $\pm$ 8.04 |  | 29.41 $\pm$ 3.78 | 40.28 $\pm$ 10.79 |  | 31.53 $\pm$ 3.53 | 42.45 $\pm$ 11.09
Task 4 | 61.72 $\pm$ 15.71 | 90.20 $\pm$ 26.10 |  | 57.83 $\pm$ 15.01 | 85.13 $\pm$ 23.85 |  | 19.38 $\pm$ 2.43 | 27.41 $\pm$ 8.04 |  | 29.41 $\pm$ 3.78 | 40.28 $\pm$ 10.79
Fascia Closure |  | 119.38 $\pm$ 27.64 | 180.71 $\pm$ 52.59 |  | 74.39 $\pm$ 15.81 | 125.18$\pm$ 37.06 |  | 102.37 $\pm$ 22.92 | 154.63 $\pm$ 44.04 |  | 111.61$\pm$25.53 | 169.57$\pm$48.52

Table 4: Hand Path Length in the Planes XY, YZ and XZ (Meter) divided to
experts and residents

##  4 Discussion and Conclusion

Depth cameras have several advantages over traditional RGB cameras. These
advantages include resilience to lighting and positional variations,
simplified data compression, and enhanced privacy. In this study, we assessed
if they provide a viable and competent alternative to RGB cameras for tasks
like object detection and action segmentation in surgical environments.

Analysis of motion using depth data revealed statistically significant
differences between experts and residents. Additionally, our study into the
effects of camera angles on measurement accuracy indicates that depth cameras
are less susceptible to variations in camera position and setup. Consequently,
readily available and affordable depth cameras could offer a new, accessible
approach for self-directed and independent training, coupled with objective
feedback.

It is essential to acknowledge the limitations of our study. The data set size
is crucial when using deep learning tools and statistical tests. More accurate
results could have been achieved with the availability of a more extensive
dataset. Due to the high memory demands of the Viterbi algorithm, we had to
use a simplified version. This is a balanced approach and a common solution in
applications involving the Viterbi algorithm. While the full algorithm might
provide slightly more precision in certain cases, the simplified version
aligned well with our research requirements. Also, while a larger dataset
could potentially offer finer details, rigorous methods were employed to
ensure the validity of our study given the available data.

In conclusion, our research contributes to the field of surgical skill
assessment. By championing the adoption of depth cameras, we provide a more
accurate, privacy-conscious, and robust approach to evaluating surgical
proficiency. The advantages of depth cameras, combined with our empirical
findings, underscore their potential to alter how surgical skills are assessed
and trained, offering a solid foundation for future advancements in this
domain.

Acknowledgments This research was funded by a grant from the University of
Wisconsin-Madison Department of Surgery. The authors would like to thank Abbie
Schafer for assisting in data annotation.

## Declarations

Conflict of interest The authors declare that they have no conflict of
interest.

Ethical approval This study was granted an exemption by the University of
Wisconsin-Madison Institutional Review Board.

Informed consent Informed consent was obtained from all individual
participants included in the study.

## References

  *   * Reznick and MacRae [2006] Reznick, R.K., MacRae, H.: Teaching surgical skills—changes in the wind. New England Journal of Medicine 355(25), 2664–2669 (2006)
  * Dosis et al. [2005] Dosis, A., Aggarwal, R., Bello, F., Moorthy, K., Munz, Y., Gillies, D., Darzi, A.: Synchronized video and motion analysis for the assessment of procedures in the operating theater. Archives of Surgery 140(3), 293–299 (2005)
  * Smith et al. [2002] Smith, S., Torkington, J., Brown, T., Taffinder, N., Darzi, A.: Motion analysis: a tool for assessing laparoscopic dexterity in the performance of a laboratory-based laparoscopic cholecystectomy. Surgical endoscopy 16, 640–645 (2002)
  * D’Angelo et al. [2016] D’Angelo, A.-L.D., Rutherford, D.N., Ray, R.D., Laufer, S., Mason, A., Pugh, C.M.: Working volume: validity evidence for a motion-based metric of surgical efficiency. The American Journal of Surgery 211(2), 445–450 (2016)
  * Goldbraikh et al. [2022] Goldbraikh, A., D’Angelo, A.-L., Pugh, C.M., Laufer, S.: Video-based fully automatic assessment of open surgery suturing skills. International Journal of Computer Assisted Radiology and Surgery 17(3), 437–448 (2022)
  * Fathabadi et al. [2021] Fathabadi, F.R., Grantner, J.L., Shebrain, S.A., Abdel-Qader, I.: Multi-class detection of laparoscopic instruments for the intelligent box-trainer system using faster r-cnn architecture. In: 2021 IEEE 19th World Symposium on Applied Machine Intelligence and Informatics (SAMI), pp. 000149–000154 (2021). IEEE
  * Goldbraikh et al. [2022] Goldbraikh, A., Avisdris, N., Pugh, C.M., Laufer, S.: Bounded future ms-tcn++ for surgical gesture recognition. In: European Conference on Computer Vision, pp. 406–421 (2022). Springer
  * Halperin et al. [2023] Halperin, L., Sroka, G., Zuckerman, I., Laufer, S.: Automatic performance evaluation of the intracorporeal suture exercise. International Journal of Computer Assisted Radiology and Surgery, 1–4 (2023)
  * Bkheet et al. [2023] Bkheet, E., D’Angelo, A.-L., Goldbraikh, A., Laufer, S.: Using hand pose estimation to automate open surgery training feedback. International Journal of Computer Assisted Radiology and Surgery, 1–7 (2023)
  * Dascalaki et al. [2009] Dascalaki, E.G., Gaglia, A.G., Balaras, C.A., Lagoudi, A.: Indoor environmental quality in hellenic hospital operating rooms. Energy and Buildings 41(5), 551–560 (2009) https://doi.org/10.1016/j.enbuild.2008.11.023
  * Likitlersuang et al. [2017] Likitlersuang, J., Sumitro, E.R., Theventhiran, P., Kalsi-Ryan, S., Zariffa, J.: Views of individuals with spinal cord injury on the use of wearable cameras to monitor upper limb function in the home and community. The journal of spinal cord medicine 40(6), 706–714 (2017)
  * Haque et al. [2020] Haque, A., Milstein, A., Fei-Fei, L.: Illuminating the dark spaces of healthcare with ambient intelligence. Nature 585(7824), 193–202 (2020)
  * Sun et al. [2023] Sun, Z., Ke, Q., Rahmani, H., Bennamoun, M., Wang, G., Liu, J.: Human action recognition from various data modalities: A review. IEEE Transactions on Pattern Analysis and Machine Intelligence 45(3), 3200–3225 (2023) https://doi.org/10.1109/TPAMI.2022.3183112
  * Yeung et al. [2019] Yeung, S., Rinaldo, F., Jopling, J., Liu, B., Mehra, R., Downing, N.L., Guo, M., Bianconi, G.M., Alahi, A., Lee, J., et al.: A computer vision system for deep learning-based detection of patient mobilization activities in the icu. NPJ digital medicine 2(1), 11 (2019)
  * Martinez-Martin et al. [2021] Martinez-Martin, N., Luo, Z., Kaushal, A., Adeli, E., Haque, A., Kelly, S.S., Wieten, S., Cho, M.K., Magnus, D., Fei-Fei, L., et al.: Ethical issues in using ambient intelligence in health-care settings. The lancet digital health 3(2), 115–123 (2021)
  * Siddiqi et al. [2021] Siddiqi, M.H., Almashfi, N., Ali, A., Alruwaili, M., Alhwaiti, Y., Alanazi, S., Kamruzzaman, M.: A unified approach for patient activity recognition in healthcare using depth camera. IEEE Access 9, 92300–92317 (2021)
  * Williams et al. [2020] Williams, T.P., Snyder, C.L., Hancock, K.J., Iglesias, N.J., Sommerhalder, C., DeLao, S.C., Chacin, A.C., Perez, A.: Development of a low-cost, high-fidelity skin model for suturing. Journal of Surgical Research 256, 618–622 (2020)
  * Buckarma et al. [2016] Buckarma, E., et al.: The How To Book of Low Cost Surgical Simulation, (2016). https://surgicaleducation.mayo.edu/how-to-book/
  * Biewald [2020] Biewald, L.: Experiment Tracking with Weights and Biases. Software available from wandb.com (2020). https://www.wandb.com/
  * Jocher et al. [2023] Jocher, G., Chaurasia, A., Qiu, J.: YOLO by Ultralytics. https://github.com/ultralytics/ultralytics
  * Behrmann et al. [2022] Behrmann, N., Golestaneh, S.A., Kolter, Z., Gall, J., Noroozi, M.: Unified fully and timestamp supervised temporal action segmentation via sequence to sequence translation. In: European Conference on Computer Vision, pp. 52–68 (2022). Springer
  * Li et al. [2020] Li, S.-J., AbuFarha, Y., Liu, Y., Cheng, M.-M., Gall, J.: Ms-tcn++: Multi-stage temporal convolutional network for action segmentation. IEEE transactions on pattern analysis and machine intelligence (2020)
  * Carreira and Zisserman [2017] Carreira, J., Zisserman, A.: Quo vadis, action recognition? a new model and the kinetics dataset. In: Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6299–6308 (2017)
  * Kay et al. [2017] Kay, W., Carreira, J., Simonyan, K., Zhang, B., Hillier, C., Vijayanarasimhan, S., Viola, F., Green, T., Back, T., Natsev, P., et al.: The kinetics human action video dataset. arXiv preprint arXiv:1705.06950 (2017)
  * Forney [1973] Forney, G.D.: The viterbi algorithm. Proceedings of the IEEE 61(3), 268–278 (1973)
  * Chmarra et al. [2008] Chmarra, M.K., Jansen, F.W., Grimbergen, C.A., Dankelman, J.: Retracting and seeking movements during laparoscopic goal-oriented movements. is the shortest path length optimal? Surgical endoscopy 22, 943–949 (2008)
  * Zhou et al. [2018] Zhou, Q.-Y., Park, J., Koltun, V.: Open3D: A modern library for 3D data processing. arXiv:1801.09847 (2018)
  * Lefor et al. [2020] Lefor, A.K., Harada, K., Dosis, A., Mitsuishi, M.: Motion analysis of the jhu-isi gesture and skill assessment working set using robotics video and motion assessment software. International Journal of Computer Assisted Radiology and Surgery 15, 2017–2025 (2020)