# Towards Representation Alignment and Uniformity in Collaborative Filtering

Chenyang Wang  DCST, BNRist, Tsinghua UniversityBeijing 100084China
<EMAIL_ADDRESS>,  Yuanqing Yu  DCST, BNRist, Tsinghua
UniversityBeijing 100084China<EMAIL_ADDRESS>,  Weizhi Ma  AIR,
Tsinghua UniversityBeijing 100084China<EMAIL_ADDRESS>,  Min Zhang
DCST, BNRist, Tsinghua UniversityBeijing 100084China<EMAIL_ADDRESS>,
Chong Chen  DCST, BNRist, Tsinghua UniversityBeijing 100084China
<EMAIL_ADDRESS>,  Yiqun Liu  DCST, BNRist, Tsinghua
UniversityBeijing 100084China<EMAIL_ADDRESS>and  Shaoping Ma  DCST,
BNRist, Tsinghua UniversityBeijing 100084China<EMAIL_ADDRESS>
(2022)

###### Abstract.

Collaborative filtering (CF) plays a critical role in the development of
recommender systems. Most CF methods utilize an encoder to embed users and
items into the same representation space, and the Bayesian personalized
ranking (BPR) loss is usually adopted as the objective function to learn
informative encoders. Existing studies mainly focus on designing more powerful
encoders (e.g., graph neural network) to learn better representations.
However, few efforts have been devoted to investigating the desired properties
of representations in CF, which is important to understand the rationale of
existing CF methods and design new learning objectives. In this paper, we
measure the representation quality in CF from the perspective of alignment and
uniformity on the hypersphere. We first theoretically reveal the connection
between the BPR loss and these two properties. Then, we empirically analyze
the learning dynamics of typical CF methods in terms of quantified alignment
and uniformity, which shows that better alignment or uniformity both
contribute to higher recommendation performance. Based on the analyses
results, a learning objective that directly optimizes these two properties is
proposed, named DirectAU. We conduct extensive experiments on three public
datasets, and the proposed learning framework with a simple matrix
factorization model leads to significant performance improvements compared to
state-of-the-art CF methods. Our implementations are publicly
available111https://github.com/THUwangcy/DirectAU.

Recommender Systems, Collaborative Filtering, Representation Learning,
Alignment and Uniformity

††journalyear: 2022††copyright: rightsretained††conference: Proceedings of the
28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining; August
14–18, 2022; Washington, DC, USA††booktitle: Proceedings of the 28th ACM
SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22), August
14–18, 2022, Washington, DC, USA††isbn: 978-1-4503-9385-0/22/08††doi:
10.1145/3534678.3539253††ccs: Information systems Recommender
systems††*Corresponding author.

##  1\. Introduction

Recommender system has become an essential part of users’ engagements with web
services, such as product recommendation (McAuley et al., 2015), video
recommendation (Covington et al., 2016), and so on. To help users discover
potential items of interests, collaborative filtering (CF) is widely adopted
in personalized recommendation (Schafer et al., 2007). The core idea of CF is
that similar users tend to have similar preferences. Compared to content-based
recommendation methods, CF only relies on past user behaviors to predict
users’ preferences on candidate items. The simplicity and effectiveness of CF
make it a canonical technique in recommender systems (Su and Khoshgoftaar,
2009).

Most CF methods utilize an encoder to embed users and items to a shared space
and then optimize an objective function to learn informative user and item
representations (Mao et al., 2021). The simplest encoder can be an embedding
table that directly maps user and item IDs to embeddings (Koren et al., 2009),
and Bayesian personalized ranking (BPR) (Rendle et al., 2009) is usually
adopted as the objective function to discriminate between positive
interactions and unobserved ones. Existing studies about CF mainly focus on
designing more powerful encoders to model complex collaborative signals
between users and items. Specifically, neural-based interaction encoders
emerge in recent years, such as multi-layer perceptron (MLP) (He et al.,
2017), attention mechanism (Chen et al., 2017), graph neural network (GNN)
(Wang et al., 2019a; He et al., 2020), and so on. Meanwhile, some recent works
point out that the nowadays complex encoders in CF actually lead to marginal
performance improvements (Mao et al., 2021). As a result, researchers also
begin to investigate other objective functions beyond the common pairwise BPR
loss (e.g., InfoNCE loss (Zhou et al., 2021), cosine contrastive loss (Mao et
al., 2021)), which have been shown to bring more robust improvements than
complex encoders.

However, few research efforts have been devoted to investigating the desired
properties of user and item representations derived by the encoder. This is
important to justify the rationale behind existing CF methods and design new
learning objectives that favor these properties. Intuitively, representations
of positive-related user-item pairs should be close to each other, and each
representation should preserve as much information about the user/item itself
as possible. Assuming all the representations are $l_{2}$ normalized, these
two properties can be referred to as 1) alignment and 2) uniformity on the
unit hypersphere (Wang and Isola, 2020). To learn informative user and item
representations, both alignment and uniformity are of great importance. If
only alignment is considered, perfectly aligned encoders are easy to be
achieved by mapping all the users and items to the same embedding. The goal of
existing loss functions in CF can be seen to avoid such trivial constants
(i.e., preserving uniformity) while optimizing for better alignment. In
practice, negative samples are usually utilized to achieve this goal. For
example, the BPR loss (Rendle et al., 2009) pairs each positive interaction
with a randomly sampled negative item, and the predicted score of the
interacted item is encouraged to be higher than the negative one.

In this work, we analyze the alignment and uniformity properties in CF
inspired by recent progress in contrastive representation learning (Wang and
Isola, 2020; Gao et al., 2021). We first theoretically show that the BPR loss
actually favors these two properties, and perfectly aligned and uniform
encoders form the exact minimizers of the BPR loss. Then, we empirically
analyze the learning dynamics of typical CF methods in terms of alignment and
uniformity via corresponding quantifying metrics proposed in (Wang and Isola,
2020). We find different CF methods demonstrate distinct learning
trajectories, and either better alignment or better uniformity benefits the
representation quality. For instance, the simplest BPR quickly converges to
promising alignment and mainly improves uniformity afterwards. Other advanced
methods achieve better alignment or uniformity via various techniques, such as
hard negative samples and graph-based encoders, which lead to better
performance accordingly. Based on the analyses results, we propose a learning
objective that directly optimizes these two properties, named DirectAU.
Extensive experiments are conducted on three public real-world datasets.
Experimental results show that a simple matrix factorization based encoder
(i.e., embedding table) that optimizes the proposed DirectAU loss yields
remarkable improvements (up to 14%) compared to state-of-the-art CF methods.

The main contributions of this work can be summarized as follows:

  * •

We theoretically show that perfectly aligned and uniform encoders form the
exact minimizers of the BPR loss. We also empirically analyze the learning
dynamics of typical CF methods in terms of quantified alignment and
uniformity.

  * •

Based on the analyses results, a simple but effective learning objective that
directly optimizes these two properties is proposed, named DirectAU.

  * •

Extensive experiments on three public datasets show that the proposed DirectAU
well balances between alignment and uniformity. When optimizing the DirectAU
objective, even the simplest matrix factorization based encoder leads to
significant performance improvements compared to state-of-the-art CF methods.

##  2\. Preliminaries

In this section, we first formulate the collaborative filtering problem. Then
we introduce how to measure alignment and uniformity based on recent progress
in self-supervised learning (Wang and Isola, 2020).

###  2.1. Collaborative Filtering

Let $\mathcal{U}$ and $\mathcal{I}$ denote the user and item set,
respectively. Given a set of observed user-item interactions
$\mathcal{R}=\\{(u,i)~{}|~{}u~{}{\rm interacted~{}with}~{}i\\}$, CF methods
aim to infer the score $s(u,i)\in\mathbb{R}$ for each unobserved user-item
pair indicating how likely the user $u$ tends to interact with the item $i$.
Then, items with the highest scores for each user will be recommended based on
the predictions.

In general, most CF methods use an encoder network $f(\cdot)$ that maps each
user and item into a low-dimensional representation
$f(u),f(i)\in\mathbb{R}^{d}$ ($d$ is the dimension of the latent space). For
example, the encoder in matrix factorization models is usually an embedding
table, which directly maps each user and item to a latent vector based on
their IDs. The encoder in graph-based models further utilizes the neighborhood
information. Then, the predicted score is defined as the similarity between
the user and item representation (e.g., dot product, $s(u,i)=f(u)^{T}f(i)$).
As for the learning objective, most studies adopt the pairwise BPR (Rendle et
al., 2009) loss to train the model:

(1)
$\mathcal{L}_{BPR}=\dfrac{1}{|\mathcal{R}|}\sum_{(u,i)\in\mathcal{R}}-\log\left[{\rm
sigmoid}\left(s(u,i)-s(u,i^{-})\right)\right],$

where $i^{-}$ is a randomly sampled negative item that the user has not
interacted with. This loss function aims to optimize the probability that the
target item gets a higher score than random negative items.

###  2.2. Alignment and Uniformity

Recent studies (Wang and Isola, 2020; Gao et al., 2021) in unsupervised
contrastive representation learning identify that the quality of
representations is highly related to two key properties, i.e., alignment and
uniformity. Given the distribution of data $p_{\rm data}(\cdot)$ and the
distribution of positive pairs $p_{\rm pos}(\cdot,\cdot)$, alignment is
straightforwardly defined as the expected distance between normalized
embeddings of positive pairs:

(2) $l_{\rm align}\triangleq\mathop{\mathbb{E}}_{(x,x^{+})\sim p_{\rm
pos}}||\tilde{f(x)}-\tilde{f(x^{+})}||^{2},$

where $\tilde{f(\cdot)}$ indicates $l_{2}$ normalized representations. On the
other hand, the uniformity loss is defined as the logarithm of the average
pairwise Gaussian potential:

(3) $l_{\rm uniform}\triangleq\log\mathop{\mathbb{E}}_{x,y\sim p_{\rm
data}}e^{-2||\tilde{f(x)}-\tilde{f(y)}||^{2}}.$

These two metrics are well aligned with the objective of representation
learning: positive instances should be close to each other while random
instances should scatter on the hypersphere. In this work, we will connect the
BPR loss with these two metrics and use them to analyze the learning dynamics
of typical CF methods.

##  3\. Alignment and Uniformity in Collaborative Filtering

In this section, we first theoretically show that the BPR loss favors
representation alignment and uniformity on the hypersphere. Then, we
empirically observe how these two properties evolve during training for
different CF methods.

###  3.1. Theoretical Analyses

Assuming the distribution of positive user-item pairs is $p_{\rm pos}$, and
the distribution of users and items is denoted as $p_{\rm user}$ and $p_{\rm
item}$ respectively, we first define the notion of optimality for alignment
and uniformity in CF as follows:

###### Definition 0 (Perfect Alignment).

An encoder $f$ is perfectly aligned if $\tilde{f(u)}=\tilde{f(i)}$ a.s. over
$(u,i)\sim p_{\rm pos}$.

###### Definition 0 (Perfect Uniformity).

An encoder $f$ is perfectly uniform if the distribution of $\tilde{f(u)}$ for
$u\sim p_{\rm user}$ and the distribution of $\tilde{f(i)}$ for $i\sim p_{\rm
item}$ are the uniform distribution $\sigma_{d-1}$ on $\mathcal{S}^{d-1}$.

Here $\mathcal{S}^{d-1}=\\{x\in\mathbb{R}^{d}:||x||=1\\}$ is the surface of
the $d$-dimensional unit ball. Note that perfectly aligned encoders can be
easily achieved by mapping all the inputs to the same representation, at the
cost of the worst uniformity. Perfectly uniform encoders can also be achieved
considering the number of users/items is usually large and $d$ is small in
real-world applications. The following theorem shows that the BPR loss favors
these two properties if perfect alignment and uniformity are realizable.

###### Theorem 3.

If perfectly aligned and uniform encoders exist, they form the exact
minimizers of the BPR loss $\mathcal{L}_{BPR}$.

###### Proof.

Assuming the similarity function $s(u,i)$ is cosine similarity (user/item
representations are normalized), we have

$\displaystyle\mathcal{L}_{BPR}$ $\displaystyle=\mathop{\mathbb{E}}_{(u,i)\sim
p_{\rm pos}}-\log{\rm sigmoid}\left(s(u,i)-s(u,i^{-})\right)$
$\displaystyle=\mathop{\mathbb{E}}_{(u,i)\sim p_{\rm
pos}}-\log\left(\dfrac{e^{\tilde{f(u)}^{T}\tilde{f(i)}}}{e^{\tilde{f(u)}^{T}\tilde{f(i)}}+e^{\tilde{f(u)}^{T}\tilde{f(i^{-})}}}\right)$
$\displaystyle=\mathop{\mathbb{E}}_{(u,i)\sim p_{\rm
pos}}-\tilde{f(u)}^{T}\tilde{f(i)}+\log\left(e^{\tilde{f(u)}^{T}\tilde{f(i)}}+e^{\tilde{f(u)}^{T}\tilde{f(i^{-})}}\right)$
(4) $\displaystyle\geq\mathop{\mathbb{E}}_{(u,i)\sim p_{\rm
pos}}\left[-1+\log\left(e^{1}+e^{\tilde{f(u)}^{T}\tilde{f(i^{-})}}\right)\right]$
(5)
$\displaystyle\geq-1+\int_{\mathcal{S}^{d-1}}\int_{\mathcal{S}^{d-1}}\log\left(e+e^{x^{T}y}\right){\rm
d}\sigma_{d-1}(x){\rm d}\sigma_{d-1}(y).$

According to the definition of perfect alignment, the equality in Equation (4)
is satisfied if and only if $f$ is perfectly aligned. According to Lemma 2 in
(Wang and Isola, 2020), Equation (5) is satisfied if and only if the feature
distribution induced by $f$ is $\sigma_{d-1}$ ($f$ is perfectly uniform).
Therefore, $\mathcal{L}_{BPR}\geq$ a constant independent of $f$, where
equality is satisfied if and only if $f$ is perfectly aligned and uniform. ∎

Considering the quantified metrics in Section 2.2 have been shown to be well
aligned with perfect alignment and uniformity (Wang and Isola, 2020), this
theorem shows that the BPR loss indeed favors lower $l_{\rm align}$ and
$l_{\rm uniform}$. Next, we will empirically show the learning dynamics of
different CF methods in terms of alignment and uniformity.

Figure 1. The trends of $l_{\rm align}$ and $l_{\rm uniform}$ during training
(left) and the learning curve (right) when optimizing the BPR loss on the
Beauty dataset.

###  3.2. Empirical Observations

We use the BPR loss to train a matrix factorization (MF) model on the Beauty
dataset222More information about the dataset and metrics will be detailed in
Section 5.1.. The encoder here is a simple embedding table that maps IDs to
embeddings. Figure 1 shows how these two properties333Calculations of
alignment and uniformity losses in CF will be detailed in Appendix., the BPR
loss, and the recommendation performance (NDCG@20), change during training.
First, we find the randomly initialized encoder is poorly aligned but well
uniform (the initial uniformity loss is low). With the optimization of the BPR
loss, the alignment loss decreases quickly and results in the increase of the
uniformity loss. As the alignment loss becomes stable, the uniformity loss
begins to decrease. Overall, the recommendation performance improves as better
alignment and uniformity are achieved. This empirically validates the analyses
in Section 3.1 that the BPR loss indeed optimizes for lower $l_{\rm align}$
and $l_{\rm uniform}$.

Besides the simplest MF encoder with the BPR loss (BPRMF), different CF
methods may have distinct learning trajectories. We further visualize the
alignment and uniformity metrics every epoch444The start point of each method
is the status after 5 epochs of training. We do not draw the first few points
because they are far away from the main area. for 4 typical CF methods on
Beauty, as shown in Figure 2. BPRMF denotes the simplest MF encoder with the
BPR loss as mentioned above. BPR-DS (Rendle and Freudenthaler, 2014) enhances
BPRMF by adopting a dynamic negative sampling strategy that makes the sampling
probability proportional to the predicted score. LGCN (He et al., 2020)
utilizes graph neural network (GNN) as the encoder and uses the standard BPR
training strategy. ENMF (Chen et al., 2020) leverages all the negative
interactions and devises an efficient approach to optimize the mean squared
error (MSE) loss. The stars in Figure 2 indicate the converged points of
different models, and we annotate NDCG@20 in parentheses. We mainly have the
following observations:

  * •

The optimization of BPR focuses more on uniformity (discriminating between
positive and negative interactions) but does not continuously pushes positive
user-item pairs closer.

  * •

BPR-DS samples more difficult negative items, and hence leads to lower
uniformity loss and better performance. But hard negatives also make it
difficult to align positive user-item pairs (higher alignment loss).

  * •

LGCN aggregates the neighborhood information and hence achieves superior
alignment even in the beginning. This explains why LGCN generally performs
well with the BPR loss. The GNN encoder structure is good at alignment, while
the BPR loss does well on uniformity. Although the training procedure hurts
alignment and the final uniformity is worse than BPRMF, the ending alignment
is still remarkable, which leads to better performance accordingly.

  * •

Different from the above pairwise methods, ENMF directly optimizes MSE and
leverages all the negative interactions, which pushes the scores of positive
user-item pairs to 1 but not just greater than negative pairs like BPR. This
whole-data based training benefits the optimization of alignment to a large
extent while maintaining promising uniformity, and hence yields superior
performance. But such pointwise optimization also hurts uniformity at the
later training stage.

Figure 2. $l_{\rm align}$-$l_{\rm uniform}$ plot for different CF methods
during training. We visualize these two metrics every epoch, and the stars
indicate the converged points. We also annotate NDCG@20 for each model in
parentheses (higher numbers are better). For $l_{\rm align}$ and $l_{\rm
uniform}$, lower numbers are better.

According to the above observations, we find different CF methods have
distinct learning dynamics in terms of alignment and uniformity. Compared to
the standard BPRMF, BPR-DS is better at uniformity but leads to worse
alignment; LGCN is better at alignment but yields worse uniformity, while both
BPR-DS and LGCN achieve higher recommendation performance than BPRMF. ENMF
further gets the best performance with both promising alignment and
uniformity. This shows that user and item representations in CF indeed favor
these two properties. Achieving better alignment or uniformity both contribute
to higher recommendation performance, and it can be beneficial to optimize
them simultaneously.

##  4\. Directly Optimizing Alignment and Uniformity (DirectAU)

Figure 3. Overview of the proposed DirectAU. We directly optimize 1)
representation alignment for positive user-item pairs and 2) in-batch
uniformity for users/items.

The above analyses demonstrate that both alignment and uniformity are
essential to learn informative user and item representations. This motivates
us to design a new learning objective that directly optimizes these two
properties to achieve better recommendation performance, named DirectAU.

Figure 1 illustrates the overall structure of the proposed framework. The
input positive user-item pairs are first encoded to embeddings and
l2-normalized to the hypersphere. We use a simple embedding table (mapping
user/item IDs to embeddings) as the default encoder555Combinations with other
encoders like graph neural networks will be tested in Section 5.4, and we find
a simple embedding table yields remarkable performance.. Then, we quantify
alignment and uniformity in CF as follows:

(6) $\begin{split}l_{\rm align}=&\mathop{\mathbb{E}}_{(u,i)\sim p_{\rm
pos}}||\tilde{f(u)}-\tilde{f(i)}||^{2}\\\ l_{\rm
uniform}=&~{}\log\mathop{\mathbb{E}}_{u,u^{\prime}\sim p_{\rm
user}}e^{-2||\tilde{f(u)}-\tilde{f(u^{\prime})}||^{2}}/~{}2~{}+\\\
&~{}\log\mathop{\mathbb{E}}_{i,i^{\prime}\sim p_{\rm
item}}e^{-2||\tilde{f(i)}-\tilde{f(i^{\prime})}||^{2}}/~{}2.\end{split}$

The alignment loss pushes up the similarity between representations of
positive-related user-item pairs, while the uniformity loss measures how well
the representations scatter on the hypersphere. We separately calculate the
uniformity within user representations and item representations because the
data distribution of user and item might be diverse, which is more suitable to
be measured respectively. Finally, we jointly optimize these two objectives
with a trade-off hyperparameter $\gamma$:

(7) $\mathcal{L}_{\rm DirectAU}=l_{\rm align}+\gamma l_{\rm uniform}.$

The weight $\gamma$ controls the desired degree of uniformity, which is
dependent on the characteristic of each dataset. The learning algorithm of
DirectAU can be found in Appendix.

Note that previous CF methods usually rely on negative sampling to
discriminate between positive and negative interactions, while DirectAU does
not need additional negative samples and only uses the input batch of positive
user-item pairs. The uniformity loss is calculated based on the in-batch
pairwise distances between representations. Using in-batch instances makes it
more consistent with the actual data distribution of users and items (i.e.,
$p_{\rm user},p_{\rm item}$), which has been shown to help reduce exposure
bias in recommender systems (Zhou et al., 2021). Compared to existing CF
methods, DirectAU is easy to implement in the absence of negative samples, and
there is only one hyper-parameter to tune (no need to consider the number of
negative samples the sampling strategy). This makes DirectAU easy to work with
various application scenarios. As for the score function, we use the dot
product between user and item representations to calculate ranking scores and
make recommendations, which is common in the literature (Wang et al., 2020c;
He et al., 2017, 2020).

##  5\. Experiments

In this section, we conduct extensive experiments on three public datasets to
validate the effectiveness of DirectAU. We first describe the experimental
settings (Section 5.1) and compare the overall top-K recommendation
performance of DirectAU with other state-of-the-art CF methods (Section 5.2).
Then, we show the learning curves when only optimizing alignment or uniformity
to verify the importance of both properties (Section 5.3). We also investigate
the performance of DirectAU when integrated with other CF encoders (Section
5.4). Finally, we provide the efficiency analyses (Section 5.5) and parameter
sensitivity of DirectAU (Section 5.6).

###  5.1. Experimental Settings

####  5.1.1. Datasets

We use three public datasets in real-world scenarios. All the datasets are
publicly available and widely adopted in previous studies (Wang et al., 2019a;
He et al., 2020; Wang et al., 2020b, 2022).

  * •

Beauty666https://jmcauley.ucsd.edu/data/amazon/links.html: This is one of the
series of product review datasets crawled from Amazon. The data is split into
separate datasets by the top-level product category.

  * •

Gowalla777http://snap.stanford.edu/data/loc-gowalla.html: This is a check-in
dataset (Liang et al., 2016) obtained from Gowalla, where users share their
locations by checking-in.

  * •

Yelp2018888https://www.yelp.com/dataset: This is a business recommendation
dataset, including restaurants, bars and so on. We use the transaction records
after Jan. 1st, 2018 following previous work (Wang et al., 2019a; He et al.,
2020).

For preprocessing the datasets, we remove repeated interactions and ensure
each user and item to have at least 5 associated interactions. This strategy
is also widely adopted in previous work (Lee et al., 2021; Wang et al.,
2019b). The statistics of datasets after preprocessing are summarized in Table
1.

####  5.1.2. Baselines

We compare the performance of DirectAU with various state-of-the-art CF
methods:

  * •

BPRMF (Rendle et al., 2009): This is a typical negative-sampling method that
optimizes MF with a pairwise ranking loss, where the negative item is randomly
sampled from the item set.

  * •

BPR-DS (Rendle and Freudenthaler, 2014): This method enhances BPRMF by
adopting the dynamic sampling strategy, where negative items with higher
prediction scores are more likely to be sampled.

  * •

ENMF (Chen et al., 2020): This is a MF-based model that uses all the
unobserved interactions as negative samples without negative sampling. An
efficient learning algorithm that minimizes the MSE loss is introduced to
learn from the whole data.

  * •

RecVAE (Shenbin et al., 2020): This method is based on the variational
autoencoder that reconstructs partially-observed user vectors, which
introduces several techniques to improve M-VAE (Liang et al., 2018).

  * •

LGCN (He et al., 2020): This is a simplified graph convolution network for CF
that performs linear propagation between neighbors on the user-item bipartite
graph.

  * •

DGCF (Wang et al., 2020a): This is a state-of-the-art GNN-based method that
introduces disentanglement on top of LGCN, which models the intent-aware
interaction graphs and encourages independence of different intents.

  * •

BUIR (Lee et al., 2021): This is a state-of-the-art negative-sample-free CF
method that learns user and item embeddings with only positive interactions.

  * •

CLRec (Zhou et al., 2021): This is a recently proposed method based on
contrastive learning, which adopts the InfoNCE loss to address the exposure
bias in recommender systems.

Table 1. Statistics of datasets. Dataset |  | #user
---
($|\mathcal{U}|$)
| #item
---
($|\mathcal{I}|$)
| #inter.
---
($|\mathcal{R}|$)
| avg. inter.
---
per user
density
Beauty | 22.4k | 12.1k | 198.5k | 8.9 | 0.07%
Gowalla | 29.9k | 41.0k | 1027.4k | 34.4 | 0.08%
Yelp2018 | 31.7k | 38.0k | 1561.4k | 49.3 | 0.13%

####  5.1.3. Evaluation Protocols

Following the common practice (Wang et al., 2019a; He et al., 2020, 2017), for
each dataset, we randomly split each user’s interactions into
training/validation/test sets with the ratio of 80%/10%/10%. To evaluate the
performance of top-K recommendation, we employ Recall and Normalized
Discounted Cumulative Gain (NDCG) as evaluation metrics. Recall@K measures how
many target items are retrieved in the recommendation result, while NDCG@K
further concerns about their positions in the ranking list. Note that we
consider the ranking list of all items (except for the training items in the
user history) instead of ranking a smaller set of random items together with
the target items, as suggested by recent work (Krichene and Rendle, 2020). We
repeat each experiment 5 times with different random seeds and report the
average score.

####  5.1.4. Implementation Details

We use the RecBole (Zhao et al., 2020) framework to implement all the methods
for fair comparisons. Adam is used as the default optimizer and the maximum
number of epochs is set to 300. Early stop is adopted if NDCG@20 on the
validation dataset continues to drop for 10 epochs. We set the embedding size
to $64$ and the learning rate to 1e-3 for all the methods. The training batch
size is set to 256 on Beauty and 1024 on the other two datasets. The weight
decay is tuned among [0, 1e-8, 1e-6, 1e-4]. The default encoder $f$ in
DirectAU is a simple embedding table that maps user/item IDs to embeddings.
The weight $\gamma$ of $l_{\rm uniform}$ in DirectAU is tuned within [0.2,
0.5, 1, 2, 5, 10]. As for baseline-specific hyper-parameters, we tune them in
the ranges suggested by the original paper. All the parameters are initialized
by xavier initialization. Codes are publicly
available999https://github.com/THUwangcy/DirectAU.

Table 2. Top-K recommendation performance on three datasets. The best results are in bold face, and the best baselines are underlined. The superscripts ∗∗ indicate $p\leq 0.01$ for the paired t-test of DirectAU vs. the best baseline (the relative improvements are denoted as Improv.). Setting | Baseline Methods | Ours
---|---|---
Dataset | Metric | BPRMF | BPR-DS | ENMF | RecVAE | LGCN | DGCF | BUIR | CLRec | DirectAU | Improv.
Beauty |  Recall@10 | 0.0806 | 0.0816 | 0.0915 | 0.0824 | 0.0863 | 0.0897 | 0.0816 | 0.0937 | 0.1002∗∗ | 6.94%
|  Recall@20 | 0.1153 | 0.1181 | 0.1282 | 0.1145 | 0.1201 | 0.1283 | 0.1204 | 0.1337 | 0.1400∗∗ | 4.74%
|  Recall@50 | 0.1763 | 0.1745 | 0.1914 | 0.1712 | 0.1819 | 0.1958 | 0.1866 | 0.1996 | 0.2062∗∗ | 3.33%
|  NDCG@10 | 0.0444 | 0.0459 | 0.0511 | 0.0486 | 0.0484 | 0.0501 | 0.0457 | 0.0547 | 0.0582∗∗ | 6.44%
|  NDCG@20 | 0.0534 | 0.0554 | 0.0606 | 0.0570 | 0.0581 | 0.0600 | 0.0556 | 0.0651 | 0.0686∗∗ | 5.38%
|  NDCG@50 | 0.0658 | 0.0670 | 0.0736 | 0.0686 | 0.0699 | 0.0738 | 0.0692 | 0.0786 | 0.0820∗∗ | 4.33%
Gowalla |  Recall@10 | 0.0866 | 0.1132 | 0.1149 | 0.1211 | 0.1289 | 0.1301 | 0.0798 | 0.1215 | 0.1394∗∗ | 7.15%
|  Recall@20 | 0.1263 | 0.1637 | 0.1671 | 0.1771 | 0.1871 | 0.1889 | 0.1164 | 0.1755 | 0.2014∗∗ | 6.63%
|  Recall@50 | 0.2040 | 0.2593 | 0.2675 | 0.2768 | 0.2934 | 0.2919 | 0.1917 | 0.2813 | 0.3127∗∗ | 6.56%
|  NDCG@10 | 0.0622 | 0.0814 | 0.0797 | 0.0845 | 0.0930 | 0.0939 | 0.0570 | 0.0868 | 0.0991∗∗ | 5.56%
|  NDCG@20 | 0.0736 | 0.0961 | 0.0953 | 0.1007 | 0.1097 | 0.1099 | 0.0676 | 0.1022 | 0.1170∗∗ | 6.44%
|  NDCG@50 | 0.0926 | 0.1196 | 0.1200 | 0.1251 | 0.1356 | 0.1358 | 0.0858 | 0.1281 | 0.1442∗∗ | 6.20%
Yelp2018 |  Recall@10 | 0.0416 | 0.0533 | 0.0596 | 0.0495 | 0.0508 | 0.0519 | 0.0444 | 0.0547 | 0.0684∗∗ | 14.83%
|  Recall@20 | 0.0693 | 0.0864 | 0.0957 | 0.0820 | 0.0833 | 0.0849 | 0.0737 | 0.0890 | 0.1096∗∗ | 14.55%
|  Recall@50 | 0.1293 | 0.1572 | 0.1710 | 0.1494 | 0.1534 | 0.1575 | 0.1386 | 0.1606 | 0.1935∗∗ | 13.16%
|  NDCG@10 | 0.0335 | 0.0423 | 0.0482 | 0.0395 | 0.0406 | 0.0409 | 0.0349 | 0.0436 | 0.0553∗∗ | 14.77%
|  NDCG@20 | 0.0428 | 0.0534 | 0.0603 | 0.0504 | 0.0514 | 0.0521 | 0.0448 | 0.0551 | 0.0691∗∗ | 14.53%
|  NDCG@50 | 0.0602 | 0.0740 | 0.0821 | 0.0698 | 0.0717 | 0.0732 | 0.0636 | 0.0758 | 0.0933∗∗ | 13.67%

###  5.2. Overall Performance

Table 2 shows the performance of different baseline CF methods and our
DirectAU. From the experimental results, we mainly have the following
observations.

Firstly, it is surprising that directly optimizing alignment and uniformity
yields such impressive performance improvements, given that most baselines
come from studies in recent two years. This demonstrates that these two
properties strongly agree with the representation quality in CF, and current
models might not address both alignment and uniformity well, which leads to
inferior results. Compared to state-of-the-art CF methods, DirectAU is not
only conceptually simple but also empirically effective.

Secondly, we find the best baseline varies in different datasets. The
contrastive learning based CLRec is effective on Beauty; while the GNN-based
DGCF takes advantage on Gowalla; and ENMF achieves remarkable performance on
the largest dataset Yelp2018. This shows that the characteristics of different
CF models may suit different application scenarios. On the contrary, DirectAU
is capable of directly adjusting the balance between alignment and uniformity,
leading to consistently the best performance on all three datasets.

Thirdly, comparing different kinds of baselines, methods with more complex
encoders do not always benefit the performance. The most complex model DGCF is
only the most effective on Gowalla but generally costs much more time for
training. Differently, methods focusing on the learning objective (e.g., ENMF,
CLRec) are more robust and usually yields promising results. This shows the
importance of designing suitable loss functions rather than sophisticated
encoders. The effectiveness of DirectAU also suggests that it is useful to
understand the desired properties of representations in CF, which benefit the
design of more powerful loss functions.

Figure 4. $l_{\rm align}$-$l_{\rm uniform}$ plot of different CF models on
Beauty. For both $l_{\rm align}$ and $l_{\rm uniform}$, lower numbers are
better. Colors and numbers in parentheses indicate NDCG@20. Figure 5. Learning
curves when only optimizing the alignment loss (left), only optimizing the
uniformity loss (middle), and optimizing both of the losses (right) on
Yelp2018. $l_{\rm uniform}$ is exponentiated for better visualization. The
encoder yields poor performance when only one of alignment and uniformity is
optimized. Both of the properties are important to learn high-quality user and
item representations.

Furthermore, in Figure 4, we show the alignment and uniformity of different CF
methods101010RecVAE is not included because it is a generative method without
item embeddings. The alignment and uniformity metrics are invalid under our
definition. along with their recommendation performance on Beauty. Overall, we
can see methods with both better alignment and uniformity achieve better
performance. ENMF and CLRec become two strong baselines because of the balance
between these two properties. DGCF mainly improves uniformity on top of LGCN
by introducing disentanglement to the representations. The recent novel method
BUIR achieves promising results without negative samples mainly due to the
superiority in alignment. But without the supervision signals from negative
samples, the uniformity of BUIR is poor. Compared to state-of-the-art CF
methods, DirectAU achieves the lowest alignment and uniformity losses and
yields the best performance. This verifies the causal effect of alignment and
uniformity on the representation quality in CF.

Table 3. Performance comparison of different encoders when integrated with the proposed DirectAU loss. Method | Beauty | Gowalla
---|---|---
Recall@20 | NDCG@20 | Recall@20 | NDCG@20
BPRMF | 0.1153 | 0.0534 | 0.1263 | 0.0736
+DirectAU | 0.1400∗∗ | 0.0686∗∗ | 0.2014∗∗ | 0.1170∗∗
LGCN-1 | 0.1211 | 0.0560 | 0.1769 | 0.1033
+DirectAU | 0.1444∗∗ | 0.0700∗∗ | 0.2036∗∗ | 0.1184∗∗
LGCN-2 | 0.1201 | 0.0581 | 0.1871 | 0.1097
+DirectAU | 0.1455∗∗ | 0.0707∗∗ | 0.2043∗∗ | 0.1191∗∗

###  5.3. Importance of Both Alignment and Uniformity Losses

To show that both properties are important to learn informative encoders,
Figure 5 gives the learning curves when 1) only optimizing the alignment loss,
2) only optimizing the uniformity loss, and 3) optimizing both of the losses
on Yelp2018. If only alignment is considered (left), the encoder achieves
perfect alignment ($l_{\rm align}$ approaches 0) but suffers a degeneration in
uniformity. As a result, the recommendation performance only improves a little
at the beginning and then converges to poor results. If only uniformity is
considered (middle), the encoder maintains uniformity (randomly initialized
embeddings are well uniform) but does not improve alignment. Hence, the
performance is even worse than only optimizing $l_{\rm align}$. Differently,
when optimizing both alignment and uniformity (right), the encoder keeps
promising uniformity and continuously improves alignment at the same time. As
a result, the representation quality steadily increases and boosts the
recommendation performance. These trends demonstrate the importance of
addressing both alignment and uniformity in CF.

Table 4. Efficiency comparison on Yelp2018, including the average training time per epoch, the number of epochs to converge, and the total training time (s: second, m: minute, h: hour). Method | time/epoch | #epoch | total time
---|---|---|---
BPRMF | 29.8s | 59 | 29m
ENMF | 24.8s | 89 | 36m
LGCN | 228.6s | 107 | 6h48m
DirectAU | 37.3s | 50 | 31m

###  5.4. Integration with Other CF Encoders

In the main experiments (Table 2), we optimize the DirectAU loss with a simple
MF encoder (i.e., embedding table). This raises the question that whether it
is also beneficial to directly optimize alignment and uniformity for other CF
encoders. Here we take MF and LGCN with different numbers of layers as the
interaction encoder, respectively. Table 3 shows the performance of these
methods with their original losses and corresponding variants with the
DirectAU loss. LGCN-X means the LGCN encoder with X GNN layers. We can see
DirectAU consistently brings remarkable improvements to each encoder. Besides,
when integrated with more powerful encoders like LGCN-2, DirectAU achieves
higher performance than the default MF encoder. This shows the generalization
ability of the proposed learning framework. Meanwhile, we find the relative
improvements are the most significant for the simplest MF encoder. In the
Gowalla dataset, it is impressive that MF+DirectAU leads to 59.2% improvements
than the original MF on average; while LGCN-2+DirectAU only brings around 8.9%
improvements. This verifies the importance of choosing proper learning
objectives in CF. With the help of the DirectAU loss, a simple MF encoder can
also learn high-quality representations, and hence achieves comparable results
with the complex LGCN-2 encoder. Considering the balance of effectiveness and
efficiency, we still choose MF as the default encoder in the following
analyses.

###  5.5. Efficiency Analyses

Here we compare the training efficiency of DirectAU with BPRMF and other two
state-of-the-art CF models, i.e., ENMF and LGCN, which are both relatively
efficient in their respective categories. In Table 4, we present the average
training time per epoch, the number of epochs to converge, and the total
training time on the largest dataset Yelp2018. The efficiency experiments are
conducted on the same machine (Intel Core 12-core CPU of 3.5GHz and single
NVIDIA GeForce GTX 1080 Ti GPU). We compare different methods under the same
implementation framework and the setting of batch size is fixed to 256 to
ensure fairness. The results show that ENMF is the most efficient in terms of
the training time per epoch, which results from the specifically designed
learning algorithm. The graph-based LGCN is much slower because of the
neighborhood aggregation in each iteration, even if LGCN performs linear
propagation for simplicity. Our DirectAU needs a little more training time per
epoch than BPRMF and ENMF mainly due to the calculation of the uniformity
loss. However, DirectAU generally converges fast and the total time is similar
with BPRMF and ENMF, which is much faster than LGCN. Thus, DirectAU is
relatively efficient for its simplicity, and we believe the performance gains
justify the runtime costs in practice.

###  5.6. Parameter Sensitivity

DirectAU introduces a hyper-parameter $\gamma$ that controls the weight of the
uniformity loss. It is worth noting that this is the only hyper-parameter to
tune for DirectAU, which does not rely on negative sampling like previous CF
methods. Therefore, there is no need to consider the number of negative
samples and the sampling strategy. This makes DirectAU easy to use in real-
world applications. Figure 6 shows how the performance changes when varying
this hyper-parameter on the three datasets. We can observe a similar trend
that the performance increases first and then decreases. Different datasets
suit different degrees of uniformity, which depend on the characteristics of
datasets. We find higher uniformity weights might be preferable for datasets
with more average interactions per user (i.e., Gowalla, Yelp2018), in which
case representations might be more likely to be pushed closer due to the
alignment loss. Note that the range of $\gamma$ is not restricted from 0.2 to
10, which may need wider ranges and fine-grained steps in practice.

##  6\. Related Work

###  6.1. Collaborative Filtering

Collaborative filtering (CF) plays an essential role in recommender systems
(Schafer et al., 2007). The core idea of CF is that similar users tend to have
similar preferences. Different from content-based filtering methods, CF does
not rely on user and item profiles to make recommendations, and hence is
flexible to work in various domains. One of the primary methods for CF is the
latent factor model, which learns latent user and item representations from
observed interactions. The predicted score of an unobserved user-item pair is
derived by the similarity (e.g., dot product) between the user and item
representation. Traditional methods are mainly based on matrix factorization
(MF) (Koren, 2008; Koren et al., 2009). With the development of neural
networks, neural CF models begin to emerge to learn more powerful user/item
representations (He et al., 2017; Zhang et al., 2019). Besides, graph neural
networks attract increasing attention recently, and a number of graph-based CF
models have been proposed (He et al., 2020; Wang et al., 2019a, 2020a; Wu et
al., 2019). The observed user-item interactions are taken as a bipartite
graph, and graph neural networks help to capture high-order connection
information.

Figure 6. Parameter sensitivity with regard to the weight of $l_{\rm uniform}$
in DirectAU.

Existing studies in CF mainly focus on the model structure of the encoder but
pay less attention to other components like the learning objective and the
negative sampling strategy, which also contribute to the final performance.
Some recent works (Chen et al., 2020; Lee et al., 2021; Liu et al., 2021; Mao
et al., 2021) begin to investigate alternative learning paradigms. For
example, ENMF (Chen et al., 2020) devises an efficient approach to optimize
the MSE loss based on the whole data. BUIR (Lee et al., 2021) presents a novel
asymmetric structure to learn from positive-only data. CLRec (Liu et al.,
2021) adopts the InfoNCE loss in contrastive learning to address the exposure
bias in recommender systems. In this paper, we also focus on the learning
objective in CF. Differently, we are the first to investigate the desired
properties of representations in CF from the perspective of alignment and
uniformity. And a new loss function that directly optimizes these two
properties is proposed based on the analyses results.

###  6.2. Alignment and Uniformity in Contrastive Representation Learning

Unsupervised contrastive representation learning has witnessed great success
in recent years (Grill et al., 2020). Studies in this literature usually aim
to learn informative representations on the unit hypersphere based on self-
supervised tasks. Recent work (Wang and Isola, 2020) identifies two key
properties related to the quality of representations, namely alignment and
uniformity. Similar instances are expected to have similar representations
(alignment), and the distribution of representations is preferred to preserve
as much information as possible (uniformity). Alignment is usually easy to be
achieved (e.g., mapping all the inputs to the same representations), but it is
hard to maintain uniformity at the same time. Previous representation learning
strategies can be seen to preserve uniformity in different ways, such as
discriminating from negative samples (Gao et al., 2021) and feature
decorrelation (Zbontar et al., 2021). Directly matching uniformly sampled
points on the unit hypersphere is also shown to provide good representations
(Bojanowski and Joulin, 2017). However, to the best of our knowledge, there
still lacks thorough investigations towards alignment and uniformity in CF.
This work theoretically shows the connection between the typical BPR loss and
these two properties. Besides, our analyses towards learning dynamics of
different CF methods help understand the rationales of existing CF methods and
design new learning objectives.

##  7\. Conclusion

In this paper, we investigate the desired properties of representations in
collaborative filtering (CF). Specifically, we propose to measure the
representation quality in CF from the perspective of alignment and uniformity,
inspired by recent progress in contrastive representation learning. We first
theoretically reveal the connection between the commonly adopted BPR loss and
these two properties. Then, we empirically analyze the learning dynamics of
typical CF methods in terms of alignment and uniformity. We find different
methods may be good at different aspects, while either better alignment or
better uniformity leads to higher recommendation performance. Based on the
analyses results, a loss function that directly optimizes these two properties
is proposed and experimented to be effective. A simple matrix factorization
model with the proposed loss function achieves superior performance compared
to state-of-the-art CF methods. We hope this work could inspire the CF
community to pay more attention to the learning paradigm via in-depth analyses
towards the representation quality.

In the future, we will investigate other learning objectives that also favor
alignment and uniformity to further improve effectiveness and efficiency.

###### Acknowledgements.

This work is supported by the Natural Science Foundation of China (Grant No.
U21B2026, 62002191) and Tsinghua University Guoqiang Research Institute. We
would like to thank the VMWare gift funding’s partly support to the authors.

## References

  * (1)
  * Bojanowski and Joulin (2017) Piotr Bojanowski and Armand Joulin. 2017.  Unsupervised learning by predicting noise. In _International Conference on Machine Learning_. PMLR, 517–526.
  * Chen et al. (2020) Chong Chen, Min Zhang, Yongfeng Zhang, Yiqun Liu, and Shaoping Ma. 2020.  Efficient Neural Matrix Factorization without Sampling for Recommendation.  _ACM Transactions on Information Systems (TOIS)_ 38, 2 (2020), 1–28.
  * Chen et al. (2017) Jingyuan Chen, Hanwang Zhang, Xiangnan He, Liqiang Nie, Wei Liu, and Tat-Seng Chua. 2017\.  Attentive collaborative filtering: Multimedia recommendation with item-and component-level attention. In _Proceedings of the 40th International ACM SIGIR conference on Research and Development in Information Retrieval_. ACM, 335–344.
  * Covington et al. (2016) Paul Covington, Jay Adams, and Emre Sargin. 2016\.  Deep neural networks for youtube recommendations. In _Proceedings of the 10th ACM conference on recommender systems_. 191–198.
  * Gao et al. (2021) Tianyu Gao, Xingcheng Yao, and Danqi Chen. 2021.  SimCSE: Simple Contrastive Learning of Sentence Embeddings.  _arXiv preprint arXiv:2104.08821_ (2021).
  * Grill et al. (2020) Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Daniel Guo, Mohammad Gheshlaghi Azar, et al. 2020\.  Bootstrap your own latent: A new approach to self-supervised learning.  _arXiv preprint arXiv:2006.07733_ (2020).
  * He et al. (2020) Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. 2020\.  Lightgcn: Simplifying and powering graph convolution network for recommendation. In _Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval_. 639–648.
  * He et al. (2017) Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua. 2017\.  Neural collaborative filtering. In _Proceedings of the 26th International Conference on World Wide Web_. International World Wide Web Conferences Steering Committee, 173–182.
  * Koren (2008) Yehuda Koren. 2008\.  Factorization meets the neighborhood: a multifaceted collaborative filtering model. In _Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining_. 426–434.
  * Koren et al. (2009) Yehuda Koren, Robert Bell, and Chris Volinsky. 2009\.  Matrix factorization techniques for recommender systems.  _Computer_ 42, 8 (2009), 30–37.
  * Krichene and Rendle (2020) Walid Krichene and Steffen Rendle. 2020.  On sampled metrics for item recommendation. In _Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining_. 1748–1757.
  * Lee et al. (2021) Dongha Lee, SeongKu Kang, Hyunjun Ju, Chanyoung Park, and Hwanjo Yu. 2021.  Bootstrapping User and Item Representations for One-Class Collaborative Filtering.  _arXiv preprint arXiv:2105.06323_ (2021).
  * Liang et al. (2016) Dawen Liang, Laurent Charlin, James McInerney, and David M Blei. 2016.  Modeling user exposure in recommendation. In _Proceedings of the 25th international conference on World Wide Web_. 951–961.
  * Liang et al. (2018) Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara. 2018.  Variational autoencoders for collaborative filtering. In _Proceedings of the 2018 world wide web conference_. 689–698.
  * Liu et al. (2021) Zhuang Liu, Yunpu Ma, Yuanxin Ouyang, and Zhang Xiong. 2021\.  Contrastive Learning for Recommender System.  _arXiv preprint arXiv:2101.01317_ (2021).
  * Mao et al. (2021) Kelong Mao, Jieming Zhu, Jinpeng Wang, Quanyu Dai, Zhenhua Dong, Xi Xiao, and Xiuqiang He. 2021.  SimpleX: A Simple and Strong Baseline for Collaborative Filtering. In _Proceedings of the 30th ACM International Conference on Information & Knowledge Management_. 1243–1252.
  * McAuley et al. (2015) Julian McAuley, Christopher Targett, Qinfeng Shi, and Anton Van Den Hengel. 2015.  Image-based recommendations on styles and substitutes. In _Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval_. 43–52.
  * Rendle and Freudenthaler (2014) Steffen Rendle and Christoph Freudenthaler. 2014.  Improving pairwise learning for item recommendation from implicit feedback. In _Proceedings of the 7th ACM international conference on Web search and data mining_. 273–282.
  * Rendle et al. (2009) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme. 2009.  BPR: Bayesian personalized ranking from implicit feedback. In _Proceedings of the 25th conference on uncertainty in artificial intelligence_. AUAI Press, 452–461.
  * Schafer et al. (2007) J Ben Schafer, Dan Frankowski, Jon Herlocker, and Shilad Sen. 2007.  Collaborative filtering recommender systems.  In _The adaptive web_. Springer, 291–324.
  * Shenbin et al. (2020) Ilya Shenbin, Anton Alekseev, Elena Tutubalina, Valentin Malykh, and Sergey I Nikolenko. 2020\.  Recvae: A new variational autoencoder for top-n recommendations with implicit feedback. In _Proceedings of the 13th International Conference on Web Search and Data Mining_. 528–536.
  * Su and Khoshgoftaar (2009) Xiaoyuan Su and Taghi M Khoshgoftaar. 2009.  A survey of collaborative filtering techniques.  _Advances in artificial intelligence_ 2009 (2009).
  * Wang et al. (2022) Chenyang Wang, Weizhi Ma, and Chong Chen. 2022.  Sequential Recommendation with Multiple Contrast Signals.  _ACM Transactions on Information Systems (TOIS)_ (2022).
  * Wang et al. (2020b) Chenyang Wang, Weizhi Ma, Min Zhang, Chong Chen, Yiqun Liu, and Shaoping Ma. 2020b.  Toward Dynamic User Intention: Temporal Evolutionary Effects of Item Relations in Sequential Recommendation.  _ACM Transactions on Information Systems (TOIS)_ 39, 2 (2020), 1–33.
  * Wang et al. (2020c) Chenyang Wang, Weizhi Ma, Min Zhang, Yiqun Liu, and Shaoping Ma. 2020c.  Make It a Chrous: Knowledge- and Time-aware Item Modeling for Sequential Recommendation. In _Proceedings of the 43th International ACM SIGIR conference_. ACM.
  * Wang et al. (2019b) Chenyang Wang, Min Zhang, Weizhi Ma, Yiqun Liu, and Shaoping Ma. 2019b.  Modeling Item-Specific Temporal Dynamics of Repeat Consumption for Recommender Systems. In _The World Wide Web Conference_. ACM, 1977–1987.
  * Wang and Isola (2020) Tongzhou Wang and Phillip Isola. 2020.  Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In _International Conference on Machine Learning_. PMLR, 9929–9939.
  * Wang et al. (2019a) Xiang Wang, Xiangnan He, Meng Wang, Fuli Feng, and Tat-Seng Chua. 2019a.  Neural graph collaborative filtering. In _Proceedings of the 42nd international ACM SIGIR conference on Research and development in Information Retrieval_. 165–174.
  * Wang et al. (2020a) Xiang Wang, Hongye Jin, An Zhang, Xiangnan He, Tong Xu, and Tat-Seng Chua. 2020a.  Disentangled graph collaborative filtering. In _Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval_. 1001–1010.
  * Wu et al. (2019) Shu Wu, Yuyuan Tang, Yanqiao Zhu, Liang Wang, Xing Xie, and Tieniu Tan. 2019\.  Session-based recommendation with graph neural networks. In _Proceedings of the AAAI Conference on Artificial Intelligence_ , Vol. 33. 346–353.
  * Zbontar et al. (2021) Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. 2021.  Barlow twins: Self-supervised learning via redundancy reduction.  _arXiv preprint arXiv:2103.03230_ (2021).
  * Zhang et al. (2019) Shuai Zhang, Lina Yao, Aixin Sun, and Yi Tay. 2019\.  Deep learning based recommender system: A survey and new perspectives.  _ACM Computing Surveys (CSUR)_ 52, 1 (2019), 1–38.
  * Zhao et al. (2020) Wayne Xin Zhao, Shanlei Mu, Yupeng Hou, Zihan Lin, Kaiyuan Li, Yushuo Chen, Yujie Lu, Hui Wang, Changxin Tian, Xingyu Pan, Yingqian Min, Zhichao Feng, Xinyan Fan, Xu Chen, Pengfei Wang, Wendi Ji, Yaliang Li, Xiaoling Wang, and Ji-Rong Wen. 2020.  RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms.  _arXiv preprint arXiv:2011.01731_ (2020).
  * Zhou et al. (2021) Chang Zhou, Jianxin Ma, Jianwei Zhang, Jingren Zhou, and Hongxia Yang. 2021.  Contrastive learning for debiased candidate generation in large-scale recommender systems. In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_. 3985–3995.

##  Appendix A Appendix

In the appendix, we first show the learning algorithm of the proposed
DirectAU. Then, we detail the calculation of the alignment and uniformity
losses when measuring the entire learned embeddings in CF.

###  A.1. Learning Algorithm of DirectAU

Algorithm 1 shows the learning algorithm of DirectAU. PyTorch-style
pseudocodes to calculate alignment and uniformity losses during training are
also given to facilitate reproducibility.

Algorithm 1 Learning algorithm of DirectAU (PyTorch style)

1:user-item interactions data $\mathcal{R}$; structure of encoder network $f$;
weight of the uniformity loss $\gamma$; embedding dimension $d$.

2:encoder parameters $\theta$

3:Randomly initialize all parameters.

4:for each mini-batch with $n$ user-item pairs $(u,i)\in\mathcal{R}$ do

5: Get user and item embeddings $f(u),f(i)$

6: $\mathbf{x}=f(u)~{}/~{}||f(u)||,~{}~{}~{}\mathbf{y}=f(i)~{}/~{}||f(i)||$

7: $\mathcal{L}_{\rm
DirectAU}=\textsc{Align}(\mathbf{x},\mathbf{y})+\gamma\cdot(\textsc{Uni}(\mathbf{x})+\textsc{Uni}(\mathbf{y}))~{}/~{}2$

8: Update the encoder $f$ by gradient descent

9:end for

10:

11:function align(x, y) # alignment loss

12: return (x - y).norm(dim=1).pow(2).mean()

13:end function

14:function uni(x) # uniformity loss

15: dist = torch.pdist(x, p=2).pow(2)

16: return dist.mul(-2).exp().mean().log()

17:end function

###  A.2. Alignment and Uniformity Calculation

According to our definitions for alignment and uniformity in CF, i.e., Eq.(6),
user-item pairs to calculate the alignment loss should sample from the
distribution of positive interactions $p_{\rm pos}$, and user-user (item-item)
pairs to calculate the uniformity loss should sample from the corresponding
user/item distribution $p_{\rm user}/p_{\rm item}$. Given the learned
embeddings of all the users and items, the alignment loss can be directly
calculated as follows:

(8) $l_{\rm
align}=\dfrac{1}{|\mathcal{R}|}\sum_{(u,i)\in\mathcal{R}}||\tilde{f(u)}-\tilde{f(i)}||^{2},$

where $\mathcal{R}$ is the set of observed user-item interactions as mentioned
in Section 2.1. We only need to traverse all the $(u,i)$ pairs in
$\mathcal{R}$, and the time complexity is $O(|\mathcal{R}|)$.

As for the calculation of uniformity, a naive and intuitive method is to
sample $u,u^{\prime}\in\mathcal{U}$ and $i,i^{\prime}\in\mathcal{I}$. However,
this is not consistent with the definition that $u,u^{\prime}\sim p_{\rm
user}$ and $i,i^{\prime}\sim p_{\rm item}$. Notice that the calculation of the
uniformity loss during training follows the actual $p_{\rm user}$ and $p_{\rm
item}$ because the training batch is constructed based on positive
interactions. When measure the overall uniformity of the learned embeddings,
we should also sample two interactions from $\mathcal{R}$ and retain the
user/item side as the input pair, which ensures that $u,u^{\prime}$ and
$i,i^{\prime}$ are sampled from corresponding distribution:

(9) $\begin{split}l_{\rm
uniform}=&~{}\left(\log\dfrac{1}{|\mathcal{R}|\left(|\mathcal{R}|-1\right)}\sum_{(u,i),(u^{\prime},i^{\prime})\in\mathcal{R}}e^{-2||\tilde{f(u)}-\tilde{f(u^{\prime})}||^{2}}\right)/~{}2~{}+\\\
&~{}\left(\log\dfrac{1}{|\mathcal{R}|\left(|\mathcal{R}|-1\right)}\sum_{(u,i),(u^{\prime},i^{\prime})\in\mathcal{R}}e^{-2||\tilde{f(i)}-\tilde{f(i^{\prime})}||^{2}}\right)/~{}2.\end{split}$

Meanwhile, this calculation method is time-consuming and contains many
redundant computations. We need to traverse the entire interaction set
$\mathcal{R}$ twice and the time complexity is $O(|\mathcal{R}|^{2})$, which
is usually intractable in practice. To solve this problem, we devise a method
to calculate the uniformity loss by directly sampling from the user/item set
together with a popularity-weighting strategy:

(10) $\begin{split}l_{\rm
uniform}=&~{}\left(\log\sum_{u,u^{\prime}\in\mathcal{U}}\dfrac{p(u)p(u^{\prime})}{P_{U}}\cdot
e^{-2||\tilde{f(u)}-\tilde{f(u^{\prime})}||^{2}}\right)/~{}2~{}+\\\
&~{}\left(\log\sum_{i,i^{\prime}\in\mathcal{I}}\dfrac{p(i)p(i^{\prime})}{P_{I}}\cdot
e^{-2||\tilde{f(i)}-\tilde{f(i^{\prime})}||^{2}}\right)/~{}2,\end{split}$

where $p(\cdot)$ returns the number of related interactions in $\mathcal{R}$
(i.e., popularity). $P_{U}=\sum_{u\in\mathcal{U}}p(u)$ and
$P_{I}=\sum_{i\in\mathcal{I}}p(i)$ is the normalization factor, respectively.
It is easy to show that Eq.(9) and Eq.(10) are exactly equivalent, while the
latter reduces the computational cost to a large extent because the scale of
$\mathcal{U}/\mathcal{I}$ is usually much smaller than $\mathcal{R}$. In this
way, we can measure both alignment and uniformity of the learned embeddings
efficiently.

This popularity-weighting strategy also explicitly suggests that the
uniformity loss focuses more on the distances between popular users/items as
expected. Those popular users and items are more likely to be aligned very
close, and it is reasonable to encourage them to scatter on the hypersphere.


# Calculation of Femur Caput Collum Diaphyseal angle for X-Rays images using
Semantic Segmentation

Deepak Bhatia, Muhammad Abdullah, Anne Querfurth, Mahdi Mantash  University of
Erlangen-Nürnberg
<EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS><EMAIL_ADDRESS>
###### Abstract

This paper investigates the use of deep learning approaches to estimate the
femur caput-collum-diaphyseal (CCD) angle from X-ray images. The CCD angle is
an important measurement in the diagnosis of hip problems, and correct
prediction can help in the planning of surgical procedures. Manual measurement
of this angle, on the other hand, can be time intensive and vulnerable to
inter-observer variability. In this paper, we present a deep-learning
algorithm that can reliably estimate the femur CCD angle from X-ray images. To
train and test the performance of our model, we employed an X-ray image
dataset with associated femur CCD angle measurements.Furthermore we built a
prototype to display the resulting predictions and to allow the user to
interact with the predictions. As this is happening in a sterile setting
during a surgery, we expanded our interface to the possibility of being used
only by voice commands. Our results show that our deep learning model predicts
the femur CCD angle on X-ray images with great accuracy, with a mean absolute
error of 4.3 degrees on left femur and 4.9 degrees on right femur on test
dataset. Our results suggest that deep learning has the potential to give a
more efficient and accurate technique for predicting the femur CCD angle,
which might have substantial therapeutic implications for the diagnosis and
management of hip problems.

###### Index Terms:

biomedical imaging, fully convolutional neural network, femur imaging, heatmap
regression, ransac, speech-to-text

##  I Introduction

Hip fractures are a common injury among the elderly, with high morbidity and
death rates. Correct hip fracture diagnosis and treatment are crucial for
achieving positive clinical results. The caput-collum-diaphyseal (CCD) angle,
which measures the angle formed between the femoral head and neck and the
femoral shaft, is one of the major factors utilized in the diagnosis and
therapy of hip fractures. The accurate measurement of the CCD angle is
critical for identifying the best treatment strategy for hip fractures,
including implant selection and surgical technique. [1] Manual CCD angle
measurement by radiologists can be time-consuming and vulnerable to inter-
observer variability.
Deep learning-based algorithms have shown promising outcomes in medical image
analysis applications in recent years. Convolutional neural networks (CNNs)
are used in these approaches to extract features. The effectiveness of these
approaches, however, is largely dependent on the quality and amount of the
training dataset. We present a deep learning-based technique for estimating
the CCD angle on X-ray images in this research. Our solution uses a U-Net
architecture [2] to learn information from X-ray images and predict the CCD
angle. The suggested method is evaluated using a set of X-ray images of
patients that have been labelled with the associated CCD angles. With an MAE
of 4.3 degrees on left femur and 4.9 degrees on right femur, our experimental
findings suggest that the proposed technique performs quite well in terms of
accuracy.
The remainder of this paper is organized as follows. Section II discusses the
theoretical aspects of the research, such as the segmentation model utilized
and the tasks performed, as well as the speech recognition model deployed. We
describe the proposed method in section III, including the dataset used for
training and evaluation, the training procedure, evaluation, and the design
and functioning of the user interface of our prototype. Section IV presents
the experimental findings. Section V, we conclude and discusses the advantages
and limits of the proposed technique. Finally, Section VI suggests future
research areas.

##  II Theory

###  II-A Segmentation model

Image segmentation is the process of classifying each pixel to a class. It can
be further divided into instance segmentation and semantic segmentation.
Semantic segmentation cannot distinguish between different instances of the
same class while instance segmentation is capable of doing it.
Image segmentation has many applications in medical imaging where it is used
to identify irregular structures and abnormalities. Recent developments in
deep-learning also entail performance enhancement of image segmentation
networks. Many new deep-learning-based segmentation architectures have been
proposed in the recent past. It includes Unet[2], MASK-RCNN[3], SegNet [4],
DeepLabv3+[5]. After the success of transformers for natural language tasks,
they have also been introduced for vision tasks as well where they are labeled
as vision transformers. [6]Swin Transformer is one such example. Our work does
not include performance comparison of different segmentation architectures.
We decided to use UNet as it generalizes well even with the limited amount of
training data. Since training data is always sparse in medical imaging tasks,
UNet has been widely used in medical imaging for segmentation tasks.
UNet is a fully convolutional network[7]. It includes a contracting path for
context learning and a symmetric expanding path for precise localisation.
Localisation is important in biomedical images. 2D transposed convolution is
used for upsampling in expanding paths.

Figure 1: U-shaped UNet architecture

###  II-B Segmentation task

Our goal is to calculate the CCD angle. CCD angle is measured between the
femur neck centerline and the femur shaft centerline. These centerlines can be
predicted directly or can be estimated from lateral and medial lines both for
the neck and shaft. Moreover, the femur neck centerline always passes through
the centre of the femur head. This extra information can also be incorporated
for the estimation of the femur neck centerline.
Lines can be regressed directly or indirectly via segmentation of the Gaussian
distributed line heatmap. Heatmap regression has performed better in pose-
estimation tasks where the goal is to detect the location of human joints or
key points[8]. Heatmap regression learns a probability distribution by taking
neighbouring pixels into account. It is more robust and generalizes well. Our
work is also based on regressing lines indirectly via heatmaps.
Considering the possibility of more than one way to estimate the femur neck
centerline and femur shaft centerline, we decided to train a 12 output channel
UNet network incorporating 3 lines (medial, centerline, lateral) both for the
neck and shaft. We also predict CCD angle both for the left femur and right
femur. So, in total, 6 lines for the left femur and 6 lines for the right
femur.

###  II-C Motivation for Touchless Interface

The working environment in which our application is used is the operating room
and therefore comes with special challenges. These include limited space, time
pressure to keep the patient’s anaesthesia as short as possible, and
sterility. The interaction with a touch-based system requires the doctor to
become non-sterile which leads to the necessity of making oneself sterile
again. This costs valuable time and material.
New technologies can help, as there are already a large number of touch-less
input devices that allow interaction without touching. These include eye
tracking, gesture recognition and voice recognition. All of these devices
represent a great opportunity to reduce time in the OR and allow a steady
workflow. [9]
We considered using gesture recognition. However, compared to voice
recognition it has many disadvantages. For example, the user would need to
have free hands and the correct visibility angle and lighting for the camera
to recognise their gestures successfully. Furthermore, using gestures to
control an application is not intuitive and requires more time to become
proficient and know all of the gestures by heart. Considering all of those
error sources, we decided to go with voice recognition instead.
Voice recognition as an input method holds further advantages. The
additionally required hardware is only a microphone. Nowadays microphones are
small, light and provide nevertheless high-quality sound recordings. This
makes it possible to be used in a large variety of spatial settings as the
surgeon can move around the room while for example having the microphone
attached to them. A relevant factor in employing a new application or workflow
in the hospital setting is the intuitivity and simplicity of using and
learning how to handle the new system. Voice recognition provides this aspect,
as the commands are displayed on the monitor as well as being regular commands
that you would give a human to perform the wanted task. The required time to
get proficient at using the system is therefore minimal.
Overall, voice recognition provides a needed addition to our interface to make
it valuable for the OR setting.

###  II-D Voice model

OpenAI Whisper is a speech recognition deep learning model that combines a
convolutional neural network and a transformer-based language model. The CNN
component of the model encodes the temporal and spectral properties of the
input audio signal, while the transformer-based language model generates
transcriptions based on the stored features. [10] One of OpenAI Whisper’s key
advantages is its capacity to tolerate loud, noisy and difficult acoustic
situations which suit very well to our requirements for a speech recognition
model in a busy interventional setting. We used OpenAI Whisper for speech-to-
text recognition and implement voice control feature to control our user
interface.

##  III Methods

###  III-A Data

The data comprised 201 hip X-ray images of 166 different patients with both
left and right femurs. The images were annotated for 18 different labels, 9
for each femur bone at the start of the project by the 8 participants to look
for different strategies to predict the CCD angle. The labels comprised the
following: Femur Right, Femoral Head Right, Femoral Head Circle Right, Femoral
Shaft Medial Right, Femoral Shaft Lateral Right, Femoral Shaft Centerline
Right, Femoral Neck Medial Right, Femoral Neck Lateral Right, Femoral Neck
Centerline Right, Femur Left, Femoral Head Left, Femoral Head Circle Left,
Femoral Shaft Medial Left, Femoral Shaft Lateral Left, Femoral Shaft
Centerline Left, Femoral Neck Medial Left, Femoral Neck Lateral Left, Femoral
Neck Centerline Left. Our team found some problems with the neck lines in the
annotations so we re-annotated the neck lines.

Figure 2: Labeled femur

###  III-B Training and Implementation

For the training of the UNet, we created 8:1:1 train, validation, and test
split. Since, in few cases, a single patient has more than one X-Rays images,
we keep all images of the same patient into the same split to avoid data leak
into validation and test split. We also applied colorjitter(contrast) and
affine transformations(translate, scale, rotate) in an online augmentations
fashion to counter over-fitting. Input was resized (512x512) and normalised
before feeding to the network. We used z-score normalisation because most of
the pixels were either zero and near to zero except the pixels in the heatmap
with greater probabilities. It helped to use sigmoid on predictions to remove
the low probability pixels. We used a sigmoid cutoff of 0.9. Since, input mask
contains pixel probabilities, we used MSELoss to calculate pixel wise mean
squared error.
Since our input masks were line heatmaps with varying probabilities, We could
use pixel-wise crossentropy loss or MSELoss, we selected MSELoss for
regression between input mask and predicted mask because segmentation values
are continuous and the magnitude of the errors is important to consider. For
the optimization, we used Adam optimizer without weight decay and with
learning rate 8e-5. We kept the learning rate same for the training as network
was able to converge in very few epochs, 25 in our case. We used the batch
size of 4 as UNet can also generalize well even on smaller batch sizes.

###  III-C Post-processing of Heatmaps

After predicting the line heatmap with UNet. we apply a sigmoid cut-off to
remove points with low probabilities. Then we fit the remaining points via
linear regression. Since, outliers can impact the performance of the linear
fit, robust linear regression requires outliers suppression. HuborRegressor,
TheilSenRegressor and RansacRegressor can be used to fit a robust linear model
while suppressing the impact of outliers. RANSAC[11] performs better when data
is highly contaminated by outliers. Therefore, we decided to use RANSAC for
outliers removal.
RANSAC separates the data points into outliers and inliers groups. Only
inliers are used to fit the linear regression model. HuberRegressor can be
applied to the inliers set which can further lower the effect of outliers
points.

Figure 3: 1st column contains input X-Rays images with masks overlayed. 2nd
column contains input mask that is fed to the UNet. 3rd Column contains masks
predicted after the sigmoid cutoff applied at the output. 4th colums contains
predicted lines after fitting the linear fit.

###  III-D Evaluation

We use MSELoss for training the UNet but it cannot be used for model
assessment as two models with same loss can have very different accuracy when
CCD angle are computed because angle computation includes outliers removal and
a linear fit on the inliers using RANSAC. For the model assessment, we
computed euclidean distance between centroids of input mask and predicted mask
for individual lines. Only centroid difference is not enough. We also computed
the angular error for individual lines to calculate the orientation error.
As a final criterion for evaluation, we also computed the final CCD angle for
both left and right femurs on test dataset.

###  III-E User Interface

####  III-E1 Design and Functionality

The interface, which is shown in fig. 4, can be roughly divided into two
sections. The image and a toolbar. The central element of the application is
the displayed CT image. This is displayed large and takes up 4/5 of the entire
screen so that the user can see as much detail of the image as possible. On
the right, there is the toolbar, which has all the buttons for the functions
and a text field with the CCD angle. The icon that indicates the system state
of the voice control function is also shown here, more about this in the next
section.

Figure 4: Screenshot of user interface with a diagnostic CT scan of the hips
opened

For a successful procedure, it is important to know the angles of both sides
of the femur in order to be able to compare them. That´s why the CCD angles
are displayed one below the other, to make comparing the numbers as easy as
possible. The displayed angle in top is for the right side, because when
looking at the CT image from left to right (read-direction), the right hip
comes first. Another option for displaying the angles would have been
underneath the image, each on the according side (left/right), but this would
separate the two numbers spatially, and capturing both at one glance would not
be possible. Therefore we did not opt for this variant.
The functionality of the interface is elementary and limited to the essential
and most important tasks that are necessary during the intervention. These are
intentionally kept simple in order not to overwhelm the user with a flood of
new functions and to make the use of the interface as intuitive as possible.
This also keeps the necessary training and familiarisation to the tool
limited. The functions include opening new images, and zooming in on the left
or right femur by clicking on a button. After opening, the lines are
immediately drawn onto the image, the angle is calculated and displayed in the
text box on the right. The lines can be edited by dragging and dropping the
endpoints. To make it easier for the user and to draw attention to this
option, the colour of the points changes when the mouse is hovered over them.
The angle is automatically recalculated with each movement of the lines. In
addition, a screenshot of the scene can be saved with the image, the lines,
the angle and the possibility to add additional text about the case.
As the goal of this application is to use it in an intra-operative setting,
not only diagnostic images need to be displayed but also intra-operative.
Those can be taken for example with a mobile c-arm which has a lot smaller
field-of-view, compared to conventional diagnostic CT scanners, which do not
cover the entire pelvis and therefore only one side of the hip is displayed.
In order to still show both sides, there is the possibility to open two images
next to each other. In this case, the picture with the right femur is
automatically arranged on the left and the left femur on the right. All
functions behave the same as when only one image is open. The only exception
is zooming, where the respective picture is maximised.

####  III-E2 Voice control

A great advantage of our application is that it can be operated without any
touch, only by voice. A microphone records what is spoken and executes
functions triggered by certain words. All commands that the system recognises
are listed in the table I. The keywords that are spoken are all intentionally
short and intuitive for easy use without having to learn many words.
Since a lot is said during an intervention, the system does not always execute
the recognised functions, but only if it has been activated beforehand.
Activation also happens when a keyword is spoken. When choosing a keyword, it
is most important that it is a word that is not used by chance in the
operating room, which would accidentally activate the application.
Furthermore, the word should be easy to pronounce and simple to remember.
Considering these aspects, we decided on the word activate.

Figure 5: System state: The system only executes commands after activation.
Once activated the system using the key word activate, the system state will
change to be listening and waits until it recognizes any commands. After
execution of a command or after waiting for more than 5 seconds it
automatically goes back to the default, not listening state.

The transition between the states of the system, from ”sleeping” to ”waiting”,
is shown in fig. 5. In the default state, the system is ”sleeping” and does
not react to commands. After activation by saying activate, the system is in a
listening state, waiting for further commands, which are then executed. After
executing them, the system falls back into the ”sleeping” state. Likewise, if
none of the cues are recognized within 5 seconds, it goes back into the
”sleeping” state. This is because the commands are all very short. Thus, it
can be assumed that if no command was recognised within these 5 seconds, the
system was either activated by mistake or the command was not recognised
correctly. In order to avoid triggering unwanted behaviour, the system falls
back into the ”sleeping” state.
In order for the user to know at all times what state the system is in, an
icon was chosen and placed in the center of the toolbar on the right-hand
side. If the icon is light grey, the interface is ”asleep” and changes its
color to bright red as soon as it is activated. This is also shown in fig. 5.
When used exclusively with the voice, however, the functionality of the system
is slightly limited, as the points of the lines cannot be edited. Otherwise,
all functions are possible, just like when using a mouse.

command | execution
---|---
left | Zoom in on left femur. If two images are open the left image is maximized.
right | Zoom in on right femur. If two images are open the right image is maximized.
out or both | Zoom out to show both sides of the hip at the same time.
open |  The The next image is opened from a previously selected folder (before starting the application) sorted by time. The first time the command is used, the latest image is opened.
save |  Waits for additional input (like patient information). When finishing by saying ok function is finished, and a screenshot of the scene with date, time and the dictated patient information is saved to a previously selected (before starting the application) save folder.
TABLE I: Keywords which are recognized by the speech model and what
functionality is triggered by them

####  III-E3 Implementation

The user interface was implemented using the python package PyQt which is
based on the GUI toolkit Qt. The CT image and the predicted lines are
visualized in a QGraphicsScene which allows the user to interact with the
lines and update them accordingly. Each line is added to the scene as an item.
Each of the lines is the parent item to two endpoints (inherited from
QGraphicsPointItem) which are therefore also visible on the scene. The lines
themselves can not be selected and moved, but the two points can. Whenever the
lines are updated (via the editing the points), the CCD angle is recalculated,
using the formula and the displayed number is updated.
Several threads are needed to allow everything to run parallel. The main
interface and all updates on the interface are executed in the main thread.
When starting the application there are 2 more threads started, which are
required to allow speech control. Communication of the sub-threads with the
main thread is done with signals and slots from PyQt.

####  III-E4 User Study Design

To get a feeling for the usability and feedback on our system, we conducted a
user study with our prototype. Four participants were briefly introduced to
the medical background and the use of the application and then left alone to
test all functions and compare the functionality of using it with mouse and
solely with speech recognition.
Since we only have very few participants in our study, all participants used
the interface with both voice and mouse. They all used the mouse first to
become familiar with the system. The study was conducted in a public space,
with background noise, to mimic the possible background noise in the operating
room.
After using the system, all subjects were given a questionnaire. These
questionnaires were created using different standardised questionnaires, such
as SUS [12] and UEQ [13], and adapted to suite this application. In the first
part, statements have to be rated with numbers from 1 (strongly agree) to 5
(strongly disagree). Those questions are on purpose designed differently, some
are positives, others have a rather negative tone. The first part of the
questionnaire was answered regarding the interface and functionality in
general and the usage with mouse. In the second part, open questions are asked
to give the subjects the possibility of detailed feedback, and to compare the
option of using the interface by voice to using the mouse.

##  IV Results

###  IV-A Algorithm

Although we trained our UNet on 12 femur lines, 6 for each side. We ended up
just using the shaft and neck centerlines for measuring the CCD angle. Table
III contains performance of neck and shaft centerlines both for left and right
femur. Smaller angular error for individual lines indicates the success of out
approach in predicting lines. Euclidean distance between centroids is bit
larger than expected but it does not contribute much towards final angle
calculation as it is calculated before RANSAC.
Finally, In table II, we compute the CCD angle on test dataset. We got the
mean absolute error of 4.3 on left femur and 4.9 on right femur which
indicates the success of our approach.

Femur | Mean Absolute Error (degrees)
---|---
Left Femur | 4.3
Right Femur | 4.9
TABLE II: Mean absolute error of CCD Femur line | Mean Centroids Euclidean distance | Mean Angular error(degrees)
---|---|---
Left shaft centerline | 9.6 | 1.9
Left neck centerline | 14 | 2.7
Right neck centerline | 7 | 5.0
Right shaft centerline | 12.6 | 2.0
TABLE III: Evaluation of individual femur centerlines

###  IV-B User Study

To evaluate our user study in the usability and user experience of our
application, the SUS score was calculated. table IV shows the SUS scores of
the 4 users who were interviewed. All participants seem to be quite satisfied
with the application, and achieve a score of 80-90%. This is clearly above the
average of 68%, which is given in [12].

| user 1 | user 2 | user 3 | user 4
---|---|---|---|---
SUS score [in %] | 80 | 85 | 80 | 90
TABLE IV: SUS score in % for four user participating in our user study

Even more interesting are the results of the open questions. Many important
hints and ideas for further developments were given there. The function of
adding and deleting lines if the system does not predict exactly the two
centre lines (this happens only very rarely in split screen mode with one
femur per side) was introduced. The possibility of using the interface with
voice was consistently praised by all users, but only one of the users stated
that he would prefer to use voice over mouse. The icon that changes color to
represent the status of the system was positively highlighted. It was also
noted that there might be better words for activation, e.g. a fantasy name.
A big concern that was noticed during the user study is that the speech model
is biased heavily. The pronunciation is very important, and with a strong
accent, the speech recognition feature was almost unusable, as it took many
attempts until the correct command was recognized, or in some cases was not
recognized at all.

##  V Conclusion

We proposed a deep learning-based technique for estimating the CCD angle on
X-ray images in patients with hip fractures in this report. Our experimental
results showed that the proposed method achieved good accuracy results, with
an MAE of 4.3 degrees on left femur and 4.9 degrees on right femur. The
proposed method has the potential to enhance patient outcomes by assisting in
the faster, more accurate and efficient identification of hip disorders.
One of the main advantages of the proposed method is that it can automatically
learn the important features from hip X-ray images to determine the
appropriate CCD angle, reducing the need for manual effort. The angle is
determined by small changes in bone structure, which can be difficult to
discover precisely using standard manual methods, which are also subject to
human error and inter-reader variability.
Another advantage of the proposed approach is that it is simple to integrate
into the clinical workflow with the voice command feature in our user
interface providing ease of use in critical conditions of an interventional
setting for hip fractures correction procedures, allowing for quick and
precise calculation and visualizations of the CCD angle from X-ray images.
This automated strategy can minimize radiologists’ and surgeons’ workloads and
increase diagnostic and treatment process efficiency.
Our proposed method, however, has certain drawbacks. One of the constraints is
that training requires a big annotated dataset, which may be difficult to get.
The dataset we utilized was too small for the model to generalize well. There
were also images containing implants, which might have hampered the model’s
learning process. Moreover, our technique may not generalize well to
populations with different demographics or to cases with severe bone
deformities, malalignment or images with implants and fractures.

##  VI Future Work

In future work, we intend to evaluate the performance of the proposed method
on a bigger and more diversified dataset. Additionally, we plan to test the
proposed method’s generalization ability on different types of cases with
severe bone abnormalities or malalignment, or cases with implants, and
fractures. Lastly, we intend to explore the proposed method’s integration into
the clinical workflow and assess its influence on the diagnostic process and
patient outcomes.

There are a number of aspects of the user interface that can be improved in
future versions. These include more functions such as adding and deleting
additional lines. Furthermore medical specialists were wishing for more
freedom when opening two images side by side. It would also make sense, for
example, to display a diagnostic image showing both sides of the hip together
with an intraoperative image showing only one femur. These considerations
raise further questions, such as how to then display the angles (since there
are now a total of three to be seen).
It would also be very interesting to do a larger user study to gain a more
detailed insight into how useful the speech recognition function actually is.
This could also include other conditions of everyday clinical life, such as
typical background noises or a sterile environment.

## References

  * [1] Kim Edward LeBlanc, Herbert L Muncie Jr, and Leanne L LeBlanc.  Hip fracture: diagnosis, treatment, and secondary prevention.  American family physician, 89(12):945–951, 2014.
  * [2] Olaf Ronneberger, Philipp Fischer, and Thomas Brox.  U-net: Convolutional networks for biomedical image segmentation, 2015\.  cite arxiv:1505.04597Comment: conditionally accepted at MICCAI 2015.
  * [3] Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick.  Mask r-cnn, 2018.
  * [4] Vijay Badrinarayanan, Alex Kendall, and Roberto Cipolla.  Segnet: A deep convolutional encoder-decoder architecture for image segmentation, 2016.
  * [5] Liang-Chieh Chen, Yukun Zhu, George Papandreou, Florian Schroff, and Hartwig Adam.  Encoder-decoder with atrous separable convolution for semantic image segmentation, 2018.
  * [6] Ze Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo.  Swin transformer: Hierarchical vision transformer using shifted windows, 2021.
  * [7] Jonathan Long, Evan Shelhamer, and Trevor Darrell.  Fully convolutional networks for semantic segmentation, 2015.
  * [8] Bin Xiao, Haiping Wu, and Yichen Wei.  Simple baselines for human pose estimation and tracking, 2018.
  * [9] André Mewes, Bennet Hensen, Frank Wacker, and Christian Hansen.  Touchless interaction with software in interventional radiology and surgery: a systematic literature review.  International journal of computer assisted radiology and surgery, 12(2):291–305, February 2017.
  * [10] Rui Li, Kaisheng Zhang, Shuang Li, Minh-Thang Luong, and Quoc V Le.  Openai whisper: Speech-to-text transformer.  arXiv preprint arXiv:2006.04558, 2020.
  * [11] Martin A. Fischler and Robert C. Bolles.  Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography.  Commun. ACM, 24(6):381–395, jun 1981.
  * [12] John Brooke.  Sus: A quick and dirty usability scale.  Usability Eval. Ind., 189, 11 1995.
  * [13] Bettina Laugwitz, Theo Held, and Martin Schrepp.  Construction and evaluation of a user experience questionnaire.  volume 5298, pages 63–76, 11 2008.