# 추천시스템을 위한 신뢰도 보정 및 활용

권  Kweon<EMAIL_ADDRESS>
# Confidence Calibration for Recommender Systems and Its Applications

권  Kweon<EMAIL_ADDRESS>
###### Abstract

Personalized recommendations have a significant impact on various daily
activities such as shopping, news, search, videos, and music. However, most
recommender systems only display the top-scored items to the user without
providing any indication of confidence in the recommendation results. The
semantic of the same ranking position differs for each user; one user might
like his third item with a probability of 30%, whereas the other user may like
her third item with 90%.

Despite the importance of having a measure of confidence in recommendation
results, it has been surprisingly overlooked in the literature compared to the
accuracy of the recommendation. In this dissertation, I propose a model
calibration framework for recommender systems for estimating accurate
confidence in recommendation results based on the learned ranking scores.
Moreover, I subsequently introduce two real-world applications of confidence
on recommendations: (1) Training a small student model by treating the
confidence of a big teacher model as additional learning guidance, (2)
Adjusting the number of presented items based on the expected user utility
estimated with calibrated probability.

Obtaining Calibrated Confidence. I investigate various parametric
distributions and propose two parametric calibration methods, namely Gaussian
calibration and Gamma calibration. Each proposed method can be seen as a post-
processing function that maps the ranking scores of pre-trained models to
well-calibrated preference probabilities, without affecting the recommendation
performance.

Bidirectional Distillation. I propose Bidirectional Distillation (BD)
framework whereby both the teacher and the student collaboratively improve
with each other. Specifically, each model is trained with the distillation
loss that makes it follow the other’s prediction confidence along with its
original loss function. Trained in a bidirectional way, it turns out that both
the teacher and the student are significantly improved compared to when being
trained separately.

Top-Personalized-K Recommendation. I introduce Top-Personalized-$K$
Recommendation (PerK), a new recommendation task aimed at generating a
personalized-sized ranking list to maximize individual user satisfaction. PerK
estimates the expected user utility by leveraging calibrated interaction
probabilities, subsequently selecting the recommendation size that maximizes
this expected utility. We expect that Top-Personalized-$K$ recommendation has
the potential to offer enhanced solutions for various real-world
recommendation scenarios, based on its great compatibility with existing
models.

원 빈 Wonbin [major]유 환 조Hwanjo Yunosign CITE 20192689 [1]Hwanjo Yu [2]Young
Myoung Ko [3]Gary Geunbae Lee [4]Dongwoo Kim [5]Namhoon Lee 20231207 20231207
2024

###### Contents

  1. 1 Introduction
  2. 2 Obtaining Calibrated Probabilities with Personalized Ranking Models
    1. 2.1 Introduction
    2. 2.2 Preliminary & Related Work
      1. 2.2.1 Personalized Ranking
      2. 2.2.2 Calibrated Probability
      3. 2.2.3 Calibration Method
    3. 2.3 Proposed Calibration Method
      1. 2.3.1 Revisiting Platt Scaling
      2. 2.3.2 Gaussian Calibration
      3. 2.3.3 Gamma Calibration
      4. 2.3.4 Other Distributions
      5. 2.3.5 Monotonicity for Proposed Desiderata
    4. 2.4 Unbiased Parameter Fitting
      1. 2.4.1 Naive Log-loss
      2. 2.4.2 Ideal Log-loss for Preference Estimation
      3. 2.4.3 Unbiased Empirical Risk Minimization
    5. 2.5 Experiment
      1. 2.5.1 Experimental Setup
      2. 2.5.2 Comparing Calibration Performance
      3. 2.5.3 Reliability Diagram
      4. 2.5.4 Score Distribution & Fitted Function
      5. 2.5.5 Propensity Estimation
      6. 2.5.6 User Degree vs Calibrated Confidence
      7. 2.5.7 Case Study
    6. 2.6 Conclusion
  3. 3 Bidirectional Distillation for Top-K Recommender System
    1. 3.1 Introduction
    2. 3.2 Analysis: Teacher vs Student
    3. 3.3 Problem Formulation
    4. 3.4 Method
      1. 3.4.1 Overview
      2. 3.4.2 Distillation Loss
      3. 3.4.3 Rank Discrepancy-aware Sampling
      4. 3.4.4 Model Training
    5. 3.5 Experiments
      1. 3.5.1 Experimental Setup
      2. 3.5.2 Performance Comparison
      3. 3.5.3 Model Size Analysis
      4. 3.5.4 Synchronization Analysis
      5. 3.5.5 Sampling Scheme Analysis
      6. 3.5.6 Hyperparameter Analysis
      7. 3.5.7 Additional Result
    6. 3.6 Related Work
    7. 3.7 Conclusion & Future Work
  4. 4 Top-Personalized-K Recommendation
    1. 4.1 Introduction
    2. 4.2 Related Work
    3. 4.3 Preliminaries
      1. 4.3.1 Recommendation with Implicit Feedback
      2. 4.3.2 User Utility
      3. 4.3.3 Top-K Recommendation
    4. 4.4 Proposed Task
    5. 4.5 Proposed Framework
      1. 4.5.1 Overview
      2. 4.5.2 Expected User Utility
      3. 4.5.3 Calibrated Interaction Probability
      4. 4.5.4 Optimization Procedure
    6. 4.6 Experiments
      1. 4.6.1 Experiment Setup
      2. 4.6.2 Comparison of User Utility
      3. 4.6.3 Personalized Recommendation Size
      4. 4.6.4 User Degree vs Personalized Size
      5. 4.6.5 Ablation Study for Calibration
      6. 4.6.6 Space and Time analysis
      7. 4.6.7 Case Study
    7. 4.7 Conclusion
  5. 5 Conclusion

###### List of Tables

  1. 2.1 Satisfaction of proposed desiderata for each calibration method. $\checkmark$ indicates the satisfaction and a blank denotes the unsatisfaction.
  2. 2.2 Ranking performance of each personalized ranking model. Numbers in boldface are the best results.
  3. 2.3 Expected Calibration Error of each calibration method applied on five personalized ranking models. Numbers in boldface are the best results and Improv denotes the improvement of the best proposed method over the best competitor (Platt or Beta with $\mathcal{L}_{\text{UERM}}$).
  4. 2.4 Maximum Calibration Error with $M=15$ of each calibration method applied on five personalized ranking models. Numbers in boldface are the best results.
  5. 2.5 Negative Log-Likelihood of each calibration method applied on five personalized ranking models. Numbers in boldface are the best results.
  6. 3.1 Data Statistics
  7. 3.2 Performance comparison. Improv.T denotes the improvement of the teacher over the original teacher model (”Teacher” in this table), Improv.B denotes the improvement of the student over the best competitive method and Improv.S denotes the improvement of the student over the original student model (”Student” in this table). $*$ and $**$ indicate $p\leq$ 0.05 and $p\leq$ 0.01 for the paired t-test of BD vs. the best competitor.
  8. 3.3 Model size and online inference efficiency. Time denotes the wall time used for generating recommendation list for every user.
  9. 3.4 Average Rank Difference before and after the training with BD.
  10. 3.5 Recommendation performance (H@50) of BD with different sampling schemes on CiteULike. Numbers in bold face are the best results.
  11. 3.6 Performance comparison with CD.
  12. 4.1 Data statistics after the preprocessing.
  13. 4.2 Average user utility of recommendations produced by compared methods when they are optimized for each target utility measure. Improv. denotes the improvement of PerK over MtCut. The numbers in boldface denote the best result in each setting. *We conduct the paired t-test with a 0.05 level and every Improv. is statistically significant.
  14. 4.3 Average user utility of recommendations produced by compared methods when they are optimized for each target utility measure. Improv. denotes the improvement of PerK over MtCut. The numbers in boldface denote the best result in each setting. *We conduct the paired t-test with a 0.05 level and every Improv. is statistically significant.
  15. 4.4 Average user utility of recommendations produced by compared methods when they are optimized for each target utility measure. Improv. denotes the improvement of PerK over MtCut. The numbers in boldface denote the best result in each setting. *We conduct the paired t-test with a 0.05 level and every Improv. is statistically significant.
  16. 4.5 Calibration error and user utility for each calibration method. User utilities are computed when each calibration method is applied to PerK. Lower is better for ECE.
  17. 4.6 Space and Time analysis. #Params. denotes the number of learnable parameters and Time denotes the wall time (in ms) used for generating a ranking list for a user.

###### List of Figures

  1. 2.1 Reliability diagram of each calibration method. Gap denotes the discrepancy between the accuracy and the average calibrated probability for each bin. The grey dashed line is a diagonal function that indicates the ideal reliability line where the blue accuracy bar should meet.
  2. 2.2 Ranking score distributions of negative and positive pairs.
  3. 2.3 Fitted function of each calibration method.
  4. 2.4 ECE with various propensity estimation techniques. In this work, we utilize item popularities to estimate the propensity scores.
  5. 2.5 User degree versus mean top-10 confidence.
  6. 2.6 Case study. Top-5 items for each user with ranking score $s$ and calibrated probability $g_{\phi}(s)$.
  7. 3.1 Rank difference between teacher and student. Rank differences are computed on the test interactions and the test interactions (x-axis) are sorted in increasing order by the rank difference.
  8. 3.2 Rank comparison between teacher and student. Each dot represents a sampled unobserved interaction.
  9. 3.3 Illustration of Bidirectional Distillation (BD) for top-$K$ recommender systems.
  10. 3.4 Performance of teacher with student of varying size. The blue dashed line indicates the performance of the model trained separately.
  11. 3.5 Performance of student with teacher of varying size. The blue dashed line indicates the performance of the model trained separately.
  12. 3.6 Rank difference after the training with BD.
  13. 3.7 Recommendation performance (H@50) of teacher and student with varying lambda.
  14. 3.8 Recommendation performance (H@50) of teacher and student with varying smoothing factor.
  15. 3.9 Recommendation performance (H@50) of teacher and student with varying updating period.
  16. 4.1 An example of top-personalized-$K$ recommendation. 1 and 0 represent the relevant and the irrelevant items, respectively. Note that these labels are not available at the inference phase. Newly added items have red outlines.
  17. 4.2 Distribution of recommendation sizes from each method on MovieLens 25M. Three figures in the same row share the y-axis.
  18. 4.3 Distribution of recommendation sizes from each method on Amazon Books. Three figures in the same row share the y-axis.
  19. 4.4 User degree versus Personalized Recommendation Size. Red dots denote the average x (user degree) for each y (personalized K).
  20. 4.5 Hyperparameter study of $M$
  21. 4.6 Case study. 1 and 0 represent the relevant and the irrelevant labels, which are unavailable at the inference phase. Decimal numbers indicate the F1 score.
  22. 5.1 An overview of this dissertation.

##  Chapter 1 Introduction

Personalized recommendations have a significant impact on various daily
activities such as shopping, news, search, videos, and music. With an enormous
number of items in the system, users cannot possibly look up the entire item
set, and therefore they need to rely on the recommendations provided by the
system. However, most recommender systems only display the top-scored items to
the user without providing any indication of confidence in the recommendation
results. The semantic of the same ranking position differs for each user; one
user might like his third item with a probability of 30%, whereas the other
user may like her third item with 90%. Also, recommendation accuracy differs
for each user based on their activeness and preference complexity. Therefore,
I claim that real-world applications need to indicate their confidence in
their recommendation results to provide reliability and interpretability to
users.

Despite the importance of having a measure of confidence in recommendation
results, it has been surprisingly overlooked in the literature compared to the
accuracy of the recommendation. Recommender models aim to learn the ranking
score for each user-item pair, so as to produce a ranking list of items for
each user. This ranking score cannot be used as a measure of probability or
confidence as it is usually an unbounded real number used for the item sorting
process. In this dissertation, I propose a model calibration framework for
recommender systems for estimating accurate confidence in recommendation
results based on the learned ranking scores. Estimated confidence can be
presented along with the recommendation results to increase reliability and
interoperability. Moreover, I subsequently introduce two real-world
applications of confidence on recommendations: (1) Training a small student
model by treating the confidence of a big teacher model as additional learning
guidance, (2) Adjusting the number of presented items based on the expected
user utility estimated with calibrated probability.

(Section ii@) Obtaining Calibrated Probabilities with Personalized Ranking
Models. In this paper, we aim to estimate the calibrated probability of how
likely a user will prefer an item. We investigate various parametric
distributions and propose two parametric calibration methods, namely Gaussian
calibration and Gamma calibration. Each proposed method can be seen as a post-
processing function that maps the ranking scores of pre-trained models to
well-calibrated preference probabilities, without affecting the recommendation
performance. We also design the unbiased empirical risk minimization framework
that guides the calibration methods to the learning of true preference
probability from the biased user-item interaction dataset. Extensive
evaluations with various personalized ranking models on real-world datasets
show that both the proposed calibration methods and the unbiased empirical
risk minimization significantly improve the calibration performance.

(Section iii@) Bidirectional Distillation for Top-K Recommendation.
Recommender systems (RS) have started to employ knowledge distillation, which
is a model compression technique training a compact model (student) with the
knowledge transferred from a cumbersome model (teacher). The state-of-the-art
methods rely on unidirectional distillation transferring the knowledge only
from the teacher to the student, with an underlying assumption that the
teacher is always superior to the student. However, we demonstrate that the
student performs better than the teacher on a significant proportion of the
test set, especially for RS. Based on this observation, we propose
Bidirectional Distillation (BD) framework whereby both the teacher and the
student collaboratively improve with each other. Specifically, each model is
trained with the distillation loss that makes it follow the other’s prediction
confidence along with its original loss function. For effective bidirectional
distillation, we propose rank discrepancy-aware sampling scheme to distill
only the informative knowledge that can fully enhance each other. The proposed
scheme is designed to effectively cope with a large performance gap between
the teacher and the student. Trained in the bidirectional way, it turns out
that both the teacher and the student are significantly improved compared to
when being trained separately. Our extensive experiments on real-world
datasets show that our proposed framework consistently outperforms the state-
of-the-art competitors. We also provide analyses for an in-depth understanding
of BD and ablation studies to verify the effectiveness of each proposed
component.

(Section iv@) Top-Personalized-K Recommendation. The conventional top-K
recommendation, which presents the top-K items with the highest ranking
scores, is a common practice for generating personalized ranking lists.
However, is this fixed-size top-$K$ recommendation the optimal approach for
every user’s satisfaction? Not necessarily. We point out that providing fixed-
size recommendations without taking into account user utility can be
suboptimal, as it may unavoidably include irrelevant items or limit the
exposure to relevant ones. To address this issue, we introduce Top-
Personalized-$K$ Recommendation, a new recommendation task aimed at generating
a personalized-sized ranking list to maximize individual user satisfaction. As
a solution to the proposed task, we develop a model-agnostic framework named
PerK. PerK estimates the expected user utility by leveraging calibrated
interaction probabilities, subsequently selecting the recommendation size that
maximizes this expected utility. Through extensive experiments on real-world
datasets, we demonstrate the superiority of PerK in Top-Personalized-$K$
recommendation task. We expect that Top-Personalized-$K$ recommendation has
the potential to offer enhanced solutions for various real-world
recommendation scenarios, based on its great compatibility with existing
models.

##  Chapter 2 Obtaining Calibrated Probabilities with Personalized Ranking
Models

For personalized ranking models, the well-calibrated probability of an item
being preferred by a user has great practical value. While existing work shows
promising results in image classification, probability calibration has not
been much explored for personalized ranking. In this paper, we aim to estimate
the calibrated probability of how likely a user will prefer an item. We
investigate various parametric distributions and propose two parametric
calibration methods, namely Gaussian calibration and Gamma calibration. Each
proposed method can be seen as a post-processing function that maps the
ranking scores of pre-trained models to well-calibrated preference
probabilities, without affecting the recommendation performance. We also
design the unbiased empirical risk minimization framework that guides the
calibration methods to the learning of true preference probability from the
biased user-item interaction dataset. Extensive evaluations with various
personalized ranking models on real-world datasets show that both the proposed
calibration methods and the unbiased empirical risk minimization significantly
improve the calibration performance. This work was published at AAAI
Conference on Artificial Intelligence (AAAI 2022) with an oral presentation
[1].

###  2.1 Introduction

Personalized ranking models aim to learn the ranking scores of items, so as to
produce a ranked list of them for the recommendation [2]. However, their
prediction results provide an incomplete estimation of the user’s potential
preference for each item; the semantic of the same ranking position differs
for each user. One user might like his third item with the probability of 30%,
whereas the other user likes her third item with 90%. Accurately estimating
the probability of an item being preferred by a user has great practical value
[3]. The preference probability can help the user choose the items with high
potential preference and the system can raise user satisfaction by pruning the
ranked list by filtering out items with low confidence [4]. To ensure
reliability, the predicted probabilities need to be calibrated so that they
can accurately indicate their ground truth correctness likelihood. In this
paper, our goal is to obtain the well-calibrated probability of an item
matching a user’s preference based on the ranking score of the pre-trained
model, without affecting the ranking performance.

While recent methods [5, 6, 7] have successfully achieved model calibration
for image classification, it has remained a long-standing problem for
personalized ranking. A pioneering work [3] firstly proposed to predict
calibrated probabilities from the scores of pre-trained ranking models by
using isotonic regression [8], which is a simple non-parametric method that
fits a monotonically increasing function. Although it has shown some
effectiveness, there is no subsequent study about parametric calibration
methods in the field of personalized ranking despite their richer
expressiveness than non-parametric methods.

In this paper, we investigate various parametric distributions, and from which
we propose two calibration methods that can best model the score distributions
of the ranking models. First, we define three desiderata that a calibration
function for ranking models should meet, and show that existing calibration
methods have the insufficient capability to model the diverse populations of
the ranking score. We then propose two parametric methods, namely Gaussian
calibration and Gamma calibration, that satisfy all the desiderata. We
demonstrate that the proposed methods have a larger expressive power in terms
of the parametric family and also effectively handles the imbalanced nature of
ranking score populations compared to the existing methods [9, 5]. Our methods
are post-processing functions with three learnable parameters that map the
ranking scores of pre-trained models to calibrated posterior probabilities.

To optimize the parameters of the calibration functions, we can use the log-
loss on the held-out validation sets [5]. The challenge here is that the user-
item interaction datasets are implicit and missing-not-at-random [10, 11]. For
each user-item pair, the label is 1 if the interaction is observed, 0
otherwise. An unobserved interaction, however, does not necessarily mean a
negative preference, but the item might have not been exposed to the user yet.
Therefore, if we fit the calibration function with the log-loss computed
naively on the implicit datasets, the mapped probabilities may indicate biased
likelihoods of users’ preference on items. To tackle this problem, we design
an unbiased empirical risk minimization framework by adopting Inverse
Propensity Scoring [12]. We first decompose the interaction variable into two
variables for observation and preference, and adopt an inverse propensity-
scored log-loss that guides the calibration functions toward the true
preference probability.

Extensive evaluations with various personalized ranking models on real-world
datasets show that the proposed calibration methods produce more accurate
probabilities than existing methods in terms of calibration measures like ECE,
MCE, and NLL. Our unbiased empirical risk minimization framework successfully
estimates the ideal empirical risk, leading to performance gain over the naive
log-loss. Furthermore, reliability diagrams show that Gaussian calibration and
Gamma calibration predict well-calibrated probabilities across all probability
ranges. Lastly, we provide an in-depth analysis that supports the superiority
of the proposed methods over the existing methods.

###  2.2 Preliminary & Related Work

####  2.2.1 Personalized Ranking

Let $\mathcal{U}$ and $\mathcal{I}$ denote the user space and the item space,
respectively. For each user-item pair $(u,i)$ of $u\in\mathcal{U}$ and
$i\in\mathcal{I}$, a label $Y_{u,i}$ is given as 1 if their interaction is
observed and 0 otherwise. It is worth noting that unobserved interaction
($Y_{u,i}=0$) may indicate the negative preference or the unawareness, or
both. A personalized ranking model
$f_{\theta}:\mathcal{U}\times\mathcal{I}\rightarrow\mathbb{R}$ learns the
ranking scores of user-item pairs to produce a ranked list of items for each
user. $f_{\theta}$ is mostly trained with pairwise loss that makes the model
put a higher score on the observed pair than the unobserved pair:

$\mathcal{L}_{\textup{pair}}=\sum_{u\in\mathcal{U},i,j\in\mathcal{I}}\ell(f_{\theta}(u,i),f_{\theta}(u,j))Y_{u,i}(1-Y_{u,j}),$
(2.1)

where $\ell(\cdot,\cdot)$ is some convex loss function such as BPR loss [2] or
Margin Ranking loss [13]. Note that the ranking score
$f_{\theta}(u,i)\in\mathbb{R}$ is not bounded in $[0,1]$ and therefore cannot
be used as a probability.

####  2.2.2 Calibrated Probability

To estimate $P(Y_{u,i}=1|f_{\theta}(u,i))$, which is the probability of item
$i$ being interacted with user $u$ given the pre-trained ranking score, we
need a post-processing calibration function $g_{\phi}(s)$ that maps the
ranking score $s=f_{\theta}(u,i)$ to the calibrated probability $p$. Here, the
calibration function for the personalized ranking has to meet the following
desiderata: (1) the function $g_{\phi}:\mathbb{R}\rightarrow[0,1]$ needs to
take an input from the unbounded range of the ranking score to output a
probability; (2) the function should be monotonically increasing so that the
item with a higher ranking score gets a higher preference probability; (3) the
function needs enough expressiveness to represent diverse score distributions
with asymmetricity.

We say the probability $p$ is well-calibrated if it indicates the ground-truth
correctness likelihood [14]:

$\mathbb{E}[Y|g_{\phi}(s)=p]=p,\;\;\;\forall p\in[0,1].$ (2.2)
$g_{\phi}(f_{\theta}(u,i))=p.$ (2.3)

For example, if we have 100 predictions with $p=0.3$, we expect 30 of them to
indeed have $Y=1$ when the probabilities are calibrated. Using this
definition, we can measure the miscalibration of a model with Expected
Calibration Error (ECE) [15]:

$\text{ECE}(g_{\phi})=\mathbb{E}\big{[}\absolutevalue{\mathbb{E}[Y|g_{\phi}(s)=p]-p}\big{]}.$
(2.4)

However, since we only have finite samples, we cannot directly compute ECE
with Eq.3. Instead, we partition the [0,1] range of $p$ into $M$ equi-spaced
bins and aggregate the value of each bin:

$\text{ECE}_{M}(g_{\phi})=\sum_{m=1}^{M}\frac{\absolutevalue{B_{m}}}{N}\left|\frac{\sum_{k\in
B_{m}}Y_{k}}{\absolutevalue{B_{m}}}-\frac{\sum_{k\in
B_{m}}p_{k}}{\absolutevalue{B_{m}}}\right|,$ (2.5)

where $B_{m}$ is $m$-th bin and $N$ is the number of samples. The first term
in the absolute value symbols denotes the ground-truth proportion of positive
samples (accuracy) in $B_{m}$ and the second term denotes the average
calibrated probability (confidence) of $B_{m}$. Similarly, Maximum Calibration
Error (MCE) is defined as follows:

$\text{MCE}_{M}(g_{\phi})=\max_{m\in\\{1,..,M\\}}\left|\frac{\sum_{k\in
B_{m}}Y_{k}}{\absolutevalue{B_{m}}}-\frac{\sum_{k\in
B_{m}}p_{k}}{\absolutevalue{B_{m}}}\right|.$ (2.6)

MCE measures the worst-case discrepancy between the accuracy and the
confidence. Besides the above calibration measures, Negative Log-Likelihood
(NLL) also can be used as a calibration measure [5].

####  2.2.3 Calibration Method

Existing methods for model calibration are categorized into two groups: non-
parametric and parametric methods. Non-parametric methods mostly adopt the
binning scheme introduced by the histogram binning [16]. The histogram binning
divides the uncalibrated model outputs into $B$ equi-spaced bins and samples
in each bin take the proportion of positive samples in the bin as the
calibrated probability. Subsequently, isotonic regression [3] adjusts the
number of bins and their width, Bayesian binning into quantiles (BBQ) [15]
takes an average of different binning models for the better generalization.
The non-parametric calibration methods (Hist, Isotonic, and BBQ) lack rich
expressiveness since they rely on the binning scheme, which maps the ranking
scores to the probabilities in a discrete manner. Also, histogram binning [16]
and BBQ [15] do not guarantee the monotonicity and BBQ takes the input only
from [0,1]. From the perspective of our desiderata, none of them meets all
three conditions.

Method | (1) | (2) | (3)
---|---|---|---
Input range | Monotonicity | Asymmetricity
Hist | ✓ |  | ✓
Isotonic | ✓ | ✓ | ✓
BBQ |  |  | ✓
Platt | ✓ | ✓ |
Temp. S | ✓ | ✓ |
Beta |  | ✓ | ✓
Gaussian | ✓ | ✓ | ✓
Gamma | ✓ | ✓ | ✓
Table 2.1: Satisfaction of proposed desiderata for each calibration method.
$\checkmark$ indicates the satisfaction and a blank denotes the
unsatisfaction.

The parametric methods try to fit calibration functions that map the output
scores to the calibrated probabilities. Temperature scaling ($\sigma(s/T)$)
[5], a well-known technique for calibrating deep neural networks, is a
simplified version of Platt scaling ($\sigma(as+b)$) [9] that adopts Gaussian
distributions with the same variance for the positive and the negative
classes. Beta calibration [14] utilizes Beta distribution for the binary
classification and Dirichlet calibration [6] generalizes it for the multi-
class classification. While recent work [7, 17] is focusing on parametric
methods and shows promising results for image classification, they cannot be
directly adopted for personalized ranking. In this paper, we propose two
parametric calibration methods that satisfy all the desiderata for the
personalized ranking models.

###  2.3 Proposed Calibration Method

####  2.3.1 Revisiting Platt Scaling

Platt scaling [9] is widely used parametric calibration method, which is a
generalized form of the temperature scaling [5]:

$g_{\phi}^{\text{Platt}}(s)=\sigma(bs+c),$ (2.7)

where $\phi=\\{b,c\\}$ are learnable parameters and
$\sigma(x)=1/(1+\text{exp}(-x))$ is the sigmoid function. In this section, we
show that Platt scaling can be derived from the assumption that the class-
conditional scores follow Gaussian distributions with the same variance.

We first set the class-conditional score distribution for the positive and the
negative classes:

$\begin{split}&p(s|Y=0)=(\sqrt{2\pi}\sigma_{0})^{-1}\text{exp}[-(s-\mu_{0})^{2}/2\sigma_{0}^{2}],\\\
&p(s|Y=1)=(\sqrt{2\pi}\sigma_{1})^{-1}\text{exp}[-(s-\mu_{1})^{2}/2\sigma_{1}^{2}],\\\
\end{split}$ (2.8)

where $\mu_{0},\mu_{1}\in\mathbb{R}$,
$\sigma_{0}^{2},\sigma_{1}^{2}\in\mathbb{R}^{+}$ are the mean and the variance
of each Gaussian distribution. Then, the posterior is computed as follows:

$\begin{split}P(Y=1|s)&=\frac{\pi_{1}p(s|Y=1)}{\pi_{1}p(s|Y=1)+\pi_{0}p(s|Y=0)}\\\
&=\frac{1}{1+\pi_{0}p(s|Y=0)/\pi_{1}p(s|Y=1)}\\\
&=\frac{1}{1+\text{exp}\big{[}(\frac{1}{2\sigma_{1}^{2}}-\frac{1}{2\sigma_{0}^{2}})s^{2}+(\frac{\mu_{0}}{\sigma_{0}^{2}}-\frac{\mu_{1}}{\sigma_{1}^{2}})s-c\big{]}}\\\
&=\sigma(as^{2}+bs+c),\end{split}$ (2.9)

where $\pi_{0}$ and $\pi_{1}$ are the prior probability for each class,
$a=(2\sigma_{0}^{2})^{-1}-(2\sigma_{1}^{2})^{-1}$,
$b=\mu_{1}/\sigma_{1}^{2}-\mu_{0}/\sigma_{0}^{2}$, and
$c=\mu_{1}^{2}/(2\sigma_{1}^{2})-\mu_{0}^{2}/(2\sigma_{0}^{2})+\text{log}(\pi_{0}\sigma_{1})-\text{log}(\pi_{1}\sigma_{0})\in\mathbb{R}$.
We can see that Platt scaling is a special case of Eq.2.9 with the assumption
$a=0$ (i.e., the same variance for both class-conditional score
distributions).

####  2.3.2 Gaussian Calibration

For personalized ranking, however, the usage of the same variance for both
class-conditional score distributions is not desirable, because a severe
imbalance between the two classes exists in user-item interaction datasets.
Since users have distinct preferences for item categories, preferred items
take only a small portion (up to 10% in real-world datasets) of the entire
itemset. Therefore, the score distribution of diverse unpreferred items and
that of distinct preferred items are likely to have disparate variances.

To tackle this problem, we let the variance of each class-conditional score
distribution be optimized with datasets, without any naive assumption of the
same variance for both classes:

$g_{\phi}^{\text{Gaussian}}(s)=\sigma(as^{2}+bs+c),$ (2.10)

where $\phi=\\{a,b,c\\}$ are learnable parameters and can be any real numbers.
Since $a=(2\sigma_{0}^{2})^{-1}-(2\sigma_{1}^{2})^{-1}$ can capture the
different deviations of two classes during the training, we can handle the
distinct distribution of each class.

####  2.3.3 Gamma Calibration

Gamma distribution is also widely adopted to model the score distribution of
ranking models [18]. Unlike Gaussian distribution that is symmetric about its
mean, Gamma distribution can capture the skewed population of ranking scores
that might exist in the datasets. In this section, we set the class-
conditional score distribution to Gamma distribution:

$\begin{split}&p(s|Y=0)=\Gamma(\alpha_{0})^{-1}\beta_{0}^{\alpha_{0}}s^{\alpha_{0}-1}\text{exp}(-\beta_{0}s),\\\
&p(s|Y=1)=\Gamma(\alpha_{1})^{-1}\beta_{1}^{\alpha_{1}}s^{\alpha_{1}-1}\text{exp}(-\beta_{1}s),\\\
\end{split}$ (2.11)

where $\Gamma(\cdot)$ is the Gamma function,
$\alpha_{0},\alpha_{1},\beta_{0},\beta_{1}\in\mathbb{R}^{+}$ are the shape and
the rate parameters of each Gamma distribution. Then, the posterior is
computed as follows:

$\begin{split}P(Y=1|s)&=\frac{1}{1+\pi_{0}p(s|Y=0)/\pi_{1}p(s|Y=1)}\\\
&=\frac{1}{1+\frac{\pi_{0}\beta_{0}^{\alpha_{0}}\Gamma(\alpha_{1})}{\pi_{1}\beta_{1}^{\alpha_{1}}\Gamma(\alpha_{0})}s^{\alpha_{0}-\alpha_{1}}\text{exp}[(\beta_{1}-\beta_{0})s]}\\\
&=\frac{1}{1+\text{exp}\big{[}(\alpha_{0}-\alpha_{1})\text{log}s+(\beta_{1}-\beta_{0})s-c\big{]}}\\\
&=\sigma(a\text{log}s+bs+c),\end{split}$ (2.12)

where $a=\alpha_{1}-\alpha_{0}$, $b=\beta_{0}-\beta_{1}$, and
$c=\text{log}(\pi_{1}\beta_{1}^{\alpha_{1}}\Gamma(\alpha_{0})/\pi_{0}\beta_{0}^{\alpha_{0}}\Gamma(\alpha_{1}))\in\mathbb{R}$.
Therefore, Gamma calibration can be formalized as follows:

$g_{\phi}^{\text{Gamma}}(s)=\sigma(a\text{log}s+bs+c),$ (2.13)

where $\phi=\\{a,b,c\\}$ are learnable parameters. Since Gamma distribution is
defined only for the positive real number, we need to shift the score to make
all the inputs positive: $s\leftarrow s-s_{\text{min}}$, where
$s_{\text{min}}$ is the minimum ranking score.

####  2.3.4 Other Distributions

Besides Gaussian and Gamma distribution, Swets [19] adopts Exponential
distributions for the score distribution of both classes:

$\begin{split}&p(s|Y=0)=\lambda_{0}\text{exp}(-\lambda_{0}s),\\\
&p(s|Y=1)=\lambda_{1}\text{exp}(-\lambda_{1}s),\\\ \end{split}$ (2.14)

where $\lambda_{0},\lambda_{1}\in\mathbb{R}^{+}$ are the rate parameters for
each Exponential distribution. Note that for Gamma distribution and
Exponential distribution, we need to shift the score to make all the inputs
positive: $s\leftarrow s-s_{min}$, where $s_{min}$ is the minimum ranking
score. Then, the posterior can be computed as follows:

$\begin{split}P(Y=1|s)&=\frac{1}{1+\pi_{0}p(s|Y=0)/\pi_{1}p(s|Y=1)}\\\
&=\frac{1}{1+\text{exp}[(\lambda_{1}-\lambda_{0})s+\text{log}(\pi_{0}\lambda_{0}/\pi_{1}\lambda_{1})]}\\\
&=\sigma(bs+c),\end{split}$ (2.15)

where $b=\lambda_{0}-\lambda_{1}$ and
$c=\text{log}(\pi_{1}\lambda_{1}/\pi_{0}\lambda_{0})\in\mathbb{R}$. It is the
exact same form as Platt scaling.

On the other hand, Manmatha [20] proposes Exponential distribution for the
negative class and Gaussian distribution for the positive class:

$\begin{split}&p(s|Y=0)=\lambda_{0}\text{exp}(-\lambda_{0}s),\\\
&p(s|Y=1)=(\sqrt{2\pi}\sigma_{1})^{-1}\text{exp}[-(s-\mu_{1})^{2}/2\sigma_{1}^{2}],\\\
\end{split}$ (2.16)

where $\lambda_{0},\mu_{1},\sigma_{1}^{2}\in\mathbb{R}^{+}$. Then, we have the
posterior as follows:

$\begin{split}P(Y=1|s)&=\frac{1}{1+\pi_{0}p(s|Y=0)/\pi_{1}p(s|Y=1)}\\\
&=\frac{1}{1+\frac{\pi_{0}\lambda_{0}\sqrt{2\pi}\sigma_{1}}{\pi_{1}}\text{exp}[(s-\mu_{1})^{2}/2\sigma_{1}^{2}-\lambda_{0}s]}\\\
&=\frac{1}{1+\text{exp}[(2\sigma_{1}^{2})^{-1}s^{2}-(\lambda_{0}+\mu_{1}/\sigma_{1}^{2})s-c]}\\\
&=\sigma(as^{2}+bs+c),\end{split}$ (2.17)

where $a=(-2\sigma_{1}^{2})^{-1}\in\mathbb{R}^{-}$,
$b=\lambda_{0}+\mu_{1}/\sigma_{1}^{2}\in\mathbb{R}^{+}$, and
$c=\text{log}(\pi_{1}/\pi_{0}\lambda_{0}\sqrt{2\pi}\sigma_{1})-\mu_{1}^{2}/(2\sigma_{1}^{2})\in\mathbb{R}$.
With the constraints $a<0$ and $b>0$, the calibration function in Eq.2.17
cannot be monotonically increasing on $s>0$ with any parameters.

Lastly, Kanoulas [21] proposes gamma distribution for the negative class and
Gaussian distribution for the positive class:

$\begin{split}&p(s|Y=0)=\rho(\alpha_{0})^{-1}\beta_{0}^{\alpha_{0}}s^{\alpha_{0}-1}\text{exp}(-\beta_{0}s),\\\
&p(s|Y=1)=(\sqrt{2\pi}\sigma_{1})^{-1}\text{exp}[-(s-\mu_{1})^{2}/2\sigma_{1}^{2}],\\\
\end{split}$ (2.18)

where $\alpha_{0}$, $\beta_{0}$, $\mu_{1}$, $\sigma_{1}^{2}\in\mathbb{R}^{+}$.
The posterior for these likelihoods can be computed as follows:

$\begin{split}P(Y=1|s)&=\frac{1}{1+\pi_{0}p(s|Y=0)/\pi_{1}p(s|Y=1)}\\\
&=\frac{1}{1+\text{exp}[\frac{1}{2\sigma_{1}^{2}}s^{2}-(\beta_{0}+\frac{\mu_{1}}{\sigma_{1}^{2}})s+(\alpha_{0}-1)\text{log}s-c]}\\\
&=\sigma(as^{2}+bs+b^{\prime}\text{log}s+c),\end{split}$ (2.19)

where $a=(-2\sigma_{1}^{2})^{-1}\in\mathbb{R}^{-}$,
$b=\beta_{0}+\mu_{1}/\sigma_{1}^{2}\in\mathbb{R}^{+}$,
$b^{\prime}=1-\alpha_{0}\in\mathbb{R}$, and
$c=\text{log}(\pi_{1}\rho(\alpha_{0})/\pi_{0}\beta_{0}^{\alpha_{0}}\sigma_{1}\sqrt{2\pi})-\mu_{1}^{2}/(2\sigma_{1}^{2})\in\mathbb{R}$.
For this function to be monotonically increasing, we need a non-linear
constraint $b^{\prime}-\frac{b^{2}}{8a}<0$ (derivation of this constraint can
be easily done by taking the derivative of Eq.2.19 w.r.t. the score $s$).
Since the optimization of logistic regression with non-linear constraints is
not straightforward, it is hard for this posterior to satisfy the second
desideratum.

####  2.3.5 Monotonicity for Proposed Desiderata

The proposed calibration methods naturally satisfy the first and the third of
our desiderata: (1) the proposed methods take the unbounded ranking scores and
produce calibrated probabilities; (2) the proposed methods have richer
expressiveness than Platt scaling or temperature scaling, since they have a
larger capacity to express asymmetric distributions. The last condition that
our calibration methods need to meet is that they should be monotonically
increasing to maintain the ranking order. To this end, we need linear
constraints on the parameters of each method: $2as+b>0$ for Gaussian
calibration and $a/s+b>0$ for Gamma calibration. Since these constraints are
linear and we have only three learnable parameters, the optimization of
constrained logistic regression is easily done in at most a few minutes with
the existing module of Scipy [22].

######  Proposition 2.1.

Gaussian calibration $g_{\phi}(s)=\sigma(as^{2}+bs+c)$ is monotonically
increasing only and only if the parameter $a$ and $b$ satisfy the constraint
$2as+b<0$ for $s_{\textup{min}}$ and $s_{\textup{max}}$.

###### Proof.

$\begin{split}&g_{\phi}^{\prime}(s)=\sigma^{\prime}(as^{2}+bs+c)\cdot(2as+b)>0\\\
&\Longleftrightarrow 2as+b>0\\\
&\because\;\sigma^{\prime}(x)=\sigma(x)(1-\sigma(x))>0\;\;\textup{for
all}\;x\in\mathbb{R}\\\ &\Longleftrightarrow 2as_{\textup{min}}+b>0\textup{
and }2as_{\textup{max}}+b>0\\\ &\because 2as+b\textup{ is a linear function of
s in }[s_{\textup{min}},s_{\textup{max}}].\\\ \end{split}$

∎

######  Proposition 2.2.

Gamma calibration $g_{\phi}(s)=\sigma(a\textup{log}s+bs+c)$ is monotonically
increasing only and only if the parameter $a$ and $b$ satisfy the constraint
$a/s+b<0$ for $s_{\textup{min}}$ and $s_{\textup{max}}$.

###### Proof.

$\begin{split}&g_{\phi}^{\prime}(s)=\sigma^{\prime}(a\textup{log}s+bs+c)\cdot(a/s+b)>0\\\
&\Longleftrightarrow a/s+b>0\\\
&\because\;\sigma^{\prime}(x)=\sigma(x)(1-\sigma(x))>0\;\;\textup{for
all}\;x\in\mathbb{R}\\\ &\Longleftrightarrow a/s_{\textup{min}}+b>0\textup{
and }a/s_{\textup{max}}+b>0\\\ &\because a/s+b\textup{ is a monotonic function
of s in }[s_{\textup{min}},s_{\textup{max}}].\\\ \end{split}$

∎

###  2.4 Unbiased Parameter Fitting

####  2.4.1 Naive Log-loss

After we formalize Gaussian Calibration and Gamma Calibration, we need to
optimize their learnable parameters $\phi$. A well-known way to fit them is to
use log-loss on the held-out validation set, which can be the same set used
for the hyperparameter tuning [5, 14]. Since we only observe the interaction
indicator $Y_{u,i}$, the naive negative log-likelihood is computed for a user-
item pair as follows:

$\mathcal{L}_{\text{naive}}=-Y_{u,i}\,\text{log}(g_{\phi}(s_{u,i}))-(1-Y_{u,i})\text{log}(1-g_{\phi}(s_{u,i})).$
(2.20)

where $s_{u,i}=f_{\theta}(u,i)$ is the ranking score for the user-item pair.
Note that during the fitting of the calibration function $g_{\phi}(s)$, the
parameters of the pre-trained ranking model $f_{\theta}(u,i)$ are fixed.

####  2.4.2 Ideal Log-loss for Preference Estimation

The observed interaction label $Y_{u,i}$, however, indicates the presence of
user-item interaction, not the user’s preference on the item. Therefore,
$Y_{u,i}=0$ does not necessarily mean the user’s negative preference, but it
can be that the user is not aware of the item. If we fit the calibration
function with $\mathcal{L}_{\textup{naive}}$, mapped probabilities could be
biased towards the negative preference by treating the unobserved positive
pair as the negative pair. To handle this implicit interaction process, we
borrow the idea of decomposing the interaction variable $Y_{u,i}$ into two
independent binary variables [10]:

$\begin{split}Y_{u,i}&=O_{u,i}\cdot R_{u,i},\\\
P(Y_{u,i}=1)&=P(O_{u,i}=1)\cdot P(R_{u,i}=1)\\\
&=\omega_{u,i}\cdot\rho_{u,i},\end{split}$ (2.21)

where $O_{u,i}$ is a binary random variable representing whether the item $i$
is observed by user $u$, and $R_{u,i}$ is a binary random variable
representing whether the item $i$ is preferred by user $u$. The user-item pair
interacts ($Y_{u,i}=1$) when the item is observed ($O_{u,i}=1$) and preferred
($R_{u,i}=1$) by the user.

The goal of this paper is to estimate the probability of an item being
preferred by a user, not the probability of an item being interacted by a
user. Therefore, we need to train $g_{\phi}(s)$ for predicting $P(R=1|s)$
instead of $P(Y=1|s)$111We can replace $Y_{u,i}$ with $R_{u,i}$ in Eq.2.2 -
2.12.. To this end, we need a new ideal loss function that can guide the
optimization towards the true preference probability:

$\mathcal{L}_{\text{ideal}}=-R_{u,i}\,\text{log}(g_{\phi}(s_{u,i}))-(1-R_{u,i})\text{log}(1-g_{\phi}(s_{u,i})).$
(2.22)

The ideal loss function enables the calibration function to learn the unbiased
preference probability. However, since we cannot observe the variable
$R_{u,i}$ from the training set, the ideal log-loss cannot be computed
directly.

####  2.4.3 Unbiased Empirical Risk Minimization

In this section, we design an unbiased empirical risk minimization (UERM)
framework to obtain the ideal empirical risk minimizer. We deploy the Inverse
Propensity Scoring (IPS) estimator [12], which is a technique for estimating
the counterfactual outcome of a subject under a particular treatment. The IPS
estimator is widely adopted for the unbiased rating prediction [10, 23] and
the unbiased pairwise ranking [24, 11]. For a user-item pair, the inverse
propensity-scored log-loss for the unbiased empirical risk minimization is
defined as follows:

$\mathcal{L}_{\text{UERM}}=-\frac{Y_{u,i}}{\omega_{u,i}}\,\text{log}(g_{\phi}(s_{u,i}))-(1-\frac{Y_{u,i}}{\omega_{u,i}})\text{log}(1-g_{\phi}(s_{u,i})),$
(2.23)

where $\omega_{u,i}=P(O_{u,i}=1)$ is called propensity score.

######  Proposition 2.3.

$\hat{\mathcal{R}}_{\textup{UERM}}(g_{\phi}|\omega)$, which is the empirical
risk of $\mathcal{L}_{\textup{UERM}}$ on validation set with true propensity
score $\omega$, is equal to $\hat{\mathcal{R}}_{\textup{ideal}}(g_{\phi})$,
which is the ideal empirical risk.

###### Proof.

$\begin{split}\hat{\mathcal{R}}_{\text{UERM}}(g_{\phi}|\omega)=&\>\mathbb{E}_{(u,i)\in\mathcal{D}_{\text{val}}}\big{[}\mathcal{L}_{\text{IPS}}(u,i)\big{]}\\\
=&-\frac{1}{\absolutevalue{\mathcal{D}_{\text{val}}}}\sum_{(u,i)\in\mathcal{D}_{\text{val}}}\mathbb{E}\left[\frac{Y_{u,i}}{\omega_{u,i}}\text{log}(g_{\phi}(s_{u,i}))\right.+\left.(1-\frac{Y_{u,i}}{\omega_{u,i}})\text{log}(1-g_{\phi}(s_{u,i}))\right]\\\
=&-\frac{1}{\absolutevalue{\mathcal{D}_{\text{val}}}}\sum_{(u,i)\in\mathcal{D}_{\text{val}}}\frac{\omega_{u,i}\rho_{u,i}}{\omega_{u,i}}\text{log}(g_{\phi}(s_{u,i}))+(1-\frac{\omega_{u,i}\rho_{u,i}}{\omega_{u,i}})\text{log}(1-g_{\phi}(s_{u,i}))\\\
=&-\frac{1}{\absolutevalue{\mathcal{D}_{\text{val}}}}\sum_{(u,i)\in\mathcal{D}_{\text{val}}}\rho_{u,i}\text{log}(g_{\phi}(s_{u,i}))+(1-\rho_{u,i})\text{log}(1-g_{\phi}(s_{u,i}))\\\
=&\>\mathbb{E}_{(u,i)\in\mathcal{D}_{\text{val}}}\big{[}\mathcal{L}_{\text{ideal}}(u,i)\big{]}\\\
=&\>\hat{\mathcal{R}}_{\text{ideal}}(g_{\phi}).\end{split}$

∎

This proposition shows that we can get the unbiased empirical risk minimizer
by
$\phi^{\textup{UERM}}=\textup{argmin}_{\phi}\\{\hat{\mathcal{R}}_{\text{UERM}}(g_{\phi}|\omega)\\}$
when only $Y_{u,i}$ is observed. The remaining challenge is to estimate the
propensity score $\omega_{u,i}$ from the dataset. There have been proposed
several techniques for estimating the propensity score such as Naive Bayes
[10] or logistic regression [25]. However, the Naive Bayes needs unbiased
held-out data for the missing-at-random condition and the logistic regression
needs additional information like user demographics and item categories. In
this paper, we adopt a simple way that utilizes the popularity of items as
done in [11]:
$\hat{\omega}_{u,i}=(\sum_{u\in\mathcal{U}}Y_{u,i}/\text{max}_{i\in\mathcal{I}}\sum_{u\in\mathcal{U}}Y_{u,i})^{0.5}.$
While one can be concerned that this estimate of propensity score may be
inaccurate, Schnabel [10] shows that we merely need to estimate better than
the naive uniform assumption. We provide an experimental result that
demonstrates our estimate of the propensity score shows comparable performance
with Naive Bayes and Logistic Regression that use additional information.

For deeper insights into the variability of the estimated empirical risk, we
investigate the bias when the propensity scores are inaccurately estimated.

######  Proposition 2.4.

The bias of $\,\hat{\mathcal{R}}_{\textup{UERM}}(g_{\phi}|\hat{\omega})$
induced by the inaccurately estimated propensity scores $\hat{\omega}$ is
$\frac{1}{\absolutevalue{\mathcal{D}_{\textup{val}}}}\sum_{(u,i)\in\mathcal{D}_{\textup{val}}}\rho_{u,i}\left(\frac{\omega_{u,i}}{\hat{\omega}_{u,i}}-1\right)\textup{log}\left(\frac{g_{\phi}(s_{u,i})}{1-g_{\phi}(s_{u,i})}\right)$.

###### Proof.

$\begin{split}\text{bias}=&\,\hat{\mathcal{R}}_{\text{UERM}}(g_{\phi}|\hat{\omega})-\hat{\mathcal{R}}_{\text{ideal}}(g_{\phi})\\\
=&\,\frac{1}{\absolutevalue{\mathcal{D}_{\text{val}}}}\sum_{(u,i)\in\mathcal{D}_{\text{val}}}-\frac{\omega_{u,i}\rho_{u,i}}{\hat{\omega}_{u,i}}\text{log}(g_{\phi}(s_{u,i}))\\\
&-(1-\frac{\omega_{u,i}\rho_{u,i}}{\hat{\omega}_{u,i}})\text{log}(1-g_{\phi}(s_{u,i}))+\rho_{u,i}\text{log}(g_{\phi}(s_{u,i}))+(1-\rho_{u,i})\text{log}(1-g_{\phi}(s_{u,i}))\\\
=&\,\frac{1}{\absolutevalue{\mathcal{D}_{\text{val}}}}\sum_{(u,i)\in\mathcal{D}_{\text{val}}}(\omega_{u,i}/\hat{\omega}_{u,i}-1)[-\rho_{u,i}\text{log}(g_{\phi}(s_{u,i}))+\rho_{u,i}\text{log}(1-g_{\phi}(s_{u,i}))]\\\
=&\,\frac{1}{\absolutevalue{\mathcal{D}_{\text{val}}}}\sum_{(u,i)\in\mathcal{D}_{\text{val}}}\rho_{u,i}\left(\frac{\omega_{u,i}}{\hat{\omega}_{u,i}}-1\right)\text{log}\left(\frac{1-g_{\phi}(s_{u,i})}{g_{\phi}(s_{u,i})}\right).\\\
\end{split}$

∎

Obviously, the bias is zero when the propensity score is correctly estimated.
Furthermore, we can see that the magnitude of the bias is affected by the
inverse of the estimated propensity score. This finding is consistent with the
previous work [26] that proposes to adopt a propensity clipping technique to
reduce the variability of the bias. In this work, we use a simple clipping
technique $\hat{\omega}_{u,i}\leftarrow\max\\{\hat{\omega}_{u,i},0.1\\}$ that
can prevent the item with extremely low popularity from amplifying the bias
[11].

###  2.5 Experiment

####  2.5.1 Experimental Setup

Our source code is publicly
available222https://github.com/WonbinKweon/CalibratedRankingModels_AAAI2022.

Datasets. To evaluate the calibration quality of predicted preference
probability, we need an unbiased test set where we can directly observe the
preference variable $R_{u,i}$ without any bias from the observation process
$O_{u,i}$. To the best of our knowledge, there are two real-world datasets
that have separate unbiased test sets where the users are asked to rate
uniformly sampled items (i.e., $O_{u,i}=1$ for test sets). Note that in the
training set, we only observe the interaction $Y_{u,i}$.
Yahoo!R3333http://research.yahoo.com/Academic_Relations has over 300K
interactions in the training set and 54K preferences in the test set from
15.4K users and 1K songs. Coat [10] has over 7K interactions in the training
set and 4.6K preferences in the test set from 290 users and 300 coats. We hold
out 10% of the training set as the validation set for the hyperparameter
tuning of the base models and the optimization of the calibration methods. For
the training and the test set, we convert the ratings over 3 to $Y=1$, and the
ratings under 4 to $Y=0$ as done in the conventional papers [27, 11].

Base models. For rigorous evaluation, we apply the calibration methods on
several widely-used personalized ranking models with various model
architectures and loss functions: Bayesian Personalized Ranking (BPR) [2],
Neural Collaborative Filtering (NCF) [27], Collaborative Metric Learning (CML)
[28], Unbiased BPR (UBPR) [11], and LightGCN (LGCN) [29]. BPR [2] learns the
user and the item embeddings with BPR loss:

$\mathcal{L}_{\textup{BPR}}=\sum_{u\in\mathcal{U},i,j\in\mathcal{I}}-\text{log}\sigma(f_{\theta}(u,i)-f_{\theta}(u,j))Y_{u,i}(1-Y_{u,j}),$
(2.24)

where $f_{\theta}(u,i)=\textbf{e}_{u}\cdot\textbf{e}_{i}$ and
$\textbf{e}_{u}$, $\textbf{e}_{i}$ are user and item embeddings. NCF [27]
learns the ranking score of a user-item pair with the binary cross-entropy
loss:

$\begin{split}\mathcal{L}_{\textup{NCF}}&=\sum_{u\in\mathcal{U},i\in\mathcal{I}}-Y_{u,i}\textup{log}\sigma(f_{\theta}(u,i))\\\
&-(1-Y_{u,i})\textup{log}(1-\sigma(f_{\theta}(u,i))),\end{split}$ (2.25)

where $f_{\theta}(u,i)=\textup{NeuralNet}(\textbf{e}_{u},\textbf{e}_{i})$. CML
[28] learns the user and the item embeddings with the triplet loss:

$\mathcal{L}_{\textup{CML}}=\sum_{u\in\mathcal{U},i,j\in\mathcal{I}}\textup{max}(0,m+f_{\theta}(u,i)-f_{\theta}(u,j))Y_{u,i}(1-Y_{u,j}),$
(2.26)

where $m$ is the margin and
$f_{\theta}(u,i)=||\textbf{e}_{u}-\textbf{e}_{i}||^{2}$. UBPR [11] learns the
user and the item embeddings with the inverse propensity-scored BPR loss:

$\mathcal{L}_{\textup{UBPR}}=\sum_{u\in\mathcal{U},i,j\in\mathcal{I}}-\text{log}\sigma(f_{\theta}(u,i)-f_{\theta}(u,j))\frac{Y_{u,i}}{\omega_{u,i}}(1-\frac{Y_{u,i}}{\omega_{u,i}}),$
(2.27)

where $f_{\theta}(u,i)=\textbf{e}_{u}\cdot\textbf{e}_{i}$ and $\omega_{u,i}$
is the propensity score estimated by the same technique used in our framework.
LGCN [29] learns the user and the item embeddings with BPR loss:

$\mathcal{L}_{\textup{LGCN}}=\sum_{u\in\mathcal{U},i,j\in\mathcal{I}}-\text{log}\sigma(f_{\theta}(u,i)-f_{\theta}(u,j))Y_{u,i}(1-Y_{u,j}),$
(2.28)

where
$f_{\theta}(u,i)=\textup{LGCN}(\textbf{e}_{u})\cdot\textup{LGCN}(\textbf{e}_{i})$
and $\textup{LGCN}(\cdot)$ is simplified Graph Convolutional Networks (GCN).

| Yahoo!R3 | Coat
---|---|---
Metric | BPR | NCF | CML | UBPR | LGCN | BPR | NCF | CML | UBPR | LGCN
NDCG@1 | 0.5070 | 0.5181 | 0.5259 | 0.5328 | 0.5443 | 0.3924 | 0.5341 | 0.5865 | 0.3982 | 0.4008
NDCG@3 | 0.5519 | 0.5812 | 0.5716 | 0.5919 | 0.6002 | 0.3761 | 0.5129 | 0.5020 | 0.3962 | 0.3973
NDCG@5 | 0.6176 | 0.6467 | 0.6379 | 0.6555 | 0.6637 | 0.4302 | 0.5452 | 0.5142 | 0.4418 | 0.4301
Recall@1 | 0.3126 | 0.3213 | 0.3286 | 0.3345 | 0.3395 | 0.1321 | 0.2132 | 0.2334 | 0.1354 | 0.1395
Recall@3 | 0.5743 | 0.6098 | 0.5918 | 0.6207 | 0.6280 | 0.2852 | 0.4166 | 0.3847 | 0.3181 | 0.3113
Recall@5 | 0.7428 | 0.7779 | 0.7613 | 0.7820 | 0.7915 | 0.4638 | 0.5146 | 0.4847 | 0.4757 | 0.4816
Table 2.2: Ranking performance of each personalized ranking model. Numbers in
boldface are the best results.

For the base models, we basically follow the source code of the authors. NCF
has 2-layer MLP for MLP module and 1-layer MLP for the prediction module. LGCN
has a 2-layer simplified GCN for the training and the inference. We use 128
for the size of user and item embeddings for all base models, except NCF which
adopts 64 for the embedding size. The batch size is 512, the learning rate is
0.001, the weight decay rate is 0.001, and the negative sample rate is 1. Each
model is trained until the convergence and their ranking performance can be
found in Table 2.2. LGCN shows the best performance for Yahoo!R3 dataset and
NCF or CML shows the best performance for Coat dataset.

Calibration methods compared. We evaluate the proposed calibration methods
with various calibration methods. For the naive baseline, we adopt the minmax
normalizer and the sigmoid function which simply re-scale the scores into
[0,1] without calibration. For non-parametric methods, we adopt Histogram
binning [16], Isotonic regression [3], and BBQ [15]. For Histogram binning, we
set the number of bins as 50 for Yahoo!R3, 15 for Coat. For BBQ, we set the
number of binning methods as 4 and the number of bins of each binning method
is {10,20,50,100}. For those that do not meet the first desideratum (BBQ and
Beta calibration), we adopt the sigmoid function to re-scale the input into
[0,1]. For parametric methods, we adopt Platt scaling [9] and Beta calibration
[14]. Note that we do not compare recent work designed for multi-class
classification [6, 7], since they are either the generalized version of Beta
calibration or cannot be directly adopted for the personalized ranking models.

Evaluation metrics. We adopt well-known calibration metrics like ECE, MCE with
$M=15$, and NLL as done in recent work [6, 7]. We also plot the reliability
diagram that shows the discrepancy between the accuracy and the average
calibrated probability of each probability interval. Note that evaluation
metrics are computed on $R_{u,i}$ which is observed only from the test set.

Evaluation process. We first train the base personalized ranking model
$f_{\theta}(u,i)$ with $Y_{u,i}$ on the training set. Second, we compute
ranking score $s_{u,i}=f_{\theta}(u,i)$ for user-item pairs in the validation
set. Third, we optimize the calibration method $g_{\phi}(s)$ on the validation
set with the computed $s_{u,i}$ and the estimated $\hat{\omega}_{u,i}$, with
$f_{\theta}(u,i)$ fixed. Lastly, we evaluate the calibrated probability
$p=g_{\phi}(s_{u,i})$ with $R_{u,i}$ from the unbiased test set by using the
above evaluation metrics.

Computing infrastructures We adopt a Titan X GPU and an Intel(R) Core(TM)
i7-7820X 3.60GHz CPU. Optimization of all calibration methods is done in at
most a few minutes.

Table 2.3: Expected Calibration Error of each calibration method applied on five personalized ranking models. Numbers in boldface are the best results and Improv denotes the improvement of the best proposed method over the best competitor (Platt or Beta with $\mathcal{L}_{\text{UERM}}$). |  | Yahoo!R3 | Coat
---|---|---|---
Type | Methods | BPR | NCF | CML | UBPR | LGCN | BPR | NCF | CML | UBPR | LGCN
uncalibrated | MinMax | 0.4929 | 0.4190 | 0.3152 | 0.3004 | 0.2258 | 0.1790 | 0.4624 | 0.1834 | 0.1920 | 0.2350
Sigmoid | 0.3065 | 0.0729 | 0.0526 | 0.2516 | 0.3024 | 0.2196 | 0.1422 | 0.0647 | 0.1415 | 0.0508
non-parametric | Hist | 0.0161 | 0.0133 | 0.0641 | 0.0130 | 0.0194 | 0.0552 | 0.0230 | 0.0161 | 0.0514 | 0.0470
Isotonic | 0.0146 | 0.0130 | 0.0635 | 0.0127 | 0.0154 | 0.0474 | 0.0159 | 0.0160 | 0.0490 | 0.0453
BBQ | 0.0136 | 0.0137 | 0.0634 | 0.0140 | 0.0165 | 0.0552 | 0.0178 | 0.0198 | 0.0459 | 0.0494
parametric w/ $\mathcal{L}_{\textup{naive}}$ | Platt | 0.0126 | 0.0146 | 0.0515 | 0.0107 | 0.0099 | 0.0441 | 0.0245 | 0.0203 | 0.0423 | 0.0407
Beta | 0.0127 | 0.0144 | 0.0504 | 0.0105 | 0.0150 | 0.0451 | 0.0258 | 0.0270 | 0.0416 | 0.0407
Gaussian | 0.0129 | 0.0104 | 0.0486 | 0.0105 | 0.0073 | 0.0436 | 0.0264 | 0.0245 | 0.0410 | 0.0404
Gamma | 0.0108 | 0.0145 | 0.0512 | 0.0107 | 0.0098 | 0.0424 | 0.0239 | 0.0208 | 0.0405 | 0.0406
| Platt | 0.0106 | 0.0129 | 0.0303 | 0.0100 | 0.0070 | 0.0411 | 0.0120 | 0.0155 | 0.0354 | 0.0224
parametric | Beta | 0.0109 | 0.0132 | 0.0305 | 0.0094 | 0.0076 | 0.0414 | 0.0075 | 0.0183 | 0.0375 | 0.0266
w/ $\mathcal{L}_{\textup{UERM}}$ | Gaussian | 0.0106 | 0.0096 | 0.0285 | 0.0070 | 0.0061 | 0.0393 | 0.0062 | 0.0147 | 0.0323 | 0.0208
| Gamma | 0.0100 | 0.0117 | 0.0287 | 0.0085 | 0.0065 | 0.0390 | 0.0061 | 0.0148 | 0.0326 | 0.0215
| Improv | 5.85% | 25.35% | 5.94% | 25.85% | 12.86% | 5.21% | 18.67% | 5.41% | 8.81% | 7.14%

Table 2.4: Maximum Calibration Error with $M=15$ of each calibration method applied on five personalized ranking models. Numbers in boldface are the best results. |  | Yahoo!R3 | Coat
---|---|---|---
Type | Methods | BPR | NCF | CML | UBPR | LGCN | BPR | NCF | CML | UBPR | LGCN
uncalibrated | MinMax | 0.4929 | 0.4190 | 0.3152 | 0.3004 | 0.2258 | 0.1790 | 0.4624 | 0.1834 | 0.1920 | 0.2350
Sigmoid | 0.5726 | 0.5248 | 0.2571 | 0.7103 | 0.5082 | 0.3868 | 0.6223 | 0.6707 | 0.2699 | 0.4642
non-parametric | Hist | 0.2620 | 0.2369 | 0.6184 | 0.1349 | 0.1366 | 0.2492 | 0.4083 | 0.4924 | 0.2910 | 0.3940
Isotonic | 0.2136 | 0.2173 | 0.5252 | 0.1909 | 0.1270 | 0.2495 | 0.4246 | 0.4812 | 0.2547 | 0.3500
BBQ | 0.2116 | 0.2489 | 0.7076 | 0.1701 | 0.1157 | 0.2545 | 0.3749 | 0.3423 | 0.3358 | 0.2589
parametric w/ $\mathcal{L}_{\textup{naive}}$ | Platt | 0.2032 | 0.2876 | 0.3254 | 0.3117 | 0.3068 | 0.3664 | 0.1844 | 0.5796 | 0.4079 | 0.4633
Beta | 0.2563 | 0.2575 | 0.5049 | 0.2705 | 0.3033 | 0.4071 | 0.2175 | 0.5782 | 0.4352 | 0.3367
Gaussian | 0.2387 | 0.2366 | 0.3737 | 0.3282 | 0.2200 | 0.3177 | 0.1678 | 0.5878 | 0.4807 | 0.3854
Gamma | 0.2123 | 0.2260 | 0.3149 | 0.3126 | 0.2962 | 0.3616 | 0.1702 | 0.5968 | 0.4185 | 0.3480
| Platt | 0.1951 | 0.2504 | 0.2293 | 0.1257 | 0.1143 | 0.2625 | 0.1882 | 0.4423 | 0.2708 | 0.2225
parametric | Beta | 0.2004 | 0.2453 | 0.2734 | 0.1317 | 0.1905 | 0.3278 | 0.2085 | 0.4150 | 0.2574 | 0.2501
w/ $\mathcal{L}_{\textup{UERM}}$ | Gaussian | 0.1816 | 0.2259 | 0.2451 | 0.1231 | 0.1064 | 0.2380 | 0.1236 | 0.3390 | 0.2419 | 0.2117
| Gamma | 0.1592 | 0.2074 | 0.2268 | 0.1231 | 0.1118 | 0.2543 | 0.1236 | 0.4663 | 0.2478 | 0.2023

Table 2.5: Negative Log-Likelihood of each calibration method applied on five personalized ranking models. Numbers in boldface are the best results. |  | Yahoo!R3 | Coat
---|---|---|---
Type | Methods | BPR | NCF | CML | UBPR | LightGCN | BPR | NCF | CML | UBPR | LightGCN
uncalibrated | MinMax | 0.4929 | 0.4190 | 0.3152 | 0.3004 | 0.2258 | 0.1790 | 0.4624 | 0.1834 | 0.1920 | 0.2350
Sigmoid | 0.7030 | 0.4641 | 0.3072 | 0.7083 | 0.6890 | 0.6727 | 1.0978 | 0.4819 | 0.6780 | 0.6904
non-parametric | Hist | 0.2753 | 0.2717 | 0.3361 | 0.2728 | 0.2769 | 0.6264 | 0.5116 | 0.5412 | 0.6177 | 0.5077
Isotonic | 0.2726 | 0.2723 | 0.3240 | 0.2728 | 0.2683 | 0.5110 | 0.4953 | 0.4711 | 0.5224 | 0.5386
BBQ | 0.2725 | 0.2693 | 0.3517 | 0.2749 | 0.2691 | 0.4867 | 0.4790 | 0.4895 | 0.5092 | 0.4813
parametric w/ $\mathcal{L}_{\textup{naive}}$ | Platt | 0.2756 | 0.2700 | 0.3184 | 0.2735 | 0.2671 | 0.4748 | 0.4771 | 0.4747 | 0.4735 | 0.4758
Beta | 0.2755 | 0.2697 | 0.3250 | 0.2738 | 0.2673 | 0.4766 | 0.4796 | 0.4747 | 0.4741 | 0.4776
Gaussian | 0.2748 | 0.2676 | 0.3196 | 0.2725 | 0.2671 | 0.4766 | 0.4768 | 0.4755 | 0.4730 | 0.4761
Gamma | 0.2735 | 0.2699 | 0.3181 | 0.2735 | 0.2672 | 0.4754 | 0.4764 | 0.4739 | 0.4747 | 0.4749
| Platt | 0.2749 | 0.2684 | 0.3034 | 0.2723 | 0.2675 | 0.4746 | 0.4655 | 0.4654 | 0.4716 | 0.4741
parametric | Beta | 0.2747 | 0.2685 | 0.3022 | 0.2721 | 0.2672 | 0.4744 | 0.4684 | 0.4665 | 0.4713 | 0.4751
w/ $\mathcal{L}_{\textup{UERM}}$ | Gaussian | 0.2743 | 0.2666 | 0.3021 | 0.2715 | 0.2671 | 0.4735 | 0.4619 | 0.4653 | 0.4711 | 0.4727
| Gamma | 0.2723 | 0.2681 | 0.3035 | 0.2720 | 0.2674 | 0.4742 | 0.4649 | 0.4651 | 0.4711 | 0.4733

####  2.5.2 Comparing Calibration Performance

Table 2.3, 2.4, and 2.5 show ECE, MCE, and NLL of each calibration method
applied on the various personalized ranking models, respectively. ECE
indicates how well the calibrated probabilities and ground-truth likelihoods
match on the test set across all probability ranges. First, the minmax
normalizer and the sigmoid function produce poorly calibrated preference
probabilities. It is obvious because the ranking scores do not have any
probabilistic meaning and naively re-scaling them cannot reflect the score
distribution.

Second, the parametric methods better calibrate the preference probabilities
than the non-parametric methods in most cases. This is consistent with recent
work [5, 6] for image classification. The non-parametric calibration methods
lack rich expressiveness since they rely on the binning scheme, which maps the
ranking scores to the probabilities in a discrete manner. On the other hand,
the parametric calibration methods fit the continuous functions based on the
parametric distributions. Therefore, they have a more granular mapping from
the ranking scores to the preference probabilities.

Third, every parametric calibration method benefits from adopting
$\mathcal{L}_{\textup{UERM}}$ instead of $\mathcal{L}_{\textup{naive}}$ for
the parameter fitting. The naive log-loss treats all the unobserved pairs as
negative pairs and makes the calibration methods produce biased preference
probabilities. On the contrary, inverse propensity-scored log-loss handles
such problem and enables us to compute the ideal empirical risk indirectly. As
a result, ECE decreases by 7.40%-76.52% for all parametric methods compared to
when the naive log-loss is used for the optimization.

Lastly, Gaussian calibration and Gamma calibration with
$\mathcal{L}_{\textup{UERM}}$ show the best calibration performance across all
base models and datasets. Platt scaling can be seen as a special case of the
proposed methods with $a=0$, so it has less expressiveness in terms of the
capacity of parametric family. Beta distribution is only defined in [0,1], so
it cannot represent the unbounded ranking scores. To adopt Beta calibration,
we need to re-scale the ranking score, however, it is not verified for the
optimality [3]. As a result, our calibration methods improve ECE by
5.21%-25.85% over the best competitor. Also, since our proposed models have a
larger capacity of expressiveness, they show larger improvement on Yahoo!R3,
which has more samples to fit the parameters than Coat.

Figure 2.1: Reliability diagram of each calibration method. Gap denotes the
discrepancy between the accuracy and the average calibrated probability for
each bin. The grey dashed line is a diagonal function that indicates the ideal
reliability line where the blue accuracy bar should meet.

####  2.5.3 Reliability Diagram

Figure 2.1 shows the reliability diagram [5] for each calibration method
applied on LGCN for Yahoo!R3. We partition the calibrated probabilities
$g_{\phi}(s)$ into 10 equi-spaced bins and compute the accuracy and the
average calibrated probability for each bin (i.e., the first and the second
term in Eq.2.5, respectively). The accuracy is the same with the ground-truth
proportion of positive samples for the positive bins (i.e., probability over
0.5) and the ground-truth proportion of negative samples for the negative bins
(i.e., probability under 0.5). Note that the bar does not exist if the bin
does not have any prediction in it.

First, the non-parametric calibration methods do not produce the probability
over 0.6. It is because they can easily be overfitted to the unbalanced user-
item interaction datasets since they do not have any prior distribution. On
the other hand, the parametric calibration methods produce probabilities
across all ranges by avoiding such overfitting problem with the prior
parametric distributions.

Second, the parametric calibration methods with UERM produce well-calibrated
probabilities especially for the positive preference ($p>0.5$). The naive log-
loss makes the calibration methods biased towards the negative preference, by
treating all the unobserved pairs as the negative pairs. As a result, the
parametric methods with the naive log-loss (upper-right three diagrams of
Figure 1) show large gaps in the positive probability range ($p>0.5$). On the
contrary, UERM framework successfully alleviates this problem and produces
much smaller gaps for the positive preference (lower-right three diagrams of
Figure 2.1). Lastly, it is quite a natural result that parametric methods with
UERM do not produce the probability over 0.9, considering that the users
prefer only a few items among a large number of items.

####  2.5.4 Score Distribution & Fitted Function

Figure 2.2: Ranking score distributions of negative and positive pairs.

Figure 2.3: Fitted function of each calibration method.

Figure 2.2 shows the distribution of ranking scores trained by NCF and LGCN on
Yahoo!R3. We can see that the class-conditional score distributions have
different deviations ($\sigma_{0}>\sigma_{1}$) and skewed shapes (left tails
are longer than the right tails). This indicates that Platt scaling (or
temperature scaling) assuming the same variance for both classes cannot
effectively handle these score distributions. Figure 2.3 shows the fitted
calibration function of each parametric method adopted on LGCN and optimized
with UERM. Since most of the user-item pairs are negative in the interaction
datasets, all three functions are fitted to produce the low probability under
0.1 for a wide bottom range to reflect the dominant negative preferences.
Platt scaling is forced to have the symmetric shape due to its parametric
family, so it produces the high probability over 0.9 which is symmetrical to
that of under 0.1. On the other hand, Gaussian calibration and Gamma
calibration, which have a larger expressive power, learn asymmetric shapes
tailored to the score distributions having different deviations and skewness.
This result shows that they effectively handle the imbalance of user-item
interaction datasets and supports the experimental superiority of the proposed
methods.

####  2.5.5 Propensity Estimation

Figure 2.4: ECE with various propensity estimation techniques. In this work,
we utilize item popularities to estimate the propensity scores.

Figure 2.4 shows the ECE of the proposed methods adopted on BPR for Coat
dataset with various propensity estimation techniques. Random estimation
produces random propensity scores from [0,1]. Uniform denotes $\omega=1$,
which is equivalent to the naive log-loss. In this work, we utilize item
popularities to estimate the propensity scores. Naive Bayes exploits the test
set to compute the conditional probabilities. Lastly, Logistic Regression uses
user demographics and item categories to estimate the propensity scores. Note
that Naive Bayes and Logistic Regression utilize the additional information
that is not available in our setting, and their propensity scores are provided
by the author [10]. Surprisingly, merely utilizing popularity is enough for
the estimation and it shows a comparable performance with Naive Bayes and
Logistic Regression which use additional information.

####  2.5.6 User Degree vs Calibrated Confidence

Figure 2.5: User degree versus mean top-10 confidence.

Figure 2.5 shows the relation between user degree (number of interactions) and
mean calibrated confidence on top-10 items. The base model is BPR. $r_{x,y}$
denotes the Pearson correlation coefficient. Since both correlation
coefficients are positive and larger than 0.5, there is a positive correlation
between user degree and mean top-10 calibrated confidence. If a user has
sufficient interactions, the recommender model can capture stable and certain
preferences for that user. Therefore, generally speaking, the mean confidence
on top-10 items increases. On the other hand, users with fewer interactions
also may get confident recommendations (Upper left of Figure 2.5). In this
case, if a user has a distinct preference for small interactions, the
recommender model also may capture certain preferences for that user. For
example, if a user has only three interactions for The Avengers, Avengers: Age
of Ultron, and Avengers: Infinity War, the recommender system can provide
Avengers: Endgame with very high confidence.

####  2.5.7 Case Study

Figure 2.6: Case study. Top-5 items for each user with ranking score $s$ and
calibrated probability $g_{\phi}(s)$.

Figure 2.6 shows the case study on Yahoo!R3 with Gaussian calibration adopted
on LGCN. The personalized ranking model first learns the ranking scores and
produces a top-5 ranking list for each user. Then, Gaussian calibration
transforms the ranking scores to the well-calibrated preference probabilities.
For the first user $u_{4506}$, the method produces high preference
probabilities for all top-5 items. In this case, we can recommend them to him
with confidence. On the other hand, for the second user $u_{8637}$, all top-5
items have low preference probabilities, and the last user $u_{2940}$ has a
wide range of preference probabilities. For these users, merely recommending
all the top-ranked items without consideration of potential preference degrade
their satisfaction. It is also known that the unsatisfactory recommendations
even make the users leave the platform [30]. Therefore, instead of
recommending items with low confidence, the system should take other
strategies, such as requesting additional user feedback [31].

###  2.6 Conclusion

In this paper, we aim to obtain calibrated probabilities with personalized
ranking models. We investigate various parametric distributions and propose
two parametric calibration methods, namely Gaussian calibration and Gamma
calibration. We also design the unbiased empirical risk minimization framework
that helps the calibration methods to be optimized towards true preference
probability with the biased user-item interaction dataset. Our extensive
evaluation demonstrates that the proposed methods and framework significantly
improve calibration metrics and have a richer expressiveness than existing
methods. Lastly, our case study shows that the calibrated probability provides
an objective criterion for the reliability of recommendations, allowing the
system to take various strategies to increase user satisfaction.

##  Chapter 3 Bidirectional Distillation for Top-K Recommender System

Recommender systems (RS) have started to employ knowledge distillation, which
is a model compression technique training a compact model (student) with the
knowledge transferred from a cumbersome model (teacher). The state-of-the-art
methods rely on unidirectional distillation transferring the knowledge only
from the teacher to the student, with an underlying assumption that the
teacher is always superior to the student. However, we demonstrate that the
student performs better than the teacher on a significant proportion of the
test set, especially for RS. Based on this observation, we propose
Bidirectional Distillation (BD) framework whereby both the teacher and the
student collaboratively improve with each other. Specifically, each model is
trained with the distillation loss that makes to follow the other’s prediction
along with its original loss function. For effective bidirectional
distillation, we propose rank discrepancy-aware sampling scheme to distill
only the informative knowledge that can fully enhance each other. The proposed
scheme is designed to effectively cope with a large performance gap between
the teacher and the student. Trained in the bidirectional way, it turns out
that both the teacher and the student are significantly improved compared to
when being trained separately. Our extensive experiments on real-world
datasets show that our proposed framework consistently outperforms the state-
of-the-art competitors. We also provide analyses for an in-depth understanding
of BD and ablation studies to verify the effectiveness of each proposed
component. This work was published at The Web Conference (WWW 2021) with an
oral presentation [32].

###  3.1 Introduction

Nowadays, the size of recommender systems (RS) is continuously increasing, as
they have adopted deep and sophisticated model architectures to better
understand the complex relationships between users and items [33, 34]. A large
recommender with many learning parameters usually has better performance due
to its high capacity, but it also has high computational costs and long
inference time. This problem is exacerbated for web-scale applications having
numerous users and items, since the number of the learning parameters
increases proportionally to the number of users and items. Therefore, it is
challenging to adopt such a large recommender for real-time and web-scale
platforms.

To tackle this problem, a few recent work [33, 34, 35] has adopted Knowledge
Distillation (KD) to RS. KD is a model-agnostic strategy that trains a compact
model (student) with the guidance of a pre-trained cumbersome model (teacher).
The distillation is conducted in two stages; First, the large teacher
recommender is trained with user-item interactions in the training set with
binary labels (i.e., $0$ for unobserved interaction and $1$ for observed
interaction.). Second, the compact student recommender is trained with the
recommendation list predicted by the teacher along with the binary training
set. Specifically, in [33], the student is trained to give high scores on the
top-ranked items of the teacher’s recommendation list. Similarly, in [34], the
student is trained to imitate the teacher’s prediction scores with particular
emphasis on the high-ranked items in the teacher’s recommendation list. The
teacher’s predictions provide additional supervision, which is not explicitly
revealed from the binary training set, to the student. By distilling the
teacher’s knowledge, the student can achieve comparable performance to the
teacher with fewer learning parameters and lower inference latency.

Despite their effectiveness, they still have some limitations. First, they
rely on unidirectional knowledge transfer, which distills knowledge only from
the teacher to the student, with an underlying assumption that the teacher is
always superior to the student. However, based on the in-depth analyses
provided in Section 2, we argue that the knowledge of the student also could
be useful for the teacher. Indeed, we demonstrate that the teacher is not
always superior to the student and further the student performs better than
the teacher on a significant proportion of the test set. In specific, the
student recommender better predicts 36$\sim$42% of the test interactions than
the teacher recommender. These proportions are remarkably large compared to
4$\sim$9% from our experiment on the computer vision task. In this regard, we
claim that both the student and the teacher can take advantage of each other’s
complementary knowledge, and be improved further. Second, the existing methods
have focused on distilling knowledge of items ranked highly by the teacher.
However, we observe that most of the items ranked highly by the teacher are
already ranked highly by the student (See Section 2 for details). Therefore,
merely high-ranked items may not be informative to fully enhance the other
model, which leads to limiting the effectiveness of the distillation.

We propose a novel Bidirectional Distillation (BD) framework for RS. Unlike
the existing methods that transfer the knowledge only from the pre-trained
teacher recommender, both the teacher and the student transfer their knowledge
to each other within our proposed framework. Specifically, they are trained
simultaneously with the distillation loss that makes to follow the other’s
predictions along with the original loss function. Trained in the
bidirectional way, it turns out that both the teacher and the student are
significantly improved compared to when being trained separately. In addition,
the student recommender trained with BD achieves superior performance compared
to the student trained with conventional distillation from a pre-trained
teacher recommender.

For effective bidirectional distillation, the remaining challenge is to design
an informative distillation strategy that can fully enhance each other,
considering the different capacities of the teacher and the student. As
mentioned earlier, items merely ranked highly by the teacher cannot give much
information to the student. Also, it is obvious that the student’s knowledge
is not always helpful to improve the teacher, as the teacher has a much better
overall performance than the student. To tackle this challenge, we propose
rank discrepancy-aware sampling scheme differently tailored for the student
and the teacher. In the scheme, the probability of sampling an item is defined
based on its rank discrepancy between the two recommenders. Specifically, each
recommender focuses on learning the knowledge of the items ranked highly by
the other recommender but ranked lowly by itself. Taking into account the
performance gap between the student and the teacher, we enable the teacher to
focus on the items that the student has very high confidence in, whereas
making the student learn the teacher’s broad knowledge on more diverse items.

The proposed BD framework can be applicable in many application scenarios.
Specifically, it can be used to maximize the performance of the existing
recommender in the scenario where there is no constraint on the model size (in
terms of the teacher), or to train a small but powerful recommender as
targeted in the conventional KD methods (in terms of the student). The key
contributions of our work are as follows:

  * •

Through our exhaustive analyses on real-world datasets, we demonstrate that
the knowledge of the student recommender also could be useful for the teacher
recommender. Based on the results, we also point out the limitation of the
existing distillation methods for RS under the assumption that the teacher is
always superior to the student.

  * •

We propose a novel bidirectional KD framework for RS, named BD, enabling that
the teacher and the student can collaboratively improve with each other during
the training. BD also adopts the rank discrepancy-aware sampling scheme
differently designed for the teacher and the student considering their
capacity gap.

  * •

We validate the superiority of the proposed framework by extensive experiments
on real-world datasets. BD considerably outperforms the state-of-the-art KD
competitors. We also provide both qualitative and quantitative analyses to
verify the effectiveness of each proposed component111We provide the source
code of BD at https://github.com/WonbinKweon/BD_WWW2021.

###  3.2 Analysis: Teacher vs Student

In this section, we provide detailed analyses of comparison between the
teacher and the student on two real-world datasets: CiteULike [36] and
Foursquare [37]. Following [34], we hold out the last interaction of each user
as test interaction, and the rest of the user-item interactions are used for
training data (see Section 3.5.1 for details of the experiment setup). We use
NeuMF [27] as a base model, and adopt NeuMF-50 as the teacher and NeuMF-5 as
the student. The number indicates the dimension of the user and item
embedding; the number of learning parameters of the teacher is 10 times bigger
than that of the student, and accordingly, the teacher shows a much better
overall performance than the student (reported in Table ???). Note that the
teacher and the student are trained separately without any distillation
technique in the analyses.

Figure 3.1: Rank difference between teacher and student. Rank differences are
computed on the test interactions and the test interactions (x-axis) are
sorted in increasing order by the rank difference.

The teacher is not always superior to the student. Figure 3.1 shows bar graphs
of the rank difference between the teacher and the student on the test
interactions (the bars in the graphs are sorted in increasing order along with
the x-axis). The rank difference on a test interaction $(u,i)$ is defined as
follows:

$\text{Rank diff.}^{u}(i)=rank_{T}^{u}(i)-rank_{S}^{u}(i),$ (3.1)

where $rank_{T}^{u}(i)$ and $rank_{S}^{u}(i)$ denote the ranks assigned by the
teacher and the student on the item $i$ for the user $u$ respectively, and
$rank_{*}^{u}(i)=1$ is the highest ranking. If the rank difference is bigger
than zero, it means that the student assigns a higher rank than the teacher on
the test interaction; the student better predicts the test interaction than
the teacher. We observe that the student model assigns higher ranks than the
teacher model on the significant proportion of the entire test interactions:
36% for CiteULike and 42% for Foursquare. These proportions are remarkably
large compared to 4% for CIFAR-10 and 9% for CIFAR-100 from our experiments on
computer vision task222We compare the predictions of the teacher and the
student on the test set. We adopt ResNet-80 as the teacher and ResNet-8 as the
student [38] and train them separately for the image classification task on
CIFAR-10 and CIFAR-100 dataset [39]..

The results raise a few questions. Why the student can perform better than the
teacher on the test data and why this phenomenon is intensified for RS? We
provide some possible reasons as follows: Firstly, not all user-item
interactions require sophisticated calculations in high-dimensional space for
correct prediction. As shown in the computer vision [40], a simple image
without complex background or patterns can be better predicted at the lower
layer than the final layer in a deep neural network. Likewise, some user-item
relationships can be better captured based on simple operations without
expensive computations. Moreover, unlike the image classification task, RS has
very high ambiguity in nature; in many applications, the user’s feedback on an
item is given in the binary form: 1 for observed interaction, 0 for unobserved
interaction. However, the binary labels do not explicitly show the user’s
preference. For example, “0” does not necessarily mean the user’s negative
preference for the item. It can be that the user may not be aware of the item.
Due to the ambiguity in the ground-truth labels, the lower training loss does
not necessarily guarantee a better ranking performance. Specifically, in the
case of NeuMF, which is trained with the binary cross-entropy loss, the
teacher can better minimize the training loss and thus better discriminate the
observed/unobserved interactions in the training set. However, this may not
necessarily result in better predicting user’s preference on all the
unobserved items due to the ambiguity of supervision.

Figure 3.2: Rank comparison between teacher and student. Each dot represents a
sampled unobserved interaction.

The high-ranked items are not that informative as expected. Figure 3.2 shows
the rank comparison between the teacher and the student. We plot the high-
ranked items in the recommendation list from the teacher and that from the
student for all users333We sample the items by using the rank-aware sampling
scheme suggested in [34]. Note that in [34], the sampled items are used for
the distillation.. Each point corresponds to
$(rank^{u}_{S}(i),rank^{u}_{T}(i))$. We observe that most of the points are
located near the lower-left corner, which means that most of the items ranked
highly by the teacher are already ranked highly by the student. In this
regard, unlike the motivation of the previous work [33, 34], items merely
ranked highly by the teacher may be not informative enough to fully enhance
the student, and thus may limit the effectiveness of the distillation. We
argue that each model can be further improved by focusing on learning the
knowledge of items ranked highly by the other recommender but ranked lowly by
itself (e.g., points located along the x-axis and y-axis).

In summary, the teacher is not always superior to the student, so the teacher
can also learn from the student. Their complementarity should be importantly
considered for effective distillation in RS. Also, the distillation should be
conducted with consideration of the rank discrepancy between the two
recommenders. Based on the observations, we are motivated to design a KD
framework that the teacher and the student can collaboratively improve with
each other based on the rank discrepancy.

###  3.3 Problem Formulation

Let the set of users and items be denoted as
$\mathcal{U}=\\{u_{1},u_{2},...,u_{n}\\}$ and
$\mathcal{I}=\\{i_{1},i_{2},...,i_{m}\\}$, where $n$ is the number of users
and $m$ is the number of items. Let $\textbf{R}\in\\{0,1\\}^{n\times m}$ be
the user-item interaction matrix, where $r_{ui}=1$ if the user $u$ has an
interaction with the item $i$, otherwise $r_{ui}=0$. Also, for a user $u$,
$\mathcal{I}_{u}^{-}=\\{i\in\mathcal{I}|r_{ui}=0\\}$ denotes the set of
unobserved items and $\mathcal{I}_{u}^{+}=\\{i\in\mathcal{I}|r_{ui}=1\\}$
denotes the set of interacted items. A top-$K$ recommender system aims to find
a recommendation list of unobserved items for each user. To make the
recommendation list, the system predicts the score
$\hat{r}_{ui}=P(r_{ui}=1|u,i)$ for each item $i$ in $\mathcal{I}_{u}^{-}$ for
each user $u$, then ranks the unobserved items according to their scores.

###  3.4 Method

Figure 3.3: Illustration of Bidirectional Distillation (BD) for top-$K$
recommender systems.

We propose a novel Bidirectional Distillation (BD) framework for top-$K$
recommender systems. Within our proposed framework, both the teacher and the
student transfer their knowledge to each other. We first provide an overview
of BD (Section 3.4.1). Then, we formalize the distillation loss to transfer
the knowledge between the two recommenders (Section 3.4.2). We also propose
rank discrepancy-aware sampling scheme differently tailored for the teacher
and the student to fully enhance each other (Section 3.4.3). Lastly, we
provide the details of the end-to-end training process within the proposed
framework (Section 3.4.4).

####  3.4.1 Overview

Figure 3.3 shows an overview of BD. Unlike the existing KD methods for RS,
both the teacher and the student are trained simultaneously by using each
other’s knowledge (i.e., recommendation list) along with the binary training
set (i.e., user-item interactions). First, the teacher and the student produce
the recommendation list for each user. Second, BD decides what knowledge to be
transferred for each distillation direction based on the rank discrepancy-
aware sampling scheme. Lastly, the teacher and the student are trained with
the distillation loss along with the original collaborative filtering loss. To
summarize, each of them is trained as follows:

$\begin{split}&\mathcal{L}_{T}(\theta_{T})=\mathcal{L}_{CF}(\theta_{T})+\lambda_{S\rightarrow
T}\cdot\mathcal{L}_{BD}(\theta_{T};\theta_{S}),\\\
&\mathcal{L}_{S}(\theta_{S})=\mathcal{L}_{CF}(\theta_{S})+\lambda_{T\rightarrow
S}\cdot\mathcal{L}_{BD}(\theta_{S};\theta_{T}),\end{split}$ (3.2)

where $T$ and $S$ denote the teacher and the student respectively,
$\theta_{*}$ is the model parameters. $\mathcal{L}_{CF}$ is the collaborative
filtering loss depending on the base model which can be any existing
recommender, and $\mathcal{L}_{BD}$ is the bidirectional distillation loss.
Lastly, $\lambda_{S\rightarrow T}$ and $\lambda_{T\rightarrow S}$ are the
hyperparameters that control the effects of the distillation loss in each
direction.

Within BD, the teacher and the student are collaboratively improved with each
other based on their complementarity. Also, during the training, the knowledge
distilled between the teacher and the student gets gradually evolved along
with the recommenders; the improvement of the teacher leads to the
acceleration of the student’s learning, and the accelerated student again
improves the teacher’s learning. Trained in the bidirectional way, both the
teacher and the student are significantly improved compared to when being
trained separately. As a result, the student trained with BD outperforms the
student model trained with the conventional distillation that relies on a pre-
trained and fixed teacher.

####  3.4.2 Distillation Loss

We formalize the distillation loss that transfers the knowledge between the
recommenders. By following the original distillation loss that matches the
class distributions of two classifiers for a given image [41], we design the
distillation loss for the user $u$ as follows:

$\displaystyle\mathcal{L}_{BD}(\theta_{T};\theta_{S})$
$\displaystyle=\sum_{j\in\mathcal{RDS}_{S\rightarrow
T}(\mathcal{I}_{u}^{-})}\mathcal{L}_{BCE}(\hat{r}^{T}_{uj},\hat{r}^{S}_{uj})$
(3.3) $\displaystyle\mathcal{L}_{BD}(\theta_{S};\theta_{T})$
$\displaystyle=\sum_{j\in\mathcal{RDS}_{T\rightarrow
S}(\mathcal{I}_{u}^{-})}\mathcal{L}_{BCE}(\hat{r}^{S}_{uj},\hat{r}^{T}_{uj}),$

where $\mathcal{L}_{BCE}(p,q)=q\log p+(1-q)\log(1-p)$ is the binary cross-
entropy loss, $\hat{r}_{uj}=P(r_{uj}=1|u,j)$ is the prediction of a
recommender and $\mathcal{RDS}_{*}(\mathcal{I}_{u}^{-})$ is a set of the
unobserved items sampled by the rank discrepancy-aware sampling.
$\hat{r}_{uj}$ is computed by $\sigma(z_{uj}/T)$ where $\sigma(\cdot)$ is the
sigmoid function, $z_{uj}$ is the logit, and $T$ is the temperature that
controls the smoothness.

Our distillation loss is a binary version of the original KD loss function.
Similar to the original KD loss transferring the knowledge of class
probabilities, our loss transfers a user’s potential positive and negative
preferences on the unobserved items. Specifically, in the binary training set,
the unobserved interaction $r_{uj}$ is only labeled as “0”. However, as
mentioned earlier, it is ambiguous whether the user actually dislikes the item
or potentially likes the item. Through the distillation, each recommender can
get the other’s opinion of how likely (and unlikely) the user would be
interested in the item, and such information helps the recommenders to better
cope with the ambiguous nature of RS.

####  3.4.3 Rank Discrepancy-aware Sampling

We propose the rank discrepancy-aware sampling scheme that decides what
knowledge to be transferred for each distillation direction. As we observed in
Section 2, most of the items ranked highly by the teacher are already ranked
highly by the student and vice versa. Thus, the existing methods [34, 33] that
simply choose the high-ranked items cannot give enough information to the
other recommender. Moreover, for effective bidirectional distillation, the
performance gap between the teacher and the student should be carefully
considered in deciding what knowledge to be transferred, as it is obvious that
not all knowledge of the student is helpful to improve the teacher.

In this regard, we develop a sampling scheme based on the rank discrepancy of
the teacher and the student, and tailor it differently for each distillation
direction with consideration of their different capacities. The underlying
idea of the scheme is that each recommender can get informative knowledge by
focusing on the items ranked highly by the other recommender, but ranked lowly
by itself. The sampling strategy for each distillation direction is defined as
follows:

Distillation from the teacher to the student. As the teacher has a much better
overall performance than the student, the opinion of the teacher should be
considered more reliable than that of the student in most cases. Thus, for
this direction of the distillation, we make the student follow the teacher’s
predictions on many rank-discrepant items. Formally, for each user $u$, the
probability of an item $i$ to be sampled is computed as follows:

$p_{T\rightarrow S}(i)\propto
tanh(\text{max}((rank_{S}(i)-rank_{T}(i))\cdot\epsilon_{t},0)),$ (3.4)

where $rank_{T}(i)$ and $rank_{S}(i)$ denote the ranks assigned by the teacher
and the student on the item $i$ for the user $u$ respectively, and
$rank_{*}(i)=1$ is the highest ranking444we omit the superscript $u$ from
$rank_{*}^{u}(i)$ for the simplicity.. We use a hyper-parameter $\epsilon_{t}$
$(>0)$ to control the smoothness of the probability. With this probability
function, we sample the items ranked highly by the teacher but ranked lowly by
the student. Since $tanh(\cdot)$ is a saturated function, items with rank
discrepancy above a particular threshold would be sampled almost uniformly. As
a result, the student learns the teacher’s broad knowledge of most of the
rank-discrepant items.

Distillation from the student to the teacher. As shown in Section 3.2, the
teacher is not always superior to the student, especially for RS. That is, the
teacher can be also further improved by learning from the student. However, at
the same time, the large performance gap between the teacher and the student
also needs to be considered for effective distillation. For this direction of
the distillation, we make the teacher follow the student’s predictions on only
a few selectively chosen rank-discrepant items. The distinct probability
function is defined as follows:

$p_{S\rightarrow T}(i)\propto
exp((rank_{T}(i)-rank_{S}(i))\cdot\epsilon_{e}),$ (3.5)

where $\epsilon_{e}$ $(>0)$ is a hyper-parameter to control the smoothness of
the probability. We use the exponential function to put particular emphasis on
the items that have large rank discrepancies. Therefore, this probability
function enables the teacher to follow the student’s predictions only on the
rank-discrepant items that the student has very high confidence in.

####  3.4.4 Model Training

Input : Training data $\mathcal{D}$, the number of total epochs $e$, rank
updating period $p$

Output : Teacher model $(\theta_{T})$, Student model $(\theta_{S})$

1 Warm up $\theta_{T}$ and $\theta_{S}$ with only $\mathcal{L}_{CF}$

2 for  _$t=0,1,...,(e-1)$_ do

3 if  _$t\text{ }\%\text{ }p==0$_ then

4 Teacher and Student update their recommendation lists

5 for  _$(u,i)\in\mathcal{D}$_ do

/* Train Teacher */

6 Draw $\mathcal{RDS}_{S\rightarrow T}(\mathcal{I}_{u}^{-})$ with probability
$p_{S\rightarrow T}(\cdot)$

7 Compute $\mathcal{L}_{CF}(\theta_{T})$ and
$\mathcal{L}_{BD}(\theta_{T};\theta_{S})$

Update $\theta_{T}$ /* Train Student */

8 Draw $\mathcal{RDS}_{T\rightarrow S}(\mathcal{I}_{u}^{-})$ with probability
$p_{T\rightarrow S}(\cdot)$

9 Compute $\mathcal{L}_{CF}(\theta_{S})$ and
$\mathcal{L}_{BD}(\theta_{S};\theta_{T})$

10 Update $\theta_{S}$

11

Algorithm 1 Bidirectional Distillation Framework.

Algorithm 1 describes a pseudo code for the end-to-end training process within
BD framework. The training data $\mathcal{D}$ consists of observed
interactions $(u,i)$. First, the model parameters $\theta_{T}$ and
$\theta_{S}$ are warmed up only with the collaborative filtering loss (line
1), as the predictions during the first few epochs are very unstable. Second,
we make the recommenders produce the recommendation lists for the subsequent
sampling. Since it is time-consuming to produce the recommendation lists every
epoch, we conduct this step every $p$ epochs (line 3-4). Next, we decide what
knowledge to be transferred in each distillation direction via the rank
discrepancy-aware sampling. It is worth noting that the unobserved items
sampled by the rank discrepancy-aware sampling can be used also for the
collaborative filtering loss. Finally, we compute the losses with the sampled
items for the teacher and the student, respectively, and update the model
parameters.

###  3.5 Experiments

In this section, we validate our proposed framework on 9 experiment settings
(3 real-world datasets $\times$ 3 base models). We first introduce our
experimental setup (Section 3.5.1). Then, we provide a performance comparison
supporting the superiority of the BD (Section 3.5.2). We also provide two
analyses for the in-depth understanding of BD (Section 3.5.3, 3.5.4). Lastly,
We provide an ablation study to verify the effectiveness of rank discrepancy-
aware sampling (Section 3.5.5) and analyses for important hyperparameters of
BD (Section 3.5.6).

####  3.5.1 Experimental Setup

Datasets. We use three real-world datasets:
CiteULike555https://github.com/changun/CollMetric/tree/master/citeulike-t
[36], Foursquare666https://sites.google.com/site/yangdingqi/home/foursquare-
dataset (Tokyo Check-in) [37] and
Yelp777https://github.com/hexiangnan/sigir16-eals/blob/master/data/yelp.rating
[42]. We only keep users who have at least five ratings for CiteULike and
Foursquare, ten ratings for Yelp as done in [27, 2]. Data statistics after the
preprocessing are presented in Table 3.1. We also report the experimental
results on ML100K and AMusic, which are used for CD [34] for the direct
comparison.

Table 3.1: Data Statistics Dataset | #Users | #Items | #Ratings | Sparsity
---|---|---|---|---
CiteULike | 5,219 | 25,187 | 130,788 | 99.90%
Foursquare | 2,293 | 61,858 | 537,167 | 99.62%
Yelp | 25,677 | 25,815 | 730,623 | 99.89%

Evaluation Protocol and Metrics. We adopt the widely used leave-one-out
evaluation protocol. For each user, we hold out the last interacted item for
testing and the second last interacted item for validation as done in [34,
27]. If there is no timestamp in the dataset, we randomly take two observed
items for each user. Then, we evaluate how well each method can rank the test
item higher than all the unobserved items for each user (i.e.,
$\mathcal{I}_{u}^{-}$). Note that instead of randomly choosing a predefined
number of candidates (e.g., 99), we adopt the full-ranking evaluation that
uses all the unobserved items as candidates. Although it is time-consuming, it
enables a more thorough evaluation compared to using random candidates [43,
34].

As we focus on top-$K$ recommendation for implicit feedback, we employ two
widely used metrics for evaluating the ranking performance of recommenders:
Hit Ratio (H@$K$) [44] and Normalized Discounted Cumulative Gain (N@$K$) [45].
H@$K$ measures whether the test item is present in the top-$K$ list and N@$K$
assigns a higher score to the hits at higher rankings in the top-$K$ list. We
compute those two metrics for each user, then compute the average score.
Lastly, we report the average value of five independent runs for all methods.

Base Models. BD is a model-agnostic framework applicable for any top-$K$ RS.
We validate BD with three base models that have different model architectures
and learning strategies. Specifically, we choose two widely used deep learning
models and one latent factor model as follows:

  * •

NeuMF [27]: A deep recommender that adopts Matrix Factorization (MF) and
Multi-Layer Perceptron (MLP) to capture complex and non-linear user-item
relationships. NeuMF uses the point-wise loss function for the optimization.

  * •

CDAE [46]: A deep recommender that adopts Denoising Autoencoders (DAE) [47]
for the collaborative filtering. CDAE uses the point-wise loss function for
the optimization.

  * •

BPR [2]: A learing-to-rank recommender that adopts MF [48] to model the user-
item interaction. BPR uses the pair-wise loss function for the optimization
under the assumption that observed items are more preferred than unobserved
items.

Methods Compared. The proposed framework is compared with the state-of-the-art
KD methods for top-$K$ RS.

  * •

Ranking Distillation (RD) [33]: A pioneering KD method for top-$K$ RS. RD
makes the student give high scores on top-ranked items by the teacher.

  * •

Collaborative Distillation (CD) [34]: A state-of-the-art KD method for top-$K$
RS. CD makes the student imitate the teacher’s scores on the items ranked
highly by the teacher.

Implementation Details. For all the base models and baselines, we use PyTorch
[49] for the implementation. For each dataset, hyperparameters are tuned by
using grid searches on the validation set. We use Adam optimizer [50] with L2
regularization and the learning rate is chosen from $\\{$0.00001, 0.0001,
0.001, 0.002$\\}$, and we set the batch size as 128. For NeuMF, we use 2-layer
MLP for the network. For CDAE, we use 2-layer MLP for the encoder and the
decoder, and the dropout ratio is set to 0.5. The number of negative samples
is set to 1 for NeuMF and BPR, $5*|\mathcal{I}_{u}^{+}|$ for CDAE as suggested
in the original paper [46].

For the distillation, we adopt as many learning parameters as possible for the
teacher model until the ranking performance is no longer increased on each
dataset. Then, we build the student model by employing only one-tenth of the
learning parameters used by the teacher. The number of model parameters of
each base model is reported in Table 3. For KD competitors (i.e., RD, CD),
$\lambda_{KD}$ is chosen from $\\{$0.01, 0.1, 0.5, 1$\\}$, the number of items
sampled for the distillation is chosen from $\\{$10, 15, 20, 30, 40, 50$\\}$,
and the temperature $T$ for logits is chosen from $\\{$1, 1.5, 2$\\}$. For
other hyperparameters, we use the values recommended from the public
implementation and the original papers [33, 34]. For BD,
$\lambda_{T\rightarrow S}$ and $\lambda_{S\rightarrow T}$ are set to 0.5, the
number of items sampled by rank discrepancy-aware sampling is chosen from
$\\{$1, 5, 10$\\}$, $\epsilon_{t}$ is chosen from
$\\{10^{-2},10^{-3},10^{-4},10^{-5}\\}$, $\epsilon_{e}$ is chosen from
$\\{10^{-3},10^{-4},10^{-5}\\}$, the temperature $T$ for logits is chosen from
$\\{$2, 5, 10$\\}$ and the rank updating period $p$ (in Algorithm 1) is set to
10.

Table 3.2: Performance comparison. Improv.T denotes the improvement of the
teacher over the original teacher model (”Teacher” in this table), Improv.B
denotes the improvement of the student over the best competitive method and
Improv.S denotes the improvement of the student over the original student
model (”Student” in this table). $*$ and $**$ indicate $p\leq$ 0.05 and
$p\leq$ 0.01 for the paired t-test of BD vs. the best competitor.

Base Model | Method | CiteULike | Foursquare | Yelp
---|---|---|---|---
H@50 | H@100 | N@50 | N@100 | H@50 | H@100 | N@50 | N@100 | H@50 | H@100 | N@50 | N@100
| Teacher | 0.1566 | 0.2272 | 0.0385 | 0.0487 | 0.1612 | 0.2066 | 0.0615 | 0.0685 | 0.0782 | 0.1341 | 0.0200 | 0.0290
| BD-Teacher | 0.1722 | 0.2443 | 0.0431 | 0.0548 | 0.1704 | 0.2232 | 0.0653 | 0.0737 | 0.0836 | 0.1473 | 0.0217 | 0.0319
| BD-Student | 0.0924 | 0.1425 | 0.0242 | 0.0325 | 0.1657 | 0.2181 | 0.0637 | 0.0721 | 0.0779 | 0.1315 | 0.0203 | 0.0289
| CD | 0.0820 | 0.1318 | 0.0227 | 0.0307 | 0.1548 | 0.2063 | 0.0592 | 0.0677 | 0.0712 | 0.1225 | 0.0189 | 0.0263
NeuMF | RD | 0.0755 | 0.1242 | 0.0197 | 0.0268 | 0.1442 | 0.1819 | 0.0547 | 0.0611 | 0.0662 | 0.1134 | 0.0161 | 0.0237
| Student | 0.0703 | 0.1167 | 0.0173 | 0.0248 | 0.1264 | 0.1674 | 0.0512 | 0.0578 | 0.0615 | 0.1018 | 0.0153 | 0.0221
| Improv.T | 9.96% | 7.53% | 11.95% | 12.53% | 5.73% | 8.03% | 6.21% | 7.59% | 6.91% | 9.84% | 8.50% | 10.00%
| Improv.B | 12.68%** | 8.16%** | 6.84%** | 5.87%** | 7.05%** | 5.74%** | 7.53%** | 6.50%** | 9.41%** | 7.35%** | 7.41%** | 9.89%**
| Improv.S | 31.44% | 22.11% | 39.88% | 30.85% | 31.11% | 30.31% | 24.34% | 24.74% | 26.67% | 29.17% | 32.68% | 30.77%
| Teacher | 0.1710 | 0.2445 | 0.0492 | 0.0611 | 0.1653 | 0.2281 | 0.0650 | 0.0743 | 0.0894 | 0.1523 | 0.0229 | 0.0331
| BD-Teacher | 0.1983 | 0.2702 | 0.0588 | 0.0674 | 0.1740 | 0.2368 | 0.0680 | 0.0766 | 0.0993 | 0.1681 | 0.0255 | 0.0366
| BD-Student | 0.0943 | 0.1470 | 0.0269 | 0.0355 | 0.1721 | 0.2242 | 0.0614 | 0.0709 | 0.0745 | 0.1323 | 0.0191 | 0.0268
| CD | 0.0861 | 0.1332 | 0.0250 | 0.0324 | 0.1629 | 0.2104 | 0.0553 | 0.0651 | 0.0667 | 0.1183 | 0.0176 | 0.0236
CDAE | RD | 0.0848 | 0.1266 | 0.0241 | 0.0318 | 0.1670 | 0.2146 | 0.0572 | 0.0675 | 0.0657 | 0.1118 | 0.0157 | 0.0231
| Student | 0.0724 | 0.1090 | 0.0198 | 0.0257 | 0.1491 | 0.2041 | 0.0552 | 0.0641 | 0.0613 | 0.1061 | 0.0151 | 0.0218
| Improv.T | 15.96% | 10.51% | 13.41% | 10.31% | 5.26% | 3.81% | 4.62% | 3.10% | 11.07% | 10.37% | 11.35% | 10.57%
| Improv.B | 9.52%** | 10.36%** | 7.60%** | 9.57%** | 3.05%* | 4.47%* | 7.34%** | 5.04%** | 11.86%** | 11.83%** | 8.52%** | 13.56%**
| Improv.S | 30.25% | 34.86% | 35.86% | 38.13% | 15.43% | 9.85% | 11.23% | 10.61% | 21.53% | 24.69% | 26.49% | 22.94%
| Teacher | 0.1224 | 0.1892 | 0.0328 | 0.0415 | 0.1746 | 0.2307 | 0.0662 | 0.0764 | 0.0896 | 0.1383 | 0.0231 | 0.0291
| BD-Teacher | 0.1479 | 0.2144 | 0.0390 | 0.0498 | 0.1834 | 0.2449 | 0.0713 | 0.0812 | 0.0946 | 0.1488 | 0.0245 | 0.0302
| BD-Student | 0.0853 | 0.1378 | 0.0213 | 0.0298 | 0.1611 | 0.1991 | 0.0610 | 0.0687 | 0.0733 | 0.1283 | 0.0171 | 0.0243
| CD | 0.0803 | 0.1271 | 0.0201 | 0.0277 | 0.1515 | 0.1851 | 0.0567 | 0.0622 | 0.0660 | 0.1137 | 0.0144 | 0.0221
BPR | RD | 0.0782 | 0.1234 | 0.0185 | 0.0259 | 0.1404 | 0.1696 | 0.0519 | 0.0574 | 0.0661 | 0.1126 | 0.0155 | 0.0215
| Student | 0.0717 | 0.1217 | 0.0171 | 0.0251 | 0.1252 | 0.1618 | 0.0469 | 0.0528 | 0.0559 | 0.1039 | 0.0127 | 0.0199
| Improv.T | 20.79% | 13.32% | 18.90% | 19.88% | 5.04% | 6.13% | 7.63% | 6.22% | 5.58% | 7.59% | 6.06% | 3.78%
| Improv.B | 6.16%** | 8.38%** | 5.97%** | 7.40%** | 6.34%** | 7.56%** | 7.58%** | 10.45%** | 10.89%** | 13.94%** | 10.32%** | 13.02%**
| Improv.S | 18.90% | 13.19% | 24.56% | 18.53% | 28.67% | 23.05% | 30.06% | 30.11% | 31.13% | 23.48% | 34.65% | 22.11%

####  3.5.2 Performance Comparison

Table 3.2 shows the recommendation performance of each KD method on three
real-world datasets and three different base models. In Table 3.2, ”Teacher”
and ”Student” indicate the base models trained separately without any
distillation technique, ”BD-Teacher” and ”BD-Student” are the teacher and the
student trained simultaneously with BD. ”CD” and ”RD” denote the student
trained with CD and RD, respectively. We analyze the experimental results from
various perspectives.

We first observe that the teacher recommender is consistently improved by the
knowledge transferred from the student with BD by up to 20.79% (for H@50 on
CiteULike). This result verifies our claim that the knowledge of the student
also could be useful for improving the teacher. Also, this result strongly
indicates that the distillation process of BD effectively resolves the
challenge of a large performance gap, and successfully transfer the
informative knowledge from the student. In specific, the ranking discrepancy-
aware sampling enables the teacher to focus on the items that the student has
very high confidence in. This strategy can minimize the adverse effects from
the performance gap, making the teacher be further improved based on the
complementary knowledge from the student.

Second, the student recommender is significantly improved by the knowledge
transferred from the teacher with BD by up to 39.88% (for N@50 on CiteULike).
Especially, the student trained with BD considerably outperforms the student
trained with the existing KD methods (i.e., RD, CD) by up to 13.94% (for H@100
on Yelp). The superiority of BD comes from the two contributions; BD makes the
student follow the teacher’s predictions on the rank-discrepant items which
are more informative than the merely high-ranked items. Also, within BD, the
improvement of the teacher leads to the acceleration of the student’s
learning, and the accelerated student again improves the teacher’s learning.
With better guidance from the improved teacher, the student with BD can
achieve superior performance than the student with the existing KD methods. To
verify the effectiveness of each contribution, we conduct in-depth analyses in
the next sections.

Table 3.3: Model size and online inference efficiency. Time denotes the wall time used for generating recommendation list for every user. Base Model | CiteULike | Foursquare | Yelp
---|---|---|---
#Params. | Time | #Params. | Time | #Params. | Time
NeuMF-T | 3.05M | 17.39s | 6.42M | 14.77s | 5.15M | 85.12s
NeuMF-S | 0.30M | 14.12s | 0.64M | 7.99s | 0.51M | 69.53s
CDAE-T | 2.80M | 14.73s | 6.36M | 7.95s | 3.89M | 64.45s
CDAE-S | 0.30M | 10.35s | 0.69M | 5.78s | 0.41M | 53.39s
BPR-T | 1.52M | 9.11s | 3.21M | 7.71s | 2.57M | 46.83s
BPR-S | 0.15M | 7.94s | 0.32M | 5.39s | 0.26M | 40.49s

Lastly, Table 3.3 shows the model size and online inference efficiency for
each base model. For making the inferences, we use PyTorch with CUDA on GTX
Titan X GPU and Intel i7-7820X CPU. As the student has only one-tenth of the
learning parameters used by the teacher, the student requires less
computational costs, thus achieves lower inference latency. Moreover, the
student trained with BD shows comparable or even better recommendation
performance to the teacher (e.g., NeuMF and CDAE on Foursquare). On CiteULike,
we observe that the student achieves comparable performance by employing
20$\sim$30% of learning parameters used by the teacher. These results show
that BD can be effectively adopted to train a small but powerful recommender.
Moreover, as mentioned earlier, BD is also applicable in setting where there
is no constraint on the model size or inference time. Specifically, BD can be
adopted to maximize the performance of a large recommender with numerous
learning parameters (i.e., the teacher).

####  3.5.3 Model Size Analysis

We control the size of the teacher and the student to analyze the effects of
the capacity gap between two recommenders on BD. We report the results of a
deep model and a latent factor model (i.e., NeuMF and BPR) on CiteULike.
Figure 3.4 shows the performance of the teacher trained with the student of
varying sizes and Figure 3.5 shows the performance of the student trained with
the teacher of varying sizes. “S” indicates the size of the student (10% of
the teacher), “M” indicates the medium size (50% of the teacher), and “T”
indicates the size of the teacher. Also, “fixed S” refers to the pre-trained
student recommender with the size of ”S”, “fixed T” refers to the pre-trained
teacher recommender with the size of ”T”. Note that ”fixed S/T” are not
updated during the training (i.e., unidirectional knowledge distillation).

Figure 3.4: Performance of teacher with student of varying size. The blue
dashed line indicates the performance of the model trained separately.

Figure 3.5: Performance of student with teacher of varying size. The blue
dashed line indicates the performance of the model trained separately.

First, we observe that both the teacher and the student achieves the greatest
performance gain when the capacity gap between two recommenders is largest;
the teacher shows the best performance with the smallest student (i.e., S in
Fig. 3.4) and the student performs best with the largest teacher (i.e., T in
Fig. 3.5). As mentioned in Section 3.2, the teacher and the student have
complementarity as some user-item relationships can be better captured without
expensive computations. In this regard, such complementarity can be maximized
when the capacity gap between the two recommenders is large enough. This
result supports our claim that the performance gain comes from the different
but reciprocal knowledge of two recommenders. Also, it is worth noting that
there are still performance improvements when two recommenders have identical
sizes (i.e., T in Fig. 3.4, S in Fig. 3.5). This can be understood as a kind
of self-distillation effect [51, 52] when the teacher has the same size as the
student. Although they have very similar kinds of knowledge, they can still
regularize each other, preventing its counterpart from being overfitted to a
few observed interactions.

Second, the teacher is more improved when it is trained along with the
learning student than when it is trained with the fixed student (i.e., S vs.
fixed S in Fig. 3.4). Similarly, the student is more improved when it is
trained together with the learning teacher than when it is trained with the
fixed teacher (i.e., T vs. fixed T in Fig. 3.5). In the unidirectional
distillation, the pre-trained recommender (i.e., fixed S/T) is no longer
improved, thus always conveys the same knowledge during the training. On the
contrary, within BD, both recommenders are trained together, thus the
knowledge distilled between the recommenders gets gradually evolved as they
are improved during the training. As a result, each recommender can be further
improved based on the evolved knowledge of its counterparts. This result again
shows the superiority of our bidirectional distillation over the
unidirectional distillation of the existing methods.

####  3.5.4 Synchronization Analysis

We perform in-depth analysis to investigate how well the teacher and the
student learn each other’s complementary knowledge within the proposed
framework. Specifically, we evaluate how much synchronized the two
recommenders are within BD, which shows that they are improved based on each
other’s complementarity. To quantify the degree of the synchronization, we
define (normalized) Average Rank Difference as follows:

$\text{Average Rank Diff.}=\frac{1}{n\cdot
m}\sum_{\mathcal{D}_{test}}\absolutevalue{\text{Rank Diff.}^{u}(i)},$ (3.6)

where $n$ and $m$ are the numbers of users and items, respectively, and
$\mathcal{D}_{test}$ is the test set that contains the held-out observed
interaction for each user. The rank difference ($\text{Rank Diff.}^{u}(i)$) is
defined in Equation 1.

Table 3.4: Average Rank Difference before and after the training with BD. Base Model | CiteULike | Foursquare | Yelp
---|---|---|---
NeuMF | before BD | 0.1944 | 0.1020 | 0.0918
after BD | 0.1323 | 0.0957 | 0.0824
CDAE | before BD | 0.1190 | 0.0963 | 0.0544
after BD | 0.0871 | 0.0883 | 0.0472
BPR | before BD | 0.1380 | 0.1180 | 0.0560
after BD | 0.1064 | 0.1082 | 0.0511

Figure 3.6: Rank difference after the training with BD.

Table 3.4 shows the change in the average rank difference after the training
with BD. We observe that the average rank difference gets consistently
decreased on all the datasets. This result indicates that the teacher and the
student get synchronized by transferring their knowledge to each other. Figure
4 shows the rank difference between the teacher and the student after the
training with BD. We adopt NeuMF as the base model as done in Section 3.2.
Note that the (normalized) average rank difference is proportional to the
extent of the blue area. We observe that the blue area in Figure 3.6 shrinks
compared to that in Figure 3.1 after the training with BD. Interestingly, we
observe that the teacher is significantly improved on CiteULike dataset (by up
to 20.79%) which has the largest average rank difference change before and
after the training with BD. The large change indicates that the two
recommenders get synchronized well during the training, which leads to
significant improvements based on each other’s knowledge.

####  3.5.5 Sampling Scheme Analysis

We examine the effects of diverse sampling schemes on the performance of BD to
verify the superiority of the proposed sampling scheme. Note that the schemes
decide what knowledge to be distilled within BD. We compare five different
sampling schemes as follows: 1) Rank discrepancy-aware sampling, 2) Rank-aware
sampling [34], 3) Top-$N$ selection [33], 4) Uniform sampling, 5) Swapped rank
discrepancy-aware sampling.

The rank discrepancy-aware sampling, which is our proposed scheme, focuses on
the rank-discrepant items between the teacher and the student. On the other
hand, the rank-aware sampling (adopted in CD) and top-$N$ selection (adopted
in RD) focus on the items ranked highly by one recommender. The uniform
sampling randomly chooses items from the entire recommendation list. Finally,
the swapped rank discrepancy-aware sampling, which is the ablation of the
proposed scheme, swaps the sampling probability function of each distillation
direction; we make the teacher follow the student on most of the rank-
discrepant items (with tanh), and make the student imitate only a few
predictions of the teacher (with exp).

Table 3.5: Recommendation performance (H@50) of BD with different sampling schemes on CiteULike. Numbers in bold face are the best results. Base Model | Sampling Scheme | Teacher | Student
---|---|---|---
NeuMF | Rank discrepancy-aware | 0.1722 | 0.0924
Rank-aware [34] | 0.1617 | 0.0812
Top-$N$ selection [33] | 0.1480 | 0.0766
Uniform | 0.1590 | 0.0726
Swapped rank discrepancy-aware | 0.1512 | 0.0747
CDAE | Rank discrepancy-aware | 0.1983 | 0.0943
Rank-aware [34] | 0.1818 | 0.0891
Top-$N$ selection [33] | 0.1757 | 0.0870
Uniform | 0.1788 | 0.0819
Swapped rank discrepancy-aware | 0.1733 | 0.0851
BPR | Rank discrepancy-aware | 0.1479 | 0.0853
Rank-aware [34] | 0.1367 | 0.0805
Top-$N$ selection [33] | 0.1264 | 0.0772
Uniform | 0.1306 | 0.0749
Swapped rank discrepancy-aware | 0.1281 | 0.0803

Table 3.5 shows the performance of BD with different sampling schemes. We
first observe that the proposed scheme achieves the best result both for the
teacher and the student. This result verifies the effectiveness of two
components of the proposed scheme: 1) sampling based on the rank discrepancy,
2) probability functions differently designed for each distillation direction.
Specifically, we can see the effectiveness of rank discrepant-based sampling
by comparing our sampling scheme with rank-aware sampling. As observed in
Section 3.2, the items merely ranked highly by the teacher may be not
informative enough to fully enhance the student. With the proposed scheme, BD
focuses on the rank-discrepant items, thus can further improve the
recommenders. Also, we observe that swapping the probability function of each
distillation direction (i.e., the swapped rank discrepancy-aware)
significantly degrades the performance. This result strongly indicates that
each probability function is well designed to effectively cope with the large
performance gap of two recommenders.

####  3.5.6 Hyperparameter Analysis

In this section, we provide thorough analyses that examine the effects of
important hyperparameters on BD. For the sake of the space, we report the
results of NeuMF on CiteULike dataset. We observe similar tendencies with
other base models and datasets.

Figure 3.7: Recommendation performance (H@50) of teacher and student with
varying lambda.

Figure 3.8: Recommendation performance (H@50) of teacher and student with
varying smoothing factor.

Figure 3.9: Recommendation performance (H@50) of teacher and student with
varying updating period.

First, Figure 3.7 shows the performance of the teacher and the student with
varying $\lambda_{T\rightarrow S}$ and $\lambda_{S\rightarrow T}$ which
control the effects of the distillation losses. The best performance is
achieved when both $\lambda_{T\rightarrow S}$ and $\lambda_{S\rightarrow T}$
are around 0.5. Also, we observe that both the recommenders are considerably
improved when $\lambda_{T\rightarrow S}$ and $\lambda_{S\rightarrow T}$ have
similar values (i.e., diagonal entries). Moreover, we observe that the
performance of the student is more robust with respect to $\lambda$ than that
of the teacher. We believe that this is because the student has a lower
overall performance than the teacher, thus can easily take advantage of the
teacher in broad settings.

Second, Figure 3.8 shows the performance of the teacher and the student with
varying $\epsilon_{t}$ and $\epsilon_{e}$ which control the smoothness of the
probability functions in rank discrepancy-aware sampling. When $\epsilon_{t}$
and $\epsilon_{e}$ are small, the probability functions become smooth. The
best performance is achieved when both parameters are around $10^{-4}$. When
$\epsilon_{e}$ is bigger than $10^{-3}$, the probability $p_{S\rightarrow
T}(\cdot)$ gets too sharp. Thus, the teacher cannot fully learn the student’s
complementary knowledge, which leads to degraded performance.

Lastly, Figure 3.9 shows the performance of the teacher and the student with
varying rank updating period $p$ (in Algorithm 1). Since it is time-consuming
to generate the recommendation list every epoch, we update the recommendation
list of the two models every $p$ epoch. In this paper, we use $p=10$ which
shows the comparable performance to the upper-bound ($p=1$).

####  3.5.7 Additional Result

In this section, we report the experimental results on ML100K and AMusic,
which are used for CD [34], for the direct comparison with CD. We do not
include this result in our main table, because we consider those datasets are
relatively small to simulate the real-world evaluation (ML100K has only 943
users and 1682 items). We adopt CDAE as the base model and we employ 2-layer
MLP for the encoder and the decoder of CDAE. Experimental results on their
experimental settings are as follows:

Table 3.6: Performance comparison with CD. Model | ML100K | AMusic
---|---|---
H@50 | N@50 | H@50 | N@50
Teacher | 0.3966 | 0.1259 | 0.1748 | 0.0533
BD-Teacher | 0.4317 | 0.1442 | 0.1936 | 0.0607
BD-Student | 0.4111 | 0.1321 | 0.1723 | 0.0545
CD | 0.3786 | 0.1232 | 0.1650 | 0.0506
Student | 0.3503 | 0.1078 | 0.1265 | 0.0416
Improv.T | 8.85% | 14.54% | 10.76% | 13.88%
Improv.B | 8.59% | 7.22% | 4.42% | 7.71%

All results are the average of five iterations and statistically significant
with p=0.01. The proposed approach (BD) still outperforms the best competitor
(CD) both on ML100K and AMusic. Moreover, the ranking performance of the
teacher also increases with BD.

###  3.6 Related Work

Reducing inference latency of RS. Several methods have been proposed for
reducing the model size and inference time of recommender systems. First, a
few work adopt discretization techniques to reduce the size of recommenders
[53, 54, 55, 56, 57]. They learn discrete representations of users and items
to make portable recommenders and successfully reduce the model size. However,
their recommendation performance is highly limited due to the restricted
capability and thus the loss of recommendation performance is unavoidable
[34]. Second, several methods try to accelerate the inference phase by
adopting model-dependent techniques [58, 59, 60]. For example, the order-
preserving transformations [58] and the pruning techniques [59, 60] have been
adopted to reduce the computational costs of the inner product. Although they
can reduce the inference latency, they are applicable only to specific models
(e.g., inner product-based models).

To tackle this challenge, a few recent methods [33, 34, 35] have adopted
knowledge distillation to RS. KD is a model-agnostic strategy and we can
employ any recommender system as the base model. RD [33] firstly proposes a KD
method that makes the student give high scores on the top-ranked items of the
teacher’s recommendation list. Similarly, CD [34] makes the student imitate
the teacher’s prediction scores with particular emphasis on the items ranked
highly by the teacher. The most recent work, RRD [35], formulates the
distillation process as a relaxed ranking matching problem between the ranking
list of the teacher and that of the student. Since it is daunting for the
small student to learn all the prediction results from the large teacher, they
focus on the high-ranked items, which can affect the top-$K$ recommendation
performance [35]. By using such supplementary supervisions from the teacher,
they have successfully improved the performance of the student. However, they
have some critical limitations in that 1) they rely on the unidirectional
distillation, 2) the high-ranked items are not informative enough. These
limitations are thoroughly analyzed and resolved in this work.

Training multiple models together. There have been successful attempts to
train multiple models simultaneously for better generalization performance in
computer vision and natural language processing [61, 62, 63]. In computer
vision, Deep Mutual Learning (DML) [61] trains a cohort of multiple
classifiers simultaneously in a peer-teaching scenario where each classifier
is trained to follow the predictions of the other classifiers along with its
original loss. Since each classifier starts from a different initial
condition, each classifier’s predicted probabilities of the next most likely
class vary. DML claims that those secondary quantities provide extra
information to the other classifiers that can help them to converge to a more
robust minima. Also, Collaborative Learning [62] trains several classifier
heads of the same network simultaneously by sharing intermediate-level
representations. Collaborative Learning argues that the consensus of multiple
views from different heads on the same data provides both supplementary
information and regularization to each classifier head. In natural language
processing, Dual Learning [63] proposes a learning mechanism that two machine
translators teach each other to reduce the costs of human labeling. Pointing
out that the machine translation can be considered as a dual-task forming a
closed loop (e.g., English-to-French/ French-to-English), Dual Learning
generates informative feedback by transferring their outputs to each other and
achieves considerable performance improvements without the involvement of a
human labeler. Although the aforementioned methods have effectively increased
performance by training multiple models simultaneously, they mostly employ
models with the same sizes focusing only on achieving better generalization
based on the consensus of various views.

In this paper, we propose a novel Bidirectional Distillation framework whereby
two recommenders of different sizes are trained together by transferring their
knowledge to each other. Pointing out that the large recommender and the small
recommender have different but complementary knowledge, we design an effective
distillation mechanism that considers their huge capacity gap.

###  3.7 Conclusion & Future Work

We propose a novel Bidirectional Distillation framework for top-$K$
recommender systems whereby the teacher and the student are collaboratively
improved with each other during the training. Within our proposed framework,
both the teacher and the student transfer their knowledge to each other with
the distillation loss. Also, our framework considers the capacity gap between
the teacher and the student with differently tailored rank discrepancy-aware
sampling. Our extensive experiments on real-world datasets show that BD
significantly outperforms the state-of-the-art competitors in terms of the
student recommender. Furthermore, the teacher model also gets the benefit of
the student and performs better than when being trained separately. We also
provide analyses for the in-depth understanding of BD and verifying the
effectiveness of each proposed component.

Moreover, we can adopt our confidence calibration approach for knowledge
distillation. Since the prediction confidence of the student and the teacher
serves as an additional soft target label, the confidence calibration
technique can be readily utilized in bidirectional distillation. As the
calibrated confidence can provide more reliable uncertainty on the prediction
of the student and the teacher, we expect the confidence calibration may
enhance the performance of the bidirectional distillation.

##  Chapter 4 Top-Personalized-K Recommendation

The conventional top-K recommendation, which presents the top-K items with the
highest ranking scores, is a common practice for generating personalized
ranking lists. However, is this fixed-size top-$K$ recommendation the optimal
approach for every user’s satisfaction? Not necessarily. We point out that
providing fixed-size recommendations without taking into account user utility
can be suboptimal, as it may unavoidably include irrelevant items or limit the
exposure to relevant ones. To address this issue, we introduce Top-
Personalized-$K$ Recommendation, a new recommendation task aimed at generating