a personalized-sized ranking list to maximize individual user satisfaction. As
a solution to the proposed task, we develop a model-agnostic framework named
PerK. PerK estimates the expected user utility by leveraging calibrated
interaction probabilities, subsequently selecting the recommendation size that
maximizes this expected utility. Through extensive experiments on real-world
datasets, we demonstrate the superiority of PerK in Top-Personalized-$K$
recommendation task. We expect that Top-Personalized-$K$ recommendation has
the potential to offer enhanced solutions for various real-world
recommendation scenarios, based on its great compatibility with existing
models.

###  4.1 Introduction

Figure 4.1: An example of top-personalized-$K$ recommendation. 1 and 0
represent the relevant and the irrelevant items, respectively. Note that these
labels are not available at the inference phase. Newly added items have red
outlines.

Personalized recommendations have a significant impact on various daily
activities such as shopping, advertising, watching videos, and listening to
music. To generate personalized ranking lists of items, recommender systems
utilize the top-$K$ recommendation approach [64], which presents the $K$ items
with the highest ranking scores, sorted in descending order. This approach has
become a common practice in recent recommender systems [65, 66, 67] due to its
optimality with the globally fixed recommendation size [68]. However, while
tremendous efforts have been made on recommender models, an important question
has been overlooked in the previous literature: is the fixed-size top-$K$
recommendation the optimal approach for ensuring every user’s satisfaction?

To elucidate the potential drawbacks of the top-$K$ recommendation approach,
we start with a motivating example with $K=5$ in Figure 4.1. For user 1, the
top-$K$ recommendation unavoidably includes two irrelevant items in the tail,
as the ranking list could not be filled with enough relevant items. Moreover,
for user 2, the recommendation could be further improved by including one
additional relevant item in the tail, although user 2 receives a more accurate
top-$K$ recommendation with four relevant items. As such, globally fixing the
recommendation sizes results in (1) exposing users to irrelevant items that
could lead to ad blindness [69] or user attrition [30], and (2) limiting
chances to provide relevant items, which can curtail user engagement and
revenue [70]. In this context, we argue the top-$K$ recommendation with a
globally fixed recommendation size is not the optimal approach for both user
satisfaction and system efficacy.

Instead of globally fixed recommendation sizes, a recommendation size that is
personalized for individual user satisfaction can create enhanced solutions
for various recommendation scenarios. Back in Figure 4.1, by adopting
personalized recommendation sizes, the system can increase both users’
satisfaction by reducing the effort spent inspecting irrelevant items (user 1)
and providing more relevant items (user 2). Furthermore, the personalized
recommendation size paves a way to further increase user satisfaction in
various applications, especially for systems with limited resources for making
recommendations: (1) Multi-domain recommender systems [71] that display items
from various domains on a single constrained screen can strike a balance in
recommendation sizes for the maximum overall user satisfaction, by employing
adapted-sized ranking lists from each domain (Figure 1 bottom). (2) In the
case of sponsored advertisements [69], advertisers can achieve higher user
engagement with the same promotion expenses by adjusting the number of
promoted items based on each user’s expected utility. (3) In the context of
the prefetching mechanism [72], which caches the initial few seconds of videos
expected to be clicked in order to reduce startup delay, the system can
minimize cache size and prevent cache pollution by adjusting the number of
items to be cached. In this sense, we claim that embracing personalized
recommendation sizes not only enhances user satisfaction but also unlocks
diverse optimization possibilities in real-world systems.

In this paper, we propose Top-Personalized-$K$ Recommendation, a new
recommendation task resolving the limitation of the top-$K$ recommendation.
Formally, the top-personalized-$K$ recommendation refers to providing a
ranking list of the variable size that maximizes individual user satisfaction,
which can be quantitatively measured by user utility [73, 74]. As a solution
to the proposed task, we develop PerK, a framework to determine the
personalized recommendation size with any existing recommender model. PerK
first formulates a bi-level optimization problem where the objective is to
determine the recommendation size that maximizes each user’s utility (Sec
4.4). To solve this optimization, we introduce the concept of expected user
utility (Sec 4.5.2), as it is not feasible to compute the true user utility
during the inference phase. We treat the interaction labels of unobserved
items as Bernoulli random variables and derive the expectation of the user
utility for various widely-used utility measures. Derived expected user
utilities can be computed with the interaction probability of user-item pairs.

The remaining challenge is obtaining accurate interaction probability with an
arbitrary recommender model. The recommender models do not necessarily output
the accurate interaction probability [5, 1]. They often output unbounded
ranking scores that cannot be treated as probabilities [2, 29] or
miscalibrated interaction probabilities that do not accurately reflect the
true likelihood of user-item interactions [27, 75]. To address this problem,
we propose the use of calibrated interaction probability obtained through the
user-wise calibration function (Sec 4.5.3). The calibration function maps the
ranking scores of the recommender model to well-calibrated interaction
probabilities [1]. We adopt Platt scaling [9] and instantiate it for each user
to consider the different distributions of the ranking score across users. We
train the calibration function to predict the interactions between pairs in a
held-out calibration set. As a result, the output of the function accurately
indicates the true likelihood of interaction, leading to accurate expected
user utility.

In summary, we aim to find the optimal recommendation size for each user by
(1) obtaining calibrated interaction probability with user-wise calibration,
(2) estimating the expected user utility, and (3) determining the
recommendation size that results in the maximum expected user utility. The
main contributions of our work can be described as follows:

  * •

We highlight the necessity of personalized recommendation size based on its
practical advantages in real-world scenarios, which has not been studied well
in the previous literature.

  * •

We propose Top-Personalized-$K$ Recommendation, a new recommendation task
where the recommendation size can be adjusted for each user to enhance
individual user satisfaction.

  * •

We develop PerK, a framework to determine the personalized recommendation size
by estimating the user’s expected utility with the calibrated interaction
probability.

  * •

We conduct comprehensive experiments with three base recommenders on four
real-world datasets, demonstrating the superiority of PerK in the top-
personalized-$K$ recommendation task.

###  4.2 Related Work

To the best of our knowledge, personalized recommendation size has not been
studied well in the previous literature. The nearest research line is the
document list truncation [76, 77] that aims to determine the optimal cutoff
position for retrieved documents. Document list truncation has been applied in
various domains, including legal document retrieval [78, 79] and searching
[80, 81]. The recent methods [82, 77, 80, 81] formulate the truncation problem
as a classification task that predicts the optimal position among the
candidate cut-off positions. The target label for this classification is given
as a $K$-dimensional vector and each element indicates the probability of
being the optimal position. Then, they deploy large and deep models (e.g., Bi-
LSTM [83], Transformer [84]) for the classification.

Limitations. While this task is related to ours, directly applying truncation
methods to the top-personalized-$K$ recommendation leads to poor performance
for several reasons. First, document retrieval datasets [85, 86] have
sufficient relevant documents, which provide rich target labels for
classification. In contrast, recommendation datasets suffer from severe
sparsity and only a small fraction of items is relevant for each user.
Additionally, recommendation datasets often include hidden-relevant items
among unobserved ones, providing inaccurate and noisy target labels. Second,
the document retrieval task leverages high-dimensional features extracted from
text contents (e.g., tf-idf [87] or doc2vec [88]) to train large transformer
models. However, in our scenarios, these additional features (e.g., review
texts) are not always available for unobserved items [89], and scalar ranking
scores are insufficient for training large transformer models. As a result,
state-of-the-art truncation models (e.g., AttnCut [80] and MtCut [81]) show
limited performance when applied to the top-personalized-$K$ task, and a
solution tailored to recommender systems is required.

###  4.3 Preliminaries

####  4.3.1 Recommendation with Implicit Feedback

Implicit Feedback. In this paper, we focus on the recommendation with implicit
feedback [90], a widely adopted scenario for top-$K$ recommendation [64]. Let
$\mathcal{U}$ and $\mathcal{I}$ denote a set of users and a set of items,
respectively. For a pair of $u\in\mathcal{U}$ and $i\in\mathcal{I}$, an
interaction label $Y_{u,i}$ is given as 1 if their interaction is observed and
0 otherwise. It is noted that when $Y_{u,i}=0$, it indicates either that the
item is irrelevant to the user or that it can be a hidden-relevant item of the
user [91]. A dataset $\mathcal{D}=\\{(u,i)|Y_{u,i}=1\\}$ consists of positive
pairs, and is split into a training set $\mathcal{D}_{\textup{tr}}$ and a
validation set $\mathcal{D}_{\textup{val}}$.
$\mathcal{I}^{-}_{u}=\\{i|(u,i)\notin\mathcal{D}_{\textup{tr}}\\}$ denotes the
unobserved itemset of $u$.

Recommender Model. A recommender model
$f_{\theta}:\mathcal{U}\times\mathcal{I}\rightarrow\mathbb{R}$ learns to
assign a ranking score to each user-item pair. In the literature, a variety of
model architectures for $f_{\theta}$ have been deployed, including matrix
factorization [48], neural networks [27, 75], and graph neural networks [29,
65]. To train recommender models, point-wise loss (e.g., binary cross-entropy,
mean squared error), pair-wise loss (e.g., BPR [2], Margin Ranking loss [13]),
and list-wise loss (e.g., InfoNCE [92], Sampled Softmax [93]) have been
adopted. After the recommender model is trained, we produce a ranking list
$\pi$ with unobserved items in $\mathcal{I}_{u}^{-}$ by sorting the ranking
scores $f_{\theta}(u,i)$ in descending order.

####  4.3.2 User Utility

In this paper, we adopt user utility for the quantitative measurement of user
satisfaction yielded by recommendation [73, 74]. User utility evaluates the
ranking list $\pi$ based on how much the recommended items are exposed and
relevant to the user:

$U(\pi|u)=\sum_{i\in\pi}\omega(\textup{rank}(i|\pi))\rho(Y_{u,i}).$ (4.1)

The function $\omega(\cdot)$ maps an item’s rank to the item’s exposure based
on the position-bias model [94], and $\rho(\cdot)$ casts the relevance of an
item to the user utility.111In the evaluation phase, we have the true
relevance (i.e., irrelevant or hidden-relevant) for unobserved items.
$\textup{rank}(i|\pi)$ is the rank of item $i$ in the ranking list $\pi$.
Depending on the formulation of $\omega(\cdot)$ and $\rho(\cdot)$, we can
represent various utility measures. For example, Discounted Cumulative Gain
(DCG) [45] can be represented with $\omega(r)=\frac{1}{\textup{log}_{2}(1+r)}$
and $\rho(Y)=\mathbbm{1}_{\\{Y=1\\}}(Y)$. Additionally, other utility
measures, such as Normalized Discounted Cumulative Gain [45], Penalized
Discounted Cumulative Gain [45, 80], F1 score [95], and Truncated Precision
[95, 75] have been adopted in the literature and will be investigated in this
work.

####  4.3.3 Top-K Recommendation

######  Definition 1 (Top-$K$ Recommendation).

The top-$K$ recommendation refers to providing a ranking list of $K$ items
with the highest ranking scores.

Typically, the recommendation size ($K$) is globally fixed and predefined by
systems, taking into account platform constraints such as screen size,
thumbnail dimensions, and promotion expense. The probability ranking principle
[68] guarantees that this approach maximizes user utility under a fixed value
of $K$ and for any decreasing function of $\omega(\cdot)$ in Eq.4.1. That is,
we get:

$\pi_{K}=\operatorname*{argsort}_{i\in\mathcal{I}_{u}^{-}}f_{\theta}(u,i)[:K]=\operatorname*{argmax}_{\pi\in\Pi_{K}}U(\pi|u),$
(4.2)

where $[:K]$ denotes to take the first $K$ elements of the list and $\Pi_{K}$
is a set of all possible ranking lists, each with a size of $K$. In the rest
of this paper, $\pi_{K}$ represents the sorted top-$K$ ranking list obtained
by Eq.4.2 (e.g., $\pi_{m}$ denotes the top-$m$ ranking list).

Limitations. Despite the prevalence and advancements in the top-$K$
recommendation, as discussed in Section 4.1, the top-$K$ recommendation has
limitations in that it provides a fixed-size recommendation without
consideration of user utility. In this case, users must inspect irrelevant
items to filter them out, which can be time-consuming, especially in domains
with lengthy inspection times (e.g., movies). This process can negatively
impact user satisfaction [96], resulting in users ignoring future
recommendations [69] or even leaving the system [30]. Moreover, the fixed size
scheme may further degrade the efficiency of real-world applications, such as
presenting an equal number of items from each domain without taking into
account user preferences or promoting an equal number of items to each user
without considering the users’ expected utility. Nevertheless, the methodology
for determining the appropriate number of items to present remains unexplored.

###  4.4 Proposed Task

We here firstly propose a new recommendation task, named Top-Personalized-$K$
Recommendation, as a means to overcome the limitation of the top-$K$
recommendation.

######  Definition 2 (Top-Personalized-$K$ Recommendation).

The top-personalized-$K$ recommendation refers to providing a ranking list
where the recommendation size is optimized in $[K]$ for each user, to maximize
individual user utility.222$[K]=\\{1,2,3,...,K\\}$

This approach ensures that each user receives a tailored-sized recommendation,
helps avoid presenting irrelevant items and providing more number of relevant
items. The problem of finding the optimal recommendation size
$k_{\textup{max}}$ can be formulated as the following bi-level optimization
problem:

$\displaystyle k_{\textup{max}}=\operatorname*{argmax}_{k\in[K]}U(\pi_{k}|u)$
(4.3)
$\displaystyle\textup{s.t.}\quad\pi_{k}=\operatorname*{argmax}_{\pi\in\Pi_{k}}U(\pi|u)$
$\displaystyle\quad\quad\quad\
=\operatorname*{argsort}_{i\in\mathcal{I}_{u}^{-}}f_{\theta}(u,i)[:k],\ \ \
\forall k\in[K].$

Fortunately, the inner optimization can be done readily since the top-$k$
ranking list $\pi_{k}$ is the optimal ranking list for each $k$. Then, in the
outer optimization, we would like to select the $k$ where $\pi_{k}$ yields the
highest user utility. However, it is noted that directly computing the user
utility is infeasible, since we do not have access to the true relevance of
unobserved items in the inference phase.

Applications. While our work primarily focuses on the technical aspects, the
proposed task has several implications for real-world applications. Various
scenarios can adopt personalized recommendation sizes by modifying the
constraint in Eq.4.3. For instance, in multi-domain recommender systems [71,
97], the total number of recommended items from various domains is constrained
due to the single limited screen. Instead of displaying the equivalent number
of items from each domain, we can adjust the recommendation size of each
domain to maximize the overall utility under the constraint. Similarly, in
sponsored advertisement [69, 98], the advertiser has a budget constraint on
the promotion expenses. In this case, instead of promoting the same number of
items to all users, the system can present personalized numbers of promotions
depending on each user’s utility while still satisfying the global budget
constraint.

PerK on Multi-Domain Scenario. In the multi-domain scenario, PerK generates
top-personalized-$K$ recommendations for each domain by slightly modifying
Eq.4.3:

$\displaystyle\\{k^{x}_{\textup{max}}\\}_{x\in\mathcal{S}}=\operatorname*{argmax}_{\\{k^{x}\\}_{x\in\mathcal{S}}}\sum_{x\in\mathcal{S}}U(\pi_{k^{x}}|u,x)$
(4.4)
$\displaystyle\textup{s.t.}\quad\pi_{k^{x}}=\operatorname*{argsort}_{i\in\mathcal{I}_{u,x}^{-}}f^{x}_{\theta}(u,i)[:k],\
\ \ \forall k^{x}\in[K],x\in\mathcal{S}$
$\displaystyle\quad\quad\sum_{x\in\mathcal{S}}k^{x}\leq N$

$k^{x}$ is the recommendation size for domain $x$ and $\mathcal{S}$ is set of
all domains. $U(\pi_{k^{x}}|u,x)$ is user utility of user $u$ and domain $x$.
$\mathcal{I}_{u,x}^{-}$ is the unobserved itemset of user $u$ and domain $x$,
$f^{x}_{\theta}$ is the recommender model for domain $x$ (it can be any
cross/multi-domain recommender model). $N$ is the total recommendation size
and $\mathbb{N}$ is the natural numbers set. To briefly explain, PerK finds
the recommendation size for each domain to maximize the average user utility
across all domains under the constraint that the total recommendation size
should not exceed $N$. This optimization problem is a variant of the Knapsack
problem [99] and can be solved by dynamic programming. We present a case study
of the multi-domain scenario on the four largest domains of Amazon datasets in
Section 4.6.6.

###  4.5 Proposed Framework

####  4.5.1 Overview

We propose PerK, a novel framework to find the optimal recommendation size for
the top-personalized-$K$ recommendation. PerK is a model-agnostic framework,
allowing it to be adapted for any item recommendation scenario with existing
recommenders. To solve the bi-level optimization problem in Eq.4.3, PerK
utilizes:

  * •

(Sec 4.5.2) Expected User Utility: Expected user utility can be estimated by
treating the interaction labels for unobserved items as Bernoulli random
variables. PerK derives the computational form of the expected user utility
for widely-used utility functions, which can be computed with the interaction
probabilities.

  * •

(Sec 4.5.3) Calibrated Interaction Probability: To obtain accurate interaction
probabilities, PerK utilizes user-wise calibration functions instantiated and
trained for each user. The calibration function maps the ranking scores of the
recommender model to the calibrated interaction probabilities.

To sum up, given a pre-trained recommender model, (1) PerK trains the user-
wise calibration functions and gets the calibrated interaction probabilities
for unobserved items (Sec 4.5.3), (2) PerK estimates the expected user utility
for each candidate size $k$ with the calibrated probability (Sec 4.5.2), (3)
PerK selects the size with the maximum expected user utility, and provides the
recommendation list having the selected size (Sec 4.5.4).

####  4.5.2 Expected User Utility

We cannot compute the true user utility in Eq.4.3, since we do not have access
to the true relevance of unobserved items in the inference phase. To overcome
this issue, PerK estimates the expected user utility instead of the true value
by treating the interaction label $Y$ for unobserved items as Bernoulli random
variables. We defined the expected user utility as follows:

$\mathbb{E}_{Y}[U(\pi|u)]=\mathbb{E}_{Y}\Big{[}\sum_{i\in\pi}\omega(\textup{rank}(i|\pi))\rho(Y_{u,i})\Big{]}.$
(4.5)

For simplicity, we transform the above formalization for the top-$k$ ranking
list $\pi_{k}$ as follows:

$\mathbb{E}_{Y}[U(\pi_{k}|u)]=\mathbb{E}_{Y}\Big{[}\sum_{r=1}^{k}\omega(r)\rho(Y_{u,r})\Big{]}.$
(4.6)

With slight abuse of terminology, let $Y_{u,r}$ denote the interaction
variable for user $u$ and the $r^{\textup{th}}$ item in $\pi_{k}$. In the rest
of this section, we derive the computational form of expected user utility for
four widely-adopted utility measures.

Normalized Discounted Cumulative Gain (NDCG). NDCG [45], one of the most
established utility measures, is formulated as:

$U_{\textup{NDCG}}(\pi_{k}|u)=\frac{U_{\textup{DCG}}(\pi_{k}|u)}{U_{\textup{IDCG}}(\pi_{k}|u)}=\frac{\sum_{r=1}^{k}\frac{\mathbbm{1}_{\\{Y=1\\}}(Y_{u,r})}{\textup{log}_{2}(1+r)}}{\sum_{r=1}^{\textup{min}(S_{Y}^{u},k)}\frac{1}{\textup{log}_{2}(1+r)}}.$
(4.7)

$S_{Y}^{u}=\sum_{i\in\mathcal{I}_{u}^{-}}Y_{u,i}$ is the sum of all
interaction variables for unobserved items of user $u$. The expected NDCG with
respect to the random variable $Y$ is:

$\displaystyle\mathbb{E}_{Y}[U_{\textup{NDCG}}(\pi_{k}|u)]$
$\displaystyle=\mathbb{E}_{Y}\Big{[}\frac{U_{\textup{DCG}}(\pi_{k}|u)}{U_{\textup{IDCG}}(\pi_{k}|u)}\Big{]}$
(4.8)
$\displaystyle=\sum_{Y_{u,1},...,Y_{u,|\mathcal{I}_{u}^{-}|}}P(Y_{u,1},...,Y_{u,|\mathcal{I}_{u}^{-}|})\frac{U_{\textup{DCG}}(\pi_{k}|u)|_{Y_{u,*}}}{U_{\textup{IDCG}}(\pi_{k}|u)|_{Y_{u,*}}}.$

$U_{\textup{DCG}}(\pi_{k}|u)$ and $U_{\textup{IDCG}}(\pi_{k}|u)$ are
conditionally independent given $Y_{u,*}$. Since investigating all possible
combinations of $Y_{u,r}$ is intractable, we re-formulate the above equation
with the summation over possible $S_{Y}^{u}$ by adopting the total expectation
theorem [100].

$\displaystyle\mathbb{E}_{Y}[U_{\textup{NDCG}}(\pi_{k}|u)]$
$\displaystyle=\sum_{m=1}^{|\mathcal{I}_{u}^{-}|}P(S_{Y}^{u}=m)\
\mathbb{E}_{Y|S_{Y}^{u}=m}\Big{[}\frac{U_{\textup{DCG}}(\pi_{k}|u)}{U_{\textup{IDCG}}(\pi_{k}|u)}\Big{]}$
(4.9)
$\displaystyle=\sum_{m=1}^{|\mathcal{I}_{u}^{-}|}P(S_{Y}^{u}=m)\frac{\mathbb{E}_{Y|S_{Y}^{u}=m}[U_{\textup{DCG}}(\pi_{k}|u)]}{U_{\textup{IDCG}}(\pi_{k}|u)|_{S_{Y}^{u}=m}}.$

$U_{\textup{DCG}}(\pi_{k}|u)$ and $U_{\textup{IDCG}}(\pi_{k}|u)$ are
conditionally independent given $S_{Y}^{u}$, and
$U_{\textup{IDCG}}(\pi_{k}|u)|_{S_{Y}^{u}=m}=\sum_{r=1}^{\textup{min}(m,k)}\frac{1}{\textup{log}(1+r)}$.
The expected DCG with respect to $Y$ conditioned on $S_{Y}^{u}=m$ can be
computed as follows:

$\displaystyle\mathbb{E}_{Y|{S_{Y}^{u}=m}}[U_{\textup{DCG}}(\pi_{k}|u)]$
$\displaystyle=\mathbb{E}_{Y|{S_{Y}^{u}=m}}\Big{[}\sum_{r=1}^{k}\frac{\mathbbm{1}_{\\{Y=1\\}}(Y_{u,r})}{\textup{log}_{2}(1+r)}\Big{]}$
(4.10)
$\displaystyle=\sum_{r=1}^{k}\frac{P(Y_{u,r}=1|S_{Y}^{u}=m)}{\textup{log}_{2}(1+r)}$
$\displaystyle=\sum_{r=1}^{k}\frac{P(Y_{u,r}=1)P(S_{Y_{-r}}^{u}=m-1)}{P(S_{Y}^{u}=m)\textup{log}_{2}(1+r)},$

where $S_{Y_{-r}}^{u}=S_{Y}^{u}-Y_{u,r}$. After aggregating Eq.4.9 and
Eq.4.10, we get

$\mathbb{E}_{Y}[U_{\textup{NDCG}}(\pi_{k}|u)]=\sum_{m=1}^{|\mathcal{I}_{u}^{-}|}\frac{\sum_{r=1}^{k}\frac{P(Y_{u,r}=1)P(S_{Y_{-r}}^{u}=m-1)}{\textup{log}_{2}(1+r)}}{\sum_{r=1}^{\textup{min}(m,k)}\frac{1}{\textup{log}_{2}(1+r)}}.$
(4.11)

For scalability, we adopt two simple approximations and get

$\mathbb{E}_{Y}[U_{\textup{NDCG}}(\pi_{k}|u)]\approx\sum_{m=1}^{M}\frac{\sum_{r=1}^{k}\frac{P(Y_{u,r}=1)P(S_{Y}^{u}=m-1)}{\textup{log}_{2}(1+r)}}{\sum_{r=1}^{\textup{min}(m,k)}\frac{1}{\textup{log}_{2}(1+r)}}.$
(4.12)

(1) We aggregate the summation only to $M\leq 2000$ rather than
$|\mathcal{I}_{u}^{-}|$ (here, $M$ is a hyperparameter), since the users are
likely to interact with only a few items among the unobserved items. (2) We
replace $P(S_{Y_{-r}}^{u}=m-1)$ with $P(S_{Y}^{u}=m-1)$ as we confirmed that
the effect of one interaction for $S_{Y}^{u}$ is negligible. These simple
techniques make the expected user utility can be estimated in real-time.
$P(S_{Y}^{u}=m-1)=P(\sum_{i\in\mathcal{I}_{u}^{-}}Y_{u,i}=m-1)$ follows the
Poisson-Binomial distribution [101], and can be computed only with the
interaction probabilities $P(Y_{u,i}=1)$ for $i\in\mathcal{I}_{u}^{-}$.

Penalized Discounted Cumulative Gain (PDCG). PDCG [45], a utility measure
based on DCG, has a penalizing term for the irrelevant items in the ranking
list.333It is called DCG in the document retrieval field [80, 81], however, we
call it PDCG to distinguish it from DCG in the item recommendation field in
Eq.4.7.

$U_{\textup{PDCG}}(\pi_{k}|u)=\sum_{r=1}^{k}\frac{\mathbbm{1}_{\\{Y=1\\}}(Y_{u,r})-\mathbbm{1}_{\\{Y=0\\}}(Y_{u,r})}{\textup{log}_{2}(1+r)}.$
(4.13)

The expected PDCG with respect to interaction variable $Y$ is computed as
follows:

$\displaystyle\mathbb{E}_{Y}[U_{\textup{PDCG}}(\pi_{k}|u)]$
$\displaystyle=\mathbb{E}_{Y}\Big{[}\sum_{r=1}^{k}\frac{\mathbbm{1}_{\\{Y=1\\}}(Y_{u,r})-\mathbbm{1}_{\\{Y=0\\}}(Y_{u,r})}{\textup{log}_{2}(1+r)}\Big{]}$
(4.14) $\displaystyle=\sum_{r=1}^{k}\frac{2\cdot
P(Y_{u,r}=1)-1}{\textup{log}_{2}(1+r)}.$

F1 Score (F1). F1 [95] is a utility measure computed as the harmonic mean of
Precision and Recall.

$\displaystyle U_{\textup{F1}}(\pi_{k}|u)$
$\displaystyle=\frac{2\cdot\frac{\sum_{r=1}^{k}\mathbbm{1}_{\\{Y=1\\}}(Y_{u,r})}{k}\cdot\frac{\sum_{r=1}^{k}\mathbbm{1}_{\\{Y=1\\}}(Y_{u,r})}{S_{Y}^{u}}}{\frac{\sum_{r=1}^{k}\mathbbm{1}_{\\{Y=1\\}}(Y_{u,r})}{k}+\frac{\sum_{r=1}^{k}\mathbbm{1}_{\\{Y=1\\}}(Y_{u,r})}{S_{Y}^{u}}}$
(4.15)
$\displaystyle=\frac{2\cdot\sum_{r=1}^{k}\mathbbm{1}_{\\{Y=1\\}}(Y_{u,r})}{S_{Y}^{u}+k}.$

The expected F1 with respect to the interaction variable $Y$ is computed as
follows:

$\displaystyle\mathbb{E}_{Y}[U_{\textup{F1}}(\pi_{k}|u)]$
$\displaystyle=\sum_{m=1}^{|\mathcal{I}_{u}^{-}|}P(S_{Y}^{u}=m)\
\mathbb{E}_{Y|S_{Y}^{u}=m}[U_{\textup{F1}}(\pi_{k}|u)]$ (4.16)
$\displaystyle=\sum_{m=1}^{|\mathcal{I}_{u}^{-}|}P(S^{u}_{Y}=m)\frac{2\cdot\sum_{r=1}^{k}P(Y_{u,r}=1|S^{u}_{Y}=m)}{m+k}$
$\displaystyle=\sum_{m=1}^{|\mathcal{I}_{u}^{-}|}\frac{2\cdot\sum_{r=1}^{k}P(Y_{u,r}=1)P(S_{Y_{-r}}^{u}=m-1)}{m+k}$
$\displaystyle\approx\sum_{m=1}^{M}\frac{2\cdot\sum_{r=1}^{k}P(Y_{u,r}=1)P(S^{u}_{Y}=m-1)}{m+k}.$

Here, we use the total expectation theorem and apply the same approximations
as done in Eq.4.12.

Truncated Precision (TP). TP [95, 75] is a utility measure that addresses the
limitations of Recall and Precision.444For example, Precision cannot have a
value of 1 if $k$ is larger than $S_{Y}^{u}$. TP is referred to as Recall in
[75]. However, in this work, we use the term Truncated Precision to
distinguish it from the standard definition of Recall.

$U_{\textup{TP}}(\pi_{k}|u)=\frac{\sum_{r=1}^{k}\mathbbm{1}_{\\{Y=1\\}}(Y_{u,r})}{\textup{min}(k,S_{Y}^{u})}.$
(4.17)

The expected TP with respect to the interaction variable $Y$ is computed as
follows:

$\displaystyle\mathbb{E}_{Y}[U_{\textup{TP}}(\pi_{k}|u)]$
$\displaystyle=\sum_{m=1}^{|\mathcal{I}_{u}^{-}|}P(S_{Y}^{u}=m)\
\mathbb{E}_{Y|S_{Y}^{u}=m}[U_{\textup{TP}}(\pi_{k}|u)]$ (4.18)
$\displaystyle=\sum_{m=1}^{|\mathcal{I}_{u}^{-}|}P(S^{u}_{Y}=m)\frac{\sum_{r=1}^{k}P(Y_{u,r}=1|S^{u}_{Y}=m)}{\textup{min}(k,m)}$
$\displaystyle=\sum_{m=1}^{|\mathcal{I}_{u}^{-}|}\frac{\sum_{r=1}^{k}P(Y_{u,r}=1)P(S_{Y_{-r}}^{u}=m-1)}{\textup{min}(k,m)}$
$\displaystyle\approx\sum_{m=1}^{M}\frac{\sum_{r=1}^{k}P(Y_{u,r}=1)P(S^{u}_{Y}=m-1)}{\textup{min}(k,m)}.$

Here, we use the total expectation theorem and apply the same approximations
as done in Eq.4.12.

Interaction Probability. Up to this point, we have formulated the
computational forms of expected user utility for four widely-adopted utility
measures: NDCG (Eq.4.12), PDCG (Eq.4.14), F1 (Eq.4.16), and TP (Eq.4.18). In
common, these measures all require the interaction probability $P(Y_{u,r}=1)$
for the estimation. In the following subsection, we present our solution to
obtain accurate interaction probabilities with an arbitrary recommender model.

####  4.5.3 Calibrated Interaction Probability

Recommender models do not necessarily output accurate interaction probability.
They often output the ranking score that can have any value of an unbounded
real number [2, 29, 65], making it difficult to treat it as a probability.
Furthermore, even when a model is trained to output probabilities [27, 75], it
has been demonstrated that these probabilities may not accurately reflect the
true likelihood (i.e., model miscalibration) [5, 1].

To address this, we introduce a post-processing calibration function
$g_{\phi}:\mathbb{R}\rightarrow[0,1]$ that maps the ranking score
$s_{u,i}=f_{\theta}(u,i)$ of a pre-trained recommender to the calibrated
interaction probability $g_{\phi}(s_{u,i})=\hat{P}(Y_{u,i}=1|s_{u,i})$. A
probability $p$ is regarded calibrated if it indicates the ground-truth
correctness likelihood [14]: $\mathbb{E}[Y|g_{\phi}(s)=p]=p$. For example, if
we have 100 user-item pairs with $p=0.2$, we expect 20 of them to have
interactions $(Y=1)$.

We adopt Platt scaling $g_{\phi}(s)=\sigma(as+b)$ [9], a generalized form of
temperature scaling [5]. This calibration function has been deployed
effectively for model calibration in computer vision [102, 103], natural
language processing [104], and recommender system [1]. The key difference is
that PerK instantiates the calibration function for each user, while previous
calibration work [1] deploys one global calibration function covering all
users. That is, we have:

$g_{\phi_{u}}(s)=\sigma(a_{u}s+b_{u}),\quad\forall u\in\mathcal{U}.$ (4.19)

The user-specific parameters $\phi_{u}=\\{a_{u},b_{u}\\}$ are related to the
distribution of the ranking score of each user [14, 1]. Therefore, the user-
wise calibration function can consider the different distributions of the
ranking score across users.

We train the calibration function to predict the interactions of pairs in a
calibration set constructed for each user. It is a common practice to adopt
the validation set as the calibration set [5, 1]. In this work, the
calibration set for user $u$ is constructed as follows:

$\mathcal{D}_{u}^{\textup{cal}}=\\{(u,i,Y_{u,i}=1)|i\in\mathcal{I}_{u}^{\textup{val}}\\}\cup\\{(u,i,Y_{u,i}=0)|i\in\mathcal{I}_{u}^{-}\setminus\mathcal{I}_{u}^{\textup{val}}\\},$
(4.20)

where $\mathcal{I}_{u}^{\textup{val}}\
=\\{i|(u,i)\in\mathcal{D}_{\textup{val}}\\}$. We also use the binary cross-
entropy loss, a widely-adopted loss function for the calibration of binary
classifiers [5, 1, 14], as our calibration loss.

$\mathcal{L}_{u}^{\textup{cal}}=\sum_{(u,i,Y_{u,i})\in\mathcal{D}^{\textup{cal}}_{u}}-Y_{u,i}\text{log}(g_{\phi_{u}}(s_{u,i}))-(1-Y_{u,i})\text{log}(1-g_{\phi_{u}}(s_{u,i})).$
(4.21)

During the fitting of the calibration function, the base recommender model
$f_{\theta}$ is fixed and only $g_{\phi_{u}}$ is updated. It is worth
mentioning that Platt scaling with binary cross-entropy is mathematically
equivalent to logistic regression and can be efficiently solved. Additionally,
as we only have two learnable parameters for each user, our calibration
functions have a negligible impact on space complexity.

####  4.5.4 Optimization Procedure

Input : Training set $\mathcal{D}_{\textup{tr}}$, validation set
$\mathcal{D}_{\textup{val}}$, pre-trained recommender model $f_{\theta}$,
maximum recommendation size $K$, expected user utility
$\mathbb{E}_{Y}[U(\pi_{k}|u)]$

Output : Top-personalized-$K$ recommendation $\pi^{\textup{perK}}$

1 Fit user-wise calibration $g_{\phi_{u}}$ on $f_{\theta}$ $\triangleright$
Section 4.5.3
$\pi_{K}=\operatorname*{argsort}_{i\in\mathcal{I}_{u}^{-}}f_{\theta}(u,i)[:K]$
$\triangleright$ Top-$K$ Recommendation

2 for  _$k\in[K]$_ do

3 $\pi_{k}=\pi_{K}[:k]$

4 Compute $\mathbb{E}_{Y}[U(\pi_{k}|u)]$ with $g_{\phi_{u}}$ $\triangleright$
Section 4.5.2

5$k_{\textup{max}}=\operatorname*{argmax}_{k\in[K]}\mathbb{E}_{Y}[U(\pi_{k}|u)]$.

6 $\pi^{\textup{perK}}=\pi_{k_{\textup{max}}}$

Algorithm 2 Top-Personalized-K Recommendation with PerK

Algorithm 2 presents the entire procedure of PerK solving the bi-level
optimization in Eq.4.3 for the top-personalized-$K$ recommendation. (line 1):
PerK first fits the user-wise calibration function $g_{\phi_{u}}$ with
$\mathcal{D}_{u}^{\textup{cal}}$ and $\mathcal{L}_{u}^{\textup{cal}}$ on top
of pre-trained and fixed recommender $f_{\theta}$. (line 2): PerK generates
top-$K$ recommendation by sorting the ranking score. (line 3-5): For each
candidate size $k$, PerK estimates the expected user utility
$\mathbb{E}_{Y}[U(\pi_{k}|u)]$ by using the calibrated interaction probability
$g_{\phi_{u}}(s_{u,i})=\hat{P}(Y_{u,i}=1|s_{u,i})$. (line 6-7): Lastly, PerK
select $k_{\textup{max}}$, the recommendation size with the highest expected
user utility, and provide the top-personalized-$K$ recommendation
$\pi^{\textup{perK}}=\pi_{k_{\textup{max}}}$ to user $u$.

###  4.6 Experiments

####  4.6.1 Experiment Setup

We provide a summary of the experiment setup due to limited space. We will
publicly provide the GitHub repository of this work in the final version.

Table 4.1: Data statistics after the preprocessing. Dataset | #Users | #Items | #Interactions | Sparsity
---|---|---|---|---
MovieLens 10M | 69,838 | 8,939 | 9,985,038 | 98.40%
CiteULike | 3,277 | 16,807 | 178,187 | 99.68%
MovieLens 25M | 162,414 | 18,424 | 24,811,113 | 99.17%
Amazon Books | 157,809 | 112,048 | 8,460,428 | 99.95%

Datasets. We use four real-world datasets including MovieLens
10M,555https://grouplens.org/datasets/movielens/ CiteULike [36], MovieLens
25M, and Amazon Books [105]. These datasets are publicly available and have
been widely used in the literature [27, 29, 106, 107]. We adopt the 20-core
setting for all datasets as done in MovieLens datasets. We randomly split each
user’s interactions into a training set (60%), a validation set (20%), and a
test set (20%).

Methods Compared. We compare PerK with various traditional and recent methods.
Specifically, we adopt

  * •

Oracle: It uses the ground-truth labels of the test set to determine the
optimal recommendation size for each user.

and three traditional methods:

  * •

Top-$k$: It denotes the top-$k$ recommendation with globally fixed
recommendation size. We adopt $k\in\\{1,5,10,20,50\\}$.

  * •

Rand: It randomly selects the recommendation size for each user. It represents
the lower bound of the performance.

  * •

Val-$k$: It selects the recommendation size that maximizes validation utility
for each user.

and two state-of-the-art methods for truncating document retrieval results:

  * •

AttnCut [80]: It deploys a classification model with Bi-LSTM and Transformer
encoder to predict the best cut-off position.

  * •

MtCut (MMoECut) [81]: It deploys MMoE [108] on top of AttnCut architecture and
adopt multi-task learning.

and the proposed framework:

  * •

PerK (ours): We determine the personalized recommendation size by estimating
the expected user utility with calibrated interaction probability.

For AttnCut and MtCut, we use the source code of the
authors.666https://github.com/Woody5962/Ranked-List-Truncation We use a
$K$-dimentional ranking score vector for the input of the models. We have
tried to use the item embeddings as additional features for the models, but it
did not increase the performance. For each dataset, hyperparameters are tuned
by using grid searches on the validation set. We use Adam optimizer with
learning rate in $\\{10^{-2},10^{-3},10^{-4}\\}$ and weight decay in
$\\{0,10^{-1},10^{-3},10^{-5},10^{-7},10^{-9}\\}$. We set the batch size to 64
and the embedding size to 64. The number of layers and the number of
transformer heads are chosen from $\\{1,2\\}$ and the dropout ratio is set to
0.2. Each model is trained until the convergence of validation performa For a
quantitative comparison of user utility, we set the maximum recommendation
size to 50 for all methods. We also tried 100 and observed a similar
performance improvement for PerK.

Base Recommender Models ($f_{\theta}$). We adopt three widely-used recommender
models with various model architectures and loss functions. Bayesian
Personalized Ranking (BPR) [2] captures the user-item relevance by the inner
product of the user and the item embeddings, and is trained with the loss
function that makes the model put the higher ranking score on the observed
pair than the unobserved pair. Neural Collaborative Filtering (NCF) [27]
adopts the feed-forward neural networks to output the ranking score of a user-
item pair and is trained with the binary cross-entropy loss. LightGCN (LGCN)
[29] adopts simplified Graph Convolutional Networks (GCN) [109] to capture the
high-order interaction between the user and the item, and is trained with the
loss function of BPR. Since PerK is a model-agnostic framework, other models
can be also adopted for PerK in future work.

For all the base recommender models, we basically follow the source code of
the authors and use PyTorch [49] for the implementation. For each dataset,
hyperparameters are tuned by using grid searches on the validation set. We use
Adam optimizer [50] with learning rate in $\\{10^{-2},10^{-3},10^{-4}\\}$ and
weight decay in $\\{0,10^{-1},10^{-3},10^{-5},10^{-7},10^{-9}\\}$. We set the
batch size to 8192 and the embedding size is chosen from $\\{64,128\\}$ for
all base models. For NCF and LGCN, the number of layers is chosen from
$\\{1,2\\}$. The negative sample rate is set to 1 for all models. Each model
is trained until the convergence of validation performance. The ranking score
from the base recommender model serves as input for PerK, AttnCut, and MtCut.

####  4.6.2 Comparison of User Utility

Table 4.2: Average user utility of recommendations produced by compared
methods when they are optimized for each target utility measure. Improv.
denotes the improvement of PerK over MtCut. The numbers in boldface denote the
best result in each setting. *We conduct the paired t-test with a 0.05 level
and every Improv. is statistically significant.

Base | Method | MovieLens 10M | CiteULike | MovieLens 25M | Amazon Books
---|---|---|---|---|---
Model | NDCG | PDCG | F1 | TP | NDCG | PDCG | F1 | TP | NDCG | PDCG | F1 | TP | NDCG | PDCG | F1 | TP
| Oracle | 0.6190 | 0.6702 | 0.3152 | 0.7078 | 0.3020 | -0.3677 | 0.1806 | 0.3940 | 0.5961 | 0.6089 | 0.2883 | 0.6812 | 0.1895 | -0.7438 | 0.1133 | 0.2643
| Top-1 | 0.4549 | -0.0902 | 0.0546 | 0.4549 | 0.1837 | -0.6326 | 0.0338 | 0.1837 | 0.4392 | -0.1216 | 0.0492 | 0.4392 | 0.0961 | -0.8078 | 0.0216 | 0.0961
| Top-5 | 0.3925 | -0.6428 | 0.1541 | 0.3751 | 0.1489 | -2.0787 | 0.0842 | 0.1401 | 0.3729 | -0.7561 | 0.1357 | 0.3547 | 0.0786 | -2.4938 | 0.0507 | 0.0741
| Top-10 | 0.3725 | -1.3715 | 0.1984 | 0.3584 | 0.1495 | -3.3650 | 0.1024 | 0.1483 | 0.3496 | -1.5372 | 0.1757 | 0.3328 | 0.0828 | -3.9401 | 0.0582 | 0.0852
| Top-20 | 0.3734 | -2.8434 | 0.2228 | 0.3849 | 0.1668 | -5.4853 | 0.1108 | 0.1907 | 0.3460 | -3.0494 | 0.2013 | 0.3510 | 0.0955 | -6.2660 | 0.0584 | 0.1155
BPR | Top-50 | 0.4057 | -7.1636 | 0.2117 | 0.4841 | 0.2021 | -10.8747 | 0.0930 | 0.2834 | 0.3740 | -7.3813 | 0.1983 | 0.4421 | 0.1203 | -11.8590 | 0.0496 | 0.1826
| Rand | 0.3872 | -3.6387 | 0.2046 | 0.4144 | 0.1738 | -6.2622 | 0.1003 | 0.2058 | 0.3611 | -3.8266 | 0.1865 | 0.3801 | 0.0998 | -7.1022 | 0.0535 | 0.1284
| Val-$k$ | 0.4562 | -0.0640 | 0.2270 | 0.4906 | 0.1922 | -0.6162 | 0.1061 | 0.2266 | 0.4368 | -0.0532 | 0.2052 | 0.4347 | 0.1093 | -0.8378 | 0.0563 | 0.1362
| AttnCut | 0.4604 | -0.0702 | 0.2371 | 0.4969 | 0.2024 | -0.6142 | 0.1063 | 0.2835 | 0.4392 | -0.1214 | 0.2087 | 0.4392 | 0.1203 | -0.8078 | 0.0566 | 0.1826
| MtCut | 0.4621 | -0.0167 | 0.2383 | 0.5037 | 0.2034 | -0.6022 | 0.1071 | 0.2838 | 0.4413 | -0.0616 | 0.2140 | 0.4639 | 0.1212 | -0.8078 | 0.0568 | 0.1826
| PerK | 0.4901 | 0.2087 | 0.2538 | 0.5711 | 0.2159 | -0.4971 | 0.1117 | 0.2993 | 0.4687 | 0.1876 | 0.2300 | 0.5401 | 0.1261 | -0.7952 | 0.0619 | 0.1894
| Improv. | 6.06%* | - | 6.50%* | 13.38%* | 6.15%* | - | 4.30%* | 5.46%* | 6.21%* | - | 7.48%* | 16.43%* | 4.04%* | - | 8.98%* | 3.72%*

Table 4.3: Average user utility of recommendations produced by compared
methods when they are optimized for each target utility measure. Improv.
denotes the improvement of PerK over MtCut. The numbers in boldface denote the
best result in each setting. *We conduct the paired t-test with a 0.05 level
and every Improv. is statistically significant.

Base | Method | MovieLens 10M | CiteULike | MovieLens 25M | Amazon Books
---|---|---|---|---|---
Model | NDCG | PDCG | F1 | TP | NDCG | PDCG | F1 | TP | NDCG | PDCG | F1 | TP | NDCG | PDCG | F1 | TP
| Oracle | 0.5760 | 0.4792 | 0.2896 | 0.6688 | 0.2813 | -0.4076 | 0.1685 | 0.3690 | 0.5310 | 0.3544 | 0.2513 | 0.6157 | 0.1444 | -0.8246 | 0.0885 | 0.2076
| Top-1 | 0.4081 | -0.1839 | 0.0474 | 0.4081 | 0.1700 | -0.6601 | 0.0304 | 0.1700 | 0.3761 | -0.2479 | 0.0393 | 0.3761 | 0.0666 | -0.8668 | 0.0150 | 0.0666
| Top-5 | 0.3521 | -0.8801 | 0.1350 | 0.3368 | 0.1403 | -2.1273 | 0.0786 | 0.1330 | 0.3206 | -1.0628 | 0.1117 | 0.3055 | 0.0563 | -2.6231 | 0.0370 | 0.0537
| Top-10 | 0.3360 | -1.6822 | 0.1764 | 0.3251 | 0.1384 | -3.4438 | 0.0942 | 0.1368 | 0.3011 | -1.9434 | 0.1474 | 0.2873 | 0.0606 | -4.1048 | 0.0434 | 0.0638
| Top-20 | 0.3397 | -3.2150 | 0.2023 | 0.3545 | 0.1529 | -5.5941 | 0.1015 | 0.1733 | 0.2980 | -3.5640 | 0.1721 | 0.3033 | 0.0712 | -6.4683 | 0.0443 | 0.0888
NCF | Top-50 | 0.3721 | -7.6021 | 0.1966 | 0.4525 | 0.1866 | -11.0026 | 0.0870 | 0.2618 | 0.3237 | -8.0277 | 0.1744 | 0.3870 | 0.0916 | -12.1148 | 0.0384 | 0.1437
| Rand | 0.3527 | -3.9939 | 0.1866 | 0.3821 | 0.1609 | -6.4157 | 0.0910 | 0.1884 | 0.3102 | -4.3422 | 0.1603 | 0.3298 | 0.0749 | -7.3212 | 0.0407 | 0.0987
| Val-$k$ | 0.4123 | -0.1030 | 0.2071 | 0.4554 | 0.1794 | -0.5801 | 0.0989 | 0.2101 | 0.3769 | -0.5071 | 0.1729 | 0.3842 | 0.0791 | -0.8935 | 0.0422 | 0.1018
| AttnCut | 0.4154 | -0.0839 | 0.2068 | 0.4612 | 0.1812 | -0.5527 | 0.0991 | 0.2619 | 0.3762 | -0.1166 | 0.1755 | 0.3994 | 0.0918 | -0.8668 | 0.0437 | 0.1438
| MtCut | 0.4195 | 0.0805 | 0.2153 | 0.4798 | 0.1891 | -0.5499 | 0.1006 | 0.2627 | 0.3798 | -0.0142 | 0.1837 | 0.4079 | 0.0923 | -0.8667 | 0.0441 | 0.1441
| PerK | 0.4482 | 0.0928 | 0.2293 | 0.5335 | 0.2061 | -0.5177 | 0.1080 | 0.2764 | 0.4056 | 0.0047 | 0.1960 | 0.4742 | 0.0982 | -0.8646 | 0.0468 | 0.1553
| Improv. | 6.84%* | - | 6.50%* | 11.19%* | 8.99%* | - | 7.36%* | 5.22%* | 6.79%* | - | 6.70%* | 16.25%* | 6.39%* | - | 6.12%* | 7.77%*

Table 4.4: Average user utility of recommendations produced by compared
methods when they are optimized for each target utility measure. Improv.
denotes the improvement of PerK over MtCut. The numbers in boldface denote the
best result in each setting. *We conduct the paired t-test with a 0.05 level
and every Improv. is statistically significant.

Base | Method | MovieLens 10M | CiteULike | MovieLens 25M | Amazon Books
---|---|---|---|---|---
Model | NDCG | PDCG | F1 | TP | NDCG | PDCG | F1 | TP | NDCG | PDCG | F1 | TP | NDCG | PDCG | F1 | TP
| Oracle | 0.6249 | 0.7459 | 0.3094 | 0.7051 | 0.3471 | -0.2352 | 0.2104 | 0.4400 | 0.5893 | 0.4840 | 0.2864 | 0.6772 | 0.1782 | -0.7571 | 0.1061 | 0.2454
| Top-1 | 0.4749 | -0.0502 | 0.0565 | 0.4749 | 0.2182 | -0.5636 | 0.0420 | 0.2182 | 0.4236 | -0.1527 | 0.0490 | 0.4236 | 0.0919 | -0.8162 | 0.0213 | 0.0919
| Top-5 | 0.4004 | -0.5959 | 0.1531 | 0.3799 | 0.1829 | -1.8801 | 0.1068 | 0.1735 | 0.3603 | -0.8313 | 0.1349 | 0.3427 | 0.0744 | -2.5187 | 0.0487 | 0.0699
| Top-10 | 0.3763 | -1.3345 | 0.1952 | 0.3584 | 0.1829 | -3.1084 | 0.1270 | 0.1816 | 0.3391 | -1.6383 | 0.1741 | 0.3237 | 0.0783 | -3.9772 | 0.0552 | 0.0803
| Top-20 | 0.3734 | -2.8186 | 0.2180 | 0.3793 | 0.1999 | -5.1979 | 0.1303 | 0.2244 | 0.3379 | -3.1805 | 0.1988 | 0.3449 | 0.0898 | -6.3212 | 0.0543 | 0.1077
LightGCN | Top-50 | 0.3997 | -7.1674 | 0.2068 | 0.4673 | 0.2354 | -10.5738 | 0.1037 | 0.3174 | 0.3687 | -7.5533 | 0.1951 | 0.4396 | 0.1117 | -11.9466 | 0.0449 | 0.1663
| Rand | 0.3875 | -3.5835 | 0.2001 | 0.4062 | 0.2075 | -6.0557 | 0.1182 | 0.2450 | 0.3527 | -3.9406 | 0.1839 | 0.3735 | 0.0938 | -7.1826 | 0.0497 | 0.1187
| Val-$k$ | 0.4652 | 0.0417 | 0.2225 | 0.4894 | 0.2296 | -0.4288 | 0.1286 | 0.2620 | 0.4253 | -0.0864 | 0.1953 | 0.4439 | 0.1017 | -0.8469 | 0.0532 | 0.1247
| AttnCut | 0.4749 | -0.0412 | 0.2237 | 0.4899 | 0.2260 | -0.5616 | 0.1291 | 0.3186 | 0.4242 | -0.1527 | 0.2041 | 0.4359 | 0.1121 | -0.8162 | 0.0516 | 0.1707
| MtCut | 0.4761 | 0.1171 | 0.2318 | 0.4949 | 0.2369 | -0.5487 | 0.1319 | 0.3223 | 0.4317 | 0.0649 | 0.2113 | 0.4440 | 0.1185 | -0.8162 | 0.0536 | 0.1761
| PerK | 0.4993 | 0.3309 | 0.2489 | 0.5702 | 0.2551 | -0.3989 | 0.1383 | 0.3438 | 0.4543 | 0.0876 | 0.2277 | 0.4742 | 0.1261 | -0.7912 | 0.0571 | 0.1878
| Improv. | 4.87%* | - | 7.38%* | 15.22%* | 7.68%* | - | 4.85%* | 6.67%* | 5.24%* | - | 7.76%* | 6.80%* | 6.41%* | - | 6.53%* | 6.64%*

Table 4.2, 4.3, and 4.4 present the average user utility yielded by the
recommendation of each compared method. For a fair comparison, we adopt the
same base model and the same target utility throughout the training of
AttnCut, MtCut, and PerK in each experimental configuration. We report the
average result of three independent runs.

Benefits of Top-Personalized-$K$ Recommendation. We first observe that
personalizing the recommendation size results in higher user utility than the
globally fixed size. Oracle shows significantly higher utility compared to the
best value of Top-$k$. This upper bound on user utility highlights the
importance of the proposed task in improving user satisfaction. Accordingly,
methods determining the personalized recommendation sizes (i.e., AttnCut,
MtCut, and PerK) generally outperform Top-$k$. Furthermore, we notice that
Top-$k$ shows a large performance deviation depending on the recommended size.
Therefore, the system should avoid naively setting a globally fixed
recommendation size and instead determine the personalized size for improved
user satisfaction.

Effectiveness of PerK. We observe that PerK outperforms the competitors in the
top-personalized-$K$ recommendation task. Val-$k$ does not perform well since
overfitting to the validation utility is not effective due to the sparse and
noisy interactions. Furthermore, the effectiveness of the methods for document
list truncation (i.e., AttnCut and MtCut) remains limited, since the target
label for the classification does not provide enough supervision due to the
hidden-positive and noisy interactions. Indeed, we discovered that they can be
easily overfitted to select the size that works well globally (Figure 4.2 and
4.3), resulting in comparable performance to the best of Top-$k$. In contrast,
PerK does not rely on a deep model and considers the hidden-relevant items, to
estimate the expected user utility. PerK directly computes the expected user
utility in mathematical form with the calibrated interaction probability and
significantly outperforms AttnCut and MtCut in the top-personalized-$K$ task.

Figure 4.2: Distribution of recommendation sizes from each method on MovieLens
25M. Three figures in the same row share the y-axis.

Figure 4.3: Distribution of recommendation sizes from each method on Amazon
Books. Three figures in the same row share the y-axis.

####  4.6.3 Personalized Recommendation Size

Figure 4.2 and 4.3 show the distributions of recommendation sizes determined
by Oracle, PerK, and MtCut. The base model is BPR and the target user utility
is F1. We have the following findings: (1) The distributions of Oracle show
that the optimal recommendation size for maximum user utility differs for each
user. (2) The distributions of MtCut are severely skewed towards $k\in[10,20]$
and have a high peak in that range. The reason is that MtCut is overfitted to
select the globally well-performing $k$ which falls into the range of [10, 20]
(refer to F1 on MovieLens 25M and Amazon Books with BPR in Table 4.2). (3) The
distributions of PerK are smooth and fairly close to those of Oracle. It is
noted that we can set the constraints for the minimum recommendation size in
Eq.4.3, if the system requires it.

####  4.6.4 User Degree vs Personalized Size

Figure 4.4: User degree versus Personalized Recommendation Size. Red dots
denote the average x (user degree) for each y (personalized K).

Figure 4.4 shows the relation between user degree (number of interactions) and
selected personalized recommendation size. The base model is BPR and the
target utility is F1. $r_{x,y}$ denotes the Pearson correlation coefficient.
Since both correlation coefficients are positive, there is a positive
correlation between user degree and selected personalized recommendation size.
If a user has sufficient interactions, the recommender model can capture
stable and certain preferences for that user. Therefore, generally speaking,
the mean confidence in top items increases and more number of recommendations
is provided. On the other hand, users with fewer interactions also may get
larger recommendations (Upper left of Figure 4.4). In this case, if a user has
a distinct preference for small interactions, the recommender model also may
provide many recommendations for that distinct preference.

####  4.6.5 Ablation Study for Calibration

Table 4.5: Calibration error and user utility for each calibration method.
User utilities are computed when each calibration method is applied to PerK.
Lower is better for ECE.

Dataset | Calibration | ECE $\downarrow$ | NDCG | PDCG | F1 | TP
---|---|---|---|---|---|---
ML10M | uncalibrated | 0.1284 | 0.4083 | -0.1071 | 0.1966 | 0.4481
global | 0.0046 | 0.4255 | 0.0838 | 0.2175 | 0.4925
user-wise | 0.0011 | 0.4482 | 0.0928 | 0.2293 | 0.5335
CiteULike | uncalibrated | 0.0480 | 0.1811 | -0.5667 | 0.0975 | 0.2512
global | 0.0017 | 0.1940 | -0.5306 | 0.1054 | 0.2639
user-wise | 0.0003 | 0.2061 | -0.5177 | 0.1080 | 0.2764
ML25M | uncalibrated | 0.1572 | 0.3612 | -0.2311 | 0.1669 | 0.3812
global | 0.0422 | 0.3761 | -0.1015 | 0.1757 | 0.4007
user-wise | 0.0098 | 0.4056 | 0.0047 | 0.1960 | 0.4742
ABooks | uncalibrated | 0.3371 | 0.0812 | -0.8864 | 0.0427 | 0.1391
global | 0.0581 | 0.0946 | -0.8655 | 0.0449 | 0.1477
user-wise | 0.0171 | 0.0982 | -0.8646 | 0.0468 | 0.1553

Table 4.5 present the ablation study on the calibration method with NCF [27]
as a base model. We compare (1) the calibration performance, and (2) user
utility when it is applied to PerK. The calibration performance is measured by
Expected Calibration Error (ECE) [15], a widely used metric for measuring the
gap between the output probability and true likelihood of interaction [1, 5].
We observe that the proposed user-wise calibration function shows lower ECE
than the global calibration function. Accordingly, PerK yields higher user
utilities when it adopts user-wise calibration, demonstrating the superiority
of the proposed user-wise calibration over the global calibration.

####  4.6.6 Space and Time analysis

Table 4.6: Space and Time analysis. #Params. denotes the number of learnable
parameters and Time denotes the wall time (in ms) used for generating a
ranking list for a user.

Base | Method | ML10M | CiteULike | ML25M | ABooks
---|---|---|---|---|---
model | #Params. | Time | #Params. | Time | #Params. | Time | #Params. | Time
BPR | Top-k | 5.041M | 2.871 | 1.291M | 2.086 | 11.574M | 2.374 | 34.542M | 2.636
PerK | 5.181M | 3.022 | 1.292M | 2.241 | 11.898M | 2.652 | 34.857M | 2.973
NCF | Top-k | 10.092M | 4.092 | 2.581M | 2.566 | 23.156M | 3.067 | 69.117M | 7.767
PerK | 10.232M | 4.406 | 2.588M | 2.817 | 23.480M | 3.378 | 69.432M | 8.871
LightGCN | Top-k | 5.042M | 3.195 | 2.571M | 2.493 | 23.147M | 2.628 | 34.542M | 3.781
PerK | 5.181M | 3.449 | 2.577M | 2.759 | 23.472M | 2.969 | 34,857M | 4.106

Table 4.6 shows the number of learnable parameters and inference time of
Top-$k$ and PerK. The target user utility is NDCG for PerK. We use PyTorch
[49] with CUDA on GTX Titan Xp GPU and Intel Xeon(R) E5-2640 v4 CPU.

First, PerK does not significantly increase the number of learnable parameters
from Top-$k$. PerK only has two additional parameters for each user’s
calibration function, and it has a negligible impact considering the size of
the user embedding is typically selected in the range of 64-128.

(a) NDCG

(b) F1

Figure 4.5: Hyperparameter study of $M$

Second, the inference time is increased by about 10%. To speed up the
estimation of expected user utility, (1) We perform the user-wise calibration
for all users together with a few matrix operations and estimate the expected
user utility with various $k$ in a parallel way, and (2) We adopt two
approximation techniques for fast computation of the expected user utility in
Eq.4.7. The hyperparameter study for $M$, which is used for this
approximation, with BPR is presented in Figure 4.5. We can see that it is
enough to aggregate the summation in Eq.7d just to $M=2000$ rather than to the
number of all unobserved items $|\mathcal{I}_{u}^{-}|$.

####  4.6.7 Case Study

(a) Single-domain scenario on Amazon Books

(b) Multi-domain scenario on Amazon Figure 4.6: Case study. 1 and 0 represent
the relevant and the irrelevant labels, which are unavailable at the inference
phase. Decimal numbers indicate the F1 score.

We present a case study on Amazon dataset to provide concrete examples of how
PerK can be applied in real-world applications.

Single-domain scenario. Figure 4.6 (a) shows a case study of the single-domain
scenario on Amazon Books with BPR as a base model and F1 as a target utility.
Real-world recommender systems often display trending items alongside
personalized ones, to create a balanced experience that encompasses both
popular choices and individual preferences [110]. We determine the number of
personalized items with PerK, and the remaining slots are then populated with
trending items. As a result, F1 for the personalized items increases by a
large margin for both users, and more trending items can be presented to the
second user. In this context, the top-personalized-$K$ recommendation allows
the system to strike a balance between the exploration-exploitation trade-off
by effectively adjusting the number of personalized items.

Multi-domain scenario. Figure 4.6 (b) shows a case study of the multi-domain
scenario on the four largest domains of Amazon datasets with BPR as a base
model and F1 as a target utility. Here, we have a constraint on the total
recommendation size considering the system’s limited resources, such as screen
size and thumbnail dimensions. We generate top-personalized-$K$
recommendations for each domain by slightly modifying Eq.4.3, to maximize the
average F1 across all domains under the constraint that the total
recommendation size should not exceed 40. As a result, we can see that the
average F1 score increases as we adopt the top-personalized-$K$ recommendation
scheme. This example demonstrates that the top-personalized-$K$ recommendation
can be adopted to multi-domain systems for displaying the optimized number of
items from each domain on a single constrained page.

###  4.7 Conclusion

We first highlight the necessity of personalized recommendation size based on
its practical advantages in real-world scenarios, which has not been studied
well in the previous literature. Then, we propose Top-Personalized-$K$
Recommendation, a new recommendation task that aims to find the optimal
recommendation size for each user to maximize individual user satisfaction. As
a solution to the top-personalized-$K$ recommendation, we propose PerK, a
framework determining the recommendation size that maximizes the expected user
utility estimated by using calibrated interaction probabilities. In our
thorough experiments on real-world datasets, PerK outperforms recent
competitors in the top-personalized-$K$ recommendation task. We believe that
the top-personalized-$K$ recommendation can provide enhanced solutions for
various item recommendation scenarios and anticipate future work on
applications including multi-domain recommender systems, sponsored
advertisements, and prefetching mechanisms.

##  Chapter 5 Conclusion

In this dissertation, I investigated confidence calibration in recommendations
and introduced two real-world applications of confidence on recommendations.
Specifically, (1) I proposed Gaussian calibration and Gamma calibration that
map the ranking scores of pre-trained models to well-calibrated preference
probabilities, without affecting the recommendation performance (Section 2).
(2) I propose a Bidirectional Distillation framework whereby both the teacher
and the student collaboratively improve with each other by treating the
confidence of the counterpart as additional learning guidance (Section 3). (3)
I introduce Top-Personalized-$K$ Recommendation, a new recommendation task
adjusting the number of presented items based on the expected user utility
estimated with calibrated probability (Section 4). We validate the superiority
of each proposed method with extensive experiments on real-world datasets. We
also provide in-depth analyses to verify the effectiveness of each proposed
component. In the future, I plan to continue my work on model calibration for
recommender systems, particularly for fairness and large language models of
recommendations.

Figure 5.1: An overview of this dissertation.

개인화된 추천은 쇼핑, 뉴스, 검색, 동영상, 음악 등 다양한 일상 활동에 큰 영향을 미친다. 대부분의 추천 시스템은 추천 결과에 대한
신뢰도를 전혀 제공하지 않고 사용자에게 상위 항목만 표시하고 있다. 하지만, 동일한 순위 위치의 의미는 사용자마다 다르다. 한 사용자는
30%의 확률로 첫 번째 항목을 좋아할 수 있는 반면, 다른 사용자는 90%의 확률로 첫 번째 항목을 좋아할 수 있다. 이 논문에서는 추천
시스템의 추천 결과에 대한 신뢰도를 추정할 수 있는 방법을 제시하고, 추정된 신뢰도를 활용하는 두 가지 실제 응용법을 소개한다. 구체적으로,
(1) 사전 훈련된 추천 모델의 순위 점수를 잘 보정된 신뢰도로 매핑하는 가우스 보정과 감마 보정을 제안한다 (Section 2). (2)
큰 교사 모델과 작은 학생 모델이 서로의 신뢰도를 추가적인 학습 레이블로 사용하여 학습하는 양방향 지식 증류 프레임워크를 제안한다
(Section 3). (3) 보정된 신뢰도를 이용하여 추정된 예상 사용자 효용을 기반으로 추천 항목 수를 조정하는 새로운 추천 패러다임인
Top-Personalized-K Recommendation를 제안한다 (Section 4). 본 논문에서는 제안된 모델들의 성능을 실제
데이터셋에 대한 실험으로 평가하였으며, 여러 분석을 통해 그 타당성을 입증하였다. 본 논문에서 소개되는 추천시스템의 보정된 신뢰도는 최근
주목 받고 있는 추천시스템의 형평성과 추천시스템을 위한 초거대 언어모델에도 활용가능하며, 앞으로 이에대한 연구를 진행 할 예정이다.

## References

  * [1] Wonbin Kweon, SeongKu Kang, and Hwanjo Yu.  Obtaining calibrated probabilities with personalized ranking models.  In AAAI, 2022.
  * [2] Steffen Rendle, Christoph Freudenthaler, Zeno Gantner, and Lars Schmidt-Thieme.  Bpr: Bayesian personalized ranking from implicit feedback.  In UAI, 2009.
  * [3] Aditya Krishna Menon, Xiaoqian J Jiang, Shankar Vembu, Charles Elkan, and Lucila Ohno-Machado.  Predicting accurate probabilities with a ranking loss.  In ICML, 2012.
  * [4] Avi Arampatzis, Jaap Kamps, and Stephen Robertson.  Where to stop reading a ranked list? threshold optimization using truncated score distributions.  In SIGIR, 2009.
  * [5] Chuan Guo, Geoff Pleiss, Yu Sun, and Kilian Q Weinberger.  On calibration of modern neural networks.  In ICML, 2017.
  * [6] Meelis Kull, Miquel Perello Nieto, Markus Kängsepp, Telmo Silva Filho, Hao Song, and Peter Flach.  Beyond temperature scaling: Obtaining well-calibrated multi-class probabilities with dirichlet calibration.  In NeurIPS, 2019.
  * [7] Amir Rahimi, Amirreza Shaban, Ching-An Cheng, Richard Hartley, and Byron Boots.  Intra order-preserving functions for calibration of multi-class neural networks.  In NeurIPS, 2020.
  * [8] Richard E Barlow and Hugh D Brunk.  The isotonic regression problem and its dual.  Journal of the American Statistical Association, 67(337):140–147, 1972.
  * [9] John Platt, Alexander Smola, Peter Bartlett, Bernhard Scholkopf, and Dale Schuurmans.  Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods.  Advances in large margin classifiers, 10(3):61–74, 1999.
  * [10] Tobias Schnabel, Adith Swaminathan, Ashudeep Singh, Navin Chandak, and Thorsten Joachims.  Recommendations as treatments: Debiasing learning and evaluation.  In ICML, 2016.
  * [11] Yuta Saito.  Unbiased pairwise learning from implicit feedback.  In NeurIPS 2019 Workshop on Causal Machine Learning, 2019.
  * [12] James M Robins, Andrea Rotnitzky, and Lue Ping Zhao.  Estimation of regression coefficients when some regressors are not always observed.  Journal of the American statistical Association, 89(427):846–866, 1994.
  * [13] Markus Weimer, Alexandros Karatzoglou, Quoc Le, and Alex Smola.  Cofirank-maximum margin matrix factorization for collaborative ranking.  In NeurIPS, 2007.
  * [14] Meelis Kull, Telmo Silva Filho, and Peter Flach.  Beta calibration: a well-founded and easily implemented improvement on logistic calibration for binary classifiers.  In AISTATS, 2017.
  * [15] Mahdi Pakdaman Naeini, Gregory Cooper, and Milos Hauskrecht.  Obtaining well calibrated probabilities using bayesian binning.  In AAAI, 2015.
  * [16] Bianca Zadrozny and Charles Elkan.  Obtaining calibrated probability estimates from decision trees and naive bayesian classifiers.  In ICML, 2001.
  * [17] Jishnu Mukhoti, Viveka Kulharia, Amartya Sanyal, Stuart Golodetz, Philip Torr, and Puneet Dokania.  Calibrating deep neural networks using focal loss.  In NeurIPS, 2020.
  * [18] Christoph Baumgarten.  A probabilistic solution to the selection and fusion problem in distributed information retrieval.  In SIGIR, 1999.
  * [19] John A Swets.  Effectiveness of information retrieval methods.  American Documentation, 20(1):72–89, 1969.
  * [20] Raghavan Manmatha, Toni Rath, and Fangfang Feng.  Modeling score distributions for combining the outputs of search engines.  In SIGIR, 2001.
  * [21] Evangelos Kanoulas, Keshi Dai, Virgil Pavlu, and Javed A Aslam.  Score distribution models: assumptions, intuition, and robustness to score manipulation.  In SIGIR, 2010.
  * [22] F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay.  Scikit-learn: Machine learning in Python.  Journal of Machine Learning Research, 12:2825–2830, 2011.
  * [23] Xiaojie Wang, Rui Zhang, Yu Sun, and Jianzhong Qi.  Doubly robust joint learning for recommendation on data missing not at random.  In ICML, 2019.
  * [24] Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel.  Unbiased learning-to-rank with biased feedback.  In WSDM, 2017.
  * [25] Paul R Rosenbaum.  Overt bias in observational studies.  In Observational studies, pages 71–104. Springer, 2002.
  * [26] Yi Su, Lequn Wang, Michele Santacatterina, and Thorsten Joachims.  Cab: Continuous adaptive blending for policy evaluation and learning.  In ICML, 2019.
  * [27] Xiangnan He, Lizi Liao, Hanwang Zhang, Liqiang Nie, Xia Hu, and Tat-Seng Chua.  Neural collaborative filtering.  In WWW, 2017.
  * [28] Cheng-Kang Hsieh, Longqi Yang, Yin Cui, Tsung-Yi Lin, Serge Belongie, and Deborah Estrin.  Collaborative metric learning.  In WWW, 2017.
  * [29] Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang.  Lightgcn: Simplifying and powering graph convolution network for recommendation.  In SIGIR, 2020.
  * [30] Sean M McNee, John Riedl, and Joseph A Konstan.  Being accurate is not enough: how accuracy metrics have hurt recommender systems.  In CHI’06 extended abstracts on Human factors in computing systems, pages 1097–1101, 2006.
  * [31] Wonbin Kweon, Seongku Kang, Junyoung Hwang, and Hwanjo Yu.  Deep rating elicitation for new users in collaborative filtering.  In WWW, 2020.
  * [32] Wonbin Kweon, SeongKu Kang, and Hwanjo Yu.  Bidirectional distillation for top-k recommender system.  In WWW, 2021.
  * [33] Jiaxi Tang and Ke Wang.  Ranking distillation: Learning compact ranking models with high performance for recommender system.  In KDD, 2018.
  * [34] J. Lee, M. Choi, J. Lee, and H. Shim.  Collaborative distillation for top-n recommendation.  In ICDM, 2019.
  * [35] SeongKu Kang, Junyoung Hwang, Wonbin Kweon, and Hwanjo Yu.  De-rrd: A knowledge distillation framework for recommender system.  In CIKM, 2020.
  * [36] Hao Wang, Binyi Chen, and Wu-Jun Li.  Collaborative topic regression with social regularization for tag recommendation.  In IJCAI, 2013.
  * [37] Dingqi Yang, Daqing Zhang, Vincent W Zheng, and Zhiyong Yu.  Modeling user activity preference by leveraging user spatial temporal characteristics in lbsns.  IEEE Transactions on Systems, Man, and Cybernetics: Systems, 45(1):129–142, 2014.
  * [38] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun.  Deep residual learning for image recognition.  In CVPR, 2016.
  * [39] Alex Krizhevsky, Geoffrey Hinton, et al.  Learning multiple layers of features from tiny images.  2009\.
  * [40] Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras.  Shallow-deep networks: Understanding and mitigating network overthinking.  In ICML, 2019.
  * [41] Geoffrey Hinton, Oriol Vinyals, and Jeff Dean.  Distilling the knowledge in a neural network.  arXiv preprint arXiv:1503.02531, 2015.
  * [42] Xiangnan He, Hanwang Zhang, Min-Yen Kan, and Tat-Seng Chua.  Fast matrix factorization for online recommendation with implicit feedback.  In SIGIR, 2016.
  * [43] Walid Krichene and Steffen Rendle.  On sampled metrics for item recommendation.  In KDD, 2020.
  * [44] Huayu Li, Richang Hong, Defu Lian, Zhiang Wu, Meng Wang, and Yong Ge.  A relaxed ranking-based factor model for recommender system from implicit feedback.  In IJCAI, 2016.
  * [45] Kalervo Järvelin and Jaana Kekäläinen.  Cumulated gain-based evaluation of ir techniques.  ACM Transactions on Information Systems (TOIS), pages 422–446, 2002\.
  * [46] Yao Wu, Christopher DuBois, Alice X Zheng, and Martin Ester.  Collaborative denoising auto-encoders for top-n recommender systems.  In WSDM, 2016.
  * [47] Pascal Vincent, Hugo Larochelle, Yoshua Bengio, and Pierre-Antoine Manzagol.  Extracting and composing robust features with denoising autoencoders.  In ICML, 2008.
  * [48] Yehuda Koren, Robert Bell, and Chris Volinsky.  Matrix factorization techniques for recommender systems.  Computer, 42(8):30–37, 2009.
  * [49] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al.  Pytorch: An imperative style, high-performance deep learning library.  In NeurIPS, 2019.
  * [50] Diederik P Kingma and Jimmy Ba.  Adam: A method for stochastic optimization.  In ICLR, 2015.
  * [51] Tommaso Furlanello, Zachary C Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar.  Born again neural networks.  In ICML, 2018.
  * [52] Linfeng Zhang, Jiebo Song, Anni Gao, Jingwei Chen, Chenglong Bao, and Kaisheng Ma.  Be your own teacher: Improve the performance of convolutional neural networks via self distillation.  In CVPR, 2019.
  * [53] Defu Lian, Rui Liu, Yong Ge, Kai Zheng, Xing Xie, and Longbing Cao.  Discrete content-aware matrix factorization.  In KDD, 2017.
  * [54] Hanwang Zhang, Fumin Shen, Wei Liu, Xiangnan He, Huanbo Luan, and Tat-Seng Chua.  Discrete collaborative filtering.  In SIGIR, 2016.
  * [55] Yan Zhang, Defu Lian, and Guowu Yang.  Discrete personalized ranking for fast collaborative filtering from implicit feedback.  In AAAI, 2017.
  * [56] Ke Zhou and Hongyuan Zha.  Learning binary codes for collaborative filtering.  In KDD, 2012.
  * [57] Wang-Cheng Kang and Julian McAuley.  Candidate generation with binary codes for large-scale top-n recommendation.  In CIKM, 2019.
  * [58] Yoram Bachrach, Yehuda Finkelstein, Ran Gilad-Bachrach, Liran Katzir, Noam Koenigstein, Nir Nice, and Ulrich Paquet.  Speeding up the xbox recommender system using a euclidean transformation for inner-product spaces.  In RecSys, 2014.
  * [59] Christina Teflioudi, Rainer Gemulla, and Olga Mykytiuk.  Lemp: Fast retrieval of large entries in a matrix product.  In SIGMOD, 2015.
  * [60] Hui Li, Tsz Nam Chan, Man Lung Yiu, and Nikos Mamoulis.  Fexipro: fast and exact inner product retrieval in recommender systems.  In SIGMOD, 2017.
  * [61] Ying Zhang, Tao Xiang, Timothy M Hospedales, and Huchuan Lu.  Deep mutual learning.  In CVPR, 2018.
  * [62] Guocong Song and Wei Chai.  Collaborative learning for deep neural networks.  In NeurIPS, 2018.
  * [63] Di He, Yingce Xia, Tao Qin, Liwei Wang, Nenghai Yu, Tie-Yan Liu, and Wei-Ying Ma.  Dual learning for machine translation.  In NeurIPS, 2016.
  * [64] Paolo Cremonesi, Yehuda Koren, and Roberto Turrin.  Performance of recommender algorithms on top-n recommendation tasks.  In RecSys, 2010.
  * [65] Jiancan Wu, Xiang Wang, Fuli Feng, Xiangnan He, Liang Chen, Jianxun Lian, and Xing Xie.  Self-supervised graph learning for recommendation.  In SIGIR, 2021.
  * [66] Yankai Chen, Huifeng Guo, Yingxue Zhang, Chen Ma, Ruiming Tang, Jingjie Li, and Irwin King.  Learning binarized graph representations with multi-faceted quantization reinforcement for top-k recommendation.  In KDD, 2022.
  * [67] Lianghao Xia, Chao Huang, and Chuxu Zhang.  Self-supervised hypergraph transformer for recommender systems.  In KDD, 2022.
  * [68] Stephen E Robertson.  The probability ranking principle in ir.  Journal of documentation, 33(4):209–304, 1977.
  * [69] Andrei Broder, Massimiliano Ciaramita, Marcus Fontoura, Evgeniy Gabrilovich, Vanja Josifovski, Donald Metzler, Vanessa Murdock, and Vassilis Plachouras.  To swing or not to swing: learning when (not) to advertise.  In CIKM, 2008.
  * [70] Dietmar Jannach and Michael Jugovac.  Measuring the business value of recommender systems.  ACM Transactions on Management Information Systems (TMIS), 10(4):1–23, 2019.
  * [71] Jie Tang, Sen Wu, Jimeng Sun, and Hang Su.  Cross-domain collaboration recommendation.  In KDD, 2012.
  * [72] Stefan Wilk, Dominik Schreiber, Denny Stohr, and Wolfgang Effelsberg.  On the effectiveness of video prefetching relying on recommender systems for mobile devices.  In 2016 13th IEEE Annual Consumer Communications & Networking Conference (CCNC), pages 429–434. IEEE, 2016.
  * [73] Ashudeep Singh and Thorsten Joachims.  Fairness of exposure in rankings.  In KDD, 2018.
  * [74] Yuta Saito and Thorsten Joachims.  Fair ranking as fair division: Impact-based individual fairness in ranking.  In KDD, 2022.
  * [75] Dawen Liang, Rahul G Krishnan, Matthew D Hoffman, and Tony Jebara.  Variational autoencoders for collaborative filtering.  In WWW, 2018.
  * [76] Avi Arampatzis, Jaap Kamps, and Stephen Robertson.  Where to stop reading a ranked list? threshold optimization using truncated score distributions.  In SIGIR, 2009.
  * [77] Dara Bahri, Yi Tay, Che Zheng, Donald Metzler, and Andrew Tomkins.  Choppy: Cut transformer for ranked list truncation.  In SIGIR, 2020.
  * [78] Stephen Tomlinson, Douglas W Oard, Jason R Baron, and Paul Thompson.  Overview of the trec 2007 legal track.  In TREC, 2007.
  * [79] Yixiao Ma, Qingyao Ai, Yueyue Wu, Yunqiu Shao, Yiqun Liu, Min Zhang, and Shaoping Ma.  Incorporating retrieval information into the truncation of ranking lists for better legal search.  In SIGIR, 2022.
  * [80] Chen Wu, Ruqing Zhang, Jiafeng Guo, Yixing Fan, Yanyan Lan, and Xueqi Cheng.  Learning to truncate ranked lists for information retrieval.  In AAAI, 2021.
  * [81] Dong Wang, Jianxin Li, Tianchen Zhu, Haoyi Zhou, Qishan Zhu, Yuxin Wen, and Hongming Piao.  Mtcut: A multi-task framework for ranked list truncation.  In WSDM, 2022.
  * [82] Yen-Chieh Lien, Daniel Cohen, and W Bruce Croft.  An assumption-free approach to the dynamic truncation of ranked lists.  In SIGIR, 2019.
  * [83] Sepp Hochreiter and Jürgen Schmidhuber.  Long short-term memory.  Neural computation, 9(8):1735–1780, 1997.
  * [84] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin.  Attention is all you need.  In NeurIPS, 2017.
  * [85] Tao Qin, Tie-Yan Liu, Jun Xu, and Hang Li.  Letor: A benchmark collection for research on learning to rank for information retrieval.  Information Retrieval, 13(4):346–374, 2010.
  * [86] Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng.  Ms marco: A human generated machine reading comprehension dataset.  In CoCo@NeurIPS, 2016.
  * [87] Juan Ramos et al.  Using tf-idf to determine word relevance in document queries.  In Proceedings of the first instructional conference on machine learning, volume 242, pages 29–48. Citeseer, 2003.
  * [88] Quoc Le and Tomas Mikolov.  Distributed representations of sentences and documents.  In ICML, 2014.
  * [89] Zhiyong Cheng, Ying Ding, Lei Zhu, and Mohan Kankanhalli.  Aspect-aware latent factor model: Rating prediction with ratings and reviews.  In WWW, 2018.
  * [90] Yifan Hu, Yehuda Koren, and Chris Volinsky.  Collaborative filtering for implicit feedback datasets.  In ICDM, 2008.
  * [91] Yuta Saito, Suguru Yaginuma, Yuta Nishino, Hayato Sakata, and Kazuhide Nakata.  Unbiased recommender learning from missing-not-at-random implicit feedback.  In WSDM, 2020.
  * [92] Aaron van den Oord, Yazhe Li, and Oriol Vinyals.  Representation learning with contrastive predictive coding.  arXiv preprint arXiv:1807.03748, 2018.
  * [93] An Zhang, Wenchang Ma, Xiang Wang, and Tat-Seng Chua.  Incorporating bias-aware margins into contrastive loss for collaborative filtering.  In NeurIPS, 2022.
  * [94] Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey.  An experimental comparison of click position-bias models.  In WWW, 2008.
  * [95] David MW Powers.  Evaluation: from precision, recall and f-measure to roc, informedness, markedness and correlation.  Journal of Machine Learning Technologies, 2:37–63, 2011.
  * [96] Enrique Amigó, Stefano Mizzaro, and Damiano Spina.  Ranking interruptus: When truncated rankings are better and how to measure that.  In SIGIR, 2022.
  * [97] Tong Man, Huawei Shen, Xiaolong Jin, and Xueqi Cheng.  Cross-domain recommendation: An embedding and mapping approach.  In IJCAI, 2017.
  * [98] Song Yao and Carl F Mela.  A dynamic model of sponsored search advertising.  Marketing Science, 30(3):447–468, 2011.
  * [99] Harvey M Salkin and Cornelis A De Kluyver.  The knapsack problem: a survey.  Naval Research Logistics Quarterly, 22(1):127–144, 1975.
  * [100] Patrick Billingsley.  Probability and measure.  John Wiley & Sons, 2008.
  * [101] Lucien Le Cam.  An approximation theorem for the poisson binomial distribution.  Pacific Journal of Mathematics, 10(4):1181–1197, 1960.
  * [102] Matthias Minderer, Josip Djolonga, Rob Romijnders, Frances Hubis, Xiaohua Zhai, Neil Houlsby, Dustin Tran, and Mario Lucic.  Revisiting the calibration of modern neural networks.  In NeurIPS, 2021.
  * [103] Zhipeng Ding, Xu Han, Peirong Liu, and Marc Niethammer.  Local temperature scaling for probability calibration.  In ICCV, 2021.
  * [104] Shrey Desai and Greg Durrett.  Calibration of pre-trained transformers.  In EMNLP, 2020.
  * [105] Jianmo Ni, Jiacheng Li, and Julian McAuley.  Justifying recommendations using distantly-labeled reviews and fine-grained aspects.  In EMNLP-IJCNLP, 2019.
  * [106] Hao Wang, Naiyan Wang, and Dit-Yan Yeung.  Collaborative deep learning for recommender systems.  In KDD, 2015.
  * [107] Xiaopeng Li and James She.  Collaborative variational autoencoder for recommender systems.  In KDD, 2017.
  * [108] Jiaqi Ma, Zhe Zhao, Xinyang Yi, Jilin Chen, Lichan Hong, and Ed H Chi.  Modeling task relationships in multi-task learning with multi-gate mixture-of-experts.  In KDD, 2018.
  * [109] Thomas N Kipf and Max Welling.  Semi-supervised classification with graph convolutional networks.  In ICLR, 2017.
  * [110] Sarika Jain, Anjali Grover, Praveen Singh Thakur, and Sourabh Kumar Choudhary.  Trends, problems and solutions of recommender system.  In International conference on computing, communication & automation, pages 955–958. IEEE, 2015.

[korean] 2015년에 포항에 와서, 9년 동안 공부하고 연구하다가, 박사 학위를 받게 되었습니다. 지난 5년 동안 지도해주신 유환조
교수님, 자유로운 연구 환경을 조성해 주시면서도 깊은 통찰을 말씀해 주실 때마다 많이 배웠습니다. 바쁘신 와중에도 박사학위 논문 심사를
맡아주신 이근배 교수님, 김동우 교수님, 이남훈 교수님, 고영명 교수님께도 감사의 말씀을 드립니다. 교수님들이 주신 피드백을 바탕으로 더
탄탄한 연구를 할 수 있었고, 앞으로도 좋은 연구를 하도록 노력하겠습니다.

학부 4년, 대학원 5년을 다시금 돌아보니 많은 분들에게 도움을 받으며, 배우며, 성장했던 것 같습니다. 모난 사람 없이 서로서로 잘 챙겼던
데이터 인텔리전스 연구실 가족들, 학부 때부터 같이 지낸 산경과 식구들, 축구보다 술을 더 같이한 기창/준혁/가종, 동기들보다 더 편하게 잘
지낸 분반 형님들, 신년회 때마다 보는 경남과학고 동기들, 등산메이트 성호와 홍준, 포항에서 지낸다고 자주 보지 못했던 제일중 친구들까지,
모두가 각자의 장점을 가지고 있어 하나씩 배우려고 노력했습니다. 다들 고맙습니다. 같이 만나 웃고 떠들며 대학원 생활을 잘 보낼 수
있었습니다. 마지막으로, 항상 응원해 주는 가족들에게도 감사의 인사를 전합니다. 앞으로 보답하려 노력하겠습니다. 제 곁에 머물렀던 모두가
바라는 일 이루기를 바라고, 시간이 지나 건강한 모습으로 다시 만났으면 좋겠습니다.

[korean] Wonbin Kweon

Department of Industrial and Management Engineering,
Pohang University of Science and Technology (B.S.)

Department of Convergence IT Engineering,
Pohang University of Science and Technology (Ph.D.)

AAAI’23-24, KDD’23, NeurIPS’23, ICLR’24, WWW’24, ICML’24

Fourth Prize, The 3rd POSTECH Research Award, 2023
AAAI Student Scholarship, 2022
Third Prize, Kakao Music Playlist Recommendation Competition, 2020
WWW Student Scholarship, 2020

Publications

  1. 1.

Wonbin Kweon, SeongKu Kang, Junyoung Hwang, Hwanjo Yu. Deep Rating Elicitation
for New Users in Collaborative Filtering. WWW 2020.

  2. 2.

SeongKu Kang, Junyoung Hwang, Wonbin Kweon, Hwanjo Yu. DE-RRD: A Knowledge
Distillation Framework for Recommender System. CIKM 2020.

  3. 3.

Wonbin Kweon, SeongKu Kang, Hwanjo Yu. Bidirectional Distillation for Top-K
Recommender System. WWW 2021.

  4. 4.

SeongKu Kang, Junyoung Hwang, Wonbin Kweon, Hwanjo Yu. Topology Distillation
for Recommender System. KDD 2021.

  5. 5.

SeongKu Kang, Dongha Lee, Wonbin Kweon, Junyoung Hwang, Hwanjo Yu. Consensus
Learning from Heterogeneous Objectives for One-Class Collaborative Filtering.
WWW 2022.

  6. 6.

Wonbin Kweon, SeongKu Kang, Hwanjo Yu. Obtaining Calibrated Probabilities with
Personalized Ranking Models. AAAI 2022 (oral presentation).

  7. 7.

SeongKu Kang, Wonbin Kweon, Dongha Lee, Jianxun Lian, Xing Xie, Hwanjo Yu.
Distillation from Heterogeneous Models for Top-K Recommendation. WWW 2023.

  8. 8.

Wonbin Kweon, SeongKu Kang, Sanghwan Jang, Hwanjo Yu. Top-Personalized-K
Recommendation. WWW 2024.

  9. 9.

Wonbin Kweon, Hwanjo Yu. Doubly Calibrated Estimator for Recommendation on
Data Missing Not At Random. WWW 2024.