# Introducing a Framework and a Decision Protocol to Calibrate Recommender
Systems

Diego Corrêa da Silva<EMAIL_ADDRESS>0000-0001-7132-1977 Federal
University of BahiaAv. Milton Santos, s/nSalvadorBahiaBrazil40170-110 and
Frederico Araújo Durão<EMAIL_ADDRESS>0000-0002-7766-6666 Federal University
of BahiaAv. Milton Santos, s/nSalvadorBahiaBrazil40170-110

(2022)

###### Abstract.

Recommender Systems use the user’s profile to generate a recommendation list
with unknown items to a target user. Although the primary goal of traditional
recommendation systems is to deliver the most relevant items, such an effort
unintentionally can cause collateral effects including low diversity and
unbalanced genres or categories, benefiting particular groups of categories.
This paper proposes an approach to create recommendation lists with a
calibrated balance of genres, avoiding disproportion between the user’s
profile interests and the recommendation list. The calibrated recommendations
consider concomitantly the relevance and the divergence between the genres
distributions extracted from the user’s preference and the recommendation
list. The main claim is that calibration can contribute positively to generate
fairer recommendations. In particular, we propose a new trade-off equation,
which considers the users’ bias to provide a recommendation list that seeks
for the users’ tendencies. Moreover, we propose a conceptual framework and a
decision protocol to generate more than one thousand combinations of
calibrated systems in order to find the best combination. We compare our
approach against state-of-the-art approaches using multiple domain datasets,
which are analyzed by rank and calibration metrics. The results indicate that
the trade-off, which considers the users’ bias, produces positive effects on
the precision and to the fairness, thus generating recommendation lists that
respect the genre distribution and, through the decision protocol, we also
found the best system for each dataset.

Calibration, Fairness, Framework, Protocol, Recommender System

††copyright: acmcopyright††journalyear: 2022††doi: XXXXXXX.XXXXXXX††journal:
JACM††journalvolume: 1††journalnumber: 1††article: 1††publicationmonth:
1††ccs: Information systems Recommender systems

##  1\. Introduction

Usually, recommender systems seek to recommend the most relevant items
according to the user’s profile. The recommendation is provided as a list
ordered by the most relevant items, i. e., the more relevant the more on top.
However, the focus on just recommending relevant items brings some drawbacks
such as low diversity of categories (Kaminskas and Bridge, 2016), unbalanced
genres (Abdollahpouri et al., 2020), popularity bias (Abdollahpouri et al.,
2017) or miscalibration (Steck, 2018). Therefore, in this paper we investigate
the calibration recommendation context.

A calibrated recommender system seeks to create a recommendation list,
following the proportion of each area (genres, classes, or tags) in the user’s
preferences. To this end, the system needs to understand all user’s interests
as a distribution. This distribution will be used as target to the
recommendation list, which is designed to be as little divergent as possible.
When the recommendation list distribution is highly divergent with user’s
preferences distribution, the state-of-the-art calls it miscalibration (Steck,
2018; Lin et al., 2020; da Silva et al., 2021). The miscalibration could
simply mean that the system fails to provide accurate personalized items
together with the fair treatment of the classes in the user’s preferences
(Abdollahpouri et al., 2020; da Silva et al., 2021). Thus, a calibrated system
creates a recommendation list considering relevant items and fair
representation of the user’s interest classes. The state-of-the-art defines
calibration as a way to provide some degree of fairness in which all user’s
interests are respected (Steck, 2018; Abdollahpouri et al., 2020; da Silva et
al., 2021).

As an example of calibration, in the song context, a user’s preferences
composed of 60% Samba, 30% Rock and 10% K-Pop will create a recommendation
list with these proportions. However, it is expected some divergence between
the proportions on the user’s preferences and the user’s recommendation list,
in the example it can be 56% Samba, 31% Rock and 13% K-Pop. As introduced, the
lower the divergence the better the calibration.

In an implementation point-of-view, the calibrated recommendations (Steck,
2018) are described as a post-processing, which rearranges the recommendation
list from the recommender algorithm. The state-of-the-art (Section 4) develops
many implementations, each one focused on their own proposals with workflows
assigned by the system specialist. Based on that, our first research claims is
a conceptual framework. We use the pattern of pre-processing, processing, and
post-processing, commonly used in the field (Pitoura et al., 2021), to
decompose the calibrated system in these three steps and twelve components
(Section 2). To test the framework reproducibility, we implement 1092
calibrated recommendation systems (Section 5). In this sense, our second
proposal is based on experiments that implement many calibrated systems and
need to decide which is the best one. To automatically decide and indicate the
best system to each dataset, we introduce the decision protocol and its
coefficients (Section 3). Our third claim is a new formula to control the
degree of relevant items and divergence (between the user’s preferences and
the user’s recommendation list).

From the claims, we will conduct the discussion following the research
questions (Section 6):

  * •

RQ1) How to find the best calibrated system to each domain?

  * •

RQ2) Do the calibrated systems have the same performance on the datasets? Are
there differences among the systems behavior?

  * •

RQ3) Is it possible to create a framework model for calibrated systems? What
are the important components to these systems?

  * •

RQ4) When the user’s bias is considered in the calibration trade-off, is it
possible to improve the system performance?

##  2\. Calibrated Recommendations

In this section, we cover the first part of our claims, the conceptual
framework to calibrated recommendation systems. The framework is inspired by
other studies (Steck, 2018; Kaya and Bridge, 2019; Abdollahpouri et al., 2020;
da Silva et al., 2021) that exploit singular points of the calibrated system.
These works don’t present a generalized description of the calibrated system.
The lack of specifications to produce modular systems may lead the specialist
to avoid exploring new possibilities and to improve the system.

In addition to the conceptual framework, a new equation is proposed to balance
the list of recommendations with relevant and fair items. As we proceed in the
section, we describe the framework and introduce the equations used in our
experiments.

###  2.1. The Framework

As introduced, we claim a conceptual framework to calibrated recommendations.
The framework is systematically divided into stages and components, which is
described in order.

####  2.1.1. System steps

The calibrated recommendation systems can be described in three major stages:
pre-processing, processing and post-processing. Each one has its restricted
function in the system. The final stage is the most important, because that’s
where the fairer list of recommendations is created. Each stage is outlined as
follows:

  1. (1)

The pre-processing deals with data treatment. This first step receives a raw
input data set and returns one that can be used by the calibrated
recommendation system. The usable data have the requirements to be used during
the process stage. In Section 5.1, we present the two datasets, showing their
numbers of users, items and interactions, before and after the pre-processing;

  2. (2)

Processing is the stage at which the recommender algorithm is implemented. It
is common for the recommendation systems to produce a recommendation list with
the recommender output. However, in calibrated recommendation systems, the
output of the recommendation algorithm serves as the input for post-
processing. In this paper, we refer to the recommendation list created by the
recommender algorithm as candidate items. In Section 5.2, we discuss the seven
recommenders used in our experiments;

  3. (3)

The post-processing is responsible for receiving the candidate items and using
them to create the fair recommendation list. In particular, this stage is
described along this section.

####  2.1.2. System Components

The pre-processing step is divided into three components: cleaning, filtering
and modeling. Our framework defines three core components to the pre-
processing, due to the calibration requires a class/genre in addition to the
user interactions (rating/likes). The cleaning component removes data that
cannot be used by the calibrated system, such as missing and incorrect
information. The filtering component blocks data that can provide noise to the
system. In the end, the pre-processing models the data to be used in the
processing step.

For instance, Table 1 provides an example of songs modeled under our
conceptual framework. The information used to calibrate will be the genre of
the songs. Table 2 presents three users and their information. We only use
users’ preferences, i. e., the interactions with the recommendation system. It
is possible to apply other users’ information depending on the recommendation
technique. However, our framework is ready to work in the collaborative
filtering (Steck, 2018; da Silva et al., 2021) and content-based filtering
(Starychfojtu and Peska, 2020). Table 3 shows the preferences of three modeled
users, with the user-item-rating composing one tuple. We introduce two formal
definitions in these examples: first the set of classes $C$ and a single class
$c$, and second is the rating (or score, weight, times played) represented as
$w_{ui}$.

Table 1. Eight items and their genres as classes.

UID | $c\in C$
---|---
I-001 | Pop$|$Rock
I-002 | Pop
I-003 | Blues
I-004 | Samba
I-005 | Funk
I-006 | K-Pop
I-007 | MPB$|$Funk
I-008 | Pop$|$Rock$|$Pagode$|$Funk

Source: Table created by the author.

Table 2. Three hypothetical users.

UID
---
U-001
U-002
U-003

Source: Table created by the author.

Table 3. Three users’ preference set.

User UID | Item UID | $w_{ui}$
---|---|---
U-001 | I-001 | 1
U-001 | I-002 | 4
U-001 | I-008 | 5
U-002 | I-004 | 2
U-002 | I-007 | 10
U-003 | I-001 | 9
U-003 | I-003 | 11
U-003 | I-004 | 3

Source: Table created by the author.

The processing step is composed only by the recommender algorithm component,
which is responsible for executing the recommender algorithm and producing the
candidate items. In this second step any technique can be implemented as
collaborative filtering, content-based filtering or hybrid technique, as well
as its recommender algorithms. Most of the related works implement
collaborative filtering. Thus, inspired by the state-of-the-art (Steck, 2018;
Kaya and Bridge, 2019; da Silva et al., 2021; Abdollahpouri et al., 2020), we
implement this technique. In some of these works the recommenders provide a
rank prediction (Steck, 2018; Kaya and Bridge, 2019) and others provide
ratings predictions (da Silva et al., 2021). Regardless of the recommender
algorithm implementation the output of this step is the user’s candidate
items. Table 4 provides an example of three users and their candidate items.
The formal representation of the predicted rating is $\hat{w}_{u,i}$. For each
unknown item by the user, the recommender algorithm predicts a possible weight
(e. g. rating) that the user will attribute to the item.

Table 4. Three hypothetical users and their candidate items.

User | Item | $\hat{w}_{u,i}$
---|---|---
U-001 | I-001 | 1
U-001 | I-002 | 4
U-001 | I-008 | 5
U-002 | I-004 | 2
U-002 | I-007 | 10
U-003 | I-001 | 9
U-003 | I-003 | 11
U-003 | I-004 | 3

Source: Table created by the author.

Post-processing is the most critical step for the calibrated recommendation
system. It receives from the processing step a set of candidate items and uses
them to create a final recommendation list. The objective of this step is to
select a subset of items from the user’s candidate items that are relevant and
fair to the user. To achieve that many formulas are used to compose the
workflow. Thus, the state-of-the-art (Steck, 2018; Kaya and Bridge, 2019;
Abdollahpouri et al., 2020; Starychfojtu and Peska, 2020; da Silva et al.,
2021) implements different systems. Along this section, we investigate the
basic component as well as some generalized equations.

The eight basic components of the post-processing are described as follows: i)
filtering, ii) modeling, iii) distributions, iv) calibration measure (aka
divergence measure), v) relevance measure, vi) trade-off weight ($\lambda$),
vii) trade-off balance and viii) select item algorithm. The filtering
component, on the post-processing, will filter the inputted candidate items.
The state-of-the-art (Kaya and Bridge, 2019; da Silva et al., 2021; Steck,
2018) uses this component to get the 100 most relevant candidate items or the
1000 top candidate items (Seymen et al., 2021). In some cases, it is not
possible to use all candidate items, due to the time-complexity which is a NP-
hard problem (Steck, 2018; da Silva et al., 2021). Thus, inspired by the
state-of-the-art we filter the 100 top relevant items in the candidate items
set, i. e., the 100 items with high value in $\hat{w}_{u,i}$. The modeling
component is responsible for moldeling the candidate items as a usable
entrance to the post-processing. A predicted weight $\hat{w}_{u,i}$ is needed
in each candidate item, thus the component prepares the items with the
requested information and structure. All other components will be presented in
sequence as sections. Figure 1 shows the conceptual framework with its steps
and components as described so far.

Figure 1. The conceptual framework with its steps and components.

###  2.2. Notations

The mathematical notations used in the following sections are formally
presented in Table 5. Along this section all notations are contextualized.

Table 5. Formal representation of the notations used in this study.

Notation | Description
---|---
$U$ | Set of all users
$u$ | A user
$I$ | Set of all items
$i$ | An item
$G$ | Set of all genres
$g$ | A genre
$I(u)$ | Set of items in the user preference $u$
$CI(u)$ | Set of candidate items to be recommended to user $u$
$L$ | A generic list with ordered items
$W(i)$ | All feedbacks weights values over an item $i$
$w_{u,i}$ | Feedback weight value given by $u$ over $i$
$\hat{w}_{u,i}$ | Predicted feedback weight value for $u$ over $i$
$N$ | Length of the final recommended list
$R^{*}$ | Final recommended list to a user

Source: Table created by the author.

###  2.3. Class Distributions

As introduced, the calibration recommendation system aims to create a list of
recommendations that tracks the distribution of user’s preferences, i. e., the
system extracts from the user’s preferences items a target distribution which
the recommendation list must achieve. Distributions are obtained from the item
classes. Extracting is done in different ways by the state-of-the-art, some
cases implements genres (Steck, 2018; da Silva et al., 2021), popularity
(Abdollahpouri et al., 2021) or sub-profiles (Kaya and Bridge, 2019).

####  2.3.1. Class Approach

Inspired by the state-of-the-art (Steck, 2018; da Silva et al., 2021), we
consider as part of the distribution equation the genre probability of being
chosen in an item. p(g$|$i) receives a genre ($g$) and an item ($i$) and
verifies if the genre composes the item. If the genre belongs to the item, the
equation returns values a value from the computation described as:

(1) $p(g|i)=\frac{1}{|classesIn(i)|}.$

####  2.3.2. Target Distribution

It is based on the user’s preferences items $I(u)$. $p$ is the formal
representation of the Target Distribution (Sacharidis et al., 2019) and it is
described as:

(2) $p(g\text{\textbar}u)=\frac{\sum_{i\in I(u)}\mathbb{1}(g\in i)w_{u,i}\cdot
p(g\text{\textbar}i)}{\sum_{i\in I(u)}\mathbb{1}(g\in i)w_{u,i}},$

where $w_{u,i}$ is the item relevance weight value (aka rating) from the user
$u$ over an item $i$; $p(g\text{\textbar}i)$ is the genre probability value
presented in Equation 1; $\mathbb{1}(g\in i)$ indicates that if the genre $g$
is present in an item $i$, than the result returns 1, if it is not, the result
returns 0.

####  2.3.3. Realized Distribution

It represents the distribution from the recommendation list $L$, which the
system aims to produce in less divergence with $p$. $q$ is the formal
representation of the Realized Distribution (Sacharidis et al., 2019) as
follows:

(3) $q(g\text{\textbar}u)=\frac{\sum_{i\in L(u)}\mathbb{1}(g\in
i)\hat{w}_{u,i}\cdot p(g\text{\textbar}i)}{\sum_{i\in CI(u)}\mathbb{1}(g\in
i)\hat{w}_{u,i}},$

where $q$ is similar to $p$. The difference between them is the weighing
strategy $\hat{w}_{u,i}$, which in our approach is the predicted relevance
weight from the recommender algorithm.

Steck (Steck, 2018) proposes $\tilde{q}$ instead of $q$ to lead with zero
values and it is presented as:

(4) $\tilde{q}(g\text{\textbar}u)=(1-\alpha)\cdot
q(g\text{\textbar}u)+\alpha\cdot p(g\text{\textbar}u).$

In the same sense of Steck, we use $\tilde{q}=q$.

###  2.4. Calibration Measure

Likewise (da Silva et al., 2021), in order to measure how much calibrated $p$
and $q$ are, we use three divergence calibration measures: 1) Kullback-Leibler
($F_{KL}$), 2) Hellinger ($F_{HE}$) and 3) Pearson Chi-Square ($F_{CHI}$). All
measures work in an interval between $[0,\infty]\in\mathbb{R}$. High values
indicate miscalibration between $p$ and $q$ and low values indicate
calibration. In some works (Zhao et al., 2021; Seymen et al., 2021), the
divergence measure is called calibration measure.

####  2.4.1. Kullback-Leibler

The KL-divergence equation is presented as:

(5) $F_{KL}(p,q)=\sum_{g\in
G}p(g\text{\textbar}u)\log_{2}\frac{p(g\text{\textbar}u)}{\tilde{q}(g\text{\textbar}u)}.$

The studies (Steck, 2018; Kaya and Bridge, 2019; da Silva et al., 2021)
implement the same equation.

####  2.4.2. Hellinger

Steck (Steck, 2018) alerts in his work that Hellinger can be used as a
calibration measure. Consequently, Silva et al. (da Silva et al., 2021)
implement it in their work as described below:

(6) $F_{HE}(p,q)=\sqrt[]{2\cdot\sum_{g\in
G}(\sqrt[]{p(g\text{\textbar}u)}-\sqrt[]{q(g\text{\textbar}u)})^{2}}.$

####  2.4.3. Pearson Chi Square

Silva et al. (da Silva et al., 2021) propose to use the Pearson Chi Square. By
that we applied that measure too as follows:

(7) $F_{CHI}(p,q)=\sum_{g\in
G}\frac{(p(g\text{\textbar}u)-\tilde{q}(g\text{\textbar}u))^{2}}{\tilde{q}(g\text{\textbar}u)}.$

###  2.5. Relevance Rank Measure

Similarly to (Steck, 2018; Kaya and Bridge, 2019; da Silva et al., 2021), we
use an equation to obtain a relevance rank weight as follows:

(8) $Sim(L)=\sum_{i\in L}w_{r(u,i)},$

where the $Sim(L)$ is the sum of the predicted relevance weight given by the
recommender algorithm.

###  2.6. Personalized Trade-Off Weight

Most of related works (Steck, 2018; Kaya and Bridge, 2019; Zhao et al., 2020,
2021) use constant trade-off weights ($\lambda$). Silva et al. (da Silva et
al., 2021) propose two personalized ways to find the trade-off weight. As
proposed by our conceptual framework, the trade-off weight is a component of
the post-processing step and such component acts as a parameter optimization.
The state-of-the-art shows that the $\lambda$ produces changes on the
recommendation list, affecting the system performance.

####  2.6.1. Normalized Variance

The equation calculates the distance along all values in the distribution,
each of that value is a genre/class representation and the values can be
between $[0,1]\in\mathbb{R}$. It is described as:

(9) $mp(u)=\frac{\sum_{g\in
G}p(g\text{\textbar}u)}{\text{\textbar}G\text{\textbar}},$

where the result from $mp(u)$ is an average of the values in each genre. That
value represents a center point among all genres.

(10) $\lambda_{u}=1-\frac{\sum_{g\in
G}\text{\textbar}p(g\text{\textbar}u)-mp(u)\text{\textbar}^{2}}{\text{\textbar}G\text{\textbar}},$

where $\lambda_{u}$ is an average of the summation over all genres, computing
the squared difference between a genre and the center point.

####  2.6.2. Genre count

A second equation to find the personalized trade-off weight is described as:

(11) $\lambda_{u}=\frac{\sum_{g\in G}\mathbb{1}(g\in
I(u))}{\text{\textbar}G\text{\textbar}},$

where $\mathbb{1}(g\in I(u))$ returns $1$ if the genre $g$ is in an item of
the user preference, otherwise $0$. After the summation, the average is
computed and that final value indicates the degree of calibration/fairness
with the user’s preferences.

###  2.7. Trade-Off Balance

All equations presented so far are to compose the trade-off balance. We
introduced ways to achieve the relevant and the fair portion of the trade-off
balance, which brings all parties together to generate a fair list of
recommendations. In our study, we implement the Steck’s (Steck, 2018) claims,
used by (Kaya and Bridge, 2019; da Silva et al., 2021) and propose a new
trade-off balance.

The Maximum Marginal Relevance (MMR) (Carbonell and Goldstein, 1998) is
commonly used to maximize the choice of items in recommendation. It tends to
get the optimal set of recommendation items $R^{*}$ from the set of candidate
items $CI(u)$. The state-of-the-art (Steck, 2018; Kaya and Bridge, 2019; da
Silva et al., 2021) also uses the MMR. The MMR is used as the base to the
trade-off balance and in our system $(\forall u\in U)$ is represented as:

(12) $R^{*}=\underset{\text{\textbar}L\text{\textbar}=N,L\subset
CI_{u}}{\arg\max}Trade(\lambda_{u},L,p),$

where the $Trade(\lambda_{u},L,p)$ is any trade-off balance. $L$ is a sub list
from the candidate items, that list is an intermediate list used when building
the final recommendation list $R^{*}$.

####  2.7.1. Linear

Steck (Steck, 2018) proposes a way to calibrate the recommendation list with a
trade-off between relevance and calibration. In our study, we call it linear
trade-off and is described as:

(13) $TradeLin(\lambda_{u},L,p)=(1-\lambda_{u})\cdot Sim(L)-\lambda_{u}\cdot
F(p,q(L)),$

where: $\lambda_{u}$ is any personalized trade-off weight from Section 2.6 (or
constant weight to all users), Sim(L) is from Section 2.5 and F(p,q(L)) is any
divergence/calibration measure from Section 2.4.

####  2.7.2. Logarithmic bias

We claim a new trade-off balance to calibrate the recommendation list. Our
balance considers the users’ bias as a way to improve the calibration with the
user tendencies, leading to a more specialized distribution without the super
specialization problem. It is presented as:

(14)
$TradeLog(\lambda_{u},L,p)=sign(TradeLin)\cdot\log^{abs(TradeLin)+1}+\hat{b}_{u}(L),$

where $sign(TradeLin)$ is a function that returns the signal from the value
obtained in the $TradeLin$ (Eq. 13). The $\log^{abs(TradeLin)+1}$ is a
logarithm from the $TradeLin$ absolute value (i.e. casting to positive), due
to the log proprieties do not comprehend negative values and the $+1$ is to
avoid the log from returning zero. The $\hat{b}_{u}(L)$ is presented in
sequence.

Koren and Bell (Koren and Bell, 2015) present a user and item bias equations.
The item bias is described as:

(15) $b_{i}=\frac{\sum_{u\in
W(i)}(w_{u,i}-\mu)}{\alpha+\text{\textbar}W(i)\text{\textbar}}.$

Our proposed users’ bias is based on the items with the predicted relevance
weight $w_{r(u,i)}$ in the place of relevance weight value $w_{u,i}$, this
adaptation turns possible to apply the users’ bias in the post-processing
step. Our claim on the user’s bias equation indicates how much the rating
prediction by the recommender algorithm is close to the real ratings. The
$\mu$ is an average of the relevance weights among all users’ preferences.
$\alpha$ and $\sigma$ values are to avoid division by zero and we adopt $0.01$
to both. The $\hat{b}_{u}(L)$ receives the list $L$ and finds the user bias
for that list. The formal representation of the users’ bias with the proposed
adaptation is:

(16) $\hat{b}_{u}(L)=\frac{\sum_{i\in L}(w_{r(u,i)}-\mu-
b_{i})}{\sigma+\text{\textbar}L\text{\textbar}}.$

###  2.8. Select items algorithm

The last component is the select items algorithm. Here many re-ranking or
selection algorithms can be applied. Most of related works implement the
Surrogate Submodular (Nemhauser et al., 1978), and similar to (Steck, 2018;
Kaya and Bridge, 2019; da Silva et al., 2021), we also implement it. The
Submodular algorithm workflow is:

  * •

$R^{*}$ starts with an empty list {};

  * •

for each $N-1$ interaction with the candidate items, if an item $i$ tested as
optimal in temporally list $L$ it is the one that maximizes $R^{*}$, then $i$
is appended into the list;

  * •

in the end, the list has $N-1$ items and the last optimal item is appended,
completing as well the top-n items $R^{*}$.

##  3\. Decision Protocol

The main contribution of our work is a protocol for deciding which system is
the best to be implemented driven by data domain. The decision protocol
expects as an input two or more calibrated systems and, based on the
coefficient results, it suggests the best one. These systems are combinations
of calibrated one based on the proposed framework.

###  3.1. Metrics

The protocol uses three well-known metrics: The Mean Average Precision (MAP),
a traditional metric used by the Recommendation Systems community; The Mean
Average Calibration Error (MACE), and the Mean Rank MisCalibration (MRMC),
proposed by (da Silva et al., 2021). Each calibrated system needs to be
evaluated by these metrics.

###  3.2. Coefficients

From those evaluation metrics values, we can obtain two coefficients: i) the
Coefficient of Calibration Error (CCE) and ii) the Coefficient of
MisCalibration (CMC). Both coefficients are part of our approach and
determinant for the effectiveness of the protocol.

The Coefficient of Calibration Error (CCE) can be described as:

(17) $CCE=\frac{\overline{MACE}}{\overline{MAP}}.$

The Coefficient of MisCalibration (CMC) can be described as:

(18) $CMC=\frac{\overline{MRMC}}{\overline{MAP}}.$

The proposal is a division between the MACE or MRMC (da Silva et al., 2021) by
the MAP. All metrics evaluate the results of each position in the recommended
list. In the Equations 17 and 18, the numerator is the MACE or MRMC results
average and the denominator is the MAP results average, and all results are
obtained over the same calibrated system. It’s worth mentioning that when the
MACE or MRMC is higher and the MAP is lower the obtained CCE or CMC value is
higher, this means the system has more error or miscalibration than precision;
and when the MACE or MRMC is lower and the MAP is higher the obtained CCE or
CMC value is lower, which means the system has less errors (or miscalibration)
and is more precise. In Equations 17 and 18, the variables
$\overline{MACE},\overline{MRMC}$ and $\overline{MAP}$ are representations of
each metric average, computed over each system running.

###  3.3. The decision

From the coefficients, the decision protocol indicates which system
combination is the best to be implemented. At this point we have, for each
calibrated system, two coefficient values. The protocol uses the coefficients
in an addition as:

(19) $s_{i}=CCE+CMC,$

where the system $i$ has its performance ($s_{i}$) measured with the
coefficients, representing its total of error and miscalibration over the
precision as previously debated.

The final decision is obtained by:

(20) $S=\min(s_{1},\ s_{2},\ s_{3}...\ s_{n}),$

where the $n$ calibrated systems are judged and the system with the lower
value of $s$ is chosen as the best calibrated system $S$.

In the next sections we will describe the experimental setup and results. Our
research implements more than one thousand calibrated systems (considering the
parameter optimization) and at the end suggests the best implementation based
on the proposed decision protocol.

##  4\. Related Work

In recent years, the calibrated recommendation has attracted attention as a
means of achieving fairness. This topic of research addresses different
points-of-view, for instance, some state-of-the-art analyses the context of
calibration (Abdollahpouri et al., 2020; Lin et al., 2020), others focused on
creating a calibrated recommendation list in a user-centric view (Steck, 2018;
Kaya and Bridge, 2019; Abdollahpouri et al., 2021; da Silva et al., 2021) and
others seek to provide more fairness in an item-centric view (Zhao et al.,
2020). Our research focuses on the user-centered view, i.e., a C-fairness
system (da Silva et al., 2021). In a sequence, we discuss several related
works that generate calibrated recommendation list by the post-processing.

Steck (Steck, 2018) proposes the calibrated recommendations as a way to
provide fairness, in which the recommendation list is generated to be the
least divergent with the movie genres in the user’s profile. The author
implements the collaborative filtering technique through a matrix
factorization recommendation algorithm. This algorithm provides a list of
recommendations that is post-processed to ensure a certain degree of fairness.
Steck implements the calibrated system as a unique system divide in three
steps. The results show the importance of calibrating recommendations where
the objective is to obtain a certain degree of fairness, but the accuracy is
slightly reduced. Inspired by this work (Steck, 2018), we implement the linear
trade-off, the relevant sum, the distribution, the surrogate submodular and
the Kullback-Leibler. So Steck’s work is covered by our findings. Furthermore,
the author suggests that the Hellinger divergence measure may be implemented
as fairness measures, consequently we follow the Steck suggestion. The work
uses only movie genres and doesn’t cover other domains as music or books.

In a similar fashion with Steck (Steck, 2018) and expanding the study about
calibrated recommendations, Kaya and Bridge (Kaya and Bridge, 2019)
investigate if calibrated recommendations can improve diversification in the
recommendation list. Beyond the calibrated systems, the authors implement the
Intent-aware approach, seeking to compare the results. As the authors explain,
the Intent-aware is an approach originally from the Information Retrieval (IR)
and adapted to the Recommendation System. The approach works with ambiguous
representation, for instance, the requisition for “Jaguar”, the system can
have multiple representations for the requisition as cats, cars or operating
systems. By that the approach seeks to include the greatest number of
representations as possible in the list. Thus, the study (Kaya and Bridge,
2019) implements two Intent-aware adaptations to recommendation systems the
Query Aspect Diversification (xQUAD) and Subprofile-Aware Diversification -
(SPAD). The authors compare these two Intent-aware forms with the Steck’s
(Steck, 2018) calibrated system and extend the system adding a new way to find
the user profile distribution, as well as comparing four systems. The authors
evaluate the results in three perspectives: precision, diversity and
miscalibration. As part of methodology the authors use: the Movielens 20M and
the Taste Profile as datasets, different trade-off weight values from (Steck,
2018) and a fast alternative least-squares matrix factorization recommender.
The results indicate a relation that intent-aware calibrates the
recommendation list to some degree and calibrated recommendations somehow
imply a diverse recommendation. To the Taste Profile dataset, the calibrated
system with the distribution based on subprofile increases the precision when
the trade-off weight is high. As Steck (Steck, 2018), Kaya and Bridge (Kaya
and Bridge, 2019) results indicate that the higher the trade-off value, the
more calibrated the list will be. From (Kaya and Bridge, 2019), we cover all
implementations inspired by (Steck, 2018). We do not cover diversification in
our study.

Abdollahpouri et al. (Abdollahpouri et al., 2020) study a connection between
popularity bias and calibration. The short paper is focused on analyzing the
calibration aspect and its fairness. It is argued that the groups which
receive popularity bias amplification will receive a higher miscalibration
degree too. Beyond, it is discussed that if the popularity bias is fixed it is
possible that the disparate miscalibration can be fixed too, turning the
system fairer. To create the groups, the authors divide all users in 10 groups
based on the degree of popular items each user likes. They do not implement a
post-processing step; the connection is between the user’s preferences set and
the recommendations provided by five chosen recommender algorithms.
Abdollahpouri et al. (Abdollahpouri et al., 2020) use a different genre
distribution formulation without probability and rank weight, as proposed in
previous calibrated work. As a dataset they use Movielens 1M and core-10 Yahoo
Movie and evaluate with miscalibration metric proposed by (Steck, 2018) and
algorithmic popularity lift proposed by the authors in previous work. The
results indicate that the users with less popular items are affected by the
popularity bias and receive more miscalibrated recommendations.

Lin et al. (Lin et al., 2020) analyses several categories of profile
characteristics over many factors that contribute to miscalibration for some
users but not others. These characteristics are provided by the users profiles
and can be interpreted from the users’ preference. The category described in
(Lin et al., 2020) is a generic word to class, tag or a genre as in our work.
To evaluate the categories authors use the miscalibration proposed by (Steck,
2018) as well as the normalized Discounted Cumulative Gain (NDCG). As datasets
they use the Movielens 1M and Yelp.com. As (Abdollahpouri et al., 2020), (Lin
et al., 2020) do not implement the post-processing step and use five well-
known algorithms. The results show that for all algorithms and for both
datasets the miscalibration is present in different degrees, where they
analyze the characteristics depending on it the miscalibration present at
different degrees too.

Seymen et al. (Seymen et al., 2021) propose a new trade-off balance called
Calib-Opt. As calibration measure, the authors use the proposed Weighted Total
Variation. In the rank measure they consider two news constraints. The
selector algorithm used by (Steck, 2018) called surrogate submodular is
substituted by Gurobi branch&bound. The results indicate that the proposed
Calib-Opt obtained a better performance than (Steck, 2018).

Silva et al. (da Silva et al., 2021) propose a set of calibrated
recommendations systems and metrics. The authors use the divergence measure
Kullback-labler similar to (Steck, 2018; Kaya and Bridge, 2019) and propose
the Hellinger and Pearson Chi Square. To evaluate the systems (da Silva et
al., 2021) propose two new metrics to the calibrated context, the Mean Average
Calibration Error (MACE) and the Mean Rank MisCalibration (MRMC). To evaluate
the authors use six well-known collaborative filtering recommender algorithms.
The results indicate that the Pearson Chi Square can obtain a better
performance than other divergence measures.

Pitoura et al. (Pitoura et al., 2021) conduct an overview about fairness in
ranking and recommendations. The authors argue that calibrated recommendations
are one model of fairness, which has six in total: demographic parity,
conditional parity, equalized odds, fairness through awareness, counterfactual
fairness and calibration-based fairness. In the overview, the authors define
five viewpoints on fairness: fairness for the recommended items, fairness for
the users, fairness for groups of users, fairness for the item providers and
the recommendation platform. The study considers that the post-processing is
the core for fairness systems, due to be the step that ensures the fairness,
modifying the recommender algorithm output.

##  5\. Experimental Setup

In this section, we illustrate the experimental setup applied in our study. We
describe the used datasets and methodology. Our setup is modeled to provide a
better understanding of the system behavior. The datasets and methodology are
inspired by related works in the state of art (Steck, 2018; Kaya and Bridge,
2019; da Silva et al., 2021).

###  5.1. Datasets

Inspired by (Kaya and Bridge, 2019), we use two public datasets for evaluating
the proposal. Table 6 shows a data description from both datasets.

Table 6. Datasets before and after the pre-processing. Datasets | —U— | —I— | —W— | —G—
---|---|---|---|---
Raw ML20M | 138,493 | 27,278 | 20,000,263 | 19
Used ML20M | 80,672 | 15,400 | 9,013,655 | 19
Raw Taste Profile | 1,019,318 | 999,056 | 48,373,586 | 16
Used Taste Profile | 94,611 | 98,305 | 4,807,615 | 16

For the movie domain, we use the MovieLens 20M dataset
(ML20M)111https://www.kaggle.com/grouplens/movielens-20m-dataset(Harper and
Konstan, 2015), which is the same dataset used by (Steck, 2018; Kaya and
Bridge, 2019; da Silva et al., 2021). Table 6 show the numbers of users (—U—),
items (—I—), feedbacks (—W—) and genres (—G—) are presented. As defined in the
proposed framework the pre-processing applies cleaning, filtering and modeling
to the data. Based on that and inspired by (Steck, 2018; Kaya and Bridge,
2019; da Silva et al., 2021), we apply a rating cut by 4 and dropping all
lower ratings, the removal of movies without genre information and the removal
of movies without user interactions. Unlike (Steck, 2018; Kaya and Bridge,
2019; da Silva et al., 2021), we dropped users with a preference set size
smaller than 30 items. In addition, we dropped items with less than 3
interactions with users to avoid the item cold-start.

For the song domain, we use the Taste
Profile222http://millionsongdataset.com/tasteprofile/ dataset and the Tagtraum
genre annotations333http://www.tagtraum.com/genres/msd_tagtraum_cd2.cls.zip,
which is the same dataset used by (Kaya and Bridge, 2019), though is not the
same genre annotation. Table 6 shows the numbers of users, items,
feedbacks/transactions and genres are presented. For obtaining pre-processed
data as indicated in the framework and similarly to (Kaya and Bridge, 2019) we
applied: the elimination of songs without genre information. Differently from
(Kaya and Bridge, 2019), we dropped users with a preference set size smaller
than 30 items, whose value was defined is defined based on the size of the
recommendation list and a cut by the most played songs where it discards the
ones played less than 3 times. In addition, we dropped items with less than 3
interactions with users to avoid the item cold-start.

###  5.2. Methodology

With the pre-processed data we ran the Grid Search method. Each recommender
algorithm used in our experimentation searches the best hyper parameters with
a 3-fold cross validation methodology. In order to choose the best hyper
parameters (HP), we use the Mean Average Error (MAE) as a decision metric,
i.e., the HP combination with the smallest error is chosen. The MAE measures
the absolute error between the real and predicted weight, thus the smaller the
MAE error, the smaller the error propagation for post-processing. To better
understand the effects of the post-processing and test our decision protocol,
as well as our framework, we select seven well-known recommender algorithms as
shown in the processing step. They are:

  1. (1)

Basic UserKNN: User-based K Nearest Neighbors (Koren, 2010). The best HP is:
$k=30$ as the number of neighbors and Mean Squared Difference as similarity
for both datasets;

  2. (2)

Basic ItemKNN: Item-based K Nearest Neighbors (Koren, 2010). The best HP is:
$k=30$ as the number of neighbors and Person Correlation as similarity for
both datasets;

  3. (3)

Slope One: (Lemire and Maclachlan, 2007) presents the algorithm which has no
HP;

  4. (4)

NMF: Non-negative Matrix Factorization presented by (Luo et al., 2014). The
best HP are: i) for ML20M the number of epochs $ne=50$, number of latent
factors $f=50$, user and item learning rate $\gamma_{u}=0.005$ and
$\gamma_{i}=0.005$, user and item regularization constant $\lambda_{u}=0.005$
and $\lambda_{i}=0.05$, and user and item regularization bias
$\lambda_{bu}=0.005$ and $\lambda_{bi}=0.005$; ii) for Taste Profile the
number of epochs $ne=30$, number of latent factors $f=50$, user and item
learning rate $\gamma_{u}=0.003$ and $\gamma_{i}=0.005$, user and item
regularization constant $\lambda_{u}=0.003$ and $\lambda_{i}=0.05$, and user
and item regularization bias $\lambda_{bu}=0.003$ and $\lambda_{bi}=0.005$;

  5. (5)

SVD: Singular Value Decomposition proposed by (Koren et al., 2009). The best
HP are: i) for ML20M the number of epochs $ne=50$, number of latent factors
$f=50$, user and item learning rate $\gamma_{u}=0.005$ and $\gamma_{i}=0.005$,
and user and item regularization $\lambda_{u}=0.01$ and $\lambda_{i}=0.01$;
ii) for Taste Profile the number of epochs $ne=30$, number of latent factors
$f=150$, user and item learning rate $\gamma_{u}=0.001$ and
$\gamma_{i}=0.001$, and user and item regularization $\lambda_{u}=0.05$ and
$\lambda_{i}=0.05$;

  6. (6)

SVD++: An extension of SVD presented by (Koren, 2008). The best HP are: for
ML20M and Taste Profile the number of epochs $ne=50$, number of latent factors
$f=50$, user and item learning rate $\gamma_{u}=0.005$ and $\gamma_{i}=0.005$,
and user and item regularization constant $\lambda_{u}=0.01$ and
$\lambda_{i}=0.01$;

  7. (7)

Co Clustering: Presented by (George and Merugu, 2005), the Co Clustering is a
recommender based on clusterization. The best HP is: i) for ML20M the number
of epochs $ne=50$, number of user and item clusters $\overline{C_{u}}=3$ and
$\overline{C_{i}}=7$; ii) for Taste Profile the number of epochs $ne=10$,
number of user and item clusters $\overline{C_{u}}=3$ and
$\overline{C_{i}}=3$.

This study does not propose any change in the recommender algorithms workflow.
Therefore, we use the Python Surprise library (Hug, 2020) implementation to
perform our experiments. The selected collaborative filtering recommenders
used in the experiment are presented in the literature by (Abdollahpouri et
al., 2020; Lin et al., 2020; da Silva et al., 2021).

The datasets used in the experiments are randomly divided for each user in 70%
of training data and 30% of test data. This methodology of validation is use
by the-state-of-the-art (Steck, 2018; Kaya and Bridge, 2019; da Silva et al.,
2021). All test data is mixed with all unknown items by the user and it is
expected the test items to compose the final recommendation list. The training
data are given as an input to the recommender in order to learn about the
users’ preference. Having the recommender totally trained, the algorithm
predicts the rating to all unknown+test items for each user. After predicting
all possible ratings to the users-item matrix, similarly to (Steck, 2018; Kaya
and Bridge, 2019; da Silva et al., 2021), we get the top 100 candidate items
with the highest rating predicted by the recommender algorithm. The post-
processing to each user starts with the top-100 candidate items, from which
the system will select the final recommendation list.

In addition to the two personalized trade-off weights, we use constant values
of $\lambda$. These same weighing values are also used by (Kaya and Bridge,
2019; da Silva et al., 2021) and they are $\lambda\in[0.0;0.1;...;0.9;1.0]$.

All the system combinations are generated taking into account two datasets,
seven recommender algorithms, two trade-off balances, two personalized weights
(VAR, CGR) plus eleven constant values (totaling 13 trade-off weights), and
the three divergence measures. Each configuration combines a different way to
generate the final recommendation list, so in total we have $2\cdot 7\cdot
2\cdot 13\cdot 3=1092$ different combinations to be evaluated by each metric.
We run each of that combinations 3 times with different training and test
data.

The final recommendation list $R^{*}$ is generated with the top-10 fair items
given by the post-processing. Finally, we evaluate the recommendation list in
all $[1;10]\in\mathbb{N}$ positions.

##  6\. Experimental Results

In this section we present the experimental results based on the setup
described previously. In particular, we present the cross-metric between MAP
and MACE as well as CCE (Subsection 6.1); the crossed results between MAP and
MRMC, plus CMC (Subsection 6.2); and the protocol decision (Subsection 6.3).
Section 6.4 answers all the research questions presented in the paper
introduction. Section 6.5 discuss the overall findings.

###  6.1. MAP x MACE

First we will analyze the results obtained from the MAP and MACE metrics. To
do that, we combine the results of each system combination, where each dot in
the lines is a $\lambda$. Figure 2 presents the cross-metric MAPxMACE for the
Movielens, as well as Table 7 presents the CCE values. Figure 3 presents the
cross-metric for the Taste Profile, whereas Table 8 presents the CCE values.

####  6.1.1. Movielens

Table 7. Movielens - CCE - Results from $LIN$ and $LOG$ trade-offs combined
with the KL, HE and CHI divergence measure for each recommender algorithm.

Divergence | Trade-off | Co Clustering | Item KNN | NMF | Slope One | SVD | SVD++ | User KNN
---|---|---|---|---|---|---|---|---
KL | $LIN$ | 60.44 | 90.73 | 3.8 | 63.52 | 4.95 |  3.52 | 430.91
KL | $LOG$ | 62.94 | 110.46 | 5.76 | 83.46 | 4.31 |  3.05 | 130.58
HE | $LIN$ | 34.7 | 57.4 | 2.51 | 40.28 | 3.68 |  2.38 | 711.24
HE | $LOG$ | 57.84 | 157.32 | 8.31 | 104.41 | 3.06 |  2.08 | 60.73
CHI | $LIN$ | 38.26 | 63.33 |  2.92 | 47.41 | 4.29 | 3.06 | 364.44
CHI | $LOG$ | 50.63 | 88.66 | 4.83 | 71.43 | 4.3 |  3.06 | 120.71

(a) KL and $LIN$

(b) KL and $LOG$

(c) Hellinger and $LIN$

(d) Hellinger and $LOG$

(e) $\chi^{2}$ and $LIN$

(f) $\chi^{2}$ and $LOG$

Figure 2. Movielens - MAP (x) and MACE (y) - Results from $LIN$ and $LOG$
trade-offs combined with the KL-divergence, Hellinger and Pearson Chi Square.

We start the dataset analysis with the system combinations that implement the
KL-divergence. Figures 2(a) and 2(b) and Table 7 (Lines 1 and 2) show that the
best trade-off balance is the $LOG$. KL-LOG-SVD++ combination achieves the
best performance. This find can be seen in Figure 2(b). The second and third
places are the other matrix factorizations. As to the $LOG$ trade-off (Figure
2(b)), the second and third places are achieved sequentially by SVD and NMF
and to $LIN$ trade-off (Figure 2(a)) the recommenders change the positions to
NMF and SVD sequentially. As indicated by the Table 7 (Lines 1 and 2) the best
performances, in the KL-divergence, are all matrix factorization approaches,
specially to the SVD++ with $CCE=3.05$ ($LOG$) and $CCE=3.52$ ($LIN$). The
worst results are held by the KNNs approaches that consider the KL-divergence
as a fixed divergence measure.

The results achieved by the Hellinger combinations are shown in Figure 2(c),
in Figure 2(d) and in Table 7 (Lines 3 and 4). The results indicate that the
$LOG$ trade-off (Figure 2(d)), similar to the KL analysis, achieves the best
performance and such behavior remain in all matrix factorization approaches.
It is possible to verify that the worst performance remains with the KNNs
approaches. The Hellinger achieves lower MACE values than the KL-divergence,
indicating that the HE produces a recommended list with a low degree of
calibration error. The results on Table 7 (Lines 3 and 4) show that the SVD++,
in the Hellinger, achieved the $CCE=2.08$ to $LOG$ and $CCE=2.38$ to $LIN$.

The last MAPxMACE results that we will analyze are the Pearson Chi Square
($\chi^{2}$) combinations. Figures 2(e) and 2(f) with Table 7 (Lines 5 and 6)
show that the $\chi^{2}$ follows the same behavior found in the two previous
measures analyzed. The best performances are all matrix factorization
approaches, as the KNNs obtain the worst performance. The best combination
performance, in the $\chi^{2}$ is obtained by the CHI-LIN-NMF with a
$CCE=2.92$ (Table 7 Lines 5 and 6), followed by the SVD++ in second place and
the SVD in third place. In the $LOG$ trade-off the places follow the sequence
SVD++, SVD and NMF.

The results indicate that if the system is focused on MAP and MACE and
according to our experimental setup, that the best combination to be
implemented in the Movielens dataset is the HE-LOG-SVD++ with a $CCE=2.08$ as
presented in Table 7 Line 4. We note that the $LOG$ trade-off is a part of the
proposal of this research. The second best performance based on the CCE
belongs to the HE-LIN-SVD++ with $CCE=2.38$ and the third place belongs to HE-
LIN-NMF with $CCE=2.51$ (Table 7 Line 3). We also can observe that the
Hellinger divergence measure achieves all best performances on the CCE. We can
also observe that there is no correlation between the MAP and MACE.

####  6.1.2. Taste Profile

Table 8. Taste Profile - CCE - Results from $LIN$ and $LOG$ trade-offs
combined with the KL, HE and CHI divergence measure for each recommender
algorithm.

Divergence | KL | Co Clustering | Item KNN | NMF | Slope One | SVD | SVD++ | User KNN
---|---|---|---|---|---|---|---|---
KL | $LIN$ | 5093.61 |  25.73 | 667.67 | 1471.03 | 489.45 | 329.39 | 1175.62
KL | $LOG$ | 5199.86 |  26.68 | 2307.22 | 1714.39 | 721.64 | 559.37 | 1232.02
HE | $LIN$ | 5072.75 |  14.39 | 459.35 | 1406.45 | 303.95 | 278.98 | 1148.17
HE | $LOG$ | 5057.98 |  21.32 | 2336.09 | 1709.8 | 709.28 | 552.41 | 1234.66
CHI | $LIN$ | 4308.4 |  23.29 | 469.9 | 1438.43 | 308.4 | 283.38 | 1047.84
CHI | $LOG$ | 4442.51 |  27.64 | 2287.83 | 1712.22 | 659.49 | 531.97 | 1223.63

(a) KL and $LIN$

(b) KL and $LOG$

(c) Hellinger and $LIN$

(d) Hellinger and $LOG$

(e) $\chi^{2}$ and $LIN$

(f) $\chi^{2}$ and $LOG$

Figure 3. Taste Profile - MAP (x) and MACE (y) - Results from $LIN$ and $LOG$
trade-offs combined with the KL-divergence, Hellinger and Pearson Chi Square.

We will start our results analysis in the Taste Profile by the KL-divergence.
Figures 3(a) and 3(b) and Table 8 (Lines 1 and 2) show that the best
recommender performance, in the KL-divergence, belongs to the Item-KNN both in
$LIN$ and $LOG$. It is possible to verify that the Item-KNN results in the
$LIN$ have more variability than in the $LOG$ trade-off. The CCE obtained by
the combination KL-LIN-Item-KNN is $25.73$ against $26.68$ from KL-LOG-Item-
KNN (Table 8 Lines 1 and 2). The second and third place are the SVD++ and SVD,
both matrix factorization approaches. The worst performance belongs to Co
Clustering.

Figures 3(c) and 3(d) support our analysis of the divergence measure Hellinger
in the Taste Profile, besides Table 8 in Lines 3 and 4 presents the CCE
performances. Similar to KL-divergence on Taste Profile, the Item-KNN obtains
the best performances in the $LIN$ with $CCE=14.39$ and $LOG$ with
$CCE=21.32$, as well as the $LOG$ trade-off has less variability in the
results when compared with the $LIN$. It is possible to verify, through Table
8 Lines 3 and 4, that the second and third place are the SVD++ and SVD, both
matrix factorization approaches. The worst performance remains with the Co
Clustering approach.

The last results to the Taste Profile are the Pearson Chi Square ($\chi^{2}$)
combinations. Figures 3(e) and 3(f) combined with Table 8 Lines 5 and 6
present the results. The $\chi^{2}$ follows the same behavior found in the two
previous measures analyzed in the dataset. The best performance belongs to
Item-KNN. The $LOG$ trade-off has less variability in the performance and the
$LIN$ obtained the best performance in the trade-off balances. The CHI-LIN-
Item-KNN obtained the $CCE=23.29$ and the CHI-LOG-Item-KNN obtained
$CCE=27.64$. The SVD++ obtained the second place and the SVD the third place
in both trade-off balances.

From the CCE results in the Taste Profile, we can observe that if the system
is focused on MAP and MACE, the best performance refers to the combination HE-
LIN-Item-KNN with a $CCE=14.39$, followed by the combination HE-LOG-Item-KNN
with a $CCE=21.32$. These results can be verified in Table 8 at lines 3 and 4.
We remind that results are restricted to the CCE (MAP and MACE metric
crossed).

Table 9. Movielens - CMC - Results from $LIN$ and $LOG$ trade-offs combined
with the KL, HE and CHI divergence measure for each recommender algorithm.

Divergence | Trade-off | Co Clustering | Item KNN | NMF | Slope One | SVD | SVD++ | User KNN
---|---|---|---|---|---|---|---|---
KL | $LIN$ | 60.44 | 90.73 | 3.8 | 63.52 | 4.95 |  3.52 | 430.91
KL | $LOG$ | 62.94 | 110.46 | 5.76 | 83.46 | 4.31 |  3.05 | 130.58
HE | $LIN$ | 298.94 | 454.46 |  20.22 | 278.07 | 31.78 | 20.56 | 5319.06
HE | $LOG$ | 576.33 | 1654.34 | 83.42 | 982.12 | 31.01 |  20.95 | 616.98
CHI | $LIN$ | 156.53 | 228.18 | 10.52 | 136.21 | 13.21 |  9.39 | 1367.03
CHI | $LOG$ | 184.39 | 294.33 | 15.84 | 181.31 | 12.8 |  9.08 | 391.45

(a) KL and $LIN$

(b) KL and $LOG$

(c) Hellinger and $LIN$

(d) Hellinger and $LOG$

(e) $\chi^{2}$ and $LIN$

(f) $\chi^{2}$ and $LOG$

Figure 4. Movielens - MAP (x) and MRMC (y) - Results from $LIN$ and $LOG$
trade-offs combined with the KL-divergence, Hellinger and Pearson Chi Square.

###  6.2. MAP x MRMC

Another crossed-metric that we will analyze is the results from MAP and MRMC.
Similar to the MAPxMACE, we want to understand the behavior of each system
combination. Figure 4 presents the results from the Movielens, as well as
Table 9 presents the CMC. Figure 5 presents the results from the Taste
Profile, as well as Table 10 presents the CMC values.

####  6.2.1. Movielens

Similar to the MAPxMACE, we will start our analysis by the KL-divergence
system combinations. Figures 4(a) and 4(b) along with Table 9 (Lines 1 and 2)
show the results from MAP and MRMC. It is possible to verify that the matrix
factorization approaches obtained the best performance in the measure (KL). In
Table 9 (Lines 1 and 2), we can observe that the KL-LOG-SVD++ obtained the
best performance with a $CMC=3.05$. In the $LOG$ trade-off the second and
third place is the SVD ($CMC=4.31$) and NMF ($CMC=5.76$) sequentially. In the
$LIN$, the SVD++ obtained the best performance, whereas the combination KL-
LIN-SVD++ ($CMC=3.52$) obtained the second best performance when we observe
all the KL-divergence combinations. As we can verify that the MAPxMRMC and the
MAPxMACE obtain the same recommender places to the KL-divergence.

For the Hellinger divergence measure, we will analyze Figures 4(c) and 4(d)
along with Table 9 (Lines 3 and 4). Similar to the KL-divergence, the matrix
factorization approaches obtained all the best performances. However, the HE-
LIN-NMF obtained the best performance to the measure with a $CMC=20.22$,
followed by the HE-LIN-SVD++ with a $CMC=20.56$. To the $LOG$ trade-off, the
best performance is HE-LOG-SVD++ with a $CMC=20.95$.

As the last analysis we will observe the Pearson Chi Square ($\chi^{2}$)
combinations take into account Figures 4(e) and 4(f) and Table 9 (Lines 5 and
6). Similar to the MAPxMACE results, as well as the previously presented
measures, the matrix factorization approaches obtained all best performances
in the $\chi^{2}$. However, in this measure the first place belongs to the
SVD++ in the $LOG$ with a $CMC=9.08$ and in the $LIN$ with a $CMC=9.39$.

Based on the $CMC$ results we can observe that if the system is focused on MAP
and MRMC and according to our experimental setup, the best combination to be
implemented in the Movielens dataset is the KL-LOG-SVD++ with a $CMC=3.05$ as
presented in the Table 9 Line 2. The second best performance belongs to KL-
LIN-SVD++. When we observe the Figure 4, it is possible to evidence that the
metrics are not directly correlated. We note that the $LOG$ trade-off is part
of this research proposes.

Table 10. Taste Profile - CMC - Results from $LIN$ and $LOG$ trade-offs
combined with the KL, HE and CHI divergence measure for each recommender
algorithm.

Divergence | Trade-off | Co Clustering | Item KNN | NMF | Slope One | SVD | SVD++ | User KNN
---|---|---|---|---|---|---|---|---
KL | $LIN$ | 25124.25 |  76.12 | 2299.61 | 6598.41 | 953.96 | 624.99 | 5331.51
KL | $LOG$ | 22937.04 |  115.76 | 11416.07 | 8539.23 | 3251.54 | 2971.04 | 6144.5
HE | $LIN$ | 42028.72 |  119.23 | 3775.94 | 11175.3 | 1635.21 | 1452.92 | 9181.55
HE | $LOG$ | 42531.68 |  239.19 | 19002.44 | 13655.52 | 5243.78 | 4729.24 | 9891.66
CHI | $LIN$ | 12640.08 |  68.46 | 1987.01 | 3937.01 | 798.85 | 687.41 | 2939.81
CHI | $LOG$ | 11985.59 |  86.13 | 11219.71 | 8244.53 | 2818.63 | 2696.87 | 5937.93

(a) KL and $LIN$

(b) KL and $LOG$

(c) Hellinger and $LIN$

(d) Hellinger and $LOG$

(e) $\chi^{2}$ and $LIN$

(f) $\chi^{2}$ and $LOG$

Figure 5. Taste Profile - MAP (x) and MRMC (y) - Results from $LIN$ and $LOG$
trade-offs combined with the KL-divergence, Hellinger and Pearson Chi Square.

####  6.2.2. Taste Profile

Similar to the Movielens analysis, in the Taste Profile we will start our
results by the KL-divergence. Figures 5(a) and 5(b) and Table 10 (Lines 1 and
2) show that the best recommender performance, in the KL-divergence, belongs
to the Item-KNN both in $LIN$ with a $CMC=76.12$ and $LOG$ with a $CMC=115.76$
respectively. The second and third best recommender performances are the
matrix factorization SVD++ and SVD. The Item-KNN system combinations obtains
the best performance with a large difference to the second and third place.
The worst performance in the KL belongs to the recommender Co Clustering.

For the Hellinger measure in the Taste Profile, Figures 5(c) and 5(d) and
Table 10 (Lines 3 and 4) present the results to the measure. Similar to the
KL-divergence, the best performance of the Hellinger is the Item-KNN with the
$LIN$ trade-off, as well as the second and third places are the matrix
factorization approaches. The Co Clustering approach achieves with the worst
performance.

As to the Pearson Chi Square presented in Figures 5(e) and 5(f) and Table 10
(lines 5 and 6), similar to the previous measures and to the MAPxMACE results
of the Taste Profile, the Item-KNN obtained the best performance to the
$\chi^{2}$. The achieved values to the $LIN$ trade-off by the $CMC$ is $68.46$
and to the $LOG$ trade-off is $CMC=86.13$. The second and third place are the
matrix factorizations, as well as the worst performance remains with the Co
Clustering approach.

Based on the results we can observe that the CHI-LIN-Item-KNN is the best
combination to be implemented with the Taste Profile. This analysis is based
on the $CMC=68.46$. When we observe the table, it is possible to verify that
the second best performance belongs to the KL-LIN-Item-KNN combination with
$CMC=76.12$, followed by the combination CHI-LOG-Item-KNN.

###  6.3. The Decision Protocol

The results presented so far help understanding the performance of each system
combination. Nevertheless, only the CCE and CMC do not indicate the best
combination to be implemented to a given domain. Based on this, we will apply
the decision protocol to chose the best combination.

Table 11. Protocol performance values to each system combination using the
Movielens dataset.

Divergence | Trade-off | Co Clustering | Item KNN | NMF | Slope One | SVD | SVD++ | User KNN
---|---|---|---|---|---|---|---|---
KL | LIN | 244.1 | 365.4 | 16.7 | 224.86 | 19.83 | 14.06 | 1682.22
KL | LOG | 268.94 | 484.4 | 26.8 | 321.3 | 18.65 | 13.14 | 575
HE | LIN | 333.64 | 511.86 | 22.73 | 318.35 | 35.46 | 22.94 | 6030.3
HE | LOG | 634.17 | 1811.66 | 91.73 | 1086.53 | 34.07 | 23.03 | 677.71
CHI | LIN | 194.79 | 291.51 | 13.44 | 183.62 | 17.5 |  12.45 | 1731.47
CHI | LOG | 235.02 | 382.99 | 20.67 | 252.74 | 17.1 |  12.14 | 512.16

####  6.3.1. Movielens

Table 11 presents the systems combinations performance values to the Movielens
dataset. Based on the performance we can observe that the best combination is
CHI-LOG-SVD++ ($s=12.14$). The Pearson Chi Square divergence obtained the best
results due to its moderate performance in the CCE and in some results the
best performance in the CMC. The $LOG$ trade-off balance, a part of our
proposal, obtained the best performance in many results in the CCE and CMC
evaluation. The SVD++ remains in the first position in the majority of the
results of the Movielens. The Table 11 presents in bold the best performance
to each recommender. The CHI-LIN-SVD++ achieves the second best performance
when all performances are compared. The worst performance are the KNNs.

Table 12. Protocol performance values to each system combination using the
Taste Profile dataset.

Divergence | Trade-off | Co Clustering | Item KNN | NMF | Slope One | SVD | SVD++ | User KNN
---|---|---|---|---|---|---|---|---
KL | LIN | 30217.86 | 101.85 | 2967.28 | 8069.44 | 1443.41 | 954.38 | 6507.13
KL | LOG | 28136.9 | 142.44 | 13723.29 | 10253.62 | 3973.18 | 3530.41 | 7376.52
HE | LIN | 47101.47 | 133.62 | 4235.29 | 12581.75 | 1939.16 | 1731.9 | 10329.72
HE | LOG | 47589.66 | 260.51 | 21338.53 | 15365.32 | 5953.06 | 5281.65 | 11126.32
CHI | LIN | 16948.48 |  91.75 | 2456.91 | 5375.44 | 1107.25 | 970.79 | 3987.65
CHI | LOG | 16428.1 | 113.77 | 13507.54 | 9956.75 | 3478.12 | 3228.84 | 7161.56

####  6.3.2. Taste Profile

Table 12 presents the systems’ combinations performance values to the Taste
Profile dataset. According to the results, we observe that the best
combination is CHI-LIN-Item-KNN ($s=91.75$). The Pearson Chi Square divergence
achieved the best results due to its best performance in the CCE and in the
CMC. The $LIN$ trade-off balance, original (Steck, 2018) proposals, obtained
the best performance in all results (first and second place). The Item-KNN
remains in the first position in all results of the Taste Profile. Similar to
the Movielens tests, the Table 12 presents in bold the best performance to
each recommender and underlined the best performance. The worst performance
belongs to the Co Clustering approach.

###  6.4. Answering research questions

In this section we debate each research question presented in Section 1.

RQ1: How to find the best calibrated system to each domain? As a research
proposition we present a decision protocol, which helps us finding the best
calibrated system to each used dataset.

RQ2: Do the calibrated systems have the same performance on the datasets? Are
there differences among the systems behavior? Among the results we can observe
that each calibrated system has a different behavior and performance on the
datasets, i. e., there are no best calibrated systems through the datasets,
each dataset has its system that performs better. This affirmation enhances
the necessity of the decision protocol, due to each dataset presenting
different results.

RQ3: Is it possible to create a framework model for calibrated systems? What
are the important components to these systems? As we see along this research
the proposed framework can be reproducible and provides an N-calibrated
system. As well, the system components can be changed to search for the best
performance on the domain dataset.

RQ4: When the user’s bias is taken into account in the calibration trade-off,
is it possible to improve the system performance? As the Movielens and Taste
Profile results show when we consider the users’ bias in the formulation the
performance is increased. This behavior is more observable in the Movielens
results.

###  6.5. Discussion

The results show that the matrix factorization approaches are prominent
recommenders when used with the calibration methods. In the Taste Profile, the
Item-KNN obtains the best performance followed by the matrix factorization. As
we can see in the results in some results the calibration method increases the
precision as well as reduces the miscalibration.

The $LOG$ trade-off approach shows that it is possible to consider the users’
bias in the equation and increase the overall performance. The framework can
be easily applied and creates thousands of systems combinations. The right
combination to be implemented can be chosen by the proposed protocol, thus
facilitating the exploitation of optimal performances.

The claim made by (Steck, 2018; Kaya and Bridge, 2019) about the drop of
precision can be partially evidenced from our results. Other colleagues claim
the same, in which the Pearson achieves competitive results such as (da Silva
et al., 2021). Likewise (da Silva et al., 2021), our findings point towards
the possibility to implement and test new divergence measures in the
calibrated recommendations.

##  7\. Conclusion and Future Work

In this research, we exploited the calibrated recommendations in a user-
centric view. We show that the state-of-the-art does not consider the users’
bias in the trade-off balance. Based on this premise, we propose the $LOG$
trade-off, showing that it is possible to increase the MAP, MACE and MRMC
performance. We also showed that the state-of-the-art explores different
numbers of implementations without an automatic way to decide the best
implementation. In order to tackle this problem, we propose two coefficients
and a decision protocol that indicates the best implementation to the dataset.
In addition, we showed the possibilities of implementing the calibrated system
following the framework division in components to provide an easier way to be
implemented and tested.

As a future work, we plan to investigate other calibration measures and
compare the effect of the post-processing in the collaborative filtering and
content-based filtering. Another extension is to explore different types of
distribution, for instance, compare the normalized against the non-normalized
ones. In addition, we will investigate other relevance measures.

###### Acknowledgements.

This study was financed by the Coordenação de Aperfeiçoamento de Pessoal de
Nível Superior - Brasil (CAPES) - Finance Code 88887.502736 / 2020-00. The
authors acknowledge the National Laboratory for Scientific Computing
(LNCC/MCTI, Brazil) for providing HPC resources of the SDumont supercomputer,
which has contributed to the research results reported within this paper. URL:
http://sdumont.lncc.br

## References

  * (1)
  * Abdollahpouri et al. (2017) Himan Abdollahpouri, Robin Burke, and Bamshad Mobasher. 2017\.  Controlling Popularity Bias in Learning-to-Rank Recommendation. In _Proceedings of the Eleventh ACM Conference on Recommender Systems_ (Como, Italy) _(RecSys ’17)_. ACM, New York, NY, USA, 42–46.  https://doi.org/10.1145/3109859.3109912
  * Abdollahpouri et al. (2020) Himan Abdollahpouri, Masoud Mansoury, Robin Burke, and Bamshad Mobasher. 2020.  The Connection Between Popularity Bias, Calibration, and Fairness in Recommendation. In _Fourteenth ACM Conference on Recommender Systems_ (Virtual Event, Brazil) _(RecSys ’20)_. Association for Computing Machinery, New York, NY, USA, 726–731.  https://doi.org/10.1145/3383313.3418487
  * Abdollahpouri et al. (2021) Himan Abdollahpouri, Masoud Mansoury, Robin Burke, Bamshad Mobasher, and Edward Malthouse. 2021.  _User-Centered Evaluation of Popularity Bias in Recommender Systems_.  Association for Computing Machinery, New York, NY, USA, 119–129.  https://doi.org/10.1145/3450613.3456821
  * Carbonell and Goldstein (1998) Jaime Carbonell and Jade Goldstein. 1998.  The Use of MMR, Diversity-based Reranking for Reordering Documents and Producing Summaries. In _21st Annual International ACM SIGIR Conference on Research and Development in Information Retrieval_ (Melbourne, Australia) _(SIGIR ’98)_. ACM, New York, NY, USA, 335–336.  https://doi.org/10.1145/290941.291025
  * da Silva et al. (2021) Diego Corrêa da Silva, Marcelo Garcia Manzato, and Frederico Araújo Durão. 2021.  Exploiting personalized calibration and metrics for fairness recommendation.  _Expert Systems with Applications_ 181 (2021), 115112.  https://doi.org/10.1016/j.eswa.2021.115112
  * George and Merugu (2005) T. George and S. Merugu. 2005.  A scalable collaborative filtering framework based on co-clustering. In _Fifth IEEE International Conference on Data Mining (ICDM’05)_ _(ICDM ’05)_. IEEE Computer Society, USA, 625–628.  https://doi.org/10.1109/ICDM.2005.14
  * Harper and Konstan (2015) F. Maxwell Harper and Joseph A. Konstan. 2015.  The MovieLens Datasets: History and Context.  _ACM Trans. Interact. Intell. Syst._ 5, 4, Article 19 (Dec. 2015), 19 pages.  https://doi.org/10.1145/2827872
  * Hug (2020) Nicolas Hug. 2020\.  Surprise: A Python library for recommender systems.  _Journal of Open Source Software_ 5 (08 2020), 2174\.  https://doi.org/10.21105/joss.02174
  * Kaminskas and Bridge (2016) Marius Kaminskas and Derek Bridge. 2016.  Diversity, Serendipity, Novelty, and Coverage: A Survey and Empirical Analysis of Beyond-Accuracy Objectives in Recommender Systems.  _ACM Trans. Interact. Intell. Syst._ 7, 1, Article 2 (Dec. 2016), 42 pages.  https://doi.org/10.1145/2926720
  * Kaya and Bridge (2019) Mesut Kaya and Derek Bridge. 2019.  A Comparison of Calibrated and Intent-Aware Recommendations. In _13th ACM Conference on Recommender Systems_ (Copenhagen, Denmark) _(RecSys ’19)_. Association for Computing Machinery, New York, NY, USA, 151–159.  https://doi.org/10.1145/3298689.3347045
  * Koren (2008) Yehuda Koren. 2008\.  Factorization Meets the Neighborhood: A Multifaceted Collaborative Filtering Model. In _14th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining_ (Las Vegas, Nevada, USA) _(KDD ’08)_. ACM, New York, NY, USA, 426–434.  https://doi.org/10.1145/1401890.1401944
  * Koren (2010) Yehuda Koren. 2010\.  Factor in the Neighbors: Scalable and Accurate Collaborative Filtering.  _ACM Trans. Knowl. Discov. Data_ 4, 1, Article 1 (Jan. 2010), 24 pages.  https://doi.org/10.1145/1644873.1644874
  * Koren and Bell (2015) Yehuda Koren and Robert Bell. 2015.  Advances in Collaborative Filtering.  In _Recommender Systems Handbook_ , Francesco Ricci, Lior Rokach, and Bracha Shapira (Eds.). Springer US, Boston, MA, 77–118.  https://doi.org/10.1007/978-1-4899-7637-6_3
  * Koren et al. (2009) Yehuda Koren, Robert Bell, and Chris Volinsky. 2009\.  Matrix Factorization Techniques for Recommender Systems.  _Computer_ 42, 8 (Aug. 2009), 30–37.  https://doi.org/10.1109/MC.2009.263
  * Lemire and Maclachlan (2007) Daniel Lemire and Anna Maclachlan. 2007.  Slope One Predictors for Online Rating-Based Collaborative Filtering.  _2005 SIAM International Conference on Data Mining, SDM 2005_ 5 (02 2007).  https://doi.org/10.1137/1.9781611972757.43
  * Lin et al. (2020) Kun Lin, Nasim Sonboli, Bamshad Mobasher, and Robin Burke. 2020\.  Calibration in Collaborative Filtering Recommender Systems: A User-Centered Analysis. In _31st ACM Conference on Hypertext and Social Media_ (Virtual Event, USA) _(HT ’20)_. Association for Computing Machinery, New York, NY, USA, 197–206.  https://doi.org/10.1145/3372923.3404793
  * Luo et al. (2014) X. Luo, M. Zhou, Y. Xia, and Q. Zhu. 2014\.  An Efficient Non-Negative Matrix-Factorization-Based Approach to Collaborative Filtering for Recommender Systems.  _IEEE Transactions on Industrial Informatics_ 10, 2 (May 2014), 1273–1284.  https://doi.org/10.1109/TII.2014.2308433
  * Nemhauser et al. (1978) G. L. Nemhauser, L. A. Wolsey, and M. L. Fisher. 1978\.  An Analysis of Approximations for Maximizing Submodular Set functions–I.  _Math. Program._ 14, 1 (Dec. 1978), 265–294.  https://doi.org/10.1007/BF01588971
  * Pitoura et al. (2021) Evaggelia Pitoura, Kostas Stefanidis, and Georgia Koutrika. 2021\.  Fairness in rankings and recommendations: an overview.  _The VLDB Journal_ (02 Oct 2021).  https://doi.org/10.1007/s00778-021-00697-y
  * Sacharidis et al. (2019) Dimitris Sacharidis, Kyriakos Mouratidis, and Dimitrios Kleftogiannis. 2019.  A Common Approach for Consumer and Provider Fairness in Recommendations. In _13th ACM Conference on Recommender Systems_ _(RecSys ’19)_. ACM, New York, NY, USA.  http://ceur-ws.org/Vol-2431/paper1.pdf
  * Seymen et al. (2021) Sinan Seymen, Himan Abdollahpouri, and Edward C. Malthouse. 2021\.  A Constrained Optimization Approach for Calibrated Recommendations. In _Fifteenth ACM Conference on Recommender Systems_ (Amsterdam, Netherlands) _(RecSys ’21)_. Association for Computing Machinery, New York, NY, USA, 607–612.  https://doi.org/10.1145/3460231.3478857
  * Starychfojtu and Peska (2020) Josef Starychfojtu and Ladislav Peska. 2020.  SmartRecepies: Towards Cooking and Food Shopping Integration via Mobile Recipes Recommender System. In _Proceedings of the 22nd International Conference on Information Integration and Web-Based Applications; Services_ (Chiang Mai, Thailand) _(iiWAS ’20)_. Association for Computing Machinery, New York, NY, USA, 144–148.  https://doi.org/10.1145/3428757.3429096
  * Steck (2018) Harald Steck. 2018\.  Calibrated Recommendations. In _12th ACM Conference on Recommender Systems_ (Vancouver, British Columbia, Canada) _(RecSys ’18)_. ACM, New York, NY, USA, 154–162.  https://doi.org/10.1145/3240323.3240372
  * Zhao et al. (2020) Xing Zhao, Ziwei Zhu, Majid Alfifi, and James Caverlee. 2020\.  Addressing the Target Customer Distortion Problem in Recommender Systems. In _The Web Conference 2020_ (Taipei, Taiwan) _(WWW ’20)_. Association for Computing Machinery, New York, NY, USA, 2969–2975.  https://doi.org/10.1145/3366423.3380065
  * Zhao et al. (2021) Xing Zhao, Ziwei Zhu, and James Caverlee. 2021.  _Rabbit Holes and Taste Distortion: Distribution-Aware Recommendation with Evolving Interests_.  Association for Computing Machinery, New York, NY, USA, 888–899.  https://doi.org/10.1145/3442381.3450099