[fieldsource=doi, match={
_}, replace=_] [fieldsource=author, match={́
i}, replace=í]

# Can We Trust Recommender System Fairness Evaluation? The Role of Fairness
and Relevance

Theresia Veronika Rampisela<EMAIL_ADDRESS>0000-0003-1233-7690 University of
CopenhagenCopenhagenDenmark ,  Tuukka Ruotsalo<EMAIL_ADDRESS>0000-0002-2203-4928 University of CopenhagenCopenhagenDenmark LUT
UniversityLahtiFinland ,  Maria Maistro<EMAIL_ADDRESS>0000-0002-7001-4817
University of CopenhagenCopenhagenDenmark and  Christina Lioma
<EMAIL_ADDRESS>0000-0003-2600-2701 University of CopenhagenCopenhagenDenmark

(2024)

###### Abstract.

Relevance and fairness are two major objectives of recommender systems (RSs).
Recent work proposes measures of RS fairness that are either independent from
relevance (fairness-only) or conditioned on relevance (joint measures). While
fairness-only measures have been studied extensively, we look into whether
joint measures can be trusted. We collect all joint evaluation measures of RS
relevance and fairness, and ask: How much do they agree with each other? To
what extent do they agree with relevance/fairness measures? How sensitive are
they to changes in rank position, or to increasingly fair and relevant
recommendations? We empirically study for the first time the behaviour of
these measures across 4 real-world datasets and 4 recommenders. We find that
most of these measures: i) correlate weakly with one another and even
contradict each other at times; ii) are less sensitive to rank position
changes than relevance- and fairness-only measures, meaning that they are less
granular than traditional RS measures; and iii) tend to compress scores at the
low end of their range, meaning that they are not very expressive. We counter
the above limitations with a set of guidelines on the appropriate usage of
such measures, i.e., they should be used with caution due to their tendency to
contradict each other and of having a very small empirical range.

fairness and relevance evaluation; recommender systems

††journalyear: 2024††copyright: rightsretained††conference: Proceedings of the
47th International ACM SIGIR Conference on Research and Development in
Information Retrieval; July 14–18, 2024; Washington, DC, USA.††booktitle:
Proceedings of the 47th International ACM SIGIR Conference on Research and
Development in Information Retrieval (SIGIR ’24), July 14–18, 2024,
Washington, DC, USA††isbn: 979-8-4007-0431-4/24/07††doi:
10.1145/3626772.3657832††ccs: Information systems Evaluation of retrieval
results††ccs: General and reference Evaluation††ccs: Information systems
Recommender systems

##  1\. Introduction

Recent increased focus on fairness in recommender systems (RSs) has led to
studies on how to evaluate different notions of fairness in RS. A recent
survey (Wang2022, 38) shows that prior work on fairness evaluation in RS
mainly focuses on group fairness (e.g., (Raj2022MeasuringResults, 29, 1, 44)),
but less so on individual fairness. Individual fairness is commonly understood
as treating similar individuals similarly (Dwork2012FairnessAwareness, 9).
Unlike group fairness evaluation, evaluating individual fairness does not
require information on sensitive attributes (e.g., gender, age) to identify
protected groups (Lazovich2022MeasuringMetrics, 18). Such information is often
unavailable due to privacy and legal issues. Further, intersectionality
between different group characteristics complicates group fairness
(Crenshaw1991MappingColor, 6, 10). Individual fairness is known to lead into
group fairness, but not vice versa (Bower2021IndividuallyRanking, 4). Overall,
individual fairness gives a broader view by assessing distribution across all
individuals in the population (Lazovich2022MeasuringMetrics, 18). For all
these reasons, we focus on individual fairness, particularly individual item
fairness, which is typically broadly defined w.r.t. exposure received by
items, i.e., how uniform the exposure distribution between items is
(Rampisela2023EvaluationStudy, 30). Yet, fairness beyond exposure also
matters, i.e., the exposure should be proportional to item relevance
(Patro2022FairDirections, 27, 35, 2).

Individual item fairness is measured by measures that (i) are detached from
relevance (fairness-only measures, defined by exposure); or (ii) are
conditioned on relevance (joint measures considering exposure w.r.t.
relevance). Measures of type (i) have been extensively analysed
(Rampisela2023EvaluationStudy, 30), but to our knowledge, this is not the case
for measures of type (ii). The growing number of measures of type (ii)
necessitates a thorough look into their usage in RS evaluation.

We present a comprehensive study into the empirical properties of all joint
measures of individual item fairness and relevance, motivated by the question
of how much can we practically trust these measures, particularly: RQ1. To
what extent do the joint measures agree with existing relevance- and fairness-
only measures? RQ2. To what extent do the joint measures agree with each
other? RQ3. How sensitive are the joint measures across decreasing rank
positions? and RQ4. How sensitive are the joint measures given increasingly
fair and relevant recommendations?

We identify some alarming limitations in the measures, and we reflect on their
best usage in practice. This is the first in-depth study on individual item
fairness measures that consider relevance in RS.

##  2\. Individual item fairness & relevance

We present the notation ($\S$2.1) and all existing joint evaluation measures
of individual item fairness and relevance ($\S$2.2).

###  2.1. Notation and definitions

Given a set of $n$ items, $I=\\{i_{1},i_{2},\dots,i_{n}\\}$, and a set of $m$
users, $U=\\{u_{1},u_{2},\dots,u_{m}\\}$, an ordered list of the $n$ items is
created for each $u\in U$. This list is created in each recommendation round
$w$, where $w\in\\{1,2,\dots,W\\}$; a round means an occurrence in which a
user receives a list of recommendations. If an item $i$ is _relevant_ to user
$u$, we write $r_{u,i}=1$, otherwise $r_{u,i}=0$. Relevance can also be
denoted as real values, $r_{u,i}\in[0,1]$. The list of user $u$’s top $k$
recommended items in round $w$ is $L_{u,w}$ and the rank position of item $i$
in user $u$’s recommendation list is $z(u,i,w)$. For cases with only one
recommendation round, user $u$’s list of top $k$ recommended items is $L_{u}$
and the rank position of item $i$ for user $u$ is $z(u,i)$.

While several different definitions of fairness exist, the definitions
commonly used in prior work on individual item fairness are closely linked to
item exposure (Rampisela2023EvaluationStudy, 30, 1, 44). An item is _exposed_
when it is recommended at the top $k$ to a user. The probability of a user
seeing an item exposed to them can be modelled using various _examination
functions_ , $e(\cdot)$. Examination functions typically assume that the
viewing probability depends only on the position $z(u,i,w)$ or $z(u,i)$. This
is a common choice across all measures in $\S$2.2.

The examination functions used in this work are shown in Tab. 1: the linear
examination function, $e_{\text{li}}$ and its min-max normalised version
$\tilde{e}_{\text{li}}$ apply a linear discount to each rank position up to
$k$ (Borges2019EnhancingAutoencoders, 3). Meanwhile, discounts based on
Discounted Cumulative Gain (DCG) (Jarvelin2002CumulatedTechniques, 16) and
Rank-Biased Precision (RBP) (Moffat2008Rank-biasedEffectiveness, 24) are used
in $e_{\text{DCG}}$ (Singh2019PolicyRanking, 34, 26) and $e_{\text{RBP}}$
(Wu2022JointRecommendation, 39, 17) respectively. The parameter $\gamma$ in
$e_{\text{RBP}}$ is the user’s patience, i.e., the probability of the user
examining the next ranked item. The user patience parameter is commonly set at
e.g., $\gamma\in\\{0.8,0.9\\}$ (Wu2022JointRecommendation, 39, 17). In the
inverse examination function $e_{\text{inv}}$, the inverse of the rank
position is used as a discount factor (Saito2022FairRanking, 32). Overall, we
use three types of examination functions (linear, discounted, and inverse),
which assume that item exposure diminishes with decreasing ranking either
linearly, or with an increasing penalty that is either proportional to the
rank decrease or the inverse of the rank. Generally, the most punishing is the
inverse function, and the least punishing is the DCG-based discount function.

Table 1. Examination functions used in this work. $\tilde{e}$ is the min-max
normalised examination function.

| Equation | Measure | Reference
---|---|---|---
linear |  $e_{\text{li}}(u,i,w)=k+1-z(u,i,w)$ $\tilde{e}_{\text{li}}(u,i,w)=\frac{e_{\text{li}}(u,i,w)-1}{k-1}=\frac{k-z(u,i,w)}{k-1}$ | IAA | (Borges2019EnhancingAutoencoders, 3)
DCG | $e_{\text{DCG}}(u,i,w)=1/\log_{2}(z(u,i,w)+1)$ | IFD | (Singh2019PolicyRanking, 34, 26)
RBP | $e_{\text{RBP}}(u,i,w)=\gamma^{z(u,i,w)-1}$ | HD, II-F, AI-F | (Wu2022JointRecommendation, 39, 17)
inverse | $e_{\text{inv}}(u,i)=1/z(u,i)$ | MME, IBO/IWO | (Saito2022FairRanking, 32)

###  2.2. Joint measures of fairness and relevance

We present measures that evaluate fairness considering relevance (Fair+Rel or
joint measures henceforth). To our knowledge, we include all Fair+Rel measures
for RSs published up to October 2023. Each measure uses an exposure function,
which is linked to the fairness of item distribution in the recommendation
and, therefore, measures item fairness jointly with relevance. We use
$\uparrow$ for measures where the higher the score, the fairest the
recommendation, and vice versa for $\downarrow$. All measures–except HD–are
defined for multiple recommendation rounds or stochastic rankings, where a
distribution over rankings is considered (Biega2018EquityRankings, 2).

####  2.2.1. Inequity of Amortized Attention (IAA) (Biega2018EquityRankings,
2)

$\downarrow$IAA111This measure is called IAA in (Raj2022MeasuringResults, 29)
and L1-norm in (Wang2022, 38). measures fairness as the aggregated difference
between item exposure and its relevance in a series of rankings that have been
generated by a stochastic process (Biega2018EquityRankings, 2). The intuition
behind IAA is that for a sequence of rankings to be fair, items should be
allocated exposure proportional to their relevance to the user. The item
position is a proxy of its exposure level. IAA was modified in
(Borges2019EnhancingAutoencoders, 3) to account for multiple recommendation
rounds (stochastic rankings):

(1) $\text{IAA}=\frac{1}{m}\sum\limits_{u\in U}\text{IAA}(u)$ (2)
$\text{IAA}(u)=\frac{1}{n}\frac{1}{W}\sum_{i\in
I}\left|\sum_{w=1}^{W}1_{L_{u,w}}(i)\cdot\tilde{e_{.}}(u,i,w)-\tilde{r}(u,i,w)\right|$

In (Borges2019EnhancingAutoencoders, 3), $\tilde{e}_{(\cdot)}(u,i,w)$ is the
min-max normalised linear examination function $\tilde{e}_{\text{li}}(u,i,w)$
(see Tab. 1) and $\tilde{r}(u,i,w)\in[0,1]$ is the min-max normalised
relevance value of item $i$ for user $u$ in round $w$, $r_{u,i,w}$.222Note
that the normalised exposure value for a recommended item at $k$ is zero. Both
the min and max relevance values are taken from the values associated with all
items for each user per round, i.e., $\min_{i\in I}r_{u,i,w}$. This value is
the aggregated relevance over all items for user $u$ in round $w$; the higher
the relevance, the closer to 1. The higher the relevance value differs from
item exposure, the more unfair. The range of IAA is $[0,1]$.

####  2.2.2. Individual Fairness Disparity (IFD) (Singh2019PolicyRanking, 34,
26)

$\downarrow$IFD is the average pairwise difference of the combined value of
item exposure and item merit. Merit is defined as a function of
relevance.333We use the item relevance value as the item merit, as per
(Singh2019PolicyRanking, 34). Similar to IAA, IFD follows the principle of
allocating exposure to an item based on its relevance. While IAA computes the
difference between the exposure and relevance of each item, IFD computes the
disparity of exposure allocation between item pairs. Based on how exposure and
merit are combined, two variations of IFD exist: IFD÷, where item exposure is
divided by item relevance (Singh2019PolicyRanking, 34), and IFD×, where the
division is replaced by multiplication (Oosterhuis2021ComputationallyFairness,
26). The term IFD(⋅) or IFD refers to the measure in general. The two versions
slightly differ in the pairwise difference computation, the formation of set
of item pairs, and the exposure weighting scheme.444Exposure is weighed
proportional to $e_{\text{DCG}}$ in (Singh2019PolicyRanking, 34); to simplify,
we use $e_{\text{DCG}}$ directly. Both IFD versions have been used to measure
fairness in ranking (Singh2019PolicyRanking, 34, 26, 41, 42).

(3) $\text{IFD}_{(\cdot)}=\frac{1}{m}\sum\limits_{u\in
U}\text{IFD}_{(\cdot)}(u)$ (4)
$\text{IFD}_{\div}(u)=\frac{1}{|H_{u}|}\sum_{(i,i^{\prime})\in
H_{u}}\max{\left\\{0,J_{\div}(u,i)-J_{\div}(u,i^{\prime})\right\\}}$ (5)
$\text{IFD}_{\times}(u)=\frac{1}{n(n-1)}\sum_{i\in I}\sum_{i^{\prime}\in
I\setminus{i}}\left[J_{\times}(u,i)-J_{\times}(u,i^{\prime})\right]^{2}$ (6)
$J_{\div}(u,i)=\frac{1}{W}\sum\limits_{w=1}^{W}e_{\text{DCG}}(u,i,w)/r_{u,i,w}$
(7) $J_{\times}(u,i)=\frac{1}{W}\sum\limits_{w=1}^{W}r_{u,i,w}\cdot
1_{L_{u,w}}(i)\cdot e_{\text{DCG}}(u,i,w)$

$J_{(\cdot)}(u,i)$ is the function combining the expected exposure and
relevance of item $i$ for user $u$ and $H_{u}=\\{(i,i^{\prime})\in I\ |\
r_{u,i}\geq r_{u,i^{\prime}}>0\\}$. The range of IFD÷ is $[0,\infty)$ and it
is 0 only when the exposure received by each relevant item is exactly
proportional to its relevance (Singh2019PolicyRanking, 34). The range of IFD×
is $[0,\infty)$ based on empirical results (Yang2023FARA:Optimization, 41).

####  2.2.3. Hellinger Distance (HD) (Jeunen2021Top-KExposure, 17)

$\downarrow$HD has been used as a measure of individual item fairness in top
$k$ contextual bandits, by quantifying the difference between the relevance-
and click-distributions of the top $k$ items sorted according to (ground
truth) relevance (Jeunen2021Top-KExposure, 17). The click probability is based
on user patience, system-allocated item exposure, and item relevance. A
recommendation is fair based on HD when the click probability of an item is
proportional to the relevance probability of that item. To compute the
relevance and click distributions, a list of top $k$ items is created for each
user by sorting items based on their (ground truth) relevance; this list is
the reference list used in the next step. Another list of items is created
based on system prediction and used to get the click probability. For each
item in the reference list, we compute its click probability based on its
order in the second list. Next, the relevance probabilities of items at the
same position in the reference list are aggregated across users and similarly
for the click probabilities. For each rank position, two aggregated values are
obtained: relevance and click. The aggregated values are the inputs to the
distance metric (Eq. (8)).

(8)
$\text{HD}=\frac{1}{\sqrt{2}}\sqrt{\sum_{p=1}^{k}\left(\sqrt{q_{p}^{\prime}}-\sqrt{c_{p}^{\prime}}\right)^{2}}$
(9) $q_{p}^{\prime}=\frac{1}{m}\sum_{u\in U}\sum_{i\in
I}\delta\left(z^{*}(u,i)=p\right)\cdot r^{\prime}_{u,i}$ (10)
$c_{p}^{\prime}=\frac{1}{m}\sum_{u\in
U}\frac{c^{*}_{u,p}}{\sum_{\ell=1}^{k}c^{*}_{u,\ell}}$ (11)
$c^{*}_{u,p}=\sum_{i\in I}\delta\left(z^{*}(u,i)=p\right)\cdot c^{full}_{u,i}$
(12) $c^{full}_{u,i}=c^{\prime}_{u,p}\,\text{if }\exists p:z(u,i)=p\,\text{,
otherwise }0$ (13) $c_{u,p}=\sum_{i\in L_{u}}\delta\left(z(u,i)=p\right)\cdot
r_{u,i}\cdot\gamma\ e_{\text{RBP}}(u,i)\cdot s_{u,p}$ (14)
$s_{u,p}=\prod_{1\leq j<p}1-\sum_{i\in L_{u}}\delta(z(u,i)=j)\cdot r_{u,i}$

where $q_{p}^{\prime}$ and $c_{p}^{\prime}$ are the normalised relevance and
click probability of the item at position $j$ respectively, where click
depends on both relevance and exposure. The position of item $i$ based on
ground-truth relevance is $z^{*}(u,i)$. The click probability of user $u$ for
item at position $p$, $c_{u,p}$ depends on $s_{u,p}$, the probability that
items before position $p$ were irrelevant to the user, and the user patience
$\gamma\ e_{\text{RBP}}(u,i)$. $r^{\prime}_{u,i}=r_{u,i}/\sum_{i\in I}r_{u,i}$
is the user-wise normalised relevance value of item $i$ to user $u$, and
$c^{\prime}_{u,p}=c_{u,p}/\sum^{k}_{p=1}c_{u,p}$ is the user-wise normalised
click probability. The value of $\delta(\cdot)=1$ when the expression $\cdot$
is True and 0 otherwise. HD ranges between $[0,\infty)$.

####  2.2.4. Mean Max Envy (MME) (Saito2022FairRanking, 32)

$\downarrow$MME uses the concept of envy-freeness, where a recommendation is
fair when each item is not disadvantaged by its own exposure allocation
compared to being allocated the exposure of any other item. In other words,
MME computes unfairness as the disadvantage suffered by the item, if the
exposure allocation of an item is swapped with another item. The disadvantage
is computed based on an impact score that uses exposure and relevance: given
full recommendation lists (size $n$) across all users, we swap each item $i$
with another item $i^{\prime}$ and compute the impact score before and after
the swap for all rank positions and users. If the score of the item $i$ before
the swap is greater or equal to its score after the swap, we have envy-
freeness for item $i$ w.r.t. item $i^{\prime}$. MME thus computes the average
maximum difference of impact imposed if item $i$ is replaced with another item
$i^{\prime}$. E.g., let $L_{u_{1}}=[i_{1},i_{2}],\ L_{u_{2}}=[i_{1},i_{3}]$
and let us swap item $i_{3}$ with $i_{1}$. Item $i_{3}$ will be exposed to
both users at the top position, like $i_{1}$ did, and then impact is
recomputed. MME is computed as follows:

(15) $\text{MME}=\frac{1}{n}\sum_{i\in I}\left\\{\max_{i^{\prime}\in
I}Imp_{i}(i^{\prime})-Imp_{i}(i)\right\\}$ (16)
$Imp_{i}(i^{\prime})=\sum\limits_{u\in U}\sum\limits_{p=1}^{k}r_{u,i}\cdot
e_{\text{inv}}(u,i^{\prime})\cdot X_{u,i^{\prime},p}$ (17)
$X_{u,i^{\prime},p}=\frac{1}{W}\frac{1}{m}\sum\limits_{w=1}^{W}1_{L_{u,w}}(i^{\prime})\cdot\delta(z(u,i^{\prime},w)=p)$

where $Imp_{i}(i^{\prime})$ is the impact when we allocate the exposure of
item $i^{\prime}$ to item $i$, $X_{u,i^{\prime},p}$ is the probability that
item $i^{\prime}$ is recommended to user $u$ at position $p$ in $W$ rounds of
recommendations, and $e_{\text{inv}}(u,i^{\prime})$ is the exposure weight of
item $i^{\prime}$ to user $u$, based on the inverse examination function (see
Tab. 1). MME ranges within $[0,\infty)$.

####  2.2.5. Item Better-Off (IBO) & Item Worse-Off (IWO)
(Saito2022FairRanking, 32)

$\uparrow$IBO and $\downarrow$IWO use the principle of dominance over uniform
ranking, where fairness means each item has a better impact (as defined in
MME) under the current ranking policy, than if it were under the uniform
random ranking policy $\pi_{unif}$, which samples all possible permutations of
items uniformly at random. IBO/IWO measures the percentage of items for which
our current ranking policy increases/decreases impact by at least 10% compared
to $\pi_{unif}$555In (Saito2022FairRanking, 32), 10% is hard coded, but this
can be a variable. We also use 10%.:

(18) $\text{IBO}=\frac{100}{|I^{-}|}\sum_{i\in
I^{-}}\delta\left(Imp_{i}(i)\geq 1.1\cdot Imp_{i}^{unif}\right)$ (19)
$\text{IWO}=\frac{100}{|I^{-}|}\sum_{i\in I^{-}}\delta\left(Imp_{i}(i)\leq
0.9\cdot Imp_{i}^{unif}\right)$ (20)
$Imp^{unif}_{i}=\frac{1}{m}\frac{1}{n}\sum\limits_{p=1}^{k}\frac{1}{p}\cdot\sum\limits_{u\in
U}r_{u,i}$

where $I^{-}$ is the set of items with at least one user that finds the item
relevant. This ensures that the set of items that cause $\delta(\cdot)=1$ in
IBO is disjoint from that in IWO.666We exclude items with no relevant users,
as for these items $Imp_{i}(i)=Imp^{unif}_{i}=0$, causing the same items being
considered ‘better-off’ and ‘worse-off’ at the same time. $Imp_{i}(i)$ is as
per Eq. (16) and $Imp^{unif}_{i}$ is the impact if item $i$ is exposed
according to $\pi_{unif}$ using $e_{\text{inv}}(u,i)$ as examination function
(see Tab. 1). Note that the above definitions are modifications to the
formulation of (Saito2022FairRanking, 32) to avoid computational issues that
result in division by zero (undefinedness limitation
(Rampisela2023EvaluationStudy, 30)).777We move the divisor $Imp^{unif}_{i}$
from the left-hand side to the right-hand side. IBO/IWO ranges between
$[0,100]$.

####  2.2.6. Individual-user-to-individual-item fairness (II-F)
(Wu2022JointRecommendation, 39)

$\downarrow$II-F was first defined by (Diaz2020EvaluatingExposure, 8) to
quantify unfairness as the disparity between system exposure and target
exposure in individual queries and individual items. II-F was redefined by
(Wu2022JointRecommendation, 39) for RSs as:

(21) $\text{II-F}=\frac{1}{m}\frac{1}{n}\sum\limits_{u\in U}\sum\limits_{i\in
I}\left(E_{u,i}-E_{u,i}^{*}\right)^{2}$ (22)
$E_{u,i}=\frac{1}{W}\sum\limits_{w=1}^{W}1_{L_{u,w}}(i)\cdot
e_{\text{RBP}}(u,i,w)$ (23)
$E_{u,i}^{*}=\frac{r_{u,i}}{|R_{u}^{*}|}\cdot\frac{1-\gamma^{|R_{u}^{*}|}}{1-\gamma}\,\text{if
}|R_{u}^{*}|>0\,\text{, otherwise }0$

where $E_{u,i}$ is the expected exposure of $i$ to $u$ as per a stochastic
ranking policy. $E_{u,i}^{*}$ is the expected exposure of $i$ to $u$ as per an
ideal stochastic ranking policy, which assumes that relevant items get equal
expected exposure (Diaz2020EvaluatingExposure, 8). Thus, the recommendation is
fair based on II-F if the system exposure matches the exposure allocated to
items under an ideal ranking policy. The examination function based on RBP
(see Tab. 1) is used in $E_{u,i}$ and the equation of $E_{u,i}^{*}$ is derived
based on the same examination function (Wu2022JointRecommendation, 39).
$|R_{u}^{*}|$ is the number of relevant items for user $u$. II-F ranges
between $[0,1]$.

####  2.2.7. All-users-to-individual-item fairness (AI-F)
(Wu2022JointRecommendation, 39)

$\downarrow$AI-F evaluates how much RSs under/overexpose an item to all users
as the mean deviation of overall system exposure over target exposure:

(24) $\text{AI-F}=\frac{1}{n}\sum\limits_{i\in
I}\left(\frac{1}{m}\sum\limits_{u\in U}E_{u,i}-\frac{1}{m}\sum\limits_{u\in
U}E_{u,i}^{*}\right)^{2}$

where $E_{u,i}$, $E_{u,i}^{*}$ are as per Eq. (22)–(23). Similar to II-F, AI-F
also quantifies fairness based on how close the system exposure is to the
target exposure. In II-F, this disparity is computed individually between each
user-item pair, while in AI-F item exposure is first aggregated across users
prior to computing the difference in exposure. Due to this aggregation, AI-F
would have a better fairness score than II-F when there is a greater number of
unique items in the recommendation, as opposed to having the same few items
exposed to all users. The range of AI-F is $[0,1]$.

##  3\. Experimental setup

We study the above Fair+Rel measures across different recommenders and
datasets. Our general experimental setup follows, and we provide the
description of the experiments in $\S$4.888Our code:
https://github.com/theresiavr/can-we-trust-recsys-fairness-evaluation.

Datasets. We use four real-world datasets of varying sizes and domains: Lastfm
(music) (Cantador20112nd2011, 5), Amazon Luxury Beauty, i.e., Amazon-lb
(e-commerce) (Ni2019JustifyingAspects, 25), QK-video (videos)
(Yuan2022Tenrec:Systems, 43), and ML-10M (movies) (Harper2015TheContext, 14).
QK-video is as provided by (Yuan2022Tenrec:Systems, 43), and the rest are as
provided by (Zhao2021RecBole:Algorithms, 45). For QK-video, we use only the
‘sharing’ interactions.

Table 2. Statistics of the preprocessed datasets.

dataset | #users ($m$) | #items ($n$) | #interactions | sparsity (%)
---|---|---|---|---
Lastfm (Cantador20112nd2011, 5) | 1,859 | 2,823 | 71,355 | 98.64%
Amazon-lb (Ni2019JustifyingAspects, 25) | 1,054 | 791 | 12,397 | 98.51%
QK-video (Yuan2022Tenrec:Systems, 43) | 4,656 | 6,423 | 51,777 | 99.83%
ML-10M (Harper2015TheContext, 14) | 49,378 | 9,821 | 5,362,685 | 98.89%

Preprocessing. We keep only users and items with at least 5 interactions
(5-core filtering). When there are duplicate interactions, we keep the most
recent one. Ratings equal/above 3 are converted to 1, and the rest are
discarded for Amazon-lb and ML-10M, as their ratings range between $[1,5]$ and
$[0.5,5]$ respectively. No conversions are done for Lastfm and QK-video as
they only have implicit feedback. Tab. 2 presents statistics of the
preprocessed datasets.

Data splits. Global temporal splits (Meng2020ExploringModels, 23) with a ratio
of 6:2:2 form the train/val/test sets from the preprocessed datasets for
Amazon-lb and ML-10M. Global random splits with the same ratio are used for
Lastfm and QK-video as they have no timestamps. Only users with at least 5
interactions in the train set are kept in all splits.

Recommenders. We use four well-known top $k$ recommenders: item-based
K-Nearest Neighbour (ItemKNN) (Deshpande2004Item-basedAlgorithms, 7), Bayesian
Personalised Ranking (BPR), (RendleBPR:Feedback, 31), Variational Autoencoder
with multinomial likelihood (MultiVAE) (Liang2018VariationalFiltering, 20),
and Neighbourhood-enriched Contrastive Learning (NCL)
(Lin2022ImprovingLearning, 21). We train BPR, MultiVAE, and NCL using RecBole
(Zhao2021RecBole:Algorithms, 45) for 300 epochs with early stopping. The
configuration with the best NDCG@10 during validation is taken as the final
model.999The hyperparameter search space and best values are in the code
repository. During testing, all unobserved items are selected as candidates
for recommendation and each user’s train/val items are excluded from their own
recommendations.

Fair re-ranker. As the models are not directly optimised for fairness, we use
a re-ranker to obtain fairer recommendations. The top $k^{\prime}$ items are
re-ranked to provide exposure to items that were outside the top $k$, where
$k^{\prime}$ is ideally larger than the cut-off $k=10$. In RS datasets,
normally there are very few relevant items per user, so $k^{\prime}$ should
not be too big (e.g., 100). We choose $k^{\prime}=25$ for all datasets and
models. The re-ranking is done per user with COMBMNZ (CM)
(Lee1997AnalysesCombination, 19) as a robust rank fusion method.101010Other
re-rankers exist but do not suit our setup, e.g., (Wang2022ProvidingSystems,
37) requires computing item similarity, but true similarity is challenging to
obtain (Dwork2012FairnessAwareness, 9, 36). CM fuses two lists of scores, one
based on relevance and one based on fairness, to create a new ranking for each
user. The relevance-based score is the min-max normalised predicted relevance
score. The fairness-based score is first obtained from the coverage score of
each top $k^{\prime}$ items based on their appearance in the top $k$. Then, we
compute 1 minus the normalised coverage to allocate higher score for items
with lower exposure, thus increasing fairness. The combined scores are sorted
to generate the final fused ranking of relevance and fairness.

Measures. Recommendation models are evaluated using all the joint measures of
relevance and fairness (Fair+Rel) presented in $\S$2.2.111111For IAA, the
ground truth relevance is used to compute the relevance score. For HD,
$\gamma=0.9$ as per (Jeunen2021Top-KExposure, 17). For II-F and AI-F,
$\gamma=0.8$ as per (Wu2022JointRecommendation, 39). Note that IBO/IWO are
normalised to [0,1] for consistency with the other measures. As comparison to
the joint measures, we evaluate relevance only (Rel) with: Hit Rate (HR), MRR,
Precision (P), Recall (R), MAP, and NDCG. We also evaluate fairness only
(Fair) with:121212We use the modified versions of these measures as per
(Rampisela2023EvaluationStudy, 30). Jain Index (Jain) (jain1984quantitative,
15, 46), Qualification Fairness (QF) (Zhu2020FARM:APPs, 46), Entropy (Ent)
(Patro2020FairRec:Platforms, 28, 33), Fraction of Satisfied Items (FSat)
(Patro2020FairRec:Platforms, 28), and Gini Index (Gini)
(Gini1912VariabilitaMutabilita, 13, 22). Unless otherwise stated, all measures
are computed at $k=10$.

##  4\. Empirical analysis

We present the evaluation results of all Fair+Rel, Rel, and Fair measures, in
$\S$4.1. We study their correlation in $\S$4.2, their sensitivity across
different top-$k$ positions in $\S$4.3 and across increasing levels of
relevance and fairness in $\S$4.4.

###  4.1. Evaluation results of all measures

Table 3. Relevance (Rel), fairness (Fair), and joint Fair+Rel scores at
$\mathbf{k=10}$ without and with re-ranking the top $\mathbf{k^{\prime}=25}$
items using COMBMNZ (CM). Bold marks the most relevant/fair score per measure.
The score 0.000 does not mean the scores are exactly 0; this is due to the
measures having small scores ($\mathbf{<10^{-3}}$) and rounding to 3 d.p.

|  | model | ItemKNN | BPR | MultiVAE | NCL
---|---|---|---|---|---|---
|  | re-ranker | - | CM | - | CM | - | CM | - | CM
Lastfm |  Rel |  $\uparrow$ HR | 0.765 | 0.581 | 0.773 | 0.587 | 0.778 | 0.523 | 0.793 | 0.571
$\uparrow$ MRR | 0.484 | 0.270 | 0.492 | 0.280 | 0.476 | 0.232 | 0.503 | 0.260
$\uparrow$ P | 0.172 | 0.089 | 0.178 | 0.092 | 0.176 | 0.076 | 0.184 | 0.087
$\uparrow$ MAP | 0.137 | 0.053 | 0.141 | 0.058 | 0.138 | 0.045 | 0.148 | 0.050
$\uparrow$ R | 0.218 | 0.114 | 0.224 | 0.119 | 0.224 | 0.098 | 0.234 | 0.110
$\uparrow$ NDCG | 0.245 | 0.119 | 0.252 | 0.126 | 0.247 | 0.102 | 0.261 | 0.115
Fair |  $\uparrow$ Jain | 0.042 | 0.094 | 0.058 | 0.140 | 0.097 | 0.222 | 0.082 | 0.215
$\uparrow$ QF | 0.474 | 0.679 | 0.362 | 0.528 | 0.517 | 0.678 | 0.453 | 0.657
$\uparrow$ Ent | 0.589 | 0.735 | 0.610 | 0.740 | 0.707 | 0.826 | 0.671 | 0.810
$\uparrow$ FSat | 0.129 | 0.216 | 0.147 | 0.228 | 0.202 | 0.321 | 0.178 | 0.286
$\downarrow$ Gini | 0.904 | 0.790 | 0.910 | 0.818 | 0.839 | 0.696 | 0.872 | 0.728
Fair+Rel |  $\uparrow$ IBO | 0.209 | 0.256 | 0.208 | 0.253 | 0.261 | 0.278 | 0.242 | 0.292
$\downarrow$ IWO | 0.791 | 0.744 | 0.792 | 0.747 | 0.739 | 0.722 | 0.758 | 0.708
$\downarrow$ IAA | 0.004 | 0.004 | 0.004 | 0.004 | 0.004 | 0.004 | 0.004 | 0.004
$\downarrow$ $\text{IFD}_{\div}$ | 0.074 | 0.053 | 0.075 | 0.054 | 0.073 | 0.049 | 0.076 | 0.052
$\downarrow$ $\text{IFD}_{\times}$ | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
$\downarrow$ HD | 0.099 | 0.177 | 0.104 | 0.174 | 0.095 | 0.203 | 0.092 | 0.177
$\downarrow$ MME | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001
$\downarrow$ II-F | 0.001 | 0.002 | 0.001 | 0.002 | 0.001 | 0.002 | 0.001 | 0.002
$\downarrow$ AI-F | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
Amazon-lb |  Rel |  $\uparrow$ HR | 0.046 | 0.016 | 0.011 | 0.021 | 0.039 | 0.014 | 0.034 | 0.011
$\uparrow$ MRR | 0.020 | 0.011 | 0.003 | 0.007 | 0.023 | 0.004 | 0.022 | 0.003
$\uparrow$ P | 0.005 | 0.002 | 0.001 | 0.002 | 0.004 | 0.002 | 0.004 | 0.001
$\uparrow$ MAP | 0.006 | 0.004 | 0.002 | 0.004 | 0.006 | 0.003 | 0.006 | 0.001
$\uparrow$ R | 0.013 | 0.005 | 0.005 | 0.010 | 0.010 | 0.008 | 0.012 | 0.003
$\uparrow$ NDCG | 0.011 | 0.005 | 0.003 | 0.006 | 0.010 | 0.004 | 0.011 | 0.002
Fair |  $\uparrow$ Jain | 0.271 | 0.431 | 0.223 | 0.359 | 0.035 | 0.097 | 0.026 | 0.080
$\uparrow$ QF | 0.650 | 0.612 | 0.549 | 0.594 | 0.222 | 0.286 | 0.229 | 0.310
$\uparrow$ Ent | 0.802 | 0.839 | 0.747 | 0.809 | 0.418 | 0.558 | 0.371 | 0.534
$\uparrow$ FSat | 0.370 | 0.438 | 0.314 | 0.376 | 0.114 | 0.152 | 0.091 | 0.138
$\downarrow$ Gini | 0.665 | 0.598 | 0.747 | 0.660 | 0.949 | 0.899 | 0.959 | 0.910
Fair+Rel |  $\uparrow$ IBO | 0.062 | 0.029 | 0.019 | 0.038 | 0.029 | 0.029 | 0.038 | 0.024
$\downarrow$ IWO | 0.938 | 0.971 | 0.981 | 0.962 | 0.971 | 0.971 | 0.962 | 0.976
$\downarrow$ IAA | 0.011 | 0.011 | 0.011 | 0.011 | 0.011 | 0.011 | 0.011 | 0.011
$\downarrow$ $\text{IFD}_{\div}$ | 0.005 | 0.003 | 0.003 | 0.002 | 0.005 | 0.002 | 0.005 | 0.003
$\downarrow$ $\text{IFD}_{\times}$ | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
$\downarrow$ HD | 0.580 | 0.630 | 0.661 | 0.626 | 0.597 | 0.653 | 0.598 | 0.667
$\downarrow$ MME | 0.001 | 0.001 | 0.001 | 0.001 | 0.003 | 0.001 | 0.004 | 0.001
$\downarrow$ II-F | 0.006 | 0.006 | 0.006 | 0.006 | 0.006 | 0.006 | 0.006 | 0.006
$\downarrow$ AI-F | 0.000 | 0.000 | 0.000 | 0.000 | 0.001 | 0.000 | 0.002 | 0.000

|  | model | ItemKNN | BPR | MultiVAE | NCL
---|---|---|---|---|---|---
|  | re-ranker | - | CM | - | CM | - | CM | - | CM
QK-video |  Rel |  $\uparrow$ HR | 0.040 | 0.047 | 0.099 | 0.045 | 0.109 | 0.061 | 0.130 | 0.077
$\uparrow$ MRR | 0.013 | 0.013 | 0.039 | 0.015 | 0.039 | 0.021 | 0.048 | 0.024
$\uparrow$ P | 0.004 | 0.005 | 0.011 | 0.005 | 0.012 | 0.006 | 0.014 | 0.008
$\uparrow$ MAP | 0.005 | 0.005 | 0.017 | 0.006 | 0.018 | 0.009 | 0.022 | 0.010
$\uparrow$ R | 0.014 | 0.019 | 0.043 | 0.019 | 0.051 | 0.027 | 0.061 | 0.033
$\uparrow$ NDCG | 0.009 | 0.010 | 0.029 | 0.011 | 0.031 | 0.016 | 0.038 | 0.019
Fair |  $\uparrow$ Jain | 0.483 | 0.589 | 0.081 | 0.379 | 0.012 | 0.032 | 0.020 | 0.071
$\uparrow$ QF | 0.901 | 0.790 | 0.625 | 0.823 | 0.100 | 0.163 | 0.201 | 0.365
$\uparrow$ Ent | 0.933 | 0.937 | 0.755 | 0.903 | 0.420 | 0.547 | 0.507 | 0.674
$\uparrow$ FSat | 0.443 | 0.547 | 0.212 | 0.382 | 0.052 | 0.090 | 0.077 | 0.150
$\downarrow$ Gini | 0.472 | 0.442 | 0.807 | 0.570 | 0.982 | 0.959 | 0.966 | 0.902
Fair+Rel |  $\uparrow$ IBO | 0.033 | 0.038 | 0.054 | 0.036 | 0.031 | 0.036 | 0.043 | 0.054
$\downarrow$ IWO | 0.967 | 0.962 | 0.946 | 0.964 | 0.969 | 0.964 | 0.957 | 0.946
$\downarrow$ IAA | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001
$\downarrow$ $\text{IFD}_{\div}$ | 0.009 | 0.007 | 0.014 | 0.008 | 0.014 | 0.009 | 0.015 | 0.010
$\downarrow$ $\text{IFD}_{\times}$ | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
$\downarrow$ HD | 0.576 | 0.560 | 0.490 | 0.565 | 0.478 | 0.535 | 0.457 | 0.519
$\downarrow$ MME | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
$\downarrow$ II-F | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001 | 0.001
$\downarrow$ AI-F | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
ML-10M |  Rel |  $\uparrow$ HR | 0.487 | 0.443 | 0.512 | 0.386 | 0.417 | 0.387 | 0.521 | 0.402
$\uparrow$ MRR | 0.282 | 0.225 | 0.299 | 0.185 | 0.237 | 0.191 | 0.302 | 0.203
$\uparrow$ P | 0.137 | 0.105 | 0.146 | 0.088 | 0.107 | 0.096 | 0.154 | 0.094
$\uparrow$ MAP | 0.089 | 0.060 | 0.095 | 0.047 | 0.067 | 0.054 | 0.101 | 0.052
$\uparrow$ R | 0.022 | 0.018 | 0.025 | 0.012 | 0.020 | 0.016 | 0.026 | 0.013
$\uparrow$ NDCG | 0.150 | 0.113 | 0.160 | 0.092 | 0.119 | 0.100 | 0.167 | 0.100
Fair |  $\uparrow$ Jain | 0.011 | 0.027 | 0.037 | 0.115 | 0.003 | 0.006 | 0.024 | 0.069
$\uparrow$ $\text{QF}^{*}$ | 0.044 | 0.068 | 0.145 | 0.216 | 0.014 | 0.025 | 0.086 | 0.132
$\uparrow$ Ent | 0.407 | 0.514 | 0.596 | 0.716 | 0.238 | 0.324 | 0.519 | 0.638
$\uparrow$ $\text{FSat}^{*}$ | 0.044 | 0.068 | 0.145 | 0.216 | 0.014 | 0.025 | 0.086 | 0.132
$\downarrow$ Gini | 0.987 | 0.971 | 0.945 | 0.879 | 0.997 | 0.993 | 0.969 | 0.930
Fair+Rel |  $\uparrow$ IBO | 0.031 | 0.046 | 0.069 | 0.091 | 0.012 | 0.018 | 0.054 | 0.074
$\downarrow$ IWO | 0.969 | 0.954 | 0.931 | 0.909 | 0.988 | 0.982 | 0.946 | 0.926
$\downarrow$ IAA | 0.008 | 0.009 | 0.008 | 0.009 | 0.009 | 0.009 | 0.008 | 0.009
$\downarrow$ $\text{IFD}_{\div}$ | 0.018 | 0.012 | 0.019 | 0.011 | 0.016 | 0.010 | 0.020 | 0.012
$\downarrow$ $\text{IFD}_{\times}$ | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
$\downarrow$ HD | 0.221 | 0.255 | 0.226 | 0.262 | 0.265 | 0.273 | 0.218 | 0.257
$\downarrow$ MME | 0.001 | 0.001 | 0.001 | 0.001 | 0.003 | 0.001 | 0.001 | 0.001
$\downarrow$ II-F | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000
$\downarrow$ AI-F | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000 | 0.000

*For ML-10M, QF $\equiv$ FSat, as QF is computed based on the % of recommended items from all items, which in this case is equivalent to FSat.

Tab. 3 shows the scores of all Fair+Rel, Rel, and Fair measures, per dataset
and recommender/re-ranking. $\uparrow$ means the higher the score, the better,
and vice versa for $\downarrow$. Overall we observe the following.

Best model agreement. We aim to study whether the measures agree on the same
best model. We note two main trends. First, for all datasets, the best model
based on Rel measures is always different from the one based on Fair measures,
except for QF in Amazon-lb. This means that the fairest model is not
necessarily the best in terms of relevance. Second, while all Rel measures
agree on the same best model per dataset (except MRR and MAP for Amazon-lb)
and all the Fair measures always agree on the same best model (except QF), the
Fair+Rel measures disagree on the best model. Occasionally, some Fair+Rel
measures agree with another more often (e.g., IBO with IWO, or IAA with HD and
II-F, or MME with AI-F and sometimes IFD), but there is no overall
consistency. The agreement between some joint measures may be due to their
similar formulations: both IBO/IWO are the fractions of items with an impact
score greater/lower than a threshold; MME/AI-F aggregate exposure across users
prior to computing the exposure difference, while IAA/HD/II-F do not; and
MME/IFD are pairwise measures.

Range of scores. We identify three issues on the score range of the Fair+Rel
measures: (1) extremely small scales for several joint measures; (2) scale
mismatch between single-aspect measures and joint measures; and (3) scale
mismatch between joint measures. About (1), for all datasets and models,
several $\downarrow$Fair+Rel scores are extremely small ($\leq 10^{-3}$), and
these scores do not allow to distinguish across models per dataset. For
example, IFD× is always close to 0 across all datasets, as the term Eq. (7) is
often 0 due to the low number of relevant items per user.131313For all four
datasets, the median number of relevant items per user is at most 46. For MME
and II-F/AI-F, Eq. (16) and Eq. (23) often result in 0 for the same reason as
IFD×. About (2), while the above Fair+Rel scores differ in the fourth or later
decimal point, the differences in the Rel and Fair scores are in the second
decimal point or before. E.g., the NDCG (Rel score) of MultiVAE-CM and NCL for
Lastfm differs by $\sim$0.16 and their Jain (Fair score) differs by
$\sim$0.14. These examples imply non-negligible differences, but the joint
scores of IAA/IFD×/MME/II-F/AI-F only differ by $\leq 10^{-3}$, which may seem
negligible.141414We use NDCG and Jain as they are more sensitive to changes
than HR and QF. These inconsistencies in the difference of magnitude make the
scores hard to understand. About (3), we see large gaps in the score range of
all joint measures, e.g., between IWO, HD, and AI-F, despite all of them being
lower-is-better measures. E.g., in ML-10M, $\downarrow$IWO $\approx 1$ (very
unfair) based on its theoretical $[0,1]$-range, $\downarrow$HD is about a
quarter of the $\downarrow$IWO score (somewhat fair), while $\downarrow$AI-F
$\approx 0$ (extremely fair). This discrepancy causes confusion in score
interpretation.

Finally, we group all Fair+Rel measures into 3 clusters: (i) IAA /HD/II-F,
which align more with Rel measures; (ii) IFD/MME/AI-F, which align more with
Fair measures; and (iii) IBO/IWO, which do not consistently align with any
single-aspect measure. Within the same cluster, especially in (i), measures
often have large differences in their score ranges (up to $\Delta\approx
0.7$).

###  4.2. Correlation between measures (RQ1 & RQ2)

Figure 1. Kendall’s $\tau$ correlation between joint Fair+Rel measures, Rel,
and Fair measures.

We compute Kendall’s $\tau$ correlation between the orderings of the
recommenders produced by the scores of each measure, to study how much the
Fair+Rel measures agree among themselves but also with Rel-only and Fair-only
measures, when ranking recommenders (Fig. 1). As Rel and Fair measures do not
always correlate strongly with each other (Rampisela2023EvaluationStudy, 30),
we do not expect Fair+Rel measures to correlate strongly with either Rel or
Fair measures.

RQ1. Agreement of joint and single-aspect measures. Overall, there is no
consistent correlation between Rel and Fair+Rel measures. IBO/IWO’s
correlations vary wildly ( $\tau\in[-0.64,0.77]$); IAA, HD, and II-F have
moderate-to-strong positive correlations ($\tau\in[0.57,1]$); IFD and MME have
weak-to-strong negative correlations ($\tau\in[-1,-0.29]$ for IFD and
$\tau\in[-0.79,-0.14]$ for MME); and AI-F has non-positive correlations
($\tau\in[-0.64,0]$).

The correlations between Fair and Fair+Rel measures are inconsistent. The
correlations of IBO/IWO vary largely again, albeit less than with Rel
measures. IAA/HD/II-F have two distinct trends across groups of datasets: they
have negative moderate-to-strong correlations ($\tau\in[-0.79,-0.57]$) for
Lastfm and QK-video, but weak correlations for Amazon-lb and ML-10M
($\tau\in[-0.29,0.14]$). Similarly, IFD has high correlations for Lastfm and
QK-video (except with QF for QK-video), $\tau\in[0.57,0.86]$, and weak or zero
correlations for the other datasets ($\tau\in[0,0.29]$). Conversely, MME and
AI-F have strong correlations except with QF for Lastfm ($\tau\in[0.5,1]$).

Note that Fair+Rel measures strongly agreeing with Rel measures do not always
strongly disagree with Fair measures, and vice versa. E.g., IAA/HD/II-F
strongly correlates with Rel measures for Amazon-lb, but they correlate weakly
with Fair measures.

RQ2. Agreement between joint measures. Overall we find that the three clusters
of joint measures identified in $\S$4.1 show strong positive correlations
between measures inside the same cluster and strong negative correlations
between measures from different clusters. E.g., IBO always perfectly
correlates with IWO, due to their similar formulation. IAA, HD, and II-F agree
strongly with one another, $\tau\in[0.57,1]$. IFD÷ correlates highly with
IFD×, $\tau\in[0.5,1]$, as their formulations are similar. MME always agrees
strongly with AI-F, $\tau\in[0.79,0.93]$. IFD sometimes has moderate-to-strong
correlations with MME and AI-F, $\tau\in[0.43,0.86]$ for Lastfm and QK-video,
but the correlations are weaker for Amazon-lb and ML-10M,
$\tau\in[0.07,0.29]$. In contrast, IAA/HD/II-F strongly disagrees with IFD,
$\tau\in[-0.71,-0.5]$ except for IFD÷ in Amazon-lb ($\tau=-0.43$).

Based on the above, we conclude that: IBO/IWO has inconsistent relationships
with single-aspect and joint measures; IAA/HD/II-F do not align with fairness;
and IFD/MME/AI-F highly disagree with relevance (even if IFD sometimes
disagrees with Fair measures too). Among the joint measures, IBO/IWO weakly
correlate with the single-aspect measures for QK-video, and similarly with
IFD÷ for Amazon-lb, but this is not consistent. We thus argue that no joint
measures reliably account for both relevance and fairness.

###  4.3.  Measure sensitivity at different ranks (RQ3)

Figure 2. Sliding window evaluation ($\mathbf{k=5}$) of NCL for Lastfm,
Amazon-lb, and ML-10M. The last column is in exponential scale.

We now study how sensitive the joint measures are at decreasing rank
positions, compared to Rel and Fair measures. When moving down the rank, Rel
scores are known to decrease while Fair scores are known to improve
(Rampisela2023EvaluationStudy, 30). For this analysis, we use only the runs of
the non-reranked NCL model as it generally has the best Rel scores. We compute
all measures at $k=5$ for each sliding window, where the windows consist of
items at decreasing rank positions: 1–5, 2–6, $\dots$, 5–9. Fig. 2 shows the
results for Lastfm, Amazon-lb, and ML-10M, which represent the overall trends
in all our datasets; results for QK-video are shown in the appendix (in our
code repository).

We find that, as expected, as we move down the rank, Rel overall decreases and
Fair improves. However, the joint measures are notably less sensitive to
changes in rank position. Changes with decreasing rank position in the single-
aspect scores are up two magnitudes greater than in the joint measures, and
the latter do not reflect these differences to the same scale. We posit that
the insensitivity is due to the effect of changing relevance being masked by
that of fairness and vice versa. This masking makes the scores hard to
interpret. Further, the very small scores of $\downarrow$IAA, IFD×, MME, II-F,
and AI-F imply extremely fair recommendations (we explain the reasons for this
in $\S$4.1), even if Rel and Fair scores are low. Thus, these joint measures
do not account well for relevance and fairness simultaneously. Last, we note
that as we move down the rank, IAA/HD/II-F worsen, IFD/MME/AI-F improve, and
IBO/IWO are inconsistent across datasets. This follows the three groups of
joint measures discussed above.

###  4.4. Artificial insertion of items (RQ4)

Figure 3. Artificial insertion of items with $\mathbf{m=1000}$ (users).

Lastly, we study how sensitive the joint measures are to different proportions
of relevant items and item fairness in the ranking. Assessing this sensitivity
is important as it affects score interpretation; if a joint measure is
unresponsive to significant changes in relevance and fairness distribution,
its score may not reflect both the fairness and relevance of the
recommendations accurately.

We start with a recommendation list having the worst Rel and Fair scores, and
gradually insert more relevant and fair items to it (we explain ‘fair items’
below). We observe how the joint measures respond to these changes, compared
to the Rel and Fair measures.

We cannot use real-life datasets for this analysis, so we build a synthetic
dataset with $m=1000$ and $n=10000$, and artificially generate rankings of
items per user, as per (Rampisela2023EvaluationStudy, 30). The initial ranking
contains the same $k=10$ items for all users, to whom these items are
irrelevant, except for one user.151515This is to keep the number of items
exactly $km$. In each iteration, an item from the bottom of each user’s top
$k$ ranking is replaced by a relevant item having less exposure (hence more
fair). The final ranking thus contains $km$ unique items across all users,
where each item is relevant only to the user that receives that item in the
top $k$. We expect all measures to initially score the worst possible, and
then gradually improve as more relevant and fair items enter the ranks.

Fig. 3 shows the results of this analysis. Overall, we see that most joint
measures are not very sensitive to changes in Rel and Fair scores, i.e., they
may vary, but negligibly. This verifies the scale mismatch between most joint
measures and the single-aspect measures observed in $\S$4.1 & $\S$4.3. While
the overall change is negligible for most measures, a common observation
between the joint measures is that their scores become (slightly) better as
Rel and Fair scores improve. An exception to this is IFD. This is because IFD
measures fairness based on the pairwise difference in the combined value of
exposure and relevance. Thus, when the relevant items start to be moved to the
top $k$, the gap between the exposure weight of relevant items in and outside
the top $k$ increases, and so does unfairness. Among joint measures that
(slightly) improve with more insertion, there are also differences. IBO/IWO
improve linearly; as both measures are percentages of items, the change is
proportional to the amount of inserted items. HD also improves, but its
improvement fluctuates due to randomness introduced by the unstable sort in
the computation, as per the original implementation in (Jeunen2021Top-
KExposure, 17). $\downarrow$IAA/IFD×/MME/II-F/AI-F improve non-linearly.
However, their scores are extremely close to 0, i.e., on the scale of
$10^{-3}$ or less. The lower bound of the measure is 0, hence these small
scores indicate that the recommendation is close to the fairest, even at the
start of the process where the Rel and Fair scores are the worst in the entire
progression. These joint measures are also rather insensitive to changes in
Rel and Fair scores. Here, their score range is (0, 0.0015), while the range
of Rel, Fair, and IBO/IWO scores is [0,1].

##  5\. Related work

Fairness evaluation in RSs. Among prior work on fairness evaluation in RSs
(Wang2022, 38, 1, 44, 29, 30), our study is close to (Amigo2023ASystems, 1)
(Amigo2023ASystems, 1), who study RS relevance and fairness for
groups/individuals and between items and users. Yet, the focus of our work,
i.e., individual item fairness, is not covered in (Amigo2023ASystems, 1).
(Raj2022MeasuringResults, 29) (Raj2022MeasuringResults, 29) overview
evaluation measures for item group fairness. Their study includes the IAA
measure as a group fairness measure (whereas we focus on individual item
fairness). Lastly, (Rampisela2023EvaluationStudy, 30)
(Rampisela2023EvaluationStudy, 30) survey individual item fairness measures
that are exclusively linked with fairness and identify the limitations within
them, while we focus on measures that jointly account for both fairness and
relevance.

Joint measures of relevance and fairness. Outside the strict domain of
individual item fairness for RS, there exist other measures that quantify
relevance and fairness jointly: (Gao2022FAIR:Evaluation, 11)
(Gao2022FAIR:Evaluation, 11) present a measure combining KL-divergence and
IDCG to jointly quantify relevance and group fairness in IR
(Gao2022FAIR:Evaluation, 11). In (Xu2023P-MMF:System, 40), utility and
provider fairness in RSs are simultaneously evaluated with a weighted sum
between relevance and fairness. Another approach used in (Garcia-
Soriano2021Maxmin-FairConstraints, 12) to evaluate individual fairness in
ranking is to compare item position based on ground truth relevance against
its position in system-produced rankings. None of the joint measures in our
work is a combination of two single-aspect measures as in
(Gao2022FAIR:Evaluation, 11) or in the form of weighted sum as in
(Xu2023P-MMF:System, 40). The measure in (Garcia-Soriano2021Maxmin-
FairConstraints, 12) is similar to HD (Jeunen2021Top-KExposure, 17). However,
we do not use it in our work because it was not defined for RS fairness, and
considerable modifications and assumptions are required prior to using it to
evaluate RS fairness.

##  6\. Appropriate usage of joint measures

We find that joint measures of relevance and fairness (1) tend to align
differently with single-aspect measures; (2) most of them consistently score
almost perfect fairness, even when recommendations are highly irrelevant and
unfair based on single-aspect measures; and (3) are rather unresponsive to
changes in the recommendation relevance and fairness, especially compared to
single-aspect measures. Next, we suggest how to best use these joint measures.

Avoid using similar joint measures. In $\S$4.2 & $\S$4.3 we find three groups
of similar joint measures: (i) IAA/HD/II-F, (ii) IFD/MME/AI-F, and (iii)
IBO/IWO. Only one measure per group should be used. Yet, considering that
typically recommendations are evaluated with Rel measures, we discourage using
measures in (i), as they are highly aligned with Rel measures. Measures in
(ii) correlate strongly with Fair measures, and can be viable options, and
likewise for measures in (iii) that do not consistently correlate with single-
aspect measures. However, we argue that measures in (iii) are more useful than
those in (ii). Measures in (ii) can be replaced by Fair measures, which are
faster to compute and do not need complete relevance judgements, while still
achieving highly similar conclusions.

Be aware of the unintuitive or inconsistent behaviour, insensitivity, and
computational complexity of the measures. We recommend that practitioners be
aware of the measure limitations in groups (ii) and (iii). Specifically, both
IFD versions worsen with higher percentages of jointly relevant and fair
recommendations, while the opposite should happen in a healthy measure. IFD÷
is unaffected by different cut-off $k$ values, as its original formulation
only considers full rankings, so it should not be used when different $k$
matters. MME is costly to compute as it is a pairwise measure ($\sim$30 mins
for the larger datasets), and the same applies to $\downarrow$IFD×, albeit to
a lesser extent. Further, IFD×/MME/AI-F tend to have extremely small scores,
which are therefore hard to interpret and discriminate across runs. They are
also rather insensitive to changes reflected by single-aspect measures,
meaning that their overall expressiveness is limited. IBO/IWO is sensitive in
this aspect. Considering the limitations and the redundancy between measures,
IBO/IWO seem to be the most viable measure out of the existing ones, but it is
the least consistent between all other measures, due to varying alignments for
different datasets, so it should be interpreted cautiously.

Avoid score misinterpretation in measures with small empirical scales. Due to
the small empirical scales of $\downarrow$IAA/IFD/MME/II-F/AI-F, their scores
tend not to represent fairness, or relevance and fairness jointly, i.e.,
scoring very close to 0 even if the recommendation is very irrelevant and
unfair based on Rel and Fair measures. Moreover, different systems can have
very similar scores which do not translate to similar performance. E.g., two
models differing in scores by only 0.001 can be interpreted as performing the
same, even though the measure has a small empirical range and is not very
sensitive to begin with. This issue can be fixed via apriori/posthoc
normalisation based on experimental values of the measures
(Wu2022JointRecommendation, 39).

Measure fairness separately from relevance. As most joint measures
(IAA/IFD/MME/II-F/AI-F) are difficult to interpret because their scores tend
to be compressed in a very low range, and are also rather insensitive to
changes in fairness and relevance, we recommend measuring individual item
fairness and relevance separately. Otherwise, the joint scores can be close to
the theoretical fairest value even if Rel and Fair scores are low ($\S$4.4).
Overall, the above joint measures have unreliable scores, are not as sensitive
as the Fair measures, and are subject to more under/overestimation of fairness
than Fair measures which have more consistent empirical range. The remaining
joint measures are not reliable either: IBO/IWO aligns inconsistently to the
single-aspect measures, while HD is almost always consistent with Rel measures
and thus does not add another dimension of fairness measurement. It is also
unstable due to sorting of items with identical relevance level.

Overall, the joint measures cannot be compared easily as they have different
scales, and they quantify two aspects that are hard to combine due to
mismatching scales. The measures tend to correlate highly with either Rel or
Fair measures, instead of having a good balance between them. As such,
optimising for a joint measure directly may not result in a simultaneously
optimal recommendation based on Rel and Fair scores. Another obstacle in
measuring fairness is the need to consider user-item relevance in the entire
dataset (not just the recommended items), which can be an issue with extremely
sparse datasets. It is thus inherently difficult to devise a measure that can
jointly quantify relevance and fairness.

##  7\. Conclusions and future work

We presented a novel empirical study on the properties of all evaluation
measures that jointly account for individual item fairness and relevance in
recommender systems. We found that out of 9 joint measures, 3 align with
traditional relevance-only measures, 4 agree more with fairness-only measures,
and the rest behave inconsistently. We also found that only a few joint
measures are sensitive to a simultaneous decrease in relevance and increased
fairness in the recommendation. Most surprisingly, nearly all joint measures
are almost unresponsive to increases in relevance and fairness. Even worse,
the majority tend to compress scores at the low end of their range, giving the
illusion of an extremely fair recommendation, even when the relevance- and
fairness-only scores are close to the theoretical worst value. Based on these
findings, we formulated recommendations on the appropriate usage of these
measures.

Future work includes improving the design of joint measures by addressing or
mitigating the limitations of the current measures outlined, to have a single
score that reflects recommendation relevance and fairness more accurately and
in a more balanced way. The individual fairness and relevance measures can
also be optimised jointly with multi-objective approach, to obtain both fair
and relevant recommendations. Future work can also investigate whether the
findings hold when the models are directly optimised for fairness, or when
different family of models are used.

###### Acknowledgements.

Funded by Algorithms, Data & Democracy (Villum & Velux funds).

## References

  * (1) Enrique Amigó, Yashar Deldjoo, Stefano Mizzaro and Alejandro Bellogín  “A unifying and general account of fairness measurement in recommender systems”  In _Information Processing & Management_ 60.1 Pergamon, 2023, pp. 103115  DOI: 10.1016/J.IPM.2022.103115
  * (2) Asia J. Biega, Krishna P. Gummadi and Gerhard Weikum  “Equity of attention: Amortizing individual fairness in rankings”  In _41st International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR 2018_ 18 Association for Computing Machinery, Inc, 2018, pp. 405–414  DOI: 10.1145/3209978.3210063
  * (3) Rodrigo Borges and Kostas Stefanidis  “Enhancing Long Term Fairness in Recommendations with Variational Autoencoders”  In _Proceedings of the 11th International Conference on Management of Digital EcoSystems_ New York, NY, USA: ACM, 2019  DOI: 10.1145/3297662
  * (4) Amanda Bower, Hamid Eftekhari, Mikhail Yurochkin and Yuekai Sun  “Individually Fair Ranking”  In _ICLR 2021 - 9th International Conference on Learning Representations_ OpenReview.net, 2021  URL: https://openreview.net/forum?id=71zCSP_HuBN
  * (5) Iván Cantador, Peter Brusilovsky and Tsvi Kuflik  “2nd Workshop on Information Heterogeneity and Fusion in Recommender Systems (HetRec 2011)”  In _Proceedings of the 5th ACM conference on Recommender systems_ , RecSys 2011  New York, NY, USA: ACM, 2011
  * (6) Kimberle Crenshaw  “Mapping the Margins: Intersectionality, Identity Politics, and Violence against Women of Color”  In _Stanford Law Review_ 43.6 Stanford Law Review, 1991, pp. 1241–1299  URL: http://www.jstor.org/stable/1229039
  * (7) Mukund Deshpande and George Karypis  “Item-based top-N recommendation algorithms”  In _ACM Transactions on Information Systems_ 22.1 ACM, 2004, pp. 143–177  DOI: 10.1145/963770.963776
  * (8) Fernando Diaz et al.  “Evaluating Stochastic Rankings with Expected Exposure”  In _Proceedings of the 29th ACM International Conference on Information & Knowledge Management_ New York, NY, USA: ACM, 2020  DOI: 10.1145/3340531
  * (9) Cynthia Dwork et al.  “Fairness through awareness”  In _ITCS 2012 - Innovations in Theoretical Computer Science Conference_ , 2012, pp. 214–226  DOI: 10.1145/2090236.2090255
  * (10) Michael D. Ekstrand, Anubrata Das, Robin Burke and Fernando Diaz  “Fairness in Recommender Systems”  In _Recommender Systems Handbook_ , 2022, pp. 679–707  DOI: 10.1007/978-1-0716-2197-4˙18
  * (11) Ruoyuan Gao, Yingqiang Ge and Chirag Shah  “FAIR: Fairness-aware information retrieval evaluation”  In _Journal of the Association for Information Science and Technology_ 73.10 John Wiley & Sons, Ltd, 2022, pp. 1461–1473  DOI: 10.1002/ASI.24648
  * (12) David García-Soriano and Francesco Bonchi  “Maxmin-Fair Ranking: Individual Fairness under Group-Fairness Constraints”  In _Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_, KDD ’21  New York, NY, USA: Association for Computing Machinery, 2021, pp. 436–446  DOI: 10.1145/3447548.3467349
  * (13) C. Gini  “Variabilità e mutabilità”  In _Contributo allo Studio delle Distribuzioni e delle Relazioni Statistiche_ , 1912
  * (14) F Maxwell Harper and Joseph A Konstan  “The MovieLens datasets: History and context”  In _ACM Trans. Interact. Intell. Syst. 5, 4, Article_ 19, 2015  DOI: 10.1145/2827872
  * (15) Rajendra K Jain, Dah-Ming W Chiu and William R Hawe  “A Quantitative Measure Of Fairness And Discrimination For Resource Allocation In Shared Computer Systems”  In _Eastern Research Laboratory, Digital Equipment Corporation, Hudson, MA_ 21, 1998  URL: http://arxiv.org/abs/cs/9809099
  * (16) Kalervo Järvelin and Jaana Kekäläinen  “Cumulated gain-based evaluation of IR techniques”  In _ACM Transactions on Information Systems_ 20.4, 2002, pp. 422–446  DOI: 10.1145/582415.582418
  * (17) Olivier Jeunen and Bart Goethals  “Top-K Contextual Bandits with Equity of Exposure”  In _Proceedings of the 15th ACM Conference on Recommender Systems_ , RecSys ’21  New York, NY, USA: Association for Computing Machinery, 2021, pp. 310–320  DOI: 10.1145/3460231.3474248
  * (18) Tomo Lazovich et al.  “Measuring disparate outcomes of content recommendation algorithms with distributional inequality metrics”  In _Patterns_ 3.8 Elsevier, 2022  DOI: 10.1016/j.patter.2022.100568
  * (19) Joon Ho Lee  “Analyses of multiple evidence combination”  In _SIGIR ’97: Proceedings of the 20th annual international ACM SIGIR conference on Research and development in information retrieval_ Philadelphia: Association for Computing Machinery (ACM), 1997, pp. 267–276  DOI: 10.1145/258525.258587
  * (20) Dawen Liang, Rahul G. Krishnan, Matthew D. Hoffman and Tony Jebara  “Variational autoencoders for collaborative filtering”  In _The Web Conference 2018 - Proceedings of the World Wide Web Conference, WWW 2018_ 10 Association for Computing Machinery, Inc, 2018, pp. 689–698  DOI: 10.1145/3178876.3186150
  * (21) Zihan Lin, Changxin Tian, Yupeng Hou and Wayne Xin Zhao  “Improving Graph Collaborative Filtering with Neighborhood-enriched Contrastive Learning”  In _WWW 2022 - Proceedings of the ACM Web Conference 2022_ Virtual Event, Lyon, France: Association for Computing Machinery, Inc, 2022, pp. 2320–2329  DOI: 10.1145/3485447.3512104
  * (22) Masoud Mansoury et al.  “FairMatch: A Graph-based Approach for Improving Aggregate Diversity in Recommender Systems”  In _UMAP 2020 - Proceedings of the 28th ACM Conference on User Modeling, Adaptation and Personalization_ 20 ACM, 2020, pp. 154–162  DOI: 10.1145/3340631.3394860
  * (23) Zaiqiao Meng, Richard McCreadie, Craig MacDonald and Iadh Ounis  “Exploring Data Splitting Strategies for the Evaluation of Recommendation Models”  In _RecSys 2020 - 14th ACM Conference on Recommender Systems_ Virtual Event, Brazil: Association for Computing Machinery, Inc, 2020, pp. 681–686  DOI: 10.1145/3383313.3418479
  * (24) Alistair Moffat and Justin Zobel  “Rank-biased precision for measurement of retrieval effectiveness”  In _ACM Transactions on Information Systems_ 27.1 ACM, 2008  DOI: 10.1145/1416950.1416952
  * (25) Jianmo Ni, Jiacheng Li and Julian McAuley  “Justifying Recommendations using Distantly-Labeled Reviews and Fine-Grained Aspects”  In _EMNLP-IJCNLP 2019 - 2019 Conference on Empirical Methods in Natural Language Processing and 9th International Joint Conference on Natural Language Processing, Proceedings of the Conference_ Association for Computational Linguistics, 2019, pp. 188–197  DOI: 10.18653/V1/D19-1018
  * (26) Harrie Oosterhuis  “Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairness”  In _Proceedings of the 44th International ACM SIGIR Conference on Research and Development in Information Retrieval_ , SIGIR ’21  New York, NY, USA: Association for Computing Machinery, 2021, pp. 1023–1032  DOI: 10.1145/3404835.3462830
  * (27) Gourab K Patro et al.  “Fair Ranking: A Critical Review, Challenges, and Future Directions”  In _Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency_ , FAccT ’22  New York, NY, USA: Association for Computing Machinery, 2022, pp. 1929–1942  DOI: 10.1145/3531146.3533238
  * (28) Gourab K. Patro et al.  “FairRec: Two-Sided Fairness for Personalized Recommendations in Two-Sided Platforms”  In _The Web Conference 2020 - Proceedings of the World Wide Web Conference, WWW 2020_ Association for Computing Machinery, Inc, 2020, pp. 1194–1204  DOI: 10.1145/3366423.3380196
  * (29) Amifa Raj and Michael D. Ekstrand  “Measuring Fairness in Ranked Results”  In _Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ New York, NY, USA: ACM, 2022, pp. 726–736  DOI: 10.1145/3477495.3532018
  * (30) Theresia Veronika Rampisela, Maria Maistro, Tuukka Ruotsalo and Christina Lioma  “Evaluation Measures of Individual Item Fairness for Recommender Systems: A Critical Study”  In _ACM Trans. Recomm. Syst._ New York, NY, USA: Association for Computing Machinery, 2023  DOI: 10.1145/3631943
  * (31) Steffen Rendle, Christoph Freudenthaler, Zeno Gantner and Lars Schmidt-Thieme  “BPR: Bayesian Personalized Ranking from Implicit Feedback”  In _Proceedings of the Twenty-Fifth Conference on Uncertainty in Artificial Intelligence_ , UAI ’09  Arlington, Virginia, USA: AUAI Press, 2009, pp. 452–461
  * (32) Yuta Saito and Thorsten Joachims  “Fair Ranking as Fair Division: Impact-Based Individual Fairness in Ranking”  In _Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining (KDD ’22), August 14-18, 2022, Washington, DC, USA_ 1 ACM, 2022, pp. 1514–1524  DOI: 10.1145/3534678.3539353
  * (33) C E Shannon  “A Mathematical Theory of Communication”  In _The Bell System Technical Journal_ 27, 1948, pp. 623–656
  * (34) Ashudeep Singh and Thorsten Joachims  “Policy learning for fairness in ranking”  In _Advances in Neural Information Processing Systems_ 32, 2019, pp. 5427–5437  URL: https://proceedings.neurips.cc/paper/2019/hash/9e82757e9a1c12cb710ad680db11f6f1-Abstract.html
  * (35) Jessie J Smith, Lex Beattie and Henriette Cramer  “Scoping Fairness Objectives and Identifying Fairness Metrics for Recommender Systems: The Practitioners’ Perspective”  In _Proceedings of the ACM Web Conference 2023_ , WWW ’23  New York, NY, USA: Association for Computing Machinery, 2023, pp. 3648–3659  DOI: 10.1145/3543507.3583204
  * (36) Leonidas Tsepenekas, Ivan Brugere, Freddy Lecue and Daniele Magazzeni  “Comparing Apples to Oranges: Learning Similarity Functions for Data Produced by Different Distributions”  In _Thirty-seventh Conference on Neural Information Processing Systems_ , 2023  URL: http://papers.nips.cc/paper_files/paper/2023/hash/59056767478c7df64e6250eadfeb0a04-Abstract-Conference.html%20https://openreview.net/forum?id=zjpjsJeVJZ
  * (37) Xiuling Wang and Wendy Hui Wang  “Providing Item-side Individual Fairness for Deep Recommender Systems”  In _ACM International Conference Proceeding Series_ 22 Association for Computing Machinery, 2022, pp. 117–127  DOI: 10.1145/3531146.3533079
  * (38) Yifan Wang et al.  “A Survey on the Fairness of Recommender Systems”  In _ACM Trans. Inf. Syst._ 41.3 New York, NY, USA: Association for Computing Machinery, 2023, pp. 1–43  DOI: 10.1145/3547333
  * (39) Haolun Wu et al.  “Joint Multisided Exposure Fairness for Recommendation”  In _SIGIR 2022 - Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval_ Association for Computing Machinery, Inc, 2022, pp. 703–714  DOI: 10.1145/3477495.3532007
  * (40) Chen Xu et al.  “P-MMF: Provider Max-min Fairness Re-ranking in Recommender System”  In _ACM Web Conference 2023 - Proceedings of the World Wide Web Conference, WWW 2023_ Association for Computing Machinery, Inc, 2023, pp. 3701–3711  DOI: 10.1145/3543507.3583296
  * (41) Tao Yang, Zhichao Xu, Zhenduo Wang and Qingyao Ai  “FARA: Future-Aware Ranking Algorithm for Fairness Optimization”  In _Proceedings of the 32nd ACM International Conference on Information and Knowledge Management_ , CIKM ’23  New York, NY, USA: Association for Computing Machinery, 2023, pp. 2906–2916  DOI: 10.1145/3583780.3614877
  * (42) Tao Yang et al.  “Marginal-Certainty-Aware Fair Ranking Algorithm”  In _Proceedings of the Sixteenth ACM International Conference on Web Search and Data Mining_ , WSDM ’23  New York, NY, USA: Association for Computing Machinery, 2023, pp. 24–32  DOI: 10.1145/3539597.3570474
  * (43) Guanghu Yuan et al.  “Tenrec: A Large-scale Multipurpose Benchmark Dataset for Recommender Systems”  In _Advances in Neural Information Processing Systems_ 35, 2022, pp. 11480–11493  URL: https://www.tencent.com/en-us/
  * (44) Meike Zehlike, Ke Yang and Julia Stoyanovich  “Fairness in Ranking, Part II: Learning-to-Rank and Recommender Systems”  In _ACM Computing Surveys_ 55.6 ACM PUB27 New York, NY, 2022  DOI: 10.1145/3533380
  * (45) Wayne Xin Zhao et al.  “RecBole: Towards a Unified, Comprehensive and Efficient Framework for Recommendation Algorithms”  In _International Conference on Information and Knowledge Management, Proceedings_ New York, NY, USA: ACM, 2021, pp. 4653–4664  DOI: 10.1145/3459637.3482016
  * (46) Qiliang Zhu, Qibo Sun, Zengxiang Li and Shangguang Wang  “FARM: A Fairness-Aware Recommendation Method for High Visibility and Low Visibility Mobile APPs”  In _IEEE Access_ 8 Institute of ElectricalElectronics Engineers Inc., 2020, pp. 122747–122756  DOI: 10.1109/ACCESS.2020.3007617