Copyright for this paper by its authors. Use permitted under Creative Commons
License Attribution 4.0 International (CC BY 4.0).

Preprint under peer-review.

[orcid=0000-0001-6747-3898<EMAIL_ADDRESS>]

[orcid=0000-0003-1792-1706<EMAIL_ADDRESS>]

[orcid=0000-0001-6368-2510<EMAIL_ADDRESS>]

# Understanding the Influence of Data Characteristics on the Performance of
Point-of-Interest Recommendation Algorithms

Linus W. Dietz  King’s College London, UK  Pablo Sánchez  Instituto de
Investigación Tecnológica (IIT), Universidad Pontificia Comillas, Spain
Alejandro Bellogín  Universidad Autónoma de Madrid, Spain

(2023)

###### Abstract

The performance of recommendation algorithms is closely tied to key
characteristics of the data sets they use, such as sparsity, popularity bias,
and preference distributions. In this paper, we conduct a comprehensive
explanatory analysis to shed light on the impact of a broad range of data
characteristics within the point-of-interest (POI) recommendation domain. To
accomplish this, we extend prior methodologies used to characterize
traditional recommendation problems by introducing new explanatory variables
specifically relevant to POI recommendation. We subdivide a POI recommendation
data set on New York City into domain-driven subsamples to measure the effect
of varying these characteristics on different state-of-the-art POI
recommendation algorithms in terms of accuracy, novelty, and item exposure.
Our findings, obtained through the application of an explanatory framework
employing multiple-regression models, reveal that the relevant independent
variables encompass all categories of data characteristics and account for as
much as $R^{2}=$ 85–90% of the accuracy and item exposure achieved by the
algorithms. Our study reaffirms the pivotal role of prominent data
characteristics, such as density, popularity bias, and the distribution of
check-ins in POI recommendation. Additionally, we unveil novel factors, such
as the proximity of user activity to the city center and the duration of user
activity. In summary, our work reveals why certain POI recommendation
algorithms excel in specific recommendation problems and, conversely, offers
practical insights into which data characteristics should be modified (or
explicitly recognized) to achieve better performance.

###### keywords:

point-of-interest recommendation offline evaluation regression analysis

UI

    user interface
LBSN

    location-based social network
RS

    recommender system
POI

    point of interest
EV

    explanatory variable
URM

    user-rating matrix
UCM

    user-check-in matrix
VIF

    variance inflation factor

##  1 Introduction

Understanding which recommendation algorithm is most effective for a specific
data set is crucial, as it has been widely acknowledged that no single
recommender can achieve optimal performance in all scenarios (Im and Hars,
2007, Anelli et al., 2022). Apart from the performance variation of the
algorithms across different data sets, it should be taken into account that
the quality of any recommender can be evaluated through a wide array of
different dimensions (Gunawardana and Shani, 2015). While accuracy may be the
primary concern when recommending items that a user might actually consume, it
is equally important to prioritize recommending different items for the users
(diversity), items that may be unfamiliar to users and, hence, surprise them
(novelty), or ensure that our recommendations are not biased towards
individual users (fairness) (Castells et al., 2015, Ekstrand et al., 2022).
However, designing models that perform well in all these (and more) dimensions
is challenging, as they need to deal with, for example, the accuracy-diversity
trade-off (Isufi et al., 2021).

Whereas such analysis of accuracy versus novelty, diversity, and other
dimensions has been conducted in traditional recommendation scenarios like
movies or books, within the point-of-interest recommendation domain, the
problem has so far not been fully analyzed. Besides, while existing research
has primarily focused on evaluating the quality of recommendations based on
the number of recommended venues that the users have visited in the test set
(i.e., using ranking accuracy metrics like Precision or Recall), there remains
a lack of consensus on the other crucial aspects of evaluation methodology,
such as data sets, data filtering, data partitioning, and other evaluation
metrics (Sánchez and Bellogín, 2022). In addition, it is important to consider
that users can be grouped into different clusters based on both the items they
consume and the patterns they exhibit while interacting with them.
Specifically, in the point-of-interest recommendation domain, these groups can
be established considering domain-specific variables, such as geographical
influence or the categories of visited POIs. In fact, in this domain, the
recommendation performance may fluctuate substantially depending on the user
group a user belongs to, especially between local users and visiting tourists
(Sánchez and Dietz, 2022).

Hence, in this paper, we investigate the success factors of classical and
point of interest (POI) recommendation algorithms through the lens of data
characteristics of a POI recommendation data set. Previously, Deldjoo et al.
(2021) proposed a method to analyze the impact of different data
characteristics on the accuracy and fairness of matrix factorization
algorithms in the movie and book recommendation domains. As opposed to such
classical recommendation problems, POI recommendations are subject to further
effects, such as seasonality, geographical influences of the venues to be
recommended, and the type of users of such system (Sánchez and Bellogín,
2022). In consequence, we incorporate some of these influences in our
framework derived from Deldjoo et al. (2021) to analyze the effect they have
on the performance of a set of recommenders.

###  1.1 Data Characteristics & Recommender Performance

When proposing a new recommendation model, researchers often rely on
intuitions or anecdotal evidence to justify why their model is effective.
However, this approach of defending the quality of recommendations is not
accurate enough and may be the reason why some models obtain excellent results
on some data sets while, at the same time, they perform poorly in others, as
recent efforts on reproducibility have demonstrated (Dacrema et al., 2019,
Said and Bellogín, 2014).

It should be noted that some studies have analyzed the effect of different
aspects in the recommendations, like the data partitioning (Meng et al., 2020)
and the hyperparameters of the models (Anelli et al., 2019). However, this
research is based on the approach presented by the works of Deldjoo et al.
(2021, 2020), where the authors defined a set of explanatory variables that
model the characteristics of the data set (e.g., ratings per user, per item,
population bias, etc.). On the basis of those works, herein, we use the same
explanatory variables as those proposed by Deldjoo et al. (2021). However, we
extend our analysis by incorporating additional variables that capture the
unique dynamics of the POI recommendation domain. In our research, we do not
only study the impact of our explanatory variables in terms of ranking
quality, measured by nDCG, but we also analyze the effect of the
recommendations in terms of both novelty and item exposure, i.e., by comparing
that the number of times that each POI is being recommended is similar to the
number of times that POI is expected to be visited (according to the test
set).

Hence, in this work, we perform a similar approach as the one presented in
Deldjoo et al. (2021) to explain the recommendation outcome in the POI
recommendation domain through the lens of data characteristics. We start from
a check-in data set based on a popular Location-Based Social Network
(Foursquare) and generate domain-driven subsamples: each subsample contains
POI visits based on novel subsampling rules specifically created for this
work. We then extend how this approach was handled in other scenarios,
exploiting inherent attributes and characteristics of the POI recommendation
domain. As we shall show throughout the paper, some of these rules are based
on the domain, e.g., the origin of the user and the travel season. In
contrast, others are based on typical measures that are undertaken to improve
the recommendation, such as removing popular venues or enforcing a k-core,
which removes inactive users and infrequently visited venues.

###  1.2 Impact on e-Tourism

This research can have significant implications for the tourism industry. POI
recommendations play a crucial role in shaping tourists’ experiences and
guiding their choices of which places to visit; thus, they are a key factor in
decision-making. We offer valuable insights that can enhance the tourism
industry’s ability to provide more tailored and satisfying experiences to
travelers by determining which recommendation algorithms should be used in
which The analysis of the beyond-accuracy metrics, i.e., novelty and item
exposure, offers valuable perspectives that cater to the user needs and the
community. Novelty is especially relevant to local users, as it fosters the
exploration of their city, whereas item exposure is important to ensure that
the flow of visitors is dissipated on many venues. By harnessing data
characteristics such as density, popularity bias, and user activity duration,
tourism stakeholders can fine-tune their recommendations to align with users’
preferences and expectations. Moreover, the identification of novel factors,
such as the distance of user activity to city centers, highlights
opportunities for businesses to understand the impact of their location on the
activity of certain users. Thus, understanding how user behavior varies in
different parts of a destination can enable businesses to tailor their
offerings and marketing efforts more effectively. Our research underscores the
value of data-driven decision-making in the tourism sector. By leveraging the
insights gained from this study, both providers of POI recommendation
platforms and the tourism industry can enhance their ability to provide
personalized and engaging experiences to users.

###  1.3 Contributions

We extend the previous approaches of Deldjoo et al. (2021) and Adomavicius and
Zhang (2012) in several aspects in the POI recommendation domain. More
precisely, the main contributions of this paper are the following:

  1. 1.

We introduce a domain-specific subsampling algorithm for POI recommendation.
This algorithm ensures that the simulated data sets are realistic and grounded
in the domain instead of random subsampling, as done in previous works.

  2. 2.

We extend the corpus of explanatory variables for analyzing the effect of
different data characteristics, including geographic aspects, in a varied set
of state-of-the-art POI recommendation algorithms. This analysis considers
three complementary evaluation dimensions: ranking accuracy, novelty, and item
exposure.

  3. 3.

We perform a comprehensive analysis of a set of recommendation algorithms by
considering $144$ different simulations. Each simulation corresponds to a
subsampled recommendation data set of a specific city within the Foursquare
check-in data set. In this way, we conduct an analysis of different samples
with disparate characteristics to detect which explanatory variables help us
to better explain the performance of the recommenders.

###  1.4 Overview of the Proposed Methodology

To further analyze the aforementioned considerations in the POI recommendation
domain, we undertake the following steps:

  1. 1.

We provide an overview of existing explanatory variables to quantify data
characteristics in recommender systems (Section 3.2) and augment them with
further explanatory variables relevant to the POI recommendation domain
(Section 3.2.4).

  2. 2.

We propose a method to create domain-specific subsamples with different data
characteristics (Section 4).

  3. 3.

Having established the theory, we perform the following steps in our
experiments (Section 5):

    1. (a)

We determine suitable data sets to analyze this problem.

    2. (b)

We systematically record the recommendation outcomes of the different
subsamples with a representative collection of recommendation models.

    3. (c)

We identify the relevant explanatory variables.

    4. (d)

We perform a regression analysis of the subsamples’ data characteristics and
recommendation metrics.

  4. 4.

Finally, we disseminate the results of this regression analysis to understand
the impact of the data characteristics on the recommendation outcome (Section
6).

##  2 Background

###  2.1 Point-of-Interest Recommendation

The point-of-interest (POI) recommendation problem is typically defined as
recommending new venues to visit to the users when they are in a city or
particular region (Sánchez and Bellogín, 2022). These venues have a specific
location, typically expressed as latitude and longitude, and might be varied
in nature, including museums, parks, restaurants, or bars (among others). As
in the traditional recommendation scenario, the objective is to maximize the
number of relevant items (in this case, venues) that are being recommended to
the user.

Formally, as pointed in a recent survey by Sánchez and Bellogín (2022), the
POI recommendation problem can be formulated as follows:

$p^{*}(u)=\arg\max_{p\in\mathcal{P}}g(u,p,C)$ (1)

where $\mathcal{P}$ denotes all POIs available in the region, and $C$
represents the set of influences, also referred to as contextual information
in some works (Adomavicius and Tuzhilin, 2015). This contextual information
should be considered to perform meaningful POI recommendations (Manotumruksa
et al., 2018).

Temporal, sequential, social, categorical, and, most importantly, geographical
information are normally exploited in most POI recommendation approaches (Li
et al., 2015, Griesner et al., 2015, Zhang and Chow, 2015, Liu et al., 2014).
In order to perform POI recommendations, researchers may use the information
available in location-based social networks. Foursquare, Gowalla, or Yelp are
examples of this type of applications where users are allowed to register the
check-ins they perform at the different venues they visit and, in some cases,
to establish social links with other registered users. These friendship links,
along with other information like the geographical coordinates of the venues,
their categories, and the timestamps of the check-ins, can be used to model
the aforementioned influences and generate potentially interesting
recommendations to the users who are new in a specific geographical region.

It is important to note that LBSNs typically contain check-ins from different
cities around the world. However, as this type of recommendation is affected
by all the aforementioned influences (especially the geographical
information), many researchers perform recommendations considering each
city/region as independent data sets (Liu et al., 2014, Li et al., 2015). This
strategy is not only practical from an experimental point of view, but it is
also quite reasonable since when a user is in a particular city, she will be
interested in visiting venues belonging to that region and not from distant
cities. Furthermore, each city may exhibit a unique distribution of POIs to
visit, along with distinct cultural characteristics and specific urban
planning. In fact, this is one of the reasons why POI recommendation is
closely related to the tourism industry (Wang et al., 2020, Santos et al.,
2019), since tourists, when they arrive to a new city, are usually interested
in visiting the most relevant venues of that specific city and immerse
themselves in the local culture. Besides, we need to consider that tourism is
the base of many economies due to the large number of stakeholders involved
(including tourists, venue owners, and local residents), such as some
countries in southern Europe (Cortés-Jiménez, 2008).

However, despite the fact that there is a large number of works linking the
POI recommendation problem with tourism, this aspect is not considered when
modeling the users. In fact, in previous studies, it is observed that most
users from LBSNs should actually be considered as “locals” and not “tourists”
in those cities since most of their check-ins belong to the same city (Sánchez
and Dietz, 2022). At the same time, there are works that use data from other
applications not directly related to tourism – such as Flickr or Instagram –
in order to analyze the behavior of tourists (Brilhante et al., 2013, Domènech
et al., 2020). In fact, such approaches also present some limitations.
Typically, user trajectories are reconstructed by leveraging the photos taken
on these platforms – where accurate location data associated with these photos
is not always guaranteed, since users may choose not to enable location data
when uploading photos, or the location data provided may be imprecise or
inaccurate. Furthermore, while a photo may capture a specific location, it
does not necessarily provide a comprehensive depiction of the actual path
taken to reach that location. Therefore, relying solely on photo data for
trajectory reconstruction may result in incomplete or unreliable
representations of user movement.

###  2.2 Specific Considerations of the POI Recommendation Problem

When addressing the POI recommendation problem, several domain-specific
considerations and problems need to be regarded. Some of them include:

##### Sparsity:

The ratio between the stored preferences in the data set and all the possible
interactions between the users and the venues is extremely high. While the
densities of LBSNs data sets from Foursquare and Gowalla are approximately
0.0034%, the density of the Netflix and Movielens20M data sets (typically used
in classical recommendation) are around 1.77%.

##### Additional influences:

As discussed in Section 2.1, POI recommendation is significantly impacted by
various factors, including geographical coordinates of visited venues, social
connections with other users, and temporal information (such as the timestamp
of check-ins). When data sparsity is a critical concern, leveraging additional
information to enhance the algorithm’s performance is crucial. Among all the
information sources, geographical influence plays the most important role
since users often prefer to visit nearby POIs. However, temporal influence can
also provide valuable insights, such as the duration of users’ visits and
their movement patterns between POIs. Therefore, exploiting all available
information sources is essential to improve the accuracy and effectiveness of
POI recommendation algorithms.

##### Implicit information:

Traditionally, in the classical recommendation problem, the information
encoded in the user-item interaction matrix has been modeled using ratings.
However, in most POI recommendation data sets, such as Brightkite, Gowalla, or
Foursquare, we only have information about the specific moment in time when a
user visited a POI. In fact, users may have checked in more than once at the
same POI, something not considered (or ignored) in classical recommendation
(Ning et al., 2015). To model these repeated preferences, researchers have
developed frequency matrices, where each entry represents the number of times
a user checked in at a venue. Therefore, in POI recommendation, implicit
information is more commonly used as compared to explicit ratings-based
information in classical recommender systems. By leveraging these frequency
matrices, POI recommender systems can capture latent user preferences and
provide better recommendations.

##### Popularity bias:

Popularity bias is a well-studied problem in the recommender systems domain
that occurs when popular items are recommended more frequently than less
popular ones, regardless of whether they actually match the interests of the
target user (Abdollahpouri et al., 2019). The effect of popularity bias is
evident in multiple layers within the context of POI recommendation. Firstly,
at the city level, an analysis of the original Foursquare data set reveals
that out of the 415 cities worldwide, the top 1% of the most popular cities
(based on the highest number of check-ins) represent the 20% of the total
recommendationscheck-ins in the data set. However, when considering the top 2%
of the most popular cities, this percentage increases to 28% of the total
check-ins. Additionally, within each specific city, we can observe the impact
of popularity bias on individual POIs. Taking New York City and Tokyo as
examples, two extensively studied cities in the Foursquare data set, we find
that in New York City, the top 1% of the most popular venues are responsible
for 27% of all check-ins, while the top 2% of venues account for 36% of the
total check-ins. Similarly, in Tokyo, the top 1% of venues encompass 48% of
all check-ins, and the top 2% represent 57% of the check-ins in the city.

###  2.3 Offline Evaluation in Point-Of-Interest Recommendation

When evaluating POI recommendation approaches using an offline evaluation
methodology, most works follow the same evaluation protocols reported in the
traditional recommendation scenario. Firstly, we start with a complete data
set containing all the check-ins between the users and the venues. From this
full data set, we obtain a subset to evaluate the performance of the models
(this set is known as the test set or ground truth) and leave the rest of the
check-ins in the training set. Sometimes we might also create an additional
subset that will be used to tune the parameters of the models (validation
set). We use this training set to train our recommendation model(s) and then,
for every user that appears in the test set, we generate a list of
recommendations ordered according to the predicted (estimated) satisfaction
each item would provide to the user, also known as the top-N recommendation
task (Cremonesi et al., 2010).

A large number of metrics have been proposed to evaluate the quality of the
recommendation algorithms, although the most common ones are focused on
measuring the accuracy of the recommendations (i.e., how many matches are
between the recommended venues and the venues visited by the user in the test
set). Thus, the more venues appearing in both lists, the better the quality of
the recommender.

For generating the different subsets, as in classical Machine Learning, it is
common to perform random splits with training, validation, and test or cross-
validation from the original data set (Cheng et al., 2016, Wang et al., 2020).
However, in recent years, many researchers have taken into account the
temporal dimension when performing these splits (Huang et al., 2020, Zhao et
al., 2019). In this sense, it is common to find two main types of temporal
splits: per user (where the $n$ oldest interactions for each user are selected
for training and the rest for validation and testing) and a global split,
where a specific timestamp is selected so that all the interactions that
occurred before that timestamp are selected for training and the rest for
validation and test (Sánchez and Bellogín, 2022). This second type is more
natural, as it mimics how a recommender system would be evaluated at a
production scale, while it also avoids data leaking to the test (Ji et al.,
2022).

As mentioned before, to analyze the quality of the recommendations produced in
both traditional and POI recommendation, most researchers report ranking
accuracy metrics oriented to measure the relevance of the recommendations,
such as Precision, Recall, or nDCG (Gunawardana and Shani, 2015). However, the
community is aware that focusing too much on recommending items that appear in
the ground truth may disregard other important evaluation dimensions for the
users, such as novelty (recommending items that are not so popular), diversity
(recommending items that are different), or serendipity (recommending items
that are novel and not easy to discover) (Castells et al., 2015). On top of
that, as the POI recommendation domain is affected by categorical,
geographical, and social influences, some researchers also use additional
metrics to consider these aspects in the evaluation. For example, Zhao et al.
(2015) used a category-level accuracy metric to measure the quality of the
recommendations, whereas Yin et al. (2015) incorporated the error made in the
geographical distance between the recommended POIs and those actually visited
in the test set by the users.

Another aspect that has received considerable attention in recent years in the
recommender systems community, as in other areas of Artificial Intelligence
and Machine Learning, is trying to understand the inherent biases learned by
these systems and how they get reinforced by the recommendations. Thus, many
researchers have focused on analyzing potential biases that may be found in
either the data sets or the recommendations produced by the models. These
biases vary widely, ranging from gender (Ekstrand and Kluver, 2021, Melchiorre
et al., 2021) to popularity bias (Abdollahpouri et al., 2019, Cañamares and
Castells, 2017). Moreover, whereas in the classical recommendation scenario it
is (currently) established that analyzing different types of biases is
important, the POI recommendation domain appears to lack comprehensive
exploration in this regard. To the best of our knowledge, only a limited
number of studies have analyzed this aspect. For example, Sánchez and Dietz
(2022) observed biases in the recommendations provided to different groups of
tourists and locals; and Weydemann et al. (2019) studied three types of
fairness in this domain. First, they analyzed the fairness regarding the
popularity of the venues, examining wether the recommender amplifies the
popularity bias. Then, they investigated fairness with respect to the
nationalities of the users by checking if the users received similar
recommendations first, and then, if each user group follows the same
distribution exhibited before. Finally, they extended the analysis to location
categories, by assessing if the recommendations are aligned with the category
distribution observed in the previous visits.

##  3 Explanatory Framework for POI Recommendation

The overall goal of this work is to understand which factors influence the
performance of different recommendation models in the POI recommendation
domain in terms of different evaluation dimensions such as accuracy and
fairness. Similar to previous approaches addressing this challenge
(Adomavicius and Zhang, 2012, Deldjoo et al., 2021), we use data
characteristics to describe subsamples of a recommendation data set and use a
regression model to capture the impact of each feature on the recommendation
outcome. The idea behind analyzing different subsamples of the same data set
is that the regression analysis would reveal the influence of variations in
data characteristics on dependent variables. The main difference to previous
studies is that our analysis is about POI recommendation, which enables us to
define further explanatory variables capturing the geographical influences of
users visiting venues in a city. Furthermore, we are able to use a domain-
driven subsampling approach, which yields additional insights into the
performance of recommenders in realistic sub-groups.

In this section, we describe the explanatory framework, which is a regression
model applied to a series of metrics – data characteristics – capturing the
interactions of users with the respective venues in a city. These data
characteristics are computed for each subsample independently using a domain-
driven approach outlined in Section 4.

###  3.1 Regression Model

Given all subsamples, we aim to model the relationship between the data
characteristics and the recommendation performance of each individual
recommendation model. This allows us to test hypotheses regarding which
explanatory variables are able to describe the variations in the dependent
variables. Equation 2 shows the regression model, which is the core of the
explanatory analysis.

$y=\epsilon+\theta_{0}+\sum_{ev\in EV}\theta_{ev}x_{ev}$ (2)

where $\theta_{ev}$ is the regression coefficient of the respective
explanatory variable $ev$ (among the set of variables $EV$), $x_{ev}$
represents the value of the explanatory variable in the current training
example (current training data set), and $y$ is the value of the dependent
variable according to the recommendation models. In particular, this means
that, for a specific recommender system $r$, the dependent variables will be
potentially different for each $r$ (i.e., its performance value $y$ would be,
in fact, $y^{r}$). Then, the performance would be modeled upon the set of
explanatory variables $x_{ev}$, that will depend on the characteristics of the
data set (as we shall explain later). Based on this, the regression model will
produce coefficients $\theta_{k}^{r}$ specifically tailored for this
particular recommender, as it will consider the explanatory variables and the
dependent variables at the same time.

###  3.2 Explanatory Variables

In this section, we list all explanatory variables which serve as independent
variables in the regression analysis. Note that, opposed to the approaches of
Adomavicius and Zhang (2012) and Deldjoo et al. (2021), we do not have a user-
rating matrix, but a user-check-in matrix, since the interaction between the
users and POI is a visit and not a rating. While the user-check-in matrix
(UCM) is conceptually very similar to a user-rating matrix (URM), the UCM is
established based on unique visits, i.e., the cell values of the UCM are $1$
if a user has visited a venue; otherwise, it is $0$. Multiple visits of one
user to the same venue also result in a value of $1$. Despite this small
conceptual difference, most EVs proposed and used by Adomavicius and Zhang
(2012) and Deldjoo et al. (2021) can also be computed for a UCM. Since there
is a significant geographic influence in POI recommendation (Li et al., 2015),
we also propose some further EVs that capture such geographic information
about the visited venues. Thus, the EVs we use can be categorized in the
following four categories:

  1. 1.

EVs that describe the structure of the UCM.

  2. 2.

EVs that describe the check-in distribution of the UCM.

  3. 3.

EVs that are based on item and user properties in the UCM.

  4. 4.

EVs that capture the underlying user activity and mobility.

####  3.2.1 EVs Based on the Structure of the UCM

These EVs capture the general structure of the UCM and are well established to
describe properties of recommendation data sets. Thus, we keep the discussion
around them rather succinct.

###### Definition 1

(SpaceSize) Given a UCM, SpaceSize is defined as:

$ev_{1}=SpaceSize(UCM)=|U|\cdot|I|$ (3)

We use the SpaceSize instead of its components, the number of users, $|U|$,
and the number of items $|I|$ since it reduces the number of variables.

###### Definition 2

(Shape) Given a UCM, Shape is defined as:

$ev_{2}=Shape(UCM)=\frac{|U|}{|I|}$ (4)

The ratio between the number of users and the number of items can be an
initial indicator of whether user-based collaborative filtering or item-based
collaborative filtering approaches might be more successful (Nikolakopoulos et
al., 2022).

###### Definition 3

(Density) Given a UCM, Density is defined as:

$ev_{3}=Density(UCM)=\frac{|C|}{|U|\times|I|}$ (5)

Density, or its inverse, sparsity, is a commonly reported metric to give an
estimation of the recommendation difficulty for collaborative recommendation
algorithms. Here we use $C$ to refer to all the check-ins performed by the
users and registered in a data set. Generally, the higher the density, the
more signal is available for the algorithm to compute fitting recommendations.
Density typically varies a lot depending on the data set and the domain, as
mentioned in Section 2.2.

###### Definition 4

($Cp_{u},Cp_{i}$) Given a UCM, check-ins per user ($Cp_{u}$) and check-ins per
item ($Cp_{i}$) are defined as:

$ev_{4}=Cp_{u}(UCM)=\frac{|C|}{|U|}$ (6) $ev_{5}=Cp_{i}(UCM)=\frac{|C|}{|I|}$
(7)

The number of interactions per user/item is also a simple but effective
measure to put the recommendation quality into perspective. If the number of
interactions is remarkably small for a user or an item, it can be regarded as
“cold”, indicating that there is not sufficient information to compute
meaningful recommendations. Given the high sparsity of the data sets in the
POI recommendation domain, it is very common to impose a minimum number of
interactions for both items and users, i.e., enforcing a $k$-core, cf. Section
4.1. This is done to avoid evaluating cold-start recommendations.

####  3.2.2 EVs Based on the Check-in Distribution of the UCM

Naturally, some users are more active than others, and not all items get the
same attention. In the POI recommendation domain, this is a very natural
phenomenon since major highlights inherently attract more visits. This is a
challenge for the fairness of the recommendations, as there is a delicate
trade-off between recommending popular POI and items in the long-tail (Rahmani
et al., 2022).

###### Definition 5

($Gini_{u},Gini_{i}$) Given a UCM, $|C_{u}|$ and $|C_{i}|$ be the number of
check-ins associated with user $u$ and item $i$ and the users/items are sorted
according to $C_{u}$ and $C_{i}$, respectively; then $Gini_{u}$ and $Gini_{i}$
are defined as follows:

$ev_{6}=Gini_{u}(UCM)=1-2\sum_{i=1}^{|I|}\frac{|I|+1-i}{|I|+1}\times\frac{|C_{i}|}{|C|}$
(8)

$ev_{7}=Gini_{i}(UCM)=1-2\sum_{u=1}^{|U|}\frac{|U|+1-u}{|U|+1}\times\frac{|C_{u}|}{|C|}$
(9)

The Gini coefficient captures the frequency distribution of the check-ins for
users or items. It is scaled between $[0;1]$, where a score of $0$ would
correspond to a uniform popularity distribution and $1$ to the extreme case of
all check-ins being concentrated on one user/item.

####  3.2.3 EVs Based on the Item and User Properties

The following two EVs, Popularity Bias and Long Tail Items, have been
motivated by Deldjoo et al. (2021) to be included in their explanatory
framework. Arguably, the POI recommendation domain is even more severely
impacted by popularity bias (Massimo and Ricci, 2021, Sánchez and Dietz,
2022), thus, it is imperative to include them in our framework. Given the
distribution of the metrics with outliers, we augment the aggregation methods
used in Deldjoo et al. (2021) (mean, standard deviation, skewness, and
kurtosis) with the median value, as the median is more robust against outliers
compared to the mean value.

###### Definition 6

(Popularity Bias) We follow the commonly accepted definition of popularity
bias proposed by Abdollahpouri et al. (2019).

$ev_{8:12}=f\Bigg{(}\Bigg{\\{}\frac{\sum_{i\in
C_{u}}\phi(i)}{|C_{u}|}\Bigg{\\}}_{u}\Bigg{)}$ (10)

where $\phi(i)$ is the popularity scoring function for an item $i$. An item’s
popularity score is defined as the number of users who visited $i$ over the
entire number of users, and $|C_{u}|$ is the number of check-ins of user $u$.
The term $f$ is an aggregation operator over users to capture inter-user
differences in the popularity profiles of users. They include average
popularity bias ($ev_{8}$, APB), median popularity bias ($ev_{9}$, MedPB),
standard deviation of popularity bias scores ($ev_{10}$, StPB), skewness
popularity bias ($ev_{11}$, SkPB), and kurtosis popularity bias ($ev_{12}$,
KuPB).

###### Definition 7

(Long Tail Items) Analyzing the popularity of items, they can be separated
into a short-head and a long-tail. We perform the split at 20% in terms of the
number of users that have visited the item:

$ev_{13:17}=f\Bigg{(}\Bigg{\\{}\frac{|i,i\in(C_{u}\cap\Gamma)|}{|C_{u}|}\Bigg{\\}}_{u}\Bigg{)}$
(11)

where $C_{u}$ are, again, the check-ins of user $u$, $\Gamma$, on the other
hand, represents long-tail items, which is determined by splitting the items
into short-head and long-tail items. We define the split between short-head
and long-tail in terms of the number of different users that have visited the
item. In the literature, typical cutoffs for separating the short-head from
the long-tail are at 20%–80%, cf. (Yin et al., 2012, Abdollahpouri et al.,
2017, Deldjoo et al., 2021), which we also use in our experiments.

As before, $f$ is an aggregation operator over users to capture inter-user
differences in long-tail profiles of users. They include average long tail
items ($ev_{13}$, ALT), median long tail items ($ev_{14}$, MedLT), standard
deviation of long-tail items scores ($ev_{14}$, StLT), skewness long tail
items ($ev_{15}$, SkLT), and kurtosis long tail items ($ev_{17}$, KuLT).

####  3.2.4 EVs Based on the User Activity and Mobility

In this section, we introduce a family of data characteristics that are
specific to the POI recommendation domain. The radius of gyration captures the
size of a user’s activity area. The distance from the city center is defined
similarly, but the information it captures is more about how central the
check-ins of a user are. Finally, the user’s activity duration is interesting
to include as well, as one can assume that the longer a user is within a city,
the more familiar she becomes with the city, and it might result in
increasingly difficult for recommendation algorithms to propose interesting
items.

###### Definition 8

(Radius of Gyration) This is a common metric to capture the geographic extent
of user mobility.

$ev_{18:22}=f\Bigg{(}\Bigg{\\{}\sqrt{\frac{\sum_{i\in
C_{u}}\text{haversine}(i,c_{u})|}{|C_{u}|}}\Bigg{\\}}_{u}\Bigg{)}$ (12)

where $c_{u}$ is the centroid of all the user’s visited venues
$\frac{\sum_{i\in C_{u}}i}{|C_{u}|}$ (González et al., 2008).

Again, $f$ is an aggregation operator over users to capture inter-user
differences in the radius of gyration of users. They include average radius of
gyration ($ev_{18}$, ARG), median radius of gyration ($ev_{19}$, MedRG),
standard deviation of radius of gyration scores ($ev_{20}$, StRG), skewness of
radius of gyration ($ev_{21}$, SkRG), and kurtosis of the radius of gyration
($ev_{22}$, KuRG).

###### Definition 9

(Distance to City Center) This EV is very similar to the radius of gyration,
however, the center is not set to the centroid of the venues visited by the
user, but to the center of the city. It is useful to differentiate between
users who perform activities near to the center of a city – typically of
historic significance – and users who are more active in the outskirts.

$ev_{23:27}=f\Bigg{(}\Bigg{\\{}\sqrt{\frac{\sum_{i\in
C_{u}}\text{haversine}(i,cc)|}{|C_{u}|}}\Bigg{\\}}_{u}\Bigg{)}$ (13)

where $cc$ is the geographic location of the city center. Again, we use
different aggregation functions: the average distance to the city center
($ev_{23}$, ADCC), median distance to the city center ($ev_{24}$, MedDCC),
standard deviation of the distances to the city center ($ev_{25}$, StDCC),
skewness of distance to the city center ($ev_{26}$, SkDCC), and kurtosis of
the distance to the city center ($ev_{27}$, KuDCC).

###### Definition 10

(Duration Active) This EV is useful to provide insights into the effects of
users being active for a shorter or longer duration.

$ev_{28:32}=f\Bigg{(}\Bigg{\\{}t(i)_{l}-t(i)_{f}|i\in
C_{u}\Bigg{\\}}_{u}\Bigg{)}$ (14)

where $t(i)_{f},t(i)_{l}$ is the time of the first and last check-in of the
items $i$ the user $u$ has visited, respectively. Note that in this case, we
took the duration from all check-ins of the user, including repeated check-
ins. Again, we use different aggregation functions: the average activity
duration ($ev_{28}$, ADA), median activity duration ($ev_{29}$, MedDA),
standard deviation of activity duration ($ev_{30}$, StDA), skewness of
activity duration ($ev_{31}$, SkDA), and kurtosis of the activity duration
($ev_{32}$, KuDA).

###  3.3 Dependent Variables

For the dependent variables, we decided to analyze three different dimensions
of the recommendations:

  * •

Ranking accuracy: we will focus on measuring how many recommended items
actually match the ground truth of the user. For this purpose, we will use the
nDCG metric (Järvelin and Kekäläinen, 2002) that is defined in Equations 15
and 16.

$\mbox{nDCG@k}=\frac{1}{U}\sum_{u\in
U}\frac{\mbox{DCG}(u)@k}{\mbox{IDCG}(u)@k}$ (15)
$\mbox{DCG}(u)@k=\sum_{n=1}^{k}\frac{2^{rel_{n}}-1}{\log_{2}(n+1)}$ (16)

where $rel_{n}$ denotes the real relevance of item $n$. In explicit rating
data sets, this relevance value is normally bounded in the [0,5] interval,
with 0 representing a non-relevant value; in our experiments, as we do not
have explicit ratings but check-ins, the relevance of the items appearing in
the test set will always be 1. IDCG represents the ideal DCG, and it is
computed in the same way as DCG but using the ground truth as the ranking.
Higher values in nDCG mean that more relevant recommendations are being
provided to the users; that is, that more recommended venues are actually
visited by the user according to the test set.

  * •

Novelty: the novelty of a recommendation can be assessed by measuring the
proportion of popular venues being recommended. If a high percentage of the
recommended venues are already well-known or frequently consumed/visited, it
indicates that the recommendations lack novelty. To measure novelty,
popularity is often used as a proxy, especially in offline evaluation where
direct feedback from users is not possible. This is because it is generally
assumed that whatever is popular within a community is likely to be known by
most users and, thus, not novel. To measure this dimension, we use the
Expected Popularity Complement (EPC) metric (Vargas and Castells, 2011),
defined in the following equation:

$\small EPC@k=\frac{1}{U}\sum_{u\in U}Z(u)\sum_{i_{k}\in
RL_{u}@k}(1-p(\mbox{seen}\mid i_{k}))$ (17)

where $RL_{u}$ is the recommendation list for user $u$, and $Z(u)$ is a
normalizing constant (generally $Z(u)=1/\sum_{i\in RL_{u}@k}{1}$). Please,
note that the original definition of the metric provided by Vargas and
Castells (2011) also incorporates a discount model (as the one used in the
nDCG metric) and a relevance model, in order to measure both the relevance and
the novelty of the recommendations together. However, in our work, as we are
evaluating ranking accuracy with nDCG, we will use the pure definition of EPC.
According to this, higher values in EPC implies that more novel
recommendations are provided to the users.

  * •

Item exposure: In the traditional recommendation domain, it is well-known that
some algorithms recommend only a few items from the entire catalog (Liu and
Zheng, 2020, Abdollahpouri et al., 2019), and hence, it has been observed that
many models suffer from a popularity bias (see Section 2.3). As a result, in
our work, we are going to focus on item exposure. We define item exposure as
the difference between the number of times an item should be recommended
according to the test set and the number of times a specific algorithm has
recommended such an item. We will use the following simple metric based on the
expected exposure loss from (Ekstrand et al., 2022) to measure the item
exposure:

$IE@k=\sum_{i\in
I}\left|\frac{U_{test}(i)}{U_{test}}-\frac{Rec@k(i)}{U_{test}}\right|$ (18)

where $U_{test}$ denotes the number of users in the test set, $U_{test}(i)$
refers to the number of users that visited item $i$ in the test set, and
$Rec@k(i)$ is the number of times item $i$ has been recommended considering
all recommendations (i.e., rankings) until position $k$, i.e., at cutoff $@k$.
As we are comparing the recommended exposure and the exposure of the items in
the test set, the lower the values obtained in this metric, the better the
performance of the recommenders in terms of item exposure111We are aware that
some works have treated the item exposure as a concept similar to the
popularity of an item (Shih et al., 2016), but we argue that they are two
different concepts. In particular, we understand that an algorithm suffers
from popularity bias if it focuses on recommending the most popular items
(regardless of the ground truth), while for an algorithm to have poor item
exposure performance, it means that it is over- or under-representing items in
the recommendations based on the test set. Hence, to measure item exposure, we
should check if the number of times a venue is recommended is close to the
number of interactions such item has received in the test set..

##  4 Constructing Subsamples with Different Data Characteristics

To apply the explanatory framework described in the previous section, it is
necessary to obtain several subsamples from a larger recommendation data set.
We now discuss our approach to constructing various data sets with different
characteristics that will serve as inputs for the explanatory framework. This
aspect poses a challenge since the recommendation data sets, specifically the
user-item interaction matrices, exhibit interdependencies that are complex to
disentangle. The approach proposed by Deldjoo et al. (2020) constructs
subsamples by selecting a random number of users and items while enforcing
certain constraints, such as predefined data set densities.

As opposed to prior studies (Adomavicius and Zhang, 2012, Deldjoo et al.,
2021), we propose to use realistic subsamples that are grounded in the domain
instead of randomly sampling users with constraints. Generating the subsamples
in such a way has two advantages: first, the subsamples retain meaningful
semantics, which provides additional analytic insights into the domain;
second, it allows providers of recommender systems to understand in which
real-world situations different recommendation models are advantageous or
unfavorable. This allows for fine-grained hybridization of algorithmic models
within a larger recommender system to cater to all populations and situations
(Sánchez and Dietz, 2022).

In the following section, we describe our proposed domain-driven subsampling
procedure, which is based on subsetting a recommendation data set by factors
that might be relevant to recommending points of interest within a city.

###  4.1 Variables for Creating Domain-driven Subsamples

Our design goals when developing the methods to construct subsamples are that
i) the subsamples must have a common basis to enable fair and meaningful
comparisons,  ii) while at the same time, they show variability in terms of
their resulting data characteristics.  iii) Further, we need to generate a
sufficiently large number of subsamples to obtain robust results from the
regression model, but  iv) each subsample should be a tractable recommendation
problem, i.e., there is sufficient signal for the individual recommendation
models to produce sensible recommendations.

As mentioned before, we take a different approach to construct subsamples
compared to the approaches in the literature (Deldjoo et al., 2021, 2020). We
formulate hypotheses regarding relevant factors that might have an influence
on the outcome of POI recommendations. The core idea is to introduce a number
of variables relevant to the domain of POI recommendation and use them as
filters to include an interaction in the UCM or not. Thus, each variable
represents the explicit hypothesis that changing its value has an influence on
the recommendation outcome.

The set of all subsamples is the cross-product of the variable values applied
to the original data set. This means that the generation of the subsamples is
not only controlled by the outcome of the random processes that define the
number of items and users in the interaction matrix, but by meaningful
subsetting of groups of users, items, or interactions.

In the following subsections, we propose different variables for subsampling,
as the POI recommendation domain offers possibility for more complex
hypotheses, since – unlike most classic recommendation data sets – there is
the temporal (when a venue is visited) and the geographical (where the venue
is and where the user is from) aspects to analyze. We leverage these aspects
to formulate hypotheses along with common strategies employed in the
evaluation of POI recommendation data sets to shape the recommendation
outcome.

####  4.1.1 Enforce a Minimum $k$-core

To mitigate the extreme sparsity of typical POI recommendation data sets, it
is common to remove interactions from the UCM until all users and venues have
at least $k$ interactions. This is done to achieve a certain – higher – level
of density in the UCM and, thus, fewer ‘cold’ items/users which usually
results in higher accuracy metrics for interaction-based algorithms. Typical
values for $k$ are $2$ (cf. Sánchez and Bellogín (2021), Gao et al. (2015)),
$5$ (cf. Yuan et al. (2013, 2014), Yao et al. (2015)), or $10$ (cf. to Nunes
and Marinho (2014), Feng et al. (2015), Li et al. (2016)).

####  4.1.2 Drop Top $n$% Popular Venues

As previously discussed, popularity bias plays a large role in POI
recommendation with a substantial interplay between the popularity of items
and recommendation accuracy (Abdollahpouri et al., 2019, Massimo and Ricci,
2021, Sánchez et al., 2023). As check-in-based data sets usually do not come
with rating information, we limit the scope of popularity to the number of
people that have visited a venue (Jannach et al., 2015).

The method to analyze the impact of popularity bias is to generate different
subsamples by removing the most popular venues in the data set. We propose to
drop the most $n$% popular items from the data set to obtain different
distributions of the item popularity (Abdollahpouri et al., 2017). The
concrete values depend on the data set at hand, but as a general guideline, we
propose values for $n$ between $0$% and $5$% for the specific point-of-
interest recommendation domain after considering the popularity bias discussed
in Section 2.2.

####  4.1.3 Filter by Day Time

The temporal aspects of visiting points of interest is another interesting
variable for the subsampling as it has a practical influence on the
feasibility of visiting a recommended POI (Yuan et al., 2013, Yao et al.,
2016). We propose to subdivide the data set into the different times of day,
as one can expect that the user behavior will be different in the morning
compared to after sunset. Several subdivisions are possible: we propose to
subsample by considering all check-ins, check-ins before noon, check-ins in
the afternoon, and check-ins during the night.

####  4.1.4 Filter by Season

Seasonality is also a relevant factor that undoubtedly has an influence on the
visited venues both by locals and tourists (Liu et al., 2011). The exact split
between seasons can be tricky to make as a high granularity (e.g., weeks or
months) can result in very small subsamples. Further, seasons are not the same
in different regions, potentially requiring different segmentations for
destinations in different climate zones (Trattner et al., 2018). In the
context of an explainability study, we recommend using broad season
categories, such as a two-season (warmer and colder months) or a four-season
model.

####  4.1.5 Filter by User Residence

In the context of POI recommendation, different groups of users exhibit
different behavior (Sánchez and Dietz, 2022). Due to the differences in
behavior between locals and visitors, we argue that it is very promising to
use such a division as a subsampling variable. If the information about the
user’s home is available in the recommendation data set, the typical groups to
analyze would be the locals of the city, domestic visitors, or international
travelers.

####  4.1.6 Further Contextual Factors

The suitability of a POI recommendation is influenced by many contextual
factors, which has been the focus of various approaches in scientific
literature (Wörndl et al., 2011, Cai et al., 2017, Yang et al., 2017, Zhao et
al., 2019). Generally, spatio-temporal aspects are most significant to POI
recommendation, of which we have discussed the most relevant ones. These could
be utilized in many further ways, such as subdividing by different districts
of a destination city or analyzing temporal aspects that might have an impact
on the social behavior of people, such as week days versus weekends.

Further, the general groups of users proposed in Section 4.1.5 can be refined
into increasingly specialized groups. For example, literature on traveler
types (Gibson and Yiannakis, 2002, Neidhardt et al., 2014, Dietz et al., 2020)
offers various roles that one could use as subsampling variables.

Finally, context factors, such as the weather on a given day, could be a
promising variable for the subsampling if available (Trattner et al., 2018).

###  4.2 Discussion

In this section, we have formulated various hypotheses of what influences POI
recommendations. These hypotheses are manifested in different variables to
enable a rigorous computational analysis. In the choice of variables, we
discussed the ones that we deem to be most relevant based on the literature on
the analysis POI recommendation algorithms.

When setting up the experiments, it is still necessary to analyze the data
characteristics of the resulting subsamples to understand which value ranges
of the variables to test. Here, it is important to retain tractable
recommendation problems, i.e., not result in too sparse or small subsamples.
Also, depending on the data set, it is not always possible to test all
variables discussed in Section 4.1. For example, removing resident users in a
city that is not very active as a tourist destination could be
counterproductive in making interesting recommendations that may attract more
tourists.

###  4.3 Visualizing the Subsampling Variations

(a) $k$-core $=2$, origin $=$ NYC

(b) $k$-core $=5$, origin $=$ NYC

(c) $k$-core $=10$, origin $=$ NYC

(d) $k$-core $=2$, origin $=$ US

(e) $k$-core $=5$, origin $=$ US

(f) $k$-core $=10$, origin $=$ US

(g) $k$-core $=2$, origin $=$ other

(h) $k$-core $=5$, origin $=$ other

(i) $k$-core $=10$, origin $=$ other

Figure 1: Heat map of the visited venues in New York with different parameters
for the $k$-core and the origin of the users (better viewed in color).

To exemplify the effect of subsampling variables, Figure 1 visualizes the
interplay of two hypotheses: the $k$-core and the origin of the users. In this
heat map showing the density of check-ins in New York City, USA, the
difference between the behavior of the locals and travelers becomes apparent:
travelers tend to visit venues in Manhattan (with the exception of the
airports), while the locals naturally have check-ins all over the map. A
higher value for $k$-core leads to a higher density in the UCM but eliminates
many venues, which can be observed in the map visualization.

##  5 Experimental Setup

In this section, we describe the used data set and the process of selecting
the subsampling variables to obtain the subsampled recommendation data sets.
We provide details about the data preprocessing, how we conducted the
recommendation experiments, and give an overview of the outcomes. Finally, we
document the procedure of variable selection for the regression model, which
is the core of the explanatory framework.

###  5.1 Selecting a Suitable Data Set

To evaluate our proposed approach, it is essential to have a sufficiently
large data set for POI recommendation that enables us to perform meaningful
subsampling. Revisiting the literature (Bao et al., 2015, Sánchez and
Bellogín, 2022), we decided to use the Foursquare data set published by Yang
et al., which has about 33 million check-ins in 415 different cities of the
world (Yang et al., 2015) and has been frequently used to benchmark POI
recommendation performance222Data set is available from
https://sites.google.com/site/yangdingqi/home/foursquare-dataset. Although the
data set contains check-ins from many cities, we conclude that it is
infeasible to include multiple cities in the scope of the analysis. The number
of check-ins per city is influenced by the number of people and the popularity
of Foursquare in the city, which results in a few cities having a large number
of check-ins but many not having sufficient interactions to further subsample
them. Furthermore, the behavior of users is influenced by the topological
realities of cities (such as centralized vs. decentralized cities). Therefore,
each city needs to be analyzed separately to account for the different
geographic influences.

In this study, we focus on New York City (NYC), NY, USA, as it is one of the
most active cities on Foursquare and, hence, has been a common subject of
analysis in the POI recommendation domain, e.g., in Albanna et al. (2016),
Maroulis et al. (2016), Jiao et al. (2019). Concretely, the scope of our
analysis is the New York City Metropolitan Area, which comprises the five
boroughs of New York City and Newark, NJ. The geographical scope of the
analysis is visualized in Figure 1.To adapt the data set for the POI
recommendation domain, we eliminated venues of the “Residences” category, as
we do not consider them as interesting POI to recommend. Finally, we also
removed duplicated check-ins, i.e., check-ins at the same venue and identical
timestamps.

###  5.2 Generation of Subsampled Recommendation Data sets

In Section 4, we discussed various possible subsampling variables to be used
in an explanatory framework for POI recommendation. In the context of this
study, not all variables were feasible, thus, we made a selection based on the
data set at hand and our experience in POI recommendation. The subsampling
variables selected are: the origin of the user, the season of the year, the
minimum k-core of the recommendation data sets, and the exclusion of the most
popular venues.

####  5.2.1 Subsampling Variables

Given the size and nature of the Foursquare data set, it is infeasible to use
all variables discussed in Section 4.1, as the individual subsamples would
become too small. Instead, we focus on an informed subset of the most relevant
aspects of POI recommendation problems.

##### User Origin

The Foursquare data set contains check-ins from users from all around the
world. Although we are only running the recommendation experiments with check-
ins in New York City, we can use the complete data set to determine the home
city of the people in the recommendation data set. To achieve this, we use the
open-source tripmining library333https://github.com/LinusDietz/tripmining to
obtain the residence of the different users (Dietz et al., 2020). This library
converts the check-in stream of users into periods of being at home and on
travel. It uses the plurality strategy to determine which is the user’s home
city, which has been shown to be accurate in a ground-truth study (Kariryaa et
al., 2018).

We use this user home label as a subsampling variable with the following
values:

  * •

all: considering all users.

  * •

US: only domestic visitors from the United States, but not citizens of New
York City.

  * •

NYC: citizens of New York City.

  * •

other: travelers from outside of the US.

The intuition behind using the users’ home as a subsampling variable is that
the behavior of locals is different than the one of visitors. This also has a
significant influence on the recommendation outcome, as shown by Sánchez and
Dietz (2022).

##### Season

The effect of seasonality on the recommendation outcome has not been analyzed
in depth so far, providing us with the opportunity to analyze this within the
explanatory study. The oceanic climate of New York City comes with relatively
similar precipitation throughout the year, thus, we used the temperature
aspect of the climate diagram to subdivide the year into the following groups:

  * •

all: All check-ins irrespective of the season.

  * •

summer: check-ins during the warmer months in New York City from May to
October.

  * •

winter: check-ins during the colder months in New York City from November to
April.

By using only two groups, we hope to achieve a clear separation and keep the
size of the resulting subsamples larger.

#####  UCM Density

Traditionally, density (i.e., the inverse of sparsity) has been a key metric
to quantify the difficulty of a recommendation problem. The sparsity is
normally referred as the situation where most of the user-item interactions
are not observed in the training data (Idrissi and Zellou, 2020).

However, density is a dependent variable, which is typically adjusted by
enforcing a $k$-core, i.e., requiring at least $k$ interactions for each user
and venue and discarding users and venues that do not fulfill this threshold.

Following the practice in literature, we create subsamples using the following
values for $k$:

  * •

2: Enforce a $k=2$-core.

  * •

5: Enforce a $k=5$-core.

  * •

10: Enforce a $k=10$-core.

##### Item Popularity

Due to the large popularity bias in POI recommendation, we argue that it is
important to analyze the effect of disregarding the most popular venues. We
used the following values to analyze this effect:

  * •

0.5: Drop the most popular 0.5% venues from the current data set.

  * •

1: Drop the most popular 1% venues from the current data set.

  * •

2: Drop the most popular 2% venues from the current data set.

  * •

5: Drop the most popular 5% venues from the current data set.

####  5.2.2 Summary

Using these variables, we generate 144 recommendation subsamples in the form
of user-check-in matrices. The result of the cross-product of applying the
aforementioned subsampling variables as filters on the original data set is
144:

{origin = [all, NYC, US, international]} $\times$ {season= [all,
summer,winter]} $\times$
$\times$ {k-core = [2, 5, 10]} $\times$ {drop top venues = [0.5, 1, 2, 5]}

####  5.2.3 Training and Test Set Generation

For each of the subsamples, we perform a temporal split per user, where 80% of
the oldest interactions of each user are sent to the training set and the rest
to the test set. Foursquare users might have performed check-ins at the same
venue more than once, but the algorithms we use are meant to recommend new
items, which means that we discard all duplicate check-ins of a user in the
same venue both in the training and the test set. We decided to proceed like
this in the test set because the goal of recommender systems is to recommend
new venues to users to explore, not venues that the user already knows, which
is common practice in the POI recommendation domain.

Table 1: Summary statistics of the impact of the subsampling on the
explanatory variables. Note that for space reasons the table only contains the
independent subsamples and not the combination of the subsampling variables.

subsample | SpaceSize | Shape | Density | $Cp_{u}$ | $Cp_{i}$ | $Gini_{u}$ | $Gini_{i}$ | APB | ALT | ARG | ADCC | ADA
---|---|---|---|---|---|---|---|---|---|---|---|---
k = 2 | 5.57E+08 | 0.36 | 4.45E-04 | 17.44 | 6.32 | 0.62 | 0.66 | 254.24 | 0.02 | 5.12 | 6.39 | 119.12
k = 5 | 1.97E+08 | 0.47 | 1.04E-03 | 21.28 | 10.07 | 0.51 | 0.62 | 176.72 | 0.02 | 5.74 | 5.93 | 158.40
k = 10 | 7.04E+07 | 0.55 | 2.24E-03 | 25.27 | 13.96 | 0.40 | 0.59 | 120.08 | 0.02 | 5.96 | 5.96 | 200.77
dtv = 0.5% | 5.29E+08 | 0.35 | 3.85E-04 | 15.02 | 5.22 | 0.65 | 0.59 | 22.50 | 0.02 | 3.93 | 6.03 | 110.57
dtv = 1% | 5.15E+08 | 0.34 | 3.68E-04 | 14.27 | 4.88 | 0.65 | 0.57 | 17.15 | 0.03 | 3.89 | 6.18 | 109.86
dtv = 2% | 4.98E+08 | 0.34 | 3.43E-04 | 13.18 | 4.45 | 0.65 | 0.54 | 13.00 | 0.03 | 3.83 | 6.35 | 108.96
dtv = 5% | 4.56E+08 | 0.33 | 3.02E-04 | 11.25 | 3.70 | 0.64 | 0.48 | 8.41 | 0.03 | 3.77 | 6.77 | 108.86
o = NYC | 1.99E+08 | 0.14 | 9.66E-04 | 36.41 | 5.09 | 0.44 | 0.62 | 71.39 | 0.04 | 6.38 | 7.41 | 215.42
o = US | 5.09E+07 | 0.46 | 6.01E-04 | 6.31 | 2.91 | 0.46 | 0.56 | 104.55 | 0.06 | 4.68 | 6.55 | 81.60
o = other | 3.27E+07 | 0.51 | 7.81E-04 | 6.23 | 3.20 | 0.48 | 0.61 | 88.59 | 0.05 | 4.03 | 4.89 | 39.66
s = summer | 3.67E+08 | 0.33 | 3.80E-04 | 12.62 | 4.22 | 0.59 | 0.61 | 154.02 | 0.03 | 4.75 | 6.96 | 108.58
s = winter | 3.29E+08 | 0.32 | 4.02E-04 | 12.84 | 4.15 | 0.57 | 0.61 | 140.04 | 0.04 | 4.70 | 6.71 | 105.17

Key for the subsample column: k: value for the k-core, dtv: drop top venues
(in terms of popularity), o: origin, s: season (cf. Section 5.2.2) The
definition of the EV acronyms in the header row can be found in Section 3.2.

Table 1 tabulates statistics regarding the subsamples with the average
explanatory variables. For space reasons, we only tabulate the 12 subsampling
variables independently to get an impression of their individual impact. The
experiments for the explanatory analysis used the cross-product of the
subsampling variables, resulting in 144 subsamples.

###  5.3 Algorithms for POI Recommendation

In this section, we explain in detail the state-of-the-art algorithms that we
have used in our experiments. Due to the nature of the application of the
different algorithms, we will divide them into two main groups: classical
recommendation algorithms and those specifically designed for point-of-
interest recommendation.

  * •

Classical recommendation algorithms:

    * –

Random: performs recommendations of venues randomly.

    * –

Pop: recommends to the target user the venues ordered by decreasing popularity
(popularity measured by the number of different users that have visited that
venue).

    * –

UB: user-based neighborhood. Non-normalized $k$-nn algorithm that recommends
to the target user venues that other similar users visited before (Ning et
al., 2015, Aiolli, 2013). We used the classical cosine similarity and Jaccard
Index as similarity metrics.

    * –

IB: item-based neighborhood. Non-normalized $k$-nn that recommends to the
target user venues similar to the ones that she visited previously (Ning et
al., 2015, Aiolli, 2013). We use the corresponding item variations of the same
similarity metrics used for UB.

    * –

HKV: matrix factorization (MF) algorithm that uses Alternate Least Squares for
optimization proposed by Hu et al. (2008).

    * –

BPRMF: matrix factorization (MF) algorithm that uses the pairwise Bayesian
Personalized Ranking loss proposed by Rendle et al. (2009) as optimization
algorithm. For our experiments. we used the version from
MyMedialite’s444MyMedialite library: http://www.mymedialite.net/ library.

  * •

Point-of-interest recommendation algorithms:

    * –

IRENMF: weighted matrix factorization method proposed by Liu et al. (2014).
This algorithm incorporates geographical information by assuming that users
tend to visit neighboring venues (instance-level influence) and also by
considering that the users check-ins are shared in the same geographical
region (region-level influence).

    * –

GeoBPRMF: geographical Bayesian Personalized Ranking matrix factorization.
Algorithm proposed by Yuan et al. (2016) that assumes that the target user
will prefer to visit new venues that are close to the ones she visited
previously.

    * –

RankGeoFM: a ranking-based matrix factorization model proposed by Li et al.
(2015). They also incorporate the geographical influence in the
recommendations by exploiting the neighboring venues (by geographical
distance) with respect to the target POI.

    * –

PopGeoNN: hybrid algorithm combining popularity (Pop), a user-based
neighborhood method (UB), and a simple geographical component that recommends
to the target user the venues closer to the average geographical position of
all the venues visited by the user in the training set.

The final score is an aggregation of every item score provided by each
recommender after normalizing its values by the maximum score of each method.

To achieve optimal parameters for each recommendation subsample, we
systematically chose the optimal hyperparameters for each recommendation model
by nDCG@5. The tested hyperparameter ranges are shown in Table 2.

Table 2: Hyperparameters tested in the recommenders. The best configurations are selected by maximizing nDCG@5. Rec |  Hyperparameters
---|---
Random |  None
Pop |  None
UB |  Sim = {Vector Cosine, Set Jaccard}, $k=\\{20,40,60,80,100,120\\}$
IB |  Sim = {Vector Cosine, Set Jaccard}, $k=\\{20,40,60,80,100,120\\}$
HKV |  Iter = $20$, Factors = $\\{10,50,100\\}$, $\lambda=\\{0.1,1,10\\}$, $\alpha=\\{0.1,1,10,100\\}$
BPRMF |  Factors = $\\{10,50,100\\}$, BiasReg = $\\{0,0.5,1\\}$, LearnRate = $0.05$, Iter = $50$, RegU = RegI = $\\{0.0025,0.001,0.005,0.01,0.1\\}$, RegJ = RegU$/10$
GeoBPRMF |  Factors = $\\{10,50,100\\}$, BiasReg = $\\{0,0.5,1\\}$, LearnRate = $0.05$, Iter = $50$, RegU = RegI = $\\{0.0025,0.001,0.005,0.01,0.1\\}$, RegJ = RegU$/10$ maxDist=$\\{1,4\\}$
IRENMF |  Factors = $\\{50,100\\}$, geo-$\alpha=\\{0.4,0.6\\}$, $\lambda_{3}=\\{0.1,1\\}$, clusters = {5, 50}
RankGeoFM |  Factors = $\\{50,100\\}$, $\alpha=\\{0.1,0.2\\}$, c = 1, $\epsilon=0.3$, neighs = $\\{10,50,100,200\\}$ iter = 120, decay = 1, boldDriver = True, learnRate = 0.001
PopGeoNN |  Sim = {Vector Cosine, Set Jaccard}, $k=\\{20,40,60,80,100,120\\}$

###  5.4 Recommendation Results on Subsamples

As described in Section 4 and shown in Section 5.2.2, we subdivided the data
set into 144 subsamples. Using the recommendation models presented in Section
5.3 with the optimal parameters, we achieve the following recommendation
outcomes regarding nDCG (Figure 2a), EPC (Figure 2b), and Item Exposure
(Figure 2c). We also report in Table 3 the average (denoted as avg) and the
standard deviation (denoted as std) obtained in the 144 subsamples by each
recommender in nDCG, EPC, and Item Exposure, all of them measured at ranking
cutoff 5.

Table 3: Mean value of the performance results of the recommenders in all subsamples. All values @5. We represent in bold the best value obtained when computing the mean (mean) in each metric. Family | Recommender | nDCG | EPC | Item Exposure
---|---|---|---|---
mean | std | mean | std | mean | std
Classic | Rnd | $0.000$ | $0.001$ | 0.999 | $0.001$ | $5.602$ | $0.737$
Pop | $0.004$ | $0.004$ | $0.987$ | $0.009$ | $7.820$ | $1.430$
UB | 0.011 | $0.006$ | $0.997$ | $0.003$ | $6.128$ | $1.230$
IB | $0.010$ | $0.005$ | 0.999 | $0.001$ | $6.137$ | $1.026$
HKV | $0.003$ | $0.005$ | $0.998$ | $0.002$ | $7.689$ | $1.482$
BPRMF | $0.010$ | $0.008$ | $0.994$ | $0.005$ | $6.618$ | $1.223$
POI | IRENMF | 0.011 | $0.009$ | $0.996$ | $0.004$ | $6.818$ | $1.089$
GeoBPRMF | $0.009$ | $0.010$ | $0.995$ | $0.004$ | $7.223$ | $1.348$
RankGeoFM | $0.003$ | $0.004$ | $0.998$ | $0.002$ | 5.336 | $0.708$
PopGeoNN | $0.009$ | $0.006$ | $0.992$ | $0.006$ | $6.784$ | $1.123$

Table 3 reveals that the performance of the recommenders, in terms of nDCG, is
relatively low. This is a common phenomenon in the POI recommendation domain,
given the vast number of candidate POIs and the scarcity of user visits to
train the models. However, some recommenders, like HKV or RankGeoFM,
consistently show a lower performance overall, with limited standard
deviation. With respect to novelty, we generally obtain high results (even in
the Pop recommender). This can be attributed to the data sparsity, where
although certain popular POIs have a significant number of visits, they have
been explored by only a small percentage of users relative to the total number
of potential users. Notably, the Pop algorithm exhibits the highest deviation,
indicating that each subsample may feature different popular POIs. This also
shows that the popularity bias is different in each subsample, being more
exaggerated in those subsamples where we remove a smaller percentage of
popular items (dtv=0.5). This popularity bias impacts the performance of other
recommenders with a bias towards popularity, such as BPRMF or PopGeoNN (in
fact, this recommender has a pure popularity component contributing to the
recommendations generated), while others like IB or RankGeoFM are less
affected. In terms of Item Exposure, we can observe notable variations among
the algorithms. While the Pop recommender achieves higher exposure results by
focusing solely on recommending popular venues, other models such as
RankGeoFM, UB, or IB can offer recommendations that encompass a more diverse
range of POIs. The behavior of the IB recommender is particularly interesting:
as derived from the results, it ranks second in terms of item exposure,
highlighting its ability to provide fair recommendations for items appearing
in the test sets; simultaneously, its performance in novelty stands out while
it also obtains competitive level of relevance, falling slightly behind the UB
and IRENMF recommenders. However, both UB and IRENMF do not obtain results as
competitive as IB in both item exposure and novelty.

Figure 2: The recommendation outcomes using the following metrics: a) nDCG@5,
b) EPC@5, and c) Item Exposure@5. The boxplot indicates the 25%, the median,
and the 75% quantiles. Overlayed is a violin plot emphasizing the density of
values and the overall range of the outcomes and the mean value with an x. The
dashed line in the nDCG plot of Figure 2a) indicates the mean value of the
Popularity algorithm.

###  5.5 Observations

As we mentioned before, we show in Figure 2 the violin plots of the results
obtained by each recommender in the 144 subsamples in terms of accuracy,
novelty, and diversity. As we can observe, the recommendation accuracy
measured in nDCG@5 varies substantially between the 144 subsamples, which is
an expected and desired outcome (Figure 2a). The random recommender is the one
with the least variance and performance, however, the HKV matrix factorization
model is also consistently low in performance, indicating that it can not deal
well with many of the smaller subsamples. Furthermore, the Pop, IB, UB, and
RankGeoFM models seem to be more robust regarding their outcomes compared to
the purely matrix factorization-based approaches that do not consider the
geographical component, as their interquartile ranges are smaller.

There are no surprises regarding the novelty of the recommendations, which we
measure using the EPC metric, cf. Section 3.3. The models that involve some
aspect of popularity (Popularity, PopGeoNN) and the BPR models produce
relatively fewer novel recommendations, which is expected due to how the
recommendations are computed (Figure 2b). On the contrary, as mentioned
before, the behavior of IB is interesting in terms of novelty since it is the
second-best model (after Rnd) in all subsamples, obtaining relatively
competitive accuracy results. This makes the IB a model that should be
considered to try to achieve a balance between accuracy and novelty.

Finally, the Item Exposure (Figure 2c) shows a low variation between most
models, with the quantiles all being between $5$ and $7.5$. Again, the
recommenders that generated fewer novel recommendations or exhibited a higher
popularity bias, such as Pop, BPRMF, GeoBPRMF, or PopGeoNN, also achieved
higher scores in terms of item exposure. This indicates a notable disparity in
the distribution of recommended POIs compared to the POIs that the user has
visited during the test set. This is consistent with previous work (Sánchez et
al., 2023), where different biases are analyzed in the POI recommendation
domain, and the effect on the performance of a set of recommenders in
different cities around the world is compared.

###  5.6 Interpretation

The plots in Figure 2 also tell much about the outcome of the subsampling
process. With respect to the nDCG and the EPC metrics, the mass of the density
plots is relatively compact, with some outliers to the top or bottom,
respectively. In the Item Exposure plot, the shapes of the density plots are
strung out, generally matching the interquartile ranges better. An interesting
observation is that in the recommendation outcomes of the worse-performing
models in terms of accuracy, i.e., Random, Pop, HKV, and RankGeoFM two groups
become visible, i.e., the density plot looks tapered just below the mean
value, meaning that the subsamples could be divided into 2 groups according to
their performance. Even though a deep analysis of this aspect is out of the
scope of this paper, it might be interesting to understand in the future which
samples belong to each group and the impact the number (and frequency) of
these groups may have on the explanatory power of the methodology followed in
this work.

###  5.7 Excluding Low-performing Recommendation Models from the Explanatory
Study

The pure Popularity-based recommendation algorithm is a very simple,
parameter-less model, but nevertheless a useful baseline in POI recommendation
due to the inherent popularity bias of the domain (Bellogín et al., 2017).
Even though computing the recommendations solely on the popularity of the
items contradicts the principle of personalization, visitors tend to visit the
popular highlights of a destination. Thus, we argue that any model in POI
recommendation should at least outperform the Popularity model in terms of
accuracy.

When it comes to the explanatory analysis, we remove the Random, HKV, and
RankGeoFM models from the pool of algorithms for the explanatory study since
the mean recommendation accuracy over all subsamples is lower than the simple
Popularity baseline. The reason for this is that the purpose of the
explanatory study is that we want to learn what the success factors of
recommendation models are in terms of their data characteristics measured
using EVs. By including models that are not “successful” (by outperforming the
popularity baseline in terms of nDCG), we would analyze the factors
contributing to poor recommendations, which is a meaningless endeavor.

This sets the final pool of 7 recommendation models to Pop, IB, UB, BPRMF,
GeoBPRMF, IRENMF, and PopGeoNN.

###  5.8 Selecting Relevant Explanatory Variables

1 $\text{features}\leftarrow\\{EVs\\}$;

2 $TR\\_VIF\leftarrow 10$;

3 while  _argmax(VIF( features)) $<$ TR_VIF_ do

4 corr_mat $\leftarrow$ PCC(features);

5 $c_{1},c_{2}\leftarrow$ argmax( | corr_mat | );

6 remove $\leftarrow$ argmax( argmax(| PCC($c_{1}$, { features \ $c_{2}$ } ) |), argmax(| PCC($c_{2}$, { features \ $c_{1}$ } ) | ) ) ;

7 features $\leftarrow$ features \ remove ;

8

9 end while

10return features;

Algorithm 1 Elimination of EVs Based on Correlation Analysis

In Section 3.2, we defined 32 potential explanatory variables that can be used
to explain the dependent variables (Section 3.3) using the regression model.
Naturally, not all independent variables possess equal levels of informative
signal, i.e., they can be noisy. The regression model is useful in identifying
noisy or unrelated independent variables as these will not be statistically
significant coefficients with respect to the target variable. However, when
using multiple variables in a regression model, multicollinearity can arise,
i.e., two or more explanatory variables being correlated with each other.
While this does not impede the outcome of the regression model,
multicollinearity in the explanatory variables decreases the predictive
contribution of the individual coefficients. To obtain meaningful results in
the significance analysis of the coefficients, we mitigate collinearity by
eliminating such redundant variables.

To do this in a reproducible way, we propose a procedure to remove highly
correlated variables until the collinearity is mitigated to an acceptable
level. The multicollinearity of a regression model is measured by the variance
inflation factor (VIF), but the scientific literature is divided on what
maximum VIF value is acceptable (Robinson and Schumacker, 2009, O’brien, 2007,
Stine, 1995). Reflecting on this, we strive to achieve a VIF of 10 in our
regression model.

Our proposed procedure is useful with an increasing amount of variables, as it
removes human judgement from the process. This is an improvement on the
previous work (Adomavicius and Zhang, 2012, Deldjoo et al., 2020, 2021), where
this step was not precisely specified.

Table 4: Final Explanatory Variables after controlling for multicollinearity EV | VIF before | VIF after
---|---|---
SpaceSize | 20.05 | 3.10
Shape | 44.14 | 5.17
Density | 19.37 | 3.77
$Cp_{u}$ | 44.59 | -
$Cp_{i}$ | 66.96 | 6.50
$Gini_{i}$ | 59.48 | 3.72
$Gini_{u}$ | 43.95 | -
APB | 1124.91 | -
MedPB | 273.57 | -
StPB | 401.88 | 1.70
SkPB | 27.29 | -
KuPB | 20.89 | 1.34
ALT | 464.46 | -
StLT | 553.90 | -
SkLT | 442.75 | -
KuLT | 334.39 | -
ADCC | 2687.53 | -
MedDCC | 550.11 | 7.68
StDCC | 938.92 | -
SkDCC | 221.20 | -
KuDCC | 153.63 | -
ARG | 1496.70 | -
MedRG | 315.72 | -
StRG | 47.40 | 7.30
SkRG | 275.60 | -
KuRG | 71.18 | -
ADA | 471.15 | -
MedDA | 51.81 | -
StDA | 46.89 | -
SkDA | 232.10 | -
KuDA | 97.35 | 2.89

The goal is to determine a set of input variables for the regression analysis
that have low collinearity, which is measured by the variance inflation factor
(VIF). To determine variables that cause unwanted collinearity, a correlation
analysis is required to discard correlated variables. Algorithm 1 describes
our proposed procedure: while there is still an EV with a VIF over the
threshold of $10$, we compute all pairwise Pearson Correlation Coefficients
(PCC) of the features, obtaining the correlation matrix $(corr\\_mat)$, and
determining the two different features with the highest positive or negative
correlation as our candidates for elimination. From these two candidates
$(c_{1},c_{2})$, we eliminate the candidate that has the highest correlation
to any other feature in the remaining features. This elimination of EVs is
repeated until the VIF values of all remaining EVs satisfy the threshold.

Figure 3: Pairwise Pearson Correlations between all explanatory variables
Figure 4: Pearson Correlation between explanatory variables after removing
highly correlated variables using Algorithm 1.

Table 4 and Figures 3 and 4 show the outcome of applying Algorithm 1 to the
data from the experiments comprising the recommendation outcome of the well-
performing recommenders as established previously in Section 5.7 on the 144
subsamples. The target metric of the linear model to compute the VIF was
nDCG@5. We retain 10 EVs, namely SpaceSize, Shape, Density, $Cp_{i}$,
$Gini_{i}$, StPB, KuPB, MedDCC, STRG, and KuDA. This outcome is interesting as
it puts emphasis on the EVs that capture the structure of the user-check-in
matrix, such as the SpaceSize, Shape, Density, and check-ins per item
($Cp_{i}$). This is not surprising, as these are very common metrics to
quantify the difficulty of a recommendation problem.

Furthermore, some aspect of all other families of EVs was included, with the
exception of the long-tail items: the Gini coefficient for the items is
retained as a measure for the inequality in the check-in distribution of the
UCM and the standard deviation and the kurtosis of the popularity bias (StPB
and KuPB) represent the EVs that are based on the item and user profiles. The
EVs based on user mobility are represented by the median distance to the city
center (MedDCC) and the standard deviation of the radius of gyration (StRG),
demonstrating that geographical variables are critical in the context of POI
recommendation. Finally, the kurtosis of the duration active (KuDA) is also
retained.

This shows that all these EVs describing the recommendation subsamples are
sufficiently different from each other measured in terms of VIF and also the
maximum pairwise Pearson Correlation Coefficent of $0.69$ between StRG and
MedDCC (cf. Figure 4). While we will come to the explanatory power of the EVs
in the following section, this result alone underlines that the newly
introduced EVs regarding mobility and user activity broaden the perspective of
POI recommendation problems.

The only statistical measure of the EVs that was not included was the skewness
of the EVs. In Figure 3, we can see that it is highly correlated with the
kurtosis of all EVs. Given that the EVs of our recommendation problems
typically do not follow a symmetrical distribution, the skewness is not a very
relevant metric. We nevertheless included it to confirm this and to maintain
comparability with Deldjoo et al. (2021).

##  6 Results

Recall from Section 3 that in its core, the explanatory framework is a linear
regression (cf. Equation 2) with the data characteristics of the 144
subsamples quantified in the explanatory variables as input variables and
nDCG, EPC, and Item Exposure as dependent variables. What we are interested in
are the coefficients of the model ($\theta_{ev}$ in Equation 2), as these
coefficients quantify the impact of the individual EV on the outcome variable.
We run the explanatory framework for each recommendation model that produced
competitive results (cf. Section 5.7) independently using the EVs that did not
suffer from multicollinearity (cf. Section 5.8).

In the analysis of the results, we assessed three key aspects: Accuracy,
Novelty, and Item Exposure. Our findings revealed notably high $R^{2}$ values
for each of these aspects. This suggests that, even after discarding numerous
explanatory variables, our regression models still have substantial
explanatory power. In this section, we delve into a detailed discussion of our
results, using a cutoff value of 5 for each metric, as in the POI
recommendation domain, it is common to report smaller cutoffs (Liu et al.,
2014, Li et al., 2015, Yuan et al., 2016). For additional results at cutoff
values of 10 and 20, please refer to Appendix A and the supplementary
material.

Our structure for presenting the experimental results follows this format:
firstly, we tabulate the regression coefficients ($\theta_{ev}$) and then
visualize these coefficients through coefficient plots. The result tables,
i.e., Table 5, Table 6, and Table 7, provide an overview of the goodness-of-
fit with the $R^{2}$ and adjusted $R^{2}$ values. Subsequently, we present the
coefficients $\theta_{ev}$ for each model. These coefficients are annotated
with stars, which indicate the significance level of the relationship between
the explanatory variable and the outcome variable. The significance levels and
the corresponding p-values used are *** to indicate that $p<0.001$, ** for
$p<0.01$, and * for $p<0.05$.

Furthermore, we visualize the values of the tables in coefficient plots
(Figures 5, 6, and 7). The dots show the coefficient $\theta_{ev}$; the
whiskers span the 95% confidence interval.

###  6.1 Accuracy

First, we analyze the recommendation accuracy in terms of nDCG in Table 5. The
$R^{2}$ coefficients of determination of the regression models are all between
$0.80$ and $0.86$, indicating that the 10 EVs could explain 80%–86% of the
variation of the nDCG@5. This result is consistent with the two previous
studies of Adomavicius and Zhang (2012) and Deldjoo et al. (2021). The least
predictable algorithm was the IB recommendation model, while the BPRMF
algorithm had the highest $R^{2}$.

Table 5: Coefficients of the Regression Model for nDCG@5

name
---
$R^{2}$
$R^{2}$ (adj)
SpaceSize
Shape
Density
$Cp_{i}$
$Gini_{i}$
StPB
KuPB
MedCC
StRG
KuDA

Popularity
---
0.824
0.813
-0.049***
0.034***
0.09***
-0.076***
0.078***
0.117***
0.002
0.027*
0.048***
0.045***

IB
---
0.799
0.787
-0.003
-0.001
0.007***
0.013***
0.016***
0.008***
0.001
0
-0.003
0.023***

UB
---
0.833
0.823
-0.034***
0.028***
0.072***
-0.062***
0.078***
0.059***
-0.007
0.018*
0.035***
0.055***

BPRMF
---
0.86
0.852
-0.049***
0.035***
0.089***
-0.061***
0.09***
0.095***
-0.006
0.033***
0.038***
0.056***

GeoBPRMF
---
0.823
0.813
-0.053***
0.041***
0.096***
-0.057***
0.085***
0.076***
0.002
0.03**
0.049***
0.047***

IRENMF
---
0.825
0.814
-0.039***
0.021**
0.049***
-0.03**
0.056***
0.098***
-0.006
0.024**
0.017*
0.02**

PopGeoNN
---
0.841
0.832
-0.043***
0.03***
0.082***
-0.065***
0.078***
0.099***
-0.001
0.023*
0.038***
0.054***

***: $p<0.001$, **: $p<0.01$, *: $p<0.05$

When it comes to the general patterns in the coefficients (cf. Figure 5), the
IB model is the most robust one against changes in the data characteristics,
which can be observed by the tendency that the coefficients are closest to 0
and the confidence intervals being the shortest. The only other recommendation
model that shows a similar discernible pattern is the IRENMF algorithm;
however, it is only outside of the main group of models in the following EVs:
Density, Check-ins per Item ($Cp_{i}$), Gini Index of Items ($Gini_{i}$), and
the Kurtosis of the Duration Active (KuDA). The overall variability in the
coefficient values is the largest in the Standard deviation of the Popularity
bias (StPB), with the highest observable coefficient belonging to the
Popularity recommendation model. The coefficients of the other recommendation
models are generally grouped together with clearly overlapping confidence
intervals, indicating that the success factors of all recommendation models
are similar. With the exception of the Item-based recommender and the Kurtosis
of the Popularity Bias (KuPB), the coefficients of all other EVs and models
share the same sign of values. A higher value in Shape, Density, $Gini_{i}$,
StPB, MedDCC, StRG, and KuDA leads to higher accuracy, whereas a higher value
in Space Size and $Cp_{i}$ deteriorates the recommendation accuracy.

Figure 5: Coefficient plot for nDCG@5. SpaceSize and $Cp_{i}$ have a negative
influence on nDCG@5, KuPB is neutral, whereas the other EVs have a positive
impact. The IB algorithm is close to neutral for all EVs, indicating that its
results are very robust against data characteristics. The opposite effect is
observed in the Pop algorithm.

Turning our attention to the significance levels of the coefficients regarding
the outcome variable, we observe that Density, $Cp_{i}$, $Gini_{i}$, Standard
deviation of the Popularity Bias (StPB), and the Kurtosis of the Duration
Active (KuDA) is always highly significant with $p<0.01$ for all
recommendation models. These consistently low p-values across the board of all
recommendation models underline their importance for the success of POI
recommendation algorithms. On the contrary, the Kurtosis of the Popularity
Bias (KuPB) coefficient was not a significant predictor towards the nDCG@5.

###  6.2 Novelty

The second aspect of our analysis is novelty, which we measure using the EPC
metric. Again, we tabulate the coefficients in Table 6. Comparing the $R^{2}$
values to the ones in Table 5 (accuracy), we see a worse regression fit with
values ranging from $0.54$ (Item-based model) to $0.77$ (Popularity and IRENMF
models). This might have several reasons: traditionally, recommendation
algorithms were developed with accuracy goals in mind; thus, our initial
selection of models might still reflect this publication bias toward accuracy
(in fact, we selected the best model parameters based on nDCG@5, which would
corroborate this bias). For the same reason, the recommendation outcome might
carry more unexplainable variation. In the coefficient section of Table 6, we
see higher values in the scores compared to Table 5.

Table 6: Coefficients ($\theta_{ev}$) of the Regression Model for EPC@5

name
---
$R^{2}$
$R^{2}$ (adj)
SpaceSize
Shape
Density
$Cp_{i}$
$Gini_{i}$
StPB
KuPB
MedCC
StRG
KuDA

Popularity
---
0.772
0.758
0.043**
-0.031*
-0.111***
0.071***
-0.101***
-0.158***
-0.038***
-0.051**
0
-0.035*

IB
---
0.543
0.516
0.002
-0.003
-0.013***
0.001
-0.009***
-0.017***
0
-0.005
0.004
-0.006.

UB
---
0.672
0.653
0.028*
-0.037**
-0.085***
0.081***
-0.081***
-0.071***
0
-0.042**
-0.009
-0.027*

BPRMF
---
0.736
0.72
0.045**
-0.032.
-0.099***
0.072**
-0.094***
-0.169***
-0.038**
-0.052**
-0.008
-0.032*

GeoBPRMF
---
0.658
0.638
0.041**
-0.038**
-0.087***
0.069***
-0.067***
-0.103***
-0.017.
-0.052**
-0.022
-0.031*

IRENMF
---
0.768
0.755
0.033**
-0.016
-0.046***
0.036*
-0.052***
-0.148***
-0.032***
-0.034*
0.001
-0.011

PopGeoNN
---
0.735
0.719
0.034**
-0.032*
-0.092***
0.069***
-0.084***
-0.118***
-0.022*
-0.048**
0.001
-0.028*

***: $p<0.001$, **: $p<0.01$, *: $p<0.05$

Figure 6: Coefficient plot for novelty measured using EPC@5. We observe an
inverse relationship compared to nDCG@5 with Shape, Density, $Gini_{i}$, StPB,
MedCC, KuDA, KuPB having a negative impact and SpaceSize and $Cp_{i}$ having a
positive impact on the EPC metrics.

Analyzing the patterns within the coefficients presented in Figure 6, we can
discern a noteworthy observation: our experiments reveal an inverse
relationship between novelty and accuracy. For all considered EVs, except for
the statistically non-significant KuPB, and across various recommender models
— except the Item-based model — a distinct change in the sign of coefficients
is evident. Specifically, these coefficients switch from a positive
association to a negative one and vice versa. The coefficients for the Item-
based model once again converge near zero, while the IRENMF model exhibits a
more consistent behavior compared to the nDCG outcome variable. Among the EVs,
SpaceSize and Check-ins per Item display positive coefficients for EPC@5.
Conversely, the other EVs exhibit a negative impact (Shape, Density,
$Gini_{i}$, StPB, MedDCC, KuDA, KuPB) or a neutral influence (StRG) on this
particular outcome variable.

The observation of a neutral influence of the Standard Deviation of the Radius
of Gyration is bolstered by the finding that its coefficients do not achieve
statistical significance across any of the recommendation models. In contrast,
we note that certain Explanatory Variables (EVs) exhibit a high level of
significance ($p<0.001$) across all recommendation models. Specifically, these
influential EVs are Density, $Gini_{i}$, and the Standard Deviation of the
Popularity Bias (StPB).

###  6.3 Item Exposure

Lastly, we shift our focus to the assessment of item exposure, as measured by
the metric defined in Section 3.3. Notably, the $R^{2}$ values for this
analysis are remarkably high, exceeding $0.88$ in all cases, except for BPRMF,
where the EVs can still account for $0.78$ of the variance, as detailed in
Table 7. We would like to emphasize that higher values in the item exposure
metric indicate a larger disparity between the number of times items are
recommended and the number of times they should ideally be recommended (as
defined in Equation 18).

Table 7: Coefficients ($\theta_{ev}$) of the Regression Model for Average Item
Exposure@5

name
---
$R^{2}$
$R^{2}$ (adj)
SpaceSize
Shape
Density
$Cp_{i}$
$Gini_{i}$
StPB
KuPB
MedCC
StRG
KuDA

Popularity
---
0.934
0.93
0.68*
-3.758***
-1.7***
6.609***
0.474.
-1.396***
-0.239
1.013**
-0.425
1.595***

IB
---
0.897
0.891
-0.043
-3.88***
-1.313***
3.582***
0.912***
-0.747**
-0.157
-0.002
0.341
1.068***

UB
---
0.909
0.904
-0.487.
-4.052***
-1.08***
4.923***
0.965***
0.752*
-0.029
0.871*
-0.098
1.436***

PopGeoNN
---
0.884
0.877
0.294
-3.68***
-1.212***
5.302***
0.017
-0.011
-0.239
1.07**
-0.347
1.854***

GeoBPRMF
---
0.852
0.844
1.424***
-3.736***
-1.385***
4.082***
0.652.
-0.229
-0.12
1.388**
-0.164
2.155***

BPRMF
---
0.78
0.767
0.867.
-3.913***
-0.895.
5.353***
-0.316
1.64**
1.023**
0.101
0.717
1.8***

IRENMF
---
0.893
0.886
1.356***
-3.192***
-2.092***
3.748***
-0.319
-0.155
0.601**
1.16***
-0.273
1.487***

***: $p<0.001$, **: $p<0.01$, *: $p<0.05$

The coefficients that exhibit high levels of significance across all
recommendation models include Shape, Check-ins per Item ($Cp_{i}$), and the
Kurtosis of the Duration Active (KuDA). Density also emerges as highly
significant in most cases, except in BPRMF. In contrast, the Standard
Deviation of the Radius of Gyration (StRG) fails to achieve significance in
any of the models. Lastly, the significance of the coefficient for $Gini_{i}$
in the IB and UB models is noteworthy as this EV quantifies the inequality in
the frequency distribution of item check-ins.

Upon analyzing the grouping of coefficients and models in Figure 7, again
certain patterns emerge: Shape and Density consistently exert a negative
influence on Item Exposure within all recommendation models. As their values
increase, the item exposure metric decreases, which means that the item
exposure is closer to the expected item exposure from the test set. This
observation is in line with the intuition that a relatively higher number of
check-ins within the user-check-in matrix results in a more dispersed
distribution, which in turn helps mitigate the influence of popularity bias.
Conversely, the Kurtosis of the Duration Active (KuDA) consistently exhibits a
positive impact on the item exposure metric across all the recommendation
models. One plausible explanation for this trend is that when users spend
longer periods in a city, they are more likely to explore and visit a large
number of popular POIs, further accentuating the effect of popularity bias.
Unsurprisingly, the Check-ins per Item ($Cp_{i}$) coefficient is the largest
in the pure Popularity model, which signifies the issue with this model
regarding item exposure. As for the remaining Explanatory Variables, their
influence on the recommendation outcomes in terms of item exposure appears to
be minor and variable.

Figure 7: Coefficient plot for Average Item Exposure@5. Check-ins per item and
the kurtosis of the Duration Active have unisono positive coefficients, while
density and shape exert a negative influence on average item exposure, meaning
that the difference to the expected item exposure is smaller, which is better.

We emphasize that these metrics highlight the evident impact of popularity
bias within the domain. Notably, the Item Exposure metric in the Popularity
models is negatively influenced by a low Standard Deviation of the Popularity
Bias (StPB). Consequently, as the Standard Deviation of popularity bias
increases, the Pop recommender achieves improved results in terms of item
exposure.

##  7 Discussion

The obtained results shed light on the strengths and weaknesses of POI
recommendation models in terms of the data characteristics of the
recommendation problem. Using the domain-driven subsampling approach to create
specific subsamples from a large POI recommendation data set, we noticed that
several established POI recommendation models are unsuited for smaller
problems (cf. Figure 2a), which warranted their exclusion from the explanatory
analysis.

In terms of the quality of the linear model, it is striking that it was
possible to explain 80–86% variation of the nDCG with the remaining 10 out of
32 EVs after the elimination of the collinear EVs. Besides, we also obtained
high results in terms of explaining item exposure (77–93%) and novelty
(52–76%) variations. This result confirms that the collinearity analysis is
necessary and helps to focus on the interesting variables without losing
explainability. Interestingly, despite being the second best-performing
algorithm (cf. Figure 2a), the Item-based recommendation model is the least
impacted one from the data characteristics of the subsamples as it has the
lowest coefficients in the linear model for nDCG (Table 5). This suggests that
this model could be a promising option when dealing with situations where data
characteristics are either unknown or subject to rapid changes. This is
particularly relevant in cold-start scenarios or recommenders that exhibit a
strong variability in item characteristics and user preferences. This could be
the case with significant seasonal trends or destinations that are heavily
influenced by event tourism (Getz and Page, 2016).

In terms of novelty of recommendations, we see a similar trend as for the
accuracy; however, our results showed once more that these two concepts are
inverse due to the popularity bias and related trade-offs discussed in the
literature (Kaminskas and Bridge, 2016, Zhao et al., 2019). Unsurprisingly, we
could show that a higher variance in the popularity bias in the interaction
data helps to promote novel recommendations. Thus, these two targets still
need to be balanced in any POI recommender system in accordance with business
needs. Emphasizing these dimensions is also important from the user’s point of
view, as it would allow the system to surprise them with recommendations that
are different from what they may be familiar with beforehand.

Regarding Item Exposure, we observe negative coefficients of Density on this
metric, signifying that higher-density subsamples tend to yield lower values
of the item exposure metric. In this context, lower values are preferable, as
it means that they are closer to the exposed values in the unbiased test set.
However, we should also consider that an increased number of Check-ins per
Item leads to a higher item exposure than desired, as this variable always
obtains positive coefficients in the regression model. Therefore, a careful
trade-off should be found between Density and Check-ins per Item.
Consequently, a higher Standard Deviation of the Popularity Bias can alleviate
this issue. Platforms need to monitor such effects closely, as the exposure of
small local travel enterprises in the recommendations can determine whether
they have means of existence in the market.

###  7.1 Practical and Theoretical Implications

In this paper, we go beyond evaluating recommendation models by employing a
framework that reveals the associations between various data characteristics
and different dimensions of performance. While this framework is not without
its imperfections, as discussed below, its application carries significant
implications for the training and evaluation of POI recommender systems. For
instance, our observations highlight the impact of explanatory variables such
as SpaceSize and check-ins per item, which exhibit a negative influence on
nDCG@5, as opposed to EVs like StPB, $Gini_{i}$, and density, which display a
positive influence. Based on our findings, developers, analysts, and
researchers of POI recommender systems should be aware of the importance of
these data characteristics. Neglecting these variables may introduce bias into
the results, leading to overly optimistic or pessimistic outcomes, depending
on the actual characteristics of the data.

Furthermore, it is worth noting that the effect of these explanatory variables
is contingent on the specific recommendation algorithm employed. Researchers
should be mindful of this when comparing baseline methods against proposed
models. Unintentionally, they might report comparisons that are not fair
because some methods are either positively or negatively impacted by the data
characteristics. Notably, this observation extends to beyond-accuracy metrics,
as we have observed similar trends in novelty and item exposure.

From a more theoretical perspective, this work introduces the prospect of
learning these data dependencies directly from the data itself and
incorporating them into recommendation models and user profiles. As our
approach is entirely data-driven, the only ad-hoc decisions we made were
related to the selection of the original explanatory variables, which could
vary depending on the specific target domain. However, once the explanatory
and dependent variables are established, it becomes conceivable to integrate
the methodology presented here into a reinforcement learning framework. In
such a setup, data could be fed to the recommenders in accordance with the
predicted effects they are anticipated to have, thus optimizing recommendation
outcomes.

Even in the absence of a reinforcement learning framework, the current
methodology can provide valuable insights into designing and enhancing
existing recommendation algorithms, at least in the POI recommendation domain.
For instance, we observed that IB is less sensitive to Density than GeoBPRMF
across the three metrics analyzed. Given that their accuracy metrics are
similar, one way to leverage this insight is to explore how to make GeoBPRMF
more resilient in scenarios where data sets exhibit varying levels of density.
To achieve this, researchers and practitioners could consider incorporating
strategies from IB or similar robust methods into an improved version of the
GeoBPRMF algorithm. This integration might involve adapting the underlying
model or introducing additional mechanisms that allow GeoBPRMF to handle data
sets with different density levels more effectively. By doing so, GeoBPRMF
could become more versatile and capable of delivering consistent performance
across a wider range of data set characteristics, enhancing its practical
applicability in diverse POI recommendation scenarios.

Most importantly, whether it is hotels, restaurants, or other POI, this work
gives guidance to recommendation platforms of the e-tourism sector to
characterize their recommendation data and understand the benefits and
drawbacks of their recommendation model from the perspective of different
users groups and businesses. It can also inform methods to self-audit biases
in the recommendations of platforms, e.g., with regards to the expected and
achieved item exposure of items in certain recommendation models (Srba et al.,
2023).

###  7.2 Limitations

A common drawback of offline analyses of POI recommendation algorithms using
location-based social network data is a gap between the user behavior and what
is recorded in LBSNs – the type of data all studies in this field used.
Although this is universal for all recommendation models in all studies, it is
worth mentioning that such studies analyze a proxy concept of check-ins that
have been actively submitted instead of the actual ground truth of the
temporal visitation of all POIs, venues, and other places. Thus, the
generalization of the analyses in this study is – just as in all other studies
– constrained by the limited availability of high-quality POI recommendation
data sets. Although we utilized the widely employed global Foursquare data set
containing 33 million check-ins (Sánchez and Bellogín, 2022), its sparsity
posed a significant challenge since the actual number of check-ins per
destination dwindled, limiting our ability to create meaningful subsamples for
most of the cities within the data set. To prevent biases stemming from
geographic influences of the topological features of different cities, we
focused our study on a single destination, the New York City metropolitan
area, due to its prominence and the highest number of check-ins in the data
set. This decision to focus on one city was essential because many less
popular destinations on Foursquare lacked the volume of check-in data to
support the explanatory analysis with subsamples of sufficient size, which
would lead to a deterioration in recommendation quality and lower statistical
significance. In fact, while New York City provided ample interaction data for
addressing the recommendation problem, it is a city less susceptible to
seasonal variations in travel behavior, as evidenced by the minor differences
in data characteristics between the summer and winter subsamples shown in
Table 1, hence the obtained results cannot be considered universal, since
cities from different nature and touristic behavior were missing from our
analysis.

##  8 Conclusions & Future Work

In both the classical recommendation domain and specialized fields, such as
point of interest recommendation, a multitude of algorithms have been
proposed. However, the effectiveness of these algorithms often varies
significantly depending on the data set under evaluation. While addressing
this challenge has been explored previously in classical recommendation
scenarios, there remains a research gap within the POI recommendation domain.
This gap is particularly significant, given the relevance of POI
recommendations in the tourism sector, affecting a multitude of businesses and
consumers. Unlike the traditional recommendation domain, like purchasing a
book or watching a movie, recommending POI involves algorithms that utilize
various signals, such as popularity and geographic locations of venues.

In this paper, we expanded upon the framework introduced by Deldjoo et al.
(2021) incorporating additional explanatory variables specific to the POI
recommendation domain. Our objective was to investigate how data
characteristics affect the performance of both classical and state-of-the-art
POI recommendation algorithms. We assessed these algorithms across three
dimensions: accuracy, novelty, and item exposure. The results we obtained shed
light on the robustness of recommendation models concerning the data
characteristics inherent in recommendation data sets, offering valuable
insights for the field. Among the various explanatory variables we analyzed,
it became evident that certain factors pertaining to the data structure
(SpaceSize, Shape, Density, and $Cp_{i}$), as well as those associated with
the distribution of check-ins ($Gini_{i}$, StPB, and KuPB), along with
geographical influence variables (MedDCC, STRG, and KuDA) are critical for
explaining the performance of recommendation systems in terms of ranking
accuracy. Furthermore, Density and $Cp_{i}$ also exert a significant influence
on both novelty and item exposure, among others. For instance, StPB
significantly influences novelty, while KuDA plays a more substantial role in
item exposure. The significance of these newly proposed explanatory variables
underlines our conclusion that the well-known data characteristics from the
analysis of classical recommendation domains are insufficient to explain
algorithmic performance in the POI recommendation domain.

In our future work, we intend to integrate the explanatory variables with
further user groups that can be identified in the data sets. Specifically,
since it has been shown that locals and tourists showcase different behavior
in the same city (Sánchez and Dietz, 2022), it would be worthwhile identifying
which explanatory variables are more relevant for various user groups. We also
plan to extend this analysis towards users with highly irregular activity in
the system, which is a distinct challenge within the POI recommendation (Chen
et al., 2022). Finally, we plan to extend our analysis to diverse travel
destinations, including summer beach destinations and historically-rich
cities. This would determine the generalizability of our findings but would
also enable us to study the performance of recommendation models in
destinations with diverging characteristics shaped by seasonal mass tourism.

Unfortunately, large-scale open data sets in this domain are generally scarce
and stem from location-based social networks, such as Foursquare (used in our
work), instead of POI recommendation systems. Transitioning towards data sets
that reflect user interactions from actual POI recommendation systems would be
a crucial next step in advancing the field.

## Data Availability

The Foursquare data set used in this study is available from Dingqi Yang’s web
page https://sites.google.com/site/yangdingqi/home/foursquare-dataset. Further
intermediate results are available in the supplementary material shared with
this paper.

## Conflict of Interest

The authors have no competing interests to declare that are relevant to the
content of this article.

## References

  * Im and Hars (2007) I. Im, A. Hars,  Does a one-size recommendation system fit all? the effectiveness of collaborative filtering based recommendation systems across different domains and search modes,  ACM Transactions on Information Systems 26 (2007) 4. doi:10.1145/1292591.1292595.
  * Anelli et al. (2022) V. W. Anelli, A. Bellogín, T. D. Noia, D. Jannach, C. Pomo,  Top-n recommendation algorithms: A quest for the state-of-the-art,  in: 30th ACM Conference on User Modeling, Adaptation and Personalization, UMAP’22, ACM, New York, NY, USA, 2022. doi:10.1145/3503252.3531292.
  * Gunawardana and Shani (2015) A. Gunawardana, G. Shani,  Evaluating recommender systems,  in: Recommender Systems Handbook, Springer, 2015, pp. 265–308. doi:10.1007/978-1-4899-7637-6_8.
  * Castells et al. (2015) P. Castells, N. J. Hurley, S. Vargas,  Novelty and diversity in recommender systems,  in: Recommender Systems Handbook, Springer, 2015, pp. 881–918. doi:10.1007/978-1-4899-7637-6_26.
  * Ekstrand et al. (2022) M. D. Ekstrand, A. Das, R. Burke, F. Diaz,  Fairness in recommender systems,  in: Recommender Systems Handbook, Springer US, 2022, pp. 679–707. doi:10.1007/978-1-0716-2197-4_18.
  * Isufi et al. (2021) E. Isufi, M. Pocchiari, A. Hanjalic,  Accuracy-diversity trade-off in recommender systems via graph convolutions,  Inf. Process. Manag. 58 (2021) 102459. doi:10.1016/j.ipm.2020.102459.
  * Sánchez and Bellogín (2022) P. Sánchez, A. Bellogín,  Point-of-interest recommender systems based on location-based social networks: A survey from an experimental perspective,  ACM Computing Surveys (2022). doi:10.1145/3510409.
  * Sánchez and Dietz (2022) P. Sánchez, L. W. Dietz,  Travelers vs. locals: The effect of cluster analysis in point-of-interest recommendation,  in: 30th ACM Conference on User Modeling, Adaptation and Personalization, UMAP’22, ACM, New York, NY, USA, 2022, pp. 132–142. doi:10.1145/3503252.3531320.
  * Deldjoo et al. (2021) Y. Deldjoo, A. Bellogín, T. D. Noia,  Explaining recommender systems fairness and accuracy through the lens of data characteristics,  Inf. Process. Manag. 58 (2021) 102662. doi:10.1016/j.ipm.2021.102662.
  * Dacrema et al. (2019) M. F. Dacrema, P. Cremonesi, D. Jannach,  Are we really making much progress? A worrying analysis of recent neural recommendation approaches,  in: RecSys, ACM, 2019, pp. 101–109. doi:10.1145/3298689.3347058.
  * Said and Bellogín (2014) A. Said, A. Bellogín,  Comparative recommender system evaluation: benchmarking recommendation frameworks,  in: RecSys, ACM, 2014, pp. 129–136. doi:10.1145/2645710.2645746.
  * Meng et al. (2020) Z. Meng, R. McCreadie, C. Macdonald, I. Ounis,  Exploring data splitting strategies for the evaluation of recommendation models,  in: RecSys, ACM, 2020, pp. 681–686. doi:10.1145/3383313.3418479.
  * Anelli et al. (2019) V. W. Anelli, T. D. Noia, E. D. Sciascio, C. Pomo, A. Ragone,  On the discriminative power of hyper-parameters in cross-validation and how to choose them,  in: RecSys, ACM, 2019, pp. 447–451. doi:10.1145/3298689.3347010.
  * Deldjoo et al. (2020) Y. Deldjoo, T. D. Noia, E. D. Sciascio, F. A. Merra,  How dataset characteristics affect the robustness of collaborative recommendation models,  in: SIGIR, ACM, 2020, pp. 951–960. doi:10.1145/3397271.3401046.
  * Adomavicius and Zhang (2012) G. Adomavicius, J. Zhang,  Impact of data characteristics on recommender systems performance,  ACM Transactions on Management Information Systems 3 (2012) 1–17. doi:10.1145/2151163.2151166.
  * Adomavicius and Tuzhilin (2015) G. Adomavicius, A. Tuzhilin,  Context-aware recommender systems,  in: Recommender Systems Handbook, Springer, 2015, pp. 191–226. doi:10.1007/978-1-4899-7637-6_6.
  * Manotumruksa et al. (2018) J. Manotumruksa, C. Macdonald, I. Ounis,  A contextual attention recurrent architecture for context-aware venue recommendation,  in: SIGIR, ACM, 2018, pp. 555–564. doi:10.1145/3209978.3210042.
  * Li et al. (2015) X. Li, G. Cong, X. Li, T. N. Pham, S. Krishnaswamy,  Rank-geofm: A ranking based geographical factorization method for point of interest recommendation,  in: SIGIR, ACM, 2015, pp. 433–442. doi:10.1145/2766462.2767722.
  * Griesner et al. (2015) J. Griesner, T. Abdessalem, H. Naacke,  POI recommendation: Towards fused matrix factorization with geographical and temporal influences,  in: RecSys, ACM, 2015, pp. 301–304. doi:10.1145/2792838.2799679.
  * Zhang and Chow (2015) J. Zhang, C. Chow,  Geosoca: Exploiting geographical, social and categorical correlations for point-of-interest recommendations,  in: SIGIR, ACM, 2015, pp. 443–452.
  * Liu et al. (2014) Y. Liu, W. Wei, A. Sun, C. Miao,  Exploiting geographical neighborhood characteristics for location recommendation,  in: CIKM, ACM, 2014, pp. 739–748. doi:10.1145/2661829.2662002.
  * Wang et al. (2020) Q. Wang, H. Yin, T. Chen, Z. Huang, H. Wang, Y. Zhao, N. Q. V. Hung,  Next point-of-interest recommendation on resource-constrained mobile devices,  in: WWW, ACM / IW3C2, 2020, pp. 906–916. doi:10.1145/3366423.3380170.
  * Santos et al. (2019) F. Santos, A. de Almeida, C. Martins, R. Gonçalves, J. Martins,  Using POI functionality and accessibility levels for delivering personalized tourism recommendations,  Comput. Environ. Urban Syst. 77 (2019). doi:10.1016/j.compenvurbsys.2017.08.007.
  * Cortés-Jiménez (2008) I. Cortés-Jiménez,  Which type of tourism matters to the regional economic growth? the cases of spain and italy,  International Journal of Tourism Research 10 (2008) 127–139. doi:10.1002/jtr.646.
  * Brilhante et al. (2013) I. R. Brilhante, J. A. F. de Macêdo, F. M. Nardini, R. Perego, C. Renso,  Where shall we go today?: planning touristic tours with tripbuilder,  in: CIKM, ACM, 2013, pp. 757–762. doi:10.1145/2505515.2505643.
  * Domènech et al. (2020) A. Domènech, I. Mohino, B. Moya-Gómez,  Using flickr geotagged photos to estimate visitor trajectories in world heritage cities,  ISPRS Int. J. Geo Inf. 9 (2020) 646. doi:10.3390/ijgi9110646.
  * Ning et al. (2015) X. Ning, C. Desrosiers, G. Karypis,  A comprehensive survey of neighborhood-based recommendation methods,  in: Recommender Systems Handbook, Springer, 2015, pp. 37–76. doi:10.1007/978-1-4899-7637-6_2.
  * Abdollahpouri et al. (2019) H. Abdollahpouri, R. Burke, B. Mobasher,  Managing popularity bias in recommender systems with personalized re-ranking,  in: FLAIRS Conference, AAAI Press, 2019, pp. 413–418.
  * Cremonesi et al. (2010) P. Cremonesi, Y. Koren, R. Turrin,  Performance of recommender algorithms on top-n recommendation tasks,  in: RecSys, ACM, 2010, pp. 39–46. doi:10.1145/1864708.1864721.
  * Cheng et al. (2016) C. Cheng, H. Yang, I. King, M. R. Lyu,  A unified point-of-interest recommendation framework in location-based social networks,  ACM TIST 8 (2016) 10:1–10:21. doi:10.1145/2901299.
  * Wang et al. (2020) W. Wang, J. Chen, J. Wang, J. Chen, J. Liu, Z. Gong,  Trust-enhanced collaborative filtering for personalized point of interests recommendation,  IEEE Trans. Ind. Informatics 16 (2020) 6124–6132. doi:10.1109/TII.2019.2958696.
  * Huang et al. (2020) L. Huang, Y. Ma, Y. Liu, A. K. Sangaiah,  Multi-modal bayesian embedding for point-of-interest recommendation on location-based cyber-physical-social networks,  Future Gener. Comput. Syst. 108 (2020) 1119–1128. doi:10.1016/j.future.2017.12.020.
  * Zhao et al. (2019) P. Zhao, H. Zhu, Y. Liu, J. Xu, Z. Li, F. Zhuang, V. S. Sheng, X. Zhou,  Where to go next: A spatio-temporal gated network for next POI recommendation,  in: AAAI, AAAI Press, 2019, pp. 5877–5884. doi:10.1609/aaai.v33i01.33015877.
  * Ji et al. (2022) Y. Ji, A. Sun, J. Zhang, C. Li,  A critical study on data leakage in recommender system offline evaluation,  ACM Transactions on Information Systems (2022). doi:10.1145/3569930.
  * Zhao et al. (2015) X. Zhao, X. Li, L. Liao, D. Song, W. K. Cheung,  Crafting a time-aware point-of-interest recommendation via pairwise interaction tensor factorization,  in: KSEM, volume 9403 of Lecture Notes in Computer Science, Springer, 2015, pp. 458–470. doi:10.1007/978-3-319-25159-2_41.
  * Yin et al. (2015) H. Yin, B. Cui, L. Chen, Z. Hu, C. Zhang,  Modeling location-based user rating profiles for personalized recommendation,  TKDD 9 (2015) 19:1–19:41. doi:10.1145/2663356.
  * Ekstrand and Kluver (2021) M. D. Ekstrand, D. Kluver,  Exploring author gender in book rating and recommendation,  User Model. User Adapt. Interact. 31 (2021) 377–420. doi:10.1007/s11257-020-09284-2.
  * Melchiorre et al. (2021) A. B. Melchiorre, N. Rekabsaz, E. Parada-Cabaleiro, S. Brandl, O. Lesota, M. Schedl,  Investigating gender fairness of recommendation algorithms in the music domain,  Information Processing & Management 58 (2021) 102666. doi:10.1016/j.ipm.2021.102666.
  * Cañamares and Castells (2017) R. Cañamares, P. Castells,  A probabilistic reformulation of memory-based collaborative filtering: Implications on popularity biases,  in: SIGIR, ACM, 2017, pp. 215–224. doi:10.1145/3077136.3080836.
  * Weydemann et al. (2019) L. Weydemann, D. Sacharidis, H. Werthner,  Defining and measuring fairness in location recommendations,  in: LocalRec@SIGSPATIAL, ACM, 2019, pp. 6:1–6:8. doi:10.1145/3356994.3365497.
  * Nikolakopoulos et al. (2022) A. N. Nikolakopoulos, X. Ning, C. Desrosiers, G. Karypis, Trust Your Neighbors: A Comprehensive Survey of Neighborhood-based Methods for Recommender Systems, Springer, New York, NY, 2022, pp. 39–89. doi:10.1007/978-1-0716-2197-4_2.
  * Rahmani et al. (2022) H. A. Rahmani, Y. Deldjoo, A. Tourani, M. Naghiaei,  The unfairness of active users and popularity bias in point-of-interest recommendation,  in: L. Boratto, S. Faralli, M. Marras, G. Stilo (Eds.), Advances in Bias and Fairness in Information Retrieval, Springer, Cham, 2022, pp. 56–68. doi:10.1007/978-3-031-09316-6_6.
  * Massimo and Ricci (2021) D. Massimo, F. Ricci,  Popularity, novelty and relevance in point of interest recommendation: An experimental analysis,  Information Technology & Tourism 23 (2021) 473–508. doi:10.1007/s40558-021-00214-5.
  * Yin et al. (2012) H. Yin, B. Cui, J. Li, J. Yao, C. Chen,  Challenging the long tail recommendation,  VLDB Endowment 5 (2012) 896–907. doi:10.14778/2311906.2311916.
  * Abdollahpouri et al. (2017) H. Abdollahpouri, R. Burke, B. Mobasher,  Controlling popularity bias in learning-to-rank recommendation,  in: 11th ACM Conference on Recommender Systems, RecSys’17, ACM, New York, NY, USA, 2017, pp. 42–46. doi:10.1145/3109859.3109912.
  * González et al. (2008) M. C. González, C. A. Hidalgo, A.-L. Barabási,  Understanding individual human mobility patterns,  Nature 453 (2008) 779–782. doi:10.1038/nature06958.
  * Järvelin and Kekäläinen (2002) K. Järvelin, J. Kekäläinen,  Cumulated gain-based evaluation of IR techniques,  ACM Trans. Inf. Syst. 20 (2002) 422–446. doi:10.1145/582415.582418.
  * Vargas and Castells (2011) S. Vargas, P. Castells,  Rank and relevance in novelty and diversity metrics for recommender systems,  in: RecSys, ACM, 2011, pp. 109–116. doi:10.1145/2043932.2043955.
  * Liu and Zheng (2020) S. Liu, Y. Zheng,  Long-tail session-based recommendation,  in: RecSys, ACM, 2020, pp. 509–514. doi:10.1145/3383313.3412222.
  * Shih et al. (2016) T. Shih, T. Hou, J. Jiang, Y. Lien, C. Lin, P. Cheng,  Dynamically integrating item exposure with rating prediction in collaborative filtering,  in: SIGIR, ACM, 2016, pp. 813–816. doi:10.1145/2911451.2914769.
  * Sánchez and Bellogín (2021) P. Sánchez, A. Bellogín,  On the effects of aggregation strategies for different groups of users in venue recommendation,  Information Processing & Management 58 (2021) 102609. doi:10.1016/j.ipm.2021.102609.
  * Gao et al. (2015) H. Gao, J. Tang, X. Hu, H. Liu,  Content-aware point of interest recommendation on location-based social networks,  in: AAAI, AAAI Press, 2015, pp. 1721–1727. doi:10.1609/aaai.v31i1.9462.
  * Yuan et al. (2013) Q. Yuan, G. Cong, Z. Ma, A. Sun, N. Magnenat-Thalmann,  Time-aware point-of-interest recommendation,  in: SIGIR, ACM, 2013, pp. 363–372. doi:10.1145/2484028.2484030.
  * Yuan et al. (2014) Q. Yuan, G. Cong, A. Sun,  Graph-based point-of-interest recommendation with geographical and temporal influences,  in: CIKM, ACM, 2014, pp. 659–668. doi:10.1145/2661829.2661983.
  * Yao et al. (2015) L. Yao, Q. Z. Sheng, Y. Qin, X. Wang, A. Shemshadi, Q. He,  Context-aware point-of-interest recommendation using tensor factorization with social regularization,  in: SIGIR, ACM, 2015, pp. 1007–1010. doi:10.1145/2766462.2767794.
  * Nunes and Marinho (2014) I. Nunes, L. B. Marinho,  A personalized geographic-based diffusion model for location recommendations in LBSN,  in: LA-WEB, IEEE Computer Society, 2014, pp. 59–67.
  * Feng et al. (2015) S. Feng, X. Li, Y. Zeng, G. Cong, Y. M. Chee, Q. Yuan,  Personalized ranking metric embedding for next new POI recommendation,  in: IJCAI, AAAI Press, 2015, pp. 2069–2075.
  * Li et al. (2016) H. Li, Y. Ge, R. Hong, H. Zhu,  Point-of-interest recommendations: Learning potential check-ins from friends,  in: KDD, ACM, 2016, pp. 975–984. doi:10.1145/2939672.2939767.
  * Sánchez et al. (2023) P. Sánchez, A. Bellogín, L. Boratto,  Bias characterization, assessment, and mitigation in location-based recommender systems,  Data Min. Knowl. Discov. 37 (2023) 1885–1929. doi:10.1007/s10618-022-00913-5.
  * Jannach et al. (2015) D. Jannach, L. Lerche, I. Kamehkhosh, M. Jugovac,  What recommenders recommend: an analysis of recommendation biases and possible countermeasures,  User Modeling and User-Adapted Interaction 25 (2015) 427–491. doi:10.1007/s11257-015-9165-3.
  * Yao et al. (2016) Z. Yao, Y. Fu, B. Liu, Y. Liu, H. Xiong,  POI recommendation: A temporal matching between POI popularity and user regularity,  in: 16th International Conference on Data Mining, IEEE, 2016, pp. 549–558. doi:10.1109/icdm.2016.0066.
  * Liu et al. (2011) Q. Liu, Y. Ge, Z. Li, E. Chen, H. Xiong,  Personalized travel package recommendation,  in: IEEE 11th International Conference on Data Mining, ICDM ’11, IEEE, Vancouver, BC, Canada, 2011, pp. 407–416. doi:10.1109/icdm.2011.118.
  * Trattner et al. (2018) C. Trattner, A. Oberegger, L. B. Marinho, D. Parra,  Investigating the utility of the weather context for point of interest recommendations,  Information Technology & Tourism 19 (2018) 117–150. doi:10.1007/s40558-017-0100-9.
  * Wörndl et al. (2011) W. Wörndl, J. Hübner, R. Bader, D. Gallego-Vico,  A model for proactivity in mobile, context-aware recommender systems,  in: Fifth ACM Conference on Recommender Systems, RecSys’11, ACM, 2011. doi:10.1145/2043932.2043981.
  * Cai et al. (2017) L. Cai, J. Xu, J. Liu, T. Pei,  Integrating spatial and temporal contexts into a factorization model for POI recommendation,  International Journal of Geographical Information Science 32 (2017) 524–546. doi:10.1080/13658816.2017.1400550.
  * Yang et al. (2017) C. Yang, L. Bai, C. Zhang, Q. Yuan, J. Han,  Bridging collaborative filtering and semi-supervised learning: A neural approach for poi recommendation,  in: 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, KDD’17, ACM, New York, NY, USA, 2017, p. 1245–1254. doi:10.1145/3097983.3098094.
  * Gibson and Yiannakis (2002) H. Gibson, A. Yiannakis,  Tourist roles: Needs and the lifecourse,  Annals of Tourism Research 29 (2002) 358–383.
  * Neidhardt et al. (2014) J. Neidhardt, R. Schuster, L. Seyfang, H. Werthner,  Eliciting the users’ unknown preferences,  in: 8th ACM Conference on Recommender Systems, RecSys ’14, ACM, New York, NY, USA, 2014, pp. 309–312. doi:10.1145/2645710.2645767.