# Measuring the Eccentricity of Items

Chanyoung Park  School of Integrated Technology
Yonsei University
Incheon, Korea
Email<EMAIL_ADDRESS>Songkuk Kim  School of Integrated Technology
Yonsei University
Incheon, Korea
Email<EMAIL_ADDRESS>
###### Abstract

The long-tail phenomenon tells us that there are many items in the tail.
However, not all tail items are the same. Each item acquires different kinds
of users. Some items are loved by the general public, while some items are
consumed by eccentric fans. In this paper, we propose a novel metric, item
eccentricity, to incorporate this difference between consumers of the items.
Eccentric items are defined as items that are consumed by eccentric users. We
used this metric to analyze two real-world datasets of music and movies and
observed the characteristics of items in terms of eccentricity. The results
showed that our defined eccentricity of an item does not change much over
time, and classified eccentric and noneccentric items present significantly
distinct characteristics. The proposed metric effectively separates the
eccentric and noneccentric items mixed in the tail, which could not be done
with the previous measures, which only consider the popularity of items.

©2017 IEEE. Personal use of this material is permitted. Permission from IEEE
must be obtained for all other uses, in any current or future media, including
reprinting/republishing this material for advertising or promotional purposes,
creating new collective works, for resale or redistribution to servers or
lists, or reuse of any copyrighted component of this work in other works.

##  I Introduction

It is widely acknowledged that items in various markets follow a long-tailed
distribution. A long-tailed distribution suggests the importance of tail
items, which in aggregate make up a significant portion of the total market.
Such less-popular items are important not only because of their market size
but also because of the information they give about the users. Every person
consumes their own special items, and these eccentric ones may imply more
about the users’ preference than other popular items.

However, not every less-popular item is special. The tail of the market has
many items, and they are not all the same. Each item acquires different kinds
of users. Even with the same number of users, items may have different types
of users. Some items are loved by the general public, while some are mostly
consumed by eccentric fans. This difference in consuming users is more
prominent in less-popular items because they do not have as many users as
popular items do, and are more likely to appeal to a certain group of users.
Despite this intuitive difference in consumers between less-popular items,
there have not been many studies that explored and captured this
characteristic.

The importance of understanding unpopular items has been widely recognized by
the field of recommender systems (RS). Novelty has been regarded as an
important aspect of RS because people seek not only relevant but also
unexpected recommendations. Several user studies have shown that a small
increase in accuracy does not necessarily lead to a higher user satisfaction
[1, 2], while serendipity notably influences user contentment. Therefore, many
approaches have been proposed to make novel recommendations [3, 4].

In the previous studies on the serendipity of RS, a trade-off between accuracy
and diversity has been reported [5, 6, 7, 8]. To mitigate the accuracy loss of
incorporating serendipity, various methods to manage tail items have been
proposed. As one of the successful approaches, Park et al. [5] have clustered
tail items into several groups and performed a regression within the group.
The improvement made by clustering tail items has implied that there are
different kinds of items mixed in the long tail. This supports our earlier
explanation on the less-popular items.

In this paper, we therefore propose a new metric of item characteristics
called item eccentricity rather than a simple popularity represented by the
number of feedbacks an item has. We assumed that the core difference between
the tail items lies in who are the consuming users. An item is called rare
when a small number of people have consumed the item. Eccentric items have
something more. Their users must be eccentric too. The coconsumer’s usual
behavior, like whether they are following mainstream or being an enthusiast of
a niche genre, implies the degree of an item’s eccentricity. The item
eccentricity we propose considers the characteristics of users, and captures
the significant difference between two similar terms, rarity and eccentricity.

Accordingly, we first start by defining the users’ personal long-term degree
of preference on rare items and refer to it as user eccentricity. We then
define the item eccentricity as the weighted average of user eccentricity
(Section III). Using the proposed metrics, we analyze the two real-world
datasets, Last.fm and MovieLens, to demonstrate that our metric effectively
separates the different tail items in terms of eccentricity (Section IV). We
investigate whether the eccentricity of items is inherent and therefore
maintained regardless of time, and see what is different between the eccentric
and noneccentric items. Finally, we conclude in Section V.

##  II Related Work

In this section, we briefly review related works. We first explain how other
studies have worked to incorporate serendipity in recommendations and why a
metric to measure such a characteristic is needed. Afterward, we investigate
the previous approaches of defining and measuring the item eccentricity.

Popularity Bias and Serendipitous Recommendation Many studies have shown that
the item popularity in various markets follows a long-tailed distribution [9,
10, 11]. Items in the head dominate the interaction between users and items,
while a large number of items are located in the tail. People are more likely
to consume and produce feedback on the head items than on the tail items. We
call this bias to popular items popularity bias [12]. Many RSs also tend to
recommend popular items over unpopular items because they have enough ratings
to be used by recommender algorithms, thus they can make a more accurate
recommendation. Several researchers have demonstrated the trade-off between
accuracy and serendipity in the recommendation [6, 7, 12].

Despite the relatively low performance on tail items, the importance of
serendipity in RS has been shown by several user studies [13, 14, 2, 15]. In
the studies, users are reported to seek both surprising and interesting
recommendations, which we call serendipity [16]. The less-popular items may
take up a small part of the overall interaction between users and items.
However, each person still consumes their own unpopular but unique items for a
significant proportion. To reduce the decrease in accuracy of including
serendipity in RS, various methods have been proposed. Steck [12] used a
popularity-stratified training method to correct for popularity bias in the
dataset. Lu [17] proposed a serendipitous personalized ranking model, which
optimizes both accuracy and serendipity in a single learning procedure. Park
and Tuzhilin [5] grouped tail items using clustering methods and improved the
recommendation accuracy. The above works imply that there are groups of
different items that are mixed in the tail part, and we need different ranking
systems and methods to incorporate the differences between items. Besides the
simple popularity, the need for defining the new metric that shows a different
perspective of items has been recognized.

Item Eccentricity There have been many approaches to measuring the extent to
which the recommendations are serendipitous [18, 19]. However, the metric on
the item novelty has a relatively small number of studies, although it is
important in understanding tail items. Nakatsuji et al. [20] proposed the
Random Walk and Restart method over the user graph, and suggested how to find
the novel items from relevant users, yet did not quantitatively measure the
novelty. Marques et al. [21] saw the novelty of items as having two other
aspects besides being unexplored by a user: familiarity and mainstreamness.
They defined the mainstreamness of music items as the log of the overall
popularity measured by total play count. Similarly, Vargas [19] defined item
novelty as the log of the inverse popularity. These metrics regarded every
item with few users as novel and a deviation from the mainstream. However, not
every unpopular item is an eccentric item. Even with the same number of users,
items may have different types of users. The two terms, rarity and
eccentricity, are often used interchangeably, yet they are different because
the core concept of niche or eccentric depends on who are the coconsumers.
Therefore, to measure the eccentricity of items, a metric that takes
characteristics of interacted users into account is required.

Figure 1: Kernel density function of eccentricity metrics in the two datasets.
30 and ML in the legend denotes the 30Music and MovieLens datasets,
respectively.

##  III Definition

In this section, we present the definition of item eccentricity. We first
introduce the item rarity, which represents the popularity of items as the
well-known IDF term. This is similar to the previous works [19, 21], but we
use this term to indicate the rarity, not the novelty of items. We then define
the user eccentricity as a user’s long-term tendency toward tail items. We
define it by how much a user consumes unpopular items and how much they like
the unpopular ones. This assumes that users who consistently consume and give
positive ratings on unpopular items are eccentric. Finally, we propose the
item eccentricity as the average of user eccentricity of consumers weighted by
their ratings. We elaborate the definition of the three metrics in the
following subsections.

###  III-A Item Rarity

We assume that there is a set of users $U={u_{1},...,u_{N}}$, a set of items
$I={i_{1},...,i_{M}}$, a set of time windows $T={t_{1},...,t_{L}}$, and the
set of known feedbacks $F={f_{1,1,1},...,f_{u,i,t},...,f_{N,M,L}}$ produced by
the customers in $U$ for the items in $I$ in the time windows in $T$. Time
windows are set to reflect the rise and fall nature of item popularity. Many
items become less popular after the release and consuming the items in a
different period should be treated differently in later processes. The size of
the time window can be determined differently depending on which dataset is
being used. We conducted preliminary research to find out which size provides
the highest stability of popularity within the time window and moderate
variability between the time windows. Our results demonstrated that four weeks
was the most suitable for the window size among the choices 1,2,3,4,6,12
weeks. Therefore, in the rest of our analysis, we set the time windows to four
weeks, where we approximately refer to it as one month. The feedbacks can be
either explicit, like five-scale ratings, or implicit, such as the number of
plays on a track.

With explained notations, the rarity of an item $i$ in time window $t$ can be
defined as the negative log of the number of feedbacks it has:

$Item\,\,Rarity_{i,t}\,(IR_{i,t})=-log\,|F_{i,t}|,$ (1)

where $|F_{i,t}|$ indicates the number of feedbacks on item $i$ in time window
$t$. In the definition, we used the number of feedbacks on each item rather
than a sum or average of feedbacks because the number of users is more
directly related to the popularity. After applying equation 1, we standardized
the resulting values to a $z$-score. Unless otherwise noted, we use this
standardized value when we speak of the item rarity for the remainder of the
paper. When we refer to one representative rarity value of an item, $IR_{i}$,
it is the average value of the rarity over all time windows of the item.

###  III-B User Eccentricity

Eccentric users are characterized by their long-term tendency toward rare
items. It can be measured by two factors: degree of consuming unpopular items
over popular items and preference on such items presented by user’s feedback.
On each user’s item list, the degree of consuming unpopular items can be
measured as an average rarity of the items and use the feedback of the user to
give the weight on each item’s rarity:

$User\,\,Eccentricity_{u}\,(UE_{u})=\frac{\sum_{i\in I_{u},t\in
T}f_{u,i,t}\cdot IR_{i,t}}{\sum_{i\in I_{u},t\in T}f_{u,i,t}}$ (2)

, where $I_{u}$ indicates the set of items that user $u$ has consumed and the
$IR_{i,t}$ is the item rarity of item $i$ in time window $t$. The value of
user eccentricity is also standardized to a $z$-score as the item rarity was.
Users with higher eccentricity consume a greater number of rare items and give
more positive feedback on such items.

###  III-C Item Eccentricity

Based on the user eccentricity measured, we can finally define the item
eccentricity. We can express the item eccentricity as how much an item is
consumed and liked by eccentric users, which is an average user eccentricity
weighted by the feedback on the items:

$Item\,\,Eccentricity_{i}\,(IE_{i})=\frac{\sum_{u\in U_{i}}f_{u,i}\cdot
UE_{u}}{\sum_{u\in U_{i}}f_{u,i}}$ (3)

. $U_{i}$ is the set of users who have consumed the item $i$ and the $UE_{u}$
is the measured eccentricity of user $u$. Item eccentricity is also
standardized as other previous metrics were.

##  IV Analysis on Eccentric Items

In this section, we present the analysis results on two real-world datasets
using the proposed metrics. Through the analysis, we show the stability of
item eccentricity and the difference between eccentric and noneccentric items
to demonstrate our metrics’ validity.

###  IV-A Datasets

In our analysis, we use two datasets of different domains: the 30Music dataset
[22] and the MovieLens 20M dataset. Table I describes the detailed statistics
of the datasets.

30Music Dataset A publicly available dataset of user behavior logs collected
from Last.fm, an online music streaming service. It provides data about users,
tracks, and the occurred music listening events as tuples of the form <user,
track, timestamp>. It also provides a small amount of loved track information
collected from users.

MovieLens 20M A user-movie ratings dataset collected from MovieLens. It
represents each rating tuple of the form <user, movie, rating, timestamp>.

The 30Music dataset exhibits a greater number of items and a more extreme long
tail. Another difference between the two datasets is the type of feedback
given by users. MovieLens provides explicit feedbacks as 0.5–5 rating while
Last.fm only provides implicit feedbacks like listening events. As we need a
set of numeric feedbacks $F$ in our metric definitions, for the 30Music, we
define the $f_{u,i,t}$ as the number of times the user $u$ listened to the
track $i$ in the time window $t$.

(a) 30Music Dataset

(b) MovieLens Dataset

Figure 2: Graph of item eccentricity against item rarity. The red line
indicates the regression line of the points. TABLE I: Statistical information
of the two datasets. Row corresponding to feedback density shows the data
density of the user-item matrix.

| 30Music | MovieLens
---|---|---
the number of logs | 31.4M | 20M
$|U|$ | 45K | 138K
$|I|$ | 5.7M | 27K
value range of $f$ | 1-1504 | 0.5-5
average value of $f$ | 1.45 | 3,53
feedback density(%) | 0.009 | 0.540

###  IV-B Distribution of Eccentricity

In our analysis, we excluded items whose rarity value is zero for every time
window. These items have only one user for each time window and are not
suitable to be analyzed for eccentricity because such items do not have
coconsumers and solely depend on one user’s eccentricity. Figure 1 shows the
kernel density of standardized user eccentricity and item eccentricity in the
two datasets. It presents the number of items and users on each eccentricity
value. All four eccentricity metrics from the two datasets have the same mean
of zero, and the same standard deviation of one because all of them are
standardized $z$-scores.

Because the user eccentricity is an average of item rarity weighted by the
feedback, the user eccentricity does not follow the long-tail distribution as
the item rarity does. It is relatively equally distributed over a wide range,
which shows that users have different degrees of preferring the tail items.
The item eccentricity is the average user eccentricity weighted by users’
feedback. A shift of mode to lower value in item eccentricity illustrates the
popularity bias explained earlier. People tend to give better feedback on
noneccentric items, and the most general eccentricity of the items becomes
less eccentric compared with user eccentricity. Therefore, the item
eccentricity becomes more concentrated on the less eccentric region. However,
the degree of concentration on mode differed by the dataset. Items in the
30Music dataset were more concentrated on the mode compared with the items in
the MovieLens.

###  IV-C Rarity and Eccentricity

In this subsection, we compare the rarity and eccentricity of items and
determine whether the eccentric and noneccentric items are genuinely
different.

People do not make sharp distinctions between rarity and eccentricity in the
real world. While they do seem to be similar to each other, our research
suggests that the two metrics are not the same. The most apparent distinction
between rarity and eccentricity would be the distribution. The item rarity
follows the long-tailed distribution, and the eccentricity follows the bell-
curved distribution. From the distribution, we can see that most items are
rare, but only some items are eccentric.

(a) 30Music Dataset

(b) MovieLens Dataset

Figure 3: Probability table of item eccentricity measured in the first period
changing to the one measured in the second period. Higher group number
indicates higher eccentricity.

Figure 2 shows the item eccentricity versus rarity. Each point in the figure
represents an item in the dataset. We only plotted the items that have
feedbacks for more than four time windows for stability. Therefore, the
extremely unpopular items are excluded in the graph. In the figure, we can
observe that as items become unpopular, their eccentricity value diverges. It
shows that there are items that have the same rarities but different
eccentricities. Considering that the item eccentricity is the weighted average
of user eccentricity, this is a natural result of the central limit theorem.

Because unpopular items have a small number of users and their average
eccentricity becomes the items’ eccentricity, it is possible that high
eccentricity is due to randomness, not due to the items’ inherent
characteristics. If some unpopular items are achieving high or low
eccentricity just by chance, they would not necessarily appear to be eccentric
in every period. On the other hand, if one item is consistently consumed by
eccentric users and thus really is eccentric, it should exhibit a high
eccentricity value regardless of the time being measured. Therefore, to
demonstrate that eccentric items identified by our metric are genuinely
eccentric, we examine item eccentricities of two separated periods and compare
the two results.

First, we divided each dataset in two according to its timestamp. For the
MovieLens dataset, we got two datasets from 2011–2012 and 2013–2014. For the
30Music dataset, we got two datasets from February–July and August–January.
Within the separated datasets, we calculated the standardized item
eccentricities. We then divided items into seven septile groups by their
eccentricity. We then kept the items that have been consumed by more than 10
users in both periods, and that have not been released during the test period
to avoid including items that have a too different condition in the two
periods.

Figure 3a shows the relationship between the two eccentricities at different
times on the same item. Each number in the box shows the probability of the
septile group pair. If the item eccentricity is obtained randomly, the
probability of moving from one septile to other septiles in a different period
should be the same across the groups. However, in Figure 3a, items that had
high eccentricity in the first period had a higher possibility of having high
eccentricity in the second period. This shows the consistency of the item
eccentricity, and proves that there are items that are naturally favored by
eccentric users and items that are not.

###  IV-D Comparison Between Tail-Item Groups

TABLE II: Average feature value of groups separated by rarity and eccentricity
in the 30Music dataset. All pairs showed a significant difference (p$<$0.01).

IR |  IE Group |  listened to
once users |  # of plays |  # of
loved users |  # of user
of artists
---|---|---|---|---|---
0.6 |  Ecc. |  69% |  2.11 |  1.03 |  241
2-6[0.5pt/1pt] |  Non-ecc. |  73% |  1.83 |  0.66 |  957
0.7 |  Ecc. |  68% |  1.83 |  0.42 |  145
2-6[0.5pt/1pt] |  Non-ecc. |  74% |  1.64 |  0.29 |  897
0.8 |  Ecc. |  66% |  2.21 |  0.60 |  188
2-6[0.5pt/1pt] |  Non-ecc. |  71% |  1.83 |  0.39 |  844
0.9 |  Ecc. |  66% |  2.11 |  0.58 |  169
2-6[0.5pt/1pt] |  Non-ecc. |  72% |  1.83 |  0.38 |  828
0.99 |  Ecc. |  62% |  2.11 |  0.73 |  177
2-6[0.5pt/1pt] |  Non-ecc. |  70% |  1.83 |  0.52 |  839

TABLE III: Statistics of movies separated by rarity and eccentricity in the
MovieLens dataset. The asterisk (*) indicates a significant difference
(p$<$0.01).

IR |  IE Group |  rating |  multimodality
of ratings |  first two months’ ratings
---|---|---|---|---
0.6 |  Ecc. |  3.15* |  0.13* |  45%*
2-5[0.5pt/1pt] |  Non-ecc. |  2.90* |  0.11* |  20%*
0.7 |  Ecc. |  3.35* |  0.12* |  25%*
2-5[0.5pt/1pt] |  Non-ecc. |  2.87* |  0.10* |  11%*
0.8 |  Ecc. |  2.98 |  0.41* |  84%*
2-5[0.5pt/1pt] |  Non-ecc. |  3.04 |  0.37* |  75%*
0.9 |  Ecc. |  2.87 |  0.40 |  82%*
2-5[0.5pt/1pt] |  Non-ecc. |  2.99 |  0.38 |  80%*
0.99 |  Ecc. |  2.90 |  0.41 |  84%*
2-5[0.5pt/1pt] |  Non-ecc. |  2.99 |  0.38 |  77%*

The previous experiments showed that there are consistently eccentric and
noneccentric items, and they are mixed in the tail part. We now investigate
these different groups of items in detail and show how they are different. We
first extracted various features of items, such as an average number of users,
mean feedback score, and popularity of the artist. We then kept the first
quintile (top 20%) and the last quintile (bottom 20%) groups of items divided
by their eccentricity value, and respectively labeled them as eccentric and
noneccentric groups. We isolated these two IE groups on each 0.6, 0.7, 0.8,
0.9, 0.99 IR group. The $X$ IR group is defined as the set of items whose
rarity value is located between plus–minus one percentile from $X$. For
example, the 0.6 IR group contains the items whose IR value locates between 59
percentile and 61 percentile. After dividing the groups, we finally averaged
several features of items in each group. We conducted statistical tests to see
whether the two eccentricity groups in the same rarity group are significantly
different. Tables II, III present the most significant and consistent
differences between the eccentric and noneccentric groups among the several
features.

Attributes of music items presented in Table II are the listened to once
users, # of plays, # of loved users, # of users of artists. The listened to
once users indicates the percentage of users who only listened to the track
once, and did not listen again. The eccentric track groups had smaller
listened to once users and had higher # of plays than the noneccentric groups.
This signifies that the eccentric tracks are more likely to be replayed by
users in future. Higher average # of loved users of eccentric tracks indicates
that the tracks were not only listened to by users in greater number but also
were favored by people. The last column, # of users of artists, was the most
prominent difference between the two groups. Noneccentric tracks were from
more popular artists compared with eccentric tracks. This implies that the
major difference between eccentric and noneccentric items may lie in the
artists.

For the movies, the differences between eccentric and noneccentric groups were
well captured by three features: rating, multimodality of ratings, and first
two months’ ratings. However, the difference between the two eccentricity
groups was not as evident as the music because of the difference in the value
range of feedbacks. The users’ feedback is used to give weight to their
eccentric behavior and increase the ability to separate items’ eccentricity
value. However, the range of ratings is limited to 0.5–5 for movies while
users can replay tracks as much as they want. Therefore, in the extreme tail,
the IR 0.9 and 0.99 groups of the MovieLens dataset did not show a clear
difference.

Considering the significant results presented in Table III, we could observe
that the eccentric movies have higher ratings and their ratings tend to follow
the multimodal Gaussian distribution. Multimodality of ratings was measured by
Hartigan’s Dip test statistic [23]. This implies that the ratings are more
polarized for eccentric and rare items. The last column, first two months’
ratings, was the only feature where all IR groups had a significant difference
between the two eccentricity groups. The result shows that the ratings of
eccentric items are more concentrated on the time they were released, while
the noneccentric items were also consumed after the release.

##  V Conclusion

Our work has proposed a novel metric to measure the eccentricity of items. The
main intuition behind our metric was that the eccentricity of items is related
to who are the consumers, not to the number of people who have used the item.
Using the proposed metric, we have analyzed two real-world datasets. Through
the analysis, we showed that the eccentric and noneccentric items classified
by our metric maintained their eccentricity value regardless of the time being
measured, and showed a significant difference on various features.

## Acknowledgment

This work was supported by ICT R&D program of MSIP/IITP (R7124-16-0004,
Development of Intelligent Interaction Technology Based on Context Awareness
and Human Intention Understanding).

## References

  * [1] J. A. Konstan and J. Riedl, “Recommender systems: from algorithms to user experience,” _User Modeling and User-Adapted Interaction_ , vol. 22, no. 1, pp. 101–123, 2012. [Online]. Available: http://dx.doi.org/10.1007/s11257-011-9112-x
  * [2] S. M. McNee, J. Riedl, and J. A. Konstan, “Being accurate is not enough: how accuracy metrics have hurt recommender systems,” in _CHI’06 extended abstracts on Human factors in computing systems_. ACM, 2006, pp. 1097–1101.
  * [3] K. Onuma, H. Tong, and C. Faloutsos, “Tangent: a novel,’surprise me’, recommendation algorithm,” in _Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining_. ACM, 2009, pp. 657–666.
  * [4] K. Kapoor, V. Kumar, L. Terveen, J. A. Konstan, and P. Schrater, “I like to explore sometimes: Adapting to dynamic user novelty preferences,” in _Proceedings of the 9th ACM Conference on Recommender Systems_. ACM, 2015, pp. 19–26.
  * [5] Y.-J. Park and A. Tuzhilin, “The long tail of recommender systems and how to leverage it,” in _Proc. of the 2008 ACM Conference on Recommender Systems_ , ser. RecSys ’08. New York, NY, USA: ACM, 2008, pp. 11–18. [Online]. Available: http://doi.acm.org/10.1145/1454008.1454012
  * [6] M. Ge, C. Delgado-Battenfeld, and D. Jannach, “Beyond accuracy: evaluating recommender systems by coverage and serendipity,” in _Proceedings of the fourth ACM conference on Recommender systems_. ACM, 2010, pp. 257–260.
  * [7] G. Adomavicius and Y. Kwon, “Improving aggregate recommendation diversity using ranking-based techniques,” _IEEE Transactions on Knowledge and Data Engineering_ , vol. 24, no. 5, pp. 896–911, 2012.
  * [8] C.-N. Ziegler, S. M. McNee, J. A. Konstan, and G. Lausen, “Improving recommendation lists through topic diversification,” in _Proc. of the 14th international conference on World Wide Web_. ACM, 2005, pp. 22–32.
  * [9] Ò. Celma and P. Cano, “From hits to niches?: or how popular artists can bias music recommendation and discovery,” in _Proc. of the 2nd KDD Workshop on Large-Scale Recommender Systems and the Netflix Prize Competition_. ACM, 2008, p. 5.
  * [10] S. Goel, A. Broder, E. Gabrilovich, and B. Pang, “Anatomy of the long tail: Ordinary people with extraordinary tastes,” in _Proc. of the 3rd ACM International Conference on Web Search and Data Mining_ , ser. WSDM ’10. New York, NY, USA: ACM, 2010, pp. 201–210. [Online]. Available: http://doi.acm.org/10.1145/1718487.1718513
  * [11] N. R. Sastry, “How to tell head from tail in user-generated content corpora.” in _ICWSM_ , 2012.
  * [12] H. Steck, “Item popularity and recommendation accuracy,” in _Proceedings of the fifth ACM conference on Recommender systems_. ACM, 2011, pp. 125–132.
  * [13] E. Tacchini, “Serendipitous mentorship in music recommender systems,” Ph.D. dissertation, Ph. D. Thesis, 2012.
  * [14] Y. C. Zhang, D. Ó. Séaghdha, D. Quercia, and T. Jambor, “Auralist: introducing serendipity into music recommendation,” in _Proc. of the fifth ACM international conference on Web search and data mining_. ACM, 2012, pp. 13–22.
  * [15] J. H. Lee and R. Price, “Understanding users of commercial music services through personas: Design implications.” in _ISMIR_ , 2015, pp. 476–482.
  * [16] J. L. Herlocker, J. A. Konstan, L. G. Terveen, and J. T. Riedl, “Evaluating collaborative filtering recommender systems,” _ACM Transactions on Information Systems (TOIS)_ , vol. 22, no. 1, pp. 5–53, 2004.
  * [17] Q. Lu, T. Chen, W. Zhang, D. Yang, and Y. Yu, “Serendipitous personalized ranking for top-n recommendation,” in _Proceedings of the The 2012 IEEE/WIC/ACM International Joint Conferences on Web Intelligence and Intelligent Agent Technology-Volume 01_. IEEE Computer Society, 2012, pp. 258–265.
  * [18] N. Hurley and M. Zhang, “Novelty and diversity in top-n recommendation – analysis and evaluation,” _ACM Trans. Internet Technol._ , vol. 10, no. 4, pp. 14:1–14:30, Mar. 2011. [Online]. Available: http://doi.acm.org/10.1145/1944339.1944341
  * [19] S. Vargas and P. Castells, “Rank and relevance in novelty and diversity metrics for recommender systems,” in _Proc. of the Fifth ACM Conference on Recommender Systems_ , ser. RecSys ’11. New York, NY, USA: ACM, 2011, pp. 109–116. [Online]. Available: http://doi.acm.org/10.1145/2043932.2043955
  * [20] M. Nakatsuji, Y. Fujiwara, A. Tanaka, T. Uchiyama, K. Fujimura, and T. Ishida, “Classical music for rock fans?: novel recommendations for expanding user interests,” in _Proceedings of the 19th ACM international conference on Information and knowledge management_. ACM, 2010, pp. 949–958.
  * [21] A. Marques, N. Andrade, and L. Balby, “Exploring the relation between novelty aspects and preferences in music listening,” in _Proc. of the 14th International Society for Music Information Retrieval Conference_ , 2013, pp. 407–412. [Online]. Available: http://ismir2013.ismir.net/wp-content/uploads/2013/09/257_Paper.pdf
  * [22] Q. M. C. A. P. R. . C. P. Turrin, R., “30music listening and playlists dataset,” in _Poster Proc. of the 9th ACM Conference on Recommender Systems_ , 2015.
  * [23] J. A. Hartigan and P. Hartigan, “The dip test of unimodality,” _The Annals of Statistics_ , pp. 70–84, 1985.