# To Recommend or Not?
A Model-Based Comparison of Item-Matching Processes

Serina Chang
Stanford University
<EMAIL_ADDRESS>&Johan Ugander
Stanford University
<EMAIL_ADDRESS>
###### Abstract

Recommender systems are central to modern online platforms, but a popular
concern is that they may be pulling society in dangerous directions (e.g.,
towards filter bubbles). However, a challenge with measuring the effects of
recommender systems is how to compare user outcomes under these systems to
outcomes under a credible counterfactual world without such systems. We take a
model-based approach to this challenge, introducing a dichotomy of process
models that we can compare: (1) a “recommender” model describing a generic
item-matching process under a personalized recommender system and (2) an
“organic” model describing a baseline counterfactual where users search for
items without the mediation of any system. Our key finding is that the
recommender and organic models result in dramatically different outcomes at
both the individual and societal level, as supported by theorems and
simulation experiments with real data. The two process models also induce
different trade-offs during inference, where standard performance-improving
techniques such as regularization/shrinkage have divergent effects. Shrinkage
improves the mean squared error of matches in both settings, as expected, but
at the cost of less diverse (less radical) items chosen in the recommender
model but more diverse (more radical) items chosen in the organic model. These
findings provide a formal language for how recommender systems may be
fundamentally altering how we search for and interact with content, in a world
increasingly mediated by such systems.

Personalized recommender systems guide the modern online experience. These
systems recommend movies and music on content platforms (Nguyen et al., 2014;
Anderson et al., 2020; Holtz et al., 2020), suggest friendships and groups on
social networking sites (Gupta et al., 2013; Kloumann and Kleinberg, 2014),
select advertisements for target audiences (Lambrecht and Tucker, 2019), and
filter news to consumers (Das et al., 2007). As recommender systems
increasingly shape the content we consume, we have become more critical of
their potential for unintended societal consequences. For example, one concern
that has received considerable attention by academics (e.g., (Nguyen et al.,
2014; Flaxman, Goel, and Rao, 2016; Rastegarpanah, Gummadi, and Crovella,
2019)) and the public (e.g., (Singer, 2011)) alike is that these systems may
be pulling society towards “filter bubble” dynamics, where individuals become
isolated from viewpoints besides their own (Pariser, 2011). However, a key
challenge with measuring the true effects of recommender systems is that we
only observe user outcomes under recommender systems, but we are missing a
credible assessment of user outcomes in the absence of these systems. To
understand definitively whether recommender systems have a significant impact
on filter bubbles, or any other societal phenomena, we need a systematic way
to compare these two scenarios and their outcomes against each other.

To address this need, we introduce a dichotomy of process models that
describes a world with and a world without recommender systems. Consider the
example of a user who is new to a movie distribution platform, and seeking to
watch one movie for the night. Under our organic model, the user searches for
movies “organically” without the mediation of any recommender system. She
wants to watch the movie that best fits her interests, but she does not know
the contents of any of the candidate movies. During this search process she
has access only to noisy signals of each movie (e.g., by watching trailers or
reading reviews), and based on these noisy signals, she must estimate each
movie’s content. To complete the matching process, she chooses to watch the
movie that she estimates best matches her own interests.

Compare this first model against the case where the platform has a
personalized recommender system. Under our recommender model, the situation is
now flipped: whereas the user knew her own interests well but previously had
to estimate the contents of the movies, now the system knows the movies on the
platform well, but must estimate the user’s interests. In a similar manner,
the system gathers a noisy sample from the user (e.g., by asking her to name
one movie she recently enjoyed). Based on this noisy sample, the system then
estimates her interests and recommends her the movie it believes is the best
match. In this dichotomy, the organic model functions as a baseline against
which the matching behavior of a recommender system can be compared.

What connects these two settings is the act of trying to match users to items
with only noisy information about one side; what differs, however, is which
information is known and who is doing the matching (the user themselves vs.
the recommender system). We find that this difference in perspective between
the two models results in significant differences in outcomes. We characterize
these differences through the lenses of both (1) individual metrics (what is
the expected loss for a given user? Does this differ across users?) and (2)
population metrics (what is the average user loss? Which items tend to be
selected overall?). Furthermore, we document the notably varying effects of
regularization in each of the models, where the users or system apply
shrinkage to their estimates. We will see that shrinkage can reduce average
user loss in both models, but it also introduces new effects that cause the
models to diverge further.

Thus, even in these distilled settings, the intersection of our models,
metrics, and algorithm design decisions (such as shrinkage) creates a rich
environment in which we can investigate the effects of recommender systems. We
summarize our contributions as:

  1. 1.

A framework of two contrasting models that capture organic search and
recommended item-matching as symmetric, comparable processes;

  2. 2.

Theorems proving key differences between the models from both the individual
and population perspectives;

  3. 3.

Simulations demonstrating that our findings translate from the theorem
settings to realistic data (MovieLens).

Collectively, our work lays out a formal model-based language for how the
effect of recommender systems can be meaningfully analyzed relative a
counterfactual world without such systems.

##  1 Related Work

The widespread adoption of personalized recommender systems has led to diverse
investigations of the potential societal consequences of these systems. One
body of literature tackles bias in recommender systems (Chen et al., 2020;
Bozdag, 2013), examining how they may systematically underrepresent minority
views (Stoica and Chaintreau, 2019), serve predictions of uneven quality
across user groups (Yao and Huang, 2017), or fail to recommend valuable items
to certain users, e.g., showing job opportunities in STEM fields to fewer
women than men (Lambrecht and Tucker, 2019). Empirical observations, however,
have often been mixed in nature, e.g., documenting how systems sometimes favor
long-tail items (Fleder and Hosanagar, 2009; Brynjolfsson, Hu, and Simester,
2011) or (over-)favor popular items (Abdollahpouri, Burke, and Mobasher,
2017).

Another topic of societal concern is the possibility that personalized
recommendations are pushing individuals into “filter bubbles” (Pariser, 2011).
Social media users are known to selectively share content and connect to
friends who agree with their existing opinions (An et al., 2014; Garimella et
al., 2018), and there is some evidence that recommender systems exacerbate
this dynamic of ideological segregation (Bakshy, Messing, and Adamic, 2015;
Flaxman, Goel, and Rao, 2016). Recommender systems seem to have a narrowing
effect in other contexts as well: e.g., the sets of movies (Nguyen et al.,
2014) and songs (Anderson et al., 2020) recommended to users tend to be less
diverse than the content that users find on their own. However, some studies
have pointed out that the role of recommender systems is modest compared to
the impact of user choice (e.g., whether to click on a recommended story) on
narrowing consumption diversity (Bakshy, Messing, and Adamic, 2015). Other
works have made the case that recommender systems actually increase diversity
in exposure (Flaxman, Goel, and Rao, 2016) and widen users’ interests
(Hosanagar et al., 2013).

Part of the reason why it is so challenging to measure the true effects of
recommender systems—and perhaps why empirical studies have not been able to
reach a consensus—is that we typically only observe user outcomes under the
recommender system, but we cannot assess user outcomes in the absence of these
systems. Claims, e.g., that users are entering filter bubbles, must be
understood relative to some counterfactual baseline. Thus, it is often helpful
to design models that enable us to “observe“ and compare these hypothetical
realities. For example, Dandekar, Goel, and Lee (2013) show with a model of
opinion formation that if users are biased in how they process evidence
(drawing undue support for their initial opinion), recommender systems can
cause a polarizing effect on users’ opinions. Stoica and Chaintreau (2019)
show under their model that recommender systems accelerate hegemonic dynamics
on social media (i.e., when a single viewpoint receives undue attention).
Perra and Rocha (2019) model the impact of algorithmic personalization on user
opinions, although the effect also depends on the structure of the users’
social network, As a final example, Chaney, Stewart, and Engelhardt (2018) use
models to analyze the consequences of feedback loops, when the recommender
system is trained on data that was influenced by the system.

Using models to study the impact of recommender systems can be powerful when
it is difficult to observe “what-ifs” in real life, e.g., what if we removed
the feedback loop, what if recommendations were not personalized. In the
present work, we take a model-based approach to ask arguably the most
fundamental “what-if” question about recommender systems: what if there were
no recommender system at all? We analyze our models using first-principles
measures such as the expected item match for each user, their expected loss,
and the variance over the population of matched items (Section 2.2). These
measures form the building blocks for many of the downstream phenomena of
interest related to recommender systems, including polarization, filter
bubbles, user satisfaction, user retention, and bias and fairness. Implicitly
embedded in these more complex processes are the metrics we study, and thus,
our work forms a foundation upon which future researchers can build.

##  2 Models and Metrics

Our models describe two contrasting processes through which users could be
matched with items of choice, e.g., movies, news articles, or consumer
products. In Section 2.1, we introduce the notation and formal dynamics of our
models. In Section 2.2, we define the individual and population-level metrics
we will use to compare the models’ outcomes.

###  2.1 Model definition

In both models, we will have $m$ users and $n$ items. Each user and each item
has a latent position; for example, this could represent movie attributes in
the movie context or ideology in the context of news articles. We denote user
positions as $x_{i}\in\mathbb{R}^{d}$ and item positions as
$y_{j}\in\mathbb{R}^{d}$, where $d$ represents the dimensionality of the
latent space. Overall, the set $\mathcal{X}=\\{x_{i}\\}_{i=1}^{m}$ represents
all user positions and the set $\mathcal{Y}=\\{y_{j}\\}_{j=1}^{n}$ represents
all item positions. Our analysis will make very weak distance-based
assumptions about the interest/utilities of users for items: we simply assume
that user $i$’s enjoyment of item $j$ is monotonically decreasing to the
distance between $x_{i}$ and $y_{j}$. Thus, under both models, the decision-
making agent (either the user or the recommender system) wants to match the
user to the item whose position is closest to hers.

##### Organic model.

In this model, the user will “organically” search through the collection of
items and choose one for herself (Figure 1a). When surveying each item $j$,
she only has access to a noisy sample of its true position, $y_{j}$. She draws
her sample $z_{j}^{(i)}$ from $N(y_{j},\Sigma_{i})$, where $\Sigma_{i}$
represents the covariance in her noise. Then, the user makes an estimate
$\hat{y}_{j}^{(i)}$ of item $j$’s true position. In the simplest case, the
user takes the maximum likelihood estimate (MLE) based on her single sample of
$y_{j}$:

$\displaystyle\hat{y}_{j}^{(i,\text{MLE})}=z_{j}^{(i)}.$ (1)

We compare this MLE to an alternate case where the user applies some form of
shrinkage (James and Stein, 1961) to her estimates. For example, if the user
assumes a Gaussian prior on the item positions, she could take a maximum a
posteriori (MAP) estimate of each, shrinking the estimate towards the item
mean:

$\displaystyle\hat{y}_{j}^{(i,\text{MAP})}=(\Sigma_{\text{item}}^{-1}+\Sigma_{i}^{-1})^{-1}(\Sigma_{\text{item}}^{-1}\mu_{\text{item}}+\Sigma_{i}^{-1}z_{j}^{(i)}),$
(2)

where $\mu_{\text{item}}$ and $\Sigma_{\text{item}}$ are the mean and
covariance of the item distribution, respectively. This style of shrinkage can
also be thought of as regularization towards the item mean. In an empirical
setting without a clear prior, Empirical Bayes shrinkage (Efron and Morris,
1976) would be preferred.

After surveying all $n$ items, the user will have constructed estimates for
every item’s position. Since we assume utility is monotonically decreasing in
distance, the user will then choose the item whose estimate is closest to her
own position, $x_{i}$. Let $k_{i}^{\text{org}}$ represent the item chosen by
user $i$ under the organic setting. Then,

$\displaystyle
k_{i}^{\text{org}}=\operatorname*{arg\,min}_{j\in[n]}||x_{i}-\hat{y}_{j}^{(i)}||.$
(3)

We let $y_{k_{i}}^{\text{org}}$ denote the position of the chosen item
$k_{i}^{\text{org}}$, and will later study properties of
$y_{k_{i}}^{\text{org}}$ as a random variable that inherits its randomness
from the user’s estimates, $\hat{y}_{j}^{(i)}$.

Figure 1: Model schematics. (a) In the organic model, the user samples and
estimates item positions, and chooses the estimate closest to her own
position. When shrinkage is applied, the user’s estimates of the items shifts
inward, and the user’s choice switches from the inner to outer item. (b) In
the recommender model, the system samples and estimates the user’s position,
and chooses the item closest to its estimate. When shrinkage is applied, the
system’s estimate of the user shifts inward, and the system’s choice switches
from the outer to inner item.

#### Recommender model.

In this model, a recommender system takes on the burden of search instead of
the user (Figure 1b). When user $i$ comes onto the platform, the system
gathers a noisy sample of $i$’s true position $x_{i}$. This sample
$z_{i}^{(r)}$ is drawn from $N(x_{i},\Sigma_{r})$, where $\Sigma_{r}$
represents the covariance in the recommender’s noise. Then, just as the user
estimated item positions based on her noisy samples, the system makes an
estimate $\hat{x}_{i}^{(r)}$ of user $i$’s true position based on its sample
of the user. Again, we first consider the case where the recommender system
takes the MLE:

$\displaystyle\hat{x}_{i}^{(r,\text{MLE})}=z_{i}^{(r)}.$ (4)

We will again compare the MLE to the case where the recommender system applies
shrinkage to its estimates of user positions. With a Gaussian prior, the MAP
estimate of user $i$’s position is

$\displaystyle\hat{x}_{i}^{(r,\text{MAP})}=(\Sigma_{\text{user}}^{-1}+\Sigma_{r}^{-1})^{-1}(\Sigma_{\text{user}}^{-1}\mu_{\text{user}}+\Sigma_{r}^{-1}z_{i}^{(r)}),$
(5)

where $\mu_{\text{user}}$ and $\Sigma_{\text{user}}$ are the mean and
covariance of the user distribution, respectively. In the absence of a prior,
Empirical Bayes shrinkage would again be preferred.

As a reversal of the organic model, we assume that the recommender system has
perfect knowledge of $\mathcal{Y}$, the set of true item positions, but can
only estimate the user’s true position. Let $k_{i}^{\text{rec}}$ represent the
chosen item for user $i$ under the recommender model. Similar to the organic
model, the system will choose the item whose position is closest to
$\hat{x}_{i}^{(r)}$:

$\displaystyle k_{i}^{\text{rec}}$
$\displaystyle=\operatorname*{arg\,min}_{j\in[n]}||\hat{x}_{i}^{(r)}-y_{j}||.$
(6)

Here we again let $y_{k_{i}}^{\text{rec}}$ denote the position of the matched
item $k_{i}^{\text{rec}}$, where $y_{k_{i}}^{\text{rec}}$ is a random variable
that inherits its randomness from the system’s estimates, $\hat{x}_{i}^{(r)}$.

We deliberately set up the recommendation process to be as similar as possible
to the organic process—the structure of the noise and logic in the choice
function are identical—so that we can compare the models on the basis of
who/what controls the matching and what information they have access to. We do
not, for example, model the process of users reacting to recommendations.
Instead, we assume that $k_{i}^{\text{rec}}$ will be the user’s match,
comparing it directly to the match $k_{i}^{\text{org}}$ under the organic
model. Furthermore, we note that both models are made up of distinct modules
(sampling, estimation, choice), each of which could be swapped out for more
complex processes. For example, in the organic model, one might be interested
in other decision-making rules for the user, such as those that make
explore/exploit trade-offs or capture risk aversion (in the case of non-
uniform sample noise), or in the recommender model, the user estimation
process could be derived from a specific algorithm. Thus, our models describe
flexible item-matching processes that can be extended to encompass many real-
world systems.

###  2.2 Metrics

We are interested in comparing the outcomes of these models through two
lenses, at the individual and at the population levels. At the individual
level, we will treat the squared distance between the user and their matched
item as a user-level loss, since we model utility as monotonically decreasing
in distance. Let $\mathcal{Y}_{k}=\\{y_{k_{1}},y_{k_{2}},\cdots,y_{k_{m}}\\}$
represent the multiset of matched item positions over all users; note that
$|\mathcal{Y}_{k}|=|\mathcal{X}|=m$, and that each item position $y_{j}$ might
appear 0 times, once, or multiple times in $\mathcal{Y}_{k}$. Then, the loss
for user $i$ is defined as

$\displaystyle l_{i}(\mathcal{X},\mathcal{Y}_{k})=||y_{k_{i}}-x_{i}||^{2}.$
(7)

As we analyze $l_{i}(\mathcal{X},\mathcal{Y}_{k})$ in Section 3, we will also
derive $\mathbb{E}[y_{k_{i}}]$ and $\text{Var}[y_{k_{i}}]$ along the way; that
is, the expected position and variance of the matched item for user $i$. We
choose these metrics to study because they form the building blocks of many
downstream phenomena of interest. For example, if the expected match for a
user is not the user’s own position, the matching process might eventually
shift user preferences or opinions, and if different users systematically
experience different losses (as we will see happens under certain settings of
our models), this implies inequities in the model as it provides matches of
differential quality to users on the basis of their preferences.

The population-level analysis, meanwhile, is concerned with average user loss
and the overall collection of matched items. Observe that the average loss
over users simply becomes the mean squared error (MSE) of the matches:

$\displaystyle\text{MSE}(\mathcal{X},\mathcal{Y}_{k})=\frac{1}{m}\sum_{i=1}^{m}l_{i}(\mathcal{X},\mathcal{Y}_{k})=\frac{1}{m}\sum_{i=1}^{m}||y_{k_{i}}-x_{i}||^{2}.$
(8)

These matches are based on estimates, where shrinkage is known to reduce the
MSE of an estimator. As such, there are opportunities in both models to reduce
mean user loss by having the user / system apply shrinkage during estimation.
However, while shrinkage might reduce MSE, that is not its only effect. In
Figure 1, we illustrate a basic intuition for a key result: due to the
reversal in who is doing the estimating and what is being estimated, shrinkage
acts as a “diversifying” force in the organic model, but it serves as a
“homogenizing” force in the recommender model. We see that shrinkage in the
organic model shifts the user’s estimates of the items inward (i.e., towards
0), so items that are further out now have a better chance of being chosen.
Meanwhile, shrinkage in the recommender model shifts the system’s estimate of
the user inward, so now items closer to the center are likelier to be chosen.

To quantify this intuition, we also study $\text{Var}[\mathcal{Y}_{k}]$ as
another population-level metric, the variance of the matched item positions
$\mathcal{Y}_{k}$. This variance has numerous interpretations in the real
world: for example, seeing what kinds of items are being selected may guide
content creators as they decide what to generate next. Furthermore, there is
evidence that users adjust their preferences to better align with content that
they are matched with, whether because they were persuaded by the content
(Diehl, Weeks, and Zúñiga, 2015), or by the very fact that it was recommended
to them (Summers, Smith, and Reczek, 2016). Such dynamics suggest that users
may become more heterogeneous if they are matched with a more diverse set of
items, and more homogeneous if they are all matched to similar content. With
repeated rounds of matching and opinion formation, a great level of
heterogeneity in users could lead to polarization or radicalization;
conversely, increasing homogeneity could result in filter bubbles and a lack
of diverse perspectives for those on the platform.

##  3 Model Analysis

To develop a theoretical understanding of how these two models behave, we
begin with simplified instances, where we assume that the user and item
positions come from Gaussian distributions in a one-dimensional space. For
now, we will assume that user and item positions are drawn independently from
$N(0,\sigma^{2}_{\text{user}})$ and $N(0,\sigma^{2}_{\text{item}})$,
respectively (since $d=1$, we replace covariance matrices $\Sigma$ with scalar
variances, $\sigma^{2}$). We also focus here on the asymptotic setting where
the number of items $n$ approaches $\infty$ (in Section 4.1 we examine
simulations with different finite values of $n$). Interestingly, we will see
that even this simplified setting is sufficient to induce the phenomena we
seek to understand, and that the results remain qualitatively similar when we
explore non-Gaussian multidimensional user and item positions learned from
data (Section 4.2).

In this section, we organize our results into a series of theorems. In
Theorems 1–2, we will analyze how the models differ in terms of individual
metrics, by deriving 3 quantities for each model: (1) $\mathbb{E}[y_{k_{i}}]$,
the expected item match; (2) $\text{Var}[y_{k_{i}}]$, the variance in the
match; (3) $\mathbb{E}[l_{i}]$, the expected loss; all for any user $i$ at
position $x_{i}$. In Theorems 3–4, we will then study how the models differ at
the population level, showing that even though shrinkage can reduce MSE for
both processes, a key difference is that shrinkage increases the variance of
matched items under the organic model but decreases this variance under the
recommender model.

##### Theorem 1.1.

In the organic MLE model as $n\rightarrow\infty$,

  1. 1.

$\mathbb{E}[y_{k_{i}}^{(\text{org})}]\rightarrow\frac{\sigma^{2}_{\text{item}}}{\sigma^{2}_{\text{item}}+\sigma^{2}_{i}}x_{i}$

  2. 2.

$\text{Var}[y_{k_{i}}^{(\text{org})}]\rightarrow\left(\frac{1}{\sigma^{2}_{\text{item}}}+\frac{1}{\sigma^{2}_{i}}\right)^{-1}$

  3. 3.

$\mathbb{E}[l_{i}^{\textrm{(org)}}]\rightarrow\left(\frac{\sigma^{2}_{\text{item}}}{\sigma^{2}_{\text{item}}+\sigma^{2}_{i}}-1\right)^{2}x_{i}^{2}+\left(\frac{1}{\sigma^{2}_{\text{item}}}+\frac{1}{\sigma^{2}_{i}}\right)^{-1}$.

###### Proof.

In the organic MLE model, the user makes an estimate of each item position,
$\hat{y}_{j}^{(i,\text{MLE})}=z_{j}^{(i)}$, where $z_{j}^{(i)}$ is a sample
drawn from $N(y_{j},\sigma^{2}_{i})$. Since there are an infinite number of
items and the support of the item and noise distributions is $\mathbb{R}$,
then there must be at least one item estimate located at any given point in
$\mathbb{R}$. Thus, when the user chooses the estimate that is closest to her
own position, $x_{i}$, we can expect that the chosen estimate lies at $x_{i}$.
Given the knowledge that the user’s chosen item produced a sample at $x_{i}$,
and that the chosen item’s position $y_{k_{i}}^{(\text{org})}$ was drawn from
$N(0,\sigma^{2}_{\text{item}})$, we can derive a closed form solution for the
posterior mean and variance of $y_{k_{i}}^{(\text{org})}$ (since both
distributions are Gaussian, forming a conjugate pair), forming the expressions
above for $\mathbb{E}[y_{k_{i}}^{(\text{org})}]$ and
$\text{Var}[y_{k_{i}}^{(\text{org})}]$.

Finally, recall that in estimation, mean squared error can be decomposed into
the sum of the squared bias and the variance of the estimator. If we view the
position of the matched item, $y_{k_{i}}$, as an estimate for user $i$’s true
position, $x_{i}$, then the expected squared distance between the two is
simply
$(\mathbb{E}[y_{k_{i}}^{(\text{org})}]-x_{i})^{2}+\text{Var}[y_{k_{i}}^{(\text{org})}]$,
thus yielding $\mathbb{E}[l_{i}^{\textrm{(org)}}]$. ∎

##### Theorem 1.2.

In the organic MAP model as $n\rightarrow\infty$,

  1. 1.

$\mathbb{E}[y_{k_{i}}^{(\text{org},\text{MAP})}]\rightarrow x_{i}$

  2. 2.

$\text{Var}[y_{k_{i}}^{(\text{org},\text{MAP})}]\rightarrow\left(\frac{1}{\sigma^{2}_{\text{item}}}+\frac{1}{\sigma^{2}_{i}}\right)^{-1}$

  3. 3.

$\mathbb{E}[l_{i}^{\textrm{(org,
MAP)}}]\rightarrow\left(\frac{1}{\sigma^{2}_{\text{item}}}+\frac{1}{\sigma^{2}_{i}}\right)^{-1}$.

###### Proof.

When the user uses MAP, her estimate of item $j$’s position becomes
$\hat{y}_{j}^{(i,\text{MAP})}=(\sigma^{2}_{\text{item}}/(\sigma^{2}_{\text{item}}+\sigma^{2}_{i}))z_{j}^{(i)}$.
Since there are an infinite number of items, she will still choose an estimate
that is equal to $x_{i}$; however, this means that the chosen item must have
produced a sample at
$((\sigma^{2}_{\text{item}}+\sigma^{2}_{i})/\sigma^{2}_{\text{item}})x_{i}$.
Again, we can derive a closed form solution for the posterior distribution of
the chosen item’s position $y_{k_{i}}^{(\text{org},\text{MAP})}$, knowing that
it was also drawn from $N(0,\sigma^{2}_{\text{item}})$. We find that the terms
cancel out, giving us $\mathbb{E}[y_{k_{i}}^{\text{org}}]=x_{i}$, and the
variance does not change compared to the MLE version of the model. Since
$\mathbb{E}[y_{k_{i}}^{\text{org}}]$ is now unbiased, the user’s expected loss
becomes only the variance. ∎

##### Theorem 2.1.

In the recommender MLE model as $n\rightarrow\infty$,

  1. 1.

$\mathbb{E}[y_{k_{i}}^{(\text{rec})}]\rightarrow x_{i}$

  2. 2.

$\text{Var}[y_{k_{i}}^{(\text{rec})}]\rightarrow\sigma^{2}_{r}$

  3. 3.

$\mathbb{E}[l_{i}^{\textrm{(rec)}}]\rightarrow\sigma^{2}_{r}$.

###### Proof.

In the recommender MLE model, the recommender system makes an estimate of the
user’s true position, $\hat{x}_{i}^{(r,\text{MLE})}=z_{i}^{(r)}$, where
$z_{i}^{(r)}$ is a sample drawn from $N(x_{i},\sigma^{2}_{r})$. Then, the
system finds the item whose position is closest to
$\hat{x}_{i}^{(r,\text{MLE})}$. Echoing the proof in Theorem 1, if there are
an infinite number of items, then there must be at least one item located at
any given point in $\mathbb{R}$. Thus, the chosen item position,
$y_{k_{i}}^{(\text{rec})}$, must be equal to $\hat{x}_{i}^{(r,\text{MLE})}$,
and so the distribution of $y_{k_{i}}^{(\text{rec})}$ reduces to the
distribution of $\hat{x}_{i}^{(r,\text{MLE})}$, which has an expected value
$x_{i}$ and variance $\sigma^{2}_{r}$. Since $y_{k_{i}}^{(\text{rec})}$ forms
an unbiased estimate for $x_{i}$, the expected loss for the user is again just
the variance. ∎

##### Theorem 2.2.

In the recommender MAP model as $n\rightarrow\infty$,

  1. 1.

$\mathbb{E}[y_{k_{i}}^{(\text{rec},\text{MAP})}]\rightarrow\frac{\sigma^{2}_{\text{user}}}{\sigma^{2}_{\text{user}}+\sigma^{2}_{r}}x_{i}$

  2. 2.

$\text{Var}[y_{k_{i}}^{(\text{rec},\text{MAP})}]\rightarrow\left(\frac{\sigma^{2}_{\text{user}}}{\sigma^{2}_{\text{user}}+\sigma^{2}_{r}}\right)^{2}\sigma^{2}_{r}$

  3. 3.

$\mathbb{E}[l_{i}^{\textrm{(rec,
MAP)}}]\rightarrow\left(\frac{\sigma^{2}_{\text{user}}}{\sigma^{2}_{\text{user}}+\sigma^{2}_{r}}-1\right)^{2}x_{i}^{2}+\left(\frac{\sigma^{2}_{\text{user}}}{\sigma^{2}_{\text{user}}+\sigma^{2}_{r}}\right)^{2}\sigma^{2}_{r}$.

###### Proof.

In the MAP setting, the recommender model’s estimate of the user’s position
becomes
$\hat{x}_{i}^{(r,\text{MAP})}=(\sigma^{2}_{\text{user}}/(\sigma^{2}_{\text{user}}+\sigma^{2}_{r}))z_{i}^{(r)}$.
Since there are an infinite number of items, it is still true that the chosen
item’s position, $y_{k_{i}}^{(\text{rec},\text{MAP})}$, will match
$\hat{x}_{i}^{(r,\text{MAP})}$ perfectly, and so the distribution of
$y_{k_{i}}^{(\text{rec},\text{MAP})}$ reduces to the distribution of
$\hat{x}_{i}^{(r,\text{MAP})}$. Since $\hat{x}_{i}^{(r,\text{MAP})}$ is simply
a sample drawn from $N(x_{i},\sigma^{2}_{r})$ scaled by
$\sigma^{2}_{\text{user}}/(\sigma^{2}_{\text{user}}+\sigma^{2}_{r})$, then
$\hat{x}_{i}^{(r,\text{MAP})}$ must be normally distributed with mean
$(\sigma^{2}_{\text{user}}/(\sigma^{2}_{\text{user}}+\sigma^{2}_{r}))x_{i}$
and variance
$(\sigma^{2}_{\text{user}}/(\sigma^{2}_{\text{user}}+\sigma^{2}_{r}))^{2}\sigma^{2}_{r}$.
This provides the expressions above for
$\mathbb{E}[y_{k_{i}}^{(\text{rec},\text{MAP})}]$ and
$\text{Var}[y_{k_{i}}^{(\text{rec},\text{MAP})}]$, and the expected squared
loss is the sum of the squared bias,
$(\mathbb{E}[y_{k_{i}}^{(\text{rec},\text{MAP})}]-x_{i})^{2}$, and the
variance $\text{Var}[y_{k_{i}}^{(\text{rec},\text{MAP})}]$. ∎

Figure 2: Expected user loss as a function of user position (Theorems 1–2);
$\sigma^{2}_{\text{item}}=1$ and $\sigma^{2}_{i}=\sigma^{2}_{r}=0.5$.

Theorems 1 and 2 imply that if we treat the matched item position $y_{k_{i}}$
as an estimate of the user position $x_{i}$, then the organic MLE model forms
biased matches for users, while the recommender MLE model does not. In
particular, the organic MLE model is biased in that, relative to $x_{i}$, the
expected match for user $i$ is contracted towards the center of the item
distribution (Theorem 1.1). Furthermore, since the bias term grows with
$x_{i}^{2}$, this means that users closer to the center will be less impacted
by the bias than users who are farther out. This is a fundamental difference
between the MLE versions of the processes: the organic model favors users who
are closer to the center of the item distribution, while the recommender model
is agnostic to user position, as we show in Figure 2. Interestingly, the trend
reverses when we switch from MLE to MAP: the organic model becomes unbiased,
and the recommender model becomes favorable to central users. Some of this
effect is particular to our form of shrinkage here (namely, that the user
perfectly erases the bias in the organic model by taking the MAP based on the
true prior distribution of items), but in general, shrinkage will always
increase bias for the recommender model, and, if the item distribution is
heaviest around its mean, shrinkage will also reduce bias for the organic
model.

From Theorems 1 and 2, we can also establish that switching from MLE to MAP
will always reduce mean user loss (i.e., MSE) in the organic model, and that
it can sometimes reduce MSE in the recommender model. First, Theorem 1 shows
that in the organic model, expected loss strictly decreases for every user
(with non-zero $x_{i}$) from MLE to MAP; thus, the mean loss must fall as
well. From Theorem 2.2, we can imagine the conditions under which MAP will
achieve a lower MSE in the recommender model: if the user variance
$\sigma^{2}_{\text{user}}$ is small, then the bias terms will be smaller
because all of the $x_{i}^{2}$’s will be closer to $0$, and the relative
reduction that MAP achieves on the variance, scaling by a factor of
$(\frac{\sigma^{2}_{\text{user}}}{\sigma^{2}_{\text{user}}+\sigma^{2}_{r}})^{2}$,
will be larger. More generally, we can conclude that there are certainly
reasons to switch from MLE to MAP for both models, which makes the
consequences of the switch all the more interesting – the individual-level
consequences which we have analyzed, in terms of biased matches and favored
users, and now population-level consequences, which we analyze below.

##### Theorem 3.

In the organic model as $n\rightarrow\infty$,

  1. 1.

$\text{Var}[\mathcal{Y}_{k}^{(\text{org})}]\rightarrow\frac{m-1}{m}\left[(\frac{\sigma^{2}_{\text{item}}}{\sigma^{2}_{\text{item}}+\sigma^{2}_{i}})^{2}\sigma^{2}_{\text{user}}+(\frac{1}{\sigma^{2}_{\text{item}}}+\frac{1}{\sigma^{2}_{i}})^{-1}\right]$

  2. 2.

$\text{Var}[\mathcal{Y}_{k}^{(\text{org},\text{MAP})}]\rightarrow\frac{m-1}{m}\left[\sigma^{2}_{\text{user}}+(\frac{1}{\sigma^{2}_{\text{item}}}+\frac{1}{\sigma^{2}_{i}})^{-1}\right]$.

###### Proof.

In general, we can think of each chosen item position
$y_{k_{i}}\in\mathcal{Y}_{k}$ as the sum of its expected value conditioned on
$x_{i}$, $\mathbb{E}[y_{k_{i}}]$, and some independent random noise $\epsilon$
with variance $\text{Var}[y_{k_{i}}]$ (which does not depend on $x_{i}$ in any
of our models).

Using this approach, the process of generating terms in
$\mathcal{Y}_{k}^{(\text{org})}$ in the organic MLE model can be seen as first
drawing a random user position $x$ from $N(0,\sigma^{2}_{\text{user}})$,
scaling it by
$\sigma^{2}_{\text{item}}/(\sigma^{2}_{\text{item}}+\sigma^{2}_{i})$, then
adding this noise term $\epsilon$. We scale $x$ because
$\mathbb{E}[y_{k_{i}}^{(\text{org})}]=\sigma^{2}_{\text{item}}/(\sigma^{2}_{\text{item}}+\sigma^{2}_{i})x_{i}$
(Theorem 1.1). This output becomes the sum of two random variables, where the
first term is the scaled $x$ and the second is $\epsilon$. The variances of
these terms are
$(\sigma^{2}_{\text{item}}/(\sigma^{2}_{\text{item}}+\sigma^{2}_{i}))^{2}\sigma^{2}_{\text{user}}$
and
$\text{Var}[y_{k_{i}}^{\text{org}}]=(\frac{1}{\sigma^{2}_{\text{item}}}+\frac{1}{\sigma^{2}_{i}})^{-1}$,
respectively, so the expected variance of $\mathcal{Y}_{k}^{(\text{org})}$ is
the sum of these expressions multiplied by $(m-1)/m$, because there are $m$
elements in $\mathcal{Y}_{k}^{(\text{org})}$.

In the organic MAP case, the process of generating terms in
$\mathcal{Y}_{k}^{(\text{org},\text{MAP})}$ can also be seen as first drawing
a random user position $x$ from $N(0,\sigma^{2}_{\text{user}})$, then adding a
noise term $\epsilon$. We do not scale $x$ in this case, since
$\mathbb{E}[y_{k_{i}}^{(\text{org},\text{MAP})}]=x_{i}$ (Theorem 1.2). The
output is again the sum of two random variables, where the first term is $x$
and the second is $\epsilon$. Their variances are $\sigma^{2}_{\text{user}}$
and $(\frac{1}{\sigma^{2}_{\text{item}}}+\frac{1}{\sigma^{2}_{i}})^{-1}$,
respectively, so the expected variance of
$\mathcal{Y}_{k}^{(\text{org},\text{MAP})}$ is their sum scaled by $(m-1)/m$.
∎

##### Theorem 4.

In the recommender model as $n\rightarrow\infty$,

  1. 1.

$\text{Var}[\mathcal{Y}_{k}^{(\text{rec})}]\rightarrow\frac{m-1}{m}(\sigma^{2}_{\text{user}}+\sigma^{2}_{r})$

  2. 2.

$\text{Var}[\mathcal{Y}_{k}^{(\text{rec},\text{MAP})}]\rightarrow\frac{m-1}{m}\left[\frac{\sigma^{2}_{\text{user}}}{\sigma^{2}_{\text{user}}+\sigma^{2}_{r}}\sigma^{2}_{\text{user}}\right]$.

###### Proof.

We can use the same approach from Theorem 3 to derive
$\text{Var}[\mathcal{Y}_{k}]$ here. In the recommender MLE model, the process
of generating terms in $\mathcal{Y}_{k}^{(\text{rec})}$ can be seen as drawing
a random user position $x$ from $N(0,\sigma^{2}_{\text{user}})$ then adding a
noise term $\epsilon$ that has variance
$\text{Var}[y_{k_{i}}^{(\text{rec})}]=\sigma^{2}_{r}$ (Theorem 2.1). We do not
scale $x$ here either because $\mathbb{E}[y_{k_{i}}^{(\text{rec})}]=x_{i}$.
The sum of these terms’ variances is
$\sigma^{2}_{\text{user}}+\sigma^{2}_{r}$, and again we scale this by
$(m-1)/m$.

In the recommender MAP model, to generate terms in
$\mathcal{Y}_{k}^{(\text{rec},\text{MAP})}$, we draw a random user position
$x$ from $N(0,\sigma^{2}_{\text{user}})$, scale it by
$\sigma^{2}_{\text{user}}/(\sigma^{2}_{\text{user}}+\sigma^{2}_{r})$, then add
a noise term $\epsilon$ (Theorem 2.2). The variance of the scaled $x$ is
$(\sigma^{2}_{\text{user}}/(\sigma^{2}_{\text{user}}+\sigma^{2}_{r}))^{2}\sigma^{2}_{\text{user}}$
and the variance of $\epsilon$ is
$\text{Var}[y_{k_{i}}^{(\text{rec},\text{MAP})}]=(\sigma^{2}_{\text{user}}/(\sigma^{2}_{\text{user}}+\sigma^{2}_{r}))^{2}\sigma^{2}_{r}$.
Adding these terms together, simplifying, and multiplying by $(m-1)/m$ yields
expression for $\text{Var}[\mathcal{Y}_{k}^{(\text{rec},\text{MAP})}]$. ∎

Figure 3: Variance of matched items $\text{Var}[\mathcal{Y}_{k}]$ (Theorems
3–4); $\sigma^{2}_{i}=\sigma^{2}_{r}=0.5$ and $m=300$. On the left, we fix
$\sigma^{2}_{\text{item}}=1$ and vary $\sigma^{2}_{\text{user}}$; on the
right, we fix $\sigma^{2}_{\text{user}}=1$ and vary
$\sigma^{2}_{\text{item}}$.

From Theorems 3 and 4, we can see that the variance of matched items will
always increase in the organic model when switching from MLE to MAP, but it
will always decrease in the recommender model when making the same switch. We
demonstrate these two effects in Figure 3: across different values for the
user variance $\sigma^{2}_{\text{user}}$ and item variance
$\sigma^{2}_{\text{item}}$, the matched item variance for the recommender MLE
model remains well above that of the recommender MAP, and the matched item
variance for the organic MLE model is always below that of the organic MAP.
This quantifies our earlier intuition from Figure 1 that shrinkage acts as a
diversifying force in the organic model (increasing variance), but a
homogenizing force in the recommender model (decreasing variance), and we have
proven that this is always true in this setting.

##  4 Simulations

Figure 4: Simulation results over different values of $n$. As before, we set
$\sigma^{2}_{\text{user}}=\sigma^{2}_{\text{item}}=1$,
$\sigma^{2}_{i}=\sigma^{2}_{r}=0.5$, and $m=300$. For the individual metrics,
we run 5000 trials for each model and $n$, and evaluate the mean (left) and
variance (middle) of the matched item position $y_{k_{i}}$ for a user
positioned at $x_{i}=0.75$. For the population metric, we run 500 trials for
each setting and evaluate the mean variance over the matched item positions
$\mathcal{Y}_{k}$ (right).

Theorems 1–4 demonstrated that even the simplest versions of the organic and
recommender processes result in vastly different matching behavior. However,
one may wonder how sensitive these results are to our assumptions (studying
the asymptotic case, with single-dimensional, normally distributed user and
item positions). In this section, we remove each of these assumptions, and
show via simulation experiments that the key qualitative results we
established in this section hold for much more realistic settings as well.

In Section 4.1, we implement the single-dimensional Gaussian model described
in Section 3, and test how the individual-level and population-level metrics
converge to their limits as we increase the number of available items. In
Section 4.2, we augment our process models to integrate multidimensional user
and item embeddings learned from real ratings data.

###  4.1 Exploring finite numbers of items

First, in the single-dimensional Gaussian setting we analyze the behavior of
our proposed metrics as the number of items $n$ grows, complementing our
analytical results in Section 3, where we focused on the asymptotic setting
where $n\rightarrow\infty$. We consider both the MLE and MAP versions of the
organic and recommender models and $n\in\\{4,6,\cdots,200\\}$. For the
individual metrics, we simulate 5000 stochastic trials for each model and
value of $n$, re-sampling per trial the positions of items and noisy samples,
but fixing the location of the single hypothetical user to $x_{i}=0.75$. For
the population metric, we simulate 500 stochastic trials, since there is far
less variance in this metric, and re-sample positions of users, items, and
noisy samples per trial.

As shown in Figure 4, these simulations are consistent with and extend our
earlier analyses. In the organic model, when the user employs the MLE, a
biased match is formed and the average $y_{k_{i}}$ converges inward of
$x_{i}=0.75$; as expected, converging at $0.5$, since
$\sigma^{2}_{\text{item}}=1$ and $\sigma^{2}_{i}$ = 0.5, and
$\mathbb{E}[y_{k_{i}}^{(\text{org})}]=(\sigma^{2}_{\text{item}}/(\sigma^{2}_{\text{item}}+\sigma^{2}_{i}))x_{i}$
(Theorem 1). Meanwhile, the recommender MLE model forms an unbiased match, and
the average $y_{k_{i}}$ converges to $0.75$ when the number of items $n$
reaches around 100. When MAP is used, the roles are reversed, as the organic
model becomes unbiased and the recommender model becomes biased. Furthermore,
every model converges to a different matched item variance: the recommender
MLE model has the highest variance, followed by organic MAP, then organic MLE,
then finally recommender MAP (following the order we would expect from Figure
3, at $\sigma^{2}_{\text{user}}=\sigma^{2}_{\text{item}}=1$), and this
ordering is observed when there are as few as 10 items.

We also observe that as we increase the number of items $n$, the average
$y_{k_{i}}$ and $\text{Var}[\mathcal{Y}_{k}]$ converge to their limit from the
“inside” (the side closer to 0). To provide some intuition for this behavior,
at the core of our asymptotic analyses for $\mathbb{E}[y_{k_{i}}]$ is the
understanding that when there are infinite items, the user will choose an
estimate that lies exactly at $x_{i}$, regardless of the shape of the item
distribution. However, with fewer (finite) items available, the more the item
density influences $\mathbb{E}[y_{k_{i}}]$ and contracts it towards the item
mean, which is 0. In the most extreme case, if there is only one item, then
$\mathbb{E}[y_{k_{i}}]=0$ for all users and $\text{Var}[\mathcal{Y}_{k}]=0$,
because every user will be matched to the same item. Thus, as the number of
items $n$ grows, $\mathbb{E}[y_{k_{i}}]$ and $\text{Var}[\mathcal{Y}_{k}]$
converge from the side closer to 0.

###  4.2 Empirical analysis with MovieLens data

In this section, we further extend our simulations by incorporating
multidimensional user and item positions fitted on real ratings data. We use
the MovieLens 1M dataset111https://grouplens.org/datasets/movielens/1m/.,
which contains 1,000,209 ratings from approximately 6,000 MovieLens users on
3,900 movies (Harper and Konstan, 2015). First, we filter the ratings matrix
to only keep users that have rated at least 50 movies, then filter to keep
movies with at least 50 ratings. After filtering, we are left with a rating
matrix $R\in\mathbb{R}^{m^{\prime}\times n}$, where we have $m^{\prime}=4,297$
users and $n=2,514$ movies remaining.

##### Learning user and item distributions from data.

We apply a collaborative filtering algorithm to $R$ in order to infer latent
embeddings $\mathcal{U}=[u_{i}]_{i=1}^{m^{\prime}}$, $u_{i}\in\mathbb{R}^{d}$,
for each user $i$, and $\mathcal{V}=[v_{j}]_{j=1}^{n}$,
$v_{j}\in\mathbb{R}^{d}$, for each movie $j$. Collaborative filtering uses the
history of interactions between all users and items to infer latent
representations, where users and items are encoded into low-dimensional spaces
such that if a user has given an item a high rating, their representations
should be “similar”, and if a user has given an item a low rating, their
representations should be further apart. Similarity can be measured in
different ways (e.g., inner products, Euclidean distance); since our models
take a distance-based perspective to utility, we implement a collaborative
filtering procedure (Khoshneshin and Street, 2010) that embeds users and items
into a unified Euclidean space where items that are closer to users are more
attractive to them.

In this framework, given a user embedding $u_{i}$ and movie embedding $v_{j}$,
their predicted rating $\hat{r}_{ij}$ is

$\displaystyle\hat{r}_{ij}=\mu+b_{i}-||u_{i}-v_{j}||^{2},$ (9)

where $\mu$ is the global average rating in $R$, $b_{i}$ is the user bias term
capturing users that tend to rate higher or lower, and $||u_{i}-v_{j}||^{2}$
is the squared distance between the two embeddings. This framework naturally
integrates our definition of user loss—the squared distance between a user and
her matched item—since a user’s predicted rating is exactly the negative
squared distance, translated by $\mu+b_{i}$ (which remains constant per user).
Furthermore, this formulation fits with the choice function in our models:
just as we assume that the recommender system will choose the item that
minimizes distance to its estimate of the user, that very same item here would
be the one that maximizes predicted rating.

To learn these embeddings, we define the following objective function, where
we aim to minimize the regularized loss over all observed ratings in $R$:

$\displaystyle\min_{\mathcal{U},\mathcal{V},\mathcal{B}}\sum_{i,j}w_{ij}[(r_{ij}-\hat{r}_{ij})^{2}]+\lambda(||u_{i}-v_{j}||^{2}+b_{i}^{2}).$
(10)

Here $\mathcal{B}$ represents the set of all user biases $b_{i}$, and
$w_{ij}\in\\{0,1\\}$ indicates whether the rating $r_{ij}$ is observed in $R$.
We then use gradient descent to update the latent parameters $\mathcal{U}$,
$\mathcal{V}$, and $\mathcal{B}$ with respect to the regularized loss. After
learning $\mathcal{U}$, i.e., the embeddings of the users in the MovieLens
dataset, we sample from this empirical distribution to generate $m$ “test”
users. This creates a new matrix of user positions,
$\mathcal{X}\in\mathbb{R}^{m\times d}$, which we use in our simulations to
represent unseen users. Our simulations do not rely on unseen movies, however,
so we can directly set $\mathcal{Y}$, the item positions in our simulations,
to $\mathcal{V}$, the movie positions directly inferred from the MovieLens
dataset.

#### Simulations with MovieLens embeddings.

Figure 5: Distribution of learned user (top) and movie (bottom) embeddings
from the MovieLens 1M dataset.

Using this approach, we learn user and movie embeddings of dimension $d=5$
from the MovieLens dataset. In Figure 5, we display the learned user and movie
distributions. We see that the user distribution is less dispersed than the
movie distribution in each dimension, and the movie distribution is sometimes
asymmetric (clearly non-Gaussian). For both embeddings, the covariance between
dimensions is low.

We then simulate the organic and recommender models with $m=300$ users and
$n=2,514$ movies (the number of movies in the filtered MovieLens dataset). For
the organic model, we fix each user’s noise covariance $\Sigma_{i}$ to
$0.5\cdot\Sigma_{\text{item}}$, where $\Sigma_{\text{item}}$ was the empirical
covariance fitted on $\mathcal{Y}$, the movie embeddings. Similarly, for the
recommender model, we fixed the system’s noise covariance $\Sigma_{r}$ to
$0.5\cdot\Sigma_{\text{user}}$, where $\Sigma_{\text{user}}$ was the empirical
covariance fitted on $\mathcal{X}$, the user embeddings. We scaled the user /
movie covariance in this way, as opposed to using spherical noise, so that the
size of the noise per dimension would scale with the variance of the estimated
population in that dimension, which we believed was more realistic. When
shrinkage is applied within each model, we no longer have the user or system
construct MAP estimates with Gaussian priors: the prior distribution is
neither Gaussian nor known. Instead, we implemented shrinkage parameterized by
a scalar $\alpha\in[0,1]$, which interpolates between the MLE estimate and the
mean over all MLE estimates. That is, user $i$’s shrunken estimate
$\hat{y}_{j}^{(i,S)}$ for movie $j$ becomes

$\displaystyle\hat{y}_{j}^{(i,S)}=(1-\alpha)\hat{y}_{j}^{(i,\text{MLE})}+\alpha(\frac{1}{n}\sum_{k=1}^{n}\hat{y}_{k}^{(i,\text{MLE})}),$
(11)

and the recommender system’s shrunken estimate of user $i$ is the equivalent
interpolation between $\hat{x}_{i}^{(r,\text{MLE})}$ and the mean MLE estimate
over all users. This family of estimators generalizes James–Stein shrinkage
and Empirical Bayes estimators, which correspond to specific recipes for
choosing $\alpha$ (or $\alpha_{i}$, different for each user).

Figure 6: Individual-level results from MovieLens experiments. We plot each
user’s average squared distance to their 10 nearest movies (as a measure of
centrality) vs. the user’s loss, averaged over 500 trials. In order to see
data points more clearly, we truncated outliers beyond the 95th percentile of
the x-axis (showing 285 out of 300 users). Figure 7: Population-level results
from MovieLens experiments. We assess the effect of varying the shrinkage
parameter $\alpha$ on the MSE and variance of matched item positions
$\text{Var}[\mathcal{Y}_{k}]$, running 500 trials for each model and value of
$\alpha$.

Despite the lopsided movie distribution and expanded number of dimensions, we
find that the key qualitative results from our theoretical analyses hold. In
Figure 6, we see that the organic MLE model continues to strongly favor more
central users, compared to the recommender MLE model, which is much more even-
handed across user positions. In our earlier setting with items drawn from
$N(0,\sigma^{2}_{\text{item}})$, the centrality of a user could be summarized
by the absolute value of $x_{i}$, but here we need a more empirical measure;
for example, we use the user’s average squared distance to their 10 nearest
movies (where higher average distance corresponds to lower centrality).

Secondly, we simulate different amounts of shrinkage
($\alpha\in\\{0.05,0.1,\cdots,0.95\\}$) and evaluate the impacts on our
population-level metrics, the mean user loss (MSE) and the variance of matched
item positions $\text{Var}[\mathcal{Y}_{k}]$. In multiple dimensions, we
compute the generalized variance as the product of the eigenvalues of the
covariance matrix of $\mathcal{Y}_{k}$; we log-transform this quantity to make
it more interpretable. As we increase the amount of shrinkage, at first this
improves MSE for both models, but eventually MSE begins to increase; the
optimal level of shrinkage seems to fall around $\alpha\approx 0.4$ for both
models. However, even though the MSE curves look similar, the
$\text{Var}[\mathcal{Y}_{k}]$ curves completely diverge: the more we increase
$\alpha$, the more $\text{Var}[\mathcal{Y}_{k}]$ grows in the organic model,
and the more it shrinks in the recommender model. This matches our earlier
findings that shrinkage has a “diversifying” effect on the organic model but a
“homogenizing” effect on the recommenders.

##  5 Conclusion

We have introduced two contrasting models of item-matching processes: one
describing a generic personalized recommender system, and the other describing
a setting where users search for items organically without the mediation of
any system. In both cases, we have a decision-making agent trying to match
users to items with limited information, but the difference lies in who/what
is doing the matching, and what information they have access to vs. what is
being estimated. Comparing the two, we have seen that this simple switch in
perspective results in dramatic differences at both the individual and
population levels. For example, in the MLE versions of the models, the
recommender model forms unbiased item matches for the users, while the organic
model does not, and the organic model favors central users, while the
recommender model does not. Applying shrinkage has notably diverging effects
on the two models: while it can reduce MSE in both cases, shrinkage leads the
recommender model to choose increasingly similar sets of items, while leading
the organic model to diversify its selections. These findings, which are
robust to changes in user and item distribution, single and multiple
dimensions, and asymptotic and finite settings, indicate pervasive differences
between recommended and organic processes of item-matching.

We have worked to study one of the core counterfactual questions about
recommender systems: if there were no recommender system, how would the world
be different? Our analyses provide evidence that the use of recommender
systems fundamentally alters how humans interact with content. Recommender
systems are usually touted as reducing search frictions in markets, but
through our work we find that such reductions in search friction comes with
myriad subtle trade-offs. We hope that future studies can build upon the
framework established in this paper, using our models as building blocks to
analyze diverse potential effects of recommender systems, such as
polarization, filter bubbles, user retention rates, or fairness across users
or products. As a specific direction, by modeling repeated choices and
incorporating learning dynamics where users learn from the items they consume
(Diehl, Weeks, and Zúñiga, 2015), it should be possible to analyze how the
phenomena we document here, pertaining to single choices, compound over time
and affect the evolution of users’ opinions and behaviors. Finally, we also
hope in future work to tie the theory of our model to real-world experiments;
for example, through randomized trials where participants are assigned to
unmediated organic search versus system-mediated search processes, to
understand the differences most broadly.

## Acknowledgements

This work was supported in part by funding from the Stanford Program on
Democracy and the Internet and ARO MURI award #W911NF-20-1-0252. We thank Jan
Overgoor, Amel Awadelkarim, Arjun Seshadri, and Kaitlyn Zhou for helpful
feedback and discussions.

## References

  * Abdollahpouri, Burke, and Mobasher (2017) Abdollahpouri, H.; Burke, R.; and Mobasher, B.  2017\.  Controlling popularity bias in learning-to-rank recommendation.  In RecSys ’17, 224–232.
  * An et al. (2014) An, J.; Quercia, D.; Cha, M.; Gummadi, K.; and Crowcroft, J.  2014\.  Sharing political news: the balancing act of intimacy and socialization in selective exposure.  EPJ Data Science 3(12).
  * Anderson et al. (2020) Anderson, A.; Maystre, L.; Anderson, I.; Mehrotra, R.; and Lalmas, M.  2020\.  Algorithmic effects on the diversity of consumption on spotify.  In WWW ’20, 2155–2165.
  * Bakshy, Messing, and Adamic (2015) Bakshy, E.; Messing, S.; and Adamic, L.  2015\.  Exposure to ideologically diverse news and opinion on facebook.  Science 348(6238):1130–1132.
  * Bozdag (2013) Bozdag, E.  2013\.  Bias in algorithmic filtering and personalization.  Ethics and information technology 15:209–227.
  * Brynjolfsson, Hu, and Simester (2011) Brynjolfsson, E.; Hu, Y.; and Simester, D.  2011\.  Goodbye pareto principle, hello long tail: The effect of search costs on the concentration of product sales.  Management Science 57(8):1373–1386.
  * Chaney, Stewart, and Engelhardt (2018) Chaney, A.; Stewart, B.; and Engelhardt, B.  2018\.  How algorithmic confounding in recommendation systems increases homogeneity and decreases utility.  In RecSys ’18, 224–232.
  * Chen et al. (2020) Chen, J.; Dong, H.; Wang, X.; Feng, F.; Wang, M.; and He, X.  2020\.  Bias and debias in recommender system: A survey and future directions.  arXiv preprint arXiv:2010.03240.
  * Dandekar, Goel, and Lee (2013) Dandekar, P.; Goel, A.; and Lee, D. T.  2013\.  Biased assimilation, homophily, and the dynamics of polarization.  PNAS 110(15):5791–5796.
  * Das et al. (2007) Das, A. S.; Datar, M.; Garg, A.; and Rajarm, S.  2007\.  Google news personalization: scalable online collaborative filtering.  In WWW ’07, 271–280.
  * Diehl, Weeks, and Zúñiga (2015) Diehl, T.; Weeks, B.; and Zúñiga, H.  2015\.  Political persuasion on social media: Tracing direct and indirect effects of news use and social interaction.  New Media & Society.
  * Efron and Morris (1976) Efron, B., and Morris, C.  1976\.  Multivariate empirical bayes and estimation of covariance matrices.  The Annals of Statistics 4(1):22–32.
  * Flaxman, Goel, and Rao (2016) Flaxman, S.; Goel, S.; and Rao, J. M.  2016\.  Filter bubbles, echo chambers, and online news consumption.  Public Opinion Quarterly 80:298–320.
  * Fleder and Hosanagar (2009) Fleder, D., and Hosanagar, K.  2009\.  Blockbuster culture’s next rise or fall: The impact of recommender systems on sales diversity.  Management science 55(5):697–712.
  * Garimella et al. (2018) Garimella, K.; Morales, G. D. F.; Gionis, A.; and Mathioudakis, M.  2018\.  Political discourse on social media: Echo chambers, gatekeepers, and the price of bipartisanship.  In WWW ’18, 913–922.
  * Gupta et al. (2013) Gupta, P.; Goel, A.; Lin, J.; Sharma, A.; Wang, D.; and Zadeh, R.  2013\.  WTF: The who to follow service at twitter.  In WWW, 505–514.
  * Harper and Konstan (2015) Harper, F. M., and Konstan, J. A.  2015\.  The movielens datasets: History and context.  In ACM Transactions on Interactive Intelligent Systems.
  * Holtz et al. (2020) Holtz, D.; Carterette, B.; Chandar, P.; Nazari, Z.; Cramer, H.; and Aral, S.  2020\.  The engagement-diversity connection: Evidence from a field experiment on spotify.  In EC ’20, 75–76.
  * Hosanagar et al. (2013) Hosanagar, K.; Fleder, D.; Lee, D.; and Buja, A.  2013\.  Will the global village fracture into tribes? recommender systems and their effects on consumer fragmentation.  Management Science 60(4).
  * James and Stein (1961) James, W., and Stein, C.  1961\.  Estimation with quadratic loss.  In Proc. Fourth Berkeley Symp. Math. Statist. Prob., 1961, volume 1, 361–380.
  * Khoshneshin and Street (2010) Khoshneshin, M., and Street, W. N.  2010\.  Collaborative filtering via euclidean embedding.  In RecSys ’10, 87–94.
  * Kloumann and Kleinberg (2014) Kloumann, I. M., and Kleinberg, J. M.  2014\.  Community membership identification from small seed sets.  In KDD ’14, 1366–1375.
  * Lambrecht and Tucker (2019) Lambrecht, A., and Tucker, C.  2019\.  Algorithmic bias? an empirical study of apparent gender-based discrimination in the display of stem career ads.  Management Science 65(7).
  * Nguyen et al. (2014) Nguyen, T. T.; Hui, P.; Harper, F. M.; Terveen, L.; and Konstan, J. A.  2014\.  Exploring the filter bubble: the effect of using recommender systems on content diversity.  In WWW ’14, 677–686.
  * Pariser (2011) Pariser, E.  2011\.  The filter bubble: What the Internet is hiding from you.  Penguin.
  * Perra and Rocha (2019) Perra, N., and Rocha, L.  2019\.  Modelling opinion dynamics in the age of algorithmic personalisation.  Nature Scientific Reports 9(7261).
  * Rastegarpanah, Gummadi, and Crovella (2019) Rastegarpanah, B.; Gummadi, K. P.; and Crovella, M.  2019\.  Fighting fire with fire: Using antidote data to improve polarization and fairness of recommender systems.  In WSDM ’19, 231–239.
  * Singer (2011) Singer, N.  2011\.  The trouble with the echo chamber online.  New York Times.
  * Stoica and Chaintreau (2019) Stoica, A., and Chaintreau, A.  2019\.  Hegemony in social media and the effect of recommendations.  In 1st FATES Workshop, WWW’19, 575–580.
  * Summers, Smith, and Reczek (2016) Summers, C.; Smith, R.; and Reczek, R.  2016\.  An audience of one: Behaviorally targeted ads as implied social labels.  J. Consumer Research 43(1):156–178.
  * Yao and Huang (2017) Yao, S., and Huang, B.  2017\.  Beyond parity: Fairness objectives for collaborative filtering.  In NeurIPS ’17, 2925–2934.