# Revenue, Relevance, Arbitrage and More: Joint Optimization Framework for
Search Experiences in Two-Sided Marketplaces

Andrew Stanton, Akhila Ananthram, Congzhe Su, Liangjie Hong  Etsy, IncNew
YorkU.S.A astanton, aananthram, csu, lhong @ etsy.com

(2019)

###### Abstract.

Two-sided marketplaces such as eBay, Etsy and Taobao have two distinct groups
of customers: buyers who use the platform to seek the most relevant and
interesting item to purchase and sellers who view the same platform as a tool
to reach out to their audience and grow their business. Additionally,
platforms have their own objectives ranging from growing both buyer and seller
user bases to revenue maximization. It is not difficult to see that it would
be challenging to obtain a globally favorable outcome for all parties. Taking
the search experience as an example, any interventions are likely to impact
either buyers or sellers unfairly to course correct for a greater perceived
need. In this paper, we address how a company-aligned search experience can be
provided with competing business metrics that E-commerce companies typically
tackle. As far as we know, this is a pioneering work to consider multiple
different aspects of business indicators in two-sided marketplaces to optimize
a search experience. We demonstrate that many problems are difficult or
impossible to decompose down to credit assigned scores on individual
documents, rendering traditional methods inadequate. Instead, we express
market-level metrics as constraints and discuss to what degree multiple
potentially conflicting metrics can be tuned to business needs. We further
explore the use of policy learners in the form of Evolutionary Strategies to
jointly optimize both group-level and market-level metrics simultaneously,
side-stepping traditional cascading methods and manual interventions. We
empirically evaluate the effectiveness of the proposed method on Etsy data and
demonstrate its potential with insights.

datasets, neural networks, learning to rank, evolutionary strategies,
e-commerce

††ccs: Information systems Learning to rank††ccs: Information systems
Information retrieval diversity

##  1\. Introduction

As online shopping becomes a dominant avenue for global buyers, E-commerce
companies strive to meet a wide range of often conflicting goals when showing
products for their communities. While optimizing for buyer Conversion Rate,
and therefore higher general revenue or Gross-Merchandise-Value (GMV), is
commonly considered the top objective, it is usually far from driving a
healthy and growing business. In fact, many E-commerce companies, especially
those with two-sided marketplaces, face a number of challenges due to over-
shackling to GMV optimization alone. Without appropriate tempering, such a
platform is usually unable to satisfy the short term needs of buyers and
sellers as well as the long term needs of the business.

A typical two-sided marketplace such as eBay, Etsy and Taobao has two distinct
groups of customers where buyers use the platform to seek the most relevant
and interesting item to purchase and sellers view the same platform as a tool
to reach out to their audience and grow their business. On top of that, the
platform normally would have its own objectives ranging from growing both
buyer and seller user bases to GMV maximization. It is not difficult to see
that it would be challenging to obtain a globally favorable outcome for all
parties.

Take showing relevant products to a buyer through the search experience as an
example. For a particular purchase intent, or sometimes with a specific item
in mind, a buyer would likely discover a spectrum of product listings from
multiple sellers in a typical two-sided marketplace. Some seem to be more
relevant than others and some even might look the same. A buyer has to decide
among these items with a positive experience such that he/she would return to
the marketplace next time. For sellers, however, they view search result pages
as prominent real estate to gain customers’ attention and therefore
potentially increase their market share. Maximizing their success in search
benefits both their brand and take home pay, regardless of buyers’ overall
experience on the site or ramifications to other sellers. On the mission of
growing a marketplace, the platform generally needs to step in and sometimes
artificially advantage under represented segments of sellers to give them more
exposure, creating a reasonably fair competition. However, it might be equally
risky to put established sellers in disadvantaged situations, who originally
rank well on their own merit, and provide a sub-standard experience to buyers
as potentially fewer relevant and lower quality goods are exposed higher
through search results.

This scenario exemplifies a microcosm of the marketplace: any interventions
are likely to impact either buyers or sellers unfairly to course correct for a
greater perceived need. On another hand, a two-sided platform also needs to be
cautious about the situation where the rich get richer and the poor get
poorer, namely the Matthew Effect, a factor of constant battle in most
marketplaces (Berbeglia and Van Hentenryck, 2017). While sellers often perform
well due to better commercial strategies and product fit than their peers, a
healthy marketplace cannot be solely dominated by a small segment of power
sellers and continue to grow. To make things even more complicated, platforms,
often operating as modern corporations, subsequently attempt to compensate for
these inefficiencies with organizations and teams devoted to their respective
customer: for example, buyer, seller, and core market. Hence, as each team
attempts to solve a particular problem set, competing needs are demanded of
the search experience with each team expecting tuning for their particular
business focus. As often these asks are ill-defined and heuristically
measured, a grand challenge for building a search ranking algorithm to satisfy
all fronts is presented.

In this paper, we address how a company-aligned search experience can be
provided with competing business metrics that E-commerce companies typically
tackle. As far as we know, this is a pioneering work to consider multiple
different aspects of business metrics in two-sided marketplaces to optimize a
search experience. We demonstrate that many problems are difficult or
impossible to decompose down to credit assigned scores on individual
documents, rendering standard point-wise approaches to multi-objective (Svore
et al., 2011) or standard diversity-based(Santos et al., 2015) learning to
rank algorithms inadequate. Instead, we express market-level metrics as
constraints and discuss to what degree multiple potentially conflicting
objectives can be tuned to business needs. In addition, we propose a policy
learner in the form of Evolutionary Strategies to jointly optimize both group-
level and market-level metrics simultaneously, side-stepping traditional
cascading methods and manual interventions.

The paper is organized as follows. In $\S$2, we discuss related work in
several different directions. We then formulate and define a wide range of
metrics including relevancy metrics, diversity metrics and a number of newly
proposed market-level metrics relevant to E-commerce interests in Section
$\S$3. We follow up with a proposed set of policies to optimize the above
metrics in $\S$4. Finally in $\S$5, we empirically evaluate the effectiveness
of proposed method on Etsy search logs data, showing how different weightings
influence the ultimately delivered rankings.

##  2\. Related Work

There are number of different facets of work that need to be considered when
ranking across a variety of soft constraints.

###  2.1. Diversity

Diversity in Learning to Rank has long and storied past as it relates to Web
Search. The simplest solution typically falls under a heuristic based
approach. Carbonell and Goldstein(Carbonell and Goldstein, 1998) formulated
the problem as a selection: Documents are chosen greedily based on a linear
combination of query-document relevance and maximal margin relevance (MMR),
with each step picking the document which the highest combined score. After
selection, the MMR scores are updated to reflect the newly picked document.
MMR is rooted in the idea of novelty, the idea that maximizing the differences
in similarity between the set of already selected documents and the remaining
set results in a more diverse outcome. Subsequent work attempts to define
better heuristics for selection (Santos et al., 2015). Dang and Croft propose
PM-2 (Dang and Croft, 2012), a diversification method based on
proportionality; they argue diversification should be biased toward the
overall subtopic proportionality of the entire query-set rather than
attempting to balance it uniformly. xQuAD (Santos et al., 2010) attempts to
understand diversity as a combination of an originating query and derived sub-
queries.

In the learning space, the closest related work comes from Xia et al. who
describe PAMM (Xia et al., 2015), a method for optimizing diversity and
relevancy via a Perceptron. Novel to the paper is the idea of direct
optimization of the evaluation metrics rather than utilizing heuristics or
optimizing surrogate functions. PAMM works generally by sampling both
positively and negatively ranked lists and attempts to maximize the margin
between them.

Crucially, all of the above attempt to solve the challenge of query ambiguity;
the idea that redundancy is undesired due to lack of strong conviction of the
topicality of a given query. This positions the frameworks to satisfy the
needs of the searcher, with the underlying objective of improving the
browser’s experience. While two-sided marketplaces face similar challenges to
query ambiguity, they also need to balance the needs of the seller: diversity
for the sake of mitigating bias. To our knowledge, none of the proposed work
attempt to optimize for market-level diversity metrics.

###  2.2. Policy Learning

Policy optimization in LTR space has come in a few flavors over the years,
with most of its history focused in online LTR. Radlinski et al. (Radlinski et
al., 2008) first discussed diversity-based online optimization in the form of
using multi-armed bandits to minimize page abandonment. They condition the
expected reward on previous documents selected, considering each remaining
document an ”arm”, allowing for suitable exploration/exploitation trade-off
against the expected reward. More recent work proposes modeling user behavior
as an MDP(Hu et al., 2018) with the goal of learning how browser sessions can
be utilized in re-ranking. They proceed to describe a policy gradient method
to learn optimal ranking policies given the learned SS-MDP.

Most applicable to our proposed policy is utilizing black box optimization in
learning to rank. Salimans et al. recently showed that Evolutionary Strategies
(Salimans et al., 2017) were well suited for learning reinforcement problems,
applying a variation of ES known as Natural Evolutionary Strategies(Wierstra
et al., 2008) to Atari game learning. They proposed a scalable algorithm and
demonstrated the optimizer’s tolerance to stochastic environments. (Chrabaszcz
et al., 2018) showed competitive results to Salimans via Canonical ES \- a
simpler version of the ($\lambda$, 1) variants of Evolutionary Strategies.
Concurrently, Ibrahim et al. applied perhaps the simplest type of
(1+1)-Evolutionary Strategies to learn policies on linear models to directly
optimize the average nDCG across all query sets(Ibrahim and Landa-Silva,
2018).

###  2.3. Popularity Bias and the Matthew Effect

Measuring market level performance within search is fairly under researched in
the space of E-Commerce. Perhaps closest to our particular use case is from
Hentenryck et al. (Van Hentenryck et al., 2016) who analyzed the impacts of
social influence on a trial-offer market, showing that ranking on conditional
purchase rate lead to natural monopolies by the highest quality products.
Follow up work attempts to compensate for the Matthew Effect (Berbeglia and
Van Hentenryck, 2017) by intervening with a stochastic policy to randomize
products of similar quality. They show how segmenting products into different
”worlds” and conditioning popularity on which world a user observes results in
a stable market where products of similar quality obtain equivalent market
share.

##  3\. Metrics For Optimization

In this section, we outline metrics for optimization from a typical two-sided
marketplace. We review classic relevancy metrics in $\S$3.1 as well as
diversity metrics in $\S$3.2, serving the foundation of metrics for modern
search ranking. We introduce a new class of market-level metrics in $\S$3.3.
We list all notations used through out the paper in Table 1.

Table 1. Notation Notation | Description
---|---
$Q=\\{q_{1},...,q_{n}\\}$ | Unique query set
$N_{i}$ | Number of documents in query set $q_{i}$
$D=\\{d_{1},...,d_{j}\\}$ | Document set
$Feats(d_{i})$ | Feature vector for document $d_{i}$
$Y\in\\{1,2,3,4,5\\}$ | Relevance grades
$y_{i,j}\in Y$ | Relevance score for $q_{i}$ and $d_{j}$
$\pi$ | Ranking policy
$R(\pi,q_{i})=<d_{1},\ldots)>$ | Ranked documents for query $q_{i}$
$V(\pi,q_{i},d_{j})$ | Evaluator for query-doc set
$S(\pi,Q,D)\in[0,1]$ | Market-level objective function

###  3.1. Relevancy Metrics

First and foremost is the concept of relevancy, rooted originally in the well-
known Probability Ranking Principle (PRP) framework (Robertson, 1977; Cooper,
1971) which states that documents should be ordered independently in
decreasing presentation of relevance. That is, the most relevant document for
a query should be placed first. To measure that principal, industry has
standardized around two core metrics for evaluating the efficacy of their
ranking systems: NDCG (Järvelin and Kekäläinen, 2000) and ERR (Chapelle et
al., 2009). We now give a brief overview of their formulation.

Normalized Discounted Cumulative Gain (NDCG): NDCG is an ordered relevance
metric measuring the agreement between a goldset list of documents and the
permutation return by the ranking policy. It is typically evaluated to some
position $K$, indicating only the first $K$ documents should be considered for
evaluation. Usually values are small, emphasizing the importance of getting
the first few documents correct. Given a query $q_{i}$:

(1)
$\displaystyle\begin{split}\mathrm{DCG@K}&=\sum_{j=1}^{K}\frac{2^{y_{i,j}}-1}{\log_{2}(i+1)}\\\
\mathrm{NDCG@K}&=\frac{\mathrm{DCG@K}}{\mathrm{IDCG@K}}\in[0,1]\end{split}$

Where IDCG is the best possible DCG score for the given query-doc set. We
typically evaluate the average NDCG@K across all queries.

Expected Reciprocal Rank (ERR): ERR (Chapelle et al., 2009) is proposed as an
adjustment to NDCG, attempting to factor in a prior to how users actually
consider documents for engagement. While graded labels are still assigned to
documents independently, ERR is grounded on the idea of the cascade user model
(Craswell et al., 2008): that previous evaluations of documents influences the
likelihood that a buyer will continue browsing. While a full discussion on ERR
is out of the scope of this paper, we provide the following formulation used
in evaluation. Given some mapping $R$ of relevance grades to probability of
relevance, we can define ERR as:

$\displaystyle R(g)$ $\displaystyle=\frac{2^{g}-1}{2^{\max(Y)}}$
$\displaystyle p_{0}$ $\displaystyle=1,p_{j}=p_{j-1}(1-R(y_{i,j}))$
$\displaystyle\mathrm{ERR}_{0}$
$\displaystyle=0,\mathrm{ERR}_{j}=\mathrm{ERR}_{j-1}+p_{j-1}\frac{R(y_{i,j})}{j}$

###  3.2. Diversity Metrics

Evaluation of sub-topic diversity is a rich field with many contributions
(Zhai et al., 2015; Agrawal et al., 2009; Chapelle et al., 2011; Clarke et
al., 2008). Generally, the metrics revolve around the idea that there exist
ambiguity in the intent behind a query. For example, while there is likely a
strong relationship between the query “lace bridal veil” and the
/clothing/wedding/accessory/veils taxonomy, for other queries there is less
implicit understanding. On Etsy, we serve a large number of inspirational
(cheerful, happy, beautiful), stylistic (geometric, upcycled, animal print),
and occasion (gifts for him, bridesmaid presents, stocking stuffers) queries
which have high taxonomic (interchangeably used with topicality) ambiguity. In
cases where there is low certainty of strong topicality, ranking benefits from
increasing coverage of different sub topics early on in the presented result
set. Indeed, empirical results have shown increasing diversity improves user
engagement metrics (Raman et al., 2012).

ERR-IA: ERR-IA (Chapelle et al., 2009) is an extension to ERR that
incorporates the notion of diversity. Let $\Pr(t\mid q)$ be the probability of
a topic, t, for a given query, q. We can define ERR-IA as:

$R(y_{i},t)=\begin{cases}R(y_{i})&i\in t\\\ 0&else\end{cases}$
$\mathrm{ERRIA@K}=\sum_{j=1}^{K}\frac{1}{j}\sum_{t}\Pr(t\mid
q)\prod_{i=1}^{j-1}(1-R(y_{i},t))R(y_{j},t)$

Or, using the ERR formula from before, we can define it as:

(2) $\begin{split}\mathrm{scores}=[y_{i}*\mathbbm{1}{[i\in cat_{t}]},\forall
i\in[1,K]]\\\ \mathrm{ERRIA@K}=\sum_{t}\Pr(t\mid
q)*\mathrm{ERR@K}(\mathrm{scores})\end{split}$

###  3.3. Market-Level Metrics

Two sided marketplaces, unlike traditional web search, suffers from multiple
challenges typically framed in the form of inequality; the realization that
there exists some skew in the marketplace we wish to correct. In this sub
section we introduce a wide range of different types of market corrections and
present potential metrics for optimization. Before we introduce the first
proposed metric, we discuss the notion of query-set dependence below.

Query-Set Dependence: At first glance, many metrics seem like they could be
modeled as group-wise diversity problems. Let us consider the case of
balancing between highly successful power sellers and new sellers on the site.
While majority of the sales will come from a fraction of the entire seller
base, an over saturation of a small proportion of sellers in search has the
potential to squeeze out new shops on the site. As there is strong evidence
that faster time to first sale increases a seller’s Life Time Value (LTV), we
would ideally like to improve their exposure. Applying a diversity constraint
between power sellers and new sellers seems reasonable: for each query set,
increase coverage between the two distributions in the top $K$ spots. However,
this ignores a fundamental problem: different queries have different amounts
of traffic associated with them, preventing us from properly balancing the
market. Much like the PRP models before, diversity metrics assume query-level
independence: we only consider how the impact of diversification adjusts the
metrics within each query set, not its contribution to the overall market
place. Another simple example can illustrate the differences. If we have two
ranked sets of items, where $N$ indicates a new seller and $P$ indicates a
power seller, for rankings $\\{P,P,N,N\\}$ and $\\{N,N,P,P\\}$, the average
diversity scores for both queries will be low. However, from an inequality
perspective, both power sellers and new sellers have equal representation in
top spots, satisfying the market level needs. Overall, we can write our
general market indicator function with only a slight abuse of notation as:

(3) $I(S|Q_{\pi}),S\in[0,1]$

where $Q_{\pi}$ is the rankings of all documents across all queries by policy
$\pi$.

####  3.3.1. Weighted Importance Ranking

This leads us to our first proposed metric: Weighted Importance Ranking. Under
the assumption that we want to balance some scoring function $S_{sub}$
according to some importance weighting, such as traffic volume, we can
describe our score function S as:

(4)
$S(\pi)=\frac{\sum_{i=1}^{N}W_{i}*S_{sub}(\pi,q_{i},d_{i})}{\sum_{i=1}^{N}W_{i}}$

This provides a useful feedback mechanism to the policy learner: much like the
intuition that rankers optimizing for NDCG or ERR should focus their energy on
improving the rank of documents higher in the page, providing importance
feedback provides guidance to the policy on which queries it should focus time
on optimizing.

####  3.3.2. Outlier Skew

When dealing with implicit feedback data, chronic cold starts, or otherwise
uncertainty in the relevance set, optimizing for the expected NDCG across all
queries can lead to an over-sensitivity to outliers: query sets that are
either trivial or impossible to successfully rank given the features.

To address the influence, we propose a simple change to the optimization
function to maximize the scores at given percentiles instead of the mean.
Given the set of scores, $M$, a set of percentiles to evaluate (e.g. 25th,
75th, etc.), $\mathrm{Percentiles}$:

(5) $\begin{split}M&=\\{V(\pi,q_{1},d_{q}),...,V(\pi,q_{N},d_{N})\\}\\\
S(\pi)&=\frac{\sum_{p\in\mathrm{Percentiles}}M_{(}p)}{|\mathrm{Percentiles}|}\end{split}$

Empirically, we have found that optimizing quantiles can provide a smoother
distribution.

####  3.3.3. Incentives

Many cases where business wishes to correct market bias can boil down to
minimizing arbitrage. For example, it might be observed that Buyers find items
with low list prices attractive and yet are surprised when confronted with
costly shipping. Sellers will often discover these user behaviors can yield a
higher collection of clicks, a standard ranking signal, and will list items
with artificially low prices to improve their ranking in Search.

Incorporating incentives in a principled way is difficult. As a marketplace
which caters to many different price points, many of which are materially
cheap to produce, this rules out simple heuristics or other hard rules.
Analysis might come up with some rule, say listing prices lower than the
median value of a product should be down weighted in search (also known as
burying), but that goes too far: we now penalize sellers producing goods more
efficiently.

Rather than exert a hard penalty as a group-wise metric, we instead propose a
simple maximization approach across the top ranked documents for all queries.
Given some user-provided behavior function, $B$, indicating that a document
for a given query exhibits a quality we wish to incentivize, a ranking of
documents for query $q_{i}$, and the number of positions, $K$, to consider:

(6) $\begin{split}pos&\in\\{1,...,N_{i}\\}\\\ B(q_{i},d_{j})&={0,1}\\\
S(\pi)&=\frac{\sum_{i=1}^{|Q|}\sum_{p=0}^{K}B(q_{i},R(\pi,q_{i})@p)}{K|Q|}\end{split}$

Combined with Weighted Importance Ranking in Equation 4, we can influence
desired behaviors conditioned on relevance and group-level diversity.

####  3.3.4. Inequality

Figure 1. Seller Distribution

The final class of market level metrics fall under the guise of inequality:
some imbalance in wealth distribution across tiers of sellers that we wish to
correct. Without interventions, marketplaces often fall under the phenomena of
the Matthew Effect; wealth accumulates in a small segment of the population,
squeezing out other Sellers. Etsy is no different: Figure 1 shows how
visibility accumulates in the wealthy few: the top few% of sellers account for
the bulk of sales. Indeed, it can be shown that models that maximizing
conditional purchase rate will result in natural monopolies (Van Hentenryck et
al., 2016) due to social signals present in the marketplace (reviews, best
sellers, number of sales, etc.). While beneficial to maximizing sales, it
carries inherent business risk: any loss of sellers occupying the monopoly
positions for popular queries will have outsize effect on bottom line KPIs.

While work has been done examining how different rankings can impact a
market’s health (e.g. maximizing number of purchases) (Berbeglia and Van
Hentenryck, 2017), there is little discussion on improving proportional
representation across tiers of sellers jointly.

Given our desire to influence inequality, we examine two methods for
flattening the distribution, depending on the amount of prior information we
have.

Gini Index: In the case where we can estimate or measure the wealth
distribution apriori, we propose minimizing the Gini Index(Gini, 1912), also
known as the Gini Coefficient, across all query sets. Based on the Lorenz
Curve, it estimates the difference between full income equality and the actual
observed wealth distribution of different sub populations. Given a cumulative
proportion of the population, $X_{i}$ and a cumulative proportion of wealth
$W_{i}$, ordered such that $w_{i}/x_{i}\leq w_{i+1}/x_{i+1}$ we can define the
Gini Index indicator:

$\displaystyle\mathrm{Gini}$
$\displaystyle=1-\sum_{i=2}^{|X|}(X_{i}-X_{i-1})(\mathrm{W}_{i}+\mathrm{W}_{i-1})$
$\displaystyle S(\pi)$ $\displaystyle=1-\mathrm{Gini}$

There are a number of different ways to compute the population: total count of
different sub populations across all inventory, such as seller deciles, the
subset represented in the train dataset, etc. Similarly, there are many ways
we can measure wealth; for E-commerce businesses, purchase count per query can
be a reasonable approximation for seller wealth where, given some function
$\mathrm{IsSubPop}$:

$\displaystyle w_{i}$
$\displaystyle=\sum_{j=1}^{|Q|}\mathrm{IsSubPop}(\pi,q_{j},i)\cdot
Purchases(q_{j})$

In the case where we are more interested in the traffic distribution, W can be
set to the amount of query volume. For simplicity, in our experiments, we
compute Gini for rank=1. However, computing the Gini Coefficient over multiple
rank positions is similarly straight forward if we have access to the
observation probability O of a document at position p (such as from a click
model):

(7) $\begin{split}O(\mathrm{Pos})&\in[0,1]\\\
S_{\mathrm{pos}}(\pi)&=O(p)*\mathrm{Gini}\end{split}$

$\mathcal{\chi}^{2}$ Uniformity: In cases where P and W are unknown, we can
use a simpler method for measuring inequality: maximizing the inverse of the
$\chi^{2}$ fit against a uniform distribution:

(8)
$\begin{split}\mathrm{Count}(\pi,cat)&={\sum_{p=1}^{K}\sum_{i=1}^{|Q|}\mathrm{IsCat}(\pi,R(\pi,q_{i},p),\mathrm{cat})}\\\
\mathcal{\chi}_{uniform}^{2}&=\frac{\sum_{c\in
C}(\mathrm{Count}(\pi,c)-\frac{|Q|}{|C|})^{2}}{\frac{|Q|}{|C|}}\\\
S(\pi)&=\frac{1}{1+\mathcal{\chi}_{\mathrm{Uniform}}^{2}}\end{split}$

##  4\. Algorithm Overview

Learning a ranking policy requires multiple levels of information: individual
scores in the case of relevancy, group-level metrics for diversity, and
market-level to account for a variety of skews. Further difficulty arises from
the large number of rank orderings: many metrics are neither continuous or
differentiable.

To optimize the policies, we compose a linear combination of all metrics (e.g.
NDCG, incentives, gini, etc) into a final Fitness Function, which our proposed
optimizer tries to maximize.

(9) $F(\pi)=\frac{\sum_{i=1}^{|S|}W_{i}\cdot
S_{i}(\pi)}{\sum_{i=1}^{|S|}W_{i}}$

Below we describe a greedy, group-level policy optimizing either a static or
stochastic value function followed by a proposed optimizer to learn the
policies.

###  4.1. Greedy Algorithm

As has been shown through numerous previous works(Santos et al., 2015), the
assumption of independence during prediction is violated when considering
group-level diversity. Consequently, most work on diversity utilize a
heuristic, second pass algorithm to select subsequent documents during the
ranking process. However, Zhai et al. showed that a simple greedy algorithm
performed well in maximizing MMR(Zhai et al., 2015)1:

1 Input:  parameters $\sigma$, value function $v$, documents $D$

2 Output:  $\langle d_{1},d_{2},...,d_{k}\rangle$

3 for  _i = 1, 2, 3, …, K_ do

4 $d_{i}=\underset{d\in
D}{\operatorname{arg}\,\operatorname{max}}\;v(\sigma,d_{i},\langle
d_{1},d_{2},...,d_{i-1}\rangle)$

5 $D=D-\\{d_{i}\\}$

6 end for

Algorithm 1 Greedy Algorithm

####  4.1.1. Static Value Functions

While the greedy policy classically focuses on a heuristic document similarity
as the selection criteria for the value function, we instead learn a
parameterization over the greedy algorithm and utilize a simple average over
the features of previously selected documents to represent aggregate state.

We define a static value function as:

(10) $\begin{split}s_{1}=0,s_{i}&=\frac{\sum_{j=1}^{i-1}Feats(d_{i})}{i-1}\\\
v(\sigma,d_{i},\langle
d_{1},..,d_{i-1}\rangle)&=\Phi(\sigma,s_{i-1}-Feats(d_{i}))\end{split}$

where $\Phi$ is a fully connected neural network.

####  4.1.2. Stochastic Value Functions

In this section we introduce a stochastic value function, SVF for short.
Queries are assumed independent much the same way that the PRP assumed
documents can be arranged independently of each other. However, it’s easy to
see how this assumption is violated.

Revisiting the previous example of new sellers and power sellers, we present a
set of rankings from two policies $\pi_{1}$ and $\pi_{2}$:

$\displaystyle Q1_{\pi_{1}}$
$\displaystyle=\\{P,N,P,N\\},Q2_{\pi_{1}}=\\{P,N,P,N\\}$ $\displaystyle
Q1_{\pi_{2}}$ $\displaystyle=\\{N,P,N,P\\},Q2_{\pi_{2}}=\\{N,P,N,P\\}$

It is clear that the group level diversity metrics for each policy are optimal
given these two seller categories, but also equally clear that the diversity
of sellers occupying the first position is poor. In expectation, one can also
see how blending the two policies would result in the highest overall reward:

$\pi=\underset{p\in\\{\pi_{1},\pi_{2}\\}}{\operatorname{arg}\,\operatorname{max}}\;\operatorname{Uniform}(0,1))$

Inspired by the observation that neural networks can be viewed as an
exponential set of sub-networks (Srivastava et al., 2014) and the above
observation, we introduce a simple stochastic feature into the network to
allow for the blending of learned sub-policies:

(11) $\begin{split}f&\sim\operatorname{Uniform}(0,1)\\\ v(\sigma,d_{i},\langle
d_{1},..,d_{i-1}\rangle)&=\Phi(\sigma,(s_{i-1}-Feats(d_{i}))\oplus
f)\end{split}$

Rather than break apart $\theta$ into explicit policy sets, we rely on feature
masking in 2 to produce thinned, sub-networks for optimization, deferring to
the optimizer to learn how best to incorporate the noise.

###  4.2. Evolutionary Strategies

Learning a policy that devolves into various forms of sorting is difficult;
given slight variations to the underlying parameters can result in large
swings in scores. To solve this challenge, we reach for recent work using ES
from the LTR and Reinforcement Learning space to maximize our desired
objectives. Below we provide a high level description of the $(1+\lambda)$
variety of ES.

Given a parameter set $\theta$ (henceforth known as the parent), a fitness
function $f$, and shaping function $H$, we sample $\lambda$ search gradients
from the Normal distribution:
$i\in\\{1,2,..,\lambda\\},\epsilon_{i}\sim\mathcal{N}(\mu,I)$ where $\mu=0$
and $I=1$ are typical parameters for the noise distribution; while both
Salimans and Chrabaszcz explored adjusting $I$, neither found it significantly
changed the results. We further augment the algorithm by masking parameters
with probability $({p})$, reducing the effective search space per pass
(Ibrahim and Landa-Silva, 2018). For each search gradient, we compute its
fitness with respect to the parent
$\mathrm{Fitness}_{i}=f(\theta_{\mathrm{parent}}+\epsilon_{i})$. We proceed to
run all scores through a shaping function which scales each gradient by some
rank function to smooth out the impact of outlier fitness scores:
$\epsilon^{\prime}_{i}=\epsilon_{i}*H(\mathrm{Fitness}_{i},\mathrm{Fitness}_{*})$.
We compute our candidate parent as the sum of gradients scaled by $\sigma$ and
compare it to the previous parent, replacing the parent if the candidate
improves.

$\displaystyle P$
$\displaystyle=\\{\theta_{\mathrm{parent}},\theta_{\mathrm{parent}}+\sigma*\sum_{i=1}^{\lambda}\epsilon^{\prime}_{i}\\}$
$\displaystyle\theta$ $\displaystyle={\underset{p\in
P}{\operatorname{arg}\,\operatorname{max}}\;F(p)}$

There are a few variations which are commonly used. The first is always
updating $\theta_{\mathrm{parent}}=\theta_{\mathrm{candidate}}$ regardless of
improvement of fitness. The second is with respect to the shaping function:
Wierstra et al. explored the impact of fitness shaping functions in Natural
Evolutionary Strategies (Wierstra et al., 2008) and found that so long as they
were monotonic with respect to utility rank, they improved the robustness. The
final one is the number of Search gradients used during candidate
construction, which also correspond to the theoretical underpinnings: Salimans
et al. used all gradients as part of its computation due to assumptions made
in NES whereas (Chrabaszcz et al., 2018) implement a canonical variant which
only uses the best $\mu$ children.

We consolidate all of these variants into the following generalized
$(1+\lambda)-ES$ algorithm in 2.

1 Input:  $\theta_{0}$ \- parameters, F - fitness function, H - shaping
function, $\left({p}\right)$ \- mask probability, update
$\in\\{True,False\\}$, $\lambda\in\mathcal{I}^{+}$, $\mu\in\mathcal{I}^{+}$,
iters $\in\mathcal{I}^{+}$

2 for  _i ={1, 2, .., iters}_ do

3 for  _c ={1..$\lambda$}_ do

4 $\epsilon_{c}\sim\mathcal{N}(0,1)\cdot\operatorname{Bern}\left({p}\right)$

5 $s_{c}=F(\theta_{i-1}+\epsilon_{c})$

6

7 end for

8 $Sort(\epsilon_{*},s_{*})\text{ in non-increasing order }\\{s_{1}\geq
s_{2}\geq..s_{\lambda}\\}$

9 $\theta_{candidate}=\theta_{i-1}+\sum_{j=1}^{\mu}\epsilon_{j}\cdot
H(s_{j},\\{s_{1},s_{2},..,s_{\mu}\\})$

10 if  _update_ then

11 $\theta_{i}\leftarrow\theta_{candidate}$

12

13 else

14 $P=\\{\theta_{i-1},\theta_{candidate}\\}$

15 $\theta_{i}\leftarrow P_{\underset{p\in
P}{\operatorname{arg}\,\operatorname{max}}\;F(p)}$

16

17 end if

18

19 end for

Algorithm 2 Generalized $(1-\lambda)$-ES

We add a few additions on top of the Generalized ES algorithm. First we
observe that the Greedy policy complexity is $\mathcal{O}(N^{2})$, leading to
severe slowdown on large document sets during training. To mitigate this, we
uniformly sub-sample the document set each pass, for each query. While left as
a hyper parameter, we found setting it to twice the K value used to compute
NDCG sufficient for convergence. Secondly, we add batching instead of
optimizing the entire dataset at once. Finally, we utilize the the masking
strategy as in ES-Rank(Ibrahim and Landa-Silva, 2018) except that we sample
from the Bernoulli distribution with some probability $\left({p}\right)$.

##  5\. Experiments

In this section, we firstly discuss the data we use in experiments in $\S$5.1
and then we discuss our baselines and chosen implementation details in $\S$5.2
and finally we analyze results in $\S$5.3.

###  5.1. Data

Figure 2. Population vs Price

Our training set is extracted from production search logs, containing the top
5,000 queries observed over an eight day period. Document features are a
combination of query relevancy, historical performance, and taxonomic
information. We further augment the dataset with a variety of metadata:

Population, Wealth, Observation: Seller population scores are based on GMV
vigintiles: Etsy, like most e-commerce sites, exhibits a heavy power law
distribution with respect to dollar shares. We compute the observation model
by summing the number of purchases over a week at each rank position and
dividing by the total number of purchases observed. Wealth is calculated by
summing the total number of purchases per query over the same week as the
observation model.

Diversity: Listings taxonomy is used as the source of diversity in queries.
Overall, it has 175 different categories with a large class imbalance.

Incentives: We binarize our product prices by the mean listing value in our
sampled search results. Price has a sharp skew (3) toward lower cost items,
often times washing out higher quality, more labor intensive products. We add
an incentive toward premium prices to boost high quality listings closer to
the top of the rankings.

Gini vs Price: As can be seen in figure (2), listing price and population
counts are inversely correlated, illustrating a common problem faced by
e-commerce: improving the Gini Index naturally results in a reduction in
listing price.

Figure 3. Price Distribution

###  5.2. Algorithms

Table 2. Weight Variants Variant | Relevance | Group Diversity | Gini Index | Incentive
---|---|---|---|---
-{0,0} | 1.00 | 0.00 | 0.00 | 0.00
-{0.05,0.05} | 0.85 | 0.05 | 0.05 | 0.05
-{0.1,0.1} | 0.70 | 0.10 | 0.10 | 0.10
-{0.17,0.17} | 0.49 | 0.17 | 0.17 | 0.17
-{0.25,0.25} | 0.25 | 0.25 | 0.25 | 0.25
-{0.3,0.3} | 0.10 | 0.30 | 0.30 | 0.30
-{0.05,0} | 0.90 | 0.00 | 0.05 | 0.05
-{0.1,0} | 0.80 | 0.00 | 0.10 | 0.10
-{0.25,0} | 0.50 | 0.00 | 0.25 | 0.25
-{0.33,0} | 0.33 | 0.00 | 0.33 | 0.33
-{0.4,0} | 0.20 | 0.00 | 0.40 | 0.40

Baselines: We compare two different baselines to affirm efficacy of our
approach. We first look at the venerable Maximal Marginal Relevance (Carbonell
and Goldstein, 1998) where the relevance model is learned using LambdaMART
(Burges, 2010) optimized for NDCG@10. Document similarity is based on the
Jaccard of each document’s taxonomy. We tune the blend parameter $\lambda$ by
maximizing the weighted sum of scores between all indicators. The second model
is optimized using the same generalized ES algorithm above, however we replace
the greedy policy with a standard pointwise inference policy, sorting
documents based on their learned scores.

Evolutionary Strategies: In our experiments we compare ES policies trained
against a variety of different metrics:

  * •

Relevancy scores measured as NDCG@10 (1)

  * •

Groupwise diversity using ERR-IA@10 (2) across different taxonomic groups

  * •

Market indicators: $\mathrm{Gini}_{1}$ and $\mathrm{Incentives}_{1}$ (6)

We combine these into our fitness function, $F$, via a weighted linear
combination in EQ 9 with weights described in Table 2. We explore two
different policies: a standard point-wise baseline and a greedy policy (1). We
use the generalized form of Canonical Evolutionary Strategies, fixing
$\lambda=768$ and $\mu=50$ for all tests. We set update to True and fix
$\left({p}\right)$ at 0.05. We compare three different variations of our
proposed models: Greedy with a static value function, Greedy with a stochastic
value function, and Pointwise with a stochastic value function. For $\Phi$, we
use a small fully connected neural net (20 $\rightarrow$ 20 $\rightarrow$ 1)
utilizing the ReLU(Jarrett et al., 2009) non-linearity. When utilizing
stochastic value functions, we evaluate the test set 5 times with different
random seeds to determine expected performance.

###  5.3. Experimental Analysis

Figure 4. Lorenz Curve (SG-ES)

To verify efficacy of our approach, we examine a few different axes in
fitness. To tease apart how different weights impact the overall model, we
first examine how market indicator constraints impact relevancy across the
different policies. We follow it up by examining to what degree relevancy,
group-level, and indicators can jointly optimize all metrics simultaneously.
We finally compare stochastic vs static policies and to what degree it
improves the model.

Figure 5. Gini - SG-ES vs. SP-ES

Comparison to Baselines: Unsurprisingly, we see in Table 3 that while MMR does
best with respect to ERR-IA and NDCG, it is at the expense of both Incentives
and Gini index which it is unable to optimize. Similarly, while the Pointwise-
ES baseline does well on Gini and Incentives, it performs worse of all the
policies on group level diversity, which is not surprising due to document
independence assumptions. Importantly, all ES policies were able to optimize
all metrics compared to the baseline, providing evidence of its efficacy.

Table 3. Policy Comparison of Baselines Across All Metrics Variant (2) | Metric | Validation | Test Mean | Test Std
---|---|---|---|---
MMR-LambdaMART | ERR-IA | 0.487 | 0.480 | -
P-ES-{0.17,0.17} | ERR-IA | 0.481 | 0.467 | -
SP-ES-{0.17,0.17} | ERR-IA | 0.484 | 0.468 | 0.001
G-ES-{0.17,0.17} | ERR-IA | 0.489 | 0.475 | -
SG-ES-{0.17,0.17} | ERR-IA | 0.478 | 0.471 | 0.002
MMR-LambdaMART | Gini | 0.795 | 0.800 | -
P-ES-{0.17,0.17}* | Gini | 0.925 | 0.891 | -
SP-ES-{0.17,0.17}* | Gini | 0.894 | 0.888 | 0.029
G-ES-{0.17,0.17}* | Gini | 0.883 | 0.881 | -
SG-ES-{0.17,0.17}* | Gini | 0.911 | 0.889 | 0.011
MMR-LambdaMART | Incentive | 0.396 | 0.403 | -
P-ES-{0.17,0.17}* | Incentive | 0.466 | 0.525 | -
SP-ES-{0.17,0.17}* | Incentive | 0.466 | 0.518 | 0.002
G-ES-{0.17,0.17}* | Incentive | 0.470 | 0.543 | -
SG-ES-{0.17,0.17}* | Incentive | 0.466 | 0.543 | 0.009
MMR-LambdaMART | NDCG | 0.692 | 0.679 | -
P-ES-{0.17,0.17} | NDCG | 0.655 | 0.637 | -
SP-ES-{0.17,0.17} | NDCG | 0.652 | 0.634 | 0.001
G-ES-{0.17,0.17} | NDCG | 0.662 | 0.651 | -
SG-ES-{0.17,0.17} | NDCG | 0.652 | 0.642 | 0.001
* indicates stat. sig. compared to MMR (P ¡ 0.005). |

Influence on Market Indicators: We evaluated variants by adjusting the
importance weight for the Gini Index and Incentives in 5. Compared to the
baselines, we were able to progressively and smoothly improve both the Gini
Index and Incentive indicators with both stochastic and static Greedy
variants. We find that weight does indeed improve the overall equality of the
system compared to its unconstrained form 4.

Table 4. Gini Index: Stochastic vs. Static (Variant ES-{0.4,0}) Variant | Validation | Test Mean | Test Std
---|---|---|---
G-ES-{0.4,0} | 0.906 | 0.798 | 0.000
SG-ES-{0.4,0} | 0.922 | 0.903 | 0.008

Figure 6. Fitness scores from each iteration - SG-ES-{0.0,0.0} (top) and SG-
ES-{0.17,0.17} (bottom). The horizontal dashed lines display the baseline MMR-
LambdaMART scores for each metric matched by color.

Joint Optimization of all Metrics: We found we were able to optimize multiple
metrics simultaneously as seen in Figure 6. Despite the conflicting nature of
the metrics, ES was able to find policies that improved the underlying metrics
compared to the baselines.

Stochastic Features: Stochastic value functions were competitive with their
static brethren. We find that the additional noise had a few benefits: first,
it helped regularize the networks; we find the difference between train and
test scores were narrower than the static variants. Furthermore, the
stochastic variants were smoother: they had lower variance as importance
weighting increased. Table 4 exemplifies the differences when considering only
market indicators and relevance. Table 5 shows how the stochastic policy is
more reliable with generalization from validation to the test set. Unlike the
greedy variant, SVFs applied to pointwise models were unable to stably improve
market indicators. Figure 5 compares stochastic pointwise and greedy
algorithms on different importance weightings - while greedy is fairly smooth,
the pointwise model displays high variance both within policy and across
different weightings.

Table 5. Gini Index: Stochastic vs. Static Variant (2) | Validation | Test Mean | Difference
---|---|---|---
G-ES-{0,0} | 0.740 | 0.849 | -0.109
SG-ES-{0,0} | 0.734 | 0.789 | -0.065
G-ES-{0,0.05} | 0.815 | 0.883 | -0.068
SG-ES-{0,0.05} | 0.854 | 0.853 | 0.001
G-ES-{0,0.1} | 0.825 | 0.899 | -0.074
SG-ES-{0,0.1} | 0.881 | 0.861 | 0.020
G-ES-{0,0.17} | 0.883 | 0.881 | 0.002
SG-ES-{0,0.17} | 0.911 | 0.889 | 0.012
G-ES-{0,0.25} | 0.924 | 0.861 | 0.063
SG-ES-{0,0.25} | 0.901 | 0.879 | 0.022
G-ES-{0,0.3} | 0.945 | 0.876 | 0.069
SG-ES-{0,0.3} | 0.925 | 0.874 | 0.051

##  6\. Conclusion and Future Work

In this paper we defined types of market indicators critical for creating
healthy, two-sided marketplaces and proposed strategies for learning policies
to jointly maximize those desired market characteristics. We showed that we
can influence these market-level metrics via our models, resulting in a method
for imposing business needs while eliminating many of the common forms of
interventions that lead to sub-par search experiences.

There are many possible directions for future work. One is to explore
stochastic models in online environments versus just offline demonstrated in
this paper. Another possible direction is to explore more efficient algorithms
as ES comes at significant cost of sample efficiency, often using orders of
magnitude more compute than other, more efficient policy gradient optimizers.
The last one is to see how these ideas can be utilized in a production system.

## References

  * (1)
  * Agrawal et al. (2009) Rakesh Agrawal, Sreenivas Gollapudi, Alan Halverson, and Samuel Ieong. 2009.  Diversifying search results. In _Proceedings of the second ACM international conference on web search and data mining_. ACM, 5–14.
  * Berbeglia and Van Hentenryck (2017) Franco Berbeglia and Pascal Van Hentenryck. 2017.  Taming the Matthew Effect in Online Markets with Social Influence.. In _AAAI_. 10–16.
  * Burges (2010) Christopher JC Burges. 2010\.  From ranknet to lambdarank to lambdamart: An overview.  _Learning_ 11, 23-581 (2010), 81.
  * Carbonell and Goldstein (1998) Jaime Carbonell and Jade Goldstein. 1998.  The use of MMR, diversity-based reranking for reordering documents and producing summaries. In _Proceedings of the 21st annual international ACM SIGIR conference on Research and development in information retrieval_. ACM, 335–336.
  * Chapelle et al. (2011) Olivier Chapelle, Shihao Ji, Ciya Liao, Emre Velipasaoglu, Larry Lai, and Su-Lin Wu. 2011\.  Intent-based diversification of web search results: metrics and algorithms.  _Information Retrieval_ 14, 6 (2011), 572–592.
  * Chapelle et al. (2009) Olivier Chapelle, Donald Metlzer, Ya Zhang, and Pierre Grinspan. 2009.  Expected reciprocal rank for graded relevance. In _Proceedings of the 18th ACM conference on Information and knowledge management_. ACM, 621–630.
  * Chrabaszcz et al. (2018) Patryk Chrabaszcz, Ilya Loshchilov, and Frank Hutter. 2018\.  Back to Basics: Benchmarking Canonical Evolution Strategies for Playing Atari.  _arXiv preprint arXiv:1802.08842_ (2018).
  * Clarke et al. (2008) Charles LA Clarke, Maheedhar Kolla, Gordon V Cormack, Olga Vechtomova, Azin Ashkan, Stefan Büttcher, and Ian MacKinnon. 2008.  Novelty and diversity in information retrieval evaluation. In _Proceedings of the 31st annual international ACM SIGIR conference on Research and development in information retrieval_. ACM, 659–666.
  * Cooper (1971) W.S. Cooper. 1971\.  The inadequacy of probability of usefulness as a ranking criterion for retrieval system output.  (1971).
  * Craswell et al. (2008) Nick Craswell, Onno Zoeter, Michael Taylor, and Bill Ramsey. 2008.  An experimental comparison of click position-bias models. In _Proceedings of the 2008 international conference on web search and data mining_. ACM, 87–94.
  * Dang and Croft (2012) Van Dang and W Bruce Croft. 2012.  Diversity by proportionality: an election-based approach to search result diversification. In _Proceedings of the 35th international ACM SIGIR conference on Research and development in information retrieval_. ACM, 65–74.
  * Gini (1912) Corrado Gini. 1912\.  Italian: Variabilità e Mutabilità (Variability and Mutability).  _Cuppini, Bologna_ (1912).
  * Hu et al. (2018) Yujing Hu, Qing Da, Anxiang Zeng, Yang Yu, and Yinghui Xu. 2018.  Reinforcement Learning to Rank in E-Commerce Search Engine: Formalization, Analysis, and Application.  _arXiv preprint arXiv:1803.00710_ (2018).
  * Ibrahim and Landa-Silva (2018) Osman Ali Sadek Ibrahim and Dario Landa-Silva. 2018.  An evolutionary strategy with machine learning for learning to rank in information retrieval.  _Soft Computing_ 22, 10 (2018), 3171–3185.
  * Jarrett et al. (2009) Kevin Jarrett, Koray Kavukcuoglu, Yann LeCun, et al. 2009\.  What is the best multi-stage architecture for object recognition?. In _Computer Vision, 2009 IEEE 12th International Conference on_. IEEE, 2146–2153.
  * Järvelin and Kekäläinen (2000) Kalervo Järvelin and Jaana Kekäläinen. 2000.  IR evaluation methods for retrieving highly relevant documents. In _Proceedings of the 23rd annual international ACM SIGIR conference on Research and development in information retrieval_. ACM, 41–48.
  * Radlinski et al. (2008) Filip Radlinski, Robert Kleinberg, and Thorsten Joachims. 2008\.  Learning diverse rankings with multi-armed bandits. In _Proceedings of the 25th international conference on Machine learning_. ACM, 784–791.
  * Raman et al. (2012) Karthik Raman, Pannaga Shivaswamy, and Thorsten Joachims. 2012\.  Learning to diversify from implicit feedback. In _WSDM Workshop on Diversity in Document Retrieval_.
  * Robertson (1977) Stephen E Robertson. 1977\.  The probability ranking principle in IR.  _Journal of documentation_ 33, 4 (1977), 294–304.
  * Salimans et al. (2017) Tim Salimans, Jonathan Ho, Xi Chen, Szymon Sidor, and Ilya Sutskever. 2017.  Evolution strategies as a scalable alternative to reinforcement learning.  _arXiv preprint arXiv:1703.03864_ (2017).
  * Santos et al. (2015) Rodrygo LT Santos, Craig Macdonald, Iadh Ounis, et al. 2015\.  Search result diversification.  _Foundations and Trends® in Information Retrieval_ 9, 1 (2015), 1–90.
  * Santos et al. (2010) Rodrygo LT Santos, Jie Peng, Craig Macdonald, and Iadh Ounis. 2010.  Explicit search result diversification through sub-queries. In _European Conference on Information Retrieval_. Springer, 87–99.
  * Srivastava et al. (2014) Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. 2014.  Dropout: a simple way to prevent neural networks from overfitting.  _The Journal of Machine Learning Research_ 15, 1 (2014), 1929–1958.
  * Svore et al. (2011) Krysta M Svore, Maksims N Volkovs, and Christopher JC Burges. 2011\.  Learning to rank with multiple objective functions. In _Proceedings of the 20th international conference on World wide web_. ACM, 367–376.
  * Van Hentenryck et al. (2016) Pascal Van Hentenryck, Andrés Abeliuk, Franco Berbeglia, Felipe Maldonado, and Gerardo Berbeglia. 2016.  Aligning Popularity and Quality in Online Cultural Markets.. In _ICWSM_. 398–407.
  * Wierstra et al. (2008) Daan Wierstra, Tom Schaul, Jan Peters, and Juergen Schmidhuber. 2008.  Natural evolution strategies. In _Evolutionary Computation, 2008. CEC 2008.(IEEE World Congress on Computational Intelligence). IEEE Congress on_. IEEE, 3381–3387.
  * Xia et al. (2015) Long Xia, Jun Xu, Yanyan Lan, Jiafeng Guo, and Xueqi Cheng. 2015.  Learning maximal marginal relevance model via directly optimizing diversity evaluation measures. In _Proceedings of the 38th international ACM SIGIR conference on research and development in information retrieval_. ACM, 113–122.
  * Zhai et al. (2015) ChengXiang Zhai, William W Cohen, and John Lafferty. 2015\.  Beyond independent relevance: methods and evaluation metrics for subtopic retrieval. In _ACM SIGIR Forum_ , Vol. 49. ACM, 2–9.