# Economic Recommender Systems – A Systematic Review

Alvise De Biasio<EMAIL_ADDRESS>Department of Mathematics,
University of Padova, Via Trieste 63, Padova, 35131 PD, Italy  R&D Department,
estilos srl, Via Ca’ Marcello 67/D, Venezia, 30172 VE, Italy  Nicolò Navarin
<EMAIL_ADDRESS>Dietmar Jannach<EMAIL_ADDRESS>Department of
AI & Cybersecurity, University of Klagenfurt, Universitätsstraße 65–67, 9020
Klagenfurt, Austria

###### Abstract

Many of today’s online services provide personalized recommendations to their
users. Such recommendations are typically designed to serve certain user
needs, e.g., to quickly find relevant content in situations of information
overload. Correspondingly, the academic literature in the field largely
focuses on the value of recommender systems for the end user. In this context,
one underlying assumption is that the improved service that is achieved
through the recommendations will in turn positively impact the organization’s
goals, e.g., in the form of higher customer retention or loyalty. However, in
reality, recommender systems can be used to target organizational economic
goals _more directly_ by incorporating monetary considerations such as price
awareness and profitability aspects into the underlying recommendation models.
In this work, we survey the existing literature on what we call _Economic
Recommender Systems_ based on a systematic review approach that helped us
identify 133 relevant papers. We first categorize existing works along
different dimensions and then review the most important technical approaches
from the literature. Furthermore, we discuss common methodologies to evaluate
such systems and finally outline the limitations of today’s research and
future directions.

###### keywords:

Recommendations , Business Value , Price and Profit , Multistakeholder ,
Survey

##  1 Introduction

Recommender Systems (RSs) [147] have become an integral part of many modern
online services, for example, on Amazon, Netflix, YouTube, or Spotify.
Typically, the recommendations provided by the system are designed to serve
certain user needs. On the mentioned e-commerce and media streaming sites, for
example, these systems support users in navigating large information spaces,
thereby helping them discover relevant content that they were previously not
aware of.

The academic literature on recommender systems has traditionally focused on
the different types of value that such systems create for _users_ , in
particular by proposing increasingly sophisticated machine learning models to
predict which items are relevant for them in a given situation. An underlying
assumption of this user-centric perspective is that by creating value for
consumers through personalized recommendations, providers expect certain
benefits for the organization as well, for example, through increased customer
engagement, loyalty, and retention [76, 92, 127, 106].

Only during the last few years, researchers increasingly emphasize the fact
that in practical applications of recommender systems, the interests of
multiple stakeholders have to be _explicitly_ taken into account.
Correspondingly, the underlying systems have to be designed to create value
both for consumers, recommendation providers, and maybe even further
stakeholders [1, 140, 146].

In practice, the business value a recommender system creates for providers is
measured through various key performance indicators (KPIs), see [70, 144].
Besides the mentioned indirect effects of recommendations on customer
engagement and retention, organizations rely on various forms of conversion
rates to gauge the effectiveness of a system. In many cases, firms directly
assess the impact of recommendations by analyzing the effects on sales
numbers.

Therefore, it becomes desirable for companies to incorporate relevant
knowledge into the underlying algorithms so that the resulting recommendations
can drive these KPIs more directly in the desired direction. One important
domain-independent approach in this context is to consider purchase-related
information in the recommendation models, in particular regarding the profit
that results from individual purchase transactions, see [141]. Moreover, such
algorithms may implement several other theories and mechanisms from the
economics and marketing literature, considering, for example, the role of
promotions and discounts, price sensitivity, or consumer utility.

These approaches, which we call _Economic Recommender Systems_ (_ECRSs_), are
highly relevant in practice. Unfortunately, the literature on this topic is
largely scattered. With this paper, we provide a systematic review of the
field, which should serve researchers and practitioners alike as a starting
point to understand the state-of-the-art in the area. Our systematic
literature search surfaced more than one hundred relevant papers, which we
categorize into five main dimensions of analysis, see Section 3. In the main
part of the survey, Section 4, we then discuss existing ECRSs technical
approaches. Afterward, we analyze existing methodologies to evaluate such
systems in Section 5. The paper ends with a discussion of open challenges.

##  2 Background and Related Work

In this section, we first provide more background on the business value of
recommendations. We then characterize the concept of economic recommender
systems in more depth. Finally we discuss the relationships of ECRSs to
neighboring topics in recommender systems research.

###  2.1 Business Value of Recommender Systems

Recommender systems, as mentioned above, are often designed to serve both user
[234, 40, 55, 124, 136] and organizational purposes [140, 44, 107, 192].
Regarding the organizational purposes, there are various ways in which an RS
can generate value for a business [9, 19, 106, 18, 231, 33], considering
economics and marketing aspects [245, 118, 206, 168, 224].

In the literature, a number of general categories were identified to
characterize how RSs may create business value and how the business value can
be measured [70, 144]. The typical measures and corresponding KPIs include
[144]:

  * •

The number of user clicks on the recommendations, often measured by the click-
through rate (CTR);

  * •

The degree of user adoption of the system, often measured by the conversion
rate (CVR);

  * •

The overall revenue generated from the sales of the firm’s products and
services;

  * •

The possible effects on the sales distribution of the items sold, e.g.,
shifting toward more profitable items;

  * •

The overall degree of user engagement with the platform, as indicator of
customer satisfaction.

Depending on the business industry (e.g., retail, entertainment,
manufacturing) or the revenue model (e.g., transaction-based, advertising,
subscription) [227, 221, 123, 197, 174, 54, 126] the company may want to
optimize certain business values rather than others. For example, in the case
where the revenue model is primarily transaction-based (e.g., Walmart), since
there is a direct link between purchases and revenue, the company might be
interested in shifting the customer behavior towards the purchase of the more
profitable items [210]. In contrast, in case the organization’s revenue model
is based on ads (e.g., YouTube), the company may be interested in increasing
the number of clicks [69] as this is directly related to the consumption of
ads that providers pay to see their brand advertised. Finally, a company might
also be interested in optimizing user engagement [106] in the case of
subscription-based models (e.g., Netflix) as this positively correlates with
retention.

###  2.2 Economic Recommender Systems

While there are various ways in which an RS can create value for users and
providers, and while there are several KPIs that firms might seek to optimize,
ultimately, the provision of a recommendation service almost always serves
some economic goal of the organization such as profit and growth. However, we
note that some forms of value creation are more directly targeting
profitability aspects than others. An increase in revenue through
recommendations or a shift in the sales distribution toward the most lucrative
items is almost directly reflected in a profit improvement [130, 52, 216]. On
the other hand, a growth in user engagement, as in the case of Netflix [106],
with more customers joining and fewer leaving, is sometimes only indirectly
reflected in higher long-term profits for the organization.

In this survey, we focus on the first type of the described recommendation
approaches, i.e., those that target economic effects in a more direct way.
Typical examples in this context are: RSs that consider company profit and
customer relevance in a balanced way [201, 62, 43, 52, 162]; systems that
leverage discounts and pricing algorithms to trigger purchases [149, 5, 289,
145, 151]; or methods that consider customers’ price sensitivity to recommend
items more in line with their price preferences [97, 51, 295, 294, 283]. We
call such systems economic recommender systems, and we informally characterize
them as follows:

> _An Economic Recommender System (ECRS) is an RS that exploits price and
> profit information and related concepts from marketing and economics to
> directly optimize an organization’s profitability._

Later in this work (Section 3), we identify five key approaches from the
literature to build ECRSs, which we divide into customer and organization-
oriented ones, depending on the focus of the underlying algorithms. Customer-
oriented approaches in the literature, for instance, integrate purchasing
behavior mechanisms (e.g., price sensitivity) into the models to generate more
relevant recommendations that will automatically lead to more profit.
Organization-oriented ones, on the other hand, apply particular organizational
strategies (e.g., profit awareness, promotional pricing) to optimize profit.

Since most recommender systems may at least indirectly target some profit-
related or growth-related goal, the boundaries between an economic RS and a
“traditional” one may sometimes appear blurry. However, a clear distinction
can often be made depending on the underlying revenue model [54, 227, 126].
For example, click-through rate maximization may be seen as an indirect method
for profit optimization in case it is only about increasing site interactions
[115, 267]. However, it may also be considered as an ECRS method in case there
is some revenue associated with each click event (e.g., commissions suppliers
pay to marketplaces for each generated impression), as in the case of an
advertising revenue model [189, 244, 288].

Concluding our characterization of ECRSs, it is important to note that
considering certain types of economic information to an inappropriate extent
may also lead to _unintended negative effects_ and _behavioral harms_ of
recommendations [8, 121, 79]. Specifically, it is vital to ensure that an ECRS
does not negatively impact the user’s trust [178] in the organization [130,
101, 210, 29]. Indeed, trust is one of the most important factors driving
adoption [34, 161] and purchase intention [205, 214]. Recommendations that are
irrelevant [48, 255, 281, 202], manipulative [171, 6, 8, 65, 111, 269, 247],
or poorly explainable [262, 64, 280] because they are too biased towards the
profitable items [263] can harm trust, leading customers to reactance [86,
274] or churning.

Besides trust, there are also other possible harms that may emerge in case the
recommendation strategy is oriented too strongly toward profit. While
algorithms are often designed to improve sales diversity [170, 10] or to
stimulate the sales or consumption of niche items [194, 275], they in practice
might sometimes nudge users to buy the most popular ones [89, 87, 172, 129,
173, 88]. Such effects may in turn have profit implications considering that
popular items sometimes have lower margins [93]. Finally, competition effects
[102, 175, 299] may also be important to consider, since rewarding higher-
margin items could push sellers to increase prices [298], thus impacting
customers’ willingness-to-pay [7], and market demand [282, 20].

###  2.3 Related Areas in RSs Research

Economic recommender systems are related to other important research areas,
including the following:

  * •

Multi-Stakeholder Recommender Systems [1, 2]: where the system is designed to
meet the interests of multiple stakeholders (e.g., consumers, providers,
suppliers);

  * •

Multi-Objective Recommender Systems [297, 15]: where the system is designed to
optimize several objectives simultaneously (e.g., accuracy, diversity);

  * •

Fair Recommender Systems [71, 276, 277, 215, 219]: where the system is
designed to avoid possible discrimination against certain user or item groups.

The relationships between ECRSs and these other areas can be characterized as
follows. Regarding multi-stakeholder RSs, we note that probably any ECRS in
practice does not _exclusively_ focus on provider profitability but considers
the interests of other stakeholders—in e-commerce, in particular, those of
consumers or suppliers as well [43, 52]. Such multi-stakeholder considerations
mean that ECRSs in practice are _multi-objective_ RSs that consider different
competing objectives, e.g., profitability vs. consumer value [62, 101] or
short-term vs. long-term profits. However, not every multi-stakeholder RS
necessarily is an economic one, e.g., considering that an RS may also be
designed to recommend users to other users (e.g., on dating platforms).
Likewise, a multi-objective RS could also optimize non-economic goals, e.g.,
popularity, which may in turn have a direct inverse relationship with
profitability under certain circumstances [93]. Finally, in terms of fairness,
when building an ECRS there is always the possibility that by designing a
system too biased [49] toward profitable items [52, 200], the organization
might risk being perceived as unfair by consumers. However, there are various
other application areas of fair recommender systems, which are not related to
economic aspects or firm profitability, e.g., when the recommender system is
designed to avoid discrimination of underrepresented groups in the
recommendations.

Various surveys have been published in the mentioned areas of multi-
stakeholder [1, 2] and multi-objective [297, 15] RSs, and on related topics
such as fairness [276, 277, 215, 219], diversity [170], trust [77], and
explainability [285, 246]. We refer the readers to these important works for
in-depth coverage of the respective topics. The present survey has certain
affinities with previous reviews on value-aware [70] and price- and profit-
aware [141] RSs. It however differs from these previous works in various ways.
First, our study is the first systematic review of economic recommender
systems based on PRISMA guidelines [209]. Moreover, existing value-aware RSs
research [70] investigated how to generically optimize business value through
RSs, whereas our research on ECRSs is focused on direct optimization of
profitability. In addition, the present research also embraces customer-
oriented approaches (e.g., price-sensitive recommendations). Previous work on
price- and profit-aware RSs [141] also focused on profitability optimization.
However, this earlier research did not cover a number of important approaches
that were identified in this survey (e.g., economic utility modeling methods).
Finally, our present review also discusses methodological questions (e.g.,
performance evaluation methods) that were not addressed in previous works.

##  3 Methodology

The present study follows a systematic review process based on Preferred
Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) [209]
guidelines. The PRISMA article selection process is recognized throughout the
scientific community as a rigorous and reliable methodology. The process aims
to identify, evaluate, and interpret all available research relevant to a
particular research question, topic area, or phenomenon of interest while
ensuring high reproducibility of results. In the following, we report: the
dimensions of analysis considered in the study, the underlying research
questions, the eligibility criteria for article inclusion, the search queries
used, the overall article analysis and selection process, and the possible
limitations of the survey.

###  3.1 Decomposing Economic Recommender Systems into Different Dimensions of
Analysis

basic/.style = draw, thin, drop shadow, font=, align=center, upupper
style/.style = basic, edge+=-, line width=.4pt, fill=black!24, upper
style/.style = basic, edge+=-, line width=.4pt, fill=black!12, lower
style/.style = basic, edge+=-, line width=.4pt, fill=black!2, where
level<=2upper style, edge path’= (!u.parent anchor) – +(0,-7pt) -| (.child
anchor) , lower style, , where level<=1parent anchor=children, child
anchor=parent, if=isodd(n_children())calign=child edge, calign primary
child/.process= O+nw+nn children(#1+1)/2 , calign=edge midpoint, , folder,
grow’=0, , [Economic Recommender Systems

Dimensions of Analysis, upupper style [Customer Oriented

Approaches, folder, grow’=0, for children=lower style, before drawing tree=
tempdima/.option=!r2.max y, tempdima-/.option=max y, for tree=
y+/.register=tempdima, , [Price Sensitivity] [Economic Utility

Modeling] ] [Organization Oriented

Approaches, folder, grow’=0, for children=lower style, before drawing tree=
tempdima/.option=!r2.max y, tempdima-/.option=max y, for tree=
y+/.register=tempdima, , [Profit Awareness] [Promotional] [Long-Term Value

Sustainability] ] ]

Figure 1: Economic recommender systems dimensions of analysis.

Economic recommender systems can be characterized by several interrelated
topics. To identify relevant articles, we therefore followed an inductive
process starting from two related surveys [141, 70], decomposing ECRSs into
different dimensions of analysis (DAs). As Figure 1 shows, we identified five
types of approaches that can be divided into customer and organization-
oriented ones, depending on their main focus. Customer-oriented approaches aim
to integrate RSs models with purchasing behavior mechanisms to generate more
relevant recommendations that could in turn lead to more value for the firm.
Instead, organization-oriented ones make use of specific organizational
strategies to directly or indirectly optimize business KPIs. Below, we explain
the rationale behind each of them.

  * •

DA1: Price Sensitivity approaches aim to explicitly consider customers’ price
preferences in the recommendation process. In fact, price is one of the
variables that most strongly influence customers’ buying behavior [179, 16].
For example, customers are often willing to pay more for certain types of
items based on presumed greater utility, better aesthetics, brand prestige,
supplier reliability, or a combination of various factors [145]. By
considering customers’ price sensitivity in the algorithms [141], more
accurate and relevant recommendations could directly increase the probability
of purchase and thus lead to higher sales revenue for the organization.

  * •

DA2: Economic Utility Modeling approaches aim to explicitly consider the
utility of recommendations for the customer in accordance with an economic
perspective. There are many utilitarian dynamics [233] related to the
particular type of purchased products [258, 99]. For example, if a customer
has just purchased a computer or a smartphone, it is very likely that he or
she will not purchase the same or a similar product again within a short time.
Conversely, there are other products, such as dog food or diapers, for which
he or she is very likely to continue buying repeatedly for an extended period
of time. Generating more relevant recommendations by considering the
customer’s utilitarian behavior could increase conversion rates and generate
more profits for the firm.

  * •

DA3: Profit Awareness approaches aim to directly incorporate profit
information into the recommendation models. In fact, profit (i.e., sales
revenue minus costs) is one of the most important business KPIs for a
successful enterprise [243]. Depending on the particular level of this
indicator, a company may or may not invest in research and development to grow
the business, attract investors to finance its operations, obtain possible
financing from banks, and many other issues of strategic interest to
entrepreneurs and managers [100]. Overall, generating more profitable
recommendations by explicitly considering profit information could directly
optimize the organization’s economic goals.

  * •

DA4: Promotional approaches generate recommendations while strategically
setting the prices of certain products or focusing the customer’s attention on
certain brands or promotions. For example, the company can offer certain
products at a discounted price (individually or in bundles) to incentivize
impulsive buying behaviors [199, 104]. Similarly, the firm can make customers
aware of certain products that they would be unlikely to discover on their own
and indirectly trigger a possible purchase in the future [165]. Both
approaches can be integrated into the recommendation process to optimize
profit.

  * •

DA5: Long-Term Value Sustainability approaches aim to generate recommendations
considering a long-term economic perspective. In fact, long-term sustainable
business growth is one of the most important aspects for a company [186, 222,
190]. For example, a company may be interested in making customers
progressively purchase more and more products and services over time to
increase their customer lifetime value. Generating recommendations by
considering such long-term economic goals of the company thus has the
potential to stimulate business growth in a sustainable way over time.

ID |  Dimension |  Search Query | Scopus | IEEE | Springer | ACM | Total
---|---|---|---|---|---|---|---
DA1 |  Price Sensitivity |  ((“recommender system”) AND (“price preference” OR “price sensitivity” OR “price elasticity” OR “willingness to pay” OR “price-aware”)) | 670 | 2 | 469 | 57 | 1198
DA2 |  Economic Utility
Modeling |  ((“recommender system”) AND (“economic”) AND (“utility theory”)) | 104 | 0 | 188 | 17 | 309
DA3 |  Profit Awareness |  ((“recommender system”) AND (“multi-stakeholder” OR “profit-aware” OR “value-aware”)) | 351 | 5 | 290 | 63 | 709
DA4 |  Promotional |  ((“recommender system”) AND (“dynamic pricing” OR “price personalization” OR “product bundling”)) | 483 | 0 | 423 | 29 | 935
DA5 |  Long-Term Value
Sustainability |  ((“recommender system”) AND (“customer lifetime value” OR “RFM” OR “cumulative profit” OR “long-term value”)) | 619 | 4 | 450 | 35 | 1108
Table 1: Search queries and results divided by online database of the
different dimensions of analysis on which this article focuses. Queries were
run on May 12, 2023 looking for all documents published since January 1, 2000.
Figure 2: PRISMA flow diagram.
2002200320042005200620072008200920102011201220132014201520162017201820192020202120222023$0$$5$$10$$15$Number
of Studies Price Sensitivity Economic Utility Modeling Profit Awareness
Promotional Long-Term Value Sustainability Figure 3: Distribution of surveyed
papers per year divided by dimension of analysis.

###  3.2 Research Questions

Having identified these dimensions of analysis, the goal of our work is to
review the state-of-the-art of current ECRSs research. More specifically, the
present survey aims to answer the following research questions (RQs):

  * •

RQ1: What technical approaches are used to build ECRSs?

  * •

RQ2: What evaluation methods are used to assess the performance of an ECRS?

  * •

RQ3: What are the main challenges and future research directions in the area
of ECRSs?

###  3.3 Search Queries

As mandated by the PRISMA guidelines, our survey aims to answer previous RQs
by systematically querying online libraries. In particular, we queried
Elsevier Scopus, IEEE Xplore, Springer Link, and ACM Digital Library to
identify relevant articles. We created a search query for each of the previous
DAs by analyzing the most recurring key terms identified in a series of
specialized articles extracted from the literature of two related surveys
[141, 70]. In Table 1, we report the used search queries and the number of
identified documents.

###  3.4 Eligibility Criteria

To be included in the review, articles must pass a rigorous analysis process.
Specifically, articles must meet the following eligibility criteria (EC):

  * •

EC1: Articles must focus on research questions related to one of the dimension
of analysis of ECRSs.

  * •

EC2: Articles must explicitly mention the business KPIs included in the search
queries.

  * •

EC3: Articles must be unique, written in English, and the full content must be
accessible to the authors.

  * •

EC4: Articles must be peer-reviewed by either scientific journals or
conferences.

  * •

EC5: Graduate theses and doctoral dissertations are not included.

###  3.5 Article Selection Process

As shown in the PRISMA flow diagram in Figure 2, we followed a multi-stage
process to identify all the relevant resources included in this review. In the
first identification phase $2227$ articles from Elsevier Scopus, $11$ articles
from IEEE Xplore, $1820$ articles from Springer Link, and $201$ articles from
ACM Digital Library were identified for subsequent analyses. In this phase,
$329$ duplicated records and $39$ non-English articles were identified and
removed. In the second screening phase, the titles and abstracts of the
remaining $3891$ articles were analyzed, and $3479$ records were removed
because the covered topics were not relevant to the present review. In this
phase, $414$ articles were then sought for retrieval and assessed for
eligibility, excluding $350$ articles after full text reading. From this
subset of $64$ eligible articles, an additional $616$ articles were identified
by searching for references in their bibliographies. These articles were then
assessed for eligibility, removing $547$ records after reading the full text.
At the end of this overall process, $133$ studies were included in the review.
In Figure 3, we show some statistics of the references obtained at the end of
the analysis process by reporting the distribution by year of surveyed papers
divided by subdimension of analysis. As can be seen from the figure, there is
a growing interest in the literature for all ECRSs dimensions.

###  3.6 Study Limitations

The possible study limitations (SL) are the following:

  * •

SL1: Articles were primarily selected from Elsevier Scopus, IEEE Xplore,
Springer Link, ACM Digital Library, and from reference searches in the
bibliographies of articles that passed the screening stage. Additional online
libraries may be considered in future research.

  * •

SL2: The study does not cover preprints, non-English articles, non-accessible
articles, graduate theses, doctoral dissertations, industry products, and
demos.

  * •

SL3: Other dimensions of analysis of ECRSs beyond those identified in Section
3.1 are left for possible future extensions of this work.

Algorithmic
Approach |  Price Sensitivity |  Economic Utility
Modeling |  Profit Awareness |  Promotional |  Long-Term Value
Sustainability
---|---|---|---|---|---
In-Processing |  [283, 265, 191, 294, 57, 295, 254, 253, 109, 110, 51, 230, 97, 50] |  [191, 270, 254, 99, 290, 286, 287, 73, 258] |  [62, 78, 201, 177, 187, 180, 43, 266, 220, 134, 108, 218, 13, 58, 260, 41, 261, 259] |  [47, 239, 23, 12, 103, 254, 74, 46, 166, 27, 183, 96, 230, 193, 268, 154, 167, 94, 95] |  [139, 138, 279, 122, 278, 148, 115, 292, 303, 216, 267, 153, 244]
Post-Processing |  [162, 45, 252, 238, 114, 272, 145, 25] |  [67, 293, 223, 296, 272, 289, 4] |  [101, 235, 162, 224, 133, 189, 188, 184, 288, 272, 24, 257, 256, 68, 52, 200, 185] |  [235, 5, 158, 116, 82, 145, 72, 31, 289, 151, 301, 149, 150, 25, 26] |  [131, 130, 210, 29]
Table 2: ECRSs studies divided by dimension of analysis and algorithmic approach. Notation | Definition
---|---
$u$ | user
$i$ | item
$p_{i}$ | item’s price
$c_{i}$ | item’s cost
$v_{i}=p_{i}-c_{i}$ | item’s profit
$m$ | number of overall users
$n$ | number of overall items
$k$ | number of items to recommend
$\mathcal{U}=\\{u_{1},\dots,u_{m}\\}$ | set of users
$\mathcal{I}=\\{i_{1},\dots,i_{n}\\}$ | set of items
$\mathbf{X}\in\\{0,1\\}^{m\times n}$ | user-item interaction matrix
$x_{u,i}\in\\{0,1\\}$ | user-item feedback
$\mathbf{\Theta}$ | set of model parameters
$\mathbf{\hat{X}}(\mathbf{\Theta})$ | scoring function
$\hat{x}_{u,i}(\mathbf{\Theta})\in[0,1]$ | user-item predicted interest
$\mathcal{Y}_{u,k}$ | recommendations list
$\mathcal{T}(\mathcal{Y}_{u,k})$ | utility function
$\rho_{u,i}$ | user-item interaction utility
Table 3: Main notation.

##  4 Technical Approaches

In this section, we discuss the underlying algorithmic approaches to each of
the ECRSs dimensions of analysis introduced in Section 3.1, i.e., price
sensitivity, profit awareness, promotional, long-term value sustainability,
and economic utility modeling.

In Table 2 we report the studies identified by the present survey that propose
technical approaches, categorized by dimension of analysis and algorithmic
method. The approaches can be divided into in- and post-processing111Pre-
processing methods may also exist in industry, e.g., when a recommendation
provider wants to rule out certain unprofitable items. Our literature search,
however, did not surface such approaches. methods, depending on the time the
economic value optimization occurs. In-processing approaches aim to
incorporate economic aspects directly into the models, either by extending the
objective function of known algorithms (e.g., by introducing new variables or
regularizers) or by developing entirely new algorithms. The underlying
algorithms may be based, for example, on supervised or reinforcement learning
paradigms. Post-processing approaches, on the other hand, can be mounted on
top of any recommender and aim to transform the recommendations generated from
the baselines by applying specific heuristic economic criteria. These may
incorporate the economic value by simply re-ranking the output of the original
algorithm or by exploiting also additional learning models.

Analyzing the distribution of the studies in Table 2, we can make some
observations. In particular, it can be noted that there are several relevant
works for all the dimensions of analysis. In addition, in-processing and post-
processing methods are equally used across all dimensions. This implies that
the research field is broad and that there are various important lines of
active research. Overall, given that there is a substantial number of works in
each dimension of analysis, we are confident that our categorization scheme
properly reflects the various types of activities in this research area.

##### Notation

In the following, we introduce the main notation used in the paper, see Table
3. Formally, the vast majority of the approaches we discuss in this survey
refer to the top-k recommendation problem [225], i.e., the problem of
determining the best $k$ items to recommend to each user. All algorithms
designed to address this particular problem consider a set
$\mathcal{U}=\\{u_{1},\dots,u_{m}\\}$ of $m$ users, a set
$\mathcal{I}=\\{i_{1},\dots,i_{n}\\}$ of $n$ items, and a user-item
interaction matrix $\mathbf{X}$, where each entry $x_{u,i}$ represents the
feedback from a user $u$ towards an item $i$. With very few exceptions, the
feedback considered is almost always implicit (i.e., $x_{u,i}\in\\{0,1\\}$).
This indicates a positive or missing interaction, depending if the user
interacted with the item or not (e.g., purchased it). Generally, it is assumed
that purchased items are those that are relevant (and maybe satisfactory) for
consumers.

Algorithms are often designed [225] to learn a scoring function
$\mathbf{\hat{X}}(\mathbf{\Theta}):\mathbf{X}\xrightarrow{}\\{\hat{x}(\mathbf{\Theta})\in\mathbb{R}:0\leq\hat{x}(\mathbf{\Theta})\leq
1\\}^{m\times n}$ to predict the missing entries of $\mathbf{X}$. The scoring
function is parameterized by a set $\mathbf{\Theta}\in\mathbb{R}^{o}$ of model
parameters222For some algorithms, such as User-Based Collaborative Filtering
based on Nearest-Neighbor techniques [204], we assume
$\mathbf{\Theta}=\emptyset$ since there are no model parameters. \- where $o$
is the number of parameters. Hence, $\hat{x}_{u,i}(\mathbf{\Theta})$
represents the expected interest of the user toward an item he or she has
never interacted with.

In the general top-$k$ recommendation problem [9], an ordered list
$\mathcal{Y}_{u,k}$ of $k$ items to be recommended to each user $u$ is
determined optimizing a specific utility function
$\mathcal{T}(\mathcal{Y}_{u,k}):\mathcal{Y}_{u,k}\xrightarrow{}\mathbb{R}$.
More formally:

$\underset{\mathcal{Y}_{u,k}}{\mathrm{argmax}}\quad\mathcal{T}(\mathcal{Y}_{u,k})$
(1)

The utility function can be implemented in arbitrary ways (e.g., including
relevance, profitability, and other aspects).

Given $\rho_{u,i}$ as the utility of the user-item interaction, the vast
majority of the studies in the RSs literature operationalize the utility
function as:

$\displaystyle\mathcal{T}(\mathcal{Y}_{u,k})=\sum_{i\in\mathcal{Y}_{u,k}}\rho_{u,i}$
(2)

optimizing directly:

$\underset{\mathcal{Y}_{u,k}}{\mathrm{argmax}}\sum_{i\in\mathcal{Y}_{u,k}}\hat{x}_{u,i}(\mathbf{\Theta})$
(3)

and thus considering the utility of the potential interaction as the expected
interest, i.e., $\rho_{u,i}=\hat{x}_{u,i}(\mathbf{\Theta})$.

However, although this user-focused utilitarian conception is currently the
most widely used one in the literature, a recommendation provider may have
different goals. In the context of ECRSs, instead, the utility functions may
be operationalized considering the item’s price $p_{i}$, and profit
$v_{i}=p_{i}-c_{i}$, where $c_{i}$ is the item’s cost. For example, algorithms
belonging to the profit-aware subdomain that we discuss in Section 4.3 are
often developed to find the most profitable, yet relevant, items for the
company, and these may clearly differ from the _most_ relevant ones.

###  4.1 Price-Sensitivity Methods

Price is one of the variables that most influence customers’ buying behavior
[179, 16]. Accordingly, many studies in the literature [97, 51, 109, 294, 162]
propose algorithms to explicitly consider customers’ price sensitivity, as
more accurate and relevant recommendations (i.e., in terms of being in the
right price range) could directly increase the probability of purchase and
thus lead to higher sales revenue for the organization. Below, we give some
insights on how these methods work by discussing a set of selected articles.

####  4.1.1 In-Processing Price-Sensitivity Methods

Most of the approaches used to generate price-sensitive recommendations are
based on in-processing algorithms. The main characteristic of these algorithms
is that price sensitivity is incorporated directly into the model.

In particular, this methodology proved particularly flexible to be applied to
the well-known Matrix Factorization (MF) [163, 164] model. The original model
estimates the expected interest of the user toward a given item via the dot
product of latent factor vectors. These are traditionally learned through a
dimensionality reduction algorithm applied to the user-item interaction
matrix. Considering price-sensitive methodologies based on MF, for example,
one paper [97] proposes to incorporate cost factors333Note that here we
respect the original paper’s terminology by referring to the cost, but
actually the cost for the user is simply the item’s price. into the model’s
objective function to generate more accurate travel tour recommendations. The
experiments reported by the authors indicate that explicitly incorporating
cost factors improves the overall accuracy of the recommendations when
compared with a plain MF model. Also, extending MF, other papers in the
literature [50, 51] propose incorporating customers’ price preferences
explicitly into the objective function through the use of particular
regularizers. However, whereas previously the purpose was to enhance the
overall performance of the system, here the study is about the use of price
preferences to make recommendations in product categories that the user has
never explored (transfer learning). In particular, according to the authors,
generating recommendations for customers’ unexplored product categories can
cause significant performance drops (-$40\%$) if traditional algorithms are
used, since the learned product user preferences are difficult to transfer
from one category to another. Instead, explicitly incorporating customers’
price preferences into the objective function can help to significantly
improve (+$43\%$) performance on unexplored categories compared to state-of-
the-art baselines.

Other studies in the literature [110, 109] propose incorporating customers’
price preferences within existing _context-aware_ recommendation algorithms
[169]. According to an experimental study with real customers in the food &
beverage field [110], explicitly incorporating discount sensitivity into the
algorithms can help to significantly improve performance in a coupon
recommendation task when compared to the CAMF method [28], i.e., a context-
aware variant of matrix factorization. Specifically, in the domain of
location-based deals, the analysis shows that the most important feature for
predicting purchase probability is the discount-to-distance ratio: the higher
the discount offered by the store, the more likely the customer is to travel
longer distances to obtain it. However, as is well known in the literature,
context variables often depend on the considered business domain. In
particular, eBay.com has some unique characteristics [109]. In this multi-
seller platform, the same products are offered at various prices
simultaneously by various sellers with different reputation scores. According
to a study [109], in this business domain, incorporating customers’
willingness-to-pay (WTP), discounting, and seller reputation features into a
context-aware recommender can help to significantly improve the accuracy of
predictions, with an $84\%$ improvement over MF models.

In addition, recent studies [295, 294, 283] propose incorporating customers’
price preferences into algorithms by exploiting Graph Neural Networks (GNNs)
[91]. Specifically, in two related studies [295, 294], it is proposed to
construct a GNN-based recommender by building a heterogeneous graph consisting
of different types of nodes: customers, items, prices, and product categories.
The key idea is to propagate price influence from prices to users by
leveraging items as a bridge so that price preferences are implicitly encoded
into the embeddings. The use of price-sensitive GNNs is also exploited in the
field of session-based recommendations [283]. For all studies based on GNNs
[295, 294, 283], the models are able to generate slightly more relevant
recommendations than the baselines. However, as various authors pointed out,
it is difficult to handle heterogeneous information and model complex
relationships underlying customer buying behavior, and research still offers
many opportunities to develop better-performing models that can fully exploit
the potential of GNN-based algorithms.

####  4.1.2 Post-Processing Price-Sensitivity Methods

Ref |  Re-Ranking Method
---|---
[238] |  $\underset{\mathcal{Y}_{u,k}}{\mathrm{argmax}}\sum_{i\in\mathcal{Y}_{u,k}}{\hat{x}_{u,i}(\mathbf{\Theta})\cdot s_{u,i}(\mathbf{\Phi})}$ (4)
[252]* |  $\underset{\mathcal{Y}_{u,k}}{\mathrm{argmax}}\sum_{i\in\mathcal{Y}_{u,k}}w_{1}(\mathbf{\Psi})\cdot\hat{x}_{u,i}(\mathbf{\Theta})+w_{2}(\mathbf{\Psi})\cdot s_{u}(\mathbf{\Phi})$ (5)
[162] |  $\underset{\mathcal{Y}_{u,k}}{\mathrm{argmax}}\sum_{i\in\mathcal{Y}_{u,k}}\hat{x}_{u,i}(\mathbf{\Theta})\cdot\Biggl{(}\Bigl{(}1+\log_{10}\bigl{(}0.1+\frac{0.9\cdot p_{i}}{c_{i}}\bigl{)}\Bigl{)}^{\beta}+\\\ +\Bigl{(}1+\log_{10}\bigl{(}0.1+\frac{0.9\cdot p_{i}}{\bar{\mathbf{p}}^{u}}\bigl{)}\Bigl{)}^{\gamma}\Biggl{)}$ (6)
Table 4: Price-sensitive re-ranking methods. *The formula captures the main
essence of the described approaches.

A number of price-sensitive recommendation algorithms also make use of post-
processing methods. The latter are primarily re-ranking algorithms, which can
be applied on top of any price-agnostic recommender baseline.

In this domain, it is proposed, for example, to generate recommendations by
weighting the expected interest $\hat{x}_{u,i}(\mathbf{\Theta})$ by the price-
sensitivity $s_{u,i}(\mathbf{\Phi})$. The latter is a particular variable,
learned through a different model parameterized by $\mathbf{\Phi}$, indicating
how price-sensitive a given user is to a given item (Eq. 4) [238]. A similar
approach is also proposed in another study [252]. However, in this case, the
price-sensitivity variable $s_{u}(\mathbf{\Phi})$ depends only on the customer
and not on the item (Eq. 5). In addition, it is necessary to use another
regression model (parameterized by $\mathbf{\Psi}$) to learn how to properly
weigh (through $w_{1}(\mathbf{\Psi})$, $w_{2}(\mathbf{\Psi})$ coefficients)
the price-sensitivity with the user’s expected interest. Both studies show
that through the use of price-sensitivity algorithms, more relevant
recommendations can be obtained.

Recently, another research [162] proposes a hybrid approach (Eq. 6) combining
the price-sensitive and the profit-aware444We discuss profit-aware methods in
Section 4.3. subdomains. This approach weighs the expected interest
$\hat{x}_{u,i}(\mathbf{\Theta})$ by balancing a user price preference factor
$\frac{p_{i}}{\bar{\mathbf{p}}^{u}}$ with a profitability factor
$\frac{p_{i}}{c_{i}}$, where $\beta,\gamma\in[-1,1]$ in Eq. 6 are
regularization parameters. In particular, considering $\bar{\mathbf{p}}^{u}$
as the average user price, the first factor captures the difference between
the customer’s typical price level and the actual item’s price. The second
factor, $\frac{p_{i}}{c_{i}}=1+\frac{v_{i}}{c_{i}}$, captures how much an
item’s sale is able to repay the underlying cost and bring profit to the
organization. In this way, it becomes possible to effectively balance the
interests of customers with those of the organization because the increase in
profitability that traditionally adversely affects the relevance of
recommendations is more than offset by the increase in the latter due to the
influence of price preferences.

In Table 4, we formally characterize the three discussed price-sensitive re-
ranking methods.

###  4.2 Economic Utility Modeling Methods

Ref |  Name |  Utility Function
---|---|---
[258] |  Standard |  $\displaystyle\mathcal{T}(\mathcal{Y}_{u,k})=\sum_{i\in\mathcal{Y}_{u,k}}\rho_{u,i}$ (7)
[135] |  Multi-Attribute |  $\displaystyle\mathcal{T}(\mathcal{Y}_{u,k})=\sum_{i\in\mathcal{Y}_{u,k}}\sum_{g\in\mathcal{G}}f_{u,g}\cdot\rho_{i,g}$ (8)
[258]* |  Constant Elasticity of Substitution |  $\displaystyle\mathcal{T}(\mathcal{Y}_{u,k})=\sum_{i\in\mathcal{Y}_{u,k}}\rho_{u,i}\cdot q_{u,i}^{\xi_{i}}$ (9)
[287] |  King-Plosser-Rebelo |  $\displaystyle\mathcal{T}(\mathcal{Y}_{u,k})=\sum_{i\in\mathcal{Y}_{u,k}}\rho_{u,i}\cdot\ln(1+q_{u,i})$ (10)
[290] |  Multi-Product |  $\displaystyle\mathcal{T}(\mathcal{Y}_{u,k})=\frac{1}{|\mathcal{Y}_{u,k}|}\sum_{i,j\in\mathcal{Y}_{u,k}:i\neq j}\Bigl{(}a_{i,j}\cdot q_{u,i}^{1-b_{i,j}}+$ (11) $\displaystyle\qquad+(1-a_{i,j})\cdot q_{u,j}^{1-b_{i,j}}\Bigr{)}^{\frac{1}{1-b_{i,j}}}$ (11)
[99] |  Marginal Utility per Dollar |  $\displaystyle\mathcal{T}(\mathcal{Y}_{u,k})=\sum_{i\in\mathcal{Y}_{u,k}}\frac{\tanh{(\rho_{u,i})}\cdot r_{i,u}}{(1+q_{u,i})\cdot\sigma(p_{i})}$ (12)
Table 5: Economic utility functions from rational choice theory. *The formulas
capture the main essence of the described approaches.

In the economic literature [195], user behavior is often modeled using
utilitarian theories to construct systems that can describe and/or optimize
certain dynamics. According to the Rational Choice Theory (RCT), at each time
instant, a rational user, when faced with a set of alternatives, will choose
those with the highest utility for him or her [233]. Accordingly, many studies
in the literature [258, 99, 290] propose algorithms that explicitly consider
the customer’s utilitarian behavior to generate more useful recommendations
that can in turn increase conversion rates and profitability. Below we give
some insights on how these methods work by discussing a few selected articles
focused, respectively, on multi-attribute, repurchase, and complementary
recommendations.

In the field of RSs, many studies in the literature assume that the utility
$\rho_{u,i}$ of a product to a customer depends on his or her purchase history
[258]. Most existing RSs recommend for each user $u$ a list
$\mathcal{Y}_{u,k}$ consisting of the top-$k$ items (Eq. 3) with the highest
predicted scores $\hat{x}_{u,i}(\mathbf{\Theta})$. The list
$\mathcal{Y}_{u,k}$ is traditionally selected from a set of items with which
the user has never interacted before. Interpreting this assumption from the
perspective of economic utility theory (Eq. 7) [258], then, the utility
$\mathcal{T}(\mathcal{Y}_{u,k})$ of a recommendation $\mathcal{Y}_{u,k}$ is
nothing but the sum of the predicted scores, i.e.,
$\rho_{u,i}=\hat{x}_{u,i}(\mathbf{\Theta})$. In this case, a recommendation
$\mathcal{Y}_{u,k}$ generated by optimizing the total utility of a set of $k$
recommended items optimizes the expected user interest estimated by any
recommendation algorithm.

However, in addition to the previous utility definition, alternative
definitions are recently emerging in the literature. For example, in the field
of Multi-Criteria Recommendation Systems [14], in the presence of a set
$\mathcal{G}$ of attributes associated with items, various studies in the
literature [135, 80, 73, 232, 293] propose to generate recommendations by
exploiting the Multi-Attribute Utility Theory (MAUT) [157]. MAUT is one of the
most widely used utilitarian theories in decision making, which aims to weigh
a set of relevant variables to determine the overall utility of each
alternative. In the context of recommendations, in particular, the overall
optimized utility (Eq. 8) in this case depends on the utility $\rho_{i,g}$ of
the single attribute $g$ of item $i$, and a weight $f_{u,g}$ that each user
can provide to indicate the importance of that attribute.

Other studies focus on the problem of repeated purchase recommendations [258,
287, 290, 99]. Unlike traditional RSs, algorithms developed for this task
generate recommendations by also considering items that the user already
purchased in the past. In particular, it is observed that the repurchase cycle
of some products may follow the Law of Diminishing Marginal Utility [258,
195]. According to this theory, many products have decreasing utility for the
user as the quantity of purchased products increases (e.g., computers, cell
phones), while others, instead, are likely to be purchased frequently over
time (e.g., baby diapers, pet food). Using the standard utilitarian criterion
in Eq. 7 it is not possible to model this behavior. Indeed, in this case, the
usefulness of recommendations for the user highly depends on the quantity
$q_{u,i}$ of item $i$ purchased by him or her until a specific time. In this
context, promising results can be obtained by modeling the repurchase cycle
through the Constant Elasticity of Substitution Utility Function [248]. This
allows the decreasing marginal utility of product $i$ to be properly modeled
through a parameter $\xi_{i}\in[0,1]$ associated with item $i$ (Eq. 9). This
parameter can be learned by extending the MF objective function. In this way,
the algorithm can explicitly consider the decreasing utility of certain
products for the user and generate more relevant recommendations.

With similar methodologies, other utilitarian functions are also used in the
literature to model customer purchasing behavior [287, 290, 99]. However,
these studies focus on different objectives. For example, one study [287]
proposes three different business cases (i.e., e-commerce, P2P lending,
freelancing) that exploit the King-Plosser-Rebelo Utility Function (Eq. 10) to
optimize the Total Surplus, i.e., an indicator that considers both the
usefulness of the recommendations for the customer and the profit for the
producer. Another study [290] in contrast propose to use the Multi-Product
Utility Function (Eq. 11) in order to also consider any complementary and
substitutability relationships among the recommended products. In the
equation, the variables $a_{i,j}$ and $b_{i,j}$ are additional parameters that
the recommendation algorithm can jointly learn with the latent factors to
model the indifference curves between pairs of products, i.e., how much the
increase in one product affects the relative marginal utility of another
product. Finally, one study [99] proposes using the Marginal Utility per
Dollar Function (Eq. 12). This function considers the price $p_{i}$ of item
$i$ and a risk attitude coefficient $r_{i,u}$ to model customers’ risk-
aversion, i.e., the tendency of consumers to spend only a small portion of
their total wealth on a single purchase.

In Table 5, we formally characterize the utility criteria discussed above.

###  4.3 Profit Aware Methods

Profit is one of the most important business KPIs for a successful enterprise
[243]. Accordingly, many studies in the literature [52, 162, 108, 177, 201,
62] propose profit-aware recommendation algorithms to directly optimize the
firm’s profitability. Below we give some insights on how these methods work by
discussing a few selected articles.

####  4.3.1 In-Processing Profit-Aware Methods

Profit-aware in-processing approaches in the literature are quite
heterogeneous, scattered, and there are several parallel lines of research.
Below, we offer a brief overview of major research directions in this area.

Some early studies [218, 134, 272] exploit Association Rules [125]. According
to this particular methodology [211], recommendations are generated through a
frequentist approach based on statistical support and confidence constructs
[208]. One of Amazon’s most prominent recommenders, i.e., “customers who
bought this item also bought”, is seemingly based on association rules. In
particular, many studies in the literature [261, 259, 260, 58] propose to
generate association rules while also optimizing profitability. The main
methods incorporate profit considerations when weighting the rules [42].
However, unlike modern RSs based on collaborative and content-based filtering
algorithms, association rules [261] are not personalized, i.e., different
users do not get different recommendations. In addition, association rules may
generally face challenges when the total number of recommendable items is very
large.

Other earlier studies [41, 13, 220] propose graph-based approaches. In
particular, one research [13] focuses on social networks. The proposed
algorithm is designed to explicitly optimize the value of recommendations in
customer-product graphs. However, in the study, profit is operationalized
through non-monetary metrics. Another relatively recent approach based on
graphs [220] is developed specifically for the taxi industry. In this
particular application domain, if we assume an hourly rate, a taxi driver’s
profit depends solely on the hours billed to customers: simply put, it is
critical for a taxi driver to minimize the distance to find a customer and
maximize the distance traveled with a customer on board. The proposed
algorithm recommends pick-up points for taxi drivers in order to maximize the
profit of driving routes while balancing the potential congestion resulting
from multiple requests from different customers at the same location.

More recently, a study [43] proposes a profit-aware RS based on collaborative
filtering. The algorithm is based on an extension of the well-known neighbor
selection criterion of the user-based nearest-neighbor collaborative filtering
model [204]. The original algorithm calculates the predicted score based on a
weighted sum of similarities between users belonging to a given neighborhood.
The authors of [43] instead propose to calculate the predicted scores by
selecting the neighbors that would allow the generation of the highest value-
weighted expected interest. Although the focus of the paper is on shilling
attacks, i.e., attacks by malicious users who generate biased ratings to
influence recommendations for their interests, the subprocedure for selecting
the most valuable users can be used to generate more profitable
recommendations.

Other recent approaches [266, 180, 177] are based on Learning To Rank (LTR)
[60]. This is a well-known technique in Information Retrieval (IR) [160, 32].
IR algorithms aim to help users to find the most relevant items based on
specific search queries. In particular, one study [266] uses this methodology
in a product search application context. The proposed algorithm integrates the
price into the objective function in order to optimize the overall sales
revenue of an e-commerce. This technique is later applied [180, 177] also to
generate recommendations without the need to anchor them to an underlying
search query. For example, one study [180] proposes a multi-objective
algorithm that is able to optimize multiple objective functions simultaneously
through LTR. The algorithm is Pareto-efficient, i.e., it optimizes each
objective (e.g., CTR and GMV555We provide the definition of the most
frequently used online metrics in Table 9.) one at a time, with the constraint
that no single objective can be further improved without affecting others.

Finally, some studies [201, 62] propose using profit-aware multi-objective
evolutionary algorithms [128, 297]. One of these [201] is based on Non-
dominated Sorting Genetic Algorithm II (NSGA-II). A more recent one [62] is
based on Multi-Objective Artificial Bee Colony (MOABC). In these cases, the
optimization target is a combination of the item’s profit and the user’s
expected interest. Both algorithms obtained very promising offline results on
the overall profit improvement, although the comparison was performed
exclusively with a traditional user-based collaborative filtering algorithm
[204].

####  4.3.2 Post-Processing Profit-Aware Methods

Ref |  Re-Ranking Method
---|---
[52] |  $\underset{\mathcal{Y}_{u,k}}{\mathrm{argmax}}\sum_{i\in\mathcal{Y}_{u,k}}{\hat{x}_{u,i}(\mathbf{\Theta})\cdot v_{i}}$ (13)
[68] |  $\displaystyle\underset{\mathcal{Y}_{u,k}}{\mathrm{argmax}}\sum_{i\in\mathcal{Y}_{u,k}}{\hat{x}_{u,i}(\mathbf{\Theta})\cdot v_{i}}$ (14) $\displaystyle\text{s.t. }\text{Dice }(\hat{\mathbf{x}}_{u}(\mathbf{\Theta}),\hat{\mathbf{x}}_{u}(\mathbf{\Theta})^{\intercal}\mathbf{v})\geq\eta$ (14)
[257]* |  $\displaystyle\underset{\mathcal{Y}_{u,k}}{\mathrm{argmax}}\sum_{i\in\mathcal{Y}_{u,k}}{v_{i}}$ (15) $\displaystyle\text{s.t. }\hat{x}_{u,i}(\mathbf{\Theta})\geq\zeta,\quad p_{i}\leq\lambda_{u}$ (15)
[141]* |  $\displaystyle\underset{\mathcal{Y}_{u,k}}{\mathrm{argmax}}\sum_{i\in\mathcal{Y}_{u,k}}{\hat{x}_{u,i}(\mathbf{\Theta})\cdot v_{i}}$ (16) $\displaystyle\text{s.t. }\hat{x}_{u,i}(\mathbf{\Theta})\geq\zeta$ (16)
[101] |  $\underset{\mathcal{Y}_{u,k}}{\mathrm{argmax}}\sum_{i\in\mathcal{Y}_{u,k}}{\delta\cdot\hat{x}_{u,i}(\mathbf{\Theta})+(1-\delta)\cdot v_{i}}$ (17)
[189]* |  $\displaystyle\underset{\mathcal{Y}}{\mathrm{argmax}}\sum_{u\in\mathcal{U}}\sum_{i\in\mathcal{I}}{\delta\cdot y_{u,i}\cdot\hat{x}_{u,i}(\mathbf{\Theta})+(1-\delta)\cdot y_{u,i}\cdot d_{u,i}}$ (18) $\displaystyle\text{s.t. }\sum_{i\in\mathcal{I}}\mathbbm{1}(y_{u,i}\cdot d_{u,i}>0)\leq l\quad(\forall u\in\mathcal{U})$ (18)
Table 6: Profit-aware re-ranking methods. *The formulas capture the main
essence of the described approaches.

In the context of this survey, many profit-aware approaches rely on post-
processing re-ranking methods. As mentioned earlier, these approaches consider
the recommender baseline as a black box and generate recommendations by
exploiting a combination of certain heuristics.

All examined profit-aware approaches are based on a simple but important
assumption [141, 70]: the items most relevant to the user are often not those
of the highest business value to the organization. Consequently, prioritizing
the highest-profit items in recommendations would allow for increased business
profitability as a result of actual user purchases of those items. In one of
the earliest approaches [200, 52] it is proposed to weight the probability of
purchase (i.e., the estimated expected interest) by profitability in order to
maximize an average expected profit (Eq. 13). This approach should make it
possible to provide more profitable recommendations than those generated by
traditional RSs. Experiments in a synthetic dataset based on a subset of
groceries transactions show encouraging results: the proposed algorithm was
able to increase profitability without excessively impacting the relevance of
recommendations. However, as also reported by the authors [200, 52], the
interests of customers and the organization must be balanced appropriately. In
fact, the organization could risk losing loyal customers should they feel
dissatisfied with overly biased recommendations toward higher-value items and
decide to leave the platform.

To mitigate this drawback, and thus to avoid providing completely irrelevant
recommendations, various studies propose more or less straightforward
extensions of Eq. 13 based on constrained optimization techniques. One of the
earliest papers [68] proposes a constrained re-ranking method based on the
Dice coefficient (Eq. 14). This can help prevent the system from providing
recommendations that are too dissimilar from the original ones based on a
threshold $\eta$. However, the study is based on various simplifying
assumptions and does not provide an empirical evaluation of the approach. In
two related studies [256, 257] instead, it is proposed to maximize
profitability under customer satisfaction and budget constraints (Eq. 15),
where $\zeta$ and $\lambda_{u}$ are two thresholds used to keep the
probability of purchase and the price of items within certain ranges,
respectively. In particular, an expert system is proposed where different
optimization goals can be specified in order to optimize profitability or
balance profitability and satisfaction in order to achieve a win-win situation
for suppliers and customers. A similar variant of this approach (Eq. 16, 17)
is also proposed in two related studies [141, 101] where the short- and long-
term profit-relevance tradeoff is investigated through the use of simulations.
In Eq. 17, $\delta\in[0,1]$ is a regularization parameter.

In addition, other studies [189, 188, 288] propose algorithms to address the
problem of sponsored recommendations. In this scenario, a supplier who decides
to sponsor its products pays the platform for each user interaction. One study
[189] in particular proposes a multi-objective post-processing re-ranking
algorithm (Eq. 18). In the equation, $y_{u,i}$ is a decision variable
($y_{u,i}=1$ iff $i\in\mathcal{Y}_{u,k}$), $d_{u,i}$ is the ad revenue that
the organization gets from suppliers if the user interacts with the item, and
$l<k$ is the maximum number of sponsored items that can be included in the
recommendation list. The algorithm is designed to balance the recommendation
of high ad revenue sponsored items with the user’s interests.

In Table 6, we formally characterize the profit-aware re-ranking methods
discussed above.

###  4.4 Promotional Methods

Promotional methods [199, 198] aim to increase sales figures by promoting
products and services to the most appropriate customer segments. We identify
three main strategies in the RSs literature that can be used to optimize
profit and related business KPIs. Pricing methods, can be used to offer
products at a discounted price or to strategically adjust prices in order to
increase the market demand. Bundling methods, are special pricing methods that
are applied to product bundles. Brand-awareness methods, finally, can be used
to focus customers’ attention on the organization’s products in order to
generate extra sales. Below we give some insights on how these methods work by
discussing a few selected articles in each category.

####  4.4.1 Pricing Methods

As discussed earlier in Section 4.1, price is one of the most influential
variables of customer buying behavior, and considering this variable
explicitly would allow for recommendations more in line with customers’
interests. However, while the previous section focuses on customer-oriented
methodologies that integrate price sensitivity as additional information in
order to generate more relevant recommendations, in this section we instead
discuss promotional techniques that an organization might want to apply to
incentivize the purchase of certain products by strategically setting the
prices [36, 103]. In the following, we describe two organizational strategies
referring to: (a) occasional discounting; (b) personalized dynamic pricing.

One of the most commonly used promotional strategies to incentivize product
purchases is to offer occasional discounts [56], for example at certain times
of the year (e.g., winter sales) or special events (e.g., Black Friday). In
the context of RSs many studies [268, 145, 254, 230, 149, 151, 116] aim to
generate recommendations while considering discounts. Some studies propose,
for example, to use re-ranking algorithms [145] to promote products on sale or
in-processing methods [254] based on adaptations of MF-based models to
explicitly consider customers’ discount sensitivity [230]. Another method is
proposed in two related studies [149, 151]. In particular, as noted by the
authors, there may be inter- and cross-category effects when discount products
are bought. Thus, especially in e-commerce, organizations can exploit RSs to
incentivize customers to buy discount products but also those products that
are related to them but not on sale (e.g., camera on sale and full-price
lens). A similar analysis is also made [116] to determine the optimal
shipping-fee discount to attract customers to the platform and encourage them
to purchase products related to the discounted ones.

While discounts may be occasional and the same for all customers, some
methodologies are proposed in the RSs literature to generate dynamic customer-
specific prices in order to strategically promote certain products and
generate higher profits. In this context, some initial studies [25, 26]
propose to use survey-based techniques (conjoint analysis) to estimate
customer WTP (recall Section 4.1.1) and filter items that are priced higher
than WTP in the ranking. The authors also discuss some possible configurations
of the algorithm to set the prices based on WTP in order to generate more
profit for the organization. However, the proposed pricing model is only
theoretical as it is not validated by empirical experiments. Another study
[154] proposes a system that classifies customers based on whether they would
buy products only if discounted or not. Based on the type of customer, the
system can offer a discount in order to incentivize purchases. However, as
discussed later [193], the study is based on assumptions that are not feasible
in practice: all products have the same price; only two price values are
available (i.e., standard and discounted price). Another work [289] proposes a
different methodology. The study focuses on a lottery-based mechanism that
aims to obtain the exact WTP for one subset of products and then to exploit
this information to predict the WTP for another subset of products. In this
way, the system can offer a personalized promotion to increase the conversion
rate of the latter product subset. The authors report significant results on
the potential ability of this system to increase profit over conventional
systems. However, the experiments are based on a user study with a low number
of users. Finally, another study [5] proposes a dynamic personalized pricing
recommender system for information goods (e.g., digital movie rentals). These
goods differ from physical goods in that their production and distribution
costs are negligible and they can be copied, rented, and resold easily. In
this context, traditional markup-based pricing methods (i.e., cost plus
margin) are not effective because there is no true underlying unit cost. The
proposed system first classifies customers according to their WTP and quality
sensitivity (e.g., whether they prefer a premium version of the same product).
Then it calculates a personalized price to incentivize purchase.

####  4.4.2 Bundling Methods

One frequently used promotional strategy [250] to increase sales revenue of
certain products is to offer them at a discount if purchased in bundles [119].
In the literature [271] it is proposed, for example, to include in the
bundles: (a) products that are complementary to each other in order to
incentivize cross-selling; (b) products that are uncorrelated, for example, to
clear the stock in the warehouse; (c) the same product in multiple quantities
(e.g., 2x1 promotion). Specifically, in RSs research [176], one branch of the
literature focuses on recommending bundles to optimize profit by exploiting
price modeling techniques. The other branch, in contrast, does not exploit
such techniques and focuses solely on optimizing relevance666Relevance-based
bundling algorithms [239] can be based for example on association rules [273,
83, 152], graph-based approaches [183, 96, 183, 74, 27], GNNs [46, 12, 47] and
transformers [23].. In this review, we focus only on bundling approaches that
aim to explicitly optimize business KPIs.

Concerning price modeling bundle recommendation techniques, two related
earlier studies [95, 94] focus on the development of a shopbot (i.e.,
comparison shopping agents) capable of offering bundles at a discounted price
based on an integer linear programming model. The proposed algorithm is
validated using data from Amazon.com and Buy.com reporting significant results
from the perspective of potential economic savings of price-sensitive bundle
purchasing customers. However, the data sample used is very small, and
optimization of business KPIs is not explicitly considered. In contrast, two
other studies [150, 301] leverage similar integer programming-based approaches
to recommend bundles with the goal of optimizing profitability [150] or any
business objective [301]. In particular, considering the case where the bundle
can be created directly by the customer by selecting the products of his or
her preference, the first study [150] proposes a multistage approach that can
dynamically determine the price of the added products in real-time with the
goal of maximizing profits for the organization. In contrast, the second study
[301] investigates how to incorporate product compatibility and potential cost
savings to generate bundles that, if recommended, could optimize certain
business objectives (e.g., profitability, revenue, and others). Both studies
report results regarding the potential ability of the proposed systems of
increasing profitability and conversion rates. In addition, two other
approaches [31, 82] are proposed recently. The first approach [31] is based on
a collaborative filtering algorithm that integrates demand estimation and
price modeling techniques to make recommendations with the goal of jointly
maximizing purchase probability and sales revenue considering the customer
WTP. The second approach [82] is based on an algorithm that can recommend
bundles with customized discounts to customers considering also inventory
levels. However, in the former case, the bundle does not offer an additional
discount over the full price of the individual products. Instead, the bundle
is created exclusively so that the total price of the products inside it is
aligned with the customer’s WTP to meet his or her price preferences. In the
latter case, on the other hand, the evaluation is based on a simulation
focused on the aviation industry with a large number of assumptions.

####  4.4.3 Brand-Awareness Methods

Some methods in the literature can be used to promote the organization’s
products and services, raise brand awareness, and increase profitability in
the long run. These methods can be interpreted by referring to the sales
funnel [249]. The sales funnel is a theoretical model that describes the
customer journey in different stages according to the type of customer
interaction with the organization [212]. Depending on the status of the
customer in the sales funnel, it might be advisable to design an RS with
different purposes.

If the customer has not yet made the first purchase (which is referred to as
the prospect state), it might be promising to maximize the conversion rate by
closing the first deal as quickly as possible [156]. At this early stage,
recommending the most popular products may not be the best strategy. Since
many popular products are commonly purchased together, customers would
discover them on their own without the need of a recommendation. Instead, it
could be more beneficial to present still popular but unrelated products,
optimizing coverage. In this way, it may be possible to attract more customers
to the platform and increase the probability they make their first purchase.

Once the customer has made the first purchase, the company can exploit
mechanisms to optimize profits in the long-run [39, 108]. One option could be
to mainly recommend items with high consumer ratings [147]. However, similarly
to the previous case, this may not be the best choice either, as many
customers might search for and buy such items anyway. Instead, it might be
more valuable to stimulate the purchase of products of possible interest that
are likely unknown to the customer [39], e.g., products that do not fall in
the top-$k$ but have medium-high ranking positions. This way, the company may
get both the revenue from the purchases of products that the customer would
discover on their own without the recommendations, and an additional revenue
through the purchases that were triggered by the recommendations. With similar
objectives, it might also be worthwhile for the company to leverage an RS
[108] to launch a marketing campaign with the purpose of promoting new
products in the market. Such a system could be designed to select a set of
seed consumers for the marketing campaign such that if these seed consumers
provide relatively high ratings, the number of other consumers to whom the new
product is recommended is maximized.

###  4.5 Long-Term Value Sustainability Methods

It is very important for organizations to grow sustainably over time [186,
222]. Accordingly, a number of studies in the literature [130, 139, 216]
propose recommendation algorithms that consider temporal dynamics to optimize
long-term business value. Many of them rely on the Customer Lifetime Value
(CLV) [38, 37] and other related conceptual models (e.g., Recency Frequency
Monetary \- RFM) from the business literature. CLV represents the expected
business value of all future cash flows attributed to a specific customer
discounted to the present time.

Similarly to what is found for bundling methods (in Section 4.4.2), some RSs
studies propose to exploit CLV to optimize long-term profit [216, 139] while
others exploit it solely to optimize relevance777Typically, RSs that rely on
CLV-related models to optimize relevance [182, 181, 237, 236, 242, 59, 264,
155, 38] follow a common workflow. Algorithms first group users into similar
customer value segments. Then they generate recommendations through
association rules or collaborative filtering leveraging this additional
information. [236, 242]. In this review, we focus only on algorithms that aim
to optimize long-term business KPIs. Below we give some insights on how these
methods work by discussing a few selected articles. In particular, we first
discuss in- and post-processing methods based on supervised learning and then
we delve into recent algorithms based on reinforcement learning.

####  4.5.1 Post-Processing and Supervised Learning Methods for Long-Term
Business Value Optimization

Some studies [130, 131, 29, 210] propose post-processing algorithms to
maximize the long-term business value of recommendations by exploiting
heuristic criteria. In particular, Hosanagar [130] proposes an algorithm
following this simple but effective intuition: when a customer trusts an RS,
the system should bias the recommendations to increase profitability; instead,
when the customer trust is below a certain threshold, the system should
recommend the most relevant products to restore trust at the expense of
profitability. The original study [130] proposes only a theoretical assessment
of the profit surplus that can be generated using this algorithm. However, the
algorithm’s performance is also evaluated in an online study [210] and in a
recent post-hoc econometric analysis [29]. These recent studies demonstrate
both the effectiveness of the proposed methods in generating higher sales
revenue than a content-based filtering algorithm [210] and how trust is
positively correlated with higher sales revenue [29].

Other approaches based on supervised machine learning algorithms are also
studied to explicitly optimize the long-term business value of
recommendations. In particular, in two related studies [138, 139], a
recommendation system is proposed to explicitly maximize CLV. The algorithm is
designed specifically for subscription-based [138] and transaction-based [139]
revenue models. In particular, survival analysis techniques are used to
identify frequent purchasing patterns among higher CLV users. Then,
recommendations are generated to match those patterns as closely as possible.
The algorithms are evaluated using real data from a mobile cartoon provider
with a subscription-based revenue model [138] and an online music provider
with a transaction-based revenue model [139], both from Japan. However,
although results regarding the improvement of the subscription period and the
number of items purchased over time are reported, the evaluation is only based
on a simulation system of user purchasing behavior.

####  4.5.2 Reinforcement Learning Recommendation Methods for Long-Term
Business Value Optimization

Recent studies propose methodologies based on Reinforcement Learning (RL) for
optimizing the long-term business value of recommendations [241]. RL is a
learning approach that aims to learn an optimal policy (i.e., recommendation
strategy) based on the sequential interaction between an agent and the
environment through trial and error to maximize a reward. This methodology is
used many times in the literature [303, 292, 148, 115, 267, 244, 153, 216,
122] to optimize the customer lifetime value.

A few studies propose algorithms to directly optimize profit [153, 216]. These
studies focus on the transaction-based revenue model where each customer
purchase brings a certain profit to the organization. Specifically, in this
context, one study [216] considers that a certain profit share can be
allocated to each user action (i.e., click, add-to-cart, pay). Hence, the
overall profitability can be maximized by optimizing the sum of the profit
allocated to each user action considering the probability that such an action
will occur given the recommendations. Other studies [303, 292, 148, 115, 267,
244], in contrast, propose algorithms to optimize user engagement, or more
generally some strategic interrelated business indicators [122]. One study
[244] is based on the advertising revenue model. In this particular context,
advertisers are used to pay the platform a certain monetary amount for each
click or conversion generated. Hence, in this case, by optimizing user
engagement, profit is directly optimized. Instead, other works [303, 292, 148,
115, 267], although they similarly propose to optimize user engagement, are
not based on advertising revenue models. Therefore, in these cases, the
relationship with profitability is indirect, as user engagement positively
correlates with retention.

##  5 Evaluation Methodologies

In this section, we review the evaluation methodologies used in the surveyed
papers. First, we give some insights into the different methodologies that are
used to evaluate algorithms. Next, we discuss the metrics used in offline
evaluation. Then we discuss the results that have been obtained in the real
world from ECRSs algorithms by analyzing in detail those studies that report
online performance. Finally, we analyze related topics concerning public
datasets and the current level of reproducibility.

###  5.1 Evaluation Approaches

Static Pred.Sim. StudiesEcon. Anal.User StudiesA/B
Tests$0$$10$$20$$30$$40$$50$$60$$70$$80$$90$Number of Studies Price
Sensitivity Economic Utility Modeling Profit Awareness Promotional Long-Term
Value Sustainability Figure 4: Distribution of evaluation methods in the
surveyed literature organized by dimension of analysis.

In the field of RSs, several methods are proposed to evaluate the performance
of algorithms and systems. Depending on the objective of the study, the
evaluation may vary in order to assess specific aspects of the recommendations
and the system. We identify five methods that are used in the surveyed
literature. Some of these are used for offline evaluation (e.g., static
predictions, simulation studies, and econometric analyses) [291], while others
are used for online evaluation (e.g., user studies, and A/B tests) [53]. While
offline methods aim to give a plausible estimate of the performance the system
could achieve under real circumstances if certain assumptions are verified,
online ones are instead based on real user interactions. In Figure 4 we report
the distribution of evaluation methods in the literature according to the
subdomain of analysis. As can be seen, offline methods are used more
frequently than online ones. Moreover, among offline methods, static
predictions is the most frequently used method.

##### Static Predictions

The most commonly used evaluation method in the RSs literature is to hide some
data (e.g., ratings, interactions) from a particular dataset, train a model on
the remaining data, and then predict the hidden data [144, 291]. After
constructing a dataset that contains all the necessary information, the
adopted standard is to measure the performance of the system with respect to
some underlying objectives [140, 17] with the help of certain metrics. In
terms of metrics, given the underlying purposes of ECRSs, the surveyed
literature often measures not only relevance prediction metrics (e.g.,
precision, MRR, NDCG) [112], but also business value metrics888We discuss the
most frequently used offline metrics in Section 5.2. (e.g., profit, revenue)
[144, 141, 189].

##### Simulation Studies

While static predictions methods [291] are mainly used to obtain an estimate
of RSs performance in the short term, other studies propose to use dynamic
simulations to assess long-term performance [101]. The methodology first
involves building a simulator to mimic user behavior [216, 244, 279]. Next,
the simulator is used to train and test RSs algorithms on the simulated
behavior [116, 267]. Simulators are often adopted to evaluate the performance
of reinforcement learning-based recommendation algorithms [11] (e.g., RecoGym
[229], RecSim [137]). Moreover, in the surveyed literature there are also
simulators [139, 185, 101] created to evaluate supervised learning algorithms.
One of these [101], based on agent-based modeling, is designed to
realistically mimic customer behavior considering various factors known in the
literature to have a high correlation with purchase probability (e.g., trust).

##### Econometric Analyses

For some algorithms in the surveyed literature [68, 130, 151], performance is
assessed with the help of econometric analyses [3, 207]. These are
quantitative approaches based on statistical or mathematical methods used to
estimate the impact of the system on certain variables of interest (e.g.,
profit [130]), considering some underlying assumptions. For example, one study
[130] investigates the impact of recommendations on corporate profit and
consumer welfare by modeling the behavior of a system that considers the
simplified case in which the company can sell only two products.

##### User Studies

In many cases, the impact of the system on certain factors (e.g., user
satisfaction) is difficult to model through offline methods. This occurs
because in some cases it is not possible to find a good proxy for the target
variable, while in other cases it would be necessary to use a large number of
assumptions. Especially when the factors are qualitative and the response is
subjective (e.g., perceived fairness), the literature adopts user studies as a
research methodology [53]. These methods typically involve recruiting a group
of users (e.g., through emails or through crowdsourcing platforms like Amazon
Mechanical Turk), randomly splitting them into distinct groups, requiring them
to perform a particular task (e.g., interacting with an RS designed for the
study), observing their (objective) behavior, and asking them about their
subjective perceptions. In the surveyed literature these methods are used
[210, 24] for example to determine the impact of algorithms on profitability
and user trust.

##### A/B Tests

When it is necessary to measure the performance of recommender systems in
real-world circumstances, A/B tests are often performed [53]. In such tests,
two (or more) versions of a system are deployed for a certain period of time
and users either interact with one or the other version [144]. Although these
tests are often complex to execute and require significant effort, the main
advantage is that they are able to directly measure business KPIs (e.g.,
revenue, profit) [53] and to compare different algorithms in production. These
tests are used many times in the surveyed literature999We discuss results of
A/B tests in Section 5.3. since algorithms are often designed to optimize such
KPIs. For example, A/B tests are used to measure the effects of a profit-aware
algorithm deployed on Alibaba’s AliOS appstore [288] and the CTR of a
reinforcement learning-based algorithm deployed on a large e-commerce platform
[216].

###  5.2 Metrics Used in Offline Evaluations

Refs |  Metric |  Type |  Definition
---|---|---|---
[112] |  $Prec@k=\frac{1}{|\mathcal{U}|}\sum_{u\in\mathcal{U}}\frac{\sum_{j=1}^{k}{rel_{u,j}^{\mathcal{Y}}}}{k}$ (19) |  Relevance |  Precision at position $k$ is the number of relevant items in the top-$k$ recommendations over the number of recommended ones.
[112] |  $Rec@k=\frac{1}{|\mathcal{U}|}\sum_{u\in\mathcal{U}}\frac{\sum_{j=1}^{k}{rel_{u,j}^{\mathcal{Y}}}}{\sum_{i=1}^{n}{x_{u,i}}}$ (20) |  Relevance |  Recall at position $k$ is the number of relevant items in the top-$k$ recommendations over the total number of relevant ones.
[112] |  $HR@k=\frac{1}{|\mathcal{U}|}\sum_{u\in\mathcal{U}}\begin{cases}1&\text{if }\quad\sum_{j=1}^{k}{rel_{u,j}^{\mathcal{Y}}}\geq 1\\\ 0&\text{otherwhise}\end{cases}$ (21) |  Relevance |  Hit-Rate at position $k$ is the fraction of users for which the recommendations list contains at least one relevant item.
[112] |  $MRR@k=\frac{1}{|\mathcal{U}|}\sum_{u\in\mathcal{U}}\frac{1}{j_{u}^{1}}$ (22) |  Relevance |  Mean Reciprocal Rank at position $k$ is the mean rank of the first relevant item in the recommendations list. In the equation, $j_{u}^{1}$ is the rank (position) of the first item relevant to user $u$.
[112] |  $NDCG@k=\frac{1}{|\mathcal{U}|}\sum_{u\in\mathcal{U}}\frac{\sum_{j=1}^{k}{\frac{rel_{u,j}^{\mathcal{Y}}}{log_{2}(j+1)}}}{IDCG_{u}@k}$ (23) |  Relevance |  Normalized Discounted Cumulative Gain at position $k$ is the inverse log reward on all positions with relevant items among the top-$k$ recommended ones. In the equation, $IDCG_{u}@k$ is the Ideal Discounted Cumulative Gain obtained by sorting all the items relevant to the user in descending order.
[189]* |  $Revenue@k=\sum_{u\in\mathcal{U}}\sum_{j=1}^{k}{rel_{u,j}^{\mathcal{Y}}\cdot p_{j}}$ (24) |  Value |  Revenue at position $k$ is the revenue from relevant items in the recommendations list.
[141]* |  $Profit@k=\sum_{u\in\mathcal{U}}\sum_{j=1}^{k}{rel_{u,j}^{\mathcal{Y}}\cdot v_{j}}$ (25) |  Value |  Profit at position $k$ is the profit from relevant items in the recommendations list.
[43]* |  $EP@k=\sum_{u\in\mathcal{U}}\sum_{j=1}^{k}{\hat{x}_{u,j}(\mathbf{\Theta})\cdot v_{j}}$ (26) |  Value |  Expected Profit at position $k$ is the statistical profit it is expected to achieve by the recommendations considering the expected user interest $\hat{x}_{u,j}(\mathbf{\Theta})$. $EP@k$ is referred to as statistical profit (compared with $Profit@k$ in Eq. 25), because the probability that the user accepts the recommendations instead of the actual ground truth relevance information is considered.
[162]** |  $PAH@k=\frac{1}{|\mathcal{U}|}\cdot\frac{Profit@k}{Volume@k}$ (27) |  Value |  Profit-at-Hit at position $k$ is the average profit per user from relevant items in the recommendations list.
[184]* |  $P\text{-}NDCG@k=\frac{1}{|\mathcal{U}|}\sum_{u\in\mathcal{U}}\frac{\sum_{j=1}^{k}{\frac{rel_{u,j}^{\mathcal{Y}}\cdot p_{j}}{log_{2}(j+1)}}}{P\text{-}IDCG_{u}@k}$ (28) |  Value |  Price-Based Normalized Discounted Cumulative Gain at position $k$ is defined as $NDCG@k$, where the gain is given by the items price. In the equation, $P\text{-}IDCG_{u}@k$ is the Price-Based Ideal Discounted Cumulative Gain obtained by sorting the prices of all relevant items to the user in descending order.
Table 7: Most frequently used offline evaluation metrics in the surveyed
literature. *Note that for the sake of notation we used $p_{j}$ and $v_{j}$ as
variables to indicate the price and profit of the recommended item at position
$j$, but these variables depend only on the item and not on the position.
**The formulas capture the main essence of the metrics.

A variety of metrics are used in the literature in offline evaluations,
including both accuracy metrics to assess the relevance prediction performance
as well as metrics aimed to investigate organizational value.

Given $rel_{u,j}^{\mathcal{Y}}$ as a ground truth relevance101010The relevance
of each item typically corresponds to the value of the ground truth
$x_{u,i}\in\\{0,1\\}$, i.e., assuming $x_{u,i}=1$ if the item was actually
purchased by the user, and $x_{u,i}=0$ if not. variable that indicates whether
the item recommended at position $j$ in the ordered ranking
$\mathcal{Y}_{u,k}$ is relevant or not for user $u$, we report in Table 7 the
metrics used for offline evaluations in the literature. In the table we
indicate for each metric the reference, the formula, the type, as well as its
definition. In the following, we mainly focus on value metrics, since
relevance prediction metrics (e.g., Precision, Hit-Rate, NDCG - see Eq. 19,
21, 23), are already widely known [112] and do not require further discussion
here.

The general principle is the same for all value metrics. Similarly to
relevance prediction metrics, first a list of top-$k$ recommendations is
generated for each user. Then the recommendations are compared to the ground
truth and certain value-related aspects are collected. Those value-related
aspects are connected to the price and profit (and in more general terms also
to the utility) of each recommended item. In particular, differently from
prediction relevance metrics, value ones do not only count the hits but
multiply that hit by the items’ price and profit. We briefly introduce the
most frequently used value metrics as follows:

  * •

Revenue@k (Eq. 24) [188, 189, 24] indicates the total revenue from the sale of
recommended products actually purchased by users;

  * •

Profit@k (Eq. 25) [141, 101, 62, 201] indicates the total profit from the sale
of recommended products actually purchased by users;

  * •

EP@k (Eq. 26) [43] indicates the statistical expected profit from the
recommendation. EP@k compared to Profit@k is referred to as statistical
because the probability of the user accepting the recommendations is
considered rather than the ground truth information;

  * •

PAH@k (Eq. 27) [162] indicates the overall profit generated by the
recommendation per user divided by the number of items sold;

  * •

P-NDCG@k (Eq. 28) [184, 180] indicates the total revenue generated on average
per user from the recommendation compared to the theoretically achievable
maximum revenue. P-NDCG@k, like NDCG@k (Eq. 23) [112], gives more importance
to the higher-priced items positioned on the top of the ranking111111Note
that, as in IR [160, 32], value metrics can be rank-agnostic (e.g., Revenue@k,
Profit@k) or rank-aware (e.g. P-NDCG@k), depending on whether the position of
the recommended items in the ranking is considered for evaluation or not..

However, analyzing the surveyed articles, some open issues can be identified.
In particular, we observe that the literature is mostly scattered,
application-specific, and there are no well-defined standards in offline
assessment of business value [141, 70]. Often the same metric is referred to
by different names (e.g., Price-Based NDCG [184], vs. G-DCG [180]). Other
times, researchers report results that are not comparable to each other
because application-specific metrics are proposed in the article to
investigate certain types of value (e.g., perishability [235], marginal
utility per dollar [99]). In fact, under certain circumstances, it would not
even be possible to use certain metrics. For example, in the case where the
underlying dataset carries only price information and not profit information
(e.g., Amazon [203], Tmall [300]), the metrics related to the latter would not
be computable without using synthetic profit distributions of the
dataset121212We discuss the synthetic profit issue in Section 5.4.. Finally,
in cases where simulations are used, the calculation of value metrics may be
based on assumptions. The main assumption that can be found [141, 101] is that
in some studies the user is supposed to always buy at least one item of the
top-$k$ recommended ones. In these cases, since the user may not have actually
purchased any of the recommended items if his or her purchase history is
analyzed, the underlying ground truth information may be unrealistic.

###  5.3 Real-World A/B Tests and User Studies

Ref |  Year |  Eval. |  Channel |  Subjects |  Durat. |  Baseline |  $\Delta\%$IPV |  $\Delta\%$CTR |  $\Delta\%$CVR |  $\Delta\%$GMV |  $\Delta\%$Rev. |  $\Delta\%$Prof.
---|---|---|---|---|---|---|---|---|---|---|---|---
[191] |  2022 |  A/B Test |  E-Commerce Platform (Walmart) |  36M sessions |  - |  Walmart Ranker |  |  |  |  +0.71% |  |
[45] |  2022 |  A/B Test |  Booking Platform |  1M searches |  20 days |  Platform Ranker |  |  [-0.50%, +2.00%] |  |  |  |
[12] |  2022 |  A/B Test |  E-Commerce Platform |  - |  - |  Co-Purchase |  |  |  |  |  +35.0% |
[177] |  2021 |  A/B Test |  Online Insurance Platform |  - |  1 week |  LogReg |  |  |  +[1.05%, 3.98%] |  |  +[2.7%, 16.2%] |
[148] |  2021 |  A/B Test |  E-Commerce Platform (Taobao) |  - |  1 week |  Vanilla-CTR |  +[6.25%, 8.67%] |  |  |  +[12.31%, 18.03%] |  |
[74] |  2020 |  A/B Test |  Video Game Platform (NetEase) |  - |  1 year |  Platform Ranker |  |  |  +60.0% |  +15.0% |  |
[216] |  2019 |  A/B Test |  E-Commerce Platform |  1M users |  1 week |  Item KNN |  +8.80% |  +8.20% |  |  +27.90% |  |
[180] |  2019 |  A/B Test |  E-Commerce Platform |  - |  3 days |  LETORIF |  +23.76% |  +13.80% |  |  +3.62% |  |
[288] |  2017b |  A/B Test |  Appstore (Alibaba AliOS) |  1M users |  2 weeks |  LinDP |  |  |  -6.00% |  |  |  +32.0%
[210]* |  2016 |  User Study |  Mail Campaign |  260 users |  9 weeks |  CBF |  |  |  |  |  +94.39% |  +137%
[289] |  2015 |  User Study |  Amazon Mechanical Turk |  79 users |  - |  Amazon Price |  |  |  |  |  |  +[241%, 248%]
[301] |  2014 |  User Study |  Mail Campaign |  few users |  1 week |  Markov Model |  |  +7.43% |  +48.92% |  |  |
[24] |  2013 |  User Study |  Amazon Mechanical Turk |  245 users |  - |  Pers. NonCF |  |  |  |  |  +28.57% |
Table 8: Results of real-world A/B tests and user studies in the surveyed
literature. In the table we report: the channel used to convey the
recommendations; the number of subjects (i.e., users, searches, or sessions);
the overall duration of the study (e.g. 20 days, 1 year); the baseline against
which the proposed algorithm in the study is compared; and the relative
improvements in online metrics of the proposed algorithm compared to the
baseline. *The relative improvements are determined by analyzing the sentence
“Overall revenue generated during the experiment was €428 for the content-
based group, €832 for the profit-based group” and Figure 11b in the original
paper [210].

Many authors evaluate the performance of ECRSs algorithms using A/B tests or
user studies. As is known in the literature, offline evaluation results are
not necessarily a valid indicator of online performance [144, 143]. This is
often due to the fact that different metrics are used for the two types of
experimental evaluation [75, 234]. While offline metrics are often used to
measure relevance prediction accuracy (e.g., Precision, NDCG), online metrics
are used instead to measure business value (e.g., CTR, GMV, Revenue) [159,
92]. Companies are usually much more interested in assessing how algorithms
impact real-world business KPIs exploiting online metrics.

In Table 8 we list the studies in the surveyed literature that measure the
performance of the proposed systems through A/B tests or user studies. In
Table 9 we then briefly summarize the meaning of each online metric that is
considered for the analysis131313Some niche metrics used to measure certain
application-specific factors reported in the studies are not considered (i.e.,
IPV, CTR, CVR, GMV, Revenue, Profit). We refer readers to a recent survey
[144] on this topic for further insights into online metrics.

Metric |  Meaning
---|---
IPV |  Individual Page View is the overall number of clicked items on the platform.
CTR |  Click-Through Rate is the number of user clicks divided by the number of items shown.
CVR |  Click-Conversion Rate is the number of purchases (or other events) divided by the number of clicks.
GMV* |  Gross Merchandise Value is the number of items sold multiplied by their price.
Revenue* |  Revenue is equal to GMV minus any commissions from item sellers.
Profit** |  Profit is equal to Revenue minus any item costs.
Table 9: Most frequently used online metrics in the surveyed literature [144]. *GMV and Revenue almost always indicate the same measure except in B2C marketplaces like eBay. **Depending on the type of subtracted cost (e.g., raw materials, marketing), profit can be gross, net, or have additional nuances (e.g., EBITDA, EBIT).  Ref | Dataset | #User | #Item | #Inter | Density |  Event |  Date |  Dem. |  Cat. |  Price |  Profit
---|---|---|---|---|---|---|---|---|---|---|---
[22] | Cosmetics | $1\,639\,358$ | $54\,571$ | $20\,692\,840$ |  0.023 % |  View, Add-to-Cart, Remove-From-Cart, Purchase |  ✓ |  |  ✓ |  ✓ |
[61] | Diginetica | $204\,789$ | $184\,047$ | $993\,483$ |  0.002 % |  Query, Click, Purchase |  ✓ |  |  ✓ |  ✓ |
[203] | Amazon(2018)* | - | $15\,500\,000$ | $233\,000\,000$ | - |  Review, Ratings |  ✓ |  |  ✓ |  ✓ |
[284] | Yelp(Full)* | $5\,556\,436$ | $539\,254$ | $289\,088\,240$ |  0.009 % |  Review, Ratings |  ✓ |  |  ✓ |  ✓ |
[81] | Yahoo!Music | $1\,948\,882$ | $98\,211$ | $11\,557\,943$ |  0.006 % |  Ratings |  |  |  |  |
[132] | Ta-Feng | $32\,266$ | $23\,812$ | $817\,741$ |  0.106 % |  Purchase |  ✓ |  ✓ |  ✓ |  ✓ |
[120] | MovieLens(20M)* | $138\,493$ | $27\,278$ | $20\,000\,263$ |  0.529 % |  Ratings |  ✓ |  ✓ |  ✓ |  |
[35] | Netflix Prize | $480\,189$ | $17\,770$ | $100\,480\,507$ |  1.177 % |  Ratings |  ✓ |  |  |  |
[90] | SPMF | - | $16\,470$ | $88\,162$ | - |  Purchase |  ✓ |  |  |  ✓ |
[180] | EC-REC | - | - | - | - |  View, Click, Purchase |  ✓ |  |  |  ✓ |
[302] | Book-Crossing | $105\,284$ | $340\,557$ | $1\,149\,780$ |  0.003 % |  Ratings |  |  ✓ |  |  |
[63] | Foodmart | $8842$ | $1559$ | $261\,130$ |  1.894 % |  Purchase |  ✓ |  ✓ |  ✓ |  ✓ |  ✓
[196] | Last.fm | $1892$ | $176\,322$ | $92\,834$ |  0.027 % |  Listen |  ✓ |  |  ✓ |  |
[228] | Epinions | $226\,570$ | $231\,637$ | $1\,132\,367$ |  0.002 % |  Ratings, Graph |  ✓ |  |  |  |
[105] | Jester | $73\,421$ | $101$ | $4\,136\,360$ |  55.779 % |  Ratings |  |  |  |  |
[213] | Steam | $2\,567\,538$ | $32\,135$ | $7\,793\,069$ |  0.009 % |  Purchase |  ✓ |  |  ✓ |  ✓ |
[300] | Tmall | $963\,923$ | $2\,353\,207$ | $44\,528\,127$ |  0.001 % |  View, Add-to-Cart, Add-to-Wishlist, Purchase |  ✓ |  |  ✓ |  |
[216] | REC-RL | $49\,000\,000$ | $200\,000\,000$ | $763\,000\,000$ |  $7.785\,714\,285\,714\,29\text{\times}{10}^{-06}$ % |  Click, Add-to-Cart, Add-to-Wishlist, Purchase |  ✓ |  |  |  ✓ |
[251] | Dunnhumby | $2500$ | $92\,400$ | $2\,600\,000$ |  1.125 % |  Purchase |  ✓ |  ✓ |  ✓ |  ✓ |
[217] | Chainstore | - | $46\,086$ | $1\,112\,949$ | - |  Purchase |  |  |  |  ✓ |
[21] | NetEase | $18\,528$ | $123\,628$ | $1\,128\,065$ |  0.049 % |  Playlist |  |  |  |  |
Table 10: Most used datasets in the surveyed literature. *Since there may be
multiple versions of the same dataset, we report the statistics of the most
recent one.

Analyzing Table 8 we can make some interesting observations. Some
considerations depend on the nature of the particular evaluation methodology
(i.e., A/B test vs. user study). For example, considering the recommendations
channel and the number of subjects, we note that user studies typically
involve few users recruited through e-mail campaigns [210, 301] or Amazon
Mechanical Turk [24, 289]. Instead, A/B tests are typically performed on a
large scale, exploiting existing systems with large customer bases [288, 216],
some of well-known brands (e.g., Walmart, Taobao, Alibaba, NetEase) [191, 288,
74, 148]. Moreover, from a performance point of view, all the studies, whether
they are based on user studies or A/B tests, show that ECRSs are able to
potentially bring huge business value to the firm. In fact, increases in
online metrics are reported in all studies. In some cases, the authors report
significant performance improvements141414To ensure evaluation reliability,
many authors test the proposed algorithm in different configuration
environments reporting different results for each of them [177]. In these
cases, Table 8 shows a range instead of a single value in metrics improvement.
(e.g., +$48.92\%$ CVR [301], +$35\%$ revenue [12], +$32\%$ profit [288]).

However, there may be some limitations regarding the insights we can get from
the studies. For example, most of the A/B tests last a very short time, i.e.,
less than three weeks151515Performing long-term A/B tests on a real platform
is complex [144] and significant effort is required both in the planning and
analysis phases. Often the test could cause financial damage to the brand as
users could lose trust in the company due to ineffective recommendations.
Other times, it is necessary to re-run the test because of bugs. Or again,
certain events (e.g., Easter, Super Bowl) or global macroenomic circumstances
(e.g., 2020 COVID-19 crisis, 2022 Ukranian war) may impact performance. [45,
177, 148, 216, 180, 288, 289, 301, 24]. In some cases, the baselines are
proprietary algorithms and their internal mechanisms are unknown [191, 45, 74,
24] (e.g., Walmart Ranker). In other cases, results depend on assumptions. For
example, a study [289] based on Amazon Mechanical Turk uses synthetic profit
information, as the authors did not have product costs available. Another
study [210] uses some proxies for offline purchases in addition to explicit
purchase data from the firm’s online site to measure revenue. In that specific
context, offline purchases cannot be connected to the online identities of
experiment participants. In particular, the authors treated items that
received high ratings by users after they clicked on the “see more details”
link as purchases to calculate profit.

###  5.4 Available Datasets

Analyzing the ECRSs literature, our survey reveals that many studies report
results based on proprietary datasets. This is mainly due to the fact that
certain types of information (e.g., prices, profits, purchases, demographics)
are of strategic importance to companies, and uncontrolled sharing could
create significant economic damage. For example, some information is sensitive
to the user, and non-anonymized sharing could have major legal implications
due to privacy laws, as well as significant impact on brand reputation. In
addition, competitors could make use of economic data related to purchasing
and profitability to study weaknesses in the business model and take away
market share. However, especially recently, several studies also report
results based on public datasets.

In Table 10 we report the most frequently used public datasets in the surveyed
literature. Specifically, in addition to statistical information such as the
number of users, items, interactions, and the density of the dataset, we also
report the type of event/interaction (e.g., click, add-to-cart, purchase,
rating), and the presence of relevant features for ECRSs algorithms, i.e.,
date, user demographics, product category, price, and profit.

Analyzing the reported information, we can make some observations. First, both
the datasets’ density and size, i.e., the number of interactions, vary
greatly. Some of them are quite sparse (e.g., REC-RL [216]), whereas others
are dense (e.g., Jester [105]). Some are quite small (e.g., Foodmart [63]),
while others are large (e.g., Amazon [203]). In addition, as expected, most of
the datasets contain economic information related to actual purchases, as well
as prices and possibly profit of products (e.g., in the Cosmetics [22],
Diginetica [61], Ta-Feng [132], and Tmall [300] datasets). Indeed, as
discussed earlier, economic information is typically used for both algorithmic
and evaluation purposes. However, as can be noted, some datasets do not
contain prices (e.g., MovieLens [120], Netflix Prize [35], Book-Crossing
[302], Epinions [228], Last.fm [196]) and currently only Foodmart [63]
contains profit. We observe that those datasets are the most frequently used
in RSs research. In particular, although profit is very important especially
to train profit-aware models, we note various studies [141, 101, 41, 13, 218,
185, 43, 201, 62] assuming some synthetic profit distribution, e.g., normal
[101], or random [201]. This assumption would allow to overcome the profit
availability issue. However, as reported in almost all the studies, this also
constitutes an important limitation. In fact, under real circumstances, the
profit distribution could be very different from the synthetic one used for
the experiments, and the results could vary considerably.

###  5.5 Reproducibility Maturity

Ref |  Year |  Dimension |  Link
---|---|---|---
[47] |  2023 |  Promotional |  https://github.com/cjx0525/BGCN
[283] |  2022b |  Price-Sensitivity |  https://github.com/Zhang-xiaokun/CoHHN
[265] |  2022 |  Price-Sensitivity |  https://github.com/PCNet-Code
[101] |  2022 |  Profit-Awareness |  https://github.com/nadaa/rec-strategies-abm
[23] |  2022 |  Promotional |  https://github.com/tzoof/BRUCE
[12] |  2022 |  Promotional |  https://github.com/muhanzhang/SEAL
[278] |  2021 |  Long-Term Value Sustainability |  https://github.com/google-research/google-research/tree/master/recs_ecosystem_creator_rl
[295] |  2020 |  Price-Sensitivity |  https://github.com/DavyMorgan/ICDE20-PUP
[270] |  2020 |  Economic Utility Modeling |  https://github.com/zhichaoxu-shufe/E-commerce-Rec-with-WEU
[98] |  2020 |  Economic Utility Modeling |  https://github.com/TobyGE/Risk-Aware-Recommnedation-Model
[67] |  2020 |  Economic Utility Modeling |  https://github.com/xydaisjtu/U-rank
[46] |  2020 |  Promotional |  https://github.com/cjx0525/BGCN
[216] |  2019 |  Long-Term Value Sustainability |  https://github.com/rec-agent/rec-rl
[180] |  2019 |  Profit-Awareness |  https://github.com/weberrr/PE-LTR
[99] |  2019 |  Economic Utility Modeling |  https://github.com/TobyGE/Maximizing-Marginal-Utility-per-Dollar-for-Economic-Recommendation
Table 11: Studies in the surveyed literature that provide the code.

The impact of reproducibility on the progress of science is undeniable.
However, although there has generally been an increase in reproducible papers
in AI over the years [113], many of them are still not sufficiently well
documented to reproduce the results of the reported experiments [117]. This
problem is observed several times in the field of RSs [30, 66], with well-
known cases regarding articles that proposed neural algorithms [226, 85],
highlighting for example: non-uniform and lax standards in adopting the
correct experimental evaluation methodologies [240]; questionable choices on
the use and fine-tuning of baselines for comparative experiments [84].

In particular, by reviewing the ECRSs literature, we note several limitations
concerning the reproducibility of the studies. As reported in Table 11, only a
very small subset of 15 articles, out of 133 ($11.27$%) identified by the
present systematic review share the implementation code161616We did not dive
into the code details because even if the code is shared, it was found earlier
in the RSs literature [240, 84, 66] that in many cases important information
is missing to ensure reproducibility (e.g., pre-processing code).. Notably, as
can be seen from the table, we find no article that publicly share the code
prior to 2019. In addition, the level of reproducibility is quite uneven when
considering the different subdomains of ECRSs. In particular, we note the
following critical issues: there are many articles published in the profit-
awareness subdomain but only two of them share the code; all the articles
published in the field of promotional strategies refer to relevance-based
bundling methods (i.e., there is no code shared about brand-awareness and
pricing methods); the code of articles concerning price-sensitivity and long-
term value methods is published only for the most recent and advanced GNN- and
RL-based algorithms. Consequently, it would be beneficial and significantly
accelerate progress in this field if researchers would pay special attention
to increasing the level of reproducibility.

##  6 Current Challenges and Future Research

In this section we discuss current challenges of ECRSs and possible future
research directions.

##### Comparing Different Algorithmic Approaches

A multitude of algorithmic approaches for optimizing business value are
proposed in the literature. In this paper, we categorize them at a high level
into in-processing and post-processing methods [70] considering five
subdimensions of analysis. However, most of the approaches are never compared
with each other and may have specificities that may make them preferable in
certain circumstances over others. For example, no study has yet compared in-
processing with post-processing approaches. In addition, different types of
in-processing algorithms are found in the literature. In particular, it is
proposed for example to extend the objective function of MF [97, 50, 51, 230],
or to use GNNs [295, 294, 283] to generate price-sensitive recommendations.
Moreover, value neighbor selection [43], graph-based [41, 13, 220] or
evolutionary [201, 62] profit-aware algorithms are proposed as well. However,
some types of methods are applied only to certain dimensions of analysis. For
example, although feasible in practice, no profit-aware MF objective function
extensions or GNNs were surfaced through our study. Similarly, no neighbor
selection or evolutionary price-sensitive algorithm was found so far.
Therefore, it might be useful for the future both to compare in-processing and
post-processing approaches and to implement theoretically feasible algorithms
not yet found in the literature, comparing them with existing ones.

##### Optimizing Business Value Trade-Offs

Business value optimization is complex, and the systems must consider multiple
trade-offs [141, 70] in the optimization process. For example, considering
real-world businesses based on an advertising revenue model (e.g., YouTube,
Alibaba’s AliOS), it is very important to find the right balance between the
ad revenue generated by sponsored items and the actual interests of the user
[288, 189]. In particular, special care must be taken not to compromise user
trust [210, 130]. In fact, it is shown both through simulations [101], in user
studies [205] and subsequent A/B tests [210] that trust is positively
correlated with propensity to purchase. A system that is too biased toward
higher-value items that provides irrelevant recommendations to the user [141,
70] could risk impacting the organization’s reputation and driving away
customers. To address this issue various studies [24, 256, 185, 189, 288, 162]
propose algorithms with the goal of balancing the interests of multiple
stakeholders [1, 2], particularly considering the profitability/relevance
trade-off [141], and optimizing short- or long-term value [131]. Furthermore,
as various studies pointed out, algorithms should take care also of
explainability [285, 246], fairness [276, 277, 215, 219], and diversity [170,
210] since they are directly related to trust [74]. However, the current
literature has not thoroughly investigated the impact of many of these factors
on business value. Hence, providing efficient algorithms to simultaneously
optimize multiple business value trade-offs (e.g., profit, fairness, and
trust) could be a valuable research direction for the future.

##### Comprehensive Purpose-Oriented Offline and Online Evaluation

Evaluating ECRSs often requires the use of methods that are different from
those used for traditional RSs [291, 53]. As a result, there are still many
open challenges in order to to be able to evaluate ECRSs in a comprehensive,
purpose-oriented way [140, 17] (i.e., that considers the purposes for which
the system is designed). Several of these challenges follow from the analysis
presented in this paper. For example, in offline evaluation, it is necessary
to use business value metrics besides the widely adopted relevance prediction
metrics [112]. Studies often exploit a variety of metrics [116, 43, 141, 162,
184] albeit with similar objectives, and the results reported are not
comparable with each other. In addition, offline evaluation methodologies are
not standardized and often are designed ad-hoc according to specific needs
[130]. Moreover, besides a few exceptions [99, 101, 283, 278, 216, 270, 180],
most studies are difficult to reproduce and are often based on proprietary
datasets or public datasets with synthetic data [201, 62]. In fact, most
datasets [120, 35, 22, 203, 132, 228, 302] do not contain information such as
profitability [63], which is however needed for model training. Regarding A/B
tests on the other hand, many of them last for a short time [177, 216, 180]
and involve a small set of users [301, 24, 289, 210] to avoid potential
economic risks [144, 210] for the organization hosting the test. Hence, there
could be several future research directions in the field of evaluation. For
example, it is necessary to develop better offline value metrics that are
indicative of online performance in a given (prototype) scenario. In addition,
large-scale A/B tests (i.e., involving many users) and reproducibility studies
are also required.

##### Design of Holistic Algorithmic Methods

In this work, decomposing the literature on ECRSs into five different
dimensions of analysis, various algorithmic approaches for optimizing business
value are explored. However, most of the existing methods [52, 216, 51, 99,
289] focus exclusively on one of the five perspectives. There are a few
exceptions [162, 191] that involve more than one dimension of analysis that
study, for example [162], how to combine price-sensitivity with profit-
awareness to generate more profit while keeping relevance high. A very small
subset of studies [72, 235, 82], on the other hand, provide broader reasoning
by also discussing inventory management techniques that might be useful for
analogous purposes. Currently, the literature lacks holistic methods capable
of leveraging multiple approaches simultaneously [70, 141] complementing each
other to optimize different nuances of business value [144] while also
considering the interrelationship [122] between them. In addition, it is also
necessary to consider the relationship between sales and marketing processes
with operational [235, 72, 82] and financial processes so as to propose
methods for improving the entire business ecosystem, e.g., reducing raw
material costs, minimizing logistics delays, or optimizing cash flows.

##  7 Conclusion

In this paper, we review the existing literature on economic RSs. Unlike
traditional RSs, economic ones aim to directly optimize profitability by
exploiting purchase information (e.g., price and profit) and related concepts
from economics and marketing. This topic is highly important because
organizations aim to optimize (long-term) profit. Accordingly, economic RSs
are well-suited for use in commercial applications such as e-commerces, media
streaming sites, and advertising platforms, as they offer various benefits for
organizations to increase their business KPIs. In this survey, we identify a
number of relevant works addressing a multitude of related issues on economic
RSs. In particular, although the literature is highly scattered, five
different approaches that jointly consider the interests of customers and
organizations are identified in this paper (e.g., price sensitivity, profit
awareness). This review shall help academic scholars and industry partners to
navigate the existing literature and understand the state-of-the-art. We hope
this work will serve as a valuable starting point to foster future research
and shift academic efforts towards more impactful RSs research that matters
[140, 142].

##  8 Acknowledgments

This work was partially funded by estilos srl.

## References

  * Abdollahpouri et al. [2020] Abdollahpouri, H., Adomavicius, G., Burke, R., Guy, I., Jannach, D., Kamishima, T., Krasnodebski, J., Pizzato, L., 2020\.  Multistakeholder recommendation: Survey and research directions.  User Modeling and User-Adapted Interaction 30, 127–158.  doi:10.1007/s11257-019-09256-1.
  * Abdollahpouri and Burke [2022] Abdollahpouri, H., Burke, R., 2022\.  Multistakeholder Recommender Systems, in: Ricci, F., Rokach, L., Shapira, B. (Eds.), Recommender Systems Handbook, pp. 647–677.  doi:10.1007/978-1-0716-2197-4_17.
  * Adamopoulos and Tuzhilin [2015a] Adamopoulos, P., Tuzhilin, A., 2015a.  The business value of recommendations: A privacy-preserving econometric analysis, in: Proceedings of the International Conference on Information Systems.
  * Adamopoulos and Tuzhilin [2015b] Adamopoulos, P., Tuzhilin, A., 2015b.  On Unexpectedness in Recommender Systems: Or How to Better Expect the Unexpected.  ACM Transactions on Intelligent Systems and Technology 5, 1–32.  doi:10.1145/2559952.
  * Adelnia Najafabadi et al. [2022] Adelnia Najafabadi, H., Shekarchizadeh, A., Nabiollahi, A., Khani, N., Rastgari, H., 2022.  Dynamic pricing for information goods using revenue management and recommender systems.  Journal of Revenue and Pricing Management 21, 153–163.  doi:10.1057/s41272-020-00276-w.
  * Adomavicius et al. [2013] Adomavicius, G., Bockstedt, J.C., Curley, S.P., Zhang, J., 2013\.  Do Recommender Systems Manipulate Consumer Preferences? A Study of Anchoring Effects.  Information Systems Research 24, 956–975.  doi:10.1287/isre.2013.0497.
  * Adomavicius et al. [2018] Adomavicius, G., Bockstedt, J.C., Curley, S.P., Zhang, J., 2018\.  Effects of Online Recommendations on Consumers’ Willingness to Pay.  Information Systems Research 29, 84–102.  doi:10.1287/isre.2017.0703.
  * Adomavicius et al. [2022] Adomavicius, G., Bockstedt, J.C., Curley, S.P., Zhang, J., 2022\.  Recommender systems, ground truth, and preference pollution.  AI Magazine 43, 177–189.  doi:10.1002/aaai.12055.
  * Adomavicius and Tuzhilin [2005] Adomavicius, G., Tuzhilin, A., 2005\.  Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions.  IEEE Transactions on Knowledge and Data Engineering 17, 734–749.  doi:10.1109/TKDE.2005.99.
  * Adomavicius and YoungOk Kwon [2012] Adomavicius, G., YoungOk Kwon, 2012\.  Improving Aggregate Recommendation Diversity Using Ranking-Based Techniques.  IEEE Transactions on Knowledge and Data Engineering 24, 896–911.  doi:10.1109/TKDE.2011.15.
  * Afsar et al. [2022] Afsar, M.M., Crump, T., Far, B., 2022.  Reinforcement Learning based Recommender Systems: A Survey.  ACM Computing Surveys 55, 1–38.  doi:10.1145/3543846.
  * Agarwal et al. [2022] Agarwal, S., Das Gupta, A., Pande, A., 2022.  Improving Bundles Recommendation Coverage in Sparse Product Graphs, in: Companion Proceedings of the Web Conference, pp. 1037–1045.  doi:10.1145/3487553.3524715.
  * Akoglu and Faloutsos [2010] Akoglu, L., Faloutsos, C., 2010\.  ValuePick: Towards a Value-Oriented Dual-Goal Recommender System, in: Proceedings of the IEEE International Conference on Data Mining, pp. 1151–1158.  doi:10.1109/ICDMW.2010.68.
  * Al-Ghuribi and Mohd Noah [2019] Al-Ghuribi, S.M., Mohd Noah, S.A., 2019\.  Multi-Criteria Review-Based Recommender System–The State of the Art.  IEEE Access 7, 169446–169468.  doi:10.1109/ACCESS.2019.2954861.
  * Alhijawi et al. [2023] Alhijawi, B., Awajan, A., Fraihat, S., 2023.  Survey on the Objectives of Recommender Systems: Measures, Solutions, Evaluation Methodology, and New Perspectives.  ACM Computing Surveys 55, 1–38.  doi:10.1145/3527449.
  * Ali and Anwar [2021] Ali, B.J., Anwar, G., 2021.  Marketing Strategy: Pricing strategies and its influence on consumer purchasing decision.  International Journal of Rural Development, Environment and Health Research 5, 26–39.  doi:10.22161/ijreh.5.2.4.
  * Alslaity and Tran [2021] Alslaity, A., Tran, T., 2021\.  Goal Modeling-based Evaluation for Personalized Recommendation Systems, in: Adjunct Proceedings of the ACM Conference on User Modeling, Adaptation and Personalization, pp. 276–283.  doi:10.1145/3450614.3464619.
  * Amatriain [2013] Amatriain, X., 2013.  Beyond data: from user information to business value through personalized recommendations and consumer science, in: Proceedings of the ACM International Conference on Information and Knowledge Management, pp. 2201–2208.  doi:10.1145/2505515.2514701.
  * Amatriain and Basilico [2016] Amatriain, X., Basilico, J., 2016\.  Past, Present, and Future of Recommender Systems: An Industry Perspective, in: Proceedings of the ACM Conference on Recommender Systems, pp. 211–214.  doi:10.1145/2959100.2959144.
  * Aridor and Gonçalves [2022] Aridor, G., Gonçalves, D., 2022\.  Recommenders’ originals: The welfare effects of the dual role of platforms as producers and recommender systems.  International Journal of Industrial Organization 83, 102845.  doi:10.1016/j.ijindorg.2022.102845.
  * [no authors] [2018] [no authors], 2018.  NetEase Music, BX Book and Douban Movie Recommendation Datasets and Model Codes.  doi:10.7910/DVN/T2GWX4.
  * [no authors] [2019] [no authors], 2019.  eCommerce Events History in Cosmetics Shop.  URL: https://www.kaggle.com/datasets/mkechinov/ecommerce-events-history-in-cosmetics-shop.
  * Avny Brosh et al. [2022] Avny Brosh, T., Livne, A., Sar Shalom, O., Shapira, B., Last, M., 2022.  BRUCE: Bundle Recommendation Using Contextualized item Embeddings, in: Proceedings of the ACM Conference on Recommender Systems, pp. 237–245.  doi:10.1145/3523227.3546754.
  * Azaria et al. [2013] Azaria, A., Hassidim, A., Kraus, S., Eshkol, A., Weintraub, O., Netanely, I., 2013\.  Movie recommender system for profit maximization, in: Proceedings of the ACM Conference on Recommender Systems, pp. 121–128.  doi:10.1145/2507157.2507162.
  * Backhaus et al. [2010a] Backhaus, K., Becker, J., Beverungen, D., Frohs, M., Mueller, O., Weddeling, M., 2010a.  Incorporating Willingness-to-Pay Data into Online Recommendations for Value-Added Services, in: Proceedings of the European Conference on Information Systems, p. 16.
  * Backhaus et al. [2010b] Backhaus, K., Becker, J., Beverungen, D., Frohs, M., Müller, O., Weddeling, M., Knackstedt, R., Steiner, M., 2010b.  Enabling individualized recommendations and dynamic pricing of value-added services through willingness-to-pay data.  Electronic Markets 20, 131–146.  doi:10.1007/s12525-010-0032-0.
  * Bai et al. [2019] Bai, J., Zhou, C., Song, J., Qu, X., An, W., Li, Z., Gao, J., 2019.  Personalized Bundle List Recommendation, in: Proceedings of the International Conference on World Wide Web, pp. 60–71.  doi:10.1145/3308558.3313568.
  * Baltrunas et al. [2011] Baltrunas, L., Ludwig, B., Ricci, F., 2011.  Matrix factorization techniques for context aware recommendation, in: Proceedings of the ACM Conference on Recommender Systems, pp. 301–304.  doi:10.1145/2043932.2043988.
  * Basu [2021] Basu, S., 2021.  Personalized product recommendations and firm performance.  Electronic Commerce Research and Applications 48, 101074.  doi:10.1016/j.elerap.2021.101074.
  * Beel et al. [2016] Beel, J., Breitinger, C., Langer, S., Lommatzsch, A., Gipp, B., 2016.  Towards reproducibility in recommender-systems research.  User Modeling and User-Adapted Interaction 26, 69–101.  doi:10.1007/s11257-016-9174-x.
  * Beladev et al. [2016] Beladev, M., Rokach, L., Shapira, B., 2016.  Recommender systems for product bundling.  Knowledge-Based Systems 111, 193–206.  doi:10.1016/j.knosys.2016.08.013.
  * Bellogín and Said [2019] Bellogín, A., Said, A., 2019\.  Information Retrieval and Recommender Systems, in: Said, A., Torra, V. (Eds.), Data Science in Practice. volume 46, pp. 79–96.  doi:10.1007/978-3-319-97556-6_5.
  * Belluf et al. [2012] Belluf, T., Xavier, L., Giglio, R., 2012.  Case study on the business value impact of personalized recommendations on a large online retailer, in: Proceedings of the ACM Conference on Recommender Systems, pp. 277–280.  doi:10.1145/2365952.2366014.
  * Benbasat and Wang [2005] Benbasat, I., Wang, W., 2005\.  Trust In and Adoption of Online Recommendation Agents.  Journal of the Association for Information Systems 6, 72–101.  doi:10.17705/1jais.00065.
  * Bennett et al. [2007] Bennett, J., Lanning, S., et al., 2007\.  The Netflix prize, in: Proceedings of the KDD cup, p. 35.
  * Bergemann and Ozmen [2006] Bergemann, D., Ozmen, D., 2006\.  Optimal pricing with recommender systems, in: Proceedings of the ACM Conference on Electronic Commerce, pp. 43–51.  doi:10.1145/1134707.1134713.
  * Bhaduri and Fogarty [2016] Bhaduri, S.N., Fogarty, D., 2016\.  Customer Lifecycle Value—Past, Present, and Future, in: Advanced Business Analytics, pp. 139–156.  doi:10.1007/978-981-10-0727-9_11.
  * Blattberg et al. [2009] Blattberg, R.C., Malthouse, E.C., Neslin, S.A., 2009.  Customer Lifetime Value: Empirical Generalizations and Some Conceptual Questions.  Journal of Interactive Marketing 23, 157–168.  doi:10.1016/j.intmar.2009.02.005.
  * Bodapati [2008] Bodapati, A.V., 2008.  Recommendation Systems with Purchase Data.  Journal of Marketing Research 45, 77–93.  doi:10.1509/jmkr.45.1.77.
  * Bollen et al. [2010] Bollen, D., Knijnenburg, B.P., Willemsen, M.C., Graus, M., 2010\.  Understanding choice overload in recommender systems, in: Proceedings of the ACM Conference on Recommender Systems, pp. 63–70.  doi:10.1145/1864708.1864724.
  * Brand [2005] Brand, M., 2005.  A random walks perspective on maximizing satisfaction and profit, in: Proceedings of the SIAM International Conference on Data Mining, pp. 12–19.  doi:10.1137/1.9781611972757.2.
  * Cai et al. [1998] Cai, C., Fu, A., Cheng, C., Kwong, W., 1998.  Mining association rules with weighted items, in: Proceedings of the International Symposium Database Engineering and Applications, pp. 68–77.  doi:10.1109/IDEAS.1998.694360.
  * Cai and Zhu [2019] Cai, Y., Zhu, D., 2019.  Trustworthy and profit: A new value-based neighbor selection method in recommender systems under shilling attacks.  Decision Support Systems 124, 113112\.  doi:10.1016/j.dss.2019.113112.
  * van Capelleveen et al. [2019] van Capelleveen, G., Amrit, C., Yazan, D.M., Zijm, H., 2019\.  The recommender canvas: A model for developing and documenting recommender system design.  Expert Systems with Applications 129, 97–117.  doi:10.1016/j.eswa.2019.04.001.
  * Cavenaghi et al. [2022] Cavenaghi, E., Camaione, L., Minasi, P., Sottocornola, G., Stella, F., Zanker, M., 2022\.  An Online Experiment of a Price-Based Re-Rank Algorithm, in: Proceedings of the Workshop on Recommenders in Tourism.
  * Chang et al. [2020] Chang, J., Gao, C., He, X., Jin, D., Li, Y., 2020\.  Bundle Recommendation with Graph Convolutional Networks, in: Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 1673–1676.  doi:10.1145/3397271.3401198.
  * Chang et al. [2023] Chang, J., Gao, C., He, X., Jin, D., Li, Y., 2023\.  Bundle Recommendation and Generation with Graph Neural Networks.  IEEE Transactions on Knowledge and Data Engineering , 1–1doi:10.1109/TKDE.2021.3114586.
  * Chau et al. [2013] Chau, P.Y., Ho, S.Y., Ho, K.K., Yao, Y., 2013.  Examining the effects of malfunctioning personalized services on online users’ distrust and behaviors.  Decision Support Systems 56, 180–191.  doi:10.1016/j.dss.2013.05.023.
  * Chen et al. [2023] Chen, J., Dong, H., Wang, X., Feng, F., Wang, M., He, X., 2023.  Bias and Debias in Recommender System: A Survey and Future Directions.  ACM Transactions on Information Systems 41, 1–39.  doi:10.1145/3564284.
  * Chen et al. [2014] Chen, J., Jin, Q., Zhao, S., Bao, S., Zhang, L., Su, Z., Yu, Y., 2014.  Does product recommendation meet its Waterloo in unexplored categories?: No, price comes to help, in: Proceedings of the International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 667–676.  doi:10.1145/2600428.2609608.
  * Chen et al. [2017] Chen, J., Jin, Q., Zhao, S., Bao, S., Zhang, L., Su, Z., Yu, Y., 2017.  Boosting Recommendation in Unexplored Categories by User Price Preference.  ACM Transactions on Information Systems 35, 1–27.  doi:10.1145/2978579.
  * Chen et al. [2008] Chen, L., Hsu, F., Chen, M., Hsu, Y., 2008.  Developing recommender systems with the consideration of product profitability for sellers.  Information Sciences 178, 1032–1048.  doi:10.1016/j.ins.2007.09.027.
  * Chen [2017] Chen, M., 2017.  Performance Evaluation of Recommender Systems.  International Journal of Performability Engineering doi:10.23940/ijpe.17.08.p7.12461256.