# Practical Bandits: An Industry Perspective

Bram van den Akker  Booking.comAmsterdamThe Netherlands ,  Olivier Jeunen
ShareChatEdinburghUnited Kingdom ,  Ying Li  NetflixLos GatosCA, USA ,  Ben
London  AmazonSeattleWA, USA ,  Zahra Nazari  SpotifyNew YorkNY, USA and
Devesh Parekh  NetflixLos GatosCA, USA

(2023)

###### Abstract.

The bandit paradigm provides a unified modeling framework for problems that
require decision-making under uncertainty. Because many business metrics can
be viewed as _rewards_ (a.k.a. _utilities_) that result from _actions_ ,
bandit algorithms have seen a large and growing interest from industrial
applications, such as search, recommendation and advertising. Indeed, with the
bandit lens comes the promise of direct optimisation for the metrics we care
about.

Nevertheless, the road to successfully applying bandits in production is not
an easy one. Even when the action space and rewards are well-defined,
practitioners still need to make decisions regarding multi-arm or contextual
approaches, on- or off-policy setups, delayed or immediate feedback, myopic or
long-term optimisation, etc. To make matters worse, industrial platforms
typically give rise to large action spaces in which existing approaches tend
to break down. The research literature on these topics is broad and vast, but
this can overwhelm practitioners, whose primary aim is to solve practical
problems, and therefore need to decide on a specific instantiation or approach
for each project. This tutorial will take a step towards filling that gap
between the theory and practice of bandits. Our goal is to present a unified
overview of the field and its existing terminology, concepts and
algorithms—with a focus on problems relevant to industry. We hope our
industrial perspective will help future practitioners who wish to leverage the
bandit paradigm for their application.

††journalyear: 2023††copyright: none††conference: Proceedings of the ACM Web
Conference; April 30–May 4, 2023; Austin, TX, USA††booktitle: Proceedings of
the ACM Web Conference (WWW ’23), April 30–May 4, 2023, Austin, TX, USA

##  1\. Introduction & Motivation

Modern-day platforms on the web consist of many moving parts, all making many
decisions over time. These decisions can include ranking of content on a
homepage, recommendations, notifications and pricing—all personalised to the
user being targeted. Because of the complexity of such systems, manually
optimising these decisions to achieve a common goal (such as retention,
revenue, user satisfaction, and others) quickly becomes an unmanageable task.
For this reason, many businesses have adopted data-driven approaches to
decision-making, so as to scale up while optimising for the desired success
metrics.

As a research problem, sequential algorithmic decision-making under
uncertainty predates modern web technology, with some of its earliest
formulations being credited to Robbins (Robbins, 1952). In essence, a
decision-maker needs to iteratively take an _action_ from a certain action
space for a number of _rounds_ , with the goal of maximising the cumulative
_rewards_ (a.k.a. _returns_ or _utilities_) that the decision-maker obtains.
This seminal work has inspired a rich body of literature focusing on all
aspects of what is now known as “the bandit problem,”111This nomenclature
originates from an analogy with a gambler who aims to maximise winnings across
slot machines, often referred to as “one-armed bandits.” ultimately laying the
groundwork for recent impressive advances in reinforcement learning (Silver et
al., 2018; Vinyals et al., 2019). As such, there is promise in adopting the
bandit perspective for problems faced by the online businesses.

There is an abundance of high quality literature introducing researchers and
practitioners to the theory behind the bandit paradigm and its algorithms
(Slivkins, 2019; Lattimore and Szepesvári, 2020; Sutton and Barto, 1998).
However, identifying work relevant to the everyday challenges faced by
industry practitioners can be cumbersome. Indeed, algorithmic advances that
lead to empirical progress typically stem from fundamental theoretical
groundwork, which can be hard to navigate for those who wish to use these
algorithms to their fullest potential, whilst remaining focused on practice.

It is our goal to take a step towards bridging this gap between the theory and
practice of bandits. This tutorial is the product of an industry-wide
collaboration, giving rise to a diverse industrial perspective of using
bandits in practice, for a wide range of applications.

##  2\. Proposal Details

###  2.1. Topic & Relevance

All authors have first-hand experience using bandit algorithms for various
applications in large-scale web platforms, such as e-commerce and content
streaming.

We have curated a family of topics based on the challenges practitioners face.
Each topic has seen a large number of recent contributions from both academia
and industry, which can feel intimidating to navigate for those with limited
experience or exposure to the bandit paradigm and its many facets.

####  2.1.1. The Bandit Setting

A first step in navigating the appropriate literature is to properly
characterise the problem setting (Liu and Li, 2021).

In its most general form, a bandit setting arises when learning without full-
information (i.e., partial supervision) (Cesa-Bianchi and Lugosi, 2006). The
appropriate algorithms vary when shifting between a _full-bandit_ setting,
where only one of the actions is taken at the time, or a form of _semi-
bandits_ where this is extended to multiple actions (Lattimore and Szepesvári,
2020). Actions can be combinatorial, ranking, discrete or continuous; rewards
can be single- or multi-objective, delayed or immediate. We aim to provide a
general overview, clarifying concepts and guiding practitioners to the
relevant literature for their problem. As much as possible, we will ground
each bandit formalism in a real application (e.g., by mapping ranking bandits
to a recommender system example).

Note that bandits can be seen as a special case of Reinforcement Learning (RL)
where state space is restricted to a single state (Sutton and Barto, 1998).
While many bandit settings can naturally be extended into a more general RL
formulation, we will not venture into this area of research too deeply.
However, we will make it clear when the limitations of the bandit formalism
can potentially be overcome with RL, and provide pointers to relevant
literature when applicable.

####  2.1.2. On-policy

The canonical bandit setting is the online, _on-policy_ setting, wherein an
agent learns a policy by interacting directly with the environment in a
sequence of decisions. This gives rise to the well-known _explore_ -_exploit_
dilemma, in which the policy must weigh the informational value of taking an
action with the reward it will yield. Seminal practical applications of such
methods in web platforms make use of Upper-Confidence-Bounds (UCB) (Li et al.,
2010) or Thompson Sampling (TS) (Chapelle and Li, 2011). These strategies can
provably attain optimal regret, which intuitively makes them attractive (Auer
et al., 2002). Nevertheless, elegant theoretical results often rely on overly
restrictive assumptions—about the stationarity of the environment, and the
policy’s ability to _immediately_ observe the outcomes and update
parameters—which rarely translate to real applications. Indeed, deploying such
real-time updates at scale is far from trivial, and it typically ignores that
an on-policy bandit is often only part of a larger dynamic system. As such,
while on-policy algorithms are well-studied in the literature, and our
understanding of them contributes to advances in related settings, examples of
_off-policy_ methods being used in practice are much more abundant.

####  2.1.3. Off-policy

In practice, on-policy algorithms can be expensive to run and difficult to
make tractable. Furthermore, such instantly self-updating methods can give
rise to safety and fairness concerns when they cannot be monitored
appropriately (Jagerman et al., 2020).

As such, there are many good reasons to move the model development stage off-
line. Here, we leverage logged data (_contexts_ , _actions_ and _rewards_)
from an existing policy in production (often referred to as the _logging_
policy), to learn (Swaminathan and Joachims, 2015; Joachims et al., 2018; Wang
et al., 2016) and evaluate (Saito and Joachims, 2021; Swaminathan et al.,
2017) new policies, This type of setting is called the “off-policy” setting —
and when the logging policy differs significantly from the policies we are
trying to evaluate, this task becomes challenging. Indeed, acess to logged
data suitable for off-policy learning is not guaranteed (Wang et al., 2016;
Joachims et al., 2017; London and Joachims, 2022). Designing a good strategy
for collecting counterfactuals through randomisation (Cesa-Bianchi et al.,
2017) while balancing exploration (Schnabel et al., 2018) is both important
and non-trivial.

####  2.1.4. Pure Exploration

Pure exploration is a family of methods that use the bandit paradigm to extend
traditional A/B testing. The goal is to identify the best arm (Garivier and
Kaufmann, 2016) or an approximation (Jamieson et al., 2014) while minimising
the exposure to each arm. We will briefly cover pure exploration as it is
commonly applied in industry.

####  2.1.5. Rewards

Theoretical works in the bandit literature commonly assume rewards to be
predefined and given. However, the design of good rewards is both essential
and non-trivial when working with Bandits in practise. Designing a reward is
subject to balancing long and short-term metrics (Wu et al., 2017), as well as
competing metrics coming from different stakeholders (Mehrotra et al., 2020;
Drugan and Nowe, 2013). Moreover, rewards are generally observed under some
delay (Ktena et al., 2019; Lancewicki et al., 2021) requiring practitioners to
estimate and mitigate the corresponding biases.

####  2.1.6. Large Action Spaces

Classical work in the bandit literature typically deals with relatively small
action spaces — especially when compared to recommendation setups where the
cardinality of the item catalogue spans the order of millions. Because bounds
on regret typically depend on the size of the action space, this is to be
expected. Successful applications have been reported for bandit-based
decision-making when the number of options is limited: for example, choosing
which playlists from a predefined set to show on a user’s homepage (Gruson et
al., 2019; Bendada et al., 2020). Recent advances provide ways forward, where
action features can be leveraged to transfer signal between arms, and possibly
to new arms (Saito and Joachims, 2022; London and Joachims, 2020). When the
action space is continuous (Jeunen et al., 2022b) or consists of rankings (Sen
et al., 2021; Kiyohara et al., 2022), we need to explicitly reason about the
structure of the problem to allow for sample-efficient learning and
estimation.

Additionally, approximate nearest-neighbour retrieval methods are often of
paramount importance for systems to scale appropriately (Malkov and Yashunin,
2020). We aim to provide an overview of recent advances in this important
area, broadening the scope of problems to which bandit methods can be applied
effectively and efficiently.

####  2.1.7. Practical Considerations

When applying bandit algorithms at scale, issues arise that are not always
covered by the academic literature. Approximations must be made in order to
meet strict latency demands. Online model updates typically happen in batches
at a regular cadence, rather than immediately after each interaction.
Moreover, in production systems that retrain and deploy new models without
human intervention, automated validations are crucial. We can evaluate the
“goodness” of a policy using logged data, but these evaluations critically
depend on the propensities of the data collection policy—which are not always
available or reliable, and therefore must be approximated. We will discuss
these practical issues, the measures taken to address them and their
repercussions.

###  2.2. Style & Duration

This will be a 3-hour hands-on tutorial. First, we aim to cover the
fundamentals and introduce the state-of-the-art in research for practical
bandit applications. In doing so, it is our hope to bridge the scientific
research and applied practitioner communities interested in this field.
Second, we will dive deeper into specific applications of bandits in practice,
outlining how theoretical approaches map to practical instantiations. Third,
we will cover hands-on Jupyter notebooks that provide insights into how these
approaches can be implemented in the OpenBandit Pipeline (Saito et al.,
2021)222github.com/st-tech/zr-obp, and how one might deal with typical
industrial challenges (van den Akker et al., 2022). These notebooks will run
on Google Colaboratory333colab.research.google.com, allowing attendees to run
examples in their browsers without requiring the installation of any software.

We will build to advanced algorithms from the ground up, making this tutorial
an adequate starting point for novice as well as experienced researchers and
practitioners in this fast-growing field.

###  2.3. Audience

This tutorial targets every researcher and practitioner that has considered
the use of bandit algorithms to solve a practical problem. The material
requires an expected background knowledge on a MSc. level in computer science,
machine learning or related fields, and will build its way up to advanced
algorithms from fundamentals. Because of our practical focus, fundamental
theory will be linked to practical considerations when applicable. We expect
attendees to feel empowered to devise practical bandit-based solutions to
machine learning problems they will be faced with in the future.

###  2.4. Previous Editions

This will be the first edition of this tutorial, but the presenters have
experience in teaching material covering similar topics in the past. We
anticipate to present (an extended version of) this tutorial at similar
conferences in the future.

###  2.5. Tutorial Material

We will provide all the slides as well as a curated reading list covering
relevant literature to participants. The notebooks used for the hands-on
session will additionally be made available, all hosted in a public GitHub
repository. If possible, we would like to record our tutorial and share the
recordings with interested participants afterwards.

###  2.6. Equipment

No additional equipment will be necessary—but it would be preferable if
recording equipment is available in the room. Otherwise, we will use a
recording setup with Zoom.

###  2.7. Video Teaser

The video teaser can be found at https://youtu.be/lHva_kgRqq4.

###  2.8. Organisation Details

We will host all materials on a publicly available GitHub repository. This
includes the slides that will be covered during the tutorial, as well as the
notebooks for the hands-on sessions. Because the material will be self-
contained and the notebooks will run on Google Colaboratory, this lends itself
to an asynchronous setup where interested participants can go through the
materials at their own pace at a later date. Additionally, we wish to record
the presentations so they can be distributed afterwards.

##  3\. Organiser Biographies

The workshop organisers and their biographies are listed here in alphabetical
order:

Bram van den Akker:

<EMAIL_ADDRESS>is a Senior Machine Learning Scientist at
Booking.com with a background in Computer Science and Artificial Intelligence
from the University of Amsterdam. At Booking.com, Bram’s work focuses on
bridging the gap between applied research and practical requirements for
Bandits all across the company. Previously, Bram has held positions at Shopify
& Panasonic, and has peer reviewed contributions to conferences and workshops
such as TheWebConf, RecSys, and KDD of which one has been awarded with a best-
paper award (van den Akker et al., 2019).

Olivier Jeunen:

<EMAIL_ADDRESS>is a Lead Decision Scientist at ShareChat with a PhD
from the University of Antwerp (Jeunen, 2021), having previously held
positions at Amazon, Spotify, Facebook and Criteo. Olivier’s research focuses
on applying ideas from causal and counterfactual inference to recommendation
and advertising problems, which have led to 20+ peer reviewed contributions to
top-tier journals, conferences, and workshops at NeurIPS, KDD, RecSys and
WSDM, of which two have been recognised with best paper awards (Jeunen and
Goethals, 2021; Jeunen et al., 2022b). He is an active PC member for KDD,
RecSys, The WebConf, WSDM and SIGIR, whilst reviewing for several journals and
workshops—which has led to two outstanding reviewer awards. He currently
serves as co-Web Chair for RecSys, co-chaired the Dutch-Belgian Information
Retrieval Workshop ’20 and the CONSEQUENCES Workshop at RecSys ’22 (Jeunen et
al., 2022a), and co-lectured tutorials at the RecSys Summer School ’19, UMAP
’20 (Vasile et al., 2020) and The WebConf ’21 (Vasile et al., 2021).

Ying Li:

<EMAIL_ADDRESS>is a Senior Research Scientist at Netflix. She obtained
her Ph.D. from the University of California, Los Angeles, and B.S. from Peking
University. At Netflix, her research interest focuses on large-scale search
and recommendation systems, bandit and long-term user satisfaction
optimization. Prior to Netflix, she was an Applied Scientist in Amazon,
focusing on cold-start classification and large-scale extreme classification
using NLP. She has co-chaired the REVEAL (Reinforcement learning-based
recommender systems at scale) workshop at RecSys 2022 (Liaw et al., 2022).

Ben London:

<EMAIL_ADDRESS>is a Sr. Scientist at Amazon Music. He earned his Ph.D.
in 2015 at the University of Maryland, where he was advised by Lise Getoor and
worked closely with Ben Taskar and Bert Huang. His research investigates
machine learning theory and algorithms, with a focus on generalization
guarantees, structured prediction, recommendation, contextual bandits, and
evaluation/learning with logged bandit feedback. His work has been published
in JMLR, ICML, NeurIPS, AISTATS, UAI and RecSys. He was a co-organizer of the
NeurIPS 2019 Workshop on ML with Guarantees, an area chair for ICML (2020,
2022) and NeurIPS (2020, 2021), a senior PC member for IJCAI, and has reviewed
for numerous conferences and journals.

Zahra Nazari:

<EMAIL_ADDRESS>is a Sr. Scientist at Spotify working on the design and
evaluation of recommender systems with a focus on sequential and long term
optimization using reinforcement learning. Prior to Spotify, she had held
positions at Google, Twitter and Framehawk and earned her Ph.D. at the
University of Southern California with a focus on preference elicitation and
human behaviour modelling in complex situations such as negotiations. She has
published 20+ papers in top conferences such as IJCAI, AAMAS, SIGIR, WSDM and
The Web Conference. Zahra is serving as the program committee member in
conferences such as SIGIR, The Web Conference, CIKM and has co-organized the
workshop on Multi-Objective Recommender Systems in Recsys for two consecutive
years (2021 and 2022).

Devesh Parekh:

<EMAIL_ADDRESS>is a Staff Research Engineer at Netflix working on
simplifying the deployment of contextual bandit solutions to new problems in
member-facing applications. Devesh has previously worked on causal inference
and contextual bandit problems in the areas of programmatic marketing, signup
optimization, and member engagement messaging at Netflix.

## References

  * (1)
  * Auer et al. (2002) Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. 2002\.  Finite-time Analysis of the Multiarmed Bandit Problem.  _Machine Learning_ 47, 2 (2002), 235–256.
  * Bendada et al. (2020) W. Bendada, G. Salha, and T. Bontempelli. 2020.  Carousel Personalization in Music Streaming Apps with Contextual Bandits. In _Proc. of the Fourteenth ACM Conference on Recommender Systems_ _(RecSys ’20)_. ACM, 420–425.
  * Cesa-Bianchi et al. (2017) Nicolò Cesa-Bianchi, Claudio Gentile, Gábor Lugosi, and Gergely Neu. 2017.  Boltzmann exploration done right.  _Advances in neural information processing systems_ 30 (2017).
  * Cesa-Bianchi and Lugosi (2006) Nicolo Cesa-Bianchi and Gábor Lugosi. 2006.  _Prediction, learning, and games_.  Cambridge university press.
  * Chapelle and Li (2011) O. Chapelle and L. Li. 2011\.  An Empirical Evaluation of Thompson Sampling. In _Proc. of the 24th International Conference on Neural Information Processing Systems_ _(NIPS’11)_. 2249–2257.
  * Drugan and Nowe (2013) Madalina M Drugan and Ann Nowe. 2013.  Designing multi-objective multi-armed bandits algorithms: A study. In _The 2013 International Joint Conference on Neural Networks (IJCNN)_. IEEE, 1–8.
  * Garivier and Kaufmann (2016) Aurélien Garivier and Emilie Kaufmann. 2016.  Optimal best arm identification with fixed confidence. In _Conference on Learning Theory_. PMLR, 998–1027.
  * Gruson et al. (2019) A. Gruson, P. Chandar, C. Charbuillet, J. McInerney, S. Hansen, D. Tardieu, and B. Carterette. 2019.  Offline Evaluation to Make Decisions About Playlist Recommendation Algorithms. In _Proceedings of the 12th ACM International Conference on Web Search and Data Mining_ _(WSDM ’19)_. ACM, 420–428.
  * Jagerman et al. (2020) Rolf Jagerman, Ilya Markov, and Maarten De Rijke. 2020\.  Safe Exploration for Optimizing Contextual Bandits.  _ACM Trans. Inf. Syst._ 38, 3, Article 24 (apr 2020), 23 pages.  https://doi.org/10.1145/3385670
  * Jamieson et al. (2014) Kevin Jamieson, Matthew Malloy, Robert Nowak, and Sébastien Bubeck. 2014.  lil’ucb: An optimal exploration algorithm for multi-armed bandits. In _Conference on Learning Theory_. PMLR, 423–439.
  * Jeunen (2021) O. Jeunen. 2021\.  _Offline Approaches to Recommendation with Online Success_.  Ph. D. Dissertation. University of Antwerp.
  * Jeunen and Goethals (2021) O. Jeunen and B. Goethals. 2021.  Pessimistic Reward Models for Off-Policy Learning in Recommendation. In _Proc. of the Fifteenth ACM Conference on Recommender Systems_ _(RecSys ’21)_. ACM, 63–74.
  * Jeunen et al. (2022a) O. Jeunen, T. Joachims, H. Oosterhuis, Y. Saito, and F. Vasile. 2022a.  CONSEQUENCES — Causality, Counterfactuals and Sequential Decision-Making for Recommender Systems. In _Proc. of the 16th ACM Conference on Recommender Systems_ _(RecSys ’22)_. ACM, 654–657.
  * Jeunen et al. (2022b) O. Jeunen, S. Murphy, and B. Allison. 2022b.  Learning to Bid with AuctionGym. In _Proc. of the AdKDD Workshop at the 28th ACM SIGKDD Conference on Knowledge Discovery & Data Mining_ _(AdKDD ’22)_. ACM.
  * Joachims et al. (2018) T. Joachims, A. Swaminathan, and M. de Rijke. 2018\.  Deep Learning with Logged Bandit Feedback. In _Proc. of the 6th International Conference on Learning Representations_ _(ICLR ’18)_.
  * Joachims et al. (2017) Thorsten Joachims, Adith Swaminathan, and Tobias Schnabel. 2017\.  Unbiased learning-to-rank with biased feedback. In _Proceedings of the tenth ACM international conference on web search and data mining_. 781–789.
  * Kiyohara et al. (2022) Haruka Kiyohara, Yuta Saito, Tatsuya Matsuhiro, Yusuke Narita, Nobuyuki Shimizu, and Yasuo Yamamoto. 2022\.  Doubly Robust Off-Policy Evaluation for Ranking Policies under the Cascade Behavior Model. In _Proceedings of the Fifteenth ACM International Conference on Web Search and Data Mining_ _(WSDM ’22)_. ACM, 487–497.  https://doi.org/10.1145/3488560.3498380
  * Ktena et al. (2019) S. Ktena, A. Tejani, L. Theis, P. Myana, D. Dilipkumar, F. Huszár, S. Yoo, and W. Shi. 2019.  Addressing Delayed Feedback for Continuous Training with Neural Networks in CTR Prediction. In _Proc. of the 13th ACM Conference on Recommender Systems_ (Copenhagen, Denmark) _(RecSys ’19)_. ACM, 187–195.
  * Lancewicki et al. (2021) Tal Lancewicki, Shahar Segal, Tomer Koren, and Yishay Mansour. 2021.  Stochastic multi-armed bandits with unrestricted delay distributions. In _International Conference on Machine Learning_. PMLR, 5969–5978.
  * Lattimore and Szepesvári (2020) Tor Lattimore and Csaba Szepesvári. 2020.  _Bandit algorithms_.  Cambridge University Press.
  * Li et al. (2010) L. Li, W. Chu, J. Langford, and R. E. Schapire. 2010\.  A Contextual-Bandit Approach to Personalized News Article Recommendation. In _Proc. of the 19th International Conference on World Wide Web_ _(WWW ’10)_. ACM, 661–670.
  * Liaw et al. (2022) Richard Liaw, Paige Bailey, Ying Li, Maria Dimakopoulou, and Yves Raimond. 2022.  REVEAL 2022: Reinforcement Learning-Based Recommender Systems at Scale. In _Proceedings of the 16th ACM Conference on Recommender Systems_. 684–685.
  * Liu and Li (2021) Yi Liu and Lihong Li. 2021\.  A map of bandits for e-commerce. In _KDD 2021 Workshop on Multi-Armed Bandits and Reinforcement Learning (MARBLE)_.
  * London and Joachims (2020) B. London and T. Joachims. 2020.  Offline Policy Evaluation with New Arms. In _Offline Reinforcement Learning Workshop at Neural Information Processing Systems_.
  * London and Joachims (2022) Ben London and Thorsten Joachims. 2022.  Control Variate Diagnostics for Detecting Problems in Logged Bandit Feedback. In _Proc. of the CONSEQUENCES+REVEAL Workshop at the 16th ACM Conference on Recommender Systems_ _(CONSEQUENCES+REVEAL ’22)_. ACM.
  * Malkov and Yashunin (2020) Y. A. Malkov and D. A. Yashunin. 2020.  Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs.  _IEEE Transactions on Pattern Analysis and Machine Intelligence_ 42, 4 (2020), 824–836.
  * Mehrotra et al. (2020) Rishabh Mehrotra, Niannan Xue, and Mounia Lalmas. 2020\.  Bandit based optimization of multiple objectives on a music streaming platform. In _Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining_. 3224–3233.
  * Robbins (1952) Herbert Robbins. 1952\.  Some aspects of the sequential design of experiments.  _Bull. Amer. Math. Soc._ 58, 5 (1952), 527–535.
  * Saito et al. (2021) Y. Saito, S. Aihara, M. Matsutani, and Y. Narita. 2021\.  Open Bandit Dataset and Pipeline: Towards Realistic and Reproducible Off-Policy Evaluation. In _Proc. of the Neural Information Processing Systems Track on Datasets and Benchmarks_ _(NeurIPS ’21)_.
  * Saito and Joachims (2021) Yuta Saito and Thorsten Joachims. 2021.  Counterfactual learning and evaluation for recommender systems: Foundations, implementations, and recent advances. In _Fifteenth ACM Conference on Recommender Systems_. 828–830.
  * Saito and Joachims (2022) Yuta Saito and Thorsten Joachims. 2022.  Off-Policy Evaluation for Large Action Spaces via Embeddings. In _Proceedings of the 39th International Conference on Machine Learning_ _(Proceedings of Machine Learning Research, Vol. 162)_. PMLR, 19089–19122.
  * Schnabel et al. (2018) Tobias Schnabel, Paul N Bennett, Susan T Dumais, and Thorsten Joachims. 2018.  Short-term satisfaction and long-term coverage: Understanding how users tolerate algorithmic exploration. In _Proceedings of the Eleventh ACM International Conference on Web Search and Data Mining_. 513–521.
  * Sen et al. (2021) Rajat Sen, Alexander Rakhlin, Lexing Ying, Rahul Kidambi, Dean Foster, Daniel N Hill, and Inderjit S Dhillon. 2021.  Top-k extreme contextual bandits with arm hierarchy. In _International Conference on Machine Learning_. PMLR, 9422–9433.
  * Silver et al. (2018) David Silver, Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Matthew Lai, Arthur Guez, Marc Lanctot, Laurent Sifre, Dharshan Kumaran, Thore Graepel, Timothy Lillicrap, Karen Simonyan, and Demis Hassabis. 2018.  A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play.  _Science_ 362, 6419 (2018), 1140–1144.
  * Slivkins (2019) Aleksandrs Slivkins. 2019\.  Introduction to Multi-Armed Bandits.  _Found. Trends Mach. Learn._ 12, 1-2 (2019), 1–286.  https://doi.org/10.1561/2200000068
  * Sutton and Barto (1998) R. S. Sutton and A. G. Barto. 1998.  _Introduction to reinforcement learning_. Vol. 135.
  * Swaminathan and Joachims (2015) A. Swaminathan and T. Joachims. 2015.  Counterfactual Risk Minimization: Learning from Logged Bandit Feedback. In _Proc. of the 32nd International Conference on International Conference on Machine Learning_ (Lille, France) _(ICML’15)_. JMLR.org, 814–823.
  * Swaminathan et al. (2017) Adith Swaminathan, Akshay Krishnamurthy, Alekh Agarwal, Miro Dudik, John Langford, Damien Jose, and Imed Zitouni. 2017.  Off-policy evaluation for slate recommendation.  _Advances in Neural Information Processing Systems_ 30 (2017).
  * van den Akker et al. (2019) Bram van den Akker, Ilya Markov, and Maarten de Rijke. 2019\.  ViTOR: learning to rank webpages based on visual features. In _The world wide web conference_. 3279–3285.
  * van den Akker et al. (2022) B. van den Akker, N. Weber, F. Moraes, and D. Goldenberg. 2022\.  Extending Open Bandit Pipeline to Simulate Industry Challenges. In _Proc. of the CONSEQUENCES+REVEAL Workshop at the 16th ACM Conference on Recommender Systems_ _(CONSEQUENCES+REVEAL ’22)_. ACM.
  * Vasile et al. (2020) F. Vasile, D. Rohde, O. Jeunen, and A. Benhalloum. 2020\.  A Gentle Introduction to Recommendation as Counterfactual Policy Learning. In _Proc. of the 28th ACM Conference on User Modeling, Adaptation and Personalization_ _(UMAP ’20)_. ACM, 392–393.
  * Vasile et al. (2021) F. Vasile, D. Rohde, O. Jeunen, A. Benhalloum, and O. Sakhi. 2021.  Recommender Systems through the Lens of Decision Theory. In _Companion Proceedings of the Web Conference 2021_ _(WWW ’21)_. ACM, 727–733.
  * Vinyals et al. (2019) Oriol Vinyals, Igor Babuschkin, Wojciech M. Czarnecki, Michaël Mathieu, Andrew Dudzik, Junyoung Chung, David H. Choi, Richard Powell, Timo Ewalds, Petko Georgiev, Junhyuk Oh, Dan Horgan, Manuel Kroiss, Ivo Danihelka, Aja Huang, Laurent Sifre, Trevor Cai, John P. Agapiou, Max Jaderberg, Alexander S. Vezhnevets, Rémi Leblond, Tobias Pohlen, Valentin Dalibard, David Budden, Yury Sulsky, James Molloy, Tom L. Paine, Caglar Gulcehre, Ziyu Wang, Tobias Pfaff, Yuhuai Wu, Roman Ring, Dani Yogatama, Dario Wünsch, Katrina McKinney, Oliver Smith, Tom Schaul, Timothy Lillicrap, Koray Kavukcuoglu, Demis Hassabis, Chris Apps, and David Silver. 2019.  Grandmaster level in StarCraft II using multi-agent reinforcement learning.  _Nature_ 575, 7782 (2019), 350–354.
  * Wang et al. (2016) Xuanhui Wang, Michael Bendersky, Donald Metzler, and Marc Najork. 2016.  Learning to rank with selection bias in personal search. In _Proceedings of the 39th International ACM SIGIR conference on Research and Development in Information Retrieval_. 115–124.
  * Wu et al. (2017) Qingyun Wu, Hongning Wang, Liangjie Hong, and Yue Shi. 2017\.  Returning is believing: Optimizing long-term user engagement in recommender systems. In _Proceedings of the 2017 ACM on Conference on Information and Knowledge Management_. 1927–1936.